OpenRaiser/Intern-Atlas huggingface dataset:
NEV-NCD: Negative Learning, Entropy, and Variance regularization based novel action categories discovery
Venue: arXiv cs.CV 2023
Year: 2023

Novel Categories Discovery (NCD) facilitates learning from a partially annotated label space and enables deep learning (DL) models to operate in an open-world setting by identifying and differentiating instances of novel classes based on the labeled data notions. One of the primary assumptions of NCD is that the novel label space is perfectly disjoint and can be equipartitioned, but it is rarely realized by most NCD approaches in practice. To better align with this assumption, we propose a novel single-stage joint optimization-based NCD method, Negative learning, Entropy, and Variance regularization NCD (NEV-NCD). We demonstrate the efficacy of NEV-NCD in previously unexplored NCD applications of video action recognition (VAR) with the public UCF101 dataset and a curated in-house partial action-space annotated multi-view video dataset. We perform a thorough ablation study by varying the composition of final joint loss and associated hyper-parameters. During our experiments with UCF101 and multi-view action dataset, NEV-NCD achieves ~ 83% classification accuracy in test instances of labeled data. NEV-NCD achieves ~ 70% clustering accuracy over unlabeled data outperforming both naive baselines (by ~ 40%) and state-of-the-art pseudo-labeling-based approaches (by ~ 3.5%) over both datasets. Further, we propose to incorporate optional view-invariant feature learning with the multiview dataset to identify novel categories from novel viewpoints. Our additional view-invariance constraint improves the discriminative accuracy for both known and unknown categories by ~ 10% for novel viewpoints.
Exact Subspace Diffusion for Decentralized Multitask Learning
Venue: arXiv cs.LG 2023
Year: 2023

Classical paradigms for distributed learning, such as federated or decentralized gradient descent, employ consensus mechanisms to enforce homogeneity among agents. While these strategies have proven effective in i.i.d. scenarios, they can result in significant performance degradation when agents follow heterogeneous objectives or data. Distributed strategies for multitask learning, on the other hand, induce relationships between agents in a more nuanced manner, and encourage collaboration without enforcing consensus. We develop a generalization of the exact diffusion algorithm for subspace constrained multitask learning over networks, and derive an accurate expression for its mean-squared deviation when utilizing noisy gradient approximations. We verify numerically the accuracy of the predicted performance expressions, as well as the improved performance of the proposed approach over alternatives based on approximate projections.
PTW: Pivotal Tuning Watermarking for Pre-Trained Image Generators
Venue: arXiv cs.LG 2023
Year: 2023

Deepfakes refer to content synthesized using deep generators, which, when misused, have the potential to erode trust in digital media. Synthesizing high-quality deepfakes requires access to large and complex generators only a few entities can train and provide. The threat is malicious users that exploit access to the provided model and generate harmful deepfakes without risking detection. Watermarking makes deepfakes detectable by embedding an identifiable code into the generator that is later extractable from its generated images. We propose Pivotal Tuning Watermarking (PTW), a method for watermarking pre-trained generators (i) three orders of magnitude faster than watermarking from scratch and (ii) without the need for any training data. We improve existing watermarking methods and scale to generators $4 \times$ larger than related work. PTW can embed longer codes than existing methods while better preserving the generator's image quality. We propose rigorous, game-based definitions for robustness and undetectability, and our study reveals that watermarking is not robust against an adaptive white-box attacker who controls the generator's parameters. We propose an adaptive attack that can successfully remove any watermarking with access to only 200 non-watermarked images. Our work challenges the trustworthiness of watermarking for deepfake detection when the parameters of a generator are available. The source code to reproduce our experiments is available at https://github.com/nilslukas/gan-watermark.
CoMaL: Conditional Maximum Likelihood Approach to Self-supervised Domain Adaptation in Long-tail Semantic Segmentation
Venue: arXiv cs.CV 2023
Year: 2023

The research in self-supervised domain adaptation in semantic segmentation has recently received considerable attention. Although GAN-based methods have become one of the most popular approaches to domain adaptation, they have suffered from some limitations. They are insufficient to model both global and local structures of a given image, especially in small regions of tail classes. Moreover, they perform bad on the tail classes containing limited number of pixels or less training samples. In order to address these issues, we present a new self-supervised domain adaptation approach to tackle long-tail semantic segmentation in this paper. Firstly, a new metric is introduced to formulate long-tail domain adaptation in the segmentation problem. Secondly, a new Conditional Maximum Likelihood (CoMaL) approach in an autoregressive framework is presented to solve the problem of long-tail domain adaptation. Although other segmentation methods work under the pixel independence assumption, the long-tailed pixel distributions in CoMaL are generally solved in the context of structural dependency, as that is more realistic. Finally, the proposed method is evaluated on popular large-scale semantic segmentation benchmarks, i.e., "SYNTHIA to Cityscapes" and "GTA to Cityscapes", and outperforms the prior methods by a large margin in both the standard and the proposed evaluation protocols.
A Reconfigurable Linear RF Analog Processor for Realizing Microwave Artificial Neural Network
Venue: arXiv cs.AR 2023
Year: 2023

Owing to the data explosion and rapid development of artificial intelligence (AI), particularly deep neural networks (DNNs), the ever-increasing demand for large-scale matrix-vector multiplication has become one of the major issues in machine learning (ML). Training and evaluating such neural networks rely on heavy computational resources, resulting in significant system latency and power consumption. To overcome these issues, analog computing using optical interferometric-based linear processors have recently appeared as promising candidates in accelerating matrix-vector multiplication and lowering power consumption. On the other hand, radio frequency (RF) electromagnetic waves can also exhibit similar advantages as the optical counterpart by performing analog computation at light speed with lower power. Furthermore, RF devices have extra benefits such as lower cost, mature fabrication, and analog-digital mixed design simplicity, which has great potential in realizing affordable, scalable, low latency, low power, near-sensor radio frequency neural network (RFNN) that may greatly enrich RF signal processing capability. In this work, we propose a 2X2 reconfigurable linear RF analog processor in theory and experiment, which can be applied as a matrix multiplier in an artificial neural network (ANN). The proposed device can be utilized to realize a 2X2 simple RFNN for data classification. An 8X8 linear analog processor formed by 28 RFNN devices are also applied in a 4-layer ANN for Modified National Institute of Standards and Technology (MNIST) dataset classification.
Proof-of-Turn: Blockchain consensus using a round-robin procedure as one possible solution for cutting costs in mobile games
Venue: arXiv cs.DC 2023
Year: 2023

This master thesis deals with Blockchain Technology in mobile turn based peer to peer games. First, it investigates the capabilities of Blockchain Technology to be used for gaming applications. In this regard, among others, Proof-of-Mechanisms, Vote-based Consensus and several Performance Improvements are described. Second, several smart contracts are introduced to show the general feasibility of turn based games hosted on Blockchain Technology. More specific, Hidden transactions, Randomization, Piles of Cards, Fog of War elements, Data allocation improvements and other smart contracts are specified. Third, a special Proof-of-Turn consensus mechanism, based on the Blockchain Technology, is defined to enable game publishers to cut costs in the means of their provided game servers. Herein, Byzantine Fault Tolerance, Peering, the CAP Theorem, Interoperability among other characteristics are covered. Last, these measures shall additionally raise the trust level among the players in mobile turn based games.
Shape of You: Precise 3D shape estimations for diverse body types
Venue: arXiv cs.CV 2023
Year: 2023

This paper presents Shape of You (SoY), an approach to improve the accuracy of 3D body shape estimation for vision-based clothing recommendation systems. While existing methods have successfully estimated 3D poses, there remains a lack of work in precise shape estimation, particularly for diverse human bodies. To address this gap, we propose two loss functions that can be readily integrated into parametric 3D human reconstruction pipelines. Additionally, we propose a test-time optimization routine that further improves quality. Our method improves over the recent SHAPY method by 17.7% on the challenging SSP-3D dataset. We consider our work to be a step towards a more accurate 3D shape estimation system that works reliably on diverse body types and holds promise for practical applications in the fashion industry.
Text-Conditional Contextualized Avatars For Zero-Shot Personalization
Venue: arXiv cs.CV 2023
Year: 2023

Recent large-scale text-to-image generation models have made significant improvements in the quality, realism, and diversity of the synthesized images and enable users to control the created content through language. However, the personalization aspect of these generative models is still challenging and under-explored. In this work, we propose a pipeline that enables personalization of image generation with avatars capturing a user's identity in a delightful way. Our pipeline is zero-shot, avatar texture and style agnostic, and does not require training on the avatar at all - it is scalable to millions of users who can generate a scene with their avatar. To render the avatar in a pose faithful to the given text prompt, we propose a novel text-to-3D pose diffusion model trained on a curated large-scale dataset of in-the-wild human poses improving the performance of the SOTA text-to-motion models significantly. We show, for the first time, how to leverage large-scale image datasets to learn human 3D pose parameters and overcome the limitations of motion capture datasets.
An elaborated pattern-based method of identifying data oscillations from mobile device location data
Venue: arXiv cs.DB 2023
Year: 2023

In recent years, passively collected GPS data have been popularly applied in various transportation studies, such as highway performance monitoring, travel behavior analysis, and travel demand estimation. Despite multiple advantages, one of the issues is data oscillations (aka outliers or data jumps), which are unneglectable since they may distort mobility patterns and lead to wrongly or biased conclusions. For transportation studies driven by GPS data, assuring the data quality by removing noises caused by data oscillations is undoubtedly important. Most GPS-based studies simply remove oscillations by checking the high speed. However, this method can mistakenly identify normal points as oscillations. Some other studies specifically discuss the removal of outliers in GPS data, but they all have limitations and do not fit passively collected GPS data. Many studies are well developed for addressing the ping-pong phenomenon in cellular data, or cellular tower data, but the oscillations in passively collected GPS data are very different for having much more various and complicated patterns and being more uncertain. Current methods are insufficient and inapplicable to passively collected GPS data. This paper aims to address the oscillated points in passively collected GPS data. A set of heuristics are proposed by identifying the abnormal movement patterns of oscillations. The proposed heuristics well fit the features of passively collected GPS data and are adaptable to studies of different scales, which are also computationally cost-effective in comparison to current methods.
Fully Scalable Massively Parallel Algorithms for Embedded Planar Graphs
Venue: arXiv cs.DS 2023
Year: 2023

We consider the massively parallel computation (MPC) model, which is a theoretical abstraction of large-scale parallel processing models such as MapReduce. In this model, assuming the widely believed 1-vs-2-cycles conjecture, solving many basic graph problems in $O(1)$ rounds with a strongly sublinear memory size per machine is impossible. We improve on the recent work of Holm and Tětek [SODA 2023] that bypass this barrier for problems when a planar embedding of the graph is given. In the previous work, on graphs of size $n$ with $O(n/\mathcal{S})$ machines, the memory size per machine needs to be at least $\mathcal{S} = n^{2/3+Ω(1)}$, whereas we extend their work to the fully scalable regime, where the memory size per machine can be $\mathcal{S} = n^δ$ for any constant $0< δ< 1$. We give the first constant round fully scalable algorithms for embedded planar graphs for the problems of (i) connectivity and (ii) minimum spanning tree (MST). Moreover, we show that the $\varepsilon$-emulator of Chang, Krauthgamer, and Tan [STOC 2022] can be incorporated into our recursive framework to obtain constant-round $(1+\varepsilon)$-approximation algorithms for the problems of computing (iii) single source shortest path (SSSP), (iv) global min-cut, and (v) $st$-max flow. All previous results on cuts and flows required linear memory in the MPC model. Furthermore, our results give new algorithms for problems that implicitly involve embedded planar graphs. We give as corollaries constant round fully scalable algorithms for (vi) 2D Euclidean MST using $O(n)$ total memory and (vii) $(1+\varepsilon)$-approximate weighted edit distance using $\widetilde{O}(n^{2-δ})$ memory. Our main technique is a recursive framework combined with novel graph drawing algorithms to compute smaller embedded planar graphs in constant rounds in the fully scalable setting.
Intent-aware Ranking Ensemble for Personalized Recommendation
Venue: arXiv cs.IR 2023
Year: 2023

Ranking ensemble is a critical component in real recommender systems. When a user visits a platform, the system will prepare several item lists, each of which is generally from a single behavior objective recommendation model. As multiple behavior intents, e.g., both clicking and buying some specific item category, are commonly concurrent in a user visit, it is necessary to integrate multiple single-objective ranking lists into one. However, previous work on rank aggregation mainly focused on fusing homogeneous item lists with the same objective while ignoring ensemble of heterogeneous lists ranked with different objectives with various user intents. In this paper, we treat a user's possible behaviors and the potential interacting item categories as the user's intent. And we aim to study how to fuse candidate item lists generated from different objectives aware of user intents. To address such a task, we propose an Intent-aware ranking Ensemble Learning~(IntEL) model to fuse multiple single-objective item lists with various user intents, in which item-level personalized weights are learned. Furthermore, we theoretically prove the effectiveness of IntEL with point-wise, pair-wise, and list-wise loss functions via error-ambiguity decomposition. Experiments on two large-scale real-world datasets also show significant improvements of IntEL on multiple behavior objectives simultaneously compared to previous ranking ensemble models.
Layph: Making Change Propagation Constraint in Incremental Graph Processing by Layering Graph
Venue: arXiv cs.DC 2023
Year: 2023

Real-world graphs are constantly evolving, which demands updates of the previous analysis results to accommodate graph changes. By using the memoized previous computation state, incremental graph computation can reduce unnecessary recomputation. However, a small change may propagate over the whole graph and lead to large-scale iterative computations. To address this problem, we propose Layph, a two-layered graph framework. The upper layer is a skeleton of the graph, which is much smaller than the original graph, and the lower layer has some disjointed subgraphs. Layph limits costly global iterative computations on the original graph to the small graph skeleton and a few subgraphs updated with the input graph changes. In this way, many vertices and edges are not involved in iterative computations, significantly reducing the communication overhead and improving incremental graph processing performance. Our experimental results show that Layph outperforms current state-of-the-art incremental graph systems by 9.08X on average (up to 36.66X) in response time.
Communication and Energy Efficient Wireless Federated Learning with Intrinsic Privacy
Venue: arXiv cs.LG 2023
Year: 2023

Federated Learning (FL) is a collaborative learning framework that enables edge devices to collaboratively learn a global model while keeping raw data locally. Although FL avoids leaking direct information from local datasets, sensitive information can still be inferred from the shared models. To address the privacy issue in FL, differential privacy (DP) mechanisms are leveraged to provide formal privacy guarantee. However, when deploying FL at the wireless edge with over-the-air computation, ensuring client-level DP faces significant challenges. In this paper, we propose a novel wireless FL scheme called private federated edge learning with sparsification (PFELS) to provide client-level DP guarantee with intrinsic channel noise while reducing communication and energy overhead and improving model accuracy. The key idea of PFELS is for each device to first compress its model update and then adaptively design the transmit power of the compressed model update according to the wireless channel status without any artificial noise addition. We provide a privacy analysis for PFELS and prove the convergence of PFELS under general non-convex and non-IID settings. Experimental results show that compared with prior work, PFELS can improve the accuracy with the same DP guarantee and save communication and energy costs simultaneously.
Generating an interactive online map of future sea level rise along the North Shore of Vancouver: methods and insights on enabling geovisualisation for coastal communities
Venue: arXiv cs.CV 2023
Year: 2023

Contemporary sea level rise (SLR) research seldom considers enabling effective geovisualisation for the communities. This lack of knowledge transfer impedes raising awareness on climate change and its impacts. The goal of this study is to produce an online SLR map accessible to the public that allows them to interact with evolving high-resolution geospatial data and techniques. The study area was the North Shore of Vancouver, British Columbia, Canada. While typically coarser resolution (10m+/pixel) Digital Elevation Models have been used by previous studies, we explored an open access airborne 1 metre LiDAR which has a higher resolution and vertical accuracy and can penetrate tree cover at a higher degree than most satellite imagery. A bathtub method model with hydrologic connectivity was used to delineate the inundation zones for various SLR scenarios which allows for a not overly complex model and process using standard tools such as ArcGIS and QGIS with similar levels of accuracy as more complex models, especially with the high-resolution data. Deep Learning and 3D visualizations were used to create past, present, and modelled future Land Use/Land Cover and 3D flyovers. Analysis of the possible impacts of 1m, 2m, 3m, and 4m SLR over the unique coastline, terrain and land use was detailed. The generated interactive online map helps local communities visualise and understand the future of their coastlines. We have provided a detailed methodology and the methods and results are easily reproducible for other regions. Such initiatives can help popularise community-focused geovisualisation to raise awareness about SLR.
OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantization
Venue: arXiv cs.AR 2023
Year: 2023

Transformer-based large language models (LLMs) have achieved great success with the growing model size. LLMs' size grows by $240\times$ every two years, which outpaces the hardware progress and makes model inference increasingly costly. Model quantization is a promising approach to mitigate the widening gap between LLM size and hardware capacity. However, the existence of outliers, values with significant magnitudes, in LLMs makes existing quantization methods less effective. Prior outlier-aware quantization schemes adopt sparsity encoding techniques to separate outliers from normal values where the process requires global coordination (e.g., a global sparsity coordination list). This incurs complex encoding/decoding hardware logics and an extra orchestration controller for the computation between outlier and normal values. As such, it is not hardware-efficient and hence only achieves sub-optimal quantization benefits. We propose OliVe, an algorithm/architecture co-designed solution that adopts an outlier-victim pair (OVP) quantization and handles outlier values locally with low hardware overheads and high performance gains. The key insight of OliVe is that outliers are important while the normal values next to them are not. Thus those normal values (called victims) can be sacrificed to accommodate outliers. This enables a memory-aligned OVP encoding scheme, which can be efficiently integrated to the existing hardware accelerators like systolic array and tensor core. As a result, OliVe-based accelerator surpasses the existing outlier-aware accelerator, GOBO, by 4.5$\times$ speedup and 4.0$\times$ energy reduction, respectively, with a superior model accuracy.
Hierarchical and Contrastive Representation Learning for Knowledge-aware Recommendation
Venue: arXiv cs.IR 2023
Year: 2023

Incorporating knowledge graph into recommendation is an effective way to alleviate data sparsity. Most existing knowledge-aware methods usually perform recursive embedding propagation by enumerating graph neighbors. However, the number of nodes' neighbors grows exponentially as the hop number increases, forcing the nodes to be aware of vast neighbors under this recursive propagation for distilling the high-order semantic relatedness. This may induce more harmful noise than useful information into recommendation, leading the learned node representations to be indistinguishable from each other, that is, the well-known over-smoothing issue. To relieve this issue, we propose a Hierarchical and CONtrastive representation learning framework for knowledge-aware recommendation named HiCON. Specifically, for avoiding the exponential expansion of neighbors, we propose a hierarchical message aggregation mechanism to interact separately with low-order neighbors and meta-path-constrained high-order neighbors. Moreover, we also perform cross-order contrastive learning to enforce the representations to be more discriminative. Extensive experiments on three datasets show the remarkable superiority of HiCON over state-of-the-art approaches.