OpenRaiser/Intern-Atlas huggingface dataset:
Fairness in Contextual Resource Allocation Systems: Metrics and Incompatibility Results
Venue: arXiv cs.CY 2022
Year: 2022

We study critical systems that allocate scarce resources to satisfy basic needs, such as homeless services that provide housing. These systems often support communities disproportionately affected by systemic racial, gender, or other injustices, so it is crucial to design these systems with fairness considerations in mind. To address this problem, we propose a framework for evaluating fairness in contextual resource allocation systems that is inspired by fairness metrics in machine learning. This framework can be applied to evaluate the fairness properties of a historical policy, as well as to impose constraints in the design of new (counterfactual) allocation policies. Our work culminates with a set of incompatibility results that investigate the interplay between the different fairness metrics we propose. Notably, we demonstrate that: 1) fairness in allocation and fairness in outcomes are usually incompatible; 2) policies that prioritize based on a vulnerability score will usually result in unequal outcomes across groups, even if the score is perfectly calibrated; 3) policies using contextual information beyond what is needed to characterize baseline risk and treatment effects can be fairer in their outcomes than those using just baseline risk and treatment effects; and 4) policies using group status in addition to baseline risk and treatment effects are as fair as possible given all available information. Our framework can help guide the discussion among stakeholders in deciding which fairness metrics to impose when allocating scarce resources.
Statistical Physics of Deep Neural Networks: Initialization toward Optimal Channels
Venue: arXiv cs.LG 2022
Year: 2022

In deep learning, neural networks serve as noisy channels between input data and its representation. This perspective naturally relates deep learning with the pursuit of constructing channels with optimal performance in information transmission and representation. While considerable efforts are concentrated on realizing optimal channel properties during network optimization, we study a frequently overlooked possibility that neural networks can be initialized toward optimal channels. Our theory, consistent with experimental validation, identifies primary mechanics underlying this unknown possibility and suggests intrinsic connections between statistical physics and deep learning. Unlike the conventional theories that characterize neural networks applying the classic mean-filed approximation, we offer analytic proof that this extensively applied simplification scheme is not valid in studying neural networks as information channels. To fill this gap, we develop a corrected mean-field framework applicable for characterizing the limiting behaviors of information propagation in neural networks without strong assumptions on inputs. Based on it, we propose an analytic theory to prove that mutual information maximization is realized between inputs and propagated signals when neural networks are initialized at dynamic isometry, a case where information transmits via norm-preserving mappings. These theoretical predictions are validated by experiments on real neural networks, suggesting the robustness of our theory against finite-size effects. Finally, we analyze our findings with information bottleneck theory to confirm the precise relations among dynamic isometry, mutual information maximization, and optimal channel properties in deep learning.
Controllable Image Captioning via Prompting
Venue: arXiv cs.CV 2022
Year: 2022

Despite the remarkable progress of image captioning, existing captioners typically lack the controllable capability to generate desired image captions, e.g., describing the image in a rough or detailed manner, in a factual or emotional view, etc. In this paper, we show that a unified model is qualified to perform well in diverse domains and freely switch among multiple styles. Such a controllable capability is achieved by embedding the prompt learning into the image captioning framework. To be specific, we design a set of prompts to fine-tune the pre-trained image captioner. These prompts allow the model to absorb stylized data from different domains for joint training, without performance degradation in each domain. Furthermore, we optimize the prompts with learnable vectors in the continuous word embedding space, avoiding the heuristic prompt engineering and meanwhile exhibiting superior performance. In the inference stage, our model is able to generate desired stylized captions by choosing the corresponding prompts. Extensive experiments verify the controllable capability of the proposed method. Notably, we achieve outstanding performance on two diverse image captioning benchmarks including COCO Karpathy split and TextCaps using a unified model.
Clustering Permutations: New Techniques with Streaming Applications
Venue: arXiv cs.DS 2022
Year: 2022

We study the classical metric $k$-median clustering problem over a set of input rankings (i.e., permutations), which has myriad applications, from social-choice theory to web search and databases. A folklore algorithm provides a $2$-approximate solution in polynomial time for all $k=O(1)$, and works irrespective of the underlying distance measure, so long it is a metric; however, going below the $2$-factor is a notorious challenge. We consider the Ulam distance, a variant of the well-known edit-distance metric, where strings are restricted to be permutations. For this metric, Chakraborty, Das, and Krauthgamer [SODA, 2021] provided a $(2-δ)$-approximation algorithm for $k=1$, where $δ\approx 2^{-40}$. Our primary contribution is a new algorithmic framework for clustering a set of permutations. Our first result is a $1.999$-approximation algorithm for the metric $k$-median problem under the Ulam metric, that runs in time $(k \log (nd))^{O(k)}n d^3$ for an input consisting of $n$ permutations over $[d]$. In fact, our framework is powerful enough to extend this result to the streaming model (where the $n$ input permutations arrive one by one) using only polylogarithmic (in $n$) space. Additionally, we show that similar results can be obtained even in the presence of outliers, which is presumably a more difficult problem.
Melody transcription via generative pre-training
Venue: arXiv cs.SD 2022
Year: 2022

Despite the central role that melody plays in music perception, it remains an open challenge in music information retrieval to reliably detect the notes of the melody present in an arbitrary music recording. A key challenge in melody transcription is building methods which can handle broad audio containing any number of instrument ensembles and musical styles - existing strategies work well for some melody instruments or styles but not all. To confront this challenge, we leverage representations from Jukebox (Dhariwal et al. 2020), a generative model of broad music audio, thereby improving performance on melody transcription by $20$% relative to conventional spectrogram features. Another obstacle in melody transcription is a lack of training data - we derive a new dataset containing $50$ hours of melody transcriptions from crowdsourced annotations of broad music. The combination of generative pre-training and a new dataset for this task results in $77$% stronger performance on melody transcription relative to the strongest available baseline. By pairing our new melody transcription approach with solutions for beat detection, key estimation, and chord recognition, we build Sheet Sage, a system capable of transcribing human-readable lead sheets directly from music audio. Audio examples can be found at https://chrisdonahue.com/sheetsage and code at https://github.com/chrisdonahue/sheetsage .
Joint Self-Supervised Image-Volume Representation Learning with Intra-Inter Contrastive Clustering
Venue: arXiv cs.CV 2022
Year: 2022

Collecting large-scale medical datasets with fully annotated samples for training of deep networks is prohibitively expensive, especially for 3D volume data. Recent breakthroughs in self-supervised learning (SSL) offer the ability to overcome the lack of labeled training samples by learning feature representations from unlabeled data. However, most current SSL techniques in the medical field have been designed for either 2D images or 3D volumes. In practice, this restricts the capability to fully leverage unlabeled data from numerous sources, which may include both 2D and 3D data. Additionally, the use of these pre-trained networks is constrained to downstream tasks with compatible data dimensions. In this paper, we propose a novel framework for unsupervised joint learning on 2D and 3D data modalities. Given a set of 2D images or 2D slices extracted from 3D volumes, we construct an SSL task based on a 2D contrastive clustering problem for distinct classes. The 3D volumes are exploited by computing vectored embedding at each slice and then assembling a holistic feature through deformable self-attention mechanisms in Transformer, allowing incorporating long-range dependencies between slices inside 3D volumes. These holistic features are further utilized to define a novel 3D clustering agreement-based SSL task and masking embedding prediction inspired by pre-trained language models. Experiments on downstream tasks, such as 3D brain segmentation, lung nodule detection, 3D heart structures segmentation, and abnormal chest X-ray detection, demonstrate the effectiveness of our joint 2D and 3D SSL approach. We improve plain 2D Deep-ClusterV2 and SwAV by a significant margin and also surpass various modern 2D and 3D SSL approaches.
Graph Representation Learning for Wireless Communications
Venue: arXiv cs.IT 2022
Year: 2022

Wireless networks are inherently graph-structured, which can be utilized in graph representation learning to solve complex wireless network optimization problems. In graph representation learning, feature vectors for each entity in the network are calculated such that they capture spatial and temporal dependencies in their local and global neighbourhoods. Graph neural networks (GNNs) are powerful tools to solve these complex problems because of their expressive representation and reasoning power. In this paper, the potential of graph representation learning and GNNs in wireless networks is presented. An overview of graph learning is provided which covers the fundamentals and concepts such as feature design over graphs, GNNs, and their design principles. Potential of graph representation learning in wireless networks is presented via few exemplary use cases and some initial results on the GNN-based access point selection for cell-free massive MIMO systems.
6G Digital Twin Networks: From Theory to Practice
Venue: arXiv cs.NI 2022
Year: 2022

Digital twin networks (DTNs) are real-time replicas of physical networks. They are emerging as a powerful technology for design, diagnosis, simulation, what-if-analysis, and artificial intelligence (AI)/machine learning (ML) driven real-time optimization and control of the sixth-generation (6G) wireless networks. Despite the great potential of what digital twins can offer for 6G, realizing the desired capabilities of 6G DTNs requires tackling many design aspects including data, models, and interfaces. In this article, we provide an overview of 6G DTNs by presenting prominent use cases and their service requirements, describing a reference architecture, and discussing fundamental design aspects. We also present a real-world example to illustrate how DTNs can be built upon and operated in a real-time reference development platform - Omniverse.
Empirical Study of Co-Renamed Identifiers
Venue: arXiv cs.SE 2022
Year: 2022

Background: The renaming of program identifiers is the most common refactoring operation. Because some identifiers are related to each other, developers may need to rename related identifiers together. Aims: To understand how developers rename multiple identifiers simultaneously, it is necessary to consider the relationships between identifiers in the program and the brief matching for non-identical but semantically similar identifiers. Method: We investigate the relationships between co-renamed identifiers and identify the types of their relationships that contribute to improving the recommendation using more than 1M of renaming instances collected from the histories of open-source software projects. We also evaluate and compare the impact of co-renaming and the relationships between identifiers when inflections occur in the words in identifiers are taken into account. Results: We revealed several relationships of identifiers that are frequently found in the co-renamed identifiers, such as the identifiers of methods in the same class or an identifier defining a variable and another used for initializing the variable, depending on the type of the renamed identifiers. Additionally, the consideration of inflections did not affect the tendency of the relationships. Conclusion: These results suggest an approach that prioritizes the identifiers to be recommended depending on their types and the type of the renamed identifier.
RGB-L: Enhancing Indirect Visual SLAM using LiDAR-based Dense Depth Maps
Venue: arXiv cs.RO 2022
Year: 2022

In this paper, we present a novel method for integrating 3D LiDAR depth measurements into the existing ORB-SLAM3 by building upon the RGB-D mode. We propose and compare two methods of depth map generation: conventional computer vision methods, namely an inverse dilation operation, and a supervised deep learning-based approach. We integrate the former directly into the ORB-SLAM3 framework by adding a so-called RGB-L (LiDAR) mode that directly reads LiDAR point clouds. The proposed methods are evaluated on the KITTI Odometry dataset and compared to each other and the standard ORB-SLAM3 stereo method. We demonstrate that, depending on the environment, advantages in trajectory accuracy and robustness can be achieved. Furthermore, we demonstrate that the runtime of the ORB-SLAM3 algorithm can be reduced by more than 40 % compared to the stereo mode. The related code for the ORB-SLAM3 RGB-L mode will be available as open-source software under https://github.com/TUMFTM/ORB SLAM3 RGBL.
Accelerating Interactive Human-like Manipulation Learning with GPU-based Simulation and High-quality Demonstrations
Venue: arXiv cs.RO 2022
Year: 2022

Dexterous manipulation with anthropomorphic robot hands remains a challenging problem in robotics because of the high-dimensional state and action spaces and complex contacts. Nevertheless, skillful closed-loop manipulation is required to enable humanoid robots to operate in unstructured real-world environments. Reinforcement learning (RL) has traditionally imposed enormous interaction data requirements for optimizing such complex control problems. We introduce a new framework that leverages recent advances in GPU-based simulation along with the strength of imitation learning in guiding policy search towards promising behaviors to make RL training feasible in these domains. To this end, we present an immersive virtual reality teleoperation interface designed for interactive human-like manipulation on contact rich tasks and a suite of manipulation environments inspired by tasks of daily living. Finally, we demonstrate the complementary strengths of massively parallel RL and imitation learning, yielding robust and natural behaviors. Videos of trained policies, our source code, and the collected demonstration datasets are available at https://maltemosbach.github.io/interactive_ human_like_manipulation/.
Wearable-based Human Activity Recognition with Spatio-Temporal Spiking Neural Networks
Venue: arXiv cs.NE 2022
Year: 2022

We study the Human Activity Recognition (HAR) task, which predicts user daily activity based on time series data from wearable sensors. Recently, researchers use end-to-end Artificial Neural Networks (ANNs) to extract the features and perform classification in HAR. However, ANNs pose a huge computation burden on wearable devices and lack temporal feature extraction. In this work, we leverage Spiking Neural Networks (SNNs)--an architecture inspired by biological neurons--to HAR tasks. SNNs allow spatio-temporal extraction of features and enjoy low-power computation with binary spikes. We conduct extensive experiments on three HAR datasets with SNNs, demonstrating that SNNs are on par with ANNs in terms of accuracy while reducing up to 94% energy consumption. The code is publicly available in https://github.com/Intelligent-Computing-Lab-Yale/SNN_HAR
A Hybrid Statistical-Machine Learning Approach for Analysing Online Customer Behavior: An Empirical Study
Venue: arXiv cs.LG 2022
Year: 2022

We apply classical statistical methods in conjunction with the state-of-the-art machine learning techniques to develop a hybrid interpretable model to analyse 454,897 online customers' behavior for a particular product category at the largest online retailer in China, that is JD. While most mere machine learning methods are plagued by the lack of interpretability in practice, our novel hybrid approach will address this practical issue by generating explainable output. This analysis involves identifying what features and characteristics have the most significant impact on customers' purchase behavior, thereby enabling us to predict future sales with a high level of accuracy, and identify the most impactful variables. Our results reveal that customers' product choice is insensitive to the promised delivery time, but this factor significantly impacts customers' order quantity. We also show that the effectiveness of various discounting methods depends on the specific product and the discount size. We identify product classes for which certain discounting approaches are more effective and provide recommendations on better use of different discounting tools. Customers' choice behavior across different product classes is mostly driven by price, and to a lesser extent, by customer demographics. The former finding asks for exercising care in deciding when and how much discount should be offered, whereas the latter identifies opportunities for personalized ads and targeted marketing. Further, to curb customers' batch ordering behavior and avoid the undesirable Bullwhip effect, JD should improve its logistics to ensure faster delivery of orders.
Counteracting Eavesdropper Attacks Through Reconfigurable Intelligent Surfaces: A New Threat Model and Secrecy Rate Optimization
Venue: arXiv cs.IT 2022
Year: 2022

The potential of Reconfigurable Intelligent Surfaces (RISs) for energy-efficient and performance-boosted wireless communications is recently gaining remarkable research attention, motivating their consideration for various $5$-th Generation (5G) Advanced and beyond applications. In this paper, we consider a Multiple-Input Multiple-Output (MIMO) Physical Layer Security (PLS) system with multiple data streams including one legitimate passive RIS and one malicious passive RIS, with the former being transparent to the multi-antenna eavesdropper and the latter's presence being unknown at the legitimate multi-antenna transceivers. We first present a novel threat model for the RIS-boosted eavesdropping system and design a joint optimization framework for the eavesdropper's receive combining matrix and the reflection coefficients of the malicious RIS. Focusing next on the secrecy rate maximization problem, we present an RIS-empowered PLS scheme that jointly designs the legitimate precoding matrix and number of data streams, the Artificial Noise (AN) covariance matrix, the receive combining matrix, and the reflection coefficients of the legitimate RIS. The proposed optimization algorithms, whose convergence to at least local optimum points is proved, are based on alternating maximization, minorization-maximization, and manifold optimization, including semi-closed form expressions for the optimization variables. Our extensive simulation results for two representative system setups reveal that, in the absence of a legitimate RIS, transceiver spatial filtering and AN are incapable of offering non-zero secrecy rates, even for malicious RISs with small numbers of elements. However, when an $L$-element legitimate RIS is deployed, confidential communication can be safeguarded against eavesdropping systems possessing even more than a $5L$-element malicious RIS.
Lossy Compression for Robust Unsupervised Time-Series Anomaly Detection
Venue: arXiv cs.LG 2022
Year: 2022

A new Lossy Causal Temporal Convolutional Neural Network Autoencoder for anomaly detection is proposed in this work. Our framework uses a rate-distortion loss and an entropy bottleneck to learn a compressed latent representation for the task. The main idea of using a rate-distortion loss is to introduce representation flexibility that ignores or becomes robust to unlikely events with distinctive patterns, such as anomalies. These anomalies manifest as unique distortion features that can be accurately detected in testing conditions. This new architecture allows us to train a fully unsupervised model that has high accuracy in detecting anomalies from a distortion score despite being trained with some portion of unlabelled anomalous data. This setting is in stark contrast to many of the state-of-the-art unsupervised methodologies that require the model to be only trained on "normal data". We argue that this partially violates the concept of unsupervised training for anomaly detection as the model uses an informed decision that selects what is normal from abnormal for training. Additionally, there is evidence to suggest it also effects the models ability at generalisation. We demonstrate that models that succeed in the paradigm where they are only trained on normal data fail to be robust when anomalous data is injected into the training. In contrast, our compression-based approach converges to a robust representation that tolerates some anomalous distortion. The robust representation achieved by a model using a rate-distortion loss can be used in a more realistic unsupervised anomaly detection scheme.
Improved Convergence Guarantees for Shallow Neural Networks
Venue: arXiv cs.LG 2022
Year: 2022

We continue a long line of research aimed at proving convergence of depth 2 neural networks, trained via gradient descent, to a global minimum. Like in many previous works, our model has the following features: regression with quadratic loss function, fully connected feedforward architecture, RelU activations, Gaussian data instances and network initialization, adversarial labels. It is more general in the sense that we allow both layers to be trained simultaneously and at {\em different} rates. Our results improve on state-of-the-art [Oymak Soltanolkotabi 20] (training the first layer only) and [Nguyen 21, Section 3.2] (training both layers with Le Cun's initialization). We also report several simple experiments with synthetic data. They strongly suggest that, at least in our model, the convergence phenomenon extends well beyond the ``NTK regime''.