OpenRaiser/Intern-Atlas huggingface dataset:
Approximate Inference via Clustering
Venue: arXiv cs.LG 2022
Year: 2022

In recent years, large-scale Bayesian learning draws a great deal of attention. However, in big-data era, the amount of data we face is growing much faster than our ability to deal with it. Fortunately, it is observed that large-scale datasets usually own rich internal structure and is somewhat redundant. In this paper, we attempt to simplify the Bayesian posterior via exploiting this structure. Specifically, we restrict our interest to the so-called well-clustered datasets and construct an \emph{approximate posterior} according to the clustering information. Fortunately, the clustering structure can be efficiently obtained via a particular clustering algorithm. When constructing the approximate posterior, the data points in the same cluster are all replaced by the centroid of the cluster. As a result, the posterior can be significantly simplified. Theoretically, we show that under certain conditions the approximate posterior we construct is close (measured by KL divergence) to the exact posterior. Furthermore, thorough experiments are conducted to validate the fact that the constructed posterior is a good approximation to the true posterior and much easier to sample from.
Schema matching using Gaussian mixture models with Wasserstein distance
Venue: arXiv cs.LG 2022
Year: 2022

Gaussian mixture models find their place as a powerful tool, mostly in the clustering problem, but with proper preparation also in feature extraction, pattern recognition, image segmentation and in general machine learning. When faced with the problem of schema matching, different mixture models computed on different pieces of data can maintain crucial information about the structure of the dataset. In order to measure or compare results from mixture models, the Wasserstein distance can be very useful, however it is not easy to calculate for mixture distributions. In this paper we derive one of possible approximations for the Wasserstein distance between Gaussian mixture models and reduce it to linear problem. Furthermore, application examples concerning real world data are shown.
Hypothesis Testing of Mixture Distributions using Compressed Data
Venue: arXiv cs.IT 2022
Year: 2022

In this paper we revisit the binary hypothesis testing problem with one-sided compression. Specifically we assume that the distribution in the null hypothesis is a mixture distribution of iid components. The distribution under the alternative hypothesis is a mixture of products of either iid distributions or finite order Markov distributions with stationary transition kernels. The problem is studied under the Neyman-Pearson framework in which our main interest is the maximum error exponent of the second type of error. We derive the optimal achievable error exponent and under a further sufficient condition establish the maximum $ε$-achievable error exponent. It is shown that to obtain the latter, the study of the exponentially strong converse is needed. Using a simple code transfer argument we also establish new results for the Wyner-Ahlswede-K{ö}rner problem in which the source distribution is a mixture of iid components.
Collective Intelligence for Deep Learning: A Survey of Recent Developments
Venue: arXiv cs.NE 2022
Year: 2022

In the past decade, we have witnessed the rise of deep learning to dominate the field of artificial intelligence. Advances in artificial neural networks alongside corresponding advances in hardware accelerators with large memory capacity, together with the availability of large datasets enabled practitioners to train and deploy sophisticated neural network models that achieve state-of-the-art performance on tasks across several fields spanning computer vision, natural language processing, and reinforcement learning. However, as these neural networks become bigger, more complex, and more widely used, fundamental problems with current deep learning models become more apparent. State-of-the-art deep learning models are known to suffer from issues that range from poor robustness, inability to adapt to novel task settings, to requiring rigid and inflexible configuration assumptions. Collective behavior, commonly observed in nature, tends to produce systems that are robust, adaptable, and have less rigid assumptions about the environment configuration. Collective intelligence, as a field, studies the group intelligence that emerges from the interactions of many individuals. Within this field, ideas such as self-organization, emergent behavior, swarm optimization, and cellular automata were developed to model and explain complex systems. It is therefore natural to see these ideas incorporated into newer deep learning methods. In this review, we will provide a historical context of neural network research's involvement with complex systems, and highlight several active areas in modern deep learning research that incorporate the principles of collective intelligence to advance its current capabilities. We hope this review can serve as a bridge between the complex systems and deep learning communities.
A new Sinkhorn algorithm with Deletion and Insertion operations
Venue: arXiv cs.LG 2022
Year: 2022

This technical report is devoted to the continuous estimation of an epsilon-assignment. Roughly speaking, an epsilon assignment between two sets V1 and V2 may be understood as a bijective mapping between a sub part of V1 and a sub part of V2 . The remaining elements of V1 (not included in this mapping) are mapped onto an epsilon pseudo element of V2 . We say that such elements are deleted. Conversely, the remaining elements of V2 correspond to the image of the epsilon pseudo element of V1. We say that these elements are inserted. As a result our method provides a result similar to the one of the Sinkhorn algorithm with the additional ability to reject some elements which are either inserted or deleted. It thus naturally handles sets V1 and V2 of different sizes and decides mappings/insertions/deletions in a unified way. Our algorithms are iterative and differentiable and may thus be easily inserted within a backpropagation based learning framework such as artificial neural networks.
DeepGate: Learning Neural Representations of Logic Gates
Venue: arXiv cs.LG 2022
Year: 2022

Applying deep learning (DL) techniques in the electronic design automation (EDA) field has become a trending topic. Most solutions apply well-developed DL models to solve specific EDA problems. While demonstrating promising results, they require careful model tuning for every problem. The fundamental question on "How to obtain a general and effective neural representation of circuits?" has not been answered yet. In this work, we take the first step towards solving this problem. We propose DeepGate, a novel representation learning solution that effectively embeds both logic function and structural information of a circuit as vectors on each gate. Specifically, we propose transforming circuits into unified and-inverter graph format for learning and using signal probabilities as the supervision task in DeepGate. We then introduce a novel graph neural network that uses strong inductive biases in practical circuits as learning priors for signal probability prediction. Our experimental results show the efficacy and generalization capability of DeepGate.
Evaluation of Machine Learning Techniques for Forecast Uncertainty Quantification
Venue: arXiv cs.LG 2022
Year: 2022

Ensemble forecasting is, so far, the most successful approach to produce relevant forecasts with an estimation of their uncertainty. The main limitations of ensemble forecasting are the high computational cost and the difficulty to capture and quantify different sources of uncertainty, particularly those associated with model errors. In this work we perform toy-model and state-of-the-art model experiments to analyze to what extent artificial neural networks (ANNs) are able to model the different sources of uncertainty present in a forecast. In particular those associated with the accuracy of the initial conditions and those introduced by the model error. We also compare different training strategies: one based on a direct training using the mean and spread of an ensemble forecast as target, the other ones rely on an indirect training strategy using an analyzed state as target in which the uncertainty is implicitly learned from the data. Experiments using the Lorenz'96 model show that the ANNs are able to emulate some of the properties of ensemble forecasts like the filtering of the most unpredictable modes and a state-dependent quantification of the forecast uncertainty. Moreover, ANNs provide a reliable estimation of the forecast uncertainty in the presence of model error. Preliminary experiments conducted with a state-of-the-art forecasting system also confirm the ability of ANNs to produce a reliable quantification of the forecast uncertainty.
Verifying Transactional Consistency of MongoDB
Venue: arXiv cs.DC 2022
Year: 2022

MongoDB is a popular general-purpose, document-oriented, distributed NoSQL database. It supports transactions in three different deployments: single-document transactions utilizing the WiredTiger storage engine in a standalone node, multi-document transactions in a replica set which consists of a primary node and several secondary nodes, and distributed transactions in a sharded cluster which is a group of multiple replica sets, among which data is sharded. A natural and fundamental question about MongoDB transactions is: What transactional consistency guarantee do MongoDB Transactions in each deployment provide? However, it lacks both concise pseudocode of MongoDB transactions in each deployment and formal specification of the consistency guarantees which MongoDB claimed to provide. In this work, we formally specify and verify the transactional consistency protocols of MongoDB. Specifically, we provide a concise pseudocode for the transactional consistency protocols in each MongoDB deployment, namely WIREDTIGER, REPLICASET, and SHARDEDCLUSTER, based on the official documents and source code. We then prove that WIREDTIGER, REPLICASET, and SHARDEDCLUSTER satisfy different variants of snapshot isolation, namely Strong-SI, Realtime-SI, and Session-SI, respectively. We also propose and evaluate efficient white-box checking algorithms for MongoDB transaction protocols against their consistency guarantees, effectively circumventing the NP-hard obstacle in theory.
Classification of animal sounds in a hyperdiverse rainforest using Convolutional Neural Networks
Venue: arXiv cs.LG 2022
Year: 2022

To protect tropical forest biodiversity, we need to be able to detect it reliably, cheaply, and at scale. Automated species detection from passively recorded soundscapes via machine-learning approaches is a promising technique towards this goal, but it is constrained by the necessity of large training data sets. Using soundscapes from a tropical forest in Borneo and a Convolutional Neural Network model (CNN) created with transfer learning, we investigate i) the minimum viable training data set size for accurate prediction of call types ('sonotypes'), and ii) the extent to which data augmentation can overcome the issue of small training data sets. We found that even relatively high sample sizes (> 80 per call type) lead to mediocre accuracy, which however improves significantly with data augmentation, including at extremely small sample sizes, regardless of taxonomic group or call characteristics. Our results suggest that transfer learning and data augmentation can make the use of CNNs to classify species' vocalizations feasible even for small soundscape-based projects with many rare species. Retraining our open-source model requires only basic programming skills which makes it possible for individual conservation initiatives to match their local context, in order to enable more evidence-informed management of biodiversity.
Continuous conditional generative adversarial networks for data-driven solutions of poroelasticity with heterogeneous material properties
Venue: arXiv cs.CE 2022
Year: 2022

Machine learning-based data-driven modeling can allow computationally efficient time-dependent solutions of PDEs, such as those that describe subsurface multiphysical problems. In this work, our previous approach of conditional generative adversarial networks (cGAN) developed for the solution of steady-state problems involving highly heterogeneous material properties is extended to time-dependent problems by adopting the concept of continuous cGAN (CcGAN). The CcGAN that can condition continuous variables is developed to incorporate the time domain through either element-wise addition or conditional batch normalization. We note that this approach can accommodate other continuous variables (e.g., Young's modulus) similar to the time domain, which makes this framework highly flexible and extendable. Moreover, this framework can handle training data that contain different timestamps and then predict timestamps that do not exist in the training data. As a numerical example, the transient response of the coupled poroelastic process is studied in two different permeability fields: Zinn \& Harvey transformation and a bimodal transformation. The proposed CcGAN uses heterogeneous permeability fields as input parameters while pressure and displacement fields over time are model output. Our results show that the model provides sufficient accuracy with computational speed-up. This robust framework will enable us to perform real-time reservoir management and robust uncertainty quantification in realistic problems.
Aerial Images Meet Crowdsourced Trajectories: A New Approach to Robust Road Extraction
Venue: arXiv cs.CV 2022
Year: 2022

Land remote sensing analysis is a crucial research in earth science. In this work, we focus on a challenging task of land analysis, i.e., automatic extraction of traffic roads from remote sensing data, which has widespread applications in urban development and expansion estimation. Nevertheless, conventional methods either only utilized the limited information of aerial images, or simply fused multimodal information (e.g., vehicle trajectories), thus cannot well recognize unconstrained roads. To facilitate this problem, we introduce a novel neural network framework termed Cross-Modal Message Propagation Network (CMMPNet), which fully benefits the complementary different modal data (i.e., aerial images and crowdsourced trajectories). Specifically, CMMPNet is composed of two deep Auto-Encoders for modality-specific representation learning and a tailor-designed Dual Enhancement Module for cross-modal representation refinement. In particular, the complementary information of each modality is comprehensively extracted and dynamically propagated to enhance the representation of another modality. Extensive experiments on three real-world benchmarks demonstrate the effectiveness of our CMMPNet for robust road extraction benefiting from blending different modal data, either using image and trajectory data or image and Lidar data. From the experimental results, we observe that the proposed approach outperforms current state-of-the-art methods by large margins.Our source code is resealed on the project page http://lingboliu.com/multimodal_road_extraction.html.
SimpleTRON: Simple Transformer with O(N) Complexity
Venue: arXiv cs.CL 2022
Year: 2022

In this paper, we propose that the dot product pairwise matching attention layer, which is widely used in Transformer-based models, is redundant for the model performance. Attention, in its original formulation, has to be seen rather as a human-level tool to explore and/or visualize relevancy scores in sequential data. However, the way how it is constructed leads to significant computational complexity. Instead, we present SimpleTRON: Simple Transformer with O(N) Complexity, a simple and fast alternative without any approximation that, unlike other approximation models, does not have any architecture-related overhead and therefore can be seen as a purely linear Transformer-like model. This architecture, to the best of our knowledge, outperforms existing sub-quadratic attention approximation models on several tasks from the Long-Range Arena benchmark. Moreover, we show, that SimpleTRON can benefit from weight transfer from pretrained large language models, as its parameters can be fully transferable.
Robust Partial-to-Partial Point Cloud Registration in a Full Range
Venue: arXiv cs.CV 2022
Year: 2022

Point cloud registration for 3D objects is a challenging task due to sparse and noisy measurements, incomplete observations and large transformations. In this work, we propose \textbf{G}raph \textbf{M}atching \textbf{C}onsensus \textbf{Net}work (\textbf{GMCNet}), which estimates pose-invariant correspondences for full-range Partial-to-Partial point cloud Registration (PPR) in the object-level registration scenario. To encode robust point descriptors, \textbf{1)} we first comprehensively investigate transformation-robustness and noise-resilience of various geometric features. \textbf{2)} Then, we employ a novel {T}ransformation-robust {P}oint {T}ransformer (\textbf{TPT}) module to adaptively aggregate local features regarding the structural relations, which takes advantage from both handcrafted rotation-invariant ({\textit{RI}}) features and noise-resilient spatial coordinates. \textbf{3)} Based on a synergy of hierarchical graph networks and graphical modeling, we propose the {H}ierarchical {G}raphical {M}odeling (\textbf{HGM}) architecture to encode robust descriptors consisting of i) a unary term learned from {\textit{RI}} features; and ii) multiple smoothness terms encoded from neighboring point relations at different scales through our TPT modules. Moreover, we construct a challenging PPR dataset (\textbf{MVP-RG}) based on the recent MVP dataset that features high-quality scans. Extensive experiments show that GMCNet outperforms previous state-of-the-art methods for PPR. Notably, GMCNet encodes point descriptors for each point cloud individually without using cross-contextual information, or ground truth correspondences for training. Our code and datasets are available at: https://github.com/paul007pl/GMCNet.
Building extraction with vision transformer
Venue: arXiv cs.CV 2022
Year: 2022

As an important carrier of human productive activities, the extraction of buildings is not only essential for urban dynamic monitoring but also necessary for suburban construction inspection. Nowadays, accurate building extraction from remote sensing images remains a challenge due to the complex background and diverse appearances of buildings. The convolutional neural network (CNN) based building extraction methods, although increased the accuracy significantly, are criticized for their inability for modelling global dependencies. Thus, this paper applies the Vision Transformer for building extraction. However, the actual utilization of the Vision Transformer often comes with two limitations. First, the Vision Transformer requires more GPU memory and computational costs compared to CNNs. This limitation is further magnified when encountering large-sized inputs like fine-resolution remote sensing images. Second, spatial details are not sufficiently preserved during the feature extraction of the Vision Transformer, resulting in the inability for fine-grained building segmentation. To handle these issues, we propose a novel Vision Transformer (BuildFormer), with a dual-path structure. Specifically, we design a spatial-detailed context path to encode rich spatial details and a global context path to capture global dependencies. Besides, we develop a window-based linear multi-head self-attention to make the complexity of the multi-head self-attention linear with the window size, which strengthens the global context extraction by using large windows and greatly improves the potential of the Vision Transformer in processing large-sized remote sensing images. The proposed method yields state-of-the-art performance (75.74% IoU) on the Massachusetts building dataset. Code will be available.
CryoCiM: Cryogenic Compute-in-Memory based on the Quantum Anomalous Hall Effect
Venue: arXiv cs.ET 2022
Year: 2022

The scaling of the already-matured CMOS technology is steadily approaching its physical limit, motivating the quest for a suitable alternative. Cryogenic operation offers a promising pathway towards continued improvement in computing speed and energy efficiency without aggressive scaling. However, the memory wall bottleneck of the traditional von-Neumann architecture persists even at cryogenic temperature. That is where a compute-in-memory (CiM) architecture, that embeds computing within the memory unit, comes into play. Computations within the memory unit help reduce the expensive data transfer between the memory and the computing units. Therefore, CiM provides extreme energy efficiency that can enable lower cooling cost at cryogenic temperature. In this work, we demonstrate CryoCiM, a cryogenic compute-in-memory framework utilizing a non-volatile memory system based on the quantum anomalous Hall effect (QAHE). Our design can perform memory read/write, and universal binary logic operations (NAND, NOR, and XOR). We design a novel peripheral circuit assembly that can perform the read/write, and single-cycle in-memory logic operations. The utilization of a QAHE-based memory system promises robustness against process variations, through the usage of topologically protected resistive states for data storage. CryoCiM is the first step towards utilizing exclusively cryogenic phenomena to serve the dual purpose of storage and computation with ultra-low power (nano-watts) operations.
A Machine Learning Analysis of COVID-19 Mental Health Data
Venue: arXiv cs.LG 2022
Year: 2022

In late December 2019, the novel coronavirus (Sars-Cov-2) and the resulting disease COVID-19 were first identified in Wuhan China. The disease slipped through containment measures, with the first known case in the United States being identified on January 20th, 2020. In this paper, we utilize survey data from the Inter-university Consortium for Political and Social Research and apply several statistical and machine learning models and techniques such as Decision Trees, Multinomial Logistic Regression, Naive Bayes, k-Nearest Neighbors, Support Vector Machines, Neural Networks, Random Forests, Gradient Tree Boosting, XGBoost, CatBoost, LightGBM, Synthetic Minority Oversampling, and Chi-Squared Test to analyze the impacts the COVID-19 pandemic has had on the mental health of frontline workers in the United States. Through the interpretation of the many models applied to the mental health survey data, we have concluded that the most important factor in predicting the mental health decline of a frontline worker is the healthcare role the individual is in (Nurse, Emergency Room Staff, Surgeon, etc.), followed by the amount of sleep the individual has had in the last week, the amount of COVID-19 related news an individual has consumed on average in a day, the age of the worker, and the usage of alcohol and cannabis.