OpenRaiser/Intern-Atlas huggingface dataset:
Negative Feedback System as Optimizer for Machine Learning Systems
Venue: arXiv cs.LG 2024
Year: 2024

With high forward gain, a negative feedback system has the ability to perform the inverse of a linear or non-linear function that is in the feedback path. This property of negative feedback systems has been widely used in analog electronic circuits to construct precise closed-loop functions. This paper describes how the function-inverting process of a negative feedback system serves as a physical analogy of the optimization technique in machine learning. We show that this process is able to learn some non-differentiable functions in cases where a gradient descent-based method fails. We also show that the optimization process reduces to gradient descent under the constraint of squared error minimization. We derive the backpropagation technique and other known optimization techniques of deep networks from the properties of negative feedback system independently of the gradient descent method. This analysis provides a novel view of neural network optimization and may provide new insights on open problems.
Tuned Compositional Feature Replays for Efficient Stream Learning
Venue: arXiv cs.CV 2024
Year: 2024

Our brains extract durable, generalizable knowledge from transient experiences of the world. Artificial neural networks come nowhere close to this ability. When tasked with learning to classify objects by training on non-repeating video frames in temporal order (online stream learning), models that learn well from shuffled datasets catastrophically forget old knowledge upon learning new stimuli. We propose a new continual learning algorithm, Compositional Replay Using Memory Blocks (CRUMB), which mitigates forgetting by replaying feature maps reconstructed by combining generic parts. CRUMB concatenates trainable and re-usable "memory block" vectors to compositionally reconstruct feature map tensors in convolutional neural networks. Storing the indices of memory blocks used to reconstruct new stimuli enables memories of the stimuli to be replayed during later tasks. This reconstruction mechanism also primes the neural network to minimize catastrophic forgetting by biasing it towards attending to information about object shapes more than information about image textures, and stabilizes the network during stream learning by providing a shared feature-level basis for all training examples. These properties allow CRUMB to outperform an otherwise identical algorithm that stores and replays raw images, while occupying only 3.6% as much memory. We stress-tested CRUMB alongside 13 competing methods on 7 challenging datasets. To address the limited number of existing online stream learning datasets, we introduce 2 new benchmarks by adapting existing datasets for stream learning. With only 3.7-4.1% as much memory and 15-43% as much runtime, CRUMB mitigates catastrophic forgetting more effectively than the state-of-the-art. Our code is available at https://github.com/MorganBDT/crumb.git.
Context-self contrastive pretraining for crop type semantic segmentation
Venue: arXiv cs.CV 2024
Year: 2024

In this paper, we propose a fully supervised pre-training scheme based on contrastive learning particularly tailored to dense classification tasks. The proposed Context-Self Contrastive Loss (CSCL) learns an embedding space that makes semantic boundaries pop-up by use of a similarity metric between every location in a training sample and its local context. For crop type semantic segmentation from Satellite Image Time Series (SITS) we find performance at parcel boundaries to be a critical bottleneck and explain how CSCL tackles the underlying cause of that problem, improving the state-of-the-art performance in this task. Additionally, using images from the Sentinel-2 (S2) satellite missions we compile the largest, to our knowledge, SITS dataset densely annotated by crop type and parcel identities, which we make publicly available together with the data generation pipeline. Using that data we find CSCL, even with minimal pre-training, to improve all respective baselines and present a process for semantic segmentation at super-resolution for obtaining crop classes at a more granular level. The code and instructions to download the data can be found in https://github.com/michaeltrs/DeepSatModels.
AutoGL: A Library for Automated Graph Learning
Venue: arXiv cs.LG 2024
Year: 2024

Recent years have witnessed an upsurge in research interests and applications of machine learning on graphs. However, manually designing the optimal machine learning algorithms for different graph datasets and tasks is inflexible, labor-intensive, and requires expert knowledge, limiting its adaptivity and applicability. Automated machine learning (AutoML) on graphs, aiming to automatically design the optimal machine learning algorithm for a given graph dataset and task, has received considerable attention. However, none of the existing libraries can fully support AutoML on graphs. To fill this gap, we present Automated Graph Learning (AutoGL), the first dedicated library for automated machine learning on graphs. AutoGL is open-source, easy to use, and flexible to be extended. Specifically, we propose a three-layer architecture, consisting of backends to interface with devices, a complete automated graph learning pipeline, and supported graph applications. The automated machine learning pipeline further contains five functional modules: auto feature engineering, neural architecture search, hyper-parameter optimization, model training, and auto ensemble, covering the majority of existing AutoML methods on graphs. For each module, we provide numerous state-of-the-art methods and flexible base classes and APIs, which allow easy usage and customization. We further provide experimental results to showcase the usage of our AutoGL library. We also present AutoGL-light, a lightweight version of AutoGL to facilitate customizing pipelines and enriching applications, as well as benchmarks for graph neural architecture search. The codes of AutoGL are publicly available at https://github.com/THUMNLab/AutoGL.
Understanding Prediction Discrepancies in Machine Learning Classifiers
Venue: arXiv cs.LG 2024
Year: 2024

A multitude of classifiers can be trained on the same data to achieve similar performances during test time, while having learned significantly different classification patterns. This phenomenon, which we call prediction discrepancies, is often associated with the blind selection of one model instead of another with similar performances. When making a choice, the machine learning practitioner has no understanding on the differences between models, their limits, where they agree and where they don't. But his/her choice will result in concrete consequences for instances to be classified in the discrepancy zone, since the final decision will be based on the selected classification pattern. Besides the arbitrary nature of the result, a bad choice could have further negative consequences such as loss of opportunity or lack of fairness. This paper proposes to address this question by analyzing the prediction discrepancies in a pool of best-performing models trained on the same data. A model-agnostic algorithm, DIG, is proposed to capture and explain discrepancies locally, to enable the practitioner to make the best educated decision when selecting a model by anticipating its potential undesired consequences. All the code to reproduce the experiments is available.
Are Word Embedding Methods Stable and Should We Care About It?
Venue: arXiv cs.CL 2024
Year: 2024

A representation learning method is considered stable if it consistently generates similar representation of the given data across multiple runs. Word Embedding Methods (WEMs) are a class of representation learning methods that generate dense vector representation for each word in the given text data. The central idea of this paper is to explore the stability measurement of WEMs using intrinsic evaluation based on word similarity. We experiment with three popular WEMs: Word2Vec, GloVe, and fastText. For stability measurement, we investigate the effect of five parameters involved in training these models. We perform experiments using four real-world datasets from different domains: Wikipedia, News, Song lyrics, and European parliament proceedings. We also observe the effect of WEM stability on three downstream tasks: Clustering, POS tagging, and Fairness evaluation. Our experiments indicate that amongst the three WEMs, fastText is the most stable, followed by GloVe and Word2Vec.
Proportionality and Strategyproofness in Multiwinner Elections
Venue: arXiv cs.GT 2024
Year: 2024

Multiwinner voting rules can be used to select a fixed-size committee from a larger set of candidates. We consider approval-based committee rules, which allow voters to approve or disapprove candidates. In this setting, several voting rules such as Proportional Approval Voting (PAV) and Phragmén's rules have been shown to produce committees that are proportional, in the sense that they proportionally represent voters' preferences; all of these rules are strategically manipulable by voters. On the other hand, a generalisation of Approval Voting gives a non-proportional but strategyproof voting rule. We show that there is a fundamental tradeoff between these two properties: we prove that no multiwinner voting rule can simultaneously satisfy a weak form of proportionality (a weakening of justified representation) and a weak form of strategyproofness. Our impossibility is obtained using a formulation of the problem in propositional logic and applying SAT solvers; a human-readable version of the computer-generated proof is obtained by extracting a minimal unsatisfiable set (MUS). We also discuss several related axiomatic questions in the domain of committee elections.
Semi-Decentralized Federated Edge Learning for Fast Convergence on Non-IID Data
Venue: arXiv cs.NI 2024
Year: 2024

Federated edge learning (FEEL) has emerged as an effective approach to reduce the large communication latency in Cloud-based machine learning solutions, while preserving data privacy. Unfortunately, the learning performance of FEEL may be compromised due to limited training data in a single edge cluster. In this paper, we investigate a novel framework of FEEL, namely semi-decentralized federated edge learning (SD-FEEL). By allowing model aggregation across different edge clusters, SD-FEEL enjoys the benefit of FEEL in reducing the training latency, while improving the learning performance by accessing richer training data from multiple edge clusters. A training algorithm for SD-FEEL with three main procedures in each round is presented, including local model updates, intra-cluster and inter-cluster model aggregations, which is proved to converge on non-independent and identically distributed (non-IID) data. We also characterize the interplay between the network topology of the edge servers and the communication overhead of inter-cluster model aggregation on the training performance. Experiment results corroborate our analysis and demonstrate the effectiveness of SD-FFEL in achieving faster convergence than traditional federated learning architectures. Besides, guidelines on choosing critical hyper-parameters of the training algorithm are also provided.
Free Vibration analysis of Curvilinearly Stiffened Composite plates with an arbitrarily shaped cutout using Isogeometric Analysis
Venue: arXiv cs.CE 2024
Year: 2024

This paper focuses on the isogeometric vibration analysis of curvilinearly stiffened composite panels. The stiffness matrices and the mass matrices are derived using the first-order shear deformation theory (FSDT). The present method models the plate and the stiffener separately, which allows the stiffener element nodes to not coincide with the plate shell-element nodes. The stiffness and mass matrices of a stiffener are transformed to those of the plate through the displacement compatibility conditions at the plate/stiffener interface by interpolation using NURBS basis functions. Cutouts are modeled using a single NURBS patch generated by creating a ruled surface between two curves. The proposed formulation is first validated by comparing it with available literature. The effects of width-to-thickness ratio, fiber orientation, ply layups, shape and size of the cutouts and the boundary conditions on the response of stiffened composite plates are then analyzed and the numerical results are used to derive useful conclusions.
A Review of the Non-Invasive Techniques for Monitoring Different Aspects of Sleep
Venue: arXiv cs.HC 2024
Year: 2024

Quality sleep is very important for a healthy life. Nowadays, many people around the world are not getting enough sleep which is having negative impacts on their lifestyles. Studies are being conducted for sleep monitoring and have now become an important tool for understanding sleep behavior. The gold standard method for sleep analysis is polysomnography (PSG) conducted in a clinical environment but this method is both expensive and complex for long-term use. With the advancements in the field of sensors and the introduction of off-the-shelf technologies, unobtrusive solutions are becoming common as alternatives for in-home sleep monitoring. Various solutions have been proposed using both wearable and non-wearable methods which are cheap and easy to use for in-home sleep monitoring. In this paper, we present a comprehensive survey of the latest research works (2015 and after) conducted in various categories of sleep monitoring including sleep stage classification, sleep posture recognition, sleep disorders detection, and vital signs monitoring. We review the latest works done using the non-invasive approach and cover both wearable and non-wearable methods. We discuss the design approaches and key attributes of the work presented and provide an extensive analysis based on 10 key factors, to give a comprehensive overview of the recent developments and trends in all four categories of sleep monitoring. We also present some publicly available datasets for different categories of sleep monitoring. In the end, we discuss several open issues and provide future research directions in the area of sleep monitoring.
The music box operad: Random generation of musical phrases from patterns
Venue: arXiv cs.SD 2024
Year: 2024

We introduce the notion of multi-patterns, a combinatorial abstraction of polyphonic musical phrases. The interest of this approach in encoding musical phrases lies in the fact that it becomes possible to compose multi-patterns in order to produce new ones. This composition is parameterized by a monoid structure on the scale degrees. This embeds the set of the musical phrases into an algebraic framework since the set of the multi-patterns is endowed with the structure of an operad. Operads are algebraic structures offering a formalization and an abstraction of the notion of operators and their compositions. Seeing musical phrases as operators allows us to perform computations on phrases and admits applications in generative music. Indeed, given a set of initial multi-patterns, we propose various algorithms to randomly generate a new and longer phrase emulating the style suggested by the inputted multi-patterns. The designed algorithms use types of grammars working with operads and colored operads, known as bud generating systems.
Regularizing Explanations in Bayesian Convolutional Neural Networks
Venue: arXiv cs.LG 2024
Year: 2024

Neural networks are powerful function approximators with tremendous potential in learning complex distributions. However, they are prone to overfitting on spurious patterns. Bayesian inference provides a principled way to regularize neural networks and give well-calibrated uncertainty estimates. It allows us to specify prior knowledge on weights. However, specifying domain knowledge via distributions over weights is infeasible. Furthermore, it is unable to correct models when they focus on spurious or irrelevant features. New methods within explainable artificial intelligence allow us to regularize explanations in the form of feature importance to add domain knowledge and correct the models' focus. Nevertheless, they are incompatible with Bayesian neural networks, as they require us to modify the loss function. We propose a new explanation regularization method that is compatible with Bayesian inference. Consequently, we can quantify uncertainty and, at the same time, have correct explanations. We test our method using four different datasets. The results show that our method improves predictive performance when models overfit on spurious features or are uncertain of which features to focus on. Moreover, our method performs better than augmenting training data with samples where spurious features are removed through masking. We provide code, data, trained weights, and hyperparameters.
Efficient Masked Face Recognition Method during the COVID-19 Pandemic
Venue: arXiv cs.CV 2024
Year: 2024

The coronavirus disease (COVID-19) is an unparalleled crisis leading to a huge number of casualties and security problems. In order to reduce the spread of coronavirus, people often wear masks to protect themselves. This makes face recognition a very difficult task since certain parts of the face are hidden. A primary focus of researchers during the ongoing coronavirus pandemic is to come up with suggestions to handle this problem through rapid and efficient solutions. In this paper, we propose a reliable method based on occlusion removal and deep learning-based features in order to address the problem of the masked face recognition process. The first step is to remove the masked face region. Next, we apply three pre-trained deep Convolutional Neural Networks (CNN) namely, VGG-16, AlexNet, and ResNet-50, and use them to extract deep features from the obtained regions (mostly eyes and forehead regions). The Bag-of-features paradigm is then applied to the feature maps of the last convolutional layer in order to quantize them and to get a slight representation comparing to the fully connected layer of classical CNN. Finally, Multilayer Perceptron (MLP) is applied for the classification process. Experimental results on Real-World-Masked-Face-Dataset show high recognition performance compared to other state-of-the-art methods.
When Automated Program Repair Meets Regression Testing -- An Extensive Study on 2 Million Patches
Venue: arXiv cs.SE 2024
Year: 2024

In recent years, Automated Program Repair (APR) has been extensively studied in academia and even drawn wide attention from industry. However, APR techniques can be extremely time consuming since (1) a large number of patches can be generated for a given bug, and (2) each patch needs to be executed on the original tests to ensure its correctness. In the literature, various techniques (e.g., based on learning, mining, and constraint solving) have been proposed/studied to reduce the number of patches. Intuitively, every patch can be treated as a software revision during regression testing; thus, traditional Regression Test Selection (RTS) techniques can be leveraged to only execute the tests affected by each patch (as the other tests would keep the same outcomes) to further reduce patch execution time. However, few APR systems actually adopt RTS and there is still a lack of systematic studies demonstrating the benefits of RTS and the impact of different RTS strategies on APR. To this end, this paper presents the first extensive study of widely-used RTS techniques at different levels (i.e., class/method/statement levels) for 12 state-of-the-art APR systems on over 2M patches. Our study reveals various practical guidelines for bridging the gap between APR and regression testing.
Fully Adaptive Self-Stabilizing Transformer for LCL Problems
Venue: arXiv cs.DC 2024
Year: 2024

The first generic self-stabilizing transformer for local problems in a constrained bandwidth model is introduced. This transformer can be applied to a wide class of locally checkable labeling (LCL) problems, converting a given fault free synchronous algorithm that satisfies certain conditions into a self-stabilizing synchronous algorithm for the same problem. The resulting self-stabilizing algorithms are anonymous, size-uniform, and \emph{fully adaptive} in the sense that their time complexity is bounded as a function of the number $k$ of nodes that suffered faults (possibly at different times) since the last legal configuration. Specifically, for graphs whose degrees are up-bounded by $Δ$, the algorithms produced by the transformer stabilize in time proportional to $\log (k + Δ)$ in expectation, independently of the number of nodes in the graph. As such, the transformer is applicable also for infinite graphs (with degree bound $Δ$). Another appealing feature of the transformer is its small message size overhead. The transformer is applied to known algorithms (or simple variants thereof) for some classic LCL problems, producing the first anonymous size-uniform self-stabilizing algorithms for these problems that are provably fully adaptive. From a technical point of view, the transformer's key design feature is a novel probabilistic tool that allows different nodes to act in synchrony even though their clocks may have been adversarially manipulated.
Ising Machines' Dynamics and Regularization for Near-Optimal Large and Massive MIMO Detection
Venue: arXiv cs.NI 2024
Year: 2024

Optimal MIMO detection has been one of the most challenging and computationally inefficient tasks in wireless systems. We show that the new analog computing techniques like Coherent Ising Machines (CIM) are promising candidates for performing near-optimal MIMO detection. We propose a novel regularized Ising formulation for MIMO detection that mitigates a common error floor problem and further evolves it into an algorithm that achieves near-optimal MIMO detection. Massive MIMO systems, that have a large number of antennas at the Access point (AP), allow linear detectors to be near-optimal. However, the simplified detection in these systems comes at the cost of overall throughput, which could be improved by supporting more users. By means of numerical simulations, we show that in principle a MIMO detector based on a hybrid use of a CIM would allow us to add more transmitter antennas/users and increase the overall throughput of the cell by a significant factor. This would open up the opportunity to operate using more aggressive modulation and coding schemes and hence achieve high throughput: for a $16\times16$ large MIMO system, we estimate around 2.5$\times$ more throughput in mid-SNR regime ($\approx 12 dB$) and 2$\times$ more throughput in high-SNR regime( $>$ 20dB) than the industry standard, Minimum-Mean Square Error decoding (MMSE).
The statistical advantage of automatic NLG metrics at the system level
Venue: arXiv cs.CL 2024
Year: 2024

Estimating the expected output quality of generation systems is central to NLG. This paper qualifies the notion that automatic metrics are not as good as humans in estimating system-level quality. Statistically, humans are unbiased, high variance estimators, while metrics are biased, low variance estimators. We compare these estimators by their error in pairwise prediction (which generation system is better?) using the bootstrap. Measuring this error is complicated: predictions are evaluated against noisy, human predicted labels instead of the ground truth, and metric predictions fluctuate based on the test sets they were calculated on. By applying a bias-variance-noise decomposition, we adjust this error to a noise-free, infinite test set setting. Our analysis compares the adjusted error of metrics to humans and a derived, perfect segment-level annotator, both of which are unbiased estimators dependent on the number of judgments collected. In MT, we identify two settings where metrics outperform humans due to a statistical advantage in variance: when the number of human judgments used is small, and when the quality difference between compared systems is small. The data and code to reproduce our analyses are available at https://github.com/johntzwei/metric-statistical-advantage .