OpenRaiser/Intern-Atlas huggingface dataset:
Communication-Efficient Federated Bilevel Optimization with Local and Global Lower Level Problems
Venue: arXiv cs.LG 2024
Year: 2024

Bilevel Optimization has witnessed notable progress recently with new emerging efficient algorithms. However, its application in the Federated Learning setting remains relatively underexplored, and the impact of Federated Learning's inherent challenges on the convergence of bilevel algorithms remain obscure. In this work, we investigate Federated Bilevel Optimization problems and propose a communication-efficient algorithm, named FedBiOAcc. The algorithm leverages an efficient estimation of the hyper-gradient in the distributed setting and utilizes the momentum-based variance-reduction acceleration. Remarkably, FedBiOAcc achieves a communication complexity $O(ε^{-1})$, a sample complexity $O(ε^{-1.5})$ and the linear speed up with respect to the number of clients. We also analyze a special case of the Federated Bilevel Optimization problems, where lower level problems are locally managed by clients. We prove that FedBiOAcc-Local, a modified version of FedBiOAcc, converges at the same rate for this type of problems. Finally, we validate the proposed algorithms through two real-world tasks: Federated Data-cleaning and Federated Hyper-representation Learning. Empirical results show superior performance of our algorithms.
Tolerance to Asynchrony in Algorithms for Multiplication and Modulo
Venue: arXiv cs.DC 2024
Year: 2024

In this article, we study some parallel processing algorithms for multiplication and modulo operations. We demonstrate that the state transitions that are formed under these algorithms satisfy lattice-linearity, where these algorithms induce a lattice among the global states. Lattice-linearity implies that these algorithms can be implemented in asynchronous environments, where the nodes are allowed to read old information from each other. It means that these algorithms are guaranteed to converge correctly without any synchronization overhead. These algorithms also exhibit snap-stabilizing properties, i.e., starting from an arbitrary state, the sequence of state transitions made by the system strictly follows its specification.
Interpretable Deep Learning Methods for Multiview Learning
Venue: arXiv cs.LG 2024
Year: 2024

Technological advances have enabled the generation of unique and complementary types of data or views (e.g. genomics, proteomics, metabolomics) and opened up a new era in multiview learning research with the potential to lead to new biomedical discoveries. We propose iDeepViewLearn (Interpretable Deep Learning Method for Multiview Learning) for learning nonlinear relationships in data from multiple views while achieving feature selection. iDeepViewLearn combines deep learning flexibility with the statistical benefits of data and knowledge-driven feature selection, giving interpretable results. Deep neural networks are used to learn view-independent low-dimensional embedding through an optimization problem that minimizes the difference between observed and reconstructed data, while imposing a regularization penalty on the reconstructed data. The normalized Laplacian of a graph is used to model bilateral relationships between variables in each view, therefore, encouraging selection of related variables. iDeepViewLearn is tested on simulated and two real-world data, including breast cancer-related gene expression and methylation data. iDeepViewLearn had competitive classification results and identified genes and CpG sites that differentiated between individuals who died from breast cancer and those who did not. The results of our real data application and simulations with small to moderate sample sizes suggest that iDeepViewLearn may be a useful method for small-sample-size problems compared to other deep learning methods for multiview learning.
Towards Fair Machine Learning Software: Understanding and Addressing Model Bias Through Counterfactual Thinking
Venue: arXiv cs.SE 2024
Year: 2024

The increasing use of Machine Learning (ML) software can lead to unfair and unethical decisions, thus fairness bugs in software are becoming a growing concern. Addressing these fairness bugs often involves sacrificing ML performance, such as accuracy. To address this issue, we present a novel counterfactual approach that uses counterfactual thinking to tackle the root causes of bias in ML software. In addition, our approach combines models optimized for both performance and fairness, resulting in an optimal solution in both aspects. We conducted a thorough evaluation of our approach on 10 benchmark tasks using a combination of 5 performance metrics, 3 fairness metrics, and 15 measurement scenarios, all applied to 8 real-world datasets. The conducted extensive evaluations show that the proposed method significantly improves the fairness of ML software while maintaining competitive performance, outperforming state-of-the-art solutions in 84.6% of overall cases based on a recent benchmarking tool.
DDCNN: A Promising Tool for Simulation-To-Reality UAV Fault Diagnosis
Venue: arXiv cs.RO 2024
Year: 2024

Identifying the fault in propellers is important to keep quadrotors operating safely and efficiently. The simulation-to-reality (sim-to-real) UAV fault diagnosis methods provide a cost-effective and safe approach to detecting propeller faults. However, due to the gap between simulation and reality, classifiers trained with simulated data usually underperform in real flights. In this work, a novel difference-based deep convolutional neural network (DDCNN) model is presented to address the above issue. It uses the difference features extracted by deep convolutional neural networks to reduce the sim-to-real gap. Moreover, a new domain adaptation (DA) method is presented to further bring the distribution of the real-flight data closer to that of the simulation data. The experimental results demonstrate that the DDCNN+DA model can increase the accuracy from 52.9% to 99.1% in real-world UAV fault detection.
A Superdirective Beamforming Approach with Impedance Coupling and Field Coupling for Compact Antenna Arrays
Venue: arXiv cs.IT 2024
Year: 2024

In most multiple-input multiple-output (MIMO) communication systems, antennas are spaced at least half a wavelength apart to reduce mutual coupling. In this configuration, the maximum array gain is equal to the number of antennas. However, when the antenna spacing is significantly reduced, the array gain of a compact array can become proportional to the square of the number of antennas, greatly exceeding that of traditional MIMO systems. Achieving this "superdirectivity" requires complex calculations of the excitation coefficients (beamforming vector), which is a challenging task. In this paper, we address this problem with a novel double coupling-based superdirective beamforming method. In particular, we categorize the antenna coupling effects to impedance coupling and field coupling. By characterizing these two coupling in model, we derive the beamforming vector for superdirective arrays. We prove that the field coupling matrix has the unique solution for an antenna array, and itself has the ability to fully characterize the distorted coupling field. Based on this proven theorem, we propose a method that accurately calculates the coupling matrix using only a number of angle sampling points on the order of the number of antennas. Moreover, a prototype of an independently-controlled superdirective antenna array is developed. Full-wave electromagnetic simulations and real-world experiments validate the effectiveness of our proposed approaches, and superdirectivity is achieved in reality by a compact array with 4 and 8 dipole antennas.
Updates on the Low-Level Abstraction of Memory Access
Venue: arXiv cs.PF 2024
Year: 2024

Choosing the best memory layout for each hardware architecture is increasingly important as more and more programs become memory bound. For portable codes that run across heterogeneous hardware architectures, the choice of the memory layout for data structures is ideally decoupled from the rest of a program. The low-level abstraction of memory access (LLAMA) is a C++ library that provides a zero-runtime-overhead abstraction layer, underneath which memory mappings can be freely exchanged to customize data layouts, memory access and access instrumentation, focusing on multidimensional arrays of nested, structured data. After its scientific debut, several improvements and extensions have been added to LLAMA. This includes compile-time array extents for zero-memory-overhead views, support for computations during memory access, new mappings for bit-packing, switching types, byte-splitting, memory access instrumentation, and explicit SIMD support. This contribution provides an overview of recent developments in the LLAMA library.
The Complexity of Graph Exploration Games
Venue: arXiv cs.CC 2024
Year: 2024

Graph Exploration problems ask a searcher to explore an unknown environment. The environment is modeled as a graph, where the searcher needs to visit each vertex beginning at some vertex. Treasure Hunt problems are a variation of Graph Exploration, in which the searcher needs to find a hidden treasure, which is located at a designated vertex. Usually these problems are modeled as online problems, and any online algorithm performs poorly because it has too little knowledge about the instance to react adequately to the requests of the adversary. Thus, the impact of a priori knowledge is of interest. One form of a priori knowledge is an unlabeled map, which is an isomorphic copy of the graph. We analyze Graph Exploration and Treasure Hunt problems with an unlabeled map that is provided to the searcher. For this, we formulate decision variants of both problems by interpreting the online problems as a game between the online algorithm (the searcher) and the adversary. The map, however, is not controllable by the adversary. The question is whether the searcher is able to explore the graph completely or find the treasure for all possible decisions of the adversary. We analyze these games in multiple settings, with and without costs on the edges, on directed and undirected graphs and with different constraints (allowing multiple visits to vertices or edges) on the solution. We prove PSPACE-completeness for most of these games. Additionally, we analyze the complexity of related problems that have additional constraints on the solution.
THC: Accelerating Distributed Deep Learning Using Tensor Homomorphic Compression
Venue: arXiv cs.LG 2024
Year: 2024

Deep neural networks (DNNs) are the de facto standard for essential use cases, such as image classification, computer vision, and natural language processing. As DNNs and datasets get larger, they require distributed training on increasingly larger clusters. A main bottleneck is the resulting communication overhead where workers exchange model updates (i.e., gradients) on a per-round basis. To address this bottleneck and accelerate training, a widely-deployed approach is compression. However, previous deployments often apply bi-directional compression schemes by simply using a uni-directional gradient compression scheme in each direction. This results in significant computational overheads at the parameter server and increased compression error, leading to longer training and lower accuracy. We introduce Tensor Homomorphic Compression (THC), a novel bi-directional compression framework that enables the direct aggregation of compressed values and thus eliminating the aforementioned computational overheads. Moreover, THC is compatible with in-network aggregation (INA), which allows for further acceleration. Our evaluation shows that training representative vision and language models with THC reaches target accuracy by 1.40x to 1.47x faster using INA and 1.28x to 1.33x faster using a software PS compared with state-of-the-art systems.
SplitOut: Out-of-the-Box Training-Hijacking Detection in Split Learning via Outlier Detection
Venue: arXiv cs.LG 2024
Year: 2024

Split learning enables efficient and privacy-aware training of a deep neural network by splitting a neural network so that the clients (data holders) compute the first layers and only share the intermediate output with the central compute-heavy server. This paradigm introduces a new attack medium in which the server has full control over what the client models learn, which has already been exploited to infer the private data of clients and to implement backdoors in the client models. Although previous work has shown that clients can successfully detect such training-hijacking attacks, the proposed methods rely on heuristics, require tuning of many hyperparameters, and do not fully utilize the clients' capabilities. In this work, we show that given modest assumptions regarding the clients' compute capabilities, an out-of-the-box outlier detection method can be used to detect existing training-hijacking attacks with almost-zero false positive rates. We conclude through experiments on different tasks that the simplicity of our approach we name \textit{SplitOut} makes it a more viable and reliable alternative compared to the earlier detection methods.
Coordinating Multiple Intelligent Reflecting Surfaces without Channel Information
Venue: arXiv cs.IT 2024
Year: 2024

Conventional beamforming methods for intelligent reflecting surfaces (IRSs) or reconfigurable intelligent surfaces (RISs) typically entail the full channel state information (CSI). However, the computational cost of channel acquisition soars exponentially with the number of IRSs. To bypass this difficulty, we propose a novel strategy called blind beamforming that coordinates multiple IRSs by means of statistics without knowing CSI. Blind beamforming only requires measuring the received signal power at the user terminal for a sequence of randomly generated phase shifts across all IRSs. The main idea is to extract the key statistical quantity for beamforming by exploring only a small portion of the whole solution space of phase shifts. We show that blind beamforming guarantees a signal-to-noise ratio (SNR) boost of Theta(N^{2L}) under certain conditions, where L is the number of IRSs and N is the number of reflecting elements per IRS. The proposed conditions for achieving the optimal SNR boost of Theta(N^{4}) in a double-IRS system are much easier to satisfy than the existing ones in the literature. Most importantly, the proposed conditions can be extended to a fully general L-IRS system. The above result significantly improves upon the state of the art in the area of multi-IRS-assisted communication. Moreover, blind beamforming is justified via field tests and simulations. In particular, as shown in our field tests at 2.6 GHz, our method yields up to 17 dB SNR boost; to the best of our knowledge, this is the first time that the use of multiple IRSs gets verified in the real world.
Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey
Venue: arXiv cs.CV 2024
Year: 2024

With the urgent demand for generalized deep models, many pre-trained big models are proposed, such as BERT, ViT, GPT, etc. Inspired by the success of these models in single domains (like computer vision and natural language processing), the multi-modal pre-trained big models have also drawn more and more attention in recent years. In this work, we give a comprehensive survey of these models and hope this paper could provide new insights and helps fresh researchers to track the most cutting-edge works. Specifically, we firstly introduce the background of multi-modal pre-training by reviewing the conventional deep learning, pre-training works in natural language process, computer vision, and speech. Then, we introduce the task definition, key challenges, and advantages of multi-modal pre-training models (MM-PTMs), and discuss the MM-PTMs with a focus on data, objectives, network architectures, and knowledge enhanced pre-training. After that, we introduce the downstream tasks used for the validation of large-scale MM-PTMs, including generative, classification, and regression tasks. We also give visualization and analysis of the model parameters and results on representative downstream tasks. Finally, we point out possible research directions for this topic that may benefit future works. In addition, we maintain a continuously updated paper list for large-scale pre-trained multi-modal big models: https://github.com/wangxiao5791509/MultiModal_BigModels_Survey. This paper has been published by the journal Machine Intelligence Research (MIR), https://link.springer.com/article/10.1007/s11633-022-1410-8, DOI: 10.1007/s11633-022-1410-8, vol. 20, no. 4, pp. 447-482, 2023.
Towards Universal Fake Image Detectors that Generalize Across Generative Models
Venue: arXiv cs.CV 2024
Year: 2024

With generative models proliferating at a rapid rate, there is a growing need for general purpose fake image detectors. In this work, we first show that the existing paradigm, which consists of training a deep network for real-vs-fake classification, fails to detect fake images from newer breeds of generative models when trained to detect GAN fake images. Upon analysis, we find that the resulting classifier is asymmetrically tuned to detect patterns that make an image fake. The real class becomes a sink class holding anything that is not fake, including generated images from models not accessible during training. Building upon this discovery, we propose to perform real-vs-fake classification without learning; i.e., using a feature space not explicitly trained to distinguish real from fake images. We use nearest neighbor and linear probing as instantiations of this idea. When given access to the feature space of a large pretrained vision-language model, the very simple baseline of nearest neighbor classification has surprisingly good generalization ability in detecting fake images from a wide variety of generative models; e.g., it improves upon the SoTA by +15.07 mAP and +25.90% acc when tested on unseen diffusion and autoregressive models.
Correlation Clustering with Active Learning of Pairwise Similarities
Venue: arXiv cs.LG 2024
Year: 2024

Correlation clustering is a well-known unsupervised learning setting that deals with positive and negative pairwise similarities. In this paper, we study the case where the pairwise similarities are not given in advance and must be queried in a cost-efficient way. Thereby, we develop a generic active learning framework for this task that benefits from several advantages, e.g., flexibility in the type of feedback that a user/annotator can provide, adaptation to any correlation clustering algorithm and query strategy, and robustness to noise. In addition, we propose and analyze a number of novel query strategies suited to this setting. We demonstrate the effectiveness of our framework and the proposed query strategies via several experimental studies.
Deep Convolutional Framelet Denoising for Panoramic by Mixed Wavelet Integration
Venue: arXiv cs.CV 2024
Year: 2024

Enhancing quality and removing noise during preprocessing is one of the most critical steps in image processing. X-ray images are created by photons colliding with atoms and the variation in scattered noise absorption. This noise leads to a deterioration in the graph's medical quality and, at times, results in repetition, thereby increasing the patient's effective dose. One of the most critical challenges in this area has consistently been lowering the image noise. Techniques like BM3d, low-pass filters, and Autoencoder have taken this step. Owing to their structural design and high rate of repetition, neural networks employing diverse architectures have, over the past decade, achieved noise reduction with satisfactory outcomes, surpassing the traditional BM3D and low-pass filters. The combination of the Hankel matrix with neural networks represents one of these configurations. The Hankel matrix aims to identify a local circle by separating individual values into local and non-local components, utilizing a non-local matrix. A non-local matrix can be created using the wave or DCT. This paper suggests integrating the waveform with the Daubechies (D4) wavelet due to its higher energy concentration and employs the u-Net neural network architecture, which incorporates the waveform exclusively at each stage. The outcomes were evaluated using the PSNR and SSIM criteria, and the outcomes were verified by using various waves. The effectiveness of a one-wave network has increased from 0.5% to 1.2%, according to studies done on other datasets
Classy Ensemble: A Novel Ensemble Algorithm for Classification
Venue: arXiv cs.LG 2024
Year: 2024

We present Classy Ensemble, a novel ensemble-generation algorithm for classification tasks, which aggregates models through a weighted combination of per-class accuracy. Tested over 153 machine learning datasets we demonstrate that Classy Ensemble outperforms two other well-known aggregation algorithms -- order-based pruning and clustering-based pruning -- as well as the recently introduced lexigarden ensemble generator. We then present three enhancements: 1) Classy Cluster Ensemble, which combines Classy Ensemble and cluster-based pruning; 2) Deep Learning experiments, showing the merits of Classy Ensemble over four image datasets: Fashion MNIST, CIFAR10, CIFAR100, and ImageNet; and 3) Classy Evolutionary Ensemble, wherein an evolutionary algorithm is used to select the set of models which Classy Ensemble picks from. This latter, combining learning and evolution, resulted in improved performance on the hardest dataset.
FedST: Secure Federated Shapelet Transformation for Time Series Classification
Venue: arXiv cs.LG 2024
Year: 2024

This paper explores how to build a shapelet-based time series classification (TSC) model in the federated learning (FL) scenario, that is, using more data from multiple owners without actually sharing the data. We propose FedST, a novel federated TSC framework extended from a centralized shapelet transformation method. We recognize the federated shapelet search step as the kernel of FedST. Thus, we design a basic protocol for the FedST kernel that we prove to be secure and accurate. However, we identify that the basic protocol suffers from efficiency bottlenecks and the centralized acceleration techniques lose their efficacy due to the security issues. To speed up the federated protocol with security guarantee, we propose several optimizations tailored for the FL setting. Our theoretical analysis shows that the proposed methods are secure and more efficient. We conduct extensive experiments using both synthetic and real-world datasets. Empirical results show that our FedST solution is effective in terms of TSC accuracy, and the proposed optimizations can achieve three orders of magnitude of speedup.
Causal Explanations for Sequential Decision-Making in Multi-Agent Systems
Venue: arXiv cs.AI 2024
Year: 2024

We present CEMA: Causal Explanations in Multi-Agent systems; a framework for creating causal natural language explanations of an agent's decisions in dynamic sequential multi-agent systems to build more trustworthy autonomous agents. Unlike prior work that assumes a fixed causal structure, CEMA only requires a probabilistic model for forward-simulating the state of the system. Using such a model, CEMA simulates counterfactual worlds that identify the salient causes behind the agent's decisions. We evaluate CEMA on the task of motion planning for autonomous driving and test it in diverse simulated scenarios. We show that CEMA correctly and robustly identifies the causes behind the agent's decisions, even when a large number of other agents is present, and show via a user study that CEMA's explanations have a positive effect on participants' trust in autonomous vehicles and are rated as high as high-quality baseline explanations elicited from other participants. We release the collected explanations with annotations as the HEADD dataset.