OpenRaiser/Intern-Atlas huggingface dataset:
Towards Fairness-Aware Federated Learning
Venue: arXiv cs.LG 2023
Year: 2023

Recent advances in Federated Learning (FL) have brought large-scale collaborative machine learning opportunities for massively distributed clients with performance and data privacy guarantees. However, most current works focus on the interest of the central controller in FL,and overlook the interests of the FL clients. This may result in unfair treatment of clients that discourages them from actively participating in the learning process and damages the sustainability of the FL ecosystem. Therefore, the topic of ensuring fairness in FL is attracting a great deal of research interest. In recent years, diverse Fairness-Aware FL (FAFL) approaches have been proposed in an effort to achieve fairness in FL from different perspectives. However, there is no comprehensive survey that helps readers gain insight into this interdisciplinary field. This paper aims to provide such a survey. By examining the fundamental and simplifying assumptions, as well as the notions of fairness adopted by existing literature in this field, we propose a taxonomy of FAFL approaches covering major steps in FL, including client selection, optimization, contribution evaluation and incentive distribution. In addition, we discuss the main metrics for experimentally evaluating the performance of FAFL approaches, and suggest promising future research directions towards FAFL.
A Theoretical Analysis on Independence-driven Importance Weighting for Covariate-shift Generalization
Venue: arXiv cs.LG 2023
Year: 2023

Covariate-shift generalization, a typical case in out-of-distribution (OOD) generalization, requires a good performance on the unknown test distribution, which varies from the accessible training distribution in the form of covariate shift. Recently, independence-driven importance weighting algorithms in stable learning literature have shown empirical effectiveness to deal with covariate-shift generalization on several learning models, including regression algorithms and deep neural networks, while their theoretical analyses are missing. In this paper, we theoretically prove the effectiveness of such algorithms by explaining them as feature selection processes. We first specify a set of variables, named minimal stable variable set, that is the minimal and optimal set of variables to deal with covariate-shift generalization for common loss functions, such as the mean squared loss and binary cross-entropy loss. Afterward, we prove that under ideal conditions, independence-driven importance weighting algorithms could identify the variables in this set. Analysis of asymptotic properties is also provided. These theories are further validated in several synthetic experiments.
Contextual Semantic Parsing for Multilingual Task-Oriented Dialogues
Venue: arXiv cs.CL 2023
Year: 2023

Robust state tracking for task-oriented dialogue systems currently remains restricted to a few popular languages. This paper shows that given a large-scale dialogue data set in one language, we can automatically produce an effective semantic parser for other languages using machine translation. We propose automatic translation of dialogue datasets with alignment to ensure faithful translation of slot values and eliminate costly human supervision used in previous benchmarks. We also propose a new contextual semantic parsing model, which encodes the formal slots and values, and only the last agent and user utterances. We show that the succinct representation reduces the compounding effect of translation errors, without harming the accuracy in practice. We evaluate our approach on several dialogue state tracking benchmarks. On RiSAWOZ, CrossWOZ, CrossWOZ-EN, and MultiWOZ-ZH datasets we improve the state of the art by 11%, 17%, 20%, and 0.3% in joint goal accuracy. We present a comprehensive error analysis for all three datasets showing erroneous annotations can lead to misguided judgments on the quality of the model. Finally, we present RiSAWOZ English and German datasets, created using our translation methodology. On these datasets, accuracy is within 11% of the original showing that high-accuracy multilingual dialogue datasets are possible without relying on expensive human annotations. We release our datasets and software open source.
Universal Private Estimators
Venue: arXiv cs.CR 2023
Year: 2023

We present \textit{universal} estimators for the statistical mean, variance, and scale (in particular, the interquartile range) under pure differential privacy. These estimators are universal in the sense that they work on an arbitrary, unknown continuous distribution $\mathcal{P}$ over $\mathbb{R}$, while yielding strong utility guarantees except for ill-behaved $\mathcal{P}$. For certain distribution families like Gaussians or heavy-tailed distributions, we show that our universal estimators match or improve existing estimators, which are often specifically designed for the given family and under \textit{a priori} boundedness assumptions on the mean and variance of $\mathcal{P}$. This is the first time these boundedness assumptions are removed under pure differential privacy. The main technical tools in our development are instance-optimal empirical estimators for the mean and quantiles over the unbounded integer domain, which can be of independent interest.
Parallel Global Edge Switching for the Uniform Sampling of Simple Graphs with Prescribed Degrees
Venue: arXiv cs.DS 2023
Year: 2023

The uniform sampling of simple graphs matching a prescribed degree sequence is an important tool in network science, e.g. to construct graph generators or null-models. Here, the Edge Switching Markov Chain (ES-MC) is a common choice. Given an arbitrary simple graph with the required degree sequence, ES-MC carries out a large number of small changes, called edge switches, to eventually obtain a uniform sample. In practice, reasonably short runs efficiently yield approximate uniform samples. In this work, we study the problem of executing edge switches in parallel. We discuss parallelizations of ES-MC, but find that this approach suffers from complex dependencies between edge switches. For this reason, we propose the Global Edge Switching Markov Chain (G-ES-MC), an ES-MC variant with simpler dependencies. We show that G-ES-MC converges to the uniform distribution and design shared-memory parallel algorithms for ES-MC and G-ES-MC. In an empirical evaluation, we provide evidence that G-ES-MC requires not more switches than ES-MC (and often fewer), and demonstrate the efficiency and scalability of our parallel G-ES-MC implementation.
Physics-Informed Neural Operator for Learning Partial Differential Equations
Venue: arXiv cs.LG 2023
Year: 2023

In this paper, we propose physics-informed neural operators (PINO) that combine training data and physics constraints to learn the solution operator of a given family of parametric Partial Differential Equations (PDE). PINO is the first hybrid approach incorporating data and PDE constraints at different resolutions to learn the operator. Specifically, in PINO, we combine coarse-resolution training data with PDE constraints imposed at a higher resolution. The resulting PINO model can accurately approximate the ground-truth solution operator for many popular PDE families and shows no degradation in accuracy even under zero-shot super-resolution, i.e., being able to predict beyond the resolution of training data. PINO uses the Fourier neural operator (FNO) framework that is guaranteed to be a universal approximator for any continuous operator and discretization-convergent in the limit of mesh refinement. By adding PDE constraints to FNO at a higher resolution, we obtain a high-fidelity reconstruction of the ground-truth operator. Moreover, PINO succeeds in settings where no training data is available and only PDE constraints are imposed, while previous approaches, such as the Physics-Informed Neural Network (PINN), fail due to optimization challenges, e.g., in multi-scale dynamic systems such as Kolmogorov flows.
SIG-VC: A Speaker Information Guided Zero-shot Voice Conversion System for Both Human Beings and Machines
Venue: arXiv cs.SD 2023
Year: 2023

Nowadays, as more and more systems achieve good performance in traditional voice conversion (VC) tasks, people's attention gradually turns to VC tasks under extreme conditions. In this paper, we propose a novel method for zero-shot voice conversion. We aim to obtain intermediate representations for speaker-content disentanglement of speech to better remove speaker information and get pure content information. Accordingly, our proposed framework contains a module that removes the speaker information from the acoustic feature of the source speaker. Moreover, speaker information control is added to our system to maintain the voice cloning performance. The proposed system is evaluated by subjective and objective metrics. Results show that our proposed system significantly reduces the trade-off problem in zero-shot voice conversion, while it also manages to have high spoofing power to the speaker verification system.
Succinct Data Structure for Path Graphs
Venue: arXiv cs.DS 2023
Year: 2023

We consider the problem of designing a succinct data structure for {\it path graphs} (which are a proper subclass of chordal graphs and a proper superclass of interval graphs) on $n$ vertices while supporting degree, adjacency, and neighborhood queries efficiently. We provide the following two solutions for this problem: - an $n \log n+o(n \log n)$-bit succinct data structure that supports adjacency query in $O(\log n)$ time, neighborhood query in $O(d \log n)$ time and finally, degree query in $\min\{O(\log^2 n), O(d \log n)\}$ where $d$ is the degree of the queried vertex. - an $O(n \log^2 n)$-bit space-efficient data structure that supports adjacency and degree queries in $O(1)$ time, and the neighborhood query in $O(d)$ time where $d$ is the degree of the queried vertex. Central to our data structures is the usage of the classical heavy path decomposition by Sleator and Tarjan~\cite{ST}, followed by a careful bookkeeping using an orthogonal range search data structure using wavelet trees~\cite{Makinen2007} among others, which maybe of independent interest for designing succinct data structures for other graph classes.
Attack time analysis in dynamic attack trees via integer linear programming
Venue: arXiv cs.CR 2023
Year: 2023

Attack trees (ATs) are an important tool in security analysis, and an important part of AT analysis is computing metrics. However, metric computation is NP-complete in general. In this paper, we showcase the use of mixed integer linear programming (MILP) as a tool for quantitative analysis. Specifically, we use MILP to solve the open problem of calculating the min time metric of dynamic ATs, i.e., the minimal time to attack a system. We also present two other tools to further improve our MILP method: First, we show how the computation can be sped up by identifying the modules of an AT, i.e. subtrees connected to the rest of the AT via only one node. Second, we define a general semantics for dynamic ATs that significantly relaxes the restrictions on attack trees compared to earlier work, allowing us to apply our methods to a wide variety of ATs. Experiments on a synthetic testing set of large ATs verify that both the integer linear programming approach and modular analysis considerably decrease the computation time of attack time analysis.
Towards Green Automated Machine Learning: Status Quo and Future Directions
Venue: arXiv cs.LG 2023
Year: 2023

Automated machine learning (AutoML) strives for the automatic configuration of machine learning algorithms and their composition into an overall (software) solution - a machine learning pipeline - tailored to the learning task (dataset) at hand. Over the last decade, AutoML has developed into an independent research field with hundreds of contributions. At the same time, AutoML is being criticised for its high resource consumption as many approaches rely on the (costly) evaluation of many machine learning pipelines, as well as the expensive large scale experiments across many datasets and approaches. In the spirit of recent work on Green AI, this paper proposes Green AutoML, a paradigm to make the whole AutoML process more environmentally friendly. Therefore, we first elaborate on how to quantify the environmental footprint of an AutoML tool. Afterward, different strategies on how to design and benchmark an AutoML tool wrt. their "greenness", i.e. sustainability, are summarized. Finally, we elaborate on how to be transparent about the environmental footprint and what kind of research incentives could direct the community into a more sustainable AutoML research direction. Additionally, we propose a sustainability checklist to be attached to every AutoML paper featuring all core aspects of Green AutoML.
Does the Brain Infer Invariance Transformations from Graph Symmetries?
Venue: arXiv cs.NE 2023
Year: 2023

The invariance of natural objects under perceptual changes is possibly encoded in the brain by symmetries in the graph of synaptic connections. The graph can be established via unsupervised learning in a biologically plausible process across different perceptual modalities. This hypothetical encoding scheme is supported by the correlation structure of naturalistic audio and image data and it predicts a neural connectivity architecture which is consistent with many empirical observations about primary sensory cortex.
Learning Augmentation Distributions using Transformed Risk Minimization
Venue: arXiv cs.LG 2023
Year: 2023

We propose a new \emph{Transformed Risk Minimization} (TRM) framework as an extension of classical risk minimization. In TRM, we optimize not only over predictive models, but also over data transformations; specifically over distributions thereof. As a key application, we focus on learning augmentations; for instance appropriate rotations of images, to improve classification performance with a given class of predictors. Our TRM method (1) jointly learns transformations and models in a \emph{single training loop}, (2) works with any training algorithm applicable to standard risk minimization, and (3) handles any transforms, such as discrete and continuous classes of augmentations. To avoid overfitting when implementing empirical transformed risk minimization, we propose a novel regularizer based on PAC-Bayes theory. For learning augmentations of images, we propose a new parametrization of the space of augmentations via a stochastic composition of blocks of geometric transforms. This leads to the new \emph{Stochastic Compositional Augmentation Learning} (SCALE) algorithm. The performance of TRM with SCALE compares favorably to prior methods on CIFAR10/100. Additionally, we show empirically that SCALE can correctly learn certain symmetries in the data distribution (recovering rotations on rotated MNIST) and can also improve calibration of the learned model.
Enabling human-centered AI: A new junction and shared journey between AI and HCI communities
Venue: arXiv cs.HC 2023
Year: 2023

Artificial intelligence (AI) has brought benefits, but it may also cause harm if it is not appropriately developed. Current development is mainly driven by a "technology-centered" approach, causing many failures. For example, the AI Incident Database has documented over a thousand AI-related accidents. To address these challenges, a human-centered AI (HCAI) approach has been promoted and has received a growing level of acceptance over the last few years. HCAI calls for combining AI with user experience (UX) design will enable the development of AI systems (e.g., autonomous vehicles, intelligent user interfaces, or intelligent decision-making systems) to achieve its design goals such as usable/explainable AI, human-controlled AI, and ethical AI. While HCAI promotion continues, it has not specifically addressed the collaboration between AI and human-computer interaction (HCI) communities, resulting in uncertainty about what action should be taken by both sides to apply HCAI in developing AI systems. This Viewpoint focuses on the collaboration between the AI and HCI communities, which leads to nine recommendations for effective collaboration to enable HCAI in developing AI systems.
Some Doxastic Łukasiewicz Logic
Venue: arXiv cs.LO 2023
Year: 2023

We propose a doxastic Łukasiewicz logic \textbf{BŁ} that is sound and complete with respect to the class of Kripke-based models in which atomic propositions and accessibility relations are both infinitely valued in the standard MV-algebra [0,1]. We also introduce some extensions of \textbf{BŁ} corresponding to axioms \textbf{D}, \textbf{4}, and \textbf{T} of classical epistemic logic. Furthermore, completeness of these extensions are established corresponding to the appropriate classes of models.
Pattern-Division Multiplexing for Multi-User Continuous-Aperture MIMO
Venue: arXiv cs.IT 2023
Year: 2023

In recent years, thanks to the advances in meta-materials, the concept of continuous-aperture MIMO (CAP-MIMO) is reinvestigated to achieve improved communication performance with limited antenna apertures. Unlike the classical MIMO composed of discrete antennas, CAP-MIMO has a quasi-continuous antenna surface, which is expected to generate any current distribution (i.e., pattern) and induce controllable spatial electromagnetic (EM) waves. In this way, the information is directly modulated on the EM waves, which makes it promising to approach the ultimate capacity of finite apertures. The pattern design is the key factor to determine the communication performance of CAP-MIMO, but it has not been well studied in the literature. In this paper, we develop pattern-division multiplexing (PDM) to design the patterns for CAP-MIMO. Specifically, we first study and model a typical multi-user CAP-MIMO system, which allows us to formulate the sum-rate maximization problem. Then, we develop a general PDM technique to transform the design of the continuous pattern functions to the design of their projection lengths on finite orthogonal bases, which can overcome the challenge of functional programming. Utilizing PDM, we further propose a block coordinate descent (BCD) based pattern design scheme to solve the formulated sum-rate maximization problem. Simulation results show that, the sum-rate achieved by the proposed scheme is higher than that achieved by benchmark schemes, which demonstrates the effectiveness of the developed PDM for CAP-MIMO.
Unicode at Gigabytes per Second
Venue: arXiv cs.PL 2023
Year: 2023

We often represent text using Unicode formats (UTF-8 and UTF-16). The UTF-8 format is increasingly popular, especially on the web (XML, HTML, JSON, Rust, Go, Swift, Ruby). The UTF-16 format is most common in Java, .NET, and inside operating systems such as Windows. Software systems frequently have to convert text from one Unicode format to the other. While recent disks have bandwidths of 5 GiB/s or more, conventional approaches transcode non-ASCII text at a fraction of a gigabyte per second. We show that we can validate and transcode Unicode text at gigabytes per second on current systems (x64 and ARM) without sacrificing safety. Our open-source library can be ten times faster than the popular ICU library on non-ASCII strings and even faster on ASCII strings.
Deep Neural Networks for Rank-Consistent Ordinal Regression Based On Conditional Probabilities
Venue: arXiv cs.LG 2023
Year: 2023

In recent times, deep neural networks achieved outstanding predictive performance on various classification and pattern recognition tasks. However, many real-world prediction problems have ordinal response variables, and this ordering information is ignored by conventional classification losses such as the multi-category cross-entropy. Ordinal regression methods for deep neural networks address this. One such method is the CORAL method, which is based on an earlier binary label extension framework and achieves rank consistency among its output layer tasks by imposing a weight-sharing constraint. However, while earlier experiments showed that CORAL's rank consistency is beneficial for performance, it is limited by a weight-sharing constraint in a neural network's fully connected output layer, which may restrict the expressiveness and capacity of a network trained using CORAL. We propose a new method for rank-consistent ordinal regression without this limitation. Our rank-consistent ordinal regression framework (CORN) achieves rank consistency by a novel training scheme. This training scheme uses conditional training sets to obtain the unconditional rank probabilities through applying the chain rule for conditional probability distributions. Experiments on various datasets demonstrate the efficacy of the proposed method to utilize the ordinal target information, and the absence of the weight-sharing restriction improves the performance substantially compared to the CORAL reference approach. Additionally, the suggested CORN method is not tied to any specific architecture and can be utilized with any deep neural network classifier to train it for ordinal regression tasks.
Hierarchical Knowledge Guided Learning for Real-world Retinal Diseases Recognition
Venue: arXiv cs.CV 2023
Year: 2023

In the real world, medical datasets often exhibit a long-tailed data distribution (i.e., a few classes occupy the majority of the data, while most classes have only a limited number of samples), which results in a challenging long-tailed learning scenario. Some recently published datasets in ophthalmology AI consist of more than 40 kinds of retinal diseases with complex abnormalities and variable morbidity. Nevertheless, more than 30 conditions are rarely seen in global patient cohorts. From a modeling perspective, most deep learning models trained on these datasets may lack the ability to generalize to rare diseases where only a few available samples are presented for training. In addition, there may be more than one disease for the presence of the retina, resulting in a challenging label co-occurrence scenario, also known as \textit{multi-label}, which can cause problems when some re-sampling strategies are applied during training. To address the above two major challenges, this paper presents a novel method that enables the deep neural network to learn from a long-tailed fundus database for various retinal disease recognition. Firstly, we exploit the prior knowledge in ophthalmology to improve the feature representation using a hierarchy-aware pre-training. Secondly, we adopt an instance-wise class-balanced sampling strategy to address the label co-occurrence issue under the long-tailed medical dataset scenario. Thirdly, we introduce a novel hybrid knowledge distillation to train a less biased representation and classifier. We conducted extensive experiments on four databases, including two public datasets and two in-house databases with more than one million fundus images. The experimental results demonstrate the superiority of our proposed methods with recognition accuracy outperforming the state-of-the-art competitors, especially for these rare diseases.