OpenRaiser/Intern-Atlas huggingface dataset:
FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning
Venue: arXiv cs.LG 2021
Year: 2021

Federated learning aims to collaboratively train a strong global model by accessing users' locally trained models but not their own data. A crucial step is therefore to aggregate local models into a global model, which has been shown challenging when users have non-i.i.d. data. In this paper, we propose a novel aggregation algorithm named FedBE, which takes a Bayesian inference perspective by sampling higher-quality global models and combining them via Bayesian model Ensemble, leading to much robust aggregation. We show that an effective model distribution can be constructed by simply fitting a Gaussian or Dirichlet distribution to the local models. Our empirical studies validate FedBE's superior performance, especially when users' data are not i.i.d. and when the neural networks go deeper. Moreover, FedBE is compatible with recent efforts in regularizing users' model training, making it an easily applicable module: you only need to replace the aggregation method but leave other parts of your federated learning algorithm intact. Our code is publicly available at https://github.com/hongyouc/FedBE.
Imbalanced Image Classification with Complement Cross Entropy
Venue: arXiv cs.CV 2021
Year: 2021

Recently, deep learning models have achieved great success in computer vision applications, relying on large-scale class-balanced datasets. However, imbalanced class distributions still limit the wide applicability of these models due to degradation in performance. To solve this problem, in this paper, we concentrate on the study of cross entropy which mostly ignores output scores on incorrect classes. This work discovers that neutralizing predicted probabilities on incorrect classes improves the prediction accuracy for imbalanced image classification. This paper proposes a simple but effective loss named complement cross entropy based on this finding. The proposed loss makes the ground truth class overwhelm the other classes in terms of softmax probability, by neutralizing probabilities of incorrect classes, without additional training procedures. Along with it, this loss facilitates the models to learn key information especially from samples on minority classes. It ensures more accurate and robust classification results on imbalanced distributions. Extensive experiments on imbalanced datasets demonstrate the effectiveness of the proposed method.
FnF-BFT: Exploring Performance Limits of BFT Protocols
Venue: arXiv cs.DC 2021
Year: 2021

We introduce FnF-BFT, a parallel-leader byzantine fault-tolerant state-machine replication protocol for the partially synchronous model with theoretical performance bounds during synchrony. By allowing all replicas to act as leaders and propose requests independently, FnF-BFT parallelizes the execution of requests. Leader parallelization distributes the load over the entire network -- increasing throughput by overcoming the single-leader bottleneck. We further use historical data to ensure that well-performing replicas are in command. FnF-BFT's communication complexity is linear in the number of replicas during synchrony and thus competitive with state-of-the-art protocols. Finally, with FnF-BFT, we introduce a BFT protocol with performance guarantees in stable network conditions under truly byzantine attacks. A prototype implementation of \prot outperforms (state-of-the-art) HotStuff's throughput, especially as replicas increase, showcasing \prot's significantly improved scaling capabilities.
Witches' Brew: Industrial Scale Data Poisoning via Gradient Matching
Venue: arXiv cs.CV 2021
Year: 2021

Data Poisoning attacks modify training data to maliciously control a model trained on such data. In this work, we focus on targeted poisoning attacks which cause a reclassification of an unmodified test image and as such breach model integrity. We consider a particularly malicious poisoning attack that is both "from scratch" and "clean label", meaning we analyze an attack that successfully works against new, randomly initialized models, and is nearly imperceptible to humans, all while perturbing only a small fraction of the training data. Previous poisoning attacks against deep neural networks in this setting have been limited in scope and success, working only in simplified settings or being prohibitively expensive for large datasets. The central mechanism of the new attack is matching the gradient direction of malicious examples. We analyze why this works, supplement with practical considerations. and show its threat to real-world practitioners, finding that it is the first poisoning method to cause targeted misclassification in modern deep networks trained from scratch on a full-sized, poisoned ImageNet dataset. Finally we demonstrate the limitations of existing defensive strategies against such an attack, concluding that data poisoning is a credible threat, even for large-scale deep learning systems.
AIMx: An Extended Adaptive Integral Method for the Fast Electromagnetic Modeling of Complex Structures
Venue: arXiv cs.CE 2021
Year: 2021

Surface integral equation (SIE) methods are of great interest for the efficient electromagnetic modeling of various devices, from integrated circuits to antenna arrays. Existing acceleration algorithms for SIEs, such as the adaptive integral method (AIM), enable the fast approximation of interactions between well-separated mesh elements. Nearby interactions involve the singularity of the kernel, and must instead be computed accurately with direct integration at each frequency of interest, which can be computationally expensive. We propose a novel algorithm for reducing the cost-per-frequency of near-region computations for both homogeneous and layered background media. In the proposed extended AIM (AIMx), the SIE operators are decomposed into a frequency-independent term containing the singularity of the kernel, and a nonsingular frequency-dependent term. Direct integration is only required for the frequency-independent term, and can be reused at each frequency, leading to significantly faster frequency sweeps. The frequency-dependent term is captured with good accuracy via fast Fourier transform (FFT)-based acceleration even in the near region, as confirmed with an error analysis. The accuracy and efficiency of the proposed method are demonstrated through numerical examples drawn from several applications, and CPU times are significantly reduced by factors ranging from three to 16.
BANANA at WNUT-2020 Task 2: Identifying COVID-19 Information on Twitter by Combining Deep Learning and Transfer Learning Models
Venue: arXiv cs.CL 2021
Year: 2021

The outbreak COVID-19 virus caused a significant impact on the health of people all over the world. Therefore, it is essential to have a piece of constant and accurate information about the disease with everyone. This paper describes our prediction system for WNUT-2020 Task 2: Identification of Informative COVID-19 English Tweets. The dataset for this task contains size 10,000 tweets in English labeled by humans. The ensemble model from our three transformer and deep learning models is used for the final prediction. The experimental result indicates that we have achieved F1 for the INFORMATIVE label on our systems at 88.81% on the test set.
Self-Supervised Contrastive Learning for Code Retrieval and Summarization via Semantic-Preserving Transformations
Venue: arXiv cs.SE 2021
Year: 2021

We propose Corder, a self-supervised contrastive learning framework for source code model. Corder is designed to alleviate the need of labeled data for code retrieval and code summarization tasks. The pre-trained model of Corder can be used in two ways: (1) it can produce vector representation of code which can be applied to code retrieval tasks that do not have labeled data; (2) it can be used in a fine-tuning process for tasks that might still require label data such as code summarization. The key innovation is that we train the source code model by asking it to recognize similar and dissimilar code snippets through a contrastive learning objective. To do so, we use a set of semantic-preserving transformation operators to generate code snippets that are syntactically diverse but semantically equivalent. Through extensive experiments, we have shown that the code models pretrained by Corder substantially outperform the other baselines for code-to-code retrieval, text-to-code retrieval, and code-to-text summarization tasks.
Alternating Beamforming with Intelligent Reflecting Surface Element Allocation
Venue: arXiv cs.IT 2021
Year: 2021

Intelligent reflecting surface (IRS) has become a promising technology to aid next generation wireless communication systems. In this paper, we develop an alternating beamforming technique with a novel concept of IRS element allocation for multiple-input multiple-output systems when a base station supports multiple single antenna users aided with a single IRS. Specifically, we allocate each IRS element separately to each user, thus, in the beamforming stage allowing the IRS element only consider a single user at a time. In result to this separation, the complexity is vastly decreased. The proposed beamforming technique tries to maximize the minimum rate of all users with minimal complexity. In the numerical results, we show that the proposed technique is comparable to the convex optimization-based benchmark with sufficiently less complexity.
Analysing Twitter Semantic Networks: the case of 2018 Italian Elections
Venue: arXiv cs.SI 2021
Year: 2021

Social media play a key role in shaping citizens' political opinion. According to the Eurobarometer, the percentage of EU citizens employing online social networks on a daily basis has increased from 18% in 2010 to 48% in 2019. The entwinement between social media and the unfolding of political dynamics has motivated the interest of researchers for the analysis of users online behavior - with particular emphasis on group polarization during debates and echo-chambers formation. In this context, attention has been predominantly directed towards the study of online relations between users while semantic aspects have remained under-explored. In the present paper, we aim at filling this gap by adopting a two-steps approach. First, we identify the discursive communities animating the political debate in the run up of the 2018 Italian Elections as groups of users with a significantly-similar retweeting behavior. Second, we study the semantic mechanisms that shape their internal discussions by monitoring, on a daily basis, the structural evolution of the semantic networks they induce. Above and beyond specifying the semantic peculiarities of the Italian electoral competition, our approach innovates studies of online political discussions in two main ways. On the one hand, it grounds semantic analysis within users' behaviors by implementing a method, rooted in statistical theory, that guarantees that our inference of socio-semantic structures is not biased by any unsupported assumption about missing information; on the other, it is completely automated as it does not rest upon any manual labelling (either based on the users' features or on their sharing patterns). These elements make our method applicable to any Twitter discussion regardless of the language or the topic addressed.
Automatic detection of microsleep episodes with deep learning
Venue: arXiv cs.LG 2021
Year: 2021

Brief fragments of sleep shorter than 15 s are defined as microsleep episodes (MSEs), often subjectively perceived as sleepiness. Their main characteristic is a slowing in frequency in the electroencephalogram (EEG), similar to stage N1 sleep according to standard criteria. The maintenance of wakefulness test (MWT) is often used in a clinical setting to assess vigilance. Scoring of the MWT in most sleep-wake centers is limited to classical definition of sleep (30-s epochs), and MSEs are mostly not considered in the absence of established scoring criteria defining MSEs but also because of the laborious work. We aimed for automatic detection of MSEs with machine learning, i.e. with deep learning based on raw EEG and EOG data as input. We analyzed MWT data of 76 patients. Experts visually scored wakefulness, and according to recently developed scoring criteria MSEs, microsleep episode candidates (MSEc), and episodes of drowsiness (ED). We implemented segmentation algorithms based on convolutional neural networks (CNNs) and a combination of a CNN with a long-short term memory (LSTM) network. A LSTM network is a type of a recurrent neural network which has a memory for past events and takes them into account. Data of 53 patients were used for training of the classifiers, 12 for validation and 11 for testing. Our algorithms showed a good performance close to human experts. The detection was very good for wakefulness and MSEs and poor for MSEc and ED, similar to the low inter-expert reliability for these borderline segments. We provide a proof of principle that it is feasible to reliably detect MSEs with deep neuronal networks based on raw EEG and EOG data with a performance close to that of human experts. Code of algorithms ( https://github.com/alexander-malafeev/microsleep-detection ) and data ( https://zenodo.org/record/3251716 ) are available.
Information Theoretic Meta Learning with Gaussian Processes
Venue: arXiv cs.LG 2021
Year: 2021

We formulate meta learning using information theoretic concepts; namely, mutual information and the information bottleneck. The idea is to learn a stochastic representation or encoding of the task description, given by a training set, that is highly informative about predicting the validation set. By making use of variational approximations to the mutual information, we derive a general and tractable framework for meta learning. This framework unifies existing gradient-based algorithms and also allows us to derive new algorithms. In particular, we develop a memory-based algorithm that uses Gaussian processes to obtain non-parametric encoding representations. We demonstrate our method on a few-shot regression problem and on four few-shot classification problems, obtaining competitive accuracy when compared to existing baselines.
On principal types and well-foundedness of the cummulativity relation in ECC
Venue: arXiv cs.LO 2021
Year: 2021

When we investigate a type system, it is helpful if we can establish the well-foundedness of types or terms with respect to a certain hierarchy, and the Extended Calculus of Constructions (called $ECC$, defined and studied comprehensively in [Luo,1994]) is no exception. However, under a very natural hierarchy relation (called the cumulativity relation in [Luo,1994]), the well-foundedness of the hierarchy does not hold generally. In this article,we show that the cumulativity relation is well-founded if it is restricted to one of the following two natural families of terms: \begin{enumerate} \item types in a valid context \item terms having normal forms \end{enumerate} Also, we give an independent proof of the existence of principal types in $ECC$ since it is used in the proof of well-foundedness of cumulativity relation in a valid context although it is often proved by utilizing the well-foundedness of the hierarchy, which would make our argument circular if adopted.
A Distance-preserving Matrix Sketch
Venue: arXiv cs.HC 2021
Year: 2021

Visualizing very large matrices involves many formidable problems. Various popular solutions to these problems involve sampling, clustering, projection, or feature selection to reduce the size and complexity of the original task. An important aspect of these methods is how to preserve relative distances between points in the higher-dimensional space after reducing rows and columns to fit in a lower dimensional space. This aspect is important because conclusions based on faulty visual reasoning can be harmful. Judging dissimilar points as similar or similar points as dissimilar on the basis of a visualization can lead to false conclusions. To ameliorate this bias and to make visualizations of very large datasets feasible, we introduce two new algorithms that respectively select a subset of rows and columns of a rectangular matrix. This selection is designed to preserve relative distances as closely as possible. We compare our matrix sketch to more traditional alternatives on a variety of artificial and real datasets.
Attribute Privacy: Framework and Mechanisms
Venue: arXiv cs.CR 2021
Year: 2021

Ensuring the privacy of training data is a growing concern since many machine learning models are trained on confidential and potentially sensitive data. Much attention has been devoted to methods for protecting individual privacy during analyses of large datasets. However in many settings, global properties of the dataset may also be sensitive (e.g., mortality rate in a hospital rather than presence of a particular patient in the dataset). In this work, we depart from individual privacy to initiate the study of attribute privacy, where a data owner is concerned about revealing sensitive properties of a whole dataset during analysis. We propose definitions to capture \emph{attribute privacy} in two relevant cases where global attributes may need to be protected: (1) properties of a specific dataset and (2) parameters of the underlying distribution from which dataset is sampled. We also provide two efficient mechanisms and one inefficient mechanism that satisfy attribute privacy for these settings. We base our results on a novel use of the Pufferfish framework to account for correlations across attributes in the data, thus addressing "the challenging problem of developing Pufferfish instantiations and algorithms for general aggregate secrets" that was left open by \cite{kifer2014pufferfish}.
Tactical Decision Making for Emergency Vehicles Based on A Combinational Learning Method
Venue: arXiv cs.AI 2021
Year: 2021

Increasing the response time of emergency vehicles(EVs) could lead to an immeasurable loss of property and life. On this account, tactical decision making for EVs' microscopic control remains an indispensable issue to be improved. In this paper, a rule-based avoiding strategy(AS) is devised, that CVs in the prioritized zone ahead of EV should accelerate or change their lane to avoid it. Besides, a novel DQN method with speed-adaptive compact state space (SC-DQN) is put forward to fit in EVs' high-speed feature and generalize in various road topologies. Afterward, the execution of AS feedback to the input of SC-DQN so that they joint organically as a combinational method. The following approach reveals that DRL could complement rule-based avoiding strategy in generalization, and on the contrary, the rule-based avoiding strategy could complement DRL in stability, and their combination could lead to less response time, lower collision rate and smoother trajectory.
Optimal Trajectory Planning for Cinematography with Multiple Unmanned Aerial Vehicles
Venue: arXiv cs.RO 2021
Year: 2021

This paper presents a method for planning optimal trajectories with a team of Unmanned Aerial Vehicles (UAVs) performing autonomous cinematography. The method is able to plan trajectories online and in a distributed manner, providing coordination between the UAVs. We propose a novel non-linear formulation for this challenging problem of computing multi-UAV optimal trajectories for cinematography; integrating UAVs dynamics and collision avoidance constraints, together with cinematographic aspects like smoothness, gimbal mechanical limits and mutual camera visibility. We integrate our method within a hardware and software architecture for UAV cinematography that was previously developed within the framework of the MultiDrone project; and demonstrate its use with different types of shots filming a moving target outdoors. We provide extensive experimental results both in simulation and field experiments. We analyze the performance of the method and prove that it is able to compute online smooth trajectories, reducing jerky movements and complying with cinematography constraints.
Graph-based keyword search in heterogeneous data sources
Venue: arXiv cs.DB 2021
Year: 2021

Data journalism is the field of investigative journalism which focuses on digital data by treating them as first-class citizens. Following the trends in human activity, which leaves strong digital traces, data journalism becomes increasingly important. However, as the number and the diversity of data sources increase, heterogeneous data models with different structure, or even no structure at all, need to be considered in query answering. Inspired by our collaboration with Le Monde, a leading French newspaper, we designed a novel query algorithm for exploiting such heterogeneous corpora through keyword search. We model our underlying data as graphs and, given a set of search terms, our algorithm nds links between them within and across the heterogeneous datasets included in the graph. We draw inspiration from prior work on keyword search in structured and unstructured data, which we extend with the data heterogeneity dimension, which makes the keyword search problem computationally harder. We implement our algorithm and we evaluate its performance using synthetic and real-world datasets.
Improved Exploration in Factored Average-Reward MDPs
Venue: arXiv cs.LG 2021
Year: 2021

We consider a regret minimization task under the average-reward criterion in an unknown Factored Markov Decision Process (FMDP). More specifically, we consider an FMDP where the state-action space $\mathcal X$ and the state-space $\mathcal S$ admit the respective factored forms of $\mathcal X = \otimes_{i=1}^n \mathcal X_i$ and $\mathcal S=\otimes_{i=1}^m \mathcal S_i$, and the transition and reward functions are factored over $\mathcal X$ and $\mathcal S$. Assuming known factorization structure, we introduce a novel regret minimization strategy inspired by the popular UCRL2 strategy, called DBN-UCRL, which relies on Bernstein-type confidence sets defined for individual elements of the transition function. We show that for a generic factorization structure, DBN-UCRL achieves a regret bound, whose leading term strictly improves over existing regret bounds in terms of the dependencies on the size of $\mathcal S_i$'s and the involved diameter-related terms. We further show that when the factorization structure corresponds to the Cartesian product of some base MDPs, the regret of DBN-UCRL is upper bounded by the sum of regret of the base MDPs. We demonstrate, through numerical experiments on standard environments, that DBN-UCRL enjoys substantially improved regret empirically over existing algorithms that have frequentist regret guarantees.