OpenRaiser/Intern-Atlas huggingface dataset:
Performance Analysis of Out-of-Distribution Detection on Various Trained Neural Networks
Venue: arXiv cs.LG 2021
Year: 2021

Several areas have been improved with Deep Learning during the past years. For non-safety related products adoption of AI and ML is not an issue, whereas in safety critical applications, robustness of such approaches is still an issue. A common challenge for Deep Neural Networks (DNN) occur when exposed to out-of-distribution samples that are previously unseen, where DNNs can yield high confidence predictions despite no prior knowledge of the input. In this paper we analyse two supervisors on two well-known DNNs with varied setups of training and find that the outlier detection performance improves with the quality of the training procedure. We analyse the performance of the supervisor after each epoch during the training cycle, to investigate supervisor performance as the accuracy converges. Understanding the relationship between training results and supervisor performance is valuable to improve robustness of the model and indicates where more work has to be done to create generalized models for safety critical applications.
Graph Classification by Mixture of Diverse Experts
Venue: arXiv cs.LG 2021
Year: 2021

Graph classification is a challenging research problem in many applications across a broad range of domains. In these applications, it is very common that class distribution is imbalanced. Recently, Graph Neural Network (GNN) models have achieved superior performance on various real-world datasets. Despite their success, most of current GNN models largely overlook the important setting of imbalanced class distribution, which typically results in prediction bias towards majority classes. To alleviate the prediction bias, we propose to leverage semantic structure of dataset based on the distribution of node embedding. Specifically, we present GraphDIVE, a general framework leveraging mixture of diverse experts (i.e., graph classifiers) for imbalanced graph classification. With a divide-and-conquer principle, GraphDIVE employs a gating network to partition an imbalanced graph dataset into several subsets. Then each expert network is trained based on its corresponding subset. Experiments on real-world imbalanced graph datasets demonstrate the effectiveness of GraphDIVE.
A city of cities: Measuring how 15-minutes urban accessibility shapes human mobility in Barcelona
Venue: arXiv cs.SI 2021
Year: 2021

As cities expand, human mobility has become a central focus of urban planning and policy making to make cities more inclusive and sustainable. Initiatives such as the "15-minutes city" have been put in place to shift the attention from monocentric city configurations to polycentric structures, increasing the availability and diversity of local urban amenities. Ultimately they expect to increase local walkability and increase mobility within residential areas. While we know how urban amenities influence human mobility at the city level, little is known about spatial variations in this relationship. Here, we use mobile phone, census, and volunteered geographical data to measure geographic variations in the relationship between origin-destination flows and local urban accessibility in Barcelona. Using a Negative Binomial Geographically Weighted Regression model, we show that, globally, people tend to visit neighborhoods with better access to education and retail. Locally, these and other features change in sign and magnitude through the different neighborhoods of the city in ways that are not explained by administrative boundaries, and that provide deeper insights regarding urban characteristics such as rental prices. In conclusion, our work suggests that the qualities of a 15-minutes city can be measured at scale, delivering actionable insights on the polycentric structure of cities, and how people use and access this structure.
Aperiodic two-way transducers and FO-transductions
Venue: arXiv cs.FL 2021
Year: 2021

Deterministic two-way transducers on finite words have been shown by Engelfriet and Hoogeboom to have the same expressive power as MSO-transductions. We introduce a notion of aperiodicity for these transducers and we show that aperiodic transducers correspond exactly to FO-transductions. This lifts to transducers the classical equivalence for languages between FO-definability, recognition by aperiodic monoids and acceptance by counter-free automata.
A Note on the Join of Varieties of Monoids with LI
Venue: arXiv cs.FL 2021
Year: 2021

In this note, we give a characterisation in terms of identities of the join of $\mathbf{V}$ with the variety of finite locally trivial semigroups $\mathbf{LI}$ for several well-known varieties of finite monoids $\mathbf{V}$ by using classical algebraic-automata-theoretic techniques. To achieve this, we use the new notion of essentially-$\mathbf{V}$ stamps defined by Grosshans, McKenzie and Segoufin and show that it actually coincides with the join of $\mathbf{V}$ and $\mathbf{LI}$ precisely when some natural condition on the variety of languages corresponding to $\mathbf{V}$ is verified.This work is a kind of rediscovery of the work of J. C. Costa around 20 years ago from a rather different angle, since Costa's work relies on the use of advanced developments in profinite topology, whereas what is presented here essentially uses an algebraic, language-based approach.
Pursuer Assignment and Control Strategies in Multi-agent Pursuit-Evasion Under Uncertainties
Venue: arXiv cs.RO 2021
Year: 2021

We consider a pursuit-evasion problem with a heterogeneous team of multiple pursuers and multiple evaders. Although both the pursuers (robots) and the evaders are aware of each others' control and assignment strategies, they do not have exact information about the other type of agents' location or action. Using only noisy on-board sensors the pursuers (or evaders) make probabilistic estimation of positions of the evaders (or pursuers). Each type of agent use Markov localization to update the probability distribution of the other type. A search-based control strategy is developed for the pursuers that intrinsically takes the probability distribution of the evaders into account. Pursuers are assigned using an assignment algorithm that takes redundancy (i.e., an excess in the number of pursuers than the number of evaders) into account, such that the total or maximum estimated time to capture the evaders is minimized. In this respect we assume the pursuers to have clear advantage over the evaders. However, the objective of this work is to use assignment strategies that minimize the capture time. This assignment strategy is based on a modified Hungarian algorithm as well as a novel algorithm for determining assignment of redundant pursuers. The evaders, in order to effectively avoid the pursuers, predict the assignment based on their probabilistic knowledge of the pursuers and use a control strategy to actively move away from those pursues. Our experimental evaluation shows that the redundant assignment algorithm performs better than an alternative nearest-neighbor based assignment algorithm.
Competing Adaptive Networks
Venue: arXiv cs.MA 2021
Year: 2021

Adaptive networks have the capability to pursue solutions of global stochastic optimization problems by relying only on local interactions within neighborhoods. The diffusion of information through repeated interactions allows for globally optimal behavior, without the need for central coordination. Most existing strategies are developed for cooperative learning settings, where the objective of the network is common to all agents. We consider in this work a team setting, where a subset of the agents form a team with a common goal while competing with the remainder of the network. We develop an algorithm for decentralized competition among teams of adaptive agents, analyze its dynamics and present an application in the decentralized training of generative adversarial neural networks.
Memory Enhanced Embedding Learning for Cross-Modal Video-Text Retrieval
Venue: arXiv cs.CV 2021
Year: 2021

Cross-modal video-text retrieval, a challenging task in the field of vision and language, aims at retrieving corresponding instance giving sample from either modality. Existing approaches for this task all focus on how to design encoding model through a hard negative ranking loss, leaving two key problems unaddressed during this procedure. First, in the training stage, only a mini-batch of instance pairs is available in each iteration. Therefore, this kind of hard negatives is locally mined inside a mini-batch while ignoring the global negative samples among the dataset. Second, there are many text descriptions for one video and each text only describes certain local features of a video. Previous works for this task did not consider to fuse the multiply texts corresponding to a video during the training. In this paper, to solve the above two problems, we propose a novel memory enhanced embedding learning (MEEL) method for videotext retrieval. To be specific, we construct two kinds of memory banks respectively: cross-modal memory module and text center memory module. The cross-modal memory module is employed to record the instance embeddings of all the datasets for global negative mining. To avoid the fast evolving of the embedding in the memory bank during training, we utilize a momentum encoder to update the features by a moving-averaging strategy. The text center memory module is designed to record the center information of the multiple textual instances corresponding to a video, and aims at bridging these textual instances together. Extensive experimental results on two challenging benchmarks, i.e., MSR-VTT and VATEX, demonstrate the effectiveness of the proposed method.
The Sample Complexity of Distribution-Free Parity Learning in the Robust Shuffle Model
Venue: arXiv cs.LG 2021
Year: 2021

We provide a lowerbound on the sample complexity of distribution-free parity learning in the realizable case in the shuffle model of differential privacy. Namely, we show that the sample complexity of learning $d$-bit parity functions is $Ω(2^{d/2})$. Our result extends a recent similar lowerbound on the sample complexity of private agnostic learning of parity functions in the shuffle model by Cheu and Ullman. We also sketch a simple shuffle model protocol demonstrating that our results are tight up to $poly(d)$ factors.
Evaluation of Correctness in Unsupervised Many-to-Many Image Translation
Venue: arXiv cs.CV 2021
Year: 2021

Given an input image from a source domain and a guidance image from a target domain, unsupervised many-to-many image-to-image (UMMI2I) translation methods seek to generate a plausible example from the target domain that preserves domain-invariant information of the input source image and inherits the domain-specific information from the guidance image. For example, when translating female faces to male faces, the generated male face should have the same expression, pose and hair color as the input female image, and the same facial hairstyle and other male-specific attributes as the guidance male image. Current state-of-the art UMMI2I methods generate visually pleasing images, but, since for most pairs of real datasets we do not know which attributes are domain-specific and which are domain-invariant, the semantic correctness of existing approaches has not been quantitatively evaluated yet. In this paper, we propose a set of benchmarks and metrics for the evaluation of semantic correctness of these methods. We provide an extensive study of existing state-of-the-art UMMI2I translation methods, showing that all methods, to different degrees, fail to infer which attributes are domain-specific and which are domain-invariant from data, and mostly rely on inductive biases hard-coded into their architectures.
Retraining DistilBERT for a Voice Shopping Assistant by Using Universal Dependencies
Venue: arXiv cs.AI 2021
Year: 2021

In this work, we retrained the distilled BERT language model for Walmart's voice shopping assistant on retail domain-specific data. We also injected universal syntactic dependencies to improve the performance of the model further. The Natural Language Understanding (NLU) components of the voice assistants available today are heavily dependent on language models for various tasks. The generic language models such as BERT and RoBERTa are useful for domain-independent assistants but have limitations when they cater to a specific domain. For example, in the shopping domain, the token 'horizon' means a brand instead of its literal meaning. Generic models are not able to capture such subtleties. So, in this work, we retrained a distilled version of the BERT language model on retail domain-specific data for Walmart's voice shopping assistant. We also included universal dependency-based features in the retraining process further to improve the performance of the model on downstream tasks. We evaluated the performance of the retrained language model on four downstream tasks, including intent-entity detection, sentiment analysis, voice title shortening and proactive intent suggestion. We observed an increase in the performance of all the downstream tasks of up to 1.31% on average.
Privacy and Trust Redefined in Federated Machine Learning
Venue: arXiv cs.CR 2021
Year: 2021

A common privacy issue in traditional machine learning is that data needs to be disclosed for the training procedures. In situations with highly sensitive data such as healthcare records, accessing this information is challenging and often prohibited. Luckily, privacy-preserving technologies have been developed to overcome this hurdle by distributing the computation of the training and ensuring the data privacy to their owners. The distribution of the computation to multiple participating entities introduces new privacy complications and risks. In this paper, we present a privacy-preserving decentralised workflow that facilitates trusted federated learning among participants. Our proof-of-concept defines a trust framework instantiated using decentralised identity technologies being developed under Hyperledger projects Aries/Indy/Ursa. Only entities in possession of Verifiable Credentials issued from the appropriate authorities are able to establish secure, authenticated communication channels authorised to participate in a federated learning workflow related to mental health data.
Towards a Personalisation Framework for Cyber-Physical-Social System (CPSS)
Venue: arXiv cs.HC 2021
Year: 2021

A Cyber-Physical-Social System (CPSS) is an emerging paradigm often understood as a physical and virtual space of interaction which is cohabited by humans and sensor-enabled smart devices. In such settings, human interaction behaviour is often different from person to person and is guided by complex environmental and natural factors that are not yet fully explored. Thus, ensuring a seamless human-machine interaction in CPSS calls for efficient means of handling human dynamics and bringing interaction experience to a personal level. To this end in this paper, we propose a personalisation framework to support the design of CPSS in recognising and addressing human/social aspects.
Industry Scale Semi-Supervised Learning for Natural Language Understanding
Venue: arXiv cs.CL 2021
Year: 2021

This paper presents a production Semi-Supervised Learning (SSL) pipeline based on the student-teacher framework, which leverages millions of unlabeled examples to improve Natural Language Understanding (NLU) tasks. We investigate two questions related to the use of unlabeled data in production SSL context: 1) how to select samples from a huge unlabeled data pool that are beneficial for SSL training, and 2) how do the selected data affect the performance of different state-of-the-art SSL techniques. We compare four widely used SSL techniques, Pseudo-Label (PL), Knowledge Distillation (KD), Virtual Adversarial Training (VAT) and Cross-View Training (CVT) in conjunction with two data selection methods including committee-based selection and submodular optimization based selection. We further examine the benefits and drawbacks of these techniques when applied to intent classification (IC) and named entity recognition (NER) tasks, and provide guidelines specifying when each of these methods might be beneficial to improve large scale NLU systems.
Comparison of different convolutional neural network activation functions and methods for building ensembles
Venue: arXiv cs.CV 2021
Year: 2021

Recently, much attention has been devoted to finding highly efficient and powerful activation functions for CNN layers. Because activation functions inject different nonlinearities between layers that affect performance, varying them is one method for building robust ensembles of CNNs. The objective of this study is to examine the performance of CNN ensembles made with different activation functions, including six new ones presented here: 2D Mexican ReLU, TanELU, MeLU+GaLU, Symmetric MeLU, Symmetric GaLU, and Flexible MeLU. The highest performing ensemble was built with CNNs having different activation layers that randomly replaced the standard ReLU. A comprehensive evaluation of the proposed approach was conducted across fifteen biomedical data sets representing various classification tasks. The proposed method was tested on two basic CNN architectures: Vgg16 and ResNet50. Results demonstrate the superiority in performance of this approach. The MATLAB source code for this study will be available at https://github.com/LorisNanni.
Learning Under Adversarial and Interventional Shifts
Venue: arXiv cs.LG 2021
Year: 2021

Machine learning models are often trained on data from one distribution and deployed on others. So it becomes important to design models that are robust to distribution shifts. Most of the existing work focuses on optimizing for either adversarial shifts or interventional shifts. Adversarial methods lack expressivity in representing plausible shifts as they consider shifts to joint distributions in the data. Interventional methods allow more expressivity but provide robustness to unbounded shifts, resulting in overly conservative models. In this work, we combine the complementary strengths of the two approaches and propose a new formulation, RISe, for designing robust models against a set of distribution shifts that are at the intersection of adversarial and interventional shifts. We employ the distributionally robust optimization framework to optimize the resulting objective in both supervised and reinforcement learning settings. Extensive experimentation with synthetic and real world datasets from healthcare demonstrate the efficacy of the proposed approach.
The open access effect in social media exposure of scholarly articles: A matched-pair analysis
Venue: arXiv cs.DL 2021
Year: 2021

Scholarly journals are increasingly using social media to share their latest research publications and communicate with their readers. Having a presence on social media gives journals a platform to raise their profile and promote their content. This study compares the number of clicks received when journals provide two types of links to subscription articles: open access (OA) and paid content links. We examine the OA effect using unique matched-pair data for the journal Nature Materials. Our study finds that OA links perform better than paid content links. In particular, when the journal does not indicate that a link to an article is an OA link, there is an obvious drop in performance against clicks on links indicating OA status. OA has a positive effect on the number of clicks in all countries, but its positive impact is slightly greater in developed countries. The results suggest that free content is more attractive to users than paid content. Social media exposure of scholarly articles promotes the use of research outputs. Combining social media dissemination with OA appears to enhance the reach of scientific information. However, extensive further efforts are needed to remove barriers to OA.
Theory-Guided Machine Learning for Process Simulation of Advanced Composites
Venue: arXiv cs.LG 2021
Year: 2021

Science-based simulation tools such as Finite Element (FE) models are routinely used in scientific and engineering applications. While their success is strongly dependent on our understanding of underlying governing physical laws, they suffer inherent limitations including trade-off between fidelity/accuracy and speed. The recent rise of Machine Learning (ML) proposes a theory-agnostic paradigm. In complex multi-physics problems, however, creating large enough datasets for successful training of ML models has proven to be challenging. One promising strategy to bridge the divide between these approaches and take advantage of their respective strengths is Theory-Guided Machine Learning (TGML) which aims to integrate physical laws into ML algorithms. In this paper, three case studies on thermal management during processing of advanced composites are presented and studied using FE, ML and TGML. A structured approach to incrementally adding increasingly complex physics to training of TGML model is presented. The benefits of TGML over ML models are seen in more accurate predictions, particularly outside the training region, and ability to train with small datasets. One benefit of TGML over FE is significant speed improvement to potentially develop real-time feedback systems. A recent successful implementation of a TGML model to assess producibility of aerospace composite parts is presented.
Self-supervised Image-text Pre-training With Mixed Data In Chest X-rays
Venue: arXiv cs.CV 2021
Year: 2021

Pre-trained models, e.g., from ImageNet, have proven to be effective in boosting the performance of many downstream applications. It is too demanding to acquire large-scale annotations to build such models for medical imaging. Meanwhile, there are numerous clinical data (in the form of images and text reports) stored in the hospital information systems. The paired image-text data from the same patient study could be utilized for the pre-training task in a weakly supervised manner. However, the integrity, accessibility, and amount of such raw data vary across different institutes, e.g., paired vs. unpaired (image-only or text-only). In this work, we introduce an image-text pre-training framework that can learn from these raw data with mixed data inputs, i.e., paired image-text data, a mixture of paired and unpaired data. The unpaired data can be sourced from one or multiple institutes (e.g., images from one institute coupled with texts from another). Specifically, we propose a transformer-based training framework for jointly learning the representation of both the image and text data. In addition to the existing masked language modeling, multi-scale masked vision modeling is introduced as a self-supervised training task for image patch regeneration. We not only demonstrate the feasibility of pre-training across mixed data inputs but also illustrate the benefits of adopting such pre-trained models in 3 chest X-ray applications, i.e., classification, retrieval, and image regeneration. Superior results are reported in comparison to prior art using MIMIC-CXR, NIH14-CXR, and OpenI-CXR datasets.