OpenRaiser/Intern-Atlas huggingface dataset:
Meta-Solver for Neural Ordinary Differential Equations
Venue: arXiv cs.LG 2021
Year: 2021

A conventional approach to train neural ordinary differential equations (ODEs) is to fix an ODE solver and then learn the neural network's weights to optimize a target loss function. However, such an approach is tailored for a specific discretization method and its properties, which may not be optimal for the selected application and yield the overfitting to the given solver. In our paper, we investigate how the variability in solvers' space can improve neural ODEs performance. We consider a family of Runge-Kutta methods that are parameterized by no more than two scalar variables. Based on the solvers' properties, we propose an approach to decrease neural ODEs overfitting to the pre-defined solver, along with a criterion to evaluate such behaviour. Moreover, we show that the right choice of solver parameterization can significantly affect neural ODEs models in terms of robustness to adversarial attacks. Recently it was shown that neural ODEs demonstrate superiority over conventional CNNs in terms of robustness. Our work demonstrates that the model robustness can be further improved by optimizing solver choice for a given task. The source code to reproduce our experiments is available at https://github.com/juliagusak/neural-ode-metasolver.
The Effect of Domain and Diacritics in Yorùbá-English Neural Machine Translation
Venue: arXiv cs.CL 2021
Year: 2021

Massively multilingual machine translation (MT) has shown impressive capabilities, including zero and few-shot translation between low-resource language pairs. However, these models are often evaluated on high-resource languages with the assumption that they generalize to low-resource ones. The difficulty of evaluating MT models on low-resource pairs is often due to lack of standardized evaluation datasets. In this paper, we present MENYO-20k, the first multi-domain parallel corpus with a special focus on clean orthography for Yorùbá--English with standardized train-test splits for benchmarking. We provide several neural MT benchmarks and compare them to the performance of popular pre-trained (massively multilingual) MT models both for the heterogeneous test set and its subdomains. Since these pre-trained models use huge amounts of data with uncertain quality, we also analyze the effect of diacritics, a major characteristic of Yorùbá, in the training data. We investigate how and when this training condition affects the final quality and intelligibility of a translation. Our models outperform massively multilingual models such as Google ($+8.7$ BLEU) and Facebook M2M ($+9.1$ BLEU) when translating to Yorùbá, setting a high quality benchmark for future research.
Counting and Sampling Perfect Matchings in Regular Expanding Non-Bipartite Graphs
Venue: arXiv cs.DS 2021
Year: 2021

We show that the ratio of the number of near perfect matchings to the number of perfect matchings in $d$-regular strong expander (non-bipartite) graphs, with $2n$ vertices, is a polynomial in $n$, thus the Jerrum and Sinclair Markov chain [JS89] mixes in polynomial time and generates an (almost) uniformly random perfect matching. Furthermore, we prove that such graphs have at least $Ω(d)^n$ any perfect matchings, thus proving the Lovasz-Plummer conjecture [LP86] for this family of graphs.
Axis-Aligned Square Contact Representations
Venue: arXiv cs.CG 2021
Year: 2021

We introduce a new class $\mathcal{G}$ of bipartite plane graphs and prove that each graph in $\mathcal{G}$ admits a proper square contact representation. A contact between two squares is \emph{proper} if they intersect in a line segment of positive length. The class $\mathcal{G}$ is the family of quadrangulations obtained from the 4-cycle $C_4$ by successively inserting a single vertex or a 4-cycle of vertices into a face. For every graph $G\in \mathcal{G}$, we construct a proper square contact representation. The key parameter of the recursive construction is the aspect ratio of the rectangle bounded by the four outer squares. We show that this aspect ratio may continuously vary in an interval $I_G$. The interval $I_G$ cannot be replaced by a fixed aspect ratio, however, as we show, the feasible interval $I_G$ may be an arbitrarily small neighborhood of any positive real.
One-Time Pads from the Digits of Pi
Venue: arXiv cs.OH 2021
Year: 2021

I present a method for generating one-time pads from the digits of pi. Computer code is given to generate such pads from passphrases in a method having an extremely low probability (<10^-53) of a successful discovery of the one-time pads by a brute-force attack. The advantages and disadvantages of this method are discussed.
Predicting Opioid Use Disorder from Longitudinal Healthcare Data using Multi-stream Transformer
Venue: arXiv cs.LG 2021
Year: 2021

Opioid Use Disorder (OUD) is a public health crisis costing the US billions of dollars annually in healthcare, lost workplace productivity, and crime. Analyzing longitudinal healthcare data is critical in addressing many real-world problems in healthcare. Leveraging the real-world longitudinal healthcare data, we propose a novel multi-stream transformer model called MUPOD for OUD identification. MUPOD is designed to simultaneously analyze multiple types of healthcare data streams, such as medications and diagnoses, by attending to segments within and across these data streams. Our model tested on the data from 392,492 patients with long-term back pain problems showed significantly better performance than the traditional models and recently developed deep learning models.
EX-RAY: Distinguishing Injected Backdoor from Natural Features in Neural Networks by Examining Differential Feature Symmetry
Venue: arXiv cs.LG 2021
Year: 2021

Backdoor attack injects malicious behavior to models such that inputs embedded with triggers are misclassified to a target label desired by the attacker. However, natural features may behave like triggers, causing misclassification once embedded. While they are inevitable, mis-recognizing them as injected triggers causes false warnings in backdoor scanning. A prominent challenge is hence to distinguish natural features and injected backdoors. We develop a novel symmetric feature differencing method that identifies a smallest set of features separating two classes. A backdoor is considered injected if the corresponding trigger consists of features different from the set of features distinguishing the victim and target classes. We evaluate the technique on thousands of models, including both clean and trojaned models, from the TrojAI rounds 2-4 competitions and a number of models on ImageNet. Existing backdoor scanning techniques may produce hundreds of false positives (i.e., clean models recognized as trojaned). Our technique removes 78-100% of the false positives (by a state-of-the-art scanner ABS) with a small increase of false negatives by 0-30%, achieving 17-41% overall accuracy improvement, and facilitates achieving top performance on the leaderboard. It also boosts performance of other scanners. It outperforms false positive removal methods using L2 distance and attribution techniques. We also demonstrate its potential in detecting a number of semantic backdoor attacks.
GraphSMOTE: Imbalanced Node Classification on Graphs with Graph Neural Networks
Venue: arXiv cs.LG 2021
Year: 2021

Node classification is an important research topic in graph learning. Graph neural networks (GNNs) have achieved state-of-the-art performance of node classification. However, existing GNNs address the problem where node samples for different classes are balanced; while for many real-world scenarios, some classes may have much fewer instances than others. Directly training a GNN classifier in this case would under-represent samples from those minority classes and result in sub-optimal performance. Therefore, it is very important to develop GNNs for imbalanced node classification. However, the work on this is rather limited. Hence, we seek to extend previous imbalanced learning techniques for i.i.d data to the imbalanced node classification task to facilitate GNN classifiers. In particular, we choose to adopt synthetic minority over-sampling algorithms, as they are found to be the most effective and stable. This task is non-trivial, as previous synthetic minority over-sampling algorithms fail to provide relation information for newly synthesized samples, which is vital for learning on graphs. Moreover, node attributes are high-dimensional. Directly over-sampling in the original input domain could generates out-of-domain samples, which may impair the accuracy of the classifier. We propose a novel framework, GraphSMOTE, in which an embedding space is constructed to encode the similarity among the nodes. New samples are synthesize in this space to assure genuineness. In addition, an edge generator is trained simultaneously to model the relation information, and provide it for those new samples. This framework is general and can be easily extended into different variations. The proposed framework is evaluated using three different datasets, and it outperforms all baselines with a large margin.
Semi-Supervised Graph-to-Graph Translation
Venue: arXiv cs.LG 2021
Year: 2021

Graph translation is very promising research direction and has a wide range of potential real-world applications. Graph is a natural structure for representing relationship and interactions, and its translation can encode the intrinsic semantic changes of relationships in different scenarios. However, despite its seemingly wide possibilities, usage of graph translation so far is still quite limited. One important reason is the lack of high-quality paired dataset. For example, we can easily build graphs representing peoples' shared music tastes and those representing co-purchase behavior, but a well paired dataset is much more expensive to obtain. Therefore, in this work, we seek to provide a graph translation model in the semi-supervised scenario. This task is non-trivial, because graph translation involves changing the semantics in the form of link topology and node attributes, which is difficult to capture due to the combinatory nature and inter-dependencies. Furthermore, due to the high order of freedom in graph's composition, it is difficult to assure the generalization ability of trained models. These difficulties impose a tighter requirement for the exploitation of unpaired samples. Addressing them, we propose to construct a dual representation space, where transformation is performed explicitly to model the semantic transitions. Special encoder/decoder structures are designed, and auxiliary mutual information loss is also adopted to enforce the alignment of unpaired/paired examples. We evaluate the proposed method in three different datasets.
Closed-Loop Error Learning Control for Uncertain Nonlinear Systems With Experimental Validation on a Mobile Robot
Venue: arXiv cs.RO 2021
Year: 2021

This paper develops a Closed-Loop Error Learning Control (CLELC) algorithm for feedback linearizable systems with experimental validation on a mobile robot. Traditional feedback and feedforward controllers are designed based on the nominal model by using Feedback Linearization Control (FLC) method. Then, an intelligent controller is designed based on sliding mode learning algorithm that utilizes closed-loop error dynamics to learn the system behavior. The controllers are working in parallel, and the intelligent controller can gradually replace the feedback controller from the control of the system. In addition to the stability of the sliding mode learning algorithm, the closed-loop stability of an $n$th order feedback linearizable system is proven. The simulation results demonstrate that CLELC algorithm can improve control performance (e.g., smaller rise time, settling time and overshoot) in the absence of uncertainties, and also provides robust control performance in the presence of uncertainties as compared to traditional FLC method. To test the efficiency and efficacy of CLELC algorithm, the trajectory tracking problem of a tracked mobile robot is studied in real-time. The experimental results demonstrate that CLELC algorithm ensures high-accurate trajectory tracking performance than traditional FLC method.
Few Shot System Identification for Reinforcement Learning
Venue: arXiv cs.RO 2021
Year: 2021

Learning by interaction is the key to skill acquisition for most living organisms, which is formally called Reinforcement Learning (RL). RL is efficient in finding optimal policies for endowing complex systems with sophisticated behavior. All paradigms of RL utilize a system model for finding the optimal policy. Modeling dynamics can be done by formulating a mathematical model or system identification. Dynamic models are usually exposed to aleatoric and epistemic uncertainties that can divert the model from the one acquired and cause the RL algorithm to exhibit erroneous behavior. Accordingly, the RL process sensitive to operating conditions and changes in model parameters and lose its generality. To address these problems, Intensive system identification for modeling purposes is needed for each system even if the model dynamics structure is the same, as the slight deviation in the model parameters can render the model useless in RL. The existence of an oracle that can adaptively predict the rest of the trajectory regardless of the uncertainties can help resolve the issue. The target of this work is to present a framework for facilitating the system identification of different instances of the same dynamics class by learning a probability distribution of the dynamics conditioned on observed data with variational inference and show its reliability in robustly solving different instances of control problems with the same model in model-based RL with maximum sample efficiency.
Learned Gradient Compression for Distributed Deep Learning
Venue: arXiv cs.LG 2021
Year: 2021

Training deep neural networks on large datasets containing high-dimensional data requires a large amount of computation. A solution to this problem is data-parallel distributed training, where a model is replicated into several computational nodes that have access to different chunks of the data. This approach, however, entails high communication rates and latency because of the computed gradients that need to be shared among nodes at every iteration. The problem becomes more pronounced in the case that there is wireless communication between the nodes (i.e. due to the limited network bandwidth). To address this problem, various compression methods have been proposed including sparsification, quantization, and entropy encoding of the gradients. Existing methods leverage the intra-node information redundancy, that is, they compress gradients at each node independently. In contrast, we advocate that the gradients across the nodes are correlated and propose methods to leverage this inter-node redundancy to improve compression efficiency. Depending on the node communication protocol (parameter server or ring-allreduce), we propose two instances of the LGC approach that we coin Learned Gradient Compression (LGC). Our methods exploit an autoencoder (i.e. trained during the first stages of the distributed training) to capture the common information that exists in the gradients of the distributed nodes. We have tested our LGC methods on the image classification and semantic segmentation tasks using different convolutional neural networks (ResNet50, ResNet101, PSPNet) and multiple datasets (ImageNet, Cifar10, CamVid). The ResNet101 model trained for image classification on Cifar10 achieved an accuracy of 93.57%, which is lower than the baseline distributed training with uncompressed gradients only by 0.18%.
An ASIC Implementation and Evaluation of a Profiled Low-Energy Instruction Set Architecture Extension
Venue: arXiv cs.AR 2021
Year: 2021

This paper presents an extension to an existing instruction set architecture, which gains considerable reduction in power consumption. The reduction in power consumption is achieved through coding of the most commonly executed instructions in a short format done by the compiler based on a profile of previous executions. This leads to fewer accesses to the instruction cache and that more instructions can fit in the cache. As a secondary effect, this turned out to be very beneficial in terms of power. Another major advantage, which is the main concern of this paper is the reduction in the number of instruction fetch cycles which will also contribute significantly towards reduction in power consumption. The work involves implementing the new processor architecture in ASIC and estimation of power-consumption compared to the normal architecture.
Combining Morphological and Histogram based Text Line Segmentation in the OCR Context
Venue: arXiv cs.CV 2021
Year: 2021

Text line segmentation is one of the pre-stages of modern optical character recognition systems. The algorithmic approach proposed by this paper has been designed for this exact purpose. Its main characteristic is the combination of two different techniques, morphological image operations and horizontal histogram projections. The method was developed to be applied on a historic data collection that commonly features quality issues, such as degraded paper, blurred text, or presence of noise. For that reason, the segmenter in question could be of particular interest for cultural institutions, that want access to robust line bounding boxes for a given historic document. Because of the promising segmentation results that are joined by low computational cost, the algorithm was incorporated into the OCR pipeline of the National Library of Luxembourg, in the context of the initiative of reprocessing their historic newspaper collection. The general contribution of this paper is to outline the approach and to evaluate the gains in terms of accuracy and speed, comparing it to the segmentation algorithm bundled with the used open source OCR software.
MLOps Challenges in Multi-Organization Setup: Experiences from Two Real-World Cases
Venue: arXiv cs.SE 2021
Year: 2021

The emerging age of connected, digital world means that there are tons of data, distributed to various organizations and their databases. Since this data can be confidential in nature, it cannot always be openly shared in seek of artificial intelligence (AI) and machine learning (ML) solutions. Instead, we need integration mechanisms, analogous to integration patterns in information systems, to create multi-organization AI/ML systems. In this paper, we present two real-world cases. First, we study integration between two organizations in detail. Second, we address scaling of AI/ML to multi-organization context. The setup we assume is that of continuous deployment, often referred to DevOps in software development. When also ML components are deployed in a similar fashion, term MLOps is used. Towards the end of the paper, we list the main observations and draw some final conclusions. Finally, we propose some directions for future work.
Managing layers of risk: Uncertainty in large development programs combining agile software development and traditional project management
Venue: arXiv cs.SE 2021
Year: 2021

How risks are managed implicitly and explicitly at multiple levels of agile projects has not been extensively studied and there is a need to investigate how risk management can be used in large agile projects. This is the objective of this exploratory study which investigates the following research question: How does a large software/hardware development project using agile practices manage uncertainty at project/subproject and work package levels?
Implementation of Technology Acceptance Model (TAM) and Importance Performance Analysis (IPA) in Testing the Ease and Usability of E-wallet Applications
Venue: arXiv cs.CY 2021
Year: 2021

Digital payment innovation is currently increasingly needed by the community, especially in making non-cash payment transactions. The purpose of this research is to know and measure the ease and usefulness of e-wallet digital wallet services, especially in the GoPay application. The population in this study are users of the Go-Pay service on the GO-JEK platform. The sample of this study consisted of 124 respondents from distributing questionnaires in Depok, West Java using a modified Technology Acceptance Model (TAM) based on existing references. The data processing in this research uses Importance Performance Analysis (IPA) analysis. The results show that based on the gap analysis, it is found that in general Go-Pay users are not satisfied with the current service quality. Based on the IPA analysis, the priority scale of E-Wallet Go-Pay quality improvement can be mapped, where quadrant I is the highest priority scale according to the user's perspective: [1], [4], [5], and [6]. These three items must be upgraded immediately by the manager to meet user expectations. Areas that become the achievements or advantages of the GoPay E-Wallet that must be maintained are in quadrant II, namely: [2] and [3]. From this explanation, it can be concluded that in general the E-Wallet GoPay Service must be improved to improve its service performance.
Exploring the Assessment List for Trustworthy AI in the Context of Advanced Driver-Assistance Systems
Venue: arXiv cs.CY 2021
Year: 2021

Artificial Intelligence (AI) is increasingly used in critical applications. Thus, the need for dependable AI systems is rapidly growing. In 2018, the European Commission appointed experts to a High-Level Expert Group on AI (AI-HLEG). AI-HLEG defined Trustworthy AI as 1) lawful, 2) ethical, and 3) robust and specified seven corresponding key requirements. To help development organizations, AI-HLEG recently published the Assessment List for Trustworthy AI (ALTAI). We present an illustrative case study from applying ALTAI to an ongoing development project of an Advanced Driver-Assistance System (ADAS) that relies on Machine Learning (ML). Our experience shows that ALTAI is largely applicable to ADAS development, but specific parts related to human agency and transparency can be disregarded. Moreover, bigger questions related to societal and environmental impact cannot be tackled by an ADAS supplier in isolation. We present how we plan to develop the ADAS to ensure ALTAI-compliance. Finally, we provide three recommendations for the next revision of ALTAI, i.e., life-cycle variants, domain-specific adaptations, and removed redundancy.
Selective Intervention Planning using Restless Multi-Armed Bandits to Improve Maternal and Child Health Outcomes
Venue: arXiv cs.LG 2021
Year: 2021

India has a maternal mortality ratio of 113 and child mortality ratio of 2830 per 100,000 live births. Lack of access to preventive care information is a major contributing factor for these deaths, especially in low resource households. We partner with ARMMAN, a non-profit based in India employing a call-based information program to disseminate health-related information to pregnant women and women with recent child deliveries. We analyze call records of over 300,000 women registered in the program created by ARMMAN and try to identify women who might not engage with these call programs that are proven to result in positive health outcomes. We built machine learning based models to predict the long term engagement pattern from call logs and beneficiaries' demographic information, and discuss the applicability of this method in the real world through a pilot validation. Through a pilot service quality improvement study, we show that using our model's predictions to make interventions boosts engagement metrics by 61.37%. We then formulate the intervention planning problem as restless multi-armed bandits (RMABs), and present preliminary results using this approach.