OpenRaiser/Intern-Atlas huggingface dataset:
A Brief Overview of the UML Scientific Profile
Venue: arXiv cs.DC 2021
Year: 2021

This article provides a brief overview of the UML SP (UML Scientific Profile). It is an Object-Oriented Simulation Language and may find usage in OOS and ABS. UML SP allows for the application of Unified Process methodology for the development of simulation software. It supports the first three stages of simulation: (1) definition of simulation objectives, definition of requirements for simulation software, (2) conceptual modelling, (3) formal description and partially, (4) programming. UML SPs positioning is that of a language for scientific simulation. It can easily be implemented in UML editors that support UML profiles. The project is available as open source on the GitHub platform.
BCNN: Binary Complex Neural Network
Venue: arXiv cs.NE 2021
Year: 2021

Binarized neural networks, or BNNs, show great promise in edge-side applications with resource limited hardware, but raise the concerns of reduced accuracy. Motivated by the complex neural networks, in this paper we introduce complex representation into the BNNs and propose Binary complex neural network -- a novel network design that processes binary complex inputs and weights through complex convolution, but still can harvest the extraordinary computation efficiency of BNNs. To ensure fast convergence rate, we propose novel BCNN based batch normalization function and weight initialization function. Experimental results on Cifar10 and ImageNet using state-of-the-art network models (e.g., ResNet, ResNetE and NIN) show that BCNN can achieve better accuracy compared to the original BNN models. BCNN improves BNN by strengthening its learning capability through complex representation and extending its applicability to complex-valued input data. The source code of BCNN will be released on GitHub.
Class-Incremental Learning with Generative Classifiers
Venue: arXiv cs.LG 2021
Year: 2021

Incrementally training deep neural networks to recognize new classes is a challenging problem. Most existing class-incremental learning methods store data or use generative replay, both of which have drawbacks, while 'rehearsal-free' alternatives such as parameter regularization or bias-correction methods do not consistently achieve high performance. Here, we put forward a new strategy for class-incremental learning: generative classification. Rather than directly learning the conditional distribution p(y|x), our proposal is to learn the joint distribution p(x,y), factorized as p(x|y)p(y), and to perform classification using Bayes' rule. As a proof-of-principle, here we implement this strategy by training a variational autoencoder for each class to be learned and by using importance sampling to estimate the likelihoods p(x|y). This simple approach performs very well on a diverse set of continual learning benchmarks, outperforming generative replay and other existing baselines that do not store data.
UIT-ISE-NLP at SemEval-2021 Task 5: Toxic Spans Detection with BiLSTM-CRF and ToxicBERT Comment Classification
Venue: arXiv cs.CL 2021
Year: 2021

We present our works on SemEval-2021 Task 5 about Toxic Spans Detection. This task aims to build a model for identifying toxic words in whole posts. We use the BiLSTM-CRF model combining with ToxicBERT Classification to train the detection model for identifying toxic words in posts. Our model achieves 62.23% by F1-score on the Toxic Spans Detection task.
ds-array: A Distributed Data Structure for Large Scale Machine Learning
Venue: arXiv cs.DC 2021
Year: 2021

Machine learning has proved to be a useful tool for extracting knowledge from scientific data in numerous research fields, including astrophysics, genomics, and molecular dynamics. Often, data sets from these research areas need to be processed in distributed platforms due to their magnitude. This can be done using one of the various distributed machine learning libraries available. One of these libraries is dislib, a distributed machine learning library for Python especially designed to process large scale data sets on HPC clusters, which makes dislib an ideal candidate for analyzing scientific data. However, dislib's main distributed data structure, called Dataset, has some limitations, including poor performance in certain operations and low flexibility and usability. In this paper, we propose a novel distributed data structure for dislib, called ds-array, that addresses dislib's main limitations in data management. Ds-arrays simplify distributed data management in dislib by exposing a NumPy-like API, provide more flexibility, and reduce the computational complexity of some operations. This results in performance improvements of up to two orders of magnitude over Datasets, while also greatly improving scalability and usability.
Development of a dynamic type 2 diabetes risk prediction tool: a UK Biobank study
Venue: arXiv cs.LG 2021
Year: 2021

Diabetes affects over 400 million people and is among the leading causes of morbidity worldwide. Identification of high-risk individuals can support early diagnosis and prevention of disease development through lifestyle changes. However, the majority of existing risk scores require information about blood-based factors which are not obtainable outside of the clinic. Here, we aimed to develop an accessible solution that could be deployed digitally and at scale. We developed a predictive 10-year type 2 diabetes risk score using 301 features derived from 472,830 participants in the UK Biobank dataset while excluding any features which are not easily obtainable by a smartphone. Using a data-driven feature selection process, 19 features were included in the final reduced model. A Cox proportional hazards model slightly overperformed a DeepSurv model trained using the same features, achieving a concordance index of 0.818 (95% CI: 0.812-0.823), compared to 0.811 (95% CI: 0.806-0.815). The final model showed good calibration. This tool can be used for clinical screening of individuals at risk of developing type 2 diabetes and to foster patient empowerment by broadening their knowledge of the factors affecting their personal risk.
Phase Transition Adaptation
Venue: arXiv cs.LG 2021
Year: 2021

Artificial Recurrent Neural Networks are a powerful information processing abstraction, and Reservoir Computing provides an efficient strategy to build robust implementations by projecting external inputs into high dimensional dynamical system trajectories. In this paper, we propose an extension of the original approach, a local unsupervised learning mechanism we call Phase Transition Adaptation, designed to drive the system dynamics towards the `edge of stability'. Here, the complex behavior exhibited by the system elicits an enhancement in its overall computational capacity. We show experimentally that our approach consistently achieves its purpose over several datasets.
Understanding the Context of IoT Software Systems in DevOps
Venue: arXiv cs.SE 2021
Year: 2021

The growing demand for connected devices and the increase in investments in the Internet of Things (IoT) sector induce the growth of the market for this technology. IoT permeates all areas of life of an individual, from smartwatches to entire home assistants and solutions in different areas. The IoT concept is gradually increasing all over the globe. IoT projects induce an articulation of studies in software engineering to prepare the development and operation of software systems materialized in physical objects and structures interconnected with embedded software and hosted in clouds. IoT projects have boundaries between development and operation stages. This study search for evidence in scientific literature to support these boundaries through Development and Operations (DevOps) principles. We rely on a Systematic Literature Review to investigate the relations of DevOps in IoT software systems. As a result, we identify concepts, characterize the benefits and challenges in the context of knowledge previously reported in primary studies in the literature. The main contributions of this paper are: (i) discussion of benefits and challenges for DevOps in IoT software systems, (ii) identification of tools, concepts, and programming languages used, and, (iii) perceived pipeline for this kind of software development.
Episodic Memory Model for Learning Robotic Manipulation Tasks
Venue: arXiv cs.RO 2021
Year: 2021

Machine learning, artificial intelligence and especially deep learning based approaches are often used to simplify or eliminate the burden of programming industrial robots. Using these approaches robots inherently learn a skill instead of being programmed using strict and tedious programming instructions. While deep learning is effective in making robots learn skills, it does not offer a practical route for teaching a complete task, such as assembly or machine tending, where a complex logic must be understood and related sub-tasks need to be performed. We present a model similar to an episodic memory that allows robots to comprehend sequences of actions using single demonstration and perform them properly and accurately. The algorithm identifies and recognizes the changes in the states of the system and memorizes how to execute the necessary tasks in order to make those changes. This allows the robot to decompose the tasks into smaller sub-tasks, retain the essential steps, and remember how they have been performed.
Concept Drift Detection from Multi-Class Imbalanced Data Streams
Venue: arXiv cs.LG 2021
Year: 2021

Continual learning from data streams is among the most important topics in contemporary machine learning. One of the biggest challenges in this domain lies in creating algorithms that can continuously adapt to arriving data. However, previously learned knowledge may become outdated, as streams evolve over time. This phenomenon is known as concept drift and must be detected to facilitate efficient adaptation of the learning model. While there exists a plethora of drift detectors, all of them assume that we are dealing with roughly balanced classes. In the case of imbalanced data streams, those detectors will be biased towards the majority classes, ignoring changes happening in the minority ones. Furthermore, class imbalance may evolve over time and classes may change their roles (majority becoming minority and vice versa). This is especially challenging in the multi-class setting, where relationships among classes become complex. In this paper, we propose a detailed taxonomy of challenges posed by concept drift in multi-class imbalanced data streams, as well as a novel trainable concept drift detector based on Restricted Boltzmann Machine. It is capable of monitoring multiple classes at once and using reconstruction error to detect changes in each of them independently. Our detector utilizes a skew-insensitive loss function that allows it to handle multiple imbalanced distributions. Due to its trainable nature, it is capable of following changes in a stream and evolving class roles, as well as it can deal with local concept drift occurring in minority classes. Extensive experimental study on multi-class drifting data streams, enriched with a detailed analysis of the impact of local drifts and changing imbalance ratios, confirms the high efficacy of our approach.
Revisiting The Evaluation of Class Activation Mapping for Explainability: A Novel Metric and Experimental Analysis
Venue: arXiv cs.CV 2021
Year: 2021

As the request for deep learning solutions increases, the need for explainability is even more fundamental. In this setting, particular attention has been given to visualization techniques, that try to attribute the right relevance to each input pixel with respect to the output of the network. In this paper, we focus on Class Activation Mapping (CAM) approaches, which provide an effective visualization by taking weighted averages of the activation maps. To enhance the evaluation and the reproducibility of such approaches, we propose a novel set of metrics to quantify explanation maps, which show better effectiveness and simplify comparisons between approaches. To evaluate the appropriateness of the proposal, we compare different CAM-based visualization methods on the entire ImageNet validation set, fostering proper comparisons and reproducibility.
Network Defense is Not a Game
Venue: arXiv cs.CR 2021
Year: 2021

Research seeks to apply Artificial Intelligence (AI) to scale and extend the capabilities of human operators to defend networks. A fundamental problem that hinders the generalization of successful AI approaches -- i.e., beating humans at playing games -- is that network defense cannot be defined as a single game with a fixed set of rules. Our position is that network defense is better characterized as a collection of games with uncertain and possibly drifting rules. Hence, we propose to define network defense tasks as distributions of network environments, to: (i) enable research to apply modern AI techniques, such as unsupervised curriculum learning and reinforcement learning for network defense; and, (ii) facilitate the design of well-defined challenges that can be used to compare approaches for autonomous cyberdefense. To demonstrate that an approach for autonomous network defense is practical it is important to be able to reason about the boundaries of its applicability. Hence, we need to be able to define network defense tasks that capture sets of adversarial tactics, techniques, and procedures (TTPs); quality of service (QoS) requirements; and TTPs available to defenders. Furthermore, the abstractions to define these tasks must be extensible; must be backed by well-defined semantics that allow us to reason about distributions of environments; and should enable the generation of data and experiences from which an agent can learn. Our approach named Network Environment Design for Autonomous Cyberdefense inspired the architecture of FARLAND, a Framework for Advanced Reinforcement Learning for Autonomous Network Defense, which we use at MITRE to develop RL network defenders that perform blue actions from the MITRE Shield matrix against attackers with TTPs that drift from MITRE ATT&CK TTPs.
A windowed correlation based feature selection method to improve time series prediction of dengue fever cases
Venue: arXiv cs.LG 2021
Year: 2021

The performance of data-driven prediction models depends on the availability of data samples for model training. A model that learns about dengue fever incidence in a population uses historical data from that corresponding location. Poor performance in prediction can result in places with inadequate data. This work aims to enhance temporally limited dengue case data by methodological addition of epidemically relevant data from nearby locations as predictors (features). A novel framework is presented for windowing incidence data and computing time-shifted correlation-based metrics to quantify feature relevance. The framework ranks incidence data of adjacent locations around a target location by combining the correlation metric with two other metrics: spatial distance and local prevalence. Recurrent neural network-based prediction models achieve up to 33.6% accuracy improvement on average using the proposed method compared to using training data from the target location only. These models achieved mean absolute error (MAE) values as low as 0.128 on [0,1] normalized incidence data for a municipality with the highest dengue prevalence in Brazil's Espirito Santo. When predicting cases aggregated over geographical ecoregions, the models achieved accuracy improvements up to 16.5%, using only 6.5% of incidence data from ranked feature sets. The paper also includes two techniques for windowing time series data: fixed-sized windows and outbreak detection windows. Both of these techniques perform comparably, while the window detection method uses less data for computations. The framework presented in this paper is application-independent, and it could improve the performances of prediction models where data from spatially adjacent locations are available.
Robust shape estimation with false-positive contact detection
Venue: arXiv cs.RO 2021
Year: 2021

We propose a means of omni-directional contact detection using accelerometers instead of tactile sensors for object shape estimation using touch. Unlike tactile sensors, our contact-based detection method tends to induce a degree of uncertainty with false-positive contact data because the sensors may react not only to actual contact but also to the unstable behavior of the robot. Therefore, it is crucial to consider a robust shape estimation method capable of handling such false-positive contact data. To realize this, we introduce the concept of heteroscedasticity into the contact data and propose a robust shape estimation algorithm based on Gaussian process implicit surfaces (GPIS). We confirmed that our algorithm not only reduces shape estimation errors caused by false-positive contact data but also distinguishes false-positive contact data more clearly than the GPIS through simulations and actual experiments using a quadcopter.
Evidential Cyber Threat Hunting
Venue: arXiv cs.CR 2021
Year: 2021

A formal cyber reasoning framework for automating the threat hunting process is described. The new cyber reasoning methodology introduces an operational semantics that operates over three subspaces -- knowledge, hypothesis, and action -- to enable human-machine co-creation of threat hypotheses and protective recommendations. An implementation of this framework shows that the approach is practical and can be used to generalize evidence-based multi-criteria threat investigations.
Measuring economic activity from space: a case study using flying airplanes and COVID-19
Venue: arXiv cs.CV 2021
Year: 2021

This work introduces a novel solution to measure economic activity through remote sensing for a wide range of spatial areas. We hypothesized that disturbances in human behavior caused by major life-changing events leave signatures in satellite imagery that allows devising relevant image-based indicators to estimate their impacts and support decision-makers. We present a case study for the COVID-19 coronavirus outbreak, which imposed severe mobility restrictions and caused worldwide disruptions, using flying airplane detection around the 30 busiest airports in Europe to quantify and analyze the lockdown's effects and post-lockdown recovery. Our solution won the Rapid Action Coronavirus Earth observation (RACE) upscaling challenge, sponsored by the European Space Agency and the European Commission, and now integrates the RACE dashboard. This platform combines satellite data and artificial intelligence to promote a progressive and safe reopening of essential activities. Code and CNN models are available at https://github.com/maups/covid19-custom-script-contest
Improving Weakly-supervised Object Localization via Causal Intervention
Venue: arXiv cs.CV 2021
Year: 2021

The recent emerged weakly supervised object localization (WSOL) methods can learn to localize an object in the image only using image-level labels. Previous works endeavor to perceive the interval objects from the small and sparse discriminative attention map, yet ignoring the co-occurrence confounder (e.g., bird and sky), which makes the model inspection (e.g., CAM) hard to distinguish between the object and context. In this paper, we make an early attempt to tackle this challenge via causal intervention (CI). Our proposed method, dubbed CI-CAM, explores the causalities among images, contexts, and categories to eliminate the biased co-occurrence in the class activation maps thus improving the accuracy of object localization. Extensive experiments on several benchmarks demonstrate the effectiveness of CI-CAM in learning the clear object boundaries from confounding contexts. Particularly, in CUB-200-2011 which severely suffers from the co-occurrence confounder, CI-CAM significantly outperforms the traditional CAM-based baseline (58.39% vs 52.4% in top-1 localization accuracy). While in more general scenarios such as ImageNet, CI-CAM can also perform on par with the state of the arts.
Improving Test Distance for Failure Clustering with Hypergraph Modelling
Venue: arXiv cs.SE 2021
Year: 2021

Automated debugging techniques, such as Fault Localisation (FL) or Automated Program Repair (APR), are typically designed under the Single Fault Assumption (SFA). However, in practice, an unknown number of faults can independently cause multiple test case failures, making it difficult to allocate resources for debugging and to use automated debugging techniques. Clustering algorithms have been applied to group the test failures according to their root causes, but their accuracy can often be lacking due to the inherent limits in the distance metrics for test cases. We introduce a new test distance metric based on hypergraphs and evaluate their accuracy using multi-fault benchmarks that we have built on top of Defects4J and SIR. Results show that our technique, Hybiscus, can automatically achieve perfect clustering (i.e., the same number of clusters as the ground truth number of root causes, with all failing tests with the same root cause grouped together) for 418 out of 605 test runs with multiple test failures. Better failure clustering also allows us to separate different root causes and apply FL techniques under SFA, resulting in saving up to 82% of the total wasted effort when compared to the state-of-the-art technique for multiple fault localisation.
PALI at SemEval-2021 Task 2: Fine-Tune XLM-RoBERTa for Word in Context Disambiguation
Venue: arXiv cs.AI 2021
Year: 2021

This paper presents the PALI team's winning system for SemEval-2021 Task 2: Multilingual and Cross-lingual Word-in-Context Disambiguation. We fine-tune XLM-RoBERTa model to solve the task of word in context disambiguation, i.e., to determine whether the target word in the two contexts contains the same meaning or not. In the implementation, we first specifically design an input tag to emphasize the target word in the contexts. Second, we construct a new vector on the fine-tuned embeddings from XLM-RoBERTa and feed it to a fully-connected network to output the probability of whether the target word in the context has the same meaning or not. The new vector is attained by concatenating the embedding of the [CLS] token and the embeddings of the target word in the contexts. In training, we explore several tricks, such as the Ranger optimizer, data augmentation, and adversarial training, to improve the model prediction. Consequently, we attain first place in all four cross-lingual tasks.
Towards Corruption-Agnostic Robust Domain Adaptation
Venue: arXiv cs.CV 2021
Year: 2021

Big progress has been achieved in domain adaptation in decades. Existing works are always based on an ideal assumption that testing target domain are i.i.d. with training target domains. However, due to unpredictable corruptions (e.g., noise and blur) in real data like web images, domain adaptation methods are increasingly required to be corruption robust on target domains. In this paper, we investigate a new task, Corruption-agnostic Robust Domain Adaptation (CRDA): to be accurate on original data and robust against unavailable-for-training corruptions on target domains. This task is non-trivial due to large domain discrepancy and unsupervised target domains. We observe that simple combinations of popular methods of domain adaptation and corruption robustness have sub-optimal CRDA results. We propose a new approach based on two technical insights into CRDA: 1) an easy-to-plug module called Domain Discrepancy Generator (DDG) that generates samples that enlarge domain discrepancy to mimic unpredictable corruptions; 2) a simple but effective teacher-student scheme with contrastive loss to enhance the constraints on target domains. Experiments verify that DDG keeps or even improves performance on original data and achieves better corruption robustness that baselines.