OpenRaiser/Intern-Atlas huggingface dataset:
All-You-Can-Fit 8-Bit Flexible Floating-Point Format for Accurate and Memory-Efficient Inference of Deep Neural Networks
Venue: arXiv cs.LG 2021
Year: 2021

Modern deep neural network (DNN) models generally require a huge amount of weight and activation values to achieve good inference outcomes. Those data inevitably demand a massive off-chip memory capacity/bandwidth, and the situation gets even worse if they are represented in high-precision floating-point formats. Effort has been made for representing those data in different 8-bit floating-point formats, nevertheless, a notable accuracy loss is still unavoidable. In this paper we introduce an extremely flexible 8-bit floating-point (FFP8) format whose defining factors - the bit width of exponent/fraction field, the exponent bias, and even the presence of the sign bit - are all configurable. We also present a methodology to properly determine those factors so that the accuracy of model inference can be maximized. The foundation of this methodology is based on a key observation - both the maximum magnitude and the value distribution are quite dissimilar between weights and activations in most DNN models. Experimental results demonstrate that the proposed FFP8 format achieves an extremely low accuracy loss of $0.1\%\sim 0.3\%$ for several representative image classification models even without the need of model retraining. Besides, it is easy to turn a classical floating-point processing unit into an FFP8-compliant one, and the extra hardware cost is minor.
Microfluidic-based Bacterial Molecular Computing on a Chip
Venue: arXiv cs.ET 2021
Year: 2021

Biocomputing systems based on engineered bacteria can lead to novel tools for environmental monitoring and detection of metabolic diseases. In this paper, we propose a Bacterial Molecular Computing on a Chip (BMCoC) using microfluidic and electrochemical sensing technologies. The computing can be flexibly integrated into the chip, but we focus on engineered bacterial AND Boolean logic gate and ON-OFF switch sensors that produces secondary signals to change the pH and dissolved oxygen concentrations. We present a prototype with experimental results that shows the electrochemical sensors can detect small pH and dissolved oxygen concentration changes created by the engineered bacterial populations' molecular signals. Additionally, we present a theoretical model analysis of the BMCoC computation reliability when subjected to unwanted effects, i.e., molecular signal delays and noise, and electrochemical sensors threshold settings that are based on either standard or blind detectors. Our numerical analysis found that the variations in the production delay and the molecular output signal concentration can impact on the computation reliability for the AND logic gate and ON-OFF switch. The molecular communications of synthetic engineered cells for logic gates integrated with sensing systems can lead to a new breed of biochips that can be used for numerous diagnostic applications.
Enhanced LSTM-based Service Decomposition for Mobile Augmented Reality
Venue: arXiv cs.NI 2021
Year: 2021

Undoubtedly, Mobile Augmented Reality (MAR) applications for 5G and Beyond wireless networks are witnessing a notable attention recently. However, they require significant computational and storage resources at the end device and/or the network via Edge Cloud (EC) support. In this work, a MAR service is considered under the lenses of microservices where MAR service components can be decomposed and anchored at different locations ranging from the end device to different ECs in order to optimize the overall service and network efficiency. To this end, we propose a mobility aware MAR service decomposition using a Long Short Term Memory (LSTM) deep neural network to provide efficient pro-active decision making in real-time. More specifically, the LSTM deep neural network is trained with optimal solutions derived from a mathematical programming formulation in an offline manner. Then, decision making at the inference stage is used to optimize service decomposition of MAR services. A wide set of numerical investigations reveal that the mobility aware LSTM deep neural network manage to outperform recently proposed schemes in terms of both decision making quality as well as computational time.
D-Cliques: Compensating for Data Heterogeneity with Topology in Decentralized Federated Learning
Venue: arXiv cs.LG 2021
Year: 2021

The convergence speed of machine learning models trained with Federated Learning is significantly affected by heterogeneous data partitions, even more so in a fully decentralized setting without a central server. In this paper, we show that the impact of label distribution skew, an important type of data heterogeneity, can be significantly reduced by carefully designing the underlying communication topology. We present D-Cliques, a novel topology that reduces gradient bias by grouping nodes in sparsely interconnected cliques such that the label distribution in a clique is representative of the global label distribution. We also show how to adapt the updates of decentralized SGD to obtain unbiased gradients and implement an effective momentum with D-Cliques. Our extensive empirical evaluation on MNIST and CIFAR10 demonstrates that our approach provides similar convergence speed as a fully-connected topology, which provides the best convergence in a data heterogeneous setting, with a significant reduction in the number of edges and messages. In a 1000-node topology, D-Cliques require 98% less edges and 96% less total messages, with further possible gains using a small-world topology across cliques.
BERT based Transformers lead the way in Extraction of Health Information from Social Media
Venue: arXiv cs.CL 2021
Year: 2021

This paper describes our submissions for the Social Media Mining for Health (SMM4H)2021 shared tasks. We participated in 2 tasks:(1) Classification, extraction and normalization of adverse drug effect (ADE) mentions in English tweets (Task-1) and (2) Classification of COVID-19 tweets containing symptoms(Task-6). Our approach for the first task uses the language representation model RoBERTa with a binary classification head. For the second task, we use BERTweet, based on RoBERTa. Fine-tuning is performed on the pre-trained models for both tasks. The models are placed on top of a custom domain-specific processing pipeline. Our system ranked first among all the submissions for subtask-1(a) with an F1-score of 61%. For subtask-1(b), our system obtained an F1-score of 50% with improvements up to +8% F1 over the score averaged across all submissions. The BERTweet model achieved an F1 score of 94% on SMM4H 2021 Task-6.
Tracking entities in technical procedures -- a new dataset and baselines
Venue: arXiv cs.CL 2021
Year: 2021

We introduce TechTrack, a new dataset for tracking entities in technical procedures. The dataset, prepared by annotating open domain articles from WikiHow, consists of 1351 procedures, e.g., "How to connect a printer", identifies more than 1200 unique entities with an average of 4.7 entities per procedure. We evaluate the performance of state-of-the-art models on the entity-tracking task and find that they are well below the human annotation performance. We describe how TechTrack can be used to take forward the research on understanding procedures from temporal texts.
Simultaneous Multi-Pivot Neural Machine Translation
Venue: arXiv cs.CL 2021
Year: 2021

Parallel corpora are indispensable for training neural machine translation (NMT) models, and parallel corpora for most language pairs do not exist or are scarce. In such cases, pivot language NMT can be helpful where a pivot language is used such that there exist parallel corpora between the source and pivot and pivot and target languages. Naturally, the quality of pivot language translation is more inferior to what could be achieved with a direct parallel corpus of a reasonable size for that pair. In a real-time simultaneous translation setting, the quality of pivot language translation deteriorates even further given that the model has to output translations the moment a few source words become available. To solve this issue, we propose multi-pivot translation and apply it to a simultaneous translation setting involving pivot languages. Our approach involves simultaneously translating a source language into multiple pivots, which are then simultaneously translated together into the target language by leveraging multi-source NMT. Our experiments in a low-resource setting using the N-way parallel UN corpus for Arabic to English NMT via French and Spanish as pivots reveals that in a simultaneous pivot NMT setting, using two pivot languages can lead to an improvement of up to 5.8 BLEU.
A High-fidelity, Machine-learning Enhanced Queueing Network Simulation Model for Hospital Ultrasound Operations
Venue: arXiv cs.LG 2021
Year: 2021

We collaborate with a large teaching hospital in Shenzhen, China and build a high-fidelity simulation model for its ultrasound center to predict key performance metrics, including the distributions of queue length, waiting time and sojourn time, with high accuracy. The key challenge to build an accurate simulation model is to understanding the complicated patient routing at the ultrasound center. To address the issue, we propose a novel two-level routing component to the queueing network model. We apply machine learning tools to calibrate the key components of the queueing model from data with enhanced accuracy.
Comparison of the FCFS and PS discipline in Redundancy Systems
Venue: arXiv cs.PF 2021
Year: 2021

We consider the c.o.c. redundancy system with $N$ parallel servers where incoming jobs are immediately replicated to $d$ servers chosen uniformly at random (without replacement). A job finishes service as soon as the first replica is completed, after which all the remaining replicas are abandoned. We compare the performance of the first-come first-served (FCFS) and processor-sharing (PS) discipline based on the stability condition, the tail behavior of the latency and the expected latency.
Automated Conformance Testing for JavaScript Engines via Deep Compiler Fuzzing
Venue: arXiv cs.SE 2021
Year: 2021

JavaScript (JS) is a popular, platform-independent programming language. To ensure the interoperability of JS programs across different platforms, the implementation of a JS engine should conform to the ECMAScript standard. However, doing so is challenging as there are many subtle definitions of API behaviors, and the definitions keep evolving. We present COMFORT, a new compiler fuzzing framework for detecting JS engine bugs and behaviors that deviate from the ECMAScript standard. COMFORT leverages the recent advance in deep learning-based language models to automatically generate JS test code. As a departure from prior fuzzers, COMFORT utilizes the well-structured ECMAScript specifications to automatically generate test data along with the test programs to expose bugs that could be overlooked by the developers or manually written test cases. COMFORT then applies differential testing methodologies on the generated test cases to expose standard conformance bugs. We apply COMFORT to ten mainstream JS engines. In 200 hours of automated concurrent testing runs, we discover bugs in all tested JS engines. We had identified 158 unique JS engine bugs, of which 129 have been verified, and 115 have already been fixed by the developers. Furthermore, 21 of the Comfort-generated test cases have been added to Test262, the official ECMAScript conformance test suite.
Symbolic Time and Space Tradeoffs for Probabilistic Verification
Venue: arXiv cs.LO 2021
Year: 2021

We present a faster symbolic algorithm for the following central problem in probabilistic verification: Compute the maximal end-component (MEC) decomposition of Markov decision processes (MDPs). This problem generalizes the SCC decomposition problem of graphs and closed recurrent sets of Markov chains. The model of symbolic algorithms is widely used in formal verification and model-checking, where access to the input model is restricted to only symbolic operations (e.g., basic set operations and computation of one-step neighborhood). For an input MDP with $n$ vertices and $m$ edges, the classical symbolic algorithm from the 1990s for the MEC decomposition requires $O(n^2)$ symbolic operations and $O(1)$ symbolic space. The only other symbolic algorithm for the MEC decomposition requires $O(n \sqrt{m})$ symbolic operations and $O(\sqrt{m})$ symbolic space. A main open question is whether the worst-case $O(n^2)$ bound for symbolic operations can be beaten. We present a symbolic algorithm that requires $\widetilde{O}(n^{1.5})$ symbolic operations and $\widetilde{O}(\sqrt{n})$ symbolic space. Moreover, the parametrization of our algorithm provides a trade-off between symbolic operations and symbolic space: for all $0<ε\leq 1/2$ the symbolic algorithm requires $\widetilde{O}(n^{2-ε})$ symbolic operations and $\widetilde{O}(n^ε)$ symbolic space ($\widetilde{O}$ hides poly-logarithmic factors). Using our techniques we present faster algorithms for computing the almost-sure winning regions of $ω$-regular objectives for MDPs. We consider the canonical parity objectives for $ω$-regular objectives, and for parity objectives with $d$-priorities we present an algorithm that computes the almost-sure winning region with $\widetilde{O}(n^{2-ε})$ symbolic operations and $\widetilde{O}(n^ε)$ symbolic space, for all $0 < ε\leq 1/2$.
A public key infrastructure for compliant spectrum use by wireless devices
Venue: arXiv cs.NI 2021
Year: 2021

A multitude of wireless devices for radio local area networks (RLAN) use frequency bands that are shared with other radio services or applications. European market surveillance activities on 5 GHz RLAN indicate a growing number of non-compliant devices that do not fulfill all radio regulation requirements and may cause interference. This paper describes a concept and method for mitigating the use of non-compliant equipment based on a public key infrastructure. If every RLAN device holds a security certificate, which is revoked when non-compliance is found, then the connection establishment between devices can be made dependent on compliance. The aim is that legitimate equipment rejects connections to non-compliant devices, so that non-compliant devices will become isolated.
Advanced Lane Detection Model for the Virtual Development of Highly Automated Functions
Venue: arXiv cs.RO 2021
Year: 2021

Virtual development and prototyping has already become an integral part in the field of automated driving systems (ADS). There are plenty of software tools that are used for the virtual development of ADS. One such tool is CarMaker from IPG Automotive, which is widely used in the scientific community and in the automotive industry. It offers a broad spectrum of implementation and modelling possibilities of the vehicle, driver behavior, control, sensors, and environmental models. Focusing on the virtual development of highly automated driving functions on the vehicle guidance level, it is essential to perceive the environment in a realistic manner. For the longitudinal and lateral path guidance line detection sensors are necessary for the determination of the relevant perceiving vehicle and for the planning of trajectories. For this purpose, a lane sensor model was developed in order to efficiently detect lanes in the simulation environment of CarMaker. The so-called advanced lane detection model (ALDM) is optimized regarding the calculation time and is for the lateral and longitudinal vehicle guidance in CarMaker.
IndT5: A Text-to-Text Transformer for 10 Indigenous Languages
Venue: arXiv cs.CL 2021
Year: 2021

Transformer language models have become fundamental components of natural language processing based pipelines. Although several Transformer models have been introduced to serve many languages, there is a shortage of models pre-trained for low-resource and Indigenous languages. In this work, we introduce IndT5, the first Transformer language model for Indigenous languages. To train IndT5, we build IndCorpus--a new dataset for ten Indigenous languages and Spanish. We also present the application of IndT5 to machine translation by investigating different approaches to translate between Spanish and the Indigenous languages as part of our contribution to the AmericasNLP 2021 Shared Task on Open Machine Translation. IndT5 and IndCorpus are publicly available for research
Agent-based Framework for Self-Organization of Collective and Autonomous Shuttle Fleets
Venue: arXiv cs.MA 2021
Year: 2021

The mobility of people is at the center of transportation planning and decision-making of the cities of the future. In order to accelerate the transition to zero-emissions and to maximize air quality benefits, smart cities are prioritizing walking, cycling, shared mobility services and public transport over the use of private cars. Extensive progress has been made in autonomous and electric cars. Autonomous Vehicles (AV) are increasingly capable of moving without full control of humans, automating some aspects of driving, such as steering or braking. For these reasons, cities are investing in the infrastructure and technology needed to support connected, multi-modal transit networks that include shared electric Autonomous Vehicles (AV). The relationship between traditional public transport and new mobility services is in the spotlight and need to be rethought. This paper proposes an agent-based simulation framework that allows for the creation and simulation of mobility scenarios to investigate the impact of new mobility modes on a city daily life. It lets traffic planners explore the cooperative integration of AV using a decentralized control approach. A prototype has been implemented and validated with data of the city of Trento.
Unmasking the Mask -- Evaluating Social Biases in Masked Language Models
Venue: arXiv cs.CL 2021
Year: 2021

Masked Language Models (MLMs) have shown superior performances in numerous downstream NLP tasks when used as text encoders. Unfortunately, MLMs also demonstrate significantly worrying levels of social biases. We show that the previously proposed evaluation metrics for quantifying the social biases in MLMs are problematic due to following reasons: (1) prediction accuracy of the masked tokens itself tend to be low in some MLMs, which raises questions regarding the reliability of the evaluation metrics that use the (pseudo) likelihood of the predicted tokens, and (2) the correlation between the prediction accuracy of the mask and the performance in downstream NLP tasks is not taken into consideration, and (3) high frequency words in the training data are masked more often, introducing noise due to this selection bias in the test cases. To overcome the above-mentioned disfluencies, we propose All Unmasked Likelihood (AUL), a bias evaluation measure that predicts all tokens in a test case given the MLM embedding of the unmasked input. We find that AUL accurately detects different types of biases in MLMs. We also propose AUL with attention weights (AULA) to evaluate tokens based on their importance in a sentence. However, unlike AUL and AULA, previously proposed bias evaluation measures for MLMs systematically overestimate the measured biases, and are heavily influenced by the unmasked tokens in the context.
Street-Map Based Validation of Semantic Segmentation in Autonomous Driving
Venue: arXiv cs.CV 2021
Year: 2021

Artificial intelligence for autonomous driving must meet strict requirements on safety and robustness, which motivates the thorough validation of learned models. However, current validation approaches mostly require ground truth data and are thus both cost-intensive and limited in their applicability. We propose to overcome these limitations by a model agnostic validation using a-priori knowledge from street maps. In particular, we show how to validate semantic segmentation masks and demonstrate the potential of our approach using OpenStreetMap. We introduce validation metrics that indicate false positive or negative road segments. Besides the validation approach, we present a method to correct the vehicle's GPS position so that a more accurate localization can be used for the street-map based validation. Lastly, we present quantitative results on the Cityscapes dataset indicating that our validation approach can indeed uncover errors in semantic segmentation masks.
Innovation for Sustainability in the Global South: Bibliometric findings from management & business and STEM (Science, Technology, Engineering and Mathematics) fields in developing countries
Venue: arXiv cs.DL 2021
Year: 2021

Research on innovation and sustainability is prolific but fragmented. This study integrates the research on innovation in management and business and STEM fields for sustainability in a unified framework for the case of developing countries (i.e., the Global South). It presents and discusses the output, impact, and structure of such research based on a sample of 14,000+ articles and conference proceedings extracted from the bibliographic database Scopus. The findings reveal research output inflections after global announcements such as UN-Earth Summits. The study also reveals the indisputable leadership of China in overall output and research agenda-setting. Nonetheless, countries such as India, Mexico, and Nigeria are either more efficient or impactful. GS research published in highly reputable journals is still scarce but increasing modestly. Central topic clusters (e.g., knowledge management) remain peripheral to the global Sustainable Development Goals (SDGs) research landscape. Finally, academic-corporate collaboration is in its infancy and limited to particular economic sectors: energy, pharmaceuticals, and high-tech.
AffectiveNet: Affective-Motion Feature Learningfor Micro Expression Recognition
Venue: arXiv cs.MM 2021
Year: 2021

Micro-expressions are hard to spot due to fleeting and involuntary moments of facial muscles. Interpretation of micro emotions from video clips is a challenging task. In this paper we propose an affective-motion imaging that cumulates rapid and short-lived variational information of micro expressions into a single response. Moreover, we have proposed an AffectiveNet:affective-motion feature learning network that can perceive subtle changes and learns the most discriminative dynamic features to describe the emotion classes. The AffectiveNet holds two blocks: MICRoFeat and MFL block. MICRoFeat block conserves the scale-invariant features, which allows network to capture both coarse and tiny edge variations. While MFL block learns micro-level dynamic variations from two different intermediate convolutional layers. Effectiveness of the proposed network is tested over four datasets by using two experimental setups: person independent (PI) and cross dataset (CD) validation. The experimental results of the proposed network outperforms the state-of-the-art approaches with significant margin for MER approaches.