OpenRaiser/Intern-Atlas huggingface dataset:
Prompt-tuning in ASR systems for efficient domain-adaptation
Venue: arXiv cs.CL 2021
Year: 2021

Automatic Speech Recognition (ASR) systems have found their use in numerous industrial applications in very diverse domains. Since domain-specific systems perform better than their generic counterparts on in-domain evaluation, the need for memory and compute-efficient domain adaptation is obvious. Particularly, adapting parameter-heavy transformer-based language models used for rescoring ASR hypothesis is challenging. In this work, we overcome the problem using prompt-tuning, a methodology that trains a small number of domain token embedding parameters to prime a transformer-based LM to a particular domain. With just a handful of extra parameters per domain, we achieve much better perplexity scores over the baseline of using an unadapted LM. Despite being parameter-efficient, these improvements are comparable to those of fully-fine-tuned models with hundreds of millions of parameters. We replicate our findings in perplexity numbers to Word Error Rate in a domain-specific ASR system for one such domain.
EventBERT: A Pre-Trained Model for Event Correlation Reasoning
Venue: arXiv cs.CL 2021
Year: 2021

Event correlation reasoning infers whether a natural language paragraph containing multiple events conforms to human common sense. For example, "Andrew was very drowsy, so he took a long nap, and now he is very alert" is sound and reasonable. In contrast, "Andrew was very drowsy, so he stayed up a long time, now he is very alert" does not comply with human common sense. Such reasoning capability is essential for many downstream tasks, such as script reasoning, abductive reasoning, narrative incoherence, story cloze test, etc. However, conducting event correlation reasoning is challenging due to a lack of large amounts of diverse event-based knowledge and difficulty in capturing correlation among multiple events. In this paper, we propose EventBERT, a pre-trained model to encapsulate eventuality knowledge from unlabeled text. Specifically, we collect a large volume of training examples by identifying natural language paragraphs that describe multiple correlated events and further extracting event spans in an unsupervised manner. We then propose three novel event- and correlation-based learning objectives to pre-train an event correlation model on our created training corpus. Empirical results show EventBERT outperforms strong baselines on four downstream tasks, and achieves SoTA results on most of them. Besides, it outperforms existing pre-trained models by a large margin, e.g., 6.5~23%, in zero-shot learning of these tasks.
Simple Attention Module based Speaker Verification with Iterative noisy label detection
Venue: arXiv cs.SD 2021
Year: 2021

Recently, the attention mechanism such as squeeze-and-excitation module (SE) and convolutional block attention module (CBAM) has achieved great success in deep learning-based speaker verification system. This paper introduces an alternative effective yet simple one, i.e., simple attention module (SimAM), for speaker verification. The SimAM module is a plug-and-play module without extra modal parameters. In addition, we propose a noisy label detection method to iteratively filter out the data samples with a noisy label from the training data, considering that a large-scale dataset labeled with human annotation or other automated processes may contain noisy labels. Data with the noisy label may over parameterize a deep neural network (DNN) and result in a performance drop due to the memorization effect of the DNN. Experiments are conducted on VoxCeleb dataset. The speaker verification model with SimAM achieves the 0.675% equal error rate (EER) on VoxCeleb1 original test trials. Our proposed iterative noisy label detection method further reduces the EER to 0.643%.
EIHW-MTG DiCOVA 2021 Challenge System Report
Venue: arXiv cs.SD 2021
Year: 2021

This paper aims to automatically detect COVID-19 patients by analysing the acoustic information embedded in coughs. COVID-19 affects the respiratory system, and, consequently, respiratory-related signals have the potential to contain salient information for the task at hand. We focus on analysing the spectrogram representations of coughing samples with the aim to investigate whether COVID-19 alters the frequency content of these signals. Furthermore, this work also assesses the impact of gender in the automatic detection of COVID-19. To extract deep learnt representations of the spectrograms, we compare the performance of a cough-specific, and a Resnet18 pre-trained Convolutional Neural Network (CNN). Additionally, our approach explores the use of contextual attention, so the model can learn to highlight the most relevant deep learnt features extracted by the CNN. We conduct our experiments on the dataset released for the Cough Sound Track of the DiCOVA 2021 Challenge. The best performance on the test set is obtained using the Resnet18 pre-trained CNN with contextual attention, which scored an Area Under the Curve (AUC) of 70.91 at 80% sensitivity.
Color Counting for Fashion, Art, and Design
Venue: arXiv cs.CV 2021
Year: 2021

Color modelling and extraction is an important topic in fashion, art, and design. Recommender systems, color-based retrieval, decorating, and fashion design can benefit from color extraction tools. Research has shown that modeling color so that it can be automatically analyzed and / or extracted is a difficult task. Unlike machines, color perception, although very subjective, is much simpler for humans. That being said, the first step in color modeling is to estimate the number of colors in the item / object. This is because color models can take advantage of the number of colors as the seed for better modelling, e.g., to make color extraction further deterministic. We aim in this work to develop and test models that can count the number of colors of clothing and other items. We propose a novel color counting method based on cumulative color histogram, which stands out among other methods. We compare the method we propose with other methods that utilize exhaustive color search that uses Gaussian Mixture Models (GMMs) and K-Means as bases for scoring the optimal number of colors, in addition to another method that relies on deep learning models. Unfortunately, the GMM, K-Means, and Deep Learning models all fail to accurately capture the number of colors. Our proposed method can provide the color baseline that can be used in AI-based fashion applications, and can also find applications in other areas, for example, interior design. To the best of our knowledge, this work is the first of its kind that addresses the problem of color-counting machine.
Expert-driven Trace Clustering with Instance-level Constraints
Venue: arXiv cs.LG 2021
Year: 2021

Within the field of process mining, several different trace clustering approaches exist for partitioning traces or process instances into similar groups. Typically, this partitioning is based on certain patterns or similarity between the traces, or driven by the discovery of a process model for each cluster. The main drawback of these techniques, however, is that their solutions are usually hard to evaluate or justify by domain experts. In this paper, we present two constrained trace clustering techniques that are capable to leverage expert knowledge in the form of instance-level constraints. In an extensive experimental evaluation using two real-life datasets, we show that our novel techniques are indeed capable of producing clustering solutions that are more justifiable without a substantial negative impact on their quality.
A Time Encoding approach to training Spiking Neural Networks
Venue: arXiv cs.NE 2021
Year: 2021

While Spiking Neural Networks (SNNs) have been gaining in popularity, it seems that the algorithms used to train them are not powerful enough to solve the same tasks as those tackled by classical Artificial Neural Networks (ANNs). In this paper, we provide an extra tool to help us understand and train SNNs by using theory from the field of time encoding. Time encoding machines (TEMs) can be used to model integrate-and-fire neurons and have well-understood reconstruction properties. We will see how one can take inspiration from the field of TEMs to interpret the spike times of SNNs as constraints on the SNNs' weight matrices. More specifically, we study how to train one-layer SNNs by solving a set of linear constraints, and how to train two-layer SNNs by leveraging the all-or-none and asynchronous properties of the spikes emitted by SNNs. These properties of spikes result in an alternative to backpropagation which is not possible in the case of simultaneous and graded activations as in classical ANNs.
Masader: Metadata Sourcing for Arabic Text and Speech Data Resources
Venue: arXiv cs.CL 2021
Year: 2021

The NLP pipeline has evolved dramatically in the last few years. The first step in the pipeline is to find suitable annotated datasets to evaluate the tasks we are trying to solve. Unfortunately, most of the published datasets lack metadata annotations that describe their attributes. Not to mention, the absence of a public catalogue that indexes all the publicly available datasets related to specific regions or languages. When we consider low-resource dialectical languages, for example, this issue becomes more prominent. In this paper we create \textit{Masader}, the largest public catalogue for Arabic NLP datasets, which consists of 200 datasets annotated with 25 attributes. Furthermore, We develop a metadata annotation strategy that could be extended to other languages. We also make remarks and highlight some issues about the current status of Arabic NLP datasets and suggest recommendations to address them.
The perception of Architectural Smells in industrial practice
Venue: arXiv cs.SE 2021
Year: 2021

Architectural Technical Debt (ATD) is considered as the most significant type of TD in industrial practice. In this study, we interview 21 software engineers and architects to investigate a specific type of ATD, namely architectural smells (AS). Our goal is to understand the phenomenon of AS better and support practitioners to better manage it and researchers to offer relevant support. The findings of this study provide insights on how practitioners perceive AS and how they introduce them, the maintenance and evolution issues they experienced and associated to the presence of AS, and what practices and tools they adopt to manage AS.
Optical Character Recognition of 19th Century Classical Commentaries: the Current State of Affairs
Venue: arXiv cs.DL 2021
Year: 2021

Together with critical editions and translations, commentaries are one of the main genres of publication in literary and textual scholarship, and have a century-long tradition. Yet, the exploitation of thousands of digitized historical commentaries was hitherto hindered by the poor quality of Optical Character Recognition (OCR), especially on commentaries to Greek texts. In this paper, we evaluate the performances of two pipelines suitable for the OCR of historical classical commentaries. Our results show that Kraken + Ciaconna reaches a substantially lower character error rate (CER) than Tesseract/OCR-D on commentary sections with high density of polytonic Greek text (average CER 7% vs. 13%), while Tesseract/OCR-D is slightly more accurate than Kraken + Ciaconna on text sections written predominantly in Latin script (average CER 8.2% vs. 8.4%). As part of this paper, we also release GT4HistComment, a small dataset with OCR ground truth for 19th classical commentaries and Pogretra, a large collection of training data and pre-trained models for a wide variety of ancient Greek typefaces.
Safe Driving via Expert Guided Policy Optimization
Venue: arXiv cs.AI 2021
Year: 2021

When learning common skills like driving, beginners usually have domain experts standing by to ensure the safety of the learning process. We formulate such learning scheme under the Expert-in-the-loop Reinforcement Learning where a guardian is introduced to safeguard the exploration of the learning agent. While allowing the sufficient exploration in the uncertain environment, the guardian intervenes under dangerous situations and demonstrates the correct actions to avoid potential accidents. Thus ERL enables both exploration and expert's partial demonstration as two training sources. Following such a setting, we develop a novel Expert Guided Policy Optimization (EGPO) method which integrates the guardian in the loop of reinforcement learning. The guardian is composed of an expert policy to generate demonstration and a switch function to decide when to intervene. Particularly, a constrained optimization technique is used to tackle the trivial solution that the agent deliberately behaves dangerously to deceive the expert into taking over. Offline RL technique is further used to learn from the partial demonstration generated by the expert. Safe driving experiments show that our method achieves superior training and test-time safety, outperforms baselines with a substantial margin in sample efficiency, and preserves the generalizabiliy to unseen environments in test-time. Demo video and source code are available at: https://decisionforce.github.io/EGPO/
A Review on Human Pose Estimation
Venue: arXiv cs.CV 2021
Year: 2021

The phenomenon of Human Pose Estimation (HPE) is a problem that has been explored over the years, particularly in computer vision. But what exactly is it? To answer this, the concept of a pose must first be understood. Pose can be defined as the arrangement of human joints in a specific manner. Therefore, we can define the problem of Human Pose Estimation as the localization of human joints or predefined landmarks in images and videos. There are several types of pose estimation, including body, face, and hand, as well as many aspects to it. This paper will cover them, starting with the classical approaches to HPE to the Deep Learning based models.
Sim2Ls: FAIR simulation workflows and data
Venue: arXiv cs.DL 2021
Year: 2021

Just like the scientific data they generate, simulation workflows for research should be findable, accessible, interoperable, and reusable (FAIR). However, while significant progress has been made towards FAIR data, the majority of science and engineering workflows used in research remain poorly documented and often unavailable, involving ad hoc scripts and manual steps, hindering reproducibility and stifling progress. We introduce Sim2Ls (pronounced simtools) and the Sim2L Python library that allow developers to create and share end-to-end computational workflows with well-defined and verified inputs and outputs. The Sim2L library makes Sim2Ls, their requirements, and their services discoverable, verifies inputs and outputs, and automatically stores results in a globally-accessible simulation cache and results database. This simulation ecosystem is available in nanoHUB, an open platform that also provides publication services for Sim2Ls, a computational environment for developers and users, and the hardware to execute runs and store results at no cost. We exemplify the use of Sim2Ls using two applications and discuss best practices towards FAIR simulation workflows and associated data.
Audio-Visual Scene-Aware Dialog and Reasoning using Audio-Visual Transformers with Joint Student-Teacher Learning
Venue: arXiv cs.CL 2021
Year: 2021

In previous work, we have proposed the Audio-Visual Scene-Aware Dialog (AVSD) task, collected an AVSD dataset, developed AVSD technologies, and hosted an AVSD challenge track at both the 7th and 8th Dialog System Technology Challenges (DSTC7, DSTC8). In these challenges, the best-performing systems relied heavily on human-generated descriptions of the video content, which were available in the datasets but would be unavailable in real-world applications. To promote further advancements for real-world applications, we proposed a third AVSD challenge, at DSTC10, with two modifications: 1) the human-created description is unavailable at inference time, and 2) systems must demonstrate temporal reasoning by finding evidence from the video to support each answer. This paper introduces the new task that includes temporal reasoning and our new extension of the AVSD dataset for DSTC10, for which we collected human-generated temporal reasoning data. We also introduce a baseline system built using an AV-transformer, which we released along with the new dataset. Finally, this paper introduces a new system that extends our baseline system with attentional multimodal fusion, joint student-teacher learning (JSTL), and model combination techniques, achieving state-of-the-art performances on the AVSD datasets for DSTC7, DSTC8, and DSTC10. We also propose two temporal reasoning methods for AVSD: one attention-based, and one based on a time-domain region proposal network.
Teaching Models new APIs: Domain-Agnostic Simulators for Task Oriented Dialogue
Venue: arXiv cs.CL 2021
Year: 2021

We demonstrate that large language models are able to simulate Task Oriented Dialogues in novel domains, provided only with an API implementation and a list of goals. We show these simulations can formulate online, automatic metrics that correlate well with human evaluations. Furthermore, by checking for whether the User's goals are met, we can use simulation to repeatedly generate training data and improve the quality of simulations themselves. With no human intervention or domain-specific training data, our simulations bootstrap end-to-end models which achieve a 37\% error reduction in previously unseen domains. By including as few as 32 domain-specific conversations, bootstrapped models can match the performance of a fully-supervised model with $10\times$ more data. To our knowledge, this is the first time simulations have been shown to be effective at bootstrapping models without explicitly requiring any domain-specific training data, rule-engineering, or humans-in-the-loop.
DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries
Venue: arXiv cs.CV 2021
Year: 2021

We introduce a framework for multi-camera 3D object detection. In contrast to existing works, which estimate 3D bounding boxes directly from monocular images or use depth prediction networks to generate input for 3D object detection from 2D information, our method manipulates predictions directly in 3D space. Our architecture extracts 2D features from multiple camera images and then uses a sparse set of 3D object queries to index into these 2D features, linking 3D positions to multi-view images using camera transformation matrices. Finally, our model makes a bounding box prediction per object query, using a set-to-set loss to measure the discrepancy between the ground-truth and the prediction. This top-down approach outperforms its bottom-up counterpart in which object bounding box prediction follows per-pixel depth estimation, since it does not suffer from the compounding error introduced by a depth prediction model. Moreover, our method does not require post-processing such as non-maximum suppression, dramatically improving inference speed. We achieve state-of-the-art performance on the nuScenes autonomous driving benchmark.
Block Contextual MDPs for Continual Learning
Venue: arXiv cs.LG 2021
Year: 2021

In reinforcement learning (RL), when defining a Markov Decision Process (MDP), the environment dynamics is implicitly assumed to be stationary. This assumption of stationarity, while simplifying, can be unrealistic in many scenarios. In the continual reinforcement learning scenario, the sequence of tasks is another source of nonstationarity. In this work, we propose to examine this continual reinforcement learning setting through the block contextual MDP (BC-MDP) framework, which enables us to relax the assumption of stationarity. This framework challenges RL algorithms to handle both nonstationarity and rich observation settings and, by additionally leveraging smoothness properties, enables us to study generalization bounds for this setting. Finally, we take inspiration from adaptive control to propose a novel algorithm that addresses the challenges introduced by this more realistic BC-MDP setting, allows for zero-shot adaptation at evaluation time, and achieves strong performance on several nonstationary environments.
Masking Effects in Combined Hardness and Stiffness Rendering Using an Encountered-Type Haptic Display
Venue: arXiv cs.RO 2021
Year: 2021

Rendering stable hard surfaces is an important problem in haptics for many tasks, including training simulators for orthopedic surgery or dentistry. Current impedance devices cannot provide enough force and stiffness to render a wall, and the high friction and inertia of admittance devices make it difficult to render free space. We propose to address these limitations by combining haptic augmented reality, untethered haptic interaction, and an encountered-type haptic display. We attach a plate with the desired hardness on the kinesthetic device's end-effector, which the user interacts with using an untethered stylus. This method allows us to directly change the hardness of the end-effector based on the rendered object. In this paper, we evaluate how changing the hardness of the end-effector can mask the device's stiffness and affect the user's perception. The results of our human subject experiment indicate that when the end-effector is made of a hard material, it is difficult for users to perceive when the underlying stiffness being rendered by the device is changed, but this stiffness change is easy to distinguish while the end-effector is made of a soft material. These results show promise for our approach in avoiding the limitations of haptic devices when rendering hard surfaces.
The Design and Simulation of Biomimetic Fish Robot for Aquatic Creature Study
Venue: arXiv cs.RO 2021
Year: 2021

In the application of underwater creature study, comparing with propeller-powered ROVs and servo motor actuated robotic fish, novel biomimetic fish robot designs with soft actuation structure could interact with aquatic creatures closely and record authentic habitats and behaviours. This final project report presents the detailed design process of a hydraulic soft actuator powered robotic fish for aquatic creature study capable of swimming along the 3D trajectory. The robotic fish is designed based on the analysis of the pro and cons of existing designs. Except for the mechanical and electronic designs and manufacturing method of crucial components, a simplified open-loop control algorithm was designed to check the functionality of the application board and microcontroller in the Proteus simulation environment. As the key component of the robotic fish, Finite Element Method (FEM) simulations were conducted to visualise the soft actuator's deformation under different pressure to validate the design. Computational Fluid Dynamics (CFD) simulations were also conducted to improve the hydrodynamic efficiency of the shape of robotic fish. Although physical manufacturing is impossible due to the pandemic, the simulations show overall good performance in terms of control, actuation, and hydrodynamic efficiency.
6G: Connectivity in the Era of Distributed Intelligence
Venue: arXiv cs.IT 2021
Year: 2021

The confluence of 5G and AI is transforming wireless networks to deliver diverse services at the Edge, driving towards a vision of pervasive distributed intelligence. Future 6G networks will need to deliver quality of experience through seamless integration of communication, computation and AI. Therefore, networks must become intelligent, distributed, scalable, and programmable platforms across the continuum of data delivery to address the ever-increasing service requirements and deployment complexity. We present novel results across three research directions that are expected to be integral to 6G systems and also discuss newer 6G metrics.