OpenRaiser/Intern-Atlas huggingface dataset:
Understanding Medical Conversations: Rich Transcription, Confidence Scores & Information Extraction
Venue: arXiv cs.LG 2021
Year: 2021

In this paper, we describe novel components for extracting clinically relevant information from medical conversations which will be available as Google APIs. We describe a transformer-based Recurrent Neural Network Transducer (RNN-T) model tailored for long-form audio, which can produce rich transcriptions including speaker segmentation, speaker role labeling, punctuation and capitalization. On a representative test set, we compare performance of RNN-T models with different encoders, units and streaming constraints. Our transformer-based streaming model performs at about 20% WER on the ASR task, 6% WDER on the diarization task, 43% SER on periods, 52% SER on commas, 43% SER on question marks and 30% SER on capitalization. Our recognizer is paired with a confidence model that utilizes both acoustic and lexical features from the recognizer. The model performs at about 0.37 NCE. Finally, we describe a RNN-T based tagging model. The performance of the model depends on the ontologies, with F-scores of 0.90 for medications, 0.76 for symptoms, 0.75 for conditions, 0.76 for diagnosis, and 0.61 for treatments. While there is still room for improvement, our results suggest that these models are sufficiently accurate for practical applications.
Hippocampus-heuristic Character Recognition Network for Zero-shot Learning
Venue: arXiv cs.CV 2021
Year: 2021

The recognition of Chinese characters has always been a challenging task due to their huge variety and complex structures. The latest research proves that such an enormous character set can be decomposed into a collection of about 500 fundamental Chinese radicals, and based on which this problem can be solved effectively. While with the constant advent of novel Chinese characters, the number of basic radicals is also expanding. The current methods that entirely rely on existing radicals are not flexible for identifying these novel characters and fail to recognize these Chinese characters without learning all of their radicals in the training stage. To this end, this paper proposes a novel Hippocampus-heuristic Character Recognition Network (HCRN), which references the way of hippocampus thinking, and can recognize unseen Chinese characters (namely zero-shot learning) only by training part of radicals. More specifically, the network architecture of HCRN is a new pseudo-siamese network designed by us, which can learn features from pairs of input training character samples and use them to predict unseen Chinese characters. The experimental results show that HCRN is robust and effective. It can accurately predict about 16,330 unseen testing Chinese characters relied on only 500 trained Chinese characters. The recognition accuracy of HCRN outperforms the state-of-the-art Chinese radical recognition approach by 15% (from 85.1% to 99.9%) for recognizing unseen Chinese characters.
HBert + BiasCorp -- Fighting Racism on the Web
Venue: arXiv cs.CL 2021
Year: 2021

Subtle and overt racism is still present both in physical and online communities today and has impacted many lives in different segments of the society. In this short piece of work, we present how we're tackling this societal issue with Natural Language Processing. We are releasing BiasCorp, a dataset containing 139,090 comments and news segment from three specific sources - Fox News, BreitbartNews and YouTube. The first batch (45,000 manually annotated) is ready for publication. We are currently in the final phase of manually labeling the remaining dataset using Amazon Mechanical Turk. BERT has been used widely in several downstream tasks. In this work, we present hBERT, where we modify certain layers of the pretrained BERT model with the new Hopfield Layer. hBert generalizes well across different distributions with the added advantage of a reduced model complexity. We are also releasing a JavaScript library and a Chrome Extension Application, to help developers make use of our trained model in web applications (say chat application) and for users to identify and report racially biased contents on the web respectively.
Non-autoregressive Mandarin-English Code-switching Speech Recognition
Venue: arXiv cs.CL 2021
Year: 2021

Mandarin-English code-switching (CS) is frequently used among East and Southeast Asian people. However, the intra-sentence language switching of the two very different languages makes recognizing CS speech challenging. Meanwhile, the recent successful non-autoregressive (NAR) ASR models remove the need for left-to-right beam decoding in autoregressive (AR) models and achieved outstanding performance and fast inference speed, but it has not been applied to Mandarin-English CS speech recognition. This paper takes advantage of the Mask-CTC NAR ASR framework to tackle the CS speech recognition issue. We further propose to change the Mandarin output target of the encoder to Pinyin for faster encoder training and introduce the Pinyin-to-Mandarin decoder to learn contextualized information. Moreover, we use word embedding label smoothing to regularize the decoder with contextualized information and projection matrix regularization to bridge that gap between the encoder and decoder. We evaluate these methods on the SEAME corpus and achieved exciting results.
Learning from Self-Discrepancy via Multiple Co-teaching for Cross-Domain Person Re-Identification
Venue: arXiv cs.CV 2021
Year: 2021

Employing clustering strategy to assign unlabeled target images with pseudo labels has become a trend for person re-identification (re-ID) algorithms in domain adaptation. A potential limitation of these clustering-based methods is that they always tend to introduce noisy labels, which will undoubtedly hamper the performance of our re-ID system. To handle this limitation, an intuitive solution is to utilize collaborative training to purify the pseudo label quality. However, there exists a challenge that the complementarity of two networks, which inevitably share a high similarity, becomes weakened gradually as training process goes on; worse still, these approaches typically ignore to consider the self-discrepancy of intra-class relations. To address this issue, in this paper, we propose a multiple co-teaching framework for domain adaptive person re-ID, opening up a promising direction about self-discrepancy problem under unsupervised condition. On top of that, a mean-teaching mechanism is leveraged to enlarge the difference and discover more complementary features. Comprehensive experiments conducted on several large-scale datasets show that our method achieves competitive performance compared with the state-of-the-arts.
Balancing Predictive Relevance of Ligand Biochemical Activities
Venue: arXiv cs.LG 2021
Year: 2021

In this paper, we present a technique for balancing predictive relevance models related to supervised modelling ligand biochemical activities to biological targets. We train uncalibrated models employing conventional supervised machine learning technique, namely Support Vector Machines. Unfortunately, SVMs have a serious drawback. They are sensitive to imbalanced datasets, outliers and high multicollinearity among training samples, which could be a cause of preferencing one group over another. Thus, an additional calibration could be required for balancing a predictive relevance of models. As a technique for this balancing, we propose the Platt's scaling. The achieved results were demonstrated on single-target models trained on datasets exported from the ExCAPE database. Unlike traditional used machine techniques, we focus on decreasing uncertainty employing deterministic solvers.
Underground Astronauts: Understanding the Sporting Science of Speleology and its Implications for HCI
Venue: arXiv cs.HC 2021
Year: 2021

In this paper, we present a qualitative study on speleology that aims to widen the current understanding of people's practices in Nature and identify a design space for technology that supports such practices. Speleology is a practice based on the discovery, study, and dissemination of natural cavities. Speleologists are amateur experts who often collaborate with scientists and local institutions to understand the geology, hydrology, and biology of a territory. Their skills are at the same time physical, technical, and theoretical; this is why speleology is defined as a 'sporting science'. Being at the boundary between outdoor adventure sports and citizen science, speleology is an interesting case study for investigating the variety and complexity of activities carried out in the natural context. We interviewed 15 experienced speleologists to explore their goals, routines, vision of the outdoors, and attitude towards technology. From our study, it emerged that i) the excitement of discovery and the unpredictability of an explorative trip are the strongest motivations for people to engage in speleology; ii) physical skilfulness is a means for knowledge generation; iii) the practice is necessarily collective and requires group coordination. From these findings, an ambivalent attitude towards technology emerged: on the one hand, the scientific vocation of speleology welcomes technology supporting the development of knowledge; on the other hand, aspects typical of adventure sports lead to resistance to technology facilitating the physical performance. We conclude the article by presenting design considerations for devices supporting speleology, as well as a few reflections on how communities of speleologists can inspire citizen science projects.
A Latent space solver for PDE generalization
Venue: arXiv cs.LG 2021
Year: 2021

In this work we propose a hybrid solver to solve partial differential equation (PDE)s in the latent space. The solver uses an iterative inferencing strategy combined with solution initialization to improve generalization of PDE solutions. The solver is tested on an engineering case and the results show that it can generalize well to several PDE conditions.
Jolie & LEMMA: Model-Driven Engineering and Programming Languages Meet on Microservices
Venue: arXiv cs.SE 2021
Year: 2021

In the field of microservices, Model-Driven Engineering has emerged as a powerful methodology for architectural design, and new programming languages have introduced language abstractions to deal with microservice development more effectively. In this article, we present the first preliminary investigation of how the two approaches can be married, taking the LEMMA framework and the Jolie programming language as respective representatives. By developing a conceptual metamodel for Jolie, we elicit a strong link between the two approaches, which shows that there is much to gain. We discuss a few low-hanging fruits that come from our finding and present some interesting future directions that arise from our new viewpoint.
COVID-19 Detection in Cough, Breath and Speech using Deep Transfer Learning and Bottleneck Features
Venue: arXiv cs.SD 2021
Year: 2021

We present an experimental investigation into the effectiveness of transfer learning and bottleneck feature extraction in detecting COVID-19 from audio recordings of cough, breath and speech. This type of screening is non-contact, does not require specialist medical expertise or laboratory facilities and can be deployed on inexpensive consumer hardware. We use datasets that contain recordings of coughing, sneezing, speech and other noises, but do not contain COVID-19 labels, to pre-train three deep neural networks: a CNN, an LSTM and a Resnet50. These pre-trained networks are subsequently either fine-tuned using smaller datasets of coughing with COVID-19 labels in the process of transfer learning, or are used as bottleneck feature extractors. Results show that a Resnet50 classifier trained by this transfer learning process delivers optimal or near-optimal performance across all datasets achieving areas under the receiver operating characteristic (ROC AUC) of 0.98, 0.94 and 0.92 respectively for all three sound classes (coughs, breaths and speech). This indicates that coughs carry the strongest COVID-19 signature, followed by breath and speech. Our results also show that applying transfer learning and extracting bottleneck features using the larger datasets without COVID-19 labels led not only to improve performance, but also to minimise the standard deviation of the classifier AUCs among the outer folds of the leave-$p$-out cross-validation, indicating better generalisation. We conclude that deep transfer learning and bottleneck feature extraction can improve COVID-19 cough, breath and speech audio classification, yielding automatic classifiers with higher accuracy.
Adversarial Semi-supervised Learning for Corporate Credit Ratings
Venue: arXiv cs.LG 2021
Year: 2021

Corporate credit rating is an analysis of credit risks within a corporation, which plays a vital role during the management of financial risk. Traditionally, the rating assessment process based on the historical profile of corporation is usually expensive and complicated, which often takes months. Therefore, most of the corporations, which are lacking in money and time, can't get their own credit level. However, we believe that although these corporations haven't their credit rating levels (unlabeled data), this big data contains useful knowledge to improve credit system. In this work, its major challenge lies in how to effectively learn the knowledge from unlabeled data and help improve the performance of the credit rating system. Specifically, we consider the problem of adversarial semi-supervised learning (ASSL) for corporate credit rating which has been rarely researched before. A novel framework adversarial semi-supervised learning for corporate credit rating (ASSL4CCR) which includes two phases is proposed to address these problems. In the first phase, we train a normal rating system via a normal machine-learning algorithm to give unlabeled data pseudo rating level. Then in the second phase, adversarial semi-supervised learning is applied uniting labeled data and pseudo-labeled data. To demonstrate the effectiveness of the proposed ASSL4CCR, we conduct extensive experiments on the Chinese public-listed corporate rating dataset, which proves that ASSL4CCR outperforms the state-of-the-art methods consistently.
End-To-End Bias Mitigation: Removing Gender Bias in Deep Learning
Venue: arXiv cs.LG 2021
Year: 2021

Machine Learning models have been deployed across many different aspects of society, often in situations that affect social welfare. Although these models offer streamlined solutions to large problems, they may contain biases and treat groups or individuals unfairly based on protected attributes such as gender. In this paper, we introduce several examples of machine learning gender bias in practice followed by formalizations of fairness. We provide a survey of fairness research by detailing influential pre-processing, in-processing, and post-processing bias mitigation algorithms. We then propose an end-to-end bias mitigation framework, which employs a fusion of pre-, in-, and post-processing methods to leverage the strengths of each individual technique. We test this method, along with the standard techniques we review, on a deep neural network to analyze bias mitigation in a deep learning setting. We find that our end-to-end bias mitigation framework outperforms the baselines with respect to several fairness metrics, suggesting its promise as a method for improving fairness. As society increasingly relies on artificial intelligence to help in decision-making, addressing gender biases present in deep learning models is imperative. To provide readers with the tools to assess the fairness of machine learning models and mitigate the biases present in them, we discuss multiple open source packages for fairness in AI.
DCANet: Dense Context-Aware Network for Semantic Segmentation
Venue: arXiv cs.CV 2021
Year: 2021

As the superiority of context information gradually manifests in advanced semantic segmentation, learning to capture the compact context relationship can help to understand the complex scenes. In contrast to some previous works utilizing the multi-scale context fusion, we propose a novel module, named Dense Context-Aware (DCA) module, to adaptively integrate local detail information with global dependencies. Driven by the contextual relationship, the DCA module can better achieve the aggregation of context information to generate more powerful features. Furthermore, we deliberately design two extended structures based on the DCA modules to further capture the long-range contextual dependency information. By combining the DCA modules in cascade or parallel, our networks use a progressive strategy to improve multi-scale feature representations for robust segmentation. We empirically demonstrate the promising performance of our approach (DCANet) with extensive experiments on three challenging datasets, including PASCAL VOC 2012, Cityscapes, and ADE20K.
Visual Camera Re-Localization Using Graph Neural Networks and Relative Pose Supervision
Venue: arXiv cs.CV 2021
Year: 2021

Visual re-localization means using a single image as input to estimate the camera's location and orientation relative to a pre-recorded environment. The highest-scoring methods are "structure based," and need the query camera's intrinsics as an input to the model, with careful geometric optimization. When intrinsics are absent, methods vie for accuracy by making various other assumptions. This yields fairly good localization scores, but the models are "narrow" in some way, eg., requiring costly test-time computations, or depth sensors, or multiple query frames. In contrast, our proposed method makes few special assumptions, and is fairly lightweight in training and testing. Our pose regression network learns from only relative poses of training scenes. For inference, it builds a graph connecting the query image to training counterparts and uses a graph neural network (GNN) with image representations on nodes and image-pair representations on edges. By efficiently passing messages between them, both representation types are refined to produce a consistent camera pose estimate. We validate the effectiveness of our approach on both standard indoor (7-Scenes) and outdoor (Cambridge Landmarks) camera re-localization benchmarks. Our relative pose regression method matches the accuracy of absolute pose regression networks, while retaining the relative-pose models' test-time speed and ability to generalize to non-training scenes.
The Tangent Search Algorithm for Solving Optimization Problems
Venue: arXiv cs.NE 2021
Year: 2021

This article proposes a new population-based optimization algorithm called the Tangent Search Algorithm (TSA) to solve optimization problems. The TSA uses a mathematical model based on the tangent function to move a given solution toward a better solution. The tangent flight function has the advantage to balance between the exploitation and the exploration search. Moreover, a novel escape procedure is used to avoid to be trapped in local minima. Besides, an adaptive variable step size is also integrated in this algorithm to enhance the convergence capacity. The performance of TSA is assessed in three classes of tests: classical tests, CEC benchmarks, and engineering optimization problems. Moreover, several studies and metrics have been used to observe the behavior of the proposed TSA. The experimental results show that TSA algorithm is capable to provide very promising and competitive results on most benchmark functions thanks to better balance between exploration and exploitation of the search space. The main characteristics of this new optimization algorithm is its simplicity and efficiency and it requires only a small number of user-defined parameters.
Global Software Engineering in the Age of GitHub and Zoom
Venue: arXiv cs.SE 2021
Year: 2021

Much has changed since the inaugural ICGSE conference in 2006. Tools have improved, awareness of cultural differences is widespread, and developments such as the foregrounding of open source have all enhanced our ability to work across geographic divides. But the pervasive and profound impact of software in the world -- especially for societal scale systems such as social media -- forces new and deeply challenging responsibilities on both developers and academics. We must find better ways of incorporating ethics into our development practices and pay far more attention to harmful unintended consequences as deployed systems interact with and often disrupt crucial social systems.
Weakly-supervised Audio-visual Sound Source Detection and Separation
Venue: arXiv cs.CV 2021
Year: 2021

Learning how to localize and separate individual object sounds in the audio channel of the video is a difficult task. Current state-of-the-art methods predict audio masks from artificially mixed spectrograms, known as Mix-and-Separate framework. We propose an audio-visual co-segmentation, where the network learns both what individual objects look and sound like, from videos labeled with only object labels. Unlike other recent visually-guided audio source separation frameworks, our architecture can be learned in an end-to-end manner and requires no additional supervision or bounding box proposals. Specifically, we introduce weakly-supervised object segmentation in the context of sound separation. We also formulate spectrogram mask prediction using a set of learned mask bases, which combine using coefficients conditioned on the output of object segmentation , a design that facilitates separation. Extensive experiments on the MUSIC dataset show that our proposed approach outperforms state-of-the-art methods on visually guided sound source separation and sound denoising.
ASTANA: Practical String Deobfuscation for Android Applications Using Program Slicing
Venue: arXiv cs.CR 2021
Year: 2021

Software obfuscation is widely used by Android developers to protect the source code of their applications against adversarial reverse-engineering efforts. A specific type of obfuscation, string obfuscation, transforms the content of all string literals in the source code to non-interpretable text and inserts logic to deobfuscate these string literals at runtime. In this work, we demonstrate that string obfuscation is easily reversible. We present ASTANA, a practical tool for Android applications to recovers the human-readable content from obfuscated string literals. ASTANA makes minimal assumptions about the obfuscation logic or application structure. The key idea is to execute the deobfuscation logic for a specific (obfuscated) string literal, which yields the original string value. To obtain the relevant deobfuscation logic, we present a lightweight and optimistic algorithm, based on program slicing techniques. By an experimental evaluation with 100 popular real-world financial applications, we demonstrate the practicality of ASTANA. We verify the correctness of our deobfuscation tool and provide insights in the behaviour of string obfuscators applied by the developers of the evaluated Android applications.
Local Metrics for Multi-Object Tracking
Venue: arXiv cs.CV 2021
Year: 2021

This paper introduces temporally local metrics for Multi-Object Tracking. These metrics are obtained by restricting existing metrics based on track matching to a finite temporal horizon, and provide new insight into the ability of trackers to maintain identity over time. Moreover, the horizon parameter offers a novel, meaningful mechanism by which to define the relative importance of detection and association, a common dilemma in applications where imperfect association is tolerable. It is shown that the historical Average Tracking Accuracy (ATA) metric exhibits superior sensitivity to association, enabling its proposed local variant, ALTA, to capture a wide range of characteristics. In particular, ALTA is better equipped to identify advances in association independent of detection. The paper further presents an error decomposition for ATA that reveals the impact of four distinct error types and is equally applicable to ALTA. The diagnostic capabilities of ALTA are demonstrated on the MOT 2017 and Waymo Open Dataset benchmarks.