OpenRaiser/Intern-Atlas huggingface dataset:
Improving low-resource ASR performance with untranscribed out-of-domain data
Venue: arXiv cs.CL 2021
Year: 2021

Semi-supervised training (SST) is a common approach to leverage untranscribed/unlabeled speech data to improve automatic speech recognition performance in low-resource languages. However, if the available unlabeled speech is mismatched to the target domain, SST is not as effective, and in many cases performs worse than the original system. In this paper, we address the issue of low-resource ASR when only untranscribed out-of-domain speech data is readily available in the target language. Specifically, we look to improve performance on conversational/telephony speech (target domain) using web resources, in particular YouTube data, which more closely resembles news/topical broadcast data. Leveraging SST, we show that while in some cases simply pooling the out-of-domain data with the training data lowers word error rate (WER), in all cases, we see improvements if we train first with the out-of-domain data and then fine-tune the resulting model with the original training data. Using 2000 hours of speed perturbed YouTube audio in each target language, with semi-supervised transcripts, we show improvements on multiple languages/data sets, of up to 16.3% relative improvement in WER over the baseline systems and up to 7.4% relative improvement in WER over a system that simply pools the out-of-domain data with the training data.
A Privacy-Preserving and Trustable Multi-agent Learning Framework
Venue: arXiv cs.LG 2021
Year: 2021

Distributed multi-agent learning enables agents to cooperatively train a model without requiring to share their datasets. While this setting ensures some level of privacy, it has been shown that, even when data is not directly shared, the training process is vulnerable to privacy attacks including data reconstruction and model inversion attacks. Additionally, malicious agents that train on inverted labels or random data, may arbitrarily weaken the accuracy of the global model. This paper addresses these challenges and presents Privacy-preserving and trustable Distributed Learning (PT-DL), a fully decentralized framework that relies on Differential Privacy to guarantee strong privacy protections of the agents' data, and Ethereum smart contracts to ensure trustability. The paper shows that PT-DL is resilient up to a 50% collusion attack, with high probability, in a malicious trust model and the experimental evaluation illustrates the benefits of the proposed model as a privacy-preserving and trustable distributed multi-agent learning system on several classification tasks.
Assessing the Reliability of Deep Learning Classifiers Through Robustness Evaluation and Operational Profiles
Venue: arXiv cs.LG 2021
Year: 2021

The utilisation of Deep Learning (DL) is advancing into increasingly more sophisticated applications. While it shows great potential to provide transformational capabilities, DL also raises new challenges regarding its reliability in critical functions. In this paper, we present a model-agnostic reliability assessment method for DL classifiers, based on evidence from robustness evaluation and the operational profile (OP) of a given application. We partition the input space into small cells and then "assemble" their robustness (to the ground truth) according to the OP, where estimators on the cells' robustness and OPs are provided. Reliability estimates in terms of the probability of misclassification per input (pmi) can be derived together with confidence levels. A prototype tool is demonstrated with simplified case studies. Model assumptions and extension to real-world applications are also discussed. While our model easily uncovers the inherent difficulties of assessing the DL dependability (e.g. lack of data with ground truth and scalability issues), we provide preliminary/compromised solutions to advance in this research direction.
On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers
Venue: arXiv cs.CL 2021
Year: 2021

How much information do NLP tasks really need from a transformer's attention mechanism at application-time (inference)? From recent work, we know that there is sparsity in transformers and that the floating-points within its computation can be discretized to fewer values with minimal loss to task accuracies. However, this requires retraining or even creating entirely new models, both of which can be expensive and carbon-emitting. Focused on optimizations that do not require training, we systematically study the full range of typical attention values necessary. This informs the design of an inference-time quantization technique using both pruning and log-scaled mapping which produces only a few (e.g. $2^3$) unique values. Over the tasks of question answering and sentiment analysis, we find nearly 80% of attention values can be pruned to zeros with minimal ($< 1.0\%$) relative loss in accuracy. We use this pruning technique in conjunction with quantizing the attention values to only a 3-bit format, without retraining, resulting in only a 0.8% accuracy reduction on question answering with fine-tuned RoBERTa.
A Leader-Follower Game Theoretic Approach to Arrest Cascading Failure in Smart Grid
Venue: arXiv cs.GT 2021
Year: 2021

The Smart Grid System (SGS) is a joint network comprising the power and the communication network. In this paper, the underlying intra-and-interdependencies between entities for a given SGS is captured using a dependency model called Modified Implicative Interdependency Model (MIIM) [1]. Given an integer K, the K-contingency list problem gives the list of K-most critical entities, failure of which maximizes the network damage at the current time. The problem being NP complete [2] and owing to the higher running time of the given Integer Linear Programming (ILP) based solution [3], a much faster heuristic solution to generate an event driven self-updating K-contingency list [4] is also given in this paper. Based on the contingency lists obtained from both the solutions, this paper proposes an adaptive entity hardening technique based on a leader-follower game theoretic approach that arrests the cascading failure of entities in the SGS after an initial failure of entities. The validation of the work is done by comparing the contingency lists using both types of solutions, obtained for different K values using the MIIM model on a smart grid of IEEE 14-Bus system with that obtained by simulating the smart grid using a co-simulation system formed by MATPOWER and Java Network Simulator (JNS). The K-contingency list obtained for a smart grid of IEEE 14-Bus system also indicate that the network damage predicted by both the ILP based solution and heuristic solution using MIIM are more realistic compared to that obtained using another dependency model called Implicative Interdependency Model (IIM) [2]. Advantage of using the MIIM based heuristic solution is also shown in this paper when larger SGS of IEEE 118-Bus is considered. Finally, it is shown how the adaptive hardening helps in improving the network performance.
Evaluating the Efficacy of Summarization Evaluation across Languages
Venue: arXiv cs.CL 2021
Year: 2021

While automatic summarization evaluation methods developed for English are routinely applied to other languages, this is the first attempt to systematically quantify their panlinguistic efficacy. We take a summarization corpus for eight different languages, and manually annotate generated summaries for focus (precision) and coverage (recall). Based on this, we evaluate 19 summarization evaluation metrics, and find that using multilingual BERT within BERTScore performs well across all languages, at a level above that for English.
Dagger: Accelerating RPCs in Cloud Microservices Through Tightly-Coupled Reconfigurable NICs
Venue: arXiv cs.AR 2021
Year: 2021

The ongoing shift of cloud services from monolithic designs to microservices creates high demand for efficient and high performance datacenter networking stacks, optimized for fine-grained workloads. Commodity networking systems based on software stacks and peripheral NICs introduce high overheads when it comes to delivering small messages. We present Dagger, a hardware acceleration fabric for cloud RPCs based on FPGAs, where the accelerator is closely-coupled with the host processor over a configurable memory interconnect. The three key design principle of Dagger are: (1) offloading the entire RPC stack to an FPGA-based NIC, (2) leveraging memory interconnects instead of PCIe buses as the interface with the host CPU, and (3) making the acceleration fabric reconfigurable, so it can accommodate the diverse needs of microservices. We show that the combination of these principles significantly improves the efficiency and performance of cloud RPC systems while preserving their generality. Dagger achieves 1.3-3.8x higher per-core RPC throughput compared to both highly-optimized software stacks, and systems using specialized RDMA adapters. It also scales up to 84 Mrps with 8 threads on 4 CPU cores, while maintaining state-of-the-art us-scale tail latency. We also demonstrate that large third-party applications, like memcached and MICA KVS, can be easily ported on Dagger with minimal changes to their codebase, bringing their median and tail KVS access latency down to 2.8 - 3.5us and 5.4 - 7.8us, respectively. Finally, we show that Dagger is beneficial for multi-tier end-to-end microservices with different threading models by evaluating it using an 8-tier application implementing a flight check-in service.
What and How long: Prediction of Mobile App Engagement
Venue: arXiv cs.IR 2021
Year: 2021

User engagement is crucial to the long-term success of a mobile app. Several metrics, such as dwell time, have been used for measuring user engagement. However, how to effectively predict user engagement in the context of mobile apps is still an open research question. For example, do the mobile usage contexts (e.g.,~time of day) in which users access mobile apps impact their dwell time? Answers to such questions could help mobile operating system and publishers to optimize advertising and service placement. In this paper, we first conduct an empirical study for assessing how user characteristics, temporal features, and the short/long-term contexts contribute to gains in predicting users' app dwell time on the population level. The comprehensive analysis is conducted on large app usage logs collected through a mobile advertising company. The dataset covers more than 12K anonymous users and 1.3 million log events. Based on the analysis, we further investigate a novel mobile app engagement prediction problem -- can we predict simultaneously what app the user will use next and how long he/she will stay on that app? We propose several strategies for this joint prediction problem and demonstrate that our model can improve the performance significantly when compared with the state-of-the-art baselines. Our work can help mobile system developers in designing a better and more engagement-aware mobile app user experience.
MedNLI Is Not Immune: Natural Language Inference Artifacts in the Clinical Domain
Venue: arXiv cs.CL 2021
Year: 2021

Crowdworker-constructed natural language inference (NLI) datasets have been found to contain statistical artifacts associated with the annotation process that allow hypothesis-only classifiers to achieve better-than-random performance (Poliak et al., 2018; Gururanganet et al., 2018; Tsuchiya, 2018). We investigate whether MedNLI, a physician-annotated dataset with premises extracted from clinical notes, contains such artifacts (Romanov and Shivade, 2018). We find that entailed hypotheses contain generic versions of specific concepts in the premise, as well as modifiers related to responsiveness, duration, and probability. Neutral hypotheses feature conditions and behaviors that co-occur with, or cause, the condition(s) in the premise. Contradiction hypotheses feature explicit negation of the premise and implicit negation via assertion of good health. Adversarial filtering demonstrates that performance degrades when evaluated on the difficult subset. We provide partition information and recommendations for alternative dataset construction strategies for knowledge-intensive domains.
Adjacency List Oriented Relational Fact Extraction via Adaptive Multi-task Learning
Venue: arXiv cs.CL 2021
Year: 2021

Relational fact extraction aims to extract semantic triplets from unstructured text. In this work, we show that all of the relational fact extraction models can be organized according to a graph-oriented analytical perspective. An efficient model, aDjacency lIst oRiented rElational faCT (DIRECT), is proposed based on this analytical framework. To alleviate challenges of error propagation and sub-task loss equilibrium, DIRECT employs a novel adaptive multi-task learning strategy with dynamic sub-task loss balancing. Extensive experiments are conducted on two benchmark datasets, and results prove that the proposed model outperforms a series of state-of-the-art (SoTA) models for relational triplet extraction.
Discriminative Reasoning for Document-level Relation Extraction
Venue: arXiv cs.CL 2021
Year: 2021

Document-level relation extraction (DocRE) models generally use graph networks to implicitly model the reasoning skill (i.e., pattern recognition, logical reasoning, coreference reasoning, etc.) related to the relation between one entity pair in a document. In this paper, we propose a novel discriminative reasoning framework to explicitly model the paths of these reasoning skills between each entity pair in this document. Thus, a discriminative reasoning network is designed to estimate the relation probability distribution of different reasoning paths based on the constructed graph and vectorized document contexts for each entity pair, thereby recognizing their relation. Experimental results show that our method outperforms the previous state-of-the-art performance on the large-scale DocRE dataset. The code is publicly available at https://github.com/xwjim/DRN.
Irregularly Clipped Sparse Regression Codes
Venue: arXiv cs.IT 2021
Year: 2021

Recently, it was found that clipping can significantly improve the section error rate (SER) performance of sparse regression (SR) codes if an optimal clipping threshold is chosen. In this paper, we propose irregularly clipped SR codes, where multiple clipping thresholds are applied to symbols according to a distribution, to further improve the SER performance of SR codes. Orthogonal approximate message passing (OAMP) algorithm is used for decoding. Using state evolution, the distribution of irregular clipping thresholds is optimized to minimize the SER of OAMP decoding. As a result, optimized irregularly clipped SR codes achieve a better tradeoff between clipping distortion and noise distortion than regularly clipped SR codes. Numerical results demonstrate that irregularly clipped SR codes achieve 0.4 dB gain in signal-to-noise-ratio (SNR) over regularly clipped SR codes at code length$\,\approx2.5\!\times\! 10^4$ and SER$\,\approx10^{-5}$. We further show that irregularly clipped SR codes are robust over a wide range of code rates.
A Provably-Efficient Model-Free Algorithm for Constrained Markov Decision Processes
Venue: arXiv cs.LG 2021
Year: 2021

This paper presents the first model-free, simulator-free reinforcement learning algorithm for Constrained Markov Decision Processes (CMDPs) with sublinear regret and zero constraint violation. The algorithm is named Triple-Q because it includes three key components: a Q-function (also called action-value function) for the cumulative reward, a Q-function for the cumulative utility for the constraint, and a virtual-Queue that (over)-estimates the cumulative constraint violation. Under Triple-Q, at each step, an action is chosen based on the pseudo-Q-value that is a combination of the three "Q" values. The algorithm updates the reward and utility Q-values with learning rates that depend on the visit counts to the corresponding (state, action) pairs and are periodically reset. In the episodic CMDP setting, Triple-Q achieves $\tilde{\cal O}\left(\frac{1 }δH^4 S^{\frac{1}{2}}A^{\frac{1}{2}}K^{\frac{4}{5}} \right)$ regret, where $K$ is the total number of episodes, $H$ is the number of steps in each episode, $S$ is the number of states, $A$ is the number of actions, and $δ$ is Slater's constant. Furthermore, Triple-Q guarantees zero constraint violation, both on expectation and with a high probability, when $K$ is sufficiently large. Finally, the computational complexity of Triple-Q is similar to SARSA for unconstrained MDPs and is computationally efficient.
The Emotion coding and Propagation based on improved Genetic algorithm
Venue: arXiv cs.CY 2021
Year: 2021

Computational communication research on information has been prevalent in recent years, as people are progressively inquisitive in social behavior and public opinion. Nevertheless, it is of great significance to analyze the direction of predominant sentiment from the sentiment communication perspective. In this paper, the information emotion propagation model is established by introducing revamp genetic algorithms into information emotion. In the process of information dissemination, both the information emotions and the network emotions are dynamic. For this model, the information emotions and the network nodes emotions are quantified as binary codes. The convergence effects, crossover and mutation algorithms are introduced. These factors all act on the transmission process via dynamic propagation rate, and the improved genetic algorithm also acts on the emotion transmission. In particular, the latter two algorithms are different from the existing biological domain. Based on the existing research results in other manuscripts, we perform simulation described above on the hybrid network. The simulation results demonstrate that the trend approximate to the actual data. As a result, our work can prove that our proposed model is essentially consistent with the actual emotion transmission phenomenon.
Semantic-Aware Contrastive Learning for Multi-object Medical Image Segmentation
Venue: arXiv cs.CV 2021
Year: 2021

Medical image segmentation, or computing voxelwise semantic masks, is a fundamental yet challenging task to compute a voxel-level semantic mask. To increase the ability of encoder-decoder neural networks to perform this task across large clinical cohorts, contrastive learning provides an opportunity to stabilize model initialization and enhance encoders without labels. However, multiple target objects (with different semantic meanings) may exist in a single image, which poses a problem for adapting traditional contrastive learning methods from prevalent 'image-level classification' to 'pixel-level segmentation'. In this paper, we propose a simple semantic-aware contrastive learning approach leveraging attention masks to advance multi-object semantic segmentation. Briefly, we embed different semantic objects to different clusters rather than the traditional image-level embeddings. We evaluate our proposed method on a multi-organ medical image segmentation task with both in-house data and MICCAI Challenge 2015 BTCV datasets. Compared with current state-of-the-art training strategies, our proposed pipeline yields a substantial improvement of 5.53% and 6.09% on Dice score for both medical image segmentation cohorts respectively (p-value<0.01). The performance of the proposed method is further assessed on natural images via the PASCAL VOC 2012 dataset, and achieves a substantial improvement of 2.75% on mIoU (p-value<0.01).
ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language Generation
Venue: arXiv cs.CL 2021
Year: 2021

Despite the recent advancement in NLP research, cross-lingual transfer for natural language generation is relatively understudied. In this work, we transfer supervision from high resource language (HRL) to multiple low-resource languages (LRLs) for natural language generation (NLG). We consider four NLG tasks (text summarization, question generation, news headline generation, and distractor generation) and three syntactically diverse languages, i.e., English, Hindi, and Japanese. We propose an unsupervised cross-lingual language generation framework (called ZmBART) that does not use any parallel or pseudo-parallel/back-translated data. In this framework, we further pre-train mBART sequence-to-sequence denoising auto-encoder model with an auxiliary task using monolingual data of three languages. The objective function of the auxiliary task is close to the target tasks which enriches the multi-lingual latent representation of mBART and provides good initialization for target tasks. Then, this model is fine-tuned with task-specific supervised English data and directly evaluated with low-resource languages in the Zero-shot setting. To overcome catastrophic forgetting and spurious correlation issues, we applied freezing model component and data argumentation approaches respectively. This simple modeling approach gave us promising results.We experimented with few-shot training (with 1000 supervised data points) which boosted the model performance further. We performed several ablations and cross-lingual transferability analyses to demonstrate the robustness of ZmBART.
Grounding Complex Navigational Instructions Using Scene Graphs
Venue: arXiv cs.LG 2021
Year: 2021

Training a reinforcement learning agent to carry out natural language instructions is limited by the available supervision, i.e. knowing when the instruction has been carried out. We adapt the CLEVR visual question answering dataset to generate complex natural language navigation instructions and accompanying scene graphs, yielding an environment-agnostic supervised dataset. To demonstrate the use of this data set, we map the scenes to the VizDoom environment and use the architecture in \citet{gatedattention} to train an agent to carry out these more complex language instructions.
Improving the Transferability of Adversarial Examples with New Iteration Framework and Input Dropout
Venue: arXiv cs.LG 2021
Year: 2021

Deep neural networks(DNNs) is vulnerable to be attacked by adversarial examples. Black-box attack is the most threatening attack. At present, black-box attack methods mainly adopt gradient-based iterative attack methods, which usually limit the relationship between the iteration step size, the number of iterations, and the maximum perturbation. In this paper, we propose a new gradient iteration framework, which redefines the relationship between the above three. Under this framework, we easily improve the attack success rate of DI-TI-MIM. In addition, we propose a gradient iterative attack method based on input dropout, which can be well combined with our framework. We further propose a multi dropout rate version of this method. Experimental results show that our best method can achieve attack success rate of 96.2\% for defense model on average, which is higher than the state-of-the-art gradient-based attacks.
Few-shot Knowledge Graph-to-Text Generation with Pretrained Language Models
Venue: arXiv cs.CL 2021
Year: 2021

This paper studies how to automatically generate a natural language text that describes the facts in knowledge graph (KG). Considering the few-shot setting, we leverage the excellent capacities of pretrained language models (PLMs) in language understanding and generation. We make three major technical contributions, namely representation alignment for bridging the semantic gap between KG encodings and PLMs, relation-biased KG linearization for deriving better input representations, and multi-task learning for learning the correspondence between KG and text. Extensive experiments on three benchmark datasets have demonstrated the effectiveness of our model on KG-to-text generation task. In particular, our model outperforms all comparison methods on both fully-supervised and few-shot settings. Our code and datasets are available at https://github.com/RUCAIBox/Few-Shot-KG2Text.