OpenRaiser/Intern-Atlas huggingface dataset:
Citations are not opinions: a corpus linguistics approach to understanding how citations are made
Venue: arXiv cs.DL 2021
Year: 2021

Citation content analysis seeks to understand citations based on the language used during the making of a citation. A key issue in citation content analysis is looking for linguistic structures that characterize distinct classes of citations for the purposes of understanding the intent and function of a citation. Previous works have focused on modeling linguistic features first and drawn conclusions on the language structures unique to each class of citation function based on the performance of a classification task or inter-annotator agreement. In this study, we start with a large sample of a pre-classified citation corpus, 2 million citations from each class of the scite Smart Citation dataset (supporting, disputing, and mentioning citations), and analyze its corpus linguistics in order to reveal the unique and statistically significant language structures belonging to each type of citation. By generating comparison tables for each citation type we present a number of interesting linguistic features that uniquely characterize citation type. What we find is that within citation collocates, there is very low correlation between citation type and sentiment. Additionally, we find that the subjectivity of citation collocates across classes is very low. These findings suggest that the sentiment of collocates is not a predictor of citation function and that due to their low subjectivity, an opinion-expressing mode of understanding citations, implicit in previous citation sentiment analysis literature, is inappropriate. Instead, we suggest that citations can be better understood as claims-making devices where the citation type can be explained by understanding how two claims are being compared. By presenting this approach, we hope to inspire similar corpus linguistic studies on citations that derive a more robust theory of citation from an empirical basis using citation corpora
Shared-Memory n-level Hypergraph Partitioning
Venue: arXiv cs.DS 2021
Year: 2021

We present a shared-memory algorithm to compute high-quality solutions to the balanced $k$-way hypergraph partitioning problem. This problem asks for a partition of the vertex set into $k$ disjoint blocks of bounded size that minimizes the connectivity metric (i.e., the sum of the number of different blocks connected by each hyperedge). High solution quality is achieved by parallelizing the core technique of the currently best sequential partitioner KaHyPar: the most extreme $n$-level version of the widely used multilevel paradigm, where only a single vertex is contracted on each level. This approach is made fast and scalable through intrusive algorithms and data structures that allow precise control of parallelism through atomic operations and fine-grained locking. We perform extensive experiments on more than 500 real-world hypergraphs with up to $140$ million vertices and two billion pins (sum of hyperedge sizes). We find that our algorithm computes solutions that are on par with a comparable configuration of KaHyPar while being an order of magnitude faster on average. Moreover, we show that recent non-multilevel algorithms specifically designed to partition large instances have considerable quality penalties and no clear advantage in running time.
Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media
Venue: arXiv cs.CL 2021
Year: 2021

Language use differs between domains and even within a domain, language use changes over time. For pre-trained language models like BERT, domain adaptation through continued pre-training has been shown to improve performance on in-domain downstream tasks. In this article, we investigate whether temporal adaptation can bring additional benefits. For this purpose, we introduce a corpus of social media comments sampled over three years. It contains unlabelled data for adaptation and evaluation on an upstream masked language modelling task as well as labelled data for fine-tuning and evaluation on a downstream document classification task. We find that temporality matters for both tasks: temporal adaptation improves upstream and temporal fine-tuning downstream task performance. Time-specific models generally perform better on past than on future test sets, which matches evidence on the bursty usage of topical words. However, adapting BERT to time and domain does not improve performance on the downstream task over only adapting to domain. Token-level analysis shows that temporal adaptation captures event-driven changes in language use in the downstream task, but not those changes that are actually relevant to task performance. Based on our findings, we discuss when temporal adaptation may be more effective.
Benchmarking Machine Learning Techniques for THz Channel Estimation Problems
Venue: arXiv cs.NI 2021
Year: 2021

Terahertz communication is one of the most promising wireless communication technologies for 6G generation and beyond. For THz systems to be practically adopted, channel estimation is one of the key issues. We consider the problem of channel modeling and estimation with deterministic channel propagation and the related physical characteristics of THz bands, and benchmark various machine learning algorithms to estimate THz channel, including neural networks (NN), logistic regression (LR), and projected gradient ascent (PGA). Numerical results show that PGA algorithm yields the most promising performance at SNR=0 dB with NMSE of -12.8 dB.
A simple and robust Abaqus implementation of the phase field fracture method
Venue: arXiv cs.CE 2021
Year: 2021

The phase field fracture method is attracting significant interest. Phase field approaches have enabled predicting - on arbitrary geometries and dimensions - complex fracture phenomena such as crack branching, coalescence, deflection and nucleation. In this work, we present a simple and robust implementation of the phase field fracture method in the commercial finite element package Abaqus. The implementation exploits the analogy between the phase field evolution law and the heat transfer equation, enabling the use of Abaqus' in-built features and circumventing the need for defining user elements. The framework is general, and is shown to accommodate different solution schemes (staggered and monolithic), as well as various constitutive choices for preventing damage under compression. The robustness and applicability of the numerical framework presented is demonstrated by addressing several 2D and 3D boundary value problems of particular interest. Focus is on the solution of paradigmatic case studies that are known to be particularly demanding from a convergence perspective. The results reveal that our phase field fracture implementation can be readily combined with other advanced computational features, such as contact, and deliver robust and precise solutions. The code developed can be downloaded from www.empaneda.com/codes.
Uncertainty Surrogates for Deep Learning
Venue: arXiv cs.LG 2021
Year: 2021

In this paper we introduce a novel way of estimating prediction uncertainty in deep networks through the use of uncertainty surrogates. These surrogates are features of the penultimate layer of a deep network that are forced to match predefined patterns. The patterns themselves can be, among other possibilities, a known visual symbol. We show how our approach can be used for estimating uncertainty in prediction and out-of-distribution detection. Additionally, the surrogates allow for interpretability of the ability of the deep network to learn and at the same time lend robustness against adversarial attacks. Despite its simplicity, our approach is superior to the state-of-the-art approaches on standard metrics as well as computational efficiency and ease of implementation. A wide range of experiments are performed on standard datasets to prove the efficacy of our approach.
Word2rate: training and evaluating multiple word embeddings as statistical transitions
Venue: arXiv cs.CL 2021
Year: 2021

Using pretrained word embeddings has been shown to be a very effective way in improving the performance of natural language processing tasks. In fact almost any natural language tasks that can be thought of has been improved by these pretrained embeddings. These tasks range from sentiment analysis, translation, sequence prediction amongst many others. One of the most successful word embeddings is the Word2vec CBOW model proposed by Mikolov trained by the negative sampling technique. Mai et al. modifies this objective to train CMOW embeddings that are sensitive to word order. We used a modified version of the negative sampling objective for our context words, modelling the context embeddings as a Taylor series of rate matrices. We show that different modes of the Taylor series produce different types of embeddings. We compare these embeddings to their similar counterparts like CBOW and CMOW and show that they achieve comparable performance. We also introduce a novel left-right context split objective that improves performance for tasks sensitive to word order. Our Word2rate model is grounded in a statistical foundation using rate matrices while being competitive in variety of language tasks.
Why Machine Learning Integrated Patient Flow Simulation?
Venue: arXiv cs.LG 2021
Year: 2021

Patient flow analysis can be studied from a clinical and or operational perspective using simulation. Traditional statistical methods such as stochastic distribution methods have been used to construct patient flow simulation submodels such as patient inflow, Length of Stay (LoS), Cost of Treatment (CoT) and Clinical Pathway (CP) models. However, patient inflow demonstrates seasonality, trend and variation over time. LoS, CoT and CP are significantly determined by attributes of patients and clinical and laboratory test results. For this reason, patient flow simulation models constructed using traditional statistical methods are criticized for ignoring heterogeneity and their contribution to personalized and value based healthcare. On the other hand, machine learning methods have proven to be efficient to study and predict admission rate, LoS, CoT, and CP. This paper, hence, describes why coupling machine learning with patient flow simulation is important and proposes a conceptual architecture that shows how to integrate machine learning with patient flow simulation.
Flexible Instance-Specific Rationalization of NLP Models
Venue: arXiv cs.CL 2021
Year: 2021

Recent research on model interpretability in natural language processing extensively uses feature scoring methods for identifying which parts of the input are the most important for a model to make a prediction (i.e. explanation or rationale). However, previous research has shown that there is no clear best scoring method across various text classification tasks while practitioners typically have to make several other ad-hoc choices regarding the length and the type of the rationale (e.g. short or long, contiguous or not). Inspired by this, we propose a simple yet effective and flexible method that allows selecting optimally for each data instance: (1) a feature scoring method; (2) the length; and (3) the type of the rationale. Our method is inspired by input erasure approaches to interpretability which assume that the most faithful rationale for a prediction should be the one with the highest difference between the model's output distribution using the full text and the text after removing the rationale as input respectively. Evaluation on four standard text classification datasets shows that our proposed method provides more faithful, comprehensive and highly sufficient explanations compared to using a fixed feature scoring method, rationale length and type. More importantly, we demonstrate that a practitioner is not required to make any ad-hoc choices in order to extract faithful rationales using our approach.
ALF -- A Fitness-Based Artificial Life Form for Evolving Large-Scale Neural Networks
Venue: arXiv cs.NE 2021
Year: 2021

Machine Learning (ML) is becoming increasingly important in daily life. In this context, Artificial Neural Networks (ANNs) are a popular approach within ML methods to realize an artificial intelligence. Usually, the topology of ANNs is predetermined. However, there are problems where it is difficult to find a suitable topology. Therefore, Topology and Weight Evolving Artificial Neural Network (TWEANN) algorithms have been developed that can find ANN topologies and weights using genetic algorithms. A well-known downside for large-scale problems is that TWEANN algorithms often evolve inefficient ANNs and require long runtimes. To address this issue, we propose a new TWEANN algorithm called Artificial Life Form (ALF) with the following technical advancements: (1) speciation via structural and semantic similarity to form better candidate solutions, (2) dynamic adaptation of the observed candidate solutions for better convergence properties, and (3) integration of solution quality into genetic reproduction to increase the probability of optimization success. Experiments on large-scale ML problems confirm that these approaches allow the fast solving of these problems and lead to efficient evolved ANNs.
Modeling Fuzzy Cluster Transitions for Topic Tracing
Venue: arXiv cs.CL 2021
Year: 2021

Twitter can be viewed as a data source for Natural Language Processing (NLP) tasks. The continuously updating data streams on Twitter make it challenging to trace real-time topic evolution. In this paper, we propose a framework for modeling fuzzy transitions of topic clusters. We extend our previous work on crisp cluster transitions by incorporating fuzzy logic in order to enrich the underlying structures identified by the framework. We apply the methodology to both computer generated clusters of nouns from tweets and human tweet annotations. The obtained fuzzy transitions are compared with the crisp transitions, on both computer generated clusters and human labeled topic sets.
Data Augmentation for Voice-Assistant NLU using BERT-based Interchangeable Rephrase
Venue: arXiv cs.CL 2021
Year: 2021

We introduce a data augmentation technique based on byte pair encoding and a BERT-like self-attention model to boost performance on spoken language understanding tasks. We compare and evaluate this method with a range of augmentation techniques encompassing generative models such as VAEs and performance-boosting techniques such as synonym replacement and back-translation. We show our method performs strongly on domain and intent classification tasks for a voice assistant and in a user-study focused on utterance naturalness and semantic similarity.
An Analysis of a BERT Deep Learning Strategy on a Technology Assisted Review Task
Venue: arXiv cs.IR 2021
Year: 2021

Document screening is a central task within Evidenced Based Medicine, which is a clinical discipline that supplements scientific proof to back medical decisions. Given the recent advances in DL (Deep Learning) methods applied to Information Retrieval tasks, I propose a DL document classification approach with BERT or PubMedBERT embeddings and a DL similarity search path using SBERT embeddings to reduce physicians' tasks of screening and classifying immense amounts of documents to answer clinical queries. I test and evaluate the retrieval effectiveness of my DL strategy on the 2017 and 2018 CLEF eHealth collections. I find that the proposed DL strategy works, I compare it to the recently successful BM25 plus RM3 model, and conclude that the suggested method accomplishes advanced retrieval performance in the initial ranking of the articles with the aforementioned datasets, for the CLEF eHealth Technologically Assisted Reviews in Empirical Medicine Task.
Motion Prediction Performance Analysis for Autonomous Driving Systems and the Effects of Tracking Noise
Venue: arXiv cs.CV 2021
Year: 2021

Autonomous driving consists of a multitude of interacting modules, where each module must contend with errors from the others. Typically, the motion prediction module depends upon a robust tracking system to capture each agent's past movement. In this work, we systematically explore the importance of the tracking module for the motion prediction task and ultimately conclude that the overall motion prediction performance is highly sensitive to the tracking module's imperfections. We explicitly compare models that use tracking information to models that do not across multiple scenarios and conditions. We find that the tracking information plays an essential role and improves motion prediction performance in noise-free conditions. However, in the presence of tracking noise, it can potentially affect the overall performance if not studied thoroughly. We thus argue practitioners should be mindful of noise when developing and testing motion/tracking modules, or that they should consider tracking free alternatives.
Accelerating Sparse Deep Neural Networks
Venue: arXiv cs.LG 2021
Year: 2021

As neural network model sizes have dramatically increased, so has the interest in various techniques to reduce their parameter counts and accelerate their execution. An active area of research in this field is sparsity - encouraging zero values in parameters that can then be discarded from storage or computations. While most research focuses on high levels of sparsity, there are challenges in universally maintaining model accuracy as well as achieving significant speedups over modern matrix-math hardware. To make sparsity adoption practical, the NVIDIA Ampere GPU architecture introduces sparsity support in its matrix-math units, Tensor Cores. We present the design and behavior of Sparse Tensor Cores, which exploit a 2:4 (50%) sparsity pattern that leads to twice the math throughput of dense matrix units. We also describe a simple workflow for training networks that both satisfy 2:4 sparsity pattern requirements and maintain accuracy, verifying it on a wide range of common tasks and model architectures. This workflow makes it easy to prepare accurate models for efficient deployment on Sparse Tensor Cores.
Unsupervised Extractive Summarization by Human Memory Simulation
Venue: arXiv cs.CL 2021
Year: 2021

Summarization systems face the core challenge of identifying and selecting important information. In this paper, we tackle the problem of content selection in unsupervised extractive summarization of long, structured documents. We introduce a wide range of heuristics that leverage cognitive representations of content units and how these are retained or forgotten in human memory. We find that properties of these representations of human memory can be exploited to capture relevance of content units in scientific articles. Experiments show that our proposed heuristics are effective at leveraging cognitive structures and the organization of the document (i.e.\ sections of an article), and automatic and human evaluations provide strong evidence that these heuristics extract more summary-worthy content units.
Yelp Dataset Analysis using Scalable Big Data
Venue: arXiv cs.DC 2021
Year: 2021

Yelp has served and will continue to serve as a data-driven application. Yelp has published a dataset containing business information, reviews, user information, and check-in information. This paper will examine this dataset to provide descriptive analytics to understand business performance, geo-spatial distribution of businesses, reviewers' rating and other characteristics, and temporal distribution of check-ins in business premises. With these analysis we are able to establish that yelp reviews, tips, elite users and check ins have started to plummet over the years. Coincidentally, the paper also establishes that Canadians have a more stable star ratings as well as sentiment ratings when compared to Americans.
Architectural Archipelagos: Technical Debt in Long-Lived Software Research Platforms
Venue: arXiv cs.SE 2021
Year: 2021

This paper identifies a model of software evolution that is prevalent in large, long-lived academic research tool suites (3L-ARTS). This model results in an "archipelago" of related but haphazardly organized architectural "islands", and inherently induces technical debt. We illustrate the archipelago model with examples from two 3L-ARTS archipelagos identified in literature.
Learning on a Budget via Teacher Imitation
Venue: arXiv cs.LG 2021
Year: 2021

Deep Reinforcement Learning (RL) techniques can benefit greatly from leveraging prior experience, which can be either self-generated or acquired from other entities. Action advising is a framework that provides a flexible way to transfer such knowledge in the form of actions between teacher-student peers. However, due to the realistic concerns, the number of these interactions is limited with a budget; therefore, it is crucial to perform these in the most appropriate moments. There have been several promising studies recently that address this problem setting especially from the student's perspective. Despite their success, they have some shortcomings when it comes to the practical applicability and integrity as an overall solution to the learning from advice challenge. In this paper, we extend the idea of advice reusing via teacher imitation to construct a unified approach that addresses both advice collection and advice utilisation problems. We also propose a method to automatically tune the relevant hyperparameters of these components on-the-fly to make it able to adapt to any task with minimal human intervention. The experiments we performed in 5 different Atari games verify that our algorithm either surpasses or performs on-par with its top competitors while being far simpler to be employed. Furthermore, its individual components are also found to be providing significant advantages alone.
Three-level Hierarchical Transformer Networks for Long-sequence and Multiple Clinical Documents Classification
Venue: arXiv cs.CL 2021
Year: 2021

We present a Three-level Hierarchical Transformer Network (3-level-HTN) for modeling long-term dependencies across clinical notes for the purpose of patient-level prediction. The network is equipped with three levels of Transformer-based encoders to learn progressively from words to sentences, sentences to notes, and finally notes to patients. The first level from word to sentence directly applies a pre-trained BERT model as a fully trainable component. While the second and third levels both implement a stack of transformer-based encoders, before the final patient representation is fed into a classification layer for clinical predictions. Compared to conventional BERT models, our model increases the maximum input length from 512 tokens to much longer sequences that are appropriate for modeling large numbers of clinical notes. We empirically examine different hyper-parameters to identify an optimal trade-off given computational resource limits. Our experiment results on the MIMIC-III dataset for different prediction tasks demonstrate that the proposed Hierarchical Transformer Network outperforms previous state-of-the-art models, including but not limited to BigBird.