OpenRaiser/Intern-Atlas huggingface dataset:
Learning Representation over Dynamic Graph using Aggregation-Diffusion Mechanism
Venue: arXiv cs.LG 2021
Year: 2021

Representation learning on graphs that evolve has recently received significant attention due to its wide application scenarios, such as bioinformatics, knowledge graphs, and social networks. The propagation of information in graphs is important in learning dynamic graph representations, and most of the existing methods achieve this by aggregation. However, relying only on aggregation to propagate information in dynamic graphs can result in delays in information propagation and thus affect the performance of the method. To alleviate this problem, we propose an aggregation-diffusion (AD) mechanism that actively propagates information to its neighbor by diffusion after the node updates its embedding through the aggregation mechanism. In experiments on two real-world datasets in the dynamic link prediction task, the AD mechanism outperforms the baseline models that only use aggregation to propagate information. We further conduct extensive experiments to discuss the influence of different factors in the AD mechanism.
From indexation policies through citation networks to normalized citation impacts: Web of Science, Scopus, and Dimensions as varying resonance chambers
Venue: arXiv cs.DL 2021
Year: 2021

Dimensions was introduced as an alternative bibliometric database to the well-established Web of Science (WoS) and Scopus, however all three databases have fundamental differences in coverage and content, resultant from their owners' indexation philosophies. In light of these differences, we explore here, using a citation network analysis and assessment of normalised citation impact of "duplicate" publications, whether the three databases offer structurally different perspectives of the bibliometric landscape or if they are essentially homogenous substitutes. Our citation network analysis of core and exclusive 2016-2018 publications revealed a large set of core publications indexed in all three databases that are highly self-referential. In comparison, each database selected a set of exclusive publications that appeared to hold similarly low levels of relevance to the core set and to one another, with slightly more internal communication between exclusive publications in Scopus and Dimensions than WoS. Our comparison of normalised citations for 41,848 publications indexed in all three databases found that German sectors were valuated as more impactful in Scopus and Dimensions compared to WoS, particularly for sectors with an applied research focus. We conclude that the databases do present structurally different perspectives, although Scopus and Dimensions with their additional circle of applied research vary more from the more base research-focused WoS than they do from one another.
Fingerprinting Fine-tuned Language Models in the Wild
Venue: arXiv cs.CL 2021
Year: 2021

There are concerns that the ability of language models (LMs) to generate high quality synthetic text can be misused to launch spam, disinformation, or propaganda. Therefore, the research community is actively working on developing approaches to detect whether a given text is organic or synthetic. While this is a useful first step, it is important to be able to further fingerprint the author LM to attribute its origin. Prior work on fingerprinting LMs is limited to attributing synthetic text generated by a handful (usually < 10) of pre-trained LMs. However, LMs such as GPT2 are commonly fine-tuned in a myriad of ways (e.g., on a domain-specific text corpus) before being used to generate synthetic text. It is challenging to fingerprinting fine-tuned LMs because the universe of fine-tuned LMs is much larger in realistic scenarios. To address this challenge, we study the problem of large-scale fingerprinting of fine-tuned LMs in the wild. Using a real-world dataset of synthetic text generated by 108 different fine-tuned LMs, we conduct comprehensive experiments to demonstrate the limitations of existing fingerprinting approaches. Our results show that fine-tuning itself is the most effective in attributing the synthetic text generated by fine-tuned LMs.
GMAIR: Unsupervised Object Detection Based on Spatial Attention and Gaussian Mixture
Venue: arXiv cs.CV 2021
Year: 2021

Recent studies on unsupervised object detection based on spatial attention have achieved promising results. Models, such as AIR and SPAIR, output "what" and "where" latent variables that represent the attributes and locations of objects in a scene, respectively. Most of the previous studies concentrate on the "where" localization performance; however, we claim that acquiring "what" object attributes is also essential for representation learning. This paper presents a framework, GMAIR, for unsupervised object detection. It incorporates spatial attention and a Gaussian mixture in a unified deep generative model. GMAIR can locate objects in a scene and simultaneously cluster them without supervision. Furthermore, we analyze the "what" latent variables and clustering process. Finally, we evaluate our model on MultiMNIST and Fruit2D datasets and show that GMAIR achieves competitive results on localization and clustering compared to state-of-the-art methods.
Exploiting co-execution with oneAPI: heterogeneity from a modern perspective
Venue: arXiv cs.DC 2021
Year: 2021

Programming efficiently heterogeneous systems is a major challenge, due to the complexity of their architectures. Intel oneAPI, a new and powerful standards-based unified programming model, built on top of SYCL, addresses these issues. In this paper, oneAPI is provided with co-execution strategies to run the same kernel between different devices, enabling the exploitation of static and dynamic policies. On top of that, static and dynamic load-balancing algorithms are integrated and analyzed. This work evaluates the performance and energy efficiency for a well-known set of regular and irregular HPC benchmarks, using an integrated GPU and CPU. Experimental results show that co-execution is worthwhile when using dynamic algorithms, improving efficiency even more when using unified shared memory.
Auto-tagging of Short Conversational Sentences using Transformer Methods
Venue: arXiv cs.CL 2021
Year: 2021

The problem of categorizing short speech sentences according to their semantic features with high accuracy is a subject studied in natural language processing. In this study, a data set created with samples classified in 46 different categories was used. Examples consist of sentences taken from chat conversations between a company's customer representatives and the company's website visitors. The primary purpose is to automatically tag questions and requests from visitors in the most accurate way for 46 predetermined categories for use in a chat application to generate meaningful answers to the questions asked by the website visitors. For this, different BERT models and one GPT-2 model, pre-trained in Turkish, were preferred. The classification performances of the relevant models were analyzed in detail and reported accordingly.
Bayesian Classifier Fusion with an Explicit Model of Correlation
Venue: arXiv cs.LG 2021
Year: 2021

Combining the outputs of multiple classifiers or experts into a single probabilistic classification is a fundamental task in machine learning with broad applications from classifier fusion to expert opinion pooling. Here we present a hierarchical Bayesian model of probabilistic classifier fusion based on a new correlated Dirichlet distribution. This distribution explicitly models positive correlations between marginally Dirichlet-distributed random vectors thereby allowing explicit modeling of correlations between base classifiers or experts. The proposed model naturally accommodates the classic Independent Opinion Pool and other independent fusion algorithms as special cases. It is evaluated by uncertainty reduction and correctness of fusion on synthetic and real-world data sets. We show that a change in performance of the fused classifier due to uncertainty reduction can be Bayes optimal even for highly correlated base classifiers.
Topological Anomaly Detection in Dynamic Multilayer Blockchain Networks
Venue: arXiv cs.CR 2021
Year: 2021

Motivated by the recent surge of criminal activities with cross-cryptocurrency trades, we introduce a new topological perspective to structural anomaly detection in dynamic multilayer networks. We postulate that anomalies in the underlying blockchain transaction graph that are composed of multiple layers are likely to also be manifested in anomalous patterns of the network shape properties. As such, we invoke the machinery of clique persistent homology on graphs to systematically and efficiently track evolution of the network shape and, as a result, to detect changes in the underlying network topology and geometry. We develop a new persistence summary for multilayer networks, called stacked persistence diagram, and prove its stability under input data perturbations. We validate our new topological anomaly detection framework in application to dynamic multilayer networks from the Ethereum Blockchain and the Ripple Credit Network, and demonstrate that our stacked PD approach substantially outperforms state-of-art techniques.
Cloud-Enabled High-Altitude Platform Systems: Challenges and Opportunities
Venue: arXiv cs.NI 2021
Year: 2021

Augmenting ground-level communications with flying networks, such as the high-altitude platform system (HAPS), is among the major innovative initiatives of the next generation of wireless systems (6G). Given HAPS quasi-static positioning at the stratosphere, HAPS-to-ground and HAPS-to-air connectivity frameworks are expected to be prolific in terms of data acquisition and computing, especially given the mild weather and quasi-constant wind speed characteristics of the stratospheric layer. This paper explores the opportunities stemming from the realization of cloud-enabled HAPS in the context of telecommunications applications and services. The paper first advocates for the potential physical advantages of deploying HAPS as flying data-centers, also known as super-macro base stations. The paper then describes various cloud services that can be offered from the HAPS and the merits that can be achieved by this integration, such as enhancing the quality, speed, and range of the offered services. The proposed services span a wide range of fields, including satellites, Internet of Things (IoT), ad hoc networks (such as sensor; vehicular; and aerial networks), gaming, and social networks. For each service, the paper illustrates the methods that would be used by cloud providers to offload the service data to the HAPS and enable the cloud customers to consume the service. The paper further sheds light on the challenges that need to be addressed for realizing practical cloud-enabled HAPS, mainly, those related to high energy, processing power, quality of service (QoS), and security considerations. Finally, the paper discusses some open issues on the topic, namely, HAPS mobility and message routing, HAPS security via blockchain and machine learning, artificial intelligence-based resource allocation in cloud-enabled HAPS, and integration with vertical heterogeneous networks.
Breaking the Cubic Barrier for (Unweighted) Tree Edit Distance
Venue: arXiv cs.DS 2021
Year: 2021

The (unweighted) tree edit distance problem for $n$ node trees asks to compute a measure of dissimilarity between two rooted trees with node labels. The current best algorithm from more than a decade ago runs in $O(n ^ 3)$ time [Demaine, Mozes, Rossman, and Weimann, ICALP 2007]. The same paper also showed that $O(n ^ 3)$ is the best possible running time for any algorithm using the so-called decomposition strategy, which underlies almost all the known algorithms for this problem. These algorithms would also work for the weighted tree edit distance problem, which cannot be solved in truly sub-cubic time under the APSP conjecture [Bringmann, Gawrychowski, Mozes, and Weimann, SODA 2018]. In this paper, we break the cubic barrier by showing an $O(n ^ {2.9546})$ time algorithm for the unweighted tree edit distance problem. We consider an equivalent maximization problem and use a dynamic programming scheme involving matrices with many special properties. By using a decomposition scheme as well as several combinatorial techniques, we reduce tree edit distance to the max-plus product of bounded-difference matrices, which can be solved in truly sub-cubic time [Bringmann, Grandoni, Saha, and Vassilevska Williams, FOCS 2016].
Anticipative Video Transformer
Venue: arXiv cs.CV 2021
Year: 2021

We propose Anticipative Video Transformer (AVT), an end-to-end attention-based video modeling architecture that attends to the previously observed video in order to anticipate future actions. We train the model jointly to predict the next action in a video sequence, while also learning frame feature encoders that are predictive of successive future frames' features. Compared to existing temporal aggregation strategies, AVT has the advantage of both maintaining the sequential progression of observed actions while still capturing long-range dependencies--both critical for the anticipation task. Through extensive experiments, we show that AVT obtains the best reported performance on four popular action anticipation benchmarks: EpicKitchens-55, EpicKitchens-100, EGTEA Gaze+, and 50-Salads; and it wins first place in the EpicKitchens-100 CVPR'21 challenge.
Oblivious Stacking and MAX $k$-CUT for Circle Graphs
Venue: arXiv cs.DS 2021
Year: 2021

Stacking is an important process within logistics. Some notable examples of items to be stacked are steel bars or steel plates in a steel yard or containers in a container terminal or on a ship. We say that two items are conflicting if their storage time intervals overlap in which case one of the items needs to be rehandled if the items are stored at the same LIFO storage location. We consider the problem of stacking items using $k$ LIFO locations with a minimum number of conflicts between items sharing a location. We present an extremely simple online stacking algorithm that is oblivious to the storage time intervals and storage locations of all other items when it picks a storage location for an item. The risk of assigning the same storage location to two conflicting items is proved to be of the order $1/k^2$ under mild assumptions on the distribution of the storage time intervals for the items. Intuitively, it seems natural to pick a storage location uniformly at random in the oblivious setting implying a risk of $1/k$ so the risk for our algorithm is surprisingly low. Our results can also be expressed within the context of the MAX $k$-CUT problem for circle graphs. The results indicate that circle graphs on average have relatively big $k$-cuts compared to the total number of edges.
Interactive Communication in Bilateral Trade
Venue: arXiv cs.GT 2021
Year: 2021

We define a model of interactive communication where two agents with private types can exchange information before a game is played. The model contains Bayesian persuasion as a special case of a one-round communication protocol. We define message complexity corresponding to the minimum number of interactive rounds necessary to achieve the best possible outcome. Our main result is that for bilateral trade, agents don't stop talking until they reach an efficient outcome: Either agents achieve an efficient allocation in finitely many rounds of communication; or the optimal communication protocol has infinite number of rounds. We show an important class of bilateral trade settings where efficient allocation is achievable with a small number of rounds of communication.
A Unified Bi-directional Model for Natural and Artificial Trust in Human-Robot Collaboration
Venue: arXiv cs.RO 2021
Year: 2021

We introduce a novel capabilities-based bi-directional multi-task trust model that can be used for trust prediction from either a human or a robotic trustor agent. Tasks are represented in terms of their capability requirements, while trustee agents are characterized by their individual capabilities. Trustee agents' capabilities are not deterministic; they are represented by belief distributions. For each task to be executed, a higher level of trust is assigned to trustee agents who have demonstrated that their capabilities exceed the task's requirements. We report results of an online experiment with 284 participants, revealing that our model outperforms existing models for multi-task trust prediction from a human trustor. We also present simulations of the model for determining trust from a robotic trustor. Our model is useful for control authority allocation applications that involve human-robot teams.
Barcode Method for Generative Model Evaluation driven by Topological Data Analysis
Venue: arXiv cs.CV 2021
Year: 2021

Evaluating the performance of generative models in image synthesis is a challenging task. Although the Fréchet Inception Distance is a widely accepted evaluation metric, it integrates different aspects (e.g., fidelity and diversity) of synthesized images into a single score and assumes the normality of embedded vectors. Recent methods such as precision-and-recall and its variants such as density-and-coverage have been developed to separate fidelity and diversity based on k-nearest neighborhood methods. In this study, we propose an algorithm named barcode, which is inspired by the topological data analysis and is almost free of assumption and hyperparameter selections. In extensive experiments on real-world datasets as well as theoretical approach on high-dimensional normal samples, it was found that the 'usual' normality assumption of embedded vectors has several drawbacks. The experimental results demonstrate that barcode outperforms other methods in evaluating fidelity and diversity of GAN outputs. Official codes can be found in https://github.com/minjeekim00/Barcode.
History Encoding Representation Design for Human Intention Inference
Venue: arXiv cs.CV 2021
Year: 2021

In this extended abstract, we investigate the design of learning representation for human intention inference. In our designed human intention prediction task, we propose a history encoding representation that is both interpretable and effective for prediction. Through extensive experiments, we show our prediction framework with a history encoding representation design is successful on the human intention prediction problem.
A Nearly Optimal All-Pairs Min-Cuts Algorithm in Simple Graphs
Venue: arXiv cs.DS 2021
Year: 2021

We give an $n^{2+o(1)}$-time algorithm for finding $s$-$t$ min-cuts for all pairs of vertices $s$ and $t$ in a simple, undirected graph on $n$ vertices. We do so by constructing a Gomory-Hu tree (or cut equivalent tree) in the same running time, thereby improving on the recent bound of $\tilde{O}(n^{2.5})$ by Abboud et al. (STOC 2021). Our running time is nearly optimal as a function of $n$.
Discovery of Causal Additive Models in the Presence of Unobserved Variables
Venue: arXiv cs.LG 2021
Year: 2021

Causal discovery from data affected by unobserved variables is an important but difficult problem to solve. The effects that unobserved variables have on the relationships between observed variables are more complex in nonlinear cases than in linear cases. In this study, we focus on causal additive models in the presence of unobserved variables. Causal additive models exhibit structural equations that are additive in the variables and error terms. We take into account the presence of not only unobserved common causes but also unobserved intermediate variables. Our theoretical results show that, when the causal relationships are nonlinear and there are unobserved variables, it is not possible to identify all the causal relationships between observed variables through regression and independence tests. However, our theoretical results also show that it is possible to avoid incorrect inferences. We propose a method to identify all the causal relationships that are theoretically possible to identify without being biased by unobserved variables. The empirical results using artificial data and simulated functional magnetic resonance imaging (fMRI) data show that our method effectively infers causal structures in the presence of unobserved variables.
ERNIE-Tiny : A Progressive Distillation Framework for Pretrained Transformer Compression
Venue: arXiv cs.CL 2021
Year: 2021

Pretrained language models (PLMs) such as BERT adopt a training paradigm which first pretrain the model in general data and then finetune the model on task-specific data, and have recently achieved great success. However, PLMs are notorious for their enormous parameters and hard to be deployed on real-life applications. Knowledge distillation has been prevailing to address this problem by transferring knowledge from a large teacher to a much smaller student over a set of data. We argue that the selection of thee three key components, namely teacher, training data, and learning objective, is crucial to the effectiveness of distillation. We, therefore, propose a four-stage progressive distillation framework ERNIE-Tiny to compress PLM, which varies the three components gradually from general level to task-specific level. Specifically, the first stage, General Distillation, performs distillation with guidance from pretrained teacher, gerenal data and latent distillation loss. Then, General-Enhanced Distillation changes teacher model from pretrained teacher to finetuned teacher. After that, Task-Adaptive Distillation shifts training data from general data to task-specific data. In the end, Task-Specific Distillation, adds two additional losses, namely Soft-Label and Hard-Label loss onto the last stage. Empirical results demonstrate the effectiveness of our framework and generalization gain brought by ERNIE-Tiny.In particular, experiments show that a 4-layer ERNIE-Tiny maintains over 98.0%performance of its 12-layer teacher BERT base on GLUE benchmark, surpassing state-of-the-art (SOTA) by 1.0% GLUE score with the same amount of parameters. Moreover, ERNIE-Tiny achieves a new compression SOTA on five Chinese NLP tasks, outperforming BERT base by 0.4% accuracy with 7.5x fewer parameters and9.4x faster inference speed.
Towards offensive language detection and reduction in four Software Engineering communities
Venue: arXiv cs.SE 2021
Year: 2021

Software Engineering (SE) communities such as Stack Overflow have become unwelcoming, particularly through members' use of offensive language. Research has shown that offensive language drives users away from active engagement within these platforms. This work aims to explore this issue more broadly by investigating the nature of offensive language in comments posted by users in four prominent SE platforms - GitHub, Gitter, Slack and Stack Overflow (SO). It proposes an approach to detect and classify offensive language in SE communities by adopting natural language processing and deep learning techniques. Further, a Conflict Reduction System (CRS), which identifies offence and then suggests what changes could be made to minimize offence has been proposed. Beyond showing the prevalence of offensive language in over 1 million comments from four different communities which ranges from 0.07% to 0.43%, our results show promise in successful detection and classification of such language. The CRS system has the potential to drastically reduce manual moderation efforts to detect and reduce offence in SE communities.