OpenRaiser/Intern-Atlas huggingface dataset:
Human Mobility Modeling During the COVID-19 Pandemic via Deep Graph Diffusion Infomax
Venue: arXiv cs.LG 2022
Year: 2022

Non-Pharmaceutical Interventions (NPIs), such as social gathering restrictions, have shown effectiveness to slow the transmission of COVID-19 by reducing the contact of people. To support policy-makers, multiple studies have first modeled human mobility via macro indicators (e.g., average daily travel distance) and then studied the effectiveness of NPIs. In this work, we focus on mobility modeling and, from a micro perspective, aim to predict locations that will be visited by COVID-19 cases. Since NPIs generally cause economic and societal loss, such a micro perspective prediction benefits governments when they design and evaluate them. However, in real-world situations, strict privacy data protection regulations result in severe data sparsity problems (i.e., limited case and location information). To address these challenges, we formulate the micro perspective mobility modeling into computing the relevance score between a diffusion and a location, conditional on a geometric graph. we propose a model named Deep Graph Diffusion Infomax (DGDI), which jointly models variables including a geometric graph, a set of diffusions and a set of locations.To facilitate the research of COVID-19 prediction, we present two benchmarks that contain geometric graphs and location histories of COVID-19 cases. Extensive experiments on the two benchmarks show that DGDI significantly outperforms other competing methods.
Adaptive Low-Precision Training for Embeddings in Click-Through Rate Prediction
Venue: arXiv cs.LG 2022
Year: 2022

Embedding tables are usually huge in click-through rate (CTR) prediction models. To train and deploy the CTR models efficiently and economically, it is necessary to compress their embedding tables at the training stage. To this end, we formulate a novel quantization training paradigm to compress the embeddings from the training stage, termed low-precision training (LPT). Also, we provide theoretical analysis on its convergence. The results show that stochastic weight quantization has a faster convergence rate and a smaller convergence error than deterministic weight quantization in LPT. Further, to reduce the accuracy degradation, we propose adaptive low-precision training (ALPT) that learns the step size (i.e., the quantization resolution) through gradient descent. Experiments on two real-world datasets confirm our analysis and show that ALPT can significantly improve the prediction accuracy, especially at extremely low bit widths. For the first time in CTR models, we successfully train 8-bit embeddings without sacrificing prediction accuracy. The code of ALPT is publicly available.
Automated Variable Renaming: Are We There Yet?
Venue: arXiv cs.SE 2022
Year: 2022

Identifiers, such as method and variable names, form a large portion of source code. Therefore, low-quality identifiers can substantially hinder code comprehension. To support developers in using meaningful identifiers, several (semi-)automatic techniques have been proposed, mostly being data-driven (e.g. statistical language models, deep learning models) or relying on static code analysis. Still, limited empirical investigations have been performed on the effectiveness of such techniques for recommending developers with meaningful identifiers, possibly resulting in rename refactoring operations. We present a large-scale study investigating the potential of data-driven approaches to support automated variable renaming. We experiment with three state-of-the-art techniques: a statistical language model and two DL-based models. The three approaches have been trained and tested on three datasets we built with the goal of evaluating their ability to recommend meaningful variable identifiers. Our quantitative and qualitative analyses show the potential of such techniques that, under specific conditions, can provide valuable recommendations and are ready to be integrated in rename refactoring tools. Nonetheless, our results also highlight limitations of the experimented approaches that call for further research in this field.
Resistance Distances in Simplicial Networks
Venue: arXiv cs.SI 2022
Year: 2022

It is well known that in many real networks, such as brain networks and scientific collaboration networks, there exist higher-order nonpairwise relations among nodes, i.e., interactions between among than two nodes at a time. This simplicial structure can be described by simplicial complexes and has an important effect on topological and dynamical properties of networks involving such group interactions. In this paper, we study analytically resistance distances in iteratively growing networks with higher-order interactions characterized by the simplicial structure that is controlled by a parameter q. We derive exact formulas for interesting quantities about resistance distances, including Kirchhoff index, additive degree-Kirchhoff index, multiplicative degree-Kirchhoff index, as well as average resistance distance, which have found applications in various areas elsewhere. We show that the average resistance distance tends to a q-dependent constant, indicating the impact of simplicial organization on the structural robustness measured by average resistance distance.
Interactive introduction to self-calibrating interfaces
Venue: arXiv cs.HC 2022
Year: 2022

This interactive paper aims to provide an intuitive understanding of the self-calibrating interface paradigm. Under this paradigm, you can choose how to use an interface which can adapt to your preferences on the fly. We introduce a PIN entering task and gradually release constraints, moving from a pre-calibrated interface to a self-calibrating interface while increasing the complexity of input modalities from buttons, to points on a map, to sketches, and finally to spoken words. This is not a traditional research paper with a hypothesis and experimental results to support claims; the research supporting this work has already been done and we refer to it extensively in the later sections. Instead, our aim is to walk you through an intriguing interaction paradigm in small logical steps with supporting illustrations, interactive demonstrations, and videos to reinforce your learning. We designed this paper for the enjoyments of curious minds of any backgrounds, it is written in plain English and no prior knowledge is necessary. All demos are available online at openvault.jgrizou.com and linked individually in the paper.
P-Transformer: Towards Better Document-to-Document Neural Machine Translation
Venue: arXiv cs.CL 2022
Year: 2022

Directly training a document-to-document (Doc2Doc) neural machine translation (NMT) via Transformer from scratch, especially on small datasets usually fails to converge. Our dedicated probing tasks show that 1) both the absolute position and relative position information gets gradually weakened or even vanished once it reaches the upper encoder layers, and 2) the vanishing of absolute position information in encoder output causes the training failure of Doc2Doc NMT. To alleviate this problem, we propose a position-aware Transformer (P-Transformer) to enhance both the absolute and relative position information in both self-attention and cross-attention. Specifically, we integrate absolute positional information, i.e., position embeddings, into the query-key pairs both in self-attention and cross-attention through a simple yet effective addition operation. Moreover, we also integrate relative position encoding in self-attention. The proposed P-Transformer utilizes sinusoidal position encoding and does not require any task-specified position embedding, segment embedding, or attention mechanism. Through the above methods, we build a Doc2Doc NMT model with P-Transformer, which ingests the source document and completely generates the target document in a sequence-to-sequence (seq2seq) way. In addition, P-Transformer can be applied to seq2seq-based document-to-sentence (Doc2Sent) and sentence-to-sentence (Sent2Sent) translation. Extensive experimental results of Doc2Doc NMT show that P-Transformer significantly outperforms strong baselines on widely-used 9 document-level datasets in 7 language pairs, covering small-, middle-, and large-scales, and achieves a new state-of-the-art. Experimentation on discourse phenomena shows that our Doc2Doc NMT models improve the translation quality in both BLEU and discourse coherence. We make our code available on Github.
BeautyREC: Robust, Efficient, and Content-preserving Makeup Transfer
Venue: arXiv cs.CV 2022
Year: 2022

In this work, we propose a Robust, Efficient, and Component-specific makeup transfer method (abbreviated as BeautyREC). A unique departure from prior methods that leverage global attention, simply concatenate features, or implicitly manipulate features in latent space, we propose a component-specific correspondence to directly transfer the makeup style of a reference image to the corresponding components (e.g., skin, lips, eyes) of a source image, making elaborate and accurate local makeup transfer. As an auxiliary, the long-range visual dependencies of Transformer are introduced for effective global makeup transfer. Instead of the commonly used cycle structure that is complex and unstable, we employ a content consistency loss coupled with a content encoder to implement efficient single-path makeup transfer. The key insights of this study are modeling component-specific correspondence for local makeup transfer, capturing long-range dependencies for global makeup transfer, and enabling efficient makeup transfer via a single-path structure. We also contribute BeautyFace, a makeup transfer dataset to supplement existing datasets. This dataset contains 3,000 faces, covering more diverse makeup styles, face poses, and races. Each face has annotated parsing map. Extensive experiments demonstrate the effectiveness of our method against state-of-the-art methods. Besides, our method is appealing as it is with only 1M parameters, outperforming the state-of-the-art methods (BeautyGAN: 8.43M, PSGAN: 12.62M, SCGAN: 15.30M, CPM: 9.24M, SSAT: 10.48M).
The free group in R
Venue: arXiv cs.SC 2022
Year: 2022

Here I present the freegroup package for working with the free group on a finite set of symbols. The package is vectorised; internally it uses an efficient matrix-based representation for free group objects but uses a configurable print method. A range of R-centric functionality is provided. It is available on CRAN at https://CRAN.R-project.org/package=freegroup.
Parameter-Efficient Finetuning of Transformers for Source Code
Venue: arXiv cs.CL 2022
Year: 2022

Pretrained Transformers achieve state-of-the-art performance in various code-processing tasks but may be too large to be deployed. As software development tools often incorporate modules for various purposes which may potentially use a single instance of the pretrained model, it appears relevant to utilize parameter-efficient fine-tuning for the pretrained models of code. In this work, we test two widely used approaches, adapters and LoRA, which were initially tested on NLP tasks, on four code-processing tasks. We find that though the efficient fine-tuning approaches may achieve comparable or higher performance than the standard, full, fine-tuning in code understanding tasks, they underperform full fine-tuning in code-generative tasks. These results underline the importance of testing efficient fine-tuning approaches on other domains than NLP and motivate future research in efficient fine-tuning for source code.
Human Digital Twin: A Survey
Venue: arXiv cs.HC 2022
Year: 2022

Digital twin has recently attracted growing attention, leading to intensive research and applications. Along with this, a new research area, dubbed as "human digital twin" (HDT), has emerged. Similar to the conception of digital twin, HDT is referred to as the replica of a physical-world human in the digital world. Nevertheless, HDT is much more complicated and delicate compared to digital twins of any physical systems and processes, due to humans' dynamic and evolutionary nature, including physical, behavioral, social, physiological, psychological, cognitive, and biological dimensions. Studies on HDT are limited, and the research is still in its infancy. In this paper, we first examine the inception, development, and application of the digital twin concept, providing a context within which we formally define and characterize HDT based on the similarities and differences between digital twin and HDT. Then we conduct an extensive literature review on HDT research, analyzing underpinning technologies and establishing typical frameworks in which the core HDT functions or components are organized. Built upon the findings from the above work, we propose a generic architecture for the HDT system and describe the core function blocks and corresponding technologies. Following this, we present the state of the art of HDT technologies and applications in the healthcare, industry, and daily life domain. Finally, we discuss various issues related to the development of HDT and point out the trends and challenges of future HDT research and development.
A Survey and Comparison of Industrial and Academic Research on the Evolution of Software Product Lines
Venue: arXiv cs.SE 2022
Year: 2022

Past research on software product lines has focused on the initial development of reusable assets and related challenges, such as cost estimation and implementation issues. Naturally, as software product lines are increasingly adopted throughout industry, their ongoing maintenance and evolution are getting more attention as well. However, it is not clear to what degree research is following this trend, and where the interests and demands of the industry lie. In this technical report, we provide a survey and comparison of selected publications on software product line maintenance and evolution at SPLC. In particular, we analyze and discuss similarities and differences of these papers with regard to their affiliation with industry and academia. From this, we infer directions for future research that pave the way for systematic and organized evolution of software product lines, from which industry may benefit as well.
GenSyn: A Multi-stage Framework for Generating Synthetic Microdata using Macro Data Sources
Venue: arXiv cs.LG 2022
Year: 2022

Individual-level data (microdata) that characterizes a population, is essential for studying many real-world problems. However, acquiring such data is not straightforward due to cost and privacy constraints, and access is often limited to aggregated data (macro data) sources. In this study, we examine synthetic data generation as a tool to extrapolate difficult-to-obtain high-resolution data by combining information from multiple easier-to-obtain lower-resolution data sources. In particular, we introduce a framework that uses a combination of univariate and multivariate frequency tables from a given target geographical location in combination with frequency tables from other auxiliary locations to generate synthetic microdata for individuals in the target location. Our method combines the estimation of a dependency graph and conditional probabilities from the target location with the use of a Gaussian copula to leverage the available information from the auxiliary locations. We perform extensive testing on two real-world datasets and demonstrate that our approach outperforms prior approaches in preserving the overall dependency structure of the data while also satisfying the constraints defined on the different variables.
A Cross-Layer Descent Approach for Resilient Network Operations of Proliferated LEO Satellites
Venue: arXiv cs.NI 2022
Year: 2022

With the proliferated low-Earth-orbit (LEO) satellites in mega-constellations, the future Internet will be able to reach any place on Earth, providing high-quality services to everyone. However, high-quality operations in terms of timeliness and resilience are lacking in the current solutions. This paper proposes a multi-layer networking approach called "Cross-Layer Descent (CLD)". Based on the proposed system model, principles, and measures, CLD can support foundational services such as telecommand (TC) transmissions for various network operation missions for LEO satellites compliant with the Consultative Committee for Space Data Systems (CCSDS) standards. The CLD approach enhances timing and resilience requirements using advanced communication payloads. From the simulation-based analysis, the proposed scheme outperforms other classical ones in resilience and latency for typical TC missions. The future work and conclusive remarks are discussed at the end.
Dirichlet-Survival Process: Scalable Inference of Topic-Dependent Diffusion Networks
Venue: arXiv cs.LG 2022
Year: 2022

Information spread on networks can be efficiently modeled by considering three features: documents' content, time of publication relative to other publications, and position of the spreader in the network. Most previous works model up to two of those jointly, or rely on heavily parametric approaches. Building on recent Dirichlet-Point processes literature, we introduce the Houston (Hidden Online User-Topic Network) model, that jointly considers all those features in a non-parametric unsupervised framework. It infers dynamic topic-dependent underlying diffusion networks in a continuous-time setting along with said topics. It is unsupervised; it considers an unlabeled stream of triplets shaped as \textit{(time of publication, information's content, spreading entity)} as input data. Online inference is conducted using a sequential Monte-Carlo algorithm that scales linearly with the size of the dataset. Our approach yields consequent improvements over existing baselines on both cluster recovery and subnetworks inference tasks.
Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization
Venue: arXiv cs.LG 2022
Year: 2022

Knowledge Distillation (KD) has been extensively used for natural language understanding (NLU) tasks to improve a small model's (a student) generalization by transferring the knowledge from a larger model (a teacher). Although KD methods achieve state-of-the-art performance in numerous settings, they suffer from several problems limiting their performance. It is shown in the literature that the capacity gap between the teacher and the student networks can make KD ineffective. Additionally, existing KD techniques do not mitigate the noise in the teacher's output: modeling the noisy behaviour of the teacher can distract the student from learning more useful features. We propose a new KD method that addresses these problems and facilitates the training compared to previous techniques. Inspired by continuation optimization, we design a training procedure that optimizes the highly non-convex KD objective by starting with the smoothed version of this objective and making it more complex as the training proceeds. Our method (Continuation-KD) achieves state-of-the-art performance across various compact architectures on NLU (GLUE benchmark) and computer vision tasks (CIFAR-10 and CIFAR-100).
Degreewidth: a New Parameter for Solving Problems on Tournaments
Venue: arXiv cs.DM 2022
Year: 2022

In the paper, we define a new parameter for tournaments called degreewidth which can be seen as a measure of how far is the tournament from being acyclic. The degreewidth of a tournament $T$ denoted by $Δ(T)$ is the minimum value $k$ for which we can find an ordering $\langle v_1, \dots, v_n \rangle$ of the vertices of $T$ such that every vertex is incident to at most $k$ backward arcs (\textit{i.e.} an arc $(v_i,v_j)$ such that $j<i$). Thus, a tournament is acyclic if and only if its degreewidth is zero. Additionally, the class of sparse tournaments defined by Bessy et al. [ESA 2017] is exactly the class of tournaments with degreewidth one. We first study computational complexity of finding degreewidth. Namely, we show it is NP-hard and complement this result with a $3$-approximation algorithm. We also provide a cubic algorithm to decide if a tournament is sparse. Finally, we study classical graph problems \textsc{Dominating Set} and \textsc{Feedback Vertex Set} parameterized by degreewidth. We show the former is fixed parameter tractable whereas the latter is NP-hard on sparse tournaments. Additionally, we study \textsc{Feedback Arc Set} on sparse tournaments.
Reconstructing Humpty Dumpty: Multi-feature Graph Autoencoder for Open Set Action Recognition
Venue: arXiv cs.CV 2022
Year: 2022

Most action recognition datasets and algorithms assume a closed world, where all test samples are instances of the known classes. In open set problems, test samples may be drawn from either known or unknown classes. Existing open set action recognition methods are typically based on extending closed set methods by adding post hoc analysis of classification scores or feature distances and do not capture the relations among all the video clip elements. Our approach uses the reconstruction error to determine the novelty of the video since unknown classes are harder to put back together and thus have a higher reconstruction error than videos from known classes. We refer to our solution to the open set action recognition problem as "Humpty Dumpty", due to its reconstruction abilities. Humpty Dumpty is a novel graph-based autoencoder that accounts for contextual and semantic relations among the clip pieces for improved reconstruction. A larger reconstruction error leads to an increased likelihood that the action can not be reconstructed, i.e., can not put Humpty Dumpty back together again, indicating that the action has never been seen before and is novel/unknown. Extensive experiments are performed on two publicly available action recognition datasets including HMDB-51 and UCF-101, showing the state-of-the-art performance for open set action recognition.
Where To Start? Transferring Simple Skills to Complex Environments
Venue: arXiv cs.RO 2022
Year: 2022

Robot learning provides a number of ways to teach robots simple skills, such as grasping. However, these skills are usually trained in open, clutter-free environments, and therefore would likely cause undesirable collisions in more complex, cluttered environments. In this work, we introduce an affordance model based on a graph representation of an environment, which is optimised during deployment to find suitable robot configurations to start a skill from, such that the skill can be executed without any collisions. We demonstrate that our method can generalise a priori acquired skills to previously unseen cluttered and constrained environments, in simulation and in the real world, for both a grasping and a placing task.
In Defense of Cross-Encoders for Zero-Shot Retrieval
Venue: arXiv cs.IR 2022
Year: 2022

Bi-encoders and cross-encoders are widely used in many state-of-the-art retrieval pipelines. In this work we study the generalization ability of these two types of architectures on a wide range of parameter count on both in-domain and out-of-domain scenarios. We find that the number of parameters and early query-document interactions of cross-encoders play a significant role in the generalization ability of retrieval models. Our experiments show that increasing model size results in marginal gains on in-domain test sets, but much larger gains in new domains never seen during fine-tuning. Furthermore, we show that cross-encoders largely outperform bi-encoders of similar size in several tasks. In the BEIR benchmark, our largest cross-encoder surpasses a state-of-the-art bi-encoder by more than 4 average points. Finally, we show that using bi-encoders as first-stage retrievers provides no gains in comparison to a simpler retriever such as BM25 on out-of-domain tasks. The code is available at https://github.com/guilhermemr04/scaling-zero-shot-retrieval.git