OpenRaiser/Intern-Atlas huggingface dataset:
Towards Universal Dense Retrieval for Open-domain Question Answering
Venue: arXiv cs.CL 2021
Year: 2021

In open-domain question answering, a model receives a text question as input and searches for the correct answer using a large evidence corpus. The retrieval step is especially difficult as typical evidence corpora have \textit{millions} of documents, each of which may or may not have the correct answer to the question. Very recently, dense models have replaced sparse methods as the de facto retrieval method. Rather than focusing on lexical overlap to determine similarity, dense methods build an encoding function that captures semantic similarity by learning from a small collection of question-answer or question-context pairs. In this paper, we investigate dense retrieval models in the context of open-domain question answering across different input distributions. To do this, first we introduce an entity-rich question answering dataset constructed from Wikidata facts and demonstrate dense models are unable to generalize to unseen input question distributions. Second, we perform analyses aimed at better understanding the source of the problem and propose new training techniques to improve out-of-domain performance on a wide variety of datasets. We encourage the field to further investigate the creation of a single, universal dense retrieval model that generalizes well across all input distributions.
Distiller: A Systematic Study of Model Distillation Methods in Natural Language Processing
Venue: arXiv cs.CL 2021
Year: 2021

We aim to identify how different components in the KD pipeline affect the resulting performance and how much the optimal KD pipeline varies across different datasets/tasks, such as the data augmentation policy, the loss function, and the intermediate representation for transferring the knowledge between teacher and student. To tease apart their effects, we propose Distiller, a meta KD framework that systematically combines a broad range of techniques across different stages of the KD pipeline, which enables us to quantify each component's contribution. Within Distiller, we unify commonly used objectives for distillation of intermediate representations under a universal mutual information (MI) objective and propose a class of MI-$α$ objective functions with better bias/variance trade-off for estimating the MI between the teacher and the student. On a diverse set of NLP datasets, the best Distiller configurations are identified via large-scale hyperparameter optimization. Our experiments reveal the following: 1) the approach used to distill the intermediate representations is the most important factor in KD performance, 2) among different objectives for intermediate distillation, MI-$α$ performs the best, and 3) data augmentation provides a large boost for small training datasets or small student networks. Moreover, we find that different datasets/tasks prefer different KD algorithms, and thus propose a simple AutoDistiller algorithm that can recommend a good KD pipeline for a new dataset.
Exploiting Curriculum Learning in Unsupervised Neural Machine Translation
Venue: arXiv cs.CL 2021
Year: 2021

Back-translation (BT) has become one of the de facto components in unsupervised neural machine translation (UNMT), and it explicitly makes UNMT have translation ability. However, all the pseudo bi-texts generated by BT are treated equally as clean data during optimization without considering the quality diversity, leading to slow convergence and limited translation performance. To address this problem, we propose a curriculum learning method to gradually utilize pseudo bi-texts based on their quality from multiple granularities. Specifically, we first apply cross-lingual word embedding to calculate the potential translation difficulty (quality) for the monolingual sentences. Then, the sentences are fed into UNMT from easy to hard batch by batch. Furthermore, considering the quality of sentences/tokens in a particular batch are also diverse, we further adopt the model itself to calculate the fine-grained quality scores, which are served as learning factors to balance the contributions of different parts when computing loss and encourage the UNMT model to focus on pseudo data with higher quality. Experimental results on WMT 14 En-Fr, WMT 16 En-De, WMT 16 En-Ro, and LDC En-Zh translation tasks demonstrate that the proposed method achieves consistent improvements with faster convergence speed.
An algorithm for reconstructing level-2 phylogenetic networks from trinets
Venue: arXiv cs.DS 2021
Year: 2021

Evolutionary histories for species that cross with one another or exchange genetic material can be represented by leaf-labelled, directed graphs called phylogenetic networks. A major challenge in the burgeoning area of phylogenetic networks is to develop algorithms for building such networks by amalgamating small networks into a single large network. The level of a phylogenetic network is a measure of its deviation from being a tree; the higher the level of network, the less treelike it becomes. Various algorithms have been developed for building level-1 networks from small networks. However, level-1 networks may not be able to capture the complexity of some data sets. In this paper, we present a polynomial-time algorithm for constructing a rooted binary level-2 phylogenetic network from a collection of 3-leaf networks or trinets. Moreover, we prove that the algorithm will correctly reconstruct such a network if it is given all of the trinets in the network as input. The algorithm runs in time $O(t\cdot n+n^4)$ with $t$ the number of input trinets and $n$ the number of leaves. We also show that there is a fundamental obstruction to constructing level-3 networks from trinets, and so new approaches will need to be developed for constructing level-3 and higher level-networks.
Semi-Supervised Imitation Learning with Mixed Qualities of Demonstrations for Autonomous Driving
Venue: arXiv cs.RO 2021
Year: 2021

In this paper, we consider the problem of autonomous driving using imitation learning in a semi-supervised manner. In particular, both labeled and unlabeled demonstrations are leveraged during training by estimating the quality of each unlabeled demonstration. If the provided demonstrations are corrupted and have a low signal-to-noise ratio, the performance of the imitation learning agent can be degraded significantly. To mitigate this problem, we propose a method called semi-supervised imitation learning (SSIL). SSIL first learns how to discriminate and evaluate each state-action pair's reliability in unlabeled demonstrations by assigning higher reliability values to demonstrations similar to labeled expert demonstrations. This reliability value is called leverage. After this discrimination process, both labeled and unlabeled demonstrations with estimated leverage values are utilized while training the policy in a semi-supervised manner. The experimental results demonstrate the validity of the proposed algorithm using unlabeled trajectories with mixed qualities. Moreover, the hardware experiments using an RC car are conducted to show that the proposed method can be applied to real-world applications.
Smartphone-based geolocation of Internet hosts
Venue: arXiv cs.NI 2021
Year: 2021

The location of Internet hosts is frequently used in distributed applications and networking services. Examples include customized advertising, distribution of content, and position-based security. Unfortunately the relationship between an IP address and its position is in general very weak. This motivates the study of measurement-based IP geolocation techniques, where the position of the target host is actively estimated using the delays between a number of landmarks and the target itself. This paper discusses an IP geolocation method based on crowdsourcing where the smartphones of users operate as landmarks. Since smartphones rely on wireless connections, a specific delay-distance model was derived to capture the characteristics of this novel operating scenario.
Nine Challenges in Artificial Intelligence and Wireless Communications for 6G
Venue: arXiv cs.IT 2021
Year: 2021

In recent years, techniques developed in artificial intelligence (AI), especially those in machine learning (ML), have been successfully applied in various areas, leading to a widespread belief that AI will collectively play an important role in future wireless communications. To accomplish the aspiration, we present nine challenges to be addressed by the interdisciplinary areas of AI/ML and wireless communications, with particular focus towards the sixth generation (6G) wireless networks. Specifically, this article classifies the nine challenges into computation in AI, distributed neural networks and learning, and ML enabled semantic communications.
An optimized application-context relocation approach for Connected and Automated Mobility (CAM)
Venue: arXiv cs.NI 2021
Year: 2021

In this paper, we study and present a management and orchestration framework for vehicular communications, which enables service continuity for the vehicle via an optimized application-context relocation approach. To optimize the transfer of the application-context for Connected and Automated Mobility (CAM) services, our MEC orchestrator performs prediction of resource availability in the edge infrastructure based on the Long Short-Term Memory (LSTM) model, and it makes a final decision on relocation by calculating the outcome of a Multi-Criteria Decision-Making (MCDM) algorithm, taking into account the i) resource prediction, ii) latency and bandwidth on the communication links, and iii) geographical locations of the vehicle and edge hosts in the network infrastructure. Furthermore, we have built a proof-of-concept for the orchestration framework in a real-life distributed testbed environment, to showcase the efficiency in optimizing the edge host selection and application context relocation towards achieving continuity of a service that informs vehicle about the driving conditions on the road.
Energy efficient distributed analytics at the edge of the network for IoT environments
Venue: arXiv cs.DC 2021
Year: 2021

Due to the pervasive diffusion of personal mobile and IoT devices, many "smart environments" (e.g., smart cities and smart factories) will be, generators of huge amounts of data. Currently, analysis of this data is typically achieved through centralised cloud-based services. However, according to many studies, this approach may present significant issues from the standpoint of data ownership, as well as wireless network capacity. In this paper, we exploit the fog computing paradigm to move computation close to where data is produced. We exploit a well-known distributed machine learning framework (Hypothesis Transfer Learning), and perform data analytics on mobile nodes passing by IoT devices, in addition to fog gateways at the edge of the network infrastructure. We analyse the performance of different configurations of the distributed learning framework, in terms of (i) accuracy obtained in the learning task and (ii) energy spent to send data between the involved nodes. Specifically, we consider reference wireless technologies for communication between the different types of nodes we consider, e.g. LTE, Nb-IoT, 802.15.4, 802.11, etc. Our results show that collecting data through the mobile nodes and executing the distributed analytics using short-range communication technologies, such as 802.15.4 and 802.11, allows to strongly reduce the energy consumption of the system up to $94\%$ with a loss in accuracy w.r.t. a centralised cloud solution up to $2\%$.
An Efficient Simulation-Based Travel Demand Calibration Algorithm for Large-Scale Metropolitan Traffic Models
Venue: arXiv cs.MA 2021
Year: 2021

Metropolitan scale vehicular traffic modeling is used by a variety of private and public sector urban mobility stakeholders to inform the design and operations of road networks. High-resolution stochastic traffic simulators are increasingly used to describe detailed demand-supply interactions. The design of efficient calibration techniques remains a major challenge. This paper considers a class of high-dimensional calibration problems known as origin-destination (OD) calibration. We formulate the problem as a continuous simulation-based optimization problem. Our proposed algorithm builds upon recent metamodel methods that tackle the simulation-based problem by solving a sequence of approximate analytical optimization problems, which rely on the use of analytical network models. In this paper, we formulate a network model defined as a system of linear equations, the dimension of which scales linearly with the number of roads with field data and independently of the dimension of the route choice set. This makes the approach suitable for large-scale metropolitan networks. The approach has enhanced efficiency compared with past metamodel formulations that are based on systems of nonlinear, rather than linear, equations. It also has enhanced efficiency compared to traditional calibration methods that resort to simulation-based estimates of traffic assignment matrices, while the proposed approach uses analytical approximations of these matrices. We benchmark the approach considering a peak period Salt Lake City case study and calibrate based on field vehicular count data. The new formulation yields solutions with good performance and is suitable for large-scale road networks.
Remarks on MOBS and cryptosystems using semidirect products
Venue: arXiv cs.CR 2021
Year: 2021

Recently, several cryptosystems have been proposed based semidirect products of various algebraic structures. Efficient attacks against several of them have already been given, along with a very general attack. The purpose of this note is to provide an observation that can be used as a point-of-attack for similar systems, and show how it can be used to efficiently cryptanalyze the MOBS system.
Learning Dynamics from Noisy Measurements using Deep Learning with a Runge-Kutta Constraint
Venue: arXiv cs.LG 2021
Year: 2021

Measurement noise is an integral part while collecting data of a physical process. Thus, noise removal is a necessary step to draw conclusions from these data, and it often becomes quite essential to construct dynamical models using these data. We discuss a methodology to learn differential equation(s) using noisy and sparsely sampled measurements. In our methodology, the main innovation can be seen in of integration of deep neural networks with a classical numerical integration method. Precisely, we aim at learning a neural network that implicitly represents the data and an additional neural network that models the vector fields of the dependent variables. We combine these two networks by enforcing the constraint that the data at the next time-steps can be given by following a numerical integration scheme such as the fourth-order Runge-Kutta scheme. The proposed framework to learn a model predicting the vector field is highly effective under noisy measurements. The approach can handle scenarios where dependent variables are not available at the same temporal grid. We demonstrate the effectiveness of the proposed method to learning models using data obtained from various differential equations. The proposed approach provides a promising methodology to learn dynamic models, where the first-principle understanding remains opaque.
Diversity by Design: Balancing Protection and Inclusion in Social Networks
Venue: arXiv cs.SI 2021
Year: 2021

The unreflected promotion of diversity as a value in social interactions -- including technology-mediated ones -- risks emphasizing the benefits of inclusion at the cost of not recognizing the potential harm from failing to protect stigmatized or marginalized individuals. Adopting the stance that technology is not value-neutral, we attempt to answer the question of how technology-mediated social platforms could accommodate \emph{diversity by design}, by balancing the often competing values of protection and inclusion. This short paper presents our research agenda as well as initial analysis and outcomes. Building on approaches from scenario planning and the methodology of Value Sensitive Design, we identify ethical principles and arguments on how to curate diversity, which we seek to operationalize through formal argumentation.
Community Formation and Detection on GitHub Collaboration Networks
Venue: arXiv cs.SI 2021
Year: 2021

This paper studies community formation in OSS collaboration networks. While most current work examines the emergence of small-scale OSS projects, our approach draws on a large-scale historical dataset of 1.8 million GitHub users and their repository contributions. OSS collaborations are characterized by small groups of users that work closely together, leading to the presence of communities defined by short cycles in the underlying network structure. To understand the impact of this phenomenon, we apply a pre-processing step that accounts for the cyclic network structure by using Renewal-Nonbacktracking Random Walks (RNBRW) and the strength of pairwise collaborations before implementing the Louvain method to identify communities within the network. Equipping Louvain with RNBRW and the contribution strength provides a more assertive approach for detecting small-scale teams and reveals nontrivial differences in community detection such as users tendencies toward preferential attachment to more established collaboration communities. Using this method, we also identify key factors that affect community formation, including the effect of users location and primary programming language, which was determined using a comparative method of contribution activities. Overall, this paper offers several promising methodological insights for both open-source software experts and network scholars interested in studying team formation.
Revisiting the Uniform Information Density Hypothesis
Venue: arXiv cs.CL 2021
Year: 2021

The uniform information density (UID) hypothesis posits a preference among language users for utterances structured such that information is distributed uniformly across a signal. While its implications on language production have been well explored, the hypothesis potentially makes predictions about language comprehension and linguistic acceptability as well. Further, it is unclear how uniformity in a linguistic signal -- or lack thereof -- should be measured, and over which linguistic unit, e.g., the sentence or language level, this uniformity should hold. Here we investigate these facets of the UID hypothesis using reading time and acceptability data. While our reading time results are generally consistent with previous work, they are also consistent with a weakly super-linear effect of surprisal, which would be compatible with UID's predictions. For acceptability judgments, we find clearer evidence that non-uniformity in information density is predictive of lower acceptability. We then explore multiple operationalizations of UID, motivated by different interpretations of the original hypothesis, and analyze the scope over which the pressure towards uniformity is exerted. The explanatory power of a subset of the proposed operationalizations suggests that the strongest trend may be a regression towards a mean surprisal across the language, rather than the phrase, sentence, or document -- a finding that supports a typical interpretation of UID, namely that it is the byproduct of language users maximizing the use of a (hypothetical) communication channel.
Obstacle-aware Waypoint Generation for Long-range Guidance of Deep-Reinforcement-Learning-based Navigation Approaches
Venue: arXiv cs.RO 2021
Year: 2021

Navigation of mobile robots within crowded environments is an essential task in various use cases, such as delivery, health care, or logistics. Deep Reinforcement Learning (DRL) emerged as an alternative method to replace overly conservative approaches and promises more efficient and flexible navigation. However, Deep Reinforcement Learning is limited to local navigation due to its myopic nature. Previous research works proposed various ways to combine Deep Reinforcement Learning with conventional methods but a common problem is the complexity of highly dynamic environments due to the unpredictability of humans and other objects within the environment. In this paper, we propose a hierarchical waypoint generator, which considers moving obstacles and thus generates safer and more robust waypoints for Deep-Reinforcement-Learning-based local planners. Therefore, we utilize Delaunay Triangulation to encode obstacles and incorporate an extended hybrid A-Star approach to efficiently search for an optimal solution in the time-state space. We compared our waypoint generator against two baseline approaches and outperform them in terms of safety, efficiency, and robustness.
WebFlow: Scalable and Decentralized Routing for Payment Channel Networks with High Resource Utilization
Venue: arXiv cs.NI 2021
Year: 2021

Payment channel networks (PCNs) have been designed and utilized to address the scalability challenge and throughput limitation of blockchains. Routing is a core problem of PCNs. An ideal PCN routing method needs to achieve 1) high scalability that can maintain low per-node memory and communication cost for large PCNs, 2) high resource utilization of payment channels, and 3) the privacy of users. However, none of the existing PCN systems consider all these requirements. In this work, we propose WebFlow, a distributed routing solution for PCNs, which only requires each user to maintain localized information and can be used for massive-scale networks with high resource utilization. We make use of two distributed data structures: multi-hop Delaunay triangulation (MDT) originally proposed for wireless networks and our innovation called distributed Voronoi diagram. We propose new protocols to generate a virtual Euclidean space in order to apply MDT to PCNs and use the distributed Voronoi diagram to enhance routing privacy. We conduct extensive simulations and prototype implementation to further evaluate WebFlow. The results using real and synthetic PCN topologies and transaction traces show that WebFlow can achieve extremely low per-node overhead and a high success rate compared to existing methods.
Simple and Effective Zero-shot Cross-lingual Phoneme Recognition
Venue: arXiv cs.CL 2021
Year: 2021

Recent progress in self-training, self-supervised pretraining and unsupervised learning enabled well performing speech recognition systems without any labeled data. However, in many cases there is labeled data available for related languages which is not utilized by these methods. This paper extends previous work on zero-shot cross-lingual transfer learning by fine-tuning a multilingually pretrained wav2vec 2.0 model to transcribe unseen languages. This is done by mapping phonemes of the training languages to the target language using articulatory features. Experiments show that this simple method significantly outperforms prior work which introduced task-specific architectures and used only part of a monolingually pretrained model.