OpenRaiser/Intern-Atlas huggingface dataset:
Improving Candidate Generation for Low-resource Cross-lingual Entity Linking
Venue: arXiv cs.CL 2020
Year: 2020

Cross-lingual entity linking (XEL) is the task of finding referents in a target-language knowledge base (KB) for mentions extracted from source-language texts. The first step of (X)EL is candidate generation, which retrieves a list of plausible candidate entities from the target-language KB for each mention. Approaches based on resources from Wikipedia have proven successful in the realm of relatively high-resource languages (HRL), but these do not extend well to low-resource languages (LRL) with few, if any, Wikipedia pages. Recently, transfer learning methods have been shown to reduce the demand for resources in the LRL by utilizing resources in closely-related languages, but the performance still lags far behind their high-resource counterparts. In this paper, we first assess the problems faced by current entity candidate generation methods for low-resource XEL, then propose three improvements that (1) reduce the disconnect between entity mentions and KB entries, and (2) improve the robustness of the model to low-resource scenarios. The methods are simple, but effective: we experiment with our approach on seven XEL datasets and find that they yield an average gain of 16.9% in Top-30 gold candidate recall, compared to state-of-the-art baselines. Our improved model also yields an average gain of 7.9% in in-KB accuracy of end-to-end XEL.
Meta-Embeddings Based On Self-Attention
Venue: arXiv cs.CL 2020
Year: 2020

Creating meta-embeddings for better performance in language modelling has received attention lately, and methods based on concatenation or merely calculating the arithmetic mean of more than one separately trained embeddings to perform meta-embeddings have shown to be beneficial. In this paper, we devise a new meta-embedding model based on the self-attention mechanism, namely the Duo. With less than 0.4M parameters, the Duo mechanism achieves state-of-the-art accuracy in text classification tasks such as 20NG. Additionally, we propose a new meta-embedding sequece-to-sequence model for machine translation, which to the best of our knowledge, is the first machine translation model based on more than one word-embedding. Furthermore, it has turned out that our model outperform the Transformer not only in terms of achieving a better result, but also a faster convergence on recognized benchmarks, such as the WMT 2014 English-to-French translation task.
OTFS Based Receiver Scheme With Multi-Antennas in High-Mobility V2X Systems
Venue: arXiv cs.IT 2020
Year: 2020

Vehicle-to-everything (V2X) is considered as one of the most important applications of future wireless communication networks. However, the Doppler effect caused by the vehicle mobility may seriously deteriorate the performance of the vehicular communication links, especially when the channels exhibit a large number of Doppler frequency offsets (DFOs). Orthogonal time frequency space (OTFS) is a new waveform designed in the delay-Doppler domain, and can effectively convert a doubly dispersive channel into an almost non-fading channel, which makes it very attractive for V2X communications. In this paper, we design a novel OTFS based receiver with multi-antennas to deal with the high-mobility challenges in V2X systems. We show that the multiple DFOs associated with multipaths can be separated with the high-spatial resolution provided by multi-antennas, which leads to an enhanced sparsity of the OTFS channel in the delay-Doppler domain and bears a potential to reduce the complexity of the message passing (MP) detection algorithm. Based on this observation, we further propose a joint MP-maximum ration combining (MRC) iterative detection for OTFS, where the integration of MRC significantly improves the convergence performance of the iteration and gains an excellent system error performance. Finally, we provide numerical simulation results to corroborate the superiorities of the proposed scheme.
Pattern Similarity-based Machine Learning Methods for Mid-term Load Forecasting: A Comparative Study
Venue: arXiv cs.LG 2020
Year: 2020

Pattern similarity-based methods are widely used in classification and regression problems. Repeated, similar-shaped cycles observed in seasonal time series encourage to apply these methods for forecasting. In this paper we use the pattern similarity-based methods for forecasting monthly electricity demand expressing annual seasonality. An integral part of the models is the time series representation using patterns of time series sequences. Pattern representation ensures the input and output data unification through trend filtering and variance equalization. Consequently, pattern representation simplifies the forecasting problem and allows us to use models based on pattern similarity. We consider four such models: nearest neighbor model, fuzzy neighborhood model, kernel regression model and general regression neural network. A regression function is constructed by aggregation output patterns with weights dependent on the similarity between input patterns. The advantages of the proposed models are: clear principle of operation, small number of parameters to adjust, fast optimization procedure, good generalization ability, working on the newest data without retraining, robustness to missing input variables, and generating a vector as an output. In the experimental part of the work the proposed models were used to forecasting the monthly demand for 35 European countries. The model performances were compared with the performances of the classical models such as ARIMA and exponential smoothing as well as state-of-the-art models such as multilayer perceptron, neuro-fuzzy system and long short-term memory model. The results show high performance of the proposed models which outperform the comparative models in accuracy, simplicity and ease of optimization.
FlexServe: Deployment of PyTorch Models as Flexible REST Endpoints
Venue: arXiv cs.DC 2020
Year: 2020

The integration of artificial intelligence capabilities into modern software systems is increasingly being simplified through the use of cloud-based machine learning services and representational state transfer architecture design. However, insufficient information regarding underlying model provenance and the lack of control over model evolution serve as an impediment to the more widespread adoption of these services in many operational environments which have strict security requirements. Furthermore, tools such as TensorFlow Serving allow models to be deployed as RESTful endpoints, but require error-prone transformations for PyTorch models as these dynamic computational graphs. This is in contrast to the static computational graphs of TensorFlow. To enable rapid deployments of PyTorch models without intermediate transformations we have developed FlexServe, a simple library to deploy multi-model ensembles with flexible batching.
Evaluation Framework For Large-scale Federated Learning
Venue: arXiv cs.LG 2020
Year: 2020

Federated learning is proposed as a machine learning setting to enable distributed edge devices, such as mobile phones, to collaboratively learn a shared prediction model while keeping all the training data on device, which can not only take full advantage of data distributed across millions of nodes to train a good model but also protect data privacy. However, learning in scenario above poses new challenges. In fact, data across a massive number of unreliable devices is likely to be non-IID (identically and independently distributed), which may make the performance of models trained by federated learning unstable. In this paper, we introduce a framework designed for large-scale federated learning which consists of approaches to generating dataset and modular evaluation framework. Firstly, we construct a suite of open-source non-IID datasets by providing three respects including covariate shift, prior probability shift, and concept shift, which are grounded in real-world assumptions. In addition, we design several rigorous evaluation metrics including the number of network nodes, the size of datasets, the number of communication rounds and communication resources etc. Finally, we present an open-source benchmark for large-scale federated learning research.
Scalable Tactile Sensing for an Omni-adaptive Soft Robot Finger
Venue: arXiv cs.RO 2020
Year: 2020

Robotic fingers made of soft material and compliant structures usually lead to superior adaptation when interacting with the unstructured physical environment. In this paper, we present an embedded sensing solution using optical fibers for an omni-adaptive soft robotic finger with exceptional adaptation in all directions. In particular, we managed to insert a pair of optical fibers inside the finger's structural cavity without interfering with its adaptive performance. The resultant integration is scalable as a versatile, low-cost, and moisture-proof solution for physically safe human-robot interaction. In addition, we experimented with our finger design for an object sorting task and identified sectional diameters of 94\% objects within the $\pm$6mm error and measured 80\% of the structural strains within $\pm$0.1mm/mm error. The proposed sensor design opens many doors in future applications of soft robotics for scalable and adaptive physical interactions in the unstructured environment.
Volumetric landmark detection with a multi-scale shift equivariant neural network
Venue: arXiv cs.CV 2020
Year: 2020

Deep neural networks yield promising results in a wide range of computer vision applications, including landmark detection. A major challenge for accurate anatomical landmark detection in volumetric images such as clinical CT scans is that large-scale data often constrain the capacity of the employed neural network architecture due to GPU memory limitations, which in turn can limit the precision of the output. We propose a multi-scale, end-to-end deep learning method that achieves fast and memory-efficient landmark detection in 3D images. Our architecture consists of blocks of shift-equivariant networks, each of which performs landmark detection at a different spatial scale. These blocks are connected from coarse to fine-scale, with differentiable resampling layers, so that all levels can be trained together. We also present a noise injection strategy that increases the robustness of the model and allows us to quantify uncertainty at test time. We evaluate our method for carotid artery bifurcations detection on 263 CT volumes and achieve a better than state-of-the-art accuracy with mean Euclidean distance error of 2.81mm.
Impact of Spatially Consistent Channels on Digital Beamforming for Millimeter-Wave Systems
Venue: arXiv cs.IT 2020
Year: 2020

The premise of massive multiple-input multiple-output (MIMO) is based around coherent transmission and detection. Majority of the vast literature on massive MIMO presents performance evaluations over simplified statistical propagation models. All such models are drop-based and do not ensure continuity of channel parameters. In this paper, we quantify the impact of spatially consistent (SC) models on beamforming for massive MIMO systems. We focus on the downlink of a 28GHz multiuser urban microcellular scenario. Using the recently standardized Third Generation Partnership Project 38.901 SC-I procedure, we evaluate the signal-to-interference-plus-noise ratio of a user equipment and the system ergodic sum spectral efficiency with zero-forcing, block diagonalization, and signal-to-leakage-plus-noise ratio beamforming. Our results disclose that at practical signal-to-noise ratio levels, SC channels yield a significant performance loss relative to the case without SC due to substantial spatial correlation across the channel parameters.
Sparse Tiling through Overlap Closures for Termination of String Rewriting
Venue: arXiv cs.LO 2020
Year: 2020

We over-approximate reachability sets in string rewriting by languages defined by admissible factors, called tiles. A sparse set of tiles contains only those that are reachable in derivations, and is constructed by completing an automaton. Using the partial algebra defined by a sparse tiling for semantic labelling, we obtain a transformational method for proving local termination. With a known result on forward closures, and a new characterisation of overlap closures, we obtain methods for proving termination and relative termination, respectively. We report on experiments showing the strength of these methods.
Dynamic Graph Operations: A Consistent Non-blocking Approach
Venue: arXiv cs.DC 2020
Year: 2020

Graph algorithms enormously contribute to the domains such as blockchains, social networks, biological networks, telecommunication networks, and several others. The ever-increasing demand of data-volume, as well as speed of such applications, have essentially transported these applications from their comfort zone: static setting, to a challenging territory of dynamic updates. At the same time, mainstreaming of multi-core processors have entailed that the dynamic applications should be able to exploit concurrency as soon as parallelization gets inhibited. Thus, the design and implementation of efficient concurrent dynamic graph algorithms have become significant. This paper reports a novel library of concurrent shared-memory algorithms for breadth-first search (BFS), single-source shortest-path (SSSP), and betweenness centrality (BC) in a dynamic graph. The presented algorithms are provably non-blocking and linearizable. We extensively evaluate C++ implementations of the algorithms through several micro-benchmarks. The experimental results demonstrate the scalability with the number of threads. Our experiments also highlight the limitations of static graph analytics methods in a dynamic setting.
Automatic Hyper-Parameter Optimization Based on Mapping Discovery from Data to Hyper-Parameters
Venue: arXiv cs.LG 2020
Year: 2020

Machine learning algorithms have made remarkable achievements in the field of artificial intelligence. However, most machine learning algorithms are sensitive to the hyper-parameters. Manually optimizing the hyper-parameters is a common method of hyper-parameter tuning. However, it is costly and empirically dependent. Automatic hyper-parameter optimization (autoHPO) is favored due to its effectiveness. However, current autoHPO methods are usually only effective for a certain type of problems, and the time cost is high. In this paper, we propose an efficient automatic parameter optimization approach, which is based on the mapping from data to the corresponding hyper-parameters. To describe such mapping, we propose a sophisticated network structure. To obtain such mapping, we develop effective network constrution algorithms. We also design strategy to optimize the result futher during the application of the mapping. Extensive experimental results demonstrate that the proposed approaches outperform the state-of-the-art apporaches significantly.
On Emergent Communication in Competitive Multi-Agent Teams
Venue: arXiv cs.AI 2020
Year: 2020

Several recent works have found the emergence of grounded compositional language in the communication protocols developed by mostly cooperative multi-agent systems when learned end-to-end to maximize performance on a downstream task. However, human populations learn to solve complex tasks involving communicative behaviors not only in fully cooperative settings but also in scenarios where competition acts as an additional external pressure for improvement. In this work, we investigate whether competition for performance from an external, similar agent team could act as a social influence that encourages multi-agent populations to develop better communication protocols for improved performance, compositionality, and convergence speed. We start from Task & Talk, a previously proposed referential game between two cooperative agents as our testbed and extend it into Task, Talk & Compete, a game involving two competitive teams each consisting of two aforementioned cooperative agents. Using this new setting, we provide an empirical study demonstrating the impact of competitive influence on multi-agent teams. Our results show that an external competitive influence leads to improved accuracy and generalization, as well as faster emergence of communicative languages that are more informative and compositional.
SeMemNN: A Semantic Matrix-Based Memory Neural Network for Text Classification
Venue: arXiv cs.CL 2020
Year: 2020

Text categorization is the task of assigning labels to documents written in a natural language, and it has numerous real-world applications including sentiment analysis as well as traditional topic assignment tasks. In this paper, we propose 5 different configurations for the semantic matrix-based memory neural network with end-to-end learning manner and evaluate our proposed method on two corpora of news articles (AG news, Sogou news). The best performance of our proposed method outperforms the baseline VDCNN models on the text classification task and gives a faster speed for learning semantics. Moreover, we also evaluate our model on small scale datasets. The results show that our proposed method can still achieve better results in comparison to VDCNN on the small scale dataset. This paper is to appear in the Proceedings of the 2020 IEEE 14th International Conference on Semantic Computing (ICSC 2020), San Diego, California, 2020.
Privacy-preserving Learning via Deep Net Pruning
Venue: arXiv cs.LG 2020
Year: 2020

This paper attempts to answer the question whether neural network pruning can be used as a tool to achieve differential privacy without losing much data utility. As a first step towards understanding the relationship between neural network pruning and differential privacy, this paper proves that pruning a given layer of the neural network is equivalent to adding a certain amount of differentially private noise to its hidden-layer activations. The paper also presents experimental results to show the practical implications of the theoretical finding and the key parameter values in a simple practical setting. These results show that neural network pruning can be a more effective alternative to adding differentially private noise for neural networks.
Ising-based Consensus Clustering on Specialized Hardware
Venue: arXiv cs.LG 2020
Year: 2020

The emergence of specialized optimization hardware such as CMOS annealers and adiabatic quantum computers carries the promise of solving hard combinatorial optimization problems more efficiently in hardware. Recent work has focused on formulating different combinatorial optimization problems as Ising models, the core mathematical abstraction used by a large number of these hardware platforms, and evaluating the performance of these models when solved on specialized hardware. An interesting area of application is data mining, where combinatorial optimization problems underlie many core tasks. In this work, we focus on consensus clustering (clustering aggregation), an important combinatorial problem that has received much attention over the last two decades. We present two Ising models for consensus clustering and evaluate them using the Fujitsu Digital Annealer, a quantum-inspired CMOS annealer. Our empirical evaluation shows that our approach outperforms existing techniques and is a promising direction for future research.
MoVi: A Large Multipurpose Motion and Video Dataset
Venue: arXiv cs.CV 2020
Year: 2020

Human movements are both an area of intense study and the basis of many applications such as character animation. For many applications, it is crucial to identify movements from videos or analyze datasets of movements. Here we introduce a new human Motion and Video dataset MoVi, which we make available publicly. It contains 60 female and 30 male actors performing a collection of 20 predefined everyday actions and sports movements, and one self-chosen movement. In five capture rounds, the same actors and movements were recorded using different hardware systems, including an optical motion capture system, video cameras, and inertial measurement units (IMU). For some of the capture rounds, the actors were recorded when wearing natural clothing, for the other rounds they wore minimal clothing. In total, our dataset contains 9 hours of motion capture data, 17 hours of video data from 4 different points of view (including one hand-held camera), and 6.6 hours of IMU data. In this paper, we describe how the dataset was collected and post-processed; We present state-of-the-art estimates of skeletal motions and full-body shape deformations associated with skeletal motion. We discuss examples for potential studies this dataset could enable.
Adaptive Online Distributed Optimal Control of Very-Large-Scale Robotic Systems
Venue: arXiv cs.MA 2020
Year: 2020

This paper presents an adaptive online distributed optimal control approach that is applicable to optimal planning for very-large-scale robotics systems in highly uncertain environments. This approach is developed based on the optimal mass transport theory. It is also viewed as an online reinforcement learning and approximate dynamic programming approach in the Wasserstein-GMM space, where a novel value functional is defined based on the probability density functions of robots and the time-varying obstacle map functions describing the changing environmental information. The proposed approach is demonstrated on the path planning problem of very-largescale robotic systems where the approximated layout of obstacles in the workspace is incrementally updated by the observations of robots, and compared with some existing state-of-the-art approaches. The numerical simulation results show that the proposed approach outperforms these approaches in aspects of the average traveling distance and the energy cost.
Conic Formation in Presence of Faulty Robots
Venue: arXiv cs.DC 2020
Year: 2020

Pattern formation is one of the most fundamental problems in distributed computing, which has recently received much attention. In this paper, we initiate the study of distributed pattern formation in situations when some robots can be \textit{faulty}. In particular, we consider the well-established \textit{look-compute-move} model with oblivious, anonymous robots. We first present lower bounds and show that any deterministic algorithm takes at least two rounds to form simple patterns in the presence of faulty robots. We then present distributed algorithms for our problem which match this bound, \textit{for conic sections}: in at most two rounds, robots form lines, circles and parabola tolerating $f=2,3$ and $4$ faults, respectively. For $f=5$, the target patterns are parabola, hyperbola and ellipse. We show that the resulting pattern includes the $f$ faulty robots in the pattern of $n$ robots, where $n \geq 2f+1$, and that $f < n < 2f+1$ robots cannot form such patterns. We conclude by discussing several relaxations and extensions.
Gaussianization Flows
Venue: arXiv cs.LG 2020
Year: 2020

Iterative Gaussianization is a fixed-point iteration procedure that can transform any continuous random vector into a Gaussian one. Based on iterative Gaussianization, we propose a new type of normalizing flow model that enables both efficient computation of likelihoods and efficient inversion for sample generation. We demonstrate that these models, named Gaussianization flows, are universal approximators for continuous probability distributions under some regularity conditions. Because of this guaranteed expressivity, they can capture multimodal target distributions without compromising the efficiency of sample generation. Experimentally, we show that Gaussianization flows achieve better or comparable performance on several tabular datasets compared to other efficiently invertible flow models such as Real NVP, Glow and FFJORD. In particular, Gaussianization flows are easier to initialize, demonstrate better robustness with respect to different transformations of the training data, and generalize better on small training sets.