OpenRaiser/Intern-Atlas huggingface dataset:
Benchmarking machine learning models on multi-centre eICU critical care dataset
Venue: arXiv cs.LG 2021
Year: 2021

Progress of machine learning in critical care has been difficult to track, in part due to absence of public benchmarks. Other fields of research (such as computer vision and natural language processing) have established various competitions and public benchmarks. Recent availability of large clinical datasets has enabled the possibility of establishing public benchmarks. Taking advantage of this opportunity, we propose a public benchmark suite to address four areas of critical care, namely mortality prediction, estimation of length of stay, patient phenotyping and risk of decompensation. We define each task and compare the performance of both clinical models as well as baseline and deep learning models using eICU critical care dataset of around 73,000 patients. This is the first public benchmark on a multi-centre critical care dataset, comparing the performance of clinical gold standard with our predictive model. We also investigate the impact of numerical variables as well as handling of categorical variables on each of the defined tasks. The source code, detailing our methods and experiments is publicly available such that anyone can replicate our results and build upon our work.
State Representation Learning from Demonstration
Venue: arXiv cs.LG 2021
Year: 2021

Robots could learn their own state and world representation from perception and experience without supervision. This desirable goal is the main focus of our field of interest, state representation learning (SRL). Indeed, a compact representation of such a state is beneficial to help robots grasp onto their environment for interacting. The properties of this representation have a strong impact on the adaptive capability of the agent. In this article we present an approach based on imitation learning. The idea is to train several policies that share the same representation to reproduce various demonstrations. To do so, we use a multi-head neural network with a shared state representation feeding a task-specific agent. If the demonstrations are diverse, the trained representation will eventually contain the information necessary for all tasks, while discarding irrelevant information. As such, it will potentially become a compact state representation useful for new tasks. We call this approach SRLfD (State Representation Learning from Demonstration). Our experiments confirm that when a controller takes SRLfD-based representations as input, it can achieve better performance than with other representation strategies and promote more efficient reinforcement learning (RL) than with an end-to-end RL strategy.
Towards a Definitive Compressibility Measure for Repetitive Sequences
Venue: arXiv cs.DS 2021
Year: 2021

Unlike in statistical compression, where Shannon's entropy is a definitive lower bound, no such clear measure exists for the compressibility of repetitive sequences. Since statistical entropy does not capture repetitiveness, ad-hoc measures like the size $z$ of the Lempel--Ziv parse are frequently used to estimate it. The size $b \le z$ of the smallest bidirectional macro scheme captures better what can be achieved via copy-paste processes, though it is NP-complete to compute and it is not monotonic upon symbol appends. Recently, a more principled measure, the size $γ$ of the smallest string \emph{attractor}, was introduced. The measure $γ\le b$ lower bounds all the previous relevant ones, yet length-$n$ strings can be represented and efficiently indexed within space $O(γ\log\frac{n}γ)$, which also upper bounds most measures. While $γ$ is certainly a better measure of repetitiveness than $b$, it is also NP-complete to compute and not monotonic, and it is unknown if one can always represent a string in $o(γ\log n)$ space. In this paper, we study an even smaller measure, $δ\le γ$, which can be computed in linear time, is monotonic, and allows encoding every string in $O(δ\log\frac{n}δ)$ space because $z = O(δ\log\frac{n}δ)$. We show that $δ$ better captures the compressibility of repetitive strings. Concretely, we show that (1) $δ$ can be strictly smaller than $γ$, by up to a logarithmic factor; (2) there are string families needing $Ω(δ\log\frac{n}δ)$ space to be encoded, so this space is optimal for every $n$ and $δ$; (3) one can build run-length context-free grammars of size $O(δ\log\frac{n}δ)$, whereas the smallest (non-run-length) grammar can be up to $Θ(\log n/\log\log n)$ times larger; and (4) within $O(δ\log\frac{n}δ)$ space we can not only...
The CEO problem with inter-block memory
Venue: arXiv cs.IT 2021
Year: 2021

An $n$-dimensional source with memory is observed by $K$ isolated encoders via parallel channels, who compress their observations to transmit to the decoder via noiseless rate-constrained links while leveraging their memory of the past. At each time instant, the decoder receives $K$ new codewords from the observers, combines them with the past received codewords, and produces a minimum-distortion estimate of the latest block of $n$ source symbols. This scenario extends the classical one-shot CEO problem to multiple rounds of communication with communicators maintaining the memory of the past. We extend the Berger-Tung inner and outer bounds to the scenario with inter-block memory, showing that the minimum asymptotically (as $n \to \infty$) achievable sum rate required to achieve a target distortion is bounded by minimal directed mutual information problems. For the Gauss-Markov source observed via $K$ parallel AWGN channels, we show that the inner bound is tight and solve the corresponding minimal directed mutual information problem, thereby establishing the minimum asymptotically achievable sum rate. Finally, we explicitly bound the rate loss due to a lack of communication among the observers; that bound is attained with equality in the case of identical observation channels. The general coding theorem is proved via a new nonasymptotic bound that uses stochastic likelihood coders and whose asymptotic analysis yields an extension of the Berger-Tung inner bound to the causal setting. The analysis of the Gaussian case is facilitated by reversing the channels of the observers.
Sample Elicitation
Venue: arXiv cs.LG 2021
Year: 2021

It is important to collect credible training samples $(x,y)$ for building data-intensive learning systems (e.g., a deep learning system). Asking people to report complex distribution $p(x)$, though theoretically viable, is challenging in practice. This is primarily due to the cognitive loads required for human agents to form the report of this highly complicated information. While classical elicitation mechanisms apply to eliciting a complex and generative (and continuous) distribution $p(x)$, we are interested in eliciting samples $x_i \sim p(x)$ from agents directly. We coin the above problem "sample elicitation". This paper introduces a deep learning aided method to incentivize credible sample contributions from self-interested and rational agents. We show that with an accurate estimation of a certain $f$-divergence function we can achieve approximate incentive compatibility in eliciting truthful samples. We then present an efficient estimator with theoretical guarantees via studying the variational forms of the $f$-divergence function. We also show a connection between this sample elicitation problem and $f$-GAN, and how this connection can help reconstruct an estimator of the distribution based on collected samples. Experiments on synthetic data, MNIST, and CIFAR-10 datasets demonstrate that our mechanism elicits truthful samples. Our implementation is available at https://github.com/weijiaheng/Credible-sample-elicitation.git.
Concealing-Restoring System for Physical Layer Data Based on Stochastic Filtering Theory
Venue: arXiv cs.IT 2021
Year: 2021

We propose a concealing-restoring system (CRS) for data on physical layer of the OSI reference model. CRS conceals those data by disturbing them with some random noises, and restores the data from the concealed ones to the original ones by using the noise elimination based on a proper stochastic filtering theory. Although we introduced the outline of the almost linear version of CRS in our previous work [Fujii & Hirokawa, Math. Industry, Springer (2020)], we explain its details, and study its nonlinearization to improve the security of CRS in this paper.
MVFST-RL: An Asynchronous RL Framework for Congestion Control with Delayed Actions
Venue: arXiv cs.LG 2021
Year: 2021

Effective network congestion control strategies are key to keeping the Internet (or any large computer network) operational. Network congestion control has been dominated by hand-crafted heuristics for decades. Recently, ReinforcementLearning (RL) has emerged as an alternative to automatically optimize such control strategies. Research so far has primarily considered RL interfaces which block the sender while an agent considers its next action. This is largely an artifact of building on top of frameworks designed for RL in games (e.g. OpenAI Gym). However, this does not translate to real-world networking environments, where a network sender waiting on a policy without sending data leads to under-utilization of bandwidth. We instead propose to formulate congestion control with an asynchronous RL agent that handles delayed actions. We present MVFST-RL, a scalable framework for congestion control in the QUIC transport protocol that leverages state-of-the-art in asynchronous RL training with off-policy correction. We analyze modeling improvements to mitigate the deviation from Markovian dynamics, and evaluate our method on emulated networks from the Pantheon benchmark platform. The source code is publicly available at https://github.com/facebookresearch/mvfst-rl.
Structured Pruning of Large Language Models
Venue: arXiv cs.CL 2021
Year: 2021

Large language models have recently achieved state of the art performance across a wide variety of natural language tasks. Meanwhile, the size of these models and their latency have significantly increased, which makes their usage costly, and raises an interesting question: do language models need to be large? We study this question through the lens of model compression. We present a generic, structured pruning approach by parameterizing each weight matrix using its low-rank factorization, and adaptively removing rank-1 components during training. On language modeling tasks, our structured approach outperforms other unstructured and block-structured pruning baselines at various compression levels, while achieving significant speedups during both training and inference. We also demonstrate that our method can be applied to pruning adaptive word embeddings in large language models, and to pruning the BERT model on several downstream fine-tuning classification benchmarks.
Interaction Relational Network for Mutual Action Recognition
Venue: arXiv cs.CV 2021
Year: 2021

Person-person mutual action recognition (also referred to as interaction recognition) is an important research branch of human activity analysis. Current solutions in the field -- mainly dominated by CNNs, GCNs and LSTMs -- often consist of complicated architectures and mechanisms to embed the relationships between the two persons on the architecture itself, to ensure the interaction patterns can be properly learned. Our main contribution with this work is by proposing a simpler yet very powerful architecture, named Interaction Relational Network, which utilizes minimal prior knowledge about the structure of the human body. We drive the network to identify by itself how to relate the body parts from the individuals interacting. In order to better represent the interaction, we define two different relationships, leading to specialized architectures and models for each. These multiple relationship models will then be fused into a single and special architecture, in order to leverage both streams of information for further enhancing the relational reasoning capability. Furthermore we define important structured pair-wise operations to extract meaningful extra information from each pair of joints -- distance and motion. Ultimately, with the coupling of an LSTM, our IRN is capable of paramount sequential relational reasoning. These important extensions we made to our network can also be valuable to other problems that require sophisticated relational reasoning. Our solution is able to achieve state-of-the-art performance on the traditional interaction recognition datasets SBU and UT, and also on the mutual actions from the large-scale dataset NTU RGB+D. Furthermore, it obtains competitive performance in the NTU RGB+D 120 dataset interactions subset.
L-Platooning: A Protocol for Managing a Long Platoon with DSRC
Venue: arXiv cs.NI 2021
Year: 2021

Vehicle platooning is an automated driving technology that enables a group of vehicles to travel very closely together as a single unit to improve fuel efficiency and driver safety and reduces CO2 emission. The significant benefits of platooning attracted huge interests from academia and industry, especially from logistics companies for utilizing platoons of "long-body" trailer trucks because of the huge cost savings. In this paper, we demonstrate that existing DSRC-based platooning solutions, however, fail to support formation of such "long" platoons consisting of typical trailer trucks because of the limited communication range of DSRC. To address this problem, we propose L-Platooning, the first platooning protocol that enables seamless, reliable, and rapid formation of a long platoon. We introduce a novel concept called Virtual Leader that refers to a vehicle that acts like a platoon leader to extend the coverage of the original platoon leader. A virtual leader election algorithm is developed to effectively designate a virtual leader based on the novel metric called the Virtual Leader Quality Index (VLQI) which quantifies the effectiveness of a vehicle serving as a platoon leader. We also develop mechanisms for L-Platooning to support the vehicle join and leave maneuvers specifically for a long platoon. Through extensive simulations using the combination of Veins (Plexe) and SUMO, we demonstrate that L-Platooning enables long-body trailer trucks to form a long platoon effectively and maintain the desired inter-vehicle distance precisely. We also show that L-Platooning handles seamlessly the vehicle join and leave maneuvers for a long platoon.
SiPPing Neural Networks: Sensitivity-informed Provable Pruning of Neural Networks
Venue: arXiv cs.LG 2021
Year: 2021

We introduce a pruning algorithm that provably sparsifies the parameters of a trained model in a way that approximately preserves the model's predictive accuracy. Our algorithm uses a small batch of input points to construct a data-informed importance sampling distribution over the network's parameters, and adaptively mixes a sampling-based and deterministic pruning procedure to discard redundant weights. Our pruning method is simultaneously computationally efficient, provably accurate, and broadly applicable to various network architectures and data distributions. Our empirical comparisons show that our algorithm reliably generates highly compressed networks that incur minimal loss in performance relative to that of the original network. We present experimental results that demonstrate our algorithm's potential to unearth essential network connections that can be trained successfully in isolation, which may be of independent interest.
Learning Lenient Parsing & Typing via Indirect Supervision
Venue: arXiv cs.SE 2021
Year: 2021

Both professional coders and teachers frequently deal with imperfect (fragmentary, incomplete, ill-formed) code. Such fragments are common in STACKOVERFLOW; students also frequently produce ill-formed code, for which instructors, TAs (or students themselves) must find repairs. In either case, the developer experience could be greatly improved if such code could somehow be parsed & typed; this makes such code more amenable to use within IDEs and allows early detection and repair of potential errors. We introduce a lenient parser, which can parse & type fragments, even ones with simple errors. Training a machine learner to leniently parse and type imperfect code requires a large training set including many pairs of imperfect code and its repair; such training sets are limited by human effort and curation. In this paper, we present a novel, indirectly supervised, approach to train a lenient parser, without access to such human-curated training data. We leverage the huge corpus of mostly correct code available on Github, and the massive, efficient learning capacity of Transformer-based NN architectures. Using GitHub data, we first create a large dataset of fragments of code and corresponding tree fragments and type annotations; we then randomly corrupt the input fragments by seeding errors that mimic corruptions found in STACKOVERFLOW and student data. Using this data, we train high-capacity transformer models to overcome both fragmentation and corruption. With this novel approach, we can achieve reasonable performance on parsing & typing STACKOVERFLOW fragments; we also demonstrate that our approach performs well on shorter student error program and achieves best-in-class performance on longer programs that have more than 400 tokens. We also show that by blending Deepfix and our tool, we could achieve 77% accuracy, which outperforms all previously reported student error correction tools.
Symmetric binary Steinhaus triangles and parity-regular Steinhaus graphs
Venue: arXiv cs.DM 2021
Year: 2021

A binary Steinhaus triangle is a triangle of zeroes and ones that points down and with the same local rule as the Pascal triangle modulo 2. A binary Steinhaus triangle is said to be rotationally symmetric, horizontally symmetric or dihedrally symmetric if it is invariant under the 120 degrees rotation, the horizontal reflection or both, respectively. The first part of this paper is devoted to the study of linear subspaces of rotationally symmetric, horizontally symmetric and dihedrally symmetric binary Steinhaus triangles. We obtain simple explicit bases for each of them by using elementary properties of the binomial coefficients. A Steinhaus graph is a simple graph with an adjacency matrix whose upper-triangular part is a binary Steinhaus triangle. A Steinhaus graph is said to be even or odd if all its vertex degrees are even or odd, respectively. One of the main results of this paper is the existence of an isomorphism between the linear subspace of even Steinhaus graphs and a certain linear subspace of dihedrally symmetric binary Steinhaus triangles. This permits us to give, in the second part of this paper, an explicit basis for even Steinhaus graphs and for the vector space of parity-regular Steinhaus graphs; i.e., the linear subspace of Steinhaus graphs that are even or odd. Finally, in the last part of this paper, we consider the generalized Pascal triangles, that are triangles of zeroes and ones, that point up now, and always with the same local rule as the Pascal triangle modulo 2. New simple bases for each linear subspace of symmetric generalized Pascal triangles are deduced from the results of the first part.
Topological Navigation Graph Framework
Venue: arXiv cs.RO 2021
Year: 2021

We focus on the utilisation of reactive trajectory imitation controllers for goal-directed mobile robot navigation. We propose a topological navigation graph (TNG) - an imitation-learning-based framework for navigating through environments with intersecting trajectories. The TNG framework represents the environment as a directed graph composed of deep neural networks. Each vertex of the graph corresponds to a trajectory and is represented by a trajectory identification classifier and a trajectory imitation controller. For trajectory following, we propose the novel use of neural object detection architectures. The edges of TNG correspond to intersections between trajectories and are all represented by a classifier. We provide empirical evaluation of the proposed navigation framework and its components in simulated and real-world environments, demonstrating that TNG allows us to utilise non-goal-directed, imitation-learning methods for goal-directed autonomous navigation.
Generic Bounds on the Maximum Deviations in Sequential Prediction: An Information-Theoretic Analysis
Venue: arXiv cs.LG 2021
Year: 2021

In this paper, we derive generic bounds on the maximum deviations in prediction errors for sequential prediction via an information-theoretic approach. The fundamental bounds are shown to depend only on the conditional entropy of the data point to be predicted given the previous data points. In the asymptotic case, the bounds are achieved if and only if the prediction error is white and uniformly distributed.
DeepGCNs: Making GCNs Go as Deep as CNNs
Venue: arXiv cs.CV 2021
Year: 2021

Convolutional Neural Networks (CNNs) have been very successful at solving a variety of computer vision tasks such as object classification and detection, semantic segmentation, activity understanding, to name just a few. One key enabling factor for their great performance has been the ability to train very deep networks. Despite their huge success in many tasks, CNNs do not work well with non-Euclidean data, which is prevalent in many real-world applications. Graph Convolutional Networks (GCNs) offer an alternative that allows for non-Eucledian data input to a neural network. While GCNs already achieve encouraging results, they are currently limited to architectures with a relatively small number of layers, primarily due to vanishing gradients during training. This work transfers concepts such as residual/dense connections and dilated convolutions from CNNs to GCNs in order to successfully train very deep GCNs. We show the benefit of using deep GCNs (with as many as 112 layers) experimentally across various datasets and tasks. Specifically, we achieve very promising performance in part segmentation and semantic segmentation on point clouds and in node classification of protein functions across biological protein-protein interaction (PPI) graphs. We believe that the insights in this work will open avenues for future research on GCNs and their application to further tasks not explored in this paper. The source code for this work is available at https://github.com/lightaime/deep_gcns_torch and https://github.com/lightaime/deep_gcns for PyTorch and TensorFlow implementation respectively.
CRISLoc: Reconstructable CSI Fingerprintingfor Indoor Smartphone Localization
Venue: arXiv cs.NI 2021
Year: 2021

Channel state information (CSI) based fingerprinting for WIFI indoor localization has attracted lots of attention very recently.The frequency diverse and temporally stable CSI better represents the location dependent channel characteristics than the coarsereceived signal strength (RSS). However, the acquisition of CSI requires the cooperation of access points (APs) and involves only dataframes, which imposes restrictions on real-world deployment. In this paper, we present CRISLoc, the first CSI fingerprinting basedlocalization prototype system using ubiquitous smartphones. CRISLoc operates in a completely passive mode, overhearing thepackets on-the-fly for his own CSI acquisition. The smartphone CSI is sanitized via calibrating the distortion enforced by WiFi amplifiercircuits. CRISLoc tackles the challenge of altered APs with a joint clustering and outlier detection method to find them. A novel transferlearning approach is proposed to reconstruct the high-dimensional CSI fingerprint database on the basis of the outdated fingerprintsand a few fresh measurements, and an enhanced KNN approach is proposed to pinpoint the location of a smartphone. Our studyreveals important properties about the stability and sensitivity of smartphone CSI that has not been reported previously. Experimentalresults show that CRISLoc can achieve a mean error of around 0.29m in a6m times 8mresearch laboratory. The mean error increases by 5.4 cm and 8.6 cm upon the movement of one and two APs, which validates the robustness of CRISLoc against environment changes.