OpenRaiser/Intern-Atlas huggingface dataset:
Bandits in Matching Markets: Ideas and Proposals for Peer Lending
Venue: arXiv cs.GT 2023
Year: 2023

Motivated by recent applications of sequential decision making in matching markets, in this paper we attempt at formulating and abstracting market designs for P2P lending. We describe a paradigm to set the stage for how peer to peer investments can be conceived from a matching market perspective, especially when both borrower and lender preferences are respected. We model these specialized markets as an optimization problem and consider different utilities for agents on both sides of the market while also understanding the impact of equitable allocations to borrowers. We devise a technique based on sequential decision making that allow the lenders to adjust their choices based on the dynamics of uncertainty from competition over time and that also impacts the rewards in return for their investments. Using simulated experiments we show the dynamics of the regret based on the optimal borrower-lender matching and find that the lender regret depends on the initial preferences set by the lenders which could affect their learning over decision making steps.
Estimating Total Correlation with Mutual Information Estimators
Venue: arXiv cs.IT 2023
Year: 2023

Total correlation (TC) is a fundamental concept in information theory that measures statistical dependency among multiple random variables. Recently, TC has shown noticeable effectiveness as a regularizer in many learning tasks, where the correlation among multiple latent embeddings requires to be jointly minimized or maximized. However, calculating precise TC values is challenging, especially when the closed-form distributions of embedding variables are unknown. In this paper, we introduce a unified framework to estimate total correlation values with sample-based mutual information (MI) estimators. More specifically, we discover a relation between TC and MI and propose two types of calculation paths (tree-like and line-like) to decompose TC into MI terms. With each MI term being bounded, the TC values can be successfully estimated. Further, we provide theoretical analyses concerning the statistical consistency of the proposed TC estimators. Experiments are presented on both synthetic and real-world scenarios, where our estimators demonstrate effectiveness in all TC estimation, minimization, and maximization tasks. The code is available at https://github.com/Linear95/TC-estimation.
ShadowNet: A Secure and Efficient On-device Model Inference System for Convolutional Neural Networks
Venue: arXiv cs.CR 2023
Year: 2023

With the increased usage of AI accelerators on mobile and edge devices, on-device machine learning (ML) is gaining popularity. Thousands of proprietary ML models are being deployed today on billions of untrusted devices. This raises serious security concerns about model privacy. However, protecting model privacy without losing access to the untrusted AI accelerators is a challenging problem. In this paper, we present a novel on-device model inference system, ShadowNet. ShadowNet protects the model privacy with Trusted Execution Environment (TEE) while securely outsourcing the heavy linear layers of the model to the untrusted hardware accelerators. ShadowNet achieves this by transforming the weights of the linear layers before outsourcing them and restoring the results inside the TEE. The non-linear layers are also kept secure inside the TEE. ShadowNet's design ensures efficient transformation of the weights and the subsequent restoration of the results. We build a ShadowNet prototype based on TensorFlow Lite and evaluate it on five popular CNNs, namely, MobileNet, ResNet-44, MiniVGG, ResNet-404, and YOLOv4-tiny. Our evaluation shows that ShadowNet achieves strong security guarantees with reasonable performance, offering a practical solution for secure on-device model inference.
Robust Quadruped Jumping via Deep Reinforcement Learning
Venue: arXiv cs.RO 2023
Year: 2023

In this paper, we consider a general task of jumping varying distances and heights for a quadrupedal robot in noisy environments, such as off of uneven terrain and with variable robot dynamics parameters. To accurately jump in such conditions, we propose a framework using deep reinforcement learning that leverages and augments the complex solution of nonlinear trajectory optimization for quadrupedal jumping. While the standalone optimization limits jumping to take-off from flat ground and requires accurate assumptions of robot dynamics, our proposed approach improves the robustness to allow jumping off of significantly uneven terrain with variable robot dynamical parameters and environmental conditions. Compared with walking and running, the realization of aggressive jumping on hardware necessitates accounting for the motors' torque-speed relationship as well as the robot's total power limits. By incorporating these constraints into our learning framework, we successfully deploy our policy sim-to-real without further tuning, fully exploiting the available onboard power supply and motors. We demonstrate robustness to environment noise of foot disturbances of up to 6 cm in height, or 33% of the robot's nominal standing height, while jumping 2x the body length in distance.
Deep Learning for Joint Channel Estimation and Feedback in Massive MIMO Systems
Venue: arXiv cs.IT 2023
Year: 2023

The great potentials of massive Multiple-Input Multiple-Output (MIMO) in Frequency Division Duplex (FDD) mode can be fully exploited when the downlink Channel State Information (CSI) is available at base stations. However, the accurate CSI is difficult to obtain due to the large amount of feedback overhead caused by massive antennas. In this paper, we propose a deep learning based joint channel estimation and feedback framework, which comprehensively realizes the estimation, compression, and reconstruction of downlink channels in FDD massive MIMO systems. Two networks are constructed to perform estimation and feedback explicitly and implicitly. The explicit network adopts a multi-Signal-to-Noise-Ratios (SNRs) technique to obtain a single trained channel estimation subnet that works well with different SNRs and employs a deep residual network to reconstruct the channels, while the implicit network directly compresses pilots and sends them back to reduce network parameters. Quantization module is also designed to generate data-bearing bitstreams. Simulation results show that the two proposed networks exhibit excellent performance of reconstruction and are robust to different environments and quantization errors.
Predictions For Pre-training Language Models
Venue: arXiv cs.CL 2023
Year: 2023

Language model pre-training has proven to be useful in many language understanding tasks. In this paper, we investigate whether it is still helpful to add the self-training method in the pre-training step and the fine-tuning step. Towards this goal, we propose a learning framework that making best use of the unlabel data on the low-resource and high-resource labeled dataset. In industry NLP applications, we have large amounts of data produced by users or customers. Our learning framework is based on this large amounts of unlabel data. First, We use the model fine-tuned on manually labeled dataset to predict pseudo labels for the user-generated unlabeled data. Then we use the pseudo labels to supervise the task-specific training on the large amounts of user-generated data. We consider this task-specific training step on pseudo labels as a pre-training step for the next fine-tuning step. At last, we fine-tune on the manually labeled dataset upon the pre-trained model. In this work, we first empirically show that our method is able to solidly improve the performance by 3.6%, when the manually labeled fine-tuning dataset is relatively small. Then we also show that our method still is able to improve the performance further by 0.2%, when the manually labeled fine-tuning dataset is relatively large enough. We argue that our method make the best use of the unlabel data, which is superior to either pre-training or self-training alone.
EWareNet: Emotion Aware Human Intent Prediction and Adaptive Spatial Profile Fusion for Social Robot Navigation
Venue: arXiv cs.RO 2023
Year: 2023

We present EWareNet, a novel intent and affect-aware social robot navigation algorithm among pedestrians. Our approach predicts the trajectory-based pedestrian intent from gait sequence, which is then used for intent-guided navigation taking into account social and proxemic constraints. We propose a transformer-based model that works on commodity RGB-D cameras mounted onto a moving robot. Our intent prediction routine is integrated into a mapless navigation scheme and makes no assumptions about the environment of pedestrian motion. Our navigation scheme consists of a novel obstacle profile representation methodology that is dynamically adjusted based on the pedestrian pose, intent, and affect. The navigation scheme is based on a reinforcement learning algorithm that takes pedestrian intent and robot's impact on pedestrian intent into consideration, in addition to the environmental configuration. We outperform current state-of-art algorithms for intent prediction from 3D gaits.
Comb inequalities for typical Euclidean TSP instances
Venue: arXiv cs.DS 2023
Year: 2023

We prove that even in average case, the Euclidean Traveling Salesman Problem exhibits an integrality gap of $(1+ε)$ for $ε>0$ when the Held-Karp Linear Programming relaxation is augmented by all comb inequalities of bounded size. This implies that large classes of branch-and-cut algorithms take exponential time for the Euclidean TSP, even on random inputs.
EBFT: Simplifying BFT Consensus Through Egalitarianism
Venue: arXiv cs.DC 2023
Year: 2023

We present Egalitarian BFT (EBFT), a simple and high-performance framework of BFT consensus protocols for decentralized systems like blockchains. The key innovation in EBFT is egalitarian block generation: nodes randomly and non-interactively propose blocks containing client transactions, rather than relying on a leader to do so. Apart from deterministic safety and liveness guarantees standard in BFT protocols, the egalitarian design provides two novel features: (i) EBFT is resilient to attacks targeting the leader, such as bribery and targeted DoS attacks, and (ii) EBFT does not require any fail-over protocol to detect and replace the faulty leader. EBFT consists of three protocols: EBFT-Syn for synchronous networks, EBFT-PSyn for partially synchronous networks, and EBFT-Turbo that builds on EBFT for high performance. We implement EBFT and evaluate its performance on AWS. To compare EBFT with state-of-the-art BFT protocols, we build EBFT-PSyn based on Bamboo, an open-source platform for prototyping partially synchronous BFT protocols. We evaluate EBFT-PSyn and HotStuff on EC2 with up to 16 nodes. The evaluation shows that EBFT-PSyn achieves better throughput and latency than HotStuff. To demonstrate its simplicity and practicality, we build EBFT on the Go version of Bitcoin, btcd. We implemented EBFT-Syn, EBFT-PSyn and EBFT-Turbo in about 920 LoCs in total. This indicates that EBFT can be built on top of existing blockchains with relatively little effort. We evaluate these protocols on EC2 instances with up to 256 nodes. Our evaluation shows that EBFT-Syn (resp. EBFT-PSyn) achieves a latency of 6 (resp. 1) seconds, and an optimized version of EBFT-PSyn processes up to 3.6k transactions per second and has a latency of 8 seconds.
Benchmarking Automated Clinical Language Simplification: Dataset, Algorithm, and Evaluation
Venue: arXiv cs.CL 2023
Year: 2023

Patients with low health literacy usually have difficulty understanding medical jargon and the complex structure of professional medical language. Although some studies are proposed to automatically translate expert language into layperson-understandable language, only a few of them focus on both accuracy and readability aspects simultaneously in the clinical domain. Thus, simplification of the clinical language is still a challenging task, but unfortunately, it is not yet fully addressed in previous work. To benchmark this task, we construct a new dataset named MedLane to support the development and evaluation of automated clinical language simplification approaches. Besides, we propose a new model called DECLARE that follows the human annotation procedure and achieves state-of-the-art performance compared with eight strong baselines. To fairly evaluate the performance, we also propose three specific evaluation metrics. Experimental results demonstrate the utility of the annotated MedLane dataset and the effectiveness of the proposed model DECLARE.
What Makes a "Good" Data Augmentation in Knowledge Distillation -- A Statistical Perspective
Venue: arXiv cs.CV 2023
Year: 2023

Knowledge distillation (KD) is a general neural network training approach that uses a teacher model to guide the student model. Existing works mainly study KD from the network output side (e.g., trying to design a better KD loss function), while few have attempted to understand it from the input side. Especially, its interplay with data augmentation (DA) has not been well understood. In this paper, we ask: Why do some DA schemes (e.g., CutMix) inherently perform much better than others in KD? What makes a "good" DA in KD? Our investigation from a statistical perspective suggests that a good DA scheme should reduce the covariance of the teacher-student cross-entropy. A practical metric, the stddev of teacher's mean probability (T. stddev), is further presented and well justified empirically. Besides the theoretical understanding, we also introduce a new entropy-based data-mixing DA scheme, CutMixPick, to further enhance CutMix. Extensive empirical studies support our claims and demonstrate how we can harvest considerable performance gains simply by using a better DA scheme in knowledge distillation.
Galloping in fast-growth natural merge sorts
Venue: arXiv cs.DS 2023
Year: 2023

We study the impact of merging routines in merge-based sorting algorithms. More precisely, we focus on the galloping routine that TimSort uses to merge monotonic sub-arrays, hereafter called runs, and on the impact on the number of element comparisons performed if one uses this routine instead of a naïve merging routine. This routine was introduced in order to make TimSort more efficient on arrays with few distinct values. Alas, we prove that, although it makes TimSort sort array with two values in linear time, it does not prevent TimSort from requiring up to $Θ(n \log(n))$ element comparisons to sort arrays of length~$n$ with three distinct values. However, we also prove that slightly modifying TimSort's galloping routine results in requiring only $\mathcal{O}(n + n \log(σ))$ element comparisons in the worst case, when sorting arrays of length $n$ with $σ$ distinct values. We do so by focusing on the notion of dual runs, which was introduced in the 1990s, and on the associated dual run-length entropy. This notion is both related to the number of distinct values and to the number of runs in an array, which came with its own run-length entropy that was used to explain TimSort's otherwise "supernatural" efficiency. We also introduce new notions of fast- and middle-growth for natural merge sorts (i.e., algorithms based on merging runs), which are found in several merge sorting algorithms similar to TimSort. We prove that algorithms with the fast- or middle-growth property, provided that they use our variant of TimSort's galloping routine for merging runs, are as efficient as possible at sorting arrays with low run-induced or dual-run-induced complexities.
A wireless signal-based sensing framework for robotics
Venue: arXiv cs.RO 2023
Year: 2023

In this paper we develop the analytical framework for a novel Wireless signal-based Sensing capability for Robotics (WSR) by leveraging robots' mobility. It allows robots to primarily measure relative direction, or Angle-of-Arrival (AOA), to other robots, while operating in non-line-of-sight unmapped environments and without requiring external infrastructure. We do so by capturing all of the paths that a wireless signal traverses as it travels from a transmitting to a receiving robot in the team, which we term as an AOA profile. The key intuition behind our approach is to enable a robot to emulate antenna arrays as it moves freely in 2D and 3D space. The small differences in the phase of the wireless signals are thus processed with knowledge of robots' local displacement to obtain the profile, via a method akin to Synthetic Aperture Radar (SAR). The main contribution of this work is the development of i) a framework to accommodate arbitrary 2D and 3D motion, as well as continuous mobility of both signal transmitting and receiving robots, while computing AOA profiles between them and ii) a Cramer-Rao Bound analysis, based on antenna array theory, that provides a lower bound on the variance in AOA estimation as a function of the geometry of robot motion. We show that allowing robots to use their full mobility in 3D space while performing SAR, results in more accurate AOA profiles and thus better AOA estimation. All analytical developments are substantiated by extensive simulation and hardware experiments on air/ground robot platforms using 5 GHz WiFi. Our experimental results bolster our analytical findings, demonstrating that 3D motion provides enhanced and consistent accuracy, with total AOA error of less than 10 degree for 95% of trials. We also analytically characterize the impact of displacement estimation errors on the measured AOA.
Intrinsically Motivated Compositional Language Emergence
Venue: arXiv cs.CL 2023
Year: 2023

Recently, there has been a great deal of research in emergent communication on artificial agents interacting in simulated environments. Recent studies have revealed that, in general, emergent languages do not follow the compositionality patterns of natural language. To deal with this, existing works have proposed a limited channel capacity as an important constraint for learning highly compositional languages. In this paper, we show that this is not a sufficient condition and propose an intrinsic reward framework for improving compositionality in emergent communication. We use a reinforcement learning setting with two agents -- a \textit{task-aware} Speaker and a \textit{state-aware} Listener that are required to communicate to perform a set of tasks. Through our experiments on three different referential game setups, including a novel environment gComm, we show intrinsic rewards improve compositionality scores by $\approx \mathbf{1.5-2}$ times that of existing frameworks that use limited channel capacity.
Discriminating Between Similar Nordic Languages
Venue: arXiv cs.CL 2023
Year: 2023

Automatic language identification is a challenging problem. Discriminating between closely related languages is especially difficult. This paper presents a machine learning approach for automatic language identification for the Nordic languages, which often suffer miscategorisation by existing state-of-the-art tools. Concretely we will focus on discrimination between six Nordic languages: Danish, Swedish, Norwegian (Nynorsk), Norwegian (Bokmål), Faroese and Icelandic.
Consequences of Slow Neural Dynamics for Incremental Learning
Venue: arXiv cs.LG 2023
Year: 2023

In the human brain, internal states are often correlated over time (due to local recurrence and other intrinsic circuit properties), punctuated by abrupt transitions. At first glance, temporal smoothness of internal states presents a problem for learning input-output mappings (e.g. category labels for images), because the internal representation of the input will contain a mixture of current input and prior inputs. However, when training with naturalistic data (e.g. movies) there is also temporal autocorrelation in the input. How does the temporal "smoothness" of internal states affect the efficiency of learning when the training data are also temporally smooth? How does it affect the kinds of representations that are learned? We found that, when trained with temporally smooth data, "slow" neural networks (equipped with linear recurrence and gating mechanisms) learned to categorize more efficiently than feedforward networks. Furthermore, networks with linear recurrence and multi-timescale gating could learn internal representations that "un-mixed" quickly-varying and slowly-varying data sources. Together, these findings demonstrate how a fundamental property of cortical dynamics (their temporal autocorrelation) can serve as an inductive bias, leading to more efficient category learning and to the representational separation of fast and slow sources in the environment.
Bayesian graph neural networks for strain-based crack localization
Venue: arXiv cs.CE 2023
Year: 2023

A common shortcoming of vibration-based damage localization techniques is that localized damages, i.e. small cracks, have a limited influence on the spectral characteristics of a structure. In contrast, even the smallest of defects, under particular loading conditions, cause localized strain concentrations with predictable spatial configuration. However, the effect of a small defect on strain decays quickly with distance from the defect, making strain-based localization rather challenging. In this work, an attempt is made to approximate, in a fully data-driven manner, the posterior distribution of a crack location, given arbitrary dynamic strain measurements at arbitrary discrete locations on a structure. The proposed technique leverages Graph Neural Networks (GNNs) and recent developments in scalable learning for Bayesian neural networks. The technique is demonstrated on the problem of inferring the position of an unknown crack via patterns of dynamic strain field measurements at discrete locations. The dataset consists of simulations of a hollow tube under random time-dependent excitations with randomly sampled crack geometry and orientation.
Computation Against a Neighbour: Addressing Large-Scale Distribution and Adaptivity with Functional Programming and Scala
Venue: arXiv cs.DC 2023
Year: 2023

Recent works in contexts like the Internet of Things (IoT) and large-scale Cyber-Physical Systems (CPS) propose the idea of programming distributed systems by focussing on their global behaviour across space and time. In this view, a potentially vast and heterogeneous set of devices is considered as an "aggregate" to be programmed as a whole, while abstracting away the details of individual behaviour and exchange of messages, which are expressed declaratively. One such a paradigm, known as aggregate programming, builds on computational models inspired by field-based coordination. Existing models such as the field calculus capture interaction with neighbours by a so-called "neighbouring field" (a map from neighbours to values). This requires ad-hoc mechanisms to smoothly compose with standard values, thus complicating programming and introducing clutter in aggregate programs, libraries and domain-specific languages (DSLs). To address this key issue we introduce the novel notion of "computation against a neighbour", whereby the evaluation of certain subexpressions of the aggregate program are affected by recent corresponding evaluations in neighbours. We capture this notion in the neighbours calculus (NC), a new field calculus variant which is shown to smoothly support declarative specification of interaction with neighbours, and correspondingly facilitate the embedding of field computations as internal DSLs in common general-purpose programming languages -- as exemplified by a Scala implementation, called ScaFi. This paper formalises NC, thoroughly compares it with respect to the classic field calculus, and shows its expressiveness by means of a case study in edge computing, developed in ScaFi.