OpenRaiser/Intern-Atlas huggingface dataset:
CLUE-AI: A Convolutional Three-stream Anomaly Identification Framework for Robot Manipulation
Venue: arXiv cs.RO 2022
Year: 2022

Robot safety has been a prominent research topic in recent years since robots are more involved in daily tasks. It is crucial to devise the required safety mechanisms to enable service robots to be aware of and react to anomalies (i.e., unexpected deviations from intended outcomes) that arise during the execution of these tasks. Detection and identification of these anomalies is an essential step towards fulfilling these requirements. Although several architectures are proposed for anomaly detection, identification is not yet thoroughly investigated. This task is challenging since indicators may appear long before anomalies are detected. In this paper, we propose a ConvoLUtional threE-stream Anomaly Identification (CLUE-AI) framework to address this problem. The framework fuses visual, auditory and proprioceptive data streams to identify everyday object manipulation anomalies. A stream of 2D images gathered through an RGB-D camera placed on the head of the robot is processed within a self-attention enabled visual stage to capture visual anomaly indicators. The auditory modality provided by the microphone placed on the robot's lower torso is processed within a designed convolutional neural network (CNN) in the auditory stage. Last, the force applied by the gripper and the gripper state are processed within a CNN to obtain proprioceptive features. These outputs are then combined with a late fusion scheme. Our novel three-stream framework design is analyzed on everyday object manipulation tasks with a Baxter humanoid robot in a semi-structured setting. The results indicate that the framework achieves an f-score of 94% outperforming the other baselines in classifying anomalies that arise during runtime.
PosePipe: Open-Source Human Pose Estimation Pipeline for Clinical Research
Venue: arXiv cs.CV 2022
Year: 2022

There has been significant progress in machine learning algorithms for human pose estimation that may provide immense value in rehabilitation and movement sciences. However, there remain several challenges to routine use of these tools for clinical practice and translational research, including: 1) high technical barrier to entry, 2) rapidly evolving space of algorithms, 3) challenging algorithmic interdependencies, and 4) complex data management requirements between these components. To mitigate these barriers, we developed a human pose estimation pipeline that facilitates running state-of-the-art algorithms on data acquired in clinical context. Our system allows for running different implementations of several classes of algorithms and handles their interdependencies easily. These algorithm classes include subject identification and tracking, 2D keypoint detection, 3D joint location estimation, and estimating the pose of body models. The system uses a database to manage videos, intermediate analyses, and data for computations at each stage. It also provides tools for data visualization, including generating video overlays that also obscure faces to enhance privacy. Our goal in this work is not to train new algorithms, but to advance the use of cutting-edge human pose estimation algorithms for clinical and translation research. We show that this tool facilitates analyzing large numbers of videos of human movement ranging from gait laboratories analyses, to clinic and therapy visits, to people in the community. We also highlight limitations of these algorithms when applied to clinical populations in a rehabilitation setting.
Pre-Trained Multilingual Sequence-to-Sequence Models: A Hope for Low-Resource Language Translation?
Venue: arXiv cs.CL 2022
Year: 2022

What can pre-trained multilingual sequence-to-sequence models like mBART contribute to translating low-resource languages? We conduct a thorough empirical experiment in 10 languages to ascertain this, considering five factors: (1) the amount of fine-tuning data, (2) the noise in the fine-tuning data, (3) the amount of pre-training data in the model, (4) the impact of domain mismatch, and (5) language typology. In addition to yielding several heuristics, the experiments form a framework for evaluating the data sensitivities of machine translation systems. While mBART is robust to domain differences, its translations for unseen and typologically distant languages remain below 3.0 BLEU. In answer to our title's question, mBART is not a low-resource panacea; we therefore encourage shifting the emphasis from new models to new data.
ORCA: A Network and Architecture Co-design for Offloading us-scale Datacenter Applications
Venue: arXiv cs.AR 2022
Year: 2022

Responding to the "datacenter tax" and "killer microseconds" problems for datacenter applications, diverse solutions including Smart NIC-based ones have been proposed. Nonetheless, they often suffer from high overhead of communications over network and/or PCIe links. To tackle the limitations of the current solutions, this paper proposes ORCA, a holistic network and architecture co-design solution that leverages current RDMA and emerging cache-coherent off-chip interconnect technologies. Specifically, ORCA consists of four hardware and software components: (1) unified abstraction of inter- and intra-machine communications managed by one-sided RDMA write and cache-coherent memory write; (2) efficient notification of requests to accelerators assisted by cache coherence; (3) cache-coherent accelerator architecture directly processing requests received by NIC; and (4) adaptive device-to-host data transfer for modern server memory systems consisting of both DRAM and NVM exploiting state-of-the-art features in CPUs and PCIe. We prototype ORCA with a commercial system and evaluate three popular datacenter applications: in-memory key-value store, chain replication-based distributed transaction system, and deep learning recommendation model inference. The evaluation shows that ORCA provides 30.1~69.1% lower latency, up to 2.5x higher throughput, and 3x higher power efficiency than the current state-of-the-art solutions.
Adversarial Support Alignment
Venue: arXiv cs.LG 2022
Year: 2022

We study the problem of aligning the supports of distributions. Compared to the existing work on distribution alignment, support alignment does not require the densities to be matched. We propose symmetric support difference as a divergence measure to quantify the mismatch between supports. We show that select discriminators (e.g. discriminator trained for Jensen-Shannon divergence) are able to map support differences as support differences in their one-dimensional output space. Following this result, our method aligns supports by minimizing a symmetrized relaxed optimal transport cost in the discriminator 1D space via an adversarial process. Furthermore, we show that our approach can be viewed as a limit of existing notions of alignment by increasing transportation assignment tolerance. We quantitatively evaluate the method across domain adaptation tasks with shifts in label distributions. Our experiments show that the proposed method is more robust against these shifts than other alignment-based baselines.
Memorizing Transformers
Venue: arXiv cs.LG 2022
Year: 2022

Language models typically need to be trained or finetuned in order to acquire new knowledge, which involves updating their weights. We instead envision language models that can simply read and memorize new data at inference time, thus acquiring new knowledge immediately. In this work, we extend language models with the ability to memorize the internal representations of past inputs. We demonstrate that an approximate kNN lookup into a non-differentiable memory of recent (key, value) pairs improves language modeling across various benchmarks and tasks, including generic webtext (C4), math papers (arXiv), books (PG-19), code (Github), as well as formal theorems (Isabelle). We show that the performance steadily improves when we increase the size of memory up to 262K tokens. On benchmarks including code and mathematics, we find that the model is capable of making use of newly defined functions and theorems during test time.
Towards True Detail Restoration for Super-Resolution: A Benchmark and a Quality Metric
Venue: arXiv cs.CV 2022
Year: 2022

Super-resolution (SR) has become a widely researched topic in recent years. SR methods can improve overall image and video quality and create new possibilities for further content analysis. But the SR mainstream focuses primarily on increasing the naturalness of the resulting image despite potentially losing context accuracy. Such methods may produce an incorrect digit, character, face, or other structural object even though they otherwise yield good visual quality. Incorrect detail restoration can cause errors when detecting and identifying objects both manually and automatically. To analyze the detail-restoration capabilities of image and video SR models, we developed a benchmark based on our own video dataset, which contains complex patterns that SR models generally fail to correctly restore. We assessed 32 recent SR models using our benchmark and compared their ability to preserve scene context. We also conducted a crowd-sourced comparison of restored details and developed an objective assessment metric that outperforms other quality metrics by correlation with subjective scores for this task. In conclusion, we provide a deep analysis of benchmark results that yields insights for future SR-based work.
Traffic-Aware UAV Placement Using a Generalizable Deep Reinforcement Learning Methodology
Venue: arXiv cs.NI 2022
Year: 2022

Unmanned Aerial Vehicles (UAVs) acting as Flying Access Points (FAPs) are being used to provide on-demand wireless connectivity in extreme scenarios. Despite ongoing research, the optimization of UAVs' positions according to dynamic users' traffic demands remains challenging. We propose the Traffic-aware UAV Placement Algorithm (TUPA), which positions a UAV acting as FAP according to the users' traffic demands, in order to maximize the network utility. Using a DRL approach enables the FAP to autonomously learn and adapt to dynamic conditions and requirements of networking scenarios. Moreover, the proposed DRL methodology allows TUPA to generalize knowledge acquired during training to unknown combinations of users' positions and traffic demands, with no additional training. TUPA is trained and evaluated using network simulator ns-3 and ns3-gym framework. The results demonstrate that TUPA increases the network utility, compared to baseline solutions, increasing the average network utility up to 4x in scenarios with heterogeneous traffic demands.
Translating Canonical SQL to Imperative Code in Coq
Venue: arXiv cs.PL 2022
Year: 2022

SQL is by far the most widely used and implemented query language. Yet, on some key features, such as correlated queries and NULL value semantics, many implementations diverge or contain bugs. We leverage recent advances in the formalization of SQL and query compilers to develop DBCert, the first mechanically verified compiler from SQL queries written in a canonical form to imperative code. Building DBCert required several new contributions which are described in this paper. First, we specify and mechanize a complete translation from SQL to the Nested Relational Algebra which can be used for query optimization. Second, we define Imp, a small imperative language sufficient to express SQL and which can target several execution languages including JavaScript. Finally, we develop a mechanized translation from the nested relational algebra to Imp, using the nested relational calculus as an intermediate step.
The Data Airlock: infrastructure for restricted data informatics
Venue: arXiv cs.IR 2022
Year: 2022

Data science collaboration is problematic when access to operational data or models from outside the data-holding organisation is prohibited, for a variety of legal, security, ethical, or practical reasons. There are significant data privacy challenges when performing collaborative data science work against such restricted data. In this paper we describe a range of causes and risks associated with restricted data along with the social, environmental, data, and cryptographic measures that may be used to mitigate such issues. We then show how these are generally inadequate for restricted data contexts and introduce the 'Data Airlock' - secure infrastructure that facilitates 'eyes-off' data science workloads. After describing our use-case we detail the architecture and implementation of a first, single-organisation version of the Data Airlock infrastructure. We conclude with outcomes and learning from this implementation, and outline requirements for a second, federated version.
Example-Based Vulnerability Detection and Repair in Java Code
Venue: arXiv cs.CR 2022
Year: 2022

The Java libraries JCA and JSSE offer cryptographic APIs to facilitate secure coding. When developers misuse some of the APIs, their code becomes vulnerable to cyber-attacks. To eliminate such vulnerabilities, people built tools to detect security-API misuses via pattern matching. However, most tools do not (1) fix misuses or (2) allow users to extend tools' pattern sets. To overcome both limitations, we created Seader-an example-based approach to detect and repair security-API misuses. Given an exemplar <insecure, secure>code pair, Seader compares the snippets to infer any API-misuse template and corresponding fixing edit. Based on the inferred info, given a program, Seader performs inter-procedural static analysis to search for security-API misuses and to propose customized fixes. For evaluation, we applied Seader to 28 <insecure, secure> codepairs; Seader successfully inferred 21 unique API-misuse templates and related fixes. With these <vulnerability, fix> patterns, we applied SEADER to a program benchmark that has 86 known vulnerabilities. Seader detected vulnerabilities with 95% precision, 72% recall, and82% F-score. We also applied Seader to 100 open-source projects and manually checked 77 suggested repairs; 76 of the repairs were correct. Seader can help developers correctly use security APIs.
Coordinated Power Control for Network Integrated Sensing and Communication
Venue: arXiv cs.IT 2022
Year: 2022

This correspondence paper studies a network integrated sensing and communication (ISAC) system that unifies the interference channel for communication and distributed radar sensing. In this system, a set of distributed ISAC transmitters send individual messages to their respective communication users (CUs), and at the same time cooperate with multiple sensing receivers to estimate the location of one target. We exploit the coordinated power control among ISAC transmitters to minimize their total transmit power while ensuring the minimum signal-to-interference-plus-noise ratio (SINR) constraints at individual CUs and the maximum Cramér-Rao lower bound (CRLB) requirement for target location estimation. Although the formulated coordinated power control problem is non-convex and difficult to solve in general, we propose two efficient algorithms to obtain high-quality solutions based on the semi-definite relaxation (SDR) and CRLB approximation, respectively. Numerical results show that the proposed designs achieve substantial performance gains in terms of power reduction, as compared to the benchmark with a heuristic separate communication-sensing design.
Phased Flight Trajectory Prediction with Deep Learning
Venue: arXiv cs.LG 2022
Year: 2022

The unprecedented increase of commercial airlines and private jets over the next ten years presents a challenge for air traffic control. Precise flight trajectory prediction is of great significance in air transportation management, which contributes to the decision-making for safe and orderly flights. Existing research and application mainly focus on the sequence generation based on historical trajectories, while the aircraft-aircraft interactions in crowded airspace especially the airspaces near busy airports have been largely ignored. On the other hand, there are distinct characteristics of aerodynamics for different flight phases, and the trajectory may be affected by various uncertainties such as weather and advisories from air traffic controllers. However, there is no literature fully considers all these issues. Therefore, we proposed a phased flight trajectory prediction framework. Multi-source and multi-modal datasets have been analyzed and mined using variants of recurrent neural network (RNN) mixture. To be specific, we first introduce spatio temporal graphs into the low-altitude airway prediction problem, and the motion constraints of an aircraft are embedded to the inference process for reliable forecasting results. In the en-route phase, the dual attention mechanism is employed to adaptively extract much more important features from overall datasets to learn the hidden patterns in dynamical environments. The experimental results demonstrate our proposed framework can outperform state-of-the-art methods for flight trajectory prediction for large passenger/transport airplanes.
GATE: Graph CCA for Temporal SElf-supervised Learning for Label-efficient fMRI Analysis
Venue: arXiv cs.LG 2022
Year: 2022

In this work, we focus on the challenging task, neuro-disease classification, using functional magnetic resonance imaging (fMRI). In population graph-based disease analysis, graph convolutional neural networks (GCNs) have achieved remarkable success. However, these achievements are inseparable from abundant labeled data and sensitive to spurious signals. To improve fMRI representation learning and classification under a label-efficient setting, we propose a novel and theory-driven self-supervised learning (SSL) framework on GCNs, namely Graph CCA for Temporal self-supervised learning on fMRI analysis GATE. Concretely, it is demanding to design a suitable and effective SSL strategy to extract formation and robust features for fMRI. To this end, we investigate several new graph augmentation strategies from fMRI dynamic functional connectives (FC) for SSL training. Further, we leverage canonical-correlation analysis (CCA) on different temporal embeddings and present the theoretical implications. Consequently, this yields a novel two-step GCN learning procedure comprised of (i) SSL on an unlabeled fMRI population graph and (ii) fine-tuning on a small labeled fMRI dataset for a classification task. Our method is tested on two independent fMRI datasets, demonstrating superior performance on autism and dementia diagnosis.
DU-VLG: Unifying Vision-and-Language Generation via Dual Sequence-to-Sequence Pre-training
Venue: arXiv cs.CV 2022
Year: 2022

Due to the limitations of the model structure and pre-training objectives, existing vision-and-language generation models cannot utilize pair-wise images and text through bi-directional generation. In this paper, we propose DU-VLG, a framework which unifies vision-and-language generation as sequence generation problems. DU-VLG is trained with novel dual pre-training tasks: multi-modal denoising autoencoder tasks and modality translation tasks. To bridge the gap between image understanding and generation, we further design a novel commitment loss. We compare pre-training objectives on image captioning and text-to-image generation datasets. Results show that DU-VLG yields better performance than variants trained with uni-directional generation objectives or the variant without the commitment loss. We also obtain higher scores compared to previous state-of-the-art systems on three vision-and-language generation tasks. In addition, human judges further confirm that our model generates real and relevant images as well as faithful and informative captions.
Gaussian Multi-head Attention for Simultaneous Machine Translation
Venue: arXiv cs.CL 2022
Year: 2022

Simultaneous machine translation (SiMT) outputs translation while receiving the streaming source inputs, and hence needs a policy to determine where to start translating. The alignment between target and source words often implies the most informative source word for each target word, and hence provides the unified control over translation quality and latency, but unfortunately the existing SiMT methods do not explicitly model the alignment to perform the control. In this paper, we propose Gaussian Multi-head Attention (GMA) to develop a new SiMT policy by modeling alignment and translation in a unified manner. For SiMT policy, GMA models the aligned source position of each target word, and accordingly waits until its aligned position to start translating. To integrate the learning of alignment into the translation model, a Gaussian distribution centered on predicted aligned position is introduced as an alignment-related prior, which cooperates with translation-related soft attention to determine the final attention. Experiments on En-Vi and De-En tasks show that our method outperforms strong baselines on the trade-off between translation and latency.
Energy-Efficient Power Control and Beamforming for Reconfigurable Intelligent Surface-Aided Uplink IoT Networks
Venue: arXiv cs.IT 2022
Year: 2022

Recently, reconfigurable intelligent surface (RIS), a planar metasurface consisting of a large number of low-cost reflecting elements, has received much attention due to its ability to improve both the spectrum and energy efficiencies by reconfiguring the wireless propagation environment. In this paper, we propose a RIS phase shift and BS beamforming optimization technique that minimizes the uplink transmit power of a RIS-aided IoT network. Key idea of the proposed scheme, referred to as Riemannian conjugate gradient-based joint optimization (RCG-JO), is to jointly optimize the RIS phase shifts and the BS beamforming vectors using the Riemannian conjugate gradient technique. By exploiting the product Riemannian manifold structure of the sets of unit-modulus phase shifts and unit-norm beamforming vectors, we convert the nonconvex uplink power minimization problem into the unconstrained problem and then find out the optimal solution on the product Riemannian manifold. From the performance analysis and numerical evaluations, we demonstrate that the proposed RCG-JO technique achieves $94\%$ reduction of the uplink transmit power over the conventional scheme without RIS.