OpenRaiser/Intern-Atlas huggingface dataset:
Clover: Towards A Unified Video-Language Alignment and Fusion Model
Venue: arXiv cs.CV 2022
Year: 2022

Building a universal Video-Language model for solving various video understanding tasks (\emph{e.g.}, text-video retrieval, video question answering) is an open challenge to the machine learning field. Towards this goal, most recent works build the model by stacking uni-modal and cross-modal feature encoders and train it with pair-wise contrastive pre-text tasks. Though offering attractive generality, the resulted models have to compromise between efficiency and performance. They mostly adopt different architectures to deal with different downstream tasks. We find this is because the pair-wise training cannot well \emph{align} and \emph{fuse} features from different modalities. We then introduce \textbf{Clover}\textemdash a Correlated Video-Language pre-training method\textemdash towards a universal Video-Language model for solving multiple video understanding tasks with neither performance nor efficiency compromise. It improves cross-modal feature alignment and fusion via a novel tri-modal alignment pre-training task. Additionally, we propose to enhance the tri-modal alignment via incorporating learning from semantic masked samples and a new pair-wise ranking loss. Clover establishes new state-of-the-arts on multiple downstream tasks, including three retrieval tasks for both zero-shot and fine-tuning settings, and eight video question answering tasks. Codes and pre-trained models will be released at \url{https://github.com/LeeYN-43/Clover}.
Mutual Adaptive Reasoning for Monocular 3D Multi-Person Pose Estimation
Venue: arXiv cs.CV 2022
Year: 2022

Inter-person occlusion and depth ambiguity make estimating the 3D poses of monocular multiple persons as camera-centric coordinates a challenging problem. Typical top-down frameworks suffer from high computational redundancy with an additional detection stage. By contrast, the bottom-up methods enjoy low computational costs as they are less affected by the number of humans. However, most existing bottom-up methods treat camera-centric 3D human pose estimation as two unrelated subtasks: 2.5D pose estimation and camera-centric depth estimation. In this paper, we propose a unified model that leverages the mutual benefits of both these subtasks. Within the framework, a robust structured 2.5D pose estimation is designed to recognize inter-person occlusion based on depth relationships. Additionally, we develop an end-to-end geometry-aware depth reasoning method that exploits the mutual benefits of both 2.5D pose and camera-centric root depths. This method first uses 2.5D pose and geometry information to infer camera-centric root depths in a forward pass, and then exploits the root depths to further improve representation learning of 2.5D pose estimation in a backward pass. Further, we designed an adaptive fusion scheme that leverages both visual perception and body geometry to alleviate inherent depth ambiguity issues. Extensive experiments demonstrate the superiority of our proposed model over a wide range of bottom-up methods. Our accuracy is even competitive with top-down counterparts. Notably, our model runs much faster than existing bottom-up and top-down methods.
Chimera: A Hybrid Machine Learning Driven Multi-Objective Design Space Exploration Tool for FPGA High-Level Synthesis
Venue: arXiv cs.AR 2022
Year: 2022

In recent years, hardware accelerators based on field-programmable gate arrays (FPGAs) have been widely adopted, thanks to FPGAs' extraordinary flexibility. However, with the high flexibility comes the difficulty in design and optimization. Conventionally, these accelerators are designed with low-level hardware descriptive languages, which means creating large designs with complex behavior is extremely difficult. Therefore, high-level synthesis (HLS) tools were created to simplify hardware designs for FPGAs. They enable the user to create hardware designs using high-level languages and provide various optimization directives to help to improve the performance of the synthesized hardware. However, applying these optimizations to achieve high performance is time-consuming and usually requires expert knowledge. To address this difficulty, we present an automated design space exploration tool for applying HLS optimization directives, called Chimera, which significantly reduces the human effort and expertise needed for creating high-performance HLS designs. It utilizes a novel multi-objective exploration method that seamlessly integrates active learning, evolutionary algorithm, and Thompson sampling, making it capable of finding a set of optimized designs on a Pareto curve with only a small number of design points evaluated during the exploration. In the experiments, in less than 24 hours, this hybrid method explored design points that have the same or superior performance compared to highly optimized hand-tuned designs created by expert HLS users from the Rosetta benchmark suite. In addition to discovering the extreme points, it also explores a Pareto frontier, where the elbow point can potentially save up to 26\% of Flip-Flop resource with negligibly higher latency.
Consistency of Implicit and Explicit Features Matters for Monocular 3D Object Detection
Venue: arXiv cs.CV 2022
Year: 2022

Low-cost autonomous agents including autonomous driving vehicles chiefly adopt monocular 3D object detection to perceive surrounding environment. This paper studies 3D intermediate representation methods which generate intermediate 3D features for subsequent tasks. For example, the 3D features can be taken as input for not only detection, but also end-to-end prediction and/or planning that require a bird's-eye-view feature representation. In the study, we found that in generating 3D representation previous methods do not maintain the consistency between the objects' implicit poses in the latent space, especially orientations, and the explicitly observed poses in the Euclidean space, which can substantially hurt model performance. To tackle this problem, we present a novel monocular detection method, the first one being aware of the poses to purposefully guarantee that they are consistent between the implicit and explicit features. Additionally, we introduce a local ray attention mechanism to efficiently transform image features to voxels at accurate 3D locations. Thirdly, we propose a handcrafted Gaussian positional encoding function, which outperforms the sinusoidal encoding function while retaining the benefit of being continuous. Results show that our method improves the state-of-the-art 3D intermediate representation method by 3.15%. We are ranked 1st among all the reported monocular methods on both 3D and BEV detection benchmark on KITTI leaderboard as of th result's submission time.
On the Subjectivity of Emotions in Software Projects: How Reliable are Pre-Labeled Data Sets for Sentiment Analysis?
Venue: arXiv cs.SE 2022
Year: 2022

Social aspects of software projects become increasingly important for research and practice. Different approaches analyze the sentiment of a development team, ranging from simply asking the team to so-called sentiment analysis on text-based communication. These sentiment analysis tools are trained using pre-labeled data sets from different sources, including GitHub and Stack Overflow. In this paper, we investigate if the labels of the statements in the data sets coincide with the perception of potential members of a software project team. Based on an international survey, we compare the median perception of 94 participants with the pre-labeled data sets as well as every single participant's agreement with the predefined labels. Our results point to three remarkable findings: (1) Although the median values coincide with the predefined labels of the data sets in 62.5% of the cases, we observe a huge difference between the single participant's ratings and the labels; (2) there is not a single participant who totally agrees with the predefined labels; and (3) the data set whose labels are based on guidelines performs better than the ad hoc labeled data set.
Local Stochastic Algorithms for Alignment in Self-Organizing Particle Systems
Venue: arXiv cs.DC 2022
Year: 2022

We present local distributed, stochastic algorithms for \emph{alignment} in self-organizing particle systems (SOPS) on two-dimensional lattices, where particles occupy unique sites on the lattice, and particles can make spatial moves to neighboring sites if they are unoccupied. Such models are abstractions of programmable matter, composed of individual computational particles with limited memory, strictly local communication abilities, and modest computational capabilities. We consider oriented particle systems, where particles are assigned a vector pointing in one of $q$ directions, and each particle can compute the angle between its direction and the direction of any neighboring particle, although without knowledge of global orientation with respect to a fixed underlying coordinate system. Particles move stochastically, with each particle able to either modify its direction or make a local spatial move along a lattice edge during a move. We consider two settings: (a) where particle configurations must remain simply connected at all times and (b) where spatial moves are unconstrained and configurations can disconnect. Taking inspiration from the Potts and clock models from statistical physics, we prove that for any $q \geq 2,$ these self-organizing particle systems can be made to collectively align along a single dominant direction (analogous to a solid or ordered state) or remain non-aligned, in which case the fraction of particles oriented along any direction is nearly equal (analogous to a gaseous or disordered state). Moreover, we show that with appropriate settings of the input parameters, we can achieve \emph{compression} and \emph{expansion}, controlling how tightly gathered the particles are, as well as \emph{alignment} or \emph{nonalignment}, producing a single dominant orientation or not.
Online Prediction in Sub-linear Space
Venue: arXiv cs.DS 2022
Year: 2022

We provide the first sub-linear space and sub-linear regret algorithm for online learning with expert advice (against an oblivious adversary), addressing an open question raised recently by Srinivas, Woodruff, Xu and Zhou (STOC 2022). We also demonstrate a separation between oblivious and (strong) adaptive adversaries by proving a linear memory lower bound of any sub-linear regret algorithm against an adaptive adversary. Our algorithm is based on a novel pool selection procedure that bypasses the traditional wisdom of leader selection for online learning, and a generic reduction that transforms any weakly sub-linear regret $o(T)$ algorithm to $T^{1-α}$ regret algorithm, which may be of independent interest. Our lower bound utilizes the connection of no-regret learning and equilibrium computation in zero-sum games, leading to a proof of a strong lower bound against an adaptive adversary.
Indivisible Participatory Budgeting under Weak Rankings
Venue: arXiv cs.GT 2022
Year: 2022

Participatory budgeting (PB) has attracted much attention in recent times due to its wide applicability in social choice settings. In this paper, we consider indivisible PB which involves allocating an available, limited budget to a set of indivisible projects, each having a certain cost, based on the preferences of agents over projects. The specific, important, research gap that we address in this paper is to propose classes of rules for indivisible PB with weak rankings (i.e., weak ordinal preferences) and investigate their key algorithmic and axiomatic issues. We propose two classes of rules having distinct significance and motivation. The first is layered approval rules which enable weak rankings to be studied by carefully translating them into approval votes. The second is need-based rules which enable to capture fairness issues. Under layered approval rules, we study two natural families of rules: greedy-truncation rules and cost-worthy rules. The paper has two parts. In the first part, we investigate algorithmic and complexity related issues for the proposed rules. In the second part, we present a detailed axiomatic analysis of these rules, for which, we examine and generalize axioms in the literature and also introduce a new axiom, pro-affordability. The paper helps to highlight the trade-offs among practical appeal, computational complexity, and axiomatic compliance of these rules.
Signed Cumulative Distribution Transform for Parameter Estimation of 1-D Signals
Venue: arXiv cs.IT 2022
Year: 2022

We describe a method for signal parameter estimation using the signed cumulative distribution transform (SCDT), a recently introduced signal representation tool based on optimal transport theory. The method builds upon signal estimation using the cumulative distribution transform (CDT) originally introduced for positive distributions. Specifically, we show that Wasserstein-type distance minimization can be performed simply using linear least squares techniques in SCDT space for arbitrary signal classes, thus providing a global minimizer for the estimation problem even when the underlying signal is a nonlinear function of the unknown parameters. Comparisons to current signal estimation methods using $L_p$ minimization shows the advantage of the method.
Relative Position Estimation in Multi-Agent Systems Using Attitude-Coupled Range Measurements
Venue: arXiv cs.RO 2022
Year: 2022

The ability to accurately estimate the position of robotic agents relative to one another, in possibly GPS-denied environments, is crucial to execute collaborative tasks. Inter-agent range measurements are available at a low cost, due to technologies such as ultra-wideband radio. However, the task of three-dimensional relative position estimation using range measurements in multi-agent systems suffers from unobservabilities. This letter presents a sufficient condition for the observability of the relative positions, and satisfies the condition using a simple framework with only range measurements, an accelerometer, a rate gyro, and a magnetometer. The framework has been tested in simulation and in experiments, where 40-50 cm positioning accuracy is achieved using inexpensive off-the-shelf hardware.
Federated Deep Reinforcement Learning for RIS-Assisted Indoor Multi-Robot Communication Systems
Venue: arXiv cs.RO 2022
Year: 2022

Indoor multi-robot communications face two key challenges: one is the severe signal strength degradation caused by blockages (e.g., walls) and the other is the dynamic environment caused by robot mobility. To address these issues, we consider the reconfigurable intelligent surface (RIS) to overcome the signal blockage and assist the trajectory design among multiple robots. Meanwhile, the non-orthogonal multiple access (NOMA) is adopted to cope with the scarcity of spectrum and enhance the connectivity of robots. Considering the limited battery capacity of robots, we aim to maximize the energy efficiency by jointly optimizing the transmit power of the access point (AP), the phase shifts of the RIS, and the trajectory of robots. A novel federated deep reinforcement learning (F-DRL) approach is developed to solve this challenging problem with one dynamic long-term objective. Through each robot planning its path and downlink power, the AP only needs to determine the phase shifts of the RIS, which can significantly save the computation overhead due to the reduced training dimension. Simulation results reveal the following findings: I) the proposed F-DRL can reduce at least 86% convergence time compared to the centralized DRL; II) the designed algorithm can adapt to the increasing number of robots; III) compared to traditional OMA-based benchmarks, NOMA-enhanced schemes can achieve higher energy efficiency.
Real Time Vehicle Identification
Venue: arXiv cs.DC 2022
Year: 2022

Identification of the vehicles passing over the roads is a very important component of an Intelligent Transportation System. However, due to the presence of multiple vehicles together and their velocity, it gets hard to accurately identify and record them in real-time. Solutions based on Computer-vision use heavyweight equipment making them quiet inflexible, costly and hence unsuitable for wide-area coverage. Solutions based on RFID, although are lightweight and cost-effective, lack of fast and efficient communication protocol pertains to their inability to record multiple moving vehicles at the same time. We propose an IoT-assisted solution that leverages Synchronous-Transmission based communication to bridge these gaps. Through extensive experiments we demonstrate that our strategy can consistently record upto an average of 40 vehicles running at speed range 30-90 Km/h with at least 97.5% accuracy.
Nonmyopic Distilled Data Association Belief Space Planning Under Budget Constraints
Venue: arXiv cs.AI 2022
Year: 2022

Autonomous agents operating in perceptually aliased environments should ideally be able to solve the data association problem. Yet, planning for future actions while considering this problem is not trivial. State of the art approaches therefore use multi-modal hypotheses to represent the states of the agent and of the environment. However, explicitly considering all possible data associations, the number of hypotheses grows exponentially with the planning horizon. As such, the corresponding Belief Space Planning problem quickly becomes unsolvable. Moreover, under hard computational budget constraints, some non-negligible hypotheses must eventually be pruned in both planning and inference. Nevertheless, the two processes are generally treated separately and the effect of budget constraints in one process over the other was barely studied. We present a computationally efficient method to solve the nonmyopic Belief Space Planning problem while reasoning about data association. Moreover, we rigorously analyze the effects of budget constraints in both inference and planning.
Achieve Optimal Adversarial Accuracy for Adversarial Deep Learning using Stackelberg Game
Venue: arXiv cs.LG 2022
Year: 2022

Adversarial deep learning is to train robust DNNs against adversarial attacks, which is one of the major research focuses of deep learning. Game theory has been used to answer some of the basic questions about adversarial deep learning such as the existence of a classifier with optimal robustness and the existence of optimal adversarial samples for a given class of classifiers. In most previous work, adversarial deep learning was formulated as a simultaneous game and the strategy spaces are assumed to be certain probability distributions in order for the Nash equilibrium to exist. But, this assumption is not applicable to the practical situation. In this paper, we give answers to these basic questions for the practical case where the classifiers are DNNs with a given structure, by formulating the adversarial deep learning as sequential games. The existence of Stackelberg equilibria for these games are proved. Furthermore, it is shown that the equilibrium DNN has the largest adversarial accuracy among all DNNs with the same structure, when Carlini-Wagner's margin loss is used. Trade-off between robustness and accuracy in adversarial deep learning is also studied from game theoretical aspect.
Federated Learning and catastrophic forgetting in pervasive computing: demonstration in HAR domain
Venue: arXiv cs.LG 2022
Year: 2022

Federated Learning has been introduced as a new machine learning paradigm enhancing the use of local devices. At a server level, FL regularly aggregates models learned locally on distributed clients to obtain a more general model. In this way, no private data is sent over the network, and the communication cost is reduced. However, current solutions rely on the availability of large amounts of stored data at the client side in order to fine-tune the models sent by the server. Such setting is not realistic in mobile pervasive computing where data storage must be kept low and data characteristic (distribution) can change dramatically. To account for this variability, a solution is to use the data regularly collected by the client to progressively adapt the received model. But such naive approach exposes clients to the well-known problem of catastrophic forgetting. The purpose of this paper is to demonstrate this problem in the mobile human activity recognition context on smartphones.
BusyBot: Learning to Interact, Reason, and Plan in a BusyBoard Environment
Venue: arXiv cs.RO 2022
Year: 2022

We introduce BusyBoard, a toy-inspired robot learning environment that leverages a diverse set of articulated objects and inter-object functional relations to provide rich visual feedback for robot interactions. Based on this environment, we introduce a learning framework, BusyBot, which allows an agent to jointly acquire three fundamental capabilities (interaction, reasoning, and planning) in an integrated and self-supervised manner. With the rich sensory feedback provided by BusyBoard, BusyBot first learns a policy to efficiently interact with the environment; then with data collected using the policy, BusyBot reasons the inter-object functional relations through a causal discovery network; and finally by combining the learned interaction policy and relation reasoning skill, the agent is able to perform goal-conditioned manipulation tasks. We evaluate BusyBot in both simulated and real-world environments, and validate its generalizability to unseen objects and relations. Video is available at https://youtu.be/EJ98xBJZ9ek.
Gradients should stay on Path: Better Estimators of the Reverse- and Forward KL Divergence for Normalizing Flows
Venue: arXiv cs.LG 2022
Year: 2022

We propose an algorithm to estimate the path-gradient of both the reverse and forward Kullback-Leibler divergence for an arbitrary manifestly invertible normalizing flow. The resulting path-gradient estimators are straightforward to implement, have lower variance, and lead not only to faster convergence of training but also to better overall approximation results compared to standard total gradient estimators. We also demonstrate that path-gradient training is less susceptible to mode-collapse. In light of our results, we expect that path-gradient estimators will become the new standard method to train normalizing flows for variational inference.
Autonomous Navigation in Confined Waters -- A COLREGs Rule 9 Compliant Framework
Venue: arXiv cs.RO 2022
Year: 2022

Fully or partial autonomous marine vessels are actively being developed by many industry actors. In many cases, the autonomous vessels will be operating close to shore, and within range of a Remote Control Center (RCC). Close to shore operation requires that the autonomous vessel is able to navigate in close proximity to other autonomous or manned vessels, and possibly in confined waters, while obeying the COLREGs on equal terms as any other vessel at sea. In confined waters however, certain COLREGs rules apply, which might alter the expected actions (give-way or stand-on), depending on the manoeuvrability of the vessels. This paper presents a Situation Awareness (SAS) framework for autonomous navigation that complies with COLREGs rule 9 (Narrow Channels). The proposed solution comprises a method for evaluating the manoeuvrability of a vessel in confined waters, for assessing the applicability of COLREGs rule 9. This feature is then integrated into an already existing SAS framework for facilitating COLREGs-compliant navigation in restricted waters. The applicability of the proposed method is demonstrated in simulation using a case study of a small autonomous passenger ferry.