Miracle-dz/UniFAR-FTUs huggingface dataset:
Deep Masked Graph Matching for Correspondence Identification in Collaborative Perception

Correspondence identification (CoID) is an essential component for collaborative perception in multi-robot systems, such as connected autonomous vehicles. The goal of CoID is to identify the correspondence of objects observed by multiple robots in their own field of view in order for robots to consistently refer to the same objects. CoID is challenging due to perceptual aliasing, object non-covisibility, and noisy sensing. In this paper, we introduce a novel deep masked graph matching approach to enable CoID and address the challenges. Our approach formulates CoID as a graph matching problem and we design a masked neural network to integrate the multimodal visual, spatial, and GPS information to perform CoID. In addition, we design a new technique to explicitly address object non-covisibility caused by occlusion and the vehicle's limited field of view. We evaluate our approach in a variety of street environments using a high-fidelity simulation that integrates the CARLA and SUMO simulators. The experimental results show that our approach outperforms the previous approaches and achieves state-of-the- art CoID performance in connected autonomous driving applications. Our work is available at: https://github.com/gaopeng5/DMGM.git.
Bayes Complexity of Learners vs Overfitting

We introduce a new notion of complexity of functions and we show that it has the following properties: (i) it governs a PAC Bayes-like generalization bound, (ii) for neural networks it relates to natural notions of complexity of functions (such as the variation), and (iii) it explains the generalization gap between neural networks and linear schemes. While there is a large set of papers which describes bounds that have each such property in isolation, and even some that have two, as far as we know, this is a first notion that satisfies all three of them. Moreover, in contrast to previous works, our notion naturally generalizes to neural networks with several layers. Even though the computation of our complexity is nontrivial in general, an upper-bound is often easy to derive, even for higher number of layers and functions with structure, such as period functions. An upper-bound we derive allows to show a separation in the number of samples needed for good generalization between 2 and 4-layer neural networks for periodic functions.
DiffuseRoll: Multi-track multi-category music generation based on diffusion model

Recent advancements in generative models have shown remarkable progress in music generation. However, most existing methods focus on generating monophonic or homophonic music, while the generation of polyphonic and multi-track music with rich attributes is still a challenging task. In this paper, we propose a novel approach for multi-track, multi-attribute symphonic music generation using the diffusion model. Specifically, we generate piano-roll representations with a diffusion model and map them to MIDI format for output. To capture rich attribute information, we introduce a color coding scheme to encode note sequences into color and position information that represents pitch,velocity, and instrument. This scheme enables a seamless mapping between discrete music sequences and continuous images. We also propose a post-processing method to optimize the generated scores for better performance. Experimental results show that our method outperforms state-of-the-art methods in terms of polyphonic music generation with rich attribute information compared to the figure methods.
Guided Speech Enhancement Network

High quality speech capture has been widely studied for both voice communication and human computer interface reasons. To improve the capture performance, we can often find multi-microphone speech enhancement techniques deployed on various devices. Multi-microphone speech enhancement problem is often decomposed into two decoupled steps: a beamformer that provides spatial filtering and a single-channel speech enhancement model that cleans up the beamformer output. In this work, we propose a speech enhancement solution that takes both the raw microphone and beamformer outputs as the input for an ML model. We devise a simple yet effective training scheme that allows the model to learn from the cues of the beamformer by contrasting the two inputs and greatly boost its capability in spatial rejection, while conducting the general tasks of denoising and dereverberation. The proposed solution takes advantage of classical spatial filtering algorithms instead of competing with them. By design, the beamformer module then could be selected separately and does not require a large amount of data to be optimized for a given form factor, and the network model can be considered as a standalone module which is highly transferable independently from the microphone array. We name the ML module in our solution as GSENet, short for Guided Speech Enhancement Network. We demonstrate its effectiveness on real world data collected on multi-microphone devices in terms of the suppression of noise and interfering speech.
GoNet: An Approach-Constrained Generative Grasp Sampling Network

This work addresses the problem of learning approach-constrained data-driven grasp samplers. To this end, we propose GoNet: a generative grasp sampler that can constrain the grasp approach direction to a subset of SO(3). The key insight is to discretize SO(3) into a predefined number of bins and train GoNet to generate grasps whose approach directions are within those bins. At run-time, the bin aligning with the second largest principal component of the observed point cloud is selected. GoNet is benchmarked against GraspNet, a state-of-the-art unconstrained grasp sampler, in an unconfined grasping experiment in simulation and on an unconfined and confined grasping experiment in the real world. The results demonstrate that GoNet achieves higher success-over-coverage in simulation and a 12%-18% higher success rate in real-world table-picking and shelf-picking tasks than the baseline.
FingerSLAM: Closed-loop Unknown Object Localization and Reconstruction from Visuo-tactile Feedback

In this paper, we address the problem of using visuo-tactile feedback for 6-DoF localization and 3D reconstruction of unknown in-hand objects. We propose FingerSLAM, a closed-loop factor graph-based pose estimator that combines local tactile sensing at finger-tip and global vision sensing from a wrist-mount camera. FingerSLAM is constructed with two constituent pose estimators: a multi-pass refined tactile-based pose estimator that captures movements from detailed local textures, and a single-pass vision-based pose estimator that predicts from a global view of the object. We also design a loop closure mechanism that actively matches current vision and tactile images to previously stored key-frames to reduce accumulated error. FingerSLAM incorporates the two sensing modalities of tactile and vision, as well as the loop closure mechanism with a factor graph-based optimization framework. Such a framework produces an optimized pose estimation solution that is more accurate than the standalone estimators. The estimated poses are then used to reconstruct the shape of the unknown object incrementally by stitching the local point clouds recovered from tactile images. We train our system on real-world data collected with 20 objects. We demonstrate reliable visuo-tactile pose estimation and shape reconstruction through quantitative and qualitative real-world evaluations on 6 objects that are unseen during training.
On Decentralized Nonsmooth Optimization

In decentralized optimization, several nodes connected by a network collaboratively minimize some objective function. For minimization of Lipschitz functions lower bounds are known along with optimal algorithms. We study a specific class of problems: linear models with nonsmooth loss functions. Our algorithm combines regularization and dual reformulation to get an effective optimization method with complexity better than the lower bounds.
Recovering Arrhythmic EEG Transients from Their Stochastic Interference

Traditionally, the neuronal dynamics underlying electroencephalograms (EEG) have been understood as arising from \textit{rhythmic oscillators with varying degrees of synchronization}. This dominant metaphor employs frequency domain EEG analysis to identify the most prominent populations of neuronal current sources in terms of their frequency and spectral power. However, emerging perspectives on EEG highlight its arrhythmic nature, which is primarily inferred from broadband EEG properties like the ubiquitous $1/f$ spectrum. In the present study, we use an \textit{arrhythmic superposition of pulses} as a metaphor to explain the origin of EEG. This conceptualization has a fundamental problem because the interference produced by the superpositions of pulses generates colored Gaussian noise, masking the temporal profile of the generating pulse. We solved this problem by developing a mathematical method involving the derivative of the autocovariance function to recover excellent approximations of the underlying pulses, significantly extending the analysis of this type of stochastic processes. When the method is applied to spontaneous mouse EEG sampled at $5$ kHz during the sleep-wake cycle, specific patterns -- called $\Psi$-patterns -- characterizing NREM sleep, REM sleep, and wakefulness are revealed. $\Psi$-patterns can be understood theoretically as \textit{power density in the time domain} and correspond to combinations of generating pulses at different time scales. Remarkably, we report the first EEG wakefulness-specific feature, which corresponds to an ultra-fast ($\sim 1$ ms) transient component of the observed patterns. By shifting the paradigm of EEG genesis from oscillators to random pulse generators, our theoretical framework pushes the boundaries of traditional Fourier-based EEG analysis, paving the way for new insights into the arrhythmic components of neural dynamics.
MetaTroll: Few-shot Detection of State-Sponsored Trolls with Transformer Adapters

State-sponsored trolls are the main actors of influence campaigns on social media and automatic troll detection is important to combat misinformation at scale. Existing troll detection models are developed based on training data for known campaigns (e.g. the influence campaign by Russia’s Internet Research Agency on the 2016 US Election), and they fall short when dealing with novel campaigns with new targets. We propose MetaTroll, a text-based troll detection model based on the meta-learning framework that enables high portability and parameter-efficient adaptation to new campaigns using only a handful of labelled samples for few-shot transfer. We introduce campaign-specific transformer adapters to MetaTroll to “memorise” campaign-specific knowledge so as to tackle catastrophic forgetting, where a model “forgets” how to detect trolls from older campaigns due to continual adaptation. Our experiments demonstrate that MetaTroll substantially outperforms baselines and state-of-the-art few-shot text classification models. Lastly, we explore simple approaches to extend MetaTroll to multilingual and multimodal detection. Source code for MetaTroll is available at: https://github.com/ltian678/metatroll-code.git
A Simple Baseline for Supervised Surround-view Depth Estimation

Depth estimation has been widely studied and serves as the fundamental step of 3D perception for intelligent vehicles. Though significant progress has been made in monocular depth estimation in the past decades, these attempts are mainly conducted on the KITTI benchmark with only front-view cameras, which ignores the correlations across surround-view cameras. In this paper, we propose S3Depth, a Simple Baseline for Supervised Surround-view Depth Estimation, to jointly predict the depth maps across multiple surrounding cameras. Specifically, we employ a global-to-local feature extraction module which combines CNN with transformer layers for enriched representations. Further, the Adjacent-view Attention mechanism is proposed to enable the intra-view and inter-view feature propagation. The former is achieved by the self-attention module within each view, while the latter is realized by the adjacent attention module, which computes the attention across multi-cameras to exchange the multi-scale representations across surround-view feature maps. Extensive experiments show that our method achieves superior performance over existing state-of-the-art methods on both DDAD and nuScenes datasets.
X-ReCoSa: Multi-Scale Context Aggregation For Multi-Turn Dialogue Generation

In multi-turn dialogue generation, responses are not only related to the topic and background of the context but also related to words and phrases in the sentences of the context. However, currently widely used hierarchical dialog models solely rely on context representations from the utterance-level encoder, ignoring the sentence representations output by the word-level encoder. This inevitably results in a loss of information while decoding and generating. In this paper, we propose a new dialog model X-ReCoSa totackle this problem which aggregates multi-scale context information for hierarchical dialog models. Specifically, we divide the generation decoder into upper and lower parts, namely the intention part andthe generation part. Firstly, the intention part takes context representations as input to generate the intention of the response. Then the generation part generates words depending on sentence representations. Therefore, the hierarchical information has been fused into response generation. we conduct experiments on the English dataset DailyDialog. Experimental results exhibit that our methodoutperforms baseline models on both automatic metric-based and human-based evaluations.
Context-based Ontology Modelling for Database: Enabling ChatGPT for Semantic Database Management

This research paper explores the use of ChatGPT in database management. ChatGPT, an AI-powered chatbot, has limitations in performing tasks related to database management due to the lack of standardized vocabulary and grammar for representing database semantics. To address this limitation, the paper proposes a solution that involves developing a set of syntaxes that can represent database semantics in natural language. The syntax is used to convert database schemas into natural language formats, providing a new application of ChatGPT in database management. The proposed solution is demonstrated through a case study where ChatGPT is used to perform two tasks, semantic integration, and tables joining. Results demonstrate that the use of semantic database representations produces more precise outcomes and avoids common mistakes compared to cases with no semantic representation. The proposed method has the potential to speed up the database management process, reduce the level of understanding required for database domain knowledge, and enable automatic database operations without accessing the actual data, thus illuminating privacy protection concerns when using AI. This paper provides a promising new direction for research in the field of AI-based database management.
Spin-$s$ Rational $Q$-system

Bethe ansatz equations for spin-$s$ Heisenberg spin chain with $s\ge1$ are significantly more difficult to analyze than the spin-$\tfrac{1}{2}$ case, due to the presence of repeated roots. As a result, it is challenging to derive extra conditions for the Bethe roots to be physical and study the related completeness problem. In this paper, we propose the rational $Q$-system for the XXX$_s$ spin chain. Solutions of the proposed $Q$-system give all and only physical solutions of the Bethe ansatz equations required by completeness. This is checked numerically and proved rigorously. The rational $Q$-system is equivalent to the requirement that the solution and the corresponding dual solution of the $TQ$-relation are both polynomials, which we prove rigorously. Based on this analysis, we propose the extra conditions for solutions of the XXX$_s$ Bethe ansatz equations to be physical.
ForDigitStress: A multi-modal stress dataset employing a digital job interview scenario

We present a multi-modal stress dataset that uses digital job interviews to induce stress. The dataset provides multi-modal data of 40 participants including audio, video (motion capturing, facial recognition, eye tracking) as well as physiological information (photoplethysmography, electrodermal activity). In addition to that, the dataset contains time-continuous annotations for stress and occurred emotions (e.g. shame, anger, anxiety, surprise). In order to establish a baseline, five different machine learning classifiers (Support Vector Machine, K-Nearest Neighbors, Random Forest, Long-Short-Term Memory Network) have been trained and evaluated on the proposed dataset for a binary stress classification task. The best-performing classifier achieved an accuracy of 88.3% and an F1-score of 87.5%.
V2V4Real: A Real-World Large-Scale Dataset for Vehicle-to-Vehicle Cooperative Perception

Modern perception systems of autonomous vehicles are known to be sensitive to occlusions and lack the capability of long perceiving range. It has been one of the key bottlenecks that prevents Level 5 autonomy. Recent research has demonstrated that the Vehicle-to-Vehicle (V2V) cooperative perception system has great potential to revolutionize the autonomous driving industry. However, the lack of a real-world dataset hinders the progress of this field. To facilitate the development of cooperative perception, we present V2V4Real, the first large-scale real-world multi-modal dataset for V2V perception. The data is collected by two vehicles equipped with multi-modal sensors driving together through diverse scenarios. Our V2V4Real dataset covers a driving area of 410 km, comprising 20K LiDAR frames, 40K RGB frames, 240K annotated 3D bounding boxes for 5 classes, and HDMaps that cover all the driving routes. V2V4Real introduces three perception tasks, including cooperative 3D object detection, cooperative 3D object tracking, and Sim2Real domain adaptation for cooperative perception. We provide comprehensive benchmarks of recent cooperative perception algorithms on three tasks. The V2V4Real dataset can be found at research.seas. ucla.edu/mobility-lab/v2v4real/.
Development of a non-contact PZT excitation and sensing technology via laser

In recent years, guided wave based structural health monitoring (SHM) techniques have attracted much attention, because they are not only sensitive to small defects but also capable to cover a wide range in plate and pipe like structures. The guided waves in a structure can be generated and sensed by a variety of techniques. This study proposes a new wireless scheme for PZT excitation and sensing where power as well as data can be transmitted via laser. A generated waveform by modulation of a laser is wirelessly transmitted to a photodiode connected to a PZT on the structures. Then, the photodiode converts the light into an electrical signal and excite the PZT and the structure. Then, the reflected response signal received at the sensing PZT is re-converted into a laser, which is wirelessly transmitted back to another photodiode located in the data acquisition unit for damage diagnosis. The feasibility of the proposed power and data transmission scheme has been experimentally investigated in a laboratory setup. Using the proposed technology, a PZT transducer can be attached to a structure without complex electronic components and a power supply.
Improving Java Deserialization Gadget Chain Mining via Overriding-Guided Object Generation

Java (de)serialization is prone to causing security-critical vulnerabilities that attackers can invoke existing methods (gadgets) on the application's classpath to construct a gadget chain to perform malicious behaviors. Several techniques have been proposed to statically identify suspicious gadget chains and dynamically generate injection objects for fuzzing. However, due to their incomplete support for dynamic program features (e.g., Java runtime polymorphism) and ineffective injection object generation for fuzzing, the existing techniques are still far from satisfactory. In this paper, we first performed an empirical study to investigate the characteristics of Java deserialization vulnerabilities based on our manually collected 86 publicly known gadget chains. The empirical results show that 1) Java deserialization gadgets are usually exploited by abusing runtime polymorphism, which enables attackers to reuse serializable overridden methods; and 2) attackers usually invoke exploitable overridden methods (gadgets) via dynamic binding to generate injection objects for gadget chain construction. Based on our empirical findings, we propose a novel gadget chain mining approach, GCMiner, which captures both explicit and implicit method calls to identify more gadget chains, and adopts an overriding-guided object generation approach to generate valid injection objects for fuzzing. The evaluation results show that GCMiner significantly outperforms the state-of-the-art techniques, and discovers 56 unique gadget chains that cannot be identified by the baseline approaches.
Essays on sequence analysis for life course trajectories

The thesis is articulated in three chapters in which I explore methodological aspects of sequence analysis for life course studies and I present some empirical analyses. In the first chapter, I study the reliability of two holistic methods used in life-course methodology. Using simulated data, I compare the goodness of classification of Latent Class Analysis and Sequence Analysis techniques. I first compare the consistency of the classification obtained via the two techniques using an actual dataset on the life course trajectories of young adults. Then, I adopt a simulation approach to measure the ability of these two methods to correctly classify groups of life course trajectories when specific forms of “random” variability are introduced within pre-specified classes in an artificial datasets. In order to do so, I introduce simulation operators that have a life course and/or obser- vational meaning. In the second chapter, I propose a method to study the heterogeneity in life course trajectories. Using a non parametric approach, I evaluate the association between Optimal Matching distances and a set of categorical variables. Using data from the National Longitudinal Study of Adolescent Health (Add-Health), I study the hetero- geneity of early family trajectories in young women. In particular, I investigate if the OM distances can be partially explained by family characteristics and geographical context experienced during adolescence. The statistical methodology is a generalization of the analysis of variance (ANOVA) to any metric measure. In the last chapter, I present an application of sequence analysis. Using family transitions from Wave I to Wave IV of Add-health, I investigate the association between life trajectories and health outcomes at Wave IV. In particular, I am interested in exploring how differences in timing, quantum and order of family formation transitions are connected to self-reported health, depres- sion and risky behaviors in young women. Using lagged-value regression models, I take into account selection and the effect of confounding variables.