OpenRaiser/Intern-Atlas huggingface dataset:
Graph-Jigsaw Conditioned Diffusion Model for Skeleton-based Video Anomaly Detection
Venue: arXiv cs.CV 2024
Year: 2024

Skeleton-based video anomaly detection (SVAD) is a crucial task in computer vision. Accurately identifying abnormal patterns or events enables operators to promptly detect suspicious activities, thereby enhancing safety. Achieving this demands a comprehensive understanding of human motions, both at body and region levels, while also accounting for the wide variations of performing a single action. However, existing studies fail to simultaneously address these crucial properties. This paper introduces a novel, practical and lightweight framework, namely Graph-Jigsaw Conditioned Diffusion Model for Skeleton-based Video Anomaly Detection (GiCiSAD) to overcome the challenges associated with SVAD. GiCiSAD consists of three novel modules: the Graph Attention-based Forecasting module to capture the spatio-temporal dependencies inherent in the data, the Graph-level Jigsaw Puzzle Maker module to distinguish subtle region-level discrepancies between normal and abnormal motions, and the Graph-based Conditional Diffusion model to generate a wide spectrum of human motions. Extensive experiments on four widely used skeleton-based video datasets show that GiCiSAD outperforms existing methods with significantly fewer training parameters, establishing it as the new state-of-the-art.
Bootstrapping Reinforcement Learning with Imitation for Vision-Based Agile Flight
Venue: arXiv cs.RO 2024
Year: 2024

Learning visuomotor policies for agile quadrotor flight presents significant difficulties, primarily from inefficient policy exploration caused by high-dimensional visual inputs and the need for precise and low-latency control. To address these challenges, we propose a novel approach that combines the performance of Reinforcement Learning (RL) and the sample efficiency of Imitation Learning (IL) in the task of vision-based autonomous drone racing. While RL provides a framework for learning high-performance controllers through trial and error, it faces challenges with sample efficiency and computational demands due to the high dimensionality of visual inputs. Conversely, IL efficiently learns from visual expert demonstrations, but it remains limited by the expert's performance and state distribution. To overcome these limitations, our policy learning framework integrates the strengths of both approaches. Our framework contains three phases: training a teacher policy using RL with privileged state information, distilling it into a student policy via IL, and adaptive fine-tuning via RL. Testing in both simulated and real-world scenarios shows our approach can not only learn in scenarios where RL from scratch fails but also outperforms existing IL methods in both robustness and performance, successfully navigating a quadrotor through a race course using only visual information. Videos of the experiments are available at https://rpg.ifi.uzh.ch/bootstrap-rl-with-il/index.html.
A Unified Model for Longitudinal Multi-Modal Multi-View Prediction with Missingness
Venue: arXiv cs.CV 2024
Year: 2024

Medical records often consist of different modalities, such as images, text, and tabular information. Integrating all modalities offers a holistic view of a patient's condition, while analyzing them longitudinally provides a better understanding of disease progression. However, real-world longitudinal medical records present challenges: 1) patients may lack some or all of the data for a specific timepoint, and 2) certain modalities or views might be absent for all patients during a particular period. In this work, we introduce a unified model for longitudinal multi-modal multi-view prediction with missingness. Our method allows as many timepoints as desired for input, and aims to leverage all available data, regardless of their availability. We conduct extensive experiments on the knee osteoarthritis dataset from the Osteoarthritis Initiative for pain and Kellgren-Lawrence grade prediction at a future timepoint. We demonstrate the effectiveness of our method by comparing results from our unified model to specific models that use the same modality and view combinations during training and evaluation. We also show the benefit of having extended temporal data and provide post-hoc analysis for a deeper understanding of each modality/view's importance for different tasks.
IKSPARK: An Inverse Kinematics Solver using Semidefinite Relaxation and Rank Minimization
Venue: arXiv cs.RO 2024
Year: 2024

Inverse kinematics (IK) is a fundamental problem frequently occurred in robot control and motion planning. However, the problem is nonconvex because the kinematic map between the configuration and task spaces is generally nonlinear, which makes it challenging for fast and accurate solutions. The problem can be more complicated with the existence of different physical constraints imposed by the robot structure. In this paper, we develop an inverse kinematics solver named IKSPARK (Inverse Kinematics using Semidefinite Programming And RanK minimization) that can find solutions for robots with various structures, including open/closed kinematic chains, spherical, revolute, and/or prismatic joints. The solver works in the space of rotation matrices of the link reference frames and involves solving only convex semidefinite problems (SDPs). Specifically, the IK problem is formulated as an SDP with an additional rank-1 constraint on symmetric matrices with constant traces. The solver first solves this SDP disregarding the rank constraint to get a start point and then finds the rank-1 solution iteratively via a rank minimization algorithm with proven local convergence. Compared to other work that performs SDP relaxation for IK problems, our formulation is simpler, and uses variables with smaller sizes. We validate our approach via simulations on different robots, comparing against a standard IK method.
BeRGeR: Byzantine-Robust Geometric Routing
Venue: arXiv cs.DC 2024
Year: 2024

We present BeRGeR: the first asynchronous geometric routing algorithm that guarantees delivery of a message despite a Byzantine fault without relying on cryptographic primitives or randomization. The communication graph is a planar embedding that remains three-connected if all edges intersecting the source-target line segment are removed. We prove the algorithm correct and estimate its message complexity.
Energy Consumption in RES-Aware 5G Networks
Venue: arXiv cs.NI 2024
Year: 2024

In this work, the impact of using Renewable Energy Source (RES) generators in next-generation (5G) cellular systems on total power consumption (PC) has been investigated. The paper highlights the gain related to the use of photovoltaic (PV) panels and wind turbines (WTs) in the form of two factors - the average extension of battery lifetime (AEBL) powering a single network cell and the average reduction in energy consumption (AREC) within the whole network. The examination has been conducted for four different seasons of the year and various configurations of available power sources. The provided system scenario was based on real data on weather conditions, building placement, and implemented mobile networks for the city of Poznan in Poland. Used RES generators were designed in accordance with the specifications of real devices.
A Comparative Investigation of Compositional Syntax and Semantics in DALL-E 2
Venue: arXiv cs.CL 2024
Year: 2024

In this study we compared how well DALL-E 2 visually represented the meaning of linguistic prompts also given to young children in comprehension tests. Sentences representing fundamental components of grammatical knowledge were selected from assessment tests used with several hundred English-speaking children aged 2-7 years for whom we had collected original item-level data. DALL-E 2 was given these prompts five times to generate 20 cartoons per item, for 9 adult judges to score. Results revealed no conditions in which DALL-E 2-generated images that matched the semantic accuracy of children, even at the youngest age (2 years). DALL-E 2 failed to assign the appropriate roles in reversible forms; it failed on negation despite an easier contrastive prompt than the children received; it often assigned the adjective to the wrong noun; it ignored implicit agents in passives. This work points to a clear absence of compositional sentence representations for DALL-E 2.
Leveraging Large Language Models to Extract Information on Substance Use Disorder Severity from Clinical Notes: A Zero-shot Learning Approach
Venue: arXiv cs.CL 2024
Year: 2024

Substance use disorder (SUD) poses a major concern due to its detrimental effects on health and society. SUD identification and treatment depend on a variety of factors such as severity, co-determinants (e.g., withdrawal symptoms), and social determinants of health. Existing diagnostic coding systems used by American insurance providers, like the International Classification of Diseases (ICD-10), lack granularity for certain diagnoses, but clinicians will add this granularity (as that found within the Diagnostic and Statistical Manual of Mental Disorders classification or DSM-5) as supplemental unstructured text in clinical notes. Traditional natural language processing (NLP) methods face limitations in accurately parsing such diverse clinical language. Large Language Models (LLMs) offer promise in overcoming these challenges by adapting to diverse language patterns. This study investigates the application of LLMs for extracting severity-related information for various SUD diagnoses from clinical notes. We propose a workflow employing zero-shot learning of LLMs with carefully crafted prompts and post-processing techniques. Through experimentation with Flan-T5, an open-source LLM, we demonstrate its superior recall compared to the rule-based approach. Focusing on 11 categories of SUD diagnoses, we show the effectiveness of LLMs in extracting severity information, contributing to improved risk assessment and treatment planning for SUD patients.
EffiPerception: an Efficient Framework for Various Perception Tasks
Venue: arXiv cs.CV 2024
Year: 2024

The accuracy-speed-memory trade-off is always the priority to consider for several computer vision perception tasks. Previous methods mainly focus on a single or small couple of these tasks, such as creating effective data augmentation, feature extractor, learning strategies, etc. These approaches, however, could be inherently task-specific: their proposed model's performance may depend on a specific perception task or a dataset. Targeting to explore common learning patterns and increasing the module robustness, we propose the EffiPerception framework. It could achieve great accuracy-speed performance with relatively low memory cost under several perception tasks: 2D Object Detection, 3D Object Detection, 2D Instance Segmentation, and 3D Point Cloud Segmentation. Overall, the framework consists of three parts: (1) Efficient Feature Extractors, which extract the input features for each modality. (2) Efficient Layers, plug-in plug-out layers that further process the feature representation, aggregating core learned information while pruning noisy proposals. (3) The EffiOptim, an 8-bit optimizer to further cut down the computational cost and facilitate performance stability. Extensive experiments on the KITTI, semantic-KITTI, and COCO datasets revealed that EffiPerception could show great accuracy-speed-memory overall performance increase within the four detection and segmentation tasks, in comparison to earlier, well-respected methods.
Sim2Real in Reconstructive Spectroscopy: Deep Learning with Augmented Device-Informed Data Simulation
Venue: arXiv cs.LG 2024
Year: 2024

This work proposes a deep learning (DL)-based framework, namely Sim2Real, for spectral signal reconstruction in reconstructive spectroscopy, focusing on efficient data sampling and fast inference time. The work focuses on the challenge of reconstructing real-world spectral signals under the extreme setting where only device-informed simulated data are available for training. Such device-informed simulated data are much easier to collect than real-world data but exhibit large distribution shifts from their real-world counterparts. To leverage such simulated data effectively, a hierarchical data augmentation strategy is introduced to mitigate the adverse effects of this domain shift, and a corresponding neural network for the spectral signal reconstruction with our augmented data is designed. Experiments using a real dataset measured from our spectrometer device demonstrate that Sim2Real achieves significant speed-up during the inference while attaining on-par performance with the state-of-the-art optimization-based methods.
U-Net Kalman Filter (UNetKF): An Example of Machine Learning-assisted Ensemble Data Assimilation
Venue: arXiv cs.LG 2024
Year: 2024

Machine learning techniques have seen a tremendous rise in popularity in weather and climate sciences. Data assimilation (DA), which combines observations and numerical models, has great potential to incorporate machine learning and artificial intelligence (ML/AI) techniques. In this paper, we use U-Net, a type of convolutional neutral network (CNN), to predict the localized ensemble covariances for the Ensemble Kalman Filter (EnKF) algorithm. Using a 2-layer quasi-geostrophic model, U-Nets are trained using data from EnKF DA experiments. The trained U-Nets are then used to predict the flow-dependent localized error covariance matrices in U-Net Kalman Filter (UNetKF) experiments, which are compared to traditional 3-dimensional variational (3DVar), ensemble 3DVar (En3DVar) and EnKF methods. The performance of UNetKF can match or exceed that of 3DVar, En3DVar or EnKF. We also demonstrate that trained U-Nets can be transferred to a higher-resolution model for UNetKF implementation, which again performs competitively to 3DVar and EnKF, particularly for small ensemble sizes.
Cross-Domain Pre-training with Language Models for Transferable Time Series Representations
Venue: arXiv cs.LG 2024
Year: 2024

Advancements in self-supervised pre-training (SSL) have significantly advanced the field of learning transferable time series representations, which can be very useful in enhancing the downstream task. Despite being effective, most existing works struggle to achieve cross-domain SSL pre-training, missing valuable opportunities to integrate patterns and features from different domains. The main challenge lies in the significant differences in the characteristics of time-series data across different domains, such as variations in the number of channels and temporal resolution scales. To address this challenge, we propose CrossTimeNet, a novel cross-domain SSL learning framework to learn transferable knowledge from various domains to largely benefit the target downstream task. One of the key characteristics of CrossTimeNet is the newly designed time series tokenization module, which could effectively convert the raw time series into a sequence of discrete tokens based on a reconstruction optimization process. Besides, we highlight that predicting a high proportion of corrupted tokens can be very helpful for extracting informative patterns across different domains during SSL pre-training, which has been largely overlooked in past years. Furthermore, unlike previous works, our work treats the pre-training language model (PLM) as the initialization of the encoder network, investigating the feasibility of transferring the knowledge learned by the PLM to the time series area. Through these efforts, the path to cross-domain pre-training of a generic time series model can be effectively paved. We conduct extensive experiments in a real-world scenario across various time series classification domains. The experimental results clearly confirm CrossTimeNet's superior performance.
Hierarchical Digital Twin for Efficient 6G Network Orchestration via Adaptive Attribute Selection and Scalable Network Modeling
Venue: arXiv cs.NI 2024
Year: 2024

Achieving a holistic and long-term understanding through accurate network modeling is essential for orchestrating future networks with increasing service diversity and infrastructure complexities. However, due to unselective data collection and uniform processing, traditional modeling approaches undermine the efficacy and timeliness of network orchestration. Additionally, temporal disparities arising from various modeling delays further impair the centralized decision-making with distributed models. In this paper, we propose a new hierarchical digital twin paradigm adapting to real-time network situations for problem-centered model construction. Specifically, we introduce an adaptive attribute selection mechanism that evaluates the distinct modeling values of diverse network attributes, considering their relevance to current network scenarios and inherent modeling complexity. By prioritizing critical attributes at higher layers, an efficient evaluation of network situations is achieved to identify target areas. Subsequently, scalable network modeling facilitates the inclusion of all identified elements at the lower layers, where more fine-grained digital twins are developed to generate targeted solutions for user association and power allocation. Furthermore, virtual-physical domain synchronization is implemented to maintain accurate temporal alignment between the digital twins and their physical counterparts, spanning from the construction to the utilization of the proposed paradigm. Extensive simulations validate the proposed approach, demonstrating its effectiveness in efficiently identifying pressing issues and delivering network orchestration solutions in complex 6G HetNets.
Dexterous Functional Pre-Grasp Manipulation with Diffusion Policy
Venue: arXiv cs.RO 2024
Year: 2024

In real-world scenarios, objects often require repositioning and reorientation before they can be grasped, a process known as pre-grasp manipulation. Learning universal dexterous functional pre-grasp manipulation requires precise control over the relative position, orientation, and contact between the hand and object while generalizing to diverse dynamic scenarios with varying objects and goal poses. To address this challenge, we propose a teacher-student learning approach that utilizes a novel mutual reward, incentivizing agents to optimize three key criteria jointly. Additionally, we introduce a pipeline that employs a mixture-of-experts strategy to learn diverse manipulation policies, followed by a diffusion policy to capture complex action distributions from these experts. Our method achieves a success rate of 72.6\% across more than 30 object categories by leveraging extrinsic dexterity and adjusting from feedback.
On the effectiveness of Large Language Models for GitHub Workflows
Venue: arXiv cs.SE 2024
Year: 2024

GitHub workflows or GitHub CI is a popular continuous integration platform that enables developers to automate various software engineering tasks by specifying them as workflows, i.e., YAML files with a list of jobs. However, engineering valid workflows is tedious. They are also prone to severe security issues, which can result in supply chain vulnerabilities. Recent advancements in Large Language Models (LLMs) have demonstrated their effectiveness in various software development tasks. However, GitHub workflows differ from regular programs in both structure and semantics. We perform the first comprehensive study to understand the effectiveness of LLMs on five workflow-related tasks with different levels of prompts. We curated a set of $\sim$400K workflows and generated prompts with varying detail. We also fine-tuned LLMs on GitHub workflow tasks. Our evaluation of three state-of-the-art LLMs and their fine-tuned variants revealed various interesting findings on the current effectiveness and drawbacks of LLMs.
Intention Action Anticipation Model with Guide-Feedback Loop Mechanism
Venue: arXiv cs.CV 2024
Year: 2024

Anticipating human intention from videos has broad applications, such as automatic driving, robot assistive technology, and virtual reality. This study addresses the problem of intention action anticipation using egocentric video sequences to estimate actions that indicate human intention. We propose a Hierarchical Complete-Recent (HCR) information fusion model that makes full use of the features of the entire video sequence (i.e., complete features) and the features of the video tail sequence (i.e., recent features). The HCR model has two primary mechanisms. The Guide-Feedback Loop (GFL) mechanism is proposed to model the relation between one recent feature and one complete feature. Based on GFL, the MultiComplete-Recent Feature Aggregation (MCRFA) module is proposed to model the relation of one recent feature with multiscale complete features. Based on GFL and MCRFA, the HCR model can hierarchically explore the rich interrelationships between multiscale complete features and multiscale recent features. Through comparative and ablation experiments, we validate the effectiveness of our model on two well-known public datasets: EPIC-Kitchens and EGTEA Gaze+.
A Hybrid Transformer-Sequencer approach for Age and Gender classification from in-wild facial images
Venue: arXiv cs.CV 2024
Year: 2024

The advancements in computer vision and image processing techniques have led to emergence of new application in the domain of visual surveillance, targeted advertisement, content-based searching, and human-computer interaction etc. Out of the various techniques in computer vision, face analysis, in particular, has gained much attention. Several previous studies have tried to explore different applications of facial feature processing for a variety of tasks, including age and gender classification. However, despite several previous studies having explored the problem, the age and gender classification of in-wild human faces is still far from the achieving the desired levels of accuracy required for real-world applications. This paper, therefore, attempts to bridge this gap by proposing a hybrid model that combines self-attention and BiLSTM approaches for age and gender classification problems. The proposed models performance is compared with several state-of-the-art model proposed so far. An improvement of approximately 10percent and 6percent over the state-of-the-art implementations for age and gender classification, respectively, are noted for the proposed model. The proposed model is thus found to achieve superior performance and is found to provide a more generalized learning. The model can, therefore, be applied as a core classification component in various image processing and computer vision problems.
WMMSE-Based Rate Maximization for RIS-Assisted MU-MIMO Systems
Venue: arXiv cs.IT 2024
Year: 2024

Reconfigurable intelligent surface (RIS) technology, given its ability to favorably modify wireless communication environments, will play a pivotal role in the evolution of future communication systems. This paper proposes rate maximization techniques for both single-user and multiuser MIMO systems, based on the well-known weighted minimum mean square error (WMMSE) criterion. Using a suitable weight matrix, the WMMSE algorithm tackles an equivalent weighted mean square error (WMSE) minimization problem to achieve the sum-rate maximization. By considering a more practical RIS system model that employs a tensor-based representation enforced by the electromagnetic behavior exhibited by the RIS panel, we detail both the sum-rate maximizing and WMSE minimizing strategies for RIS phase shift optimization by deriving the closed-form gradient of the WMSE and the sum-rate with respect to the RIS phase shift vector. Our simulations reveal that the proposed rate maximization technique, rooted in the WMMSE algorithm, exhibits superior performance when compared to other benchmarks.