Miracle-dz/UniFAR-FTUs huggingface dataset:
DriveTrack: A Benchmark for Long-Range Point Tracking in Real-World Videos

This paper presents DriveTrack, a new benchmark and data generation framework for long-range keypoint tracking in real-world videos. DriveTrack is motivated by the observation that the accuracy of state-of-the-art trackers depends strongly on visual attributes around the selected keypoints, such as texture and lighting. The problem is that these artifacts are especially pronounced in real-world videos, but these trackers are unable to train on such scenes due to a dearth of annotations. DriveTrack bridges this gap by building a framework to automatically annotate point tracks on autonomous driving datasets. We release a dataset consisting of 1 billion point tracks across 24 hours of video, which is seven orders of magnitude greater than prior real-world benchmarks and on par with the scale of synthetic benchmarks. DriveTrack unlocks new use cases for point tracking in real-world videos. First, we show that fine-tuning keypoint trackers on DriveTrack improves accuracy on real-world scenes by up to 7%. Second, we analyze the sensitivity of trackers to visual artifacts in real scenes and motivate the idea of running assistive keypoint selectors alongside trackers.
Concept Prerequisite Relation Prediction by Using Permutation-Equivariant Directed Graph Neural Networks

This paper studies the problem of CPRP, concept prerequisite relation prediction, which is a fundamental task in using AI for education. CPRP is usually formulated into a link-prediction task on a relationship graph of concepts and solved by training the graph neural network (GNN) model. However, current directed GNNs fail to manage graph isomorphism which refers to the invariance of non-isomorphic graphs, reducing the expressivity of resulting representations. We present a permutation-equivariant directed GNN model by introducing the Weisfeiler-Lehman test into directed GNN learning. Our method is then used for CPRP and evaluated on three public datasets. The experimental results show that our model delivers better prediction performance than the state-of-the-art methods.
SeeThruFinger: See and Grasp Anything with a Soft Touch

We present SeeThruFinger, a soft robotic finger with an in-finger vision for multi-modal perception, including visual perception and tactile sensing, for geometrically adaptive and real-time reactive grasping. Multi-modal perception of intrinsic and extrinsic interactions is critical in building intelligent robots that learn. Instead of adding various sensors for different modalities, a preferred solution is to integrate them into one elegant and coherent design, which is a challenging task. This study leverages the Soft Polyhedral Network design as a robotic finger, capable of omni-directional adaptation with an unobstructed view of the finger's spatial deformation from the inside. By embedding a miniature camera underneath, we achieve the visual perception of the external environment by inpainting the finger mask using E2FGV, which can be used for object detection in the downstream tasks for grasping. After contacting the objects, we use real-time object segmentation algorithms, such as XMem, to track the soft finger's spatial deformations. We also learned a Supervised Variational Autoencoder to enable tactile sensing of 6D forces and torques for reactive grasp. As a result, we achieved multi-modal perception, including visual perception and tactile sensing, and soft, adaptive object grasping within a single vision-based soft finger design compatible with multi-fingered robotic grippers.
Nonlinear enhancement of coherent magnetization dynamics

Magnets are interesting materials for classical and quantum information technologies. However, the short decoherence and dephasing times that determine the scale and speed of information networks, severely limit the appeal of employing the ferromagnetic resonance. Here we show that the lifetime and coherence of the uniform Kittel mode can be enhanced by 3-magnon interaction-induced mixing with the long-lived magnons at the minima of the dispersion relation. Analytical and numerical calculations based on this model explain recent experimental results and predict experimental signatures of quantum coherence.
A new lower bound for sphere packing

We show there exists a packing of identical spheres in $\mathbb{R}^d$ with density at least \[ (1-o(1))\frac{d \log d}{2^{d+1}}\, , \] as $d\to\infty$. This improves upon previous bounds for general $d$ by a factor of order $\log d$ and is the first asymptotically growing improvement to Rogers' bound from 1947.
Deep Unsupervised Domain Adaptation for Time Series Classification: a Benchmark

Unsupervised Domain Adaptation (UDA) aims to harness labeled source data to train models for unlabeled target data. Despite extensive research in domains like computer vision and natural language processing, UDA remains underexplored for time series data, which has widespread real-world applications ranging from medicine and manufacturing to earth observation and human activity recognition. Our paper addresses this gap by introducing a comprehensive benchmark for evaluating UDA techniques for time series classification, with a focus on deep learning methods. We provide seven new benchmark datasets covering various domain shifts and temporal dynamics, facilitating fair and standardized UDA method assessments with state of the art neural network backbones (e.g. Inception) for time series data. This benchmark offers insights into the strengths and limitations of the evaluated approaches while preserving the unsupervised nature of domain adaptation, making it directly applicable to practical problems. Our paper serves as a vital resource for researchers and practitioners, advancing domain adaptation solutions for time series data and fostering innovation in this critical field. The implementation code of this benchmark is available at https://github.com/EricssonResearch/UDA-4-TSC.
Assume-Guarantee Reinforcement Learning

We present a modular approach to reinforcement learning (RL) in environments consisting of simpler components evolving in parallel. A monolithic view of such modular environments may be prohibitively large to learn, or may require unrealizable communication between the components in the form of a centralized controller. Our proposed approach is based on the assume-guarantee paradigm where the optimal control for the individual components is synthesized in isolation by making assumptions about the behaviors of neighboring components, and providing guarantees about their own behavior. We express these assume-guarantee contracts as regular languages and provide automatic translations to scalar rewards to be used in RL. By combining local probabilities of satisfaction for each component, we provide a lower bound on the probability of satisfaction of the complete system. By solving a Markov game for each component, RL can produce a controller for each component that maximizes this lower bound. The controller utilizes the information it receives through communication, observations, and any knowledge of a coarse model of other agents. We experimentally demonstrate the efficiency of the proposed approach on a variety of case studies.
On the optimal arrangement of $2d$ lines in $\mathbb{C}^d$

We show the optimal coherence of $2d$ lines in $\mathbb{C}^{d}$ is given by the Welch bound whenever a skew Hadamard of order $d+1$ exists. Our proof uses a variant of Hadamard doubling that converts any equiangular tight frame of size $\tfrac{d-1}{2} \times d$ into another one of size $d \times 2d$. Among $d<150$, this produces equiangular tight frames of new sizes when $d = 11$, $35$, $39$, $43$, $47$, $59$, $67$, $71$, $83$, $95$, $103$, $107$, $111$, $119$, $123$, $127$, $131$, and $143$.
Diagnosing and Rectifying Fake OOD Invariance: A Restructured Causal Approach

Invariant representation learning (IRL) encourages the prediction from invariant causal features to labels deconfounded from the environments, advancing the technical roadmap of out-of-distribution (OOD) generalization. Despite spotlights around, recent theoretical result verified that some causal features recovered by IRLs merely pretend domain-invariantly in the training environments but fail in unseen domains. The fake invariance severely endangers OOD generalization since the trustful objective can not be diagnosed and existing causal remedies are invalid to rectify. In this paper, we review a IRL family (InvRat) under the Partially and Fully Informative Invariant Feature Structural Causal Models (PIIF SCM /FIIF SCM) respectively, to certify their weaknesses in representing fake invariant features, then, unify their causal diagrams to propose ReStructured SCM (RS-SCM). RS-SCM can ideally rebuild the spurious and the fake invariant features simultaneously. Given this, we further develop an approach based on conditional mutual information with respect to RS-SCM, then rigorously rectify the spurious and fake invariant effects. It can be easily implemented by a small feature selection subnet introduced in the IRL family, which is alternatively optimized to achieve our goal. Experiments verified the superiority of our approach to fight against the fake invariant issue across a variety of OOD generalization benchmarks.
How Does User Behavior Evolve During Exploratory Visual Analysis?

Exploratory visual analysis (EVA) is an essential stage of the data science pipeline, where users often lack clear analysis goals at the start and iteratively refine them as they learn more about their data. Accurate models of users' exploration behavior are becoming increasingly vital to developing responsive and personalized tools for exploratory visual analysis. Yet we observe a discrepancy between the static view of human exploration behavior adopted by many computational models versus the dynamic nature of EVA. In this paper, we explore potential parallels between the evolution of users' interactions with visualization tools during data exploration and assumptions made in popular online learning techniques. Through a series of empirical analyses, we seek to answer the question: how might users' exploration behavior evolve in response to what they have learned from the data during EVA? We present our findings and discuss their implications for the future of user modeling for system design.
Benchmarking the Full-Order Model Optimization Based Imitation in the Humanoid Robot Reinforcement Learning Walk

When a gait of a bipedal robot is developed using deep reinforcement learning, reference trajectories may or may not be used. Each approach has its advantages and disadvantages, and the choice of method is up to the control developer. This paper investigates the effect of reference trajectories on locomotion learning and the resulting gaits. We implemented three gaits of a full-order anthropomorphic robot model with different reward imitation ratios, provided sim-to-sim control policy transfer, and compared the gaits in terms of robustness and energy efficiency. In addition, we conducted a qualitative analysis of the gaits by interviewing people, since our task was to create an appealing and natural gait for a humanoid robot. According to the results of the experiments, the most successful approach was the one in which the average value of rewards for imitation and adherence to command velocity per episode remained balanced throughout the training. The gait obtained with this method retains naturalness (median of 3.6 according to the user study) compared to the gait trained with imitation only (median of 4.0), while remaining robust close to the gait trained without reference trajectories.
Detecting Grasping Sites in a Martian Lava Tube: Multi-Stage Perception Trade Study for ReachBo

This paper presents a trade study analysis to design and evaluate the perception system architecture for ReachBot. ReachBot is a novel robotic concept that uses grippers at the end of deployable booms for navigation of rough terrain such as walls of caves and lava tubes. Previous studies on ReachBot have discussed the overall robot design, placement and number of deployable booms, and gripper mechanism design; however, analysis of the perception and sensing system remains underdeveloped. Because ReachBot can extend and interact with terrain over long distances on the order of several meters, a robust perception and sensing strategy is crucial to identify grasping locations and enable fully autonomous operation. This trade study focuses on developing the perception trade space and realizing such perception capabilities for a physical prototype. This work includes analysis of: (1) multiple-range sensing strategies for ReachBot, (2) sensor technologies for subsurface climbing robotics, (3) criteria for sensor evaluation, (4) positions and modalities of sensors on ReachBot, and (5) map representations of grasping locations. From our analysis, we identify the overall perception strategy and hardware configuration for a fully-instrumented case study mission to a Martian lava tube, and identify specific sensors for a hardware prototype. The final result of our trade study is a system design conducive to benchtop testing and prototype hardware development.
CGS-Mask: Making Time Series Predictions Intuitive for All

Artificial intelligence (AI) has immense potential in time series prediction, but most explainable tools have limited capabilities in providing a systematic understanding of important features over time. These tools typically rely on evaluating a single time point, overlook the time ordering of inputs, and neglect the time-sensitive nature of time series applications. These factors make it difficult for users, particularly those without domain knowledge, to comprehend AI model decisions and obtain meaningful explanations. We propose CGS-Mask, a post-hoc and model-agnostic cellular genetic strip mask-based saliency approach to address these challenges. CGS-Mask uses consecutive time steps as a cohesive entity to evaluate the impact of features on the final prediction, providing binary and sustained feature importance scores over time. Our algorithm optimizes the mask population iteratively to obtain the optimal mask in a reasonable time. We evaluated CGS-Mask on synthetic and real-world datasets, and it outperformed state-of-the-art methods in elucidating the importance of features over time. According to our pilot user study via a questionnaire survey, CGS-Mask is the most effective approach in presenting easily understandable time series prediction results, enabling users to comprehend the decision-making process of AI models with ease.
System Integration of Xilinx DPU and HDMI for Real-Time inference in PYNQ Environment with Image Enhancement

Use of edge computing in application of Computer Vision (CV) is an active field of research. Today, most CV applications make use of Convolutional Neural Networks (CNNs) to inference on and interpret video data. These edge devices are responsible for several CV related tasks, such as gathering, processing and enhancing, inferencing on, and displaying video data. Due to ease of reconfiguration, computation on FPGA fabric is used to achieve such complex computation tasks. Xilinx provides the PYNQ environment as a user-friendly interface that facilitates in Hardware/Software system integration. However, to the best of authors' knowledge there is no end-to-end framework available for the PYNQ environment that allows Hardware/Software system integration and deployment of CNNs for real-time input feed from High Definition Multimedia Interface (HDMI) input to HDMI output, along with insertion of customized hardware IPs. In this work we propose an integration of rea\textbf{L}-time image \textbf{E}nancement IP with \textbf{A}I inferencing engine in the \textbf{P}ynq environment (\textbf{LEAP}), that integrates HDMI, AI acceleration, image enhancement in the PYNQ environment for Xilinx's Microprocessor on Chip (MPSoC) platform. We evaluate our methodology with two well know CNN models, Resnet50 and YOLOv3. To validate our proposed methodology, LEAP, a simple image enhancement algorithm, histogram equalization, is designed and integrated in the FPGA fabric along with Xilinx's Deep Processing Unit (DPU). Our results show successful implementation of end-to-end integration using completely open source information.
TF-CLIP: Learning Text-free CLIP for Video-based Person Re-Identification

Large-scale language-image pre-trained models (e.g., CLIP) have shown superior performances on many cross-modal retrieval tasks. However, the problem of transferring the knowledge learned from such models to video-based person re-identification (ReID) has barely been explored. In addition, there is a lack of decent text descriptions in current ReID benchmarks. To address these issues, in this work, we propose a novel one-stage text-free CLIP-based learning framework named TF-CLIP for video-based person ReID. More specifically, we extract the identity-specific sequence feature as the CLIP-Memory to replace the text feature. Meanwhile, we design a Sequence-Specific Prompt (SSP) module to update the CLIP-Memory online. To capture temporal information, we further propose a Temporal Memory Diffusion (TMD) module, which consists of two key components: Temporal Memory Construction (TMC) and Memory Diffusion (MD). Technically, TMC allows the frame-level memories in a sequence to communicate with each other, and to extract temporal information based on the relations within the sequence. MD further diffuses the temporal memories to each token in the original features to obtain more robust sequence features. Extensive experiments demonstrate that our proposed method shows much better results than other state-of-the-art methods on MARS, LS-VID and iLIDS-VID.
Integral Fractional Ornstein-Uhlenbeck Process Model for Animal Movement

Modeling the trajectories of animals is challenging due to the complexity of their behaviors, the influence of unpredictable environmental factors, individual variability, and the lack of detailed data on their movements. Additionally, factors such as migration, hunting, reproduction, and social interactions add additional layers of complexity when attempting to accurately forecast their movements. In the literature, various models exits that aim to study animal telemetry, by modeling the velocity of the telemetry, the telemetry itself or both processes jointly through a Markovian process. In this work, we propose to model the velocity of each coordinate axis for animal telemetry data as a fractional Ornstein-Uhlenbeck (fOU) process. Then, the integral fOU process models position data in animal telemetry. Compared to traditional methods, the proposed model is flexible in modeling long-range memory. The Hurst parameter $H \in (0,1)$ is a crucial parameter in integral fOU process, as it determines the degree of dependence or long-range memory. The integral fOU process is nonstationary process. In addition, a higher Hurst parameter ($H>0.5$) indicates a stronger memory, leading to trajectories with transient trends, while a lower Hurst parameter ($H<0.5$) implies a weaker memory, resulting in trajectories with recurring trends. When H = 0.5, the process reduces to a standard integral Ornstein-Uhlenbeck process. We develop a fast simulation algorithm of telemetry trajectories using an approach via finite-dimensional distributions. We also develop a maximum likelihood method for parameter estimation and its performance is examined by simulation studies. Finally, we present a telemetry application of Fin Whales that disperse over the Gulf of Mexico.
Bayesian Inference of Initial Conditions from Non-Linear Cosmic Structures using Field-Level Emulators

Analysing next-generation cosmological data requires balancing accurate modeling of non-linear gravitational structure formation and computational demands. We propose a solution by introducing a machine learning-based field-level emulator, within the Hamiltonian Monte Carlo-based Bayesian Origin Reconstruction from Galaxies (BORG) inference algorithm. Built on a V-net neural network architecture, the emulator enhances the predictions by first-order Lagrangian perturbation theory to be accurately aligned with full $N$-body simulations while significantly reducing evaluation time. We test its incorporation in BORG for sampling cosmic initial conditions using mock data based on non-linear large-scale structures from $N$-body simulations and Gaussian noise. The method efficiently and accurately explores the high-dimensional parameter space of initial conditions, fully extracting the cross-correlation information of the data field binned at a resolution of $1.95h^{-1}$ Mpc. Percent-level agreement with the ground truth in the power spectrum and bispectrum is achieved up to the Nyquist frequency $k_\mathrm{N} \approx 2.79h \; \mathrm{Mpc}^{-1}$. Posterior resimulations - using the inferred initial conditions for $N$-body simulations - show that the recovery of information in the initial conditions is sufficient to accurately reproduce halo properties. In particular, we show highly accurate $M_{200\mathrm{c}}$ halo mass function and stacked density profiles of haloes in different mass bins $[0.853,16]\times 10^{14}M_{\odot}h^{-1}$. As all available cross-correlation information is extracted, we acknowledge that limitations in recovering the initial conditions stem from the noise level and data grid resolution. This is promising as it underscores the significance of accurate non-linear modeling, indicating the potential for extracting additional information at smaller scales.
YouTube Video Analytics for Patient Health Literacy: Evidence from Colonoscopy Preparation Videos

Videos can be an effective way to deliver contextualized, just-in-time medical information for patient education. However, video analysis, from topic identification and retrieval to extraction and analysis of medical information and understandability from a patient perspective are extremely challenging tasks. This study utilizes data analysis methods to retrieve medical information from YouTube videos concerning colonoscopy to manage health conditions. We first use the YouTube Data API to collect metadata of desired videos on select search keywords and use Google Video Intelligence API to analyze texts, frames and objects data. Then we annotate the YouTube video materials on medical information, video understandability annotation and recommendation. We develop a bidirectional long short-term memory (BLSTM) model to identify medical terms in videos and build three classifiers to group videos based on the level of encoded medical information, video understandability level and whether the videos are recommended. Our study provides healthcare practitioners and patients with guidelines for generating new educational video content and enabling management of health conditions.