OpenRaiser/Intern-Atlas huggingface dataset:
MotionTrace: IMU-based Field of View Prediction for Smartphone AR Interactions
Venue: arXiv cs.HC 2024
Year: 2024

For handheld smartphone AR interactions, bandwidth is a critical constraint. Streaming techniques have been developed to provide a seamless and high-quality user experience despite these challenges. To optimize streaming performance in smartphone-based AR, accurate prediction of the user's field of view is essential. This prediction allows the system to prioritize loading digital content that the user is likely to engage with, enhancing the overall interactivity and immersion of the AR experience. In this paper, we present MotionTrace, a method for predicting the user's field of view using a smartphone's inertial sensor. This method continuously estimates the user's hand position in 3D-space to localize the phone position. We evaluated MotionTrace over future hand positions at 50, 100, 200, 400, and 800ms time horizons using the large motion capture (AMASS) and smartphone-based full-body pose estimation (Pose-on-the-Go) datasets. We found that our method can estimate the future phone position of the user with an average MSE between 0.11 - 143.62 mm across different time horizons.
Sólo Escúchame: Spanish Emotional Accompaniment Chatbot
Venue: arXiv cs.CL 2024
Year: 2024

According to the World Health Organization (WHO), suicide was the fourth leading cause of death in the world for individuals aged 15 to 29 in 2019. Given the rapid increase in mental health issues, providing psychological support is both crucial and urgent. In this paper: (1) we propose Sólo Escúchame, the first open-source Spanish emotional assistance chatbot, based on LLaMA-2-7b-Chat. (2) We introduced the HEAR (Hispanic Emotional Accompaniment Responses) dataset, compiled from multiple English sources translated into Spanish, as well as generic data generated using ChatGPT-3.5-Turbo. Finally, (3) we propose an evaluation metric based on two semi-automatic assessment methods. Our system outperforms a range of state-of-the-art models in providing psychological assistance in Spanish. Our models and datasets are publicly available to facilitate reproducibility.
Graph Unfolding and Sampling for Transitory Video Summarization via Gershgorin Disc Alignment
Venue: arXiv cs.CV 2024
Year: 2024

User-generated videos (UGVs) uploaded from mobile phones to social media sites like YouTube and TikTok are short and non-repetitive. We summarize a transitory UGV into several keyframes in linear time via fast graph sampling based on Gershgorin disc alignment (GDA). Specifically, we first model a sequence of $N$ frames in a UGV as an $M$-hop path graph $\mathcal{G}^o$ for $M \ll N$, where the similarity between two frames within $M$ time instants is encoded as a positive edge based on feature similarity. Towards efficient sampling, we then "unfold" $\mathcal{G}^o$ to a $1$-hop path graph $\mathcal{G}$, specified by a generalized graph Laplacian matrix $\mathcal{L}$, via one of two graph unfolding procedures with provable performance bounds. We show that maximizing the smallest eigenvalue $λ_{\min}(\mathbf{B})$ of a coefficient matrix $\mathbf{B} = \textit{diag}\left(\mathbf{h}\right) + μ\mathcal{L}$, where $\mathbf{h}$ is the binary keyframe selection vector, is equivalent to minimizing a worst-case signal reconstruction error. We maximize instead the Gershgorin circle theorem (GCT) lower bound $λ^-_{\min}(\mathbf{B})$ by choosing $\mathbf{h}$ via a new fast graph sampling algorithm that iteratively aligns left-ends of Gershgorin discs for all graph nodes (frames). Extensive experiments on multiple short video datasets show that our algorithm achieves comparable or better video summarization performance compared to state-of-the-art methods, at a substantially reduced complexity.
Batch Active Learning in Gaussian Process Regression using Derivatives
Venue: arXiv cs.LG 2024
Year: 2024

We investigate the use of derivative information for Batch Active Learning in Gaussian Process regression models. The proposed approach employs the predictive covariance matrix for selection of data batches to exploit full correlation of samples. We theoretically analyse our proposed algorithm taking different optimality criteria into consideration and provide empirical comparisons highlighting the advantage of incorporating derivatives information. Our results show the effectiveness of our approach across diverse applications.
An Overview of Integration of the Virtualization of Network Functions in the Context of Information Centric Networks
Venue: arXiv cs.NI 2024
Year: 2024

Nowadays, the new network architectures with respect to the traditional network topologies must manage more data, which entails having increasingly robust and scalable network structures. As there is a process of growth, adaptability, and change in traditional data networks, faced with the management of large volumes of information, it is necessary to incorporate the virtualization of network functions in the context of information content networks, in such a way that there is a balance between the user and the provider at cost level and profit, the functions on the network. In turn, NFVs (Network Functions Virtualization) are considered network structures designed based on IT virtualization technologies, which allow virtualizing of the functions that can be found in the network nodes, which are connected through routing tables, which allows offering communication services for various types of customers. Therefore, information-centric networks (IC, unlike traditional data networks which proceed to exchange information between hosts using data packets and TCP/IP communication protocols, use the content of the data for this purpose where the data travels through the network is stored in a routing table located in the CR (Content Router) of the router temporality to be reused later, which allows for reducing operation costs and capital costs . The purpose of this work is to analyze how the virtualization of network functions is integrated into the field of information-centric networks. Also, the advantages and disadvantages of both architectures are considered and presented as a critical analysis when considering the current difficulties and future trends of both network topologies.
User Experience of Visualizations in Motion: A Case Study and Design Considerations
Venue: arXiv cs.HC 2024
Year: 2024

We present a systematic review, an empirical study, and a first set of considerations for designing visualizations in motion, derived from a concrete scenario in which these visualizations were used to support a primary task. In practice, when viewers are confronted with embedded visualizations, they often have to focus on a primary task and can only quickly glance at a visualization showing rich, often dynamically updated, information. As such, the visualizations must be designed so as not to distract from the primary task, while at the same time being readable and useful for aiding the primary task. For example, in games, players who are engaged in a battle have to look at their enemies but also read the remaining health of their own game character from the health bar over their character's head. Many trade-offs are possible in the design of embedded visualizations in such dynamic scenarios, which we explore in-depth in this paper with a focus on user experience. We use video games as an example of an application context with a rich existing set of visualizations in motion. We begin our work with a systematic review of in-game visualizations in motion. Next, we conduct an empirical user study to investigate how different embedded visualizations in motion designs impact user experience. We conclude with a set of considerations and trade-offs for designing visualizations in motion more broadly as derived from what we learned about video games. All supplemental materials of this paper are available at https://osf.io/3v8wm/}.
PanicleNeRF: low-cost, high-precision in-field phenotypingof rice panicles with smartphone
Venue: arXiv cs.CV 2024
Year: 2024

The rice panicle traits significantly influence grain yield, making them a primary target for rice phenotyping studies. However, most existing techniques are limited to controlled indoor environments and difficult to capture the rice panicle traits under natural growth conditions. Here, we developed PanicleNeRF, a novel method that enables high-precision and low-cost reconstruction of rice panicle three-dimensional (3D) models in the field using smartphone. The proposed method combined the large model Segment Anything Model (SAM) and the small model You Only Look Once version 8 (YOLOv8) to achieve high-precision segmentation of rice panicle images. The NeRF technique was then employed for 3D reconstruction using the images with 2D segmentation. Finally, the resulting point clouds are processed to successfully extract panicle traits. The results show that PanicleNeRF effectively addressed the 2D image segmentation task, achieving a mean F1 Score of 86.9% and a mean Intersection over Union (IoU) of 79.8%, with nearly double the boundary overlap (BO) performance compared to YOLOv8. As for point cloud quality, PanicleNeRF significantly outperformed traditional SfM-MVS (structure-from-motion and multi-view stereo) methods, such as COLMAP and Metashape. The panicle length was then accurately extracted with the rRMSE of 2.94% for indica and 1.75% for japonica rice. The panicle volume estimated from 3D point clouds strongly correlated with the grain number (R2 = 0.85 for indica and 0.82 for japonica) and grain mass (0.80 for indica and 0.76 for japonica). This method provides a low-cost solution for high-throughput in-field phenotyping of rice panicles, accelerating the efficiency of rice breeding.
MedSyn: LLM-based Synthetic Medical Text Generation Framework
Venue: arXiv cs.CL 2024
Year: 2024

Generating synthetic text addresses the challenge of data availability in privacy-sensitive domains such as healthcare. This study explores the applicability of synthetic data in real-world medical settings. We introduce MedSyn, a novel medical text generation framework that integrates large language models with a Medical Knowledge Graph (MKG). We use MKG to sample prior medical information for the prompt and generate synthetic clinical notes with GPT-4 and fine-tuned LLaMA models. We assess the benefit of synthetic data through application in the ICD code prediction task. Our research indicates that synthetic data can increase the classification accuracy of vital and challenging codes by up to 17.8% compared to settings without synthetic data. Furthermore, to provide new data for further research in the healthcare domain, we present the largest open-source synthetic dataset of clinical notes for the Russian language, comprising over 41k samples covering 219 ICD-10 codes.
A Demand-aware Networked System Using Telemetry and ML with ReactNET
Venue: arXiv cs.NI 2024
Year: 2024

Emerging network applications ranging from video streaming to virtual/augmented reality need to provide stringent quality-of-service (QoS) guarantees in complex and dynamic environments with shared resources. A promising approach to meeting these requirements is to automate complex network operations and create self-adjusting networks. These networks should automatically gather contextual information, analyze how to efficiently ensure QoS requirements, and adapt accordingly. This paper presents ReactNET, a self-adjusting networked system designed to achieve this vision by leveraging emerging network programmability and machine learning techniques. Programmability empowers ReactNET by providing fine-grained telemetry information, while machine learning-based classification techniques enable the system to learn and adjust the network to changing conditions. Our preliminary implementation of ReactNET in P4 and Python demonstrates its effectiveness in video streaming applications.
PromptSAM+: Malware Detection based on Prompt Segment Anything Model
Venue: arXiv cs.CR 2024
Year: 2024

Machine learning and deep learning (ML/DL) have been extensively applied in malware detection, and some existing methods demonstrate robust performance. However, several issues persist in the field of malware detection: (1) Existing work often overemphasizes accuracy at the expense of practicality, rarely considering false positive and false negative rates as important metrics. (2) Considering the evolution of malware, the performance of classifiers significantly declines over time, greatly reducing the practicality of malware detectors. (3) Prior ML/DL-based efforts heavily rely on ample labeled data for model training, largely dependent on feature engineering or domain knowledge to build feature databases, making them vulnerable if correct labels are scarce. With the development of computer vision, vision-based malware detection technology has also rapidly evolved. In this paper, we propose a visual malware general enhancement classification framework, `PromptSAM+', based on a large visual network segmentation model, the Prompt Segment Anything Model(named PromptSAM+). Our experimental results indicate that 'PromptSAM+' is effective and efficient in malware detection and classification, achieving high accuracy and low rates of false positives and negatives. The proposed method outperforms the most advanced image-based malware detection technologies on several datasets. 'PromptSAM+' can mitigate aging in existing image-based malware classifiers, reducing the considerable manpower needed for labeling new malware samples through active learning. We conducted experiments on datasets for both Windows and Android platforms, achieving favorable outcomes. Additionally, our ablation experiments on several datasets demonstrate that our model identifies effective modules within the large visual network.
Opportunities and Challenges of Urban Agetech: from an Automated City to an Ageing-Friendly City
Venue: arXiv cs.CY 2024
Year: 2024

Caring for the elderly, aging-in-place, and enabling the elderly to maintain a good life continue to be topics of increasing importance, especially in countries with a higher percentage of older people, as people live longer, and care-giving costs rise. This position paper proposes the concept of urban agetech, where agetech services beyond the home can be an integral part of a modern ageing-friendly city, and where support for the elderly, where needed, in the form of automated systems (e.g., robots and automated vehicles) would be a normal city function/service, akin to the rather commonplace public transport services today.
FDiff-Fusion:Denoising diffusion fusion network based on fuzzy learning for 3D medical image segmentation
Venue: arXiv cs.CV 2024
Year: 2024

In recent years, the denoising diffusion model has achieved remarkable success in image segmentation modeling. With its powerful nonlinear modeling capabilities and superior generalization performance, denoising diffusion models have gradually been applied to medical image segmentation tasks, bringing new perspectives and methods to this field. However, existing methods overlook the uncertainty of segmentation boundaries and the fuzziness of regions, resulting in the instability and inaccuracy of the segmentation results. To solve this problem, a denoising diffusion fusion network based on fuzzy learning for 3D medical image segmentation (FDiff-Fusion) is proposed in this paper. By integrating the denoising diffusion model into the classical U-Net network, this model can effectively extract rich semantic information from input medical images, thus providing excellent pixel-level representation for medical image segmentation. ... Finally, to validate the effectiveness of FDiff-Fusion, we compare it with existing advanced segmentation networks on the BRATS 2020 brain tumor dataset and the BTCV abdominal multi-organ dataset. The results show that FDiff-Fusion significantly improves the Dice scores and HD95 distance on these two datasets, demonstrating its superiority in medical image segmentation tasks.
Improving Neural Surface Reconstruction with Feature Priors from Multi-View Image
Venue: arXiv cs.CV 2024
Year: 2024

Recent advancements in Neural Surface Reconstruction (NSR) have significantly improved multi-view reconstruction when coupled with volume rendering. However, relying solely on photometric consistency in image space falls short of addressing complexities posed by real-world data, including occlusions and non-Lambertian surfaces. To tackle these challenges, we propose an investigation into feature-level consistent loss, aiming to harness valuable feature priors from diverse pretext visual tasks and overcome current limitations. It is crucial to note the existing gap in determining the most effective pretext visual task for enhancing NSR. In this study, we comprehensively explore multi-view feature priors from seven pretext visual tasks, comprising thirteen methods. Our main goal is to strengthen NSR training by considering a wide range of possibilities. Additionally, we examine the impact of varying feature resolutions and evaluate both pixel-wise and patch-wise consistent losses, providing insights into effective strategies for improving NSR performance. By incorporating pre-trained representations from MVSFormer and QuadTree, our approach can generate variations of MVS-NeuS and Match-NeuS, respectively. Our results, analyzed on DTU and EPFL datasets, reveal that feature priors from image matching and multi-view stereo outperform other pretext tasks. Moreover, we discover that extending patch-wise photometric consistency to the feature level surpasses the performance of pixel-wise approaches. These findings underscore the effectiveness of these techniques in enhancing NSR outcomes.
First Order Stochastic Optimization with Oblivious Noise
Venue: arXiv cs.DS 2024
Year: 2024

We initiate the study of stochastic optimization with oblivious noise, broadly generalizing the standard heavy-tailed noise setup. In our setting, in addition to random observation noise, the stochastic gradient may be subject to independent oblivious noise, which may not have bounded moments and is not necessarily centered. Specifically, we assume access to a noisy oracle for the stochastic gradient of $f$ at $x$, which returns a vector $\nabla f(γ, x) + ξ$, where $γ$ is the bounded variance observation noise and $ξ$ is the oblivious noise that is independent of $γ$ and $x$. The only assumption we make on the oblivious noise $ξ$ is that $\mathbf{Pr}[ξ= 0] \ge α$ for some $α\in (0, 1)$. In this setting, it is not information-theoretically possible to recover a single solution close to the target when the fraction of inliers $α$ is less than $1/2$. Our main result is an efficient list-decodable learner that recovers a small list of candidates, at least one of which is close to the true solution. On the other hand, if $α= 1-ε$, where $0< ε< 1/2$ is sufficiently small constant, the algorithm recovers a single solution. Along the way, we develop a rejection-sampling-based algorithm to perform noisy location estimation, which may be of independent interest.
Abstraction in Neural Networks
Venue: arXiv cs.NE 2024
Year: 2024

We show how brain networks, modeled as Spiking Neural Networks, can be viewed at different levels of abstraction. Lower levels include complications such as failures of neurons and edges. Higher levels are more abstract, making simplifying assumptions to avoid these complications. We show precise relationships between executions of networks at different levels, which enables us to understand the behavior of lower-level networks in terms of the behavior of higher-level networks. We express our results using two abstract networks, A1 and A2, one to express firing guarantees and the other to express non-firing guarantees, and one detailed network D. The abstract networks contain reliable neurons and edges, whereas the detailed network has neurons and edges that may fail, subject to some constraints. Here we consider just initial stopping failures. To define these networks, we begin with abstract network A1 and modify it systematically to obtain the other two networks. To obtain A2, we simply lower the firing thresholds of the neurons. To obtain D, we introduce failures of neurons and edges, and incorporate redundancy in the neurons and edges in order to compensate for the failures. We also define corresponding inputs for the networks, and corresponding executions of the networks. We prove two main theorems, one relating corresponding executions of A1 and D and the other relating corresponding executions of A2 and D. Together, these give both firing and non-firing guarantees for the detailed network D. We also give a third theorem, relating the effects of D on an external reliable actuator neuron to the effects of the abstract networks on the same actuator neuron.
OntoForms: User interface structure from a domain ontology
Venue: arXiv cs.SE 2024
Year: 2024

This paper presents a software component that generates a user interface structure for populating a domain ontology. The core of this work is an algorithm that takes an ontology and returns a structure describing the user interface. The component also provides functions for populating the ontology and editing existing individuals. Unlike previous approaches, this method can be implemented without any configuration. Additionally, it offers an easy-to-use configuration mechanism that allows irrelevant classes to be hidden and automatically populated. What distinguishes this work is that, instead of exploring the ontology using syntactic methods or queries, our algorithm employs services that implement description logic inference mechanisms. This work illustrates the proposed approach using the well-known wine ontology.
ExoViP: Step-by-step Verification and Exploration with Exoskeleton Modules for Compositional Visual Reasoning
Venue: arXiv cs.CV 2024
Year: 2024

Compositional visual reasoning methods, which translate a complex query into a structured composition of feasible visual tasks, have exhibited a strong potential in complicated multi-modal tasks. Empowered by recent advances in large language models (LLMs), this multi-modal challenge has been brought to a new stage by treating LLMs as few-shot/zero-shot planners, i.e., vision-language (VL) programming. Such methods, despite their numerous merits, suffer from challenges due to LLM planning mistakes or inaccuracy of visual execution modules, lagging behind the non-compositional models. In this work, we devise a "plug-and-play" method, ExoViP, to correct errors in both the planning and execution stages through introspective verification. We employ verification modules as "exoskeletons" to enhance current VL programming schemes. Specifically, our proposed verification module utilizes a mixture of three sub-verifiers to validate predictions after each reasoning step, subsequently calibrating the visual module predictions and refining the reasoning trace planned by LLMs. Experimental results on two representative VL programming methods showcase consistent improvements on five compositional reasoning tasks on standard benchmarks. In light of this, we believe that ExoViP can foster better performance and generalization on open-domain multi-modal challenges.