OpenRaiser/Intern-Atlas huggingface dataset:
Soft Hoeffding Tree: A Transparent and Differentiable Model on Data Streams
Venue: arXiv cs.LG 2024
Year: 2024

We propose soft Hoeffding trees (SoHoT) as a new differentiable and transparent model for possibly infinite and changing data streams. Stream mining algorithms such as Hoeffding trees grow based on the incoming data stream, but they currently lack the adaptability of end-to-end deep learning systems. End-to-end learning can be desirable if a feature representation is learned by a neural network and used in a tree, or if the outputs of trees are further processed in a deep learning model or workflow. Different from Hoeffding trees, soft trees can be integrated into such systems due to their differentiability, but are neither transparent nor explainable. Our novel model combines the extensibility and transparency of Hoeffding trees with the differentiability of soft trees. We introduce a new gating function to regulate the balance between univariate and multivariate splits in the tree. Experiments are performed on 20 data streams, comparing SoHoT to standard Hoeffding trees, Hoeffding trees with limited complexity, and soft trees applying a sparse activation function for sample routing. The results show that soft Hoeffding trees outperform Hoeffding trees in estimating class probabilities and, at the same time, maintain transparency compared to soft trees, with relatively small losses in terms of AUROC and cross-entropy. We also demonstrate how to trade off transparency against performance using a hyperparameter, obtaining univariate splits at one end of the spectrum and multivariate splits at the other.
End-to-end Inception-Unet based Generative Adversarial Networks for Snow and Rain Removals
Venue: arXiv cs.CV 2024
Year: 2024

The superior performance introduced by deep learning approaches in removing atmospheric particles such as snow and rain from a single image; favors their usage over classical ones. However, deep learning-based approaches still suffer from challenges related to the particle appearance characteristics such as size, type, and transparency. Furthermore, due to the unique characteristics of rain and snow particles, single network based deep learning approaches struggle in handling both degradation scenarios simultaneously. In this paper, a global framework that consists of two Generative Adversarial Networks (GANs) is proposed where each handles the removal of each particle individually. The architectures of both desnowing and deraining GANs introduce the integration of a feature extraction phase with the classical U-net generator network which in turn enhances the removal performance in the presence of severe variations in size and appearance. Furthermore, a realistic dataset that contains pairs of snowy images next to their groundtruth images estimated using a low-rank approximation approach; is presented. The experiments show that the proposed desnowing and deraining approaches achieve significant improvements in comparison to the state-of-the-art approaches when tested on both synthetic and realistic datasets.
Enhancing Missing Data Imputation through Combined Bipartite Graph and Complete Directed Graph
Venue: arXiv cs.LG 2024
Year: 2024

In this paper, we aim to address a significant challenge in the field of missing data imputation: identifying and leveraging the interdependencies among features to enhance missing data imputation for tabular data. We introduce a novel framework named the Bipartite and Complete Directed Graph Neural Network (BCGNN). Within BCGNN, observations and features are differentiated as two distinct node types, and the values of observed features are converted into attributed edges linking them. The bipartite segment of our framework inductively learns embedding representations for nodes, efficiently utilizing the comprehensive information encapsulated in the attributed edges. In parallel, the complete directed graph segment adeptly outlines and communicates the complex interdependencies among features. When compared to contemporary leading imputation methodologies, BCGNN consistently outperforms them, achieving a noteworthy average reduction of 15% in mean absolute error for feature imputation tasks under different missing mechanisms. Our extensive experimental investigation confirms that an in-depth grasp of the interdependence structure substantially enhances the model's feature embedding ability. We also highlight the model's superior performance in label prediction tasks involving missing data, and its formidable ability to generalize to unseen data points.
Structure Matters: Dynamic Policy Gradient
Venue: arXiv cs.LG 2024
Year: 2024

In this work, we study $γ$-discounted infinite-horizon tabular Markov decision processes (MDPs) and introduce a framework called dynamic policy gradient (DynPG). The framework directly integrates dynamic programming with (any) policy gradient method, explicitly leveraging the Markovian property of the environment. DynPG dynamically adjusts the problem horizon during training, decomposing the original infinite-horizon MDP into a sequence of contextual bandit problems. By iteratively solving these contextual bandits, DynPG converges to the stationary optimal policy of the infinite-horizon MDP. To demonstrate the power of DynPG, we establish its non-asymptotic global convergence rate under the tabular softmax parametrization, focusing on the dependencies on salient but essential parameters of the MDP. By combining classical arguments from dynamic programming with more recent convergence arguments of policy gradient schemes, we prove that softmax DynPG scales polynomially in the effective horizon $(1-γ)^{-1}$. Our findings contrast recent exponential lower bound examples for vanilla policy gradient.
Fed-LDR: Federated Local Data-infused Graph Creation with Node-centric Model Refinement
Venue: arXiv cs.LG 2024
Year: 2024

The rapid acceleration of global urbanization has introduced novel challenges in enhancing urban infrastructure and services. Spatio-temporal data, integrating spatial and temporal dimensions, has emerged as a critical tool for understanding urban phenomena and promoting sustainability. In this context, Federated Learning (FL) has gained prominence as a distributed learning paradigm aligned with the privacy requirements of urban IoT environments. However, integrating traditional and deep learning models into the FL framework poses significant challenges, particularly in capturing complex spatio-temporal dependencies and adapting to diverse urban conditions. To address these challenges, we propose the Federated Local Data-Infused Graph Creation with Node-centric Model Refinement (Fed-LDR) algorithm. Fed-LDR leverages FL and Graph Convolutional Networks (GCN) to enhance spatio-temporal data analysis in urban environments. The algorithm comprises two key modules: (1) the Local Data-Infused Graph Creation (LDIGC) module, which dynamically reconfigures adjacency matrices to reflect evolving spatial relationships within urban environments, and (2) the Node-centric Model Refinement (NoMoR) module, which customizes model parameters for individual urban nodes to accommodate heterogeneity. Evaluations on the PeMSD4 and PeMSD8 datasets demonstrate Fed-LDR's superior performance over six baseline methods. Fed-LDR achieved the lowest Mean Absolute Error (MAE) values of 20.15 and 17.30, and the lowest Root Mean Square Error (RMSE) values of 32.30 and 27.15, respectively, while maintaining a high correlation coefficient of 0.96 across both datasets. Notably, on the PeMSD4 dataset, Fed-LDR reduced MAE and RMSE by up to 81\% and 78\%, respectively, compared to the best-performing baseline FedMedian.
M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding
Venue: arXiv cs.CV 2024
Year: 2024

Document visual question answering (DocVQA) pipelines that answer questions from documents have broad applications. Existing methods focus on handling single-page documents with multi-modal language models (MLMs), or rely on text-based retrieval-augmented generation (RAG) that uses text extraction tools such as optical character recognition (OCR). However, there are difficulties in applying these methods in real-world scenarios: (a) questions often require information across different pages or documents, where MLMs cannot handle many long documents; (b) documents often have important information in visual elements such as figures, but text extraction tools ignore them. We introduce M3DocRAG, a novel multi-modal RAG framework that flexibly accommodates various document contexts (closed-domain and open-domain), question hops (single-hop and multi-hop), and evidence modalities (text, chart, figure, etc.). M3DocRAG finds relevant documents and answers questions using a multi-modal retriever and an MLM, so that it can efficiently handle single or many documents while preserving visual information. Since previous DocVQA datasets ask questions in the context of a specific document, we also present M3DocVQA, a new benchmark for evaluating open-domain DocVQA over 3,000+ PDF documents with 40,000+ pages. In three benchmarks (M3DocVQA/MMLongBench-Doc/MP-DocVQA), empirical results show that M3DocRAG with ColPali and Qwen2-VL 7B achieves superior performance than many strong baselines, including state-of-the-art performance in MP-DocVQA. We provide comprehensive analyses of different indexing, MLMs, and retrieval models. Lastly, we qualitatively show that M3DocRAG can successfully handle various scenarios, such as when relevant information exists across multiple pages and when answer evidence only exists in images.
BitNet a4.8: 4-bit Activations for 1-bit LLMs
Venue: arXiv cs.CL 2024
Year: 2024

Recent research on the 1-bit Large Language Models (LLMs), such as BitNet b1.58, presents a promising direction for reducing the inference cost of LLMs while maintaining their performance. In this work, we introduce BitNet a4.8, enabling 4-bit activations for 1-bit LLMs. BitNet a4.8 employs a hybrid quantization and sparsification strategy to mitigate the quantization errors introduced by the outlier channels. Specifically, we utilize 4-bit activations for inputs to the attention and feed-forward network layers, while sparsifying intermediate states followed with 8-bit quantization. Extensive experiments demonstrate that BitNet a4.8 achieves performance comparable to BitNet b1.58 with equivalent training costs, while being faster in inference with enabling 4-bit (INT4/FP4) kernels. Additionally, BitNet a4.8 activates only 55% of parameters and supports 3-bit KV cache, further enhancing the efficiency of large-scale LLM deployment and inference.
Noisy Zero-Shot Coordination: Breaking The Common Knowledge Assumption In Zero-Shot Coordination Games
Venue: arXiv cs.LG 2024
Year: 2024

Zero-shot coordination (ZSC) is a popular setting for studying the ability of reinforcement learning (RL) agents to coordinate with novel partners. Prior ZSC formulations assume the $\textit{problem setting}$ is common knowledge: each agent knows the underlying Dec-POMDP, knows others have this knowledge, and so on ad infinitum. However, this assumption rarely holds in complex real-world settings, which are often difficult to fully and correctly specify. Hence, in settings where this common knowledge assumption is invalid, agents trained using ZSC methods may not be able to coordinate well. To address this limitation, we formulate the $\textit{noisy zero-shot coordination}$ (NZSC) problem. In NZSC, agents observe different noisy versions of the ground truth Dec-POMDP, which are assumed to be distributed according to a fixed noise model. Only the distribution of ground truth Dec-POMDPs and the noise model are common knowledge. We show that a NZSC problem can be reduced to a ZSC problem by designing a meta-Dec-POMDP with an augmented state space consisting of all the ground-truth Dec-POMDPs. For solving NZSC problems, we propose a simple and flexible meta-learning method called NZSC training, in which the agents are trained across a distribution of coordination problems - which they only get to observe noisy versions of. We show that with NZSC training, RL agents can be trained to coordinate well with novel partners even when the (exact) problem setting of the coordination is not common knowledge.
Leveraging Transfer Learning and Multiple Instance Learning for HER2 Automatic Scoring of H\&E Whole Slide Images
Venue: arXiv cs.CV 2024
Year: 2024

Expression of human epidermal growth factor receptor 2 (HER2) is an important biomarker in breast cancer patients who can benefit from cost-effective automatic Hematoxylin and Eosin (H\&E) HER2 scoring. However, developing such scoring models requires large pixel-level annotated datasets. Transfer learning allows prior knowledge from different datasets to be reused while multiple-instance learning (MIL) allows the lack of detailed annotations to be mitigated. The aim of this work is to examine the potential of transfer learning on the performance of deep learning models pre-trained on (i) Immunohistochemistry (IHC) images, (ii) H\&E images and (iii) non-medical images. A MIL framework with an attention mechanism is developed using pre-trained models as patch-embedding models. It was found that embedding models pre-trained on H\&E images consistently outperformed the others, resulting in an average AUC-ROC value of $0.622$ across the 4 HER2 scores ($0.59-0.80$ per HER2 score). Furthermore, it was found that using multiple-instance learning with an attention layer not only allows for good classification results to be achieved, but it can also help with producing visual indication of HER2-positive areas in the H\&E slide image by utilising the patch-wise attention weights.
Intellectual Property Protection for Deep Learning Model and Dataset Intelligence
Venue: arXiv cs.CR 2024
Year: 2024

With the growing applications of Deep Learning (DL), especially recent spectacular achievements of Large Language Models (LLMs) such as ChatGPT and LLaMA, the commercial significance of these remarkable models has soared. However, acquiring well-trained models is costly and resource-intensive. It requires a considerable high-quality dataset, substantial investment in dedicated architecture design, expensive computational resources, and efforts to develop technical expertise. Consequently, safeguarding the Intellectual Property (IP) of well-trained models is attracting increasing attention. In contrast to existing surveys overwhelmingly focusing on model IPP mainly, this survey not only encompasses the protection on model level intelligence but also valuable dataset intelligence. Firstly, according to the requirements for effective IPP design, this work systematically summarizes the general and scheme-specific performance evaluation metrics. Secondly, from proactive IP infringement prevention and reactive IP ownership verification perspectives, it comprehensively investigates and analyzes the existing IPP methods for both dataset and model intelligence. Additionally, from the standpoint of training settings, it delves into the unique challenges that distributed settings pose to IPP compared to centralized settings. Furthermore, this work examines various attacks faced by deep IPP techniques. Finally, we outline prospects for promising future directions that may act as a guide for innovative research.
Fingernail-Based Tangential Force Simulation for Enhanced Dexterous Manipulation in Virtual Reality
Venue: arXiv cs.HC 2024
Year: 2024

This study introduces a novel haptic device for enhancing dexterous manipulation in virtual reality. By stimulating mechanoreceptors on both sides of the fingernail, our lightweight system simulates tangential force sensations. We employ mechanical stimulation for more natural tactile feedback. A preliminary "balancing grasp challenge" experiment shows that users make more frequent micro-adjustments with our device, indicating improved precision. This research aims to advance haptic feedback in VR, potentially leading to more immersive and realistic virtual interactions.
Enhancing Medical Anatomy Education through Virtual Reality (VR): Design, Development, and Evaluation
Venue: arXiv cs.HC 2024
Year: 2024

Modern medicine demands innovations in medical education, particularly in the learning of human anatomy, traditionally taught through textbooks, dissections, and lectures. Virtual Reality (VR) has emerged as a promising tool to address the limitations of these conventional methods by emphasising vision-based and active learning. However, current VR educational tools are often inaccessible due to high costs and specialised equipment requirements. This paper details the design and development of an accessible, desktop-based VR system aimed at enhancing anatomy education by leveraging the user's visual perception to promote a meaningful and interactive learning experience. The Virtual Anatomy Lab was designed to enable students to interact with a 3D Skull model to complete tasks virtually via an interactive user interface (UI) with the help of common devices like a mouse and keyboard. As part of the study, a group of medical students from prestigious medical schools throughout Malaysia were invited to evaluate the built system to offer feedback and determine its overall efficiency and usability in fulfilling their learning goals. The results and findings from user evaluations were then analysed to discuss its effectiveness and areas for future improvement.
Haptic Color Patterns for Visually Impaired People-Pilot Study for a Learning Color Wheel
Venue: arXiv cs.HC 2024
Year: 2024

This study proposes a tactile diagram pattern for visually impaired people to recognize color information. The pattern uses the principle of three primary colors, with different patterns representing red, blue, and yellow. The size of tactile elements on these patterns indicates the proportion of the color mixing. A preliminary experiment showed that even a sighted participant could understand and reconstruct the tactile diagram. Future experiments will target visually impaired people to confirm the effectiveness of this method.
Development of fall prevention training device that can provide external disturbance to the ankle with pneumatic gel muscles (PGM) while walking
Venue: arXiv cs.HC 2024
Year: 2024

Although the average life expectancy in Japan has been increasing in recent years, the problem of the large gap between healthy life expectancy and average life expectancy is still unresolved. Among the factors that lead to the need for nursing care, injuries due to falls account for a certain percentage of the total. In this paper, we developed boots that can provide external disturbance to the ankle with pneumatic gel muscles (PGM) while walking. We experimented using an angular velocity and acceleration of the heel as evaluation indices to evaluate the effectiveness of fall prevention training using this device, which is smaller and more wearable than conventional devices. In this study, we confirmed that the developed system has enough training intensity to significantly affect the gait waveform.
Mouse Embedded Soft Vibration Actuator for Exploring Surface Textures
Venue: arXiv cs.HC 2024
Year: 2024

We aimed to develop a tactile display that allows users to actively explore the virtual texture of a surface. We developed a tactile display embedded in an optical mouse that provides a wide range of frequency vibrations to the user's fingertip in response to its movement. We conducted an experiment to confirm the degree of fineness based on a stripe pattern identification task using a trial system. The results showed that the subject could identify a fine spatial resolution up to 0.16 mm width.
A Comparison of Violin Bowing Pressure and Position among Expert Players and Beginners
Venue: arXiv cs.HC 2024
Year: 2024

The violin is one of the most popular musical instruments, but mastering it requires a significant amount of practice time. The bowing action (pressure, position, speed) of the right hand is crucial in determining tonal quality, but this is difficult to master. This study compared the bowing movements, specifically bow pressure, bow position, and bow speed, of experienced players with those of beginners. Identifying common bowing characteristics of experienced violin players can aid the evaluation of beginners' skills and the development of a feedback system that supports self-practice and instruction.
Edge shape sensation presented in a noncontact manner using airborne ultrasound
Venue: arXiv cs.HC 2024
Year: 2024

To perceive 3D shapes such as pyramids, the perception of planes and edges as tactile sensations is an essential component. This is difficult to perceive with the conventional vibrotactile sensation used in ultrasound haptics because of its low spatial resolution. Recently, it has become possible to produce a high-resolution pressure sensation using airborne ultrasound. By using this pressure sensation, it is now possible to reproduce a linear, sharp-edged sensation in the area of a fingerpad. In this study, it is demonstrated that this pressure sensation can be used to reproduce the feeling of fine, sharp edges, and its effectiveness is confirmed by comparing it with conventional vibrotactile sensation. In the demonstration, participants can experience the contact sensation of several types of edges with different curvatures.
Innovative Weight Simulation in Virtual Reality Cube Games: A Pseudo-Haptic Approach
Venue: arXiv cs.HC 2024
Year: 2024

This paper presents an innovative pseudo-haptic model for weight simulation in virtual reality (VR) environments. By integrating visual feedback with voluntary exerted force through a passive haptic glove, the model creates haptic illusions of weight perception. Two VR cube games were developed to evaluate the model's effectiveness. The first game assesses participants' ability to discriminate relative weights, while the second evaluates their capability to estimate absolute weights. Twelve participants, aged 18 to 59, tested the games. Results suggest that the pseudo-haptic model is effective for relative weight discrimination tasks and holds potential for various VR applications. Further research with a larger participant group and more complex scenarios is recommended to refine and validate the model.
Hold and Feel! A Multiplayer Video Game System with Interpersonal Vibrotactile Feedback via Bracelet Controllers
Venue: arXiv cs.HC 2024
Year: 2024

Recent multiplayer video games (MPVGs) have increasingly incorporated social behaviors, such as gathering in the same place and facing each other, to enhance player interaction. This study aims to develop an MPVG system that facilitates social interaction through interpersonal touch between players. We present an MPVG system where two players wearing bracelet-type game controllers control a player character through touch to catch items and feel the position of the player character through vibrotactile sensations between their hands.