OpenRaiser/Intern-Atlas huggingface dataset:
SAUC: Sparsity-Aware Uncertainty Calibration for Spatiotemporal Prediction with Graph Neural Networks
Venue: arXiv cs.LG 2024
Year: 2024

Quantifying uncertainty is crucial for robust and reliable predictions. However, existing spatiotemporal deep learning mostly focuses on deterministic prediction, overlooking the inherent uncertainty in such prediction. Particularly, highly-granular spatiotemporal datasets are often sparse, posing extra challenges in prediction and uncertainty quantification. To address these issues, this paper introduces a novel post-hoc Sparsity-awar Uncertainty Calibration (SAUC) framework, which calibrates uncertainty in both zero and non-zero values. To develop SAUC, we firstly modify the state-of-the-art deterministic spatiotemporal Graph Neural Networks (ST-GNNs) to probabilistic ones in the pre-calibration phase. Then we calibrate the probabilistic ST-GNNs for zero and non-zero values using quantile approaches.Through extensive experiments, we demonstrate that SAUC can effectively fit the variance of sparse data and generalize across two real-world spatiotemporal datasets at various granularities. Specifically, our empirical experiments show a 20\% reduction in calibration errors in zero entries on the sparse traffic accident and urban crime prediction. Overall, this work demonstrates the theoretical and empirical values of the SAUC framework, thus bridging a significant gap between uncertainty quantification and spatiotemporal prediction.
Exploring SSL Discrete Tokens for Multilingual ASR
Venue: arXiv cs.CL 2024
Year: 2024

With the advancement of Self-supervised Learning (SSL) in speech-related tasks, there has been growing interest in utilizing discrete tokens generated by SSL for automatic speech recognition (ASR), as they offer faster processing techniques. However, previous studies primarily focused on multilingual ASR with Fbank features or English ASR with discrete tokens, leaving a gap in adapting discrete tokens for multilingual ASR scenarios. This study presents a comprehensive comparison of discrete tokens generated by various leading SSL models across multiple language domains. We aim to explore the performance and efficiency of speech discrete tokens across multiple language domains for both monolingual and multilingual ASR scenarios. Experimental results demonstrate that discrete tokens achieve comparable results against systems trained on Fbank features in ASR tasks across seven language domains with an average word error rate (WER) reduction of 0.31% and 1.76% absolute (2.80% and 15.70% relative) on dev and test sets respectively, with particularly WER reduction of 6.82% absolute (41.48% relative) on the Polish test set.
TabKANet: Tabular Data Modeling with Kolmogorov-Arnold Network and Transformer
Venue: arXiv cs.LG 2024
Year: 2024

Tabular data is the most common type of data in real-life scenarios. In this study, we propose the TabKANet model for tabular data modeling, which targets the bottlenecks in learning from numerical content. We constructed a Kolmogorov-Arnold Network (KAN) based Numerical Embedding Module and unified numerical and categorical features encoding within a Transformer architecture. TabKANet has demonstrated stable and significantly superior performance compared to Neural Networks (NNs) across multiple public datasets in binary classification, multi-class classification, and regression tasks. Its performance is comparable to or surpasses that of Gradient Boosted Decision Tree models (GBDTs). Our code is publicly available on GitHub: https://github.com/AI-thpremed/TabKANet.
AutoIRT: Calibrating Item Response Theory Models with Automated Machine Learning
Venue: arXiv cs.LG 2024
Year: 2024

Item response theory (IRT) is a class of interpretable factor models that are widely used in computerized adaptive tests (CATs), such as language proficiency tests. Traditionally, these are fit using parametric mixed effects models on the probability of a test taker getting the correct answer to a test item (i.e., question). Neural net extensions of these models, such as BertIRT, require specialized architectures and parameter tuning. We propose a multistage fitting procedure that is compatible with out-of-the-box Automated Machine Learning (AutoML) tools. It is based on a Monte Carlo EM (MCEM) outer loop with a two stage inner loop, which trains a non-parametric AutoML grade model using item features followed by an item specific parametric model. This greatly accelerates the modeling workflow for scoring tests. We demonstrate its effectiveness by applying it to the Duolingo English Test, a high stakes, online English proficiency test. We show that the resulting model is typically more well calibrated, gets better predictive performance, and more accurate scores than existing methods (non-explanatory IRT models and explanatory IRT models like BERT-IRT). Along the way, we provide a brief survey of machine learning methods for calibration of item parameters for CATs.
Optimized Design of A Haptic Unit for Vibrotactile Amplitude Modulation
Venue: arXiv cs.RO 2024
Year: 2024

Communicating information to users is a crucial aspect of human-machine interaction. Vibrotactile feedback encodes information into spatiotemporal vibrations, enabling users to perceive tactile sensations. It offers advantages such as lightweight, wearability, and high stability, with broad applications in sensory substitution, virtual reality, education, and healthcare. However, existing haptic unit designs lack amplitude modulation capabilities, which limits their applications. This paper proposed an optimized design of the haptic unit from the perspective of vibration amplitude modulation. A modified elastic model was developed to describe the propagation and attenuation mechanisms of vibration in the skin. Based on the model, two types of hierarchical architectural design were proposed. The design incorporated various materials arranged in multiple layers to amplify or attenuate the vibration amplitude as it traveled through the structure. An experimental platform was built to evaluate the performance of the optimized design.
Vertex identification to a forest
Venue: arXiv cs.DS 2024
Year: 2024

Let $\mathcal{H}$ be a graph class and $k\in\mathbb{N}$. We say a graph $G$ admits a \emph{$k$-identification to $\mathcal{H}$} if there is a partition $\mathcal{P}$ of some set $X\subseteq V(G)$ of size at most $k$ such that after identifying each part in $\mathcal{P}$ to a single vertex, the resulting graph belongs to $\mathcal{H}$. The graph parameter ${\sf id}_{\mathcal{H}}$ is defined so that ${\sf id}_{\mathcal{H}}(G)$ is the minimum $k$ such that $G$ admits a $k$-identification to $\mathcal{H}$, and the problem of \textsc{Identification to $\mathcal{H}$} asks, given a graph $G$ and $k\in\mathbb{N}$, whether ${\sf id}_{\mathcal{H}}(G)\le k$. If we set $\mathcal{H}$ to be the class $\mathcal{F}$ of acyclic graphs, we generate the problem \textsc{Identification to Forest}, which we show to be {\sf NP}-complete. We prove that, when parameterized by the size $k$ of the identification set, it admits a kernel of size $2k+1$. For our kernel we reveal a close relation of \textsc{Identification to Forest} with the \textsc{Vertex Cover} problem. We also study the combinatorics of the \textsf{yes}-instances of \textsc{Identification to $\mathcal{H}$}, i.e., the class $\mathcal{H}^{(k)}:=\{G\mid {\sf id}_{\mathcal{H}}(G)\le k\}$, {which we show to be minor-closed for every $k$} when $\mathcal{H}$ is minor-closed. We prove that the minor-obstructions of $\mathcal{F}^{(k)}$ are of size at most $2k+4$. We also prove that every graph $G$ such that ${\sf id}_{\mathcal{F}}(G)$ is sufficiently big contains as a minor either a cycle on $k$ vertices, or $k$ disjoint triangles, or the \emph{$k$-marguerite} graph, that is the graph obtained by $k$ disjoint triangles by identifying one vertex of each of them into the same vertex.
Visual Language Tracking with Multi-modal Interaction: A Robust Benchmark
Venue: arXiv cs.CV 2024
Year: 2024

Visual Language Tracking (VLT) enhances tracking by mitigating the limitations of relying solely on the visual modality, utilizing high-level semantic information through language. This integration of the language enables more advanced human-machine interaction. The essence of interaction is cognitive alignment, which typically requires multiple information exchanges, especially in the sequential decision-making process of VLT. However, current VLT benchmarks do not account for multi-round interactions during tracking. They provide only an initial text and bounding box (bbox) in the first frame, with no further interaction as tracking progresses, deviating from the original motivation of the VLT task. To address these limitations, we propose a novel and robust benchmark, VLT-MI (Visual Language Tracking with Multi-modal Interaction), which introduces multi-round interaction into the VLT task for the first time. (1) We generate diverse, multi-granularity texts for multi-round, multi-modal interaction based on existing mainstream VLT benchmarks using DTLLM-VLT, leveraging the world knowledge of LLMs. (2) We propose a new VLT interaction paradigm that achieves multi-round interaction through text updates and object recovery. When multiple tracking failures occur, we provide the tracker with more aligned texts and corrected bboxes through interaction, thereby expanding the scope of VLT downstream tasks. (3) We conduct comparative experiments on both traditional VLT benchmarks and VLT-MI, evaluating and analyzing the accuracy and robustness of trackers under the interactive paradigm. This work offers new insights and paradigms for the VLT task, enabling a fine-grained evaluation of multi-modal trackers. We believe this approach can be extended to additional datasets in the future, supporting broader evaluations and comparisons of video-language model capabilities.
Latent Space Score-based Diffusion Model for Probabilistic Multivariate Time Series Imputation
Venue: arXiv cs.LG 2024
Year: 2024

Accurate imputation is essential for the reliability and success of downstream tasks. Recently, diffusion models have attracted great attention in this field. However, these models neglect the latent distribution in a lower-dimensional space derived from the observed data, which limits the generative capacity of the diffusion model. Additionally, dealing with the original missing data without labels becomes particularly problematic. To address these issues, we propose the Latent Space Score-Based Diffusion Model (LSSDM) for probabilistic multivariate time series imputation. Observed values are projected onto low-dimensional latent space and coarse values of the missing data are reconstructed without knowing their ground truth values by this unsupervised learning approach. Finally, the reconstructed values are fed into a conditional diffusion model to obtain the precise imputed values of the time series. In this way, LSSDM not only possesses the power to identify the latent distribution but also seamlessly integrates the diffusion model to obtain the high-fidelity imputed values and assess the uncertainty of the dataset. Experimental results demonstrate that LSSDM achieves superior imputation performance while also providing a better explanation and uncertainty analysis of the imputation mechanism. The website of the code is \textit{https://github.com/gorgen2020/LSSDM\_imputation}.
Modeling Rational Adaptation of Visual Search to Hierarchical Structures
Venue: arXiv cs.HC 2024
Year: 2024

Efficient attention deployment in visual search is limited by human visual memory, yet this limitation can be offset by exploiting the environment's structure. This paper introduces a computational cognitive model that simulates how the human visual system uses visual hierarchies to prevent refixations in sequential attention deployment. The model adopts computational rationality, positing behaviors as adaptations to cognitive constraints and environmental structures. In contrast to earlier models that predict search performance for hierarchical information, our model does not include predefined assumptions about particular search strategies. Instead, our model's search strategy emerges as a result of adapting to the environment through reinforcement learning algorithms. In an experiment with human participants we test the model's prediction that structured environments reduce visual search times compared to random tasks. Our model's predictions correspond well with human search performance across various set sizes for both structured and unstructured visual layouts. Our work improves understanding of the adaptive nature of visual search in hierarchically structured environments and informs the design of optimized search spaces.
Mechanism Design for Extending the Accessibility of Facilities
Venue: arXiv cs.GT 2024
Year: 2024

We study a variation of facility location problems (FLPs) that aims to improve the accessibility of agents to the facility within the context of mechanism design without money. In such a variation, agents have preferences on the ideal locations of the facility on a real line, and the facility's location is fixed in advance where (re)locating the facility is not possible due to various constraints (e.g., limited space and construction costs). To improve the accessibility of agents to facilities, existing mechanism design literature in FLPs has proposed to structurally modify the real line (e.g., by adding a new interval) or provide shuttle services between two points when structural modifications are not possible. In this paper, we focus on the latter approach and propose to construct an accessibility range to extend the accessibility of the facility. In the range, agents can receive accommodations (e.g., school buses, campus shuttles, or pickup services) to help reach the facility. Therefore, the cost of each agent is the distance from their ideal location to the facility (possibility) through the range. We focus on designing strategyproof mechanisms that elicit true ideal locations from the agents and construct accessibility ranges (intervals) to approximately minimize the social cost or the maximum cost of agents. For both social and maximum costs, we design group strategyproof mechanisms with asymptotically tight bounds on the approximation ratios.
SGFormer: Single-Layer Graph Transformers with Approximation-Free Linear Complexity
Venue: arXiv cs.LG 2024
Year: 2024

Learning representations on large graphs is a long-standing challenge due to the inter-dependence nature. Transformers recently have shown promising performance on small graphs thanks to its global attention for capturing all-pair interactions beyond observed structures. Existing approaches tend to inherit the spirit of Transformers in language and vision tasks, and embrace complicated architectures by stacking deep attention-based propagation layers. In this paper, we attempt to evaluate the necessity of adopting multi-layer attentions in Transformers on graphs, which considerably restricts the efficiency. Specifically, we analyze a generic hybrid propagation layer, comprised of all-pair attention and graph-based propagation, and show that multi-layer propagation can be reduced to one-layer propagation, with the same capability for representation learning. It suggests a new technical path for building powerful and efficient Transformers on graphs, particularly through simplifying model architectures without sacrificing expressiveness. As exemplified by this work, we propose a Simplified Single-layer Graph Transformers (SGFormer), whose main component is a single-layer global attention that scales linearly w.r.t. graph sizes and requires none of any approximation for accommodating all-pair interactions. Empirically, SGFormer successfully scales to the web-scale graph ogbn-papers100M, yielding orders-of-magnitude inference acceleration over peer Transformers on medium-sized graphs, and demonstrates competitiveness with limited labeled data.
AutoGeo: Automating Geometric Image Dataset Creation for Enhanced Geometry Understanding
Venue: arXiv cs.LG 2024
Year: 2024

With the rapid advancement of large language models, there has been a growing interest in their capabilities in mathematical reasoning. However, existing research has primarily focused on text-based algebra problems, neglecting the study of geometry due to the lack of high-quality geometric datasets. To address this gap, this paper introduces AutoGeo, a novel approach for automatically generating mathematical geometric images to fulfill the demand for large-scale and diverse geometric datasets. AutoGeo facilitates the creation of AutoGeo-100k, an extensive repository comprising 100k high-quality geometry image-text pairs. By leveraging precisely defined geometric clauses, AutoGeo-100k contains a wide variety of geometric shapes, including lines, polygons, circles, and complex spatial relationships, etc. Furthermore, this paper demonstrates the efficacy of AutoGeo-100k in enhancing the performance of multimodal large language models through fine-tuning. Experimental results indicate significant improvements in the model's ability in handling geometric images, as evidenced by enhanced accuracy in tasks such as geometric captioning and mathematical reasoning. This research not only fills a critical gap in the availability of geometric datasets but also paves the way for the advancement of sophisticated AI-driven tools in education and research. Project page: https://autogeo-official.github.io/.
Evaluating the Performance of Large Language Models in Competitive Programming: A Multi-Year, Multi-Grade Analysis
Venue: arXiv cs.SE 2024
Year: 2024

This study explores the performance of large language models (LLMs) in solving competitive programming problems from the Romanian Informatics Olympiad at the county level. Romania, a leading nation in computer science competitions, provides an ideal environment for evaluating LLM capabilities due to its rich history and stringent competition standards. We collected and analyzed a dataset comprising 304 challenges from 2002 to 2023, focusing on solutions written by LLMs in C++ and Python for these problems. Our primary goal is to understand why LLMs perform well or poorly on different tasks. We evaluated various models, including closed-source models like GPT-4 and open-weight models such as CodeLlama and RoMistral, using a standardized process involving multiple attempts and feedback rounds. The analysis revealed significant variations in LLM performance across different grades and problem types. Notably, GPT-4 showed strong performance, indicating its potential use as an educational tool for middle school students. We also observed differences in code quality and style across various LLMs
Redefining Data-Centric Design: A New Approach with a Domain Model and Core Data Ontology for Computational Systems
Venue: arXiv cs.DC 2024
Year: 2024

This paper presents an innovative data-centric paradigm for designing computational systems by introducing a new informatics domain model. The proposed model moves away from the conventional node-centric framework and focuses on data-centric categorization, using a multimodal approach that incorporates objects, events, concepts, and actions. By drawing on interdisciplinary research and establishing a foundational ontology based on these core elements, the model promotes semantic consistency and secure data handling across distributed ecosystems. We also explore the implementation of this model as an OWL 2 ontology, discuss its potential applications, and outline its scalability and future directions for research. This work aims to serve as a foundational guide for system designers and data architects in developing more secure, interoperable, and scalable data systems.
Batched Online Contextual Sparse Bandits with Sequential Inclusion of Features
Venue: arXiv cs.LG 2024
Year: 2024

Multi-armed Bandits (MABs) are increasingly employed in online platforms and e-commerce to optimize decision making for personalized user experiences. In this work, we focus on the Contextual Bandit problem with linear rewards, under conditions of sparsity and batched data. We address the challenge of fairness by excluding irrelevant features from decision-making processes using a novel algorithm, Online Batched Sequential Inclusion (OBSI), which sequentially includes features as confidence in their impact on the reward increases. Our experiments on synthetic data show the superior performance of OBSI compared to other algorithms in terms of regret, relevance of features used, and compute.
Seed-Music: A Unified Framework for High Quality and Controlled Music Generation
Venue: arXiv cs.SD 2024
Year: 2024

We introduce Seed-Music, a suite of music generation systems capable of producing high-quality music with fine-grained style control. Our unified framework leverages both auto-regressive language modeling and diffusion approaches to support two key music creation workflows: controlled music generation and post-production editing. For controlled music generation, our system enables vocal music generation with performance controls from multi-modal inputs, including style descriptions, audio references, musical scores, and voice prompts. For post-production editing, it offers interactive tools for editing lyrics and vocal melodies directly in the generated audio. We encourage readers to listen to demo audio examples at https://team.doubao.com/seed-music "https://team.doubao.com/seed-music".
Optimal Control Approach for Gait Transition with Riemannian Splines
Venue: arXiv cs.RO 2024
Year: 2024

Robotic locomotion often relies on sequenced gaits to efficiently convert control input into desired motion. Despite extensive studies on gait optimization, achieving smooth and efficient gait transitions remains challenging. In this paper, we propose a general solver based on geometric optimal control methods, leveraging insights from previous works on gait efficiency. Building upon our previous work, we express the effort to execute the trajectory as distinct geometric objects, transforming the optimization problems into boundary value problems. To validate our approach, we generate gait transition trajectories for three-link swimmers across various fluid environments. This work provides insights into optimal trajectory geometries and mechanical considerations for robotic locomotion.
Cross-Entropy Optimization for Hyperparameter Optimization in Stochastic Gradient-based Approaches to Train Deep Neural Networks
Venue: arXiv cs.LG 2024
Year: 2024

In this paper, we present a cross-entropy optimization method for hyperparameter optimization in stochastic gradient-based approaches to train deep neural networks. The value of a hyperparameter of a learning algorithm often has great impact on the performance of a model such as the convergence speed, the generalization performance metrics, etc. While in some cases the hyperparameters of a learning algorithm can be part of learning parameters, in other scenarios the hyperparameters of a stochastic optimization algorithm such as Adam [5] and its variants are either fixed as a constant or are kept changing in a monotonic way over time. We give an in-depth analysis of the presented method in the framework of expectation maximization (EM). The presented algorithm of cross-entropy optimization for hyperparameter optimization of a learning algorithm (CEHPO) can be equally applicable to other areas of optimization problems in deep learning. We hope that the presented methods can provide different perspectives and offer some insights for optimization problems in different areas of machine learning and beyond.