OpenRaiser/Intern-Atlas huggingface dataset:
iNNspector: Visual, Interactive Deep Model Debugging
Venue: arXiv cs.HC 2024
Year: 2024

Deep learning model design, development, and debugging is a process driven by best practices, guidelines, trial-and-error, and the personal experiences of model developers. At multiple stages of this process, performance and internal model data can be logged and made available. However, due to the sheer complexity and scale of this data and process, model developers often resort to evaluating their model performance based on abstract metrics like accuracy and loss. We argue that a structured analysis of data along the model's architecture and at multiple abstraction levels can considerably streamline the debugging process. Such a systematic analysis can further connect the developer's design choices to their impacts on the model behavior, facilitating the understanding, diagnosis, and refinement of deep learning models. Hence, in this paper, we (1) contribute a conceptual framework structuring the data space of deep learning experiments. Our framework, grounded in literature analysis and requirements interviews, captures design dimensions and proposes mechanisms to make this data explorable and tractable. To operationalize our framework in a ready-to-use application, we (2) present the iNNspector system. iNNspector enables tracking of deep learning experiments and provides interactive visualizations of the data on all levels of abstraction from multiple models to individual neurons. Finally, we (3) evaluate our approach with three real-world use-cases and a user study with deep learning developers and data analysts, proving its effectiveness and usability.
Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption
Venue: arXiv cs.CL 2024
Year: 2024

Large Language Models (LLMs), epitomized by ChatGPT's release in late 2022, have revolutionized various industries with their advanced language comprehension. However, their efficiency is challenged by the Transformer architecture's struggle with handling long texts. KV Cache has emerged as a pivotal solution to this issue, converting the time complexity of token generation from quadratic to linear, albeit with increased GPU memory overhead proportional to conversation length. With the development of the LLM community and academia, various KV Cache compression methods have been proposed. In this review, we dissect the various properties of KV Cache and elaborate on various methods currently used to optimize the KV Cache space usage of LLMs. These methods span the pre-training phase, deployment phase, and inference phase, and we summarize the commonalities and differences among these methods. Additionally, we list some metrics for evaluating the long-text capabilities of large language models, from both efficiency and capability perspectives. Our review thus sheds light on the evolving landscape of LLM optimization, offering insights into future advancements in this dynamic field. Links to the papers mentioned in this review can be found in our Github Repo https://github.com/zcli-charlie/Awesome-KV-Cache.
Optimal Broadcast Schedules in Logarithmic Time with Applications to Broadcast, All-Broadcast, Reduction and All-Reduction
Venue: arXiv cs.DC 2024
Year: 2024

We give optimally fast $O(\log p)$ time (per processor) algorithms for computing round-optimal broadcast schedules for message-passing parallel computing systems. This affirmatively answers difficult questions posed in a SPAA 2022 BA and a CLUSTER 2022 paper. We observe that the computed schedules and circulant communication graph can likewise be used for reduction, all-broadcast and all-reduction as well, leading to new, round-optimal algorithms for these problems. These observations affirmatively answer open questions posed in a CLUSTER 2023 paper. The problem is to broadcast $n$ indivisible blocks of data from a given root processor to all other processors in a (subgraph of a) fully connected network of $p$ processors with fully bidirectional, one-ported communication capabilities. In this model, $n-1+\lceil\log_2 p\rceil$ communication rounds are required. Our new algorithms compute for each processor in the network receive and send schedules each of size $\lceil\log_2 p\rceil$ that determine uniquely in $O(1)$ time for each communication round the new block that the processor will receive, and the already received block it has to send. Schedule computations are done independently per processor without communication. The broadcast communication subgraph is an easily computable, directed, $\lceil\log_2 p\rceil$-regular circulant graph also used elsewhere. We show how the schedule computations can be done in optimal time and space of $O(\log p)$, improving significantly over previous results of $O(p\log^2 p)$ and $O(\log^3 p)$, respectively. The schedule computation and broadcast algorithms are simple to implement, but correctness and complexity are not obvious. The schedules are used for new implementations of the MPI (Message-Passing Interface) collectives MPI_Bcast, MPI_Allgatherv, MPI_Reduce and MPI_Reduce_scatter. Preliminary experimental results are given.
Egocentric Robots in a Human-Centric World? Exploring Group-Robot-Interaction in Public Spaces
Venue: arXiv cs.RO 2024
Year: 2024

The deployment of social robots in real-world scenarios is increasing, supporting humans in various contexts. However, they still struggle to grasp social dynamics, especially in public spaces, sometimes resulting in violations of social norms, such as interrupting human conversations. This behavior, originating from a limited processing of social norms, might be perceived as robot-centered. Understanding social dynamics, particularly in group-robot-interactions (GRI), underscores the need for further research and development in human-robot-interaction (HRI). Enhancing the interaction abilities of social robots, especially in GRIs, can improve their effectiveness in real-world applications on a micro-level, as group interactions lead to increased motivation and comfort. In this study, we assessed the influence of the interaction condition (dyadic vs. triadic) on the perceived extraversion (ext.) of social robots in public spaces. The research involved 40 HRIs, including 24 dyadic (i.e., one human and one robot) interactions and 16 triadic interactions, which involve at least three entities, including the robot.
Learning mental states estimation through self-observation: a developmental synergy between intentions and beliefs representations in a deep-learning model of Theory of Mind
Venue: arXiv cs.NE 2024
Year: 2024

Theory of Mind (ToM), the ability to attribute beliefs, intentions, or mental states to others, is a crucial feature of human social interaction. In complex environments, where the human sensory system reaches its limits, behaviour is strongly driven by our beliefs about the state of the world around us. Accessing others' mental states, e.g., beliefs and intentions, allows for more effective social interactions in natural contexts. Yet, these variables are not directly observable, making understanding ToM a challenging quest of interest for different fields, including psychology, machine learning and robotics. In this paper, we contribute to this topic by showing a developmental synergy between learning to predict low-level mental states (e.g., intentions, goals) and attributing high-level ones (i.e., beliefs). Specifically, we assume that learning beliefs attribution can occur by observing one's own decision processes involving beliefs, e.g., in a partially observable environment. Using a simple feed-forward deep learning model, we show that, when learning to predict others' intentions and actions, more accurate predictions can be acquired earlier if beliefs attribution is learnt simultaneously. Furthermore, we show that the learning performance improves even when observed actors have a different embodiment than the observer and the gain is higher when observing beliefs-driven chunks of behaviour. We propose that our computational approach can inform the understanding of human social cognitive development and be relevant for the design of future adaptive social robots able to autonomously understand, assist, and learn from human interaction partners in novel natural environments and tasks.
GaussianSR: High Fidelity 2D Gaussian Splatting for Arbitrary-Scale Image Super-Resolution
Venue: arXiv cs.CV 2024
Year: 2024

Implicit neural representations (INRs) have significantly advanced the field of arbitrary-scale super-resolution (ASSR) of images. Most existing INR-based ASSR networks first extract features from the given low-resolution image using an encoder, and then render the super-resolved result via a multi-layer perceptron decoder. Although these approaches have shown promising results, their performance is constrained by the limited representation ability of discrete latent codes in the encoded features. In this paper, we propose a novel ASSR method named GaussianSR that overcomes this limitation through 2D Gaussian Splatting (2DGS). Unlike traditional methods that treat pixels as discrete points, GaussianSR represents each pixel as a continuous Gaussian field. The encoded features are simultaneously refined and upsampled by rendering the mutually stacked Gaussian fields. As a result, long-range dependencies are established to enhance representation ability. In addition, a classifier is developed to dynamically assign Gaussian kernels to all pixels to further improve flexibility. All components of GaussianSR (i.e., encoder, classifier, Gaussian kernels, and decoder) are jointly learned end-to-end. Experiments demonstrate that GaussianSR achieves superior ASSR performance with fewer parameters than existing methods while enjoying interpretable and content-aware feature aggregations.
Difficulty Estimation and Simplification of French Text Using LLMs
Venue: arXiv cs.CL 2024
Year: 2024

We leverage generative large language models for language learning applications, focusing on estimating the difficulty of foreign language texts and simplifying them to lower difficulty levels. We frame both tasks as prediction problems and develop a difficulty classification model using labeled examples, transfer learning, and large language models, demonstrating superior accuracy compared to previous approaches. For simplification, we evaluate the trade-off between simplification quality and meaning preservation, comparing zero-shot and fine-tuned performances of large language models. We show that meaningful text simplifications can be obtained with limited fine-tuning. Our experiments are conducted on French texts, but our methods are language-agnostic and directly applicable to other foreign languages.
Revealing urban area from mobile positioning data
Venue: arXiv cs.SI 2024
Year: 2024

Researchers face the trade-off between publishing mobility data along with their papers while simultaneously protecting the privacy of the individuals. In addition to the fundamental anonymization process, other techniques, such as spatial discretization and, in certain cases, location concealing or complete removal, are applied to achieve these dual objectives. The primary research question is whether concealing the observation area is an adequate form of protection or whether human mobility patterns in urban areas are inherently revealing of location. The characteristics of the mobility data, such as the number of activity records or the number of unique users in a given spatial unit, reveal the silhouette of the urban landscape, which can be used to infer the identity of the city in question. It was demonstrated that even without disclosing the exact location, the patterns of human mobility can still reveal the urban area from which the data was collected. The presented locating method was tested on other cities using different open data sets and against coarser spatial discretization units. While publishing mobility data is essential for research, it was demonstrated that concealing the observation area is insufficient to prevent the identification of the urban area. Furthermore, using larger discretization units alone is an ineffective solution to the problem of the observation area re-identification. Instead of obscuring the observation area, noise should be added to the trajectories to prevent user identification.
DINOv2 Rocks Geological Image Analysis: Classification, Segmentation, and Interpretability
Venue: arXiv cs.CV 2024
Year: 2024

Recent advancements in computer vision have significantly improved image analysis tasks. Yet, deep learning models often struggle when applied to domains outside their training distribution, such as in geosciences, where domain-specific data can be scarce. This study investigates the classification, segmentation, and interpretability of CT-scan images of rock samples, focusing on the application of modern computer vision techniques to geoscientific tasks. We compare a range of segmentation methods to assess their efficacy, efficiency, and adaptability in geological image analysis. The methods evaluated include Otsu thresholding, clustering techniques (K-means, fuzzy C-means), a supervised machine learning approach (Random Forest), and deep learning models (UNet, ResNet152, and DINOv2), using ten binary sandstone datasets and three multi-class calcite datasets. DINOv2 was selected for its promising results in feature extraction and its potential applicability in geoscientific tasks, prompting further assessment of its interpretability and effectiveness in processing CT-scanned rock data. For classification, a non-fine-tuned DINOv2 demonstrates strong performance in classifying rock images, even when the CT-scans are outside its original training set. In segmentation tasks, thresholding and clustering techniques, though computationally efficient, produce subpar results despite preprocessing efforts. In contrast, supervised methods achieve better performance. While deep learning methods demand greater computational resources, they require minimal intervention and offer superior generalization. A LoRA fine-tuned DINOv2, in particular, excels in out-of-distribution segmentation and outperforms other methods in multi-class tasks, even with limited data. Notably, the segmentation masks generated by DINOv2 often appear more accurate than the original targets, based on visual inspection.
PIR Codes, Unequal-Data-Demand Codes, and the Griesmer Bound
Venue: arXiv cs.IT 2024
Year: 2024

Unequal Error-Protecting (UEP) codes are error-correcting (EC) codes designed to protect some parts of the encoded data better than other parts. Here, we introduce a similar generalization of PIR codes that we call Unequal-Data-Demand (UDD) PIR codes. These codes are PIR-type codes designed for the scenario where some parts of the encoded data are in higher demand than other parts. We generalize various results for PIR codes to UDD codes. Our main contribution is a new approach to the Griesmer bound for linear EC codes involving an Integer Linear Programming (ILP) problem that generalizes to linear UEP codes and linear UDD PIR codes.
Estimating Earthquake Magnitude in Sentinel-1 Imagery via Ranking
Venue: arXiv cs.CV 2024
Year: 2024

Earthquakes are commonly estimated using physical seismic stations, however, due to the installation requirements and costs of these stations, global coverage quickly becomes impractical. An efficient and lower-cost alternative is to develop machine learning models to globally monitor earth observation data to pinpoint regions impacted by these natural disasters. However, due to the small amount of historically recorded earthquakes, this becomes a low-data regime problem requiring algorithmic improvements to achieve peak performance when learning to regress earthquake magnitude. In this paper, we propose to pose the estimation of earthquake magnitudes as a metric-learning problem, training models to not only estimate earthquake magnitude from Sentinel-1 satellite imagery but to additionally rank pairwise samples. Our experiments show at max a 30%+ improvement in MAE over prior regression-only based methods, particularly transformer-based architectures.
Dallah: A Dialect-Aware Multimodal Large Language Model for Arabic
Venue: arXiv cs.CL 2024
Year: 2024

Recent advancements have significantly enhanced the capabilities of Multimodal Large Language Models (MLLMs) in generating and understanding image-to-text content. Despite these successes, progress is predominantly limited to English due to the scarcity of high quality multimodal resources in other languages. This limitation impedes the development of competitive models in languages such as Arabic. To alleviate this situation, we introduce an efficient Arabic multimodal assistant, dubbed Dallah, that utilizes an advanced language model based on LLaMA-2 to facilitate multimodal interactions. Dallah demonstrates state-of-the-art performance in Arabic MLLMs. Through fine-tuning six Arabic dialects, Dallah showcases its capability to handle complex dialectal interactions incorporating both textual and visual elements. The model excels in two benchmark tests: one evaluating its performance on Modern Standard Arabic (MSA) and another specifically designed to assess dialectal responses. Beyond its robust performance in multimodal interaction tasks, Dallah has the potential to pave the way for further development of dialect-aware Arabic MLLMs.
Semi-Classical Subspaces, The No Synchronization Law, and More
Venue: arXiv cs.CC 2024
Year: 2024

This paper looks at the intersection of algorithmic information theory and physics, namely quantum mechanics, thermodynamics, and black holes. We discuss theorems which characterize the barrier between the quantum world and the classical realm. The notion of a ``semi-classical subspace'' is introduced. Partial signals and partial information cloning can be obtained on quantum states in semi-classical subspaces. The No Synchronization Law is detailed, which says separate and isolated physical systems evolving over time cannot have algorithmic thermodynamic entropies that are in synch. We look at future work involving the Kolmogorov complexity of black holes.
An NKCS Model of Bookchins Communalism
Venue: arXiv cs.NE 2024
Year: 2024

The NKCS model was introduced to explore coevolutionary systems, that is, systems in which multiple species are closely interconnected. The fitness landscapes of the species are coupled to a controllable amount, where the underlying properties of the individual landscapes are also controllable. No previous work has explored the use of hierarchical control within the model. This paper explores the effects of using a confederation, based on Bookchins communalism, and a single point of global control. Significant changes in behaviour from the traditional model are seen across the parameter space.
Classification-Based Automatic HDL Code Generation Using LLMs
Venue: arXiv cs.AR 2024
Year: 2024

While large language models (LLMs) have demonstrated the ability to generate hardware description language (HDL) code for digital circuits, they still suffer from the hallucination problem, which leads to the generation of incorrect HDL code or misunderstanding of specifications. In this work, we introduce a human-expert-inspired method to mitigate the hallucination of LLMs and improve the performance in HDL code generation. We first let LLMs classify the type of the circuit based on the specifications. Then, according to the type of the circuit, we split the tasks into several sub-procedures, including information extraction and human-like design flow using Electronic Design Automation (EDA) tools. Besides, we also use a search method to mitigate the variation in code generation. Experimental results show that our method can significantly improve the functional correctness of the generated Verilog and reduce the hallucination of LLMs.
HPAC-ML: A Programming Model for Embedding ML Surrogates in Scientific Applications
Venue: arXiv cs.DC 2024
Year: 2024

Recent advancements in Machine Learning (ML) have substantially improved its predictive and computational abilities, offering promising opportunities for surrogate modeling in scientific applications. By accurately approximating complex functions with low computational cost, ML-based surrogates can accelerate scientific applications by replacing computationally intensive components with faster model inference. However, integrating ML models into these applications remains a significant challenge, hindering the widespread adoption of ML surrogates as an approximation technique in modern scientific computing. We propose an easy-to-use directive-based programming model that enables developers to seamlessly describe the use of ML models in scientific applications. The runtime support, as instructed by the programming model, performs data assimilation using the original algorithm and can replace the algorithm with model inference. Our evaluation across five benchmarks, testing over 5000 ML models, shows up to 83.6x speed improvements with minimal accuracy loss (as low as 0.01 RMSE).
Needle Segmentation Using GAN: Restoring Thin Instrument Visibility in Robotic Ultrasound
Venue: arXiv cs.RO 2024
Year: 2024

Ultrasound-guided percutaneous needle insertion is a standard procedure employed in both biopsy and ablation in clinical practices. However, due to the complex interaction between tissue and instrument, the needle may deviate from the in-plane view, resulting in a lack of close monitoring of the percutaneous needle. To address this challenge, we introduce a robot-assisted ultrasound (US) imaging system designed to seamlessly monitor the insertion process and autonomously restore the visibility of the inserted instrument when misalignment happens. To this end, the adversarial structure is presented to encourage the generation of segmentation masks that align consistently with the ground truth in high-order space. This study also systematically investigates the effects on segmentation performance by exploring various training loss functions and their combinations. When misalignment between the probe and the percutaneous needle is detected, the robot is triggered to perform transverse searching to optimize the positional and rotational adjustment to restore needle visibility. The experimental results on ex-vivo porcine samples demonstrate that the proposed method can precisely segment the percutaneous needle (with a tip error of $0.37\pm0.29mm$ and an angle error of $1.19\pm 0.29^{\circ}$). Furthermore, the needle appearance can be successfully restored under the repositioned probe pose in all 45 trials, with repositioning errors of $1.51\pm0.95mm$ and $1.25\pm0.79^{\circ}$. from latex to text with math symbols
Effect of Duration and Delay on the Identifiability of VR Motion
Venue: arXiv cs.CR 2024
Year: 2024

Social virtual reality is an emerging medium of communication. In this medium, a user's avatar (virtual representation) is controlled by the tracked motion of the user's headset and hand controllers. This tracked motion is a rich data stream that can leak characteristics of the user or can be effectively matched to previously-identified data to identify a user. To better understand the boundaries of motion data identifiability, we investigate how varying training data duration and train-test delay affects the accuracy at which a machine learning model can correctly classify user motion in a supervised learning task simulating re-identification. The dataset we use has a unique combination of a large number of participants, long duration per session, large number of sessions, and a long time span over which sessions were conducted. We find that training data duration and train-test delay affect identifiability; that minimal train-test delay leads to very high accuracy; and that train-test delay should be controlled in future experiments.