OpenRaiser/Intern-Atlas huggingface dataset:
Checker Bug Detection and Repair in Deep Learning Libraries
Venue: arXiv cs.SE 2024
Year: 2024

Checker bugs in Deep Learning (DL) libraries are critical yet not well-explored. These bugs are often concealed in the input validation and error-checking code of DL libraries and can lead to silent failures, incorrect results, or unexpected program behavior in DL applications. Despite their potential to significantly impact the reliability and performance of DL-enabled systems built with these libraries, checker bugs have received limited attention. We present the first comprehensive study of DL checker bugs in two widely-used DL libraries, i.e., TensorFlow and PyTorch. Initially, we automatically collected a dataset of 2,418 commits from TensorFlow and PyTorch repositories on GitHub from Sept. 2016 to Dec. 2023 using specific keywords related to checker bugs. Through manual inspection, we identified 527 DL checker bugs. Subsequently, we analyzed these bugs from three perspectives, i.e., root causes, symptoms, and fixing patterns. Using the knowledge gained via root cause analysis of checker bugs, we further propose TensorGuard, a proof-of-concept RAG-based LLM-based tool to detect and fix checker bugs in DL libraries via prompt engineering a series of ChatGPT prompts. We evaluated TensorGuard's performance on a test dataset that includes 92 buggy and 135 clean checker-related changes in TensorFlow and PyTorch from January 2024 to July 2024. Our results demonstrate that TensorGuard has high average recall (94.51\%) using Chain of Thought prompting, a balanced performance between precision and recall using Zero-Shot prompting and Few-Shot prompting strategies. In terms of patch generation, TensorGuard achieves an accuracy of 11.1\%, which outperforms the state-of-the-art bug repair baseline by 2\%. We have also applied TensorGuard on the latest six months' checker-related changes (493 changes) of the JAX library from Google, which resulted in the detection of 64 new checker bugs.
From Generalist to Specialist: Adapting Vision Language Models via Task-Specific Visual Instruction Tuning
Venue: arXiv cs.CV 2024
Year: 2024

Large vision language models (VLMs) combine large language models with vision encoders, demonstrating promise across various tasks. However, they often underperform in task-specific applications due to domain gaps between pre-training and fine-tuning. We introduce VITask, a novel framework that enhances task-specific adaptability of VLMs by integrating task-specific models (TSMs). VITask employs three key strategies: exemplar prompting (EP), response distribution alignment (RDA), and contrastive response tuning (CRT) to improve the task-specific performance of VLMs by adjusting their response distributions. EP allows TSM features to guide VLMs, while RDA enables VLMs to adapt without TSMs during inference by learning from exemplar-prompted models. CRT further optimizes the ranking of correct image-response pairs, thereby reducing the risk of generating undesired responses. Experiments on 12 medical diagnosis datasets across 9 imaging modalities show that VITask outperforms both vanilla instruction-tuned VLMs and TSMs, showcasing its ability to integrate complementary features from both models effectively. Additionally, VITask offers practical advantages such as flexible TSM integration and robustness to incomplete instructions, making it a versatile and efficient solution for task-specific VLM tuning. Our code are available at https://github.com/baiyang4/VITask.
A Benchmark on Directed Graph Representation Learning in Hardware Designs
Venue: arXiv cs.LG 2024
Year: 2024

To keep pace with the rapid advancements in design complexity within modern computing systems, directed graph representation learning (DGRL) has become crucial, particularly for encoding circuit netlists, computational graphs, and developing surrogate models for hardware performance prediction. However, DGRL remains relatively unexplored, especially in the hardware domain, mainly due to the lack of comprehensive and user-friendly benchmarks. This study presents a novel benchmark comprising five hardware design datasets and 13 prediction tasks spanning various levels of circuit abstraction. We evaluate 21 DGRL models, employing diverse graph neural networks and graph transformers (GTs) as backbones, enhanced by positional encodings (PEs) tailored for directed graphs. Our results highlight that bidirected (BI) message passing neural networks (MPNNs) and robust PEs significantly enhance model performance. Notably, the top-performing models include PE-enhanced GTs interleaved with BI-MPNN layers and BI-Graph Isomorphism Network, both surpassing baselines across the 13 tasks. Additionally, our investigation into out-of-distribution (OOD) performance emphasizes the urgent need to improve OOD generalization in DGRL models. This benchmark, implemented with a modular codebase, streamlines the evaluation of DGRL models for both hardware and ML practitioners
Towards Self-Improvement of LLMs via MCTS: Leveraging Stepwise Knowledge with Curriculum Preference Learning
Venue: arXiv cs.LG 2024
Year: 2024

Monte Carlo Tree Search (MCTS) has recently emerged as a powerful technique for enhancing the reasoning capabilities of LLMs. Techniques such as SFT or DPO have enabled LLMs to distill high-quality behaviors from MCTS, improving their reasoning performance. However, existing distillation methods underutilize the rich trajectory information generated by MCTS, limiting the potential for improvements in LLM reasoning. In this paper, we propose AlphaLLM-CPL, a novel pairwise training framework that enables LLMs to self-improve through MCTS behavior distillation. AlphaLLM-CPL efficiently leverages MCTS trajectories via two key innovations: (1) AlphaLLM-CPL constructs stepwise trajectory pairs from child nodes sharing the same parent in the search tree, providing step-level information for more effective MCTS behavior distillation. (2) AlphaLLM-CPL introduces curriculum preference learning, dynamically adjusting the training sequence of trajectory pairs in each offline training epoch to prioritize critical learning steps and mitigate overfitting. Experimental results on mathematical reasoning tasks demonstrate that AlphaLLM-CPL significantly outperforms previous MCTS behavior distillation methods, substantially boosting the reasoning capabilities of LLMs.
In-Band Full-Duplex MIMO Systems for Simultaneous Communications and Sensing: Challenges, Methods, and Future Perspectives
Venue: arXiv cs.IT 2024
Year: 2024

In-band Full-Duplex (FD) Multiple-Input Multiple-Output (MIMO) systems offer a significant opportunity for Integrated Sensing and Communications (ISAC) due to their capability to realize simultaneous signal transmissions and receptions. This feature has been recently exploited to devise spectrum-efficient simultaneous information transmission and monostatic sensing operations, a line of research typically referred to as MIMO FD-ISAC. In this article, capitalizing on a recent FD MIMO architecture with reduced complexity analog cancellation, we present an FD-enabled framework for simultaneous communications and sensing using data signals. In contrast to communications applications, the framework's goal is not to mitigate self interference, since it includes reflections of the downlink data transmissions from targets in the FD node's vicinity, but to optimize the system parameters for the intended dual functionality. The unique characteristics and challenges of a generic MIMO FD-ISAC system are discussed along with a broad overview of state-of-the-art special cases, including numerical investigations. Several directions for future work on FD-enabled ISAC relevant to signal processing communities are also provided.
MotionRL: Align Text-to-Motion Generation to Human Preferences with Multi-Reward Reinforcement Learning
Venue: arXiv cs.CV 2024
Year: 2024

We introduce MotionRL, the first approach to utilize Multi-Reward Reinforcement Learning (RL) for optimizing text-to-motion generation tasks and aligning them with human preferences. Previous works focused on improving numerical performance metrics on the given datasets, often neglecting the variability and subjectivity of human feedback. In contrast, our novel approach uses reinforcement learning to fine-tune the motion generator based on human preferences prior knowledge of the human perception model, allowing it to generate motions that better align human preferences. In addition, MotionRL introduces a novel multi-objective optimization strategy to approximate Pareto optimality between text adherence, motion quality, and human preferences. Extensive experiments and user studies demonstrate that MotionRL not only allows control over the generated results across different objectives but also significantly enhances performance across these metrics compared to other algorithms.
Efficient and Robust Knowledge Distillation from A Stronger Teacher Based on Correlation Matching
Venue: arXiv cs.LG 2024
Year: 2024

Knowledge Distillation (KD) has emerged as a pivotal technique for neural network compression and performance enhancement. Most KD methods aim to transfer dark knowledge from a cumbersome teacher model to a lightweight student model based on Kullback-Leibler (KL) divergence loss. However, the student performance improvements achieved through KD exhibit diminishing marginal returns, where a stronger teacher model does not necessarily lead to a proportionally stronger student model. To address this issue, we empirically find that the KL-based KD method may implicitly change the inter-class relationships learned by the student model, resulting in a more complex and ambiguous decision boundary, which in turn reduces the model's accuracy and generalization ability. Therefore, this study argues that the student model should learn not only the probability values from the teacher's output but also the relative ranking of classes, and proposes a novel Correlation Matching Knowledge Distillation (CMKD) method that combines the Pearson and Spearman correlation coefficients-based KD loss to achieve more efficient and robust distillation from a stronger teacher model. Moreover, considering that samples vary in difficulty, CMKD dynamically adjusts the weights of the Pearson-based loss and Spearman-based loss. CMKD is simple yet practical, and extensive experiments demonstrate that it can consistently achieve state-of-the-art performance on CIRAR-100 and ImageNet, and adapts well to various teacher architectures, sizes, and other KD methods.
Disturbance Observer-based Control Barrier Functions with Residual Model Learning for Safe Reinforcement Learning
Venue: arXiv cs.RO 2024
Year: 2024

Reinforcement learning (RL) agents need to explore their environment to learn optimal behaviors and achieve maximum rewards. However, exploration can be risky when training RL directly on real systems, while simulation-based training introduces the tricky issue of the sim-to-real gap. Recent approaches have leveraged safety filters, such as control barrier functions (CBFs), to penalize unsafe actions during RL training. However, the strong safety guarantees of CBFs rely on a precise dynamic model. In practice, uncertainties always exist, including internal disturbances from the errors of dynamics and external disturbances such as wind. In this work, we propose a new safe RL framework based on disturbance rejection-guarded learning, which allows for an almost model-free RL with an assumed but not necessarily precise nominal dynamic model. We demonstrate our results on the Safety-gym benchmark for Point and Car robots on all tasks where we can outperform state-of-the-art approaches that use only residual model learning or a disturbance observer (DOB). We further validate the efficacy of our framework using a physical F1/10 racing car. Videos: https://sites.google.com/view/res-dob-cbf-rl
Data-informed modeling of the formation, persistence, and evolution of social norms and conventions
Venue: arXiv cs.SI 2024
Year: 2024

Social norms and conventions are commonly accepted and adopted behaviors and practices within a social group that guide interactions -- e.g., how to spell a word or how to greet people -- and are central to a group's culture and identity. Understanding the key mechanisms that govern the formation, persistence, and evolution of social norms and conventions in social communities is a problem of paramount importance for a broad range of real-world applications, spanning from preparedness for future emergencies to promotion of sustainable practices. In the past decades, mathematical modeling has emerged as a powerful tool to reproduce and study the complex dynamics of norm and convention change, gaining insights into their mechanisms, and ultimately deriving tools to predict their evolution. The first goal of this chapter is to introduce some of the main mathematical approaches for modeling social norms and conventions, including population models and agent-based models relying on the theories of dynamical systems, evolutionary dynamics, and game theory. The second goal of the chapter is to illustrate how quantitative observations and empirical data can be incorporated into these mathematical models in a systematic manner, establishing a data-based approach to mathematical modeling of formation, persistence, and evolution of social norms and conventions. Finally, current challenges and future opportunities in this growing field of research are discussed.
Enhancing Multimodal LLM for Detailed and Accurate Video Captioning using Multi-Round Preference Optimization
Venue: arXiv cs.CV 2024
Year: 2024

Videos contain a wealth of information, and generating detailed and accurate descriptions in natural language is a key aspect of video understanding. In this paper, we present video-SALMONN 2, an advanced audio-visual large language model (LLM) with low-rank adaptation (LoRA) designed for enhanced video (with paired audio) captioning through directed preference optimization (DPO). We propose new metrics to evaluate the completeness and accuracy of video descriptions, which are optimized using DPO. To further improve training, we introduce a novel multi-round DPO (mrDPO) approach, which involves periodically updating the DPO reference model, merging and re-initializing the LoRA module as a proxy for parameter updates after each training round (1,000 steps), and incorporating guidance from ground-truth video captions to stabilize the process. To address potential catastrophic forgetting of non-captioning abilities due to mrDPO, we propose rebirth tuning, which finetunes the pre-DPO LLM by using the captions generated by the mrDPO-trained model as supervised labels. Experiments show that mrDPO significantly enhances video-SALMONN 2's captioning accuracy, reducing global and local error rates by 40\% and 20\%, respectively, while decreasing the repetition rate by 35\%. The final video-SALMONN 2 model, with just 7 billion parameters, surpasses leading models such as GPT-4o and Gemini-1.5-Pro in video captioning tasks, while maintaining competitive performance to the state-of-the-art on widely used video question-answering benchmark among models of similar size. Upon acceptance, we will release the code, model checkpoints, and training and test data. Demos are available at \href{https://video-salmonn-2.github.io}{https://video-salmonn-2.github.io}.
Barter Exchange with Bounded Trading Cycles
Venue: arXiv cs.GT 2024
Year: 2024

Consider a barter exchange problem over a finite set of agents, where each agent owns an item and is also associated with a (privately known) wish list of items belonging to the other agents. An outcome of the problem is a (re)allocation of the items to the agents such that each agent either keeps her own item or receives an item from her (reported) wish list, subject to the constraint that the length of the trading cycles induced by the allocation is up-bounded by a prespecified length bound k. The utility of an agent from an allocation is 1 if she receives an item from her (true) wish list and 0 if she keeps her own item (the agent incurs a large dis-utility if she receives an item that is neither hers nor belongs to her wish list). In this paper, we investigate the aforementioned barter exchange problem from the perspective of mechanism design without money, aiming for truthful (and individually rational) mechanisms whose objective is to maximize the social welfare. As the construction of a social welfare maximizing allocation is computationally intractable for length bounds k \geq 3, this paper focuses on (computationally efficient) truthful mechanisms that approximate the (combinatorially) optimal social welfare.We also study a more general version of the barter exchange problem, where the utility of an agent from participating in a trading cycle of length 2 \leq \ell \leq k is lambda(\ell), where λis a general (monotonically non-increasing) length function. Our results include upper and lower bounds on the guaranteed approximation ratio, expressed in terms of the length bound k and the length function λ. On the technical side, our main contribution is an algorithmic tool that can be viewed as a truthful version of the local search paradigm. As it turns out, this tool can be applied to more general (bounded size) coalition formation problems.
Fourier-based Action Recognition for Wildlife Behavior Quantification with Event Cameras
Venue: arXiv cs.CV 2024
Year: 2024

Event cameras are novel bio-inspired vision sensors that measure pixel-wise brightness changes asynchronously instead of images at a given frame rate. They offer promising advantages, namely a high dynamic range, low latency, and minimal motion blur. Modern computer vision algorithms often rely on artificial neural network approaches, which require image-like representations of the data and cannot fully exploit the characteristics of event data. We propose approaches to action recognition based on the Fourier Transform. The approaches are intended to recognize oscillating motion patterns commonly present in nature. In particular, we apply our approaches to a recent dataset of breeding penguins annotated for "ecstatic display", a behavior where the observed penguins flap their wings at a certain frequency. We find that our approaches are both simple and effective, producing slightly lower results than a deep neural network (DNN) while relying just on a tiny fraction of the parameters compared to the DNN (five orders of magnitude fewer parameters). They work well despite the uncontrolled, diverse data present in the dataset. We hope this work opens a new perspective on event-based processing and action recognition.
Orchestrating the Execution of Serverless Functions in Hybrid Clouds
Venue: arXiv cs.DC 2024
Year: 2024

In recent years, serverless computing, especially Function as a Service (FaaS), is rapidly growing in popularity as a cloud programming model. The serverless computing model provides an intuitive interface for developing cloud-based applications, where the development and deployment of scalable microservices has become easier and cost-effective. An increasing number of batch-processing applications are deployed as pipelines that comprise a sequence of functions that must meet their deadline targets to be practical. In this paper, we present our Hybrid Cloud Scheduler (HCS) for orchestrating the execution of serverless batch-processing pipelines deployed over heterogeneous infrastructures. Our framework enables developers to (i) automatically schedule and execute batch-processing applications in heterogeneous environments such as the private edge and public cloud serverless infrastructures, (ii) benefit from cost reduction through the utilization of their own resources in a private cluster, and (iii) significantly improves the probability of meeting the deadline requirements of their applications. Our experimental evaluation demonstrates the efficiency and benefits of our approach.
Evaluating the Impact of Point Cloud Colorization on Semantic Segmentation Accuracy
Venue: arXiv cs.CV 2024
Year: 2024

Point cloud semantic segmentation, the process of classifying each point into predefined categories, is essential for 3D scene understanding. While image-based segmentation is widely adopted due to its maturity, methods relying solely on RGB information often suffer from degraded performance due to color inaccuracies. Recent advancements have incorporated additional features such as intensity and geometric information, yet RGB channels continue to negatively impact segmentation accuracy when errors in colorization occur. Despite this, previous studies have not rigorously quantified the effects of erroneous colorization on segmentation performance. In this paper, we propose a novel statistical approach to evaluate the impact of inaccurate RGB information on image-based point cloud segmentation. We categorize RGB inaccuracies into two types: incorrect color information and similar color information. Our results demonstrate that both types of color inaccuracies significantly degrade segmentation accuracy, with similar color errors particularly affecting the extraction of geometric features. These findings highlight the critical need to reassess the role of RGB information in point cloud segmentation and its implications for future algorithm design.
LR Parsing of Permutation Phrases
Venue: arXiv cs.FL 2024
Year: 2024

This paper presents an efficient method for LR parsing of permutation phrases. In practical cases, the proposed algorithm constructs an LR(0) automaton that requires significantly fewer states to process a permutation phrase compared to the standard construction. For most real-world grammars, the number of states is typically reduced from $Ω(n!)$ to $O(2^{n})$, resulting in a much more compact parsing table. The state reduction increases with longer permutation phrases and a higher number of permutation phrases within the right-hand side of a rule. We demonstrate the effectiveness of this method through its application to parsing a JSON document.
Cooperate or Compete: Coalition Formation in Congestion Games
Venue: arXiv cs.GT 2024
Year: 2024

This paper investigates the potential benefits of cooperation in scenarios where finitely many agents compete for shared resources, leading to congestion and thereby reduced rewards. By appropriate coordination the members of the cooperating group (a.k.a., coalition) can minimize the congestion losses due to inmates, while efficiently facing the competition from outsiders (coalitions indulge in a non-cooperative congestion game). The quest in this paper is to identify the stable partition of coalitions that are not challenged by a new coalition. In contrast to the traditional cooperative games, the worth of a coalition in our game also depends upon the arrangement of the opponents. Every arrangement leads to a partition and a corresponding congestion game; the resultant Nash equilibria (NEs) dictate the `worth'. The analysis is further complicated due to the presence of multiple NEs for each such game.
Shap-Select: Lightweight Feature Selection Using SHAP Values and Regression
Venue: arXiv cs.LG 2024
Year: 2024

Feature selection is an essential process in machine learning, especially when dealing with high-dimensional datasets. It helps reduce the complexity of machine learning models, improve performance, mitigate overfitting, and decrease computation time. This paper presents a novel feature selection framework, shap-select. The framework conducts a linear or logistic regression of the target on the Shapley values of the features, on the validation set, and uses the signs and significance levels of the regression coefficients to implement an efficient heuristic for feature selection in tabular regression and classification tasks. We evaluate shap-select on the Kaggle credit card fraud dataset, demonstrating its effectiveness compared to established methods such as Recursive Feature Elimination (RFE), HISEL (a mutual information-based feature selection method), Boruta and a simpler Shapley value-based method. Our findings show that shap-select combines interpretability, computational efficiency, and performance, offering a robust solution for feature selection.