OpenRaiser/Intern-Atlas huggingface dataset:
Loss-to-Loss Prediction: Scaling Laws for All Datasets
Venue: arXiv cs.LG 2024
Year: 2024

While scaling laws provide a reliable methodology for predicting train loss across compute scales for a single data distribution, less is known about how these predictions should change as we change the distribution. In this paper, we derive a strategy for predicting one loss from another and apply it to predict across different pre-training datasets and from pre-training data to downstream task data. Our predictions extrapolate well even at 20x the largest FLOP budget used to fit the curves. More precisely, we find that there are simple shifted power law relationships between (1) the train losses of two models trained on two separate datasets when the models are paired by training compute (train-to-train), (2) the train loss and the test loss on any downstream distribution for a single model (train-to-test), and (3) the test losses of two models trained on two separate train datasets (test-to-test). The results hold up for pre-training datasets that differ substantially (some are entirely code and others have no code at all) and across a variety of downstream tasks. Finally, we find that in some settings these shifted power law relationships can yield more accurate predictions than extrapolating single-dataset scaling laws.
I Can Tell What I am Doing: Toward Real-World Natural Language Grounding of Robot Experiences
Venue: arXiv cs.RO 2024
Year: 2024

Understanding robot behaviors and experiences through natural language is crucial for developing intelligent and transparent robotic systems. Recent advancement in large language models (LLMs) makes it possible to translate complex, multi-modal robotic experiences into coherent, human-readable narratives. However, grounding real-world robot experiences into natural language is challenging due to many reasons, such as multi-modal nature of data, differing sample rates, and data volume. We introduce RONAR, an LLM-based system that generates natural language narrations from robot experiences, aiding in behavior announcement, failure analysis, and human interaction to recover failure. Evaluated across various scenarios, RONAR outperforms state-of-the-art methods and improves failure recovery efficiency. Our contributions include a multi-modal framework for robot experience narration, a comprehensive real-robot dataset, and empirical evidence of RONAR's effectiveness in enhancing user experience in system transparency and failure analysis.
Quadratic Programming Optimization for Bio-Inspired Thruster-Assisted Bipedal Locomotion on Inclined Slopes
Venue: arXiv cs.RO 2024
Year: 2024

Our work aims to make significant strides in understanding unexplored locomotion control paradigms based on the integration of posture manipulation and thrust vectoring. These techniques are commonly seen in nature, such as Chukar birds using their wings to run on a nearly vertical wall. In this work, we show quadratic programming with contact constraints which is then given to the whole body controller to map on robot states to produce a thruster-assisted slope walking controller for our state-of-the-art Harpy platform. Harpy is a bipedal robot capable of legged-aerial locomotion using its legs and thrusters attached to its main frame. The optimization-based walking controller has been used for dynamic locomotion such as slope walking, but the addition of thrusters to perform inclined slope walking has not been extensively explored. In this work, we derive a thruster-assisted bipedal walking with the quadratic programming (QP) controller and implement it in simulation to study its performance.
Oblivious Algorithms for Maximum Directed Cut: New Upper and Lower Bounds
Venue: arXiv cs.DS 2024
Year: 2024

In the maximum directed cut problem, the input is a directed graph $G=(V,E)$, and the goal is to pick a partition $V = S \cup (V \setminus S)$ of the vertices such that as many edges as possible go from $S$ to $V\setminus S$. Oblivious algorithms, introduced by Feige and Jozeph (Algorithmica'17), are a simple class of algorithms for this problem. These algorithms independently and randomly assign each vertex $v$ to either $S$ or $V \setminus S$, and the distribution of $v$'s assignment is determined using only extremely local information about $v$: its bias, i.e., the relative difference between its out- and in-degrees. These algorithms have natural implementations in certain graph streaming models, where they have important implications (Saxena, Singer, Sudan, and Velusamy, SODA'23, FOCS'23, Kallaugher, Parekh, and Voronova, STOC'24). In this work, we narrow the gap between upper and lower bounds on the best approximation ratio achievable by oblivious algorithms for Max-Directed-Cut. We show that there exists an oblivious algorithm achieving an approximation ratio of at least $0.4853$, while every oblivious algorithm obeying a natural symmetry property achieves an approximation ratio of at most $0.4889$. The previous known bounds were $0.4844$ and $0.4899$, due to Singer (APPROX'23) and Feige and Jozeph, respectively. Our techniques involve designing principled parameterizations of the spaces of algorithms and lower bounds and then executing computer searches through these spaces.
LLMSteer: Improving Long-Context LLM Inference by Steering Attention on Reused Contexts
Venue: arXiv cs.LG 2024
Year: 2024

As large language models (LLMs) show impressive performance on complex tasks, they still struggle with longer contextual understanding and high computational costs. To balance efficiency and quality, we introduce LLMSteer, a fine-tuning-free framework that enhances LLMs through query-independent attention steering. Tested on popular LLMs and datasets, LLMSteer narrows the performance gap with baselines by 65.9% and reduces the runtime delay by up to 4.8x compared to recent attention steering methods.
Scalable Deep Metric Learning on Attributed Graphs
Venue: arXiv cs.LG 2024
Year: 2024

We consider the problem of constructing embeddings of large attributed graphs and supporting multiple downstream learning tasks. We develop a graph embedding method, which is based on extending deep metric and unbiased contrastive learning techniques to 1) work with attributed graphs, 2) enabling a mini-batch based approach, and 3) achieving scalability. Based on a multi-class tuplet loss function, we present two algorithms -- DMT for semi-supervised learning and DMAT-i for the unsupervised case. Analyzing our methods, we provide a generalization bound for the downstream node classification task and for the first time relate tuplet loss to contrastive learning. Through extensive experiments, we show high scalability of representation construction, and in applying the method for three downstream tasks (node clustering, node classification, and link prediction) better consistency over any single existing method.
Breaking the Cycle of Recurring Failures: Applying Generative AI to Root Cause Analysis in Legacy Banking Systems
Venue: arXiv cs.SE 2024
Year: 2024

Traditional banks face significant challenges in digital transformation, primarily due to legacy system constraints and fragmented ownership. Recent incidents show that such fragmentation often results in superficial incident resolutions, leaving root causes unaddressed and causing recurring failures. We introduce a novel approach to post-incident analysis, integrating knowledge-based GenAI agents with the "Five Whys" technique to examine problem descriptions and change request data. This method uncovered that approximately 70% of the incidents previously attributed to management or vendor failures were due to underlying internal code issues. We present a case study to show the impact of our method. By scanning over 5,000 projects, we identified over 400 files with a similar root cause. Overall, we leverage the knowledge-based agents to automate and elevate root cause analysis, transforming it into a more proactive process. These agents can be applied across other phases of the software development lifecycle, further improving development processes.
Chanel-Orderer: A Channel-Ordering Predictor for Tri-Channel Natural Images
Venue: arXiv cs.CV 2024
Year: 2024

This paper shows a proof-of-concept that, given a typical 3-channel images but in a randomly permuted channel order, a model (termed as Chanel-Orderer) with ad-hoc inductive biases in terms of both architecture and loss functions can accurately predict the channel ordering and knows how to make it right. Specifically, Chanel-Orderer learns to score each of the three channels with the priors of object semantics and uses the resulting scores to predict the channel ordering. This brings up benefits into a typical scenario where an \texttt{RGB} image is often mis-displayed in the \texttt{BGR} format and needs to be corrected into the right order. Furthermore, as a byproduct, the resulting model Chanel-Orderer is able to tell whether a given image is a near-gray-scale image (near-monochromatic) or not (polychromatic). Our research suggests that Chanel-Orderer mimics human visual coloring of our physical natural world.
Attentive Contextual Attention for Cloud Removal
Venue: arXiv cs.CV 2024
Year: 2024

Cloud cover can significantly hinder the use of remote sensing images for Earth observation, prompting urgent advancements in cloud removal technology. Recently, deep learning strategies have shown strong potential in restoring cloud-obscured areas. These methods utilize convolution to extract intricate local features and attention mechanisms to gather long-range information, improving the overall comprehension of the scene. However, a common drawback of these approaches is that the resulting images often suffer from blurriness, artifacts, and inconsistencies. This is partly because attention mechanisms apply weights to all features based on generalized similarity scores, which can inadvertently introduce noise and irrelevant details from cloud-covered areas. To overcome this limitation and better capture relevant distant context, we introduce a novel approach named Attentive Contextual Attention (AC-Attention). This method enhances conventional attention mechanisms by dynamically learning data-driven attentive selection scores, enabling it to filter out noise and irrelevant features effectively. By integrating the AC-Attention module into the DSen2-CR cloud removal framework, we significantly improve the model's ability to capture essential distant information, leading to more effective cloud removal. Our extensive evaluation of various datasets shows that our method outperforms existing ones regarding image reconstruction quality. Additionally, we conducted ablation studies by integrating AC-Attention into multiple existing methods and widely used network architectures. These studies demonstrate the effectiveness and adaptability of AC-Attention and reveal its ability to focus on relevant features, thereby improving the overall performance of the networks. The code is available at \url{https://github.com/huangwenwenlili/ACA-CRNet}.
Generalized Ping-Pong: Off-Chip Memory Bandwidth Centric Pipelining Strategy for Processing-In-Memory Accelerators
Venue: arXiv cs.AR 2024
Year: 2024

Processing-in-memory (PIM) is a promising choice for accelerating deep neural networks (DNNs) featuring high efficiency and low power. However, the rapid upscaling of neural network model sizes poses a crucial challenge for the limited on-chip PIM capacity. When the PIM presumption of "pre-loading DNN weights/parameters only once before repetitive computing" is no longer practical, concurrent writing and computing techniques become necessary for PIM. Conventional methods of naive ping-pong or in~situ concurrent write/compute scheduling for PIM cause low utilization of off-chip memory bandwidth, subsequently offsetting the efficiency gain brought by PIM technology. To address this challenge, we propose an off-chip memory bandwidth centric pipelining strategy, named "generalized ping-pong", to maximize the utilization and performance of PIM accelerators toward large DNN models. The core idea of the proposed generalized ping-pong strategy is to evenly distribute the active time and fully utilize the off-chip memory bandwidth. Based on a programmable and scalable SRAM PIM architecture, we quantitatively analyze and compare the generalized ping-pong with the conventional scheduling strategies of naive ping-pong and in-situ write/compute for PIM. Experiments show that the generalized ping-pong strategy achieves acceleration of over 1.67 times when fully utilizing the off-chip memory bandwidth. When further limiting the off-chip memory bandwidth ranging in 8~256 bytes per clock cycle, the proposed generalized ping-pong strategy accelerates 1.22~7.71 times versus naive ping-pong. The developed PIM accelerator design with the generalized ping-poing strategy is open-sourced at https://github.com/rw999creator/gpp-pim.
Using ChatGPT-4 for the Identification of Common UX Factors within a Pool of Measurement Items from Established UX Questionnaires
Venue: arXiv cs.HC 2024
Year: 2024

Measuring User Experience (UX) with standardized questionnaires is a widely used method. A questionnaire is based on different scales that represent UX factors and items. However, the questionnaires have no common ground concerning naming different factors and the items used to measure them. This study aims to identify general UX factors based on the formulation of the measurement items. Items from a set of 40 established UX questionnaires were analyzed by Generative AI (GenAI) to identify semantically similar items and to cluster similar topics. We used the LLM ChatGPT-4 for this analysis. Results show that ChatGPT-4 can classify items into meaningful topics and thus help to create a deeper understanding of the structure of the UX research field. In addition, we show that ChatGPT-4 can filter items related to a predefined UX concept out of a pool of UX items.
Virtual Staining of Label-Free Tissue in Imaging Mass Spectrometry
Venue: arXiv cs.CV 2024
Year: 2024

Imaging mass spectrometry (IMS) is a powerful tool for untargeted, highly multiplexed molecular mapping of tissue in biomedical research. IMS offers a means of mapping the spatial distributions of molecular species in biological tissue with unparalleled chemical specificity and sensitivity. However, most IMS platforms are not able to achieve microscopy-level spatial resolution and lack cellular morphological contrast, necessitating subsequent histochemical staining, microscopic imaging and advanced image registration steps to enable molecular distributions to be linked to specific tissue features and cell types. Here, we present a virtual histological staining approach that enhances spatial resolution and digitally introduces cellular morphological contrast into mass spectrometry images of label-free human tissue using a diffusion model. Blind testing on human kidney tissue demonstrated that the virtually stained images of label-free samples closely match their histochemically stained counterparts (with Periodic Acid-Schiff staining), showing high concordance in identifying key renal pathology structures despite utilizing IMS data with 10-fold larger pixel size. Additionally, our approach employs an optimized noise sampling technique during the diffusion model's inference process to reduce variance in the generated images, yielding reliable and repeatable virtual staining. We believe this virtual staining method will significantly expand the applicability of IMS in life sciences and open new avenues for mass spectrometry-based biomedical research.
Domain Adaptive Unfolded Graph Neural Networks
Venue: arXiv cs.LG 2024
Year: 2024

Over the last decade, graph neural networks (GNNs) have made significant progress in numerous graph machine learning tasks. In real-world applications, where domain shifts occur and labels are often unavailable for a new target domain, graph domain adaptation (GDA) approaches have been proposed to facilitate knowledge transfer from the source domain to the target domain. Previous efforts in tackling distribution shifts across domains have mainly focused on aligning the node embedding distributions generated by the GNNs in the source and target domains. However, as the core part of GDA approaches, the impact of the underlying GNN architecture has received limited attention. In this work, we explore this orthogonal direction, i.e., how to facilitate GDA with architectural enhancement. In particular, we consider a class of GNNs that are designed explicitly based on optimization problems, namely unfolded GNNs (UGNNs), whose training process can be represented as bi-level optimization. Empirical and theoretical analyses demonstrate that when transferring from the source domain to the target domain, the lower-level objective value generated by the UGNNs significantly increases, resulting in an increase in the upper-level objective as well. Motivated by this observation, we propose a simple yet effective strategy called cascaded propagation (CP), which is guaranteed to decrease the lower-level objective value. The CP strategy is widely applicable to general UGNNs, and we evaluate its efficacy with three representative UGNN architectures. Extensive experiments on five real-world datasets demonstrate that the UGNNs integrated with CP outperform state-of-the-art GDA baselines.
GraphCL: Graph-based Clustering for Semi-Supervised Medical Image Segmentation
Venue: arXiv cs.CV 2024
Year: 2024

Semi-supervised learning (SSL) has made notable advancements in medical image segmentation (MIS), particularly in scenarios with limited labeled data and significantly enhancing data utilization efficiency. Previous methods primarily focus on complex training strategies to utilize unlabeled data but neglect the importance of graph structural information. Different from existing methods, we propose a graph-based clustering for semi-supervised medical image segmentation (GraphCL) by jointly modeling graph data structure in a unified deep model. The proposed GraphCL model enjoys several advantages. Firstly, to the best of our knowledge, this is the first work to model the data structure information for semi-supervised medical image segmentation (SSMIS). Secondly, to get the clustered features across different graphs, we integrate both pairwise affinities between local image features and raw features as inputs. Extensive experimental results on three standard benchmarks show that the proposed GraphCL algorithm outperforms state-of-the-art semi-supervised medical image segmentation methods.
Learning Time-Optimal and Speed-Adjustable Tactile In-Hand Manipulation
Venue: arXiv cs.RO 2024
Year: 2024

In-hand manipulation with multi-fingered hands is a challenging problem that recently became feasible with the advent of deep reinforcement learning methods. While most contributions to the task brought improvements in robustness and generalization, this paper addresses the critical performance measure of the speed at which an in-hand manipulation can be performed. We present reinforcement learning policies that can perform in-hand reorientation significantly faster than previous approaches for the complex setting of goal-conditioned reorientation in SO(3) with permanent force closure and tactile feedback only (i.e., using the hand's torque and position sensors). Moreover, we show how policies can be trained to be speed-adjustable, allowing for setting the average orientation speed of the manipulated object during deployment. To this end, we present suitable and minimalistic reinforcement learning objectives for time-optimal and speed-adjustable in-hand manipulation, as well as an analysis based on extensive experiments in simulation. We also demonstrate the zero-shot transfer of the learned policies to the real DLR-Hand II with a wide range of target speeds and the fastest dextrous in-hand manipulation without visual inputs.
Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
Venue: arXiv cs.CL 2024
Year: 2024

Efficient inference in large language models (LLMs) has become a critical focus as their scale and complexity grow. Traditional autoregressive decoding, while effective, suffers from computational inefficiencies due to its sequential token generation process. Speculative decoding addresses this bottleneck by introducing a two-stage framework: drafting and verification. A smaller, efficient model generates a preliminary draft, which is then refined by a larger, more sophisticated model. This paper provides a comprehensive survey of speculative decoding methods, categorizing them into draft-centric and model-centric approaches. We discuss key ideas associated with each method, highlighting their potential for scaling LLM inference. This survey aims to guide future research in optimizing speculative decoding and its integration into real-world LLM applications.
Guided Object-Oriented Development
Venue: arXiv cs.SE 2024
Year: 2024

To improve the quality of programs we provide an approach to guidance in the process of program development. At the higher level the various activities and their dependencies to structure the process are identified. At the lower level, detailed, practical rules are given for the decision-making in the development steps during these activities. The approach concentrates on structure and behavior of a single class. It includes design and specification and is compatible with methodologies for programming in the large. Informal specifications are introduced to help develop correct and robust code as well as corresponding tests. A strict distinction is made between external design and specification on one hand and internal design and specification on the other hand, which helps in keeping control over complexity. The approach also exploits the separation of success and failure scenarios. A worked-out example is provided.
Simultaneous Communication and Tracking using Fused Bistatic Measurements
Venue: arXiv cs.IT 2024
Year: 2024

In this paper, we propose a bistatic sensing-assisted beam tracking method for simultaneous communication and tracking of user vehicles navigating arbitrary-shaped road trajectories. Prior work on simultaneous communication and tracking assumes a colocated radar receiver at the transmitter for sensing measurements using the reflected Integrated Sensing and Communication (ISAC) signals in the mmWave band. Full isolation between transmitter and receiver is required here to avoid self-interference. We consider the bistatic setting where the sensing receivers are not colocated and can be realized in practice using traditional half-duplex transmit or receive nodes. First, we process the echoes reflected from the vehicle at multiple multi-antenna nodes at various locations, facilitating estimation of the vehicle's current position. Then, we propose selection criteria for the estimates and a maximum likelihood (ML) fusion scheme to fuse these selected estimates based on the estimated error covariance matrices of these measurements. This fusion scheme is important in bistatic and multistatic settings as the localization error depends significantly on the geometry of the transmitter, target, and receiver locations. Finally, we predict the vehicle's next location using a simple kinematic equation-based model. Through extensive simulation, we study the average spectral efficiency of communication with a moving user using the proposed simultaneous communication and tracking scheme. The proposed fusion-based scheme achieves almost the same average spectral efficiency as an ideal scheme that knows the exact trajectory. We also show that the proposed scheme can be easily extended to systems with Hybrid Digital-Analog architectures and performs similarly even in these systems.