OpenRaiser/Intern-Atlas huggingface dataset:
Theseus: Exploring Efficient Wafer-Scale Chip Design for Large Language Models
Venue: arXiv cs.AR 2024
Year: 2024

The emergence of the large language model~(LLM) poses an exponential growth of demand for computation throughput, memory capacity, and communication bandwidth. Such a demand growth has significantly surpassed the improvement of corresponding chip designs. With the advancement of fabrication and integration technologies, designers have been developing Wafer-Scale Chips~(WSCs) to scale up and exploit the limits of computation density, memory capacity, and communication bandwidth at the level of a single chip. Existing solutions have demonstrated the significant advantages of WSCs over traditional designs, showing potential to effectively support LLM workloads. Despite the benefits, exploring the early-stage design space of WSCs for LLMs is a crucial yet challenging task due to the enormous and complicated design space, time-consuming evaluation methods, and inefficient exploration strategies. To address these challenges, we propose Theseus, an efficient WSC design space exploration framework for LLMs. We construct the design space of WSCs with various constraints considering the unique characteristics of WSCs. We propose efficient evaluation methodologies for large-scale NoC-based WSCs and introduce multi-fidelity Bayesian optimization to efficiently explore the design space. Evaluation results demonstrate the efficiency of Theseus that the searched Pareto optimal results outperform GPU cluster and existing WSC designs by up to 62.8\%/73.7\% in performance and 38.6\%/42.4\% in power consumption for LLM training, while improving up to 23.2$\times$ and 15.7$\times$ for the performance and power of inference tasks. Furthermore, we conduct case studies to address the design tradeoffs in WSCs and provide insights to facilitate WSC designs for LLMs.
Exploring Federated Learning Dynamics for Black-and-White-Box DNN Traitor Tracing
Venue: arXiv cs.CR 2024
Year: 2024

As deep learning applications become more prevalent, the need for extensive training examples raises concerns for sensitive, personal, or proprietary data. To overcome this, Federated Learning (FL) enables collaborative model training across distributed data-owners, but it introduces challenges in safeguarding model ownership and identifying the origin in case of a leak. Building upon prior work, this paper explores the adaptation of black-and-white traitor tracing watermarking to FL classifiers, addressing the threat of collusion attacks from different data-owners. This study reveals that leak-resistant white-box fingerprints can be directly implemented without a significant impact from FL dynamics, while the black-box fingerprints are drastically affected, losing their traitor tracing capabilities. To mitigate this effect, we propose increasing the number of black-box salient neurons through dropout regularization. Though there are still some open problems to be explored, such as analyzing non-i.i.d. datasets and over-parameterized models, results show that collusion-resistant traitor tracing, identifying all data-owners involved in a suspected leak, is feasible in an FL framework, even in early stages of training.
Non-Terrestrial Networks for 6G: Integrated, Intelligent and Ubiquitous Connectivity
Venue: arXiv cs.NI 2024
Year: 2024

Universal connectivity has been part of past and current generations of wireless systems, but as we approach 6G, the subject of social responsibility is being built as a core component. Given the advent of Non-Terrestrial Networks (NTN), reaching these goals will be much closer to realization than ever before. Owing to the benefits of NTN, the integration NTN and Terrestrial Networks (TN) is still infancy, where the past, the current and the future releases in the 3$^{\text{rd}}$ Generation Partnership Project (3GPP) provide guidelines to adopt a successfully co-existence/integration of TN and NTN. Therefore, in this article, we have illustrated through 3GPP guidelines, on how NTN and TN can effectively be integrated. Moreover, the role of beamforming and Artificial Intelligence (AI) algorithms is highlighted to achieve this integration. Finally the usefulness of integrating NTN and TN is validated through experimental analysis.
Virtually Objective Quantification of in vitro Wound Healing Scratch Assays with the Segment Anything Model
Venue: arXiv cs.CV 2024
Year: 2024

The in vitro scratch assay is a widely used assay in cell biology to assess the rate of wound closure related to a variety of therapeutic interventions. While manual measurement is subjective and vulnerable to intra- and interobserver variability, computer-based tools are theoretically objective, but in practice often contain parameters which are manually adjusted (individually per image or data set) and thereby provide a source for subjectivity. Modern deep learning approaches typically require large annotated training data which complicates instant applicability. In this paper, we make use of the segment anything model, a deep foundation model based on interactive point-prompts, which enables class-agnostic segmentation without tuning the network's parameters based on domain specific training data. The proposed method clearly outperformed a semi-objective baseline method that required manual inspection and, if necessary, adjustment of parameters per image. Even though the point prompts of the proposed approach are theoretically also a source for subjectivity, results attested very low intra- and interobserver variability, even compared to manual segmentation of domain experts.
Categorized Grid and Unknown Space Causes for LiDAR-based Dynamic Occupancy Grids
Venue: arXiv cs.RO 2024
Year: 2024

Occupancy Grids have been widely used for perception of the environment as they allow to model the obstacles in the scene, as well as free and unknown space. Recently, there has been a growing interest in the unknown space due to the necessity of better understanding the situation. Although Occupancy Grids have received numerous extensions over the years to address emerging needs, currently, few works go beyond the delimitation of the unknown space area and seek to incorporate additional information. This work builds upon the already well-established LiDAR-based Dynamic Occupancy Grid to introduce a complementary Categorized Grid that conveys its estimation using semantic labels while adding new insights into the possible causes of unknown space. The proposed categorization first divides the space by occupancy and then further categorizes the occupied and unknown space. Occupied space is labeled based on its dynamic state and reliability, while the unknown space is labeled according to its possible causes, whether they stem from the perception system's inherent constraints, limitations induced by the environment, or other causes. The proposed Categorized Grid is showcased in real-world scenarios demonstrating its usefulness for better situation understanding.
Concurrent Binary Trees for Large-Scale Game Components
Venue: arXiv cs.GR 2024
Year: 2024

A concurrent binary tree (CBT) is a GPU-friendly data-structure suitable for the generation of bisection based terrain tessellations, i.e., adaptive triangulations over square domains. In this paper, we expand the benefits of this data-structure in two respects. First, we show how to bring bisection based tessellations to arbitrary polygon meshes rather than just squares. Our approach consists of mapping a triangular subdivision primitive, which we refer to as a bisector, to each halfedge of the input mesh. These bisectors can then be subdivided adaptively to produce conforming triangulations solely based on halfedge operators. Second, we alleviate a limitation that restricted the triangulations to low subdivision levels. We do so by using the CBT as a memory pool manager rather than an implicit encoding of the triangulation as done originally. By using a CBT in this way, we concurrently allocate and/or release bisectors during adaptive subdivision using shared GPU memory. We demonstrate the benefits of our improvements by rendering planetary scale geometry out of very coarse meshes. Performance-wise, our triangulation method evaluates in less than 0.2ms on console-level hardware.
Sign Language Recognition Based On Facial Expression and Hand Skeleton
Venue: arXiv cs.CV 2024
Year: 2024

Sign language is a visual language used by the deaf and dumb community to communicate. However, for most recognition methods based on monocular cameras, the recognition accuracy is low and the robustness is poor. Even if the effect is good on some data, it may perform poorly in other data with different interference due to the inability to extract effective features. To solve these problems, we propose a sign language recognition network that integrates skeleton features of hands and facial expression. Among this, we propose a hand skeleton feature extraction based on coordinate transformation to describe the shape of the hand more accurately. Moreover, by incorporating facial expression information, the accuracy and robustness of sign language recognition are finally improved, which was verified on A Dataset for Argentinian Sign Language and SEU's Chinese Sign Language Recognition Database (SEUCSLRD).
FedIA: Federated Medical Image Segmentation with Heterogeneous Annotation Completeness
Venue: arXiv cs.CV 2024
Year: 2024

Federated learning has emerged as a compelling paradigm for medical image segmentation, particularly in light of increasing privacy concerns. However, most of the existing research relies on relatively stringent assumptions regarding the uniformity and completeness of annotations across clients. Contrary to this, this paper highlights a prevalent challenge in medical practice: incomplete annotations. Such annotations can introduce incorrectly labeled pixels, potentially undermining the performance of neural networks in supervised learning. To tackle this issue, we introduce a novel solution, named FedIA. Our insight is to conceptualize incomplete annotations as noisy data (i.e., low-quality data), with a focus on mitigating their adverse effects. We begin by evaluating the completeness of annotations at the client level using a designed indicator. Subsequently, we enhance the influence of clients with more comprehensive annotations and implement corrections for incomplete ones, thereby ensuring that models are trained on accurate data. Our method's effectiveness is validated through its superior performance on two extensively used medical image segmentation datasets, outperforming existing solutions. The code is available at https://github.com/HUSTxyy/FedIA.
Learning Bipedal Walking on a Quadruped Robot via Adversarial Motion Priors
Venue: arXiv cs.RO 2024
Year: 2024

Previous studies have successfully demonstrated agile and robust locomotion in challenging terrains for quadrupedal robots. However, the bipedal locomotion mode for quadruped robots remains unverified. This paper explores the adaptation of a learning framework originally designed for quadrupedal robots to operate blind locomotion in biped mode. We leverage a framework that incorporates Adversarial Motion Priors with a teacher-student policy to enable imitation of a reference trajectory and navigation on tough terrain. Our work involves transferring and evaluating a similar learning framework on a quadruped robot in biped mode, aiming to achieve stable walking on both flat and complicated terrains. Our simulation results demonstrate that the trained policy enables the quadruped robot to navigate both flat and challenging terrains, including stairs and uneven surfaces.
Efficient Sparse Attention needs Adaptive Token Release
Venue: arXiv cs.CL 2024
Year: 2024

In recent years, Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide array of text-centric tasks. However, their `large' scale introduces significant computational and storage challenges, particularly in managing the key-value states of the transformer, which limits their wider applicability. Therefore, we propose to adaptively release resources from caches and rebuild the necessary key-value states. Particularly, we accomplish this by a lightweight controller module to approximate an ideal top-$K$ sparse attention. This module retains the tokens with the highest top-$K$ attention weights and simultaneously rebuilds the discarded but necessary tokens, which may become essential for future decoding. Comprehensive experiments in natural language generation and modeling reveal that our method is not only competitive with full attention in terms of performance but also achieves a significant throughput improvement of up to 221.8%. The code for replication is available on the https://github.com/WHUIR/ADORE.
Fast, Scalable, Energy-Efficient Non-element-wise Matrix Multiplication on FPGA
Venue: arXiv cs.AR 2024
Year: 2024

Modern Neural Network (NN) architectures heavily rely on vast numbers of multiply-accumulate arithmetic operations, constituting the predominant computational cost. Therefore, this paper proposes a high-throughput, scalable and energy efficient non-element-wise matrix multiplication unit on FPGAs as a basic component of the NNs. We firstly streamline inter-layer and intra-layer redundancies of MADDNESS algorithm, a LUT-based approximate matrix multiplication, to design a fast, efficient scalable approximate matrix multiplication module termed "Approximate Multiplication Unit (AMU)". The AMU optimizes LUT-based matrix multiplications further through dedicated memory management and access design, decoupling computational overhead from input resolution and boosting FPGA-based NN accelerator efficiency significantly. The experimental results show that using our AMU achieves up to 9x higher throughput and 112x higher energy efficiency over the state-of-the-art solutions for the FPGA-based Quantised Neural Network (QNN) accelerators.
Real Time Collision Avoidance with GPU-Computed Distance Maps
Venue: arXiv cs.RO 2024
Year: 2024

This paper presents reactive obstacle and self-collision avoidance of redundant robotic manipulators within real time kinematic feedback control using GPU-computed distance transform. The proposed framework utilizes discretized representation of the robot and the environment to calculate 3D Euclidean distance transform for task-priority based kinematic control. The environment scene is represented using a 3D GPU-voxel map created and updated from a live pointcloud data while the robotic link model is converted into a voxels offline and inserted into the voxel map according to the joint state of the robot to form the self-obstacle map. The proposed approach is evaluated using the Tiago robot, showing that all obstacle and self collision avoidance constraints are respected within one framework even with fast moving obstacles while the robot performs end-effector pose tracking in real time. A comparison of related works that depend on GPU and CPU computed distance fields is also presented to highlight the time performance as well as accuracy of the GPU distance field.
Consistency Flow Matching: Defining Straight Flows with Velocity Consistency
Venue: arXiv cs.CV 2024
Year: 2024

Flow matching (FM) is a general framework for defining probability paths via Ordinary Differential Equations (ODEs) to transform between noise and data samples. Recent approaches attempt to straighten these flow trajectories to generate high-quality samples with fewer function evaluations, typically through iterative rectification methods or optimal transport solutions. In this paper, we introduce Consistency Flow Matching (Consistency-FM), a novel FM method that explicitly enforces self-consistency in the velocity field. Consistency-FM directly defines straight flows starting from different times to the same endpoint, imposing constraints on their velocity values. Additionally, we propose a multi-segment training approach for Consistency-FM to enhance expressiveness, achieving a better trade-off between sampling quality and speed. Preliminary experiments demonstrate that our Consistency-FM significantly improves training efficiency by converging 4.4x faster than consistency models and 1.7x faster than rectified flow models while achieving better generation quality. Our code is available at: https://github.com/YangLing0818/consistency_flow_matching
Assessing the Code Clone Detection Capability of Large Language Models
Venue: arXiv cs.SE 2024
Year: 2024

This study aims to assess the performance of two advanced Large Language Models (LLMs), GPT-3.5 and GPT-4, in the task of code clone detection. The evaluation involves testing the models on a variety of code pairs of different clone types and levels of similarity, sourced from two datasets: BigCloneBench (human-made) and GPTCloneBench (LLM-generated). Findings from the study indicate that GPT-4 consistently surpasses GPT-3.5 across all clone types. A correlation was observed between the GPTs' accuracy at identifying code clones and code similarity, with both GPT models exhibiting low effectiveness in detecting the most complex Type-4 code clones. Additionally, GPT models demonstrate a higher performance identifying code clones in LLM-generated code compared to humans-generated code. However, they do not reach impressive accuracy. These results emphasize the imperative for ongoing enhancements in LLM capabilities, particularly in the recognition of code clones and in mitigating their predisposition towards self-generated code clones--which is likely to become an issue as software engineers are more numerous to leverage LLM-enabled code generation and code refactoring tools.
Tiny-PULP-Dronets: Squeezing Neural Networks for Faster and Lighter Inference on Multi-Tasking Autonomous Nano-Drones
Venue: arXiv cs.RO 2024
Year: 2024

Pocket-sized autonomous nano-drones can revolutionize many robotic use cases, such as visual inspection in narrow, constrained spaces, and ensure safer human-robot interaction due to their tiny form factor and weight -- i.e., tens of grams. This compelling vision is challenged by the high level of intelligence needed aboard, which clashes against the limited computational and storage resources available on PULP (parallel-ultra-low-power) MCU class navigation and mission controllers that can be hosted aboard. This work moves from PULP-Dronet, a State-of-the-Art convolutional neural network for autonomous navigation on nano-drones. We introduce Tiny-PULP-Dronet: a novel methodology to squeeze by more than one order of magnitude model size (50x fewer parameters), and number of operations (27x less multiply-and-accumulate) required to run inference with similar flight performance as PULP-Dronet. This massive reduction paves the way towards affordable multi-tasking on nano-drones, a fundamental requirement for achieving high-level intelligence.
Comparative Evaluation of Learning Models for Bionic Robots: Non-Linear Transfer Function Identifications
Venue: arXiv cs.RO 2024
Year: 2024

The control and modeling of robot dynamics have increasingly adopted model-free control strategies using machine learning. Given the non-linear elastic nature of bionic robotic systems, learning-based methods provide reliable alternatives by utilizing numerical data to establish a direct mapping from actuation inputs to robot trajectories without complex kinematics models. However, for developers, the method of identifying an appropriate learning model for their specific bionic robots and further constructing the transfer function has not been thoroughly discussed. Thus, this research introduces a comprehensive evaluation strategy and framework for the application of model-free control, including data collection, learning model selection, comparative analysis, and transfer function identification to effectively deal with the multi-input multi-output (MIMO) robotic data.
Meta 3D TextureGen: Fast and Consistent Texture Generation for 3D Objects
Venue: arXiv cs.CV 2024
Year: 2024

The recent availability and adaptability of text-to-image models has sparked a new era in many related domains that benefit from the learned text priors as well as high-quality and fast generation capabilities, one of which is texture generation for 3D objects. Although recent texture generation methods achieve impressive results by using text-to-image networks, the combination of global consistency, quality, and speed, which is crucial for advancing texture generation to real-world applications, remains elusive. To that end, we introduce Meta 3D TextureGen: a new feedforward method comprised of two sequential networks aimed at generating high-quality and globally consistent textures for arbitrary geometries of any complexity degree in less than 20 seconds. Our method achieves state-of-the-art results in quality and speed by conditioning a text-to-image model on 3D semantics in 2D space and fusing them into a complete and high-resolution UV texture map, as demonstrated by extensive qualitative and quantitative evaluations. In addition, we introduce a texture enhancement network that is capable of up-scaling any texture by an arbitrary ratio, producing 4k pixel resolution textures.
A New Achievable Region of the $K$-User MAC Wiretap Channel with Confidential and Open Messages Under Strong Secrecy
Venue: arXiv cs.IT 2024
Year: 2024

This paper investigates the achievable region of a $K$-user discrete memoryless (DM) multiple access wiretap (MAC-WT) channel, where each user transmits both secret and open messages. All these messages are intended for Bob, while Eve is only interested in the secret messages. In the achievable coding strategy, the confidential information is protected by open messages and also by the introduction of auxiliary messages. When introducing an auxiliary message, one has to ensure that, on one hand, its rate is large enough for protecting the secret message from Eve and, on the other hand, the resulting sum rate (together with the secret and open message rate) does not exceed Bob's decoding capability. This yields an inequality structure involving the rates of all users' secret, open, and auxiliary messages. To obtain the rate region, the auxiliary message rates must be eliminated from the system of inequalities. A direct application of the Fourier-Motzkin elimination procedure is elusive since a) it requires that the number of users $K$ is explicitly given, and b) even for small $K = 3, 4, \ldots$, the number of inequalities becomes extremely large. We prove the result for general $K$ through the combined use of Fourier-Motzkin elimination procedure and mathematical induction. This paper adopts the strong secrecy metric, characterized by information leakage. To prove the achievability under this criterion, we analyze the resolvability region of a $K$-user DM-MAC channel. In addition, we show that users with zero secrecy rate can play different roles and use different strategies in encoding their messages. These strategies yield non-redundant rate inequalities. By considering all possible coding strategies, we provide a new achievable region for the considered channel, and show that it strictly improves those already known in the existing literature by considering a specific example.