OpenRaiser/Intern-Atlas huggingface dataset:
Integrating Large Language Models with Internet of Things Applications
Venue: arXiv cs.AI 2024
Year: 2024

This paper identifies and analyzes applications in which Large Language Models (LLMs) can make Internet of Things (IoT) networks more intelligent and responsive through three case studies from critical topics: DDoS attack detection, macroprogramming over IoT systems, and sensor data processing. Our results reveal that the GPT model under few-shot learning achieves 87.6% detection accuracy, whereas the fine-tuned GPT increases the value to 94.9%. Given a macroprogramming framework, the GPT model is capable of writing scripts using high-level functions from the framework to handle possible incidents. Moreover, the GPT model shows efficacy in processing a vast amount of sensor data by offering fast and high-quality responses, which comprise expected results and summarized insights. Overall, the model demonstrates its potential to power a natural language interface. We hope that researchers will find these case studies inspiring to develop further.
Learning ID-free Item Representation with Token Crossing for Multimodal Recommendation
Venue: arXiv cs.IR 2024
Year: 2024

Current multimodal recommendation models have extensively explored the effective utilization of multimodal information; however, their reliance on ID embeddings remains a performance bottleneck. Even with the assistance of multimodal information, optimizing ID embeddings remains challenging for ID-based Multimodal Recommender when interaction data is sparse. Furthermore, the unique nature of item-specific ID embeddings hinders the information exchange among related items and the spatial requirement of ID embeddings increases with the scale of item. Based on these limitations, we propose an ID-free MultimOdal TOken Representation scheme named MOTOR that represents each item using learnable multimodal tokens and connects them through shared tokens. Specifically, we first employ product quantization to discretize each item's multimodal features (e.g., images, text) into discrete token IDs. We then interpret the token embeddings corresponding to these token IDs as implicit item features, introducing a new Token Cross Network to capture the implicit interaction patterns among these tokens. The resulting representations can replace the original ID embeddings and transform the original ID-based multimodal recommender into ID-free system, without introducing any additional loss design. MOTOR reduces the overall space requirements of these models, facilitating information interaction among related items, while also significantly enhancing the model's recommendation capability. Extensive experiments on nine mainstream models demonstrate the significant performance improvement achieved by MOTOR, highlighting its effectiveness in enhancing multimodal recommendation systems.
Stopping Set Analysis for Concatenated Polar Code Architectures
Venue: arXiv cs.IT 2024
Year: 2024

This paper investigates properties of concatenated polar codes and their potential applications. We start with reviewing previous work on stopping set analysis for conventional polar codes, which we extend in this paper to concatenated architectures. Specifically, we present a stopping set analysis for the factor graph of concatenated polar codes, deriving an upper bound on the size of the minimum stopping set. To achieve this bound, we propose new bounds on the size of the minimum stopping set for conventional polar code factor graphs. The tightness of these proposed bounds is investigated empirically and analytically. We show that, in some special cases, the exact value of the minimum stopping set can be determined with a time complexity of $O(N)$, where $N$ is the codeword length. The stopping set analysis motivates a novel construction method for concatenated polar codes. This method is used to design outer polar codes for two previously proposed concatenated polar code architectures: augmented polar codes and local-global polar codes. Simulation results demonstrate the advantage of the proposed codes over previously proposed constructions based on density evolution (DE).
MambaCPU: Enhanced Correlation Mining with State Space Models for CPU Performance Prediction
Venue: arXiv cs.PF 2024
Year: 2024

Forecasting CPU performance, which involves estimating performance scores based on hardware characteristics during operation, is crucial for computational system design and resource management. This research field currently faces two primary challenges. First, the diversity of CPU products and the specialized nature of hardware characteristics make real-world data collection difficult. Second, existing approaches, whether reliant on hardware simulation models or machine learning, suffer from significant drawbacks, such as lengthy simulation cycles, low prediction accuracy, and neglect of characteristic correlations. To address these issues, we first gathered, preprocessed, and standardized historical data from the 4th Generation Intel Xeon Scalable Processors across various benchmark suites to create a new dataset named PerfCastDB. Subsequently, we developed a novel network, MambaCPU (MaC), as the baseline model for the PerfCastDB dataset. This model employs the mamba structure to explore global dependencies and correlations among multiple characteristics. The use of intra- and inter-group attention mechanisms further refines correlations within and between characteristic groups. These techniques enhance MaC's capability to analyze and mine complex multivariate correlations. Comparative experiments on the PerfCastDB dataset demonstrate that MaC surpasses existing methods, confirming its effectiveness. Additionally, we have open-sourced part of the dataset and the MaC code at \url{https://github.com/xiaoman-liu/MaC} to facilitate further research.
Flow Generator Matching
Venue: arXiv cs.CV 2024
Year: 2024

In the realm of Artificial Intelligence Generated Content (AIGC), flow-matching models have emerged as a powerhouse, achieving success due to their robust theoretical underpinnings and solid ability for large-scale generative modeling. These models have demonstrated state-of-the-art performance, but their brilliance comes at a cost. The process of sampling from these models is notoriously demanding on computational resources, as it necessitates the use of multi-step numerical ordinary differential equations (ODEs). Against this backdrop, this paper presents a novel solution with theoretical guarantees in the form of Flow Generator Matching (FGM), an innovative approach designed to accelerate the sampling of flow-matching models into a one-step generation, while maintaining the original performance. On the CIFAR10 unconditional generation benchmark, our one-step FGM model achieves a new record Fréchet Inception Distance (FID) score of 3.08 among few-step flow-matching-based models, outperforming original 50-step flow-matching models. Furthermore, we use the FGM to distill the Stable Diffusion 3, a leading text-to-image flow-matching model based on the MM-DiT architecture. The resulting MM-DiT-FGM one-step text-to-image model demonstrates outstanding industry-level performance. When evaluated on the GenEval benchmark, MM-DiT-FGM has delivered remarkable generating qualities, rivaling other multi-step models in light of the efficiency of a single generation step.
BitPipe: Bidirectional Interleaved Pipeline Parallelism for Accelerating Large Models Training
Venue: arXiv cs.LG 2024
Year: 2024

With the increasing scale of models, the need for efficient distributed training has become increasingly urgent. Recently, many synchronous pipeline parallelism approaches have been proposed to improve training throughput. However, these approaches still suffer from two major issues, i.e., pipeline bubbles caused by periodic flushing and extra communication due to the increasing number of pipeline stages. To this end, we propose BitPipe, a bidirectional interleaved pipeline parallelism for accelerating large models training. Specifically, a hybrid scheme of fusing interleaved pipelines with bidirectional pipelines is proposed to reduce the computational time of each single micro-batch and multiply the number of devices executing simultaneously. A V-shaped schedule with eager gradient synchronization is introduced to reduce and overlap the communication between devices. Experiments conducted on up to 32 GPUs show that BitPipe improves the training throughput of GPT-style and BERT-style models by 1.05x-1.28x compared to the state-of-the-art synchronous approaches. The code of our implementation is available at https://github.com/wuhouming/BitPipe.
Notes on the Mathematical Structure of GPT LLM Architectures
Venue: arXiv cs.LG 2024
Year: 2024

An exposition of the mathematics underpinning the neural network architecture of a GPT-3-style LLM.
Visual Imitation Learning of Non-Prehensile Manipulation Tasks with Dynamics-Supervised Models
Venue: arXiv cs.RO 2024
Year: 2024

Unlike quasi-static robotic manipulation tasks like pick-and-place, dynamic tasks such as non-prehensile manipulation pose greater challenges, especially for vision-based control. Successful control requires the extraction of features relevant to the target task. In visual imitation learning settings, these features can be learnt by backpropagating the policy loss through the vision backbone. Yet, this approach tends to learn task-specific features with limited generalizability. Alternatively, learning world models can realize more generalizable vision backbones. Utilizing the learnt features, task-specific policies are subsequently trained. Commonly, these models are trained solely to predict the next RGB state from the current state and action taken. But only-RGB prediction might not fully-capture the task-relevant dynamics. In this work, we hypothesize that direct supervision of target dynamic states (Dynamics Mapping) can learn better dynamics-informed world models. Beside the next RGB reconstruction, the world model is also trained to directly predict position, velocity, and acceleration of environment rigid bodies. To verify our hypothesis, we designed a non-prehensile 2D environment tailored to two tasks: "Balance-Reaching" and "Bin-Dropping". When trained on the first task, dynamics mapping enhanced the task performance under different training configurations (Decoupled, Joint, End-to-End) and policy architectures (Feedforward, Recurrent). Notably, its most significant impact was for world model pretraining boosting the success rate from 21% to 85%. Although frozen dynamics-informed world models could generalize well to a task with in-domain dynamics, but poorly to a one with out-of-domain dynamics.
Multi-Agent Reinforcement Learning with Selective State-Space Models
Venue: arXiv cs.LG 2024
Year: 2024

The Transformer model has demonstrated success across a wide range of domains, including in Multi-Agent Reinforcement Learning (MARL) where the Multi-Agent Transformer (MAT) has emerged as a leading algorithm in the field. However, a significant drawback of Transformer models is their quadratic computational complexity relative to input size, making them computationally expensive when scaling to larger inputs. This limitation restricts MAT's scalability in environments with many agents. Recently, State-Space Models (SSMs) have gained attention due to their computational efficiency, but their application in MARL remains unexplored. In this work, we investigate the use of Mamba, a recent SSM, in MARL and assess whether it can match the performance of MAT while providing significant improvements in efficiency. We introduce a modified version of MAT that incorporates standard and bi-directional Mamba blocks, as well as a novel "cross-attention" Mamba block. Extensive testing shows that our Multi-Agent Mamba (MAM) matches the performance of MAT across multiple standard multi-agent environments, while offering superior scalability to larger agent scenarios. This is significant for the MARL community, because it indicates that SSMs could replace Transformers without compromising performance, whilst also supporting more effective scaling to higher numbers of agents. Our project page is available at https://sites.google.com/view/multi-agent-mamba .
A Uniform Framework for Problems on Context-Free Grammars
Venue: arXiv cs.FL 2024
Year: 2024

In a paper published in Information Processing Letters in 2000, Bouajjani et al. presented an automata-based approach to a number of elementary problems on context-free grammars. This approach is of pedagogical interest since it provides a uniform solution to decision procedures usually solved by independent algorithms in textbooks. This paper improves upon the work by Bouajjani et al. in a number of ways. We present a new algorithm which not only has a better space complexity but is also (in our opinion) easier to read and understand. Moreover, a closer inspection reveals that the new algorithm is competitive to well-known solutions for most (but not all) standard problems.
Evolving choice hysteresis in reinforcement learning: comparing the adaptive value of positivity bias and gradual perseveration
Venue: arXiv cs.NE 2024
Year: 2024

The tendency of repeating past choices more often than expected from the history of outcomes has been repeatedly empirically observed in reinforcement learning experiments. It can be explained by at least two computational processes: asymmetric update and (gradual) choice perseveration. A recent meta-analysis showed that both mechanisms are detectable in human reinforcement learning. However, while their descriptive value seems to be well established, they have not been compared regarding their possible adaptive value. In this study, we address this gap by simulating reinforcement learning agents in a variety of environments with a new variant of an evolutionary algorithm. Our results show that positivity bias (in the form of asymmetric update) is evolutionary stable in many situations, while the emergence of gradual perseveration is less systematic and robust. Overall, our results illustrate that biases can be adaptive and selected by evolution, in an environment-specific manner.
Gradient Descent Efficiency Index
Venue: arXiv cs.LG 2024
Year: 2024

Gradient descent is a widely used iterative algorithm for finding local minima in multivariate functions. However, the final iterations often either overshoot the minima or make minimal progress, making it challenging to determine an optimal stopping point. This study introduces a new efficiency metric, Ek, designed to quantify the effectiveness of each iteration. The proposed metric accounts for both the relative change in error and the stability of the loss function across iterations. This measure is particularly valuable in resource-constrained environments, where costs are closely tied to training time. Experimental validation across multiple datasets and models demonstrates that Ek provides valuable insights into the convergence behavior of gradient descent, complementing traditional performance metrics. The index has the potential to guide more informed decisions in the selection and tuning of optimization algorithms in machine learning applications and be used to compare the "effectiveness" of models relative to each other.
Intelligent Understanding of Large Language Models in Traditional Chinese Medicine Based on Prompt Engineering Framework
Venue: arXiv cs.CL 2024
Year: 2024

This paper explores the application of prompt engineering to enhance the performance of large language models (LLMs) in the domain of Traditional Chinese Medicine (TCM). We propose TCM-Prompt, a framework that integrates various pre-trained language models (PLMs), templates, tokenization, and verbalization methods, allowing researchers to easily construct and fine-tune models for specific TCM-related tasks. We conducted experiments on disease classification, syndrome identification, herbal medicine recommendation, and general NLP tasks, demonstrating the effectiveness and superiority of our approach compared to baseline methods. Our findings suggest that prompt engineering is a promising technique for improving the performance of LLMs in specialized domains like TCM, with potential applications in digitalization, modernization, and personalized medicine.
Project Lx Conventos: Travelling through space and time in Lisbon's religious buildings
Venue: arXiv cs.HC 2024
Year: 2024

Project Lx Conventos aims to study, in a systematic and integrated manner, the impact of the dissolution of religious orders in the dynamics of urban transformation in nineteenth century Lisbon. After the liberal revolution and the civil war, in the 19th century, the dissolution of religious orders led to the alienation, in Lisbon, of nearly 130 religious buildings which were then given profane occupations (mainly public services) or demolished and divided in plots, originating new urban realities. Project Lx Conventos thus aims to show that the extinction of the convents was decisive in the urban development of Lisbon, in the eighteen hundreds. The project stands on a large set of multimedia data which includes historic and contemporary cartography and geo-referenced photos, videos and 3D models, provided by the projects partners, Lisbon Municipality and the Portuguese National Archive, Torre do Tombo. Supported by these materials, the project's team is creating an online system that will implement a spatial and temporal navigation of these resources integrated in an interactive Map of Lisbon. Besides spatially locating and analyzing the data available for each of the religious buildings considered in the project, the tool integrates cutting edge interaction technology for: 1) Enabling a temporal voyage over the available traces of religious buildings; 2) Analyzing the evolution of religious buildings and their surroundings, through available data; 3) Using 3D representations of the buildings for accessing related data, through time. In this paper, the tools under development in the context of Lx Conventos are described, as well as the technologies supporting them. The current status of the system is presented and future developments are proposed.
Efficient D-2-D via Leader Election: Arbitrary Initial Configuration and No Global Knowledge
Venue: arXiv cs.DC 2024
Year: 2024

Distance-2-Dispersion (D-2-D) problem aims to disperse $k$ mobile agents starting from an arbitrary initial configuration on an anonymous port-labeled graph $G$ with $n$ nodes such that no two agents occupy adjacent nodes in the final configuration, though multiple agents may occupy a single node if there is no other empty node whose all adjacent nodes are also empty. In the existing literature, this problem is solved starting from a rooted configuration for $k$ $(\geq 1)$ agents in $O(mΔ)$ synchronous rounds with a total of $O(\log n)$ memory per agent, where $m$ is the number of edges and $Δ$ is the maximum degree of the graph. In this work we study the D-2-D problem using $n$ mobile agents starting from an arbitrary initial configuration. Solving D-2-D with $n$ agents is equivalent to finding a maximal independent set of the graph as size of any maximal independent set must be less than $n$. We solve this problem and terminate in $O(max\{n\log^2 n, m\})$ rounds using $O(\log n)$ memory per agent. The agents do not have any prior knowledge of any graph parameters. During the run of our algorithm, we also solve the leader election problem that elects an agent as a leader in $O(max\{n\log^2 n, m\})$ rounds with $O(\log n)$ bits of memory at each agent without requiring any prior global knowledge.
A neural network approach for solving the Monge-Ampère equation with transport boundary condition
Venue: arXiv cs.LG 2024
Year: 2024

This paper introduces a novel neural network-based approach to solving the Monge-Ampère equation with the transport boundary condition, specifically targeted towards optical design applications. We leverage multilayer perceptron networks to learn approximate solutions by minimizing a loss function that encompasses the equation's residual, boundary conditions, and convexity constraints. Our main results demonstrate the efficacy of this method, optimized using L-BFGS, through a series of test cases encompassing symmetric and asymmetric circle-to-circle, square-to-circle, and circle-to-flower reflector mapping problems. Comparative analysis with a conventional least-squares finite-difference solver reveals the competitive, and often superior, performance of our neural network approach on the test cases examined here. A comprehensive hyperparameter study further illuminates the impact of factors such as sampling density, network architecture, and optimization algorithm. While promising, further investigation is needed to verify the method's robustness for more complicated problems and to ensure consistent convergence. Nonetheless, the simplicity and adaptability of this neural network-based approach position it as a compelling alternative to specialized partial differential equation solvers.
COR-MP: Conservation of Resources Model for Maneuver Planning
Venue: arXiv cs.RO 2024
Year: 2024

Decision-making for automated driving remains a challenging task. For their integration into real platforms, these algorithms must guarantee passenger safety and comfort while ensuring interpretability and an appropriate computational time. To model and solve this decision-making problem, we have developed a novel approach called COR-MP (Conservation of Resources model for Maneuver Planning). This model is based on the Conservation of Resources theory, a psychological concept applied to human behavior. COR-MP is based on various driving parameters, such as comfort, safety, or energy, and provides in real-time a profit value that enables us to quantify the impact of a decision on the decision-maker. Our method has been tested and validated through closed-loop simulations using RTMaps middleware, and preliminary results have been obtained by testing COR-MP on a real vehicle.
Detection of Human and Machine-Authored Fake News in Urdu
Venue: arXiv cs.CL 2024
Year: 2024

The rise of social media has amplified the spread of fake news, now further complicated by large language models (LLMs) like ChatGPT, which ease the generation of highly convincing, error-free misinformation, making it increasingly challenging for the public to discern truth from falsehood. Traditional fake news detection methods relying on linguistic cues also becomes less effective. Moreover, current detectors primarily focus on binary classification and English texts, often overlooking the distinction between machine-generated true vs. fake news and the detection in low-resource languages. To this end, we updated detection schema to include machine-generated news with focus on the Urdu language. We further propose a hierarchical detection strategy to improve the accuracy and robustness. Experiments show its effectiveness across four datasets in various settings.
Dynamic O(arboricity) coloring in polylogarithmic worst-case time
Venue: arXiv cs.DS 2024
Year: 2024

A recent work by Christiansen, Nowicki, and Rotenberg provides dynamic algorithms for coloring sparse graphs, concretely as a function of the arboricity alpha of the input graph. They give two randomized algorithms: O({alpha} log {alpha}) implicit coloring in poly(log n) worst-case update and query times, and O(min{{alpha} log {alpha}, {alpha} log log log n}) implicit coloring in poly(log n) amortized update and query times (against an oblivious adversary). We improve these results in terms of the number of colors and the time guarantee: First, we present an extremely simple algorithm that computes an O({alpha})-implicit coloring with poly(log n) amortized update and query times. Second, and as the main technical contribution of our work, we show that the time complexity guarantee can be strengthened from amortized to worst-case. That is, we give a dynamic algorithm for implicit O({alpha})-coloring with poly(log n) worst-case update and query times (against an oblivious adversary).