OpenRaiser/Intern-Atlas huggingface dataset:
Moderating New Waves of Online Hate with Chain-of-Thought Reasoning in Large Language Models
Venue: arXiv cs.CL 2024
Year: 2024

Online hate is an escalating problem that negatively impacts the lives of Internet users, and is also subject to rapid changes due to evolving events, resulting in new waves of online hate that pose a critical threat. Detecting and mitigating these new waves present two key challenges: it demands reasoning-based complex decision-making to determine the presence of hateful content, and the limited availability of training samples hinders updating the detection model. To address this critical issue, we present a novel framework called HATEGUARD for effectively moderating new waves of online hate. HATEGUARD employs a reasoning-based approach that leverages the recently introduced chain-of-thought (CoT) prompting technique, harnessing the capabilities of large language models (LLMs). HATEGUARD further achieves prompt-based zero-shot detection by automatically generating and updating detection prompts with new derogatory terms and targets in new wave samples to effectively address new waves of online hate. To demonstrate the effectiveness of our approach, we compile a new dataset consisting of tweets related to three recently witnessed new waves: the 2022 Russian invasion of Ukraine, the 2021 insurrection of the US Capitol, and the COVID-19 pandemic. Our studies reveal crucial longitudinal patterns in these new waves concerning the evolution of events and the pressing need for techniques to rapidly update existing moderation tools to counteract them. Comparative evaluations against state-of-the-art tools illustrate the superiority of our framework, showcasing a substantial 22.22% to 83.33% improvement in detecting the three new waves of online hate. Our work highlights the severe threat posed by the emergence of new waves of online hate and represents a paradigm shift in addressing this threat practically.
Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inference
Venue: arXiv cs.LG 2024
Year: 2024

Recent advancements in large language models (LLMs) boasting billions of parameters have generated a significant demand for efficient deployment in inference workloads. The majority of existing approaches rely on temporal architectures that reuse hardware units for different network layers and operators. However, these methods often encounter challenges in achieving low latency due to considerable memory access overhead. This paper investigates the feasibility and potential of model-specific spatial acceleration for LLM inference on FPGAs. Our approach involves the specialization of distinct hardware units for specific operators or layers, facilitating direct communication between them through a dataflow architecture while minimizing off-chip memory accesses. We introduce a comprehensive analytical model for estimating the performance of a spatial LLM accelerator, taking into account the on-chip compute and memory resources available on an FPGA. Through our analysis, we can determine the scenarios in which FPGA-based spatial acceleration can outperform its GPU-based counterpart. To enable more productive implementations of an LLM model on FPGAs, we further provide a library of high-level synthesis (HLS) kernels that are composable and reusable. This library will be made available as open-source. To validate the effectiveness of both our analytical model and HLS library, we have implemented BERT and GPT2 on an AMD Alveo U280 FPGA device. Experimental results demonstrate our approach can achieve up to 13.4x speedup when compared to previous FPGA-based accelerators for the BERT model. For GPT generative inference, we attain a 2.2x speedup compared to DFX, an FPGA overlay, in the prefill stage, while achieving a 1.9x speedup and a 5.7x improvement in energy efficiency compared to the NVIDIA A100 GPU in the decode stage.
LLM-Powered Hierarchical Language Agent for Real-time Human-AI Coordination
Venue: arXiv cs.AI 2024
Year: 2024

AI agents powered by Large Language Models (LLMs) have made significant advances, enabling them to assist humans in diverse complex tasks and leading to a revolution in human-AI coordination. LLM-powered agents typically require invoking LLM APIs and employing artificially designed complex prompts, which results in high inference latency. While this paradigm works well in scenarios with minimal interactive demands, such as code generation, it is unsuitable for highly interactive and real-time applications, such as gaming. Traditional gaming AI often employs small models or reactive policies, enabling fast inference but offering limited task completion and interaction abilities. In this work, we consider Overcooked as our testbed where players could communicate with natural language and cooperate to serve orders. We propose a Hierarchical Language Agent (HLA) for human-AI coordination that provides both strong reasoning abilities while keeping real-time execution. In particular, HLA adopts a hierarchical framework and comprises three modules: a proficient LLM, referred to as Slow Mind, for intention reasoning and language interaction, a lightweight LLM, referred to as Fast Mind, for generating macro actions, and a reactive policy, referred to as Executor, for transforming macro actions into atomic actions. Human studies show that HLA outperforms other baseline agents, including slow-mind-only agents and fast-mind-only agents, with stronger cooperation abilities, faster responses, and more consistent language communications.
Understanding normalization in contrastive representation learning and out-of-distribution detection
Venue: arXiv cs.CV 2024
Year: 2024

Contrastive representation learning has emerged as an outstanding approach for anomaly detection. In this work, we explore the $\ell_2$-norm of contrastive features and its applications in out-of-distribution detection. We propose a simple method based on contrastive learning, which incorporates out-of-distribution data by discriminating against normal samples in the contrastive layer space. Our approach can be applied flexibly as an outlier exposure (OE) approach, where the out-of-distribution data is a huge collective of random images, or as a fully self-supervised learning approach, where the out-of-distribution data is self-generated by applying distribution-shifting transformations. The ability to incorporate additional out-of-distribution samples enables a feasible solution for datasets where AD methods based on contrastive learning generally underperform, such as aerial images or microscopy images. Furthermore, the high-quality features learned through contrastive learning consistently enhance performance in OE scenarios, even when the available out-of-distribution dataset is not diverse enough. Our extensive experiments demonstrate the superiority of our proposed method under various scenarios, including unimodal and multimodal settings, with various image datasets.
Exploring the Capabilities of ChatGPT in Ancient Chinese Translation and Person Name Recognition
Venue: arXiv cs.CL 2024
Year: 2024

ChatGPT's proficiency in handling modern standard languages suggests potential for its use in understanding ancient Chinese. This paper explores ChatGPT's capabilities on ancient Chinese via two tasks: translating ancient Chinese to modern Chinese and recognizing ancient Chinese names. A comparison of ChatGPT's output with human translations serves to evaluate its comprehension of ancient Chinese. The findings indicate that: (1.)the proficiency of ancient Chinese by ChatGPT is yet to reach a satisfactory level; (2.) ChatGPT performs the best on ancient-to-modern translation when feeding with three context sentences. To help reproduce our work, we display the python code snippets used in this study.
Greedy Grammar Induction with Indirect Negative Evidence
Venue: arXiv cs.CL 2024
Year: 2024

This paper offers a fresh look at the pumping lemma constant as an upper bound on the information required for learning Context Free Grammars. An objective function based on indirect negative evidence considers the occurrences, and non-occurrences, of a finite number of strings, encountered after a sufficiently long presentation. This function has optimal substructure in the hypotheses space, giving rise to a greedy search learner in a branch and bound method. A hierarchy of learnable classes is defined in terms of the number of production rules that must be added to interim solutions in order to incrementally fit the input. Efficiency strongly depends on the position of the target grammar in the hierarchy and on the richness of the input.
On Safety and Liveness Filtering Using Hamilton-Jacobi Reachability Analysis
Venue: arXiv cs.RO 2024
Year: 2024

Hamilton-Jacobi (HJ) reachability-based filtering provides a powerful framework to co-optimize performance and safety (or liveness) for autonomous systems. Under this filtering scheme, a nominal controller is minimally modified to ensure system safety or liveness. However, the resulting controllers can exhibit abrupt switching and bang-bang behavior, which is not suitable for applications of autonomous systems in the real world. This work presents a novel, unifying framework to design safety and liveness filters through reachability analysis. We explicitly characterize the maximal set of control inputs that ensures safety (or liveness) at a given state. Different safety filters can then be constructed using different subsets of this maximal set along with a projection operator to modify the nominal controller. We use the proposed framework to design three safety filters, each balancing performance, computation time, and smoothness differently. We highlight their relative strengths and limitations by applying these filters to autonomous navigation and rocket landing scenarios and on a physical robot testbed. We also discuss practical aspects associated with implementing these filters on real-world autonomous systems. Our research advances the understanding and potential application of reachability-based controllers on real-world autonomous systems.
Closed and asymptotic formulae for harmonic and quadratic harmonic sums
Venue: arXiv cs.DM 2024
Year: 2024

We present here a large collection of harmonic and quadratic harmonic sums, that can be useful in applied questions, e.g., probabilistic ones. We find closed-form formulae, that we were not able to locate in the literature.
A-SDM: Accelerating Stable Diffusion through Redundancy Removal and Performance Optimization
Venue: arXiv cs.CV 2024
Year: 2024

The Stable Diffusion Model (SDM) is a popular and efficient text-to-image (t2i) generation and image-to-image (i2i) generation model. Although there have been some attempts to reduce sampling steps, model distillation, and network quantization, these previous methods generally retain the original network architecture. Billion scale parameters and high computing requirements make the research of model architecture adjustment scarce. In this work, we first explore the computational redundancy part of the network, and then prune the redundancy blocks of the model and maintain the network performance through a progressive incubation strategy. Secondly, in order to maintaining the model performance, we add cross-layer multi-expert conditional convolution (CLME-Condconv) to the block pruning part to inherit the original convolution parameters. Thirdly, we propose a global-regional interactive (GRI) attention to speed up the computationally intensive attention part. Finally, we use semantic-aware supervision (SAS) to align the outputs of the teacher model and student model at the semantic level. Experiments show that this method can effectively train a lightweight model close to the performance of the original SD model, and effectively improve the model speed under limited resources. Experiments show that the proposed method can effectively train a light-weight model close to the performance of the original SD model, and effectively improve the model speed under limited resources. After acceleration, the UNet part of the model is 22% faster and the overall speed is 19% faster.
ITEACH-Net: Inverted Teacher-studEnt seArCH Network for Emotion Recognition in Conversation
Venue: arXiv cs.MM 2024
Year: 2024

There remain two critical challenges that hinder the development of ERC. Firstly, there is a lack of exploration into mining deeper insights from the data itself for conversational emotion tasks. Secondly, the systems exhibit vulnerability to random modality feature missing, which is a common occurrence in realistic settings. Focusing on these two key challenges, we propose a novel framework for incomplete multimodal learning in ERC, called "Inverted Teacher-studEnt seArCH Network (ITEACH-Net)." ITEACH-Net comprises two novel components: the Emotion Context Changing Encoder (ECCE) and the Inverted Teacher-Student (ITS) framework. Specifically, leveraging the tendency for emotional states to exhibit local stability within conversational contexts, ECCE captures these patterns and further perceives their evolution over time. Recognizing the varying challenges of handling incomplete versus complete data, ITS employs a teacher-student framework to decouple the respective computations. Subsequently, through Neural Architecture Search, the student model develops enhanced computational capabilities for handling incomplete data compared to the teacher model. During testing, we design a novel evaluation method, testing the model's performance under different missing rate conditions without altering the model weights. We conduct experiments on three benchmark ERC datasets, and the results demonstrate that our ITEACH-Net outperforms existing methods in incomplete multimodal ERC. We believe ITEACH-Net can inspire relevant research on the intrinsic nature of emotions within conversation scenarios and pave a more robust route for incomplete learning techniques. Codes will be made available.
A Grating Based High-Frequency Motion Stimulus Paradigm for Steady-State Motion Visual Evoked Potentials
Venue: arXiv cs.HC 2024
Year: 2024

Objective: This paper proposes a novel type of stimulus in the shape of sinusoidal gratings displayed with an imperceptibly high-frequency motion. The stimulus has been designed for use in BCI (Brain Computer Interface) applications that employ visually evoked potentials (VEPs) in an effort to mitigate discomfort associated with VEPs. The stimuli set included traditional VEP stimuli, already established in the literature, allowing comparative analyses. We conducted analyses of signal distinction measures by calculating the signal-to-noise ratio and the classification performance of its evoked potentials. Methods: Fourteen participants were seated in a dimly lit room facing a display. Participants' fixation on the central stimulus was controlled by means of a desktop eye tracker. Participants attended a flicker-based steady-state VEP (SSVEP) task, a motion-based steady-state-motion VEP (SSMVEP) task, and the novel stimulus task (the imperceptible grating SSMVEP). Participants were asked to complete behavioral fatigue scale tasks. Results: A significant effect of stimulus type was observed, accompanied by insignificant differences in prediction accuracy. Partially significant task effects were obtained in fatigue scale tasks. Conclusion: The study revealed that the imperceptible grating SSMVEP stimulus successfully evoked SSMVEP responses within acceptable margins in the related cortical regions. This novel stimulus contributes to BCI research by providing an imperceptible interface, improving already established stimuli design in the SSVEP and the SSMVEP literature. Significance: The present paper provides a novel SSMVEP stimulus type that may inform the future design of effective VEP-based BCI paradigms that allow seamless interaction with computer interfaces.
A Unified Generative Framework for Realistic Lidar Simulation in Autonomous Driving Systems
Venue: arXiv cs.CV 2024
Year: 2024

Simulation models for perception sensors are integral components of automotive simulators used for the virtual Verification and Validation (V\&V) of Autonomous Driving Systems (ADS). These models also serve as powerful tools for generating synthetic datasets to train deep learning-based perception models. Lidar is a widely used sensor type among the perception sensors for ADS due to its high precision in 3D environment scanning. However, developing realistic Lidar simulation models is a significant technical challenge. In particular, unrealistic models can result in a large gap between the synthesised and real-world point clouds, limiting their effectiveness in ADS applications. Recently, deep generative models have emerged as promising solutions to synthesise realistic sensory data. However, for Lidar simulation, deep generative models have been primarily hybridised with conventional algorithms, leaving unified generative approaches largely unexplored in the literature. Motivated by this research gap, we propose a unified generative framework to enhance Lidar simulation fidelity. Our proposed framework projects Lidar point clouds into depth-reflectance images via a lossless transformation, and employs our novel Controllable Lidar point cloud Generative model, CoLiGen, to translate the images. We extensively evaluate our CoLiGen model, comparing it with the state-of-the-art image-to-image translation models using various metrics to assess the realness, faithfulness, and performance of a downstream perception model. Our results show that CoLiGen exhibits superior performance across most metrics. The dataset and source code for this research are available at https://github.com/hamedhaghighi/CoLiGen.git.
Lower Bounds for Set-Multilinear Branching Programs
Venue: arXiv cs.CC 2024
Year: 2024

In this paper, we prove super-polynomial lower bounds for the model of \emph{sum of ordered set-multilinear algebraic branching programs}, each with a possibly different ordering ($\sum \mathsf{smABP}$). Specifically, we give an explicit $nd$-variate polynomial of degree $d$ such that any $\sum \mathsf{smABP}$ computing it must have size $n^{ω(1)}$ for $d$ as low as $ω(\log n)$. Notably, this constitutes the first such lower bound in the low degree regime. Moreover, for $d = \poly(n)$, we demonstrate an exponential lower bound. This result generalizes the seminal work of Nisan (STOC, 1991), which proved an exponential lower bound for a single ordered set-multilinear ABP. The significance of our lower bounds is underscored by the recent work of Bhargav, Dwivedi, and Saxena (to appear in TAMC, 2024), which showed that super-polynomial lower bounds against a sum of ordered set-multilinear branching programs -- for a polynomial of sufficiently low degree -- would imply super-polynomial lower bounds against general ABPs, thereby resolving Valiant's longstanding conjecture that the permanent polynomial can not be computed efficiently by ABPs. More precisely, their work shows that if one could obtain such lower bounds when the degree is bounded by $O(\log n/ \log \log n)$, then it would imply super-polynomial lower bounds against general ABPs. Our results strengthen the works of Arvind \& Raja (Chic. J. Theor. Comput. Sci., 2016) and Bhargav, Dwivedi \& Saxena (to appear in TAMC, 2024), as well as the works of Ramya \& Rao (Theor. Comput. Sci., 2020) and Ghoshal \& Rao (International Computer Science Symposium in Russia, 2021), each of which established lower bounds for related or restricted versions of this model. They also strongly answer a question from the former two, which asked to prove super-polynomial lower bounds for general $\sum \mathsf{smABP}$.
On two-dimensional minimal linear codes over the rings $\mathbb{Z}_{p^n}$
Venue: arXiv cs.IT 2024
Year: 2024

In this paper we study two dimensional minimal linear code over the ring $\mathbb{Z}_{p^n}$(where $p$ is prime). We show that if the generator matrix $G$ of the two dimensional linear code $M$ contains $p^n+p^{n-1}$ column vector of the following type {\scriptsize{$u_{l_1}\begin{pmatrix} 1\\ 0 \end{pmatrix}$, $u_{l_2}\begin{pmatrix} 0\\1 \end{pmatrix}$, $u_{l_3}\begin{pmatrix} 1\\u_1 \end{pmatrix}$, $u_{l_4}\begin{pmatrix} 1\\u_2 \end{pmatrix}$,...,$u_{l_{p^n-p^{n-1}+2}} \begin{pmatrix} 1\\u_{p^n-p^{n-1}} \end{pmatrix}$, $u_{l_{p^n-p^{n-1}+3}}\begin{pmatrix} d_1 \\ 1 \end{pmatrix}$, $u_{l_{p^n-p^{n-1}+4}}\begin{pmatrix} d_2\\ 1 \end{pmatrix}$,..., $u_{l_{p^n+1}}\begin{pmatrix} d_{p^{n-1}-1}\\1 \end{pmatrix}$, $u_{l_{p^n+2}}\begin{pmatrix} 1\\d_1 \end{pmatrix}$, $u_{l_{p^n+3}}\begin{pmatrix} 1\\d_2 \end{pmatrix}$,...,$u_{l_{p^n+p^{n-1}}}\begin{pmatrix} 1 \\d_{p^{n-1}-1} \end{pmatrix}$}}, where $u_i$ and $d_j$ are distinct units and zero divisors respectively in the ring $\mathbb{Z}_{p^n}$ for $1\leq i \leq p^n+p^{n-1}$, $1\leq j \leq p^{n-1}-1$ and additionally, denote $u_{l_i}$ as units in $\mathbb{Z}_{p^n}$, then the module generated by $G$ is a minimal linear code. Also we show that if any one column vector of the above types are not present entirely in $G$, then the generated module is not a minimal linear code.
Dual-scale Enhanced and Cross-generative Consistency Learning for Semi-supervised Medical Image Segmentation
Venue: arXiv cs.CV 2024
Year: 2024

Medical image segmentation plays a crucial role in computer-aided diagnosis. However, existing methods heavily rely on fully supervised training, which requires a large amount of labeled data with time-consuming pixel-wise annotations. Moreover, accurately segmenting lesions poses challenges due to variations in shape, size, and location. To address these issues, we propose a novel Dual-scale Enhanced and Cross-generative consistency learning framework for semi-supervised medical image Segmentation (DEC-Seg). First, we propose a Cross-level Feature Aggregation (CFA) module that integrates cross-level adjacent layers to enhance the feature representation ability across different resolutions. To address scale variation, we present a scale-enhanced consistency constraint, which ensures consistency in the segmentation maps generated from the same input image at different scales. This constraint helps handle variations in lesion sizes and improves the robustness of the model. Furthermore, we propose a cross-generative consistency scheme, in which the original and perturbed images can be reconstructed using cross-segmentation maps. This consistency constraint allows us to mine effective feature representations and boost the segmentation performance. To further exploit the scale information, we propose a Dual-scale Complementary Fusion (DCF) module that integrates features from two scale-specific decoders operating at different scales to help produce more accurate segmentation maps. Extensive experimental results on multiple medical segmentation tasks (polyp, skin lesion, and brain glioma) demonstrate the effectiveness of our DEC-Seg against other state-of-the-art semi-supervised segmentation approaches. The implementation code will be released at https://github.com/taozh2017/DECSeg.