OpenRaiser/Intern-Atlas huggingface dataset:
SAM Decoding: Speculative Decoding via Suffix Automaton
Venue: arXiv cs.CL 2024
Year: 2024

Speculative decoding (SD) has been demonstrated as an effective technique for lossless LLM inference acceleration. Retrieval-based SD methods, one kind of model-free method, have yielded promising speedup, but they often rely on incomplete retrieval resources, inefficient retrieval methods, and are constrained to certain domains. This paper presents a novel retrieval-based speculative decoding method that adapts suffix automaton (SAM) for efficient and accurate draft generation by utilizing common text corpus and dynamic text sequence. Unlike existing $n$-gram matching methods, SAM-Decoding finds the exact longest suffix match, achieving an average time complexity of O(1) per generation step of SAM update and suffix retrieval. It can also integrate with existing methods, adaptively selecting a draft generation strategy based on match length to generalize to broader domains. Extensive experiments on Spec-Bench show that our method is $18\%+$ faster than other retrieval-based SD methods. Additionally, when combined with advanced EAGLE-2, it provides an additional speedup of $3.28\%$ -- $11.13\%$ across various-sized LLM backbones. Our code is available at our \href{https://github.com/hyx1999/SAM-Decoding}{repository}.
I'm Spartacus, No, I'm Spartacus: Measuring and Understanding LLM Identity Confusion
Venue: arXiv cs.CR 2024
Year: 2024

Large Language Models (LLMs) excel in diverse tasks such as text generation, data analysis, and software development, making them indispensable across domains like education, business, and creative industries. However, the rapid proliferation of LLMs (with over 560 companies developing or deploying them as of 2024) has raised concerns about their originality and trustworthiness. A notable issue, termed identity confusion, has emerged, where LLMs misrepresent their origins or identities. This study systematically examines identity confusion through three research questions: (1) How prevalent is identity confusion among LLMs? (2) Does it arise from model reuse, plagiarism, or hallucination? (3) What are the security and trust-related impacts of identity confusion? To address these, we developed an automated tool combining documentation analysis, self-identity recognition testing, and output similarity comparisons--established methods for LLM fingerprinting--and conducted a structured survey via Credamo to assess its impact on user trust. Our analysis of 27 LLMs revealed that 25.93% exhibit identity confusion. Output similarity analysis confirmed that these issues stem from hallucinations rather than replication or reuse. Survey results further highlighted that identity confusion significantly erodes trust, particularly in critical tasks like education and professional use, with declines exceeding those caused by logical errors or inconsistencies. Users attributed these failures to design flaws, incorrect training data, and perceived plagiarism, underscoring the systemic risks posed by identity confusion to LLM reliability and trustworthiness.
HELENE: Hessian Layer-wise Clipping and Gradient Annealing for Accelerating Fine-tuning LLM with Zeroth-order Optimization
Venue: arXiv cs.LG 2024
Year: 2024

Fine-tuning large language models (LLMs) poses significant memory challenges, as the back-propagation process demands extensive resources, especially with growing model sizes. Recent work, MeZO, addresses this issue using a zeroth-order (ZO) optimization method, which reduces memory consumption by matching the usage to the inference phase. However, MeZO experiences slow convergence due to varying curvatures across model parameters. To overcome this limitation, we introduce HELENE, a novel scalable and memory-efficient optimizer that integrates annealed A-GNB gradients with a diagonal Hessian estimation and layer-wise clipping, serving as a second-order pre-conditioner. This combination allows for faster and more stable convergence. Our theoretical analysis demonstrates that HELENE improves convergence rates, particularly for models with heterogeneous layer dimensions, by reducing the dependency on the total parameter space dimension. Instead, the method scales with the largest layer dimension, making it highly suitable for modern LLM architectures. Experimental results on RoBERTa-large and OPT-1.3B across multiple tasks show that HELENE achieves up to a 20x speedup compared to MeZO, with average accuracy improvements of 1.5%. Furthermore, HELENE remains compatible with both full parameter tuning and parameter-efficient fine-tuning (PEFT), outperforming several state-of-the-art optimizers. The codes will be released after reviewing.
Poster: Reliable 3D Reconstruction for Ad-hoc Edge Implementations
Venue: arXiv cs.CV 2024
Year: 2024

Ad-hoc edge deployments to support real-time complex video processing applications such as, multi-view 3D reconstruction often suffer from spatio-temporal system disruptions that greatly impact reconstruction quality. In this poster paper, we present a novel portfolio theory-inspired edge resource management strategy to ensure reliable multi-view 3D reconstruction by accounting for possible system disruptions.
FlowScope: Enhancing Decision Making by Time Series Forecasting based on Prediction Optimization using HybridFlow Forecast Framework
Venue: arXiv cs.LG 2024
Year: 2024

Time series forecasting is crucial in several sectors, such as meteorology, retail, healthcare, and finance. Accurately forecasting future trends and patterns is crucial for strategic planning and making well-informed decisions. In this case, it is crucial to include many forecasting methodologies. The strengths of Auto-regressive Integrated Moving Average (ARIMA) for linear time series, Seasonal ARIMA models (SARIMA) for seasonal time series, Exponential Smoothing State Space Models (ETS) for handling errors and trends, and Long Short-Term Memory (LSTM) Neural Network model for complex pattern recognition have been combined to create a comprehensive framework called FlowScope. SARIMA excels in capturing seasonal variations, whereas ARIMA ensures effective handling of linear time series. ETS models excel in capturing trends and correcting errors, whereas LSTM networks excel in reflecting intricate temporal connections. By combining these methods from both machine learning and deep learning, we propose a deep-hybrid learning approach FlowScope which offers a versatile and robust platform for predicting time series data. This empowers enterprises to make informed decisions and optimize long-term strategies for maximum performance. Keywords: Time Series Forecasting, HybridFlow Forecast Framework, Deep-Hybrid Learning, Informed Decisions.
Transforming Teacher Education in Developing Countries: The Role of Generative AI in Bridging Theory and Practice
Venue: arXiv cs.CY 2024
Year: 2024

This study examines the transformative potential of Generative AI (GenAI) in teacher education within developing countries, focusing on Ghana, where challenges such as limited pedagogical modeling, performance-based assessments, and practitioner-expertise gaps hinder progress. GenAI has the capacity to address these issues by supporting content knowledge acquisition, a role that currently dominates teacher education programs. By taking on this foundational role, GenAI allows teacher educators to redirect their focus to other critical areas, including pedagogical modeling, authentic assessments, and fostering digital literacy and critical thinking. These roles are interconnected, creating a ripple effect where pre-service teachers (PSTs) are better equipped to enhance K-12 learning outcomes and align education with workforce needs. The study emphasizes that GenAI's roles are multifaceted, directly addressing resistance to change, improving resource accessibility, and supporting teacher professional development. However, it cautions against misuse, which could undermine critical thinking and creativity, essential skills nurtured through traditional teaching methods. To ensure responsible and effective integration, the study advocates a scaffolding approach to GenAI literacy. This includes educating PSTs on its supportive role, training them in ethical use and prompt engineering, and equipping them to critically assess AI-generated content for biases and validity. The study concludes by recommending empirical research to explore these roles further and develop practical steps for integrating GenAI into teacher education systems responsibly and effectively.
Comparison of Multilingual and Bilingual Models for Satirical News Detection of Arabic and English
Venue: arXiv cs.CL 2024
Year: 2024

Satirical news is real news combined with a humorous comment or exaggerated content, and it often mimics the format and style of real news. However, satirical news is often misunderstood as misinformation, especially by individuals from different cultural and social backgrounds. This research addresses the challenge of distinguishing satire from truthful news by leveraging multilingual satire detection methods in English and Arabic. We explore both zero-shot and chain-of-thought (CoT) prompting using two language models, Jais-chat(13B) and LLaMA-2-chat(7B). Our results show that CoT prompting offers a significant advantage for the Jais-chat model over the LLaMA-2-chat model. Specifically, Jais-chat achieved the best performance, with an F1-score of 80\% in English when using CoT prompting. These results highlight the importance of structured reasoning in CoT, which enhances contextual understanding and is vital for complex tasks like satire detection.
AnimateAnything: Consistent and Controllable Animation for Video Generation
Venue: arXiv cs.CV 2024
Year: 2024

We present a unified controllable video generation approach AnimateAnything that facilitates precise and consistent video manipulation across various conditions, including camera trajectories, text prompts, and user motion annotations. Specifically, we carefully design a multi-scale control feature fusion network to construct a common motion representation for different conditions. It explicitly converts all control information into frame-by-frame optical flows. Then we incorporate the optical flows as motion priors to guide final video generation. In addition, to reduce the flickering issues caused by large-scale motion, we propose a frequency-based stabilization module. It can enhance temporal coherence by ensuring the video's frequency domain consistency. Experiments demonstrate that our method outperforms the state-of-the-art approaches. For more details and videos, please refer to the webpage: https://yu-shaonian.github.io/Animate_Anything/.
NeuroNURBS: Learning Efficient Surface Representations for 3D Solids
Venue: arXiv cs.CV 2024
Year: 2024

Boundary Representation (B-Rep) is the de facto representation of 3D solids in Computer-Aided Design (CAD). B-Rep solids are defined with a set of NURBS (Non-Uniform Rational B-Splines) surfaces forming a closed volume. To represent a surface, current works often employ the UV-grid approximation, i.e., sample points uniformly on the surface. However, the UV-grid method is not efficient in surface representation and sometimes lacks precision and regularity. In this work, we propose NeuroNURBS, a representation learning method to directly encode the parameters of NURBS surfaces. Our evaluation in solid generation and segmentation tasks indicates that the NeuroNURBS performs comparably and, in some cases, superior to UV-grids, but with a significantly improved efficiency: for training the surface autoencoder, GPU consumption is reduced by 86.7%; memory requirement drops by 79.9% for storing 3D solids. Moreover, adapting BrepGen for solid generation with our NeuroNURBS improves the FID from 30.04 to 27.24, and resolves the undulating issue in generated surfaces.
The Competition Complexity of Prophet Secretary
Venue: arXiv cs.GT 2024
Year: 2024

We study the classic single-choice prophet secretary problem through a resource augmentation lens. Our goal is to bound the $(1-ε)$-competition complexity for different classes of online algorithms. This metric asks for the smallest $k$ such that the expected value of the online algorithm on $k$ copies of the original instance, is at least a $(1 - ε)$-approximation to the expected offline optimum on the original instance (without added copies). We consider four natural classes of online algorithms: single-threshold, time-based threshold, activation-based, and general algorithms. We show that for single-threshold algorithms the $(1-ε)$-competition complexity is $Θ(\ln(\frac{1}ε))$ (as in the i.i.d. case). Additionally, we demonstrate that time-based threshold and activation-based algorithms (which cover all previous approaches for obtaining competitive-ratios for the classic prophet secretary problem) yield a sub-optimal $(1-ε)$-competition complexity of $Θ\left(\frac{\ln(\frac{1}ε)}{\ln\ln(\frac{1}ε)}\right)$, which is strictly better than the class of single-threshold algorithms. Finally, we find that the $(1-ε)$-competition complexity of general adaptive algorithms is $Θ(\sqrt{\ln(\frac{1}ε)})$, which is in sharp contrast to $Θ(\ln\ln(\frac{1}ε))$ in the i.i.d. case.
SPICA: Retrieving Scenarios for Pluralistic In-Context Alignment
Venue: arXiv cs.CL 2024
Year: 2024

When different groups' values differ, one approach to model alignment is to steer models at inference time towards each group's preferences. However, techniques like in-context learning only consider similarity when drawing few-shot examples and not cross-group differences in values. We propose SPICA, a framework that accounts for group-level differences during in-context example retrieval. SPICA introduces three designs: scenario banks, group-informed retrieval metrics, and in-context alignment prompts. From an evaluation of SPICA on an alignment task collecting inputs from four demographic groups ($n = 544$), our metrics retrieve in-context examples that more closely match observed preferences, with the best prompt configuration using multiple contrastive responses to demonstrate examples. In an end-to-end evaluation ($n = 120$), we observe that SPICA is higher rated than similarity-based retrieval, with groups seeing up to a +0.16 point improvement on a 5 point scale. Additionally, gains from SPICA were more uniform, with all groups benefiting from alignment rather than only some. Finally, we find that while a group-agnostic approach can align to aggregated values, it is not most suited for divergent groups.
Multi-Modal Self-Supervised Learning for Surgical Feedback Effectiveness Assessment
Venue: arXiv cs.LG 2024
Year: 2024

During surgical training, real-time feedback from trainers to trainees is important for preventing errors and enhancing long-term skill acquisition. Accurately predicting the effectiveness of this feedback, specifically whether it leads to a change in trainee behavior, is crucial for developing methods for improving surgical training and education. However, relying on human annotations to assess feedback effectiveness is laborious and prone to biases, underscoring the need for an automated, scalable, and objective method. Creating such an automated system poses challenges, as it requires an understanding of both the verbal feedback delivered by the trainer and the visual context of the real-time surgical scene. To address this, we propose a method that integrates information from transcribed verbal feedback and corresponding surgical video to predict feedback effectiveness. Our findings show that both transcribed feedback and surgical video are individually predictive of trainee behavior changes, and their combination achieves an AUROC of 0.70+/-0.02, improving prediction accuracy by up to 6.6%. Additionally, we introduce self-supervised fine-tuning as a strategy for enhancing surgical video representation learning, which is scalable and further enhances prediction performance. Our results demonstrate the potential of multi-modal learning to advance the automated assessment of surgical feedback.
A Resilience Perspective on C-V2X Communication Networks under Imperfect CSI
Venue: arXiv cs.IT 2024
Year: 2024

Cellular vehicle-to-everything (C-V2X) networks provide a promising solution to improve road safety and traffic efficiency. One key challenge in such systems lies in meeting different quality-of-service (QoS) requirements of coexisting vehicular communication links, particularly under imperfect channel state information (CSI) conditions caused by the highly dynamic environment. In this paper, a novel analytical framework for examining the resilience of C-V2X networks in face of imperfect CSI is proposed. In this framework, the adaptation phase of the C-V2X network is studied, in which an adaptation power scheme is employed and the probability distribution function (PDF) of the imperfect CSI is estimated. Then, the resilience of C-V2X networks is studied through two principal dimensions: remediation capability and adaptation performance, both of which are defined, quantified, and analyzed for the first time. Particularly, an upper bound on the estimation's mean square error (MSE) is explicitly derived to capture the C-V2X's remediation capability, and a novel metric named hazard rate (HR) is exploited to evaluate the C-V2X's adaptation performance. Afterwards, the impact of the adaptation power scheme on the C-V2X's resilience is examined, revealing a tradeoff between the C-V2X's remediation capability and adaptation performance. Simulation results validate the framework's superiority in capturing the interplay between adaptation and remediation, as well as the effectiveness of the two proposed metrics in guiding the design of the adaptation power scheme to enhance the system's resilience.
Analyzing Pokémon and Mario Streamers' Twitch Chat with LLM-based User Embeddings
Venue: arXiv cs.CL 2024
Year: 2024

We present a novel digital humanities method for representing our Twitch chatters as user embeddings created by a large language model (LLM). We cluster these embeddings automatically using affinity propagation and further narrow this clustering down through manual analysis. We analyze the chat of one stream by each Twitch streamer: SmallAnt, DougDoug and PointCrow. Our findings suggest that each streamer has their own type of chatters, however two categories emerge for all of the streamers: supportive viewers and emoji and reaction senders. Repetitive message spammers is a shared chatter category for two of the streamers.
Beamforming Design and Multi-User Scheduling in Transmissive RIS Enabled Distributed Cooperative ISAC Networks with RSMA
Venue: arXiv cs.IT 2024
Year: 2024

In this paper, we propose a novel transmissive reconfigurable intelligent surface (TRIS) transceiver-empowered distributed cooperative integrated sensing and communication (ISAC) network to enhance coverage as well as to enhance wireless environment understanding. Based on the network requirements, the users are categorized into cooperative users (CUEs) and destination users (DUEs), and the CUEs utilize their own resources to serve the DUEs. To realize cooperation, we implement rate-splitting multiple access (RSMA) at the base station (BS), where the common stream is decoded and reencoded at the CUEs and forwarded to the DUEs, while the private stream satisfies the CUEs' own communication requirements. We construct an optimization problem with maximum minimum radar mutual information (RMI) as the objective function to optimize the BS beamforming matrix, the CUE beamforming matrices, the common stream rate vectors, and the user scheduling vectors. Due to the coupling of the optimization variables and non-convex operation, the proposed problem is a non-convex optimization problem that cannot be solved directly. To address the above challenges, we adopt a consensus alternating direction method of multipliers (ADMM) framework to decouple the optimization variables and solve it. Specifically, the problem is decoupled into multiple subproblems and solved by iterative optimization independently until overall convergence is achieved. Finally, numerical results validate the superiority of the proposed scheme in terms of improving communication sum-rate and RMI, and greatly reduce the algorithm complexity.
Timing-driven Approximate Logic Synthesis Based on Double-chase Grey Wolf Optimizer
Venue: arXiv cs.AR 2024
Year: 2024

With the shrinking technology nodes, timing optimization becomes increasingly challenging. Approximate logic synthesis (ALS) can perform local approximate changes (LACs) on circuits to optimize timing with the cost of slight inaccuracy. However, existing ALS methods that focus solely on critical path depth reduction (depth-driven methods) or area minimization (area-driven methods) are inefficient in achieving optimal timing improvement. %based on double-chase grey wolf optimizer (DCGWO). where we employ a double-chase grey wolf optimizer to explore and apply LACs, simultaneously bringing excellent critical path shortening and area reduction under error constraints. According to experiments on open-source circuits with TSMC 28nm technology, compared to the SOTA method, our framework can generate approximate circuits with greater critical path delay reduction under different error and area constraints.
From Crime to Hypercrime: Evolving Threats and Law Enforcement's New Mandate in the AI Age
Venue: arXiv cs.CY 2024
Year: 2024

The paper examines the trajectory of crime, tracing its evolution from traditional forms to digital manifestations in cybercrime, and proposes "Hypercrime" as the latest frontier. Leveraging insights from Michael McGuire's "Hypercrime: The New Geometry of Harm," the study calls for a paradigm shift in law enforcement strategies to meet the challenges posed by AI-driven hypercrime. Emphasis is placed on understanding hypercrime's complexity, developing proactive policies, and embracing technological tools to mitigate risks associated with AI misuse.
Gadgetless Lifting Beats Round Elimination: Improved Lower Bounds for Pointer Chasing
Venue: arXiv cs.CC 2024
Year: 2024

We prove an Ω(n/k+k) communication lower bound on (k-1)-round distributional complexity of the k-step pointer chasing problem under uniform input distribution, improving the Ω(n/k - k log n) lower bound due to Yehudayoff (Combinatorics Probability and Computing, 2020). Our lower bound almost matches the upper bound of O(n/k + k) communication by Nisan and Wigderson (STOC 91). As part of our approach, we put forth gadgetless lifting, a new framework that lifts lower bounds for a family of restricted protocols into lower bounds for general protocols. A key step in gadgetless lifting is choosing the appropriate definition of restricted protocols. In this paper, our definition of restricted protocols is inspired by the structure-vs-pseudorandomness decomposition by Göös, Pitassi, and Watson (FOCS 17) and Yang and Zhang (STOC 24). Previously, round-communication trade-offs were mainly obtained by round elimination and information complexity. Both methods have some barriers in some situations, and we believe gadgetless lifting could potentially address these barriers.