OpenRaiser/Intern-Atlas huggingface dataset:
Learning Multimodal Cues of Children's Uncertainty
Venue: arXiv cs.CL 2024
Year: 2024

Understanding uncertainty plays a critical role in achieving common ground (Clark et al.,1983). This is especially important for multimodal AI systems that collaborate with users to solve a problem or guide the user through a challenging concept. In this work, for the first time, we present a dataset annotated in collaboration with developmental and cognitive psychologists for the purpose of studying nonverbal cues of uncertainty. We then present an analysis of the data, studying different roles of uncertainty and its relationship with task difficulty and performance. Lastly, we present a multimodal machine learning model that can predict uncertainty given a real-time video clip of a participant, which we find improves upon a baseline multimodal transformer model. This work informs research on cognitive coordination between human-human and human-AI and has broad implications for gesture understanding and generation. The anonymized version of our data and code will be publicly available upon the completion of the required consent forms and data sheets.
Efficient Vision-Language Models by Summarizing Visual Tokens into Compact Registers
Venue: arXiv cs.CV 2024
Year: 2024

Recent advancements in vision-language models (VLMs) have expanded their potential for real-world applications, enabling these models to perform complex reasoning on images. In the widely used fully autoregressive transformer-based models like LLaVA, projected visual tokens are prepended to textual tokens. Oftentimes, visual tokens are significantly more than prompt tokens, resulting in increased computational overhead during both training and inference. In this paper, we propose Visual Compact Token Registers (Victor), a method that reduces the number of visual tokens by summarizing them into a smaller set of register tokens. Victor adds a few learnable register tokens after the visual tokens and summarizes the visual information into these registers using the first few layers in the language tower of VLMs. After these few layers, all visual tokens are discarded, significantly improving computational efficiency for both training and inference. Notably, our method is easy to implement and requires a small number of new trainable parameters with minimal impact on model performance. In our experiment, with merely 8 visual registers--about 1% of the original tokens--Victor shows less than a 4% accuracy drop while reducing the total training time by 43% and boosting the inference throughput by 3.3X.
Optimizing Retrieval-Augmented Generation with Elasticsearch for Enhanced Question-Answering Systems
Venue: arXiv cs.IR 2024
Year: 2024

This study aims to improve the accuracy and quality of large-scale language models (LLMs) in answering questions by integrating Elasticsearch into the Retrieval Augmented Generation (RAG) framework. The experiment uses the Stanford Question Answering Dataset (SQuAD) version 2.0 as the test dataset and compares the performance of different retrieval methods, including traditional methods based on keyword matching or semantic similarity calculation, BM25-RAG and TF-IDF- RAG, and the newly proposed ES-RAG scheme. The results show that ES-RAG not only has obvious advantages in retrieval efficiency but also performs well in key indicators such as accuracy, which is 0.51 percentage points higher than TF-IDF-RAG. In addition, Elasticsearch's powerful search capabilities and rich configuration options enable the entire question-answering system to better handle complex queries and provide more flexible and efficient responses based on the diverse needs of users. Future research directions can further explore how to optimize the interaction mechanism between Elasticsearch and LLM, such as introducing higher-level semantic understanding and context-awareness capabilities, to achieve a more intelligent and humanized question-answering experience.
Speciesism in Natural Language Processing Research
Venue: arXiv cs.CL 2024
Year: 2024

Natural Language Processing (NLP) research on AI Safety and social bias in AI has focused on safety for humans and social bias against human minorities. However, some AI ethicists have argued that the moral significance of nonhuman animals has been ignored in AI research. Therefore, the purpose of this study is to investigate whether there is speciesism, i.e., discrimination against nonhuman animals, in NLP research. First, we explain why nonhuman animals are relevant in NLP research. Next, we survey the findings of existing research on speciesism in NLP researchers, data, and models and further investigate this problem in this study. The findings of this study suggest that speciesism exists within researchers, data, and models, respectively. Specifically, our survey and experiments show that (a) among NLP researchers, even those who study social bias in AI, do not recognize speciesism or speciesist bias; (b) among NLP data, speciesist bias is inherent in the data annotated in the datasets used to evaluate NLP models; (c) OpenAI GPTs, recent NLP models, exhibit speciesist bias by default. Finally, we discuss how we can reduce speciesism in NLP research.
Formal Explanations for Neuro-Symbolic AI
Venue: arXiv cs.AI 2024
Year: 2024

Despite the practical success of Artificial Intelligence (AI), current neural AI algorithms face two significant issues. First, the decisions made by neural architectures are often prone to bias and brittleness. Second, when a chain of reasoning is required, neural systems often perform poorly. Neuro-symbolic artificial intelligence is a promising approach that tackles these (and other) weaknesses by combining the power of neural perception and symbolic reasoning. Meanwhile, the success of AI has made it critical to understand its behaviour, leading to the development of explainable artificial intelligence (XAI). While neuro-symbolic AI systems have important advantages over purely neural AI, we still need to explain their actions, which are obscured by the interactions of the neural and symbolic components. To address the issue, this paper proposes a formal approach to explaining the decisions of neuro-symbolic systems. The approach hinges on the use of formal abductive explanations and on solving the neuro-symbolic explainability problem hierarchically. Namely, it first computes a formal explanation for the symbolic component of the system, which serves to identify a subset of the individual parts of neural information that needs to be explained. This is followed by explaining only those individual neural inputs, independently of each other, which facilitates succinctness of hierarchical formal explanations and helps to increase the overall performance of the approach. Experimental results for a few complex reasoning tasks demonstrate practical efficiency of the proposed approach, in comparison to purely neural systems, from the perspective of explanation size, explanation time, training time, model sizes, and the quality of explanations reported.
Nova: An Iterative Planning and Search Approach to Enhance Novelty and Diversity of LLM Generated Ideas
Venue: arXiv cs.AI 2024
Year: 2024

Scientific innovation is pivotal for humanity, and harnessing large language models (LLMs) to generate research ideas could transform discovery. However, existing LLMs often produce simplistic and repetitive suggestions due to their limited ability in acquiring external knowledge for innovation. To address this problem, we introduce an enhanced planning and search methodology designed to boost the creative potential of LLM-based systems. Our approach involves an iterative process to purposely plan the retrieval of external knowledge, progressively enriching the idea generation with broader and deeper insights. Validation through automated and human assessments indicates that our framework substantially elevates the quality of generated ideas, particularly in novelty and diversity. The number of unique novel ideas produced by our framework is 3.4 times higher than without it. Moreover, our method outperforms the current state-of-the-art, generating at least 2.5 times more top-rated ideas based on 170 seed papers in a Swiss Tournament evaluation.
MoDification: Mixture of Depths Made Easy
Venue: arXiv cs.CL 2024
Year: 2024

Long-context efficiency has recently become a trending topic in serving large language models (LLMs). And mixture of depths (MoD) is proposed as a perfect fit to bring down both latency and memory. In this paper, however, we discover that MoD can barely transform existing LLMs without costly training over an extensive number of tokens. To enable the transformations from any LLMs to MoD ones, we showcase top-k operator in MoD should be promoted to threshold-p operator, and refinement to architecture and data should also be crafted along. All these designs form our method termed MoDification. Through a comprehensive set of experiments covering model scales from 3B to 70B, we exhibit MoDification strikes an excellent balance between efficiency and effectiveness. MoDification can achieve up to ~1.2x speedup in latency and ~1.8x reduction in memory compared to original LLMs especially in long-context applications.
Takin-ADA: Emotion Controllable Audio-Driven Animation with Canonical and Landmark Loss Optimization
Venue: arXiv cs.CV 2024
Year: 2024

Existing audio-driven facial animation methods face critical challenges, including expression leakage, ineffective subtle expression transfer, and imprecise audio-driven synchronization. We discovered that these issues stem from limitations in motion representation and the lack of fine-grained control over facial expressions. To address these problems, we present Takin-ADA, a novel two-stage approach for real-time audio-driven portrait animation. In the first stage, we introduce a specialized loss function that enhances subtle expression transfer while reducing unwanted expression leakage. The second stage utilizes an advanced audio processing technique to improve lip-sync accuracy. Our method not only generates precise lip movements but also allows flexible control over facial expressions and head motions. Takin-ADA achieves high-resolution (512x512) facial animations at up to 42 FPS on an RTX 4090 GPU, outperforming existing commercial solutions. Extensive experiments demonstrate that our model significantly surpasses previous methods in video quality, facial dynamics realism, and natural head movements, setting a new benchmark in the field of audio-driven facial animation.
TiMePReSt: Time and Memory Efficient Pipeline Parallel DNN Training with Removed Staleness
Venue: arXiv cs.DC 2024
Year: 2024

DNN training is time-consuming and requires efficient multi-accelerator parallelization, where a single training iteration is split over available accelerators. Current approaches often parallelize training using intra-batch parallelization. Combining inter-batch and intra-batch pipeline parallelism is common to further improve training throughput. In this article, we develop a system, called TiMePReSt, that combines them in a novel way which helps to better overlap computation and communication, and limits the amount of communication. The traditional pipeline-parallel training of DNNs maintains similar working principle as sequential or conventional training of DNNs by maintaining consistent weight versions in forward and backward passes of a mini-batch. Thus, it suffers from high GPU memory footprint during training. In this paper, experimental study demonstrates that compromising weight consistency doesn't decrease prediction capability of a parallelly trained DNN. Moreover, TiMePReSt overcomes GPU memory overhead and achieves zero weight staleness. State-of-the-art techniques often become costly in terms of training time. In order to address this issue, TiMePReSt introduces a variant of intra-batch parallelism that parallelizes the forward pass of each mini-batch by decomposing it into smaller micro-batches. A novel synchronization method between forward and backward passes reduces training time in TiMePReSt. The occurrence of multiple sequence problem and its relation with version difference have been observed in TiMePReSt. This paper presents a mathematical relationship between the number of micro-batches and worker machines, highlighting the variation in version difference. A mathematical expression has been developed to calculate version differences for various combinations of these two without creating diagrams for all combinations.
Scalable Field-Aligned Reparameterization for Trimmed NURBS
Venue: arXiv cs.CG 2024
Year: 2024

In engineering design, one of the most daunting problems in the design-through-analysis workflow is to deal with trimmed NURBS (Non-Uniform Rational B-Splines), which often involve topological/geometric issues and lead to inevitable gaps and overlaps in the model. Given the dominance of the trimming technology in CAD systems, reconstructing such a model as a watertight representation is highly desired. While remarkable progress has been made in recent years, especially with the advancement of isogeometric analysis (IGA), there still lack a fully automatic and scalable tool to achieve this reconstruction goal. To address this issue, we present a semi-automatic and scalable reparameterization pipeline based on a scalable and feature-aligned meshing tool, QuadriFlow [1]. On top of it, we provide support for open surfaces to deal with engineering shell structures, and perform sophisticated patch simplification to remove undesired tiny/slender patches. As a result, we obtain a watertight spline surface (multi-patch NURBS or unstructured splines) with a simple quadrilateral layout. Through several challenging models from industry applications, we demonstrate the efficacy and efficiency of the proposed pipeline as well as its integration with IGA. Our source code is publicly available on GitHub [2].
ChartifyText: Automated Chart Generation from Data-Involved Texts via LLM
Venue: arXiv cs.HC 2024
Year: 2024

Text documents with numerical values involved are widely used in various applications such as scientific research, economy, public health and journalism. However, it is difficult for readers to quickly interpret such data-involved texts and gain deep insights. To fill this research gap, this work aims to automatically generate charts to accurately convey the underlying data and ideas to readers, which is essentially a challenging task. The challenges originate from text ambiguities, intrinsic sparsity and uncertainty of data in text documents, and subjective sentiment differences. Specifically, we propose ChartifyText, a novel fully-automated approach that leverages Large Language Models (LLMs) to convert complex data-involved texts to expressive charts. It consists of two major modules: tabular data inference and expressive chart generation. The tabular data inference module employs systematic prompt engineering to guide the LLM (e.g., GPT-4) to infer table data, where data ranges, uncertainties, missing data values and corresponding subjective sentiments are explicitly considered. The expressive chart generation module augments standard charts with intuitive visual encodings and concise texts to accurately convey the underlying data and insights. We extensively evaluate the effectiveness of ChartifyText on real-world data-involved text documents through case studies, in-depth interviews with three visualization experts, and a carefully-designed user study with 15 participants. The results demonstrate the usefulness and effectiveness of ChartifyText in helping readers efficiently and effectively make sense of data-involved texts.
Optimizing over FP/EDF Execution Times: Known Results and Open Problems
Venue: arXiv cs.OS 2024
Year: 2024

In many use cases the execution time of tasks is unknown and can be chosen by the designer to increase or decrease the application features depending on the availability of processing capacity. If the application has real-time constraints, such as deadlines, then the necessary and sufficient schedulability test must allow the execution times to be left unspecified. By doing so, the designer can then perform optimization of the execution times by picking the schedulable values that minimize any given cost. In this paper, we review existing results on the formulation of both the Fixed Priority and Earliest Deadline First exact schedulability constraints. The reviewed formulations are expressed by a combination of linear constraints, which enables then optimization routines.
Enhancing Cryptocurrency Market Forecasting: Advanced Machine Learning Techniques and Industrial Engineering Contributions
Venue: arXiv cs.LG 2024
Year: 2024

Cryptocurrencies, as decentralized digital assets, have experienced rapid growth and adoption, with over 23,000 cryptocurrencies and a market capitalization nearing \$1.1 trillion (about \$3,400 per person in the US) as of 2023. This dynamic market presents significant opportunities and risks, highlighting the need for accurate price prediction models to manage volatility. This chapter comprehensively reviews machine learning (ML) techniques applied to cryptocurrency price prediction from 2014 to 2024. We explore various ML algorithms, including linear models, tree-based approaches, and advanced deep learning architectures such as transformers and large language models. Additionally, we examine the role of sentiment analysis in capturing market sentiment from textual data like social media posts and news articles to anticipate price fluctuations. With expertise in optimizing complex systems and processes, industrial engineers are pivotal in enhancing these models. They contribute by applying principles of process optimization, efficiency, and risk mitigation to improve computational performance and data management. This chapter highlights the evolving landscape of cryptocurrency price prediction, the integration of emerging technologies, and the significant role of industrial engineers in refining predictive models. By addressing current limitations and exploring future research directions, this chapter aims to advance the development of more accurate and robust prediction systems, supporting better-informed investment decisions and more stable market behavior.
SignAttention: On the Interpretability of Transformer Models for Sign Language Translation
Venue: arXiv cs.CL 2024
Year: 2024

This paper presents the first comprehensive interpretability analysis of a Transformer-based Sign Language Translation (SLT) model, focusing on the translation from video-based Greek Sign Language to glosses and text. Leveraging the Greek Sign Language Dataset, we examine the attention mechanisms within the model to understand how it processes and aligns visual input with sequential glosses. Our analysis reveals that the model pays attention to clusters of frames rather than individual ones, with a diagonal alignment pattern emerging between poses and glosses, which becomes less distinct as the number of glosses increases. We also explore the relative contributions of cross-attention and self-attention at each decoding step, finding that the model initially relies on video frames but shifts its focus to previously predicted tokens as the translation progresses. This work contributes to a deeper understanding of SLT models, paving the way for the development of more transparent and reliable translation systems essential for real-world applications.
Blockchain-Based Trust and Transparency in Airline Reservation Systems using Microservices Architecture
Venue: arXiv cs.SE 2024
Year: 2024

This research gives a detailed analysis of the application of blockchain technology to the airline reservation systems in order to bolster trust, transparency, and operational efficiency by overcoming several challenges including customer control and data integrity issues. The study investigates the major components of blockchain technology such as decentralised databases, permanent records of transactions and transactional clauses executed via codes of programs and their impacts on automated systems and real-time tracking of audits. The results show a 30% decrease in booking variations together with greater data synchronization as a result of consensus processes and resistant data formations. The approach to the implementation of a blockchain technology for the purpose of this paper includes many APIs for the automatic multi-faceted record-keeping system including the smart contract execution and controllable end-users approach. Smart contracts organized the processes improving the cycle times by 40% on the average while guaranteeing no breach of agreements. In addition to this, the architecture of the system has no single point failure with over 98% reliability while measures taken to improve security have led to 85% of the customers expressing trust in the services provided. In summation, the results suggest that reservations in the airline sector stand a chance of being redefined with blockchain through savoring the benefits of a single source of truth while attempting to resolve this intrinsic problem of overcomplexity. Although the system improves the experience of customers and the level of operational transparency, issues concerning scalability and regulatory adherence. This research is also a stepping stone for further studies that are intended to address these challenges and more applicable to the airline industry.
Assessing the Impact of AR-Assisted Warnings on Roadway Workers' Stress Under Different Workload Conditions
Venue: arXiv cs.HC 2024
Year: 2024

Recent data from the Federal Highway Administration highlights an alarming increase in fatalities and injuries in roadway work zones, emphasizing the need for enhanced worker safety measures. This study addresses this concern by evaluating stress levels among roadway workers equipped with AR-assisted multi-sensory warning technology during varying work intensities. The research leverages a high-fidelity Virtual Reality environment to simulate realistic work scenarios, enabling safe evaluation of high-risk situations. Unlike previous studies focusing on external factors, this research investigates the internal physiological impact on workers. Utilizing wearable sensors, the study collected physiological data, including photoplethysmography (PPG), electrodermal activity (EDA), and skin temperature (ST), to assess stress levels continuously and non-invasively. Our findings from 18 participants reveal significant differences between light- and medium-intensity activities in heart rate variability metrics. These metrics commonly used to assess autonomic nervous system function and stress levels, included mean heart rate, NN50, pNN50, and HF-HRV. By examining the relationship between AR-enabled warnings, work intensity, and stress levels, the study contributes to enhancing worker safety and well-being. The proposed methodology offers potential for active stress monitoring in the field, contributing to enhanced safety practices and worker productivity in construction sites. By providing real-time physiological data, this approach enables informed stress management and more effective hazard warning systems in roadway work zones. This research bridges a gap in understanding the physiological impacts of AR-assisted warnings on roadway workers. The insights gained from this study can inform future safety interventions and guide the development of more effective warning systems.