OpenRaiser/Intern-Atlas huggingface dataset:
Perceptual Analysis of Groups of Virtual Humans Animated using Interactive Platforms
Venue: arXiv cs.HC 2024
Year: 2024

Virtual humans (VH) have been used in Computer Graphics (CG) for many years, and perception studies have been applied to understand how people perceive them. Some studies have already examined how realism impacts the comfort of viewers. In some cases, the user's comfort is related to human identification. For example, people from a specific group may look positively at others from the same group. Gender is one of those characteristics that have in-group advantages. For example, in terms of VHs, studies have shown that female humans are more likely to recognize emotions in female VHs than in male VHs. However, there are many other variables that can impact the user perception. To aid this discussion, we conducted a study on how people perceive comfort and realism in relation to interactive VHs with different genders and expressing negative, neutral, or positive emotions in groups. We created a virtual environment for participants to interact with groups of VHs, which are interactive and should evolve in real-time, using a popular game engine. To animate the characters, we opted for cartoon figures that are animated by tracking the facial expressions of actors, using available game engine platforms to conduct the driven animation. Our results indicate that the emotion of the VH group impacts both comfort and realism perception, even by using simple cartoon characters in an interactive environment. Furthermore, the findings suggest that individuals reported feeling better with a positive emotion compared to a negative emotion, and that negative emotion recognition is impacted by the gender of the VHs group. Additionally, although we used simple characters, the results are consistent with the perception obtained when analysing realistic the state-of-the-art virtual humans, which positive emotions tend to be more correctly recognized than negative ones.
Performance Evaluation of Tokenizers in Large Language Models for the Assamese Language
Venue: arXiv cs.CL 2024
Year: 2024

Training of a tokenizer plays an important role in the performance of deep learning models. This research aims to understand the performance of tokenizers in five state-of-the-art (SOTA) large language models (LLMs) in the Assamese language of India. The research is important to understand the multi-lingual support for a low-resourced language such as Assamese. Our research reveals that the tokenizer of SUTRA from Two AI performs the best with an average Normalized Sequence Length (NSL) value of 0.45, closely followed by the tokenizer of GPT-4o from Open AI with an average NSL value of 0.54, followed by Gemma 2, Meta Llama 3.1, and Mistral Large Instruct 2407 with an average NSL value of 0.82, 1.4, and 1.48 respectively.
Beyond Scalar Reward Model: Learning Generative Judge from Preference Data
Venue: arXiv cs.CL 2024
Year: 2024

Learning from preference feedback is a common practice for aligning large language models~(LLMs) with human value. Conventionally, preference data is learned and encoded into a scalar reward model that connects a value head with an LLM to produce a scalar score as preference or reward. However, scalar models lack interpretability and are known to be susceptible to biases in datasets. This paper investigates leveraging the generation capability of LLMs to address both limitations in one shot. Specifically, we prompt the pre-trained LLM to generate positive and negative judgments, both supported with rationales in natural language form. The self-generated contrastive judgment pairs are used to train the generative judge with Direct Preference Optimization (DPO). This proposal of training the generative Judge using self-generated Contrastive judgments (Con-J) ensures natural interpretability due to the generated rationales together with the judgments, as well as high robustness against bias without the need for an additional reward head. Experimental results show that the performance of Con-J is comparable to the scalar reward model trained on the same collection of preference data, and demonstrate its superior interpretability and robustness in encoding human preferences.
Enhancing Retrieval in QA Systems with Derived Feature Association
Venue: arXiv cs.CL 2024
Year: 2024

Retrieval augmented generation (RAG) has become the standard in long context question answering (QA) systems. However, typical implementations of RAG rely on a rather naive retrieval mechanism, in which texts whose embeddings are most similar to that of the query are deemed most relevant. This has consequences in subjective QA tasks, where the most relevant text may not directly contain the answer. In this work, we propose a novel extension to RAG systems, which we call Retrieval from AI Derived Documents (RAIDD). RAIDD leverages the full power of the LLM in the retrieval process by deriving inferred features, such as summaries and example questions, from the documents at ingest. We demonstrate that this approach significantly improves the performance of RAG systems on long-context QA tasks.
Parameter Estimation of Long Memory Stochastic Processes with Deep Neural Networks
Venue: arXiv cs.LG 2024
Year: 2024

We present a purely deep neural network-based approach for estimating long memory parameters of time series models that incorporate the phenomenon of long-range dependence. Parameters, such as the Hurst exponent, are critical in characterizing the long-range dependence, roughness, and self-similarity of stochastic processes. The accurate and fast estimation of these parameters holds significant importance across various scientific disciplines, including finance, physics, and engineering. We harnessed efficient process generators to provide high-quality synthetic training data, enabling the training of scale-invariant 1D Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) models. Our neural models outperform conventional statistical methods, even those augmented with neural networks. The precision, speed, consistency, and robustness of our estimators are demonstrated through experiments involving fractional Brownian motion (fBm), the Autoregressive Fractionally Integrated Moving Average (ARFIMA) process, and the fractional Ornstein-Uhlenbeck (fOU) process. We believe that our work will inspire further research in the field of stochastic process modeling and parameter estimation using deep learning techniques.
AI-rays: Exploring Bias in the Gaze of AI Through a Multimodal Interactive Installation
Venue: arXiv cs.HC 2024
Year: 2024

Data surveillance has become more covert and pervasive with AI algorithms, which can result in biased social classifications. Appearance offers intuitive identity signals, but what does it mean to let AI observe and speculate on them? We introduce AI-rays, an interactive installation where AI generates speculative identities from participants' appearance which are expressed through synthesized personal items placed in participants' bags. It uses speculative X-ray visions to contrast reality with AI-generated assumptions, metaphorically highlighting AI's scrutiny and biases. AI-rays promotes discussions on modern surveillance and the future of human-machine reality through a playful, immersive experience exploring AI biases.
M2AR: A Web-based Modeling Environment for the Augmented Reality Workflow Modeling Language
Venue: arXiv cs.HC 2024
Year: 2024

This paper introduces M2AR, a new web-based, two- and three-dimensional modeling environment that enables the modeling and execution of augmented reality applications without requiring programming knowledge. The platform is based on a 3D JavaScript library and the mixed reality immersive web standard WebXR. For a first demonstration of its feasibility, the previously introduced Augmented Reality Workflow Modeling Language (ARWFML) has been successfully implemented using this environment. The usefulness of the new modeling environment is demonstrated by showing use cases of the ARWFML on M2AR.
Enhanced Digital Twin for Human-Centric and Integrated Lighting Asset Management in Public Libraries: From Corrective to Predictive Maintenance
Venue: arXiv cs.HC 2024
Year: 2024

Lighting asset management in public libraries has traditionally been reactive, focusing on corrective maintenance, addressing issues only when failures occur. Although standards now encourage preventive measures, such as incorporating a maintenance factor, the broader goal of human centric, sustainable lighting systems requires a shift toward predictive maintenance strategies. This study introduces an enhanced digital twin model designed for the proactive management of lighting assets in public libraries. By integrating descriptive, diagnostic, predictive, and prescriptive analytics, the model enables a comprehensive, multilevel view of asset health. The proposed framework supports both preventive and predictive maintenance strategies, allowing for early detection of issues and the timely resolution of potential failures. In addition to the specific application for lighting systems, the design is adaptable for other building assets, providing a scalable solution for integrated asset management in various public spaces.
Explaining the (Not So) Obvious: Simple and Fast Explanation of STAN, a Next Point of Interest Recommendation System
Venue: arXiv cs.IR 2024
Year: 2024

A lot of effort in recent years have been expended to explain machine learning systems. However, some machine learning methods are inherently explainable, and thus are not completely black box. This enables the developers to make sense of the output without a developing a complex and expensive explainability technique. Besides that, explainability should be tailored to suit the context of the problem. In a recommendation system which relies on collaborative filtering, the recommendation is based on the behaviors of similar users, therefore the explanation should tell which other users are similar to the current user. Similarly, if the recommendation system is based on sequence prediction, the explanation should also tell which input timesteps are the most influential. We demonstrate this philosophy/paradigm in STAN (Spatio-Temporal Attention Network for Next Location Recommendation), a next Point of Interest recommendation system based on collaborative filtering and sequence prediction. We also show that the explanation helps to "debug" the output.
Using Prompts to Guide Large Language Models in Imitating a Real Person's Language Style
Venue: arXiv cs.CL 2024
Year: 2024

Large language models (LLMs), such as GPT series and Llama series have demonstrated strong capabilities in natural language processing, contextual understanding, and text generation. In recent years, researchers are trying to enhance the abilities of LLMs in performing various tasks, and numerous studies have proved that well-designed prompts can significantly improve the performance of LLMs on these tasks. This study compares the language style imitation ability of three different large language models under the guidance of the same zero-shot prompt. It also involves comparing the imitation ability of the same large language model when guided by three different prompts individually. Additionally, by applying a Tree-of-Thoughts (ToT) Prompting method to Llama 3, a conversational AI with the language style of a real person was created. In this study, three evaluation methods were used to evaluate LLMs and prompts. The results show that Llama 3 performs best at imitating language styles, and that the ToT prompting method is the most effective to guide it in imitating language styles. Using a ToT framework, Llama 3 was guided to interact with users in the language style of a specific individual without altering its core parameters, thereby creating a text-based conversational AI that reflects the language style of the individual.
Detecting Machine-Generated Long-Form Content with Latent-Space Variables
Venue: arXiv cs.CL 2024
Year: 2024

The increasing capability of large language models (LLMs) to generate fluent long-form texts is presenting new challenges in distinguishing machine-generated outputs from human-written ones, which is crucial for ensuring authenticity and trustworthiness of expressions. Existing zero-shot detectors primarily focus on token-level distributions, which are vulnerable to real-world domain shifts, including different prompting and decoding strategies, and adversarial attacks. We propose a more robust method that incorporates abstract elements, such as event transitions, as key deciding factors to detect machine versus human texts by training a latent-space model on sequences of events or topics derived from human-written texts. In three different domains, machine-generated texts, which are originally inseparable from human texts on the token level, can be better distinguished with our latent-space model, leading to a 31% improvement over strong baselines such as DetectGPT. Our analysis further reveals that, unlike humans, modern LLMs like GPT-4 generate event triggers and their transitions differently, an inherent disparity that helps our method to robustly detect machine-generated texts.
The Wallpaper is Ugly: Indoor Localization using Vision and Language
Venue: arXiv cs.CV 2024
Year: 2024

We study the task of locating a user in a mapped indoor environment using natural language queries and images from the environment. Building on recent pretrained vision-language models, we learn a similarity score between text descriptions and images of locations in the environment. This score allows us to identify locations that best match the language query, estimating the user's location. Our approach is capable of localizing on environments, text, and images that were not seen during training. One model, finetuned CLIP, outperformed humans in our evaluation.
LLM-TOPLA: Efficient LLM Ensemble by Maximising Diversity
Venue: arXiv cs.CL 2024
Year: 2024

Combining large language models during training or at inference time has shown substantial performance gain over component LLMs. This paper presents LLM-TOPLA, a diversity-optimized LLM ensemble method with three unique properties: (i) We introduce the focal diversity metric to capture the diversity-performance correlation among component LLMs of an ensemble. (ii) We develop a diversity-optimized ensemble pruning algorithm to select the top-k sub-ensembles from a pool of $N$ base LLMs. Our pruning method recommends top-performing LLM subensembles of size $S$, often much smaller than $N$. (iii) We generate new output for each prompt query by utilizing a learn-to-ensemble approach, which learns to detect and resolve the output inconsistency among all component LLMs of an ensemble. Extensive evaluation on four different benchmarks shows good performance gain over the best LLM ensemble methods: (i) In constrained solution set problems, LLM-TOPLA outperforms the best-performing ensemble (Mixtral) by 2.2\% in accuracy on MMLU and the best-performing LLM ensemble (MoreAgent) on GSM8k by 2.1\%. (ii) In generative tasks, LLM-TOPLA outperforms the top-2 performers (Llama70b/Mixtral) on SearchQA by $3.9\mathrm{x}$ in F1, and on XSum by more than $38$ in ROUGE-1. Our code and dataset, which contains outputs of 8 modern LLMs on 4 benchmarks is available at https://github.com/git-disl/llm-topla
UDE-III: An Enhanced Unified Differential Evolution Algorithm for Constrained Optimization Problems
Venue: arXiv cs.NE 2024
Year: 2024

In this paper, an enhanced unified differential evolution algorithm, named UDE-III, is presented for real parameter-constrained optimization problems (COPs). The proposed UDE-III is a significantly enhanced version of the Improved UDE (i.e., IUDE or UDE-II), which secured the 1st rank in the CEC 2018 competition on real parameter COPs. To design UDE-III, we extensively targeted the weaknesses of UDE-II. Specifically, UDE-III uses three trial vector generation strategies - DE/rand/1, DE/current-to-rand/1, and DE/current-to-pbest/1. It is based on a dual population approach, and for each generation, it divides the current population into two sub-populations. In the top sub-population, it employs all three trial vector generation strategies on each target vector. On the other hand, the bottom sub-population employs strategy adaptation and one trial vector generation strategy is implemented on each target vector. The mutation operation in UDE-III is based on ranking-based mutation. Further, it employs the parameter adaptation principle of SHADE. The constraint handling principle in UDE-III is based on a combination of the feasibility rule and epsilon-constraint handling technique. We observed that stagnation is a major weakness of UDE-II. To overcome this weakness, we took inspiration from the best-discarded vector selection (BDVS) strategy proposed in the literature and integrated a novel strategy in UDE-III to address stagnation. Additionally, unlike UDE-II, UDE-III considers the size of the two sub-populations to be a design element. Moreover, in comparison to UDE-II, UDE-III improves upon the strategy adaptation, ranking-based mutation, and the constraint handling technique. The proposed UDE-III algorithm is tested on the 28 benchmark 30D problems provided for the CEC 2024 competition on real parameter COPs. The experimental results demonstrate the superiority of UDE-III over UDE-II.
Enhancing the Travel Experience for People with Visual Impairments through Multimodal Interaction: NaviGPT, A Real-Time AI-Driven Mobile Navigation System
Venue: arXiv cs.HC 2024
Year: 2024

Assistive technologies for people with visual impairments (PVI) have made significant advancements, particularly with the integration of artificial intelligence (AI) and real-time sensor technologies. However, current solutions often require PVI to switch between multiple apps and tools for tasks like image recognition, navigation, and obstacle detection, which can hinder a seamless and efficient user experience. In this paper, we present NaviGPT, a high-fidelity prototype that integrates LiDAR-based obstacle detection, vibration feedback, and large language model (LLM) responses to provide a comprehensive and real-time navigation aid for PVI. Unlike existing applications such as Be My AI and Seeing AI, NaviGPT combines image recognition and contextual navigation guidance into a single system, offering continuous feedback on the user's surroundings without the need for app-switching. Meanwhile, NaviGPT compensates for the response delays of LLM by using location and sensor data, aiming to provide practical and efficient navigation support for PVI in dynamic environments.
Development of a Mouse for Individuals Without Upper Limbs Using Arduino Technology
Venue: arXiv cs.HC 2024
Year: 2024

This project focuses on the design and construction of a prototype mouse based on the Arduino platform, intended for individuals without upper limbs to use computers more effectively. The prototype comprises a microcontroller responsible for processing signals from the MPU-6050 sensor, used as a reference for cursor position, and foot-operated buttons for right and left-click functions. Its design enables cursor control through head movements, providing users with an easy and intuitive way to interact with the computer's graphical interface. Feasibility testing was conducted through experimental trials, resulting in ideal accuracy and precision. These trials indicate that the device is viable for use in individuals without upper limbs.
SyllableLM: Learning Coarse Semantic Units for Speech Language Models
Venue: arXiv cs.CL 2024
Year: 2024

Language models require tokenized inputs. However, tokenization strategies for continuous data like audio and vision are often based on simple heuristics such as fixed sized convolutions or discrete clustering, which do not necessarily align with the semantic structure of the data. For speech in particular, the high resolution of waveforms (16,000 samples/second or more) presents a significant challenge as speech-based language models have had to use several times more tokens per word than text-based language models. In this work, we introduce a controllable self-supervised technique to merge speech representations into coarser syllable-like units while still preserving semantic information. We do this by 1) extracting noisy boundaries through analyzing correlations in pretrained encoder losses and 2) iteratively improving model representations with a novel distillation technique. Our method produces controllable-rate semantic units at as low as 5Hz and 60bps and achieves SotA in syllabic segmentation and clustering. Using these coarse tokens, we successfully train SyllableLM, a Speech Language Model (SpeechLM) that matches or outperforms current SotA SpeechLMs on a range of spoken language modeling tasks. SyllableLM also achieves significant improvements in efficiency with a 30x reduction in training compute and a 4x wall-clock inference speedup.
Gamified crowd-sourcing of high-quality data for visual fine-tuning
Venue: arXiv cs.AI 2024
Year: 2024

This paper introduces Gamified Adversarial Prompting (GAP), a framework that crowd-sources high-quality data for visual instruction tuning of large multimodal models. GAP transforms the data collection process into an engaging game, incentivizing players to provide fine-grained, challenging questions and answers that target gaps in the model's knowledge. Our contributions include (1) an approach to capture question-answer pairs from humans that directly address weaknesses in a model's knowledge, (2) a method for evaluating and rewarding players that successfully incentivizes them to provide high-quality submissions, and (3) a scalable, gamified platform that succeeds in collecting this data from over 50,000 participants in just a few weeks. Our implementation of GAP has significantly improved the accuracy of a small multimodal model, namely MiniCPM-Llama3-V-2.5-8B, increasing its GPT score from 0.147 to 0.477 on our dataset, approaching the benchmark set by the much larger GPT-4V. Moreover, we demonstrate that the data generated using MiniCPM-Llama3-V-2.5-8B also enhances its performance across other benchmarks, and exhibits cross-model benefits. Specifically, the same data improves the performance of QWEN2-VL-2B and QWEN2-VL-7B on the same multiple benchmarks.
Pseudo-Deterministic Construction of Irreducible Polynomials over Finite Fields
Venue: arXiv cs.DS 2024
Year: 2024

We present a polynomial-time pseudo-deterministic algorithm for constructing irreducible polynomial of degree $d$ over finite field $\mathbb{F}_q$. A pseudo-deterministic algorithm is allowed to use randomness, but with high probability it must output a canonical irreducible polynomial. Our construction runs in time $\tilde{O}(d^4 \log^4{q})$. Our construction extends Shoup's deterministic algorithm (FOCS 1988) for the same problem, which runs in time $\tilde{O}(d^4 p^{\frac{1}{2}} \log^4{q})$ (where $p$ is the characteristic of the field $\mathbb{F}_q$). Shoup had shown a reduction from constructing irreducible polynomials to factoring polynomials over finite fields. We show that by using a fast randomized factoring algorithm, the above reduction yields an efficient pseudo-deterministic algorithm for constructing irreducible polynomials over finite fields.
Cross-Lingual Query-by-Example Spoken Term Detection: A Transformer-Based Approach
Venue: arXiv cs.LG 2024
Year: 2024

Query-by-example spoken term detection (QbE-STD) is typically constrained by transcribed data scarcity and language specificity. This paper introduces a novel, language-agnostic QbE-STD model leveraging image processing techniques and transformer architecture. By employing a pre-trained XLSR-53 network for feature extraction and a Hough transform for detection, our model effectively searches for user-defined spoken terms within any audio file. Experimental results across four languages demonstrate significant performance gains (19-54%) over a CNN-based baseline. While processing time is improved compared to DTW, accuracy remains inferior. Notably, our model offers the advantage of accurately counting query term repetitions within the target audio.