OpenRaiser/Intern-Atlas huggingface dataset:
TCNFormer: Temporal Convolutional Network Former for Short-Term Wind Speed Forecasting
Venue: arXiv cs.LG 2024
Year: 2024

Global environmental challenges and rising energy demands have led to extensive exploration of wind energy technologies. Accurate wind speed forecasting (WSF) is crucial for optimizing wind energy capture and ensuring system stability. However, predicting wind speed remains challenging due to its inherent randomness, fluctuation, and unpredictability. This study proposes the Temporal Convolutional Network Former (TCNFormer) for short-term (12-hour) wind speed forecasting. The TCNFormer integrates the Temporal Convolutional Network (TCN) and transformer encoder to capture the spatio-temporal features of wind speed. The transformer encoder consists of two distinct attention mechanisms: causal temporal multi-head self-attention (CT-MSA) and temporal external attention (TEA). CT-MSA ensures that the output of a step derives only from previous steps, i.e., causality. Locality is also introduced to improve efficiency. TEA explores potential relationships between different sample sequences in wind speed data. This study utilizes wind speed data from the NASA Prediction of Worldwide Energy Resources (NASA POWER) of Patenga Sea Beach, Chittagong, Bangladesh (latitude 22.2352° N, longitude 91.7914° E) over a year (six seasons). The findings indicate that the TCNFormer outperforms state-of-the-art models in prediction accuracy. The proposed TCNFormer presents a promising method for spatio-temporal WSF and may achieve desirable performance in real-world applications of wind power systems.
When Fire Attacks: How does Concrete Stand up to Heat ?
Venue: arXiv cs.CE 2024
Year: 2024

Fire is a process that generates both light and heat, posing a significant threat to life and infrastructure. Buildings and structures are neither inherently susceptible to fire nor completely fire-resistant; their vulnerability largely depends on the specific causes of the fire, which can stem from natural events or human-induced hazards. High temperatures in structures can lead to severe health risks for those directly affected, discomfort due to smoke, and compromised safety if the structure fails to meet safety standards. Elevated temperatures can also cause significant structural damage, becoming the primary cause of casualties, economic losses, and material damage. This study aims to investigate the thermal and structural behavior of concrete beams when exposed to extreme fire conditions. It examines the effects of different temperatures on plain and reinforced concrete (PCC and RCC, respectively) using finite element method (FEM) simulations. Additionally, the study explores the performance of various concrete grades under severe conditions. The analysis reveals that higher-grade concrete exhibits greater displacement, crack width, stress, and strain but has lower thermal conductivity compared to lower-grade concrete. These elevated temperatures can induce severe stresses in the concrete, leading to expansion, spalling, and the potential failure of the structure. Reinforced concrete, on the other hand, shows lower stress concentrations and minimal strain up to 250°C. These findings contribute to the existing knowledge and support the development of improved fire safety regulations and performance-based design methodologies.
Evaluating and Comparing Crowd Simulations: Perspectives from a Crowd Authoring Tool
Venue: arXiv cs.GR 2024
Year: 2024

Crowd simulation is a research area widely used in diverse fields, including gaming and security, assessing virtual agent movements through metrics like time to reach their goals, speed, trajectories, and densities. This is relevant for security applications, for instance, as different crowd configurations can determine the time people spend in environments trying to evacuate them. In this work, we extend WebCrowds, an authoring tool for crowd simulation, to allow users to build scenarios and evaluate them through a set of metrics. The aim is to provide a quantitative metric that can, based on simulation data, select the best crowd configuration in a certain environment. We conduct experiments to validate our proposed metric in multiple crowd simulation scenarios and perform a comparison with another metric found in the literature. The results show that experts in the domain of crowd scenarios agree with our proposed quantitative metric.
Neural Network Equalizers and Successive Interference Cancellation for Bandlimited Channels with a Nonlinearity
Venue: arXiv cs.IT 2024
Year: 2024

Neural networks (NNs) inspired by the forward-backward algorithm (FBA) are used as equalizers for bandlimited channels with a memoryless nonlinearity. The NN-equalizers are combined with successive interference cancellation (SIC) to approach the information rates of joint detection and decoding (JDD) with considerably less complexity than JDD and other existing equalizers. Simulations for short-haul optical fiber links with square-law detection illustrate the gains.
Started Off Local, Now We're in the Cloud: Forensic Examination of the Amazon Echo Show 15 Smart Display
Venue: arXiv cs.CR 2024
Year: 2024

Amazon Echo is one of the most popular product families of smart speakers and displays. Considering their growing presence in modern households as well as the digital traces associated with residents' interactions with these devices, analyses of Echo products are likely to become more common for forensic investigators at "smart home" crime scenes. With this in mind, we present the first forensic examination of the Echo Show 15, Amazon's largest smart display running on Fire OS and the first Echo device with Visual ID, a face recognition feature. We unveil a non-invasive method for accessing the unencrypted file system of the Echo Show 15 based on an undocumented pinout for the eMMC interface which we discovered on the main logic board. On the device, we identify various local usage artifacts, such as searched products, streamed movies, visited websites, metadata of photos and videos as well as logged events of Visual ID about movements and users detected by the built-in camera. Furthermore, we utilize an insecurely stored token on the Echo Show 15 to obtain access to remote user artifacts in Amazon's cloud, including Alexa voice requests, calendars, contacts, conversations, photos, and videos. In this regard, we also identify new Amazon APIs through network traffic analysis of two companion apps, namely Alexa and Photos. Overall, in terms of practical relevance, our findings demonstrate a non-destructive way of data acquisition for Echo Show 15 devices as well as how to lift the scope of forensic traces from local artifacts on the device to remote artifacts stored in the cloud.
A Survey on Evaluation of Multimodal Large Language Models
Venue: arXiv cs.CV 2024
Year: 2024

Multimodal Large Language Models (MLLMs) mimic human perception and reasoning system by integrating powerful Large Language Models (LLMs) with various modality encoders (e.g., vision, audio), positioning LLMs as the "brain" and various modality encoders as sensory organs. This framework endows MLLMs with human-like capabilities, and suggests a potential pathway towards achieving artificial general intelligence (AGI). With the emergence of all-round MLLMs like GPT-4V and Gemini, a multitude of evaluation methods have been developed to assess their capabilities across different dimensions. This paper presents a systematic and comprehensive review of MLLM evaluation methods, covering the following key aspects: (1) the background of MLLMs and their evaluation; (2) "what to evaluate" that reviews and categorizes existing MLLM evaluation tasks based on the capabilities assessed, including general multimodal recognition, perception, reasoning and trustworthiness, and domain-specific applications such as socioeconomic, natural sciences and engineering, medical usage, AI agent, remote sensing, video and audio processing, 3D point cloud analysis, and others; (3) "where to evaluate" that summarizes MLLM evaluation benchmarks into general and specific benchmarks; (4) "how to evaluate" that reviews and illustrates MLLM evaluation steps and metrics; Our overarching goal is to provide valuable insights for researchers in the field of MLLM evaluation, thereby facilitating the development of more capable and reliable MLLMs. We emphasize that evaluation should be regarded as a critical discipline, essential for advancing the field of MLLMs.
LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Models
Venue: arXiv cs.AI 2024
Year: 2024

Large Language Models (LLMs) have demonstrated notable capabilities across various tasks, showcasing complex problem-solving abilities. Understanding and executing complex rules, along with multi-step planning, are fundamental to logical reasoning and critical for practical LLM agents and decision-making systems. However, evaluating LLMs as effective rule-based executors and planners remains underexplored. In this paper, we introduce LogicGame, a novel benchmark designed to evaluate the comprehensive rule understanding, execution, and planning capabilities of LLMs. Unlike traditional benchmarks, LogicGame provides diverse games that contain a series of rules with an initial state, requiring models to comprehend and apply predefined regulations to solve problems. We create simulated scenarios in which models execute or plan operations to achieve specific outcomes. These game scenarios are specifically designed to distinguish logical reasoning from mere knowledge by relying exclusively on predefined rules. This separation allows for a pure assessment of rule-based reasoning capabilities. The evaluation considers not only final outcomes but also intermediate steps, providing a comprehensive assessment of model performance. Moreover, these intermediate steps are deterministic and can be automatically verified. LogicGame defines game scenarios with varying difficulty levels, from simple rule applications to complex reasoning chains, in order to offer a precise evaluation of model performance on rule understanding and multi-step execution. Utilizing LogicGame, we test various LLMs and identify notable shortcomings in their rule-based logical reasoning abilities.
An Empirical Study of API Misuses of Data-Centric Libraries
Venue: arXiv cs.SE 2024
Year: 2024

Developers rely on third-party library Application Programming Interfaces (APIs) when developing software. However, libraries typically come with assumptions and API usage constraints, whose violation results in API misuse. API misuses may result in crashes or incorrect behavior. Even though API misuse is a well-studied area, a recent study of API misuse of deep learning libraries showed that the nature of these misuses and their symptoms are different from misuses of traditional libraries, and as a result highlighted potential shortcomings of current misuse detection tools. We speculate that these observations may not be limited to deep learning API misuses but may stem from the data-centric nature of these APIs. Data-centric libraries often deal with diverse data structures, intricate processing workflows, and a multitude of parameters, which can make them inherently more challenging to use correctly. Therefore, understanding the potential misuses of these libraries is important to avoid unexpected application behavior. To this end, this paper contributes an empirical study of API misuses of five data-centric libraries that cover areas such as data processing, numerical computation, machine learning, and visualization. We identify misuses of these libraries by analyzing data from both Stack Overflow and GitHub. Our results show that many of the characteristics of API misuses observed for deep learning libraries extend to misuses of the data-centric library APIs we study. We also find that developers tend to misuse APIs from data-centric libraries, regardless of whether the API directive appears in the documentation. Overall, our work exposes the challenges of API misuse in data-centric libraries, rather than only focusing on deep learning libraries. Our collected misuses and their characterization lay groundwork for future research to help reduce misuses of these libraries.
LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillation
Venue: arXiv cs.CV 2024
Year: 2024

We introduce LLaVA-MoD, a novel framework designed to enable the efficient training of small-scale Multimodal Language Models (s-MLLM) by distilling knowledge from large-scale MLLM (l-MLLM). Our approach tackles two fundamental challenges in MLLM distillation. First, we optimize the network structure of s-MLLM by integrating a sparse Mixture of Experts (MoE) architecture into the language model, striking a balance between computational efficiency and model expressiveness. Second, we propose a progressive knowledge transfer strategy to ensure comprehensive knowledge migration. This strategy begins with mimic distillation, where we minimize the Kullback-Leibler (KL) divergence between output distributions to enable the student model to emulate the teacher network's understanding. Following this, we introduce preference distillation via Direct Preference Optimization (DPO), where the key lies in treating l-MLLM as the reference model. During this phase, the s-MLLM's ability to discriminate between superior and inferior examples is significantly enhanced beyond l-MLLM, leading to a better student that surpasses its teacher, particularly in hallucination benchmarks. Extensive experiments demonstrate that LLaVA-MoD outperforms existing models across various multimodal benchmarks while maintaining a minimal number of activated parameters and low computational costs. Remarkably, LLaVA-MoD, with only 2B activated parameters, surpasses Qwen-VL-Chat-7B by an average of 8.8% across benchmarks, using merely 0.3% of the training data and 23% trainable parameters. These results underscore LLaVA-MoD's ability to effectively distill comprehensive knowledge from its teacher model, paving the way for the development of more efficient MLLMs. The code will be available on: https://github.com/shufangxun/LLaVA-MoD.
Exploring the potential of AI in nurturing learner empathy, prosocial values and environmental stewardship
Venue: arXiv cs.HC 2024
Year: 2024

With Artificial Intelligence (AI) becoming a powerful tool for education (Zawacki-Richter et al., 2019), this chapter describes the concept of combining generative and traditional AI, citizen-science physiological, neuroergonomic wearables and environmental sensors into activities for learners to understand their own well-being and emotional states better with a view to developing empathy and environmental stewardship. Alongside bespoke and affordable wearables (DIY EEG headsets and biometric wristbands), interpretable AI and data science are used for learners to explore how the environment affects them physiologically and mentally in authentic environments. For example, relationships between environmental changes (e.g. poorer air quality) and their well-being (e.g. cognitive functioning) can be discovered. This is particularly crucial, as relevant knowledge can influence the way people treat the environment, as suggested by the disciplines of environmental neuroscience and environmental psychology (Doell et al., 2023). Yet, according to Palme and Salvati, there have been relatively few studies on the relationships between microclimates and human health and emotions (Palme and Salvati, 2021). As anthropogenic environmental pollution is becoming a prevalent problem, our research also aims to leverage on generative AI to introduce hypothetical scenarios of the environment as emotionally strong stimuli of relevance to the learners. This would provoke an emotional response for them to learn about their own physiological and neurological responses (using neuro-physiological data). Ultimately, we hope to establish a bidirectional understanding of how the environment affects humans physiologically and mentally; after which, to gain insights as to how AI can be used to effectively foster empathy, pro-environmental attitudes and stewardship.
Leveraging Open Knowledge for Advancing Task Expertise in Large Language Models
Venue: arXiv cs.CV 2024
Year: 2024

The cultivation of expertise for large language models (LLMs) to solve tasks of specific areas often requires special-purpose tuning with calibrated behaviors on the expected stable outputs. To avoid huge cost brought by manual preparation of instruction datasets and training resources up to hundreds of hours, the exploitation of open knowledge including a wealth of low rank adaptation (LoRA) models and instruction datasets serves as a good starting point. However, existing methods on model and data selection focus on the performance of general-purpose capabilities while neglecting the knowledge gap exposed in domain-specific deployment. In the present study, we propose to bridge such gap by introducing few human-annotated samples (i.e., K-shot) for advancing task expertise of LLMs with open knowledge. Specifically, we develop an efficient and scalable pipeline to cost-efficiently produce task experts where K-shot data intervene in selecting the most promising expert candidates and the task-relevant instructions. A mixture-of-expert (MoE) system is built to make the best use of individual-yet-complementary knowledge between multiple experts. We unveil the two keys to the success of a MoE system, 1) the abidance by K-shot, and 2) the insistence on diversity. For the former, we ensure that models that truly possess problem-solving abilities on K-shot are selected rather than those blind guessers. Besides, during data selection, instructions that share task-relevant contexts with K-shot are prioritized. For the latter, we highlight the diversity of constituting experts and that of the fine-tuning instructions throughout the model and data selection process. Extensive experimental results confirm the superiority of our approach over existing methods on utilization of open knowledge across various tasks. Our codes will be available at https://github.com/Yaphabates/Rocket.
In-Context Imitation Learning via Next-Token Prediction
Venue: arXiv cs.RO 2024
Year: 2024

We explore how to enhance next-token prediction models to perform in-context imitation learning on a real robot, where the robot executes new tasks by interpreting contextual information provided during the input phase, without updating its underlying policy parameters. We propose In-Context Robot Transformer (ICRT), a causal transformer that performs autoregressive prediction on sensorimotor trajectories without relying on any linguistic data or reward function. This formulation enables flexible and training-free execution of new tasks at test time, achieved by prompting the model with sensorimotor trajectories of the new task composing of image observations, actions and states tuples, collected through human teleoperation. Experiments with a Franka Emika robot demonstrate that the ICRT can adapt to new tasks specified by prompts, even in environment configurations that differ from both the prompt and the training data. In a multitask environment setup, ICRT significantly outperforms current state-of-the-art next-token prediction models in robotics on generalizing to unseen tasks. Code, checkpoints and data are available on https://icrt.dev/
Artificial Neural Network and Deep Learning: Fundamentals and Theory
Venue: arXiv cs.LG 2024
Year: 2024

"Artificial Neural Network and Deep Learning: Fundamentals and Theory" offers a comprehensive exploration of the foundational principles and advanced methodologies in neural networks and deep learning. This book begins with essential concepts in descriptive statistics and probability theory, laying a solid groundwork for understanding data and probability distributions. As the reader progresses, they are introduced to matrix calculus and gradient optimization, crucial for training and fine-tuning neural networks. The book delves into multilayer feed-forward neural networks, explaining their architecture, training processes, and the backpropagation algorithm. Key challenges in neural network optimization, such as activation function saturation, vanishing and exploding gradients, and weight initialization, are thoroughly discussed. The text covers various learning rate schedules and adaptive algorithms, providing strategies to optimize the training process. Techniques for generalization and hyperparameter tuning, including Bayesian optimization and Gaussian processes, are also presented to enhance model performance and prevent overfitting. Advanced activation functions are explored in detail, categorized into sigmoid-based, ReLU-based, ELU-based, miscellaneous, non-standard, and combined types. Each activation function is examined for its properties and applications, offering readers a deep understanding of their impact on neural network behavior. The final chapter introduces complex-valued neural networks, discussing complex numbers, functions, and visualizations, as well as complex calculus and backpropagation algorithms. This book equips readers with the knowledge and skills necessary to design, and optimize advanced neural network models, contributing to the ongoing advancements in artificial intelligence.
Ranking evaluation metrics from a group-theoretic perspective
Venue: arXiv cs.IR 2024
Year: 2024

Confronted with the challenge of identifying the most suitable metric to validate the merits of newly proposed models, the decision-making process is anything but straightforward. Given that comparing rankings introduces its own set of formidable challenges and the likely absence of a universal metric applicable to all scenarios, the scenario does not get any better. Furthermore, metrics designed for specific contexts, such as for Recommender Systems, sometimes extend to other domains without a comprehensive grasp of their underlying mechanisms, resulting in unforeseen outcomes and potential misuses. Complicating matters further, distinct metrics may emphasize different aspects of rankings, frequently leading to seemingly contradictory comparisons of model results and hindering the trustworthiness of evaluations. We unveil these aspects in the domain of ranking evaluation metrics. Firstly, we show instances resulting in inconsistent evaluations, sources of potential mistrust in commonly used metrics; by quantifying the frequency of such disagreements, we prove that these are common in rankings. Afterward, we conceptualize rankings using the mathematical formalism of symmetric groups detaching from possible domains where the metrics have been created; through this approach, we can rigorously and formally establish essential mathematical properties for ranking evaluation metrics, essential for a deeper comprehension of the source of inconsistent evaluations. We conclude with a discussion, connecting our theoretical analysis to the practical applications, highlighting which properties are important in each domain where rankings are commonly evaluated. In conclusion, our analysis sheds light on ranking evaluation metrics, highlighting that inconsistent evaluations should not be seen as a source of mistrust but as the need to carefully choose how to evaluate our models in the future.