OpenRaiser/Intern-Atlas huggingface dataset:
Generative Technology for Human Emotion Recognition: A Scope Review
Venue: arXiv cs.LG 2024
Year: 2024

Affective computing stands at the forefront of artificial intelligence (AI), seeking to imbue machines with the ability to comprehend and respond to human emotions. Central to this field is emotion recognition, which endeavors to identify and interpret human emotional states from different modalities, such as speech, facial images, text, and physiological signals. In recent years, important progress has been made in generative models, including Autoencoder, Generative Adversarial Network, Diffusion Model, and Large Language Model. These models, with their powerful data generation capabilities, emerge as pivotal tools in advancing emotion recognition. However, up to now, there remains a paucity of systematic efforts that review generative technology for emotion recognition. This survey aims to bridge the gaps in the existing literature by conducting a comprehensive analysis of over 320 research papers until June 2024. Specifically, this survey will firstly introduce the mathematical principles of different generative models and the commonly used datasets. Subsequently, through a taxonomy, it will provide an in-depth analysis of how generative techniques address emotion recognition based on different modalities in several aspects, including data augmentation, feature extraction, semi-supervised learning, cross-domain, etc. Finally, the review will outline future research directions, emphasizing the potential of generative models to advance the field of emotion recognition and enhance the emotional intelligence of AI systems.
ConPR: Ongoing Construction Site Dataset for Place Recognition
Venue: arXiv cs.RO 2024
Year: 2024

Place recognition, an essential challenge in computer vision and robotics, involves identifying previously visited locations. Despite algorithmic progress, challenges related to appearance change persist, with existing datasets often focusing on seasonal and weather variations but overlooking terrain changes. Understanding terrain alterations becomes critical for effective place recognition, given the aging infrastructure and ongoing city repairs. For real-world applicability, the comprehensive evaluation of algorithms must consider spatial dynamics. To address existing limitations, we present a novel multi-session place recognition dataset acquired from an active construction site. Our dataset captures ongoing construction progress through multiple data collections, facilitating evaluation in dynamic environments. It includes camera images, LiDAR point cloud data, and IMU data, enabling visual and LiDAR-based place recognition techniques, and supporting sensor fusion. Additionally, we provide ground truth information for range-based place recognition evaluation. Our dataset aims to advance place recognition algorithms in challenging and dynamic settings. Our dataset is available at https://github.com/dongjae0107/ConPR.
Neural Probabilistic Logic Learning for Knowledge Graph Reasoning
Venue: arXiv cs.AI 2024
Year: 2024

Knowledge graph (KG) reasoning is a task that aims to predict unknown facts based on known factual samples. Reasoning methods can be divided into two categories: rule-based methods and KG-embedding based methods. The former possesses precise reasoning capabilities but finds it challenging to reason efficiently over large-scale knowledge graphs. While gaining the ability to reason over large-scale knowledge graphs, the latter sacrifices reasoning accuracy. This paper aims to design a reasoning framework called Neural Probabilistic Logic Learning(NPLL) that achieves accurate reasoning on knowledge graphs. Our approach introduces a scoring module that effectively enhances the expressive power of embedding networks, striking a balance between model simplicity and reasoning capabilities. We improve the interpretability of the model by incorporating a Markov Logic Network based on variational inference. We empirically evaluate our approach on several benchmark datasets, and the experimental results validate that our method substantially enhances the accuracy and quality of the reasoning results.
Decoupled Access-Execute enabled DVFS for tinyML deployments on STM32 microcontrollers
Venue: arXiv cs.AR 2024
Year: 2024

Over the last years the rapid growth Machine Learning (ML) inference applications deployed on the Edge is rapidly increasing. Recent Internet of Things (IoT) devices and microcontrollers (MCUs), become more and more mainstream in everyday activities. In this work we focus on the family of STM32 MCUs. We propose a novel methodology for CNN deployment on the STM32 family, focusing on power optimization through effective clocking exploration and configuration and decoupled access-execute convolution kernel execution. Our approach is enhanced with optimization of the power consumption through Dynamic Voltage and Frequency Scaling (DVFS) under various latency constraints, composing an NP-complete optimization problem. We compare our approach against the state-of-the-art TinyEngine inference engine, as well as TinyEngine coupled with power-saving modes of the STM32 MCUs, indicating that we can achieve up to 25.2% less energy consumption for varying QoS levels.
Energy-Efficient Probabilistic Semantic Communication over Space-Air-Ground Integrated Networks
Venue: arXiv cs.IT 2024
Year: 2024

Space-air-ground integrated networks (SAGINs) are emerging as a pivotal element in the evolution of future wireless networks. Despite their potential, the joint design of communication and computation within SAGINs remains a formidable challenge. In this paper, the problem of energy efficiency in SAGIN-enabled probabilistic semantic communication (PSC) system is investigated. In the considered model, a satellite needs to transmit data to multiple ground terminals (GTs) via an unmanned aerial vehicle (UAV) acting as a relay. During transmission, the satellite and the UAV can use PSC technique to compress the transmitting data, while the GTs can automatically recover the missing information. The PSC is underpinned by shared probability graphs that serve as a common knowledge base among the transceivers, allowing for resource-saving communication at the expense of increased computation resource. Through analysis, the computation overhead function in PSC is a piecewise function with respect to the semantic compression ratio. Therefore, it is important to make a balance between communication and computation to achieve optimal energy efficiency. The joint communication and computation problem is formulated as an optimization problem aiming to minimize the total communication and computation energy consumption of the network under latency, power, computation capacity, bandwidth, semantic compression ratio, and UAV location constraints. To solve this non-convex non-smooth problem, we propose an iterative algorithm where the closed-form solutions for computation capacity allocation and UAV altitude are obtained at each iteration. Numerical results show the effectiveness of the proposed algorithm.
Assessing Consensus of Developers' Views on Code Readability
Venue: arXiv cs.SE 2024
Year: 2024

The rapid rise of Large Language Models (LLMs) has changed software development, with tools like Copilot, JetBrains AI Assistant, and others boosting developers' productivity. However, developers now spend more time reviewing code than writing it, highlighting the importance of Code Readability for code comprehension. Our previous research found that existing Code Readability models were inaccurate in representing developers' notions and revealed a low consensus among developers, highlighting a need for further investigations in this field. Building on this, we surveyed 10 Java developers with similar coding experience to evaluate their consensus on Code Readability assessments and related aspects. We found significant agreement among developers on Code Readability evaluations and identified specific code aspects strongly correlated with Code Readability. Overall, our study sheds light on Code Readability within LLM contexts, offering insights into how these models can align with developers' perceptions of Code Readability, enhancing software development in the AI era.
M5 -- A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Tasks
Venue: arXiv cs.CL 2024
Year: 2024

Since the release of ChatGPT, the field of Natural Language Processing has experienced rapid advancements, particularly in Large Language Models (LLMs) and their multimodal counterparts, Large Multimodal Models (LMMs). Despite their impressive capabilities, LLMs often exhibit significant performance disparities across different languages and cultural contexts, as demonstrated by various text-only benchmarks. However, current research lacks such benchmarks for multimodal visio-linguistic settings. This work fills this gap by introducing M5, the first comprehensive benchmark designed to evaluate LMMs on diverse vision-language tasks within a multilingual and multicultural context. M5 includes eight datasets covering five tasks and $41$ languages, with a focus on underrepresented languages and culturally diverse images. Furthermore, we introduce two novel datasets, M5-VGR and M5-VLOD, including a new Visio-Linguistic Outlier Detection task, in which all evaluated open-source models fail to significantly surpass the random baseline. Through extensive evaluation and analyses, we highlight substantial task-agnostic performance disparities between high- and low-resource languages. Moreover, we show that larger models do not necessarily outperform smaller ones in a multilingual setting.
Finetuning End-to-End Models for Estonian Conversational Spoken Language Translation
Venue: arXiv cs.CL 2024
Year: 2024

This paper investigates the finetuning of end-to-end models for bidirectional Estonian-English and Estonian-Russian conversational speech-to-text translation. Due to the limited availability of speech translation data for Estonian, we created additional training data by web scraping and synthesizing data from speech recognition datasets using machine translation. We evaluated three publicly available end-to-end models: Whisper, OWSM 3.1, and SeamlessM4T. Our results indicate that fine-tuning with synthetic data enhances translation accuracy by a large margin, with SeamlessM4T matching or surpassing cascaded speech translation systems that use state-of-the-art speech recognition and machine translation models.
Use of Mobile Devices in the Classroom to Increase Motivation and Participation of Engineering University Students
Venue: arXiv cs.CY 2024
Year: 2024

The aim of this study was to see whether student participation increased when mobile devices were used in the classroom. We measured the amount of student participative actions when the Socrative tool was used and when it was not used. Our experiment involved a total of 192 students, corresponding to 4 different subjects of Computer Engineering at the Universitat de les Illes Balears, during 2012/2013 and 2013/2014 courses. An independent paired t-test was performed on the measurements. The analysis results show that student participation increases with the use of mobile devices for theory classes and students are willing to participate in class activities and share their own results.
StreamLTS: Query-based Temporal-Spatial LiDAR Fusion for Cooperative Object Detection
Venue: arXiv cs.CV 2024
Year: 2024

Cooperative perception via communication among intelligent traffic agents has great potential to improve the safety of autonomous driving. However, limited communication bandwidth, localization errors and asynchronized capturing time of sensor data, all introduce difficulties to the data fusion of different agents. To some extend, previous works have attempted to reduce the shared data size, mitigate the spatial feature misalignment caused by localization errors and communication delay. However, none of them have considered the asynchronized sensor ticking times, which can lead to dynamic object misplacement of more than one meter during data fusion. In this work, we propose Time-Aligned COoperative Object Detection (TA-COOD), for which we adapt widely used dataset OPV2V and DairV2X with considering asynchronous LiDAR sensor ticking times and build an efficient fully sparse framework with modeling the temporal information of individual objects with query-based techniques. The experiment results confirmed the superior efficiency of our fully sparse framework compared to the state-of-the-art dense models. More importantly, they show that the point-wise observation timestamps of the dynamic objects are crucial for accurate modeling the object temporal context and the predictability of their time-related locations. The official code is available at \url{https://github.com/YuanYunshuang/CoSense3D}.
10 Years of Fair Representations: Challenges and Opportunities
Venue: arXiv cs.LG 2024
Year: 2024

Fair Representation Learning (FRL) is a broad set of techniques, mostly based on neural networks, that seeks to learn new representations of data in which sensitive or undesired information has been removed. Methodologically, FRL was pioneered by Richard Zemel et al. about ten years ago. The basic concepts, objectives and evaluation strategies for FRL methodologies remain unchanged to this day. In this paper, we look back at the first ten years of FRL by i) revisiting its theoretical standing in light of recent work in deep learning theory that shows the hardness of removing information in neural network representations and ii) presenting the results of a massive experimentation (225.000 model fits and 110.000 AutoML fits) we conducted with the objective of improving on the common evaluation scenario for FRL. More specifically, we use automated machine learning (AutoML) to adversarially "mine" sensitive information from supposedly fair representations. Our theoretical and experimental analysis suggests that deterministic, unquantized FRL methodologies have serious issues in removing sensitive information, which is especially troubling as they might seem "fair" at first glance.
Computational Thinking through Design Patterns in Video Games
Venue: arXiv cs.MM 2024
Year: 2024

Prior research has explored potential applications of video games in programming education to elicit computational thinking skills. However, existing approaches are often either too general, not taking into account the diversity of genres and mechanisms between video games, or too narrow, selecting tools that were specifically designed for educational purposes. In this paper we propose a more fundamental approach, defining beneficial connections between individual design patterns present in video games and computational thinking skills. We argue that video games have the capacity to elicit these skills and even to potentially train them. This could be an effective method to solidify a conceptual base which would make programming education more effective.
FedSat: A Statistical Aggregation Approach for Class Imbalanced Clients in Federated Learning
Venue: arXiv cs.LG 2024
Year: 2024

Federated learning (FL) has emerged as a promising paradigm for privacy-preserving distributed machine learning, but faces challenges with heterogeneous data distributions across clients. This paper presents FedSat, a novel FL approach specifically designed to simultaneously handle three forms of data heterogeneity, namely label skewness, missing classes, and quantity skewness, by proposing a prediction-sensitive loss function and a prioritized-class based weighted aggregation scheme. While the prediction-sensitive loss function enhances model performance on minority classes, the prioritized-class based weighted aggregation scheme ensures client contributions are weighted based on both statistical significance and performance on critical classes. Extensive experiments across diverse data-heterogeneity settings demonstrate that FedSat significantly outperforms state-of-the-art baselines, with an average improvement of 1.8% over the second-best method and 19.87% over the weakest-performing baseline. The approach also demonstrates faster convergence compared to existing methods. These results highlight FedSat's effectiveness in addressing the challenges of heterogeneous federated learning and its potential for real-world applications.
DSMix: Distortion-Induced Sensitivity Map Based Pre-training for No-Reference Image Quality Assessment
Venue: arXiv cs.CV 2024
Year: 2024

Image quality assessment (IQA) has long been a fundamental challenge in image understanding. In recent years, deep learning-based IQA methods have shown promising performance. However, the lack of large amounts of labeled data in the IQA field has hindered further advancements in these methods. This paper introduces DSMix, a novel data augmentation technique specifically designed for IQA tasks, aiming to overcome this limitation. DSMix leverages the distortion-induced sensitivity map (DSM) of an image as prior knowledge. It applies cut and mix operations to diverse categories of synthetic distorted images, assigning confidence scores to class labels based on the aforementioned prior knowledge. In the pre-training phase using DSMix-augmented data, knowledge distillation is employed to enhance the model's ability to extract semantic features. Experimental results on both synthetic and authentic IQA datasets demonstrate the significant predictive and generalization performance achieved by DSMix, without requiring fine-tuning of the full model. Code is available at \url{https://github.com/I2-Multimedia-Lab/DSMix}.
Hybrid NOMA Assisted OFDMA Uplink Transmission
Venue: arXiv cs.IT 2024
Year: 2024

Hybrid non-orthogonal multiple access (NOMA) has recently received significant research interest due to its ability to efficiently use resources from different domains and also its compatibility with various orthogonal multiple access (OMA) based legacy networks. Unlike existing studies on hybrid NOMA that focus on combining NOMA with time-division multiple access (TDMA), this work considers hybrid NOMA assisted orthogonal frequency-division multiple access (OFDMA). In particular, the impact of a unique feature of hybrid NOMA assisted OFDMA, i.e., the availability of users' dynamic channel state information, on the system performance is analyzed from the following two perspectives. From the optimization perspective, analytical results are developed which show that with hybrid NOMA assisted OFDMA, the pure OMA mode is rarely adopted by the users, and the pure NOMA mode could be optimal for minimizing the users' energy consumption, which differs from the hybrid TDMA case. From the statistical perspective, two new performance metrics, namely the power outage probability and the power diversity gain, are developed to quantitatively measure the performance gain of hybrid NOMA over OMA. The developed analytical results also demonstrate the ability of hybrid NOMA to meet the users' diverse energy profiles.
DiCTI: Diffusion-based Clothing Designer via Text-guided Input
Venue: arXiv cs.CV 2024
Year: 2024

Recent developments in deep generative models have opened up a wide range of opportunities for image synthesis, leading to significant changes in various creative fields, including the fashion industry. While numerous methods have been proposed to benefit buyers, particularly in virtual try-on applications, there has been relatively less focus on facilitating fast prototyping for designers and customers seeking to order new designs. To address this gap, we introduce DiCTI (Diffusion-based Clothing Designer via Text-guided Input), a straightforward yet highly effective approach that allows designers to quickly visualize fashion-related ideas using text inputs only. Given an image of a person and a description of the desired garments as input, DiCTI automatically generates multiple high-resolution, photorealistic images that capture the expressed semantics. By leveraging a powerful diffusion-based inpainting model conditioned on text inputs, DiCTI is able to synthesize convincing, high-quality images with varied clothing designs that viably follow the provided text descriptions, while being able to process very diverse and challenging inputs, captured in completely unconstrained settings. We evaluate DiCTI in comprehensive experiments on two different datasets (VITON-HD and Fashionpedia) and in comparison to the state-of-the-art (SoTa). The results of our experiments show that DiCTI convincingly outperforms the SoTA competitor in generating higher quality images with more elaborate garments and superior text prompt adherence, both according to standard quantitative evaluation measures and human ratings, generated as part of a user study.
On the Connectivity of the Flip Graph of Plane Spanning Paths
Venue: arXiv cs.CG 2024
Year: 2024

Flip graphs of non-crossing configurations in the plane are widely studied objects, e.g., flip graph of triangulations, spanning trees, Hamiltonian cycles, and perfect matchings. Typically, it is an easy exercise to prove connectivity of a flip graph. In stark contrast, the connectivity of the flip graph of plane spanning paths on point sets in general position has been an open problem for more than 16 years. In order to provide new insights, we investigate certain induced subgraphs. Firstly, we provide tight bounds on the diameter and the radius of the flip graph of spanning paths on points in convex position with one fixed endpoint. Secondly, we show that so-called suffix-independent paths induce a connected subgraph. Consequently, to answer the open problem affirmatively, it suffices to show that each path can be flipped to some suffix-independent path. Lastly, we investigate paths where one endpoint is fixed and provide tools to flip to suffix-independent paths. We show that these tools are strong enough to show connectivity of the flip graph of plane spanning paths on point sets with at most two convex layers.
TwinLab: a framework for data-efficient training of non-intrusive reduced-order models for digital twins
Venue: arXiv cs.CE 2024
Year: 2024

Purpose: Simulation-based digital twins represent an effort to provide high-accuracy real-time insights into operational physical processes. However, the computation time of many multi-physical simulation models is far from real-time. It might even exceed sensible time frames to produce sufficient data for training data-driven reduced-order models. This study presents TwinLab, a framework for data-efficient, yet accurate training of neural-ODE type reduced-order models with only two data sets. Design/methodology/approach: Correlations between test errors of reduced-order models and distinct features of corresponding training data are investigated. Having found the single best data sets for training, a second data set is sought with the help of similarity and error measures to enrich the training process effectively. Findings: Adding a suitable second training data set in the training process reduces the test error by up to 49% compared to the best base reduced-order model trained only with one data set. Such a second training data set should at least yield a good reduced-order model on its own and exhibit higher levels of dissimilarity to the base training data set regarding the respective excitation signal. Moreover, the base reduced-order model should have elevated test errors on the second data set. The relative error of the time series ranges from 0.18% to 0.49%. Prediction speed-ups of up to a factor of 36,000 are observed. Originality: The proposed computational framework facilitates the automated, data-efficient extraction of non-intrusive reduced-order models for digital twins from existing simulation models, independent of the simulation software.