OpenRaiser/Intern-Atlas huggingface dataset:
AI-Tutoring in Software Engineering Education
Venue: arXiv cs.SE 2024
Year: 2024

With the rapid advancement of artificial intelligence (AI) in various domains, the education sector is set for transformation. The potential of AI-driven tools in enhancing the learning experience, especially in programming, is immense. However, the scientific evaluation of Large Language Models (LLMs) used in Automated Programming Assessment Systems (APASs) as an AI-Tutor remains largely unexplored. Therefore, there is a need to understand how students interact with such AI-Tutors and to analyze their experiences. In this paper, we conducted an exploratory case study by integrating the GPT-3.5-Turbo model as an AI-Tutor within the APAS Artemis. Through a combination of empirical data collection and an exploratory survey, we identified different user types based on their interaction patterns with the AI-Tutor. Additionally, the findings highlight advantages, such as timely feedback and scalability. However, challenges like generic responses and students' concerns about a learning progress inhibition when using the AI-Tutor were also evident. This research adds to the discourse on AI's role in education.
Spatial Summation of Localized Pressure for Haptic Sensory Prostheses
Venue: arXiv cs.HC 2024
Year: 2024

A host of medical conditions, including amputations, diabetes, stroke, and genetic disease, result in loss of touch sensation. Because most types of sensory loss have no pharmacological treatment or rehabilitative therapy, we propose a haptic sensory prosthesis that provides substitutive feedback. The wrist and forearm are compelling locations for feedback due to available skin area and not occluding the hands, but have reduced mechanoreceptor density compared to the fingertips. Focusing on localized pressure as the feedback modality, we hypothesize that we can improve on prior devices by invoking a wider range of stimulus intensity using multiple points of pressure to evoke spatial summation, which is the cumulative perceptual experience from multiple points of stimuli. We conducted a preliminary perceptual test to investigate this idea and found that just noticeable difference is reduced with two points of pressure compared to one, motivating future work using spatial summation in sensory prostheses.
MaiNLP at SemEval-2024 Task 1: Analyzing Source Language Selection in Cross-Lingual Textual Relatedness
Venue: arXiv cs.CL 2024
Year: 2024

This paper presents our system developed for the SemEval-2024 Task 1: Semantic Textual Relatedness (STR), on Track C: Cross-lingual. The task aims to detect semantic relatedness of two sentences in a given target language without access to direct supervision (i.e. zero-shot cross-lingual transfer). To this end, we focus on different source language selection strategies on two different pre-trained languages models: XLM-R and Furina. We experiment with 1) single-source transfer and select source languages based on typological similarity, 2) augmenting English training data with the two nearest-neighbor source languages, and 3) multi-source transfer where we compare selecting on all training languages against languages from the same family. We further study machine translation-based data augmentation and the impact of script differences. Our submission achieved the first place in the C8 (Kinyarwanda) test set.
Leveraging Swarm Intelligence to Drive Autonomously: A Particle Swarm Optimization based Approach to Motion Planning
Venue: arXiv cs.RO 2024
Year: 2024

Motion planning is an essential part of autonomous mobile platforms. A good pipeline should be modular enough to handle different vehicles, environments, and perception modules. The planning process has to cope with all the different modalities and has to have a modular and flexible design. But most importantly, it has to be safe and robust. In this paper, we want to present our motion planning pipeline with particle swarm optimization (PSO) at its core. This solution is independent of the vehicle type and has a clear and simple-to-implement interface for perception modules. Moreover, the approach stands out for being easily adaptable to new scenarios. Parallel calculation allows for fast planning cycles. Following the principles of PSO, the trajectory planer first generates a swarm of initial trajectories that are optimized afterward. We present the underlying control space and inner workings. Finally, the application to real-world automated driving is shown in the evaluation with a deeper look at the modeling of the cost function. The approach is used in our automated shuttles that have already driven more than 3.500 km safely and entirely autonomously in sub-urban everyday traffic.
Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models
Venue: arXiv cs.CL 2024
Year: 2024

Kullback-Leiber divergence has been widely used in Knowledge Distillation (KD) to compress Large Language Models (LLMs). Contrary to prior assertions that reverse Kullback-Leibler (RKL) divergence is mode-seeking and thus preferable over the mean-seeking forward Kullback-Leibler (FKL) divergence, this study empirically and theoretically demonstrates that neither mode-seeking nor mean-seeking properties manifest in KD for LLMs. Instead, RKL and FKL are found to share the same optimization objective and both converge after a sufficient number of epochs. However, due to practical constraints, LLMs are seldom trained for such an extensive number of epochs. Meanwhile, we further find that RKL focuses on the tail part of the distributions, while FKL focuses on the head part at the beginning epochs. Consequently, we propose a simple yet effective Adaptive Kullback-Leiber (AKL) divergence method, which adaptively allocates weights to combine FKL and RKL. Metric-based and GPT-4-based evaluations demonstrate that the proposed AKL outperforms the baselines across various tasks and improves the diversity and quality of generated responses. Codes are available at \href{https://github.com/wutaiqiang/LLM_KD_AKL}{github}.
Adversarial Attacks and Dimensionality in Text Classifiers
Venue: arXiv cs.LG 2024
Year: 2024

Adversarial attacks on machine learning algorithms have been a key deterrent to the adoption of AI in many real-world use cases. They significantly undermine the ability of high-performance neural networks by forcing misclassifications. These attacks introduce minute and structured perturbations or alterations in the test samples, imperceptible to human annotators in general, but trained neural networks and other models are sensitive to it. Historically, adversarial attacks have been first identified and studied in the domain of image processing. In this paper, we study adversarial examples in the field of natural language processing, specifically text classification tasks. We investigate the reasons for adversarial vulnerability, particularly in relation to the inherent dimensionality of the model. Our key finding is that there is a very strong correlation between the embedding dimensionality of the adversarial samples and their effectiveness on models tuned with input samples with same embedding dimension. We utilize this sensitivity to design an adversarial defense mechanism. We use ensemble models of varying inherent dimensionality to thwart the attacks. This is tested on multiple datasets for its efficacy in providing robustness. We also study the problem of measuring adversarial perturbation using different distance metrics. For all of the aforementioned studies, we have run tests on multiple models with varying dimensionality and used a word-vector level adversarial attack to substantiate the findings.
Scalable Model Editing via Customized Expert Networks
Venue: arXiv cs.CL 2024
Year: 2024

Addressing the issues of hallucinations and outdated knowledge in large language models is critical for their reliable application. Model Editing presents a promising avenue for mitigating these challenges in a cost-effective manner. However, existing methods often suffer from unsatisfactory generalization and unintended effects on non-edited samples. To overcome these limitations, we introduce a novel approach: Scalable Model Editing via Customized Expert Networks (SCEN), which is a two-stage continuous training paradigm. Specifically, in the first stage, we train lightweight expert networks individually for each piece of knowledge that needs to be updated. Subsequently, we train a corresponding indexing neuron for each expert to control the activation state of that expert. We conducted a series of experiments on the ZsRE and Hallucination benchmarks by tuning the advanced open-source LLM, Llama2, achieving state-of-the-art results compared to current mainstream methods. Our code is available at https://github.com/TAL-auroraX/SCEN.
Optimizing Peak Age of Information in MEC Systems: Computing Preemption and Non-preemption
Venue: arXiv cs.IT 2024
Year: 2024

The freshness of information in real-time monitoring systems has received increasing attention, with Age of Information (AoI) emerging as a novel metric for measuring information freshness. In many applications, update packets need to be computed before being delivered to a destination. Mobile edge computing (MEC) is a promising approach for efficiently accomplishing the computing process, where the transmission process and computation process are coupled, jointly affecting freshness. In this paper, we aim to minimize the average peak AoI (PAoI) in an MEC system. We consider the generate-at-will source model and study when to generate a new update in two edge server setups: 1) computing preemption, where the packet in the computing process will be preempted by the newly arrived one, and 2) non-preemption, where the newly arrived packet will wait in the queue until the current one completes computing. We prove that the fixed threshold policy is optimal in a non-preemptive system for arbitrary transmission time and computation time distributions. In a preemptive system, we show that the transmission-aware threshold policy is optimal when the computing time follows an exponential distribution. Our numerical simulation results not only validate the theoretical findings but also demonstrate that: 1) in our problem, preemptive systems are not always superior to non-preemptive systems, even with exponential distribution, and 2) as the ratio of the mean transmission time to the mean computation time increases, the optimal threshold increases in preemptive systems but decreases in non-preemptive systems.
PSCodec: A Series of High-Fidelity Low-bitrate Neural Speech Codecs Leveraging Prompt Encoders
Venue: arXiv cs.SD 2024
Year: 2024

Neural speech codecs have recently emerged as a focal point in the fields of speech compression and generation. Despite this progress, achieving high-quality speech reconstruction under low-bitrate scenarios remains a significant challenge. In this paper, we propose PSCodec, a series of neural speech codecs based on prompt encoders, comprising PSCodec-Base, PSCodec-DRL-ICT, and PSCodec-CasAN, which are capable of delivering high-performance speech reconstruction with low bandwidths. Specifically, we first introduce PSCodec-Base, which leverages a pretrained speaker verification model-based prompt encoder (VPP-Enc) and a learnable Mel-spectrogram-based prompt encoder (MelP-Enc) to effectively disentangle and integrate voiceprint and Mel-related features in utterances. To further enhance feature utilization efficiency, we propose PSCodec-DRL-ICT, incorporating a structural similarity (SSIM) based disentangled representation loss (DRL) and an incremental continuous training (ICT) strategy. While PSCodec-DRL-ICT demonstrates impressive performance, its reliance on extensive hyperparameter tuning and multi-stage training makes it somewhat labor-intensive. To circumvent these limitations, we propose PSCodec-CasAN, utilizing an advanced cascaded attention network (CasAN) to enhance representational capacity of the entire system. Extensive experiments show that our proposed PSCodec-Base, PSCodec-DRL-ICT, and PSCodec-CasAN all significantly outperform several state-of-the-art neural codecs, exhibiting substantial improvements in both speech reconstruction quality and speaker similarity under low-bitrate conditions.
Building test batteries based on analysing random number generator tests within the framework of algorithmic information theory
Venue: arXiv cs.IT 2024
Year: 2024

The problem of testing random number generators is considered and it is shown that an approach based on algorithmic information theory allows us to compare the power of different tests in some cases where the available methods of mathematical statistics do not distinguish between the tests. In particular, it is shown that tests based on data compression methods using dictionaries should be included in the test batteries.
Learning Sequence Attractors in Recurrent Networks with Hidden Neurons
Venue: arXiv cs.NE 2024
Year: 2024

The brain is targeted for processing temporal sequence information. It remains largely unclear how the brain learns to store and retrieve sequence memories. Here, we study how recurrent networks of binary neurons learn sequence attractors to store predefined pattern sequences and retrieve them robustly. We show that to store arbitrary pattern sequences, it is necessary for the network to include hidden neurons even though their role in displaying sequence memories is indirect. We develop a local learning algorithm to learn sequence attractors in the networks with hidden neurons. The algorithm is proven to converge and lead to sequence attractors. We demonstrate that the network model can store and retrieve sequences robustly on synthetic and real-world datasets. We hope that this study provides new insights in understanding sequence memory and temporal information processing in the brain.
Wideband Beamforming for Near-Field Communications with Circular Arrays
Venue: arXiv cs.IT 2024
Year: 2024

The beamforming performance of the uniform circular array (UCA) in near-field wideband communication systems is investigated. Compared to uniform linear array (ULA), UCA exhibits uniform effective array aperture in all directions, thus enabling more users to benefit from near-field communications. In this paper, the unique beam squint effect in near-field wideband UCA systems is comprehensively analyzed in both the distance and angular domains. It is rigorously demonstrated that the beam focal point only exists at a specific frequency in wideband UCA systems, resulting in significant beamforming loss. To alleviate this unique beam squint effect, the true-time delay (TTD)-based beamforming architecture is exploited. In particular, two wideband beamforming optimization approaches leveraging TTD units are proposed. 1) Analytical approach: In this approach, the phase shifters (PSs) and the time delay of TTD units are designed based on the analytical formula for beamforming gain. Following this design, the minimum number of TTD units required to achieve a predetermined beamforming gain is quantified. 2) Joint-optimization approach: In this method, the PSs and the TTD units are jointly optimized under practical maximum delay constraints to approximate the optimal unconstrained analog beamformer. Specifically, an efficient alternating optimization algorithm is proposed, where the PSs and the TTD units are alternately updated using either the closed-form solution or the low-complexity linear search approach. Extensive numerical results demonstrate that 1) the proposed beamforming schemes effectively mitigate the beam squint effect, and 2) the joint-optimization approach outperforms the analytical approach in terms of array gain and achievable spectral efficiency.
A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approaches
Venue: arXiv cs.RO 2024
Year: 2024

Task and Motion Planning (TAMP) integrates high-level task planning and low-level motion planning to equip robots with the autonomy to effectively reason over long-horizon, dynamic tasks. Optimization-based TAMP focuses on hybrid optimization approaches that define goal conditions via objective functions and are capable of handling open-ended goals, robotic dynamics, and physical interaction between the robot and the environment. Therefore, optimization-based TAMP is particularly suited to solve highly complex, contact-rich locomotion and manipulation problems. This survey provides a comprehensive review on optimization-based TAMP, covering (i) planning domain representations, including action description languages and temporal logic, (ii) individual solution strategies for components of TAMP, including AI planning and trajectory optimization (TO), and (iii) the dynamic interplay between logic-based task planning and model-based TO. A particular focus of this survey is to highlight the algorithm structures to efficiently solve TAMP, especially hierarchical and distributed approaches. Additionally, the survey emphasizes the synergy between the classical methods and contemporary learning-based innovations such as large language models. Furthermore, the future research directions for TAMP is discussed in this survey, highlighting both algorithmic and application-specific challenges.
An Error-Bounded Lossy Compression Method with Bit-Adaptive Quantization for Particle Data
Venue: arXiv cs.IT 2024
Year: 2024

This paper presents error-bounded lossy compression tailored for particle datasets from diverse scientific applications in cosmology, fluid dynamics, and fusion energy sciences. As today's high-performance computing capabilities advance, these datasets often reach trillions of points, posing significant visualization, analysis, and storage challenges. While error-bounded lossy compression makes it possible to represent floating-point values with strict pointwise accuracy guarantees, the lack of correlations in particle data's storage ordering often limits the compression ratio. Inspired by quantization-encoding schemes in SZ lossy compressors, we dynamically determine the number of bits to encode particles of the dataset to increase the compression ratio. Specifically, we utilize a k-d tree to partition particles into subregions and generate ``bit boxes'' centered at particles for each subregion to encode their positions. These bit boxes ensure error control while reducing the bit count used for compression. We comprehensively evaluate our method against state-of-the-art compressors on cosmology, fluid dynamics, and fusion plasma datasets.
Retrieving Examples from Memory for Retrieval Augmented Neural Machine Translation: A Systematic Comparison
Venue: arXiv cs.CL 2024
Year: 2024

Retrieval-Augmented Neural Machine Translation (RAMT) architectures retrieve examples from memory to guide the generation process. While most works in this trend explore new ways to exploit the retrieved examples, the upstream retrieval step is mostly unexplored. In this paper, we study the effect of varying retrieval methods for several translation architectures, to better understand the interplay between these two processes. We conduct experiments in two language pairs in a multi-domain setting and consider several downstream architectures based on a standard autoregressive model, an edit-based model, and a large language model with in-context learning. Our experiments show that the choice of the retrieval technique impacts the translation scores, with variance across architectures. We also discuss the effects of increasing the number and diversity of examples, which are mostly positive across the board.
AI-augmented Automation for Real Driving Prediction: an Industrial Use Case
Venue: arXiv cs.SE 2024
Year: 2024

The risen complexity of automotive systems requires new development strategies and methods to master the upcoming challenges. Traditional methods need thus to be changed by an increased level of automation, and a faster continuous improvement cycle. In this context, current vehicle performance tests represent a very time-consuming and expensive task due to the need to perform the tests in real driving conditions. As a consequence, agile/iterative processes like DevOps are largely hindered by the necessity of triggering frequent tests. This paper reports on a practical experience of developing an AI-augmented solution based on Machine Learning and Model-based Engineering to support continuous vehicle development and testing. In particular, historical data collected in real driving conditions is leveraged to synthesize a high-fidelity driving simulator and hence enable performance tests in virtual environments. Based on this practical experience, this paper also proposes a conceptual framework to support predictions based on real driving behavior.
Fragmented Moments, Balanced Choices: How Do People Make Use of Their Waiting Time?
Venue: arXiv cs.HC 2024
Year: 2024

Everyone spends some time waiting every day. HCI research has developed tools for boosting productivity while waiting. However, little is known about how people naturally spend their waiting time. We conducted an experience sampling study with 21 working adults who used a mobile app to report their daily waiting time activities over two weeks. The aim of this study is to understand the activities people do while waiting and the effect of situational factors. We found that participants spent about 60% of their waiting time on leisure activities, 20% on productive activities, and 20% on maintenance activities. These choices are sensitive to situational factors, including accessible device, location, and certain routines of the day. Our study complements previous ones by demonstrating that people purpose waiting time for various goals beyond productivity and to maintain work-life balance. Our findings shed light on future empirical research and system design for time management.
Universal Functional Regression with Neural Operator Flows
Venue: arXiv cs.LG 2024
Year: 2024

Regression on function spaces is typically limited to models with Gaussian process priors. We introduce the notion of universal functional regression, in which we aim to learn a prior distribution over non-Gaussian function spaces that remains mathematically tractable for functional regression. To do this, we develop Neural Operator Flows (OpFlow), an infinite-dimensional extension of normalizing flows. OpFlow is an invertible operator that maps the (potentially unknown) data function space into a Gaussian process, allowing for exact likelihood estimation of functional point evaluations. OpFlow enables robust and accurate uncertainty quantification via drawing posterior samples of the Gaussian process and subsequently mapping them into the data function space. We empirically study the performance of OpFlow on regression and generation tasks with data generated from Gaussian processes with known posterior forms and non-Gaussian processes, as well as real-world earthquake seismograms with an unknown closed-form distribution.
When Digital Twin Meets Generative AI: Intelligent Closed-Loop Network Management
Venue: arXiv cs.NI 2024
Year: 2024

Generative artificial intelligence (GAI) and digital twin (DT) are advanced data processing and virtualization technologies to revolutionize communication networks. Thanks to the powerful data processing capabilities of GAI, integrating it into DT is a potential approach to construct an intelligent holistic virtualized network for better network management performance. To this end, we propose a GAI-driven DT (GDT) network architecture to enable intelligent closed-loop network management. In the architecture, various GAI models can empower DT status emulation, feature abstraction, and network decision-making. The interaction between GAI-based and model-based data processing can facilitate intelligent external and internal closed-loop network management. To further enhance network management performance, three potential approaches are proposed, i.e., model light-weighting, adaptive model selection, and data-model-driven network management. We present a case study pertaining to data-model-driven network management for the GDT network, followed by some open research issues.