OpenRaiser/Intern-Atlas huggingface dataset:
On Measuring Context Utilization in Document-Level MT Systems
Venue: arXiv cs.CL 2024
Year: 2024

Document-level translation models are usually evaluated using general metrics such as BLEU, which are not informative about the benefits of context. Current work on context-aware evaluation, such as contrastive methods, only measure translation accuracy on words that need context for disambiguation. Such measures cannot reveal whether the translation model uses the correct supporting context. We propose to complement accuracy-based evaluation with measures of context utilization. We find that perturbation-based analysis (comparing models' performance when provided with correct versus random context) is an effective measure of overall context utilization. For a finer-grained phenomenon-specific evaluation, we propose to measure how much the supporting context contributes to handling context-dependent discourse phenomena. We show that automatically-annotated supporting context gives similar conclusions to human-annotated context and can be used as alternative for cases where human annotations are not available. Finally, we highlight the importance of using discourse-rich datasets when assessing context utilization.
XAI for Skin Cancer Detection with Prototypes and Non-Expert Supervision
Venue: arXiv cs.CV 2024
Year: 2024

Skin cancer detection through dermoscopy image analysis is a critical task. However, existing models used for this purpose often lack interpretability and reliability, raising the concern of physicians due to their black-box nature. In this paper, we propose a novel approach for the diagnosis of melanoma using an interpretable prototypical-part model. We introduce a guided supervision based on non-expert feedback through the incorporation of: 1) binary masks, obtained automatically using a segmentation network; and 2) user-refined prototypes. These two distinct information pathways aim to ensure that the learned prototypes correspond to relevant areas within the skin lesion, excluding confounding factors beyond its boundaries. Experimental results demonstrate that, even without expert supervision, our approach achieves superior performance and generalization compared to non-interpretable models.
Objective and subjective evaluation of speech enhancement methods in the UDASE task of the 7th CHiME challenge
Venue: arXiv cs.SD 2024
Year: 2024

Supervised models for speech enhancement are trained using artificially generated mixtures of clean speech and noise signals. However, the synthetic training conditions may not accurately reflect real-world conditions encountered during testing. This discrepancy can result in poor performance when the test domain significantly differs from the synthetic training domain. To tackle this issue, the UDASE task of the 7th CHiME challenge aimed to leverage real-world noisy speech recordings from the test domain for unsupervised domain adaptation of speech enhancement models. Specifically, this test domain corresponds to the CHiME-5 dataset, characterized by real multi-speaker and conversational speech recordings made in noisy and reverberant domestic environments, for which ground-truth clean speech signals are not available. In this paper, we present the objective and subjective evaluations of the systems that were submitted to the CHiME-7 UDASE task, and we provide an analysis of the results. This analysis reveals a limited correlation between subjective ratings and several supervised nonintrusive performance metrics recently proposed for speech enhancement. Conversely, the results suggest that more traditional intrusive objective metrics can be used for in-domain performance evaluation using the reverberant LibriCHiME-5 dataset developed for the challenge. The subjective evaluation indicates that all systems successfully reduced the background noise, but always at the expense of increased distortion. Out of the four speech enhancement methods evaluated subjectively, only one demonstrated an improvement in overall quality compared to the unprocessed noisy speech, highlighting the difficulty of the task. The tools and audio material created for the CHiME-7 UDASE task are shared with the community.
SMLP: Symbolic Machine Learning Prover
Venue: arXiv cs.LG 2024
Year: 2024

Symbolic Machine Learning Prover (SMLP) is a tool and a library for system exploration based on data samples obtained by simulating or executing the system on a number of input vectors. SMLP aims at exploring the system based on this data by taking a grey-box approach: SMLP combines statistical methods of data exploration with building and exploring machine learning models in close feedback loop with the system's response, and exploring these models by combining probabilistic and formal methods. SMLP has been applied in industrial setting at Intel for analyzing and optimizing hardware designs at the analog level. SMLP is a general purpose tool and can be applied to systems that can be sampled and modeled by machine learning models.
Different Tastes of Entities: Investigating Human Label Variation in Named Entity Annotations
Venue: arXiv cs.CL 2024
Year: 2024

Named Entity Recognition (NER) is a key information extraction task with a long-standing tradition. While recent studies address and aim to correct annotation errors via re-labeling efforts, little is known about the sources of human label variation, such as text ambiguity, annotation error, or guideline divergence. This is especially the case for high-quality datasets and beyond English CoNLL03. This paper studies disagreements in expert-annotated named entity datasets for three languages: English, Danish, and Bavarian. We show that text ambiguity and artificial guideline changes are dominant factors for diverse annotations among high-quality revisions. We survey student annotations on a subset of difficult entities and substantiate the feasibility and necessity of manifold annotations for understanding named entity ambiguities from a distributional perspective.
A Reinforcement Learning-Boosted Motion Planning Framework: Comprehensive Generalization Performance in Autonomous Driving
Venue: arXiv cs.RO 2024
Year: 2024

This study introduces a novel approach to autonomous motion planning, informing an analytical algorithm with a reinforcement learning (RL) agent within a Frenet coordinate system. The combination directly addresses the challenges of adaptability and safety in autonomous driving. Motion planning algorithms are essential for navigating dynamic and complex scenarios. Traditional methods, however, lack the flexibility required for unpredictable environments, whereas machine learning techniques, particularly reinforcement learning (RL), offer adaptability but suffer from instability and a lack of explainability. Our unique solution synergizes the predictability and stability of traditional motion planning algorithms with the dynamic adaptability of RL, resulting in a system that efficiently manages complex situations and adapts to changing environmental conditions. Evaluation of our integrated approach shows a significant reduction in collisions, improved risk management, and improved goal success rates across multiple scenarios. The code used in this research is publicly available as open-source software and can be accessed at the following link: https://github.com/TUM-AVS/Frenetix-RL.
Non-Linear Analog Processing Gains in Task-Based Quantization
Venue: arXiv cs.IT 2024
Year: 2024

In task-based quantization, a multivariate analog signal is transformed into a digital signal using a limited number of low-resolution analog-to-digital converters (ADCs). This process aims to minimize a fidelity criterion, which is assessed against an unobserved task variable that is correlated with the analog signal. The scenario models various applications of interest such as channel estimation, medical imaging applications, and object localization. This work explores the integration of analog processing components -- such as analog delay elements, polynomial operators, and envelope detectors -- prior to ADC quantization. Specifically, four scenarios, involving different collections of analog processing operators are considered: (i) arbitrary polynomial operators with analog delay elements, (ii) limited-degree polynomial operators, excluding delay elements, (iii) sequences of envelope detectors, and (iv) a combination of analog delay elements and linear combiners. For each scenario, the minimum achievable distortion is quantified through derivation of computable expressions in various statistical settings. It is shown that analog processing can significantly reduce the distortion in task reconstruction. Numerical simulations in a Gaussian example are provided to give further insights into the aforementioned analog processing gains.
Closing the Gap in Human Behavior Analysis: A Pipeline for Synthesizing Trimodal Data
Venue: arXiv cs.CV 2024
Year: 2024

In pervasive machine learning, especially in Human Behavior Analysis (HBA), RGB has been the primary modality due to its accessibility and richness of information. However, linked with its benefits are challenges, including sensitivity to lighting conditions and privacy concerns. One possibility to overcome these vulnerabilities is to resort to different modalities. For instance, thermal is particularly adept at accentuating human forms, while depth adds crucial contextual layers. Despite their known benefits, only a few HBA-specific datasets that integrate these modalities exist. To address this shortage, our research introduces a novel generative technique for creating trimodal, i.e., RGB, thermal, and depth, human-focused datasets. This technique capitalizes on human segmentation masks derived from RGB images, combined with thermal and depth backgrounds that are sourced automatically. With these two ingredients, we synthesize depth and thermal counterparts from existing RGB data utilizing conditional image-to-image translation. By employing this approach, we generate trimodal data that can be leveraged to train models for settings with limited data, bad lightning conditions, or privacy-sensitive areas.
SLYKLatent: A Learning Framework for Gaze Estimation Using Deep Facial Feature Learning
Venue: arXiv cs.CV 2024
Year: 2024

In this research, we present SLYKLatent, a novel approach for enhancing gaze estimation by addressing appearance instability challenges in datasets due to aleatoric uncertainties, covariant shifts, and test domain generalization. SLYKLatent utilizes Self-Supervised Learning for initial training with facial expression datasets, followed by refinement with a patch-based tri-branch network and an inverse explained variance-weighted training loss function. Our evaluation on benchmark datasets achieves a 10.9% improvement on Gaze360, supersedes top MPIIFaceGaze results with 3.8%, and leads on a subset of ETH-XGaze by 11.6%, surpassing existing methods by significant margins. Adaptability tests on RAF-DB and Affectnet show 86.4% and 60.9% accuracies, respectively. Ablation studies confirm the effectiveness of SLYKLatent's novel components.
Spiking Music: Audio Compression with Event Based Auto-encoders
Venue: arXiv cs.SD 2024
Year: 2024

Neurons in the brain communicate information via punctual events called spikes. The timing of spikes is thought to carry rich information, but it is not clear how to leverage this in digital systems. We demonstrate that event-based encoding is efficient for audio compression. To build this event-based representation we use a deep binary auto-encoder, and under high sparsity pressure, the model enters a regime where the binary event matrix is stored more efficiently with sparse matrix storage algorithms. We test this on the large MAESTRO dataset of piano recordings against vector quantized auto-encoders. Not only does our "Spiking Music compression" algorithm achieve a competitive compression/reconstruction trade-off, but selectivity and synchrony between encoded events and piano key strikes emerge without supervision in the sparse regime.
Training Adversarial yet Safe Agent to Characterize Safety Performance of Highly Automated Vehicles
Venue: arXiv cs.RO 2024
Year: 2024

This paper focuses on safety performance testing and characterization of black-box highly automated vehicles (HAV). Existing testing approaches typically obtain the testing outcomes by deploying the HAV into a specific testing environment. Such a testing environment can involve various passively given testing strategies presented by other traffic participants such as (i) the naturalistic driving policy learned from human drivers, (ii) extracted concrete scenarios from real-world driving data, and (iii) model-based or data-driven adversarial testing methodologies focusing on forcing safety-critical events. The safety performance of HAV is further characterized by analyzing the obtained testing outcomes with a particular selected measure, such as the observed collision risk. The aforementioned testing practices suffer from the scarcity of safety-critical events, have limited operational design domain (ODD) coverage, or are biased toward long-tail unsafe cases. This paper presents a novel and informative testing strategy that differs from these existing practices. The proposal is inspired by the intuition that a relatively safer HAV driving policy would allow the traffic vehicles to exhibit a higher level of aggressiveness to achieve a certain fixed level of an overall safe outcome. One can specifically characterize such a HAV and traffic interactive strategy and use it as a safety performance indicator for the HAV. Under the proposed testing scheme, the HAV is evaluated under its full ODD with a reward function that represents a trade-off between safety and adversity in generating safety-critical events. The proposed methodology is demonstrated in simulation with various HAV designs under different operational design domains.
TrustAgent: Towards Safe and Trustworthy LLM-based Agents
Venue: arXiv cs.CL 2024
Year: 2024

The rise of LLM-based agents shows great potential to revolutionize task planning, capturing significant attention. Given that these agents will be integrated into high-stake domains, ensuring their reliability and safety is crucial. This paper presents an Agent-Constitution-based agent framework, TrustAgent, with a particular focus on improving the LLM-based agent safety. The proposed framework ensures strict adherence to the Agent Constitution through three strategic components: pre-planning strategy which injects safety knowledge to the model before plan generation, in-planning strategy which enhances safety during plan generation, and post-planning strategy which ensures safety by post-planning inspection. Our experimental results demonstrate that the proposed framework can effectively enhance an LLM agent's safety across multiple domains by identifying and mitigating potential dangers during the planning. Further analysis reveals that the framework not only improves safety but also enhances the helpfulness of the agent. Additionally, we highlight the importance of the LLM reasoning ability in adhering to the Constitution. This paper sheds light on how to ensure the safe integration of LLM-based agents into human-centric environments. Data and code are available at https://github.com/agiresearch/TrustAgent.
Bisimulations and Logics for Higher-Dimensional Automata
Venue: arXiv cs.LO 2024
Year: 2024

Higher-dimensional automata (HDAs) are models of non-interleaving concurrency for analyzing concurrent systems. There is a rich literature that deals with bisimulations for concurrent systems, and some of them have been extended to HDAs. However, no logical characterizations of these relations are currently available for HDAs. In this work, we address this gap by introducing Ipomset modal logic, a Hennessy-Milner type logic over HDAs, and show that it characterizes Path-bisimulation, a variant of the standard ST-bisimulation. We also define a notion of Cell-bisimulation, using the open-maps framework of Joyal, Nielsen, and Winskel, and establish the relationship between these bisimulations (and also their "strong" variants, which take restrictions into account). In our work, we rely on the new categorical definition of HDAs as presheaves over concurrency lists and on track objects.
Stochastic Two Points Method for Deep Model Zeroth-order Optimization
Venue: arXiv cs.LG 2024
Year: 2024

Large foundation models, such as large language models, have performed exceptionally well in various application scenarios. Building or fully fine-tuning such large models is usually prohibitive due to either hardware budget or lack of access to backpropagation. The zeroth-order methods offer a promising direction for tackling this challenge, where only forward passes are needed to update the model. This paper introduces an efficient Stochastic Two-Point (S2P) approach within the gradient-free regime. We present the theoretical convergence properties of S2P under the general and relaxed smoothness assumptions, and the derived results help understand and inherently connect the two popular types of zeroth-order methods, basic random search and stochastic three-point method. The theoretical properties also shed light on a Variant of S2P (VS2P), through exploiting our new convergence properties that better represent the dynamics of deep models in training. Our comprehensive empirical results show that VS2P is highly effective in optimizing objectives for deep models. It outperforms or achieves competitive performance compared to standard methods across various model types and scales.
Recent Innovations in Footwear Sensors: Role of Smart Footwear in Healthcare -- A Survey
Venue: arXiv cs.CY 2024
Year: 2024

Smart shoes have ushered in a new era of personalised health monitoring and assistive technology. The shoe leverages technologies such as Bluetooth for data collection and wireless transmission and incorporates features such as GPS tracking, obstacle detection, and fitness tracking. This article provides an overview of the current state of smart shoe technology, highlighting the integration of advanced sensors for health monitoring, energy harvesting, assistive features for the visually impaired, and deep learning for data analysis. The study discusses the potential of smart footwear in medical applications, particularly for patients with diabetes, and the ongoing research in this field. Current footwear challenges are also discussed, including complex construction, poor fit, comfort, and high cost.
Forecasting Imports in OECD Member Countries and Iran by Using Neural Network Algorithms of LSTM
Venue: arXiv cs.CY 2024
Year: 2024

Artificial Neural Networks (ANN) which are a branch of artificial intelligence, have shown their high value in lots of applications and are used as a suitable forecasting method. Therefore, this study aims at forecasting imports in OECD member selected countries and Iran for 20 seasons from 2021 to 2025 by means of ANN. Data related to the imports of such countries collected over 50 years from 1970 to 2019 from valid resources including World Bank, WTO, IFM,the data turned into seasonal data to increase the number of collected data for better performance and high accuracy of the network by using Diz formula that there were totally 200 data related to imports. This study has used LSTM to analyse data in Pycharm. 75% of data considered as training data and 25% considered as test data and the results of the analysis were forecasted with 99% accuracy which revealed the validity and reliability of the output. Since the imports is consumption function and since the consumption is influenced during Covid-19 Pandemic, so it is time-consuming to correct and improve it to be influential on the imports, thus the imports in the years after Covid-19 Pandemic has had a fluctuating trend.
Using ChatGPT for Science Learning: A Study on Pre-service Teachers' Lesson Planning
Venue: arXiv cs.CY 2024
Year: 2024

Despite the buzz around ChatGPT's potential, empirical studies exploring its actual utility in the classroom for learning remain scarce. This study aims to fill this gap by analyzing the lesson plans developed by 29 pre-service elementary teachers from a Korean university and assessing how they integrated ChatGPT into science learning activities. We first examined how the subject domains and teaching and learning methods/strategies were integrated with ChatGPT in the lesson plans. We then evaluated the lesson plans using a modified TPACK-based rubric. We further examined pre-service teachers' perceptions and concerns about integrating ChatGPT into science learning. Results show diverse applications of ChatGPT in different science domains. Fourteen types of teaching and learning methods/strategies were identified in the lesson plans. On average, the pre-service teachers' lesson plans scored high on the modified TPACK-based rubric, indicating a reasonable envisage of integrating ChatGPT into science learning, particularly in 'instructional strategies & ChatGPT'. However, they scored relatively lower on exploiting ChatGPT's functions toward its full potential compared to other aspects. The study also identifies both appropriate and inappropriate use cases of ChatGPT in lesson planning. Pre-service teachers anticipated ChatGPT to afford high-quality questioning, self-directed learning, individualized learning support, and formative assessment. Meanwhile, they also expressed concerns about its accuracy and the risks that students may be overly dependent on ChatGPT. They further suggested solutions to systemizing classroom dynamics between teachers and students. The study underscores the need for more research on the roles of generative AI in actual classroom settings and provides insights for future AI-integrated science learning.
A Systematic Mapping Study of Digital Twins for Diagnosis in Transportation
Venue: arXiv cs.CY 2024
Year: 2024

In recent years, digital twins have been proposed and implemented in various fields with potential applications ranging from prototyping to maintenance. Going forward, they are to enable numerous efficient and sustainable technologies, among them autonomous cars. However, despite a large body of research in many fields, academics have yet to agree on what exactly a digital twin is -- and as a result, what its capabilities and limitations might be. To further our understanding, we explore the capabilities of digital twins concerning diagnosis in the field of transportation. We conduct a systematic mapping study including digital twins of vehicles and their components, as well as transportation infrastructure. We discovered that few papers on digital twins describe any diagnostic process. Furthermore, most existing approaches appear limited to system monitoring or fault detection. These findings suggest that we need more research for diagnostic reasoning utilizing digital twins.
Beyond Behaviorist Representational Harms: A Plan for Measurement and Mitigation
Venue: arXiv cs.CY 2024
Year: 2024

Algorithmic harms are commonly categorized as either allocative or representational. This study specifically addresses the latter, focusing on an examination of current definitions of representational harms to discern what is included and what is not. This analysis motivates our expansion beyond behavioral definitions to encompass harms to cognitive and affective states. The paper outlines high-level requirements for measurement: identifying the necessary expertise to implement this approach and illustrating it through a case study. Our work highlights the unique vulnerabilities of large language models to perpetrating representational harms, particularly when these harms go unmeasured and unmitigated. The work concludes by presenting proposed mitigations and delineating when to employ them. The overarching aim of this research is to establish a framework for broadening the definition of representational harms and to translate insights from fairness research into practical measurement and mitigation praxis.