OpenRaiser/Intern-Atlas huggingface dataset:
Real World Conversational Entity Linking Requires More Than Zeroshots
Venue: arXiv cs.CL 2024
Year: 2024

Entity linking (EL) in conversations faces notable challenges in practical applications, primarily due to the scarcity of entity-annotated conversational datasets and sparse knowledge bases (KB) containing domain-specific, long-tail entities. We designed targeted evaluation scenarios to measure the efficacy of EL models under resource constraints. Our evaluation employs two KBs: Fandom, exemplifying real-world EL complexities, and the widely used Wikipedia. First, we assess EL models' ability to generalize to a new unfamiliar KB using Fandom and a novel zero-shot conversational entity linking dataset that we curated based on Reddit discussions on Fandom entities. We then evaluate the adaptability of EL models to conversational settings without prior training. Our results indicate that current zero-shot EL models falter when introduced to new, domain-specific KBs without prior training, significantly dropping in performance. Our findings reveal that previous evaluation approaches fall short of capturing real-world complexities for zero-shot EL, highlighting the necessity for new approaches to design and assess conversational EL models to adapt to limited resources. The evaluation setup and the dataset proposed in this research are made publicly available.
Remote telepresence over large distances via robot avatars: case studies
Venue: arXiv cs.RO 2024
Year: 2024

This paper discusses the necessary considerations and adjustments that allow a recently proposed avatar system architecture to be used with different robotic avatar morphologies (both wheeled and legged robots with various types of hands and kinematic structures) for the purpose of enabling remote (intercontinental) telepresence under communication bandwidth restrictions. The case studies reported involve robots using both position and torque control modes, independently of their software middleware.
Mix Testing: Specifying and Testing ABI Compatibility of C/C++ Atomics Implementations
Venue: arXiv cs.PL 2024
Year: 2024

The correctness of complex software depends on the correctness of both the source code and the compilers that generate corresponding binary code. Compilers must do more than preserve the semantics of a single source file: they must ensure that generated binaries can be composed with other binaries to form a final executable. The compatibility of composition is ensured using an Application Binary Interface (ABI), which specifies details of calling conventions, exception handling, and so on. Unfortunately, there are no official ABIs for concurrent programs, so different atomics mappings, although correct in isolation, may induce bugs when composed. Indeed, today, mixing binaries generated by different compilers can lead to an erroneous resulting binary. We present mix testing: a new technique designed to find compiler bugs when the instructions of a C/C++ test are separately compiled for multiple compatible architectures and then mixed together. We define a class of compiler bugs, coined mixing bugs, that arise when parts of a program are compiled separately using different mappings from C/C++ atomic operations to assembly sequences. To demonstrate the generality of mix testing, we have designed and implemented a tool, atomic-mixer, which we have used: (a) to reproduce one existing non-mixing bug that state-of-the-art concurrency testing tools are limited to being able to find (showing that atomic-mixer at least meets the capabilities of these tools), and (b) to find four previously-unknown mixing bugs in LLVM and GCC, and one prospective mixing bug in mappings proposed for the Java Virtual Machine. Lastly, we have worked with engineers at Arm to specify, for the first time, an atomics ABI for Armv8, and have used atomic-mixer to validate the LLVM and GCC compilers against it.
Nuclear Atomic Norm for parametric estimation of sparse channels
Venue: arXiv cs.IT 2024
Year: 2024

Parametric channel estimation in mmWave not only enables the anticipated large spectral efficiency gains of \acs{MIMO} systems but also reveals important propagation parameters, allowing for a low complexity representation of the channel matrix. In this work, we propose to use atomic norm as a gridless multidimensional spectral estimation approach to address parametric channel estimation where both AoD and AoA are identified. The conditions for recovery of the propagation parameters are given depending on properties of the measurement matrix, and on structural features such as the antenna geometry or the number of scatters to resolve. The proposed methodology is compared against several state-of-the-art parametric approaches.
Development and Validation of a Modular Sensor-Based System for Gait Analysis and Control in Lower-Limb Exoskeletons
Venue: arXiv cs.RO 2024
Year: 2024

With rapid advancements in exoskeleton hardware technologies, successful assessment and accurate control remain challenging. This study introduces a modular sensor-based system to enhance biomechanical evaluation and control in lower-limb exoskeletons, utilizing advanced sensor technologies and fuzzy logic. We aim to surpass the limitations of current biomechanical evaluation methods confined to laboratories and to address the high costs and complexity of exoskeleton control systems. The system integrates inertial measurement units, force-sensitive resistors, and load cells into instrumented crutches and 3D-printed insoles. These components function both independently and collectively to capture comprehensive biomechanical data, including the anteroposterior center of pressure and crutch ground reaction forces. This data is processed through a central unit using fuzzy logic algorithms for real-time gait phase estimation and exoskeleton control. Validation experiments with three participants, benchmarked against gold-standard motion capture and force plate technologies, demonstrate our system's capability for reliable gait phase detection and precise biomechanical measurements. By offering our designs open-source and integrating cost-effective technologies, this study advances wearable robotics and promotes broader innovation and adoption in exoskeleton research.
Integrating End-to-End and Modular Driving Approaches for Online Corner Case Detection in Autonomous Driving
Venue: arXiv cs.AI 2024
Year: 2024

Online corner case detection is crucial for ensuring safety in autonomous driving vehicles. Current autonomous driving approaches can be categorized into modular approaches and end-to-end approaches. To leverage the advantages of both, we propose a method for online corner case detection that integrates an end-to-end approach into a modular system. The modular system takes over the primary driving task and the end-to-end network runs in parallel as a secondary one, the disagreement between the systems is then used for corner case detection. We implement this method on a real vehicle and evaluate it qualitatively. Our results demonstrate that end-to-end networks, known for their superior situational awareness, as secondary driving systems, can effectively contribute to corner case detection. These findings suggest that such an approach holds potential for enhancing the safety of autonomous vehicles.
SBOM Generation Tools in the Python Ecosystem: an In-Detail Analysis
Venue: arXiv cs.CR 2024
Year: 2024

Software Bills of Material (SBOMs), which improve transparency by listing the components constituting software, are a key countermeasure to the mounting problem of Software Supply Chain attacks. SBOM generation tools take project source files and provide an SBOM as output, interacting with the software ecosystem. While SBOMs are a substantial improvement for security practitioners, providing a complete and correct SBOM is still an open problem. This paper investigates the causes of the issues affecting SBOM completeness and correctness, focusing on the PyPI ecosystem. We analyze four popular SBOM generation tools using the CycloneDX standard. Our analysis highlights issues related to dependency versions, metadata files, remote dependencies, and optional dependencies. Additionally, we identified a systematic issue with the lack of standards for metadata in the PyPI ecosystem. This includes inconsistencies in the presence of metadata files as well as variations in how their content is formatted.
Real-time Accident Anticipation for Autonomous Driving Through Monocular Depth-Enhanced 3D Modeling
Venue: arXiv cs.CV 2024
Year: 2024

The primary goal of traffic accident anticipation is to foresee potential accidents in real time using dashcam videos, a task that is pivotal for enhancing the safety and reliability of autonomous driving technologies. In this study, we introduce an innovative framework, AccNet, which significantly advances the prediction capabilities beyond the current state-of-the-art (SOTA) 2D-based methods by incorporating monocular depth cues for sophisticated 3D scene modeling. Addressing the prevalent challenge of skewed data distribution in traffic accident datasets, we propose the Binary Adaptive Loss for Early Anticipation (BA-LEA). This novel loss function, together with a multi-task learning strategy, shifts the focus of the predictive model towards the critical moments preceding an accident. {We rigorously evaluate the performance of our framework on three benchmark datasets--Dashcam Accident Dataset (DAD), Car Crash Dataset (CCD), and AnAn Accident Detection (A3D), and DADA-2000 Dataset--demonstrating its superior predictive accuracy through key metrics such as Average Precision (AP) and mean Time-To-Accident (mTTA).
One-Index Vector Quantization Based Adversarial Attack on Image Classification
Venue: arXiv cs.CV 2024
Year: 2024

To improve storage and transmission, images are generally compressed. Vector quantization (VQ) is a popular compression method as it has a high compression ratio that suppresses other compression techniques. Despite this, existing adversarial attack methods on image classification are mostly performed in the pixel domain with few exceptions in the compressed domain, making them less applicable in real-world scenarios. In this paper, we propose a novel one-index attack method in the VQ domain to generate adversarial images by a differential evolution algorithm, successfully resulting in image misclassification in victim models. The one-index attack method modifies a single index in the compressed data stream so that the decompressed image is misclassified. It only needs to modify a single VQ index to realize an attack, which limits the number of perturbed indexes. The proposed method belongs to a semi-black-box attack, which is more in line with the actual attack scenario. We apply our method to attack three popular image classification models, i.e., Resnet, NIN, and VGG16. On average, 55.9% and 77.4% of the images in CIFAR-10 and Fashion MNIST, respectively, are successfully attacked, with a high level of misclassification confidence and a low level of image perturbation.
How local constraints influence network diameter and applications to LCL generalizations
Venue: arXiv cs.DC 2024
Year: 2024

In this paper, we investigate how local rules enforced at every node can influence the topology of a network. More precisely, we establish several results on the diameter of trees as a function of the number of nodes, as listed below. These results have important consequences on the landscape of locally checkable labelings (LCL) on \emph{unbounded} degree graphs, a case in which our lack of knowledge is in striking contrast with that of \emph{bounded degree graphs}, that has been intensively studied recently. [See paper for full abstract.]
Spectron: Target Speaker Extraction using Conditional Transformer with Adversarial Refinement
Venue: arXiv cs.SD 2024
Year: 2024

Recently, attention-based transformers have become a de facto standard in many deep learning applications including natural language processing, computer vision, signal processing, etc.. In this paper, we propose a transformer-based end-to-end model to extract a target speaker's speech from a monaural multi-speaker mixed audio signal. Unlike existing speaker extraction methods, we introduce two additional objectives to impose speaker embedding consistency and waveform encoder invertibility and jointly train both speaker encoder and speech separator to better capture the speaker conditional embedding. Furthermore, we leverage a multi-scale discriminator to refine the perceptual quality of the extracted speech. Our experiments show that the use of a dual path transformer in the separator backbone along with proposed training paradigm improves the CNN baseline by $3.12$ dB points. Finally, we compare our approach with recent state-of-the-arts and show that our model outperforms existing methods by $4.1$ dB points on an average without creating additional data dependency.
Correlating Time Series with Interpretable Convolutional Kernels
Venue: arXiv cs.LG 2024
Year: 2024

This study addresses the problem of convolutional kernel learning in univariate, multivariate, and multidimensional time series data, which is crucial for interpreting temporal patterns in time series and supporting downstream machine learning tasks. First, we propose formulating convolutional kernel learning for univariate time series as a sparse regression problem with a non-negative constraint, leveraging the properties of circular convolution and circulant matrices. Second, to generalize this approach to multivariate and multidimensional time series data, we use tensor computations, reformulating the convolutional kernel learning problem in the form of tensors. This is further converted into a standard sparse regression problem through vectorization and tensor unfolding operations. In the proposed methodology, the optimization problem is addressed using the existing non-negative subspace pursuit method, enabling the convolutional kernel to capture temporal correlations and patterns. To evaluate the proposed model, we apply it to several real-world time series datasets. On the multidimensional rideshare and taxi trip data from New York City and Chicago, the convolutional kernels reveal interpretable local correlations and cyclical patterns, such as weekly seasonality. In the context of multidimensional fluid flow data, both local and nonlocal correlations captured by the convolutional kernels can reinforce tensor factorization, leading to performance improvements in fluid flow reconstruction tasks. Thus, this study lays an insightful foundation for automatically learning convolutional kernels from time series data, with an emphasis on interpretability through sparsity and non-negativity constraints.
Content, Nudges and Incentives: A Study on the Effectiveness and Perception of Embedded Phishing Training
Venue: arXiv cs.CR 2024
Year: 2024

A common form of phishing training in organizations is the use of simulated phishing emails to test employees' susceptibility to phishing attacks, and the immediate delivery of training material to those who fail the test. This widespread practice is dubbed embedded training; however, its effectiveness in decreasing the likelihood of employees falling for phishing again in the future is questioned by the contradictory findings of several recent field studies. We investigate embedded phishing training in three aspects. First, we observe that the practice incorporates different components -- knowledge gains from its content, nudges and reminders from the test itself, and the deterrent effect of potential consequences -- our goal is to study which ones are more effective, if any. Second, we explore two potential improvements to training, namely its timing and the use of incentives. Third, we analyze employees' reception and perception of the practice. For this, we conducted a large-scale mixed-methods (quantitative and qualitative) study on the employees of a partner company. Our study contributes several novel findings on the training practice: in particular, its effectiveness comes from its nudging effect, i.e., the periodic reminder of the threat rather than from its content, which is rarely consumed by employees due to lack of time and perceived usefulness. Further, delaying training to ease time pressure is as effective as currently established practices, while rewards do not improve secure behavior. Finally, some of our results support previous findings with increased ecological validity, e.g., that phishing is an attention problem, rather than a knowledge one, even for the most susceptible employees, and thus enforcing training does not help.
Healthcare system resilience and adaptability to pandemic disruptions in the United States
Venue: arXiv cs.SI 2024
Year: 2024

Understanding healthcare system resilience has become paramount, particularly in the wake of the COVID-19 pandemic, which imposed unprecedented burdens on healthcare services and severely impacted public health. Resilience is defined as the system's ability to absorb, recover from, and adapt to disruptions; however, despite extensive studies on this subject, we still lack empirical evidence and mathematical tools to quantify its adaptability (the ability of the system to adjust to and learn from disruptions). By analyzing millions of patients' electronic medical records across US states, we find that the COVID-19 pandemic caused two successive waves of disruptions within the healthcare systems, enabling natural experiment analysis of the adaptive capacity for each system to adapt to past disruptions. We generalize the quantification framework and find that the US healthcare systems exhibit substantial adaptability but only a moderate level of resilience. When considering system responses across racial groups, Black and Hispanic groups were more severely impacted by pandemic disruptions than White and Asian groups. Physician abundance is the key characteristic for determining healthcare system resilience. Our results offer vital guidance in designing resilient and sustainable healthcare systems to prepare for future waves of disruptions akin to COVID-19 pandemics.
Semantic Segmentation from Image Labels by Reconstruction from Structured Decomposition
Venue: arXiv cs.CV 2024
Year: 2024

Weakly supervised image segmentation (WSSS) from image tags remains challenging due to its under-constraint nature. Most mainstream work focus on the extraction of class activation map (CAM) and imposing various additional regularization. Contrary to the mainstream, we propose to frame WSSS as a problem of reconstruction from decomposition of the image using its mask, under which most regularization are embedded implicitly within the framework of the new problem. Our approach has demonstrated promising results on initial experiments, and shown robustness against the problem of background ambiguity. Our code is available at \url{https://github.com/xuanrui-work/WSSSByRec}.
Less is more: concatenating videos for Sign Language Translation from a small set of signs
Venue: arXiv cs.CV 2024
Year: 2024

The limited amount of labeled data for training the Brazilian Sign Language (Libras) to Portuguese Translation models is a challenging problem due to video collection and annotation costs. This paper proposes generating sign language content by concatenating short clips containing isolated signals for training Sign Language Translation models. We employ the V-LIBRASIL dataset, composed of 4,089 sign videos for 1,364 signs, interpreted by at least three persons, to create hundreds of thousands of sentences with their respective Libras translation, and then, to feed the model. More specifically, we propose several experiments varying the vocabulary size and sentence structure, generating datasets with approximately 170K, 300K, and 500K videos. Our results achieve meaningful scores of 9.2% and 26.2% for BLEU-4 and METEOR, respectively. Our technique enables the creation or extension of existing datasets at a much lower cost than the collection and annotation of thousands of sentences providing clear directions for future works.
On the Design Space Between Transformers and Recursive Neural Nets
Venue: arXiv cs.LG 2024
Year: 2024

In this paper, we study two classes of models, Recursive Neural Networks (RvNNs) and Transformers, and show that a tight connection between them emerges from the recent development of two recent models - Continuous Recursive Neural Networks (CRvNN) and Neural Data Routers (NDR). On one hand, CRvNN pushes the boundaries of traditional RvNN, relaxing its discrete structure-wise composition and ends up with a Transformer-like structure. On the other hand, NDR constrains the original Transformer to induce better structural inductive bias, ending up with a model that is close to CRvNN. Both models, CRvNN and NDR, show strong performance in algorithmic tasks and generalization in which simpler forms of RvNNs and Transformers fail. We explore these "bridge" models in the design space between RvNNs and Transformers, formalize their tight connections, discuss their limitations, and propose ideas for future research.
Benchmarking Cognitive Domains for LLMs: Insights from Taiwanese Hakka Culture
Venue: arXiv cs.CL 2024
Year: 2024

This study introduces a comprehensive benchmark designed to evaluate the performance of large language models (LLMs) in understanding and processing cultural knowledge, with a specific focus on Hakka culture as a case study. Leveraging Bloom's Taxonomy, the study develops a multi-dimensional framework that systematically assesses LLMs across six cognitive domains: Remembering, Understanding, Applying, Analyzing, Evaluating, and Creating. This benchmark extends beyond traditional single-dimensional evaluations by providing a deeper analysis of LLMs' abilities to handle culturally specific content, ranging from basic recall of facts to higher-order cognitive tasks such as creative synthesis. Additionally, the study integrates Retrieval-Augmented Generation (RAG) technology to address the challenges of minority cultural knowledge representation in LLMs, demonstrating how RAG enhances the models' performance by dynamically incorporating relevant external information. The results highlight the effectiveness of RAG in improving accuracy across all cognitive domains, particularly in tasks requiring precise retrieval and application of cultural knowledge. However, the findings also reveal the limitations of RAG in creative tasks, underscoring the need for further optimization. This benchmark provides a robust tool for evaluating and comparing LLMs in culturally diverse contexts, offering valuable insights for future research and development in AI-driven cultural knowledge preservation and dissemination.