OpenRaiser/Intern-Atlas huggingface dataset:
Model Comparison for Fast Domain Adaptation in Table Service Scenario
Venue: arXiv cs.RO 2024
Year: 2024

In restaurants, many aspects of customer service, such as greeting customers, taking orders, and processing payments, are automated. Due to the various cuisines, required services, and different standards of each restaurant, one challenging part of making the entire automated process is inspecting and providing appropriate services at the table during a meal. In this paper, we demonstrate an approach for automatically checking and providing services at the table. We initially construct a base model to recognize common information to comprehend the context of the table, such as object category, remaining food quantity, and meal progress status. After that, we add a service recognition classifier and retrain the model using a small amount of local restaurant data. We gathered data capturing the restaurant table during the meal in order to find a suitable service recognition classifier. With different inputs, combinations, time series, and data choices, we carried out a variety of tests. Through these tests, we discovered that the model with few significant data points and trainable parameters is more crucial in the case of sparse and redundant retraining data.
ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment
Venue: arXiv cs.CV 2024
Year: 2024

Diffusion models have demonstrated remarkable performance in the domain of text-to-image generation. However, most widely used models still employ CLIP as their text encoder, which constrains their ability to comprehend dense prompts, encompassing multiple objects, detailed attributes, complex relationships, long-text alignment, etc. In this paper, we introduce an Efficient Large Language Model Adapter, termed ELLA, which equips text-to-image diffusion models with powerful Large Language Models (LLM) to enhance text alignment without training of either U-Net or LLM. To seamlessly bridge two pre-trained models, we investigate a range of semantic alignment connector designs and propose a novel module, the Timestep-Aware Semantic Connector (TSC), which dynamically extracts timestep-dependent conditions from LLM. Our approach adapts semantic features at different stages of the denoising process, assisting diffusion models in interpreting lengthy and intricate prompts over sampling timesteps. Additionally, ELLA can be readily incorporated with community models and tools to improve their prompt-following capabilities. To assess text-to-image models in dense prompt following, we introduce Dense Prompt Graph Benchmark (DPG-Bench), a challenging benchmark consisting of 1K dense prompts. Extensive experiments demonstrate the superiority of ELLA in dense prompt following compared to state-of-the-art methods, particularly in multiple object compositions involving diverse attributes and relationships.
Learning Expressive And Generalizable Motion Features For Face Forgery Detection
Venue: arXiv cs.CV 2024
Year: 2024

Previous face forgery detection methods mainly focus on appearance features, which may be easily attacked by sophisticated manipulation. Considering the majority of current face manipulation methods generate fake faces based on a single frame, which do not take frame consistency and coordination into consideration, artifacts on frame sequences are more effective for face forgery detection. However, current sequence-based face forgery detection methods use general video classification networks directly, which discard the special and discriminative motion information for face manipulation detection. To this end, we propose an effective sequence-based forgery detection framework based on an existing video classification method. To make the motion features more expressive for manipulation detection, we propose an alternative motion consistency block instead of the original motion features module. To make the learned features more generalizable, we propose an auxiliary anomaly detection block. With these two specially designed improvements, we make a general video classification network achieve promising results on three popular face forgery datasets.
Adversarial Sparse Teacher: Defense Against Distillation-Based Model Stealing Attacks Using Adversarial Examples
Venue: arXiv cs.LG 2024
Year: 2024

We introduce Adversarial Sparse Teacher (AST), a robust defense method against distillation-based model stealing attacks. Our approach trains a teacher model using adversarial examples to produce sparse logit responses and increase the entropy of the output distribution. Typically, a model generates a peak in its output corresponding to its prediction. By leveraging adversarial examples, AST modifies the teacher model's original response, embedding a few altered logits into the output while keeping the primary response slightly higher. Concurrently, all remaining logits are elevated to further increase the output distribution's entropy. All these complex manipulations are performed using an optimization function with our proposed Exponential Predictive Divergence (EPD) loss function. EPD allows us to maintain higher entropy levels compared to traditional KL divergence, effectively confusing attackers. Experiments on CIFAR-10 and CIFAR-100 datasets demonstrate that AST outperforms state-of-the-art methods, providing effective defense against model stealing while preserving high accuracy. The source codes will be made publicly available here soon.
ROUGE-K: Do Your Summaries Have Keywords?
Venue: arXiv cs.CL 2024
Year: 2024

Keywords, that is, content-relevant words in summaries play an important role in efficient information conveyance, making it critical to assess if system-generated summaries contain such informative words during evaluation. However, existing evaluation metrics for extreme summarization models do not pay explicit attention to keywords in summaries, leaving developers ignorant of their presence. To address this issue, we present a keyword-oriented evaluation metric, dubbed ROUGE-K, which provides a quantitative answer to the question of -- \textit{How well do summaries include keywords?} Through the lens of this keyword-aware metric, we surprisingly find that a current strong baseline model often misses essential information in their summaries. Our analysis reveals that human annotators indeed find the summaries with more keywords to be more relevant to the source documents. This is an important yet previously overlooked aspect in evaluating summarization systems. Finally, to enhance keyword inclusion, we propose four approaches for incorporating word importance into a transformer-based model and experimentally show that it enables guiding models to include more keywords while keeping the overall quality. Our code is released at https://github.com/sobamchan/rougek.
Evaluation of Road User Radio-Frequency Exposure Levels in an Urban Environment from Vehicular Antennas and the Infrastructure in ITS-G5 5.9 GHz Communication
Venue: arXiv cs.NI 2024
Year: 2024

This study aims to investigate the variability of exposure levels among road users generated in a realistic urban scenario by Vehicle-to-Vehicle (V2V) and Vehicle-to-Infrastructure (V2I) communication technologies operating at 5.9 GHz. The exposure levels were evaluated in terms of whole-body Specific Absorption Rate (wbSAR) [W/kg] in three different human models, ranging from children to adults. We calculated the electromagnetic field exposure level generated by V2V and V2I using raytracing and we assessed wbSAR resulting in urban exposure scenarios with an increasing number of transmitting antennas. Whole-body SAR was generally very low, on the order of 10^-4 W/kg. The maximum wbSAR, of 4.9x10^-4 W/kg, was obtained in the worst-case exposure condition comprising more than one transmitting vehicle and was found in the adult model for a distance within 10 m from the transmitting cars. We found that the height of the human model highly impacted the exposure level. Namely, the child (which is the shortest human model) was generally much less exposed than adults. All the wbSAR values found by varying the number of transmitting antennas, the distance of the road user from the antennas, and the type of human model (adult vs. child) were very well below the limits set by the ICNIRP and IEEE guidelines of 0.08 W/kg for human exposure in the 100 kHz - 300 GHz range.
AQuA: Automated Question-Answering in Software Tutorial Videos with Visual Anchors
Venue: arXiv cs.HC 2024
Year: 2024

Tutorial videos are a popular help source for learning feature-rich software. However, getting quick answers to questions about tutorial videos is difficult. We present an automated approach for responding to tutorial questions. By analyzing 633 questions found in 5,944 video comments, we identified different question types and observed that users frequently described parts of the video in questions. We then asked participants (N=24) to watch tutorial videos and ask questions while annotating the video with relevant visual anchors. Most visual anchors referred to UI elements and the application workspace. Based on these insights, we built AQuA, a pipeline that generates useful answers to questions with visual anchors. We demonstrate this for Fusion 360, showing that we can recognize UI elements in visual anchors and generate answers using GPT-4 augmented with that visual information and software documentation. An evaluation study (N=16) demonstrates that our approach provides better answers than baseline methods.
Understanding Hybrid Spaces: Designing a Spacetime Model to Represent Dynamic Topologies of Hybrid Spaces
Venue: arXiv cs.CY 2024
Year: 2024

This paper develops a spatiotemporal model for the visualization of dynamic topologies of hybrid spaces. The visualization of spatiotemporal data is a well-known problem, for example in digital twins in urban planning. There is also a lack of a basic ontology for understanding hybrid spaces. The developed spatiotemporal model has three levels: a level of places and media types, a level of perception and a level of time and interaction. Existing concepts and types of representation of hybrid spaces are presented. The space-time model is tested on the basis of an art exhibition. Two hypotheses guide the accompanying online survey: (A) there are correlations between media use (modality), the participants' interactions (creativity) and their perception (understanding of art) and (B) individual parameters (demographic data, location and situation, individual knowledge) influence perception (understanding of art). The range, the number of interactions and the response rate were also evaluated. The online survey generally showed a positive correlation between media use (modality) and individual activity (creativity). However, due to the low participation rate ($P_{TN} = 14$), the survey is unfortunately not very representative. Various dynamic topologies of hybrid spaces were successfully visualized. The joint representation of real and virtual places and media types conveys a new basic understanding of place, range and urban density. Relationships between modality, Mobility and communicative interaction become visible. The current phenomenon of multilocality has been successfully mapped. The space-time model enables more precise class and structure formation, for example in the development of digital twins. Dynamic topologies of hybrid spaces, such as in social media, at events or in urban development, can thus be better represented and compared.
Fairness-Aware Interpretable Modeling (FAIM) for Trustworthy Machine Learning in Healthcare
Venue: arXiv cs.LG 2024
Year: 2024

The escalating integration of machine learning in high-stakes fields such as healthcare raises substantial concerns about model fairness. We propose an interpretable framework - Fairness-Aware Interpretable Modeling (FAIM), to improve model fairness without compromising performance, featuring an interactive interface to identify a "fairer" model from a set of high-performing models and promoting the integration of data-driven evidence and clinical expertise to enhance contextualized fairness. We demonstrated FAIM's value in reducing sex and race biases by predicting hospital admission with two real-world databases, MIMIC-IV-ED and SGH-ED. We show that for both datasets, FAIM models not only exhibited satisfactory discriminatory performance but also significantly mitigated biases as measured by well-established fairness metrics, outperforming commonly used bias-mitigation methods. Our approach demonstrates the feasibility of improving fairness without sacrificing performance and provides an a modeling mode that invites domain experts to engage, fostering a multidisciplinary effort toward tailored AI fairness.
vSPACE: Voting in a Scalable, Privacy-Aware and Confidential Election
Venue: arXiv cs.CR 2024
Year: 2024

The vSPACE experimental proof-of-concept (PoC) on the TrueElect[Anon][Creds] protocol presents a novel approach to secure, private, and scalable elections, extending the TrueElect and ElectAnon protocols with the integration of AnonCreds SSI (Self-Sovereign Identity). Such a protocol PoC is situated within a Zero-Trust Architecture (ZTA) and leverages confidential computing, continuous authentication, multi-party computation (MPC), and well-architected framework (WAF) principles to address the challenges of cybersecurity, privacy, and trust over IP (ToIP) protection. Employing a Kubernetes confidential cluster within an Enterprise-Scale Landing Zone (ESLZ), vSPACE integrates Distributed Ledger Technology (DLT) for immutable and certifiable audit trails. The Infrastructure as Code (IaC) model ensures rapid deployment, consistent management, and adherence to security standards, making vSPACE a future-proof solution for digital voting systems.
ContrastDiagnosis: Enhancing Interpretability in Lung Nodule Diagnosis Using Contrastive Learning
Venue: arXiv cs.CV 2024
Year: 2024

With the ongoing development of deep learning, an increasing number of AI models have surpassed the performance levels of human clinical practitioners. However, the prevalence of AI diagnostic products in actual clinical practice remains significantly lower than desired. One crucial reason for this gap is the so-called `black box' nature of AI models. Clinicians' distrust of black box models has directly hindered the clinical deployment of AI products. To address this challenge, we propose ContrastDiagnosis, a straightforward yet effective interpretable diagnosis framework. This framework is designed to introduce inherent transparency and provide extensive post-hoc explainability for deep learning model, making them more suitable for clinical medical diagnosis. ContrastDiagnosis incorporates a contrastive learning mechanism to provide a case-based reasoning diagnostic rationale, enhancing the model's transparency and also offers post-hoc interpretability by highlighting similar areas. High diagnostic accuracy was achieved with AUC of 0.977 while maintain a high transparency and explainability.
Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks
Venue: arXiv cs.LG 2024
Year: 2024

In this work, we investigate the effect of momentum on the optimisation trajectory of gradient descent. We leverage a continuous-time approach in the analysis of momentum gradient descent with step size $γ$ and momentum parameter $β$ that allows us to identify an intrinsic quantity $λ= \frac{ γ}{ (1 - β)^2 }$ which uniquely defines the optimisation path and provides a simple acceleration rule. When training a $2$-layer diagonal linear network in an overparametrised regression setting, we characterise the recovered solution through an implicit regularisation problem. We then prove that small values of $λ$ help to recover sparse solutions. Finally, we give similar but weaker results for stochastic momentum gradient descent. We provide numerical experiments which support our claims.
A Collaborative Robot-Assisted Manufacturing Assembly Process
Venue: arXiv cs.RO 2024
Year: 2024

An effective human-robot collaborative process results in the reduction of the operator's workload, promoting a more efficient, productive, safer and less error-prone working environment. However, the implementation of collaborative robots in industry is still challenging. In this work, we compare manual and robot-assisted assembly processes to evaluate the effectiveness of collaborative robots while featuring different modes of operation (coexistence, cooperation and collaboration). Results indicate an improvement in ergonomic conditions and ease of execution without substantially compromising assembly time. Furthermore, the robot is intuitive to use and guides the user on the proper sequencing of the process.
OccFusion: Depth Estimation Free Multi-sensor Fusion for 3D Occupancy Prediction
Venue: arXiv cs.CV 2024
Year: 2024

3D occupancy prediction based on multi-sensor fusion,crucial for a reliable autonomous driving system, enables fine-grained understanding of 3D scenes. Previous fusion-based 3D occupancy predictions relied on depth estimation for processing 2D image features. However, depth estimation is an ill-posed problem, hindering the accuracy and robustness of these methods. Furthermore, fine-grained occupancy prediction demands extensive computational resources. To address these issues, we propose OccFusion, a depth estimation free multi-modal fusion framework. Additionally, we introduce a generalizable active training method and an active decoder that can be applied to any occupancy prediction model, with the potential to enhance their performance. Experiments conducted on nuScenes-Occupancy and nuScenes-Occ3D demonstrate our framework's superior performance. Detailed ablation studies highlight the effectiveness of each proposed method.
On the Certification of the Kinematics of 3-DOF Spherical Parallel Manipulators
Venue: arXiv cs.RO 2024
Year: 2024

This paper aims to study a specific kind of parallel robot: Spherical Parallel Manipulators (SPM) that are capable of unlimited rolling. A focus is made on the kinematics of such mechanisms, especially taking into account uncertainties (e.g. on conception & fabrication parameters, measures) and their propagations. Such considerations are crucial if we want to control our robot correctly without any undesirable behavior in its workspace (e.g. effects of singularities). In this paper, we will consider two different approaches to study the kinematics and the singularities of the robot of interest: symbolic and semi-numerical. By doing so, we can compute a singularity-free zone in the work- and joint spaces, considering given uncertainties on the parameters. In this zone, we can use any control law to inertially stabilize the upper platform of the robot.
Survival probability of structures under fatigue: a data-based approach
Venue: arXiv cs.CE 2024
Year: 2024

This article addresses the probabilistic nature of fatigue life in structures subjected to cyclic loading with variable amplitude. Drawing on the formalisation of Miner's cumulative damage rule that we introduced in the recent article [Cartiaux, Ehrlacher, Legoll, Libal and Reygner, Prob. Eng. Mech. 2023], we apply our methodology to estimate the survival probability of an industrial structure using experimental data. The study considers both the randomness in the initial state of the structure and in the amplitude of loading cycles. The results indicate that the variability of loading cycles can be captured through the concept of deterministic equivalent damage, providing a computationally efficient method for assessing the fatigue life of the structure. Furthermore, the article highlights that the usual combination of Miner's rule and of the weakest link principle systematically overestimates the structure's fatigue life. On the case study that we consider, this overestimation reaches a multiplicative factor of more than two. We then describe how the probabilistic framework that we have introduced offers a remedy to this overestimation.
SIRST-5K: Exploring Massive Negatives Synthesis with Self-supervised Learning for Robust Infrared Small Target Detection
Venue: arXiv cs.CV 2024
Year: 2024

Single-frame infrared small target (SIRST) detection aims to recognize small targets from clutter backgrounds. Recently, convolutional neural networks have achieved significant advantages in general object detection. With the development of Transformer, the scale of SIRST models is constantly increasing. Due to the limited training samples, performance has not been improved accordingly. The quality, quantity, and diversity of the infrared dataset are critical to the detection of small targets. To highlight this issue, we propose a negative sample augmentation method in this paper. Specifically, a negative augmentation approach is proposed to generate massive negatives for self-supervised learning. Firstly, we perform a sequential noise modeling technology to generate realistic infrared data. Secondly, we fuse the extracted noise with the original data to facilitate diversity and fidelity in the generated data. Lastly, we proposed a negative augmentation strategy to enrich diversity as well as maintain semantic invariance. The proposed algorithm produces a synthetic SIRST-5K dataset, which contains massive pseudo-data and corresponding labels. With a rich diversity of infrared small target data, our algorithm significantly improves the model performance and convergence speed. Compared with other state-of-the-art (SOTA) methods, our method achieves outstanding performance in terms of probability of detection (Pd), false-alarm rate (Fa), and intersection over union (IoU).
EVD4UAV: An Altitude-Sensitive Benchmark to Evade Vehicle Detection in UAV
Venue: arXiv cs.CV 2024
Year: 2024

Vehicle detection in Unmanned Aerial Vehicle (UAV) captured images has wide applications in aerial photography and remote sensing. There are many public benchmark datasets proposed for the vehicle detection and tracking in UAV images. Recent studies show that adding an adversarial patch on objects can fool the well-trained deep neural networks based object detectors, posing security concerns to the downstream tasks. However, the current public UAV datasets might ignore the diverse altitudes, vehicle attributes, fine-grained instance-level annotation in mostly side view with blurred vehicle roof, so none of them is good to study the adversarial patch based vehicle detection attack problem. In this paper, we propose a new dataset named EVD4UAV as an altitude-sensitive benchmark to evade vehicle detection in UAV with 6,284 images and 90,886 fine-grained annotated vehicles. The EVD4UAV dataset has diverse altitudes (50m, 70m, 90m), vehicle attributes (color, type), fine-grained annotation (horizontal and rotated bounding boxes, instance-level mask) in top view with clear vehicle roof. One white-box and two black-box patch based attack methods are implemented to attack three classic deep neural networks based object detectors on EVD4UAV. The experimental results show that these representative attack methods could not achieve the robust altitude-insensitive attack performance.