OpenRaiser/Intern-Atlas huggingface dataset:
Adversarial Attacks to Multi-Modal Models
Venue: arXiv cs.CR 2024
Year: 2024

Multi-modal models have gained significant attention due to their powerful capabilities. These models effectively align embeddings across diverse data modalities, showcasing superior performance in downstream tasks compared to their unimodal counterparts. Recent study showed that the attacker can manipulate an image or audio file by altering it in such a way that its embedding matches that of an attacker-chosen targeted input, thereby deceiving downstream models. However, this method often underperforms due to inherent disparities in data from different modalities. In this paper, we introduce CrossFire, an innovative approach to attack multi-modal models. CrossFire begins by transforming the targeted input chosen by the attacker into a format that matches the modality of the original image or audio file. We then formulate our attack as an optimization problem, aiming to minimize the angular deviation between the embeddings of the transformed input and the modified image or audio file. Solving this problem determines the perturbations to be added to the original media. Our extensive experiments on six real-world benchmark datasets reveal that CrossFire can significantly manipulate downstream tasks, surpassing existing attacks. Additionally, we evaluate six defensive strategies against CrossFire, finding that current defenses are insufficient to counteract our CrossFire.
Stratospheric aerosol source inversion: Noise, variability, and uncertainty quantification
Venue: arXiv cs.LG 2024
Year: 2024

Stratospheric aerosols play an important role in the earth system and can affect the climate on timescales of months to years. However, estimating the characteristics of partially observed aerosol injections, such as those from volcanic eruptions, is fraught with uncertainties. This article presents a framework for stratospheric aerosol source inversion which accounts for background aerosol noise and earth system internal variability via a Bayesian approximation error approach. We leverage specially designed earth system model simulations using the Energy Exascale Earth System Model (E3SM). A comprehensive framework for data generation, data processing, dimension reduction, operator learning, and Bayesian inversion is presented where each component of the framework is designed to address particular challenges in stratospheric modeling on the global scale. We present numerical results using synthesized observational data to rigorously assess the ability of our approach to estimate aerosol sources and associate uncertainty with those estimates.
Shadow Removal Refinement via Material-Consistent Shadow Edges
Venue: arXiv cs.CV 2024
Year: 2024

Shadow boundaries can be confused with material boundaries as both exhibit sharp changes in luminance or contrast within a scene. However, shadows do not modify the intrinsic color or texture of surfaces. Therefore, on both sides of shadow edges traversing regions with the same material, the original color and textures should be the same if the shadow is removed properly. These shadow/shadow-free pairs are very useful but hard-to-collect supervision signals. The crucial contribution of this paper is to learn how to identify those shadow edges that traverse material-consistent regions and how to use them as self-supervision for shadow removal refinement during test time. To achieve this, we fine-tune SAM, an image segmentation foundation model, to produce a shadow-invariant segmentation and then extract material-consistent shadow edges by comparing the SAM segmentation with the shadow mask. Utilizing these shadow edges, we introduce color and texture-consistency losses to enhance the shadow removal process. We demonstrate the effectiveness of our method in improving shadow removal results on more challenging, in-the-wild images, outperforming the state-of-the-art shadow removal methods. Additionally, we propose a new metric and an annotated dataset for evaluating the performance of shadow removal methods without the need for paired shadow/shadow-free data.
The existence of stealth corrections in scientific literature -- a threat to scientific integrity
Venue: arXiv cs.DL 2024
Year: 2024

Introduction: Thorough maintenance of the scientific record is needed to ensure the trustworthiness of its content. This can be undermined by a stealth correction, which is at least one post-publication change made to a scientific article, without providing a correction note or any other indicator that the publication was temporarily or permanently altered. In this paper we provide several examples of stealth corrections in order to demonstrate that these exist within the scientific literature. As far as we are aware, no documentation of such stealth corrections was previously reported in the scientific literature. Methods: We identified stealth corrections ourselves, or found already reported ones on the public database pubpeer.com or through social media accounts of known science sleuths. Results: In total we report 131 articles that were affected by stealth corrections and were published between 2005 and 2024. These stealth corrections were found among multiple publishers and scientific fields. Conclusion: and recommendations Stealth corrections exist in the scientific literature. This needs to end immediately as it threatens scientific integrity. We recommend the following: 1) Tracking all changes to the published record by all publishers in an open, uniform and transparent manner, preferably by online submission systems that log every change publicly, making stealth corrections impossible; 2) Clear definitions and guidelines on all types of corrections; 3) Support sustained vigilance of the scientific community to publicly register stealth corrections.
ExIQA: Explainable Image Quality Assessment Using Distortion Attributes
Venue: arXiv cs.CV 2024
Year: 2024

Blind Image Quality Assessment (BIQA) aims to develop methods that estimate the quality scores of images in the absence of a reference image. In this paper, we approach BIQA from a distortion identification perspective, where our primary goal is to predict distortion types and strengths using Vision-Language Models (VLMs), such as CLIP, due to their extensive knowledge and generalizability. Based on these predicted distortions, we then estimate the quality score of the image. To achieve this, we propose an explainable approach for distortion identification based on attribute learning. Instead of prompting VLMs with the names of distortions, we prompt them with the attributes or effects of distortions and aggregate this information to infer the distortion strength. Additionally, we consider multiple distortions per image, making our method more scalable. To support this, we generate a dataset consisting of 100,000 images for efficient training. Finally, attribute probabilities are retrieved and fed into a regressor to predict the image quality score. The results show that our approach, besides its explainability and transparency, achieves state-of-the-art (SOTA) performance across multiple datasets in both PLCC and SRCC metrics. Moreover, the zero-shot results demonstrate the generalizability of the proposed approach.
AssistTaxi: A Comprehensive Dataset for Taxiway Analysis and Autonomous Operations
Venue: arXiv cs.CV 2024
Year: 2024

The availability of high-quality datasets play a crucial role in advancing research and development especially, for safety critical and autonomous systems. In this paper, we present AssistTaxi, a comprehensive novel dataset which is a collection of images for runway and taxiway analysis. The dataset comprises of more than 300,000 frames of diverse and carefully collected data, gathered from Melbourne (MLB) and Grant-Valkaria (X59) general aviation airports. The importance of AssistTaxi lies in its potential to advance autonomous operations, enabling researchers and developers to train and evaluate algorithms for efficient and safe taxiing. Researchers can utilize AssistTaxi to benchmark their algorithms, assess performance, and explore novel approaches for runway and taxiway analysis. Addition-ally, the dataset serves as a valuable resource for validating and enhancing existing algorithms, facilitating innovation in autonomous operations for aviation. We also propose an initial approach to label the dataset using a contour based detection and line extraction technique.
Representation Tuning
Venue: arXiv cs.LG 2024
Year: 2024

Activation engineering is becoming increasingly popular as a means of online control of large language models (LLMs). In this work, we extend the idea of inference-time steering with vectors that represent a behavioral direction of interest to tuning those vectors directly into the model, obviating the need for online control. First, we identify activation vectors related to honesty in an open-source LLM (Llama-2-13b-chat). Next, we demonstrate that model output can be made more or less honest by adding positive or negative multiples of these vectors to residual stream activations during generation. Then, we show that a similar effect can be achieved by fine-tuning the vectors directly into the model, by use of a dual loss function based on the cosine similarity of residual stream activations to the vectors combined with a standard token-based loss ("representation tuning"). Finally, we compare the generations in response to honesty-probing prompts from the resulting models to those from models fine-tuned with a token-based loss alone, and to those from the untuned model subjected to online steering. Overall, fine-tuning the vectors into the models using the cosine similarity plus token loss showed a stronger effect than online steering, and generalized better than using the standard loss, suggesting the potential utility of this approach as a safety measure. Code and data are available at https://github.com/cma1114/representation_tuning. Tuned models are available at https://huggingface.co/collections/cackerman/representation-tuning-66da1e5ab41cd1b824687d9f.
Intrapartum Ultrasound Image Segmentation of Pubic Symphysis and Fetal Head Using Dual Student-Teacher Framework with CNN-ViT Collaborative Learning
Venue: arXiv cs.CV 2024
Year: 2024

The segmentation of the pubic symphysis and fetal head (PSFH) constitutes a pivotal step in monitoring labor progression and identifying potential delivery complications. Despite the advances in deep learning, the lack of annotated medical images hinders the training of segmentation. Traditional semi-supervised learning approaches primarily utilize a unified network model based on Convolutional Neural Networks (CNNs) and apply consistency regularization to mitigate the reliance on extensive annotated data. However, these methods often fall short in capturing the discriminative features of unlabeled data and in delineating the long-range dependencies inherent in the ambiguous boundaries of PSFH within ultrasound images. To address these limitations, we introduce a novel framework, the Dual-Student and Teacher Combining CNN and Transformer (DSTCT), which synergistically integrates the capabilities of CNNs and Transformers. Our framework comprises a Vision Transformer (ViT) as the teacher and two student mod ls one ViT and one CNN. This dual-student setup enables mutual supervision through the generation of both hard and soft pseudo-labels, with the consistency in their predictions being refined by minimizing the classifier determinacy discrepancy. The teacher model further reinforces learning within this architecture through the imposition of consistency regularization constraints. To augment the generalization abilities of our approach, we employ a blend of data and model perturbation techniques. Comprehensive evaluations on the benchmark dataset of the PSFH Segmentation Grand Challenge at MICCAI 2023 demonstrate our DSTCT framework outperformed ten contemporary semi-supervised segmentation methods. Code available at https://github.com/jjm1589/DSTCT.
Pyramid-Monozone Synergistic Grasping Policy in Dense Clutter
Venue: arXiv cs.RO 2024
Year: 2024

Grasping a diverse range of novel objects in dense clutter poses a great challenge to robotic automation mainly due to the occlusion problem. In this work, we propose the Pyramid-Monozone Synergistic Grasping Policy (PMSGP) that enables robots to effectively handle occlusions during grasping. Specifically, we initially construct the Pyramid Sequencing Policy (PSP) to sequence each object in cluttered scenes into a pyramid structure. By isolating objects layer-by-layer, the grasp detection model is allowed to focus on a single layer during each grasp. Then, we devise the Monozone Sampling Policy (MSP) to sample the grasp candidates in the top layer. Through this manner, each grasp targets the topmost object, thereby effectively avoiding most occlusions. We performed more than 7,000 real-world grasping in densely cluttered scenes with 300 novel objects, demonstrating that PMSGP significantly outperforms seven competitive grasping methods. More importantly, we tested the grasping performance of PMSGP in extremely cluttered scenes involving 100 different household goods, and found that PMSGP pushed the grasp success rate to 84.9\%. To the best of our knowledge, no previous work has demonstrated similar performance. All grasping videos are available at: https://www.youtube.com/@chenghaoli4532/playlists.
Operational State Complexity of Block Languages
Venue: arXiv cs.FL 2024
Year: 2024

In this paper we consider block languages, namely sets of words having the same length, and study the deterministic and nondeterministic state complexity of several operations on these languages. Being a subclass of finite languages, the upper bounds of operational state complexity known for finite languages apply for block languages as well. However, in several cases, smaller values were found. Block languages can be represented as bitmaps, which are a good tool to study their minimal finite automata and their operations, as we illustrate here.
Winning Strategies for the Synchronization Game on Subclasses of Finite Automata
Venue: arXiv cs.FL 2024
Year: 2024

We exhibit a winning strategy for Synchronizer in the synchronization game on every synchronizing automaton in whose transition monoid the regular D-classes form subsemigroups
Various Types of Comet Languages and their Application in External Contextual Grammars
Venue: arXiv cs.FL 2024
Year: 2024

In this paper, we continue the research on the power of contextual grammars with selection languages from subfamilies of the family of regular languages. We investigate various comet-like types of languages and compare such language families to some other subregular families of languages (finite, monoidal, nilpotent, combinational, (symmetric) definite, ordered, non-counting, power-separating, suffix-closed, commutative, circular, or union-free languages). Further, we compare the language families defined by these types for the selection with each other and with the families of the hierarchy obtained for external contextual grammars. In this way, we extend the existing hierarchy by new language families.
The VoiceMOS Challenge 2024: Beyond Speech Quality Prediction
Venue: arXiv cs.SD 2024
Year: 2024

We present the third edition of the VoiceMOS Challenge, a scientific initiative designed to advance research into automatic prediction of human speech ratings. There were three tracks. The first track was on predicting the quality of ``zoomed-in'' high-quality samples from speech synthesis systems. The second track was to predict ratings of samples from singing voice synthesis and voice conversion with a large variety of systems, listeners, and languages. The third track was semi-supervised quality prediction for noisy, clean, and enhanced speech, where a very small amount of labeled training data was provided. Among the eight teams from both academia and industry, we found that many were able to outperform the baseline systems. Successful techniques included retrieval-based methods and the use of non-self-supervised representations like spectrograms and pitch histograms. These results showed that the challenge has advanced the field of subjective speech rating prediction.
Situated Visualization in Motion
Venue: arXiv cs.HC 2024
Year: 2024

We contribute a first design space on visualizations in motion and the design of a pilot study we plan to run in the fall. Visualizations can be useful in contexts where either the observation is in motion or the whole visualization is moving at various speeds. Imagine, for example, displays attached to an athlete or animal that show data about the wearer -- for example, captured from a fitness tracking band; or a visualization attached to a moving object such as a vehicle or a soccer ball. The ultimate goal of our research is to inform the design of visualizations under motion.
Enabling Shared-Control for A Riding Ballbot System
Venue: arXiv cs.RO 2024
Year: 2024

This study introduces a shared-control approach for collision avoidance in a self-balancing riding ballbot, called PURE, marked by its dynamic stability, omnidirectional movement, and hands-free interface. Integrated with a sensor array and a novel Passive Artificial Potential Field (PAPF) method, PURE provides intuitive navigation with deceleration assistance and haptic/audio feedback, effectively mitigating collision risks. This approach addresses the limitations of traditional APF methods, such as control oscillations and unnecessary speed reduction in challenging scenarios. A human-robot interaction experiment, with 20 manual wheelchair users and able-bodied individuals, was conducted to evaluate the performance of indoor navigation and obstacle avoidance with the proposed shared-control algorithm. Results indicated that shared-control significantly reduced collisions and cognitive load without affecting travel speed, offering intuitive and safe operation. These findings highlight the shared-control system's suitability for enhancing collision avoidance in self-balancing mobility devices, a relatively unexplored area in assistive mobility research.
Situated Visualization in Motion for Video Games
Venue: arXiv cs.HC 2024
Year: 2024

We contribute a systematic review of situated visualizations in motion in the context of video games. Video games produce rich dynamic datasets during gameplay that are often visualized to help players succeed in a game. Often these visualizations are moving either because they are attached to moving game elements or due to camera changes. We want to understand to what extent this motion and contextual game factors impact how players can read these visualizations. In order to ground our work, we surveyed 160 visualizations in motion and their embeddings in the game world. Here, we report on our analysis and categorization of these visualizations.
Extensions to BIER Tree Engineering (BIER-TE) for Large Multicast Domains and 1:1 Protection: Concept, Implementation and Performance
Venue: arXiv cs.NI 2024
Year: 2024

Bit Index Explicit Replication (BIER) has been proposed by the IETF as a stateless multicast transport technology. BIER adds a BIER header containing a bitstring indicating receivers of an IP multicast (IPMC) packet within a BIER domain. BIER-TE extends BIER with tree engineering capabilities, i.e., the bitstring indicates both receivers as well as links over which the packet is transmitted. As the bitstring is of limited size, e.g., 256 bits, only that number of receivers can be addressed within a BIER packet. To scale BIER to larger networks, the receivers of a BIER domain have been assigned to subsets that can be addressed by a bitstring with a subset ID. This approach is even compliant with fast reroute (FRR) mechanisms for BIER. In this work we tackle the challenge of scaling BIER-TE to large networks as the subset mechanism of BIER is not sufficient for that purpose. A major challenge is the support of a protection mechanism in this context. We describe how existing networking concepts like tunneling, egress protection and BIER-TE-FRR can be combined to achieve the goal. Then, we implement the relevant BIER-TE components on the P4-programmable Tofino ASIC which builds upon an existing implementation for BIER. Finally, we consider the forwarding performance of the prototype and explain how weaknesses can be improved from remedies that are well-known for BIER implementations.
RSVP for VPSA : A Meta Design Study on Rapid Suggestive Visualization Prototyping for Visual Parameter Space Analysis
Venue: arXiv cs.HC 2024
Year: 2024

Visual Parameter Space Analysis (VPSA) enables domain scientists to explore input-output relationships of computational models. Existing VPSA applications often feature multi-view visualizations designed by visualization experts for a specific scenario, making it hard for domain scientists to adapt them to their problems without professional help. We present RSVP, the Rapid Suggestive Visualization Prototyping system encoding VPSA knowledge to enable domain scientists to prototype custom visualization dashboards tailored to their specific needs. The system implements a task-oriented, multi-view visualization recommendation strategy over a visualization design space optimized for VPSA to guide users in meeting their analytical demands. We derived the VPSA knowledge implemented in the system by conducting an extensive meta design study over the body of work on VPSA. We show how this process can be used to perform a data and task abstraction, extract a common visualization design space, and derive a task-oriented VisRec strategy. User studies indicate that the system is user-friendly and can uncover novel insights.
Cross-Refine: Improving Natural Language Explanation Generation by Learning in Tandem
Venue: arXiv cs.CL 2024
Year: 2024

Natural language explanations (NLEs) are vital for elucidating the reasoning behind large language model (LLM) decisions. Many techniques have been developed to generate NLEs using LLMs. However, like humans, LLMs might not always produce optimal NLEs on first attempt. Inspired by human learning processes, we introduce Cross-Refine, which employs role modeling by deploying two LLMs as generator and critic, respectively. The generator outputs a first NLE and then refines this initial explanation using feedback and suggestions provided by the critic. Cross-Refine does not require any supervised training data or additional training. We validate Cross-Refine across three NLP tasks using three state-of-the-art open-source LLMs through automatic and human evaluation. We select Self-Refine (Madaan et al., 2023) as the baseline, which only utilizes self-feedback to refine the explanations. Our findings from automatic evaluation and a user study indicate that Cross-Refine outperforms Self-Refine. Meanwhile, Cross-Refine can perform effectively with less powerful LLMs, whereas Self-Refine only yields strong results with ChatGPT. Additionally, we conduct an ablation study to assess the importance of feedback and suggestions. Both of them play an important role in refining explanations. We further evaluate Cross-Refine on a bilingual dataset in English and German.