OpenRaiser/Intern-Atlas huggingface dataset:
CoSAM: Self-Correcting SAM for Domain Generalization in 2D Medical Image Segmentation
Venue: arXiv cs.CV 2024
Year: 2024

Medical images often exhibit distribution shifts due to variations in imaging protocols and scanners across different medical centers. Domain Generalization (DG) methods aim to train models on source domains that can generalize to unseen target domains. Recently, the segment anything model (SAM) has demonstrated strong generalization capabilities due to its prompt-based design, and has gained significant attention in image segmentation tasks. Existing SAM-based approaches attempt to address the need for manual prompts by introducing prompt generators that automatically generate these prompts. However, we argue that auto-generated prompts may not be sufficiently accurate under distribution shifts, potentially leading to incorrect predictions that still require manual verification and correction by clinicians. To address this challenge, we propose a method for 2D medical image segmentation called Self-Correcting SAM (CoSAM). Our approach begins by generating coarse masks using SAM in a prompt-free manner, providing prior prompts for the subsequent stages, and eliminating the need for prompt generators. To automatically refine these coarse masks, we introduce a generalized error decoder that simulates the correction process typically performed by clinicians. Furthermore, we generate diverse prompts as feedback based on the corrected masks, which are used to iteratively refine the predictions within a self-correcting loop, enhancing the generalization performance of our model. Extensive experiments on two medical image segmentation benchmarks across multiple scenarios demonstrate the superiority of CoSAM over state-of-the-art SAM-based methods.
Outliers resistant image classification by anomaly detection
Venue: arXiv cs.CV 2024
Year: 2024

Various technologies, including computer vision models, are employed for the automatic monitoring of manual assembly processes in production. These models detect and classify events such as the presence of components in an assembly area or the connection of components. A major challenge with detection and classification algorithms is their susceptibility to variations in environmental conditions and unpredictable behavior when processing objects that are not included in the training dataset. As it is impractical to add all possible subjects in the training sample, an alternative solution is necessary. This study proposes a model that simultaneously performs classification and anomaly detection, employing metric learning to generate vector representations of images in a multidimensional space, followed by classification using cross-entropy. For experimentation, a dataset of over 327,000 images was prepared. Experiments were conducted with various computer vision model architectures, and the outcomes of each approach were compared.
SEAGULL: No-reference Image Quality Assessment for Regions of Interest via Vision-Language Instruction Tuning
Venue: arXiv cs.CV 2024
Year: 2024

Existing Image Quality Assessment (IQA) methods achieve remarkable success in analyzing quality for overall image, but few works explore quality analysis for Regions of Interest (ROIs). The quality analysis of ROIs can provide fine-grained guidance for image quality improvement and is crucial for scenarios focusing on region-level quality. This paper proposes a novel network, SEAGULL, which can SEe and Assess ROIs quality with GUidance from a Large vision-Language model. SEAGULL incorporates a vision-language model (VLM), masks generated by Segment Anything Model (SAM) to specify ROIs, and a meticulously designed Mask-based Feature Extractor (MFE) to extract global and local tokens for specified ROIs, enabling accurate fine-grained IQA for ROIs. Moreover, this paper constructs two ROI-based IQA datasets, SEAGULL-100w and SEAGULL-3k, for training and evaluating ROI-based IQA. SEAGULL-100w comprises about 100w synthetic distortion images with 33 million ROIs for pre-training to improve the model's ability of regional quality perception, and SEAGULL-3k contains about 3k authentic distortion ROIs to enhance the model's ability to perceive real world distortions. After pre-training on SEAGULL-100w and fine-tuning on SEAGULL-3k, SEAGULL shows remarkable performance on fine-grained ROI quality assessment. Code and datasets are publicly available at the https://github.com/chencn2020/Seagull.
Evaluating Text-to-Image Diffusion Models for Texturing Synthetic Data
Venue: arXiv cs.RO 2024
Year: 2024

Building generic robotic manipulation systems often requires large amounts of real-world data, which can be dificult to collect. Synthetic data generation offers a promising alternative, but limiting the sim-to-real gap requires significant engineering efforts. To reduce this engineering effort, we investigate the use of pretrained text-to-image diffusion models for texturing synthetic images and compare this approach with using random textures, a common domain randomization technique in synthetic data generation. We focus on generating object-centric representations, such as keypoints and segmentation masks, which are important for robotic manipulation and require precise annotations. We evaluate the efficacy of the texturing methods by training models on the synthetic data and measuring their performance on real-world datasets for three object categories: shoes, T-shirts, and mugs. Surprisingly, we find that texturing using a diffusion model performs on par with random textures, despite generating seemingly more realistic images. Our results suggest that, for now, using diffusion models for texturing does not benefit synthetic data generation for robotics. The code, data and trained models are available at \url{https://github.com/tlpss/diffusing-synthetic-data.git}.
Try-On-Adapter: A Simple and Flexible Try-On Paradigm
Venue: arXiv cs.CV 2024
Year: 2024

Image-based virtual try-on, widely used in online shopping, aims to generate images of a naturally dressed person conditioned on certain garments, providing significant research and commercial potential. A key challenge of try-on is to generate realistic images of the model wearing the garments while preserving the details of the garments. Previous methods focus on masking certain parts of the original model's standing image, and then inpainting on masked areas to generate realistic images of the model wearing corresponding reference garments, which treat the try-on task as an inpainting task. However, such implements require the user to provide a complete, high-quality standing image, which is user-unfriendly in practical applications. In this paper, we propose Try-On-Adapter (TOA), an outpainting paradigm that differs from the existing inpainting paradigm. Our TOA can preserve the given face and garment, naturally imagine the rest parts of the image, and provide flexible control ability with various conditions, e.g., garment properties and human pose. In the experiments, TOA shows excellent performance on the virtual try-on task even given relatively low-quality face and garment images in qualitative comparisons. Additionally, TOA achieves the state-of-the-art performance of FID scores 5.56 and 7.23 for paired and unpaired on the VITON-HD dataset in quantitative comparisons.
Reachability Analysis of the Domain Name System
Venue: arXiv cs.CR 2024
Year: 2024

The high complexity of DNS poses unique challenges for ensuring its security and reliability. Despite continuous advances in DNS testing, monitoring, and verification, protocol-level defects still give rise to numerous bugs and attacks. In this paper, we provide the first decision procedure for the DNS verification problem, establishing its complexity as $\mathsf{2ExpTime}$, which was previously unknown. We begin by formalizing the semantics of DNS as a system of recursive communicating processes extended with timers and an infinite message alphabet. We provide an algebraic abstraction of the alphabet with finitely many equivalence classes, using the subclass of semigroups that recognize positive prefix-testable languages. We then introduce a novel generalization of bisimulation for labelled transition systems, weaker than strong bisimulation, to show that our abstraction is sound and complete. Finally, using this abstraction, we reduce the DNS verification problem to the verification problem for pushdown systems. To show the expressiveness of our framework, we model two of the most prominent attack vectors on DNS, namely amplification attacks and rewrite blackholing.
Tangi: a Tool to Create Tangible Artifacts for Sharing Insights from 360$^\circ$ Video
Venue: arXiv cs.HC 2024
Year: 2024

Designers often engage with video to gain rich, temporal insights about the context of users, collaboratively analyzing it to gather ideas, challenge assumptions, and foster empathy. To capture the full visual context of users and their situations, designers are adopting 360$^\circ$ video, providing richer, more multi-layered insights. Unfortunately, the spherical nature of 360$^\circ$ video means designers cannot create tangible video artifacts such as storyboards for collaborative analysis. To overcome this limitation, we created Tangi, a web-based tool that converts 360$^\circ$ images into tangible 360$^\circ$ video artifacts, that enable designers to embody and share their insights. Our evaluation with nine experienced designers demonstrates that the artifacts Tangi creates enable tangible interactions found in collaborative workshops and introduce two new capabilities: spatial orientation within 360$^\circ$ environments and linking specific details to the broader 360$^\circ$ context. Since Tangi is an open-source tool, designers can immediately leverage 360$^\circ$ video in collaborative workshops.
Morpho-Aware Global Attention for Image Matting
Venue: arXiv cs.CV 2024
Year: 2024

Vision Transformers (ViTs) and Convolutional Neural Networks (CNNs) face inherent challenges in image matting, particularly in preserving fine structural details. ViTs, with their global receptive field enabled by the self-attention mechanism, often lose local details such as hair strands. Conversely, CNNs, constrained by their local receptive field, rely on deeper layers to approximate global context but struggle to retain fine structures at greater depths. To overcome these limitations, we propose a novel Morpho-Aware Global Attention (MAGA) mechanism, designed to effectively capture the morphology of fine structures. MAGA employs Tetris-like convolutional patterns to align the local shapes of fine structures, ensuring optimal local correspondence while maintaining sensitivity to morphological details. The extracted local morphology information is used as query embeddings, which are projected onto global key embeddings to emphasize local details in a broader context. Subsequently, by projecting onto value embeddings, MAGA seamlessly integrates these emphasized morphological details into a unified global structure. This approach enables MAGA to simultaneously focus on local morphology and unify these details into a coherent whole, effectively preserving fine structures. Extensive experiments show that our MAGA-based ViT achieves significant performance gains, outperforming state-of-the-art methods across two benchmarks with average improvements of 4.3% in SAD and 39.5% in MSE.
Fill in the blanks: Rethinking Interpretability in vision
Venue: arXiv cs.CV 2024
Year: 2024

Model interpretability is a key challenge that has yet to align with the advancements observed in contemporary state-of-the-art deep learning models. In particular, deep learning aided vision tasks require interpretability, in order for their adoption in more specialized domains such as medical imaging. Although the field of explainable AI (XAI) developed methods for interpreting vision models along with early convolutional neural networks, recent XAI research has mainly focused on assigning attributes via saliency maps. As such, these methods are restricted to providing explanations at a sample level, and many explainability methods suffer from low adaptability across a wide range of vision models. In our work, we re-think vision-model explainability from a novel perspective, to probe the general input structure that a model has learnt during its training. To this end, we ask the question: "How would a vision model fill-in a masked-image". Experiments on standard vision datasets and pre-trained models reveal consistent patterns, and could be intergrated as an additional model-agnostic explainability tool in modern machine-learning platforms. The code will be available at \url{https://github.com/BoTZ-TND/FillingTheBlanks.git}
Moving Forward: A Review of Autonomous Driving Software and Hardware Systems
Venue: arXiv cs.RO 2024
Year: 2024

With their potential to significantly reduce traffic accidents, enhance road safety, optimize traffic flow, and decrease congestion, autonomous driving systems are a major focus of research and development in recent years. Beyond these immediate benefits, they offer long-term advantages in promoting sustainable transportation by reducing emissions and fuel consumption. Achieving a high level of autonomy across diverse conditions requires a comprehensive understanding of the environment. This is accomplished by processing data from sensors such as cameras, radars, and LiDARs through a software stack that relies heavily on machine learning algorithms. These ML models demand significant computational resources and involve large-scale data movement, presenting challenges for hardware to execute them efficiently and at high speed. In this survey, we first outline and highlight the key components of self-driving systems, covering input sensors, commonly used datasets, simulation platforms, and the software architecture. We then explore the underlying hardware platforms that support the execution of these software systems. By presenting a comprehensive view of autonomous driving systems and their increasing demands, particularly for higher levels of autonomy, we analyze the performance and efficiency of scaled-up off-the-shelf GPU/CPU-based systems, emphasizing the challenges within the computational components. Through examples showcasing the diverse computational and memory requirements in the software stack, we demonstrate how more specialized hardware and processing closer to memory can enable more efficient execution with lower latency. Finally, based on current trends and future demands, we conclude by speculating what a future hardware platform for autonomous driving might look like.
Probabilistic Prior Driven Attention Mechanism Based on Diffusion Model for Imaging Through Atmospheric Turbulence
Venue: arXiv cs.CV 2024
Year: 2024

Atmospheric turbulence introduces severe spatial and geometric distortions, challenging traditional image restoration methods. We propose the Probabilistic Prior Turbulence Removal Network (PPTRN), which combines probabilistic diffusion-based prior modeling with Transformer-driven feature extraction to address this issue. PPTRN employs a two-stage approach: first, a latent encoder and Transformer are jointly trained on clear images to establish robust feature representations. Then, a Denoising Diffusion Probabilistic Model (DDPM) models prior distributions over latent vectors, guiding the Transformer in capturing diverse feature variations essential for restoration. A key innovation in PPTRN is the Probabilistic Prior Driven Cross Attention mechanism, which integrates the DDPM-generated prior with feature embeddings to reduce artifacts and enhance spatial coherence. Extensive experiments validate that PPTRN significantly improves restoration quality on turbulence-degraded images, setting a new benchmark in clarity and structural fidelity.
Towards Automatic Evaluation of Task-Oriented Dialogue Flows
Venue: arXiv cs.CL 2024
Year: 2024

Task-oriented dialogue systems rely on predefined conversation schemes (dialogue flows) often represented as directed acyclic graphs. These flows can be manually designed or automatically generated from previously recorded conversations. Due to variations in domain expertise or reliance on different sets of prior conversations, these dialogue flows can manifest in significantly different graph structures. Despite their importance, there is no standard method for evaluating the quality of dialogue flows. We introduce FuDGE (Fuzzy Dialogue-Graph Edit Distance), a novel metric that evaluates dialogue flows by assessing their structural complexity and representational coverage of the conversation data. FuDGE measures how well individual conversations align with a flow and, consequently, how well a set of conversations is represented by the flow overall. Through extensive experiments on manually configured flows and flows generated by automated techniques, we demonstrate the effectiveness of FuDGE and its evaluation framework. By standardizing and optimizing dialogue flows, FuDGE enables conversational designers and automated techniques to achieve higher levels of efficiency and automation.
Towards Geometry-Preserving Reductions Between Constraint Satisfaction Problems (and other problems in NP)
Venue: arXiv cs.CC 2024
Year: 2024

Motivated by phase transitions in combinatorial optimization problems, we define two kinds of geometry-preserving reductions between constraint satisfaction problems and other NP-search problems. We give a couple of examples and counterexamples for these reductions.
Biotic Browser: Applying StreamingLLM as a Persistent Web Browsing Co-Pilot
Venue: arXiv cs.HC 2024
Year: 2024

This paper presents "Biotic Browser," an innovative AI assistant leveraging StreamingLLM to transform web navigation and task execution. Characterized by its ability to simulate the experience of a passenger in an autonomous vehicle, the Biotic Browser excels in managing extended interactions and complex, multi-step web-based tasks. It marks a significant advancement in AI technology, particularly in the realm of long-term context management, and offers promising applications for enhancing productivity and efficiency in both personal and professional settings.
Reply to: Limitations in odour recognition and generalisation in a neuromorphic olfactory circuit
Venue: arXiv cs.NE 2024
Year: 2024

Dennler et al. submit that they have discovered limitations affecting some of the conclusions drawn in our 2020 paper, Rapid online learning and robust recall in a neuromorphic olfactory circuit. Specifically, they assert (1) that the public dataset we used suffers from sensor drift and a nonrandomized measurement protocol, (2) that our neuromorphic EPL network is limited in its ability to generalize over repeated presentations of an odorant, and (3) that our EPL network results can be performance matched by using a more computationally efficient distance measure. Though they are correct in their description of the limitations of that public dataset, they do not acknowledge in their first two assertions how our utilization of those data sidestepped these limitations. Their third claim arises from flaws in the method used to generate their distance measure. We respond below to each of these three claims in turn.
Detecting Student Disengagement in Online Classes Using Deep Learning: A Review
Venue: arXiv cs.HC 2024
Year: 2024

Student disengagement in online learning has become a critical challenge, particularly post-pandemic. This review explores deep learning techniques used to detect disengagement, emphasizing computer vision and affective computing as effective approaches. We examine recent studies focusing on facial expressions, eye movements, and posture to assess student attention, along with non-face-based indicators like mouse activity. A systematic review of 38 selected studies outlines the indicators, methods, and models employed in this field, providing insights for future research on real-time engagement monitoring in online classrooms
Object-Centric Local Process Models
Venue: arXiv cs.DB 2024
Year: 2024

Process mining is a technology that helps understand, analyze, and improve processes. It has been present for around two decades, and although initially tailored for business processes, the spectrum of analyzed processes nowadays is evermore growing. To support more complex and diverse processes, subdisciplines such as object-centric process mining and behavioral pattern mining have emerged. Behavioral patterns allow for analyzing parts of the process in isolation, while object-centric process mining enables combining different perspectives of the process. In this work, we introduce \emph{Object-Centric Local Process Models} (OCLPMs). OCLPMs are behavioral patterns tailored to analyzing complex processes where no single case notion exists and we leverage object-centric Petri nets to model them. Additionally, we present a discovery algorithm that starts from object-centric event logs, and implement the proposed approach in the open-source framework ProM. Finally, we demonstrate the applicability of OCLPMs in two case studies and evaluate the approach on various event logs.
Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Survey
Venue: arXiv cs.LG 2024
Year: 2024

Building effective machine learning (ML) workflows to address complex tasks is a primary focus of the Automatic ML (AutoML) community and a critical step toward achieving artificial general intelligence (AGI). Recently, the integration of Large Language Models (LLMs) into ML workflows has shown great potential for automating and enhancing various stages of the ML pipeline. This survey provides a comprehensive and up-to-date review of recent advancements in using LLMs to construct and optimize ML workflows, focusing on key components encompassing data and feature engineering, model selection and hyperparameter optimization, and workflow evaluation. We discuss both the advantages and limitations of LLM-driven approaches, emphasizing their capacity to streamline and enhance ML workflow modeling process through language understanding, reasoning, interaction, and generation. Finally, we highlight open challenges and propose future research directions to advance the effective application of LLMs in ML workflows.
The Noisy Work of Uncertainty Visualisation Research: A Review
Venue: arXiv cs.HC 2024
Year: 2024

Uncertainty visualisation is quickly becomming a hot topic in information visualisation. Exisiting reviews in the field take the definition and purpose of an uncertainty visualisation to be self evident which results in a large amout of conflicting information. This conflict largely stems from a conflation between uncertainty visualisations designed for decision making and those designed to prevent false conclusions. We coin the term "signal suppression" to describe a visualisation that is designed for preventing false conclusions, as the approach demands that the signal (i.e. the collective take away of the estimates) is suppressed by the noise (i.e. the variance on those estimates). We argue that the current standards in visualisation suggest that uncertainty visualisations designed for decision making should not be considered uncertainty visualisations at all. Therefore, future work should focus on signal suppression. Effective signal suppression requires us to communicate the signal and the noise as a single "validity of signal" variable, and doing so proves to be difficult with current methods. We illustrate current approaches to uncertainty visualisation by showing how they would change the visual apprearance of a choropleth map. These maps allow us to see why some methods succeed at signal suppression, while others fall short. Evaluating visualisations on how well they perform signal suppression also proves to be difficult, as it involves measuring the effect of noise, a variable we typically try to ignore. We suggest authors use qualitative studies or compare uncertainty visualisations to the relevant hypothesis tests.