OpenRaiser/Intern-Atlas huggingface dataset:
Unlearning or Concealment? A Critical Analysis and Evaluation Metrics for Unlearning in Diffusion Models
Venue: arXiv cs.LG 2024
Year: 2024

Recent research has seen significant interest in methods for concept removal and targeted forgetting in text-to-image diffusion models. In this paper, we conduct a comprehensive white-box analysis showing the vulnerabilities in existing diffusion model unlearning methods. We show that existing unlearning methods lead to decoupling of the targeted concepts (meant to be forgotten) for the corresponding prompts. This is concealment and not actual forgetting, which was the original goal. This paper presents a rigorous theoretical and empirical examination of five commonly used techniques for unlearning in diffusion models, while showing their potential weaknesses. We introduce two new evaluation metrics: Concept Retrieval Score (\textbf{CRS}) and Concept Confidence Score (\textbf{CCS}). These metrics are based on a successful adversarial attack setup that can recover \textit{forgotten} concepts from unlearned diffusion models. \textbf{CRS} measures the similarity between the latent representations of the unlearned and fully trained models after unlearning. It reports the extent of retrieval of the \textit{forgotten} concepts with increasing amount of guidance. CCS quantifies the confidence of the model in assigning the target concept to the manipulated data. It reports the probability of the \textit{unlearned} model's generations to be aligned with the original domain knowledge with increasing amount of guidance. The \textbf{CCS} and \textbf{CRS} enable a more robust evaluation of concept erasure methods. Evaluating existing five state-of-the-art methods with our metrics, reveal significant shortcomings in their ability to truly \textit{unlearn}. Source Code: \color{blue}{https://respailab.github.io/unlearning-or-concealment}
Segmentation by Factorization: Unsupervised Semantic Segmentation for Pathology by Factorizing Foundation Model Features
Venue: arXiv cs.CV 2024
Year: 2024

We introduce Segmentation by Factorization (F-SEG), an unsupervised segmentation method for pathology that generates segmentation masks from pre-trained deep learning models. F-SEG allows the use of pre-trained deep neural networks, including recently developed pathology foundation models, for semantic segmentation. It achieves this without requiring additional training or finetuning, by factorizing the spatial features extracted by the models into segmentation masks and their associated concept features. We create generic tissue phenotypes for H&E images by training clustering models for multiple numbers of clusters on features extracted from several deep learning models on The Cancer Genome Atlas Program (TCGA), and then show how the clusters can be used for factorizing corresponding segmentation masks using off-the-shelf deep learning models. Our results show that F-SEG provides robust unsupervised segmentation capabilities for H&E pathology images, and that the segmentation quality is greatly improved by utilizing pathology foundation models. We discuss and propose methods for evaluating the performance of unsupervised segmentation in pathology.
The Influence of Task and Group Disparities over Users' Attitudes Toward Using Large Language Models for Psychotherapy
Venue: arXiv cs.HC 2024
Year: 2024

The population suffering from mental health disorders has kept increasing in recent years. With the advancements in large language models (LLMs) in diverse fields, LLM-based psychotherapy has also attracted increasingly more attention. However, the factors influencing users' attitudes to LLM-based psychotherapy have rarely been explored. As the first attempt, this paper investigated the influence of task and group disparities on user attitudes toward LLM-based psychotherapy tools. Utilizing the Technology Acceptance Model (TAM) and Automation Acceptance Model (AAM), based on an online survey, we collected and analyzed responses from 222 LLM-based psychotherapy users in mainland China. The results revealed that group disparity (i.e., mental health conditions) can influence users' attitudes toward LLM tools. Further, one of the typical task disparities, i.e., the privacy concern, was not found to have a significant effect on trust and usage intention. These findings can guide the design of future LLM-based psychotherapy services.
Referring Expression Generation in Visually Grounded Dialogue with Discourse-aware Comprehension Guiding
Venue: arXiv cs.CL 2024
Year: 2024

We propose an approach to referring expression generation (REG) in visually grounded dialogue that is meant to produce referring expressions (REs) that are both discriminative and discourse-appropriate. Our method constitutes a two-stage process. First, we model REG as a text- and image-conditioned next-token prediction task. REs are autoregressively generated based on their preceding linguistic context and a visual representation of the referent. Second, we propose the use of discourse-aware comprehension guiding as part of a generate-and-rerank strategy through which candidate REs generated with our REG model are reranked based on their discourse-dependent discriminatory power. Results from our human evaluation indicate that our proposed two-stage approach is effective in producing discriminative REs, with higher performance in terms of text-image retrieval accuracy for reranked REs compared to those generated using greedy decoding.
ReL-SAR: Representation Learning for Skeleton Action Recognition with Convolutional Transformers and BYOL
Venue: arXiv cs.CV 2024
Year: 2024

To extract robust and generalizable skeleton action recognition features, large amounts of well-curated data are typically required, which is a challenging task hindered by annotation and computation costs. Therefore, unsupervised representation learning is of prime importance to leverage unlabeled skeleton data. In this work, we investigate unsupervised representation learning for skeleton action recognition. For this purpose, we designed a lightweight convolutional transformer framework, named ReL-SAR, exploiting the complementarity of convolutional and attention layers for jointly modeling spatial and temporal cues in skeleton sequences. We also use a Selection-Permutation strategy for skeleton joints to ensure more informative descriptions from skeletal data. Finally, we capitalize on Bootstrap Your Own Latent (BYOL) to learn robust representations from unlabeled skeleton sequence data. We achieved very competitive results on limited-size datasets: MCAD, IXMAS, JHMDB, and NW-UCLA, showing the effectiveness of our proposed method against state-of-the-art methods in terms of both performance and computational efficiency. To ensure reproducibility and reusability, the source code including all implementation parameters is provided at: https://github.com/SafwenNaimi/Representation-Learning-for-Skeleton-Action-Recognition-with-Convolutional-Transformers-and-BYOL
Creativity and Visual Communication from Machine to Musician: Sharing a Score through a Robotic Camera
Venue: arXiv cs.HC 2024
Year: 2024

This paper explores the integration of visual communication and musical interaction by implementing a robotic camera within a "Guided Harmony" musical game. We aim to examine co-creative behaviors between human musicians and robotic systems. Our research explores existing methodologies like improvisational game pieces and extends these concepts to include robotic participation using a PTZ camera. The robotic system interprets and responds to nonverbal cues from musicians, creating a collaborative and adaptive musical experience. This initial case study underscores the importance of intuitive visual communication channels. We also propose future research directions, including parameters for refining the visual cue toolkit and data collection methods to understand human-machine co-creativity further. Our findings contribute to the broader understanding of machine intelligence in augmenting human creativity, particularly in musical settings.
LSVOS Challenge Report: Large-scale Complex and Long Video Object Segmentation
Venue: arXiv cs.CV 2024
Year: 2024

Despite the promising performance of current video segmentation models on existing benchmarks, these models still struggle with complex scenes. In this paper, we introduce the 6th Large-scale Video Object Segmentation (LSVOS) challenge in conjunction with ECCV 2024 workshop. This year's challenge includes two tasks: Video Object Segmentation (VOS) and Referring Video Object Segmentation (RVOS). In this year, we replace the classic YouTube-VOS and YouTube-RVOS benchmark with latest datasets MOSE, LVOS, and MeViS to assess VOS under more challenging complex environments. This year's challenge attracted 129 registered teams from more than 20 institutes across over 8 countries. This report include the challenge and dataset introduction, and the methods used by top 7 teams in two tracks. More details can be found in our homepage https://lsvos.github.io/.
Transformer-Enhanced Iterative Feedback Mechanism for Polyp Segmentation
Venue: arXiv cs.CV 2024
Year: 2024

Colorectal cancer (CRC) is the third most common cause of cancer diagnosed in the United States and the second leading cause of cancer-related death among both genders. Notably, CRC is the leading cause of cancer in younger men less than 50 years old. Colonoscopy is considered the gold standard for the early diagnosis of CRC. Skills vary significantly among endoscopists, and a high miss rate is reported. Automated polyp segmentation can reduce the missed rates, and timely treatment is possible in the early stage. To address this challenge, we introduce \textit{\textbf{\ac{FANetv2}}}, an advanced encoder-decoder network designed to accurately segment polyps from colonoscopy images. Leveraging an initial input mask generated by Otsu thresholding, FANetv2 iteratively refines its binary segmentation masks through a novel feedback attention mechanism informed by the mask predictions of previous epochs. Additionally, it employs a text-guided approach that integrates essential information about the number (one or many) and size (small, medium, large) of polyps to further enhance its feature representation capabilities. This dual-task approach facilitates accurate polyp segmentation and aids in the auxiliary classification of polyp attributes, significantly boosting the model's performance. Our comprehensive evaluations on the publicly available BKAI-IGH and CVC-ClinicDB datasets demonstrate the superior performance of FANetv2, evidenced by high dice similarity coefficients (DSC) of 0.9186 and 0.9481, along with low Hausdorff distances of 2.83 and 3.19, respectively. The source code for FANetv2 is available at https://github.com/xxxxx/FANetv2.
Bridging Research and Practice Through Conversation: Reflecting on Our Experience
Venue: arXiv cs.HC 2024
Year: 2024

While some research fields have a long history of collaborating with domain experts outside academia, many quantitative researchers do not have natural avenues to meet experts in areas where the research is later deployed. We explain how conversations -- interviews without a specific research objective -- can bridge research and practice. Using collaborative autoethnography, we reflect on our experience of conducting conversations with practitioners from a range of different backgrounds, including refugee rights, conservation, addiction counseling, and municipal data science. Despite these varied backgrounds, common lessons emerged, including the importance of valuing the knowledge of experts, recognizing that academic research and practice have differing objectives and timelines, understanding the limits of quantification, and avoiding data extractivism. We consider the impact of these conversations on our work, the potential roles we can serve as researchers, and the challenges we anticipate as we move forward in these collaborations.
Continuous Learning of Transformer-based Audio Deepfake Detection
Venue: arXiv cs.SD 2024
Year: 2024

This paper proposes a novel framework for audio deepfake detection with two main objectives: i) attaining the highest possible accuracy on available fake data, and ii) effectively performing continuous learning on new fake data in a few-shot learning manner. Specifically, we conduct a large audio deepfake collection using various deep audio generation methods. The data is further enhanced with additional augmentation methods to increase variations amidst compressions, far-field recordings, noise, and other distortions. We then adopt the Audio Spectrogram Transformer for the audio deepfake detection model. Accordingly, the proposed method achieves promising performance on various benchmark datasets. Furthermore, we present a continuous learning plugin module to update the trained model most effectively with the fewest possible labeled data points of the new fake type. The proposed method outperforms the conventional direct fine-tuning approach with much fewer labeled data points.
A Comprehensive Comparison Between ANNs and KANs For Classifying EEG Alzheimer's Data
Venue: arXiv cs.NE 2024
Year: 2024

Alzheimer's Disease is an incurable cognitive condition that affects thousands of people globally. While some diagnostic methods exist for Alzheimer's Disease, many of these methods cannot detect Alzheimer's in its earlier stages. Recently, researchers have explored the use of Electroencephalogram (EEG) technology for diagnosing Alzheimer's. EEG is a noninvasive method of recording the brain's electrical signals, and EEG data has shown distinct differences between patients with and without Alzheimer's. In the past, Artificial Neural Networks (ANNs) have been used to predict Alzheimer's from EEG data, but these models sometimes produce false positive diagnoses. This study aims to compare losses between ANNs and Kolmogorov-Arnold Networks (KANs) across multiple types of epochs, learning rates, and nodes. The results show that across these different parameters, ANNs are more accurate in predicting Alzheimer's Disease from EEG signals.
OciorCOOL: Faster Byzantine Agreement and Reliable Broadcast
Venue: arXiv cs.DC 2024
Year: 2024

COOL (Chen'21) is an error-free and deterministic Byzantine agreement protocol that achieves consensus on an $\ell$-bit message with a communication complexity of $O(\max\{n\ell, n t \log t \})$ bits in four phases, given $n\geq 3t + 1$, for a network of $n$ nodes, where up to $t$ nodes may be dishonest. In this work we show that COOL can be optimized by reducing one communication round. The new protocol is called OciorCOOL. Additionally, building on OciorCOOL, we design an optimal reliable broadcast protocol that requires only six communication rounds.
MemoVis: A GenAI-Powered Tool for Creating Companion Reference Images for 3D Design Feedback
Venue: arXiv cs.HC 2024
Year: 2024

Providing asynchronous feedback is a critical step in the 3D design workflow. A common approach to providing feedback is to pair textual comments with companion reference images, which helps illustrate the gist of text. Ideally, feedback providers should possess 3D and image editing skills to create reference images that can effectively describe what they have in mind. However, they often lack such skills, so they have to resort to sketches or online images which might not match well with the current 3D design. To address this, we introduce MemoVis, a text editor interface that assists feedback providers in creating reference images with generative AI driven by the feedback comments. First, a novel real-time viewpoint suggestion feature, based on a vision-language foundation model, helps feedback providers anchor a comment with a camera viewpoint. Second, given a camera viewpoint, we introduce three types of image modifiers, based on pre-trained 2D generative models, to turn a text comment into an updated version of the 3D scene from that viewpoint. We conducted a within-subjects study with feedback providers, demonstrating the effectiveness of MemoVis. The quality and explicitness of the companion images were evaluated by another eight participants with prior 3D design experience.
Compute-Forward Multiple Access for Gaussian MIMO Channels
Venue: arXiv cs.IT 2024
Year: 2024

Compute-forward multiple access (CFMA) is a multiple access transmission scheme based on Compute-and-Forward (CF) which allows the receiver to first decode linear combinations of the transmitted signals and then solve for individual messages. This paper extends the CFMA scheme to a two-user Gaussian multiple-input multiple-output (MIMO) multiple access channel (MAC). We propose the CFMA serial coding scheme (SCS) and the CFMA parallel coding scheme (PCS) with nested lattice codes. We first derive the expression of the achievable rate pair for MIMO MAC with CFMA-SCS. We prove a general condition under which CFMA-SCS can achieve the sum capacity of the channel. Furthermore, this result is specialized to single-input multiple-output (SIMO) and $2$-by-$2$ diagonal MIMO multiple access channels, for which more explicit sum capacity-achieving conditions on power and channel matrices are derived. We construct an equivalent SIMO model for CFMA-PCS and also derive the achievable rates. Its sum capacity achieving conditions are then analysed. Numerical results are provided for the performance of CFMA-SCS and CFMA-PCS in different channel conditions. In general, CFMA-PCS has better sum capacity achievability with higher coding complexity.
Human Impedance Modulation to Improve Visuo-Haptic Perception
Venue: arXiv cs.HC 2024
Year: 2024

Humans activate muscles to shape the mechanical interaction with their environment, but can they harness this control mechanism to best sense the environment? We investigated how participants adapt their muscle activation to visual and haptic information when tracking a randomly moving target with a robotic interface. The results exhibit a differentiated effect of these sensory modalities, where participants' muscle cocontraction increases with the haptic noise and decreases with the visual noise, in apparent contradiction to previous results. These results can be explained, and reconciled with previous findings, when considering muscle spring like mechanics, where stiffness increases with cocontraction to regulate motion guidance. Increasing cocontraction to more closely follow the motion plan favors accurate visual over haptic information, while decreasing it avoids injecting visual noise and relies on accurate haptic information. We formulated this active sensing mechanism as the optimization of visuo-haptic information and effort. This OIE model can explain the adaptation of muscle activity to unimodal and multimodal sensory information when interacting with fixed or dynamic environments, or with another human, and can be used to optimize human-robot interaction.
On the Weaknesses of Backdoor-based Model Watermarking: An Information-theoretic Perspective
Venue: arXiv cs.CR 2024
Year: 2024

Safeguarding the intellectual property of machine learning models has emerged as a pressing concern in AI security. Model watermarking is a powerful technique for protecting ownership of machine learning models, yet its reliability has been recently challenged by recent watermark removal attacks. In this work, we investigate why existing watermark embedding techniques particularly those based on backdooring are vulnerable. Through an information-theoretic analysis, we show that the resilience of watermarking against erasure attacks hinges on the choice of trigger-set samples, where current uses of out-distribution trigger-set are inherently vulnerable to white-box adversaries. Based on this discovery, we propose a novel model watermarking scheme, In-distribution Watermark Embedding (IWE), to overcome the limitations of existing method. To further minimise the gap to clean models, we analyze the role of logits as watermark information carriers and propose a new approach to better conceal watermark information within the logits. Experiments on real-world datasets including CIFAR-100 and Caltech-101 demonstrate that our method robustly defends against various adversaries with negligible accuracy loss (< 0.1%).
NOVI : Chatbot System for University Novice with BERT and LLMs
Venue: arXiv cs.CL 2024
Year: 2024

To mitigate the difficulties of university freshmen in adapting to university life, we developed NOVI, a chatbot system based on GPT-4o. This system utilizes post and comment data from SKKU 'Everytime', a university community site. Developed using LangChain, NOVI's performance has been evaluated with a BLEU score, Perplexity score, ROUGE-1 score, ROUGE-2 score, ROUGE-L score and METEOR score. This approach is not only limited to help university freshmen but is also expected to help various people adapting to new environments with different data. This research explores the development and potential application of new educational technology tools, contributing to easier social adaptation for beginners and settling a foundation for future advancement in LLM studies.
RealisDance: Equip controllable character animation with realistic hands
Venue: arXiv cs.CV 2024
Year: 2024

Controllable character animation is an emerging task that generates character videos controlled by pose sequences from given character images. Although character consistency has made significant progress via reference UNet, another crucial factor, pose control, has not been well studied by existing methods yet, resulting in several issues: 1) The generation may fail when the input pose sequence is corrupted. 2) The hands generated using the DWPose sequence are blurry and unrealistic. 3) The generated video will be shaky if the pose sequence is not smooth enough. In this paper, we present RealisDance to handle all the above issues. RealisDance adaptively leverages three types of poses, avoiding failed generation caused by corrupted pose sequences. Among these pose types, HaMeR provides accurate 3D and depth information of hands, enabling RealisDance to generate realistic hands even for complex gestures. Besides using temporal attention in the main UNet, RealisDance also inserts temporal attention into the pose guidance network, smoothing the video from the pose condition aspect. Moreover, we introduce pose shuffle augmentation during training to further improve generation robustness and video smoothness. Qualitative experiments demonstrate the superiority of RealisDance over other existing methods, especially in hand quality.
Denoising: A Powerful Building-Block for Imaging, Inverse Problems, and Machine Learning
Venue: arXiv cs.LG 2024
Year: 2024

Denoising, the process of reducing random fluctuations in a signal to emphasize essential patterns, has been a fundamental problem of interest since the dawn of modern scientific inquiry. Recent denoising techniques, particularly in imaging, have achieved remarkable success, nearing theoretical limits by some measures. Yet, despite tens of thousands of research papers, the wide-ranging applications of denoising beyond noise removal have not been fully recognized. This is partly due to the vast and diverse literature, making a clear overview challenging. This paper aims to address this gap. We present a clarifying perspective on denoisers, their structure, and desired properties. We emphasize the increasing importance of denoising and showcase its evolution into an essential building block for complex tasks in imaging, inverse problems, and machine learning. Despite its long history, the community continues to uncover unexpected and groundbreaking uses for denoising, further solidifying its place as a cornerstone of scientific and engineering practice.