OpenRaiser/Intern-Atlas huggingface dataset:
Towards a Comprehensive, Efficient and Promptable Anatomic Structure Segmentation Model using 3D Whole-body CT Scans
Venue: arXiv cs.CV 2024
Year: 2024

Segment anything model (SAM) demonstrates strong generalization ability on natural image segmentation. However, its direct adaptation in medical image segmentation tasks shows significant performance drops. It also requires an excessive number of prompt points to obtain a reasonable accuracy. Although quite a few studies explore adapting SAM into medical image volumes, the efficiency of 2D adaptation methods is unsatisfactory and 3D adaptation methods are only capable of segmenting specific organs/tumors. In this work, we propose a comprehensive and scalable 3D SAM model for whole-body CT segmentation, named CT-SAM3D. Instead of adapting SAM, we propose a 3D promptable segmentation model using a (nearly) fully labeled CT dataset. To train CT-SAM3D effectively, ensuring the model's accurate responses to higher-dimensional spatial prompts is crucial, and 3D patch-wise training is required due to GPU memory constraints. Therefore, we propose two key technical developments: 1) a progressively and spatially aligned prompt encoding method to effectively encode click prompts in local 3D space; and 2) a cross-patch prompt scheme to capture more 3D spatial context, which is beneficial for reducing the editing workloads when interactively prompting on large organs. CT-SAM3D is trained using a curated dataset of 1204 CT scans containing 107 whole-body anatomies and extensively validated using five datasets, achieving significantly better results against all previous SAM-derived models. Code, data, and our 3D interactive segmentation tool with quasi-real-time responses are available at https://github.com/alibaba-damo-academy/ct-sam3d.
Active Learning for Regression based on Wasserstein distance and GroupSort Neural Networks
Venue: arXiv cs.LG 2024
Year: 2024

This paper addresses a new active learning strategy for regression problems. The presented Wasserstein active regression model is based on the principles of distribution-matching to measure the representativeness of the labeled dataset. The Wasserstein distance is computed using GroupSort Neural Networks. The use of such networks provides theoretical foundations giving a way to quantify errors with explicit bounds for their size and depth. This solution is combined with another uncertainty-based approach that is more outlier-tolerant to complete the query strategy. Finally, this method is compared with other classical and recent solutions. The study empirically shows the pertinence of such a representativity-uncertainty approach, which provides good estimation all along the query procedure. Moreover, the Wasserstein active regression often achieves more precise estimations and tends to improve accuracy faster than other models.
SYNCS: Synthetic Data and Contrastive Self-Supervised Training for Central Sulcus Segmentation
Venue: arXiv cs.CV 2024
Year: 2024

Bipolar disorder (BD) and schizophrenia (SZ) are severe mental disorders with profound societal impact. Identifying risk markers early is crucial for understanding disease progression and enabling preventive measures. The Danish High Risk and Resilience Study (VIA) focuses on understanding early disease processes, particularly in children with familial high risk (FHR). Understanding structural brain changes associated with these diseases during early stages is essential for effective interventions. The central sulcus (CS) is a prominent brain landmark related to brain regions involved in motor and sensory processing. Analyzing CS morphology can provide valuable insights into neurodevelopmental abnormalities in the FHR group. However, segmenting the central sulcus (CS) presents challenges due to its variability, especially in adolescents. This study introduces two novel approaches to improve CS segmentation: synthetic data generation to model CS variability and self-supervised pre-training with multi-task learning to adapt models to new cohorts. These methods aim to enhance segmentation performance across diverse populations, eliminating the need for extensive preprocessing.
EndoGSLAM: Real-Time Dense Reconstruction and Tracking in Endoscopic Surgeries using Gaussian Splatting
Venue: arXiv cs.CV 2024
Year: 2024

Precise camera tracking, high-fidelity 3D tissue reconstruction, and real-time online visualization are critical for intrabody medical imaging devices such as endoscopes and capsule robots. However, existing SLAM (Simultaneous Localization and Mapping) methods often struggle to achieve both complete high-quality surgical field reconstruction and efficient computation, restricting their intraoperative applications among endoscopic surgeries. In this paper, we introduce EndoGSLAM, an efficient SLAM approach for endoscopic surgeries, which integrates streamlined Gaussian representation and differentiable rasterization to facilitate over 100 fps rendering speed during online camera tracking and tissue reconstructing. Extensive experiments show that EndoGSLAM achieves a better trade-off between intraoperative availability and reconstruction quality than traditional or neural SLAM approaches, showing tremendous potential for endoscopic surgeries. The project page is at https://EndoGSLAM.loping151.com
Gradient-based Sampling for Class Imbalanced Semi-supervised Object Detection
Venue: arXiv cs.CV 2024
Year: 2024

Current semi-supervised object detection (SSOD) algorithms typically assume class balanced datasets (PASCAL VOC etc.) or slightly class imbalanced datasets (MS-COCO, etc). This assumption can be easily violated since real world datasets can be extremely class imbalanced in nature, thus making the performance of semi-supervised object detectors far from satisfactory. Besides, the research for this problem in SSOD is severely under-explored. To bridge this research gap, we comprehensively study the class imbalance problem for SSOD under more challenging scenarios, thus forming the first experimental setting for class imbalanced SSOD (CI-SSOD). Moreover, we propose a simple yet effective gradient-based sampling framework that tackles the class imbalance problem from the perspective of two types of confirmation biases. To tackle confirmation bias towards majority classes, the gradient-based reweighting and gradient-based thresholding modules leverage the gradients from each class to fully balance the influence of the majority and minority classes. To tackle the confirmation bias from incorrect pseudo labels of minority classes, the class-rebalancing sampling module resamples unlabeled data following the guidance of the gradient-based reweighting module. Experiments on three proposed sub-tasks, namely MS-COCO, MS-COCO to Object365 and LVIS, suggest that our method outperforms current class imbalanced object detectors by clear margins, serving as a baseline for future research in CI-SSOD. Code will be available at https://github.com/nightkeepers/CI-SSOD.
On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyond
Venue: arXiv cs.LG 2024
Year: 2024

This paper aims to clearly distinguish between Stochastic Gradient Descent with Momentum (SGDM) and Adam in terms of their convergence rates. We demonstrate that Adam achieves a faster convergence compared to SGDM under the condition of non-uniformly bounded smoothness. Our findings reveal that: (1) in deterministic environments, Adam can attain the known lower bound for the convergence rate of deterministic first-order optimizers, whereas the convergence rate of Gradient Descent with Momentum (GDM) has higher order dependence on the initial function value; (2) in stochastic setting, Adam's convergence rate upper bound matches the lower bounds of stochastic first-order optimizers, considering both the initial function value and the final error, whereas there are instances where SGDM fails to converge with any learning rate. These insights distinctly differentiate Adam and SGDM regarding their convergence rates. Additionally, by introducing a novel stopping-time based technique, we further prove that if we consider the minimum gradient norm during iterations, the corresponding convergence rate can match the lower bounds across all problem hyperparameters. The technique can also help proving that Adam with a specific hyperparameter scheduler is parameter-agnostic, which hence can be of independent interest.
Experimental Studies of Metaverse Streaming
Venue: arXiv cs.MM 2024
Year: 2024

Metaverse aims to construct a large, unified, immersive, and shared digital realm by combining various technologies, namely XR (extended reality), blockchain, and digital twin, among others. This article explores the Metaverse from the perspective of multimedia communication by conducting and analyzing real-world experiments on four different Metaverse platforms: VR (virtual reality) Vircadia, VR Mozilla Hubs, VRChat, and MR (mixed reality) Virtual City. We first investigate the traffic patterns and network performance in the three VR platforms. After raising the challenges of the Metaverse streaming and investigating the potential methods to enhance Metaverse performance, we propose a remote rendering architecture and verify its advantages through a prototype involving the campus network and MR multimodal interaction by comparison with local rendering.
Event Temporal Relation Extraction based on Retrieval-Augmented on LLMs
Venue: arXiv cs.CL 2024
Year: 2024

Event temporal relation (TempRel) is a primary subject of the event relation extraction task. However, the inherent ambiguity of TempRel increases the difficulty of the task. With the rise of prompt engineering, it is important to design effective prompt templates and verbalizers to extract relevant knowledge. The traditional manually designed templates struggle to extract precise temporal knowledge. This paper introduces a novel retrieval-augmented TempRel extraction approach, leveraging knowledge retrieved from large language models (LLMs) to enhance prompt templates and verbalizers. Our method capitalizes on the diverse capabilities of various LLMs to generate a wide array of ideas for template and verbalizer design. Our proposed method fully exploits the potential of LLMs for generation tasks and contributes more knowledge to our design. Empirical evaluations across three widely recognized datasets demonstrate the efficacy of our method in improving the performance of event temporal relation extraction tasks.
Fundus: A Simple-to-Use News Scraper Optimized for High Quality Extractions
Venue: arXiv cs.CL 2024
Year: 2024

This paper introduces Fundus, a user-friendly news scraper that enables users to obtain millions of high-quality news articles with just a few lines of code. Unlike existing news scrapers, we use manually crafted, bespoke content extractors that are specifically tailored to the formatting guidelines of each supported online newspaper. This allows us to optimize our scraping for quality such that retrieved news articles are textually complete and without HTML artifacts. Further, our framework combines both crawling (retrieving HTML from the web or large web archives) and content extraction into a single pipeline. By providing a unified interface for a predefined collection of newspapers, we aim to make Fundus broadly usable even for non-technical users. This paper gives an overview of the framework, discusses our design choices, and presents a comparative evaluation against other popular news scrapers. Our evaluation shows that Fundus yields significantly higher quality extractions (complete and artifact-free news articles) than prior work. The framework is available on GitHub under https://github.com/flairNLP/fundus and can be simply installed using pip.
Multi-Review Fusion-in-Context
Venue: arXiv cs.CL 2024
Year: 2024

Grounded text generation, encompassing tasks such as long-form question-answering and summarization, necessitates both content selection and content consolidation. Current end-to-end methods are difficult to control and interpret due to their opaqueness. Accordingly, recent works have proposed a modular approach, with separate components for each step. Specifically, we focus on the second subtask, of generating coherent text given pre-selected content in a multi-document setting. Concretely, we formalize Fusion-in-Context (FiC) as a standalone task, whose input consists of source texts with highlighted spans of targeted content. A model then needs to generate a coherent passage that includes all and only the target information. Our work includes the development of a curated dataset of 1000 instances in the reviews domain, alongside a novel evaluation framework for assessing the faithfulness and coverage of highlights, which strongly correlate to human judgment. Several baseline models exhibit promising outcomes and provide insightful analyses. This study lays the groundwork for further exploration of modular text generation in the multi-document setting, offering potential improvements in the quality and reliability of generated content. Our benchmark, FuseReviews, including the dataset, evaluation framework, and designated leaderboard, can be found at https://fusereviews.github.io/.
CoLLEGe: Concept Embedding Generation for Large Language Models
Venue: arXiv cs.CL 2024
Year: 2024

Current language models are unable to quickly learn new concepts on the fly, often requiring a more involved finetuning process to learn robustly. Prompting in-context is not robust to context distractions, and often fails to confer much information about the new concepts. Classic methods for few-shot word learning in NLP, relying on global word vectors, are less applicable to large language models. In this paper, we introduce a novel approach named CoLLEGe (Concept Learning with Language Embedding Generation) to modernize few-shot concept learning. CoLLEGe is a meta-learning framework capable of generating flexible embeddings for new concepts using a small number of example sentences or definitions. Our primary meta-learning objective is simply to facilitate a language model to make next word predictions in forthcoming sentences, making it compatible with language model pretraining. We design a series of tasks to test new concept learning in challenging real-world scenarios, including new word acquisition, definition inference, and verbal reasoning, and demonstrate that our method succeeds in each setting without task-specific training. Code and data for our project can be found at https://college-concept-learning.github.io/
Harmonic Decomposition in Data Sketches
Venue: arXiv cs.DS 2024
Year: 2024

In the turnstile streaming model, a dynamic vector $\mathbf{x}=(\mathbf{x}_1,\ldots,\mathbf{x}_n)\in \mathbb{Z}^n$ is updated by a stream of entry-wise increments/decrements. Let $f\colon\mathbb{Z}\to \mathbb{R}_+$ be a symmetric function with $f(0)=0$. The \emph{$f$-moment} of $\mathbf{x}$ is defined to be $f(\mathbf{x}) := \sum_{v\in[n]}f(\mathbf{x}_v)$. We revisit the problem of constructing a \emph{universal sketch} that can estimate many different $f$-moments. Previous constructions of universal sketches rely on the technique of sampling with respect to the $L_0$-mass (uniform samples) or $L_2$-mass ($L_2$-heavy-hitters), whose universality comes from being able to evaluate the function $f$ over the samples. In this work we take a new approach to constructing a universal sketch that does not use \emph{any} explicit samples but relies on the \emph{harmonic structure} of the target function $f$. The new sketch ($\textsf{SymmetricPoissonTower}$) \emph{embraces} hash collisions instead of avoiding them, which saves multiple $\log n$ factors in space, e.g., when estimating all $L_p$-moments ($f(z) = |z|^p,p\in[0,2]$). For many nearly periodic functions, the new sketch is \emph{exponentially} more efficient than sampling-based methods. We conjecture that the $\textsf{SymmetricPoissonTower}$ sketch is \emph{the} universal sketch that can estimate every tractable function $f$.
Closing the Information Gap in Unidentified Anomalous Phenomena (UAP) Studies
Venue: arXiv cs.DL 2024
Year: 2024

Unidentified Anomalous Phenomena (UAP), also known as Unidentified Flying Objects (UFOs), has shifted from being a stigmatized topic on the fringes of scientific inquiry to a legitimate subject of scientific interest with a need for high quality, curated data, and rigorous scientific investigation. This paper presents a preliminary scoping review and analysis of scholarly literature related to UAP from 1967 until 2023, exploring a diverse range of research areas across disciplines to illustrate scholarly discourse about the topic. The paper focuses on characterizing papers published in recent years and notes that Library & Information Science is unrepresented in the current UAP literature. The paper also discusses how researchers across the iFields can contribute to UAP studies through inherent expertise such as data curation and data science as well as information behavior and information literacy, among others. The paper concludes by emphasizing that UAP Studies offer a rich intellectual realm for information science research, with the iFields well positioned to play a crucial role in supporting and engaging in the study of UAP.
CapsF: Capsule Fusion for Extracting psychiatric stressors for suicide from twitter
Venue: arXiv cs.CL 2024
Year: 2024

Along with factors such as cancer, blood pressure, street accidents and stroke, suicide has been one of Iran main causes of death. One of the main reasons for suicide is psychological stressors. Identifying psychological stressors in an at risk population can help in the early prevention of suicidal and suicidal behaviours. In recent years, the widespread popularity and flow of real time information sharing of social media have allowed for potential early intervention in large scale and even small scale populations. However, some automated approaches to extract psychiatric stressors from Twitter have been presented, but most of this research has been for non Persian languages. This study aims to investigate the techniques of detecting psychological stress related to suicide from Persian tweets using learning based methods. The proposed capsule based approach achieved a binary classification accuracy of 0.83.
Efficient Weighting Schemes for Auditing Instant-Runoff Voting Elections
Venue: arXiv cs.CY 2024
Year: 2024

Various risk-limiting audit (RLA) methods have been developed for instant-runoff voting (IRV) elections. A recent method, AWAIRE, is the first efficient approach that can take advantage of but does not require cast vote records (CVRs). AWAIRE involves adaptively weighted averages of test statistics, essentially "learning" an effective set of hypotheses to test. However, the initial paper on AWAIRE only examined a few weighting schemes and parameter settings. We explore schemes and settings more extensively, to identify and recommend efficient choices for practice. We focus on the case where CVRs are not available, assessing performance using simulations based on real election data. The most effective schemes are often those that place most or all of the weight on the apparent "best" hypotheses based on already seen data. Conversely, the optimal tuning parameters tended to vary based on the election margin. Nonetheless, we quantify the performance trade-offs for different choices across varying election margins, aiding in selecting the most desirable trade-off if a default option is needed. A limitation of the current AWAIRE implementation is its restriction to a small number of candidates -- up to six in previous implementations. One path to a more computationally efficient implementation would be to use lazy evaluation and avoid considering all possible hypotheses. Our findings suggest that such an approach could be done without substantially compromising statistical performance.
AI Ethics and Governance in Practice: An Introduction
Venue: arXiv cs.CY 2024
Year: 2024

AI systems may have transformative and long-term effects on individuals and society. To manage these impacts responsibly and direct the development of AI systems toward optimal public benefit, considerations of AI ethics and governance must be a first priority. In this workbook, we introduce and describe our PBG Framework, a multi-tiered governance model that enables project teams to integrate ethical values and practical principles into their innovation practices and to have clear mechanisms for demonstrating and documenting this.
Fine-Tuning Pre-trained Language Models to Detect In-Game Trash Talks
Venue: arXiv cs.CL 2024
Year: 2024

Common problems in playing online mobile and computer games were related to toxic behavior and abusive communication among players. Based on different reports and studies, the study also discusses the impact of online hate speech and toxicity on players' in-game performance and overall well-being. This study investigates the capability of pre-trained language models to classify or detect trash talk or toxic in-game messages The study employs and evaluates the performance of pre-trained BERT and GPT language models in detecting toxicity within in-game chats. Using publicly available APIs, in-game chat data from DOTA 2 game matches were collected, processed, reviewed, and labeled as non-toxic, mild (toxicity), and toxic. The study was able to collect around two thousand in-game chats to train and test BERT (Base-uncased), BERT (Large-uncased), and GPT-3 models. Based on the three models' state-of-the-art performance, this study concludes pre-trained language models' promising potential for addressing online hate speech and in-game insulting trash talk.
Isometric Neural Machine Translation using Phoneme Count Ratio Reward-based Reinforcement Learning
Venue: arXiv cs.CL 2024
Year: 2024

Traditional Automatic Video Dubbing (AVD) pipeline consists of three key modules, namely, Automatic Speech Recognition (ASR), Neural Machine Translation (NMT), and Text-to-Speech (TTS). Within AVD pipelines, isometric-NMT algorithms are employed to regulate the length of the synthesized output text. This is done to guarantee synchronization with respect to the alignment of video and audio subsequent to the dubbing process. Previous approaches have focused on aligning the number of characters and words in the source and target language texts of Machine Translation models. However, our approach aims to align the number of phonemes instead, as they are closely associated with speech duration. In this paper, we present the development of an isometric NMT system using Reinforcement Learning (RL), with a focus on optimizing the alignment of phoneme counts in the source and target language sentence pairs. To evaluate our models, we propose the Phoneme Count Compliance (PCC) score, which is a measure of length compliance. Our approach demonstrates a substantial improvement of approximately 36% in the PCC score compared to the state-of-the-art models when applied to English-Hindi language pairs. Moreover, we propose a student-teacher architecture within the framework of our RL approach to maintain a trade-off between the phoneme count and translation quality.