OpenRaiser/Intern-Atlas huggingface dataset:
First-frame Supervised Video Polyp Segmentation via Propagative and Semantic Dual-teacher Network
Venue: arXiv cs.CV 2024
Year: 2024

Automatic video polyp segmentation plays a critical role in gastrointestinal cancer screening, but the cost of frameby-frame annotations is prohibitively high. While sparse-frame supervised methods have reduced this burden proportionately, the cost remains overwhelming for long-duration videos and large-scale datasets. In this paper, we, for the first time, reduce the annotation cost to just a single frame per polyp video, regardless of the video's length. To this end, we introduce a new task, First-Frame Supervised Video Polyp Segmentation (FSVPS), and propose a novel Propagative and Semantic Dual-Teacher Network (PSDNet). Specifically, PSDNet adopts a teacher-student framework but employs two distinct types of teachers: the propagative teacher and the semantic teacher. The propagative teacher is a universal object tracker that propagates the first-frame annotation to subsequent frames as pseudo labels. However, tracking errors may accumulate over time, gradually degrading the pseudo labels and misguiding the student model. To address this, we introduce the semantic teacher, an exponential moving average of the student model, which produces more stable and time-invariant pseudo labels. PSDNet merges the pseudo labels from both teachers using a carefully-designed back-propagation strategy. This strategy assesses the quality of the pseudo labels by tracking them backward to the first frame. High-quality pseudo labels are more likely to spatially align with the firstframe annotation after this backward tracking, ensuring more accurate teacher-to-student knowledge transfer and improved segmentation performance. Benchmarking on SUN-SEG, the largest VPS dataset, demonstrates the competitive performance of PSDNet compared to fully-supervised approaches, and its superiority over sparse-frame supervised state-of-the-arts with a minimum improvement of 4.5% in Dice score.
Context-Aware Outlier Rejection for Robust Multi-View 3D Tracking of Similar Small Birds in An Outdoor Aviary
Venue: arXiv cs.CV 2024
Year: 2024

This paper presents a novel approach for robust 3D tracking of multiple birds in an outdoor aviary using a multi-camera system. Our method addresses the challenges of visually similar birds and their rapid movements by leveraging environmental landmarks for enhanced feature matching and 3D reconstruction. In our approach, outliers are rejected based on their nearest landmark. This enables precise 3D-modeling and simultaneous tracking of multiple birds. By utilizing environmental context, our approach significantly improves the differentiation between visually similar birds, a key obstacle in existing tracking systems. Experimental results demonstrate the effectiveness of our method, showing a $20\%$ elimination of outliers in the 3D reconstruction process, with a $97\%$ accuracy in matching. This remarkable accuracy in 3D modeling translates to robust and reliable tracking of multiple birds, even in challenging outdoor conditions. Our work not only advances the field of computer vision but also provides a valuable tool for studying bird behavior and movement patterns in natural settings. We also provide a large annotated dataset of 80 birds residing in four enclosures for 20 hours of footage which provides a rich testbed for researchers in computer vision, ornithologists, and ecologists. Code and the link to the dataset is available at https://github.com/airou-lab/3D_Multi_Bird_Tracking
Anchor Learning with Potential Cluster Constraints for Multi-view Clustering
Venue: arXiv cs.CV 2024
Year: 2024

Anchor-based multi-view clustering (MVC) has received extensive attention due to its efficient performance. Existing methods only focus on how to dynamically learn anchors from the original data and simultaneously construct anchor graphs describing the relationships between samples and perform clustering, while ignoring the reality of anchors, i.e., high-quality anchors should be generated uniformly from different clusters of data rather than scattered outside the clusters. To deal with this problem, we propose a noval method termed Anchor Learning with Potential Cluster Constraints for Multi-view Clustering (ALPC) method. Specifically, ALPC first establishes a shared latent semantic module to constrain anchors to be generated from specific clusters, and subsequently, ALPC improves the representativeness and discriminability of anchors by adapting the anchor graph to capture the common clustering center of mass from samples and anchors, respectively. Finally, ALPC combines anchor learning and graph construction into a unified framework for collaborative learning and mutual optimization to improve the clustering performance. Extensive experiments demonstrate the effectiveness of our proposed method compared to some state-of-the-art MVC methods. Our source code is available at https://github.com/whbdmu/ALPC.
Batch Selection for Multi-Label Classification Guided by Uncertainty and Dynamic Label Correlations
Venue: arXiv cs.LG 2024
Year: 2024

The accuracy of deep neural networks is significantly influenced by the effectiveness of mini-batch construction during training. In single-label scenarios, such as binary and multi-class classification tasks, it has been demonstrated that batch selection algorithms preferring samples with higher uncertainty achieve better performance than difficulty-based methods. Although there are two batch selection methods tailored for multi-label data, none of them leverage important uncertainty information. Adapting the concept of uncertainty to multi-label data is not a trivial task, since there are two issues that should be tackled. First, traditional variance or entropy-based uncertainty measures ignore fluctuations of predictions within sliding windows and the importance of the current model state. Second, existing multi-label methods do not explicitly exploit the label correlations, particularly the uncertainty-based label correlations that evolve during the training process. In this paper, we propose an uncertainty-based multi-label batch selection algorithm. It assesses uncertainty for each label by considering differences between successive predictions and the confidence of current outputs, and further leverages dynamic uncertainty-based label correlations to emphasize instances whose uncertainty is synergistically expressed across multiple labels. Empirical studies demonstrate the effectiveness of our method in improving the performance and accelerating the convergence of various multi-label deep learning models.
Prior2Posterior: Model Prior Correction for Long-Tailed Learning
Venue: arXiv cs.CV 2024
Year: 2024

Learning-based solutions for long-tailed recognition face difficulties in generalizing on balanced test datasets. Due to imbalanced data prior, the learned \textit{a posteriori} distribution is biased toward the most frequent (head) classes, leading to an inferior performance on the least frequent (tail) classes. In general, the performance can be improved by removing such a bias by eliminating the effect of imbalanced prior modeled using the number of class samples (frequencies). We first observe that the \textit{effective prior} on the classes, learned by the model at the end of the training, can differ from the empirical prior obtained using class frequencies. Thus, we propose a novel approach to accurately model the effective prior of a trained model using \textit{a posteriori} probabilities. We propose to correct the imbalanced prior by adjusting the predicted \textit{a posteriori} probabilities (Prior2Posterior: P2P) using the calculated prior in a post-hoc manner after the training, and show that it can result in improved model performance. We present theoretical analysis showing the optimality of our approach for models trained with naive cross-entropy loss as well as logit adjusted loss. Our experiments show that the proposed approach achieves new state-of-the-art (SOTA) on several benchmark datasets from the long-tail literature in the category of logit adjustment methods. Further, the proposed approach can be used to inspect any existing method to capture the \textit{effective prior} and remove any residual bias to improve its performance, post-hoc, without model retraining. We also show that by using the proposed post-hoc approach, the performance of many existing methods can be improved further.
Metagoals Endowing Self-Modifying AGI Systems with Goal Stability or Moderated Goal Evolution: Toward a Formally Sound and Practical Approach
Venue: arXiv cs.AI 2024
Year: 2024

We articulate here a series of specific metagoals designed to address the challenge of creating AGI systems that possess the ability to flexibly self-modify yet also have the propensity to maintain key invariant properties of their goal systems 1) a series of goal-stability metagoals aimed to guide a system to a condition in which goal-stability is compatible with reasonably flexible self-modification 2) a series of moderated-goal-evolution metagoals aimed to guide a system to a condition in which control of the pace of goal evolution is compatible with reasonably flexible self-modification The formulation of the metagoals is founded on fixed-point theorems from functional analysis, e.g. the Contraction Mapping Theorem and constructive approximations to Schauder's Theorem, applied to probabilistic models of system behavior We present an argument that the balancing of self-modification with maintenance of goal invariants will often have other interesting cognitive side-effects such as a high degree of self understanding Finally we argue for the practical value of a hybrid metagoal combining moderated-goal-evolution with pursuit of goal-stability -- along with potentially other metagoals relating to goal-satisfaction, survival and ongoing development -- in a flexible fashion depending on the situation
A Large-scale Empirical Study on Fine-tuning Large Language Models for Unit Testing
Venue: arXiv cs.SE 2024
Year: 2024

Unit testing plays a pivotal role in software development, improving software quality and reliability. However, generating effective test cases manually is time-consuming, prompting interest in unit testing research. Recently, Large Language Models (LLMs) have shown potential in various unit testing tasks, including test generation, assertion generation, and test evolution, but existing studies are limited in scope and lack a systematic evaluation of the effectiveness of LLMs. To bridge this gap, we present a large-scale empirical study on fine-tuning LLMs for unit testing. Our study involves three unit testing tasks, five benchmarks, eight evaluation metrics, and 37 popular LLMs across various architectures and sizes, consuming over 3,000 NVIDIA A100 GPU hours. We focus on three key research questions: (1) the performance of LLMs compared to state-of-the-art methods, (2) the impact of different factors on LLM performance, and (3) the effectiveness of fine-tuning versus prompt engineering. Our findings reveal that LLMs outperform existing state-of-the-art approaches on all three unit testing tasks across nearly all metrics, highlighting the potential of fine-tuning LLMs in unit testing tasks. Furthermore, large-scale, decoder-only models achieve the best results across tasks, while encoder-decoder models perform better under the same parameter scale. Additionally, the comparison of the performance between fine-tuning and prompt engineering approaches reveals the considerable potential capability of the prompt engineering approach in unit testing tasks. We then discuss the concerned issues on the test generation task, including data leakage issues, bug detection capabilities, and metrics comparisons. Finally, we further pinpoint carious practical guidelines for LLM-based approaches to unit testing tasks in the near future.
Generalizable Articulated Object Perception with Superpoints
Venue: arXiv cs.CV 2024
Year: 2024

Manipulating articulated objects with robotic arms is challenging due to the complex kinematic structure, which requires precise part segmentation for efficient manipulation. In this work, we introduce a novel superpoint-based perception method designed to improve part segmentation in 3D point clouds of articulated objects. We propose a learnable, part-aware superpoint generation technique that efficiently groups points based on their geometric and semantic similarities, resulting in clearer part boundaries. Furthermore, by leveraging the segmentation capabilities of the 2D foundation model SAM, we identify the centers of pixel regions and select corresponding superpoints as candidate query points. Integrating a query-based transformer decoder further enhances our method's ability to achieve precise part segmentation. Experimental results on the GAPartNet dataset show that our method outperforms existing state-of-the-art approaches in cross-category part segmentation, achieving AP50 scores of 77.9% for seen categories (4.4% improvement) and $39.3\%$ for unseen categories (11.6% improvement), with superior results in 5 out of 9 part categories for seen objects and outperforming all previous methods across all part categories for unseen objects.
CyberSentinel: Efficient Anomaly Detection in Programmable Switch using Knowledge Distillation
Venue: arXiv cs.CR 2024
Year: 2024

The increasing volume of traffic (especially from IoT devices) is posing a challenge to the current anomaly detection systems. Existing systems are forced to take the support of the control plane for a more thorough and accurate detection of malicious traffic (anomalies). This introduces latency in making decisions regarding fast incoming traffic and therefore, existing systems are unable to scale to such growing rates of traffic. In this paper, we propose CyberSentinel, a high throughput and accurate anomaly detection system deployed entirely in the programmable switch data plane; making it the first work to accurately detect anomalies at line speed. To detect unseen network attacks, CyberSentinel uses a novel knowledge distillation scheme that incorporates "learned" knowledge of deep unsupervised ML models (\textit{e.g.}, autoencoders) to develop an iForest model that is then installed in the data plane in the form of whitelist rules. We implement a prototype of CyberSentinel on a testbed with an Intel Tofino switch and evaluate it on various real-world use cases. CyberSentinel yields similar detection performance compared to the state-of-the-art control plane solutions but with an increase in packet-processing throughput by $66.47\%$ on a $40$ Gbps link, and a reduction in average per-packet latency by $50\%$.
FAP-CD: Fairness-Driven Age-Friendly Community Planning via Conditional Diffusion Generation
Venue: arXiv cs.AI 2024
Year: 2024

As global populations age rapidly, incorporating age-specific considerations into urban planning has become essential to addressing the urgent demand for age-friendly built environments and ensuring sustainable urban development. However, current practices often overlook these considerations, resulting in inadequate and unevenly distributed elderly services in cities. There is a pressing need for equitable and optimized urban renewal strategies to support effective age-friendly planning. To address this challenge, we propose a novel framework, Fairness-driven Age-friendly community Planning via Conditional Diffusion generation (FAP-CD). FAP-CD leverages a conditioned graph denoising diffusion probabilistic model to learn the joint probability distribution of aging facilities and their spatial relationships at a fine-grained regional level. Our framework generates optimized facility distributions by iteratively refining noisy graphs, conditioned on the needs of the elderly during the diffusion process. Key innovations include a demand-fairness pre-training module that integrates community demand features and facility characteristics using an attention mechanism and min-max optimization, ensuring equitable service distribution across regions. Additionally, a discrete graph structure captures walkable accessibility within regional road networks, guiding model sampling. To enhance information integration, we design a graph denoising network with an attribute augmentation module and a hybrid graph message aggregation module, combining local and global node and edge information. Empirical results across multiple metrics demonstrate the effectiveness of FAP-CD in balancing age-friendly needs with regional equity, achieving an average improvement of 41% over competitive baseline models.
Business Analysis: User Attitude Evaluation and Prediction Based on Hotel User Reviews and Text Mining
Venue: arXiv cs.CY 2024
Year: 2024

In the post-pandemic era, the hotel industry plays a crucial role in economic recovery, with consumer sentiment increasingly influencing market trends. This study utilizes advanced natural language processing (NLP) and the BERT model to analyze user reviews, extracting insights into customer satisfaction and guiding service improvements. By transforming reviews into feature vectors, the BERT model accurately classifies emotions, uncovering patterns of satisfaction and dissatisfaction. This approach provides valuable data for hotel management, helping them refine service offerings and improve customer experiences. From a financial perspective, understanding sentiment is vital for predicting market performance, as shifts in consumer sentiment often correlate with stock prices and overall industry performance. Additionally, the study addresses data imbalance in sentiment analysis, employing techniques like oversampling and undersampling to enhance model robustness. The results offer actionable insights not only for the hotel industry but also for financial analysts, aiding in market forecasts and investment decisions. This research highlights the potential of sentiment analysis to drive business growth, improve financial outcomes, and enhance competitive advantage in the dynamic tourism and hospitality sectors, thereby contributing to the broader economic landscape.
SilVar: Speech Driven Multimodal Model for Reasoning Visual Question Answering and Object Localization
Venue: arXiv cs.CV 2024
Year: 2024

Visual Language Models have demonstrated remarkable capabilities across tasks, including visual question answering and image captioning. However, most models rely on text-based instructions, limiting their effectiveness in human-machine interactions. Moreover, the quality of language models depends on reasoning and prompting techniques, such as COT, which remain underexplored when using speech instructions. To address these challenges, we propose SilVar, a novel end-to-end multimodal model that uses speech instructions for reasoning in visual question answering. In addition, we investigate reasoning techniques with levels including conversational, simple, and complex speech instruction. SilVar is built upon CLIP, Whisper, and LLaMA 3.1-8B, enabling intuitive interactions by allowing users to provide verbal or text instructions. To this end, we introduce a dataset designed to challenge models with speech-based reasoning tasks for object localization. This dataset enhances the model ability to process and explain visual scenes from spoken input, moving beyond object recognition to reasoning-based interactions. The experiments show that SilVar achieves SOTA performance on the MMMU and ScienceQA benchmarks despite the challenge of speech-based instructions. We believe SilVar will inspire next-generation multimodal reasoning models, toward expert artificial general intelligence. Our code and dataset are available here.
TelegramScrap: A comprehensive tool for scraping Telegram data
Venue: arXiv cs.CY 2024
Year: 2024

[WhitePaper] The TelegramScrap tool provides a robust and versatile solution for extracting and analyzing data from Telegram channels and groups, addressing the increasing demand for efficient methods to study digital ecosystems. This white paper outlines the tool's development, capabilities, and applications in academic and scientific research, including studies on disinformation, political communication, and thematic patterns in online communities. Built with flexibility and user accessibility in mind, the tool allows researchers to customize scraping parameters, handle large datasets, and produce structured outputs in formats such as Excel and Parquet. Its modular architecture, real-time progress tracking, and error-handling mechanisms ensure reliability and scalability for diverse research needs. Emphasizing ethical data collection, the tool aligns with Telegram's terms of service and data privacy regulations, encouraging responsible use. Released under an open-source license, TelegramScrap invites the academic community to explore, adapt, and improve the tool while providing appropriate credit. This paper demonstrates the tool's impact through its application in multiple studies, showcasing its potential to advance computational social science and enhance understanding of digital interactions and societal trends [ Code available on GitHub: https://github.com/ergoncugler/web-scraping-telegram ].
Inflation of 2D boundary ghosts and digital watermarking
Venue: arXiv cs.DM 2024
Year: 2024

Projection ghosts are discrete arrays of signed values positioned so that their discrete projections vanish for some chosen set of n projection angles. Minimal ghosts are designed to be compact, with no internal pixels having value zero. Here we control the shape, number of boundary pixels and area that each minimal ghost encloses. Binary minimal ghosts and their boundaries can themselves be inflated by tiling copies of themselves to make ghosts with larger sizes and different shapes, whilst still retaining the same set of n zero projection angles. The intricate perimeters of minimal ghosts are formed by three strings of connected pixels that are defined by the minimal projection angles. We show that large changes to the ghost areas can be made whilst keeping the length of their segmented perimeters fixed. These inflated boundary ghosts may prove useful as secure watermarks to embed into digital image data. Boundary ghosts may also help guide the selection of angles used to reconstruct images where the object domain is confined to oval shaped arcs.
Quantum-Like Contextuality in Large Language Models
Venue: arXiv cs.CL 2024
Year: 2024

Contextuality is a distinguishing feature of quantum mechanics and there is growing evidence that it is a necessary condition for quantum advantage. In order to make use of it, researchers have been asking whether similar phenomena arise in other domains. The answer has been yes, e.g. in behavioural sciences. However, one has to move to frameworks that take some degree of signalling into account. Two such frameworks exist: (1) a signalling-corrected sheaf theoretic model, and (2) the Contextuality-by-Default (CbD) framework. This paper provides the first large scale experimental evidence for a yes answer in natural language. We construct a linguistic schema modelled over a contextual quantum scenario, instantiate it in the Simple English Wikipedia and extract probability distributions for the instances using the large language model BERT. This led to the discovery of 77,118 sheaf-contextual and 36,938,948 CbD contextual instances. We proved that the contextual instances came from semantically similar words, by deriving an equation between degrees of contextuality and Euclidean distances of BERT's embedding vectors. A regression model further reveals that Euclidean distance is indeed the best statistical predictor of contextuality. Our linguistic schema is a variant of the co-reference resolution challenge. These results are an indication that quantum methods may be advantageous in language tasks.