OpenRaiser/Intern-Atlas huggingface dataset:
Leveraging Foundation models for Unsupervised Audio-Visual Segmentation
Venue: arXiv cs.CV 2023
Year: 2023

Audio-Visual Segmentation (AVS) aims to precisely outline audible objects in a visual scene at the pixel level. Existing AVS methods require fine-grained annotations of audio-mask pairs in supervised learning fashion. This limits their scalability since it is time consuming and tedious to acquire such cross-modality pixel level labels. To overcome this obstacle, in this work we introduce unsupervised audio-visual segmentation with no need for task-specific data annotations and model training. For tackling this newly proposed problem, we formulate a novel Cross-Modality Semantic Filtering (CMSF) approach to accurately associate the underlying audio-mask pairs by leveraging the off-the-shelf multi-modal foundation models (e.g., detection [1], open-world segmentation [2] and multi-modal alignment [3]). Guiding the proposal generation by either audio or visual cues, we design two training-free variants: AT-GDINO-SAM and OWOD-BIND. Extensive experiments on the AVS-Bench dataset show that our unsupervised approach can perform well in comparison to prior art supervised counterparts across complex scenarios with multiple auditory objects. Particularly, in situations where existing supervised AVS methods struggle with overlapping foreground objects, our models still excel in accurately segmenting overlapped auditory objects. Our code will be publicly released.
VEATIC: Video-based Emotion and Affect Tracking in Context Dataset
Venue: arXiv cs.CV 2023
Year: 2023

Human affect recognition has been a significant topic in psychophysics and computer vision. However, the currently published datasets have many limitations. For example, most datasets contain frames that contain only information about facial expressions. Due to the limitations of previous datasets, it is very hard to either understand the mechanisms for affect recognition of humans or generalize well on common cases for computer vision models trained on those datasets. In this work, we introduce a brand new large dataset, the Video-based Emotion and Affect Tracking in Context Dataset (VEATIC), that can conquer the limitations of the previous datasets. VEATIC has 124 video clips from Hollywood movies, documentaries, and home videos with continuous valence and arousal ratings of each frame via real-time annotation. Along with the dataset, we propose a new computer vision task to infer the affect of the selected character via both context and character information in each video frame. Additionally, we propose a simple model to benchmark this new computer vision task. We also compare the performance of the pretrained model using our dataset with other similar datasets. Experiments show the competing results of our pretrained model via VEATIC, indicating the generalizability of VEATIC. Our dataset is available at https://veatic.github.io.
ProMap: Datasets for Product Mapping in E-commerce
Venue: arXiv cs.LG 2023
Year: 2023

The goal of product mapping is to decide, whether two listings from two different e-shops describe the same products. Existing datasets of matching and non-matching pairs of products, however, often suffer from incomplete product information or contain only very distant non-matching products. Therefore, while predictive models trained on these datasets achieve good results on them, in practice, they are unusable as they cannot distinguish very similar but non-matching pairs of products. This paper introduces two new datasets for product mapping: ProMapCz consisting of 1,495 Czech product pairs and ProMapEn consisting of 1,555 English product pairs of matching and non-matching products manually scraped from two pairs of e-shops. The datasets contain both images and textual descriptions of the products, including their specifications, making them one of the most complete datasets for product mapping. Additionally, the non-matching products were selected in two phases, creating two types of non-matches -- close non-matches and medium non-matches. Even the medium non-matches are pairs of products that are much more similar than non-matches in other datasets -- for example, they still need to have the same brand and similar name and price. After simple data preprocessing, several machine learning algorithms were trained on these and two the other datasets to demonstrate the complexity and completeness of ProMap datasets. ProMap datasets are presented as a golden standard for further research of product mapping filling the gaps in existing ones.
Autoencoder-Based Visual Anomaly Localization for Manufacturing Quality Control
Venue: arXiv cs.CV 2023
Year: 2023

Manufacturing industries require efficient and voluminous production of high-quality finished goods. In the context of Industry 4.0, visual anomaly detection poses an optimistic solution for automatically controlled product quality with high precision. In general, automation based on computer vision is a promising solution to prevent bottlenecks at the product quality checkpoint. We considered recent advancements in machine learning to improve visual defect localization, but challenges persist in obtaining a balanced feature set and database of the wide variety of defects occurring in the production line. Hence, this paper proposes a defect localizing autoencoder with unsupervised class selection by clustering with k-means the features extracted from a pre-trained VGG16 network. Moreover, the selected classes of defects are augmented with natural wild textures to simulate artificial defects. The study demonstrates the effectiveness of the defect localizing autoencoder with unsupervised class selection for improving defect detection in manufacturing industries. The proposed methodology shows promising results with precise and accurate localization of quality defects on melamine-faced boards for the furniture industry. Incorporating artificial defects into the training data shows significant potential for practical implementation in real-world quality control scenarios.
Regular Representations of Uniform TC^0
Venue: arXiv cs.LO 2023
Year: 2023

The circuit complexity class DLOGTIME-uniform AC^0 is known to be a modest subclass of DLOGTIME-uniform TC^0. The weakness of AC^0 is caused by the fact that AC^0 is not closed under restricting AC^0-computable queries into simple subsequences of the input. Analogously, in descriptive complexity, the logics corresponding to DLOGTIME-uniform AC^0 do not have the relativization property and hence they are not regular. This weakness of DLOGTIME-uniform AC^0 has been elaborated in the line of research on the Crane Beach Conjecture. The conjecture (which was refuted by Barrington, Immerman, Lautemann, Schweikardt and Th{é}rien) was that if a language L has a neutral letter, then L can be defined in first-order logic with the collection of all numerical built-in relations, if and only if L can be already defined in FO with order. In the first part of this article we consider logics in the range of AC^0 and TC^0. First we formulate a combinatorial criterion for a cardinality quantifier C_S implying that all languages in DLOGTIME-uniform TC^0 can be defined in FO(C_S). For instance, this criterion is satisfied by C_S if S is the range of some polynomial with positive integer coefficients of degree at least two. In the second part of the paper we first adapt the key properties of abstract logics to accommodate built-in relations. Then we define the regular interior R-int(L) and regular closure R-cl(L), of a logic L, and show that the Crane Beach Conjecture can be interpreted as a statement concerning the regular interior of first-order logic with built-in relations B. We show that if B={+}, or B contains only unary relations besides the order, then R-int(FO_B) collapses to FO with order. In contrast, our results imply that if B contains the order and the range of a polynomial of degree at least two, then R-cl(FO_B) includes all languages in DLOGTIME-uniform TC^0.
Setting the Right Expectations: Algorithmic Recourse Over Time
Venue: arXiv cs.LG 2023
Year: 2023

Algorithmic systems are often called upon to assist in high-stakes decision making. In light of this, algorithmic recourse, the principle wherein individuals should be able to take action against an undesirable outcome made by an algorithmic system, is receiving growing attention. The bulk of the literature on algorithmic recourse to-date focuses primarily on how to provide recourse to a single individual, overlooking a critical element: the effects of a continuously changing context. Disregarding these effects on recourse is a significant oversight, since, in almost all cases, recourse consists of an individual making a first, unfavorable attempt, and then being given an opportunity to make one or several attempts at a later date - when the context might have changed. This can create false expectations, as initial recourse recommendations may become less reliable over time due to model drift and competition for access to the favorable outcome between individuals. In this work we propose an agent-based simulation framework for studying the effects of a continuously changing environment on algorithmic recourse. In particular, we identify two main effects that can alter the reliability of recourse for individuals represented by the agents: (1) competition with other agents acting upon recourse, and (2) competition with new agents entering the environment. Our findings highlight that only a small set of specific parameterizations result in algorithmic recourse that is reliable for agents over time. Consequently, we argue that substantial additional work is needed to understand recourse reliability over time, and to develop recourse methods that reward agents' effort.
Remote Inference of Cognitive Scores in ALS Patients Using a Picture Description
Venue: arXiv cs.CL 2023
Year: 2023

Amyotrophic lateral sclerosis is a fatal disease that not only affects movement, speech, and breath but also cognition. Recent studies have focused on the use of language analysis techniques to detect ALS and infer scales for monitoring functional progression. In this paper, we focused on another important aspect, cognitive impairment, which affects 35-50% of the ALS population. In an effort to reach the ALS population, which frequently exhibits mobility limitations, we implemented the digital version of the Edinburgh Cognitive and Behavioral ALS Screen (ECAS) test for the first time. This test which is designed to measure cognitive impairment was remotely performed by 56 participants from the EverythingALS Speech Study. As part of the study, participants (ALS and non-ALS) were asked to describe weekly one picture from a pool of many pictures with complex scenes displayed on their computer at home. We analyze the descriptions performed within +/- 60 days from the day the ECAS test was administered and extract different types of linguistic and acoustic features. We input those features into linear regression models to infer 5 ECAS sub-scores and the total score. Speech samples from the picture description are reliable enough to predict the ECAS subs-scores, achieving statistically significant Spearman correlation values between 0.32 and 0.51 for the model's performance using 10-fold cross-validation.
Breaking the k/log k Barrier in Collective Tree Exploration via Tree-Mining
Venue: arXiv cs.DS 2023
Year: 2023

In collective tree exploration, a team of $k$ mobile agents is tasked to go through all edges of an unknown tree as fast as possible. An edge of the tree is revealed to the team when one agent becomes adjacent to that edge. The agents start from the root and all move synchronously along one adjacent edge in each round. Communication between the agents is unrestricted, and they are, therefore, centrally controlled by a single exploration algorithm. The algorithm's guarantee is typically compared to the number of rounds required by the agents to go through all edges if they had known the tree in advance. This quantity is at least $\max\{2n/k,2D\}$ where $n$ is the number of nodes and $D$ is the tree depth. Since the introduction of the problem by [FGKP04], two types of guarantees have emerged: the first takes the form $r(k)(n/k+D)$, where $r(k)$ is called the competitive ratio, and the other takes the form $2n/k+f(k,D)$, where $f(k,D)$ is called the competitive overhead. In this paper, we present the first algorithm with linear-in-$D$ competitive overhead, thereby reconciling both approaches. Specifically, our bound is in $2n/k + O(k^{\log_2(k)-1} D)$ and leads to a competitive ratio in $O(k/\exp(\sqrt{\ln 2\ln k}))$. This is the first improvement over $O(k/\ln k)$ since the introduction of the problem, twenty years ago. Our algorithm is developed for an asynchronous generalization of collective tree exploration (ACTE). It belongs to a broad class of locally-greedy exploration algorithms that we define. We show that the analysis of locally-greedy algorithms can be seen through the lens of a 2-player game that we call the tree-mining game and which could be of independent interest.
Beyond original Research Articles Categorization via NLP
Venue: arXiv cs.CL 2023
Year: 2023

This work proposes a novel approach to text categorization -- for unknown categories -- in the context of scientific literature, using Natural Language Processing techniques. The study leverages the power of pre-trained language models, specifically SciBERT, to extract meaningful representations of abstracts from the ArXiv dataset. Text categorization is performed using the K-Means algorithm, and the optimal number of clusters is determined based on the Silhouette score. The results demonstrate that the proposed approach captures subject information more effectively than the traditional arXiv labeling system, leading to improved text categorization. The approach offers potential for better navigation and recommendation systems in the rapidly growing landscape of scientific research literature.
Cryptography: Against AI and QAI Odds
Venue: arXiv cs.CR 2023
Year: 2023

Artificial Intelligence (AI) presents prodigious technological prospects for development, however, all that glitters is not gold! The cyber-world faces the worst nightmare with the advent of AI and quantum computers. Together with Quantum Artificial Intelligence (QAI), they pose a catastrophic threat to modern cryptography. It would also increase the capability of cryptanalysts manifold, with its built-in persistent and extensive predictive intelligence. This prediction ability incapacitates the constrained message space in device cryptography. With the comparison of these assumptions and the intercepted ciphertext, the code-cracking process will considerably accelerate. Before the vigorous and robust developments in AI, we have never faced and never had to prepare for such a plaintext-originating attack. The supremacy of AI can be challenged by creating ciphertexts that would give the AI attacker erroneous responses stymied by randomness and misdirect them. AI threat is deterred by deviating from the conventional use of small, known-size keys and pattern-loaded ciphers. The strategy is vested in implementing larger secret size keys, supplemented by ad-hoc unilateral randomness of unbound limitations and a pattern-devoid technique. The very large key size can be handled with low processing and computational burden to achieve desired unicity distances. The strategy against AI odds is feasible by implementing non-algorithmic randomness, large and inexpensive memory chips, and wide-area communication networks. The strength of AI, i.e., randomness and pattern detection can be used to generate highly optimized ciphers and algorithms. These pattern-devoid, randomness-rich ciphers also provide a timely and plausible solution for NIST's proactive approach toward the quantum challenge.
Human-Machine Co-Creativity with Older Adults -- A Learning Community to Study Explainable Dialogues
Venue: arXiv cs.HC 2023
Year: 2023

This position paper is part of a long-term research project on human-machine co-creativity with older adults. The goal is to investigate how robots and AI-generated content can contribute to older adults' creative experiences, with a focus on collaborative drawing and painting. The research has recently started, and current activities are centred around literature studies, interviews with seniors and artists, and developing initial prototypes. In addition, a course "Drawing with Robots", is being developed to establish collaboration between human and machine learners: older adults, artists, students, researchers, and artificial agents. We present this course as a learning community and as an opportunity for studying how explainable AI and creative dialogues can be intertwined in human-machine co-creativity with older adults.
Text-Guided Generation and Editing of Compositional 3D Avatars
Venue: arXiv cs.CV 2023
Year: 2023

Our goal is to create a realistic 3D facial avatar with hair and accessories using only a text description. While this challenge has attracted significant recent interest, existing methods either lack realism, produce unrealistic shapes, or do not support editing, such as modifications to the hairstyle. We argue that existing methods are limited because they employ a monolithic modeling approach, using a single representation for the head, face, hair, and accessories. Our observation is that the hair and face, for example, have very different structural qualities that benefit from different representations. Building on this insight, we generate avatars with a compositional model, in which the head, face, and upper body are represented with traditional 3D meshes, and the hair, clothing, and accessories with neural radiance fields (NeRF). The model-based mesh representation provides a strong geometric prior for the face region, improving realism while enabling editing of the person's appearance. By using NeRFs to represent the remaining components, our method is able to model and synthesize parts with complex geometry and appearance, such as curly hair and fluffy scarves. Our novel system synthesizes these high-quality compositional avatars from text descriptions. The experimental results demonstrate that our method, Text-guided generation and Editing of Compositional Avatars (TECA), produces avatars that are more realistic than those of recent methods while being editable because of their compositional nature. For example, our TECA enables the seamless transfer of compositional features like hairstyles, scarves, and other accessories between avatars. This capability supports applications such as virtual try-on.
LCReg: Long-Tailed Image Classification with Latent Categories based Recognition
Venue: arXiv cs.CV 2023
Year: 2023

In this work, we tackle the challenging problem of long-tailed image recognition. Previous long-tailed recognition approaches mainly focus on data augmentation or re-balancing strategies for the tail classes to give them more attention during model training. However, these methods are limited by the small number of training images for the tail classes, which results in poor feature representations. To address this issue, we propose the Latent Categories based long-tail Recognition (LCReg) method. Our hypothesis is that common latent features shared by head and tail classes can be used to improve feature representation. Specifically, we learn a set of class-agnostic latent features shared by both head and tail classes, and then use semantic data augmentation on the latent features to implicitly increase the diversity of the training sample. We conduct extensive experiments on five long-tailed image recognition datasets, and the results show that our proposed method significantly improves the baselines.
Solving Recurrence Relations using Machine Learning, with Application to Cost Analysis
Venue: arXiv cs.PL 2023
Year: 2023

Automatic static cost analysis infers information about the resources used by programs without actually running them with concrete data, and presents such information as functions of input data sizes. Most of the analysis tools for logic programs (and other languages) are based on setting up recurrence relations representing (bounds on) the computational cost of predicates, and solving them to find closed-form functions that are equivalent to (or a bound on) them. Such recurrence solving is a bottleneck in current tools: many of the recurrences that arise during the analysis cannot be solved with current solvers, such as Computer Algebra Systems (CASs), so that specific methods for different classes of recurrences need to be developed. We address such a challenge by developing a novel, general approach for solving arbitrary, constrained recurrence relations, that uses machine-learning sparse regression techniques to guess a candidate closed-form function, and a combination of an SMT-solver and a CAS to check whether such function is actually a solution of the recurrence. We have implemented a prototype and evaluated it with recurrences generated by a cost analysis system (the one in CiaoPP). The experimental results are quite promising, showing that our approach can find closed-form solutions, in a reasonable time, for classes of recurrences that cannot be solved by such a system, nor by current CASs.
Reliability-based cleaning of noisy training labels with inductive conformal prediction in multi-modal biomedical data mining
Venue: arXiv cs.LG 2023
Year: 2023

Accurately labeling biomedical data presents a challenge. Traditional semi-supervised learning methods often under-utilize available unlabeled data. To address this, we propose a novel reliability-based training data cleaning method employing inductive conformal prediction (ICP). This method capitalizes on a small set of accurately labeled training data and leverages ICP-calculated reliability metrics to rectify mislabeled data and outliers within vast quantities of noisy training data. The efficacy of the method is validated across three classification tasks within distinct modalities: filtering drug-induced-liver-injury (DILI) literature with title and abstract, predicting ICU admission of COVID-19 patients through CT radiomics and electronic health records, and subtyping breast cancer using RNA-sequencing data. Varying levels of noise to the training labels were introduced through label permutation. Results show significant enhancements in classification performance: accuracy enhancement in 86 out of 96 DILI experiments (up to 11.4%), AUROC and AUPRC enhancements in all 48 COVID-19 experiments (up to 23.8% and 69.8%), and accuracy and macro-average F1 score improvements in 47 out of 48 RNA-sequencing experiments (up to 74.6% and 89.0%). Our method offers the potential to substantially boost classification performance in multi-modal biomedical machine learning tasks. Importantly, it accomplishes this without necessitating an excessive volume of meticulously curated training data.
Energy-Constrained Active Exploration Under Incremental-Resolution Symbolic Perception
Venue: arXiv cs.RO 2023
Year: 2023

In this work, we consider the problem of autonomous exploration in search of targets while respecting a fixed energy budget. The robot is equipped with an incremental-resolution symbolic perception module wherein the perception of targets in the environment improves as the robot's distance from targets decreases. We assume no prior information about the total number of targets, their locations as well as their possible distribution within the environment. This work proposes a novel decision-making framework for the resulting constrained sequential decision-making problem by first converting it into a reward maximization problem on a product graph computed offline. It is then solved online as a Mixed-Integer Linear Program (MILP) where the knowledge about the environment is updated at each step, combining automata-based and MILP-based techniques. We demonstrate the efficacy of our approach with the help of a case study and present empirical evaluation in terms of expected regret. Furthermore, the runtime performance shows that online planning can be efficiently performed for moderately-sized grid environments.