Reviewerly/exHarmony huggingface dataset:
Challenges and Opportunities in Neuro-Symbolic Composition of Foundation Models

Trustworthy, resilient, and interpretable artificial intelligence (AI) is essential for effective operation of the Internet of Things (IoT) in adversarial environments. Such a robust and interpretable AI is needed to improve tactical coordination through scalability, corroboration, and context-aware intelligence. It is crucial to have robust machine learning (ML) models with characteristics such as low-supervision adaptability, decision explanations, and adaptive inference. Pre-trained large language models (LLMs) and foundation models (FMs) address some of these challenges, but are unpredictable and cannot directly solve complex tasks in mission-critical scenarios. However, their generalization capabilities make them potential building blocks for high-assurance AI/ML systems that compose multiple FMs and LLMs. In this paper, we propose combining neural foundation models (FMs) using symbolic programs that results in a more effective AI for adversarial conditions. Neuro-symbolic composition of FMs to solve complex tasks requires interactive and unambiguous specification of the intent, task decomposition into subtasks that can be solved by individual FMs, program synthesis for composing FMs, and neuro-symbolic inference that schedules inference of different FMs and combines their results. We give examples of such neuro-symbolic programs using foundation models to solve visual question-answering tasks such as out-of-context detection. This position paper identifies the challenges and opportunities in the neuro-symbolic composition of the large language models and foundation models.
Repetition-aware Image Sequence Sampling for Recognizing Repetitive Human Actions

In the field of video-based human action recognition (HAR), standard hand-crafted and deep learning-based approaches are constrained by the computational and memory requirements of their models and the length of the input sequence that can be processed during learning. Sampling techniques employing a windowed or a random clip cropping have been the simplest and most effective ways to cope with limitations on the maximum possible length of the input sequence. However, such designs do not guarantee that the correct ordering of the action steps is captured, or require several learning iterations. In this work we address this problem for the class of repetitive actions. Specifically, given a temporal segmentation of a repetitive action into its repetitive segments, we propose and develop novel approaches for ranking and selecting/sampling segments so as to improve learning in deep models for HAR. We show that by employing the proposed repetition-aware sampling schemes in state-of-the-art deep models for HAR, the action recognition accuracy is increased. The proposed approach is evaluated on existing datasets and on a new dataset that is tailored to the quantitative evaluation of the task at hand. The obtained results reveal how our approach performs in relation to various characteristics of the observed repetitive actions (repetition frequency, their effects on scene objects, etc) and demonstrate the performance improvements.
Unseen And Adverse Outdoor Scenes Recognition Through Event-based Captions

This paper presents EventCAP, i.e., event-based captions, for refined and enriched qualitative and quantitative captions by Deep Learning (DL) models and Vision Language Models (VLMs) with different tasks in a complementary manner. Indoor and outdoor images are used for object recognition and captioning. However, outdoor images in events change in wide ranges due to natural phenomena, i.e., weather changes. Such dynamical changes may degrade segmentation by illumination and object shape changes. This increases unseen objects and scenes under such adverse conditions. On the other hand, single state-of-art (SOTA) DLs and VLMs work with single or limited tasks, Therefore, this paper proposes EventCAP with captions with physical scales and objects’ surface properties. Moreover, an iterative VQA model is proposed to refine in-complete segmented images with the prompts. A higher se-mantic level in captions for real-world scene descriptions is experimentally shown compared to SOTA VLMs.
Just Ask Plus: Using Transcripts for VideoQA

Social-IQ 2.0 challenge is designed to benchmark recent AI technologies' skills to reason about social interactions, which is referred as Artificial Social Intelligence in the form of a VideoQA task. In this work, we use Just Ask and SpeechT5 models as feature extractors, and reason by adding one attention layer and two transformer encoders. Our best configuration reaches 53.35% accuracy on the validation set. The code is publicly available on GitHub.
OMG-Attack: Self-Supervised On-Manifold Generation of Transferable Evasion Attacks

Evasion Attacks (EA) are used to test the robustness of trained neural networks by distorting input data to misguide the model into incorrect classifications. Creating these attacks is a challenging task, especially with the ever increasing complexity of models and datasets. In this work, we introduce a self-supervised, computationally economical method for generating adversarial examples, designed for the unseen black-box setting. Adapting techniques from representation learning, our method generates on-manifold EAs that are encouraged to resemble the data distribution. These attacks are comparable in effectiveness compared to the state-of-the-art when attacking the model trained on, but are significantly more effective when attacking unseen models, as the attacks are more related to the data rather than the model itself. Our experiments consistently demonstrate the method is effective across various models, unseen data categories, and even defended models, suggesting a significant role for on-manifold EAs when targeting unseen models.
LEMMS: Label Estimation of Multi-feature Movie Segments

In the last few years, there has been an increasing amount of methods and algorithms that approach and automate different video and image editing tasks. A task that so far has not been investigated too much in depth is the analysis of video editing patterns. In this work, we present LEMMS (Label Estimation of Multi-feature Movie Segments), a methodology to analyze and label 30-second-long movie editing patterns based on the following editing features: shot size, shot subject, editing pace, and editing trend. LEMMS can identify more or less fine-grained editing classes using a multi-clustering approach. To evaluate the robustness of LEMMS in assigning correct labels the performance of an LSTM classifier is analyzed. For our study, we extracted 24 363 segments of movie scenes from the AVE [1] dataset. The performance of LEMMS in semi-automatic label identification for 30-second long movie segments is accurate, as the proposed approach has an overall accuracy of 92.8% for 50 classes.
Video Attribute Prototype Network: A New Perspective for Zero-Shot Video Classification

Video attributes, which leverage video contents to instantiate class semantics, play a critical role in diversifying semantics in zero-shot video classification, thereby facilitating semantic transfer from seen to unseen classes. However, few presences discuss video attributes, and most methods consider class names as class semantics that tend to be loosely defined. In this paper, we propose a Video Attribute Prototype Network (VAPNet) to generate video attributes that learns in-context semantics between video captions and class semantics. Specifically, we introduce a cross-attention module in the Transformer decoder by considering video captions as queries to probe and pool semantic-associated class-wise features. To alleviate noises in pre-extracted captions, we learn caption features through a stochastic representation derived from a Gaussian representation where the variance encodes uncertainties. We utilize a joint video-to-attribute and video-to-video contrastive loss to calibrate visual and semantic features. Experiments show that VAPNet significantly outperforms SoTA by relative improvements of 14.3% on UCF101 and 8.8% on HMDB51, and further surpasses the pre-trained vision-language SoTA by 4.1% and 17.2%. Code is available <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
MatchMakerNet: Enabling Fragment Matching for Cultural Heritage Analysis

Automating the reassembly of fragmented objects is a complex task with applications in cultural heritage preservation, paleontology, and medicine. However, the matching subtask of the reassembly process has received limited attention, despite its crucial role in reducing the alignment search space. To address this gap, we propose Match-MakerNet, a network architecture designed to automate the pairing of object fragments for reassembly. By taking two point clouds as input and leveraging graph convolution alongside a simplified version of DGCNN, MatchMakerNet achieves remarkable results. After training on the Artifact (synthetic) dataset, we achieve an accuracy of 87.31% in all-to-all comparisons between the fragments. In addition, it demonstrates robust generalization capabilities, achieving 86.93% accuracy on the Everyday (synthetic) dataset and 83.03% on the Puzzles 3D (real-world) dataset. These findings highlight the effectiveness and versatility of Match-MakerNet in solving the matching subtask.
A Simple Signal for Domain Shift

Test time domain adaptation has come to the forefront as a challenging scenario in recent times. Although single domain test-time adaptation has been well studied and shown impressive performance, this can be limiting when the model is deployed in a dynamic test environment. We explore this continual domain test time adaptation problem here. Specifically, we question if we can translate the effectiveness of single domain adaptation methods to continuous test-time adaptation scenario. We take a step towards bridging the gap between these two settings by proposing a domain shift detection mechanism and hence allowing us to employ the current test-time adaptation methods even in a continual setting. We propose to use the given source domain trained model to continually measure the similarity between the feature representations of the consecutive batches. A domain shift is detected when this measure crosses a certain threshold, which we use as a trigger to reset the model back to source and continue test-time adaptation. We demonstrate the effectiveness of our method by performing experiments across datasets, batch sizes and different single domain test-time adaptation baselines.
CLIP-Decoder : ZeroShot Multilabel Classification using Multimodal CLIP Aligned Representations

Multi-label classification is an essential task utilized in a wide variety of real-world applications. Multi-label zero-shot learning is a method for classifying images into multiple unseen categories for which no training data is available, while in general zero-shot situations, the test set may include observed classes. The CLIP-Decoder is a novel method based on the state-of-the-art ML-Decoder attention-based head. We introduce multi-modal representation learning in CLIP-Decoder, utilizing the text encoder to extract text features and the image encoder for image feature extraction. Furthermore, we minimize semantic mismatch by aligning image and word embeddings in the same dimension and comparing their respective representations using a combined loss, which comprises classification loss and CLIP loss. This strategy outperforms other methods and we achieve cutting-edge results on zero-shot multilabel classification tasks using CLIP-Decoder. Our method achieves an absolute increase of 3.9% in performance compared to existing methods for zero-shot learning multi-label classification tasks. Additionally, in the generalized zero-shot learning multi-label classification task, our method shows an impressive increase of almost 2.3%.
Knowledge Informed Sequential Scene Graph Verification Using VQA

We propose a new task, non localized scene graph verification, whose objective is to provide a justified expression of inconsistencies between the visual content of the image and its non-localized scene graph in order to diagnose errors or anticipate corrections. We introduce a sequential algorithm capable of detecting and proposing plausible corrections, taking into account the information already present in the scene graph and exploiting knowledge priors. Instead of relying on object detection that requires bounding box annotations, we use a simple visual question answering (VQA) as a proxy for visual content analysis. We show on the VG150 dataset that our strategy is efficient compared to a baseline adapted from a caption editing approach. We also show that our algorithm is able to efficiently correct corrupted scene graphs.
SC<sup>2</sup>GAN: Rethinking Entanglement by Self-correcting Correlated GAN Space

Generative Adversarial Networks (GANs) can synthesize realistic images, with the learned latent space shown to encode rich semantic information with various interpretable directions. However, due to the unstructured nature of the learned latent space, it inherits the bias from the training data where specific groups of visual attributes that are not causally related tend to appear together, a phenomenon also known as spurious correlations, e.g., age and eyeglasses or women and lipsticks. Consequently, the learned distribution often lacks the proper modelling of the missing examples. The interpolation following editing directions for one attribute could result in entangled changes with other attributes. To address this problem, previous works typically adjust the learned directions to minimize the changes in other attributes, yet they still fail on strongly correlated features. In this work, we study the entanglement issue in both the training data and the learned latent space for the StyleGAN2-FFHQ model. We propose a novel framework SC <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> GAN that achieves disentanglement by re-projecting low-density latent code samples in the original latent space and correcting the editing directions based on both the high-density and low-density regions. By leveraging the original meaningful directions and semantic region-specific layers, our framework interpolates the original latent codes to generate images with attribute combination that appears infrequently, then inverts these samples back to the original latent space. We apply our framework to pre-existing methods that learn meaningful latent directions and showcase its strong capability to disentangle the attributes with small amounts of low-density region samples added.
GPS-GLASS: Learning Nighttime Semantic Segmentation Using Daytime Video and GPS data

Semantic segmentation for autonomous driving should be robust against various in-the-wild environments. Nighttime semantic segmentation is especially challenging due to a lack of annotated nighttime images and a large domain gap from daytime images with sufficient annotation. In this paper, we propose a novel GPS-based training framework for nighttime semantic segmentation. Given GPS-aligned pairs of daytime and nighttime images, we perform cross-domain correspondence matching to obtain pixel-level pseudo supervision. Moreover, we conduct flow estimation between daytime video frames and apply GPS-based scaling to acquire another pixel-level pseudo supervision. Using these pseudo supervisions with a confidence map, we train a nighttime semantic segmentation network without any annotation from nighttime images. Experimental results demonstrate the effectiveness of the proposed method on several nighttime semantic segmentation datasets.
Deterministic Neural Illumination Mapping for Efficient Auto-White Balance Correction

Auto-white balance (AWB) correction is a critical operation in image signal processors for accurate and consistent color correction across various illumination scenarios. This paper presents a novel and efficient AWB correction method that achieves at least 35 times faster processing with equivalent or superior performance on high-resolution images for the current state-of-the-art methods. Inspired by deterministic color style transfer, our approach introduces deterministic illumination color mapping, leveraging learnable projection matrices for both canonical illumination form and AWB-corrected output. It involves feeding high-resolution images and corresponding latent representations into a mapping module to derive a canonical form, followed by another mapping module that maps the pixel values to those for the corrected version. This strategy is designed as resolution-agnostic and also enables seamless integration of any pre-trained AWB network as the backbone. Experimental results confirm the effectiveness of our approach, revealing significant performance improvements and reduced time complexity compared to state-of-the-art methods. Our method provides an efficient deep learning-based AWB correction solution, promising real-time, high-quality color correction for digital imaging applications. Source code is available at https://github.com/birdortyedi/DeNIM/
You can have your ensemble and run it too – Deep Ensembles Spread Over Time

Ensembles of independently trained deep neural networks yield uncertainty estimates that rival Bayesian networks in performance. They also offer sizable improvements in terms of predictive performance over single models. However, deep ensembles are not commonly used in environments with limited computational budget – such as autonomous driving – since the complexity grows linearly with the number of ensemble members. An important observation that can be made for robotics applications, such as autonomous driving, is that data is typically sequential. For instance, when an object is to be recognized, an autonomous vehicle typically observes a sequence of images, rather than a single image. This raises the question, could the deep ensemble be spread over time?In this work, we propose and analyze Deep Ensembles Spread Over Time (DESOT). The idea is to apply only a single ensemble member to each data point in the sequence, and fuse the predictions over a sequence of data points. We implement and experiment with DESOT for traffic sign classification, where sequences of tracked image patches are to be classified. We find that DESOT obtains the benefits of deep ensembles, in terms of predictive and uncertainty estimation performance, while avoiding the added computational cost. Moreover, DESOT is simple to implement and does not require sequences during training. Finally, we find that DESOT, like deep ensembles, outperform single models for out-of-distribution detection.
Noise Detection and Removing in Heart Sound Signals via Nuclear Norm Minimization Problems

Heart sound is an electric signal affected by some factors during the signal's recording process, which adds unwanted information to the signal. Recently, many studies have been interested in noise removal and signal recovery problems. The first step in signal processing is noise removal; many filters are used and proposed for treating this problem. Here, the Hankel matrix is implemented from a given signal and tries to clean the signal by overcoming unwanted information from the Hankel matrix. The first step is detecting unwanted information by defining a binary operator. This operator is defined under some threshold. The unwanted information replaces by zero, and the wanted information keeping in the estimated matrix. The resulting matrix contains zeros, so the problem is to find a low-rank matrix. Matrix completion is a heuristic NP-hard problem. It is a minimization problem defined by the matrix nuclear norm. In this paper, nuclear norm, and weighted nuclear norm minimization problems are derived to find a low-rank matrix of implemented Hankel matrix from the signal. A Robust Principal Component used to solve a low-rank-sparse matrix finds a low-rank Hankel matrix by solving a minimization problem numerically. The results show that the given methods are efficient in reconstructing and recovering the signals with a rate of more than 96%, with small values of mean square errors.
VAST: Vivify Your Talking Avatar via Zero-Shot Expressive Facial Style Transfer

Current talking face generation methods mainly focus on speech-lip synchronization. However, insufficient investigation on the facial talking style leads to a lifeless and monotonous avatar. Most previous works fail to imitate expressive styles from arbitrary video prompts and ensure the authenticity of the generated video. This paper proposes an unsupervised variational style transfer model (VAST) to vivify the neutral photo-realistic avatars. Our model consists of three key components: a style encoder that extracts facial style representations from the given video prompts; a hybrid facial expression decoder to model accurate speech-related movements; a variational style enhancer that enhances the style space to be highly expressive and meaningful. With our essential designs on facial style learning, our model is able to flexibly capture the expressive facial style from arbitrary video prompts and transfer it onto a personalized image renderer in a zero-shot manner. Experimental results demonstrate the proposed approach contributes to a more vivid talking avatar with higher authenticity and richer expressiveness.
Contrastive Image Synthesis and Self-supervised Feature Adaptation for Cross-Modality Biomedical Image Segmentation

This work presents a novel framework CISFA (Contrastive Image synthesis and Self-supervised Feature Adaptation) that builds on image domain translation and unsupervised feature adaptation for cross-modality biomedical image segmentation. Different from existing approaches, our method employs a one-sided generative model and incorporates a weighted patch-wise contrastive loss between sampled patches of the input image and the corresponding synthetic image, which serves as shape constraints. Furthermore, we notice that the generated images and input images share similar structural information but are in different modalities. To address this, we enforce contrastive losses on the generated images and the input images to train the encoder of a segmentation model to minimize the discrepancy between paired images in the learned embedding space. Compared with existing works that rely on adversarial learning for feature adaptation, such a method enables the encoder to learn domain-independent features in a more explicit way. We extensively evaluate our methods on segmentation tasks containing CT and MRI images for abdominal cavities and whole hearts. Experimental results show that the proposed framework not only outputs synthetic images with less distortion of organ shapes, but also outperforms state-of-the-art domain adaptation methods.