Reviewerly/exHarmony huggingface dataset:
Hashing for Structure-Based Anomaly Detection

We focus on the problem of identifying samples in a set that do not conform to structured patterns represented by low-dimensional manifolds. An effective way to solve this problem is to embed data in a high dimensional space, called Preference Space, where anomalies can be identified as the most isolated points. In this work, we employ Locality Sensitive Hashing to avoid explicit computation of distances in high dimensions and thus improve Anomaly Detection efficiency. Specifically, we present an isolation-based anomaly detection technique designed to work in the Preference Space which achieves state-of-the-art performance at a lower computational cost. Code is publicly available at https://github.com/ineveLoppiliF/Hashing-for-Structure-based-Anomaly-Detection .
CarPatch: A Synthetic Benchmark for Radiance Field Evaluation on Vehicle Components

Neural Radiance Fields (NeRFs) have gained widespread recognition as a highly effective technique for representing 3D reconstructions of objects and scenes derived from sets of images. Despite their efficiency, NeRF models can pose challenges in certain scenarios such as vehicle inspection, where the lack of sufficient data or the presence of challenging elements (e.g. reflections) strongly impact the accuracy of the reconstruction. To this aim, we introduce CarPatch, a novel synthetic benchmark of vehicles. In addition to a set of images annotated with their intrinsic and extrinsic camera parameters, the corresponding depth maps and semantic segmentation masks have been generated for each view. Global and part-based metrics have been defined and used to evaluate, compare, and better characterize some state-of-the-art techniques. The dataset is publicly released at https://aimagelab.ing.unimore.it/go/carpatch and can be used as an evaluation guide and as a baseline for future work on this challenging topic.
Evaluation of 3D Reconstruction Pipelines Under Varying Imaging Conditions

We present IVL-SYNTHSFM-v2, a dataset of 4000 images depicting five objects from different viewpoints and under different lighting conditions and acquisition setups. The images have been rendered from 3D scenes with varying camera positions, lights, camera depth of field, and motion blur. Images depict one of five objects each acquired using eight different acquisition setups (scenes). 100 images have been rendered from each scene. The dataset is intended to be used to evaluate 3D reconstruction algorithms. The varying imaging conditions are introduced to challenge the algorithms to address realistic, non-ideal situations. The dataset provides pixel-precise ground truth to perform accurate evaluations. We demonstrate the usefulness of IVL-SYNTHSFM-v2 by assessing state-of-the-art 3D reconstruction algorithms.
Large Class Separation is Not What You Need for Relational Reasoning-Based OOD Detection

Standard recognition approaches are unable to deal with novel categories at test time. Their overconfidence on the known classes makes the predictions unreliable for safety-critical applications such as healthcare or autonomous driving. Out-Of-Distribution (OOD) detection methods provide a solution by identifying semantic novelty. Most of these methods leverage a learning stage on the known data, which means training (or fine-tuning) a model to capture the concept of normality. This process is clearly sensitive to the amount of available samples and might be computationally expensive for on-board systems. A viable alternative is that of evaluating similarities in the embedding space produced by large pre-trained models without any further learning effort. We focus exactly on such a fine-tuning-free OOD detection setting. This works presents an in-depth analysis of the recently introduced relational reasoning pre-training and investigates the properties of the learned embedding, highlighting the existence of a correlation between the inter-class feature distance and the OOD detection accuracy. As the class separation depends on the chosen pre-training objective, we propose an alternative loss function to control the inter-class margin, and we show its advantage with thorough experiments.
LCMV: Lightweight Classification Module for Video Domain Adaptation

Video action recognition models exhibit high performance on in-distribution data but struggle with distribution shifts in test data. To mitigate this issue, Unsupervised Domain Adaptation (UDA) methods have been proposed, consisting in training on labeled data from a source domain and incorporating unlabeled test data from a target domain to reduce the domain gap. This requires simultaneous access to data from both domains, which may not be practical in real-world scenarios due to privacy issues. A more practical approach called Source-Free Domain Adaptation (SFDA) has been recently proposed, which consists in adapting a well-trained source model using only unlabeled target data. However, existing SFDA methods are computationally intensive and designed for specific architectures. In this paper, we propose an approach called Lightweight Classification Module for Video Domain Adaptation (LCMV). LCMV is based on a backpropagation-free prototypical algorithm, which efficiently adapts a source model using unlabeled target data only. Results on two popular datasets, HMDB-UCF $$_{full}$$ and EPIC-Kitchens-55, show significant improvements of LCMV compared to the previous state-of-the-art SFDA methods, and competitive results when compared to state-of-the-art UDA methods.
Narrative Perspectives and Embodiment in Cinematic Virtual Reality

Along with the technological advancements of virtual reality over the years, has come the emergence of Cinematic Virtual Reality (CVR), where immersive 360-degree video approaches the high-quality found in feature film. Extensive research has been done on embodiment and presence in relation to Virtual Reality (VR), however, there is a lack of existing literature on the narrative effects of embodiment and perspective in narrative VR films. Exploring the concept of viewer embodiment and its connection to the cinematic concept of narrative perspective, we conduct a review of literature in relation to CVR and flat screen cinema, selecting five CVR films to conduct an analysis of how cinematic techniques for establishing perspective and embodiment can be translated from flat screen cinema. Considering embodiment and perspective in CVR, we propose a spectrum of embodiment between extreme distancing from and extreme identification with characters in the narrative. Areas for future exploration are considered in light of the lack of research in this area.
Dynamic Local Filters in Graph Convolutional Neural Networks

Over the last few years, we have seen increasing data generated from non-Euclidean domains, usually represented as graphs with complex relationships. Graph Neural Networks (GNN) have gained a high interest because of their potential in processing graph-structured data. In particular, there is a strong interest in performing convolution on graphs using specific GNN architectures, generally called Graph Convolutional Neural Networks (GCNN). This paper presents a novel method to adapt the behaviour of a GCNN using an input-based dynamically generated filter. Notice that the idea of adapting the network behaviour to the inputs they process to maximize the total performances has aroused much interest in the neural networks literature over the years. The experimental assessment confirms the capabilities of the proposed approach, achieving promising results using simple architectures with a low number of filters.
FERMOUTH: Facial Emotion Recognition from the MOUTH Region

People use various nonverbal communicative channels to convey emotions, among which facial expressions are considered the most important ones. Consequently, automatic Facial Expression Recognition (FER) is a crucial task for enhancing computers' perceptive abilities, particularly in human-computer interaction. Although state-of-the-art FER systems can identify emotions from the entire face, situations may arise where occlusions prevent the entire face from being visible. During the COVID-19 pandemic, many FER systems have been developed for recognizing emotions from the eye region due to the obligation to wear a mask. However, in many situations, the eyes may be covered, for instance, by sunglasses or virtual reality devices. In this paper, we faced the problem of developing a FER system that solely considers the mouth region and classifies emotions using only the lower part of the face. We tested the effectiveness of this FER system in recognizing emotions from the lower part of the face and compared the results to a FER system trained on the same datasets using the same approach on the entire face. As expected, emotions primarily associated with the mouth region (e.g., happiness, surprise) were recognized with minimal loss compared to the entire face. Nevertheless, even though most negative emotions were not accurately detected using only the mouth region, in cases where the face is partially covered, this area may still provide some information about the displayed emotion.
Quasi-Online Detection of Take and Release Actions from Egocentric Videos

In this paper, we considered the problem of detecting object take and release actions from untrimmed egocentric videos in an industrial domain. Rather than requiring that actions are recognized as they are observed, in an online fashion, we propose a quasi-online formulation in which take and release actions can be recognized shortly after they are observed, but keeping a low latency. We contribute a problem formulation, an evaluation protocol, and a baseline approach that relies on state-of-the-art components. Experiments on ENIGMA, a newly collected dataset of egocentric untrimmed videos of human-object interactions in an industrial scenario, and on THUMOS'14 show that the proposed approach achieves promising performance on quasi-online take/release action recognition and outperforms methods for online detection of action start on THUMOS'14 by $$+8.64\%$$ when an average latency of 2.19s is allowed. Code and supplementary material are available at https://github.com/fpv-iplab/Quasi-Online-Detection-Take-Release .
GLR: Gradient-Based Learning Rate Scheduler

Training a neural network is a complex and time-consuming process because of many combinations of hyperparameters that have to be adjusted and tested. One of the most crucial hyperparameters is the learning rate which controls the speed and direction of updates to the weights during training. We proposed an adaptive scheduler called Gradient-based Learning Rate scheduler (GLR) that significantly reduces the tuning effort thanks to a single user-defined parameter. GLR achieves competitive results in a very wide set of experiments compared to the state-of-the-art schedulers and optimizers. The computational cost of our method is trivial and can be used to train different network topologies.
Benchmarking of Blind Video Deblurring Methods on Long Exposure and Resource Poor Settings

This paper presents a benchmark evaluation of blind video deblurring methods in specific challenging settings. The employed videos are affected by severe deblurring artifacts and acquisition conditions (e.g., low resolution, high exposure, camera motion, complex scene motion, etc.). An in depth state of the art investigation has been carried out. Then, a specific set of methods based on mathematical optimization with image priors has been involved in our benchmark evaluation. The selected methods have been evaluated quantitatively and qualitatively.
Time-Aware Circulant Matrices for Question-Based Temporal Localization

Episodic memory involves the ability to recall specific events, experiences, and locations from one's past. Humans use this ability to understand the context and significance of past events, while also being able to plan for future endeavors. Unfortunately, episodic memory can decline with age and certain neurological conditions. By using machine learning and computer vision techniques, it could be possible to "observe" the daily routines of elderly individuals from their point of view and provide customized healthcare and support. For example, it could help an elderly person remember whether they have taken their daily medication or not. Therefore, considering the important impact on healthcare and societal assistance, this problem has been recently discussed in the research community, naming it Episodic Memory via Natural Language Queries. Recent approaches to this problem mostly rely on the literature related to similar fields, but contextual information from past and future clips is often unexplored. To address this limitation, in this paper we propose the Time-aware Circulant Matrices technique, which aims at introducing awareness of the surrounding clips into the model. In the experimental results, we present the robustness of our method by ablating its components, and confirm its effectiveness on the Ego4D public dataset, achieving an absolute improvement of more than 1% on R@5.
Consensus Ranking for Efficient Face Image Retrieval: A Novel Method for Maximising Precision and Recall

Efficient face image retrieval, i.e. searching for existing photographs of a person in unlabelled photo collections using a query photo, is evaluated for a novel method to find the top n results for Consensus Ranking. The approach aims to maximise precision and recall by using the retrieved photos, all ranked on similarity. The proposed method aims to retrieve all photos of the queried person while excluding images of other individuals. To achieve this, the method uses the top n results as temporary queries, recalculates similarities, and combines the obtained ranked lists to produce a better overall ranking. The method includes a novel and reliable procedure for selecting n, which is evaluated on two datasets, and considers the impact of age variation in the datasets.
FEAD-D: Facial Expression Analysis in Deepfake Detection

As the development of deep learning (DL) techniques has progressed, the creation of convincing synthetic media, known as deepfakes, has become increasingly easy, raising significant concern about the use of these videos to spread false information and potentially manipulate public opinion. In recent years, deep neural networks, such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), have been used for deepfake detection systems, exploiting the inconsistencies and the artifacts introduced by generation algorithms. Taking into account the main limitation of fake videos to realistically reproduce the natural human emotion patterns, in this paper, we present FEAD-D, a publicly available tool for deepfake detection performing facial expression analysis. Our system exploits data from the DeepFake Detection Challenge (DFDC) and consists of a model based on bidirectional Long Short-Term Memory (BiLSTM) capable of detecting a fake video in about two minutes with an overall accuracy of 84.29% on the test set (i.e. comparable with the current state-of-the-art, while consisting of fewer parameters), showing that emotional analysis can be used as a robust and reliable method for deepfake detection.
LBKENet:Lightweight Blur Kernel Estimation Network for Blind Image Super-Resolution

Blind image super-resolution (Blind-SR) is the process of leveraging a low-resolution (LR) image, with unknown degradation, to generate its high-resolution (HR) version. Most of the existing blind SR techniques use a degradation estimator network to explicitly estimate the blur kernel to guide the SR network with the supervision of ground truth (GT) kernels. To solve this issue, it is necessary to design an implicit estimator network that can extract discriminative blur kernel representation without relying on the supervision of ground-truth blur kernels. We design a lightweight (LBKENet) approach for blind super-resolution (Blind-SR) that estimates the blur kernel and restores the HR image based on a deep convolutional neural network (CNN) and a deep super-resolution residual convolutional generative adversarial network. Since the blur kernel for blind image SR is unknown, following the image formation model of the blind super-resolution problem, we first introduce a neural network-based model to estimate the blur kernel. This is achieved by (i) a Super Resolver that, from a low-resolution input, generates the corresponding SR image; and (ii) an Estimator Network generating the blur kernel from the input datum. The output of both models is used in a novel loss formulation. The proposed network is end-to-end trainable. The methodology proposed is substantiated by both quantitative and qualitative experiments. Results on benchmarks demonstrate that our computationally efficient approach (12 $$\times $$ fewer parameters than the state-of-the-art models) performs favorably with respect to approaches that have less number of parameters and can be used on devices with limited computational capabilities.
Smoothing and Transition Matrices Estimation to Learn with Noisy Labels

In recent years, there has been impressive progress in learning with noisy labels, particularly in leveraging a small set of clean data. Meta-learning-based label correction techniques have further advanced performance by correcting noisy labels during training. However, these methods require multiple back-propagation steps, which considerably slows down the training process. Alternatively, some researchers have attempted to estimate the label transition matrix on-the-fly to address the issue of noisy labels. These approaches are more robust and faster than meta-learning-based techniques. The use of the transition matrix makes the classifier skeptical about all corrected samples, thereby mitigating the problem of label noise. We propose a novel three-head architecture that can efficiently estimate the label transition matrix and two new label smoothing matrices at each iteration. Our approach enables the estimated matrices to closely follow the shifting noise and reduce over-confidence on classes during classifier model training. We report extensive experiments on synthetic and real world noisy datasets, achieving state of the art performance on synthetic variants of CIFAR-10/100 and on the challenging Clothing1M datasets. Code at https://github.com/z3n0e/STM .
BLUES: Before-reLU-EStimates Bayesian Inference for Crowd Counting

Ensuring the trustworthiness of artificial intelligence and machine learning systems is becoming a crucial requirement given their widespread applications, including crowd counting, which we focus on in this work. This is often addressed by integrating uncertainty measures into their predictions. Most Bayesian uncertainty quantification techniques use a Gaussian approximation of the output, whose variance is interpreted as the uncertainty measure. However, in the case of neural network models for crowd counting based on density estimation, where the ReLU activation function is used for the output units, such a prior may lead to an approximated distribution with a significant mass on negative values, although they cannot be produced by the ReLU activation. Interestingly, we found that this is related to "false positive" pedestrian localisation errors in the density map. We propose to address this issue by shifting the Bayesian Inference Before the reLU EStimates (BLUES). This modification allows us to estimate a probability distribution both on the people density and the people presence in each pixel. This allows us to compute a crowd segmentation map, which we exploit for filtering out false positive localisations. Results on several benchmark data sets provide evidence that our BLUES approach allows for improving the accuracy of the estimated density map and the quality of the corresponding uncertainty measure.
Enhancing Hierarchical Vector Quantized Autoencoders for Image Synthesis Through Multiple Decoders

Vector Quantized Variational Autoencoders (VQ-VAEs) have gained popularity in recent years due to their ability to represent images as discrete sequences of tokens that index a learned codebook of vectors, enabling efficient image compression. One variant of particular interest is VQ-VAE 2, which extends previous works by representing images as a hierarchy of sequences, resulting in finer-grained representations. In this study, we further enhance such hierarchical autoencoder approach by introducing multiple decoders, which allow to represent images as a sum of multi-scale contributions in the pixel space. Our proposed model, the Multi Scale (MS) VQ-VAE, not only enables better control over the encoding of each sequence (resulting in improved explainability and codebook usage) but, as a consequence, also shows advantages in image synthesis. Our experiments demonstrate that the MS-VQVAE achieves comparable or superior reconstructions on various datasets and resolutions, as well as greater stability across runs. Moreover, we include a proof-of-concept trial to showcase the potential applications of our model in image synthesis.
Medical Transformers for Boosting Automatic Grading of Colon Carcinoma in Histological Images

Developing computer-aided approaches for cancer diagnosis and grading is receiving an uprising demand since this could take over intra- and inter-observer inconsistency, speed up the screening process, allow early diagnosis, and improve the accuracy and consistency of the treatment planning processes. The third most common cancer worldwide and the second most common in women is ColoRectal Cancer (CRC). Grading CRC is a key task in planning appropriate treatments and estimating the response to them. Automatic systems have the potential to speed up and make it more robust but, unfortunately, the most recent and promising machine learning techniques have not been applied for automatic CRC grading so far. For example, there is no work exploiting transformer networks, which outperform convolutional neural networks (CNN) and are replacing them in many applications, for CRC detection and grading at a large scale. To fill this gap, in this work, a transformer-based network endowed with an additional control mechanism in the self-attention module is exploited to understand discriminative regions in large histological images. These relevant regions have been used to train the most suited Convolutional Neural Network (as emerged from recent research findings) for the automatic grading of CRC. The experimental proofs on the largest publicly available CRC dataset demonstrated marked improvement with respect to the leading state-of-the-art approaches relying on CNN.
Hierarchical Pretrained Backbone Vision Transformer for Image Classification in Histopathology

Histopathology plays a crucial role in clinical diagnosis, treatment planning, and research by enabling the examination of diseases in tissues and organs. However, the manual analysis of histopathological images is time-consuming and labor-intensive, requiring expert pathologists. To address this issue, this work proposes a novel architecture called Hierarchical Pretrained Backbone Vision Transformer for automated histopathological image classification, a critical tool in clinical diagnosis, treatment planning, and research. Current deep learning-based methods for image classification require a large amount of labeled data and significant computational resources to be trained effectively. By leveraging pretrained Visual Transformer backbones, our approach can classify histopathology images, achieve state-of-the-art performance, and take advantage of the pretrained backbones' weights. We evaluated it on the Chaoyang histopathology dataset, comparing it with other state-of-the-art Visual Transformers. The experimental results demonstrate that the proposed architecture outperforms the others, indicating its potential to be an effective tool for histopathology image classification.