Reviewerly/exHarmony huggingface dataset:
Recurrent Information Synthesis for Down-Sampling Based Video Coding

With the popularity of high-definition videos, down-sampling based video coding has emerged a proven efficient method. In this paper, we non-uniformly down-sample the videos for compression with having intra-coded frames at the original resolution and inter-coded frames at a down-scaled resolution. After decoding, a recurrent information synthesis (RIS) module is proposed to super-resolve the down-sampled frames using a reference pool. Such a reference pool is initialized using decoded intra-coded frame first, and then recurrently updated by measuring the multi-scale features. Experimental results show that the proposed RIS achieves an average 5.73% BD-Rate gain against the VVC anchor in Low-delay P (LDP) setting, outperforming the state-of-the-art approaches. Additionally, our method requires fewer parameters and provides faster inference, demonstrating improved computational efficiency.
Field-informed Reinforcement Learning of Collective Tasks with Graph Neural Networks

Coordinating a multi-agent system of intelligent situated agents is a traditional research problem, impacted by the challenges posed by the very notion of distributed intelligence. These problems arise from agents acquiring information locally, sharing their knowledge, and acting accordingly in their environment to achieve a common, global goal. These issues are even more evident in large-scale collective adaptive systems, where agent interactions are necessarily proximity-based, thus making the emergence of controlled global collective behaviour harder.In this context, two main approaches have been proposed for creating distributed controllers out of macro-level task/goal descriptions: manual design, in which programmers build the controllers directly, and automatic design, which involves synthesizing programs using machine learning methods. In this paper, we consider a new hybrid approach called Field-Informed reinforcement learning (FIRL). We utilise manually designed computational fields (globally distributed data structures) to manage global agent coordination. Then, using Deep Q-learning in combination with Graph Neural Networks we enable the agents to learn the necessary local behaviour automatically to solve collective tasks, relying on those fields through local perception. We demonstrate the effectiveness of this new approach in simulated use cases where tracking and covering tasks for swarm robotics are successfully solved.
Using a Diverse Neural Network to Predict the Quality of Light Field Images

Light Fields (LF) capture both angular and spatial information of light rays traveling through free space, resulting in a richer representation of a scene from multiple viewpoints. However, the high dimensionality of LF data poses a challenge for compression and transmission algorithms, which can degrade visual quality. To address this issue, we propose a novel no-reference LF Image Quality Assessment (LF-IQA) method that accurately predicts the quality of complex LF content. Our method is based on a diverse neural network architecture that includes Convolutional Neural Network (CNN) blocks, Atrous Convolutional Layers (ACLs), and Long Short-Term Memory (LSTM) layers. The model architecture consists of two streams, each containing CNN, ACL, and LSTM layers, which take the horizontal and vertical epipolar plane LF images as input. CNN blocks extract basic input features, while ACL blocks extract high-level features. The LSTM layers are used to capture long-term dependencies and relationships among distortion-related features. Finally, the outputs of both streams are concatenated and fed into the regression block for quality prediction. Our results demonstrate that the proposed LF-IQA method is robust and outperforms current state-of-the-art methods, even for complex LF content.
FORT: Fisheye Online Realtime Tracking with an Improved Kalman Filter

The goal of human tracking is to detect people in a scene and assign them a unique identifier that the tracker will follow across multiple frames. Our tracker, FORT, implements deep learning solutions such as, YOLOv7 for detection, ResNeXt-50 for feature extraction and re-identification and an adapted Kalman filter for tracking. The goal is to present a real time tracking solution for the complex environment of top-view, fisheye images. The proposed solution is then compared with the BoT-SORT and StrongSORT trackers on a custom fisheye Multiple Object Tracking (MOT) Challenge dataset.
AVT-VQDB-UHD-1-Appeal: A UHD-1/4K Open Dataset for Video Quality and Appeal Assessment Using Modern Video Codecs

A number of factors play an important role in the perception of video quality for streaming and other services, key among them being encoding-related degradations. Hence, newer codecs are developed with the goal of optimizing video quality for a given encoding setting. Here, subjective studies are an efficient method to evaluate the performance of such newer codecs. Furthermore, contextual factors impact the perception of video quality, e.g., the appeal of the content itself. To this end, this paper presents a subjective study targeting both quality and appeal assessment of videos. For this purpose, a subjective study consisting of three different parts is conducted. Firstly, participants were asked to rate the appeal of the uncompressed UHD-1/4K source content with a duration of 8 - 10s each. Following this, the video quality of these source videos individually encoded with either the HEVC/H.265, AV1, or VVC/H.266 video codec was rated. A wide range of encoding conditions in terms of resolution (360p to 2160p) and bitrate (100kbps to 15mbps) is used to encode the videos, so as to enable the applicability of the data to real-world settings. In the last part, subjects are again asked to rate the appeal of the uncompressed source content. The results are analyzed to assess the impact of different encoding conditions on perceived video quality. In addition, the impact of appeal on video quality and vice-versa is also investigated. Furthermore, an objective quality assessment with different state-of-the-art full-reference, bitstream-based, and hybrid models including the newer codecs AV1 and VVC is presented. The subjective dataset including test design, subjective results, sources, and encoded audiovisual contents are made publicly available following an open science approach.
Comparison of Metrics for Predicting Image and Video Quality at Varying Viewing Distances

Viewing distance and display resolution have ar-guably a significant impact on perceived image quality; images seen on a mobile phone with high pixel density reveal fewer distortions than the same images seen on a large TV from a close distance. However, only a few image and video quality metrics account for the effect of viewing distance and resolution. Those that do, typically rely on contrast sensitivity functions (CSFs) of the visual system. Other metrics can be potentially adapted to different viewing distances by rescaling input images. In this paper, we investigate the performance of such adapted metrics together with those that natively account for viewing distance. The results for three testing datasets indicate that there is no evidence that the metrics based on the CSF outperform those that rely on rescaled images. Moreover, we found that both methods are not successful to account for the changes in quality introduced by the change in viewing distance. We conclude that accounting for viewing distances requires better models.
Retrieval-Based Methodology for Few-Sample Logo Recognition

Logo recognition describes the challenging task of detecting and classifying logos in digital images and videos. Former works approach logo recognition as a closed-set problem. However, this approach is accompanied by several shortcomings linked with its incapability of recognising new classes. In this paper, we propose an open-set logo recognition method, named REtrieval-based methodology For FEw-sample LOgo Recognition (REFELOR). REFELOR is composed by a generic logo detector and a feature extractor, allowing the generalization on unseen classes, using only a few samples per logo. That is, a single-stage generic logo detector is trained to detect logos in an input image. Then, feature representations for the detected logos are extracted, using the feature extractor, while the feature representations of a database containing only a few samples per class are also extracted. Finally, the detected logo representations are classified to the corresponding class based a similarity search in the representations of the aforementioned database. In addition, a regularization technique is applied to the feature extractor, providing further improvements. The experimental evaluation validates the effectiveness of the proposed method, outperforming current state-of-the-art logo recognition methods.
Energy Efficient Scheduling for Serverless Systems

Serverless computing, also referred to as Function-as-a-Service (FaaS), is a cloud computing model that has attracted significant attention and has been widely adopted in recent years. The serverless computing model offers an intuitive, event-based interface that makes the development and deployment of scalable cloud-based applications easier and cost-effective. An important aspect that has not been examined in these systems is their energy consumption during the application execution. One way to deal with this issue is to schedule the function invocations in an energy-efficient way. However, efficient scheduling of applications in a multi-tenant environment, like FaaS systems, poses significant challenges. The trade-off between the server’s energy usage and the hosted functions’ performance requirements needs to be taken into consideration. In this work, we propose an Energy Efficient Scheduler for orchestrating the execution of serverless functions so that it minimizes energy consumption while it satisfies the applications’ performance demands. Our approach considers real-time performance measurements and historical data and applies a novel DVFS technique to minimize energy consumption. Our detailed experimental evaluation using realistic workloads on our local cluster illustrates the working and benefits of our approach.
Quality-of-Experience Assessment for Ultra-Low Latency Live Streaming Videos

The quality-of-experience (QoE) assessment is vitally important for optimizing video streaming systems. Different from traditional streaming video, ultra-low latency live streaming (ULL-LS) video is susceptible to fluctuations in compression quality, playback smoothness, and latency in the time-varying network environment, which makes QoE evaluation extremely complicated. However, most existing streaming QoE researches focus on the effect of video compression quality and rebuffering events, without considering the joint impact of compression quality, playback smoothness, and latency on the user QoE, resulting in inaccurate quality assessment for ULL-LS videos. In this paper, we establish the first continuous-time subjective evaluation database for ULL-LS videos (ULL-LSVD), aiming to study the comprehensive influence of various compression and transmission distortions on the user's QoE. Additionally, we propose an objective assessment model for ULL-LS videos based on correlation analysis. Compared with other well-known objective metrics, our model shows superiority in the QoE evaluation of ULL-LS videos. We believe that our database and analysis will promote the QoE modeling research for ULL-LS videos, and we will make it public soon.
ConViViT - A Deep Neural Network Combining Convolutions and Factorized Self-Attention for Human Activity Recognition

The Transformer architecture has gained significant popularity in computer vision tasks due to its capacity to generalize and capture long-range dependencies. This characteristic makes it well-suited for generating spatiotemporal tokens from videos. On the other hand, convolutions serve as the fundamental backbone for processing images and videos, as they efficiently aggregate information within small local neighborhoods to create spatial tokens that describe the spatial dimension of a video. While both CNN-based architectures and pure transformer architectures are extensively studied and utilized by researchers, the effective combination of these two backbones has not received comparable attention in the field of activity recognition. In this research, we propose a novel approach that leverages the strengths of both CNNs and Transformers in an hybrid architecture for performing activity recognition using RGB videos. Specifically, we suggest employing a CNN network to enhance the video representation by generating a 128-channel video that effectively separates the human performing the activity from the background. Subsequently, the output of the CNN module is fed into a transformer to extract spatiotemporal tokens, which are then used for classification purposes. Our architecture has achieved new SOTA results with 90.05 %, 99.6%, and 95.09% on HMDB51. UCF101. and ETRI-Activity3D respectively.
Evolving Processing Pipelines for Industrial Imaging with Cartesian Genetic Programming

The reconfiguration of machine vision systems heavily depends on the collection and availability of large datasets, rendering them inflexible and vulnerable to even minor changes in the data. This paper proposes a refinement of Miller’s Cartesian Genetic Programming methodology, aimed at generating filter pipelines for image processing tasks. The approach is based on CGP-IP, but specifically adapted for image processing in industrial monitoring applications. The suggested method allows for retraining of filter pipelines using small datasets; this concept of self-adaptivity renders high-precision machine vision more resilient to faulty machine settings or changes in the environment and provides compact programs. A dependency graph is introduced to rule out invalid pipeline solutions. Furthermore, we suggest to not only generate pipelines from scratch, but store and reapply previous solutions and re-adjust filter parameters. Our modifications are designed to increase the likelihood of early convergence and improvement in the fitness indicators. This form of self-adaptivity allows for a more resource-efficient configuration of image filter pipelines with small datasets.
NUVER: Network Based Vulnerability Visualizer

The deluge of attacks on IT infrastructure has necessitated the understanding of cybersecurity vulnerabilities for mitigation. A two-step novel model is developed to harness the time-stamped data to analyze the vulnerability trends. In the first step of the model, a text network is created from the vulnerability report. A software is constructed that analyzes vulnerabilities by leveraging natural language processing and network theory. The network theoretic study provides details of the complex relationships between vulnerabilities. The visualizations from the tool provide information for security researchers for an in-depth study of vulnerabilities.
Video-Based Stabilized 3D Face Alignment Using Temporal Multi-Discrimination

Existing 3D face alignment primarily aim to achieve accurate face alignment result for a static facial image. While these methods have strong alignment performance under large poses, occlusion, and extreme lighting conditions, they often result in trembling artifacts in video-based sequential 3D face alignment. Reducing temporal misalignment remains a challenging task because a single misaligned frame can propagate errors to other frames along the temporal axis. To address this issue, we propose a novel temporal discriminating scheme that learns the distribution gap between the face alignment results and ground truth face animation. By leveraging the discrimination results as a guide, the proposed method can effectively align the 3D faces to the input video by reducing temporal trembling artifacts. To effectively learn the distribution gap, we introduce a multi-discriminating scheme that separately discriminates facial animation based on identity and expression changes. It enables the proposed method to produce a stabilized alignment result, especially in dynamic and fast movement. Through extensive experiments in both qualitative and quantitative evaluations, it is confirmed that our method outperforms state-of-the-art 3D face alignment methods by animating stabilized results in the video.
MetaHumans Help to Evaluate Deepfake Generators

The progress achieved in deepfake technology has been remarkable; however, evaluating the resulting videos and comparing different generators remains challenging. A primary concern arises from the lack of ground-truth data, except for self-reenactment scenarios. Additionally, available datasets may have inherent limitations, such as lacking expected animations or demonstrating inadequate subject diversity. Furthermore, there are ethical and privacy concerns when using real individuals' faces in such applications. This paper goes beyond the state-of-the-art dealing with the evaluation of deepfake generators by introducing an innovative dataset featuring MetaHumans. Our dataset ensures the availability of ground-truth data and encompasses diverse facial expressions, variations in pose and illumination conditions, and combinations of these factors. Additionally, we meticulously control and verify the expected animations within the dataset. The proposed dataset enables accurate evaluation of cross-reenactment generated images. By utilizing various established metrics, we demonstrate a high degree of correlation between the generator's scores obtained from deep-fake videos of Metahumans and those obtained from deepfake videos of real persons. The synthesized MetaHuman dataset can be accessed at: https://github.com/SaharHusseini/MMSP_2023
Plug-and-Play Joint Image Deblurring and Detection

Object detection is a widely researched topic in computer vision; however, current models often struggle with processing degraded images with adverse imaging conditions like low light, blur, and haze. Conventional approaches involve a separate image recovery network prior to detection, resulting in a large network that is sub-optimal in performance. Alternatively, in this study, we propose a lightweight plug-and-play solution to improve the performance of object detectors on degraded images, without the need for retraining the vision task, i.e., detector network. This solution utilizes an image enhancement plug-in subnetwork that can be turned on and off for the main vision task network, leading to improved detection accuracy without sacrificing inference time. Empirically, our proposed model achieved a 48.9% mean average precision (mAP) on a degraded Pascal VOC dataset, compared to the baseline model at 26.7%.
INDigo: An INN-Guided Probabilistic Diffusion Algorithm for Inverse Problems

Recently it has been shown that using diffusion models for inverse problems can lead to remarkable results. However, these approaches require a closed-form expression of the degradation model and can not support complex degradations. To overcome this limitation, we propose a method (INDigo) that combines invertible neural networks (INN) and diffusion models for general inverse problems. Specifically, we train the forward process of INN to simulate an arbitrary degradation process and use the inverse as a reconstruction process. During the diffusion sampling process, we impose an additional data-consistency step that minimizes the distance between the intermediate result and the INN-optimized result at every iteration, where the INN-optimized image is composed of the coarse information given by the observed degraded image and the details generated by the diffusion process. With the help of INN, our algorithm effectively estimates the details lost in the degradation process and is no longer limited by the requirement of knowing the closed-form expression of the degradation model. Experiments demonstrate that our algorithm obtains competitive results compared with recently leading methods both quantitatively and visually. Moreover, our algorithm performs well on more complex degradation models and real-world low-quality images.
Measuring Underwater Image Quality with Depthwise-Separable Convolutions and Global-Sparse Attention

Underwater Image Quality Assessment (UIQA) poses specific challenges due to blur, dispersion of light, and color garbling caused by water turbulence. Images captured by Autonomous Underwater Vehicles (AVS) correlate poorly along their RGB channels, as blue and green channel components dominate. We propose a novel lightweight no-reference (NR) UIQA architecture based on depthwise-separable convolutions and global sparse attention that surpasses the existing deep learning architectures in terms of performance as well as parameter count and Multiply-Accumulate operations (MACs). Proposed model gives 3.64% increase in SRCC and 6.11% increase in PLCC in comparison to state-of-the-art for UEQAB dataset. Our work also illustrates the effectiveness of depthwise separable convolutions for underwater image quality assessment. Our empirical analysis confirms less redundancy among the channels of a feature map when depthwise separable convolutions are used compared to standard convolutions by increasing representational efficiency.
Improved Transform Structures for Learned Wavelet-Like Fully Scalable Image Compression

This paper studies features and performance of different structures for learned wavelet-like transforms in fully scalable image compression. Specifically, we explore different neural network topologies and various arrangements of lifting steps to improve the existing wavelet transform. Experimental results strongly suggest that the proposed proposal-opacity network topology, comprising a collection of linear predictions modulated by non-linear opacities, performs better than the other considered designs. Results also show that augmenting a good base wavelet transform with two learned lifting steps performs significantly better than other learned lifting structures, achieving 26.8% bit-rate savings over JPEG 2000.
Discriminative Deep Feature Visualization for Explainable Face Recognition

Despite the huge success of deep convolutional neural networks in face recognition (FR) tasks, current methods lack explainability for their predictions because of their "blackbox" nature.In recent years, studies have been carried out to give an interpretation of the decision of a deep FR system.However, the affinity between the input facial image and the extracted deep features has not been explored.This paper contributes to the problem of explainable face recognition by first conceiving a face reconstruction-based explanation module, which reveals the correspondence between the deep feature and the facial regions.To further interpret the decision of an FR model, a novel visual saliency explanation algorithm has been proposed.It provides insightful explanation by producing visual saliency maps that represent similar and dissimilar regions between input faces.A detailed analysis has been presented for the generated visual explanation to show the effectiveness of the proposed method.