Reviewerly/exHarmony huggingface dataset:
FRR-Net: A Real-Time Blind Face Restoration and Relighting Network

Face restoration models that mitigate low light, mixed lighting, poor camera quality conditions can benefit various applications, including video conferencing, image capture apps, among other uses. Many different models exist to address this problem. Although recent models generate impressive and high-fidelity faces, several important challenges remain, such as model efficiency, realistic texture and facial components, low-light environments, and screen illumination on the face. To tackle these challenges, we propose a simple, yet effective model called Face Restoration and Relighting Network (FRR-Net). The FRR-Net architecture includes an encoder-decoder model with a parallel distortion classifier which predicts the distortion types during training. This model is systematically scaled to balance network depth and width for better performance and efficiency trade-off. In addition, to generate the enhanced facial region, FRR-Net also utilizes a facial segmentation mask during the training, which not only helps the model performance but can also be used for further post-production uses. Furthermore, this work integrates a wide range of data degradation techniques to generate data for training to tackle both face enhancement and relighting. We demonstrate the effectiveness of our method by comparing it with several recent face restoration models. FRR-Net is computationally efficient and can perform inference at 13ms per frame on a low-powered Neural Processing Unit making it suitable for real-time face restoration applications.
QuickSRNet: Plain Single-Image Super-Resolution Architecture for Faster Inference on Mobile Platforms

In this work, we present QuickSRNet, an efficient super-resolution architecture for real-time applications on mobile platforms. Super-resolution clarifies, sharpens, and upscales an image to higher resolution. Applications such as gaming and video playback along with the ever-improving display capabilities of TVs, smartphones, and VR headsets are driving the need for efficient upscaling solutions. While existing deep learning-based super-resolution approaches achieve impressive results in terms of visual quality, enabling real-time DL-based super-resolution on mobile devices with compute, thermal, and power constraints is challenging. To address these challenges, we propose QuickSRNet, a simple yet effective architecture that provides better accuracy-to-latency trade-offs than existing neural architectures for single-image super-resolution. We present training tricks to speed up existing residual-based super-resolution architectures while maintaining robustness to quantization. Our proposed architecture produces 1080p outputs via 2× upscaling in 2.2 ms on a modern smartphone, making it ideal for high-fps real-time applications.
Gatha: Relational Loss for enhancing text-based style transfer

Text-based style transfer is a promising area of research that enables the generation of stylistic images from plain text descriptions. However, the existing text-based style transfer techniques do not account for the subjective nature of prompt descriptions or the nuances of style-specific vocabulary during the optimization process. This severely limits the stylistic expression of the predominant models. In this paper, we address this gap by proposing Gatha, which incorporates subjectivity by introducing an additional loss function that enforces the relationship between stylized images and a proxy style set to be similar to the relationship between the text description and the proxy style set. We substantiate the effectiveness of Gatha through both qualitative and quantitative analysis against the existing state-of-the-art models and show that our approach allows for consistently improved stylized images.
High-Resolution Synthetic RGB-D Datasets for Monocular Depth Estimation

Accurate depth maps are essential in various applications, such as autonomous driving, scene reconstruction, point-cloud creation, etc. However, monocular-depth estimation (MDE) algorithms often fail to provide enough texture & sharpness, and also are inconsistent for homogeneous scenes. These algorithms mostly use CNN or vision transformer-based architectures requiring large datasets for supervised training. But, MDE algorithms trained on available depth datasets do not generalize well and hence fail to perform accurately in diverse real-world scenes. Moreover, the ground-truth depth maps are either lower resolution or sparse leading to relatively inconsistent depth maps. In general, acquiring a high-resolution ground truth dataset with pixel-level precision for accurate depth prediction is an expensive, and time-consuming challenge.In this paper, we generate a high-resolution synthetic depth dataset (HRSD) of dimension 1920 × 1080 from Grand Theft Auto (GTA-V), which contains 100,000 color images and corresponding dense ground truth depth maps. The generated datasets are diverse and have scenes from indoors to outdoors, from homogeneous surfaces to textures. For experiments and analysis, we train the DPT algorithm, a state-of-the-art transformer-based MDE algorithm on the proposed synthetic dataset, which significantly increases the accuracy of depth maps on different scenes by 9%. Since the synthetic datasets are of higher resolution, we propose adding a feature extraction module in the transformer’s encoder and incorporating an attention-based loss, further improving the accuracy by 15 %.
CFDP: Common Frequency Domain Pruning

As the saying goes, sometimes less is more – and when it comes to neural networks, that couldn’t be more true. Enter pruning, the art of selectively trimming away unnecessary parts of a network to create a more streamlined, efficient architecture. In this paper, we introduce a novel end-to-end pipeline for model pruning via the frequency domain. This work aims to shed light on the interoperability of intermediate model outputs and their significance beyond the spatial domain. Our method, dubbed Common Frequency Domain Pruning (CFDP) aims to extrapolate common frequency characteristics defined over the feature maps to rank the individual channels of a layer based on their level of importance in learning the representation. By harnessing the power of CFDP, we have achieved state-of-the-art results on CIFAR-10 with GoogLeNet reaching an accuracy of 95.25%, that is, +0.2% from the original model. We also outperform all benchmarks and match the original model’s performance on ImageNet, using only 55% of the trainable parameters and 60% of the FLOPs. In addition to notable performances, models produced via CFDP exhibit robustness to a variety of configurations including pruning from untrained neural architectures, and resistance to adversarial attacks. The implementation code can be found at https://github.com/Skhaki18/CFDP.
Impact of Pseudo Depth on Open World Object Segmentation with Minimal User Guidance

Pseudo depth maps are depth map predicitions which are used as ground truth during training. In this paper we leverage pseudo depth maps in order to segment objects of classes that have never been seen during training. This renders our object segmentation task an open world task. The pseudo depth maps are generated using pretrained networks, which have either been trained with the full intention to generalize to downstream tasks (LeRes and MiDaS), or which have been trained in an unsupervised fashion on video sequences (MonodepthV2). In order to tell our network which object to segment, we provide the network with a single click on the object’s surface on the pseudo depth map of the image as input. We test our approach on two different scenarios: One without the RGB image and one where the RGB image is part of the input. Our results demonstrate a considerably better generalization performance from seen to unseen object types when depth is used. On the Semantic Boundaries Dataset we achieve an improvement from 61.57 to 69.79 IoU score on unseen classes, when only using half of the training classes during training and performing the segmentation on depth maps only.
A Geometric and Photometric Exploration of GAN and Diffusion Synthesized Faces

Classic computer-generated imagery is produced by modeling 3D scene geometry, the surrounding illumination, and a virtual camera. As a result, rendered images accurately capture the geometry and physics of natural scenes. In contrast, AI-generated imagery is produced by learning the statistical distribution of natural scenes from a large set of real images. Without an explicit 3D model of the world, we wondered how accurately synthesized content captures the 3D geometric and photometric properties of natural scenes. From a diverse set of real, GAN- and diffusion-synthesized faces, we estimate a 3D geometric model of the face, from which we estimate the surrounding 3D photometric environment. We also analyze 2D facial features – eyes and mouth – that have been traditionally difficult to accurately render. Using these models, we provide a quantitative analysis of the 3D and 2D realism of synthesized faces.
Background Enhancement of Exercise and Performance Learning Assistant System Platform for Yoga Pose Practices

To support exercise or performance learning at home, we have developed Exercise and Performance Learning Assistant System (EPLAS) as a web application system using Node.js. In EPLAS, the user can practice Yoga exercises by following the model performance video of the instructor. Then, it evaluates his/her performance by comparing keypoints between the user and the instructor that will be extracted by OpenPose for feedback. However, the current background of EPLAS is not immersive for the user to concentrate in the exercise and keep the motivation. In this paper, we enhance the background of EPLAS by adopting a panoramic photo or a computer graphic (CG) animation with the instrumental music. For evaluations, we prepared five backgrounds using panoramic photos and CG animations, and asked 15 students in Okayama University, Japan, to use it. The questionnaire results confirmed the effectiveness.
DPOSE: Online Keypoint-CAM Guided Inference for Driver Pose Estimation with GMM-based Balanced Sampling

Human pose estimation (HPE) is an essential component of Driving Monitoring Systems (DMS) for real-time recognition of driving behavior. To achieve this, HPE is typically integrated with other tasks such as detection and head pose regression, into a single lightweight model that can be easily deployed on edge-side devices. However, oversimplified designs of lightweight HPE models may cause overfitting on generalized samples, rendering them unable to handle rare samples, particularly in the case of the dataset with the imbalanced distribution. In this paper, we propose an optimization scheme for a proprietary HPE task in DMS scenarios. Our method involves a pose-wise hard mining strategy to balance the pose distribution. Additionally, we introduce an online keypoint independent grad-cam loss, which constrains the gradient-based activation feature map of each keypoint prediction to its corresponding semantic region. We evaluate our approach using a benchmark dataset for DMS tasks and achieve outstanding results. Our code will be publicly available <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Diffusion-Enhanced PatchMatch: A Framework for Arbitrary Style Transfer with Diffusion Models

Diffusion models have gained immense popularity in recent years due to their impressive ability to generate high-quality images. The opportunities that diffusion models provide are numerous, from text-to-image synthesis to image restoration and enhancement, as well as image compression and inpainting. However, expressing image style in words can be a challenging task, making it difficult for diffusion models to generate images with specific style without additional optimization techniques. In this paper, we present a novel method, Diffusion-Enhanced PatchMatch (DEPM), that leverages Stable Diffusion for style transfer without any finetuning or pretraining. DEPM captures high-level style features while preserving the fine-grained texture details of the original image. By enabling the transfer of arbitrary styles during inference, our approach makes the process more flexible and efficient. Moreover, its optimization-free nature makes it accessible to a wide range of users.
Exploring Diversified Adversarial Robustness in Neural Networks via Robust Mode Connectivity

This paper proposes a new method called robust mode connectivity (RMC) to enhance the adversarial robustness of neural networks (NNs) by exploring a wider range of parameter space. While adversarial training methods have shown promising results in enhancing the robustness of NNs against perturbations, they are limited by considering only a single type of perturbation during training and having limited search capability. RMC aims to address this limitation by considering multiple ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p</inf> norm perturbations (p = 1, 2, ∞) and building on the concept of mode connectivity to identify a path of NNs with high robustness against different types of perturbations. The proposed method employs a multi steepest descent (MSD) algorithm to explore the parameter space and achieve diversified adversarial robustness. Experimental results on various datasets and architectures demonstrate the effectiveness of RMC.
Implications of Solution Patterns on Adversarial Robustness

Empirical robustness evaluation (RE) of deep learning models against adversarial perturbations involves solving non-trivial constrained optimization problems. Recent work has shown that these RE problems can be reliably solved by a general-purpose constrained-optimization solver, PyGRANSO with Constraint-Folding (PWCF). In this paper, we take advantage of PWCF and other existing numerical RE algorithms to explore distinct solution patterns in solving RE problems with various combinations of losses, perturbation models, and optimization algorithms. We then provide extensive discussions on the implications of these patterns on current robustness evaluation and adversarial training. A comprehensive version of this work can be found in [19].
Compensation Learning in Semantic Segmentation

Label noise and ambiguities between similar classes are challenging problems in developing new models and annotating new data for semantic segmentation. In this paper, we propose Compensation Learning in Semantic Segmentation, a framework to identify and compensate ambiguities as well as label noise. More specifically, we add a ground truth depending and globally learned bias to the classification logits and introduce a novel uncertainty branch for neural networks to induce the compensation bias only to relevant regions. Our method is employed into state-of-the-art segmentation frameworks and several experiments demonstrate that our proposed compensation learns inter-class relations that allow global identification of challenging ambiguities as well as the exact localization of subsequent label noise. Additionally, it enlarges robustness against label noise during training and allows target-oriented manipulation during inference. We evaluate the proposed method on Cityscapes, KITTI-STEP, ADE20k, and COCO-stuff10k.
In Defense of Structural Symbolic Representation for Video Event-Relation Prediction

Understanding event relationships in videos requires a model to understand the underlying structures of events (i.e. the event type, the associated argument roles, and corresponding entities) and factual knowledge for reasoning. Structural symbolic representation (SSR) based methods directly take event types and associated argument roles/entities as inputs to perform reasoning. However, the state-of-the-art video event-relation prediction system shows the necessity of using continuous feature vectors from input videos; existing methods based solely on SSR inputs fail completely, even when given oracle event types and argument roles. In this paper, we conduct an extensive empirical analysis to answer the following questions: 1) why SSR-based method failed; 2) how to understand the evaluation setting of video event relation prediction properly; 3) how to uncover the potential of SSR-based methods. We first identify suboptimal training settings as causing the failure of previous SSR-based video event prediction models. Then through qualitative and quantitative analysis, we show how evaluation that takes only video as inputs is currently unfeasible, as well as the reliance on oracle event information to obtain an accurate evaluation. Based on these findings, we propose to further contextualize the SSR-based model to an Event-Sequence Model and equip it with more factual knowledge through a simple yet effective way of reformulating external visual commonsense knowledge bases into an event-relation prediction pretraining dataset. The resultant new state-of-the-art model eventually establishes a 25% Macro-accuracy performance boost.
Improving Cross-Domain Detection with Self-Supervised Learning

Cross-Domain Detection (XDD) aims to train a domain-adaptive object detector using unlabeled images from a target domain and labeled images from a source domain. Existing approaches achieve this either by transferring the style of source images to that of target images, or by aligning the features of images from the two domains. In this paper, rather than proposing another method following the existing lines, we introduce a new framework complementary to existing methods. Our framework unifies some popular Self-Supervised Learning (SSL) techniques (e.g., rotation angle prediction, strong/weak data augmentation, mean teacher modeling) and adapts them to the XDD task. Our basic idea is to leverage the unsupervised nature of these SSL techniques and apply them simultaneously across domains (source and target) and models (student and teacher). These SSL techniques can thus serve as shared bridges that facilitate knowledge transfer between domains. More importantly, as these techniques are independently applied in each domain, they are complementary to existing domain alignment techniques that relies on interactions between domains (e.g., adversarial alignment). We perform extensive analyses on these SSL techniques and show that they significantly improve the performance of existing methods. In addition, we reach comparable or even better performance than the state-of-the-art methods when integrating our framework with an old well-established method.
Multi-modal Aerial View Image Challenge: Translation from Synthetic Aperture Radar to Electro-Optical Domain Results - PBVS 2023

This paper unveils the discoveries and outcomes of the inaugural iteration of the Multi-modal Aerial View Image Challenge (MAVIC) aimed at image translation. The primary objective of this competition is to stimulate research efforts towards the development of models capable of translating co-aligned images between multiple modalities. To accomplish the task of image translation, the competition utilizes images obtained from both synthetic aperture radar (SAR) and electro-optical (EO) sources. Specifically, the challenge centers on the translation from the SAR modality to the EO modality, an area of research that has garnered attention. The inaugural challenge demonstrates the feasibility of the task. The dataset utilized in this challenge is derived from the UNIfied COincident Optical and Radar for recognitioN (UNICORN) dataset. We introduce an new version of the UNICORN dataset that is focused on enabling the sensor translation task. Performance evaluation is conducted using a combination of measures to ensure high fidelity and high accuracy translations.
Hierarchical Blockchain-enabled Federated Learning with Reputation Management for Mobile Internet of Vehicles

Federated learning (FL), as a distributed technology, has great application potential in autonomous driving. However, due to the lack of a quality verification mechanism for the client, it faces the problem of malicious users attacking the global model, which reduces the accuracy of FL model. In response to this problem, this paper proposes a hierarchical blockchain-enabled FL with reputation management scheme, which improves the efficiency and accuracy of FL while ensuring privacy and security. In this paper, we first propose a hierarchical blockchain framework and design a blockchain consensus algorithm based on parameter proof of quality (PoQ), which can provide FL workers with a secure and efficient data storage environment in a decentralized manner, while making up for the poor scalability of traditional single blockchains. On this basis, we design a reputation management method based on Bayesian theory and multiple subjective logic models to select high-quality local participating users. In particular, the weights of factors such as interactive activity and interactive position are taken into account to improve the accuracy of reputation calculations. Extensive simulations validate the performance of our scheme in improving FL accuracy and efficiency.
Learning to See in Nighttime Driving Scenes with Inter-frequency Priors

Currently, image-to-image translation methods have achieved significant performance with the help of deep CNNs and GANs, but most existing models are not suitable for autonomous driving scenarios due to their interpretability. In this paper, we develop a high-quality image translator (i.e. night → day), N2D-LPNet, to facilitate nighttime driving scene dperception. Instead of pursuing a complex network structure, we first attempt to explore the inter-frequency relation knowledge to simplify image translation process. Specifically, the lightweight Laplace pyramid is introduced as the backbone architecture to decompose the feature maps of nighttime image into high- and low-frequency components. Considering the similar morphological properties of different frequency components, we design a flexible inter-frequency guiding strategy which utilizes each lower frequency information to refine the higher frequency feature in a progressive manner. Benefiting from the advantage of constraint from inter-frequency priors, our method can process the nighttime image better under the practical driving system while still persevere competitive results. We also discuss the potential value of N2D-LPNet for other high-level vision tasks.
Vision + Language Applications: A Survey

Text-to-image generation has attracted significant interest from researchers and practitioners in recent years due to its widespread and diverse applications across various industries. Despite the progress made in the domain of vision and language research, the existing literature remains relatively limited, particularly with regard to advancements and applications in this field. This paper explores a relevant research track within multimodal applications, including text, vision, audio, and others. In addition to the studies discussed in this paper, we are also committed to continually updating the latest relevant papers, datasets, application projects and corresponding information at https://github.com/Yutong-Zhou-cv/Awesome-Text-to-Image.