Reviewerly/exHarmony huggingface dataset:
Local Color Distributions Prior for Image Enhancement

Existing image enhancement methods are typically designed to address either the over- or under-exposure problem in the input image. When the illumination of the input image contains both over- and under-exposure problems, these existing methods may not work well. We observe from the image statistics that the local color distributions (LCDs) of an image suffering from both problems tend to vary across different regions of the image, depending on the local illuminations. Based on this observation, we propose in this paper to exploit these LCDs as a prior for locating and enhancing the two types of regions (i.e., over-/under-exposed regions). First, we leverage the LCDs to represent these regions, and propose a novel local color distribution embedded (LCDE) module to formulate LCDs in multi-scales to model the correlations across different regions. Second, we propose a dual-illumination learning mechanism to enhance the two types of regions. Third, we construct a new dataset to facilitate the learning process, by following the camera image signal processing (ISP) pipeline to render standard RGB images with both under-/over-exposures from raw data. Extensive experiments demonstrate that the proposed method outperforms existing state-of-the-art methods quantitatively and qualitatively. Codes and dataset are in https://hywang99.github.io/lcdpnet/ .
UIA-ViT: Unsupervised Inconsistency-Aware Method Based on Vision Transformer for Face Forgery Detection

Intra-frame inconsistency has been proved to be effective for the generalization of face forgery detection. However, learning to focus on these inconsistency requires extra pixel-level forged location annotations. Acquiring such annotations is non-trivial. Some existing methods generate large-scale synthesized data with location annotations, which is only composed of real images and cannot capture the properties of forgery regions. Others generate forgery location labels by subtracting paired real and fake images, yet such paired data is difficult to collected and the generated label is usually discontinuous. To overcome these limitations, we propose a novel Unsupervised Inconsistency-Aware method based on Vision Transformer, called UIA-ViT, which only makes use of video-level labels and can learn inconsistency-aware feature without pixel-level annotations. Due to the self-attention mechanism, the attention map among patch embeddings naturally represents the consistency relation, making the vision Transformer suitable for the consistency representation learning. Based on vision Transformer, we propose two key components: Unsupervised Patch Consistency Learning (UPCL) and Progressive Consistency Weighted Assemble (PCWA). UPCL is designed for learning the consistency-related representation with progressive optimized pseudo annotations. PCWA enhances the final classification embedding with previous patch embeddings optimized by UPCL to further improve the detection performance. Extensive experiments demonstrate the effectiveness of the proposed method.
Amending Facial Expression Representation via De-albino

This paper is the first to investigate the flawed nature of padding in the convolutional layer. We find that the accumulation of padding in convolutional networks has an erosive effect on features, resulting in a partial loss of recognition performance. We block the detrimental impacts of padding to some extent at the extraction stage of the representation by secondary processing of the high-dimensional features at the network's back end, despite the fact that no solution for total padding replacement has been established. Our proposed module named Amend Representation Module (ARM) weakens the weights of degraded feature pixels and obtains more discriminative representations from eroded features. Experiments on public benchmarks prove that our ARM based on ResNet-18 boosts the performance of FER remarkably.
Decentral and Incentivized Federated Learning Frameworks: A Systematic Literature Review

The advent of federated learning (FL) has sparked a new paradigm of parallel and confidential decentralized machine learning (ML) with the potential of utilizing the computational power of a vast number of Internet of Things (IoT), mobile, and edge devices without data leaving the respective device, thus ensuring privacy by design. Yet, simple FL frameworks (FLFs) naively assume an honest central server and altruistic client participation. In order to scale this new paradigm beyond small groups of already entrusted entities toward mass adoption, FLFs must be: 1) truly decentralized and 2) incentivized to participants. This systematic literature review is the first to analyze FLFs that holistically apply both, the blockchain technology to decentralize the process and reward mechanisms to incentivize participation. 422 publications were retrieved by querying 12 major scientific databases. After a systematic filtering process, 40 articles remained for an in-depth examination following our five research questions. To ensure the correctness of our findings, we verified the examination results with the respective authors. Although having the potential to direct the future of distributed and secure artificial intelligence, none of the analyzed FLFs is production ready. The approaches vary heavily in terms of use cases, system design, solved issues, and thoroughness. We provide a systematic approach to classify and quantify differences between FLFs, expose limitations of current works and derive future directions for research in this novel domain.
Fine-Grained Scene Graph Generation with Data Transfer

Scene graph generation (SGG) is designed to extract (subject, predicate, object) triplets in images. Recent works have made a steady progress on SGG, and provide useful tools for high-level vision and language understanding. However, due to the data distribution problems including long-tail distribution and semantic ambiguity, the predictions of current SGG models tend to collapse to several frequent but uninformative predicates (e.g., on, at), which limits practical application of these models in downstream tasks. To deal with the problems above, we propose a novel Internal and External Data Transfer (IETrans) method, which can be applied in a plug-and-play fashion and expanded to large SGG with 1,807 predicate classes. Our IETrans tries to relieve the data distribution problem by automatically creating an enhanced dataset that provides more sufficient and coherent annotations for all predicates. By applying our proposed method, a Neural Motif model doubles the macro performance for informative SGG. The code and data are publicly available at https://github.com/waxnkw/IETrans-SGG.pytorch .
NLX-GPT: A Model for Natural Language Explanations in Vision and Vision-Language Tasks

Natural language explanation (NLE) models aim at explaining the decision-making process of a black box system via generating natural language sentences which are human-friendly, high-level and fine-grained. Current NLE models <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Throughout this paper, we refer to NLE models as Natural Language Explanation models aimed for vision and vision-language tasks. explain the decision-making process of a vision or vision-language model (a.k.a., task model), e.g., a VQA model, via a language model (a.k.a., explanation model), e.g., GPT. Other than the additional memory resources and inference time required by the task model, the task and explanation models are completely independent, which disassociates the explanation from the reasoning process made to predict the answer. We introduce NLX-GPT, a general, compact and faithful language model that can simultaneously predict an answer and explain it. We first conduct pre-training on large scale data of image-caption pairs for general understanding of images, and then formulate the answer as a text prediction task along with the explanation. Without region proposals nor a task model, our resulting overall framework attains better evaluation scores, contains much less parameters and is 15× faster than the current SoA model. We then address the problem of evaluating the explanations which can be in many times generic, data-biased and can come in several forms. We therefore design 2 new evaluation measures: (1) explain-predict and (2) retrieval-based attack, a selfevaluation framework that requires no labels. Code is at: https://github.com/fawazsammani/nlxgpt.
Diversity Matters: Fully Exploiting Depth Clues for Reliable Monocular 3D Object Detection

As an inherently ill-posed problem, depth estimation from single images is the most challenging part of monocular 3D object detection (M3OD). Many existing methods rely on preconceived assumptions to bridge the missing spatial information in monocular images, and predict a sole depth value for every object of interest. However, these assumptions do not always hold in practical applications. To tackle this problem, we propose a depth solving system that fully explores the visual clues from the subtasks in M3OD and generates multiple estimations for the depth of each target. Since the depth estimations rely on different assumptions in essence, they present diverse distributions. Even if some assumptions collapse, the estimations established on the remaining assumptions are still reliable. In addition, we develop a depth selection and combination strategy. This strategy is able to remove abnormal estimations caused by collapsed assumptions, and adaptively combine the remaining estimations into a single one. In this way, our depth solving system becomes more precise and robust. Exploiting the clues from multiple subtasks of M3OD and without introducing any extra information, our method surpasses the current best method by more than 20% relatively on the Moderate level of test split in the KITTI 3D object detection benchmark, while still maintaining real-time efficiency.
GaTector: A Unified Framework for Gaze Object Prediction

Gaze object prediction is a newly proposed task that aims to discover the objects being stared at by humans. It is of great application significance but still lacks a unified solution framework. An intuitive solution is to incorporate an object detection branch into an existing gaze prediction method. However, previous gaze prediction methods usually use two different networks to extract features from scene image and head image, which would lead to heavy network architecture and prevent each branch from joint optimization. In this paper, we build a novel framework named GaTector to tackle the gaze object prediction problem in a unified way. Particularly, a specific-general-specific (SGS) feature extractor is firstly proposed to utilize a shared backbone to extract general features for both scene and head images. To better consider the specificity of inputs and tasks, SGS introduces two input-specific blocks before the shared backbone and three task-specific blocks after the shared backbone. Specifically, a novel Defocus layer is designed to generate object-specific features for the object detection task without losing information or requiring extra computations. Moreover, the energy aggregation loss is introduced to guide the gaze heatmap to concentrate on the stared box. In the end, we propose a novel wUoC metric that can reveal the difference between boxes even when they share no overlapping area. Extensive experiments on the GOO dataset verify the superiority of our method in all three tracks, i.e. object detection, gaze estimation, and gaze object prediction.
Tree Energy Loss: Towards Sparsely Annotated Semantic Segmentation

Sparsely annotated semantic segmentation (SASS) aims to train a segmentation network with coarse-grained (i.e., point-, scribble-, and block-wise) supervisions, where only a small proportion of pixels are labeled in each image. In this paper, we propose a novel tree energy loss for SASS by providing semantic guidance for unlabeled pixels. The tree energy loss represents images as minimum spanning trees to model both low-level and high-level pair-wise affini-ties. By sequentially applying these affinities to the net-work prediction, soft pseudo labels for unlabeled pixels are generated in a coarse-to-fine manner, achieving dynamic online self-training. The tree energy loss is effective and easy to be incorporated into existing frameworks by com-bining it with a traditional segmentation loss. Compared with previous SASS methods, our method requires no multi-stage training strategies, alternating optimization proce-dures, additional supervised data, or time-consuming post-processing while outperforming them in all SASS settings. Code is available at https://github.com/megvii-research/TreeEnergyLoss.
PPDL: Predicate Probability Distribution based Loss for Unbiased Scene Graph Generation

Scene Graph Generation (SGG) has attracted more and more attention from visual researchers in recent years, since Scene Graph (SG) is valuable in many downstream tasks due to its rich structural-semantic details. However, the ap-plication value of SG on downstream tasks is severely lim-ited by the predicate classification bias, which is caused by long-tailed data and presented as semantic bias of predicted relation predicates. Existing methods mainly reduce the prediction bias by better aggregating contexts and integrating external priori knowledge, but rarely take the semantic similarities between predicates into account. In this paper, we propose a Predicate Probability Distribution based Loss (PPDL) to train the biased SGG models and obtain unbi-ased Scene Graphs ultimately. Firstly, we propose a predi-cate probability distribution as the semantic representation of a particular predicate class. Afterwards, we re-balance the biased training loss according to the similarity between the predicted probability distribution and the estimated one, and eventually eliminate the long-tailed bias on predicate classification. Notably, the PPDL training method is model- agnostic, and extensive experiments and qualitative anal-yses on the Visual Genome dataset reveal significant per-formance improvements of our method on tail classes compared to the state-of-the-art methods.
Metaverse and Posthuman Animated Avatars for Teaching-Learning Process: Interperception in Virtual Universe for Educational Transformation

It has been quite some time that the attention of teachers has been focused on the incorporation of Metaverse in education. Since Facebook's announcement that it will be rebranding and promoting itself as Meta, there has been much interest in this subject. To the best of our knowledge, no research as of date has systematically synthesised the results associated with the implementation of Metaverse in education, even though several studies have conducted literature reviews on Metaverse in general. This research does a detailed literature assessment on how the Metaverse might be used in the classroom in order to fill this knowledge gap. Research trends, focal points, and limitations of this 'study subject' are revealed using both bibliometric analysis and text analysis. It is evident from the collected data that lifelogging for educational metaverse merits additional research. The results indicate that research into mobile-, hybrid-, and micro-learning environments are still in its infancy. There is no study on the effectiveness of utilising metaverse to educate children with special needs. In order to widen the attraction of metaverse to educators throughout the globe and to improve their capacity to assist successful learning and teaching in the virtual environment, more in-depth research shall be carried out along the proposed roadmap as suggested by the results of the current investigation.
Interpretable Image Classification with Differentiable Prototypes Assignment

Existing prototypical-based models address the black-box nature of deep learning. However, they are sub-optimal as they often assume separate prototypes for each class, require multi-step optimization, make decisions based on prototype absence (so-called negative reasoning process), and derive vague prototypes. To address those shortcomings, we introduce ProtoPool, an interpretable prototype-based model with positive reasoning and three main novelties. Firstly, we reuse prototypes in classes, which significantly decreases their number. Secondly, we allow automatic, fully differentiable assignment of prototypes to classes, which substantially simplifies the training process. Finally, we propose a new focal similarity function that contrasts the prototype from the background and consequently concentrates on more salient visual features. We show that ProtoPool obtains state-of-the-art accuracy on the CUB-200-2011 and the Stanford Cars datasets, substantially reducing the number of prototypes. We provide a theoretical analysis of the method and a user study to show that our prototypes capture more salient features than those obtained with competitive methods. We made the code available at https://github.com/gmum/ProtoPool .
Super-resolution generative adversarial networks of randomly-seeded fields

Reconstruction of field quantities from sparse measurements is a problem arising in a broad spectrum of applications. This task is particularly challenging when the mapping between sparse measurements and field quantities is performed in an unsupervised manner. Further complexity is added for moving sensors and/or random on–off status. Under such conditions, the most straightforward solution is to interpolate the scattered data onto a regular grid. However, the spatial resolution achieved with this approach is ultimately limited by the mean spacing between the sparse measurements. In this work, we propose a super-resolution generative adversarial network framework to estimate field quantities from random sparse sensors. The algorithm exploits random sampling to provide incomplete views of the high-resolution underlying distributions. It is hereby referred to as the randomly seeded super-resolution generative adversarial network (RaSeedGAN). The proposed technique is tested on synthetic databases of fluid flow simulations, ocean surface temperature distribution measurements and particle-image velocimetry data of a zero-pressure-gradient turbulent boundary layer. The results show excellent performance even in cases with high sparsity or noise level. This generative adversarial network algorithm provides full-field high-resolution estimation from randomly seeded fields with no need of full-field high-resolution representations for training. The problem of reconstructing full-field quantities from incomplete observations arises in various real-world applications. Güemes and colleagues propose a super-resolution algorithm based on a generative adversarial network that can achieve reconstruction of the underlying field from random sparse measurements without requiring full-field high-resolution training data.
Does the selection of virtual reality video matter? A laboratory experimental study of the influences of arousal

Despite the increasing number of studies on virtual tourism, there is currently little knowledge regarding the significance of the selection of appropriate virtual reality video (e.g., content stimulus) to marketing. This study explored how virtual reality video stimulation (arousal) elicited user flow experience and subsequent satisfaction using the stimulus-organism-response (SOR) paradigm. Analysis of the data revealed that self-reported arousal had a statistically significant impact on enjoyment, time distortion, and focused attention. Similar effects on enjoyment and time distortion were found by respiration rate. The results also demonstrated that user satisfaction was influenced by enjoyment and control. The PLS-SEM analysis further demonstrated that enjoyment partially mediated the effects of self-reported arousal and respiration rate on satisfaction. According to multigroup analysis, self-reported arousal (heart rate) exhibited a greater influence on enjoyment (time distortion) across two groups (weak versus strong stimulation). Theoretical and practical implications of these findings were discussed.
Benchmarking the performance of portfolio optimization with QAOA

Abstract We present a detailed study of portfolio optimization using different versions of the quantum approximate optimization algorithm (QAOA). For a given list of assets, the portfolio optimization problem is formulated as quadratic binary optimization constrained on the number of assets contained in the portfolio. QAOA has been suggested as a possible candidate for solving this problem (and similar combinatorial optimization problems) more efficiently than classical computers in the case of a sufficiently large number of assets. However, the practical implementation of this algorithm requires a careful consideration of several technical issues, not all of which are discussed in the present literature. The present article intends to fill this gap and thereby provides the reader with a useful guide for applying QAOA to the portfolio optimization problem (and similar problems). In particular, we will discuss several possible choices of the variational form and of different classical algorithms for finding the corresponding optimized parameters. Viewing at the application of QAOA on error-prone NISQ hardware, we also analyse the influence of statistical sampling errors (due to a finite number of shots) and gate and readout errors (due to imperfect quantum hardware). Finally, we define a criterion for distinguishing between ‘easy’ and ‘hard’ instances of the portfolio optimization problem.
Tri-modality consistency optimization with heterogeneous augmented images for visible-infrared person re-identification

Visible-infrared person re-identification (VI-ReID) is a challenging technology due to the large gap between daytime visible modality and night-time infrared modality. Previous studies mainly investigate the invariant modality-shared information by the feature-level constraint. They hardly eliminate the large discrepancy between both the inter- and intra- modality, obtaining suboptimal results. In this paper, we propose a novel Tri-modality Consistency Optimization Model (TCOM) to adequately decrease inter- and intra- modality discrepancies for VI-ReID. To this end, a pivotal heterogeneous augmented modality is generated by fusing visible images and infrared images. For tackling the distribution discrepancy, we design a Triplet Center Loss (TCL) to maintain the feature consistency by mitigating the relative distance among different modalities. Furthermore, we define a regularization term named Compact Intra-modality Constraint (CIC) that forces the same pedestrian within each modality to possess the compact feature distribution. With the two invariant constraints, TCOM explores the inter- and intra- modality space-invariant representation and compels the feature distribution from different modalities to be close to each other. Extensive experiments on mainstream databases demonstrate that TCOM achieves superior performance.