Reviewerly/exHarmony huggingface dataset:
Scene Flow Estimation from Point Clouds with Contrastive Loss and Dual Pseudo Labels

Scene flow estimation aims to extract the 3D motion vector between each surface point in two consecutive point clouds. Pseudo-label-based approaches usually exploit point-to-point relations and 3D geometry information to generate the pseudo label for self-supervised learning. However, unreasonable results are still obtained due to the unexploited information of negative samples. Moreover, previous approaches are limited by the fact that pseudo labels are only generated along the forward direction, ignoring the backward direction that has strong spatiotemporal correlations with the forward direction. In this paper, we address these issues in a simple yet effective manner. Specifically, we introduce a contrastive loss to exploit both the information of positive samples and negative samples. Furthermore, we design a dual pseudo labels generation strategy to provide a bidirectional self-supervision for scene flow estimation. Experiments on FlyingThings3D and KITTI datasets show that our method can achieve competitive performance compared to recent self-supervised methods.
Video Question Answering Using Clip-Guided Visual-Text Attention

Cross-modal learning of video and text plays a key role in Video Question Answering (VideoQA). In this paper, we propose a visual-text attention mechanism to utilize the Contrastive Language-Image Pre-training (CLIP) trained on lots of general domain language-image pairs to guide the cross-modal learning for VideoQA. Specifically, we first extract video features using a TimeSformer and text features using a BERT from the target application domain, and utilize CLIP to extract a pair of visual-text features from the general-knowledge domain through the domain-specific learning. We then propose a Cross-domain Learning to extract the attention information between visual and linguistic features across the target domain and general domain. The set of CLIP-guided visual-text features are integrated to predict the answer. The proposed method is evaluated on MSVD-QA and MSRVTTQA datasets and outperforms state-of-the-art methods.
ELEGAN: An Efficient Low Light Enhancement GAN for Unpaired Supervision

Low-light image enhancement using deep learning methods in real-world scenarios is challenging, as capturing an enormous paired training dataset is impractical. Many unsupervised low-light image-enhancement methods have recently been explored, although most of them are computationally intense with a complex network structure. This paper proposes a lightweight attention-guided generative adversarial network called ELEGAN for fast low-light image enhancement in a fully unsupervised fashion. We introduce a self-regularized illumination attention map-guided three-layer U-type generator that offers significant speedup by utilizing parallel processing on the encoder side to reduce the inter-dependency between the layers. Additionally, we propose a modified residual dense block for better restoration of features with lesser parameters. We incorporate a global-local discriminator structure in our model. Both visual and quantitative results demonstrate that ELEGAN outperforms state-of-the-art methods in illumination restoration, noise reduction and structure recovery of low-light images.
Fusing Explicit and Implicit Flow for Optical Flow Estimation

Estimating optical flow for large movement remains a challenging issue due to inconsistency in features between frames. To resolve this challenge, we propose a novel sequence-based deep learning network that jointly trains explicit flow and implicit flow to accurately estimate optical flow. To do this, we implemented three submodules: explicit flow embedder, implicit flow embedder, and flow fusion network. Explicit flow embedder learns the pair-wise correlation between visible pixels based on the spatial attention made per image. Implicit flow embedder learns implicit flow based on the temporal context of motion from all frames in the sequence. To effectively learn the implicit flow, we give a longer sequence of frames as input. Flow fusion network fuses features from explicit and implicit embedder to output the final optical flow. Through extensive experiments, our model demonstrates its robustness against the large motion while providing accurate flow estimation for pixels without pairs in the next frame.
Items Outperform Adjectives in a Computational Model of Binary Semantic Classification

Semantic memory encompasses one's knowledge about the world. Distributional semantic models, which construct vector spaces with embedded words, are a proposed framework for understanding the representational structure of human semantic knowledge. Unlike some classic semantic models, distributional semantic models lack a mechanism for specifying the properties of concepts, which raises questions regarding their utility for a general theory of semantic knowledge. Here, we develop a computational model of a binary semantic classification task, in which participants judged target words for the referent's size or animacy. We created a family of models, evaluating multiple distributional semantic models, and mechanisms for performing the classification. The most successful model constructed two composite representations for each extreme of the decision axis (e.g., one averaging together representations of characteristically big things and another of characteristically small things). Next, the target item was compared to each composite representation, allowing the model to classify more than 1,500 words with human-range performance and to predict response times. We propose that when making a decision on a binary semantic classification task, humans use task prompts to retrieve instances representative of the extremes on that semantic dimension and compare the probe to those instances. This proposal is consistent with the principles of the instance theory of semantic memory.
Controlling Facial Attribute Synthesis by Disentangling Attribute Feature Axes in Latent Space

In this study, we propose a novel approach to synthesize high-resolution and hyper-realistic face images with controlled attributes. Firstly, by training an attribute classifier to assign attribute labels to given synthesized face images, we build the links between latent vectors and face attributes. Secondly, we adapt the regression method to match the distributions of latent vectors with the corresponding face attributes, to control the attribute synthesis in the face images. Finally, we use the Gram-Schmidt orthogonalization algorithm to disentangle the attribute feature axes in latent space, such that a change in one attribute will not cause any changes in other attributes. Extensive experiments demonstrate the effectiveness of the proposed approach for high-quality face image synthesis with controlled attributes.
DFT-CAM: Discrete Fourier Transform Driven Class Activation Map

Class activation map methods (CAMs) serve one of the key roles in explainable artificial intelligence (XAI) and are recently being applied to weakly-supervised object localization, showing great potential in many applications. However, the current CAM methods still have room for improvement regarding the weakly-supervised object localization task. In this paper, we proposed DFT-CAM, a novel CAM method that combines discrete Fourier transform based feature encoding with an orthogonality-based feature selection scheme. DFT-CAM doesn’t require any training, better captures semantic information, aggregates only the most representative convolutional features, and improves the downstream tasks such as weakly-supervised object localization. The experimental results show promising performance compared to other CAM methods across different deep learning classification networks.
Deformation Robust Text Spotting with Geometric Prior

The goal of text spotting is to perform text detection and recognition simultaneously. Although the diversity of luminosity and orientation in scene texts has been widely studied, the font diversity and shape variance of the same character are ignored in recent works, since most characters in natural images are rendered in standard fonts. To solve this problem, we present a Chinese Artistic Dataset, termed as ARText, which contains 33, 000 artistic images with rich shape deformation and font diversity. Based on this database, we develop a deformation robust text spotting method (DR TextSpotter) to solve the recognition problem of complex deformation of characters in different fonts. Specifically, we propose a geometric prior module to highlight the important features based on the unsupervised landmark detection sub-network. A graph convolution network is further constructed to fuse the character features and landmark features, and then performs semantic reasoning to enhance the discrimination for different characters. The experiments are conducted on ARText and IC19-ReCTS datasets. Our results demonstrate the effectiveness of our proposed method. The datasets and models will become publicly available after publication.
Artifact: A Large-Scale Dataset With Artificial And Factual Images For Generalizable And Robust Synthetic Image Detection

Synthetic image generation has opened up new opportunities but has also created threats in regard to privacy, authenticity, and security. Detecting fake images is of paramount importance to prevent illegal activities, and previous research has shown that generative models leave unique patterns in their synthetic images that can be exploited to detect them. However, the fundamental problem of generalization remains, as even state-of-the-art detectors encounter difficulty when facing generators never seen during training. To assess the generalizability and robustness of synthetic image detectors in the face of real-world impairments, this paper presents a large-scale dataset <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> named ArtiFact, comprising diverse generators, object categories, and real-world challenges. Moreover, the proposed multi-class classification scheme, combined with a filter stride reduction strategy addresses social platform impairments and effectively detects synthetic images from both seen and unseen generators. The proposed solution significantly outperforms other top teams by 8.34% on Test 1, 1.26% on Test 2, and 15.08% on Test 3 in the IEEE VIP Cup challenge at ICIP 2022, as measured by the accuracy metric.
Hierarchical Terrain Attention and Multi-Scale Rainfall Guidance for Flood Image Prediction

With the deterioration of climate, the phenomenon of rain-induced flooding has become frequent. To mitigate its impact, recent works adopt convolutional neural network or its variants to predict the floods. However, these methods directly force the model to reconstruct the raw pixels of flood images through a global constraint, overlooking the underlying information contained in terrain features and rainfall patterns. To address this, we present a novel framework for precise flood map prediction, which incorporates hierarchical terrain spatial attention to help the model focus on spatially-salient areas of terrain features and constructs multi-scale rainfall embedding to extensively integrate rainfall pattern information into generation. To better adapt the model in various rainfall conditions, we leverage a rainfall regression loss for both the generator and the discriminator as additional supervision. Extensive evaluations on real catchment datasets demonstrate the superior performance of our method, which greatly surpasses the previous arts under different rainfall conditions.
Graphrpe: Relative Position Encoding Graph Transformer for 3d Human Pose Estimation

The graph neural network has been playing increasingly important roles in 2D-3D lifting based single-frame human pose estimation. However, it still suffers from inferior modeling of local and global associations between 2D nodes. To this end, we introduce two relative position encoding approaches and propose a novel graph-transformer structure "GraphRPE." The model consists of two components: graph relative position encoding (GRPE) and universal relative position encoding (URPE). GRPE embeds graph structure prior information into the attention map to correct the self-attention weights and prompt appropriate interactions between 2D nodes, both locally and globally. On the other hand, URPE introduces the Toeplitz matrix to address the limited representation capability of the transformer. In addition, we investigate several graph edge types and their impacts on the results. Extensive experimental results demonstrate that our proposed method achieves SOTA performance on the Human3.6M dataset.
A Multichannel Localization Method for Camouflaged Object Detection

This paper proposes a multichannel method for discriminative region localization in Camouflaged Object Detection (COD) tasks. In one channel, processing the phase and amplitude of 2-D Fourier spectra generate a modified form of the original image, used later for a pixel-wise optimal local entropy analysis. The other channel implements a class activation map (CAM) and Global Average Pooling (GAP) for object localization. We combine the channels linearly to form the final localized version of the COD images. Experimentation in multiple COD datasets demonstrates that the proposed method successfully localizes regions containing more than 80% of the camouflaged objects. Our proposed method does not require memory-intensive devices or prior training on particular image features, making it easily integrable into a resource-constrained environment. The proposed approach is also applicable to non-COD images.
TRICKVOS: A Bag of Tricks for Video Object Segmentation

Space-time memory (STM) network methods have been dominant in semi-supervised video object segmentation (SVOS) due to their remarkable performance. In this work, we identify three key aspects where we can improve such methods; i) supervisory signal, ii) pretraining and iii) spatial awareness. We then propose TrickVOS; a generic, method-agnostic bag of tricks addressing each aspect with i) a structure-aware hybrid loss, ii) a simple decoder pretraining regime and iii) a cheap tracker that imposes spatial constraints in model predictions. Finally, we propose a lightweight network and show that when trained with TrickVOS, it achieves competitive results to state-of-the-art methods on DAVIS and YouTube benchmarks, while being one of the first STM-based SVOS methods that can run in real-time on a mobile device.
Flash Compensated Low-Light Enhancement Via Hierarchical Network Prediction

Photography in low-light conditions suffers from dense noise and insufficient light. Flash photography, introducing extra light sources, performs better at suppressing noise and revealing details, while being interrupted by unnatural ambient illumination. This paper offers an analysis of the pros and cons to utilize low-light and flash images for enhancement, which inspires us to design a unified sample-adaptive CNN to capture diverse focuses from different inputs in a complementary way. Specifically, a Flash Compensated Dynamic Filtering Network is proposed to utilize the revealed details of flash images to compensate for fine structure reconstruction in low-light enhancement. To adaptively fuse information from misaligned low-light and flash image pairs, our network is designed with three distinctive features. Firstly, we adopt a layer-wise regression strategy, where results are predicted from the single input first and then fused to sufficiently leverage complementary information. Secondly, we employ a sample-adaptive mechanism, where each pixel is estimated with its distinctive parameters augmented by weighted residual connections. Finally, we utilize a coarse-to-fine architecture, where features are extracted by diversified receptive fields to utilize hierarchical contextual information. Experimental results demonstrate that the three design principles lead to the significant superiority of the proposed method over state-of-the-art methods.
Towards Robustness: Enhancing Deep Learning Models Through Meta-Learning and Bilevel Optimization for Accurate Car Damage Classification

The article focuses on the difficulties of training deep learning models for real-world scenarios, where the data is often noisy and improperly labeled. Despite the progress made in developing sophisticated models, they require extensive, well-labeled data sets to perform optimally. To tackle this challenge, various approaches have been proposed for training with noisy data, including our previous work, Co-Meta, which proposed a meta-learning approach. In this study, we employed Co-Meta to train a damage classification model with a noisy dataset. We demonstrate how the selection of meta data sets can significantly influence the results and present an efficient method for addressing this issue. Furthermore, we tackle the problem of high memory usage associated with Co-Meta by utilizing alternative bi-level optimization algorithms that decrease the computational resources needed. Our experiments demonstrate that the choice of optimization algorithm does not have a significant impact on the final accuracy, yet it has a considerable effect on the memory requirements. These modifications enhance Co-Meta performances significantly. Our selection strategy resulted in an accuracy improvement of over 1%, while the new optimization methods reduced the memory requirements by half.
YOLO-MAXVOD for Real-Time Video Object Detection

Video Object Detection (VOD) is one of the fundamental problems in video understanding with applications ranging from surveillance to autonomous driving. But many such real-world applications are unable to leverage the existing VOD models owing to their higher computational complexity which reduces inference speed. Single-stage still-image object detection models are naively used without any use of video information. In this paper, we present YOLOX based VOD model, YOLO-MaxVOD, which provides a better trade-off between accuracy and inference time than the current real-time VOD solutions. Specifically, we propose a temporal fusion module that integrates within the YOLOX architecture to take advantage of the high speed that the YOLOX model offers. In our experimentation on the Imagenet-VID dataset, we show that YOLO-MaxVOD shows 4.4-5.6% AP50 improvement over the baseline YOLOX, across different versions, with just a 1-2 ms increase in latency on NVIDIA 1080Ti GPU.
Degradation Conditioned GAN for Degradation Generalization of Face Restoration Models

Face restoration models are usually trained on synthetic degraded data to output an image that matches the clean version of itself. Most previous methods use a single model to deal with all the degradation levels, resulting in a domain generalization problem. We explore the value of degradation information and propose a Degradation Conditioned GAN (DeCGAN). The architecture consists of modulated convolution, bias, and fusion modules, inspired by deblurring, denoising and super-resolution. The whole network can be modulated by the degradation levels to achieve delicate and precise restoration effects. Experiments are conducted on conventional and modulated face restoration tasks. DeCGAN can achieve more faithful restoration and better metrics (FID, LPIPS, etc.) than previous methods do. Moreover, our model performs well on real-world low-quality face images.
Audio-Visual Quality Assessment for User Generated Content: Database and Method

With the explosive increase of User Generated Content (UGC), UGC video quality assessment (VQA) becomes more and more important for improving users’ Quality of Experience (QoE). However, most existing UGC VQA studies only focus on the visual distortions of videos, ignoring that the user’s QoE also depends on the accompanying audio signals. In this paper, we conduct the first study to address the problem of UGC audio and video quality assessment (AVQA). Specifically, we construct the first UGC AVQA database named the SJTU-UAV database, which includes 520 in-the-wild UGC audio and video (A/V) sequences, and conduct a user study to obtain the mean opinion scores of the A/V sequences. The content of the SJTU-UAV database is then analyzed from both the audio and video aspects to show the database characteristics. We also design a family of AVQA models, which fuse the popular VQA methods and audio features via support vector regressor (SVR). We validate the effectiveness of the proposed models on the three databases. The experimental results show that with the help of audio signals, the VQA models can evaluate the perceptual quality more accurately. The database will be released to facilitate further research.
Attention-Guided Contrastive Masked Image Modeling for Transformer-Based Self-Supervised Learning

Self-supervised learning with vision transformer (ViT) has gained much attention recently. Most existing methods rely on either contrastive learning or masked image modeling. The former is suitable for global feature extraction but underperforms in fine-grained tasks. The later explores the internal structure of images but ignores the high information sparsity and unbalanced information distribution. In this paper, we propose a new approach called Attention-guided Contrastive Masked Image Modeling (ACoMIM), which integrates the merits of both paradigms and leverages the attention mechanism of ViT for effective representation. Specifically, it has two pretext tasks, predicting the features of masked regions guided by attention and comparing the global features of masked and unmasked images. We show that these two pretext tasks complement each other and improve our method’s performance. The experiments demonstrate that our model transfers well to various downstream tasks such as classification and object detection. Code is available at https://github.com/yczhan/ACoMIM.
Training Cartoonization Network without Cartoon

Photo cartoonization aims to translate a real-world photo into a cartoon image. Previous learning-based cartoonization studies have shown promising results, however, an alternative approach is still necessary because of the following limitations. First, the deficiency of training datasets necessitates the additional dataset acquisition process, which yields uneven network training results. Second, we observe that the created images using existing works have color transition problems. In this paper, we propose a new approach on photo cartoonization which does not use cartoon datasets and does not suffer these limitations. By focusing on the two important aspects of cartoon style, regions and edges, our work enables us to produce cartoonized images without using a manually collected dataset. We show that our framework can generate high-quality cartoonized images and achieve competitive performance with comparable cartoonization networks even without training cartoon dataset.
CTI-Unet: Hybrid Local Features and Global Representations Efficiently

Recent advancements in medical image segmentation have demonstrated superior performance by combining Transformer and U-Net due to the Transformer’s exceptional ability to capture long-range semantic dependencies. However, existing approaches mostly replace or concatenate the Convolutional Neural Networks (CNNs) and Transformers in series, which limits the potential of their combination. In this paper, we introduce a dual-branch feature encoder, CTI-UNet, that effectively fuses the global representations and local features of the CNN and Transformer branches at different scales through bidirectional feature interaction. Our proposed method outperforms existing approaches on multiple medical datasets, demonstrating state-of-the-art performance. The code for CTI-UNet is publicly available at https://github.com/huhaigen/CTI-UNet.