Reviewerly/exHarmony huggingface dataset:
Pose Calibrated Feature Aggregation for Face Set Recognition

This paper presents Pose Calibrated Feature Aggregation Network (PCFAN), an architecture for set/video face recognition. Using stacked attention blocks and a multi-stream architecture, it automatically assigns adaptive weights to every instance in the set, based on both the recognition embeddings and the associated face metadata. It uses these weights to produce a single, compact feature vector for the set. The model automatically learns to advocate for features from images with more favorable qualities and poses, which inherently hold more information. Our block can be inserted on top of any standard recognition model for set prediction and improved performance, particularly in unconstrained scenarios where subject pose and image quality vary considerably between frames. We test our approach on two challenging video face-recognition datasets, IJB-A and IJB-B to report state-of-the-art results. Moreover, a comparison with top aggregation methods as our baselines demonstrates that PCFAN is the superior approach.
Query Learning of Both Thing and Stuff for Panoptic Segmentation

Starting from DETR, query based detection and segmentation methods achieve comparable results as previous works with a simplified and elegant pipeline. In this work, a novel, simple and unified baseline, named QueryPanSeg, is proposed for panoptic segmentation. QueryPanSeg represents both things and stuff as learnable queries separately. For thing query, we propose to encode each instance mask into compact mask vectors and perform classification, box regression and mask encoding regression simultaneously. For stuff query, we propose a residual interactive learning where each stuff query is responsible for one semantic category and performs pixel interaction via one multi-head attention layer. With this approach, instance-wise and semantically consistent properties for things and stuff can be unified in one framework. Compared with the original DETR, our approach results in a nearly 10 times shorter training schedule to converge. Compared with previous box-based and box-free methods, our proposed approach outperforms many state-of-the-art results with much simpler pipeline without handcrafted components.
Automatic Dataset Generation for Specific Object Detection

In the past decade, object detection tasks are defined mostly by large public datasets. However, building object detection datasets is not scalable due to inefficient image collecting and labeling. Furthermore, most labels are still in the form of bounding boxes, which provide much less information than the real human visual system. In this paper, we present a method to synthesize object-in-scene images, which can preserve the objects' detailed features without bringing irrelevant information. In brief, given a set of images containing a target object, our algorithm first trains a model to find an approximate center of the object as an anchor, then makes an outline regression to estimate its boundary, and finally blends the object into a new scene. Our result shows that in the synthesized image, the boundaries of objects blend very well with the background. Experiments also show that SOTA segmentation models work well with our synthesized data.
Sequential Cross Attention Based Multi-Task Learning

In multi-task learning (MTL) for visual scene understanding, it is crucial to transfer useful information between multiple tasks with minimal interferences. In this paper, we propose a novel architecture that effectively transfers informative features by applying the attention mechanism to the multi-scale features of the tasks. Since applying the attention module directly to all possible features in terms of scale and task requires a high complexity, we propose to apply the attention module sequentially for the task and scale. The cross-task attention module (CTAM) is first applied to facilitate the exchange of relevant information between the multiple task features of the same scale. The cross-scale attention module (CSAM) then aggregates useful information from feature maps at different resolutions in the same task. Also, we attempt to capture long range dependencies through the self-attention module in the feature extraction network. Extensive experiments demonstrate that our method achieves state-of-the-art performance on the NYUD-v2 and PASCAL-Context dataset. Our code is available at https://github.com/kimsunkyung/SCA-MTL
Robust Temporally-Coherent Strategy for Few-shot Video Instance Segmentation

Traditional video instance segmentation (VIS) aims to detect, segment, and track object instances from a known class set in videos. In real-world applications, however, video instance segmentation typically need to cope with novel-class instances and to fast adapt with a few labeled videos. In this work, we aim to tackle the task of few-shot video instance segmentation (FVIS), which is challenging due to large variations in object appearance and motion. We propose a robust temporally coherent strategy, termed as VTFA, based on a two-stage fine-tuning approach. VTFA enforces the instance segmentation of novel classes to be temporally smooth and reduces the classification bias between novel and base classes. The proposed Memory-aware Temporal Context Encoding Module (MTCE) in VTFA encodes the temporal context information, which contributes to the consistency in the final predictions. We also propose a loss named Instance-level Pair-wise Contrastive (IPC) Loss on both the novel and base classes to enhance the robustness of instance classification. To validate our method, we develop a YouTube-VIS-FS benchmark to compare our method with several baselines. The experimental evaluation shows that our strategy is superior or competitive to those strong baselines.
Accurate and Robust Image Correspondence for Structure-From-Motion and its Application to Multi-View Stereo

In this paper, we propose a robust and accurate image correspondence method by combining SuperPoint + SuperGlue (SP+SG) and Local feature matching with TRansformers (LoFTR). The proposed method finds corresponding points on regions with rich texture by SP+SG and those with poor texture by LoFTR since SP+SG exhibits high localization accuracy of image correspondence and LoFTR exhibits high robustness against poor texture regions. The proposed method can be used for image correspondence in SfM to not only improve the estimation accuracy of camera parameters in SfM, but also to improve the reconstruction accuracy and expand the reconstruction area in MVS. Through experiments on the ETH3D dataset, we demonstrate that the proposed method achieves more accurate 3D reconstruction than conventional methods, and also show the impact of image correspondence accuracy in SfM on multi-view 3D reconstruction.
Taxonomy Driven Learning Of Semantic Hierarchy Of Classes

Standard pre-trained convolutional neural networks are deployed on different task-specific limited class applications. These applications require classifying images of a much smaller subset of classes than that of the original large domain dataset on which the network is pre-trained. Therefore, a computationally inefficient and over-represented network is obtained. Hierarchically Self Decomposing CNN (HSD-CNN) addresses this issue by dissecting the network into sub-networks in an automated hierarchical fashion such that each sub-network is useful for classifying images of closely related classes. However, visual similarities are not always well-aligned with the semantic understanding of humans. In this paper, we propose a method that aids the pre-trained network to learn the hierarchy of classes derived from standard taxonomy, WordNet and, produce sub-networks corresponding to semantically meaningful classes upon decomposition. Experimental results show that the cluster of classes obtained for each sub-network is semantically closer according to WordNet hierarchy without degradation in overall accuracy.
Adaptive Warping Network for Transferable Adversarial Attacks

Deep Neural Networks (DNNs) are extremely susceptible to adversarial examples, which are crafted by intentionally adding imperceptible perturbations to clean images. Due to potential threats of adversarial attacks in practice, black-box transfer-based attacks are carefully studied to identify the vulnerability of DNNs. Unfortunately, transfer-based attacks often fail to achieve high transferability because the adversarial examples tend to overfit the source model. Applying input transformation is one of the most effective methods to avoid such overfitting. However, most previous input transformation methods obtain limited transferability because these methods utilize fixed transformations for all images. To solve the problem, we propose an Adaptive Warping Network (AWN), which searches for appropriate warping to the individual data. Specifically, AWN optimizes the warping, which mitigates the effect of adversarial perturbations in each iteration. The adversarial examples are generated to become robust against such strong transformations. Extensive experimental results on the ImageNet dataset demonstrate that AWN outperforms the existing input transformation methods in terms of transferability.
Multi-Field De-Interlacing Using Deformable Convolution Residual Blocks and Self-Attention

Although deep learning has made significant impact on image/video restoration and super-resolution, learned deinterlacing has so far received less attention in academia or industry. This is despite deinterlacing is well-suited for supervised learning from synthetic data since the degradation model is known and fixed. In this paper, we propose a novel multi-field full frame-rate deinterlacing network, which adapts the state-of-the-art superresolution approaches to the deinterlacing task. Our model aligns features from adjacent fields to a reference field (to be deinterlaced) using both deformable convolution residual blocks and self attention. Our extensive experimental results demonstrate that the proposed method provides state-of-the-art deinterlacing results in terms of both numerical and perceptual performance. At the time of writing, our model ranks first in the Full FrameRate LeaderBoard at https://videoprocessing.ai/benchmarks/deinterlacer.html
Smart Learning of Click and Refine for Nuclei Segmentation on Histology Images

Deep learning has proven to be a very efficient tool to help pathologists analyze Whole Slide Images (WSI) toward automated classification or segmentation of detailed structures such as nuclei, glands or glomeruli. These objects are particularly relevant for disease diagnosis and staging. Many deep learning methods have shown impressive performance but are still imperfect, while manual segmentation has poor inter-rater agreement. In this paper, we propose a patch-level automated correction of a given baseline initial segmentation, based on deep-learning of segmentation errors and downstream local refinements. Results on the MoNuSeg and PanNuke test datasets show significant improvement of nuclei segmentation quality.
Rotation-Equivariant Graph Convolutional Networks For Spherical Data Via Global-Local Attention

Graph convolutional networks (GCNs) are widely adopted for spherical data processing, striking a balance between rotation equivariance and computation efficiency. However, current GCN-based approaches either apply radial filters with a fixed convolution kernel, or adopt attention schemes to learn a variable kernel based only on node features, which lack expressiveness and may suffer over-smoothing problem. To address this, in this paper, we propose a global-local attention-based spherical graph convolution (GlasGC) for spherical data representation. Specifically, the designed graph convolution includes a structure-enhanced local attention module to estimate the structure correlation efficiently based on topological characteristics of the given spherical graph. The structure correlation is then incorporated with the feature correlation to form the local attention that determines importance of a neighbor node to the center node. We also design a global attention module to obtain the importance scores of different nodes based on both node features and graph topology, which acts as a message-passing enhancer for informative nodes and can relieve the over-smoothing problem. Based on our proposed GlasGC, we design specific GlasGCNs for classification and semantic segmentation of spherical data. Empirical evaluations on these tasks demonstrate that our proposed GlasGCNs can generally achieve better classification and segmentation performance.
Unsupervised Generative Variational Continual Learning

Continual learning aims at learning a sequence of tasks without forgetting any task. While most of the existing literature in continual learning is aimed at class incremental learning in a supervised setting, there is an enormous potential for unsupervised continual learning using generative models. This paper proposes a combination of architectural pruning and neuron addition in generative variational models toward unsupervised generative continual learning (UGCL). Evaluations on standard benchmark data sets demonstrate the superior generative ability of the proposed method.
Efficient Transformer with Locally Shared Attention for Video Quality Assessment

Transformer has shown outstanding performance in time-series data processing, which can definitely facilitate quality assessment of video sequences. However, the quadratic time and memory complexities of Transformer potentially impede its application to long video sequences. In this work, we study a mechanism of sharing attention across video clips in video quality assessment (VQA) scenario. Consequently, an efficient architecture based on integrating shared multi-head attention (MHA) into Transformer is proposed for VQA, which greatly ease the time and memory complexities. A long video sequence is first divided into individual clips. The quality features derived by an image quality model on each frame in a clip are aggregated by a shared MHA layer. The aggregated features across all clips are then fed into a global Transformer encoder for quality prediction at sequence level. The proposed model with a lightweight architecture demonstrates promising performance in no-reference VQA (NR-VQA) modelling on publicly available databases. The source code can be found at https://github.com/junyongyou/lagt_vqa.
Light Field Integral Image Coding Optimization under 2D Hierarchical Coding Structure

In integral photography, one important acquisition method of light field, a 3D scene is captured from different perspectives and the data with repetitive views is contained in one integral image. To encode such integral images, 2D hierarchical coding structure (2D-HCS) is used to exploit the dependency among images of different views. However, the fixed 2D-HCS cannot fully explore the inter-view dependency, making the coding not optimal. Therefore, in this paper, an inter-view dependent rate-distortion optimization (RDO) method is proposed for light field integral image coding under 2D-HCS. Experimental results demonstrate that, compared to 2D-HCS, the proposed method can achieve averaged {Y, U, V} BD-rate savings of {13.2%, 9.2%, 12.0%} only by adapting the Lagrange Multiplier and up to {21.6%, 18.4%, 21.8%} together with QP adaptation.
Deep Image Debanding

Banding or false contour is an annoying visual artifact whose impact negatively degrades the perceptual quality of visual content. Since users are increasingly expecting better visual quality from such content and banding leads to deteriorated quality-of-experience, the area of banding removal or debanding has taken paramount importance. Existing debanding approaches are mostly knowledge-driven, while data-driven debanding approaches remain surprisingly missing. In this work, we construct a large-scale dataset of 51,490 pairs of corresponding pristine and banded image patches, which enables us to make one of the first attempts at developing a deep learning based banding artifact removal method for images that we name deep debanding network (deepDeband). We also develop a bilateral weighting scheme that fuses patch-level debanding results to full-size images. Extensive performance evaluation shows that deepDeband is successful at greatly reducing banding artifacts in images, outperforming existing methods both quantitatively and visually. The proposed algorithm and dataset are made publicly available. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
Cross Domain Low-Dose CT Image Denoising With Semantic Information Alignment

Recently, cross domain adaptation has been applied into quite a few image restoration tasks. While promising performance has been achieved, the domain shift problem between the training set (a.k.a., source domain) and the testing set (a.k.a., target domain) in Low-dose Computed Tomography (LDCT) image denoising tasks is typically ignored by most existing methods. This is prone to the degradation of the denoising performance due to large discrepancy of feature distribution in each dataset from various vendors. Therefore, a simple yet effective LDCT denoising approach has been proposed in this paper to alleviate the domain shift between source and target domains through a novel semantic information alignment. Specifically, we first propose an adaptive version of random frequency mask (RFM) to extract the shared semantic information of cross domains. Then, we incorporate the mask into the existing denoiser to construct a semantic-information-guided objective. Experiments on synthetic and real datasets show our proposed method achieves impressive performance.
Local and Global Fusion Network For Learned Image Compression

Convolution autoencoder is a widely utilized framework for learned image compression. However, it is facing performance bottleneck because the convolution with limited receptive field mainly extracts image local information. In this paper, we propose a novel neural network based image compression framework by removing both local and global redundancy. Herein, convolution autoencoder and Generative flow (Glow) are utilized to extract image local and global information respectively. Glow is a lossless invertible neural network and can facilitate global information extraction. Furthermore, we design a DenseNet module to fuse the local and global information extracted from convolution autoencoder and Glow. Extensive experimental results show that the proposed framework outperform the intra-frame coding of Versatile Video Coding (VVC) and state-of-the-art neural network based image compression methods.
Intra Prediction of Regular and Near-Regular Textures Via Graph-Based Inpainting

Intra prediction is an important technique to improve coding efficiency by exploiting the spatial redundancy present in typical video sequences. In video coding standards such as H.264/AVC, HEVC and VVC, directional predictors are utilized to generate prediction along a single direction within a block to be coded. However, these predictors fail to generate an accurate prediction when the block contains complex patterns such as periodic textures. In this paper, we propose a graph-based inpainting method that can handle both regular and near-regular textures. The proposed inpainting method utilizes a total variation model associated with the Laplacian matrix of a graph, whose edge weights are a function of pixel patch distance. We evaluate the performance of our proposed method as an additional prediction mode combined with the H.264/AVC coding standard. Experimental results show that the proposed method can significantly outperform H.264/AVC predictors in areas with high frequency periodic patterns.
UCF-CAP, Video Captioning in the Wild

Recent technological advances in the fields of data and computer science have improved significantly the everyday life of people. However, technological advances are also being adopted by criminals to facilitate and expand their illicit actions. The Deep Learning (DL) paradigm has shown a significant potential in analysing complex structured data. However, in the crime detection domain, a limited number of public datasets is available, constrained to specific tasks only, which hinders the research and development of accurate and robust DL-assisted tools. The goal of this work is to extend the well-known UCF-crime dataset to the case of video captioning. To the best of our knowledge, this is the first publicly available crime-related video captioning dataset. A new proposed video captioning approach is compared to a plethora of state-of- the-art-methods in this dataset, while qualitative and quantitative characteristics of the latter are presented.
Task-Aware Few-Shot Visual Classification with Improved Self-Supervised Metric Learning

Few-shot learning strategies are developed for training a reliable model on even a limited amount of data, but few-shot learning tasks usually lead to the over-fitting dilemma and result in a task-level inductive bias. In contrast to conventional few-shot learning techniques following the meta-learning framework design, recent few-shot learning studies aim to derive a reliable feature extractors via a self-supervised learning mechanism for solving the dilemma. Therefore, we proposed in this paper a task-aware few-shot visual classification framework by articulating meta-learning, traditional supervised classification, and self-supervised learning schemes. The proposed mechanism learns to transform an initial feature embedding into a more general and representative space so that classification performance can be boosted. Extensive experiments show that the proposed method can solve the over-fitting dilemma and outperforms previous state-of-the-art few-shot learning methods.
Adversarial Training with Channel Attention Regularization

Adversarial attack shows that deep neural networks (DNNs) are highly vulnerable to small perturbation. Currently, one of the most effective ways to defend against adversarial attacks is adversarial training, which generates adversarial examples during training and induces the models to classify them correctly. To further increase robustness, various techniques such as exploiting additional unlabeled data and novel training loss have been proposed. In this paper, we propose a novel regularization method that exploits latent features, which can be easily combined with existing approaches. We discover that particular channels are more sensitive to adversarial perturbation, motivating us to propose regularizing these channels. Specifically, we attach a channel attention module for adjusting sensitivity of each channel by reducing the difference between the latent feature of the natural image and that of the adversarial image, which we call Channel Attention Regularization (CAR). CAR can be combined with the existing adversarial training framework, showing that it improves the robustness of state-of-the-art defense models. Experiments on various existing adversarial training methods against diverse attacks show the effectiveness of our methods. Codes are available at https://github.com/sgmath12/Adversarial-Training-CAR.