Reviewerly/exHarmony huggingface dataset:
Spatial-Semantic Attention for Grounded Image Captioning

Grounded image captioning models usually process high-dimensional vectors from the feature extractor to generate descriptions. However, mere vectors do not provide adequate information. The model needs more explicit information for grounded image captioning. Besides high dimensional vectors, the feature extractor also predicts the locations and categories of the objects, which contains low-level spatial information and high-level semantic information. To this end, we propose a new attention module called Spatial-Semantic (SS) Attention, which utilizes the predictions from the backbone network to help the model attend to the correct objects. Specifically, the SS attention module collects the position of proposals and the class probabilities from the feature extractor as spatial and semantic information to assist attention weighting. In addition, we propose a grounding loss to supervise the SS attention. Our method achieves high performance on captioning and grounding metrics and outperforms some powerful previous models on the Flickr30k Entities dataset.
Multi-Modal Transformer for RGB-D Salient Object Detection

The main focus of existing RGB-D salient object detection models is achieving effective multi-modal fusion. Due to the limited receptive field of conventional convolutional neural networks (CNNs), CNN-based multi-modal fusion strategies fail to extensively model the correlation between the two modalities (appearance information from the RGB image and geometric information from the depth data). Given the success of transformer networks for long-range dependency modeling, we investigate multi-modal transformer networks for RGB-D salient object detection. Specifically, a transformer-based multi-modal fusion module is presented to effectively fuse appearance features and geometric features. Experimental results on six challenging benchmark RGB-D salient object detection datasets demonstrate the effectiveness of our approach.
Refining Self-Supervised Learning in Imaging: Beyond Linear Metric

We introduce in this paper a new statistical perspective, exploiting the Jaccard similarity metric, as a measure-based metric to effectively invoke non-linear features in the loss of self-supervised contrastive learning. Specifically, our proposed metric may be interpreted as a dependence measure between two adapted projections learned from the so-called latent representations. This is in contrast to the cosine similarity measure in the conventional contrastive learning model, which accounts for correlation information. To the best of our knowledge, this effectively non-linearly fused information embedded in the Jaccard similarity, is novel to self-supervision learning with promising results. The proposed approach is compared to two state-of-the-art self-supervised contrastive learning methods on three image datasets. We not only demonstrate its amenable applicability in current ML problems, but also its improved performance and training efficiency.
A Robust Misalignment Estimation Approach in Non-Aligned Double JPEG Compression Scenario

The estimation of the misalignment between consecutive JPEG compressions is a really important task that can be useful in forensics investigation for first quantization matrix estimation and forgery localization. Based on the analysis of DCT histograms obtained applying a third compression, an effective shift estimation solution has been designed. More-over, to increase the overall robustness in challenging conditions (e.g., small patches) several fusion strategies combining all the DCT coefficient information have been investigated. Finally, the effectiveness of the proposed solution has been demonstrated considering several scenarios (i.e., different patch sizes and quantization matrices) and comparisons with state-of-the-art solutions.
Reference Guided Reflection Removal Using Deep Visual Attribute Cues

Reflections in images are typically caused due to presence of glass like reflective objects or surfaces that affect the overall visual appeal and hence undesirable. There has been extensive interest in the past to use data driven approaches for both single image as well as multi image reflection removal. However, recently there has been only minor incremental improvements in single image reflection removal given the challenging ill-posed nature of the problem. Reference based methods has yielded state of the art performance in areas such as super resolution, however has been unexplored for reflection removal. In this paper, we propose a novel multi-stage deep learning based method for reference based reflection removal. We also propose a novel visual attribute cue that represents the reflection free semantic content of the input image. This cue is generated using the reference image while maintaining the geometric structure of the input image. We formulate the reference based reflection removal problem as extraction of visual attribute cues followed by a guided image restoration. We perform qualitative and quantitative evaluation to demonstrate the superiority of the proposed approach over the existing state of the art single image reflection removal methods.
Human-Centric Image Retrieval with Gaze-Based Image Captioning

This paper presents human-centric image retrieval with gaze-based image captioning. Although the development of cross-modal embedding techniques has enabled advanced image retrieval, many methods have focused only on the information obtained from the contents such as image and text. For further extending the image retrieval, it is necessary to construct retrieval techniques that directly reflect human intentions. In this paper, we propose a new retrieval approach via image captioning based on gaze information by focusing on the fact that the gaze information obtained from humans contains semantic information. Specifically, we construct a transformer, connect caption and gaze trace (CGT) model that learns the relationship among images, captioning provided by humans and gaze traces. Our CGT model enables transformer-based learning by dividing the gaze traces into several bounding boxes, and thus, gaze-based image captioning becomes feasible. By using the obtained captioning for cross-modal retrieval, we can achieve human-centric image retrieval. The technical contribution of this paper is transforming the gaze trace into the captioning via the transformer-based encoder. In the experiments, by comparing the cross-modal embedding method, the effectiveness of the proposed method is proved.
Towards Lightweight Neural Network-based Chroma Intra Prediction for Video Coding

In video compression the luma channel can be useful for predicting chroma channels (Cb, Cr), as has been demonstrated with the Cross-Component Linear Model (CCLM) used in Versatile Video Coding (VVC) standard. More recently, it has been shown that neural networks can even better capture the relationship among different channels. In this paper, a new attention-based neural network is proposed for cross-component intra prediction. With the goal to simplify neural network design, the new framework consists of four branches: boundary branch and luma branch for extracting features from reference samples, attention branch for fusing the first two branches, and prediction branch for computing the predicted chroma samples. The proposed scheme is integrated into VVC test model together with one additional binary block-level syntax flag which indicates whether a given block makes use of the proposed method. Experimental results demonstrate 0.31%/2.36%/2.00% BD-rate reductions on Y/Cb/Cr components, respectively, on top of the VVC Test Model (VTM) 7.0 which uses CCLM.
Interpretable Concept-Based Prototypical Networks for Few-Shot Learning

Few-shot learning aims at recognizing new instances from classes with limited samples. This challenging task is usually alleviated by performing meta-learning on similar tasks. However, the resulting models are black-boxes. There has been growing concerns about deploying black-box machine learning models and FSL is not an exception in this regard. In this paper, we propose a method for FSL based on a set of human-interpretable concepts. It constructs a set of metric spaces associated with the concepts and classifies samples of novel classes by aggregating concept-specific decisions. The proposed method does not require concept annotations for query samples. This interpretable method achieved results on a par with six previously state-of-the-art black-box FSL methods on the CUB fine-grained bird classification dataset.
Ques-to-Visual Guided Visual Question Answering

Visual question answering (VQA) answers text-based questions about images. The difficulty of VQA lies in the accurate localization of the region related to the question. In this paper, we introduce the ques-to-visual (q2v) feature as the additional input of VQA to tackle this problem. The q2v feature is generated according to the semantics of the question, containing visual semantics that is helpful to locate the region related to the question. We then use self-attention to model the intra-relationship in each modality to enhance different features, i.e., q2v, image, and text features. The enhanced features are then fused by spatial guided-attention and multi-scale channel attention modules for the answer prediction. Experimental results on the VQA2.0 benchmark dataset show that our method achieves higher performance when compared with other methods.
Deep Unrolling of Diffusion Process with Morphological Laplacian and its Implementation with SIMD Instructions

This paper presents a deep network based on unrolling of a diffusion process with morphological Laplacian. The diffusion process is an iterative algorithm that represents time evolution of a partial differential equation with Laplacian. We introduce the morphological Laplacian to the basic diffusion and unwrap to the deep network. The morphological filters are nonlinear operators and have parameters that are referred to as structuring elements. By the training using error back propagation, the network of the morphology can be adapted to image processing applications. Since the morphological filters is realized with addition, max and min, the error due to bit-length of data is not amplified. By this property, the morphological parts of the network are implemented in unsigned 8-bit integer with Single instruction multiple data set (SIMD) to achieve fast computation on the small micro processors. We applied the proposed network to Gaussian denoising and image completion. The results are compared with other de-noising algorithm and networks.
Automatic Detection of Sentimentality from Facial Expressions

Emotion recognition has received considerable attention from the Computer Vision community in the last 20 years. However, most of the research focused on analyzing the six basic emotions (e.g. joy, anger, surprise), with a limited work directed to other affective states. In this paper, we tackle sentimentality (strong feeling of heartwarming or nostalgia), a new emotional state that has few works in the literature, and no guideline defining its facial markers. To this end, we first collect a dataset of 4.9K videos of participants watching some sentimental and non-sentimental ads, and then we label the moments evoking sentimentality in the ads. Second, we use the ad-level labels and the facial Action Units (AUs) activation across different frames for defining some weak frame-level sentimentality labels. Third, we train a Multilayer Perceptron (MLP) using the AUs activation for sentimentality detection. Finally, we define two new ad-level metrics for evaluating our model performance. Quantitative and qualitative results show promising results for sentimentality detection. To the best of our knowledge this is the first work to address the problem of sentimentality detection.
Frequency-Relevant Residual Learning for Multi-Modal Image Denoising

Recently, multi-modal image processing has shown its great potential in boosting denoising performance in terms of both accuracy and visual quality. However, current studies generally face the challenge of achieving a good balance between noise removal and detail preservation. In this work, we introduce patch-wise frequency decomposition into convolutional neural networks and propose a novel multi-modal image denoising (MID) algorithm. Integrating the frequency-domain information of images from different modalities, our network first predicts a frequency-relevant residual and then regresses the denoised result using a learnable reconstruction kernel. Benefiting from the distinctive properties of noise and true signals as well as the correlation between multi-modal images in the frequency domain, the proposed algorithm can effectively remove noise and simultaneously reconstruct fine details. Extensive experiments demonstrate the superiority and generalizability of our algorithm over state-of-the-art competing algorithms on various MID tasks, including near-infrared guided RGB image denoising, flash guided no-flash image denoising, and RGB guided depth image denoising. Code is available at https://github.com/liuxw11/FRL.
Improving Self-Supervised Learning for Out-Of-Distribution Task via Auxiliary Classifier

In real world scenarios, out-of-distribution (OOD) datasets may have a large distributional shift from training datasets. This phenomena generally occurs when a trained classifier is deployed on varying dynamic environments, which causes a significant drop in performance. To tackle this issue, we are proposing an end-to-end deep multi-task network in this work. Observing a strong relationship between rotation prediction (self-supervised) accuracy and semantic classification accuracy on OOD tasks, we introduce an additional auxiliary classification head in our multi-task network along with semantic classification and rotation prediction head. To observe the influence of this addition classifier in improving the rotation prediction head, our proposed learning method is framed into bi-level optimisation problem where the upper-level is trained to update the parameters for semantic classification and rotation prediction head. In the lower-level optimisation, only the auxiliary classification head is updated through semantic classification head by fixing the parameters of the semantic classification head. The proposed method has been validated through three unseen OOD datasets where it exhibits a clear improvement in semantic classification accuracy than other two baseline methods. Our code is available on GitHub https://github.com/harshita-555/OSSL
Efficient and Accurate Skeleton-Based Two-Person Interaction Recognition Using Inter-and Intra-Body Graphs

Skeleton-based two-person interaction recognition has been gaining increasing attention as advancements are made in pose estimation and graph convolutional networks. Although the accuracy has been gradually improving, the increasing computational complexity makes it more impractical for a real-world environment. There is still room for accuracy improvement as the conventional methods do not fully represent the relationship between inter-body joints. In this paper, we propose a lightweight model for accurately recognizing two-person interactions. In addition to the architecture, which incorporates middle fusion, we introduce a factorized convolution technique to reduce the weight parameters of the model. We also introduce a network stream that accounts for relative distance changes between inter-body joints to improve accuracy. Experiments using two large-scale datasets, NTU RGB+D 60 and 120, show that our method simultaneously achieved the highest accuracy and relatively low computational complexity compared with the conventional methods.
Self-Supervised Cooperative Colorization of Achromatic Faces

Despite the recent progress in deep learning-based face image colorization techniques, there is still much room for improvement. One of the significant challenges is the bias toward specific skin color. Moreover, the conventional face colorization approaches aim to produce colored 2D face images, whereas the generation of colored 3D faces from monocular achromatic (gray-scale) images is beyond the scope of these methods despite having immense potential applications. To address these issues, we propose Self-Supervised COoperative COlorizaTion of Achromatic Faces (COCOTA) framework that contains chromatic and achromatic pipelines to jointly estimate the color and shape of 3D faces using monocular achromatic face images without inducing any specific color bias. On the challenging CelebA test dataset, COCOTA out-performs the current state-of-the-art method by a large margin (e.g., for 3D color-based error, a reduction from 5.12 ± 0.13 to 3.09 ± 0.08 leading to an improvement of 39.6%), demonstrating the effectiveness of the proposed method.
Multi-Frame Video Prediction with Learnable Motion Encodings

Predicting multiple future frames from a given video is a challenging problem due to several factors such as changing camera, dynamically moving objects, occlusions, etc. While recent deep learning methods have made significant progress on the video prediction problem, most methods predict the immediate or a fixed number of future frames. To obtain longer-term frame predictions, existing techniques usually process the predicted frames iteratively, resulting in blurry or inconsistent predictions. In this work, we present a new approach that can predict an arbitrary number of future video frames with a single forward pass through the network. Instead of directly predicting a fixed number of future optical flows or frames, we learn temporal motion encodings, i.e., temporal motion basis vectors and a network to predict the coefficients. The learned motion basis can be easily extended to arbitrary length at inference time, enabling us to predict an arbitrary number of future frames. Experiments on benchmark datasets show that our approach performs favorably against several competitive techniques even for the next frame prediction setting. When evaluated under 5-frame or 10-frame prediction settings, the proposed method achieves higher performance gains over the state-of-the-art techniques that iteratively process the predictions.
Image Enhancement for Improved Visibility of Digital Displays Under The Sunlight

Under the sunlight, images displayed on a digital device are generally perceived to be darker than the original, which leads to a decrease in visibility. So, global luminance compensation or tone mapping adaptive to ambient lighting is required. However, global luminance compensation schemes usually have limitations in chrominance compensation as well as local contrast enhancement. This paper proposes a piece-wise linear curve (PLEC)-based image enhancement to enhance both luminance and chrominance. PLECs are regressed through deep learning. In addition, we present a local contrast enhancement scheme for further visibility improvement. Experimental results show that the proposed method outperforms a prior art in terms of subjective/objective visual quality, with significant run-time reduction.
On Quantization of Image Classification Neural Networks for Compression Without Retraining

We studied the quantization of neural networks for their compression and representation without retraining. The goal is to facilitate neural network representation and deployment in standard formats so that general networks may have their weights quantized and entropy coded within the deployment format. We relate weight entropy and model accuracy and try to evaluate distribution of weights against known distributions. Many scalar quantization strategies were tested. We have found that weights are typically approximated by a Laplacian distribution for which optimal quantizers are approximated by entropy-coded uniform quantizers with dead-zones. Results indicate that it is possible to reduce 8-fold the size of the popular image classification networks with accuracy losses near 1%.
Interactive Image Segmentation with Transformers

In this paper, we address the click-based interactive segmentation task with a novel transformer network. Transformer-based approaches show promising results in various computer vision tasks. However, all modern interactive segmentation methods are still based on convolutional networks. We propose a transformer network for interactive segmentation and explore three different ways to feed click information into neural networks. Through extensive evaluation, we show that our model trained on a combination of COCO and LVIS sets a new click-based state-of-the-art on GrabCut, Berkeley, SBD, DAVIS, and Pascal VOC in terms of NoC (Number of Clicks) and mIoU. The source code is available at https://github.com/SamsungLabs/saic-is.
SVBR-Net: A Non-Blind Spatially Varying Defocus Blur Removal Network

Defocus blur is a physical consequence of the optical sensors used in most cameras. Although it can be used as a photographic style, it is commonly viewed as an image degradation modeled as the convolution of a sharp image with a spatially-varying blur kernel. Motivated by the advance of blur estimation methods in the past years, we propose a non-blind approach for image deblurring that can deal with spatially-varying kernels. We introduce two encoder-decoder sub-networks that are fed with the blurry image and the estimated blur map, respectively, and produce as output the deblurred (deconvolved) image. Each sub-network presents several skip connections that allow data propagation from layers spread apart, and also inter-subnetwork skip connections that ease the communication between the modules. The network is trained with synthetically blur kernels that are augmented to emulate blur maps produced by existing blur estimation methods, and our experimental results show that our method works well when combined with a variety of blur estimation methods. Our code will be available at https: //github.com/alikaraali/SVBR_Net_icip.
A Similarity Distillation Guided Feature Refinement Network for Few-Shot Semantic Segmentation

Few-shot semantic segmentation is a challenging task of predicting object categories in pixel-wise with only few annotated samples. Existing methods mainly have two problems, which are representation inconsistency of query and support images and semantic-level feature insufficient. To tackle the two problems, we propose a similarity distillation guided feature refinement network (SD-FRNet). Specifically, we first use support label to generate support similarity feature map and coarse prediction of query image. Then, we use this coarse prediction to generate query similarity feature. To compensate feature representation inconsistency, we conduct knowledge distillation mechanism to align similarity features of query and support images. To enrich semantic-level feature, we further design a feature refinement module, which achieves high-quality segmentation. Extensive experiments show the effectiveness of SD-FRNet. On benchmark datasets, PASCAL-5 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sup> and COCO-20 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sup> , our proposed SD-FRNet outperform the previous SOTA (state-of-the-art) results.