Reviewerly/exHarmony huggingface dataset:
Pre-Training with Fractal Images Facilitates Learned Image Quality Estimation

Today’s image quality estimation is widely dominated by learning-based approaches. The availability of annotated, i.e. rated, images is often a bottleneck in training data-driven visual quality models and hinders their generalization power. This paper proposed a novel pre-training scheme for learning-based quality estimation that does not rely on human-annotated datasets, but leverages synthetic fractal images. These images can be synthesized inexhaustibly and are inherently labeled during generation. We evaluate the pre-training strategy on a popular neural network-based quality model and show that the training effort can be reduced significantly, resulting in better final accuracy and faster convergence speed.
A Privacy-Preserving Approach for Multi-Source Domain Adaptive Object Detection

A new research topic, multi-source domain adaptive object detection (MSDAOD) under privacy-preserving constraint is explored in this paper, where the clients can only access their own source data while the server can only access unlabeled target data. Accordingly, a novel MSDAOD framework is proposed wherein the clients employ a source-only Probabilistic Faster R-CNN (PFRCNN) to generate models with localization uncertainty, while a Multi-teacher Pseudo-label Ensemble Network (MPEN) is developed on the server side. In MPEN, FedMA-based algorithm aggregates the above models to a domain-invariant backbone while a novel pseudo-label ensemble (PLE) scheme is employed to reduce false positives arising from domain specific parts, and enhance the overall system performance using target domain information. Experiments demonstrate that our method outperforms other state-of-the-art MSDAOD and privacy-preserving methods by 10%~16% in average precision (AP).
Rectangular-Output Image Stitching

Image stitching aims to combine two images with overlapping fields to expand the field-of-view (FoV). However, the stitched images of existing methods are irregular, and need to be processed by rectangling methods, which is time-consuming and prone to be unnatural. In this paper, we propose the first end-to-end framework, Rectangular-output Deep Image Stitching Network (RDISNet), to directly stitch two images into a standard rectangular image while learning color consistency between image pairs and maintaining the authenticity of the content. To further preserve the structure of large objects in the stitched image, we design a dilated BN-RCU block to expand the receptive field of RDISNet for extracting enriched spatial context. Furthermore, we design a novel data synthesis pipeline and build the first rectangular-output deep image stitching dataset (RDIS-D) for jointing image stitching and rectangling. Experimental results demonstrate that RDISNet performs favorably against the state-of-the-art methods.
MEGL: Multi-Experts Guided Learning Network for Single Camera Training Person Re-Identification

The time-saving single-camera training(SCT) person re-identification aims to learn camera-invariant information without cross-camera pedestrian annotations. To address this challenging task, we propose a novel approach called Multi-Experts Guided Learning Network (MEGL-Net) for SCT-ReID that can obtain features not influenced by camera views at the global and local levels under the guidance of multi-camera experts. Firstly, to obtain camera-invariant features, an adaptive feature integration module (AFI) is introduced to adaptively integrate expert-guided features from different camera branches. Then, the proposed camera-local interactive module (CLI) facilitates interaction between the local branch and the camera experts branch for automatically extracting discriminative, domain-invariant features at a fine-grained level. Finally, our framework aggregates expert-guided features with global features and enhanced local features in the testing stage for pedestrian retrieval. Under the Market-SCT and Duke-SCT datasets, experimental results demonstrate that our approach significantly improves ReID performance and outperforms existing state-of-the-art (SOTA) methods.
Efficient Convolution and Transformer-based Network for Video Frame Interpolation

Video frame interpolation is an increasingly important research task with several key industrial applications in the video coding, broadcast and production sectors. Recently, transformers have been introduced to the field resulting in substantial performance gains. However, this comes at a cost of greatly increased memory usage, training and inference time. In this paper, a novel method integrating a transformer encoder and convolutional features is proposed. This network reduces the memory burden by close to 50% and runs up to four times faster during inference time compared to existing transformer-based interpolation methods. A dual-encoder architecture is introduced which combines the strength of convolutions in modelling local correlations with those of the transformer for long-range dependencies. Quantitative evaluations are conducted on various benchmarks with complex motion to showcase the robustness of the proposed method, achieving competitive performance compared to state-of-the-art interpolation networks.
A Large Scale Multi-View RGBD Visual Affordance Learning Dataset

The physical and textural attributes of objects have been widely studied for recognition, detection and segmentation tasks in computer vision. A number of datasets, such as large scale ImageNet, have been proposed for feature learning using data hungry deep neural networks and for hand-crafted feature extraction. To intelligently interact with objects, robots and intelligent machines need the ability to infer beyond the traditional physical/textural attributes, and understand/learn visual cues, called visual affordances, for affordance recognition, detection and segmentation. To date there is no publicly available large dataset for visual affordance understanding and learning. In this paper, we introduce a large scale multi-view RGBD visual affordance learning dataset, a benchmark of 47210 RGBD images from 37 object categories, annotated with 15 visual affordance categories. To the best of our knowledge, this is the first ever and the largest multi-view RGBD visual affordance learning dataset. We benchmark the proposed dataset for affordance segmentation and recognition tasks using popular Vision Transformer and Convolutional Neural Networks. Several state-of-the-art deep learning networks are evaluated each for affordance recognition and segmentation tasks. Our experimental results showcase the challenging nature of the dataset and present definite prospects for new and robust affordance learning algorithms. The dataset is publicly available at https://sites.google.com/view/afaqshah/dataset.
Subjective Assessment of the Impact of a Content Adaptive Optimiser for Compressing 4K HDR Content With AV1

Since 2015 video dimensionality has expanded to higher spatial and temporal resolutions and a wider colour gamut. This High Dynamic Range (HDR) content has gained traction in the consumer space as it delivers an enhanced quality of experience. At the same time, the complexity of codecs is growing. This has driven the development of tools for content-adaptive optimisation that achieve optimal rate-distortion performance for HDR video at 4K resolution. While improvements of just a few percentage points in BD-Rate (1-5%) are significant for the streaming media industry, the impact on subjective quality has been less studied especially for HDR/AV1. In this paper, we conduct a subjective quality assessment (42 subjects) of 4K HDR content with a per-clip optimisation strategy. We correlate these subjective scores with existing popular objective metrics used in standard development and show that some perceptual metrics correlate surprisingly well even though they are not tuned for HDR. We find that the DSQCS protocol is too insensitive to categorically compare the methods but the data allows us to make recommendations about the use of experts vs non-experts in HDR studies, and explain the subjective impact of film grain in HDR content under compression.
Perception-Oriented Omnidirectional Image Super-Resolution Based on Transformer Network

Omnidirectional image (ODI) super-resolution (SR) is an important technique in augmented reality and virtual reality applications to address the low-resolution problem caused by limitations in capturing devices or bandwidth. The ODI projection distortion makes it challenging to apply existing SR methods. In this paper, we propose an ODI SR method by leveraging the characteristics of ODIs and human visual characteristics. Specifically, we firstly design a perception-orientated adaptive loss function by jointly utilizing saliency map and latitude map. In our proposed ODI-SR network, we introduce an attention module to aggregate multi-scale information and leverage spherical convolution to adapt to the spheric format of ODIs. Furthermore, we design a data augmentation strategy for ODIs according to viewpoint distribution to further improve the visual quality of SR images. Extensive experimental results demonstrate that the proposed method achieves state-of-the-art performance according to both qualitative and quantitative evaluations.
Video Summarization Through Fine-Grained Hierarchical Modeling with Multi-Dimensional Features

Video summarization aims to shorten the video length while maintaining the original video content, which facilitates large-scale video searching and browsing. Most of the existing methods simply take static image features as input, which causes the loss of temporal action information of successive frames. Additionally, the use of two-stage temporal modeling aggravates the loss of temporal relationship. In this paper, we propose a framework based on Fine-Grained Hierarchical Modeling (FGHM) employing multi-dimensional features. Firstly, the multi-dimensional features extractor extracts static image features and dynamic video features. Then dynamic temporal modeling is carried out to model the temporal dependency of the entire video. We also investigate the effects of spatial-temporal features extracted by various 3D features extractors. Extensive experiments demonstrate the effectiveness of FGHM against state-of-the-art methods.
A Light-Weight Hybrid Network for Image dehazing

Single image dehazing is a challenging task in the field of computer vision and image processing. Although several advanced dehazing algorithms based on convolutional neural networks (CNNs) have achieved promising results, their limited receptive fields leave room for improvement. Transformers, another new neural network architecture, have shown significant performance in high-level vision tasks, but applying them directly for image reconstruction tasks may lead to coarse details in results. To address this issue, traditional CNN blocks and a Transformer module are combined, then a light-weight dehazing model is proposed. In the proposed model, convolution layers and a Transformer module are combined as an encoder, in which the convolution layers extract local features (details) while the Transformer module gains global features (structures). Finally, a decoder consisting of traditional CNN blocks restores the haze-free image by merging features of both types. The proposed method is evaluated on the SOTS outdoor dataset and real hazy images and outperforms existing methods with less parameters.
Contour-Assisted Long-Range Perceptual Network for Camouflaged Instance Segmentation

High quality instance segmentation has shown emerging significance in computer vision, especially in complex camouflaged situations. For this reason, we propose a single-stage segmentation model named Contour-assisted Long-range Perceptual Network (CLPNet). By following SOLOv2 model, based on deformable transformer, we propose Aggregate and Optimize multi-layer Transformer for generating refining features. Secondly, through iterative optimization, the full use of long-range context dependencies makes the internal of the instance give strong response, and contour of object tightly surrounds the instance. Experiments on camouflaged object detection dateset show that our method reaches 41.7% AP. Compared with the previous instance segmentation method, it is obviously more effective in small object detection.
Finding Camouflaged Object Guided by Contour and Attention

Camouflaged object detection aims to detect objects closely blended into the background. Inspired by visual mechanism of human being, we define detection process as an end-to-end task consisted of locating and refining. To this end, we propose Contour Supervision and Initial Locating Guidance Network (CSIGNet) to effectively segment camouflaged objects from background. Specifically, our method fully explores the contribution of semantic contour to the binary segmentation task. In addition, attention mechanism is used for our final prediction. Experiments show that it has achieved excellent results on public datasets and the model can accurately segment camouflage objects. Codes will be made available at https://github.com/RecKono/CSIGNet.
LKBQ: Pushing the Limit of Post-Training Quantization to Extreme 1 bit

Recent advances have shown the potential for post-training quantization (PTQ) to reduce excessive hardware resources and quantize deep models to low bits in a short time, compared with Quantization-Aware Training (QAT). However, existing PTQ approaches lose a lot of accuracies when quantizing the model to extremely low bits, e.g., 1 bit. In this work, we propose layer-by-layer self-knowledge distillation binary post-training quantization (LKBQ), the first method capable of quantizing the weights of neural networks to 1 bit in PTQ domain. We show that careful use of layer-by-layer self-distillation within the LKBQ can provide a significant performance boost. Furthermore, our evaluation results show that the initialization of quantized network weights can have a huge impact on the results. Then we propose three methods for weight initialization. Finally, in light of the characteristics of the binarized network, we propose a method named gradient scaling to further improve efficiency. Our experiments show that LKBQ pushes the limit of PTQ to extreme 1-bit for the first time.
TSFC: Texture and Structure Features Coupling for Image Inpainting

Image inpainting has made significant progress benefiting from the advantages of convolutional neural networks (CNNs). Deep learning-based methods have shown extraordinary performance in this field. In this paper, we propose a novel image inpainting architecture with pure CNN that can jointly reconstruct the structure and texture of the image. Our generative network architecture (TSFC) consists of two parallel stages: structure generation and texture generation. In the structure generation stage, we use the large convolution kernel, which is highly neglected in modern networks, using the effective perceptual field of the large convolution kernel to enhance the perception of overall structural features. In the texture generation stage, we use the small convolution kernel to extract local texture features. Qualitative and quantitative experimental results on CelebA-HQ and Paris Street View datasets demonstrate the effectiveness and superiority of our method.
Feature fusion attention network for underwater image enhancement

The attenuation, scattering, and absorption of light in water make underwater imaging a challenging task, and global contextual information is key to enhancing underwater images. However, most existing algorithms rely on simple convolutional neural networks (CNNs), which inherently have locality due to convolutional operations, are difficult to directly obtain global contextual information. The problem is addressed by embedding Swin-Transformer, which has global modeling capability, into U-Net. Specifically, a feature fusion module is constructed to change the way feature propagation occurs between network layers and to fuse context feature information between different network layers. The proposed method is evaluated on the constructed dataset. Experimental results show that the proposed method outperforms existing methods in objective metrics such as peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM), and significantly improved the quality and visibility of underwater images.
Comparison of machine learning techniques for self-collisions checking of manipulating robots

In this paper, we deal with the problem of self-collision detection for a mobile-manipulating robot. Typically, this problem is solved by the method that precisely checks the collision between triangles in the 3D meshes. In this case, the iterative methods for collision checking use techniques like Bounding Volume Hierarchy that reduce the computation time. However, collision checking is still time-consuming during motion planning when this procedure is executed multiple times. To deal with this problem, we propose to define collision detection as a binary classification problem. Then, we show how to collect samples to train the machine learning model for classification. We systematically compare a set of techniques and evaluate them in the task of motion planning for a robotic arm taking into account the accuracy and computation time. The obtained collision classifier is implemented and verified in the Robot Operating System.
SEM-CS: Semantic Clipstyler for Text-Based Image Style Transfer

CLIPStyler demonstrated image style transfer with realistic textures using only a style text description (instead of requiring a reference style image). However, the ground semantics of objects in the style transfer output is lost due to style spill-over on salient and background objects (content mismatch) or over-stylization. To solve this, we propose Semantic CLIP-Styler (Sem-CS), that performs semantic style transfer.Sem-CS first segments the content image into salient and non-salient objects and then transfers artistic style based on a given style text description. The semantic style transfer is achieved using global foreground loss (for salient objects) and global background loss (for non-salient objects). Our empirical results, including DISTS, NIMA and user study scores, show that our proposed framework yields superior qualitative and quantitative performance. Our code is available at github.com/chandagrover/sem-cs.
3D-CSL: Self-Supervised 3D Context Similarity Learning for Near-Duplicate Video Retrieval

In this paper, we introduce 3D-CSL, a compact pipeline for Near-Duplicate Video Retrieval (NDVR), and explore a novel self-supervised learning strategy for video similarity learning. Most previous NDVR methods depend a lot on pair-wise labeled data, so that be limited by the scale of datasets and cannot optimize complex but efficient backbones, e.g., 3D transformers. In order to break this limitation, we explore the self-supervised similarity learning for the NDVR task and propose FCS loss, a novel triplet loss, and ShotMix, a novel video-specific augmentation, which enhances the self-supervised video similarity learning significantly. With this premise, the compact 3D pipeline we propose shows a great advantage in extracting global spatiotemporal dependencies in videos and achieves the best balance between efficiency and effectiveness. Furthermore, we also propose PredMAE to pretrain the 3D transformer with video prediction task as a pretext task to boost the downstream NDVR task without any human labels. The experiments on FIVR-200K and CC_WEB_VIDEO demonstrate the superiority and reliability of our method, which achieves the state-of-the-art performance on clip-level NDVR. Code is released in https://github.com/dun-research/3D-CSL
Improving Adversarial Transferability Via Feature Translation

Deep Neural Networks (DNNs) are vulnerable to adversarial examples, which are crafted to cause the model to make wrong predictions. In real-world scenario, since adversary cannot access to target models, black-box attack has attracted great attention. Among them, many studies have been conducted on transfer-based attacks because they can effectively attack unknown target model. However, transfer-based attacks often fail to fool other models which have slightly different activation maps because adversarial examples tend to overfit to the source model. To alleviate this problem, we introduce Feature Translation Attack (FTA), which applies translation on intermediate features during optimization process. Specifically, FTA generates a new adversarial example whose feature is similar to the ensemble of translated features from the existing adversarial example. We achieved better performance than state-of-the-art methods in extensive experiments.
Interpretable deep learning approach for classification of breast cancer - a comparative analysis of multiple instance learning models

Breast cancer is the most frequent female cancer. Its early diagnosis increases the chances of a complete cure for the patient. Suitably designed deep learning algorithms can be an excellent tool for quick screening analysis and support radiologists and oncologists in diagnosing breast cancer.The design of a deep learning-based system for automated breast cancer diagnosis is not easy due to the lack of annotated data, especially at pixel level, the large size of the images with relatively small cancer lesion sizes and class imbalance, a wide diversity of cancerous lesions, a variety of breasts, both in size and density, make the training of the neural models challenging. Moreover, clinicians are often concerned about using these black-box models because of the lack of transparency in their inference. To address these issues, we propose an approach taking advantage of Multiple Instance Learning (MIL), supported by attention mechanisms. We researched Attention-based MIL (AMIL), Gated AMIL (GAMIL), Dual Stream MIL (DSMIL) and CLustering-constrained AMIL (CLAM) models trained in a weakly-supervised manner and compared them with a common model in image classification tasks, ResNet18.The approach described in this paper is multimodal and combines two mammographic projections (CC and MLO) in the training process. The developed neural system achieved high classification efficiency. Furthermore, exploiting the generated attentional maps allowed the localisation of cancerous lesions, thus increasing the interpretability of the algorithm. Thanks to this mechanism, we were also able to detect artifacts in the analyzed database, difficult to spot but drastically skewing the algorithm’s performance.
Machine Learning Detects a Biopsy Needle in Ultrasound Images

Localization of a biopsy needle in ultrasound (US) images is an important medical image analysis task, as it may help clinicians reduce the risk of damaging the tissue surrounding the cancer and spreading cancerous cells. Despite numerous studies dedicated to segmenting the needle from US, virtually all of them build upon the strong assumption that the needle is present in the image, which does not hold in clinical settings. We address this research gap and propose an end-to-end machine learning approach for biopsy needle detection in US images. The rigorous experimental study revealed that our approach delivers high-quality and fast operation, while offering a high level of flexibility—not only does it allow to update all blocks of the pipeline, but also to build a detection cascade for multi-scale analysis which dramatically reduces the number of sub-images undergoing classification, hence speeds up the detection process.