Reviewerly/exHarmony huggingface dataset:
Disentangling Noise from Images: A Flow-Based Image Denoising Neural Network

The prevalent convolutional neural network (CNN)-based image denoising methods extract features of images to restore the clean ground truth, achieving high denoising accuracy. However, these methods may ignore the underlying distribution of clean images, inducing distortions or artifacts in denoising results. This paper proposes a new perspective to treat image denoising as a distribution learning and disentangling task. Since the noisy image distribution can be viewed as a joint distribution of clean images and noise, the denoised images can be obtained via manipulating the latent representations to the clean counterpart. This paper also provides a distribution-learning-based denoising framework. Following this framework, we present an invertible denoising network, FDN, without any assumptions on either clean or noise distributions, as well as a distribution disentanglement method. FDN learns the distribution of noisy images, which is different from the previous CNN-based discriminative mapping. Experimental results demonstrate FDN's capacity to remove synthetic additive white Gaussian noise (AWGN) on both category-specific and remote sensing images. Furthermore, the performance of FDN surpasses that of previously published methods in real image denoising with fewer parameters and faster speed.
Estimating Egocentric 3D Human Pose in the Wild with External Weak Supervision

Egocentric 3D human pose estimation with a single fisheye camera has drawn a significant amount of attention recently. However, existing methods struggle with pose estimation from in-the-wild images, because they can only be trained on synthetic data due to the unavailability of large-scale in-the-wild egocentric datasets. Furthermore, these methods easily fail when the body parts are occluded by or interacting with the surrounding scene. To address the shortage of in-the-wild data, we collect a large-scale in-the-wild egocentric dataset called Egocentric Poses in the Wild (EgoPW). This dataset is captured by a head-mounted fisheye camera and an auxiliary external camera, which provides an additional observation of the human body from a third-person perspective during training. We present a new egocentric pose estimation method, which can be trained on the new dataset with weak external supervision. Specifically, we first generate pseudo labels for the EgoPW dataset with a spatio-temporal optimization method by incorporating the external-view supervision. The pseudo labels are then used to train an egocentric pose estimation network. To facilitate the network training, we propose a novel learning strategy to supervise the egocentric features with the high-quality features extracted by a pretrained external-view pose estimation model. The experiments show that our method predicts accurate 3D poses from a single in-the-wild egocentric image and outperforms the state-of-the-art methods both quantitatively and qualitatively.
Blending Spaces: Cross-Reality Interaction Techniques for Object Transitions Between Distinct Virtual and Augmented Realities

Cross-Reality (CR) involves interaction between different modalities and levels of immersion such as Virtual and Augmented Reality, as we explore in this paper. Whereas previous work assumed similarity between their respective Virtual and Augmented Environment (VE and AE), we explore the case in which VE and AE are distinct. This gives rise to novel and critical problems, such as how to visualise and interact with the other environment. In this context we investigate the fundamental interaction of transitioning an object across environments, to which we contribute five interaction techniques. Two are inspired by literature: Virtual Magic Lens and Binary Transition; while the other three are entirely novel: Auto Blended Space, Manual Blended Space - Button Transition and Manual Blended Space - Touch Transition. In a study evaluating the first four techniques, we found that participants (N=20) performed a CR object manipulation and transition task significantly faster using our Auto Blended Space technique. We then modified Manual Blended Space - Button Transition into Manual Blended Space - Touch Transition in response to these results, and reassessed the four techniques in a more complex object manipulation task (N=16). We found that this type of task was better suited to manual transition methods rather than automatic methods. Taken together, our final contribution are five blended space design factors, and timely Cross-Reality transition design guidelines.
Multigranularity Self-Attention Network for Fine-Grained Ship Detection in Remote Sensing Images

Fine-grained ship detection is very important in the remote sensing field.Most previous remote sensing object detection works only utilize the global features for fine-grained object detection, which ignores the local information, deteriorating the detection performance.In this article, we propose a multigranularity self-attention network (MGANet), which can exploit both the global and local features for fine-grained ship detection.The MGANet consists of two modules: a local-global features alignment module (LAM) and a multigranularity self-attention module (MSM).The LAM is designed to align features of object parts and features of the object by using the convolution with different strides.The MSM introduces a self-attention mechanism that can effectively fuse the global and local features for fine-grained ship detection.In addition, we launched an oriented bounding box-based fine-grained ship detection (OFSD) dataset which is the largest fine-grained ship dataset to test and verify the effectiveness of the proposed MGANet method.Comprehensive evaluations on the OFSD and ShipRSImageNet datasets demonstrate the superiority of our proposed MGANet method over existing state-of-the-art methods for fine-grained ship detection in remote sensing images.
Knowledge Condensation Distillation

Knowledge Distillation (KD) transfers the knowledge from a high-capacity teacher network to strengthen a smaller student. Existing methods focus on excavating the knowledge hints and transferring the whole knowledge to the student. However, the knowledge redundancy arises since the knowledge shows different values to the student at different learning stages. In this paper, we propose Knowledge Condensation Distillation (KCD). Specifically, the knowledge value on each sample is dynamically estimated, based on which an Expectation-Maximization (EM) framework is forged to iteratively condense a compact knowledge set from the teacher to guide the student learning. Our approach is easy to build on top of the off-the-shelf KD methods, with no extra training parameters and negligible computation overhead. Thus, it presents one new perspective for KD, in which the student that actively identifies teacher’s knowledge in line with its aptitude can learn to learn more effectively and efficiently. Experiments on standard benchmarks manifest that the proposed KCD can well boost the performance of student model with even higher distillation efficiency. Code is available at https://github.com/dzy3/KCD .
Time-rEversed DiffusioN tEnsor Transformer: A New TENET of Few-Shot Object Detection

In this paper, we tackle the challenging problem of Few-shot Object Detection. Existing FSOD pipelines (i) use average-pooled representations that result in information loss; and/or (ii) discard position information that can help detect object instances. Consequently, such pipelines are sensitive to large intra-class appearance and geometric variations between support and query images. To address these drawbacks, we propose a Time-rEversed diffusioN tEnsor Transformer (TENET), which i) forms high-order tensor representations that capture multi-way feature occurrences that are highly discriminative, and ii) uses a transformer that dynamically extracts correlations between the query image and the entire support set, instead of a single average-pooled support embedding. We also propose a Transformer Relation Head (TRH), equipped with higher-order representations, which encodes correlations between query regions and the entire support set, while being sensitive to the positional variability of object instances. Our model achieves state-of-the-art results on PASCAL VOC, FSOD, and COCO.
Chaos Based Encryption Technique for Compressed H264/AVC Videos

Saving videos streaming from the unlawful users became an urgent claim. Cryptography and Steganography are the most important and indispensable processes to protect and embed the multimedia data during the transfer or storage process. Chaotic-based cryptography has been proposed to be an efficient encryption technique for multimedia applications and has played a vital role in information security. While the chaotic maps have a long history of researching as a nonlinear dynamic system, visual encryption is useful for concealing messages in videos and safeguarding private content from unauthorized intrusion. In this paper, two new proposed algorithms for compressed videos by the advanced H.264/AVC video coding are presented. First algorithm approach has been implemented by robust video encryption algorithm based on the chaos maps with random key to be tested under different attacks. The second algorithm approach has been implemented by using the hybrid of both steganography and cryptography based on chaotic maps for video frames. The proposed algorithms are conducted on luminance component Y of a set of different YUV video sequences with different resolution using MATLAB software. The simulation results of the proposed algorithms are evaluated using various performance metrics comparing to that with the state-of-art techniques. The suggested approaches have shown to be more resistant to many sorts of attacks and is recommended for future using in real-time applications.
Saliency-Based Centernet for Ship Detection in SAR Images

Ship detection in SAR images plays a significant role in civilian and military fields. The CNN-based detection algorithms have been widely used for ship detection in SAR images and the CenterNet has received extensive attention as an anchor-free detector. However, for the detection method based on CNN, the diversity of data can determine the generalization ability of the detection algorithm. Saliency detection can highlight target regions and suppress background clutter which can produce high-quality SAR images. In this paper, we integrate saliency detection with the CNN network and achieve an end-to-end detection network to complete the ship detection in SAR images. Our proposed saliency-based CenterNet contains two parts, saliency detection and CNN-based detection. The saliency detection can produce high quality slices in which the ship region is retained well, while most sea and land regions around the ship are suppressed. The following CNN-based network CenterNet can automatically extract the features of the ships without human intervention and can effectively distinguish ships and the strong scattering targets on land. Combining saliency detection with CNN-based detection methods can obtain a more accurate target detection model. The experimental results on SSDD demonstrate the effectiveness of the proposed method. Compared with the original CenterNet network, our method improves the mAP by 1.11% on all images of SSDD, and especially improves the mAP by 6.71% for inshore ships detection.
HuMMan: Multi-modal 4D Human Dataset for Versatile Sensing and Modeling

4D human sensing and modeling are fundamental tasks in vision and graphics with numerous applications. With the advances of new sensors and algorithms, there is an increasing demand for more versatile datasets. In this work, we contribute HuMMan, a large-scale multi-modal 4D human dataset with 1000 human subjects, 400k sequences and 60M frames. HuMMan has several appealing properties: 1) multi-modal data and annotations including color images, point clouds, keypoints, SMPL parameters, and textured meshes; 2) popular mobile device is included in the sensor suite; 3) a set of 500 actions, designed to cover fundamental movements; 4) multiple tasks such as action recognition, pose estimation, parametric human recovery, and textured mesh reconstruction are supported and evaluated. Extensive experiments on HuMMan voice the need for further study on challenges such as fine-grained action recognition, dynamic human mesh reconstruction, point cloud-based parametric human recovery, and cross-device domain gaps (Homepage: https://caizhongang.github.io/projects/HuMMan/ ).
Analysis of Protein Structure for Drug Repurposing Using Computational Intelligence and ML Algorithm

Proteins are fundamental compounds in biological processes during the analysis of drug target indication for drug repurposing. The identification of relevant features is a necessary step in determining protein structure. A classification technique is used to identify the most important features in a dataset, which is why feature selection is so important. For protein structure prediction, recent research has developed a wide range of new methods to improve accuracy. The authors use principal component analysis (PCA) with correlation-matrix-based feature selection to analyse breast cancer data. In this paper, they discussed a therapeutic agent that is used to reduce the dataset by reduction-based algorithm and after that applied reduced dataset labelled as Standard Gold Dataset on machine learning model to analyze drug target indication. They get the higher accuracy of 92.8%, 93.9%, and 95.3%, each of the three datasets with 200, 500, and 1000 features with SVM with RBF kernel function. Also they found the best result, 97.8%, with the same classifier.
Few-Shot Head Swapping in the Wild

The head swapping task aims at flawlessly placing a source head onto a target body, which is of great importance to various entertainment scenarios. While face swapping has drawn much attention, the task of head swapping has rarely been explored, particularly under the few-shot setting. It is inherently challenging due to its unique needs in head modeling and background blending. In this paper, we present the Head Swapper (HeSer), which achieves few-shot head swapping in the wild through two delicately de-signed modules. Firstly, a Head2Head Aligner is devised to holistically migrate pose and expression information from the target to the source head by examining multi-scale in-formation. Secondly, to tackle the challenges of skin color variations and head-background mismatches in the swapping procedure, a Head2Scene Blender is introduced to si-multaneously modify facial skin color and fill mismatched gaps on the background around the head. Particularly, seamless blending is achieved with the help of a Semantic-Guided Color Reference Creation procedure and a Blending UNet. Extensive experiments demonstrate that the proposed method produces superior head swapping results on a variety of scenes.
DCCF: Deep Comprehensible Color Filter Learning Framework for High-Resolution Image Harmonization

Image color harmonization algorithm aims to automatically match the color distribution of foreground and background images captured in different conditions. Previous deep learning based models neglect two issues that are critical for practical applications, namely high resolution (HR) image processing and model comprehensibility. In this paper, we propose a novel Deep Comprehensible Color Filter (DCCF) learning framework for high-resolution image harmonization. Specifically, DCCF first downsamples the original input image to its low-resolution (LR) counter-part, then learns four human comprehensible neural filters (i.e. hue, saturation, value and attentive rendering filters) in an end-to-end manner, finally applies these filters to the original input image to get the harmonized result. Benefiting from the comprehensible neural filters, we could provide a simple yet efficient handler for users to cooperate with deep model to get the desired results with very little effort when necessary. Extensive experiments demonstrate the effectiveness of DCCF learning framework and it outperforms state-of-the-art post-processing method on iHarmony4 dataset on images’ full-resolutions by $$7.63\%$$ and $$1.69\%$$ relative improvements on MSE and PSNR, respectively. Our code is available at https://github.com/rockeyben/DCCF .
Interactive Image Synthesis with Panoptic Layout Generation

Interactive image synthesis from user-guided input is a challenging task when users wish to control the scene structure of a generated image with ease. Although remarkable progress has been made on layout-based image synthesis approaches, existing methods require high-precision inputs such as accurately placed bounding boxes, which might be constantly violated in an interactive setting. When placement of bounding boxes is subject to perturbation, layout-based models suffer from “missing regions” in the constructed semantic layouts and hence undesirable artifacts in the generated images. In this work, we propose Panoptic Layout Generative Adversarial Network (PLGAN) to address this challenge. The PLGAN employs panoptic theory which distinguishes object categories between “stuff” with amorphous boundaries and “things” with well-defined shapes, such that stuff and instance layouts are constructed through separate branches and later fused into panoptic layouts. In particular, the stuff layouts can take amorphous shapes and fill up the missing regions left out by the instance layouts. We experimentally compare our PLGAN with state-of-the-art layout-based models on the COCO-Stuff, Visual Genome, and Landscape datasets. The advantages of PLGAN are not only visually demonstrated but quantitatively verified in terms of inception score, Fréchet inception distance, classification accuracy score, and coverage. The code is available at https://github.com/wb-finalking/PLGAN.
Unsupervised Domain Adaptation for Monocular 3D Object Detection via Self-training

Monocular 3D object detection (Mono3D) has achieved unprecedented success with the advent of deep learning techniques and emerging large-scale autonomous driving datasets. However, drastic performance degradation remains an unwell-studied challenge for practical cross-domain deployment as the lack of labels on the target domain. In this paper, we first comprehensively investigate the significant underlying factor of the domain gap in Mono3D, where the critical observation is a depth-shift issue caused by the geometric misalignment of domains. Then, we propose STMono3D, a new self-teaching framework for unsupervised domain adaptation on Mono3D. To mitigate the depth-shift, we introduce the geometry-aligned multi-scale training strategy to disentangle the camera parameters and guarantee the geometry consistency of domains. Based on this, we develop a teacher-student paradigm to generate adaptive pseudo labels on the target domain. Benefiting from the end-to-end framework that provides richer information of the pseudo labels, we propose the quality-aware supervision strategy to take instance-level pseudo confidences into account and improve the effectiveness of the target-domain training process. Moreover, the positive focusing training strategy and dynamic threshold are proposed to handle tremendous FN and FP pseudo samples. STMono3D achieves remarkable performance on all evaluated datasets and even surpasses fully supervised results on the KITTI 3D object detection dataset. To the best of our knowledge, this is the first study to explore effective UDA methods for Mono3D.
Towards Efficient Adversarial Training on Vision Transformers

Vision Transformer (ViT), as a powerful alternative to Convolutional Neural Network (CNN), has received much attention. Recent work showed that ViTs are also vulnerable to adversarial examples like CNNs. To build robust ViTs, an intuitive way is to apply adversarial training since it has been shown as one of the most effective ways to accomplish robust CNNs. However, one major limitation of adversarial training is its heavy computational cost. The self-attention mechanism adopted by ViTs is a computationally intense operation whose expense increases quadratically with the number of input patches, making adversarial training on ViTs even more time-consuming. In this work, we first comprehensively study fast adversarial training on a variety of vision transformers and illustrate the relationship between the efficiency and robustness. Then, to expediate adversarial training on ViTs, we propose an efficient Attention Guided Adversarial Training mechanism. Specifically, relying on the specialty of self-attention, we actively remove certain patch embeddings of each layer with an attention-guided dropping strategy during adversarial training. The slimmed self-attention modules accelerate the adversarial training on ViTs significantly. With only $$65\%$$ of the fast adversarial training time, we match the state-of-the-art results on the challenging ImageNet benchmark.
Detecting Generated Images by Real Images

The widespread of generative models have called into question the authenticity of many things on the web. In this situation, the task of image forensics is urgent. The existing methods examine generated images and claim a forgery by detecting visual artifacts or invisible patterns, resulting in generalization issues. We observed that the noise pattern of real images exhibits similar characteristics in the frequency domain, while the generated images are far different. Therefore, we can perform image authentication by checking whether an image follows the patterns of authentic images. The experiments show that a simple classifier using noise patterns can easily detect a wide range of generative models, including GAN and flow-based models. Our method achieves state-of-the-art performance on both low- and high-resolution images from a wide range of generative models and shows superior generalization ability to unseen models. The code is available at https://github.com/Tangsenghenshou/Detecting-Generated-Images-by-Real-Images .
The Dark Side of Metaverse Learning Environments

What is knows about the Metaverse, which is intended to be monopolized by high-end technology corporations, that can be introduced as the true owners of the new world? Allegedly, the Metaverse is a platform made up of digital twins, modeled environments, and advanced reality technologies that will go beyond what we currently understand. The crucial aspect here is talking about things “beyond what we know.” In addition to numerous potential advantages, the metaverse—which might be highlighted as the final benefit provided by the internet revolution to humanity—also poses significant risks and threats that must not be overlooked. This chapter aims to enlighten readers on the threats and risks associated with using Metaverse learning environments, even though this was not their intended use from their design. It is crucial to highlight that Metaverse, which is currently solely organized under the control of a few high-end technology corporations, has the potential to become a dystopian nightmare for everyone, thus we must be vigilant to avoid this from happening.