Reviewerly/exHarmony huggingface dataset:
A novel image encryption algorithm based on hybrid chaotic maps using a key image

One of the most important data currently shared on the Internet is images of various sizes and types, gray and color, private and public, civilian and military, medical and scientific, commercial, and others. The biggest challenge and problem is the secure transmission of these images in this insecure environment. In this context, this paper proposes a new image encryption algorithm using a hybrid of three modified and improved chaotic one-dimensional (1D) maps to avoid the shortcomings of 1D maps and multidimensional (MD) maps. A key image is used to initialize the chaotic maps and is also used as a mask in the diffusion phase with the eXclusive OR (XOR) operator. The encryption process uses ExtraParam extracted from the original image, which is very sensitive to bit changes during the initialization of the chaotic maps. The proposed encryption process consists of two main phases: confusion and diffusion processes. The proposed image encryption algorithm has successfully passed numerous tests and cryptanalysis. We also measure the proposed algorithm with various analysis experiments such as histogram and entropy analysis, the number of pixel change rate (NPCR), the unified average changing intensity (UACI), the mean square error(MSE), the peak signal-to-noise ratio (PSNR) and Key Space. The experimental results and comparative analysis show that the proposed encryption algorithm has excellent performance, is strong enough to withstand various attacks, and provides excellent privacy to digital images.
TETFN: A text enhanced transformer fusion network for multimodal sentiment analysis

Multimodal sentiment analysis (MSA), which aims to recognize sentiment expressed by speakers in videos utilizing textual, visual and acoustic cues, has attracted extensive research attention in recent years. However, textual, visual and acoustic modalities often contribute differently to sentiment analysis. In general, text contains more intuitive sentiment-related information and outperforms nonlinguistic modalities in MSA. Seeking a strategy to take advantage of this property to obtain a fusion representation containing more sentiment-related information and simultaneously preserving inter- and intra-modality relationships becomes a significant challenge. To this end, we propose a novel method named Text Enhanced Transformer Fusion Network (TETFN), which learns text-oriented pairwise cross-modal mappings for obtaining effective unified multimodal representations. In particular, it incorporates textual information in learning sentiment-related nonlinguistic representations through text-based multi-head attention. In addition to preserving consistency information by cross-modal mappings, it also retains the differentiated information among modalities through unimodal label prediction. Furthermore, the vision pre-trained model Vision-Transformer is utilized to extract visual features from the original videos to preserve both global and local information of a human face. Extensive experiments on benchmark datasets CMU-MOSI and CMU-MOSEI demonstrate the superior performance of the proposed TETFN over state-of-the-art methods.
Automatic translation of sign language with multi-stream 3D CNN and generation of artificial depth maps

Sign languages play an essential role in the cognitive and social development of the deaf, consisting of a natural form of communication and being a symbol of identity and culture. However, hearing loss has a severe social impact due to an existing communication barrier, preventing access to essential services such as education and health. A bi-directional sign language translation may be the solution to bridging the communication gap between the deaf and the listener, completing a two-way communication cycle. Virtual personal assistants can benefit from this technology by extending how users interact with the intelligent system. With this idea, in this work we develop a multi-stream deep learning model to recognize signs of Brazilian (BSL), Indian (ISL), and Korean (KSL) Sign Languages. We combine different types of information for the classification task, using single-stream and multi-stream 3D Convolutional Neural Networks. In addition, considering the largest source of sign data globally – the internet – we propose a depth sensor-free classification method, with depth maps artificially generated through Generative Adversarial Networks. In order to consider the main parameters that encode sign languages, the final architecture is composed of a multi-stream network that receives the segmented hands, the faces, the distances and speeds of the points of articulation, and the RGB frames associated with artificial depth maps. Finally, we provide a visual explanation to understand which regions were important for model decision-making. The best models were obtained using the multi-stream network, presenting an accuracy of 0.91 ± 0.07, and f1-score of 0.90 ± 0.08 on publicly available BSL data set. The results suggest that the multi-stream network with artificially generated depth maps is suitable for the task of sign recognition in different languages.
Diverse approaches to learning with immersive Virtual Reality identified from a systematic review

To investigate how learning in immersive Virtual Reality was designed in contemporary educational studies, this systematic literature review identified nine design features and analysed 219 empirical studies on the designs of learning activities with immersive Virtual Reality. Overall, the technological features for physical presence were more readily implemented and investigated than pedagogical features for learning engagement. Further analysis with k-means clustering revealed five approaches with varying levels of interactivity and openness in learning tasks, from watching virtual worlds passively to responding to personalised prompts. Such differences in the design appeared to stem from different practical and educational priorities, such as accessibility, interactivity, and engagement. This review highlights the diversity in the learning task designs in immersive Virtual Reality and illustrates how researchers are navigating practical and educational concerns. We recommend future empirical studies recognise the different approaches and priorities when designing and evaluating learning with immersive Virtual Reality. We also recommend that future systematic reviews investigate immersive Virtual Reality-based learning not only by learning topics or learner demographics, but also by task designs and learning experiences.
What the Literature on Medicine, Nursing, Public Health, Midwifery, and Dentistry Reveals: An Overview of the Rapidly Approaching Metaverse

What is being done in the metaverse literature in a variety of health-related fields, including Medicine, Nursing, Public Health, Midwifery, and Dentistry, and what topics are explored in this literature? In this study, a content analysis of the studies linked with the term metaverse in all health-related literature is presented. The studies were carried out on the documents accessed in Web of Science and Scopus searches made with the words "Metaverse" and "metaverse" on 06/05/2022. For 312 papers, data was retrieved from the Scopus bibliometric database, and for 240 publications, data was retrieved from the Web of Science data source. In the field of health, there haven't been many direct studies on metaverse technology. This is because metaverse technology is made up of many different technologies that work together and are always getting better. Some of these technologies are augmented reality, social networks, virtual reality, blockchain technology, artificial intelligence, and avatar. The study also tells field researchers about metaverse studies that are important in the health sector and about changes in the literature as a whole. According to the research areas, the documents found had to do with the field of health, and only the researches in the field of health were evaluated. It is interesting to think about how metaverse technology will change scientific research, health services, and health-related processes. The study looks at the current state of the literature on metaverse technology, as well as the future of the field and what it can do. Virtual reality or augmented reality applications have defined themselves as augmented reality under the metaverse. These applications are most useful in critical situations in the health sector and in processes that are hard and must be done by trial and error. In addition to the tremendous advances in technologies (blockchain, computer vision, haptic devices, sensors, computer networks) with which metaverse technology is associated in recent years, the mechanical resolution of the sense of touch in the Nobel Peace Prize in physiology and medicine received in 2021 can be seen as one of the field's critical milestones. The growth of the field and past experiences show that many real-world applications can be moved to the metaverse universe in the future. One of the fields that will be affected the most by this interaction is health care.
Cross-domain Few-shot Learning with Task-specific Adapters

In this paper, we look at the problem of cross-domain few-shot classification that aims to learn a classifier from previously unseen classes and domains withfew labeled samples. Recent approaches broadly solve this problem by pa-rameterizing their few-shot classifiers with task-agnostic and task-specific weights where the former is typically learned on a large training set and the latter is dynamically predicted through an auxiliary network conditioned on a small support set. In this work, we focus on the estimation of the latter, and propose to learn task-specific weights from scratch directly on a small support set, in contrast to dynamically estimating them. In particular, through systematic analysis, we show that task-specific weights through parametric adapters in matrix form with residual connections to multiple intermediate layers of a backbone network significantly improves the per-formance of the state-of-the-art models in the Meta-Dataset benchmark with minor additional cost.
GIRAFFE HD: A High-Resolution 3D-aware Generative Model

3D-aware generative models have shown that the introduction of 3D information can lead to more controllable image generation. In particular, the current state-of-the-art model GIRAFFE [38] can control each object's rotation, translation, scale, and scene camera pose without corresponding supervision. However, GIRAFFE only operates well when the image resolution is low. We propose GIRAFFE HD, a high-resolution 3D-aware generative model that inherits all of GIRAFFE's controllable features while generating high-quality, high-resolution images (512 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> resolution and above). The key idea is to leverage a style- based neural renderer, and to independently generate the foreground and background to force their disentanglement while imposing consistency constraints to stitch them together to composite a coherent final image. We demonstrate state-of-the-art 3D controllable high-resolution image generation on multiple natural image datasets.
InOut: Diverse Image Outpainting via GAN Inversion

Image outpainting seeks for a semantically consistent extension of the input image beyond its available content. Compared to inpainting - filling in missing pixels in a way coherent with the neighboring pixels - outpainting can be achieved in more diverse ways since the problem is less constrained by the surrounding pixels. Existing image outpainting methods pose the problem as a conditional image-to-image translation task, often generating repetitive structures and textures by replicating the content available in the input image. In this work, we formulate the problem from the perspective of inverting generative adversarial networks. Our generator renders micro-patches conditioned on their joint latent code as well as their individual positions in the image. To outpaint an image, we seek for multiple latent codes not only recovering available patches but also synthesizing diverse outpainting by patch-based generation. This leads to richer structure and content in the outpainted regions. Furthermore, our formulation allows for outpainting conditioned on the categorical input, thereby enabling flexible user controls. Extensive experimental results demonstrate the proposed method performs favorably against existing in- and outpainting methods, featuring higher visual quality and diversity.
Source-Free Domain Adaptation via Distribution Estimation

Domain Adaptation aims to transfer the knowledge learned from a labeled source domain to an unlabeled target domain whose data distributions are different. However, the training data in source domain required by most of the existing methods is usually unavailable in real-world applications due to privacy preserving policies. Recently, Source-Free Domain Adaptation (SFDA) has drawn much attention, which tries to tackle domain adaptation problem without using source data. In this work, we propose a novel framework called SFDA-DE to address SFDA task via source Distribution Estimation. Firstly, we produce robust pseudo-labels for target data with spherical k-means clustering, whose initial class centers are the weight vectors (anchors) learned by the classifier of pretrained model. Furthermore, we propose to estimate the class-conditioned feature distribution of source domain by exploiting target data and corresponding anchors. Finally, we sample surrogate features from the estimated distribution, which are then utilized to align two domains by minimizing a contrastive adaptation loss function. Extensive experiments show that the proposed method achieves state-of-the-art performance on multiple DA benchmarks, and even outperforms traditional DA methods which require plenty of source data.
Cross-patch Dense Contrastive Learning for Semi-supervised Segmentation of Cellular Nuclei in Histopathologic Images

We study the semi-supervised learning problem, using a few labeled data and a large amount of unlabeled data to train the network, by developing a cross-patch dense contrastive learning framework, to segment cellular nuclei in histopathologic images. This task is motivated by the expensive burden on collecting labeled data for histopathologic image segmentation tasks. The key idea of our method is to align features of teacher and student networks, sampled from cross-image in both patch- and pixel-levels, for enforcing the intra-class compactness and inter-class separability of features that as we shown is helpful for extracting valuable knowledge from unlabeled data. We also design a novel optimization framework that combines consistency regularization and entropy minimization techniques, showing good property in eviction of gradient vanishing. We assess the proposed method on two publicly available datasets, and obtain positive results on extensive experiments, outperforming the state-of-the-art methods. Codes are available at https://github.com/zzw-szu/CDCL.
FILM: Frame Interpolation for Large Motion

We present a frame interpolation algorithm that synthesizes an engaging slow-motion video from near-duplicate photos which often exhibit large scene motion. Near-duplicates interpolation is an interesting new application, but large motion poses challenges to existing methods. To address this issue, we adapt a feature extractor that shares weights across the scales, and present a “scale-agnostic” motion estimator. It relies on the intuition that large motion at finer scales should be similar to small motion at coarser scales, which boosts the number of available pixels for large motion supervision. To inpaint wide disocclusions caused by large motion and synthesize crisp frames, we propose to optimize our network with the Gram matrix loss that measures the correlation difference between features. To simplify the training process, we further propose a unified single-network approach that removes the reliance on additional optical-flow or depth network and is trainable from frame triplets alone. Our approach outperforms state-of-the-art methods on the Xiph large motion benchmark while performing favorably on Vimeo-90K, Middlebury and UCF101. Source codes and pre-trained models are available at https://film-net.github.io .
Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformer

Recent developments in transformer models for visual data have led to significant improvements in recognition and detection tasks. In particular, using learnable queries in place of region proposals has given rise to a new class of one-stage detection models, spearheaded by the Detection Transformer (DETR). Variations on this one-stage approach have since dominated human-object interaction (HOI) detection. However, the success of such one-stage HOI detectors can largely be attributed to the representation power of transformers. We discovered that when equipped with the same transformer, their two-stage counterparts can be more performant and memory-efficient, while taking a fraction of the time to train. In this work, we propose the Unary-Pairwise Transformer, a two-stage detector that exploits unary and pairwise representations for HOIs. We observe that the unary and pairwise parts of our transformer network specialise, with the former preferentially increasing the scores of positive examples and the latter decreasing the scores of negative examples. We evaluate our method on the HICO-DET and V-COCO datasets, and significantly outperform state-of-the-art approaches. At inference time, our model with ResNet50 approaches real-time performance on a single GPU.
CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation

Top-down methods dominate the field of 3D human pose and shape estimation, because they are decoupled from human detection and allow researchers to focus on the core problem. However, cropping, their first step, discards the location information from the very beginning, which makes themselves unable to accurately predict the global rotation in the original camera coordinate system. To address this problem, we propose to Carry Location Information in Full Frames (CLIFF) into this task. Specifically, we feed more holistic features to CLIFF by concatenating the cropped-image feature with its bounding box information. We calculate the 2D reprojection loss with a broader view of the full frame, taking a projection process similar to that of the person projected in the image. Fed and supervised by global-location-aware information, CLIFF directly predicts the global rotation along with more accurate articulated poses. Besides, we propose a pseudo-ground-truth annotator based on CLIFF, which provides high-quality 3D annotations for in-the-wild 2D datasets and offers crucial full supervision for regression-based methods. Extensive experiments on popular benchmarks show that CLIFF outperforms prior arts by a significant margin, and reaches the first place on the AGORA leaderboard (the SMPL-Algorithms track). The code and data are available at https://github.com/huawei-noah/noah-research/tree/master/CLIFF .
Perceiving and Modeling Density for Image Dehazing

In the real world, the degradation of images taken under haze can be quite complex, where the spatial distribution of haze varies from image to image. Recent methods adopt deep neural networks to recover clean scenes from hazy images directly. However, due to the generic design of network architectures and the failure in estimating an accurate haze degradation model, the generalization ability of recent dehazing methods on real-world hazy images is not ideal. To address the problem of modeling real-world haze degradation, we propose a novel Separable Hybrid Attention (SHA) module to perceive haze density by capturing positional-sensitive features in the orthogonal directions to achieve this goal. Moreover, a density encoding matrix is proposed to model the uneven distribution of the haze explicitly. The density encoding matrix generates positional encoding in a semi-supervised way – such a haze density perceiving and modeling strategy captures the unevenly distributed degeneration at the feature-level effectively. Through a suitable combination of SHA and density encoding matrix, we design a novel dehazing network architecture, which achieves a good complexity-performance trade-off. Comprehensive evaluation on both synthetic datasets and real-world datasets demonstrates that the proposed method surpasses all the state-of-the-art approaches with a large margin both quantitatively and qualitatively. The code is released in https://github.com/Owen718/ECCV22-Perceiving-and-Modeling-Density-for-Image-Dehazing .
Siamese Contrastive Embedding Network for Compositional Zero-Shot Learning

Compositional Zero-Shot Learning (CZSL) aims to recognize unseen compositions formed from seen state and object during training. Since the same state may be various in the visual appearance while entangled with different objects, CZSL is still a challenging task. Some methods recognize state and object with two trained classifiers, ignoring the impact of the interaction between object and state; the other methods try to learn the joint representation of the state-object compositions, leading to the domain gap between seen and unseen composition sets. In this paper, we propose a novel Siamese Contrastive Embedding Network (SCEN) <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Code: https://github.com/XDUxyLi/SCEN-master for unseen composition recognition. Considering the entanglement between state and object, we embed the visual feature into a Siamese Contrastive Space to capture prototypes of them separately, alleviating the interaction between state and object. In addition, we design a State Transition Module (STM) to increase the diversity of training compositions, improving the robustness of the recognition model. Extensive experiments indicate that our method significantly outperforms the state-of-the-art approaches on three challenging benchmark datasets, including the recent proposed C-QGA dataset.
ClusterGNN: Cluster-based Coarse-to-Fine Graph Neural Network for Efficient Feature Matching

Graph Neural Networks (GNNs) with attention have been successfully applied for learning visual feature matching. However, current methods learn with complete graphs, resulting in a quadratic complexity in the number of features. Motivated by a prior observation that self- and cross- attention matrices converge to a sparse representation, we propose ClusterGNN, an attentional GNN architecture which operates on clusters for learning the feature matching task. Using a progressive clustering module we adaptively divide keypoints into different subgraphs to reduce redundant connectivity, and employ a coarse-to-fine paradigm for mitigating miss-classification within images. Our approach yields a 59.7% reduction in runtime and 58.4% reduction in memory consumption for dense detection, compared to current state-of-the-art GNN-based matching, while achieving a competitive performance on various computer vision tasks.