Reviewerly/exHarmony huggingface dataset:
Image-Text Retrieval With Cross-Modal Semantic Importance Consistency

Cross-modal image-text retrieval is an important area of Vision-and-Language task that models the similarity of image-text pairs by embedding features into a shared space for alignment. To bridge the heterogeneous gap between the two modalities, current approaches achieve inter-modal alignment and intra-modal semantic relationship modeling through complex weighted combinations between items. In the intra-modal association and inter-modal interaction processes, the higher-weight items have a higher contribution to the global semantics. However, the same item always produces different contributions in the two processes, since most traditional approaches only focus on the alignment. This usually results in semantic changes and misalignment. To address this issue, this paper proposes Cross-modal Semantic Importance Consistency (CSIC) which achieves invariance in the semantic of items during aligning. The proposed technique measures the semantic importance of items obtained from intra-modal and inter-modal self-attention and learns a more reasonable representation vector by inter-calibrating the importance distribution to improve performance. We conducted extensive experiments on the Flickr30K and MS COCO datasets. The results show that our approach can significantly improve retrieval performance, proving the proposed approach’s superiority and rationality.
The Extreme Counts: Modeling the Performance Uncertainty of Cloud Resources with Extreme Value Theory

Although Cloud techniques developed rapidly in the last decade, most of the applications running on Cloud are still web-based. It is the performance uncertainty of Cloud resources that hinders the further migration of other applications, such as quality critical applications. Hence, an accurate Cloud performance model is crucial for optimized resource allocation to satisfy the quality requirements of the quality critical applications. However, the existing efforts of Cloud performance modeling focus more on the mean and variance, which cannot be leveraged to guarantee meeting the deadline miss rate of quality critical applications. To tackle the issue, a new modeling method is proposed to build performance uncertainty model of Cloud resources based on Extreme Value Theory, which can generate a proper threshold to guarantee the application's Quality of Service (QoS). Based on our experimental data and studies, the threshold calculated by our proposed model can make the average miss rate become lower than the required 5% deadline miss rate and reduced by 77% compared with the traditional modeling method. The number of times that the deadline miss rate cannot be satisfied is also reduced by 84%.
Split Hierarchical Variational Compression

Variational autoencoders (VAEs) have witnessed great success in performing the compression of image datasets. This success, made possible by the bits-back coding framework, has produced competitive compression performance across many benchmarks. However, despite this, VAE architectures are currently limited by a combination of coding practicalities and compression ratios. That is, not only do state-of the-art methods, such as normalizing flows, often demonstrate out-performance, but the initial bits required in coding makes single and parallel image compression challenging. To remedy this, we introduce Split Hierarchical Variational Compression (SHVC). SHVC introduces two novelties. Firstly, we propose an efficient autoregressive prior, the autoregressive sub-pixel convolution, that allows a generalisation between per-pixel autoregressions and fully factorised probability models. Secondly, we define our coding framework, the autoregressive initial bits, that flexibly supports parallel coding and avoids -for the first time - many of the practicalities commonly associated with bits-back coding. In our experiments, we demonstrate SHVC is able to achieve state-of the-art compression performance across full-resolution lossless image compression tasks, with up to 100x fewer model parameters than competing VAE approaches.
Energy-efficient Oriented Approximate Quantization Scheme for Fine-Grained Sparse Neural Network Acceleration

For edge-side applications with severe power constraints, using pruning and quantization to compress models while maintaining network accuracy has become a widely deployed form of Convolutional Neural Networks (CNNs). For the same model accuracy, fine-grained non-regular pruning can bring higher model compression rate than coarse-grained regular pruning, but also introduces a larger indexing overhead. Besides, this overhead increases dramatically as the pruning granularity decreases. In this work, an approximate quantization scheme for fine-grained pruning is proposed. By reusing part of the quantized data bits, the proposed scheme can merge quantized data and indexes approximately, reducing the indexing overhead as well. Meanwhile, since the approximate compensation of index bits, the proposed scheme achieves an effective model accuracy improvement compared to the case of direct quantization to low bit-width. Experimental results show that, for 2:4 fine-grained pruning and 8-bit quantization scenario, the proposed method can save 20% of memory space and transmission cost. Compared with the direct quantization to 6-bit approach, the proposed scheme improves the accuracy by nearly 0.5% in the simulation of ImageNet dataset on ResNet50, despite occupying the same storage space. When deploying Yolov2-tiny at 16 × compression ratio, the energy efficiency of the CNN accelerator with the proposed approximate quantization is 1.33-3.82× that of other state-of-the-art designs.
Learning Feature Decomposition for Domain Adaptive Monocular Depth Estimation

Monocular depth estimation (MDE) has attracted intense study due to its low cost and critical functions for robotic tasks such as localization, mapping and obstacle detection. Supervised approaches have led to great success with the advance of deep learning, but they rely on large quantities of ground-truth depth annotations that are expensive to acquire. Unsupervised domain adaptation (UDA) transfers knowledge from labeled source data to unlabeled target data, so as to relax the constraint of supervised learning. However, existing UDA approaches may not completely align the domain gap across different datasets because of the domain shift problem. We believe better domain alignment can be achieved via well-designed feature decomposition. In this paper, we propose a novel UDA method for MDE, referred to as Learning Feature Decomposition for Adaptation (LFDA), which learns to decompose the feature space into content and style components. LFDA only attempts to align the content component since it has a smaller domain gap. Meanwhile, it excludes the style component which is specific to the source domain from training the primary task. Furthermore, LFDA uses separate feature distribution estimations to further bridge the domain gap. Extensive experiments on three domain adaptative MDE scenarios show that the proposed method achieves superior accuracy and lower computational cost compared to the state-of-the-art approaches.
The Exploration on Ethical Problems of Educational Metaverse

In recent years, the Metaverse is gaining traction. It is playing an increasingly important role in our lives. The year of 2021 is referred to the first year of the Metaverse. Researches all over the world on the Metaverse showed an explosive force beyond imagination. Beginning with the concept of the Metaverse and the application in educational Metaverse as the guidance, this paper discusses the ethical issues of educational Metaverse based on background of the Metaverse. Ultimately the biggest problem of the Metaverse is a human beings problem. That is to realize the purpose of people-oriented education. This paper mainly examines ethical problems of Metaverse from two aspects. Firstly, the problem of individual autonomy in the educational Metaverse is clearly specified. Secondly, we investigate the issues of privacy in the educational Metaverse. In addition, a series of minor problems such as data distortion, technical training and challenges for teachers are also discussed in detail.
Predicting Long-Term Field of View in 360-Degree Video Streaming

Many virtual reality (VR) and augmented reality (AR) applications are closely associated with the streaming of 360-degree videos. The user only views a portion of the 360-degree video within his field of view (FoV). Thus, long-term prediction of the user's future FoV can save bandwidth resources in on-demand video streaming, minimize video rebuffering time caused by significant bandwidth fluctuations in the network, and reduce unnecessary delays between the server and the client. In this article, we predict the target user's FoV based on the user's past FoV trajectory as well as the other users' future FoV trajectory. Firstly, we propose an eye-gaze-based heatmap to describe the historical movement trajectory of the user's viewport more accurately. Then, we use a method based on other user's eye gaze positions to select similar users. Finally, we introduce the squeeze-and-excitation (SE) network and U-Net (SE-Unet) to combine the historical trajectory of the target user with the future trajectory of other users to predict long-term FoV. Experimental results show that the prediction model proposed in our research is more accurate and durable.
Design and Implementation of Analyzer Management System Based on Elasticsearch

Elasticsearch is one of the most popular full-text search and analytics engine. It can store, search, and analyze big volumes of data in near real time. Before searching, Elasticsearch will build multiple inverted indexes for the data. The analyzer plays a crucial role in this process. An appropriate analyzer can segment text into semantically meaningful words, which can significantly improve the query accuracy. However, the default analyzer has limited performance in the Chinese context. Existing methods generally replace the default analyzer with manual configuration to optimize the query effect. The cost of service downtime caused by manually updating analyzers is often unacceptable in production environments. Based on Elasticsearch's Restful-API, we have implemented a framework for dynamic configuration of analyzers in a cluster environment. The framework supports common Chinese analyzers and provides a visual interface. Experiments show that the framework proposed in this paper reduces the update and maintenance time cost of the analyzer in the online environment by 94% compared to manual update. At the same time, compared with the default analyzer configuration of Elasticsearch, the accuracy of the system based on this framework is improved by 30%.
Causal Influences Decouple From Their Underlying Network Structure In Echo State Networks

Echo State Networks (ESN) are versatile recurrent neural network models in which the hidden layer remains unaltered during training. Interactions among nodes of this static backbone (the network structure) produce diverse representations (i.e., network dynamics) of the given stimuli that are harnessed by a read-out mechanism to perform computations needed for solving a given task (i.e., behavior). Moreover, ESNs are accessible models of neuronal circuits, since they are relatively inexpensive to train. Therefore, ESNs have become attractive for neuroscientists studying the relationship between neural structure, function, and behavior. For instance, it is not yet clear how distinctive connectivity patterns of brain networks (structure) support effective interactions among their nodes (dynamics) and how these patterns of interactions give rise to computation (behavior). To address this question, we employed an ESN with a biologically inspired structure and used a systematic multi-site lesioning framework to quantify the causal contribution of each node to the network's output, thus providing a causal link between network structure and behavior. We then focused on the structure-function relationship and decomposed the causal influence of each node on all other nodes, using the same lesioning framework. We found that nodes in a properly engineered ESN interact with each other largely irrespective of the network's underlying structure. However, in a network with the same topology where the ESN's leakage rate is non-optimal and the dynamics are diminished, the underlying connectivity patterns determine the node interactions. Our results suggest that causal structure-function relations in ESNs can be decomposed into two components, direct and indirect interactions. The former are based on influences relying on structural connections. The latter describe the effective communication between any two nodes through other intermediate nodes. These widely distributed indirect interactions may crucially contribute to the efficient performance of ESNs.
Safety Assurance of Artificial Intelligence-Based Systems: A Systematic Literature Review on the State of the Art and Guidelines for Future Work

The objective of this research is to present the state of the art of the safety assurance of Artificial Intelligence (AI)-based systems and guidelines on future correlated work. For this purpose, a Systematic Literature Review comprising 5090 peer-reviewed references relating safety to AI has been carried out, with focus on a 329-reference subset in which the safety assurance of AI-based systems is directly conveyed. From 2016 onwards, the safety assurance of AI-based systems has experienced significant effervescence and leaned towards five main approaches: performing black-box testing, using safety envelopes, designing fail-safe AI, combining white-box analyses with explainable AI, and establishing a safety assurance process throughout systems’ lifecycles. Each of these approaches has been discussed in this paper, along with their features, pros and cons. Finally, guidelines for future research topics have also been presented. They result from an analysis based on both the cross-fertilization among the reviewed references and the authors’ experience with safety and AI. Among 15 research themes, these guidelines reinforce the need for deepening guidelines for the safety assurance of AI-based systems by, e.g., analyzing datasets from a safety perspective, designing explainable AI, setting and justifying AI hyperparameters, and assuring the safety of hardware-implemented AI-based systems.
How does a virtual human earn your trust?

Virtual humans demonstrate the ability to act as non-judgmental conversational partners, eliciting greater self-disclosure. However, it is unclear what virtual human and conversational characteristics are important when self-disclosing. To address this gap, we conducted a set of qualitative, semi-formal interviews (n = 17) among computer science students to investigate participant mental models of willingness to disclose to virtual humans and characteristics of virtual humans that affect their self-disclosure. Our findings indicate that participants' mental models of virtual humans are largely inconsistent with current literature. This inconsistency appears to eliciting hesitancy and discomfort with virtual humans. Furthermore, trust and listening were identified as two primary characteristics of a virtual human interaction that are valuable towards willingness to disclose. Additionally, these characteristics were also valued in different ways for virtual humans in comparison to real humans. From the interviews, we identify and provide guidelines of designing virtual human interactions and conversations to elicit greater willingness to disclose.
Learning Representations for New Sound Classes With Continual Self-Supervised Learning

In this paper, we work on a sound recognition system that continually incorporates new sound classes. Our main goal is to develop a framework where the model can be updated without relying on labeled data. For this purpose, we propose adopting representation learning, where an encoder is trained using unlabeled data. This learning framework enables the study and implementation of a practically relevant use case where only a small amount of the labels is available in a continual learning context. We also make the empirical observation that a similarity-based representation learning method within this framework is robust to forgetting even if no explicit mechanism against forgetting is employed. We show that this approach obtains similar performance compared to several distillation-based continual learning methods when employed on self-supervised representation learning methods.
Versatile Multi-Modal Pre-Training for Human-Centric Perception

Human-centric perception plays a vital role in vision and graphics. But their data annotations are prohibitively expensive. Therefore, it is desirable to have a versatile pretrain model that serves as a foundation for data-efficient downstream tasks transfer. To this end, we propose the Human-Centric Multi-Modal Contrastive Learning framework HCMoCo that leverages the multi-modal nature of human data (e.g. RGB, depth, 2D keypoints) for effective representation learning. The objective comes with two main challenges: dense pre-train for multi-modality data, efficient usage of sparse human priors. To tackle the challenges, we design the novel Dense Intra-sample Contrastive Learning and Sparse Structure-aware Contrastive Learning targets by hierarchically learning a modal-invariant latent space featured with continuous and ordinal feature distribution and structure-aware semantic consistency. HCMoCo provides pre-train for different modalities by combining heterogeneous datasets, which allows efficient usage of existing task-specific human data. Extensive experiments on four downstream tasks of different modalities demonstrate the effectiveness of HCMoCo, especially under data-efficient settings (7.16% and 12% improvement on DensePose Estimation and Human Parsing). Moreover, we demonstrate the versatility of HCMoCo by exploring cross-modality super-vision and missing-modality inference, validating its strong ability in cross-modal association and reasoning. Codes are available at https://github.com/hongfz16/HCMoCo.
Fine-grained Data Distribution Alignment for Post-Training Quantization

While post-training quantization receives popularity mostly due to its evasion in accessing the original complete training dataset, its poor performance also stems from scarce images. To alleviate this limitation, in this paper, we leverage the synthetic data introduced by zero-shot quantization with calibration dataset and propose a fine-grained data distribution alignment (FDDA) method to boost the performance of post-training quantization. The method is based on two important properties of batch normalization statistics (BNS) we observed in deep layers of the trained network, i.e., inter-class separation and intra-class incohesion. To preserve this fine-grained distribution information: 1) We calculate the per-class BNS of the calibration dataset as the BNS centers of each class and propose a BNS-centralized loss to force the synthetic data distributions of different classes to be close to their own centers. 2) We add Gaussian noise into the centers to imitate the incohesion and propose a BNS-distorted loss to force the synthetic data distribution of the same class to be close to the distorted centers. By utilizing these two fine-grained losses, our method manifests the state-of-the-art performance on ImageNet, especially when both the first and last layers are quantized to the low-bit. Code is at https://github.com/zysxmu/FDDA .
Recommender Systems in E-Commerce

Recommendations as a social process play an important role when people rely on external knowledge to make decisions about the artifact of interest. A recommendation system is an intelligent program that generates a high-quality list of items that a user may be interested in. Today, there are a huge variety of different methods and algorithms for filtering data and providing recommendations. Recommendations can be divided into three main categories: Collaborative Sorting, Content-Based Recommendations and Integrated Recommendations. This paper compares and describes these methods and discusses their limitations by describing the problems plagued by recommendations.E-commerce platform makes easy to sell and buy products on a large customer base. Customers now search on Google instead of directly going certain e-commerce site. E-commerce website helps customers to narrow down their comprehensive ideas and be able to complete the product.
Text Enhancement Network for Cross-Domain Scene Text Detection

Conventional scene text detection approaches essentially assume that training and test data are drawn from the same distribution and have achieved compelling results. However, scene text detectors often suffer from performance degradation in real-world applications, since the feature distribution of training images is different from that of test images obtained from a new scene. To address the above problems, we propose a novel method called Text Enhancement Network (TEN) based on adversarial learning for cross-domain scene text detection. Specifically, we first design a Multi-adversarial Feature Alignment (MFA) module to maximally align features of the source and target data from low-level texture to high-level semantics. Second, we develop the Text Attention Enhancement (TAE) module to re-weigh the importance of text regions and accordingly enhance the corresponding features, in order to improve the robustness against noisy background. Additionally, we design a self-training strategy to further boost the performance of our TEN. We conduct extensive experiments on five benchmarks, and the experimental results demonstrate the effectiveness of our TEN.
A Completely Blind Video Quality Evaluator

Automatic video quality assessment of user-generated content (UGC) has gained increased interest recently, due to the ubiquity of shared video clips uploaded and circulated on social media platforms across the globe. Most existing video quality models developed for this vast content are trained on large numbers of samples labeled during large-scale subjective studies, which are often fail to exhibit adequate generalization abilities on unseen data. Moreover, large labeled video quality datasets are not always available for every scenario, and may not address the coincident evaluation of social videos and the distortions that afflict them. Because of this, it is also desirable to develop opinion-unaware, “completely blind” video quality models, that are free of training, yet can compete with existing learning-based models. Here we propose such a model called VIQE (VIdeo Quality Evaluator), which we designed based on a comprehensive analysis of patch- and frame-wise video statistics, as well as of space-time statistical regularities of videos. The statistical features desired from the analysis capture complementary predictive aspects of perceptual quality, which are aggregated to obtain final video quality scores. Extensive experiments on recent large-scale video quality databases demonstrate that VIQE is even competitive with state-of-the-art opinion-aware models. The source code is being made available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/uniqzheng/Complete-Blind-VQA</uri> .