Reviewerly/exHarmony huggingface dataset:
Hybrid Relation Guided Set Matching for Few-shot Action Recognition

Current few-shot action recognition methods reach impressive performance by learning discriminative features for each video via episodic training and designing various temporal alignment strategies. Nevertheless, they are limited in that (a) learning individual features without considering the entire task may lose the most relevant information in the current episode, and (b) these alignment strategies may fail in misaligned instances. To overcome the two limitations, we propose a novel Hybrid Relation guided Set Matching (HyRSM) approach that incorporates two key components: hybrid relation module and set matching metric. The purpose of the hybrid relation module is to learn task-specific embeddings by fully exploiting associated relations within and cross videos in an episode. Built upon the task-specific features, we reformulate distance measure between query and support videos as a set matching problem and further design a bidirectional Mean Hausdorff Metric to improve the resilience to misaligned instances. By this means, the proposed HyRSM can be highly informative and flexible to predict query categories under the few-shot settings. We evaluate HyRSM on six challenging benchmarks, and the experimental results show its superiority over the state-of-the-art methods by a convincing margin. Project page: https://hyrsm-cvpr2022.github.io/.
Catching Both Gray and Black Swans: Open-set Supervised Anomaly Detection

Despite most existing anomaly detection studies assume the availability of normal training samples only, a few labeled anomaly examples are often available in many real-world applications, such as defect samples identified during random quality inspection, lesion images confirmed by radiologists in daily medical screening, etc. These anomaly examples provide valuable knowledge about the application-specific abnormality, enabling significantly improved detection of similar anomalies in some recent models. However, those anomalies seen during training often do not illustrate every possible class of anomaly, rendering these models ineffective in generalizing to unseen anomaly classes. This paper tackles open-set supervised anomaly detection, in which we learn detection models using the anomaly examples with the objective to detect both seen anomalies (‘gray swans’) and unseen anomalies (‘black swans’). We propose a novel approach that learns disentangled representations of abnormalities illustrated by seen anomalies, pseudo anomalies, and latent residual anomalies (i.e., samples that have unusual residuals compared to the normal data in a latent space), with the last two abnormalities designed to detect unseen anomalies. Extensive experiments on nine real-world anomaly detection datasets show superior performance of our model in detecting seen and unseen anomalies under diverse settings. Code and data are available at: https://github.com/choubo/DRA
Thin-Plate Spline Motion Model for Image Animation

Image animation brings life to the static object in the source image according to the driving video. Recent works attempt to perform motion transfer on arbitrary objects through unsupervised methods without using a priori knowledge. However, it remains a significant challenge for current unsupervised methods when there is a large pose gap between the objects in the source and driving images. In this paper, a new end-to-end unsupervised motion transfer framework is proposed to overcome such issues. Firstly, we propose thin-plate spline motion estimation to produce a more flexible optical flow, which warps the feature maps of the source image to the feature domain of the driving image. Secondly, in order to restore the missing regions more realistically, we leverage multi-resolution occlusion masks to achieve more effective feature fusion. Finally, additional auxiliary loss functions are designed to ensure that there is a clear division of labor in the network modules, encouraging the network to generate high-quality images. Our method <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Our source code is publicly available: https://github.com/yoyo-nb/Thin-Plate-Spline-Motion-Model. can animate a variety of objects, including talking faces, human bodies, and pixel animations. Experiments demonstrate that our method performs better on most benchmarks than the state of the art with visible improvements in motion-related metrics.
$\beta$-DARTS: Beta-Decay Regularization for Differentiable Architecture Search

Neural Architecture Search (NAS) has attracted increasingly more attention in recent years because of its capability to design deep neural network automatically. Among them, differential NAS approaches such as DARTS, have gained popularity for the search efficiency. However, they suffer from two main issues, the weak robustness to the performance collapse and the poor generalization ability of the searched architectures. To solve these two problems, a simple-but-efficient regularization method, termed as Beta-Decay, is proposed to regularize the DARTS-based NAS searching process. Specifically, Beta-Decay regularization can impose constraints to keep the value and variance of activated architecture parameters from too large. Furthermore, we provide in-depth theoretical analysis on how it works and why it works. Experimental results on NAS-Bench-201 show that our proposed method can help to stabilize the searching process and makes the searched network more transferable across different datasets. In addition, our search scheme shows an outstanding property of being less dependent on training time and data. Comprehensive experiments on a variety of search spaces and datasets validate the effectiveness of the proposed method. The code is available at https://github.com/Sunshine-Ye/Beta-DARTS.
AP-BSN: Self-Supervised Denoising for Real-World Images via Asymmetric PD and Blind-Spot Network

Blind-spot network (BSN) and its variants have made significant advances in self-supervised denoising. Never-theless, they are still bound to synthetic noisy inputs due to less practical assumptions like pixel-wise independent noise. Hence, it is challenging to deal with spatially corre-lated real-world noise using self-supervised BSN. Recently, pixel-shuffle downsampling (PD) has been proposed to re-move the spatial correlation of real-world noise. However, it is not trivial to integrate PD and BSN directly, which prevents the fully self-supervised denoising model on real-world images. We propose an Asymmetric PD (AP) to ad-dress this issue, which introduces different P D stride factors for training and inference. We systematically demonstrate that the proposed AP can resolve inherent trade-offs caused by specific PD stride factors and make BSN applicable to practical scenarios. To this end, we develop AP-BSN, a state-of-the-art self-supervised denoising method for real-world sRGB images. We further propose random-replacing refinement, which significantly improves the performance of our AP-BSN without any additional parameters. Extensive studies demonstrate that our method outperforms the other self-supervised and even unpaired denoising methods by a large margin, without using any additional knowledge, e.g., noise level, regarding the underlying unknown noise.
EMOCA: Emotion Driven Monocular Face Capture and Animation

As 3D facial avatars become more widely used for communication, it is critical that they faithfully convey emotion. Unfortunately, the best recent methods that regress parametric 3D face models from monocular images are unable to capture the full spectrum of facial expression, such as subtle or extreme emotions. We find the standard reconstruction metrics used for training (landmark reprojection error, photometric error, and face recognition loss) are insufficient to capture high-fidelity expressions. The result is facial geometries that do not match the emotional content of the input image. We address this with EMOCA (EMOtion Capture and Animation), by introducing a novel deep perceptual emotion consistency loss during training, which helps ensure that the reconstructed 3D expression matches the expression depicted in the input image. While EMOCA achieves 3D reconstruction errors that are on par with the current best methods, it significantly outperforms them in terms of the quality of the reconstructed expression and the perceived emotional content. We also directly regress levels of valence and arousal and classify basic expressions from the estimated 3D face parameters. On the task of in-the-wild emotion recognition, our purely geometric approach is on par with the best image-based methods, highlighting the value of 3D geometry in analyzing human behavior. The model and code are publicly available at https://emoca.is.tue.mpg.de.
MiniViT: Compressing Vision Transformers with Weight Multiplexing

Vision Transformer (ViT) models have recently drawn much attention in computer vision due to their high model capability. However, ViT models suffer from huge number of parameters, restricting their applicability on devices with limited memory. To alleviate this problem, we propose MiniViT, a new compression framework, which achieves parameter reduction in vision transformers while retaining the same performance. The central idea of MiniViT is to multiplex the weights of consecutive transformer blocks. More specifically, we make the weights shared across layers, while imposing a transformation on the weights to increase diversity. Weight distillation over self-attention is also applied to transfer knowledge from large-scale ViT models to weight-multiplexed compact models. Comprehensive experiments demonstrate the efficacy of MiniViT, showing that it can reduce the size of the pre-trained Swin-B transformer by 48%, while achieving an increase of 1.0% in Top-1 accuracy on ImageNet. Moreover, using a single-layer of parameters, MiniViT is able to compress DeiT-B by 9.7 times from 86M to 9M parameters, without seriously compromising the performance. Finally, we verify the transferability of MiniViT by reporting its performance on downstream benchmarks. Code and models are available at here.
CHEX: CHannel EXploration for CNN Model Compression

Channel pruning has been broadly recognized as an effective technique to reduce the computation and memory cost of deep convolutional neural networks. However, conventional pruning methods have limitations in that: they are restricted to pruning process only, and they require a fully pre-trained large model. Such limitations may lead to sub-optimal model quality as well as excessive memory and training cost. In this paper, we propose a novel Channel Exploration methodology, dubbed as CHEX, to rectify these problems. As opposed to pruning-only strategy, we propose to repeatedly prune and regrow the channels throughout the training process, which reduces the risk of pruning important channels prematurely. More exactly: From intra-Layer's aspect, we tackle the channel pruning problem via a well-known column subset selection (CSS) formulation. From inter-Layer's aspect, our regrowing stages open a path for dynamically re-allocating the number of channels across all the layers under a global channel sparsity constraint. In addition, all the exploration process is done in a single training from scratch without the need of a pre-trained large model. Experimental results demonstrate that CHEX can effectively reduce the FLOPs of diverse CNN architectures on a variety of computer vision tasks, including image classification, object detection, instance segmentation, and 3D vision. For example, our compressed ResNet-50 model on ImageNet dataset achieves 76% top-l accuracy with only 25% FLOPs of the original ResNet-50 model, outperforming previous state-of-the-art channel pruning methods. The checkpoints and code are available at here.
Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval

Sketching enables many exciting applications, notably, image retrieval. The fear-to-sketch problem (i.e., “I can't sketch”) has however proven to be fatal for its widespread adoption. This paper tackles this “fear” head on, and for the first time, proposes an auxiliary module for existing retrieval models that predominantly lets the users sketch without having to worry. We first conducted a pilot study that revealed the secret lies in the existence of noisy strokes, but not so much of the “I can't sketch”. We consequently design a stroke subset selector that detects noisy strokes, leaving only those which make a positive contribution towards successful retrieval. Our Reinforcement Learning based formulation quantifies the importance of each stroke present in a given subset, based on the extent to which that stroke contributes to retrieval. When combined with pre-trained retrieval models as a pre-processing module, we achieve a significant gain of 8%-10% over standard baselines and in turn report new state-of-the-art performance. Last but not least, we demonstrate the selector once trained, can also be used in a plug-and-play manner to empower various sketch applications in ways that were not previously possible.
PNP: Robust Learning from Noisy Labels by Probabilistic Noise Prediction

Label noise has been a practical challenge in deep learning due to the strong capability of deep neural networks in fitting all training data. Prior literature primarily resorts to sample selection methods for combating noisy labels. However, these approaches focus on dividing samples by order sorting or threshold selection, inevitably introducing hyperparameters (e.g., selection ratio / threshold) that are hard-to-tune and dataset-dependent. To this end, we propose a simple yet effective approach named PNP (Probabilistic Noise Prediction) to explicitly model label noise. Specifically, we simultaneously train two networks, in which one predicts the category label and the other predicts the noise type. By predicting label noise probabilistically, we identify noisy samples and adopt dedicated optimization objectives accordingly. Finally, we establish a joint loss for network update by unifying the classification loss, the auxiliary constraint loss, and the in-distribution consistency loss. Comprehensive experimental results on synthetic and realworld datasets demonstrate the superiority of our proposed method. The source code and models have been made available at https://github.com/NUST-Machine-Intelligence-Laboratory/PNP.
Network Page Building Methodical Reviews Using Involuntary Manuscript Classification Procedures Founded on Deep Learning

The final phase is classifying new documents on the basis of association-based classifier rules (ABCR), which are obtained from the previous phase. The keys to construct the newly document vector is the use of the like HTML parser and string token that are used in the index level. Specific, the parser is used to compute the general text and term-ranked “important” text. The string tokenizer process is used to parse the terms from the document. Then, the “useful” terms are placed into a table to allow their frequencies newly document to be computed. same heuristic for the term “usefulness” is used in the indexing to find the document frequency of each term. Thus, the tf.idf values for each term in the newly document vector can be computed. ABCR are organized in descending order of support. When a new document is classified, the first matching rule in the sequence conducts the prediction, which is the OABCR. This sequence is usually tied by a class. When no rule in the sequence matches the coming document, the class of the document is predicted by scanning other class label rules. If no class rule matches the document attributes, then the document class label becomes null.
From Emotion AI to Cognitive AI

Survey/review study From Emotion AI to Cognitive AI Guoying Zhao *, Yante Li , and Qianru Xu University of Oulu, Pentti Kaiteran Katu 1, Linnanmaa 90570, Finland * Correspondence: guoying.zhao@oulu.fi Received: 22 September 2022 Accepted: 28 November 2022 Published: 22 December 2022 Abstract: Cognitive computing is recognized as the next era of computing. In order to make hardware and software systems more human-like, emotion artificial intelligence (AI) and cognitive AI which simulate human intelligence are the core of real AI. The current boom of sentiment analysis and affective computing in computer science gives rise to the rapid development of emotion AI. However, the research of cognitive AI has just started in the past few years. In this visionary paper, we briefly review the current development in emotion AI, introduce the concept of cognitive AI, and propose the envisioned future of cognitive AI, which intends to let computers think, reason, and make decisions in similar ways that humans do. The important aspect of cognitive AI in terms of engagement, regulation, decision making, and discovery are further discussed. Finally, we propose important directions for constructing future cognitive AI, including data and knowledge mining, multi-modal AI explainability, hybrid AI, and potential ethical challenges.
Semantic-Preserved Communication System for Highly Efficient Speech Transmission

Deep learning (DL) based semantic communication methods have been explored for the efficient transmission of images, text, and speech in recent years. In contrast to traditional wireless communication methods that focus on the transmission of abstract symbols, semantic communication approaches attempt to achieve better transmission efficiency by only sending the semantic-related information of the source data. In this paper, we consider semantic-oriented speech transmission which transmits only the semantic-relevant information over the channel for the speech recognition task, and a compact additional set of semantic-irrelevant information for the speech reconstruction task. We propose a novel end-to-end DL-based transceiver which extracts and encodes the semantic information from the input speech spectrums at the transmitter and outputs the corresponding transcriptions from the decoded semantic information at the receiver. In particular, we employ a soft alignment module and a redundancy removal module to extract only the text-related semantic features while dropping semantically redundant content, greatly reducing the amount of semantic redundancy compared to existing methods. We also propose a semantic correction module to further correct the predicted transcription with semantic knowledge by leveraging a pretrained language model. For the speech to speech transmission, we further include a CTC alignment module that extracts a small number of additional semantic-irrelevant but speech-related information, such as duration, pitch, power and speaker identification of the speech for the better reconstruction of the original speech signals at the receiver. We also introduce a two-stage training scheme which speeds up the training of the proposed DL model. The simulation results confirm that our proposed method outperforms current methods in terms of the accuracy of the predicted text for the speech to text transmission and the quality of the recovered speech signals for the speech to speech transmission, and significantly improves transmission efficiency. More specifically, the proposed method only sends 16% of the amount of the transmitted symbols required by the existing methods while achieving about a 10% reduction in WER for the speech to text transmission. For the speech to speech transmission, it results in an even more remarkable improvement in terms of transmission efficiency with only 0.2% of the amount of the transmitted symbols required by the existing method while preserving the comparable quality of the reconstructed speech signals.
A 100-pixel photon-number-resolving detector unveiling photon statistics

Single-photon detectors are ubiquitous in quantum information science and quantum sensing. They are key enabling technologies for numerous scientific discoveries and fundamental tests of quantum optics. Photon-number-revolving detectors are the ultimate measurement tool of light; however, few detectors so far can provide high-fidelity photon number resolution at few-photon levels. Here we demonstrate an on-chip detector that can resolve up to 100 photons by spatiotemporally multiplexing an array of superconducting nanowires along a single optical waveguide. The unparalleled photon number resolution paired with the high-speed response exclusively allows us to unveil the quantum photon statistics of a true thermal light source at an unprecedented level, which is realized by direct measurement of the higher-order correlation function g(N) (with values of N up to 15), observation of photon-subtraction-induced photon number enhancement and quantum-limited state discrimination against a coherent light source. Our detector provides a viable route towards various important applications, including photonic quantum computation and quantum metrology. Up to 100 photons are resolved by a waveguide-integrated and hybrid spatiotemporal-multiplexed superconducting nanowire single-photon detector array. Using this detector array, photon statistical behaviour for a true thermal and a coherent light source is verified.
Computation at the speed of light: metamaterials for all-optical calculations and neural networks

The explosion in the amount of information that is being processed is prompting the need for new computing systems beyond existing electronic computers. Photonic computing is emerging as an attractive alternative due to performing calculations at the speed of light, the change for massive parallelism, and also extremely low energy consumption. We review the physical implementation of basic optical calculations, such as differentiation and integration, using metamaterials, and introduce the realization of all-optical artificial neural networks. We start with concise introductions of the mathematical principles behind such optical computation methods and present the advantages, current problems that need to be overcome, and the potential future directions in the field. We expect that our review will be useful for both novice and experienced researchers in the field of all-optical computing platforms using metamaterials.
Systematic literature review: The adaptation of distance learning process during the COVID-19 pandemic using virtual educational spaces in metaverse

Virtual education has started to grab interest since technology emerges in everything in our life. Metaverse is one of the technologies that has been conceived since the late 90s and is currently being renewed and renovated to suit current changes. Yet, many teachers and students still do not know how to utilize Metaverse to create a new atmosphere in the learning process. Therefore, this study aims to show the results of using Metaverse in the form of virtual space in the educational field and how teachers and students respond to the process. In this study, the authors would like to conduct a Systematic Literature Review (SLR) on the advantages and disadvantages of virtual educational spaces in Metaverse based on many previous studies related to this topic. Survey results conducted by previous studies show that students mostly enjoy using Metaverse as their learning method and can comprehend several lessons better when using Metaverse than traditional learning- textbook-based and face-to-face learning. Unfortunately, only several studies focus on finding preferred subjects to teach using the Metaverse and for which educational level this method suits best. In general, the authors conclude that Metaverse has excellent potential in the future to be explored profoundly in the education field due to the development of skills in the use of technology and a significant increase in student practice scores. However, the guidance of teachers and parents is still needed so that students can avoid the disadvantages.