Reviewerly/exHarmony huggingface dataset:
An Empirical study of Data-Free Quantization’s Tuning Robustness

Deep convolutional neural networks are now performing increasingly superior in various fields, while the network parameters are getting massive as the advanced neural networks tend to be deeper. Among various model compression methods, quantization is one of the most potent approaches to compress neural networks by compacting model weights and activations to lower bit-width. The data-free quantization method is also proposed, which is specialized for some privacy and security scenarios and enables quantization without access to real data. In this work, we find that the tuning robustness of existing data-free quantization is flawed, progressing an empirical study and determining some hyperparameter settings that can converge the model stably in the data-free quantization process. Our study aims to evaluate the overall tuning robustness of the current data-free quantization system, which is existing methods are significantly affected by parameter fluctuations in tuning. We also expect data-free quantification methods with tuning robustness to appear in the future.
HiMODE: A Hybrid Monocular Omnidirectional Depth Estimation Model

Monocular omnidirectional depth estimation is receiving considerable research attention due to its broad applications for sensing 360° surroundings. Existing approaches in this field suffer from limitations in recovering small object details and data lost during the ground-truth depth map acquisition. In this paper, a novel monocular omnidirectional depth estimation model, namely HiMODE is proposed based on a hybrid CNN+Transformer (encoder-decoder) architecture whose modules are efficiently designed to mitigate distortion and computational cost, without performance degradation. Firstly, we design a feature pyramid network based on the HNet block to extract high-resolution features near the edges. The performance is further improved, benefiting from a self and cross attention layer and spatial/temporal patches in the Transformer encoder and decoder, respectively. Besides, a spatial residual block is employed to reduce the number of parameters. By jointly passing the deep features extracted from an input image at each backbone block, along with the raw depth maps predicted by the transformer encoder-decoder, through a context adjustment layer, our model can produce resulting depth maps with better visual quality than the ground-truth. Comprehensive ablation studies demonstrate the significance of each individual module. Extensive experiments conducted on three datasets; Stanford3D, Matterport3D, and SunCG, demonstrate that HiMODE can achieve state-of-the-art performance for 360° monocular depth estimation. Complete project code and supplementary materials are available at https://github.com/himode5008/HiMODE.
Faster, Lighter, Robuster: A Weakly-Supervised Crowd Analysis Enhancement Network and A Generic Feature Extraction Framework

With bounding box labels needed for training, object detection is viewed unfavorably in terms of crowd analysis, due to the intensive labor for labeling and the unsatisfactory performance in clutters and severe occlusions. Another feasible method, density-based regression, despite its proficiency in counting and only point-level labels used for training, cannot get the location of each person, and the time and space consumption is relatively high. In this paper, we propose a generic feature extraction framework, Adaptive Pyramid Score (APS), based on object detection and designed specifically for extracting quantitative and spatial-semantic features. Moreover, as an intuitive and feasible solution regarding crowd analysis, we propose the weakly-supervised Confidence-Threshold-Foresight Network (CTFNet) under our APS feature extraction framework, which only needs count-level labels for training and improves the performance of various methods dramatically. Our system realizes the triple enhancement of counting, localization, and detection, which is also proved to be faster than advanced crowd analysis methods, lighter to be transplanted to various object detection methods, and robuster to tackle tasks of extreme scenes. Furthermore, the weakly-supervised paradigm leverage the intensive labor for labeling profoundly.
DNAS:A Decoupled Global Neural Architecture Search Method

Neural Architecture Search (NAS) can automatically design model architecture with better performance. Current researchers have searched for local architecture similar to block, then stacked to construct entire models, or searched the entire model based on a manually designed benchmark module. There is no method to directly search the architecture of the global(entire) model at the operation level. The purpose of this article is to search the entire model directly in the operation level search space. We analyzed the search space of past methods which searching for local architectures, then a working mode for global model architecture search named CAM is proposed. Proposed CAM decouples the architectural parameters of the entire model which can complete the entire model architecture search with few architecture parameters. In the experiment, the test error 2.68 % in CIFAR-10 is obtained by the proposed method at the global architecture level, which can compare with the stage-of-art local architecture search methods.
Does Interference Exist When Training a Once-For-All Network?

The Once-For-All (OFA) method offers an excellent pathway to deploy a trained neural network model into multiple target platforms by utilising the supernet-subnet architecture. Once trained, a subnet can be derived from the supernet (both architecture and trained weights) and deployed directly to the target platform with little to no retraining or fine-tuning. To train the subnet population, OFA uses a novel training method called Progressive Shrinking (PS) which is designed to limit the negative impact of interference during training. It is believed that higher interference during training results in lower subnet population accuracies. In this work we take a second look at this interference effect. Surprisingly, we find that interference mitigation strategies do not have a large impact on the overall subnet population performance. Instead, we find the subnet architecture selection bias during training to be a more important aspect. To show this, we propose a simple-yet-effective method called Random Subnet Sampling (RSS), which does not have mitigation on the interference effect. Despite no mitigation, RSS is able to produce a better performing subnet population than PS in four small-to-medium-sized datasets; suggesting that the interference effect does not play a pivotal role in these datasets. Due to its simplicity, RSS provides a 1.9× reduction in training times compared to PS. A 6.1× reduction can also be achieved with a reasonable drop in performance when the number of RSS training epochs are reduced. Code available at https://github.com/Jordan-HS/RSS-Interference-CVPRW2022
Rank in Style: A Ranking-based Approach to Find Interpretable Directions

Recent work such as StyleCLIP aims to harness the power of CLIP embeddings for controlled manipulations. Although these models are capable of manipulating images based on a text prompt, the success of the manipulation often depends on careful selection of the appropriate text for the desired manipulation. This limitation makes it particularly difficult to perform text-based manipulations in do-mains where the user lacks expertise, such as fashion. To address this problem, we propose a method for automatically determining the most successful and relevant text-based edits using a pre-trained StyleGAN model. Our approach consists of a novel mechanism that uses CLIP to guide beam-search decoding, and a ranking method that identifies the most relevant and successful edits based on a list of keywords. We also demonstrate the capabilities of our framework in several domains, including fashion.
A3D: Studying Pretrained Representations with Programmable Datasets

Rendered images have been used to debug models, study inductive biases, and understand transfer learning. To scale up rendered datasets, we construct a pipeline with 40 classes of images including furniture and consumer products, backed by 48,716 distinct object models, 480 environments, and 563 materials. We can easily vary dataset diversity along four axes—object diversity, environment, material, and camera angle, making the dataset "programmable". Using this ability, we systematically study how these axes of data characteristics influence pretrained representations. We generate 21 datasets by reducing diversity along different axes, and study performance on five downstream tasks. We find that reducing environment has the biggest impact on performance and is harder to recover after fine-tuning. We corroborate this by visualizing the models’ representations, findings that models trained on diverse environments learn more visually meaningful features.
Deep Image Retrieval is not Robust to Label Noise

Large-scale datasets are essential for the success of deep learning in image retrieval. However, manual assessment errors and semi-supervised annotation techniques can lead to label noise even in popular datasets. As previous works primarily studied annotation quality in image classification tasks, it is still unclear how label noise affects deep learning approaches to image retrieval. In this work, we show that image retrieval methods are less robust to label noise than image classification ones. Furthermore, we, for the first time, investigate different types of label noise specific to image retrieval tasks and study their effect on model performance.
ActAR: Actor-Driven Pose Embeddings for Video Action Recognition

Human action recognition (HAR) in videos is one of the core tasks of video understanding. Based on video sequences, the goal is to recognize actions performed by humans. While HAR has received much attention in the visible spectrum, action recognition in infrared videos is little studied. Accurate recognition of human actions in the infrared domain is a highly challenging task because of the redundant and indistinguishable texture features present in the sequence. Furthermore, in some cases, challenges arise from the irrelevant information induced by the presence of multiple active persons not contributing to the actual action of interest. Therefore, most existing methods consider a standard paradigm that does not take into account these challenges, which is in some part due to the ambiguous definition of the recognition task in some cases. In this paper, we propose a new method that simultaneously learns to recognize efficiently human actions in the infrared spectrum, while automatically identifying the key-actors performing the action without using any prior knowledge or explicit annotations. Our method is composed of three stages. In the first stage, optical flow-based key-actor identification is performed. Then for each key-actor, we estimate key-poses that will guide the frame selection process. A scale-invariant encoding process along with embedded pose filtering are performed in order to enhance the quality of action representations. Experimental results on InfAR dataset show that our proposed model achieves promising recognition performance and learns useful action representations.
Bridging the Gap Between Automated and Human Facial Emotion Perception

Understanding the complex relationship between emotions and facial expressions is important for both psychologists and computer scientists. A large body of research in psychology investigates facial expressions, emotions, and how emotions are perceived from facial expressions. As computer scientists look to incorporate this research into automatic emotion perception systems, it is important to understand the nature and limitations of human emotion perception. These principles of emotion science affect the way datasets are created, methods are implemented, and results are interpreted in automated emotion perception. This paper aims to distill and align prior work in automated and human facial emotion perception to facilitate future discussions and research at the intersection of the two disciplines.
Self-Supervised Learning of Pose-Informed Latents

Siamese network architectures trained for self-supervised instance recognition can learn powerful visual representations that are useful in various tasks. Many such approaches maximize the similarity between representations of augmented images of the same object. In this paper, we depart from traditional self-supervised learning benchmarks by defining a novel methodology for new challenging tasks such as zero shot pose estimation. Our goal is to show that common Siamese networks can effectively be trained on frame pairs from video sequences to generate pose-informed representations. Unlike parallel efforts that focus on introducing new image-space operators for data augmentation, we argue that extending the augmentation strategy by using different frames of a video leads to more powerful representations. To show the effectiveness of this approach, we use the Objectron and UCF101 datasets to learn representations and evaluate them on pose estimation, action recognition, and object re-identification. Furthermore, we carefully validate our method against a number of baselines.
Momentum Contrastive Pruning

Momentum contrast [16] (MoCo) for unsupervised visual representation learning has a close performance to supervised learning, but it sometimes possesses excess parameters. Extracting a subnetwork from an over-parameterized unsupervised network without sacrificing performance is of particular interest to accelerate inference speed. Typical pruning methods are not applicable for MoCo, because in the fine-tune stage after pruning, the slow update of the momentum encoder will undermine the pretrained encoder. In this paper, we propose a Momentum Contrastive Pruning (MCP) method, which prunes the momentum encoder instead to obtain a momentum subnet. It maintains an un-pruned momentum encoder as a smooth transition scheme to alleviate the representation gap between the encoder and momentum subnet. To fulfill the sparsity requirements of the encoder, alternating direction method of multipliers [40] (ADMM) is adopted. Experiments prove that our MCP method can obtain a momentum subnet that has almost equal performance as the over-parameterized MoCo when transferred to downstream tasks, meanwhile has much less parameters and float operations per second (FLOPs).
Methods of Reducing Etching Defects in Back End of Line for Semiconductor in 28NM Technology

As technology keeps shrinking to 28nm and below, Plasma etching is a key method to define high-resolution patterns in integrated chip manufacturing. To create structures in a chip, a pattern is formed in a photoresist by lithography and then the pattern is transferred to the device materials by plasma etching. In this paper, the defect adders in hard mask post dry etching is identified, and the removal characteristics of these defects in the back-end etching process are studied. Here we describe three ways to reduce the etching defects. Firstly, Argon purge and chamber pump down are adopted after process etching to remove wafer edge particle. Secondly, Performing reverse bias voltage discharge on the chuck of the wafer, the purse is to inhibit charge adsorption and prevent etching defects from adhering to the surface. Thirdly, the addition of scrubber after hard mask process also helps to remove large-size defects, with a removal rate of about 60%, which is conducive to reducing the impact on yield loss.
PaintInStyle: One-Shot Discovery of Interpretable Directions by Painting

The search for interpretable directions in latent spaces of pre-trained Generative Adversarial Networks (GANs) has become a topic of interest. These directions can be utilized to perform semantic manipulations on the GAN generated images. The discovery of such directions is performed either in a supervised way, which requires manual annotation or pre-trained classifiers, or in an unsupervised way, which requires the user to interpret what these directions represent. In this work, we propose a framework that finds a specific manipulation direction using only a single simple sketch drawn on an image. Our method finds directions consisting of channels in the style space of the StyleGAN2 architecture responsible for the desired edits and performs image manipulations comparable with state-of-the-art methods.
Few-Shot Image Classification Along Sparse Graphs

Few-shot learning remains a challenging problem, with unsatisfactory 1-shot accuracies for most real-world data. Here, we present a new perspective for data distributions in the feature space of a deep network and show how to exploit this perspective for few-shot learning. First, we observe that nearest neighbors in the feature space are with high probability members of the same class while generally two random points from one class are not much closer to each other than two points between classes. This observation suggests that classes in feature space form sparse, loosely connected graphs instead of dense clusters. To exploit this property, we propose using label propagation to the nearest unlabeled data and then using a kernel PCA reconstruction error as decision boundary in feature-space for the data distribution of each class. Using this method, which we call "K-Prop," we demonstrate largely improved few-shot learning performances (e.g., 83% accuracy for 1-shot 5-way classification on the RESISC45 satellite-images dataset) for datasets for which a backbone network can be trained to produce high within-class nearest-neighbor probabilities. We demonstrate this relationship using six different datasets.
Mitigating Paucity of Data in Sinusoid Characterization Using Generative Synthetic Noise

Although the remarkable breakthrough offered by Deep Learning (DL) models in numerous computer vision tasks, the need to acquire large amounts of high-quality natural data and fine-grained annotations is a shortcoming that fundamentally increases the cost and time devoted to training these models in real-world applications. Hence, synthetic datasets are considered reliable alternatives that can reduce the data acquisition by replacing or merging with natural data or effective pre-training of the models. To this end, in this work, we propose a novel approach to integrate structural data structures with the synthetic noise structures learned by unsupervised models that mimic the noise structures in natural data. Based on the proposed approach, we introduce the Sinusoid Feature Recognition (SFR) dataset, which contains hard-to-detect fixed-period sinusoid waves. While the previous works in this regard use generative models to sample synthetic data to inflate the training set, we instead apply unsupervised learning models to generate deep synthetic noise which makes training models in the proposed dataset more challenging. We evaluate the segmentation, image reconstruction, and sinusoid characterization models pre-trained or fully trained on the synthetic SFR dataset on a private dataset of grayscale Acoustic Tele-Viewer (ATV) images. Experimental results show that supervision on our proposed synthetic dataset can improve the accuracy of the models by 3-4% via pre-training, and by 17-27% via ad-hoc training while dealing with challenging, realistic real-world images.
The Role of Shape for Domain Generalization on Sparsely-Textured Images

State-of-the-art object recognition methods do not generalize well to unseen domains. Work in domain generalization has attempted to bridge domains by increasing feature compatibility, but has focused on standard, appearance-based representations. We show the potential of shape-based representations to increase domain robustness. We compare two types of shape-based representations: one trains a convolutional network over edge features, and another computes a soft, dense medial axis transform. We show the complementary strengths of these representations for different types of domains, and the effect of the amount of texture that is preserved. We show that our shape-based techniques better leverage data augmentations for domain generalization, and are more effective at texture bias mitigation than shape-inducing augmentations. Finally, we show that when the convolutional network in state-of-the-art domain generalization methods is replaced with one that explicitly captures shape, we obtain improved results.
A Soft-ranked Index Fusion Framework with Saliency Weighting for Image Quality Assessment

The compression technique is widely adopted for efficient data storage and transmission. Accurate image quality assessment (IQA) measures are urgently desired to evaluate the compression performance. To obtain a more robust evaluation, we propose a soft-ranked index fusion framework for the perceptual preference prediction task, with a combination of different quality measures. The derived soft-ranked indices are fully leveraged to provide the strong discriminability of ranking information. Furthermore, a saliency weighting approach is utilized to investigate the impact of visual attention on our framework. Experimental results indicate that our method achieves a promising prediction accuracy compared with the state-of-the-art quality measures.
Transformer Based Multimodal Scene Recognition in Soccer Videos

This paper presents a transformer-based multimodal soccer scene recognition method for both visual and audio modalities. Our approach directly uses the original video frames and audio spectrogram from the soccer video as the input of the transformer model, which can capture the spatial information of the action at a moment and the contextual temporal information between different actions in the soccer videos. We fuse both video frames and audio spectrogram information output from the transformer model in order to better identify scenes that occur in real soccer matches. The late fusion performs a weighted average of visual and audio estimation results to obtain complete information of a soccer scene. We evaluate the proposed method on SoccerNet-V2 dataset and confirm that our method achieves the best performance compared with the recent and state-of-the-art methods.
Incremental Text Clustering Algorithm For Cloud-Based Data Management In Scientific Research Papers

This study aims to build clusters of similar research papers. Text clustering for research articles is challenging because re-clustering is necessary to handle newly added papers. An incremental clustering algorithm is presented to find similar research papers for COVID-19 related literature. The proposed approach uses an incremental word embedding generation technique to extract feature vectors of the papers. The initial clustering is done by using the K-means algorithm by two NLP feature extraction models; TF-IDF and Word2vec. The clustering results show that the Word2vec outperforms the TF-IDF model. With increasing COVID-19 literature continuously, the ultimate focus is to add the newly published papers to the existing clusters without re-clustering. Title, abstract, and full body of papers are considered for testing the proposed incremental algorithm. Clustering quality is evaluated by the Microsoft language similarity package, which shows clustering of the full-text body outperforms the abstract and title of papers.
Smileverse : A VR Experience

SmileVerse is a continuation of our previous work, Smiling Buddha [1], which aims to achieve emotional contagion. In the interactive installation of Smiling Buddha, we have designed a natural interactive process to let the smile that can be contagious. Based on the innovative VR/AI technologies, we upgraded our previous artwork to build a virtual universe and use facial trackers to detect the user’s expressions and let the virtual characters respond interactively to achieve smiles that can be contagious.