Reviewerly/exHarmony huggingface dataset:
Neural Architecture Search with Representation Mutual Information

Performance evaluation strategy is one of the most important factors that determine the effectiveness and efficiency in Neural Architecture Search (NAS). Existing strategies, such as employing standard training or performance predictor, often suffer from high computational complexity and low generality. To address this issue, we propose to rank architectures by Representation Mutual Information (RMI). Specifically, given an arbitrary architecture that has decent accuracy, architectures that have high RMI with it always yield good accuracies. As an accurate performance indicator to facilitate NAS, RMI not only generalizes well to different search spaces, but is also efficient enough to evaluate architectures using only one batch of data. Building upon RMI, we further propose a new search algorithm termed RMI-NAS, facilitating with a theorem to guarantee the global optimal of the searched architecture. In particular, RMI-NAS first randomly samples architectures from the search space, which are then effectively classified as positive or negative samples by RMI. We then use these samples to train a random forest to explore new regions, while keeping track of the distribution of positive architectures. When the sample size is sufficient, the architecture with the largest probability from the aforementioned distribution is selected, which is theoretically proved to be the optimal solution. The architectures searched by our method achieve remarkable top-1 accuracies with the magnitude times faster search process. Besides, RMI-NAS also generalizes to different datasets and search spaces. Our code has been made available at https://git.openi.org.cn/PCL_AutoML/XNAS.
Improving Robustness Against Stealthy Weight Bit-Flip Attacks by Output Code Matching

Deep neural networks (DNNs) have been shown to be vulnerable against adversarial weight bit-flip attacks through hardware-induced fault-injection methods on the memory systems where network parameters are stored. Recent attacks pose the further concerning threat of finding minimal targeted and stealthy weight bit-flips that preserve expected behavior for untargeted test samples. This renders the attack undetectable from a DNN operation perspective. We propose a DNN defense mechanism to improve robustness in such realistic stealthy weight bit-flip attack scenarios. Our output code matching networks use an output coding scheme where the usual one-hot encoding of classes is replaced by partially overlapping bit strings. We show that this encoding significantly reduces attack stealthiness. Importantly, our approach is compatible with existing defenses and DNN architectures. It can be efficiently implemented on pre-trained models by simply re-defining the output classification layer and finetuning. Experimental benchmark evaluations show that output code matching is superior to existing regularized weight quantization based defenses, and an effective defense against stealthy weight bit-flip attacks.
Fast support vector classifier with generalization-memorization kernel

Recently, Vladimir Vapnik et al. proposed a generalization-memorization kernel for SVC, which significantly improves the memorization and generalization performance of SVC. However, it requires more memory on large data sets. In this paper, we propose a fast support vector classifier with generalization-memorization kernel. It uses class prototypes instead of all samples and an efficient closed-form solution without iterations. Thus, it greatly reduces the learning time and memory usage. The effectiveness and fast learning speed of our method are confirmed in experiments compared with other methods.
Mining the Potential Relationships Between Cancer Cases and Industrial Pollution Based on High-Influence Ordered-Pair Patterns

Co-location pattern mining aims to discover the relationships between spatial features. Traditional co-location patterns are based on clique relationships and only consider the prevalence of patterns. However, pollution sources and cancer cases do not satisfy the clique relationship, and users focus on the influence of pollution sources on cancer cases. Therefore, we propose high-influence ordered-pair patterns to study their relationships. First, we measure the influence of pollution sources on cancer cases. Then, to efficiently mine high-influence ordered-pair patterns, we propose a basic algorithm with two pruning strategies and an optimizing algorithm based on participating instances. Extensive experiments on real and synthetic datasets show that our mining results are more reasonable than existing algorithms and can provide guidance for cancer prevention. Moreover, our algorithm is also highly efficient and scalable.
ViewBirdiformer: Learning to Recover Ground-Plane Crowd Trajectories and Ego-Motion From a Single Ego-Centric View

We introduce a novel learning-based method for view birdification [1], the task of recovering ground-plane trajectories of pedestrians of a crowd and their observer in the same crowd just from the observed ego-centric video. View birdification becomes essential for mobile robot navigation and localization in dense crowds where the static background is hard to see and reliably track. It is challenging mainly for two reasons; i) absolute trajectories of pedestrians are entangled with the movement of the observer which needs to be decoupled from their observed relative movements in the ego-centric video, and ii) a crowd motion model describing the pedestrian movement interactions is specific to the scene yet unknown a priori. For this, we introduce a Transformer-based network referred to as ViewBirdiformer which implicitly models the crowd motion through self-attention and decomposes relative 2D movement observations onto the ground-plane trajectories of the crowd and the camera through cross-attention between views. Most important, ViewBirdiformer achieves view birdification in a single forward pass which opens the door to accurate real-time, always-on situational awareness. Extensive experimental results demonstrate that ViewBirdiformer achieves accuracy similar to or better than state-of-the-art with three orders of magnitude reduction in execution time.
Advancements in deep structured learning based medical image interpretation

Deep learning is a fast flattering highest development in the machine learning technique leading to augment performance in diverse medical applications. The tributary artificial intelligence is rapidly becoming metamorphic in the domain of Healthcare Technology rendering facility to analyze the medical data with high rapidity and exactness. Due to the overwhelming development of artificial intelligence (AI), the contemporary encroachments of deep learning employing advanced and futuristic deep learning methods has become a vivacious research domain for processing medical data in the health care industries. This paper presents a complete analysis about the deep structured learning based methods used in medical image analysis and also elucidate about the advancement of convolutional neural network which is a major class of deep neural networks used in medical image processing. It also gives insight about the fundamentals of deep learning methods and reviews in image interpretation. It also explains about the implementation of an integrated framework for medical image synthesis combining Speckle GAN and Di-Conv-AE-Net.
Adaptive Spatial-BCE Loss for Weakly Supervised Semantic Segmentation

For Weakly-Supervised Semantic Segmentation (WSSS) with image-level annotation, mostly relies on the classification network to generate initial segmentation pseudo-labels. However, the optimization target of classification networks usually neglects the discrimination between different pixels, like insignificant foreground and background regions. In this paper, we propose an adaptive Spatial Binary Cross-Entropy (Spatial-BCE) Loss for WSSS, which aims to enhance the discrimination between pixels. In Spatial-BCE Loss, we calculate the loss independently for each pixel, and heuristically assign the optimization directions for foreground and background pixels separately. An auxiliary self-supervised task is also proposed to guarantee the Spatial-BCE Loss working as envisaged. Meanwhile, to enhance the network's generalization for different data distributions, we design an alternate training strategy to adaptively generate thresholds to divide the foreground and background. Benefiting from high-quality initial pseudo-labels by Spatial-BCE Loss, our method also reduce the reliance on post-processing, thereby simplifying the pipeline of WSSS. Our method is validated on the PASCAL VOC 2012 and COCO 2014 datasets, and achieves the new state-of-the-arts. Code is available at https://github.com/allenwu97/Spatial-BCE .
Unsupervised and Semi-supervised Bias Benchmarking in Face Recognition

We introduce Semi-supervised Performance Evaluation for Face Recognition (SPE-FR). SPE-FR is a statistical method for evaluating the performance and algorithmic bias of face verification systems when identity labels are unavailable or incomplete. The method is based on parametric Bayesian modeling of the face embedding similarity scores. SPE-FR produces point estimates, performance curves, and confidence bands that reflect uncertainty in the estimation procedure. Focusing on the unsupervised setting wherein no identity labels are available, we validate our method through experiments on a wide range of face embedding models and two publicly available evaluation datasets. Experiments show that SPE-FR can accurately assess performance on data with no identity labels, and confidently reveal demographic biases in system performance.
Spatial Domain-Based Robust Watermarking Framework for Cultural Images

Heritage multimedia, which include photographs, customs, knowledge, arts, rituals, audio, cultural information, and music, are valuable artifacts of any region. The most important attribute of heritage media is the transmission of important features of past generations, which reflect their way of living, innovative attitude, and diversity in archaeological and historical perspectives. However, the proliferation of the Internet has made such data exchange more challenging than ever, allowing unauthorized users to easily access such information. Under such circumstances, securing cultural heritage (CH) media is essential. In that regard, herein, we present a spatial domain-based blind and robust watermarking scheme for the ownership verification of colored CH images; this scheme uses DC coefficient modification. In this scheme, the “Y” element of the YCbCr space is used for inserting a watermark. The “Y” element of a host image is divided into non-overlapping blocks with sizes of 8 × 8. Each 8 × 8 block is then divided into two 4 × 8 subblocks. Instead of calculating the DC coefficients using the discrete cosine transform, we independently calculate the DC coefficient of every 4 × 8 subblock in the spatial domain. We test our method based on standard test images obtained from the USC-SIPI dataset and a self-created dataset of cultural images. Our scheme demonstrates improved robustness and lower computational complexity than frequency-domain-based techniques. The average peak signal-to-noise ratio of the proposed technique for test images is 40.0830 dB, and the structural similarity index matrix value is closer to one under no attack, ensuring the imperceptibility of the technique. Further, we prove the resilience of the proposed algorithm by comparing it with various state-of-the-art techniques.
Learning Algebraic Representation for Systematic Generalization in Abstract Reasoning

Is intelligence realized by connectionist or classicist? While connectionist approaches have achieved superhuman performance, there has been growing evidence that such task-specific superiority is particularly fragile in systematic generalization. This observation lies in the central debate between connectionist and classicist, wherein the latter continually advocates an algebraic treatment in cognitive architectures. In this work, we follow the classicist's call and propose a hybrid approach to improve systematic generalization in reasoning. Specifically, we showcase a prototype with algebraic representation for the abstract spatial-temporal reasoning task of Raven's Progressive Matrices (RPM) and present the ALgebra-Aware Neuro-Semi-Symbolic (ALANS) learner. The ALANS learner is motivated by abstract algebra and the representation theory. It consists of a neural visual perception frontend and an algebraic abstract reasoning backend: the frontend summarizes the visual information from object-based representation, while the backend transforms it into an algebraic structure and induces the hidden operator on the fly. The induced operator is later executed to predict the answer's representation, and the choice most similar to the prediction is selected as the solution. Extensive experiments show that by incorporating an algebraic treatment, the ALANS learner outperforms various pure connectionist models in domains requiring systematic generalization. We further show the generative nature of the learned algebraic representation; it can be decoded by isomorphism to generate an answer.
Active Instance Selection for Few-Shot Classification

Few-shot learning aims to develop well-trained models by using only a few annotated samples. However, the performance of few-shot learning deteriorates if inappropriate support samples are selected. This happens because of its large dependency on only a few support samples, which limits the stable upgrade of performance even when additional training samples are provided. To overcome this issue, we introduce an active instance selection mechanism where effective support instances are chosen from the unlabeled pool to improve the performance of few-shot classification while maintaining the same annotation cost. Our mechanism comprises four processing steps: feature pre-processing, feature distribution fitting, instance scoring, and instance selection. First, the feature pre-processing normalizes the scattered features to obtain a well-defined distribution stably. Then, the feature distribution fitting finds the distributional relationship between the support images and the unlabeled query samples. Finally, in instance scoring and instance selection, we select the instances based on the instance-wise score estimated by using the fitted distributions. To verify the validity of the proposed mechanism, we employ various algorithms for the feature distribution fitting and the instance scoring. We perform several experiments by integrating the proposed mechanism into a variety of few-shot classification methods. Furthermore, we validate the effectiveness of our mechanism’s components through extensive ablation studies. The experimental results verify that the proposed algorithm presents stable performance improvement with some additional labeling cost for the few-shot classification task. We also present the optimal methods to select effective samples to be labeled for the target few-shot classification algorithms.
Target-Absent Human Attention

The prediction of human gaze behavior is important for building human-computer interaction systems that can anticipate the user's attention. Computer vision models have been developed to predict the fixations made by people as they search for target objects. But what about when the target is not in the image? Equally important is to know how people search when they cannot find a target, and when they would stop searching. In this paper, we propose a data-driven computational model that addresses the search-termination problem and predicts the scanpath of search fixations made by people searching for targets that do not appear in images. We model visual search as an imitation learning problem and represent the internal knowledge that the viewer acquires through fixations using a novel state representation that we call Foveated Feature Maps (FFMs). FFMs integrate a simulated foveated retina into a pretrained ConvNet that produces an in-network feature pyramid, all with minimal computational overhead. Our method integrates FFMs as the state representation in inverse reinforcement learning. Experimentally, we improve the state of the art in predicting human target-absent search behavior on the COCO-Search18 dataset. Code is available at: https://github.com/cvlab-stonybrook/Target-absent-Human-Attention.
A Unified Framework for Domain Adaptive Pose Estimation

While pose estimation is an important computer vision task, it requires expensive annotation and suffers from domain shift. In this paper, we investigate the problem of domain adaptive 2D pose estimation that transfers knowledge learned on a synthetic source domain to a target domain without supervision. While several domain adaptive pose estimation models have been proposed recently, they are not generic but only focus on either human pose or animal pose estimation, and thus their effectiveness is somewhat limited to specific scenarios. In this work, we propose a unified framework that generalizes well on various domain adaptive pose estimation problems. We propose to align representations using both input-level and output-level cues (pixels and pose labels, respectively), which facilitates the knowledge transfer from the source domain to the unlabeled target domain. Our experiments show that our method achieves state-of-the-art performance under various domain shifts. Our method outperforms existing baselines on human pose estimation by up to 4.5 percent points (pp), hand pose estimation by up to 7.4 pp, and animal pose estimation by up to 4.8 pp for dogs and 3.3 pp for sheep. These results suggest that our method is able to mitigate domain shift on diverse tasks and even unseen domains and objects (e.g., trained on horse and tested on dog). Our code will be publicly available at: https://github.com/VisionLearningGroup/UDA_PoseEstimation .
Prediction of Types in Python with Pre-trained Graph Neural Networks

The application of Graph Neural Networks for pre-training models for source code is not well studied. We experimented with pre-training a Graph Neural Network model for Python on tasks of Name Prediction and Edge Prediction. Then, we used pre-trained weights to initialize a model for variable type prediction. Our preliminary results suggest that pre-training on these tasks brings neither improvements in type prediction performance nor training dynamics. Possible ways to fix this are discussed in the concluding section of the paper. Additionally, we performed an ablation study to see whether type prediction is overreliant on some parts of the graph. Results suggest, that type prediction model does not significantly rely on obvious shortcuts and could be a useful proxy for evaluating pre-trained graph embeddings.
Synthetic to Real Gap Estimation of Autonomous Driving Datasets using Feature Embedding

Recent advances in autonomous driving using deep learning have drawn immense attention from robotics and computer vision communities. Training generalized deep learning models for autonomous driving tasks like visual odometry, segmentation, and object detection requires large amounts of data. Acquiring real-world data with accurate annotations is time-consuming and expensive. Due to this challenge, synthetic datasets are increasingly being used for training and testing deep learning models. Synthetic data lacks the appearance and contextual properties of real-world datasets. Several works have been shown to reduce this gap between synthetic and real-world images. However, evaluating the gap between the synthetic and real-world datasets is a longstanding challenge because of its highly not deterministic nature. This research proposes the use of feature embedding techniques to address this synthetic to reality gap in the form of distance between different data clusters. From the experiments, the proposed approach estimated the distance between real-world to enhanced virtual datasets is 6-10 times the distance between real-world to virtual datasets.
The Blockchain and Homomorphic Encryption Data Sharing Method in Privacy-Preserving Computing

As globalization takes place, multiple data resources need to be integrated, and the degree of information flow between regions determines the degree of integration development between regions. Data sharing is even more sensitive when it comes to national and individual interests and is a matter of national security and national interest. Although Privacy-Preserving Computing achieves privacy protection for input data in the process of multi-party collaborative computing. However, the original data, computing process, and results face verification problems. As a result, sharing data on the blockchain will leak private information. To ensure the security of personal privacy information and recording in monitoring systems, This paper studies the use of homomorphic encryption for the secure transfer of blockchain data. Using our solution, the problem of data leakage in data sharing can be addressed by assuming that the data cannot be tampered with. This method has high transmission accuracy and a short transmission time, which effectively prevents data tampering caused by long transmission times.
Repurposing Knowledge Graph Embeddings for Triple Representation via Weak Supervision

The majority of knowledge graph embedding techniques treat entities and predicates as separate embedding matrices, using aggregation functions to build a representation of the input triple. However, these aggregations are lossy, i.e. they do not capture the semantics of the original triples, such as information contained in the predicates. To combat these shortcomings, current methods learn triple embeddings from scratch without utilizing entity and predicate embeddings from pre-trained models. In this paper, we design a novel fine-tuning approach for learning triple embeddings by creating weak supervision signals from pre-trained knowledge graph embeddings. We develop a method for automatically sampling triples from a knowledge graph and estimating their pairwise similarities from pre-trained embedding models. These pairwise similarity scores are then fed to a Siamese-like neural architecture to fine-tune triple representations. We evaluate the proposed method on two widely studied knowledge graphs and show consistent improvement over other state-of-the-art triple embedding methods on triple classification and triple clustering tasks.
Dual Cost-sensitive Graph Convolutional Network

In graph node classification tasks, traditional graph neural network (GNN) models assume that different types of misclassification have equal loss and thus seek to maximize the posterior probability of sample nodes under labeled classes. However, the graph data in realistic scenarios tend to follow unbalanced long-tail class distributions, making it difficult for GNN to accurately represent the minority class samples because of the overfitting tendency to the majority class features. To address this problem, in this paper we propose a novel GNN model, named Dual Cost-sensitive Graph Convolutional Network (DCSGCN). The DCSGCN is a two-tower model containing two sub-networks that compute the posterior probability and the misclassification cost separately. It uses the cost as complementary information in classification to correct the posterior probability under the minimal risk perspective. Furthermore, we propose a series of new methods to compute node cost labels based on graph topological information and node class distribution. Extensive experiments lead to the observation that DCSGCN outperforms the state-of-the-art model on diverse real-world imbalanced graphs.