Angular Visual Hardness

Although convolutional neural networks ( CNNs ) are inspired by the mechanisms behind human visual systems , they diverge on many measures such as ambiguity or hardness . In this paper , we make a surprising discovery : there exists a ( nearly ) universal score function for CNNs whose correlation is statistically significant than the widely used model confidence with human visual hardness . We term this function as angular visual hardness ( AVH ) which is given by the normalized angular distance between a feature embedding and the classifier weights of the corresponding target category in a CNN . We conduct an in-depth scientific study . We observe that CNN models with the highest accuracy also have the best AVH scores . This agrees with an earlier finding that state-of-art models tend to improve on classification of harder training examples . We find that AVH displays interesting dynamics during training : it quickly reaches a plateau even though the training loss keeps improving . This suggests the need for designing better loss functions that can target harder examples more effectively . Finally , we empirically show significant improvement in performance by using AVH as a measure of hardness in self-training methods for domain adaptation . 1 INTRODUCTION . The invention and development of Convolutional Neural Networks ( CNNs ) were inspired by natural visual processing systems . For example , artificial neurons were designed to mimic neurons taking and transforming information [ 48 ] , and neocognitron , the origin of the CNN architecture , was enlightened by early findings of receptive fields in the visual cortex [ 15 ] . CNNs have achieved a great success in pushing the boundaries in a wide range of computer vision tasks such as image classification [ 24 , 30 , 59 ] , face recognition [ 39 , 63 , 64 ] , and scene analysis [ 19 , 43 , 70 ] . Specifically , on certain large-scale benchmarks such as ImageNet [ 10 ] , CNNs have even surpassed human-level accuracy . Despite such notable progress , CNNs are still far from matching human vision on many measures such as robustness , adaptability and few-shot learning [ 23 ] , and could suffer from various biases . For example , CNNs pre-trained on Imagenet are biased towards textures [ 18 ] . These biases can result in CNNs being overconfident , or prone to domain gaps and adversarial attacks . Therefore , to fundamentally solve the above problems , the efforts should be made to improve CNN ’ s capabilities to model human visual system [ 47 , 62 ] . The popular measure of CNN confidence , softmax score , has been widely used in many applications , yet causing calibration problems and tending to make CNNs overconfident even if they are wrong [ 21 , 34 ] . However , this is not the case with human vision . Thus , there is a gap between the current measure of hard examples that appear to be ambiguous or uncertain in these two systems . We denote human visual hardness as the measure of how hard an example is to human visual system . In this paper , we bridge the gap by proposing a novel score function on CNNs that correlates closely with human visual hardness . The first piece of this puzzle starts with the question of what is a good measure of human visual hardness . Recently , [ 52 ] argued that human selection frequency is a good measure . This is the average number of times an image gets picked by a crowd of annotators , when they are asked to pick an image from a pool that belongs to a certain specified category . Intuitively , human selection frequency depends on various factors like object sizes , poses , special filters applied to images , etc . [ 52 ] collected human selection frequency scores on ImageNet validation set using the MTurk platform . In this paper , we use this dataset to verify several hypotheses on correlations between CNNs and human visual systems in section 3.2 . Moreover , an automatic detection of examples that are hard for human vision has numerous applications . [ 52 ] showed that state-of-the-art models perform better on hard examples ( i.e. , hard for humans ) . This implies that in order to improve generalization , the models need to improve accuracy on hard examples . This can be achieved through various learning algorithms such as curriculum learning [ 2 ] and self-paced learning [ 32 ] where being able to detect hard examples is crucial . Measuring sample confidence is also important in partially-supervised problems such as semi-supervised learning [ 71 , 72 ] , unsupervised domain adaptation [ 8 ] and weakly-supervised learning [ 65 ] due to their under-constrained nature . For instance , self-training [ 73 ] can easily reach to trivial solutions if one does not select pseudo-labels carefully based on correct measure of hardness . Furthermore , by identifying hard examples , one can detect various biases in current CNN models . Sample hardness can also be used to identify implicit distribution imbalance in datasets to ensure fairness and remove societal biases [ 4 ] . Our contributions are summarized as follows : . Angular visual hardness ( AVH ) : Given a CNN , we propose a novel score function that has stronger correlation with human visual hardness than softmax score . It is the normalized angular distance between the image feature embedding and the weights of the target category ( See Figure 1 ) . The normalization takes into account the angular distances to other categories . We argue that the semantic ambiguity that affects human visual hardness has stronger correlation with this score and we find empirical evidence to support this claim . The AVH score is inspired by the observation from Figure 1 and also [ 42 ] that samples from each class concentrate in a convex cone in the embedding space . In addition , some existing theoretical results [ 61 ] show that for minimization of logistic loss or cross- entropy loss which is often used in CNNs , gradient descent converges to the same direction as maximum margin solutions irrelevant to the ` 2 norm of classifier weights or feature embeddings . This also provides the intuition behind why we can show AVH score rather than current model confidence , softmax score , correlates better with human visual hardness . We validate that there is a statistically-significant stronger correlation between AVH and human selection frequency across a wide range of CNN models . Hence , it serves as its proxy on datasets where such information is not available and is beneficial to a number of downstream tasks . We observed the evolution of AVH score during training of CNN models . It plateaus early in training even if the training ( cross-entropy ) loss keeps improving . This suggests the need to design better loss functions that can improve performance on hard examples . We also validate the argument in [ 52 ] that improving on hard examples is the key to improve the generalization by verifying that the state-of-the-art models have the best average AVH scores . Finally , we empirically show the superiority of AVH with its application to self-training for unsupervised domain adaptation . With AVH being an improved confidence measure , our proposed self-training framework renders considerably improved pseudo-label selection and category estimation , leading to state-of-the-art results with significant performance gain over baselines . 2 RELATED WORK . Example hardness measures : Recently , measuring sample hardness for deep learning models has been widely studied with loss value [ 58 ] , relative Euclidean distance [ 56 , 66 ] and gradient norm [ 28 ] . On the other hand , there is a rich history in cognitive and neuroscience communities to understand human visual perception [ 6 , 7 , 14 , 45 ] , where many of them focus on mechanisms used by the human brain to translate visual information to mental representations . These representations are subject to many correspondence differences and errors and thereby are not isomorphic to the real world [ 37 ] . They can be affected by the ambiguity of different semantics [ 27 ] such as occlusion , distortion , motion blur , and inherent similarity among objects . Due to the expensive human labeling process , such detailed semantic information is typically not present in large-scale image benchmarks used to train the CNN models . Angular distance in CNNs : [ 69 ] uses the deep features to quantify the semantic difference between images , indicating that deep features contain the most crucial semantic information . It empirically shows that the angular distance between feature maps in deep neural networks is very consistent with the human in distinguishing the semantic difference . However , because of the different goal mentioned above , they have not studied or shown any strong correlation of human visual hardness and the angular distance on natural images . [ 40 ] proposes a hyperspherical neural network that constrains the parameters of neurons on a unit hypersphere and uses angular similarity to replace the inner product similarity . [ 42 ] decouples the inner product as the norm and the angle and argues that the norm corresponds to intra-class variation , and the angle corresponds to inter-class semantic difference . However , this work does not consider any human factors , while our goal is to bridge the gap between CNNs and human perception . [ 35 , 41 ] propose well-performing regularizations based on angular diversity to improve the network generalization . Image degradation : Because CNNs and humans achieve similar accuracy on a wide range of tasks on benchmark datasets , a number of works have investigated similarities and differences between CNNs and human vision [ 3 , 5 , 9 , 11 , 12 , 29 , 46 , 51 , 67 ] . Since human annotation data is hard to come by , researchers have proposed an alternative measure of visual hardness on images based on image degradation [ 37 ] . It involves adding noise or changing image properties such as contrast , blurriness , and brightness . [ 17 ] employed psychological studies to validate the degradation method as a way to measure human visual hardness . It should be noted that the artificial visual hardness introduced by degradation is a different concept from the natural visual hardness . The hardness based on degradation only reflects the hardness of a single original image with various of transformations , while natural visual hardness based on the ambiguity of human perception across a distribution of natural images . In this paper , we consider both as the surrogates of human visual hardness . Deep model calibration . Confidence calibration is the problem of predicting probability estimates representative of the true correctness likelihood [ 21 ] . It is well-known that the deep neural networks are mis-calibrated and there has been a rich literature trying to solve this problem [ 21 , 31 ] . However , this is a somewhat different issue because the confidence calibration is a problem introduced by two measurements of the model , which does not have any involvement of human visual hardness . 3 A DISCOVERY FROM SCIENTIFIC TESTING : ANGULAR VISUAL HARDNESS . 3.1 NOTATIONS AND SETUP . In order to quantify Human Visual Hardness and Model Predictions for convenience purposes in experiments , we use corresponding surrogates which are formally defined as the following throughout the paper . We use the ImageNet [ 10 ] benchmark in all following experiments . Particularly , we take advantage of the Human Selection Frequency information for validation images provided by the recent paper [ 52 ] . Recall that such information can serve as one of the proxy for Human Visual Hardness . To test if our findings with Human Selection Frequency hold on another proxy , image degradation , we create an augmented validation set based on two image degradation methods , decreasing contrast and adding noise . We label them with corresponding degradation level ( results shown in Appendix A.2 and A.5 ) . Besides , in order to verify that the our experimental results hold consistently across models instead of a particular model , we use four popular ImageNet pre-trained models AlexNet [ 30 ] , VGG19 [ 59 ] , DenseNet121 [ 26 ] , ResNet50 [ 24 ] . We select ResNet50 as the representative model for some experiments . Most importantly , we also provide experimental results on different datasets , MNIST and CIFAR10/100 , in Appendix A.3 and A.4 to better support our proposal . Denote Sn as the unit n-sphere , formally , Sn = { x ∈ Rn+1|‖x‖2 = 1 } . Below by A ( · , · ) , we denote the angular distance between two points on Sn , i.e. , A ( u , v ) = arccos ( 〈u , v〉‖u‖‖v‖ ) . Let x be the feature embeddings input for the layer before the last one of the classifier of the pretrained CNN models , eg . FC2 for VGG19 . Let C be the number of classes for a classification task . Denote W = { wi|0 < i ≤ C } as the set of weights for all C classes in the final layer of the classifier . Definition 1 ( Angular Visual Hardness ( AVH ) ) . AVH , for any ( x , y ) , is defined as : AVH ( x ) = A ( x , wy ) ∑C i=1A ( x , wi ) , which wy represents the weights of the target class . Definition 2 ( Model Confidence ) . We define model confidence on a single sample as the probability score of the true objective class output by the CNN models , formally , e wyx∑C i=1 e wix . Definition 3 ( Human Selection Frequency ) . We define one way to measure human visual hardness on pictures as Human Selection Frequency . Quantitatively , given m number of human workers in a labeling process described in [ 52 ] , if b out of m label a picture as a particular class and that class is the target class of that picture in the final dataset , then Human Selection Frequency is defined as bm .

Summary: This paper defined Angular Visual Hardness (AVH) based on the angle between image feature embedding and the weights of the target class. The authors compared the correlation between AVH and human selection frequency with model confidence and feature norm. The results show that both AVH and model confidence have correlation, but AVH has a stronger correlation than model confidence. Differently from the conjecture of [41], feature norm was not correlated with human selection frequency.