Visual Expertise and the Log-Polar Transform Explain Image Inversion Effects
Authors: Martha Gahl
Shubham Kulkarni
Nikhil Pathak
Alex Russell
Garrison W. Cottrell
Venue: Submitted to ICLR 2023 (2023)

Abstract: Visual expertise can be defined as the ability to discriminate among subordinate-level objects in homogeneous classes, such as identities  of faces within the class "face". Despite being able to discriminate many faces, subjects perform poorly at recognizing even familiar faces once inverted. This face-inversion effect is in contrast to subjects’ performance identifying inverted objects for which their experience is at a basic level, which results in less impairment.  Experimental results have suggested that when identifying mono-oriented objects, such as cars, car novices' performance is between that of faces and other objects. We build an anatomically-inspired neurocomputational model to explore this effect. Our model includes a foveated retina and the log-polar mapping from the visual field to V1. This transformation causes changes in scale to appear as horizontal translations, leading to scale equivariance. Rotation is similarly equivariant, leading to vertical translations. When fed into a standard convolutional network, this provides rotation and scale invariance. It may be surprising that a rotation-invariant network shows any inversion effect at all. This is because there is a crucial topological difference between scale and rotation: Rotational invariance is discontinuous, with V1 ranging from 90 degrees (vertically up) to 270 degrees (vertically down). Hence when a face is inverted, the configural information in the face is disrupted while feature information is relatively unaffected. We show that the inversion effect arises as a result of visual expertise, where configural information becomes relevant as more identities are learned at the subordinate level. Our model matches the classic result: faces suffer more from inversion than mono-oriented objects, which are more disrupted than non-mono-oriented objects when objects are only familiar at a basic level.

Under review as a conference paper at ICLR 2023
VISUAL EXPERTISE AND THE LOG-POLAR TRANS- FORM EXPLAIN IMAGE INVERSION EFFECTS
Anonymous authors Paper under double-blind review
ABSTRACT
Visual expertise can be defined the ability to discriminate among subordinate- level objects in homogeneous classes, such as identities of faces within the class "face". Despite being able to discriminate many faces, subjects perform poorly recognizing even familiar faces once inverted. This face-inversion effect is in con- trast to subjects' performance identifying inverted objects for which their experi- ence is at a basic level, which results in less impairment. Experimental results have suggested that when identifying mono-oriented objects, such as cars, car novices' performance is between that of faces and other objects. We build an anatomically- inspired neurocomputational model to explore this effect. Our model includes a foveated retina and the log-polar mapping from the visual field to V1. This trans- formation causes changes in scale to appear as horizontal translations, leading to scale equivariance. Rotation is similarly equivariant, leading to vertical transla- tions. When fed into a standard convolutional network, this provides rotation and scale invariance. It may be surprising that rotation-invariant network shows any inversion effect at all. This is because there is crucial topological difference be- tween scale and rotation: Rotational invariance is discontinuous, with VI ranging from 90% (vertically up) to (vertiiaay down). Hence when face is inverted, the configural information in the face disrupted while feature information is relatively unaffected. We show that the inversion effect arises as result of vi- sual expertise, where configural information becomes relevant as more identities are learned the subordinate level. Our model matches the classic result: faces suffer more from inversion than mono-oriented objects, which are more disrupted than non-mono-orienter objects when objects only familiar at basic level.
INTRODUCTION
Since 1969, researchers have been studying the effects of inverting images (Yin)
1969). Some re-
searchers have focused on defining the bounds of inversion effects: what the measurable effect is for what types of images (Farah et al., 1995| Yin 1969] Jacques et al.] [2007] Rezlescu et al. (017)) active during inver- sion tasks or what level of experience a participant had with the stimuli in the experiment (Gauthier
Others looked explain how inversion effects arise: what part of the brain
al.
2000
Gauthier & Bukach 2007: Gauthier et al., 2014 Kanwisher et al. 1997 1998 Richler
et al.
2011 Wang et al. 2014).
of images during the training phase, and then they were shown pairs of images in testing and asked to select the image that was in the study set. Trials with upright images and trials with inverted images were compared to determine the inversion effect. Using images of faces resulted in a strong and significant inversion effect performance was much mono-oriented category had a lesser, but still
In Yin (1969), participants studied
worse for inverted faces. Images of houses significant effect. Images of airplanes had an insignificant inversion effect. We draw two conclusions from this work: the effects inversion on performance are greater when images of faces are used as the stimuli and insignificant when images of certain objects (e.g., planes, that are less mono-oriented than houses) are used as the stimuli (Yin) 1969). The second conclusion is that not all objects produce the same inversion effects. Mono-oriented objects, which are objects that are typically seen in only one orientation such as the houses in Yin's 1969 work, do show an inversion effect, though is smaller than that of faces (Yin) 1969).

---

Under review as a conference paper at ICLR 2023
Since Yin's 1969 paper on inversion, great deal of research has focused explaining inversion effects. Why is it that different stimuli faces, objects, mono-oriented objects produce different inversion effeets? One explanation of inversion effects, supported by brain imaging and behavioral studies, is that visual expertise changes the way we process visual stimuli. Faces are processed holis- tically, which means that not just the features, but the configuration of the features matters. When such stimuli are inverted, the configuration is disrupted, and we are left with featural processing (Gauthier et al., [0000] 1999 2003). Similar inversion effects have been observed in experts of other domains, such as dog show judges or bird watchers (Diamond & Carey 1986; Gauthier et al. 2000).
Visual expertise is defined with respect to Rosch's basic level categories. In a category hierarchy, the basic level is the level at which objects are most commonly labeled, such as "chair", "tree", or "car"" Basic level categories define broad categories of objects that share properties such as general appearance, function, and common parts (Rosch et al., 1976). For example, cars can look very different from each other, but they all have wheels, an enclosed space for passengers, and are used for ground transportation. Visual expertise is defined as having proficiency in differentiating subordinate-level sub-classes of basic level categories. For example, subordinates of the basic level category "tree" could include "sugar maple", "american elm", "northern red oak".
Most people are face experts in this sense. It has been estimated that we are able to identify on the order of 5,000 different people (Jenkins al. 2018). Identity is subordinate because faces share the same features (eyes, nose, mouth, ears, etc.) in the same general configu- ration. We also process faces holistically (Gauthier & Bukach, 2007). This means that instead of just using the features of face to recognize person, we use the configural information, such as the distance between the eyes, or the distance from the nose the mouth. Hence, expertise fine grained discrimination of homogeneous categories. The research into expertise suggests that experts other domains, such as cars or birds, also use configural information when viewing basic level categories in which the participants are experts (Gauthier et al. 2000).
judgment
We conduct experiments in order ask if there is a way characterize inversion effects in dif- ferent stimuli in terms of levels of expertise. In doing so, we explore the changes in visual signal processing that occur between novice level and expert level. To do this, We build an anatomically inspired network that incorporates foveation high resolution central vision and low resolution pe- ripheral vision and the log polar mapping between the visual field and the primary visual cortex (Polimeni et al. 2006). The log polar mapping causes changes in image scale to appear as horizontal translations. When presented as input to a convolutional neural network (CNN), which is translation invariant, the log polar mapping makes the network relatively scale invariant. Image rotation is sim- ilarly equivariant the two differ topologically: pixels that shift vertically can "fall off" the edge of the image and wrap around to the opposite edge. This causes a rearranging of features in the image, hence a disruption of configural information.
the log-polar representation, because it leads to vertical translations. However,
Using this model, we test the inversion effects of different types of stimuli across increasing expertise in order to gain an insight into how and why visual processing changes based on the visual stimulus. Our model is consistent with the view that expertise plays significant role in the way we process visual inputs, and leads the inversion effects seen in previous work.
2
METHODS
2.1
MODEL
We use ResNet-50 (He et al., 2016) to perform all experiments, trained from scratch with the foveated, log-polar representation. We call this LPnet. We compare our results to a "vanilla" ResNet- 50 with standard images. Unless otherwise noted, all experiments use the Adam optimizer, an initial learning rate of 1e-4, and minibatch size of 48.
2.2 DATA
To test the effects of expertise in visual processing, we use four different datasets. To model experts, the first three datasets are images of faces, cars, and dogs, generally mono-oriented objects, with targets at the subordinate level. To model novices, who mainly know basic-level labels, the fourth
2

---

Under review as a conference paper at ICLR 2023
LL
A
B
D
IC
RD
Figure 1: Example images used for recognition experiments from four datasets: (A) lab sab-gathered face dataset, (B) Comprehensive Cars dataset, (C) ImageNet (dog categories only), and (D) Ima- geNet (dog categories excluded).
dataset contains 128 categories with basic-level targets. 124 are ImageNet categories, containing objects that are seen in variety of orientations in natural scenes. The remaining four are mono- oriented: faces, cars, houses, and dogs. We randomly chose our with 80/10/10 split.
Faces We used a dataset collected by lab members which includes 128 separate identities and ap- proximately 200 example images per identity. The images for each identity portray the person in variety of contexts, with differing backgrounds, lighting conditions, orientations, and facial expres- sions. Example images from the dataset are shown Figure TA
Subordinate level mono-triented objects: Cars Cars are an appropriate choice for mono-oriente objects because they are almost exclusively seen upright in natural scenes. "Car" is also a basic level category with a number of sub-classes. Car expertise is associated with discrimination at the level of model, e.g., 2010 Toyota Camry. The dataset used is the Comprehensive Cars dataset (Yang et al., 2015). These images include cars of variety of makes, models, and years. The dataset contains 136,726 total images across 163 car makes and 1,716 car models. The cars are varying orientations, lighting conditions, and backgrounds. Examples of cars from the Comprehensive Cars dataset are shown in Figure 3
Subordinate level mono-oriented objects: Dogs In the same way that models of cars subor- dinates of the category "car", dog breeds are subordinates of the category "dog". We use 117 dog breeds included in ImageNet (Deng et al. 2009). Like all of ImageNet, these images are highly variable in pose, context, and scale. Because of this we use all available images from ImageNet in these 1 117 categories. Examples of dogs in the dataset are shown in Figure[[..
Basic level categories We use a subset of 124 categories from ImageNet (Deng et al. 2009) which chosen specifically because they are naturally viewed in multiple orientations, such "ladle". "screwdriver"". or or "dumbbell" (Figure [D). We avoided categories such as "clock" or "candle" which, although can be oriented multiple ways, are naturally seen primarily in a limited number of orienta- tions. The labels for the stimuli in this category are at the basic level, instead of at the subordinate level as in the previous two datasets (Rosch al. 1976). Again, due to the within-category variance, we used all examples in ImageNet in order to achieve acceptable performance. We also included categories (bringing the total number of categories to 128) that are mono-oriented. They are: "face", "car", "dog". and "house". Using these mono-oriented objects in experiment with basic level categorization allows us to compare how performance changes between experts and novices as vi- sual expertise general increases. We aggregate data from each of the subordinates of "face", "car", and "dog" to get a varied sample of images for these categories. We matched the number of images approximately to the number of images the ImageNet categories included.

---

Under review as a conference paper at ICLR 2023
Crop 1
Crop 2
0
159
180
0"
15
180
C
Standard
RD
Log Polar
Figure 2: A visualization of the log polar transformation for two crops of the same image. The top row shows standard images, or a depiction of the visual field. The bottom row shows images that have been transformed, using the log polar mapping that approximates the representation of the visual field in VI. For each crop, we show both image types three amounts of rotation: 09, 15% and 180
2.3 DATA TRANSFORMATIONS
Cropping We augment our data by performing random cropping on the images. We take four random crops of each image, with each crop including approximately 65% of the pixels in the image. The crops cannot extend past the edge of the image, so none of the crops include any padding.
Rotation We randomly rotated our training images to be between and 15", because scenes may be viewed with some small amount of rotation from the tilting of the viewers own head. To study the effects of inversion in the network, our validation images are shown at Onand 180"
Foveation For LPNet only, we foveate each crop using the algorithm described in Jiang al. (2015). The foveation leaves the center of the crop (the point of fixation) at high resolution and transforms the periphery to be at lower resolution. The further pixel is from the center the image, the greater the degree of blurring. This mimics the foveation of the retina (Jiang al., 2015).
Log-polar transform For LPNet only, we further preprocessed our images to create an anatomicaaly--rallitiic mapping of the visual field onto the visual cortex. Previous work has shown the validity of using log polar transformations as 2D approximation to the mapping of the visual field onto the visual cortex in primates (Polimeni et al., (006) and in computational models of facial recognition (Anonymous). For each case of rotation, we first rotated the image and then took the log polar transformation of the image. After the images have undergone log polar transformation, the changes in degrees of rotation appear as changes in vertical translation. This is in contrast to the shifts that occur when scaled images undergo a log polar transformation, which are in the horizontal direction. This vertical shift causes pixels to "fall off" the edge of V1. Those pixels wrap around the opposite side of VI. Instead of appearing as simple translation, changes in rotation im- ages that have undergone log polar transformation result in a fundamental rearranging of image components. A visualization of the log polar transform is provided in Figure
All transformations cropping, rotation, and foveation and log- polariizizaion for LPNet are done at the beginning every epoch. For LPNet, the cropping has a large effect, as the fixation point changes with each crop. To see this, note the difference in the two fixations in Figure 2
3 EXPERIMENT I: FACES AND OBJECTS
Based on Yin 1969 we first explore the difference in the effect of inversion based on whether a participant is viewing faces (objects of expertise) or objects for which the subject is novice, only categorized at the basic level. In that work, Yin found that images of faces produced a significant and large inversion effect, while images of airplanes did not produce significant inversion effect.
4

---

Under review as a conference paper at ICLR 2023
3.1 EXPERIMENTAL SETUP
We define expertise in our model as the ability to differentiate between subordinates of basic level categories. When using subordinate level visual stimuli, such as individual face identities, the net- work has to learn to discriminate between very similar visual stimuli. For example, faces share fea- tures and the same overall organization of features. Just as with humans, being able to discriminate very few subordinates demonstrates a low level of expertise with particular basic level category of visual stimuli. Being able to discriminate visually between many subordinates demonstrates high visual expertise with that category. Our network learned to perform categorization tasks using 4, 8, 16, 32, 64, or 128 categories of either faces or non mono-oriented objects.
Over training, we increase the number of classifications the network makes in order to mimic the increase of visual expertise over time, similar to first knowing only family members, then adding family friends, then pre-K, K, ctc. The network is trained for 40 epochs with 4 category outputs (iden- tities for faces, object categories for the "novice" network). After 40 epochs, new categories are added. There is an immediate sharp drop in accuracy, but during the next 40 epochs the network learns the 8 categories. This continues for a total of 240 epochs, across 4, 8, 16, 32, 64, and 128 category outputs. Each phase of training is essentially performing pretraining of the network for the next phase of training by learning features that are helpful for discrimination.
During training, cropped images are rotated randomly between and 15% Testing images are presented in two conditions: upright (09) and inverted (1809). The upright condition provides a baseline so that we can measure the amount the accuracy decreases after inversion. The amount the accuracy drops in the inverted condition is relative to the accuracy of the upright condition, so we report the percent drop in accuracy due to inversion, i.e.,
Acctost = ( (Accup - Accsne) ) AcCup
By looking at the percent drop between the accuracies for the two testing conditions, we can deter- mine the effect inversion had on the network's 's recognition capabilities.
This experiment includes four configurations of networks and data: (1) CNN with face dataset (2) CNN with object dataset (3) LPNet with face dataset and (4) L.NNet with object dataset.
3.2 RESULTS
We ran all experiments five times and averaged the results. Figure 3 shows the accuracy over train- ing- The first row is for standard CNNs and the second row LPett. The effect of the stimuli and the category level of the stimuli, either basic level or subordinate level, is clear. The green line is the training accuracy, the yellow line is the validation accuracy for upright images, and the magenta line is the validation accuracy for inverted images. For the face stimuli, even in early phases of train- ing, there is a performance gap between the upright validation accuracy and the inverted validation accuracy. As the network learns differentiate more identities, this performance gap continues to increase. When using objects as the training stimuli, the difference in performance between the two validation conditions is overall much smaller, with no apparent gap during the first phase of training. In addition, the size of the gap changes less throughout training.
The results on the CNN and LPNet have similar trends in that both show performance gaps between the two validation conditions which increase as the number of identities increases. This gap is larger for the CNN, in part because the inverted condition is more difficult for it. With inverted faces, the CNN fails nearly completely on inverted images of faces with an accuracy of approximately 4%. This is because the log polar transform provides some rotation invariance that standard CNNs do not have. Hence I.Pnet is more representative of human ability on inverted images.
Figure shows the percent accuracy lost as in Eq. D i.e., the accuracy lost relative to the upright accuracy, directly measuring how much inverting an image will disrupt performance in any given experiment. Figure 4 is clear that image inversion has a significant impact with face stimuli, and much smaller impact with object stimuli. This mirrors the first conclusion of Yin 1969. One of the categories included in the object experiment is houses. In order to recreate Yin's 's experiment, we also plot the percent accuracy lost on just the house category. The percent accuracy lost for houses is much lower than that of faces, but slightly higher than objects. This mirrors the second conclusion of Yin 1 1969. Faloo shows how the percent accuracy lost on standard CNN is higher than the
5

---

Under review as a conference paper at ICLR 2023
Face Stimuli
Object Stimuli
1.0
1.0
0.8
0.8
CNN
0.4
0.4
Training
Training
0.2
0.2
Upright Val
Upright Val
Inverted Val
Inverted Val
0.0
0.0
40
240
240
Epochs
Epochs
1.0
1.0
0.8
0.8
LPNet
0.6
0.6
Mmm
Training Upright Val
Training
0.2
0.2
Upright Val
Inverted Val
Inverted Val
0.0
0.0
40
80
120
160
200
240
40
80
120
160
200
240
Epochs
Epochs
Figure 3: Accuracy throughout training on standard CNNs and LPNet for face and object stimuli. The green line is training accuracy, the orange line is accuracy on validation images 0%, and the magenta line is accuracy on validation images rotated 180". Shaded regions are +/-1 standard deviation.
CNN
LPNet
1.0
L.0
Faces
0.8
0.8
Dogs
Houses
0.6
0.6
Ohjects
0.4
Faces
Cars
Dogs
0.2
Houses
Objects
0.0
0.0
0
20
40
100
120
60
80
100
120
Number of Categories
Number Categories
Figure 4: Percent accuracy lost across all phases of training standard CNN and LPNet. Shown for faces, car models, and dog breeds (all expert networks), and houses and objects from the basic- level categorizer.
corresponding values LPnet. This is because there is such significant performance decrease inverted images; the standard CNN losing a larger proportion the upright validation accuracy.
4 EXPERIMENT I: EXPERTISE EFFECTS WITH CARS MODELS AND DOG BREEDS
In the previous experiment we saw that the inversion effect for faces was significant and the inversion effect for objects using network with ability the basic level was not significant. Here explore

---

Under review as a conference paper at ICLR 2023
expertise with mono- oriootrreette objects, cars and dogs, to compare the inversion effect of faces that of mono-oriented objects identified by models with expert level knowledge. We can also determine if the degree of expertise of the network changes the magnitude of inversion effects.
4.1 EXPERIMENT SETUP
To study expertise in mono-oriented objects, we trained two different sets of networks: one to dis- tinguish the sub-class of car models and one to distinguish the sub-class of dog breeds. Being able to differentiate between different car models (e.g. Toyota Camry, Hyundai Santa Fe) is indicative of a car "expert". We again run experiments with both standard CNNs and L.Net.
Aside from the data chosen, this experiment follows the same setup and procedure as the previous experiment. We increase the number of identities being differentiated during each phase training. For car models, our phases of training include 4, 8, 16, 32, 64, and 128 category outputs. The mean number of examples per car model used in our experiments is 151 images. For dogs, our phases of training include 4, 8, 16, 32, 64, and 117 category outputs. This is limited by the number of dog breeds available ImageNet. These images inherently have more variation in pose (dogs sitting, standing, jumping, rolling around, etc.), ImageNet, which averages to approximately 1500 images per dog breed.
used all available images from the dog categories of
4.2 RESULTS
The results for these experiments are shown in Figure 5 with the first row of plots representing results on a standard CNN and the second row representing results on LPNet. For all plots, the validation accuracy gap much smaller at the beginning of training, but continues to increase as more categories are added. Like the previous experiment, it is much more difficult for the CNN to distinguish inverted images than it is for LPNet, because of the rotation invariance provided by LPNet. Looking just at LPNet plots, the and dog upright validation accuracies (Figure 55) and the face upright validation accuracies (Figure B vary widely. In addition, the number of percentage points lost because of inversion in these experiments also varies. This is because of differences in data that are hard to control for, like more complicated backgrounds for cars and dogs, similar contexts for faces, and amount of data available train on. Despite these differences, when looking at the percent accuracy lost in Figure 4 it is clear that the network experienced very similar inversion effect in each expert LPNet network. This is because percent accuracy lost measures how much of the upright validation accuracy was lost due to inversion (how big of an impact inversion had), not the net number of percentage points lost.
Discriminating between different subordinate categories may become harder or easier for the net- work depending on the stimulus class itself or the context of the images. However, the percent accuracy lost shows that image inversion affects LPNet trained as an expert on car models or dogs in almost the same way it affects LPNet trained on faces. This consistent with the Diamond & Carey (186) results.
EXPERIMENT III: RECREATION OF HUMAN EXPERIMENT: COMPLETE COMPOSITE PARADIGM
To further compare network performance more directly to human data, we recreate the complete composite paradigm (Gauthier Bukach 2007, Meinhardt et al. 2014) for LPNet with face data and for a standard CNN with face data. The complete composite paradigm is a way to measure holistic face processing. Subjects view two faces and are asked to attend only to the top half of the face. They are then asked to determine if the top halves are the same or different. This is done in four conditions. In congruent trials the answer for the top and bottom images are the same: both the top and bottom halves are the same or both the top and bottom halves are different. Incongruent trials have the same top half the face with different bottom halves the face or vice versa. Holistic processing is measured as the difference between congruent and incongruent trials. Figure 6A shows a diagram of the complete composite paradigm from (Meinhardt et al.) (2014) and Figure 63 shows example images used in our experiment.
7

---

Under review as a conference paper at ICLR 2023
Car Model Stimuli
Dog Breed Stimuli
1.0
1.0
0.8
0.8
ferree
CNN
0.4
0.4
Training
Training
0.2
0.2
Upright Val
Upright Val
Inverted Val
Inverted Val
0.0
0.0
40
80
240
Epochs
Epochs
1.0
1.0
0.8
0.8
LPNet
Mgr M
0.6
0.6
Training Upright Val
Training
0.2
Upright Val
Inverted Val
Inverted Val
0.0
0.0
40
80
120
160
200
240
40
80
120
160
200
240
Epochs
Epochs
Figure 5: Accuracy throughout training on standard CNNs and LPNet for car model stimuli and dog
breed stimuli. LPNet dog breed plot averaged over three runs.
A
B
Same Trials
Differert Trials
Same
Different
Congruent
A
A
A
Congruent
B
B
B
Incongruent
Incongruent
A
A
A
B
C
B
Figure 6: (A) figure from (Meihhartt al. 2014) showing different combinations of facial com- ponents included in the complete composite paradigm. (B) Images of whole and mismatched faces used recreate the complete composite paradigm our network. Here, "Same" judgments are on the top half of the face.
We compare the output from the last convolutional layer of the network for different images and calculate the cosine similarity for a pair of images. The fixation point for the network on the images is between the eyes (as subjects are instructed to attend to the top potion of the face) and three pixels left of center, which comes from data that the first fixation people make when viewing a face is to the left of center (Hsiao & Cottrell 2008). We perform three trials for each network using three sets of images and average over the trials to get the final cosine similarities. This experiment is done for both upright and inverted images.
8

---

Under review as a conference paper at ICLR 2023
Upright Images
Inverted Images
Network
Trial type
ccongruent incongruent
congruent incongruent
CNN
Same Different
0.3086
0,4061
CNN
0.0913
0.0661
LPNet LPNet
Same
0.2463
0.
Different
0.0848
0.1
Table 1: Difference in cosine similarity between congruent and incongruent trials for CNN and LPNet.
5.1 RESULTS
In Gauthier (2007), the holistic processing effect is calculated the difference between the congru- ent and incongruent trials for or the hit rate. Because we do not have the data to calculate a hit rate, we instead subtract our cosine similarities between congruent and incongruent trials. As seen in Table DI both LPNet and the CNN show holistic processing of upright faces with difference between congruent and incongruent trials being larger than zero. The CNN shows a larger effect, meaning it preforms holistic processing to greater degree. This helps explain why CNNs preform so poorly on inverted images; they rely on the configural information but do not have any rotation invariance. The CNN has the same performance with inverted faces. LPNet, however, sees drastic decrease holistic processing for inverted faces, just as people do [REF].
CONCLUSION
We explored image inversion effects and the impact of visual expertise on performance. By using images of faces, objects, cars, and dogs, we were able show that LPNet, a convolutional neural network that includes a foveated retina and the log polar mapping from the visual field to V1, can reproduce experimental results of image inversion despite being nominally rotation invariant. We showed that there is larger effect on performance from inverting images of faces than images of objects, which increases as the number of categories being discriminated increases. We then explored the result that images of mono-oriented objects have inversion effect between that of faces and objects by showing performance of a house-novice network. Expert networks on mono- oreinted objects do not show effect between that of faces and objects. We showed that LPNet trained to distinguish car models or dog breeds showed similar inversion effects to faces. This suggests that inversion effects are not dependent the stimulus, but rather on the level of visual expertise with the stimulus and on the categorization level of the stimulus (expert or novice viewing subordinate basic level categories).
Image inversion effects are dependent on the level of expertise because of an expert's reliance on configural information to do fine grain discrimination. These effects occur with both CNNs and LPNet. LPNet is more realistic to human vision for the inclusion of the log polar transform, which provides scale and rotation invariance. This is seen with inverted images, when CNNs nearly com- pletely fail to perform differentiation while LPNet is still able to perform some differentiation. We have shown that, when using the log polar transform, the inversion of an image causes a rearrange- ment of features and loss of configural information.
The log polar transform approximates the mapping from the visual field VI, meaning that dis- ruption of configural information occurs before VI before the cortex gets the visual input. When LPnet is not an expert, the loss of configural information has a minimal impact on discrimination ability, and it is able to perform the task with only minor performance changes. As LPNet is asked to do more fine grain discrimination between categories, relies more heavily on the configural information held within the image.
Our results support the hypothesis that the source of the inversion effect in visual expertise, including face expertise, is disruption of configural information at the level of V1.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Anonymous.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255,200 2009.
R Diamond and S Carey. Why faces are and are not special: An effect of expertise. Journal of Experimental Psychology: General, 115(2):007-117, 1986.
Martha J. Farah, James W. Tanaka, and H. Maxwell Drain. What causes the face inversion effect?
Journal of Experimental Psychology: Human Perception and Performance, 21:628-634, 1995.
Isabel Gauthier and Cindy Bukach. Should we reject the expertise hypothesis? Cognition, 103: 322-330, 2007.
Isabel Gauthier, Michael Tarr, Adam Anderson, Pawel Skudlarski, and John Gore. Activation of the middle fusiform 'face area' increases with expertise in recognizing novel objects. Nature neuroscience, 2:568-73, 07 1999.
Isabel Gauthier, Pawel Skudlarskiand John C. Gore, and Adam Anderson. Expertise for and birds recruits brain areas involved in face recognition. Nature Neuroxcience, 3:191-197, 2000.
Isabel Gauthier, Tim Curran, Kim M Curby, and Daniel Collins. Perceptual interference supports non-modular account of face processing. Nature Neuroscience, 6:428 432,,4322200 2003.
Isabel Gauthier, Rankin W. McGugin, Jennifer J. Richler, Grit Herzmann, Magen Speegle, and Ana E. Van Gulick. Experience moderates overlap between object and facerecognition, suggesting a common ability. Journal of Vision, 14:1-12, 2014.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Janet Huiwen Hsiao and Garrison Cottrell. Two fixations suffice face recognition. Psychological science, 19:998-1006, 2008
Corentin Jacques, Olivier d' Arripe, and Bruno Rossion. The time course of the inversion effect during individual face discrimination. Journal of Vision, 7, 6 2007.
R. Jenkins, A. J. Dowsett, and M. Burton. How many faces do people know? Proceedings of the Royal Society B: Biological Sciences, 285), 2018.
Ming Jiang, Shengsheng Huang, Juanyong Duan, and Qi Zhao. Salicon: Saliency in context. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1072-1080, 2015.
N. Kanwisher, F. Tong, and K. Nakayama. The effect of face inversion the human fusiform face area. Cognition, 68, 1998.
Nancy Kanwisher. Josh McDermott, and Marvin M. Chun. The fusiform face area: A module in human extrastriate cortex specialized for face perception. Journal of Neuroscience, 17(11): 4302-4311, 1997.
Günter Meinhardt, Bozana Meinhardt-Injern and Malte Persike. The complete design the com- posite face paradigm: role of response bias, target certainty, and feedback. Frontiers in human neuroscience, 2014.
J.R. Polimeni, M. Balasubramanian. and E.L. Schwartz. Multi-area visuotopic map complexes in macaque striate and extra-striate cortex. Vision Research, 2006.
C. Rezleseu, T. Susilo, Jeremy B. Wilmer, and A. Caramazza. The inversion, part-whole, and com- posite effects reflect distinct perceptual mechanisms with varied relationships to face recognition. Journal of Experimental Psychology: Human Perception and Performance, 43:1961-1973, 2017.
10

---

Under review as a conference paper at ICLR 2023
Jennifer J. Richler, Olivia S. Cheung, and Isabel Gauthier. Holistic processing predicts face recog-
nition. Psychological Science, 22:464-4711 471, 2011.
Eleanor Rosch, Carolyn B Mervis, Wayne D Gray, David M Johnson, and Penny Boyes-Braent
Basic objects in natural categories. Cognitive Psychology, 8(3):382-439, 1976.
Panqu Wang, Isabel Gauthier, and Garrison Cottrell, Experience matters: Modeling the relationship between face and object recognition. Proceedings of the 36th Annual Conference of the Cognitive Science Society, 2014.
Linjie Yang. Ping Luo, Chen Change Loy, and Xiaoou Tang. A large-scale car dataset for fine- grained categorization and verification. 2015 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 3973-3981, 2015.
Robert K Yin. Looking at upside-down faces. Journal of Experimental Psychology, 81:141-145, 1969.


Reviews:
[{"review_id": "dg1SCAYNZR", "reviewer": "Reviewer_UAVL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present an study in order to check whether an standard CNN (Resnet-50) or an standard CNN in which both a foveated rendering from the retina and a log-polar representation have benn added behaves as humans in the image inversion effects. In order to do that they perform a variaty of experiments on different datasets, from faces, to dogs, cars, and a general one.", "summary_of_the_review": "The paper is of interest and is nicely written. However, I believe it is still not at the level for a conference like ICLR.", "strength_and_weaknesses": "Strengths:\n\n- The paper is of interest, as it shows how to modify a CNN in order to perform image inversion in a similar eway to humans. If we aim at creating neural nets that ressemble most to humans, these are lines to follow.\n- The paper is very nicely written, and easy to read.\n\nWeaknesses:\n-First, I am surprised that the authors completely ignore in the paper that these ideas of looking wether a CNN performs as humans in different tasks has been around for some time. Clear examples of this are [1,2] regarding color and brightness illusions, the Scilliation rig [3], the perception on liquids [4], or contrast sentitivity functions [5]. I believe the authors need a state-of.the-art section including all this line of research.\n\n- The authors perform all the experiments in ResNet-50. I believe that, for a paper as this one, at least a couple of different neural networks architectures should be used. If not, how can we sure that this results do not only hold true for the ResNet architecture.\n\n- The authors in page 2 say that CNN are translational invariant. This is currently not fully agreed by the community, as recent studies have shown that this is not the case. See [6]. For this reason, authors need to comment on this fact.\n\n- The last experiment is not clear. Why cosine similarity? Why not to use any other metric? Can the authors ellaborate?\n\n- Finally, I recommend the authors to remove the last line of the paper. I do not agree that the inversion effect happens in V1. They prove that a log-polar transformation helps, that is suppose to be in V1; however, there is still a long way to claim for that. What happens for example if we modify the architecture.\n\n[1] Convolutional neural networks can be deceived by visual illusions\nA Gomez-Villa, A Martin, J Vazquez-Corral, M Bertalmío\nCVPR, 2019\n\n[2] Color illusions also deceive CNNs for low-level vision tasks: Analysis and implications\nA Gomez-Villa, A Martín, J Vazquez-Corral, M Bertalmío, J Malo\nVision Research 176, 156-174, 2021\n\n[3] ImageNet-trained deep neural networks exhibit illusion-like response to the Scintillating grid\nED Sun, R Dekel - Journal of Vision, 2021 \n\n[4] Visual perception of liquids: Insights from deep neural networks\nJJR van Assen, S Nishida, RW Fleming\nPLoS computational biology 16 (8), e1008018, 2019\n\n[5] Contrast sensitivity functions in autoencoders\nQ Li, A Gomez-Villa, M Bertalmío, J Malo\nJournal of Vision 22 (6), 8-8, 2022\n\n[6] Convolutional neural networks are not invariant to translation, but they can learn to be\nV Biscione, JS Bowers, JMLR 2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "cfz_aWw9aX3", "reviewer": "Reviewer_8Y2A", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors explored image inversion effects and the impact of visual expertise on performance. By using images of faces, objects, cars, and dogs, the authors were able to show that LPNet, a convolutional neural network that includes a foveated retina and the log polar mapping from the visual field to V1, can reproduce experimental results of image inversion despite being nominally rotation invariant.", "summary_of_the_review": "See Strength And Weaknesses", "strength_and_weaknesses": "Pros: \n1. The motivation is clear. \n2. The paper is well-written and organized. \nCons: \n1. The main contributions are not clear. \n2. Some related works are missing, e.g., 3D Face Reconstruction from A Single Image Assisted by 2D Face Images in the Wild.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_gBvMm_iGwF", "reviewer": "Reviewer_rdQQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Through of a set of \"machine psychophysics\" experiments, the authors probe on the image inversion effects comparing faces and other objects with a family of neural networks that are \"classical\", and other that incorporate both foveation + log-polar mapping to simulate visual processes up to V1. Authors find that their log-polar + foveation neural network model explains the image inversion effect while modern CNN's do not.", "summary_of_the_review": "I think the topic of this paper is quite exciting as it can greatly advance out understanding of face perception in humans, and overall let us know what makes faces so special compared to everyday objects (and this could in turn be helpful for machine vision). \n\nHowever: while the experiments in this paper are done with great precision and detail (error bars, optimizers are well documented to train the networks presumably trained from scratch, multiple classes & datasets), I think the premise of explaining the image inversion effect through early staged of visual processing is quite bold, (but exciting!), however I do not know what foveation has to do with this. If humans can recognize faces without foveation, then how does that results fit into this methodology? I could argue the same for log-polar mapping.\n\nPerhaps a missing experiment that remains to be done here is to re-do the experiments with the faces shrunken to the foveal area of the log-polar mapping and later evaluate the outcomes? Quite frankly, I'm also a bit confused, as I'm not sure how to make sense of these results even by having the log-polar network as a control. Maybe there is a confounding variable in this analysis, but I can't wrap my head around what the confounding variable is or could be. \n\nIt occurs to me that maybe some additional experiments could be done with scenes, but I'm not sure what is to be proven: perhaps that there is no image inversion effects in scenes? (I'm arguing for scenes here because that is a believable scenario where I can see having a model that incorporates a wide field of view through a log-polar transform be plausible -- similar to Deza & Konkle, ArXiv 2020.)\n\nMissing references:\n* Brain-like functional specialization emerges spontaneously in deep neural networks, by Dobbs et al. Science Advances 2022.\n(In general several works from Dobbs that involve training neural networks on faces and objects and comparing these resuilts)\n\n", "strength_and_weaknesses": "See below for Summary of Review. TLDR: I think the main strength of this paper is the scientific question and trying to understand what are the computational mechanisms of human perception that when simulated can disrupt face perception. The main weakness of this paper is that I do not understand the premise of using a foveated field of view to account for holistic processing of faces when faces can be accurately identified when they also lie within our fovea (!)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "10kbEhOsq5_", "reviewer": "Reviewer_yrmA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This article is interested in the so-called \"Inversion effect\" in human vision: the fact that although humans are as a rule excellent at recognising and discriminating faces, performance falls to very low levels when faces are seen upside down. This is an interesting effect as we are generally apt at recognising objects in various poses. \n\nThe authors' two hypotheses are that i) the log-polar mapping of the early visual system (retina+V1) plays a role in this effect (this is implicit in the paper's methodology); and 2) that this effect is explained by visual expertise: ie, that face recognition is a highly specialised competency that depends on complex configurations that are disrupted when a face is seen upside-down. \n\nThe paper's hypotheses are examined by recognition experiments using standard CNNs and a proposed log-polar neural network (so-called LPNet), on recognition tasks using faces and other objects as a baseline. Visual expertise is modeled as the number of classes the networks have to discriminate. \n\nResults show that: \n- validation performance drops with the number of classes in all cases. \n- validation performance drops faster for inverted stimuli\n- performance on inverted stimuli tends to be worse for the CNN compared to LPNet\n- CNNs appear to be more reliant on global configurations than LPNet", "summary_of_the_review": "In summary, I find the research idea interesting, but I have a number of concerns about the paper as it stands: \n- Although the research question and hypotheses are interesting, it is not fully clear to me that the experiments really validate the hypotheses, and that the contrived experimental setup allows for conclusions that are not already well known. \n- To justify presentation at ICLR, it would be important that the authors clarify what are the contributions and implications of their findings for computer vision and neural networks research. If the main contribution is psychological in nature maybe this is not the best venue for this paper. \n- Code and data would need to be made available to allow reproduction of the results. ", "strength_and_weaknesses": "### Strengths\n- The paper is generally clear and well written, with a clear description of the  experimental design. \n- The hypotheses are well framed with respect to the literature on the topic\n- The experimental setup is generally clear and well chosen. \n- Effects such as the inversion effect are interesting to study as they could link the high performance of the human visual system to specialisation in processing.\n\n### Weaknesses\n- Relevance to ICLR: I am not fully convinced that ICLR is the right venue for this work. Although the methodology involves neural networks, the hypothesis and conclusion are relevant to the human visual system and psychology in particular. Due to the experimental choices (mainly, artificially limited data augmentation), the results have limited bearing to the study of artificial neural networks or computer vision in general. \n- Some of the experimental results seem fairly obvious: A CNN trained with limited rotation fails at recognising examples under large rotation/inversion. A network that had some additional rotation training due to design (log-polar mapping) performs better. \n- The central argument that the inversion effect is due to the fact that recognition is mediated by configurations that are disrupted by inversion is not really supported by the experimental setup or results: the configurations learnt by a CNN are not robust to rotation if only limited rotations were included in the training data. This is a consequence of known limitations of CNN and of experimental design, but what does it say about humans? We see faces in much a wider variety of poses than suggested by the dataset? \n- Is the conclusion any deeper than: learning systems can recognise patters in similar conditions than the ones they were exposed to during training? \n- In Experiment III, it is not clear how large each \"set of images\" is. It would be good also to have a measure of variance on those numbers, as I expect it may be quite large depending on the face configuration.  \n- The argument about visual expertise could be refined in my view. In Figure 4, the loss in accuracy due to inversion climbs up to 20 classes and to a lesser extent up to 60, but seems fairly stable after that. It is unclear why visual expertise would plateau in this manner (error bars are also extremely large on the LPNet graph). ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject