COCO-Periph: Bridging the Gap Between Human and Machine Perception in the Periphery
Authors: Anne Harrington
Vasha DuTell
Mark Hamilton
Ayush Tewari
Simon Stent
William T. Freeman
Ruth Rosenholtz
Venue: ICLR 2024 poster (2024)
Keywords: peripheral vision
object detection
dataset
foveation
psychophysics
Primary area: applications to neuroscience & cognitive science
TL;DR: comparing DNNs to human peripheral vision

Abstract: Evaluating deep neural networks (DNNs) as models of human perception has given rich insights into both human visual processing and representational properties of DNNs. We extend this work by analyzing how well DNNs perform compared to humans when constrained by peripheral vision -- which limits human performance on a variety of tasks, but also benefits the visual system significantly. We evaluate this by (1) modifying the Texture Tiling Model (TTM), a well tested model of peripheral vision to be more flexibly used with DNNs, (2) generating a large dataset which we call COCO-Periph that contains images transformed to capture the information available in human peripheral vision, and (3) comparing DNNs to humans at peripheral object detection using a psychophysics experiment. Our results show that common DNNs underperform at object detection compared to humans when simulating peripheral vision with TTM. Training on COCO-Periph begins to reduce the gap between human and DNN performance and leads to small increases in corruption robustness, but DNNs still struggle to capture human-like sensitivity to peripheral clutter. Our work brings us closer to accurately modeling human vision, and paves the way for DNNs to mimic and sometimes benefit from properties of human visual processing.

Published as a conference paper at ICLR 2024
COCO-PERIPH:
BRIDGING THE GAP BETWEEN HUMAN AND MACHINE PERCEPTION IN THE PERIPHERY
Anne Harrington Vasha 2 Mark Hamilton'
Ayush Tewari*
Simon Stent William T. Freeman"
Ruth Rosenholtz
MIT CSAIL MIT Brain and Cognitive Sciences 3 Toyota Research Institute
fannekh, vasha) emin edu
ABSTRACT
Evaluating deep neural networks (DNNs) as models of human perception has given rich insights into both human visual processing and representational prop- erties of DNNs. We extend this work by analyzing how well DNNs perform com- pared to humans when constrained by peripheral vision - which limits human per- formance variety of tasks, but also benefits the visual system significantly. We evaluate this by (1) modifying the texture tiling model (TTM), a well tested model of peripheral vision, be more flexibly used with DNNs, (2) generating large dataset which we call COCO-Periph that contains images transformed to capture the information available in human peripheral vision, and (3) comparing DNNs to humans at peripheral object detection using a psychophysics experiment. Our results show that common DNNs underperform object detection compared humans when simulating peripheral vision with TTM. Training on COCO- begins to reduce the gap between human and DNN performance and leads to small increases in corruption robustness, but DNNs still struggle to capture human-like sensitivity to peripheral clutter. Our work brings us closer to accurately modeling human vision, and paves the way for DNNs to mimic and sometimes benefit from properties of human visual processing.
INTRODUCTION
Deep neural networks (DNNs) have shown great promise as models of human visual perception, en- abling the prediction of both neural response patterns (Yamins et al., 2014 Rajalingham al. 2015)
Yamins & DiCarlo 2016) Kell & McDermoti 2019) and aspects of visual task performance ( Yamiiss
al.] 2014]] Geirhos et al.) [0188] Mehrer et al.] 2021). However, there are still critical differences in how computer vision DNNs process information compared to humans (Rajalingham et al. 2015
Geirhos et al. 2020 Wichmann & Geirhos 2023). These differences are evident in psychoph
experiments (Berardino et al. Feahher et al. 2019 Hénaff et al. 2019 Harrington al. (022)
and adversarial examples (Szegedy et al. 2013 Elsayed et al. [018]] lyas et al. 2219). One differ- ence between DNNs and humans that has gained recent interest is the existence of peripheral vision in humans. Peripheral vision describes the process in which human vision represents the world with decreasing fidelity at greater eccentricities, i.e. farther from the point of fixation. Over 99% of the human visual field is represented by peripheral vision. While it is thought to be a mechanism for dealing with capacity limits from the size of the optic nerve and visual cortex, peripheral vision has also been shown to serve as critical determinant of human performance for wide range of visual tasks (Whitey & Levi 2011 Rosenholtz 2016).
The benefits of modeling peripheral vision in DNNs are two-fold. For applications that require pre- dicting or mimicking human performance on a visual task like predicting if driver will detect hazard DNNs in computer vision must capture aspects of human peripheral vision that drive task performance. For applications in representation learning, peripheral vision represents a bio- logical strategy that presumably evolved to efficiently and robustly solve variety of tasks in spite of information loss due to significant constraints on the system. DNNs might benefit from these representational strategies in areas such as robustness, where a link between adversarial robustness and human visual representations has already been seen (Engstrom et al. 2019 Iyyas et al. 2019 Harrington & Deza 2022

---

Published as a conference paper at ICLR 2024
Original (0*)
5
10
15
20
100%
100%
100%
perspers
ba
peper
u pe
eebir
person 98%
person 97sperson 95%
person 96%
teddy bear 61 ttadd beB
Godog 65%
bottle
bed 69%
bottle
bottle 97%
leddy
teddy
teddy
teddy
The COCO-Periph Dataset contains MS-COCO images that have been transformed to visualize the loss of information in human peripheral vision due visual crowding (not just acuity loss) at various eccentricities. This loss of visual information causes difficulties for computer vision systems: models such as Faster-RCNN (shown here) perform poorly at tasks like object detection as simulated eccentricity increases. In contrast, human performance is known to fail gracefully, raising the question: how can we close this gap?
Figure 1:
Accurately modeling peripheral vision in DNNs, however, is challenging. Current DNN approaches disjoint and a number of them require specialized architectures (Jonnalagadda et al. 2021 Min
iet al. (0022), only model a loss of resolution (Pramod et al. 2022) which is insufficient to predict effects of peripheral vision like crowding (Balas et al.) 2009), rely on style transfer approaches (Deza & Konkle 2020) which are not well tested as statistical models. In human vision science, peripheral vision has been well-modeled with multi- scale pyramid- based image transformations that, particular task, instead output images transformed to rep- resent the information available peripheral vision. Humans viewing these transformed images perform visual tasks with an accuracy that well predicts human performance while fixating the orig- inal images (Ehinger& & Rosenholtz 2016 Rosenholtz et al. 2012b Freeman & Simoncelli, 2011).
rather than predicting performance
In this work, we leverage one of these pyramid- based peripheral vision models, the Texture Tiling Model (TTM) (Rosenholtz et al.) 2012b), to simulate peripheral vision in a variety of DNN models. uniform rather than foveated pooling operation (uniform-
We do by modifying TTM TTM); this allows us to model a single point in the periphery without having to choose a fixation. We use uniform TTM to render a popular object dataset, MS-COCO (Lin et al.) 2014), simulate peripheral vision at the input level for DNNs we call the transformed dataset COCO-Periph. To understand the effect that peripheral vision-like inputs have on DNN performance, we perform human psychophysics experiment measuring object detection in the periphery, and then design machine psychophysics experiment to test DNNs on the same task. We compare detection results between humans and DNNs and show gap in performance between the two. This gap can be reduced by training on COCO-Periph, but we still see noticeable differences in sensitivity clutter.
The CoCo-Periph dataset is one of the largest datasets for studying peripheral vision in DNNs, and our analysis represents one of the broadest evaluations of peripheral vision modern DNNs to date. COCO-Periph itself is a significant contribution, representing over 6 months of compute time that makes it computationally feasible to test TTM in DNNs and standardizes the evaluation of peripheral vision in computer vision. We publicly release COCO- COCO-Periph dataset, along with code for uniform TTM and the psychophysics analyses at hhtpp://githhub.e to enable further research into human and machine perception in the periphery paving the way for DNNs to mimic and benefit from properties of human visual processing.
2 BACKGROUND AND RELATED WORK
2.1 PERIPHERAL VISION
Often misunderstood simple loss of acuity, peripheral vision in reality involves much more complex processes. While the retina does display progressive reduction of photoreceptor density a function of eccentricity, most of the information loss in peripheral vision occurs downstream
2

---

Published as a conference paper at ICLR 2024
in the visual cortex. The phenomenon of visual crowding exemplifies this where human peripheral performance degrades in the face of clutter from the spacing of nearby objects and the features of local image regions Vater et al. 2022
Peripheral vision has been successfully modeled as a loss of information in representation space (Rosenholtz et al.] 22122b Freeman & Simoncelli 2011), where models like TTM (Rosenholtz et al.) 20126 Rosenholtz 2020) perform a texture- pra.uuueeroocssingiiiii computation of local summary statistics within pooling regions that grow with eccentricity and tile the visual field. TTM relies on the Portilla and Simoncelli statistic set (Portilla & Simoncelli] 2000). very similar to the Freeman and Simoncelli model (Freeman Simoncelli] 2011)) Some more recent models evaluate different statistic and show strong performance on metameric tasks (Deza et al. 2017]] Wallii et al. 2019 Broderick et al. 2023 Among these, TTM is one of the only to be validated against human performance on an extensive number of behavioral tasks, including peripheral object recognition, visual search, and a variety of scene perception tasks (Ehinger Rosenholtz, 2016). Although TTM is powerful, the computational requirements of synthesizing TTM transforms make it impractical to use online at the large scale of DNNs. Synthesizing a single TTM transform image can take 5+ hours. This has been addressed in part by (Brown et al., 2021). which modified the optimization process for transform generation with gradient descent, allowing GPU-optimization, and (Deza et al.) (017) and (Wallis) jet al. (017) which incorporated style-transfer into the procedure. However, these models not as well validated on human performance as TTM, and most are still not fast enough to use during DNN training. To facilitate large experiments, we create COCO-Periph large-scale dataset that pre-computes these images with more flexible fixation scheme.
2.2 HUMAN-INSPIRED DEEP NEURAL NETWORKS
Extensive work has been done in creating poologicalll-inspiee object recognition models. A number of these models have been shown to impart representational benefits such as robustness to occlusion
(Dzza& & Konkle 2020), generalization across scale (Zhang et al. 2019 Han et al. 2020). and adversarial robustness /Vuyyuru et al., 2020 Dapello et al. [201]] Guo et al. (022). has also been suggested that ad adversarial training alone can improve human perceptual alignment (Dapello et al.)
2020 Feather et al.] 2022 Ilyas et al. 2019 Harrington & Deza 2022). Research in this domain overall has greatly benefited from DNN benchmarks such as BrainScore (Schrimpf et al. 2020 that
compare models to humans using neural and behavioral data.
Despite clear benefits on recognition tasks, modeling human vision is less explored in more complex tasks like object detection. One exception to this includes FoveaBox which takes inspiration from foveation in human vision to simultaneously predict object position and boundaries without anchors (Kong et al. 2020). Additionally, training on a stylized version of COCO (Michaelis et al., (019)) (much like the stylized ImageNet work which reduced texture bias and increased shape bias in recognition models (Geirhos et al.) 2018)) was shown to increase corruption robustness in object detection DNNs. Peripheral vision, however, is thought to use texture-like representations, and is critically involved in tasks where context matters like detection. Testing peripheral vision in tasks like detection is key to understanding the benefits and trade-offs of modeling human vision in DNNs.
3 UNIFORM TEXTURE TILING MODEL
To model critical aspects of peripheral vision without assuming a fixation point, we use modified version of TTM that relies on uniform, rather than foveated pooling. In original TTM (Rosenholtz) 2020), pooling regions grow linearly with eccentricity, but for uniform TTM, we fix the pooling region size to match the ring of pooling regions at single eccentricity (see Appendix Sec. 1. For example, we set the pooling region size to correspond to 15" eccentricity, as in Figure 2 With that uniform pooling, we can create images that show the information available as if each pooling region appeared at the same eccentricity. Though this represents an impossible situation, it provides two practical advantages: (1) the ability to shift the modeled fixation by stitching together pre-computed uniformly transformed images to create secudo-foveatee images (see Appendix Sec. [A.2), and (2)
to evaluate both human and machine performance for entire image at single eccentricity.
3

---

Published as a conference paper at ICLR 2024
Figure 2: Original vs. Uniform Texture Tiling Model (TTM). An original image (a) can be pro- cessed using the Original TTM (b) which is foveated, meaning that information is pooled in regions that grow farther from am assumed fixation point (green dot). To create our dataset we adapt TTM use fixed uniform pooling region size everywhere in the image, shown in (c) 15 eccentricity. We can efficiently stitch together multiple uniform TTM images from our pre-computed dataset to recreate the foveated effect (d).
COCO-PERIPH DATASET
We apply Uniform TTM to the COCO dataset, creating COCO-Periph which contains images trans- formed like peripheral vision. COCO-Periph allows use the highly tested Texture Tiling as an input pre-processing step to train and evaluate DNNs. In COCO-Periph, we rendered images that capture the amount of information available humans at (5", 10 15", 20%) in the periphery, as- suming 16 pixels per degree. (For reference, the width of a full-screen image on a laptop at a typical viewing distance subtends 20 40%). COCO-Periphl contains the entire COCO 2017 validation and test set, and over 74K, 117K, 118K, and 97K of the 118K total training images transformed to model 5", 10", 15 and 20 eccentricity, respectively.
Measuring object detection performance on COCO-Periph using the original COCO ground truth labels, we see in Table [ for variety of pre-trained models that average precision (AP) degrades with distance in the periphery. De-noising models, which have the highest baseline scores, perform the best overall compared to the other architectures measured. Performance likely degrades because COCO-Periph is potentially out of distribution for models, and at farther eccentricities, objects have a greater potential to move due to the larger pooling regions used in uniform TTM. To understand how the degradation in performance experiment in Section In the psychophysics analysis, we address box localization issues (see Sec () and train Faster-RCNN on COCO-Periph (See Table TableeDottonn bottom row and Sec. 7.I
see compares to human vision, we conduct psychophysies
5 HUMAN PSYCHOPHYSICS: OBJECT DETECTION IN THE PERIPHERY
To compare DNN performance to humans in the periphery, we first collected human psychophysics data on an object detection task. We choose detection rather than a recognition task because humans can guess object identity quite well based on context alone, i.e. even when the object itself occluded (Wijntjes & Rosenholtz] 2018). our detection task, we present two images on every trial, identical except for the presence absence of particular object, and ask human subject to judge which of the two images contained a target object. For the object present images, we choose 26 images from the COCO validation set that have one instance of object. For the absent image, we remove that object via in- poianinnin (see Appendix Sec. A.3.I). We selected images with a variety of small to medium sized objects in different scenes.
In each trial, 10 eye-tracked subjects fixated at a specified location either 5 10 15,, or 20 away from the target object, and viewed object present and absent image in random order. Subjects
4

---

Published as a conference paper at ICLR 2024
Model Areh
AP
AP 5
10
AP 15
AP 20
DINO-FocalNet-aage (Zhang ct al.
2022
58.4
51.6
44.4
20.2
15.0
DINO-Swin-Tinys (Zhang al. 2022)
51.3
44.0
34.1
11.1
7.6
Detr-R50 (Carion ét al. 2020
42.0
35.2
25.1
6.9
4.5
RetinaNet-R50 (Lin al. [2017
38.7
31.5
22.1
6.9
5.0
FoveaBox (Kong et al. 2020)
40.4
33.2
23.4
7.5
5.3
Faster- Faster-RCNN-XIIn RCNN-XI01 (Ren al.
2015
39.6
32.6
21.8
6.6
4.7
Faster-RCNN-R50 (Ren et al.
2015
36.7
29.4
19.9
5.9
4.2
All Train Faster-RCNN-R500
33.8
30.5
28.1
15.8
12.7
All o FT Faster-RCNN-RS0
36.1
31.8
27.7
13.9
10.8
Table I: Average Precision (AP) COCO and COCO-Periph Validation Set. 0" refers to per- formance on unchanged MS-COCO data. The other AP values correspond to different eccentricities of uniform TTM transform images.
were asked to report which image contained the specified object in a ee--interaal-fore---hhiee paradigm (2IFC), viewing 10 present/absent image pairs each eccentricity. As control,
also tested subjects on the same task but on uniform TTM transformed images. Details on these experimental setups can be found the Appendix Sec. A.3.2 A.3.3 and A.4
We find overall that human object detection performance always degrades progressively with in- creasing eccentricity (Figure blue line). Detection ability is consistently strong 5 However, for some images observers reach near chance performance at 20 eccentricity, whereas few image pairs have objects that are easily detected at all eccentricities. Often, high color contrast between the object and its background and lack of clutter from other nearby objects made target objects more easily detected in the periphery, leading to better performance, which consistent with the crowding literature. See Appendix Sec. A.4.1 for per-image human accuracy.
MACHINE PSYCHOPHYSICS
0.9
Accuracy
0.8
0.5
Focalies PACCN 850
0.5
10
15
20
b.
Eccentricity
Figure 3: Example Easy Object Detection. (a) Original image with target object bounding box, and (b) TTM transform for 15" (240 pixels) with extended bounding box (used to perform machine object detection task). (c) Human accuracy for peripheral viewing of example image averaged over all subjects with error bars reporting SEM (blue), compared with accuracy on TTM image for pre- trained (pink) and trained on COCO-Periph (gray) Faster RCNN R50 model, and a DINO FocalNet model (green). Psychometric curves are fit with an inverse cumulative normal distribution.
To compare human and DNN performance, we have DNN object detection models perform the same two-alternan wwo-aleenative/ineeew forced choice task given to human subjects. We do this by first using uniform TTM to generate 10 different peripheral transform images for each of the object present and absent images, at each of the tested eccentricities (5", 10", 15 20%). Because TTM is stochastic model that under-constrained compared to image pixel values, each of the 10 TTM transform images differ from one another. This gives 100 unique present/absent transform pairs for each original image/ecccntrriiity combination. For each pairing, we input a transformed image to the object detection model with low detection threshold (0.01) to get proposed bounding boxes and object scores. We then determine if the proposed box overlaps with a padded ground truth box of the target object; we pad the ground truth box by half the width of pooling region account for
5

---

Published as a conference paper at ICLR 2024
1.0
20
Critical Eccentricity
0.9
lu 113
ACCUROO
Human UTTN rale
Human
10
Haman UTTN all
DRO Swit Try
0.7
RETIA RSD
DPRO Focalhet
DETI RS0
RACNN KI01
Human UTTTA all
Human UTTH UTT14 neivo Human
All FRICH
All FT FYACIN
DINO rocalltes
Fovestiox
FenCHN R50
FrACIN x1o1
R50
RACNN R50
All TR FIRCMN
DINO Swin
RETINA
All RACK
Foveebox
2.5
5.0
12.5
Eccentricit (degrees)
Figure 4: Defection Accuracy and Critical Eccentricity for Humans and DNNs averaged over all images. Accuracy averaged across images and over subjects for human data is reported by eccentricity (left). We show data for humans viewing original images (Human), and humans viewing uniform TTM (uTTMM images (hatched blue bars: 3 naive subjects, 5 all subjects). To this data, we fit psychometric functions. We summarize performance across eccentricity as the critical fall-off point y (right). In both plots, we observe noticeable drop in performance near large eccentricities.
position uncertainty introduced in human peripheral vision and TTM (see Figure B and b for an example of 15 padding). To measure how strongly the DNN predicts there is object in padded box region, we sum the total scores of all objects (regardless of predicted class) that overlap at least 75 percent (intersection of area with respect to the proposed box). We score the model correct on a trial if the total object scores for the present image are greater than the absent. We score incorrect the absent is greater and give half score if present and absent are equal. We take the average over all the present-absent pairing accuracies for each eccentricity. See Appendix Sec. A.5 for pseudo-code and Figure 14 for the general workflow.
To keep the comparison between DNNs and humans fair, we do not enforce that the model must predict the correct object identity when scoring predictions at each trial. Because we use forced- choice paradigm, humans subjects can give correct response by simply detecting the presence of any object at the approximate right location, rather the specified one. Although we specify an object class to human subjects, this strategy is likely to happen when peripheral information is poor.
7 HUMAN VS MACHINE PERFORMANCE AT PERIPHERAL OBJECT DETECTION
Like the human observers, DNNs' response accuracies are highly image-dependent, with some pairs resulting poor performance for all models. While human performance falls gradually for most im- ages, DNN object detectors can often retain good accuracy for the 5" eccentricity TTM transforms, but many show sharp falloffs in accuracy to chance performance soon after (See Figure 3 for representative example).
To quantitatively compare performance, we fit both human and DNN performance data across eccen- tricity to a psychometric function for each image. We use a reverse cumulative Gaussian distribution which determines the critical (75% correct, halfway between perfect and chance performance) ec- centricity by the mean of the distribution (μ), and the performance falloff rate by (a) (Strasburger) 2020).
For all images tested, humans outperform all object detection models, with critical thresholds more than 50 greater than detection models (Figure 4 We find generally weak correlations between between DNN and human performance for critical eccentricity (μ) (see Appendix Fig, [7]. Among the pre-trained models, DINO detectors have the closest critical eccentricity to humans and have the strongest correlation.

---

Published as a conference paper at ICLR 2024
Human FIRENN R5O All 'TR FYRCNI
Criticall Eccentricity (u Fiit
20
15
10
5
b.
Figure 5: Detection Recognition. (a) Fine-tuning and training COCO-Peripe increases base- line Faster-RCNN-R50 performance at the object detection psychophysies task. (b) Machine psy- chophysics performance recognition. We pass the classification head of Faster-RCNN models the groundtruth bounding box and score performance based on recognition in that region. Data is averaged over all experiment images.
7.1 TRAINING ON COCO-
To reduce the gap between human and DNN performance, we fine-tune and train ResNet-50 back- bone Faster-RCNN-FPPy detection model on COCO-Periph (See Appendix Sec. [.88 for AP results and training details, and Appendix Fig. [2 for bounding box examples). For fine-tuning (plotted as All FT RCNN) we start from detectron2's 2's (Wu al. (2099) pre-trained model and use 3x iteration scheme with lowered learning rate. When training from scratch (plotted as All FT RCNN), we use the default 3x training configuration in detectron2.
find that training a model from scratch with all eccentricities COCO-Periph plus original COCO images (0%) produces the best performing model in the psychophysics evaluation (see Fig- ure 4. The model trained with COCO-Periph has a critical eccentricity of nearly 5 greater than the pre-trained baseline (Figure 4 5 The fine-tuned model, however, under-performs the trained model which we suspect is because of the lowered learning rate during training and a decrease in baseline average precision. In addition, report that DNN psychophysics performance similar on uniform, original TTM, and pscudofoveated TTM (see Appendix Fig. 15 16..
To better understand the impact training on COCO-Periph has on the psychophysics performance, we additionally evaluate object recognition in the machine psychophysics (Figure 5). We give the classification head of Faster-RCNN- R550 models the padded ground truth bounding box of the target object. We then score models based which image. present or absent, has the highest classification probability for the target object. Unlike the detection version of the task, we find that training from scratch performs worse than baseline. This could indicate the trained model improved more at localizing objects rather identifying them the periphery.
Model
mAP. c
severity=
severity=
brightness
elastic transform
Faster-RCNN 17.34
25.08
9.57
28.29
11.94
All FineTune
17.47
24.80
9.84
27.31
14.48
All Train
16.72
23.28
9.88
25.31
15.02
Table 2: Corruption Robustness Average Precision (mAP,) .) on COCO Validation Set. All mod-
els are Faster-RCNN ResNe150 FPN architecture (Ren et al. 2015). (mAP.) refers to AP over 15 corruptions at 5 severity levels (Michaelis et al.) 2019). Columns (severity= 1) and (severity= 5) report AP over corruptions at a single severity level. Last two columns (brightness, elastic) report AP for the worst and best performing corruption compared to baseline.
In order to see if there were benefits from training COCO-Periph beyond getting closer to human performance, we evaluated trained models for corruption robustness using the COCO-C suite of corruptions (Michaelis et al.] 2019). We find that corruption robustness improves most noticeably for geometric transformations like glass blur' and 'elastic transform'. Interestingly, performance is slightly lower than baseline for like corruptions and ones that change contrast (See Appendix Fig. [6). Averaged over all corruptions, average precision is slightly higher for the model trained from seratch on COCO-Periph than baseline.

---

Published as a conference paper at ICLR 2024
50
50
Human
DINO Swin Tiny
DINO FocalNet
40
RETINA R50
DETR R50
FIRONN X101
Fit (*)
30
30
FrRCNN R50
Fit
All
20
20
All FT FYRCNN
10
10
1000
2000
3000
4000
5000
10 15 20
25
30
35
a.
Bounding Box Size (Pixel Area)
b.
Number of Annotations
Figure 6: Object Size and Clutter. Object size predicts critical eccentricity for human observers (blue), but remains low for large objects in all object detection models tested. b: In humans (blue), critical eccentricity is highest for images with few objects, with performance decreasing as images become more crowded with objects. This relationship is not observed for tested object detection models, where critical eccentricities remain low.
7.2 EFFECTS OF OBJECT SIZE AND CLUTTER
Since both human and DNN performance strongly varied by image, we looked for image properties that might predict performance, and asked if these had similar effects for humans and computer vision DNNs. Examining critical eccentricity as function of object size, humans have higher critical eccentricity for larger objects; that is, human performance increases with progressively larger target objects (Figure 6) Surprisingly, this relationship does not appear to hold for any object detection model, even the ones trained on COCO-Periph which have higher AP-large and AP-small than baseline on TTM transform images (see Appendix Sec. [..8.I).
Human object detection performance in the periphery is known to be strongly mediated by the amount of clutter. One measure of clutter is the number of objects near the target. To test if this holds true for object detection models in our experiment, we used the number of ground truth COCO annotations the image as a proxy for clutter (note clutter can be present in specific sub-regions of an image, and that COCO annotations do not label all objects in many scenes). As expected, human performance decreases as images become more cluttered (Figure (). Performance object detection models does not show strong relationship with clutter. Interestingly, this is true for models trained on COCO-Periph, which should reflect the degrading effect of clutter on the peripheral representation, according to TTM.
DISCUSSION
To evaluate the effects of peripheral vision on DNNs, we modified a summary statistic model of peripheral vision (TTM) to be more flexibly pre-computed as an image transformation. We then rendered a large subset of the COCO dataset (COCO-Periph) to model 5", 15", and 20 degrees in the periphery to feed into object detection DNNs. To compare performance against humans, we collected human data on detection in the periphery and measured performance against DNNs.
8.1 UNDERSTANDING DIFFERENCES IN HUMAN AND DNN PERFORMANCE
Our results expose gap in performance between humans and computer vision DNNs in the periph- ery. When we restrict DNNs to input matching human peripheral vision, detection performance matches humans for some models at low eccentricities, but quickly becomes brittle, degrading sharply with eccentricity, whereas human performance falls off smoothly. What underlies the noted differences in performance and could explain this? Can rule out limitations of TTM itself? While TTM is widely-tested like all other peripheral vision models, sometimes under- aeder-predicts human performance, and validation has primarily been in greyscale images (although TTM-like models have been tested for color under the metamerism task (Brown et al.) (02T). and see (Jagadeesh & Gardner 2022)). We argue that while these limitations of I'M may put ceiling on model perfor-
8

---

Published as a conference paper at ICLR 2024
mance, TTM uses the same statistics all eccentricities, which cannot explain the sharp falloff in performance at high eccentricities as compared to humans. This along with differences in perfor- mance between models indicate that gaps between human and machine performance are unlikely to be explained by stimuli alone. Furthermore, we note multiple aspects in which we aimed to conser- vatively design our machine psychophysics experiment, making the comparison between humans and machine as fair as possible, and the task as easy as possible for models to perform well on (See Section Sc.tion(A.S). Finally, and most notably, we validate in a control experiment that human detection performance on TTM images closely follows performance on original images (Fig. (4). Thus, it is unlikely that DNNs were disadvantaged compared to humans in terms of the amount of information available.
8.2 EFFECT OF PERIPHERAL TRAINING
Training on the peripheral images in COCO--Peripl reduces the gap in object detection performance. Interestingly, see evidence that training helps more with object localization rather than identi- fication (Figure 5) this may explain why models trained on COCO-Periph do better on the psy- chophysics tasks than some models that have a higher AP on COCO-Periph. One role of peripheral vision is to guide where to fixate next, and favoring localization over identification aligns with the goal of guiding fixation. Despite these improvements, we observe that models trained on COCO- Periph still exhibit drops in performance that are greater than humans. This suggests that the be- havior we see is not solely attributable to a domain shift. The long-term purpose of our dataset and experiments are to build new ways of matching human behavior beyond fine-tuning alone. Our re- sults imply that task formulation is critical area to explore in aligning DNNs and humans. TTM as model suggests that one general representation can explain behavior on variety of visual tasks. We believe an important future direction in bridging the gap between humans and DNNs is to opti- mize for generalization across variety of tasks rather than maximizing for accuracy on a single task. Current benchmarks in computer vision do not encourage this, and we hope that our dataset and experiments can facilitate research in this direction.
Training on COCO-Periph also increases robustness to geometric corruptions, but decreases ro- bustness noise corruptions. Although the texture-like representations of peripheral vision may contribute to human robustness to adversarial noise (Harrington & Dezal 2022). the TTM-transform itself more closely resembles geometric corruptions and this is evident in our robustness evaluations. While we do not evaluate the adversarial robustness of our trained models, it appears that more work needed fully understand the relationship between peripheral vision and robustness.
8.3 COCO-PERIPH NEW BENCHMARK WITH REAL- WORLD APPLICATIONS
A key contribution of our work is COCO-Periph, one of the first large scale datasets for study- ing peripheral vision in DNNs. We present an application of COCO-Periph in object detection, but COCO-Periph provides unified and flexible framework for modeling peripheral vision in any modern DNN. By building the dataset on COCO, peripheral vision can be evaluated for the first time
on tasks that go beyond simple crowding measures (Volokitin et al. 2017 Lonnqvist et al.) 2020 and object/scene recognition (Deza & Konkle 2020: Jonnalagadda et al. 2021 Pramod et al. 2022
Min et al. 2022). We present COCO-Periph, along with our psychophysies experiment, as tool to the community to build DNNs that can predict human performance and benefit from properties of
human vision.
Peripheral vision is useful to model and predict because plays a key role in guiding visual attention in humans, enabling fast, efficient detection of features and motion over a wide visual field. Peripheral representations in machine learning give performance boosts for object recognition (Pramod et al.) 2022 Jonnalagadda et al. 2021; Min et al.) (022), and we extend this work to object detection. Modeling this enables us to better predict human visual behavior. This has applications in many areas including: (1) driver safety where we could predict a person sees a hazard, (2) content memorability where we could optimize images to capture attention, (3) UV/UX to create displays that easy to view, and (4) foveated rendering, and (5) compression where peripheral vision could help models perform under reduced visual information.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
Our study contributes to research modeling human visual behavior. Although we do not achieve human level performance, potential harms of such system could include difficulty distinguishing AI from humans and tracking human attention patterns. Reducing the harm of both these risks could be achieved through watermarking or other signatures to make clear when a model is being used. Regarding our human subject experiment, all participants provided informed consent prior participation, compliance with the Common Rule (45 CFR 46). This study was assessed as exempt from review by an Institutional Review Board, pursuant to 45 CFR 46.1 (0) There were no risks discomforts associated with the study, beyond what normally expected using a standard computer or video game. The eye tracker used was non-invasive and did not require special contact lenses. To minimize visual and motor fatigue, participants were given breaks every 15 minutes and were informed that they could exit the experiment any point for any reason. All participant data was anonymized and kept a locked room under password protection. See Appendix section A.3.2 for for further details.
ACKNOWLEDGMENTS
This work funded by the Toyota Research Institute, CSAIL MEnTorEd Opportunities in Re- search (METEOR) Fellowship. US National Science Foundation under grant number 1955219, and National Science Foundation Grant (CCS-1826755 to PI Rosenholtz. The authors acknowledge the MIT SuperCloud Reuther et al. (018) and Lincoln Laboratory Supercomputing Center for provid- ing HPC resources that have contributed to the research results reported within this paper.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Benjamin Balas, Lisa Nakano, and Ruth Rosenholtz. A summary-statissic representation in periph- eral vision explains visual crowding. Journal of vision, 9(12):13-33 2009.
Alexander Berardino, Valero Laparra, Johannes Ballé, and Eero Simoncelli. Eigen-distortions of hierarchical representations. Advances in neural information processing systems, 30, 2017.
William Broderick, Gizem Rufo, Jonathan Winawer, and Eero Simoncelli. Foveated metamers of the early visual system. bioRxiv, pp. 2023-05, 2023.
Rachel Brown, Vasha DuTell, Bruce Walter, Ruth Rosenholtz, Peter Shirley, Morgan McGuire, and David Luebke. Efficient dataflow modeling of peripheral encoding in the human visual system. arXiv preprint arXiv:2107.11500 2021.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Aision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pp. 213-229. Springer, 2020.
Joel Dapello, Tiago Marques, Martin Schrimpf, Franziska Geiger, David Cox, and James J DiCarlo. Simulating a primary visual cortex the front of cmns improves robustness image perturbations. Advances in Neural Information Processing Systems, 33: 13073-13088, 2020.
Joel Dapello, Jenelle Feather, Hang Le, Tiago Marques, David Cox, Josh McDermott, James J DiCarlo, and Sue Chung. Neural population geometry reveals the role of stochasticity in robust perception. Advances in Neural Information Processing Systems, 34:15595 15607, 2021.
detrex contributors.. detrex: An research platform for transformer- based object detection algorithms.
https:///itthut com Research/ rex 2022.
Arturo Deza and Talia Konkle. Emergent properties of foveated perceptual systems. arXiv preprint arXiv:2006607991, 2020.
Arturo Deza, Aditya Jonnalagadda, and Miguel Eckstein. Towards metamerism via foveated style transfer. arXiv preprint wrXiv:1705. 10041, 2017.
Krista A Ehinger and Ruth Rosenholtz. general account of peripheral encoding also predicts scene perception performance. Journal of Vision, 16(2):13-13, 2016.
Gamaleldin Elsayed, Shreya Shankar, Brian Cheung, Nicolas Papernot, Alexey Kurakin, Ian Good- fellow, and Jascha Sohl-Dickstein. Adversarial examples that fool both computer vision and time-limited humans. Advances in neural information processing systems, 31, 2018.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Alek- sander Madry. Learning wenceettalll--alinnee representations via adversarial robustness. ArXiv preprint ar arXiv:1906.0090 2019.
Jenelle Feather, Alex Durango, Ray Gonzalez, and Josh McDermott. Metamers of neural networks reveal divergence from human perceptual systems. Advances in Neural Information Processing Systems, 32, 2019.
Jenelle Feather, Guillaume Leclere, Aleksander Madry, and Josh H McDermott. Model metamers illuminate divergences between biological and artificial neural networks. bioRxiv, 2022.
Jeremy Freeman and Bero P Simoncelli. Metamers of the ventral stream. Nature neuroscience, 14 (9):1195-122)) 1201, 2011.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis. Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias im- proves accuracy and robustness. arXiv preprint 1xX.v:/8/1.12233 2018.
Robert Geirhos, Kristof Meding, and Felix A Wichmann. Beyond accuracy: quantifying trial-by- trial behaviour cnns and humans by measuring error consistency. Advances in Neural Informa- fion Processing Systems, 33: 11990-1399,, 2020.

---

Published as a conference paper at ICLR 2024
Wilson S Geisler and Jeffrey S Perry. Real-time simulation of arbitrary visual fields. In Proceedings of the 2002 symposium on Eye tracking research & applications, pp. 83-87, 2002.
Chong Guo, Michael J Lee, Guillaume Leclere, Joel Dapello, Yug Rao, Aleksander Madry, and James J DiCarlo. Adversarially trained neural representations may already be as robust corre-
sponding biological neural representations. arXiv preprint arXiv:2206. 1228, 2022.
Yena Han, Gemma Roig, Gad Geiger, and Tomaso Poggio. Scale and translation-invariann for
novel objects in human vision. Scientific reports, 10(1):1411, 2020.
Anne Harrington and Arturo Deza. Finding biological plausibility for adversarially robust features via metameric tasks. In International Conference on Learning Representations, 2022. URL
eew net/ wwt.eewwweeweeeeeeeeeeeee Forum?id=yaa_ _
Anne Harrington, Vasha DuTell, Ayush Tewari, Mark Hamilton, Simon Stent, Ruth Rosenholtz, and William T Freeman. Exploring perceptual straightness in learned visual representations. In The Eleventh International Conference Learning Representations, 2022.
Olivier J Hénaff, Robbe LT Goris, and Eero P Simoncelli. Perceptual straightening of natural videos.
Nature neuroscience, 22(6))984-999 2019.
Andrew Ilas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Alcksander Madry. Adversarial examples are not bugs, they features. Advances in neural information processing systems, 32, 2019.
Akshay Jagadeesh and Justin Gardner. Texture- representation of objects in human visual cortex. Proceedings of the National Academy of Sciences, 19((7))e211530211 2022.
Aditya Jonnalagadda, William Yang Wang. BS Manjunath, and Miguel P Eckstein. Foveater:
Foveated transformer for image classification. arXiv preprint 1rXiv:2105.14177. 2021.
Alexander JE Kell and Josh McDermott. Deep neural network models of sensory systems: win- dows onto the role of task constraints. Current opinion in neurobiology. 55:121-1322 2019.
Tao Kong, Fuchun Sun, Huaping Liu, Yuning Jiang, Lei Li, and Jianbo Shi. Foveabox: Beyound unchor-based object detection. IEEE Transactions on Image Processing, 29:7389-7399 2020.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland. September 6- 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pp- 2980-2988, 2017.
Ben Lonnqvist, Alasdair DF Clarke, and Ramakrishna Chakravarthi. Crowding in humans is unlike that convolutional neural networks. Neural Networks, 126:262-274, 2020.
Johannes Mehrer, Courtney J Spoerer, Emer C Jones, Nikolaus Kriegeskorte, and Tim C Kietzmann. An ecologically motivated image dataset for deep learning yields better models of human vision. Proceedings of the National Academy of Sciences, 1 118(8):e201 1417118, 2021.
Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexan- der S. Ecker, Matthias Bethge, and Wieland Brendel. Benchmarking robustness in object detec- tion: Autonomous driving when winter is coming. arXiv preprint arXiv: 1907.074844 2019.
Juhong Min, Yucheng Zhao, Chong Luo, and Minsu Cho. Peripheral vision transformer. arXiv preprint arXiv:2206. 220.:2206000001,, 2022.
Jeffrey S Perry and Wilson S Geisler. Gaze-contingent real-time simulation of arbitrary visual fields. In Human vision and electronic imaging VII, volume 4662, pp. 57-69. SPIE, 2002.
Javier Portilla and Eero P Simoncelli. A parametric texture model based on joint statistics of com- plex wavelet coefficients. International journal of computer vision, 700 2000.
12

---

Published as a conference paper at ICLR 2024
RT Pramod, Harish Katti, and SP Arun. Human peripheral blur is optimal for object recognition. Vision research, 200: 108083, 2022.
Rishi Rajalingham, Kailyn Schmidt, and James DiCarlo. Comparison of object recognition behav- ior in human and monkey. Journal of Neuroscience, 35(35): 2015.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Albert Reuther, Jeremy Kepner, Chansup Byun, Siddharth Samsi, William Arcand, David Bestor, Bill Bergeron, Vijay Gadepally, Michael Houle, Matthew Hubbell, Michael Jones, Anna Klein, Lauren Milechin, Julia Mullen, Andrew Prout, Antonio Rosa, Charles Yee, and Peter Michaleas. Interactive supercomputing on 40,000 cores for machine learning and data analysis. In 2018 IEEE High Performance extreme Computing Conference (HPEC), pp. 1-6. IEEE, 2018.
Ruth Rosenholtz. Capabilities and limitations of peripheral vision. Annual review of vision science, 2:437-457, 2016.
Ruth Rosenholtz. Texture tiling model, 2020. URL https://aspacc.mitttco handul//
1721.1/1240
Ruth Rosenholtz, Jie Huang, and Krista A Ehinger. Rethinking the role of top-down attention in vision: Effects attributable to lossy representation peripheral vision. Frontiers in psychology. 3:13, 2012a.
Ruth Rosenholtz, Jie Huang, Alvin Raj, Benjamin Balas, and Livia Ilie. A summary statistic representation in peripheral vision explains visual search. Journal of vision, 14-14, 2012b.
Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J Majaj, Rishi Rajalingham, Elias B Issa, Ko- hitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, pp. 407007, 2020.
Hans Strasburger. Seven myths on crowding and peripheral vision. i-Perception, 11(3):
2041669520913052, 2020.
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksci Silvestrov, Nacjin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution-robust large mask inpainting with fourier convolutions. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2149-2159, 2022.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint 1rXivv1312.6199 2013.
Christian Vater, Benjamin Wolfe, and Ruth Rosenholtz. Peripheral vision in real-world tasks: A systematie review. Psychonomic bulletin & review, 29(5):1531-153 2022.
Anna Volokitin, Gemma Roig, and Tomaso A Poggio. Do deep neural networks suffer from crowd- ing? Advances in neural information processing systems, 30, 2017.
Manish Reddy Vuyyuru, Andrzej Banburski, Nishka Pant, and Tomaso Poggio. Biologically in- spired mechanisms for adversarial robustness. Advances in Neural Information Processing Sys- tems, 33:2135-2146, 2020.
Thomas Wallis, Christina Funke, Alexander Ecker, Leon Gatys, Felix Wichmann, and Matthias Bethge. Towards matching peripheral appearance for arbitrary natural images using deep features. Journal of Vision, 17(10):786-788 2017.
Thomas SA Wallis, Christina M Funke, Alexander $ Ecker, Leon Gatys, Felix A Wichmann, and Matthias Bethge. Image content is more important than bouma's law for scene metamers. ELife, 8:e42512, 2019.
David Whitney and Dennis M Levi. Visual crowding: A fundamental limit on conscious perception and object recognition. Trendy in cognitive sciences, 1504))160-166, 2011.
13

---

Published as a conference paper at ICLR 2024
Felix A Wichmann and Robert Geirhos. Are deep neural networks adequate behavioral models of human visual perception? Anmual Review of Vision Science, 9, 2023.
Maarten WA Wijntjes and Ruth Rosenholtz. Context mitigates crowding: Peripheral object recog- nition in real-world images. Cognition, 180:158-164, 2018.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. 2019.
Daniel LK Yamins and James J DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience, 19(3):55-3365, 2016.
Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James DiCarlo. Performance-optimiza hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 1 111(3)::619-8624 2014.
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M. Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection, 2022.
Jiaxuan Zhang, Yena Han, Tomaso Poggio, and Gemma Roig. Eccentricity dependent neural net- work with recurrent attention for scale, translation and clutter invariance. Journal of Vision, 19 (10):209-22) 209, 2019.
14

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 UNIFORM TTM METHODS
Original TTM
Uniform TTM
Figure 7: Pooling Regions in Original and Uniform Texture Tiling Models (TTM). Original TTM (Rosenholtz et al., 2012a) foveated, so its pooling regions are small around the fixation point and grow farther from the fixation. We adapt TTM to use a fixed pooling region size everywhere in the image (Uniform TTM). The size is determined by the distance in the periphery being modeled.
In the original TTM model, the pooling region size is determined by pooling rate, and distance from the fovea, d. For the uniform version, we fix d for a certain eccentricity rather than varying like the original model. We set the overlap between pooling regions to be 60%, and we arrange the uniform pooling regions a rhombic lattice to make it as close as possible to original TTM. We use the same synthesis procedure original TTM (matching statistics like those defined in (Portilla & Simoncelli 2000) for each pooling region iteratively from noise). The uniform TTM transforms take between 2 3 hours to synthesize take hours on 1 core). Closer eccentricities like 5 take longer to run than large ones because the pooling region size is small. For COCO-Periph, we create uniform TTM transforms for 5,10,15 15, or 20". For all TTM transforms, we assume that there are 16 pixels per degree, which is standard for original TTM.
CPU core (compared to the original TTM transforms, which
When changing to uniform pooling, we also change the ordering of pooling region optimization from foveated TTM. Foveated TTM alternates spiraling from fovea to edge of periphery and back. This caused artifacts in uniformly-tile -iied TTM. Therefore, we opted for randomly ordered optimization of the pooling regions, eliminating the optimization artifacts.
A.2 PSELDO-FOVEATION
Using the uniform TTM transforms, can quickly simulate fixation anywhere in an image, essentially generating a foveated TTM transform at low computational cost. This has been done previously for blur (Perry & Geisler 2002 Geisler & Perry 2002). We call the process pseudo- foveation. To create pseudo-foveated images, we stitch together uniform TTM transforms rendered multiple eccentricities (Figure (). For the fovea, we insert a circle crop of the original image. Then we add cropped rings from the uniform TTM transforms, centering each ring at the eccentricity it was rendered at (5 degree TTM transform is centered at 5" eccentricity in the image). To reduce edge effects, we linearly blend the border between uniform TTM transform crops; all borders are weighted equally in the blending. With pre-computed transforms, this process of eseudo-foveation takes only 50ms per image, more efficient at generating images than foveated TTM, making it fea- sible to incorporate into the dataloading loop for training and testing DNNs.
15

---

Published as a conference paper at ICLR 2024
20
15
5
10
10
20
Figure 8: Pseudo-Foveet images created by stitching together Uniform TTM images. Pseudo-foveation allows us to simulate a fixation at any point in an image. The top row shows TTM transforms for a single eccentricity, and the bottom row shows two pseudo-foveated TTM transforms for different fixation points.
A.3 HUMAN PSYCHOPHYSICS EXPERIMENT
A.3.1 PRESENT/ ABSENT EXPERIMENT IMAGE PAIRS
Original
Mask
In-Painted
Figure 9: Example of LaMa (Suvorov et al. 2022) inpainting on COCO validation set image.
We inpainting create the object absent version of each image in the psychophysies experiment. Here, the wine-glass is removed with few artifacts.
To create pairs of images where a given object was both present and absent, we used images from the MS-COCO validation set (Lin al. (014) (such that they would be novel to both humans and to trained networks). We found images in landscape orientation where an object from a COCO object category appeared and labeled only once in the image, and the object
detected with
at least 50% confidence in the original image with the detectron2 (Wu et al. (019)) object detection model (faster.renn_r5)) renn_r50_fpnn) From this set, we hand-sclected 26 images that spanned range of conditions that would affect the difficulty of the peripheral detection task (object identity and size, variation in luminance and color contrast from background, amount of crowding around object, etc). We then used the LaMa image in- tn-painting model to inpaint the chosen object (Suvorov et al.) 2022), with a hand-drawn in-painting mask rather than the entire bounding box, as to avoid in- painting nearby objects in crowded scenes. In addition to the in-painted images, for each COCO image we also created a size matched 1/f pink noise mask to eliminate any motion transients and after-image effects during the experiment. These 26 image pairs were used for our object detection experiments. Note that figures reflect 24 images, as 2 images were removed from analysis because of poor psychometric eurves fits (see Figures 20 and 21 to view the final image set).
16

---

Published as a conference paper at ICLR 2024
500
400 ms
500
1500
Figure 10: Human psychophysics experiment trial. Subjects complete a 21FC (2 interval forced choice) task where they determine if target object appears disappears in sequence. Red arrows indicate the location of the target object. Subjects fixate at a cross that is placed at 5, 10, 15 or 20 away from the object.
A.3.2 EXPERIMENTAL SETUP
All participants provided informed consent prior to participation, in compliance with the Common Rule (45 CFR 46), and this study assessed exempt from review by MIT's Institutional Review Board, pursuant to 45 CFR 46. (b)(2).). Participants took approximately 2 hours to complete the study and were paid $40 Amazon gift card for their participation.
12 subjects participated in the human psychophysics experiment. We discarded the data from subjects due computer malfunction and difficulty eye-tracking with a strong contact lens pre- scription. The remaining subjects consisted of 4 Male, Female, and Non-binary subjects ranging age from 19 31. All had self-reported normal or corrected normal visual acuity with contact lenses, with no history of eye surgery. 2 subjects had corrective lenses for myopia with correction less than -1.25, but did not normally wear glasses or contacts (and did not during the experiment); We included these subjects as the viewing distance was only 82cm.
Subjects were seated and head placed the chinrest of an EyeLink 1000 in tower-mount configu- ration. Subjects were 82 em from a monitor screen, and their left eye position tracked. Nine-point calibration was performed and validated to within 1 degree at each point. The experiment allowed for fixation within 2 degrees, displaying small dot on the screen for real-time feedback of measured fixation location. Subjects were asked to pause the trial block to re-calibrate if measured fixation did not reflect fixation location, they had difficulty with the system recognizing their fixation.
A.3.3 EXPERIMENTAL PARADIGM
The experiment consisted of 2IFC (two interval forced choice) task where subjects report which out of two images contains a target object. Each subject saw 26 image pairs 10 times at different fixation locations (5, 10, 15, 20%) away from target object, where the fixation location vector computed from the target object location towards the center-point of the image. The order of each presentation randomized across the whole experiment. Fach subject saw the present/absent
the
17

---

Published as a conference paper at ICLR 2024
Moving Average Accuracy (Window 150)
1.0
Accuracy
0.7
0.6
ecc: 20
160
180
200
220
240
260
Trial Number at Eccentricity
Figure 11: Learning Over Experiment. Despite including practice trials before the experiment and excluding correct/incorrect feedback during the experiment, human responses did exhibit a small learning effeet over the course of the experiment. Accuracy is plotted against x-axis ordered by the number of times subject had seen any image at a given eccentricity. Bold lines show moving window averaged accuracy over all subjects, and pale lines show individual subject's data.
image pair 5 times present-first and times absent-first. No correct/incorrect feedback was given to subject.
Subjects maintained fixation on a cross presented at either 5, 10, 15, 20 degrees from the object location, and were eye-tracked to ensure fixation was maintained within degrees. Attention was directed to the object location with latitude/longitude arrows. After presentation, subjects were given the original COCO object category, and prompted to report which image contained the the object by reporting if the object 'appeared" (was in the 2nd image but not the first) "disappeared" (was in the 1st image but not the second).
before proceeding. If the subject broke fixation anytime an image or mask is shown, the trial was aborted and shuffled to the end of experiment. Each image after for 500ms to eliminate visible flicker of appearance of disappearance. Subjects were given response window of 1.5 seconds, and the image pair shuffled to the end of the experiment time-out.
Each trial waited begin until the subject fixated
shown for 400ms, and size-matched pink noise mask shown
The experiment total was 1040 trials long. Subjects were given break every 150 trials, re- calibrating after each break and before starting a new block. Before beginning the experiment, subjects completed practice round consisting of 15 trials of very easy image pairs. 2 subjects needed to do the practice round second time before they reported being comfortable with the task.
The images in the practice round were much larger than those in the actual experiment to make the task easier (10+ degrees). This may have contributed to a learning effect we observed in some subjects where performance improves with the number of trials completed (Figure IT).
A.4 HUMAN CONTROL EXPERIMENT: MEASURING HUMAN PERFORMANCE ON UNIFORM TTM IMAGES
To rule out the hypothesis that DNN under- peleer-peffommnce could be due to TTM throwing away too much information, performed control experiment measuring human detection performance uniform TTM images. Again, all participants were provided informed consent prior to participation. This experiment consisted of 3 naive subjects and 2 expert subjects, 3 males, 2 females.
18

---

Published as a conference paper at ICLR 2024
In this experiment, used the same 21FC task the original image experiment. Instead of asking participants to fixate a set distance away from a target object, participants were able to freely view Uniform TTM images simulating 5, 10, 15, 20 viewing conditions. Images were presented in 22 pixels per degree set-up, similar to the original experiment. To avoid a search task, we cued the location of the target object before each trial. Stimulus, mask, and response timings were the same as the original experiment (400ms, 500ms, 1.5s respectively). Like the original, the experiment was 1040 trials total, consisting of 26 image pairings shown for 4 eccentricities for 10 repetitions. 5 of the repetitions showed object present images first and the other 5 showed object absent first. For these repetitions, we used unique seeds of Uniform TTM images. Participants were given breaks every 150 trials.
A.4.1 HUMAN PSYCHOPHYSICS RESULTS
Accuracy by Eccentriticy and Image for 10 Subjects
1.0
Average Accuracy
0.8
0.6
0.2
15
20
0.0
000000009590
000000009769
000000011197
000000015439
000000018150
000000018380
000000019042
000000061268
000000063602
000000067616
000000119445
000000119514
08600018622
000000203864
0000002095300
000000222299
000000226417
000000255165
000000281000
000000110003
00000322333
000000484351
000000491008
000005092230
000000445000
Figure 12: Per-Image Accuracy over all Subjects. When viewing original images varying
eccentricities, human performance was very image-dependent, but decayed as eccentricity increased for all images
Accuracy by Eccentriticy and Image for Subjects
Average Accuracy
1.0
0.5
000000009590
090000009762
000000011533
000000016439
000000015150
000000018380
000000019042
000000061268
000093063602
000000067616
00000011 19445
0000001 19516
000000186440 86422
000000203864
000000209530
000000222299
000090226417
000000255165
000000281400
0000000000
100000484351
000000 000
000000491008
0000003092338
000000529148
000000545100
Figure 13: Per-Image Accuracy for UniformTTM, images over all Subjects. When viewing uni-
form TTM images, human performance was also image-dependent, and also decayed as eccentricity increased for all images.
Overall, subjects performed well at the task for most images, only reaching chance performance for approximately 30% of the images (Figure 12). Images 000000009769 and 000000067616 were removed from downstream analysis due to poor fitting of psychometric function. The difficulty was extremely image-dependent images were extremely difficult and that they were guessing (though results show they often per- formed better than chance, despite this), while other images extremely easy. The hardest and
subjects reported similar feedback during debriefing that certain
19

---

Published as a conference paper at ICLR 2024
I0A 0.75
present absent
correct
Scora
present
TTM {...}
DNN
absent
Score
DNN
TTM
(...)
Original Images
10 uniform TTM samples
present, absent pair
detection
accuracy
Workflow for Machine Psychophysics Experiment. To simulate trials for detection DNNs, we generate 10 uniform TTM transform images for each present/absent image in the ex- periment. We run inference on the transforms and sum the scores of all box predictions that have an intersection over area (IOA) greater than 0.75%. If the summed score is greater on the present transform, the DNN is recorded correct for that pairing. White boxes indicate the ground truth and padded ground truth. Green boxes show predictions that meet the area condition, red do not.
Figure 14:
casiest reported images tended to be those least and most crowded, and those with the most and least background contrast, respectively. We see similar results for subjects viewing TTM images directly (Figure
A.S MACHINE PSYCHOPHYSICS EXPERIMENT
We provide pseudo-code for the machine psychophysics procedure. We tried a variety detection criteria including, enforcing that predictions match the target category, enforcing the size the box predictions to be no more than half or twice the size of the padded ground truth, and taking the average score over all boxes that overlap the padded ground truth. We arrived at the summing approach described in the Algorithml because it yielded the highest critical j scores and showed similar trends in performance to the other approaches. For some models, there were 1-2 images for which the psychometric fit did not converge, and therefore a value for μ could not be determined.
In addition, potential edge case in DNN psychophysics setup is when the present image prob- ability equals the absent image probability. We investigated how often this happened and found that for the baseline R50-RCNN model, this represents only cases where the prediction probability is zero (no predictions above the 1% threshold) in both the present and absent mongrel pair. Over the image set, it occurs least once over the 100 present/absent pairings, in 10/26 images, and for each eccentricity (80", 11000 22400 (320),, occurs 770,26 times over the 2600 pairings per eccentricity, for a total of (2.7%.11 11..1.1....999 99.2% overall. For the baseline RCNN, we observe that aprob never equals pprob because the model had equivalent predictions in the present and absent images. This only occurs when no objects are predicted in either image.
Making an apples to apples comparison between humans and machines, especially for object detec- tion, is non-trivial; in formulating a psychophysics experiment for these models, which are trained to perform very specific tasks, and report only specific information, design choices inherently affect performance. We took a conservative approach, aiming to give models as much of an chance as possible at performing on-par to their human counterparts. We note here various ways in which we allowed models flexibility in the machine psychophysics experiment:
For the experiment, we chose images with present/absent objects for which models were able detect confidently for the original (0") image.
20

---

Published as a conference paper at ICLR 2024
Algorithm For each object present/absent image in the human experiment, we create 100 pairings of uniform TTM transform images (P and A). We simulate trials by looking at the box predictions (bores) of detection DNN for each pairing (p, We sum the total box scores that overlap with the target object box at least 0.75% IOA (intersection over area). To determine this overlap, we take the target object ground truth box (gt) and pad it with by half pooling region (pr). If the total score for the present image (pprob) is higher than the absent (aprob), we record the DNN model as having correct response. We average over all 100 pairings for final accuracy (acc).
procedure GETMODELACCURAAC
acc 0
initialize accuracy > loop through all present/absent pairings for one object get score for target object
for p. a E A) do pprob GeefTargeetDeteciioonoooo//// gt. pr)
aprob = GelTTangetDerrcciinnSooeee gt. pr)
if pprob uprob then ace acc + 0.5
get per trial accuracy
if pprob > aprob then ace ace +
acc = acc + trials
D take average over all trials
function GETTAARGTTDETECCTONSSCCOEEE
gt. pr)
glx gt + 0.5 size(pr) boxes, scores = DNN(im) prob =
expand ground truth bounding box by half pooling region
& get box proposals initialize total score of overlapping proposals
for b, 8 (bozes, scores) do
ioa b) > 0.75 then prob prob +
D check boxes that overlap expanded ground truth
return prob
return sum of overlapping scores
2. We utilized padded bounding box when evaluating predictions, TTM has location invariance on the order of pooling region width that can move objects transformed images. To compensate for this, we included any predictions with IOA of 20.75. In addition, human subjects were primed with an object location, but were not not restricted within specific region of the image to attend when making the present/absent determination.
3.
We did not enforce models to match labels when performing the present/absent task, allow- ing thee prediction of any object to be counted. Humans may have been able to perform the present/absent task by simply reporting the present image the with more 'ob- jectness', without regard to label. We also found removing the label requirement improved model performance.
4. We measured total probability as the sum of all prediction probabilities that overlapped the bounding box, rather than the maximum. This was chosen for the same reason as (1), in that humans could have used a similar metric. In addition, we found that summed probabilities showed better model performance as opposed to maximum probabilities (Figure
A.6 CONTROL EXPERIMENTS: ORIGINAL TTM, PSUEDO-FOVEATEED, AND CORRELATION WITH HUMAN PERFORMANCE
For our main results, we report DNN performance using uniform TTM images. As control we also run machine psychophysics performance on original TTM (Figures [5] and ggeudo-ooveater (Figure 16). We see similar trends in performance to Figure of the main text. We also look the correlation between the human and DNN performance across each image set in Figures
18 19
21

---

Published as a conference paper at ICLR 2024
Critical Eccentricity
20
(u Fit)
15
10
5
0
Human
All TR FrRCNN
All FT FrRCNN
DINO FocalNet
DINO Swin Tiny
FrRCNN R50
FrRCNN X101
RETINA R50
DETR R50
Figure 15: Critical Eccentricity using original TTM the machine psychophysics.
Critical Eccentricity
20
(4 (itt)
15
10
5
0
Human
All "R FrRCNN
All FT FIRCNN
DINO Swin Tiny
DINO FocalNet
FrRCNN R50
RETINA R50
FrRCNN X101
DETR R50
Figure 16: Critical Eccentricity using pseudo-foveated TTM in the machine psychophysies.
22

---

Published as a conference paper at ICLR 2024
30
DINO Swin Tiny: 0000001
DINO FocalNet: =000001
25
RETINA R50: 2²=0.0
DETR R50: ~²=0.0
Machine Fit
20
FrRCNN X101: r²=0.0
FrRCNN R50: =0=0.01
15
All TR FrRCNN: 00001
AT FRRCNN: r =0.33
10
5
0
10
20
30
40
50
60
Human μ Fit
Figure 17: Correlation between Human and Machine Critical Eccentricity using uniform TTM in the machine psychophysics.
DINO Swin Tiny: 2=0003
20
DINO FocalNet: 2=0000
RETINA R50: 2==000
DETR R50: 2=0001
Machine Fit
FrRCNN X101: 7=0004
FrRCNN R50: =005
All FrRCNN: r -0006
10
All FFRCNN: 2.0
0
10
20
30
40
50
60
Human Ir Fit
Figure 18: Correlation between Human and Machine Critical Eccentricity using original TTM in the machine psychophysics.
23

---

Published as a conference paper at ICLR 2024
DINO Swin Tiny: =.0.0
30
DINO FocalNet: ~²=0.0
RETINA R50: 2==000
25
DETR R50: 02=0.03
Machine μ Fit
FrRCNN X101: =20005
20
FrRCNN R50: =
All' FrRCNN: 2=
15
All" FrRCNN: 2=0.01
10
5
0
10
20
30
40
50
60
Human μ Fit
Figure 19: Correlation between Human and Machine Critical Eccentricity using pseudo- foveated TTM in the machine psychophysics.
24

---

Published as a conference paper at ICLR 2024
Present Experiment Images
Figure 20: Object Present Images. We selected 24 images from the original COCO validation set use in psychophysics experiment measuring human object detection in the periphery.
25

---

Published as a conference paper at ICLR 2024
Absent Experiment Images
Figure 21:
Object Absent Images. We in-painted out target object from 24 original COCO validation set images use in the human object detection experiment.
A.7 FINE- TINE-TNING OBJECT DETECTION MODELS
A.8 TRAINING PROCEDURE
Model
Img
5" UTTM
10 UTTM
15" UTIM
20t UTTM
0' OTFT Ece
100%
FTT FT Ecc
50%
50%
10 FT
Ecc
50%
50%
15a FT Ecc
50%
50%
20 FT Ecc
50%
50%
20%
20%
20%
20%
20%
TI Bec
20%
20%
20%
20%
20%
Table 3: Distribution Original COCO training set images (Img) and varying eccentricity TTM transform images (UTTM) used for fine-tuned Object detection models. Transform images are derived from COCO training set images.
We fine-tuned the Faster R-CNN model from the Detectron2 library Wu et al., 2019) (faster renn r50 fpn) using mixture of original training images, and TTM transforms for varying
26

---

Published as a conference paper at ICLR 2024
eccentricities. In training, selected 55,000 images from each of the COCO and COCO-Periph training sets at each of the 4 eccentricities 15,20,15,20) We train an additional model on eccen- tricities using 55,000 images for each eccentricity (see Table [ for image distributions: the model fine-tuned on 5", for example, is trained on 55,000 COCO images and 55,000 COCO-Periph 5 transform images). All models
trained for 180,000 iterations starting from the weights of pre- trained R-CNN from (Wu et al. 2019). We set the solver to step at 120,000 and 160,000. We 10 A. All other training parameters are the same R-CNN training parameters in (w et al., 2019) as the baseline model.
the base learning rate to 3
We also train one model from scratch on all eccentricities (55,000 images from each eccentricity as well as 55,000 original COCO images). We use the same 3x training schedule provided in (Wu et al.) 2019) for Faster RCNN R50 FPN models (starting from an ImageNet trained ResNet50 backhone, training for 270.000 iters, 16 images per batch). Due to computational limiations, we do not fine- tune or train any transformer-ben based object detection models; we only analyze their baseline models which obtain from the detrex library (detrex contributors 2022). In the main text, report results for fine-tuning on all eccentricities since that model is the top performer.
Original (09)
5
10
15
20
Baseline
person 9016
parson 67%
bruck
Trained
penson 6.
Baseline
person 96%
bed 69%
person person 95%
person
reddy bear
%% Eaddy bear
BDoog 65%
bellie 98%
bottle 97%
teddy
leddy bear
Tediy bear
teady bear
person 93%
person 91% person 97% 67peeson 9US9 person 93%
person DOR
87%
cat 51%
person 669S
Baseline Trained
bettle 98%
bottle 93%
BOOK 57%
book 68%
teddy bear
Reday bear
teddy bear
bear L00%
Uber 100%
Blee S850
bear BO%
bird B5
Binnd R0
bird
Trained
95%
200%
bear 99%
bear 99%
bear
Dird 35
DI 95
Dind 9
Figure 22: Training improves object detection Uniform TTM images. Faster-RCNN R50 FPN trained from scratch on peripherally transformed images retains more stable predictions with more accurate bounding boxes as eccentricity increases when compared the baseline RaNN R50 FPN model.
27

---

Published as a conference paper at ICLR 2024
A.8.1 AVERAGE PRECISION OVER ALL, SMALL, & LARGE BOUNDING BOXES
We evaluate all fine-tuned models on the COCO-Periph validation set and compare detection perfor- mance for small and large bounding box objects. We find that fine-tuning on uniform TTM images improves performance on the COCO-Periph validation set more for large objects than small objects.
Bounding Box Average Precision (AP)
35
Rinttune t
netune 5
Finetune
30
Finetune
Finetuna
25
Finetune All
20
15
0
5
10
15
20
Eccentricity (*)
Figure 23: AP COCO and COCO-Periph validation for COCO-Periph Trained Models. We train and fine-tune Faster-RCNN-RSO-FF model on TTM transformed images from the COCO- Periph train set, and show improvement in AP score COCO-Periph validation, with minimal reduction performance on original COCO validation set. Finetune 0 is trained on original COCO training set, Train All is trained from only the ResNet-50 backbone on COCO-Periph training images.
Bounding BOx Average Precision (APs)
20.0
finetune_ecc
finetune_ acc10
15.0
finetune_ece ecc15
finetune ecc20
12.5
finetune all eces
10.0
5.0
2.5
0.0
Eccentricity (")
Figure 24: Faster R-CNN R50 Object Detection Bounding Box AP for Small Objects. We fine-tune an R-CNN on TTM transform images and compare their AP small to the baseline model. All models generally do poorly on smaller objects. Fine-tuning does not improve performance original COCO images, but it does slightly improve performance small eccentricity COCO- Periph images like 5 and 10
28

---

Published as a conference paper at ICLR 2024
Bounding Box Average Precision (API)
50
baseline
finetune, aacc
45
finetune_ect
finetune, poc10
40
finetune, ecc15
35
finetune all occs
25
20
10
0
5
10 Eccentricity (*)
15
20
Figure 25: Faster R-CNN R50 Object Detection Bounding Box AP for Large Objects. All models generally do better large objects than small objects, and fine-tuning uniform transform images improves large object performance on COCO-Periph images much more than small objects at farther eccentricities.
A.9 ROBUSTNESS TO CORRUPTION IN FINE-TUNED TRAINED FASTER R-CNN
Fine-tuning and training from scratch uniform TTM transform images improves robustness. to some unseen geometric corruptions, but not contrast and noise-like corruptions (Figure 26
gaussion_ nolse
shos_nolse
Impulse_naie nolse
defocus_bluu blur
glass_blur blur
30
PT
4B FT
19
40 27
motion blur
zoom
fog
brightness
contrast
elastic elastic_transfemm
pixelate
jpag_ compresspiinnnn
40 17
W01 7R
A0 F7
a0 7R
A0 F7
a 77
e01 1R
Model
Figure 26: Corruption Robustness Tune-Tuned and Trained models. We report AP averaged over all 5 severity levels for each corruption in COCO-C (Michaelis et al. 2019). BL refers to the pre-trained Faster RCNN R50 model (blue). The FT model is one fine-tined on COCO-Periph (orange). The TR model is trained from scratch on COCO-Periph (green).
29

---

Published as a conference paper at ICLR 2024
A.10
MAXIMUM PROBABILITY AS PREDICTION REDUCES PERFORMANCE
Human FYRCNN R50 AII'TT FT FIRCNN 'TR FIRCNN
Critical Eccentricity
20
Fit)
15
3
10
Figure 27: Sum of Probabilities vs Maximum Probability Using the sum of the prediction proba- bilities (opaque) as the prediction results improved performance on the 2IFC task as compared to using the maximum predicted probability (translucent). This holds for both the baseline, fine-tuned, and trained RCNN models.
30


Reviews:
[{"review_id": "uBKiCWGweX", "reviewer": "Reviewer_ogr4", "summary": "This paper presents a new dataset generated by a modified version of the texture tiling model (TTM) to simulate the information available to human peripheral vision via image synthesis. The paper generates a new dataset, COCO-Periph, which is MS COCO processed through the TTM at different eccentricities. Humans performed a forced-choice object detection experiment, viewing the target object location at various eccentricities. Human accuracy was high well into the periphery (out to the 20 deg tested). Object detection models performing the same task using the TTM synthesised images showed performance drops to chance that occurred at smaller eccentricities than humans. Training on COCO-Periph raised this somewhat, but still not to human performance. The paper also presents some mixed results on noise robustness. The conclusions are that (1) the tested models are insufficient to match human peripheral performance (2) future model architectures should try to close this gap and (3) the COCO-Periph dataset is a valuable benchmark for achieving this.", "questions": "1. Is there something I have overlooked that rules out the alternative explanation for the results proposed above?\n\n2. How often does `if pprob = aprob then acc = acc + 0.5` occur?\n\n3. Is it possible that training on detection makes the model worse at the 2IFC task for some images? e.g. if a single confident object detection yields a lower summed \"probability\" (`pprob` and `aprob`) than multiple less-confident detections that overlap the padded box?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The Pseudo-foveation is a clever way to produce computationally-tractable \"fixated\" images when the individual TTM eccentricities take so long to synthesise. The paper may consider citing Geisler and Perry [refs 1, 2], as they did this for blur.\n\n2. The method to turn an object detection CNN into a 2IFC CNN is clever. It took me some time to understand the pooling of detection probabilities over the bounding box, but this is a nice way to compare the human and machine performance.\n\n3. The 2IFC detection images created with inpainting is clever. I like the manipulation for creating target-absent images in a labelled object dataset.\n\n### References\n\n\n[1] J. S. Perry and W. S. Geisler, “Gaze-contingent real-time simulation of arbitrary visual fields,” in Proc. SPIE, Jun. 2002, pp. 57–69. doi: [https://doi.org/10.1117/12.469554](https://doi.org/10.1117/12.469554).\n\n[2] W. S. Geisler and J. S. Perry, “Real-time simulation of arbitrary visual fields,” in Proceedings of the 2002 symposium on Eye tracking research & applications, ACM, 2002, pp. 83–87. Accessed: Nov. 11, 2014. [http://dl.acm.org/citation.cfm?id=507090](http://dl.acm.org/citation.cfm?id=507090)", "weaknesses": "### An alternative explanation for the results is that the TTM is an inadequate model of human peripheral vision \n\nIf the TTM discards more information than the human visual system in fact preserves, then we would expect this pattern of results: humans have better performance out to larger eccentricities than DNNs trained on COCO-Periph. This is not because of the model architecture, but because the COCO-Periph training set distorts images too much. If this were the case, then closing the gap between human and machine peripheral vision may not be about testing more advanced model architectures, but about finding a better representation for human peripheral vision.\n\nThe paper currently gives the impression that the TTM and related approaches have unequivocal support as a model of human peripheral vision:\n\n> Humans viewing these transformed images perform visual tasks with an accuracy that well predicts human performance while fixating the original images\n\n> This model has been tested to well predict human performance on an extensive number of behavioral tasks, including peripheral object recognition, visual search, and a variety of scene perception tasks\n\n> rendered images that capture the amount of information available to humans at (5◦, 10◦, 15◦, 20◦)\n\n> previous work demonstrated that TTM is a close match to human peripheral vision (Ehinger & Rosenholtz, 2016), so differences in performance cannot be explained by stimuli alone.\n\nHowever, there is other work that would rather seem to support the alternative explanation above [refs 3, 4, 5, 6]. \nUnder some circumstances, humans are more sensitive to image distortions than predicted by these texture models.\nWhile there are undoubtedly tasks for which TTM does correlate well with human performance (e.g. visual search, scene gist recognition, etc), object detection as tested here may not be one of them.\nIndeed, it is generally considered that crowding is about discrimination, not detection. \nIf the present paper had instead used an object discrimination task then the correlation between human and TTM-trained-DNNs might be much higher.\nIn any case, it remains uncertain whether the \"gap in performance between humans and computer vision DNNs in the periphery\" can or should be addressed through the development of more advanced DNN architectures and training on COCO-Periph (e.g. \"In future, it would be interesting to train and evaluate transformer based- architectures.\"). \nRather, the adequacy of the TTM as a model of the human periphery for this specific purpose remains an open question.\n\n### A more comprehensive description of the TTM should be provided\n\nThe paper lacks a comprehensive description of what the Texture Tiling Model (TTM) computes, which hinders a clear understanding of its workings. \nThe reference given for the original TTM is Rosenholtz et al., 2012a (e.g. Figure 8 caption). \nThat paper also does not contain a detailed description of what the TTM actually computes, that would be sufficient to re-implement the model.\nIt is also unclear whether the authors intend to release the TTM code, as the paper mentions the public release of the COCO-Periph dataset \"along with all analysis code\" but doesn't specify whether this includes the TTM.\nFor a more in-depth understanding of the TTM's summary statistics, reference to Portilla and Simoncelli (2000) is necessary, although this reference is absent from the paper and should be included.\n\n### Data analysis and quantification\n\nThe absence of uncertainty indications in the data plots is a noticeable shortcoming. \nAccuracies and psychometric function parameter estimates (Figure 3c, 5, 6, etc) should give indications of uncertainty in the data and parameter estimates (e.g. confidence intervals). \nAre the psychometric function parameters estimated from all human data pooled? \nFor each participant individually? \nOtherwise it's hard to evaluate the precision of the data.\n\nFor instance, Figure 5 shows human criticial eccentricity $\\mu$ values of around 22. \nHowever, if this is a fit to the accuracy data in Figure 3c (as it seems to be) then this value seems implausible: the psychometric function threshold is likely far higher. \nThis point seems to be supported by the fact that the two images that were removed (A. 3.4, Figure 13) appear to show basically no eccentricity effect -- including these images would further support a far higher $\\mu$ value.\n\nMoreover, the step size of TTM eccentricities might be insufficient to offer robust constraints on the model's $\\mu$ values. \nFigure 3c illustrates multiple almost-equally-plausible psychometric curves that could be drawn between the relevant points of DINO FocalNet, potentially altering the $\\mu$ estimates. These issues highlight the importance of disclosing uncertainty in parameter estimates, making it difficult to evaluate the improvement gained from training or fine-tuning on COCO-Periph.\n\n\n### References\n\n\n[3] T. S. A. Wallis, M. Bethge, and F. A. Wichmann, “Testing models of peripheral encoding using metamerism in an oddity paradigm,” Journal of Vision, vol. 16, no. 2, p. 4, Mar. 2016, doi: [https://doi.org/10.1167/16.2.4](https://doi.org/10.1167/16.2.4).\n\n[4] A. Deza, A. Jonnalagadda, and M. P. Eckstein, “Towards metamerism via foveated style transfer,” ICLR 2019. [https://openreview.net/forum?id=BJzbG20cFQ](https://openreview.net/forum?id=BJzbG20cFQ)\n\n[5] T. S. A. Wallis, C. M. Funke, A. S. Ecker, L. A. Gatys, F. A. Wichmann, and M. Bethge, “Image content is more important than Bouma’s Law for scene metamers,” eLife, vol. 8, p. e42512, Apr. 2019, doi: [https://doi.org/10.7554/eLife.42512](https://doi.org/10.7554/eLife.42512).\n\n[6] W. F. Broderick, G. Rufo, J. Winawer, and E. P. Simoncelli, “Foveated metamers of the early visual system”. bioRxiv. doi: [https://doi.org/10.1101/2023.05.18.541306](https://doi.org/10.1101/2023.05.18.541306)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NEuWxNOgeK", "reviewer": "Reviewer_xwTZ", "summary": "The paper introduces COCO-periph, a novel benchmark dataset tailored for advancing human-centric computer vision, building upon the well known  COCO dataset. This dataset is introduced to evaluate Convolutional Neural Networks' (CNNs) efficacy in recognizing peripheral objects. To create it they  utilize the Texture Tiling Model (TTM) to emulate the degradation of information with increasing object eccentricity.  CNNs’ considerably underperfom in this task, furthermore the behavior of models and humans is widely different. For instance,  in humans the drop in performance is gradual as the eccentricity augments, in contrast, CNNs drop sharpely once certain threshold is passed.", "questions": "I appreciate the authors' innovative approach with the COCO-periph dataset and the incorporation of the Texture Tiling Model (TTM) to simulate information loss in peripheral object recognition in the COCO dataset. However, I would like to request further elaboration on the decision not to present the TTM-generated images directly to human participants. My concern revolves around potential artifacts that could have been introduced during the experiments, possibly influencing the outcomes.\n\nWhile previous results seem to corroborate a substantial match with human performance, it's crucial to note that the stimuli utilized in their study differ in certain aspects from those in previous research, such as the grayscale images presented in Ehinger & Rosenholtz (2016). Additionally, there seems to be a noticeable discrepancy in the texture quality of the images.\n\nAs well, although data-based solutions seems to shorten the distance  between CNNs and human performance,  I am eager to learn about the authors' insights and opinions regarding the fundamental mechanisms absent in CNNs but present in humans, which might account for these observed disparities. What specific aspects of human cognition and perception do the authors believe are not adequately captured by current CNN models, and how might future research bridge this gap to enhance the performance of computer vision systems?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper proposes an interesting  benchmark that can bridge the gap or at least understand the limitations of CNNs.  I think it constitutes an excellent example of human-centered computer vision and its application in both understanding mechanisms to carry on this computations in humans and how models can increase performance if they are crafted with these effects in mind. The paper reads very well, and the authors provide a clear insight into what is the contribution of the paper.", "weaknesses": "I missed some references, that seems to be central for this problem, for instance (Zhang & Han ,2019) (Han, 2020). Perhaps linked to this the main missing point in the paper in my opinion was some baseline model that incorporates some of the mechanisms  or hypothesis from human vision that aim to solve this problem ( like EEN).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZQkO3bJHYN", "reviewer": "Reviewer_eW1M", "summary": "The paper explores the performance of deep neural networks when constrained by human peripheral vision, using a modified TTM model and proposes a new dataset called COCO-Periph. A psychophysics experiment with humans is conducted, aiming to highlight difference between human and artificial models in periphery-sensible tasks.", "questions": "- Could you clarify the specific contributions of the COCO-Periph dataset, as it appears to have limited value in the context of the study? - The results in Table 1 show that the fine-tuned model still underperforms compared to DINO. Can you provide more insights into why this performance gap exists and whether there were any unexpected challenges in fine-tuning?\n- Could you explain the reasoning behind fig. 5 and why a direct comparison of model performance with DINO in absolute values was not included?\n- Can you provide more context on potential real-world applications or implications of this results?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "- The paper aims at investigating a very relevant topic, somehow underexplored in the past, when models of foveated vision normally only take into account loss of acuity.\n- The paper introduces an efficient variation of TTM.", "weaknesses": "- The paper introduces a new dataset called COCO-Periph. However, it is unclear how this dataset significantly contributes to the existing knowledge. Given that the primary focus is on modeling peripheral vision in DNNs, the dataset itself does not seem to offer substantial value to the research community. A better approach would be to release the code for the community to experiment with, rather than creating a dataset that doesn't seem to add much to the field.\n- If we interpret peripheral image transformation via TTM as a form of image perturbation, results are unsurprising. The observation that models perform poorly on shifted domains (in this case, TTM generated images) and become more robust after training on the new domain (COCO-Periph) is not surprising. The results are, in essence, not only predictable but do not provide an insight on periphearl vision. To be clear: the exact same results could have be obtain with ANY image transformation resulting in a domain shift.\n- The paper fails to provide a clear explanation for why the proposed model, even after fine-tuning on the entire dataset, still performs significantly worse than DINO. This raises questions about the efficacy of the proposed model and whether it addresses the problem of peripheral vision adequately.\n- Figure 5, which assesses the model's resilience, is presented misleadingly. It only demonstrates the model's performance across various eccentricities, which is expected, given that the model was trained on the same data. Still, as we see from Table 1, the proposed finetuned approach performs worse than DINO. It would be interesting to see absolute performance on 20 degs, for DINO and the proposed model (I imagine, that would not show any advantage)\n- The paper includes an experiment comparing humans presented with objects at different eccentricities with a model presented with a uniform peripheral transformation obtained with TTM. This comparison is not fair. This essentially tests the model on out-of-distribution data, making the poor performance of the models obvious. In no way I see a support for any conclusions regarding properties of the peripheral vision. \n- The paper lacks a clear explanation of the practical importance of a peripheral vision model. The only advantage seems to be an increase in robustness for very related image transformations...", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)