Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces a fixation-based method based on a hybrid transformer-CNN architecture, which demonstrates many of the strengths of sequential processing, namely computational efficiency and adversarial robustness. This model largely outperforms a comparative full resolution model, and has a number of attractive behavioural qualities.
main_review: Strengths:

- Overall, I found this paper both an interesting read and quite a novel approach. It is great to see practical sequential models being developed, and the inclusion of robustness evaluation via adversarial attacks was very helpful in bolstering the paper's claims.

- I appreciate the effort made to make the model comparisons more fair (i.e. based on a similar number of parameters). 

- The components of the model, training steps, and experiments were clearly described.

Weaknesses and Questions:

- Why restrict the adversarial attacks to only the transformer component? We know adversarial attacks affect CNNs, too, so I was unclear on the rationale for this design choice. Also, since Deit-Ti is a purely transformer-based method, I found it unclear if this meant that the comparison of the adversarial attack on Deit-Ti against the two hybrid methods proposed in the paper was fully equivalent.

- What is Deit-Ti (distilled)? I could not find an explanation for this model in the text, and it only appears in Table 2.

- Why is the Dynamic Stop version of the foveated model not included in the adversarial robustness investigation?

- In Table 1, one column uses "PC" as a column head, but I couldn't find an explanation of what this stood for. It is possible I missed it, though. I am assuming from context it is performance.

- In Section 3.2, I found it somewhat unclear to follow. Are there two 50% decay mechanisms (1. the attention weights of the previous confidence map, 2. the IOR map of previously fixated locations)?

- The line directly below equation (2) has a typo: "fixatiothe n"

- In Section 2: Sequential processing, the way the three main advantages are interspersed with the rest of the paragraph text is a little hard to follow. Also, an additional advantage of sequential processing with eye movements is articulated by Tsotsos (2011) A Computational Perspective on Visual Attention, which is to compensate for the "Boundary Problem" introduced by the loss of a convolution kernel half-width of fully defined output for each layer of convolutional filtering. The relevance of this issue to CNNs was demonstrated by Alsallakh et al., Mind the Pad - CNNs Can Develop Blind Spots, ICLR 2021.

- In Section 2: Computational models of categorization and eye movements, it is mentioned that saliency-based models do not incorporate foveal vision, but Wloka et al., Active Fixation Control to Predict Saccade Sequences, CVPR 2018 provides a foveated model of eye movements based on an underlying saliency map.
summary_of_the_review: This is an overall strong paper. It proposes an interesting new approach, and does a good job of exploring the behaviour of this approach with respect to alternative architectures. The weaknesses identified are largely minor, points of clarity, or easily fixable.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a called Foveater which uses a foveated module to extract the information from the feature map with different levels of details and different locations. This proposed method has the architecture that makes sense for the image classification task. However, I found it is difficult to find significant technical novelty in this paper.
main_review: [Paper strength]
- The method is well-motivated. The fovea in computer vision can save the computation, improve the robustness of the prediction, and improve the model performance.

- The proposed method achieves better performance than baselines in the adversarial robustness experiments. The author shows the results for the robustness experiments in table 2 that the proposed method achieves better performance than baselines. The author then shows the contributions of the foveated natural of the proposed model in figure 4 for the adversarial robustness.

- The proposed architecture make sense to me. The proposed method uses the foveation module to extract coarse and fine information from the feature map, and then change the fovea locations with the fixation sequence prediction, and transformer in the middle to extract feature with attention. Essentially, there are three components of attention mechanisms: foveation module, sequential fixation, and transformer. 

- The author shows the proposed method can run faster than other baselines methods while maintaining a similar performance in terms of the accuracy of the classification tasks.

- The author provides detailed information about the proposed network architecture that is possible for re-implementation.

[Paper weakness]
- The main weakness of this paper is the lack of technical novelty. Compared with the previous DeiT-Tiny, the main contribution of this paper is the foveation module. However, the foveation module is very much like the dilated convolutions [a] and the pyramid of dilated convolutions [b], except here the filter/kernel is average pooling. If the author wants to claim novelty over the family of dilated convolutions, they should provide more discussions, analysis and experiments to show the advantage of the proposed method. Current, I cannot judge from the paper. 
The sequential fixation is the recurrent CNN, and there is no other technical contribution as far as I can see.
Essentially, the proposed method is a combination of recurrent CNN, transformer, and dilated convolutions. If the author would like to claim this combination itself is the contribution, there should be more ablation experiments on the contribution of each component.

    [a] Multi-scale context aggregation by dilated convolutions. ICLR 2016.

    [b] ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation. ECCV 2018

- It is not clear to me how to train the sequential fixations. Since the fixation point is a spatial location, directly retrieving it from the attention weights map in the paper is not differentiable. I am confused about how the author does the backpropagation, i.e. how to judge one fixation position is better than another one? As far as I know, this is hard attention implemented by recurrent neural networks output the location, which usually can be solved with spatial transformer[c], LSTM, or other reinforcement learning loss. The task itself is not easy to train.

    [c] Spatial transformer networks. NeurIPS 2015.

- There is no significant performance improvement. As shown in Tables 1 and 2, there is no improvement from the proposed method over other baselines in terms of the accuracy of the image classification task. Although the proposed method can run faster than baselines, I do not think the running speed performance is superior since it is actually slower than the ResNet-18 in table 2.

- The initial fixation should start at the centre of the image. I can see from figure 4 that a random initial fixation point is helpful for the proposed method since it can have multiple choices to choose the ideal locations. However, it is unfair for the full-resolution baseline to have a random initial location since it cannot achieve good performance if the random location is not ideal. 

- It is not clear to me how much of the class token could be helpful for the final performance, or does it matter at all?

- Equation 2 seems like the loss is averaged after the classification layer, which is a conflict with figure 1 that the averaging happens before the classification layer. Please ignore this comment if I am wrong.
summary_of_the_review: I agree with the author that the foveation module could be helpful for the computer vision task. However, it is hard for me to see the technical novelty of the proposed method. Therefore, I lean to reject this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper shows a method of Foveated Transformer motivated by a human foveal vision where the spatial resolution varies depending on the focused point. Namely, the method mainly consists of the spatial attention model that controls the gaze point and spatial-varying convolutional filter (dense pooling in center and sparse pooling in peripheral). The former simulates human gaze control and the latter mimics the human retina. The experimental validation is performed by the object recognition task (ImageNet) and adversarial attack tasks and performed better than the Deit-Tiny model that uses a similar number of parameters.
main_review: The method presents a unique idea of computer vision tasks based on the human perceptional system. Since humans archive vision processings with limited computing resources, employing the idea may work for computing tasks with the limited resource, thus, the idea proposed in this paper may give an impact on the community. However, I still wonder about the effectiveness due to the lack of comparison with existing methods.

- Performance of object recognition:
The performance of ImageNet task is far from SOTA. I understand the main claim of the paper is introducing a new scheme that increases the performance of the existing algorithm, but if so, the question is why the paper used Deit-Tiny for only the reference. It is highly desired to combine other methods as well as the proposed foveated scheme.

- Attention model:
Discussion (and hopefully experimental validation) need about the difference between the attention model and the proposed gaze control. 


- Foveated transformer:
I wonder which is better between the proposed foveated transformer v.s. dilated convolution since they have similar computational models. Rather, dilated convolution is much similar to the human vision for its gradient-density nature. 
summary_of_the_review: The method presents an interesting and unique approach that is motivated by human vision perception system including gaze and attention control as well as the foveated vision. However, experimental validation is limited, and thus difficult to understand the pure effectiveness. I think other application scenarios such as real-time video recognition may be suit for the scheme than the object recognition tasks for its low-cost computing nature.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors present an approach to computer vision combining Transformer networks and foveated vision. They contrast their approach, which uses a recurrent attention model to simulate eye saccades to explore a scene, against full resolution deep CNN models for classification problems to demonstrate reduced computation and greater resistance to adversarial attacks.
main_review: While recent success in convolutional processing has dominated the field, certain aspects of vision common to many biological vision systems have had limited treatment. The proposed work is interesting in this respect, as it combines a foveated vision model within the popular transformer model framework. It demonstrates a model for active vision with predicted eye movements using recurrent network to predict the next area of foveated attention as a simulated eye movement. 

The introduction provides motivation for foveated vision and review of prior art in this area. However, the most relevant works are not cited and the paper falsely claims to be the first for combining transformer networks and foveated vision: "No papers have evaluated the additional potential gains of incorporating a foveated architecture into transfomers." The authors should correct this claim, and take note of the following references to explain what is unique in their approach and how it differs from other foveated vision models:

* Harris, Ethan William Albert, Mahesan Niranjan, and Jonathon Hare. "Foveated convolutions: improving spatial transformer networks by modelling the retina." (2019).
* Dabane, Ghassan, Laurent Perrinet, and Emmanuel Daucé. "What You See Is What You Transform: Foveated Spatial Transformers as a bio-inspired attention mechanism." (2021).

The authors reference 2017 work of Akbas and Eckstein as another foveated object detector, but do not provide distinctives that set their model apart from the work. The current paper would benefit from direct comparison to these other foveated vision methods. The only comparisons to other methods are against full-resolution models.

It is unclear from the introduction why adversarial attacks are a focus of this paper. The motivation of the paper starts on the basis of biologically inspired vision systems, yet such systems do not have to address adversarial attack other than overcoming natural camouflage (which is not at all similar to the types of adversarial attacks compared in the paper involving gradient informed adversarial examples), and in any case there is not explanation of why adversarial robustness should be a feature of foveated vision, or why to evaluate foveated vision systems on this basis.

Similarly, while the connection between transformer networks and foveated vision systems seems more supportable, it is lacking. There is an obvious opportunity and need to provide some description of transformer networks and why they are particularly well suited a framework to support the foveated vision system the authors wish to demonstrate. 

Eye movements seem limited by a 14x14 grid. Explanation of the foveation pooling and aggregation approach was a little confusing. Is the movement restricted to an inner region, or is the zero-padding extend to potentially half the visual field when the attention is placed near the edge of the image. It would be interesting to visualize the visual field for a given fixation. Given the biological motivation, I would be interesting to know how this compares to biological systems. 

Classification results seem a little underwhelming. While it barely beats ResNet-18, 70.2% vs 77.1% against EfficientNet-B0 is disappointing. Again, without direct comparisons to other foveated vision systems it is difficult to tell how much this improves (if at all) on current art.

I was particularly intrigued by what could be learned from the saccades themselves, but relatively little was discussed in that regard. The examples in Fig 3 were helpful to see but limited to a few hand-picked examples, and difficult to know how well movements relate to object content. It would be informative to compare eye movements predicted by the model to biological movements in gaze-tracking datasets. 
Such comparisons could support the biological motivation of the paper and show that the model is learning beneficial movements. Comparing the only to full resolution does not sufficiently answer this. Another comparison to illustrate this would be to compare the existing model against random movements. 

I think thresholding the classification probability is a good idea to determine a stopping point for the dynamic stopping of fixation exploration, and the average fixations per class is interesting but I would like to understand if this correlates in some way to actual object complexity. The conclusion claims the learned model can allocate computational resources based on the difficulty of an image, but I did not see any comparisons that defended this. Can you show that more fixations were used by the model for images that were more complex by some entropy or other complexity metric? Or compare to human gaze experiments?

Other specific comments:
Pg. 2: "reach that approaches" -> "approach"
Pg. 2: "using which locations far apart in the image will interact" -> this sentence is unclear as to its meaning, but needs revision
Pg. 4: "squared pooling regions...for computational speed-up": what kind of speedup is obtained, and by how much, and compared to what?
Pg. 6: description of dynamic-stop of fixation exploration: Is this only for inference, or is dynamic stopping performed during training?
Figure 4: Requires color to interpret. It may be helpful to use different marker and line styles in case this is viewed in black and white




summary_of_the_review: This is an interesting area of research, but I do not think that the authors did a sufficient job of motivating the model and experiments. They did not motivate the use of transformer networks, or explain why adversarial attacks are an appropriate problem for comparing foveated vision systems. They also left some important questions answered or not fully answered (such as whether the fixation movements were actually benefiting the classification). Experiments provided do not do direct comparison with equivalent models with and without the predictive attention which could easily determine the contribution for this part of the model. Claims in the paper are not sufficiently supported. A priority claim made in the paper as the first work to combine foveated vision and transformer networks is false and ignores published work by Harris et al. 2019 that uses spatial transformer networks for foveated convolutions. Direct comparisons to other foveated vision systems are lacking, and without these it is not possible to assess how well their particular approach compares to the most similar prior art.
review_title: null

Meta-review: This paper introduces an architecture that uses pooling regions and
eye movements to sequentially build up an object representation.  A 
confidence threshold is used to allow recognition in less time for
easier images.

There was a lot of disagreement on this paper.  Those in favor argued 
that it is a worthy endeavor to explore new biologically motivated
architectures and foveated eye movements are an important aspect of  
human vision that is worth exploring for computer vision.  Another pro
was the improved robustness to some adversarial attacks.  Those
arguing for not accepting the paper, argued that classification
performance is not improved over SOTA and that more ablation studies 
should be done to better understand the role and importance of the
various aspects of the model and how they differ from other
architectural designs with dilated convolutions instead of the 
foveation module.
 
I agree that more ablation studies would be useful to better
understand the role of the different model components. While I 
feel that this novel sequential processing algorithm is worth publishing to
increase activity in this area, I feel it would be best received after further 
studies help clarify the importance of different aspects of the model.
I recommend resubmission after further analysis.
Decision: Reject