Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The problem of solving computer vision tasks with neural networks on large images is considered. Authors propose a method that splits an image into patches and then applying greedy iterative approach (IPS iterative patch selection) to select the representative ones. Selection is performed by creating a buffer and iteratively updating it by scoring them. Final buffer is then transformed via a convex combination of embeddings. The approach is more heursitic than theoretically justified. Results are demonstrated on multiple dataset where high resolution matters, the method is compared with multiple others. 
strength_and_weaknesses: Strength:

- The problem is very relevant and considers important problem outside of the main focus. Solving CV tasks at high resolution is challenging and authors provide a working (according to results) solution. 
- Overall novelty of the paper is in 2 components: (i) iterative patch selection and (ii) patch aggregation.
- The idea of using transformer attention for scoring that is learned without gradient is novel and seem to work fine.
- The related work section is quite broad and gives a nice overview for the person not in the field.

Weaknesses/questions:

1) Significance of the method and novelty. The approach seem to be hand designed and heuristic. We don't see much of the theory to back it up. The contribution of (i) iterative patch selection can be considered as a straightforward and not novel. Patch aggregation with the idea of scoring patches is also not groundbreaking. The way patch scores are training is an in interesting point.
2) The explanation of how attention scorer is trained is a bit vague. Paper states multiple times that there is no gradient propagation (no-gradient mode), but then it is not clear how to train the scorer. There is a second network that aggregates patches, but it seem to take patch + positional encoding meaning previous scores are not needed. If both blocks share the same attention then we can train only the second one, and use attention scores for the selector. A more detailed explanation of the novel part is needed. 
3) The implementation uses a relatively heavy network resnet50 to embed patches. This network by itself can potentially solve the problem if information from all patches is aggregated by some way of pooling (max for example). Probably learning an embedder will be useful for the work. 


clarity,_quality,_novelty_and_reproducibility: Clarity - can be improved, specifically novel parts such as learning a scorer and what is fed to the task transformer.
Quality - paper is well written.
Originality of the work - potentially low as the approach is a hand-designed system with little theoretical justification. 
summary_of_the_review: Paper tackles interesting problem of working with high resolution images. The solution is a system that selects informative patches and processes them with transformer. The novelty of the method is on the weak side and some components that can be considered interesting to the community are not well explained. Given little novelty and no significant theoretical justification of the method my recommendation is top reject the paper. 


------Post rebuttal------
Thank authors for addressing raised points. The way attention module is trained without gradients is more clear now and updates to the figure are great. Reading other reviews it seems they are more positive. On my end, paper has only empirical contribution is it is not groundbreaking, however, given other reviews I will increase my score to 6 and will not object if the paper is accepted.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes  a simple method, Iterative Patch Selection (IPS), which can select the image patches from a high-resolution image and meet the memory constraint. Two-stage DNN is used to produce the final result, the first part of the DNN is used to select the input patches in autoregressive fashion. The selected patches are then processed by the second part of the DNN to produce the final results. The results indicate that ISP can lead to superior accuracy with minimum level of memory footprint. 
strength_and_weaknesses: Strong point:
+ Image patch searching for high-resolution image is an interesting and practical problem.
+ The proposed solution is technically sound.
+ The evaluation results are comprehensive.

Weak points:
- Compared with "Differentiable Patch Selection (DPS)" which also use transformer based image patch selection, the contribution is not that significant.
- The no-gradient mode is confusing to me. Why can't you use the encoder transformer to compute the scores for all the patches, then use some approximation function (e.g., softmax) to estimate the top-k function. This will make the whole DNN differentiable.
- Not sure whether the proposed architecture will bring additional hardware constraint due to the storage of the intermediate results within the DNN. The author should justify that.
clarity,_quality,_novelty_and_reproducibility: The paper is presented in a clear way. Figures are very easy and helpful. The novelty is limited. Code is not provided, so reproducibility is a question mark.
summary_of_the_review: Image patch selection for large-scale image is an interesting and practical problem. The paper adopts the transformer based DNN to perform both image patch selection and classification. However, I am a bit concerned about the novelty of the work and the reproducibility of the work, given that no code has been submitted.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new method of multiple-instance learning from high resolution images, which uses a patch selection scheme that processes available patches in mini-batches, scores them, then aggregates the top-M highest-scoring patches’ embeddings to reach a bag/image-level prediction. The proposed IPS Transformer out-performs state-of-the-art methods on 3 different high resolution image benchmarks. It is shown that IPS Transformer can be less sensitive to the choice of pool size (M), use less memory, while performing similarly to existing methods.
strength_and_weaknesses: The manuscript is written quite well and the related work is handled particularly well. The experimental settings are clear and seem to be fair, with many related works being reproduced by the authors themselves. The IPS Transformer clearly performs well while being less sensitive to the choice of M, uses lower GPU RAM, and has competitive inference times.

Though the proposed algorithm is not excessively complex, the concrete steps of how to train IPS Transformer is scattered in the text of Sec. 2. It is therefore difficult to clearly understand the steps involved and which modules are trained when. The paper may benefit from either an algorithm figure in the main paper or a pipeline/workflow diagram figure.

The experimental results and ablation studies look complete.
clarity,_quality,_novelty_and_reproducibility: The manuscript is easy to read and understand and is of high quality.

The proposed method is novel in its clever re-use of the multi-head cross-attention.

The algo figure in Appendix 1 as well as other details allows for one to reproduce the approach.
summary_of_the_review: The paper presents a practical yet simple solution to MIL on high-resolution images and experimentally validates its performance on 3 benchmarks. The presentation and analysis are of high quality, warranting the paper’s acceptance.

Edit after rebuttal: I did not previously have major concerns regarding this submission, and after carefully reading the other reviews as well as the authors' diligent response and action, I maintain my rating of 8 as no further concerns exist from my side.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors deal with the problem of training neural networks when the input images are very large and thus cannot fit entirely into GPU memory (ie, cases where even a batch size of 1 is too large). They propose an Iterative Patch Selection algorithm which selects only the most salient patches which are then aggregated into a global representation for image recognition. A transformer is used for both patch selection and aggregation. Their experiments are able to process >16 gigapixel images on a 5GB GPU.
strength_and_weaknesses: Strengths:

S1: The paper addresses an important problem that plagues graduate students and researchers who may not have access to the latest and most expensive clusters of GPUs, namely that of being able to do vision under limited resources. It should therefore be of interest to the ICLR community.

S2: the paper is well written and is up to ICLR standards. However as I indicate below, in certain places the authors should move some of the details from the appendix into the main text.


Weaknesses:

W1: Fig 1: it is unclear how the patch scores are shown in this figure

W2: under Eq.1, X_1,...,X_M is not well defined.

W3: page 2, last paragraph: the relevant information is really in the appendix. I suggest the authors add more details here. This is currently too high level.

W4: under Eq 3, the variables just under Eq 3 need to be discussed in more detail

W5: in the paper's prior work, the authors should make a better connection to early approaches for simulating saccades in vision and attention beams. Why do humans have attention? 

W6: Appendix A pseudocode: this is not pseudocode, it is actual code. I suggest edits to make it more readable.
clarity,_quality,_novelty_and_reproducibility: Overall the paper is up to ICLR standards. There are some clarity issues, which I highlight above, that could improve the paper's readability.
summary_of_the_review: Overall I think this paper should be of interest to the ICLR community as it helps enable big scale deep learning under limited resources. The main weakness I see is that the authors need to do a better job positioning the paper with respect to early work on attention.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a strategy for training deep networks on very large input images by selecting only salient parts of those images and training on those parts only. This strategy substantially reduces how much memory is required for training, hence solving an important practical obstacle for the use of deep networks on high-resolution images. The paper presents experiments on high-resolution traffic images and medical images, validating the efficacy of the approach relative to prior work.

Some reviewers question the paper’s novelty because the strategy presented is relatively simple. The AC disagrees with this view: a simple method that works well is much more likely to be widely adopted than a complex method.

The manuscript’s presentation of the IPS method is confusing at times, as pointed out by multiple reviewers. The authors are encouraged to improve the presentation in the camera-ready version of the paper.
justification_for_why_not_higher_score: The paper presents a simple and practical method that will help researchers train models on high-resolution images. At present, use of high-resolution images for training is a bit of a niche use case though so I believe this work is best presented to a targeted audience interested in that niche use case via a poster presentation.
justification_for_why_not_lower_score: The paper presents a simple and practical method that will help researchers train models on high-resolution images. The method will surely be of interest to (part of) the ICLR audience.
Decision: Accept: poster