Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper tackles the cross-view geo-localization problem, which seeks to match a given ground-level image to aerial images that cover the same area, selected from a fixed database.

The paper is the first to apply transformers to the problem, and reports state-of-the-art results. Notably, the transformer’s attention mechanism largely frees them from having to warp the aerial image to match the ground-level image’s panoramic projection, a procedure that can go awry in the face of significant occlusion in the ground-level panorama, and/or errors in the ground pano’s orientation data. The resulting accuracy improves the state of the art by a significant margin.

The paper also makes a few adjustments to the ViT transformer, performing ablation studies on some of them.
main_review: # Originality
The paper is the first to apply the transformer attention architecture to this problem. In addition, it modifies the ViT (Vision Transformer) [4] in a few ways: it replaces image patches with single-pixel columns through the feature maps, which strikes me as an improvement in elegance. It also computes the keys of layer i from the values of layer i-1. Finally, it introduces a ResNet-like pass-through connection through the transformer stack. The paper presents ablation studies of the effects of the first two changes above.

# Quality
The submission is technically sound, and analyses presented show good evidence of its accuracy advantage over its predecessors. The ablation studies are useful and well presented. 

There are some claims that are unsupported by evidence. Notably, the explanation in lines 180-198, of how self-cross attention affects feature learning, seems weak. 

Lines 181-185 mention an architectural similarity to ResNet. In the paragraph below labeled “Clarity”, I talk about how the wiring of Fig 1b is messy. As it stands, the figure obscures any parallel with ResNet. Once fixed, this parallel should be more visually obvious. 

In lines 189-191: “As shown, during the early training stage, the localization performance of the self-attention-based model fluctuates a lot.” It isn’t obvious that the early stage fluctuations of the red curve are significantly worse than those of the black curve. Could you quantify this, e.g. by computing a variance envelope over a moving time window? In any case, the final performance gap between the red and black curves could be attributable to many many things, and it’s not at all obvious why the early stability advantage of the black training curve (assuming there is one)  should be singled out as an explanation. Please remove the statement “As a result, the stable training of the EgoTR could improve the network performance.” Without direct evidence, this is purely speculative. In any case, even if the early stability did lead to better final performance, this only replaces one question (“what causes the better performance”) with a speculative statement (“stable training causes better performance”) tacked onto another question (“what causes the stability?”). That question is just as open as the original question.

“Compared to the self-attention in Eq. 2, our proposed self-cross attention creates short path between adjacent layers,”

# Clarity 
There are a number of places where the clarity could be improved:

## Abstract

“Evolving”:
I found the paper’s use of the term “Evolving” to be confusing, in the sense that it led me astray from what is actually being described. In a field where evolution mechanisms are closely studied as a part of the learning toolbox, I would stay away from using “evolve” to describe anything that doesn’t involve Darwinian (or even Lamarckian!) improvement. Even when taking the more relaxed view that “evolve” can refer to any sequential improvement process, it seems misapplied to the EvoTR. While the keys of layer i are a function of the features of layer i-1, It’s hard to say that the keys are “improved” from one layer to the next. Each set of keys are specific to that layer’s features, and comparing them across layers would be comparing apples to oranges. In that case, what is being “evolved”, i.e. sequentially improved over iterations? 

I recommend renaming EvoTR to replace “evolve” with something closer to what is being described, for example, L2LT (Layer-to-layer Transformer).

## Section 3.2
The wiring in Fig. 1b is messy. At the bottom, x_{i-1} goes into Layer Norm, the output of which should be z_l according to Eq 3. This z_l is unnecessarily merged with the wire labeled x_{l-2} (did you mean for that to be z_{l-1}?). These wires should be kept separate: the z_l wire should go to the Q and V blocks, while the z_{l-1} wire should go to the K block. Most importantly, under the present figure it’s unclear where the x_{i-1} output at the top of the figure comes from. It can’t be the output of the Layer Norm; that’s z_l. It’s not the x_{l-2} input at the bottom; the index is different. Please fix the labels if necessary, and when two wires merge, make sure they only do so by going into a labeled block (e.g. “Add”).

In “Preliminaries: Vision Transformer”, please explain what “class” means, in the context of the cross-view geo-localization problem. Most readers will be familiar with “class” as used in image classification, so it is unusual to see “class” be part of the input data, as opposed to being a target output.

As you describe the ViT here, please be explicit about which parts correspond to the different dashed boxes in Fig 1a (even understanding that Fig 1a contains a modified ViT, rather than the vanilla ViT being described here).

In “Learnable positional embedding”, please describe why the positional embedding is added to the feature maps, rather than concatenated as additional channels. It’s a strikingly surprising design choice, at least to me.

Please separate Equations 2 and 3 into separate numbered equations. If you must keep the equations for Q_l, K_l, and V_l on a single line, please add more whitespace after the commas, to keep them visually separate.

## Section 4.1
Please provide more details on the difference between CVACT vs CVACT_test. It sounds like the former provides just pairs of ground images with their corresponding aerial images. Is this correct? CVACT sounds the same, except that both images now have “geo-tags”, which I assume are like GPS coordinates? Is this correct? If so, How is this extra information used, during inference, training, and evaluation? Also, why is the test set (CVACT_test) ~3x the size of the CVACT training set?

## Section 4.2
Consider moving section 4.2 (“implementation detail”) to the beginning of section 4 (as 4.1).

Please provide more detail on the pretraining. In particular:
* Did you train a single encoder to classify ImageNet images, then transfer those same trained weights to both the aerial and ground encoder networks? 
* Did you augment ImageNet images with positional information? 
* How does the network do without pre-training?

## Page 8
Most of the page makes reference to Table 4. Please try to move that figure to this page.

## Section 4.5
Please go into more detail on how to interpret the distance matrices of figure 4. In distance image D, is D[i, j] the dot product of the positional embeddings of pixel V[i] and pixel V[j], where V is the flattened version of the ground/aerial image? Where does the checkerboard pattern come from? Without having this nailed down, it’s hard to evaluate the claims being made in this section, especially the one in lines 375-378. Lines 379-382 need an image illustrating the effect they describe, as it’s hard to visualize.

## Nit
Throughout the paper, there are misused articles, mostly in the form of unnecessary or missing “the'''s. Before submission, please run the paper by a colleague who speaks English natively, or if none are available, through a grammar checker (e.g. Grammarly).

# Significance
The results are significant. The architecture presented is an improvement in accuracy over competing methods. They largely do away with the need to preprocess the aerial image through a panoramic projection, a potentially brittle step that we are well rid of. The claim is that the transformer’s attention mechanism handles the correspondence problem instead. If this is true, this work may also open the door to future work in geolocating planar ground-level images, which are far more common than ground panoramas. Unfortunately, this is not studied or mentioned in this paper.

limitations_and_societal_impact: In section 5, the authors highlight that the main limitation of EgoTR is its large demand on GPU memory. Please quantify this demand and compare it to that of the closest competitors.

Line 391-392 says “Moreover, the EgoTR is built on top of the pretrained Transformer, which requires a large amount of data for training.” Is it the pre-training or the fine-tuning that takes a lot of data? Please quantify the data demands and compare it to that of the closest competitors, for example by comparing the number of epochs it takes to converge.

Can you foresee any limitations that would impede success when using this method to match planar aerial images to planar ground images?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes EgoTR for cross-view geo-localization with evolving transformer. Specifically, it utilizes the properties of self-attention in transformers to model global dependencies, which reduces visual ambiguities in cross-view geo-localization. The positional encoding is also used in the proposed EgoTR to understand and correspond geometric configurations between ground and aerial images. 
main_review: - This paper is well-written and easy to follow.

- The paper claims that the use of positional encoder removes the need for strong assumptions on geometry knowledge required in some existing cross-view geo-localization methods. It is however not very obvious from the paper how this is achieved over the cross-view setting. It seems like the positional encoding mentioned in Line 160-171 is based on the image patches in the respective domains. Thus, shouldn't the positional knowledge be just within the image in the same domain, i.e. self-attention? How would the Siamese network architecture and the Triplet loss help in enhancing the cross-view geometry knowledge via the positional encoder? 

- The individual components of the proposed framework is not new. Transformer is from ViT [4], and the loss function is from [8]. The only new thing is the attention across different layers in the network proposed in Section 3.3. Therefore, this paper can be seen as another additional paper that introduces transformer into a well-established task / framework. 

- What is the explanation on self-attention across different layers can help cross-view localization? Similar to the positional encoding, it is also done within the same view. The only interaction across the two views is the final feature embeddings and the triplet loss. 

- Using transformers mean that the memory consumption might increase a lot compared to other existing methods. 

- The results on the two benchmark datasets show good performances on the proposed method. This seems to empirically suggest that the method that adopted transformer is effective on the task of cross-view geo-localization. 

- The ablation studies in Section 4.4 also seem to empirically support that claim that the positional encoder and self-cross attention across different layers are effective. However, as mentioned earlier, a more concrete explanation on why positional encoder and self-attention across layers can help cross-view geo-localization is missing. 


limitations_and_societal_impact: Limitations are discussed in conclusion, but not societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: a) This paper brings transformer to solve the cross-view geo-localization problem;
b) The main contributions are two folds: one using transformer and the other using a modified attention scheme;
c) Results are good and outperform previous method.


main_review: Strong points:

a) The paper is well-written;

b) Using transformer for cross-view geo-localization is novel;

c) Though modifying the attention equation is engineered, it is effective for this task;

d) Experimental results are good.


Weak points:

a) I would say the claim of the superiority of learned position embeddings is too strong and expect the following experiments.

Randomly orientate the ground-view images and compare the proposed method with respect to [17].

Without this experiment, the proposed method can also benefit from the regularized configuration of aligned cross-view images.


b)  Self-cross attention.

x_{l-1} and x_{l-2} is used  in equation (3), can we use x_{l-3} to replace x_{l-2}?

or, jointly using x_{l-1}, x_{l-2} and x_{l-3}?

x_{l-3} can be used to build to Q_l.


c) Time and model complexity comparisons.

Though the code length [line 300] is short for the proposed method, as the transformer is time-consuming, I would expect detailed time and model complexity comparisons with respect to previous methods.



limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes to use transformer for cross-view image geo-localization. It adopts a cross-layer attention block to further improve the performance. Experiments on CVUSA and CVACT show improvement over existing methods.
main_review: Strengths:
1.	The idea is simple and easy to follow.
2.	The tables are clear.
3.	The performance is higher than existing papers.

Weaknesses:
1.	Transformer has been adopted for lots of NLP and vision tasks, and it is no longer novel in this field. Although the authors made a modification on the transformer, i.e. cross-layer, it does not bring much insight in aspect of machine learning. Besides, in ablation study (table4 and 5), the self-cross attention brings limited improvement (<1%). I don’t think this should be considered as significant improvement. It seems that the main improvements over other methods come from using a naïve transformer instead of adding the proposed modification. 
2.	This work only focuses on a niche task, which is more suitable for CV conference like CVPR rather than machine learning conference. The audience should be more interested in techniques that can work for general tasks, like general image retrieval.
3.	The proposed method uses AdamW with cosine lr for training, while comparing methods only use adam with fixed lr. Directly comparing with their numbers in paper is unfair. It would be better to reproduce their results using the same setting, since most of the recent methods have their code released.

limitations_and_societal_impact: There is no discussion about societal impacts.
review_title: null

Meta-review: This paper proposes a siamese convolutional + vision transformer architecture for cross-view geolocalisation, from ground view equirectangular panorama images to satellite views (cartesian or polar coordinates). The transformer has a cross-attention mechanism between consecutive layers (I was confused by the word “evolving”, since no genetic algorithms were involved). The method is thoroughly evaluated on a US cross-modal localisation dataset where it achieves state-of-the-art results.

Reviewer kVJu praised the idea and significance of results but criticised some speculative or unsubstantiated claims in the paper, as well as the clarity, including the usage of term “evolution”; the authors responded with a long list of modifications and additional results. Reviewer JyiQ was uncertain why positional encoding and cross-layer self-attention specifically helped with the localisation task (the authors’ response was only empiriical but did not provide intuition about why it worked). Reviewer Bnto had questions about ground view image orientations, which were answered by the authors through additional experiments. Reviewer 9M5m made the usual comment about novelty (which I will dismiss) and narrowness of scope (again, that’s an easy critique to make), as well as a claim about learning rate (addressed by the authors).

All scores but for reviewer 9M5m (4) were positive (6, 6, 7). Reviewer 9M5m said they would update their score to 5 (which makes the paper average equal to 6). I have dismissed claims of “novelty” and “narrowness of scope” and decided to accept the paper.

Decision: Accept (Poster)