CATs: Cost Aggregation Transformers for Visual Correspondence
Authors: Seokju Cho
Sunghwan Hong
Sangryul Jeon
Yunsung Lee
Kwanghoon Sohn
Seungryong Kim
Venue: NeurIPS 2021 Poster (2021)
Keywords: Semantic correspondence
cost aggregation
Transformers
Visual correspondence
Dense correspondence

Abstract: We propose a novel cost aggregation network, called Cost Aggregation Transformers (CATs), to find dense correspondences between semantically similar images with additional challenges posed by large intra-class appearance and geometric variations. Cost aggregation is a highly important process in matching tasks, which the matching accuracy depends on the quality of its output. Compared to hand-crafted or CNN-based methods addressing the cost aggregation, in that either lacks robustness to severe deformations or inherit the limitation of CNNs that fail to discriminate incorrect matches due to limited receptive fields, CATs explore global consensus among initial correlation map with the help of some architectural designs that allow us to fully leverage self-attention mechanism. Specifically, we include appearance affinity modeling to aid the cost aggregation process in order to disambiguate the noisy initial correlation maps and propose multi-level aggregation to efficiently capture different semantics from hierarchical feature representations. We then combine with swapping self-attention technique and residual connections not only to enforce consistent matching, but also to ease the learning process, which we find that these result in an apparent performance boost. We conduct experiments to demonstrate the effectiveness of the proposed model over the latest methods and provide extensive ablation studies. Code and trained models are available at https://sunghwanhong.github.io/CATs/.

CATs: Cost A Aggregation Transformers for Visual Correspondence
Seokju Cho" Yonsei University
Sunghwan Hong" Korea University
Sangryul Jeon Yonsei University
Yunsung Lee Korea University
Kwanghoon Sohn Yonsei University
Seungryong Kim
Korea University
Abstract
We propose novel cost aggregation network, called Cost Aggregation Transform- ers (CATs), to find dense correspondences between semantically similar images with additional challenges posed by large intra-class appearance and geometric variations. Cost aggregation is highly important process in matching tasks, which the matching accuracy depends on the quality of its output. Compared to hand- crafted or CNN-based methods addressing the cost aggregation, in that either lacks robustness to severe deformations or inherit the limitation CNNs that fail to discriminate incorrect matches due to limited receptive fields, CATs explore global consensus among initial correlation map with the help of some architectural de- signs that allow to fully leverage self-attention mechanism. Specifically, include appearance affinity modeling to aid the cost aggregation process in order to disambiguate the noisy initial correlation maps and propose multi-level aggregation to efficiently capture different semantics from hierarchical feature representations. We then combine with swapping self-attentics technique and residual connections not only to enforce consistent matching, but also to case the learning process, which we find that these result in an apparent performance boost. We conduct experiments to demonstrate the effectiveness of the proposed model over the latest methods and provide extensive ablation studies. Code and trained models are available at htttps://sunghvanhon. gg github. /ombii//ATT//
Introduction
Establishing dense correspondences across semantically similar images facilitate many Computer Vision applications, including semantic segmentation [46] [44] 36), object detection [29], and image editing (53 30 28 255. Unlike classical dense correspondence problems that consider visually similar
images taken under the geometrically constrained settings [16] 19 (50 18), semantic correspondence
poses additional challenges from large intra-class appearance and geometric variations caused by the unconstrained settings of given image pair.
51 35 addressed these challenges by carefully designing deep convolute lutiona neural networks (CNNs)- bbseed models analogously to the 41. feature extraction, cost aggregation, and flow estimation. Several 47 51] focused on the feature extraction stage, as it has been proven that the more
Recent approaches 42
45
45
3
37
31 58
57
classical matching pipeline 48
works 24
37
39
powerful feature representation the model learns, the more robust matching is obtained 24 51). However, solely relying on the matching similarity between features without any prior often suffers
"Equal contribution
Corresponding author
35th Conference on Neural Information Processing Systems (NcurIPS 2021)

---

from the challenges due to ambiguities generated by repetitive patterns or background clutters 42
24 [6]] On the other hand, some methods 42
26 [88 focused on flow estimation stage
either by designing additional CNN as an ad-hoc regressor that predicts the parameters of a single
global transformation [2 finding confident matches from correlation maps (20. 26], or directly
feeding the correlation maps into the decoder to infer dense correspondences [58] However, these
methods highly rely on the quality of the initial correlation maps.
The latest methods 45
31 27 35| have focused on the second stage, highlighting the
importance of cost aggregation. Since the quality of correlation maps is of prime importance, they proposed to refine the matching scores by formulating the task as optimal transport problem [47 [1], re- weighting matching scores by Hough space voting for geometric consistency [37 [39], or utilizing
high- digh-nieensional 4D or 6D convolutions to find locally consistent matches [45] 44 27 35), Although
formulated variously, these methods either use hand-crafted techniques that are neither learnable nor
robust to severe deformations, or inherit the limitation of CNNs, e.g., limited receptive fields, failing to discriminate incorrect matches that are locally consistent.
In this work, we focus on the cost aggregation stage, and propose a novel cost aggregation network to tackle aforementioned issues. Our network, called Cost Aggregation with Transformers (CATs), is based on Transformer [0], which is renowned for its global receptive field. By considering all the matching scores computed between features of input images globally. our aggregation networks explore global consensus and thus refine the ambiguous or noisy matching scores effectively.
Specifically, based the observation that desired correspondence should be aligned discontinuities with appearance of images, we concatenate an appearance embedding with the correlation map. which helps to disambiguate the correlation map within the Transformer. To benefit from hierarchical feature representations, following [26] 39 (58), we use stack of correlation maps constructed from multi- level features, and propose effectively aggregate the scores across the multi-level correlation maps. Furthermore, we consider bidirectional nature of correlation map, and leverage the correlation map from both directions, obtaining reciprocal scores by swapping the pair of dimensions of correlation map in order to allow global consensus in both perspective. In addition to all these combined, we provide residual connections around aggregation networks in order to ease the learning process.
We demonstrate our method on several benchmarks [38 benchmarks prove the effectiveness of the proposed model over the latest methods for semantic correspondence. We also provide an extensive ablation study to validate and analyze components in CAT's
12 Experimental results on various
2 Related Work
Semantic Correspondence. Methods for semantic correspondence generally follow the classical matching pipeline 48 early efforts 7
41
including feature extraction, cost aggregation, and flow estimation. Most leveraged the hand-crafted features which are inherently limited in capturing
30
11
high-level semantics. Though using deep CNN-based features [5] 24 42] 4 2
49 26 has become
increasingly popular thanks to their invariance to deformations, without a means to refine the matching scores independently computed between the features, the performance would be rather limited.
To alleviate this, several methods focused on flow estimation stage. Rocco
proposed an end-to-end network to predict global transformation parameters from the matching scores, and their success inspired many variants 44 23 25). RTNs 23 obtain semantic correspondences through
an iterative process of estimating spatial transformations. DGC-Net [4] Semantic-GLU
[58]
and DMP [15] utilize a CNN-based decoder to directly find correspondence fields. PDC-Net [59 proposed flexible probabilistic model that jointly learns the flow estimation and its uncertainty. Arguably, directly regressing correspondences from the initial matching scores highly relies on the quality of them.
thus have focused on cost aggregation stage to refine the initial matching scores. Among hand-crafted crafted methods, SCOT 31 formulates semantic correspondence as an optimal transport problem and attempts to solve two issues, namely many to one matching and background matching. HPF 37 first computes appearance matching confidence using hyperpixel features and then uses Regularized Hough Matching (RHM) algorithm for cost aggregation to enforce geometric consistency. DHPF (39,, that replaces feature selection algorithm
Recent numerous methods 45 47 B7l
39

---

Source Appearance Affiniy
Projection
Transposse
Feature Backbone
Connection
& Soft-argmax
Correlation
Transformm Apgregator
Transpoxx
Transformer
Aggre
pred
frr
Backbone
Dr
Projection
Target Appearance Affinity
Feature Extraction
Cost Aggregation
Flow Estimation
Figure 1: Overall network architecture. Our networks consist of feature extraction, cost aggrega- tion, and flow estimation modules. We first extract multi-level dense features and construct a stack of correlation maps. We then concatenate with embedded features and feed into the Transformer-basee cost aggregator to obtain a refined correlation map. The flow is then inferred from the refined map.
of HPF [37 with trainable networks, also uses RHM. However, these hand-crafted techniques for refining the matching scores are neither learnable nor robust to severe deformations. As learning- based approaches, NC-Net [45] utilizes 4D convolution achieve local neighborhood consensus by finding locally consistent matches, and its variants 44 27 proposed more efficient methods. GOCor (57) proposed aggregation module that directly improves the correlation maps. GSF [21) formulated pruning module to suppress false positives of correspondences in order to refine the initial correlation maps. CHM [351 goes one step further, proposing a learnable geometric matching algorithm which utilizes 6D convolution. However, they are all limited in the sense that they inherit limitation of CNN-based architectures, which is local receptive fields.
Transformers in Vision. Transformer [61]. the de facto standard for Natural Language Processing (NLP) tasks, has recently imposed significant impact on various tasks in Computer Vision fields such as image classification [IO (55) object detection 62), tracking and matching (22 [1]. ViT [10], the first work to propose an end-to-end Transformer-be based architecture for the image classification task, successfully extended the receptive field, owing to its self-attention nature that can capture global relationship between features. For visual correspondence, LoFTR [5] uses cross and self-attention module to refine the feature maps conditioned on both input images, and formulate the hand-crafted aggregation layer with dual-softmax COTR 22 takes coordinates input and addresses dense correspondence task without the use of correlation map. Unlike these, for the first time, we propose a Transformer-based cost aggregation module.
45 60) and optimal transport 47 to infer correspondences.
Methodology
3.1 Motivation and Overview
Let us denote a pair of images, i.e., source and target, as T, and I,, which represent semantically similar images, and features extracted from La and L as D, and D,, respectively. Here, our goal is to establish dense correspondence field F(i) between two images that is defined for each pixel i, which warps 4 towards Le.
Estimating the correspondence with sole reliance on matching similarities between D, and D, is often challenged by the ambiguous matches due to the repetitive patterns or background clutters 42 24 26
To address this, numerous methods proposed cost aggregation techniques that focus on refining the
initial matching similarities either by formulating the task as optimal transport problem 47 31 using
regularized Hough matching to re-weight the costs 37 39 4D or 6D convolutions
45 27 35
However, these methods either use hand-crafted techniques that are weak severe deformations, or fail to discriminate incorrect matches due to limited receptive fields.

---

(a) Source
(b) Target
(c) Raw Corr.
(d) Self-Attn.
(e) Refined Corr. (l) Ground-truth
Figure 2: Visualization of correlation map and self-attention: (a) source image, (b) target image, (c) raw correlation map, (d) self-attention, (e) refined correlation map, and (f) ground truth, which bilinearly upsampled. The visualization proves that CATs successfully aggregates the costs by integrating the surrounding information of the query, represented as green circle the source.
To overcome these, present Transformer-basee cost aggregation networks that effectively integrate information present in all pairwise matching costs, dubbed CATs, as illustrated in Fig. [ As done widely in other works 42 45 50 34 (37, we follow the common practice for feature extraction and cost computation. In the following, we first explain feature extraction and cost computation, and then describe several critical design choices we made for effective aggregation of the matching costs.
3.2 Feature Extraction and Cost Computation
To extract dense feature maps from images, we follow [26 37 39 that use multi-level features for
construction of correlation maps. We use CNNs that produce sequence of L feature maps, and D represents a feature map at (-th level. As done in [37), we use different combination of multi-level features depending
the dataset trained on, e.g., PF-PASCAL [12] or SPair-71k (38). Given sequence of feature maps, we resize all the selected feature maps to Rhxwxe with height h, width w, and channels. The resized features then undergo 1-2 normalization.
Given resized dense features DA and D.. we compute a correlation map C Rhwxke using the inner product between features: C(i,j) Dz(i) D.(j) with points and in the target and source features, respectively. In this way, all pairwise feature matches are computed and stored. However, raw matching scores contain numerous ambiguous matching points exemplified in Fig. 2 which results inaccurate correspondences. To remedy this, we propose cost aggregation networks in the following that aim to refine the ambiguous noisy matching scores.
3.3 Transformer Aggregator
Renowned for its global receptive fields, one of the key elements of Transformer 61 is the self- attention mechanism, which enables finding the correlated input tokens by first feeding into scaled dot product attention function, normalizing with Layer Normalization (LN) [ and passing the
normalized values to MLP. Several works [10 3 62 51 have shown that given images or features
as input, Transformers 61 integrate the global information a flexible manner by learning to find the attention scores for all pairs of tokens.
In this paper, leverage the Transformers integrate the matching scores discover global consensus by considering global context information. Specifically, we obtain a refined cost C' by feeding the raw cost C to the Transformer T, consisting of self-attention, LN, and MLP modules:
C' TCC Epos).
where Epos denotes positional embedding. The standard Transformer receives as input a ID sequence of token embeddings. In context, reshape the correlation map C. into sequence of vectors C(k) E xhh for k E {1, hw). We visualize the refined correlation map with self-attention in Fig. where the ambiguities are significantly resolved.
Appearance Affinity Modeling. When only matching costs are considered for aggregation, self- attention layer processes the correlation map itself disregarding the noise involved in the correlation map, which may lead to inaccurate correspondences. Rather than solely relying on raw correlation map, we additionally provide an appearance embedding from input features to disambiguate the comrlation map aided by appearance affinity within the Transformer. Intuition behind is that visually similar points an image, e.g., color or feature, have similar correspondences, as proven stereo matching literature, e.g., Cost Volume Filtering (CVF) [16. 50).

---

Positional
Intra: Correlation Sa Attention
Inter Correlation Self Attontion
Embedding
LN & MLPP
LN & MLP
Projection
Figure 3: Illustration of Transformer aggregator: Given correlation maps C. with projected features, Transformer aggregation consisting of intra- and inter-correlation self-attention with LN and MLP refines the imputs not only across spatial domains but across levels.
To provide appearance affinity, we propose to concatenate embedded features projected from input features with the correlation map. We first feed the features D) into linear projection networks, and then concatenate the output along corresponding dimension, so that the correlation map is augmented such that [C. P(D)] E ho (han+p)) where | denotes concatenation, P denotes linear projection networks, and is channel dimension of embedded feature. Within the Transformer, self aelf-attention layer aggregates the correlation map and passes the output to the linear projection networks to retain the size of original correlation C.
Multi-Level Aggregation. As shown in capturing hierarchical semantic feature representations. Thus we also use multi-level features from different levels of convolutional layers to construct stack of correlation maps. Each correlation map C. computed between D' and D is concatenated with corresponding embedded features and fed into the aggregation networks. The aggregation networks now consider multiple correlations, aiming to effectively aggregates the matches by the hierarchical semantic representations.
39 58 311, leveraging multi-level features allows
As shown in Fig. stack of L augmented correlation maps, [C', P(D [R) x undergo the Translormer aggregator. For each /-th augmented correlation map, we aggregate with self-attention layer across the points in the augmented correlation map, and we refer this as intra- cera-correlation self-attention. In addition. subsequent to this, the correlation map undergoes inter- correlation self-attention across multi-level dimensions. Contrary to HPF 337 that concatenates all the multi-level features and compute a correlation map, which disregards the level-wise similarities, within the inter-correlation layer of the proposed model, the similar matching scores are explored across lulti-level dimensions. In this way, we can embrace richer semantics in different levels of feature maps, as shown in Fig.
3.4 Cost Aggregation with Transformers
By leveraging the Transformer aggregator, we present aggregation framework with following additional techniques to improve the performance.
Swapping Self- Attention. To obtain a refined correlation map invariant to order of the input images and impose consistent matching scores, we argue that reciprocal scores should be used aids infer confident correspondences. As correlation map contains bidirectional matching scores, from both target and source perspective, we can leverage matching similarities from both directions in order to obtain more reciprocal scores as done similarly in other works
45 26
As shown in Fig. I first feed the augmented correlation map to the aforementioned Transformer aggregator. Then we transpose the output, swapping the pair of dimensions in order to concatenate with the embedded feature from the other image, and feed into the subsequent another aggregator. Note that we share the parameters of the Transformer aggregators to obtain reciprocal scores. Formally, define the whole process following:
= T(C'', P(D)) + Epos); C' = ( + Epos),
where (Ti(i)) = C(j, denotes swapping the pair of dimensions corresponding to the source and target images; S denotes the intermediate correlation map before swapping the axis. Note that NC-Net (45) proposed similar procedure, but instead of processing serially, they separately process the correlation map and its transposed version and add the outputs, which is designed to produce
5

---

(a) Source
(b) Target
(c) Corr. #1
(d) Corr. #3
(c) HPF 37
(f) CATs
Figure 4: Visualization of multi-level aggregation: (a) source, (b) target images, (c), (d) multi-level correlation maps (e.g..
and = 3), respectively, and final correlation maps by (e) HPF [37 and (f) CATs. Note that HPF and CATs utilize the same feature maps. Compared to HPF, CATs successfully embrace richer semantics in different levels of feature map.
a correlation map invariant to the particular order of the input images. Unlike this, we process the correlation map serially, first aggregating one pair of dimensions and then further aggregating with respect to the other pair. In this way, the subsequent attention layer given more consistent matching scores input, allowing further reduction of inconsistent matching scores. We include an ablation study to justify our choice in Section 4.4
Residual Connection. At the initial phase when the correlation map is fed into the Transformers, noisy score maps are inferred due to mundomly-initiaizee parameters, which could complicate the learning process. To stabilize the learning process and provide a better initialization for the matching, we employ the residual connection. Specifically, we enforce the cost aggregation networks to estimate the residual correlation by adding residual connection around aggregation networks.
3.5
Training
Data Augmentation. Transformer is well known for lacking some of inductive bias and its data- hungry nature thus necessitates large quantity of training data to be fed 6T Recent methods [55]
56
32
that employ the Transformer to address Computer Vision tasks have empirically shown that
data augmentation techniques have positive impact on performance. However, in correspondence task, the question of to what extent data augmentation affect the performance has not yet been properly addressed. From the experiments, we empirically find that data augmentation has positive impacts on performance in semantic correspondence with Transformers reported in Section 4.4 To apply data augmentation [2 with predetermined probabilities to input images at random. Specifically, 50% of the time, randomly crop the input image, and independently for each augmentation function used in (6), we set the probability for applying the augmentation as 20%. More details can be found in supplementary material.
Training Objective. As in each pair of images. We first average the stack of refined correlation maps C' Rhuxxhxx obtain C" E /e k and then transform it into dense flow field Fpred using soft-argmax operator [26]. Subsequently, we compare the predicted dense flow field with the ground-truth flow field Fet obtained by following the protocol of (37) using input keypoints. For the training objective, we utilize Average End-Point Error (AEPE) [4 computed by averaging the Euclidean distance between the
39
we assume that the ground-truth keypoints are given for
ground-truth and estimated flow. We thus formulate the objective function L F|
Epred l|2-
4 Experiments
4.1
Implementation Details
14] pre-trained on ImageNet 88, and follow- ing [37], extract the features from the best subset layers. Other backbone features can also be used, which we analyze the effect of various backbone features in the following ablation study. For the hyper-parameters for Transformer encoder, we set depth as and the number of heads as We resize the spatial size of the input image pairs to 256x256 and sequence of selected features are resized to 16x 16. We use learnable positional embedding [10), instead of fixed 61]. We implemented network using PyTorch [40], and AdamW
For backbone feature extractor, we use 101
33 optimizer with an initial learning

---

Higher PCK is better. The best results are in bold, and the second best results are underlined. CATsi means CAT's without fine-tuning feature backbone. level: Feature-level, FT. feat.: Fine-tune feature.
Table 1: Quantitative evaluation on standard benchmarks 38 11
12
SPair 71k 38
PF-PASCAL 12
WILLOW II
Methods
Feat.-level
FT. feat.
Aggregation
PCK @ Cheas
PCK @ Ging
PCK Obbo
0.1
0.05
0.1
0.15
0.05
0.1
0.15
WTA
Single
25.7
35.2
53.3
62.8
24.7
46.9
59.0
CNNGco
Single
20.6
41.0
69.5
80.4
36.9
69.2
77.8
A2Nct
Single
22.3
42.8
70.8
83.3
36.3
68.8
84.4
WeakAlign
43
Single
20.9
49.0
74.8
84.0
37.0
70.2
79.9
RTNs 23
Single
X
25.7
55.2
75,9
85.2
41.3
71.9
86.2
SFNct 26
Multi
X
53.6
81.9
90.6
46.3
74.0
84.2
NC-Net 45
Single
V
4D Conv.
20.1
54.3
78.9
86.0
33.8
67.0
83.7
DCC-Net
Single
X
4D Conv.
55.6
82.3
90.5
43.6
73.8
86.5
HPF
Multi
RHM
28.2
60.1
84.8
92.7
45.9
74.4
2D Conv.
36.1
87.8
49.1
78.7
90.2
ANC-Net 27
Single
4D Conv.
86.1
DHPF 39
Multi
RHM OT-RHM
37.3
75.7
90.7
95.0
49.5
77.6
89.1
SCOT 31
Multi
35.6
63.1
85.4
92.7
47.8
76.0
87.1
CHM
35
Single
6D Conv.
46.3
80.1
91.6
94.9
52.7
79.4
87.5
CATst
Multi
X
Transformer
42.4
67.5
89.1
94.9
46.6
75.6
87.5
CAT'
Multi
Transformer
49.9
75.4
92.6
96.4
50.3
79.2
90.3
Table 2: Per-class quantitative evaluation on SPair-71k 38
benchmark.
Mathods
aero.
bike
bird
boat
boot.
Trls
cal
choi.
COS
boss.
mihik.
plan.
shhe.
Emai.
tV
all
CNNGeo 10
23.4
16.7
40.2
14.3
36.4
27.7
26.0
32.7
12.7
27.4
22.8
13.7
20.9
21.0
17.5
10.2
30.8
34.1
20.6
35.6
13.9
NC-Net
17.9
12.2
32.1
11.7
29.0
19.9
16.1
39.2
9.9
23.9
18.8
15.7
17.4
15.9
14.8
9.6
24.2
31.1
20.1
IIP 37
25.2
18.9
$2.1
15.7
38.0
22.8
19.1
52.9
17.9
33.0
32.8
20.6
24.4
27.9
21.1
15.9
31.5
35.6
28.2
SCOF 31
34.9
20.7
63.8
43.5
21.3
63.1
20.0
42.9
29.8
35.0
23.7
24.4
40.8
35.6
DHPF
10
17.3
49.6
20.3
29.7
64.9
20.3
60.1
38.9
72.2
55.5
CATsT
16.5
26.9
69.
24.3
41.3
38.5
30.2
65.7
15.9
53.7
53.3
16.7
32.7
33.2
31,2
68.0
19.1
12.1
CATs
$2.0
34.7
73.3
34.3
49.9
57.5
43.6
66.
24.4
63.2
56.5
$2.0
42.6
41.7
43.0
33.6
72.6
58.0
49.9
rate 5 for the CATs layers and 3 for the backbone features are used, which we gradually decrease during training.
4.2
Experimental Settings
In this section, we conduct comprehensive experiments for semantic correspondence. by evaluating our approach through comparisons to state-of-He oaiic--f-thhe titth-eart art methods including CNNGco [4], A2Net [49
WeakAlign 43). NC-Net
[45]. RTNs [23]. SFNet [26]. HPF (37), DCC-Net (7], ANC-Net [27).
DHPF 39 SCOT 31 GSF 21 and CHMNet 351 In Section 4.3, we first evaluate matching
results on several benchmarks with quantitative measures, and then provide an analysis of each
component in our framework in Section 4.4. For more implementation details, please refer to our implementation available at Itttps:// //tp:////tttttttt com/ /Swn///ananong//Cna /CATs
Datasets. SPair-7Ik 38 provides total 70,958 image pairs with extreme and diverse viewpoint, scale variations, and rich annotations for each image pair, e.g., keypoints, scale difference. truncation and occlusion difference, and clear data split. Previously, for semantic matching, most of the datasets are limited to a small quantity with similar viewpoints and scales [1] [2]. As our network relies on Transformer which requires large number of data for training, SPair-71k (38) makes the of Transformer in our model feasible. we also consider PF-PASCAL [12] containing 1,351 image pairs from 20 categories and PF-WILLOW [TT] containing 900 image pairs from 4 categories, each dataset providing corresponding ground-truth annotations.
Evaluation Metric. For evaluation on SPair- 71k 38), PF-WILLOW [1I. and PF-PASCAL [12]
we employ a percentage of correct keypoints (PCK), computed as the ratio of estimated keypoints within the threshold from ground-truths to the total number of keypoints. Given predicted keypoint kpred and ground-truth keypoint kot, we count the number of predicted keypoints that satisfy following condition: d(Kpred, kcT) S max H, W), where d( denotes Euclidean distance;
7

---

Figure 5: Qualitative results on SPair-71k [38] (from top bottom) keypoints transfer results by SCOT [3I], DHPF [39], and CATs. Note that green and red line denotes correct and wrong prediction, respectively, with respect to the ground-trut -ruuhh
denotes a threshold which we evaluate on PF-PASCAL with Oiing SPair-71k and PF-WILLOW with Obbox: H and W denote height and width of the object bounding box or entire image, respectively.
4.3 Matching Results
For fair comparison, we follow the evaluation protocol of [37] for SPair-7lk, which our network trained on the training split and evaluated on the test split. Similarly. for PF-PASCAL and PF-
WILLOW, following the common evaluation protocol of [13 23 37 39, we train our network on the training split of PF-PASCAL [12 and then evaluate on the test split of PF-PASCAL [2] and
PF-WILLOW [1I. All the results of other methods are reported under identical setting.
Table 1 summarizes quantitative results on SPair-71k 388 PF-PASCAL [12] and PF-WILLOW We note whether each method leverages multi-level features and fine-tunes the backbone features in order ensure fair comparison. We additionally denote the types of cost aggregation. Generally, our CATs outperform other methods over all the benchmarks. This is also confirmed by the results on SPair-7Ik, Ik, as shown Table [ where the proposed method outperforms other methods by large margin. Note that CATsi reports lower PCK than that of CHM, and this is because CHM fine-tunes its backbone networks while CATst does not. Fig. 3 visualizes qualitative results for extremely challenging image pairs. We observe that compared to current state-of sitie--ffth-arrr the methods [3]] 39), our method is capable of suppressing noisy scores and find accurate correspondences in cases with large scale and geometric variations.
It is notable that CATs generally report lower PCK PF-WILLOW compared other state- of-the-art methods. This is because the Transformer is well known for lacking some of inductive bias. When we evaluate on PF-WILLOW, we infer with the model trained on the training split of PF- PASCAL, which only contains 1,351 image pairs, and as only relatively small quantity of image pairs available within the PF-PASCAL training split, the Transformer shows low generalization power. This demonstrates that the Transformer based architecture indeed requires means to compensate for the lack of inductive bias, e.g., data augmentation.
4.4 Ablation Study
In this section show an ablation analysis validate critical components we made to design architecture, and provide an analysis on use of different backbone features, and data augmentation. We train all the variants on the training split of SPair-71k (38] when evaluating SPair-71k, lk, and train on PF-PASCAL [12] for evaluating on PF-PASCAL. We measure the PCK, and each ablation experiment is conducted under same experimental setting for fair comparison.
Network Architecture. B the analysis on key components in our architecture. There are four key components we analyze for the ablation study, including appearance modelling, multi- level aggregation, swapping self-attention and residual connection.

---

We first define the model without any of these as baseline, which simply feeds the correlation map into the self-
Table 3: Ablation study of CATs.
attention layer. We evaluate on SPair-71k k benchmark by
SPair-711
Components
progressively adding the each key component. From to
Obbos 0.1
V, we observe consistent increase in performance when
(I)
Baseline
26.8
each component is added. shows a large improvement
(lI)
Appearance Modelling
33.5
in performance, which demonstrates that the appearance
(III)
Multi-level Aggregation
35.9
(IV)
Swapping Self-Attention
38.8
modelling enabled the model to refine the ambiguous
(V) Residual Connection
42.4
noisy matching scores. Although relatively small increase in PCK for III, it proves that the proposed model
successfully aggregates the multi-level correlation maps. Furthermore, IV and V show apparent increase, proving the significance of both components.
Feature Backbone. As shown in Table we explore the impact of different feature back- bones on the performance on SPair-7Ik [38 and PF-PASCAL [12] We report the results of models with backbone networks frozen. The top two rows models with DeiT-B [55], next two rows use DINO [4, and the rest use ResNet- 101 [14 as backbone. Specifically, subscript single for DeiT-B and DINO, we use the fea- ture map extracted the last layer for the single- level, while for subscript all, every feature map from 12 layers is used for cost construction. For ResNet-101 subscript single, we use single-level feature cropped at conv4 - 23, while for multi, we use the best layer subset provided by (37]. Sum- marizing the results, we observed that leveraging multi-level features showed apparent improvements in performance, proving effectiveness of multi-level aggregation introduced by our method. It is worth noting that DINO, which more excel at dense tasks than DeiT-B, outperforms DeiT-B when applied to semantic matching. This indicates that finettnning the feature could enhance the performance. To best of our knowledge, we are the first to employ Transformer- based features for semantic matching. It would be an interesting setup to train an end-to-end Transformer-bass based networks, and we hope this work draws attention from community and made useful for future works.
Table 4: Ablation study of feature backbone.
SPair-Tk Oldore 0.1
PF-PASCAL.
Feature Backbone
Ciin = 0.1
DeiT-Bettato SS
32.1
76.5 87.5
DaiT-Ba11 55
38.2
DINO w/ ViT-B/16esagla
39.5
88.9
DINO w/ ViT-B/16.11
42.0
88.9
ResNet-101, 10l
37.4
87.3
ResNet-10laatss
42.4
89.1
Data Augmentation. In Table we compared the PCK per- formance between our variants and DHPF [39]. We note if the
Table 5: Effects of augmentation.
model is trained with augmentation. For fair comparison, we
Augment.
SPair-7Ik
abbox 0.1
evaluate both DHPF (3 and CATs trained on SPair-7Ik (38] using strong supervision, which assumes that the ground keypoints are given. The results show that compared to DHPF, a CNN-based method, data augmentation has larger influence on CAT's in terms of performance. This demonstrates that not only we eased the data-hunger problem inherent Transform- ers, but also found that applying augmentations for matching has positive effects. Augmentation technique would bring a highly likely improvements in performance, and we hope that the future works benefit from this.
DHPF 39
X
37.3
DHPF 39
V
39.4
CATs CATs
X 43.5
/
49.9
Serial swapping. It is apparent that Equation 2 is not designed for an order-invariant output. Different from NC-Net 45 we let the correlation map undergo the self-attention module in a serial manner. We conducted a simple experiment to compare the difference between each approach. From experiments, obtained the results of parallel and serial processing which are PCK of 40.8 and 42.4, respectively. In light of this, although CATs may not support order invariance, adopting serial processing inconsistent matching scores by additionally processing the already processed cost map, which we finalize the architecture to include serial processing.
SPair-71e 1k with abbox 0.1,
obtain higher PCK as it has better capability to reduce
4.5 Analysis
Visualizing Self-Attention. We visualize the multi-level attention maps obtained from the Trans- former aggregator. As shown in Fig. 6 the learned self-attention map at each level exhibits different

---

Figure 6: Visualization of sell-attention: (from left to right) source and target images, and multi- level self-attentions. Note that each attention map attends different aspects, and CATs aggregates the cost leveraging hierarchical semantic representations.
aspect. With these self-attentions, networks can leverage multi-level correlations to capture hierarchical semantic feature representations effectively.
Memory and run-time. In Table 6 we show the memory and run-time comparison to NC-
Table 6: Memory and run-time comparison. Inference time for aggregator is denoted by (+).
Net [45]. SCOT [T], DHPF [39] and CHM [35] with CATs. For fair comparison, the results are obtained using single NVIDIA GeForce RTX 2080 Ti GPU and Intel Core 17-10700 CPU. We measure the inference time for both the pro- cess without counting feature extraction, and the whole process. Thanks to Transformers' fast com- putation nature, compared to other methods, our method is beyond compare. We also find that compared to other cost aggregation methods including 4D, 6D convolutons, OT-RHM and RHM, ours show comparable efficiency in terms of computational cost. Note that NC-Net utilizes a single feature map while other methods utilize multi-level feature maps. We used the standard self-attention module for implementation, but more advanced and efficient transformer 32 architectures could reduce the overall memory consumption.
Aggregation
Memory [GB]
Kun-time [ms]
NC-Net 45
4D Conv.
1.2
193.3 (1666) 1)
SCOT
31
OT-RHM
146.5
DHPF 39
RHM
1.6
57.7 (29.5)
CHM
Conv
47.2 (38.3)
CATS
Transformer
34.5 (7.4)
4.6 Limitations
One obvious limitation that CATs possess is that when applying the method non-corresponding images, the proposed method would still deliver correspondences as it lacks power to ignore pixels that do not have correspondence at all. A straightforward solution would to consider including module to account for pixel-wise matching confidence. Another limitation of CATs would be its inability to address task of finding accurate correspondences given multi-objects or non-corresponding objects. Addressing such challenges would be promising direction for future work.
5 Conclusion
In this paper, we have proposed, for the first time, Transformer-based cost aggregation networks for semantic correspondence which enables aggregating the matching scores computed between input features, dubbed CATs. We have made several architectural designs in the network architec- ture, including appearance affinity modelling, multi-level aggregation, swapping self-attention, and residual correlation. We have shown that our method surpasses the current state-of-the-arr in several benchmarks. Moreover, we have conducted extensive ablation studies to validate our choices and explore its capacity. A natural next step, which we leave for future work, is to examine how CAT' could extend its domain to tasks including 3-D reconstruction, semantic segmentation and stitching, and to explore self-supervised learning.
Acknowledgements
This research was supported by the MSIT, Korea, under the ICT Creative Consilience program (ITTP-2021- 11001-222000--1111 and (No. 2020-0-00368, Neural-Symboolc Model for Knowledge Acquisition and Inference Techniques) supervised by the IITP and National Research Foundation of Korea
10

---

References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv:660.00455. 2016.
arXiv preprint
[2] Alexander Buslaev, Vladimir I Iglovikov, Eugene Khvedchenya, Alex Parinov, Mikhail Druzhinin, and Alexandr A Kalinin. Albumentations: fast and flexible image augmentations. Information, 2020.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers, In ECCV. Springer, 2020.
[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégon, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2104.1 140:29404.42944 2021.
[5] Christopher B Choy, Young Gwak, Silvio Savarese, and Manmohan Chandraker. Universal correspon- dence network. NeurlPS, 29:2414-2422, 2016.
[6] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced scarch space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020.
Navnect Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In CVPR Workshops), 2005.
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hicrarchical image datahase. CVPR, 2009.
[9] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and description. In CVPR, 2018.
[10] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprint arXiv:2010. /9299, 2020.
[11] Bumsub Ham, Minsu Cho, Cordelia Schmid, and Jean Ponce. Proposal flow. In CVPR, 2016.
[12] Bumsub Ham, Minsu Cho, Cordelia Schmid, and Jean Ponce. Proposal flow: Semantic correspondences from object proposals. JEEE transactions on pattern analysis and machine intelligence, 2017.
[13] Kai Han, Rafael S Rezende, Bumsub Ham, Kwan-Yee K Wong. Minsu Cho, Cordelia Schmid, and Jean Ponce. Scnet: Learning semantic correspondence. In ICCV, 2017.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[15] Sunghwan Hong and Seungryong Kim. Deep matching prior: Test-time optimization for dense corre- spondence. In Proceedings the IEEE/CVF International Conference Computer Vision (ICCV), 2021.
[16] Asmaa Hosni, Christoph Rhemann, Michael Bleyer, Carsten Rother, and Margrit Gelautz. Fast cost-volume filtering for visual correspondence and beyond. PAMI, 2012.
[17] Shuaiyi Huang, Qiuyue Wang. Songyang Zhang, Shipeng Yan, and Xuming He. Dynamic context correspondence network for semantic alignment. In ICCV, 2019.
[18] Tak-Wa Hui, Xiaoou Tang, and Chen Change Loy. Liteflownet: lightweight convolutional neural network for optical flow estimation. In CVPR, 2018.
[19] Eddy 1lg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox Flownet 2.0: Evolution of optical flow estimation with deep networks. In CVPR, 2017.
[20] Sangryul Jeon, Seungryong Kim, Dongbo Min, and Kwanghoon Sohn. Parn: Pyramidal affine regression networks for dense semantic correspondence. In ECCV, 2018.
[21] Sangryul Jeon, Dongbo Min. Seungryong Kim, Jihwan Choe, and Kwanghoon Sohn. Guided semantic flow. In ECCV. Springer, 2020.
[22] Wei Jiang, Eduard Trulls, Jan Hosang, Andrea Tagliasacchi, and Kwang Moo Yi. Cotr: Correspondence transformer for matching across images. arXiv preprint arXiv:2103.14167, 2021.
11

---

[23] Seungryong Kim, Stephen Lin, Sang Ryul Jeon, Dongbo Min, and Kwanghoon Sohn. Recurrent transformer networks for semantic correspondence. In NewrIPS, 2018.
[24] Seungryong Kim, Dongbo Min, Bumsub Ham, Sangryul Jeon, Stephen Lin, and Kwanghoon Sohn. Fess: Fully convolutional similarim for dense semantic correspondence. In CVPR, 2017.
[25] Seungryong Kim, Dongbo Min, Somi Jeong, Sunok Kim, Sangryul Jeon, and Kwanghoon Sohn. Semantic attribute matching networks. In CVPR, 2019.
[26] Junghyup Lee, Dohyung Kim, Jean Ponce, and Bumsub Ham. Sfnet: Learning object-aware semantic correspondence. In CVPR, 2019.
[27] Shuda Li, Kai Han, Theo W Costain, Henry Howard-Jenkins, and Victor Prisacariu. Correspondence networks with adaptive neighbourhood consensus. In CVPR, 2020.
[28] Jing Liao, Yuan Yao, Lu Yuan, Gang Hua, and Sing Bing Kang. Visual attribute transfer through deep image analogy. arXiv: 705.01088, 2017.
[29] Tsung-Y Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongic. Feature pyramid networks for object detection. In CVPR, 2017.
[30] Liu, Jenny Yuen, and Antonio Torralba. Sift flow: Dense correspondence across scenes and its
applications. IEEE transactions on pattern analysis and machine intelligence, 33(5)977-994, 2010.
[31] Yanbin Liu, Linchao Zhu, Makoto Yamada, and Yi Yang. Semantic correspondence as an optimal transport problem. In CVPR, 2020.
[32] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint 10X 2021.
[33] Ilya Loshehilov and Frank Hutter. Decoupled weight decay regularization. arXiv:1711.05101. 2017.
[34] laroslav Melekhov, Aleksei Tiulpin, Torsten Sattler, Mare Pollefeys, Esa Rahtu, and Juho Kannala. Dge-net:
Dense geometric correspondence network. WACV, 2019.
[35] Juhong Min and Minsu Cho. Convolutional hough matching networks. arXiv preprint arXiv21103.1633. 2021.
[36] Juhong Min, Dahyun Kang, and Minsu Cho. Hypercorrelation squeeze for few-shot segmentation. arXiv preprint 2rXiv:2104.01553 2021.
[37] Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Hyperpixel flow: Semantic correspondence with multi-layer neural features. In ICCV, 2019.
[38] Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Spair-71k: A large-scale benchmark for semantic correspondence. arXiv preprint 1908. 1908.1043, 2019.
[39] Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Learning to compose hypercolumns for visual correspondence. In ECCV, 2020.
[40] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming
Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[41] James Philbin, Ondrej Chum, Michael Isard, Josef Sivie, and Andrew Zisserman. Object retrieval with large vocabularies and fast spatial matching. In CVPR. IEEE, 2007.
[42] Ignacio Rocco, Relja Arandjelovic, and Josef Sivic. geometric matching. In CVPR, 2017.
Convolutional neural network architecture for
[43] Ignacio Rocco, Relja Arandjelovié, and Josef Sivie. End-to-end weakly-supervised semantic alignment. In CVPR, 2018.
[44] Ignacio Rocco, Relja Arandjelović, and Josef Sivic. Efficient neighbourhood consensus networks via submanifold sparse convolutions. In ECCV, 2020.
[45] Ignacio Rocco, Mireea Cimpoi, Relja Arandjelović, Akihiko Torii, Tomas Pajdla, and Josef Sivie. Neigh- bourhood consensus networks. In NewrlPS, 2018
12

---

[46] Michael Rubinstein, Armand Joulin, Johannes Kopf, and Ce Liu. Unsupervised joint object discovery and segmentation in internet images. In CVPR, 2013.
[47] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In CVPR, 2020.
[48] Daniel Scharstein and Richard Szeliski. A taxonomy and evaluation of dense two-frame stereo correspon- dence algorithms. International journal of computer vision, 2002.
[49] Paul Hongsuck Sen, Jongmin Lee, Deunsol Jung, Bohyung Han, and Minsu Cho. Attentive semantic alignment with offset-aware correlation kernels. In ECCV, 2018.
[50] Deqing Sun, Xiaodong Yang. Ming-Yo Yu Liu, and Jan Kautz. Pwe-net: Cnns for optical flow using pyramid, warping, and cost volume. In CVPR, 2018.
[51] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector- frre local feature matching with transformers. arXiv preprint arXiv:2 00680, 2021.
[52] Peize Sun, Yi Jiang, Rufeng Zhang, Enze Xie, Jinkun Can, Xinting Hu, Tao Kong, Zehuan Yuan, Changhu Wang, and Ping Luo. Transtrack: Multiple-object tracking with transformer. arXiv preprint arXiv:2012. 15460, 2020.
[53] Richard Szeliski. Image alignment and stitching: tutorial. Foundations and Trends0 Computer Graphics and Vision, 2006.
[54] Tatsunori Taniai, Sudipta N Sinha, and Yoichi Sato. Joint recovery of dense correspondence and cosegmen- tation two images. CVPR, 2016.
[55] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers and distillation through attention. arXiv preprint arXiv:2012. 12877, 2020.
[56] Hugo Touvron, Matthiuu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. arXiv preprint arXiv:2103.1723: 2021.
[57] Prune Truong, Martin Danelljan, Lue V Gool, and Radu Timofte. Gocor: Bringing globally optimized correspondence volumes into your neural network. In NeurlPS, 2020.
[58] Prune Truong, Martin Danelljan, and Radu Timofte. Glu-net: Global-local universal network for dense flow and correspondences. In CVPR, 2020.
[59] Prune Truong, Martin Danelljan, Lue Van Gool, and Radu Timofte. Learning accurate dense correspon- dences and when to trust them. arXiv preprint arXiv:2101.017/0 2021.
[60] Michat Tysrkiewicz, Pascal Fua, and Eduard Trulls. Disk: Learning local features with policy gradient. arXiv preprint arXiv:2006. 13566, 2020.
[61] Ashish Vaswani, Noam Shazcer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv: 03762, 2017.
[62] Xizhou Zhu, Weijie Su, Lewei La, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end tod object detection. arXiv preprint arX1v:001000419, 2020.
13


Reviews:
[{"review_id": "wOtDLTuyfrq", "reviewer": "Reviewer_CGvU", "summary": "In this paper, the authors propose transformer-based cost aggregation networks for semantic correspondence with challenges of large intra-class appearance and geometric variations. The overall learning pipeline hierarchically aggregates the matching scores computed between disambiguated input features from semantically similar images by combining with swapping self-attention and residual connections. Extensive experimental results on three datasets, SPairk-71k, PF-PASCAL, and PF-WILLOW, show its efficacy on finding dense semantic correspondences.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Motivation and reasoning about transformer]\nFirst, the motivation of the proposed system is described clearly, considering the necessity of leveraging transformer-based cost aggregation for dense correspondence estimation between semantically similar images. Overall, the flow of the paper is easy to understand, and the goal is straightforward.\n\n[About architectural design – aggregator]\nThe overall network architecture consists of feature extraction, cost aggregation, and flow estimation steps. Specifically, on the stage of cost aggregation, each feature is transformed with the proposed module, CAT, and aggregated with intra- and inter-correlation self-attention mechanism. Here, I have a question about the role of each self-attention module. Although in L165, the operation of each module is described well, the motivation for disentangling each module is not clear. I would like to see the reasoning of their design, and what happens if the modules are not decomposed.\n\n[Natural effectiveness of cost aggregation itself]\nInherently, the cost aggregation by inner product has the capability of finding correlation between two input features. As described in Figure 2 (c), the raw correlation map shows its strong capability of finding semantically similar features. Therefore, these results raise a doubt as to whether the proposed CAT module is effectively designed, considering the complexity of the module. I wonder if the effect of CAT can be maximized when finding semantic correspondence for cluttered scenes with high complexity, such as traffic driving environments. Currently, most of the demonstrated dataset images contain only a single object.\n\n[Minor comment]\nThe title is too ambiguous to deliver/represent the contribution of the works.\n\n### UPDATE AFTER REBUTTAL ###\nI appreciate the authors' feedback and valuable comments from other reviewers. Parts of my main concerns (e.g., reasoning about the design of the aggregator) are eased with the authors' fair reasonings. However, as commented by FB1A, those ablation studies (including the justification of transformers and cost volume) need to be discussed in the main paper. In addition, It would be better to include some failure cases or qualitative results with multiple objects for further discussion (although I appreciate the reply from the authors for the issue of multiple objects). Overall, I update my final score from 7 to 6, but still leaning towards acceptance.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I would like to see how the author would respond to the weaknesses in the previous question and the opinions of other reviewers.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "r0hs3NeHaJm", "reviewer": "Reviewer_TuyT", "summary": "The paper deals with the task of semantic correspondence, i.e., finding correspondence between semantically similar images, e.g., the front left paw of two different dogs.  In a nutshell, the paper explores the use of Transformers for this problem and demonstrate that the built-in global receptive field is very beneficial for this task.  Specifically, the authors improve the cost-aggregation part of a semantic correspondence pipeline, which refines the initial matching costs.  The authors propose to concatenate the correlation between pixels (on a feature map from a neural network) with the corresponding appearance embedding, which helps to disambiguate noisy or ambiguous matches (like the left and right eye of a cat as in Figure 2).  Results are demonstrated on standard benchmarks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality\n\nAlthough the paper is not ground-breaking, it demonstrates how Transformer modules can be beneficial for the task of semantic correspondence.\n\n\n### Clarity\n\nThe paper is well written, apart from a few typos.\n\n\n### Experiments\n\n- In Table 1, I think it makes sense to see an improvement for the dataset SPair-71k because it contains more data. But compared to CHM, the performance of the proposed method is not great; a rank-based metric to compare CHM and CATs would actually go in favor of CHM, when only considering columns where both methods have valid results.  This brings me to the next question; why are no results of CHM for PCK threshold of 0.15?\n\n\n### Minor comments\n\n- Line 4: Please split the sentence \"Compared to previous ...\" to make it easier to comprehend.\n- The abstract is vague and hard to understand without already knowing the typical pipeline for semantic correspondence frameworks.\n\n\n### Post-rebuttal update\n\nAfter having read all other reviews and the authors' response, I have decided to keep my initial rating and recommend accepting the submission. However, I also agree with the comments from other reviews on the missing justification of using Transformers and cost volumes. Irrespective of the acceptance decision, I strongly encourage the authors to integrate these additional discussions to make the paper stronger.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I did not find any mentions of limitations, but the broader impact to society is discussed in the supplemental material.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "a-lFohyKy-0", "reviewer": "Reviewer_FB1A", "summary": "The paper proposes a cost-volume refinement method using transformers to be used for a cost-volume-based semantic correspondence framework. The main innovations of the method are the use of transformers, how the cost volume is aggregated with appearance features, and the serial way in which appearance features are used. The method is tested on SPair-71K, PF-Pascal, and PF-Willow datasets. ", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper provides decent performance, outperforming the competitors on SPair-71K, while performing slightly worse in PF-Pascal and PF-Willow. However, as the main argument of the paper is about the use of Transformers for cost volume refinement, this has to be well justified, which the current paper lacks. Considering this, with the presentation quality issues, the reviewer is currently leaning towards rejection. The reviewer, however, would be very happy to be convinced otherwise.\n\n1. Why Transformers?\n\nContrary to the argument made in the paper, e.g. lines 87--88, CNN-based architectures do not necessarily suffer from local receptive fields. For example, even the featuremap in this paper, which is 16x16 at the end, would have a receptive that roughly covers the entire image. For example, resnet 101 (v1) has a receptive field size of 1027 pixels. This, combined with the post-processing or hierarchical processing makes the locality argument less convincing. Moreover, a naive way to remove the locality constraint for a 16x16 feature map would be to use MLPs. \n\nIn fact, a critical component of the transformer is that they actually relate and route features according to the pair-wise relationships -- the core idea behind the self-attention module. Hence I would suggest that the authors reconsider this argument.\n\nGiven this, what is most important in this paper is in fact, not just achieving state-of-the-art performance, but rather revealing that one should for sure head towards transformers. The architecture suggested in this paper is a perfect playground for this as the T function could be replaced with any deep network -- which is quite rare! One could make use of CNNs, (both 2D and 3D), as well as a fully connected layer. This ablation study is a must for the paper to deliver its main message, but is missing. Without a clear justification on why transformer is necessary, the contributions of the paper itself are quite incremental, as many of the components are similar to existing work (e.g. [35]). The reviewer would like to note that the ablation study in the current manuscript is great, but they are somewhat tangent to showing that the use of a transformer is essential.\n\n2. Why cost volume?\n\nAs shown in Table 3, it seems that the use of appearance, and how the transformer utilizes it seems to be critical to the final performance. However, since transformers have the ability to perform dot products, at this point, it becomes somewhat questionable whether the cost volume itself is necessary at all. For example, what would happen if everything in the pipeline remains the same but the cost volume is removed? This may create some issues related to connecting between the two images, but then the serial way in which the appearances of the two images are provided may allow information to flow through and let the transformer perform this matching by itself. Hence this is perhaps another ablation study that is required to justify the architecture.\n\n3. Missing reference\n\nFollowing up, in fact in COTR [a], the paper does something along these lines where cost volume is omitted and it is left to the transformer to make correlations among features. Their work shows that this is a highly effective strategy in forming correspondences. This work is also the first work that the reviewer is aware of on applying Transformers to the correspondence problem (their work is not on semantic correspondences though), predating also LoFTR in terms of the date it became public. While it is probably unreasonable for the paper to compare against this method as it is aiming for geometric correspondences (COTR code page does show a demo for semantic correspondence across human faces), the paper should at least differentiate itself from this method.\n\n[a] Jiang et al., \"COTR: Correspondence Transformer for Matching Across Images\", ArXiv 21\n\n4. Serial  swapping\n\nThe serial swapping suggested in Eq 2 is not order invariant. Hence, differently from NC-Net, depending on which image is used first, it will give different outcomes. Is there some mitigation strategy to avoid this from happening? Otherwise, this does not seem like a proper way to encourage cycle consistency, but another way to incorporate the two appearances.\n\n5. Presentation\n\nThe quality of the presentation requires improvement. The paper does not read well and has multiple grammatical errors that harm the paper's quality. For example, \"appearance affinity modeling\" in line 9 is unclear, as well as lines 11--14 unless the reader has already read the paper. In line 26, the paper states that it has been proven that more powerful feature representation... but this is just supported by empirical evidence. A similar mistake is in line 202, where the paper states augmentation guarantees performance boost, which is not always true. Other examples include; In line 22, \"unconstrained settings\" is unclear; line 40 \"formulated variously\"; line 65 \"without a means to refine...\" phrase seems discontinued from the previous phrase; line 131 \"by first feeding into scaled dot product attention function\" object of the sentence is missing; line 199 \"inductive bias\" could be anything and not clear as Transformers also have inductive bias coming from their particular structure; line 204 \"have empirically found\" should be present tense to be consistent; line 218 no need for \"basically\".\n\nThere are some organizational issues as well, for example, lines 104--115 are somewhat repeating what is said in the related works section, and it is the reviewer's personal opinion that they belong also either in the intro or related works as they are not directly relevant to the method. Similarly, 130--136 also discusses Transformers and Layer Normalization in general, which is again more suited in the related works section.\n\nMany important details about the method are also left to reference. For example, in lines 119--121, the paper refers to 35 for how multi-level features are extracted, but without this information, it is hard to understand the paper. For completeness, this information should be included. It seems that some of the details are present in the supplementary appendix, but in this case, the main paper should say so. Another example where detail is missing is related to augmentation in lines 206--207, as without reading [6], it's impossible to replicate the paper. \n\n6. Promises in the checklist are not delivered\n\n1(c) should be Yes, with a pointer to the appendix.\n\n3(d), 4(a), 4(b) are promised, but not provided.\n\n==== Post Rebuttal Update ====\n\nI am convinced by the new experiments that the reviewers have added, and agree that it is now enough to empirically justify the proposed method's design. However, I am concerned that incorporating these changes would require a significant amount to rewrite, which we have no means of verifying its quality after edit. I thus still vote for rejecting the paper. It seems like the submission was an unfinished product, that was completed during the rebuttal period.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper does not detail the limitations of the proposed method. One obvious limitation that the reviewer can think of would be when applying the method to non-corresponding images. It is likely that the proposed method would still deliver correspondences then. It is also somewhat unclear how the method would perform for pixels that do not have correspondence at all. Recent works, e.g. PDC-Net [57] allows such regions to be ignored. I am not sure the current method is able to do so.\n\nThe appendix does discuss broader impact. I would also imagine current work could be used to make deep fakes even more realistic, as a societal impact. Still, the work itself does not have immediate ethical concerns.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Y1Wzb1Zhjhz", "reviewer": "Reviewer_MLbN", "summary": "This paper addresses the problem of finding dense correspondences between semantically similar images. Given a feature extractor, the primary focus of this work is on the design of a matching algorithm to find the optimal matches for each spatial location. The proposed method is a learned transformer based architecture that takes advantage of multi-scale features to resolve ambiguous local matches. Results demonstrate that the performance of the proposed model is comparable to existing methods on multiple datasets. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The paper addresses an important problem of finding correspondences across semantically similar images. Majority of the research in this direction focuses on improving feature extraction methods. However, this work shows that building smarter post-processing algorithms for finding matches could bring significant improvements in the quality of correspondences. \n\n- The proposed method of using a transformer-based architecture for computing feature correlation scores is technically sound, intuitive and novel. The motivation for the presented design of the architecture has been explained thoroughly. The idea of including multi-scale appearance features and processing the intra-level and inter-level features sequentially is also novel and could be adopted in other domains. \n\n- The experimental results show that the proposed model is at least as effective as existing state-of-the-art models for finding semantic correspondences. The qualitative results presented provide insight into the functioning of the self-attention mechanisms and demonstrate that the model can generate high-quality correspondences. Furthermore, the ablative studies show that all the design choices in the transformer-based architecture lead to improved correspondences. \n\n## Concerns/Suggestions\n- The quality of writing of the paper could be significantly improved. Especially the abstract and introduction are very difficult to follow. For example, Line 48-56 uses too much jargon without any context provided. It only makes sense after reading the approach section. So I think the paper would benefit from thorough proof-reading. \n\n- Since the proposed cost-aggregation method relies on a transformer-based architecture, it would be good to know the computational cost compared to existing methods like CVF. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have not discussed the societal impact of this work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5 hours "}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper initially had mixed reviews (7,7,6,3). The reviewers seem to agree that the proposed method is interesting, new enough to warrant publications, and the experiments are solid. The negative reviewer raised concerns about the paper's presentation, including lack of justification of certain technical choices and statements. Thanks to the author's rebuttal, many of these concerned have been addressed (also thanks to the additional experiments). The negative reviewer upgraded their rating to 5, saying they are convinced by the new experiments that the reviewers have added. In a private discussion between the reviewers (not visible to the authors), an approximate consensus emerge: the amount of revision required to improve the presentation is considered small enough to warrant acceptance now. The reviewers explicitly said that they trust the authors to make the promised changes.

Decision: Accept (Poster)