Unifying Feature and Cost Aggregation with Transformers for Semantic and Visual Correspondence
Authors: Sunghwan Hong
Seokju Cho
Seungryong Kim
Stephen Lin
Venue: ICLR 2024 poster (2024)
Keywords: Feature aggregation
Cost aggregation
Transformer
Dense matching
Primary area: visualization or interpretation of learned representations

Abstract: This paper introduces a Transformer-based integrative feature and cost aggregation network designed for dense matching tasks. In the context of dense matching, many works benefit from one of two forms of aggregation: feature aggregation, which pertains to the alignment of similar features, or cost aggregation, a procedure aimed at instilling coherence in the flow estimates across neighboring pixels. In this work, we first show that feature aggregation and cost aggregation exhibit distinct characteristics and reveal the potential for substantial benefits stemming from the judicious use of both aggregation processes. We then introduce a simple yet effective architecture that harnesses self- and cross-attention mechanisms to show that our approach unifies feature aggregation and cost aggregation and effectively harnesses the strengths of both techniques. Within the proposed attention layers, the features and cost volume both complement each other, and the attention layers are interleaved through a coarse-to-fine design to further promote accurate correspondence estimation. Finally at inference, our network produces multi-scale predictions, computes their confidence scores, and selects the most confident flow for final prediction. Our framework is evaluated on standard benchmarks for semantic matching, and also applied to geometric matching, where we show that our approach achieves significant improvements compared to existing methods.

Published as a conference paper at ICLR 2024
UNIFYING FEATURE AND COST AGGREGATION WITH TRANSFORMERS FOR SEMANTIC AND VISUAL CORRE- SPONDENCE
Sunghwan Hong", Seokju Cho', Seungryong Kim
Korea University {sung hwan, seokju ju cho, seungryong ac.
Stephen Lin
Microsoft Research Asia
stevelin@ne croso
weeeeiinmicrooof...o com
ABSTRACT
This paper introduces a Transformer-base integrative feature and cost aggrega- tion network designed for dense matching tasks. In the context of dense matching, many works benefit from one of two forms of aggregation: feature aggregation, which pertains to the alignment of similar features, or cost aggregation, a proce- dure aimed at instilling coherence in the flow estimates across neighboring pixels. In this work, first show that feature aggregation and cost aggregation exhibit distinct characteristics and reveal the potential for substantial benefits stemming from the judicious use of both aggregation processes. We then introduce simple yet effective architecture that harnesses self- and cross-attention mechanisms to show that our approach unifies feature aggregation and cost aggregation and ef- fectively harnesses the strengths of both techniques. Within the proposed attention layers, the features and cost volume both complement each other, and the attention layers are interleaved through a coarse-to-fine design to further promote accurate correspondence estimation. Finally at inference, our network produces multi-scale predictions, computes their confidence scores, and selects the most confident flow for final prediction. Our framework is evaluated on standard benchmarks for se- mantic matching. and also applied geometric matching, where we show that approach achieves significant improvements compared to existing methods.
INTRODUCTION
Finding visual correspondences between images is a central problem in computer vision, with numerous applications including simultaneous localization and mapping (SLAM) (Bailey & Durrant-Whye 2006), augmented reality (AR) (Peebles et al., 202I). and structure from motion (SIM) (Schonberger & Frahm 2016). Given visually or semantically similar images, sparse cor- respondence approaches (Lowe, 2004) first detect a set of sparse points and extract corresponding descriptors to find matches across them. contrast, dense correspondence (Philbin al.) (007)) aims at finding matches for all pixels. Dense correspondence approaches typically follow the clas- sical matching pipeline feature extraction and aggregation, cost aggregation, and flow estima- tion (Scharstein & Szeliski 2002 Philbin et al. 2007).
Much recent correspondence research Sarlin et al., 2020 Sun et al. 2021 Jiang et al. 2021 Xu ct al., 2021: Li al. 2021 Cho et al. 202 1 Min et al. 202 Huang al. 2022b Cho et al. 2022a) have utilized a means to benefit from either feature aggregation or cost aggregation. Feature aggregation, as illustrated in Fig. [ (a), is process that aims not only integrate self-similar features within an image but also align similar features between the two images for matching. The advantages of feature aggregation have been made particularly evident in several attention- and
"equal contribution Corresponding author

---

Published as a conference paper at ICLR 2024
Fexture Extraction
Feature Extroction
Feature Fatraction
D
D,
Cost
Cost
Cost Computation
Cost Aggregation
Feature
Feature Extraction
Festure Estraction
Extraction
Dr
D
Dc
Feature
Integrative
Apgregation
Feature and Apgregation
(a) Feature Aggregation
(b) Cost Aggregation
(c) Integrative Aggregation (Ours)
Figure I: Intuition of the proposed method: (a) feature aggregation methods that aggregate feature descriptors, (b) cost aggregation methods that aggregate a cost volume, and (c) our integrative fea- ture and cost aggregation method, which jointly performs both aggregations find highly accurate correspondences.
Transformer- based matching networks (Vaswani et al., 2017 Sarlin et al.) 2020: Sun et al. 2021 Xu et al. 2021 Jiang et al., (021). Their accuracy in matching can be attributed to, we show Fig. 2 (e-f) and supported by previous studies (Sun et al. 2021: Amir al. 2021), the learned semantic features. While the visualization exhibits consistency among parts with the same semantics, dense matching often requires features with even greater discriminative power for more robust pixel-wise correspondence estimation, which is typically challenged by repetitive patterns and background clutters.
To compensate for, the other hand, cost aggregation, illustrated in Fig,
[b). has been adopted
by numerous works (Rocco et al.] 2018 Min et al. 20216; Cho et al. 202 Huang et al. 20226]
Cho et al. 2022a) for its favorable generalization ability (Song et al.] [021] Liu al. 2022) and ro- bustness to repetitive patterns and background clutter, which can be attributed to the matching simi- larities encoded in the cost volumes. These works can leverage the matching similarities enforce smoothness and coherence in the disparity or flow estimates across neighboring pixels. However, it important to note that, as highlighted in Fig. 3 (c-h), cost volumes often lack semantic context and exhibit relatively less consideration of spatial structure. This disparity arises due to the fact that the information encapsulated within cost volumes established on the basis of pixel pairs, which could potentially lead to challenges in scenarios where such contextual cues play a pivotal role.
In this paper, we tackle the dense correspondence task by first performing thorough exploration of feature aggregation and cost aggregation and their distinct characteristics. We then propose simple yet effective architecture that can benefit from the potential advantages stemming from a more judicious use of both aggregation processes. The proposed architecture is Transformer- based aggregation network, namely Unified Feature and Cost Aggregation Transformers (UFC), that models an integrative aggregation of feature descriptors and the cost volume, illustrated in Fig. (c).
This network consists of two stages, the first of which employs self-attention layer aggregate the descriptors and cost volume jointly. In this stage, the descriptors can help to disambiguate the noisy cost volume similarly to cost volume filtering (Hosni et al.) 2012 Sun al. 2018). 8), and the cost volume can encourage the features to account for matching probabilities as an additional factor for alignment. For the subsequent step, we design a cross-attention layer that enables further aggregation aided by the outputs from earlier aggregations. This aggregated cost volume can guide the aligmment with its sharpened matching distribution. These attention layers are interleaved. We further propose hierarchical processing to enhance the benefits one aggregation gains from the other. Finally, at inference time, our method estimates multi-scale predictions and their confidence scores to recover highly accurate flow.
We first evaluate the proposed method on the tasks of semantic matching and subsequently, we also substantiate that our framework achieves appreciable performance when applied to geometric matching. Our framework clearly outperforms prior works on all the major dense matching bench- marks, including HPatches (Baintas et al. 2017), ETH3D (Schops et al.] 2017). SPair-71k (Min) jet al. (0196)) PF-PASCAL (Ham et al., (017)) and PF- WF-WILLOW (Ham et al. (016). We provide extensive ablation and analysis to validate our design choices.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
Feature Extraction and Aggregation. Feature extraction involves detecting interest points and extracting the descriptors of the corresponding points. In traditional methods (Liu et al. 2010 Bay et al. 2006 Dalal & Triggs 2005 Tola et al. 2009), the matching performance mostly relies on the quality of the feature detection and description methods, and outlier rejection across matched points is typically determined by RANSAC (Fischler & Bolles 1981).
Learning-based feature extraction methods (DeTooee et al. 2018 Ono et al. 2018 Dusmanu al. 2019 Revaud et al. 2019) obtain dense deep features tailored for matching. These works have
demonstrated that the quality of feature descriptors contributes substantially to matching perfor-
mance. In accordance with this, recent matching networks (Sarlin et al. (2020; Min et al. [2019a] Lee
et al. 2019; Hong & Kim 202T Min al. 2020 Jiang et al. al.] [202] [Sun et al. 2021 Xu al. [021]
proposed effective means for feature aggregation. Notable sparse correspondence works include Su-
perGlue (Sarlin al.] 2020) and LOFIR (Sun et al. 202T), which employ graph or Transformer-
based self- and pross-attentiin for aggregation. Other methods are PUMP (Revaud et al. (022) and
ECO-TR (Tan al. 2022), which are follow-up works COTR (Jiang et 2202T) These methods evaluate dense correspondence benchmarks in different way from previous works, using only the sparse and quasi-dense correspondences above certain confidence levels. We refer the readers to the supplementary material for detailed discussion.
For dense correspondence. SFNet (Lee et al. 2019 and DMP (Hong & Kim 2021) introduce adap- tation layers after feature extraction learn feature maps well to matching and evaluated on dense semantic and geometric matching. DKM (Edstedt et al. 2023) adopts Gaussian Processing Kernels for dense correspondence, and it demonstrates its effectiveness in pose estimation. In optical flow, GMFlow (Xu et al. [222]] leverages Transformer for feature aggregation, and its extension Xu al. (2023) applies the method stereo matching and depth estimation. In semantic correspon- dence, notable works include SCorrSAN (Huang et al. 2022a) proposes an efficient spatial context encoder to aggregate spatial context and feature descriptors, and MMNet/Zhao et al. (202T) proposes multi-scale matching network to learn discriminative pixel-level features.
Cost Aggregation. In the dense correspondence literature, many works have designed their
chitectures for effective cost aggregation, which brings strong generalization power (Song et al.
2021
Liu et al., 2022). Recent works (Truong al. [2020b] HHng & Kim [202]] Jeon et al. [0200]
Truong et al. 2021) use 2D convolutions to establish correspondence while aggregating the cost volume with learnable kernels, while some works (Min et 2019a 2020; Liu et al. 2020) utilize handcrafted methods, which include RHM (Cho et al. 2015) and the OT solver (Sinkhorn 1967).
NC-Net (Rocco et al. (018)) was the first to propose 4D convolutions for cost aggregation, and nu- merous works (Li et al. 2020: Yang & Ramanan 2019 Huang et al.
2019 Rocco et al. 2020 Min
ctt al.
202 a b) leveraged or extended 4D convolutions.
Among bransformer-based networks, CATs (Cho et al. 2021 recently proposed to use Trans- former (Vaswani et al.) (017)) for cost aggregation, and its extension CATs++ (Cho et al.] combined convolutions and Transformer for an enhanced cost aggregation. VAT (Hong et al. proposed 4D convolutional Swin transformer for cost aggregation that benefits from better general ization power and showed its effectiivnnes for semantic correspondence. NeMF (Hong et al. 2022b)
2022a)
2022a
incorporates an implicit neural representation into semantic correspondence and implicitly repre- sents the cost volume infer correspondences defined at arbitrary resolution. FlowFrrmer (Huang let al. (202bb) and STTR (Li et al.) (022]) are Transformer- based cost aggregation networks specifi- cally designed for optical flow or stereo matching.
FEATURE AND COST AGGREGATION
In this section, examine the characteristics of feature and cost aggregation, which will later be verified empirically in Section 5.3 Fig. 2 and Fig. 3 present visualizations of feature maps and cost volumes different stages of aggregation. Although there may be different types of aggrega- tions, throughout this work, we focus on attention- based aggregations. From the visualizations, the following observations can be made.
The information encoded by features and cost volumes differ: Feature aggregation and cost aggregation thus exploit different information, which is exemplified in Fig. (c-d) and Fig. (c-e)
3

---

Published as a conference paper at ICLR 2024
(a)
(b)
(d)
(f)
(g)
(h)
Figure 2: PCA visualizations of feature maps: (a-b) source and target images. (c-d) raw feature maps. (e-f) feature maps that have undergone feature aggregation. (g-h) feature maps that have undergone integrative aggregation. Our integrative aggregation methodology enables the acquísition of more discerning feature representations while preserving both semantic and spatial structural aspects, resulting the estimation of highly accurate correspondences.
(b)
(c)
(d)
(c)
(f)
gg
(h)
Figure 3: Visualizations of cost volumes: (a-b) source and target images. (c-e) 2D slices of raw cost volumes at different levels 1. (f) cost volumes constructed using feature maps that have undergone self-attention. (g) with feature maps that have undergone both self-attention and cross-attention, and (h) the cost volume that have undergone cost aggregation. From (c-e), the noises are suppressed in (h), while (f-g) shows aggregated features help to construct less noisy cost volumes. Note that the visualizations are obtained with respect to the circled point in the target image.
where the spatial structure preserved the features while sparse spatial locations with higher similarity to the query point is highlighted in the cost volume. Due the different information encoded in their inputs, the outputs both aggregations have different characteristics. In Fig. 2 compared raw features in (c-d), the feature aggregation in (e-f) makes the features of semantic parts, e.g. legs/claws and head, more consistent between the two birds. This is in agreement with observations previous studies (Amir al. 2021 Sun al. [021). On the other hand, compared to noisy cost volumes visualized in Fig. 3 (c-e), the aggregated cost volume in (h) is less noisy and more clearly highlights the most probable region for matching while less probable regions are suppressed. We additionally observe that each type of aggregation can have apparent effects on the other. Naturally, more robust descriptors can construct less noisy cost volume shown Fig. (f-gg) and ease the subsequent cost aggregation process to promote more accurate correspondences. However, without a special model design, cost aggregation would not affect feature aggregation since it performed after feature aggregation. Motivated by this, proposed integrative aggregation, and its effects on feature maps are shown in Fig. ( (g-h), where salient features become more distinguishable from each other, i.e., the left and right claws. This exemplifies that more discriminative feature representations that are more focused and that also preserve semantics and spatial structure can be leamned and reveals that potential benefits from their integration can be realized.
From these observations, we propose in the following a simple yet effective architecture that makes prudent use of both feature and cost aggregation.
4

---

Published as a conference paper at ICLR 2024
Interaate Sat Mettmmtt
mropettion
Prolctioon
LN MEP
La VU
Softma
Usuumppe
Coneto
Intambbe Sar Serfarmentnn
Upaampe
LN MLP
Ang Prol
Projicion
Roshape
Gaovll
(a) Integrative self-attention
(b) Cross-attention with matching distribu-
Figure 4: Illustration of the proposed self- and cross-attention: (a) joint feature aggregation and cost aggregation, and (b) cross-attention layer with matching distribution.
METHODOLOGY
4.1 PROBLEM FORMULATION
Let us first denote a pair of visually or semantically similar images, i.e., the source and target, as I and Ie, the feature descriptors extracted from T, and I, D, and D., respectively, and the cost volume computed between the feature maps as C. Given 1, and 4, we aim to establish a dense correspondence field F(i) that is defined at all pixels and warps La towards 4. Given features extracted from deep CNNs (He et al. 2016) or Transformers (Dosovitskiy et al. 2020), we can 1 xxxx
construct and store cost volume that consists of all pairwise feature similarities with height h and width w: C(i,j) = D.(i) D,(i), where and j index the source and target features, respectively. The dense correspondence field, F(i). can then be determined from C(i,j) considering all
4.2 PRELIMINARIES: SELF- AND CROSS-ATTENTT ATTENTION
We briefly explain the attention mechanism, core component we extend from. Given sequence of tokens as an input, Transformer (Vaswani et al. 2017) first linearly projects tokens obtain query, key and value embeddings. These are then fed into scaled dot product attention layer, followed by Layer Normalization (LN) (Ba et al., 2016)) and a feed-forward network or MLP, to produce an output with the same shape as the input. Each token is attended to by all the other tokens. This projections are formulated as:
Q Po(x),, K Px(x(X) = Px(X),
where Pq. Px and Py denote query, key and value projections, respectively, and X denotes a token with a positional embedding. Subsequently, they pass through an attention layer:
QKT Vdk
Attention(X) softmax(
VV,
(2)
where dx is the dimension the key embedding. Note that the Attentionn.) function can be defined in various ways (Wang et al. 2020 Liu et al. 2021 Katharopoulos et al. 2020 Lu et al.) 2021 Wu et al. 2021). Self- and cross-attention attention are distinguished by their input to the key and value projections. Given pair of input tokens, e.g., X, and X,, the input to the key and value projections of self-attention for X, is the same input, X,, but for cross-atee atteetiooo across X, and X,, the inputs to the key and value projection are X,.
5

---

Published as a conference paper at ICLR 2024
4.3 UNIFIED FEATURE AND COST AGGREGATION
Integrative Self-Attention. Toward more judicious use of both aggregations, we first leverage the fact that both feature descriptors D., D, and cost volume C' encode different information. To this end, in the proposed integrative self-attention layer, as shown in Fig. we first obtain a feature cost volume [D, C| by concatenating D and C, where [ denotes concatenation.
This concatenation brings benefits from two perspectives. From the cost aggregation point of view, the feature map of the feature cost volume can disambiguate the initial noisy cost volume by refer- ring to semantic -ware aware features as demonstrated in the stereo matching literature (Yoon & Kweon 2006 Hosni al., 2012 He al.) 201T), i.e., cost volume filtering. From the feature aggregation point of view, the cost volume explicitly represents the similarity of features in one image with re- spect to the features the other, and accounting for it drives the features in each image to become more compatible with those of the other. As the iterations unfold, this process will encourage both aggregations to benefit each other. In the end, the resultant feature representations will be more robust and discriminative as shown in Fig, (g-h).
To compute self-attention, we take a different approach from other works (Sun et al. 2021 Cho et al. 202I) to define the query, key and value embeddings. Concretely, we define two independent value embeddings, specifically one for feature projection and the other for cost volume projection. Formally, define the query, key and values as:
Q == P&((D, C)), = (DD,
VD Pr)
Vc Pvv
where Vp and Ve denote the value embeddings of feature descriptors and the cost volume, respec- tively. After computing an attention map by applying softmax over the query and key dot product, use it to aggregate feature D and cost volume C with VD and Ve using Eq. ass follows:
QKT
Attention D) = softmax
VD.
ddk
QKT
Attentionaar- softmax
Vdx /dz) \VC.
Note that any type of attention computation be utilized, i.e., additive (Bahdanau al. 2014)
or dot product (Vaswani et al. 2017), while in practice we use the linear kernel dot product with the associative property of matrix products (Katharopoulos attention are denoted as D', D/, and C".
2020). The outputs of this self-
Cross-A Attentonnn with Matching Distribution. In the proposed cross-attention layer, the aggre- gated features and cost volume are explicitly used for further aggregation, and we condition both feature descriptors both input images via this layer. By exploiting the outputs of the self- self-attention layer, the cooss-tteention layer performs cross cross-atteniss attention between feature descriptors for further fea- ture aggregation using the improved feature descriptors D',, D' and enhanced cost volume C' from earlier aggregations.
As shown in Fig, attention map, since applying a softmax function over the cost volume is tantamount to obtaining the input cost volume is
first apply convolution the input cost volume and treat the output
an attention map. In this way, enhanced aggregation is enabled, transformed to represent a sharpened matching distribution. With a cross-attention map and value for the attention score defined as QK" C" and VD Pvo (D'), respectively, the subsequent attention process for cross-attention is then defined as follows:
Attention (C', D') softmax( C"
VD
(5)
vdx
The outputs of this cross-attention are denoted as D and DJ, and C" is constructed using D" and DJ. The proposed attention layers interleaved, and they are stacked N times to facilitate the aggregations and increase the model capacity.
4.4 COARSE-TO-FINE FORMULATION
To improve the robustness of fine-scale estimates and enhance the benefits one aggregation gains from the other, we extend our architecture to a coarse-to-fine tonfine approach through pyramidal process-
ing, done in (Jeon et al., 2018 Melekhov et al. 2019 Truong et al. 2020b Hong & Kim 2021

---

Published as a conference paper at ICLR 2024
Comolo
Cameid
Uppampliit
C²
Convéd
Conuid
Comed
Canvid
C
Conoed
Canved
Cost Volume
Feature
Linear
Linea
Liner
D2
Salf Altimitioon
Cross Atenton
Crose-Aitentin
Self Not--it
X
Liner
Lincar
D1-
X.
Figure 5: Overall architecture of the proposed method. Given feature maps D3 and D, and the cost volume C' as inputs, our method employs self- and cross-attention specifically designed to conduct joint feature aggregation and cost aggregation in coarse-to manner.
We first use a coarse pair of refined feature maps and aggregated cost volume, and similar to Zhaoo al. 202T) that learns complementary correspondence by adding the cost volume of the previous scale, we progressively learn complementary descriptors and correspondences and encourage the coarser outputs to enhance the subsequent aggregations.
Formally, given the outputs of the attention block each level, Drr D. and C"dd, where denotes the 1-th level, we upsample the aggregated features using bilinear interpolation and add them to the raw feature descriptors extracted from 1, and 4, defined the next level: D D + up(("")), where D is defined similarly. Note that we let the output cost volumes of self- and cross-attention at each level, Cd and C"rd,, undergo convolution and residual connections, i.e. C = C + Conv-d((C"dy, to facilitate the training process. Then, we define the next-level cost volume as C'+1 =C+11 C
Finally, given the features D and D at each level, we compute the cost volume, and the sum of all cost volumes across all levels are added up to obtain the final output C* that is used to estimate the final flow field, as shown in the bottom of Fig. S We empirically find that increasing N and I appreciably boosts correspondence performance. However, there also exists speed-accuracy trade- off of choosing N and 1, for which we provide an analysis in the supplementary material.
4.5 INFERENCE: DENSE ZOOM-IN
At the inference phase, we leverage multi-scale predictions to predict highly accurate correspon- dences. The goals of this approach two-fold: prevent large memory increase when pro- cessing high-resolution input image pairs, e.g., HD or Full HD, and to capture possible fine-grained correspondences missed by the coarse-to-fine design.
Dense zoom-in consists of three stages, For the first stage, UFC takes an input image pair and uses the output flow coarsely align the source image to the target image as similarly done (Shen et al. 2020). Unlike RANSAC-Flow (Shen et al. 2020), we do not resort to finding homography transtormation, but rather rely on the output flow itself We empirically find that for images with extreme geometric deformations, reliable homography transformations may not be found.
Subsequently, we evenly partition the coarsely aligned source image and the target image into & x local windows, where h is hyperparameter. Each pair of partitioned local windows at the same location then used to find more fine-grained correspondences by feeding them into UFC to obtain local flow fields. Note that in this stage, we also compute cycle-consistency confidence score Jiang et al. (2021) that will be used in the final Docision-maainn process. To enable multi-scale inference, choose multiple k, e.g., k material. We then perform transitive composition Zhou et al. (2016) using the coarse flow and each of the multi-scale flows. Finally, using the confidence values of composited flows at each pixel,
for which provide ablation study in the supplementary
7

---

Published as a conference paper at ICLR 2024
Teats Image
Kospant Armozziion Res.
FETASCAL
PENNILION
Methods
hell, App
Cost Agp
PUK Ohee 0.1
PUK
Cheun
POC
Res.
0.01
0,15
0.1
0.05
DRIPF
240 240
240
RHM UT RIMM
L.74
11.0
20.9
57.3
47.5
15.7
59.7
95.0
49.5
77.6
71.0
SCOT
Max 300
35.6
63.1
55.4
92.7
47.8
76.0
CHM
240x: 2401
Care
BD Core Trans.
57.5
80.1
01-6
94.9
70-4
CATS
49.9
75.4
92.6
40.7
PRiupC Nc..NN
400 400
4D
79.2
90.1
46.0
SCaOSAN
156 256
81.5
98.3
90.0
VAT
512x 512
512
4D Cans: Trans.
3.17
19.6
35.0
55.3
76.2
90.1
96.2
81.6
42.5
31.3
CAT
512x 512
512
4D Cans: Tranc
4.11
25.00
40.2
00.5
68.5
84.0
01.3
96.8
56.2
81.1
47.0
72.6
TrssherMaccher
240 240
240
Trans
53.5
80.8
91.3
76.0
65.3
NeMF
512x 512
On
4D Cons, Trans.
3.2
19.5
34.2
53.0
90.5
93.6
60.8
15.0
UFG
51264 512
On
Integrative Frandionner
8.40
34.1
64.4
12.1
88.0
94.8
07.9
58.6
81.
50.4
Table 1: Semantic matching results.
HPadhes Ouigial
ETHED
Methods
Fou.Aax
CootAgg
ARPS
PCK
AEPS
Ave
7pc
S000 15
A0g
DOTR
Trims
7.35
1.05
1.82
1:07
2.13
2:37
2.41
7.51
7.17
BCO TR
Teats
1,72
1.81
3,59
1.97
1,87
COTR, lneep.
Teans
7.98
85.33
1.70
1.52
2.36
2.47
2.55
3.23
3.76
2.59
UFCC+(F) ((F)
Ineegrative Translomer
0.87
1.29
1.37
3.19
1.92
1.75
98.76
1.45
1.59
1.84
1.76
1.82
1.90
1.48
1.73
Coss.
4.24
5.17
GLU.Nes GOCr
Hand
1.29
2.28
2.54
7,50
HMM
3D Grne.
Cons.
131
15.54
5/
11.67
65.43
10.4
63:51
3.83
5.31
4.41
1.56
5.55
0.55
14:30
6.0
PDCNe (MS)
7D Cors
1.18
7.45
11.64
71.00
10.00
15.14
91.41
1.09
1.79
3.00
1.76
2.47
2.00
3.56
1.35
GAlFoos
Teans
4.72
26.45
40.75
02.49
70.30
43.85
69.50
1.64
1.56
2.12
2.35
3.40
5.62
10.64
3.56
COTRI
Trans
19165
45.81
02.03
5:10
9.56
13:27
16.09
1.36
2.35
1.34
UFC
Ireegrative Transormer
191
6.13
5,62
6.76
19.31
7.88
19.36
1.54
1.72
1.99
7.18
3.58
2.66
3.01
2.24
Table 2: Geometric matching results. A higher scene label or rate, i.e., or 15, consists of more difficult images with extreme geometric deformations. f: Dense evaluation without zoom-in technique and confidence thresholding. (C): Confidence thresholding.
select the flow with the highest confidence score. This selection is performed for every pixel and results in a final dense flow map.
5 EXPERIMENTS
5.1
SEMANTIC MATCHING
We first evaluate ours on semantic matching, where large intra-class variations and background clut- ters pose additional challenges to matching. We three standard benchmarks: SPair-71k (Min)
ct al. 2019b), PE-PASCAL (Ham et al. 2017) and PF-WILLOW (Ham et al. 2016). We follow the evaluation protocol of (Cho et al. (022b). The results are summarized in Table I where UFC outper- forms others for all the benchmarks at almost all PCKs, showing robustness to the above challenges. While VAT (Hong et al., 2022a) and NeMF (Hong et al. (2026)) perform better at albbxx we stress that VAT evaluates at higher resolution and NeMF specializes in fine- amne-rrined correspondences. Also, we note that our performance is slightly inferior to SCOT (Liu et al., (020)) and PWarPC (Truong et al. 2022) for Ohbox kp = 0.1, but this is compensated for by the superior performance at lower al- pha, t.e., 0.05, and the fact that PF-PASCAL (Ham et al. 2017) and PF-WILLOW (Ham et al. 2016) are small-scale datasets with a limited number of image pairs. Moreover, we highlight that for SPair- 71k (Min et al. 2019b), the largest dataset in semantic correspondence with extreme viewpoint and scale difference, UFC outperforms competitors at all PCKs.
5.2 GEOMETRIC MATCHING
We next show that our method also performs very well in geometric matching. Following the eval- uation protocol of (Truong et al. 202T). we report the results on HPatches (Baintas et al. 2017) and ETH3D (Schops et al.] (017)) in Table 2 From the results, our method elearly outperforms exist- ing dense matching networks, including those that perform additional optimization (Truong et al. 2020a Hong& & Kim 202T Truong et al. (02T) and inference strategies (Jiang et al.) 2021: Truong et al. 2021). and representative optical flow method, GMFlow (Xu et al.] 2021). Note that UFC excels at finding correspondences under extreme geometric deformations, i.e., scene IV and V. In- terestingly, as UFC outperforms others at all intervals of ETH3D that consist of image sequences with varying magnitudes of geometric transformations, this indicates that it can also perform well in optical flow settings. This is supported by the results of GMFlow (Xu et al. 2021), where consistently achieve better performance. To ensure a fair comparison to COTR (Jing et al. 202T)
and its follow-up works (Revaud et al. 2022 Tan al. 2022), we present results from variant of
8

---

Published as a conference paper at ICLR 2024
method, denoted as (C). Moreover, we also include COTRi to represent a truly dense version of COTR (Jiang et al. 2021 and we observe that UFC clearly performs better.
5.3 QUANTITATIVE COMPARISON BETWEEN AGGREGATION STRATEGIES
Figures and qualitatively show that the information both aggregations exploit and the
HiFacces AFPE
SFain
(beox
(1)
Peature self atl.
78.2
36.
output they generate differ. Here, we empiri- cally show that these lead appreciable per-
(1))
Festare self-att, and cross-att.
59.9
38.5
(T)
Cost self-uit.
$1.8
33.6
formance differences. Table(3 (I-III) compares
(V)
(IV)
Fearare selr-att. cast seif-art.
Festure welf- and crossall.. self-ont.
26.2
36,8
51.7
56.5
different aggregation strategies that are trained and evaluated on both tasks. Note that for these
Table 3: Comparison of aggregation strategies.
variants, we do include any module for boosting performance, c.g., coarse-to-fine multi-level or multi-scale features, and maintain a similar number of learnable parameters. Pytorch-like pseu- docodes and additional visualizations are given in the supplementary material. From the results, we find that each aggregation strategy yields apparently different results in two tasks as reported in (I-III). A particularly illustrative comparison (I) vs. (III), where only self-attention is performed on features or the cost volume, which clearly differentiate them. As expected, we also observe that performing tross-attention improves performance.
In the last two rows, we report the results of variants that utilize both feature and cost aggregation. (V) is our integrative aggregation, and (IV) is a naive sequential aggregation where feature aggre- gation is followed by cost aggregation. They both achieve large performance boosts, expected since the resultant cost volume in Fig. B (f) and (h) includes less noisy and more concentrated scores compared (c-e). while integrative aggregation clearly performs better. From these quantitative comparisons, we highlight that the two types of aggregation serve different purposes that lead to apparent performance differences, and the potential benefits arising from their relationship can be further exploited with our proposed design.
5.4 ABLATION STUDY
In this ablation study, we verify the need for each component of method. Table I presents quantitative results of each variant on
Components
Malcches ABPE 1
Ohava =,11
SPair-1k
(1)
Bascline
36.8
51.7
Integraivve
both geometric and semantic matching. The baseline represents variant equipped with self- and cross-attention feature maps. From (II) to (VI), each proposed component is pro- gressively included. We control the number of learnable parameters for each variant be similar.
(II)
Integrative self- and cross-atl.
22.8
58.1
(IV)
mutching distribution.
18.7
59.9
(V)
hicrerchical processing
10.9
64.4
dense
64.4
Table 4: Component ablation study.
Comparing (I) and (II), we find that the proposed integrative self ceff-attention layer benefits from joint aggregation of features and cost volume, achieving improved performance on both tasks. We next find that each component clearly helps to boost performance. Interestingly, we find dramatic im- provements when cross-attention is included (III), indicating that explicit conditioning between in- put images is helpful. This further enhanced by using the aggregated cost volume cross- attentioon map (IV). We empirically find that our dense zoom-in brings marginal gain for semantic matching. We believe the sparse keypoint evaluation metric and the relatively small resolution images of the benchmark disturb harnessing the advantages of our inference strategy.
CONCLUSION
In this paper, we introduced a simple yet effective dense matching approach, Unified Feature and Cost Aggregation with Transformers (UFC), that capitalizes on the distinct advantages of the two types of aggregation. We further devise an enhanced aggregation through cross-altention with matching distribution. This method is formulated in a coarse-to-fin fine manner, yielding an appre- ciable performance boost. We have shown that approach exhibits high speed and efficiency and that it surpasses all other existing works on several benchmarks, establishing new state-of ofate-of-the-ar performance.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This research was supported by the MSIT, Korea (ITT--20242020--0011811 ICT Creative Consilience Program, RSS2023-00227592, Development of 3D Object Identification Tech- nology Robust to Viewpoint Changes), and National Research Foundation of Korea (NRF- A02RRIA6AAA0304540
REFERENCES
Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual de- scriptors, arXiv preprint arXiv:21 2(3):4, 2021.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1rx1v 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv: arXiv: 4499.4473 2014.
Tim Bailey and Hugh Burrant-Whyte Simultaneous localization and mapping (slam): Part ii. IEEE robotics & automation magazine, 13(3):108-1) 2006.
Vassileios Balntas, Karel Lenc, Andrea Vedaldi, and Krystian Mikolajczyk. Hpatches: bench- mark and evaluation of handcrafted and learned local descriptors. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5173-5182, 2017.
Herbert Bay, Tinne Tuytelaars, and Luc Van Gool. Surf: Speeded up robust features. In European conference on computer vision, pp. 404-417. Springer, 2006.
Minsu Cho, Suha Kwak, Cordelia Schmid, and Jean Ponce. Unsupervised object discovery and localization in the wild: Part-based matching with bottom-up region proposals. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1201-1210, 2015.
Scokju Cho, Sunghwan Hong, Sangryul Jeon, Yunsung Lee, Kwanghoon Sohn, and Seungryong Kim. Cats: Cost aggregation transformers for visual correspondence. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
Seokju Cho, Sunghwan Hong, and Seungryong Kim. Cats++: Boosting cost aggregation with con- volutions and transformers. IEEE Transactions on Pallern Analysis and Machine Intelligence, pp- 1-20, 2022a. doi: 10. 10.1099/PAAM.222223221117
Seokju Cho, Sunghwan Hong, and Seungryong Kim. Cats++: Boosting cost aggregation with con- volutions and transformers. arXiv preprint arXiv:2202.06817 2022b.
Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In CVPR Workshops), 2005.
Daniel DeTooe, Tomasz Malisiewic7, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and deseription. In Proceedings of the IEEE conference computer vision and pattern recognition workshops, 224-236, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Marc Pollefeys, Josef Sivie, Akihiko Torii, and Torsten Sattler. D2-net: A trainable cnn for joint description and detection of local features. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pp. 8092- 8101, 2019.
Johan Edstedt. Ioannis Athanasiadis, Märten Wadenbäck. and Michael Felsberg, Dkm: Dense ker- nelized feature matching for geometry estimation. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 17765-17775, 2023.
10

---

Published as a conference paper at ICLR 2024
Martin A Fischler and Robert C Bolles. Random sample consensus: paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24 (6):381-395, 1981.
Bumsub Ham, Minsu Cho, Cordelia Schmid, and Jean Ponce. Proposal flow. In CVPR, 2016.
Bumsub Ham, Minsu Cho, Cordelia Schmid, and Jean Ponce. Proposal flow: Semantic correspon- dences from object proposals. IEEE transactions on pattern analysis and machine intelligence, 2017.
Kaiming He, Christoph Rhemann, Carsten Rother, Xiaoou Tang, and Jian Sun. global sampling method for alpha matting. In CVPR 2011, pp. 2049-2056. IEEE, 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
Sunghwan Hong and Seungryong Kim. Deep matching prior: Test-time optimization for dense correspondence. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Sunghwan Hong, Seokju Cho, Jisu Nam, Stephen Lin, and Seungryong Kim. Cost aggregation with 4d convolutional swin transformer for few-shot segmentation. arXiv preprint arXiv:2207.10800 2022a.
Sunghwan Hong, Jisu Nam, Seokju Cho, Susung Hong. Sangryul Jeon, Dongbo Min, and Seungry- ong Kim. Neural matching fields: Implicit representation of matching fields for visual correspon- dence. Advances in Neural Information Processing Systems, 11351-135200 2022b.
Asmaa Hosni, Christoph Rhemann, Michael Bleyer, Carsten Rother, and Margrit Gelautz. Fast cost- volume filtering for visual correspondence and beyond. IEEE transactions on pattern analysis and machine intelligence, 35(2):504-511, 2012.
Shuaiyi Huang, Qiuyue Wang. Songyang Zhang, Shipeng Yan, and Xuming He. Dynamic context correspondence network for semantic alignment. In ICCV, 2019.
Shuaiyi Huang, Luyu Yang, Bo He, Songyang Zhang, Xuming He, and Abhinav Shrivastava. Learn- ing semantic correspondence with sparse annotations. In Proceedings of the European Conference on Computer Vision(ECCV), 2022a.
Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Flowformer: A transformer architecture for optical flow. arXiv preprint arXiv:2203.16194 2022b.
Sangryul Jeon, Seungryong Kim, Dongbo Min, and Kwanghoon Sohn. Parm: Pyramidal affine regression networks for dense semantic correspondence. In Proceedings of the European Confer- ence on Computer Vision (ECCV), pp. 351-366, 2018.
Sangryul Jeon, Dongbo Min, Seungryong Kim, Jihwan Choe, and Kwanghoon Sohn. Guided se- mantic flow. In ECCV. Springer, 2020.
Wei Jiang, Eduard Trulls, Jan Hosang, Andrea Tagliasacchi, and Kwang Moo Yi. Cotr: Correspon- dence transformer for matching across images. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6207-6217, 2021.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mnn:: Fast autoregressive transformers with linear attention. In International Conference on Ma- chine Learning, pp. 5156-5165. PMLR, 2020.
Junghyup Lee, Dohyung Kim, Jean Ponce, and Bumsub Ham. Sfnet: Learning object-aware seman- tic correspondence. In CVPR, 2019.
Shuda Li, Kai Han, Theo W Costain, Henry Howard-Jenking, and Victor Prisacariu. Correspondence networks with adaptive neighbourhood consensus. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 10196- 2020.

---

Published as a conference paper at ICLR 2024
Zhaoshuo Li, Xingtong Liu, Nathan Drenkow, Andy Ding, Francis X Creighton, Russell H Taylor, and Mathias Unberath. Revisiting stereo depth estimation from a sequence-to--eeqenee to-sequence perspec- tive with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6197-6206, 2021.
Biyang Liu, Huimin Yu, and Guodong Qi. Graftnet: Towards domain generalized stereo matching with a broad -pectrumm and task-oriented feature. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13012-13021, 2022.
Ce Liu, Jenny Yuen, and Antonio Torralba. Sift flow: Dense correspondence across scenes and its applications. IEEE transactions on pattern analysis and machine intelligence, 33(5)977-999,, 2010.
Yanbin Liu, Linchao Zhu, Makoto Yamada, and Yi Yang. Semantic correspondence optimal transport problem. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Ze Liu, Yutong Lin. Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint MrXiv:2103.14053 2021.
David G Lowe. Distinctive image features from acale-invariant keypoints. International journal of computer vision, 60(2):91-110, 2004.
Jiachen Lu, Jinghan Yao, Junge Zhang, Xiatian Zhu, Hang Xu, Weiguo Gao. Chunjing Xu, Tao Xiang, and Li Zhang. Soft: Softmax-free transformer with linear complexity. arXiv preprint arXiv:21 10. 19945, 2021.
laroslav Melekhov, Aleksei Tiulpin, Torsten Sattler, Marc Pollefeys, Esa Rahtu, and Juho Kannala. Dge-net: Dense geometric correspondence network. In WACV, 2019.
Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Hyperpixel flow: Semantic correspondence with multi-layer neural features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019a.
Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Spair-71k: A large-scale benchmark for semantic correspondence. arXiv preprint arXiv:1908. 10543, 2019b.
Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho. Learning to compose hypercolumns for 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XV 16. Springer, 2020.
visual correspondence. In Computer
Juhong Min, Seungwook Kim, and Minsu Cho. Convolutional hough matching networks for robust and efficient visual correspondence. arXiv preprint arXiv:2109.0522. 202
Juhong Min, Seungwook Kim, and Minsu Cho. Convolutional hough matching networks for robust and efficient visual correspondence. arXiv preprint arXiv:2109.0522) 2021b.
Yuki Ono, Eduard Trulls, Pascal Fua, and Kwang Moo Yi. Lf-net: Learning local features from images. Advances in neural information processing systems, 31, 2018.
William Peebles, Jun-Yan Zhu, Richard Zhang, Antonio Torralba, Alexei Efros, and Eli Shechtman. Gan-supervised dense visual alignment. arXiv preprint 1rXiv:2112.05143, 2021.
James Philbin, Ondrej Chum, Michael Isard, Josef Sivic, and Andrew Zisserman. Object retrieval with large vocabularies and fast spatial matching. In CVPR. IEEE, 2007.
Jerome Revaud, Philippe Weinzaepfel, César De Souza, Noe Pion, Gabriela Csurka, Yohann Cabon, and Martin Humenherger. R2d2: repeatable and reliable detector and descriptor. arXiv preprint arXiv:1906.0 190.061906006995. 2019.
Jérome Revaud, Vincent Leroy, Philippe Weinzaepfel. and Boris Chidlovskii. Pump: Pyramidal and uniqueness matching priors for unsupervised learning of local descriptors. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 3926-3936, 2022.
12

---

Published as a conference paper at ICLR 2024
Ignacio Rocco, Mircea Cimpoi, Relja Arandjelovié. Akihiko Torii, Tomas Pajdla, and Josef Sivic. Neighbourhood consensus networks. arXiv preprint arXiv: 2018.
Ignacio Rocco, Relja Arandjelović, and Josef Sivic. Efficient neighbourhood consensus networks via submanifold sparse convolutions. In ECCV, 2020.
Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In CVPR, 2020.
Daniel Scharstein and Richard Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International jourmal of computer vision, 2002.
Johannes I. Schonberger and Jan-Michael Frahm. Structure-frommmen of the IEEE conference on computer vision and pattern recognition, pp. 4104-4113, 2016.
revisited. In Proceedings
Thomas Schops, Johannes L. Schonberger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Marc Pollefeys, and Andreas Geiger. A multi-view stereo benchmark with high-resolution images and multi-camera videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3260-3269, 2017.
Xi Shen, François Darmon, Alexei A Efros, and Mathieu Aubry. Ransac-flow generic two-stage image alignment. In European Conference on Computer Vision, pp. 618-637. Springer, 2020.
Richard Sinkhorn. Diagonal equivalence to matrices with prescribed row and column sums. The American Mathematical Monthly, 1967.
Xiao Song, Guorun Yang, Xinge Zhu, Hui Zhou, Zhe Wang. and Jianping Shi. Adastereo: A simple and efficient approach for adaptive stereo matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. (0328-10337, 2021.
Deqing Sun, Xiaodong Yang, Ming-Yu Yu Liu, and Jan Kautz. Pwe-net: Cons for optical flow using pyramid, warping, and cost volume. In CVPR, 2018
Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8922-89311 2021.
Dongli Tan, Laagg-Jiagg Liu, Xingyu Chen, Chao Chen, Ruixin Zhang, Yunhang Shen, Shouhong Ding, and Rongrong Ji. Eco-tr: Efficient correspondences finding via coarse-to-fine refinement. In European Conference on Computer Vision, pp- 317-334. Springer, 2022.
Engin Tola, Vincent Lepetit, and Pascal Fua. Daisy: An efficient dense descriptor applied to wide- baseline stereo. IEEE transactions on pattern analysis and machine intelligence, 32(5):815-830, 2009.
Prune Truong, Martin Danelljan, Lue V Gool, and Radu Timofte. Gocor: mized correspondence volumes into your neural network. Advances in Neural Information Pro- cessing Systems, 11778-14299, 2020a.
Bringing globally opti-
Prune Truong, Martin Danelljan, and Radu Timofte. Glu-net: Global-local universal network for dense flow and correspondences. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 0258-6268, 2020b.
Prune Truong, Martin Danelljan, Luc Van Gool, and Radu Timofte. Learning accurate dense corre- spondences and when to trust them. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, 2021.
Prune Truong, Martin Danelljan, Fisher Yu, and Lue Van Gool. Probabilistic warp consistency for weakly-supervised semantic correspondences. arXiv preprint arXiv:2203.042799 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, 2017.
13

---

Published as a conference paper at ICLR 2024
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang. and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04766. 2020.
Chuhan Wu, Fangzhao Wu, Tao Qi, Yongfeng Huang, and Xing Xie. Fastformer: Additive attention can be all you need. arXiv preprint arXiv:2108. 09084, 2021.
Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. Gmflow: Learning optical flow via global matching. arXiv preprint arXiv:2111. 2021.
Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao, and Andreas Geiger. Unifying flow, stereo and depth estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Gengshan Yang and Deva Ramanan. Volumetric correspondence networks for optical flow. Ad- vances in neural information processing systems, 32, 2019.
Kuk-Jin Yoon and In So Kweon. Adaptive support-weight approach for correspondence search.
IEEE transactions on pattern analysis and muchine intelligence, 28(4):650-656, 2006.
Dongyang Zhao, Ziyang Song, Zhenghao Ji, Gangming Zhao, Weifeng Ge, and Yizhou Yu. Multi- scale matching networks for semantic correspondence. In Proceedings the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 3354-3364, 2021.
Tinghui Zhou, Philipp Krahenbuhl, Mathicu Aubry, Qixing Huang, and Alexei A Efros. Learning dense correspondence via 3d-guided cycle consistency. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 117-126, 2016.
14


Reviews:
[{"review_id": "iPnM5osvMB", "reviewer": "Reviewer_96VU", "summary": "The paper introduces an integrative feature and cost aggregation modules in a CNN architecture for a semantic correspondence task. The introduced module cleans up noisy matches in the cost volume and thus improves the matching accuracy. The paper demonstrates better accuracy on semantic matching and geometric matching tasks by using their method.", "questions": "- Increase of learnable parameters\n\n  How many number of learnable parameters account for the new module (i.e., integrative feature and cost aggregation module)? How significant are they compared to the number of parameters of the entire network (15.5M)? Probably it's also good to include an extra column in Table 3 and 4 for the number of network parameters.\n\n- Resolution of the cost volume\n\n  What's the resolution of the cost volume (saying the input image resolution is HxW)? At each level the features are upsampled ($D^{l}_s$), but how can the resolution of the cost volume ($C^{l})$ remain the same over different pyramid levels? Is there any reason to fix the resolution of the cost volume?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Good results\n\n  The paper achieves good accuracy on both semantic and geometric matching tasks (Table 1 and 2). It demonstrates the effectiveness of the proposed aggregation modules.\n\n- Detail analysis\n\n  The paper provides a sufficient amount of analysis. Fig. 3 visualizes the qualitative comparison of the proposed modules (from (f) to (h)). Further, the ablation study (Table 3 and 4) validates the proposed ideas.", "weaknesses": "Despite the good accuracy on both tasks, there are concerns about novelty/contributions.\n\n- Existing ideas in other literature\n\n  Similar ideas on feature and cost volume aggregation have been demonstrated in other literature such as stereo matching [a,b] and optical flow estimation [c]. Actually the related work section (Sec. 2) summarizes those relevant papers very well. Compared to the existing solutions, what would be the new technical design (in self-/cross-attention) of the proposed module, except for applying it to semantic & geometric matching problems? Can the newer technical design from the proposed module also benefit other tasks that use cost volume, eg., stereo matching, optical flow, scene flow, etc. ?\n\n   [a] Attention-Aware Feature Aggregation for Real-time Stereo Matching on Edge Devices, ACCV 2020\n   \n   [b] Attention Concatenation Volume for Accurate and Efficient Stereo Matching, CVPR 2022\n   \n   [c] GMFlow: Learning Optical Flow via Global Matching, CVPR 2022\n\n- Limitation\n\n  Discussion on the limitation is missing. What would be the limitation of the method or unsolved problems?\n\n\n- Can the paper provide more qualitative examples and discuss where the gain mainly originates?\n\n  Table 4 shows the accuracy improvement by adding more components. I am wondering if the paper can also include some qualitative examples and discuss where the gain mainly originates, such as resolving some particular matching ambiguity. It would be great if the paper can provide more insights related to its improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VZu8hjoEUk", "reviewer": "Reviewer_4EyK", "summary": "This paper presents a new vision transformer architecture to conduct feature aggregation and cost aggregation for dense matching tasks. The authors show distinct characteristics of feature aggregation and cost aggregation. and use self- and cross-attention mechanisms to unify the feature and cost aggregation. They validate the effectiveness of the proposed method with semantic matching and geometry matching.", "questions": "1. The local discriminative ability of features is also important for dense matching tasks. I would like to see an analysis of whether the proposed method causes damage in this perspective or whether this issue can be avoided in some design of the method.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea to unify feature aggregation and cost aggregation is interesting. It compensates for the lack of semantic information in cost representation and helps to drive the features in each image to become more compatible with others.\n2. They conduct extensive experiments on semantic matching and geometry matching to validate the effectiveness of the proposed method UFC. UFC can improve the matching performance. And they provide step-by-step ablations of each component.\n3. The paper is well-organized and easy to understand. The authors visualize the changes in feature maps and cost volumes, which helps understand how their method works. I see that feature aggregation can preserve semantic information and geometry structure and the cost aggregation reduces the noise in cost volumes.", "weaknesses": "1. In visualization results Figure 2, features with integrative aggregation methodology preserve the semantic information. However, it seems the proposed method damages the local discriminative ability of features.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pjApO7ILGu", "reviewer": "Reviewer_gxEA", "summary": "This paper proposes to combine feature and cost aggregation to address the dense feature matching task.\n\nThe main idea is to use cost score matrix for both self- and cross-attention feature updating, to learn more discriminative features and compute better cost matrix.\n\nExperiments on some semantic and geometric matching datasets show the effectiveness of the proposed method.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1) This paper is generally presented well;\n\n2) The idea of use cost score matrix for both self- and cross-attention feature updating is simple and effective;\n\n3) Experimental results are good.", "weaknesses": "1)  Section 5.1. It is quite blurry for me whether previous state-of-the-art methods in Table 1 and 2 are trained on the same datasets; For example, the proposed method is trained on the DPED-CityScape-ADE and MegaDepth datasets;\n\n2) The performance of the proposed method on the optical flow (KITTI, Sintel) task is blurry for me;\n\n3) It's good to see the improved matching performance on the HPatches dataset. However, I want to see whether the improved matching would lead to better Rotation and translation estimations. \n\n4) Using cost score matrix for both self- and cross-attention feature updating is good. However, this contribution may be constrained to large overlapping ratio between images. If pairwise images have small overlapping ratio, the cost score matrix is noisy, and may provide wrong guidance for feature updating. Would you please check whether the proposed method works on some challenging image pairs from the MegaDepth dataset.\n\n5) Please show some failure cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nclKPlabOl", "reviewer": "Reviewer_HY22", "summary": "This paper proposed a dense matching estimation method by unifying both feature and cost volume aggregation with transformer. The authors first analyze the merits and faults of feature and cost aggregation and then claim that using them interleavely could improve the feature representation. Experiment results show the effectiveness of the proposed method.", "questions": "As discussed in the related works, many stereo-matching and optical flow works use Transformer-based cost aggregation networks. \nThe idea proposed in this paper should be a general way to improve all feature matching-based tasks, and I think there are no enormous model differences among stereo, flow, and dense matching. So the authors should compare these SOTA stereo and flow methods in a more fair way. For example, re-training the model with the same data setting for dense matching or verifying the effectiveness of the proposed method in stereo and flow estimation benchmarks.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. To the best of my knowledge, this is the first work that discusses the relationship between feature and cost aggregation (learning). The authors carefully discuss their merits and faults in Sec.1 and Sec.3.\n2. Interleavely aggregating features between feature and cost volumes is interesting. The visualization results of Fig.2, and Fig.3 also verify the effectiveness.\n3. The authors provide complete experimental details in the supplementary.", "weaknesses": "1. The usage of \"aggregation\" is a little confusing. In my opinion, \"aggregation\" means combining multiple features into a single one, which should usually be used to describe the process of attention aggregation of ($QK^T$ and $V$). However, I am not sure whether \"aggregation\" is suitable to be used to indicate the whole learning process of cost volume learning. Because many cost volume learning is not related to attention learning.\n2. Although the authors analyze the merits of feature/cost aggregation, some claims have not been clarified. For example, feature matching is \"challenged by repetitive patterns and background clutters\", while cost volume learning enjoys \"robustness to repetitive patterns and background clutter\". No evidence is shown in this paper to support this claim.\n3. The authors did not formulate the method presentation well in Sec.4 and Fig.4, which makes the proposed method suffer from too complicated designs and difficult to follow. I strongly recommend the authors introduce the shape and reshape of the most important tensors to make the whole pipeline clearer. The concatenation in Eq(3) is operated along which dimension? Why $C'$ appears again in Eq.5 as $QK^T$, while $C'$ should be already defined as the output of the cost volume feature?\n4. The experiments are not solid enough. The proposed method needs to be compared with more recent methods. In the geometric matching results from Tab2, most competitors are from 2020 and 2021, which are far from \"state-of-the-art\". Only one flow estimation method is considered (GMFlow). However, as discussed in the supplementary, the comparison is not fair, because GMFlow is trained on Sintel rather than DPED-CityScape-ADE+MegaDepth fine-tuning. Besides, all these competitors are trained on DPED-CityScape-ADE **or** MegaDepth (the proposed method is trained with DPED-CityScape-ADE **and** MegaDepth finetuning) as said in supplementary B.1. The authors should clarify this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)