Supplementary Material:
LoFTR: Detector-Free Local Feature Matching with Transformers

1. Formulation Details

Timing(ms)Process

Positional Encoding. We use the 2D extension of the absolute sinusoidal positional encoding following [1]:sin (ωk · x) , i = 4kcos (ωk · x) , i = 4k + 1
PE ix,y = f (x, y)i :=,sin(ωk · y) , i = 4k + 2cos (ωk · y) , i = 4k + 3

Full model(OT / DS)

Small model(OT)

Local Feature CNN
Coarse-Level LoFTR
Matching Module
Coarse-to-Fine Module

48.2
33.6
21.0 / 12.0
17.2 / 19.7

37.4
33.5
9.1
13.1

Overall

121 / 116

96

Table 1: Timing measurements. Time cost for an image
pair of 640×480. Both the full model of LoFTR-OT and
LoFTR-DS and the small model of LoFTR-OT are tested.

where ωk = 100001 2k/d , d is the number of channels of features which positional encoding applied on, i is a index forfeature channels.

Unlike the original FPN, we use various widths for different levels, which are the same as the corresponding stages
in the bottom-up ResNet.

2. Timing
We test LoFTR’s speed on 100 randomly sampled ScanNet image pairs, with a size of 640 × 480, and report the
averaged results. A RTX 2080Ti GPU is used for timing. For the full model, both the optimal transport and thedual-softmax variants are reported. As illustrated in Tab. 1,
LoFTR-DS runs faster on average than LoFTR-OT. The difference comes mainly from the matching module. Whenusing a smaller model with 1/16 of the original resolution at
the coarse-level and 1/4 at the fine-level (i.e., the second ablation experiment in the main text), the speed is about 20%
faster. Note that performance may vary over different image
pairs and different matching layers, mainly caused by thedifferent number of matches for the coarse-to-fine model torefine. We notice that a large fraction of time is spent on the
Local Feature CNN, which is caused by the use of FPN anda design flaw in the widths of the FPN backbone (we use awidth of 128 for the ResNet stem and the first stage). We
expect to fix this problem in our future released models.

LoFTR Encoder Layer. The original Transformer [10]is composed of a Transformer encoder and a Transformer
decoder, both of which consist of several homogeneous encoder or decoder layers. Our LoFTR module is an encoderonly architecture consists of cascaded LoFTR encoder layers. The LoFTR encoder layer inspired by [8, 7] is a variantof the Post-LN encoder layer originally used in [10]. Fig. 1
illustrates their differences. In our experiments, the PostLN layer(Fig. 1(b)) requires careful tuning of the learning
rate and its scheduling and suffers from slow convergence.
Our variant has fast convergence without extensive hyperparameter tuning. For normalization layers, we use the layer
normalization, while batch normalization also fits into our
encoder layer and performs comparatively. As mentioned
in the main paper, we use 4 encoder layers in the coarselevel and 1 encoder layer in the fine-level. We use 8 heads
in all multi-head attention layers. Input features have 256
and 128 channels respectively for the coarse- and fine-level
LoFTR modules, both from the FPN backbone.

3. Implementation Details

Differentiable Matching with Optimal Transport.
LoFTR can utilize the optimal transport layer
proposed in [8] to extract matches in coarse-level. Similar
to [8], we express the pairwise score as the similarity ofAB:coarse-level LoFTR features F̃trand F̃tr3.1. Architecture
Local Feature CNN. The Local Feature CNN is built witha modified ResNet-18 with FPN. Different from the original
ResNet-18, we use a width of 128 for the stem and widthsof [128, 196, 256] for the subsequent three stages. We build
the FPN with levels P1 through P3 and take features fromP3 as the input features F̃tr for coarse-level LoFTR, features from P1 as the input features F̂tr for fine-level LoFTR.

S (i, j) =1
AB
· hF̃tr(i), F̃tr(j)i,D

where D is the feature dimension of F̃tr . Under the partial
assignment setup, we augment the score matrix S to S with
1

fi

fi

Add

Add&Norm

Feed-Forward&Norm

Feed-Forward

ConcatAdd&Norm

Cat&Linear&Norm

Cat&Linear<latexit sha1_base64="f4jXYSUiJn4KudYnZNr5TSSFBPI=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVRJRdCUFN11WtA+opSTTaR2aF5OJIkXwB9zqp4l/oH/hnTEFtYhOSHLm3HvOzL3XTwKRKsd5LVhz8wuLS8Xl0srq2vpGeXOrlcaZZLzJ4iCWHd9LeSAi3lRCBbyTSO6FfsDb/vhMx9s3XKYiji7VXcJ7oTeKxFAwTxF1MeyLfrniVB2z7Fng5qCCfDXi8guuMEAMhgwhOCIowgE8pPR04cJBQlwPE+IkIWHiHPcokTajLE4ZHrFj+o5o183ZiPbaMzVqRqcE9EpS2tgjTUx5krA+zTbxzDhr9jfvifHUd7ujv597hcQqXBP7l26a+V+drkVhiBNTg6CaEsPo6ljukpmu6JvbX6pS5JAQp/GA4pIwM8ppn22jSU3tureeib+ZTM3qPctzM7zrW9KA3Z/jnAWtg6p7VHXODyu103zURexgF/s0z2PUUEcDTfIe4RFPeLbqVmRl1u1nqlXINdv4tqyHD2hFkEc=</latexit><latexit sha1_base64="f4jXYSUiJn4KudYnZNr5TSSFBPI=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVRJRdCUFN11WtA+opSTTaR2aF5OJIkXwB9zqp4l/oH/hnTEFtYhOSHLm3HvOzL3XTwKRKsd5LVhz8wuLS8Xl0srq2vpGeXOrlcaZZLzJ4iCWHd9LeSAi3lRCBbyTSO6FfsDb/vhMx9s3XKYiji7VXcJ7oTeKxFAwTxF1MeyLfrniVB2z7Fng5qCCfDXi8guuMEAMhgwhOCIowgE8pPR04cJBQlwPE+IkIWHiHPcokTajLE4ZHrFj+o5o183ZiPbaMzVqRqcE9EpS2tgjTUx5krA+zTbxzDhr9jfvifHUd7ujv597hcQqXBP7l26a+V+drkVhiBNTg6CaEsPo6ljukpmu6JvbX6pS5JAQp/GA4pIwM8ppn22jSU3tureeib+ZTM3qPctzM7zrW9KA3Z/jnAWtg6p7VHXODyu103zURexgF/s0z2PUUEcDTfIe4RFPeLbqVmRl1u1nqlXINdv4tqyHD2hFkEc=</latexit><latexit sha1_base64="PsHoLMrVyi9SMBhfAbsVWk11qI4=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVZKC6EoqbrqsaB+gRZLJtA5NkzCZKKUI/oBb/TTxD/QvvDOmoBbRCUnOnHvPmbn3+kkoUuU4rwVrbn5hcam4XFpZXVvfKG9utdM4k4y3WBzGsut7KQ9FxFtKqJB3E8m9kR/yjj881fHOLZepiKMLNU54b+QNItEXzFNEnZ8EwXW54lQds+xZ4Oaggnw14/ILrhAgBkOGETgiKMIhPKT0XMKFg4S4HibESULCxDnuUSJtRlmcMjxih/Qd0O4yZyPaa8/UqBmdEtIrSWljjzQx5UnC+jTbxDPjrNnfvCfGU99tTH8/9xoRq3BD7F+6aeZ/dboWhT6OTA2CakoMo6tjuUtmuqJvbn+pSpFDQpzGAcUlYWaU0z7bRpOa2nVvPRN/M5ma1XuW52Z417ekAbs/xzkL2rWqe1B1zmqV+nE+6iJ2sIt9much6migiRZ5D/CIJzxbDSuyMuvuM9Uq5JptfFvWwwcPgJAg</latexit><latexit sha1_base64="U1rdFGwGh0pcwnJ90JyafWom9uc=">AAACyXicjVHLSsNAFD2Nr1pfVZdugkVwVZKC6EoK3QhuKtgH1CJJOq1j00ycTMRaXPkDbvXHxD/Qv/DOGEEtohOSnDn3njNz7/XjkCfKcV5y1szs3PxCfrGwtLyyulZc32gmIpUBawQiFLLtewkLecQaiquQtWPJvJEfspY/rOl465rJhIvoVI1j1h15g4j3eeApopo1ERE6L5acsmOWPQ3cDJSQrbooPuMMPQgESDECQwRFOISHhJ4OXDiIietiQpwkxE2c4Q4F0qaUxSjDI3ZI3wHtOhkb0V57JkYd0CkhvZKUNnZIIyhPEtan2SaeGmfN/uY9MZ76bmP6+5nXiFiFC2L/0n1m/lena1Ho48DUwKmm2DC6uiBzSU1X9M3tL1UpcoiJ07hHcUk4MMrPPttGk5jadW89E381mZrV+yDLTfGmb0kDdn+Ocxo0K2V3r+ycVErVw2zUeWxhG7s0z31UcYQ6GuR9iQc84sk6tq6sG+v2I9XKZZpNfFvW/TuhNZGN</latexit>h<latexit sha1_base64="YxgTbE4Tg2Dm9Lslt6jOsGRRbvU=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEsceCF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipOR6UK27VXYCsEy8nFcjRGJS/+sOYpRFKwwTVuue5ifEzqgxnAmelfqoxoWxCR9izVNIItZ8tDp2RC6sMSRgrW9KQhfp7IqOR1tMosJ0RNWO96s3F/7xeasKan3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUUWZsNiUbgrf68jppX1W9m6rbvK7Ua3kcRTiDc7gED26hDvfQgBYwQHiGV3hzHp0X5935WLYWnHzmFP7A+fwBzEmM5g==</latexit>

Attention Layer<latexit sha1_base64="TAABLtYdTuqDTFnOGcGO7GJUajc=">AAAB+HicbVDLSgNBEOz1GeMjqx69DAbBU9gNiB4jXjx4iGAekCxhdjKbDJl9MNMrrCFf4sWDIl79FG/+jbPJHjSxYKCo6u7pLj+RQqPjfFtr6xubW9ulnfLu3v5BxT48aus4VYy3WCxj1fWp5lJEvIUCJe8mitPQl7zjT25yv/PIlRZx9IBZwr2QjiIRCEbRSAO7co3Io5yTO5pxNbCrTs2Zg6wStyBVKNAc2F/9YczS0Axhkmrdc50EvSlVKJjks3I/1TyhbEJHvGdoREOuvel88Rk5M8qQBLEyL0IyV393TGmodRb6pjKkONbLXi7+5/VSDK68qYiS1FzHFh8FqSQYkzwFMhSKM5SZIZQpYXYlbEwVZWiyKpsQ3OWTV0m7XnMvas59vdpoFHGU4ARO4RxcuIQG3EITWsAghWd4hTfryXqx3q2PRemaVfQcwx9Ynz+qjZMU</latexit>

Q = WQ f i K = WK f j V = WV fj<latexit sha1_base64="4XUGxwNURwmnEZdHAYd77F3yrjU=">AAAB73icbVBNS8NAEJ34WetX1aOXxSJ4Koko9iIUvHhswX5AG8JmO2mXbjZxdyOU0j/hxYMiXv073vw3btsctPXBwOO9GWbmhang2rjut7O2vrG5tV3YKe7u7R8clo6OWzrJFMMmS0SiOiHVKLjEpuFGYCdVSONQYDsc3c389hMqzRP5YMYp+jEdSB5xRo2VOo3bdtCIAh6Uym7FnYOsEi8nZchRD0pfvX7CshilYYJq3fXc1PgTqgxnAqfFXqYxpWxEB9i1VNIYtT+Z3zsl51bpkyhRtqQhc/X3xITGWo/j0HbG1Az1sjcT//O6mYmq/oTLNDMo2WJRlAliEjJ7nvS5QmbE2BLKFLe3EjakijJjIyraELzll1dJ67LiXVfcxlW5Vs3jKMApnMEFeHADNbiHOjSBgYBneIU359F5cd6dj0XrmpPPnMAfOJ8/adiPhw==</latexit><latexit sha1_base64="8HhSfiQ1YEjRG3PALXyrVmoLveQ=">AAAB73icbVBNS8NAEJ34WetX1aOXxSJ4Koko9iIUvHisYNNCG8Jmu2nXbnbj7kYooX/CiwdFvPp3vPlv3LY5aOuDgcd7M8zMi1LOtHHdb2dldW19Y7O0Vd7e2d3brxwc+lpmitAWkVyqToQ15UzQlmGG006qKE4iTtvR6Gbqt5+o0kyKezNOaZDggWAxI9hYqeNft0M/Dh/CStWtuTOgZeIVpAoFmmHlq9eXJEuoMIRjrbuem5ogx8owwumk3Ms0TTEZ4QHtWipwQnWQz+6doFOr9FEslS1h0Ez9PZHjROtxEtnOBJuhXvSm4n9eNzNxPciZSDNDBZkvijOOjETT51GfKUoMH1uCiWL2VkSGWGFibERlG4K3+PIy8c9r3mXNvbuoNupFHCU4hhM4Aw+uoAG30IQWEODwDK/w5jw6L8678zFvXXGKmSP4A+fzB3q2j5I=</latexit><latexit sha1_base64="NFLzNNQxUfcVflQnMcUwpbaP+HI=">AAAB73icbVBNSwMxEJ34WetX1aOXYBE8lV1R7EUoeBF6qWA/oF2WbJptY7PZNckKZemf8OJBEa/+HW/+G9N2D9r6YODx3gwz84JEcG0c5xutrK6tb2wWtorbO7t7+6WDw5aOU0VZk8YiVp2AaCa4ZE3DjWCdRDESBYK1g9HN1G8/MaV5LO/NOGFeRAaSh5wSY6VO/brt10P/wS+VnYozA14mbk7KkKPhl756/ZimEZOGCqJ113US42VEGU4FmxR7qWYJoSMyYF1LJYmY9rLZvRN8apU+DmNlSxo8U39PZCTSehwFtjMiZqgXvan4n9dNTVj1Mi6T1DBJ54vCVGAT4+nzuM8Vo0aMLSFUcXsrpkOiCDU2oqINwV18eZm0zivuZcW5uyjXqnkcBTiGEzgDF66gBrfQgCZQEPAMr/CGHtELekcf89YVlM8cwR+gzx9Y8I98</latexit>

fi<latexit sha1_base64="f4jXYSUiJn4KudYnZNr5TSSFBPI=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVRJRdCUFN11WtA+opSTTaR2aF5OJIkXwB9zqp4l/oH/hnTEFtYhOSHLm3HvOzL3XTwKRKsd5LVhz8wuLS8Xl0srq2vpGeXOrlcaZZLzJ4iCWHd9LeSAi3lRCBbyTSO6FfsDb/vhMx9s3XKYiji7VXcJ7oTeKxFAwTxF1MeyLfrniVB2z7Fng5qCCfDXi8guuMEAMhgwhOCIowgE8pPR04cJBQlwPE+IkIWHiHPcokTajLE4ZHrFj+o5o183ZiPbaMzVqRqcE9EpS2tgjTUx5krA+zTbxzDhr9jfvifHUd7ujv597hcQqXBP7l26a+V+drkVhiBNTg6CaEsPo6ljukpmu6JvbX6pS5JAQp/GA4pIwM8ppn22jSU3tureeib+ZTM3qPctzM7zrW9KA3Z/jnAWtg6p7VHXODyu103zURexgF/s0z2PUUEcDTfIe4RFPeLbqVmRl1u1nqlXINdv4tqyHD2hFkEc=</latexit>

Coarse-level Supervision. Since the coarse-level LoFTR
module is applied on features at 1/8 of the original image dimension, each of which represents a grid of pixels in original images, there might exist one-to-many matches. This
makes it difficult to accurately determine the ground-truth
coarse matches. We take the mutual nearest neighbors between the central localtions of 1/8 grids of the input images
as a approximated ground-truth. In implementation, we takethe central location of a 1/8 grid in left image, project it to thethe same scale as the depth map, and index its depth. Basedon its depth value and the known camera pose, we warp thegrid center to the right image, and take its nearest neighbor as a matching candidate. The same process is repeatedagain from right to the left image. Based on two sets of
one directional nearest neighbor matches, we only keep themutual nearest ones as the final groud-truth. Note that we
do not enforce a depth consistency check or thresholding
the reprojection distances. We empirically find that further
constraints on the approximated ground-truth matches leads
to models with worse performance.

h<latexit sha1_base64="YxgTbE4Tg2Dm9Lslt6jOsGRRbvU=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEsceCF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipOR6UK27VXYCsEy8nFcjRGJS/+sOYpRFKwwTVuue5ifEzqgxnAmelfqoxoWxCR9izVNIItZ8tDp2RC6sMSRgrW9KQhfp7IqOR1tMosJ0RNWO96s3F/7xeasKan3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUUWZsNiUbgrf68jppX1W9m6rbvK7Ua3kcRTiDc7gED26hDvfQgBYwQHiGV3hzHp0X5935WLYWnHzmFP7A+fwBzEmM5g==</latexit>

Attention Layer<latexit sha1_base64="TAABLtYdTuqDTFnOGcGO7GJUajc=">AAAB+HicbVDLSgNBEOz1GeMjqx69DAbBU9gNiB4jXjx4iGAekCxhdjKbDJl9MNMrrCFf4sWDIl79FG/+jbPJHjSxYKCo6u7pLj+RQqPjfFtr6xubW9ulnfLu3v5BxT48aus4VYy3WCxj1fWp5lJEvIUCJe8mitPQl7zjT25yv/PIlRZx9IBZwr2QjiIRCEbRSAO7co3Io5yTO5pxNbCrTs2Zg6wStyBVKNAc2F/9YczS0Axhkmrdc50EvSlVKJjks3I/1TyhbEJHvGdoREOuvel88Rk5M8qQBLEyL0IyV393TGmodRb6pjKkONbLXi7+5/VSDK68qYiS1FzHFh8FqSQYkzwFMhSKM5SZIZQpYXYlbEwVZWiyKpsQ3OWTV0m7XnMvas59vdpoFHGU4ARO4RxcuIQG3EITWsAghWd4hTfryXqx3q2PRemaVfQcwx9Ynz+qjZMU</latexit>

Q = WQ f i K = WK f j V = WV fj<latexit sha1_base64="4XUGxwNURwmnEZdHAYd77F3yrjU=">AAAB73icbVBNS8NAEJ34WetX1aOXxSJ4Koko9iIUvHhswX5AG8JmO2mXbjZxdyOU0j/hxYMiXv073vw3btsctPXBwOO9GWbmhang2rjut7O2vrG5tV3YKe7u7R8clo6OWzrJFMMmS0SiOiHVKLjEpuFGYCdVSONQYDsc3c389hMqzRP5YMYp+jEdSB5xRo2VOo3bdtCIAh6Uym7FnYOsEi8nZchRD0pfvX7CshilYYJq3fXc1PgTqgxnAqfFXqYxpWxEB9i1VNIYtT+Z3zsl51bpkyhRtqQhc/X3xITGWo/j0HbG1Az1sjcT//O6mYmq/oTLNDMo2WJRlAliEjJ7nvS5QmbE2BLKFLe3EjakijJjIyraELzll1dJ67LiXVfcxlW5Vs3jKMApnMEFeHADNbiHOjSBgYBneIU359F5cd6dj0XrmpPPnMAfOJ8/adiPhw==</latexit><latexit sha1_base64="8HhSfiQ1YEjRG3PALXyrVmoLveQ=">AAAB73icbVBNS8NAEJ34WetX1aOXxSJ4Koko9iIUvHisYNNCG8Jmu2nXbnbj7kYooX/CiwdFvPp3vPlv3LY5aOuDgcd7M8zMi1LOtHHdb2dldW19Y7O0Vd7e2d3brxwc+lpmitAWkVyqToQ15UzQlmGG006qKE4iTtvR6Gbqt5+o0kyKezNOaZDggWAxI9hYqeNft0M/Dh/CStWtuTOgZeIVpAoFmmHlq9eXJEuoMIRjrbuem5ogx8owwumk3Ms0TTEZ4QHtWipwQnWQz+6doFOr9FEslS1h0Ez9PZHjROtxEtnOBJuhXvSm4n9eNzNxPciZSDNDBZkvijOOjETT51GfKUoMH1uCiWL2VkSGWGFibERlG4K3+PIy8c9r3mXNvbuoNupFHCU4hhM4Aw+uoAG30IQWEODwDK/w5jw6L8678zFvXXGKmSP4A+fzB3q2j5I=</latexit><latexit sha1_base64="NFLzNNQxUfcVflQnMcUwpbaP+HI=">AAAB73icbVBNSwMxEJ34WetX1aOXYBE8lV1R7EUoeBF6qWA/oF2WbJptY7PZNckKZemf8OJBEa/+HW/+G9N2D9r6YODx3gwz84JEcG0c5xutrK6tb2wWtorbO7t7+6WDw5aOU0VZk8YiVp2AaCa4ZE3DjWCdRDESBYK1g9HN1G8/MaV5LO/NOGFeRAaSh5wSY6VO/brt10P/wS+VnYozA14mbk7KkKPhl756/ZimEZOGCqJ113US42VEGU4FmxR7qWYJoSMyYF1LJYmY9rLZvRN8apU+DmNlSxo8U39PZCTSehwFtjMiZqgXvan4n9dNTVj1Mi6T1DBJ54vCVGAT4+nzuM8Vo0aMLSFUcXsrpkOiCDU2oqINwV18eZm0zivuZcW5uyjXqnkcBTiGEzgDF66gBrfQgCZQEPAMr/CGHtELekcf89YVlM8cwR+gzx9Y8I98</latexit>

fi<latexit sha1_base64="f4jXYSUiJn4KudYnZNr5TSSFBPI=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVRJRdCUFN11WtA+opSTTaR2aF5OJIkXwB9zqp4l/oH/hnTEFtYhOSHLm3HvOzL3XTwKRKsd5LVhz8wuLS8Xl0srq2vpGeXOrlcaZZLzJ4iCWHd9LeSAi3lRCBbyTSO6FfsDb/vhMx9s3XKYiji7VXcJ7oTeKxFAwTxF1MeyLfrniVB2z7Fng5qCCfDXi8guuMEAMhgwhOCIowgE8pPR04cJBQlwPE+IkIWHiHPcokTajLE4ZHrFj+o5o183ZiPbaMzVqRqcE9EpS2tgjTUx5krA+zTbxzDhr9jfvifHUd7ujv597hcQqXBP7l26a+V+drkVhiBNTg6CaEsPo6ljukpmu6JvbX6pS5JAQp/GA4pIwM8ppn22jSU3tureeib+ZTM3qPctzM7zrW9KA3Z/jnAWtg6p7VHXODyu103zURexgF/s0z2PUUEcDTfIe4RFPeLbqVmRl1u1nqlXINdv4tqyHD2hFkEc=</latexit>

fj<latexit sha1_base64="XHptQbu2aN/FQZticJn/tr7CI8o=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVRJRdCUFN11WtA+opSTTaY1NkzCZKKUI/oBb/TTxD/QvvDNOQS2iE5KcOfeeM3Pv9ZMwSKXjvOasufmFxaX8cmFldW19o7i51UjjTDBeZ3EYi5bvpTwMIl6XgQx5KxHcG/khb/rDMxVv3nKRBnF0KccJ74y8QRT0A+ZJoi763ZtuseSUHb3sWeAaUIJZtbj4giv0EIMhwwgcESThEB5Setpw4SAhroMJcYJQoOMc9yiQNqMsThkesUP6DmjXNmxEe+WZajWjU0J6BSlt7JEmpjxBWJ1m63imnRX7m/dEe6q7jenvG68RsRLXxP6lm2b+V6dqkejjRNcQUE2JZlR1zLhkuivq5vaXqiQ5JMQp3KO4IMy0ctpnW2tSXbvqrafjbzpTsWrPTG6Gd3VLGrD7c5yzoHFQdo/KzvlhqXJqRp3HDnaxT/M8RgVV1FAn7wEe8YRnq2pFVmbdfaZaOaPZxrdlPXwAaqWQSA==</latexit>

fj<latexit sha1_base64="XHptQbu2aN/FQZticJn/tr7CI8o=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVRJRdCUFN11WtA+opSTTaY1NkzCZKKUI/oBb/TTxD/QvvDNOQS2iE5KcOfeeM3Pv9ZMwSKXjvOasufmFxaX8cmFldW19o7i51UjjTDBeZ3EYi5bvpTwMIl6XgQx5KxHcG/khb/rDMxVv3nKRBnF0KccJ74y8QRT0A+ZJoi763ZtuseSUHb3sWeAaUIJZtbj4giv0EIMhwwgcESThEB5Setpw4SAhroMJcYJQoOMc9yiQNqMsThkesUP6DmjXNmxEe+WZajWjU0J6BSlt7JEmpjxBWJ1m63imnRX7m/dEe6q7jenvG68RsRLXxP6lm2b+V6dqkejjRNcQUE2JZlR1zLhkuivq5vaXqiQ5JMQp3KO4IMy0ctpnW2tSXbvqrafjbzpTsWrPTG6Gd3VLGrD7c5yzoHFQdo/KzvlhqXJqRp3HDnaxT/M8RgVV1FAn7wEe8YRnq2pFVmbdfaZaOaPZxrdlPXwAaqWQSA==</latexit><latexit sha1_base64="NAHOtIgEQtavlY0G60hJVcJtZF0=">AAAB6nicbVBNS8NAEJ34WetX1aOXxSLUS0lEsceCF48V7Qe0oWy2m3bpZhN2J0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekEhh0HW/nbX1jc2t7cJOcXdv/+CwdHTcMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2Mb2d++4lrI2L1iJOE+xEdKhEKRtFKDxV60S+V3ao7B1klXk7KkKPRL331BjFLI66QSWpM13MT9DOqUTDJp8VeanhC2ZgOeddSRSNu/Gx+6pScW2VAwljbUkjm6u+JjEbGTKLAdkYUR2bZm4n/ed0Uw5qfCZWkyBVbLApTSTAms7/JQGjOUE4soUwLeythI6opQ5tO0YbgLb+8SlqXVe+66t5fleu1PI4CnMIZVMCDG6jDHTSgCQyG8Ayv8OZI58V5dz4WrWtOPnMCf+B8/gCG+41E</latexit>

(a)

(b)

Figure 1: Encoder layer in LoFTR and the Post-LN variant used in [10].
a learnable slack variable (referred to as the dustbin in [8]),in order to solve the assignment solution with sinkhorn algorithm:

Fine-level Supervision. The fine-level LoFTR operates on
w × w windows centering around the established coarselevel matches. For a pair of coarse-level matches, we warp
the central location of the left grid to the right image, calculate its distance to its nearest neighbor, and check if it
locates within the corresponding fine-level window of theright grid. We only keep those coarse matches for fine-levelsupervision, whose central locations of the left grids arematchable within the fine-level windows of the right grid.

S(i, N B ) = S(N A , j) = S(N A , N B ) = z ∈ R,
A
where N A and N B are the number of features in F̃tr(i)
B
and F̃tr (j). We empirically find that three sinkhorn iterations gives a good balance between speed and the overall
performance.

3.2. Dataset Preparation
MegaDepth. We use the MegaDepth [6] dataset to train
and validate LoFTR on outdoor pose estimation and homography estimation. We remove scenes used as thetest set in the Image Matching Challenge, as well as
scenes with low-quality depth maps pointed out by [3].
Among scenes kept, we enumerate all image pairs withcovisible scores in a range of [0.1, 0.7] and further split
each scene into sub-scenes with covisible scores in ranges[0.1, 0.3], [0.3, 0.5], [0.5, 0.7] respectively. Sub-scenes usedas the validation set in the Image Matching challenge arealso grouped as a validation set, from with we sample 1500
images with each sub-scene evenly sampled. The remaining sub-scenes are used for training, leading to a training set
composed of 368 sub-scenes in total.

3.4. Training Details
We empirically find that LoFTR is not sensitive to hyperparameters, thus we didn’t extensively tune the hyperparameters for training. However, we elaborate on training
details for the results exhibited in the main paper here.

3.3. Supervision

MegaDepth. As mentioned in Section 3.2, we split the
MegaDepth dataset into sub-scenes with covisible scores indifferent ranges. Based on the data sampling scheme of [2],we randomly sample 100 pairs from each sub-scene during each epoch of training, resulting in 36800 samples perepoch. The sampling scheme also balance the sample difficulties and scene variaties of each iteration, which facilitate
convergence of networks. We train LoFTR for 30 epochs
in total. Under a batch size of 8, the initial learning rate is
set to 1 × 10−3 , with a linear learning rate warm-up in 3
epochs from 0.1 of the initial learning rate. All hyperparameters scale with batch size following the linear scaling
rule [4]. We decay the learning rate by 0.5 every 8 epochs
starting from the 8th epoch. The training takes one day on
16 V100 GPUs.

We elaborate on the generation of coarse- and fine-levelsupervisions in this section.

ScanNet. Training on ScanNet follows a similar scheme as
MegaDepth. We sample 200 image pairs per scene at each

ScanNet. We follow exactly [8] for the construction of
ScanNet dataset. About 230M image pairs from 1513
scenes with covisible scores in the range [0.4, 0.8] form thetraining set. We refer readers to the supplementary of [8]for more details.

2epoch and balance scene varianties for each iteration. This
leads to 302400 samples for each epoch. We train LoFTR
for 30 epochs, with a batch size of 64. The initial learning rate is set to 6 × 10−3 , with a linear warm up for 4800
iterations, both of which scale with the linear scaling rule.
We decay the learning rate by 0.5 every 3 epochs startingfrom the 3rd epoch. The training takes one day on 64 1080
Ti GPUs. Note that our training scheme is much shorter,
looking at about 9M samples in the entire training, than SuperGlue [8], which looks at about 360M samples.

topic with us (whether anonymously or not).

Sampling for Fine-level Training. LoFTR is trained endto-end from the FPN backbone until the fine-level LoFTR
module. To reduce and stabilize the use of GPU memory
during training, we limit the number of training samples
for fine-level LoFTR to 30% of the maximum number ofcoarse-level LoFTR matches possible. If fewer matches areproduced in the coarse-level, we pad coarse-level matches
by randomly sampling from ground-truth matches. To increase the number of usable samples and ease network convergence, we make sure there is at least 200 ground-truth
coarse-level matches for fine-level training, which is ignored if not much ground-truth matches exists.

References

5. More Qualitative Results
More comparison of LoFTR and baselines on ScanNet
and MegaDepth datasets is illustrated in Fig. 2. We alsopresent more qualitative results on the InLoc benchmark
in Fig. 3. We finally give more examples of the attention
weight visualization in self- and cross-attention, as well as
the transformed feature visualization through PCA in Fig. 4.

[1] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas
Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-toend object detection with transformers. In ECCV, 2020.
[2] Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Marc Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler. D2-Net:
A trainable cnn for joint detection and description of local
features. CVPR, 2019.
[3] Mihai Dusmanu, Johannes L. Schönberger, and Marc Pollefeys. Multi-View Optimization of Local Feature Geometry.
In Proceedings of the European Conference on Computer Vision, 2020.
[4] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch,
Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint
arXiv:1706.02677, 2017.
[5] Xinghui Li, Kai Han, Shuda Li, and Victor Prisacariu. Dualresolution correspondence networks. NeurIPS, 2020.
[6] Zhengqi Li and Noah Snavely. Megadepth: Learning singleview depth prediction from internet photos. In CVPR, 2018.
[7] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario
Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. 2019.
[8] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz,
and Andrew Rabinovich. SuperGlue: Learning feature
matching with graph neural networks. In CVPR, 2020.
[9] Hajime Taira, Masatoshi Okutomi, Torsten Sattler, Mircea
Cimpoi, Marc Pollefeys, Josef Sivic, Tomas Pajdla, and Akihiko Torii. InLoc: Indoor visual localization with densematching and view synthesis. In CVPR, 2018.
[10] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia
Polosukhin. Attention is all you need. NeurIPS, 2017.

Shared Training Setups. All setups not mentioned areshared for training on ScanNet and MegaDepth. For example, we use the Adam-W optimizer with weight decay factor
equals to 0.1. We use gradient clipping equals 0.5 to avoid
exploding gradients. We don’t use dropout for regularization as usually used when training Transformers.

4. On the Naming of This Paper
During the review period, we received some suggestions
on the naming of the paper. Reviewer 1 kindly suggested
that the idea of “local feature” is not fully accurate and conflicts with the title “detector-free”, and we should name ourpaper as a semi-dense matching method. After careful considerations, we decide to keep the original name. We elaborate the thinking behind this decision in the hope of communicating with the reviewer.
We consider the concept of “local feature” lies in thatthe information each feature vector describes is from a local region on the image. Indeed, most previous works uses
the term “local feature” in the context of the detector-based
feature extraction pipelines. However, we don’t think “local
feature” can only refer to features around the detected keypoints. Due to this reason, we believe that it’s reasonable toextract dense local features with a CNN, which has limited
receptive field and the extracted information is only related
to a local region. In our paper, we later use Transformer tomatch the two sets of local features and extend the receptive
field to the entire image pair. We hope that these explanations will resolve the questions regarding the naming of thispaper, and we welcome Reviewer 1 to further discuss this3

DRC-Net

LoFTR

DifficultOutdoor

Easy

DifficultIndoor

Easy

SuperPoint + SuperGlue

Figure 2: Qualitative image matches on ScanNet and MegaDepth. We report more results that compare SuperGlue [8],DRC-Net [5] and LoFTR in indoor and outdoor environments. The matching precision and pose error are printed in the upper
left corner. The red color indicates a match error larger than five pixels.
4

DRC-Net

LoFTR

Difficult

Easy

SuperPoint + SuperGlue

Feature PCA

Cross

Self

Figure 3: Qualitative image matches on InLoc benchmark. We report comparison results on InLoc [9] benchmark. Since
no ground-truth pose is available, we color the match with predicted confidence. Red indicates higher confidence and blue
for the opposite.

Figure 4: Visualization of self- and cross-attention weights and the transformed features. We visualize more self- andcross-attention results and PCA of feature representation.

5