SeqMatchNet: Contrastive Learning with Sequence Matching for Place Recognition & Relocalization
Authors: Sourav Garg
Madhu Vankadari
Michael Milford
Venue: CoRL2021 Oral (2021)
Keywords: Relocalization
Localization
Visual Place Recognition
Sequence Matching
Contrastive Learning

Abstract: Visual Place Recognition (VPR) for mobile robot global relocalization is a well-studied problem, where contrastive learning based representation training methods have led to state-of-the-art performance. However, these methods are mainly designed for single image based VPR, where sequential information, which is ubiquitous in robotics, is only used as a post-processing step for filtering single image match scores, but is never used to guide the representation learning process itself. In this work, for the first time, we bridge the gap between single image representation learning and sequence matching through "SeqMatchNet" which transforms the single image descriptors such that they become more responsive to the sequence matching metric. We propose a novel triplet loss formulation where the distance metric is based on "sequence matching", that is, the aggregation of temporal order-based Euclidean distances computed using single images. We use the same metric for mining negatives online during the training which helps the optimization process by selecting appropriate positives and harder negatives. To overcome the computational overhead of sequence matching for negative mining, we propose a 2D convolution based formulation of sequence matching for efficiently aggregating distances within a distance matrix computed using single images. We show that our proposed method achieves consistent gains in performance as demonstrated on four benchmark datasets. Source code available at https://github.com/oravus/SeqMatchNet.

SeqMatchNet: Contrastive Learning with Sequence Matching for Place Recognition & Relocalization
Sourav Garg QUT, Australia b.gargequt. edu.
Madhu Vankadari University of Oxford, UK
Michael Milford QUT, Australia
Abstract: Visual Place Recognition (VPR) for mobile robot global relocalization is a well-studied problem, where contrastive learning based representation train- ing methods have led to state-of-the-an performance. However, these methods are mainly designed for single image based VPR, where sequential information, which is ubiquitous in robotics, only used as post-processing step for filtering single image match scores, but is never used to guide the representation learning process itself. In this work, for the first time, we bridge the gap between single im- age representation learning and sequence matching through SeqMatchNer which transforms the single image descriptors such that they become more responsive to the sequence matching metrie. We propose novel triplet loss formulation where the distance metric is based on sequence matching, that is, the aggregation of temporal order-based Euclidean distances computed using single images. We use the same metric for mining negatives online during the training which helps the optimization process by selecting appropriate positives and harder negatives. To overcome the computational overhead of sequence matching for negative min- ing, we propose 2D) convolution based formulation of sequence matching for efficiently aggregating distances within a distance matrix computed using single images. We show that our proposed method achieves consistent gains in perfor- mance as demonstrated on four benchmark datasets. Source code available at https://githie www./ooawww//eeeaatttiaaii
Keywords: Visual Place Recognition, Localization, Contrastive Learning
Introduction
In the context of mobile robotics, Visual Place Recognition (VPR) is defined as the task of recogniz- ing previously seen places by matching currently observed images with those stored in the reference map 2, 3]. It is one of the key capabilities of a mobile robot and is required for the task of visual localization, either in the form coarse location estimation [4, 5] given a prior map or for loop clo- sures and mapping in visual SLAM [6, 7]. These VPR-driven tasks are relevant for many practical robotic applications, including driverless cars, last mile delivery and exploratory rovers.
One of the main challenges in VPR is scene appearance variations as a place can be revis- ited during a different time of day, season or weather conditions. for example, nighttime [8] or snowy conditions [9]. Under such difficult scenarios, classical feature encoding techniques often fail [10, 11, 12, 13). Hence, learning-bass based techniques have been developed for both local [14, 15] and global feature representations [4, 16, 17]. In VPR, learning single image descriptor has been accomplished by both classification [17, 18] and contrastive representation learning [4, 19] type approaches. While these single image based methods address the appearance challenges to some extent, dealing with significant perceptual aliasing typically requires the use of additional informa- tion in the form of image sequences [20, 21, 22, 23, 24).
In the VPR literature, several sequence- based models have been explored including the use of DTW [25], linear diagonal search [21], graphs [23] and particle filters [26]. Most of these meth- ods follow an underlying common principle of aggregating match scores obtained by compar- ing single image representations. The community has thus focused on both the aspects, that
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

is, learning more robust single image global descriptors through variety of deep learning tech- niques [4, 16. 19. 17. 27, 28] and developing robust sequence matching techniques [29. 30. 31, 32] but in isolation. That is, single image representations are learnt without ever considering the subse- quent task of sequence score aggregation (or sequence matching). Thus, there remains a disconnect between the two, as the latter never guides the learning process of the former. In this paper, we ad- dress this issue through triplet-loss based contrastive learning with the following key contributions:
for the first time, we integrate sequence matching in the VPR training pipeline, bridging the gap between learning single image descriptors and sequence matching;
the sequence matching metric based temporal order-presene average of Euclidean distances for both computing triplet loss and mining negatives; and
we propose
finally, to overcome the computational overhead of sequence matching for negative mining, we propose novel 2D convolutions based distance matrix computation method.
We show that transforming an existing single image descriptor by directly optimizing it for sequence matching using the latter both as a loss function and for negative mining leads to superior perfor- mance, as compared to traditional formulations based on single image based distance metrics.
2 Related Work
A detailed literature review on VPR is provided in [1, 3, 33, 2]. Here, we cover the recent VPR lit- erature involving learning- based and sequence-basee bassed techniques, and related literature that employs triplets- baste contrastive learning techniques.
2.1 Learning for VPR
Recent advances in deep learning using CNNs have led to significant progress in the VPR litera- ture, where superior performance was first achieved by simply using off-the-shelf deep-leart image encoders [34, 35] for image retrieval [36] and place recognition [37], followed by analyses of differ- ent CNN layers and training data suitability [38, 39J. Several works exist that employ off-the-shelf CNNs in unique ways to improve VPR [40, 41, 42, 31, 43, 44, 45]. Learning directly for VPR, one of the most notable works, NetVLAD [4], re-established the well-known local feature aggregation technique: VLAD (Vector of Locally Aggregated Descriptor) [13] by employing a deep-learnt image encoder, a learnable VLAD layer and triplets-based contrastive learning. The follow-up literature has explored learning methods both in the context of feature aggregation and learning method types.
Chen et al. [17] trained AMOSNet and HybridNet on a place classification task to later use the learnt features as place representations. DeLF [46] and its recent global descriptor adaptation with DeLG [18] learnt features through landmark recognition task. Sarlin al. [27] proposed HF-Net employing knowledge distillation for simultaneously learning local and global features. Different pooling techniques such as GeM [16, 19), NetBoW (47, 48), NetFV [47], pyramid pooling [49, 50), max pooling [7], and graph attention pooling [51] have also been explored to learn robust image rep- resentations. Other learning based methods include descriptor standardization [52, 39], autoencon- ders [53], CapsuleNet [54] based feature segregation [55], and omnidirectional CNNs [56]. How- ever, all these methods are specifically designed for single image based representation learning and are agnostic to any subsequent sequential processing.
2.2 Sequences for VPR
Use of sequential information in VPR is known to improve robustness against perceptual alias- ing as more visual evidence becomes available to disambiguate false matches, which are otherwise hard to prevent through limited information or occasional transient noise within single images. Ho and Newman [20] proposed dynamic programming based sub-sequence searching to detect loop closures given a matrix of single image based distances. Milford and Wyeth [21] proposed SeqS- LAM, a sequence searching VPR method based on local velocity search within a column- column-normalinn normalized score matrix. SMART [57] adapted SeqSLAM with the use of odometry information to deal with the velocity-sensitiviity problem. Along the same lines, Lynen et al. [29] transformed the distance matrix to obtain "placial indices" and computed vote density using odometry information to form "placeless" place recognition system. A number of follow-up works [22, 58, 23, 59, 60] have

---

Sequence Matching Triplet
Idenlity Matrix
Negafive
Postive
Posilive
Anchor
Ir.
Distance
Image Encoder
Anchor
Maingte
Trislt Miring
Increase Margin
Negative
(a)
Training
Negative Mining (& Testing)
Optimber
Figure SeqMatchNet: (a) Images are encoded using an existing single image representation method and then fully connected layer with bias is used the learnable linear transform. The resultant single image descriptors are then processed as sequence for computing triplet loss. Eu- clidean distance is only computed between temporally corresponding descriptors, say for the anchor and positive sequence, which is then averaged to obtain sequence matching distance dap- (b) For online negative mining, the distance matrix M Miiigge (obtained from single image descriptor compar- isons) is convolved with an Identity matrix Ir to efficiently compute sequence matching distances. Dashed blue lines show the training process.
proposed novel sequence searching techniques robustly estimate place match hypotheses, for ex- ample, by using models based on Network Flow [22], HMM [61], Directed Acyclic Graph [23], kNN-DTW [25], and Particle Filter [26, 62, 32j. However, all of these methods are agnostic to the underlying feature representation techniques and do not use any learning-bald -based mechanisms.
2.3 Triplet Loss and Contrastive Learning
Contrastive learning is used learn representation of the given signal by pulling closer the asso- ciated signals in contrast to other signals in the dataset [63, 64]. It has been employed for face verification [65], dimension reduction [66], object detection [67], video alignment [68], action recognition [69, 70], and several other tasks in scene understanding [71, 72, 73, 63, 74, 75J. Loss formulations typically differ among these methods; examples include noise-contrastive loss [76], normalized temperature based cross entropy loss [63], and max margin-based triplet loss [77] which also used in this work. In the context of sequential imagery or videos. coff-supervied triplet generation is quite common, as the positives can be sampled using either temporal proximity [78], "clips' from within the same video [79, 80], another view of the same scene [81], or different video from the same known cluster [82]. All of these methods aggregate frame-wise information explicitly by channel-wise concatenation of individual frames with 2D CNNs [70] implicitly by
encoding clips through 3D CNN 179, 69, 80]. The supervisory signal in most of these methods is either based on strict timestamps [78] or nearby frames/clips [79, 82). Contrary to that, frame-leeel correspondences are available for place recognition datasets through GPS or other sensors. This aids in defining supervised loss functions for accurate sequence alignment where order information can be explicitly enforced. unlike unordered set-based matching methods [83, 84, 85].
2.4 Learning with sequences for VPR
Recent work in VPR has started to explore learning based techniques for sequence-based methods but have mainly focused generating sequential descriptors a single summary tion a sequence of images. This includes online discriminative learning of memory cells [86], end- to-end multi-view learning via feature grouping, fusion or recurrence [87, 88], learning descriptor grouping for sequence matching optimization [89], unsupervised coresets based summarization [90], topological map learning via recurrence [91], odometry/position based sequential learning [92, 93], unsupervised change-based based descriptor adaptation using Delta Descriptors [94], and supervised 1D temporal convolution based hierarchical VPR with SeqNet [95]. In contrast SeqNet [95] which aggregates single image deseriptors within a sequence to generate summary vector representation, the proposed MatchNt aggregates single image based match scores given a pair sequences. We do not perform any descriptor aggregation in this work but learn linear transformation of sin- gle image descriptors, applied independently to each element within a sequence. Thus, sequential interaction is only introduced the sequence matching metric when calculating loss and mining
representa-
3

---

negatives. This is in contrast to SeqNet which treats aggregated descriptors as a single vector for contrastive loss optimization.
SeqMatchNet
Here, present all the components of SeqMatchNer (see Figure 1). We first explain how the isting single image representations are transformed; then we present our training framework which includes the triplet loss, sequence matching metric and convolutions- tation for negative mining; and finally, we describe the test time sequential query retrieval process.
distance matrix compu-
3.1 Image Representation
Given image I, its corresponding descriptor X E RD is assumed to be pre-computed using existing single image representation method. This descriptor is linearly transformed to X e R" using a learnable fully-connected layer with weight matrix W R Dx and bias B IR. In the supplementary material, we provide results for deeper network configurations.
X' WX
(1)
X' is 12-normalized before computing distances. W and B are shared across all the descriptors and also when computing triplets, as explained next.
3.2 Training
Here, present the training pipeline including contrastive triplet loss and the use of sequence matching as distance metric both for computing loss and mining negatives.
Contrastive Triplet Loss: Our objective is to learn X' such that for any given anchor image deseriptor X² its distance from a positive image descriptor X (belonging to a nearby physical location) is smaller than its distance from a negative image descriptor X,, (belonging to a far-off physical location) by certain margin (m). That is, the contrastive max-margin triplet loss L is defined as:
mar - dan + m,
Sequence-based Distance Metric: In most existing works, the distance computation in the afore- mentioned triplet loss formulation based on single summary vector metrics [4, 87]. In this work, we instead use a sequence of image descriptors of length L centered at any given single image descriptor to define distance metric, inspired by well-established sequence matching meth- ods [21, 57, 23, 96, 95). Thus, for temporally-ordere list of images within a sequence, order- corresponding pairwise Euclidean distances are computed between any two sequences centered at X and X and then averaged to obtain the sequence matching distance' (see Figure 1):
L/2
d;- = 1/L
Σ
X
The above sequential order-preserving distance measure is practical for many robotic applications, for example, autonomous driving, where agent navigates repeatedly through the same route, especially when shorter metric span (short sequence) is considered. Unlike several existing video- based learning methods [78, 79, 70, 69, 82). the availability of motion and position information (odometry/GPS) from robot's sensor data allows sub-sampling of image sequences using a fixed metric separation while also providing frame-to-frame correspondence across reference and query databases [6, 21, 4]. This enables simplified supervision with the help of the proposed distance metric and loss formulation for the task of visual place recognition, unlike unordered set-based matching metrics used in other domains [83, 84, 85].
Using sequence-based matching during training as proposed has a threefold effect. 1) It simulta- neously optimizes for L consecutive descriptors - this prevents the system from over-focusing on
di follows all the distance metric properties since it is average of ordered list of Euclidean distances.

---

any one single descriptor within sequence which is too hard (a positive or negative) to optimize standalone (as in traditional approaches) but might only have a minimal impact on the aggregated sequence distance. This effect can be observed through the negative distance profiles visualized in Section 5.5. 2) Since the individual deseriptors within sequence are matched in temporal-order preserving manner, the anchor-po anchor-positive positive distance so achieved is ideally the lowest this prevents the system from unnecessarily optimizing for all possible combinations of anchors and positives within a certain localization radius, c.g. trying to minimize full L x L distance matrix between anchor and a positive sequence as opposed to ideally minimizing only the diagonal elements of that ma- trix. 3) As an aggregated distance is computed between anchor and negative through sequence matching, the negatives selected online during training are harder than those which would have been selected when using single image based training this gives priority to those regions of the environment where several consecutively-occurii negatives make difficult to avoid perceptual aliasing.
Distance Matrix Computation for Negative Mining: Matching sequences of image descriptors naturally comes at an additional cost of computation. This becomes a bottleneck during training when searching for hard negatives in large reference database. Since most of the sequence- -ased VPR datasets are long vehicular trajectories of the environment, it is typical to first compute a dis- tance matrix through single image descriptor comparisons. This distance matrix is then further pro- cessed to obtain sequence matching scores [20, 21, 5J. Leveraging GPUs' highly parallelized con- volution operations, we propose an efficient 2D convolution based approach to compute sequence- based distance matrix Maeg that convolves an identity matrix Iz with the single descriptor based distance matrix Mgingle:
(1. = Ir Maingle (t,5)/L
(4)
The above formulation computes the mean of the diagonal elements of M single IL strides along its rows (reference database) and columns (query database), thus the outcome is the same as that of Equation 3. Using an identity matrix as kernel enables linear 1D warp between a given pair of image sequences, thus assuming uniform robot motion for both the sequences. However, Equation can be easily adapted to deal with non-uniform motion by employing multiple binary kernels where Is are determined by the warp to be searched such as the linear search proposed in SeqSLAM [21].
3.3 Testing
Sequential Query Retrieval: During the test time, query images are retrieved using sequences of length centered at each query image. In the batch approach of sequence matching (Equation 4) for mining negatives during training, all the queries of the train set were available beforehand. For the test time, similar to an online operation, we assume query images to be accrued incrementally. Thus, we re-use Equation 4 for computing sequence matching distances but perform convolutions only along the rows of the matrix Maingle. Using the resultant distance matrix Mang a reference image with the lowest sequence matching distance is retrieved as match for any given query sequence.
Experimental Settings
4.1 Datasets
We use four datasets in this work: i) Nordland [9] Summer vs Winter, ii) Oxford Robotcar [8] and iii) Brisbane City Loop [97]- - Day vs Night, and iv) MSLS [88]- Multi-cityMuulii-onniitoo For the Oxford and Brisbane datasets, we subsample image frames at 2 meters for both training and testing. In the Nordland dataset, consecutive frames are approximately 20 meters apart. No subsampling performed for the MSLS dataset. Additional details about the datasets and train/test splits are provided in the supplementary material.
4.2 Pre-processing and Evaluation
We used NetVLAD with PCA+ PCA+Wnitnnigg [4] as our image descriptor of size D 4096, computed by downsampling images to resolution of 640 x 320. The learnt descriptors are retained be of the same size D to enable fair comparison against vanilla NetVLAD baselines. For sequence matching,
5

---

L = 5 is used for both training and testing. For testing, localization radius of 10 meters, 20 meters and 10 frames used respectively for Oxford, Brisbane/MSLS and Nordland dataset".
We use Recall @ K [98] as the evaluation metric as also used by previous works [4, 95, 87, 99]. It is defined as the ratio of queries that correctly matched within its top-K retrieved images the total number of queries. Please refer to the supplementary material for training parameters details.
5 Results and Discussion
In this section, we first present the key results of improving VPR using sequence matching within the training pipeline both as a loss function and for mining negatives, which is compared against the vanilla method of using single image based distance metrics. Then, we present results for cross-city testing of models trained on different cities to observe generalization trends. Finally, we present results for state-of-the-art VPR comparisons considering a variety of sequence-based approaches along with an ablation study on the length of sequence used. In order to better understand the performance variations between single image and sruunce--bssed methods, we present a study on cross-testing single/sequence matching with/without sequence-based training in Section 5.2 of the supplementary material.
Table 1: Ablation study of different loss types and mining types for sequence-based VPR using Recall @K K (1/5/20).
Loss Type
Mining Type
Oxford
Nordland
Brisbane
Existing:
Single
Single
0.76/0.89/0.97
0.59/0.75/0.86
0.50/0.62/0.75
Proposed: SeqMatch Single
Single
0.79/0.90/0.97
0.61/0.76/0.8'
0.51/0.62/0.74
SeqMatch
0.76/0.89/0.97
0.66/0.81/0.91
0.51/0.63/0.75
SeqMatch
SeqMatch
0.78/0.89/0.97
0.66/0.81/0.91
0.51/0.63/0.75
5.1
Sequence Matching based Loss and Negative Mining
Table shows how the choice of matching metric affects the recall performance when defining the loss and mining negatives. For this, consider Equation 3 for computing loss and negatives during training using a sequence length of or 5, referred to as Single and SeqMatch respectively in Table 1. Results reported on the train/test splits of each of the three datasets are based on the use of sequence matching for test-time query retrieval. It can be observed that given the retrieval task of sequence matching, using SeqMatch as metric during training for both loss and mining leads superior performance for all the datasets (last row: SeqMaach-SSeqattee at all (first row: Single-Singley, Furthermore, when using cross-combination (second and third row), use of SeqMatch either retains or improves performance. In particular, it can be observed that the Oxford dataset necessarily requires SeqMatch-based loss improve performance, leading an absolute increase in Recall @ by 3%. On the other hand, the Nordland dataset benefits more from SeqMatch-bas based mining, leading to a 7% absolute performance improvement. This study shows the importance of optimizing single image descriptors for sequence matching- VPR using SeqMatch the metric (Equation 3) during training.
as compared to when not using it
Table 2: Cross-City VPR Performance: Recall K (1/5/20) when training on one city and testing on the other. Best is boldfaced and second best italicized per column.
Train City\) Test City
Oxford
Brisbane
MSLS-Amman
Vanilla NetVLAD + Seq. Matching
0.67/0.79/0.90
0.21/0.27/0.37
0,25/0.34/0.40
Oxford
0.78/0.89/0.97
0.29/0.38/0.33
0,26/0.36/0.41
Brisbane
17700..4//0.9
0.51//.63/0.75
0.26/0.35/0.41
MellouMee
0.58/0.72/0.86
0.27/0.35/0.44
II is common to use the Nordland dataset with units of localization radius as frames instead of meters [9,
98, 87, 95].

---

Table 3: Comparison against other approaches: Recall @ 1/5/20
with train/test across cities.
Nespovve Distance (Weem)
Megeliue Maing
angle
Methods
Oxford
Brishane
No Sequence:
NetVLAD [4]
0.47/0.70/0.85
0.20/0.28/ 22/0.41
03/0
Sequential
Smoothing [94]
1.59/0.72/..8 85
D. 0.2000.2500.3
Descriptor:
Delta [94]
0.37/0.55/0.7 74
0.20/0.3/0.50
SeqNet (8s) 195]
0.6200.6699.88
0.32/0.40/0055
Sequence
NetVLAD [4]
0.67/0.79/0.90
0.21/0.27/0.37
index
Matching:
S1 [95]
0.71/0.83/0.93
1.28/0.36/0:48
Query-negative distance profiles for single (orange) and sequence-based negative mining (blue).
Figure 2:
8. [95] GISM (58]
0.65/-/-
0.26/-/-
Si [95] + GRH [5]
0,34/-/-
0,18/-f-
Ours: SegMatchNet
0.70/0.84/0.44 94
0.29/0.38/0.50
Nierarchicnt:
HVPR (8,, [95]
0.7110.82/0.88
0.29/0.40/0.55
Ours: HVPR (Ss, SegMutehNet)
0.71/0.82/0.88
0.30/0.40/0.55
Table 4: Effect of Sequence Length
Oxford
Brisbanc
Method/Dataset
L=5
L==10
L=-0
L=$
L=10
1=
NV + Seq. Matching
0.67/0.79/0.
0.79/0.86/0.93
1.92/0.94/0.99 96
0,21/0.27/0.37
0.24/0.28/0.7
0.26/0.30/0
SouNet (5ş) [95]
0.62/0.76/0.88
0.64/0.75/0.89
0.58/0.71/0.8
0.32/0.40/0.55
0.33/0 39/0.52
0.32/0.37/0.4
Ours: SegMatchNet
0.70/0.82/0.88
0.84/0.92/0.96
0.94/0.97/0.99
0.29/0.3800.5
0.31/0.400052
0.380.4600.55
5.2 Cross-City VPR Performance
In Table 2, we present sequence matching results for training on one city and testing on the other. Even though the environmental conditions vary across different cities, cross-city generalization trends look promising. For example, SeqMatchNet based Recall @1 I for Oxford (0.70) and Brisbane (0.29) during cross-testing is significantly better than the use of sequence matching on top of vanilla NetVL.AD which respectively achieves 0.67 and 0.21 (in the first row, referred to as No Transfor- mation). similar trend relative performance improvements can be observed for MSL.S-Amman, where trained models from all three cities lead to higher recall.
5.3 Benchmarking against State-of-the-Arr Sequence-based Methods
Sequence-based information can be leveraged in different ways to improve single image based VPR performance. In Table 3, compare against variety of methods as following. 1) Sequential descriptors refer to single summary vector representation constructed through a sequence of images, which includes amoothing//veeraiing [94]. Delta Descriptors [94] and SeqNet [95]. 2) Sequence matching refers to aggregation of single image based distances which can either be based on the diagonal of the matrix or graph-based path search within the matrix [58. 5]. For the former, we consider SeqNet's 's single frame version S₁ which is the same as using SeqMatchNet but with L = in Equation 3, that is, without guiding single image descriptors with sequence matching metric. For the latter, we consider Graph-based Image Sequence Matcher [58], referred to as GISM, and Graph-based based Relocalization using LSH [5], referred as GRH. Hierarchical sequence matching is considered as per [95] where sequential descriptors shortlist candidates for sequence matching. The first row Table 3 does not use any sequential information and corresponds to single image based VPR baseline. For all other methods, we use sequence length of L, 5. For all methods that use training, results are presented using models trained on the other city. The best results within any block are boldfaced and the overall best are additionally italicized. The underlying single image descriptor for all the methods is either NetVLAD [4] or an adaptation of NetVLAD as cited.
It can be observed that in most cases, use of sequential information improves performance over single image descriptors as expected. SeqMatchNet establishes a new state-of-the-an on the Oxford dataset considering Recall @ K(5,20) and achieves the best results on the Brisbane dataset within the sequence matching block. Among the sequential descriptors, SeqNet [95] achieves state-of-the-art results on the Brisbane dataset but these results deteriorate when using S₃ within HVPR due to inferior sequence matching of S,. However, this not the case with the HVPR combination of Ss and SecMatchNet. where Recall@) improves over the baseline HVPR while maintaining the
7

---

overall Recall@K(5,2))) This shows the relevance of improved sequence matching with SeqMatchNet which benefits from the inclusion of sequence matching in its training pipeline.
5.4 Effect of Sequence Length
Table 4 shows the effect of sequence length on the recall performance of the proposed method as compared SeqNet [95] and vanilla Ne(VLAD with sequence matching. For both the datasets (tested using models trained on the other city), it can be observed that SeqNet's performance deteri- orates with an increase the sequence length. This can be attributed to its sequence average pooling layer which can potentially reduce the discriminative ability of the descriptors during retrieval when using longer sequences for such orderless averaging, On the other hand, in the proposed SeqMatch- Net, averaging of distances between corresponding pairs of images within a sequence enforces an explicit order (as a linear identity warp between two short time-series), leading to state-of-the-ar per- formance especially when longer sequence lengths are used. The proposed method also outperforms vanilla NetVLAD based sequence matching which shows that the learnt descriptor transformation through SeqMatchNet leads to robust representations suited to the task.
5.5 Visualizing Query-Negative Distance Profiles
In our proposed method, we use sequence matching to mine harder negatives than what is achievable through single descriptor matching. In Figure 2, we show the single image distances between query sequence and its negatives (mean of top 50 hard negatives) before averaging distances within a sequence, where negatives are computed using either single descriptors (orange) or sequence- based matching (blue). can be observed that single descriptors based matching (orange) leads to negative selection soiely based on the central elements of the sequence at query frame index 1. This is apparent from the dip in its distance profile at while disregarding the corresponding distance values in its vicinity. On the other hand, sequence- based negative mining (blue) simultaneously considers all the elements in the sequence and is also able to select negatives when the central element is not necessarily the hardest one but the overall sequence indeed is.
6 Conclusion
Recent advances in visual place recognition have been accelerated by learning based techniques, which predominantly governed by single image based distance metrics. The of sequential information, although well established in the literature, has never been explored in the context of biasing single image representations to suit the subsequent task of sequence matching. In this paper, we bridge this gap for the first time through SeqMatchNer and show that overall VPR performance can be improved by including sequence matching in the training pipeline by adapting an existing sin- gle image descriptor. For this, we proposed temporal order- preserved average of Euclidean distances as the sequence matching metric which is used to both compute the loss and mine negatives online. To reduce the overhead of sequence matching for the latter, we proposed novel 2D) convolution- based processing of the distance matrix. We established that the single image descriptors learnt through SeqMatchNet are more robust and lead to superior performance with the help of sequence matching metric as opposed the use of single image matching metrics.
In mobile robotics, localization pipelines are typically comprised of several steps of robustness in- ducing processes. The same true of VPR where single image global representation matching is followed by performance enhancement techniques like sequential processing [20, 21]. local feature matching [100, 42], feature filtering [101, 102], query expansion [103], and even dimensionality re- duction [4]. One can expect that including these subsequent processes in the representation learning process can improve the overall robustness. This work takes a step in that direction by the inclusion of sequence matching in the training itself which leads to consistent performance gains. This study will hopefully encourage the researchers to reconsider the existing VPR and localization pipelines which can benefit from end-to-end task-relevant learning.
8

---

Acknowledgments
We acknowledge the continued support from the QUT Centre for Robotics and thank the reviewers and the editor for their useful comments. We used Weights & Biases [104] for experiment tracking for this paper.
References
[1] S. Lowry, N. Sünderhauf, P. Newman, J. J. Leonard, D. Cox, P. Corke, and M. J. Milford. Visual place recognition: A survey. IEEE Transactions Robotics, 32(1):1-19, 2016.
[2] C. Masone and B. Caputo. A survey on deep visual place recognition. IEEE Access, 9:
19516-19547, 2021.
[3] S. Garg, Fischer, and M. Milford. Where is your place, visual place recognition? In IJCAI, 2021.
[4] R. Arandjelovic, P. Gronat, A. Torii, T. Pajdla, and J. Sivic. Netvlad: Cnn architecture for weakly supervised place recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
[5] O. Vysotska and C. Stachniss. Relocalization under substantial appearance changes using hashing. In Proceedings of the IROS Workshop on Planning, Perception and Navigation for Intelligent Vehicles, Vancouver, BC, Canada, volume 24, 2017.
[6] M. Cummins and Newman. Fab-map: Probabilistic localization and mapping in the space of appearance. The International Journal of Robotics Research, 27( 2008.
[7] M. Lopez-Antequera, R. Gomez-Ojeda, N. Petkov, and J. Gonzalez-Jimenzz Appearance- invariant place recognition by discriminatively training a convolutional neural network. Pat- tern Recognition Letters, 92:89-95, 2017.
[8] W. Maddern, G. Pascoe, C. Linegar, and P. Newman. 1 year, 1000 km: The oxford robotcar dataset. IJ Robotics Res., 36(1):3-15, 2017.
[9] N. Sünderhauf, Neubert, and P. Protzel. Are we there yet? challenging seqslam on 3000 km journey across all four seasons. In Proc. of Workshop on Long- Term Autonomy, IEEE International Conference on Robotics and Automation (ICRA), page 2013, 2013.
[10] D. G. Lowe. Distinetive image features from scale-invariant keypoints. International journal of computer vision, 60(2):91-110, 2004.
[11] H. Bay, T. Tuytelaars, and L. Van Gool. Surf: Speeded up robust features. Eur. Conf. Comput. Vis., pages 404 1104-417,40 2006.
[12] E. Rublee, V. Rabaud, K. Konolige, and G. Bradski. Orb: An efficient alternative to sift or surf. In Inf. Conf. Comput. Vis., pages 2564-2571, 2011.
[13] H. Jégou, M. Douze, C. Schmid, and P. Pérez. Aggregating local descriptors into a compact image representation. In IEEE Conf. Comput. Vis. Pattern Recog., pages 3304-3311, 2010.
[14] D. De'Tone, T. Malisiewicz, and A. Rabinovich. Superpoint: Self-supervised interest point detection and description. In IEEE Conf. Comput. Vis. Pattern Recog. Worksh., pages 224- 236, 2018.
[15] P.EE. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich. SuperGlue: Learning feature matching with graph neural networks. In IEEE Conf. Comput. Vis. Pattern Recog., pages 4938-4947, 2020.
[16] F. Radenović, G. Tolias, and O. Chum. Fine-tuning cnn image retrieval with no human anno- tation. IEEE Trans. Pattern Anal. Mach. Intell., 4157) 2018
[17] Z. Chen, A. Jacobson, N. Sünderhauf, B. Upcroft, L. Liu, C. Shen, I. Reid, and M. Milford. Deep learning features at scale for visual place recognition. Robotics and Automation (ICRA), 2017 IEEE International Conference on, pages 3223-3230. IEFE, 2017.

---

[18] B. Cao, A. Araujo, and J. Sim. Unifying deep local and global features for image search. In European Conference on Computer Vision, pages 726-743. Springer, 2020.
[19] J. Revaud, J. Almazán, R. S. Rezende, and C. R. d. Souza. Learning with average preci- sion: Training image retrieval with listwise loss. Proceedings of the IEEE International Conference on Computer Vision, pages 5107-5116, 2019.
[20] L.. Ho and Newman. Detecting loop closure with scene sequences. International Journal of Computer Vision, 74(3)2611226, 2007.
[21] M. J. Milford and G. F. Wyeth. Seqslam: Visual route-based navigation for sunny summer days and stormy winter nights. In Robotics and Automation (ICRA), 2012 IEEE International Conference on, pages 1643-1649. IEEE, 2012.
[22] T. Nascer, L. Spinello, W. Burgard, and C. Stachniss. Robust visual robot localization across seasons using network flows. In Twenty-Fighth AAAI Conference on Artificial Intelligence, 2014.
[23] O. Vysotska and C. Stachniss. Lazy data association for image sequences matching under substantial appearance changes. IEEE Robotics and Automation Letters, 1(1):213-220, 2016.
[24] S. Garg, N. Sünderhauf, and M. Milford. Gemantic-geomettee visual place recognition: A new perspective for reconciling opposing views. The International Journal of Robotics Research, 2019.
[25] A. Forechi, A. De Souza, C. Badue, and Oliveira-Santos, Sequential appearance-basee global localization using ensemble of knn-dtw classifiers. In Neural Networks (IJCNN), 2016 International Joint Conference on, pages 2782-2788 IEEE, 2016.
[26] E. Pepperell, P. Corke, and M. Milford. Routed roads: Probabilistic rision-based place recog- nition for changing conditions, split streets and varied viewpoints. The International Journal of Robotics Research, 35(9): 1057-1179 2016.
[27] P.-E. Sarlin, C. Cadena, R. Siegwart, and M. Dymezyk. From coarse to fine: Robust hierar- chical localization at large scale. arXiv preprint arXiv:1812.03350 2018.
[28] H. Porav, W. Maddern, and P. Newman. Adversarial training for adverse conditions: Robust metric localisation using appearance transfer. In IEEE Int. Conf. Robot. Autom., pages 1011- 1018, 2018.
[29] S. Lynen, M. Bosse, P. T. Furgale, and R. Siegwart. Placeless place-recognition. In 3DV, pages 303-310, 2014.
[30] T. Naseer, W. Burgard, and C. Stachniss. Robust visual localization across seasons. 1EEE Transactions on Robotics, 2018.
[31] O. Vysotska and C. Stachniss. Effective visual place recognition using multi-sequence maps. IEEE Robotics and Automation Letters, 2019.
[32] A.-D. Doan, T.-T. Do, Y. Latif, T.-J. Chin, and I. Reid. Practical visual localization for autonomous driving: Why not filter? arXiv preprint arXiv:1811.0806. 2018.
[33] Zhang, L. Wang, and Y. Su. Visual place recognition: A survey from deep learning perspective. Pattern Recognition, 2020.
[34] Krizhevsky, Sutskever, and G. E. Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pages 1097-
1 1105, 2012.
[35] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 770- 778, 2016.
10

---

[36] A. S. Razavian. H. Azizpour, J. Sullivan, and S. Carlsson. Can features off-the-shelff: an astounding baseline for recognition. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2014 IEEE Conference on, pages 512-519. IEEE, 2014.
[37] Z. Chen, O. Lam, A. Jacobson, and M. Milford. Convolutional neural network-based place recognition. In Australasian Conference on Robotics and Automation, volume 2, page 4, 2014.
[38] Sünderhauf, S. Shirazi, Dayoub, B. Upcroft, and M. Milford. On the performance of convnet features for place recognition. In Intelligent Robots and Systems (IROS), 2015 JEEE/RSJ International Conference on, pages 4297-4304, IEEE, 2015.
[39] S. Garg, N. Suenderhauf, and M. Milford. Don't 't look back: Robustifying place categorization for viewpoint- and condition-invariant place recognition. In IEEE International Conference on Robotics and Automation (ICRA), 2018.
[40] N. Sünderhauf, S. Shirazi, A. Jacobson, F. Dayoub, E. Pepperell, B. Uperoft, and M. Milford. Place recognition with convnet landmarks: Viewpoint-robust condition-robust, training-free. Proceedings of Robotics: Science and Systems XII, 2015.
[41] Z. Chen, Maffra, I. Sa, and M. Chli. Only look once, mining distinctive landmarks from convnet for visual place recognition. In Intelligent Robots and Systems (IROS), 2017 IEEE/RSJ International Conference on, pages 9-16. IEEE, 2017.
[42] S. Garg, N. Suenderhauf, and M. Milford. Lost? appearance invariiint place recognition for opposite viewpoints using visual semantics. In Proceedings of Robotics: Science and Systems XIV, 2018.
[43] Y. Hou, H. Zhang, and S. Zhou. Evaluation of object proposals and convnet features for landmark-based visual place recognition. Journal of Intelligent & Robotic Systems, 92(3-4): 505-520, 2018
[44] Y. Hou, H. Zhang, and Zhou, Bocnf: Efficient image matching with bag of convnet features for scalable and robust visual place recognition. Autonomous Robots, (696:1169-1185.. 2018.
[45] M. Zaffar, A. Khaliq, S. Ehsan, M. Milford, and K. McDonald-Maier. Levelling the playing field: A comprehensive comparison of visual place recognition approaches under changing conditions. arXiv preprint arXiv:1903.09100 2019.
[46] H. Noh, A. Araujo, J. Sim, T. Weyand, and B. Han. Large-scale image retrieval with attentive deep local features. In Int. Conf. Comput. Vis., pages
2017.
[47] Miech, I. Laptev, and Sivic. Learnable pooling with context gating for video classifica- tion. In IEEE Conf. Comput. Vis. Pattern Recog. Worksh., 2017.
[48] E.-J Ong, S. Husain, M. Bober-Irizan and M. Bober. Deep architectures and ensembles for semantic video classification. IEEE Trans. Circuits Syst. Video Technol., 2018.
[49] J. Yu, C. Zhu, Zhang, Q. Huang. and D. Tao. Spatial pyramid-enhancee netvlad with weighted triplet loss for place recognition. IEEE Trans. Neural Netw. Learn. Syst., 31(2): 661-674, 2019.
[50] Y. Zhu, J. Wang. L. Xie, and L. Zheng. Attention-based pyramid aggregation network for visual place recognition. In 2018 ACM Multimedia Conference on Multimedia Conference, pages 99-107. ACM, 2018.
[51] Y. Yang, B. Ma, X. Liu, I.. Zhao, and S. Huang. Gsap: A global structure attention pooling method for graph-based visual place recognition. Remote Sensing, 13(8):1467, 2021.
[52] S. Schubert, P. Neubert, and P. Protzel. Unsupervised learning methods for visual place recognition discretely and continuously changing environments. arXiv preprint arXiv:200/ 200600000.08900,, 2020.
11

---

[53] A. Saadonado-Ramiire and L. A. Menees. Learning nd-hoc compact representations from salient landmarks for visual place recognition in underwater environments. In 2019 In- ternational Conference on Robotics and Automation (ICRA), pages 5739-5745. IEEE, 2019.
[54] S. Sabour, N. Frosst, and G. E. Hinton. Dynamic routing between capsules. arXiv preprint arXiv: 2017.
[55] P. Yin, L. Xu, Li, C. Yin, Li, R. A. Srivatsan, L. Li, J. Ji, and Y. He. multi-domain feature learning method for visual place recognition. In 2019 International Conference on Robotics and Automation (ICRA), pages 319-324. IEEF, 2019.
[56] T.HH. Wang, H.-J Huang, J.-T. Lin, C.-W. Hu, K.-H. Zeng, and M. Sun. Omnidirectional cnn for visual place recognition and navigation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 2341-2348. IEEE, 2018.
[57] E. Pepperell, Corke, and M. Milford. Towards persistent visual navigation using smart. In Proceedings of Australasian Conference Robotics and Automation. ARAA, 2013,
[58] O. Vysotska, T. Naseer, L. Spinello, W. Burgard, and C. Stachniss. Efficient and effective matching of image sequences under substantial appearance changes exploiting gps priors. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 2774-2779. IEEE, 2015.
[59] D. Bai, C. Wang, B. Zhang, X. Yi, and X. Yang. Sequence searching with enn features for robust and fast visual place recognition. Computers & Graphics, 70:270-280. 2018.
[60] R. Arroyo, F. Alcantarilla, L. M. Bergasa, and E. Romera. Towards life-long visual localiza- tion using an efficient matching of binary sequences from images. In 2015 IEEE international conference on robotics and automation (ICRA), pages 6328-6688000 6335..33.55 IEEE, 2015.
[61] P. Hansen and B. Browning. Visual place recognition using hmm sequence matching. In Intelligent Robots and Systems (IROS 2014), 2014 JEFE/RSJ International Conference on, pages 4549-4555. IEEE, 2014.
[62] Y. Liu and H. Zhang, Towards improving the efficiency of sequence-based slam. In Mecha- tronics and Automation (ICMA), 2013 IEEE International Conference on, pages 1261-1266. IEEE, 2013.
[63] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[64] P. Khosla, P. Teterwak, C. Wang, A. Samaa, Y. Tian, P. Isola, A. Maschinot, C. Liu, and Krishnan. Supervised contrastive learning. arXiv preprint arXiv:2004.1136 2020.
[65] S. Chopra, R. Hadsell, and Y. LeCun. Learning similarity metric discriminatively, with application face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 05), volume 1, pages 539-546. IEEE, 2005.
[66] Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning an invariant map- ping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recogni- tion (CVPR'06) volume pages 1735-1742, IEEE, 2006.
[67] S. Pirk, M. Khansari, Y. Bai, C. Lynch, and P. Sermanet. Online object representations with 2019.
contrastive learning. arXiv preprint arXiv:
[68] S. Haresh, S. Kumar, H. Coskun, S. N. Syed, A. Konin, M. Z. Zia, and Q.-H. Tran. Learning by aligning videos in time. arXiv preprint arXiv:2 /7260, 2021.
[69] A. Singh, O. Chakraborty. A. Varshney, R. Panda, R. Feris, K. Saenko, and A. Das. Semi-supervised action recognition with temporal contrastive learning. arXiv preprint arXiv:2102.0277: 02251.....222..... 2021.
12

---

[70] G. Lorre, J. Rabarisoa, A. Orcesi, S. Ainouz, and S. Canu. Temporal contrastive pretrain- ing for video action recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 662-670, 2020.
[71] A. Shrivastava, T. Malisiewicz, A. Gupta, and A. A. Efros. Data-driven visual similarity for cross-domain image matching. In Proceedings of the 2011 SIGGRAPH Asia Conference, pages 1- 2011.
[72] A. Dosovitskiy, P. Fischer, J. T. Springenberg, M. Riedmiller, and T. Brox, Discriminative un- supervised feature learning with exemplar convolutional neural networks. IEEE transactions pattern analysis and machine intelligence, 38(9):1334-1747, 2015.
[73] T. Park, A. A. Efros, R. Zhang, and J.-Y. Zhu. Contrastive learning for unpaired image-to- image translation. European Conference on Computer Vision, pages 319-345. Springer, 2020.
[74] I. Misra and L. d. Maaten. Self-supervised learning of pretext-invarien representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6707-6717, 2020.
[75] J. Spencer, R. Bowden, and S. Hadfield. Defeat-net: General monocular depth via simulta- neous unsupervised representation learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 14402-14413, 2020.
[76] M. Gutmann and A. Hyvärimen. coise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Y. W. Teh and M. Titterington, editors, Proceed- ings of the Thirteenth International Conference on Artificial Intelligence and Statistics, vol- ume 9 of Proceedings of Muchine Learning Research, pages 297-304, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL http://proceeding/ mlr. wwt..mml..eeeeeee press/v9/ gutmann Oa. html.
[77] F. Schroff, D. Kalenichenko, and J. Philbin. Facenet: A unified embedding for face recogni- tion and clustering. In Proceedings of the IEEE conference on computer vision and patlern recognition, pages 815-823, 2015.
[78] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang. S. Schaal, S. Levine, and G. Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1 1134-1141. IEEE, 2018.
[79] I. Dave, R. Gupta, M. N. Rizve, and M. Shah. Telr: Temporal contrastive learning for video representation. arXiv preprint arXiv:2101.07974, 2021.
[80] L. Tao, X. Wang, and Yamasaki. Self-supervised video representation learning using inter- intra contrastive framework. In Proceedings of the 28th ACM International Conference on Multimedia, pages 2193-2201, 2020.
[81] N. Rai, E. Adeli, K.-H. Lee, A. Gaidon, and J. C. Niebles. Cocon: Cooperative-contrastto learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3393, 2021.
[82] Qian, T. Meng, B. Gong, M.-H. Yang, Wang, Belongie, and Y. Cui. Spatiotemporal contrastive video representation learning. arXiv preprint 2xXiv:2008.03800 2020.
[83] J. Ribera, D. Guera, Y. Chen, and E. J. Delp. Locating objects without bounding boxes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6479 64899,649,261 2019.
[84] H. Fan, H. Su, and L. J. Guibas. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 605-613, 2017.
[85] P. Fang, P. Ji, L. Petersson, and M. Harandi. Set augmented triplet loss for video person re-identification. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 464-473, 2021.
13

---

[86] V. A. Nguyen, J. A. Starzyk, and W.-B. Goh. A spatio-temporal long term memory approach for visual place recognition in mobile robotic navigation. Robotics and Autonomous Systems, 6112:1174441758 2013.
[87] J. M. Facil, D. Olid, L. Montesano, and Civera. Condition-invariant multi-view place recognition. arXiv preprini urXiv: 2019.
[88] F. Warburg, S. Hauberg, M. López-Antequera, P. Gargallo, Y. Kuang, and J. Civera. Map- illary street-level sequences: dataset for lifelong place recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2626-2635, 2020.
[89] H. Zhang, F. Han, and H. Wang. Robust multimodal sequence- sequence-based loop closure detection via structured sparsity. In Robotics: Science and systems, 2016.
[90] M. Volkov, G. Rosman, D. Feldman, J. W. Fisher, and D. Rus. Coresets for visual summariza- tion with applications loop closure. In 2015 IEEE International Conference Robotics and Automation (ICRA), pages 3638-3645. IEEE, 2015.
[91] P. Neubert, S. Schubert, and P. Protzel. A neurologically inspired sequence processing model for mobile robot place recognition. IEEE Robotics and Automation Letters, 4(4):3200-3207, 2019.
[92] M. Chancán and M. Milford. Mvp: Unified motion and visual self-supervised learning for large-scale robotic navigation. arXiv preprint arXiv:2003.0066) 2020.
[93] M. Chancán and M. Milford. Deepseqslam: A trainable cnn+ mn for joint global description and sequence-based place recognition. arXiv preprint arXiv:2011.085/8, 2020.
[94] S. Garg, B. Harwood, G. Anand, and M. Milford. Delta descriptors: Change-based place representation for robust visual localization. IEEE Robotics and Automation Letters, 5(4): 5120-5127, 2020.
[95] S. Garg and M. J. Milford. Seqnet: Learning descriptors for sequence-based hierarchical place recognition. IEEE Robotics and Automation Letters, 2021.
[96] S. Garg and M. Milford. Fast, compact and highly scalable visual place recognition through sequence-basee matching of overloaded representations. In 2020 IEEE International Confer- ence on Robotics and Automation (ICRA), 2020.
[97] M. Milford, Garg, and J. Mount. P1-007: How automated vehicles will interact with road infrastructure now and in the future. Technical report, iMove, QUT and Queensland Gov- ernment, January 2020. URL https: / /ttp///vvauutaaaaaa.aaa coot//upccnteetttt /uploads/ pdff.
6- Fana-inal--- Report-
[98] M. Zaffar, S. Garg, M. Milford, J. Kooij, D. Flynn, K. McDonald-Maier, and S. Ehsan. Vpr- bench: An open-source visual place recognition evaluation framework with quantifiable view- point and appearance change. International Journal of Computer Vision, pages 1-39, 2021.
[99] S. Hausler, S. Garg, M. Xu, M. Milford, and T. Fischer. Patch-netvlad: Multi-scale fusion of locally-globe descriptors for place recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14141-14152, 2021.
[100] M. Cummins and P. Newman. Appearance-only slam at large scale with fab-map 2.0. The International Journal of Robotics Research, 30(9):100--122: 2011.
[101] J. Knopp. J. Sivic, and T. Pajdla. Avoiding confusing features in place recognition. In Euro- pean Conference on Computer Vision, pages 741-761. Springer, 2010.
[102] S. Garg, V. Babu, T. Dharmasiri, S. Hausler, N. Suenderhauf, S. Kumar, T. Drummond, and M. Milford. Look no deeper: Recognizing places from opposing viewpoints under varying scene appearance using single-view depth estimation. In IEEE Int. Conf. Robot, Autom,, pages 4916-4923, 2019.
14

---

[103] O. Chum, A. Mikulik, M. Perdoch, and J. Matas. Total recall II: Query expansion revisited. In IEEE Conf. Comput. Vis. Pattern Recog., pages 889-8988998888088888888888888888 2011.
[104] L. Biewald. Experiment tracking with weights and biases, 2020. URthttps://var wandb. com/. Software available from wandb.com.
15


Reviews:
[{"review_id": "jzd-SMBltrR", "reviewer": "Reviewer_CkaF", "summary": "The paper proposes a sequence matching learning formulation for place recognition and relocalization. Instead of learning the image metric with single images, the authors use the distance between images of the sequences in contrast for learning the feature embedding. A negative mining method is also used to improve the training process. The final results on several commonly used benchmarks show the proposed method is very promising.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS\n\nThe paper considers the sequence information to learn image embedding for place recognition and relocalization. I agree that it is an important source of information to explore.\n\nWEAKNESSES\n\nThe feature learning with single-layer network is a curious choice. Usually, deep networks can provide better embedding with SGD. Some exploration of the embedding network, especially depth, is important for understanding the loss.\n\nThe ablation study on different training and testing schemes will be very helpful for understanding the method. The variants are 1) training on sequence and test on single images. 2) training on single images and test with a sequence?\n\nIt is unclear whether the major improvement comes from negative mining or not.\n\nFailure case analysis will also help understand the results.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Although I agree that learning from sequences can be better, there are still a lot of unknowns for the current experiments.\n\n----\n\nAfter reading author's feedback, my concerns are largely addressed. I changed my recommendation to weak accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "I am willing to improve my rating if the authors can provide strong evidence that the loss can be used for deeper embedding networks. The additional study mentioned in the weaknesses is important to make the design convincing."}}, {"review_id": "Oi5QS9LjHvg", "reviewer": "Reviewer_A1mn", "summary": "The paper presents an integrated deep-based approach for visual place recognition (VPR) that combines sequence learning of feature descriptors with sequence matching. Particularly, a Euclidean distance metric is introduced in a triplet loss, to provide order-preservation of single image descriptors in the sequence. To facilitate the training, the authors also a propose a simple method for negative mining based on the Euclidean distance metric in the sequence data.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper has a very rigorous examination of the state of the art works. I find the idea of integrating the sequence matching into the training process of VPR reasonable, The paper is well-written and easy to follow. Adding an algorithmic block describing the training process would be helpful.\nMy only concern regards the significance of the provided results and the fundamental differences of the proposed work w.r.t. SeqNet, that uses learning of image descriptors in sequences and produces hypotheses on sequence matches, to be later on refined.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "UPDATE: After rebuttal and the update of the paper with stronger ablations, I change my score to Strong accept, as I believe the paper bring significant improvement w.r.t. the prior works.\n\nMy recommendation is mainly driven by the concern described above, namely I am not convinced about the benefit of the method compared to SeqNet, as the result of table 3 does not seem significant. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "While the paper is well-written and well-organized, my concern regard the similarities and differences between SeqNet and SeqMatchNet. It seems that the main component of the method is SeqNet with an extension to the triplet loss. This is of course fine, as long as a rigorous examination of the benefits of the one method over the other is demonstrated. Firstly, it would be interesting to ablate the length of the sequence, as only a single vs. Length of 5 is provided. Moreover, the authors claim state of the art results in Table.4 however the increase in recall are marginal or same as SeqNet. Is it possible to provide evidence for the significance of the prediction improvement? \nI think this will also benefit the community regarding the research direction you are proposing (i.e., embedding sequence matching in the training process)."}}, {"review_id": "5D9d1W0euv4", "reviewer": "Reviewer_Kfyz", "summary": "The paper presents an approach for visual place recognition dubbed SeqMatchNet. The idea is to bridge the gap between sequence matching methods and robustifying image descriptors for visual place recognition. This is accomplished by introducing a contrastive triplet loss which instead of being computed on the descriptor of a single query image, is computed instead on the distance metric calculated on a sequence of image descriptors. In order to speed up computation, the distance metric can be efficiently computed by a convolution operation. The method is evaluated thoroughly on a number of datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper proposes a unique approach for bridging the gap between the different research directions in the area of visual place recognition, and demonstrates the utility of combining both a contrastive loss approach with sequence matching.\n\nThe list below should be viewed more as questions that are unclear to this reviewer rather than weaknesses of the approach:\n1) For the sequence-based distance metric: one of the advantages claimed is that negatives mined during training are harder than those selected using single image based training. Can you elaborate more on why this is the case? Is there some empirical evidence supporting this claim in your findings?\n2)  For table 1 and table 2, what is the baseline approach used? Is it a simple NetVlad model?\n3) In line 251, the good performance is attributed to the importance of optimizing single image descriptors. Where in the presented approach are the image descriptors optimized?\n4) For the cross-city VPR performance, it would be interesting to see the performance of other SOTA methods as well. Currently, it's hard to judge the performance of this approach without this comparison.\n5) Finally a bit of a naive question, but how do the approaches generalize to a new dataset after being trained on a particular training scene? The dataset and the matching matrices need to be computed from scratch for every scene correct? So where does the generalization bit come in? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I do not feel strongly about accepting nor rejecting the paper without having the above questions clarified.\n\nEDIT:\n\nAfter the author's responses I feel more confident in arguing for the acceptance of the work.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See above.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The idea proposed by the authors has been reasonably well received by the reviewers. Combining sequential information and learning of image descriptors showed to improve the results w.r.t. other baselines. While existing time contrastive learning methods have relied on sequences of images, the idea of leveraging sensor information for obtaining frame-level correspondences during training is well thought and clean.
The paper has a good technical quality and it is clear to read.

However, as pointed out by the reviewers,  certain parts of the paper have caused confusion and require more support such as the case for mining negatives and their real benefit, deeper analyses including ablation studies on sequence lengths, design choices on the network depth, and its benefits in regards to SeqNet.

Post-rebuttal ============

Thank you for the clarifications and discussions. The reviewers have been in general pleased with the answers and many of the concerns have been solved in the updated manuscript.

Decision: Accept (Oral)