# Loop Closure Detection in RGB-D SLAM by Utilizing Siamese ConvNet Features
## 2 Materials and Methods
### Siamese ConvNet Architecture for Loop Closure Detection
**While \(t\
eq N\) do**  
\(t\gets t+1\) sample \(M\) training samples from \(X\)  
\(\hat{t}^{i}=Conv(x^{i}\Big{|}\theta_{net})\)  
\(\
abla\theta_{cls}=\sum\limits_{i=1}^{M}\frac{\partial_{cls}(\hat{t}^{j}\hat{ t}_{obs})}{\partial\theta_{cls}}\)  
\(\
abla\theta_{cst}=\lambda\cdot\sum\limits_{i=1}^{M}\frac{\partial_{cost}( \hat{t}^{j}\hat{t}_{obs})}{\partial\theta_{cls}}\)  
\(\
abla\hat{t}=\frac{\partial_{cls}(\hat{t}^{j}\hat{t}_{obs})}{\partial\hat{t} }+\lambda\cdot\frac{\partial_{cls}(\hat{t}^{j}\hat{t}_{obs})}{\partial\hat{t}}\)  
\(\
abla\theta_{conv}=\sum\limits_{i=1}^{M}\frac{\partial_{conv}(\hat{x}^{j} \theta_{conv})}{\partial\theta_{cst}}\)  
**Update \(\theta_{conv}=\theta_{conv}-\eta(t)\cdot\
abla\theta_{conv}\), \(\theta_{cst}=\theta_{cst}-\eta(t)\cdot\
abla\theta_{cst}\), \(\theta_{cls}=\theta_{cls}-\eta(t)\cdot\
abla\theta_{cls}\) End while Output \(\theta_{conv},\ \theta_{cls}\) and \(\theta_{cst}\)**  
**Algorithm 1** learning strategy for loop closure detection  
**Output \(\theta_{conv},\ \theta_{cls}\) and \(\theta_{cst}\)**  
**Algorithm 1** learning strategy for loop closure detection  
**Input:** training set \(X=\left\{x^{1},x^{2},x^{3},\ldots,x^{k}\right\}\), \(x^{k}=\left\{x_{1}^{k},x_{2}^{k},\hat{t}_{1}^{k},\hat{t}_{2}^{k}\right\}\), initialized parameters \(\theta_{net}\), \(\theta_{cls}\) and \(\theta_{cst}\). learning rate \(\eta(t),\ t\gets 0\), times of iteration \(N\), batch size \(k\). **While \(t\
eq N\) do**  
\(t\gets t+1\) sample \(M\) training samples from \(X\)  
\(\hat{t}^{i}=Conv(x^{i}\Big{|}\theta_{net})\)  
\(\
abla\theta_{cls}=\sum\limits_{i=1}^{M}\frac{\partial_{cls}(\hat{t}^{j}\hat {t}_{obs})}{\partial\theta_{cls}}\)  
\(\
abla\theta_{cst}=\lambda\cdot\sum\limits_{i=1}^{M}\frac{\partial_{conv}( \hat{t}^{j}\hat{t}_{obs})}{\partial\theta_{cls}}\)  
\(\
abla\theta_{conv}=\sum\limits_{i=1}^{M}\frac{\partial_{conv}(\hat{x}^{j} \theta_{conv})}{\partial\theta_{cst}}\)  
**Update \(\theta_{conv}=\theta_{conv}-\eta(t)\cdot\
abla\theta_{conv}\), \(\theta_{cst}=\theta_{cst}-\eta(t)\cdot\
abla\theta_{cst}\), \(\theta_{cls}=\theta_{cls}-\eta(t)\cdot\
abla\theta_{cls}\) End while Output \(\theta_{conv},\ \theta_{cls}\) and \(\theta_{cst}\)**  
**Algorithm 2** learning strategy for loop closure detection  
**Input:** training set \(X=\left\{x^{1},x^{2},x^{3},\ldots,x^{k}\right\}\), \(x^{k}=\left\{x_{1}^{k},x_{2}^{k},\hat{t}_{1}^{k},\hat{t}_{2}^{k}\right\}\), initialized parameters \(\theta_{net}\), \(\theta_{cls}\) and \(\theta_{cst}\). learning rate \(\eta(t),\ t\gets 0\)  
* Mid-level fusion:  
In contrast to the early fusion, RGB images and depth images are respectively fed to the RGB stream and depth stream, and two networks are fused at intermediate layers with the same size of feature maps. Finally, the combined architecture outputs the description for detecting loops. Figure 6 shows the flowchart of the mid-level fusion pipeline. * Late fusion:  
In the late fusion strategy, the two streams are discriminatively trained for feature extraction. The generated RGB descriptions and depth descriptions are combined by averaging for the final decision, as shown in Figure 7.  
We experiment to compare the three-fusion strategy in the next Section 3. By quantifying the experiment results, we conclude that late fusion is best suited in our application of RGB-D loop closure detection.