## Appendix A
```
Input: image, intrinsics, gauge = 1.435 Output: calibrated point cloud // get label and depth from backbone models
1 label \(\leftarrow\) segmentationModel(image) \(\leftarrow\) depthModel(image) // project point cloud to 3D with intrinsic camera parameters \(\leftarrow\) positionCloud \(\leftarrow\) projection(depth,intrinsics) // choose rails from this point cloud \(\leftarrow\) pointCloud \(\leftarrow\) pointCloud \(\leftarrow\) 'rail-raised' // perform calibration \((R_{init},T_{init})\)\(\leftarrow\) initialization(rails) \((R,T,s)\)\(\leftarrow\) calibration(gauge,R_{init},T_{init},rails) // Scale cloud and transfer to ego coordinate frame \(\leftarrow\) calibratedCloud \(\leftarrow\) R \(\cdot\) (\(s\cdot\) pointCloud) \(+\) T return calibratedCloud
```  
**Algorithm A1** main metric scaling and extrinsic calibration```
Input:\(\mathit{railis3D}\) Output:\(R_{init}\),\(T_{init}\) //initialtranslationisthecenterofmass
1\(T_{init}\gets mean(\mathit{railis3D})\) //normalvectorofgroundplane
2\(cov\gets covariance(\mathit{railis3D})\) \(e_{e,z}\gets\mathit{thirdEigenvector}(cov)\) if\(T_{init}\cdot e_{e,z}<0\)then
3\(e_{e,z}\gets e_{e,z}\)
4 else
5\(e_{e,z}\leftarrow-e_{e,z}\)
6 end if //computedirectionvectorofthelineofsight.Thisistheintersectionline betweentheplaneswithnormalvectors\(\mathit{ex}_{\mathit{cam}}\)and\(\mathit{ex}_{\mathit{cam}}\leftarrow[1,0,0]\) \(e_{\mathit{los}}\leftarrow e_{\mathit{ez}}\times e_{\mathit{cam}}\) \(e_{\mathit{ex}}\gets e_{\mathit{los}}\)//Initialx-direction //y-unitvectorforright-handedcoordinateframe \(e_{\mathit{ey}}\gets e_{\mathit{los}}\times e_{\mathit{ez}}\) //buildinitialrotationmatrix \(R_{init}\leftarrow[e_{\mathit{ex},e_{\mathit{ey}},e_{\mathit{ez}}]\) return\(R_{init},T_{init}\)
```

**Algorithm A2** initialization

```
Input:\(\mathit{gauge}\),\(R_{init}\),\(T_{init}\),\(\mathit{railis3D}\) Output:\(R\),\(T\),\(s\) //Projectrailpointsontogroundplanespannedby\(e_{\mathit{ex}}\)and\(e_{\mathit{ey}}\) \(\mathit{railis2D}\leftarrow\mathit{projectToPlane}(\mathit{railis3D},e_{\mathit{ex},e_{\mathit{ey}}})\) //Computeintersectionlinebetween\(e_{\mathit{exex}}\)and\(e_{\mathit{ez}}\) \(e_{\mathit{exam}}\leftarrow[1,0,0]\) \(e_{\mathit{ez}}\gets R_{init}[2]\) \(q,w\gets getIntersectionLine(e_{\mathit{exam}},e_{\mathit{ez}})\) //Computebeginpositionalongthelineofsight \(t_{\mathit{begin}}\leftarrow\frac{\mathit{min}(\mathit{railis3D})-q,z}{w,z}\) \(\mathit{posBegin3D}\gets q+t_{\mathit{begin}}\times w\) \(\mathit{posBegin2D}\leftarrow\mathit{projectToPlane}(\mathit{posBegin3D},e_{ \mathit{ex},e_{\mathit{ey}}})\) //Optimizationloop \((\hat{\alpha},\mathit{optimalHist})\leftarrow\mathit{optimization}(\mathit{railis2D},0)\) //Clusterhistogramtofindrails \(\mathit{cluster}\gets DBSCAN(\mathit{optimalHist})\) //Findnearesttwoclusterstostartingposition \((left,right)\gets findNearestCluster(cluster,\mathit{posBegin2D}.y)\) //Computescalingfactor
```

**Algorithm A3**calibration

```
Input:\(\mathit{gauge}\),\(R_{init}\),\(T_{init}\),\(\mathit{railis3D}\) Output:\(R\),\(T\),\(s\) //Projectrailpointsontogroundplanespannedby\(e_{\mathit{ex}}\)and\(e_{\mathit{ey}}\) \(\mathit{railis2D}\leftarrow\mathit{projectToPlane}(\mathit{railis3D},e_{\mathit{ex},e_{\mathit{ey}}})\) //Computeintersectionlinebetween\(e_{\mathit{exam}}\)and\(e_{\mathit{ez}}\) \(\mathit{exam}\leftarrow[1,0,0]\) \(e_{\mathit{ez}}\gets R_{init}[2]\) \(q,w\gets getIntersectionLine(e_{\mathit{exam}},e_{\mathit{ez}})\) //Computebeginpositionalongthelineofsight \(t_{\mathit{begin}}\leftarrow\frac{\mathit{min}(\mathit{railis3D})-q,z}{w,z}\) \(\mathit{posBegin3D}\leftarrow q+t_{\mathit{begin}}\times w\) \(\mathit{posBegin3D}\leftarrow\mathit{projectToPlane}(\mathit{posBegin3D},e_{ \mathit{ex},e_{\mathit{ey}}})\) \(\mathit{//Optimizationloop}\) \((\hat{\alpha},\mathit{optimalHist})\leftarrow\mathit{optimization}(\mathit{railis2D},0)\) //Clusterhistogramtofindrails \(\mathit{cluster}\gets DBSCAN(\mathit{optimalHist})\) //Findnearesttwoclusterstostartingposition \((left,right)\gets findNearestCluster(cluster,\mathit{posBegin2D}.y)\) //Computescalingfactor
```

**Algorithm A4**calibration

```
Input:\(\mathit{gauge}\),\(R_{init}\),\(T_{init}\),\(\mathit{railis3D}\) Output:\(R\),\(T\),\(s\) //Projectrailpointsontogroundplanespannedby\(e_{\mathit{ex}}\)and\(e_{\mathit{ey}}\) \(\mathit{railis2D}\leftarrow\mathit{projectToPlane}(\mathit{railis3D},e_{\mathit{ex},e_{\mathit{ey}}})\) //Computeintersectionlinebetween\(e_{\mathit{exam}}\)and\(e_{\mathit{ez}}\) \(\mathit{exam}\leftarrow[1,0,0]\) \(e_{\mathit{ez}}\gets R_{init}[2]\) \(q,w\gets getIntersectionLine(e_{\mathit{exam}},e_{\mathit{ez}})\) //Computebeginpositionalongthelineofsight \(t_{\mathit{begin}}\leftarrow\frac{\mathit{min}(\mathit{railis3D})-q,z}{w,z}\) \(\mathit{posBegin3D}\leftarrow q+t_{\mathit{begin}}\times w\) \(\mathit{posBegin3D}\leftarrow\mathit{projectToPlane}(\mathit{posBegin3D},e_{ \mathit{ex},e_{\mathit{ey}}})\) \(\mathit{//Optimizationloop}\) \((\hat{\alpha},\mathit{optimalHist})\leftarrow\mathit{optimization}(\mathit{railis2D},0)\) //Clusterhistogramtofindrails \(\mathit{cluster}\gets DBSCAN(\mathit{optimalHist})\) //Findnearesttwoclusterstostartingposition \((left,right)\gets findNearestCluster(cluster,\mathit{posBegin2D}.y)\) //Computescalingfactor
```

**Algorithm A4**calibration

```
Input:\(\mathit{gauge}\),\(R_{init}\),\(T_{init}\),\(\mathit{railis3D}\) Output:\(R\),\(T\),\(s\) //Projectrailpointsontogroundplanespannedby\(e_{\mathit{ex}}\)and\(e_{\mathit{ey}}\) \(\mathit{railis2D}\leftarrow\mathit{projectToPlane}(\mathit{railis3D},e_{\mathit{ex},e_{\mathit{ey}}})\) //Computeintersectionlinebetween\(e_{\mathit{exam}}\)and\(e_{\mathit{ez}}\) \(\mathit{exam}\leftarrow[1,0,0]\) \(e_{\mathit{ez}}\gets R_{init}[2]\) \(q,w\gets getIntersectionLine(e_{\mathit{exam}},e_{\mathit{ez}})\) //Computebeginbeginpositionalongthelineofsight \(t_{\mathit{begin}}\leftarrow\frac{\mathit{min}(\mathit{railis3D})-q,z}{w,z}\) \(\mathit{posBegin3D}\leftarrow q+t_{\mathit{begin}}\times w\) \(\mathit{posBegin3D}\leftarrow\mathit{projectToPlane}(\mathit{posBegin3D},e_{ \mathit{ex},e_{\mathit{ey}}})\) \(\mathit{posBegin2D}\leftarrow\mathit{projectToPlane}(\mathit{posBegin3D},e_{ \mathit{ex},e_{\mathit{ey}}})\) \(\mathit{//Optimizationloop}\) \((\hat{\alpha},\mathit{optimalHist})\leftarrow\mathit{optimization}(\mathit{railis2D},0)\) //Clusterhistogramtofindrails \(\mathit{cluster}\gets DBSCAN(\mathit{optimalHist})\) //Findnearesttwoclusterstostartingposition \((left,right)\gets findNearestCluster(cluster,\mathit{posBegin2D}.y)\) //Computescalingfactor
```

**Algorithm A5**calibration

```
Input:\(\mathit{gauge}\),\(R_{init}\),\(T_{init}\),\(\mathit{railis3D}\) Output:\(R\),\(T\),\(s\) //Projectrailpointsontogroundplanespannedby\(e_{\mathit{ex}}\)and\(e_{\mathit{ey}}\) \(\mathit{railis2D}\leftarrow\mathit{projectToPlane}(\mathit{railis3D},e_{\mathit{ex},e