# Deep sensorless tracking of ultrasound probe orientation during freehand transperineal biopsy with spatial context for symmetry disambiguation

Chloé Soormally, Clément Beitone, Jocelyne Troccaz, Sandrine Voros

► To cite this version:

Chloé Soormally, Clément Beitone, Jocelyne Troccaz, Sandrine Voros. Deep sensorless tracking of ultrasound probe orientation during freehand transperineal biopsy with spatial context for symmetry disambiguation. Computers in Biology and Medicine, 2025, 196, pp.110777. 10.1016/j.compbiomed.2025.110777. hal-05247944

HAL Id: hal-05247944

https://hal.science/hal-05247944v1

Submitted on 10 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Deep sensorless tracking of ultrasound probe orientation during freehand transperineal biopsy with spatial context for symmetry disambiguation

Chloé Soormally¹,*, Clément Beitone¹, Jocelyne Troccaz¹, and Sandrine Voros¹

¹Univ. Grenoble Alpes, CNRS, UMR 5525, VetAgro Sup, Grenoble INP, INSERM, TIMC, 38000 Grenoble, France

*Corresponding author : Chloé Soormally, chloe.soormally@univ-grenoble-alpes.fr

## Abstract

**Background** Diagnosis of prostate cancer requires histopathology of tissue samples. Following an MRI to identify suspicious areas, a biopsy is performed under ultrasound (US) guidance. In existing assistance systems, 3D US information is generally available (taken before the biopsy session and/or in between samplings). However, without registration between 2D images and 3D volumes, the urologist must rely on cognitive navigation.

**Method** This work introduces a deep learning model to track the orientation of real-time US slices relative to a reference 3D US volume using only image and volume data. The dataset comprises 515 3D US volumes collected from 51 patients during routine transperineal biopsy. To generate 2D images streams, volumes are resampled to simulate three degrees of freedom rotational movements around the rectal entrance. The proposed model comprises two ResNet-based sub-modules to address the symmetry ambiguity arising from complex out-of-plane movement of the probe. The first sub-module predicts the unsigned relative orientation between consecutive slices, while the second leverages a custom similarity model and a spatial context volume to determine the sign of this relative orientation. From the sub-modules predictions, slices orientations along the navigated trajectory can then be derived in real-time.

**Results** Results demonstrate that registration error remains below 2.5 mm in 92% of cases over a 5-second trajectory, and 80% over a 25-second trajectory.

**Conclusion** These findings show that accurate, sensorless 2D/3D US registration given a spatial context is achievable with limited drift over extended navigation. This highlights the potential of AI-driven biopsy assistance to increase the accuracy of freehand biopsy.

## 1 Introduction

Prostate cancer is the most prevalent male cancer in over a hundred countries [1]. In April 2024, the *Lancet* Commission on Prostate Cancer issued a set of recommendations aimed at addressing the anticipated surge in new cases in the coming years [2]. This projected increase is attributed to an aging population and rising life expectancy. The recommendations are tailored to the resources available in each country, but a critical component in combating prostate cancer is the ability to diagnose the disease accurately and at an early stage. While various screening methods exist — such as clinical examinations, PSA blood tests, and PI-RADS scores from MRIs — the diagnosis requires a prostate biopsy followed by a histological examination (tissue analysis with a microscope for cancer detection and grading). The ultrasound-guided biopsy examination, typically performed after a pre-operative MRI, aims to collect different prostatic cores. These samples are taken from both the targeted sites identified with the MRI and systematic sites determined by a biopsy scheme.

The transrectal (TR) approach has traditionally been preferred for prostate biopsy. However, the transperineal (TP) approach has gained popularity due to recent changes in practice. Initially performed under general anesthesia, transperineal biopsy is now conducted as an outpatient procedure under local anesthesia, similar to transrectal biopsy. With a lower risk of infection and comparable cost-efficiency, transperineal biopsy presents a compelling alternative to the transrectal approach [3]. The TR and TP approaches differ in three key aspects: the ultrasound probe type (end-fire for TR, side-fire for TP), the needle entry point (rectum for TR, perineum for TP), and needle mobility (attached to the probe in TR, free in TP). Regarding the use of needle guides and probe supports, there is currently no consensus, but it is worth noting that freehand practice remains the most cost-effective option. In addition, an american study [4] suggests that
freehand practice samples the prostate better than grid-based (and therefore probe-supported) practice. Regardless of the chosen approach - transrectal or transperineal - the accuracy issues are the same. Identifying and targeting the pre-defined areas under US guidance is a difficult exercise for urologists who are novices or rarely perform this type of procedure. However, not following the biopsy scheme or poorly targeting the lesions can lead to incorrect diagnosis and treatment.

The issue of precision in prostate biopsy has given rise to a number of commercialized assistance systems based on MRI/ultrasound and US/US image fusion. These include among others the Uronav system (Phillips [5]), Artemis (Eigen [6]) and Trinity® stations (Koelis [7]). These systems can be divided into two categories, depending on whether they perform *probe-tracking* or *organ-tracking* image fusion [8, 9]. Probe-tracking systems, such as Uronav or Artemis, rely on an external localizer - respectively an electromagnetic (EM) tracker and a mechanical-position encoder. They operate in real-time with the 2D ultrasound stream, but are less accurate, as they do not take into account organ deformation or patient movement.

Trinity® is currently the only fusion system that performs organ-tracking fusion. The station is equipped with a motorized ultrasound probe enabling 3D ultrasound acquisitions - in addition to real-time 2D ultrasound streaming. After a 3D US acquisition at the very beginning of the procedure, the system performs an initial 3D MRI/3D US deformable registration followed by 3D US/3D US rigid registrations each time a new 3D US biopsy volume is acquired. This acquisition is performed either to record the location of the core with the needle in place (real biopsy), or to check the needle position before triggering (virtual biopsy). The Trinity® system is cost-efficient and minimally disruptive to existing practices but does not currently provide real-time assistance, operating only with 3D image registration which takes a few seconds.

Throughout the procedure, 3D US/3D US registrations considerably improves the precision of the biopsy. However, freehand navigation between two biopsy sites remains challenging, as it is purely cognitive, i.e. based on the urologist's mental representation of space. In this paper, we propose a deep-learning method to track in real-time probe orientation during freehand navigation between two biopsy sites. Probe tracking is based solely on image content - in line with the Trinity® system, which uses no external sensors. An illustration of the clinical context of this work and its objective is presented in Figure 1.

By assisting navigation between biopsy sites, this approach aims to help urologists reach targets more easily, reducing reliance on mental representation. This can lower the number of virtual biopsies, shorten procedure time, and make the technique more accessible to urologists with less experience or who perform it infrequently. Given the increasing incidence of prostate cancer and the critical need for early detection, such advancements are essential to improve diagnostic accuracy and procedural efficiency.

<figure><img src="image_2.png" /><figcaption>Figure 1: Clinical workflow for a sensorless freehand fusion system for prostate biopsy assistance (such as Trinity®). 3D US volumes are acquired at each biopsy site and rigidly registered with the initial 3D US volume (panorama), which is in itself elastically registered with the pre-procedure MRI. In the current registration pipeline, freehand navigation between two biopsy sites is difficult because the system does not provide real-time assistance.</figcaption></figure>
# 2 Related work

## 2.1 Iterative slice-to-volume registration for motion correction in image-guided prostate biopsy

The problem of estimating the orientation of a 2D ultrasound slice in a volume can be seen as a slice-to-volume registration problem - also known as 2D/3D registration.

Ferrante et al. [10] proposed a literature review on traditional iterative methods for slice-to-volume registration. They presented a formalization of this registration problem. Considering a slice *I* extracted from a volume *V*, registering the slice *I* to the volume *V* amounts to finding the matching function *T* that optimally aligns the slice *I* within the volume *V* by minimizing the function *M(I,V,T) + R(I,V,T)* where *M* is a similarity criterion and *R* is a regularization term. The similarity criterion - which measures the alignment between the two images - can be iconic if based on voxel intensity, or geometric if based on anatomical or salient landmarks in the images. The correspondence function can be modeled by a rigid (six degrees of freedom), linear (9 degrees of freedom), or deformable transformation (for instance Thin Plate Spline or Free Form Deformation).

In the context of image-guided prostate biopsy, a number of studies have tackled the problem of 2D US/3D US or 2D US/3D MRI registration to correct prostate motion and thus increase biopsy precision. Approaches often perform iterative 2D US/ 3D US registration [5, 6, 11, 12, 13, 14, 15] and more rarely 2D US / 3D MRI registration [16]. Most of them require probe tracking by an external localizer [5, 6, 11, 12, 13, 16] - such as an electromagnetic or optical tracker - to initialize and accelerate image-based registration. Such registration is often based on iconic similarity metrics such as SSD (Sum of Squared Difference) [5], NCC (Normalized Cross-Correlation) [13, 11] or MI (Mutual Information) [16]. Only more recent approaches have overcome the constraint of an external localizer, using a lighter inertial sensor (IMU) [14], or no sensor at all [15]. Both approaches use a hybrid similarity metric, based on iconic and geometric features. Additionally, estimated transformations are often rigid, with a non-rigid correction rarely added [12]. The methods are almost exclusively statically validated by calculating the target registration error (TRE) on a few anatomical marker on real biopsy data. Only Beitone et al. [15] proposes a dynamic evaluation along the trajectory.

The major drawbacks of these iterative registration approaches are the need for good initialization to ensure the convergence of the optimization as well as their computation time, which is not compatible with real-time use despite attempts to accelerate the processes.

## 2.2 Deep slice-to-volume registration

Medical image registration using deep learning (DL) is becoming increasingly popular [17]. Deep learning makes it possible - once the model is trained - to estimate the transformation that optimally aligns the slice in the volume in a single step, without iteration. This greatly accelerates registration, making it possible to envision real-time use.

### 2.2.1 Sensorless ultrasound reconstruction

Ultrasound volume reconstruction from a freehand ultrasound probe scan/sweep is one of the major application of slice-to-volume registration. With the advent of deep-learning, many sensorless ultrasound reconstruction studies have emerged, following the pioneering work of Prevost et al. [18]. The methods developed for this particular application are somewhat of a precursor to the deep-learning methods developed for navigation assistance during prostate biopsy.

Methods for 3D ultrasound reconstruction often use as input two consecutive slices [18, 19, 20] or more [21, 22, 23, 24, 25, 26, 27]. Some approaches [22, 24] consider inertial sensor (IMU) measurements as additional input, and combine them with the input images. The output of the models are relative rigid transformations between slices, with the exception of one work [21] which seeks to exploit correlations between consecutive slices to improve localization in the atlas. The architectures used are varied, but the authors often exploit CNN backbone architectures from the literature such as as ResNet [19, 22, 26], VGG [21], DenseNet [23] or EfficientNet [27]. Some approaches seek to exploit spatial and/or temporal correlations within image sequences using recurrent networks [22, 26, 27], attention mechanisms [21, 19] or networks derived from natural language processing such as Seq2Vec [25] or the Transformer [24]. Finally, in most networks, the last block is a multi-layer perceptron (MLP) used to estimate the parameters of the transformations. In all studies, learning is supervised and associated with an MSE (mean squared
error) or MAE (mean absolute error) cost function. However, some authors combine this function with other cost functions: self-supervised functions for online training [26], Pearson correlation [22, 26] or other customized functions for multi-task training [25, 27]. The data used to train and evaluate the models are often 2D ultrasound slice sequences (video streams) [18, 19, 20, 22, 23, 24, 25, 26, 27]. Ground-truth (GT) relative localizations for models training and evaluation are obtained with a calibrated EM, optical sensor or an inertial measurement unit (IMU) that tracks the ultrasound probe. Other studies sample slices in 3D ultrasound volumes [21, 28, 29]. GT locations are thus available directly without an external calibrated sensor.

Since the aim of these methods is reconstruction, trajectories followed by the probe are sweeps, with a defined direction for one-dimensional rotation or translation. Only Luo et al. [26] considered more complex 'loop-scan' trajectories, with a change of direction. However, the trajectory remained very simple (one translational degree of freedom) and not compatible with a perfectly free hand-held movement. The authors also pointed out that these trajectories were associated with poorer results than simple scans.

As Prevost et al. pointed out in their work [18], the constraint of very simple sweeps acquired in a fixed direction is essential for training their model. However, this limitation goes beyond their method. It is the very symmetry of the trajectory estimation problem that makes the estimation of out-of-plane transformations (translation or rotation) ambiguous. Naturally, for reconstruction purposes, it is not very constraining to impose simple trajectories with a fixed direction. On the other hand, for navigation assistance with complex hand-held probe movements without a fixed direction, this constraint is not an option.

The main challenge in adapting reconstruction work for navigation assistance is therefore to find a way to overcome the symmetry ambiguity.

### 2.2.2 Sensorless navigation assistance between biopsy sites under 2D US guidance

As explained in the above paragraph, removing symmetry ambiguity is necessary for methods aimed at assisting biopsy navigation. Literature is still scarce for this very specific application, and we will therefore present the only three existing works, which are also the closest to our problematic. Furthermore, it should also be remembered that, as part of a biopsy procedure, 3D volumes (obtained by EM tracking or motorized probe) are generally available and can therefore be exploited to assist navigation.

Guo et al. [30, 31] and Dupuy et al. [32] developed DL-based methods in the context of real-time navigation assistance for prostate biopsy. Both studies focused on transrectal biopsy, historically the most common method, which uses an end-fire probe with an interdependent needle. The studied probe trajectories involved base-to-apex sweeping motions, which are essentially rotation around one axis — a simplified representation of the potential movements in transrectal biopsy.

Dupuy et al. [32] took inspiration from the 2D CNN-based architecture from Prevost et al. [24], which they adapted to consider more inputs. In addition to positional information (previous predicted transformation, inertial sensor information), they compared the performance of various context spatial inputs (previous slice, context volume or context sub-volume). Their best model was obtained by using the slice preceding the current slice as spatial information, and the two positional information (including an IMU sensor). In addition, they proposed a validation protocol for their method based on sequential evaluation that exploits the 3D/3D registration computed by existing assistance systems (e.g. Trinity®). The main metric used to assess registration quality is a surface based registration metric, calculated with the prostate mesh supplied with the data. The approach of Dupuy et al. relied heavily on a simulated IMU sensor, which certainly removed the ambiguity, but is not strictly speaking a sensorless method.

Guo et al. worked with 619 TRUS volumes reconstructed by EM tracking. They first proposed [30] a convolutional network estimating the orientation of the current slice in its underlying volume - i.e. the volume from which it originates - taking the slice and volume as input. In their more recent study [31], they proposed to track the orientation of the current slice along the trajectory by successively predicting the relative orientation between two consecutive slices using a previously developed network [33]. The underlying volume was also used to refine the predicted orientation through an similarity-based iterative correction block. Both approaches therefore exploited a spatial context volume in addition to the 2D US stream. This certainly helped to reduce the ambiguity of symmetry, although the authors did not make this clear. However, as they used the volume from which the slices originate as a spatial context, their evaluation lacked clinical realism, due to noise and deformations that may occur during navigation. These deformations could be caused by the patient's movement or the pressure exerted by the probe or needle on the prostate. It is therefore highly uncertain whether their method can be generalized to a more realistic clinical scenario.
### 2.2.3 Specificity of biopsy navigation assistance

In conclusion, we would like to highlight two key differences between biopsy navigation assistance (this work) and 3D US reconstruction:

1. It is possible to constrain the probe trajectories for 3D US reconstruction (e.g, scan trajectories with defined direction of translation and/or rotation), which is not an option for navigation assistance as the urologist must be free to move the probe as they wish.

2. It is possible to use regularly acquired 3D ultrasound volumes as spatial context for navigation assistance, which is not the case for 3D reconstruction as it aims at reconstructing those volume in a freehand way.

## 2.3 Contributions

The contributions of this work are fourfold:

* We propose a novel deep learning method to track the orientation of an US probe rotating around a fixed point during transperineal biopsy. Our *organ-tracking* approach is based solely on US image content and does not require an external localizer, since probe orientation is defined in relation to the last 3D US volume acquired, rather than in absolute world coordinates. This means that, unlike the *sensor-tracking* methods, prostate rotation movements with respect to the probe can also be taken into account.

* Our method leverages spatial context available in the form of 3D ultrasound volumes acquired regularly during a biopsy procedure to resolve sign ambiguity caused by the symmetry of the problem. This enables the consideration of trajectories frequent change of rotation direction, making it suitable for navigation purposes. To ensure realism, the context volume used to resolve ambiguity is different from the volume from which slices are extracted during data simulation.

* A trainable iconic similarity module has been developed to overcome the limitations of conventional iconic similarity measures when the content of images differs due to noise, distortion or movement. The relevance of this module is highlighted by comparison with the SSIM (Structural Similarity Index Measure) function.

* The model architecture is kept simple, comprising two ResNet-18 [34] blocks, thereby limiting the number of trainable parameters to 32 million. This simplicity allows for quick training of the model, real-time inference time (40 Hz), and facilitates possible deployment on routine clinical equipment. By way of comparison, Guo et al. [31] propose an approach based on two ResNeXt-50 [35] models, totaling about 60 million parameters while the models proposed by Dupuy et al. [32], depending on the spatial context input they consider, have between 39 million and 140 million parameters.

# 3 Methods

## 3.1 Problem statement

We consider a transperineal biopsy procedure in which a 3D US biopsy volume $V$ has just been acquired using the *3D-mode* of a motorized side-fire probe. After volume acquisition, the urologist can switch the probe to *2D-mode* to navigate to the next biopsy site using the real time US images stream as support.

We further assume that during navigation the US probe essentially rotates around a fixed point (pivot point) at the entrance of the patient's rectum. This hypothesis is supported by previous studies in the literature [36, 37], which define the working space of an endorectal probe (end-fire probe) as a cone whose origin is the rectal entrance and whose angle is 60°. Our work is based on transperineal biopsy with a side-fire probe, whose main rotation is around the probe axis (transverse transducer), but which has essentially the same degrees of freedom as an end-fire probe.

The rigid pose $\Pi$ of the probe in relation to the volume $V$ can be described in a reference frame associated with volume $V$ - denoted $\mathcal{R}_V$ and centered on the pivot point. The pose can be parametrized by a 6-dimensional vector with three parameters of translations $T = (t_x, t_y, t_z)$ and three parameters of rotations $\Theta = (\theta_x, \theta_y, \theta_z)$ (Euler angles).

The navigation/probe trajectory can be represented by a sequence of $N$ probe poses $(\Pi_t = [T_t, \Theta_t])_{t \in [1,N]}$. Let $(I_t)_{t \in [1,N]}$ represent the US slices capturing the cross-sectional views corresponding to this sequence.
It must be emphasized that these slices result from the ultrasound stream of the 2D-mode navigation (and not from resampling of volume V). Because of noise, patient movements and/or prostate deformations, the iconic content of slice $I_t$ differs from the content of the slice resampled in volume V at $\Pi_t$ position.

When navigation starts, the initial pose of the probe in relation to V is known (see Figure 2) as the 3D volume V has just been acquired. We can therefore set $\Theta_1 = 0$ and $T_1 = 0$ without loss of generality. Furthermore, under the 'rotational movement' hypothesis, the variation of $t_x, t_y, t_z$ can be considered small, and the pose can be approximated by the vector $[0, \Theta_t]$.

The problem we address is that of estimating, for each $t$, the transformation parameters that optimally aligns slice $I_t$ within volume V. Assuming that i) the rigid displacement comes essentially from the rotation of the probe/prostate around the rectal entrance and ii) deformations of the prostate are small between two 3D acquisitions, this optimal transformation parameters can be approximated by $\Pi_t = [0, \Theta_t]$. In the remainder of this section, we propose a method for estimating the orientation $\Theta_t$ based on the slices $I_1, I_2, ... I_N$ along with the volume V.

<figure><img src="image_3.png" /><figcaption>Figure 2: Problem statement. Let $\mathcal{R}_V$ denote the reference frame (Pivot point, x, y, z) associated with volume V. When navigation starts, the orientation $\Theta_1 = 0$ (black contour) of the probe in $\mathcal{R}_V$ - and therefore of the corresponding view $I_1$ - is known. The probe axis is aligned with the z axis, while the US slice is in the (Pivot, y, z) plane. A time $t$, the orientation $\Theta_t$ (green contour) of the probe in $\mathcal{R}_V$ - and its corresponding view $I_t$ - is unknown and must be recovered.</figcaption></figure>

## 3.2 Preliminary method: Estimating $\Theta_t$ solely from the real-time ultrasound stream

We consider the scenario where the 3D volume V is not used, and only the real-time ultrasound image stream serves as input for estimating $\Theta_t$. This subsection demonstrates that estimation is theoretically possible, but in practice the method is not very robust to model prediction errors. This preliminary result justify the use of a context volume to remove the ambiguity associated with symmetry - as detailed in the next subsection.

Because of the symmetry of the problem mentioned in the literature review, and in particular highlighted by Prevost et al. [18], it is not possible to directly estimate the sign of relative angles between two consecutive slices, i.e., directly estimate the vector $\Delta\Theta_t = \Theta_t - \Theta_{t-1}$. However we demonstrate that it is possible to detect whether a change in the sign of one or several relative angles occurs during the trajectory.

A first network based on a ResNet-18 backbone [34] was developed to perform this sign change detection task. The network - which architecture is presented in Figure 3 - was trained in a supervised manner with the MSE loss. Considering three consecutive slices $I_{t-2}, I_{t-1}, I_t$ the trained model - denoted $f_0$ - estimates the three parameters of the vector $\Psi_t = |\Delta\Theta_t| \cdot \text{sign}(\Delta\Theta_{t-1} \cdot \Delta\Theta_t)$. To avoid any confusion, we specify that $|v|$ (resp. $\text{sign}(v)$) with $v = [v_1, v_2, v_3]$ corresponds to the vector $[|v_1|, |v_2|, |v_3|]$ (resp. $[\text{sign}(v_1), \text{sign}(v_2), \text{sign}(v_3)]$). Similarly $\cdot$ represents here the element-wise multiplication between two 3-dimensional vectors.

$\Theta_t$ can be estimated by determining the unsigned relative angles and whether or not the signs of these relative angles have changed with respect to time $t-1$.
<figure><img src="image_4.png" /><figcaption>Figure 3: Network architecture for $f_0$ and $f_1$. The original dense layers of the ResNet-18 have been replaced by a new MLP, whose layers are detailed in the figure, with ReLU activations.</figcaption></figure>

More formally, by setting:

$$ \left\{ \begin{array}{l} \hat{\Psi}_t^{abs} := |\hat{\Psi}_t| \\ \hat{\Psi}_t^{sign} := \text{sign}(\hat{\Psi}_t) \end{array} \right. \quad \text{with} \quad \hat{\Psi}_t = f_0([I_{t-2}, I_{t-1}, I_t]) $$

$\Theta_t$ can be estimated with the following equations:

$$ \begin{aligned} \hat{\Sigma}_1 &= \text{sign}(\Delta\Theta_1) \\ \hat{\Sigma}_t &= \hat{\Psi}_t^{\text{sign}} \cdot \hat{\Sigma}_{t-1} \\ \hat{\Theta}_t &= \hat{\Theta}_{t-1} + \hat{\Sigma}_t \cdot \hat{\Psi}_t^{abs} \end{aligned} $$

This algorithm makes it possible theoretically to overcome the symmetry problem. In practice, however, the algorithm is not robust enough due to sign misclassifications. These errors propagate along the entire trajectory resulting in mirrored trajectory as it is illustrated in subsection 5.1. To resolve the sign ambiguity in a more robust way, we propose exploiting the volume $V$ acquired just before navigation as spatial context.

## 3.3 Exploiting spatial context to resolve sign ambiguity

It is possible to leverage the 3D volume $V$ acquired just before navigation to remove ambiguity about the direction of movement. We propose in this sub-section a method that estimates the direction based on spatial context information derived from $V$ - that we will from now on refer to as spatial context volume. An overview of the method is presented in Figure 4

### 3.3.1 Estimating the unsigned relative orientation

The network architecture from section 3.2 was reused for a slightly different estimation task. Instead of predicting $\Psi_t$, the network was trained to directly predicts $\Psi_t^{abs} = |\Delta\Theta_t|$. Let $f_1$ be this new trained model, $\hat{\Psi}_t^{abs}$ is now directly the prediction of $f_1$:

$$ \hat{\Psi}_t^{abs} = f_1(I_{t-2}, I_{t-1}, I_t) $$

### 3.3.2 Estimating the sign of relative orientation

We propose a method to estimate $\Sigma_t = \text{sign}(\Delta\Theta_t)$ based on targeted resampling in the spatial context volume and on a similarity model that performs an eight-class classification.

More precisely, if we consider i) the estimate $\hat{\Psi}_t^{abs}$ of $|\Delta\Theta_t|$ from 3.3.1 and ii) an estimation of the previous orientation $\hat{\Theta}_{t-1}$, we can generate the eight orientations derived from the eight combinations of signs:

$$ \Phi_{s,t} = \hat{\Theta}_{t-1} + s \cdot \hat{\Psi}_t^{abs} $$

with $s \in S = [[1, 1, 1], [1, 1, -1], ..., [-1, -1, -1]]$ and $|S| = 2^3$.
<figure><img src="image_5.png" /><figcaption>Figure 4: Proposed network leveraging spatial context to resolve sign ambiguity. The network takes as input three consecutive slices - the current slice at time t and the two previous ones - as well as a spatial context volume V. The output is the estimate of the orientation Θ$_t$ of the current slice in V. Sub-model ① estimates $\Psi_t^{abs} = |\Delta\Theta_t|$ using the input slices. Sub-model ② estimates $\Sigma_t = \text{sign}(\Delta\Theta_t)$ through targeted resampling in the spatial context volume, using the prediction of the sub-model ① and the estimation of the previous orientation $\hat{\Theta}_{t-1}$. Finally, in ③, an estimation $\hat{\Theta}_t$ is derived by combining predictions from sub-models ① and ② and the previous orientation estimation.</figcaption></figure>

Let $(J_{\Phi_{s,t}}^V)_{s \in S}$ denote the eight corresponding slices resampled in the spatial context volume V. The objective is to determine the slice $J_{\Phi_{s,t}}^V$ that is most similar to the current slice $I_t$ as the corresponding sign s gives an estimate of $\Sigma_t = \text{sign}(\Delta\Theta_t)$. Since movements and deformations of the prostate can occur during navigation, and since the $\Theta_{t-1}$ estimate is imperfect, using an iconic similarity metric like SSIM to find the most similar slice is not robust enough. We therefore designed a network based on a shared siamese architecture with a ResNet-18 backbone to perform the comparison. An illustration of the proposed architecture is provided in Figure 5.

The network was trained in a supervised manner with the cross-entropy loss using $\Sigma_t$ (encoded with class indices) as target. Let $f_2$ denote the trained similarity model, the estimation $\hat{\Sigma}_t$ is obtained with :

$$ \hat{\Sigma}_t = S[\text{argmax}_{f_2}([I_{t-1}, I_t, (J_{\Phi_{s,t}}^V)_{s \in S}])] $$

### 3.3.3 Estimating the full orientation

Using the predictions of the two previous sub-models $f_1$ and $f_2$, the orientation estimate can be updated at each time step t:

$$ \hat{\Theta}_t = \hat{\Theta}_{t-1} + \hat{\Sigma}_t \cdot \hat{\Psi}_t^{abs} $$

where $\hat{\Psi}_t^{abs} = f_1(I_{t-2}, I_{t-1}, I_t)$ and $\hat{\Sigma}_t = S[\text{argmax}_{f_2}([I_{t-1}, I_t, (J_{\Phi_{s,t}}^V)_{s \in S}])]$.
<figure><img src="image_6.png" /><figcaption>Figure 5: Network architecture for $f_2$ (relative orientation sign estimation). The model $f_2$ takes as input the eight image triplets $([I_{t-1}, I_t, J_{\Phi_{s,t}}^V])_{s \in S}$ and estimates $\Sigma_t = \text{sign}(\Delta\Theta_t)$. The architecture is of the 'shared siamese' type, where each triplet of images is processed in turn by ResNet-18 networks (to which the classification layer has been removed) sharing weights. Each triplet is thus encoded by a vector of size 512. These vectors are concatenated into a vector of size 4096, which is processed by an MLP (with ReLU activations) before the final eight-class classification layer.</figcaption></figure>

# 4 Experimental setup

## 4.1 Clinical data

A biopsy procedure using the Trinity® system can be summarized as follows:

1. A panoramic volume (or panorama) is acquired at the very beginning of the procedure. A prostate segmentation is obtained in a semi-automatic way allowing the panoramic volume to be elastically registered with the pre-operative MRI.

2. At regular intervals during the procedure, the urologist acquires a 3D biopsy volume, which is rigidly registered with the panoramic volume. Let $T_{pano \to V_i}$ denote the rigid transformation between the i-th 3D biopsy volume and the panoramic volume. Depending on the context, the needle may either be present (real biopsy) in the biopsy volume, or absent (virtual biopsy). When absent, this transformation allows verification of the needle's position before insertion. When present, it enables recording and localizing the biopsy core within the panoramic volume.

The study is based on a set of 519 3D ultrasound volumes collected from 51 patients during transperineal biopsy procedures at Grenoble University Hospital. The volumes were acquired using the Trinity® system (Koelis, Grenoble) and a side-fire probe. Each patient is associated with i) a panoramic volume acquired at the start of the procedure ii) a set of biopsy volumes of variable size ($V_i$)$_{i \in [1, 2, ...,]}$ acquired at each real or virtual biopsy site. Segmentation in the form of a mesh is also available for each panoramic volume. The dimensions of the volumes - which are in gray scale - are 304 x 256 x 256. Resolution, which varies slightly between volumes (around 0.35 mm), is isotropic. A summary of the clinical data is shown in Figure 6.

## 4.2 Data simulation

The current clinical workflow does not provide access to the ultrasound stream or the probe trajectory, as they are not recorded by the system. To address this limitation, we simulated our working data—including the probe trajectory, ultrasound stream, and corresponding spatial context volume—using available clinical data. The simulation approach is inspired by the method proposed by Dupuy et al. [32]. The complete simulation process, illustrated in Figure 7, is detailed in this subsection.

### 4.2.1 Probe trajectories

Probe trajectories were simulated by generating random $(\Theta_t)_t$ sequences of various lengths with $\Theta_1 = [0^\circ, 0^\circ, 0^\circ]$. Examples of trajectories of 50, 100 and 500 slices are presented in Figure 8. All the details about the generating process are given in Appendix A.

### 4.2.2 Ultrasound images stream

We simulated US images streams associated with a rotational movement of the probe around the entrance of the rectum. For each sequence, US slices were sampled from a single biopsy volume at $(\Theta_t)_t$ orientations
<figure><img src="image_7.png" /><figcaption>Figure 6: Clinical data. Our dataset consists of MRI volumes (not used in this paper), panoramic volumes (associated with a mesh of the prostate) and biopsy volumes. It also contains the transformations between volumes as presented in the figure and calculated by the Trinity® system.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Figure 7: Procedure for obtaining our working data from clinical data. For each biopsy volume, a complex 3-degree-of-freedom probe trajectories with numerous changes in rotation direction was simulated. Slices were resampled from each biopsy volume along the associated trajectory to simulate real-time US streams. For a given patient, the spatial context volume associated with the US stream derived from volume #i was chosen from the biopsy volumes #i (source), #i-1 and #i+1 (neighbors) and panoramic volume. Using the neighboring or panoramic volumes is more clinically realistic as it ensures that the iconic content used to generate the 2D stream is different from the iconic content of the spatial context volume.</figcaption></figure>
<figure><img src="image_9.png" /><figcaption>Figure 8: Probe trajectories simulation. The Euler angles are bounded by the intervals [-10°, 10°] for the x and y axes and by [-30°, 30°] for the z axis. The relative Euler angles are chosen in the interval [-1°, 1°] for the x and y axes and in [-3°, 3°] for the z axis.</figcaption></figure>

where the sequence $(\Theta_t)_t$ was generated according to subsection 4.2.1 (see Figure 9). The center (pivot point) of the associated transformation was selected for each biopsy volume so as to be close to the entrance to the rectum. The sequence of $(\Theta_t)_t$ gives the GT orientation of the slices from the generated ultrasound stream.

For a given sequence, the volume from which the slices originate is referred to as the source volume ($V_{source}$). It should be emphasized that this volume does not exist in the clinical workflow, but is only required to generate the ultrasound stream.

Since we sampled in volumes, the resampled slices contained a variable amount of information. Depending on the orientation, the slice contained more or less black pixels at the borders. In order not to bias our learning process, all slices from the same volume were processed to contain the same black pixel edges - i.e. the smallest common area containing information (strictly positive pixels) was selected. In order to retain sufficient information in the images, the Euler angles of the trajectories were chosen in small ranges, namely [-10°, 10°] for the angles around x and y and [-30°, 30°] for the angle around z.

<figure><img src="image_10.png" /><figcaption>Figure 9: Ultrasound images stream simulation. Ultrasound slices are sampled from the biopsy volumes at $(\Theta_t)_t$ orientations to simulate a rotational movement with three degrees of freedom around a fixed point.</figcaption></figure>

### 4.2.3 Spatial context volume

In the clinical scenario, the spatial context volume typically corresponds to the volume acquired just before the start of navigation, referred to as V in the previous section. For this work, we chose this context volume to ensure some clinical realism. Several levels of realism were considered depending on the choice of spatial context volume: i) the source volume $V_{source}$ (from which the simulated US images stream originates) ii) the panoramic volume $V_{panorama}$ acquired at the very start of the procedure or iii) a neighboring volume $V_{neighbor}$ to the source volume (i.e. acquired just before or just after during the procedure). The first level, found in the literature (Guo et al. [15, 16]), is not very realistic, as it does not take into account possible movement and deformation of the prostate. The second level is not the most realistic either, as in practice there should not be too much displacement between the context volume and the real-time US slices. The third level, with a context volume close to but different from the source volume, allows the difficulty of noise, movement and distortion to be reasonably taken into account.

In the clinical workflow, the first stream image $I_1$ is at orientation [0°, 0°, 0°] in the context volume (see Figure 2). To reproduce this situation, given that the stream is generated from a source volume while using
another volume as context volume, it is necessary to align both volumes. Available rigid 3D/3D registrations
between each biopsy volume and the panorama enabled the spatial context volume to be rigidly registered
with the source volume.

Let $V_i$ denote the source volume corresponding to the $i$-th biopsy volume acquired during the procedure. When selecting the panoramic volume as the spatial context, the transformation $T_{pano \to V_i}$ is directly available. Alternatively, if the neighboring volume $V_{i-1}$ is chosen as the spatial context, the transformation $T_{V_{i-1} \to V_i}$ can be obtained through the composition $T_{pano \to V_i} \circ T_{pano \to V_{i-1}}^{-1}$. A similar approach applies when considering the neighboring volume $V_{i+1}$ instead.

For training only, the spatial context volume associated with an ultrasound stream generated from a source
volume was randomly selected at each time step between the panoramic volume and the two neighboring
volumes of the source volume. For inference, we compared different levels of realism by varying the spatial
context volume. We chose between i) the source volume ($V_{source}$), the panoramic volume ($V_{panorama}$) and
one of the two neighboring volumes ($V_{neighbor}$). Among the two neighboring volumes, we selected the
neighbor whose transformation to the panorama is closest to the transformation of the source volume to the
panorama.

## 4.3 Implementation details

The patients were divided into three sets: 43 patients for model training, 4 patients for validation and 4 patients for testing, corresponding to 376, 46 and 42 biopsy volumes respectively.

The sub-models $f_0$, $f_1$ and $f_2$ presented in section 3 (methods) were independently pretrained. Specific
datasets were created for this purpose. The datasets are presented in the appendix B. Sub-models were
trained with the Adam optimizer for 30 epochs, with a batch size of 50 and a learning rate of $10^{-4}$. The best-
performing weights on the validation data were retained. Performance was measured with the corresponding
loss function.

The sub-models $f_1$ and $f_2$ were then jointly fine-tuned on simulated image sequences. Two sequences of 50
slices were simulated per biopsy volume. The training parameters were consistent with those used during
pretraining, except for the batch size, which was set to 500 (comprising 10 trajectories of 50 slices each).
Best-performing weights on the validation data were retained. Performance during fine-tuning was evaluated
by computing the MSE between the ground-truth and estimated orientations. As an indication, we trained
our models on an NVIDIA A100 (40GB) GPU, and fine-tuning the two sub-models on sequences took around
45 minutes per epoch.

Furthermore, to ensure that the $f_2$ sub-model focuses only on the non-zero pixels of the images (and not
the borders related to resampling), we applied a mask to its input images so that they all contain strictly
the same amount of information. This step is only necessary because we did not have real ultrasound image
streams, and we therefore needed to resample in a 3D volume.

Finally, as the first prediction is $\hat{\Theta}_2$, we chose to initialize $\hat{\Theta}_0$ at [0°, 0°, 0°] and $\hat{\Theta}_1$ at [0.25°, 0.25°, 1°].

## 4.4 Evaluation metrics

From a clinical perspective, it has been suggested that the registration error should not exceed 2.5 mm, this
value being based on the size of the smallest treatable tumor estimated at 0.5 cc (i.e a sphere with a diameter
of approximately 5 mm) [38]. In order to evaluate our method in relation to the clinical needs, we propose
to use a surface-based registration metric. It was proposed in the literature [10, 32, 39] as a registration
evaluation metric when no anatomical marker are available to compute the TRE but a segmentation and the
ground-truth transformations are known. It is not directly related to the TRE as there is no measurement
error but note that in both cases the evaluation points (anatomical markers or mesh points) are not used in
the registration process. Moreover, in our case, the segmentation is highly localized to the area of interest
(prostate segmentation), and we therefore expect the surface-based registration error to approximate the
point-based registration error (TRE).

To assess the quality of tracking on a trajectory of *N* slices [*I*₁, *I*₂, *I*₃, ..., *I*ₙ] (corresponding to GT orientations [Θ₁, Θ₂, Θ₃, ..., Θₙ]), we computed a surface registration error (SRE) on the prostate mesh associated with the patient along the trajectory. More formally, for all *t*, if we denote $\hat{M}_t$ the transformation matrix of the estimated orientation $\hat{\Theta}_t$ and $M_t$ the transformation matrix of the ground truth (GT) orientation $\Theta_t$ and ($p_k$)$_{k \in 1...K}$ the *K* points of the mesh (coordinates in millimeters), the SRE (after registration) at slice index *t* is defined by:
$$ (\text{SRE})_t = \sqrt{\frac{1}{K} \sum_{k=1}^{K} ||M_t \cdot p_k - \hat{M}_t \cdot p_k||^2} $$

This value was compared with the SRE before registration at slice index *t* defined by :

$$ (\text{SRE before registration})_t = \sqrt{\frac{1}{K} \sum_{k=1}^{K} ||M_t \cdot p_k - p_k||^2} $$

Several metrics were derived from the SRE: the average SRE over the trajectory (Mean SRE), the SRE at index *t* = *N* (Final SRE), the ratio of *t* indices such that the SRE is below a certain threshold *x* (% indexes with SRE < *x*). Based on clinical precision requirements, we selected relevant threshold values, specifically 2.5 mm and 5 mm.

## 4.5 Experimental protocol

Several experiments were carried out to cover several key aspects of our problem (symmetry of estimation, relevance of context volume, trajectory length and drift, etc.). An overview of the experimental protocol is given in Table 1.

<table><caption>Table 1: Overview of the experimental protocol.</caption><thead><tr><th>Sub-section</th><th>Main objective</th><th>Test set</th><th>Ultrasound stream (generated from V<sub>source</sub>)</th><th>Spatial context volume</th></tr></thead><tbody><tr><td>5.1</td><td>Preliminary result : to highlight the symmetry</td><td rowspan="5">42 biopsy volumes + 4 panoramic volumes + 4 meshes</td><td>1 single trajectory<br>42 sequences (1 traj / volume)<br>100 slices (5s) / sequence</td><td>None<br>V<sub>neighbor</sub></td></tr><tr><td>5.2.1</td><td>To demonstrate the usefulness of the context volume in resolving the symmetry-related ambiguity</td><td>84 different trajectories<br>84 sequences (2 traj / volume)<br>100 slices (5s) / sequence</td><td>- V<sub>source</sub><br>- V<sub>panorama</sub><br>- V<sub>neighbor</sub></td></tr><tr><td>5.2.2</td><td>To compare the proposed module f<sub>2</sub> with (non trainable) iconic metric SSIM in connection with the choice of spatial context volume</td><td></td><td>- V<sub>source</sub><br>- V<sub>neighbor</sub></td></tr><tr><td>5.3</td><td>To compare the proposed method with the literature</td><td></td><td>V<sub>neighbor</sub></td></tr><tr><td>5.4<br>(short trajectories)<br>5.4<br>(long trajectories)</td><td>To analyze the influence of f<sub>2</sub> targets during training (inject previous prediction or not) in connection with the estimation drift along the trajectory</td><td>84 different trajectories<br>84 sequences (2 traj / volume)<br>500 slices (25s) / sequence</td><td>V<sub>neighbor</sub></td></tr></tbody></table>

# 5 Results and discussion

In this section, we present the results from the experimental protocol presented in Table ??

## 5.1 Preliminary results: Highlighting the symmetry problem

The method based solely on the 2D ultrasound image stream (i.e. without spatial context volume) from subsection 3.2 was evaluated on a single simulated trajectory. This trajectory was used to generate 42 US images sequences from the 42 biopsy volumes in the test set. This corresponds to 4200 images. The SRE before and after registration along the trajectory is shown in Figure 10a. It can be seen that the SRE increases very rapidly. Looking at a predicted trajectory in Figure 10b, we see that, as expected, this proposed context-less method is not very robust due to the symmetry of the problem. Sign changes are detected, but imperfectly, resulting sometimes in predicted trajectories that mirror the GT trajectory. It confirms the necessity to use spatial context to disambiguate the trajectory estimation.

The remainder of the **Experiments and results** section is devoted solely to the main method leveraging the spatial context volume - also referred to as the *context-based* method - presented in subsection 3.3.

## 5.2 Influence of spatial context volume in symmetry disambiguation

We studied the influence of using a spatial context volume in symmetry-related sign disambiguation as described in subsection 3.3. This context can be either the source volume $V_{source}$, the panoramic volume $V_{panorama}$, or the neighboring volume $V_{neighbor}$ (as chosen for inference in subsection 4.2.3).
<figure><img src="image_11.png" /><figcaption>(a) SRE (± standard deviation) along the trajectory.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>(b) Illustration of the symmetry problem.</figcaption></figure>

Figure 10: Performance of the *context-less* method. The model based solely on the 2D image stream can detect changes in the sign of the relative orientation vector, but is not robust to the symmetry issue yielding to mirrored trajectory.

### 5.2.1 Evaluation on a single trajectory

With $V_{neigbour}$ as context volume, we evaluated the model derived from the *context-based* method on the US images sequences from the previous subsection (all generated from a single trajectory). We observe in Figure 11 that the SRE remains relatively low along the trajectory. The mean SRE remains below 2.5 mm throughout the trajectory, meeting the clinically relevant precision threshold we established.

<figure><img src="image_13.png" /><figcaption>(a) SRE (± standard deviation) along the trajectory.</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>(b) Illustration of the tracking with spatial context.</figcaption></figure>

Figure 11: Performance of the *context-based* method. SRE ± standard deviation (before and after registration) along the trajectory is averaged over the 42 test volumes.

### 5.2.2 Evaluation with several trajectories

We evaluated further the *context-based* method by considering various trajectories of 100 slices. Although, as mentioned in subsection 4.2.3 using $V_{neighbor}$ best reproduces the clinical realism, we compared performance for different choices of spatial context volume. In addition, we wanted to highlight the relevance of the $f_2$ sub-model (proposed trainable similarity module), especially when the context volume is not the source volume. This was done by comparing the performance of the original model (with $f_2$) with that of the model in which we have replaced the $f_2$ sub-model with the (non-trainable) SSIM function. The latter takes as input the eight image doublets ($I_{t-1}, J_{\Phi_{s,t}}$)$_{s \in S}$ consecutively and returns the sign vector $\mathbf{s}$ of the doublet with the highest similarity between the two slices.

The method was evaluated on 84 different simulated trajectories. These trajectories were used to simulate 84 US images sequences by resampling slices in the 42 biopsy volumes from the 4 test patients (2 trajectories/volume). This brought the total number of images in the test set to 8400. Mean SRE over the trajectory,
final SRE, MSE and ratio of indexes with SRE below 2.5 mm and 5 mm are presented in Table 2. Values are averaged over the 84 trajectories We also report the SRE values before registration for reference.

The results highlight several interesting aspects. Firstly, the lowest level of realism ($V_{source}$ as spatial context) is associated with very good results for both the proposed model (with $f_2$) and the SSIM model. This is not surprising, as exploiting $V_{source}$ does not reproduce the clinical difficulty associated with noise, deformations and movements. It also indicates that the proposed sub-model $f_2$ does learn a similarity function, and achieves the performance of the SSIM function in the absence of noise, deformation or movement. However, when $V_{neighbor}$ or $V_{panorama}$ are exploited as spatial context, the proposed sub-model $f_2$ performs better than the SSIM function, highlighting the contribution of the former when there are noise, deformations and possible movement of the prostate. In addition, from the results in terms of MSE, we observe angle $\theta_x$ (in-plane transformation) is better tracked than angles $\theta_y$ and $\theta_z$ (out-of-plane transformations) which confirms that in-plane tracking is a simpler task than out-of-plane tracking.

<table><caption>Table 2: Performance of the *context-based* method over 84 test trajectories with different choices of spatial context volumes. The contribution of $f_2$ is highlighted by comparison with a similar model in which the sub-model $f_2$ is replaced by the SSIM similarity function.</caption><thead><tr><th rowspan="2">Context-based models</th><th rowspan="2">Mean SRE (mm)</th><th rowspan="2">Final SRE (mm)</th><th colspan="3">MSE (degree)</th><th colspan="2">% indexes with SRE &lt; x</th></tr><tr><th>θ<sub>x</sub></th><th>θ<sub>y</sub></th><th>θ<sub>z</sub></th><th>x = 2.5 mm</th><th>x = 5 mm</th></tr></thead><tbody><tr><td>Before registration</td><td>12.25 ± 6.03</td><td>13.00 ± 5.26</td><td>—</td><td>—</td><td>—</td><td>3.62 ± 5.06</td><td>11.35 ± 8.92</td></tr><tr><td colspan="8">Spatial context volume: V<sub>source</sub></td></tr><tr><td>With SSIM</td><td>0.68 ± 1.080</td><td>0.91 ± 1.36</td><td>0.07 ± 0.08</td><td>3.48 ± 14.89</td><td>3.03 ± 9.83</td><td>97.41 ± 11.48</td><td>98.79 ± 6.96</td></tr><tr><td>With f<sub>2</sub></td><td>1.15 ± 0.82</td><td>1.69 ± 1.79</td><td>0.32 ± 0.51</td><td>1.75 ± 2.34</td><td>3.51 ± 6.61</td><td>95.94 ± 14.64</td><td>99.78 ± 1.78</td></tr><tr><td colspan="8">Spatial context volume: V<sub>neighbor</sub></td></tr><tr><td>With SSIM</td><td>5.12 ± 7.06</td><td>8.09 ± 7.87</td><td>3.83 ± 7.92</td><td>13.00 ± 16.10</td><td>135.20 ± 258.25</td><td>57.40 ± 35.89</td><td>70.87 ± 32.24</td></tr><tr><td>With f<sub>2</sub></td><td>2.27 ± 4.28</td><td>3.90 ± 7.73</td><td>0.27 ± 0.29</td><td>6.22 ± 13.70</td><td>41.51 ± 108.43</td><td>86.33 ± 23.10</td><td>93.05 ± 17.83</td></tr><tr><td colspan="8">Spatial context volume: V<sub>panorama</sub></td></tr><tr><td>With SSIM</td><td>9.34 ± 9.39</td><td>14.89 ± 10.94</td><td>13.20 ± 37.99</td><td>26.99 ± 30.51</td><td>334.82 ± 389.80</td><td>31.64 ± 30.26</td><td>46.48 ± 31.41</td></tr><tr><td>With f<sub>2</sub></td><td>3.21 ± 6.09</td><td>6.530 ± 9.70</td><td>0.31 ± 0.34</td><td>8.79 ± 20.87</td><td>113.22 ± 20.87</td><td>78.00 ± 30.38</td><td>87.72 ± 21.41</td></tr></tbody></table>

To take into account the temporal (dynamic) aspect of a trajectory, Figure 12 presents a box plot of the SRE across several slice indexes. This box plot has been produced for the most clinically realistic scenario i.e. using $V_{neighbor}$ as spatial context. The box plot indicates that there are a few dozen outliers for each index studied. However, median and quartile values are very stable along the trajectory, suggesting little drift. The median stays below 2 mm until the end of the trajectory, and the third quartile remains below 2.5 mm. An analysis of drift over longer trajectories will be discussed in a subsequent subsection.

<figure><img src="image_15.png" /><figcaption>Figure 12: Box plot showing the SRE at different slice indexes. It was obtained with the *context-based* method leveraging $V_{neighbor}$ as context.</figcaption></figure>

## 5.3 Comparison to the literature

We compared the results of the proposed method with those reported by Dupuy et al. (Table 3) for the sensorless approach. It should be remembered that in the latter work, the results were based on simulated
'base-apex' trajectories for transrectal biopsy (end-fire probe). For a better comparison, we evaluated our method on simulated trajectories with one single DoF, corresponding to rotation around the probe axis (main rotation for a side-fire probe). For both works, the registration error was calculated in the same manner, i.e. the SRE on the prostate segmentation. Although we cannot make a strict comparison as we are not evaluating on the same data nor the same biopsy gesture, we note that the results of the proposed method are consistent with the literature. We remind that our aim is not to improve the performances on simple 1-DoF trajectories, but rather to develop a method that works for more complex 3-DoFs trajectories.

<table><caption>Table 3: Comparison with Dupuy et al., [32]. Mean SRE ± standard deviation before and after registration. SRE values are those reported by the authors for the sensorless approach.</caption><thead><tr><th>Biopsy data</th><th>Methods</th><th>Mean SRE before (mm)</th><th>Mean SRE after (mm)</th></tr></thead><tbody><tr><td>Transrectal (1 DoF)<br>(with V<sub>neighbor</sub>)</td><td>Dupuy et al. 2023 [32]</td><td>9.52 ± 6.49</td><td>2.74 ± 6.04</td></tr><tr><td>Transperineal (1 DoF)<br>(with V<sub>neighbor</sub>)</td><td>Proposed</td><td>9.23 ± 5.63</td><td>1.86 ± 3.27</td></tr></tbody></table>

Furthermore, we re-implemented the method of Guo et al. [31] (for transrectal biopsy) to evaluate it on our transperineal data with complex trajectories with three degrees of freedom.

We recall that their method is based on a first network - Frame2Frame - which calculated the relative rigid transformation between two consecutive slices. A (non-trainable) iterative similarity filtering (SF) module was then applied to correct the Frame2Frame prediction. Both blocks (Frame2Frame and SF) were reimplemented. Due to Frame2Frame's time-consuming training and mediocre performance, we proposed an alternative architecture for Frame2Frame, using a ResNet-18. Since Guo et al. [31] used the ultrasound volume reconstructed from the tracked ultrasound stream as the spatial context (which is analogous to using $V_{source}$ as spatial context volume), we compared the methods when using $V_{source}$ (clinically unrealistic) and volume $V_{neighbor}$ as spatial context. The results - presented in Table 4 - show a clear superiority of our method on these complex transperineal trajectories.

The values reported by the authors are not exactly comparable, as the data are not the same and the evaluation method is different, even though the size of the datasets is roughly the same. The authors measured the Distance Error, which measures the distance between the corner points of the 2D frame's ground-truth position and estimated position (in mm). For this metric, the authors reported an error (after registration) of 6.95 ± 8.03 mm on their (transrectal) data. This value is similar to the SRE value (7.40 ± 9.19mm) obtained with ResNet + SF on our transperineal data.

<table><caption>Table 4: Comparison with Guo et al. 2023 [31]. Mean SRE ± standard deviation before and after registration. SRE values are those we have obtained with our re-implementation. Guo et al. [31] report a Distance Error of 6.95 ± 8.03 mm (after registration) on their transrectal dataset.</caption><thead><tr><th>Biopsy data</th><th>Methods</th><th>Mean SRE before (mm)</th><th>Mean SRE after (mm)</th></tr></thead><tbody><tr><td rowspan="3">Transperineal (3 DoFs)<br>(with V<sub>source</sub>)</td><td>Frame2Frame ([31]) + SF ([31])</td><td rowspan="3">12.25 ± 6.03</td><td>10.14 ± 7.18</td></tr><tr><td>Proposed</td><td>1.15 ± 0.82</td></tr><tr><td>ResNet + SF ([31])</td><td>7.40 ± 9.19</td></tr><tr><td rowspan="2">Transperineal (3 DoFs)<br>(with V<sub>neighbor</sub>)</td><td>Frame2Frame ([31]) + SF ([31])</td><td rowspan="2"></td><td>11.09 ± 6.90</td></tr><tr><td>Proposed</td><td>2.27 ± 4.28</td></tr></tbody></table>

## 5.4 Longer trajectories and drift correction

This sub-section presents an analysis of the performance of the *context-based* method (leveraging $V_{neighbor}$ as spatial context volume) on longer trajectories, particularly with regard to the inherent drift associated with the accumulation of error during tracking.

The trajectories studied so far were relatively short: around 5 seconds assuming a 20 FPS (frames per second) display frequency. This navigation time is short, but coherent for an experienced clinician. However, considering a considerably longer navigation time of 25 seconds (or 500 slices at 20 FPS) is undeniably interesting from a clinical point of view, as the majority of urologists, especially beginners, need a longer
navigation time between two biopsy sites. In addition, we would point out that this 25-second time is generally an overestimate, and is more likely to be between 15 and 20 seconds.

We first considered longer trajectories, of 200 slices, and observed a passive correction in some trajectories as it is illustrated in Figure 13. By *passive* we mean that it was not an expected behavior. It seems that sometimes the estimated orientation was converging somehow to the GT orientation, instead of diverging. To encourage this unexpected but highly desirable behavior we slightly changed the targets for sub-model $f_2$. The targets were changed from $\text{sign}(\Delta\Theta_t) = \text{sign}(\Theta_t - \Theta_{t-1})$ (sign between GTs) to $\text{sign}(\Theta_t - \hat{\Theta}_{t-1})$ (sign between GT and previous estimate). The pretrained $f_1$ and $f_2$ sub-models were then jointly fine-tuned on the training sequences as before. The model trained with these targets will subsequently be referred to as *drift-corrected* model.

<figure><img src="image_16.png" /><figcaption>Figure 13: On these examples, we observe that despite a drift on the z axis, the estimation $\hat{\theta}_z$ seems to converge toward the GT orientation $\theta_z$.</figcaption></figure>

In the upper part of Table 5, the results obtained with the original model and the *drift-corrected* model are compared on the 84 short 100-slices trajectories presented in subsection 5.2. We observe that the *drift-corrected* model performs slightly better than the original model on short trajectories.

For longer trajectories, the difference in performance between the two models is more significant. We simulated 84 image sequences of 500 slices from the 42 test volumes (two trajectories/volume) to evaluate our models. This brought the total number of images in the test set to 42000. The performance of the models on the longer trajectories is shown in the lower part of the Table 5. Figure 14 presents the associated box plots showing the SRE across different slice indices for both models. We observe a significant improvement in performance with the *drift-corrected* model, in terms of final and average SRE, and ratio of indices with SRE below 2.5 mm and 5 mm. In addition, the box plots show that median and quartile values are much more stable in the *drift-corrected* model, suggesting that error accumulation is lower for the latter. For the *drift-corrected* model, the ratio of indices with a SRE below 2.5mm and 5 mm reaches 80 % and 90% respectively. In other words, 80% of the time during a long 25s trajectory, the SRE is below the established precision threshold of 2.5 mm.

<table><caption>Table 5: Performance of the original and *drift-corrected* models on short and long image sequences. Each model was evaluated on 84 trajectories simulated in 42 volumes using $V_{neighbor}$ as spatial context.</caption><thead><tr><th rowspan="2">Context-based models</th><th rowspan="2">Mean SRE (mm)</th><th rowspan="2">Final SRE (mm)</th><th colspan="3">MSE (degree)</th><th colspan="2">% indexes with SRE &lt; x</th></tr><tr><th>θ<sub>x</sub></th><th>θ<sub>y</sub></th><th>θ<sub>z</sub></th><th>x = 2.5 mm</th><th>x = 5 mm</th></tr></thead><tbody><tr><td colspan="8" style="text-align:center; font-weight:bold;">Short trajectories (100 slices)</td></tr><tr><td>Before registration</td><td>12.25 ± 6.03</td><td>13.00 ± 5.26</td><td>—</td><td>—</td><td>—</td><td>3.62 ± 5.06</td><td>11.35 ± 8.92</td></tr><tr><td>Original</td><td>2.27 ± 4.28</td><td>3.90 ± 4.28</td><td>0.27 ± 0.29</td><td>6.22 ± 13.70</td><td>41.51 ± 108.43</td><td>86.33 ± 23.10</td><td>93.05 ± 17.83</td></tr><tr><td>Drift-corrected</td><td>1.40 ± 2.17</td><td>2.19 ± 4.03</td><td>0.24 ± 0.24</td><td>3.57 ± 10.20</td><td>14.05 ± 52.65</td><td>91.72 ± 17.24</td><td>96.94 ± 9.68</td></tr><tr><td colspan="8" style="text-align:center; font-weight:bold;">Long trajectories (500 slices)</td></tr><tr><td>Before registration</td><td>12.19 ± 5.45</td><td>13.13 ± 5.62</td><td>—</td><td>—</td><td>—</td><td>1.42 ± 1.40</td><td>8.04 ± 4.14</td></tr><tr><td>Original</td><td>6.35 ± 11.74</td><td>12.86 ± 19.26</td><td>1.72 ± 2.59</td><td>31.44 ± 47.58</td><td>550.12 ± 1570.31</td><td>55.11 ± 26.28</td><td>77.96 ± 25.93</td></tr><tr><td>Drift-corrected</td><td>2.86 ± 5.74</td><td>4.50 ± 8.85</td><td>0.56 ± 0.87</td><td>16.53 ± 36.01</td><td>62.95 ± 157.03</td><td>79.45 ± 23.55</td><td>90.10 ± 16.88</td></tr></tbody></table>

# 6 Conclusion and perspectives

We proposed a method for tracking the orientation of an 2D US slice during a biopsy procedure. This orientation is calculated relative to the last 3D volume acquired during the procedure, and not absolutely in a fixed reference frame. This allows for more accurate localization as it takes into account possible movements of the patient or prostate during the procedure. Since determining the signs of relative orientation parameters is ambiguous due to the symmetry of the problem, we suggested using a 3D US volume from the procedure
<figure><img src="image_17.png" /><figcaption>(a) Complete figure.</figcaption></figure>

<figure><img src="image_18.png" /><figcaption>(b) Adapted SRE axis scale.</figcaption></figure>

<figure><img src="image_19.png" /><figcaption>(c) Complete figure.</figcaption></figure>

<figure><img src="image_20.png" /><figcaption>(d) Adapted SRE axis scale.</figcaption></figure>

Figure 14: Box plots showing the SRE at different slice indexes. The models were evaluated on 84 test trajectories of length 500 using $V_{neighbor}$ as spatial context. (a) and (b): proposed model. (c) and (d): proposed *drift-corrected* model. Zooming in the region of interest, we see that error accumulation is lower for the proposed *drift-corrected* model.

as *spatial context*. In the absence of ground-truth transformations that would localize an ultrasound slice in the last volume acquired, we proposed a method for simulating US streams by resampling slices in the biopsy volumes. To enhance clinical realism, we have chosen as *spatial context* a different volume from which the slices are extracted (source volume). We further hypothesized that a volume neighboring the source volume, i.e., acquired just before or just after during the procedure, best reproduces difficulty of the clinical scenario, but there are some limitations to this choice. Indeed, we do not know precisely how close the volumes are in terms of probe displacement, deformation, presence of a biopsy needle, etc. Furthermore, the alignment transformation between the source volume and its neighboring volume is obtained by composing two rigid transformations, which results in systematic registration errors. This misalignment between the two volumes is artificial, and although it impacts the performance of our method, it does not reflect completely the reality of the clinical scenario.

The proposed architecture is simple, with few trainable weights. We sincerely believe that a simpler model may offer more interpretability and may instill greater confidence, especially in the healthcare field. Moreover, fewer weights mean an easier deployment on routine clinical equipment, which is not always equipped with powerful GPUs, or sometimes none at all. In terms of computing time for inference, with our computer setup our method largely achieves the real-time performance required for clinical applications, with a frequency of 40 Hz.

The next immediate step of this work involves recording 2D US streams with ground-truth orientations, which could be achieved using an EM localizer. However, this approach would not constitute a true *gold standard* as the probe's orientation would be referenced to the fixed coordinate system of the operating room rather than directly relative to the prostate. We hypothesize that these recordings may contain physical information not yet simulated, which, if accurately captured by a future task-specific model, could enhance
tracking performance. In the longer term, integrating the three missing rigid transformation parameters (translations) and accounting for prostate deformations (with a DDF for instance) would be valuable advancements.

We believe that this work, although presented as part of the Koelis clinical pipeline, could have broader applications. It could be adapted to other biopsy assistance systems, which also work with 3D US volumes acquired during the procedure. Additionally, it could benefit other medical applications that seek to localize in real-time ultrasound slices in space. However, implementing this approach in a clinical setting comes with certain challenges. We have suggested that a lightweight deep learning model could facilitate its deployment on clinical routine equipment. Additionally, the design of an intuitive user interface, based on the proposed real-time slice localization will be an important aspect to consider to ensure the usability of a future navigation system. It should facilitate the urololgist's mental representation during navigation, thereby improving the efficiency of the procedure, both in terms of accuracy and procedure time.

## Acknowledgments

This work was partly supported by the French "Agence Nationale de la Recherche" Investissement d'Avenir program (grant PSPC n°9 DIANA, grant MIAI@Grenoble-Alpes under reference ANR-19-P3IA-0003 and CAMI Labex under reference ANR-11-LABX-0004).

## Declaration of generative AI and AI-assisted technologies in the writing process.

During the preparation of this work the author(s) used ChatGPT/OpenAI in order to correct the English. After using this tool/service, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the content of the published article.

## A Probe trajectory generating process

Trajectories of size less than or equal to 100 were generated according to the Table 6. Longer trajectories were split into several sub-trajectories of size 100 (+ remainder of size < 100).

<table><caption>Table 6: Simulation of probe trajectories ($\Theta_t$)$_t$ of length $\le$ 100. For each of the three angle coordinates, the sequence was divided into four stages whose order and size were randomly chosen. In addition, the sign of the rotation ($s_i$)$_i$ for periods that are not plateaus (small angles around 0°) was randomly chosen for each sequence. The table shows the uniform laws from which the three relative angles were derived for each stage.</caption><thead><tr><th>Stage</th><th>δθ<sub>x</sub></th><th>δθ<sub>y</sub></th><th>δθ<sub>z</sub></th></tr></thead><tbody><tr><td>1</td><td>s<sub>1</sub> * [0.25°, 0.75°]</td><td>s<sub>3</sub> * [0.25°, 0.75°]</td><td>s<sub>5</sub> * [0.5°, 2°]</td></tr><tr><td>2</td><td>[-0.1°, 0.1°]</td><td>[-0.1°, 0.1°]</td><td>s<sub>6</sub> * [0.5°, 2°]</td></tr><tr><td>3</td><td>s<sub>2</sub> * [0°, 1°]</td><td>s<sub>4</sub> * [0°, 1°]</td><td>[-0.5°, 0.5°]</td></tr><tr><td>4</td><td>[-0.1°, 0.1°]</td><td>[-0.1°, 0.1°]</td><td>s<sub>7</sub> * [0°, 3°]</td></tr></tbody></table>

## B Datasets for sub-models pretraining

The datasets created for the pretraining of the sub-models $f_0$, $f_1$ and $f_2$ are presented in Table 7 and Table 8 respectively.

## References

[1] Freddie Bray, Mathieu Laversanne, Hyuna Sung, Jacques Ferlay, Rebecca L. Siegel, Isabelle Soerjomataram, and Ahmedin Jemal. Global cancer statistics 2022: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. *CA: A Cancer Journal for Clinicians*, 74(3):229–263, 2024. _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.3322/caac.21834.
<table><caption>Table 7: Dataset generation for sub-model $f_0$ and $f_1$ pretraining. 120 triplets of consecutive images ($I_Θ$, $I_{Θ+ΔΘ_1}$, $I_{Θ+ΔΘ_1+ΔΘ_2}$) were sampled by volume. The table shows the uniform law intervals from which $Θ$, $ΔΘ_1$ and $ΔΘ_2$ are randomly drawn. For $f_0$ (resp. $f_1$), $I_S$ denotes the interval $[-0.5°, 0.5°]$ (resp. $[0°, 0.5°]$) and $I_L$ the interval $[-2°, 2°]$ (resp. $[0°, 2°]$).</caption><thead><tr><th></th><th>Θ</th><th>ΔΘ<sub>1</sub></th><th>ΔΘ<sub>2</sub></th></tr></thead><tbody><tr><td>Rotation around x</td><td>[-10°, 10°]</td><td rowspan="3">{I<sub>S</sub>, I<sub>L</sub>}<sup>3</sup></td></tr><tr><td>Rotation around y</td><td>[-10°, 10°]</td></tr><tr><td>Rotation around z</td><td>[-30°, 30°]</td></tr></tbody></table>

<table><caption>Table 8: Dataset generation for sub-model $f_2$ pretraining. 100 tuples of 10 slices were sampled per pair biopsy volume $V$ / associated context volume. The slices $I_Θ$ and $I_{Θ+ΔΘ}$ were sampled in $V$ while the eight slices ($J_Φ$) were sampled in the spatial context volume. The table gives the intervals of the uniform laws from which $Θ$ and $ΔΘ$ were randomly drawn.</caption><thead><tr><th></th><th>Θ</th><th>ΔΘ</th></tr></thead><tbody><tr><td>Rotation around x</td><td>[-10°, 10°]</td><td>[-2°, 2°]</td></tr><tr><td>Rotation around y</td><td>[-10°, 10°]</td><td>[-2°, 2°]</td></tr><tr><td>Rotation around z</td><td>[-30°, 30°]</td><td>[-4°, 4°]</td></tr></tbody></table>

[2] Nicholas D James, Ian Tannock, James N'Dow, Felix Feng, Silke Gillessen, Syed Adnan Ali, Blanca Trujillo, Bissan Al-Lazikani, Gerhardt Attard, Freddie Bray, Eva Compérat, Ros Eeles, Omolara Fatiregun, Emily Grist, Susan Halabi, Áine Haran, Daniel Herchenhorn, Michael S Hofman, Mohamed Jaloh, Stacy Loeb, Archie MacNair, Brandon Mahal, Larissa Mendes, Masood Moghul, Caroline Moore, Alicia Morgans, Michael Morris, Declan Murphy, Vedang Murthy, Paul L Nguyen, Anwar Padhani, Charles Parker, Hannah Rush, Mark Sculpher, Howard Soule, Matthew R Sydes, Derya Tilki, Nina Tunariu, Paul Villanti, and Li-Ping Xie. The Lancet Commission on prostate cancer: planning for the surge in cases. *The Lancet*, page S0140673624006512, April 2024.

[3] Louis Lenfant, Eric Barret, Morgan Rouprêt, Francois Rozet, Guillaume Ploussard, and Pierre Mozer. Transperineal Prostate Biopsy Is the New Black: What Are the Next Targets? *European Urology*, 82(1):3–5, July 2022.

[4] Ahmet Urkmez, Cihan Demirel, Muammer Altok, Tharakeswara K. Bathala, Daniel D. Shapiro, and John W. Davis. Freehand versus Grid-Based Transperineal Prostate Biopsy: A Comparison of Anatomical Region Yield and Complications. *The Journal of Urology*, October 2021. Publisher: Wolters KluwerPhiladelphia, PA.

[5] Sheng Xu, Jochen Kruecker, Baris Turkbey, Neil Glossop, Anurag K. Singh, Peter Choyke, Peter Pinto, and Bradford J. Wood. Real-time MRI-TRUS fusion for guidance of targeted prostate biopsies. *Computer Aided Surgery*, 13(5):255–264, January 2008.

[6] Jeffrey Bax, Derek Cool, Lori Gardi, Kerry Knight, David Smith, Jacques Montreuil, Shi Sherebrin, Cesare Romagnoli, and Aaron Fenster. Mechanically assisted 3D ultrasound guided prostate biopsy system. *Medical Physics*, 35(12):5397–5410, December 2008.

[7] Michael Baumann, Pierre Mozer, Vincent Daanen, and Jocelyne Troccaz. Prostate biopsy tracking with deformation estimation. *Medical Image Analysis*, 16(3):562–576, April 2012.

[8] G. Fiard, R. Mallet, L. Cormier, J.L. Descotes, J. Defontaines, P. Coloby, F. Kleinclauss, E. Lechevallier, J.L. Moreau, A. De La Taille, A. Ruffion, F. Rozet, and G. Fournier. Biopsies ciblées de la prostate par fusion échographie-IRM : quel dispositif choisir en 2018 ? *Progrès en Urologie - FMC*, 28(3):F66–F71, September 2018.

[9] Masatomo Kaneko, Dordaneh Sugano, Amir H. Lebastchi, Vinay Duddalwar, Jamal Nabhani, Christopher Haiman, Inderbir S. Gill, Giovanni E. Cacciamani, and Andre Luis Abreu. Techniques and Outcomes of MRI-TRUS Fusion Prostate Biopsy. *Current Urology Reports*, 22(4):27, March 2021.

[10] Enzo Ferrante and Nikos Paragios. Slice-to-volume medical image registration: A survey. *Medical Image Analysis*, 39:101–123, July 2017.

[11] Tharindu De Silva, Aaron Fenster, Derek W. Cool, Lori Gardi, Cesare Romagnoli, Jagath Samarabandu, and Aaron D. Ward. 2D-3D rigid registration to compensate for prostate
motion during 3D TRUS-guided biopsy. *Medical Physics*, 40(2):022904, 2013. eprint: https://aapm.onlinelibrary.wiley.com/doi/pdf/10.1118/1.4773873.

[12] Siavash Khallaghi, C. Antonio Sánchez, Saman Nouranian, Samira Sojoudi, Silvia Chang, Hamidreza Abdi, Lindsay Machan, Alison Harris, Peter Black, Martin Gleave, Larry Goldenberg, S. Sidney Fels, and Purang Abolmaesumi. A 2D-3D Registration Framework for Freehand TRUS-Guided Prostate Biopsy. In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro Frangi, editors, *Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015*, Lecture Notes in Computer Science, pages 272–279, Cham, 2015. Springer International Publishing.

[13] Derek J. Gillies, Lori Gardi, Tharindu De Silva, Shuang-Ren Zhao, and Aaron Fenster. Real-time registration of 3D to 2D ultrasound images for image-guided prostate biopsy. *Medical Physics*, 44(9):4708–4723, September 2017.

[14] Sonia-Yuki Selmi, Emmanuel Promayon, and Jocelyne Troccaz. Hybrid 2D–3D ultrasound registration for navigated prostate biopsy. *International Journal of Computer Assisted Radiology and Surgery*, 13(7):987–995, July 2018.

[15] Clément Beitone, Gaëlle Fiard, and Jocelyne Troccaz. Towards real-time free-hand biopsy navigation. *Medical Physics*, 48(7):3904–3915, 2021. eprint: https://aapm.onlinelibrary.wiley.com/doi/pdf/10.1002/mp.14582.

[16] Shihui Zhang, Shan Jiang, Zhiyong Yang, and Ranlu Liu. 2D Ultrasound and 3D MR Image Registration of the Prostate for Brachytherapy Surgical Navigation. *Medicine*, 94(40):e1643, October 2015.

[17] Grant Haskins, Uwe Kruger, and Pingkun Yan. Deep learning in medical image registration: a survey. *Machine Vision and Applications*, 31(1):8, January 2020.

[18] Raphael Prevost, Mehrdad Salehi, Simon Jagoda, Navneet Kumar, Julian Sprung, Alexander Ladikos, Robert Bauer, Oliver Zettinig, and Wolfgang Wein. 3D freehand ultrasound without external tracking using deep learning. *Medical Image Analysis*, 48:187–202, August 2018.

[19] Yanting Xie, Hongen Liao, Daoqiang Zhang, Lei Zhou, and Fang Chen. Image-Based 3D Ultrasound Reconstruction with Optical Flow via Pyramid Warping Network. In *2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)*, pages 3539–3542, Mexico, November 2021. IEEE.

[20] Thomas Leblanc, Florent Lalys, Quentin Tollenaere, Adrien Kaladji, Antoine Lucas, and Antoine Simon. Stretched reconstruction based on 2D freehand ultrasound for peripheral artery imaging. *International Journal of Computer Assisted Radiology and Surgery*, 17(7):1281–1288, July 2022.

[21] Pak-Hei Yeung, Moska Aliasi, Aris T. Papageorghiou, Monique Haak, Weidi Xie, and Ana I. L. Nam-burete. Learning to map 2D ultrasound images into 3D space with minimal human annotation. *Medical Image Analysis*, 70:101998, May 2021.

[22] Mingyuan Luo, Xin Yang, Hongzhang Wang, Liwei Du, and Dong Ni. Deep Motion Network for Free-hand 3D Ultrasound Reconstruction. In Linwei Wang, Qi Dou, P. Thomas Fletcher, Stefanie Speidel, and Shuo Li, editors, *Medical Image Computing and Computer Assisted Intervention – MICCAI 2022*, pages 290–299, Cham, 2022. Springer Nature Switzerland.

[23] Mahsa Mikaeili and Hasan Şakir Bilge. Trajectory estimation of ultrasound images based on convolutional neural network. *Biomedical Signal Processing and Control*, 78:103965, September 2022.

[24] Guochen Ning, Hanying Liang, Lei Zhou, Xinran Zhang, and Hongen Liao. Spatial Position Estimation Method for 3D Ultrasound Reconstruction Based on Hybrid Transfomers. In *2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI)*, pages 1–5, March 2022. ISSN: 1945-8452.

[25] Sidaty El hadramy, Juan Verde, Karl-Philippe Beaudet, Nicolas Padoy, and Stéphane Cotin. Trackerless Volume Reconstruction from Intraoperative Ultrasound Images. In Hayit Greenspan, Anant Madabhushi, Parvin Mousavi, Septimiu Salcudean, James Duncan, Tanveer Syeda-Mahmood, and Russell Taylor, editors, *Medical Image Computing and Computer Assisted Intervention – MICCAI 2023*, pages 303–312, Cham, 2023. Springer Nature Switzerland.

[26] Mingyuan Luo, Xin Yang, Hongzhang Wang, Haoran Dou, Xindi Hu, Yuhao Huang, Nishant Ravikumar, Songcheng Xu, Yuanji Zhang, Yi Xiong, Wufeng Xue, Alejandro F. Frangi, Dong Ni, and Litao Sun. RecON: Online learning for sensorless freehand 3D ultrasound reconstruction. *Medical Image Analysis*, 87:102810, July 2023.
[27] Qi Li, Ziyi Shen, Qian Li, Dean C Barratt, Thomas Dowrick, Matthew J Clarkson, Tom Vercauteren, and Yipeng Hu. Trackerless Freehand Ultrasound with Sequence Modelling and Auxiliary Transformation Over Past and Future Frames. *2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI)*, pages 1–5, April 2023. Conference Name: 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI) ISBN: 9781665473583 Place: Cartagena, Colombia Publisher: IEEE.

[28] Yuanwei Li, Bishesh Khanal, Benjamin Hou, Amir Alansary, Juan J. Cerrolaza, Matthew Sinclair, Jacqueline Matthew, Chandni Gupta, Caroline Knight, Bernhard Kainz, and Daniel Rueckert. Standard Plane Detection in 3D Fetal Ultrasound Using an Iterative Transformation Network. 11070:392–400, 2018. Book Title: Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 ISBN: 97830009274 97830009281 Place: Cham Publisher: Springer International Publishing.

[29] Chiara Di Vece, Brian Dromey, Francisco Vasconcelos, Anna L. David, Donald Peebles, and Danail Stoyanov. Deep learning-based plane pose regression in obstetric ultrasound. *International Journal of Computer Assisted Radiology and Surgery*, 17(5):833–839, May 2022.

[30] Hengtao Guo, Xuanang Xu, Sheng Xu, Bradford J. Wood, and Pingkun Yan. End-to-end Ultrasound Frame to Volume Registration. In Marleen de Bruijne, Philippe C. Cattin, Stéphane Cotin, Nicolas Padoy, Stefanie Speidel, Yefeng Zheng, and Caroline Essert, editors, *Medical Image Computing and Computer Assisted Intervention – MICCAI 2021*, pages 56–65, Cham, 2021. Springer International Publishing.

[31] Hengtao Guo, Xuanang Xu, Xinrui Song, Sheng Xu, Hanqing Chao, Joshua Myers, Baris Turkbey, Peter A. Pinto, Bradford J. Wood, and Pingkun Yan. Ultrasound Frame-to-Volume Registration via Deep Learning for Interventional Guidance. *IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control*, 70(9):1016–1025, September 2023.

[32] Tamara Dupuy, Clément Beitone, Jocelyne Troccaz, and Sandrine Voros. 2D/3D Deep Registration Along Trajectories With Spatiotemporal Context: Application to Prostate Biopsy Navigation. *IEEE Transactions on Biomedical Engineering*, 70(8):2338–2349, August 2023.

[33] Hengtao Guo, Hanqing Chao, Sheng Xu, Bradford J. Wood, Jing Wang, and Pingkun Yan. Ultrasound Volume Reconstruction From Freehand Scans Without Tracking. *IEEE Transactions on Biomedical Engineering*, 70(3):970–979, March 2023. Conference Name: IEEE Transactions on Biomedical Engineering.

[34] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In *2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 770–778, June 2016. ISSN: 1063-6919.

[35] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated Residual Transformations for Deep Neural Networks. In *2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 5987–5995, July 2017. ISSN: 1063-6919.

[36] Cécile Poquet Torterotot, Pierre Mozer, Michael Baumann, Marie-Aude Vitrani, and Guillaume Morel. Analysis of endorectal probe kinematics during prostate biopsies. page 47, 2010.

[37] Marie-Aude Vitrani, Michael Baumann, David Reversat, Guillaume Morel, Alexandre Moreau-Gaudry, and Pierre Mozer. Prostate biopsies assisted by comanipulated probe-holder: first in man. *International Journal of Computer Assisted Radiology and Surgery*, 11(6):1153–1161, June 2016.

[38] V. V. Karnik, A. Fenster, J. Bax, D. W. Cool, L. Gardi, I. Gyacskov, C. Romagnoli, and A. D. Ward. Assessment of image registration accuracy in three-dimensional transrectal ultrasound guided prostate biopsy. *Medical Physics*, 37(2):802–813, February 2010.

[39] Amber L. Simpson, Jessica Burgner, Courtenay L. Glisson, S. Duke Herrell, Burton Ma, Thomas S. Pfeiffer, Robert J. Webster III, and Michael I. Miga. Comparison Study of Intraoperative Surface Acquisition Methods for Surgical Navigation. *IEEE Transactions on Biomedical Engineering*, 60(4):1090–1099, April 2013. Conference Name: IEEE Transactions on Biomedical Engineering.