SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In practice the policy may be modelled as a neural network with parameters  $\theta$ , that predicts a distribution, from which to sample the action  $a_{t}$ . Practically, for continuous actions, the policy may be defined by two parametric functions (neural networks) with shared parameters, which specify a diagonal Gaussian distribution from which to sample the action; one function specifying the mean of the distribution  $\mu = \mu_{\theta}(s_t)$  and one specifying the standard deviation  $\sigma = \sigma_{\theta}(s_t)$ . The policy may then be given by  $\log \pi_{\theta}(a_t|s_t) = -\frac{1}{2}\left(\sum_{i=1}^{k}\left(\frac{(a_{t,i} - \mu_i)^2}{\sigma_i^2}\right) + k\log 2\pi\right)$ . However, for notational convenience in further analysis, we simply use  $\pi_{\theta}$  instead of modelling two separate networks that predict the parameters of the diagonal Gaussian distribution.
A cumulative reward over a trajectory may be used to compute optimal policy parameters  $\theta^{*}$ . The cumulative reward may be computed as a discounted sum of rewards over a trajectory, starting from time-step  $t$  and is given by:
$$
Q ^ {\pi_ {\theta}} \left(s _ {t}, a _ {t}\right) = \sum_ {k = 0} ^ {T} \gamma^ {k} R _ {t + k} \tag {2}
$$
where the discount factor  $\gamma$  discounts future rewards. An expectation of this cumulative reward may be denoted as the return:
$$
J (\theta) = \mathbb {E} _ {\pi_ {\theta}} \left[ Q ^ {\pi_ {\theta}} \left(s _ {t}, a _ {t}\right) \right] \tag {3}
$$
which may be computed over multiple trajectories.
The optimisation problem may then be summarised as:
$$
\theta^ {*} = \arg \max  _ {\theta} J (\theta), \tag {4}
$$
and the objective function is maximised using gradient ascent.
The training procedure to obtain an optimised policy  $\pi_{\theta^{*}}$  which maximises the cumulative reward, representative of navigation performance, is summarised in the Supplementary Materials (Appendix J "Reinforcement Learning Training Algorithm"). After training, this policy may be used to perform navigation intraoperatively.
5.2 Experiments and results
Data A total of 155 colon meshes were selected from the TotalSegmentator subset of the SARAMIS dataset. To define navigation targets for the RL experiment, a single colon was manually labelled for all anatomically relevant landmarks (namely the rectum, hepatic and splenic flexures, and the caecum). Subsequently, the other 154 colons were deformably registered to the manually labelled colon in order to obtain anatomically appropriate labels for these regions. Registration was performed using Coherent Point Drift [50] ( $\alpha = 1$ ,  $\beta = 10$ ) in order to propagate label annotations through the dataset. An analysis of registration accuracy is provided in Supp. Mat. Appendix H. All centerline labels were then mapped onto the procedural mesh using kd-Tree [4] search ( $n = 20$ ), given the sufficiently dense procedural meshes. This defined the navigation areas on the surface mesh for each patient. The subset was split into 91 meshes used for model training, 32 for model development and 32 meshes as a hold-out test set. Images for navigation were subsequently simulated with a Mitsuba3 renderer, with size  $200 \times 200$  pixels. Full hyperparameters are available in the provided implementation (Supp. Mat. Appendix A).
Figure 4: Summary of autonomous navigation experiment. Given a patient-derived mesh model of the colon, and defined navigation targets (A), the camera pose in the environment is used to render the synthetic view inside the colon. Using the rendering as the state  $s_t$ , a human observer or an RL agent may sample action  $a_t$  with the aim of reaching navigation structures (B). We report the success rates by sub-dataset on the hold-out test set in (C), showing good generalisation across unseen test sets.
Evaluating RL policy vs a human observer To evaluate the efficacy of the autonomous navigation, we compare the RL-learnt policy with the policy of a human observer ([S.U.S.], biomedical imaging researcher with 4 years of experience with medical imaging). The efficacy is evaluated by the number of steps taken to reach the target, across four patient volumes which were not encountered during training. The human observer policy was within the same RL environment where interaction with the environment was done by sampling actions, where the action space was limited and the step size for  $a_{t}$  was fixed (i.e., movement allowed in only orthogonal directions to the camera line of sight, controlled using arrow keys; and camera rotation controlled in the same orthogonal planes, controlled using 'W, A, S, D' keys). A visualisation of the rendered scenes from the environment are presented in Fig. 4. Example trajectories are generated in a representative navigation task from the rectum to the caecum in the Supp. Mat. Appendix. E. The average number of steps to find targets for RL and the human over 24 test cases were  $77.8 \pm 13.2$  and  $75.3 \pm 15.6$ , respectively. There was no statistically significant difference in RL vs human performance (p-value= 0.83).
Evaluating RL policy success rate for navigation We report the success rate of finding all four targets, across 594 different patient cases held out from training, with 100 randomised starting locations for each. For the random start locations experiment, a random location was picked as the starting point before navigating to the next structure. This was repeated 100 times for each patient case. For the sequential localisation task, the order in which to visit the 4 target locations was randomised 100 times per case. If the RL function failed to localise the structure, a random starting location was assigned for localising the next structure. Failure in the task is defined as the inability to navigate to the structure within 256 steps or collision with the colonic wall. Results are presented in Table 3, and performance split by sub-constituent dataset is reported in Fig. 4C. It should be noted that the lowest success rate is for the hepatic flexure and splenic flexure localisation tasks and the highest success rate is observed for the rectum localisation task. We observe a small performance decline from the TotalSegmentator datasets in comparison to the unseen datasets during training (Fig. 4 C).
Structure	Rectum	Hepatic Flexure	Splenic Flexure	Caecum	Overall
Random start locations	84.4	74.4	73.5	83.6	79.0
Sequential localisation	87.1	75.3	74.4	82.0	79.7
Table 3: Overall success rate (%) of navigating to a structure within the colon for 594 held-out subjects across the AMOS, Abdomen-1k and TotalSegmentator datasets.
The RL policy took approximately 7 days to train on a single Nvidia Tesla V100 GPU. During inference the model predictions coupled with an environment had a speed of approximately 20 iterations per second.
SARAMIS presents the first, large-scale dataset of patient-specific 3D rendering assets representing the major structures of the visceral anatomy. As reported in Table 1, in comparison to previous works, SARAMIS offers the following distinct advantages: 1) Scale: SARAMIS is over three orders of magnitude larger than any previous set of 3D rendering assets for MIS in the literature, 2) Heterogeneity: SARAMIS offers an order of magnitude larger number of anatomical targets than previous datasets, and 3) Patient variability: SARAMIS features a significantly larger number of subjects compared to previous datasets. Through multi-organ segmentation, the creation of new labelling for all assets, data curation, and camera path generation, SARAMIS enables simulation-based experimentation not possible from the underlying CT scans alone.
In addition to SARAMIS, we developed a Markov decision process environment for navigation within the colon using simulated intraoperative images derived from patient CTAP scans (i.e., not from dedicated CT colonography scans). The observed performance of the RL function and human observer in the colonoscopy navigation task across four patients was comparable, which indicates that an effective generalisable cross-patient navigation policy was learnt using our proposed training scheme. Furthermore, it is interesting to note the overall high  $(79.0\%)$  and  $79.7\%$  success rates of finding structures within the colon, within 256 steps, for the held out test set across a variety of randomised starting locations for the intraoperative imaging probe. The highest success rates were observed for the rectum and caecum, possibly due to their distinct appearances compared to the remainder of the colon, and tight curvature and blind-loop nature of the colonic flexures. While we model wall intersection constraints within our work, we do not account for all possible constraints in the endoscopic settings - for example, camera pose constraints such that the camera may not face directly opposite the direction of endoscope insertion from one step to the next, or extra-luminal boundaries imposed by surrounding visceral organs. Additionally, we qualitatively observe (Supp. Mat. Appendix E) that human trajectories are smoother than RL trajectories, which may arise from the lack of a smoothness prior or regularization on the generated actions. Accounting for these constraints represents a natural avenue of future research. Overall, whilst training was performed on a subset of the available assets, this indicates the robustness of the proposed training scheme which allows for the trained policy to be generalisable not only across patients but also across starting locations and the four target structures included during training.
A limitation of this work lies in the design of shading nodes for the procedural texturing of SARAMIS assets. Despite designing the anatomical textures under the supervision of a clinician with surgical training and by referencing 2D intra-operative images of different anatomy, it is likely that deviations from the proposed parameters in the associated shader nodes may result in non-clinically feasible renders. Whilst procedural texturing is still an industry standard in the computer-graphics community [17, 29, 19], this manual approach could be paired or replaced with learning-based texturing [25] in order to texture SARAMIS meshes in a data-driven way. Another important limitation of this work is that the dataset does not capture all structures of relevance for the full simulation of MIS/RAMIS scenes (e.g. ligaments, fatty tissue, and fluids), due to the limited resolution of CT scans, and its relatively poor detection of soft-tissue structures. This limitation presents a natural avenue for future research, which could facilitate using SARAMIS to simulate different autonomous navigation tasks within the human abdominal anatomy.
In this work we introduce SARAMIS: Simulation Assets for Robotic-Assisted and Minimally-Invasive Surgery, a large-scale dataset of 3D rendering assets composed of 3D meshes, textures and diffuse maps for over 104 human anatomical structures. We warmly invite the wider research community to use SARAMIS assets for vision tasks in RAMIS/MIS such as depth estimation, camera pose estimation, or pairing tetrahedral meshes with open-source deformation modelling environments [20, 24] to further develop surgical vision applications and autonomous navigation tasks in MIS/RAMIS research.
This work is supported by the Wellcome/EPSRC Centre for Interventional and Surgical Sciences [203145Z/16/Z]. NMB, AA, ET, AS, and SF are supported by the EPSRC-funded UCL Centre for Doctoral Training in Intelligent, Integrated Imaging in Healthcare (i4health) [EP/S021930/1]. AA is supported by an EPSRC Industrial Case grant [EP/W522077/1], and a Microsoft Research PhD Scholarship Wellcome Trust award [221915/Z/20/]. MJC, YH, NMB, and SUS are supported by [EP/T029404/1]. TD is supported by [EP/V052438/1]. ZMCB is supported by the Natural Sciences and Engineering Research Council of Canada Postgraduate Scholarships-Doctoral Program, and the University College London Overseas and Graduate Research Scholarships. This work is also supported by the International Alliance for Cancer Early Detection, an alliance between Cancer Research UK [C28070/A30912, C73666/A31378], Canary Center at Stanford University, the University of Cambridge, OHSU Knight Cancer Institute, University College London and the University of Manchester.
[1] M. Allan, S. Thompson, M. J. Clarkson, S. Ourselin, D. J. Hawkes, J. Kelly, and D. Stoyanov. 2d-3d pose tracking of rigid instruments in minimally invasive surgery. In Information Processing in Computer-Assisted Interventions: 5th International Conference, IPCAI 2014, Fukuoka, Japan, June 28, 2014. Proceedings 5, pages 1–10. Springer, 2014.
[2] J. Allard, S. Cotin, F. Faure, P.-J. Bensoussan, F. Poyer, C. Duriez, H. Delingette, and L. Grisoni. Sofa-an open source framework for medical simulation. In MMVR 15-Medicine Meets Virtual Reality, volume 125, pages 13-18. IOP Press, 2007.
[3] R. Antonova, P. Shi, H. Yin, Z. Weng, and D. K. Jensfelt. Dynamic environments with deformable objects. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[4] J. L. Bentley. K-d trees for semidynamic point sets. In Proceedings of the sixth annual symposium on Computational geometry, pages 187-197, 1990.
[5] J. Bernal, F. J. Sánchez, G. Fernández-Esparrach, D. Gil, C. Rodríguez, and F. Vilarino. Wmdova maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Computerized medical imaging and graphics, 43:99-111, 2015.
[6] H. Borgli, V. Thambawita, P. H. Smedsrud, S. Hicks, D. Jha, S. L. Eskeland, K. R. Randel, K. Pogorelov, M. Lux, D. T. D. Nguyen, et al. Hyperkvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy. Scientific data, 7(1):283, 2020.
[7] N. Bourdel, T. Collins, D. Pizarro, C. Debize, A.-s. Gremeau, A. Bartoli, and M. Canis. Use of augmented reality in laparoscopic gynecology to visualize myomas. Fertility and sterility, 107(3):737-739, 2017.
[8] A. Buia, F. Stockhausen, and E. Hanisch. Laparoscopic surgery: A qualified systematic review. World journal of methodology, 5(4):238, 2015.
[9] M. Carstens, F. M. Rinner, S. Bodenstedt, A. C. Jenke, J. Weitz, M. Distler, S. Speidel, and F. R. Kolbinger. The dresden surgical anatomy dataset for abdominal organ segmentation in surgical data science. Scientific Data, 10(1):1-8, 2023.