SARAMIS: Simulation Assets for Robotic Assisted and Minimally Invasive Surgery
Authors: Nina Montana-Brown
Shaheer U. Saeed
Ahmed Abdulaal
Thomas Dowrick
Yakup Kilic
Sophie Wilkinson
Jack Gao
Meghavi Mashar
Chloe He
Alkisti Stavropoulou
Emma L Thomson
Zachary Baum
Simone Foti
Brian Davidson
Yipeng Hu
Matthew John Clarkson
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Minimally-invasive surgery
simulation
reinforcement learning
computer vision
synthetic data

Abstract: Minimally-invasive surgery (MIS) and robot-assisted minimally invasive (RAMIS) surgery offer well-documented benefits to patients such as reduced post-operative pain and shorter hospital stays.
However, the automation of MIS and RAMIS through the use of AI has been slow due to difficulties in data acquisition and curation, partially caused by the ethical considerations of training, testing and deploying AI models in medical environments.
We introduce \texttt{SARAMIS}, the first large-scale dataset of anatomically derived 3D rendering assets of the human abdominal anatomy.
Using previously existing, open-source CT datasets of the human anatomy, we derive novel 3D meshes, tetrahedral volumes, textures and diffuse maps for over 104 different anatomical targets in the human body, representing the largest, open-source dataset of 3D rendering assets for synthetic simulation of vision tasks in MIS+RAMIS, increasing the availability of openly available 3D meshes in the literature by three orders of magnitude.
We supplement our dataset with a series of GPU-enabled rendering environments, which can be used to generate datasets for realistic MIS/RAMIS tasks.
Finally, we present an example of the use of \texttt{SARAMIS} assets for an autonomous navigation task in colonoscopy from CT abdomen-pelvis scans for the first time in the literature.
\texttt{SARAMIS} is publically made available at https://github.com/NMontanaBrown/saramis/, with assets released under a CC-BY-NC-SA license.

SARAMIS: Simulation Assets for Robotic Assisted and Minimally Invasive Surgery
Nina Banttaia-Brown....
Shaheer U. Saeed 1.2
Ahmed Abdulaal Thomas Dowrick2
Yakup Kilic*
Sophie Wilkinson
Jack Gao
Meghavi Mashar
Alkisti Stavropoulou
Emma L. Thomson'
Zachary MC Baum
Chloe Hel.2
Simone Foti
Brian Davidson Yipeng Hu1.2
Matthew J Clarkson
Centre for Medical Image Computing, UCL., London, United Kingdom
Welcome/EPSRC Centre for Interventional And Surgical Sciences, London, United Kingdom
3 University College London Hospitals
n. moottanabrowosss. ucl. ac .u
Abstract
Minimally-invasive surgery (MIS) and robot-assisted minimally invasive (RAMIS) surgery offer well-documented benefits to patients such as reduced post-operative pain and shorter hospital stays. However, the automation of MIS and RAMIS through the use of AI has been slow due to difficulties in data acquisition and curation, partially caused by the ethical considerations of training, testing and deploying AI models in medical environments. We introduce SARAMIS, the first large-scale dataset of anatomically derived 3D rendering assets the human ab- dominal anatomy. Using previously existing. open-source CT datasets of the human anatomy, we derive novel 3D meshes, tetrahedral volumes, textures and diffuse maps for over 104 different anatomical targets in the human body, representing the largest, open- open-source dataset of 3D rendering assets for synthetic simulation of vision tasks in MIS+RAMIS, increasing the availability of openly available 3D meshes in the literature by three orders of magnitude. We supplement our dataset with series of GPU-enabled rendering environments, which can be used generate datasets for realistic MIS/RAMIS tasks. Finally, we present an example the use of SARAMIS assets for an autonomous navigation task in colonoscopy from CT abdomen-pelvis scans for the first time in the literature. SARAMIS is publically made available at htttt::///// /tttps//ggitubb com/ /omn/Noonteen WWootannaBoown saramis/ with as- sets released under a CC-BY-NC-SA license.
Introduction
Laparoscopy and endoscopy techniques in surgical and medical practice which involve inserting video cameras into a patient in order to diagnose and treat a number of conditions, and have made possible to perform minimally invasive surgery (MIS). These techniques obviate the need for large incisions at the operative site, replacing them with small incisions into which cameras and tools are inserted to perform the intervention. The benefits of MIS have been well documented g
49 [Z]], and can be summarised as follows: 1) Reduced post- post-operative pain, 2) Shortened hospital stays 71 47 3) Improved rates of patient recovery [12], and 4) Lowered costs to hospital systems in number of interventions [8 66, 47 (2). Additionally, recent advances in robotics have enabled the pairing of robotic elements with laparoscopic equipment, which provides further benefits such as an improved ergonomic environment for surgeons (76) and the possibility of teleoperation [II] In tandem, (partially) autonomous robotic surgery has emerged as an increasingly important research
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

topic [10 62 26 Indeed. many surgeons consider the full automation of robot-assisted minimally invasive surgery (RAMIS) the 'end goal' of surgical practice [26].
Although there have been advances in technologies to facilitate both MIS and RAMIS 58 -Such
image overlay [64 37 77 7 3D localisation of tools and cameras relative pre-op
[1- the data collection and validation of such solutions has been limited by the equipment required for validation. This can include, for example, optical trackers, stereo cameras, and/or LIDAR-like sensors which are non-standard surgical objects that interrupt surgical workflow and are expensive to accrue and implement [27 [0].
Whilst traditional computer vision applications have long exploited such devices to create laage-scale annotated datasets for relevant tasks such camera- pose estimation reconstccntt [3] 42
the aforementioned difficulties regarding surgical logistics (e.g. sterilisation of all objects theatre requiring repeated calibration, time-sensitivity of surgical environments, and overhead equipment cost) have resulted limited datasets for these tasks in MIS/RAMIS, In parallel, synthetic data and rendering environments have emerged promising, alternative resources enable computer vision at scale [53, 688, and are important for the development and testing of safe autonomous systems. However, silico datasets for the development of deep learning algorithms and autonomous systems in MIS/RAMIS are limited in number and application [38].
In this paper, we introduce Simulation Assets for Robotic Assisted and Minimally Invasive Surgery (SARAMIS), the first large-scale, multi-organ, open-source collection of rendering assets for the simulation of robotic and minimally invasive surgery. We summarise the contributions of this work as follows:
We provide the first, large-scale database datieee-datt-derrieed rendering assets represent- ing anatomical organs, textures, and tetrahedral meshes for the simulation of abdominal minimally invasive interventions.
We integrate SARAMIS with existing open-source environments for the procedural simulation of endoscopic and laparoscopic procedures, including simulation of depth maps, stereo and monocular cameras.
We develop Markov decision process environment for navigation within the colon, ing the above-described simulations, and subsequently use this environment to train an autonomous reinforcement learning (RL) function which learns to navigate four different structures within the colon and is generalisable to different patient cases; we open-source this environment for further research and development.
blender
Openn- sour-saurce randerars
104+ classes
(A)
(B)
(C)
Figure I: Summary SARAMIS pipeline. We annotate a large-scale dataset of open open-source CT scans (A), remesh and convert them into simulation files (triangular and tetrahedral meshes, normal maps, diffuse maps) (B) that can be used with a open-source renderers (C) to produce synthetic training data for MIS applications.
2 Related Work
Open Source MIS Datasets In contrast to tasks such 3D medical image segmentation 45 35 the number of freely available, annotated datasets focussed on MIS tasks small [10]. Most available datascts focus on 2D segmentation from intra-operative images for tasks such organ 9 30

---

pathology [3 41, and tool [6] segmentation, as well as action recognition [4]. Datasets to validate steps in MIS pipelines which are critical to workflow automation, such as camera pose estimation, 3D-to-3D registration, and organ deformation, are comparatively limited. The lack of varied datasets can attributed to the comparatively high cost of label acquisition and cleaning 46 [3), which involves the introduction of (previously discussed) non-standard equipment into the surgical workflow 33]. Furthermore, unlike traditional computer vision applications, deformable object modelling is a prerequisite to achieving clinically relevant accuracies [59] 70 63 [4]. Whilst animal models [48] may be used to validate algorithms through the use of irradiating scans, few patient open-source datasets to validate deformation models exist in the literature [59 36].
Synthetic datasets for applications related to MIS represent an alternative approach. with promising results in terms of simulation-tor to-rlall transfer for deformation simulation [55] [54] 63 [57] 77, 67, 69]
segmentation [66 15 and depth estimation [60]
72 However, current work either does not release
3D assets to simulate or manipulate scenes 154
or uses non-open-source frameworks [60] 16 and
can limited in terms of application 660
16
70 69.. Existing work is further limited in the
number of anatomical variations of 3D assets due to the use of small cohort to produce the datasets
66 36 [6 69]. We summarise the existing literature of 3D assets of simulation of MIS+RAMIS
tasks in Table
Dataset
# 3D Assets
# Organs
Subjects
Open-Source?
CV3D [72]
1 (Colon)
Dowrick et al.
16
1
(Liver)
Tagliabue et al.
70
1
1 (Tissue Retraction) (Liver)
1
Suwelack al.
669
1
DEPOLL [59]
2
1 (Porcine Liver)
2
Dowrick et al.
1
(Liver, Colon 18
OpenHELP 336
18
1
SimCol [60]
1
1 (Colon) (Liver)
IRCAD 3D Liver Dataset 67
20
20
SARAMIS
114,838
106
2527
Table I: Summary of existing 3D datasets for simulation of MIS+RAMIS tasks in the literature as compared to SARAMIS
Simulation Environments for MIS Rendering frameworks are abundant in the computer vision 34] and there are a number that support physics-basee basee multi-object rigid-body 24 Whilst there has been an interest in deformable-objec interactions in robotics, and MIS tasks [55 [54), the majority of these works are not open-
literature 4
20
interactions 3
20
computer vision
43
source (70] or have limited support for realistic soft-body interactions [73). Frameworks that use finite-element modelling, required for realism and accuracy in MIS/RAMIS, are limited 770 21
Several advances have been made towards task-level automation in the field of RAMIS 28 with reported success in tasks such as path planning suturing of various structures 62 40,, and tissue retraction in an ex vivo environment 663 However, there exist significant ongoing ethical questions surrounding the regulation and deployment of autonomous surgical systems [51). This is especially the case for more complex tasks such as navigation or full surgical automation. Many studies evaluate tasks in phantom models [62) animal models (62), or a limited number of synthetic patient-specine models [63] Synthetic assessment environments are promising, but much like the MIS simulation environment literature, these suffer from very small patient cohorts to generate the datasets (38) and thus a limited representation of anatomical variance.
Autonomy in RAMIS
SARAMIS tackles a number these issues in important ways. It provides one of the largest dataset of heterogenous patient-derived meshes to date, with a total of 1 1160018 meshes from 2529 patient models over more than 104 different anatomical structures. Additionally, SARAMIS may be paired with commonly-used rendering environments to sample monocular video with different camera intrinsics, depth maps, pose labels, optical flow, and segmentation maps, such as Blender- Mitsuba3 34 we provide examples of interfacing the SARAMIS assets with Mitsuba3, Kubric, and PyBullet, which can leverage GPU simulation of finite-element modelling or particle-based dynamics deformation simulation, for RL or otherwise.
Kubric (24), or
3

---

3 Dataset Generation
Data Collection and Annotation
Three open-source, anonymised, medical image datasets of patient CT scans [4] 35. 75 were selected for analysis. A human-in-the-loop semi-automatic data
annotation strategy (Fig. [ Panel A) was used to generate 3D rendering assets from patient-specific CT
seans using 3DSlicer [56], Py PyTorhh [52], and Meshlab [33. Initially, CT scans were automatically
segmented with ToganSegmentator [75 which is composed of several nnU-Nets [32] trained to detect 104 anatomical labels from CT scans. Given that the TotalSegmentator dataset contains 3D segmentations for all anatomical organs of interest verified by radiologist, only the Abdomen- Ik [45] and AMOS [55] datasets are processed using the segmentation pipeline. The generated labels strategy involving seven trained anatomical annotators
were assessed with collaborative-iteraaie and four radiologists. Initially, all annotations were inspected by the anatomical annotators under the supervision of clinician. The following criteria were adopted to flag cases in need of further review: 1) Verify class homogeneity within an anatomical structure, 2) Flag topological errors (e.g.. slices missing, holes within an anatomical structure), 3) Flag under- or over-segmentatton, and 4) Flag potential pathology. Additionally, annotators were instructed to log the type of CT scan from the data [full-body (FBCT), chest-abdomen-peliie (ACT)). The full annotation protocol, including training practice for the junior annotators, is made available in the Supp. Mat. (Appendix B Datasheet for Datasets). Subsequently, cases that were flagged for potential errors were individually reviewed and manually corrected under the supervision of a clinician. Finally, a subset of 450 of the verified CT scans were reviewed by four radiologists, instructed to verify the correctness of the segmentations and note any other relevant pathology or errors.
(CTCAP), abdomen-pelvis (CTAP), abdominal
Colon Mesh Generation Human bowel segmentation in CT is a challenging task due to a com- bination of tortuous anatomy and inconsistent contrast (itself due to the air-fluid interfaces in the colon), which can result in an incomplete tubular segmentation in non-contrast enhanced CT scans. Therefore, in order to generate more realistic and continuous mesh models that are tubular, a pro- cedural generation approach was considered instead [15]. All colon models from FBCT, CTCAP, and CTAP scans were manually inspected in 3DSlicer to detect the presence of the rectum, hepatic and splenic flexures, and the caecum. Models were then categorised as complete segmentations (all landmarks detected and a full segmentation obtained), partial segmentations (all landmarks are detected, but may be disconnected in regions), or erroncous segmentations (2 of the previous landmarks missing). Complete and partial segmentations were then manually processed with the Vascular Modelling Toolkit (VMTK) B33 in order to extract the colon centerlines. This was achieved by the manual placement of landmarks at the beginning and end of distinct regions (e.g.. the beginning and
of the rectum) in order extract line segments deseribing the tubular structure of that region. A matching algorithm was used to find a continuous line segment describing the entire colon by defining a start point on each colon, the subsequent closest segments are matched one by one (Appendix A, Supp. Mat.). A BSpline curve was then fit to the data points for each colon to obtain smooth representation of the centerline and resampled to 1000 points each. We provide generic interfaces from which the extracted curves can be converted into mesh representations of colons. We consider the colon topologically as a closed tube by extruding the mesh along the centerline with varying radius parameters, we can obtain a patient-derived representation of the colon. Full procedural simulation parameters are summarised the Supplementary Materials, and provided open-source for future research.
Mesh Generation All anatomical segmentations bar the colon were automatically extracted and converted into 3D mesh (Fig. [ Panel B) models using marching cubes algorithm [44]. Given the voxelised nature of CT scans with varying resolution between 0.5-5mm, the resulting meshes were prst-proeesed using Laplacian smoothing [18]. Additionally, meshes are mean-centered in their patient frame-of- reffreenceeeeee as well as their local frame-of-reference. Finally, patient frame-of- reference meshes are converted into tetrahedral volumes using lTTeWild 31
Texture and normal mapping We procedurally generate bone, bowel, soft abdominal organ, and muscle normal and diffuse maps using Blender. Based on open-source images of the aforementioned textures, we create Shader nodes (Supp. Mat. Appendix D) in Blender using Principled BSDF nodes to replicate the visual appearance of the structures. Subsequently, each mesh it's corresponding

---

Table 2: Summary of CT data of three datasets from which SARAMIS is derived. FBCT = Full Body CT, CTCAP = chest-abdoomnn-elliss CT, CTAP = abdomen-pelvis CT, ACT Abdomen CT. Other refers to alternative CT scans, as described in the datasheet for 75
Type of CT Scan
Dataset
Initial
FBCT
CTCAP
CTAP
ACT
Other
Included
No changes
Abdomen-1k
1063
10
366
71
592
0
1048
526
Amos
600
220
321
140
TotalSegmentator
1200
169
197
110
0
724
1200
1200
SARAMIS
2863
179
635
401
592
724
2527
1866
normal maps and diffuse characteristics are baked in 2k resolution, Example renders can be visualised in Fig 2 B).
Procedural Colon
Synthetic Renders
Blander Renders of SARAMIS
(A)
Real Images
(B)
Figure 2: Textured and shaded assets from SARAMIS. In A), we render a procedurally generated colon, with examples of synthetic renders of the colon, well reference real images from the HyperKvasir dataset [6]. We showcase other assets from SARAMIS in B), namely the liver (top left), vertebrae (top right), pancreas (bottom left), and muscle (bottom right).
4 SARAMIS
4.1 3D Dataset Generation
Overall, SARAMIS consists of a total of 14,838 meshes, textures, and normal map tuples that derived from total of 2527 patient scans. From the initial 2863 scans, total of 336 were excluded from segmentation analysis for the following reasons: 194 due to lack of availability of test set label, 15 due significant pathology making organ differentiation difficult, 13 due to the presence of fluid in the abdomen (e.g. haemoperitoneum or ascites) occluding organs of interest, 100 due to alternative imaging modality (MRI). 2 due metallic artefacts the scan, 1 due poor quality scan, and due original file corruption leading to lack of a segmentation file. In total, 15, 321, and scans were excluded from the Abdomen- 1 AMOS, and TotalSegmentator datasets, respectively. Overall, this results in the inclusion of 1048, 279, 1200 scans from the Abdomen- 1k, AMOS, and TotalSegmentator datasets in the SARAMIS dataset. The average voxel resolution is [0. 10..7x0..77x3.28 + [x.13x0. 14x 1.14xxx.x66 1.44mmmmm Jmm (mean voxel size for Abdomen- AMOS [0. 10.70x0.70% 70x4.23Jmm each, respectively).
and TotalSegmennatoo [0.8 1x0.8 1x2.70J, [0. 70x0000xx4..... The data split is documented in Table and the dataset split by mesh type is documented in Fig The resulting dataset and full compilation, processing, and texturing instructions are provided the SARAMIS Datasheet for Datasets (Datasheet for Datasets, Supp. Mat.).
total, 637 (48 %) of reviewed seans required correction. In addition, 78 scans (17% of the meta- reviewed subset) were flagged by the radiologists and required additional corrections. A total of 343,495,44 425 voxels were edited, where 18,52 18,226,556 voxels were corrected the AMOS dataset, and 224.968.869 voxels were corrected in the Abdomen- 1k dataset. Where corrections were necessary,

---

AMOS
Number of mesites
dbdomen SoralSerreeettee
500
Our
Anatomical Stracture
Figure 3: Number of meshes per organ in SARAMIS, split by constituent datasets.
median 27,924 [IQR=7239, 98857] voxels were corrected per scan, with average 12.6 [IQR=3.0, 15.0] structures corrected per scan. The most commonly corrected structure was the liver (315 instances corrected), with the least corrected structures being vertebrae CI, C2, and C3, and the brain (1 correction each). In addition, two previously unseen labels were corrected in the dataset: L6 and T13, denoting transitional vertebrae (which can result e.g. from congenital spinal deformity, resulting in additional or fewer lumbar or thoracic vertebra). A total of 48 scans were flagged as having transitional vertebra, with a total of 8 L6 segments, 26 T13 segments, and 14 as sacralised L5. A full description of organs changed per dataset is supplied in the Supp. Mat. Appendix I. Additional analyses of mesh density, surface area, and vertex are provided in Supp. Mat. Appendix G.
Autonomous Navigation with Colonoscopy
The SARAMIS dataset provides reference set of data simulate intraoperative navigation tasks. These simulations may then be used to train autonomous agents to navigate within the anatomies of interest. One such example explored in this work, specifically in the application of colonoscopy, is detailed in the following subsections.
5.1 Methods
In this work, the navigation task in colonoscopy is formulated as a sequential decision problem modelled by finite horizon partially observable Markov Decision Process (MDP). The decision policy learnt using RL. as described in the following sections. The navigation is performed based on an image acquired from camera inside the colon, where the task is for the camera navigate to a desired target which visible from the camera pose.
5.1.1 The Markov decision process environment
The MDP environment for RL modelled tuple (8,A,,,,,,,7)))
States
Here, S is the state-space from which a state at time-step / may be sampled E S. In our formulation 8 an image acquired from a synthetic camera. In this work, the image is simulated using Mitsuba3 [34], using the previously generated textures and diffuse maps.
Actions The pose of the camera is defined as et E I and change in the pose is defined as the action 0, A E IR. where A may be denoted the continuous action space, such that a, is the sampled action at time-step I. 'he updated pose may then be defined as C++1 = c + ar, which is the pose which the new camera image 8t+1 is rendered. The state transition distribution conditioned on state-action pairs is given by p S x S x A + [0, 1] where o(st+llst, ax) denotes the probability of the next state $1+1 S given the current state s, S and action a, E pair.
S A - R produces reward at time-step / denoted by R2 = r(st, (:) given the current state 8, and action a, pair. our formulation the reward is of two parts: 1) rasat()) which is the inverse of the distance between the camera position defined by ca
Rewards The reward function
formed

---

and the target (clipped to prevent finding only the target centre); 2) N'smagel which tests conditions with the help of the image plane position and camera position, follows:
[1
if target not in image 8t
Timage (5t: ar)
=
+10
(and terminate episode) if target in image 5t
(I)
10 (and terminate episode) if camera intersects with wall
where the target detection in the image is done by checking the intersection of the camera line of sight with coordinates of the target structure (a sphere placed in the region of interest), both computed based on and additional preset camera parameters. The wall intersection of the camera is computed using er and a tolerance from centre-line coordinates of the colon. The final reward function r,
then given by R; r(s,, at) Témage (st; ae) + (St at). This reward is scaled in order to balance the constituent rewards, with further details found within the implementation (Supp. Mat.). The episode termination with high reward values triggered by the 'target in image s;" and 'camera intersects with wall' conditions prevents undesirable solutions e.g., navigating to structures through walls or hovering around a target to maximise the iettance-bssed reward.
5.1.2 The policy
The policy ((ae(s)): S A [0, 1] denotes the probability of performing action at given state 8,. An action may then be sampled using ~ (-)).
Following the state transition distribution and the policy æ, for sampling next states and current actions, respectively, together with the reward function r, we can generate a trajectory of collected states, actions and corresponding rewards for multiple time-steps ($1, 41, R1. $T ar, Rry)
If we consider the policy тyy to be parameterised by policy parameters 0 then our aim is find the optimal parameters 0² such that if you follow Tp,, the accumulated reward r is maximised.
In practice the policy may be modelled distribution, from which to sample the action ar- Practically, for continuous actions, the policy may be defined by two parametric functions (neural networks) with shared parameters, which specify diagonal Gaussian distribution from which to sample the action; one function specifying the mean of the distribution μ (s:) and one specifying the standard deviation
neural network with parameters 0, that predicts
ag (s:). The policy may
- (E (un::=1)² + k log 2xr However, for notational convenience in further analysis, we simply use instead modelling two separate networks that predict the parameters of the diagonal Gaussian distribution.
then be given by log (or(ss:)
of
cumulative reward over a trajectory may be used to compute optimal policy parameters 0*, The cumulative reward may be computed as discounted sum of rewards over trajectory, starting from time-step t and is given by:
(st, ar)
Σ
7 R
where the discount factor y discounts future rewards. An expectation of this cumulative reward may be denoted as the return:
((0) Em, (sc,
(3)
which may be computed over multiple trajectories.
The optimisation problem may then be summarised as:
6 arg max (0),
(4)
and the objective function is maximised using gradient ascent.
The training procedure to obtain an optimised policy тy- which maximises the cumulative reward, representative of navigation performance, is summarised in the Supplementary Materials (Appendix J "Reinforcemeen Learning Training Algorithm").. After training, this policy may be used to perform navigation intraoperatively.
7

---

5.2 Experiments and results
Data A total of 155 colon meshes were selected from the TotalSegmentator subset of the SARAMIS dataset. To define navigation targets for the RL experiment, single colon was manually labelled for all anatomically relevant landmarks (namely the rectum, hepatic and splenic flextures, and the caecum). Subsequently, the other 154 colons were deformably registered to the manually labelled colon in order to obtain anatomically appropriate labels for these regions. Registration was performed using Coherent Point Drift [50] (a = 1. 10) in order to propagate label annotations through the dataset. An analysis of registration accuracy is provided in Supp. Mat. Appendix H. All centerline labels were then mapped onto the procedural mesh using kd-Tree 4 search (n=20), given the sufficiently dense procedural meshes. This defined the navigation areas on the surface mesh for each patient. The subset was split into 91 meshes used for model training, 32 for model development and 32 meshes as a hold-out test set. Images for navigation were subsequently simulated with a Mitsuba3 renderer, with size 200x200 pixels. Full hyperparameters are available in the provided implementation (Supp. Mat. Appendix A).
(A)
(B)
Splenic Flexure
Hepatic Flexure
Reinforcement Jesmming or human observer
tel
Caecum
c t Oett
Rectum
Fcrl
Figure 4: Summary of autonomous navigation experiment. Given a patient-derived mesh model of the colon, and defined navigation targets (A), the camera pose in the environment is used to render the synthetic view inside the colon. Using the rendering the state 8t, a human observer or an RL agent may sample action with the aim of reaching navigation structures (B). We report the success rates by sub-dataset on the hold-out test set in (C), showing good generalisation accross unseen test sets.
Evaluating RL policy a human observer To evaluate the efficacy of the autonomous navigation, we compare the RL-learnt policy with the policy of a human observer (SS.U.S.). biomedical imaging researcher with 4 years of experience with medical imaging). The efficacy is evaluated by the number of steps taken to reach the target, across four patient volumes which were not encountered during training. The human observer policy was within the same RL environment where interaction with the environment was done by sampling actions, where the action space was limited and the step size for a was fixed (i.e., movement allowed in only orthogonal directions to the camera line of sight, controlled using arrow keys; and camera rotation controlled in the same orthogonal planes, controlled using 'W, A. S, D' keys). A visualisation of the rendered scenes from the environment are presented in Fig. 4 Example trajectories are generated in representative navigation task from the rectum to the caecum in the Supp. Mat. Appendix. E. The average number of steps to find targets for RL and the human over 24 test cases were 77.8 + 13.2 and 75.3 + 15.6, respectively. There was no statistically significant difference in RL vs human performance (p-value= 0.83).
Evaluating RL policy success rate for navigation We report the success rate of finding all four targets, across 594 different patient cases held out from training, with 100 randomised starting locations for each. For the random start locations experiment, a random location was picked as the starting point before navigating to the next structure. This was repeated 100 times for each patient case. For the sequential localisation task, the order in which to visit the 4 target locations was randomised 100 times per case. If the RL function failed localise the structure, random starting location was assigned for localising the next structure. Failure in the task is defined as the inability to navigate to the structure within 256 steps or collision with the colonic wall. Results are presented in B and performance split by sub-constituent dataset is reported in Fig. It should be noted that the lowest success rate is for the hepatic flexure and splenic flexure localisation tasks and the highest success rate is observed for the rectum localisation task. We observe small performance decline from the TotalSegmentator datasets in comparison to the unseen datasets during training (Fig. C).
8

---

Structure
Rectum
Hepatic Flexure
Splenic Flexure
Caecum
Overall
Random start locations
84.4
74.4
73.5
83.6
79.0
Sequential localisation
87.1
75.3
74.4
82.0
79.7
Table 3: Overall success rate (%) of navigating to structure within the colon for 594 held-out
subjects across the AMOS, Abdomen- Ik and TotalSegmentator datasets.
The RL policy took approximately 7 days to train on single Nvidia Tesla V100 GPU. During inference the model predictions coupled with an environment had a speed of approximately 20 iterations per second.
Discussion
SARAMIS presents the first, large-scale dataset of patient-specific 3D rendering assets representing the major structures of the visceral anatomy. As reported in Table [ in comparison to previous works, SARAMIS offers the following distinct advantages: 1) Scale: SARAMIS is over three orders of magnitude larger than any previous set of 3D rendering assets for MIS in the literature, 2) Heterogeneity: SARAMIS offers order of magnitude larger number of anatomical targets than previous datasets, and 3) Patient variability: SARAMIS features a significantly larger number of subjects compared to previous datasets. Through multi-organ segmentation, the creation of new labelling for all assets, data curation, and camera path generation, SARAMIS enables simulation-based experimentation not possible from the underlying CT scans alone.
In addition to SARAMIS, we developed a Markov decision process environment for navigation within the colon using simulated intraoperative images derived from patient CTAP scans (i.e., not from dedicated CT colonography scans). The observed performance of the RL function and human observer in the colonoscopy navigation task across four patients that an effective generalisable cross- pattiettt navigation policy was learnt using our proposed training scheme. Furthermore, it is interesting to note the overall high (79.0% and 79.7%) success rates of finding structures within the colon, within 256 steps, for the held out test set across a variety of randomised starting locations for the intraoperative imaging probe. The highest success rates were observed for the rectum and caecum, possibly due to their distinct appearances compared to the remainder of the colon, and tight curvature and blind-loop nature of the colonic flexures. While we model wall intersection constraints within our work, we do not account for all possible constraints in the endoscopic settings for example, camera pose constraints such that the camera may not face directly opposite the direction of endoscope insertion from one step to the next, or extra-luminal boundaries imposed by surrounding visceral organs. Additionally, we qualitatively observe (Supp. Mat. Appendix E) that human trajectories are smoother than RL trajectories, which may arise from the lack of smoothness prior or regularization on the generated actions. Accounting for these constraints represents natural avenue of future research. Overall, whilst training was performed on subset of the available assets, this indicates the robustness of the proposed training scheme which allows for the trained policy to be generalisable not only across patients but also across starting locations and the four target structures included during training.
comparable, which indicates
7 Limitations
A limitation of this work lies in the design of shading nodes for the procedural texturing of SARAMIS assets. Despite designing the anatomical textures under the supervision of a clinician with surgical training and by referencing 2D intra-operative images of different anatomy, is likely that deviations from the proposed parameters in the associated shader nodes may result in non- non-clinically feasible renders. Whilst procedural texturing is still an industry standard in the computer-graphics community 17 29 19). this manual approach could be paired or replaced with learning- based texturing (25 in order to texture SARAMIS meshes in a data-driven way. Another important limitation of this work is that the dataset does not capture all structures of relevance for the full simulation of MIS/RAMIS scenes (e.g. ligaments, fatty tissue, and fluids), due to the limited resolution of CT scans, and its relatively poor detection of soft-tissue structures. This limitation presents natural avenue for future research, which could facilitate using SARAMIS to simulate different autonomous navigation tasks within the human abdominal anatomy.

---

8 Conclusions
In this work we introduce SARAMIS: Simulation Assets for Robotic-Assiste and Minimally-Invasite Surgery, a large-scale dataset of 3D rendering assets composed of 3D meshes, textures and diffuse maps for over 104 human anatomical structures. We warmly invite the wider research community to use SARAMIS assets for vision tasks in RAMIS/MIS such as depth estimation, camera pose estimation, or pairing tetrahedral meshes with open-source deformation modelling environments (20 24 to further develop surgical vision applications and autonomous navigation tasks in MIS/RAMIS research.
Acknowledgments and Disclosure of Funding
This work is supported by the Wellcome/EPPRC Centre for Interventional and Surgical Sciences [2031452/16/Z]. NMB, AA, ET, AS, and SF are supported by the EPSRC-funded UCL Centre for Doctoral Training in Intelligent, Integrated Imaging in Healthcare (i4health) [EP/S021930// AA is supported by an EPSRC Industrial Case grant (EP/W522077/1), and a Microsoft Research PhD Scholarship Wellcome Trust award [221915/7/20/] MJC, YH, NMB, and SUS are supported by [EP/T029404/1]. TD is supported by (EP/VO52438/1)) ZMCB supported by the Natural Sciences and Engineering Research Council of Canada Postgraduate Scholarships-Doctee Doctoral Program, and the University College London Overseas and Graduate Research Scholarships. This work is also supported by the International Alliance for Cancer Early Detection, alliance between Cancer Research UK [C28070/A3091 C73666/A31378), Canary Center at Stanford University, the University of Cambridge, OHSU Knight Cancer Institute, University College London and the University of Manchester.
References
[1] M. Allan, S. Thompson, M. Clarkson, S. Ourselin, D. J. Hawkes, J. Kelly, and D. Stoyanov. 2d- 3d pose tracking of rigid instruments in minimally invasive surgery. In Information Processing in Computer-Assise Interventions: 5th International Conference, IPCAI 2014, Fukuoka, Japan, June 28, 2014. Proceedings 5, pages 1-10. Springer, 2014.
[2] J. Allard, S. Cotin, F. Faure, P.-J. Bensoussan, F. Poyer, C. Duriez, H. Delingette, and L. Grisoni. Sofa-an open source framework for medical simulation. In MMVR 15-Medicine Meets Virtual Reality, volume 125, pages 13-18. IOP Press, 2007.
[3] R. Antonova, P. Shi, H. Yin, Z. Weng, and D. K. Jensfelt. Dynamic environments with deformable objects. In Thirty-fitth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[4] L. Bentley. K-d trees for semidynamic point sets In Proceedings of the sixth annual symposium on Computational geometry, pages 187-197, 1990.
[5] J. Bernal, F. Sánchez, G. Fernández- Exparirccz.aaarach.. D. Gil, C. Rodríguez, and F. Vilariño. Wm- dova maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Computerized medical imaging and graphics, 43:99-111, 2015.
[6] H. Borgli, V. Thambawita, P. Smedsrud, S. Hicks, D. Jha, S. L.. Eskeland, K. R. Randel, K. Pogorelov, M. Lux, D. T. D. Nguyen, et al. Hyperkvasir, comprehensive multi-class image and video dataset for gastrointestinal endoscopy. Scientific data, 7(1):283, 2020.
[7] N. Bourdel, T. Collins, D. Pizarro, C. Debize, A.-s. Grémeau, A. Bartoli, and M. Canis. Use of augmented reality laparoscopic gynecology to visualize myomas. Fertility and sterility, 107(3):737-739, 2017.
[8] A. Buia, F. Stockhausen, and E. Hanisch. Laparoscopic surgery: A qualified systematic review. World journal of methodology, 5(4):238, 2015.
[9] M. Carstens, F. M. Rinner, S. Bodenstedr, A. C. Jenke, J. Weitz, M. Distler, S. Speidel, and F. R. Kolbinger. The dresden surgical anatomy dataset for abdominal organ segmentation in surgical data science. Scientific Data, 10(1):1-8, 2023.
10

---

[10] F. Chadebecq, L. B. Lovat, and D. Stoyanov. Artificial intelligence and automation in endoscopy and surgery, Nature Reviews Gastroenterology Hepatology, 20(3):171-182, 2023.
[11] Z. Chen, A. Deguet, R. Taylor, S. DiMaio, G. Fischer, and P. Kazanzides. An open-source hardware and software platform for telesurgical robotics research, Proceedings of the MICCAI Workshop Systems and Architecture for Computer Assisted Interventions, Nagoya, Japan, volume 2226, 2013.
[12] I. Cheng, T. Liu, G. Zhang, X. Peng, and X. Zhang. Does minimally invasive surgery improve short-term recovery in total knee arthroplasty? Clinical Orthopaedics and Related Research@, 468: 168:166516448, 2010.
[13] Cignoni, M. Callieri, M. Corsini, M. Dellepiane. F. Ganovelli, G. Ranzuglia, et al. Meshlab: open-source mesh processing tool. In Eurographics Italian chapter conference, volume 2008, pages 129-136. Salerno, Italy, 2008
[14] T. Collins, D. Pizarro, S. Gasparini, N. Bourdel, P. Chauvet, M. Canis, L. Calvet, and A. Bartoli. Augmented reality guided laparoscopic surgery of the uterus. IEEE Transactions on Medical Imaging, 10(1)3371-380 2020.
[15] T. Dowrick, L. Chen, J. Ramalhinho, J. G.-B. Puyal, and M. J. Clarkson. Procedurally generated colonoscopy and laparoscopy data for improved model training performance. In B. Bhattarai, S. Ali, A. Rau, A. Nguyen, A. Namburete, R. Caramalau, and D. Stoyanov, editors, Data Engineering Medical Imaging, pages 67-77, Cham, 2023. Springer Nature Switzerland.
[16] T. Dowrick, B. Davidson, K. Gurusamy, and M. J. Clarkson. Large scale simulation of labeled intraoperative scenes unity. International Journal of Computer Assisted Radiology and Surgery, 7(5):961-963 2022.
[17] D. Ebert. Texturing & modeling: a procedural approach. Morgan Kaufmann, 2003.
[18] D. A. Field. Laplacian smoothing and delaunay triangulations. Communications in applied numerical methods, 4(6):709-712, 1988.
[19] J. D. Foley. Computer graphics: principles and practice, volume 12110. Addison-Wesley Professional, 1996.
[20] B. Foundation. Blender.
[21] N. Gameworks. Nvidia flex, 2018.
[22] J. Gehrman, E. Angenete, I. Björholt, E. Lesén, and E. Haglind. Cost-effectiveness analysis of laparoscopic and open surgery routine swedish care for colorectal cancer. Surgical endoscopy, 34:4403 4412, 2020.
[23] A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pallern recognition, pages 3354-3361. IEEE, 2012.
[24] K. Greff, F. Belletti, L. Beyer, C. Doersch, Y. Du, D. Duckworth, D. Fleet, D. Gnanapragasam, F. Golemo, C. Hermaan, et al. Kubrie: A scalable dataset generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3749-3761, 2022.
[25] P. Guerrero, M. Hašan, K. Sunkavalli, R. Měch, T. Boubekeur, and N. J. Mitra. Matformer: A generative model for procedural materials. arXiv preprint arXiv:2207.0100 2022.
[26] A. A. Gumbs, F. Alexander, K. Karcz, E. Chouillard, R. Croner, J. Coles- Black, B. de Simone, M. Gagner, B. Gayet, V. Grasso, al. White paper: definitions artificial intelligence and autonomous actions in clinical surgery. Artificial Intelligence Surgery, 2(2))93-100, 2022.
[27] Haidegger, S. Speidel, Stoyanov, and R. M. Satava. Robot-assisted minimally invasive robotics in the data age. Proceedings of the IEEE, 110(7):835-84) 3465-846,40 2022.
11

---

[28] J. Han, J. Davids, H. Ashrafian, A. Darzi, D. S. Elson, and M. Sodergren. A systematic review of robotic surgery: From supervised paradigms to fully autonomous robotic approaches. The International Journal of Medical Robotics and Computer Assisted Surgery, 18(2):e2358, 2022.
[29] D. Hearn, M. P. Baker, and M. P. Baker. Computer graphics with OpenGI, volume 3. Pearson Prentice Hall Upper Saddle River, NJ: 2004.
[30] W.-Y. Hong, C.-L. Kao, Y.-H. Kuo, J.-R. Wang, W.-L. Chang, and C.-S. Shih. Cholecseg&k: a semantic segmentation dataset for laparoscopic cholecystectomy based on cholec80. arXiv preprint arXiv:2012.12455, 2020.
[31] Y. Hu, T. Schneider, B. Wang, D. Zorin, and D. Panozzo. Fast tetrahedral meshing in the wild. ACM Trans. Graph., 39(4), July 2020.
[32] F. Isensee, P. F. Jaeger, S. A. Kohl, J. Petersen, and K. H. Maier-Hein. nnu-net: self-configuring method for deep learning-baid -based biomedical image segmentation. Nature methods, 18(2):203- 211, 2021.
[33] R. Izzo, D. Steinman, Manini, and L. Antiga. The vascular modeling toolkit: a python library for the analysis of tubular structures in medical images. Journal of Open Source Software, 3(25):745, 2018.
[34] W. Jakob, S. Speierer, N. Roussel, and D. Vicini. Dr. jit: a just-in-time compiler for differentiable rendering. ACM Transactions on Graphics (TOG), 114):11-19 2022.
[35] Y. Ji, H. Bai, C. Ge, J. Yang. Y. Zhu, R. Zhang, Z. Li, L. Zhanng, W. Ma, X. Wan, et al. Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation. Advances in Neural Information Processing Systems, 35:36722-36732 2022.
[36] H. Kenngott, J. Wünscher, M. Wagner, A. Preukschas, A. Wekerle, P. Neher, S. Suwelack, Speidel, F. Nickel, D. Oladokun, et al. Openhelp (heidelberg laparoscopy phantom): develop- ment of an open-source surgical evaluation and training tool. Surgical endoscopy, 29:3338-3344 2015.
[37] A. Khaddad, J.-C. Bernhard, G. Margue, C. Michiels, S. Ricard, K. Chandelon, F, Bladou, N. Bourdel, and A. Bartoli. A survey of augmented reality methods guide minimally invasive partial nephrectomy. World Journal of Urology, 41(2):335-343, 2023.
[38] B. D. Killcen, S. M. Cho, M. Armand, R. H. Taylor, and M. Unberath. In silico simulation: A key enabling technology for next-generation intelligent surgical systems. Progress in Biomedical Engineering. 2023.
[39] H. U. Lemke and M. W. Vannier. The operating room and the need for an it infrastructure and standards. Int. J. Comput. Assist. Radiol. Surg., 133::177-21, 2006.
[40] S. Leonard, K. L. Wu, Y. Kim, A. Krieger, and P. C. Kim. Smart tissue anastomosis robot (star): vision-guided robotics system for laparoscopic suturing. IEEE Transactions on Biomedical Engineering, 61(4):1305-1311 2014.
[41] K. Li, M. I. Fathan, K. Patel, T. Zhang, C. Zhong, A. Bansal, A. Rastogi, J. S. Wang, and G. Wang. Colonoscopy polyp detection and classification: Dataset creation and comparative evaluations. Plos one, (6(8))00,5,,,, 2021.
[42] L. Lin, Y. Liu, Y. Hu, X. Yan, K. Xie, and H. Huang. Capturing, reconstructing, and simulating: 2022: 17th European Conference, Tel
the urbanscene3d dataset. In Computer Aviy, Israel, October 23-27, 2022, Proceedings, Part VIII, pages 93-109. Springer, 2022.
[43] X. Lin, Y. Wang. J. Olkin, and D. Held. Softgym: Benchmarking deep reinforcement learning for deformable object manipulation. In Conference on Robot Learning, pages 432-448. PML.R, 2021.
[44] W. E. Lorensen and H. Cline. Marching cubes: A high resolution surface construction algorithm. ACM siggraph computer graphics, 21(4):163-169, 1987.
12

---

[45] J. Ma, Y. Zhang, S. Gu, C. Zhu, C. Ge, Y. Zhang, X. An, C. Wang, Q. Wang, X. Liu, et al. Abdomenct- lk: Is abdominal organ segmentation solved problem? IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):6695-677 6714, 2021.
[46] L. Maier-Hein, S. S. Vedula, S. Speidel, N. Navab, R. Kikinis, A. Park, M. Eisenmann, H. Feuss- ner, G. Forestier, S. Giannarou, et al. Surgical data science for next-generation interventions. Nature Biomedical Engineering, 199):6916666, 2017.
[47] V. Minutolo, A. Licciardello, B. Di Stefano, M. Arena, G. Arena, and V. Antonacci. Outcomes and cost analysis of laparoscopic versus open appendectomy for treatment of acute appendicitis: years experience in district hospital. BMC surgery, 14(1):1-6, 2014.
[48] R. Modrzejewski, T. Collins, B. Seeliger, A. Bartoli, A. Hostettler, and J. Marescaux. An in vivo porcine dataset and evaluation methodology to measure soft-body laparoscopic liver registration accuracy with extended algorithm that handles collisions. International journnaal of computer assisted radiology and surgery, 14: 14:1237-1245, 2019.
[49] K. Mohiuddin and S. J. Swanson. Maximizing the benefit of minimally invasive surgery. Journal of surgical oncology. 10155)315-31,, 2013.
[50] A. Myronenko and X. Song. Point set registration: Coherent point drift. IEEE transactions pattern analysis and machine intelligence, 32(12):2262-23 2275, 2010.
[51] S. O'Sullivan, N. Nevejans, C. Allen, A. Blyth, S. Leonard, U. Pagallo, K. Holzinger, A. Holzinger, M. I. Sajid, and H. Ashrafian. Legal, regulatory, and ethical frameworks for development of standards in artificial intelligence (ai) and autonomous robotic surgery. The international journal of medical robotics and computer assisted surgery, 15(1)=e1968, 2019.
[52] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L.. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[53] G. Paulin and M. Ivasic-Kos. Review and analysis of synthetic dataset generation methods and techniques for application in computer vision. Artificial Intelligence Review, pages 1-45, 2023.
[54] M. Pfeiffer, I. Funke, M. R. Robu, S. Bodenstedt, L. Strenger, S. Engelhardt, T. RoB, M. J. Clarkson, K. Gurusamy, B. R. Davidson, et al. Generating large laheled data sets for laparoscopic image processing tasks using unpaired image-to-image translation. In Medical Image Computing and Computer Assisted InterventionMMICCAI 2019: 22nd International Conference, Shenzhen, China, October 13-17, 2019, Proceedings, Part V 22, pages 119-127. Springer, 2019.
[55] M. Pfeiffer, C. Riediger, S. Leger, J.-P. Kühn, D. Seppelt, R.-T. Hoffmann, J. Weitz, and S. Spei- del. Non-rigid volume to surface registration using data-driven biomechanical model. In Medical Image Computing and Computer Assisted intervention-MMCCAl 2020: 23rd Interna- tional Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part IV 23, pages 724-734. Springer, 2020.
[56] S. Pieper, M. Halle, and R. Kikinis. 3d slicer. In 2004 2nd IEEE international symposium on biomedical imaging: nano to macro (IEEE Cat No. 04EX821), pages 632 -23. IEEE, 2004.
[57] A. Pore, E. Tagliabue, M. Piccinelli, D. Dall'Alba, A. Casals, and P. Fiorini. Learning from demonstrations for autonomous soft-tissue retraction. In 2021 International Symposium on Medical Robotics (ISMR), pages 1-7, IEEE, 2021.
[58] L. Qian, Y. Wu, S. P. DiMaio, N. Navab, and Kazanzides. A review of augmented reality in robotic-assisted surgery. IEEE Transactions on Medical Robotics and Bionics, 2(1):1-16, 2019.
[59] N. Rabbani, I. Calvet, Y. Espinel, B. Le Roy, M. Ribeiro, F. Buc, and A. Bartoli. A method- ology and clinical dataset with ground-truth to evaluate registration accuracy quantitatively in computer-assisted laparoscopic liver resection. Computer Methods in Biomechanics and Biomedical Engineering: Imaging Visualization, 10(4):441-450 2022.
[60] A. Rau, B. Bhattarai, I.. Agapito, and D. Stoyanov. Bimodal camera pose prediction for endoscopy, arXiv preprint arXtv:2204.04990 2022.
13

---

[61] M. Rodrigues, M. Mayo, and P. Patros. Surgical tool datasets for machine learning research: survey, International Journal of Computer Vision, 130(9))2222-222 -2242, 2022.
[62] H. Saeidi, J. D. Opfermann, M. Kam, S. Wei, S. Léonard, M. H. Hsieh, J. U. Kang, and A. Krieger. Autonomous robotic laparoscopic surgery for intestinal anastomosis. Science robotics, 7(62))ebb)2908, 2022.
[63] P. M. Schcikl, E. Tagliabue, B. Gyenes, M. Wagner. D. Dall' Alba, P. Fiorini, and F. Mathis- Ullrich. Sim-to-real transfer for visual reinforcement learning of deformable object manipulation for robot-assisted surgery. IEEE Robotics and Automation Letters, 8(2):560-555 567, 2023.
[64] C. Schneider, S. Thompson, J. Totz, Y. Song. M. Allam, M. Sodergren, A. Desjardins, D. Barratt, S. Ourselin, K. Gurusamy, et al. Comparison of manual and semi-automatic registration in augmented reality image-guided liver surgery: clinical feasibility study. Surgical endoscopy, 34:4702-471 1, 2020.
[65] A. Segato, M. Di Marzo, Zucchelli, S. Galvan, R. Secoli, and De Momi. Inverse reinforeement learning intra-operative path planning for steerable needle. IEEE Transactions on Biomedical Engineering, 19(6):1995-205) 2021.
[66] A. Smigielski, L. Piskorz, and W. Koptas. Comparison of treatment costs of laparoscopic and open surgery. Videosurgery and Other Miniinvasive Techniques, 10(3):437-441, 2015.
[67] L. Soler, A. Hostettler, V. Agnus, A. Charnoz, J. Fasquel, J. Moreau, A. Osswald, M. Bouhadjar, and J. Marescaux. 3d image reconstruction for comparison of algorithm database: patient specific anatomical and medical image database. IRCAD, Strashourg, France, Tech. Rep, 1(1), 2010.
[68] J. Straub, T. Whelan, L. Ma, Y. Chen, E. Wijmans, S. Green, J. Engel, R. Mur-Artal, C. Ren, S. Verma, et al. The replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1966.05797 2019.
[69] S. Suwelack, S. Röhl, S. Bodenstedt, D. Reichard, R. Dillmann, T. dos Santos, L. Maier-Hein, M. Wagner, J. Wünscher, H. Kenngott, et al. Physics-based based shape matching for intraoperative image guidance. Medical physics, 41( (1(11):111901 19901, 2014.
[70] E. Tagliabue, A. Pore, D. Dall' Alba, M. Piccinelli, and P. Fiorini. Unityflexml: Training reinforcement learning agents in a simulated surgical environment.
[71] T. Tarin, A. Feifer, S. Kimm, L. Chen, D. Sjoberg, Coleman, and P. Russo. Impact of common clinical pathway on length of hospital stay in patients undergoing open and minimally invasive kidney surgery. The Journal of urology. 191(5):1225-122) 2014.
[72] B. Taylor L., M. Golhar, R. Vijayan, V. Akshintala, J. R. Garcia, and N. J. Durr. Colonoscopy 3d video dataset with paired depth from 2d-3d registration. arXiv:2206. 08903, 2022.
[73] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033, IEEE, 2012.
[74] A. Twinanda, S. Shehata, D. Mutter, J. Marescaux, M. De Mathelin, and N. Padoy. Endonet: a deep architecture for recognition tasks on laparoscopic videos. IEEE transactions on medical imaging, 36(1):86 1:86-77 2016.
[75] Wasserthal, M. Meyer, H.-C Breit, J. Cyriac, S. Yang, and M. Segeroth. Totalsegmentator: robust segmentation of 104 anatomical structures in ct images. arXiv preprint arXiv:2208.0554 05868, 2022.
[76] J. Y. Wee, L.-J. Kuo, and J. C.-Y. Ngu. A systematic review of the true benefit of robotic surgery: Ergonomics. The International Journal of Medical Robotics and Computer Assisted Surgery, 16(4):e2113, 2020.
[77] A. Yavariabdi, A. Bartoli, C. Samir, M. Artigues, and M. Canis. Mapping and characterizing endometrial implants by registering 2d transvaginal ultrasound to 3d pelvic magnetic resonance images. Computerized Medical Imaging and Graphics, 45:11-25, 2015.
14


Reviews:
[{"review_id": "7FGfP00bGm", "reviewer": "Reviewer_Mims", "summary": "", "questions": "", "limitations": "\nLimitations are briefly discussed the last paragraph of the conclusion. I recommend extending this paragraph (see above) and make it more prominently visible with a paragraph header and separation from future work and concluding statements.\nNo societal impact is discussed, but I do not expect negative impact from this work.\n\n", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "\n- The paper introduces large labeled medical dataset. Rather than introducing new data, the paper sources its data from multiple existing datasets and focuses its contribution on segmentation, labeling, curation and camera path generation.\nThis increases the dataset size and simplifies the use of multiple datasets.\n\n- Efforts like this paper are much needed to counter the notorious lack of available medical data for tackling various machine learning and computer vision problems. Highly realistic synthetic data is a great and lacking step to close the large training gap in the medical image domain.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "xn8iKfweAQ", "reviewer": "Reviewer_PbKg", "summary": "", "questions": "", "limitations": "See \"Opportunities For Improvement\"\n\nEdit: After the rebuttal, I decreased my score as the main problem of the paper remains, no code documentation and public data are available for review. The tasks and claims are unclear.", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "The idea of having a simulation for autonomous navigation in colonoscopy is interesting. The paper is quite straightforward to understand.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "MvTUj2pe4d", "reviewer": "Reviewer_1urX", "summary": "", "questions": "", "limitations": "limitations addressed ", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The team creates the world's largest MIS Dataset. The dataset is very meaningful to the biomedical community, and can be used to improve and evaluate the performance of autonomous navigation tasks. Real world data is hard to get, to tackle this the team builds high quality synethetic data with through computer graphic approaches and human expert validation. \n    - The team incorportate existing open-source software such as Blender to create realistic simulation of depth maps, stereo and monocular cameras, and deformation of soft tissue.\n    - The team explores one important application of the dataset, which is autonomous navigation tasks, setting a benchmark for future research.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "LY5b1VVxad", "reviewer": "Reviewer_zAJD", "summary": "", "questions": "", "limitations": "As mentioned above (“Opportunities for improvement”), the realism of the textures might be a bottleneck for the application of the data set. I encourage the authors to thoroughly discuss this aspect.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "Uniqueness: First openly accessible dataset that provides rendering assets of abdominal organs that can be used for in-silico validation of algorithms in the context of robot assisted and minimally invasive surgery.\n\nCuration: Careful curation of three existing CT datasets\n\nOpen science: Both the dataset with supporting datasheet and all necessary processing procedures will be provided in publicly accessible repositories which enables straightforward reproduction and extension.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Submission: SARAMIS (**#932**) Bottom line: **accept (as poster)**

**AC Special Note**: one out of the four reviewers obviously believes the authors did not engage in (sufficiently original) work to warrant consideration (score of 2), whereas the three other reviewers (scores of 7, 8, and 7) believe the work to present novel and clearly original contributions to the field; the **AC chose to accept (poster), with a confidence of 3 (somewhat confident), and would appreciate the SAC/PC to take a look at whether or not the lowest-scoring review (Reviewer PbKg) is yet a (more more credible) domain expert, and thus should carry the decision**.

**Summary**: the authors combined pre-existing datasets of primarily soft-abdominal-tissue CT scans and then **created additional digital assets together with a clear outline of use cases** for the support of developing test environments for minimally invasive surgery. The assets are well described and tested; however, the final application (fully implemented test environment for MIS) is not provided, and as such it is difficult to adjudicate just how soon this dataset will see a definitive use case (as intended). As such, next to the Special Note above, this dataset might also be more appropriate for a "surgery specialty journal", compared to the broad domain-nonspecific NeurIPS...

Pros: from reading the paper and looking through the assets in general, it seems obvious that the authors have spent quite some time preparing, describing, and working with the data they put together

Cons: while the application (simulating minimally invasive surgery) is conceptually clear, and a dataset of assets for this purpose is needed, it is somewhat unclear to the AC how much work the creation of such a simulation environment entails, and whether it is primarily restricted by the lack of a comprehensive dataset (as presented) or rather by the difficulty of developing such a system for other reasons; as such the AC finds it hard to judge the **relative value** of this dataset for the community beyond being a genuinely fascinating case of taking existing datasets and creating a (labor-intensive!) derivative dataset of assets. The value would also seem primarily for a surgical robotics community rather than within broader machine-learning or computer vision. Given my lack of domain knowledge (no experience with how CV/machine learning is applied **in the field of surgical robotic simulation**!), the AC is only somewhat confident, relying on the median (three out of four reviewers scored 7 or higher) rating.

Decision: the AC follows the median reviewer in proposing accepting the dataset -- under the proviso that the SAC and PCs confirm that the outlier review (Reviewer PbKg) isn't actually the person with the strongest domain expertise.

----

Reviewer Mims: the limitations section was highlighted by the authors (as requested by the reviewer), and other concerns seem to have been addressed satisfactorily

----

Reviewer PbKg (**high-confidence outlier with score 2**): From the entire (very lengthy) exchange between the reviewer and authors, my read is that the reviewer infers a lack of authors' contribution to **novelty** w.r.t. dataset and methods. The back and forth hinges on whether or not the work of the authors, as presented, does in fact correspond to a dataset asset that, in its present form, is "missing" from the literature. In my general opinion (as AC, without deep domain knowledge), I would say that the other three reviewers' score suggests to me that this reviewer either has a very different idea of a standard for "novel, original, and significant contribution", or has a strong domain expertise not shared by the other reviewers; it is obvious from the unchanged score that the reviewer does not believe that the authors have sufficiently addressed the concerns. As AC, I took the liberty to override this reviewer's score, but see Special Note and my other comments above!

----

Reviewer 1urX: it seems that the concerns raised were addressed, and another reviewer commented that the authors' clarifications also increased their score.

----

Reviewer zAJD: offered **many** points, but seems very satisfied with the overall rebuttals and how authors addressed the concerns.

----

In summary, I would kindly ask the SAC and PCs to take a look at the low-scoring reviewer's comments and, together with the question of fit (surgical robotics audience) make a final decision as to whether or not to accept this contribution.
Decision: Accept (Poster)