SCANNER FOR INTRAOPERATIVE APPLICATION

A tissue scanning system (1) comprising: a depth sensor (13) configured to determine distance to a surface; a pointer device (11), wherein the depth sensor (13) is mounted to the pointer device (11); a camera-based tracking system (3) configured to determine (114) relative orientation and position between an anatomical feature (9) and the pointer device (11); and at least one processing device (6). The processing device (6) is configured to: generate (116) a surface point cloud (16) of a surface (17) associated with the anatomical feature (9) based on a plurality of determined distances from the depth sensor (13) and corresponding relative orientation and position of the pointer device (11) relative to the anatomical feature (9).

TECHNICAL FIELD

The present disclosure relates to a system and method of scanning tissue during surgery.

BACKGROUND

It is important to accurately position surgical tools during surgery for effective treatment. Examples of surgery requiring accurate placements of surgical tools include surgery in relation to bones and joints, such as knee and hip replacement surgery. This may involve cutting, or otherwise shaping, bone and cartilage of the patient and securing implantable components thereto.

This requires the surgical tools to be accurately configured relative to the patient such that the surgical tool can operate in accordance with the surgical plan. This may involve apparatus and systems that assist the surgeon to guide the surgical tool to the desired position.

Consideration for anatomical features of a patient is important for the surgical plan and the resultant operation. Some of the anatomical features of the patient may be determined preoperatively based on medical imaging data, such as CT (X-ray computed tomography) or MRI (magnetic resonance imaging) images. These medical images may be analysed by a computer to construct a 3D model, such as a segmented mesh, that represents the imaging data.

However, such 3D modelled features are not always perfect and accurate. During an operation, a surgeon may find deviations from the modelled anatomic features compared to the actual anatomical features. For example, once skin and muscle are moved, the exposed bone and other tissue deviates from the 3D model used for the surgical plan. In other examples, some anatomical features may not be able to be accurately modelled preoperatively due to difficulties imaging that particular anatomical feature.

In the present description the term “position” with reference to a position of an element may include a position of the element in two, or three, dimensional space, where context permits may also include the orientation of the element.

SUMMARY

A tissue scanning system comprising: a depth sensor configured to determine distance to a surface; a pointer device, wherein the depth sensor is mounted to the pointer device; a camera-based tracking system configured to determine relative orientation and position between an anatomical feature and the pointer device; and at least one processing device. The processing device is configured to: generate a surface point cloud of a surface associated with the anatomical feature based on a plurality of determined distances from the depth sensor and corresponding relative orientation and position of the pointer device relative to the anatomical feature.

In some examples, the tissue scanning system further comprise: one or more pointer markers attached to the pointer device; and one or more tissue markers attached to the anatomical feature. To determine relative orientation and position between the anatomical feature and the pointer device, the camera-based tracking system, or the at least one processing device, is further configured to: identify the pointer markers and tissue markers in one or more fields of view of the camera-based tracking system; and based on locations of the pointer markers and tissue markers in the field of view, calculate the relative orientation and position between the anatomical feature and the pointer device.

In some examples, the pointer markers (19) and the tissue markers (21) are ArUco fiducial markers.

In some examples of the tissue scanning system, the pointer device includes a guide tip, wherein the relative position of the guide tip to the depth sensor is fixed, or selectively fixed, during use.

In some examples of the tissue scanning system, a relative distance between the guide tip and the depth sensor is selected to be within a desired operating range of the depth sensor.

In some examples of the tissue scanning system, wherein the guide tip is configured to contact an index point on the surface associated with the anatomical feature, wherein the guide tip aids in maintaining a scanning distance between the depth sensor and the surface to be within the desired operating range.

In some examples of the tissue scanning system, the contact between the index point and the guide tip forms a pivot point such that as the pointer device is moved relative to the anatomical feature around the pivot point. The depth sensor determines a corresponding depth to the surface for that relative orientation and position to generate the surface point cloud of the surface.

In some examples, the pivot point is an intermediate reference point used to determine relative orientation and position of the anatomical features and the pointer device.

In some examples, the depth sensor is selected from one or more of: a Lidar (light detection and ranging); and/or an optical rangefinder.

In some examples, the tissue scanning system further comprise: a second camera mounted to the pointer device, wherein the depth sensor is directed in a direction within a field of view of the second camera; and a graphical user interface to display at least part of an image from the second camera.

In some examples, the at least one processing device of the tissue scanning system is further configured to: receive a patient profile of the anatomical feature); determine a predicted outline of the anatomical feature based on the patient profile; generate a modified image comprising the image from the second camera superimposed with the predicted outline; wherein the graphical user interface displays the modified image to guide a user to direct the depth sensor mounted to the pointer device to surface(s) corresponding to the predicted outline of the anatomical feature.

In some examples of the tissue scanning system, the processing device is further configured to: compare the generated surface point cloud with the patient profile; and generate an updated patient profile based on a result of the comparison.

In some examples of the tissue scanning system, the depth sensor and the at least one processing device are part of a mobile communication device.

There is also provided a method of acquiring a surface point cloud of a surface associated with an anatomical feature, the method comprising: receiving a plurality of determined distances from a depth sensor, wherein each determined distance has accompanying spatial data indicative of relative orientation and position of the depth sensor to the anatomical feature; determining the relative orientation and position of the depth sensor to the anatomical feature from the spatial data; and generating a surface point cloud of the surface associated with the anatomical features based on: the plurality of determined distances from the depth sensor: and corresponding relative orientation and position of the depth sensor to the anatomical feature.

In some examples of the method, determining relative orientation and position of the depth sensor to the anatomical feature further comprises: determining the spatial data by identifying in one or more fields of view of a camera-based tracking system: pointer markers mounted relative to the depth sensor; and tissue markers mounted relative to the anatomical feature. Based on locations of the pointer markers and tissue markers in the field of view, the method further includes calculating the relative orientation and position between the anatomical feature and the depth sensor.

In some examples, the method further comprises: receiving an image from a second camera, wherein the depth sensor is directed in a direction within a field of view of the second camera; receiving, a patient profile of the anatomical feature; determining a predicted outline of the anatomical feature based on the patient profile; generating a modified image comprising the image from a second camera superimposed with the predicted outline; displaying, at a graphical user interface, the modified image to guide a user to direct the depth sensor to surface(s) corresponding to the predicted outline of the anatomical feature.

In some examples, the method further comprises: comparing the generated surface point cloud with the patient profile; and generating an updated patient profile based on a result of the comparison.

A non-transitory, tangible, computer-readable medium comprising program instructions that, when executed, cause a processing device to perform the method.

There is also provided a tissue scanning system comprising: a camera-based tracking system configured to: determine relative orientation and position between an anatomical feature and a pointer device; and a depth sensor at the pointer device configured to capture surface point cloud measurements of surface(s) associated with the anatomical feature.

There is also provided a tissue scanning system comprising: a pointer device to receive a depth sensor configured to capture surface point cloud measurements of surface(s) associated with an anatomical feature; and a camera-based tracking system configured to: determine relative orientation and position between an anatomical feature and the pointer device.

DESCRIPTION OF EMBODIMENTS

Overview

FIG.1illustrates an example of a tissue scanning system1. This includes a depth sensor13configured to determine distance to a surface, in particular a surface17of an anatomical feature9of a patient. The depth sensor13is mounted to a pointer device11. The pointer device11may include a guide tip29to contact an index point35on the surface17to aid locating the depth sensor13to within a desired operating range33.

A camera-based system3is configured to determine114relative orientation and position between the anatomical feature9and the pointer device11. This enables a processing device to generate116a surface point cloud16of the surface17based on associating the received112plurality of determined distances from the depth sensor13with the corresponding relative orientation and positions.

In some examples, the pointer device11has one or more pointer markers19and one or more tissue markers21attached to the anatomical feature9. These pointer markers19and tissue markers21, in the field of view23of the camera-based tracking system3, assist in calculating the relative orientation and position between the anatomical feature9and the pointer device11.

The tissue scanning system1may be used in a method100to capture information to generate the surface point cloud16, or other three-dimensional model, of the anatomical feature during an operation. In some other examples, the tissue scanning system1is used to supplement, or update, an existing model or medical images of the anatomical feature. For example, pre-operative medical images may have been used to generate a patient profile51for that patient's specific anatomical feature, which is incorporated into a surgical plan. The tissue scanning system1may be used to directly scan the tissue during operation to provide an updated and more accurate surface point cloud16for the patient profile.

In some examples, the scanning system1may include providing, on a graphical user interface, a modified image to guide a user to the depth sensor13to the surface. This modified image is based on a real-time, or near real-time, image49superimposed with a predicted outline of the anatomical feature9based on a patient profile51(where the patient profile51may include information from preoperative medical imaging). This modified image can assist the user to direct the depth sensor to particular areas of interest to update the patient profile. This can be useful where particular tissue(s) are difficult to accurately image preoperatively.

The components of the tissue scanning system1will now be described in detail followed by methods of implementation.

The Anatomical Features and Tissue Markers

The anatomical features9can include bone, cartilage and other tissue of a patient. The surface17of the anatomical feature9can be any surface of interest dependent on the type of surgery. In some examples, this can include the surface of the femur bone (and related tissue) during arthroplasty.

To assist determining relative orientation and position, one or more tissue markers21can be attached to the anatomical feature9. In the example ofFIG.1, the tissue marker21is attached to the femur bone, and in particular at a shaft portion that will not be removed during arthroplasty.

Although only one tissue marker21is illustrated inFIG.1, it is to be appreciated that multiple tissue markers21can be used that may improve accuracy and range for the camera-based tracking system3. Details of the tissue marker21will be described in further detail below and can include features similar to the pointer markers10.

In some other examples, tissue markers21may not be necessary if the camera-based tracking system3can determine the orientation and position of the anatomical feature9. The camera-based tracking system3may determine, from image(s), unique surfaces, outline, or other characteristics of the anatomical feature to determine the orientation and position.

Pointer Device

Referring toFIG.2, the pointer device11is configured to receive the depth sensor13. In this example, the depth sensor11is part of a mobile communication device61and the pointer device11is configured to receive the mobile communication device61. This can include a cradle62to receive the mobile communication device61. It is to be appreciated that the depth sensor11(or the mobile communication device61) can be mounted by other means such as clamps, screws, and other fastening means.

The pointer device11also includes an elongated shaft28that terminates with a guide tip29. The relative position of the guide tip29and the depth sensor13is fixed (or in alternative examples selectively fixed) during use. In some examples, the relative distance31between the guide tip29and the depth sensor13is selected to be within a desired operating range33of the depth sensor13. The guide tip29can be used to contact an index point35on the surface17of the anatomical feature. Thus the guide tip29can aid in maintaining a scanning distance37between the depth sensor13and the surface17to be scanned to be within the desired operating range.33.

Furthermore the contact between the index point35and the guide tip29can act as a pivot point39. The user can apply slight pressure so that the guide tip29stays in contact with a particular index point35whilst the pointer device11is moved into various orientation and positions relative to the anatomical feature9around that pivot point39. The depth sensor13determines the various depths that can be associated with those various relative orientation and positions for the system to generate the surface point cloud16. In some examples, the guide tip29includes a sharp point to mildly pierce and engage the surface17so that the guide tip29does not slip from the particular index point35. In other alternatives, the guide tip29may include a partially spherical surface to assist in rotation of the pointer device11around the pivot point39.

The pointer device11may have one or more pointer markers19attached. The pointer markers19may assist the camera-based tracking system3, and/or one or more processing devices6in the system to determine the orientation and position of the pointer device11. This will be discussed in further detail below.

Depth Sensor

The depth sensor13is configured to determine a distance to a surface. This can include a depth sensor using laser light, such as a Lidar (light detection and ranging) technology or other laser range finder. This can involve determining distance by time-of-flight of a light pulse that is directed to the surface17and reflected back to the depth sensor13.

In some examples, the depth sensor13can include a Lidar detector that can includes a flash Lidar to allow three dimensional image of an area with one scan. This can provide imaging Lidar technology that can determine a distance between the depth sensor13to a plurality of points on the surface17. In some examples, the depth sensor (or processor processing distance data) has a range gate to ensure only certain measurements are associated with the measured surface point cloud. For example, it would be desirable to exclude the operating table, operating room floor, and walls, as these items do not relate to the anatomical features. Thus one parameter may include excluding measurements greater than or equal to a specified distance.

In some examples, the depth sensor13is associated with a mobile communication device, smart phone, tablet computer, or other electronic device. In one example, the depth sensor13is a Lidar scanner such as provided in the iPhone 12. Pro and the iPad Pro products from Apple Inc.

In some examples, the depth sensor13includes a depth camera. This can include a system including light projectors (including projectors that project multiple dot points) and a camera to detect reflections of those dot points on the surface17, and a processor and software to create a surface point cloud16, or other representation or data of a three dimensional surface. In some examples the depth sensor13includes the TrueDepth camera and sensor system in the iPhone (iPhone X, iPhone XS, iPhone 11 Pro, iPhone 12) offered by Apple Inc. In some examples, the system may utilise software to process data from the depth sensor, such as the Scandy Pro 3D Scanner offered by Scandy. Such software may, at least in part, also function to generate a 3D mesh, surface point cloud, or other 3D model. This can include a 3D model in STL file format.

In other examples, the depth sensor can include optical rangefinders that utilise trigonometry and a plurality of spaced-apart optical sensors to determine range. This can include utilising principles from coincidence rangefinders or stereoscopic rangefinders. Such a depth sensor can include two or more optical cameras with a known spaced-apart distance whereby features of a target object in the captured image(s) are compared. The deviations of the location of the features in the captured image(s) along with the known spaced-apart distance can be used to compute the distance between the depth sensor13and the surface17.

Camera-Based Tracking System

The camera-based tracking system3is configured to determine the relative orientation and position between the anatomical feature9and the pointer device11. In one example of the camera-based tracking system3, as illustrated inFIG.1, this includes a camera with a field of view23that, in use, can detect at least part of the anatomical feature9and the pointer device11(or the corresponding tissue markers21and pointer marker19).FIG.7illustrates an image24from the field of view23of the camera-based tracking system3

In some examples, the camera-based tracking system3can include multiple cameras to provide a plurality of fields of view23. This can assist in providing greater accuracy or enable the system to be more robust. This can include enabling the camera-base tracking system3to continue operating even if the anatomical feature, pointer device11, or markers19,21are masked from the field of view from one of the cameras. Such masking may occur, for example, from the body of the surgeon or other instruments in the operating theatre. In these examples, the location of the camera(s) of the camera-based tracking system3is known and may be used to define, at least in part, a frame of reference for the system to enable determination of relative orientation and position of the anatomical feature9and the pointer device11.

In some examples, the camera-based tracking system3identifies markers19,21in the field of view23, whereby the markers can be more identifiable and distinguishable. For example the markers can include shape, colour, patterns, codes, or other unique or distinguishing features. In particular features that are in contrast to what would be found in the background of an operating room. In some example, this can include fiducial markers that can be used for identifying markers19,21and for calculating a position or point in space of the marker19,21in the field of view23. In some examples, the fiducial marker may have features to enable determining the orientation of the marker. For example, the perceived shape of the marker (from the perspective of a camera) may be skewed depending on the relative orientation to the camera and these characteristics used to calculate the relative orientation of the marker. Determining the position and/or orientation of markers enables calculation of corresponding positions and orientations for the anatomical feature or pointer device.

In some examples, multiple markers are used. Referring toFIG.2, the pointer markers19include two markers19a,19bthat are provided at different positions at the pointer device11. The use of multiple markers19a,19benable two corresponding locations of the two markers19a,19bto be determined. With known relative positions between the markers19a,19bat the pointer device11, such information can be used to calculate relative orientation of the pointer device11.

In some examples, the multiple markers19a,19binclude marks that are presented at different angles. This can be useful in some situations where the pointer device is orientated so that the one of the markers19a,19bis obscured or masked from the camera. The other marker, being orientated differently, may still be presented to visible to the camera-based tracking system.

In yet other examples, the different orientation of the markers19a,19bcan aid the camera-based tracking system to calculate the orientation of the pointer device11(or anatomical feature9) associated with the markers.

In some examples, the fiducial makers are ArUco, ARTag, ARToolKit, and/or AprilTag, fiducial markers that have been used for augmented reality technologies.

The camera-based tracking system3includes a processing device6to identify, in images captured in the field of view23, the markers19,21and their respective locations25,27. The processing device6can, based on those locations,25,27, calculate the relative orientation and positions between the anatomical features9and the pointer device. This calculation of relative orientation and position can resolving the multiple frames of reference and relative position and orientation of components in the system. For example:(1) Relative position and orientation between the anatomical feature9and the tissue marker21;(2) Relative position and orientation between the tissue marker21and the camera-based tracking system3;(3) Relative position and orientation between the camera-based tracking system3and the pointer marker21;(4) Relative position and orientation between the pointer marker19and the pointer device11;(5) Relative position and orientation between the pointer device11and the depth sensor13.(6) Relative position between the depth sensor13and the surface17based on the distance measured by the depth sensor13.

In some examples, the processor6performs a subset of the calculations noted above, and passes data to another processor to complete the calculations. In other examples, some calculations can be reduced. For example items (2) and (3) above may include a calculation of the relative position and orientation between the pointer marker19and the tissue marker21without using a frame of reference of the camera-based tracking system. This may be achieved when both the pointer markers19and the tissue marker21are both within the same field of view23of a camera.

In some examples, the camera-based tracking system may simply send images from the camera to another processing device to determine calculate the relative orientations and positions.

Second Camera and Graphical User Interface

In some examples, a second camera43is mounted to the pointer device11. The second camera43has a field of view47and the depth sensor13is directed in a direction45within that field of view47. This allows the second camera43to capture an image49of the region that the depth sensor13is sensing which, in use, will include the surface17of the anatomical feature as illustrated inFIG.3a.

A graphical user interface41can display at least part of the image49from the second camera43that can assist the surgeon in guiding the depth sensor13to various parts of the surface17. In some examples, the graphical user interface41can include a reticle36to mark the portion that the depth sensor13is actively sensing.

The graphical user interface41may also display a virtual guide to the user for manipulating the pointer device11/depth sensor13to enable measurements in specified areas of interest.

In some examples, the system includes a processing device6to generate a virtual guide for the user at the graphical user interface41. This may include a receiving105a patient profile51of the anatomical feature9. The patient profile51may comprise of earlier scans or models of the patient's anatomical feature. Such an initial patient profile51may be created pre-operatively from medical imaging, and/or with idealised models of such anatomical features. Such initial patient profiles51may not be precise, and thus the tissue scanning system1is used to update the patient profile51with refined measurements intra-operatively.

The processing device6then determines107a predicted outline53of the anatomical feature9based on the patient profile51. The predicted outline53is from the perspective of the second camera43relative to the surface17of the anatomical feature9. In some examples, data from the camera-based tracking system3can be used, at least in part, to determine the relative orientation of the anatomical feature9to the second camera43. This relative orientation, in conjunction with the patient profile51can then be used to determine the perspective and the predicted outline53.

The processing device6can then generate a modified image55(as illustrated inFIG.3b) that includes: the image49from the second camera43; and the predicted outline53superimposed on the image49. This modified image55, displayed at the graphical user interface41can be used by the surgeon to guide the depth sensor to the corresponding predicted outline53. The advantage is to obtain more accurate, and actual, measurements of the surface17of the anatomical features during surgery with the determined surface point cloud16.

The processing device6can compare121the generated surface point cloud16with the patient profile51. The result of the comparison can be used to update the patient profile51. An advantage of this system is to incorporate direct measurement of the surface17in the patient profile51that may be more accurate that a profile based only on medical imaging that may not have the same accuracy or resolution.

Mobile Communication Device

In some examples, the tissue scanning system1can include the use of a mobile communication device61, such as a smart phone. For example, the iPhone 12. Pro offered by Apple Inc includes a processing device, communications interface, a Lidar sensor (that is a depth sensor), multiple cameras (that can be function as, or augment, the second camera43and/or camera-based tracking system3). The mobile communication device61may also include inertial measurement sensors such as gyroscopes, accelerometers, and magnetometers that can assist in calculating orientation and movement of the depth sensor13and/or pointer device11.

The mobile communication device61also includes a graphical user interface41that can display the image49, the modified image55, as well as other data including representations of the patient profile51and updated patient profile59.

In some examples, one or more, or all of the functions of the processing device6are performed by the processing device in the mobile communication device61. In other examples, data from the depth sensor, cameras and/or inertial measurement sensors are sent, via the communication interface, to another processing device to perform the steps of the method.

Method of Acquiring a Surface Point Cloud

Referring toFIG.5, an example of acquiring a surface point cloud16of a surface17of the anatomical feature will be described in detail. It is to be appreciated that this is a specific example and variations of the method100may have less steps or additional steps.

The method100includes receiving112a plurality of determined distances from the depth sensor13as the pointer device11is manipulated by the surgeon. This results in obtaining distance measurements at different locations on the surface17. Each of the plurality of determined distances are associated with spatial data18, where the spatial data is indicative of relative orientation and position of the depth sensor13to the anatomical feature9. This is used to obtain each point that make up the surface point cloud16

The spatial data18can be obtained from the camera-based tracking system3. In one example, the spatial data18may include an image24(as shown inFIG.7) that includes, within the field of view23, the anatomical feature9and the pointer device11. Based on this image, the relative orientation and position of the anatomical feature9and the pointer11can be calculated. Since the position of the depth sensor13at the pointer device is known, this can be used to determine114the relative orientation and position of the depth sensor13relative to the anatomical feature.

In another example, as illustrated inFIG.5andFIG.8, the spatial data is based on identifying101pointer and tissue markers19,21in the field of view23that are attached, respectively, to the pointer device11and anatomical feature. Based on the locations25,27, of the pointer markers19and tissue markers21in the field of view23(such as in image24), the method includes calculating103the relative orientation and position between the anatomical feature9and the pointer device11. This can also include calculating the relative position and orientation with the depth sensor13).

The method100further includes generating116a surface point cloud16of the surface17associated with the anatomical features9as represented inFIG.9. The surface point cloud16is generated based on:The plurality of determined distances from the depth sensor13; andThe corresponding orientation and position of the depth sensor to the anatomical feature9for each of the determined distances.

FIG.9illustrates the surface point cloud16of parts of the surface17that have been scanned, whilst other portions that have not been scanned remain blank. A representation of the pointer device11is provided to show the spatial relationship of the pointer device11and this does not form part of the actual surface point cloud16of the surface17.

In some examples, the step of generating116a surface point cloud16may include multiple steps. In one example, the depth sensor13and system may first determine a 3D mesh, 3D point cloud, or other 3D model of the surface17relative to the depth sensor13. That is, using a frame of reference relative to the depth sensor13(which in some cases is the same as, or closely associated with, the frame of reference of the mobile communication device). The second step is to apply a transformation to a coordinate system desired by the system and user, which can include a coordinate system relative to a part of the pointer device, markers, anatomical feature9or even a reference point at the operating theatre. The transformation and selected coordinate system can allow easier relationships to be determined and calculated with respect to the anatomical features9(e.g. frame of reference relative to the femur bone).

FIG.6aillustrates various coordinate systems including: the depth sensor13coordinate system (that coincides with mobile communication device coordinate system), the pointer coordinate system, and marker19coordinate system. The depth sensor13may output distance data in a coordinate system relative to the depth sensor13. However, the system may desire distance data relative to another coordinate system, say the coordinate system of the pointer device relative to the pointer tip29. In one example, this includes applying a transformation matrix to the data for rotation and translations.

The normal vector of the coordinate system of the depth sensor relative to the pointer device is, in this example, defined by a plane: where x-coord: 16.4 mm, y-coord −63.5 mm, and z-coord 166.6 mm. This produces the transform matrix below:

To account for the translation of the depth sensor13to the pointer coordinate system, transformation further includes a correct translation of: x: 138.4 mm, y: −16.57 mm, z: 112.31 mm as illustrated inFIG.6b.

In some examples, the system may require a sequence of transformation to bring the various data in to a single desired coordinate system. In some particular examples, this includes transforming the data to be in the reference frame of the anatomical feature9, such as the bone.

Referring toFIG.10, this can include the depth sensor13and system determining201a plurality of distance measurements (which may be in raw form, or processed into, at least in part a 3D mesh or point cloud) that are relative to a coordinate system of the depth sensor13(or in the case of a mobile communication device the coordinate system specified by that device). That information needs to be transformed203to a coordinate system relative to the pointer device11(an example of which is discussed above). That information, in turn needs to be transformed207to a common frame of reference, namely that of the bone. In some examples this includes the camera-base tracking system3determining205locations in space of the respective pointer markers19and tissue markers21so that appropriate transformations can be applied to the data so that the surface point cloud16can be generated relative to a frame of reference of the anatomical feature9.

In some examples, the system may, at least in part, use the pivot point39as an intermediate reference point to assist in calculating an appropriate translation for the transform. For example, the pivot point39(calibrated to a known position and orientation to the depth sensor13) may, in use, be in contact with the surface of the anatomical feature.

Method of Visually Guiding the Pointer Device

The method100may also include providing a visual guide at a graphical user interface41. This can aid the user to scan particular areas of interest and/or areas that have not been adequately scanned.

Referring back toFIG.4, the method further includes receiving an image49from the second camera43, wherein the depth sensor13is directed in a direction45within a field of view47of the second camera43. Referring toFIG.3athat illustrates an example of the image49from the second camera43, this shows the surface17of the anatomical feature9and the guide tip29of the pointer device11.

The method further includes receiving105a patient profile51of the anatomical feature9. This patient profile may be constructed with medical imaging data of the patient, models of the patient based on the medical images and/or other scans and measurements of the patient, idealised or approximate models of anatomical features of a human. The method further includes determining107a predicted outline53of the anatomical feature based on the patient profile, which takes into consideration the perspective that the second camera43is viewing the anatomical feature9.

As illustrated inFIG.3B, a modified image55is generated109and displayed111, whereby the modified image55comprises at least part of the original image49of the anatomical features superimposed with the predicted outline53. The modified image55may also include a reticle36that represents the portion/direction45in the field of view47that the depth sensor13will be scanning.

In this example, it is desirable for the surgeon to obtain a more accurate data on the outer surface17of the anatomical feature9. As such, the user can then manipulate the pointer device11so that the reticle36is at or around the predicted outline53and the user can then trace (i.e. follow) the predicted outline53to obtain measurements along that predicted outline, which in turn causes a surface point cloud16to be generated for that corresponding traced area of the surface17.

In this example, the predicted outline53is in the form of a substantially enclosed loop. However, it is to be appreciated that in other examples, the predicted outline could be a silhouette of an area, whereby the silhouette guides the user to scan an area of interest.

In some examples, the surface point cloud16is used to update a patient profile59. In one example, the surface point cloud16becomes at least part of the updated patient profile59.

In other examples, the method includes comparing121the generated surface point cloud16with the existing patient profile. That is, identifying the points of difference between the patient profile51on record and scanned surface point cloud16. Then, the method includes updating the updated patient profile59based on a result of the comparison. By using comparisons, this may be useful in cases where only a portion of the patient profile is scanned and updated.

Other Features

The mobile communication device61and the processor therein may perform the majority, or all, of the steps of the method100described above. However, it is to be appreciated that the mobile communication device61can send outputs to other devices via one or more communications networks (including wireless networks) to another device. For example, the user may wish to use an alternative graphical user interface41(such as a larger display screen in the operator). To that end, the contents displayed at the screen of the mobile communication device61may be mirrored to that other display. In other examples, the output may include sending (including) streaming, images49and modified images55to other devices. In yet other examples, data associated with the patient profile, or updated patient profile can be sent and stored at a storage device in communication with the mobile communication device61. This includes storing the data on cloud based storage.

Variations

In some examples the camera-based tracking system may utilise one or more cameras (which may include the second camera43) at a mobile communication device. For example, a forward facing camera of the mobile communication device is configured to locate, in the field of view, the tissue marker21at the anatomical feature9. The same camera, or another camera, may be used to identify pointer markers19at the pointer device. This information can be used to determine information of the relative orientation and position of the pointer device11and, ultimately, the depth sensor13.

In some examples, the tissue scanning system may include a kit comprising: the pointer device11and the camera-based tracking system3. The pointer device11is configured to receive a depth sensor (13) configured to capture surface point cloud measurements of surface(s) associated with an anatomical feature. For example, the pointer device11is configured to receive a separately supplied mobile communication device having a depth sensor.

In other examples, the tissue scanning system may include a kit comprising: the camera-based tracking system3and the depth sensor13. The depth sensor13is located at a pointer device. The pointer device11configured to receive the mobile communication device and to aid directing and locating the depth sensor13may be supplied separately.

Processing Device

The processing device1013, as illustrated inFIG.11, includes a processor1102connected to a program memory1104, a data memory1106, a communication port1108and a user port1110. The program memory1104is a non-transitory computer readable medium, such as a hard drive, a solid state disk or CD-ROM. Software, that is, an executable program stored on program memory1104causes the processor1102to perform the method100inFIG.4.

The processor1102may receive determined distances and orientation and position data and store them in data store1106, such as on RAM or a processor register. The depth sensor data and other information may be received by the processor1102from data memory106, the communications port1108, the input port1011, and/or the user port1110.

The processor1102is connected, via the user port1110, to a display1112to show visual representations1114the images, or modified images from cameras, and/or the surface point cloud. The processor1102may also send the surface point cloud, as output signals via communication port1108to an output port1012.

Although communications port1108and user port1110are shown as distinct entities, it is to be understood that any kind of data port may be used to receive data, such as a network connection, a memory interface, a pin of the chip package of processor1102, or logical ports, such as IP sockets or parameters of functions stored on program memory1104and executed by processor1102. These parameters may be stored on data memory1106and may be handled by-value or by-reference, that is, as a pointer, in the source code.

The processor1102may receive data through all these interfaces, which includes memory access of volatile memory, such as cache or RAM, or non-volatile memory, such as an optical disk drive, hard disk drive, storage server or cloud storage. The processing device13may further be implemented within a cloud computing environment, such as a managed group of interconnected servers hosting a dynamic number of virtual machines.