Patent ID: 12220175

DETAILED DESCRIPTION

Some embodiments of this disclosure, illustrating all its features, will now be discussed in detail. The words “comprising,” “having,” “containing,” and “including,” and other forms thereof, are intended to be equivalent in meaning and be open ended in that an item or items following any one of these words is not meant to be an exhaustive listing of such item or items, or meant to be limited to only the listed item or items.

It must also be noted that as used herein and in the appended claims, the singular forms “a,” “an,” and “the” include plural references unless the context clearly dictates otherwise. Although any systems and methods similar or equivalent to those described herein can be used in the practice or testing of embodiments of the present disclosure, the preferred, systems and methods are now described.

Embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings in which like numerals represent like elements throughout the several figures, and in which example embodiments are shown. Embodiments of the claims may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. The examples set forth herein are non-limiting examples and are merely examples among other possible examples. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are explicitly contemplated herein.

FIG.1Aillustrates network connection diagram100of a system102for providing surgical assistance during a surgical procedure, according to an embodiment. The system102may be connected to a communication network104. The communication network104may further be connected with a healthcare facility106for facilitating data transfer between the system102and the healthcare facility106.

The communication network104may be a wired and/or a wireless network. The communication network104, if wireless, may be implemented using communication techniques such as Visible Light Communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), Long Term Evolution (LTE), Wireless Local Area Network (WLAN), Infrared (IR) communication, Public Switched Telephone Network (PSTN), Radio waves, and other communication techniques known in the art.

The healthcare facility106may comprise a real-time health record unit108and a group of databases for storing different information required during a surgical procedure. The group of databases may comprise a surgical outcomes database110, surgical options database112, surgical annotations database114, and a correlation database116. Different databases are used in present case; however, a single database may also be used for storing the data. Usage of the different databases may also allow segregated storage of different data and may thus reduce time to access required data.

In one embodiment, the surgical outcomes database110may be configured to store results of previous surgeries for several patients. The results may be stored in a structured manner. In one case, the results may be organized based on types of surgical procedures, followed by patients' medical histories, followed by pre-plan of surgical procedures, and before and after annotations of surgeons for the surgical procedures.

The surgical outcomes database110may further store images of the patients. The images may be any of camera images, Magnetic Resonance Imaging (MRI) images, and X-Ray images. The surgical outcomes database110may also comprise actual VR training data used by a surgeon, for pre-planning of a surgical procedure. The surgical outcomes database110may also include data by a Three-Dimensional (3D) camera during the actual surgical procedure and all related monitor data, which is displayed on a screen during the surgical procedure. The surgical outcomes database110may also include unexpected or adverse events occurring in a time-sequence of pre-plan and actual results. The surgical outcomes database110may also include annotations provided by previous surgeons.

In at least one embodiment, the surgical options database112may be implemented as a query on the surgical outcomes database110. The surgical options database112may comprise data organized in a surgical options structure. In at least one scenario facilitated by the embodiment, for a particular surgical procedure performed on a particular type of patient, the surgical options database112may store information related to each event step, such as details of the events that went well, occurrence of adverse events, frequency of the adverse events, annotations made by a surgeon, and usefulness of the annotations.

In at least one embodiment, the surgical annotations database114may be implemented as a query on the surgical outcomes database110. The surgical annotations database114may extract a surgeon's annotations from the surgical outcomes database110and may organize the surgeon's annotations. The surgical annotations database114may also input a current surgeon's annotations, made either during training and preplanning or during an actual surgical procedure by the current surgeon.

In at least one embodiment, the correlation database116may be implemented as a real-time query of all databases (surgical outcomes database110, surgical options database112, and surgical annotations database114) to find correlations based upon a current surgical procedure. For example, an adverse event such as a patient's blood pressure unexpectedly dropping may occur during a surgical procedure. Accordingly, the correlation database116may implement a search of all databases for data relating to unexpected drop in blood pressure during a similar surgical procedure. In accordance with this non-limiting example, the data stored in the correlation database116may include data indicating the percentage of patients who incur an increase in blood pressure during such a surgical procedure, as well as data correlating such occurrence of increased blood pressure to the respective patient's medical history. For example, the past history may indicate that more than 75% patients experiencing increased blood pressure were diabetic patients and that a particular injection (e.g., Hydralazine used to relax and dilate blood vessels, resulting in lowered blood pressure) proved to be effective for 90% of such patients. The correlation database116may also include a surgeon's annotations organized by frequency of usage of the database. For example, 70% of the surgeons may have annotated to provide said injection before proceeding to a next step.

FIG.1Billustrates a network connection diagram1000of a system1102for creating a surgical plan for use in a surgical procedure, according to another embodiment. The system1102may be connected to a communication network104, similar to that ofFIG.1A. The communication network104may further be connected with the healthcare facility106for facilitating data transfer between the system1102and the healthcare facility106.

The communication network104may be the same or similar to that ofFIG.1A.

The healthcare facility106may include a real-time health record unit108and a group of databases for storing different information required during the surgical procedure. The group of databases may include a surgical annotation database1110, an image abnormality database1112, and a surgical path database1114. As withFIG.1A, different databases are used in present case; however, a single database may also be used for storing the data. Usage of the different databases may also allow segregated storage of different data and may thus reduce time to access required data.

The real-time health record unit108may be configured to store medical data for a plurality of patients in real-time. The data may correspond to a respective patient's medical imaging data, diagnostic data, as well as comprehensive medical history data. The stored medical data may include medical images of an affected body part of a respective patient, as well as comprehensive medical records, including medical history, previous test results, and notes of surgeons/doctors or medical advisors.

In at least one embodiment, the surgical annotation database1110may be configured to accept annotations provided by a respective surgeon, either during surgical planning or in real-time during a surgical procedure. The surgeon may add the annotations at any time during surgical pre-operative planning, and the annotations may be stored in the surgical annotation database1110. The surgeon may add the annotations either using the system102or a user device116. A smart phone is shown as the user device1116in FIG.1B, as an example, used for displaying information related to the surgical procedures. However, user device1116may be implemented as any other device comprising a Graphical User Interface (GUI), for example, a laptop, a desktop, a tablet, a phablet, or other such devices known in the art.

In at least one embodiment, the image abnormality database1112may be configured to store data related to irregularities and/or abnormalities in the medical images of the affected body part of a respective patient. The medical images may include a plurality of images captured using an imaging system118, utilizing known medical imaging techniques, such as Magnetic Resonance Imaging (MRI), Computerized Tomography (CT), X-rays, etc. The image abnormality database1112may store data related to the abnormalities identified in the medical images during the pre-planning stage of a surgical procedure. A non-limiting example of such an irregularity and/or abnormality stored in image abnormality database1112may relate to a ruptured Achilles tendon of a right ankle of a respective patient.

In at least one embodiment, the surgical path database1114may be configured to store the plurality of surgical paths that may be followed for respective surgical procedures.

FIG.2Ashows a block diagram showing different components of system102as depicted inFIG.1A. The system102comprises a processor202, interface(s)204, and memory206. The processor202may execute an algorithm stored in the memory206for utilizing artificial intelligence to provide surgical assistance. The processor202may also be configured to decode and execute any instructions received from one or more other electronic devices or server(s). The processor202may include one or more general purpose processors (e.g., INTEL® or Advanced Micro Devices® (AMD) microprocessors) and/or one or more special purpose processors (e.g., digital signal processors or Xilinx® System On Chip (SOC) Field Programmable Gate Array (FPGA) processor). The processor202may be configured to execute one or more computer-readable program instructions, such as program instructions to carry out any of the functions described in this description.

The interface(s)204may help a user to interact with the system102. The interface(s)204of the system102may either accept an input from the user or provide an output to the user, or may perform both the actions. The interface(s)204may either be a Command Line Interface (CLI), Graphical User Interface (GUI), or a voice interface.

The memory206may include, but is not limited to, fixed (hard) drives, magnetic tape, floppy diskettes, optical disks, Compact Disc Read-Only Memories (CD-ROMs), and magneto-optical disks, semiconductor memories, such as ROMs, Random Access Memories (RAMs), Programmable Read-Only Memories (PROMs), Erasable PROMs (EPROMs), Electrically Erasable PROMs (EEPROMs), flash memory, magnetic or optical cards, or other type of media/machine-readable medium suitable for storing electronic instructions.

The memory206may comprise an Artificial Intelligence (AI) surgical assistant208, implemented as a program. The AI surgical assistant208may comprise three modules i.e., procedure planning module210, virtual reality surgical practice module212, and augmented reality assistance module214.

FIG.2Bshows a block diagram showing different components of the system1102, as depicted inFIG.1B, according to an embodiment. The system1102includes a processor202, interface(s)204, and a memory206, similar to that ofFIG.2A. Further to the description thereof pertaining toFIG.2A, the processor202may execute an algorithm stored in the memory206for creating the surgical plan for use in the surgical procedure. The processor202may also be configured to decode and execute any instructions received from one or more other electronic devices or server(s). Again, the processor202may include one or more general purpose processors (e.g., INTEL® or Advanced Micro Devices® (AMD) microprocessors) and/or one or more special purpose processors (e.g., digital signal processors or Xilinx® System On Chip (SOC) Field Programmable Gate Array (FPGA) processor). The processor202may be configured to execute one or more computer-readable program instructions, such as program instructions to carry out any of the functions described in this description.

Further to the description thereof pertaining toFIG.2A, the interface(s)204may facilitate a surgeon's interaction with the system102. The interface(s)204of the system102may accept input from the surgeon and provide an output to the surgeon. The interface(s)204may either be a Command Line Interface (CLI), Graphical User Interface (GUI), or a voice interface.

Again, similar to the description thereof pertaining toFIG.2A, the memory206may include, but is not limited to, fixed (hard) drives, magnetic tape, floppy diskettes, optical disks, Compact Disc Read-Only Memories (CD-ROMs), and magneto-optical disks, semiconductor memories, such as ROMs, Random Access Memories (RAMs), Programmable Read-Only Memories (PROMs), Erasable PROMs (EPROMs), Electrically Erasable PROMs (EEPROMs), flash memory, magnetic or optical cards, or other type of media/machine-readable medium suitable for storing electronic instructions.

In one embodiment, the memory206may include a pre-operative planning image processing module1208. The pre-operative planning image processing module1208includes a surgical planning module1210. Further, the surgical planning module1210includes a surgical annotation module212.

Functioning of the procedure planning module210, as shown inFIG.2A, will now be explained with reference to flowchart300shown inFIGS.3A and3B. One skilled in the art will appreciate that, for this and other processes and methods disclosed herein, the functions performed in the processes and methods may be implemented in differing order. Furthermore, the outlined steps and operations are only provided as examples, and some of the steps and operations may be optional, combined into fewer steps and operations, or expanded into additional steps and operations without detracting from the essence of the disclosed embodiments.

A user may log into the procedure planning module210. The term “user” may refer to a surgeon, and may henceforth be used interchangeably in the description. Optionally, the logging in procedure may be accompanied with authentication of the user. Upon logging in, the procedure planning module210may allow the user to identify a subject patient and receive identification information for the subject patient, at step302. The subject patient may refer to a patient upon whom the surgeon will operate. The procedure planning module210may be utilized to retrieve records pertaining to the subject patient from Electronic Health Records (EHR) stored in real-time health record unit108.

Thus, step304includes retrieving records of the subject patient, and identifying the surgery to be performed for the subject patient. As an example of a surgery that may be performed, the subject patient may need an Anterior Cruciate Ligament (ACL) repair of his left knee. All possible surgical procedures for repairing of the knee may thus be retrieved from the surgical options database112. Further to the example, the surgical procedure for repairing the knee may include a cadaver graft, patellar tendon graft, and different methods of performing such surgical procedures. The surgical procedures may be performed though arthroscopy, open, and/or other different surgical paths of going in on one side of the knee and another side of the knee. Also retrieved from surgical options database112at step306, along with the possible surgical procedures, are attributes of other patients operated upon using such surgical procedures as well as details of adverse events that have occurred or may occur during such surgical procedures.

At step308, the procedure planning module210may create Three-dimensional (3D) models of both the subject patient's affected area as well as that for the other patients whose comparative data was retrieved. At step310, the 3D model of the subject patient may be compared to the 3D models of the other patients to determine similarity scores. At step312, data belonging to the other patients may be filtered based on a threshold value of the determined similarity scores. As a non-limiting example, the threshold similarity score value may be set as 25%, thus, data belonging to the other patients, present in the top 25% of the similarity scores may be kept for further analysis.

Step314includes determining best fit curves for attributes of the remaining ones of the comparative patients. For example, Body Mass Index (BMI) may be used as an attribute of the remaining ones of the comparative patients. Step314may also include determining correlation coefficients for experiencing adverse events from a surgical procedure. For example, blood loss during surgery may be one such adverse event. Thereafter, step316includes storing best fit curves for each attribute in the correlation database116.

Step318includes filtering the attributes based on a threshold value of correlation coefficient. As a non-limiting example, the attributes having correlation coefficient values greater than the threshold value of correlation coefficient may be filtered for further use. Thus, if the threshold value of correlation coefficient is set at 0.6, the attributes having correlation coefficient values less than 0.6 may be discarded. Step318also includes identifying relevant adverse events based on the filtering. The attributes in which the subject patient's attribute measurement lies outside of a standard deviation from a best fit curve may also be discarded.

Step320includes scoring a severity of the relevant adverse event may be scored. As a non-limiting example, the score may range from 1 to 10, by which 1 may indicate a lowest level of severity and 10 may indicate a highest level of severity. The severity scores for adverse events may be determined based on predefined rules, as severity of any adverse event is well known to the surgeons.

At step322, adverse event scores may be determined for each surgical procedure previously identified at step306by multiplying a correlation coefficient of an attribute, present in common with the subject patient, by a severity score of the adverse event. For example, the correlation coefficient between BMI greater than 35 and bleeding during patellar tendon graft may be 0.8, and the severity score may have a value of 3. The severity of the adverse event during such case may be scored as “0.8*3=2.4.” In accordance with a non-limiting example, the score of 2.4 may be summed with scores of all other adverse events. It is evident that all such scores are pre-determined for adverse events, and thus the adverse events occurring in a real surgical procedure may have corollary scores.

FIG.4shows a non-limiting example of information related to the surgical procedures being displayed on a Graphical User Interface (GUI) of a user device120such as the smart phone shown as the user device120inFIG.1. However, the user device120used to display information related to the surgical procedures may be any other device comprising a GUI, for example, a laptop, a desktop, a tablet, a phablet, or other such devices known in the art.

Step324of flowchart300continues, with reference toFIG.4, by ranking the surgical procedures based on the respective adverse event scores. The surgical procedures may be displayed on the GUI with the adverse events being listed from top to bottom in ascending order of adverse event scores. A user may then select one surgical procedure from amongst the available surgical procedures, and step326includes a user selection of the surgical procedure being accepted through the GUI.

In one embodiment, using the GUI, the user may view plurality of information in real-time, during the surgical procedure. The plurality of information may include the possible surgical procedures, currently adopted surgical procedure, adverse events ranked on the basis of their scores, annotations related to the adverse events, associated files history for an adverse event (for example, Patellar chip in present case), procedure to be followed at a current step to move ahead, and a real-time video of each step performed. Further, all images displayed on the GUI may be overlaid, scaled, and annotated by the user. Thus, step328includes sending to Virtual Reality (VR) surgical practice module212the data related to the surgical procedure and adverse events related to the surgical procedure.

The VR surgical practice module212may allow a user to work on VR surgical practice equipment122, which may be connected to the system102, and may allow simulation of surgical procedures. The simulation may include displaying one or more virtual organs to be operated upon by the user. A vital organ may comprise multiple elements and each element may have neighbouring elements. A plurality of tensioned connections may connect the neighbouring elements with the vital organ, such that force applied on one element propagates via respective neighbouring elements and thus providing a distributed reaction over the vital organ.

The VR surgical practice equipment122may also include a physical manipulation device for manipulation by the user and a tracking arrangement for tracking the physical manipulation device and translating motion of the physical manipulation device into application of forces onto said virtual organ. The VR surgical practice equipment122facilitates simulation of moving, cutting, suturing, coagulations, and other aspects of surgical procedures for different organs. Thus, the VR surgical practice equipment122may facilitate surgical practice for a user.

FIG.5illustrates a flowchart500for a method executed by VR surgical practice module212, as shown inFIG.2A. A user may log into the system102using VR surgical practice equipment122. At step502, the VR surgical practice module212may receive data related to the surgical procedure and adverse events related to the surgical procedure. The user may start VR practice session using the augmented reality display118and the VR surgical practice equipment122, based on the data received from the procedure planning module210.

Step504, during the VR practice session, includes the data pertaining to possible adverse events being displayed to the user in synchronism with the same moments during the practice session at which the adverse events actually occurred in previous surgical procedures. At step506, the user may provide annotations related to different steps of surgical procedure training. The annotations may be made in various formats including, but not limited to, text, audio notes, instructions to pull up specific medical data from patient's Electronic Health Records (EHR), and Audio-Video files related to the procedure. At step508, the annotations may be stored in the surgical annotations database114. At step510, the medical imaging data, annotations, and other data related to the VR practice session may be sent to the Augmented Reality (AR) assistance module214.

FIG.6shows a flowchart600for a method executed by AR assistance module214inFIG.2A. The AR assistance module214may support the user while performing the real surgical procedure on the subject patient. At step602, the AR assistance module214may receive the medical imaging data, annotations, and other data related to the VR practice session. At step604, a position of the patient may be identified relative to an AR display118worn by the user. Based on the identified position, at step606, user requested data may be overlaid on the portion.

In at least one non-limiting example, at step608, the user may start performing the surgical procedure on the patient, and the annotations may be presented to the user at designated times. By at least one example embodiment, time stamps may be used to determine an appropriate time to present the annotations. For example, an annotation may be set to be presented five minutes into the surgical procedure.

Alternatively, an appropriate time to present the annotations may be determined based upon initiation of a particular step of a surgical procedure. For example, an annotation may be set to be presented during initiation of a third step of the surgical procedure. Each step of the surgical procedure may be monitored by the AR display118, and thus the system102may present the annotation to the user during a predetermined step.

The annotations may help the surgeon by storing important details related to any step of the surgical procedure, which may then be recalled during subsequent procedures. Such details may be presented to the surgeon, at designated moments, for reminding about taking due care, as annotated. Thus, the surgeons may be assisted by their own person input defined during a training session prior to the actual surgical procedure. This helps in improving accuracy of the surgeons by allowing them to utilize every minute yet essential detail, thus reducing occurrence of adverse events experienced by the patient, during a surgical procedure.

FIG.7shows a flowchart700for a method of providing surgical assistance during a surgical procedure, according to an embodiment of system102as shown inFIG.2A.

In flowchart700, each block may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing a respective specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur in a different order than that shown inFIG.7. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may be executed in reverse order, depending upon the functionality involved. Any process descriptions or blocks in flowcharts should be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps in the process, and alternate implementations are included within the scope of the example embodiments in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved. In addition, the process descriptions or blocks in flow charts should be understood as representing decisions made by a hardware structure such as a state machine. The flowchart700starts at the step702and proceeds to step712.

At step702, a medical condition of a patient may be determined. Based on the medical condition, surgical options may be determined for the patient. In one embodiment, the surgical options may be determined by processor202of the system102, as shown inFIG.2A, in communication with the surgical options database112.

At step704, an optimal surgical option may be determined from among the stored surgical options based on outcomes of the surgical options in other patients. In one non-limiting example, the optimal surgical option may be determined by the processor202.

At step706, adverse events related to the surgical procedure may be identified. In one non-limiting example, the adverse events may be identified by the processor202.

At step708, a Three-Dimensional (3D) model of the patient may be created using images of the patient gathered from one or more of different sources such as a digital camera, X-ray device, and Magnetic Resonance Imaging (MRI) device. In one non-limiting example, the 3D model may be created by the processor202.

At step710, a Virtual Reality (VR) simulation may be created for training a surgeon using the 3D model. During the training, annotations may be received from the surgeon and stored in the surgical annotations database114. The annotations may later be presented to the surgeon at designated moments during a real surgical procedure. The designated moments may be set based on time-stamps or steps of an actual surgical procedure. In one non-limiting example, the VR simulation may be created and the annotations may be accepted by the processor202.

At step712, the annotations may be presented to the surgeon at the designated moments of an actual surgical procedure. The annotations provide surgical assistance to the surgeon by providing reminders of certain important activities and details to be implemented at the designated moments. In one non-limiting example, the annotations may be visually presented by a user interface.

Functioning of the surgical planning module1210, as shown inFIG.2B, will now be explained with reference to flowchart800shown inFIG.8, with reference to system1102ofFIGS.1B and2B. The functions performed in the processes and methods may be implemented in differing order than those presently depicted and described. Furthermore, the outlined steps and operations are only provided as examples, and some of the steps and operations may be optional, combined into fewer steps and operations, or expanded into additional steps and operations without detracting from the essence of the disclosed embodiments.

At first, the surgeon may need to log-in to the surgical planning module1210. The surgeon may log-in using his credentials, i.e., a user name and a password, using, for example, the user device1116. Upon log-in, the surgical planning module1210may facilitate the surgeon identifying a subject patient and receiving identification information for the subject patient, at step802. The surgical planning module1210may be utilized to retrieve medical data pertaining to the subject patient from Electronic Health Records (EHR) stored in the real-time health record unit1108. The medical data may include medical images of an affected body part of the subject patient, as captured by the imaging system1118using one or more medical imaging techniques, such as Magnetic Resonance Imaging (MRI), Computerized Tomography (CT), and X-Ray.

At step804, a recommendation for surgery for the subject patient may be made, and the diagnosis of the subject patient may be retrieved based at least on the subject patient's medical data. At step806, any abnormalities present in the medical images may be identified; and at step808, the identified abnormalities may be correlated with the diagnosis of the subject patient.

At decision block810, the surgical planning module1210, shown inFIG.2B, may be utilized to determine whether the identified abnormalities and the subject patient's diagnosis are consistent with each other. At step812(“no”), if the identified abnormalities are not consistent with the subject patient's diagnosis, the surgical planning module1210may be utilized to alert the surgeon that a diagnosis is needed for the subject patient. At step814(“yes”), In case the identified abnormalities are determined to be consistent with the subject patient's diagnosis, the surgical planning module1210may be utilized to segment the medical images. Thereafter, at step816, the surgical planning module1210may be utilized to run the surgical annotation module1212.

Functioning of the surgical annotation module1212, as shown inFIG.2B, will now be explained with reference to the flowchart900shown inFIG.9, with reference to system1102ofFIGS.1B and2B. Once again, the functions performed in the processes and methods may be implemented in differing order. Furthermore, the outlined steps and operations are only provided as examples, and some of the steps and operations may be optional, combined into fewer steps and operations, or expanded into additional steps and operations without detracting from the essence of the disclosed embodiments.

At step902, the surgical annotation module1212may receive identification information for the subject patient, the recommended surgery for the subject patient, the subject patient's diagnosis, and the image segments, from the surgical planning module1210, shown inFIG.2B. At step904, the surgical annotation module1212may be utilized to display the image segments for the surgeon. The image segments may be displayed on the GUI of the user device1116. For example, as shown inFIG.10A, segment of an affected body part, i.e., a ruptured Achilles tendon of a right ankle of the subject patient, is displayed to the surgeon.

At step906, the surgical paths may be retrieved from the surgical path database1114based on criteria set by the surgeon, including, but not limited to, frequency of success of the surgical paths, time of operation for the surgical paths, etc. Thereafter, the surgical paths may be overlaid on the image segments of the affected body part of the subject patient.

At step908, the surgical paths that are overlaid on the image segments may be displayed for the surgeon using the GUI of the user device1116. For example, as shown inFIG.10B, the surgical paths overlaid on an image segment of the affected body part, i.e., the ruptured Achilles tendon of the right ankle of the subject patient, is displayed in different colors, such as red, green, yellow, etc.

At step910, a surgical path may be selected from the surgical paths, and a surgical step belonging to the surgical path may be selected based on the surgeon's input. For example, as shown inFIG.10B, the surgical paths may be displayed to the surgeon for selecting a surgical path, i.e., a route. Further, as shown inFIG.10C, the surgical steps may be displayed to the surgeon to select a surgical step in order to repair the right knee of the subject patient.

In one non-limiting example, the surgeon may select the surgical step based at least on criteria, including but not limited to average time of the surgical step and worst-case risks of the surgical step. In one non-limiting example, as shown inFIG.10D, the surgical annotation module1212may be utilized to display the selected surgical step, i.e., surgical step4for the surgeon on the GUI of the user device1116.

At step912, an image segment may be selected from the image segments, based on the surgeon's input. For example, as shown inFIG.10D, the surgical annotation module1212may be utilized to enable the surgeon to select an image segment using the user device1116. In a non-limiting example, the surgical annotation module1212may be utilized to enable the surgeon to include or exclude the image segments. For example, as shown inFIG.10E, the surgical annotation module1212may be utilized to enable the surgeon to include or exclude the image segments related to the right ankle.

At step914, annotations related to the surgical step may be received from the surgeon via, e.g., the GUI of the user device1116, which may provide input options for the user. For example, as shown inFIG.10F, the surgeon may provide an annotation, such as “to move a tibial nerve medially at the surgical step4to avoid contact with the tibial nerve” in order to prevent damage to the tibial nerve during execution of surgical step4.

In another example, the surgeon may want to add an annotation at a particular surgical step regarding a particular concern pertaining to a vascular tissue in the right knee. In that example scenario, the surgeon may want to include the vascular tissue and nerve tissue but exclude specific tendons that may obscure a view of the vascular tissue and the nerve tissue. Other examples of the annotations may include requesting a 30% reduction in volume at a particular surgical step, requesting a surgical implant being added at the particular surgical step, etc.

At step916, the image segment along with the annotations may be stored in the surgical annotation database1110.

FIG.11a flowchart11100showing a method for creating the surgical plan for use in the surgical procedure, according to an embodiment, explained in conjunction with the elements disclosed inFIGS.1B,2B, and10A-10F.

In flowchart11100, each block may represent a module, segment, or portion of code, which includes one or more executable instructions for implementing a respective specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur in a different order than that shown inFIG.11. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may be executed in the reverse order, depending upon the functionality involved. Any process descriptions or blocks in flowcharts should be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps in the process, and alternate implementations are included within the scope of the example embodiments in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved. In addition, the process descriptions or blocks in flow charts should be understood as representing decisions made by a hardware structure, such as a state machine. The flowchart11100starts at step11102and ends at step11116.

At step11102, a recommended surgery for a subject patient may be identified based at least on medical data of the subject patient, which may include one or more medical images of an affected body part of the subject patient.

At step11104, one or more abnormalities present in the one or more medical images of the subject patient may be identified.

At step11106, the one or more medical images may be segmented into one or more image segments.

At step11108, surgical paths may be retrieved for addressing the surgical need of the subject patient from the surgical path database1114, shown inFIG.1B.

At step11110, the surgical paths overlaid on the one or more image segments may be displayed for the surgeon on the user device1116.

At step11112, a surgical path may be selected from the surgical paths, a surgical step belonging to the surgical path may be selected, and an image segment from may be selected the one or more image segments, based on the surgeon's input.

At step11114, one or more annotations related to the surgical step may be received from the surgeon.

At step11116, the image segment may be stored along with the one or more annotations, in the surgical annotation database1110, thereby creating the surgical plan for use in the surgical procedure. Thereafter, the surgical plan may be displayed on an operating room display1120during the surgical procedure.

In an illustrative embodiment, any of the operations, processes, etc., described herein may be implemented as computer-readable instructions stored on a computer-readable medium. The computer-readable instructions can be executed by a processor of a mobile unit, a network element, and/or any other computing device.

There is little distinction left between hardware and software implementations of aspects of systems; the use of hardware or software is generally (but not always, in that in certain contexts the choice between hardware and software can become significant) a design choice representing cost vs. efficiency tradeoffs. There are various vehicles by which processes and/or systems and/or other technologies described herein can be effected, e.g., hardware, software, and/or firmware, and that the preferred vehicle will vary with the context in which the processes and/or systems and/or other technologies are deployed. For example, if an implementer determines that speed and accuracy are paramount, the implementer may opt for a mainly hardware and/or firmware vehicle; if flexibility is paramount, the implementer may opt for a mainly software implementation; or, yet again alternatively, the implementer may opt for some combination of hardware, software, and/or firmware.

The foregoing detailed description has set forth various embodiments of the devices and/or processes via the use of block diagrams, flowcharts, and/or examples. Insofar as such block diagrams, flowcharts, and/or examples contain one or more functions and/or operations, it will be understood by those within the art that each function and/or operation within such block diagrams, flowcharts, or examples can be implemented, individually and/or collectively, by a wide range of hardware, software, firmware, or virtually any combination thereof. In one embodiment, several portions of the subject matter described herein may be implemented via Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), digital signal processors (DSPs), or other integrated formats. However, those skilled in the art will recognize that some aspects of the embodiments disclosed herein, in whole or in part, can be equivalently implemented in integrated circuits, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or as virtually any combination thereof, and that designing the circuitry and/or writing the code for the software and or firmware would be well within the skill of one of skill in the art in light of this disclosure. In addition, those skilled in the art will appreciate that the mechanisms of the subject matter described herein are capable of being distributed as a program product in a variety of forms, and that an illustrative embodiment of the subject matter described herein applies regardless of the particular type of signal bearing medium used to actually carry out the distribution. Examples of a signal bearing medium include, but are not limited to, the following: a recordable type medium such as a floppy disk, a hard disk drive, a CD, a DVD, a digital tape, a computer memory, etc.; and a transmission type medium such as a digital and/or an analog communication medium (e.g., a fiber optic cable, a waveguide, a wired communications link, a wireless communication link, etc.).

The embodiments, features, systems, devices, materials, methods and techniques described herein may, in some embodiments, be similar to any one or more of the embodiments, features, systems, devices, materials, methods and techniques described in U.S. application Ser. No. 16/012,464, filed Jun. 19, 2018, titled SURGERY PLANNING, which is incorporated by reference in its entirety. In addition, the embodiments, features, systems, devices, materials, methods, and techniques described herein may, in certain embodiments, be applied to or used in connection with any one or more of the embodiments, features, systems, devices, or other matter. All of the patents and applications are incorporated by reference in their entireties.

Those skilled in the art will recognize that it is common within the art to describe devices and/or processes in the fashion set forth herein, and thereafter use engineering practices to integrate such described devices and/or processes into data processing systems. That is, at least a portion of the devices and/or processes described herein can be integrated into a data processing system via a reasonable amount of experimentation. Those having skill in the art will recognize that a typical data processing system generally includes one or more of a system unit housing, a video display device, a memory such as volatile and non-volatile memory, processors such as microprocessors and digital signal processors, computational entities such as operating systems, drivers, graphical user interfaces, and applications programs, one or more interaction devices, such as a touch pad or screen, and/or control systems including feedback loops and control motors (e.g., feedback for sensing position and/or velocity; control motors for moving and/or adjusting components and/or quantities). A typical data processing system may be implemented utilizing any suitable commercially available components, such as those typically found in data computing/communication and/or network computing/communication systems.

The herein described subject matter sometimes illustrates different components contained within, or connected with, different other components. It is to be understood that such depicted architectures are merely examples, and that in fact many other architectures can be implemented which achieve the same functionality. In a conceptual sense, any arrangement of components to achieve the same functionality is effectively “associated” such that the desired functionality is achieved. Hence, any two components herein combined to achieve a particular functionality can be seen as “associated with” each other such that the desired functionality is achieved, irrespective of architectures or intermedial components. Likewise, any two components so associated can also be viewed as being “operably connected”, or “operably coupled”, to each other to achieve the desired functionality, and any two components capable of being so associated can also be viewed as being “operably couplable”, to each other to achieve the desired functionality. Specific examples of operably couplable include but are not limited to physically mateable and/or physically interacting components and/or wirelessly interactable and/or wirelessly interacting components and/or logically interacting and/or logically interactable components.

From the foregoing, it will be appreciated that various embodiments of the present disclosure have been described herein for purposes of illustration, and that various modifications may be made without departing from the scope and spirit of the present disclosure. Accordingly, the various embodiments disclosed herein are not intended to be limiting.