Patent ID: 12207861

DETAILED DESCRIPTION

The detailed description set forth below is intended as a description of various configurations of the subject technology and is not intended to represent the only configurations in which the subject technology may be practiced. The appended drawings are incorporated herein and constitute a part of the detailed description. The detailed description includes specific details for the purpose of providing a thorough understanding of the subject technology. However, the subject technology is not limited to the specific details set forth herein and may be practiced without these specific details. In some instances, structures and components are shown in block diagram form in order to avoid obscuring the concepts of the subject technology.

Terminology

Throughout this patent disclosure, the terms “tool presence/absence detection model,” “tool presence/absence detector” “tool detection model,” “tool detector,” “ML tool-detection model,” and “ML tool detector” are used interchangeably to mean a deep-learning model constructed to predict whether a surgical tool, such as an energy tool is present or absent in a given surgical video frame and provide a confidence score to each prediction.

Overview

Embodiments described herein provide various techniques and systems for building machine-learning (ML)/deep-learning (DL) surgical tool detection models for processing surgical videos and predicting whether a surgical tool is present or absent in each video frame of a surgical video. In particular, the ML surgical tool detection models described in this disclosure include an energy tool presence/absence detection model which is trained and used to process a real-time surgical video of a laparoscopic or robotic surgery that uses an energy tool to cut and cauterize/seal tissues, and generate a real-time prediction for each video frame regarding whether the energy tool is present or absent in each video frame.

In various embodiments, the disclosed energy tool presence/absence detection model is built to detect multiple models and versions of a given type of the energy tool instead of a particular model or version of the given type of the energy tool. In some embodiments, a disclosed energy tool presence/absence detection model is built for various models and versions of an ultrasonic energy tool by one manufacturer, such as Harmonic™ scalpels by Ethicon™. However, a separate energy tool presence/absence detection model may be built for various models and versions of an ultrasonic energy tool by another manufacturer, such as Sonicision™ scalpels by Covidien™. Another disclosed energy tool presence/absence detection model can be built for various models and versions of a bipolar energy tool by one manufacturer, such as Enseal™ tissue sealers by Ethicon™. However, a separate energy tool presence/absence detection model may be built for various models and versions of a bipolar energy tool by another manufacturer, such as a Ligasure™ tissue sealers by Covidien™. The disclosed energy tool presence/absence detection model is further integrated with an energy-tool unsafe-use detection process to detect unsafe events associated with the energy tool usage during a surgery and to prevent injuries from the energy tool.

In various embodiments, the disclosed ML surgical tool detection model building techniques and systems can build a robust surgical tool detection model by first obtaining an initially trained tool detection model based on an initial training dataset, wherein the initial training dataset can include collected images of the surgical tool in different types and models, instead of just a particular type/model of the energy tool. This will allow building an initially trained tool detection model that can cover all types and models of the energy tool that are potentially in use everywhere. To increase training data diversity to cover more real-world scenarios, multiple data augmentation techniques including random color distortion and geometrical transformation can be carefully applied to the initial training dataset, while keeping the original labels of the images. The extended training dataset including both the initial training dataset and the augmented versions of the initial training images can be used to train an initial ML tool detection model and obtain the initially trained tool detection model.

The disclosed ML surgical tool detection model building techniques and systems also include mechanisms to update or further train the initially trained tool detection model based on additional training images related to the energy tool and energy tool usage. In some embodiments, the disclosed ML surgical tool detection model building techniques and systems update the initially trained tool detection model through the active learning, which can involve a training loop. This includes using the initially trained tool detection model to separate the additional training images into high-confidence-level images that are similar to images in the initial training dataset; and low-confidence-level images that are significantly different from images in the initial training dataset. The identified low-confidence-level images are then annotated by skilled annotators. Next, the disclosed ML surgical tool detection model building techniques and systems trains/updates the initially trained tool detection model using the labeled low-confidence-level images to update. The updated surgical tool detection model can have improved accuracy and precision than the initially trained tool detection model, and can also detect more diverse and more complex surgical scenarios related to the surgical tool than the initially trained tool detection model.

Using the disclosed surgical tool detection model updating techniques, the initially trained tool detection model is updated on a significantly smaller but information-rich set of additional training images, which makes the model training/updating process much more efficient than using both high-confidence-level images and low-confidence-level images. Moreover, the convergence time during the model optimization is greatly reduced comparing with a training process without using active learning when the same validation dataset is used. Note that the disclosed ML surgical tool detection model building techniques and systems can be used not only to build energy tool presence/absence detection models, but also to build surgical tool detection models for surgical tools other than energy tools.

Building a Surgical Tool Presence/Absence Detection Model Through Active Learning

Surgical videos including both laparoscopic surgery videos and robotic surgery videos captured during minimally invasive surgeries can help to improve both the efficiency and the quality of the surgeries by providing real-time visual feedback. Object detection models and techniques can leverage this visual feedback by extracting and analyzing information from a surgical video, such as detecting which surgical tools are used to enable various clinical use cases. In this disclosure, a deep-learning-based model and technique for performing frame-by-frame processing of a surgical video to detect an energy device (e.g., a Harmonic™ vessel sealer manufactured by Ethicon™) in the surgical video is disclosed.

In some embodiments, to train the disclosed deep-learning energy tool detection model, an initial training dataset of surgical images (e.g., —8000 images) related to the energy tool use are collected in the data collection phase. In some embodiments, these surgical images are collected from gastric bypass and sleeve gastrectomy procedures. The training images can be labeled by a number of resident surgeons who are highly skilled in the given surgical procedures and using the energy tool. To ensure the quality of labeled training data, annotation guideline and discussion along with a quality assurance procedure are developed. Moreover, a level of agreement around or above 90% across the number of annotators is consistently maintained.

To increase training data diversity to cover more real-world scenarios, multiple data augmentation techniques including random color distortion and geometrical transformation can be carefully applied to the initial training data set, while keeping the original labels of the images. The extended training dataset including both the initial training dataset and the augmented versions of the initial training images can be used to train an initial ML tool detection model. The initial ML tool detection model is then embedded within a training-validation loop equipped with an active learning pipeline to identify an additional training dataset of low confidence-level images. This additional training dataset (˜e.g., ˜2000-3000 images) can be subsequently labeled by the same team of annotators using the same annotation procedure and guideline, and the labeled additional training images are subsequently used to update the initial ML tool detection model.

By incorporating active learning into the disclosed ML tool detection model training procedure, the following improvements over conventional model training schemes have been achieved: (1) a significantly smaller number of training images is annotated; (2) additional training images that are significantly different from the initial training dataset can be identified from a large unprocessed image set and then used to updated the initially trained model; and (3) the convergence time during the model optimization is greatly reduced comparing with a training process without using active learning when the same validation dataset is used. Using the trained ML tool detection model obtained through the disclosed model training procedure, the following optimal F1-score, recall and precision across the validation dataset were obtained as 99.19%, 99.75% and 99.87%, respectively. A separate test dataset was also independently prepared which demonstrated best F1-score, recall and precision of 95.89%, 95.50%, 96.31%, respectively, while the datasets cover various surgical procedures.

FIG.1shows a block diagram of an exemplary machine-learning (ML) energy-tool-detection model building system100(or “ML model-building system100” hereinafter) for generating an energy tool presence/absence detection model through active learning in accordance with some embodiments described herein. As can be seen inFIG.1, ML model-building system100includes a model training module102, a trained tool presence/absence detection model (or “trained tool detection model”)130, an unlabeled data-filtering module104, and a new training dataset annotation module106which are coupled in a loop in the illustrated order. While a Harmonic™ scalpel/vessel sealer made by Ethicon™ is used as the main example of the energy tool described in conjunction with the disclosed ML model-building system100ofFIG.1, ML model-building system100can be used to train and update a tool presence/absence detection model for any model/type of a surgical tool/instrument that is used to simultaneously cut and cauterize/seal tissues during a surgery, and hence the energy tool described in this patent disclosure is not limited to Harmonic™ scalpels or a particular model/type of the energy tool.

To train an untrained energy-tool presence/absence detection model128(or “untrained detection model128”), an initial dataset including a large number of unlabeled endoscope images extracted from a collection of surgical videos recorded during surgical procedures involving the energy tool is first collected. Note that the collection of surgical videos can be collected from various surgical procedures, including but are not limited to gastric bypass and sleeve gastrectomy. A large number of training images are needed partially because high quality data annotation requires multiple data annotators to be consistent and largely agree on many different scenarios. In some embodiments, the diversity of the data sources in the initial dataset is controlled by the number of different doctors and different hospital involved as well as different surgical procedures. Note that the initial dataset can include collected images of different types and models of the energy tool, instead of just a particular type and model of the energy tool. This will allow for building an initially trained tool detection model that can cover all types and models of the energy tool that are potentially in use around the world. For example, for Harmonic vessel sealers, the initial dataset should include the surgical image data of at least of the following tool types: (1) Harmonic Ace; (2) Harmonic Ace+; (3) Harmonic Ace+7; (4) Harmonic HD 1000i; and (5) Harmonic HD 1100, among others. The different types of the energy tool can also include both ultrasonic tools and bipolar tools.

In some embodiments, ˜50% of the collected images in the initial dataset are tool-absent images (or “the first class” images) used for detecting instances when the energy tool is not visible in a given endoscope image; while the other ˜50% are tool-present images (or “the second class” images) used for detecting instances when the energy tool is visible in a given endoscope image. In some particular example, a total of 8000 raw surgical images for energy tool presence/absence detections are collected, of which 4000 images are the tool-absent/first class images while the other 4000 images are the tool-present/second class images. In other embodiments however, the ratio of the first class images and the second class images in the initial dataset can be different from 1:1. For example, another construction of the initial dataset can have ˜40% of the first class images and ˜60% of the second class images; while yet another construction of the initial dataset can have ˜60% of the first class images and ˜40% of the second class images.

In some embodiments, each class of the training images may be further broken down into a number of common subclasses/cases. For example, considering that each cutting/sealing sequence (or an “activation sequence”) by a Harmonic sealer on a tissue can typically comprise multiple shorter firing events (i.e., multiple activation events) with inactive gaps between the multiple activation events, the tool-absent/first class images may further include the following five subclasses: (A1) only anatomy, no tool images; (A2) outside activation-sequence images; (A3) in-between activation events images; (A4) during activation event images; and (A5) other surgical-tool keypoint images. In one real-world example, a dataset of 4086 tool-absent images has the following breakdown corresponding to above five subclasses A1-A5: (A1) 100 images; (A2) 1658 images; (A3) 2065 images; (A4) 24 images; and (A5) 239 fdimages. Similarly, the tool-present/second class images may further include the following three subclasses: (P1) during activation event images; (P2) between activation events images; (P3) the energy-tool keypoint images. In the real-world example, a dataset of 3996 tool-present images has the following breakdown corresponding to above three subclasses P1-P3: (P1) 623 images; (P2) 3318 images; (P3) 55 images.

After the above-described data collection process to obtain the initial dataset, the initial dataset is labeled by a number of resident surgeons (e.g., between 2-5 surgeons) who are highly skilled with the energy tool and the surgical procedure depicted in the training images. To ensure the quality of labeled data, annotation guideline and discussion along with fa quality assurance procedure should be developed. For example, the quality assurance procedure can include performing statistical analysis to uncover anomalies and to identify similarities among the group of annotators in order to increase the level of full agreement among the annotators. As a general requirement, a level of full agreement around 90% across all annotators should be consistently maintained. For those training images that involve disagreements among annotators, additional review and discussion are used to determine the cause of the disagreements, and labels with consensus are eventually obtained. In practice, it has been observed that annotators generally have very high agreement on both true tool-present cases and true tool-absent cases. However, some disagreement can occur on a few true tool-absent cases (such as cases where the tool is maneuvering around the edge of an image frame), while disagreement rarely occurs on most or all true tool-present cases.

After data annotation on the initial dataset, the labeled initial dataset120shown inFIG.1is obtained. Next, ML model-building system100uses model training module102and the labeled initial dataset120to train an untrained tool presence/absence detection model128(or “untrained detection model128”), and output a trained presence/absence detection model130. In various embodiments, untrained detection model128can be implemented using various convolutional neural network (CNN/ConvNet) architectures. For example, untrained detection model128can be implemented with a residual neural network (ResNet). In a particular embodiment, untrained detection model128is implemented with ResNet18 with all layers in the network unfrozen.

As can be seen inFIG.1, during the model training, labeled initial dataset120is divided into a training dataset122and a validation dataset124. As one practical example, 8000 images in the labeled initial dataset120is divided into the training dataset including 80% of the total images, and the validation dataset with the rest of the 20% of the total images. In addition, an independent test dataset equal to the size of validation dataset124can also be prepared. In some embodiments, model training module102is configured to train an untrained tool presence/absence detection model128through a number of N epochs (e.g., N=200) based on the training dataset122. At the end of each epoch, model training module102calculates the recall, precision, the F1-score, and accuracy values over the validation set124, and stores the set of updated model parameters. At the end of N epochs, the set of stored model parameters corresponding to a given epoch that generates the best performance metric, e.g., the highest recall value is chosen as the trained presence/absence detection model130. A person skilled in the art will appreciate that depending on which best metric value of the four performance metrics (i.e., recall, precision, the F1-score, and accuracy) is used, a slightly different trained model130can be obtained, each having its own strength and weakness. Note that from the clinical point of view, a trained model130having the highest recall value may be preferred due to the fact that the version of the trained model130can provide the best performance for capturing the most number of tool-absent (i.e., tool off-screen) events.

As can be seen inFIG.1, model training module102also includes a data augmentation submodule132. In various embodiments, data augmentation submodule132is configured to perform a number of image augmentation operations on a given image in the labeled initial dataset120to generate a number of transformed/augmented images of the initial image. Note that regardless the types of image augmentations are used on a given initial image, the resulting augmented images should have the same present/absent label as the given initial image. In various embodiments, the image augmentation operations that can be performed by data augmentation submodule132can include an image scaling function. Specifically, the image scaling function is configured to multiply the original height and width of a given initial image by one random number (but does not change the aspect). This random number can be chosen from a range provided by the data scientist, e.g., a range that is between 0 and 1. Note that this image augmentation function can be seen as a zoom-in function by a random amount. The image augmentation operations can also include an aspect ratio augmentation. Specifically, the ratio function is configured to multiply the original aspect ratio of the initial image by a random number. This random number can be chosen from a range provided by the user, e.g., a range that is between 0 and infinity. The image augmentation operations can also include a cropping/resizing function. Specifically, the cropping/resizing function is configured to first randomly crop out a portion of the initial image and subsequently resize the cropped image to a predetermined size, e.g., (244,244).

Note that data augmentation submodule132can also be configured to perform the following image augmentation functions on a given initial image: (1) rotating the image, either clockwise or counterclockwise; (2) flipping the image, either with respect to the horizontal axis or the vertical axis; (3) changing the image brightness; (4) changing the image color tunes; (5) changing the image resolution; among others. Note that although data augmentation submodule132can generate the various types of augmented images artificially based on the initial dataset120, the augmented images are generally used to mimic real world scenarios that can happen but are not necessarily included in the initial dataset120. For example, suppose that the common tool positions are on the right side of the image frames with the tool tip pointing to the left due to most of the surgeons holding the tool with the right hand, the surgical tool images from a left-handed surgeon may appear on the left side of the image frames with the tool tip point to the right. By performing a left-right imaging flipping operation on the initial images, the above scenario can be simulated. As another example, the qualities of endoscope videos from different hospitals around the world are vastly different, leading to capturing videos of various color ranges. This video color variation can be simulated by performing color manipulations in the image augmentation process. Furthermore, the surgical image rotations caused by the axis of the endoscope constantly rotating inside the abdomen of a patient during the surgery can be simulated by a random rotation of the images during the image augmentation process.

Note that through the N epochs of model training, a given image in the labeled initial dataset120may be transformed differently using different image augmentation functions in different epochs. In some embodiments, after training tool detection model128using the original labeled initial dataset120in the first epoch of N epochs of model training, model training module102is configured to transform a randomly-selected subset of the labeled initial dataset120into corresponding augmented images using data augmentation submodule132at the beginning of each subsequent epoch of model training. Specifically, to transform a given image in the randomly-selected subset, data augmentation submodule132can randomly apply one of the available image augmentation functions to the given image. After the training dataset transformation, the original labeled initial dataset120becomes a modified training dataset comprising both the augmented and labeled images and original labeled images from labeled initial dataset120. Next, model training module102is configured to train tool detection model128using the modified training dataset generated for each subsequent epoch. As such, instead of using the same labeled initial dataset120for all training epochs, the above-described training technique adds a new set of data diversities into the training dataset for each epoch of model training. Consequently, the trained detection model130at the end of N epochs is able to identify and correctly classify significantly more tool present/absent situations that can arise in surgical procedures than the trained detection model130without adding the augmented images into the training process.

Referring back toFIG.1, note that after generating trained tool detection model130, ML model-building system100then applies trained tool detection model130to an unlabeled image dataset140. Note that unlabeled dataset140may be obtained at the same time when the labeled initial dataset120was collected or unlabeled image dataset140may be obtained separately before or after the labeled initial dataset120was collected. Because image data annotation work is highly labor intensive, it is not practical to manually annotate all of the collected image data. However, after tool detection model130is trained based on a reasonably-sized initial dataset120, unlabeled image dataset140can now be processed and used to further train and improve trained tool detection model130through a process of Active Learning. In some embodiments, unlabeled image dataset140is collected after the labeled initial dataset120was collected for a new model or a new type of the energy tool, e.g., a new type of Harmonic™ scalpel that has just become commercially available but has not been built into the trained tool detection model130.

Note that the general concept of Active Learning is that, for the best training data annotation efficiency and effectiveness, data annotation resources and priority should be given to those date/images containing the most information, i.e., those scenarios unfamiliar to the trained model, referred to as “low confidence” data to the trained model. In other words, for those scenarios which are already built into the trained model, it is unnecessary and inefficient to collect and label the same types of data and update the model with such data, because they are “high confidence” data to the trained model. Furthermore, labeling these “high confidence” data and including them in the training process could also lead to an overfitting problem. Without processing, unlabeled image dataset140can contain a large amount of such high confidence images to trained tool detection model130. Hence, instead of manually labeling and retraining the model using these high confidence images, these high confidence images should be removed from unlabeled image dataset140. In contrast, those low confidence data in unlabeled dataset140should be identified and labeled as new training data.

Using the concept of active learning, trained tool detection model130and unlabeled data-filtering module104are used collectively to select a subset of unlabeled image dataset140that has the most information unfamiliar to trained tool detection model130. More specifically, trained tool detection model130processes unlabeled image dataset140and outputs a set of confidence levels150for the set of images in unlabeled image dataset140, wherein each confident level in the set of confidence levels150is between (0, 1). For a given image in unlabeled image dataset140, if the corresponding confidence level in the set of confidence levels150is very close to 1, it indicates that trained tool detection model130is highly confident that the target energy tool is detected in the given image (i.e., detecting a tool-present scenario). On the other hand, if the output confidence level for the given image very close to 0, it indicates that trained tool detection model130is highly confident that the target energy tool is absent in the given image (detecting a tool-absent scenario). However, if the output confidence level for the given image is neither close to 1 nor to 0 but lies somewhere between (0, 1), e.g., 0.6 or 0.7, it means that trained tool detection model130is not sufficiently confident on either a tool-presence or a tool-absence decision for the given image. Unlabeled data-filtering module104can then filter unlabeled image dataset140and outputs a low-confidence dataset160that includes only those images having low-confidence levels. Such images are the above-described low confidence data of the highest interest, which can now be passed to the annotators to be labeled. As such, Active Learning implemented in ML model building system100only selects those low-confidence images in unlabeled image dataset140, instead of labeling the entire unlabeled image dataset140.

Note that by using active learning in the process of training the tool detection model, the number of epochs that is needed to converge to the optimal model performance can be significantly reduced (given the same validation dataset is used). In an exemplary training process, it was observed that before using active learning, 184 epochs were needed to get the optimal recall score of the model. However, after using active learning, only 6 epochs were used to obtain the same recall score of the model. This improvement in model training performance is due to the fact that active learning can refine the training dataset by enriching the information in the training dataset and as a result, the model optimizer converges more quickly to the best answer on the same validation dataset.

To identify the low confidence data and to filter out the high confidence data, unlabeled data-filtering module104is configured with two thresholds TH1and TH2. Generally speaking, threshold TH1should be set to be very close to 0 to separate high-confidence absent images and low-confidence absent images. In contrast, threshold TH2should be set to be very close to 1 to separate high-confidence present images and low-confidence present images. Hence, the range between TH1and TH2corresponds to low confidence levels.FIG.2shows an exemplary data-filter configuration with two thresholds TH1and TH2setting the boundaries between high confidence data and low confidence data in accordance with some embodiments described herein. In the example show, TH1is set to 0.01 which means inferred confidence levels near or below 0.01 for images in unlabeled image dataset140is considered as a high-confidence tool-absent image202. Moreover, TH2is set to 0.99 which means inferred confidence levels near or above 0.99 for images in unlabeled image dataset140are considered as high-confidence tool-present images204. As such, the range (0.01, 0.99) corresponds to the low confidence levels. As a result, those images that have inferred confidence levels fall between the range (0.01, 0.99), e.g., 0.5, are identified as the low confidence images206which are selected and included in the low-confidence dataset160as the output of unlabeled-data filtering module104.

Note that the two threshold values 0.01, 0.99 are just example values that can provide sufficiently good filtering results. Generally speaking, the values of TH1and TH2are statistical determined. For example, TH1and TH2can be determined based on the determined recall score of trained tool detection model130. In this regards, for instance, the two thresholds TH1and TH2can be determined by analyzing the result of the validation dataset. More specifically, we can first gather the confidence levels of those samples in the of the validation dataset that are incorrectly inferred by the trained tool detection model130, and subsequently determine the range of gathered confidence levels that contains false negative and false positives. The two thresholds, TH1and TH2can be automatically obtained from the two boundaries of the determined range. Those unlabeled image dataset140identified in low-confidence dataset160can then be annotated/labeled by new training dataset annotation module106, which generally includes the same manual-annotation procedures by the skilled annotators as described above.

As a continuation of the practical example that started with 8000 raw surgical images for the Harmonic-sealer tool presence/absence detection, additional 12,000 raw surgical images for the Harmonic-sealer tool were collected. Instead of manual annotating these 12,000 images, they are passed through the disclosed ML model-building system100, and only ˜2700 (˜23%) of the 12,000 images were identified as low confidence images with new information, and subsequent labeled. In other words, by using Active Learning and the disclosed ML model-building system100, the additional annotation effort on the 12,000 images can be greatly reduced.

After low-confidence dataset160within unlabeled image dataset140are annotated/labeled, a new training dataset170are obtained. Using the above example, new training dataset170would include ˜2700new training image. In the disclosed ML model-building system100, new training dataset170is combined with initial training dataset122to obtain a combined training dataset that has a greater size and more diverse and complex than initial training dataset122. In some embodiments, the combined training dataset is used to update trained tool detection model130in the model training loop. This is shown inFIG.1where model training module102receives trained tool detection model130and the combined training dataset122and170as inputs and an updated tool detection model180as output. Note that updating trained tool detection model130means that the model will be trained from its present state. In other words, trained tool detection model130is not re-trained from the scratch but further trained from the present state of the model. Note that while the training dataset has grown in size and diversity, validation dataset124can remain the same so that a fair comparison can be made between the trained tool detection model130without the active learning and the updated tool detection model130based on the combined training dataset.

Note that one practical reason of updating trained tool detection model130is to obtain an updated version of the tool detection model for a new version/type of the energy tool that has not been built into the trained tool detection model130. As mentioned above, to update tool detection model130for a new version/type of the energy tool, unlabeled image dataset140can be generated for the new version/type of the energy tool, e.g., a new type of Harmonic™ sealer that has just become commercially available. Generally speaking, the new version/type of the energy tool may differ from the existing versions/types of the energy tool that have been built into the trained tool detection model130to some degree, e.g., in terms of changes in colors, in terms of changes in geometries, in terms of changes in printed text, or in terms of missing or additions certain mechanical features/parts. However, the new version/type of the energy tool is also largely the same in overall appearance as the existing versions/types of the energy tool that have been built into the trained tool detection model130. As such, it is unnecessary to re-train the trained tool detection model130on the entire unlabeled image dataset140generated for the new version/type of the energy tool. Using the disclosed ML model-building system100with an active learning loop, a subset of low-confidence images160within the unlabeled image dataset140that contains the useful information of the new version/type of the energy tool but that has not been built into the trained tool detection model130(e.g., changes in colors, geometries, printed text, and/or missing or additions certain mechanical features/parts) can be identified and used to update the trained tool detection model130. However, those images in the unlabeled image dataset140that can be inferred by the trained tool detection model130with high confidences are identified and removed.

FIG.3presents a flowchart illustrating an exemplary process for generating an energy tool presence/absence detection model through active learning in accordance with some embodiments described herein. In one or more embodiments, one or more of the steps inFIG.3may be omitted, repeated, and/or performed in a different order. Accordingly, the specific arrangement of steps shown inFIG.3should not be construed as limiting the scope of the technique.

Process300may begin by receiving a labeled training dataset including about half of all training images labeled as tool-absent images while the other half of the training images are labeled as tool-present images (step302). Note that to obtain the labeled training dataset, an initial dataset including a large number of unlabeled endoscope images recorded during the surgical tasks involving the energy tool has to be collected, and ˜50% of the collected images are tool-absent images used for detecting instances when the energy tool is not visible in a given endoscope image; while the other ˜50% are tool-present images used for detecting instances when the energy tool is visible in a given endoscope image. However, as described above, the breakdown between the tool-absent images and the tool-present images in the initial training dataset can be different from ˜50% for each type of images, for example, the breakdown can be 60%/40% or 40%/60% for the two types of labled images. The labeled training dataset is subsequently generated through a manual annotation procedure by a group of highly skilled surgeons with the energy tool and the surgical procedure depicted in the training images.

Next, process300trains a tool presence/absence detection model using the labeled training dataset to obtain a trained tool presence/absence detection model (step304). In some embodiments, when training the tool presence/absence detection model over a number of epochs based on the labeled training dataset, process300can apply data augmentations such as random color distorting and geometrical transformation on a subset of the training dataset in different epochs to increase the diversity of the training data in a realistic manner while not altering the labels of the augmented images from the original labeled images.

After generating the trained tool presence/absence detection model, process300then applies the trained tool detection model to an additional image dataset of the energy tool and generates a corresponding set of inferred confidence levels for the unlabeled image dataset (step306). Next, process300identifies a subset of low-confidence-level images among the additional image dataset that has inferred confidence levels fall between two high-confidence-level thresholds corresponding to tool-absent prediction and tool-present prediction, respectively (step308). Note that one of the two high-confidence-level thresholds is close to 0 (e.g., 0.1) indicating a high-confidence of detecting that the tool is absent in an image wherein the other confidence-level threshold is close to 1 (e.g., 0.99) indicating a high-confidence of detecting that the tool is present in an image. In various embodiments, the values of the two high-confidence-level thresholds are statistical determined. Process300then provides true presence/absence labels to the identified low-confidence-level dataset through a manual annotation procedure (step310). Process300next combines the labeled slow-confidence-level dataset with the original labeled training dataset to generate an updated training dataset (step312). Process300subsequently updates the trained presence/absence detection model using the updated training dataset (step314).

Enemy Tool Real-Time Safety Monitoring Using Tool Present/Absent Detector

FIG.4presents a flowchart illustrating an exemplary process400for preventing injuries from an energy tool used for cutting/sealing tissues during a laparoscopic or robotic surgery in accordance with some embodiments described herein. In one or more embodiments, one or more of the steps inFIG.4may be omitted, repeated, and/or performed in a different order. Accordingly, the specific arrangement of steps shown inFIG.4should not be construed as limiting the scope of the technique.

Process400may begin by receiving a real-time control signal indicating a tool activation state of an energy tool during a surgery (step402). In some embodiments, the real-time control signal is received from a tool controller. For example, in the case of a harmonic ultrasonic sealer, the control signal is generated by a Ethicon™ generator such as Gen11™. Note that however, each activation decision during the surgery is made by the surgeon and initiated by pulling on a handle on the energy tool or pressing a button on the tool. The surgery action then triggers the generator to begin generating an activation pulse which is then transmitted to the tool and energizes the jaws of the tool. Note that in conventional systems, as long as the handle is not released or the activation button is not pressed again, the activation pulse continues to be generated and the energy tool remains activated. However, the activation pulse stops when the handle is released or the activation button is pressed again, thereby by disabling the energy tool. Note that at the end of the current activation session, the activation pulse data, including the starting and ending timestamp (or alternatively the duration of the pulse) and the power settings are logged by the generator. As a result, at the end of the surgery the generator logs a sequence of activation pulses, wherein each activation pulse corresponds to a single activation/firing event of the energy tool.

While receiving the real-time control signal, process400simultaneously receives real-time endoscope video frames of the surgery captured by an endoscope camera (step404) and simultaneously applies the above-described energy tool presence/absence detector to the real-time endoscope video frames as the video frames are being received to generate a real-time tool presence/absence decision for each processed video frame as well as a confidence level associated with each presence/absence decision (step406). In various embodiments, the tool presence/absence detector described herein is generated by the disclosed ML model-building system100inFIG.1. In some embodiments, the presence/absence decision can include: (1) tool presence decision indicating the energy tool is present in the given video frame; and (2) a tool absence decision indicating the energy tool is absent in the given video frame. Note that each decision is also associated with a confidence level, and a low confidence level is generally not expected because the training data used to train the tool presence/absence detector covered potential edge cases that could appear in each image frame, such as a tool too close to the edge or an occluded tool. In contrast, a high confidence level below the confidence level threshold TH1for an absence decision generally means the tool is not visible in the given endoscope image; whereas a high confidence level above the confidence level threshold TH2for a presence decision generally means the tool is fully visible in the given endoscope image.

Next, process400checks the received real-time control signal against the real-time presence/absence decisions to identify an unsafe event that involves a tool absence decision (step408). Note that because the real-time presence/absence decisions are generated on a frame-by-frame basis, step408can also be performed on a frame-by-frame basis for each newly-generated presence/absence decision on an endoscope video image.FIG.5presents a flowchart illustrating an exemplary process500for making proper safe/unsafe determinations based on the received real-time control signal of the energy tool and the real-time presence/absence decisions on the endoscope video images from the tool presence/absence detector in accordance with some embodiments described herein. In one or more embodiments, one or more of the steps inFIG.5may be omitted, repeated, and/or performed in a different order. Accordingly, the specific arrangement of steps shown inFIG.5should not be construed as limiting the scope of the technique.

Process500begins by determining that the newly-generated (i.e., current) decision by the tool presence/absence detection model is a tool absence decision (step502). Process500next determines if the tool absence decision coincides temporally with an activation pulse in the real-time control signal (step504). For example, process can determine that the new absence decision is generated inside an activation pulse of the real-time control signal when the real-time control signal is HIGH. If the new tool absence decision coincides with the activation pulse, process500further determines if the current activation pulse is at the beginning of the corresponding activation duration (step506). In some embodiments, process500determines whether the current activation pulse is at the beginning of the current activation duration by computing a time period from when the current activation pulse first transitions to HIGH until the current time. Process500subsequently determines that the current activation pulse is at the beginning of the current activation duration if the computed time period is shorter than a predetermine time period based on an average activation duration of the energy tool during a surgical procedure.

As described above, the highly unsafe scenario that the real-time energy tool safety monitoring system is designed to avoid is when the energy tool is just beginning to fire but the tool is off screen (e.g., when both jaws of the energy tool are not in the video frame) and not visible to the user. Hence, if process500determines that the current activation pulse is at the beginning of its activation duration when the new tool absence decision is generated, process500identifies an unsafe event (step508).FIG.6Aillustrates an exemplary scenario of detecting an unsafe tool-use event by comparing an exemplary control signal602of the energy tool including an activation pulse604and an exemplary tool presence/absence signal606generated by the disclosed tool presence/absence detection model in accordance with some embodiments described herein. As can be seen inFIG.6A, tool presence/absence signal606is composed of both tool absence decisions which correspond to the LOW (˜0) values of tool presence/absence signal606; and tool presence decisions which correspond to the HIGH (˜1) values of tool presence/absence signal606. Moreover, activation pulse604in control signal602corresponds to the HIGH values in control signal602. It can be observed that tool presence/absence signal606includes a sequence of tool absence decisions608that coincides/overlaps with activation pulse604. Moreover, the time period610that the sequence of tool absence decisions608coincides/overlaps with activation pulse604is the beginning/early portion of the activation pulse604when the energy tool begins to fire. Hence, the exemplary scenario depicted inFIG.6Arepresents an unsafe tool-use event that should be detected. This unsafe tool-use event can detected by checking any absence decision (e.g., absence decision612) within absence decisions608against activation pulse604, and detecting that the absence decision falls within the early activation time period610of activation pulse604.

FIG.6Billustrates an exemplary scenario of detecting another unsafe tool-use event by comparing an exemplary control signal622of the energy tool including an activation pulse624and an exemplary tool presence/absence signal626generated by the disclosed tool presence/absence detection model in accordance with some embodiments described herein. As can be seen inFIG.6B, tool presence/absence signal626is composed of a short pulse of tool presence decision(s)628which corresponds to the HIGH (˜1) values of tool presence/absence signal626which is then followed by a long period of tool absence decisions630which correspond to the LOW (˜0) values of tool presence/absence signal626. Note that even though quick tool presence decision(s)628were generated at the beginning of activation pulse624, tool absence decisions630begin to be generated right after tool presence decision(s)628which coincide/overlap with activation pulse624. Moreover, early portion of tool absence decisions630also coincides/overlaps with the beginning/early portion of the activation pulse624when the energy tool begins to fire. Hence, the exemplary scenario depicted inFIG.6Balso represents an unsafe tool-use event that should be detected. This unsafe tool-use event can detected by checking any absence decision (e.g., absence decision650) within absence decisions630against activation pulse624, and detecting that the absence decision falls within the early activation time period of activation pulse624.

However, if process500determines that the current activation pulse is not at the beginning of the activation duration when the new tool-absence decision is generated, process500determines that the energy tool is safe to use (step510). As described above, the tool absence decisions can occur toward the end of a given activation duration, or even in the middle of the activation duration when the endoscope camera has already moved away from the location of energy tool. However, because the energy tool itself most likely still remains in place, it is reasonable to assume that the energy tool has no safety concerns.FIG.6Cillustrates an exemplary scenario of determining that the tool is safe to use by comparing an exemplary control signal632of the energy tool including an activation pulse634and an exemplary tool presence/absence signal636generated by the disclosed tool presence/absence detection model in accordance with some embodiments described herein. As can be seen inFIG.6C, tool presence/absence signal636is composed of both tool absence decisions which correspond to the LOW (˜0) values of tool presence/absence signal636; and tool presence decisions which correspond to the HIGH (˜1) values of tool presence/absence signal636. Moreover, activation pulse634in control signal632corresponds to the HIGH values in control signal632. It can be observed that tool presence/absence signal636includes a sequence of tool presence decisions638that substantially coincides/overlaps with activation pulse634. Also during the same time period of activation pulse634, there is no tool absence decision generated. Hence, the exemplary scenario depicted inFIG.6Crepresents a safe tool-use scenario.

FIG.6Dillustrates another scenario of determining that the tool is safe to use by comparing an exemplary control signal642of the energy tool including an activation pulse644and an exemplary tool presence/absence signal646generated by the disclosed tool presence/absence detection model in accordance with some embodiments described herein. As can be seen inFIG.6D, tool presence/absence signal646is composed of both tool absence decisions which correspond to the LOW (˜0) values of tool presence/absence signal646; and tool presence decisions which correspond to the HIGH (˜1) values of tool presence/absence signal646. Moreover, activation pulse644in control signal642corresponds to the HIGH values in control signal642. It can be observed that tool presence/absence signal646includes a sequence of tool absence decisions648that coincides/overlaps with activation pulse644. However, the time period660that the sequence of tool absence decisions648coincides/overlaps with activation pulse644is toward the end of activation pulse644. As described above, the exemplary scenario depicted inFIG.6Dalso represents a safe tool-use scenario. This safe tool-use decision can be made by checking any absence decision within the sequence of absence decisions648against activation pulse644, and detecting that no absence decision648falls within the early activation time period of activation pulse644.

Returning toFIG.4, if after checking the real-time control signal against the real-time presence/absence decisions, process400identifies the unsafe event that involves a newly-generated tool absence decision, process400next determines if the confidence level associated with the newly-generated tool absence decision is above a high confidence level threshold (step410). Note that a high confidence level for a tool absence decision generally means the tool is completely missing in the given endoscope image. If so, process400immediately disables the energy tool so that the tool can not fire (step412). However, if process400determines that the confidence level associated with the newly-generated tool absence decision is below the high confidence level threshold, process400can generate one or more warning/alert feedbacks as a safety guard without fully disabling the tool (step414). The types of warning/alert feedbacks can include, but are not limited: (1) generating and displaying a visual alert/feedback on an endoscope monitor; (2) generating an audio alert through a console speaker; (3) generating a mechanical vibration through the energy tool; and (4) a mechanical/tactile feedback that delays the firing of the energy tool until the user takes a further action. For example, the mechanical/tactile feedback can be implemented as an interlock design that requires two-stop activation. More specifically, the first stop of the interlock design is used for generating the activation pulse and triggering the detection of an unsafe event. In this regard, the mechanical warning is provided to the user if the tool does not fire at the first stop, or the warning can be the fact that the energy tool does not fire at the first stop which is certainly noticed by the user. At the moment, the user can choose to inspect the endoscope video before taking another action, or alternatively the user may choose to proceed to fire the tool by applying extra force on the firing handle or button until the second stop of the interlock design is reached.

Note that the exemplary process400is designed to identify an unsafe event after each newly presence/absence decision is made based on a new video frame. However, this technique can be computationally-intensive and also susceptible to false positives. Realizing that the primary unsafe event is when the energy tool just begins to fire but the tool is missing from the endoscopic view, it is possible to modify process400to obtain more efficient process to detect such unsafe event. More specifically, when the real-time control signal indicates that the energy tool is idle, i.e., no activation pulse exists (e.g., when the signal is LOW), the modified process does not have to use the ML tool-detection model to detect whether the real-time endoscope images include the energy tool or not because the tool is inherently safe. However, the modified process continues to detect a new activation pulse in the control signal. When the beginning of a new activation pulse is detected in the control signal, e.g., by detecting a signal transition from LOW to HIGH, the modified process can start applying the machine-learning model to the real-time laparoscopic video frames and to start generating real-time decisions. Note that once the tool presence/absence decisions are being generated, the rest of the modified process is substantially the same as process400between step406and step414.

FIG.7Ashows an exemplary endoscope console702displaying an endoscope image containing a target energy tool704and a visual feedback706generated by the disclosed energy tool presence/absence detection model in accordance with some embodiments described herein. Note that because the disclosed tool presence/absence detection model can successfully detect energy tool704in endoscope console702, visual feedback706is shown as a green circle indicating that the tool is visible and hence safe to use.

In contrast,FIG.7Bshows an exemplary endoscope console712displaying an endoscope image containing a wrong surgical tool714and a visual feedback716generated by the disclosed energy tool presence/absence detection model in accordance with some embodiments described herein. Note that because the disclosed tool presence/absence detection model can successfully the target energy tool is absent in endoscope console712(even when a different tool714is present), visual feedback716is shown as a yellow circle indicating that an unsafe event is identified. The disclosed detection model additionally generates a warning message718inside endoscope console712next to the visual feedback716specifying the type of unsafe event that is detected.

Other Applications of the Energy Tool Presence/Absence Detection Model

Note that based on the ML tool-detection model output from a given surgical procedure, the total time the energy tool is present in the endoscopy video during the surgical procedure can be easily determined. Hence, a percentage of time the energy tool is present over the overall duration of the surgical duration can be calculated. This percentage value can then be compared with a standard of percentage value for the energy tool presence, and from which the skills of the surgeon using the energy tool can be estimated based on whether the computed percentage value is above or below the nominal value and by how much. For example, if a surgeon typically has 10% less “presence time” of the tool in his surgeries, this could mean that the surgeon has used less energy during the surgery and hence the patient may be able to recover faster due to less damage to the patient's tissues. Note that the energy tool presence information from the tool presence/absence detection model can be collaborated with the tool activation data from the tool log, such as the number of activations/firings of the energy tool in each minute of the determined tool presence. Note that the number of activations/firings of the tool per tool presence can be another indicator of the surgeon's skill level and/or a complexity level of the surgery. Another metric that can be determined based on the tool detection model output can include the number of activations per duration of activation. For example, 10 minutes of activation could contain 20 activations (i.e., each activation, in average, has lasted for 30 seconds). In another example, 10 minutes of activations could contain 40 activations (i.e., each activation, in average, has lasted for 15 seconds). The above metric can be used to infer the age and efficiency of the device, and can also be used to infer the complexity of the human anatomy. Note that the number of activations can also be correlated with the length of the surgery, because a higher number of activations would typically mean more damage to tissue, and hence higher likelihood of complexity which could lead to longer surgery time.

In some embodiments, the output from the disclosed ML tool presence/absence detector can be collaborated with the output from another ML model trained to detect and extract different surgical phases and surgical tasks within the surgical procedure. Hence, the disclosed ML tool presence/absence detector output can further be used to make at least the following event determinations: (1) the energy tool is present in an identified surgical phase/surgical task (such as greater curve dissection); (2) the energy tool is absence from an identified surgical phase/surgical task; (3) if the energy tool is present in an identified surgical phase/surgical task, how long the energy is present in the identified surgical phase/surgical task; (4) if the energy tool is present in an identified surgical phase/surgical task that the energy tool is not supposed to be present; and (5) if the energy tool is absent from an identified surgical phase/surgical task that the energy tool is supposed to be present. Based on the above information that can be extracted from the disclosed ML tool presence/absence detector output, the skills of the surgeon using the energy tool can be further evaluated and compared with other surgeons performing the same surgical procedure. Moreover, surgical anomalies may be identified if events (4) and (5) are detected from based on the offline procedure data analytics. Moreover, because the disclosed tool presence/absence detector can continuously report the presence and absent of the energy tool, we can count the number of times that the tool leaves the endoscope view and subsequently returns to the endoscope view. This count can be correlated with the complexity level of the organ/tissue that is under the surgery as well as the skill level of the surgeon,

In some embodiments, the output from the disclosed ML tool presence/absence detector can be collaborated with the output from yet another ML model trained to detect different organs/tissues in the endoscope video of the surgical procedure. Hence, the disclosed tool presence/absence detector output can further be used to determine which organs/tissues the energy tool was used upon and for how long. Based on the above information that can be extracted from the output of the disclosed ML tool presence/absence detector, the skills of the surgeon using the energy tool can be further evaluated and compared with other surgeons performing the same surgical procedure.

In some embodiments, the output from the disclosed ML tool presence/absence detector can be collaborated with the output from yet another ML model trained to detect a bleeding event or other complication events in the endoscope video of the surgical procedure. Hence, the output of the disclosed ML tool presence/absence detector can further be used to determine if the energy tool was present during a detected bleeding event. If so, additional information related to the use of the energy tool during the surgical procedure may be collaborated with the detected event to predict or determine the cause of the bleeding. For example, the additional information of the energy tool use may include the settings, such as power level of the energy tool when the bleeding event occurs. The additional information can also include the identified surgical task detected by another ML model when the bleeding event occurs. Moreover, useful statistics can be generated in terms of what percentage of the total tool activations leads to bleeding or other complications.

Note that because the disclosed ML tool presence/absence detector was continuously trained and updated to detect different models of the energy tool with different versions (e.g., all existing versions) of a given model of the energy tool, the output from the disclosed ML tool presence/absence detector can be used during the offline procedure data analytics to generate useful statistics for the tool manufacturers. Note that the useful statistics can include collaborating above-described of bleeding and other complications statistics with the tool model/version statistics. Specifically, statistics between collected bleeding and other complication events and collected models and versions of the energy tool can be established or updated. Such statistics can then be used to evaluate and score each model and version of the energy tool.

FIG.8conceptually illustrates a computer system with which some embodiments of the subject technology can be implemented. Computer system800can be a client, a server, a computer, a smartphone, a PDA, a laptop, or a tablet computer with one or more processors embedded therein or coupled thereto, or any other sort of computing device. Such a computer system includes various types of computer-readable media and interfaces for various other types of computer-readable media. Computer system800includes a bus802, processing unit(s)812, a system memory804, a read-only memory (ROM)810, a permanent storage device808, an input device interface814, an output device interface806, and a network interface816. In some embodiments, computer system800is a part of a robotic surgical system.

Bus802collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of computer system800. For instance, bus802communicatively connects processing unit(s)812with ROM810, system memory804, and permanent storage device808.

From these various memory units, processing unit(s)812retrieves instructions to execute and data to process in order to execute various processes described in this patent disclosure, including the above-described surgical tool presence/absence detection model building techniques and techniques for detecting unsafe events during a surgery using the disclosed surgical tool presence/absence detection models inFIGS.1-6. The processing unit(s)812can include any type of processor, including, but not limited to, a microprocessor, a graphic processing unit (GPU), a tensor processing unit (TPU), an intelligent processor unit (IPU), a digital signal processor (DSP), a field-programmable gate array (FPGA), and an application-specific integrated circuit (ASIC). Processing unit(s)812can be a single processor or a multi-core processor in different implementations.

ROM810stores static data and instructions that are needed by processing unit(s)812and other modules of the computer system. Permanent storage device808, on the other hand, is a read-and-write memory device. This device is a non-volatile memory unit that stores instructions and data even when computer system800is off. Some implementations of the subject disclosure use a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive) as permanent storage device808.

Other implementations use a removable storage device (such as a floppy disk, flash drive, and its corresponding disk drive) as permanent storage device808. Like permanent storage device808, system memory804is a read-and-write memory device. However, unlike storage device808, system memory804is a volatile read-and-write memory, such as a random access memory. System memory804stores some of the instructions and data that the processor needs at runtime. In some implementations, various processes described in this patent disclosure, including the above-described surgical tool presence/absence detection model building techniques and techniques for detecting unsafe events during a surgery using the disclosed surgical tool presence/absence detection models inFIGS.1-6, are stored in system memory804, permanent storage device808, and/or ROM810. From these various memory units, processing unit(s)812retrieves instructions to execute and data to process in order to execute the processes of some implementations.

Bus802also connects to input and output device interfaces814and806. Input device interface814enables the user to communicate information to and select commands for the computer system. Input devices used with input device interface814include, for example, alphanumeric keyboards and pointing devices (also called “cursor control devices”). Output device interface806enables, for example, the display of images generated by the computer system800. Output devices used with output device interface806include, for example, printers and display devices, such as cathode ray tubes (CRT) or liquid crystal displays (LCD). Some implementations include devices such as a touchscreen that functions as both input and output devices.

Finally, as shown inFIG.8, bus802also couples computer system800to a network (not shown) through a network interface816. In this manner, the computer can be a part of a network of computers (such as a local area network (“LAN”), a wide area network (“WAN”), an intranet, or a network of networks, such as the Internet. Any or all components of computer system800can be used in conjunction with the subject disclosure.

The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed in this patent disclosure may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

The hardware used to implement the various illustrative logics, logical blocks, modules, and circuits described in connection with the aspects disclosed herein may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of receiver devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Alternatively, some steps or methods may be performed by circuitry that is specific to a given function.

In one or more exemplary aspects, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or code on a non-transitory computer-readable storage medium or non-transitory processor-readable storage medium. The steps of a method or algorithm disclosed herein may be embodied in processor-executable instructions that may reside on a non-transitory computer-readable or processor-readable storage medium. Non-transitory computer-readable or processor-readable storage media may be any storage media that may be accessed by a computer or a processor. By way of example but not limitation, such non-transitory computer-readable or processor-readable storage media may include RAM, ROM, EEPROM, flash memory, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that may be used to store desired program code in the form of instructions or data structures and that may be accessed by a computer. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above are also included within the scope of non-transitory computer-readable and processor-readable media. Additionally, the operations of a method or algorithm may reside as one or any combination or set of codes and/or instructions on a non-transitory processor-readable storage medium and/or computer-readable storage medium, which may be incorporated into a computer-program product.

While this patent document contains many specifics, these should not be construed as limitations on the scope of any disclosed technology or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular techniques. Certain features that are described in this patent document in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. Moreover, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.

Only a few implementations and examples are described, and other implementations, enhancements and variations can be made based on what is described and illustrated in this patent document.