Patent Publication Number: US-2022233056-A1

Title: Automatically controlling an on/off state of a light source for an endoscope during a surgical procedure in an operating room

Description:
PRIORITY CLAIM AND RELATED PATENT APPLICATIONS 
     This patent application is a continuation of, and hereby claims the benefit of priority under 35 U.S.C. § 120 to co-pending U.S. patent application Ser. No. 17/037,056, filed on 29 Sep. 2020 (Attorney Docket No. 10231P1077USC), entitled, “Method and System for Automatically Turning On/Off a Light Source for an Endoscope During a Surgery,” which itself is a continuation of U.S. patent application Ser. No. 16/440,552, filed on 13 Jun. 2019 (Attorney Docket No. 10231P1077US), entitled, “Method and System for Automatically Turning On/Off a Light Source for an Endoscope During a Surgery,” which has issued as U.S. Pat. No. 10,799,090 on Oct. 13, 2020. All of the above-listed applications are hereby incorporated by reference as a part of this patent document. 
    
    
     TECHNICAL FIELD 
     The present disclosure generally relates to surgical automation and, more specifically, to systems, devices and techniques for automatically turning on/off a light source of an endoscope when the endoscope is being put into/taken out of a patient&#39;s body or pointed at a white object for automatic white balancing. 
     BACKGROUND 
     During a laparoscopic or robotic surgery, the endoscope inside a patient&#39;s body may need to be taken out of the patient&#39;s body for various reasons, such as for lens cleaning, and then put back into the patient&#39;s body to continue its imaging function. The endoscope is generally equipped with a powerful light source, such as a light-emitting diode (LED) or a laser, wherein the light generated by the light source travels down to the distal end of the endoscope inside the patient&#39;s body to illuminate the body&#39;s cavity so that the endoscope camera can “see” and record. However, if this powerful light source is not turned off when the endoscope is taken out of the patient&#39; s body, accidentally pointing the endoscope at the eyes of any of the nearby surgical staff can cause significant retinal damage that can be potentially permanent. In order to avoid this potential health risk when the endoscope is outside of the patient&#39;s body, an operator of the endoscope is trained to immediately hold his/her palm against the end of the endoscope until the light source is manually turned off. Similarly, before inserting the endoscope back into the patient&#39;s body, the operator needs to manually turn the light source back on, hold his/her palm against the end of the endoscope to block off the light until the endoscope has entered the trocar or otherwise been inserted into the patient&#39;s body. 
     However, the manual control of turning on/off the endoscope light source can still pose risks to the surgical staff if the surgeon taking the endoscope out has poor coordination between blocking off the light and turning off the light source, or simply forgets to turn off the light source. Moreover, in certain operating room (OR) setups, it is the responsibility of a circulating nurse or a scrub technologist to turn on or off the light source depending on whether the surgeon is driving the endoscope into the patient&#39;s body or taking it out. These setups depend on the coordination between the surgical staff and the surgeon to ensure the safety of people in the OR, and therefore can have higher associated risks of accidental light exposure than having the surgeon controlling both the endoscope and the on/off state of the light source. 
     Hence, there is a need for an automatic process of turning on/off the endoscope light source when the endoscope is being inserted into/taken out of the patient&#39;s body without the problems described above. 
     SUMMARY 
     This patent disclosure provides various embodiments of a machine-learning-based detection/control technique for enabling automatically turning on/off a light source of an endoscope when the endoscope is being inserted into/taken out of a patient&#39;s body or pointed at a white object for automatic white balancing. In some embodiments, to enable automatically turning on/off an LED light source or a laser light source, a statistical classifier for video scene analysis, and in particular for distinguishing between scenes from inside the patent&#39;s body and scenes from outside of the patent&#39;s body was first constructed. Next, the statistical classifier can be applied to real-time video images captured by the endoscope to determine a state for the endoscope, such as whether the endoscope is inside the patient&#39;s body or outside of the patient&#39;s body. In this manner, a change of state of the endoscope can be automatically detected. 
     More specifically, if the state of the endoscope has changed from being inside the patient&#39;s body to being outside of the patient&#39;s body, a control signal can be immediately generated to turn off the light source for the endoscope. On the other hand, if the state of the endoscope has changed from being outside of the patient&#39;s body to being inside of the patient&#39;s body, a control signal can be immediately generated to turn on the light source for the endoscope. Hence, the disclosed control technique eliminates the need for manual intervention in turning on/off the light source for the endoscope camera. Note that the disclosed technique essentially detects transitions of the endoscope between the two states, and therefore the detection of the state changes is instant with a minimal latency. In other words, the disclosed technique can instantly detect the first moment when the endoscope is outside of the patient&#39;s body and immediately cause the light source to be turned off. 
     It is another objective of this patent disclosure to provide a technique for enabling automatically turning on/off the white balance mode of the endoscope for automatic white balancing. More specifically, a proposed auto-white-balancing technique is designed to detect and recognize that the endoscope is pointed at a white balancing object, such as a white card or a white sponge. The proposed auto-white-balancing technique can then trigger the white balancing operation, including causing the light source to be turned on to allow an automatic white balance to be performed. In some embodiments, the proposed auto-white-balancing technique can also detect the end of the white balancing operation by recognizing that the endoscope is pointed away from the white balancing object. For example, the proposed auto-white-balancing technique can include a video analysis tool for detecting a scene change from the images of the white balancing object to a different object. The proposed auto-white-balancing technique can then switch off the white balancing mode including causing the light source to be turned off again. Note that the light source can remain off until the endoscope being reinserted into the patient&#39;s body, wherein the proposed automatic light source control technique will have the light source turned on again to resume imaging. Hence, the proposed auto-white-balancing technique completely eliminates the need for manual intervention during white balancing, and therefore saves significant OR time and hassle. 
     In one aspect, a process for automatically turning on/off a light source of an endoscope during a surgical procedure in an operating room (OR) is disclosed. This process can begin by receiving real-time video images captured by a camera of the endoscope. The process next processes the real-time video images to determine whether the endoscope has been inserted into a patient&#39;s body or is outside of the patient&#39;s body. If the process determines that endoscope is outside of the patient&#39;s body and if the light source is turned on, the process then generates a first control signal to immediately turn off the light source to the endoscope. In this manner, the disclosed process ensures the safety of people in the OR and at the same time prevents sensitive information in the OR from being captured by the camera. 
     In some embodiments, the process determines whether the endoscope has been inserted into the patient&#39;s body or is outside of the patient&#39;s body by detecting one of the following two events based on the received real-time video images: (1) a first transitional event of transferring the endoscope from the inside of the patient&#39;s body to the outside of the patient&#39;s body; and (2) a second transitional event of transferring the endoscope from the outside of the patient&#39;s body to the inside of the patient&#39;s body. Note that the endoscope is determined to be outside of the patient&#39;s body when the first transitional event is detected; whereas the endoscope is determined to have been inserted into the patient&#39;s body when the second transitional event is detected. 
     In some embodiments, the process detects the first transitional event by receiving a first sequence of real-time video images captured by the camera of the endoscope. Next, the process applies a machine-learning classifier to the first sequence of real-time video images to classify each video image in the first sequence of real-time video images as: (1) a first class of images of scenes inside the patient&#39;s body; or (2) a second class of images of scenes outside of the patient&#39;s body. The process subsequently detects the first transitional event based on the classified first sequence of video images. 
     In some embodiments, after the light source has been turned off, the process further receives a second sequence of real-time video images captured by the camera of the endoscope while the endoscope remains outside of the patient&#39;s body. Next, the process applies the machine-learning classifier to the second sequence of real-time video images to classify each video image in the second sequence of real-time video images as: (1) the first class of images; or (2) the second class of images. The process subsequently detects the second transitional event based on the classified second sequence of video images. If the second transitional event is detected, the process then generates a second control signal to immediately turn on the light source to the endoscope. 
     In some embodiments, prior to detecting the first or the second transitional event using the machine-learning classifier, the process further includes the steps for training the machine-learning classifier by: receiving a plurality of training videos containing the first and second transitional events; for each training video in the plurality of training videos, annotating the set of video images in the training video as either the first class of images of scenes inside the patient&#39;s body or the second class of images of scenes outside of the patient&#39;s body; and training the machine-learning classifier to distinguish an input image as being the first class of images or the second class of images based on the annotated plurality of training videos. 
     In some embodiments, after the machine-learning classifier has been trained to correctly label an input image as being the first class of images or the second class of images, the process further trains the trained machine-learning classifier to detect a transitional event within an input sequence of video images and to distinguish the detected transitional event as being the first transitional event or the second transitional event. 
     In some embodiments, prior to annotating the set of video images in the training video, the process further includes the step of segmenting the training video into a set of video segments based on the first and second transitional events, so that each video segment in the set of video segments belongs to one of two predefined phases: (1) an inside phase when the endoscope has been inserted into the patient&#39;s body; and (2) an outside phase when the endoscope is outside of the patient&#39;s body. 
     In some embodiments, the process annotates the set of video images in the training video by: (1) for each video segment in the set of video segments of the inside phase, labeling video images within the video segment as the first class of images; and (2) for each video segment in the set of video segments of the outside phase, labeling video images within the video segment as the second class of images. 
     In some embodiments, when the endoscope is outside of the patient&#39;s body while the light source is turned off, the process includes steps of enabling an auto-white-balancing operation by: (1) receiving a first real-time video image captured by the camera of the endoscope; (2) processing the first real-time video image to determine whether the endoscope is pointed at a white-balancing object; and (3) if so, generating a third control signal to immediately turn on the light source to initiate an automatic white-balancing operation. 
     In some embodiments, the process determines whether the endoscope is pointed at a white-balancing object by using a second machine-learning classifier to classify the first real-time video image as: (1) a white-balancing object; or (2) a non-white balancing object. 
     In some embodiments, while performing the automatic white-balancing operation, the process further includes the steps of: (1) receiving a second real-time video image captured by the camera of the endoscope; (2) processing the second real-time video image to determine whether the endoscope is pointed at a non-white-balancing object; and (3) if so, generating a fourth control signal to immediately turn off the light source to terminate the automatic white-balancing operation. 
     In some embodiments, the process determines whether the endoscope is pointed at a non-white-balancing object by using the second machine-learning classifier to classify the second real-time video image as: (1) a white-balancing object; or (2) a non-white balancing object. 
     In another aspect, an endoscope system capable of automatically turning on/off a light source during a surgical procedure is disclosed. This endoscope system can include: an endoscope module; a light source module coupled to the endoscope module for providing a light source for the endoscope module; and a light-source control module coupled to the endoscope module and the light source module. Moreover, this light-source control module is configured to control an ON/OFF state of the light source for the endoscope module during a surgical procedure by: (1) receiving real-time video images captured by a camera of the endoscope system; (2) processing the real-time video images to determine whether the endoscope module has been inserted into a patient&#39;s body or is outside of the patient&#39;s body; and (3) if the endoscope module is determined to be outside of the patient&#39;s body and if the light source is turned on, generating a first control signal to immediately turn off the light source to the endoscope module. 
     In some embodiments, the light source module further includes a light source, such as an LED light source or a laser light source. The light source module additionally includes a power control unit coupled to the light source and the light-source control module. Moreover, the power control unit is configured to: receive light-source control signals from the light-source control module; and cause the light source to be turned on or turned off based on the received light-source control signals, thereby eliminating the need for manually turning on/off the light source during the surgical procedure. 
     In yet another aspect, an apparatus for automatically turning on/off a light source of an endoscope during a surgical procedure in an operating room (OR) is disclosed. This apparatus includes one or more processors and a memory coupled to the one or more processors. Moreover, the memory stores a set of instructions that, when executed by the one or more processors, cause the apparatus to control an ON/OFF state of the light source for the endoscope during the surgical procedure by: (1) receiving real-time video images captured by a camera of the endoscope; (2) processing the real-time video images to determine whether the endoscope has been inserted into a patient&#39;s body or is outside of the patient&#39;s body; and (3) if the endoscope is determined to be outside of the patient&#39;s body and if the light source is turned on, generating a control signal to immediately turn off the light source to the endoscope to ensure the safety of people in the OR and to prevent sensitive information in the OR from being captured. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The structure and operation of the present disclosure will be understood from a review of the following detailed description and the accompanying drawings in which like reference numerals refer to like parts and in which: 
         FIG. 1  shows a block diagram of an exemplary endoscope system including an automatic light-source on/off control subsystem in accordance with some embodiments described herein. 
         FIG. 2  presents a flowchart illustrating an exemplary process for automatically turning on/off a light source of an endoscope camera module during a surgical procedure in accordance with some embodiments described herein. 
         FIG. 3  presents a flowchart illustrating an exemplary process for training the disclosed statistical classifier for classifying surgical video images as being either inside or outside of a patient&#39;s body in accordance with some embodiments described herein. 
         FIG. 4  presents a photographic image illustrating a conventional way of manually turning on/off a light source for an endoscope camera module. 
         FIG. 5  conceptually illustrates a computer system with which some embodiments of the subject technology can be implemented. 
     
    
    
     DETAILED DESCRIPTION 
     The detailed description set forth below is intended as a description of various configurations of the subject technology and is not intended to represent the only configurations in which the subject technology may be practiced. The appended drawings are incorporated herein and constitute a part of the detailed description. The detailed description include s specific details for the purpose of providing a thorough understanding of the subject technology. However, the subject technology is not limited to the specific details set forth herein and may be practiced without these specific details. In some instances, structures and components are shown in block diagram form in order to avoid obscuring the concepts of the subject technology. 
     This patent disclosure provides various embodiments of a machine-learning-based detection/control technique for enabling automatically turning on/off a light source of an endoscope when the endoscope is being inserted into/taken out of a patient&#39;s body or pointed at a white object for automatic white balancing. In some embodiments, to enable automatically turning on/off an LED light source or a laser light source, a statistical classifier for video scene analysis, and in particular for distinguishing between scenes from inside the patent&#39;s body and scenes from outside of the patent&#39;s body was first constructed. Next, the statistical classifier can be applied to real-time video images captured by the endoscope to determine a state for the endoscope, such as whether the endoscope is inside the patient&#39;s body or outside of the patient&#39;s body. In this manner, a change of state of the endoscope can be automatically detected. 
     More specifically, if the state of the endoscope has changed from being inside the patient&#39;s body to being outside of the patient&#39;s body, a control signal can be immediately generated to turn off the light source for the endoscope. On the other hand, if the state of the endoscope has changed from being outside of the patient&#39;s body to being inside of the patient&#39;s body, a control signal can be immediately generated to turn on the light source for the endoscope. Hence, the disclosed control technique eliminates the need for manual intervention in turning on/off the light source for the endoscope camera. Note that the disclosed technique essentially detects transitions of the endoscope between the two states, and therefore the detection of the state changes is instant with a minimal latency. In other words, the disclosed technique can instantly detect the first moment when the endoscope is outside of the patient&#39;s body and immediately cause the light source to be turned off. 
     Note that prior to commencing any surgery, automatic white balancing is a mandatory step to prepare the endoscope. Moreover, during a surgical procedure, after the endoscope has been outside of the patient&#39;s body temporarily for cleaning or for other reasons (described in more details below), the endoscope needs to be re-calibrated, i.e., to have another white balancing done before the endoscope can be reinserted into the patient&#39;s body. Assuming the endoscope light source is turned off prior to performing the white balancing for the described safety reasons, the surgical staff would normally be required to manually turn on/off a white balancing mode, which includes manually turning on the light source to perform the white balancing operation and then manually turning off the light source when the white balancing is completed. 
     It is another objective of this patent disclosure to provide a technique for enabling automatically turning on/off the white balance mode of the endoscope for automatic white balancing. More specifically, a proposed auto-white-balancing technique is designed to detect and recognize that the endoscope is pointed at a white balancing object, such as a white card or a white sponge. The proposed auto-white-balancing technique can then trigger the white balancing operation, including causing the light source to be turned on to allow an automatic white balance to be performed. In some embodiments, the proposed auto-white-balancing technique can also detect the end of the white balancing operation by recognizing that the endoscope is pointed away from the white balancing object. For example, the proposed auto-white-balancing technique can include a video analysis tool for detecting a scene change from the images of the white balancing object to a different object. The proposed auto-white-balancing technique can then switch off the white balancing mode including causing the light source to be turned off again. Note that the light source can remain off until the endoscope being reinserted into the patient&#39;s body, wherein the proposed automatic light source control technique will have the light source turned on again to resume imaging. Hence, the proposed auto-white-balancing technique completely eliminates the need for manual intervention during white balancing, and therefore saves significant OR time and hassle. 
     During a laparoscopic or robotic surgery, the endoscope inside a patient&#39;s body may need to be taken out of the patient&#39;s body for various reasons, such as for lens cleaning, and then put back into the patient&#39;s body to continue its imaging function. A time period during the surgical procedure when the endoscope is taken out of the patient&#39;s body for one of a number of reasons and then put back into the patient&#39;s body may be referred to as an out-of-body (OOB) event. Note that each surgical procedure can include a number of OOB events which take place for a number of reasons. For example, an OOB event will occur if the endoscope lens has to be cleaned, e.g., when the endoscope lens is covered with blood. Another type of OOB event involves changing the endoscope lens from one scope size to another scope size for different anatomy/fields of view (FOV). Typically, during an OOB event the endoscope camera continues to record. The proposed techniques can use the video images captured during an OOB event to determine when to turn on/off the light source during the OOB event to ensure the safety of the surgical staff in the OR and when to turn on/off the light source for automatic white balancing. 
       FIG. 1  shows a block diagram of an exemplary endoscope system  100  including an automatic light-source on/off control subsystem  110  in accordance with some embodiments described herein. As can be seen in  FIG. 1 , endoscope system  100  includes an endoscope camera module  102 , a light source unit  104 , and a proposed automatic light-source on/off control subsystem  110  (or “light-source control subsystem  110 ” hereinafter). Light-source control subsystem  110  is coupled to an output of endoscope camera module  102  and configured to receive real-time surgical procedure videos and/or still images (collectively referred to as “surgical video images  120 ” hereinafter) during a live surgical procedure. Endoscope camera module  102  is typically the rod-like or tube-like structure that contains the physical imaging component such as a CCD camera, the optic lenses, and connections such as optic fiber for the light source. Light-source control subsystem  110  is also coupled to an input of light source module  104  and configured to generate and output automatic light source on/off control signals  130  (or “light-source control signals  130 ” hereinafter) during a live surgical procedure. As can be seen in  FIG. 1 , light source module  104  includes a physical light source  106  such as an LED light source or a laser light source, and a power control unit  108 . Power control unit  108  can be configured to receive light-source control signals  130  from light-source control subsystem  110  and immediately turn on or turn off light source  106  based on light-source control signals  130 . 
     Proposed light-source on/off control subsystem  110  can include one or more video-image processing modules including video-image processing modules  112  and  114 . Specifically, video-image processing module  112  is configured to perform a surgical scene analysis to determine whether endoscope camera module  102  is inside the patent&#39;s body or outside of the patient&#39;s body. Video-image processing module  112  is also configured to detect two transitional events: (1) when endoscope camera module  102  is transferred from the inside of the patient&#39;s body to the outside of the patient&#39;s body, also referred to as “the first transitional event” hereinafter; and (2) when endoscope camera module  102  is transferred from the outside of the patient&#39;s body to the inside of the patient&#39;s body, also referred to as “the second transitional event” hereinafter. Video-image processing module  112  is configured to generate a “POWER-OFF” signal  130 - 1  for turning off light source  106  in light source module  104  when the first transitional event is detected, and a “POWER-ON” signal  130 - 2  for turning on light source  106  in light source module  104  when the second transitional event is detected. Light-source control subsystem  110  is configured to output control signals  130 - 1  and  130 - 2  generated by video-image processing module  112  to light source module  104 . Upon receiving a control signal  130 , power control unit  108  of light source module  104  is configured to immediately turn off light source  106  for endoscope camera module  102  if the POWER-OFF signal  130 - 1  is received or immediately turn on light source  106  for endoscope camera module  102  if the POWER-ON signal  130 - 2  is received. 
     Proposed video-image processing module  114  is configured to perform a surgical scene analysis to detect the beginning and ending of a white balancing operation when endoscope camera module  102  is outside of the patient&#39;s body. Video-image processing module  114  is also configured to generate a “POWER-ON” signal  130 - 3  for turning on light source  106  in light source module  104  when the beginning of a white balancing operation is detected, or to generate a “POWER-OFF” signal  130 - 4  for turning off light source  106  in light source module  104  when the ending of a white balancing operation is detected. Light-source control subsystem  110  is configured to output the control signals  130 - 3  and  130 - 4  generated by video-image processing module  114  to light source module  104 . Upon receiving a control signal  130 , power control unit  108  of light source module  104  is configured to immediately turning on light source  106  for endoscope camera module  102  if the POWER-ON signal  130 - 3  is received or immediately turn off light source  106  for endoscope camera module  102  if the POWER-OFF signal  130 - 4  is received. Note that video-image processing module  114  does not need to be engaged when endoscope camera module  102  is inside the patient&#39;s body. In some embodiments, video-image processing module  114  becomes engaged after video-image processing module  112  detects the first transitional event, and becomes disengaged after video-image processing module  112  detects the second transitional event. 
     In some embodiments, video-processing module  112  can include a machine-learning-based or a computer-vision-based statistical classifier trained to distinguish between scenes from inside a patent&#39;s body and scenes from outside of the patent&#39;s body. Video-processing module  112  can apply this machine-learning or computer-vision-based statistical classifier to real-time surgical video images  120  captured by endoscope camera module  102  to classify a real-time scene depicted in surgical video images  120  as either (1) inside the patient&#39;s body or (2) outside of the patient&#39;s body. In some embodiments, the statistical classifier is configured to classify/label each received video image  120  as being either (1) inside the patient&#39;s body or (2) outside of the patient&#39;s body. Note that the first transitional event or the second transitional event is a continuous action represented by a consecutive sequence of video images. Hence, after a sequence of video images  120  has been classified/labeled, a transitional event can be detected if the sequence of video images  120  contains either the first transitional event or the second transitional event. While it is possible to detect the first and second transitional events based on just one or a few classified/labeled video frames, generating decisions based on a sequence of labeled video images  120  allows for identifying correlations among the sequence of video images to facilitate generating a more accurate decision on whether the first or second transitional event exists in the sequence of video images. 
     In some embodiments, if endoscope camera module  102  is initially inside the patient&#39;s body (and light source  106  is turned on) and the statistical classifier has classified one or more newly received surgical video images  120  as being outside of the patient&#39;s body, the first transitional event can be detected and video-processing module  112  immediately generates the “POWER-OFF” control signal  130 - 1 . Alternatively, if endoscope camera module  102  is initially inside the patient&#39;s body (and light source  106  is turned on) and the statistical classifier has classified one or more newly received surgical video images  120  as being inside the patient&#39;s body, no transitional event has occurred and video-processing module  112  continues monitoring the state of endoscope camera module  102  without generating any control signals  130 . Note that by detecting the first transitional event, video-processing module  112  can detect the moment endoscope camera module  102  is brought to the outside of the patient&#39;s body with a minimal latency to immediately cause the light source to be turned off. 
     Similarly, if endoscope camera module  102  is initially outside of the patient&#39;s body (and light source  106  is turned off) and the statistical classifier has classified one or more newly received surgical video images  120  as being inside the patient&#39;s body, the second transitional event can be detected and video-processing module  112  immediately generates the “POWER-ON” control signal  130 - 3 . Alternatively, if endoscope camera module  102  is initially outside of the patient&#39;s body (and light source  106  is turned off) and the statistical classifier has classified one or more newly received surgical video images  120  as being outside of the patient&#39;s body, no transitional event has occurred and video-processing module  112  continues monitoring the state of endoscope camera module  102  without generating any control signals  130 . 
     Note that another important application of the disclosed light-source control subsystem  110  is that, anytime endoscope camera module  102  is taken out of the patient&#39;s body, video-processing module  112  automatically causes light source  106  to be turned off. As a result, even if endoscope camera module  102  is then pointed at a whiteboard in the OR with sensitive information or another object in the OR associated with a privacy concern, the sensitive information can not be captured clearly or identified in the dark images captured by endoscope camera module  102  when light source  106  is off. 
     In some embodiments, prior to using the proposed statistical classifier to classify surgical video images  120 , the statistical classifier was trained based on a large number of surgical videos containing the above-described first and second transitional events. More specifically, a large number of surgical videos containing the first and second transitional events are first gathered. The gathered surgical videos can include actual surgical procedure videos performed by surgeons. Furthermore, the gathered surgical videos can include artificially generated procedure videos that are created to include the first and second transitional events. In some embodiments, each training video can contain sufficient video frames depicting the endoscope being both inside and outside the patient&#39;s body and the actions of transferring the endoscope from the inside/outside to the outside/inside of the patient&#39;s body (i.e., the first and second transitional events). 
     Next, for each of the training surgical videos, each of the video images is annotated/labeled as either a first class of image of inside a patient&#39;s body or a second class of image of outside of the patient&#39;s body. In some embodiments, prior to annotating the training surgical video, the training surgical video can be segmented by a phase segmentation engine into a set of video segments based on the first and second transitional events, and each video segment in the set of video segments can belong to one of the two phases, i.e., an inside phase when the endoscope is inside the patient&#39;s body or an outside phase when the endoscope is entirely outside of the patient&#39;s body. A person skilled in the art would appreciate that a surgical video including multiple  00 B events can contain multiple video segments of the inside phase and multiple video segments of the outside phase. Next, for each video segment of the set of video segments of the inside phase, video images within the video segment can be labeled as the first class of image of inside the patient&#39;s body; and for each video segment of the video segments of the outside phase, video images within the video segment can be labeled as the second class of image of outside of the patient&#39;s body. 
     Once all the training videos have been properly labeled, the statistical classifier can be trained based on the labeled set of surgical videos so that the trained statistical classifier can be used to distinguish a given input video frame as the first class of image or the second class of image. Moreover, the statistical classifier can also be trained to distinguish a transitional event within a sequence of video images  120  as being the first transitional event or the second transitional event based on the ability to correctly label each image frame as being the first class of image or the second class of image. In some embodiments, the proposed statistical classifier can include a feature-based model which can be used to distinguish an input image as being the first class of image or the second class of image based on a set of user-specified image features, such as the statistics of the color, the texture, the contrast, among other features. In other embodiments, the proposed statistical classifier can include a deep learning model that does not require the user to manually identify the features. 
     In various embodiments, the proposed statistical classifier can include a machine-learning model constructed based on a regression model, a deep neural network-based model, a support vector machine, a decision tree, a Naive Bayes classifier, a Bayesian network, or a k-nearest neighbors (KNN) model. In some embodiments, each of these machine-learning models is constructed based on a convolutional neural network (CNN) architecture, a recurrent neural network (RNN) architecture, or another form of deep neural network (DNN) architecture. 
     As mentioned above, video-image processing module  114  is configured to generate a “POWER-ON” control signal  130 - 3  for light source module  104  when the beginning of a white balancing operation is detected, or to generate a “POWER-OFF” control signal  130 - 4  for light source module  104  when the ending of a white balancing operation is detected. In some embodiments, video-processing module  114  can include a machine-learning-based or a computer-vision-based statistical classifier trained to detect a white balancing object. Some applicable white balancing objects can include a white balancing card, a white sponge, a whiteboard, a piece of white paper, a piece of white cloth, and any other suitable white medium. 
     In some embodiments, the statistical classifier is configured to classify/label each received video image  120  as being either (1) a white balancing object or (2) a non-white balancing object. 
     In some embodiments, if endoscope camera module  102  is outside of the patient&#39;s body during an  00 B event with light source  106  turned off, and the statistical classifier has classified one or more newly received surgical video images  120  as a white balancing object, video-processing module  114  immediately generates the “POWER-ON” control signal  130 - 3  to trigger the white balancing operation. Alternatively, if endoscope camera module  102  is initially pointed at a white balancing object with light source  106  turned on for white balancing, and the statistical classifier has classified one or more newly received surgical video images  120  as being a non-white balancing object, video-processing module  114  immediately generates the “POWER-OFF” control signal  130 - 4  to terminate the white balancing operation. 
     In some embodiments, prior to using the proposed statistical classifier to classify surgical video images  120  as being either a white balancing object or a non-white balancing object, the statistical classifier can be trained on a different data set to identify different types of white balancing objects and non-white-balancing objects. 
     In some embodiments, the disclosed automatic light-source control subsystem  110  can be implemented in computer software, in electronic hardware, or in combinations of both, and as a separate module from light source module  104  (as shown in  FIG. 1 ). In some other embodiments, the disclosed automatic light-source control subsystem  110  can be implemented in computer software, in electronic hardware, or in combinations of both, and as a component of light source module  104  (not shown in  FIG. 1 ). 
       FIG. 2  presents a flowchart illustrating an exemplary process  200  for automatically turning on/off a light source of an endoscope camera module during a surgical procedure in accordance with some embodiments described herein. In one or more embodiments, one or more of the steps in  FIG. 2  may be omitted, repeated, and/or performed in a different order. Accordingly, the specific arrangement of steps shown in  FIG. 2  should not be construed as limiting the scope of the technique. 
     Process  200  may begin by receiving a sequence of real-time video images captured by the endoscope camera module when the light source is turned on (step  202 ). In particular, the endoscope camera module is initially inside the patient&#39;s body. Next, process  200  analyzes a sequence of newly received video images to determine whether the endoscope camera module is inside the patient&#39;s body or outside of the patient&#39;s body (step  204 ). In some embodiments, process  200  uses a statistical classifier to classify each image in the sequence of newly received video images as either a first class of image of being inside the patient&#39;s body or a second class of image of being outside of the patient&#39;s body. Process  200  can determine a state of the endoscope camera module as being inside the patient&#39;s body or outside of the patient&#39;s body based on the classified sequence of images. More specifically, process  200  can determine the state of the endoscope camera module as being outside of the patient&#39;s body by detecting the first transitional event within the sequence of newly received video images. 
     If process  200  determines that the endoscope camera module is still inside the patient&#39;s body at step  204 , process  200  returns to step  202  and continues monitoring the state of the endoscope camera module. Otherwise, if process  200  determines that the endoscope camera module is outside of the patient&#39;s body (i.e., the first transitional event is detected), process  200  generates a control signal for turning off the light source for safety and other reasons (step  206 ). The control signal is used by a light source module to immediately turn off the light source to prevent accidental eye damages. 
     Next, when the endoscope camera module is outside of the patient&#39;s body with the light source turned off, process  200  analyzes a real-time video image to determine whether the endoscope camera module is pointed at a white balancing object (step  208 ). For example, process  200  can analyze the video image to identify a predefined white balancing object within the video image. If not, process  200  can return to step  208  and continue to look for the cue indicating the beginning of the white balancing operation. However, if a white balancing object is detected at step  208 , process  200  generates another control signal for turning on the light source for white balancing (step  210 ). 
     Next, when the endoscope camera module is outside of the patient&#39;s body with the light source turned on, process  200  continues to analyze real-time video images to determine whether the white balancing operation is completed (step  212 ). If not, process  200  can return to step  212  and continue to look for the cue indicating the ending of the white balancing operation. Otherwise, the white balancing operation is completed and process  200  generates yet another control signal for turning off the light source (step  214 ). Process  200  subsequently analyzes another sequence of newly received video images to determine if the endoscope camera module remains outside of the patient&#39;s body (step  216 ). Similarly, process  200  can use the statistical classifier to classify each image in the sequence of newly received video images as either the first class of image of being inside the patient&#39;s body or the second class of image of being outside of the patient&#39;s body. More specifically, process  200  can determine the state of the endoscope camera module as being inside the patient&#39;s body by detecting the second transitional event within the sequence of newly received video images. 
     If process  200  determines that the endoscope camera module remains outside of the patient&#39;s body at step  216 , process  200  returns to step  216  to continue monitoring the state of the endoscope camera module. Otherwise, if process  200  determines that the endoscope camera module is inside the patient&#39;s body at step  216  (i.e., the second transitional event is detected), process  200  generates yet another control signal for turning on the light source (step  218 ). The control signal is used by the light source module to immediately turn on the light source to resume endoscopic imaging. 
       FIG. 3  presents a flowchart illustrating an exemplary process  300  for training the disclosed statistical classifier for classifying surgical video images as being either inside or outside of a patient&#39;s body in accordance with some embodiments described herein. In one or more embodiments, one or more of the steps in  FIG. 3  may be omitted, repeated, and/or performed in a different order. Accordingly, the specific arrangement of steps shown in  FIG. 3  should not be construed as limiting the scope of the technique. 
     Process  300  begins by collecting a large number of training surgical videos, wherein each training surgical video contains one or more OOB events (step  302 ). As described above, each of the OOB events typically includes the first transitional event (i.e., taking the endoscope out of the patient&#39;s body) and the second transitional event (i.e., putting the endoscope back into the patient&#39;s body). Note that the collected training surgical videos can include both actual surgical procedure videos performed by surgeons and artificially generated procedure videos created specifically to provide training data for training the statistical classifier. 
     Process  300  next segments each of the training surgical videos into a set of video segments based on the associated one or more OOB events, wherein each video segment in the set of video segments belongs to either the inside phase when the endoscope is inside the patient&#39;s body or the outside phase when the endoscope is outside of the patient&#39;s body (step  304 ). Note that the set of transitional events within a surgical video provides the natural phase boundaries for segmenting the surgical video into the inside phases and the outside phases. Also note that a surgical video containing multiple OOB events can be segmented into a set of corresponding video segments that is composed of multiple inside phases and multiple outside phases. 
     Next, for each video segment of the inside phase from a segmented training surgical video, process  300  labels video images within the video segment as the first class of image of inside the patient&#39;s body; and for each video segment of the outside phase of the segmented surgical video, process  300  labels video images within the video segment as the second class of image of outside of the patient&#39;s body (step  306 ). After all the training surgical videos have been properly labeled, process  200  proceeds to train the statistical classifier based on the labeled set of training surgical videos so that the trained statistical classifier can be used to distinguish a given input video frame as the first class of image or the second class of image (step  308 ). 
       FIG. 4  presents a photographic image illustrating a conventional way of manually turning on/off a light source for an endoscope camera module  400 . As can be seen in  FIG. 4 , endoscope camera module  400  includes light source on/off button  402  located at the proximal end of endoscope camera module  400 . After a surgeon takes endoscope camera module  400  out of a patient&#39;s body, the surgeon has to hold endoscope camera module  400  on one end and manually manipulate light source on/off button  402  on the other end of endoscope camera module  400 . The disclosed automatic light-source control techniques eliminate the needs for manually turning on/off the endoscope light sources during a number of significant events within a surgical procedure, thereby significantly reducing or eliminating the chances of any of the surgical staff being harmed by the powerful endoscope light source, and saving significant OR time and hassle. 
       FIG. 5  conceptually illustrates a computer system with which some embodiments of the subject technology can be implemented. Computer system  500  can be a client, a server, a computer, a smartphone, a PDA, a laptop, or a tablet computer with one or more processors embedded therein or coupled thereto, or any other sort of computing device. Such a computer system includes various types of computer-readable media and interfaces for various other types of computer-readable media. Computer system  500  includes a bus  502 , processing unit(s)  512 , a system memory  504 , a read-only memory (ROM)  510 , a permanent storage device  508 , an input device interface  514 , an output device interface  506 , and a network interface  516 . In some embodiments, computer system  500  is a part of a robotic surgical system. 
     Bus  502  collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of computer system  500 . For instance, bus  502  communicatively connects processing unit(s)  512  with ROM  510 , system memory  504 , and permanent storage device  508 . 
     From these various memory units, processing unit(s)  512  retrieves instructions to execute and data to process in order to execute various processes described in this patent disclosure, including the above-described processes of automatically turning on/off a light source of an endoscope camera module during a surgical procedure and training the disclosed statistical classifier for classifying surgical video images as being either inside or outside of a patient&#39;s body in conjunction with  FIGS. 1-3 . The processing unit(s)  512  can include any type of processor, including, but not limited to, a microprocessor, a graphics processing unit (GPU), a tensor processing unit (TPU), an intelligent processor unit (IPU), a digital signal processor (DSP), a field-programmable gate array (FPGA), and an application-specific integrated circuit (ASIC). Processing unit(s)  512  can be a single processor or a multi-core processor in different implementations. 
     ROM  510  stores static data and instructions that are needed by processing unit(s)  512  and other modules of the computer system. Permanent storage device  508 , on the other hand, is a read-and-write memory device. This device is a non-volatile memory unit that stores instructions and data even when computer system  500  is off. Some implementations of the subject disclosure use a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive) as permanent storage device  508 . 
     Other implementations use a removable storage device (such as a floppy disk, flash drive, and its corresponding disk drive) as permanent storage device  508 . Like permanent storage device  508 , system memory  504  is a read-and-write memory device. However, unlike storage device  508 , system memory  504  is a volatile read-and-write memory, such as a random access memory. System memory  504  stores some of the instructions and data that the processor needs at runtime. In some implementations, various processes described in this patent disclosure, including the above-described processes of automatically turning on/off a light source of an endoscope camera module during a surgical procedure and training the disclosed statistical classifier for classifying surgical video images as being either inside or outside of a patient&#39;s body in conjunction with  FIGS. 1-3 , are stored in system memory  504 , permanent storage device  508 , and/or ROM  510 . From these various memory units, processing unit(s)  512  retrieve instructions to execute and data to process in order to execute the processes of some implementations. 
     Bus  502  also connects to input and output devices  514  and  506 . Input devices  514  enable the user to communicate information to and select commands for the computer system. Input devices  514  can include, for example, alphanumeric keyboards and pointing devices (also called “cursor control devices”). Output devices  506  enable, for example, the display of images generated by computer system  500 . Output devices  506  can include, for example, printers and display devices, such as cathode ray tubes (CRT) or liquid crystal displays (LCD). Some implementations include devices such as a touchscreen that functions as both input and output devices. 
     Finally, as shown in  FIG. 5 , bus  502  also couples computer system  500  to a network (not shown) through a network interface  516 . In this manner, the computer can be a part of a network of computers (such as a local area network (“LAN”), a wide area network (“WAN”), an intranet, or a network of networks, such as the Internet. Any or all components of computer system  500  can be used in conjunction with the subject disclosure. 
     The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed in this patent disclosure may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. 
     The hardware used to implement the various illustrative logics, logical blocks, modules, and circuits described in connection with the aspects disclosed herein may be implemented or performed with a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable-logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of receiver devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Alternatively, some steps or methods may be performed by circuitry that is specific to a given function. 
     In one or more exemplary aspects, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or code on a non-transitory computer-readable storage medium or non-transitory processor-readable storage medium. The steps of a method or algorithm disclosed herein may be embodied in processor-executable instructions that may reside on a non-transitory computer-readable or processor-readable storage medium. Non-transitory computer-readable or processor-readable storage media may be any storage media that may be accessed by a computer or a processor. By way of example but not limitation, such non-transitory computer-readable or processor-readable storage media may include RAM, ROM, EEPROM, flash memory, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that may be used to store desired program code in the form of instructions or data structures and that may be accessed by a computer. Disk and disc, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above are also included within the scope of non-transitory computer-readable and processor-readable media. Additionally, the operations of a method or algorithm may reside as one or any combination or set of codes and/or instructions on a non-transitory processor-readable storage medium and/or computer-readable storage medium, which may be incorporated into a computer-program product. 
     While this patent document contains many specifics, these should not be construed as limitations on the scope of any disclosed technology or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular techniques. Certain features that are described in this patent document in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination. 
     Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. Moreover, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments. 
     Only a few implementations and examples are described, and other implementations, enhancements and variations can be made based on what is described and illustrated in this patent document.