Patent Description:
This patent application claims priority to and the benefit of the filing date of <CIT>.

Inventive aspects generally relate to video content searching. In particular, they relate to the searching medical procedure video recordings for certain subject matter of interest, and the retrieval of video clips that show the subject matter of interest.

It is desirable to have the ability to locate certain subject matter of interest in a video recording, without having knowledge of the precise point in the video recording at which the sought after subject matter is shown. Thus, it is desirable to have to have video search capabilities that enable a user to perform key word or key phrase subject matter searches of video recordings.

Subject matter searches of video recordings typically involve an algorithmic approach that relies exclusively on analysis of image data. See, for example <CIT>, entitled "Method and System for Segmenting Image Data". Generally speaking, analyses of image data try to identify certain patterns that exist among the various pixels of an image (e.g., a frame of a video) that are determined to be indicative of various subject matter of interest. Image characteristics used to perform these analyses may include patterns in pixel color, pixel intensity, lines or contours in the image formed by adjacent pixels, and the like. The subject matter of an image by analysis of image data is challenging and computationally intensive, an observation with which <CIT> agrees. Additionally, the results returned are often inaccurate.

In view of the limitations associated with determining the subject matter of an image exclusively using image data analysis, an improved system and method for performing subject matter searches of video recordings is desirable. In one aspect, a video recording is associated with various system events taking place on a medical system. Information about the system events taking place on the medical system provide an additional data set that may be indicative of certain events taking place in the video recording. When searching the recorded video for specific subject matter of interest, analysis of this additional data set can be used to augment image data analysis of the video recording. This search approach would be preferred over a search approach that relies exclusively on image data analysis if it can minimize the computational intensity of the search and improve the search accuracy.

<CIT> discloses systems and methods in which local tissue diagnostic measurements are correlated with archival local tissue diagnostic data from prior tissue analyses to supplement diagnostic measurements with tissue analysis data from prior tissue analyses having similar local tissue diagnostic data. The tissue analysis data may include information such as pathology data, outcome data, and diagnosis data. The archived local tissue diagnostic data and the tissue analysis data may be stored in a database, and employed for a wide variety of methods, involving preoperative, intraoperative, and/or postoperative phases of a medical procedure. Methods and systems are also provided for displaying, on a medical image shown in a user interface, hyperlinked reference markers associated with tissue analyses, where the reference markers are shown at locations corresponding to local tissue analyses, and where associated diagnostic data and/or tissue analysis may be viewed by selecting a given reference marker.

The following summary introduces certain aspects of the inventive subject matter in order to provide a basic understanding. This summary is not an extensive overview of the inventive subject matter, and it is not intended to identify key or critical elements or to delineate the scope of the inventive subject matter. Although this summary contains information that is relevant to various aspects and embodiments of the inventive subject matter, its sole purpose is to present some aspects and embodiments in a general form as a prelude to the more detailed description below.

The present invention provides a method defined by independent claim <NUM>.

A method of searching recorded video is disclosed. The method involves receiving a user command to locate one or more video clips showing a medical event of interest from one or more procedure video recordings. Next, one or more system events likely to correspond to the occurrence of the medical event of interest are identified from one or more procedure event logs. Using one or more timestamps corresponding to each of the one or more system events, one or more candidate video clips are identified from the one or more procedure video recordings. By performing an analysis of image data, a determination is made with regards to whether each of the one or more candidate video segments is an identified video segment that contains the medical event of interest. At least one identified video segment is presented to a user.

A computing device for performing a video search is defined by independent claim <NUM>.

This description and the accompanying drawings that illustrate inventive aspects, embodiments, implementations, or applications should not be taken as limiting-the claims define the protected invention. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the scope of this description and the claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail in order not to obscure the invention.

Further, this description's terminology is not intended to limit the invention. For example, spatially relative terms-such as "beneath", "below", "lower", "above", "upper", "proximal", "distal", and the like-may be used to describe one element's or feature's relationship to another element or feature as illustrated in the figures. These spatially relative terms are intended to encompass different positions (i.e., locations) and orientations (i.e., rotational placements) of a device in use or operation in addition to the position and orientation shown in the figures. For example, if a device in the figures is turned over, elements described as "below" or "beneath" other elements or features would then be "above" or "over" the other elements or features. Thus, the exemplary term "below" can encompass both positions and orientations of above and below. A device may be otherwise oriented (rotated <NUM> degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly. Likewise, descriptions of movement along and around various axes includes various special device positions and orientations. In addition, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context indicates otherwise. And, the terms "comprises", "comprising", "includes", and the like specify the presence of stated features, steps, operations, elements, and/or components but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and/or groups. Components described as coupled may be electrically or mechanically directly coupled, or they may be indirectly coupled via one or more intermediate components.

Aspects of the invention are described primarily in terms of an implementation using a da Vinci® Surgical System (specifically, a Model IS4000, marketed as the da Vinci® Xi™ HD™ Surgical System), commercialized by Intuitive Surgical, Inc. of Sunnyvale, California. Knowledgeable persons will understand, however, that inventive aspects disclosed herein may be embodied and implemented in various ways, including robotic and, if applicable, non-robotic embodiments and implementations. Implementations on da Vinci® Surgical Systems (e.g., the Model IS4000 da Vinci® Xi™ Surgical System, the Model IS3000 da Vinci Si® Surgical System) are merely exemplary and are not to be considered as limiting the scope of the inventive aspects disclosed herein.

The present disclosure describes a medical system that includes an image capture device for recording a medical procedure performed using the medical system. Additionally, the medical system includes a computer processor, and a memory device on which procedure event logs are stored. The procedure event logs record the occurrence of certain events taking place on the medical system. The procedure event logs further record the times at which these events took place. The medical system can synchronize the system events of the procedure event logs with corresponding medical procedure video recordings.

In one aspect, a user of a medical seeks to view a specific segment of recorded medical video that shows a specific portion of a medical procedure. The user inputs via a user interface one or more keywords that are descriptive of the subject matter shown in the sought after video segment. These inputs can be made using one of a variety of known input methods, such as through an external keyboard coupled to the medical system, through a touchscreen text entry interface of the medical system, or through a voice control interface of the medical system. A computer processor in data communication with the medical system uses information stored in the procedure event logs to identify one or more candidate video segments (or "video clips") of recorded medical video that are likely to include the subject matter of interest. This identification is possible because of known associations between (<NUM>) certain types of entries in the procedure event logs, and (<NUM>) certain medical events of interest (which are captured in the video recording of the medical procedure). Next, the computer algorithm applies to the candidate video clips (and not to any of the remaining video recording) an algorithm that analyzes at the image data contained in the frames of each of identified video clip, to determine whether any frames show the subject matter of interest being searched for. If the image data analysis algorithm affirmatively identifies one or more identified video clips as containing the subject matter of interest, then these video clips are presented to the user.

Referring now to the drawings, in which like reference numerals represent like parts throughout the several views, <FIG> is a plan view of a minimally invasive teleoperated surgical system <NUM>, typically used for performing a minimally invasive diagnostic or medical procedure on a patient <NUM> who is lying on an operating table <NUM>. The system includes a surgeon's console <NUM> for use by a surgeon <NUM> during the procedure. One or more assistants <NUM> may also participate in the procedure. The minimally invasive teleoperated surgical system <NUM> further includes a patient-side cart <NUM> and an electronics cart <NUM>. The patient-side cart <NUM> can manipulate at least one removably coupled surgical instrument <NUM> through a minimally invasive incision in the body of the patient <NUM> while the surgeon <NUM> views the surgical site through the surgeon's console <NUM>. An image of the surgical site can be obtained by an endoscope <NUM>, such as a stereoscopic endoscope, which can be manipulated by the patient-side cart <NUM> to orient the endoscope <NUM>. Computer processors located on the electronics cart <NUM> can be used to process the images of the surgical site for subsequent display to the surgeon <NUM> through the surgeon's console <NUM>. The number of surgical instruments <NUM> used at one time will generally depend on the diagnostic or medical procedure and the space constraints within the operating room among other factors. If it is necessary to change one or more of the surgical instruments <NUM> being used during a procedure, an assistant <NUM> can remove the surgical instrument <NUM> from the patient-side cart <NUM>, and replace it with another surgical instrument <NUM> from a tray <NUM> in the operating room.

<FIG> is a perspective view of the surgeon's console <NUM>. The surgeon's console <NUM> includes a left eye display <NUM> and a right eye display <NUM> for presenting the surgeon <NUM> with a coordinated stereoscopic view of the surgical site that enables depth perception. The console <NUM> further includes one or more control inputs <NUM>. One or more surgical instruments installed for use on the patient-side cart <NUM> (shown in <FIG>) move in response to surgeon <NUM>'s manipulation of the one or more control inputs <NUM>. The control inputs <NUM> can provide the same mechanical degrees of freedom as their associated surgical instruments <NUM> (shown in <FIG>) to provide the surgeon <NUM> with telepresence, or the perception that the control inputs <NUM> are integral with the instruments <NUM> so that the surgeon has a strong sense of directly controlling the instruments <NUM>. To this end, position, force, and tactile feedback sensors (not shown) may be employed to transmit position, force, and tactile sensations from the surgical instruments <NUM> back to the surgeon's hands through the control inputs <NUM>.

The surgeon's console <NUM> is usually located in the same room as the patient so that the surgeon can directly monitor the procedure, be physically present if necessary, and speak to a patient-side assistant directly rather than over the telephone or other communication medium. But, the surgeon can be located in a different room, a completely different building, or other remote location from the patient allowing for remote medical procedures.

<FIG> is a perspective view of the electronics cart <NUM>. The electronics cart <NUM> can be coupled with the endoscope <NUM> and includes a computer processor to process captured images for subsequent display, such as to a surgeon on the surgeon's console, or on another suitable display located locally and/or remotely. For example, if a stereoscopic endoscope is used, a computer processor on electronics cart <NUM> can process the captured images to present the surgeon with coordinated stereo images of the surgical site. Such coordination can include alignment between the opposing images and can include adjusting the stereo working distance of the stereoscopic endoscope. As another example, image processing can include the use of previously determined camera calibration parameters to compensate for imaging errors of the image capture device, such as optical aberrations. The captured images can also be stored onto a memory device for later viewing. Optionally, equipment in electronics cart may be integrated into the surgeon's console or the patient-side cart, or it may be distributed in various other locations in the operating room.

<FIG> is a diagrammatic illustration of one embodiment of teleoperated surgical system <NUM>. A surgeon's console <NUM> (such as surgeon's console <NUM> in <FIG>) can be used by a surgeon to control a patient-side cart <NUM> (such as patent-side cart <NUM> in <FIG>) during a minimally invasive procedure. The patient-side cart <NUM> can use an imaging device, such as a stereoscopic endoscope, to capture images of a surgical site and output the captured images to a computer processor located on an electronics cart <NUM> (such as the electronics cart <NUM> in <FIG>). The computer processor typically includes one or more data algorithming boards purposed for executing computer readable code stored in a non-volatile memory device of the computer processor. The computer processor can process the captured images in a variety of ways prior to any subsequent display. For example, the computer processor can overlay the captured images with a virtual control interface prior to displaying the combined images to the surgeon via the surgeon's console <NUM>.

Additionally or in the alternative, the captured images can undergo image processing by a computer processor located outside of electronics cart <NUM>. In one embodiment, teleoperated surgical system <NUM> includes an optional a computer processor <NUM> (as indicated by dashed line) similar to the computer processor located on electronics cart <NUM>, and patient-side cart <NUM> outputs the captured images to computer processor <NUM> for image processing prior to display on the surgeon's console <NUM>. In another embodiment, captured images first undergo image processing by the computer processor on electronics cart <NUM> and then undergo additionally image processing by computer processor <NUM> prior to display on the surgeon's console <NUM>. Teleoperated surgical system <NUM> can include an optional display <NUM>, as indicated by dashed line. Display <NUM> is coupled with the computer processor located on the electronics cart <NUM> and with computer processor <NUM>, and captured images processed by these computer processors can be displayed on display <NUM> in addition to being displayed on a display of the surgeon's console <NUM>.

<FIG> is a perspective view of a patient-side cart <NUM> of a minimally invasive teleoperated surgical system, in accordance with embodiments of the present invention. The patient-side cart <NUM> includes one or more support assemblies <NUM>. A surgical instrument manipulator <NUM> is mounted at the end of each support assembly <NUM>. Additionally, each support assembly <NUM> can optionally include one or more unpowered, lockable setup joints that are used to position the attached surgical instrument manipulator <NUM> with reference to the patient for surgery. As depicted, the patient-side cart <NUM> rests on the floor. In other embodiments, operative portions of the patient-side cart can be mounted to a wall, to the ceiling, to the operating table <NUM> that also supports the patient's body <NUM>, or to other operating room equipment. Further, while the patient-side cart <NUM> is shown as including four surgical instrument manipulators <NUM>, more or fewer surgical instrument manipulators <NUM> may be used.

A functional minimally invasive teleoperated surgical system will generally include a vision system portion that enables a user of the teleoperated surgical system to view the surgical site from outside the patient's body <NUM>. The vision system typically includes a camera instrument <NUM> for capturing video images and one or more video displays for displaying the captured video images. Camera instrument <NUM> can be similar to endoscope <NUM> shown at <FIG>. In some surgical system configurations, the camera instrument <NUM> includes optics that transfer the images from a distal end of the camera instrument <NUM> to one or more imaging sensors (e.g., CCD or CMOS sensors) outside of the patient's body <NUM>. Alternatively, the imaging sensor(s) can be positioned at the distal end of the camera instrument <NUM>, and the signals produced by the sensor(s) can be transmitted along a lead or wirelessly for processing and display on the one or more video displays. One example of a video display is the stereoscopic display on the surgeon's console in surgical systems commercialized by Intuitive Surgical, Inc. , Sunnyvale, California.

Referring to <FIG>, mounted to each surgical instrument manipulator <NUM> is a surgical instrument <NUM> that operates at a surgical site within the patient's body <NUM>. Each surgical instrument manipulator <NUM> can be provided in a variety of forms that allow the associated surgical instrument to move with one or more mechanical degrees of freedom (e.g., all six Cartesian degrees of freedom, five or fewer Cartesian degrees of freedom, etc.). Typically, mechanical or control constraints restrict each manipulator <NUM> to move its associated surgical instrument around a center of motion on the instrument that stays stationary with reference to the patient, and this center of motion is typically located at the position where the instrument enters the body.

Surgical instruments <NUM> are controlled through computer-assisted teleoperation. A functional minimally invasive teleoperated surgical system includes a control input that receives inputs from a user of the teleoperated surgical system (e.g., a surgeon or other medical person). The control input is in communication with one or more computer-controlled teleoperated actuators, to which surgical instrument <NUM> is coupled. In this manner, the surgical instrument <NUM> moves in response to a medical person's movements of the control input. In one embodiment, one or more control inputs are included in a surgeon's console such as surgeon's console <NUM> shown at <FIG>. A surgeon can manipulate control inputs <NUM> of surgeon's console <NUM> to operate teleoperated actuators of patient-side cart <NUM>. The forces generated by the teleoperated actuators are transferred via drivetrain mechanisms, which transmit the forces from the teleoperated actuators to the surgical instrument <NUM>.

Referring to <FIG>, a surgical instrument <NUM> and a cannula <NUM> are removably coupled to manipulator <NUM>, with the surgical instrument <NUM> inserted through the cannula <NUM>. One or more teleoperated actuators of the manipulator <NUM> move the surgical instrument <NUM> as a whole. The manipulator <NUM> further includes an instrument carriage <NUM>. The surgical instrument <NUM> is detachably connected to the instrument carriage <NUM>. The instrument carriage <NUM> houses one or more teleoperated actuators that provide a number of controller motions that the surgical instrument <NUM> translates into a variety of movements of an end effector on the surgical instrument <NUM>. Thus the teleoperated actuators in the instrument carriage <NUM> move only one or more components of the surgical instrument <NUM> rather than the instrument as a whole. Inputs to control either the instrument as a whole or the instrument's components are such that the input provided by a surgeon or other medical person to the control input (a "master" command) is translated into a corresponding action by the surgical instrument (a "slave" response).

<FIG> shows a schematic view of a medical system <NUM>. Medical system <NUM> includes a teleoperated surgical system <NUM>. As indicated by dashed line, medical system <NUM> can optionally include a computer <NUM> in data communication with teleoperated surgical system <NUM>. In one embodiment, computer <NUM> is a personal computer. In alternate embodiments, computer <NUM> is a server or a mainframe. Data communication between computer <NUM> and teleoperated surgical system <NUM> can be via a variety of methods known in the art (e.g., an Local Area Network (LAN) connection, internet connection (wired or wireless), or similar data communication means). Additionally or in the alternative, this data communication can occur by means of data transfers from teleoperated surgical system <NUM> to computer <NUM> using a mobile hard drive or other similar data storage device. Computer <NUM> can be used to access information stored on a memory device of teleoperated surgical system <NUM>, and the teloperated surgical system <NUM> can be used to access information stored on a memory device of computer <NUM>. A keyboard of computer <NUM> can provide a user interface for text entry. The text entered using the keyboard can be operated on by a computer processor of computer <NUM>, as well as by a computer processor of teleoperated surgical system <NUM>.

Medical system <NUM> includes a memory device on which one or more procedure event logs are maintained. <FIG> shows select portions of an exemplary procedure event log. In one embodiment, the memory device is memory device <NUM> located on electronics cart <NUM>. In an alternate embodiment, the procedure event logs are maintained on a memory device located on surgeon's console <NUM>. In yet another embodiment, the procedure event logs are maintained on a memory device located on computer <NUM>.

The entries of the procedure event logs can include records of various interactions between teleoperated surgical system <NUM> and a medical person using teleoperated surgcial system <NUM>. In one embodiment, procedure event logs are automatically generated and automatically maintained in parallel with other operations performed by teleoperated surgical system <NUM> (e.g., computer-assisted teleoperation, video recording, video display, etc.). Input by a medical person is not required for the generation and maintenance of the procedure event logs of medical system <NUM>.

7A is a flow diagram of an image capture and recording algorithm <NUM> of a medical system. 7B is a flow diagram of an algorithm that populates procedure event logs of the medical system in parallel with image capturing and recording. <FIG> is a flow diagram of an algorithm that: (<NUM>) uses the procedure event log data to identify one or more candidate video segments (or "video clips") of the captured video during which certain medical events of interest sought by a user of a device likely look place; and (<NUM>) applies to the candidate video clips (and not to any of the remaining video recording) an algorithm that looks at the image data contained in the frames of each of candidate video clip, to determine whether any frames show the medical events of interest that is sought after by a device user.

In one embodiment, algorithms <NUM>, <NUM>, and <NUM> are implemented by processor <NUM> of medical system <NUM> (shown at <FIG>). In alternate embodiments, one or more of algorithms <NUM>, <NUM>, and <NUM> can be implemented by a processor other than processor <NUM>.

7A is a flow diagram showing an image capturing and recording algorithm <NUM> of a medical system. In one embodiment, the medical system includes a teleoperated surgical system. Referring to FIG. 7A, at <NUM>, video images are captured by an image capture device. In one embodiment, the image capture device is an endoscope (e.g., endoscope <NUM> at <FIG>) including an active pixel sensor (APS) array. The APS array is composed of red pixel sensors, green pixel sensors, and blue pixel sensors, which are arranged in a periodic pattern. Each pixel sensor of the APS array includes a photodetector and a light filter that limits the light frequencies to which the photodetector is exposed. The light filter of each red pixel sensor lets through only light that is in the red visible light portion of the electromagnetic (EM) radiation spectrum. Similarly, the light filter of each green pixel lets through only light that is in the green visible light portion of the EM spectrum, and the light filter of each blue pixel lets through only light that is in the blue visible light portion of the EM spectrum. Accordingly, each pixel sensor of the APS array will respond only to light that is within the EM spectrum range of its corresponding light filter. All other light frequencies that reach a pixel sensor are filtered out by its light filter and never reach its photodetector. Accordingly each pixel sensor of the APS array will generally have no response to light that is outside the EM spectrum range of its optical filter. A red pixel sensor will not respond to non-red light because the light filter of the red pixel sensor prevents all non-red light from reaching the red pixel sensor's photodetector. A green pixel sensor will not respond to non-green light because the light filter of the green pixel sensor prevents all non-green light from reaching the green pixel sensor's photodetector. And, a blue pixel sensor will not respond to non-blue light because the light filter of the blue pixel sensor prevents all non-blue light from reaching the blue pixel sensor's photodetector.

In one embodiment, the APS array described above is configured to produce raw bayer pattern images. Generally, each pixel of a raw bayer pattern image corresponds directly one-to-one to an individual pixel sensor of the APS array. As discussed previously, the APS array is composed of red pixel sensors, green pixel sensors, and blue pixel sensors, which are arranged in a periodic pattern. When the periodic arrangement of the various colored pixel sensors in the APS array is known, based on this color information, it is known what color each pixel of the raw bayer pattern image should be. If the pixel of the raw bayer pattern image corresponds to a red pixel sensor of the APS array, then the pixel should be red. If the pixel of the raw bayer pattern image corresponds to a green pixel sensor of the APS array, then the pixel should be green. If the pixel of the raw bayer pattern image corresponds to a blue pixel sensor of the APS array, then the pixel should be blue.

Pixel intensity is another piece of information needed to generate a raw bayer pattern image. This information is provided by the signals produced by the photoreceptor of each pixel sensor of the APS array. Consider for example a pixel of a raw bayer pattern image that corresponds with a red pixel sensor of the APS array. As discussed previously, the ability to determine that the pixel is associated with a red pixel sensor is based on prior knowledge of the periodic arrangement of the various colored pixel sensors of the APS array. The question, then, is whether this pixel of the raw bayer pattern image should be: (<NUM>) a faint red; or (<NUM>) a deep, saturated red. Whether this pixel is faint red or a deep, saturated red is determined by the amount of light that reaches the photoreceptor of the corresponding red pixel sensor of the APS array. As discussed previously, the light filter of the red pixel sensor blocks all non-red light, and allows only red light to pass through to the photoreceptor. Thus, whether this pixel is faint red or a deep, saturated red is determined by the amount of red light to which the red pixel sensor (or more precisely, its photoreceptor) is exposed. The more light that reaches a photoreceptor of a pixel sensor, the more intense the color of its corresponding pixel of a raw bayer pattern image. These principles apply similarly to green pixel sensors and blue pixels sensors of the APS array.

At <NUM>, raw bayer pattern image data undergoes video processing to convert (<NUM>) raw bayer pattern images as captured by the endoscope, to (<NUM>) red-green-blue (RGB) images suitable for display. In one embodiment, this video processing at <NUM> involves a de-mosaicking processing module. Generally speaking, RGB images are based on an additive color model. The color of each pixel of an RGB image is determined by adding together an appropriate intensity of red color, green color, and blue color. Starting with a raw bayer pattern image, the de-mosaicking processing module creates an RGB image by adjusting the color of each red, green, or blue pixel of a raw bayer pattern image, using the information of different colored pixels that surround it. For example, a red pixel in a raw bayer pattern image is adjusted by augmenting it with color data from (<NUM>) one or more adjacent blue pixels, and (<NUM>) one or more adjacent green pixels. Using this method, the red pixel of the raw bayer pattern image is converted to an RGB image pixel with red, green, and blue color components. These principles apply similarly to the conversion of green pixels and blue pixels of the raw bayer pattern image to RGB image pixels.

One objective of the de-mosaicking processing module's image processing algorithms is to produce, through pixel-by-pixel adjustment, an RGB image in which each pixel of the RGB image is an accurate approximation of the ratio of red, green, and blue light that the corresponding pixel sensor of the APS array (i.e., of the endoscope) is exposed to. As discussed previously, each pixel sensor of the APS array includes a light filter, which lets through only red light if the pixel sensor is a red pixel sensor, only green light if it is a green pixel sensor, or only blue light if it is a blue pixel sensor. Non-red light that reaches a red pixel sensor is filtered out by the light filter of the red pixel sensor and never reaches the photodetector of the red pixel sensor. The same is true for non-green light that reaches a green pixel sensor, and for non-blue light that reaches a blue pixel sensor. Accordingly, the information detected by the photodetector of any single pixel sensor (whether red, green, or blue) corresponds exclusively to the intensity of a single color of light (i.e., the color of light allowed to pass through the particular light filter).

The APS array is made up of red pixel sensors, green pixel sensors, and blue pixel sensors, which are arranged in a periodic pattern. Generally, pixel sensors of different colors are not arranged adjacent to one another. For example, in one embodiment, each red pixel sensor is flanked by one or more green pixel sensors and one or more blue pixel sensors. One reason for this type of pixel sensor arrangement is to enable the de-mosaicking processing module's image processing algorithms to approximate one color component of the light reaching a particular pixel sensor using information from an adjacent pixel sensor. In one example, a red pixel sensor is flanked by a green pixel sensor to its left, a green pixel sensor to its right, a blue pixel sensor above, and a blue pixel sensor below. The red pixel sensor includes a light filter that allows through only red light through to the photodetector. All non-red light (e.g., green light, blue light, etc.) that reaches the pixel sensor is filtered out, and never reaches the photodetector. Accordingly, the signal from the photodetector of the red pixel sensor is an indication of the intensity of red light to which the red pixel sensor is exposed. The photodetector of the red pixel sensor, however, provides no information about the intensity of non-red light (e.g., green light or blue light) that reaches the red pixel sensor, because all non-red light is filtered out by the light filter of the red pixel sensor and never reaches the photodetector. In this example, however, two green pixel sensors and two blue pixel sensors surround the red pixel sensor. The photodetector of each of the two green pixel sensors provides information about the intensity of green light to which each green pixel sensor is exposed. The photodetector of each of the two blue pixel sensors provides information about the intensity of blue light to which each blue pixel sensor is exposed. This information from adjacent green pixel sensors and adjacent blue pixel sensors can be used to approximate the intensity of green light and blue light to which the red pixel sensor is exposed. This approximation of green light and blue light to which the red pixel sensor is exposed, together with information about red light intensity provided by the photodetector of the red pixel sensor, is used to generate an RGB image pixel corresponding to the red pixel sensor. The colors of RGB image pixels corresponding to green pixel sensors and blue pixel sensors can be determined in a similar manner.

Subsequent to video processing <NUM>, the processed images are displayed at <NUM> to one or more users of a teleoperated surgical system (e.g., teleoperated surgical system <NUM> shown at <FIG>). Referring to <FIG>, these images can be displayed to a surgeon <NUM> seated at surgeon's console. Additionally, these images can be shown to other surgical staff on a display located on the electronics cart <NUM>. In one embodiment, the processed images make up a SXGA video output at <NUM> frames per second (fps). Image data of the processed images is communicated over DVI-D connections to the various displays. In one embodiment, at <NUM>, the processed images are additionally stored in a memory device. The processed images can be stored in their compressed and/or uncompressed format. In one embodiment, the processed images are stored on memory device <NUM> located on the electronics cart <NUM> (shown at <FIG>). In an alternate embodiment, the processed images are stored on a memory device located on the surgeon's console <NUM> or the patient-side cart <NUM>, or the storage device is a removable memory device (i.e., a "thumb drive") that is in data communication with the computer processor implementing the video processing at <NUM>. Optionally, as indicated by dashed line at <NUM>, un-processed raw bayer pattern image data generated at <NUM> can also be stored in this or another storage device.

In one embodiment, a sequence of images output by image processing at <NUM> is stored as a medical video at <NUM>. A timestamp of the time at which the video recording began is embedded with the data of the medical video. Video recording can be commanded automatically when an image capture device (e.g., endoscope) is installed for use on a teleoperated surgical system. In alternate embodiments, video recording begins when a user initiates the recording. This timestamp can be used by a computer processor of a teleoperated surgical system configured to perform content searches of recorded medical video. This aspect will be discussed in greater detail below.

Procedure event logs describing various system events occurring on a medical system (e.g., a portion of an exemplary event log is shown at <FIG>) can be used to improve the speed and accuracy with which content searching can be performed on video recordings of medical procedures performed using the medical system. Namely, system events of procedure event logs that are of clinical significance can be used to identify video segments (i.e., "video slips") of medical procedure video recordings that are likely to contain certain content of interest.

In one embodiment, the medical system includes a teleoperated surgical system. Procedure event logs of the teleoperated surgical system are generally continuously maintained by a computer processor <NUM> (shown at <FIG>) of the teleoperated surgical system, and stored on memory device <NUM> located on electronics cart <NUM> (shown at <FIG>). In alternate embodiments, one or both the computer and the memory device can be located on another component of the teleoperated surgical system. The procedure event logs preferably includes entries for system events that include the following:.

This list is by no means exhaustive. The event log can include entries corresponding to the occurrence of numerous other system events detectable by the teleoperated surgical system.

7B is a flow diagram showing an algorithm <NUM> for automatically generating and maintaining procedure event logs of a medical system. Referring to FIG. 7B, at <NUM>, algorithm <NUM> detects the occurrence of a system event. If the occurrence of a system event is detected at <NUM>, then at <NUM>, a corresponding entry is created in a procedure event log maintained on the medical system. Algorithm <NUM> updates the procedure event log by creating additional entries for each new system event that is detected, as indicated by <NUM>. In one embodiment of the procedure event log, the following fields are recorded for each entry associated with the occurrence of a clinically system event: (<NUM>) a timestamp of the event; (<NUM>) a short description of the event; and (<NUM>) a unique code associated exclusively with the particular type of event. In one example, a surgical instrument (e.g., a needle driver instrument) is installed for use on the teleoperated surgical system (e.g., at <NUM>:<NUM> AM local time), which the teleoperated surgical system detects and identifies as a system event at <NUM>. At <NUM>, an associated entry is made and entered into a procedure event log. The short description field of the event log entry will read, "Needle Driver Instrument Installed". A unique code, e.g., <NUM>, that is associated with each installation of a needle driver instrument will be recorded also. Finally, a <NUM>:<NUM> AM timestamp for the entry is generated. The local time of the event can be determined locally on the teleoperated surgical system if it was previously programmed with the local time. Alternatively, if the teleoperated surgical system is connected to the internet, the local time of the event can be determined by performing an internet query, in which the internet protocol (IP) address associated with the teleoperated surgical system can be used to provide locational information.

Portions of a procedure event log having the data structure described above are shown at <FIG>. As discussed previously, in one embodiment, the event log is automatically generated and automatically maintained with no input required, e.g., from a user of a medical system implementing algorithm <NUM>. This generation and maintenance of the event log takes place in parallel to other operations of the medical system. These other operations of the medical system can include the video recording of a medical procedure performed using the medical system, as well as the video processing and storage of these medical procedure video recordings.

<FIG> is a flow diagram showing an algorithm <NUM> for searching recorded medical procedure video. Algorithm <NUM> makes use of the entries of a procedure event log. Referring to <FIG>, at <NUM>, algorithm <NUM> receives a command issued by a user of the medical system to search recorded medical video for certain medical subject matter of interest. In one embodiment, the medical system includes a teleoperated surgical system. The user can issue this command by entering via a text entry interface certain text describing the medical event of interest, by speaking a certain phrase describing the medical event of interest through a voice-controlled user interface, or by other user interfaces known in the art. In one embodiment, this search is conducted intraoperatively by the operating surgeon (e.g., surgeon <NUM> at <FIG>), and the search is limited to the recording of the instant medical procedure. In alternate embodiments, various medical procedure video recordings and their associated event logs are saved in a memory device. The contents of these saved medical procedure video recordings can be searched intraoperatively or postoperatively by a medical person. In one example, the medical person desiring to search the contents of these saved medical procedure video recordings issues the search command by entering via a keyboard (e.g., a keyboard of computer <NUM> shown at <FIG>) certain text describing the medical event of interest.

At <NUM>, algorithm <NUM> uses the information input at <NUM> to search one or more procedure event logs containing system events relating to the instant medical procedure, for one or more system events that are indicative of the occurrence of the medical event of interest. In one embodiment, <NUM> limits the event log search to the system events of the instant medical procedure by searching only the system events that follow the most recent powering on of the teleoperated surgical system. In alternate embodiments, <NUM> searches the system events of a collection of archived procedure event logs. Each archived procedure event log is associated with a medical procedure performed using the device, and in one embodiment, a video recording of associated medical procedure (procedure video recording) is archived with its corresponding procedure event log. In one embodiment, associations between (<NUM>) various medical events of interest, and (<NUM>) system events in the event log, are preprogrammed on a device implementing algorithm <NUM> when the device is manufactured. The information on which these algorithmic associations are based can be derived from the knowledge of medically trained persons with experience using the teleoperated surgical system. As more associations between medical events of interest and system events become known to the manufacturer of the device, an updated algorithm <NUM> incorporating these associations can be provided to teleoperated surgical systems in the field using an internet connection. These additional associations may be made by the manufacturer of the device after study of medical procedure video recordings and of event log data retrieved from examples of the teleoperated surgical system used in the field.

In accordance with the above discussion, in one example, at <NUM>, a medical person using a teleoperated surgical system implementing algorithm <NUM> (e.g., surgeon <NUM> using teleoperated surgical system <NUM>, both shown at <FIG>) to perform a medical procedure wants to learn more about the circumstances surrounding an earlier part of the procedure, during which the patient experienced high blood loss. The medical person initiates a content search of a video recording of the instant medical procedure by speaking "high blood loss" into a voice-controlled search interface. The teleoperated surgical system is programmed to associate high blood loss with the use of an electrocautery instrument. Electrocautery instruments are commonly used to stop bleeding. At <NUM>, algorithm <NUM> conducts a search on the portion of the event log corresponding to the instant medical procedure, for event log entries whose descriptions include "cautery" and "install". Referring to <FIG>, in one example, the search at <NUM> produces a first system event <NUM>, a second system event <NUM>, and a third system event <NUM>. These system events indicate three separate instances in which a cautery instrument was installed on the teleoperated surgical system. At <NUM>, the timestamp for each of the search results generated by <NUM> is retrieved. In this example, the timestamps for the first system event <NUM>, the second system event <NUM>, and the third system event <NUM> are <NUM>:<NUM> AM, <NUM>:<NUM> AM, and <NUM>:<NUM> PM, respectively. At <NUM>, a timespan of interest is determined for each of the retrieved timestamps. In the case of high blood loss, the teleoperated surgical system is programmed to identify as the timespan of interest a <NUM>-minute video segment corresponding to each of the retrieved timestamps. Each <NUM>-minute video segment begins one minute before each retrieved timestamp and continues for three minutes. These video segments are identified as candidate video clips. In this example, the three <NUM>-minute video segments of interest (i.e., candidate video clips) are the following segments of the medical video: (<NUM>) the segment from <NUM>:<NUM> AM to <NUM>:<NUM> AM; (<NUM>) the segment from <NUM>:<NUM> AM to <NUM>:<NUM> AM; and (<NUM>) the segment from <NUM>:<NUM> AM to <NUM>:<NUM> AM. Different timespans of interest may be associated with different medical events of interest and with different system events.

At <NUM>, an image search algorithm that analyzes image data is applied to the frames of the video segments of interest (i.e., the candidate video clips) identified at <NUM>. In one embodiment, a first frame of each candidate video clip is located by determining the duration of time that has passed since video recording was initiated. As an example, with reference to <FIG>, video recording was initiated at <NUM>, which has a timestamp of <NUM>:<NUM> AM. <NUM> determined as the first candidate video clip the portion of the video recording taking place between <NUM>:<NUM> AM and <NUM>:<NUM> AM. Accordingly, the first frame of the first candidate video clip occurs at <NUM> minutes into the video recording. Similarly, the first frame of the second video segment of interest (the second candidate video clip) and of the third video segment of interest (the third candidate video clip) occur at <NUM> hour <NUM> minutes into the video recording and <NUM> hours and <NUM> minutes into the video recording, respectively. Likewise, the last frame of the first video segment of interest (the first candidate video clip), of the second video segment of interest (the second video clip), and the third video segment of interest occur (the third video clip) are at <NUM> hour <NUM> minutes, <NUM> hours <NUM> minutes, and <NUM> hours <NUM> minutes, respectively.

Discussion of an exemplary algorithm for determining the subject matter of an image using image data analysis can be found at <CIT>, entitled "Method and System for Segmenting Image Data" Generally, these algorithms look for certain patterns in the pixels of an image that are indicative of certain subject matter of interest. In this example, <NUM> is programmed to search the frames of the three candidate video clips identified above for pixel patterns that are indicative of high blood loss. In one embodiment, the image search algorithm positively identifies high blood loss when a sequence of frames making up a candidate video clip contain an increasing number of adjoining pixels of deep red color, as compared with the immediately preceding frame. In this example, <NUM> positively identifies high blood loss as being shown in the second candidate video clip (the video segment from <NUM>:<NUM> AM to <NUM>:<NUM> AM, corresponding to the second system event <NUM>, shown at <FIG>), and determines that high blood loss is not shown in the first candidate video clip (the <NUM>:<NUM> AM to <NUM>:<NUM> AM video segment, corresponding to the first system event <NUM>, shown at <FIG>) or in the third candidate video clip (the <NUM>:<NUM> AM to <NUM>:<NUM> PM video segment, corresponding to the third system event <NUM>, shown at <FIG>).

At <NUM>, the candidate video clip identified at <NUM> (i.e., the video segment from <NUM>:<NUM> AM to <NUM>:<NUM> AM) is presented to the medical person for viewing. In one embodiment, this video segment is displayed automatically. In an alternate embodiment, a link is provided on a display for viewing by the medical person (e.g., on a display located on surgeon's console <NUM> shown at <FIG>). The link is configured to play back to the medical person the portion of the recorded video corresponding to the identified video clip. Similarly, if more than one candidate video clip is identified at <NUM>, more than one link can be displayed. Please note that the identified video clips and/or the one or more links configured to play back identified video clips can be presented to the medical person on any of a number of displays other than the display located on the surgeon's console <NUM>. Additionally or in the alternative, this information can be displayed on a display associated with computer <NUM> (shown at <FIG>).

In another example, <NUM> of the algorithm <NUM> includes analysis of the actuation frequency of a cautery function of an electrocautery instrument. Actuation of the cautery function can occur by way of a foot-activated pedal located on a surgeon's console of a teleoperated surgical system (e.g., surgeon's console <NUM> shown at <FIG>). As discussed previously, cautery instruments are commonly used to stop bleeding. Generally, the more severe the bleeding, the more frequently the cautery functionality is actuated in the effort to stop the bleeding. In one embodiment, the teleoperated surgical system is programmed to associate high blood loss with the frequency with which the cautery function of an electrocautery instrument is actuated. In one example, a medical person viewing a medical procedure video recording seeks to learn more about the circumstances surrounding a part of the procedure during which the patient experienced high blood loss. The medical person initiates a video search in a manner similar to that discussed above. In this example, each actuation of the cautery function is a system event for which an entry is created in a procedure event log. At <NUM>, algorithm <NUM> conducts a search of the procedure event log corresponding to the instant medical procedure, searching for procedure event log entries whose descriptions include "cautery" and "actuation". In one example, the search at <NUM> yields <NUM> entries. At <NUM>, the timestamp for each of the search results generated by <NUM> is retrieved. In this example: (<NUM>) the timestamp for the first entry is <NUM>:<NUM> AM; (<NUM>) the timestamps of entries <NUM>-<NUM> are closely spaced and span the period between <NUM>:<NUM> AM and <NUM>:<NUM> AM; and (<NUM>) the timestamp for entry <NUM> is <NUM>:<NUM> PM. At <NUM>, a timespan of interest is determined using the timestamps retrieved at <NUM>. In this example, the teleoperated surgical system is programmed to identify as the timespan of interest a <NUM>-minute video segment (i.e., candidate video clip) corresponding to instances in which the cautery function is actuated more than <NUM> times in a <NUM> minute period. Each candidate video clip begins one minute before the timestamp of the event log entry corresponding to the first cautery actuation in the sequence of cautery actuations, and continues for two minutes after that timestamp. In this example, the frequency of cautery actuation between <NUM>:<NUM> AM and <NUM>:<NUM> AM is the only instance satisfying the more-than-<NUM>-times-in-<NUM>-minutes test. Accordingly, only one candidate video clip is identified: the segment from <NUM>:<NUM> AM to <NUM>:<NUM> AM. <NUM> and <NUM> operate in a manner similar to that described previously.

In another example, a medical person using a teleoperated surgical system implementing algorithm <NUM> (e.g., surgeon <NUM> using teleoperated surgical system <NUM>, both shown at <FIG>) to perform a cholecystectomy wants to learn more about the circumstances surrounding the mobilization of the gallbladder from the duodenum wall during the procedure. The medical person initiates a content search of a video recording of the cholecystectomy procedure by speaking "gallbladder mobilization" into a voice-controlled search interface. The teleoperated surgical system is programmed to associate gallbladder mobilization (a medical event of interest) with the use of a fluorescence imaging mode (a system event logged in procedure event logs). Fluorescence imaging is frequently used to improve visualization of certain anatomy during the gallbladder mobilization portion of a cholecystectomy. Accordingly, at <NUM>, algorithm <NUM> conducts a search on the portion of the event log corresponding to the instant cholecystectomy procedure, searching for event log entries whose descriptions include "fluorescence" and "on". In one example, the search at <NUM> yields a first entry, a second entry, and a third entry. At <NUM>, the timestamp for each of the search results generated by <NUM> is retrieved. In this example, the timestamps for the first entry, the second entry, and the third entry are <NUM>:<NUM> AM, <NUM>:<NUM> AM, and <NUM>:<NUM> PM, respectively. At <NUM>, a timespan of interest is determined for each of the retrieved timestamps. In the case of gallbladder mobilization, the teleoperated surgical system is programmed to identify as the timespan of interest a <NUM>-minute video segment corresponding to each of the retrieved timestamps. Each <NUM>-minute video segment begins at the time of the retrieved timestamp and continues for ten minutes after the timestamp. In this example, the three <NUM>-minute video segments of interest (candidate video clips) are the following segments of the medical video: (<NUM>) the segment from <NUM>:<NUM> AM to <NUM>:<NUM> AM; (<NUM>) the segment from <NUM>:<NUM> AM to <NUM>:<NUM> AM; and (<NUM>) the segment from <NUM>:<NUM> PM to <NUM>:<NUM> PM. <NUM> and <NUM> operate in a manner similar to that described previously.

In another example, a medical person using a teleoperated surgical system implementing algorithm <NUM> (e.g., surgeon <NUM> using teleoperated surgical system <NUM>, both shown at <FIG>) to perform a mitral valve repair procedure wants to learn more about the circumstances surrounding the reconstructive work performed during a mitral valve repair procedure. The medical person initiates a content search of a video recording of the mitral valve repair procedure by speaking "valve reconstruction" into a voice-controlled search interface. The teleoperated surgical system is programmed to associate valve reconstruction (a medical event of interest) with the use of a fine control mode (a system event logged in procedure event logs). In contrast to the coarse control mode, the fine control mode affords the user of the teleoperated surgical system the ability to command smaller movements of the surgical instruments. As discussed previously, in one embodiment of a teleoperated surgical system, surgical instruments coupled to slave robotic manipulators move in response to a surgeon's movement of two control inputs located on a surgeon's console. In a coarse control mode, a surgical instrument may move <NUM> in response to a surgeon's movement of a control input by <NUM>. In a fine control mode, the same movement <NUM> movement of the control input may produce only a <NUM> movement of the surgical instrument. A fine control mode is frequently used by a surgeon who is operating in areas of delicate anatomy. Compared to the coarse control mode, the finer scaling between (<NUM>) the movement of the control input, and (<NUM>) the movement of a corresponding surgical instrument, enables the surgeon to make smaller dissections and smaller sutures.

Accordingly, at <NUM>, algorithm <NUM> conducts a search on the portion of the event log corresponding to the instant valve repair procedure, searching for event log entries whose descriptions include "fine control mode". In one example, the search at <NUM> yields a first entry, a second entry, and a third entry. At <NUM>, the timestamp for each of the search results generated by <NUM> is retrieved. In this example, the timestamps for the first entry, the second entry, and the third entry are <NUM>:<NUM> AM, <NUM>:<NUM> AM, and <NUM>:<NUM> PM, respectively. At <NUM>, a timespan of interest is determined for each of the retrieved timestamps. In the case of valve reconstruction, the teleoperated surgical system is programmed to identify as the timespan of interest a <NUM>-minute video segment corresponding to each of the retrieved timestamps. Each <NUM>-minute video segment begins at the time of the retrieved timestamp and continues for <NUM> minutes after the timestamp. In this example, the three <NUM>-minute video segments of interest (i.e., candidate video clips) are the following segments of the medical video: (<NUM>) the segment from <NUM>:<NUM> AM to <NUM>:<NUM> AM; (<NUM>) the segment from <NUM>:<NUM> AM to <NUM>:<NUM> AM; and (<NUM>) the segment from <NUM>:<NUM> PM to <NUM>:<NUM> PM. <NUM> and <NUM> operate in a manner similar to that described previously.

For each of the preceding examples, the candidate video clips are evaluated to determine, using the image data of each of the candidate video clips, whether the pixels of the frames making up each candidate video clip are actually indicative of the subject matter of interest desired by the user. Candidate video clips for which this determination is made in the affirmative are presented to a user for viewing.

<FIG> shows a flow diagram of algorithm <NUM>, which directs the data collection for and maintenance of a data library of a device. In one embodiment, the device is a teleoperated surgical system, and the data library includes: (<NUM>) one or more video recordings (procedure video recordings) of surgical procedures performed using the surgical system; and (<NUM>) a procedure event log for each of the one or more procedure video recordings. The data library can be stored on a memory device of the teleoperated surgical system. Additionally or in the alternative, the data library can be stored: (<NUM>) in a memory device located on a remote server in data communication with the teleoperated surgical system; (<NUM>) in the memory of a cloud computing system that is in data communication with the teleoperated surgical system; or (<NUM>) any other electronic storage medium option.

In one embodiment, every time the surgical system is powered on, a new procedure event log is created. In one instance, powering on the surgical system creates a new procedure event log, but does not start the recording of video captured by an image capture device associated with the surgical system; a user is required to make an input via a user interface of the surgical system to start recording video captured by the image capture. In another instance, if an image capture device (e.g., an endoscope) is installed on the surgical system when the surgical system is powered on, then the powering on of the surgical can additionally cause the surgical system to begin recording video captured by the image capture device. In another instance, recording of video captured by an image capture device is initiated automatically when the image capture device is installed for use onto a surgical system that is already powered on. These examples are merely exemplary, and not intended to be limiting.

As shown in <FIG>, algorithm <NUM> is initiated when it receives a START signal at <NUM>. The START signal is the product of a user's input received via user interface <NUM> (e.g., actuating a power on/off switch). If the START signal is not received, then algorithm <NUM> loops back and repeats the query at <NUM>.

If the START signal is received at <NUM>, algorithm <NUM> proceeds to <NUM>, at which algorithm <NUM> queries a system clock <NUM> to read the current time. At <NUM>, algorithm <NUM> receives an image from an image capture device <NUM> in data communication with a computer processor executing algorithm <NUM>, the image being one frame of a video recording captured by the image captured device <NUM>. At <NUM>, the image received at <NUM> is digitally timestamped with the current time read at <NUM>, and the timestamped image is saved to a memory device <NUM> as one frame of a video recording.

Algorithm <NUM> also updates and maintains a procedure event log. As discussed previously, in one embodiment, a procedure event log is created each time the device performing algorithm <NUM> (e.g., a teleoperated surgical system) is powered on. At <NUM>, algorithm <NUM> queries the surgical system to determine whether one of a number of system events of interest is detected at that moment. Preferably, a memory device of the surgical system includes a look-up table containing a pre-identified list of system events to be logged in procedure event logs. If <NUM> detects the occurrence of one of the system events contained in the look-up table, the identified system event is retrieved and, at <NUM>, digitally timestamped with the current time determined at <NUM>. This system event and its associated timestamp are saved to memory device <NUM> as an entry in the procedure event log. Algorithm <NUM> then proceeds to <NUM>, which will be discussed in greater detail below.

If <NUM> determines that none of the system events contained in the look-up table are taking place, algorithm <NUM> proceeds directly to <NUM>, at which algorithm <NUM> queries whether a STOP signal has been received. If the STOP signal is received, algorithm <NUM> will stop video recording and will discontinue updates to the procedure event log for the existing surgical procedure. Preferably, the STOP signal is the product of a user input via a user interface (e.g., user interface <NUM>). In one embodiment, the STOP signal is received when the system is powered off (e.g., when a user actuates a power on/off switch). These examples are merely exemplary, and not intended to be limiting.

If a STOP signal is received, algorithm <NUM> proceeds to <NUM>, at which the images making up the various frames of the procedure video recording are compiled to produce a viewable video recording, and the various system events making up the entries of the procedure event log are compiled to form a complete procedure event log. The compiled procedure video recording and the procedure event log are then saved to memory device <NUM>, for later use. If a STOP signal is not received, algorithm <NUM> loops back to run <NUM> again.

<FIG> shows an algorithm <NUM> that performs video content searches using a combination of image data and auxiliary data. In one application, algorithm <NUM> can be used to search various procedure video recordings (e.g., procedure video recordings generated by algorithm <NUM>, shown at <FIG>) for certain subject matter of interest. In this application, the data contained in the procedure event logs, each of which is associated with a particular procedure video recording, constitutes auxiliary data that can be used to facilitate a search of the procedure video recordings for certain subject matter of interest. Algorithm <NUM> can be carried out by a computer processor of the device carrying out algorithm <NUM> (e.g., a computer processor that is part of a teleoperated surgical system). Algorithm <NUM> can also be carried out by a computer processor in data communication with the device carrying out algorithm <NUM> (e.g., a personal computing device).

In one embodiment, algorithm <NUM> makes use of known associations between (<NUM>) specific system events logged in the procedure event log, and (<NUM>) specific medical subject matter of interest, to facilitate a user's search for portions of procedure video recordings containing the specific medical subject matter of interest. As discussed previously, this approach of algorithm <NUM> improves upon various shortcomings associated with earlier video search algorithms that rely exclusively on image data analysis.

Referring to <FIG>, in one embodiment, algorithm <NUM> is initiated by a user request commanded via user interface <NUM>. At user interface <NUM>, a user requests that certain procedure video recordings be searched for certain subject matter of interest to the user. In one example, the user is a surgeon preparing to perform a particular surgical procedure involving a specific surgical technique, and the surgeon would like to view portions of archived procedure video recordings showing that specific surgical technique. User interface <NUM> may be a microphone, using which the user can command a search of previously recorded video for certain subject matter of interest. Alternatively, the user interface might be a touchpad or some alternate user interface that provides text entry means. These examples are merely exemplary, and not intended to be limiting.

In one embodiment, at <NUM>, a query mapper associates one or more terms contained in the user search requested at <NUM> with one of the system events ordinarily recorded by the procedure event logs. Preferably, query mapper <NUM> operates by searching a look-up table containing known associations between (<NUM>) various key terms or phrases, and (<NUM>) individual system events or patterns of several system events. As additionally associations become known, this look-up table can be updated to reflect the additional associations.

In one example, a user requests at <NUM> to search procedure video recordings for surgical techniques to mitigate patient bleeding during a surgical procedure. The query mapper at <NUM> operates on this user query, and determines that system events that indicate frequent applications of an electrocautery functionality associated with cautery instruments during a given surgical procedure correspond to a common surgical technique used to mitigate patient bleeding. Continuing with this example, at <NUM>, event searcher searches procedure event logs archived in a data library <NUM> for repeated applications of electrocautery in a given period (e.g., at least <NUM> applications of electrocautery energy in a <NUM>-minute time span).

Referring to <FIG>, exemplary data library <NUM> shows procedure event logs 1040a,1041a,1041a. Each of procedure event logs 1040a,1041a, 1042a is associated with one of the procedure video recordings 1040b,1041b,1042b. Continuing with the above example, event searcher <NUM> searches through procedure event logs 1040a,1041a,1042a for instances repeated applications of electrocautery in a given period, and finds that only procedure event log 1040a and procedure event log 1041a contain periods that satisfy the specified frequency of electrocautery application.

As discussed previously with respect to algorithm <NUM>, in one embodiment, each system event of a procedure event log (e.g., procedure event log 1040a,1041a,1042a) is archived with a corresponding timestamp and each frame of a procedure video recording (e.g., procedure video recording 1040a,1041a,1042a) is archived with a corresponding timestamp. Video clip extractor <NUM> operates on the output of event searcher <NUM> by using one or more timestamps associated with system events of a procedure event log to retrieve a corresponding segment of the procedure video recording. Continuing with the above example, using the output of event searcher <NUM>, video clip extractor <NUM> retrieves from procedure video recordings 1040b,1041b a candidate video clip for each of the portions of procedure event logs 1040a,1041a that satisfy the specified frequency of electrocautery application.

Claim 1:
A method comprising:
receiving, by a processing unit including one or more processors, a user command that includes a user description of a medical event of interest, to locate one or more video clips showing the medical event of interest, that is a portion of a medical procedure performed on a medical system, from one or more medical procedure video recordings of the medical procedure, each video recording including a corresponding recorded timestamp, recorded with at least one video frame of the video recording;
identifying, by the processing unit, from one or more procedure event logs, wherein each procedure event log includes one or more records, wherein the records correspond to occurrences of system events during performance of the medical procedure on the medical system, and wherein each record includes a timestamp indicating an occurrence time of a system event and a description of the system event, one or more records that include descriptions that correspond to the user description of the medical event of interest;
identifying, by the processing unit, one or more candidate video clips from the one or more medical procedure video recordings based upon one or more of the corresponding recorded timestamps and the timestamps included in the one or more identified records;
determining, by the processing unit, by searching image data of the one or more candidate video clips for patterns in pixels, in a sequence of frames making up a candidate video clip that contain an increasing number of adjoining pixels indicating the patterns, as compared with a preceding frame, that are indicative of the medical event of interest, whether each of the one or more candidate video segments is an identified video segment that contains the medical event of interest; and
presenting, by a user interface coupled to the processing unit, to a user at least one identified video segment.