Patent Description:
In some scenarios, an ultrasound probe may also be positioned within the patient to capture an ultrasound image within the patient during the surgical procedure. The ultrasound image may be presented concurrently with the endoscopic image to the surgeon (e.g., by way of the same display device that displays the endoscopic image). In this manner, the surgeon may visualize both the outer surfaces of the tissue included in the surgical area (using the endoscopic image) and structures internal to the tissue in the surgical area (using the ultrasound image) while performing the surgical procedure.

To capture a useful ultrasound image, the ultrasound probe must make good physical contact with tissue. Poor tissue contact by the ultrasound probe will result in an ultrasound image dominated by noise (e.g., noise generated by electronic components and/or signal artifacts that occur near the transducer surface of the ultrasound probe). If the ultrasound probe is not making good tissue contact, it may be distracting and/or useless to present the ultrasound image to the surgeon during the surgical procedure.

<CIT> discloses an ultrasound diagnosis apparatus comprising a memory and a processor, which is configured to determine, based on an ultrasound image captured by an ultrasound probe located within a patient (e.g. a transesophageal echocardiography probe) a contact state of the probe. The determination of the contact state is based on a ratio between a first region, for which ultrasound data is acquired, to a second region, for which ultrasound data is not acquired, or alternatively on a ratio of the first region to a whole region. <CIT> discloses determining contact states between an ultrasound probe imaging surface and a tissue surface, wherein the average luminance value of pixel values on a scanning line is compared to a predetermined threshold in order to estimate whether or not the point on the probe imaging surface from which the ultrasonic wave has emerged is in contact with the tissue surface.

In a first aspect, the present invention provides a system as defined in claim <NUM>. The system includes a memory storing instructions; and a processor communicatively coupled to the memory and configured to execute the instructions to classify each pixel in a plurality of pixels included in an ultrasound image captured by an ultrasound probe located within a patient as either showing tissue or showing non-tissue; and determine, based on the classification of each pixel in the plurality of pixels as either showing tissue or showing non-tissue, a contact state of the ultrasound probe, the contact state indicating whether the ultrasound probe is in operative physical contact with tissue of the patient.

In a second aspect, the present invention provides a method as defined in claim <NUM>. The method includes classifying, by a contact detection system, each pixel in a plurality of pixels included in an ultrasound image captured by an ultrasound probe located within a patient as either showing tissue or showing non-tissue; and determining, by the contact detection system based on the classifying of each pixel in the plurality of pixels as either showing tissue or showing non-tissue, a contact state of the ultrasound probe, the contact state indicating whether the ultrasound probe is in operative physical contact with tissue of the patient.

Particular embodiments are defined in the dependent claims.

Systems and methods for detecting tissue contact by an ultrasound probe are described herein. A contact detection system is configured to classify each pixel in a plurality of pixels included in an ultrasound image captured by an ultrasound probe located within a patient as either showing tissue or showing non-tissue, and determine, based on the classification of each pixel in the plurality of pixels as either showing tissue or showing non-tissue, a contact state of the ultrasound probe. The contact state indicates whether the ultrasound probe is in operative physical contact with tissue of the patient.

In some examples, the contact detection system may determine local descriptor values for a plurality of pixels included in an ultrasound image captured by an ultrasound probe located within a patient and use the local descriptor values to perform the classification. As described herein, the local descriptor values may characterize an intensity distribution and/or a spatial autocorrelation for each pixel in the plurality of pixels.

As used herein, operative physical contact refers to when the ultrasound probe is making sufficient enough tissue contact to capture a useful ultrasound image (i.e., an ultrasound image that includes at least a threshold amount of useful information instead of or in addition to noise, where the threshold amount may be determined in any of the ways described herein). Hence, an ultrasound probe may be in operative physical contact with tissue by being in full physical contact with the tissue or by being in partial physical contact with the tissue, as long as the partial physical contact is sufficient to render a useful ultrasound image. The ultrasound probe is not in operative physical contact with tissue when the ultrasound probe is not making sufficient enough tissue contact to capture a useful ultrasound image.

Based on the determined contact state of the ultrasound probe, the contact detection system may perform one or more operations. For example, based on the contact state of the ultrasound probe, the contact detection system may control a display of the ultrasound image within a viewable image displayed by a display device, set a parameter of an ultrasound machine connected to the ultrasound probe, and/or generate a control signal configured to be used by a computer-assisted surgical system to control a positioning of the ultrasound probe. These and other operations that may be performed by the contact detection system based on the determined contact state of the ultrasound probe are described herein.

The systems and methods described herein may provide various advantages and benefits. For example, the systems and methods described herein may intelligently prevent an ultrasound image from being included in a viewable image presented to a user (e.g., a surgeon) when the ultrasound image does not include useful information, thereby providing an improved visual experience for the user during a surgical procedure. Additionally or alternatively, the systems and methods described herein may automatically optimize one or more settings of an ultrasound machine used during a surgical procedure, thereby improving a quality of an ultrasound image generated by the ultrasound machine. Additionally or alternatively, the systems and methods described herein may facilitate optimal positioning of an ultrasound probe within a patient. Each of these operations may improve efficiency and effectiveness of a surgical procedure.

The systems and methods described herein advantageously determine whether an ultrasound probe is in operative physical contact with tissue based solely on the contents of an ultrasound image (also referred to as a B-mode image). In particular, the systems and methods described herein may be configured to distinguish between speckle (content in an ultrasound image that results from constructive and destructive interference of sound waves reflecting off of structures within the tissue) and non-useful noise included in the ultrasound image. Although speckle may visually appear similar to noise, the intensity distribution and spatial autocorrelation of speckle within an ultrasound image differs from noise. The systems and methods leverage this distinction to determine whether an ultrasound probe is in operative physical contact with tissue. This may advantageously result in substantially real-time determination of the contact state of the ultrasound probe.

These and other advantages and benefits of the systems and methods described herein will be made apparent herein.

<FIG> illustrates various components of an exemplary ultrasound imaging system <NUM>. As shown, ultrasound imaging system <NUM> may include an ultrasound machine <NUM>, an ultrasound probe <NUM>, and a display device <NUM>. Ultrasound machine <NUM> is communicatively coupled to ultrasound probe <NUM> by way of communication link <NUM> and to display device <NUM> by way of communication link <NUM>. Communication links <NUM> and <NUM> may be implemented by any suitable wired and/or wireless components. For example, communication link <NUM> may be implemented a cable, shaft, or other structure that carries one or more wires that communicatively interconnect ultrasound machine <NUM> and ultrasound probe <NUM>.

Ultrasound machine <NUM> may include computing components configured to facilitate generation of an ultrasound image. For example, ultrasound machine <NUM> may include a controller configured to control operation of ultrasound probe <NUM> by directing ultrasound probe <NUM> to emit and detect sound waves. In some examples, the controller and/or any other component of ultrasound machine <NUM> is configured to operate in accordance with one or more definable (e.g., adjustable) parameters. For example, ultrasound machine <NUM> may be configured to direct ultrasound probe <NUM> to emit sound waves having a definable frequency and/or receive sound waves at a particular gain. As another example, ultrasound machine <NUM> may also be configured to specify a fan depth of ultrasound image <NUM>.

Ultrasound machine <NUM> may additionally or alternatively include one or more image processing components configured to generate ultrasound image data <NUM> based on sound waves detected by ultrasound probe <NUM>. As shown, ultrasound machine <NUM> may transmit ultrasound image data <NUM> to display device <NUM> by way of communication link <NUM>. Display device <NUM> may use ultrasound image data <NUM> to generate and display an ultrasound image <NUM>.

In some examples, ultrasound machine <NUM> is connected to, integrated into, or implemented by a surgical system. For example, ultrasound machine <NUM> may be connected to, integrated into, or implemented by a computer-assisted surgical system that utilizes robotic and/or teleoperation technology to perform a surgical procedure (e.g., a minimally invasive surgical procedure). An exemplary computer-assisted surgical system is described herein.

Ultrasound probe <NUM> (also called a transducer) is configured to capture an ultrasound image by emitting sound waves and detecting the sound waves after they reflect from structures inside a body (e.g., structures internal to an organ or other tissue within a patient). Ultrasound probe <NUM> may have any suitable shape and/or size as may serve a particular implementation. In some examples, ultrasound probe <NUM> may have a shape and size that allow ultrasound probe <NUM> to be inserted into a patient by way of a port in a body wall of the patient. In these examples, a position of ultrasound probe <NUM> within the patient may be controlled manually (e.g., by manually manipulating a shaft to which ultrasound probe <NUM> is connected). Additionally or alternatively, the position of ultrasound probe <NUM> may be controlled in a computer-assisted manner (e.g., by a computer-assisted surgical system that utilizes robotic and/or teleoperation technology).

Display device <NUM> may be implemented by any suitable device configured to render or display ultrasound image <NUM> based on ultrasound image data <NUM>. As described herein, display device <NUM> may also be configured to display additional or alternative images and/or information. For example, in some scenarios, display device <NUM> may display a viewable image that includes ultrasound image <NUM> together with an endoscopic image acquired by an endoscope and/or a pre-operative model (e.g., a 3D model) of patient anatomy registered with the endoscopic image.

As mentioned, ultrasound probe <NUM> must be in operative physical contact with tissue of the patient to capture a useful ultrasound image. To illustrate, <FIG> illustrate different possible contact states of ultrasound probe <NUM> with respect to tissue <NUM>. Tissue <NUM> may represent any organ or anatomical feature of a patient.

<FIG> illustrates a first contact state in which ultrasound probe <NUM> is in operative physical contact with tissue <NUM>. As shown, an entire bottom surface <NUM> (which is convex-shaped in the examples provided herein) of ultrasound probe <NUM> is in physical contact with tissue <NUM>. In this contact state, there is sufficient acoustic coupling between ultrasound probe <NUM> and tissue <NUM> to capture a useful ultrasound image.

<FIG> illustrates a second contact state in which ultrasound probe <NUM> is not in operative physical contact with tissue <NUM>. As shown, bottom surface <NUM> of ultrasound probe <NUM> is separated from tissue <NUM> by a gap <NUM>. In this contact state, because there is no physical contact between ultrasound probe <NUM> and tissue <NUM>, an ultrasound image captured by ultrasound probe <NUM> will be dominated by noise, and is therefore not useful to a user.

<FIG> illustrates another instance of the second contact state in which ultrasound probe <NUM> is not in operative physical contact with tissue <NUM>. In <FIG>, only a small portion of bottom surface <NUM> of ultrasound probe <NUM> is in physical contact with tissue <NUM>. Because of this, ultrasound probe <NUM> may be determined to not be in operative physical contact with tissue <NUM> if the amount of useful information included in an ultrasound image generated by ultrasound probe <NUM> is below a particular threshold. Hence, as shown in <FIG>, ultrasound probe <NUM> may sometimes not be in operative physical contact with tissue <NUM> even though ultrasound probe <NUM> is at least partially touching tissue <NUM>.

<FIG> illustrates an exemplary contact detection system <NUM> ("system <NUM>") that is configured to detect tissue contact by an ultrasound probe (e.g., ultrasound probe <NUM>). System <NUM> may be included in, implemented by, or connected to any of surgical systems, ultrasound machines, or other computing systems described herein. For example, system <NUM> may be implemented by a computer-assisted surgical system and/or ultrasound machine <NUM>. As another example, contact detection system <NUM> may be implemented by a stand-alone computing system communicatively coupled to a computer-assisted surgical system and/or ultrasound machine <NUM>.

As shown, system <NUM> may include, without limitation, a storage facility <NUM> and a processing facility <NUM> selectively and communicatively coupled to one another. Facilities <NUM> and <NUM> may each include or be implemented by hardware and/or software components (e.g., processors, memories, communication interfaces, instructions stored in memory for execution by the processors, etc.). For example, facilities <NUM> and <NUM> may be implemented by any component in a computer-assisted surgical system. In some examples, facilities <NUM> and <NUM> may be distributed between multiple devices and/or multiple locations as may serve a particular implementation.

Storage facility <NUM> may maintain (e.g., store) executable data used by processing facility <NUM> to perform any of the operations described herein. For example, storage facility <NUM> may store instructions <NUM> that may be executed by processing facility <NUM> to perform any of the operations described herein. Instructions <NUM> may be implemented by any suitable application, software, code, and/or other executable data instance. Storage facility <NUM> may also maintain any data received, generated, managed, used, and/or transmitted by processing facility <NUM>.

Processing facility <NUM> may be configured to perform (e.g., execute instructions <NUM> stored in storage facility <NUM> to perform) various operations associated with detecting tissue contact by an ultrasound probe. For example, processing facility <NUM> may be configured to classify each pixel in a plurality of pixels included in an ultrasound image captured by an ultrasound probe located within a patient as either showing tissue or showing non-tissue. Processing facility <NUM> may be further configured to determine, based on the classification of each pixel in the plurality of pixels as either showing tissue or showing non-tissue, a contact state of the ultrasound probe. These and other operations that may be performed by processing facility <NUM> are described herein. In the description that follows, any references to operations performed by system <NUM> may be understood to be performed by processing facility <NUM> of system <NUM>.

<FIG> illustrates a detailed view of ultrasound image <NUM>. As shown, ultrasound image <NUM> includes a plurality of pixels (e.g., pixel <NUM>). Each pixel has an intensity value defined by ultrasound image data <NUM>. The term "pixel" is used herein to refer to any suitably sized and/or shaped region of ultrasound image <NUM> as may serve a particular implementation.

In some examples, system <NUM> may limit its processing of pixels for purposes of determining a contact state of ultrasound probe <NUM> to pixels within a particular region of interest. For example, <FIG> shows an exemplary region of interest <NUM> within ultrasound image <NUM>. In this case, system <NUM> may only determine local descriptor values for pixels within region of interest <NUM> and not for a set of pixels outside of region of interest <NUM>. This may limit the impact of artifacts near the ultrasound probe surface (region A in <FIG>) and signal dropout deeper in ultrasound image <NUM> (region B in <FIG>) since both of these regions can confound the results of the local descriptor value processing by system <NUM>. As shown, the left and right border regions of ultrasound image <NUM> (region C in <FIG>) may also be excluded from region of interest <NUM>. Alternatively, a border handling heuristic may be used to include border region pixels in the processing. Region of interest <NUM> may include any suitable number of pixels as may serve a particular implementation. In some examples, region of interest <NUM> includes all pixels included in ultrasound image <NUM>. In other examples, there may be multiple regions of interest included in ultrasound image <NUM>.

System <NUM> may determine one or more local descriptor values for each pixel included in region of interest <NUM> in any suitable manner. The local descriptor values characterize an intensity distribution for each pixel in region of interest <NUM> and/or a spatial autocorrelation for each pixel in region of interest <NUM>. Exemplary local descriptor values that characterize intensity distribution for a pixel include a moment (e.g., a local variance and/or a mean) of the intensity distribution for the pixel and/or any other metric representative of the intensity distribution for the pixel. Exemplary local descriptor values that characterize spatial autocorrelation include autocorrelation values (e.g., one or more terms of an autocovariance function) and/or any other metric representative of spatial autocorrelation for the pixel.

In some examples, the autocorrelation values determined by system <NUM> are spatial autocorrelation values (e.g., lag-<NUM> autocorrelation values in either the vertical (y) or horizontal (x) directions). Additionally or alternatively, the autocorrelation values may be temporal. In the examples provided herein, it will be assumed that the autocorrelation values are spatial.

In some examples, system <NUM> may determine multiple local descriptor values for each pixel in region of interest <NUM>. For example, system <NUM> may determine both a local variance value and an autocorrelation value for each pixel included in region of interest <NUM>. In some alternative examples, system <NUM> may determine only a single local descriptor value for each pixel. For example, system <NUM> may determine only a local variance value for each pixel included in region of interest <NUM>. Examples of determining local descriptor values for pixels are described herein.

System <NUM> may classify, based on the local descriptor values, pixels as either showing tissue or showing non-tissue in any suitable manner. System <NUM> may alternatively classify pixels as either showing tissue or showing non-tissue in any other suitable manner. For example, one or more image processing techniques, machine learning techniques, etc. may be used to classify pixels as either showing tissue or showing non-tissue. However, for illustrative purposes, the classification examples herein are based on local descriptor values.

For example, system <NUM> may compare the local descriptor values to one or more thresholds. To illustrate, system <NUM> may classify pixels that have local descriptor values above a local descriptor threshold as showing tissue and pixels that have local descriptor values below the local descriptor threshold as showing non-tissue.

As an example, the local descriptors determined by system <NUM> may include local variance values and autocorrelation values. In this example, system <NUM> may classify pixels that have local variance values above a variance threshold and autocorrelation values above an autocorrelation threshold as showing tissue. Likewise, system <NUM> may classify pixels that have local variance values below the variance threshold and/or autocorrelation values below the autocorrelation threshold as showing non-tissue. These thresholds may be determined in a number of different ways, some of which are described herein.

As another example, system <NUM> may determine only local variance values for each pixel. In this example, system <NUM> may classify pixels that have local variance values above the variance threshold as showing tissue and pixels that have local variance values below the variance threshold as showing non-tissue.

Additionally or alternatively, system <NUM> may classify pixels as either showing tissue or showing non-tissue by providing the local descriptor values as inputs into a machine learning model and classifying, based on an output of the machine learning model, each pixel in region of interest <NUM> as either showing tissue or showing non-tissue. The machine learning model may be supervised and/or unsupervised, and may be implemented by any suitable algorithm, such as logistic regression, classification and regression trees, random forests, and/or neural nets.

Additionally or alternatively, system <NUM> may classify pixels as either showing tissue or showing non-tissue by evaluating any other type of function as may serve a particular implementation. The function may output a binary classification of showing tissue or non-tissue or a fuzzy value indicating a probability of a pixel as showing either tissue or non-tissue. In the latter case, the probability may then be compared to a threshold to make a binary classification of showing tissue or non-tissue.

Once the pixels in region of interest <NUM> are classified as either showing tissue or showing non-tissue, system <NUM> may determine, based on the classification of each pixel as either showing tissue or showing non-tissue, a contact state of the ultrasound probe. The contact state indicates whether the ultrasound probe is in operative physical contact with tissue of the patient.

System <NUM> may use the classification of each pixel as either showing tissue or showing non-tissue to determine the contact state in any suitable manner. For example, system <NUM> may determine an average pixel classification representative of a number of pixels classified as showing tissue compared to a number of pixels classified as showing non-tissue. The average pixel classification may be a ratio of pixels classified as showing tissue to pixels classified as showing non-tissue and compare the ratio to a contact state threshold, which may be determined in a number of different ways as described herein. Additionally or alternatively, the average pixel classification could be a mean, median, or other suitable metric.

If the average pixel classification is above the contact state threshold, system <NUM> may determine that ultrasound probe <NUM> is in a first contact state that indicates that ultrasound probe <NUM> is in operative physical contact with the tissue of the patient. If the average pixel classification is below the contact state threshold, system <NUM> may determine that ultrasound probe <NUM> is in a second contact state that indicates that ultrasound probe <NUM> is not in operative physical contact with the tissue of the patient.

In accordance with the invention, two different contact state thresholds are used by system <NUM> for debouncing purposes. For example, system <NUM> may initially compare the average pixel classification to a first contact state threshold. Once the average pixel classification goes above the first contact state threshold, system <NUM> may determine that ultrasound probe <NUM> is in the first contact state that indicates that ultrasound probe <NUM> is in operative physical contact with the tissue of the patient. While ultrasound probe <NUM> is in the first contact state, the average pixel classification must go below a second contact state threshold that is lower than the first contact state threshold for system <NUM> to determine that ultrasound probe <NUM> is in the second contact state that indicates that the ultrasound probe <NUM> is not in operative physical contact with the tissue of the patient.

System <NUM> may determine the contact state of ultrasound probe <NUM> in any other suitable manner. For example, system <NUM> may provide the classifications to a machine learning model and use an output of the machine learning model to determine the contact state of ultrasound probe <NUM>. As another example, system <NUM> may evaluate any suitable function based on the classifications to determine the contact state of ultrasound probe <NUM>.

In some examples, before determining the local descriptor values, system <NUM> may optionally determine a background intensity for ultrasound image <NUM> and generate a demeaned ultrasound image by subtracting the background intensity from the ultrasound image <NUM>. System <NUM> may then determine the local descriptor values for the pixels in ultrasound image <NUM> by determining the local descriptor values for pixels included in the demeaned ultrasound image.

A particular processing heuristic that may be performed by system <NUM> in accordance with the principles described herein to determine a contact state of ultrasound probe <NUM> will now be described. It will be recognized that the processing heuristic is exemplary of a variety of different processing heuristics that may be performed by system <NUM> to determine a contact state of ultrasound probe <NUM>.

As mentioned, ultrasound images can appear to be noisy due to speckle. However, the intensity distribution and spatial autocorrelation of speckle differs from noise. Since the image content and gain settings can vary throughout the image, the autocovariance function may be estimated locally in accordance with the following equation: Wj,k(x, y) = E[(I(x, y) - µ(x,y))(I(x +j,y + k) - µ(x + j, y + k))].

In this equation, Wj,k(x, y) is the spatial autocovariance at location x and y, I(x, y) is the image intensity value, and µ(x, y) is the local mean intensity.

To distinguish between noise and speckle (which is representative of tissue), system <NUM> may determine the local mean intensity and one or more terms (also referred to as coefficients) of the autocovariance function.

For example, system <NUM> may first perform background subtraction on ultrasound image <NUM>. To illustrate, system <NUM> may use a box filter, H<NUM> (e.g., a seven by seven pixel filter) to estimate the background intensity and then subtract the background intensity from the original ultrasound image <NUM> to produce a demeaned image, Ĩ, where Ĩ(x, y) = I(x, y) - H<NUM> * I(x, y).

System <NUM> may then determine one or more terms of the autocovariance function. For example, system <NUM> may determine an estimate, Ŵj,k, within a local neighborhood around each pixel. For example, system <NUM> may use a second box filter, H<NUM>, (e.g., a <NUM> by <NUM> pixel filter) in accordance with the following equations: Ŵ<NUM>,<NUM>(x, y) = H<NUM> * Ĩ(x, y)<NUM> and Ŵ<NUM>,<NUM>(x, y) = H<NUM> * (Ĩ(x, y) · Ĩ(x, y + <NUM>)).

System <NUM> may then generate a binary tissue map, T(x, y), which shows which pixels are consistent with signal coming from ultrasound reflected or backscattered from tissue. For example, the binary tissue map may be generated in accordance with the following equation: T(x, y) = Ŵ<NUM>,<NUM>(x, y) > V & Ŵ<NUM>,<NUM>(x, y)/ Ŵ<NUM>,<NUM>(x, y) > AC1.

In this equation, V and AC1 are threshold parameters corresponding to the minimum variance and lag-<NUM> autocorrelation in the vertical direction. In the examples herein, autocorrelation is the autocovariance function normalized by the variance (i.e., Ŵj,k(x, y)/ Ŵ<NUM>,<NUM>(x, y)).

System <NUM> may optionally apply morphological processing to remove isolated pixels from T(x,y) and produce a smoother map. The morphological processing may be performed in any suitable manner.

System <NUM> may use the ratio of pixels within region of interest <NUM> where tissue is detected to determine the contact state of ultrasound probe <NUM> in accordance with the following equation: ΣROI T(x, y) /NROI.

In some examples, to prevent bouncing between contact states, two thresholds may be used by system <NUM>. For example, if ΣROI T(x, y) /NROI is goes above a first contact state threshold, system <NUM> may determine that ultrasound probe <NUM> is in a first contact state that indicates that ultrasound probe <NUM> is in operative physical contact with the tissue of the patient. Once in this state, ΣROI T(x, y) /NROI must go below a second contact state threshold lower than the first contact state threshold before system <NUM> determines that ultrasound probe <NUM> is in a second contact state that indicates that ultrasound probe <NUM> is not in operative physical contact with the tissue of the patient.

In some alternative embodiments, system <NUM> may obtain local estimates of the autocovariance function by using frequency domain approaches based on the short-time Fourier transform (STFT) or wavelet transforms. The STFT coefficients, or those of another wavelet transform, could be used directly to generate the tissue map described herein. In some examples, the coefficients of the auto-covariance function described herein may be replaced with those of an STFT or wavelet transform.

In some examples, any of the thresholds described herein (e.g., the local descriptor thresholds and the contact state thresholds described herein) may be set by system <NUM> in response to user input. In this manner, a user may manually tune the thresholds to appropriate levels. Additionally or alternatively, any of the thresholds described herein may be set based on an output of a machine learning model. The thresholds described herein may additionally or alternatively be determined in any other manner.

System <NUM> may perform various operations based on the contact state of ultrasound probe <NUM>. For example, based on the contact state of ultrasound probe <NUM>, system <NUM> may control a display of ultrasound image <NUM> within a viewable image displayed by display device <NUM>.

To illustrate, <FIG> show an exemplary viewable image <NUM> displayed by display device <NUM>. Viewable image <NUM> includes an endoscopic image of a surgical area within a patient as captured by an endoscope. As shown in both <FIG>, the endoscopic image depicts tissue <NUM> (e.g., an organ within the patient), a surgical tool <NUM> configured to manipulate tissue <NUM> in response to user input, and ultrasound probe <NUM>. While viewable image <NUM> is depicted as a two-dimensional image in <FIG>, it will be recognized that viewable image <NUM> may alternatively be a three-dimensional image in other examples.

In some examples, viewable image <NUM> may further include a pre-operative model of patient anatomy within the surgical area depicted in viewable image <NUM>. This is described more fully in <CIT>. The pre-operative model may be registered with the endoscopic image such that the model is located at a position within viewable image <NUM> that corresponds to an actual position of the patient anatomy. For example, the pre-operative model may include a three-dimensional model of structures interior to tissue <NUM> generated based on pre-operative imaging (e.g., MRI and/or CT scan imaging).

In <FIG>, ultrasound probe <NUM> is in operative physical contact with tissue <NUM>. Accordingly, system <NUM> may determine that the contact state of ultrasound probe <NUM> indicates that ultrasound probe <NUM> is in operative physical contact with tissue <NUM>. Based on this determination and as shown in <FIG>, system <NUM> may display ultrasound image <NUM> within viewable image <NUM>. As illustrated, ultrasound image <NUM> may be located within viewable image <NUM> at a position that appears to be directly beneath the bottom surface of ultrasound probe <NUM>. By positioning ultrasound image <NUM> in this manner, system <NUM> may allow a user to more readily ascertain relative positions of structures interior to tissue <NUM> and included within ultrasound image <NUM> with other content shown in viewable image <NUM>. Alternatively, ultrasound image <NUM> may be located at any other position within viewable image <NUM> as may serve a particular implementation.

<FIG> shows that ultrasound probe <NUM> has been repositioned to a location that is not in physical contact with tissue <NUM>. Such repositioning may occur in response to user manipulation of ultrasound probe <NUM> and/or in any other manner. In response to the repositioning, system <NUM> may determine that the contact state of ultrasound probe now indicates that ultrasound probe is not in operative physical contact with tissue <NUM>. In response and as shown in <FIG>, system <NUM> may abstain from displaying (e.g., by hiding or otherwise not displaying) ultrasound image <NUM> in viewable image <NUM>.

By intelligently controlling the display of ultrasound image <NUM> in this manner, system <NUM> may ensure that ultrasound image <NUM> is only displayed when it includes useful information for the user. Otherwise, ultrasound image <NUM> is hidden so as not to obscure other content in viewable image <NUM>.

In some examples, system <NUM> may display only a portion of ultrasound image <NUM> in response to determining that the contact state of ultrasound probe <NUM> indicates that ultrasound probe <NUM> is in operative physical contact with tissue <NUM>. For example, if a particular region (e.g., a pie-shaped slice) of ultrasound image <NUM> includes useful information, but the rest of ultrasound image <NUM> does not, this may be indicative of only a portion of ultrasound probe <NUM> being in operative physical contact with tissue <NUM>. In response, system <NUM> may generate and display a cropped ultrasound image that includes only a portion of ultrasound image <NUM>. The cropped ultrasound image may include the region that includes the useful information and may be determined based on the classification of the pixels as either showing tissue or showing non-tissue.

To illustrate, <FIG> shows viewable image <NUM> displayed by display device <NUM>. As shown, instead of displaying the full ultrasound image <NUM> within viewable image <NUM>, system <NUM> displays a cropped ultrasound image <NUM> within viewable image <NUM>. For the sake of comparison, a dashed outline <NUM> representative of what full ultrasound image <NUM> would look like were it displayed is shown in <FIG>. Dashed outline <NUM> may or may not be actually displayed in viewable image <NUM> as may serve a particular implementation.

<FIG> illustrates another example of displaying only a portion of ultrasound image <NUM> in viewable image <NUM>. In this example, system <NUM> displays cropped ultrasound image <NUM> instead of full ultrasound image <NUM>, which is again represented by dashed lines <NUM> that may or may not be actually displayed in viewable image <NUM>. As shown, cropped ultrasound image <NUM> does not include a distal region of full ultrasound image <NUM>. This type of cropped ultrasound image <NUM> may be beneficial to display in scenarios in which deeper regions of tissue result in signal dropout, thus causing the distal region of full ultrasound image <NUM> to include more noise than useful content.

System <NUM> may additionally or alternatively set (e.g., adjust), based on the contact state of ultrasound probe <NUM>, a parameter of ultrasound imaging machine <NUM>. For example, system <NUM> may set a frequency and/or a gain of the sound emitted or received by ultrasound probe <NUM> based on the contact state of ultrasound probe <NUM>. System <NUM> may additionally or alternatively set a fan depth for ultrasound image <NUM> based on the contact state of ultrasound probe <NUM>. By setting one or more parameters based on the contact state of ultrasound probe <NUM>, system <NUM> may be configured to automatically acquire a better quality ultrasound image <NUM>.

For example, the contact state of ultrasound probe <NUM> may indicate that ultrasound probe <NUM> is barely in operative physical contact with tissue <NUM> (e.g., if the ratio described above is barely above the contact state threshold). In this scenario, system <NUM> may increase the gain of the sound received by ultrasound probe <NUM> to improve the image quality of ultrasound image <NUM>.

System <NUM> may additionally or alternatively generate, based on the contact state of ultrasound probe <NUM>, a control signal configured to be used by a computer-assisted surgical system to control a positioning of ultrasound probe <NUM> (e.g., to achieve and/or maintain tissue contact). For example, a shaft of ultrasound probe <NUM> may be coupled to a manipulator arm of a computer-assisted surgical system. In this example, the computer-assisted surgical system may be configured to adjust a positioning of ultrasound probe <NUM> based on the control signal by repositioning the manipulator arm. As another example, a different surgical tool (e.g., graspers) controllable by computer-assisted surgical system may be configured to hold and reposition ultrasound probe <NUM>. In either example, the control signal may indicate that ultrasound probe <NUM> is not in operative physical contact with tissue <NUM>. In response, the computer-assisted surgical system may reposition ultrasound probe <NUM> until the control signal indicates that ultrasound probe <NUM> is in operative physical contact with tissue <NUM>.

<FIG> illustrates an exemplary computer-assisted surgical system <NUM> ("surgical system <NUM>"). As described herein, ultrasound machine <NUM> and system <NUM> may be implemented by surgical system <NUM>, connected to surgical system <NUM>, and/or otherwise used in conjunction with surgical system <NUM>.

As shown, surgical system <NUM> may include a manipulating system <NUM>, a user control system <NUM>, and an auxiliary system <NUM> communicatively coupled one to another. Surgical system <NUM> may be utilized by a surgical team to perform a computer-assisted surgical procedure on a patient <NUM>. As shown, the surgical team may include a surgeon <NUM>-<NUM>, an assistant <NUM>-<NUM>, a nurse <NUM>-<NUM>, and an anesthesiologist <NUM>-<NUM>, all of whom may be collectively referred to as "surgical team members <NUM>. " Additional or alternative surgical team members may be present during a surgical session as may serve a particular implementation.

While <FIG> illustrates an ongoing minimally invasive surgical procedure, it will be understood that surgical system <NUM> may similarly be used to perform open surgical procedures or other types of surgical procedures that may similarly benefit from the accuracy and convenience of surgical system <NUM>. Additionally, it will be understood that the surgical session throughout which surgical system <NUM> may be employed may not only include an operative phase of a surgical procedure, as is illustrated in <FIG>, but may also include preoperative, postoperative, and/or other suitable phases of the surgical procedure. A surgical procedure may include any procedure in which manual and/or instrumental techniques are used on a patient to investigate or treat a physical condition of the patient.

As shown in <FIG>, manipulating system <NUM> may include a plurality of manipulator arms <NUM> (e.g., manipulator arms <NUM>-<NUM> through <NUM>-<NUM>) to which a plurality of surgical instruments may be coupled. Each surgical instrument may be implemented by any suitable surgical tool (e.g., a tool having tissue-interaction functions), medical tool, imaging device (e.g., an endoscope), sensing instrument (e.g., a force-sensing surgical instrument), diagnostic instrument, or the like that may be used for a computer-assisted surgical procedure on patient <NUM> (e.g., by being at least partially inserted into patient <NUM> and manipulated to perform a computer-assisted surgical procedure on patient <NUM>). While manipulating system <NUM> is depicted and described herein as including four manipulator arms <NUM>, it will be recognized that manipulating system <NUM> may include only a single manipulator arm <NUM> or any other number of manipulator arms as may serve a particular implementation.

Manipulator arms <NUM> and/or surgical instruments attached to manipulator arms <NUM> may include one or more displacement transducers, orientational sensors, and/or positional sensors used to generate raw (i.e., uncorrected) kinematics information. One or more components of surgical system <NUM> may be configured to use the kinematics information to track (e.g., determine positions of) and/or control the surgical instruments.

User control system <NUM> may be configured to facilitate control by surgeon <NUM>-<NUM> of manipulator arms <NUM> and surgical instruments attached to manipulator arms <NUM>. For example, surgeon <NUM>-<NUM> may interact with user control system <NUM> to remotely move or manipulate manipulator arms <NUM> and the surgical instruments. To this end, user control system <NUM> may provide surgeon <NUM>-<NUM> with imagery (e.g., high-definition 3D imagery) of a surgical area associated with patient <NUM> as captured by an imaging system (e.g., any of the medical imaging systems described herein). In certain examples, user control system <NUM> may include a stereo viewer having two displays where stereoscopic images of a surgical area associated with patient <NUM> and generated by a stereoscopic imaging system may be viewed by surgeon <NUM>-<NUM>. Surgeon <NUM>-<NUM> may utilize the imagery to perform one or more procedures with one or more surgical instruments attached to manipulator arms <NUM>.

To facilitate control of surgical instruments, user control system <NUM> may include a set of master controls. These master controls may be manipulated by surgeon <NUM>-<NUM> to control movement of surgical instruments (e.g., by utilizing robotic and/or teleoperation technology). The master controls may be configured to detect a wide variety of hand, wrist, and finger movements by surgeon <NUM>-<NUM>. In this manner, surgeon <NUM>-<NUM> may intuitively perform a procedure using one or more surgical instruments.

Auxiliary system <NUM> may include one or more computing devices configured to perform primary processing operations of surgical system <NUM>. In such configurations, the one or more computing devices included in auxiliary system <NUM> may control and/or coordinate operations performed by various other components (e.g., manipulating system <NUM> and user control system <NUM>) of surgical system <NUM>. For example, a computing device included in user control system <NUM> may transmit instructions to manipulating system <NUM> by way of the one or more computing devices included in auxiliary system <NUM>. As another example, auxiliary system <NUM> may receive, from manipulating system <NUM>, and process image data representative of imagery captured by an imaging device attached to one of manipulator arms <NUM>.

In some examples, auxiliary system <NUM> may be configured to present visual content to surgical team members <NUM> who may not have access to the images provided to surgeon <NUM>-<NUM> at user control system <NUM>. To this end, auxiliary system <NUM> may include a display monitor <NUM> configured to display one or more user interfaces, such as images (e.g., 2D images) of the surgical area, information associated with patient <NUM> and/or the surgical procedure, and/or any other visual content as may serve a particular implementation. For example, display monitor <NUM> may display images of the surgical area together with additional content (e.g., graphical content, contextual information, etc.) concurrently displayed with the images. In some embodiments, display monitor <NUM> is implemented by a touchscreen display with which surgical team members <NUM> may interact (e.g., by way of touch gestures) to provide user input to surgical system <NUM>.

Manipulating system <NUM>, user control system <NUM>, and auxiliary system <NUM> may be communicatively coupled one to another in any suitable manner. For example, as shown in <FIG>, manipulating system <NUM>, user control system <NUM>, and auxiliary system <NUM> may be communicatively coupled by way of control lines <NUM>, which may represent any wired or wireless communication link as may serve a particular implementation. To this end, manipulating system <NUM>, user control system <NUM>, and auxiliary system <NUM> may each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc..

<FIG> shows an exemplary method <NUM>. One or more of the operations shown in in <FIG> may be performed by system <NUM>, any components included therein, and/or any implementation thereof.

In operation <NUM>, a contact detection system determines local descriptor values for a plurality of pixels included in an ultrasound image captured by an ultrasound probe located within a patient. Operation <NUM> may be performed in any of the ways described herein.

In operation <NUM>, the contact detection system classifies, based on the local descriptor values, each pixel in the plurality of pixels as either showing tissue or showing non-tissue. Operation <NUM> may be performed in any of the ways described herein.

In operation <NUM>, the contact detection system determines, based on the classification of each pixel in the plurality of pixels as either showing tissue or showing non-tissue, a contact state of the ultrasound probe, the contact state indicating whether the ultrasound probe is in operative physical contact with tissue of the patient. Operation <NUM> may be performed in any of the ways described herein.

<FIG> shows another exemplary method <NUM>. One or more of the operations shown in in <FIG> may be performed by system <NUM>, any components included therein, and/or any implementation thereof.

In operation <NUM>, the contact detection system determines, based on the local descriptor values, a contact state of the ultrasound probe, the contact state indicating whether the ultrasound probe is in operative physical contact with tissue of the patient. Operation <NUM> may be performed in any of the ways described herein.

In operation <NUM>, the contact detection system controls, based on the contact state of the ultrasound probe, a display of the ultrasound image within a viewable image displayed by a display device. Operation <NUM> may be performed in any of the ways described herein.

In operation <NUM>, the contact detection system sets, based on the contact state of the ultrasound probe, a parameter of an ultrasound imaging machine connected to the ultrasound probe. Operation <NUM> may be performed in any of the ways described herein.

In operation <NUM>, the contact detection system generates, based on the contact state of the ultrasound probe, a control signal configured to be used by a computer-assisted surgical system to control a positioning of the ultrasound probe. Operation <NUM> may be performed in any of the ways described herein.

In operation <NUM>, a contact detection system classifies each pixel in a plurality of pixels included in an ultrasound image captured by an ultrasound probe located within a patient as either showing tissue or showing non-tissue. Operation <NUM> may be performed in any of the ways described herein.

<FIG> illustrates an exemplary computing device <NUM> that may be specifically configured to perform one or more of the processes described herein. Any of the systems, units, computing devices, and/or other components described herein may be implemented by computing device <NUM>.

As shown in <FIG>, computing device <NUM> may include a communication interface <NUM>, a processor <NUM>, a storage device <NUM>, and an input/output ("I/O") module <NUM> communicatively connected one to another via a communication infrastructure <NUM>. While an exemplary computing device <NUM> is shown in <FIG>, the components illustrated in <FIG> are not intended to be limiting. Additional or alternative components may be used in other embodiments. Components of computing device <NUM> shown in <FIG> will now be described in additional detail.

Claim 1:
A system comprising:
a memory storing instructions; and
a processor communicatively coupled to the memory and configured to execute the instructions to:
classify each pixel in a plurality of pixels included in an ultrasound image captured by an ultrasound probe located within a patient as either showing tissue or showing non-tissue;
determine an average pixel classification representative of a number of pixels classified as showing tissue compared to a number of pixels classified as showing non-tissue;
determine, if the average pixel classification is above a first contact state threshold, that the ultrasound probe is in a first contact state that indicates that the ultrasound probe is in operative physical contact with the tissue of the patient; and
determine, if the average pixel classification is below a second contact state threshold lower than the first contact state threshold, that the ultrasound probe is in a second contact state that indicates that the ultrasound probe is not in operative physical contact with the tissue of the patient.