Patent ID: 12223629

Further details and aspects of exemplary embodiments of the disclosure are described in more detail below with reference to the appended figures. Any of the above aspects and embodiments of the disclosure may be combined without departing from the scope of the disclosure.

DETAILED DESCRIPTION OF EMBODIMENTS

Embodiments of the presently disclosed devices, systems, and methods of treatment are described in detail with reference to the drawings, in which like reference numerals designate identical or corresponding elements in each of the several views. As used herein, the term “distal” refers to that portion of a structure that is farther from a user, while the term “proximal” refers to that portion of a structure that is closer to the user. The term “clinician” refers to a doctor, nurse, or other care provider and may include support personnel.

The present disclosure is applicable where images of a surgical site are captured. Endoscope systems are provided as an example, but it will be understood that such description is exemplary and does not limit the scope and applicability of the present disclosure to other systems and procedures.

Referring initially toFIGS.1-3, an endoscope system1, in accordance with the present disclosure, includes an endoscope10, a light source20, a video system30, and a display device40. With continued reference toFIG.1, the light source20, such as an LED/Xenon light source, is connected to the endoscope10via a fiber guide22that is operatively coupled to the light source20and to an endocoupler16disposed on, or adjacent to, a handle18of the endoscope10. The fiber guide22includes, for example, fiber optic cable which extends through the elongated body12of the endoscope10and terminates at a distal end14of the endoscope10. Accordingly, light is transmitted from the light source20, through the fiber guide22, and emitted out the distal end14of the endoscope10toward a targeted internal feature, such as tissue or an organ, of a body of a patient. As the light transmission pathway in such a configuration is relatively long, for example, the fiber guide22may be about 1.0 m to about 1.5 m in length, only about 15% (or less) of the light flux emitted from the light source20is outputted from the distal end14of the endoscope10.

With reference toFIG.2andFIG.3, the video system30is operatively connected to an image sensor32mounted to, or disposed within, the handle18of the endoscope10via a data cable34. An objective lens36is disposed at the distal end14of the elongated body12of the endoscope10and a series of spaced-apart, relay lenses38, such as rod lenses, are positioned along the length of the elongated body12between the objective lens36and the image sensor32. Images captured by the objective lens36are forwarded through the elongated body12of the endoscope10via the relay lenses38to the image sensor32, which are then communicated to the video system30for processing and output to the display device40via cable39. The image sensor32is located within, or mounted to, the handle18of the endoscope10, which can be up to about 30 cm away from the distal end14of the endoscope10.

With reference toFIGS.4-7, the flow diagrams include various blocks described in an ordered sequence. However, those skilled in the art will appreciate that one or more blocks of the flow diagram may be performed in a different order, repeated, and/or omitted without departing from the scope of the present disclosure. The below description of the flow diagram refers to various actions or tasks performed by one or more video system30, but those skilled in the art will appreciate that the video system30is exemplary. In various embodiments, the disclosed operations can be performed by another component, device, or system. In various embodiments, the video system30or other component/device performs the actions or tasks via one or more software applications executing on a processor. In various embodiments, at least some of the operations can be implemented by firmware, programmable logic devices, and/or hardware circuitry. Other implementations are contemplated to be within the scope of the present disclosure.

Referring toFIG.4, there is shown a schematic configuration of a system, which may be the endoscope system ofFIG.1or may be a different type of system (e.g., visualization system, etc.). The system, in accordance with the present disclosure, includes an imaging device410, a light source420, a video system430, and a display device440. The light source420is configured to provide light to a surgical site through the imaging device410via the fiber guide422. The distal end414of the imaging device410includes an objective lens436for capturing the image at the surgical site. The objective lens436forwards the image to the image sensor432. The image is then communicated to the video system430for processing. The video system430includes an imaging device controller450for controlling the endoscope and processing the images. The imaging device controller450includes processor452connected to a computer-readable storage medium or a memory454which may be a volatile type memory, such as RAM, or a non-volatile type memory, such as flash media, disk media, or other types of memory. In various embodiments, the processor452may be another type of processor such as, without limitation, a digital signal processor, a microprocessor, an ASIC, a graphics processing unit (GPU), field-programmable gate array (FPGA), or a central processing unit (CPU).

In various embodiments, the memory454can be random access memory, read only memory, magnetic disk memory, solid state memory, optical disc memory, and/or another type of memory. In various embodiments, the memory454can be separate from the imaging device controller450and can communicate with the processor452through communication buses of a circuit board and/or through communication cables such as serial ATA cables or other types of cables. The memory454includes computer-readable instructions that are executable by the processor452to operate the imaging device controller450. In various embodiments, the imaging device controller450may include a network interface540to communicate with other computers or a server.

Referring now toFIG.5, there is shown an operation for smoke reduction in images. In various embodiments, the operation ofFIG.5can be performed by an endoscope system1described above herein. In various embodiments, the operation ofFIG.5can be performed by another type of system and/or during another type of procedure. The following description will refer to an endoscope system, but it will be understood that such description is exemplary and does not limit the scope and applicability of the present disclosure to other systems and procedures. The following description will refer to an RGB (Red, Green, Blue) image or RGB color model, but it will be understood that such description is exemplary and does not limit the scope and applicability of the present disclosure to other types of images or color models (for example, CMYK (Cyan, Magenta, Yellow, Key), CIELAB, or CIEXYZ). The image sensor32may capture raw data. The format of the raw data may be RGGB, RGBG, GRGB, or BGGR. The video system30may convert the raw data to RGB using a demosaicing algorithm. A demosaicing algorithm is a digital image process used to reconstruct a full-color image from the incomplete color samples output from an image sensor overlaid with a color filter array (CFA). It is also known as CFA interpolation or color reconstruction. The RGB image may be further converted by the video system30to another color model, such as CMYK, CIELAB, or CIEXYZ.

Initially, at step502, an image of a surgical site is captured via the objective lens36and forwarded to the image sensor32of endoscope system1. The term “image” as used herein may include still images or moving images (for example, video). In various embodiments, the captured image is communicated to the video system30for processing. For example, during an endoscopic procedure, a surgeon may cut tissue with an electrosurgical instrument. During this cutting, smoke may be generated. When the image is captured, it may include the smoke. Smoke is generally a turbid medium (such as particles, water droplets) in the atmosphere. The irradiance received by the objective lens36from the scene point is attenuated by the line of sight. This incoming light is mixed with ambient light (air-light) reflected into the line of sight by atmospheric particles such as smoke. This smoke degrades the image, making it lose contrast and color fidelity.

FIG.6shows an exemplary pixel representation of an image captured in step502. In various embodiments, the captured image may or may not have been processed during the capture process or after the capture process. In various embodiments, an image600includes a number of pixels, and the dimensions of the image600are often represented as the amount of pixels in an X by Y format, such as 500×500 pixels, for example. In accordance with aspects of the present disclosure, and as explained in more detail later herein, each pixel of the image600may be processed based on a pixel area602,610centered at that pixel, which will also be referred to herein as a patch. In various embodiments, each patch/pixel area of the image can have the same size. In various embodiments, different pixel areas or patches can have different sizes. Each pixel area or patch can be denoted as Ω(x), which is a pixel area/patch having a particular pixel x as its center pixel. In the illustrative example ofFIG.6, the pixel area602has a size of 3×3 pixels and is centered at a particular pixel x1606. If an image has 18 by 18 pixels, a patch size may be 3×3 pixels. The illustrated image size and patch size are exemplary and other image sizes and patch sizes are contemplated to be within the scope of the present disclosure.

With continuing reference toFIG.6, each pixel601in an image600may have combinations of color components612, such as red, green, and blue, which are also referred to herein as color channels. Ic(y) is used herein to denote the intensity value of a color component c of a particular pixel y in the image600. For a pixel601, each of the color components612has an intensity value representing the brightness intensity of that color component. For example, for a 24 bit RGB image, each of the color components612has 8 bits, which corresponds to each color component having 256 possible intensity values.

Referring again toFIG.5, at step504, the video system30determines a dark channel matrix for the image600. As used herein, the phrase “dark channel” of a pixel refers to the lowest color component intensity value among all pixels of a patch Ω(x)602centered at a particular pixel x. The term “dark channel matrix” of an image, as used herein, refers to a matrix of the dark channel of every pixel of the image. The dark channel of a pixel x will be denoted as I_DARK(x). In various embodiments, the video system30calculates the dark channel of a pixel as follows:
I_DARK(x)=min(min(Ic(y))), for allc∈{r,g,b}y∈Ω(x)
where y denotes a pixel of the patch Ω(x), c denotes a color component, and Ic(y) denotes the intensity value of the color component c of pixel y. Thus, the dark channel of a pixel x is the outcome of two minimum operations across two variables c and y, which together determine the lowest color component intensity value among all pixels of a patch centered at pixel x. In various embodiments, the video system30can calculate the dark channel of a pixel by acquiring the lowest color component intensity value for every pixel in the patch and then finding the minimum value among all those values. For cases where the center pixel of the patch is at or near the edge of the image, only the part of the patch in the image is used.

For example, with reference toFIG.6, for an image600that was captured in step502, the image600may have a height and width of 18×18 pixels, the pixel area (patch) size may be 3×3 pixels. For example, a 3×3 pixel area Ω(x1)602centered at x1606may have the following intensities for the R, G, and B channels for each of the 9 pixels in the patch:

[1,3,62,0,15,3,42,4,36,7,47,6,91,3,25,8,99,1⁢1,2⁢5]
In this example, for the top-left pixel in the pixel area Ω(x1)602, the R channel may have an intensity of 1, the G channel may have an intensity of 3, and the B channel may have an intensity of 6. Here, the R channel has the minimum intensity value (a value of 1) of the RGB channels for that pixel.

The minimum color component intensity value of each the pixels would be determined. For example, for the 3×3-pixel area Ω(x1)602centered at x1, the minimum color component intensity value for each of the pixels in the pixel area Ω(x1)602are:

[103246159]
Thus, the dark channel of the pixel would have an intensity value of 0 for this exemplary 3×3-pixel area Ω(x)602centered at x1.

Referring again toFIG.5, at step506, the video system30estimates an atmospheric light component for each pixel, and the atmospheric light components for all of the pixels are together referred to herein as an “atmospheric light matrix.” The estimated atmospheric light component for a pixel x will be denoted herein as A(x). In various embodiments, A(x) can be determined based on the lowest color component intensity value for each pixel y604in a pixel area Ω(x)602, which can be denoted as:
A(x)=f(min(Ic(y))), for allc∈{r,g,b}y∈Ω(x),
where f( ) is an operation for estimating the atmospheric light component, based on the lowest color component intensity value for each pixel y604in the patch Ω(x1)602. In various embodiments, the operation f( ) may determine the maximum value among min(Ic(y)), for y∈Ω(x). In various embodiments, the maximum value can be scaled by a coefficient “coef,” which in various embodiments can have a value between 0 and 1, such as 0.85. The embodiment of atmospheric light component described above may be provided as follows:
A(x)=f(min(Ic(y)))=max(min(Ic(y)))*coef, for allc∈{r,g,b}y∈Ω(x)
For example, using the same example above for intensity values in patch Ω(x1)602, the video system30determines the atmospheric light component A(x1) to be 9*coef.

At step508, the video system30determines what is referred to herein as a transmission map T. The transmission map T is determined based on the dark channel matrix and the atmospheric light matrix, which were determined in steps504and506. The transmission map includes a transmission component T(x) for each pixel x. In various embodiments, the transmission component can be determined as follows:

T⁡(x)=1-ω*I_DARK⁢(X)A⁡(X),
where ω is a parameter having a value between 0 and 1, such as 0.85. In practice, even in clear images, there are some particles. Thus, some haze exists when distant objects are observed. The presence of haze is a cue to human perception of depth. If all haze is removed, the perception of depth may be lost. Therefore, to retain some haze, the parameter ω (0<ω<=1) is introduced. In various embodiments, the value of ω can vary based on the particular application. Thus, the transmission map is equal to 1 minus ω times the dark channel of a pixel (I-DARK(x)) divided by the atmospheric light component of the pixel, A(x).

At step510, the video system30de-hazes the image based on the transmission map.FIG.7illustrates one way to perform the de-hazing operation.

With reference toFIG.7, the illustrated operation assumes that the original image is an RGB image. The operation attempts to retain the color of the original RGB image600as much as possible in the de-haze process. In various embodiments, the de-hazing operation converts the image600from the RGB color space to the YUV color space (Y is luminance, U and V are chrominance or color), and applies dehazing on the Y (luma) channel, which is generally a weighted sum of the RGB color channels.

Initially, at step702the video system30converts the RGB image600to a YUV image denoted as I-YUV. The conversion of each pixel from RGB and YUV may be performed as follows:

[YUV]=[0.21260.71520.0722-0.09991-0.336090.4360.615-0.55861-0.05639][RGB]

Next, at step704the video system30performs a de-hazing operation on the channel Y (luma) of the I-YUV image. In accordance with aspects of the present disclosure, the de-hazing operation is as following:

Y′(x)=Y⁡(x)-A⁡(x)T⁡(x)
where Y′(x) is the Y(luma) channel of de-hazed image I-Y′UV. A(x) is the estimated atmospheric light component for pixel x, and T(x) is the transmission map value for pixel x. Thus, the Y(luma) channel of de-hazed image I-Y′UV is equal to the difference of the Y(luma) channel of image I-YUV and the estimated atmospheric light component A(x) calculated in step506, divided by the transmission map value T(x) which was determined in step508.

Finally, at step706the video system30converts the YUV image I-Y′UV to an de-hazed RGB image, the conversion from YUV to RGB is as follows:

[RGB]=[101.280331-0.21482-0.3805912.127980][YUV]

In various embodiments, the video system30may communicate the resultant de-hazed RGB image on the display device40or save it to a memory or external storage device for later recall or further processing. Although the operation ofFIG.7is described with respect to an RGB image, it will be understood that the disclosed operation can be applied to other color spaces as well.

FIGS.8-10show an example result of the methods described in the previous sections.FIG.8shows an image800with smoke captured during a surgical procedure using the endoscope system1. For example, during an endoscopic procedure, a surgeon may cut tissue804with an electrosurgical instrument802. During this cutting smoke806may be generated. This smoke806would be captured in the image800.

FIG.9shows a de-hazed image900, where the image800fromFIG.8was de-hazed was based on a constant atmospheric light value. The image1000, still somewhat obscured by smoke806, may include an electrosurgical instrument802and tissue804. For example, in a case where a constant atmospheric light value A was used instead of the atmospheric light matrix A being estimated by the formula used in step506.

FIG.10shows a de-hazed RGB image1000, de-hazed using the method ofFIGS.5and7, as described herein. The de-hazed RGB image1000may include an electrosurgical instrument802and tissue804. The method may start with the capture of the image800ofFIG.8during a surgical procedure, as in step502using the endoscopic system1. For example, the image may be approximately 20×20 pixels. Next, the video system30determines the dark channel matrix of the image as in step504. For example, the size of the pixel area Ω(x) may be set to approximately 3×3 pixels.

The determined dark channel matrix of the image ofFIG.8is used by the video system30to estimate the atmospheric light matrix by estimating a maximum value among the minimum color component intensities for each pixel in a pixel area, and multiplying this maximum value by a coefficient (e.g., 0.85) as in step506. Next, as in step508the video system30calculates a transmission map (T) according to the dark channel matrix and the estimated atmospheric light matrix.

The transmission map (T) is used in a de-hazing operation as described inFIG.7. At step702the video system30converts the RGB image I to a YUV image I-YUV. Next, at step704the video system30applies the de-hazing operation on channel Y (luma) of the I-YUV image by subtracting the estimated atmospheric light component A(x) from the Y (luma) channel and then dividing this difference by the determined transmission map, creating image I-Y′UV. Finally in step706, the I-Y′UV image gets converted to a de-hazed RGB image1000(seeFIG.10).

The embodiments disclosed herein are examples of the present disclosure and may be embodied in various forms. For instance, although certain embodiments herein are described as separate embodiments, each of the embodiments herein may be combined with one or more of the other embodiments herein. Specific structural and functional details disclosed herein are not to be interpreted as limiting, but as a basis for the claims and as a representative basis for teaching one skilled in the art to variously employ the present disclosure in virtually any appropriately detailed structure. Like reference numerals may refer to similar or identical elements throughout the description of the figures.

The phrases “in an embodiment,” “in embodiments,” “in some embodiments,” or “in other embodiments” may each refer to one or more of the same or different embodiments in accordance with the present disclosure. A phrase in the form “A or B” means “(A), (B), or (A and B).” A phrase in the form “at least one of A, B, or C” means “(A); (B); (C); (A and B); (A and C); (B and C); or (A, B, and C).” The term “clinician” may refer to a clinician or any medical professional, such as a doctor, nurse, technician, medical assistant, or the like, performing a medical procedure.

The systems described herein may also utilize one or more controllers to receive various information and transform the received information to generate an output. The controller may include any type of computing device, computational circuit, or any type of processor or processing circuit capable of executing a series of instructions that are stored in a memory. The controller may include multiple processors and/or multicore central processing units (CPUs) and may include any type of processor, such as a microprocessor, digital signal processor, microcontroller, programmable logic device (PLD), field programmable gate array (FPGA), or the like. The controller may also include a memory to store data and/or instructions that, when executed by the one or more processors, causes the one or more processors to perform one or more methods and/or algorithms.

Any of the herein described methods, programs, algorithms or codes may be converted to, or expressed in, a programming language or computer program. The terms “programming language” and “computer program,” as used herein, each include any language used to specify instructions to a computer, and include (but is not limited to) the following languages and their derivatives: Assembler, Basic, Batch files, BCPL, C, C+, C++, Delphi, Fortran, Java, JavaScript, machine code, operating system command languages, Pascal, Perl, PL1, scripting languages, Visual Basic, metalanguages which themselves specify programs, and all first, second, third, fourth, fifth, or further generation computer languages. Also included are database and other data schemas, and any other meta-languages. No distinction is made between languages which are interpreted, compiled, or use both compiled and interpreted approaches. No distinction is made between compiled and source versions of a program. Thus, reference to a program, where the programming language could exist in more than one state (such as source, compiled, object, or linked) is a reference to any and all such states. Reference to a program may encompass the actual instructions and/or the intent of those instructions.

Any of the herein described methods, programs, algorithms or codes may be contained on one or more machine-readable media or memory. The term “memory” may include a mechanism that provides (for example, stores and/or transmits) information in a form readable by a machine such a processor, computer, or a digital processing device. For example, a memory may include a read only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, or any other volatile or non-volatile memory storage device. Code or instructions contained thereon can be represented by carrier wave signals, infrared signals, digital signals, and by other like signals.

It should be understood that the foregoing description is only illustrative of the present disclosure. Various alternatives and modifications can be devised by those skilled in the art without departing from the present disclosure. Accordingly, the present disclosure is intended to embrace all such alternatives, modifications and variances. The embodiments described with reference to the attached drawing figures are presented only to demonstrate certain examples of the present disclosure. Other elements, steps, methods, and techniques that are insubstantially different from those described above and/or in the appended claims are also intended to be within the scope of the present disclosure.