MEDICAL DEVICES AND RELATED SYSTEMS AND METHODS FOR AUTOMATIC IMAGE BRIGHTNESS CONTROL

A medical device system may include a control unit comprising one or more processors that implement an algorithm to enhance images obtained by a medical device. The one or more processing boards perform the steps of: receiving a first image from the first viewing element; determining a current illumination value of the first image; determining a first difference between the current illumination value and a target high illumination value if the current illumination value is greater than the target high illumination value; determining a second difference between the current illumination value and a target low illumination value if the current illumination value is less than the target low illumination value; generating a new illumination value, using at least one of the first difference and the second difference; and converting the new illumination value to a first voltage value for application to one or more illuminators of the medical device.

TECHNICAL FIELD

Various aspects of this disclosure relate generally to systems, devices, and methods for automatic image brightness control. More specifically, embodiments of this disclosure relate to imaging catheters, such as endoscope or other medical devices, configured to automatically control an illuminator and related systems and methods, among other aspects.

BACKGROUND

During endoscopic procedures, a medical professional operating an endoscope often relies on one or more illuminators to illuminate a field of view of a camera at the distal end of the endoscope. Most imaging catheters, such as endoscopes, rely on a fixed illumination output, with each of the imaging catheter's illuminators outputting a constant illumination, for example, from one or more light emitting diodes (LEDs). Such imaging catheters with a constant illumination output control the image brightness by varying the exposure and/or the gain of the image sensor. This results in a noticeable stepped response in the image brightness, as well as a slow response to changing scenes. Step response refers to the change of the output of a system when its input is a unit step function. The stepped response is due to the limited number of exposure steps available in the image sensor, and the slowed response is at least in part because the exposure values are written in single steps at the end of each image frame. For example, if the exposure needs to be adjusted by 10 steps to increase or decrease the exposure of the image sensor, then it would typically take a minimum of 10 image frames to adjust the brightness of the image, resulting in a noticeable lag (approximately 300 milliseconds for a 30 frame per second (fps) image sensor).

When a user experiences image lag in an imaging catheter system, the procedure may be prolonged, and procedural tasks may be more difficult and delayed. There is a need for alternative methods of illumination adjustment for imaging catheters and other medical devices to reduce imaging lag and address other problems with medical device illumination and imaging systems.

SUMMARY

Aspects of the disclosure relate to, among other things, systems, devices, and methods to help reduce imaging lag in medical device imaging systems, among other aspects. The systems, devices, and methods of this disclosure may decrease the time required to focus and/or properly illuminate a field of view of a camera or other imaging device of an endoscope or other medical device. Endoscopes and other medical devices incorporating the systems and methods of this disclosure may help address image lag, may help reduce the time required for procedures, and may help address other issues. Each of the aspects disclosed herein may include one or more of the features described in connection with any of the other disclosed aspects.

According to one aspect, a medical device system may include a control unit configured to be operatively coupled to a medical device. The control unit may comprise one or more processors that implement an algorithm to enhance images obtained by a first viewing element of the medical device. The one or more processing boards perform the steps of: receiving a first image from the first viewing element; determining a current illumination value of the first image; determining a first difference between the current illumination value and a target high illumination value if the current illumination value is greater than the target high illumination value; determining a second difference between the current illumination value and a target low illumination value if the current illumination value is less than the target low illumination value; generating a new illumination value, using at least one of the first difference and the second difference; and converting the new illumination value to a first voltage value for application to one or more illuminators of the medical device.

In other aspects, the medical device system may include one or more of the following features. The target high illumination value and the target low illumination value together may define a tolerance band around a target illumination value stored by the control unit. The one or more processing boards may further perform the steps of: determining if the current illumination value is below a first threshold illumination value; and if the current illumination value is below the first threshold illumination value, using a scaling factor to generate the new illumination value. The one or more processing boards may further perform the steps of: determining if the new illumination value is greater than a maximum illumination value; if the new illumination value is greater than a maximum illumination value, converting the maximum illumination value to a second voltage value for application to one or more illuminators of the medical device, and increasing a gain of the one or more imaging devices. The one or more processing boards may further perform the steps of: determining if the new illumination value is lower than the current illumination value; if the new illumination value is lower than the current illumination value, decreasing a gain of the one or more imaging devices. The medical device may be an endoscope. Determining the current illumination value of the first image may include accumulating and summing pixel values of the first image. The one or more processing boards may further perform the steps of: determining a current frame rate of the first viewing element; generating a new frame rate, using at least one of the first difference and the second difference; and applying the new frame rate to the first viewing element.

In other aspects, the medical device system may include one or more of the following features. The medical device may include a first viewing element and at least one illuminator. The one or more processors may further perform the steps of: determining a current exposure time of the first viewing element; generating a new exposure time, using at least one of the first difference and the second difference; and applying the new exposure time to the first viewing element. The one or more processors may further perform the steps of: prior to converting the new illumination value to the first voltage value, determining if the current illumination value is below or above the new illumination value; if the current illumination value is below the new illumination value, converting the new illumination value to the first voltage value; and if the current illumination value is above the new illumination value, increasing a frame rate of the first viewing element. Generating a new illumination value, using at least one of the first difference and the second difference, may include determining a first error coefficient of the first image and a second error coefficient of a second image, wherein the second image was received by the control unit prior to the first image; wherein the first error coefficient is the first difference if the current illumination value is greater than the target high illumination value; and wherein the first error coefficient is the second difference if the current illumination value is less than the target low illumination value. Generating the new illumination value may further include determining a proportional tuning constant, an integral tuning constant, and a derivative tuning constant each associated with the medical device.

In other aspects, the medical device system may include one or more of the following features. The current illumination value may be a first current illumination value, the target high illumination value may be a first target high illumination value, the target low illumination value may be a first target low illumination value, the new illumination value may be a first new illumination value, and the one or more processing boards may further perform the steps of: receiving a second image from a second viewing element; determining a second current illumination value of the second image; determining a third difference between the second current illumination value and a second target high illumination value if the second current illumination value is greater than the second target high illumination value; determining a fourth difference between the second current illumination value and a second target low illumination value if the second current illumination value is less than the second target low illumination value; generating a second new illumination value, using at least one of the third difference and the fourth difference; and converting the second new illumination value to a second voltage value for application to one or more illuminators of the medical device. The one or more processing boards may further perform the steps of: displaying, via at least one electronic display, a second image received from the first viewing element, and the second image is illuminated by the one or more illuminators receiving the first voltage.

In other aspects, a method of enhancing images obtained by a medical device system is disclosed. The medical device system may comprise (a) one or more processers, and (b) a medical device operatively coupled to the one or more processers, wherein the medical device is configured to be inserted into a body of a patient and includes a first viewing element and one or more illuminators. The method comprising the steps of: receiving a first image from the first viewing element; determining a current illumination value of the first image; determining a first difference between the current illumination value and a target high illumination value if the current illumination value is greater than the target high illumination value; determining a second difference between the current illumination value and a target low illumination value if the current illumination value is less than the target low illumination value; generating a new illumination value, using at least one of the first difference and the second difference; and converting the new illumination value to a first voltage value for application to the one or more illuminators of the medical device.

In other aspects, the method may include one or more of the following features. The method may further comprise the steps of determining if the new illumination value is greater than a maximum illumination value; if the new illumination value is greater than a maximum illumination value, converting the maximum illumination value to a second voltage value for application to one or more illuminators of the medical device, and increasing a gain of the one or more imaging devices. The method may further comprise the steps of: determining a current exposure time of the first viewing element; generating a new exposure time, using at least one of the first difference and the second difference; and applying the new exposure time to the first viewing element. The method may further comprise the steps of: determining a current frame rate of the first viewing element; generating a new frame rate, using at least one of the first difference and the second difference; and applying the new frame rate to the first viewing element.

In other aspects, a non-transitory computer readable medium may contain program instructions for causing a computer to perform a method of enhancing images obtained by a first viewing element in a medical device system, and the medical device system may comprise a processor configured to implement the process, and a medical device operatively coupled to the processor, the medical device being configured for insertion into a body of a patient and including the first viewing element and one or more illuminators. The method may comprise the steps of: receiving a first image from the first viewing element; determining a current illumination value of the first image; determining a first difference between the current illumination value and a target high illumination value if the current illumination value is greater than the target high illumination value; determining a second difference between the current illumination value and a target low illumination value if the current illumination value is less than the target low illumination value; generating a new illumination value, using at least one of the first difference and the second difference; and converting the new illumination value to a first voltage value for application to the one or more illuminators of the medical device.

DETAILED DESCRIPTION

Reference will now be made in detail to aspects of this disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same or similar reference numbers will be used through the drawings to refer to the same or like parts. The term “distal” refers to a portion farthest away from a user when introducing a device into a patient. By contrast, the term “proximal” refers to a portion closest to the user when placing the device into the patient. InFIGS.1A and1B, arrows labeled “P” and “D” are used to show the proximal and distal directions in the figure. As used herein, the terms “comprises,” “comprising,” or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements, but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. The term “exemplary” is used in the sense of “example,” rather than “ideal.” Further, relative terms such as, for example, “about,” “substantially,” “approximately,” etc., are used to indicate a possible variation of ±10% in a stated numeric value or range.

Embodiments of this disclosure seek to improve the illumination and imaging of a medical device, such as an endoscope, during a medical procedure. As non-limiting exemplary benefits, aspects of this disclosure may reduce the lag experienced with an imaging system and/or may facilitate viewing a field of view of one or more cameras of a medical device, among other aspects.

FIGS.1A and1Bshow perspective views of an exemplary endoscope system100. Endoscope system100may include an endoscope101. Although the term endoscope may be used herein, it will be appreciated that other devices, including, but not limited to, duodenoscopes, colonoscopes, ureteroscopes, bronchoscopes, laparoscopes, sheaths, catheters, or any other suitable delivery device or other type of medical device may be used in connection with the systems and methods of this disclosure, and the systems and methods discussed below may be incorporated into any of these or other medical devices.

Endoscope101may include a handle assembly106and a flexible tubular shaft108. The handle assembly106may include one or more of a biopsy port102, a biopsy cap103, an image capture button104, an elevator actuator107, a locking lever109, a locking knob110, a first control knob112, a second control knob114, a suction button116, an air/water button118, a handle body120, and an umbilicus105. All of the actuators, elevators, knobs, buttons, levers, ports, or caps of endoscope system100, such as those enumerated above, may serve any purpose and are not limited by any particular use that may be implied by the respective naming of each component used herein. The umbilicus105may extend from handle body120to auxiliary devices, such as a control unit175, water/fluid supply, and/or vacuum source. Umbilicus105may transmit signals between endoscope101and control unit175, in order to control lighting and imaging components of endoscope101and/or receive image data from endoscope101. Umbilicus105also can provide fluid for irrigation from the water/fluid supply and/or suction to a distal tip119of shaft108. Buttons116and118may control valves for suction and fluid supply (e.g., air and water), respectively. Shaft108may terminate at distal tip119. Shaft108may include an articulation section122for deflecting distal tip119in up, down, left, and/or right directions. Knobs112and114may be used for controlling such deflection. Locking lever109and locking knob110may lock knobs112and114, respectively, in desired positions.

Distal tip119may include one or more imaging devices125,126and lighting sources127-130(e.g., one or more LEDs, optical fibers, and/or other illuminators). Examples of imaging devices (or viewing elements)125,126include one or more cameras, one or more image sensors, endoscopic viewing elements, optical assemblies including one or more image sensors and one or more lenses, and any other imaging device known in the art. As shown inFIG.1A, distal tip119may include a front-facing imaging device125and a side-facing imaging device126. However, in other embodiments, distal tip119may include only one imaging device125,126, which may be front-facing or side-facing. In other examples, distal tip119may include three or more imaging devices125,126directed in different directions, and in some examples the fields of view of each imaging device125,126may overlap. Distal tip119may include one or more illuminators127-130, and one or more illuminators127,128may be front-facing illuminators (or face the distal direction), and one or more illuminators may be side-facing illuminators129,130. Side-facing imaging device126and side-facing illuminators129,130may face radially outward, perpendicularly, approximately perpendicularly, or otherwise transverse to a longitudinal axis of shaft108and distal tip119. Front-facing or forward-facing imaging device125and front-facing illuminators127,128may face approximately along a longitudinal axis of distal tip119and shaft108.

The disclosed endoscope system100may also include control unit175, as depicted inFIGS.1A and1B. Control unit175may be capable of interfacing with endoscope101to provide power and instructions for imaging devices125,126and illuminators127-130. Control unit175may also control other aspects of endoscope101, such as, for example, the application of suction, the deployment or delivery of fluid, and/or the movement of distal tip119. Control unit175may be powered by an external source such as an electrical outlet. In addition, the control unit175may include buttons, knobs, touchscreens, or other user interfaces to control the imaging devices125,126, illuminators127-130, and other features of endoscope101. The control unit175may be housed in the handle body120itself or in a separate apparatus.

Control unit175may be configured to enable the user to set or control one or more illumination and imaging parameters. For example, control unit175may enable the user to set or control an illumination level for each of illuminators127-130, gain level for each of the imaging devices125,126, exposure time for each of the imaging devices125,126, frame rate of each of the imaging devices125,126, maximum or target values for any of the illumination and imaging parameters, and/or any other parameter associates with the imaging devices125,126and illuminators127-130. In some examples, control unit175may be configured to execute one or more algorithms using one or more illumination and imaging parameters, for example to automatically adjust an illumination level of one or more of illuminators127-130and/or automatically adjust one or more parameters of imaging devices125,126. For example, control unit175may set or select an illumination level for one or more illuminators127-130based on data received from one or more imaging devices125,126.

Control unit175may include electronic circuitry configured to receive, process, and/or transmit data and signals between endoscope101and one or more other devices. For example, control unit175may be in electronic communication with a display configured to display images based on image data and/or signals processed by control unit175, which may have been generated by the imaging devices125,126of endoscope101. Control unit175may be in electronic communication with the display in any suitable manner, either via wires or wirelessly. The display may be manufactured in any suitable manner and may include touch screen inputs and/or be connected to various input and output devices such as, for example, mouse, electronic stylus, printers, servers, and/or other electronic portable devices. Control unit175may include software and/or hardware that facilitates operations such as those discussed above. For example, control unit175may include one or more algorithms, models, or the like for executing any of the methods and/or systems discussed in this disclosure, and may be configured to automatically adjust the illumination value applied to one or more illuminators127-130, and automatically adjust the gain and the frame rate applied to the one or more imaging devices125,126.

In operating endoscope system100, a user may use his/her left hand to hold handle assembly106while the right hand is used to hold accessory devices and/or operate one or more of the actuators of the handle assembly106, such as first and second control knobs112,114and locking lever109and locking knob110. When grasping handle body120, the user may use a left-hand finger to operate image capture button104, suction button116, and/or air/water button118(each by pressing). During a procedure, a user may view the field of view of one or more of imaging devices125,126on an electronic display operable connected to control unit175. The one or more illuminators127-130may provide illumination to the field of view of the one or more of imaging devices125,126.

FIGS.2-4are process flow diagrams illustrating various control-loop algorithms that may be implemented by endoscope system100or any other medical device system with one or more imaging devices and one or more illuminators. Although the algorithms described herein are discussed in relation to an endoscope system, the algorithms are not so limited and may be implemented using any medical device system known in the art that includes imaging components and illumination components. In general, the algorithms discussed herein vary the illumination level based on the changing data received from the one or more imaging devices, for example the changing field of view of an imaging device.

FIG.2illustrates an illumination control method200that may be automatically executed by control unit175of endoscope system100. Method200utilizes proportional, integral, and derivative (PID) coefficients to control the speed and accuracy of the illumination in endoscope system100. At initial step201, a target image brightness is stored in control unit175and an initial illumination value is set by control unit175. In some examples, a user may select a target image brightness, and in other examples control unit175may automatically determine a target image brightness. For example, control unit175may use prior procedure data to determine a target image brightness to use in method200. In some examples, the target image brightness may be a range of brightness values to apply to the one or more illuminators127-130, ranging from a target low (Tlow) brightness value to a target high (Thigh) brightness value. For example, the range of brightness values used for the target image brightness may be a tolerance band set around a particular target brightness value. Also during step201, the user or control unit175may set an initial illumination value to apply to the one or more illuminators127-130.

In the next step202, control unit175may determine an actual illumination value by accumulating and summing the value of the pixels in the current image frame received from one or more imaging devices125,126. In some examples, only a single image frame from a single imaging device125,126may be used to determine the initial illumination value, and in other examples a plurality of image frames may be used from one or more imaging devices125,126.

During the time between frames received from the one or more imaging devices125,126(e.g. during the vertical blanking), step203includes determining the error and PID coefficients for the current image frame. To determine the error coefficient for the current image frame, the control unit may execute the following calculation:

BCurrentis the calculated illumination value of the current image frame. Tlowis the target low illumination value, and Thighis the target high illumination value. In some examples, control unit175may determine the target range of brightness (or range of illumination values) to be a tolerance band around a target illumination value set by the user. To determine the PID coefficients, control unit175may execute the following calculations:

KPis the proportional tuning constant. KIis the integral tuning constant, and KDis the derivative tuning constant. Error is the calculated error coefficient for the current image frame, and ErrorOld is the calculated error coefficient of the immediately prior image frame. The tuning parameters (e.g. tuning constants KP, KI, and KD) are determined experimentally and are dependent on the type of illumination used, as well as the driving circuit. In some examples, tuning constants KP, KI, and KDmay be experimentally determined to achieve a targeted response time. The speed of the PID loop is directly related to the tuning constants KP, KI, and KD. Tuning constant KPadjusts the output in proportion to the current error. Tuning constant KIcontrols the static error. Tuning constant KDis based on the rate of change of the error and provides a damping effect on the output.

In step204, once the error and PID coefficients for the current image frame are determined using the above-described calculations, control unit175may determine a new illumination value to apply to the one or more illuminators127-130, for example, using the old illumination value, the PID coefficients, and a hardware scaling factor. The new illumination value may be determined the following calculation:

FScalingis a scaling factor based on the hardware used to drive the illumination, such as the type of illuminator (LED, fiber optic, etc.) and circuitry connected to the illuminator. Scaling factor FScalingmay be based on the driving circuit, and may depend on the specific hardware implementation and the desired range of allowable illumination. Temporary_Value is an intermediate value used by the control unit to determine BNew, or a new illumination value to apply to the one or more illuminators127-130. Note the calculated illumination values are digital numbers that are converted to an analog voltage, current, or any other digital value controlling illumination applied to the one or more illuminators127-130, thus allowing control unit175to control the illumination of endoscope101. In some examples, control unit175will repeat steps202-204for the next image frame from the one or more imaging devices125,126once step204is completed and BNewis applied to the one or more illuminators127-130. Thus, steps201-204are an example of a control loop algorithm to automatically adjust the illumination of endoscope system100.

In some examples, the control loop algorithm ofFIG.2may include an additional step205of determining if BNew, the new illumination value, is near the top or bottom of the range of illumination values accepted by a particular illuminator, or the range of illumination values the illuminator's hardware is able to accept. In some examples, the control loop algorithm may not include step205and may proceed from step204to step202(shown in dotted lines) to continue cycling through the loop algorithm.

FIG.3illustrates the different steps301-303that control unit175may execute if the BNewis near the bottom or top of the range of illumination values accepted by the one or more illuminators, if the BNewis at the maximum of the range of illumination values accepted by the one or more illuminators, and if the BNewis at the minimum of the range of illumination values accepted by the one or more illuminators, respectively.

In step301, if the BNewis near the bottom or top of the range of illumination values accepted by the one or more illuminators, a different scaling factor (FScaling) is used during the next cycle of the algorithm ofFIG.2. This different scaling factor (FScaling) is smaller than the previously used scaling factor (FScaling) to force the change in the illumination value to be smaller than the previous change in illumination value. Adjusting the scaling factor (FScaling) when BNewis near the bottom or top of the range of illumination values may facilitate the reduction or elimination of any oscillation in the lighting caused by nonlinearities in the hardware of the one or more illuminators.

In step302, BNewis at the maximum of the range of illumination values accepted by the one or more illuminators (e.g. the illumination value is at a maximum). Since the illumination value cannot be increased beyond the maximum of the range of illumination values accepted by the one or more illuminators, control unit175may adjust the digital gain of the one or more image sensors associated with the one or more imaging devices125,126. For example, under normal conditions where BNewis at the maximum of the range of illumination values and the illumination or brightness level of the current image frame is below a minimum target illumination value, the gain of the one or more image sensors of the one or more imaging devices125,126is increased. The gain is then increased (e.g., in a step-wise manner) until the minimum target illumination value for the current image frame is reached or the maximum gain of the one or more image sensors is reached. In some examples, if a saturation value (e.g., a value representing an intensity of color) for the current image frame falls from a target saturation value as the gain of the one or more image sensors is being increased in the step-wise manner, the gain may instead be increased to the maximum gain, while the illumination value for the current image frame is reduced.

In step303, BNewis at the minimum of the range of illumination values accepted by the one or more illuminators (e.g. the illumination value is at a minimum). Since the illumination value cannot be decreased beyond the minimum of the range of illumination values accepted by the one or more illuminators, control unit175may adjust the digital gain of the one or more image sensors associated with the one or more imaging devices125,126. For example, under normal conditions where BNewis at the minimum of the range of illumination values and the illumination or brightness level of the current image frame is above a maximum target illumination value, the gain of the one or more image sensors of the one or more imaging devices125,126is decreased. The gain is then decreased (e.g., in a step-wise manner) until the maximum target illumination value for the current image frame is reached or the minimum gain of the one or more image sensors is reached. In some examples, if a saturation value for the current image frame falls from a target saturation value as the gain of the one or more image sensors is being decreased in the step-wise manner, the gain may instead be decreased to the minimum gain, while the illumination value for the current image frame is increased. After completing any of steps301,302, or303, control unit may continue with another cycle of the control loop algorithm ofFIG.2, for example, starting with step202of determining the actual image brightness for the next image frame received from the one or more imaging devices125,126. When the control loop algorithm executed by control unit175incorporates steps205and301-303, control unit175may automatically switch between (i) adjusting the illumination value applied to the one or more illuminators127-130and (ii) adjusting the gain of the one or more image sensors of the one or more imaging devices125,126.

In some examples, to speed up the response of control unit175to extreme differences between BCurrentand a target illumination value, an extreme image brightness guard may be used to change the gain of the one or more image sensors by values greater than 1. The extreme image brightness guard may be a minimum difference between BCurrentand the target illumination value. Once the extreme image brightness guard is met (or the minimum difference between BCurrentand the target illumination level is met), the gain will be increased by control unit175by a value proportional to the difference between BCurrentand the target illumination value. For example, if the gain is in the low end of the range of gain values, and the BCurrentsuddenly drops well below the target illumination value, the gain is increased by control unit175by a value proportional to the difference between BCurrentand the target illumination value. This scaling factor may be 2, 5, 10, or any number appropriate to more quickly reach the target illumination value. By adjusting the rate at which the gain is increased or decreased, the lag time to reach the target illumination value is decreased. When the extreme image brightness guard is not reached, the gain is increased or reduced by 1 as needed to reach the target illumination value.

FIG.4illustrates another illumination control method400that may be automatically executed by control unit175of endoscope system100(or another control unit). Method400ofFIG.4utilizes proportional, integral, and derivative (PID) coefficients to control the speed and accuracy of the illumination in endoscope system100. At initial step401, a target image brightness (e.g. a target illumination value) is stored in control unit175, and an initial illumination value of the one or more illuminators127-130(e.g. a pre-set illumination value) and an initial exposure time of the one or more imaging devices125,126is determined by control unit175. The initial illumination value applied by control unit175in method400may be a specific illumination value that provides enough light such that the image brightness is adequate for average imaging volumes, at the maximum exposure available in imaging devices125,126for the desired frame rate. This specific illumination value may be set by a user or automatically applied by control unit175. For example, default illumination values may achieve approximately 40-50% image brightness on average, and a user may be able to adjust the illumination value in the range between 25-70% image brightness, based on the user preference and clinical need. In some examples, a user may select a target image brightness (e.g. target illumination value), and in other examples control unit175may automatically determine a target image brightness. For example, control unit175may use prior procedure data to determine a target image brightness to use in the method ofFIG.2. In some examples, the target image brightness may be a range of brightness values to apply to the one or more illuminators127-130, ranging from a target low (Tlow) brightness value to a target high (Thigh) brightness value.

In the next step402, in the same manner as described hereinabove in relation to step202ofFIG.2, control unit175may accumulate and sum the value of the pixels in the current image frame received from one or more imaging devices125,126. In some examples, only a single image frame from a single imaging device125,126may be used to determine the initial illumination value, and in other examples a plurality of image frames may be used from one or more imaging devices125,126.

During the time between frames received from the one or more imaging devices125,126(e.g. during the vertical blanking), step403includes determining the error and PID coefficients for the current image frame. The error and PID coefficients are determined in the same manner as the method described above in relation toFIG.2. FScalingmay change when using the PID loop with exposure time, for example, a user may want to scale the output to have smaller steps to more finely control the illumination. Once the error and PID coefficients for the current image frame are determined, control unit175may determine a new illumination value and, in step404, adjust the exposure time based on the new illumination value. For example, the exposure time is reduced when the image is too bright (e.g. BNew>BCurrent), and the exposure time is increased when the image is too dim (e.g. BCurrent>BNewt). The exposure time may be increased by a single unit, or by plurality of units, after each execution of the PID algorithm loop for the current frame. Once the exposure time reaches a maximum value and the image remains too dim (e.g. BCurrent>BNewt), the illuminator value provided to the one or more illuminators127-130may be increased using the same illuminator control algorithm described hereinabove in relation toFIG.2.

Sensor manufacturers typically have controllable registers for exposure such that the number written to the register of registers is in some fraction of a line. For example, a sensor with 480 lines running at 30 frames per second will have a line time of approximately 65 microseconds. If the exposure number in the register corresponds to 1/16 of a line, then writing the number 16 to the register would give an exposure time of approximately 65 microseconds. The maximum exposure allowed for a particular image sensor, in this example, would be approximately 30 microseconds, since any exposure longer than 30 microseconds will force the frame rate to decrease. By knowing the exposure in fractions of a line, the exposure of an image sensor may be controlled using a PID based algorithm, such as method400ofFIG.4. By adjusting the exposure time of the one or more imaging devices125,126and adjusting the illumination value applied to the one or more illuminators127-130using method400ofFIG.4, the spectral stability of the light used for illuminating target anatomy may be increased. In addition, method400may facilitate minimizing color shifts due to different lighting scenarios.

FIG.5illustrates an exemplary chart of the illumination values applied to one or more illuminators127-130in a PID control system, such as the system described in relation toFIG.2, and the exposure time values applied to one or more imaging devices125,126in a PID control system, such as the system described in relation toFIG.4. By using the slopes of the illumination values and the exposure time values, a more complex method may be utilized to adjust the brightness of an image that switches between (i) adjusting the illumination values of one or more illuminators127-130and (ii) adjusting the exposure time of the one or more imaging devices125,126. For example, if the demand for a change in brightness is large (e.g. above a certain threshold value), the illumination value applied to the illuminators127-130can be adjusted to modify the image brightness. By adjusting the illumination value applied to the illuminators127-130, the increase or decrease in brightness will take effect in less time than if the exposure time was adjusted. If the demand for a change in brightness is small (e.g. below a certain threshold value), the exposure time of the one or more imaging devices125,126can be adjusted to modify the image brightness. The increase or decrease in brightness as a result of a single step up or down in the exposure time will have a smaller effect on the image brightness than a single step up or down in the illumination value. By toggling between adjusting the illumination value applied to illuminators127-130and adjusting the exposure time applied to the imaging devices125,126, the brightness of an image may be controlled with greater precision, particularly near the low limits of the illumination control.

FIG.6illustrates another illumination control method600that may be automatically executed by control unit175of endoscope system100(or another control unit). Method600ofFIG.6utilizes proportional, integral, and derivative (PID) coefficients to control the speed and accuracy of the illumination in endoscope system100. Method600ofFIG.6incorporates control of the frame rate of imaging devices125,126as an additional aspect of controlling brightness of a received image, in combination with any of the other methods discussed hereinabove in relation toFIGS.2-5. As shown inFIG.6, at initial step601, a target image brightness (e.g. a target illumination value) is set (e.g., stored in control unit175), and an initial illumination value of the one or more illuminators127-130is determined by control unit175. Also during step601, an initial frame rate is set (e.g., stored in control unit175) at the one or more imaging devices125,126.

In the next step602, in the same manner as described hereinabove in relation to step202ofFIG.2, control unit175may determine actual image brightness by accumulating and summing the value of the pixels in the current image frame received from one or more imaging devices125,126.

During the time between frames received from the one or more imaging devices125,126(e.g. during the vertical blanking), at steps603and605, control unit175determines whether the actual illumination value of the current image frame is below (step603) or above (step605) the target illumination value.

If the actual illumination value of the current image frame is below (step603) the target illumination value, control unit175will (i) first adjust the illumination value applied to the one or more illuminators127-130until either the maximum illumination value for the one or more illuminators127-130is reached or the target illumination value is reached, and then (ii) adjust the gain value applied to the one or more imaging devices125,126until either the maximum gain value is reached or the target illumination value is reached. As shown in step604, if both the illumination value applied to the one or more illuminators and the gain value applied to the one or more imaging devices125,126are at their respective maximum values, control unit175will proceed to decrease the frame rate of the one or more imaging devices125,126, to allow for an increase in exposure time, until either the target illumination value is reached for the received image or the minimum frame rate is reached.

If the actual illumination value of the current image frame is above (step605) the target illumination value, control unit175will increase the frame rate applied to the one or more imaging devices125,126until either the target illumination value is reached for the received image or the maximum frame rate is reached. Then, if the actual illumination value is still above the target illumination value and the maximum frame rate is reached (step606), control unit175will then (i) adjust the illumination value applied to the one or more illuminators127-130until either the minimum illumination value for the one or more illuminators127-130is reached or the target illumination value is reached, and then (ii) adjust the gain value applied to the one or more imaging devices125,126until either the minimum gain value is reached or the target illumination value is reached. By combining the automatic adjustment of the illumination value applied to the one or more illuminators127-130, the gain value applied to the one or more imaging devices125,126, and the frame rate applied to the one or more imaging devices125,126, the image brightness may more efficiently be adjusted, for example, to minimize color shifts due to different lighting scenarios. A higher frame rate may also lead to a reduced video latency, as well as allowing for the appearance of a “smoother” video.

In various embodiments, any of the systems and methods described herein may include control unit175and a medical device (e.g., endoscope101), and control unit175may include a processor, in the form of one or more processors or central processing unit (“CPU”), for executing program instructions. In some examples, the one or more processors may be one or more processing boards. Control unit175may include an internal communication bus, and a storage unit (such as ROM, HDD, SDD, etc.) that may store data on a computer readable medium, although control unit175may receive programming and data via network communications. Control unit175may also have a memory (such as RAM) storing instructions for executing techniques presented herein, although the instructions may be stored temporarily or permanently within other modules of control unit175(e.g., processor and/or computer readable medium) or remotely, such as on a cloud server electronically connected with control unit175. The various system functions of control unit175may be implemented in a distributed fashion on a number of similar platforms, to distribute the processing load. Alternatively, the systems discussed herein may be implemented by appropriate programming of one computer hardware platform at control unit175.

FIG.7provides a functional block diagram illustration of general purpose computer hardware platforms.FIG.7illustrates a network or host computer platform, as may typically be used to implement a server700or a browser, or any other device executing features of the methods and systems described herein. It is believed that those skilled in the art are familiar with the structure, programming, and general operation of such computer equipment and as a result, the drawings should be self-explanatory.

A platform for the server700or the like, for example, may include a data communication interface for packet data communication760. The platform may also include a central processing unit (CPU)720, in the form of one or more processors, for executing program instructions. The platform typically includes an internal communication bus710, program storage, and data storage for various data files to be processed and/or communicated by the platform such as ROM730and RAM740, although the server700often receives programming and data via network communications770. The hardware elements, operating systems, and programming languages of such equipment are conventional in nature, and it is presumed that those skilled in the art are adequately familiar therewith. The server700also may include input and output ports750to connect with input and output devices such as keyboards, mice, touchscreens, monitors, displays, etc. Of course, the various server functions may be implemented in a distributed fashion on a number of similar platforms, to distribute the processing load. Alternatively, the servers may be implemented by appropriate programming of one computer hardware platform.

While the disclosed methods, devices, and systems are described with exemplary reference to control unit175, it should be appreciated that the disclosed embodiments may be applicable to any environment, such as a desktop or laptop computer, etc. Also, the disclosed embodiments may be applicable to any type of Internet protocol.

Thus, while certain embodiments have been described, those skilled in the art will recognize that other and further modifications may be made thereto without departing from the spirit of the invention, and it is intended to claim all such changes and modifications as falling within the scope of the invention. For example, functionality may be added or deleted from the block diagrams and operations may be interchanged among steps shown in the figures. Steps may be added or deleted to methods described within the scope of the present invention.