Patent Description:
Dual image acquisition can be a useful feature in the field of endoscopy. Two video streams of the same scene are captured, but each of the captured image streams has different associated characteristics such as a variation in light spectrum, depth of field, or light intensity. In prior dual image systems, images have generally been collected, split in image space, and then focused onto two independent detectors. Such a configuration allows for more versatility than a single image acquisition system, but is generally more expensive and complex, requiring at least two sensors and associated electronics and mounting assemblies.

Some prior art systems capture multiple images from a single image sensor using processing techniques to generate high dynamic range (HDR) images. The intensity level of image light captured varies when acquiring each of the multiple images, generally in rapid succession. A resultant or composite HDR image is created by mathematically combining the multiple images. Such designs have significant limitations due to lack of flexibility in positioning desired optical elements such as filters, lenses, and other elements in the optical path, and due to the delay time of adjusting the light level and acquiring each of the multiple images needed to create an HDR image for the frame.

<CIT> discloses an imaging device for generating a wide dynamic range image that is less blurred even in case of a moving object, wherein the imaging device comprises an imager including a plurality of first pixels including a polarizing filter having a pre-determined polarization direction as well as a plurality of second pixels not including any polarization filter, wherein a polarizing element is provided on an incident plane side of the imager. In <CIT> a similar imaging apparatus is described with an imaging section generating a pixel signal based on incident light acting as polarization pixels each having one of a plurality of polarizing directions and a polarization element attached on a side of an incident plane of the imaging section as well as a sensitivity direction section to detect a sensitivity level of the polarization pixels.

<CIT> discloses an endoscope device including a reflection-presence image acquisition unit that acquires a reflection-presence image containing a specular reflection component from a subject, a reflection-absence image acquisition unit that acquires a reflection-absence image not containing the specular reflection component from the subject and a synthesis processing unit that performs synthesis of the reflection-presence image and the reflection-absence image. Hereby, the incident light irradiated from a light guiding section is first polarized at a polarizer before falling on the subject and a polarization beamsplitter can be used in an imaging section to split the diffusion reflection component from the subject for passing to a first imaging section or a second imaging section, so that the reflection-present image is acquired.

What is needed are devices and methods to enable an endoscopic camera to acquire HDR images in a cost effective manner. What is further needed are devices allowing the use of varied existing endoscopes for HDR imaging applications, and allowing the detection of the varied characteristics in the HDR images.

It is an object of the invention to provide improved endoscopic acquisition of multiple images, and to allow the use of varied existing endoscopes for multiple imaging applications. It is another object of the invention to enable a simpler and faster method to acquire images of varying intensity necessary to generate HDR images. It is a further object of the invention to permit image analysis to make use of varying states of polarization from a single captured image for purposes such as glare reduction and polarization studies. Endoscopic camera head devices and methods are provided using light captured by an endoscope system. At least one polarizing optical element manipulates the polarization properties of image light. The resulting image light is passed to an image sensor configured with pixels equipped with differently polarizing filters to produce images of differing intensity from the light content collected at different polarization states. The resulting image or images are processed to produce an image with high dynamic range.

The problem is solved by an imaging system comprising an image sensor including a plurality of pixels arranged as a plurality of subgroups each subgroup containing at least two pixels, at least one pixel in a respective subgroup having an associated polarizer, whereby light detected by at least a first pixel of each subgroup is differently polarized than light detected by a second pixel of a subgroup; a polarizing optical element distal to the image sensor which by its polarizing characteristics is capable of controlling the relative intensity detected by the pixels in each subgroup; and a processor adapted to receive image data from the image sensor and to create a plurality of images from the received image data, each created from a set of pixels spanning a plurality of subgroups and having the same respective polarizer orientation, the processor further adapted to combine the plurality of image data from the image sensor and the second image sensor and generate a high dynamic range (HDR) image therefrom; wherein the polarizing optical element comprises a beamsplitter, splitting an image beam into a first beam with a first polarization and a second beam with a second polarization, the first beam directed to the image sensor including a plurality of pixels of subgroups, and the second beam directed to a second image sensor, wherein the imaging system further comprises a second polarizing optical element arranged along an optical path between the beamsplitter and the image sensors including a plurality of pixels of subgroups.

According to the first aspect of the invention, an imaging system is provided including an image sensor, a polarizing optical element, and a processor. The image sensor includes a plurality of pixels arranged as a plurality of subgroups each subgroup containing at least two pixels, at least one pixel in a respective subgroup having an associated polarizer, whereby light detected by at least a first pixel of each subgroup is differently polarized than light detected by a second pixel of a subgroup. The polarizing optical element is distal to the sensor which by its polarizing characteristics is capable of controlling the relative intensity detected by the pixels in each subgroup. The processor adapted to receive image data from the image sensor and generate a high dynamic range (HDR) image therefrom.

According to the invention, the processor is adapted to receive image data from the image sensor and create a plurality of images, each created from a set of pixels spanning a plurality of subgroups and having the same respective polarizer orientation. The processor is further adapted to combine the plurality of images to generate a high dynamic range (HDR) image.

In a further implementation of the imaging system, the polarizing optical element changes the relative intensity detected by the pixels in each subgroup dynamically based on the content, such as the intensity value detected at a selected plurality of pixels, of the received image. In still another implementation, the polarizing optical element is a polarizer that rotates, thereby adjusting the relative intensity detected by the various pixels in each subgroup dynamically based on the content of the received image. The polarizing optical element may also be a polarizer followed by a variable retarder, and wherein the retardance of the variable retarder is changed thereby changing the relative intensity detected by the various pixels in each subgroup dynamically based on the content of the received image. These dynamic changes can be a result of calculating at least one statistical property for the plurality of images.

In a further implementation of the imaging system, the processor is adapted to calculate at least one statistical property for the plurality of images, and based at least in part on the values of the statistical properties, generate a control signal for the polarizing optical element.

In another implementation of the imaging system, the processor is adapted to, when combining plurality of images to generate a HDR image, select a pixel intensity value for each of the plurality of subgroups based on at least one intensity value from a pixel in the subgroup.

According to the invention, the polarizing optical element comprises beamsplitter, splitting an image beam into a first beam with a first polarization and a second beam with a second polarization, the first beam directed to the image sensor including a plurality of pixels of subgroups, and the second beam directed to a second image sensor. A second polarizing optical element is included along an optical path between the beamsplitter and the image sensor including a plurality of pixels of subgroups.

According to a second aspect of the invention, a method as defined in claim <NUM> is provided for producing enhanced images from a scope. The method includes collecting image light at the scope from an illuminated scene and directing the image light along an optical channel. The method manipulates the polarization properties of the image light. The image light is focused on an image sensor. The method polarizes image light passing to individual pixels of the image sensor, the pixels arranged as a plurality of subgroups each containing at least two pixels, at least one pixel in a respective subgroup having a respective polarizer oriented in a fixed direction associated therewith. The method receives image data from the image sensor and combines a plurality of pixel data to generate a high dynamic range (HDR) image.

According to the inventive method, the combination of pixel data to generate an HDR image includes the combination of a plurality of images, each created from a set of pixels having the same respective polarizer orientation. In some implementations, the relative polarizing optical element changes the intensity detected by the various pixels in each subgroup dynamically based on the content, such as the intensity value detected at a selected plurality of pixels, of the received image. In some implementations the changing of the relative intensity is performed by modifying the angle of a rotating polarizer along an optical path, or alternatively changing of the relative intensity is performed by a polarizer followed by a variable retarder, wherein the retardance of the variable retarder is changed, thereby changing the relative intensity detected by the various pixels in each subgroup. The changes can be made dynamically in response to the content of the received image. These dynamic changes can be a result of calculating at least one statistical property for the plurality of images, and based at least in part on the values of the statistical properties, performing the step of changing the relative intensity.

According to another implementation of the method, the combining the plurality of images to generate a HDR image includes, for each of the plurality of subgroups, selecting a pixel intensity value based on at least one intensity value from a pixel in the subgroup.

In another implementation of the method, the directed image light is divided into two beams. According to the invention the directed image beam is split based on polarization characteristics, such that one resulting beam has different polarization characteristics than the other resulting beam. In some implementations of this aspect of the invention, one of the beams may be received by a second image sensor.

These and other features of the invention will be apparent from the following description of the preferred embodiments, considered along with the accompanying drawings.

The present invention will become more fully understood from the detailed description given herein and the accompanying drawings which are given by way of illustration only, and thus are not limitative of the present invention, and wherein:.

As used herein, first elements (e.g., sensors and lenses) that are "optically arranged" in relation to other elements, refers to the first elements' position along a common optical path that includes first and other elements. For example, a lens group optically arranged between an image sensor and an objective, means that the lens group occupies a portion of the optical path that light travels (e.g., from the objective to the image sensor) for capturing images or video. Directions such as upstream and downstream refer to the direction of light travel.

Because digital cameras, image sensors and related circuitry for signal capture and processing are well-known, the present description will be directed in particular to elements forming part of, or cooperating more directly with, a method and apparatus in accordance with the invention. Elements not specifically shown or described herein are selected from those known in the art. Certain aspects of the embodiments to be described are provided in software. Given the system as shown and described according to the invention in the following materials, software not specifically shown, described or suggested herein that is useful for implementation of the invention is conventional and within the ordinary skill in such arts.

<FIG> is a block diagram of a medical imaging device <NUM> according to an example embodiment of the invention. Medical imaging device <NUM> ("device <NUM>") includes a camera head <NUM> which may have an endoscope <NUM> attached via connectors <NUM> and <NUM>. In some embodiments, an endoscope <NUM> and camera head <NUM> may be integrated into a single housing with no connectors needed. In some embodiments, the device <NUM> is provided as only the camera head <NUM> adapted to be connected to a suitable endoscope. Connectors <NUM> and <NUM> in this embodiment are standard eyecup style optical connectors but may be any suitable connector allowing light to pass from endoscope <NUM> to camera head <NUM>. Various structural components supporting the depicted elements are omitted in the diagrams herein, as well as other components such as illumination lights sources and controls, which are known in the art and are not shown in order to avoid obscuring the relevant details of the example embodiments of the invention.

Camera head <NUM> includes an optical group <NUM> positioned at or behind a central window of connector <NUM> to receive and condition optical image light from the endoscope <NUM> which contains one or more focusing lenses for focusing light onto image sensor <NUM>. Optically arranged to receive image light from optical group <NUM> is a polarization control element <NUM>, which alters the polarization properties of the image light.

The optical group <NUM> is generally for shaping, directing and focusing light received from the scope <NUM> onto the image sensor <NUM>. The optical group <NUM> typically includes at least one focusing lens, with the group having a total positive power. Many suitable lenses and combinations of lenses may be used for optical group <NUM>.

Polarization control element <NUM> is preferably an adjustable polarization control element which is able to vary the polarization properties of the image light over time. For example, polarization control element <NUM> may be a rotating polarizer which in operation is rotated to vary polarization direction of the light passed to image sensor <NUM>. Polarization control element <NUM> may also be constructed with a linear polarizer followed by a variable retarder. In such an embodiment, there is no physical rotation, and polarization is controlled by varying the retardance of the variable retarder. Examples of constructing polarization control element <NUM> will be further described below.

Image sensor <NUM> is optically arranged to receive the image light from polarization control element <NUM>. Image sensor <NUM> includes a plurality of pixels that are arranged as a plurality of subgroups each subgroup containing at least two pixels, at least one of the pixels in a respective subgroup having a respective polarizer. In many implementations each pixel in a respective subgroup has an associated polarizing filter oriented in a different direction from the other polarizing filters associated with the other pixels in the subgroup. In some embodiments one or more pixels of each subgroup may have no associated polarization filter. This arrangement provides ability to differentiate image light of different polarizations, producing multiple images each created from a set of pixels having the same respective polarizer orientation. Because the pixel subgroups of image sensor <NUM> detect light of different polarizations, polarization control element <NUM> is capable of controlling the relative intensity detected by the pixels in each subgroup.

Image data from image sensor <NUM> is generally processed to treat the image data of the multiple images separately, providing images with different intensity ranges. These images are then combined to provide a high dynamic range (HDR) single combined image with higher dynamic range than any of the raw images taken individually. Such processing is described further below.

In some embodiments, system <NUM> includes an endoscope <NUM> as depicted at the left of the block diagram. The depicted endoscope is an example only, and many endoscope and borescope designs are suitable, including rigid and flexible endoscopes and borescopes. The exemplar endoscope <NUM> includes a cover glass <NUM> at its distal tip, which in this version faces directly along the longitudinal axis of the endoscope <NUM>, but may also be positioned at an angle relative to the longitudinal axis as is known in the art. Behind, or on the proximal side of, the cover glass <NUM> is shown a preferred position for the objective lens <NUM>, set against or very near cover glass <NUM> and preferably assembled together with the cover glass in construction. While a wide-angle lens is preferred for objective lens <NUM>, this is not limiting and any suitable lens may be used in various embodiments. Objective lens <NUM> may be part of an objective lens group <NUM> which may include one or more additional lenses <NUM>. The particular number and arrangement of lenses in the endoscope <NUM> will vary widely depending on the application. Optically arranged or attached at the proximal side of objective lens <NUM> or objective lens group <NUM> is a series of one or more rod lenses <NUM> that serve to pass the light down endoscope <NUM> towards its proximal end. Typically several rod lenses <NUM> are employed, which may be separated by spacers or other lenses in any suitable manner known in the art.

<FIG> is a cross section diagram of an optical assembly <NUM> of camera head <NUM> of <FIG>, that is not part of the invention. The view includes a light ray diagram showing the passage of image light through optical assembly <NUM> to image sensor <NUM>. The depicted optical elements are in diagram form only and are not drawn to scale. Optical assembly <NUM> includes an optical group <NUM> which receives light from an attached or integrated endoscope.

The image light received from the scope is directed in the single optical channel of optical group <NUM>. Optical Group <NUM> includes lenses <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM>. Concave-convex lens <NUM>-<NUM> optically arranged to receive the image light and having a slightly negative optical power spreading the image light to a desired size. Optically arranged to receive image light from lens <NUM>-<NUM> is plano-concave lens <NUM>-<NUM>, which has a negative optical power to align the image light. Optically arranged along the planar area of lens <NUM>-<NUM> is convex-plano lenses <NUM>-<NUM>, with a positive power for further conditioning the image light, a doublet achromat lens with a biconvex lens <NUM>-<NUM> and a concave-convex lens <NUM>-<NUM>. Group <NUM> has a total positive optical power as indicated by the converging effect on the depicted ray lines.

A polarization control element <NUM> is optically positioned between optical group <NUM> and image sensor <NUM>. Polarization control element <NUM> is arranged to receive the optical image light from optical group <NUM> and alter the polarization properties of the image light. As discussed above, polarization control element <NUM> is preferably adjustable and may be constructed in a number of suitable ways, such as described with respect to the examples shown in <FIG> and <FIG>. Polarization control element <NUM> varies the polarization properties of the incoming light, and this variation may be changed at different time intervals, such that the relative intensity of light at different polarizations is varied as the light is passed to image sensor <NUM>. The light at various polarizations is then detected at image sensor <NUM> and employed to produce multiple images of different intensities. An HDR image is then produced from the multiple images, with higher apparent dynamic range than either a first or second image taken individually. Further, with multiple images captured containing different polarization characteristics, further advantages may be achieved such as glare reduction and improved feature visibility over a traditional HDR imaging technique. In addition, it is possible to observe polarizing effects of various tissues and other objects under observation.

<FIG> shows a cross sectional diagram of a portion of an example sensor which may be employed in some embodiments. The diagram illustrates the structure of an image sensor chip suitable for use with the techniques disclosed herein, such as the IMX250MYR RGGB image sensor (Sony Semiconductor Solutions Corporation, Japan). In order to more easily illustrate the utility of the polarizing sensor, it may be beneficial to contrast the polarizing sensor with a standard single chip CMOS or CCD sensor employing a Bayer pattern color filter as shown in <FIG>. A standard Bayer filter image sensor utilizes four individual pixels in a single calculation unit. The image sensor shown in <FIG> comprises <NUM> individual pixels in four separate calculation units. Each of the pixels is covered by an individual color filter. Light energy collected by each individual photodetector located beneath each filter collects a given intensity of light of the color that passes through the filter. Two green pixels have been conventionally used in each calculation unit in order to mimic sensitivity of the human eye to the green part of the visible spectrum, see, for example, <CIT>), incorporated herein by reference. As a result of the filtering array, each pixel represents only a single color content, and a demosaicing algorithm is generally used to estimate a color and intensity value for each pixel, however it is helpful to remember that the actual collected image resolution is decreased as a result of the Bayer filter in contrast to that a monochromatic image, wherein each pixel represents an actual absolute measured light intensity value. As can be seen in the cross section of the polarizing sensor shown in <FIG>, according to the image sensors used in the present invention, in contrast to the simple Bayer color filter shown in <FIG>, pixels are arranged in blocks wherein each pixel in a block is filtered at an individual polarization angle, as illustrated in <FIG>. The diagram generally shows an idealized cross section of a segment of image sensor <NUM>, showing a small array with very few pixel elements in order to illustrate the construction. However, typically an image sensor will have an array of many hundreds of pixels in each dimension, for example <NUM> pixels along one dimension. The sensor construction will vary with different technologies and may include more or less layers, with this example shown for illustration purposes. Sensor <NUM> includes a cover glass <NUM>, which is typically fixed in packaging or as an element of an integrated sensor assembly. Beneath cover glass <NUM> are micro-lenses <NUM> which collect and direct the impinging light toward the photodiodes <NUM>. Each micro-lens <NUM> is typically associated with a single photodiode. A grid of polarizers <NUM> is positioned under microlenses <NUM>. <FIG> shows a top diagram view of the grid of polarizers <NUM> for a 4x4 array.

Referring to <FIG> and <FIG>, polarizers <NUM> are generally constructed with micro-wires in a micro-wire grid frame mounted to a glass substrate which sits above photodiodes <NUM>. Each polarizer includes rows of micro-wires or other structures that only pass light polarized in the direction perpendicular to that of the micro-wires, as is known in the art. Light of other polarizations is increasingly blocked the more the polarization of the incoming light varies from the orientation of the polarizer. As shown in <FIG>, in this embodiment polarizers <NUM> are arranged in groups of four, a repeating two-by-two grid, with the lower right depicted polarizer <NUM> at zero degrees orientation, the lower left at <NUM> degrees, the upper right at <NUM> degrees, and the upper left at <NUM> degrees.

Certain embodiments of this invention make use of a sensor array, a top-view portion of which is illustrated in <FIG>, wherein polarizers <NUM> rest atop a color filter layer <NUM>, which includes a color filter array such as a Bayer filter array of red, green, and blue filters discussed above, in which each color filter spans a two-by-two section of polarizers <NUM> as well as spanning a two-by-two grid of photodiodes <NUM>, illustrating a 4x4 grid, or calculation unit, wherein each polarization state in each color is represented by at least one pixel in the <NUM> square pixel grid. In this way it can be seen that a trade-off is made between resolution and polarization measurements in an analogous way to the trade-off made when utilizing a Bayer filter on a color sensor. By contrast a color, four-state monochromatic polarization detecting sensor, as shown in <FIG>, will have approximately half the effective vertical and horizontal resolution of a comparable color sensor, while effectively generating four approximately quarter resolution color images, each detecting light of a unique polarization state. Color filter layer <NUM> may be integrated with the glass substrate of polarizers <NUM>, or the order may be reversed.

A metalized interconnect layer (not shown) may be between the color filter <NUM> layer and the photodiode <NUM> layer, or between the photodiode <NUM> layer and the carrier substrate <NUM>, and in some cases may be integrated with the photodiode layer. Some sensor designs place the micro-lens <NUM> layer on-chip, that is, under the color filter <NUM> layer or under the layer of polarizers <NUM>.

<FIG> shows a diagram of the combined transmission of a polarization control element upstream of an image sensor with integrated polarizers such as those of <FIG>. The diagram shows four graphs, representing light intensity at four polarization angles of <NUM> degrees, <NUM> degrees, <NUM> degrees, and <NUM> degrees (the angles passed by the various polarizers in <FIG>). The diagram shows the effect of a polarization control element with a rotating polarizer, as the polarizer is rotated. The angle of the rotating polarizer is shown along the horizontal axis, and the light intensity of transmitted light is shown on the vertical axis.

As the polarization control element varies the polarization angle of the light, the polarization-sensitive pixels of the image sensor detect light at different intensities. This effect allows the simultaneous capture of multiple images with different intensities for use with HDR imaging. Alternately the captured image data can be otherwise manipulated for various polarization studies or for other purposes.

In non-HDR systems with a single captured image, a feature within the captured image might be washed out or obscured in shadow, depending on the illumination of the scene and the settings of the image sensing apparatus. In traditional HDR systems, an HDR image is constructed from, for example, three images of three different intensity levels including a high intensity (bright) image, a medium intensity image, and a low intensity (dim) image. These images are usually captured in rapid succession while varying the exposure by adjusting the iris, the light source, or the sensor integration time. This allows features to be distinguished even if they are very dim or very bright, by selecting ideally exposed regions of each captured images and combining them, or by using other video processing techniques known in the art. The techniques herein make use of the polarization-sensitive image sensor to capture multiple images of a single scene at various intensity levels determined by the polarization angle of each pixel as well as the state of the polarization control element. In the case of a rotating polarization control element, in the present example, four images can be captured in any of the states of the rotating polarizer. For example, at the fifteen degree rotation position shown on the chart in <FIG>, the sensor may be used to capture an image from the <NUM> degree pixels of about <NUM>% of the illumination passing through the rotating polarizer, <NUM>% for the <NUM> degree pixels, <NUM>% from the <NUM> degree pixels, and <NUM>% from the <NUM> degree pixels, providing four images at varying light intensity levels. Or, the sensor may be used to acquire one or more images at a first rotation position and then acquire one or more additional images at another rotation position. In this manner, a precise desired intensity level may be achieved by controlling exactly where on the depicted curves an image is acquired for any particular one of the four sets of polarized pixels.

<FIG> shows a cross section diagram of a polarization control element <NUM> constructed as a rotating polarizer according to some embodiments. <FIG> is a front view diagram of the same polarization control element <NUM>. The depicted implementation of a polarization control element may be used in the design of <FIG>, for example, to achieve the effect depicted in <FIG>. Polarization control element <NUM> includes a housing <NUM> forming a circular opening in which a polarizing filter <NUM> is rotatably mounted. Rotating movement of polarizing filter <NUM> is depicted by the arrow. Construction of rotating polarizers is known in the art, and may include an electric motor or other drive mechanism mounted in housing <NUM> and operable to produce the rotation using a drive gear matched to grooves along the perimeter of polarizing filter <NUM>. In operation, the drive mechanism may rotate polarizing filter <NUM> continuously or in discrete movements upon command.

<FIG> shows another diagram of the combined transmission of a polarization control element in front of an image sensor with integrated polarizers, using a different polarization control element implementation for this embodiment. Like <FIG>, the diagram <NUM> shows four graphs, representing light intensity at four polarization angles of the sensor's polarizers. The diagram shows the effect of a polarization control element implemented with a variable retarder. The retardance of the variable retarder in phase units (degrees of phase retardance) is shown along the horizontal axis, and the light intensity of transmitted light is shown on the vertical axis.

Similarly to the embodiment of <FIG>, the polarization-sensitive image sensor may be used to capture multiple images in any of the depicted states of the polarization control element.

<FIG> shows a cross section diagram of a polarization control element <NUM> including a variable retarder <NUM> and a linear polarizer <NUM> according to some embodiments. <FIG> shows a front view diagram of the variable retarder <NUM> of <FIG> shows an enlarged cross section view of the variable retarder <NUM> of <FIG>. Referring to <FIG>, polarization control element <NUM> may be used in the embodiment of <FIG> or other embodiments in which a polarization control element is optically arranged following a beamsplitter. Polarization control element <NUM> is depicted with an arrow in <FIG> showing the direction of light passage, first through linear polarizer <NUM> and then through variable retarder <NUM>.

Variable retarder <NUM> includes a frame <NUM> with control electronics, and a retarding medium <NUM>. In the cross-section of <FIG>, the construction of one example retarding medium <NUM> is shown. Many suitable variable retarder technologies may be used, with a preference for those that can be miniaturized. In this embodiment, retarding medium <NUM> is a thin layer of a liquid crystal fluid sandwiched between two transparent electrodes <NUM>. The liquid crystal molecules are highly anisotropic. When a voltage is applied to electrodes <NUM> through electrical circuitry in housing <NUM>, an electric field between electrodes <NUM> orients the liquid crystal molecules. The field-induced orientation of the liquid crystal molecules causes a change in the index of refraction of the liquid crystal layer. This change results in alteration of the optical phase of light passing through the liquid crystal layer. In operation, variable retarder <NUM> is controlled to produce at least two different polarization conditions, and typically is varied back and forth across some or all of the depicted range of retardance shown in <FIG>.

Linear polarizer <NUM> includes a frame <NUM> and a linear polarizing filter <NUM>. Many different linear polarizer technologies may be used for linear polarizer <NUM>. The shown fixed orientation of the linear polarizing filter <NUM> is selected to achieve the result of <FIG>.

As noted above, certain embodiments of the invention employ a beamsplitter as part of the optical assembly. The beamsplitter splits the beam based on polarization, as shown in <FIG>. An optical group <NUM> receives light from an attached or integrated endoscope and focuses and directs the image light to image sensors <NUM> and <NUM>. Downstream from the optical group <NUM> is a beamsplitter, <NUM>, which may be part of a beam splitting prism <NUM>. The beamsplitter splits the incoming light based on polarization, sending light of one polarization towards sensor <NUM>, while the remaining polarized light beam is directed to a second sensor <NUM>. The second sensor <NUM> in this example is a specialized sensor discussed above, wherein individual pixels are arranged in subgroups, and one or more pixels within those subgroups are equipped with a polarization filter, as shown previously in <FIG>. Between the beamsplitter <NUM> and the second sensor <NUM> is placed a polarization control element <NUM> that may take the form of any of the above mentioned variations or a fixed retarder. The polarization control element may be absent from the system, the specialized image sensor working in conjunction with the polarizing beamsplitter, to provide light of varying intensities directly to the pixels of each subgroup. Light may be split evenly or unevenly. For example, <NUM>% of the light may be directed to the second image sensor <NUM>, with the remaining light sent to the first image sensor <NUM>. It should further be noted, that in these embodiments, it may not be necessary, or even desirable, to employ a Bayer filter on the second image sensor <NUM>. Image data received by the image processor from the second image sensor may then be used to define regions of higher and lower exposure, and enhance the dynamic range of a resultant image by combining the data received by the two sensors.

<FIG> shows a block diagram of system including an endoscope device and an image capture device, wherein the image capture devicebeing not part of the invention, according to some embodiments. The invention is applicable to more than one type of device enabled for image capture, such as FI-capable endoscopes, other medical or industrial imaging devices. The preferred version is an imaging scope system, such as an endoscope.

The diagram shows a light source <NUM> which illuminates subject scene <NUM> with visible light. Fluorescent excitation light may also be used, which may be outside the visible spectrum in the ultra-violet range or the infra-red/near infrared range, or both. Light source <NUM> may include a single light emitting element configured to provide light throughout the desired spectrum, or a visible light emitting element and a one or more fluorescent excitation light emitting elements. Further, light source <NUM> may include fiber optics passing through the body of the scope, or other light emitting arrangements such as LEDs or laser diodes positioned at or near the front of the scope.

As shown in the drawing, light <NUM> reflected from (or, alternatively, as in the case of fluorescence, excitation light <NUM> absorbed and subsequently emitted by) the subject scene is input to an optical assembly <NUM>, where the light, modified as described herein, passes to image sensor assembly <NUM> where it is focused to form an image at a solid-state image sensor <NUM>. Optical assembly <NUM> includes the optics of the endoscope and of the camera head. As discussed above, portions of the optical assembly may be embodied in a camera head attached to a scope, or in a single imaging device. Image sensor <NUM> converts the incident light to an electrical signal by integrating charge for each picture element (pixel). Image sensor <NUM> includes polarization sensitive pixels as described herein.

The total amount of light <NUM> reaching the image sensor <NUM> is regulated by the light source <NUM> intensity, the optical assembly <NUM> aperture, the attenuation of optical path, the polarization filters of the pixels. The exposure can further be controlled by the time for which the image sensor <NUM> integrates charge. An exposure controller <NUM> responds to the amount of light available in the scene given the intensity and spatial distribution of digitized signals corresponding to the intensity and spatial distribution of the light focused on image sensor <NUM>. If fluorescence imaging is used, exposure controller <NUM> also controls the emission of fluorescent excitation light from light source <NUM>, and may control the visible and fluorescent light emitting elements to be on at the same time, or to alternate to allow fluoresced light frames to be captured in the absence of visible light if such is useful in the fluorescent imaging scheme employed. Exposure controller <NUM> may also control the optical assembly <NUM> aperture, and indirectly, the time for which the image sensor <NUM> integrates charge. The control connection from exposure controller <NUM> to timing generator <NUM> is shown as a dotted line because the control is typically indirect. Exposure controller <NUM> may also command the state of polarization control element <NUM>, <NUM> in optical assembly <NUM>.

Timing generator <NUM> produces various clocking signals to select rows and pixels and synchronizes the operation of image sensor <NUM>, analog signal processor <NUM>, and A/D converter <NUM>. Image sensor assembly <NUM> includes the image sensor <NUM>, the analog signal processor <NUM>, the A/D converter <NUM>, and the timing generator <NUM>. The functional elements of the image sensor assembly <NUM> can be fabricated as a single integrated circuit as is commonly done with CMOS image sensors or they can be separately-fabricated integrated circuits.

Analog signals from the image sensor <NUM> are processed by analog signal processor <NUM> and applied to analog-to-digital (A/D) converter <NUM> for digitizing the analog sensor signals. The digitized signals each representing streams of images or image representations based on the data, are fed to image processor <NUM> as image signal <NUM>. The images captured at different polarizations are typically transmitted as separate images within image signal <NUM>.

The system camera control unit (CCU) <NUM> includes image processing circuitry <NUM> performing digital image processing functions to process and filter the received images as is known in the art. Image processing circuitry may include separate, parallel pipelines for processing the multiple images separately. CCU <NUM> may be implemented in a single assembly or may include two or more camera control modules performing different functions such as communication with a specific camera model, and image processing. Such circuitry is known in the art and will not be further described here. Image processing circuitry <NUM> may provide algorithms, known in the art, for combining two images of the same view but containing different characteristics in a combined image display.

The system controller <NUM> controls the overall operation of the image capture device based on a software program stored in program memory <NUM>. This memory can also be used to store user setting selections and other data to be preserved when the camera is turned off. System controller <NUM> controls the sequence of data capture by directing exposure controller <NUM> to set the light source <NUM> intensity, the optical assembly <NUM> aperture, and controlling various filters in optical assembly <NUM> and timing that may be necessary to obtain image streams. A data bus <NUM> includes a pathway for address, data, and control signals.

Processed image data are continuously sent to video encoder <NUM> to produce a video signal. This signal is processed by display controller <NUM> and presented on image display <NUM>. This display is typically a liquid crystal display backlit with light-emitting diodes (LED LCD), although other types of displays are used as well. The processed image data can also be stored in system memory <NUM> or other internal or external memory device.

The user interface <NUM>, including all or any combination of image display <NUM>, user inputs <NUM>, and status display <NUM>, is controlled by a combination of software programs executed on system controller <NUM>. User inputs typically include some combination of typing keyboards, computer pointing devices, buttons, rocker switches, joysticks, rotary dials, or touch screens. The system controller <NUM> manages the graphical user interface (GUI) presented on one or more of the displays (e.g. on image display <NUM>). In particular, the system controller <NUM> will typically have a mode toggle user input (typically through a button on the endoscope or camera head itself, but possibly through a GUI interface), and in response transmit commands to adjust image processing circuitry <NUM> based on predetermined setting stored in system memory. Preferably a system employed with any of the device designs herein provides ability to toggle between an individual view of either single image (for example, image <NUM> or image <NUM>), all individual images, and/or a view of the combined image created with processing of data from all images or image data. Settings may be provided to adjust the manner in which characteristics from the individual images are combined to produce an HDR image, and displayed or stored. Settings may also include different settings for different models of scopes that may be attached to a camera head or other imaging device containing image sensor assembly <NUM>.

Image processing circuitry <NUM> is one of three programmable logic devices, processors, or controllers in this embodiment, in addition to a system controller <NUM> and the exposure controller <NUM>. Image processing circuitry <NUM>, controller <NUM>, exposure controller <NUM>, system and program memories <NUM> and <NUM>, video encoder <NUM>, and display controller <NUM> may be housed within CCU <NUM>.

CCU <NUM> may be responsible for powering and controlling light source <NUM>, image sensor assembly <NUM>, and/or optical assembly <NUM>. In some versions, a separate front end camera module may perform some of the image processing functions of image processing circuitry <NUM>.

Although this distribution of imaging device functional control among multiple programmable logic devices, processors, and controllers is typical, these programmable logic devices, processors, or controllers can be combinable in various ways without affecting the functional operation of the imaging device and the application of the invention. These programmable logic devices, processors, or controllers can comprise one or more programmable logic devices, digital signal processor devices, microcontrollers, or other digital logic circuits. Although a combination of such programmable logic devices, processors, or controllers has been described, it should be apparent that one programmable logic device, digital signal processor, microcontroller, or other digital logic circuit can be designated to perform all of the needed functions. All of these variations can perform the same function and fall within the scope of this invention.

<FIG> is a flowchart of a method for producing endoscopy images according to an example embodiment. The method may be performed under control of a processor such as system controller <NUM> (<FIG>), employing any of the various example embodiments of a camera head or joint endoscope and camera head devices as described herein, with a suitable camera control unit such as that described above to process the image data. Other types of medical scopes or borescopes may be used in place of the endoscope as discussed above. The method begins at process block <NUM> where it includes receiving the image light from an endoscope. The endoscope device used may be a separate device attached to a camera head or an endoscope integrated with a camera head. At process block <NUM>, the process directs the received image light along a single optical channel. At block <NUM>, the process alters the polarization properties of the image light with the polarization control element. This may be done in a time varying manner, such as by rotating a rotating polarizer or varying a variable retarder. Block <NUM> may include causing the polarizing optical element to change relative intensity detected by the pixels in each subgroup dynamically based on the content of the received image, such as a camera control unit (CCU) determining that portions of a captured first image, second image or combined image is underexposed or overexposed, and adjusting settings as appropriate in the subsequent frame. In some embodiments, the process includes calculating at least one statistical property for the plurality of images (block <NUM>) such as the intensity value over all pixels or a selected group of representative pixels, and determining which pixels in a certain polarization state have an intensity value below or above a given threshold, and based at least in part on the values of the statistical properties, generate a control signal for the polarizing optical element.

Next at process block <NUM> the process includes focusing the first portion of image light on the polarization image sensor. Note that while the steps are depicted in order, typically block <NUM> is performed on the light as it is focused toward the sensor, because the focusing lenses are preferably optically positioned ahead of the polarization control elements. At block <NUM>, the polarization image sensor creates multiple images based on the light at different polarizations as described above.

Next at process block <NUM>, image processing is performed on the image data from the sensor. The processing is generally applied to image data to generate a single combined image including at least first characteristics resulting from one of the images, and second characteristics resulting from another one of the images. The processing is performed by a CCU such as the CCU <NUM> of <FIG>, or other suitable image processing circuitry. Processing the image data with different intensity ranges provides a high dynamic range (HDR) single combined image with higher dynamic range than either the first or second image taken individually. Many suitable HDR algorithms may be used. Known image processing techniques for polarization studies may be applied in processing the multiple images from such an embodiment, permitting, for example, glare reduction in the combined image.

As used herein the terms "comprising," "including," "carrying," "having" "containing," "involving," and the like are to be understood to be openended, that is, to mean including but not limited to. Any use of ordinal terms such as "first," "second," "third," etc., in the claims to modify a claim element does not by itself connote any priority, precedence, or order of one claim element over another, or the temporal order in which acts of a method are performed. Rather, unless specifically stated otherwise, such ordinal terms are used merely as labels to distinguish one claim element having a certain name from another element having a same name (but for use of the ordinal term).

The foregoing has outlined rather broadly the features and technical advantages of the invention in order that the detailed description of the invention that follows may be better understood. It should be appreciated by those skilled in the art that the conception and specific embodiments disclosed may be readily utilized as a basis for modifying or designing other structures for carrying out the same purposes of the invention. It should also be realized by those skilled in the art that such equivalent constructions do not depart from the scope of the invention as set forth in the appended claims.

Claim 1:
An imaging system (<NUM>) comprising:
an image sensor (<NUM>) including a plurality of pixels arranged as a plurality of subgroups each subgroup containing at least two pixels, at least one pixel in a respective subgroup having an associated polarizer, whereby light detected by at least a first pixel of each subgroup is differently polarized than light detected by a second pixel of a subgroup;
a polarizing optical element (<NUM>) distal to the image sensor (<NUM>) which by its polarizing characteristics is capable of controlling the relative intensity detected by the pixels in each subgroup;
characterized in that the polarizing optical element (<NUM>) comprises a beamsplitter (<NUM>), splitting an image beam into a first beam with a first polarization and a second beam with a second polarization, the first beam directed to the image sensor (<NUM>) including a plurality of pixels of subgroups, and the second beam directed to a second image sensor (<NUM>), and in that
the imaging system (<NUM>) further comprises a processor adapted to receive image data from the image sensor and to create a plurality of images from the received image data, each created from a set of pixels spanning a plurality of subgroups and having the same respective polarizer orientation, the processor further adapted to combine the plurality of image data from the image sensor (<NUM>) and the second image sensor (<NUM>) and generate a high dynamic range (HDR) image therefrom, and
a second polarizing optical element (<NUM>) arranged along an optical path between the beamsplitter (<NUM>) and the image sensor (<NUM>) including a plurality of pixels of subgroups.