Patent Description:
Objective image quality can be a key factor when evaluating overall endoscope performance. Thus, during the development or repair process for an objective of an endoscope, there is a continuing need for the ability to "look through" the objective of an endoscope to see the image provided by the objective. However, the objective is an optical component that can be very hard to evaluate standing alone (i.e., separate or apart from the endoscope) because the image that is formed by an objective is strongly over corrected (i.e., the plane in which the image is sharp is strongly curved) and for the "naked" eye of a user, only a small portion of the image can be in focus at any given time.

Currently, to properly check the objective of an endoscope, the objective has to be mounted in the endoscope and the entire optical system of the endoscope is used during the test image evaluation. The requirement to mount the objective in the endoscope assembly when evaluating the objective can be very cumbersome, expensive and time consuming, especially if the objective is determined to deliver inferior image quality and has to be remanufactured. The need to include the entire optical system of the endoscope during testing results from the endoscope being a compound optical device. Compound optical devices create many intermediate images of the object that are strongly over or under corrected before the final image is provided by the device to the pupil of a viewer's eye or the optical coupler of a camera.

<CIT> is the basis for the preambles of claims <NUM> and <NUM>, and describes a method for calibrating an optical instrument. <CIT> describes a medical image display apparatus and method. <CIT> describes an image processing device. <CIT> describes a measuring system for measuring an imaging quality of an EUV lens. <CIT> describes a method for combining multiple overlapping images and more particularly to a method for image stitching.

The present application generally pertains to evaluating an image delivered by an objective of an endoscope by capturing a series of intermediate images from the objective that have small areas in focus at different locations in space along the optical axis and then digitally stitching the intermediate images together to form the image for evaluation. The field curvature for images created by an objective of an endoscope can be very strong. The strong field curvature of the objective can result in a curved image plane from the objective that can be difficult to evaluate with a relatively simple imaging system.

The invention is defined in independent claims <NUM> and <NUM>.

The shifting of the focus point for the image plane along the optical axis (e. , towards or away from the objective) can result in captured images having different areas in focus. For captured images that are digitally processed, areas of the corresponding images that are in relatively good focus can be identified and extracted or "cut out. " The identification of areas in good focus can be done in many different ways. For example, areas of higher spatial frequencies may be identified as areas of good focus. In another example, sharpness and/or edge detection algorithms may be employed and the intensity maxima from the algorithms may be identified as areas of good focus. Once the area of good focus is identified, the image (or just the area in good focus) can be stored in the memory and the image capture device (e.g., a camera) can be moved to the next location for another area of the image to come into focus. The process of adjusting the focus point can be repeated <NUM>-<NUM> times (or more, if necessary) to get a full image area that is covered by the focused areas or portions.

The present application also pertains to a system and method for evaluating an objective by collecting a series of differently focused images and digitally stitching the images together in order to obtain a final image that corresponds to an image as seen through the objective. The system and method can use image processing and stitching algorithms to generate the final image. The system and method may also include some movable optics and a camera that can deliver a set of images used in later image processing. The collected images can be evaluated in terms of image sharpness and the areas at which each image is in relatively good focus. Once the good focus areas have been found, the areas can be extracted and used to form a final image where most of the target scene (e.g., the formed image of a target) should be in focus, thereby permitting the objective to be evaluated.

One advantage of the present application is that objective image quality analysis can be performed without the need for all of the optical systems of the endoscope to be present.

Another advantage of the present application is that the objective can be quickly evaluated thereby permitting a large volume of endoscopic objectives from large production batches or repair batches to receive quality inspections.

Other features and advantages of the present application will be apparent from the following more detailed description of the identified embodiments, taken in conjunction with the accompanying drawings which show, by way of example, the principles of the application.

Wherever possible, the same reference numbers are used throughout the drawings to refer to the same or like parts.

<FIG> shows an embodiment of an endoscope <NUM>. The endoscope <NUM> can include a body encasing optical elements such as an objective <NUM>, a relay system <NUM> and an ocular system <NUM>. The objective <NUM> can be used to provide an image (e.g., reflected light) of a target (e.g., the object to be viewed) to the relay system <NUM>. The objective <NUM> may include one or more lenses and/or other optical components. The relay system <NUM> can be used to transfer the image from the objective <NUM> to the ocular system <NUM>. The relay system <NUM> can include one or more lenses and/or optical fibers to transfer the image from the objective <NUM>. The ocular system <NUM> permits a user of the endoscope <NUM> to view the image from the objective <NUM>. The ocular system <NUM> can include an eyepiece for direct viewing of the image and/or a camera system to capture and display the image. In some embodiments, the endoscope <NUM> may include one or more devices to transfer light from a light source to the target to illuminate the target.

<FIG> and <FIG> show an embodiment of the objective <NUM> from the endoscope <NUM> of <FIG>. Light rays <NUM> passing through the objective <NUM> form a curved focal plane P in the image space. In the embodiment shown in <FIG> and <FIG>, four bundles of light rays <NUM> are shown passing through the objective <NUM>, but more or less than four bundles of light rays may pass through the objective <NUM> in other embodiments. The curved focal plane P can be dissected by corresponding image planes P1-P3 over a distance L extending along the optical axis for the objective <NUM>. While <NUM> image planes P1-P3 are shown in <FIG> and <FIG>, the focal plane P can be dissected by any number of image planes along the distance L.

For each image plane P1-P3, a portion of the image (from the object space) captured at the corresponding image plane can be in focus, while other portions of the image remain out of focus. <FIG> shows an embodiment of an image captured at image plane P2. As can be seen in <FIG>, the portion of the image corresponding to image plane P2 is in focus, while the portions of the image corresponding to image planes P1 and P3 are out of focus.

<FIG> show embodiments of an evaluation system used to evaluate objectives <NUM> from endoscopes <NUM>. The evaluation system <NUM> can include a target <NUM>, the objective <NUM> being evaluated or tested, an image capture system <NUM> and an image processing system <NUM>. The image capture system <NUM> can include one or more optical components <NUM> (e.g., lenses, prisms, etc.) and/or a camera <NUM>. In an embodiment, the camera <NUM> may be replaced by an image sensor (or sensor array) that can be used to capture images. The image sensor can be controlled and operated by the image processing system <NUM> in an embodiment.

In one embodiment, the target <NUM>, the objective <NUM> being evaluated or tested and the image capture system <NUM> can be placed in a fixture or enclosure <NUM> that prevents or limits ambient light from reaching the camera <NUM>. The target <NUM> can include features that allow in focus areas of captured images to be easily extracted. The target <NUM> can have a pattern that is uniform and of high contrast across the whole field of view. In one embodiment, the pattern for the target <NUM> can include a set of slanted (e.g., at a <NUM>° angle) alternating black and white stripes with a width of about <NUM> (e.g., a "zebra" pattern). The use of a "zebra" pattern can be beneficial in image post processing and can be relatively easily reconstructed even on very blurry images. In an embodiment, the target <NUM> can be back illuminated to prevent parasitic reflections and the illumination level of the target <NUM> can be adjustable to obtain a desired image contrast from the camera <NUM>.

The optical components <NUM> (sometimes referred to as pickup optics) includes movable optical elements that permit the objective image (i.e., the image formed by the objective <NUM>) to be relayed onto the camera <NUM> from multiple image positions (e.g., image planes P1-P3) shifted along the longitudinal axis of the objective <NUM>. The camera <NUM> can capture multiple intermediate images of the objective image from multiple image positions and provide the captured images to the image processing system <NUM>. The image processing system <NUM> can evaluate the intermediate images from the camera <NUM> with image processing algorithms to extract image areas that are in focus. Once the intermediate images with corresponding in focus areas are extracted, the image processing system <NUM> can produce the final image used to evaluate the objective <NUM> by stitching together the extracted in focus areas from the intermediate images.

<FIG> shows an embodiment of a camera <NUM> that can be used with the evaluation system <NUM>. The camera <NUM> shown in <FIG> can include logic <NUM>, referred to herein as "camera logic," which may be implemented in software, firmware, hardware, or any combination thereof. In <FIG>, the camera logic <NUM> is implemented in software and stored in memory <NUM>. However, other configurations of the camera logic <NUM> are possible in other embodiments. The camera logic <NUM>, when implemented in software, can be stored and transported on any computer-readable medium for use by or in connection with an instruction execution apparatus that can fetch and execute instructions.

The embodiment of the camera <NUM> shown in <FIG> can include at least one conventional processor <NUM>, which can incorporate processing hardware for executing instructions stored in the memory <NUM>. As an example, the processor <NUM> may include a central processing unit (CPU), a digital signal processor (DSP), and/or a graphic processing unit (GPU). The processor <NUM> can communicate to and drive the other elements within the camera <NUM> via a local interface <NUM>, which can include at least one bus.

As shown by <FIG>, the camera <NUM> can also include an image sensor <NUM> and a communication module <NUM>. The image sensor <NUM> can be used to record, capture or obtain images from the objective <NUM> in the area surrounding or in proximity to the camera <NUM> (e.g., the image space). The communication module <NUM> can include a radio frequency (RF) radio or other device for communicating wirelessly with image processing system <NUM>. In another embodiment, the communication module <NUM> may also include an interface permitting wired communication between the camera <NUM> and the image processing system <NUM>.

The image sensor <NUM> can include one or more CCDs (charge coupled devices) and/or one or more active pixel sensors or CMOS (complementary metal-oxide-semiconductor) sensors. The images from the image sensor <NUM> can be stored as image data <NUM> in memory <NUM>. The image data <NUM> can be stored in any appropriate file format, including, but not limited to, PNG (portable network graphics), JPEG (joint photographic experts group), TIFF (tagged image file format), MPEG (moving picture experts group), WMV (Windows media video), QuickTime and GIF (graphics interchange format).

From time-to-time, the camera logic <NUM> can be configured to transmit the image data <NUM> to the image processing system <NUM>. The image data <NUM> may be analyzed by the image processing system <NUM> to determine if the objective <NUM> is acceptable for use in an endoscope <NUM>. The image data <NUM> may be time-stamped based on the time indicated by a clock (not shown) in order to indicate when the image data <NUM> was obtained.

<FIG> shows an embodiment of the image processing system <NUM>. The image processing system <NUM> may be implemented as one or more general or special-purpose computers, such as a laptop, hand-held (e.g., smartphone), desktop, or mainframe computer. The image processing system <NUM> can include logic <NUM>, referred to herein as "device logic," for generally controlling the operation of the image processing system <NUM>. The image processing system <NUM> also includes image processing logic <NUM> to determine the portions or areas of the images from the camera <NUM> that are in focus and image stitching control logic <NUM> to form an evaluation image (or final image) from the portions of the images determined to be in focus by the image processing logic <NUM>. The image processing system <NUM> further includes evaluation logic <NUM> for processing and analyzing the evaluation image to determine the acceptability of the objective <NUM>. The device logic <NUM>, the image processing logic <NUM>, the image stitching control logic <NUM> and the evaluation logic <NUM> can be implemented in software, hardware, firmware or any combination thereof. In the image processing system <NUM> shown in <FIG>, the device logic <NUM>, the image processing logic <NUM>, the image stitching control logic <NUM> and the evaluation logic <NUM> are implemented in software and stored in memory <NUM> of the image processing system <NUM>. Note that the device logic <NUM>, the image processing logic <NUM>, the image stitching control logic <NUM> and the evaluation logic <NUM>, when implemented in software, can be stored and transported on any non-transitory computer-readable medium for use by or in connection with an instruction execution apparatus that can fetch and execute instructions.

The image processing system <NUM> can include at least one conventional processor <NUM>, which has processing hardware for executing instructions stored in memory <NUM>. As an example, the processor <NUM> may include a central processing unit (CPU), a digital signal processor (DSP), and/or a graphic processing unit (GPU). The processor <NUM> communicates to and drives the other elements within the image processing system <NUM> via a local interface <NUM>, which can include at least one bus. Furthermore, an input interface <NUM>, for example, a keypad, keyboard or a mouse, can be used to input data from a user of the image processing system <NUM>, and an output interface <NUM>, for example, a printer, monitor, liquid crystal display (LCD), or other display apparatus, can be used to output data to the user. Further, a communication interface <NUM> may be used to exchange data with the camera <NUM>.

As shown by <FIG>, camera data <NUM> and evaluation data <NUM> and can be stored in memory <NUM> at the image processing system <NUM>. The camera data <NUM> can include image data <NUM> from the camera <NUM>. The evaluation data <NUM> can include evaluation images generated by the image stitching logic <NUM> and/or information relating to the in focus areas of the images from the camera <NUM> determined by image processing logic <NUM>. The camera data <NUM> and the evaluation data <NUM> can be used and/or analyzed by device logic <NUM>, the image processing logic <NUM>, the image stitching control logic <NUM> and the evaluation logic <NUM> to determine the acceptability of the objective <NUM> that generated the images.

<FIG> is a flow chart showing an embodiment of a process for evaluating an objective <NUM> of an endoscope <NUM> with evaluation system <NUM>. The objective <NUM> that is undergoing an evaluation can be positioned in the enclosure <NUM> of the system <NUM> with the target <NUM> such that the objective image formed by the objective <NUM> incorporates the target <NUM>. The process of <FIG> can be used to evaluate objectives <NUM> as part of either a manufacturing process or a repair process for endoscopes <NUM>.

Once the objective is positioned in the enclosure <NUM>, the image capture system <NUM> can be positioned along the optical axis in the enclosure <NUM> (step <NUM>) and an intermediate image can be captured (step <NUM>). The positioning of the image capture system <NUM> in the enclosure can be used to obtain intermediate images where a particular portion or area of the intermediate image is in focus and the remaining portions of the intermediate image are out of focus. For each position of the image capture system <NUM> along the optical axis, the intermediate image can be captured. In an embodiment, the target illumination level may need to be adjusted for some of the axial positions of the image capture system <NUM> to achieve the best image contrast in the intermediate image. For example, intermediate images captured from an image plane at the center of the objective image can be much brighter in the center, while the intermediate images captured from an image plane that is in focus at the periphery of the objective image are darker due to natural vignetting within tested objective. Next, a determination can be made as to whether additional intermediate images need to be collected (step <NUM>). If additional images are to be collected, the process returns to step <NUM> to position the image capture system <NUM> into another position. In one embodiment, a predetermined number of intermediate images of the objective image can be obtained from different positions along the focal plane of the objective <NUM> by adjusting the position of the image capture system <NUM>. In another embodiment, the position of the objective <NUM> can be moved to obtain the predetermined number of intermediate images. Each of the intermediate images can have a different area or portion of the objective image in focus depending on where the intermediate image is captured (by the image capture system <NUM>) with respect to the focal plane of the objective image.

In one embodiment, the image capture system <NUM> can include optical components <NUM> and camera <NUM> that are moveable within fixture <NUM>. The moveable optical components <NUM> and camera <NUM> can be fixed with respect to each other to maintain the image plane (sometimes referred to as the pick-up plane) in the best possible focus on the image sensor <NUM> of the camera <NUM>. The optical components <NUM> and camera <NUM> can be mounted on a common assembly and move in tandem such that a substantially constant focal distance between the optical components <NUM> and the camera <NUM> is maintained and a substantially constant focal distance between the optical components <NUM> and image plane of the objective image being captured is maintained. The moveable optical components <NUM> and camera <NUM> can be refocused along the optical axis at different distances from the objective <NUM> to permit multiple intermediate images of the objective image to be captured. As shown in <FIG>, the optical components <NUM> and camera <NUM> can be positioned in different positions to capture intermediate images corresponding to image planes P1-P3. In <FIG>, the optical components <NUM> and camera <NUM> can be positioned to capture intermediate image #<NUM>. In intermediate image #<NUM>, the portion of the objective image corresponding to image plane P2 is in focus and the remaining portions of the objective image (e.g., the portions corresponding to image planes P1 and P3) are out of focus. In <FIG>, the optical components <NUM> and camera <NUM> can be positioned to capture intermediate image #<NUM>. In intermediate image #<NUM>, the portion of the objective image corresponding to image plane P1 is in focus and the remaining portions of the objective image (e.g., the portions corresponding to image planes P2 and P3) are out of focus. In <FIG>, the optical components <NUM> and camera <NUM> can be positioned to capture intermediate image #<NUM>. In intermediate image #<NUM>, the portion of the objective image corresponding to image plane P3 is in focus and the remaining portions of the objective image (e.g., the portions corresponding to image planes P1 and P2) are out of focus.

In another embodiment, the image capture system <NUM> may omit moveable optical components <NUM> (or have the optical components <NUM> in a fixed position) and just move camera <NUM>. The moveable camera <NUM> can moved into different positions such that the image plane of the objective image being captured is in focus. As shown in <FIG>, the camera <NUM> can be positioned in different positions to capture intermediate images corresponding to image planes P1-P3. In <FIG>, the camera <NUM> can be positioned to capture intermediate image #<NUM>. In intermediate image #<NUM>, the portion of the objective image corresponding to image plane P2 is in focus and the remaining portions of the objective image (e.g., the portions corresponding to image planes P1 and P3) are out of focus. In <FIG>, the camera <NUM> can be positioned to capture intermediate image #<NUM>. In intermediate image #<NUM>, the portion of the objective image corresponding to image plane P1 is in focus and the remaining portions of the objective image (e.g., the portions corresponding to image planes P2 and P3) are out of focus. In <FIG>, the camera <NUM> can be positioned to capture intermediate image #<NUM>. In intermediate image #<NUM>, the portion of the objective image corresponding to image plane P3 is in focus and the remaining portions of the objective image (e.g., the portions corresponding to image planes P1 and P2) are out of focus.

Returning back to <FIG>, once all of the intermediate images have been collected, each of the intermediate images can be processed with the image processing system <NUM> to extract a portion from each captured image (step <NUM>). In an embodiment, when the intermediate images from respective image planes are collected (e.g., image planes P1-P3), the intermediate images can be evaluated in terms of image sharpness regions. While only three image planes (e.g., P1-P3) and three corresponding intermediate images are shown in the embodiments of <FIG> and <FIG>, additional image planes (e.g., <NUM>-<NUM> image planes and intermediate images) can be used in other embodiments. The number of image planes and intermediate images that may be required to generate a final image can depend on the strength of the correction for the objective <NUM> and the strength of the curvature for the focal plane of the objective image.

In one embodiment, the extracted portion of the intermediate image can correspond to the portion of the intermediate image that is in focus. Sharp or in focus portions of the intermediate image may be found using different techniques. For example, techniques involving edge and/or sharpness detection algorithms, local spatial frequency estimators, local gradients, etc. may be used. In one embodiment, Sobel matrix operators for local gradients was used. In another embodiment, the extracted portion of the intermediate image can correspond to a predetermined portion of the objective image that is related to the image plane being captured by the intermediate image.

The extracted portions of the intermediate images can be stitched (or assembled) together by the image processing system to form a final image (step <NUM>). In one embodiment, if there are areas of the final image are not covered by any extracted portion from the intermediate images, additional intermediate images may be collected to guarantee full coverage of the final image. The final focused image can be used for the evaluation of the objective <NUM> being tested and can be used to find any imperfections in optical subcomponents or the assembly of the objective <NUM>. The final image can correspond to the objective image provided by the objective <NUM>. However, the final image can differ from the image that is formed by a fully assembled endoscope <NUM>, when the rest of the endoscope optics are present. The slight difference in images can be due to a fact that some optical aberrations of the objective <NUM> may be corrected in the remaining part of the relay system <NUM> and the ocular system <NUM>. The image processing system <NUM> can then review the final image (step <NUM>) to determine the acceptability of the final image. If the acceptability of the final image can directly correspond to the acceptability of the objective <NUM> being evaluated. If the final image is acceptable, then the objective <NUM> is also acceptable. However, if the final image is not acceptable (or is rejected), then the objective <NUM> is also not acceptable and rejected.

Although the figures herein may show a specific order of method steps, the order of the steps may differ from what is depicted. Also, two or more steps may be performed concurrently or with partial concurrence. Variations in step performance can depend on the software and hardware systems chosen and on designer choice. All such variations are within the scope of the application. Software implementations could be accomplished with standard programming techniques, with rule based logic and other logic to accomplish the various connection steps, processing steps, comparison steps and decision steps.

Claim 1:
A system (<NUM>) to evaluate an objective (<NUM>) of an endoscope (<NUM>), the system (<NUM>) comprising:
an objective (<NUM>) from an endoscope (<NUM>), the objective (<NUM>) having an optical axis;
an image capture system (<NUM>) configured to capture a plurality of intermediate images of a formed image from the objective (<NUM>), the image capture system (<NUM>) being movable along the optical axis to capture the plurality of intermediate images; and characterized in
an image processing system (<NUM>) communicatively coupled to the image capture system (<NUM>) to receive the plurality of intermediate images from the image capture system (<NUM>), the image processing system (<NUM>) configured to extract at least one portion from each intermediate image of the plurality of intermediate images, the image processing system (<NUM>) configured to assemble the extracted portions from the plurality of intermediate image to form a final image, wherein the final image corresponds to the formed image from the objective (<NUM>), wherein the at least one extracted portion from the intermediate image corresponds to an in focus portion of the intermediate image, and wherein an evaluation of the final image by the image processing system (<NUM>) determines acceptability of the objective (<NUM>).