Source: http://www.google.com/patents/US5956416?dq=5,884,272
Timestamp: 2017-07-26 07:37:33
Document Index: 433464060

Matched Legal Cases: ['art 9', 'art 9', 'art 2', 'art 3', 'art 9', 'art 108', 'art 108', 'art 108', 'art 401', 'art 401', 'art 401', 'art 108', 'art 301', 'art 301', 'art 301', 'art 108', 'art 312', 'art 522']

Patent US5956416 - Endoscope image processing apparatus - Google PatentsSearch Images Maps Play YouTube News Gmail Drive More »Sign inPatentsThe endoscope image processing apparatus comprises a processing apparatus for image-processing as predetermined at least one image signal and a discriminating apparatus for discriminating regions ineffective to image-processing by the processing apparatus or to the result of the image-processing from...http://www.google.com/patents/US5956416?utm_source=gb-gplus-sharePatent US5956416 - Endoscope image processing apparatusAdvanced Patent SearchTry the new Google Patents, with machine-classified Google Scholar results, and Japanese and South Korean patents.Publication numberUS5956416 APublication typeGrantApplication numberUS 08/525,098Publication dateSep 21, 1999Filing dateSep 8, 1995Priority dateMar 23, 1989Fee statusLapsedAlso published asUS5515449Publication number08525098, 525098, US 5956416 A, US 5956416A, US-A-5956416, US5956416 A, US5956416AInventorsTakao Tsuruoka, Kazunari NakamuraOriginal AssigneeOlympus Optical Co., Ltd.Export CitationBiBTeX, EndNote, RefManPatent Citations (6), Referenced by (68), Classifications (23), Legal Events (5) External Links: USPTO, USPTO Assignment, EspacenetEndoscope image processing apparatus
US 5956416 AAbstract
The endoscope image processing apparatus comprises a processing apparatus for image-processing as predetermined at least one image signal and a discriminating apparatus for discriminating regions ineffective to image-processing by the processing apparatus or to the result of the image-processing from other effective regions. The image processing apparatus further comprises an outputting apparatus for outputting the results of processing images by the processing apparatus only on the effective regions discriminated by the discriminating apparatus and an image forming apparatus forming images based on the results of processing images by the processing apparatus only on the effective regions.
1. An endoscope image processing apparatus, comprising:a processing means for inputting at least picture image signals of a picture plane from an endoscope unit to perform predetermined picture image processing for the input picture image signals and accordingly obtaining output picture image signals; a discriminating means, operably coupled to said processing means, for discriminating regions having ineffective input picture image signals unsuitable for being processed by said processing means from effective functional information input picture image signals suitable for being processed by said processing means within a whole picture plane of said input picture image signals, wherein said ineffective input picture image signals represent images having overexposure levels which exceed an allowable maximum exposure level; and a displaying means, operably coupled to said processing means, for displaying a picture image processed by said processing means, and operably coupled to said discriminating means, for displaying said discriminated regions as ineffective regions based on a signal specified for displaying said regions having ineffective input picture image signals, said picture image processed by said processing means and said discriminated regions being simultaneously displayed to thereby distinguish said discriminated regions as ineffective regions from said picture image processed by said processing means. 2. An endoscope image processing apparatus according to claim 1, wherein said discriminating means discriminates regions exceeding at least a predetermined maximum level of a luminance level of said input picture image signals.
3. An endoscope image processing apparatus according to claim 1, wherein said discriminating means discriminates a region less than at least a predetermined minimum level of a luminance level of said input picture image signals.
4. An endoscope image processing apparatus according to claim 1, wherein said processing means performs process of only effective regions other than regions discriminated by said discriminating means.
5. An endoscope image processing apparatus according to claim 1, wherein said processing means performs whole predetermined picture image process on input picture image signals of a picture plane being inputted and outputs output picture image signals of a picture plane corresponding to the input picture image signals.
6. An endoscope image processing apparatus according to claim 1, further comprising an output means for outputting a result of picture image process by said processing means only for effective regions discriminated from regions determined to be said unsuitable regions by said discriminating means.
7. An endoscope image processing apparatus according to claim 1, further comprising a picture image forming means for forming a picture image based on a result of the picture image process by said image processing means only for effective regions discriminated from said unsuitable regions by said discriminating means.
8. An endoscope image processing apparatus according to claim 7, wherein said discriminating means discriminates between ineffective regions and said effective regions before processing picture images by said processing means.
9. An endoscope image processing apparatus according to claim 7, wherein said picture image forming means includes ineffective regions picture image forming means for forming a predetermined picture image showing ineffective regions determined to be said unsuitable regions by said discriminating means.
10. An endoscope image processing apparatus according to claim 9, wherein said ineffective regions picture image forming means forms said predetermined picture image by replacing the result of the picture image process by said processing means with a predetermined picture image data on the ineffective regions determined to be said unsuitable regions by said discriminating means.
11. An endoscope image processing apparatus according to claim 9, wherein said ineffective regions picture image forming means forms said predetermined picture image by replacing the result of picture image process by said processing means with a picture image data of a result performing different process from the process of said processing means for original input picture image signals on the ineffective regions determined to be said unsuitable regions by said discriminating means.
12. An endoscope image processing apparatus according to claim 9, wherein said ineffective regions picture image forming means forms said predetermined picture image by replacing the result of picture image process by said processing means with original input picture image signals on the ineffective regions determined to be said unsuitable regions by said discriminating means.
13. An endoscope image processing apparatus according to claim 9, wherein said ineffective regions picture image forming means differently forms said predetermined picture images depending on a cause of the ineffective regions determined to be said unsuitable regions by said discriminating means.
14. An endoscope image processing apparatus according to claim 1, wherein said discriminating means makes said discriminating for input picture images before processing picture images by said processing means.
15. An endoscope image processing apparatus according to claim 1, wherein said discriminating means includes an operating means for executing an operation using a plurality of picture image signals corresponding to a same subject.
16. An endoscope image processing apparatus according to claim 15, wherein two picture image signals among said plurality of picture image signals correspond to images of two wavelength ranges where a difference of absorbance in two wavelength ranges varies by variation of a degree of saturation with oxygen of hemoglobin.
17. An endoscope image processing apparatus according to claim 15, wherein two picture image signals among said plurality of picture image signals correspond to images of two wavelength ranges where a difference of absorbance in two wavelength ranges varies by variation in quantity of hemoglobin.
18. An endoscope image processing apparatus according to claim 1, wherein said processing means includes means for executing a plurality of operations.
19. An endoscope image processing apparatus according to claim 18, wherein said discriminating means makes said discriminating based on at least one of output signals of said means for executing said plurality of operations.
20. An endoscope image processing apparatus according to claim 18, wherein said processing means includes a logarithmic-pressing means for logarithmic-pressing at least two of said input picture image signals.
21. An endoscope image processing apparatus according to claim 20, wherein said processing means includes a difference operation means for calculating a difference between signals pressed by said logarithmic-pressing means.
22. An endoscope image processing apparatus according to claim 21, wherein said discriminating means makes said discriminating based on one of output values of said logarithmic-pressing means and said operation means.
23. An endoscope image processing apparatus according to claim 18, wherein said displaying means displays a picture image in color processed by said processing means and said discriminated regions in achromatic color.
24. An endoscope image processing apparatus according to claim 1, wherein said displaying means displays said discriminated regions in a net pattern.
25. An endoscope image processing apparatus according to claim 1, wherein said processing means inputs R, G and B picture image signals as input picture image signals and includes a calculating means for calculating L* a* b* in L* a* b* coordinate systems from the input signals.
26. An endoscope image processing apparatus according to claim 25, wherein said processing means includes a means for executing color enhancement process based on said calculated L* a* b*.
27. An endoscope image processing apparatus according to claim 26, wherein said discriminating means performs discriminating process based on the L* calculated by said processing means.
28. An endoscope image processing apparatus according to claim 27, wherein said processing means performs said process only for regions except ineffective regions discriminated by said discriminating means.
29. An endoscope image processing apparatus according to claim 1, wherein said processing means performs differential process for the input picture image signals and includes a means for extracting an edge.
30. An endoscope image processing apparatus according to claim 1, wherein said discriminating means judges regions having inferior illumination to be said unsuitable regions.
31. An endoscope image processing apparatus according to claim 1, wherein said processing means includes a means for executing an operation using a plurality of picture image signals corresponding to different time of a same subject.
32. An endoscope image processing apparatus according to claim 31, wherein said discriminating means judges unreliable-accuracy regions caused by deviation of a location of said same subject between said plurality of picture image signals to be unsuitable regions.
33. An endoscope apparatus, comprising:a light illuminating means for supplying light of different wavelength ranges to a body cavity sequentially; an endoscope unit outputting endoscope picture images of respective wavelength ranges illuminated by said light illuminating means; a processing means for inputting endoscope picture image signals from said endoscope unit to perform predetermined picture image processing for the input picture image signals and accordingly obtaining output picture image signals; a discriminating means, operably coupled to said processing means, for discriminating regions having ineffective input picture image signals unsuitable for being processed by said processing means from effective functional information input picture image signals suitable for being processed by said processing means within a whole picture plane of said input picture image signal, wherein said ineffective input picture image signals represent images having over-exposure levels which exceed an allowable maximum exposure level; and a displaying means, operably coupled to said processing means, for displaying a picture image processed by said processing means, and operably coupled to said discriminating means, for displaying said discriminated regions as ineffective regions based on a signal specified for displaying said regions having ineffective input picture image signals, said picture image processed by said processing means and said discriminated regions being simultaneously displayed to thereby distinguish said discriminated regions as ineffective regions from said picture image processed by said processing means. 34. An endoscope apparatus according to claim 33, wherein said light illuminating means includes a light source and a rotary filter having a plurality of filters letting light of different wavelength ranges pass and disposing the filters in front of said light source sequentially.
35. An endoscope apparatus according to claim 34, wherein said rotary filter includes filters letting light at least in a vicinity of 580 nm wavelength range and in a vicinity of 800 nm wavelength range pass.
36. An endoscope apparatus according to claim 35, wherein said processing means outputs output picture images showing a degree of saturation with oxygen of hemoglobin based on input picture images for at least said two wavelength ranges.
37. An endoscope apparatus according to claim 34, wherein said rotary filter includes three filters letting at least red, green and blue light pass, respectively.
38. An endoscope apparatus according to claim 37, wherein said processing means includes a calculating means for calculating L* a* b* of R, G and B signals for said red, green and blue illuminating light in L* a* b* coordinate systems.
39. An endoscope apparatus according to claim 38, wherein said processing means includes a means for performing color enhancement process based on said calculated L* a* b*.
40. An endoscope apparatus according to claim 38, wherein said discriminating means performs discriminating process based on the L* calculated by said processing means.
41. An endoscope apparatus according to claim 33, wherein said endoscope unit contains a CCD transforming a subject image into electric signals, and said endoscope apparatus stores endoscope picture images in respective wavelength ranges from said CCD between said endoscope unit and processing means and has a plurality of memory means for outputting endoscope picture images in the respective wavelength ranges to said processing means.
42. An endoscope picture image processing apparatus, comprising:a processing means for performing predetermined picture image processing for at least one original endoscope picture image; a judging means for judging ineffective regions from effective functional information image regions suitable for being processed by said processing means on results of performing a process for said original endoscope picture image and picture image process to be attributes; and a displaying means, operably coupled to said processing means and said judging means, for displaying a result processed by said processing means with image regions determined ineffective by said judging means as ineffective regions, based on a signal specified for displaying said ineffective regions to thereby distinguish said ineffective regions determined ineffective from said result processed by said processing means, wherein said image regions determined ineffective have over-exposure levels which exceed an allowable maximum exposure level. 43. An endoscope picture image processing apparatus according to claim 42, further comprising a mask signal generating circuit, the mask signal generating circuit generating mask signals in regions determined ineffective by said judging means and performing masking in said regions displayed by said displaying means.
44. An endoscope picture image processing apparatus according to claim 43, wherein said processing means inputs a plurality of picture images taken in by different timing for a same subject to process the images and said judging means judges regions having deviation between said plurality of picture images.
45. A living body function picture image displaying apparatus, comprising:an illuminating means for supplying light of at least two wavelength ranges to a living body; an image forming means for transforming a subject image illuminated by the illuminating means into at least a first and second electric image signals in said wavelength ranges, respectively; an operating means for executing operation based on said first and second electric image signals and obtaining functional information picture images showing living body function information on said living body; a picking up means, operably coupled to said operating means, for picking up regions of ineffective results unsuitable for being processed by said operating means, wherein said ineffective results are images having over-exposure levels which exceed an allowable maximum exposure level; and a displaying means, operably coupled to said operating means, for displaying the image obtained by said operating means, and operably coupled to said picking up means, for displaying the image obtained by said picking up means as an ineffective picture image based on a signal specified for displaying said regions of ineffective results to thereby distinguish the ineffective picture image obtained by the picking up means from the functional information images obtained by said operating means. 46. A living body function picture image displaying apparatus according to claim 45, wherein said living body function information is blood information within a living body and said operating means includes a calculating means for calculating a difference between said first and second electric image signals.
47. A living body function picture image displaying apparatus according to claim 46, wherein said operating means executes said difference calculation after applying logarithms to said first and second electric image signals.
48. A living body function picture image displaying apparatus according to claim 46, wherein said picking up means includes a means for finding luminance of said subject and picks up one of regions where the luminance is too high and too low.
49. An endoscope picture image displaying apparatus, comprising:an endoscope unit generating electric signals denoting a subject picture image; a processing means for performing specific functional information picture image process based on said electric signals; a detecting means for detecting at least high luminance regions having halation within said subject picture image unsuitable for being processed by said processing means; and a displaying means, operably coupled to said processing means and said detecting means, for displaying an output picture image processed by said processing means with said high luminance regions as ineffective picture images, based on a signal specified for displaying said high luminance regions to thereby distinguish said high luminance regions as the ineffective picture images from said output functional information picture image processed by said processing means. 50. An endoscope picture image displaying apparatus according to claim 49, wherein said processing means processes to find a degree of saturation with oxygen of hemoglobin.
51. An endoscope picture image displaying apparatus according to claim 50, wherein said endoscope unit generates two electric signals denoting two subject images on at least two wavelength ranges and said processing means includes a means for acquiring at least a difference between said two electric signals.
This is a continuation of application Ser. No. 07/842,769 filed Mar. 2, 1992 now U.S. Pat. No. 5,515,449 which is a continuation of application Ser. No. 07/440,620 filed Nov. 22, 1989, now abandoned.
This invention relates to image processing apparatus for processing predetermined images as more particularly to an endoscope image processing apparatus for processing images obtained by an endoscope inserted into a body cavity to non-invasively diagnose affected parts.
An example of a conventional endoscope apparatus is shown in FIG. 26. As shown in this drawing, a light emitted from a lamp 31 is time-serially separated into the respective wavelength regions of R (red), G (green) and B (blue) by a rotary filter 33 having filters 33R, 33G and 33B transmitting the light of the respective wavelength regions of red (R), green (G) and blue (B) and rotated by a motor 33 and is emitted into an endoscope light guide 23 at the entrance end. This frame sequential illuminating light is led to the endoscope tip part by the above mentioned light guide 23, is emitted from this tip part and is radiated onto an object to be imaged. The returning light from the object by this illuminating light is made to form an image on a CCD 41 provided in the endoscope tip part 9 by an image forming optical system 22. An image signal from this CCD 41 is amplified to be on a voltage level in a predetermined range by an amplifier 42. The output of this amplifier 42 has γ corrected by a γ-correcting circuit 43, is then converted into a digital signal by an A/D converter 44 and is stored in respective memories 46R, 46G and 46B through a switching switch 45. The image signals stored in the respective memories are read out by the timing of television signals and are converted into analog signals respectively by D/A converters 47R, 47G and 47B. These analog image signals together with a synchronizing signal SYNC from a synchronizing signal generating circuit 52 are transmitted to RGB signal output ends 49R, 49G and 49B. The thus obtained RGB signals are displayed on a monitor to make an endoscope observation. The above mentioned synchronizing signal is output from a synchronizing signal output end 49S and is input together with the RGB signals into the monitor.
Recently, various image processes are made for such an endoscope apparatus. As examples of such an image process, there are a) a coloration enhancing process whereby three RGB signals are converted to be in uniform color spaces of the brightness, chroma and hue and are processed to be enhanced as is shown in the publication of Japanese Patent Application Laid Open No. 173182/1988 and b) an operating process whereby a part of three RGB signals is changed to be in an infrared region and the oxygen saturated degree in the living body tissue is determined by an operation between pixels as is mentioned in U.S. Pat. No. 4,878,113.
FIG. 4 is an explanatory view for explaining an image process by this embodiment.
FIG. 7 is of explanatory views showing the results of the coloration enhancing process.
FIG. 13 is an explanatory view showing an image by a conventional edge extracting process and an output image by this embodiment.
As shown in FIG. 3, a light distributing lens 21 and an image forming optical system 22 are arranged in the above mentioned tip part 9. A light guide 117, made of a fiber bundle, is provided on the rear end side. The light guide 117 is inserted through the above mentioned insertable part 2, operating part 3 and universal cord 4 and is connected to the above mentioned connector 5 which is to be connected to the above mentioned observing apparatus 6 so that the illuminating light emitted from the light source apparatus within the observing apparatus 6 may enter the above mentioned light guide 117 at the entrance end. This light source apparatus is provided with a lamp 118 and a rotary filter 116 arranged in the illuminating light path and rotated by a motor 115. In this embodiment, the above mentioned lamp 118 is to emit ultraviolet to infrared rays. Filters 116R, 116G and 116B, transmitting, light of respective wavelength regions different from one another, are arranged in the peripheral direction in the above mentioned rotary filter 116. In this embodiment, the filter 116R transmits a red color light near 650 nm, the filter 116G transmits a green color light near 580 nm and the filter 116B transmits an infrared light near 800 nm. The light emitted from the above mentioned lamp 118 is time-serially separated by the above mentioned rotary filter 116 into the respective wavelength regions and enters the above mentioned light guide 117 at the entrance end. This illuminating light is led to the tip part 9 by the above mentioned light guide 117, is emitted from the tip surface and is radiated to the object through the light distributing lens 21.
A solid state imaging device as, for example, a CCD 101 is arranged in the image forming position of the above mentioned image forming optical system 22 so that the object image illuminated by the above mentioned frame sequential illuminating light may be formed by the above mentioned image forming optical system 22 and may be converted to an electric signal by the above mentioned CCD 101. The image signal from this CCD 101 is input into an amplifier 102 so as to be amplified into an electric signal in a predetermined range (for example, of 0 to 1 volt). The output electric signal of this amplifier 102 has γ corrected by a γ-correcting circuit 103, is then converted to a digital signal by an A/D converter 104 and is input into a selector 105 having one input and three outputs. The time-serially transmitted RGB signals are separated by this selector 105 into respective R, G and B color signals which are stored in respective memories 106R, 106G and 106B corresponding to R, G and B. The image signals read out of the respective memories are converted into analog signals respectively by D/A converters 107R, 107G and 107B and are output from respective R, G and B signal output ends 109, 110 and 111 through an image processing part 108. Together with the above mentioned R, G and B signals, a synchronizing signal SYNC from a synchronizing signal generating circuit 113 is output from a synchronizing signal output end 114. The above mentioned R, G and B signal and synchronizing signal are input into the monitor 7 and various image processing apparatuses.
An ineffective region detecting circuit 210 for detecting the overflow and underflow in each operating process is provided and is connected to the logarithmic amplifiers 204R, 204G and 204B, differential amplifiers 205 and 206 and divider 207. The output of the above mentioned ineffective region detecting circuit is input into an abnormal part data ROM 211 in which predetermined abnormal part data are to be stored. The output of this abnormal part data ROM 211 is input into the above mentioned selector 208 at the other input end. The above mentioned selector 208 is to have the input switched by the control signal from the above mentioned ineffective region detecting circuit 210. That is to say, in an ordinary case, a signal, from the divider 207 will be detected. In case a signal showing that an ineffective region is detected is output from the ineffective region detecting circuit 210, a signal from the abnormal part data ROM 211 will be selected.
An ultraviolet to infrared light emitted from the lamp 118 enters the rotary filter 116 rotated by the motor 115. As described above, this rotary filter 116 has the filter 116R transmitting a red color light near 650 nm, filter 116G transmitting a green color light near 580 nm and filter 116B transmitting an infrared light near 800 nm. Therefore, the light from the above mentioned lamp 118 is time-serially separated into lights of wavelengths corresponding to the above mentioned respective filters 116R, 116G and 116B which are led into a body cavity through the light guide 117 and are radiated as illuminating lights into a body cavity through the light distributing lens 21. The object image by the respective illuminating light is formed on the CCD 101 by the image forming optical system 22 and is converted into an electric signal. The output signal of this CCD 101 is amplified by the amplifier 102 and is converted by the γ correcting circuit 103 to be of a predetermined γ characteristic. The output of this γ-correcting circuit 103 is converted into a digital signal by the A/D converter 104, is time-serially separated into respective wavelengths through the selector 105 and is stored as images in the memories 106R, 106G and 106B. The video signals read out of these memories 106R, 106G and 106B are synchronized, are converted into analog video signals by the D/A converters 107R, 107G and 107B and are input into the image processing part 108.
This means that the difference between the video signal, corresponding to the region in which the light absorbing degree of blood hardly varies with the variation of SO2 and the video signal corresponding to the region in which the light absorbing degree of blood varies with the variation of SO2, is determined. How much oxygen is dissolved in the object, that is, the oxygen saturated degree is determined from them. The outputs of the above mentioned two differential amplifiers 205 and 206 are input into the divider 207 and SO2 is determined by a predetermined operation. The output signal of this divider has γ corrected again by the γ correcting circuit 209 through the selector 208 and is output as RGB signals. In this case, the RGB signals are identical and a black and white image is output.
In this embodiment, such abnormalities as the overflow and underflow in the respective operation processes of the logarithmic amplifiers 204R, 204G and 204B, differential amplifiers 205 and 206 and divider 207 are detected by the ineffective region detecting circuit 210. In the operation processes, for example, of the logarithmic amplifiers 204R, 204G and 204B. When the input signal is very low, the operation result will be of an abnormal value but, when it is very high, the possibility of being a halation will be so high as to be determined to be of an abnormal value. In the operation processes of the differential amplifiers 205 and 206, in case the output of the differential amplifier 205 is to be divided by the output of the differential amplifier 206 in the divider 207 in the later step, if the output value of the differential amplifier 206 is near 0, the operatable range of the divider 207 will be exceeded and therefore when the output value of the differential amplifier 206 is near 0, the output value will be determined to be abnormal. In the operation process of the divider 207, in case the operation output is minute, the part will be of a color close to a gray color and the possibility of no presence of hemoglobin in that part will be so high that such a case will be determined to be abnormal. Thus, in each operation process, in case the predetermined signal level is exceeded, a signal will be output to the ineffective region detecting circuit 210 from the operation circuit. In case an abnormal operation is made in the operation process, a signal will be transmitted to the abnormal part data ROM 211 from this ineffective region detecting circuit 210 and a predetermined abnormal part signal will be output from this abnormal part data ROM 211. The selector 208 will select the signal from the divider 207 in the ordinary case but the signal from the abnormal part data ROM 211 in case the signal is output from the ineffective region detecting circuit 210 and will output the signal to the γ correcting circuit 209.
In this embodiment, as shown in FIG. 5, there is no image processing part 108 of the first embodiment but instead a new image processing part 401 is connected to the RGB memories 106R, 106G and 106B at the output thereof. The output signals of this image processing part 401 are converted to analog signals by the D/A converters 107R, 107G and 107B and are output from the respective R, G and B signal output ends 109, 110 and 111.
Then, in Step S3, SL*+L* is made SL*, Sa* +a* is made Sa* and Sb*+b* is made Sb*.
In the process shown in FIG. 6, the values of L*, a* and b* are calculated from the data of R, G and B of the original image, their average values are determined, the functions for the enhancement are determined, the values of L*, a* and b* are again calculated from the data of R, G and B of the original image, V, C and H are determined, are enhanced and are converted to the data of R, G and B to make an image of the result of the process. The details of this process are shown in the publication of Japanese patent application laid open No. 173182/1988.
The same as in the first embodiment, in the above mentioned rotary filter 116, the filter 116R transmits a red color light near 650 nm, the filter 116G transmits a green color light near 580 nm and the filter 116B transmits an infrared light near 800 nm. The light from the lamp 118 is time-serially separated into the light of the wavelengths corresponding to the above mentioned respective filters 116R, 116G and 116B and is radiated to an object to be imaged. The output signal of the CCD 101 imaging the object image is time-seriallly separated into three RGB signals corresponding to the above mentioned respective wavelengths and is stored in the memories 106R, 106G and 106B. The respective images in the respective memories 106R, 106G and 106B are transferred to a work memory 402 within the image processing part 401.
In the above mentioned initial setting, by transferring the image data, the G component Image-- G(X-- size, Y-- size) and B component Image-- B(X-- size, Y size) of the original image are set. Also, a hemoglobin amount housing array IHb (X-- size, Y-- size) and pseudo-color data housing arrays R(X-- size, Y-- size), G(X-- size, Y-- size) and B(X-- size, Y-- size) are prepared and are respectively initialized. Also, the illuminating condition data are transferred to an array Light (X-- size, Y-- size). This Light (X-- size, Y size) has data in which a reference white color plate such as of magnesium oxide is photographed, a region in which the light amount is less than 1/2 the light amount in the center part is made 0 and the other region is made 1. Thirty-two kinds of quasi-color data and 3 kinds of ineffective region displaying data are stored respectively for RGB in an array Color (35, 3). For example, when 1 to 32 are made normal data and 33 to 35 are made ineffective region data, the data of the above mentioned Light X-- size, Y-- size) and Color (35, 3) will be read out of an auxiliary memorizing apparatus 405. Also, work variables x, y, high and low are initialized to be respectively 0, 0, 230 and 30.
Then, in the operating process, as shown in FIG. 9, first of all, in Step S31, y+1 is made y and then, in Step S32, x+1 is made x.
Then, in Step S33, if an Image-- G and Image-- B are not both 0, log {Image G(X,Y)}-log {Image-- B(X,Y)} will be determined to be IHb(X,Y).
Then, in Step S34, if IHb (X,Y)>max, IHb(X,Y) will be made max and, if IHb (X,Y)<min, IHb(X,Y) will be made min.
Then, in Step S35, it is determined whether x<X-- size or not. In the case of YES, the process will return to Step S32 but, in the case of NO, the process will proceed to the next Step S36. In this Step S36, it is determined whether y<Y-- size or not. In the case of YES, the process will return to Step S31 but, in the case of NO, the operation process will end and will proceed to a pseudo-color process.
Thus, in the operation process, the hemoglobin amount IHb is calculated on all the pixels in which the Image-- G and Image-- B are not both 0 and the maximum value and minimum value of IHb are determined respectively as max and min. The vicinity of 580 nm, to which the G image corresponds, is a wavelength region in which the light absorbing degree of blood (hemoglobin) is large and the vicinity of 800 nm, to which the B image corresponds, is a wavelength region in which the light absorbing degree of blood is small. Therefore, the hemoglobin amount is determined by the operation between these two images.
Then, in the pseudo-color process, as shown in FIG. 10, first of all, in Step S41, y+1 is made y and then, in Step S42, x+1 is made x.
Then, in Step S43, when 32 {IHb (X,Y)-min }/{max-min } is made IHb(X,Y), IHb will be normalized to be 0 to 32. The pseudo-color data Color {IHb(X,Y), 1}, Color {IHb(X,Y), 2} and Color {IHb(X,Y), 3} corresponding to this normalized IHb are read out of the color array and are substituted respectively into R (X,Y), G (X,Y) and B (X,Y).
Then, in Step S44, in case at least one of the Image-- G and Image-- B is higher than the prescribed value (for example, 230 if 8-bit data), it will be determined to be a halation part and ineffective region displaying data, that is, in this case, the 33rd data Color (33, 3), Color (33, 2) and Color (33, 3) are substituted respectively into R (X,Y), G (X,Y) and B (X,Y).
Then, in Step S45, in case at least one of the Image-- G and Image-- B is lower than the prescribed value (for example, 30 if 8-bit data), it will be determined to be a shadow part and ineffective region displaying data, that is, in this case, the 34th data Color (34, 1), Color (34, 2) and Color (34, 3) are substituted respectively into R (X,Y), G (X,Y) and B (X,Y).
Then, in Step S46, the part in which Light (X,Y) is 0, that is, the part in which the illuminating condition is deteriorated and the part in which at least one of the Image-- G and Image-- B is 0, that is, the part in which the operation process is impossible are considered to be parts in which the precision can not be guaranteed and ineffective region displaying data, that is, in this case, 35th data Color (35, 1), Color (35, 2) and Coloar (35, 3) are substituted respectively into R (X,Y), G (X,Y) and B (X,Y).
Then, in Step S47, it is determined whether x<X-- size or not. In the case of YES, the process will return to S42 but, in the case of NO, the process will proceed to the next Step S48. In this Step S48, it is determined whether y<Y-- size or not. In the case of YES, the process will return to Step S41 but, in the case of NO, the hemoglobin amount calculating process will end.
As shown in FIG. 3, the video signals corresponding to the three RGB wavelength regions are input into the image processing part 108. As the respective input signals are input respectively into the inverse γ-correcting circuits 201R, 201G and 201B and have had γ already corrected by the γ-correcting circuit 103, an inverse γ-correction is made to uncorrect the image. The output of the inverse γ-correcting circuit 201B is input into a halation detecting part 301 and the halation part is detected from the level of the B signal. In an ordinary endoscope image, the B signal exists only in the low level region. In case the B signal is in the high level region, it will be able to be determined to be of a halation. That is to say, in the halation detecting part 301, the level of the B signal is monitored so that, in case it is in the high level region, a signal will be transmitted to the ineffective region detecting circuit 210.
In case a halation is detected in the halation detecting part 301, a signal will be transmitted from the above mentioned ineffective region detecting circuit 210 to the abnormal part data ROM 211 from which predetermined part signals will be output to the selectors 303R, 303G and 303B. In these selectors 303R, 303G and 303B, in an ordinary case, signals from the differentiating circuits 302R, 302G and 302B will be selected but, in case a signal showing that a halation is detected is output from the ineffective region detecting circuit 210, a signal from the abnormal part data ROM 211 will be selected and will be output to the γ-correcting circuiuts 304R, 304G and 304B. An example of the output image by this embodiment is shown in FIG. 13(b) and, for the sake of comparison, an image by the conventional edge extracting process is shown in FIG. 13(a).
As shown in FIG. 3, video signals corresponding to the three RGB wavelength regions are input into the image processing part 108. The respective input signals are input respectively into the inverse γ-correcting circuits 201R, 201G and 201B, have had γ already corrected by the γ-correcting circuit 103 and therefore have inverse γ-corrected to uncorrect it to the original state. The outputs of the inverse γ correcting circuits 201R, 201G and 201B are input into the halation detecting part 312 wherein a luminance signal Y is determined by the following formula from the three RGB signals and the halation is detected from the level of this luminance signal Y:
A lamp 521 generating light in a wide band from ultraviolet rays to infrared rays is provided within an observing apparatus (which shall be mentioned as a video processor hereinafter). For this lamp 521 can be used a general xenon lamp or strobe lamp which generates a large amount of not only a visible light but also an ultraviolet light and infrared light. This lamp 521 is fed with an electric power by a current source part 522.
The light from the observed part illuminated by this illuminating light is made to form an image on the solid state imaging device 516 by the objective lens system 515 and is photoelectrically converted. A driving pulse from a driver circuit 531 within the above mentioned video processor is applied to this solid state imaging device 516 through the above mentioned signal line 526 so that reading-out and transfer may be made by this driving pulse. The video signal read out of this solid state imaging device 516 is input into a pre-amplifier 532 provided within the above mentioned video processor 1 or electronic endoscope 1 through the signal line 527. The video signal amplified by this pre-amplifier 532 is input into a processing circuit 533, is processed in this processing circuit 533 to correct γ and remove carriers, to have a knee characteristic against halation parts, to give a bias to dark parts and to be on a pedestal level. The output of the above mentioned processing circuit 533 is converted to a digital signal by an A/D converter 534. This digital video signal is selectively stored in a memory(1) 536a, memory(2) 536b and memory(3) 536c corresponding to respective colors, for example, of red (R), green (G) and blue (B) by a selector 535 through an output range LUT (look-up table) 545 provided with a look-up table for limiting the output range. The outputs of the above mentioned memory(1) 536a, memory(2) 536b and memory(3) 536c are simultaneously read out, are converted to analog signals by a D/A converter 537 and are output as R, G and B color signals.
Even in case the displaying device is only a television monitor or a device different in the displaying capacity is used, a reliable effective data region has not been considered. That is to say, the same image data have been input into any of the displaying such as a monitor high in the gradation characteristic and a displaying device narrow in the gradation expression as a monitor or video printer low in the gradation characteristic. Therefore, the displaying capacity has not been able to be well utilized in the displaying device high in the gradation characteristic and the displaying capacity has been so insufficient as to produce an artifact in the displaying device low in the gradation characteristic.
The R, G and B image signals from the superimposing circuit 541 are input into a color lag detecting circuit 547 and image selecting circuit 549. In the frame sequential type endoscope, as the color is time-sequentially separated, in case the movement of the object is fast, a so-called color lag in which the positions of the respective images corresponding to the respective colors will lag to show the primary colors will be generated. The above mentioned color lag detecting circuit 547 detects such color lag amount and the range in which the color lag is produced. This color lag detecting circuit 547 is formed as shown, for example, in FIG. 17. That is to say, the G image signal is input into a first integrator 571 and subtractor 574. The B image signal is input into a second integrator 572 and the above mentioned subtractor 574 through a variable gain amplifier 570. The outputs of both integrators 571 and 572 are input into a gain controlling circuit 573 controlling the gain of the above mentioned variable variable gain amplifiler 570. The output of the above mentioned subtractor 574 is input into a window comparator 575. The upper threshold value Vth and lower threshold value -Vth of this window comparator 575 are set by a window setter 576. The output of the above mentioned window comparator 575 is input into a mask signal generating circuit 548 in FIG. 16. In this color lag detecting circuit 547, the G image signal is integrated for 1 field or 1 frame period in the first integrator 571. The B image signal is amplified by the variable gain amplifier 570 and is then integrated for 1 field or 1 frame period in the second integrator 572. The respective outputs of both integrators 571 and 572 are compared with each other in the gain controlling circuit 573. The output of this gain controlling circuit 573 controls the gain of the variable gain amplifier 570 so that the respective outputs of both integrators 571 and 572 may be equal to each other. As a result, the G image signal and B image signal input into the subtractor 574 will become equal in the integrated value within 1 field or 1 frame period. The G image signal and B image signal are so high in correlation that, as shown in FIG. 18(a), the output of the above mentioned subtractor 574 will approach 0 in case no color lag is produced but the absolute value will become large in case a color lag is produced. As shown in FIG. 18(b), the window comparator 575 will output an H level signal in case the output of the above mentioned subtractor 574 deviates from the range from Vth to -Vth but will output an L level signal in the other case. Therefore, the region in which the output of the above mentioned window comparator 575 is on the H level is the region in which the color lag is large.
The image signal output from the processing circuit 533 is input into the clipping circuit 550 in which the low level image signal low in S/N and lacking in the reliability and the high level image signal such as of the over-exposure and halation part are clipped respectively to fixed values. Thus, the ineffective data are made a fixed value by the clipping circuit 550, are converted to digital data by the A/D converter 534 and are input into the LUT circuit 551. In this LUT circuit 551, the data are converted by input and output characteristics as are shown in FIG. 20. That is to say, the low level part and high level part clipped to the fixed values in the clipping circuit 550 are both converted to fixed ineffective signal levels. The thus converted image signals are input into the selector 535. The high level data among the image data are in non-linear input and output relations in the processing circuit 533 having a knee characteristic due to a halation part or very high exposure level. In the dark part, due to the lack of exposure, the image signal will be on a low level, therefore the S/N will reduce and the image data will be low in reliability. If an image is processed or measured by using such a high level or low level data, an artifact will be likely produced.
All the signals among the image signals output from the endoscope apparatus are not always effective. That is to say, such character information as of the picture frame and patient data is not required to have the image processed and, unless such information is definitely separated from the image data imaged by the endoscope, the data after the image is processed will be meaningless. If such character information as of the picture frame and patient data and the effective region are discriminated from each other by the software on the image processing apparatus side, the processing speed will be reduced.
In this embodiment, as shown in FIG. 23, the respective filters of the rotary filter 539 transmit light of narrow bands having 570 nm, 650 nm and 805 nm respectively as centers.
In the above mentioned selecting circuit 650, the ineffective region producing an artifact is replaced with a mask signal and thus the image signal in which the effective Hb distribution image region and the ineffective region producing an artifact are definitely discriminated is processed to be pseudo-colorized in response to the level of the image signal in a pseudo-colorizing circuit 651. The Hb distribution image from this pseudo-colorizing circuit 651 is input into the superimposing circuit 541 and the patient information input in the character information input circuit 542 is superimposed on the image information. The image signal to which such character information as of the displaying frame and patient information is thus added is output as R, G and B signals which are input into a matrix circuit 543. In this matrix circuit 543, the above mentioned R, G and B signals are converted to color difference signals and a luminance signal which are input into an encoder circuit 544 and are converted to an NTSC signal to be output. The above mentioned NTSC signal is or the R, G and B signals are input into the color monitor in which the Hb distribution image is displayed in the pseudo-color.
The output of the above mentioned selecting circuit 656 is processed to be an image for calculating the same Hb distribution as in the tenth embodiment in the image processing circuit 657. The output of this image processing circuit 657 is converted to an analog signal by the D/A converter 658 and is output.
With respect to neither dark part nor halation part but to the image data part which is reliable, the above mentioned selecting circuit 656 selects the image data from the memories 536a to 536c as they are stored. With respect to the dark part detected by the dark part detecting circuit 653 and the halation part detected by the halation detecting circuit 652, the selecting circuit 656 selects the image signal processed by the dark part processing circuit 654 and the image signal of the mask data output from the halation mask data generating circuit 655.
The image data selected by the selecting circuit 657 are processed by the image processing circuit 657 to make an Hb distribution image. The respective output signals from the dark part detecting circuit 653, halation detecting circuit 652 and image frame generating circuit 540 are input as control signals into the selecting circuit 657. The image processing circuit 657 does not process on the basis of the signals from the respective detecting circuits 653 and 652 the already detected dark part and halation part but outputs the signal from the dark part processing circuit 654 and the signal from the halation mask data generating circuit 655 as they are to the D/A comparator 658. Also, the image processing circuit 657 does not process the data outside the picture frame not required to be processed as an Hb distribution image.
The Hb distribution image data processed and obtained by the above mentioned image processing circuit 657 are converted to an analog image signal by the D/A converter 658 and are output to a monitor or the like.
Patent CitationsCited PatentFiling datePublication dateApplicantTitleUS4901143 *Feb 16, 1989Feb 13, 1990Olympus Optical Co., Ltd.Electronic endoscope system provided with a means of imaging frozen pictures having few picture image smearsUS4914512 *Jan 18, 1989Apr 3, 1990Kabushiki Kaisha ToshibaElectronic endoscope apparatus capable of displaying hemoglobin concentration on color imageUS4922915 *Sep 13, 1989May 8, 1990Ben A. ArnoldAutomated image detail localization methodUS4962540 *Nov 27, 1989Oct 9, 1990Olympus Optical Co., Ltd.Method and apparatus for detecting corresponding regions between picture imagesUS5078150 *May 2, 1989Jan 7, 1992Olympus Optical Co., Ltd.Spectral diagnosing apparatus with endoscopeUS5512940 *Mar 17, 1994Apr 30, 1996Olympus Optical Co., Ltd.Image processing apparatus, endoscope image sensing and processing apparatus, and image processing method for performing different displays depending upon subject quantity* Cited by examinerReferenced byCiting PatentFiling datePublication dateApplicantTitleUS6271891 *Jun 18, 1999Aug 7, 2001Pioneer Electronic CorporationVideo signal processing circuit providing optimum signal level for inverse gamma correctionUS6594074 *Aug 4, 1999Jul 15, 2003Carl Zeiss Jena GmbhMicroscope, especially laser scanning microscope with rotatable interference filtersUS6768526 *Jun 4, 2001Jul 27, 2004Industrial Technology Research InstituteTime-sequential color separator and liquid crystal projector using the sameUS7554535 *Oct 3, 2002Jun 30, 2009Nec CorporationDisplay apparatus, image display system, and terminal using the sameUS7813576 *Nov 24, 2003Oct 12, 2010Fujifilm CorporationImage processing apparatusUS7846107May 13, 2005Dec 7, 2010Boston Scientific Scimed, Inc.Endoscopic apparatus with integrated multiple biopsy deviceUS7924308 *Oct 20, 2005Apr 12, 2011Fujifilm CorporationElectronic endoscope apparatus and method of controlling image luminance for electronic endoscope apparatusUS7955255Apr 20, 2006Jun 7, 2011Boston Scientific Scimed, Inc.Imaging assembly with transparent distal capUS7967759Jan 19, 2006Jun 28, 2011Boston Scientific Scimed, Inc.Endoscopic system with integrated patient respiratory status indicatorUS8000525Feb 22, 2005Aug 16, 2011Panasonic CorporationImage processing device, image processing system, image processing method, image processing program, and integrated circuit deviceUS8052597Aug 30, 2005Nov 8, 2011Boston Scientific Scimed, Inc.Method for forming an endoscope articulation jointUS8083671Sep 29, 2005Dec 27, 2011Boston Scientific Scimed, Inc.Fluid delivery system for use with an endoscopeUS8097003May 13, 2005Jan 17, 2012Boston Scientific Scimed, Inc.Endoscopic apparatus with integrated variceal ligation deviceUS8118732Sep 30, 2004Feb 21, 2012Boston Scientific Scimed, Inc.Force feedback control system for video endoscopeUS8144993Jun 8, 2007Mar 27, 2012Olympus CorporationMedical image processing methodUS8197400Sep 28, 2005Jun 12, 2012Boston Scientific Scimed, Inc.Selectively rotatable shaft couplerUS8199187Sep 30, 2005Jun 12, 2012Boston Scientific Scimed, Inc.Adapter for use with digital imaging medical deviceUS8202265Apr 20, 2006Jun 19, 2012Boston Scientific Scimed, Inc.Multiple lumen assembly for use in endoscopes or other medical devicesUS8213676Dec 20, 2006Jul 3, 2012Ge Inspection Technologies LpInspection apparatus method and apparatus comprising motion responsive controlUS8353860Sep 30, 2005Jan 15, 2013Boston Scientific Scimed, Inc.Device for obstruction removal with specific tip structureUS8357148Sep 29, 2005Jan 22, 2013Boston Scientific Scimed, Inc.Multi-functional endoscopic system for use in electrosurgical applicationsUS8425408Sep 17, 2009Apr 23, 2013Boston Scientific Scimed, Inc.Articulation joint for video endoscopeUS8435172Dec 8, 2008May 7, 2013Boston Scientific Scimed, Inc.Automated control of irrigation and aspiration in a single-use endoscopeUS8475366Aug 24, 2009Jul 2, 2013Boston Scientific Scimed, Inc.Articulation joint for a medical deviceUS8514278Dec 29, 2006Aug 20, 2013Ge Inspection Technologies LpInspection apparatus having illumination assemblyUS8535219Mar 31, 2010Sep 17, 2013Boston Scientific Scimed, Inc.Fluid manifold for endoscope systemUS8553953 *Jun 22, 2010Oct 8, 2013National Yunlin University Of Science And TechnologyEndoscopic navigation method and endoscopic navigation systemUS8585715Dec 14, 2011Nov 19, 2013Boston Scientific Scimed, Inc.Endoscopic apparatus with integrated variceal ligation deviceUS8608648Nov 22, 2011Dec 17, 2013Boston Scientific Scimed, Inc.Articulation jointUS8622894Dec 30, 2011Jan 7, 2014Boston Scientific Scimed, Inc.Articulation jointUS8657737 *Jan 28, 2011Feb 25, 2014Fujifilm CorporationElectronic endoscope system, an electronic endoscope processor, and a method of acquiring blood vessel informationUS8810636Dec 20, 2006Aug 19, 2014Ge Inspection Technologies, LpInspection apparatus method and apparatus comprising selective frame outputUS8825125Dec 21, 2011Sep 2, 2014Fujifilm CorporationEndoscope system and processor apparatus thereof, and image generating methodUS8870753Jun 2, 2011Oct 28, 2014Boston Scientific Scimed, Inc.Imaging assembly with transparent distal capUS8888684Mar 27, 2006Nov 18, 2014Boston Scientific Scimed, Inc.Medical devices with local drug delivery capabilitiesUS8944996 *Jan 28, 2011Feb 3, 2015Fujifilm CorporationElectronic endoscope system with oxygen saturation enhancing image regionUS9358363May 22, 2012Jun 7, 2016Boston Scientific Scimed, Inc.Multiple lumen assembly for use in endoscopes or other medical devicesUS9439557Sep 22, 2011Sep 13, 2016Boston Scientific Scimed, Inc.Articulation jointUS9621808Jul 14, 2014Apr 11, 2017General Electric CompanyInspection apparatus method and apparatus comprising selective frame outputUS9629555 *Feb 2, 2015Apr 25, 2017Fujifilm CorporationEndoscope system, endoscope system processor device, operation method for endoscope system, and operation method for endoscope system processor deviceUS20020060650 *Oct 23, 2001May 23, 2002Asahi Kogaku Kogyo Kabushiki KaishaSchematic illustration drawing apparatus and methodUS20020080304 *Jun 4, 2001Jun 27, 2002Fang-Chuan HoTime-sequential color separator and liquid crystal projector using the sameUS20040101188 *Nov 24, 2003May 27, 2004Fuji Photo Film Co., Ltd.Image processing apparatusUS20040148141 *Jan 14, 2004Jul 29, 2004Fuji Photo Film Co., Ltd.Method and apparatus for displaying fluorescence images and method and apparatus for acquiring endoscope imagesUS20040246242 *Oct 3, 2002Dec 9, 2004Daigo SasakiDisplay apparatus, image display system, and terminal using the sameUS20060082646 *Oct 20, 2005Apr 20, 2006Fujinon CorporationElectronic endoscope apparatus and method of controlling image luminance for electronic endoscope apparatusUS20070183656 *Feb 22, 2005Aug 9, 2007Yasuhiro KuwaharaImage processing device, image processing system, image processing method, image processing program, and integrated circuit deviceUS20080151047 *Dec 20, 2006Jun 26, 2008General Electric CompanyInspection apparatus method and apparatus comprising selective frame outputUS20080152210 *Dec 20, 2006Jun 26, 2008General Electric CompanyInspection apparatus method and apparatus comprising motion responsive controlUS20080292154 *Jun 8, 2007Nov 27, 2008Olympus CorporationMedical image processing methodUS20110160534 *Jun 22, 2010Jun 30, 2011Tsung-Chun LeeEndoscopic navigation method and endoscopic navigation systemUS20110230715 *Jan 28, 2011Sep 22, 2011Takaaki Saitoelectronic endoscope system, an electronic endoscope processor, and a method of acquiring blood vessel informationUS20110237882 *Jan 28, 2011Sep 29, 2011Takaaki SaitoElectronic endoscope systemUS20110237884 *Jan 28, 2011Sep 29, 2011Takaaki SaitoElectronic endoscope systemUS20130113906 *Oct 23, 2012May 9, 2013Fujifilm CorporationEndoscope system, processor device thereof, and method for displaying oxygen saturation levelUS20150238086 *Feb 2, 2015Aug 27, 2015Fujifilm CorporationEndoscope system, endoscope system processor device, operation method for endoscope system, and operation method for endoscope system processor deviceUSRE46007Feb 23, 2015May 24, 2016Boston Scientific Scimed, Inc.Automated control of irrigation and aspiration in a single-use endoscopeCN102197985A *Jan 28, 2011Sep 28, 2011富士胶片株式会社Electronic endoscope systemCN102197985B *Jan 28, 2011Mar 4, 2015富士胶片株式会社电子内窥镜系统CN102860812A *Jul 2, 2012Jan 9, 2013富士胶片株式会社Electronic endoscope system, light source device, and control method of electronic endoscope systemCN102860812B *Jul 2, 2012Feb 3, 2016富士胶片株式会社电子内窥镜系统、光源装置、及电子内窥镜系统的控制方法CN103099598A *Nov 8, 2012May 15, 2013富士胶片株式会社Endoscope system, processor device thereof, and method for displaying oxygen saturation levelCN104939792A *Mar 24, 2015Sep 30, 2015富士胶片株式会社Medical image processing device and method for operating the sameEP1842481A1 *Dec 8, 2005Oct 10, 2007Olympus CorporationMedical image processing methodEP1842481A4 *Dec 8, 2005Nov 4, 2009Olympus CorpMedical image processing methodEP2476373A1Dec 20, 2011Jul 18, 2012Fujifilm CorporationEndoscope system and processor apparatus thereof, and image generating methodEP2591716A1Oct 31, 2012May 15, 2013Fujifilm CorporationEndoscope system, processor device thereof, and method for displaying oxygen saturation levelWO2007031946A2Sep 12, 2006Mar 22, 2007Dvp Technologies Ltd.Medical image processing* Cited by examinerClassifications U.S. Classification382/128, 348/65International ClassificationA61B1/04, H04N7/18, G06F19/00, G06T5/00Cooperative ClassificationG06T7/136, A61B1/00009, A61B1/0638, A61B1/043, G06T2207/10068, G06T7/0012, G06T2207/10016, G06F19/3487, G06T2207/30004, G06T5/50, G06F19/3406, G06F19/322, A61B1/0646European ClassificationG06F19/34A, G06F19/34P, G06T7/00B2, G06T5/50Legal EventsDateCodeEventDescriptionFeb 28, 2003FPAYFee paymentYear of fee payment: 4Feb 26, 2007FPAYFee paymentYear of fee payment: 8Apr 25, 2011REMIMaintenance fee reminder mailedSep 21, 2011LAPSLapse for failure to pay maintenance feesNov 8, 2011FPExpired due to failure to pay maintenance feeEffective date: 20110921RotateOriginal ImageGoogle Home - Sitemap - USPTO Bulk Downloads - Privacy Policy - Terms of Service - About Google Patents - Send FeedbackData provided by IFI CLAIMS Patent Services