Source: https://patents.justia.com/patent/7176966
Timestamp: 2019-09-18 14:07:14
Document Index: 321267759

Matched Legal Cases: ['art 3', 'art 4', 'art 4', 'art 3', 'art 30', 'art 5', 'art 40', 'art 4', 'art 5', 'art 40', 'art 5', 'art 5', 'art 4', 'art 4', 'art 5', 'art 4', 'art 5']

US Patent for Image processing device Patent (Patent # 7,176,966 issued February 13, 2007) - Justia Patents Search
Justia Patents Combined Image Signal Generator And General Image Signal ProcessingUS Patent for Image processing device Patent (Patent # 7,176,966)
Feb 15, 2002 - Olympus
Objects of the present invention are to provide an image processing device and image processing method which make it possible to process the high definition image at a moving image rate with a high speed.
The image-pickup unit 2 has a lens 6 which forms an optical image of the subject, a color separation prism 7 which subjects the light incident on the abovementioned lens 6 to color separation, four (for example) image-pickup elements 8a, 8b, 8c and 8d on which optical images are formed via the abovementioned color separation prism 7, a lens control circuit 9 which controls the lens 6 into a focused state and the like, an image-pickup element driving circuit 10 which drives the image-pickup elements 8a, 8b, 8c and 8d, a CPU 11 which controls the lens control circuit 9 and the like, and a synchronization signal generating circuit 12 which supplies a basic clock signal and horizontal and vertical synchronization signals to the image-pickup element driving circuit 10 and the like.
Furthermore, the signal processing part 3 has image-pickup signal processing circuits 13a through 13d which perform processing that converts the output signals of the image-pickup elements 8a through 8d into digital signals, an image arrangement conversion circuit 14 that performs image sequencing from the output signals of the abovementioned image-pickup signal processing circuits 13a through 13d, an image splitting circuit 15 which splits the output signal of the abovementioned image arrangement conversion circuit 14 into (for example) eight image regions, and image processing circuits 16a through 16h which perform processing that produces RGB color signals for the signals of the eight image regions produced by the splitting performed by the image splitting circuit 15.
Furthermore, the recording part 4 has image compression circuits 17a through 17h that perform image compression processing on the output signals of the image processing circuits 16a through 16h, image recording circuits 18a through 18h that perform recording processing on the output signals of the image compression circuits 17a through 17h, and recording media 19a through 19h that record the output signals of the image recording circuits 18a through 18h. Moreover, in FIG. 1, the image compression circuits 17a through 17h are included in the recording part 4; however, it would also be possible to include these circuits in the signal processing part 3.
FIG. 2 shows the configuration of the color separation prism 7 and four image-pickup element parts. The light passing through the lens 6 (i. e., the light reflected from the subject) is split by blue (B), G1 (green), G2 (green) and red (R) prisms 7a, 7b, 7c and 7d, and these split light beams are respectively formed as images on the image-pickup elements 8a, 8b, 8c and 8d.
Furthermore, as is shown in FIG. 3A, the abovementioned image-pickup elements 8a through 8d are joined to the prisms 7a, 7b, 7c and 7d in positions that are shifted by ½ of one pixel to the up-down directions and left-right directions with respect to the image that is formed in common. Moreover, a function that is equivalent to that of an image-pickup element that has four times the pixels of a single image-pickup element is achieved by means of the four image-pickup elements 8a through 8d.
In the present embodiment, image-pickup elements that have (for example) 2,000,000 pixels are respectively used as the image-pickup elements 8a through 8d; accordingly, an overall resolution of 8,000,000 pixels is obtained. As is shown in FIG. 4, image-pickup elements that have 1920 pixels in the horizontal direction and 1080 pixels in the vertical direction (in this case, an aspect of 16:9) are used.
The pixel in the upper left corner is a pixel of the image-pickup element 8a; the pixels below this pixel and to the right of this pixel are pixels of the image-pickup elements 8b and 8c, and the pixel to the right of the pixel of the image-pickup element 8b is a pixel of the image-pickup element 8d. These four pixels are arranged in a repeating pixel arrangement in the lateral direction (horizontal direction) and longitudinal direction (vertical direction). Furthermore, in order to simplify the description, in FIG. 3B, these pixels are indicated by the symbols a through d (except for eight parts) in the image-pickup elements 8a through 8d.
As is shown in FIG. 1, the image-pickup element driving circuit 10 receives a signal from the synchronization signal generating circuit 12 and generates a driving signal, and this driving signal is applied in common to the four image-pickup elements 8a through 8d. As a result, the four image-pickup elements 8a through 8d are driven in parallel, and the output signals of these image-pickup elements are respectively input into the four image-pickup signal processing circuits 13a through 13d.
The image-pickup signal processing circuits 13a through 13d process in parallel the analog signals that are simultaneously output in parallel from the image-pickup elements 8a through 8d and converts into digital signals. Then, these signals are output to the image arrangement conversion circuit 14, which converts the image arrangement.
In the present embodiment, as was described above, the four plate-type image-pickup elements 8a through 8d are driven at the same timing by a common driving signal; accordingly, the following merits are obtained: namely, the configuration of the driving system can be simplified, and the image-pickup signal processing circuits 13a through 13d can process the output signals in common, so that these circuits can also be simplified.
In the abovementioned image arrangement conversion circuit 14, the image signals of respective colors that are simultaneously output from the image-pickup signal processing circuits 13a through 13d are subjected to an arrangement conversion so that an image signal with a Bayer arrangement is produced.
Specifically, as is shown in model form in FIG. 4, image signals B, G1, G2 and R output from the image-pickup elements 8a through 8d (and processed by image signal processing circuits 13a through 13d) are input into the image arrangement conversion circuit 14. In this case, the respective image signals are image signals of 1920 pixels in the horizontal direction and 1080 pixels in the vertical direction; an arrangement conversion is performed by the image arrangement conversion circuit 14 so that an image signal with a Bayer arrangement of 3840 pixels in the horizontal direction and 2160 pixels in the vertical direction is produced.
The R, G1, G2 and B image signals that are respectively output from the image-pickup signal processing circuits 13d, 13b, 13c and 13a at 74.25 MHz are input via wiring lines into eight image splitting constituent circuits (hereafter referred to as “image splitting circuits” for the sake of simplification) 15a through 15h.
Furthermore, the image arrangement conversion circuit 14 is actually constituted to adjust the timing at which the image signals are input into the image splitting circuits 15a through 15h. Specifically, the image signal formed by combining the R, G1, G2 and B image signals that are input into the image splitting circuits 15a through 15h is an image signal with a Bayer arrangement of 3840 pixels in the horizontal direction and 2160 pixels in the vertical direction, as shown in FIG. 4, and image arrangement conversion and image splitting are actually performed in parallel.
For example, the image splitting circuit 15a is constructed from two sets of four FIFO memories, i. e., eight FIFO memories 22, four first selectors 23 (two sets SEL RG and SEL BG in FIG. 5), two second selectors 24 (two sets SEL RB in FIG. 5), and a third selector 25 (SEL F in FIG. 5). The output signal of the third selector 25 is input into the image processing circuit 16a. The other image splitting circuits 15b through 15h also have a similar configuration.
Writing into the FIFO memories 22 and reading from the FIFO memories 22 are performed under the control of a memory control unit 26. In this case, in all of the FIFO memories 22 installed in the image splitting circuits 15a through 15h, writing is performed in units of four memories. In other words, all of the FIFO memories 22 are constructed in units of four, from F1_1 to F1_8, and from F2_1 to F2_8. Here, for example, F1_1 has F1_1_R, F1_1_G1, F1_1_G2 and F1_1_B used for the simultaneous writing of the four images (signals) R, G1, G2 and B, respectively. The other FIFO memories are similar.
As is shown in FIG. 6, signals of R, G1, G2 and B are input into the FIFO memories 22 simultaneously (in parallel) from the image-pickup signal processing circuits 13d, 13b, 13c and 13a in synchronization with the horizontal synchronizing signal by a clock of 74.25 MHz. Furthermore, if the period of the 74.25 MHz clock is designated as T, then one horizontal period is 2200T, and of this period, 1920T is the signal input period of the effective pixels.
Furthermore, while writing is being performed into the FIFO memories 22 from F2_1 through F2_8, the memories F1_1_G2 and F1_1_B, and F1_1_R and F1__G1, are successively read by a read clock at 18.56 MHz, which is ¼ of 74.25 MHz.
Furthermore, being switched by the second selector 24 (in concrete terms, SEL RB) and the third selector 25 (in concrete terms, SEL F) as shown in FIG. 6, the signals B_0 and G2_0 read out from F1_1_G2 and F1_1_B are output at 37.13 MHz and input into the image processing circuit 16i (i=a to h). Similarly, the signals G1_0 and R_0 read out from F1_1_R and F1_1_G are output at 37.13 MHZ and input into the image processing circuit 16i.
The signal processing circuits 16i performs processing which produces RGB signals for the respective pixels. Namely, each of the abovementioned pixels has only a single color signal component, and lacks the other two color signal components; accordingly, the lacking color signals are produced by the image processing circuit 16i using the signals of the surrounding pixels.
FIG. 8A shows pixels in a Bayer arrangement that are input into one of the image processing circuits 16i; signals of pixels that have RGB components are produced from the pixels of this arrangement as shown in FIGS. 8B, 8C and 8D. In FIG. 8A, for example, an R pixel is designated as the center, and adjacent R signals are produced using the surrounding 5×5 pixels.
The signal that is thus subjected to image processing by the image processing circuit 16i (i=a to h) is input into the image compression circuit 17i (i=a to h) and compressed by a compression method such as JPEG, MPEG or the like, so that the signal is converted into a signal that can be recorded on the recording medium 19i by the image recording circuit 18i. Then, each signal is respectively recorded on a non-volatile, large-capacity recording medium 19i (i=a to h) such as a hard disk or the like.
Furthermore, since a plurality of recording media 19a through 19h are also used, the present invention also has the effect of allowing recording at a moving image rate even in the case of compressed data of high-definition images that would exceed the recording rate that is possible in the case of a single recording medium.
Specifically, the output signals of the image processing circuits 16i are input into the image conversion circuits 30i (i=a to h) of an image conversion circuit part 30 that forms a portion of the display part 5. Then, after being converted into images consisting of desired numbers of pixels by the image conversion circuits 30i, these signals are input into a display processing circuit 31, and signal processing which allows display by a display device 32 is performed, so that the signals can be displayed by this display device 32.
The output signals from the image conversion circuits 30a through 30h are input into two sets of eight FIFO memories each, i. e., FIFO memories (1) and FIFO memories (2) via a selector 33. Writing and reading into and from the two sets of the FIFO memories (1) and (2) are controlled by an FIFO controller 35. The signals read out from the two sets of the FIFO memories (1) and (2) pass through selectors 36a and 36b, and are further output from a common selector 37.
As is shown in FIG. 12, the horizontal periods (1100T) of the image conversion circuits 30a through 30h are ½ the horizontal period (2200T) in the high-vision television (HDTV).
Furthermore, in the period (1000T) corresponding to the first half of the horizontal period (2200T) of the HDTV, the RGB signals output from the image conversion circuits 30a through 30h are written into the FIFO memories (1) while being thinned (these signals are simultaneously written into the memories indicated by 1 through 8 in FIG. 12). In the latter half-period (1000T), writing is not performed, and the image is thinned by one line. In the next first half-period, writing is similarly performed into the other FIFO memories (2), and in the subsequent latter half-period, writing is not performed, so that the image is thinned by one line.
As is shown in FIG. 14, the horizontal periods (1100T) of the image conversion circuits 30a through 30h are ½ the horizontal period (2200T) of the high-vision television.
Furthermore, the RGB signals (e. g., images of line n) output from the four image conversion circuits 30a through 30d in the horizontal period (100T) are written into half of the FIFO memories (1) (e. g., simultaneously into the memories indicated by 1 through 4 in FIG. 14), and in the next horizontal period, the images of line n+1 are written into the remaining half of these FIFO memories (1) (simultaneously into the memories indicated by 5 through 8 in FIG. 14).
Furthermore, in the present embodiment, as was described above, image-pickup elements that have an aspect of 16:9 are used as the image-pickup elements 8a through 8d, and the embodiment is constructed so that the overall image obtained from the image-pickup elements 8a through 8d is split into eight rectangular images (the image regions indicated by {circle around (1)} through {circle around (8)} in the vertical direction by the image splitting circuit 15 as shown in FIG. 18A, and so that the images of the respective regions are further subjected to image processing in the image processing circuits 16a through 16h. Here, in the case of output to a monitor with an aspect of 16:9 used as the display device 32, the acquired images can easily be displayed on a full screen of this aspect by selecting all of the image regions indicated by {circle around (1)} through {circle around (8)}.
Specifically, in this second embodiment, eight image processing circuits 16a through 16h are used for a display device 32 with an aspect of 16:9; as a result, an image suited to the display screen can easily be produced. Alternatively, by using six image processing circuits 16b through 16g for a display device 32 with an aspect of 4:3, an image suited to the display screen of this display device can easily be produced.
Furthermore, in cases where six image processing circuits 16b through 16g are used, the power supplies for the two image processing circuits 16a and 16h can be cut off so that the electric power consumption can be reduced. Furthermore, in this second embodiment, a case was described in which the image was split into eight parts; however, it would also be possible to split the image into an integer multiple of eight parts.
The image processing device 1B of the abovementioned second embodiment had a configuration in which the output signals of the image processing circuits 16a through 16h were respectively input into the image conversion circuits 30a through 30h and displayed by the display device 32 as shown in FIG. 9. However, the present modification is arranged so that images recorded on the recording media 19a through 19h can be reproduced and displayed.
Accordingly, image reproduction circuits 41a through 41h which are connected to the recording media 19a through 19h, and which reproduce images, and image expansion circuits 42a through 42h which perform expansion processing on the output signals of the abovementioned image reproduction circuits 41a through 41h, are provided, and the system is arranged so that the output signals of the image expansion circuits 42a through 42h can be input into the image conversion circuits 30a through 30h and displayed by the display device 32.
The modification of the image processing device shown in FIG. 19 has a configuration in which a reproduction and expansion part 40 which performs reproduction and expansion processing on the recording media 19i is installed in addition to the recording part 4 and display part 5 in the image processing device of the second embodiment shown in FIG. 9. The remaining parts have the same configuration as the image processing device of the second embodiment shown in FIG. 9.
In this first modification, images recorded on the recording media 19a through 19h can be reproduced and displayed; accordingly, recorded images can be checked.
This image processing device 1D is devised so that the signals of the image processing circuits 16a through 16h are further input into the image conversion circuits 30a through 30h in the image processing device 1C shown in FIG. 19. Furthermore, this modification is devised so that images recorded on the recording media 19i can be reproduced and displayed, and so that input images from the image processing circuits 16i can also be displayed, by means of a switch or the like not shown in the figures. The remaining parts have the same configuration as the first modification shown in FIG. 19.
In this second modification, picked-up images can be displayed, and images recorded on the recording media 19a through 19h can also be reproduced and displayed.
The image processing device 1E of the third embodiment of the present invention shown in FIG. 21 has a configuration in which an image synthesizing circuit 45 which synthesizes the plurality of split images produced by the image processing circuits 16a through 16h into a single image is provided in the image processing device 1B of the second embodiment of the present invention shown in FIG. 9, and in which an image splitting circuit 46 which splits the image synthesized by the abovementioned image synthesizing circuit 45 into a plurality of images is further provided; the split images produced by the splitting performed by this image splitting circuit 46 are compressed by a plurality of image compression circuits 17a through 17h, and recording processing is further performed by image recording circuits 18a through 18h so that the images are recorded on recording media 19a through 19h.
In the eight split images produced by the respective image processing circuits 16a through 16h, as is shown in FIG. 22A, the size of the R, G and B color component images is (for example) 480×2160 pixels, and these split images are synthesized by the image synthesizing circuit 45 to form a single color image, i. e., three color component image.
In the image processing device 1F of this first modification, the reproduction and expansion part 40 shown in FIG. 19 is further installed in the image processing device 1E shown in FIG. 21. Specifically, image information that is compressed and recorded on the recording media 19a through 19h is reproduced by the image reproduction circuits 41a through 41h, and the output signals are expanded by the image expansion circuits 42a through 42h.
In the present modification, the output images of the image expansion circuits 42a through 42h are synthesized into a single image by the image synthesizing circuit 47. This image synthesizing circuit 47 is connected to the image splitting circuit 48 installed in the display part 5; the synthesize image is split into a plurality of images by this image splitting circuit 48, and these images are input into the image conversion circuits 30a through 30h. The output signals are input into the display device 32 via the image processing circuit 31.
In this first modification, the device has a configuration in which the side on which the image-pickup unit and the recording unit can be freely attached and detached. Furthermore, in this configuration, the recording unit and the display part 5 are also detachable. Accordingly, the recording media 19a through 19h of the recording part 4 can easily be changed, and the display device 32 or the like can also easily be changed. Furthermore, the recording part 4 or display part 5 that is connected and used can be changed in accordance with the use environment, so that more convenient parts can be selected and used.
This image processing device 1G is a modification of the image processing device 1F in FIG. 23, and constructed so that the output signals from the image processing circuits 16a through 16h are input into the image conversion circuits 30a through 30h, thus making it possible to display image-picked-up signals as well on the display device 32. In this second modification, in addition to the effects and merits of the abovementioned first modification, it is also possible to display image-picked-up signals on the display device 32.
Furthermore, in FIG. 24, a configuration is shown in which the output signals from the image processing circuits 16a through 16h are input into the image conversion circuits 30a through 30h; however, it would also be possible to use a configuration in which the output of the image synthesizing circuit 45 is input into the image splitting circuit 48. In other words, it is possible to connect the image splitting circuit 46 of the freely detachable recording part 4 to the image synthesizing circuit 45 so that images can be recorded on the recording media 19a through 19h, or to connect the image splitting circuit 48 of the freely detachable display part 5 to the image synthesizing circuit 45 so that picked-up images can be displayed by the display device 32.
Furthermore, in regard to the number of pixels that are incorporated by overlapping, the number of pixels that are necessary for the image processing of adjacent pixels are incorporated as a unit as shown in FIG. 27. For example, in the case shown in FIG. 27, two pixels in the vertical and horizontal directions, i. e., 2×2 pixels, constitute a unit of arrangement. In this case, therefore, an amount equal to two pixels from the boundary may be set as the amount of overlapping. Furthermore, the number of pixels that are incorporated from outside the ends of the split images {circle around (1)} and {circle around (8)} may also be the same as this amount of overlapping.
As a result of the respective split image regions thus being formed so that the respective image regions are caused to overlap (including partial regions to the outside of the regions at both ends), color signals can be produced simply and quickly even for the case of pixels at the ends of the respective image regions by using the pixels of the outside overlapping portions, wherein image processing that produces three color signals from the signals of the respective split image regions by means of the respective image processing circuits 16i from a single color component by interpolation using the color components of peripheral pixels is performed.
Even if there are no overlapping portions, the required pixels from adjacent image regions can be incorporated to produce color signals. However, the respective image processing circuits 16i cannot perform processing in parallel for this incorporation; accordingly, the processing becomes complicated, and the processing speed drops. Furthermore, since no outside pixels are incorporated in the pixels at each end of the two image regions ({circle around (1)} and {circle around (8)}) located at both ends facing the outside, the image quality also drops in this case.
In the image processing device 1I of the sixth embodiment shown in FIG. 31, a shading correction circuit 56 that performs optical shading correction on the image-pickup signal processing circuits 13a through 13d is installed in the image processing device 1B shown in FIG. 9. The remaining parts have the same configuration as in FIG. 9.
Furthermore, it is also possible to perform shading correction on the post-stage side, e. g., on the side of the image processing devices 16a through 16h; in such a case, however, correction coefficients corresponding to the respective regions of the split images processed by the image processing device 16a through 16h must be set as shown (for example) in FIG. 32E, as a result it is difficult to perform shading correction using the same circuit configuration. On the other hand, if the system is devised so that shading Correction is performed by the image-pickup signal processing circuits 13a through 13d on the pre-stage side of image splitting as shown in FIG. 31, then shading corrections can be performed using the same circuit configuration, so that the circuit configuration can also be simplified.
The image processing device 1J of the seventh embodiment shown in FIG. 33 has a configuration in which the output of the image synthesizing circuit 45 in the image processing device 1E shown in FIG. 21 is converted into a plurality of images, specifically four images in accordance with the video signals, by an image distribution circuit 61, and these images are output four display devices (1) 62a through (4) 62d. For the sake of simplicity, the display devices will be indicated as (1) through (4) below.
wherein the integrated image arrangement conversion and splitting circuit includes a first plurality N of memories, a second plurality N/2 of selectors of a first type coupled with outputs of the memories, a third plurality of N/4 selectors of a second type coupled with outputs of the selectors of the first type, and a fourth plurality of N/B selectors of a third type coupled with outputs of the selectors of the second type.
5379070 January 3, 1995 Retter et al.
5436661 July 25, 1995 Yamamoto et al.
5669010 September 16, 1997 Duluk, Jr.
5703648 December 30, 1997 Shikakura
6020922 February 1, 2000 Lee
6697115 February 24, 2004 Amano
60-154781 August 1985 JP
Patent number: 7176966
Patent Publication Number: 20020113885
Inventors: Jun Inoue (Akiruno), Osamu Inagaki (Hachioji), Shinichi Nakajima (Koganei), Hiroki Shibasaki (Hachioji)
Application Number: 10/077,445
Current U.S. Class: Combined Image Signal Generator And General Image Signal Processing (348/222.1); Lens Or Filter Substitution (348/360); Each Supplying Only One Color Signal (348/265)