Image compression associated with camera shake correction

Abstract: A signal processor corrects an image signal by removing unwanted displacement and compresses the corrected image signal. The compression may use similarity of the image over time. Where an otherwise scarcely-changing image moves as a whole due to camera shake, the apparent similarity over time is low and compression is poor. By using displacement-corrected data, the similarity over time is increased and compression is improved.

Description: BACKGROUND OF THE INVENTION Field of the Invention This invention relates to a signal processing device, and more particularly, to compression processing of image signals. Description of the Related Art Recently, so-called digital VCR's (video cassette recorders) are being considered in which digital signal processing of image signals of so-called VCR's is performed which use magnetic tapes having a width of 1/2 inches or 8 millimeters for recording moving-image signals. The digital VCR performs various kinds of image processing using correlation between images in time. One example of such processing is frame correlation image compression processing which uses correlation between images in the direction of time base. FIG. 5 is a diagram showing the principle of the frame correlation image compression processing. In FIG. 5, there are shown a picture frame n, and a picture frame n + 1 immediately after the picture frame n in the direction of time base indicated by arrow t. FIG. 5 illustrates a case in which a coded pixel (picture element) P n+1 of the next picture frame n + 1 is subjected to interframe encoding based on a target pixel P n of the picture frame n. The processing is based on a premise (an assumption) that there is strong correlation between immediately successive picture frames of a moving-image signal, and basically, pixel data at the same position in adjacent frames have substantially the same value. If encoding is performed according to simple interframe prediction, only the difference between the pixels Pn and P n+1 , which is the variance in the image deviation from the above-described premise, is encoded. Hence, it is possible to reduce the amount of data by a method, such as nonlinear quantization or the like, which is similar to one-dimensional DPCM (differential pulse-code modulation). Such a method compresses the amount of data so that, for example, only transmission of information of "the same as the preceding frame" suffices in the case of a perfectly still picture on the assumption that correlation increases as the values of pixel data are closer between frames. However, when correlation between images is low, data corresponding to the predicted error from a predicted correlation value must be transmitted. Hence, when the entire picture frame has much movement, there is a possibility of temporarily generating a large amount of data. Accordingly, in the case of a portable image recording apparatus provided with the above-described digital signal processing system, various kinds of image processing using time-base correlation between images cannot be effectively performed owing to displacement of picture frames caused by unintended hand movement. US-A-4769826 describes a video encoding apparatus in which panning movement of a video camera mounted on a tripod is detected using an X/Y motion sensor. A distortion operation member selects from a plurality of blocks of the preceding frame the block providing minimum distortion to the existing input block. A subtracter 6 estimates a difference signal series between the signal series of the input block and the signal series to give the minimum distortion and outputs the different series signal to a quantization encoding member. The quantized difference or error component forms a coded output signal, together with information representing which block is the most resembling one. A paper entitled 'A Low Bit Rate Coding Algorithm For Full Motion Video Signal' by Haghiri and Denoyelle published in Signal Processing Image Communication, Volume 2, No. 2, August 1990 at pages 187 to 199 describes one type of algorithm for predictive interframe coding. WO90/00334 and the English language abstracts for JP-A-1-109970 and JP-A-1-174076 describe methods for stabilising or removing fluctuations from image signals. In particular, WO90/00334 describes a method of and apparatus for electronically stabilising incoming video data in which a comparison is made between succeeding frames of at least the most intense pixel (the hotspot) and possibly also of the intensity of pixels surrounding the hotspot pixel to avoid errors occasioned by camera movement or to enable tracking of a moving object and maintaining the object centrally in a monitor screen. SUMMARY OF THE INVENTION In one aspect, the present invention provides an image signal processing apparatus in accordance with claim 1. In another aspect, the present invention provides a method for processing an image signal in accordance with claim 12. The present invention enables a signal processing apparatus to be provided which can efficiently perform various kinds of image processing using time-base correlation of images. Embodiments of the invention will now be described by way of example with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram of an image pickup apparatus according to a first embodiment of the present invention; FIGS. 2(a) and 2(b) are diagrams illustrating correction of shaking according to the first embodiment; FIG. 3 is a diagram illustrating a picture surface for displaying image information according to the first embodiment; FIG. 4 is a block diagram of an image pickup apparatus according to a second embodiment of the present invention; and FIG. 5 is a schematic diagram illustrating image compression according to a time-base correlation method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A signal processing device illustrating a signal processing unit provided within an image pickup apparatus including an image pickup means will now be explained. An image pickup apparatus according to a first embodiment of the present invention will be explained in detail with reference to FIGS. 1 - 3. FIG. 1 is a block diagram of the image pickup apparatus of the present embodiment. FIGS. 2(a) and 2(b) are diagrams illustrating shaking correction. FIG. 3 is a diagram illustrating a picture surface for displaying image information. In FIG. 1, there are shown an imaging optical system 1, and a solid-state image pickup device 2, such as a CCD (charge-coupled device) or the like. Incident light of a photographed image from the imaging optical system 1 is imaged onto the solid-state image pickup device 2, and is subjected to photoelectric conversion, whereby an image signal is obtained. An A/D converter 3 converts the image signal into a digital image signal. A digital signal processing circuit (DSP circuit) 4 performs well-known video camera signal processing, such as γ-conversion, knee-characteristic processing, color balance adjustment or the like. A shaking correction circuit 5 corrects shaking of the picture surface. An image memory 6 temporarily stores an image signal for at least one field. There are also shown an image compression circuit 7, and a memory 8 for storing image signals. A memory control circuit 9 controls a reading operation of stored image signals. A microcomputer 10 for controlling the system (hereinafter termed a system controller) controls the image compression circuit 7, the memory control circuit 9, an address control circuit 11 (to be described below) and the like. The address control circuit 11 controls storing of image signals in the memory 8. A shaking/panning sensor 12 detects panning which is frequently used in a camera work. There is also shown an operation key 13. An explanation will now be provided of the signal processing operations until storage of an image by the image pickup apparatus having the above-described configuration. A photographed optical image from the imaging optical system 1 is focused onto the image pickup device 2. The focused optical image is converted into an electrical image signal by the image pickup device 2. The image sigal is converted into a digital image signal by the A/D converter 3, and is input to the DSP circuit 4, in which predetermined signal processing is performed. The image signal output from the DSP circuit 4 is input to the shaking correction circuit 5, in which shaking of the photographed image is corrected. The operation of the shaking correction circuit 5 will be more specifically explained with reference to FIGS. 2(a) and 2(b). As shown in FIG. 2(a), a photographed picture frame having a size of H 0 x V 0 which has allowance both in the horizontal and vertical directions relative to a substantially effective picture frame having a size of h x v (horizontal width x vertical width) is set, and is first stored in the image memory 6. As shown in FIG. 2(b), cutting of an effective picture frame so as to cancel shaking of an image is realized by controlling the setting of a memory address. That is, the shaking correction circuit 5 includes a circuit for detecting a moving vector of an image signal, and detects a moving vector corresponding to shaking of the picture frame generated by unintended hand movement. In the example shown in FIG. 2(b), movement from point A to point B between adjacent frames is detected, and a moving vector v is input to the memory control circuit 9. The memory control circuit 9 changes the read address so as to cancel the influence of the vector v. If it is assumed that point P n to start reading of image data at the upper left position of picture frame n is P n (X, Y) = (X n , Y n ), and the moving vector v is v(X, Y) = (X v , Y v ), point P n+1 to start reading of data on picture frame n + 1 is set as P n+1 (X, Y) = (X n + X v , Y n + Y v ). By thus controlling the memory address, an image signal in which shaking of the picture frame due to unintended hand movement is removed is obtained from the image memory 6. In the stabilized image signal obtained as described above, correlation in the direction of time base is high. The above-described image signal is subjected to data compression processing including time-base correlation processing by the image compression circuit 7 to reduce the amount of image data, and is then stored in the large-capacity solid-state memory 8. By appropriately combining the above-described interframe correlation processing using temporal correlation between images, moving-vector compensation for reducing deterioration in picture quality in the interframe correlation processing, both-side predictive interframe compression in which interframe compression is performed for time periods before and after (past and future) the present time on time base, and the like, it is possible to reduce the amount of data of a digital image signal to about a fraction of a few tenths to a few hundredths of the original amount. More specifically, according to the JPEG (Joint Photographic Expert Group) method in which encoding is performed by DCT (discrete cosine transform) in units of a block comprising (8 x 8) pixels, or the so-called MPEG-I which is an algorithm proposed by the MPEG (Moving Picture Expert Group), standard picture quality substantially equivalent to that of a so-called 1/2-inch VCR can be obtained. The above-described compression and storage processing is performed by operating the system controller 10 and the address control circuit 11 controlled by the system controller 10 through the operation key 13. By further adding the shaking/panning sensor 12 in which the movement of the main body of the camera is detected by external sensors, shaking correction and image compression processing can be more efficiently performed. The shaking/panning sensor 12 comprises a plurality of one-dimensional sensors, such as accelerometers or the like, which constitute necessary detecting axes. For example, when panning which is frequently used in a camera work is detected, panning information (direction, speed, timing and the like) is transmitted from the one-dimensional sensor in the horizontal direction of the shaking/panning sensor 12 to the shaking correction circuit 5 and the system controller 10. The shaking correction circuit 5 which has received the panning information can prevent unnatural processing at the start and end of panning by interrupting correction of the panning direction during a panning operation. Correlation processing in the horizontal direction can be properly executed according to the panning information transmitted to the compression circuit 7 via the system controller 10, whereby compression processing is optimized. FIG. 3 illustrates a picture surface for displaying image information. In the present embodiment, based on a TV signal of the NTSC system (525 scanning lines), a TV signal having an aspect ratio (9:16) of the image which is close to that of a movie is provided as an input signal. If the effective ratio of the picture surface is assumed to be 94 % both in the vertical and horizontal directions, information of (495 x 880) pixels for 30 picture frames is generated per second, and is supplied to the A/D converter 3. The composite image signal input to the A/D converter 3 is converted into a digital signal, and is subjected to video signal processing of components (for example, Y:(R - Y):(B - Y) = 4:2:2) by the DSP circuit 4 whenever necessary. If luminance (Y) and color difference are quantized in 8 bits with a sampling rate of 1/2, information of about 700 G (giga) bits is obtained per hour. If a stereo sound signal is digitized at 48 kHz (kilohertz)/16 bits, information of about 5.5 G bits per hour is obtained. If the above-described compression processing is performed for these signals, information of about 4 G bits is obtained for the entire AV (audio-visual) signal. Next, an explanation will be provided of an image pickup apparatus according to a second embodiment of the present invention with reference to FIG. 4. FIG. 4 is a block diagram of the image pickup apparatus of the second present embodiment. In FIG. 4, the same components as in FIG. 1 are indicated by the same reference numerals, and explanation thereof will be omitted. In FIG. 4, an error correcting code adding circuit 14 corrects erroneous codes in a digital image signal output from the image compression circuit 7. A recording system 15 uses media, such as magnetooptical or other disks, digital VCR tapes or the like. A servo system 16 mechanically controls the recording system 15. That is, in the second embodiment, an image pickup apparatus which does not employ a solid-state memory is used, the same components as in the first embodiment are used except the above-described three components are added, and the same image signal processing as in the first embodiment is performed. As explained above in detail, since the signal processing device of the present invention performs time-base correlation processing of image data compression after removing shaking in an image signal and corelation between images is increased, it is possible to increase efficiency in data compression. More particularly, the present invention is expected to greatly remove the influence of hand movement likely to be produced when a small portable image pickup apparatus is used. By having a configuration of processing photographed information in the form of digital signals and recording the signals whenever necessary, the present invention can also prevent a reduction in the quality of information and reliability during a retrieving or copying operation. Various changes and modifications may be made without departing from the scope of the invention. For example, although in the above-described embodiments, the moving vector of an image signal is detected between adjacent frames, the moving vector of an image signal may be detected between adjacent fields. In other words, the foregoing description of the embodiments has been given for illustrative purposes only and is not to be construed as imposing any limitation in every respect. The scope of the invention is, therefore, to be determined solely by the following claims and not limited by the text of the specification and alterations that may be made within the scope of the claims fall within the scope of the invention.

Claims: An image signal processing apparatus, comprising: input means (3, 4) for inputting an image signal representing an image frame produced by an image sensing means (2), and interframe predictive encoding means (7) for encoding the image signal by encoding the difference between the image signal and a predicted image signal based on the image signal for a previous frame, characterised by correction means (5) for sensing unintended movement of the image sensing means (2) and for correcting and thereby stabilising the image signal by removing the effect of the unintended movement, the correction means being adapted to correct the image signal before the image signal is encoded by the interframe predictive encoding means (7) and the interframe predictive encoding means being adapted to carry out the interframe predictive encoding using the corrected image signal and a predicted image signal based on the corrected image signal for a previous frame. An apparatus according to claim 1, wherein said predictive encoding means (7) includes encoding means for encoding in units of blocks, each block comprising a two-dimensional arrangement of picture elements. An apparatus according to claim 1 or 2, wherein the operation of the correction means (5) is arranged to be controlled in accordance with an output of an external sensor (12). An apparatus according to claim 3, wherein said correction means (5) comprises moving vector detection means for detecting a moving vector of the image signal. An apparatus according to claim 1 or 2, wherein said correction means (5) comprises moving vector detection means for detecting a moving vector of the image signal and the apparatus further comprises: memory means (6) for storing an image signal; and memory control means (9) for reading the image signal by changing an address of said memory means (6) in accordance with the moving vector detected by said moving vector detection means. An apparatus according to claim 5, wherein said moving vector detection means is arranged to obtain the moving vector by comparing two frames of the image signal. An apparatus according to claim 5 or 6, wherein said memory means (6) is arranged to store an image signal for at least one field. An apparatus according to claim 5, 6 or 7, wherein said moving vector detection means is arranged to obtain the moving vector by comparing two fields of the image signal. An apparatus according to any one of claims 5 to 8, further comprising a sensor (12) for detecting movement of the entire apparatus. An apparatus according to claim 9, wherein said memory control means (9) is arranged to be controlled in accordance with an output of said sensor (12). A portable image recording apparatus comprising image sensing means (1,2) for converting an optical image into an image signal and an image signal processing apparatus in accordance with any one of claims 1 to 10. A method of processing an image signal, comprising: interframe predictively encoding the image signal by encoding the difference between the image signal and a predicted image signal based on the image signal for a previous frame, characterised by correcting and thereby stabilising the image signal by removing the effect of sensed unintended movement before the image signal is interframe predictively encoded and carrying out the interframe predictive encoding using the corrected image signal and a predicted image signal based on the corrected image signal for a previous frame. A method according to claim 12 which comprises storing the image signal in memory means (6), detecting a moving vector of the image signal and correcting the image signal by reading the image signal from the memory means (6) by changing an address of the memory according to the detected moving vector. A method according to claim 13, which comprises obtaining the moving vector by comparing two frames or fields of the image signal. A method according to claim 12, 13 or 14, which comprises encoding the image signal after correction in units of blocks, each block comprising a two-dimensional arrangement of picture elements.

Inventors: TAKAHASHI KOJI, TAKAHASHI, KOJI, Takahashi, Koji, c/o CANON KABUSHIKI KAISHA
Applicants: CANON KK, CANON KABUSHIKI KAISHA
Application year: 1992
Main classification: H04N5
IPC codes: H04N5, G06T9, H04N7, H04N9