Abstract:
An image signal processing apparatus includes an inputting unit, an electrical image stabilizing unit, and a compression encoding unit. The inputting unit is arranged to input an image signal representing a sequence of images of a subject produced by an image sensing unit. The electrical image stabilizing unit is arranged to detect unintentional movement of the image caused by shaking of an image sensor unit on the basis of the image signal and to correct the image signal, after the input signal is input by the input unit, by removing the effect of the unintentional movement, so as to yield a corrected sequence of images in which a position of the subject is stabilized. The compression encoding unit is arranged to interpicture encode the image signal stabilized by the electrical image stabilizing unit.

Description:
This application is a continuation of application Ser. No. 07/900,945, filed Jun. 17, 1992, now abandoned. 

   BACKGROUND OF THE INVENTION 
   1. Field of the Invention 
   This invention relates to a signal processing device, and more particularly, to compression processing of image signals. 
   2. Description of the Related Art 
   Recently, so-called digital VCRs (video cassette recorders) have been considered in which digital signal processing of image signals of so-called VCR&#39;s is performed using magnetic tape having a width of ½ inches or 8 millimeters for recording moving-image signals. 
   The digital VCR performs various kinds of image processing using correlation between image in time. One example of such processing is frame correlation image compression processing which uses correlation between images in time. 
     FIG. 5  is a diagram showing the principle of the frame correlation image compression processing. 
   In  FIG. 5 , there are shown a picture frame n, and a picture frame n+1 immediately succeeding the picture frame n in time as indicated by arrow t.  FIG. 5  illustrates a case in which a coded pixel (picture element) P n+1  of the next picture frame n+1 is subjected to interframe encoding based on a target pixel of P n  of the picture frame n. 
   The processing is based on the premise (an assumption) that there is strong correlation between immediately successive picture frames of a moving-image signal, and basically, pixel data at the same position in adjacent frames have substantially the same value. 
   If encoding is performed according to simple interframe prediction, only the difference between the pixels P n  and P n+1 , the variance in the image deviation from the above-described premise, is encoded. Hence, it is possible to reduce the amount of data by a method, such as nonlinear quantization or the like, which is similar to one-dimensional DPCM (differential pulse-code modulation). 
   Such a method compresses the data so that, for example, only transmission of information of “the same as the preceding frame” suffices in the case of a perfectly still picture on the assumption that correlation increases as the values of pixel data of succeeding frames are closer. 
   However, when correlation between images is low, data corresponding to the predicted error from a predicted correlation value must be transmitted. Hence, when the entire picture frame has much movement, it is possible to temporarily generate a large amount of data. 
   Accordingly, in the case of a portable image recording apparatus provided with the above-described digital signal processing system, various kinds of image processing using time-base correlation between images cannot be effectively performed due to displacement of picture frames caused by unintended hand movement. 
   SUMMARY OF THE INVENTION 
   The present invention has been made in consideration of the problems of the prior art. 
   It is an object of the present invention to provide a signal processing device which can efficiently perform various kinds of image processing using time-base correlation of images. 
   This object is accomplished, according to one aspect of the present invention, by a signal processing device comprising an image displacement corrector that removes image displacement in an image signal, and a compression unit that compresses the image signal output from the image displacement corrector. 
   Other objects and advantages of the invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings. 

   
     BRIEF DESCRIPTION OF THE DRAWINGS 
       FIG. 1  is a block diagram of an image pickup apparatus according to a first embodiment of the present invention; 
       FIGS. 2(   a ) and  2 ( b ) are diagrams illustrating correction of shaking according to the first embodiment; 
       FIG. 3  is a diagram illustrating a picture surface for displaying image information according to the first embodiment; 
       FIG. 4  is a block diagram of an image pickup apparatus according to a second embodiment of the present invention; and 
       FIG. 5  is a schematic diagram illustrating image compression according to a time-base correlation method. 
   

   DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
   A signal processing device illustrating a signal processing unit provided within an image pickup apparatus including an image pickup means will now be explained. 
   An image pickup apparatus according to a first embodiment of the present invention will be explained in detail with reference to  FIGS. 1–3 . 
     FIG. 1  is a block diagram of the image pickup apparatus of the present embodiment.  FIGS. 2(   a ) and  2 ( b ) are diagrams illustrating shaking correction.  FIG. 3  is a diagram illustrating a picture surface for displaying image information. 
   In  FIG. 1 , there are shown an imaging optical system  1 , and a solid-state image pickup device  2 , such as a CCD (charge-coupled device) or the like. Incident light of a photographed image from the imaging optical system  1  is directed to the solid-state image pickup device  2 , and is subjected to photoelectric conversion, whereby an image signal is obtained. 
   An A/D converter  3  converts the image signal into a digital image signal. A digital signal processing circuit (DSP circuit)  4  performs well-known video camera signal processing, such as y-conversion, knee-characteristic processing, color balance adjustment or the like. 
   A shaking correction circuit  5  corrects shaking of the picture surface. An image memory  6  temporarily stores an image signal for at least one image field. There are also shown an image compression circuit  7 , and a memory  8  for storing image signals. A memory control circuit  9  controls reading of stored image signals. A microcomputer  10  for controlling the system (hereinafter termed a system controller) controls the image compression circuit  7 , the memory control circuit  9 , an address control circuit  11  (to be described below) and the like. 
   The address control circuit  11  controls storing of image signals in the memory  8 . A shaking/panning sensor  12  detects panning which is frequently used in camera work and operation key  13  are also shown in  FIG. 1 . 
   An explanation will now be provided of the signal processing operations until storage of an image by the image pickup apparatus having the above-described configuration. 
   A photographed optical image from the imaging optical system  1  is focused on the image pickup device  2 . The focused optical image is converted into an electrical image signal by the image pickup device  2 . The image signal is converted into a digital image signal by the A/D converter  3 , and is input to the DSP circuit  4 , in which predetermined signal processing is performed. The image signal output from the DSP circuit  4  is input to the shaking correction circuit  5 , in which shaking of the photographed image is corrected. 
   The operation of the shaking correction circuit  5  will be more specifically explained with reference to  FIGS. 2(   a ) and  2 ( b ). 
   As shown in  FIG. 2(   a ), a photographed picture frame having a size of H o ×V o  which has allowance in both the horizontal and vertical directions relative to a substantially effective picture frame having a size of h×v (horizontal width×vertical width) is set, and is first stored in the image memory  6 . 
   As shown in  FIG. 2(   b ), cutting of an effective picture frame so as to cancel shaking of an image is realized by controlling the setting of a memory address. That is, the shaking correction circuit  5  includes a circuit for detecting a moving vector of an image signal, and detects a moving vector corresponding to shaking of the picture frame generated by unintended hand movement. Thus, the shaking correction circuit serves as an electrical stabilizing unit to stabilize the picture frame electrically. 
   In the example shown in  FIG. 2(   b ), movement from point A to point B between adjacent frames is detected, and a moving vector v is input to the memory control circuit  9 . The memory control circuit  9  changes the read address so as to cancel the influence of the moving vector v. 
   If it is assumed that point P n  to start reading of image data at the upper left position of picture frame n is 
   P n  (X, Y)=(X n , Y n ), 
   and that the moving vector v is 
   v(X, Y)=(X v , Y v ), 
   point P n+1  to start reading of data on picture frame n+1 is set as 
   P n+1  (X, Y)=(X n +X v , Y n +Y v ). 
   By thus controlling the memory address, an image signal in which shaking of the picture frame due to unintended hand movement is removed is obtained from the image memory  6 . 
   In the stabilized image signal obtained as described above, correlation in the direction of the time base is high. The above-described image signal is subjected to data compression processing including time-base correlation processing by the image compression circuit  7  to reduce the amount of image data, and is then stored in the large-capacity solid-state memory  8 . 
   As is well known in the art, the highly correlated image signal obtained by removing motion with the motion vector is more easily compressed so that the overall compression of the image signal is significantly improved. 
   By appropriately combining the above-described inter-frame correlation processing using temporal correlation between images, moving-vector compensation for reducing deterioration in picture quality in the interframe correlation processing, two sided predictive interframe compression in which interframe compression is performed for time periods before and after (past and future) the present time on time base, and the like, it is possible to reduce the amount of data of a digital image signal to about a fraction of a few tenths to a few hundredths of the original amount. 
   More specifically, according to the JPEG (Joint Photographic Expert Group) method in which encoding is performed by DCT (discrete cosine transform) in units of a block comprising (8×8) pixels, or the so-called MPEG-1 which is an algorithm proposed by the MPEG (Moving Picture Expert Group), standard picture quality substantially equivalent to that of a so-called ½-inch VCR can be obtained. 
   The above-described compression and storage processing is performed by operating the system controller  10  and the address control circuit  11  controlled by the system controller  10  through the operation key  13 . 
   By further adding the shaking/panning sensor  12  in which the movement of the main body of the camera is detected by external sensors, shaking correction and image compression processing can be more efficiently performed. The shaking/panning sensor  12  comprises a plurality of one-dimensional sensors, such as accelerometers or the like, which constitute necessary detecting axes. For example, when panning which is frequently used in camera work is detected, panning information (direction, speed, timing and the like) is transmitted from the one-dimensional sensor in the horizontal direction of the shaking/panning sensor  12  to the shaking correction circuit  5  and the system controller  10 . 
   The shaking correction circuit  5  which has received the panning information can prevent unnatural processing at the start and end of panning by interrupting correction in the panning direction during a panning operation. 
   Correlation processing in the horizontal direction can be properly executed according to the panning information transmitted to the compression circuit  7  via the system controller  10 , whereby compression processing is optimized. 
     FIG. 3  illustrates a picture surface for displaying image information. 
   In the present embodiment, based on a TV signal of the NTSC system (525 scanning lines), a TV signal having an aspect ratio (9:16) of the image which is close to that of a movie is provided as an input signal. If the effective ration of the picture surface is assumed to be 94% both in the vertical and horizontal directions, information of (495×880) pixels for 30 picture frames is generated per second, and is supplied to the A/D converter  3 . 
   The composite image signal input to the A/D converter  3  is converted into a digital signal, and is subjected to video signal processing of components (for example, Y:(R−Y):(B−Y)=4:2:2) by the DSP circuit  4  whenever necessary. If luminance (Y) and color difference are quantized with 8 bit quantization at a sampling rate of ½, information of about 700 G (giga) bits is obtained per hour. 
   If a stereo sound signal is digitized at 48 kHz (kilohertz)/16 bits, information of about 5.5 G bits per hour is obtained. If the above-described compression processing is performed for these signals, information of about 4 G bits is obtained for the entire AV (audio-visual) signal. 
   Next, an explanation will be provided of an image pickup apparatus according to a second embodiment of the present invention with reference to  FIG. 4 . 
     FIG. 4  is a block diagram of the image pickup apparatus of the second present embodiment. In  FIG. 4 , the same components as in  FIG. 1  are indicated by the same reference numerals, and explanation thereof will be omitted. 
   In  FIG. 4 , an error correcting code adding circuit  14  corrects erroneous codes in a digital image signal output from the image compression circuit  7 . A recording system  15  uses media, such as magnetooptical or other disks, digital VCR tapes or the like. A servo system  16  mechanically controls the recording system  15 . 
   That is, in the second embodiment, an image pickup apparatus which does not employ a solid-state memory is used, the same components as in the first embodiment are used except the above-described three components are added, and the same image signal processing as in the first embodiment is performed. 
   As explained above in detail, since the signal processing device of the present invention performs time-base correlation processing of image data compression after removing shaking in an image signal and correlation between images is increased, it is possible to increase the efficiency of data in data compression. 
   More particularly, the present invention is expected to largely remove the influence of hand movement likely to be produced when a small portable image pickup apparatus is used. 
   By having a configuration of processing photographed information in the form of digital signals and recording the signals whenever necessary, the present invention can also prevent a reduction in the quality of information and reliability during a retrieving or copying operation. 
   Various changes and modifications may be made without departing from the spirit and scope of the invention. 
   For example, although in the above-described embodiments, the moving vector of an image signal is detected between adjacent frames, the moving vector of an image signal may be detected between adjacent fields. 
   In other words, the foregoing description of the embodiments has been given for illustrative purposes only and is not to be construed as imposing any limitation in every respect. 
   The scope of invention is, therefore, to be determined solely by the following claims and not limited by the text of the specification and alterations that may be made within the scope of the claims fall within the true spirit and scope of the invention.