Motion vector detecting apparatus and image stabilizer including the same

Abstract: A motion vector detecting apparatus (50) and an image stabilizer (200) including the motion vector detecting apparatus (50), the motion vector detecting apparatus (50) including a motion vector arithmetic unit (9) for calculating a motion vector of an inputted image signal, an integrator (11) for integrating the motion vector, a still decision unit (20) for judging whether or not the inputted image signal represents a still image, a coring value determiner (21) for determining a coring value on the basis of outputs from the integrator (11) and the still decision unit (20) and a coring circuit (10) which controls the coring value on the basis of an output from the coring value determiner (21) so as to perform coring processing on an output of the motion vector arithmetic unit (9).

Description: The present invention relates to a motion vector detecting apparatus for detecting a motion vector of an inputted image, i.e. an amount of displacement of the inputted image and an image stabilizer for correcting sway of an outputted image, which includes the motion vector detecting apparatus. A motion vector detecting apparatus employing a correlational arithmetic unit is known from, for example, Japanese Patent Laid-Open Publication JP-A-2246687 and the corresponding EP-A-0 389 168 which falls under article 54 (3) EPC. Fig. 1 to 4 show the known motion vector detecting apparatus in which the correlational arithmetic unit utilizes absolute values of differences. In Figs. 1 and 3, the known motion vector detecting apparatus includes a motion vector detector 9 and a coring processor 13. The motion vector detector 9 includes a fist latch 1, a representative point memory 2, a second latch 3, a subtracter 4, an address controller 5, an absolute value converter 6, an accumulative adder 7 and a decision unit 8 for judging an address of a minimum value. Meanwhile, the coring processor 13 includes a coring circuit 10, an integrator 11 and a coring value determiner 12. The subtracter 4 may be replaced by an adder if a complement is employed. The known motion vector detecting apparatus of the above described arrangement is further described, hereinbelow. Initially, a motion vector of an image is described with reference to Figs. 5a to 5c. Fig. 5a shows an image at a time point and Fig. 5b shows an image subsequent to the image of Fig. 5a by one field or one frame. When the image is displaced in parallel by movement of an image pickup device, etc. as shown in Figs. 5a and 5b, an amount of parallel displacement of the image is expressed by a vector of the arrow in Fig. 5c and this vector is referred to as a "motion vector". Fig. 2 shows a representative point and pixcels surrounding the representative point in a so-called representative point matching method which is most popular among motion vector detecting methods. In this method, image data is disposed at a representative point in a field and a motion vector is detected by performing correlational arithmetic operation for determining to which one of the surrounding pixcels the image data is displaced in the next field. Then, operation of the known motion vector detecting apparatus employing the correlational arithmetic unit is described with reference to Figs. 1 and 2. Image data at respective representative points on a screen are received by the first latch 1 in response to a timing pulse LP1 and are, at a certain timing, written at addresses of the representative point memory 2, which correspond to the representative points, respectively. Subsequently, in the next field or the next frame, a difference between image data in a motion vector detecting area surrounding each representative point and image data of each representative point of the previous field stored in the representative point memory 2 is obtained. An absolute value of the difference is obtained by the absolute value converter 6 and is inputted to the accumulative adder 7. Data representing the absolute values of the differences each obtained on the basis of coordinates of each representative point, namely outputs of the absolute value converter 6 are accumulatively added. When accumulative addition of the data for all the representative points has been completed, location of a minimum value of the accumulative sums stored by the accumulative adder 7 is judged by the decision unit 8. In correlational decision based on the absolute values of the differences, the absolute values of the differences become smaller at locations having closer correlation. Hence, the location (address) of the minimum accumulative sum relative to the location (address) of the corresponding representative point indicates the motion vector. Since the above described operation is performed for each field (frame), the first latch 1 is provided for storing, while correlational arithmetic operation is being preformed, image data at the representative points for correlational arithmetic operation in the next field ( frame). Meanwhile, when correlation between image data at a representative point and image data surrounding the representative point is obtained, the second latch 3 stores the image data at the representative point. Then, the coring processor 13 is described. Fig. 4 shows input/output characteristics of the coring processor 13. Processing in which a minute component of an input signal, i.e. a component corresponding to a portion α of Fig. 4 is outputted as 0 is referred to as "coring processing". Since levels of noise components in a signal are generally low, coring processing is used for eliminating minute noise components. However, effective signal components may exist in minute variable components. Therefore, if the coring amount of Fig. 4 is set too large, change of the signal components will appear conspicuously. Meanwhile, since fluctuations of the detected motion vector also pose a problem, it is necessary to eliminate the fluctuations. Especially when sway of the image of the input signal is small, the fluctuations become conspicuous. Thus, a condition that sway of the image of the input signal is small is detected by integrating an x-axis component vect x and a y-axis component vect y of the motion vector by the integrator 11 and coring characteristics of the coring circuit 10 are controlled by the coring value determiner 12. Fig. 6 show attenuation characteristics of a coring value in the motion vector detector 9. As the integrated motion vector is increased, the coring value is reduced. Thus, in a region where fluctuations of the detected motion vector due to noise components are conspicuous, namely the integrated motion vector is small, the coring amount is increased so as to lessen influence of the fluctuations. On the contrary, in a region where the fluctuations of the detected motion vector due to noise components are not so conspicuous, namely the integrated motion vector is large, the coring amount is reduced such that error of the detected motion vector is lessened. As a result, elimination of the signal components due to coring processing is minimized. However, in the known motion vector detecting apparatus of the above described arrangement, in the case where an image signal of an image having close correlation in one direction is inputted, the integrated motion vector may assume a large value even when the image is a still image, so that coring processing does not function properly and thus, such a problem arises that fluctuations of the motion vector become conspicuous in a state of input of the image signal of the image having close correlation in one direction. Accordingly, an essential object of the present invention is to provide, with a view to eliminating the drawbacks of the conventional motion vector detecting apparatuses, a motion vector detecting apparatus which detects a state of input of a still image so as to control a coring value such that fluctuations of a motion vector do not become conspicuous and which controls, in the case of input of an image other than a still image, the coring value such that the coring value does not assume a small value. In order to accomplish this object of the present invention, a motion vector detecting apparatus for detecting a motion vector of an inputted image signal, in one aspect of the present invention comprises: a motion vector arithmetic unit for calculating the motion vector of the inputted image signal; an integrator for integrating the motion vector; a still decision unit for judging whether or not the inputted image signal represents a still image; a coring value determiner for determining a coring value on the basis of outputs from said integrator and said still decision unit; and a coring circuit which controls the coring value on the basis of an output from said coring value determiner so as to perform coring processing on an output of said motion vector arithmetic unit. Furthermore, in another aspect of the present invention, there is provided an image stabilizer for correcting sway of an outputted image, which includes the motion vector detecting apparatus. By the above described arrangement of the present invention, the detected motion vector is subjected to coring in accordance with magnitude of the detected motion vector and result of decision on the still image. Therefore, in accordance with the present invention, detection accuracy at the time of detection of a large motion vector is not lowered and fluctuations of the motion vector can be suppressed even when correlation in the image is close in one direction. This object and features of the present invention will become apparent from the following description taken in conjunction with the preferred embodiments thereof with reference to the accompanying drawings, in which: Fig. 1 is a block diagram of a motion vector detector employed in a prior art motion vector detecting apparatus (already referred to); Fig. 2 is a view explanatory of a representative point and pixcels surrounding the representative point (already referred to); Fig. 3 is block diagram of a coring processor employed in the prior art motion vector detecting apparatus of Fig. 1 (already referred to); Fig. 4 is a graph showing input/output characteristics of a coring circuit of the coring processor of Fig. 3 (already referred to); Figs. 5a to 5c are views explanatory of a motion vector of an image (already referred to); Fig. 6 is a graph showing attenuation characteristics of a coring value in the motion vector detector of Fig. 1 (already referred to); Fig. 7 is a block diagram of a motion vector detecting apparatus according to a first embodiment of the present invention; Fig. 8 is a block diagram of a still decision unit employed in the motion vector detecting apparatus of Fig. 7; Fig. 9 is a graph showing relation between speed of displacement of image through movement of a camera and level of signal of the still decision unit of Fig. 8; Fig. 10 is a graph showing attenuation characteristics of a coring value in the motion vector detecting apparatus of Fig. 7; Fig. 11 is a block diagram of a still decision unit employed in a motion vector detecting apparatus according to a second embodiment of the present invention; and Fig. 12 is a block diagram of an image stabilizer including the motion vector detecting apparatus of Fig. 7, in another aspect of the present invention. Before the description of the present invention proceeds, it is to be noted that like parts are designated by like reference numerals throughout several views of the accompanying drawings. Referring now to the drawings, there is shown in Fig. 7, a motion vector detecting apparatus 50 according to a first embodiment of the present invention. The motion vector detecting apparatus 50 is characterized by a still decision unit 20 and a coring value determiner 21. In the same manner as a prior art motion vector detecting apparatus of Figs. 1 and 3, the motion vector detecting apparatus 50 includes a motion vector detector 9, a coring circuit 10 and an integrator 11. The motion vector detector 9 includes a first latch 1, a representative point memory 2, a second latch 3, a subtracter 4, an address controller 5, an absolute value converter 6, an accumulative adder 7 and a decision unit 8 for judging an address of a minimum value. As shown in Fig. 8, the still decision unit 20 includes a difference detector 30, a low-pass filter 31 and a decision unit 32. For each field (frame), a minimum value obtained by the decision unit 8 is inputted to the difference detector 30 in which an absolute value of a difference between the inputted minimum value and a minimum value of a frame preceding the frame of the inputted minimum value by one frame is obtained. The low-pass filter 31 for eliminating high frequency band eliminates rapid change from an output of the difference detector 30. If an inputted image signal represents a still image, variations of the minimum value are small, so that the value obtained at the low-pass filter 31 becomes small. On the other hand, if an image is swaying, variations of the minimum value are large, so that the value obtained at the low-pass filter 31 becomes large. Fig. 9 shows relation between speed of displacement of the image (corresponding to speed of movement of a camera) and output value of the low-pass filter 31. The abscissa is expressed by the number of pixcels per field, while the ordinate is expressed by standard deviation of the output value of the low-pass filter 31. In Fig. 9, if output of the low-pass filter 31 is 50 or less, it is possible to judge that the inputted image is still. When a still state, i.e. a state of output of 50 or less from the low-pass filter 31 has been detected continuously, the decision unit 32 judges that the inputted image is still and transmits to the coring value determiner 21, a still signal indicative of the still image. In response to the still signal, the coring value determiner 21 increases a coring value irrespective of an integrated motion vector as shown in Fig. 10. Thus, the coring value is increased irrespective of the integrated motion vector only when the still condition has been established as described above. Therefore, even if correlation in the image becomes close in one direction and thus, the integrated motion vector becomes slightly large, the motion vector can be outputted by compressing fluctuations of the motion vector. Thus, by controlling the coring value by the coring value determiner 21 in this embodiment, elimination of noises from the signal is mainly performed in a region where fluctuations of the detected motion vector due to noise components are conspicuous, namely the integrated motion vector is small. On the other hand, in a region where fluctuations of the detected motion vector due to noise components are not so conspicuous, namely the integrated motion vector is large, elimination of the signal components due to coring processing is minimized such that error of the detected motion vector is lessened. As a result, when a large motion vector has been detected, detection accuracy is not lowered. Meanwhile, when a small motion vector has been detected, minute noise components of the detected motion vector are suppressed. Furthermore, only when the still condition has been established, the coring value is increased irrespective of the integrated motion vector. Therefore, even if correlation in the image becomes close in one direction and thus, the integrated motion vector becomes slightly large, the motion vector can be detected by compressing fluctuations of the motion vector. Fig. 11 shows a still decision unit 40 employed in a motion vector detecting apparatus 100 according to a second embodiment of the present invention. Since other constructions of the motion vector detecting apparatus 100 are the same as the motion vector detecting apparatus 50, description thereof is abbreviated for the sake of brevity. In Fig. 11, the still decision unit 40 includes a memory 33 for storing a part or a whole of an image signal, a subtracter 34, an absolute value converter 35, an accumulative adder 36, a low-pass filter 37 and a decision unit 38. Operation of the still decision unit 40 of the above described arrangement is described, hereinbelow. An image signal inputted to the still decision unit 40 is stored in the memory 33 temporarily and a difference between the image signals of two successive fields or frames is calculated by the subtracter 34. This calculation of the difference is not required to be performed for whole pixcels of the inputted image signals but for a small portion, e.g. 1/3 to 1/5 of the whole pixcels or a minute portion, e.g. 1/30 to 1/50 of the whole pixcels, which is restricted to low-frequency band. Absolute values of the thus obtained differences of the image signals are obtained by the absolute value converter 35 and are accumulatively added by the accumulative adder 36. Thus, a sum of the differences between the successive fields (frames) are obtained. In the case of a still image, the obtained sum is formed by noise components only and therefore, assumes a small value. On the other hand, in the case where an image has changed greatly or a camera has been panned or tilted, the sum assumes a large value. A rapid change portion in the input signal is attenuated by the low-pass filter 37. Then, based on whether an output of the low-pass filter 37 is large or small, the decision portion 38 makes a decision as to whether or not the image signal represents a still image. After a decision as to whether or not the image signal represents a still image has been made as described above, a result of the decision is transmitted to the coring value determiner 21 in the same manner as the motion vector detecting apparatus 50. When the still condition has been established, the coring value is set to a large value in the same manner as the first embodiment and the motion vector is outputted by compressing the fluctuation components of the motion vector. In this embodiment, the difference of the image signals is directly calculated as described above. Thus, even if the camera is panned or tilted slightly, it becomes possible to accurately judge whether or not the image signal represents a still image. Therefore, the motion vector can be detected stably without adversely affecting calculation of the motion vector. An image stabilizer 200 in another aspect of the present invention is described with reference to Fig. 12. The image stabilizer 200 includes the motion vector detecting apparatus 50 of Fig. 7, a memory 41, an interpolator 42 for interpolating an image disposed at an arbitrary position, a vector determiner 44 for determining a motion vector of an image from among prospective motion vectors and an output controller 43 for controlling the memory 41 and the interpolator 42 on the basis of the motion vector of the image. The memory 41 is provided for storing image signals and reading out the image signals disposed at arbitrary positions of pixcels. Operation of the image stabilizer 200 of the above described arrangement is described. In Fig. 12, since the motion vector is obtained in the same method as that of the first embodiment, description thereof is abbreviated for the sake of brevity. Prospective motion vectors of an image signal inputted from the motion vector detecting apparatus 50 are, respectively, obtained from four regions divided vertically and horizontally and a motion vector of the inputted image signal is determined in the vector determiner 44 by a median filter for selecting a median value. Then, the determined motion vector of the image signal is inputted to the output controller 43. In the case where the motion vector is expressed by an address of the memory 41 in which the image signal is stored, the address for signal read-out of the memory 41 is determined by the output controller 43 in accordance with an integral portion of the motion vector. Meanwhile, the output controller 43 controls the interpolator 42 in accordance with a decimal portion of the motion vector. The interpolator 42 performs linear interpolation in which multiplication of a weight corresponding to an inverse number of a distance is performed in both vertical and horizontal directions. Thus, the output controller 43 shifts the image in a direction opposite to that of sway of the image so as to cancel sway of the image. Meanwhile, the output controller 43 magnifies an original image to an output image about 1.1 to 1.5 times as large as the original image and shifts the image in the memory 41 vertically and horizontally such that an edge of the image is invisible when the image is outputted. In prior art image stabilizers, when correlation in the image is close in one direction, fluctuations of the motion vector are large and an outputted image sways despite of input of a still image. However, in the above described arrangement of the image stabilizer 200 by employing the motion vector detecting apparatus of the present invention, even when correlation in the image is close in one direction, fluctuations of the motion vector are compressed to a low level and sway of the outputted image is reduced such that correction of sway of the outputted image is performed stably. Meanwhile, it should be noted that Fig. 10 shows one example of characteristics of the coring value. Thus, the coring value may assume arbitrary characteristics such as that shown by a dotted line in Fig. 10. Furthermore, a method of judging whether or not an image signal represents a still image is not restricted to those of the above embodiments and may also employ, for example, an acceleration sensor, etc. Moreover, in the image stabilizer of the preser invention, the image is magnified and shifted by using the memory such that sway of the outputted image is corrected. However, this method may also be replaced by a method in which read-out of an image pickup element such as a charge coupled device is controlled in place of the memory. As is clear from the foregoing description, in the motion vector detecting apparatus of the present invention even when correlation in the image is close in one direction so as to increase fluctuations of the motion vector, the detected motion vector is subjected to coring processing of proper magnitude in accordance with magnitude of the detected motion vector or still conditions of the image. Thus, in accordance with the present invention, fluctuations of the motion vector, which are minute noise components causing detection error, are restricted to a minimum and reduction of a large motion vector can be minimized such that detection error is lessened. Meanwhile, in the image stabilizer of the present invention, even in the case where fluctuation of the motion vector are large due to poor detection conditions of the motion vector and thus, the outputted image sways despite of input of a still image, fluctuations of the motion vector are compressed to a low level and sway of the outputted image is reduced such that correction of sway of the outputted image is performed stably. Although the present invention has been fully described by way of example with reference to the accompanying drawings, it is to be noted here that various changes and modifications will be apparent to those skilled in the art within the scope of the present invention as defined in the claims.

Claims: A motion vector detecting apparatus (50) for detecting a motion vector of an inputted image signal, comprising: a motion vector arithmetic unit (9) for calculating the motion vector of the inputted image signal; an integrator (11) for integrating the motion vector; a still decision unit (20) for judging whether or not the inputted image signal represents a still image; a coring value determiner (21) for determining a coring value on the basis of outputs from said integrator (11) and said still decision unit (20); and a coring circuit (10) which controls the coring value on the basis of an output from said coring value determiner (21) so as to perform coring processing on an output of said motion vector arithmetic unit (9). A motion vector detecting apparatus (50) as claimed in Claim 1, wherein by using a minimum value of said motion vector arithmetic unit (9), said still decision unit (20) judges whether or not the inputted image signal represents the still image. A motion vector detecting apparatus (100) as claimed in Claim 1, wherein by using the inputted image signal, said still decision unit (40) judges whether or not the inputted image signal represents the still image. An image stabilizer (200) for correcting sway of an outputted image, containing a motion vector detecting apparatus (50) as defined in claims 1, 2 or 3; a vector determiner (44) for determining the motion vector of the inputted image signal in a whole screen; and an output controller (43) which magnifies and interpolates an inputted image so as to output an image at an arbitrary position. An image stabilizer (200) as claimed in claim 4 further including a memory (41) and an interpolator (42). An image stabilizer for correcting sway of an outputted image from a CCD, comprising: a motion vector detecting apparatus (50) as defined in claim 1, 2 or 3; a vector determiner (44) for determining the motion vector of the inputted image signal from a CCD in a whole screen; and an output controller (43) which controls a CCD read-out controller and an interpolator (42) so as to output an image at an arbitrary position.

Inventors: MORIMURA ATSUSHI EKUSERU HAITS, UOMORI KENYA, MORIMURA, ATSUSHI, EKUSERU HAITSU DAI, UOMORI, KENYA
Applicants: MATSUSHITA ELECTRIC IND CO LTD, MATSUSHITA ELECTRIC INDUSTRIAL CO., LTD.
Application year: 1991
Main classification: H04N5
IPC codes: H04N7, H04N5, G06T7