Abstract:
An image stabilization device includes a number of micromechanical actuators, an image recording sensor, and a computational device. The image sense generates image data of an image and is suspendedly mounted to the micromechanical actuators. The computational device receives and processes the image data generated by the image recording sensor. The computational device includes a detection device for detecting shake so as to perform an image analysis on the image data and detect changes in a time sequence of the image data. The detected changes are used to generate control signals for controlling the micromechanical actuators so as to mechanically move the image recording sensor in a manner to counteract the shake and to stabilize the image.

Description:
[0001]     Priority is claimed to German patent application no. DE 10 2004 045 430.2, the entire subject matter of which is hereby incorporated by reference herein.  
         [0002]     The present invention relates to an image stabilization device in general, and in particular to an image stabilization device having means for image analysis and shake compensation.  
       BACKGROUND  
       [0003]     The recording of moving images is becoming more and more popular. Cameras having image sensors with CCD technology (CCD: charged coupled devices) have provided a breakthrough in this field. In particular, such camera systems are becoming increasingly smaller and lighter. This has made it possible to install photo or video cameras in mobile telephones, so that mass use of video telephony, for example, for video conferencing, has come within reach. With the introduction of the code multiplexing method (UMTS), this application in the field of moblie radio communications received a new boost.  
         [0004]     However, the practical implementation involves some difficulties. In order to record one&#39;s own face, mobile telephones are typically held with one hand and with the arm half extended. This position, especially in conjunction with the small size and low weight of the mobile telephone, makes it particularly difficult to hold the mobile telephone still so as to ensure video transmission with little blur. Especially during mobile phone video conferencing, it is a problem when the camera movements and the speaker movements are different and the image is therefore blurred. Even slight, often unconscious movements of the hand or of the head already cause noticeable shaking of the image to be transmitted. However, blurred video transmission significantly reduces the perceived quality.  
         [0005]     Several shake compensation systems are already known from video camera applications.  
         [0006]     Document EP 04 98 730 A2, for example, describes a video signal processing apparatus which can be used in a video camera in conjunction with a device for compensating for vibrations caused by the hand. This apparatus is provided with means that generate a motion vector representing an amount and direction of motion between consecutive frames of a video signal.  
         [0007]     The described apparatus further has means capable of deciding whether or not the determined motion vector represents an unwanted movement of the camera. Also provided is a device for generating correction or control signals as a function of the determined motion vector to allow unwanted movement of the camera to be compensated for.  
         [0008]     However, the motion vector determination is still in need of improvement in terms of the requirements for mobile radio communications and the recording of faces with their specific features.  
         [0009]     U.S. Pat. No. 5,030,984 discloses a device which detects the motion in a scene by sampling the change in intensity of light of consecutive images at each pixel site in the image along with the change in intensity of light of a neighboring pixel site. These changes are then used to calculate both the displacement of the entire image and a directional degree of image change. Finally, a signal representing the total displacement is calculated and used, inter alia, to control the movement of a film in a direction in order to minimize blur in the recorded image.  
         [0010]     Another document, EP 0 956 695 B1, discloses a motion sensor, which performs analysis on a plurality of images and determines a scene-motion parameter therefrom. The calculation unit, in this document also referred to as “motion sensor”, can be part of an image stabilization system. According to the description, the most practical technique is to analyze two sequential, successive fields or frames to determine the scene-motion parameter.  
         [0011]     All of these systems operate fully electronically, so that, typically, part of the potential resolution is sacrificed for the sake of stabilization. This is because, in electronic image stabilization, it is common practice today to use unused edge pixels of the CCD image converter to detect movements of the video or photographer and to subsequently compensate for them mathematically. As in the case of many electronic aids, such as digital zoom or pixel interpolation, this results in a more or less significant loss in quality. This may not be perceived as too great a disadvantage in high-resolution systems.  
         [0012]     But particularly in video telephones, the reasonably transmittable image resolution and repetition rate are significantly lower, in spite of the constantly increasing data transfer rates. Yet, it is especially in the case of mobile telephones that there are, on the one hand, extreme demands on size and, on the other hand, enormous cost pressure, so that the cheapest possible, low-resolution CCD chips are used. In this connection, typical resolutions of a video telephone may be in the range of VGA.  
         [0013]     It is therefore desirable to have an image stabilization system that, other than digital image editing, substantially retains the resolution of the image recording sensor.  
         [0014]     Optical image stabilization is mainly used in binoculars, video cameras and miniature camera lenses. The movements of the user are detected by so-called “gyroscopes”. Gyroscopes are devices capable of detecting motions about the horizontal and vertical axes. Once the camera or the lens system detects the direction in which the pick-up lens is moving, a special refractive lens, which is mostly supported in a liquid, is almost simultaneously moved in exactly the opposite direction, so that camera shake is largely compensated for.  
         [0015]     Document DE 199 42 900 B4, for example, discloses a device for correcting image defects caused by camera shake. Three pairs of acceleration sensors are provided to detect camera shake in the horizontal and vertical directions as well as rotation. The correction signals supplied by the acceleration sensors are fed to multilayer piezoelectric actuators, which are connected to a CCD sensor. Using the multilayer piezoelectric actuators, the CCD sensor can be displaced in the X-direction and in the Y-direction, which disadvantageously allows only limited correction. On the other hand, the use of six acceleration sensors appears to be cumbersome, complex, and space-intensive.  
         [0016]     To compensate for unwanted rotation of the camera body, there is also provided a rotary member, for example, in the form of a rotary shaft, which cooperates with a multilayer piezoelectric actuator. However, a rotary shaft involves a high degree of constructional complexity, is prone to failure, and, at best, poorly suited for applications with tight spatial constraints, such as in mobile telephones.  
         [0017]     Disadvantageously, however, this technology does not provide for compensation for subject or object shake.  
         [0018]     These systems, which have been developed for video cameras, which are large in size compared to a mobile telephone, can generally not be easily applied to a mobile telephone because of the extreme space constraints. Moreover, the stability requirements in mobile telephones are generally higher than in cameras, which are typically handled with more care and attention.  
       SUMMARY OF THE INVENTION  
       [0019]     It is therefore an object of the present invention to provide a simple, inexpensive and compact image stabilization device that uses the resolution of the image recording sensor to the best possible extent without having to resort to mechanical gyro sensors.  
         [0020]     Another, alternative, object of the present invention is to provide an image stabilization device which uses the advantages of image recognition also to compensate for object motion.  
         [0021]     It is a further, alternative, object of the present invention to provide such a device which is suitable, in particular, for installation in hand-holdable mobile video telephones and which can be inexpensively produced in large quantities.  
         [0022]     An additional, alternative, object of the present invention is to provide such a device that is capable of compensating for a long shaking stroke.  
         [0023]     It is yet another, alternative, object of the present invention to provide such a device that avoids or at least reduces the disadvantages of known devices.  
         [0024]     The present invention provides a device for electronic image stabilization of electronic image data or video image data using digital image analysis; the device being able to be used, for example, in hand-holdable mobile video telephones. Such mobile video telephones are understood to include not only classic cell phones, but also the so-called “personal digital assistants” (PDAs), “mobile digital assistants” (MDAs), and other mobile devices which allow for video transmission and which may be prone to shake due to the characteristic of being hand-held by a user.  
         [0025]     The device includes an image recording sensor, preferably a CCD chip, for recording, i.e., generating the video image data, a computational device, such as a digital microcomputer, which receives and processes the digital video image data from the image recording sensor. The computational device further includes detection means which are designed to detect shake in the sequence of video images such that an image analysis and image recognition, in particular face recognition, are performed on the video image data, and changes in the time sequence of the video image data are detected by the detection means. The detection means are designed, in particular, as electronic and/or digital detection means and are preferably implemented by suitably designed hardware or suitably programmed software, for example, image recognition software. This digital electronic image analysis has the advantage of eliminating the need for mechanically complex and sensitive gyro sensors. This makes the device insensitive to failure, and allows it to have very small dimensions. Moreover, it is also possible to detect object motion.  
         [0026]     The detected image changes are used to generate control signals for controlling a compensating device. The compensating device compensates or at least reduces the shake. To this end, the compensating device is provided with a plurality of electro-micromechanical actuators which the image recording sensor is suspendedly mounted to. The control signals are preferably electrical control signals which are used to control the micromechanical actuators in such a manner that the image recording sensor is mechanically moved by the actuators so as to counteract the shake and to thereby stabilize the video image data. Thus, the image recording sensor is automatically and preferably continuously moved in a direction opposite to the shaking movements of the camera and of the object in order to compensate for the shake.  
         [0027]     In comparison with the software compensation often used in the prior art, this mechanical compensation has the advantage that no edge pixels must be sacrificed, so that the full resolution of the CCD chip can be used even when the shake compensation is in operation. In expensive and complex high-resolution video cameras, the loss of a few edge pixels may at first not appear to be too serious to one skilled in the art, but it is a particular disadvantage for use in mobile telephones, because, unlike video cameras, mobile telephones frequently have a lower resolution, for example, in VGA quality, anyway. Of course, the device may also be used for photo or video cameras.  
         [0028]     Preferably, the plurality of actuators includes a first and/or a second group of actuators, or, in particular, is divided into two groups of actuators; the actuators of the first group running transversely or perpendicularly to the actuators of the second group. The preferably linearly acting actuators of the first group are used to tilt the image recording sensor in a direction transverse to the normal of the image plane (optical axis) in order to compensate for tilts of the camera caused by the user and/or by object motion in a direction transverse to the optical axis in two dimensions (X- and Y-directions), i.e., two degrees of freedom. Tilting is particularly suitable for the structural and optical conditions of a video telephone, because the tilting of the image recording sensor advantageously allows for a relatively long stroke, and thus, for effective shake compensation.  
         [0029]     Moreover, the actuators of the second group are used to rotate the image recording sensor about the optical axis of the image recording sensor in order to compensate for corresponding rotations about the optical axis, i.e., with respect to a third degree of freedom.  
         [0030]     This independent control of the actuators of the first and second groups with respect to the three degrees of freedom allows for particularly effective shake compensation.  
         [0031]     Preferably, the actuators of the first group are designed to perform a linear movement along a longitudinal axis and arranged with their longitudinal axes transverse or substantially perpendicular to the image plane of the image recording sensor in order to tilt the image recording sensor in a direction transverse to the normal of the image plane.  
         [0032]     In an embodiment of the present invention, the actuators of the first group are mounted and attached to the back side of the image recording sensor opposite the optically active side. In an embodiment, the actuators of the first group are integrated into the CCD chip on the back side of the CCD chip. This allows for a small size and inexpensive mass production, which in turn is an advantage especially in the highly competitive market for video telephones.  
         [0033]     It has turned out to be advantageous for the first group to consist of exactly three actuators, which form a tripod located on the back side of the image recording sensor. Advantageously, there is no need for disturbing elements on the front side of the sensor, and the tripod allows for easy, two dimensional tilting of the image recording sensor in a direction transverse to the normal of the image plane.  
         [0034]     In accordance with another embodiment of the present invention, the actuators of the second group are designed to perform a linear movement along a longitudinal axis and arranged with their longitudinal axes transverse to the normal of the image plane or substantially parallel to the image plane of the image recording sensor in order to rotate the image recording sensor about the normal of the image plane. Preferably, the actuators of the first and second groups are controlled independently of each other and produce movements which are substantially independent of those of the respective other group.  
         [0035]     The image recording sensor, or CCD chip, preferably defines side surfaces that extend transversely to the image plane, the actuators of the second group being suspendedly mounted to the side surfaces.  
         [0036]     In an embodiment, the image recording sensor has a substantially rectangular shape in its image plane, and the second group consists of four actuators, which are each suspended at the end of a side surface and which extend with their respective longitudinal axes parallel to an adjacent side surface; the actuators of the second group being used to bi-directionally rotate the image recording sensor about the normal of the image plane. To this end, the image recording sensor is preferably located in a housing and is movably and suspendedly mounted within and to the housing via the actuators.  
         [0037]     The above-described arrangement is advantageous because the actuators of the first and/or second groups can be implemented by linearly acting piezoelectric elements which are each directly controlled and driven by a respective electrical control signal.  
         [0038]     The detection means are preferably designed such that they determine differential angles from the time sequence of the video image data, and that a distinction is made between wanted and unwanted movements based on the size of the differential angles and predefined decision criteria. Subsequently, the movements defined as unwanted can be compensated for to the extent possible; the movements defined as wanted remaining uncompensated  
         [0039]     Moreover, the detection means are preferably digital electronic detection means, for example, suitable software that is designed to perform face recognition.  
         [0040]     For face recognition, it is preferred to perform at least recognition of a plurality of prominent facial features selected from the group including the eyes, nose, and mouth of a face. To this end, each recognized facial feature (eyes, nose, mouth, etc.) is assigned a point of reference. Subsequently, each point of reference can be used as a reference point, or a central point, such as a centroid, is calculated from several, in an embodiment three, points of reference and used as a reference point for each individual image. Using this/these reference point(s), the image stabilization performed by continuously tracking the reference point(s) in the video image data.  
         [0041]     In the following, the present invention will be explained in more detail on the basis of exemplary embodiments and with reference to the drawings. Identical or similar parts are partly provided with the same reference numerals, and the features of the various exemplary embodiments may be combined with one another. 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0042]      FIG. 1  is a schematic side view of the device according to a first embodiment of the present invention.  
         [0043]      FIG. 2  shows the device of  FIG. 1  in a tilted position.  
         [0044]      FIG. 3  is a transparent top view of the device of  FIG. 1 .  
         [0045]      FIG. 4  is a transparent top view of the device according to a second embodiment of the present invention.  
         [0046]      FIG. 5  is a transparent top view showing the device of  FIG. 4  in a rotated position;  
         [0047]      FIG. 6  is a schematic representation of a human face.  
         [0048]      FIG. 7  shows the face of  FIG. 6  with a plurality of points of reference.  
         [0049]      FIG. 8  is a block-like diagram of the process sequence for the image stabilization according to the present invention. 
     
    
     DETAILED DESCRIPTION  
       [0050]      FIG. 1  shows an image recording sensor in the form of a CCD chip  10  having three actuators  14 ,  16 ,  18  arranged on and mounted to back side  12  thereof. Actuators  14 ,  16 ,  18  connect CCD chip  10  to a housing wall  20 . Back side  12  is opposite the optically active front side  13  of CCD chip  10 .  
         [0051]     Micromechanical actuators  14 ,  16 ,  18  are formed by piezoelectric elements which each contract or expand along their longitudinal axes  24 ,  26 ,  28 , which are perpendicular to image plane  30  of CCD chip  10 , in order to tilt CCD chip  10  relative to the rest normal  32  of image plane  30 . The integration of piezoelectric actuators  14 ,  16 ,  18  into CCD chip  10  on back side  12 , as well as the attachment of the piezoelectric actuators of the second group to the side surfaces of CCD chip  10  permits inexpensive mass production.  FIG. 1  shows the CCD chip in its rest position, in which it extends parallel to housing wall  20 .  
         [0052]     Referring to  FIG. 2 , CCD chip  10  is shown in a tilted position. In this tilted position, the current normal  34  of image plane  30  forms an angle α with rest normal  32 .  
         [0053]     The tilting is achieved by expansion of linear piezoelectric actuator  14  and simultaneous contraction of linear piezoelectric actuator  18 .  
         [0054]     Referring to  FIG. 3 , the triangular or tripod arrangement of the three piezoelectric actuators  14 ,  16 ,  18  is shown, which is used to effect two-dimensional tilting about the X-axis and the Y-axis, which define the image plane in the rest position.  
         [0055]     Referring to  FIG. 4 , rectangular CCD chip  10  is suspended in a surrounding housing  40  via four further actuators  42 ,  44 ,  46 ,  48  in the form of linear piezoelectric actuators within housing  40 .  
         [0056]     Accordingly, piezoelectric actuators  14 ,  16 ,  18  form a first group of actuators, which are used to tilt CCD chip  10 , and piezoelectric actuators  42 ,  44 ,  46 ,  48  form a second group used to rotate CCD chip  10  about its normal  34 , which, in this view, extends perpendicular to the image plane.  
         [0057]     The piezoelectric actuators of the second group are secured on one side to the ends of sides  10   a ,  10   b,    10   c ,  10   d , respectively, and on the other side to the inner sides of housing  40 . Longitudinal axis  52  of piezoelectric actuator  42  extends parallel to adjacent side surface  10   b . The remaining actuators of the second group are correspondingly arranged at the other sides.  
         [0058]     Referring to  FIG. 5 , CCD chip  10  is shown rotated about its normal  34  of image plane  30  as compared to the view of  FIG. 4 . The rotation is achieved by simultaneous expansion of all four actuators  42 ,  44 ,  46 ,  48  of the second group. This does indeed result in a slight twisting of actuators  14 ,  16 ,  18  of the first group, but does not significantly affect the function of the actuators of the first group.  
         [0059]     As can be seen in  FIG. 5 , independent control of the actuators of the first group and of the second group produces a two-dimensional tilting motion about the X-axis and about the Y-axis, and/or a rotation about the Z-axis formed by normal  34 . Due to the smallness of the tilt and rotation angles, the motion about the three axes mentioned above can be regarded, to a good approximation, as being independently controlled. In particular, both a movement of the object and of the camera can be compensated for by a single control signal per actuator. Because of this, by selective operation or control of the individual piezoelectric actuators, CCD chip  10  can be oriented in almost any desired direction, and thus, the unwanted movement of the camera can be compensated for, and the image stabilization is capable of effectively compensating for a plurality of different shaking movement patterns.  
         [0060]     Referring to  FIG. 6 , a face  60  including eyes  62 ,  64 , a nose  66 , a mouth  68  and two ears  70 ,  72  is schematically shown as an example of an object.  
         [0061]     Referring to  FIG. 7 , several prominent facial elements are each assigned a point of reference  74  by a digital electronic computational device in the form of a microcomputer after a suitable image recognition, which is generally known to one skilled in the art. The computational device, or more precisely, suitably programmed software calculates a polygon, in this example a triangle  76 , from several, preferably three, of the points of reference. In this example, the triangle is formed by the points of reference  74  of the two eyes  62 ,  64 , and of nose tip  66  as particularly prominent facial elements. Subsequently, the centroid  78  of triangle  76  is calculated and established as a reference point for the image stabilization.  
         [0062]     Referring to  FIG. 8 , initially, image  1  of the video image data stream is recorded, and position vector {right arrow over (P)} 1  of reference point  78  is determined. Subsequently, next image  2  of the video image data stream is recorded, and position vector {right arrow over (P)} 2  of reference point  78  is established. A displacement vector {right arrow over (v)} 1  is determined, for example, by simply subtracting the two position vectors {right arrow over (P)} 1  and {right arrow over (P)} 2 . Displacement vector {right arrow over (v)} 1  is read in by a computational and control device  80 . To this end, software which carries out the image recognition and motion tracking schematically shown in  FIG. 6  through  FIG. 8 , is programmed into the mobile terminal device, or more precisely, into computational device  80 . Subsequently, computational and control device  80  generates control signals  82  with which all seven piezoelectric actuators  14 ,  16 ,  18 ,  42 ,  44 ,  46 ,  48 , which together form the mechanical compensating device  84 , are controlled independently of each other in such a manner that, through the thereby produced two-dimensional tilting and one-dimensional rotation of CCD chip  10 , the image of the face is moved and rotated back to the same position as in image  1  to the greatest possible extent and as far as is feasible within the limits of the stroke of the actuators. Exactly one control signal which is independent of the respective other control signals is used for each piezoelectric actuator.  
         [0063]     Advantageously, because of the software-based image stabilization, there is no need for acceleration sensors.  
         [0064]     The stabilization device is installed, for example, in mobile terminal devices, in particular, DECT-, GSM-, or UMTS-standard mobile video telephones, and is part of an otherwise known camera system.  
         [0065]     Since the image recognition software is advantageously capable of detecting even very small movements, it is easily possible to distinguish conscious from unconscious movements. This is also done electronically using software.  
         [0066]     This allows movements with small differential angles to be interpreted by the computational device as unwanted movements, and thus, as shake, and to be compensated for using the micromechanical stabilization method described above. Movements with large differential angles are interpreted by the computational device as conscious movements and are therefore not compensated for.  
         [0067]     Once the software detects a movement that is interpreted as being unwanted, CCD chip  10 , which records the image, is appropriately re-oriented to compensate for the movement of the camera.  
         [0068]     It will be apparent to one skilled in the art that the above-described embodiments are meant to be exemplary, and that the present invention is not limited thereto, but can be varied in many ways without departing from the spirit thereof.