Abstract:
A real-time low frame-rate video compression system and method that allows the user to perform face-to-face communication through an extremely low bandwidth network. At the encoder side, the system is able to automatically select only a few good faces from the original sequence with high visual quality and compress and transmit them. At the decoder side, the system use image-morphing based rendering method to generate a normal frame-rate video. Experimental results show that the system is superior to more traditional video codecs for low bit-rate face-to-face communication.

Description:
BACKGROUND 
   1. Technical Field 
   The invention is related to video conferencing, and in particular, to a system and method for very low frame rate video streaming for face-to-face videoconferencing that employs novel eye tracking and blink detection techniques, as well as using image cropping and morphing to reduce frame rates. 
   2. Related Art 
   Face-to-face video communication is a potentially important component of real time communication systems. Inexpensive cameras connected to devices ranging from desktop computers to cell phones enable video conferencing in a variety of modes such as one-to-one and multi-party conferences. 
   Most video teleconference solutions are specifically designed for broadband networks and cannot be applied to low bandwidth networks. Previous face video compression techniques are not able to efficiently operate at very low bit rates because they compress and transmit the entirety of every video frame. Thus, reducing the bandwidth will of necessity degrade the image in every frame. There is a minimum for the allocated bits for each frame below which conventional compression techniques cannot produce visually acceptable results. Multi-party video conferences put an added strain on bandwidth requirements since multiple video streams need to be simultaneously transmitted in order for all of the participants to participate. 
   Different approaches have been proposed to reduce the bandwidth requirements for streaming video, such as the MPEG-4 face animation standard and H.26x video coding [1]. By taking advantage of face models, the MPEG-4 face animation standard can achieve a high compression ratio by sending only face model parameters. However, it is difficult to make the synthesized faces look natural and match the original video. H.26x waveform-based coding techniques are fully automatic and robust, but are not efficient for low bit-rate face video since their generality does not take advantage of any face models. These two types of techniques are combined together in a recently proposed low bit-rate face video streaming system [2], where prior knowledge about faces are incorporated into traditional waveform-based compression techniques to achieve better compression performance. This system is, however, not able to operate efficiently at very low bit rates (e.g., on the order of 8 kb/s). 
   Therefore, what is needed is a system and method that can provide face-to-face video conferencing at very low bit rates with natural looking results. Additionally, this system and method should be able to provide face-to-face video conferencing in real time. 
   SUMMARY 
   The very low bit rate video conferencing system and method of the invention produces a very low bit rate result by trading frame rate to maintain visual quality. This system and method is particularly appropriate for transmitting imagery of persons who are not currently speaking, although the same methods apply to all participants of a video conference. 
   The very low bit rate video conferencing system and method employs a low frame rate face video transmission strategy, which can, in real-time compress and transmit face video at very low bit-rates, such as 8 Kb/s. A main contribution of the very low bit rate video conferencing system and method of the invention is the novel compression strategy: instead of compressing and transmitting every frame at a very low quality, the very low bit rate video conferencing system and method reduces the video frame-rate and sends only important regions of carefully selected frames. The very low bit rate video conferencing system and method employs a real-time face tracking method at the encoder to provide a simple means to clip out the most relevant portion of the video. The very low bit rate video conferencing system and method further uses a template matching-based “good face finder” to select only a few good frames from the face video for transmission. A regular sampling of frames will often result in one in which the person has their eyes closed, or worse yet, half closed. We have all experienced this when taking photographs. In a multiple frame per second video stream it is fine to include blinks, but at only one frame every two seconds, such a frame is quite disturbing. The good frames are selected based on two criteria: (1) they should be visually good faces, (e.g., eyes open); and (2) they should carry as much information as possible (e.g., the face). The encoder then compresses and transmits only these good frame regions. At the decoder, the good frames are rendered by an image morphing method to create a normal frame-rate face video. The morphing between frames keeps the face “alive” even though no new information is being received between the temporally sparse frames. 
   The very low bit rate video conferencing system and method can be used individually in very low bandwidth networks, or as a complement to existing video conferencing systems. In a teleconference involving a group of people, each person&#39;s face will be captured and transmitted to others. Since there is generally only one speaker at a time, the very low bit rate video conferencing system and method can transmit the face of the speaker with a higher frame-rate, high quality video while transmitting all the listeners using low frame-rate video to save overall network bandwidth. 
   In the encoding process, an input video frame is first processed to locate the face. The very low bit rate video conferencing system and method then tracks the positions of eyes, to be used both for face evaluation and motion compensation. The current frame is evaluated and it is decided whether or not it is a good one. If it is, a feature-based image morphing is performed to align this frame to the previously transmitted frame. The current frame is subtracted from the previous frame and the residual is sent to a standard video encoder. The very low bit rate video conferencing system and method then transmits the compressed frame along with control parameters such as time stamps and face feature positions. 
   In the decoder, the residual is decoded and added to the previous decoded frame. To recover the aligned image the transmitted control parameters are used to morph the face to its original location. The very low bit rate video conferencing system and method puts the new frame into the frame buffer, and renders the current displayed frame by image morphing between consecutively received good frames. 
   It is noted that in this section and the remainder of this specification, the description refers to various individual publications identified by a numeric designator contained within a pair of brackets. For example, such a reference may be identified by reciting, “reference [1]” or simply “[1]”. A listing of the publications corresponding to each designator can be found at the end of the Detailed Description section. 

   
     DESCRIPTION OF THE DRAWINGS 
     The file of this patent or application contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the U.S. Patent and Trademark Office upon request and payment of the necessary fee. 
     The specific features, aspects, and advantages of the present invention will become better understood with regard to the following description, appended claims, and accompanying drawings where: 
       FIG. 1  is a general system diagram depicting a general-purpose computing device constituting an exemplary system wherein the very low bit rate video conferencing system and method of the invention can be practiced. 
       FIG. 2  illustrates an exemplary flow diagram of the encoding of the very low bit rate video conferencing system and method. 
       FIG. 3  illustrates an exemplary flow diagram of the decoding of the very low bit rate video conferencing system and method. 
       FIG. 4  shows examples of good faces (a and b) and not so good faces (c and d). Considering the importance of eye contact in face-to-face communication, the very low bit rate video conferencing system and method only transmits faces with open eyes. 
       FIG. 5A  illustrates an exemplary flow diagram of the creation of the eye templates employed by the very low bit rate video conferencing system and method. 
       FIG. 5B  illustrates an exemplary flow chart of the real-time eye tracking employed by the very low bit rate video conferencing system and method. 
       FIG. 6  depicts (a) An original left eye template; (b) Three feature patches computed from (a); From left to right: grayscale image, horizontal edge map and vertical edge map; and (c) Three 1D features are created by projecting the feature patches to horizontal or vertical axis. 
       FIG. 7(   a ) depicts the maximum correlation values on matching the left eye template to each frame.  FIGS. 7(   b ) and ( c ) show face tracking and eye blink detection examples. Red boxes are face tracking box and green boxes are cropping windows. Blue dots in (b) indicate detected open eyes and the absence of blue dots in (c) indicates closed eyes. 
       FIG. 8A  illustrates an exemplary flow diagram of the image morphing employed in the very low bit rate video conferencing system and method. 
       FIG. 8B  depicts an exemplary morph mesh. 
       FIG. 9  illustrates an exemplary flow diagram of the image morphing for rendering employed in the very low bit rate video conferencing system and method. 
       FIG. 10  depicts examples of morphing-based rendering.  FIGS. 10  ( a ) and ( d ) depict the first and the second decoded good faces.  FIGS. 10(   b ) and ( c ) depict two synthesized intermediate faces created by image morphing and cross dissolving.  FIGS. 10(   e ) and ( f ) depict two synthesized intermediate faces created only by cross dissolving. 
       FIG. 11  shows Top: 5 out of 30 frames in the final video with good frame selection. Bottom: the corresponding set of frames in the final video without good frame selection. It clearly suggests that image morphing is very useful to achieve a better motion compensation in low frame-rate case. 
       FIG. 12  depicts (a) One example frame. (b) Compressed by H.264 video codec (off-line) with a limited bit-rate at 12.4 Kb/S. Note that the visual quality of face is poor. (c) Using low frame-rate compression system, the very low bit rate video conferencing system and method uses only ⅓ bandwidth of dial-up connection to transmit high quality faces that look alive. 
   

   DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
   In the following description of the preferred embodiments of the present invention, reference is made to the accompanying drawings, which form a part hereof, and in which is shown by way of illustration specific embodiments in which the invention may be practiced. It is understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the present invention. 
   1.0 Exemplary Operating Environment: 
     FIG. 1  illustrates an example of a suitable computing system environment  100  on which the invention may be implemented. The computing system environment  100  is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the invention. Neither should the computing environment  100  be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment  100 . 
   The invention is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the invention include, but are not limited to, personal computers, server computers, hand-held, laptop or mobile computer or communications devices such as cell phones and PDA&#39;s, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. 
   The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer in combination with hardware modules, including components of a microphone array  198 . Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices. With reference to  FIG. 1 , an exemplary system for implementing the invention includes a general-purpose computing device in the form of a computer  110 . 
   Components of computer  110  may include, but are not limited to, a processing unit  120 , a system memory  130 , and a system bus  121  that couples various system components including the system memory to the processing unit  120 . The system bus  121  may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus also known as Mezzanine bus. 
   Computer  110  typically includes a variety of computer readable media. Computer readable media can be any available media that can be accessed by computer  110  and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, or other data. 
   Computer storage media includes, but is not limited to, RAM, ROM, PROM, EPROM, EEPROM, flash memory, or other memory technology; CD-ROM, digital versatile disks (DVD), or other optical disk storage; magnetic cassettes, magnetic tape, magnetic disk storage, or other magnetic storage devices; or any other medium which can be used to store the desired information and which can be accessed by computer  110 . Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above should also be included within the scope of computer readable media. 
   The system memory  130  includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM)  131  and random access memory (RAM)  132 . A basic input/output system  133  (BIOS), containing the basic routines that help to transfer information between elements within computer  110 , such as during start-up, is typically stored in ROM  131 . RAM  132  typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit  120 . By way of example, and not limitation,  FIG. 1  illustrates operating system  134 , application programs  135 , other program modules  136 , and program data  137 . 
   The computer  110  may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only,  FIG. 1  illustrates a hard disk drive  141  that reads from or writes to non-removable, nonvolatile magnetic media, a magnetic disk drive  151  that reads from or writes to a removable, nonvolatile magnetic disk  152 , and an optical disk drive  155  that reads from or writes to a removable, nonvolatile optical disk  156  such as a CD ROM or other optical media. Other removable/non-removable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. The hard disk drive  141  is typically connected to the system bus  121  through a non-removable memory interface such as interface  140 , and magnetic disk drive  151  and optical disk drive  155  are typically connected to the system bus  121  by a removable memory interface, such as interface  150 . 
   The drives and their associated computer storage media discussed above and illustrated in  FIG. 1 , provide storage of computer readable instructions, data structures, program modules and other data for the computer  110 . In  FIG. 1 , for example, hard disk drive  141  is illustrated as storing operating system  144 , application programs  145 , other program modules  146 , and program data  147 . Note that these components can either be the same as or different from operating system  134 , application programs  135 , other program modules  136 , and program data  137 . Operating system  144 , application programs  145 , other program modules  146 , and program data  147  are given different numbers here to illustrate that, at a minimum, they are different copies. A user may enter commands and information into the computer  110  through input devices such as a keyboard  162  and pointing device  161 , commonly referred to as a mouse, trackball, or touch pad. 
   Other input devices (not shown) may include a joystick, game pad, satellite dish, scanner, radio receiver, and a television or broadcast video receiver, or the like. These and other input devices are often connected to the processing unit  120  through a wired or wireless user input interface  160  that is coupled to the system bus  121 , but may be connected by other conventional interface and bus structures, such as, for example, a parallel port, a game port, a universal serial bus (USB), an IEEE 1394 interface, a Bluetooth™ wireless interface, an IEEE 802.11 wireless interface, etc. Further, the computer  110  may also include a speech or audio input device, such as a microphone or a microphone array  198 , as well as a loudspeaker  197  or other sound output device connected via an audio interface  199 , again including conventional wired or wireless interfaces, such as, for example, parallel, serial, USB, IEEE 1394, Bluetooth™, etc. 
   A monitor  191  or other type of display device is also connected to the system bus  121  via an interface, such as a video interface  190 . In addition to the monitor, computers may also include other peripheral output devices such as a printer  196 , which may be connected through an output peripheral interface  195 . 
   The computer  110  may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer  180 . The remote computer  180  may be a personal computer, a server, a router, a network PC, a peer device, or other common network node, and typically includes many or all of the elements described above relative to the computer  110 , although only a memory storage device  181  has been illustrated in  FIG. 1 . The logical connections depicted in  FIG. 1  include a local area network (LAN)  171  and a wide area network (WAN)  173 , but may also include other networks. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets, and the Internet. 
   When used in a LAN networking environment, the computer  110  is connected to the LAN  171  through a network interface or adapter  170 . When used in a WAN networking environment, the computer  110  typically includes a modem  172  or other means for establishing communications over the WAN  173 , such as the Internet. The modem  172 , which may be internal or external, may be connected to the system bus  121  via the user input interface  160 , or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer  110 , or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation,  FIG. 1  illustrates remote application programs  185  as residing on memory device  181 . It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used. 
   The exemplary operating environment having now been discussed, the remaining part of this description will be devoted to a discussion of the very low bit rate video conferencing system and method of the invention. 
   2.0 The Very Low Bit Rate Video Conferencing System and Method: 
   The very low bit rate video conferencing system and method can be used individually in very low bandwidth networks, or as a complement to existing video conferencing systems. In a teleconference involving a group of people, each person&#39;s face will be captured and transmitted to others. Since there is generally only one speaker at a time, the very low bit rate video conferencing system and method can transmit the face of the speaker with a higher frame-rate, high quality video while transmitting all the listeners using low frame-rate video to save overall network bandwidth. 
   The following paragraphs discuss the details of the very low bit rate video conferencing system and method of the invention. 
   2.1 The Encoder 
   The encoding process of the very low bit rate video conferencing system and method is shown in  FIG. 2 . A video frame is input, as shown in process action  202 . The input video frame is first processed to locate the face (process action  204 ). If no face is found, the very low bit rate video conferencing system and method does not process this frame any further (process actions  206 ,  208 ). If a face is found the very low bit rate video conferencing system and method tracks the positions of the eyes to be used both for face evaluation and motion compensation (process actions  206 ,  210 ). The face and eye tracking, and face evaluation are described in detail in Section 3.0. The very low bit rate video conferencing system and method then evaluates the frame and decides whether or not it is a good one (process action  212 ,  214 ). If it is, a feature-based image morphing is performed to align this frame to the previously transmitted frame (process action  216 ). The current frame is subtracted from the previous frame (process action  218 ) and the residual is sent to a standard video encoder (process action  220 ). The low bit rate video conferencing system and method then transmits the compressed residual along with control parameters such as time stamps and face feature positions, as shown in process action  222 . 
   2.2 The Decoder 
   A flow chart of the decoding process of the low bit rate video conferencing system and method is shown in  FIG. 3 . The decoder receives the encoded residual, as shown in process in  302 . The very low bit rate video conferencing system and method decodes the received residual and adds it to the previous decoded frame to recover the aligned image (process action  304 ). The transmitted control parameters are used to morph the face to its original location (process action  306 ). The new frame is put into the frame buffer (process action  308 ), and the current displayed frame is rendered by image morphing between consecutive received good frames (process action  310 ). 
   3.0 Finding Good Faces 
   The very low frame rate setting provides the freedom to choose which frame to transmit. For example, if the camera operates at 15 fps, but one wishes to transmit only one frame every 2 seconds, then one has up to 30 frames to choose from (however, in practice one may wish to limit the choice to minimize latency). Since each frame will be seen for 2 seconds, it becomes critical to select “good” frames. Choosing the specific features that distinguish a “good” frame from a “bad” one is somewhat objective. In an informal study many frames were examined and their quality was judged. It was originally hypothesized that some aspects of the eyes and mouth would correlate with the subjective judgment. In fact, only how open the eyes are had a significant correlation with “goodness.” Examining  FIG. 4 , one sees a typical set of frames from a video sequence. The openness of the eyes decreases from (a) to (b) to (c) to (d) and so did the subjective of opinion about the “goodness” of the frames. This is consistent with psychological research results [3]. No other aspect of the face had a consistent impact. If one randomly selects frames to transmit, one will frequently encounter blinking, or half-blinking eyes, as shown if  FIG. 4 . To avoid such frames, the very low bit rate video conferencing system and method of the invention employs a face feature-based eye blink detection algorithm, which distinguishes between open and closed eyes. 
   3.1 Real-Time Eye Tracking. 
   Face tracking has been extensively used for efficient face video compression [4, 5]. The very low bit rate video conferencing system and method begins with the efficient face detection algorithm proposed in [6] to locate a rectangular face box containing the face. For robustness, the low bit rate video conferencing system and method employs a template matching based method for both eye tracking and blink detection. For each video conference participant, the pupil positions are manually indicated on the first frame with wide open eyes (process action  502 ). The user is also asked to indicate the mouth corners for later morphing (process action  504 ). Two image patches are extracted at the pupil positions as templates, one for each eye, (process action  506 ), an example of which is shown in  FIG. 6(   a ). 
   On each frame, the very low bit rate video conferencing system and method iteratively matches the templates to possible locations and select the best matching positions as the eye locations. For real-time detection and tracking, for each input frame (process action  508 ), the detector scans only in the neighborhood of the face location in the previous frame to find a face box (process action  510 ). If a face is not found, the face is no longer processed (process action  512 ). Given a face box is found, the very low bit rate video conferencing system and method searches the upper part of the box for eyes and extracts two image patches, one for each eye (process action  514 ). Eyes from different people or under different illumination conditions may have significantly different appearances. For efficiency and robustness to illumination changes, the very low bit rate video conferencing system and method uses image feature based matching instead of directly comparing image patches. As shown in  FIG. 6(   b ), for an image patch, the very low bit rate video conferencing system and method computes its corresponding grayscale image, and horizontal and vertical edge maps created with Sobel filters (process action  516 ). By summing columns the low bit rate video conferencing system and method further projects the image patches to the horizontal axis (for the grayscale image and vertical edge map) or to the vertical axis by summing rows (for the horizontal edge map) to produce three 1D signals (process actions  518  and  520 ). The similarity between the eye template image patch and the extracted candidate image patch from the input frame is computed as the weighted sum of the correlations between corresponding 1D signals for each eye (process action  522 ). Mathematically, the three 1D signals for the left eye template T L  are denoted as G i   T     L   ,i=1, . . . , X L  (for grayscale image), H i   T     L   ,i=1, . . . , Y L  (for horizontal edge map) and V i   T     L   ,i=1, . . . , X L  (for vertical edge map), where X L  and Y L  are the width and height of the template. For a candidate image patch I, the three corresponding signals are denoted as G i   I , H i   I  and V i   I . The correlation between the two image patches is computed as
 
 S   P ( T   L   ,I )= w   G   ·S ( G   T     L     ,G   I )+ w   H   ·S ( H   T     L     ,H   I )+ w   v   ·S ( V   T     L     ,V   I )  (1)
 
where w G , w H  and w ν  are predefined weights. In one exemplary embodiment these predefined weights are set to be 0.4, 0.3 and 0.3, respectively.
 
   S(A,B) is the signal correlation function computed as 
                   S   ⁡     (     A   ,   B     )       =       ∑     i   =   1     L     ⁢       (       a   i     ⁢     b   i       )     /         ∑     i   =   1     L     ⁢       a   i   2     ⁢       ∑     i   =   1     L     ⁢     b   i   2                         (   2   )               
where L is the length of the signal. Equation 2 describes the function on the right hand side of equation 1. In Equation 2, A and B are two arrays, A=a 1 ,a 2 , . . . , a L , B=b 1 ,b 2 , . . . , b L  where a i  and b i  are elements in the two arrays.
 
   3.2 Eye Blink Detection 
   The advantage of template-matching based eye tracking is that it not only gives the best possible locations for the eyes, but also tells how well the templates match to these locations, indicated by the computed correlation values. Since the low bit rate video conferencing system and method uses open eyes as templates, when the eyes are blinking, the correlation values dropped significantly.  FIG. 7(   a ) shows the maximum correlation values on matching the left eye template to each frame of an example video. More particularly, for each frame the correlation values are compared to a threshold. Blinks can be clearly seen at fairly regular intervals and they can be easily detected by applying a threshold on the correlation value computed from each frame. If the correlation values are greater than the threshold, the eyes are determined to be open. In one working embodiment of the very low bit rate video conferencing system and method of the invention the threshold is set to be 0.6. 
     FIG. 7  shows some examples on face tracking and eye blink detection. The blue dots in  FIG. 7(   b ) indicates open eyes detected by the system, while the absence of blue dots in  FIG. 7(   c ) means that the system determines that the eyes are closed or half-closed. 
   3.3 Good Frame Selection. 
   Good frames for transmission are selected as follows. For each good frame F G   i , there is a time stamp t G   i . Good frames are selected from the original sequence based on the following criteria:
 
 t   min   ≦t   G   i   −t   G   i−1   ≦t   max ,  (1)
 
where t min  and t max  are parameters determining how frequently one wants to pick out good frames and essentially, the frequency of the good frames determines the required bandwidth of the transmitted video. The variables t min  and t max  can be user defined.
 
   (2) Both the face tracker and eye blink detector give positive results (e.g., the face-tracker gives a positive result when the frame contains a face and the eye blink detector gives a positive result when the eyes are open), which ensures the selected face has good visual quality (e.g., the eyes are open). 
   In cases that the user is temporally away from the camera, which means the second criterion cannot be satisfied, in on embodiment, the system sends a random frame every t max  time to keep the transmitted video alive. More specifically, in the time interval [t min , t max ], the system will search for a frame which satisfies the two criteria. However, if none of the frames in this time interval satisfies the two criteria, the system will randomly choose one frame.
 
4.0 Compression and Rendering of Faces
 
   4.1 Improved Motion Compensation 
   Selected good faces are compressed before transmission. The frames containing the good faces can be compressed using a standard video codec. Since the good faces are sparsely selected from the original video, the frame difference is typically larger than a high frame rate system making standard motion compensation less efficient. The face and eye tracking used to select frames can also inform the compression subsystem. First, by only transmitting the face region the very low bit rate video conferencing system and method avoids redundant transmission of the background. Second, the face tracking approximately aligns subsequent frames, significantly reducing the size of the interframe difference. Finally, by applying an image morph [7] that aligns the eye and mouth positions the difference between subsequent frames is further reduced. In one embodiment, the view area is limited to the area surrounding the head. However, it is also possible to send the background once or very infrequently. 
   As shown in  FIG. 8A , process actions  802  and  804 , to compensate for motion between frames, the very low bit rate video conferencing system and method calculates the differences of eye locations between two adjacent frames, and estimates the location of the mouth corners based on the initial marked frame (process action  806 ). From this the very low bit rate video conferencing system and method creates a target morph mesh (process action  808 ) as in [7].  FIG. 8B  depicts a morph mesh. To build the mesh, the image is segmented into attached triangle patches. By moving the positions of some vertices, a deformed mesh is created. The original image patches are mapped into the deformed triangles to get a deformed image. The very low bit rate video conferencing system and method morphs each frame to the immediately previously received frame and subtracts the result to obtain a residual image (process action  810 ). The morph is created by moving corresponding vertices (e.g., eye corners, mouth corners) to the positions from the last frame. The pixels within the triangles are carried along with the change in vertex positions using barycentric coordinates. In addition to coding the final residual the very low bit rate video conferencing system and method also transmits the eye and mouth locations so the process can be decoded (process action  812 ). 
   4.2 Image Morphing for Rendering. 
   As shown in  FIG. 9 , the decoder performs the opposite operations. The residual is decoded (process action  902 ) and added to the previous frame (process action  904 ). This new frame is then unmorphed based on the transmitted feature positions to reproduce the most current frame (process action  906 ). One now has the option of simply switching from the old frame to the new one. However, this would produce a jump in the viewed video. To avoid a jarring jump, one can cross dissolve from the old to new but this produces ghostlike blurry images in the transition, as shown in  FIGS. 10(   e ) and ( f ). A better solution is to use the morph information to smoothly transform the old frame geometry to the new one while also performing the cross dissolve as was done in [7]. In more detail, we may wish to smoothly morph from frame t to frame t+5 with images I(t) and I(t+5) and with sets of feature points F(t) and F(t+5). To create an in-between frame I(t+2) from these two frames, we first linear interpolate F(t) and F(t+5) to get F(t+2). (The assumption is F(t) and F(t+5) is the same set of feature points with different locations). Then we morph I(t) to I a (t+2) based on F(t) and F(t+2), and morph I(t+5) to I b (t+2) based on F(t+5) and F(t+2). The final image is created by I(t+2)=⅗·I a (t+2)+⅖·I b (t+2).) In this way one can create a new video in the same frame-rate as the original video captured at the encoder (process action  1000 ).  FIG. 10  shows two decoded frames and the synthesized intermediate frames. 
   5.0 Results 
   5.1 Good Face Finding. 
   As can be seen in  FIG. 11 , the very low bit rate video conferencing system and method is able to select only good frames to transmit. The figure shows 5 of 30 frames selected for transmission using the good frame selection criteria and by simply regularly sampling the video. The other 25 frames all looked good both with and without the selection criteria. It clearly shows that by using good frame selection, the transmitted video looks more natural, and facilitates important social-cognitive effects. 
   5.2 Compression. 
   Table 1 shows the compression result of the system on a sample video with different settings of t min  and t max , which control the desired frame rate. Note that the very low bit rate video conferencing system and method only requires a very low bit-rate to transmit a semi-alive video. The last row of the table shows the compression result of the codec of the very low bit rate video conferencing system and method without using image morphing. 
   
     
       
             
           
             
             
             
             
           
         
             
               TABLE 1 
             
           
           
             
                 
             
             
               Compression results of the low frame-rate system 
             
           
        
         
             
                 
               Codec 
               Configurations 
               Bit-Rate 
             
             
                 
                 
             
             
                 
               MPEG 2 
               640*480, 30f/s, good quality 
               322Kb/s 
             
             
                 
               H.264 
               640*480, 30f/s, lowest quality 
               12.4Kb/2  
             
             
                 
               Low-frame rate 
               240*280, t min  = 1, t max  = 3 
                7.4Kb/s 
             
             
                 
               Low-frame rate 
               240*280, t min  = 2, t max  = 4 
                3.8Kb/s 
             
             
                 
               Low-frame rate 
               240*280, t min  = 2, t max  = 4 
                5.4Kb/s 
             
             
                 
               w/o morphing 
             
             
                 
                 
             
           
        
       
     
   
   H.264 also achieves a low bit-rate compression, but the visual quality of the compressed video is significantly worse spatially, as shown in  FIG. 12 . However, to be fair, it should be noted that the increased temporal frame rate does help depict a person&#39;s motion better. Further user studies will be needed to fairly examine the trade-offs. Informally, the very low bit rate video conferencing system and method provides a better experience at a lower bit rate for viewing listeners, but may lose some important semantic meaning in the motion of speakers to maintain the better visual quality and lower bit rate. More importantly, the very low bit rate video conferencing system and method performs the face/eye tracking and encoding in real-time as compared with the H.264 encoder which took about 30 minutes to encode a ten second video. Also, the very low bit rate video conferencing system and method uses only about ⅓rd of the bandwidth of a slow dial-up line and thus is a good choice particularly when there are multiple participants in a video conference. 
   The foregoing description of the very low bit rate video conferencing system and method has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. Further, it should be noted that any or all of the aforementioned alternate embodiments may be used in any combination desired to form additional hybrid embodiments. It is intended that the scope of the invention be limited not by this detailed description, but rather by the claims appended hereto. 
   REFERENCES 
   
       
       [1] G. Cote, B. Erol, M. Gallant, F. Kossentini. 263+: Video coding at low bit rates. IEEE Trans. Circuit and Systems for Video Technology, vol. 8, no. 7, 1998. 
       [2] Z. Wen, Z. C. Liu, M. Cohen, J. Li, K. Zheng, T. Huang. Low Bit-rate Video Streaming for Face-to-face Teleconference. In Proc. IEEE Int. Conf. Multimedia and Expo, 2004. 
       [3] M. Garau, M. Slater, S. Bee, M. A. Sasse. The impact of eye gaze on communication using humanoid avatars. In Proc. SIGCHI Conf. Human factors in computing systems, 2001. 
       [4] W. Vieux, K. Schwerdt, J. Crowley. Facetracking and coding for video compression In Proc. of the First Int. Conf. on Computer Vision Systems, 1999. 
       [5] J. Crowley, F. Berard. Multi-modal tracking of faces for video communication. In Proc. IEEE Conf. Computer Vision and Patt. Recog, 1997. 
       [6] S. Z. Li, X. L. Zou, et al. Real-time multi-view face detection, tracking, pose estimation, alignment, and recognition. In IEEE CVPR Demo Summary, 2001. 
       [7] T. Beier, S. Neely. Feature-Based Image Metamorphosis. Computer Graphics, 26(2), 1992.