Abstract:
A system and method for streaming whiteboard content to computing devices in a networked environment. The invention is an extension of whiteboard image generation technology to provide network-based collaboration of a target meeting. In one aspect, each networked client can receive audio content and whiteboard content (video images). In another aspect, each networked client can transmit audio content and annotation content which is displayed separately or generated on the whiteboard image. The streaming content is built on external collaboration frameworks.

Description:
BACKGROUND  
         [0001]    This application claims priority under 35 U.S.C. Section 119(e)(1) of provisional application No. 60/449,683, filed Feb. 24, 2003.  
           [0002]    1. Technical Field  
           [0003]    This invention is directed toward a system and method for method for streaming whiteboard content to computing devices in a networked environment. More specifically, the invention is directed toward a system and method for sending a sequence of enhanced whiteboard images to one or more remote meeting participants in real-time to allow network-based collaboration for all meeting participants during a meeting  
           [0004]    2. Background Art  
           [0005]    Meetings constitute a large part of many workers&#39; working time. Making more efficient use of this time spent in meetings and the time and money spent traveling to and from meetings translates into a big increase in productivity and large cost savings.  
           [0006]    Many meeting scenarios use a whiteboard extensively for brainstorming sessions, lectures, project planning meetings, patent disclosures, and so on. Note-taking and copying what is written on the board often interferes with many participants&#39; active contribution and involvement during these meetings. As a result, some efforts have been undertaken to capture whiteboard content in some automated fashion.  
           [0007]    Every business depends on the free flow of information and ideas to improve their products and services. Effective collaboration between people both inside and outside an organization increases product quality, improves product or project development lead times, and reduces costs. However, effective collaboration is often difficult. One issue that hinders effective meetings is that often people are not physically co-located. Traveling to meetings can be very time-consuming when considering lengthy travel times, and can be expensive. For a two or three hour meeting people often will travel a day to the meeting and a day back, especially if the meeting is across country or located in another country or on another continent.  
           [0008]    Hence, there exists a great need to improve the ease of having a meeting with many participants that are not physically co-located that allows such remote participants to participate in the meeting in real time and share their ideas.  
         SUMMARY  
         [0009]    The invention is directed toward a system and method for streaming whiteboard content to computing devices in a networked environment. The invention, termed the Real-time Whiteboard Streaming system and method is an extension of whiteboard image generation technology to provide network-based collaboration during a meeting. In one embodiment of the invention, each networked client can receive audio content and enhanced whiteboard content (video images or a sequence of snapshot images) of the meeting. In another embodiment, each networked client can transmit audio content and annotation content that is displayed on a separate display from the actual whiteboard or is generated on the actual physical whiteboard itself. The streaming content is built on external collaboration frameworks, such as, for example, Microsoft® Corporation&#39;s, Windows Messenger and Net Meeting. An enhanced live video of the whiteboard is streamed to the personal computers (PCs) of remote meeting participants, and in turn; they can send back comments and annotations. The resulting system makes a natural collaborative tool for distributed meetings.  
           [0010]    A typical scenario for this system involves a brainstorming session involving a number N of people gathered in one location, normally a typical meeting room, and M (typically 1 to 3) individuals (remote persons), each of which may be in their own offices or other remote locations.  
           [0011]    The N people gather in a meeting room equipped with a whiteboard; a camera capturing the whiteboard content; a microphone device capturing meeting audio; a meeting server; a projector, television or other display, connected to the meeting server; and a loudspeaker, preferably mounted on the microphone device. Each of the M individuals sits in his or her own office or other remote location equipped with a desktop computer or notebook computer; a speaker; a microphone; and a network connection, which connects the remote person&#39;s computer to the meeting server.  
           [0012]    The N people in the meeting room use the whiteboard as the collaborative space to share their ideas by drawing on the whiteboard. The whiteboard content is captured quickly, in virtually real time, and is shared with the remote persons by transferring enhanced images of the whiteboard to them over the network. The verbal discussion is captured by the microphone, and the audio, preferably with directional information (e.g., with audio tracking using an audio fingerprint), is also sent to the remote persons via this means.  
           [0013]    The remote persons participate in the meeting through audio, which is played on the loudspeaker in the meeting room. The remote persons can also participate in the meeting by annotating a captured whiteboard frame, and the annotated whiteboard frame is shown on the display in the meeting room, and also on other remote persons&#39; computer. Additionally, a signal such as a buzzer or some visual cue may be used to alert the meeting participants that a remote participant has provided input to the whiteboard content.  
           [0014]    The whole meeting, including the annotations, may be archived for future viewing. The whiteboard and annotations are time-stamped, and are therefore synchronized with the audio.  
           [0015]    As discussed previously, the remote participants can provide annotations and verbal comments on the whiteboard content. The display in the meeting room, which is different than the whiteboard itself, shows both the whiteboard content and the annotations. An alternative scenario is to project the annotations of the remote participants directly projected onto the actual whiteboard in the meeting room. The advantage of this scenario is that the remote participants actively participate in the development of whiteboard contents.  
           [0016]    In the above scenarios, conventional PCs are used by the remote participants. However, the remote participants could alternately use Tablet PCs. Microsoft&#39;s® Tablet PC is a design for a fully-equipped personal computer that allows a user to take notes using natural handwriting on a stylus or digital pen-sensitive touch screen instead of requiring the use of a keyboard. The ink technology makes it much easier for the remote participants to make annotations—either public and/or private.  
           [0017]    As discussed above, the Real-Time Whiteboard Streaming system and method has a server-based side and client-based side. In the following paragraphs these will be discussed in more detail.  
           [0018]    The server-based side of the Real-Time Whiteboard Streaming system and method generally consists of four primary parts: 1) a classification procedure that classifies cells of an image sequence as background, foreground or stroke cells; 2) a dynamic whiteboard background initialization and update procedure that computes the whiteboard color without the foreground objects (such as people) and pen strokes; 3) an efficient real time procedure that enhances the whiteboard region in the input video sequence; and 4) an analysis procedure that extracts the newly appeared strokes on the whiteboard.  
           [0019]    The system and method according to the invention initially acquires a series of snapshots or a live whiteboard video input composed of a real-time sequence of image frames of the whiteboard. This snapshot sequence or video is processed in real-time on a frame-by-frame basis.  
           [0020]    The sequence of the whiteboard image frames, either snapshots or video, are input into a cell-based motion detection process. In this process each image frame in a given sequence of frames is divided into cells. Each corresponding cell location is compared over time to detect any changes in lighting or color. If there are significant lighting changes, the whiteboard color matching process action is performed, which will be discussed in further detail later. If there are no significant lighting changes, each of the cells of the video frame is classified as foreground, whiteboard background or stroke cells. In cell classification, the idea is to filter out portions of the image of a person or other object obstructing portions of the whiteboard, such that only the strokes of whiteboard content are visible on the whiteboard background.  
           [0021]    Cell classification is also used to create a whiteboard video stream that is an enhanced live video stream with the whiteboard and foreground objects such as a person standing in front of the whiteboard displayed. Another data stream, a whiteboard and stroke only data stream, is also created wherein only the strokes on the whiteboard are displayed on the whiteboard in an enhanced manner without foreground objects.  
           [0022]    To display an enhanced image that includes the whiteboard and the foreground objects, image enhancement is performed. In the image enhancement process, the whiteboard color is made more uniform and the stroke saturation of the whiteboard content (what is written on the whiteboard) is increased to make the strokes more vivid and legible.  
           [0023]    Another output of the cell classification is the determination as to whether strokes have been added or subtracted from the whiteboard. In this process action, the Real-time Whiteboard Streaming system and method determines whether or not the stroke content in a cell increases or decreases (i.e., corresponding to additional writing or erasures of whiteboard content). If there are changes in the stroke content of a cell these changes to content are output to the whiteboard data stream.  
           [0024]    Additionally, after cell classification, the system determines whether or not the whiteboard color model should be updated. The whiteboard color model update processing identifies gradual changes in the whiteboard color. For example, these minor changes in lighting conditions may be due to clouds passing in front of the sun or something casting a shadow on the whiteboard. If gradual lighting changes are identified the current whiteboard color model in the whiteboard color model database is updated.  
           [0025]    As discussed previously, if significant light changes are noted in the cell-based motion detection process, the system and method according to the invention determines whether or not the noted light changes are embodied in a current whiteboard color model. If the new lighting condition corresponds to a whiteboard color model in the whiteboard color model database, this model is used for the subsequent cell classification. However, if the color model is not found a new color model is initialized or estimated and added to the whiteboard color model database.  
           [0026]    The system and method according to the invention has many advantages. It allows remote meeting participants to actively participate in a meeting without the time and expense of traveling to a remote site. It provides a more legible version of a data stream of the whiteboard with foreground objects, as well as a more legible data stream of the whiteboard content without foreground objects.  
           [0027]    In addition to the just described benefits, other advantages of the present invention will become apparent from the detailed description which follows hereinafter when taken in conjunction with the drawing figures which accompany it. 
       
    
    
     DESCRIPTION OF THE DRAWINGS  
       [0028]    The file of this patent contains at least one drawing executed in color. Copies of this patent with color drawing(s) will be provided by the U.S. Patent and Trademark Office upon request and payment of the necessary fee.  
         [0029]    The specific features, aspects, and advantages of the invention will become better understood with regard to the following description, appended claims, and accompanying drawings where:  
         [0030]    [0030]FIG. 1 is a diagram depicting a general purpose computing device constituting an exemplary system for implementing the invention.  
         [0031]    [0031]FIG. 2 is a diagram of the primary system components of the Real-time Whiteboard Streaming system and method.  
         [0032]    [0032]FIG. 3 is a diagram of the primary functional components of the Real-time Whiteboard Streaming system and method.  
         [0033]    [0033]FIG. 4 is a general flow diagram of the system and method according to the invention.  
         [0034]    [0034]FIG. 5 is a diagram of a series of input images.  
         [0035]    [0035]FIG. 6 is a general flow diagram of the cell classification of whiteboard image cells used by the system and method according to the invention.  
         [0036]    [0036]FIG. 7 is a general flow diagram of the whiteboard color estimation procedure used by the system and method according to the invention.  
         [0037]    [0037]FIG. 8 is a general flow diagram of the whiteboard image enhancement of the system and method according to the invention.  
         [0038]    [0038]FIG. 9 is a general flow diagram of stroke identification and extraction of the system and method according to the invention.  
         [0039]    FIG. A- 1  is a diagram used to explain the whiteboard color estimation and filtering of the system and method according to the invention. 
     
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS  
       [0040]    In the following description of the preferred embodiments of the present invention, reference is made to the accompanying drawings that form a part hereof, and in which is shown by way of illustration specific embodiments in which the invention may be practiced. It is understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the present invention.  
       1.0 Exemplary Operating Environment  
       [0041]    [0041]FIG. 1 illustrates an example of a suitable computing system environment  100  on which the invention may be implemented. The computing system environment  100  is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the invention. Neither should the computing environment  100  be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment  100 .  
         [0042]    The invention is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the invention include, but are not limited to, personal computers, server computers, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.  
         [0043]    The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.  
         [0044]    With reference to FIG. 1, an exemplary system for implementing the invention includes a general purpose computing device in the form of a computer  110 . Components of computer  110  may include, but are not limited to, a processing unit  120 , a system memory  130 , and a system bus  121  that couples various system components including the system memory to the processing unit  120 . The system bus  121  may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus also known as Mezzanine bus.  
         [0045]    Computer  110  typically includes a variety of computer readable media. Computer readable media can be any available media that can be accessed by computer  110  and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computer  110 . Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the any of the above should also be included within the scope of computer readable media.  
         [0046]    The system memory  130  includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM)  131  and random access memory (RAM)  132 . A basic input/output system  133  (BIOS), containing the basic routines that help to transfer information between elements within computer  110 , such as during start-up, is typically stored in ROM  131 . RAM  132  typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit  120 . By way of example, and not limitation, FIG. 1 illustrates operating system  134 , application programs  135 , other program modules  136 , and program data  137 .  
         [0047]    The computer  110  may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only, FIG. 1 illustrates a hard disk drive  141  that reads from or writes to non-removable, nonvolatile magnetic media, a magnetic disk drive  151  that reads from or writes to a removable, nonvolatile magnetic disk  152 , and an optical disk drive  155  that reads from or writes to a removable, nonvolatile optical disk  156  such as a CD ROM or other optical media. Other removable/non-removable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. The hard disk drive  141  is typically connected to the system bus  121  through anon-removable memory interface such as interface  140 , and magnetic disk drive  151  and optical disk drive  155  are typically connected to the system bus  121  by a removable memory interface, such as interface  150 .  
         [0048]    The drives and their associated computer storage media discussed above and illustrated in FIG. 1, provide storage of computer readable instructions, data structures, program modules and other data for the computer  110 . In FIG. 1, for example, hard disk drive  141  is illustrated as storing operating system  144 , application programs  145 , other program modules  146 , and program data  147 . Note that these components can either be the same as or different from operating system  134 , application programs  135 , other program modules  136 , and program data  137 . Operating system  144 , application programs  145 , other program modules  146 , and program data  147  are given different numbers here to illustrate that, at a minimum, they are different copies. A user may enter commands and information into the computer  110  through input devices such as a keyboard  162  and pointing device  161 , commonly referred to as a mouse, trackball or touch pad. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit  120  through a user input interface  160  that is coupled to the system bus  121 , but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB). A monitor  191  or other type of display device is also connected to the system bus  121  via an interface, such as a video interface  190 . In addition to the monitor, computers may also include other peripheral output devices such as speakers  197  and printer  196 , which may be connected through an output peripheral interface  195 . Of particular significance to the present invention, a camera  163  (such as a digital/electronic still or video camera, or film/photographic scanner) capable of capturing a sequence of images  164  can also be included as an input device to the personal computer  110 . Further, while just one camera is depicted, multiple cameras could be included as an input device to the personal computer  110 . The images  164  from the one or more cameras are input into the computer  110  via an appropriate camera interface  165 . This interface  165  is connected to the system bus  121 , thereby allowing the images to be routed to and stored in the RAM  132 , or one of the other data storage devices associated with the computer  110 . However, it is noted that image data can be input into the computer  110  from any of the aforementioned computer-readable media as well, without requiring the use of the camera  163 .  
         [0049]    The computer  110  may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer  180 . The remote computer  180  may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer  110 , although only a memory storage device  181  has been illustrated in FIG. 1. The logical connections depicted in FIG. 1 include a local area network (LAN)  171  and a wide area network (WAN)  173 , but may also include other networks. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.  
         [0050]    When used in a LAN networking environment, the computer  110  is connected to the LAN  171  through a network interface or adapter  170 . When used in a WAN networking environment, the computer  110  typically includes a modem  172  or other means for establishing communications over the WAN  173 , such as the Internet. The modem  172 , which may be internal or external, may be connected to the system bus  121  via the user input interface  160 , or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer  110 , or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation, FIG. 1 illustrates remote application programs  185  as residing on memory device  181 . It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.  
         [0051]    The exemplary operating environment having now been discussed, the remaining parts of this description section will be devoted to a description of the program modules embodying the invention.  
       2.0 System and Method for Real—Time Whiteboard Streaming  
     2.1 General Overview  
       [0052]    The Real-time Whiteboard Streaming system and method generally has a server-based component and can have one or more remote clients, arranged in a network-based environment. Enhanced live video or an enhanced real time sequence of snapshots of the whiteboard is streamed to the PCs of remote meeting participants, and in turn, they can send back annotations on the whiteboard content, as well as verbally commenting on the meeting proceedings.  
         [0053]    As shown in FIG. 2, a typical scenario for this system involves a brainstorming session involving a number N people in one location such as a typical meeting room  202 , and M (typically 1 to 3) remote persons  204   a  . . .  204   m  each in their own offices or other remote locations.  
         [0054]    The N people gather in a meeting room  202  equipped with a whiteboard  206 ; a camera capturing the whiteboard content  208 ; a microphone  210  capturing meeting audio; a meeting server; a projector, television or other display  212 , connected to the meeting server  214 ; and a loudspeaker  216 , preferably mounted on the microphone device. Each of the M individuals sits in his or her own office or other remote location  204   a  . . .  204   m  equipped with a desktop or notebook computer  218 ; a speaker (not shown); a microphone/headset  220 ; and a network connection  222 , which connects the remote person&#39;s computer  218  to the meeting server  214 .  
         [0055]    The N people in the meeting room  202  use the whiteboard  206  as the collaborative space to share their ideas by drawing on the whiteboard. The whiteboard content is captured quickly, and is shared with the remote persons by transferring enhanced whiteboard content to their computers  218   a  . . .  218   m . The verbal discussion is captured by the microphone  210 , and the audio, preferably with directional information, is sent to the remote persons.  
         [0056]    The remote persons participate in the meeting through audio, which is played on the loudspeaker  216  in the meeting room. The remote persons can also participate in the meeting by annotating a captured whiteboard frame, and the annotated whiteboard frame is shown on the display  212  in the meeting room, and also on other remote persons&#39; computers  218   a  . . .  218   m . A signal, such as a beeper, buzzer or visual alert such as a flashing display can be used to alert the meeting participants that a remote participant has made an annotation.  
       2.2 Server-based Real-Time Whiteboard Streaming System and Method  
       [0057]    In general, as shown in FIG. 3, the Real-Time Whiteboard Streaming system and method functionally consists of four primary parts: 1) a classification procedure that classifies image cells of an image sequence as background, foreground or stroke cells (process action  302  ); 2) a dynamic whiteboard background initialization and update procedure that computes the whiteboard color without the foreground objects (such as people) and pen strokes (process action  304  ); 3) an efficient real time procedure that enhances the whiteboard region in the input image sequence (process action  306  ); and 4) an analysis procedure that extracts the newly appeared strokes on the whiteboard (process action  308 ). The output of the image enhancement procedure is an enhanced whiteboard image sequence (process action  310 ), while the output of the stroke analysis procedure is streaming whiteboard data only (process action  312 ).  
         [0058]    More specifically, as shown in FIG. 4, the system and method according to the invention initially acquires a live whiteboard input composed of a real-time sequence of image frames of the whiteboard, as shown in process action  402 . This video or sequence of snapshots or photographs is processed in real-time on a frame-by-frame basis.  
         [0059]    A sequence of the whiteboard image frames are input into a cell-based motion detection process  404 . In this process each image frame in a given sequence of frames is divided into cells. Each corresponding cell location is compared over time to detect any changes in lighting (process action  406 ). If there are significant lighting changes, the whiteboard color matching process action is performed (process action  408 ), which will be discussed in further detail later. If there are no significant lighting changes, each of the cells of the video frame are classified as foreground, whiteboard background or stroke cells (process action  410 ). In cell classification, the idea is to filter out portions of the image of a person or other object obstructing portions of the whiteboard, such that only the strokes of whiteboard content are visible on the whiteboard. Cell classification is also used to create a whiteboard image stream which is an enhanced live video or snapshot stream with the whiteboard and foreground objects such as a person standing in front of the whiteboard displayed, as shown in process action  414 . The stream is enhanced via an enhancement procedure, as shown in process action  412 . Another data stream, a whiteboard data stream, is also created wherein only the strokes on the whiteboard are displayed in an enhanced manner without foreground objects (process action  418 ), after stroke processing (process action  416 ) has been performed.  
         [0060]    To display an enhanced image that includes the whiteboard and the foreground objects, image enhancement is performed as shown in process action  412 . In the image enhancement (process action  412 ), the whiteboard color is made more uniform and the stroke saturation of the whiteboard content (what is written on the whiteboard) is increased to make the strokes more vivid and legible.  
         [0061]    Another output of the cell classification is the determination as to whether strokes have been added or subtracted from the whiteboard, as shown in process action  416 . In this process action, the Real-time Streaming Whiteboard Streaming system and method determines whether or not the stroke content in a cell increases or decreases. If there are changes in the stroke content of a cell this content is output to the whiteboard data stream (process action  418 ).  
         [0062]    Additionally, as shown in process action  420 , after cell classification the system determines whether or not the whiteboard color model should be updated. The whiteboard color model update processing identifies gradual changes in the whiteboard color due to, for example, minor changes in lighting conditions due to, for example, clouds passing in front of the sun or an object casting a shadow on the whiteboard. If changes are identified the current whiteboard color model in the whiteboard color model database  422 , a database of all whiteboard color models available, is updated (process action  420 ).  
         [0063]    As discussed previously, if significant light changes are noted (process action  406 ), the system and method according to the invention determines whether or not the noted light changes are embodied in a current whiteboard color model (process action  408 ). If the new lighting condition corresponds to a whiteboard color model in the whiteboard color model database (process action  424 ), this model is used for the subsequent cell classification. However, if the color model is not found a new color model is initialized (process action  426 ) and added to the whiteboard color model database.  
         [0064]    The general system and method according to the invention having been described, the next paragraphs provide details of the aforementioned process actions.  
         [0065]    2.2.1 Live Whiteboard Video or Snapshot Sequence Input.  
         [0066]    The system and method according to the invention initially acquires a live whiteboard video input or series of snapshots composed of a real-time sequence of image frames of the whiteboard. This video or snapshot series is processed essentially in real-time on a frame-by-frame basis. Some selected frames are shown in FIG. 5 from a sample video sequence. How many of the frames of a sequence are processed depends to some extent of the availability of the meeting server&#39;s  214  Central Processing Unit (CPU). Some, but not all, of the input frames are used for the subsequent whiteboard background color processing, based on CPU availability. The real-time processing of the foreground and stroke calculations and enhancement processing used to create the enhanced whiteboard video stream and whiteboard data stream take precedence over the whiteboard background calculations when the meeting server&#39;s processing bandwidth is constrained.  
         [0067]    2.2.2 Cell-based Motion Detection and Cell Classification.  
         [0068]    Cell classification identifies the cells as either 1) foreground; 2) the whiteboard background or 3) a stroke written on the whiteboard. The Real-Time Whiteboard Streaming system and method computes the blank whiteboard color from the input image sequence. The difficulty in this procedure is that the entire whiteboard may not be visible in any single frame, primarily due to obstructions in front of the whiteboard. Hence, the whiteboard background has to be constructed by first classifying the parts of the image frame as foreground or background. Only the background parts are used to compute the whiteboard background color.  
         [0069]    In one working embodiment of the system and method according to the invention, the foreground/background decision is performed for image blocks of 16×16 pixels, called cells. In general, two primary heuristics are relied to perform the cell classification: 1) Since the camera and the whiteboard are stationary, the whiteboard background cells are stationary throughout the sequence; 2) Although there are sometimes foreground objects (e.g., a person standing in front of the whiteboard) obscuring the whiteboard, the cells that belong to the whiteboard background are typically the majority.  
         [0070]    Thus, in one embodiment of the invention, shown in FIG. 6, an image frame cell is input (process action  602 ). The image of a cell is compared against the image of the same cells (e.g., the cells in the same location) in previous frames, as shown in process action  604 . At each frame, all the cells that have been stationary for more than N frames (4 in one working embodiment of the Real-time Whiteboard Streaming system and method) are considered to be the background candidates and fed to the Whiteboard Color Model Update module (process action  610 ). If the cell age is not greater than the age threshold (4 in one embodiment) the cell is classified as foreground cell, as shown in process action  608 , and the age of the cell is reset to 1. If there is any significant difference in the cell color, the cell is classified as a foreground cell (process action  610 ). The difference test is that the Y, U, V channels of cell color have to be within 15, 5, and 5 intensity levels from their counterparts in the whiteboard color respectively. With respect to the foreground cells, to verify that these cells are not misclassified, an additional test is performed to determine whether cells connected to other foreground cells (process action  612 ). If a group of foreground cells are isolated, their classifications are reverted and the classification process continues. The definition of isolation in one working embodiment of the invention is within a 5×5 cell neighborhood, there are less than 6 foreground cells. If the cell in question is not previously determined to be a foreground cell by the aforementioned tests, it is determined whether the cell contains edges, as shown in process action  614 . If the cell does contain edges, it is designated as a stroke cell (process action  616 ). If the cell does not contain edges, it is classified as a whiteboard background cell (process action  618 ).  
         [0071]    2.2.3 Whiteboard Color Estimation and Whiteboard Color Model Update.  
         [0072]    The whiteboard color model update (process action  420 ) looks for gradual changes in the whiteboard background color. In the most general sense, the Real-time Whiteboard Streaming system and method tests to see if there is a gradual change by determining the average color of each of the cells. If the average color in the cells is almost the same as before, then it is determined that a gradual change has occurred. The system in one embodiment applies the same test as it does in determining whether the cell is a background cell or not. When such a gradual change is noted, an existing whiteboard color model is updated, instead of a new whiteboard color model being created. Only the color model for cells that are background cells or strokes are updated in the existing whiteboard color model. Foreground cells are not updated.  
         [0073]    More specifically, as shown in FIG. 7, in one working embodiment of the invention the image of the whiteboard is divided into cells, as shown in process action  702 . The cell size should preferably be roughly the same as what the size of a single character on the whiteboard is expected to be (in one working embodiment of the invention this was 16 by 16 pixels). Since the ink absorbs the incident light, the luminance of the whiteboard pixels is higher than pen stroke pixels. The whiteboard color within the cell is therefore the color with the highest luminance. In practice, the colors of the pixels in the top 10th percentile are averaged in order to reduce the error introduced by sensor noise. Hence, the color of each cell is computed by first sorting the (e.g. 16×16=256) pixels in brightness and then taking the average of the top 10% values, as shown in process action  706 . The resulting cell colors are used as inputs to a least-media-square error algorithm, which fits a global plane over the colors and throws away the cells that contain outlier colors (the foreground colors), as shown in process action  708 . The remaining cells are considered as background cells and their colors are used to update the whiteboard background. In order to fill the holes created by the cells that are obscured by foreground objects, the cells with known colors are also propagated to the neighboring cells without the colors. These remaining cell colors are integrated with the previously computed cell colors using the color estimation and filtering technique outlined in Appendix A.  
         [0074]    To perform an update of the whiteboard color model the Real-time Whiteboard Streaming system and method takes a greater percentage of the original color model (e.g., 90%) and a smaller percentage of the new color for background or strokes (e.g. 10%).  
         [0075]    2.2.4 Significant Lighting Changes.  
         [0076]    As discussed above, significant lighting changes are considered (process action  406 ). For instance, if a light is turned off in the meeting room almost all of the cells in a given image of the whiteboard will change. If there is a significant lighting change, the cell-based motion detection (process action  404 ) will report that most of the cells are changed (e.g., 95% of the cells is the threshold used in one working embodiment of the invention to signify a significant change in the lighting). Then the whiteboard background is reset and a new color model is initialized or extracted from the whiteboard color model database. The procedure of cell classification and so on is then started over again.  
         [0077]    2.2.5 Image Enhancement.  
         [0078]    The goal of white balancing or color enhancement is to transform the input whiteboard image into an image with the same pen strokes on uniform background (usually white). For each pixel, the color value=C light , the stroke pen color=C pen , and the whiteboard background color=C wb . Since the whiteboard is physically built to be uniformly colored, it can be assumed that C wb  is constant for all the pixels. Hence, the lack of uniformity in the input image is due to different amounts of incident light to each pixel. Therefore, the first procedure in white balancing or color enhancement is to estimate C light , for each pixel, the result of which is in fact an image of the blank whiteboard, as shown in FIG. 7, process action  702  and discussed previously under the section on whiteboard color estimation.  
         [0079]    Once the blank whiteboard color is calculated, it can be used to make the whiteboard region of the input video sequence uniformly white and the color of the pen strokes can be saturated as well. As shown in FIG. 8, once the image of the blank whiteboard is computed (process action  802 ), the input image is color enhanced in two steps:  
         [0080]    1. Make the background uniformly white (process action  804 ). For each cell, the computed whiteboard color (equivalent to the incident light C light ) is used to scale the color of each pixel in the cell:  
         C   out     =       min        (     1   ,       C   input       C   light         )       .                           
 
         [0081]    2. Reduce the image noise and increase the color saturation of the pen strokes. The value of each color channel of each pixel is remapped according to an S-shaped curve: 0.5-0.5 cos(C out   p π). The steepness of the S-curve is controlled by p. In one working embodiment of the invention, p is set to 0.75 (process action  806 ).  
         [0082]    2.2.6 Stroke Identification and Extraction  
         [0083]    The strokes at the cell block level are also extracted to be output as the Whiteboard Data Stream. The strokes have the following properties: 1) they are stationary; 2) they have some edges. As shown in FIG. 9, in order to identify the edges and identify straight lines, two Sobel filters [-1-2-1; 0 0 0; 1 2 1] and [-1 0 1; -2 0 2; -1 0 1] are run on each incoming frame (process actions  902  to  906 ). The first Sobel filter is run to identify the horizontal edges, and the second Sobel filter is run to identify the vertical ones. This process identifies the edgels that exist in the input images. An edgel is defined to be the sum of the absolute values of the corresponding pixels in the two filtered images. For each cell, it has to be stationary for N frames (4 in one working embodiment of the present invention) and contain an edgel whose value is greater than a threshold (60 in one working embodiment of the present invention) to be considered a stroke cell. These edgels of the stroke cells define the edges of the stroke, as shown in process action  908 .  
         [0084]    2.2.7 Whiteboard Color Model Matching  
         [0085]    When the system is started or a lighting change event is detected, N frames are acquired (e.g. 4 frames), and the color of cells that are stationary during the period are compared with each of the whiteboard background images in the database. If the stationary cells are all matched to the corresponding cell colors in one of the images in the database (using the YUV color model and 15,5,5 for difference test), then there is a match and the system proceeds to initialize the remaining non-stationary cells with the colors from the matched image. Once the background color updating procedure stabilizes, the matching is again performed against the database. If there is no match, the current color image is added to the database.  
       2.3 Client-based Real-Time Whiteboard Streaming System and Method  
       [0086]    As discussed previously, the verbal discussion in the meeting room is captured by the microphone, and the audio, preferably with directional information, is sent to the remote persons. An enhanced whiteboard image stream and a whiteboard data stream is sent to the remote participants as discussed previously. The remote participants can chose to receive one or the other or both real time data streams.  
         [0087]    The remote persons participate in the meeting through audio, which is played on the loudspeaker in the meeting room and by annotating a captured whiteboard frame. The remote participants annotations can, for example, consist of using a mouse, or other input device, to add text, circle items written on the whiteboard or draw figures or other markings. The annotations the remote participant makes are sent over the network to the meeting server using any suitable real-time communications protocol and displayed. Only the annotations, not the whole image of the whiteboard need to be sent to the meeting server since the whiteboard information is already resident. This is advantageous in that the network bandwidth required to send the annotation data is very small.  
         [0088]    2.3.1 Annotation to a Separate Display.  
         [0089]    The annotated whiteboard frame is shown on a display in the meeting room. This display is separate from the actual whiteboard in the meeting room, but contains the content of the whiteboard and the annotations of the remote participant. The annotations of a remote participant are also displayed on other remote persons&#39; desktops. A signal, such as a buzzer or flashing screen draws the attention of the meeting participants and remote participants to the annotations a remote participant makes to the whiteboard.  
         [0090]    2.3.2 Annotation Proiected on Actual Whiteboard.  
         [0091]    Alternately, the annotations of a remote participant can be displayed on the actual physical whiteboard in the meeting room. This can be done by projecting the annotations of the remote clients onto the actual whiteboard using a projector.  
       2.4 Annotation Using a Table PC  
       [0092]    In the above scenarios, conventional PCs are used by the remote participants. However, the remote participants could alternately use Tablet PCs. Microsoft&#39;s® Tablet PC is a design for a fully-equipped personal computer that allows a user to take notes using natural handwriting on a stylus or digital pen-sensitive touch screen instead of requiring the use of a keyboard. The ink technology makes it much easier for the remote participants to make annotations—either public and/or private.  
       2.5 Archiving  
       [0093]    The whole meeting, including the annotations, may be archived for future viewing. The whiteboard and annotations are time-stamped, and are therefore synchronized with the audio. Thus, meeting participants can review the meeting at their leisure. Additionally, people who were unable to attend the meeting can view the meeting at a later date.  
         [0094]    The foregoing description of the invention has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teaching.