Abstract:
A collaborative online tele-operation system allows an audience of many participants to simultaneously share control of a single remote actor, such that the actions of the actor are based on the ongoing collective preferences of the audience.

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
     The application claim priority to the benefit of U.S. Provisional Patent Application Ser. No. 60/283,303 filed on Apr. 12, 2001, the entire contents of which are incorporated by reference herein. 
    
    
     STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT 
     The U.S. Government has a paid-up license in this invention and the right in limited circumstances to require that the patent owner to license others on reasonable terms as provided for by the terms of Grant No. IIS-0113147 by the National Science Foundation. 
    
    
     FIELD OF THE INVENTION 
     The invention relates generally to remote control of activity and more specifically to collaborative audience control and participation. 
     BACKGROUND OF THE INVENTION 
     Although so-called “reality” programs and “webcams” have captured an increasing amount of popular interest, these offer little opportunity for direct interaction with the audience. Indeed, while attempts to facilitate audience participation in television programming have a long history, such efforts have typically been based on illusion. Thus, even programming that involves some degree of audience participation is not truly collaborative—that is, the action the audience sees cannot currently be determined, in real time, by the collective preferences of the audience. 
     Efforts to involve the audience to a greater degree have included so-called “tele-operation” and “tele-robotics” systems, in which the members of the audience compete for control of the actor in the remote environment. These systems generally permit only a single member of the audience at a time to issue commands to the actor. Once the command is completed, the issuer of the command voluntarily or involuntarily gives up control and the members of the audience compete for the chance to issue the next command to the actor. 
     Accordingly, there exists a need for a system that supports collaborative control of the actor by multiple members of the audience. 
     SUMMARY OF THE INVENTION 
     The present invention provides for real-time “telepresence” that translates collaborative audience preferences into actions that the audience can perceive. In one aspect, the invention supports a “Tele-Actor”—i.e., a mechanical device such as a robot or a biological organism such as a skilled human equipped with cameras, microphones, and wireless communication systems who moves through and interacts with a remote environment. First-person video and audio is transmitted to a base station and then broadcast over the Internet to a number (e.g., tens, hundreds, or thousands) of “Tele-Directors” online. Tele-Directors not only view, but interact with each other and with the remote environment by sending motion and action requests back to the Tele-Actor by clicking on their web browsers. Requests are treated as motion or action votes and are processed at the base station to provide a single stream of commands, which are then conveyed to the Tele-Actor, who responds accordingly. 
     The group of online Tele-Directors thus collaborates rather than competes for access. The present invention allows large groups of individuals to share in remote experiences. For example, groups of students may collaboratively steer a Tele-Actor through a working steelmill in Japan or through the presidential inauguration, around a newly active volcano or through the streets of Nairobi. 
     In another aspect, the invention relates to a system for facilitating real-time remote participation in an activity performed by an actor in a remote environment by members of the audience. The environment can be real (i.e., a physical environment) or virtual (i.e., computationally created). The system includes clients for use by the audience and an aggregator in communication with the client. Typically each member of the audience has his or her own client. The client receives the progress of the actor from an aggregator and displays it the members of the audience. The client also receives commands from the members of the audience (i.e., the Tele-Directors) related to the progress of the activity. The aggregator, in turn, receives the commands from the clients and processes them to generate a consensus command, which it forwards to the actor. 
     The client can be a computational device (e.g., a computer or personal digital assistant (PDA)), and includes a display. An interface that includes, for example, a question area, a chat area, and a voting area is displayed at the client to facilitate interaction among the members of the audience and voting upon a command related to the progress of the activity of the actor. 
     In another aspect, the invention can be used in educational and journalism applications. Groups of Tele-Directors collaborate to control a resource (e.g., a mechanical device such as a camera or robot). The input from the Tele-Directors is combined to generate a control stream for the resource. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The invention is pointed out with particularity in the appended claims. The advantages of the invention may be better understood by referring to the following description taken in conjunction with the accompanying drawing in which: 
         FIG. 1  is a block diagram of an embodiment of a system in accordance with the principles of the present invention; 
         FIG. 2  is a block diagram of an embodiment of a client and the aggregator shown in  FIG. 1 ; and 
         FIGS. 3A-3E  are embodiments of a voting interface displayed at the client. 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     With reference to  FIG. 1 , a system  10  for facilitating remote participation in an activity monitorable by an audience includes a series of clients  20 A,  20 B,  20 C,  20 D,  20 E (referred to generally as client  20 ) in communication with an aggregator  30  through a first network  40 A. An aggregator  30  can include a server  50  in communication with a base station  60  through a second network  40 B. Alternatively, the server  50  and the base station  60  can be a single computer. The aggregator  30 , or alternatively the base station  60 , is in communication with an actor  70  or a controllable resource. Networks  40 A,  40 B may be, for example, different computer or telecommunications networks or the Internet. 
     In one embodiment, the server  50  can be a computer including an AMD K7 950 MHz processor with 1.2 gigabytes of memory connected to a 100 megabytes per second T3 line. The base station  60  can be a Dell laptop computer including a Pentium III 600 MHz processor with 64 megabytes of memory connected to a 10 megabytes per second T1 line. The server  50  establishes a connection with the base station  60  though the second network  40 B via a socket. The base station  60  can include a card interface, e.g., a USB video card. 
     The actor  70  can be equipped with an apparatus (not shown) for capturing and transmitting the sites and sounds of the remote environment of the actor  70 . The apparatus can, for example, include a Swann MicroCam wireless video camera that provides a 2.4 GHz analog RF output and transmits line-of-sight up to approximately 300 feet with a resolution of 380 horizontal lines. 
     The clients  20  preferably include a custom Internet browser interface based on DHTML. The base station  60  preferably includes image selection interface software. The base station  60  captures images from the actor  70  and attaches textual questions to the images, which are transmitted to the server  50  for distribution to the clients  20 . The server  50  maintains a database of the questions and images and communicates with the client  20 . 
     The base station  60  communicates with the actor  70  via a wireless interface (e.g., IEEE 802.11 or Bluetooth). Typically the actor  70  is located in an environment remote from the audience and clients  20 . In other words, the clients  20  are geographically distributed relative to the environment of the actor  70 . The Tele-Directors share control of the actor  70  from their Internet browsers running on the client  20 . 
     In operation, the Tele-Directors view and monitor the activity of the actor  70  at the clients  20 . As the actor  70  moves through the remote environment, video images are captured at the base station  60  and streamed back to the server  50  for distribution as, for example, .jpg images to the clients  20  along with audio signals received from the actor  70  over a wireless microphone or a cellular phone. Alternatively, real-time video (and, if desired, audio) is streamed to the clients  20 . The Tele-Directors respond to questions embedded within the images relating to the progress of the actor  70 . The responses are collected by the server  50  and aggregated to generate a consensus command, which is forward through the second network  40 B to the base station  60 . In one embodiment, a base station operator (not shown) relays the consensus command to the actor  70  though a wireless audio channel. Alternatively, the actor  70  may be equipped with wireless communication device, such as a PDA, which receives the consensus command directly from the base station  60 . The wireless communication device used by the actor  70  can facilitate viewing of an interface (not shown). The interface facilitates sending information relating to the remote environment for monitoring by the Tele-Directors, and receiving the consensus command. Optionally, the interface allows the actor  70  to request a vote from the Tele-Directors and provide feedback about the command to the Tele-Directors. 
     With reference to  FIG. 2 , in one embodiment of the client  20  and the aggregator  30 , the client  20  includes a first applet  80 A and a second applet  80 B. The applets are received from the aggregator  30  through the network  40 A (e.g., the Internet) via a socket and execute as running processes. The aggregator  30  includes a first web server  50 A, a second web server  50 B, a video card  90 , and an actor control module  100 . The video card  90  is in communication with a camera  110  that monitors the activity of the actor  70 . The actor control module  100  is in communication with the actor  70 . The aggregator can be a single computer or a plurality of computers (e.g., two) each performing specific functions. 
     In one embodiment, the web server  50 A runs software for providing the video images or live video. The video card can provide either full motion capture, at, for example, 30 frames per second, or single still image captures. The driver for the video card  90  facilitates configuration of the resolution rates, color resolution, hue, contrast, color, and brightness. 
     The web server  50 B handles HTML client requests. The actor control module  100  can also reside and execute on the same computer as the server  50 B. The actor control module  100  can be attached to the actor  70  (e.g., as a robotic arm) though an RS-232 serial connection. 
     In operation, the applet  80 A and the web server  50 A provide live streaming video feedback related to the progress of the actor  70  captured by the camera  110  and video card  90 . That is, the applet  80 A actually manages presentation of video (and, possibly, audio) information from the actor  70  on the client  20 . The applet  80 B and the web server  50 B, in turn, coordinate control of the actor  70 . Input from the Tele-Directors is received by the applet  80 B and forwarded to web server  50 B for aggregation. The aggregation by the web server  50 B results in a consensus command, which is forwarded to the actor  70  for performance. The resulting progress of the actor is viewed by the Tele-Directors using the camera  110  (and if desired, an audio device), the video card  90 , the web server  50 A and the applet  80 A as described above. 
       FIGS. 3A-3D  depict different embodiments of a voting interface  300  displayed at the clients  20 . With reference to  FIG. 3A , the voting interface  300  includes a voting area  310 , a question area  320 , and chat area  330 . The voting interface  300  can be realized as a graphical user interface (GUI). Questions or choices related to the future activity of the actor  70  are displayed in the question area  320 . The questions can be randomly chosen from a database of questions, provided by the Tele-Directors, or provided by a base station operator located at the base station  60 . The Tele-Directors respond to the question by placing a “votel” (i.e., an indicator associated with each Tele-Director using the system) in the vote area  310  thereby indicating their respective response to the question. Each question can have a limited response period (e.g., one minute). Within the response period, the Tele-Directors may change their responses. For example, a Tele-Director may wish to change his or her vote in response to postings displayed in the chat area  330  from other Tele-Directors lobbying for a specific response. After the response period expires, the votels are analyzed to generate the consensus command, which is in turn forwarded to the actor  70 . Alternatively, the votes from the Tele-Directors can be analyzed in a continuous, dynamic fashion to provide a prediction of the consensus command prior to the expiration of the response period. 
     With reference to  FIG. 3B , in one exemplary embodiment, the system  10  is used to control the actions of a biological organism (e.g., a snake), preferably a real or animated mammal (e.g., a human). The voting interface  300  displays a shopping environment in the voting area  310 . In this embodiment, the question area  320  and voting area  310  are integrated. A votel  340  is associated with each Tele-Director logged into the system. While a Tele-Director&#39;s votel is outside the voting area  310 , that Tele-Director may present textual information to the other Tele-Directors. The text is displayed below the Tele-Director&#39;s votel. 
     When a vote is to take place, the live audio/video stream may be paused and a still picture displayed, such as that shown in  FIG. 3B ; alternatively, streaming may continue without pause. The Tele-Directors select a portion of the picture and post, by typing, a choice (goal) for the progress of the actor. A transparent circle appears, accompanied by the text that defines the choice in a color associated with Tele-Director. The Tele-Directors vote for choices by moving their respective votels into the corresponding transparent circles. Each Tele-Director may post multiple choices but may only vote for a single choice. After the expiration of the voting period the totals for each choice are determined and the winning choice (i.e., the most popular choice) is forwarded to the actor  70 . 
     Alternatively, a simple economy in which Tele-Directors spend points to vote or post goals can be used to control the number of votes and or choices for a given voting period. Each Tele-Director starts with a limited number of points. The Tele-Directors pay to post a goal, to vote, and to change a vote. Points can be replenished over time and bonuses given for voting for, or posting, the winning choice. Different economic models may be employed for different types of applications. For example, in a distance learning context, it may be appropriate to encourage all questions; even those questions that are not chosen for immediate presentation to the speaker might facilitate useful discussion among the Tele-Directors. 
     With reference to  FIGS. 2 and 3C , in another embodiment the Tele-Directors control a robot arm which moves a planchette on a Ouija board. In operation, the applet  80 B displays a small window with a representation of the planchette (a “virtual” planchette). The applet  80 B also displays two text panels: one listing currently registered clients and another containing the question being considered. The applet  80 B establishes communication with server SOB either directly via a bus, if it is located on the same machine, or through a socket connection. Through this connection, the clients send desired force or motion vectors (as described below) to server  50  at regular time intervals (e.g., every 3 seconds). The server  50 B aggregates the force commands from all the clients and generates a consensus command, which is forwarded to the robot arm. The server  50 B also transmits information about the current question being asked and the clients currently registered back to the instance of the applet  80 B at each client  20 . 
     As described above, the applet  80 B at each client sends a desired motion vector to the server  50 B at a periodic rate. At the client, the position of a mouse (or other pointing device) associated with the client is read by a local java applet and the virtual planchette is displayed in the lower window of the voting interface. The virtual planchette tracks the motion of the mouse as it is moved by the client user. The planchette motion is preferably based on an inertial model to generate force or motion vectors. 
     In one embodiment, a vector from the center of the planchette screen to the current mouse position is treated as a force command. The user of a client i specifies desired acceleration by moving the mouse, and the acceleration is expressed in two dimensions x, y as a=(a ix ; a iy ). Frictional drag of the planchette may be modeled with a constant magnitude and a direction opposite the current velocity of the planchette. If the current velocity of the planchette in two dimensions is v 0 =(v 0x ; v 0y ) and the magnitude of the constant frictional acceleration is a f , then 
                 a   fx     =       a   f     -       v     0   ⁢   x             v     0   ⁢   x     2     +     v     0   ⁢   y     2               ,       a   ⁢           ⁢   n   ⁢           ⁢   d   ⁢           ⁢     a   fy       =       a   f     ⁢         -     v     0   ⁢   y               v     0   ⁢   x     2     +     v     0   ⁢   y     2           .               
The resulting velocity v of the planchette is v=v 0 +(a+a f )Δt. The virtual planchette is preferably updated locally  30  times a second, therefore Δt=0.03 seconds. Summing the inputs from all clients yields the consensus command (i.e., the net desired acceleration of the planchette).
 
     The consensus command is forwarded to the actor (in this case, the robot arm), and is accepted in form of a desired goal point and speed. To prevent the robot arm from moving outside the viewable region, the calculated goal point is limited to the boundary of the region. For example, with an x, y region defined by 0&lt;x&lt;W and 0&lt;y&lt;L, the current position of the robot is projected in direction v until it hits the boundary. Let θ=tan −1 (v y /v x ). To calculate the goal point, the following equation for y corresponds to each of the four possible regions of θ:
 
0°≦θ&lt;90°  y =min( L,y   0 +( W−x   0 )tan θ)
 
90°≦θ&lt;180°  y =min( L,y   0 +(− x   0 )tan θ)
 
180°≦θ&lt;270°  y =max(0, y   0 +(− x   0 )tan θ)
 
270°≦θ&lt;360°  y =max(0, y   0 +( W−x   0 )tan θ).
 
Therefore x=x 0 +[(y−y 0 )/tan θ]. The robot control module  100  is sent a move command toward goal point (x,y) with speed v=√{square root over (v x   2 +v y   2 )}. This procedure is preferably repeated every 3 seconds.
 
     With reference to  FIGS. 1 and 3D , in another embodiment a “Spatial Dynamic Voting” (SDV) interface facilitates interaction and collaboration among the remote clients  20 .  FIG. 3D  illustrates the SDV interface displayed by the browsers of all active clients. The users of the clients  20  register online to participate in collaborative control of the actor by selecting a votel color and submitting their email addresses to the server  50 , which stores this information in a database and sends back a password via email. The server  50  also maintains a tutorial and a frequently asked questions section to familiarize new clients with system operation. 
     Using the SDV interface, clients participate in a series of short (e.g., one minute) “elections.” Each election is based on a single image with a textual question. In  FIG. 3D , the actor  70  is visiting an architectural site. The election image shows a building with the question: “Where should we go next?” The clients click on their respective displays to position their votels. Using the HTTP protocol, the clients  20  transmit the positions of the votels back to the server  50  and appear in an updated election image sent to all the clients every 6-20 seconds. The updated image allows the Tele-Directors to change their votes several times during an election. When the election is completed, a clustering algorithm (described in more detail below) can analyze the pattern of the votes to determine a single command for the actor. The SDV interface differs from multiple choice polling because it allows spatially and temporally continuous inputs. 
     To facilitate client training and asynchronous testing, the system  300  can include two modes of operation, offline and online. In offline mode, all election images are extracted from a prestored library of images resident, for example, in a database at the server or the base station. In online mode, election images are sampled from the live video captured by the actor. Both offline and online SDV modes have potential for collaborative education, testing, and training. 
     The consensus command can be automatically extracted from the positions of the votels. A votel may be defined as a vector v i =[u, x, y, t], where u is a client identifier, x and y indicate a two-dimensional location in the election image, and t indicates the time when the votel was received at the server. During each election, the server collects a set of votels V. The collection V is analyzed to determine voting patterns in terms of goals and collaboration. 
     Conventional clustering algorithms can be used to identify groups of neighboring votels to thereby generate the consensus command. After votels are classified into groups, one approach is to compute the convex hull of each group with three or more votels and treat each convex polygon as a distinct response to the question. When the actor is restricted to movements on a floor, the horizontal positions of votels provide the primary navigation information. In such cases, all votels are projected onto the horizontal axis and a conventional nearest neighbor algorithm is employed to perform one-dimensional incremental interval clustering. After all votels are collected and their clusters analyzed, the goal with maximum votes (as identified by the clustering analysis) is selected for execution by the actor. 
     The invention can also provide information concerning the degree of collaboration among the Tele-Directors based on how the votels are spatially correlated. For each question i, a votel density ratio c i  is computed: 
               c   i     =         d   i     d     =           n   i       a   i         N   A       =         n   i     N     ⁢     (     A     a   i       )                 
where d i  is the votel density (votes per unit area) for goal i, d is the overall average votel density, n i  is number of votel in goal i, a i  is the area or width of the goal i, N is the total number of votes and A is the area of the election image. This metric is proportional to the ratio n/N and inversely proportional to the area of the goal region. The metric is high when many votes are concentrated in a small goal region (high collaboration) and low when votes are uniformly spread among multiple goals (low collaboration). The overall collaboration level for each election can also be computed by:
 
             c   =         ∑     n   i         ∑     a   i         ⁢     A   N             
When all votes fall into goal regions,
 
             c   =     A     ∑     a   i               
provides a measure of how focused the votels are.
 
       FIG. 3E  depicts an embodiment  400  of the voting interface which can be used in a journalistic or educational environment. The Tele-Directors can post potential commands as text. In turn, the Tele-Directors vote on these commands by using an input device (e.g., a mouse) to indicate which command they prefer. The Tele-Directors can also change or remove their votes as they desire. Each Tele-Director can have, for example, five votes to distribute as he or she wishes. That is, a Tele-Director can vote five times for a single command, or give a single vote to a number of different commands, etc. Voting is continuous and dynamic, and the Tele-Directors may chose to erase votes because a command is no longer relevant to the current situation in the remote environment displayed via the voting interface  400 . A voting round ends when the actor  70  calls for a consensus command. The command with the most votes can be chosen as the consensus command and sent to the actor  70 . In turn, the actor has the ability to reject the consensus command, and the Tele-Director who proposed (or those Tele-Directors who voted for) the rejected command are penalized by, for example, losing several votes for a specific number of subsequent voting rounds. This embodiment may employ streaming video and audio for awareness of the actor&#39;s situation. Additionally, the interface can include a chat space facilitating communication among the Tele-Directors. 
     Having shown the preferred embodiments, one skilled in the art will realize that many variations are possible within the scope and spirit of the claimed invention. It is therefore the intention to limit the invention only by the scope of the claims.