Method system and apparatus for telepresence communications utilizing video avatars

An apparatus, system and method for telepresence communications in an environment of a virtual location between two or more participants at multiple locations. First perspective data descriptive of the perspective of the virtual location environment experienced by a first participant at a first location and feature data extracted and/or otherwise captured from a second participant at a second location are processed to generate a first virtual representation of the second participant in the virtual environment from the perspective of the first participant. Likewise, second perspective data descriptive of the perspective of the virtual location environment experienced by the second participant and feature data extracted and/or otherwise captured from features of the first participant are processed to generate a second virtual representation of the first participant in the virtual environment from the perspective of the second participant. The first and second virtual representations are rendered and then displayed to the first and second participants, respectively. The first and second virtual representations are updated and redisplayed to the participants upon a change in one or more of the perspective data and feature data from which they are generated. The apparatus, system and method are scalable to two or more participants.

TECHNICAL FIELD

This invention relates to interpersonal communication using audiovisual technology and more specifically to methods and systems for the transmission and reception of audio-visual information.

BACKGROUND OF THE INVENTION

There are many situations in which one or more individuals would like to observe and possibly interact with objects or other individuals. When two or more individuals need to meet and discuss issues of mutual interest, a common approach is a physical (face-to-face) meeting. This type of meeting has the advantage of direct personal contact and gives the individuals the ability to communicate most effectively, since eye contact may be maintained, and physical gestures such as facial expressions, hand movements, and body posture are readily evident. For most meetings, this is the preferred medium of exchange since large amounts of information may be exchanged transparently if the information is at the location of the meeting.

In certain situations, such as communication over long distances, arranging such face-to-face meetings can be time-consuming or prohibitively expensive. In these situations, the most common way to exchange information is over the telephone, via e-mail or by teleconferencing. Each of these approaches has serious drawbacks. Telephone conversations provide none of the visual cues that may be important when making a business decision. Telephone conversations are also difficult to manage when more than two individuals need to be involved in the meeting. E-mail or regular postal services are much slower than an in-person meeting and provide none of the visual or even audio cues that are present in in-person meetings. The use of video teleconferencing equipment allows individuals at remote locations to meet and exchange information through the use of audio/visual communication.

There is, however, a substantial difference between an in-person meeting between two or more people and a meeting using a video teleconferencing system. The latter does not provide the same experience as the former. In an in-person meeting, we see the other person in three dimensions, in color and at the right size, and each participant at their appropriate physical position. More importantly, we have the ability to make and maintain eye contact. This visual information contributes to a sense of presence of the individual. The current state of the art in video teleconferencing provides none of these benefits. Video teleconferencing also does not provide the nuances of facial and body movement available from a personal meeting, since the entire image is transmitted at the same scale. Therefore, the in-person impact of a frown or smile is likely to be greater than when using a video teleconferencing system since the correct aspect and detail of the area around the mouth is not transmitted in a video teleconference. Moreover, exchange of non-personal information, such as reports, documents, etc., resident at a particular location to others participating in a teleconference may be limited. It is therefore difficult to transmit personal and non-personal information of a desirable quality and quantity using existing teleconferencing technology.

DETAILED DESCRIPTION

Referring now toFIG. 1, a block diagram of a system that supports a virtual teleconference at a virtual location between a first participant at a first location100and a second participant at a second location200is shown. The system provides for the collection, processing and display of audiovisual information concerning one or more remotely located participants to a local participant in a manner that has telepresence, as will be described. The collection and processing of audiovisual information from both remote participants and a local participant, as well as interaction between remote and local participants and environment description of the virtual location, allows for the generation, updating and display of one or more virtual representations or avatars of remote participants in the environment of the virtual location of the teleconference to be made to the local participant, from the perspective of the local participant. The telepresence teleconferencing of the present invention is scalable to any number of participants. As used herein, virtual representations or avatars may refer to either the video and/or the rendered representation of the user.

The functionality shown inFIG. 1may be employed for each participant beyond Participant1and Participant2that is added to the virtual teleconference, subject to available memory and latency requirements, with stored model information about the added participants made available to each of the other participants in the teleconference to facilitate the generation of virtual representations or avatars of all remote participants in the teleconference to any given local participant.

There is a capture/tracking element104,204, respectively, that captures cued data generated by features of the participants1,2. As used herein, cued data refers to data generated by certain monitored features of a participant, such as the mouth, eyes, face, etc., suitable for cuing their capture and tracking by capture/tracking elements104,204, respectively and that provide information that enhances the sense of actual presence, referred to as telepresence, experienced by participants in the virtual location of the teleconference. Cued data may be visual and/or audio. Cued visual data refers to the capture of the movement of such features may be cued by movement of the feature, such as movement of an eyebrow, the mouth, a blink, etc., or it may be cued to automatically update periodically. Cued data may have an audio component as well and capture of cued audio data may be triggered by the sound produced by a participant's mouth or movement of the mouth itself. Additional cued visual data to be collected may be movement of the hands, the head, the torso, legs, etc. of a participant. Gestures, such as head nods, hand movement, and facial expressions, are important to clarify or enhance meaning, generally augmenting the communication experience and thus are important to enhance the telepresence of the present invention. The capture elements104,204additionally may have the ability to track movement of certain of the features they are monitoring, as will be explained. Suitable capture elements may include cameras, microphones, and head tracking equipment. Any number of capture elements may be used. For instance, there may be a camera devoted to capturing and tracking movement of each eye of a participant, another devoted to capturing facial movements, such as mouth movement, and a microphone for capturing any sounds uttered by the participant. Moreover, the proximity of the data capturing devices to the participant whose movements and sounds are being captured may vary. In a head mounted display, the eye, face, mouth, head tracking, etc. capture elements may be located inside the eyewear of the head mounted display. Or, the capture elements may be a series of cameras located on a desk, table or other area proximate the participant.

One mechanism that incorporates the data capture components into a single integrated system is a specially designed pair of eyeglasses. The eyeglasses are capable of collecting eye and facial tracking information, as well as audio information through the use of a boom, a collection device that may have a single point of attachment to a head-mounted data capture element.

The cued data gathered from a participant is processed by a processing element102,104to extract recognized and selected feature data, such as pupil movement, eyebrow movement, mouth movement, etc., from the raw cued data captured by the capture elements104,204. This extracted feature data of the local participant may then be transmitted by transmit elements112,162for receipt by processors associated with remote participants, where it will be used, along with other data, such as environment and perspective data, to generate a virtual representation of the local participant in the virtual environment for viewing by one or more remote participants from the perspective of the remote participants.

In addition to capturing cued visual data from a participant, the capture elements104,204additionally are tasked with capturing perspective data from the participant1,2, respectively. It is noted that perspective data may be captured by capture elements that are different or disjoint from capture elements102,204. Perspective data refers to any orientation or movement of the participant being monitored that may affect what is experienced by the participant in the virtual environment of the teleconference. Perspective data may thus include movement of the head or a re-orientation, such as turning, of the participant's body. For instance, if the virtual environment of the teleconference is to provide the sense that participants1and2are seated across from each other at a virtual conference table, then the acts of participant1moving his head, standing up, leaning forward towards participant2, etc. may each be expected to change what is seen or heard by participant1in the virtual environment, and thus the perspective of the environment of the virtual location experienced by participant1is said to have changed. Capturing and tracking movement, re-orientation, or other perspective data of a participant provides one of the types of data that is used to process and generate a believable teleconference at the virtual location for the participant. Suitable capturing/tracking elements for capturing perspective data may include cameras or other motion tracking elements such as magnetometers that use the magnetic field of the earth and accelerometers that measure acceleration, or other devices used to determine the direction and orientation of movement of the head or other parts of the body that can affect the perspective of the virtual environment that is experienced by a participant.

Data about the one or more participants at remote locations is additionally needed to generate a virtual representation of the remote participants in the virtual environment from the perspective of a local participant. Receive elements106,206, functionally associated with participants1and2, respectively, receive cued visual data captured from one or more remote participants and transmitted over the system by remote transmit elements112,162as described above. Thus, for the simplified system ofFIG. 1, receive element1106will receive from transmit element162extracted feature data extracted and processed by processing element2152from cued visual data captured by capture/tracking element2154from Participant2and transmitted by transmit element2156. Similarly, receive element2156will receive from transmit element1112extracted feature data extracted and processed by processing element1102from cued visual data captured by capture/tracking element1104from Participant1and transmitted by transmit element1112. Again, the system ofFIG. 1is scalable, meaning there may be more than two participants, in which case the extracted feature data received by receive elements106,156would be from two or more remote participants.

With the receipt of extracted, remote feature data by the receive element associated with a participant, the local processing element now has enough information to generate one or more virtual representations of the one or more remote participants in the virtual environment from the local participant's perspective. In addition to the extracted feature data of a remote participant received by the local receive element, the processing element associated with the local participant has the perspective data captured from the local participant, a model of the remote participant, and information that defines what the visual and audio configuration of the environment of the virtual location at which the virtual teleconference takes place. The processing element thus processes this information to generate a virtual representation of the remote participant in the virtual environment as seen from the perspective of the local participant. This processing may be performed by the processing element to generate virtual representations from the perspective of the local participant in the virtual environment for each remote participant that transmits its visual and/or audio information to the local receiver element.

Visual extracted feature data of the remote participant may be put together with a model of the remote participant (108,158) that is stored and accessible to the processing element associated with the local participant. The stored model may be a two-dimensional or three-dimensional (3D) computer model upon which the received extracted feature data may be used to update the model. The model may additionally be just the head, bust or some larger model of the remote participant. It may be that only head or face portion of the model is individual to the remote participant, with the rest of the virtual representation of the remote participant being supplied by a stock avatar. The portion of the virtual representation of the remote participant that is individualized by the use of the participant-specific model108,158may well be affected by factors such as the amount and quality of cued data that is collected and the amount of processing power and time to be dedicated to this task. If only eye, mouth, and face data are captured from the remote participant, then it would sufficient to store only a participant-specific model of the head of the remote participant upon which the collected and extracted feature data may be overlaid, for example. An example of a 3D model is described in conjunction withFIG. 9.

Information about the environment110,160of the virtual location where the teleconference is to take place is also processed by the local processing element when generating the virtual representation of a remote participant. Environment data expresses the set-up of the virtual conference, with the relative positions of each of the participants in it and the visual backdrop, such as the location of conference table, windows, furniture, etc. to be experienced by the participants. Movement of a participant, by either head or body movement, by one or more teleconference participants may change the perspective from which the participant sees this environment and so must be tracked and accounted for when generating the virtual representation of the remote participant that will be displayed to the local participant. Again, the processing element that generates the virtual representation for the local participant is operable to generate virtual representations in this manner for each participant in the virtual teleconference for which cued data is received.

The processing elements1and2, shown as elements102and152, respectively, need not necessarily reside at the participants' locations. Additionally, they need not necessarily be one discrete processor and may indeed encompass many processing elements to perform the various processing functions as will be described. It is further envisioned that there may be a central processing element, which would encompass both processing element1102and processing element2152and which may further be physically located in a location different from locations100and200. This is illustrated in block diagram300ofFIG. 3in which the processing of captured local feature and perspective data and remote data need not be performed at local locations, such as Location1and Location2, and may indeed be provided by processing capabilities of communication network390. The captured data of participants1and2are transmitted remotely using communications network390. In a certain embodiment of the present invention, communications network390is a high bandwidth, low latency communications network. For instance, data may be communicated at 20 fps with a 150 mS latency over a standard Internet IP link.

Models of remote participants340,380are shown at local locations, but this is not required, particularly as the processing element or elements are to be located on the communication network; the stored model may be a 3D computer model as shown. 3D models are useful to store image information that does not rapidly change, and thereby allows the amount of data that must be transmitted across communications network390to be reduced. After receiving remote image data, data display components330and360are operable to update the 3-dimensional models340and380used to create the virtual representation.

The one or more virtual representations that have been generated in the virtual environment by the processing element are provided to a render element114,164that renders the computer generated data of the one or more virtual representations for display by a display element116,166to the local participant. The display element may be part of a head mounted display worn by the local participant or it may be any other suitable mechanisms for displaying the environment of the virtual location to the participant.

Important to maintaining the sense of actual presence or telepresence between two or more participants in the teleconference, the system has the ability to monitor or track any changes occurring with remote participants or the local participant. Any such changes will require that the virtual representation of the virtual environment and the other participants in it be changed accordingly. Thus, upon a change in the remote cued data received by a local receiver element, the perspective data collected from a local participant, or a change in the environment of the virtual location itself, will cause the one or more virtual representations of remote participants that are generated to be updated and the updated representations rendered and then displayed to the local participant.

Referring now toFIG. 2, a flow chart200which described a method of teleconferencing between at least two participants at a virtual location in accordance with certain embodiments of the present invention is shown. At block210, data generated by the features of first participant100and perspective data of first participant100are captured. At Block220, recognized patterns of the captured feature data of the first participant100are extracted. At Blocks230and240, capturing of perspective data and cued data of the second participant200and extraction of recognized data is performed. At Block250, the extracted feature data of the second participant, the perspective data of the first participant, and environment data of the virtual location are processed to generate a virtual representation of the second participant from the perspective of the first participant. At Block260, similar processing occurs to generate a virtual representation of the first participant from the perspective of the second participant. These virtual representations may then be displayed to the appropriate participant at Blocks270and280. As described above, the virtual representation is first rendered by a rendering element and the rendered virtual representation is displayed to the participant on an appropriate display means, such as a head mounted display. Finally, at Blocks290and295, the generated virtual representations are updated upon a change in any of the data used to generate them. Thus, a change in the perspective data of the local participant, the cued data captured from the remote participant, or the environment will ensure that the virtual representation is updated accordingly. It is important to note, however, that since the cued remote data and the local perspective data are being monitored and tracked continuously, the virtual representations are being updated periodically anyway, such as 15 times or more per Second, for instance. However, a change in the data may force the update process to occur sooner than it might otherwise have occurred, contributing to the sense of a “real-time” in-person telepresence environment enjoyed by participants in the virtual teleconference.

It is noted here that while the capture, extraction and processing necessary to create the virtual representation of the second participant for display to the first participant occurs prior to similar processing to generate the virtual representation of the first participant for display to the second participant, the order may be changed if so desired without departing from the spirit and scope of the invention.

Referring now toFIG. 4, a more detailed block diagram400of elements of telepresence communication architecture400is shown, in accordance with certain embodiments of the present invention. As indicated by the dashed line in the figure, the figure illustrates functionality involving data collected between first and second locations. As will be clear from the following description, the functionality concerns data collected, processed and transmitted by a sender block410at location1and data received, processed and displayed by a receiver block455at location2. However, it will be understood that to make a completely integrated system there will need to be a receiver block and a sender block to support the participant at location1and the participant at location2. The one-directional block diagram ofFIG. 4will simplify the description to follow and an extension to a fully bi- or multi-directional will be understood by those of skill in the art. It will be noted by one of ordinary skill in the art that a telepresence communication system is operable to transmit two images in full duplex using one or more communication links; while communication may occur over one or more broadband links, communication is not restricted to broadband. Thus, a remote location may comprise a sending module410as well as a receiving module455. Also, a local location may comprise a sending module410and a receiving module455. This configuration will allow two images to be tracked, transmitted, received and displayed. This is of course scalable to any number of locations and participants, subject to available processing power, storage, and latency conditions.

It can be seen by means of the dashed lines in the figure, that there are three main functions being performed by sender block410for the participant at Location1: capture/tracking, processing, and synchronizing/transmitting. The sender block410is concerned primarily with the capture/tracking at Blocks415,420,425, processing at blocks430and437, and transmitting at Blocks445,450of locally obtained participant information. At block415, local audio, such as what the location1participant is saying is captured. Head tracking block420tracks movement and orientation of the location1participant and thus supplies the perspective data of participant1. Image Capture block425captures feature data of location1participant, such as movement of participant1's mouth, eyes, face, etc. In more sophisticated capture schemes, other features of the participant may be captured, such as movement of hands, arms, legs, etc. Blocks415,520,425are all examples of capture elements104,154. In certain embodiments of the present invention, audio element415is a microphone or boom microphone, head tracking element420is a head tracker, accelerator or some combination thereof. An MPEG-4 style facial animation player with local head tracking for a space-stable view may be used if desired. Image capture element425may be a number of cameras.

FIG. 5illustrates various types of capture elements suitable for capturing and tracking cued feature data and perspective data of a participant. In this figure, features of participant510, such as eyes, mouth, face, hands, head etc. are captured by feature cameras520,550, and550, while tracking equipment530is operable to track movement of these features. Audio sensor560captures audio generated by participant510. Additionally, mouth movements may be tracked via audio analysis and embedded camera or cameras in a boom microphone if desired. According to an embodiment of the present invention, one or more of feature cameras520,540,550, tracking equipment530and audio sensor560may be located in a head mounted display, such as an eyewear display. Also according to an embodiment of the present invention, a boom having one or more boom cameras and an audio sensor560may be coupled to the pair of eyeglasses. The one or more boom cameras are operable to provide more detailed resolution of the mouth and the region around the mouth. In a certain embodiment of the present invention, infrared illumination could be employed by eye cameras to compensate for lack of visual light.

Processing is performed on the audio information and the perspective data captured by head tracking element420to generate sound information about the location1participant that can be transmitted. Sound processing block430can modify the raw audio415produced by Participant1as a function of the head movement of Participant1. Alternately, the raw audio captured at415may be simply transmitted if no locale processing is needed or desired. Computer vision recognition element437has feature extraction435and feature tracking440processing of the head tracking and cued feature data provided by elements420and425. The most important feature data contained in the captured data is extracted and will be transmitted for processing by the receiver455at a remote location2. Computer vision recognition subsystem437, for instance, can extract and track movements of the head, mouth, pupils, eyelids, eyebrows, forehead, and other features of interest. In some cases, computer vision recognition element437may use a local 3D model of the participant itself for feature tracking.

In accordance with certain embodiments of the present invention, a sense of eye-to-eye contact may be achieved by providing, during a transmission set-up period, a first one or more fixed dots on the image displayed to a first user and a second one or more fixed dots on the image displayed to a second user. During the transmission set-up period, the location of the eyes in the image displayed to the first participant is collocated with the first one or more fixed dots. Also during the transmission set-up period, the location of the eyes in the image displayed to the second participant is collocated with the second one or more fixed dots. This approach enables the participants to have the sense of eye-to-eye contact since the first one or more fixed dots and the second one or more fixed dots provide the expected location of the eyes displayed to the first participant and the second participant, respectively. Eye contact is maintained by the participants responding to the visual cues presented to them, as in a real-life in-person conversation.

Extracted feature data from block437and processed sound from block430is encoded and synchronized at block445. It is modulated at modulator450and then transmitted for receipt by demodulator463of the receiver block455associated with Location2. In a certain embodiment of the present invention, this data is transmitted using a broadband link460.

Data received from location1by demodulator463is demodulated and passed to a decoder465. Decoder465passes decoded audio and extracted feature data of the participant at location1to sound element473, view generation block475and model update block480. Movement and orientation of participant2, referred to as perspective data of participant2, from head tracking element470and the audio data received from participant1are processed by sound block473to generate an audio component of a virtual representation of participant1from the perspective of participant2that can then be provided by audio element493. Consider, for example, the following. The audio component of the virtual representation made available to participant2is affected not only by what participant1says, but also upon the orientation of participant2's body or head with respect to participant2in the virtual environment.

In certain embodiments of the present invention, encoder445encodes spatial coordinated information that enables head tracking component470to create an aspect of the remote image that is space stabilized. Note that this space stabilization is operable to occur when one or more aspects captured by head tracking equipment420and image capture equipment425are coupled to a pair of eyeglasses. In this case, the use of head tracking420and feature tracking440allows the 3-D image generated to be stabilized with respect to movement of the head.

Extracted feature data is additionally made available to view generation block475and model update block480by decoder465. It is used by model update480to update the model of the participant at location1that is stored at block483. In certain embodiments, model update block480performs a facial model update that uses facial data stored in 3-D model483to construct the virtual representation of participant1. View generation block475generates the view or views of the virtual representation of the participant1from the perspective of participant2to be rendered by render element485and then displayed to the participant at location2by display490. In certain embodiments of the present invention, two slightly different views of the virtual representation of participant1in the virtual environment are generated by view generation element475. When these slightly different views are rendered and then displayed to each eye at485and490, respectively, they result in participant2experiencing a 3D stereoscopic view of participant1.

Referring now toFIG. 6, exemplary data capture system600is shown. The capture and tracking elements illustrated in this drawing include first and second eye cameras610,620, face camera630, microphone for capturing sounds made by the participant, and tracking equipment650. Eye cameras610and620capture the movement of various eyes features, including pupil movement, blinks, and eyebrow movement. Face camera630can capture movement of the mouth, jaw, node, and head orientation. Microphone640can additionally be a boom microphone having a camera that looks at the participant's mouth. Tracking equipment650tracks these features over time. InFIG. 7, image capture flow700illustrates how the data captured by capture elements610,620,630,650of image capture block710is then processed by vision recognition and feature extraction processing at block720to extract certain valuable feature-cued data. At block730, synchronization of this extracted feature data with a time or date stamp is performed prior to the data being transmitted for receipt by a receive block455associated with a remote location.FIG. 8illustrates image generation and display flow800, in which it is illustrated that views of the virtual representation by a participant at a remote location may be generated at Block840, for one each eye for 3D stereoscopic viewing if desired, from local tracking data820, referred to as perspective data herein. Extracted feature data from the remote location810is used to animate the stored model of the remote participant at Block850. This information is then passed onto a rendering engine that renders computer images, which again may be stereo at Block860; the rendered images may include audio information as described previously. Finally, at block870, a display element, such as a display screen or a head mounted display, displays the rendered images to the local participant.

Referring now toFIG. 9, an example of a 3D model, as may be stored locally to aid in the generation of an avatar representative of a remote participant, is illustrated. On the left side of the model is the wireframe, the vertices of which are stored in memory and define the geometry of the face of the remote participant. On the right side, the texture map of the face, which shows such things as skin texture, eye color, etc., is overlaid the basic geometry of the wireframe as shown to provide a more real and compelling view of the remote participant. The updated movement, reflected in the captured feature data, is manifested in corresponding changes of the wireframe.

InFIG. 10, an illustration of a first participant at location100, a second participant at location200, and a virtual location1000is illustrated. In can be seen that, in this example, participants1and2at locations100and200, respectively, are wearing head mounted displays through which they may both experience a virtual teleconference1000in which participant11010and participant21020both experience an mutual environment1030that is not real but which does make use of eye-to-eye contact and other telepresence features to greatly facilitate the virtual meeting. In this example, the environment1030is streamlined, having a conference table, a chair for each of the participants, and the participants themselves. It is also envisioned that the virtual environment may include the virtual representations of participants set against a real backdrop of the location where they are (location1, location2, etc.), such as in the middle of a library or conference room, etc. in which the viewing participant is actually located. As has been discussed, a teleconference according to the present invention is scalable to a number of participants. As shown in virtual teleconference1100ofFIG. 11, this virtual teleconference1100is attended by at least four participants1110,1120,1130, and1140. The virtual environment1150in which the teleconference takes place is more elaborate. Also, as shown in this example and also inFIG. 12, may actually be better for a face-to-face traditional conference because it can facilitate the sharing of data during the teleconference by many participants are varying physical locations. InFIG. 12, participants1210,1220, and1230share an environment1250in which data or other objects presented during the virtual teleconference, such as view graph1260, may be offered by a participant for viewing by one or more of the other teleconference participants. The ability to share data or other object information residing at a remote location with others not at that location via the virtual teleconference provides the advantage of being able to share a quality and quantity of information that would not ordinarily be available in a traditional teleconference. When there are more than two people in the teleconference, telepresence facilitates the exchange and observation of inter-personal communications that occur between multiple participants, such as shrugs, glances, etc. which commonly form an important, non-verbal aspect of any conversation. If there is a shared object, for instance, a participant can see that other participants in the teleconference have their attention directed to the shared object or that a participant is looking away from the object, etc., to reflect the full extent of communications that might occur in an actual meeting.

In accordance with another embodiment of the present invention, rather than capturing and tracking cued data generated by the monitored features of a participant, as was discussed above in connection with capture/tracking elements104,204, for instance, an audio and/or visual recording of captured feature data of the participant may be recorded by an capture element, which may have capture audio/visual (A/V) elements. The capture element may be a video camera focused on the participant, or other recording device, such as a cellular phone, microphone, etc. The capture element may be voice driven if desired. Tracking of the participant may be accomplished separately through a tracking element, as will be described herein, rather than through a combination capture/tracking element, i.e. capture/tracking elements104,204as was described above. This audio and/or visual recording of the participant is processed by a processing element where it is encoded and then transmitted by a transmit/sync element to a receive element to be used for generating a video avatar of the participant.

Throughout the description of these further embodiments of the invention, several different types of data are described. The term user perspective modification data refers to data actively provided from the user via a user interface device or tracking element—e.g. keyboard or mouse—used to change the user's perspective in the space. The term feature data or captured feature data is data that has been passively captured from the user via a sensor or capture element, such as a camera, microphone, etc., as will be discussed at length, and may include, by way of example and not limitation, A/V feature data as will be described. The phrase encoded feature data is used to refer to the extracted feature data after is has been processed by an encoder—e.g. H.263. This data is used by the receiver and displayed on a model of the participant.

Referring now toFIG. 13, a block diagram of a system that supports a virtual teleconference at a virtual location between a first participant at a first location1300and a second participant at a second location1350is shown. The system provides for the collection, processing and display of audiovisual information concerning one or more remotely located participants to a local participant in a manner that has telepresence, as will be described. The collection and processing of audiovisual information from both remote participants and a local participant, as well as interaction between remote and local participants and environment data of the virtual location, allows for the generation, updating and display of one or more virtual representations or avatars of remote participants in the environment of the virtual location of the teleconference to be made to the local participant, from the perspective of the local participant. As used herein, virtual representations or avatars may refer to either the video and/or the rendered representation of the user.

The telepresence teleconferencing of the present invention is scalable to any number of participants. The functionality shown inFIG. 13may be employed for each participant beyond Participant1and Participant2that is added to the virtual teleconference, subject to available memory and latency requirements, with stored model information about the added participants made available to each of the other participants in the teleconference to facilitate the generation of virtual representations or avatars of all remote participants in the teleconference to any given local participant.

Referring again to the figure, capture element1304,1354, respectively, captures a recording of the participant on whom it is focused; the recording may be a streaming A/V format, an example of encoded feature data; otherwise, the recorded captured feature data may be processed by an encoder1303of processing element1302to generate encoded feature data suitable for transmission by transmit sync element11312to receive element21356. Thus, capture element11304, shown as a video camera focused on Participant1, records a recording of captured feature data of Participant1. This recording of the participant is sent to processing element1302, which contains an encoder/decoder1303, for processing and encoding (compression). Examples of video encoding standards for streaming A/V formats include H.263, H.264, MPEG-2, MPEG-4. Capture element1304,1354is capable of recording an entire image including video and audio, just video or just audio. It may look at the whole person, not just portions of the body (eyes, head, eyebrows, etc) as is the case with the cued data discussed previously.

Unlike the system discussed previously in connection withFIG. 1, in this system embodiment capture element1304,1354is not used for both capture and tracking, but rather for recording captured feature data of the participant, such as a video of the participant. This feature data is encoded by encoder/decoder element1303, and the processed and encoded feature data, such as video, may then be transmitted by transmit/sync elements1312and1362, respectively, to receiver elements1306,1356, respectively, at the receiving end. There the encoded feature data is supplied to processors1352,1302, respectively, associated with remote participants, where it will be processed, along with other data, such as environment information1310,1360, user perspective modification data provided by tracking elements1305,1355, respectively, for generating at render elements1314,1364a video avatar of the participant in the virtual environment for viewing at display elements1316,1366by one or more remote participants from the perspective of the remote participants.

The tracking of user perspective modification data from the participant is provided by tracking element1305,1355, respectively, illustrated here as a keyboard; the tracking element may be any element that allows the user to control and modify his own perspective within the virtual environment. The tracking element provides a means, other than that described previously with regard to capture/tracking element104,154, to track those movements of a participant within the virtual environment, such as the turning of the head or body, or other movement, that can be expected to alter the perspective of that participant in the teleconference. The tracking element, such as the keyboard shown here, provides a way for the participant to affect, i.e. control and modify, the perspective of his experience in the teleconference without the use of a heads-up display, previously discussed. The participant may manipulate his perspective in the virtual space by appropriate control and manipulation of his tracking element; in the example of a keyboard, this may be accomplished through manipulation of one or more keys and/or functions of the keyboard. For instance, the participant may simulate looking left in the virtual environment space by manipulating the left arrow key, while looking to the right in the space may be accomplished by use of the right arrow key. Other suitable tracking elements may include a sensor, joystick, mouse, PDA, stylus, a peripheral device, or another other technology capable of tracking the perspective of the participant, including the direction and/or orientation of movement of the body of the participant that can affect the perspective of the virtual environment that is experienced by the participant. As previously mentioned, tracking in this embodiment is not also performed by the capture element. The data used to update the perspective of the local and remote participant are referred to as the user perspective modification data, as has been described.

It is noted that user perspective modification data refers to any orientation or movement of the participant being monitored that may affect what is experienced by the participant in the virtual environment of the teleconference. User perspective modification data may thus include movement of the head or a re-orientation, such as turning, of the participant's body. For instance, if the virtual environment of the teleconference is to provide the sense that participants1and2are seated across from each other at a virtual conference table, then the acts of participant1moving his head, standing up, leaning forward towards participant2in the virtual environment, etc. may each be expected to change what is seen or heard by participant1, as well as that experienced by participant2, in the virtual environment, and thus the perspective experienced by participant1in the virtual environment is said to have changed. Thus, tracking movement, re-orientation, or other perspective data of a participant by the tracking element1305,1355provides one of the types of data that is used to process and generate a believable teleconference at the virtual location for the participant.

The received encoded feature data, video in many cases, of one or more remote participants is decoded and rendered onto the representation of the remote participant(s) in the local scene. This decoded data is rendered with the appropriate context and orientation that is determined by the tracking elements and the environment. Data about the one or more participants at remote locations is additionally used to generate a virtual representation of the remote participants in the virtual environment from the perspective of a local participant. Receive elements1306,1356, functionally associated with participants1and2, respectively, receive captured feature data captured from one or more remote participants, encoded by encoder/decoder element1303,1353, respectively, and transmitted over the system by remote transmit elements1312,1362as described above. Thus, for the simplified system ofFIG. 13, receive element11306will receive from transmit element1362encoded A/V data processed and encoded by processing element21352and encoder/decoder1353from the video recorded by capture element1354from Participant2and transmitted by transmit element21362. Similarly, receive element21356will receive from transmit element11312encoded data processed and encoded by processing element11302and encoder/decoder11303from the video recorded by capture element1304from Participant1and transmitted by transmit element11312. Again, the system ofFIG. 13is scalable, meaning there may be more than two participants, in which case the encoded feature data received by receive elements1306,1356would be from more two or more remote participants.

With the receipt of the encoded recording by the receive element associated with a participant and the user perspective modification tracking data from the tracking element associated with that participant, the local processing element now has enough information to generate one or more virtual representations of the one or more remote participants in the virtual environment from the local participant's perspective. In addition to the captured feature data, such as video recording, of a remote participant that is processed and encoded to produce encoded feature data that is transmitted to and received by the local receive element, the processing element associated with the local participant has the perspective modification data provided by and captured from the local participant in response to user manipulation and control of a tracking element, a model of the remote participant, and information that is representative of the visual and audio configuration of the environment of the virtual location at which the virtual teleconference takes place. It should be noted that the user perspective modification tracking data need only be provided by the tracking element to the processing element upon some change in this data, as triggered by any changes to the tracking element exercised by the user. The processing element processes this information to generate a virtual representation of the remote participant in the virtual environment as seen from the perspective of the local participant. This processing may be performed by the processing element to generate virtual representations from the perspective of the local participant in the virtual environment for each remote participant that transmits its visual and/or audio information to the local receiver element.

Encoded, feature data, as in the streaming A/V format, for example, of the remote participant may be put together with a model of the remote participant (1308,1358) that is stored and accessible to the processing element associated with the local participant. In this case, the received encoded feature data is combined with the stored model of the remote participant to render the avatar of the remote participant that is displayed to the local participant on display element1316,1366. The stored model may be a two-dimensional or three-dimensional (3D) computer model upon which the received A/V format data may be used to update or enhance the model. The model may additionally be just the head, bust or some larger model of the remote participant. It may be that only the head or face portion of the model is individual to the remote participant, with the rest of the virtual representation of the remote participant being supplied by a stock avatar not specific to the particular remote participant. The portion of the virtual representation of the remote participant that is individualized by the use of the participant-specific model1308,1358may well be affected by factors such as the amount and quality of streaming A/V data that is collected and the amount of processing power and time to be dedicated to this task. If only eye, mouth, and face A/V data information are captured from the remote participant, then it would be sufficient to store only a participant-specific model of the head of the remote participant upon which the captured data may be overlaid, for example. An example of a 3D model is described in conjunction withFIG. 9, discussed above.

Information about the environment1310,1360of the virtual location where the teleconference is to take place is also processed by the local processing element1302,1352when generating the virtual representation of a remote participant. Environment data expresses the set-up of the virtual conference, with the relative positions of each of the participants within it and the visual backdrop, such as the location of a conference table, windows, furniture, etc. to be experienced by the participants while in the teleconference. Movement of a participant, by either head or body movement, or as reflected by tracking elements1305,1355, by one or more teleconference participants may change the perspective from which the participant sees this environment and so must be tracked and accounted for when generating the virtual representation of the remote participant that will be displayed to the local participant. Again, the processing element that generates the virtual representation for the local participant may be operable to generate virtual representations in this manner for each participant in the virtual teleconference for which A/V data is received.

The processing elements1and2, shown as elements1302,1303and1352,1353, respectively, need not necessarily reside at the participants' locations. Additionally, they need not necessarily be one discrete processor and may indeed encompass many processing elements to perform the various processing functions as will be described. It is further envisioned that there may be a central processing element, which may encompass both processing element11302,1303and processing element21352,1353and which may further be physically located in a location different from locations1300and1350. This is illustrated in block diagram1500ofFIG. 15. The processing of captured local A/V and perspective data and remote data need not be performed at local locations, such as Location1and Location2, and may indeed be provided by processing capabilities of communication network1590. The captured data of participants1and2are transmitted remotely using communications network1590. In a certain embodiment of the present invention, communications network1590is a high bandwidth, low latency communications network. For instance, data may be communicated at 20 fps with a 150 mS latency over a standard Internet IP link. Also, while communication may occur over one or more broadband links, communication is not restricted to broadband.

Models of remote participants1540,1580are shown at local locations1,2, respectively, but this is not required, particularly as the processing element or elements are to be located on the communication network; the stored model may be a 3D computer model as shown. 3D models are useful to store image information that does not rapidly change, and thereby allows the amount of data that must be transmitted across communications network1590to be reduced. After receiving remote image data, data display components1530and1560are operable to update the 3-dimensional models1540and1580used to create the virtual representation made available to the local participant.

The one or more virtual representations that have been generated in the virtual environment by the processing element are provided to a render element1314,1364that renders the computer generated data of the one or more virtual representations for display by a display element1316,1366to the local participant. As previously described, the display element may be a monitor, computer screen, or the like, or it may be any other suitable mechanisms for displaying the environment of the virtual location to the participant. In this embodiment, the display element as a computer screen, monitor or the like, provides the benefits of telepresence without the requirement of using a heads-up display in the telepresence system.

Important to maintaining the sense of actual presence or telepresence between two or more participants in the teleconference, the system has the ability to monitor or track any changes occurring with remote participants or the local participant. Any such changes will require that the virtual representation of the virtual environment and the other participants in it be changed accordingly. As described in connection with this embodiment, either or both of the local participant and the remote participant may control at any time the perspective viewed by them by manipulation of their respective tracking elements, such as through manipulation of their keyboard, stylus, joystick, mouse, peripheral device or other suitable device. This user perspective modification data, then, is data that in addition to captured feature data, model information and environment data is used to update the avatar of remote participants as seen by a local participant. Thus, upon a change in at least one of the A/V encoded feature data of a remote participant received by a local receiver element, the user perspective modification data collected from a local participant, the user perspective modification data collected from the remote participant, or a change in the environment of the virtual location itself, the one or more virtual representations of remote participant(s) that are generated are updated and the updated representations rendered and then displayed to the local participant.

Referring now toFIG. 14, flow chart1400describes an exemplary method of teleconferencing between at least two participants at a virtual location in accordance with these certain embodiments of the present invention. At block1410, user perspective modification data generated by the first participant at first location1300are captured by tracking element1305. At Block1415, the feature data, such as streaming A/V, i.e. video, of the first participant at the first location is captured by capture element1304as discussed above. This feature data of the first participant at the first location is encoded at Block1420. Next, at Block1425, the user perspective modification data generated by the second participant at the second location1350are captured by tracking element1355. At Block1430, the feature data, such as video, of the second participant at second location1350is captured by capture element1354per the discussion above. This feature data is encoded at Block1435. Next, at Block1440, the encoded feature data, such as encoded A/V feature data, from the second participant (received from transmit element1362), the perspective data of the first participant, and the environment data1310of the virtual location is processed by processing element1302and encoder/decoder1303to generate the virtual representation of the second participant from the perspective of the first participant. This is performed with the encoded feature data from the first participant (received from transmit element1312), the perspective data of the second participant, and the environment data1360with the appropriate processing elements1352,1353, to generate the virtual representation of the first participant from the perspective of the second participant at Block1445. These virtual representations may then be displayed to the appropriate participant at Blocks1450and1455. As described above, the virtual representation is first rendered by a rendering element and the rendered virtual representation is displayed to the participant on an appropriate display means, such as a computer monitor or screen.

Finally, at Blocks1460and1465, the generated virtual representations are updated upon a change in any of the data used to generate them. Thus, a change in the user perspective modification data of the local participant, the encoded feature data captured from the remote participant, or the environment information will ensure that the virtual representation is updated accordingly. At Block1465, for instance, the virtual representation of the first participant is updated upon the occurrence of a change in condition, which may be brought about upon the occurrence of several different conditions, including upon a change in at least one of the user perspective modification tracking data of the first participant, the feature data of the second participant, and environment data of the virtual location. Any updated virtual representation is displayed to the second participant. At Block1460, a similar analysis occurs, but in this instance for displaying an updated virtual representation(s) of the second participant to the first participant.

It is to be noted that in addition to the three types of data processed to generate or update a virtual representation of a participant, another, fourth type of data may also be used—the user modification perspective data of the remote participant. Consider, for example, a change in the user perspective modification data of the second participant, such as through control and manipulation of a keyboard by the second participant. This change, which may reflect a change in where the second participant is looking in the virtual space, for instance, can be expected to affect the virtual representation of the second participant that is displayed to the first participant. The converse, i.e. that a change in the user perspective modification data of the first participant may cause the virtual representation of the first participant that is displayed to the second participant to change, may also be true.

It is noted here that while the capture, tracking and processing necessary to create the virtual representation of the second participant for display to the first participant occurs prior to similar processing to generate the virtual representation of the first participant for display to the second participant, the order may be changed if so desired without departing from the spirit and scope of the invention.

Referring now toFIG. 16, it can be seen that a so-called combinational approach may be employed in connection with certain other embodiments of the present invention. In this figure, in connection with Participant1there is a system1600that is consistent with the principals described forFIGS. 1-4above. System1600communicates with System1650, which is consistent with the principals described forFIGS. 13-15above. One point of interest, however, is that because processing element1in system1600must process, i.e. decode, feature data captured by the capture element of system1650, transmitted by transmit element2, and received by receive element1, the processing element1is in communication with an decoder block as shown. Conversely, processing element2operates with encoder, as shown, in order to be able to encode the feature data captured by the video stream of participant2at Location2.

Thus, using the combinational approach shown inFIG. 16, participant1may be interfacing with the virtual teleconference by means of a heads-up display, while participant2is being recorded by the capture element and can control his respective user perspective within the virtual environment by manipulation of a keyboard, joystick, PDA, stylus, sensor or other tracking element.

Moreover, with regard to the combination system approach, reference to the flowchart1700ofFIG. 17illustrates a combinational method of teleconferencing between at least two participants at a virtual location in accordance with certain embodiments of the present invention. Blocks1725,1730,1735,1755relate to the approach discussed in connection withFIGS. 1-4while Blocks1710,1715,1720,1740, and1760relate toFIGS. 13-15.

At block1710, user perspective modification data generated by the second participant at second location are captured by tracking element. At Block1715, the feature data, such as streaming A/V, i.e. video, of the second participant at the second location is captured by capture element as discussed above. This feature data of the second participant at the second location is encoded at Block1720. At Block1725, data generated by features of the first participant and perspective data from the first participant at the first location are captured. Recognizable feature data and patterns are extracted from the captured visual data of the second participant at Block1730. Blocks1725and1730are performed in accordance with the description related toFIG. 2above, for example. It is noted that the order in which Blocks1710-1720and1725-1730may be reversed or changed without departing from the spirit and scope of the claimed invention.

Next at blocks1735and1740virtual representations of the first and second participants, respectively, are generated. It is noted that in this combinational approach ofFIGS. 16 and 17, the data used to generate or modify the virtual representations of the participants differs. As previously mentioned, Participant1's feature and perspective data is captured and managed in accordance with the description ofFIGS. 1-4while that of Participant2is more described byFIG. 14, for example.

At Block1735, a virtual representation of the first participant from the perspective of the second participant is generated. The extracted feature data of the first participant, the perspective data of the second participant and environment data of the virtual location are processed by the second processing element associated with the second participant at the second location to generate the virtual representation of the first participant from the perspective of the second participant. At Block1740, the first processing element and the decoder associated with the first participant operate to process the encoded feature data from the second participant, the perspective data of the first participant, and the environment data of the virtual location to generate the virtual representation of the second participant from the perspective of the first participant. At Blocks1745and1750, these virtual representations may be displayed. In the case of Participant1, this may be via a heads-up display as other types of displayed as has been previously described. In the embodiment described in connection with Participant2, in which a heads-up display is not used by Participant2, the representation of participant1may be displayed to Participant2by means of a computer screen, monitor or other suitable display. It is understood that the order in which the processing of blocks1735and1740occurs is not necessarily important and may vary or such processing may occur simultaneously, particularly as the processing elements associated with each participant may be distinct.

Blocks1755and1760illustrate that the virtual representations of the participants may be updated and the updated representations displayed to the respective participants. Again, the order in which this updating and displaying occurs may vary and may occur simultaneously, particularly as the processing elements associated with each participant may be distinct. At Block1755, the virtual representation of the first participant is updated and the updated virtual representation displayed to the second participant upon a change in one or more of the following conditions: the user perspective modification data of the second participant, cued data captured from the first participant, and/or the environment data of the virtual location. At block1760, the virtual representation of the second participant is updated and the updated virtual representation displayed to the first participant upon a change in one or more of the following conditions: the user perspective modification data of the second participant, encoded feature data captured from the second participant, the perspective data of the first participant, and/or the environment data of the virtual location.

Next, at Block1440, the encoded feature data from the second participant (received from transmit element1362), the perspective data of the first participant, and the environment data1310of the virtual location is processed by processing element1302and encoder/decoder1303to generate the virtual representation of the second participant from the perspective of the first participant. This is performed with the encoded feature data from the first participant (received from transmit element1312), the perspective data of the second participant, and the environment data1360with the appropriate processing elements1352,1353, to generate the virtual representation of the first participant from the perspective of the second participant at Block1445. These virtual representations may then be displayed to the appropriate participant at Blocks1450and1455. As described above, the virtual representation is first rendered by a rendering element and the rendered virtual representation is displayed to the participant on an appropriate display means, such as a computer monitor or screen.

Finally, at Blocks1460and1465, the generated virtual representations are updated upon a change in any of the data used to generate them. Thus, a change in the user perspective modification data of the local participant, the encoded feature data captured from the remote participant, or the environment information will ensure that the virtual representation is updated accordingly. At Block1465, for instance, the virtual representation of the first participant is updated upon the occurrence of a change in condition, which may be brought about upon the occurrence of several different conditions, including upon a change in at least one of the user perspective modification data of the first participant, the feature data of the second participant, and environment data of the virtual location. Moreover, the user perspective modification data of the second participant can likewise result in the virtual representation of the first participant being updated. Any updated virtual representation is displayed to the second participant. At Block1460, a similar analysis occurs, but in this instance for displaying an updated virtual representation(s) of the second participant to the first participant.

It is noted here that while the capture, tracking and processing necessary to create the virtual representation of the second participant for display to the first participant occurs prior to similar processing to generate the virtual representation of the first participant for display to the second participant, the order may be changed if so desired without departing from the spirit and scope of the invention.

InFIG. 18, a more detailed block diagram1800of elements of telepresence communication architecture is shown, in accordance with certain embodiments of the present invention. This architecture is illustrative for the approach in which the capture element and the tracking element are separate, as discussed in connection withFIGS. 13-17. As indicated by the dashed line in the figure, the figure illustrates functionality involving data collected between first and second locations.

Referring now toFIG. 18, a more detailed block diagram of elements of telepresence communication architecture is shown, in accordance with certain embodiments of the present invention presented inFIGS. 13-17. As indicated by the dashed line in the figure, the figure illustrates functionality related to data transmitted and/or received (collected) between first and second locations. As will be clear from the following description, the functionality concerns data collected, processed and transmitted by a sender block1800at location1and data received, processed and displayed by a receiver block1850at location2. However, it will be understood that to make a completely integrated system there will need to be a receiver block and a sender block to support the participant at location1and the participant at location2. The one-directional block diagram ofFIG. 18will simplify the description to follow and an extension to a fully bi- or multi-directional will be understood by those of skill in the art. It will be noted by one of ordinary skill in the art that a telepresence communication system is operable to transmit two images in full duplex using one or more communication links; while communication may occur over one or more broadband links, communication is not restricted to broadband. Thus, a remote location may comprise a sending module1800as well as a receiving module1850. Also, a local location may comprise a sending module1800and a receiving module1850. This configuration will allow two images or virtual representations to be tracked, transmitted, received and displayed. This is of course scalable to any number of locations and participants, subject to available processing power, storage, and latency conditions.

It can be seen by means of the dashed lines in the figure, that there are three main functions being performed by sender block1800for the participant at Location1: capture and tracking, processing, and synchronizing/transmitting. The sender block1800is concerned primarily with video capture at Block1815, tracking at Block1810and audio capture at Block1805, processing of sound at Block1820and A/V at video encoder1825, and with the synchronization/transmit function at encoding/synchronization Block1830and modulator block1835locally obtained participant information. It should be noted that unlike the telepresence architecture described in connection withFIG. 4, the capture element and the tracking element are separate functional elements, thereby permitting the use of a key board or other suitable device to enable the user to easily control his perspective within the virtual environment, as reflected in the user perspective modification data.

At block180, local audio, such as what the location1participant is saying is captured; video capture occurs at Block1815. The audio and video capture functions of1805and1815may be performed by the same device. As discussed at length above, the capture element may be a videocamera or other suitable device or means for capturing the A/V information data of the participant. In this embodiment, tracking of the perspective of Participant1is performed by a separate function, shown as tracking block1810. As discussed above, the participant can readily control the perspective experienced within the virtual environment through manipulation of a tracking element, such as a keyboard, etc., as reflected in his user perspective modification data captured by tracking element1810. The processing of the A/V data occurs within processing blocks1820and1825. At Blocks1830and1835the A/V and tracking data is encoded if necessary and synchronized, and modulated, in readiness of transmission of the data to a receiver1850at location2. It is modulated at modulator1835and then transmitted for receipt by demodulator1890of the receiver block1850associated with Location2. In a certain embodiment of the present invention, this data is transmitted using a broadband link.

In accordance with certain embodiments of the present invention, a sense of eye-to-eye contact may be achieved by providing, during a transmission set-up period, a first one or more fixed dots on the image displayed to a first user and a second one or more fixed dots on the image displayed to a second user. During the transmission set-up period, the location of the eyes in the image displayed to the first participant is collocated with the first one or more fixed dots. Also during the transmission set-up period, the location of the eyes in the image displayed to the second participant is collocated with the second one or more fixed dots. This approach enables the participants to have the sense of eye-to-eye contact since the first one or more fixed dots and the second one or more fixed dots provide the expected location of the eyes displayed to the first participant and the second participant, respectively. Eye contact is maintained by the participants responding to the visual cues presented to them, as in a real-life in-person conversation.

Data received from location1by demodulator1890is demodulated and passed to a decoder1895. Decoder1895passes decoded audio and feature data of the participant at location1to sound element1870, video decoder1875, view generation block1880and model update block1885. Movement and orientation of participant2, referred to as their user perspective modification data, from tracking element1897and the A/V data and tracking data (user perspective modification data of participant1) received from participant1are processed by sound block1870to generate an audio component of a virtual representation of participant1from the perspective of participant2that can then be provided by audio element1855. Consider, for example, the following. The audio component of the virtual representation made available to participant2is affected not only by what participant1says, but also upon the orientation of participant2's body or head with respect to participant2in the virtual environment.

Encoded feature data is additionally made available to view generation block1880and model update block1885by decoder1895. It is used by model update1885to update the model of the participant at location1that is stored at block1899. In certain embodiments, model update block1885performs a facial model update that uses facial data stored in 3-D model1899to construct the virtual representation of participant1. View generation block1880generates the view or views of the virtual representation of the participant1from the perspective of participant2to be rendered by render element1865and then displayed to the participant at location2by display1860.

Telepresence telecommunications is a novel approach designed to provide a sense of presence of the person or persons with whom the communication is taking place. It is an alternative to traditional video conferencing systems that may use three-dimensional graphical avatars and animation enhancements to deliver the experience of a face-to-face conversation. Other communication methods like letter writing, telephone, email or video teleconferencing do not provide the same experience as an in-person meeting. In short, the sense of presence is absent. A telepresence teleconference attempts to deliver the experience of being in physical proximity with the other person or persons or objects with which communication is taking place.

Telepresence architecture employs an audio-visual communication system that is operable to transmit to one or more remote users a local image's likeness in three dimensions, potentially in full scale and in full color. Telepresence communication is also operable to remotely make and maintain eye contact. Mechanisms that contribute to the sense of presence with a remote person are the provision for a high resolution display for one or more specified regions of an image, the ability to track participant movements within one or more specified regions of an image, and the ability to update changes in local and/or remote participant changes in near real-time. The telepresence architecture enables one or more participants to receive likeness information and render the information as a three-dimensional image from the perspective of the local participant to a display unit, providing proper tracking and refresh rates.