Source: http://www.google.com/patents/US5736982?dq=6,034,652
Timestamp: 2014-03-08 06:22:20
Document Index: 757845224

Matched Legal Cases: ['art 51', 'art 52', 'art 51', 'art 52', 'art 52', 'art 51', 'art 52', 'art 52', 'art 51', 'art 51', 'art 12', 'art 51', 'art 12', 'art 12', 'art 12', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'arts 52', 'arts 52', 'art52', 'art 52', 'art 52', 'art 52', 'arts 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'arts 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'arts 52', 'art 52', 'art 52', 'art 5', 'art 52', 'art 52', 'art 5', 'art 5', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'arts 5', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 5', 'art 5', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 5', 'art 12', 'art 2', 'art 2', 'art 2', 'art 2', 'art 12', 'art 2', 'art 2', 'art 52', 'art 52', 'art 52', 'art 52', 'arts 5', 'art 52', 'arts 5', 'art 52', 'arts 5', 'arts 5', 'arts 5', 'arts 5', 'art 52', 'arts 5', 'art 52', 'arts 5', 'art 52', 'arts 5', 'art 12', 'art 12', 'art 2', 'arts 2', 'arts 2', 'art 12', 'art 12', 'art 12', 'art 2', 'art 12', 'art 12', 'art 12', 'art 12', 'art 12', 'arts 2', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52', 'art 52']

Patent US5736982 - Virtual space apparatus with avatars and speech - Google PatentsSearch Images Maps Play YouTube News Gmail Drive More »Sign inAdvanced Patent SearchPatentsA plurality of terminals are connected to a server via a communication network and share a predetermined common virtual space. The terminals each send to the server the position coordinates of the viewing point and direction of eyes of its user in the virtual space, and the visual field image viewed...http://www.google.com/patents/US5736982?utm_source=gb-gplus-sharePatent US5736982 - Virtual space apparatus with avatars and speechAdvanced Patent SearchPublication numberUS5736982 APublication typeGrantApplication numberUS 08/509,091Publication dateApr 7, 1998Filing dateAug 1, 1995Priority dateAug 3, 1994Fee statusPaidAlso published asCA2155254C, DE69521369D1, DE69521369T2, DE69525551D1, DE69525551T2, DE69531009D1, DE69531009T2, EP0696018A2, EP0696018A3, EP0696018B1, EP0942396A2, EP0942396A3, EP0942396B1, EP0942397A2, EP0942397A3, EP0942397B1Publication number08509091, 509091, US 5736982 A, US 5736982A, US-A-5736982, US5736982 A, US5736982AInventorsHiroyuki Arita, Machio Moriuchi, Yumi Murakami, Yoshio Nagashima, Yasuhiro Nakajima, Shohei Sugawara, Gen Suzuki, Hiroya TanigawaOriginal AssigneeNippon Telegraph And Telephone CorporationExport CitationBiBTeX, EndNote, RefManPatent Citations (9), Non-Patent Citations (24), Referenced by (178), Classifications (18), Legal Events (4) External Links: USPTO, USPTO Assignment, EspacenetVirtual space apparatus with avatars and speechUS 5736982 AAbstract A plurality of terminals are connected to a server via a communication network and share a predetermined common virtual space. The terminals each send to the server the position coordinates of the viewing point and direction of eyes of its user in the virtual space, and the visual field image viewed from that viewing point is displayed on a display. Based on the position coordinates and direction of eyes of the avatar each of the other terminals received from each of the other terminals via the server, each terminal generates an avatar image in the specified direction and at the specified position and displays it in the visual field. The server is always supplied with the latest position information of the avatar from every terminal and, when the distance between two arbitrary avatars becomes smaller than a threshold value, connects speech channels of the two terminals corresponding to these avatars.
The system of the present invention can be designed as either a distributed connection type system or a centralized one. In the distributed connection type system, as shown in FIG. 1A, a plurality of terminals 10.sub.1, 10.sub.2 and 10.sub.3 connected to the thick-lined communication network such as a LAN (local area network) are each adapted to form a common virtual space and to send and receive data to and from the other terminals as indicated by the thin-lined arrows. Each terminal sends data representing the position of the avatar of the user in the virtual space and data representing the direction of eyes of the avatar (hereinafter referred to as position information) to all the other terminals at regular time intervals or when the position or direction data changes. Upon receiving the position data and direction-of-eye data from other terminals, each terminal checks the data to see if the avatars of the other terminal users exist in the visual field of its avatar, and if so, the terminal displays the avatar images of the other terminal users at the positions specified by the position data received. Moreover, as explained with reference to an embodiment described later on, each user sends his voice or speech from his terminal to all the other terminals, and as described later in respect of another embodiment, if necessary, the user sends, for example, his facial video to other terminals by request. FIG. 1B shows a distributed connection type system where the LAN includes ISDN.
In the centralized connection type system, as depicted in FIG. 2A, the terminals 10.sub.1, 10.sub.2 and 10.sub.3 are all connected to a server 50 via a communication network such as a LAN and perform two-way communication with the server 50 as indicated by the thin-lined arrows. In this instance, each terminal sends at least the position information of the avatar of its user to the server 50; the server 50 performs required processing on the basis of the position information received from each terminal and sends the processed position information to all the terminals 10.sub.1, 10.sub.2 and 10.sub.3. FIG. 2B shows the case where the terminals 10.sub.1, 10.sub.2 and 10.sub.3 are all connected to the server 50, for example, via ISDN.
FIG. 4A schematically illustrates the architecture of a virtual space VS provided beforehand for the terminal control device 12 of the terminal unit 10.sub.1 of a user U1, positions P1 and P2 (given as coordinate values) of avatars A1 and A2 of users in the virtual space VS and the directions of eyes (indicated by the arrows ED1 and ED2) of the avatars A1 and A2. Moreover, position P1' indicates the position of the avatar A1 having moved thereto and the direction of eye at the position P1' is indicated by the arrow ED1'. On the other hand, FIG. 4B shows a visual field image that the avatar A1 observes in the direction ED1 from the position P1; this visual field image is displayed on the display 13 of the terminal unit 10.sub.1 of the user U1. FIG. 4C shows a visual field image that the avatar A1 in FIG. 4A observes at the position P1' after having moved thereto, the direction of its eyes being indicated by the arrow ED1'.
When the user U1 instructs, by a joystick or similar control device 14 of his terminal 10.sub.1, his avatar in the virtual space VS to move rightward from the position P1 to the position P1' as shown in FIG. 4A, the terminal control device 12 responds to the "move" instruction to display on the display 13 the visual field image in the virtual space VS viewed from the new position P1' (FIG. 4C) in place of the visual field image from the position P1 displayed until then (FIG. 4B), and the control device 12 sends the new position P1' from the interface 11 to the server 50 via the communication network NW. The avatar image A1 representing the user U1 in the virtual space VS is not displayed on the display 13 of the terminal 10.sub.1 of his own. In this embodiment, the avatar image A2 of the other user U2 is displayed in the visual field image viewed from the viewing point P1' (FIG. 4C).
The server 50 has, as shown in FIG. 5, a channel interface part 51, a connection control part 52 and a table memory 53. The channel interface part 51 is connected via the communication network NW to the terminal units 10.sub.1 and 10.sub.2, receives therefrom the position information of their avatars, that is, the viewing points P1 and P2 and directions of eyes ED1 and ED2 of the avatars A1 and A2, transmits the position information to all terminals except the transmitting one and controls audio and video channel connection between the terminals specified by the connection control part 52. The connection control part 52 writes the received position information, that is, the coordinates of the positions (the virtual space is three-dimensional and the position of each avatar is expressed by three-dimensional coordinates but will hereinafter be expressed by two-dimensional coordinates (x, y)) and the directions of eyes ED in a position information table 53A of the table memory 53 in correspondence with the respective terminals. According to the present invention, when the relationship between two arbitrary avatars satisfies a predetermined condition after the updating of the data stored in the memory 53, the terminals corresponding to the two avatars are connected via the channel interface part 51 to enable communication or conversation between the users of these terminals. The conversation enable condition consists of, for example, the distance between the avatars and the degree of eye-to-eye contact between them as described later with reference to other embodiments. The connection control part 52 calculates the distance d between the avatars A1 and A2, for example, in the table 53A by d.sup.2 =(x.sub.1 -x.sub.2).sup.2 +(Y.sub.1 -Y.sub.2).sup.2, and when d&lt;D (where D is a predetermined value) and the degree of eye-to-eye contact defined by the directions of eyes ED1 and ED2 of the avatars A1 and A2 satisfies a predetermined condition, the connection control part 52 instructs the channel interface part 51 to connect the channel between the terminals 10.sub.1 and 10.sub.2 corresponding to the avatars A1 and A2 and writes the state of connection (indicated by a white circle) of the avatars A1 and A2 in a state-of-connection table 53B.
The channel interface part 51 relays processed audio and video data between the terminals 10.sub.1 and 10.sub.2, that is, sends the data received from the terminal 10.sub.1 to the terminal 10.sub.2 and the data received from the latter to the former.
The terminal control part 12 of the terminal 10.sub.1 decodes the audio data received from the terminal 10.sub.2 via the channel interface part 51 and outputs the sound from the speaker SP, creates the avatar image at the position specified by the coordinate value (x.sub.2, Y.sub.2) contained in the position information received from the terminal 10.sub.2 and outputs it to the display 13 in combination with the visual field image in the virtual space being currently displayed. Similarly, the terminal 10.sub.2 processes and outputs the audio and video data received from the terminal 10.sub.1.
In the management table memory 12E, as shown in FIG. 8, there are stored the position coordinate COV and direction of eyes γ of the user's avatar inputted from the control input processing part 12D through the manipulation of the control device 14, the position coordinates COV and direction of eyes γ of other avatars, change flags CFLG and state flags SFLG received from the server 50 (or other terminals); these pieces of information are stored in correspondence with the respective avatar identifiers AID. The avatar identifier AID, the state flag SFLG, the coordinate value COV and the direction of eyes ED are the same as those used in the "move" message depicted in FIG. 6. When these pieces of data on avatars are updated, the change flag CFLG is set to "1." Now, a description will be given of the operation of the terminal control part 12 in the terminal 10.sub.1 of the user U1, for instance. The CPU 12C reads out of the management table memory 12E the position (x,y,z) and direction of eyes γ.sub.1 corresponding to the identifier AID.sub.1 of the avatar A1 of the user U1, instructs the video image generating part 12G to generate the visual field image observed in the direction of eyes γ.sub.1 from the position (x,y,z) in the virtual space stored as data in the file unit 12F, detects other avatars present in that field of vision from their coordinate values stored in the table memory 12E, generates avatar images at the positions of the detected avatars and instructs the display 13 to display thereon the avatar images in combination with the visual field image. The avatar images that are displayed on the display 13 are, for example, video images of users' faces received from the server 50 and produced in sizes corresponding to the distances from the avatar A1 of the user U1 to the other avatars to be displayed.
(a) When the distance d between the avatars A1 and A2 given their position coordinates is equal to or smaller than a predetermined value Da as shown in FIG. 9A, the server 50 interconnects the terminals 10.sub.1 and 10.sub.2 corresponding to the avatars A1 and A2, enabling transmission and reception of speech between them. In the system like this, the direction of eyes is not taken into account.
(b) When the distance d between the avatars A1 and A2 is equal to or smaller than a predetermined value Db and at least one of the avatars is in the field of vision of the other avatar as shown in FIG. 9B, the server 50 interconnects the two corresponding terminals, enabling transmission and reception of speech between them. The visual angle α is a predetermined value. In the example FIG. 9B, the avatar A1 of the user U1 is not displayed on the display unit 13 of the terminal 10.sub.2 of the user U2 but the avatar A2 of the user U2 is displayed on the terminal display unit 13 of the user U1; hence, the avatar A1 can start conversation with the avatar A2 by calling to it.
(d) In such a situation as shown in FIG. 9D wherein a third avatar A3 approaches one (A1, for example) of the avatars A1 and A2 engaged in conversation with each other in a system utilizing the above-mentioned condition (a) and a conversation enable condition (d≦Dd) is also satisfied between the avatars A1 and A3 as shown FIG. 9D, voices of the avatars A1 and A2 are sent to the terminal 10.sub.3 of the avatar A3 after being mixed, voices of the avatars A1 and A3 are sent to the terminal 10.sub.2 of the avatar A2 after being mixed and voices of the avatars A2 and A3 are sent to the terminal 10.sub.1 of the avatar A1 after being mixed so as to enable conversation among the avatars A1, A2 and A3 of the three terminal users.
The server 50 swaps speech data and position information with the terminals via channels CH.sub.1, CH.sub.2 and CH.sub.3. At first, the data received via the channels CH.sub.1, CH.sub.2 and CH.sub.3 are received in channel interface parts INF.sub.1, INF.sub.2 and INF.sub.3, respectively. The channel interface parts INF.sub.1 to INF.sub.3 analyze the received data and, in the case of speech data, transfer it to a speech switching part 52S. When the received data is position information containing position data and direction-of-eyes data, the channel interface parts INF.sub.1 to INF.sub.3 write the position data and the direction-of-eyes data in the table memory 53 and, at the same time, transfer them to a position information distributing part 52A. The position information distributing part 52A copies the position information received from the channel interface part INF.sub.1 and transfers it to the channel interface parts INF.sub.2 and INF.sub.3 and copies the position information received from the channel interface part INF.sub.2 and transfers it to the channel interface parts INF.sub.1 and INF.sub.3. Furthermore, the position information distributing part 52A copies the position information and direction-of-eyes information received from the channel interface part INF.sub.3 and transfers them to the channel interface parts INF.sub.1 and INF.sub.2. A distance decision part 52B reads out the position information from the table memory 53 and calculates all distances d.sub.ij (i,j:1,2,3, i≠j) between avatars A.sub.i and A.sub.j. The distance decision part 52B compares each distance d.sub.ij with a predetermined threshold value D and sets the distance d.sub.ij to a value 1 or 0, depending on whether 0&lt;d.sub.ij ≦D or d.sub.ij &gt;0, and transfers the value to a mixing object determining part 52D. An eye contact decision part 52C uses the position data and the direction-of-eyes data to calculate a value w.sub.ij which indicates whether either of the avatars are in the field of vision of the other. That is to say, the eye contact decision part 52C sets the value w.sub.ij to "1" or "0," depending on whether or not the viewing points (the positions of avatars) of two users Ui and Uj are each in the field of vision of the other, and transfers the value w.sub.ij to a mixing object determining part 52D. The mixing object determining part 52D calculates the product, p.sub.ij =d.sub.ij w.sub.ij, of the values d.sub.ij and w.sub.ij and instructs a switch 52S to connect the speech of the user Ui, for which the above-noted product is "1," to the channel CHj of the user Uj and the speech of the user Uj to the channel CHi of the user Ui.
Now, a description will be given, with reference to FIGS. 11 and 12, of the principles of decision in the distance decision part 52B and the eye contact deciding part 52C. As shown in FIG. 11, the distance decision part 52B comprises a distance calculating part 52B1 for calculating the distance between two avatars and a comparison part 52B2 for making a check to see if the calculated distance is within the threshold value D. The eye contact deciding part 52C comprises direction-of-avatar calculating parts 52C1 and 52C3 each of which calculates the direction of one of two avatars from the other, comparison parts 52C2 and 52C3 which compare calculated directions θ.sub.i and θ.sub.j with a predetermined visual-field threshold value α to determine if either of the avatars are in the field of vision of the other, and a coincidence deciding logical operation part52C5 which uses the two results of comparison to determine if the two avatars establish eye-to-eye contact.
As shown in FIG. 12, a coordinate axis is set in the virtual space VS; let the coordinates of the position P.sub.i of the avatar A.sub.i be (x.sub.i,y.sub.i) and the coordinates of the position p.sub.j of the avatar A.sub.j be (x.sub.j,y.sub.j). Furthermore, let the direction-of-eyes vector i of the avatar A.sub.i be (i.sub.x,i.sub.y) and the direction-of-eyes vector j of the avatar A.sub.j be (j.sub.x,j.sub.y). Incidentally, the direction-of-eyes vector is a unit vector.
The distance between the avatars A.sub.i and A.sub.j can be calculated by the following equation on the basis of the position coordinates (x.sub.i,y.sub.i) and (x.sub.j,y.sub.j) inputted into the distance calculating part 52B1.
d.sub.ij ={(x.sub.j -x.sub.i).sup.2 +(y.sub.j -y.sub.i).sup.2 }.sup.1/2(1)
The distance d.sub.ij is compared with the predetermined threshold value D, and as referred to previously, the distance d.sub.ij is set to a value "1" or "0," depending on whether 0&lt;d.sub.ij ≦D or d.sub.ij &lt;D. The distance value d.sub.ij thus set is transferred to the mixing object determining part 52D.
The coincidence deciding logical operation part 52C calculates a value w which indicates whether the fields of vision of users coincide with each other, on the basis of the position coordinates (x.sub.i,y.sub.i) and (x.sub.j,y.sub.j) and the direction-of-eyes vectors (i.sub.x,i.sub.y) and (j.sub.x,j.sub.y) inputted into the direction-of-avatar calculating parts 52C1 and 52C3.
COSθ.sub.i can be determined by calculating the inner product of the vector i and the vector P.sub.ij from the coordinate P.sub.i to P.sub.j.
i  -y.sub.i)
where which is expressed by Eq. (1). Therefore, the direction of existence θ.sub.i of the avatar A.sub.j viewed from the avatar A.sub.i can be calculated by the following equation: ##EQU1##
This calculation can also be conducted using the distance d.sub.ij calculated in the distance calculating part 52B1.
The direction θ.sub.i thus calculated by the direction-of-avatar calculating part 52C1 is compared with the predetermined visual-field threshold value α in the comparison part 52C2, and it is set to a value "1" or "0," depending on whether 0&lt;θ.sub.i ≦α or α&lt;θ.sub.i. The thus set value θ.sub.i is inputted into the coincidence deciding logical operation part 52C5.
Similarly, the direction of existence θj of the avatar A.sub.i viewed from the avatar A.sub.j can be determined by the following equation: ##EQU2## The direction θ.sub.i thus calculated in the calculating part 52C3 is compared with the predetermined visual-field threshold value α, and it is set to a value "1" or "0," depending on whether o&lt;θ.sub.j ≦α or α≦θ.sub.j. The thus set value θ.sub.i is inputted into the coincidence deciding logical operation part 52C5. FIG. 12 shows that the avatar A.sub.j is in the field of vision α of the avatar A.sub.i, whereas the avatar A.sub.i is out of the field of vision α of the avatar A.sub.j and hence is not recognized. A preferable value of the visual angle α is 45 degrees, for instance.
The direction θ.sub.i of the avatar A.sub.j viewed from the avatar A.sub.i and the direction θ.sub.j of the avatar A.sub.i viewed from the avatar A.sub.j, outputted from the comparison parts 52C2 and 52C4, respectively, are inputted into the coincidence deciding logical operation part 52C5, wherein a logical product w.sub.ij =θ.sub.i outputs a value w.sub.ij =1 which expresses the coincidence of the fields of vision of the users only when θ.sub.i indicating either of the avatars is in the field of vision of the other. When either one of the value θ.sub.i and θ.sub.j is "0," the logical operation part 52C5 outputs a value w.sub.ij =0. The output w.sub.ij from the logical operation part 52C5 is transferred to the mixing object determining part 52D. The mixing object determining part 52D uses the set value d.sub.ij from the distance decision part 52B and the set value w.sub.ij from the eye contact deciding part 52C to calculate P.sub.ij =d.sub.ij provides it to the switching part 52S.
The switching part 52S responds to the instruction of the mixing object determining part 52D select from voices received from the channel interface parts INF.sub.2 and INF.sub.3 those voices which satisfy a condition p.sub.23 =1, that is, those voices which are to be connected to the channel CH.sub.1 accommodated in the channel interface part INF.sub.1 ; the voices thus selected are mixed by a mixer 52M.sub.1 and the mixed output is provided to the channel interface part INF.sub.1. Of voices received from the channel interface parts INF.sub.1 and INF.sub.3, those voices which satisfy a condition p.sub.13 =1, that is, those voices which are to be connected to the channel CH.sub.2 accommodated in the channel interface part INF.sub.2, are selected and mixed by a mixer 52M.sub.2, thereafter being transferred to the channel interface part INF.sub.2. Similarly, of voices received from the channel interface parts INF.sub.1 and INF.sub.2, those voices which satisfy a condition p.sub.12 32 1, that is, those voices which are to be connected to the channel CH.sub.3 accommodated in the channel interface part INF.sub.3, are selected and mixed by a mixer 52M.sub.3, thereafter being transferred to the channel interface part INF.sub.3.
The channel interface parts INF.sub.1, INF.sub.2 and INF.sub.3 provide on the channels CH.sub.1, CH.sub.2 and CH.sub.3 the position information containing the position data and the direction-of-eyes data, received from the position information distributing part 52, and the speech data received from the mixers 52M.sub.1, 52M.sub.2 and 52M.sub.3.
In the case of a system which implements the aforementioned conversation enable conditions (a) and (d), the eye contact deciding part 52C in FIG. 10 need not be provided and the mixing object determining part 52D controls the switch 52S on the basis of the distance d.sub.ij alone. The conversation enable conditions (b) and (e) can be implemented by ORing, w.sub.ij =θ.sub.i +θ.sub.j, in the coincidence logical operation part 52C5 in FIG. 11.
In the above embodiments, even if the number of avatars engaged in conversation is three or more, they can each hear voices of all the other avatars in that group but cannot hear voices of an avatar who stays out of the group. This will be described in respect of such a party as shown in FIG. 13 wherein there attend in the virtual space VC seven avatars A1 to A7 corresponding to users of seven terminals 10.sub.1 to 10.sub.7. As shown, the users of the avatars A1 and A2 are talking with each other and the users of the avatars A3 to A5 are also talking with one another, but the users of the avatars A6 and A7 are not engaged in the conversation of either group. If the users of the avatars A6 and A7 could hear voices of both groups as environment sounds, they would feel the existence of the other avatars in the virtual space VC like in the real world. Similarly, the users of the avatars A1 and A2 engaged in conversation could also experience enhanced realism of the virtual space VC if they could hear, as environment sounds, the voices of the avatars A3 to A5 and sounds made by the avatars A6 and A7.
FIG. 14 shows how the speech path switch 52S and the mixer 52M in the server 50 of the apparatus of this embodiment are interconnected in the case of FIG. 13. Let it be assumed that the terminals 10.sub.1 to 10.sub.7 of the seven users are present in the same virtual space and that the group of two users corresponding to the terminals 10.sub.1 and 10.sub.2 and the group of three users corresponding to the terminals 10.sub.3, 10.sub.4 and 10.sub.5 are engaged in conversation independently of each other. In this embodiment, the mixer 52 is adaptively divided into mixing parts 52M.sub.A and 52M.sub.B corresponding to the two conversation groups, respectively, and a mixing part 52M.sub.C for all the avatars in the virtual space VC.
The switch 52S has a construction which one-way connects sounds and voices received from all the terminals 10.sub.1 to 10.sub.7 to the mixing part 52M.sub.C. A sound S.sub.C, produced by mixing the sounds and speech data D.sub.1 to D.sub.7 thus one-way transmitted from all the terminals 10.sub.1 to 10.sub.7, is attenuated by a loss inserting part 5L.sub.C down to a level appropriate for an environment sound and transmitted to the terminals 10.sub.6 and 10.sub.7 of the users who are not engaged in conversation. In this way, the users of the terminals 10.sub.6 and 10.sub.7 can hear, as environment sounds, the sounds made and voices uttered by all the users present in the shared virtual space.
On the other hand, the switch 52S two-way connects the speech data SD.sub.1 and SD.sub.2 sent from the terminals 10.sub.1 and 10.sub.2 to the mixing part 52M.sub.A and, at the same time, it one-way connects the mixed sound S.sub.C, inputted from the mixing part 52M.sub.C via the loss inserting part 5L.sub.C, to a loss inserting part 5L.sub.A to attenuate to such a sound pressure level as not to hinder conversation, after which the mixed sound S.sub.C is provided to the mixing part 52M.sub.A. The mixing part 52M.sub.A mixes the speech data SD.sub.2 from the terminal 10.sub.2 and the environment sound S.sub.C and sends the mixed sound to the terminal 10.sub.1 via the switch 52S; furthermore, the mixing part 52M.sub.A mixes the speech data SD.sub.1 from the terminal 10.sub.1 and the environment sound S.sub.C and sends the mixed sound to the terminal 10.sub.2 via the switch 52S. Thus, the users of the terminals 10.sub.1 and 10.sub.2 are capable of hearing the environment sound S.sub.C of the reduced sound pressure level while at the same time carrying on a two-way conversation as in the case of talking to each other over the telephone. As regards the group of the terminals 10.sub.3 to 10.sub.5, too, the output from the mixing part 52M.sub.C is similarly connected to the mixing part 52m.sub.b via the loss inserting parts 5L.sub.C and 5L.sub.B, and the mixing part 52M.sub.B generates speech data to be sent to each terminal by mixing speech data from all the other terminals and the environment sound S.sub.C and sends it to the terminals via the switch 25S, enabling the users of the terminals to hear the environment sound S.sub.C of the lowered sound pressure level while carrying on two-way conversation.
Turning now to FIG. 15, the server 50, which is provided with the switch 52S and the mixer 52M shown in FIG. 14, will be further described. Let it be assumed, for the sake of brevity, that the number of terminals is three and that the users of the terminals 10.sub.1 and 10.sub.2 are talking with each other, leaving the user of the terminal 10.sub.3 alone. In FIG. 14 the interface INF and the switch 52S are two-way connected, but in FIG. 15 the channel interface parts INF.sub.1 to INF.sub.3 and the switch 52S are shown to be one-way connected with a view to showing the kinds of speech data that are transmitted and received between them. In FIG. 15, the virtual space and respective terminals transmit audio data and position information data via an advanced information system INS network and the channels CH.sub.1 to CH.sub.3 in the server 50. At first, pieces of data received via the channels CH.sub.1 to CH.sub.3 are received in the channel interface parts INF.sub.1 to INF.sub.3, respectively. The channel interface part INF.sub.1 analyzes the received data and, if it is speech data SD.sub.1, transfers it to the switch 52S. Likewise, the channel interface parts INF.sub.2 and INF.sub.3 analyze the received data and, if they are speech data SD.sub.2 and SD.sub.3, transfer them to the switch 52S.
When the received data is position data and direction-of-eyes data, the channel interface parts INF.sub.1 to INF.sub.3 transfer these pieces of data to the position information distributing part 52A and write them into the table memory 53. The position information distributing part 52A copies the position data and direction data received from the channel interface part INF.sub.1 and transfers them to the channel interface parts INF.sub.2 and INF.sub.3. Similarly, the position information distributing part 52A copies the position data and direction data received from the channel interface part INF.sub.2 and transfers them to the channel interface parts INF.sub.1 and INF.sub.3 and copies the position data and direction data received from the channel interface part INF.sub.3 and transfers them to the channel interface parts INF.sub.1 and INF.sub.2.
A conversation monitoring part 52D discriminates a group of avatars that satisfies the aforementioned predetermined conversation enable conditions on the basis of the position data and direction-of-eyes data read out of the table memory 53 and defines or specifies in the mixer 52M the mixing part 52M.sub.A which mixes speech data from the terminals corresponding to the avatars of the group and the mixing part 52M.sub.B which generates an environment sound from speech data from the terminals corresponding to all avatars in the virtual space. The conversation monitoring part 52D controls the switch 52S to supply the mixing part 52M.sub.A with the speech data SD.sub.1 and SD.sub.2 received from the terminals 10.sub.1 and 10.sub.2 corresponding to the avatars of the discriminated group and the mixing part 52M.sub.B with the speech data SD.sub.1, SD.sub.2 and SD.sub.3 from all the avatars. Thus, the switch 52S transfers the speech data SD.sub.1 to SD.sub.3 received from the channel interface parts INF.sub.1 to INF.sub.3 to the mixing part 52M.sub.B. The mixing part 52M.sub.B mixes the speech data SD.sub.1 to SD.sub.3 and transfers the mixed sound S.sub.B as an environment sound to the switch 52S via a loss inserting part 5L.sub.B. The switch 52S sends the environment sound S.sub.B to the channel interface part INF.sub.3 corresponding to the terminal 10.sub.3 of the user not engaged in conversation and, at the same time, provides the sound S.sub.B via a loss inserting part 5L.sub.A to the mixing part 52M.sub.A. The mixing part 52M.sub.A mixes the sound S.sub.B with the speech data SD.sub.1 and SD.sub.2 from the channel interface parts INF.sub.1 and INF.sub.3, respectively, and sends the mixed sounds SD.sub.1 +SD.sub.B and SD.sub.2 +SD.sub.B to the channel interface parts INF.sub.2 and INF.sub.1, from which they are sent to the terminals 10.sub.2 and 10.sub.1, respectively.
As the conversation enable condition for the conversation monitoring part 52D to identify the avatars of the conversation group, it is possible to use the aforementioned conditions such as the distance between the avatars of the users, their mutual existence in the field of vision of the other, or a combination thereof. When the avatars of the conversation group end the conversation and enter a state in which the conversation enable condition is 5 not satisfied, the conversation monitoring part 52D cuts off the paths from the channel interface parts INF.sub.1 and INF.sub.2 to the mixing part 52M.sub.A and controls the switch 52S to send the environment sound S.sub.B from the mixing part 52M.sub.B to the channel interface parts INF.sub.1 to INF.sub.3 via the loss inserting part 5L.sub.B.
The FIG. 15 embodiment has been described as being applied to the centralized connection type system; in the case of the distributed connection type system, as depicted in FIG. 16 (wherein no video-related parts are shown), position information of avatars received from respective terminals is written into a table memory 12E. A conversation monitoring part 12T controls a switch 12W to supply a mixing part 2M.sub.A with voice data received from the terminals corresponding to other avatars detected from their position information read out of the table memory 12E. By this, mixed voice data of all the avatars is obtained from the mixing part 2M.sub.A, and the mixed voice data is provided to a loss inserting part 2L, wherein a predetermined loss is introduced thereinto to generate the environment sound S.sub.B, which is provided to a mixing part 2M.sub.B. On the other hand, the conversation monitoring part 12T detects other avatars which satisfy the condition for conversation direction or indirectly with the avatar of the terminal concerned on the basis of the position information of other avatars and the position information of the avatar concerned set by the control device 14 and controls the switch 12W to supply the mixing part 2M.sub.B with voice data received from the terminals corresponding to the above-mentioned other avatars satisfying the conversation enable conditions. As the result of this, the mixing part 2M.sub.B mixes the voices of the other avatars engaged in conversation with the avatar of the terminal user concerned, together with the environment sound S.sub.B, and the mixed output is provided to the speaker SP.
The example of FIG. 19 employs the angle θ between the direction of eyes or line of sight EL of the avatar A1 and a straight line joining the avatar A1 and each of the other avatars. The voices from the avatars of smaller angle are preferentially graded up to higher levels of quality. This example uses five levels of quality. That is, the sound pressure level of the voice of the avatar A4 (θ=0) on the line of sight EL of the avatar A1 is increased (loss rate: 0 dB); the voices of the avatars A7 and A5 with -45 rate of -10 dB; the voices of the avatars A3 and A6 with -90 45 voices of the avatars A9 and A2 with -135 90 and the voices of the avatars (e.g. A8) with -180 135 This processing is carried out for each of all the other remaining avatars.
In the example of FIG. 20, the avatar A1 and each of the other avatars are joined by a straight line as indicated by the broken line and the line of sight of the avatar A1 is turned until it comes into alignment with the straight line and the turning angle α is calculated. The direction of rotation in this case is the direction in which the angle α decreases. Similarly, the line of sight of the other avatar is turned until it comes into alignment with the straight line and the turning angle β is calculated. The direction of turn in this case is the direction in which the angle β decreases. Then, the sum of the two turning angles, α+β=θ, is calculated. The voices of the avatars of the smaller angles are graded up to higher levels of quality. This example uses five levels of quality. That is, the sound pressure level for the avatar A4 to which the line of sight of the avatar A1 conforms (θ=0 A3 with 0 45 90 135 and -19 dB, respectively. This processing is carried out for each of all the other avatars in the virtual space.
FIG. 21 illustrates an example of the configuration of the server 50 which effects the above-described voice quality control in the centralized connection type virtual space sharing apparatus . For the sake of simplicity, the server 50 is shown to accommodate three terminals. The server 50 is connected to terminals 10.sub.1, 10.sub.2 and 10.sub.3 (see FIGS. 2A and 2B) via the channels CH.sub.1, CH.sub.2 and CH.sub.3 and receives data therefrom in the channel interface parts INF.sub.1, INF.sub.2 and INF.sub.3, respectively.
When the received data is position data and direction-of-eye data, the channel interface parts INF.sub.1 to INF.sub.3 transfer them to the position information distributing part 52A and, at the same time, write them into the table memory 53.
As in the case of the FIG. 10 embodiment, the position information distributing part 52A copies the position data and direction-of-eyes data received from the channel interface part INF.sub.1 and transfers them to the channel interface parts INF.sub.2 and INF.sub.3, copies and transfers the position data and direction-of-eyes data received from the channel interface part INF.sub.2 to parts INF.sub.1 and INF.sub.3, and copies and transfers the position data and direction-of-eyes data received from the channel interface part INF.sub.3 to parts INF.sub.1 and INF.sub.2.
A loss determining part 52E.sub.1 uses the position data and direction-of-eyes data read out of the table memory 53 to calculate, by the methods described previously with reference to FIGS. 17 to 20, the loss rates of voices of other users to be provided to the user of the terminal accommodated in the channel interface part INF.sub.1. Based on the loss rates thus determined, the loss determining part 52E.sub.1 sends loss-inserting instructions to loss inserting parts 5L.sub.12 and 5L.sub.13 corresponding to the users of the terminals accommodated in the channel interface parts INF.sub.2 and INF.sub.3. Similarly, a loss determining part 52E.sub.2 also sends loss-inserting instructions to loss inserting parts 5L.sub.21 and 5L.sub.23 corresponding to the users of the terminals accommodated in the channel interface parts INF.sub.1 and INF.sub.3. Also a loss insertion determining part 52E.sub.3 similarly sends loss-inserting instructions to loss inserting parts 5L.sub.31 and 5L.sub.32 corresponding to the users of the terminals accommodated in the channel interface parts INF.sub.1 and INF.sub.2.
The channel interface part INF.sub.1 analyses received data and, if it is speech data, transfers the speech data SD.sub.1 to the loss inserting parts 5L.sub.21 and 5L.sub.31. Likewise, the channel interface part INF.sub.2 analyses received data and, if it is speech data, transfers the speech data SD.sub.2 to the loss inserting parts 5L.sub.12 and 5L.sub.32. Also the channel interface part INF.sub.3 similarly analyses received data and, if it is speech data, then transfers the speech data SD.sub.3 to the loss inserting parts 5L.sub.13 and 5L.sub.23. By this, the above-mentioned loss is inserted in the speech data fed to each loss inserting part.
A speech mixing part 52M.sub.1 mixes the speech data with the losses inserted therein by the loss inserting parts 5L.sub.12 and 5L.sub.13 and transfers the mixed output to the channel interface part INF.sub.1, from which it is sent to the terminal 10.sub.1 via the channel CH.sub.1. A speech mixing part 52M.sub.2 mixes the speech data with the losses inserted therein by the loss inserting parts 5L.sub.21 and 5L.sub.23 and transfers the mixed output to the channel interface part INF.sub.2, from which it is sent to the terminal 10.sub.2 via the channel CH.sub.2. Similarly, a speech mixing part 52M.sub.3 also mixes speech data with losses inserted therein by the loss inserting parts 5L.sub.31 and 5L.sub.32 and transfers the mixed output to the channel interface INF.sub.3, from which it is sent to the terminal 10.sub.3 via the channel CH.sub.3.
FIG. 23 illustrates an example of the configuration of one terminal 10 for use in the distributed connection type virtual space sharing apparatus which effects the aforementioned speech quality control. In this example, the number of terminals of other users is three. In the centralized connection type system, the terminal 10 of FIG. 22 sends and receives position data, direction-of-eyes data and speech data to and from the server 50 of FIG. 21 and voices are mixed in the server 50 in correspondence with respective users. In contrast thereto, in the distributed connection type system of FIG. 23, a speech quality of control part 12Q is provided in the terminal control part 12 of each user terminal and, based on the position data and/or direction-of-eyes data received from the other terminals and stored in the table memory 12E, the sound pressure level for each of the other users' avatars is determined in a loss determining part 2E by a desired one of the methods described previously with respect to FIGS. 17 to 20; the losses thus determined are set in loss inserting parts 2L.sub.1, 2L.sub.2 and 2L.sub.3, respectively. The pieces of speech data received from the other terminals are attenuated by the losses set in the loss inserting parts 2L.sub.1 to 2L.sub.3 and then mixed by a mixer 2M, thereafter being outputted to the speaker SP. The basic principles and operations are the same as those described previously.
The FIG. 24 embodiment has a construction in which a speech quality requesting part 12R and a speech quality request analyzing part 12S are added to the FIG. 23 embodiment. The speech quality requesting part 12R is supplied with speech quality determining parameters for respective avatars which are calculated from their position data and/or direction-of-eyes data in a loss determining part 2E to determine losses, such as distances from each avatar to the others; the speech quality determining part 12R determines the necessary speech quality corresponding to each distance and provides the information to a packet assembling and disassembling part 12H. The packet assembling and disassembling part 12H assembles into a packet a signal which requests each terminal to send speech of the determined quality and sends the packet to each terminal via the channel interface part 12A. The speech quality that is defined in terms of distance can be obtained, for example, by changing the transmission rate of speech data. For instance, four distance threshold values D.sub.1 to D.sub.4 are predetermined which bear the relationship D.sub.1 &lt;D.sub.2 &lt;D.sub.3 &lt;D.sub.4. Each avatar requests another avatar with the distance d in the range of D.sub.4 &lt;d≦D.sub.1 to send speech data of a 64 Kb/s transmission rate, another avatar with the distance d in the range of D.sub.1 &lt;d≦D.sub.2 to send speech data of a 32 Kb/s transmission rate, another avatar with the distance d in the range of D.sub.3 &lt;d≦D.sub.4 to send speech data of a 16 Kb/s transmission rate and still another avatar with the distance d in the range of D.sub.4 &lt;D to send speech data of an 8 Kb/s transmission rate.
In the packet assembling and disassembling part 12H, the speech data packets received from the respective terminals in response to the requests of the terminal concerned are disassembled into speech data of the requested bit rates, which are provided to the loss inserting parts 2L.sub.1, 2L.sub.2 and 2L.sub.3, respectively, wherein they are subjected to the same processing as described above in respect of the FIG. 23 embodiment, thereafter being mixed by the mixer 2M and then provided to the speaker SP.
FIG. 26 shows display images of the visual field image that the avatar A1 observes in the virtual space depicted in FIG. 25. The broken lines indicate distance threshold values D.sub.1, D.sub.2, D.sub.3 and D.sub.4 relative to the avatar A1 (which are not displayed in practice). The avatar images in respective regions defined by these threshold values are each displayed in the quality determined as described previously.
The server 50 sends and receives position information (position coordinates and direction of eyes) and video images to and from the terminals via channels CH.sub.1, CH.sub.2 and CH.sub.3. The data received from the channels CH.sub.1, CH.sub.2 and CH.sub.3 are received in the channel interface parts INF.sub.1, INF.sub.2 and INF.sub.3, respectively. The channel interface parts INF.sub.1 to INF.sub.3 each analyze the received data and, if it is video image data, transfer it to a video storage part 52K. The video storage part 52K writes the received video image in a memory which stores video images in correspondence with terminals accommodated. When the received data is position information (position coordinates and direction of eyes), the channel interface parts INF.sub.1 to INF.sub.3 each transfer it to a position information distributing part 52A. The position information distributing part 52A copies the position information received from the channel interface part INF.sub.1 and transfers it to the channel interface parts INF.sub.2 and INF.sub.3 ; the position information distributing part 52A copies the position information received from the channel interface part INF.sub.2 and transfers it to the INF.sub.1 and INF.sub.3 ; and the position information distributing part 52A copies the position information received from the channel interface part INF.sub.3 and transfers it to the channel interface parts INF.sub.1 and INF.sub.3. When the received data is image request information, the channel interface parts INF.sub.1 to INF.sub.3 each transfers it to an image requests analyzing part 52J. The image request analyzing part 52J analyzes the received request and informs the image storage part 52K of the requested image and, at the same time, informs video processing part 52N of the requested resolution and/or the number of frames per second and the requesting terminal. The video storage part 52K reads out of its memory the requested image specified by the image request analyzing part 52N and transfers it to the video processing part 52N. The video processing part 52N converts the video image received from the video storage part 52K to the resolution and/or the number of frames per second specified by the video image request analyzing part 52J and, on the basis of the specified requesting terminal information, sends the video image to the requesting terminal via the channel interface part INF.sub.1 and the channel CH.sub.1, the channel interface part INF.sub.2 and the channel CH.sub.2, or the channel interface part INF.sub.3 and the channel CH.sub.3.
Patent CitationsCited PatentFiling datePublication dateApplicantTitleUS5467401 *Oct 12, 1993Nov 14, 1995Matsushita Electric Industrial Co., Ltd.Sound environment simulator using a computer simulation and a method of analyzing a sound spaceUS5469511 *Oct 5, 1990Nov 21, 1995Texas Instruments IncorporatedMethod and apparatus for presentation of on-line directional soundUS5513129 *Jul 14, 1993Apr 30, 1996Fakespace, Inc.Method and system for controlling computer-generated virtual environment in response to audio signalsUS5598478 *Dec 20, 1993Jan 28, 1997Victor Company Of Japan, Ltd.Sound image localization control apparatusUS5615132 *Jan 21, 1994Mar 25, 1997Crossbow Technology, Inc.Method and apparatus for determining position and orientation of a moveable object using accelerometersUS5633993 *Feb 10, 1993May 27, 1997The Walt Disney CompanyMethod and apparatus for providing a virtual world sound systemEP0634857A2 *Jun 14, 1994Jan 18, 1995International Business Machines CorporationMultimedia communication terminalEP0659006A2 *Nov 9, 1994Jun 21, 1995International Business Machines CorporationAudio conferencing systemJPH06175942A * Title not available* Cited by examinerNon-Patent CitationsReference1Benford, "A Distributed Architecture for Large Collaborative Virtual Environments", IEEE, pp. 1-9, 1993.2 *Benford, A Distributed Architecture for Large Collaborative Virtual Environments , IEEE, pp. 1 9, 1993.3Benford, S., et al., "Supporting Cooperative Work in Virtual Environments," The Computer Journal, vol. 37, No. 8, 1994, pp. 653-667.4 *Benford, S., et al., Supporting Cooperative Work in Virtual Environments, The Computer Journal, vol. 37, No. 8, 1994, pp. 653 667.5 *Carlsson C., et al., Dive A Multi User Virtual Reality System, Institute of Electrical and Electronics Engineers, pp. 394 400, Sep. 18, 1993.6Carlsson C., et al., Dive -A Multi-User Virtual Reality System, Institute of Electrical and Electronics Engineers, pp. 394-400, Sep. 18, 1993.7Caudell, T.P., et al., "Neural Modeling of Face Animation for Telecommuting in Virtual Reality," Proceedings of the Virtual Reality Annual International Symposium, Seattle, Sep. 18, 1993, pp. 478-485.8 *Caudell, T.P., et al., Neural Modeling of Face Animation for Telecommuting in Virtual Reality, Proceedings of the Virtual Reality Annual International Symposium, Seattle, Sep. 18, 1993, pp. 478 485.9Cohen et al., "Augmented Audio Reality: Telepresence/VR Hybrid Acoustic Environments", IEEE, pp. 361-364, Jul. 1993.10Cohen et al., "Virtual Gain for Audio Windows", IEEE, pp. 85-91, 1993.11 *Cohen et al., Augmented Audio Reality: Telepresence/VR Hybrid Acoustic Environments , IEEE, pp. 361 364, Jul. 1993.12 *Cohen et al., Virtual Gain for Audio Windows , IEEE, pp. 85 91, 1993.13 *Fukuda, K., et al., Hypermedia personal computer communication system: Fujitsu Habitat, Fujitsu Scientific and Technical Journal, vol. 26, No. 3, pp. 197 206, Oct., 1990.14Fukuda, K., et al., Hypermedia personal computer communication system: Fujitsu Habitat, Fujitsu Scientific and Technical Journal, vol. 26, No. 3, pp. 197-206, Oct., 1990.15Ishii, Hiroshi, et al., "Iterative Design of Seamless Collaboration Media," Communications of the ACM, vol. 37, No. 8, Aug. 1994, pp. 83-97.16 *Ishii, Hiroshi, et al., Iterative Design of Seamless Collaboration Media, Communications of the ACM, vol. 37, No. 8, Aug. 1994, pp. 83 97.17Kobayashi et al., "Virtual Conference Room: A Metaphor for Multi-User Real-Time Conferencing Systems", IEEE, pp. 430-435, Jul. 1993.18 *Kobayashi et al., Virtual Conference Room: A Metaphor for Multi User Real Time Conferencing Systems , IEEE, pp. 430 435, Jul. 1993.19Morishima et al., "Facial Expression Synthesis Based on Natural Voice for Virtual Face-to-Face Communication with Machine", IEEE, pp. 486-491, 1993.20 *Morishima et al., Facial Expression Synthesis Based on Natural Voice for Virtual Face to Face Communication with Machine , IEEE, pp. 486 491, 1993.21 *Robinett W., Interactivity and individual viewpoint in shared virtual worlds: the big screen vs. networked personal displays, Computer Graphics, vol. 28, No. 2, pp. 127 130, May, 1994.22Robinett W., Interactivity and individual viewpoint in shared virtual worlds: the big screen vs. networked personal displays, Computer Graphics, vol. 28, No. 2, pp. 127-130, May, 1994.23Wenzel et al., "A System for Three-dimensional Acoustic Visualization in a Virtual Environment Workstation", IEEE, pp. 329-337, 1990.24 *Wenzel et al., A System for Three dimensional Acoustic Visualization in a Virtual Environment Workstation , IEEE, pp. 329 337, 1990.* Cited by examinerReferenced byCiting PatentFiling datePublication dateApplicantTitleUS5812126 *Dec 31, 1996Sep 22, 1998Intel CorporationMethod and apparatus for masquerading onlineUS5926179 *Sep 29, 1997Jul 20, 1999Sony CorporationThree-dimensional virtual reality space display processing apparatus, a three-dimensional virtual reality space display processing method, and an information providing mediumUS5956028 *Sep 11, 1996Sep 21, 1999Fujitsu Ltd.Virtual space communication system, three-dimensional image display method, and apparatus thereforUS6057856 *Sep 16, 1997May 2, 2000Sony Corporation3D virtual reality multi-user interaction with superimposed positional information display for each userUS6091410 *Nov 26, 1997Jul 18, 2000International Business Machines CorporationAvatar pointing modeUS6094196 *Jul 3, 1997Jul 25, 2000International Business Machines CorporationInteraction spheres of three-dimensional objects in three-dimensional workspace displaysUS6094677 *May 30, 1997Jul 25, 2000International Business Machines CorporationMethods, systems and computer program products for providing insertions during delays in interactive systemsUS6154211 *Sep 29, 1997Nov 28, 2000Sony CorporationThree-dimensional, virtual reality space display processing apparatus, a three dimensional virtual reality space display processing method, and an information providing mediumUS6217446 *Dec 3, 1997Apr 17, 2001Kabushi Kaisha Sega EnterprisesGame device and picture processing deviceUS6222939Jun 25, 1997Apr 24, 2001Eyematic Interfaces, Inc.Labeled bunch graphs for image analysisUS6229533 *Aug 2, 1996May 8, 2001Fujitsu LimitedGhost object for a virtual worldUS6241609 *Jan 11, 1999Jun 5, 2001U.S. Philips CorporationVirtual environment viewpoint controlUS6271843 *May 30, 1997Aug 7, 2001International Business Machines CorporationMethods systems and computer program products for transporting users in three dimensional virtual reality worlds using transportation vehiclesUS6272231Nov 6, 1998Aug 7, 2001Eyematic Interfaces, Inc.Wavelet-based facial motion capture for avatar animationUS6298374 *Mar 20, 1998Oct 2, 2001Fujitsu LimitedCommunication management apparatus with user modifiable symbol movable among virtual spaces shared by user terminals to direct current user position in real world and recording medium used thereforUS6301370Dec 4, 1998Oct 9, 2001Eyematic Interfaces, Inc.Face recognition from video imagesUS6314411 *Oct 8, 1998Nov 6, 2001Pegasus Micro-Technologies, Inc.Artificially intelligent natural language computational interface system for interfacing a human to a data processor having human-like responsesUS6329986 *Feb 21, 1998Dec 11, 2001U.S. Philips CorporationPriority-based virtual environmentUS6346956 *Sep 29, 1997Feb 12, 2002Sony CorporationThree-dimensional virtual reality space display processing apparatus, a three-dimensional virtual reality space display processing method, and an information providing mediumUS6356659Jan 16, 2001Mar 12, 2002Eyematic Interfaces, Inc.Labeled bunch graphs for image analysisUS6396509 *Feb 21, 1998May 28, 2002Koninklijke Philips Electronics N.V.Attention-based interaction in a virtual environmentUS6401122 *Dec 9, 1996Jun 4, 2002Fujitsu LimitedCommunication management apparatusUS6415188Dec 23, 1998Jul 2, 2002Dennis Sunga FernandezMethod and apparatus for multi-sensor processingUS6437777Sep 26, 1997Aug 20, 2002Sony CorporationThree-dimensional virtual reality space display processing apparatus, a three-dimensional virtual reality space display processing method, and an information providing mediumUS6437778Jun 15, 1999Aug 20, 2002Fujitsu LimitedVirtual space communication system, three-dimensional image display method, and apparatus thereforUS6453294May 31, 2000Sep 17, 2002International Business Machines CorporationDynamic destination-determined multimedia avatars for interactive on-line communicationsUS6454653Mar 29, 2000Sep 24, 2002Square Co., Ltd.Game apparatus, game display control method, and computer-readable storage mediumUS6466695Aug 4, 1999Oct 15, 2002Eyematic Interfaces, Inc.Procedure for automatic analysis of images and image sequences based on two-dimensional shape primitivesUS6539354Mar 24, 2000Mar 25, 2003Fluent Speech Technologies, Inc.Methods and devices for producing and using synthetic visual speech based on natural coarticulationUS6563950Dec 21, 2001May 13, 2003Eyematic Interfaces, Inc.Labeled bunch graphs for image analysisUS6580811May 31, 2001Jun 17, 2003Eyematic Interfaces, Inc.Wavelet-based facial motion capture for avatar animationUS6625299Apr 6, 1999Sep 23, 2003Jeffrey MeisnerAugmented reality technologyUS6714661Jul 24, 2001Mar 30, 2004Nevengineering, Inc.Method and system for customizing facial feature tracking using precise landmark finding on a neutral face imageUS6731307 *Oct 30, 2000May 4, 2004Koninklije Philips Electronics N.V.User interface/entertainment device that simulates personal interaction and responds to user's mental state and/or personalityUS6734885 *Sep 3, 1999May 11, 2004Sony CorporationInformation processing apparatus, method and computer program for virtual reality transparent avatarsUS6784901Aug 31, 2000Aug 31, 2004ThereMethod, system and computer program product for the delivery of a chat message in a 3D multi-user environmentUS6816159 *Dec 10, 2001Nov 9, 2004Christine M. SolazziIncorporating a personalized wireframe image in a computer software applicationUS6817979Jun 28, 2002Nov 16, 2004Nokia CorporationSystem and method for interacting with a user's virtual physiological model via a mobile terminalUS6832178Sep 7, 2001Dec 14, 2004Dennis S. FernandezMethod and apparatus for multi-sensor processingUS6834115Aug 13, 2001Dec 21, 2004Nevengineering, Inc.Method for optimizing off-line facial feature trackingUS6853379Aug 13, 2001Feb 8, 2005Vidiator Enterprises Inc.Method for mapping facial animation values to head mesh positionsUS6876364Aug 9, 2002Apr 5, 2005Vidiator Enterprises Inc.Method for mapping facial animation values to head mesh positionsUS6910186Dec 8, 2000Jun 21, 2005Kyunam KimGraphic chatting with organizational avatarsUS6917703Feb 28, 2001Jul 12, 2005Nevengineering, Inc.Method and apparatus for image analysis of a gabor-wavelet transformed image using a neural networkUS6922664Sep 7, 2001Jul 26, 2005Dennis Sunga FernandezMethod and apparatus for multi-sensor processingUS6928087Jan 25, 2001Aug 9, 2005Telefonaktiebolaget Lm Ericsson (Publ)Method and apparatus for automatic cross-media selection and scalingUS6933938 *Sep 3, 1999Aug 23, 2005Sony CorporationInformation processing apparatus, method and computer program for virtual reality aura settingUS6940454Aug 13, 2001Sep 6, 2005Nevengineering, Inc.Method and system for generating facial animation values based on a combination of visual and audio informationUS6954906 *Sep 29, 1997Oct 11, 2005Sony CorporationImage display processing apparatus that automatically changes position of sub-window relative to main window depending on distance at watch sub window is commanded to be displayedUS6970931Sep 14, 2000Nov 29, 2005International Business Machines CorporationMethod for translucent online interactionUS6981220 *Apr 25, 2001Dec 27, 2005Sony CorporationInformation processing apparatus and method, and storage mediumUS7006616May 15, 2000Feb 28, 2006Terayon Communication Systems, Inc.Teleconferencing bridge with EdgePoint mixingUS7007849Jul 3, 2003Mar 7, 2006Metrologic Instruments, Inc.Automatically-activated hand-supportable 2-d bar code symbol reading system employing a linear laser scanning pattern generator, an automatic bar code symbol data detector, audible data capture feedback generator, and a manually-activated data transmission activation switchUS7028904Jan 12, 2003Apr 18, 2006Metrologic Instruments, Inc.Wireless bar code symbol reading system capable of automatically collecting and storing symbol character data when hand-supportable unit is operated outside of its rf data communication range, and automatically transmitting stored symbol character data when the hand-supportable unit is operated within its rf data communication rangeUS7050624Jul 24, 2001May 23, 2006Nevengineering, Inc.System and method for feature location and tracking in multiple dimensions including depthUS7050655Aug 13, 2001May 23, 2006Nevengineering, Inc.Method for generating an animated three-dimensional video headUS7076525 *Nov 16, 2000Jul 11, 2006Sony CorporationVirtual space system, virtual space control device, virtual space control method, and recording mediumUS7080124Oct 24, 2001Jul 18, 2006Amazon Technologies, Inc.Digital media resource messagingUS7093007 *Dec 21, 2001Aug 15, 2006Hewlett-Packard Development Company, L.P.Identifying a physical device's avatar using a unique, substantially non-removable communication identifierUS7111786Jan 12, 2004Sep 26, 2006Metrologic Instruments, Inc.Automatically-activated wireless hand-supportable laser scanning bar code symbol reading system with data transmission activation switch and automatic communication range dependent controlUS7136080 *Aug 31, 2000Nov 14, 2006Sony Computer Entertainment Inc.Entertainment system, entertainment apparatus, recording medium, and program providing color coded display messagesUS7137561Jul 1, 2003Nov 21, 2006Metrologic Instruments, Inc.Wireless laser scanning bar code symbol reading system employing a low-battery protection circuit, vibrational alarm and sleep mode of operationUS7143358 *Dec 23, 1999Nov 28, 2006Yuen Henry CVirtual world internet web site using common and user-specific metricsUS7162054Sep 18, 2003Jan 9, 2007Jeffrey MeisnerAugmented reality technologyUS7172126Jul 30, 2003Feb 6, 2007Metrologic Instruments, Inc.Wireless automatic laser scanning bar code symbol reading system, wherein the RF-based transceiver chipsets within the wireless hand-supportable unit and base station thereof are automatically deactivated and said RF data communication link therebetween terminated when said system enters its power-saving sleep mode, and reactivated and re-established when re-entering its operational modeUS7181690 *Aug 3, 2000Feb 20, 2007Worlds. Com Inc.System and method for enabling users to interact in a virtual spaceUS7191191 *Apr 12, 2002Mar 13, 2007Immersion CorporationHaptic authoringUS7210104 *Feb 14, 2001Apr 24, 2007Sega CorporationInformation display method and information display system for finding another user in a plurality of users based upon registered informationUS7251788 *Dec 21, 2000Jul 31, 2007Nokia CorporationSimulated speed-of-light delay for recreational benefit applicationsUS7266532 *Jun 3, 2002Sep 4, 2007The General Hospital CorporationReconfigurable autonomous device networksUS7278578Jul 30, 2003Oct 9, 2007Metrologic Instruments, Inc.Automatically-activated wireless laser scanning 2D bar code symbol reading system capable of automatically transmitting stored symbol character data when the hand-supportable unit is operated within its RF data communication range and automatically collecting and storing symbol character data when the hand-supportable unit is operated outside of its RF data communication rangeUS7281663Jul 30, 2003Oct 16, 2007Metrologic Instruments, Inc.Wireless bar code symbol reading system having hand-supportable unit and remote base stationUS7293235 *Aug 12, 1999Nov 6, 2007British Telecommunications Public Limited CompanyPredicting avatar movement in a distributed virtual environmentUS7325740Jul 1, 2003Feb 5, 2008Metrologic Instruments, Inc.Automatically-activated laser scanning 2D bar code symbol reading system employing a visible linear-type laser scanning pattern and an audible feedback signal during scan data capturing and buffering operationsUS7386799Apr 8, 2003Jun 10, 2008Forterra Systems, Inc.Cinematic techniques in avatar-centric communication during a multi-user online simulationUS7426536 *Jul 28, 2000Sep 16, 2008Fujitsu LimitedCommunication promotion method and systemUS7464878Sep 13, 2006Dec 16, 2008Metrologic Instruments, Inc.Automatically-activated wireless hand-supportable laser scanning bar code symbol reading system with automatic communication range dependent controlUS7465212Dec 30, 2004Dec 16, 2008GanzSystem and method for toy adoption and marketingUS7493558 *Nov 2, 2006Feb 17, 2009Worlds, Inc.System and method for enabling users to interact in a virtual spaceUS7503006 *Sep 25, 2003Mar 10, 2009Microsoft CorporationVisual indication of current voice speakerUS7523867Feb 17, 2005Apr 28, 2009Metrologic Instruments, Inc.Automatically-activated hand-supportable laser scanning bar code symbol reading system with omnidirectional and unidirectional scanning modes in addition to a data transmission activation switchUS7534157Dec 30, 2004May 19, 2009GanzSystem and method for toy adoption and marketingUS7549123 *Jun 15, 2005Jun 16, 2009Apple Inc.Mixing input channel signals to generate output channel signalsUS7568964Oct 14, 2008Aug 4, 2009GanzSystem and method for toy adoption and marketingUS7604525Jan 22, 2009Oct 20, 2009GanzSystem and method for toy adoption and marketingUS7626569 *Oct 24, 2005Dec 1, 2009Graphics Properties Holdings, Inc.Movable audio/video communication interface systemUS7633487 *Sep 25, 2003Dec 15, 2009Fujitsu LimitedMultichannel information processing deviceUS7653877 *Aug 10, 2004Jan 26, 2010Sony CorporationInformation processing apparatus and method, and storage mediumUS7677948Dec 30, 2004Mar 16, 2010GanzSystem and method for toy adoption and marketingUS7685518Nov 26, 2002Mar 23, 2010Sony CorporationInformation processing apparatus, method and medium using a virtual reality spaceUS7686225Oct 29, 2007Mar 30, 2010Metrologic Instruments, Inc.Wireless code symbol reading system with automatic communication range dependent controlUS7779004Feb 22, 2006Aug 17, 2010Qurio Holdings, Inc.Methods, systems, and products for characterizing target systemsUS7789726Oct 31, 2007Sep 7, 2010GanzSystem and method for toy adoption and marketingUS7814153 *Mar 12, 2004Oct 12, 2010Prototerra, Inc.System and method for client side managed data prioritization and connectionsUS7840903Jun 8, 2007Nov 23, 2010Qurio Holdings, Inc.Group content representationsUS7849420Feb 26, 2007Dec 7, 2010Qurio Holdings, Inc.Interactive content representations enabling content sharingUS7859532 *Feb 28, 2007Dec 28, 2010Kabushiki Kaisha Square EnixGenerating character images based upon display conditionsUS7862428Jul 2, 2004Jan 4, 2011GanzInteractive action figures for gaming systemsUS7886010Feb 28, 2005Feb 8, 2011Amazon Technologies, Inc.Digital media resource messagingUS7945856Jan 13, 2009May 17, 2011Worlds.Com, Inc.System and method for enabling users to interact in a virtual spaceUS7967657Nov 5, 2008Jun 28, 2011GanzSystem and method for toy adoption and marketingUS7990387Aug 16, 2007Aug 2, 2011International Business Machines CorporationMethod and apparatus for spawning projected avatars in a virtual universeUS8002605Jan 27, 2009Aug 23, 2011GanzSystem and method for toy adoption and marketingUS8024407Oct 17, 2007Sep 20, 2011Citrix Systems, Inc.Methods and systems for providing access, from within a virtual world, to an external resourceUS8066571Jun 9, 2008Nov 29, 2011Metaplace, Inc.System and method for enabling characters to be manifested within a plurality of different virtual spacesUS8082501 *Mar 19, 2009Dec 20, 2011Worlds.Com, Inc.System and method for enabling users to interact in a virtual spaceUS8140340Jan 18, 2008Mar 20, 2012International Business Machines CorporationUsing voice biometrics across virtual environments in association with an avatar's movementsUS8145998 *Mar 19, 2009Mar 27, 2012Worlds Inc.System and method for enabling users to interact in a virtual spaceUS8161385 *Apr 8, 2011Apr 17, 2012Worlds Inc.System and method for enabling users to interact in a virtual spaceUS8164589 *Jun 22, 2005Apr 24, 2012Nhn CorporationImage resource loading system and method which carries out loading of object for renewal of game screenUS8196050Sep 17, 2007Jun 5, 2012Mp 1, Inc.System and method for embedding a view of a virtual space in a banner ad and enabling user interaction with the virtual space within the banner adUS8205158Dec 5, 2007Jun 19, 2012GanzFeature codes and bonuses in virtual worldsUS8218690Sep 29, 2008Jul 10, 2012Qualcomm Atheros, Inc.Timing offset compensation for high throughput channel estimationUS8219388 *Jun 7, 2006Jul 10, 2012Konami Digital Entertainment Co., Ltd.User voice mixing device, virtual space sharing system, computer control method, and information storage mediumUS8260266Jun 26, 2007Sep 4, 2012Qurio Holdings, Inc.Method and system for third-party discovery of proximity-based servicesUS8260860 *Jul 28, 2011Sep 4, 2012Sony Computer Entertainment America LlcDynamic bandwidth controlUS8261307Oct 25, 2007Sep 4, 2012Qurio Holdings, Inc.Wireless multimedia content brokerage service for real time selective content provisioningUS8265252Apr 11, 2008Sep 11, 2012Palo Alto Research Center IncorporatedSystem and method for facilitating cognitive processing of simultaneous remote voice conversationsUS8271475 *May 27, 2008Sep 18, 2012International Business Machines CorporationApplication of user context to searches in a virtual universeUS8292688Mar 11, 2011Oct 23, 2012GanzSystem and method for toy adoption and marketingUS8316097Jan 17, 2011Nov 20, 2012GanzMultiple-layer chat filter system and methodUS8317566Apr 23, 2009Nov 27, 2012GanzSystem and method for toy adoption and marketingUS8319777 *May 16, 2008Nov 27, 2012Konami Digital Entertainment Co., Ltd.Character display, character displaying method, information recording medium, and programUS8321513Mar 4, 2008Nov 27, 2012GanzMultiple-layer chat filter system and methodUS8323068Apr 21, 2011Dec 4, 2012GanzVillagers in a virtual world with upgrading via codesUS8334871Jun 6, 2011Dec 18, 2012International Business Machine CorporationSpawning projected avatars in a virtual universeUS8345049Aug 16, 2007Jan 1, 2013International Business Machine CorporationMethod and apparatus for predicting avatar movement in a virtual universeUS8380725Aug 3, 2010Feb 19, 2013GanzMessage filter with replacement textUS8386566 *Nov 11, 2008Feb 26, 2013International Business Machines CorporationMethod and apparatus for controlling information in virtual worldUS8402377May 1, 2012Mar 19, 2013Mp 1, Inc.System and method for embedding a view of a virtual space in a banner ad and enabling user interaction with the virtual space within the banner adUS8407592 *Apr 11, 2011Mar 26, 2013Worlds Inc.System and method for enabling users to interact in a virtual spaceUS8408963Mar 31, 2011Apr 2, 2013GanzSystem and method for toy adoption and marketingUS8441478Mar 16, 2012May 14, 2013Nhn CorporationImage resource loading system and method which carries out loading of object for renewal of game screenUS8441486 *Apr 27, 2009May 14, 2013Jagex Ltd.Position tracking in a virtual worldUS8458602 *Aug 31, 2010Jun 4, 2013GanzSystem and method for limiting the number of characters displayed in a common areaUS8460052Mar 21, 2011Jun 11, 2013GanzSystem and method for toy adoption and marketingUS8465338Mar 17, 2011Jun 18, 2013GanzSystem and method for toy adoption and marketingUS8478827 *Dec 30, 2010Jul 2, 2013Amazon Technologies, Inc.Digital media resource messagingUS8500511Mar 17, 2011Aug 6, 2013GanzSystem and method for toy adoption and marketingUS8531514 *Sep 18, 2008Sep 10, 2013Nec CorporationImage providing system and image providing methodUS8549440Oct 30, 2007Oct 1, 2013GanzSystem and method for toy adoption and marketingUS8553028 *Feb 20, 2008Oct 8, 2013Julian Michael UrbachEfficiently implementing and displaying independent 3-dimensional interactive viewports of a virtual world on multiple client devicesUS8583039 *Jun 28, 2007Nov 12, 2013Palm, Inc.Data prioritization and distribution limitation system and methodUS8585497Oct 27, 2008Nov 19, 2013GanzInteractive action figures for gaming systemsUS8595631 *Feb 20, 2008Nov 26, 2013International Business Machines CorporationAccessibility in virtual worlds using tagsUS8616970Apr 7, 2008Dec 31, 2013Palo Alto Research Center IncorporatedSystem and method for managing a multiplicity of text messages in an online gameUS8627212Mar 7, 2013Jan 7, 2014Mp 1, Inc.System and method for embedding a view of a virtual space in a banner ad and enabling user interaction with the virtual space within the banner adUS8627213 *Aug 10, 2004Jan 7, 2014Hewlett-Packard Development Company, L.P.Chat room system to provide binaural sound at a user locationUS8631332 *Sep 30, 2010Jan 14, 2014Linkedin CorporationContent visualizationUS8636588Oct 24, 2008Jan 28, 2014GanzInteractive action figures for gaming systemsUS8640028 *Jun 16, 2011Jan 28, 2014Worlds Inc.System and method for enabling users to interact in a virtual spaceUS8641471Dec 22, 2010Feb 4, 2014GanzSystem and method for toy adoption and marketingUS20070257915 *May 7, 2007Nov 8, 2007Ken KutaragiUser Interface Device, User Interface Method and Information Storage MediumUS20080014988 *Jun 28, 2007Jan 17, 2008Palm, Inc.Data prioritization and distribution limitation system and methodUS20090077463 *Sep 17, 2007Mar 19, 2009Areae, Inc.System for providing virtual spaces for access by usersUS20090077475 *Sep 17, 2007Mar 19, 2009Areae, Inc.System for providing virtual spaces with separate places and/or acoustic areasUS20090195650 *Feb 3, 2009Aug 6, 2009Olympus Imaging Corp.Virtual image generating apparatus, virtual image generating method, and recording medium storing virtual image generating programUS20090299988 *May 27, 2008Dec 3, 2009Hamilton Ii Rick AApplication of user context to searches in a virtual universeUS20100077034 *Sep 22, 2008Mar 25, 2010International Business Machines CorporationModifying environmental chat distance based on avatar population density in an area of a virtual worldUS20100146408 *Dec 10, 2008Jun 10, 2010International Business Machines CorporationSystem and method to modify audio components in an online environmentUS20100164960 *May 16, 2008Jul 1, 2010Konami Digital Entertainment Co., Ltd.Character Display, Character Displaying Method, Information Recording Medium, and ProgramUS20100214400 *Sep 18, 2008Aug 26, 2010Motoaki ShimizuImage providing system and image providing methodUS20100275136 *Apr 27, 2009Oct 28, 2010Jagex Ltd.Position tracking in a virtual worldUS20110022966 *Sep 30, 2010Jan 27, 2011Digg, Inc.Content visualizationUS20110055734 *Aug 31, 2010Mar 3, 2011GanzSystem and method for limiting the number of characters displayed in a common areaUS20110191413 *Apr 11, 2011Aug 4, 2011Dave LeahySystem and method for enabling users to interact in a virtual spaceUS20110201422 *Feb 5, 2008Aug 18, 2011Konami Digital Entertainment Co., Ltd.Game device, game device control method, program and information memory mediumUS20110219318 *May 16, 2011Sep 8, 2011Raj Vasant AbhyankerCharacter expression in a geo-spatial environmentUS20110252341 *Jun 16, 2011Oct 13, 2011Dave LeahySystem and method for enabling users to interact in a virtual spaceUS20110254837 *Apr 19, 2011Oct 20, 2011Lg Electronics Inc.Image display apparatus and method for controlling the sameUS20110302507 *Apr 8, 2011Dec 8, 2011Dave LeahySystem and method for enabling users to interact in a virtual spaceUS20130091436 *Nov 29, 2012Apr 11, 2013Linkedin CorporationContent visualizationUSRE44054Jun 19, 2007Mar 5, 2013GanzGraphic chatting with organizational avatarsCN100482149CAug 13, 2004Apr 29, 2009索尼株式会社Image display system, image display device, image display methodDE10026489A1 *May 27, 2000Dec 6, 2001Bosch Gmbh RobertVideo conference operating method has synchronised representation of video conference scene provided at each video conference terminalDE10106072A1 *Feb 9, 2001Aug 14, 2002Deutsche Telekom AgVerfahren zur Darstellung visueller Informationen in einer Telekommunikationsvorrichtung und TelekommunikationsvorrichtungEP1754393A1 *Apr 15, 2005Feb 21, 2007Smart Internet Technology Crc Pty LimitedApparatuses and methods for use in creating an audio sceneWO2009039084A1 *Sep 16, 2008Mar 26, 2009Thor AlexanderSystem for providing virtual spaces with separate places and/or acoustic areas* Cited by examinerClassifications U.S. Classification715/706, 715/848, 348/E07.83, 715/727International ClassificationG06T15/30, H04N7/14, H04N7/15, G06T17/00Cooperative ClassificationG06T17/00, G06T2200/16, G06T15/30, H04N7/15, H04N7/157, H04N7/148European ClassificationH04N7/14A4, G06T17/00, H04N7/15, G06T15/30Legal EventsDateCodeEventDescriptionSep 9, 2009FPAYFee paymentYear of fee payment: 12Sep 9, 2005FPAYFee paymentYear of fee payment: 8Oct 4, 2001FPAYFee paymentYear of fee payment: 4Aug 1, 1995ASAssignmentOwner name: NIPPON TELEGRAH AND TELEPHONE CORPORATION, JAPANFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SUZUKI, GEN;SUGAWARA, SHOHEI;TANIGAWA, HIROYA;AND OTHERS;REEL/FRAME:007633/0783Effective date: 19950720RotateOriginal ImageGoogle Home - Sitemap - USPTO Bulk Downloads - Privacy Policy - Terms of Service - About Google Patents - Send FeedbackData provided by IFI CLAIMS Patent Services©2012 Google