Sound output control apparatus, sound output control system, sound output control method, and program

Provided are a sound output control apparatus, a sound output control system, a sound output control method, and a program that can appropriately thin out the output of pieces of sound data. A sound data reception section receives a plurality of pieces of sound data transmitted from transmission apparatuses that are different from each other. A selection section selects a portion of the plurality of pieces of sound data on the basis of at least one of a result of a voice activity detection process performed on each of the pieces of sound data or moving averages of volumes of sounds represented by the pieces of sound data. A sound data transmission section outputs the selected portion of the pieces of sound data.

TECHNICAL FIELD

The present invention relates to a sound output control apparatus, a sound output control system, a sound output control method, and a program.

BACKGROUND ART

In recent years, it has increasingly become common for users to play games while voice-chatting with other users at distant locations, such as users who are playing games together or viewers of moving images representing the game play state.

SUMMARY

Technical Problems

An increase in the number of users participating in voice chat increases the load on the apparatus that each user uses to input and output speech during voice chat and a server that relays sound data. Further, the communication volume of a sound data communication path increases. This causes a decrease in the service quality of voice chat and an increase in the operational cost and communication cost of the server, for example.

Thus, in voice chat in which many users participate, it is necessary to appropriately thin out the output of pieces of sound data to appropriately suppress the above-described load and communication volume. However, this has not been done with conventional voice chat techniques.

The present invention has been made in view of the circumstances described above. It is an object of the present invention to provide a sound output control apparatus, a sound output control system, a sound output control method, and a program that can appropriately thin out the output of pieces of sound data.

Solution to Problems

In order to solve the issue described above, a sound output control apparatus according to the present invention includes a reception section configured to receive a plurality of pieces of sound data transmitted from transmission apparatuses that are different from each other; a selection section configured to select a portion of the plurality of pieces of sound data on the basis of at least one of a result of a voice activity detection process performed on each of the pieces of sound data or moving averages of volumes of sounds represented by the pieces of sound data; and an output section configured to output the selected portion of the pieces of sound data.

Further, in an aspect of the present invention, the output section transmits the selected portion of the pieces of sound data to a reception apparatus capable of voice-chatting with the transmission apparatuses.

In this aspect, from among the plurality of pieces of sound data, the selection section may select pieces of sound data in the number determined on the basis of the type of the reception apparatus.

Further, in an aspect of the present invention, a decoding section configured to decode the pieces of sound data is further included. The output section outputs the selected portion of the pieces of sound data to the decoding section.

Further, in an aspect of the present invention, from among the plurality of pieces of sound data, the selection section selects pieces of sound data in the number determined on the basis of a load on the sound output control apparatus or a communication quality of a computer network to which the sound output control apparatus is connected.

Further, a sound output control system according to the present invention includes a plurality of communication apparatuses included in a first group, a plurality of communication apparatuses included in a second group, and a relay apparatus, in which the relay apparatus includes a first reception section configured to receive a plurality of pieces of sound data transmitted from the communication apparatuses that are included in the first group and different from each other and a transmission section configured to transmit a portion selected from among the plurality of pieces of sound data received by the first reception section of the relay apparatus to the communication apparatuses other than any communication apparatus that has transmitted sound data corresponding to the portion, and each of the communication apparatuses included in the second group includes a decoding section configured to decode the pieces of sound data, a second reception section configured to receive the portion of the pieces of sound data, the portion having been transmitted from the relay apparatus, and at least one piece of sound data transmitted from any of the other communication apparatuses that are included in the second group and different from each other, and an output section configured to output a portion of a plurality of the pieces of sound data received by the second reception section of the communication apparatus to the decoding section.

Further, in an aspect of the present invention, a determination section configured to determine, on the basis of the number of the communication apparatuses that transmit and receive the pieces of sound data to and from each other, whether a communication apparatus newly added to the sound output control system is to be included in the first group or the second group is further included.

Further, a sound output control method according to the present invention includes a step of receiving a plurality of pieces of sound data transmitted from transmission apparatuses that are different from each other; a step of selecting a portion of the plurality of pieces of sound data on the basis of at least one of a result of a voice activity detection process performed on each of the pieces of sound data or moving averages of volumes of sounds represented by the pieces of sound data; and a step of outputting the selected portion of the pieces of sound data.

Further, a program according to the present invention causes a computer to execute a procedure of receiving a plurality of pieces of sound data transmitted from transmission apparatuses that are different from each other; a procedure of selecting a portion of the plurality of pieces of sound data on the basis of at least one of a result of a voice activity detection process performed on each of the pieces of sound data or moving averages of volumes of sounds represented by the pieces of sound data; and a procedure of outputting the selected portion of the pieces of sound data.

DESCRIPTION OF EMBODIMENT

FIG.1is a diagram illustrating an example of the overall configuration of a voice chat system1according to an embodiment of the present invention. As illustrated inFIG.1, the voice chat system1includes voice chat apparatuses10(10-1,10-2, . . . ,10-n), a relay apparatus12, and a management server14, all of which mainly include a computer. The voice chat apparatuses10, the relay apparatus12, and the management server14are connected to a computer network16such as the Internet. The voice chat apparatuses10, the relay apparatus12, and the management server14can communicate with each other.

The management server14is a computer such as a server that manages, for example, account information and the like of each user who uses the voice chat system1. The management server14stores, for example, a plurality of pieces of account data associated with users. Here, each account data includes, for example, a user identification (ID), real name data, email address data, and the like. The user ID is the identification information of a user. The real name data indicates the real name of the user. The email address data indicates the email address of the user.

The voice chat apparatus10is, for example, a computer, such as a game console, a portable game apparatus, a smartphone, or a personal computer, which is capable of inputting and outputting sounds in voice chat.

As illustrated inFIG.2A, the voice chat apparatus10includes, for example, a processor10a, a storage section10b, a communication section10c, a display section10d, an operation section10e, a microphone10f, a speaker10g, and an encoding and decoding section10h. It is noted that the voice chat apparatus10may include a camera.

The processor10ais, for example, a program control device such as a central processing unit (CPU) and performs various information processes according to a program stored in the storage section10b.

The storage section10bis, for example, a storage element such as a read only memory (ROM) or a random access memory (RAM), a hard disk drive, or the like.

The communication section10cis, for example, a communication interface for exchanging data with computers such as the other voice chat apparatuses10, the relay apparatus12, and the management server14via the computer network16.

The display section10dis, for example, a liquid crystal display or the like. The display section10ddisplays, for example, a screen generated by the processor10aand a moving image represented by moving image data received via the communication section10c.

The operation section10eis, for example, an operation member for performing operation input to the processor10a.

The microphone10fis a sound input device used to input, for example, sounds for voice chat.

The speaker10gis a sound output device used to output, for example, sounds in voice chat.

The encoding and decoding section10hincludes, for example, an encoder and a decoder. The encoder encodes input sound to generate sound data representing the sound. Further, the decoder decodes input sound data and outputs the sound represented by the sound data.

In the present embodiment, the relay apparatus12is, for example, a computer such as a server that relays the sound data described above.

As illustrated inFIG.2B, the relay apparatus12according to the present embodiment includes, for example, a processor12a, a storage section12b, and a communication section12c.

The processor12ais, for example, a program control device such as a CPU and performs various information processes according to a program stored in the storage section12b.

The storage section12bis, for example, a storage element such as a ROM or a RAM, a hard disk drive, or the like.

The communication section12cis a communication interface for exchanging data with computers such as the voice chat apparatuses10and the management server14.

In the present embodiment, the users of the voice chat system1can enjoy voice chat with each other. Here, for example, the voice chat may be conducted while a moving image representing the play state of a game that is being played by some or all of the users participating in the voice chat is shared.

Further, in the present embodiment, voice chat can be conducted by a plurality of users. Here, in the present embodiment, a plurality of users participating in voice chat are assumed to belong to a group called a party. By performing a predetermined operation, each user of the voice chat system1according to the present embodiment can create a new party or join a party already created.

FIG.3is a diagram schematically illustrating an example of sound data transmission in the voice chat system1according to the present embodiment.

In the example ofFIG.3, a user A, a user B, a user C, a user D, and a user E are assumed to be participating in a party. In addition, the user A to the user E are assumed to use voice chat apparatuses10-1to10-5, respectively. In addition, in the example of the sound data transmission illustrated inFIG.3, the voice chat apparatus10-5used by the user E receives sound data.

In the example ofFIG.3, pieces of sound data transmitted from the respective voice chat apparatuses10-1to10-4are transmitted to the relay apparatus12. Then, the relay apparatus12selects a portion of the plurality of pieces of sound data. The relay apparatus12then transmits the selected portion of the pieces of sound data to the voice chat apparatus10-5. In this manner, in the example ofFIG.3, the pieces of sound data to be received by the voice chat apparatus10-5are thinned out by the relay apparatus12.

It is noted that this similarly applies to each of the voice chat apparatuses10-1to10-4in the example ofFIG.3. The relay apparatus12selects a portion of a plurality of pieces of sound data each transmitted from the voice chat apparatuses10other than the relevant voice chat apparatus10. Then, the relay apparatus12transmits the selected portion of the pieces of sound data to the relevant voice chat apparatus10.

FIG.4is a diagram schematically illustrating another example of sound data transmission in the voice chat system1according to the present embodiment.

In the example ofFIG.4as well, the user A, the user B, the user C, the user D, and the user E are assumed to be participating in a party. In addition, the user A to the user E are assumed to use the voice chat apparatuses10-1to10-5, respectively. In addition, in an example of sound data transmission illustrated inFIG.4, the voice chat apparatus10-5used by the user E receives sound data.

In the example ofFIG.4, pieces of sound data transmitted from the respective voice chat apparatuses10-1to10-4are transmitted to the communication section10cof the voice chat apparatus10-5. The processor10aof the voice chat apparatus10-5then selects a portion of the plurality of pieces of sound data. The processor10aof the voice chat apparatus10-5then outputs the selected portion of the pieces of sound data to the encoding and decoding section10hof the voice chat apparatus10-5. In this manner, in the example ofFIG.4, the pieces of sound data to be input into the encoding and decoding section10hof the voice chat apparatus10-5are thinned out by the processor10aof the voice chat apparatus10-5.

It is noted that this similarly applies to each of the voice chat apparatuses10-1to10-4. The processor10aof the relevant voice chat apparatus10selects a portion of a plurality of pieces of sound data each transmitted from the voice chat apparatuses10other than the relevant voice chat apparatus10. Then, the processor10aoutputs the selected portion of the pieces of sound data to the encoding and decoding section10hof the relevant voice chat apparatus10.

FIGS.5and6are diagrams each schematically illustrating yet another example of sound data transmission in the voice chat system1according to the present embodiment.

In the examples ofFIGS.5and6, the user A, the user B, the user C, the user D, the user E, and a user F are assumed to be participating in a party. In addition, the user A to the user F are assumed to use the voice chat apparatuses10-1to10-6, respectively.

As illustrated inFIG.5, here, for example, the voice chat apparatuses10-1to10-3transmit and receive pieces of sound data to and from each other through a peer-to-peer (P2P) connection. By contrast, each of the voice chat apparatuses10-4to10-6transmits and receives pieces of sound data to and from the other voice chat apparatuses10via the relay apparatus12.

Thus, for example, the voice chat apparatus10-1transmits, not via the relay apparatus12but directly, sound data addressed to the voice chat apparatus10-2to the voice chat apparatus10-2. Further, the voice chat apparatus10-1transmits, not via the relay apparatus12but directly, sound data addressed to the voice chat apparatus10-3to the voice chat apparatus10-3.

By contrast, the voice chat apparatus10-1transmits sound data addressed to the voice chat apparatuses10-4to10-6to the relay apparatus12.

This similarly applies to the voice chat apparatuses10-2and10-3. Each of the voice chat apparatuses10-2and10-3directly transmits sound data addressed to the voice chat apparatuses10that are P2P-connected to those voice chat apparatuses10, while transmitting, to the relay apparatus12, sound data addressed to the voice chat apparatuses10that are not P2P-connected to those voice chat apparatuses10.

Further, as for the voice chat apparatuses10-4to10-6, each of the voice chat apparatuses10transmits sound data addressed to the other voice chat apparatuses10to the relay apparatus12.

In addition, in the example of sound data transmission illustrated inFIG.6, the voice chat apparatus10-1used by the user A receives sound data in the situation illustrated inFIG.5.

In the example ofFIG.6, as described above, pieces of sound data transmitted from the respective voice chat apparatuses10-4to10-6are transmitted to the relay apparatus12. The relay apparatus12then selects a portion of the plurality of pieces of sound data. The relay apparatus12then transmits the selected portion of the pieces of sound data to the voice chat apparatus10-1. In this manner, in the example ofFIG.6, the pieces of sound data to be transmitted from the relay apparatus12and received by the voice chat apparatus10-1are thinned out by the relay apparatus12.

In addition, in the example ofFIG.6, in addition to the pieces of sound data transmitted from the relay apparatus12, pieces of sound data transmitted from the respective voice chat apparatuses10-2and10-3are also transmitted to the communication section10cof the voice chat apparatus10-1. The processor10aof the voice chat apparatus10-1then selects a portion of the plurality of pieces of sound data. The processor10aof the voice chat apparatus10-1then outputs the selected portion of the pieces of sound data to the encoding and decoding section10hof the voice chat apparatus10-1. In this manner, in the example ofFIG.6, the pieces of sound data to be input into the encoding and decoding section10hof the voice chat apparatus10-1are thinned out by the processor10aof the voice chat apparatus10-1.

It is noted that this similarly applies to the voice chat apparatuses10-2and10-3. The relay apparatus12selects a portion of a plurality of pieces of sound data transmitted from the respective voice chat apparatuses10-4to10-6. The relay apparatus12then transmits the selected portion of the pieces of sound data to the relevant voice chat apparatuses10. Further, the processor10aof the relevant voice chat apparatus10selects a portion of the plurality of pieces of sound data received by the communication section10cof the relevant voice chat apparatus10. The selected portion of the pieces of sound data is then output to the encoding and decoding section10hof the relevant voice chat apparatus10.

As the number of users participating in the voice chat increases, the load on the voice chat apparatuses10and the relay apparatus12increases. Further, the communication volume of a sound data communication path increases. According to the example ofFIG.3, the pieces of sound data to be transmitted from the relay apparatus12to the voice chat apparatus10-5are thinned out by the relay apparatus12. Thus, the communication volume (network traffic) of pieces of sound data flowing through the computer network16can be suppressed. Further, for example, in a case where the relay apparatus12is such that the operational cost thereof increases in proportion to an increase in the amount of data transmission, the operational cost of the relay apparatus12can be suppressed by employing the example ofFIG.3.

Further, according to the example ofFIG.4, since the pieces of sound data to be input into the encoding and decoding section10hof the voice chat apparatus10-5are thinned out, the decoding process load in the voice chat apparatus10-5can be suppressed.

Further, according to the examples ofFIGS.5and6, the selection of pieces of sound data is dispersedly performed by the relay apparatus12and the voice chat apparatus10. As a result, the suppression of the communication volume of pieces of sound data flowing through the computer network16, the suppression of the operational cost of the relay apparatus12, and the suppression of the decoding process load in the voice chat apparatus10-1can be achieved with a good balance.

In the present embodiment, information regarding a party, such as a sound data communication path as illustrated inFIG.5, is managed by party management data stored in the management server14, for example. The party management data is exemplified inFIG.7. As illustrated inFIG.7, the party management data includes a party ID, which is the identification information of the party, and pieces of user data, which are associated with corresponding users participating in the party. In addition, each user data includes a user ID, connection destination address data, type data, a P2P connection flag, and the like.

The user ID is, for example, the identification information of the corresponding user. The connection destination address data is, for example, data indicating the address of the voice chat apparatus10used by the relevant user. The type data is, for example, data indicating the type of voice chat apparatus10used by the relevant user. Here, examples of the type of voice chat apparatus10include a game console, a portable game apparatus, a smartphone, a personal computer, and the like as described above. The P2P connection flag is, for example, a flag indicating whether or not the voice chat apparatus10used by the relevant user performs the P2P connection.

Here, for example, the voice chat apparatus10that performs the P2P connection refers to the voice chat apparatus10that performs the P2P connection with the voice chat apparatuses10used by some or all of the other users participating in the party. In the example ofFIG.5, the voice chat apparatuses10-1to10-3correspond to the voice chat apparatuses10that perform the P2P connection. Further, similarly, the voice chat apparatuses10-1to10-5illustrated inFIG.4also correspond to the voice chat apparatuses10that perform the P2P connection.

In addition, the voice chat apparatus10that does not perform the P2P connection refers to the voice chat apparatus10that is connected, via the relay apparatus12, to the voice chat apparatuses10used by all the other users participating in the party. In the example ofFIG.5, the voice chat apparatuses10-4to10-6correspond to the voice chat apparatuses10that do not perform the P2P connection. Further, similarly, the voice chat apparatuses10-1to10-5illustrated inFIG.3also correspond to the voice chat apparatuses10that do not perform the P2P connection.

Here, for example, the value of the P2P connection flag is set to 1 in the user data of the user using the voice chat apparatus10that performs the P2P connection. Further, for example, the value of the P2P connection flag is set to 0 in the user data of the user using the voice chat apparatus10that does not perform the P2P connection.

FIG.7exemplifies the party management data whose party ID is 001. This party management data is associated with a party in which six users participate. The party management data illustrated inFIG.7includes six pieces of user data each of which is associated with a corresponding user participating in the party. Hereinafter, the user whose user ID is aaa, the user whose user ID is bbb, the user whose user ID is ccc, the user whose user ID is ddd, the user whose user ID is eee, and the user whose user ID is fff will be referred to as the user A, the user B, the user C, the user D, the user E, and the user F, respectively. Further, the user A, the user B, the user C, the user D, the user E, and the user F are assumed to use the voice chat apparatuses10-1,10-2,10-3,10-4,10-5, and10-6, respectively.

Further, in the present embodiment, a copy of the party management data stored in the management server14is transmitted to the voice chat apparatuses10used by the users participating in the party associated with this party management data and the relay apparatus12. In addition, the copy of the party management data stored in the management server14is stored in the storage section10bof each voice chat apparatus10and the storage section12bof the relay apparatus12. Thus, the voice chat apparatus10used by the user participating in the party can identify the addresses of the voice chat apparatuses10used by the other users participating in this party.

Further, data indicating the address of the relay apparatus12is also stored in the storage section10bof each voice chat apparatus10. Thus, the voice chat apparatuses10can identify the address of the relay apparatus12.

Further, in the present embodiment, the party management data stored in the management server14is updated in response to, for example, a user's operation to join a party or the like. In addition, whenever the party management data stored in the management server14is updated, a copy of the updated party management data is transmitted to the voice chat apparatuses10used by the users participating in the party associated with this party management data and the relay apparatus12. Then, the copy of the party management data stored in the storage section10bof each voice chat apparatus10and the storage section12bof the relay apparatus12is updated. In this manner, in the present embodiment, the latest information indicated in the party management data is shared among the voice chat apparatuses10used by the users participating in the party associated with this party management data.

In addition, in the present embodiment, for example, each of the voice chat apparatuses10that perform the P2P connection directly transmits sound data addressed to the other voice chat apparatuses10that perform the P2P connection to the other voice chat apparatuses10. In addition, each of the voice chat apparatuses10that perform the P2P connection transmits the sound data addressed to the other voice chat apparatuses10that do not perform the P2P connection to the relay apparatus12.

In addition, in the present embodiment, for example, each of the voice chat apparatuses10that do not perform the P2P connection transmits sound data addressed to the other voice chat apparatuses10to the relay apparatus12.

By using the party management data exemplified inFIG.7in this manner, it is possible to appropriately control whether sound data is to be transmitted to the voice chat apparatuses10or the relay apparatus12.

Further, in the sound data selection described above, a portion of the plurality of pieces of sound data may be selected on the basis of at least one of the result of the voice activity detection (VAD) process performed on each of the pieces of sound data or the volumes of the sounds represented by the pieces of sound data.

For example, for each predetermined time period (e.g., 20 milliseconds, 40 milliseconds, or the like), the voice chat apparatus10may generate sound data corresponding to this time period by encoding the sound input over this time period.

Then, the voice chat apparatus10may determine whether or not the relevant sound data represents a human voice by performing a known voice activity detection process on the relevant sound data. The voice chat apparatus10may then generate VAD data indicating whether or not the relevant sound data represents a human voice. Here, for example, in a case where the relevant sound data represents a human voice, the voice chat apparatus10may generate VAD data with a value of 1, otherwise VAD data with a value of 0.

Further, the voice chat apparatus10may identify the volume of the sound represented by the relevant sound data. Then, the voice chat apparatus10may generate volume data indicating the volume of the sound represented by the relevant sound data.

In addition, the voice chat apparatus10may transmit the relevant sound data associated with the identification information of the voice chat apparatus10(e.g., the user ID corresponding to the voice chat apparatus10), the VAD data described above, and the volume data described above. Further, data indicating a time period corresponding to the relevant sound data, such as a timestamp, may be associated with the relevant sound data.

Then, the relay apparatus12receives a plurality of pieces of sound data from each of the voice chat apparatuses10that are different from each other. For each predetermined time period, the relay apparatus12may select a portion of a plurality of pieces of sound data received during this time period. Here, for example, the selection may be made on the basis of the VAD data and the volume data associated with each sound data. Then, the selected portion of the pieces of sound data may be transmitted.

Further, the voice chat apparatus10receives a plurality of pieces of sound data from each of the voice chat apparatuses10that are different from each other. For each predetermined time period, the voice chat apparatus10may select a portion of a plurality of pieces of sound data received during this time period. Here, for example, the selection may be made on the basis of the VAD data and the volume data associated with each sound data. Then, the selected portion of the pieces of sound data may be output to the encoding and decoding section10hof the relevant voice chat apparatus10.

Specific examples of the selection based on the VAD data and the volume data will be described later.

Hereinafter, the functions implemented in the voice chat system1according to the present embodiment and the processes performed in the voice chat system1according to the present embodiment will be further described, focusing on the sound data selection and the transmission of the selected pieces of sound data.

FIG.8is a functional block diagram illustrating an example of the functions implemented in the voice chat apparatus10and the relay apparatus12according to the present embodiment. It is noted that all of the functions illustrated inFIG.8do not need to be implemented in the voice chat apparatus10and the relay apparatus12according to the present embodiment. Further, functions other than those illustrated inFIG.8may also be implemented.

As illustrated inFIG.8, the voice chat apparatus10according to the present embodiment functionally includes a party management data storage section20, a party management section22, a sound acceptance section24, a VAD data generation section26, a volume data generation section28, a sound data transmission section30, a sound data reception section32, a selection section34, a selected sound data output section36, and a sound output section38, for example.

The party management data storage section20is implemented mainly by the storage section10b. The party management section22is mainly implemented by the processor10aand the communication section10c. The sound acceptance section24is mainly implemented by the microphone10fand the encoding and decoding section10h. The VAD data generation section26, the volume data generation section28, the selection section34, and the selected sound data output section36are mainly implemented by the processor10a. The sound data transmission section30and the sound data reception section32are mainly implemented by the communication section10c. The sound output section38is mainly implemented by the encoding and decoding section10hand the speaker10g.

In addition, the functions described above are implemented by the processor10aexecuting a program installed in the voice chat apparatus10, which is a computer. The program includes instructions corresponding to the above-described functions. This program is supplied to the voice chat apparatus10via, for example, a computer-readable information storage medium such as an optical disc, a magnetic disk, a magnetic tape, a magneto-optical disk, or a flash memory, or via the Internet or the like.

Further, as illustrated inFIG.8, the relay apparatus12according to the present embodiment functionally includes a party management data storage section40, a party management section42, a sound data reception section44, a selection section46, and a sound data transmission section48, for example.

The party management data storage section40is mainly implemented by the storage section12b. The party management section42is mainly implemented by the processor12aand the communication section12c. The sound data reception section44and the sound data transmission section48are mainly implemented by the communication section12c. The selection section46is mainly implemented by the processor12a.

In addition, the functions described above are implemented by the processor12aexecuting a program installed in the relay apparatus12, which is a computer. The program includes instructions corresponding to the above-described functions. This program is supplied to the relay apparatus12via, for example, a computer-readable information storage medium such as an optical disc, a magnetic disk, a magnetic tape, a magneto-optical disk, or a flash memory, or via the Internet or the like.

In the present embodiment, for example, the party management data storage section20of the voice chat apparatus10and the party management data storage section40of the relay apparatus12store the party management data exemplified inFIG.7.

In the present embodiment, for example, in response to the reception of the party management data transmitted from the management server14, the party management section22of the voice chat apparatus10updates the party management data stored in the party management data storage section20to the received party management data.

In the present embodiment, for example, in response to the reception of the party management data transmitted from the management server14, the party management section42of the relay apparatus12updates the party management data stored in the party management data storage section40to the received party management data.

For example, when the user performs an operation to join an existing party, the management server14adds the user data including the user ID of this user to the party management data including the party ID of this party. Hereinafter, this user data will be referred to as additional user data. Here, the connection destination address data of the additional user data is set to the address of the voice chat apparatus10used by this user. Further, the type data of the additional user data is set to a value indicating the type of the relevant voice chat apparatus10.

Further, the value of the P2P connection flag of the additional user data is set to 1 or 0, as described above. Here, for example, the value of the P2P connection flag of this user data may be determined on the basis of the number of voice chat apparatuses10that transmit and receive pieces of sound data representing sounds in voice chat to and from each other. For example, the value of the P2P connection flag of this user data may be determined on the basis of the number of pieces of user data included in the party management data corresponding to this party.

Specifically, for example, in a case where the number of pieces of user data included in the party management data including the additional user data is 8 or less, the value of the P2P connection flag of the additional user data may be set to 1. Further, in a case where the number of pieces of user data included in the party management data including the additional user data is equal to or greater than9, the value of the P2P connection flag of the additional user data may be set to 0.

In this manner, while there are only a small number of voice chat apparatuses10that transmit and receive pieces of sound data to and from each other, the voice chat apparatuses10transmit and receive the pieces of sound data to and from each other through the P2P connection. In this case, the relay apparatus12is not used in transmitting and receiving the pieces of sound data. Thus, in a case where there are only a small number of voice chat apparatuses10that transmit and receive pieces of sound data to and from each other, the load on the relay apparatus12can be suppressed.

On the other hand, in a case where there are many voice chat apparatuses10that transmit and receive pieces of sound data to and from each other, the communication volume of the pieces of sound data flowing through the computer network16becomes excessively large because there are many voice chat apparatuses10that are transmission destinations of a single voice chat apparatus10. Here, as described above, in response to an increase in the number of voice chat apparatuses10that transmit and receive pieces of sound data to and from each other, the relay apparatus12is also used to transmit and receive the pieces of sound data. This makes it possible to prevent the communication volume of the pieces of sound data from becoming excessively large.

It is noted that in a case where the number of pieces of user data included in the party management data is 8 or less, the voice chat apparatus10used by the user who has performed the joining operation may attempt to achieve the P2P connection to each of the voice chat apparatuses10used by the other users participating in the same party. Then, in a case where the P2P connections to all the voice chat apparatuses10have been successful, the value of the P2P connection flag of the additional user data may be set to 1. On the other hand, in a case where any of the P2P connections to the voice chat apparatuses10has failed, the value of the P2P connection flag of the additional user data may be set to 0.

As described above, in response to the addition of the additional user data to the party management data stored in the management server14in this manner, the party management data stored in each of the voice chat apparatuses10used by the users participating in this party is updated. Further, the party management data stored in the relay apparatus12is also similarly updated.

In the present embodiment, for example, the sound acceptance section24accepts sound for voice chat. By encoding the sound, the sound acceptance section24may generate sound data representing the sound.

In the present embodiment, for example, the VAD data generation section26generates the above-described VAD data on the basis of the sound data generated by the sound acceptance section24.

In the present embodiment, for example, the volume data generation section28generates the above-described volume data on the basis of the sound data generated by the sound acceptance section24.

In the present embodiment, for example, the sound data transmission section30of the voice chat apparatus10transmits sound data representing the sound accepted by the sound acceptance section24. Here, the sound data transmission section30may transmit the sound data associated with the identification information of the relevant voice chat apparatus10. Further, the sound data transmission section30may also transmit sound data associated with the identification information of the relevant voice chat apparatus10, the VAD data generated by the VAD data generation section26, and the volume data generated by the volume data generation section28. Further, data indicating a time period corresponding to this sound data, such as a timestamp, may be associated with this sound data.

Further, as described above, the sound data transmission section30of the voice chat apparatus10may transmit the sound data to the voice chat apparatuses10used by the other users participating in the same party as the user using the relevant voice chat apparatus10. Further, the sound data transmission section30of the voice chat apparatus10may transmit, to the relay apparatus12, the sound data addressed to the voice chat apparatuses10used by the other users participating in the same party as the user using the relevant voice chat apparatus10.

In the present embodiment, for example, the sound data reception section32of the voice chat apparatus10receives sound data. Here, the sound data reception section32may receive a plurality of pieces of sound data transmitted from transmission apparatuses that are different from each other. In the example described above, each of the voice chat apparatuses10used by the other users participating in the same party as the user using the relevant voice chat apparatus10or the relay apparatus12corresponds to such a transmission apparatus.

The sound data reception section32of the voice chat apparatus10may directly transmit pieces of sound data from the voice chat apparatuses10used by the other users participating in the same party as the user using the relevant voice chat apparatus10. Further, the sound data reception section32of the voice chat apparatus10may also receive pieces of sound data transmitted via the relay apparatus12from the voice chat apparatuses10used by the other users participating in the same party as the user using the relevant voice chat apparatus10.

In the present embodiment, for example, the selection section34of the voice chat apparatus10selects a portion of a plurality of pieces of sound data received by the sound data reception section32of the voice chat apparatus10. Here, as described above, the selection section34may select a portion of the plurality of pieces of sound data on the basis of at least one of the result of the voice activity detection process performed on each of the pieces of sound data or the volumes of the sounds represented by the pieces of sound data.

In the present embodiment, for example, the selected sound data output section36outputs the portion of the pieces of sound data, the portion having been selected by the selection section34of the voice chat apparatus10. Here, for example, the portion of the pieces of sound data, the portion having been selected by the selection section34of the voice chat apparatus10, is output to the sound output section38.

In the present embodiment, for example, the sound output section38decodes the sound data output from the selected sound data output section36. In addition, in the present embodiment, for example, the sound output section38outputs a sound that is generated by decoding the sound data and that represents the sound data.

In the present embodiment, for example, the sound data reception section44of the relay apparatus12receives a plurality of pieces of sound data transmitted from transmission apparatuses that are different from each other. In the above-described example, each voice chat apparatus10that transmits sound data to the relay apparatus12corresponds to such a transmission apparatus. The sound data reception section44of the relay apparatus12receives, for example, pieces of sound data transmitted from the sound data transmission sections30of the voice chat apparatuses10.

In the present embodiment, for example, the selection section46of the relay apparatus12selects a portion of the plurality of pieces of sound data received by the sound data reception section44of the relay apparatus12. Here, as described above, the selection section46may select a portion of the plurality of pieces of sound data on the basis of at least one of the result of the voice activity detection process performed on each of the pieces of sound data or the volumes of the sounds represented by the pieces of sound data.

In the present embodiment, for example, the sound data transmission section48of the relay apparatus12transmits the portion of the pieces of sound data, the portion having been selected by the selection section46of the relay apparatus12, to reception apparatuses capable of voice-chatting with the transmission apparatuses that are the transmission sources of the pieces of sound data received by the sound data reception section44of the relay apparatus12. In the above-described example, the voice chat apparatuses10correspond to such reception apparatuses. Here, on the basis of the user ID associated with sound data, the sound data transmission section48may identify the party to which the user represented by the relevant user ID belongs. Then, the sound data transmission section48may transmit the sound data to the voice chat apparatuses10used by the users participating in the party, except for the voice chat apparatus10used by the user associated with the relevant user ID.

Further, the relay apparatus12may receive a plurality of pieces of sound data that are transmitted from respective communication apparatuses that are different from each other and included in a first group. The relay apparatus12may then transmit a portion selected from among the plurality of pieces of sound data to the communication apparatuses other than any communication apparatus that has transmitted sound data corresponding to the portion. In the examples ofFIGS.5and6, the voice chat apparatuses10-4to10-6correspond to the communication apparatuses included in the first group. In other words, the voice chat apparatus10associated with the user data whose value of the P2P connection flag is 0 corresponds to the communication apparatus included in the first group.

Further, the voice chat apparatus10included in a second group may receive the above-described portion of the pieces of sound data, the portion having been transmitted from the relay apparatus12, and at least one piece of sound data transmitted from any of the other communication apparatuses that are different from each other and included in the second group. The relevant voice chat apparatus10may then output a portion of the plurality of pieces of sound data received to the sound output section38of the relevant voice chat apparatus10. In the example ofFIG.6, the voice chat apparatus10-1corresponds to the voice chat apparatus10that receives the pieces of sound data. Further, the voice chat apparatuses10-1to10-3correspond to the communication apparatuses included in the second group. In other words, the voice chat apparatus10associated with the user data whose value of the P2P connection flag is 1 corresponds to the communication apparatus included in the second group.

Further, as described above, the management server14may determine whether a communication apparatus newly added to the voice chat system1is to be included in the first group or the second group, on the basis of the number of communication apparatuses that transmit and receive pieces of sound data to and from each other. For example, as described above, the management server14may determine the value of the P2P connection flag included in the user data corresponding to the newly added voice chat apparatus10, on the basis of the number of voice chat apparatuses10that transmit and receive pieces of sound data to and from each other.

Here, an example of a flow of a sound data transmission process performed in the voice chat apparatus10according to the present embodiment is described with reference to a flow diagram exemplified inFIG.9. Processes indicated in S101to S107ofFIG.9are repeatedly performed every predetermined time period (e.g., every 20 milliseconds or every 40 milliseconds).

First, the sound acceptance section24generates sound data by encoding sound accepted in this loop period (S101).

Then, the VAD data generation section26performs a VAD process on the sound data generated in the process indicated in S101, thereby determining whether or not this sound data represents a human voice (S102).

Then, the VAD data generation section26generates the VAD data according to the result of the determination in the process indicated in S102(S103).

Then, the volume data generation section28identifies the volume of the sound represented by the sound data generated in the process indicated in S101(S104).

Then, the volume data generation section28generates volume data indicating the volume identified in the process indicated in S104(S105).

Then, the sound data transmission section30identifies communication apparatuses that are to be sound data transmission destinations, on the basis of the party management data stored in the party management data storage section20(S106). Here, for example, the sound data transmission section30identifies the addresses of the voice chat apparatuses10that are to be transmission destinations and whether or not the sound data needs to be transmitted to the relay apparatus12.

Then, the sound data transmission section30transmits the sound data generated in the process indicated in S101to the transmission destinations identified in the process indicated in S106(S107), and the process returns to the process indicated in S101. As described above, this sound data is associated with the user ID of the user using the relevant voice chat apparatus10, the VAD data generated in the process indicated in S103, and the volume data generated in the process indicated in S105. Further, data indicating this loop period, such as a timestamp, may be associated with the relevant sound data.

It is noted that, in order to prevent chattering, when it is determined in the process indicated in S102that the sound data represents a human voice, the process indicated in S102may not be performed over a predetermined time period (e.g., one second) starting from the timing of the determination. Then, over this time period, VAD data with a value of 1 may be generated in the process indicated in S103.

Next, an example of a flow of a sound output process performed in the voice chat apparatus10according to the present embodiment will be described with reference to flow diagrams exemplified inFIGS.10A and10B. Processes indicated in S201to S217ofFIGS.10A and10Bare repeatedly performed every predetermined time period (e.g., every 20 milliseconds or every 40 milliseconds). Further, in the following description, it is assumed that the maximum number of pieces of sound data that can be received by the sound output section38in the relevant voice chat apparatus10during a single time period is determined in advance. Hereinafter, the maximum number will be expressed as n1.

First, the selection section34identifies pieces of sound data received by the sound data reception section32during this loop period (S201).

Then, the selection section34checks whether or not the number of pieces of sound data identified in the process indicated in S201(hereinafter expressed as m1) is equal to or less than n1(S202).

Assume that it is confirmed in the process indicated in S202that m1is equal to or less than n1(S202: Y). In this case, the selected sound data output section36outputs all the pieces of sound data identified in the process indicated in S201to the sound output section38(S203).

Assume that it is confirmed in the process indicated in S202that m1is greater than n1(S202: N). In this case, the selection section34identifies, from among the plurality of pieces of sound data identified in the process indicated in S201, any sound data whose value of the associated VAD data is 1 (S204).

Then, the selection section34checks whether or not the number of pieces of sound data identified in the process indicated in S204(hereinafter expressed as m2) is equal to or greater than n1(S205).

Assume that it is confirmed in the process indicated in S205that m2is equal to or greater than n1(S205: Y). In this case, the selection section34identifies n1pieces of sound data in descending order of the volume indicated by the associated volume data from among the plurality of pieces of sound data identified in the process indicated in S204(S206). In a case where m2is equal to n1, all the pieces of sound data identified in the process indicated in S204are identified in the process indicated in S206.

Then, the selected sound data output section36outputs the n1pieces of sound data identified in the process indicated in S206to the sound output section38(S207).

Assume that it is confirmed in the process indicated in S205that m2is less than n1(S205: N). In this case, the selection section34identifies, from among the plurality of pieces of sound data identified in the process indicated in S201, any sound data whose value of the associated VAD data is 0 (S208).

Then, for each of the pieces of sound data identified in the process indicated in S208, the selection section34identifies a moving average of the volume of the sound represented by the sound data (S209). Here, the moving average of the volume of the sound represented by the sound data refers to, for example, the average of the volumes of the sounds represented by the pieces of sound data received from the voice chat apparatus10that is the transmission source of these pieces of sound data during the most recent predetermined round or the most recent predetermined time period (e.g., during the most recent one second). It is noted that, in identifying the moving average, the selection section34may store at least pieces of sound data received over the most recent predetermined round or the most recent predetermined time period.

Then, the selection section34identifies, from among the plurality of pieces of sound data identified in the process indicated in S208, any sound data whose moving average identified in the process indicated in S209is equal to or greater than a predetermined threshold value (e.g., −40 dBOV or greater) (S210).

Then, the selection section34compares the sum of the number of pieces of sound data identified in the process indicated in S204and the number of pieces of sound data identified in the process indicated in S210(hereinafter expressed as m3) with n1(S211).

Assume that it is confirmed in the process indicated in S211that m3is greater than n1. In this case, the selection section34identifies n1pieces of sound data in descending order of the volume indicated by the associated volume data from among the plurality of pieces of sound data identified in the process indicated in S204or S210(S212). Then, the selected sound data output section36outputs the n1pieces of sound data identified in the process indicated in S212to the sound output section38(S213).

Assume that it is confirmed in the process indicated in S211that m3is less than n1. In this case, the selection section34identifies (n1-m3) pieces of sound data in descending order of the volume indicated by the associated volume data from among the remaining pieces of sound data that have not been identified in the process indicated in S204or S210(S214). Then, the selected sound data output section36outputs a total of n1pieces of sound data identified in the process indicated in S204, S210, or S214to the sound output section38(S215).

Assume that it is confirmed in the process indicated in S211that m3is equal to n1. In this case, the selection section34outputs a total of n1pieces of sound data identified in the process indicated in S204or S210to the sound output section38(S216).

Then, the sound output section38decodes the pieces of sound data output in the process indicated in S203, S207, S213, S215, or S216and outputs the sounds represented by these pieces of sound data (S217). Then, the process returns to the process indicated in S201.

According to the processing example illustrated inFIGS.10A and10B, the selection section34of the voice chat apparatus10accurately selects, from among a plurality of pieces of sound data, pieces of sound data that are most likely to represent speeches in voice chat.

It is noted that the sound data selection process is not limited to the one illustrated in the processing example ofFIGS.10A and10B. For example, the selection section34may identify any sound data whose value of the associated VAD data is 1, or whose moving average of the volume is equal to or greater than the predetermined threshold value. Then, from among these pieces of sound data, the selection section34may identify n1pieces of sound data in descending order of the volume indicated by the associated volume data.

Next, an example of a flow of a sound data relay process performed by the relay apparatus12according to the present embodiment will be described with reference to flow diagrams exemplified inFIGS.11A and11B. Processes indicated in S301to S316ofFIGS.11A and11Bare repeatedly performed every predetermined time period (e.g., every 20 milliseconds or every 40 milliseconds). Further, in the following description, it is assumed that the maximum number of pieces of sound data that can be transmitted from the relay apparatus12during a single time period is determined in advance. Hereinafter, the maximum number will be expressed as n2.

Further, it is assumed that the selection section46of the relay apparatus12has preliminarily identified a plurality of voice chat apparatuses10that are to be sound data transmission destinations, on the basis of the party management data stored in the party management data storage section40.

First, the selection section46identifies pieces of sound data received by the sound data reception section44during this loop period (S301).

Then, the selection section46checks whether or not the number of pieces of sound data identified in the process indicated in S301(hereinafter expressed as m4) is equal to or less than n2(S302).

Assume that it is confirmed in the process indicated in S302that m4is equal to or less than n2(S302: Y). In this case, the sound data transmission section48transmits all the pieces of sound data identified in the process indicated in S301to the voice chat apparatuses10that are the transmission destinations (S303), and the process returns to the process indicated in S301.

Assume that it is confirmed in the process indicated in S302that m4is greater than n2(S302: N). In this case, the selection section46identifies, from among the plurality of pieces of sound data identified in the process indicated in S301, any sound data whose value of the associated VAD data is 1 (S304).

Then, the selection section46checks whether or not the number of pieces of sound data identified in the process indicated in S304(hereinafter expressed as m5) is equal to or greater than n2(S305).

Assume that it is confirmed in the process indicated in S305that m5is equal to or greater than n2(S305: Y). In this case, the selection section46identifies n2pieces of sound data in descending order of the volume indicated by the associated volume data from among the plurality of pieces of sound data identified in the process indicated in S204(S306). In a case where m5is equal to n2, all the pieces of sound data identified in the process indicated in S304are identified in the process indicated in S306.

Then, the sound data transmission section48transmits the n2pieces of sound data identified in the process indicated in S306to the voice chat apparatuses10that are the transmission destinations (S307), and the process returns to the process indicated in S301.

Assume that it is confirmed in the process indicated in S305that m5is less than n2(S305: N). In this case, the selection section46identifies, from among the plurality of pieces of sound data identified in the process indicated in S301, any sound data whose value of the associated VAD data is 0 (S308).

Then, for each of the pieces of sound data identified in the process indicated in S308, the selection section46identifies a moving average of the volume of the sound represented by the relevant sound data (S309). Here, the moving average of the volume of the sound represented by the sound data refers to, for example, the average of the volumes of the sounds represented by the pieces of sound data received from the voice chat apparatus10that is the transmission source of these pieces of sound data during the most recent predetermined round or the most recent predetermined time period (e.g., during the most recent one second). It is noted that, in identifying the moving average, the selection section46may store at least pieces of sound data received over the most recent predetermined round or the most recent predetermined time period.

Then, the selection section46identifies, from among the plurality of pieces of sound data identified in the process indicated in S308, any sound data whose moving average identified in the process indicated in S309is equal to or greater than a predetermined threshold value (e.g., −40 dBOV or greater) (S310).

Then, the selection section46compares the sum of the number of pieces of sound data identified in the process indicated in S304and the number of pieces of sound data identified in the process indicated in S210(hereinafter expressed as m6) with n2(S311).

Assume that it is confirmed in the process indicated in S311that m6is greater than n2. In this case, the selection section46identifies n2pieces of sound data in descending order of the volume indicated by the associated volume data from among the plurality of pieces of sound data identified in the process indicated in S304or S310(S312). Then, the sound data transmission section48transmits the n2pieces of sound data identified in the process indicated in S312to the voice chat apparatuses10that are the transmission destinations (S313), and the process returns to the process indicated in S301.

Assume that it is confirmed in the process indicated in S311that m6is less than n2. In this case, the selection section46identifies (n2-m6) pieces of sound data in descending order of the volume indicated by the associated volume data from among the remaining pieces of sound data that have not been identified in the process indicated in S304or S310(S314). Then, the selected sound data output section36transmits a total of n2pieces of sound data identified in the process indicated in S304, S310, or S314to the voice chat apparatuses10that are the transmission destinations (S315), and the process returns to the process indicated in S301.

Assume that it is confirmed in the process indicated in S311that m6is equal to n2. In this case, the selection section46transmits a total of n2pieces of sound data identified in the process indicated in S304or S310to the voice chat apparatuses10that are the transmission destinations (S316), and the process returns to the process indicated in S301.

According to the processing example illustrated inFIGS.11A and11B, the selection section46of the relay apparatus12accurately selects, from among a plurality of pieces of sound data, pieces of sound data that are most likely to represent speeches in voice chat.

The sound data selection based on the VAD data can reduce the possibility of selecting sound data representing a sound other than a human voice, such as the sound of a desk being tapped or the sound of an ambulance, for example. Further, the sound data selection based on the moving average of the volume can increase the possibility of selecting, for example, sound data that has not been selected during the sound data selection based on the VAD data but actually represents a human voice.

It is noted that the sound data selection process is not limited to the one illustrated in the processing example ofFIGS.11A and11B. For example, the selection section46may identify any sound data whose value of the associated VAD data is 1, or whose moving average of the volume is equal to or greater than the predetermined threshold value. Then, from among these pieces of sound data, the selection section46may identify n2pieces of sound data in descending order of the volume indicated by the associated volume data.

Further, as described above, the selection section34or the selection section46may select a portion of the plurality of pieces of sound data on the basis of the moving averages of the volumes of the sounds represented by the pieces of sound data. With this configuration, the sound data selection is performed in a stable manner.

Further, in the processing example illustrated inFIGS.11A and11B, from among the plurality of pieces of sound data, the selection section46may select pieces of sound data in the number determined on the basis of the type of voice chat apparatuses10that are to be sound data transmission destinations. Then, the sound data transmission section48may transmit the portion of the pieces of sound data to the relevant voice chat apparatuses10. In this case, the above-described number n2varies depending on the types of voice chat apparatuses10that are to be transmission destinations.

Since smartphones are highly likely to use a carrier's network, it is particularly necessary to suppress the communication volume of pieces of sound data flowing through the network. In light of this circumstance, it is preferable that a smaller number of pieces of sound data be transmitted to the voice chat apparatuses10that are smartphones than to the voice chat apparatuses10that are game consoles, for example.

Further, pieces of sound data in the number determined on the basis of the load on the apparatus or the communication quality of the computer network16may be selected from among the plurality of pieces of sound data.

For example, the selection section34of the voice chat apparatus10may determine the above-described value n1on the basis of the load on the relevant voice chat apparatus10.

Further, the selection section46of the relay apparatus12may determine the above-described value n2on the basis of the load on the relay apparatus12or the communication quality (e.g., communication volume) of the computer network16.

According to the voice chat system1described above, it is possible to appropriately thin out the output of pieces of sound data representing sounds in voice chat. For example, the transmission output of pieces of sound data from the relay apparatus12and the output of pieces of sound data from the processor10aof the voice chat apparatus10to the encoding and decoding section10hcan be appropriately thinned out.

It is noted that the present invention is not limited to the embodiment described above.

For example, in the process illustrated inFIGS.10A and10B, the selection section34may store a list of user IDs representing users who have recently spoken. The list may include n1user IDs.

Then, the selection section34may select a portion of pieces of sound data transmitted from the voice chat apparatuses10used by users whose user IDs are not included in the list. Here, n3, which is the number of pieces of sound data to be selected, is less than n1. Then, the selection section34may remove n3user IDs from the list. For example, n3user IDs may be removed from the list in ascending order of the timing of addition to the list. Further, n3user IDs may be deleted from the list in ascending order of the volume of the sound represented by the sound data in the most recent period.

Then, the selection section34may add the user IDs associated with the selected n3pieces of sound data to the list.

Then, the selection section34may select the pieces of sound data associated with the user IDs included in the list. In this manner, the sound data selection in the voice chat apparatus10is performed in a stable manner.

Further, similarly, in the process illustrated inFIGS.11A and11B, the selection section46may store a list including n2user IDs. Then, as with the selection section34in the above-described example, the pieces of sound data associated with the user IDs included in the list may be selected. With this configuration, the sound data selection in the relay apparatus12is performed in a stable manner.

Further, how the roles of the voice chat apparatus10and the relay apparatus12are split is not limited to those described above. For example, the relay apparatus12may determine whether or not the sound data represents a human voice and identify the volume of the sound represented by the sound data. Alternatively, for example, the voice chat apparatus10that receives sound data may determine whether or not the sound data represents a human voice and identify the volume of the sound represented by the sound data.

Further, the above-described specific character strings and numerical values and the specific character strings and numerical values in the drawings are examples, and the character strings and the numerical values are not limited thereto.