VIDEO DISPLAY SYSTEM AND VIDEO DISPLAY METHOD

A video display system includes: a VR system including a display that displays a VR space to a participant who participates in a VR experience, and an audio processor that receives and outputs voice; an explainer terminal that receives and outputs voice from and to an explainer who provides an explanation to the participant in the VR space; a video distributor that distributes a 360-degree video for the VR space to the VR system; an audio conversation unit that assists an audio conversation between the participant and the explainer; and a controller that manages an operation mode that selectively takes a first state in which questions from the participant to the explainer are not allowed, a second state in which questions are allowed, and a third state in which a question is being asked, and causes the VR system to display information specifying the operation mode on the display.

TECHNICAL FIELD

The present invention relates to video display systems and video display methods, and particularly relates to video display systems that provide services for an experience of a virtual reality (VR).

BACKGROUND ART

Conventionally, various techniques have been proposed as video display systems that provide a service for experiencing VR (virtual reality) (see, for example, Patent Literature (PTL) 1 and PTL 2).

PTL 1 proposes an at-home travel system which can give a remote user, by providing the remote user with a present actual situation of a tourist destination of choice in the form of real-time video, an impression and a realistic sensation as if the remote user is actually visiting the tourist destination.

In addition, PTL 2 proposes an information processing device related to point-of-view control of a participant in sightseeing using VR.

CITATION LIST

Patent Literature

SUMMARY OF INVENTION

Technical Problem

However, the techniques described in PTL 1 and PTL 2 are premised on the participants having high ICT (information and communication technology) competence and have a problem in that not everyone can readily participate.

In consideration thereof, an object of the present disclosure is to provide a video display system and a video display method in which everyone can readily participate.

Solution to Problem

To achieve the above object, a video display system according to one embodiment of the present disclosure is a video display system that provides a service for an experience of a virtual reality (VR), and includes: a VR system including a display that displays a VR space that is a video for the VR to a participant who participates in the experience, and an audio processor that receives and outputs voice from and to the participant; an explainer terminal that receives and outputs voice from and to an explainer, the explainer providing an explanation for the experience to the participant in the VR space; a video distributor that distributes a 360-degree video for generating the VR space to the VR system; an audio conversation unit that assists an audio conversation between the participant and the explainer via the VR system and the explainer terminal; and a controller that controls at least one of the VR system, the explainer terminal, the video distributor, or the audio conversation unit. The controller manages an operation mode that selectively takes a first state in which a question from the participant to the explainer is not allowed, a second state in which the question is allowed, and a third state in which the question is being asked, and causes the VR system to display information specifying the operation mode on the display.

To achieve the above object, a video display method according to one embodiment of the present disclosure is a video display method performed by a video display system that provides a service for an experience of a virtual reality (VR), and includes: a VR system including a display that displays a VR space that is a video for the VR to a participant who participates in the experience, and an audio processor that receives and outputs voice from and to the participant; an explainer terminal that receives and outputs voice from and to an explainer, the explainer providing an explanation for the experience to the participant in the VR space; a video distributor that distributes a 360-degree video for generating the VR space to the VR system; and an audio conversation unit that assists an audio conversation between the participant and the explainer via the VR system and the explainer terminal. The video display method includes: managing an operation mode that selectively takes a first state in which a question from the participant to the explainer is not allowed, a second state in which the question is allowed, and a third state in which the question is being asked, and causing the VR system to display information specifying the operation mode on the display.

Advantageous Effects of Invention

The present disclosure provides a video display system and a video display method in which everyone can readily participate.

DESCRIPTION OF EMBODIMENTS

Knowledge Obtained by the Inventors

VR tourism services which realize virtual experiences such as going on a sightseeing tour, attending a trade show, making an inspection tour, going on a factory tour, visiting an art museum, a museum, a zoo, an aquarium, or the like without actually being on-site are gaining momentum.

In a VR tourism service, it is important from the perspective of a tourism experience that a plurality of visitors use a VR system to participate in a VR space of a destination and listen to an explanation on the destination provided by a human (or avatar) tour guide (leader) or ask the guide questions as if on an actual sightseeing tour. Simply listening to an explanation provided by the guide in a one-sided manner is no different than viewing a tourism VR program on YouTube (registered trademark) or the like. It is difficult to charge for such a level of service due to a lack of appeal. What is crucially important to a VR tourism service is that an interactive trip experience involving listening to an explanation provided by a guide with expertise or an expert and having the guide or the expert answer questions be provided.

Since participants in a VR tourism service are people with a wide range of experiences, ICT competence, characteristics, languages, and the like and include both young and old as well as foreigners, it is important to provide a service satisfying requirements such as those presented below.

To this end, support including (1) a user interface that anybody can readily use, (2) coping with bad-mannered participants, (3) private conversations between friends, and (4) accommodating a plurality of languages and accommodating a plurality of guides is required.

In order to solve each of the four problems described above, the VR tourism system according to the present disclosure can provide a VR tourism system in which anyone can readily participate by using a 360-degree video distribution system and a digital audio conference unit and realizing means that allows VR tourism participants and guides to respectively operate three states with a simple UI (user interface) constituted of one button. In other words, based on a VR tourism system that solves the problem described in (1) above, the problems described in (2) to (4) above have been solved by adding various additional functions to such a VR tourism system. Accordingly, since a service by a simple UI in which anybody from the elderly to children can participate without knowledge of ICT can be realized in VR tourism, VR tourism can be enjoyed just like real sightseeing.

Note that the VR tourism system according to the present disclosure need not necessarily solve all of the four problems described above and need only solve at least one of the problems (1) to (4) described above as a video display system in which anyone can readily participate.

(VR Tourism System According to Reference Example)

FIG.1Ais a conceptual diagram of VR tourism system650according to a reference example which enables conversations with a guide. VR tourism system650according to the reference example enables a simulated tourism experience with a realistic sensation with respect to a location by sending, to respective VR users10to13, video and audio (ambient sound of the location, audio of guide20, and the like) obtained by photographing the location with a 360-degree camera. Note that a “VR user” may also be referred to as a “VR tourism user”, a “participant”, or simply a “user”.

In order to further improve the VR tourism experience, it is important that VR users10to13do not listen to an explanation provided by guide20in a one-sided manner but are capable of engaging in two-way conversations with guide20while wearing VR systems600to630. In consideration thereof, as illustrated inFIG.1A, VR tourism system650according to the reference example realizes a function of engaging in a conversation with other VR users10to13or engaging in a conversation with guide20even during VR tourism by appropriately combining VR audio conference unit810with 360-degree video distributor820in VR cloud800.

FIG.1Bis a block diagram illustrating a detailed configuration of VR tourism system650illustrated inFIG.1A. VR systems600and the like include VR audio conference unit605that enables an audio conversation during VR tourism. VR audio conference unit605is constituted of A/D converter108athat converts an audio signal outputted from microphone108to digital audio data, audio compressor108bthat compresses the audio data, and muter108cthat controls whether or not to transmit the compressed audio data to VR cloud800.

VR audio conference unit810included in VR cloud800includes audio decoders314ato314dthat decode audio data transmitted from VR systems600and the like, audio decoder316that decodes compressed audio data transmitted from smartphone710used by guide20, audio mixer313that mixes the audio data from audio decoders314ato314dand the audio data from audio decoder316to generate composite audio data, and audio compressor317that compresses the mixed composite audio data and transmits the compressed composite audio data to smartphone710of guide20.

360-degree video distributor820included in VR cloud800includes AV demultiplexer321that receives a 360-degree video transmitted from observation system700and demultiplexes the 360-degree video into video data and audio data, audio decoder322that decodes the demultiplexed audio data, audio mixer323that mixes the decoded audio data and composite audio data sent from VR audio conference unit810, audio compressor324that compresses the mixed audio data, and AV compositor325that composites the compressed audio data and the video data demultiplexed by AV demultiplexer321and distributes as a 360-degree video to VR system600and the like.

Smartphone710that is a guide's audio conference terminal has hardware and an application (APP) which function as VR audio conference unit750and, specifically, smartphone710is constituted of A/D converter213athat converts an audio signal outputted from a microphone of headset720to digital audio data, audio compressor213bthat compresses the audio data, muter213cthat controls whether or not to transmit the compressed audio data to VR cloud800, audio decoder214athat receives and decodes composite audio data transmitted from VR cloud800, and D/A converter214bthat converts the decoded composite audio data to an analog audio signal and outputs the analog audio signal to a loudspeaker of headset720.

In this manner, in VR tourism system650according to the reference example, function additions (1) to (3) described below are performed in order to realize conversations between VR users10to13and guide20by combining a general videoconferencing system with the video distribution system including 360-degree video distributor820.

(1) VR audio conference unit605for connecting to VR audio conference unit810of VR cloud800is added to VR systems600and the like.

(2) Guide20is caused to use smartphone710that is a guide's audio conference terminal for connecting to VR audio conference unit810of VR cloud800from headset720mounted with a microphone, an earphone, or the like via an ICT device such as a smartphone.

(3) Composite audio data obtained by mixing voices of VR users10and the like and guide20in VR audio conference unit810of VR cloud800is mixed with only audio data extracted by 360-degree video distributor820from 360-degree video data sent from the observation system and the mixed audio data is sent to VR system600or the like of each VR user10or the like.

Note that a position of muter108cis not limited to the illustrated position as long as audio captured by microphone108is muted before the audio is mixed by audio mixer313.

Furthermore, a configuration may be adopted in which muter108cis placed before audio decoders314ato314d(on an input side relative to audio decoders314ato314d) or between audio decoders314ato314dand audio mixer313of VR audio conference unit810in VR cloud800.

(Problems in VR Tourism System According to Reference Example)

In cases of videoconferencing systems such as Zoom (registered trademark) and Teams (registered trademark), since the videoconferencing systems are used for business and their users can be expected to have certain ICT skills, operations such as muting and un-muting can be performed by the users themselves. However, VR tourism users may possibly be users in a wide range of ages from the elderly to children, a wide variety of occupations, educational levels, and the like and all participants cannot be expected to be skilled in ICT. Therefore, problems such as those described below may possibly occur.

In particular, when each VR user is not speaking, defects such as mixing of noise and, at the worst, howling or the like may occur unless the VR user performs muting using a mute function of his/her own VR system. However, it is conceivably difficult to have a VR user unaccustomed to videoconferencing systems such as Zoom (registered trademark) and Teams (registered trademark) perform a mute operation or learn how to perform the operation in a VR system.

Problems to be Solved

As described above, the VR tourism system according to the reference example has the following four problems (1) to (4) to be solved.

(1) A simple user interface that enables diverse participants to use the VR tourism service must be provided.

Unlike users of a videoconferencing system that is used for business, education, and the like, people of various age groups and levels of understanding of ICT skills may participate as VR tourism users. In addition, a field of view is blocked by entering a VR space while wearing VR goggles. Therefore, operations using a controller or the like need to be easy and a simple user interface that satisfies conditions such as those described below is required. It is also necessary that a user need not learn a concept of muting or learn how to perform a muting operation. It is also necessary that VR experience does not decline even when a person unable to learn how to perform an operation, a person having forgotten how to perform an operation, or a person who does not follow rules participates. Furthermore, it is important that a description of operations of the VR system can be readily provided before the VR experience.

In addition, a main duty of guide20is to provide explanations and guide20is unable to perform system operations in a same manner as a host of a videoconferencing system. Guide20requires an extremely simple user interface that enables guide20to focus on the duty of providing explanations.

(2) Bad-mannered VR users who obstruct provision of an explanation by guide20by verbally abusive language, continuous questions, and the like must be eliminated. With a videoconferencing system, since a purpose of use is business, there is no need to consider bad-mannered participants. In addition, even when a problem occurs, a system manager can block an utterance by forced muting or the like. A VR tourism service requires a function which enables, when guide20encounters a problem, the problem to be solved (a conversation to be blocked) by a simple operation.

(3) Private conversations by participants with traveling companions must be supported. In other words, free conversations within a group must be supported without obstructing provision of explanations by guide20and without the conversations being heard by participants belonging to other groups.

(4) A plurality of languages and a plurality of guides must be accommodated with respect to participants in VR tourism in order to accommodate foreign travel and foreigners. In other words, in order to simultaneously support users who use a plurality of languages, a plurality of guides, simultaneous interpreters, machine translation, and the like must also be supported.

Hereinafter, solutions to each of the four problems listed above will be described using an embodiment and examples. Note that the embodiment and the examples to be described below each present a specific example of the present disclosure. Numerical values, shapes, materials, constituent elements, the arrangement and connection of the constituent elements, steps, an order of the steps, and the like described in the following embodiment and the following examples are merely examples and are not intended to limit the present disclosure. Furthermore, the respective drawings are not necessarily precise illustrations. In the drawings, substantially same components will be denoted by same reference signs and redundant descriptions will be either omitted or simplified.

EMBODIMENT

FIG.2Ais a block diagram illustrating a configuration of video display system1according to the embodiment. Video display system1is a system that provides a service for experiencing VR (virtual reality) and includes VR systems30ato30bused by participants10to13, explainer terminal40, observation system50, and cloud60. VR systems30ato30b, explainer terminal40, and observation system50are connected to cloud60via a communication network.

Each of VR systems30aand30bincludes display unit31that displays a VR space that is a video for VR to participants10to13who participate in the experience and audio processor32that inputs and outputs audio from and to participants10to13.

Explainer terminal40is a terminal that inputs and outputs audio of explainer20who provides explanations for the experience towards participants10to13in the VR space and includes explainer's input unit41that accepts an operation from explainer20indicating whether or not questions are to be accepted.

Observation system50is a system that generates a 360-degree video by photographing the real world and transmitting the 360-degree video to video distributor63of cloud60and includes camera51and microphone52that picks up audio of explainer20.

Cloud60refers to a computer on a communication network and includes video distributor63that distributes a 360-degree video for generating a VR space to VR systems30ato30b, audio conversation unit61that assists audio conversation between participants10to13and explainer20via VR systems30ato30band explainer terminal40, and controller62that controls at least one of VR systems30ato30b, explainer terminal40, video distributor63, and audio conversation unit61.

In this case, controller62controls VR systems30ato30bso as to manage an operation mode that selectively takes a first state in which questions from participants10to13to explainer20are not allowed, a second state in which questions are allowed, and a third state in which a question is in progress, and causes display unit31to display information specifying the operation mode. More specifically, controller62controls the plurality of VR systems30ato30bso as to manage an operation mode corresponding to each of the plurality of VR systems30ato30band to cause information specifying the operation mode to be displayed on display unit31of a corresponding VR system among VR systems30ato30b.

Accordingly, each of participants10to13can immediately know whether his/her own state is the first state in which questions cannot be asked, the second state in which questions can be asked, or the third state in which a question is being asked based on information specifying an operation mode that is displayed on display unit31, an operation for asking explainer20a question is simplified, and the problem (1) (a simple user interface that enables diverse participants to use the VR tourism service must be provided) described above is solved.

Note that video display system1does not necessarily need observation system50. Video distributor63of cloud60may read a 360-degree video created in advance by photography, CG, or the like from storage in which the 360-degree video had been stored and distribute the 360-degree video to VR systems30ato30b.

In addition, controller62need not be provided in one computer that constitutes cloud60and all of or a part of controller62may be provided in other devices such as a plurality of computers, VR systems30ato30b, and explainer terminal40. In other words, controller62may be divided into a plurality of functional modules and provided by being distributed among a plurality of devices.

When explainer's input unit41accepts an instruction to enter the first state or, in other words, a state of not accepting questions, controller62may control the plurality of VR systems30ato30bto display an image indicating the first state on display unit31of the plurality of VR systems30ato30band to mute the first state or, in other words, audio input from the plurality of participants10to13but, on the other hand, when explainer's input unit41accepts an instruction to enter the second state or, in other words, a state of accepting questions, controller62may control the plurality of VR systems30ato30bto display an image indicating the second state on display unit31of the plurality of VR systems30ato30bto create the second state or, in other words, a state in which questions are allowed and to change audio input from the plurality of participants10to13to the first state or, in other words, to mute the audio input. Accordingly, explainer20can control whether or not to allow questions with respect to the plurality of participants10to13by a simple operation using explainer's input unit41.

In addition, when participant's input unit33of any one of the plurality of VR systems30ato30breceives an operation indicating a request to ask a question and an operation mode of VR system30aor the like corresponding to participant's input unit33having received the operation is the second state, controller62may control VR system30aor the like to display information specifying the third state instead of the second state on display unit31and, at the same time, change the operation mode to the third state by un-muting audio input from the corresponding participant. Accordingly, participants10to13can ask explainer20questions by a simple operation using participant's input unit33.

Furthermore, when controller62controls VR system30aor the like so as to unmute audio input from participant10or the like or, in other words, create the third state, controller62may control VR system30aor the like so as to display an image indicating the first state on display unit31and create the first state by muting audio input from a corresponding participant13or the like with respect to another VR system30bor the like excluding VR system30aor the like. Accordingly, when one participant asks explainer20a question, the other participants are prevented from presenting questions and the other participants can listen to a conversation between the one participant and explainer20.

In this case, explainer's input unit41may be a single button for receiving an operation indicating accepting questions and an operation indicating not accepting questions. Accordingly, explainer20can switch between a state of accepting questions and a state of not accepting questions by a simple operation using a single button.

In a similar manner, participant's input unit33may be a single button for receiving an operation indicating a request to ask a question and indicating that a question is in progress and an operation indicating that a question is completed. Accordingly, participants10to13can start asking explainer20a question by a simple operation using a single button.

In addition, in place of or in addition to explainer's input unit41of explainer terminal40, audio conversation unit61may assist audio conversation between participants10to13and explainer20using audio of explainer20acquired by microphone52of observation system50. Accordingly, explainer20can provide an explanation such as tourism guidance and engage in conversation including answering questions from participants10to13using microphone52of observation system50.

FIG.2Bis a block diagram illustrating a configuration of controller62aaccording to Variation 1 of video display system1illustrated inFIG.2A. In the present variation, controller62aincludes blocking information70including permission/refusal information indicating whether or not to block a question to explainer20from each of the plurality of participants10to13and, based on blocking information70, with respect to participant10or the like indicated to be blocked in the permission/refusal information, controls VR system30aor the like so that an image indicating the first state is displayed on display unit31of VR system30aor the like corresponding to participant10or the like and the first state is created by muting audio input from participant10or the like. Accordingly, the problem (2) (bad-mannered users who obstruct provision of an explanation by guide20by verbally abusive language, continuous questions, and the like must be eliminated) described above is solved.

FIG.2Cis a block diagram illustrating a configuration of cloud60aaccording to Variation 2 of video display system1illustrated inFIG.2A. Cloud60aincludes audio conversation unit61, controller62b, video distributor63, and group audio conversation unit64. Group audio conversation unit64assists a group audio conversation among two or more participants10or the like selected from the plurality of participants10to13. Controller62bincludes group information71including affiliation information indicating whether or not each of the plurality of participants10to13belongs to a predetermined group and, based on group information71, controls group audio conversation unit64so as to assist group audio conversation only with respect to the plurality of participants10and the like that belong to a same group. Accordingly, the problem (3) (private conversations by participants with traveling companions must be supported) described above is solved.

More specifically, controller62bcontrols group audio conversation unit64so as to assist group audio conversation when the operation modes of VR systems30aand the like corresponding to the plurality of participants10and the like belonging to the same group are all the first state or the second state.

When the operation mode of VR systems30aor the like corresponding to any one of participants10or the like belonging to the same group makes a transition from the second state to the third state, controller62bcontrols VR systems30ato30bso as to stop assisting the group audio conversation and, for each of other participants13and the like excluding participant10or the like, causes display unit31of corresponding VR system30bor the like to display an image indicating the first state and mute audio input from participant13and the like. Accordingly, when one participant starts to ask explainer20a question during a group conversation, the other participants can listen to the question.

In addition, while assisting the group audio conversation, controller62bmay control a plurality of VR systems30aand the like that correspond to a plurality of participants10and the like who belong to a same group so as to adjust a volume of at least one of audio from explainer20to the plurality of participants10and the like belonging to the same group and audio of the group audio conversation. Accordingly, the participants of a group audio conversation can listen to the group audio conversation and to an explanation provided by explainer20in a state where there is a mixture of the group audio conversation and the explanation.

FIG.2Dis a block diagram illustrating a configuration of video display system1aaccording to Variation 3 of video display system1illustrated inFIG.2A. Video display system1aincludes, in video display system1illustrated inFIG.2A, first explainer terminal40aand second explainer terminal40binstead of explainer terminal40and cloud60binstead of cloud60.

First explainer terminal40ais a terminal that inputs and outputs audio to and from first explainer20awho provides explanations in a first language. Second explainer terminal40bis a terminal that inputs and outputs audio to and from second explainer20bwho provides explanations in a second language.

Cloud60bincludes audio conversation unit61, controller62c, and video distributor63. Audio conversation unit61includes first audio conversation unit61athat assists an audio conversation between participants10to13and first explainer20avia the plurality of VR systems30ato30band first explainer terminal40aand second audio conversation unit61bthat assists an audio conversation between participants10to13and second explainer20bvia the plurality of VR systems30ato30band second explainer terminal40b.

Controller62cincludes language information72that associates each of the plurality of participants10to13with a desired language of the provided explanation and controls first audio conversation unit61aand second audio conversation unit61bso that, based on language information72, with respect to participant10or the like associated with the first language, an audio conversation between participant10or the like and first explainer20ais assisted by first audio conversation unit61aand, with respect to participant13or the like associated with the second language, an audio conversation between participant13or the like and second explainer20bis assisted by second audio conversation unit61b. Accordingly, the problem (4) (a plurality of languages and a plurality of guides must be accommodated with respect to participants in VR tourism in order to accommodate foreign travel and foreigners) described above is solved.

Note that second explainer20bmay be a simultaneous interpreter who interprets between the first language and the second language. In such a case, first audio conversation unit61aalso outputs audio from first explainer20ato second explainer20b. Accordingly, a video display system that also accommodates cases including a simultaneous interpreter is realized.

FIG.2Eis a block diagram illustrating a configuration of cloud60caccording to Variation 1 of video display system1aillustrated inFIG.2D. Cloud60cincludes audio conversation unit61, controller62c, video distributor63, and audio switcher65. Audio switcher65switches between outputting audio from a simultaneous interpreter to participant10or the like associated with the second language and outputting audio to first explainer20aand participant13or the like associated with the first language. Accordingly, when second explainer20bis a simultaneous interpreter, the simultaneous interpreter can switch between speaking to a participant and speaking to first explainer20awho speaks the first language and convenience is improved.

FIG.2Fis a block diagram illustrating a configuration of cloud60daccording to Variation 2 of video display system1aillustrated inFIG.2D. Cloud60dincludes audio conversation unit61, controller62d, video distributor63, and machine translators66ato66b. Machine translators66ato66bperform machine translation between two different languages (such as the first language and the second language).

Controller62dincludes language information72that associates each of the plurality of participants10to13with a desired language of the provided explanation and controls machine translators66aand the like so that, based on language information72, with respect to participant10or the like associated with the first language, participant10or the like and explainer20engage in an audio conversation without involving machine translators66ato66band, with respect to participant13or the like associated with the second language, participant13or the like and explainer20engage in an audio conversation through machine translators66aand the like. Accordingly, a plurality of languages can be accommodated due to machine translation.

In addition, controller62may include machine translation management information73indicating two languages handled by the plurality of machine translators66ato66band, based on language information72and machine translation management information73, controller62may control the plurality of machine translators66ato66bsuch that for each of participants10to13, a machine translator that translates between a language associated with participants10to13and the first language used by the guide is selected from among the plurality of machine translators66ato66band participant10to13and explainer20engage in an audio conversation via the machine translator selected from machine translators66ato66b. Accordingly, by accommodating a plurality of different kinds of machine translation, an extremely large number of languages can be accommodated.

Moreover, as will be illustrated in operation flow examples of examples to be described later, a video display method according to the present disclosure is a video display method by video display system1that provides a service of experiencing VR (virtual reality), the video display method including: controlling VR systems30ato30bso as to manage an operation mode that selectively assumes a first state in which questions from participants10to13to explainer20are not allowed, a second state in which the questions are allowed, and a third state in which a question is in progress and cause display unit31to display information specifying the operation mode (FIG.6). More specifically, in the controlling of VR systems30ato30b, the plurality of VR systems30ato30bare controlled so as to manage an operation mode corresponding to each of the plurality of VR systems30ato30band to cause information specifying the operation mode to be displayed on display unit31of a corresponding VR system among VR systems30ato30b(FIG.6,FIG.2A).

Accordingly, each of participants10to13can immediately know whether his/her own state is the first state in which questions cannot be asked, the second state in which questions can be asked, or the third state in which a question is being asked based on information displayed on display unit31that specifies an operation mode, an operation for asking explainer20a question is simplified, and the problem (1) (a simple user interface that enables diverse participants to use the VR tourism service must be provided) described above is solved.

In addition, in the controlling of VR systems30ato30b, VR systems30ato30bare controlled so that, based on blocking information70including permission/refusal information indicating whether or not to block a question to explainer20from each of the plurality of participants10to13, with respect to participants10to13indicated to be blocked in the permission/refusal information, an image indicating the first state is displayed on display unit31of VR systems30ato30bcorresponding to participants10to13and the first state is created by muting audio input from participants10to13(FIG.13,FIG.2B). Accordingly, the problem (2) (bad-mannered users who obstruct provision of explanations by guide20by verbally abusive language, continuous questions, and the like must be eliminated) described above is solved.

In addition, the controlling of VR systems30ato30bincludes group information71including affiliation information indicating whether or not each of the plurality of participants10to13belongs to a predetermined group and, based on group information71, group audio conversation unit64is controlled so as to assist group audio conversation with respect to the plurality of participants10to13that belong to a same group (FIG.13,FIG.2C). Accordingly, the problem (3) (private conversations by participants with traveling companions must be supported) described above is solved.

In addition, in the controlling of VR systems30ato30b, based on language information72that associates each of the plurality of participants10to13with a desired language of the provided explanation, first audio conversation unit61aand second audio conversation unit61bare controlled so that with respect to participants10to13associated with the first language, an audio conversation between participants10to13and first explainer20ais assisted by first audio conversation unit61aand, with respect to participants10to13associated with the second language, an audio conversation between participants10to13and second explainer20bis assisted by second audio conversation unit61b(FIG.25,FIG.2D). Accordingly, the problem (4) (a plurality of languages and a plurality of guides must be accommodated with respect to participants in VR tourism in order to accommodate foreign travel and foreigners) described above is solved.

In addition, in the controlling of VR systems30ato30b, based on language information72that associates each of the plurality of participants10to13with a desired language of the provided explanation, machine translators66ato66bare controlled so that, with respect to participants10to13associated with the first language, participants10to13and explainer20engage in an audio conversation without involving machine translators66ato66band, with respect to participants10to13associated with the second language, participants10to13and explainer20engage in an audio conversation through machine translators66ato66b(FIG.35,FIG.2F). Accordingly, the problem (4) (a plurality of languages and a plurality of guides must be accommodated with respect to participants in VR tourism in order to accommodate foreign travel and foreigners) described above is solved.

Hereinafter, as specific examples of the embodiment, first to fourth examples that solve the problems (1) to (4) described above will be described. Every example will be described as an example of applying the video display system according to the embodiment to a VR tourism system.

First, Example 1 will be described as a specific example of solving the problem (1) (a simple user interface that enables diverse participants to use the VR tourism service must be provided) described above.

FIG.3is a schematic configuration diagram of VR tourism system2according to Example 1. VR tourism system2is constituted of: A) VR systems100,110,120, and130for VR users10to13to experience a VR tourism service; B) 360-degree video distributor320which is provided in VR cloud300and which distributes a 360-degree video for generating a VR space inside VR systems100,110,120, and130; C) VR audio conference unit310which is provided in VR cloud300and which realizes an audio conversation between each of VR systems100,110,120, and130and guide20; and D) guide's terminal210or the like constituted of smartphone210, headset220, and guide's remote controller230for guide20to engage in audio conversation with VR users10to13.

More specifically, VR system100(110,120, and130) includes display101(111,121, and131), input unit102(112,122, and132) that accepts operations, communicator104(114,124, and134) that communicates with the VR cloud, and VR space generator103(113,123, and133) that generates a VR space to be displayed on display101(111,121, and131). VR space generator103(113,123, and133) and display101(111,121, and131) are examples of display unit31according to the embodiment and input unit102(112,122, and132) is an example of participant's input unit33according to the embodiment. Note that illustration of audio processor32according to the embodiment is omitted in the present schematic configuration diagram.

VR cloud300includes VR audio conference unit310including conversation controller311and audio processor312and 360-degree video distributor320. VR audio conference unit310is an example of audio conversation unit61according to the embodiment, 360-degree video distributor320is an example of video distributor63according to the embodiment, and conversation controller311is an example of controller62according to the embodiment.

Observation system200includes communicator201, capturing unit202, camera203, remote controller204, and microphone205. Camera203is an example of camera51according to the embodiment, and microphone205is an example of microphone52according to the embodiment.

Guide's terminal210or the like is an example of explainer terminal40according to the embodiment and is constituted of smartphone210, headset220, and guide's remote controller230that is an example of explainer's input unit41according to the embodiment. Guide instruction generator211is a function of acquiring an instruction of guide20from guide's remote controller230and outputting the instruction to VR audio conference unit310and is an example of the function of acquiring an instruction of guide20from explainer's input unit41among functions of controller62according to the embodiment.

While participants can speak freely as a general rule in a general audio conference system, since VR tourism system2is for VR tourism, by having VR users10to13normally listen to explanations provided by guide20and enabling VR users10to13to speak (ask a question) only when permitted by guide20instead of allowing participants to speak freely, both VR users10to13and guide20can experience VR tourism with a simple user interface.

To this end, audio/display control (mute function, guidance display, and the like) of VR systems100,110,120, and130of respective users10to13is realized by having guide20use guide instruction generator211in guide's terminal210or the like to operate conversation controller311in VR cloud300.

FIG.4is a diagram illustrating an operation example of VR tourism system2according to Example 1. More specifically, (a) inFIG.4illustrates a display example in VR systems100,110,120, and130, display examples240aand240busing an audio control application in smartphone210of guide20, an operation example of guide's remote controller230, and an operation example of VR remote controller106by VR user10or the like.

InFIG.4, (b) illustrates operation mode explanation table410indicating a management example of operation modes by conversation controller311. In other words, when “guide's instruction” is “questions not allowed”, “questions not allowed”400ais displayed at bottom right of screen display example400of the VR system (“display state of VR” is display mode of “questions not allowed”) and the VR user becomes muted (“mute function of VR system” becomes “muted” or, in other words, the first state) while, on the other hand, when “guide's instruction” is “questions allowed”, the VR user becomes muted (“mute function of VR system” becomes “muted”) when “questions allowed”401ais displayed at bottom right of screen display example401of the VR system (“display state of VR” is display mode of “questions allowed” or, in other words, the second state) but the VR user becomes unmuted (“mute function of VR system” becomes “unmuted”) when “question in progress”402ais displayed at bottom right of screen display example402of the VR system (“display state of VR” is display mode of “question in progress” or, in other words, the third state).

As illustrated in the present drawing, by realizing three guidance display modes in VR systems100,110,120, and130and transitions of display modes with guide20and VR users10and the like using a simple UI constituted of a single button, the following four consequences are realized.

(1) Basically, since utterances of VR users10to13are to be constantly muted, there is no need for VR users10to13to learn a concept of muting or how to perform a muting operation.

(2) VR users10to13can readily understand when questions can be asked and when questions cannot be asked by a guidance display in VR space (transition of guidance display of “questions allowed” and “questions not allowed”).

(3) When VR users10to13wish to ask a question (wish to speak to guide20), pushing one button (question button) when guidance of “questions allowed” is being displayed will suffice. Therefore, since there is little to learn, there is little to forget. It is important to be aware that keeping operations simple is essential because a field of view is to be blocked when wearing the VR system.

(4) When guide20does not want to take questions (in other words, when guide20does not want to be disturbed when providing an explanation), guide20can set “questions not allowed” with respect to VR users10to13at any time only by an operation of a “questions not allowed” instruction (even when a question is in progress). In addition, when accepting questions, “questions allowed” can be set with respect to VR users10to13only by an operation of a “questions allowed” instruction. Since both operations can be performed by simple operations of a single button such as a toggle button, guide20can focus on providing explanations.

From the features described above, A) VR users10to13can readily enjoy conversation with guide20with one button operation. B) Guide20can readily control VR systems100,110,120, and130with one button operation and smoothly provide explanations without being disturbed by VR users10to13.

FIG.5is a block diagram illustrating a detailed configuration of VR tourism system2according to Example 1. As VR tourism system2, in the configuration example of VR tourism system650according to the reference example illustrated inFIG.1B, conversation controller311and guide instruction recorder315are added to VR audio conference unit310of VR cloud300, conversation controller107and guidance display controller109are added to VR audio conference unit105of VR systems100,110,120, and130, and input unit212and guide instruction generator211are added to VR audio conference unit250of smartphone210that is a guide's terminal.

In other words, input unit212that accepts input by a user and guide instruction generator211that conveys, based on the input, instruction contents of guide20to VR audio conference unit310in VR cloud300are included in VR audio conference unit250in guide's terminal210or the like. In addition, VR cloud300includes conversation controller107which receives an instruction of guide20sent from guide's terminal210or the like, which records contents of the instruction in guide instruction recorder315, which generates a control instruction for controlling VR audio conference unit105of VR systems100,110,120, and130, and which transmits the generated control instruction to VR systems100,110,120, and130.

Furthermore, VR audio conference unit105of VR systems100,110,120, and130includes conversation controller107which is responsible for setting a mute state of muter18caccording to contents of a control instruction sent from VR cloud300and user input and changing a guidance display state such as “questions allowed” by issuing an instruction to guidance display controller109.

Such a configuration realizes functions illustrated inFIG.4as an algorithm of the operation flow example of VR tourism system2according to Example 1 illustrated inFIG.6described below is executed.

FIG.6is a diagram illustrating an operation flow example of VR tourism system2(in other words, a specific example of a video display method) according to Example 1. More specifically, (a) inFIG.6illustrates an operation flow of VR systems100,110,120, and130, (b) inFIG.6illustrates same operation mode explanation table410as (b) inFIG.4for reference, and (c) inFIG.6illustrates an operation flow of VR cloud300.

As illustrated in (c) inFIG.6, VR cloud300is capable of accepting a “guide's instruction” (as transition of operation modes, a transition to “questions allowed” and a transition to “questions not allowed”) from an APP of smartphone210or from guide's remote controller230or the like via input unit212and guide instruction generator211of guide's terminal210or the like (S10).

In addition, conversation controller311determines the accepted “guide's instruction” (S11) and sorts instructions into the following two types of processing. Specifically, when the “guide's instruction” is “questions allowed” (Y in S11), the instruction is set to “questions allowed” (S13), and when the “guide's instruction” is “questions not allowed” (N in S11), the instruction is set to “questions not allowed” (S12).

In addition, conversation controller311notifies all VR systems100,110,120, and130that guide20has issued a new instruction together with instruction contents (S14) and returns to step S10.

On the other hand, while each VR system100,110,120, or130performs main processing of a VR tourism distribution service, when there are the following two types of interrupt processing (S30to S40and S20to S24) as illustrated in (a) inFIG.6, the interrupt processing is performed before returning to the main processing. If there is no interrupt processing, reception processing in the VR tourism service is continued until the VR tourism distribution service ends.

Note that the main processing of VR systems100,110,120, and130refers to processing in which VR systems100,110,120, and130provide VR users10to13with a VR space for VR tourism and is processing involving receiving a 360-degree video sent from 360-degree video distributor320, demultiplexing the received 360-degree video into video data and audio data, and after decoding the demultiplexed audio data, performing audio processing dependent on a position and an azimuth of VR observation system200and outputting to a loudspeaker as audio while, on the other hand, after decoding the demultiplexed video data, performing mapping on a 360-degree spherical video, further performing video segmentation dependent on a position and an azimuth of VR observation system200, and outputting as a video to a display.

A first type of interrupt processing is user input interrupt processing of VR systems100,110,120, and130. Conversation controller107receives a VR user input from VR remote controller106or the like (S30), and when the received VR user input is a “question in progress” request (Y in S31), conversation controller107refers to information in guide instruction recorder315(S33).

As a result, when a value of the guide's instruction is “questions not allowed” or, in other words, the first state (N in S35), the processing is ended, but when the value of the guide's instruction is “questions allowed” (Y in S35) and the display mode is “questions allowed” or, in other words, the second state (N in S36), guidance display controller109changes the display mode to “question in progress” (S37), muter108csets unmute (S38) or, in other words, changes to the third state, and ends processing.

On the other hand, when the value of the guide's instruction is “questions allowed” (Y in S35) and the display mode is “question in progress” or, in other words, the third state (Y in S36), guidance display controller109changes the display mode to “questions allowed” (S39), muter108csets to “mute” or, in other words, the second state (S40), and ends processing.

Note that when the VR user input is other than a “question in progress” request (N in S31), conversation controller107processes the VR user input (S32) and ends processing.

A second type of interrupt processing is interrupt processing from VR cloud300. Conversation controller311records the guide instruction received from VR cloud300in guide instruction recorder315.

In addition, when the guide instruction is “questions allowed” (Y in S21), guidance display controller109is caused to change the display mode to “questions allowed” or, in other words, the second state (S23) and the processing is ended. On the other hand, when the guide instruction is “questions not allowed” (N in S21), guidance display controller109changes the display mode to “questions not allowed” (S22), muter108csets to “mute” (S24) or, in other words, the first state, and ends processing.

FIG.7is a diagram illustrating a specific configuration example of VR systems100,110,120, and130of VR tourism system2according to Example 1. In the present configuration example, VR systems100,110,120, and130are constituted of a computer or a smartphone (computer/smartphone160) and an HMD (head-mounted display) or VR glasses (HMD/VR glasses140) connected thereto. Instead of the present configuration example, VR systems100,110,120, and130may be solely constituted of HMD/VR glasses140. In such a case, functions of a CPU and a GPU of both components are integrated and peripheral functions are also consolidated.

As main components, computer/smartphone160is constituted of: high-speed communication element163that supports WiFi (registered trademark) or Ethernet (registered trademark) for connecting to VR systems100,110,120, and130; GPU (Graphics Processing Unit)165that mainly performs processing of video data and graphics; CPU (Central Processing Unit)170that performs general data processing and control of entire computer/smartphone160; non-volatile memory167that is a hard disk, a flash memory, or the like for storing a program necessary for running CPU170and GPU165; RAM166used to store data necessary for running CPU170and GPU165; power supply control element169for supplying power to power switch168and the respective units; AV output terminal161for outputting video and audio signals to HMD/VR glasses140; an I/F such as USB (Universal Serial Bus) for controlling HMD/VR glasses140and acquiring data therefrom; a memory bus that connects RAM166and non-volatile memory167and enables access by CPU170and GPU165; a system bus that enables CPU170and GPU165to access AV output terminal161, USB162, and communication element163; bus converter164that connects the system bus and the memory bus to each other; a display device that is not illustrated; an input device for performing operations; other general-purpose I/Fs (interfaces), and the like.

As functions exercised by executing the program, CPU170includes: multiplexer170athat multiplexes video data and audio data; demultiplexer170bthat demultiplexes video data and audio data from a received 360-degree video; audio decoder170cthat decodes demultiplexed audio data; and audio playback controller170dthat controls playback of decoded audio data.

As functions exercised by executing the program, GPU165includes: motion/position detector165athat receives motion/position data generated by motion/position detector165aof HMD/VR glasses140and detects a motion and a position of a VR user; VR controller165bthat issues a generation instruction of a VR space based on the detected motion and position of the VR user; VR video decoder165cthat decodes video data demultiplexed by CPU170; guidance display generator165dthat instructs display and generation of guidance such as “questions not allowed”, “questions allowed”, and “question in progress”; graphics generator165ethat generates graphics of a guidance based on the instruction; and VR display controller165fthat composites video data decoded by VR video decoder165cand graphics generated by the graphics generator based on the generation instruction of a VR space from VR controller165band outputs the composite video data/graphics as a VR video to HMD/VR glasses140.

While there are cases that differ from the present example in terms of some types of processing being performed by GPU165or CPU170and also in terms of bus configurations, there is no difference in functional configurations and operations. In addition, AV output terminal161and USB162may be replaced with I/Fs such as USB_Type C (registered trademark) as high-speed two-way I/Fs. In such a case, HMD/VR glasses140are to be connected by a same I/F or connected by a converter that converts the I/F. Generally, when sending a video by USB162, since a data amount is compressed by suitable compression, suitable video compression is performed by CPU170or GPU165and a VR video is sent to HMD/VR glasses140via USB162.

On the other hand, as main components, HMD/VR glasses140in VR systems100,110,120, and130include: an audio input unit constituted of microphone157afor inputting audio, microphone amplifier157b, and ADC (A/D converter)157c; an audio output unit constituted of loudspeaker158c, headphone terminal158d, amplifier158b, and DAC (D/A converter)158a; a VR display unit constituted of two sets of lenses153aand153bfor a user to view VR video and display elements152aand152b; motion/position sensor141constituted of a motion/position detector and an azimuth detector constituted of a gyro sensor, a camera, an ultrasonic microphone, or the like; wireless communication element148such as Bluetooth (registered trademark) for communicating with a controller (not illustrated); volume button142for controlling output volume from the audio output unit; power switch143for turning on/off power of HMD/VR glasses140; power supply control element145for power supply control; a memory bus that connects EEPROM154, RAM155, and an SD card (registered trademark) with GPU151and CPU156and performs exchange of data with the memories; AV input terminal146for receiving video signals and audio signals from CPU156, GPU151, wireless communication element148, and computer/smartphone160; an I/F such as USB147for receiving a control signal from computer/smartphone160and sending video, audio signals, and data of motion/positions; CPU156that mainly performs control of audio compression (audio compressor156a), switches, and power and performs control of entire HMD/VR glasses140; GPU151that includes video display processor151bthat mainly adjusts video to the VR display unit and motion/position detector151athat corrects and shapes motion/position information to be sent to computer/smartphone160from information from motion/position sensor141; EEPROM154for storing programs and data that enable CPU156and GPU151to run; RAM155for storing data while CPU156and GPU151are running; a memory bus for connecting CPU156, GPU151, RAM155, and EEPROM154; a system bus to which CPU156, GPU151, USB147, the audio input unit, the audio output unit, and wireless communication element148are connected and which performs control and exchange of data; an I/O bus that performs control and low-speed exchange of data including the buttons described earlier, power supply control element145, motion/position sensor141and, although not illustrated, the audio input unit, the audio output unit, a VR photography camera, and the like; and several bus converters150that connect the respective buses to each other.

Furthermore, HMD/VR glasses140are configured to be connectible to VR remote controller106via BLE (Bluetooth Low Energy; registered trademark)149and enable an operation to request a “question” when VR users10to13wish to engage in a conversation with guide20.

While there are cases that differ from the present example in terms of some types of processing being performed by GPU151or CPU156and also in terms of bus configurations, there is no difference in functional configurations and operations.

In addition, since video data from AV input terminal146has a large data amount and is high-speed, the video data is illustrated as being directly loaded into GPU151when the system bus lacks sufficient speed.

Furthermore, video information photographed by the camera included in motion/position sensor141may be sent to display elements152aand152bas information used by a VR user to check a periphery of HMD/VR glasses140or sent to computer/smartphone160through USB147to monitor whether a user is not in a dangerous situation.

In addition, power supply control element145receives power supply from USB147or AV input terminal146, performs stabilization of voltage, management of battery capacity, and the like and, although not illustrated, supplies power to all constituent elements. In some cases, battery144may be provided inside or outside and battery144may be connected to power supply control element145.

A state of a button or a cursor of controllers (not illustrated) is acquired by CPU156through wireless communication element148and used to perform a button operation, a movement, or an application operation in VR space. A position and an orientation of the controllers are detected by a camera or an ultrasonic sensor included in motion/position sensor141and, after being subjected to suitable processing by motion/position sensor141, used for control by CPU156and, at the same time, sent to computer/smartphone160via USB147to be used in a program executed by CPU156or used in rendering of graphics and image processing executed by GPU151.

FIG.8is a diagram illustrating a specific configuration example of VR audio conference unit250included in guide's terminal210or the like of VR tourism system2according to Example 1. VR audio conference unit250includes: communicating means with VR cloud300(communicating means via a terminal having a communication function of using carrier communication such as 5G or Wi-Fi (registered trademark) to communicate with VR cloud300such as a PC or smartphone210); audio interface215for connecting to external devices that perform audio input/output such as microphone216and loudspeaker217; A/D converter213aand D/A converter214bwhich convert analog audio into digital data or reverse processing thereof; audio compressor213band audio decoder214afor compressing and decoding digital audio data; muter213cfor temporarily preventing an audio signal from microphone216from being sent; and guide instruction generator211that converts an input into a guide instruction and sends the guide instruction to VR cloud300.

Furthermore, guide's terminal210or the like includes communicator/CODEC for headset211awhich performs power-saving communication such as BLE (registered trademark) with external headset220and guide's remote controller230or the like and input processor211bthat issues an input instruction according to the communication and outputs the input instruction to guide instruction generator211.

While a dedicated terminal may be used as guide's terminal210or the like, since guide's terminal210or the like can be readily implemented with smartphone210and an APP of smartphone210, guide's terminal210or the like may be implemented as an APP on smartphone210(business or private use) in the possession of guide20. However, using guide's remote controller230connected by low-power consumption communication such as BLE (registered trademark) of smartphone210instead of operating smartphone210is more preferable since operations are simple and the fact that an operation is being performed can be hidden from VR users10to13.

Using guide's remote controller230enables operations to be performed by simply blind-pushing a button on guide's remote controller230being hand-held even when an explanation is being provided.

As described above, due to VR tourism system2according to Example 1, the simple user interface illustrated inFIG.4is realized and the problem (1) described earlier is solved.

Variation 1 of Example 1

Next, Variation 1 of Example 1 will be described.

FIG.9is a schematic configuration diagram of VR tourism system2aaccording to Variation 1 of Example 1. VR tourism system2ais a system that is VR tourism system2according to Example 1 being applied to a VR tourism system with a “VR sickness prevention function” and to a VR tourism system with a “lost person prevention function”.

In this case, a VR tourism system with a “VR sickness prevention function” is a VR tourism system with the following functions. Specifically, in order to avoid VR sickness that accompanies a movement of a 360-degree camera used to take photos in VR tourism or the like, a VR tourism system with a “VR sickness prevention function” has a function of avoiding (reducing) VR sickness by sending metadata (a movement of a camera that photographs a 360-degree video, a time of day of start of panning, a movement direction, or the like) in addition to the 360-degree video from a VR observation system to a VR system, generating a visual effect (a display on a screen or the like) or a non-visual effect (a sound, vibration of a controller, or the like) which appropriately notifies a viewer of a movement immediately before a camera movement or during a movement period in the VR system, and displaying an unconventional visual effect video or the like to prompt the viewer to take a precaution or to prevent the viewer from becoming sick.

In addition, a VR tourism system with a “lost person prevention function” is a VR tourism system with the following functions. Specifically, a VR tourism system with a “lost person prevention function” has a function of preventing a user from getting lost by sending, with respect to a VR user, metadata including a trigger or the like that is generated by a tour guide (leader, explainer) to a VR system and presenting a visual guidance of a position of the guide or a visual guidance indicating a position in a 3D-space of an object designated (being explained) by the guide in the form of a video, audio, vibration, or the like.

VR tourism system2aaccording to the present variation includes VR systems100,110,120, and130, VR cloud300a, observation system200a, and guide's terminals (smartphone210, guide's remote controller230).

Guide's terminal210and the like respectively send acquired pieces of metadata additional information219aand219bto VR cloud300aand observation system200a. Observation system200aincludes camera203, remote controller204, capturing unit202, and communicator201mounted to vehicle206for acquiring a 360-degree video and transmits metadata additional information219breceived from guide's terminal210and the like and a 360-degree video to VR cloud300a. VR cloud300aincludes 360-degree video distributor320athat distributes metadata additional information219areceived from guide's terminal210and the like and the 360-degree video received from observation system200ato VR systems100,110,120, and130. In VR systems100,110,120, and130, VR space generator103exercises the “VR sickness prevention function” and the “lost person prevention function” by changing displays according to the metadata received from VR cloud300a.

In essence, VR tourism system2aaccording to the present variation acquires additional information to form a basis of metadata by various methods (a movement of a camera that photographs a 360-degree video, a time of day of start of panning, a movement direction, or the like), generates metadata, and sends the metadata to respective VR systems100,110,120, and130via VR cloud300a. By generating additional information based on the metadata when reproducing a VR space using the 360-degree video, VR systems100,110,120, and130can prevent VR sickness and present appropriate guidance information.

In this case, when assuming smartphone210having functions of a GPS, an IMU (inertial measurement unit), a camera, and the like as the guide's terminal, the following two scenarios are possible: (1) acquiring data effective for metadata generation using position information, acceleration information, and the like of smartphone210, sending the acquired data to VR cloud300aand observation system200a, and having VR cloud300aand observation system200agenerate metadata; and (2) acquiring data effective for metadata generation using position information, acceleration information, and the like of smartphone210, sending the acquired data to VR cloud300a, and having VR cloud300acombine the data with metadata from observation system200ato create metadata.

In addition, by adding a movement start/stop button or the like to guide's remote controller230that is capable of communicating with smartphone210by BLE (registered trademark) and by having guide20press the button before moving or before stopping a movement, accuracy of movement detection or the like can be improved and more appropriate metadata can be generated.

Variation 2 of Example 1

Next, Variation 2 of Example 1 will be described.

FIG.10is a schematic configuration diagram of VR tourism system2baccording to Variation 2 of Example 1. VR tourism system2bcorresponds to a second specific example of solving the problem (1) (a simple user interface that enables diverse participants to use the VR tourism service must be provided) described above.

In VR tourism system2according to Example 1 illustrated inFIG.5, guide's terminal210and the like have two functions, namely, a function of listening to voices of VR users10to13and a function of conveying audio of guide20to VR users10to13via VR cloud300. In VR tourism system2baccording to the present variation, audio of guide20is acquired from microphone205of observation system200. Therefore, in VR tourism system2baccording to the present variation, a portion of transferring audio of guide20to VR systems100,110,120, and130from guide's terminals210and the like and VR audio conference unit310aof VR cloud300ahas been deleted from VR tourism system2according to Example 1.

As described above, even in VR tourism system2baccording to the present variation, only a path of acquiring audio of guide20differs from Example 1 and the problem (1) described above is solved.

Next, Example 2 will be described as a specific example of solving the problem (2) (bad-mannered users who obstruct provision of an explanation by guide20by verbally abusive language, continuous questions, and the like must be eliminated) described above.

(2) Even VR users who make inappropriate remarks and obstruct an explanation by guide20by verbally abusive language, continuous questions, and the like may possibly participate in VR tourism. Tourism experience by other good VR users is inhibited unless utterances by such participants are eliminated. When such a situation occurs, guide20must handle the situation with a simple operation. Specifically, a function of muting an utterance of the corresponding user with an instruction by guide20and enabling “utterances not allowed” to continue even in a state where “utterances are allowed” for other users in order to prevent the corresponding user from making subsequent utterances is required, and a VR tourism system having such a function will be described as a VR tourism system according to Example 2. The VR tourism system according to Example 2 is a system in which a function of solving the problem (2) has been added to VR tourism system2according to Example 1.

FIG.11is a diagram illustrating an operation example of a VR tourism system according to Example 2. More specifically, (a) inFIG.11illustrates a display example in VR systems100,110,120, and130, display examples240cand240dusing an audio control application in smartphone210of guide20, an operation example of guide's remote controller230, and an operation example of VR remote controller106by VR user10or the like. InFIG.11, (b) illustrates same operation mode explanation table410as in (b) inFIG.4. InFIG.11, (c) illustrates user management table330included in a VR audio conference unit according to the present example. User management table330is an example of blocking information70in an embodiment including permission/refusal information that indicates, with respect to each of a plurality of participants, whether or not to block questions to an explainer and, specifically, user management table330is an information table that records a “mode” indicating, for each VR user (“User-ID”), whether or not (“Block” or “OK”) the VR user is a user who is an object of the block.

As illustrated in (a) inFIG.11, VR systems100,110,120, and130having three display modes, UIs of guide's remote controller230and VR remote controller106, and internal operation processing are the same as in Example 1.

In addition, in the VR cloud, while basic functions (switching between “questions allowed”401aand “questions not allowed”400a) of guide instructions are the same as in Example 1, the following functions have been added.

Specifically, as illustrated in (c) inFIG.11, user management table330that records, for each VR user, whether the VR user is a good user or a user who makes an inappropriate remark has been added in the VR cloud. User management table330is an example of blocking information70according to Variation 1 of the embodiment.

In addition, as illustrated in display examples240cand240dand an operation example of guide's remote controller230in (a) inFIG.11, a function has been added which enables a “block” instruction for guide20to block (forced mute) utterances of a user having made an inappropriate remark and an “unblock” instruction that allows utterances of a blocked user.

Furthermore, a function has been added which, when a “block” instruction is issued, changes a value of “Mode” of a corresponding user in user management table330to “Block” and performs forced muting of the corresponding user.

In addition, a function has been added which, when guide20issues an “unblock” instruction, changes values of “Mode” of all users in user management table330to “OK”. Note that when guide20issues an “utterance allowed” instruction, display in the VR system with respect to a user of which the value of “Mode” in user management table330is “Block” remains unchanged as “questions not allowed”400a.

FIG.12is a schematic configuration diagram of VR tourism system3according to Example 2. VR tourism system3is a system in which a function of solving the problem (2) has been added to VR tourism system2according to Example 1 and includes VR systems100,110,120, and130, VR cloud300b, guide's terminal210and the like, and observation system200. VR audio conference unit310of VR cloud300bincludes user management table330in addition to components according to Example 1.

A configuration of VR systems100,110,120, and130of VR users10to13is the same as the configuration example of VR tourism system2according to Example 1 illustrated inFIG.5and operation methods of VR users10to13are also the same. In other words, VR users10to13are prevented from becoming aware of the fact that a user who makes an inappropriate remark is being blocked.

A configuration in VR audio conference unit250of guide's terminal210and the like is also the same as in Example 1. As a change, since “block” and “unblock” have been added to “questions allowed” and “questions not allowed” in VR tourism system2according to Example 1 as input received by input unit212, “guide's instruction” that is sent to VR cloud300bhas also been added.

VR audio conference unit310of VR cloud300bis an example of controller62aaccording to Variation 1 of the embodiment and receives the “guide's instruction” sent from guide's terminal210and the like in VR cloud300b, and when instruction contents thereof are “questions allowed” or “questions not allowed”, VR audio conference unit310records the “guide's instruction” in guide instruction recorder315. When the instruction from guide20is “block” and “unblock”, VR audio conference unit310updates, in accordance with the instruction, user management table330for recording a user who makes an inappropriate remark.

Furthermore, VR audio conference unit310generates a “control instruction” for controlling VR audio conference unit105of VR systems100,110,120, and130, transmits the “control instruction” to all VR systems100,110,120, and130and, as a result, in VR systems100,110,120, and130, guidance display controller109constantly displays “questions not allowed”400awith respect to a blocked user.

FIG.13is a diagram illustrating an operation flow example of VR tourism system3(in other words, a specific example of a video display method) according to Example 2. More specifically, (a) inFIG.13illustrates an operation flow of VR systems100,110,120, and130, (b) inFIG.13illustrates same operation mode explanation table410as (c) inFIG.11for reference, (c) inFIG.13illustrates an operation flow of VR cloud300, and (d) inFIG.13illustrates same user management table330as (c) inFIG.11for reference.

As illustrated in (c) inFIG.13, in VR cloud300b, conversation controller311is capable of accepting a “guide's instruction” (a transition to “questions allowed” and a transition to “questions not allowed”) from an APP of smartphone210or from guide's remote controller230or the like via input unit212and guide instruction generator211of guide's terminal210or the like (S50).

In addition, conversation controller311determines the accepted “guide's instruction” and sorts instructions into the following four types of processing (S51to S54). Specifically, when “guide's instruction” is “questions allowed” (Y in S51), the instruction is set to “questions allowed” (S60), when “guide's instruction” is “questions not allowed” (Y in S52), the instruction is set to “questions not allowed” (S61), when “guide's instruction” is “block” (Y in S53), a value of the user presently engaged in conversation in user management table330is changed to “Block” (S62), and when “guide's instruction” is “unblock” (Y in S54), values of all users in user management table330are changed to “OK” (S63).

Furthermore, conversation controller311notifies all VR systems100,110,120, and130that guide20has issued a new instruction (instruction change notification) together with instruction contents (however, a user of which the value in user management table330is “block” is notified “questions not allowed” as instruction contents) (steps S70to S75) and returns to step S50.

More specifically, in processing of transmitting the instruction change notification to each VR system (S70to S75), conversation controller311first sets variable N to1(S70) and next extracts an N-th value from user management table330and increments variable N (S71). In addition, conversation controller311determines whether or not the value extracted from user management table330is “block” (S72) and, in the case of “block” (Y in S72), sets the instruction to “questions not allowed” (S73) and transmits an instruction change notification to a corresponding VR system (S74). When the value extracted from user management table330is not “block” (N in S72), conversation controller311transmits an instruction change notification to the corresponding VR system (S74) without setting the instruction to “questions not allowed”.

Conversation controller311determines whether or not confirmation has been finished with respect to all VR users (S75) and, if not finished (N in S75), repeats steps S71to S75until confirmation is finished with respect to all VR users.

On the other hand, as illustrated in (a) inFIG.13, operations of respective VR systems100,110,120, and130are the same as operations of respective VR systems100,110,120, and130according to Example 1. Specifically, while main processing of a VR tourism distribution service is performed in each VR system100,110,120, or130, when there is interrupt processing by a guide instruction from VR cloud300as illustrated in (a) inFIG.13, interrupt processing similar to that in Example 1 is performed (S20to S25). In addition, although not illustrated inFIG.13, when there is an interrupt by a VR user input from VR remote controller106and the like, VR systems100,110,120, and130perform interrupt processing similar to that in Example 1 (S30to S40inFIG.6).

As described above, due to VR tourism system3according to Example 2, an utterance by a user who makes an inappropriate remark is eliminated using a simple user interface and the problem (2) described earlier is solved as in the operation example illustrated inFIG.11.

Next, Example 3 will be described as a specific example that solves the problem (3) (private conversations by participants with traveling companions must be supported) described above.

Before describing a specific example that solves the problem (3), first, the problem (3) will be described in detail using drawings.

FIG.14is a diagram in which a description of the problem (3) has been added to VR tourism system650according to the reference example illustrated inFIG.1A. As illustrated inFIG.14, in VR tourism system650according to the reference example, since each of VR users10to13is participating in the VR tourism service using different VR systems600,610,620, and630, VR users10to13can only engage in conversation in a shared VR space during their participation. In other words, since utterances by VR users10to13in the VR space are shared among all participants, VR users10to13cannot engage in private conversations.

In the case of a videoconferencing system for business instead of a VR tourism system, since private conversations are usually not allowed during a conference, such an application is rare. Therefore, systems such as Teams (registered while videoconferencing trademark) and Zoom (registered trademark) provide a “Breakout Room” function of going into another conference room and having a conversation, there is no function for a specific group to engage in a private conversation in a same conference room. In addition, in an interaction/conference system in a VR space constituted by 3DCG such as VRChat, while there is a function for increasing a volume of a voice of a nearby person and reducing a volume of a voice of a faraway person, a private conversation function in a group is not implemented.

During a normal real group tour, a situation where a group member can no longer engage in private conversations with other group members does not occur. Therefore, in a VR tourism service, support is required for conversations in a same group. In such a case, the conversation must be prevented from being heard by guide20or tourists outside of the group.

In consideration thereof, a function capable of supporting a private conversation between friends who travel together is required and a VR tourism system having such a function will be described as a VR tourism system according to Example 3. The VR tourism system according to Example 3 is a system in which a function of solving the problem (3) has been added to VR tourism system3according to Example 2.

FIG.15is a diagram illustrating an operation example of the VR tourism system according to Example 3. More specifically, (a) inFIG.15illustrates a display example in VR systems100,110,120, and130, smartphone210and guide's remote controller230of guide20, and VR remote controller106of VR user10or the like.

InFIG.15, (b) illustrates operation mode explanation table411according to the present example. In operation mode explanation table411, an item of “mute function for group audio conference” has been added to operation mode explanation table411according to Example 1. As indicated in the item of “mute function for group audio conference”, when a “mute function of VR system” is “muted”, the “mute function for group audio conference” becomes “group conversation allowed (unmuted)” and when the “mute function of VR system” is “unmuted”, the “mute function for group audio conference” becomes “muted”.

InFIG.15, (c) illustrates user management table331included in a VR audio conference unit according to the present example. User management table331is an information table created by adding a group ID (“G-ID”) for each VR user to user management table330according to Example 2. User management table331is an example of blocking information71according to Variation 2 of the embodiment.

When participating in a videoconference, although a conversation cannot be held with a part of the participants while preventing the conversation from being heard by other participants or a speaker, separately using a chat application such as Line (registered trademark) or WeChat (registered trademark) after muting oneself while participating in the videoconference to prevent his/her own conversation from becoming audible to the videoconference enables a group call with a part of the participants to be performed. However, participating in a videoconference using a PC and launching a chat application with smartphone210is a hassle and supporting both functions with a single VR tourism system is desired.

In the present example, based on the VR tourism system described in the operation example illustrated inFIG.4, a private conversation in a group is realized by combining a function for setting a separate audio conference in a VR cloud for each group of which a VR user submits an application when signing up (upon participation, during participation) for VR tourism. Simply combining the two systems may make operations complicated and an inappropriate use of a muter or the like may cause a group conversation to leak out to a VR space and may annoy guide20or other VR users.

In consideration thereof, in the present example, as in the operation example illustrated inFIG.15, voices of VR users10to13are not transmitted to the inside of the VR space unless VR users10to13explicitly present a question to guide20. In other words, in states of “questions not allowed” and “questions allowed”, a conversation within a group is not shared with the VR space (the “mute function for group audio conference” in (b) inFIG.15is “group conversation allowed (unmuted)”). In other words, operations by VR users are absolutely unnecessary in order to engage in a conversation within a group. On the other hand, only when guide20sets “question in progress” by an operation using guide's remote controller230in order for VR user10or the like in a group to ask guide20a question, a group audio conference of a corresponding user is muted in the VR tourism system (the “mute function for group audio conference” in (b) inFIG.15is “muted”).

Since using the VR tourism system according to the present example completely prevents a conversation within a group from being heard by other participants, group tourism in which privacy is more protected than in the real world can be realized.

FIG.16is a block diagram illustrating a detailed configuration of VR tourism system4according to Example 3. VR tourism system4is a system in which a function of solving the problem (3) has been added to VR tourism system3according to Example 2 and includes VR systems100,110,120, and130, VR cloud300c, guide's terminal210and the like, and observation system200.

In VR cloud300c, group audio conference units341ato341cthat assist an audio conference for each group and VR tourism service manager340that manages VR tourism and group audio conferences are added to VR cloud300baccording to Example 2 and user management table331to which group IDs have been added is provided in place of user management table330according to Example 2. Group audio conference units341ato341care examples of group audio conversation unit64according to Variation 2 of the embodiment and VR tourism service manager340is an example of controller62baccording to Variation 2 of the embodiment.

In addition, in VR audio conference unit105aof VR systems100,110,120, and130, AV demultiplexer183that demultiplexes a 360-degree video and audio data from a 360-degree video sent from VR cloud300c; audio decoder184that decodes the demultiplexed audio data; audio decoder182athat decodes audio data sent from group audio conference units341ato341cof VR cloud300c; audio mixer182bthat mixes the two pieces of audio data decoded by audio decoder182aand audio decoder184; and D/A converter182cthat converts the mixed audio data into an analog audio signal and outputs the analog audio signal to loudspeaker182dhave been added to the configuration of VR audio conference unit105according to Example 1 and Example 2, and first muter181that performs mute processing for VR tourism and second muter180that performs mute processing among group audio conference units341ato341care provided in place of muter108caccording to Example 1 and Example 3. Note that 360-degree video processor186corresponds to VR space generator103according to Example 1.

In other words, VR tourism system4according to the present example is obtained by adding, inside VR cloud300of the configuration example of VR tourism system2according to Example 1 illustrated inFIG.5, group audio conference units341ato341cfor realizing a conversation for each user group, VR tourism service manager340that manages an entire system, and user management table331for managing which group each user belongs to.

VR audio conference unit310that realizes a conversation between VR users10to13and guide20and group audio conference units341ato341cthat realizes a group conversation in VR cloud300care completely independent of each other and pieces of audio data managed by the two functions in VR cloud300cnever become mixed. Group audio conference units341ato341care initialized in VR cloud300cin a same number as the number of groups participating in same VR tourism when users participate in the VR tourism using VR systems100,110,120, and130.

Group audio conference units341ato341chave a simple configuration of only collecting pieces of audio data used by VR users10and the like belonging to a same group, mixing the pieces of audio data, and returning the audio data to respective VR systems100,110,120, and130. In addition, VR audio conference unit310, 360-degree video distributor320, and guide's terminal210and the like of VR cloud300chave exactly the same configurations and functions as the configuration example of VR tourism system2according to Example 1 illustrated inFIG.5.

In addition to the constituent elements of Example 1 illustrated inFIG.5, a function of sending audio data inputted from microphone108of VR systems100,110,120, and130to group audio conference units341ato341cand second muter180that performs mute control with respect to the audio data to be sent are added to VR systems100,110,120, and130. Furthermore, a function of receiving 360-degree video data sent from VR cloud300c, performing AV demultiplexing, and mixing segmented audio data with audio conversation data within groups sent from group audio conference units341ato341c(AV demultiplexer183, audio decoder184, audio decoder182a, audio mixer182b, and D/A converter182c) have been added.

While the functions described above are added to VR systems100,110,120, and130, since there are no operations by VR users10to13with respect to group audio conference units341ato341c, operations by VR users10to13are similar to Example 1 illustrated inFIG.5. However, when guide20operates guide's remote controller230in order for a VR user to set “question in progress”, the mute function of first muter181is unmuted, the mute function of second muter180is set, and audio data of a corresponding user does not flow to group audio conference units341ato341c. In other words, since audio data of the corresponding user is included in the 360-degree video data, the other VR users in the group can listen to the conversation with guide20.

FIG.17is a diagram illustrating an operation flow example of VR tourism system4(in other words, a specific example of a video display method) according to Example 3. More specifically, (a) inFIG.17illustrates an operation flow of VR systems100,110,120, and130and (b) inFIG.17illustrates an operation flow of VR cloud300c.

Main processing of VR systems100,110,120, and130is the same as in Example 1 and Example 2 (not illustrated).

Interrupt processing of input from a VR user in VR systems100,110,120, and130becomes a flow in which, in steps S30to S40according to Example 1 illustrated inFIG.6, steps S38aand S40aare respectively performed in place of steps S38and S40and steps S110and S111are newly added. In other words, guidance display controller109changes the display mode to “question in progress” (S37), and when first muter181sets unmute (S38a), second muter180sets mute (S110). In addition, guidance display controller109changes the display mode to “questions allowed” (S39), and when first muter181sets to “mute” (S40a), second muter180unmutes (S111).

Furthermore, interrupt processing from VR cloud300cin VR systems100,110,120, and130is the same as in Example 1 and Example 2 (not illustrated).

As illustrated in (a) inFIG.17, in initialization in VR systems100,110,120, and130, first, since a VR tourism service application to which VR systems100,110,120, and130are mounted is launched (S100), a VR user inputs a user ID and a password (S101), transmits the user ID and the password to VR cloud300(S102), and awaits a reply from VR cloud300(S103).

As a result, when the received reply is an error (Y in S104), a return is made to step S101, but when the received reply is normal (N in S104), a transition is made to main processing of VR systems100,110,120, and130(S105).

On the other hand, in VR cloud300c, main processing is the same as in Example 1 and Example 2 (not illustrated).

In addition, in login processing of VR cloud300c, as illustrated in (b) inFIG.17, VR tourism service manager340first acquires a user ID and a password from VR systems100,110,120, and130(S90) and checks whether a user is a legitimate user of the present tourism service (S91).

As a result, when the user is not a legitimate user (N in S91), VR tourism service manager340transmits an error to corresponding VR system100or the like (S92) and returns to step S90. When the corresponding user is a legitimate user (Y in S91), VR tourism service manager340acquires a value of the user from user management table331(S93).

When the acquired corresponding user is a solo traveler (N in S95), a jump is made to step S99. In a case of a group traveler having made a first login (N in S96), VR tourism service manager340starts (initializes) a group audio conference in which the corresponding user participates (S97).

In a case of a group traveler not having made a first login (Y in S96) and when step S97has been completed, VR tourism service manager340connects VR systems100,110,120, and130of the corresponding user to the group audio conference of the corresponding user (S98), transmits the fact that preparation for use of the tourism service has been normally completed to corresponding VR systems100,110,120, and130(S99), and ends processing.

As described above, VR tourism system4according to the present example supports a private conversation between friends who travel together among participants and solves the problem (3) described above.

Variation 1 of Example 3

Next, Variation 1 of Example 3 will be described.

FIG.18is a block diagram illustrating a detailed configuration of VR tourism system4aaccording to Variation 1 of Example 3. VR tourism system4acorresponds to a second specific example that solves the problem (3) (private conversations by participants with traveling companions must be supported) described above.

VR tourism system4aaccording to the present variation has a configuration (in other words, VR audio conference unit105b) in which volume adjuster187is added to VR audio conference unit105of VR systems100,110,120, and130in VR tourism system4according to Example 3 illustrated inFIG.16.

Volume adjuster187is an example of a function of controller62baccording to Variation 2 of the embodiment among functions of a controller according to the embodiment for controlling, when assisting a group audio conversation, a plurality of VR systems that correspond to a plurality of participants who belong to a same group so as to adjust a volume of at least one of audio from an explainer to a plurality of participants belonging to the same group and audio of the group audio conversation.

Specifically, in order to accommodate a case where audio of guide20and audio of a conversation in a group are mixed and are indiscernible, VR tourism system4aaccording to the present variation has a configuration in which volume adjuster187that adjusts a volume with respect to audio from group audio conference units341ato341chas been added to VR tourism system4according to Example 3 illustrated inFIG.16.

VR users10to13have VR remote controller with volume adjustment function106a, and volume of audio from group audio conference units341ato341ccan be adjusted by performing an operation of changing a volume of VR remote controller with volume adjustment function106aor the like.

Note that a configuration of VR systems100,110,120, and130and an operation flow of processing in VR systems100,110,120, and130and VR cloud300care the same as in Example 1.

As described above, even with VR tourism system4aaccording to the present variation, volume adjuster187supports a private conversation between friends who travel together among participants and the problem (3) described above is solved.

Next, Example 4 will be described as a specific example that solves the problem (4) (a plurality of languages and a plurality of guides must be accommodated with respect to participants in VR tourism in order to accommodate foreign travel and foreigners) described above.

Before describing a specific example that solves the problem (4), first, the problem (4) will be described in detail using drawings.

FIG.19is a diagram in which a description of the problem (4) has been added to VR tourism system650according to the reference example illustrated inFIG.1A. In the VR tourism system, providing a VR tourism experience with high resolution and a strong realistic sensation requires photography with a resolution of at least 8K video and preferably 11K video. However, photography of 8K video and 11K video involves large equipment, a high video transmission rate, and a large capacity. Therefore, both photography and distribution end up being costly. Therefore, a VR tourism service does not make sense as a business unless the VR tourism service is used by many VR users.

In order to increase the number of participants in one VR sightseeing tour, even when VR users10to13who use different languages participate, the number of object persons can be increased if explanations provided by guide20can be received in their own languages. However, as illustrated inFIG.19, since the inside of a shared VR space is used when participating in VR tourism in VR tourism system650according to the reference example, conversations with guide20or other users can only be held in one language. In other words, since a language of use of VR users in the VR space is to be shared and used among all participants, the language becomes a language of use of guide20and a plurality of languages cannot be supported at the same time.

While a multilingual service is desirably provided in order to increase the number of users by preparing a plurality of guides20for each language or even using simultaneous interpreters from the perspective of a provider of a VR tourism service, since VR tourism service650according to the reference example illustrated inFIG.19is incapable of supporting guides20of a plurality of languages, a separate tour must be organized for each language.

In consideration thereof, in order to accommodate foreign travel and foreigners, a plurality of languages and a plurality of guides must be accommodated with respect to participants in VR tourism and a VR tourism system having such a function will be described as a VR tourism system according to Example 4. The VR tourism system according to Example 4 is a system in which a function of solving the problem (4) has been added to VR tourism system2according to Example 1.

FIG.20is a diagram illustrating a use case that is desirably realized by the VR tourism system according to Example 4 or, in other words, a diagram illustrating required specifications of a VR tourism service. More specifically, (a) inFIG.20illustrates an example of a case in which Japanese users are provided with VR tourism at a foreign tourist destination. Due to a Japanese VR tourism service provider dispatching its own guide20or photography crew to Egypt420aand providing a VR tourism service, if the Japanese VR tourism service provider can set up an appliance with a business operator who performs an on-site English VR tourism service for English-speaking countries420cand subsequently change a language of use of guide20to Japanese, types of VR tourism services for Japanese (Japan420b) can be increased in an inexpensive and efficient manner.

InFIG.20, (b) illustrates an example of providing foreign users with VR tourism of tourist destinations in Japan421a. In a case of English, Chinese, and Spanish which have a large number of speakers, each language independently enables VR tourism users in various countries to be reached. However, in a case of Vietnamese (Vietnam421b) or Thai (Thailand421c), even when independently hosting VR tourism is difficult although a certain number of users can be expected, VR tourism can be hosted if a plurality of languages can be supported.

FIG.21is a schematic configuration diagram of VR tourism system5according to Example 4. VR tourism system5is a system in which a function of solving the problem (4) has been added to VR tourism system2according to Example 1 and includes VR systems100,110,120, and130, VR cloud300d, first guide's terminal210aand the like, second guide's terminal210band the like, and observation system200.

First guide's terminal210aand the like are examples of first explainer terminal40ain Variation 3 of the embodiment which performs input and output of audio with Japanese guide20abeing a first explainer who provides explanations in a first language and first guide's terminal210aand the like are constituted of smartphone210awhich accommodates the first language, headset220a, and guide's remote controller230a.

Second guide's terminal210band the like are examples of second explainer terminal40bin Variation 3 of the embodiment which performs input and output of audio with English guide20bbeing a second explainer who provides explanations in a second language and second guide's terminal210band the like are constituted of smartphone210bwhich accommodates the second language, headset220b, and guide's remote controller230b.

VR cloud300dincludes first VR audio conference unit310b, second VR audio conference unit310c, user management table330a, guide language management table332, and 360-degree video distributor320c.

First VR audio conference unit310bis an example of the first audio conversation unit according to Variation 3 of the embodiment which assists an audio conversation between participants and the first explainer via the plurality of VR systems and the first explainer terminal. Second VR audio conference unit310cis an example of the second audio conversation unit according to Variation 3 of the embodiment which assists an audio conversation between participants and the second explainer via the plurality of VR systems and the second explainer terminal.

As illustrated inFIG.21, user management table330ais an information table that records, for each VR user (“User-ID”), a language (“Language”) used by the VR user and a guide ID (“Guide-ID”) of a corresponding guide. Guide language management table332is an information table that records, for each guide ID (“Guide-ID”), a language (“Language”) used by the guide.

More specifically, a configuration example of VR tourism system according to Example 4 illustrated inFIG.21is approximately the same as the configuration example of VR tourism system2according to Example 1 illustrated inFIG.3. Main differences are that A) a VR audio conference unit (first VR audio conference unit310band second VR audio conference unit310c) is provided for each of Japanese guide20aand English guide20bin VR cloud300dand all audio data is sent using the VR audio conference unit (first VR audio conference unit310band second VR audio conference unit310c) instead of sending the audio data to 360-degree video distributor320, and B) VR cloud300dhas user management table330aand has means for connecting each VR user to suitable first VR audio conference unit310bor second VR audio conference unit310cso that when the VR user starts using the VR tourism service, the VR user can receive service by Japanese guide20aor English guide20bmatching the VR user's own language.

In the present example, while a 360-degree video that captures Japanese guide20aand English guide20bis shared, audio control is independent and each of Japanese guide20aand English guide20bperforms operation of VR system100or the like of his/her own VR user10and the like. From VR users10to13, each VR space appears to be independent and, for example, a Japanese user remains completely unaware of the presence of English guide20bor English users.

FIG.22is a diagram illustrating an example of a use case that is realized by VR tourism system5according to Example 4. More specifically, (a) inFIG.22illustrates a case where VR tourism by an English guide of a foreign (Egypt422a) tourist destination is provided to not only VR users in English-speaking countries422cbut also to VR users in Japan422b, and (b) inFIG.22illustrates a case where VR tourism by a Vietnamese guide and a Thai guide at a tourist destination in Japan423ais provided to foreign (Vietnam423band Thailand423c) VR users.

As illustrated in (a) and (b) inFIG.22, in each case, if the two guides (Japanese guide20aand English guide20b, Vietnamese guide20a1and Thai guide20b1) can provide explanations in their own languages, the use cases to be realized described in (a) and (b) inFIG.20become realizable.

FIG.23is a diagram illustrating an example of a field of view provided by VR systems100,110,120, and130in VR tourism system5according to Example 4. Here, an example of a field of view of a user of each guide (field of view424bof a Vietnamese user ((b) inFIG.23), field of view424cof a Thai user ((c) inFIG.23)) in a case (VR tourism in Japan424a) where there are two guides (Vietnamese guide20a1and Thai guide20b1) in a 360-degree video is illustrated.

In the case of a 360-degree video, since there is no frame that serves as a boundary of the video, a photographer can perform photography without worrying about positions of the guides (Vietnamese guide20a1and Thai guide20b1). In addition, VR users10to13can view any range within 360 degrees. Therefore, as illustrated in (a) inFIG.23, when a frame is set as in ordinary camera photography, both guides are positioned off-center and a sense of discomfort is created, but since VR users of each language are only conscious of the guide of his/her own language (Vietnamese guide20a1or Thai guide20b1), as illustrated in (b) and (c) inFIG.23, Vietnamese guide20a1and Thai guide20b1can be respectively placed at a center of a field of view and a sense of discomfort is not created. Therefore, by having the two guides provide explanations in their own languages and by sharing video, VR users10to13can engage in conversation with guide20or obtain an explanation from guide20who matches his/her own language and a value of VR tourism experience is enhanced.

FIG.24is a block diagram illustrating a detailed configuration of VR tourism system5according to Example 4. VR audio conference unit105aof VR systems100,110,120, and130is the same as that in Example 3. First VR audio conference unit310band second VR audio conference unit310cboth have the same configuration as VR audio conference unit310according to Example 1. VR audio conference units250included in first guide's terminal201aand the like used by Japanese guide20aand second guide's terminal210band the like used by English guide20bboth have the same configuration as VR audio conference unit250according to Example 1.

In other words, while VR tourism system5according to fourth example has approximately the same configuration as the configuration of VR tourism system2illustrated inFIG.5or VR tourism system3illustrated inFIG.12, a difference in VR cloud300dis that two independent VR tourism services are realized by two VR audio conference units (first VR audio conference unit310band second VR audio conference unit310c) which share 360-degree video distributor320.

Both guide's terminals (first guide's terminal210aand the like and second guide's terminal210band the like) are similar to guide's terminal210and the like according to Example 1 illustrated inFIG.5in terms of a configuration, functions, and operations by a guide.

Both VR audio conference units (first VR audio conference unit310band second VR audio conference unit310c) of VR cloud300dare similar to VR audio conference unit310according to Example 1 illustrated inFIG.5in terms of a configuration and functions.

Besides the addition of means for compositing audio from VR audio conference unit (first VR audio conference unit310band second VR audio conference unit310c) of VR cloud300dwith audio of a 360-degree video, VR audio conference unit105aof VR systems100,110,120, and130is similar to VR audio conference unit105according to Example 1 illustrated inFIG.5in terms of functions and operations by VR users10to13.

A difference from Example 1 is that VR tourism service manager340and user management table331are newly provided in VR cloud300d, and when VR users10to13start use of the VR tourism service, VR tourism service manager340connects each VR user to an appropriate VR audio conference unit (first VR audio conference unit310band second VR audio conference unit310c) so that the VR user can receive service of a guide matching his/her own language. VR tourism service manager340is an example of controller62caccording to Variation 3 of the embodiment and user management table331is an example of language information72according to Variation 3 of the embodiment.

FIG.25is a diagram illustrating an operation flow example of VR tourism system5(in other words, a specific example of a video display method) according to Example 4. More specifically, (a) inFIG.25illustrates an operation flow of VR systems100,110,120, and130and (b) inFIG.25illustrates an operation flow of VR cloud300d.

In each of VR systems100,110,120, and130, main processing, interrupt processing of input from VR users, and interrupt processing from VR cloud300dare the same as in Example 1 and Example 2 (not illustrated).

In addition, in VR systems100,110,120, and130, initialization processing ((a) inFIG.25) by which a VR user starts using VR tourism is the same as the operation flow (S100to S105) according to Example 3 illustrated in (b) inFIG.17and a description thereof will be omitted.

Furthermore, in VR cloud300d, main processing is the same as in Example 1 and Example 2 (not illustrated).

As login processing in VR cloud300d, as illustrated in (b) inFIG.25, VR tourism service manager340first acquires a user ID and a password from VR systems100and the like (S140) and checks whether a user is a legitimate user of the present tourism service (S141).

When the user is not a legitimate user (N in S141), an error is transmitted to corresponding VR system100or the like (S142) and a return is made to step S140. When the user is a legitimate user (Y in S141), VR tourism service manager340acquires a value of the corresponding user from user management table330(S143).

When language selection by the corresponding user is the same as the language of the first guide (for example, the language (Japanese) of Japanese guide20a) (Y in S145), VR tourism service manager340connects VR system100or the like of the corresponding user to first VR audio conference unit310b(S145a) and jumps to step S149.

On the other hand, when language selection by the corresponding user is the same as the language of the second guide (for example, the language (English) of English guide20b) (Y in S146), VR tourism service manager340connects VR system110or the like of the corresponding user to second VR audio conference unit310c(S147), transmits the fact that preparation for use of the tourism service has been normally completed to corresponding VR system110or the like (S149), and ends processing.

When language selection by the corresponding user is neither the language of the first guide (N in S145) nor the language of the second guide (N in S146), VR tourism service manager340executes default language selection processing (S148) and returns to step S145.

In default language selection processing (S148), for example, VR tourism service manager340: A) causes the corresponding user to select the language of the first guide (for example, the language (Japanese) of Japanese guide20a) or the language of the second guide (for example, the language (English) of English guide20b); B) selects English when the language of the first guide or the second guide is English; or C) selects the language of the first guide (for example, the language (Japanese) of Japanese guide20a).

As described above, VR tourism system5according to the present example realizes accommodation by a plurality of languages and accommodation by a plurality of guides with respect to participants and solves the problem (4) described above.

Variation 1 of Example 4

Next, Variation 1 of Example 4 will be described.

FIG.26is a schematic configuration diagram of VR tourism system5aaccording to Variation 1 of Example 4. VR tourism system5acorresponds to a second specific example that solves the problem (4) (a plurality of languages and a plurality of guides must be accommodated with respect to participants in VR tourism in order to accommodate foreign travel and foreigners) described above.

VR tourism system5ais a system accommodating VR tourism including a Japanese-English simultaneous interpreter who accompanies a guide and has a configuration created by including third guide's terminal210cfor Japanese-English simultaneous interpreter20cin place of second guide's terminal210band newly adding audio switcher350to VR cloud300ein the configuration of VR tourism system5according to Example 4 illustrated inFIG.21. Third guide's terminal210cis constituted of smartphone210cfor Japanese-English simultaneous interpreter20cand headset220c. Audio switcher350is an example of audio switcher65according to the embodiment that switches between outputting audio from the second explainer to a participant associated with the second language and outputting audio to the first explainer and a participant associated with the first language.

In VR tourism system5according to Example 4 illustrated inFIG.21, a Japanese VR space and an English VR space exist completely independently of one another and each guide20supports VR users10to13whom guide20is in charge of. The present variation is configured such that, although the two language spaces are independent of each other, Japanese guide20aissues instructions to all VR users10to13while Japanese-English simultaneous interpreter20conly provides a simultaneous interpreting service and is unable to issue instructions to VR users10to13.

Being aware that VR users include a Japanese user group and an English user group, Japanese guide20arequires a distinction between the two user groups in the display modes of “questions not allowed” and “questions allowed” of VR tourism system650according to the reference example and also requires the three instructions of “questions not allowed”, “questions by Japanese user group allowed”, and “questions by English user group allowed”.

Both Japanese users and English users are only conscious of Japanese guide20aand the presence of Japanese-English simultaneous interpreter20cis not explicitly visible. However, audio control is independent for each language group and although the Japanese users and the English users are aware of each others' presence, the Japanese users and the English users need not be conscious of the fact that different languages are being spoken.

FIG.27is a diagram illustrating an example of a use case that is realized by VR tourism system5aaccording to Variation 1 of Example 4. More specifically, (a) inFIG.27illustrates a case where VR tourism by an English guide at a foreign (Egypt425a) tourist destination is provided to not only VR users in English-speaking countries425cbut also to VR users in Japan425bvia a Japanese-English simultaneous interpreter, (b) inFIG.27illustrates a case where VR tourism by a Thai guide at a tourist destination in Japan426ais provided to foreign VR users (not only VR users in Thailand426cbut also VR users in Vietnam426bvia a Thai-Vietnamese simultaneous interpreter), and (c) inFIG.27illustrates a case where VR tourism by an English guide at a foreign (Egypt427a) tourist destination is provided to not only VR users in English-speaking countries427cbut also to VR users in Japan427bvia a Japanese-English simultaneous interpreter and to VR users in Vietnam427dvia an English-Vietnamese simultaneous interpreter.

If a guide and a simultaneous interpreter can cooperate with each other to provide explanations in their respective languages as in the example of the use case illustrated inFIG.27, the use case to be realized described with reference toFIG.20becomes realizable.

FIG.28is a diagram describing functions of audio switcher350included in VR tourism system5aaccording to Variation 1 of Example 4. More specifically, (a) inFIG.28illustrates state transition diagram415indicating a transition of a state of each VR user managed by VR tourism service manager340, (b) inFIG.28illustrates operation mode explanation table411describing each state illustrated in (a) inFIG.28, (c) inFIG.28illustrates a configuration example of audio switcher350, (d) inFIG.28illustrates operating states of audio switcher350in state “questions not allowed by anyone”415a, state “guide-user questions allowed”415b, and state “interpreter-user questions allowed”415d, (e) inFIG.28illustrates an operating state of audio switcher350in state “guide-user question in progress”415c, and (f) inFIG.28illustrates an operating state of audio switcher350in state “interpreter-user question in progress”415e.

In state transition diagram415illustrated in (a) inFIG.28, a solid arrow indicates a transition when Japanese guide20aissues an instruction of “questions not allowed”, a dashed arrow indicates a transition when Japanese guide20aissues an instruction of “questions allowed” to Japanese guide20a, and dotted arrow indicates a transition when Japanese guide20aissues an instruction of “questions allowed” to Japanese-English simultaneous interpreter20c.

In operation mode explanation table411illustrated in (b) inFIG.28, “state” indicates states415ato415eillustrated in (a) inFIG.28, “guide's instruction” indicates an instruction by Japanese guide20a, “VR display state (Guide)” indicates a display mode in a VR system of a VR user having selected Japanese guide20a, “VR display state (SI)” indicates a display mode in a VR system of a VR user having selected Japanese-English simultaneous interpreter20c, “VR mute function (G)” indicates a mute state of a VR user having selected Japanese guide20a, “VR mute function (SI)” indicates a mute state of a VR user having selected Japanese-English simultaneous interpreter20c, and “connection destination of microphone of SI” indicates a VR user to be a connection destination of a microphone of Japanese-English simultaneous interpreter20c. Note that “GU” stands for Japanese guide20aand “SI” stands for Japanese-English simultaneous interpreter20c.

As indicated in operation mode explanation table411, state “questions not allowed by anyone”415acorresponds to when the “guide's instruction” is “questions not allowed”, in which case “VR display state (Guide)” is “questions not allowed”, “VR display state (SI)” is “questions not allowed”, “VR mute function (G)” is “muted”, “VR mute function (SI)” is “muted”, and “connection destination of microphone of SI” is toward a VR user having selected Japanese-English simultaneous interpreter20c((d) inFIG.28).

State “guide-user questions allowed”415bcorresponds to when the “guide's instruction” is “questions allowed” with respect to Japanese guide20aand “VR display state (Guide)” is “questions allowed”, in which case “VR display state (SI)” is “questions not allowed”, “VR mute function (G)” is “muted”, “VR mute function (SI)” is “muted”, and “connection destination of microphone of SI” is toward a VR user having selected Japanese-English simultaneous interpreter20c((d) inFIG.28).

State “guide-user question in progress”415ccorresponds to when the “guide's instruction” is “questions allowed” with respect to Japanese guide20aand “VR display state (Guide)” is “question in progress”, in which case “VR display state (SI)” is “questions not allowed”, “VR mute function (G)” is “unmuted”, “VR mute function (SI)” is “muted”, and “connection destination of microphone of SI” is toward a VR user having selected Japanese-English simultaneous interpreter20c((e) inFIG.28).

State “interpreter-user questions allowed”415dcorresponds to when the “guide's instruction” is “questions allowed” with respect to Japanese-English simultaneous interpreter20c, “VR display state (Guide)” is “questions not allowed”, and “VR display state (SI)” is “questions allowed”, in which case “VR mute function (G)” is “muted”, “VR mute function (SI)” is “muted”, and “connection destination of microphone of SI” is toward a VR user having selected Japanese-English simultaneous interpreter20c((d) inFIG.28).

State “interpreter-user question in progress”415ecorresponds to when the “guide's instruction” is “questions allowed” with respect to Japanese-English simultaneous interpreter20c, “VR display state (Guide)” is “questions not allowed”, and “VR display state (SI)” is “question in progress, in which case “VR mute function (G)” is “muted”, “VR mute function (SI)” is “unmuted”, and “connection destination of microphone of SI” is toward a VR user having selected Japanese guide20a((f) inFIG.28).

FIG.29is a block diagram illustrating a detailed configuration of VR tourism system5aaccording to Variation 1 of Example 4. VR tourism system5ahas a configuration created by including third guide's terminal210cfor Japanese-English simultaneous interpreter20cin place of second guide's terminal210band newly adding audio switcher350to VR cloud300ein the configuration of VR tourism system5according to Example 4 illustrated inFIG.24.

In other words, VR tourism system5aaccording to the present variation has a similar configuration to Example 1 illustrated inFIG.24. As illustrated inFIG.28, a main difference is that the number of types of guide instructions increases from two to three, Japanese-English simultaneous interpreter20cis unable to issue instructions, there are five states due to combination of three guide instructions and operations of VR users10to13and, in accordance with the states, an operation of audio switcher350of VR cloud300eis changed by VR tourism service manager340.

FIG.30is a diagram illustrating an operation flow example of VR tourism system5a(in other words, a specific example of a video display method) according to Variation 1 of Example 4. More specifically, (a) inFIG.30illustrates an operation flow of VR systems100,110,120, and130and (b) inFIG.30illustrates an operation flow of VR cloud300e.

In each of VR systems100,110,120, and130, main processing, interrupt processing of input from VR users, and interrupt processing from VR cloud300eare the same as in Example 4 (not illustrated).

In addition, in VR systems100,110,120, and130, initialization processing ((a) inFIG.30) by which a VR user starts using VR tourism is the same as the operation flow (S100to S105) according to Example 3 illustrated in (b) inFIG.17and a description thereof will be omitted.

In VR cloud300e, as main processing, as illustrated in (b) inFIG.30, VR tourism service manager340accepts a “guide's instruction” (a transition to “questions not allowed” and a transition to “questions allowed GU”, “questions allowed IU”) from an APP of smartphone210or from guide's remote controller230or the like (S180).

In addition, VR tourism service manager340determines the accepted “guide's instruction” and sorts instructions into the following three types of processing.

Specifically, when the “guide's instruction” is “questions allowed GU” (Y in S181), VR tourism service manager340sets the “guide's instruction” to “questions allowed GU” (S185), notifies all VR systems100and the like of the GU (S186), notifies “questions not allowed” to all VR systems110and the like of the IU (S187), and returns to step S180.

In addition, when the “guide's instruction” is “questions allowed IU” (Y in S182), VR tourism service manager340sets the “guide's instruction” to “questions allowed IU” (S190), notifies all VR systems110and the like of the IU (S191), notifies “questions not allowed” to all VR systems100and the like of the GU (S192), and returns to step S180.

When the “guide's instruction” is neither “questions allowed GU” (N in S181) nor “questions allowed IU” (N in S182), VR tourism service manager340sets the “guide's instruction” to “questions not allowed” (S183), notifies all VR systems100and the like (S184), and returns to step S180.

In addition, as login processing of VR cloud300e, as illustrated in (b) inFIG.30, VR tourism service manager340acquires a user ID and a password from VR systems100,110,120, and130and checks whether a user is a legitimate user of the present tourism service (S160).

When the user is not a legitimate user (N in S161), VR tourism service manager340transmits an error to corresponding VR system100or the like (S162) and returns to step S160. When the user is a legitimate user (Y in S161), VR tourism service manager340acquires a value of the corresponding user from user management table330a(S163).

When the acquired language selection of the user is the same as the language of the first guide (the language (Japanese) of Japanese guide20a) (Y in S165), VR tourism service manager340connects VR system100or the like of the corresponding user to first VR audio conference unit310b(S166) and jumps to step S170.

On the other hand, when the acquired language selection of the user is the same as the language of Japanese-English simultaneous interpreter20c(Y in S167), VR tourism service manager340connects VR system100or the like of the corresponding user to second VR audio conference unit310c(S168) and jumps to step S170.

When the acquired language selection of the user is neither the language of the first guide (language (Japanese) of Japanese guide20a) (N in S165) nor the language of Japanese-English simultaneous interpreter20c(N in S167), VR tourism service manager340executes default language selection processing (S169) and returns to step S165.

After connecting VR system100or the like of the corresponding user to first VR audio conference unit310bor second VR audio conference unit310c(S166, S168), VR tourism service manager340transmits the fact that preparation for use of the tourism service has been normally completed to corresponding VR system100or the like (S170) and ends processing.

In default language selection processing (S169), for example, VR tourism service manager340: A) causes the corresponding user to select the language of the first guide (for example, the language (Japanese) of Japanese guide20a) or the language of the simultaneous interpreter (for example, the language (English) of Japanese-English simultaneous interpreter20c); B) selects English when the language of the first guide or the second guide is English; or C) selects the language of the first guide (for example, the language (Japanese) of Japanese guide20a).

As described above, even with VR tourism system5aaccording to the present variation, explanations and conversations accommodating a plurality of languages by a simultaneous interpreter are supported and the problem (4) described above is solved.

Variation 2 of Example 4

Next, Variation 2 of Example 4 will be described.

FIG.31is a schematic configuration diagram of VR tourism system5baccording to Variation 2 of Example 4. VR tourism system5bcorresponds to a third specific example that solves the problem (4) (a plurality of languages and a plurality of guides must be accommodated with respect to participants in VR tourism in order to accommodate foreign travel and foreigners) described above.

VR tourism system5bis a system that accommodates VR tourism with a function of machine translation and has a configuration in which, in the configuration of VR tourism system2according to Example 1 illustrated inFIG.3, machine translator360, machine translation management table333, and user management table330bare added to VR audio conference unit310dof VR cloud300f.

Machine translator360is an example of machine translators66ato66baccording to the embodiment which perform machine translation between the first language and the second language. Machine translation management table333is an example of machine translation management information73according to the embodiment and is an information table that records, for an ID (translation processing module ID “MT-ID”) of each of a plurality of translation processing modules included in machine translator360, a correspondence between the first language being a translation source (“Language-S”) and the second language being a translation destination (“Language-D”). User management table330bis an example of language information72according to the embodiment and is an information table that records a language used by a guide (“Guide Language”), a default language (“Default Language”), a language of use (“Language”) for each VR user (“User-ID”), a translation processing module ID for the guide (“MT-ID-S”), and a translation processing module ID for the VR user (“MT-ID-D”).

As illustrated inFIG.31, while VR tourism system5baccording to the present variation has approximately the same configuration as the schematic configuration diagram of VR tourism system2according to Example 1 illustrated inFIG.3, VR audio conference unit310dof VR cloud300fincludes machine translator360, machine translation management table333for selecting machine translation means to be used for each VR user10to13, and user management table330b.

In the present variation, since the guide is Japanese guide20a, VR users000001to000003are Japanese users and therefore do not use machine translation as indicated in user management table330b. VR user000004and VR user000005are English users and therefore use translation processing modules (of which MT-IDs are01and02) for Japanese-English translation. Although VR user000006is a French user, since machine translator360does not have a French-Japanese translation function, English that is the default language is selected and the same setting as VR user000004and VR user000005is applied. Similarly, with respect to VR user000008, since machine translator360does not support Thai, same Japanese-English translation as VR user000006is selected. VR user000007is a Chinese user and therefore translation processing modules (of which MT-IDs are05and06) for Japanese-Chinese translation are selected.

Using VR tourism system5bdescribed above, although there is a possibility that a problem of accuracy of machine translation may exist, both Japanese users and non-Japanese users can experience guided VR tourism in his/her own language or in a default language.

FIG.32is a diagram illustrating an example of a use case that is realized by VR tourism system5baccording to Variation 2 of Example 4. More specifically, (a) inFIG.32illustrates a case where VR tourism by an English guide at a foreign (Egypt428a) tourist destination is provided to not only VR users in English-speaking countries428cbut also to VR users in Japan428bvia machine translator360, (b) inFIG.32illustrates a case where VR tourism by a Thai guide at a tourist destination in Japan429ais provided to foreign VR users (not only VR users in Thailand429cbut also VR users in Vietnam429bvia machine translator360), and (c) inFIG.32illustrates a case where VR tourism by an English guide at a foreign (Egypt430a) tourist destination is provided to not only VR users in English-speaking countries430dbut also to VR users in Japan430bvia machine translator360, to VR users in Vietnam430cvia machine translator360, and to VR users in French-speaking countries430evia machine translator360.

As illustrated inFIG.32, with VR tourism system5baccording to the present variation, if Japanese guide20aand Japanese-English simultaneous interpreter20ccan cooperate with each other to provide explanations in their respective languages, the use case to be realized described with reference toFIG.20becomes realizable.

FIG.33is a diagram for describing detailed functions of machine translator360included in VR tourism system5baccording to Variation 2 of Example 4. More specifically, (a) inFIG.33is a block diagram illustrating a detailed configuration of VR audio conference unit310d, (b) inFIG.33illustrates a configuration (machine translators360ato360d) of translation processing module group370included in machine translator360illustrated in (a) in FIG.33, (c) inFIG.33illustrates machine translation management table333illustrated in (a) inFIG.33, and (d) inFIG.33illustrates user management table333billustrated in (a) inFIG.33.

As illustrated in (a) inFIG.33, VR audio conference unit310dincludes user management table330b, machine translation management table333, a guide instruction recorder, machine translator360, and audio mixer313. Machine translator360is a group of machine translators360ato360dthat performs machine translation from a first language to a second language for each VR user10to13.

Each of machine translators360ato360dincludes: audio decoder361athat decodes audio data sent from corresponding VR system100or the like; translation processing module group S (361b) that translates the decoded audio data into a language corresponding to a guide (in the present variation, Japanese guide20a); audio compressor361cthat compresses the obtained audio data and outputs the compressed audio data to audio mixer313; audio decoder362athat decodes audio data sent from guide's terminal210aand the like; translation processing module group D (362b) that translates the decoded audio data into a language corresponding to a VR user; and audio compressor362cthat compresses the translated audio data and sends the compressed audio data to corresponding VR system100or the like. Audio mixer313mixes pieces of audio data in translated languages outputted from machine translators360ato360dand sends the mixed audio data to guide's terminal210aand the like.

As illustrated in (b) inFIG.33, translation processing module group370is a group of translation processing modules to be used as translation processing module group S (361b) and translation processing module group D (362b) illustrated in (a) inFIG.33and, as illustrated, translation processing module group370is constituted of modules that perform translation from various first languages to second languages such as English to Japanese (“English to Japanese”) and Japanese to French (“Japanese to French”).

In this example, translation processing module group370has eight translation functions which are digitized in machine translation management table333in (c) inFIG.33. As illustrated in (a) inFIG.33, since VR systems100,110,120, and130of respective VR users10to13are connected to machine translators360ato360d, a translation processing module used for each VR user is selected based on information in user management table330b, and a result thereof is transmitted to a guide via guide's terminal210aand the like, the guide can listen to questions by each VR user in his/her own language.

An explanation provided by the guide is sent to machine translators360ato360dfor each VR user via guide's terminal210aand the like, translated by the translation processing module selected by user management table330b, and subsequently sent to each VR user, thereby enabling each VR user to listen to tourism guidance in his/her own language regardless of the language of the guide.

In the example of user management table330billustrated in (d) inFIG.33, since the language used by the guide is English and the default language is set to English, translation processing modules01and02that perform Japanese-English translation are selected for users000001to000003, translation is not required for users000004and000005being English users, translation processing modules07and08that perform English-French translation are selected for user000006being a French user, translation processing modules03and04that perform Chinese-English translation are selected for user000007being a Chinese user, and while user000008is a Thai user, since a translation processing module for Thai is unavailable, the default language of English is selected and machine translation is not set.

FIG.34is a block diagram illustrating a detailed configuration of VR tourism system5baccording to Variation 2 of Example 4. VR tourism system5bhas a configuration comparable to, in the configuration of VR tourism system2according to Example 1 illustrated inFIG.5, VR audio conference unit105aaccording to Example 4 that performs two-way audio processing being included in place of VR audio conference unit105included in VR systems100,110,120, and130and VR audio conference unit310dwith a function of performing machine translation being included in place of VR audio conference unit310of VR cloud300.

In other words, VR tourism system5baccording to the present variation includes VR cloud300fincluding VR audio conference unit310dwith a function of performing machine translation. VR audio conference unit310dincludes: conversation controller311; audio decoder361athat decodes audio data sent from VR system100or the like of each VR user; translation processing module group S (361b) that translates the decoded audio data into a language corresponding to a guide (in the present variation, Japanese guide20a); audio mixer313that mixes a plurality of pieces of translated audio data corresponding to the respective VR users; audio compressor361cthat compresses the mixed audio data and transmits the compressed audio data to guide's terminal210a; audio decoder362athat decodes audio data sent from guide's terminal210a; translation processing module group D (362b) that translates the decoded audio data into a language corresponding to each VR user; audio compressor362cthat compresses the translated audio data and transmits the compressed audio data to corresponding VR system100or the like; machine translation management table333; user management table330d; and guide instruction recorder315.

In VR tourism system5baccording to the present variation, VR systems100,110,120, and130of each user and translation processing module group370(translation processing module group S (361b) and translation processing module group D (362D)) are correctly connected based on machine translation management table333and user management table330b. In addition, with respect to a VR user of a same language as the language of use of the guide (in the present variation, Japanese guide20a), the VR user engages in conversation with the guide as-is.

In VR tourism system5baccording to the present variation, audio translation is performed by VR cloud300fand two-way communication is performed when only a guide who speaks a language that differs from one's own language is present. However, with the exception of this point, VR tourism system5baccording to the present variation has the same functions as VR tourism system2according to Example 1 illustrated inFIG.5.

Based on machine translation management table333and user management table330b, VR tourism service manager340of VR cloud300fselects a correct translation processing module from translation processing module group370using language selection of a VR user, a language of a guide, and a setting of a default language and sets values of each VR user of a translation processing module ID (“MT-ID-S”) for a guide and a translation processing module ID (“MT-ID-D”) for a VR user in user management table330.

FIG.35is a diagram illustrating an operation flow example of VR tourism system5b(in other words, a specific example of a video display method) according to Variation 2 of Example 4. More specifically, (a) inFIG.35illustrates an operation flow of VR systems100,110,120, and130and (b) inFIG.35illustrates an operation flow of VR cloud300f.

In each of VR systems100,110,120, and130, main processing, interrupt processing of input from VR users, and interrupt processing from VR cloud300fare the same as in Example 4 (not illustrated).

In addition, in VR systems100,110,120, and130, initialization processing ((a) inFIG.35) by which a VR user starts using VR tourism is the same as the operation flow (S100to S105) according to Example 3 illustrated in (b) inFIG.17and a description thereof will be omitted.

Furthermore, in VR cloud300f, main processing is the same as in Example 1 and Example 2 (not illustrated).

As login processing in VR cloud300f, as illustrated in (b) inFIG.35, VR tourism service manager340first acquires a user ID and a password (S210) from VR systems100,110,120, and130and checks whether a user is a legitimate user of the present tourism service (S210).

When the user is not a legitimate user (N in S211), VR tourism service manager340transmits an error to corresponding VR system100or the like (S212) and returns to step S210.

When the user is a legitimate user (Y in S211), VR tourism service manager340acquires a value of language selection of the corresponding user from user management table330b(S213).

As a result, when language selection by the corresponding user is the same as the language of guide20(Y in S215), VR tourism service manager340jumps to step S220.

When a pair of the language selection by the corresponding user and the language of the guide is in the translation processing modules (Y in S216), VR tourism service manager340sets a value of the selected pair as translation processing module IDs (“MT-ID-S” and “MT-ID-D”) in user management table330bof the corresponding user (S225), connects audio of corresponding VR system100or the like to a translation processing module corresponding to the selected translation processing module ID (S226), and jumps to step S218.

When the default language selection and the language of the guide are the same (Y in S217), VR tourism service manager340jumps to step S220.

When the default language selection and the language of the guide are different (N in S217) and a pair of the language of the guide and the default language is in translation processing (Y in S230), VR tourism service manager340sets the translation processing module ID in user management table330bof corresponding user as the pair of the language of the guide and the default language (S231), connects audio of corresponding VR system100or the like to a translation processing module of the selected translation processing module ID (S232), and jumps to step S218.

When the pair of the language of the guide and the default language is not in translation processing (N in S230), VR tourism service manager340jumps to step S220.

In step S220, VR tourism service manager340sets two values of translation processing module IDs in user management table330bof the corresponding user to “N/A” (S220), connects audio of corresponding VR system100or the like to guide's terminal210aand the like (S221), and jumps to step S218.

In S218, VR tourism service manager340transmits the fact that preparation for use of the tourism service has been normally completed to corresponding VR system100or the like (S218) and ends processing.

As described above, even with VR tourism system5baccording to the present variation, explanations and conversations accommodating a plurality of languages by machine translator360are supported and the problem (4) described above is solved.

Although the video display system and the video display method according to the present disclosure have been described based on the foregoing embodiment and examples, the present disclosure is not limited to the embodiment and examples. Other modes constructed by applying various variations conceivable by a person skilled in the art to the present embodiment and examples or by combining a part of constituent elements of the embodiment and the examples with each other are also included in the scope of the present disclosure unless there is a deviation from the gist of the present disclosure.

For example, while an example in which the video display system according to the present disclosure is applied to a VR tourism system has been described as examples, the video display system according to the present disclosure is not limited to VR tourism systems and can also be applied to a VR experience system that provides virtual experiences such as attending a trade show, making an inspection tour, going on a factory tour, and visiting an art museum, a museum, a zoo, an aquarium, or the like.

In addition, while conversations by audio between participant10and the like and guide20and the like are assisted by audio conversation units in the embodiment and the like described above, in addition thereto or in place thereof, the conversations may be displayed as subtitles on a display unit of a VR system. To this end, for example, the VR system may be provided with an audio recognizer that recognizes audio of a conversation in real time, a video compositor that causes the display unit of the VR system to display text obtained by the audio recognizer, and the like.

In this case, questions may be accepted by guide20from participant10and the like in the form of a chat. In a similar manner to the case of audio, guide20may control whether to accept or reject chat input using a guide's remote controller.

Furthermore, in Example 4 that accommodates a plurality of languages, machine translators may output text to VR systems instead of outputting audio. Conceivable specific configuration examples include:1) instead of converting a guide's audio in a single language into another language, converting the guide's audio into text in the same language and displaying on the VR system;2) translating the audio into a language that differs from the language of use of the guide, converting the translated audio into text, and displaying on the VR system;3) when there are a plurality of guides using different languages, converting audio of each guide into text without translating the audio and displaying the translated text on the VR system; and4) when there are a plurality of guides using different languages, selecting a guide who uses a language that is close to the language to be converted into text, translating audio of the selected guide, converting the translated audio into text, and displaying the text on the VR system. To this end, for example, a VR cloud may be provided with an audio recognizer, a compositor that composites text obtained by the audio recognizer with a 360-degree video, and the like in addition to machine translators.

Propriety of displaying subtitles can be determined by describing and managing information on whether or not subtitles are to be used in user management table330ainFIG.26, user management table330bor machine translation management table333inFIG.31, or user management table330binFIG.33or by having participant10or the like select upon use.

INDUSTRIAL APPLICABILITY

The video display system and the video display method according to the present disclosure can be used as a video display system for providing a service for experiencing VR (virtual reality) such as a VR experience system that provides virtual experiences including going on a sightseeing tour, attending a trade show, making an inspection tour, going on a factory tour, and visiting an art museum, a museum, a zoo, an aquarium, or the like.

REFERENCE SIGNS LIST