GENERATION DEVICE, GENERATION METHOD, AND GENERATION PROGRAM

A generation device includes a coefficient acquisition unit that acquires coefficient information of an object on the basis of raw material information of the object when detecting an interaction of the object mapped on cyber space, a sound source selection unit that calculates statistical information in which a type of a sound source corresponding to the interaction of the object and intensity of the sound source are associated with each other and selects sound source information. Further, there is a voice synthesis unit that generates synthesized sound source information obtained by synthesizing the sound source information on the basis of the statistical information and the sound source information and generates three-dimensional sound source information obtained by converting the synthesized sound source information into three-dimension by executing three-dimensional sound rendering on the basis of the position information to the synthesized sound source information.

TECHNICAL FIELD

The present invention relates to a generation device, a generation method, and a generation program.

BACKGROUND ART

Digital twin (DT) technique for mapping a real space target onto cyber space is realized by the progress of ICT (Information and Communication Technology) technology and is attracting attention. The DT is configured to map and accurately express a shape, a state, a function, and the like of a real world target such as a production machine in a factory, an engine of an aircraft, an automobile.

By using this DT, current analysis, future prediction, simulation of possibility, and the like regarding the target can be performed in the cyber space. Further, cyber space benefit such as intelligently controlling the real world target on the basis of the result, for example, benefit of facilitating utilization of the ICT technology, can be fed back to the real world target.

In the future, increasing use of converting various real world targets to DTs is expected to increase demand for cooperation between industries and large-scale simulations by interacting (interaction) or combining different and diverse DTs across industries.

When a user himself or herself uses the DT or the DT is used by the DT (human DT) of the user, the user or the human DT need to recognize the interaction between DTs as in the real world.

The human recognizes a real world event by using five senses. For this reason, in order to recognize the interaction between DTs, it is necessary to digitize visual, auditory, olfactory and taste information of the interaction. In this case, attention is paid to digitization of sound information related to the auditory information.

FIG.10is a diagram for explaining an image of generating the sound related to converting mono to the DT. In a concept of DTC (Digital Twin Computing), by digitizing the mono in the real world, it is possible to process and edit the mono or to interact with another mono in digital space.

In an example shown inFIG.10, a mono11is generated by digitizing a real world mono10. Also, the mono11and another mono12are made to interact to generate a sound “pock”. The mono11is processed and edited to generate a mono11a, and the mono11aand another mono12aare made to interact to generate a sound “con”. The mono11is processed and edited to generate a mono11b, and the mono11band another mono12bare made to interact to generate a sound “challin”.

In order to generate the sound described inFIG.10, a sound may be manually assigned.FIG.11is a diagram for explaining a manual assigning procedure of a sound FB. As shown inFIG.11, first, a virtual object is defined (step S10). For example, the virtual objects ob1-1, ob1-2, ob1-3, ob2-1, ob2-2and ob2-3are defined.

Subsequently, the interaction which can be performed by the virtual object is defined (step S11). In an example shown inFIG.11, an interaction int1between the virtual object2-3and the virtual object1-1is defined. Finally, a sound is assigned to the interaction int1(step S12).

Although the above-mentioned prior art is a technique for manually assigning the sound, there is a prior art for analyzing video to synthesize an effect voice.FIG.12is a flowchart showing the processing procedure of the prior art in which video is analyzed and effect voice is synthesized.

As shown inFIG.12, in the prior art, material and interaction of the object are selected (step S20). In an example shown inFIG.12, the material is selected from a plurality of candidates included in a group G1, and the interaction is selected from a plurality of candidates included in a group G2.

In the prior art, sound textures for the selected material and interaction are generated (step S21). The sound textures include sound texture so1of each material and sound texture so2of each interaction.

In the prior art, a two-dimensional video17is analyzed by using an NN (Neural Network)16, and an appropriate sound18obtained by combining sound textures so1and so2is synthesized (step S22).

CITATION LIST

Non Patent Literature

SUMMARY OF INVENTION

Technical Problem

In the prior art described inFIG.12, since the two-dimensional video17is analyzed and sound texts so1and so2for the material and the interaction of the object are synthesized, there is a problem that it is impossible to apply the prior art as it is to three-dimensional space requiring a position of a sound source.

Unlike the two-dimensional sound synthesis, in the sound synthesis in the three-dimensional space, the sound is unnatural to the user unless the position of the sound source is appropriately set.FIG.13is a diagram showing an example of the three-dimensional space. In the three-dimensional space15shown inFIG.13, an object15bexists at the position in upper right front of a user15a. The sound with the object15bas the sound source is unnatural to the user15aunless the sound is set to be heard from the upper right front of the user15a.

The present invention has been made in view of the above, and an object of the present invention is to provide a generation device, a generation method, and a generation program capable of synthesizing a realistic sound in the three-dimensional space.

Solution to Problem

In order to solve the above-described problem and achieve the object, it is characterized in that a generation device include a coefficient acquisition unit that acquires coefficient information of an object on the basis of raw material information of the object when detecting an interaction of the object mapped on cyber space, a sound source selection unit that calculates statistical information in which a type of a sound source corresponding to the interaction of the object and intensity of the sound source are associated with each other and selects sound source information corresponding to the statistical information by inputting position information, shape information, and the coefficient information of the object to a machine learning model, and a voice synthesis unit that generates synthesized sound source information obtained by synthesizing the sound source information on the basis of the statistical information and the sound source information and generates three-dimensional sound source information obtained by converting the synthesized sound source information into three-dimension by executing three-dimensional sound rendering on the basis of the position information to the synthesized sound source information.

Advantageous Effects of Invention

According to the present invention, the realistic sound can be synthesized even in the three-dimensional space.

DESCRIPTION OF EMBODIMENTS

Hereinafter, examples of the generation device, the generation method, and the generation program disclosed in the present application will be described in detail with reference to the drawings. Note that these examples are not intended to limit the scope of the present invention.

Examples

In the DT (Digital Twin) technique, a real space object is mapped onto cyber space (digital three-dimensional space). Data of the object mapped onto the cyber space is expressed as DT data. A user can view the DT data of the object mapped onto the cyber space by using VR (Virtual Reality) or AR (Augmented Reality).

FIG.1is a diagram for explaining the DT data that is used in the present example. As shown inFIG.1, the DT data includes a position, a posture, a shape, an appearance, a material, and a mass as parameters.

The position is position coordinates (x, y, z) of the object which uniquely specifies the position of the object. The posture is posture information (yaw, roll, pitch) of the object which uniquely specifies an orientation of the object. The shape is mesh information or geometry information representing the shape of a three-dimensional object to be displayed. The appearance is color information of an object surface. The material is information indicating the material of the object. The mass is information indicating the mass of the object.

A generation device according to the present example acquires coefficient information (friction coefficient, attenuation coefficient) of the object on the basis of raw material information (information including the shape, the mass, and the material) of the object when detecting an interaction of the object in the cyber space.

The generation device calculates statistical information in which a type of a sound source corresponding to the interaction of the object and intensity of the sound source are associated with each other, and selects sound source information corresponding to the statistical information from a sound source DB by inputting the position information, the shape information, and the coefficient information of the object to a trained machine learning model.

The generation device generates synthesized sound source information obtained by synthesizing the sound source information on the basis of the statistical information and the sound source information. The generation device generates three-dimensional sound source information by executing three-dimensional sound rendering on the basis of the position information of the object to the synthesized sound source information, and outputs the three-dimensional sound source information to a device such as a VR device or an AR device.

The generation device executes the above-mentioned processing to generate and output the realistic three-dimensional sound source information. The three-dimensional sound source information is a natural sound source for a user experiencing the cyber space.

Next, a configuration example of the generation device according to the present example will be described.FIG.2is a functional block diagram showing a configuration of the generation device according to the present example. As shown inFIG.2, this generation device100is connected to a device50. For example, the device50corresponds to the VR device, the AR device, and the like. The device50is a device for mapping the real space object onto the cyber space, and the DT data of each object in the cyber space is defined by the data shown inFIG.1. When each object is a dynamic object, it moves on the cyber space in accordance with preset scenario information to generate the interaction. The dynamic object may generate the interaction with another dynamic object or generate the interaction with a static object.

The device50outputs the DT data (time-series DT data) corresponding to each object to the generation device100. In addition, the device50acquires the three-dimensional sound source information generated by the generation device100, and generates the three-dimensional sound source in the cyber space.

In the example shown inFIG.2, the device50and the generation device100are separately shown, but one device having a function of the device50and a function of the generation device100may be used.

The generation device100includes an interaction detection unit110, a physical information acquisition unit110a, an object extraction unit110b, a coefficient acquisition unit120, a sound source selection unit130, and a voice synthesis unit140.

The interaction detection unit110acquires the DT data of each object from the device50, and detects the interaction of the object. It is assumed that the interaction is defined in advance. For example, the interaction detection unit110detects a collision (interaction) when a distance between a plurality of objects becomes less than a threshold value.

The interaction detection unit110outputs the target object information to the physical information acquisition unit110a, and outputs the scenario information to the object extraction unit110bwhen detecting the interaction. The target object information includes the DT data of fixed time interval before and after the detection of the interaction that is DT data of the object related to the interaction. The scenario information includes a type of the interaction, DT data of the object related to the interaction, and the like.

The physical information acquisition unit110acalculates the shape information, the position information, and movement information of the object at the time of interaction detection on the basis of the target object information. For example, the shape of the object includes information on a collision area. The position information includes information on the three-dimensional position of the object on the cyber space at a point of time when the interaction is detected. The movement information includes information on a movement speed and a movement direction immediately before the interaction is detected.

The physical information acquisition unit110aoutputs the shape information, the position information, and the movement information to the sound source selection unit130. The physical information acquisition unit110aoutputs the position information and the movement information to the voice synthesis unit140.

The object extraction unit110bextracts raw material information of the object related to the interaction on the basis of the scenario information. The raw material information includes information on the shape, the mass, and the material of the object. The object extraction unit110boutputs the extracted raw material information to the coefficient acquisition unit120and the sound source selection unit130.

The coefficient acquisition unit120has a material DB (DataBase)120aand an acquisition unit120b.

The material DB120astores information on the friction coefficient and the attenuation coefficient of substance corresponding to combination of the shape, the mass, and the material of the object.

The acquisition unit120bacquires the friction coefficient and the attenuation coefficient corresponding to the raw material information on the basis of the raw material information (shape, mass, material) and the material DB120a. The acquisition unit120bsets the acquired friction coefficient and attenuation coefficient to coefficient information, and outputs the coefficient information to the sound source selection unit130.

The sound source selection unit130has a sound source DB130aand a selection unit130b.

The sound source DB130astores a plurality of sound elements. The sound elements include sinusoidal information and sound texture information of each different frequency. The sound textures include information related to small recordings and cochleagram.

For example, the sound source DB130astores the sound sources of the sound elements of a plurality of materials and the sound sources of the sound elements of a plurality of interactions.FIG.3is a diagram showing an example of sound elements stored in the sound source DB130a. InFIG.3, sound elements related to the material se_m1, se_m2, se_m3, se_m4, . . . , se_mnand sound elements related to the interaction se_i1, se_i2, . . . , se_inare described. Each sound element is information in which frequency, time, and intensity are associated with each other. Each sound element identifies the material and the interaction.

The selection unit130bacquires the position information and the shape information of the object from the physical information acquisition unit110a. The selection unit130bacquires the coefficient information from the coefficient acquisition unit120.

The selection unit130binputs the position information, the shape information, and the coefficient information of the object to the trained machine learning model to calculate the statistical information corresponding to the interaction of the object. The selection unit130bselects the sound source information corresponding to the statistical information from the sound source DB130a. The selection unit130boutputs the statistical information and the sound source information to the voice synthesis unit140.

FIG.4is a diagram for explaining processing of the selection unit. In the example shown inFIG.4, the position information, the shape information, and the coefficient information of the object are collected to be object related information30. For example, the object related information30includes the following information (1) to (6).(1) Mass of object(2) Friction coefficient, attenuation coefficient(3) Collision area(4) Raw material of collision part(5) Speed of collision part(6) Acceleration of collision part

The selection unit130bperforms dimension reduction by performing PCA (Principal Component Analysis) to the object related information30. The selection unit130bmay perform the dimension reduction by using any conventional technique. The selection unit130bmay select any information among (1) to (6) to perform the dimension reduction.

The selection unit130bcalculates the statistical information36by inputting the object related information30subjected to dimension reduction to the machine learning model35. The machine learning model35corresponds to a Recurrent Neural Network, a Convolutional Neural Network, etc.

For example, the identification information of each sound source is associated with the intensity of the sound source in the statistical information. The statistical information36includes the sound elements related to the materials se_m120%, se_m270%, and se_m310% and the sound elements related to the interactions se_i180% and se_i220%. In the statistical information36, when the intensities of the sound elements related to the materials are summed up, the sum becomes 100%. In addition, the intensities of the sound elements related to the interactions are summed up, the sum becomes 100%.

The selection unit130bselects the sound source information corresponding to the statistical information36from the sound source DB130a. The selection unit130boutputs the statistical information (for example, the statistical information36) and the sound source information to the voice synthesis unit140.

Here, it is assumed that the machine learning model35used by the selection unit130bis trained in advance on the basis of teacher data composed of a set of input data and a correct answer label. The input data corresponds to the object related information30. The correct answer label is information in which the identification information of the sound source and the intensity of the sound source are associated with each other.

Let us return to the description ofFIG.2. The voice synthesis unit140acquires the position information and the movement information from the physical information acquisition unit110a. The voice synthesis unit140acquires the statistical information and the sound source information from the sound source selection unit130. The voice synthesis unit140generates the three-dimensional sound source information on the basis of the statistical information, the sound source information, the position information, and the movement information.

The voice synthesis unit140has a synthesis unit140aand a rendering unit140b.

The synthesis unit140agenerates synthesized sound source information obtained by synthesizing the sound source information on the basis of the statistical information and the sound source information. For example, the synthesis unit140aconnects the sound elements of the sound source information to generate the synthesized sound source information by using NPL 2.

The synthesis unit140aoutputs the generated synthesized sound source information to the rendering unit140b.

The rendering unit140bgenerates the three-dimensional sound source information (3D waveform) obtained by converting the synthesized sound source information into three-dimension by executing three-dimensional sound rendering on the basis of the synthesized sound source information, the position information and the movement information. For example, the rendering unit140bexecutes the three-dimensional sound rendering by using NPL 3. The rendering unit140boutputs the three-dimensional sound source information to a transmission unit150.

The transmission unit150transmits the three-dimensional sound source information to the device50.

Next, processing procedure of the generation device100according to the present example will be described.FIG.5is a flowchart showing the processing procedure of the generation device according to the present example. As shown inFIG.5, the interaction detection unit110of the generation device100acquires the DT data from the device50(step S101). The interaction detection unit110of the generation device100shifts to the step S101when detecting no interaction (step S102, No).

On the other hand, the interaction detection unit110shifts to a step S103when detecting the interaction (step S102, Yes). The physical information acquisition unit110aof the generation device100calculates the shape information, the position information, and the movement information related to the interacted object on the basis of the target object information acquired from the interaction detection unit110(step S103).

The object extraction unit110bof the generation device100extracts the raw material information related to the interacted object on the basis of the scenario information acquired from the interaction detection unit110(step S104).

The coefficient acquisition unit120of the generation device100executes coefficient acquisition processing (step S105). The sound source selection unit130of the generation device100executes sound source selection processing (step S106). The voice synthesis unit140of the generation device100executes voice synthesis processing (step S107).

The transmission unit150of the generation device100transmits the three-dimensional sound source information to the device50(step S108).

Next, an example of the processing procedure of the coefficient acquisition processing shown in step S105ofFIG.5will be described.FIG.6is a flowchart showing the processing procedure of the coefficient acquisition processing. As shown inFIG.6, the acquisition unit120bof the coefficient acquisition unit120acquires the raw material information of each interacted object (step S201).

The acquisition unit120bretrieves the friction coefficient and the attenuation coefficient from the material DB120aon the basis of the raw material information (step S202). The acquisition unit120boutputs the friction coefficient and the attenuation coefficient to the sound source selection unit130(step S203).

Next, an example of the processing procedure of the sound source selection processing shown in the step S106ofFIG.5will be described.FIG.7is a flowchart showing the processing procedure of the sound source selection processing. As shown inFIG.7, the selection unit130bof the sound source selection unit130acquires the position information, the shape information and the coefficient information of the object (step S301).

The selection unit130bexecutes the dimension reduction to the object related information (step S302). The selection unit130binputs the object related information subjected to the dimension reduction to the machine learning model, and calculates the statistical information (step S303).

The selection unit130bacquires the sound source information corresponding to the statistical information from the sound source DB130a(step S304). The selection unit130boutputs the statistical information and the sound source information to the voice synthesis unit140(step S305).

Next, an example of the processing procedure of the voice synthesis processing shown in the step S107ofFIG.5will be described.FIG.8is a flowchart showing the processing procedure of the voice synthesis processing. As shown inFIG.8, the synthesis unit140aof the voice synthesis unit140acquires the position information, the movement information, the statistical information, and the sound source information (step S401).

The synthesis unit140agenerates the synthesized sound source information by connecting the sound elements of the sound source information on the basis of the statistical information (step S402). The rendering unit140bof the voice synthesis unit140executes the three-dimensional sound rendering on the basis of the synthesized sound source information, the position information, and the movement information, and generates the three-dimensional sound source information (step S403).

The rendering unit140boutputs the three-dimensional sound source information (step S404).

Next, the effect of the generation device100according to the present example will be described. The generation device100acquires the coefficient information (friction coefficient, attenuation coefficient) of the object on the basis of the raw material information (information including the shape, the mass, and the material) of the object when detecting the interaction of the object on the cyber space. The generation device100calculates the statistical information in which the type of the sound source corresponding to the interaction of the object and the intensity of the sound source are associated with each other, and selects the sound source information corresponding to the statistical information from the sound source DB130aby inputting the position information, the shape information, and the coefficient information of the object to the trained machine learning model. The generation device100generates the synthesized sound source information obtained by synthesizing the sound source information on the basis of the statistical information and the sound source information, generates the three-dimensional sound source information by executing the three-dimensional sound rendering on the basis of the position information of the object on the synthesized sound source information, and outputs the synthesized sound source information to the device50.

The generation device100can generate and output the realistic three-dimensional sound source information by executing the above-mentioned processing. The three-dimensional sound source information is the natural sound source for the user experiencing the cyber space by using the device50.

The generation device100executes the dimension reduction on the position information, the shape information, and the coefficient information, and calculates the statistical information by inputting information to be an execution result of the dimension reduction into the machine learning model. Accordingly, the cost for using the machine learning model can be reduced.

The generation device100acquires the speed of the collision part and the acceleration of the collision part of the two objects, and the positions of the two objects as the position information when detecting the collision of the two objects as the interaction. By using such position information, the three-dimensional sound source information on the cyber space can be generated with high accuracy.

Subsequently, an example of a computer that performs a generation program will be described below.FIG.9is a diagram showing an example of the computer that executes the generation program. A computer1000includes a memory1010, a CPU1020, a hard disk drive interface1030, a disc drive interface1040, a serial port interface1050, a video adapter1060, and a network interface1070, for example. These respective units are connected by a bus1080.

The memory1010includes a ROM (Read Only Memory)1011and a RAM1012. The ROM1011stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface1030is connected to a hard disk drive1031. The disc drive interface1040is connected to a disc drive1041. A removable storage medium such as a magnetic disc or an optical disc is inserted into the disc drive1041, for example. A mouse1051and a keyboard1052are connected to the serial port interface1050, for example. A display1061is connected to the video adapter1060, for example.

Here, an OS1091, an application program1092, a program module1093, and program data1094are stored in the hard disk drive1031, for example. Each piece of information described in the above-mentioned embodiment is stored in the hard disk drive1031or the memory1010, for example.

In addition, the generation program is stored in the hard disk drive1031, for example, as the program module1093in which instructions executed by the computer1000are described. Specifically, the program module1093in which respective processes for executing the interaction detection unit110, the physical information acquisition unit110a, the object extraction unit110b, the coefficient acquisition unit120, the sound source selection unit130, the voice synthesis unit140, the transmission unit150are described is stored in the hard disk drive1031.

Further, data used for information processing by the generation program is stored in, for example, the hard disk drive1031as the program data1094. Thereafter, the CPU1020reads the program module1093and the program data1094stored in the hard disk drive1031into the RAM1012when necessary, and executes each procedure described above.

Note that the storage of the program module1093and the program data1094related to the generation program is not limited to the case where the program module1093and the program data1094are stored in the hard disk drive1031, and the program module1093and the program data1094may be stored in, for example, a detachable storage medium and may be read by the CPU1020via the disc drive1041. Alternatively, the program module1093and the program data1094related to the generation program may be stored in another computer connected via a network such as a LAN or a WAN (Wide Area Network) and may be read by the CPU1020via the network interface1070.

Although the embodiments to which the invention made by the present inventor is applied have been described above, the present invention is not limited by the descriptions and drawings forming a part of the disclosure of the present invention according to the embodiments. That is to say, other embodiments, examples, operation techniques, and the like made by those skilled in the art on the basis of the embodiment are all included in the scope of the present invention.

REFERENCE SIGNS LIST