Patent Description:
There are sound field sound collection generation apparatuses that drive a speaker with a drive signal generated by using a wavefront synthesis method from sound field signals that have been collected by a plurality of microphones so as to virtually reproduce the sound field at the sound-collected location (for example, see Patent Document <NUM>). Patent Document <NUM> relates to a virtual reality virtual theater system that generates or otherwise displays a virtual theater for viewing videos etc. and projects sounds of the videos etc. to virtual theater speakers. Patent Document <NUM> relates to providing binaural sound to listener, which is convolved with a head related transfer function of the listener, while the listener watches a movie such that sounds from the movie localize to a location of a character in the movie. Patent Document <NUM> relates to determining orientation angles of spherical video and rendering a sound file based on the orientation angles.

Only the reproduction of the sound field, however, cannot give a sufficient realistic feeling to a listener who is listening to acoustics, in some cases. Therefore, the present disclosure proposes an acoustic processing apparatus, an acoustic processing method, and an acoustic processing program, which are capable of enhancing the realistic feeling given to the listener who is listening to acoustics.

The problems are solved by an acoustic processing apparatus, an acoustic processing method, and an acoustic processing program according to the independent claims. Further embodiments are given in the dependent claims.

It is to be noted that in each of the following embodiments, the same parts are designated by the same reference numerals, so that duplicate description will be omitted.

<FIG> is an explanatory diagram showing an outline of an acoustic process by an acoustic processing apparatus <NUM> according to the present disclosure. The acoustic processing apparatus <NUM> is an apparatus that converts acoustic characteristics of an acoustic content such as a movie, a live music performance, a promotional video of a music piece, a television program, a music piece, or the like into acoustic characteristics of a sound-recorded location, so as to regenerate a sound field while reproducing the sound field of a sound-recorded location.

Here, the acoustic processing apparatus <NUM> cannot give a sufficient realistic feeling to a listener of the acoustic content only by reproducing the sound field of the sound-recorded location, in some cases. Specifically, in a case where a measurement location where the acoustic characteristics of the acoustic content are measured and a regeneration location of the acoustic content are the same, the acoustic processing apparatus <NUM> is capable of giving the listener as if the listener were staying at the location. However, in other cases, the realistic feeling is reduced by half.

For example, even in a case where the acoustic processing apparatus <NUM> provides a user who views and listens to a movie at home with an acoustic content of the movie in which a reverberation characteristic, an echo characteristic, and the like of a movie theater are reproduced and converted into acoustic characteristics, it is difficult to give the user the feeling of staying in a movie theater, because the staying location of the user is a user's house.

Hence, the acoustic processing apparatus <NUM> utilizes, for example, a technology such as virtual reality (hereinafter, referred to as VR: Virtual Reality) to reproduce the measurement location where acoustic information of the acoustic content has been measured, and then regenerates the sound field of the acoustic content in which the acoustic characteristics of the measurement location have been reproduced.

For example, as shown in <FIG>, the acoustic processing apparatus <NUM> stores beforehand a video content D1 of a movie, an acoustic content D2 of the movie, and acoustic information D3 of a movie theater, and in addition, a VR entire spherical video D4 inside the movie theater (step S01). The acoustic information D3 of the movie theater includes various parameters related to acoustic characteristics in the movie theater.

Then, for example, when the acoustic processing apparatus <NUM> acquires a provision request for a movie content from a user terminal <NUM> such as a smartphone carried by a user U, the acoustic processing apparatus <NUM> transmits and provides the user terminal <NUM> of the user U with the video content D1 of the movie and the acoustic content D2 of the movie.

In this situation, the acoustic processing apparatus <NUM> converts the acoustic characteristics of the acoustic content D2 of the movie into the acoustic characteristics of the movie theater on the basis of the acoustic information D3 of the movie theater, and provides the user terminal <NUM> with the acoustic characteristics together with the VR entire spherical video D4 inside the movie theater (step S02).

The VR entire spherical video D4 inside the movie theater includes an image of a screen installed in the movie theater and images of surrounding environments including audience seats, walls, a ceiling, and the like of the movie theater. The acoustic processing apparatus <NUM> adds information indicating that the position of the screen in the VR entire spherical video D4 inside the movie theater is set to a display position of the video content D1 of the movie to the video content D1 of the movie, and provides the user terminal <NUM> with the video content D1 of the movie.

Accordingly, the acoustic processing apparatus <NUM> is capable of displaying the VR entire spherical video D4 inside the movie theater on, for example, a head-mounted display <NUM> worn by the user U, and displaying the video content D1 of the movie on the screen in the VR entire spherical video D4 (step S03).

At the same time, the acoustic processing apparatus <NUM> is capable of regenerating the sound field of the acoustic content D2 of the video content D1, in which the acoustic characteristics have been converted into the acoustic characteristics in the movie theater through, for example, an earphone <NUM> worn by the user U (step S04).

In this manner, the acoustic processing apparatus <NUM> is capable of allowing the user U to listen to the acoustic content D2, while allowing the user U to visually recognize not only the video content D1 of the movie projected on the screen but also the surrounding environments of the screen such as the audience seats, the walls, the ceiling, and the like of the movie theater.

Accordingly, the acoustic processing apparatus <NUM> is capable of giving a realistic feeling to the user U who views and listens to the movie video content at home, for example, as if the user U were watching a movie in the movie theater. Here, the VR entire spherical video D4 inside the movie theater is displayed on the head-mounted display <NUM>. However, the acoustic processing apparatus <NUM> may display a three-dimensional computer graphics (3DCG) video in which the inside of the movie theater is reproduced, instead of the VR entire spherical video D4 inside the movie theater. It is to be noted that the operation of the acoustic processing apparatus <NUM>, which has been described with reference to <FIG>, is an example. Other operation examples of the acoustic processing apparatus <NUM> will be described later with reference to <FIG>.

Next, an example of the configuration of the acoustic processing apparatus <NUM> will be described with reference to <FIG> is a block diagram showing an example of the configuration of the acoustic processing apparatus <NUM> according to the present disclosure. As shown in <FIG>, the acoustic processing apparatus <NUM> includes a communication unit <NUM>, a storage unit <NUM>, and a controller <NUM>.

The communication unit <NUM> is realized by, for example, a network interface card (NIC) or the like. The communication unit <NUM> is connected to enable information communication with the user terminal <NUM>, an acoustic information creating apparatus <NUM>, and a creator terminal <NUM> in a wired or wireless manner through a communication network N such as the Internet.

The acoustic information creating apparatus <NUM> is an apparatus that creates acoustic information <NUM>, which will be described later. Further, the creator terminal <NUM> is a terminal device used by a creator who creates an acoustic content <NUM> to be provided for the user U by the acoustic processing apparatus <NUM>.

The storage unit <NUM> is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory (Flash Memory), or a storage device such as a hard disk, an optical disk, or the like. Such a storage unit <NUM> stores a video content <NUM>, the acoustic content <NUM>, VR video information <NUM>, the acoustic information <NUM>, and the like.

The video content <NUM> denotes a plurality of video data such as a movie, a live music performance, a promotional video of a music piece, a TV program, and the like, and denotes content data provided for the user terminal <NUM> by the acoustic processing apparatus <NUM>.

The acoustic content <NUM> denotes a plurality of pieces of audio data such as a movie, a live music performance, a promotional video of a music piece, a TV program, a music piece, and the like, and denotes content data to be provided for the user terminal <NUM> by the acoustic processing apparatus <NUM>.

The VR video information <NUM> includes a plurality of VR entire spherical videos that has been captured at various locations. Here, an example of a method for creating a VR entire spherical video will be described with reference to <FIG> is an explanatory diagram of the method for creating the VR entire spherical video according to the present disclosure.

As shown in <FIG>, in a case where the VR entire spherical video is created, a <NUM>-degree camera <NUM> is installed at each location where the acoustic content <NUM> is regenerated, and the <NUM>-degree camera <NUM> captures an omnidirectional image including the front, back, up, down, left, and, right of such a location, so as to capture a VR entire spherical video Vr.

Accordingly, for example, the <NUM>-degree camera <NUM> is installed in a movie theater to capture images, so as to enable creation of the VR entire spherical video Vr including the screen of the movie theater shown in <FIG> and the images of the surrounding environments of the screen such as the audience seats, the walls, the ceiling, and the like of the movie theater.

It is to be noted that in the example shown in <FIG>, the movie theater in which a single screen is installed has been given as an example. However, in the present disclosure, a VR entire spherical video Vr of a movie theater including four screens installed on a front face, left and right faces, and a bottom face can be created.

In such a case, the acoustic processing apparatus <NUM> displays the video content <NUM> on a single screen of the front face of the four screens, and displays the surrounding environments of the movie theater on the other three screens. Also in such a configuration, the acoustic processing apparatus <NUM> enables an enhancement in the realistic feeling given to the user.

Returning to <FIG>, the acoustic information <NUM> will be described. The acoustic information <NUM> includes a plurality of pieces of information regarding acoustics at each location, in which the sound field of the acoustic content <NUM> is regenerated. Here, an example of the acoustic information <NUM> will be described with reference to <FIG> is an explanatory diagram showing an example of the acoustic information <NUM> according to the present disclosure.

As shown in <FIG>, the acoustic information <NUM> is provided for every user for whom the video content <NUM> or the acoustic content <NUM> is provided by the acoustic processing apparatus <NUM>. The acoustic information <NUM> denotes information in which a user ID, a user's head related transfer function (hereinafter, referred to as an HRTF: Head Related Transfer Function), a location, a VR video, and acoustic parameters are associated with each other.

The user ID is identification information for identifying each user. The HRTF is a function information unique to each user that mathematically represents how the sound reaches user's ears from a sound source. Here, an HRTF measurement method will be described with reference to <FIG> and <FIG>.

<FIG> and <FIG> are explanatory views of the HRTF measurement method according to the present disclosure. For example, in a case of measuring the HRTF included in the acoustic information D3 of the movie theater that has been described with reference to <FIG>, the user U is asked to wear an ear microphone <NUM> on an audience seat of a movie theater Mt, and the sound of a test signal Ts is output from a speaker SP of the movie theater, as shown in <FIG>.

Then, the acoustic information creating apparatus <NUM> acquires an audio signal SL, which has been collected by the ear microphone <NUM> attached to the left ear of the user U, and an audio signal SR, which has been collected by the ear microphone <NUM> attached to the right ear of the user U.

Then, the acoustic information creating apparatus <NUM> derives an HRTF of the user U on the basis of a temporal deviation, a deviation in a signal level (intensity), a difference in resonance, and the like between the two audio signals SL and SR that have been acquired. In this manner, the acoustic information creating apparatus <NUM> actually measures the test signal Ts heard by the user U, so as to be capable of deriving an accurate HRTF of the user U.

It is to be noted that the HRTF differs depending on the location (the environment) where the user U listens to the test signal Ts. For this reason, for example, in a case where there are a plurality of locations where a user desires to view the video while listening to the acoustic content <NUM>, it is necessary for the user to come to each of the locations in order to measure and derive the HRTF. Such conducts will be a burden for the user.

Therefore, the acoustic information creating apparatus <NUM> is also capable of deriving the HRTF of the user U at a plurality of locations while reducing the burden of the user U. For example, as shown in <FIG>, in the audio signal collected by the ear microphone <NUM> attached to the ear of the user U, a sound wave characteristic has a period depending on the user U in a predetermined period part in the beginning, and after the period, the sound wave characteristic has a period depending on the location.

Therefore, for example, the user U is asked to come to one location and the sound of the test signal Ts is collected by the ear microphone <NUM>, so that the acoustic information creating apparatus <NUM> acquires an audio signal waveform of a part of the period depending on the user U. Subsequently, dummy dolls DM each equipped with the ear microphone <NUM> are installed at a plurality of locations desired by the user U, so that the acoustic information creating apparatus <NUM> acquires the audio signal waveform of a part of the period while the sound wave characteristic depends on the location.

Then, the acoustic information creating apparatus <NUM> synthesizes the audio signal waveform of the part of the period depending on the user U and the audio signal waveform of the part of the period depending on the location that has been acquired by using the dummy dolls respectively installed at the plurality of locations, and derives the HRTF of the user U at each of the locations on a bases of a synthesized signal.

Accordingly, the acoustic information creating apparatus <NUM> is capable of deriving the HRTF of the user U at a plurality of locations desired by the user U while reducing the burden on the user U, although the accuracy is slightly lower than that in the case of the actual measurements.

Further, the acoustic information creating apparatus <NUM>, for example, asks the user U to take a photo of the user's ear and transmit image data, so as to be capable of estimating and deriving the HRTF of the user U on the basis of the image data of the ear. In such a case, when the image data including the image of the ear is input, the acoustic information creating apparatus <NUM> utilizes a learning model that is machine-learned to output the HRTF corresponding to the ear to derive the HRTF of the user U.

Accordingly, the acoustic information creating apparatus <NUM> is capable of estimating and deriving the HRTF of the user without asking the user U to come to the location to measure the HRTF. Therefore, a further reduction in the burden on the user U for the HRTF measurement is enabled.

Returning to <FIG>, the acoustic information <NUM> will be described continuously. The location included in the acoustic information <NUM> is identification information for identifying the location that the user U that has been registered beforehand desires to view while listening to the acoustic content <NUM>. The VR video is identification information for identifying the VR entire spherical video corresponding to the location included in the acoustic information <NUM>.

The acoustic parameters are respectively associated with numerical values indicating a reverberation characteristic of a reverberation period or the like and an echo characteristic of a reflection coefficient or the like of an audio wave for every one of a plurality of audio output positions at each location. The acoustic information creating apparatus <NUM> measures the actual acoustics at each location, and derives the acoustic parameters on the basis of the acoustics that have been collected. Accordingly, the acoustic information creating apparatus <NUM> is capable of deriving accurate acoustic parameters corresponding to the actual location. It is to be noted that in <FIG>, data of each item is conceptually represented as "A01" or "B01". However, in reality, specific data corresponding to each item is stored in the data of each item.

The acoustic information creating apparatus <NUM> transmits the acoustic information <NUM>, which has been created, to the acoustic processing apparatus <NUM>. The acoustic processing apparatus <NUM> stores the acoustic information <NUM>, which has been received from the acoustic information creating apparatus <NUM>, in the storage unit <NUM>. It is to be noted that here, the description has been given with regard to the case where the acoustic information creating apparatus <NUM> creates the acoustic information <NUM>. However, the acoustic processing apparatus <NUM> may include a similar function and configuration as the acoustic information creating apparatus <NUM>, so that the acoustic processing apparatus <NUM> may create the acoustic information <NUM> to be stored in the storage unit <NUM>.

Returning to <FIG>, the controller <NUM> will be described. The controller <NUM> includes, for example, a microcomputer including a central processing unit (CPU), a read only memory (ROM), a random access memory (RAM), an input and output port, and various circuits.

The controller <NUM> includes an acquisition unit <NUM>, a processing unit <NUM>, and a providing unit <NUM>, which function by the CPU performing various programs (corresponding to an example of an acoustic processing program according to an embodiment) stored in the ROM, by using the RAM as a work area.

It is to be noted that the acquisition unit <NUM>, the processing unit <NUM>, and the providing unit <NUM> included in the controller <NUM> may partially or wholly include hardware such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

The acquisition unit <NUM>, the processing unit <NUM>, and the providing unit <NUM> each achieve or perform actions of an information process to be described below. It is to be noted that an internal configuration of the controller <NUM> is not limited to the configuration shown in <FIG>, and may have another configuration as long as the configuration performs the information process to be described later.

The acquisition unit <NUM> acquires, for example, a provision request for the video content <NUM> and the acoustic content <NUM> from the user U. Further, in a case of acquiring the provision request for the video content <NUM> and the acoustic content <NUM> from the user U, the acquisition unit <NUM> acquires information regarding the location to be viewed by a user who listens to the acoustic content <NUM> from the user terminal <NUM>.

For example, in a case of acquiring the provision request for the video content D1 of the movie and the acoustic content D2 of the movie from the user terminal <NUM> and acquiring the information indicating a movie theater as the location information from the user terminal <NUM>, the acquisition unit <NUM> outputs the information that has been acquired from the user terminal <NUM> to the processing unit <NUM>.

In a case where the information that has been acquired from the user terminal <NUM> is input from the acquisition unit <NUM>, the processing unit <NUM> converts the acoustic characteristics of the acoustic content <NUM> into acoustic characteristic in accordance with the location corresponding to the location information that has been acquired from the user terminal <NUM> on the basis of the acoustic information <NUM>, and outputs the acoustic characteristics that have been subject to conversion to the providing unit <NUM>.

In this situation, the processing unit <NUM> applies the HRTF of the user U for every user U to convert the acoustic characteristics of the acoustic content <NUM>. Accordingly, the processing unit <NUM> is capable of converting the acoustic characteristics of the acoustic content <NUM> so as to make optimum acoustic characteristics for the user U. The processing unit <NUM> outputs the information that has been acquired from the user terminal <NUM> to the providing unit <NUM> together with the acoustic content <NUM> that has been subject to the acoustic characteristic conversion.

The providing unit <NUM> transmits to the user terminal <NUM>, the acoustic content <NUM> input from the processing unit <NUM>, the VR entire spherical video of the location corresponding to the location information, and the video content <NUM> for which the provision request has been made by the user.

Accordingly, for example, as shown in <FIG>, the acoustic processing apparatus <NUM> is capable of allowing the user U to visually recognize not only the video content D1 of the movie projected on the screen but also the surrounding environments of the screen such as the audience seats, the walls, the ceiling, and the like of the movie theater.

Then, at the same time, the acoustic processing apparatus <NUM> is capable of allowing the user to listen to the acoustic content D2 of the movie, in which the acoustic characteristics have been converted into the acoustic characteristics of the movie theater. Therefore, the acoustic processing apparatus <NUM> is capable of giving a realistic feeling to the user U who views and listens to the movie video content at home, for example, as if the user U were watching a movie in the movie theater.

It is to be noted that heretofore, the case where the acquisition unit <NUM> acquires the provision request for the video content <NUM> and the acoustic content <NUM> from the user terminal <NUM> has been described. However, the acquisition unit <NUM> acquires an image of a staying location of the user U together with the provision request for the acoustic content <NUM> from the user terminal <NUM>, in some cases.

In such cases, the processing unit <NUM> converts the acoustic characteristics of the acoustic content <NUM> in accordance with the staying location of the user U, the providing unit <NUM> transmits the acoustic content <NUM> that has been subject to conversion to the user terminal <NUM>, and the user terminal <NUM> regenerates the sound field of the acoustic content <NUM>. Such operation examples of the acoustic processing apparatus <NUM> will be described later with reference to <FIG> and <FIG>.

In addition, the acquisition unit <NUM> acquires the provision request for the acoustic information <NUM> of the acoustic content <NUM> and the VR entire spherical video of the location in which the sound field of the acoustic content <NUM> is regenerated, from a creator who has created the acoustic content <NUM>, in some cases. An operation example of the acoustic processing apparatus <NUM> in such cases will be described later with reference to <FIG>.

Next, an operation example of the acoustic processing apparatus <NUM> will be described with reference to <FIG> are explanatory diagrams of operation examples of the acoustic processing apparatus <NUM> according to the present disclosure.

As shown in <FIG>, the acquisition unit <NUM> of the acoustic processing apparatus <NUM> acquires, for example, an image Pic1 in which the staying location of user U (here, the inside of a vehicle C) has been captured by the user terminal <NUM> when the user U stays inside the vehicle C, and the provision request for the acoustic content <NUM>, in some cases.

In such cases, the acoustic processing apparatus <NUM> predicts the acoustic characteristics of the staying location from the image Pic1 of the staying location, converts the acoustic characteristics of the acoustic content <NUM> into the acoustic characteristics that have been predicted, and causes the user terminal <NUM> to regenerate the sound field. For example, the processing unit <NUM> of the acoustic processing apparatus <NUM> conducts image recognition of the image Pic1, and determines that the staying location of the user U is a space inside the vehicle C.

Then, the processing unit <NUM> estimates a length L in a front-rear direction, a length W in a lateral direction, and a length H in a height direction in the vehicle from the image Pic1 to predict the size of the space inside the vehicle, and predicts acoustic characteristics such as an echo characteristic, a reverberation characteristic, and the like of the space inside the vehicle on the basis of the size of the space inside the vehicle.

Subsequently, the processing unit <NUM> converts the acoustic characteristics of the acoustic content <NUM> into the acoustic characteristics that have been predicted, and for example, arranges a virtual speaker SpC at a center position on the front side and virtual speakers SpL and SpR at positions respectively apart from the center by <NUM> degrees to the left and right in the vehicle.

Then, the processing unit <NUM> converts the acoustic characteristics of the acoustic content <NUM> so that the sounds can be heard as if they were outputting from the three virtual speakers PcC, SpL, and SpR, and outputs the acoustic characteristics that have been subject to conversion to the providing unit <NUM>. The processing unit <NUM> causes the providing unit <NUM> to transmit the acoustic content <NUM> to the user terminal <NUM>.

Accordingly, the acoustic processing apparatus <NUM> is capable of giving a realistic feeling to the user U, for example, as if the user U were listening to the acoustic content <NUM> with a high-quality car audio, when the user U listens to the acoustic content <NUM> with an earphone.

Further, as shown in <FIG>, the acquisition unit <NUM> acquires an image Pic2 in which the staying location has been captured by the user U when the user U stays in a living room at home, and a provision request for the video content <NUM> and the acoustic content <NUM>, in some cases.

Further, in this situation, the acquisition unit <NUM> acquires information indicating that, for example, the user U has selected a predetermined area A (here, an area surrounding a television Tv) including a visual field center of the user U from the image Pic2, in some cases.

In such cases, the processing unit <NUM> arranges virtual speakers Sp1, Sp2, Sp3, Sp4, Sp5, and Sp6, which correspond to the audio output positions of the acoustic content <NUM>, so as to surround the predetermined area A. Then, the processing unit <NUM> converts the acoustic characteristics of the acoustic content <NUM>, and outputs the acoustic characteristics that have been subject to conversion to the providing unit <NUM> so that the sounds can be heard as if they were outputting from the virtual speakers Sp1, Sp2, Sp3, Sp4, Sp5, and Sp6.

The providing unit <NUM> transmits to the user terminal <NUM>, the video content <NUM>, for which the provision request has been made by the user U, and the acoustic content <NUM>, the acoustic characteristics of which have been subject to conversion by the processing unit <NUM>, and causes the user terminal <NUM> to display the video content <NUM> and to regenerate the sound field of the acoustic content <NUM>.

Accordingly, in a case where the user U listens to the acoustic content <NUM> with the earphone, the acoustic processing apparatus <NUM> is capable of giving the user U a realistic feeling as if the user U were listening to the acoustic content <NUM> with a high-quality audio device, while displaying the video content <NUM> on the television Tv.

In addition, in this situation, for example, as shown in <FIG>, the user U is viewing and listening to an animation product Vd on the television Tv, in some cases. In such cases, the acoustic processing apparatus <NUM> is also capable of displaying augmented reality (AR: Augmented Reality) images Ca, Cc, and Cd of characters appearing in the product Vd around the user on a head-mounted display Cb. Accordingly, the acoustic processing apparatus <NUM> is capable of further enhancing the realistic feeling given to the user U.

It is to be noted that here, the staying location of the user is predicted from the image that has been captured by the user U. However, this is an example. The acquisition unit <NUM> is also capable of acquiring, for example, position information of the user U, the position of which is measured by the global positioning system (GPS) included in the user terminal <NUM>.

In such a case, the processing unit <NUM> predicts the staying location of the user from the position information of the user that has been acquired by the acquisition unit, converts the acoustic characteristics of the acoustic content <NUM> into the acoustic characteristics of the staying location that has been predicted, and regenerates the sound field. Accordingly, the processing unit <NUM> is capable of converting the acoustic characteristics of the acoustic content <NUM> into the acoustic characteristics in accordance with an accurate staying location that has been measured by the GPS.

In addition, the acquisition unit <NUM> is also capable of acquiring an image that has been selected from the images that had been captured by the user U in the past from the user terminal <NUM> or an image that has been viewed by the user U via the communication network N.

In such a case, the processing unit <NUM> predicts the acoustic characteristics of a location appearing in the image that has been acquired by the acquisition unit <NUM>, converts the acoustic characteristics of the acoustic content into the acoustic characteristics that have been predicted, and regenerates the sound field. Accordingly, the acoustic processing apparatus <NUM> gives the user U a realistic feeling, for example, as if the user U were listening to the acoustic content <NUM> at a location of memories that the user U has visited in the past or a favorite location appearing in the image that the user U has viewed in the past.

Further, the processing unit <NUM> changes the number of the virtual speakers to be arranged as the audio output positions of the acoustic content <NUM> and the audio output characteristics of the virtual speakers in accordance with the size of the space that is predicted from the image that the user U is staying. For example, the processing unit <NUM> increases the number of the virtual speakers to be arranged, as the size of the space to be predicted becomes larger.

Further, in a case where the size of the space to be predicted becomes further larger, the processing unit <NUM> arranges, for example, a virtual speaker having audio output characteristics such that the acoustic content <NUM> can be heard from a <NUM>-degree direction like a surround speaker. Accordingly, the acoustic processing apparatus <NUM> is capable of causing the user terminal <NUM> to regenerate an optimum sound field in accordance with the size of the staying location of the user U.

Further, the acquisition unit <NUM> acquires, for example, a provision request for the acoustic information <NUM> of the acoustic content <NUM> and the VR entire spherical video of the location in which the sound field of the acoustic content <NUM> is regenerated, from the creator who has created the acoustic content <NUM>, in some cases.

In such cases, as shown in <FIG>, the processing unit <NUM> of the acoustic processing apparatus <NUM> causes the providing unit <NUM> to transmit the acoustic content <NUM>, the acoustic information <NUM>, and the VR video information <NUM>, for which the provision request has been made for the creator terminal <NUM>, to the creator terminal <NUM> used by a creator CR.

Accordingly, the creator CR is able to change, for example, the acoustic information <NUM> on the basis of the creator's own creative intention, while viewing the VR entire spherical video Vr of the movie theater included in the VR video information <NUM>. For example, in the present state, the creator CR is able to change the acoustic information <NUM>, which can be heard as if virtual speakers Sp were arranged on both sides of the screen of a movie theater, to the acoustic information <NUM>, which can be heard as if the virtual speakers Sp were arranged further away from both sides of the screen.

Further, the creator CR is able to change, for example, to the acoustic information <NUM>, which can be heard as if a new virtual speaker SpU were arranged above the screen and a new virtual speaker SpD were arranged below the screen. In this situation, the creator CR listens to the acoustic content <NUM> to which the creator's own HRTF has been applied, and changes the acoustic information <NUM>.

Then, the creator CR transmits acoustic information 34a, an acoustic content 32a, and the VR video information <NUM>, which have been changed, from the creator terminal <NUM> to the acoustic processing apparatus <NUM>. The acoustic processing apparatus <NUM> causes the storage unit <NUM> to store the acoustic information 34a, the acoustic content 32a, and the VR video information <NUM>, which have been received from the creator terminal <NUM>.

Accordingly, when providing the acoustic content 32a to the user U next time, the acoustic processing apparatus <NUM> is capable of regenerating the sound field of the acoustic content 32a with the acoustic characteristics on which the creative intention of the creator CR has been reflected. In this situation, the acoustic processing apparatus <NUM> provides the user U with the acoustic content 32a to which the HRTF of the user U has been applied, and is capable of regenerating the sound filed of the acoustic content 32a having the optimum acoustic characteristics for the user U.

Next, an example of a process performed by the controller <NUM> of the acoustic processing apparatus <NUM> will be described with reference to <FIG> are flowcharts showing an example of the process performed by the controller <NUM> of the acoustic processing apparatus <NUM> according to the present disclosure.

When the controller <NUM> of the acoustic processing apparatus <NUM> acquires a provision request for a content including acoustics and a video from the user U of the user terminal <NUM>, the controller <NUM> performs the process shown in <FIG>. Specifically, when the controller <NUM> acquires the provision request for the content from the user U, the controller <NUM> first acquires the content desired by the user U and location information from the user U (step S101).

Subsequently, the controller <NUM> converts the acoustic characteristics of the acoustic content corresponding to the content desired by the user into the acoustic characteristics in accordance with the acoustic characteristics corresponding to the location information (step S102). Then, the controller <NUM> provides the user U with the video content, the acoustic content in which the acoustic characteristics have been subject to conversion, and the VR entire spherical video to be visually recognized by the user U when the acoustic content and the video content are regenerated, so as to regenerate the sound field (step S103), and ends the process.

Further, the controller <NUM> performs the process shown in <FIG>, when acquiring the provision request for the acoustic content from the user and the image of the staying location of the user that has been captured by the user. Specifically, when the controller <NUM> acquires the provision request for the content and the captured image from the user U, the controller <NUM> first predicts the acoustic characteristics of a space appearing in the captured image (step S201).

Subsequently, the controller <NUM> converts the acoustic characteristics of the acoustic content corresponding to the content desired by the user U into the acoustic characteristics of the space predicted in step S201 (step S202). Then, the controller <NUM> provides the user U with the acoustic content in which the acoustic characteristics have been subject to conversion, regenerates the sound field (step S203), and ends the process.

Further, in a case where the predetermined area A in the captured image is selected by the user, the controller <NUM> performs a process of arranging the audio output position of the acoustic content so as to surround the predetermined area A, and provides the user U with the acoustic content.

In this situation, the controller <NUM> changes the number of audio output positions and audio output characteristics of the acoustic content to be arranged in accordance with the size of the predetermined area A selected by the user, and provides the user U with the acoustic content.

It is to be noted that also when acquiring from the user U the provision request for the content including the acoustics and the video, and the captured image in which the staying location of the user appears, the controller <NUM> is capable of predicting the acoustic characteristics of the space appearing in the captured image, and providing the user U with the acoustic content in which the acoustic characteristics have been converted into the predicted ones.

Further, the controller <NUM> performs the process shown in <FIG>, when acquiring from the creator CR of the acoustic content, for example, the acoustic information of the acoustic content and the provision request for the location, in which the sound field of the acoustic content is regenerated.

Specifically, when acquiring the provision request for the acoustic information and the location from the creator CR, the controller <NUM> first provides the creator CR with the acoustic information and the VR entire spherical video of the location, in which the sound field of the acoustic content corresponding to the acoustic information is regenerated (step S301).

Subsequently, the controller <NUM> determines whether or not the changed acoustic information has been acquired from the creator CR (step S302). Then, when the controller <NUM> determines that the acoustic information has not been acquired from the creator CR (steps S302, No), the controller <NUM> repeats the determination process of step S302 until the acoustic information is acquired.

Then, when it is determined that the acoustic information has been acquired from the creator CR (step S302, Yes), the controller <NUM> stores the acoustic characteristics that have been acquired in association with the VR entire spherical video that has been provided for the creator CR (step S303), and ends the process.

Claim 1:
An acoustic processing apparatus (<NUM>) comprising:
an acquisition unit (<NUM>) configured to:
acquire a provision request for a video content (<NUM>) and a corresponding acoustic content (<NUM>) from a user terminal (<NUM>) of a user;
acquire information regarding a viewing location of the user from the user terminal (<NUM>);
a storage unit (<NUM>) configured to:
store the video content (<NUM>) and the acoustic content (<NUM>);
store a plurality of virtual reality entire spherical videos (<NUM>) captured at a plurality of locations, wherein each of the virtual reality entire spherical videos (<NUM>) includes an image of a screen for displaying the video content and an image of a surrounding environment of the screen;
store a plurality of pieces of acoustic information (<NUM>) each indicating acoustic parameters for regenerating a sound field of the acoustic content (<NUM>) at a respective one of the plurality of locations;
a processing unit (<NUM>) configured to:
convert an acoustic characteristic of the acoustic content (<NUM>) to an acoustic characteristic of the viewing location of the user by applying the acoustic parameters of the one of plurality of pieces of acoustic information (<NUM>) whose location corresponds to the acquired information regarding the viewing location of the user; and
a providing unit (<NUM>) configured to:
transmit to the user terminal (<NUM>) the virtual reality entire spherical video (<NUM>) corresponding to the acquired information regarding the viewing location of the user, the converted acoustic content (<NUM>) generated by the processing unit (<NUM>) and the video content (<NUM>) to cause the user terminal (<NUM>) to display the video content (<NUM>) on the screen in the virtual reality entire spherical video (<NUM>) corresponding to the acquired information regarding the viewing location of the user and to regenerate the sound field of the acoustic content (<NUM>) in accordance with the acoustic characteristic of the viewing location of the user.