Patent ID: 12262192

DESCRIPTION OF EMBODIMENTS

Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the appended drawings.

Note that, in this specification and the appended drawings, structural elements that have substantially the same function and structure are denoted with the same reference numerals, and repeated explanation of these structural elements is omitted.

Hereinafter, a mode for carrying out the present technology will be described. The description will proceed in the following order.1. First embodiment2. Second embodiment3. Third embodiment4. Fourth embodiment5. Modified example6. Fifth embodiment7. Sixth embodiment8. Example executed by software

1. First Embodiment

<Overview of the Present Disclosure>

An overview of an advertisement presentation system of an embodiment of the present disclosure will be described with reference toFIG.1.

An advertisement presentation system1ofFIG.1is a system which is also referred to as a digital signage and presents an advertisement constituted by an image and a sound to a pedestrian walking on a passage or the like.

As illustrated inFIG.1, the advertisement presentation system1includes an information processing device11, a sensor12, display units13-1to13-3, and a multi-speaker system14, and the display units13-1to13-3and the multi-speaker system14are installed along a passage15. Further, in a case where it is unnecessary to specify each of the display units13-1to13-3, it is referred to simply as a display unit13, and other configurations are similarly referred to.

The sensor12detects a position of a pedestrian who walks on the passage and is a listener who listens to (views) an advertisement and outputs the detected position of the pedestrian to the information processing device11. The sensor12is, for example, a camera that images the passage, and captures an image of the position of the pedestrian on the passage and outputs the captured image to the information processing device11as a detection result. Further, the sensor12may be a sensor other than a camera as long as it is a sensor for detecting the position of the pedestrian located on the passage, and, for example, an area light-shielded with walking of the pedestrian may be detected as the position of the pedestrian through a sensor that detects a position on the basis of a weight associated with the passage of the pedestrian or a sensor including a light projecting unit that projects a laser beam onto the passage and a light receiving unit. Here, the following description will proceed under the assumption that the sensor12captures an image of the passage15and outputs the captured image to the information processing device11.

The information processing device11detects a position, a moving speed, and a moving direction of the pedestrian in the passage on the basis of the image captured by the sensor12, estimates a motion of the pedestrian in the future of a predetermined time, causes an advertisement sound which the pedestrian can hear as an appropriate sound to be output from the multi-speaker system14at an estimated position estimated at a predetermined timing in the future of a predetermined time, and causes an advertisement image to be displayed in the display unit13closest to an estimated position.

The display units13-1to13-3are displays constituted by a liquid crystal display (LCD) or an organic electro luminescence (EL) and are installed along the passage15on which the pedestrian walks. Further, inFIG.1, the example in which the three display units13-1to13-3are installed as the display unit13is illustrated, but the number of display units13may not be three. Further, the number of display units13may be one, and the advertisement image may be displayed in an area on the display unit13closest to the estimated position at a predetermined timing.

The multi-speaker system14is constituted by a plurality of speakers, and causes a sound to be output as if there are sound sources at various positions including before and after a surface on which a speaker is installed through wave field synthesis using each speaker. Here, the wave field synthesis is a process of adjusting a sound output from a plurality of speakers, forming a spatial sound field by changing a position of a virtual sound source of a sound variously, and transmitting the spatial sound field.

As illustrated in a left portion ofFIG.2, in a case where a listener H1hears a sound output from the multi-speaker system14through wave field synthesis, for example, the multi-speaker system14can output a sound to be heard as if a sound source SS1is located at a position in front of a speaker surface of the multi-speaker system14through the wave field synthesis.

Further, as illustrated in a middle portion ofFIG.2, in a case where the listener H1hears the sound output from the multi-speaker system14through wave field synthesis, for example, the multi-speaker system14can output a sound to be hard as if a sound source SS2moves while drawing an S-shaped path SL ahead of or behind the speaker surface of the multi-speaker system14through the wave field synthesis.

Further, as illustrated in a right portion ofFIG.2, in a case where the listener H1hears the sound output from the multi-speaker system14through wave field synthesis, for example, the multi-speaker system14can output a sound in which individual sounds Sd1to Sd3are heard by listeners H11to H13without being mixed when the listeners H11to H13are lined up to face the speaker surface in front of the speaker surface of the multi-speaker system14through the wave field synthesis.

Further, although not illustrated, the multi-speaker system14can also output a sound as if there is a sound source at a position behind the speaker surface of the multi-speaker system14.

In other words, if a position of the pedestrian who is the listener within a space can be specified, the multi-speaker system14can cause an advertisement sound to which any one of three types of effects described above with reference toFIG.2or an effect of a combination thereof is added to be heard through the wave field synthesis. Further, a “sound” includes a sound other than a human voice such as a music or a sound effect.

Next, an overview of an operation of the advertisement presentation system1will be described with reference toFIG.1.

As illustrated in an upper portion ofFIG.1, the information processing device11detects a position, a moving speed, and a moving direction of the pedestrian H1at a time t1on the basis of the image captured by the sensor12. Then, as illustrated in a lower portion ofFIG.1, the information processing device11estimates a position, a moving speed, and a moving direction of the pedestrian H1at a time t2which is the future of a predetermined time as a motion on the basis of the detection result. Further, the motion of the pedestrian H1estimated here indicates a change in the position of the pedestrian H1which is assumed chronologically and does not indicate an action of holding and opening hands or an action of opening and closing the feet.

On the basis of the detection result of the pedestrian H1at the time t1and the motion estimation result of the pedestrian H1at the time t2, the information processing device11plans a sound reproduction schedule indicating which content is to be output by which effect among content serving as an advertisement sound to be output from the multi-speaker system14at each time between the times t1and t2. Further, at this time, the information processing device11plans an image reproduction schedule indicating which content is to be displayed at which position among content serving as an advertisement image to be displayed in the display units13-1to13-3.

The information processing device11controls the multi-speaker system14on the basis of the planned sound reproduction schedule such that the advertisement sound is output so that the advertisement sound is appropriately heard at each position in accordance with the motion of the pedestrian H1who is the moving listener at each time.

In other words, the information processing device11controls the multi-speaker system14such that a sound which the pedestrian H1(t11) located at a position at which the pedestrian H1is estimated to pass at a time t11can hear as an appropriate sound can be output through the wave field synthesis.

At the same time, the information processing device11controls the display units13-1to13-3on the basis of the planned image reproduction schedule such that the advertisement image is displayed so that the advertisement image can be viewed at each position in accordance with the motion of the pedestrian H1who is the moving listener at each time.

In other words, the information processing device11controls the display unit13-2such that an image which the pedestrian H1located at a position at which the pedestrian H1(t11) is estimated to pass at the time t11can view as an appropriate image can be displayed through the display unit13-2, for example.

Accordingly, even when the pedestrian H1moves along the passage15, the pedestrian H1can hear the advertisement sound appropriately and further view the advertisement image in accordance with the motion of the pedestrian H1.

As a result, it is possible to the advertisement including the sound and the image to be appropriately recognized in accordance with the motion of moving pedestrian H1.

<Hardware Configuration Example of Information Processing Device>

Next, a hardware configuration example of the information processing device11will be described with reference to a hardware block diagram ofFIG.3.

For example, the information processing device11is configured as a controller for the display unit13and the multi-speaker system14, and includes a control unit31, an input unit32, an output unit33, a storage unit34, a communication unit35, a drive36, and a removable storage medium37which are electrically connected to one another via a bus38.

The control unit31is constituted by a processor and a memory, and controls an overall operation of the information processing device11. Further, the control unit31includes an audio processing unit51and an image processing unit52, and controls an output of an advertisement sound and display of an advertisement image.

The input unit32is constituted by a keyboard, an operation button, or the like, and receives an operation input of a user and outputs the operation input to the control unit31via the bus38.

The output unit33outputs an image to the display unit13constituted by a display such as a liquid crystal display (LCD), an organic electro luminescence (EL), or the like so that the image is displayed and outputs a sound signal to the multi-speaker system14so that a sound is output.

The storage unit34is constituted by a hard disk drive (HDD), a solid state drive (SSD), a semiconductor memory, or the like and is controlled by the control unit31and writes or reads various types of data and programs.

The communication unit35is controlled by the control unit31and performs transmission and reception of various types of data and programs with an external server or the like (not illustrated) via a network represented by a local area network (LAN) or the like in a wired or wireless manner.

The drive36reads or writes data from or in a removable storage medium37such as a magnetic disk (including a flexible disk), an optical disk (including a compact disc-read only memory (CD-ROM), a digital versatile disc (DVD), a Blu-ray (a registered trademark) disc, and an archival disc), a magneto-optical disk (including a mini disc (MD)), or a semiconductor memory.

<Functional Configuration Example Realized by Information Processing Device>

Next, functions realized by the information processing device11will be described with reference to a functional block diagram ofFIG.4.

The functions realized by the information processing device11are realized by the audio processing unit51and the image processing unit52in the control unit31. Further, audio data111for an advertisement, image data112for an advertisement, and subtitle data113are stored in the storage unit34.

The audio processing unit51controls the output of the advertisement sound by controlling the multi-speaker system14using the audio data111for the advertisement stored in the storage unit34.

The image processing unit52controls the display of the advertisement image by controlling the display unit13using the image data112for the advertisement and the subtitle data113stored in the storage unit34.

The audio processing unit51includes a motion estimating unit71, a reproduction schedule generating unit72, an attribute estimating unit73, an audio data acquiring unit74, a sound field synthesis parameter generating unit75, a sound field synthesizing unit76, and an N-channel amplifying unit77.

The motion estimating unit71detects the position, the moving speed, and the moving direction of the pedestrian on the passage who is the listener on the basis of the image of the passage captured by the sensor12. Then, the motion estimating unit71estimates the chronological position of the pedestrian in the future of a predetermined time on the basis of the detected position, the moving speed, and the moving direction of the pedestrian on the passage as the motion, and outputs a motion estimation result to the reproduction schedule generating unit72.

The reproduction schedule generating unit72generates a reproduction schedule of the advertisement sound on the basis of information of the chronological position of the pedestrian in the future of a predetermined time in the motion estimation result and outputs the reproduction schedule of the advertisement sound to the audio data acquiring unit74and the sound field synthesis parameter generating unit75and an image reproducing unit91and a subtitle reproducing unit92of the image processing unit52.

For example, as illustrated inFIG.5, in a case where a central position of the pedestrian is detected at a position P1from the image captured at a time t1by the sensor12, a position at a time t2in the future of a predetermined time is estimated, for example, as a position P2in a predetermined direction from the position P1on the basis of the moving speed and the moving direction at the time t1.

In this case, it is estimated that the central position of the pedestrian moves while drawing a locus obtained by connecting the positions P1and P2by a straight line as indicated by a straight line L ofFIG.5. Here, the slope of the straight line L ofFIG.5is a slope corresponding to the moving speed.

Therefore, the position chronologically moving on the straight line L at the times t1and t2is estimated as the motion. When the motion estimation result is indicated by the straight line L ofFIG.5, the reproduction schedule generating unit72generates a reproduction schedule of outputting the advertisement sound to which any one of the effects described with reference toFIG.2or an effect of a combination thereof is added so that the pedestrian can hear the advertisement sound appropriately at a position corresponding to each time on the straight line L. For this reason, the reproduction schedule can be regarded as a reproduction scenario of the sound and the image. Further, the reproduction schedule here is also a reproduction schedule of causing the advertisement image which the pedestrian can view appropriately to be displayed at the position of the pedestrian corresponding to each time.

Therefore, the reproduction schedule in which, in a case where the pedestrian moves along the straight line L, the advertisement sound to which any one of the effects described with reference toFIG.2or an effect of a combination thereof is added is output at a corresponding position P11at the time t11out of the times t1and t2so that the pedestrian who is the listener can hear it appropriately, and the advertisement image is displayed at the corresponding position so that the pedestrian can view an appropriate image is generated.

The attribute estimating unit73is constituted by a recognizers or the like, and estimates an attribute into which a situation of the pedestrian (listener) hearing (viewing) the advertisement is classified on the basis of the image or the sound captured by the sensor12and outputs an attribute estimation result to then audio data acquiring unit74, the image reproducing unit91, and the subtitle reproducing unit92.

Further, a type of attribute indicating the situation of the pedestrian classified in the attribute estimating unit73is information which can be recognized on the basis of at least one of an image or a sound and is, for example, information indicating the presence or absence of a pedestrian, for example.

Further, in a case where there is a pedestrian, the type of attribute indicating the situation of the pedestrian includes, for example, information of an analysis result such as individual identification information of a pedestrian, a male, a female, an age, a nationality, a speaking language, one man, one woman, a group of a plurality of men, a group of a plurality of women, a group of mixed men and women, and a family which is recognized by analysis using at least one of the image or the sound.

Further, the type of attribute indicating the situation of the pedestrian includes, for example, information of an analysis result indicating a facial expression of a pedestrian, an emotion estimated from a facial expression or the like of a pedestrian, or the like, and a state of a behavior of a pedestrian (including standing still, walking, running, or a gesture) which is recognized by analysis using at least one of the image or the sound.

Further, type of attribute indicating the situation of the pedestrian includes, for example, information used in non-verbal cues or the like such as a facial expression, a facial color, a line of sight, a body gesture, a hand gesture, a body posture, how to put a physical distance with a counterpart, clothes, a hair style, breathing, a voice tone, or a voice quality. Further, the type of classification of the attribute indicating the situation of the pedestrian here is an example, and other types of attributes may be set. Further, the attribute estimating unit73may be realized by a recognizer having a learning function, an artificial intelligence (AI), or the like.

On the basis of the reproduction schedule supplied from the reproduction schedule generating unit72and the attribute estimation result supplied from the attribute estimating unit73, the audio data acquiring unit74accesses the storage unit34, acquires the audio data111for the corresponding advertisement, and outputs the acquired audio data111to the sound field synthesis parameter generating unit75.

The audio data111for the advertisement is audio data constituted by a plurality of types of content corresponding to the attribute such as a male, a female, an age group, a nationality, a group of men, a group of women, a group of mixed men and women, and a family. Therefore, the audio data acquiring unit74acquires the audio data (content) for the advertisement corresponding to the estimated attribute which is specified in the reproduction schedule, and outputs the acquired audio data to the sound field synthesis parameter generating unit75.

On the basis of the reproduction schedule and the audio data111supplied from the audio data acquiring unit74, the sound field synthesis parameter generating unit75generates a sound field synthesis parameter necessary for synthesizing the sound field, and outputs the sound field synthesis parameter to the sound field synthesizing unit76.

The sound field synthesizing unit76controls the N-channel amplifying unit77on the basis of the sound field synthesis parameter supplied from the sound field synthesis parameter generating unit75such that it is output to the multi-speaker system14as sound.

The N-channel amplifying unit77controls each of speakers of N channels constituting the multi-speaker system14in accordance with a control signal supplied from the sound field synthesizing unit76such that a sound is output. With a series of processes described above, the wave field synthesis for realizing the effect specified by the reproduction schedule is realized, and the advertisement sound to which the effects described with reference toFIG.2are added is output.

The image processing unit52includes the image reproducing unit91, the subtitle reproducing unit92, and a subtitle imposing unit93.

On the basis of the reproduction schedule supplied from the reproduction schedule generating unit72and the attribute estimation result supplied from the attribute estimating unit73, the image reproducing unit91accesses the storage unit34, reads and reproduces the image data112for the corresponding advertisement, and outputs the image data112for the corresponding advertisement to the subtitle imposing unit93.

The image data112for the advertisement is the image data112including a plurality of types of content corresponding to the attributes such as a male, a female, an age group, a nationality, a group of men, a group of women, a group of mixed men and women, and a family. Therefore, the image reproducing unit91responds to a predetermined timing in the reproduction schedule, reads and reproduces the image data112of the content for the advertisement corresponding to the attribute as the attribute estimation result, and outputs the image data112to the subtitle imposing unit93.

On the basis of the reproduction schedule supplied from the reproduction schedule generating unit72and the attribute estimation result supplied from the attribute estimating unit73, the subtitle reproducing unit92accesses the storage unit34and reads and reproduces the subtitle data113, and outputs the subtitle data113to the subtitle imposing unit93.

The subtitle data113for the advertisement is the subtitle data113including a plurality of types of content corresponding to the attributes such as a male, a female, an age group, a nationality, a group of men, a group of women, a group of mixed men and women, and a family. For this reason, the subtitle reproducing unit92reads and reproduces the subtitle data113of the content for the advertisement corresponding to the attribute serving as the attribute estimation result which is the subtitle data113corresponding to a predetermined timing in the reproduction schedule, and outputs the subtitle data113to the subtitle imposing unit93.

The subtitle imposing unit93imposes an image of a subtitle which is reproduced and output from the subtitle reproducing unit92on an image which is reproduced and output from the image reproducing unit91and outputs an imposition result to the target display unit13as an image.

<Output Process>

Next, an output process of the sound and the image for the advertisement in a first embodiment will be described with reference to a flowchart ofFIG.6.

In step S11, the sensor12captures the image of the passage and outputs the captured image to the audio processing unit51.

In step S12, the attribute estimating unit73of the audio processing unit51estimates the attribute of the pedestrian on the basis of the captured image, and outputs the attribute estimation result to the audio data acquiring unit74and the image reproducing unit91and subtitle reproducing unit92of the image processing unit52.

In step S13, the motion estimating unit71detects the position, the moving speed, and the moving direction of the pedestrian on the basis of the image captured by the sensor12, estimates the position of the pedestrian in the future of a predetermined time as a motion, and outputs the motion estimation result to the reproduction schedule generating unit72.

In step S14, the reproduction schedule generating unit72determines whether or not the supplied motion estimation result is different from an immediately previous motion estimation result. In other words, in a case where there is no significant change from the immediately previous motion estimation result, it is desirable to use the reproduction schedule generated on the basis of the immediately previous motion estimation result without change, and the reproduction schedule need not be generated again. To this end, the reproduction schedule generating unit72obtains a difference with the immediately previous motion estimation result, compares the difference with a predetermined threshold value, and determines whether or not the supplied motion estimation result is different from an immediately previous motion estimation result by determining whether or not the difference is larger than the threshold value or not. Further, since there is no immediately previous motion estimation result in a first process, the process proceeds while assuming that it is different from the immediately previous motion estimation result.

In a case where it is determined in step S14that it is the first process or the difference with the immediately previous motion estimation result is larger than the predetermined threshold value, and it is different from the immediately previous motion estimation result, the process proceeds to step S15.

In step S15, the reproduction schedule generating unit72generates a schedule of causing an effect to be applied to a sound and a sound in a state in which it can be appropriately heard as a sound to which an effect is applied to be output at the position at each time on the moving path estimated on the basis of the motion estimation result of the pedestrian who is the listener. Then, the reproduction schedule generating unit72outputs the information of the generated reproduction schedule to the audio data acquiring unit74, the sound field synthesis parameter generating unit75, and the image reproducing unit91and the subtitle reproducing unit92of the image processing unit52.

In step S16, the control unit31determines whether or not it is a next reproduction start time in the reproduction schedule. Here, for example, since, in the reproduction schedule, a timing to start reproduction of content of a sound and an image of a predetermined advertisement is set by a time, it is determined whether or not it is the reproduction start time of the next content.

In a case where it is determined in step S16that it is the next content reproduction start time in the reproduction schedule, the process proceeds to step S17.

In step S17, the audio data acquiring unit74accesses the storage unit34, acquires the audio data111of the content which starts to be reproduced and corresponds to the attribute serving as the attribute estimation result, and outputs the audio data111to the sound field synthesis parameter generating unit75.

In step S18, the sound field synthesis parameter generating unit75generates the sound field synthesis parameter for synthesizing the sound filed in which the sound which can be appropriately heard by the pedestrian who is the listener can be output through the wave field synthesis through the wave field synthesis on the basis of the reproduction schedule, and outputs the sound field synthesis parameter to the sound field synthesizing unit76together with the audio data111.

In step S19, the sound field synthesizing unit76synthesizes the sound field by synthesizing the audio data111for each channel through a convolutional operation on the basis of the sound field synthesis parameter and the audio data111, and outputs the sound field to the N-channel amplifying unit77.

In step S20, the N-channel amplifying unit77amplifies the audio data111synthesized for each channel, and outputs the audio data111to each speaker of the multi-speaker system14as the sound signal.

In step S21, the multi-speaker system14outputs the sound on the basis of the sound signal supplied from the N-channel amplifying unit77.

In other words, with a series of processes in steps S19to S21, the sound in which the sound field is synthesized (wave field synthesized) is output from the multi-speaker system14.

In step S22, the image reproducing unit91accesses the storage unit34and reads the image data112of the content which starts to be reproduced and corresponds to the attribute serving as the attribute estimation result.

In step S23, the image reproducing unit91reproduces the image on the basis of the read image data112and outputs the image to the subtitle imposing unit93.

In step S24, the subtitle reproducing unit92accesses the storage unit34and reads the subtitle data113of the content which starts to be reproduced and corresponds to the attribute serving as the attribute estimation result.

In step S25, the subtitle reproducing unit92reproduces the subtitle on the basis of the read subtitle data113and outputs the subtitle to the subtitle imposing unit93.

In step S26, the subtitle imposing unit93imposes the subtitle on the reproduced image.

In step S27, the subtitle imposing unit93outputs the image obtained by imposing the subtitle to the display unit13so that the image is displayed.

In step S28, it is determined whether or not an end instruction is given, and in a case where no end instruction is given, the process returns to step S11, and the subsequent processes are repeated. Then, in a case where an end instruction is given in step S28, the process ends.

Further, in a case where it is determined in step S16that it is not the next content reproduction start time in the reproduction schedule, the process of steps S17to S27is skipped.

With the above process, even when the pedestrian who is the listener continuously moves on the passage15, the pedestrian can continuously hear the advertisement sound at each position in accordance with the motion in a state in which the effect added to the audio data111can be appropriately recognized through the wave field synthesis.

Further, even when the pedestrian who is the viewer continuously moves, the pedestrian can continuously view the advertisement image displayed on the nearby display unit13which the pedestrian is passing at an appropriate timing at each position in accordance with the motion.

Further, the sound and the image (including the subtitle) presented to the pedestrian who is the listener are content of the sound and the image (including the subtitle) corresponding to the attribute of the pedestrian, and thus it is possible to cause an easily acceptable advertisement to be presented to the pedestrian who is the listener.

As a result, since it is possible to present the advertisement including the sound and the image constituted by an optimum type of content to the pedestrian who is the listener in accordance with the motion of the pedestrian, it is possible to cause an advertisement effect to be improved.

Further, it is possible to provide an advertisement including an effective sound content to the listener who is an advertisement target in a digital signage or the like.

Further, in a case where it is possible to change a language and provide an advertisement if it is possible to recognize a nationality or the like as an attribute, it is possible to improve an advertisement effect and a guidance convenience.

Further, it is possible to automatically present an advertisement using a sensor device such as a camera as the sensor12without intervention of the user, and it is possible to realize an effective advertisement and guidance corresponding to the target listener together with labor saving.

The example of presenting the advertisement to the target person who is the pedestrian on the passage15has been described, but it may be other places as long as there is a pedestrian or an advertisement target person.

Further, in the above description, a case where there is only one pedestrian has been described, but the pedestrian may be a group of a plurality of pedestrians. In the case of a group of a plurality of pedestrians, a group of men only, a group of women only, a group of mixed men and women, a family, or the like may be included in the attribute, and the audio data111, the image data112, and the subtitle data113corresponding to each group may be prepared, the advertisement sound corresponding to each group may be output, and an image including the subtitle may be displayed.

Further, the number of pedestrians or the number of groups of pedestrians may be two or more, and a sound and an image including content of an appropriate attribute may be presented to each of a pedestrian or a group of pedestrians at an appropriate timing in accordance with the motion by performing a similar process for each of a pedestrian or a group of pedestrians. In particular, through the wave field synthesis, since it is possible to output a sound in which the effects described with reference toFIG.2are added to content corresponding to each attribute to each of a plurality of pedestrians and a plurality of groups, it is possible to present an easily acceptable advertisement sound to a plurality of pedestrians or a plurality of groups.

Further, for the sound, the user may be allowed to set transition of a sound source using an authoring tool, and a sound to which an effect of enabling the transition of the sound source to be recognized through the wave field synthesis is added may be output.

The example in which the sound field synthesis parameter related to the wave field synthesis is generated each time in accordance with the position of the pedestrian which is the listener in a listening area, the sound field synthesis parameter and the audio data are synthesized by the convolutional operation for each channel, and the wave field synthesis is performed by using the synthesized audio data of each channel has been described above.

However, the listening area may be divided into a plurality of areas in advance, a wave field synthesis filter may be calculated in advance for each divided area and stored in a lookup table or the like, and when the position of the pedestrian in the listening area is detected, the wave field synthesis may be performed by reading the wave field synthesis filter of the area corresponding to the detected position and performing the filter process.

When the wave field synthesis filter is prepared in advance as described above, it is possible to reduce a computation amount and perform the process related to the wave field synthesis at a high speed. In particular, in a case where it is possible to enable a plurality of pedestrians to individually hear audio data of different languages, if a computation is performed each time, the process is likely to be delayed if a certain number of persons is exceeded, and it is possible to suppress the occurrence of delay by preparing the wave field synthesis filter.

Further, in the above process, the imposed subtitle of the pedestrian determined by the attribute of the pedestrian is displayed on display unit13together with the advertisement image, and thus even in a case where an advertisement is unable to be appropriately provided only with sound content, it is possible to reliably present the advertisement.

2. Second Embodiment

<Example of Outputting Sound and Displaying Image for Each Area>

The example of outputting the advertisement sound and displaying the image in accordance with the position of the pedestrian in the passage has been described above, but in a case where a plurality of pedestrians or a plurality of pedestrian groups are walking, there is a limitation to the number of individual persons or the number of groups to be processed individually. In this regard, the passage may be divided into a plurality of areas, and the output of the advertisement sound and the display of the image may be switched for each area in accordance with the attribute of the pedestrian or the group of pedestrians located in each area. Further, in a case where there are a plurality of groups or a plurality of pedestrians in the same area, content having a priority is decided in advance, and the content is presented. For example, content is assumed to be presented in order of English, Chinese, and Japanese in the descending order of priorities.

For example, a case where a group G1of pedestrians, a group G2of pedestrians, and a pedestrian H1are located at positions indicated in upper portions ofFIGS.7to9at times t100to t101, times t101to t102, and times t102to t103is considered. Further, it is assumed that the passage is divided into five areas Z1to Z5, and the illustration of the display unit13is omitted, but the display unit13is assumed to be located at the position corresponding to each of the areas Z1to Z5as well. Further, the attribute estimating unit73is assumed to classify the group G1of the pedestrians, the group G2of pedestrians, and the pedestrian H1into attributes indicating a group of pedestrians whose speaking language is English, a group whose speaking language is Chinese, and a pedestrian whose speaking language is Japanese.

In other words, in the times t100to t101, it is recognized on the basis of the image captured by the sensor12that the group G1is located over the areas Z1and Z2, the group G2is located over the area Z4and Z5, and the pedestrian H1is located in the area Z3as illustrated in the upper portion ofFIG.7. Further, on the basis of the image captured by the sensor12, a motion is estimated from the moving speed and the moving direction of each of the groups G1and G2and the pedestrian H1, and the position at the times t102and t103is estimated.

Further, in the times t100to t101, on the basis of the arrangement of the groups G1and G2and the pedestrian H1illustrated in the upper portion ofFIG.7, the multi-speaker system14outputs a sound of content of English to the areas Z1and Z2as the advertisement sound for the group G1, outputs a sound of content of Chinese to the areas Z4and Z5as the advertisement sound for the group G2, and outputs a sound of Japanese to the area Z3as the advertisement sound for the pedestrian H1as illustrated in a lower portion ofFIG.7.

Further, in times t101to t102, it is assumed to be estimated on the basis of an estimation result that the group G1is located over the areas Z2and Z3, the group G2is located over the areas Z4and Z5, and the pedestrian H1is located in the area Z3, for example, as illustrated in an upper portion ofFIG.8.

In this regard, in the times t101to t102, on the basis of the arrangement of the groups G1and G2and the pedestrian H1illustrated in the upper portion ofFIG.8, the multi-speaker system14outputs a sound of content of English to the area Z2as the advertisement sound for the group G1, outputs a sound of content of Chinese to the areas Z4and Z5as the advertisement sound for the group G2, and outputs a sound of Japanese to the area Z3as the advertisement sound for the pedestrian H1in accordance with the set priority as illustrated in a lower portion ofFIG.8.

Further, both the group G1whose speaking language is English and the pedestrian H1whose speaking language is Japanese are located in the area Z3ofFIG.8, but here, the sound of the content of English having a higher priority than Japanese is output to the areas Z2and Z3. As the sounds are output as described above, in the pedestrians located in the areas Z2and Z3, the advertisement can be presented to the group G1of pedestrians having English as the speaking language which are overwhelmingly in the majority, and thus the advertisement effect can be increased.

Further, in times t102to t103, it is assumed to be estimated on the basis of an estimation result that the group G1is located in the area Z4, the group G2is located in the area Z5, and the pedestrian H1is located in the area Z2, for example, as illustrated in an upper portion ofFIG.9.

In this regard, in the times t102to t103, on the basis of the arrangement of the groups G1and G2and the pedestrian H1illustrated in the upper portion ofFIG.9, the multi-speaker system14outputs a sound of content of English to the area Z4as the advertisement sound for the group G1, outputs a sound of content of Chinese to the area Z5as the advertisement sound for the group G2, and outputs a sound of content of Japanese to the area Z2as the advertisement sound for the pedestrian H1as illustrated in a lower portion ofFIG.9.

For example, a reproduction schedule illustrated inFIG.10is planned from the results of the times t101to t103described with reference toFIGS.7to9.

As illustrated inFIG.10, in the times t100to t101, as described above with reference toFIG.7, the sound of content of English is output in the areas Z1and Z2, the sound of content of Japanese is output in the area Z3, and the sound of content of Chinese is output in the areas Z4and Z5.

Further, in the times t101to t102, as described above with reference toFIG.8, the sound of content of English is output in the areas Z2and Z3, and the sound of content of Chinese is output in the areas Z4and Z5.

Further, in the times t102to t103, as described above with reference toFIG.9, the sound of content of Japanese is output in the area Z2, the sound of content of English is output in the area Z4, and the sound of content of Chinese is output in the area Z5.

Further,FIG.10illustrates an example in which, in the times t103to t105, the sound of content of Japanese is output in the areas Z1and Z2, and the sound of content of English is output in the areas Z3to Z5.

Further, although not illustrated, a corresponding image is displayed on the display unit13in a corresponding area, similarly to the sound. Further, inFIG.10, for example, only BGM may be reproduced in an area in which the reproduction schedule is not set for content of any of English, Chinese, and Japanese.

As described above, a sound corresponding to a category of the pedestrian or the group of pedestrians located in each of areas classified into a plurality of areas Z1to Z5may be output on the basis of the motion estimation result based on the position, the moving speed, and the moving direction of the pedestrian or the group of pedestrians.

<Switching Corresponding to Length of Content>

Further, the example in which, in the area in which the estimated pedestrian or the group of pedestrians are located, the output of the content of the sound of a corresponding attribute and the display of the image are performed for each area, but content of an advertisement sound having a different length is switched and output in accordance with a length of time in which the pedestrian or the group of pedestrians are estimated to be located in the area (listening area) to which the sound is output, and the image may be displayed.

In other words, for example, a case where the group G1is located in the area Z1, the pedestrian H1is located in the area Z5, and the group G1and the pedestrian H1are moving in moving directions indicated by arrows at the same speed as illustrated in an upper portion ofFIG.11is considered.

For example, it is assumed to be estimated that, in times t110to t111, the group G1is located over the areas Z1and Z2, and the pedestrian H1is not located in any of the areas Z1to Z5, in time t111to t112, the group G1is located over the areas Z2and Z3, and the pedestrian H1is not located in the area Z5, in time t112to t113, the group G1is located in the areas Z3and Z4, and the pedestrian H1is not located in the area Z4, and in time t113to t115, both the group G1and the pedestrian H1are located in the vicinity of the boundary between the areas Z3and Z4.

In this case, a reproduction schedule illustrated in a lower part ofFIG.11is planned. In other words, in the areas Z1to Z3, in times t110to t113, a sound including content E-M of English in which the length of the reproduction time is the times t110to t113(or the length of time taken for the group G1to pass the areas Z1to Z3) is output, and a corresponding image is displayed.

Further, in the areas Z4to Z5, in the times t111to t113, a sound including content JS of Japanese in which the length of the reproduction time is the times t112to t113(or the length of time taken for the pedestrian H1to pass the areas Z5and Z4) is output, and a corresponding image is displayed.

Further, in times t113to t115, the group G1and the pedestrian H1are located in the vicinity of the boundary between the areas Z3and Z4. In this state, if the sound of the content including any language is output, either the pedestrian H1or the group G1of pedestrians is unlikely to be able to be recognized. In this regard, in the times t113to t115, a sound including content Ec-L of English set as a common language and having a length of the times t113to t115is output, and a corresponding image is displayed.

As a result, the group G1and the pedestrian H1can completely hear (view) content of a corresponding attribute having an appropriate length until they are in close to each other, and further, content set as a common language is reproduced after they are located near each other.

Further, in a case where the number of pedestrians or the number of groups in the passage15exceeds a predetermined number or in a case where the number of attributes to be classified is increased, a processing load for individually reproducing the content is increased. In this regard, in this case, only the content set as the common language may be reproduced regardless of the number of pedestrians or the number of attributes. As a result, even in a case where the number of pedestrians, the number of groups, or the number of attributes to be classified is large, the processing load related to reproduction can be reduced.

<Output Process in a Case where Reproduction Schedule is Set for Each Area>

Next, an output process in a case where the reproduction schedule is set for each area will be described with reference to the flowchart ofFIG.12.

In step S31, the sensor12captures the image of the passage and outputs the image to the audio processing unit51.

In step S32, the attribute estimating unit73of the audio processing unit51estimates the attribute of the pedestrian or the group of pedestrians on the basis of the captured image, and outputs the attribute estimation result to the audio data acquiring unit74and the image reproducing unit91and the subtitle reproducing unit92of the image processing unit52.

In step S33, the motion estimating unit71detects the position, the moving speed, and the moving direction of each of the pedestrian or the group of pedestrians on the basis of the image captured by the sensor12, estimates the position of each of the pedestrian or the group of pedestrians in the future of a predetermined time, and outputs the motion estimation result to the reproduction schedule generating unit72.

In step S34, the reproduction schedule generating unit72determines whether or not the supplied motion estimation result of the pedestrian or the group of pedestrians is different from the immediately previous motion estimation result.

In a case where it is determined in step S34that it is the first process or the difference with the immediately previous motion estimation result is larger than the predetermined threshold value, and it is different from the immediately previous motion estimation result, the process proceeds to step S35.

In step S35, the reproduction schedule generating unit72generates a schedule of causing a sound which can be appropriately heard in each area as illustrated inFIGS.7to11to be reproduced on the basis of the position at each time on the estimated moving path of the pedestrian who is the listener or the group of pedestrians on the basis of the motion estimation result. Then, the reproduction schedule generating unit72outputs the information of the generated reproduction schedule to the audio data acquiring unit74, the sound field synthesis parameter generating unit75, and the image reproducing unit91and the subtitle reproducing unit92of the image processing unit52.

In step S36, the control unit31determines whether or not it is a next reproduction start time in the reproduction schedule.

For example, in the case of the reproduction schedule illustrated inFIG.10, at the time t101, reproduction of the content of Japanese and reproduction of the content of Chinese are continued in the areas Z3to Z5, and thus it is not a timing at which the next reproduction is started, but in the area Z2, but since reproduction of the content of English is newly started, the time t101is regarded as the next reproduction start time.

On the other hand, inFIG.10, at the time t104, reproduction of the content of Japanese and reproduction of the content of English are continued in all of the areas Z1to Z5, and it is not a time at which reproduction of the next content is started in any area. Therefore, the time t104is not regarded as the next reproduction start time.

In other words, in a case where it is determined in step S36that the reproduction of the next content is started at any one of all the areas, it is regarded as the next reproduction start time.

In a case where it is determined in step S36that it is the next content reproduction start time in the reproduction schedule, the process proceeds to step S37.

In step S37, the control unit31sets one of unprocessed areas as a processing target area. In other words, for example, in the example ofFIGS.7to11, any one unprocessed area among the areas Z1to Z5is set as the processing target area.

In step S38, the audio data acquiring unit74accesses the storage unit34, acquires the audio data111of the content which starts to be reproduced in the processing target area and corresponds to the attribute, and outputs the audio data111to the sound field synthesis parameter generating unit75.

The audio data111acquired here may correspond only to the attribute, and in a case where the pedestrian and the group of pedestrians having different attributes are located in the passage as described with reference toFIG.11, the audio data111of the content in which the length corresponding to the time taken until the pedestrian and the group of pedestrians having different attributes are close to each other is considered may be acquired.

Further, in a case where an area in which reproduction of new content is not started is the processing target area, the process of steps S38to S48is skipped.

In step S39, the sound field synthesis parameter generating unit75generates the parameter for synthesizing the sound filed in which the sound which can be appropriately heard by the pedestrian who is the listener in the processing target area can be output through the wave field synthesis on the basis of the reproduction schedule and the audio data111in the processing target area, and outputs the parameter to the sound field synthesizing unit76together with the audio data111.

In step S40, the sound field synthesizing unit76synthesizes the sound field by synthesizing the audio data111for each channel through a convolutional operation on the basis of the sound field synthesis parameter and the audio data111, and outputs the sound field to the N-channel amplifying unit77.

In step S41, the N-channel amplifying unit77amplifies the audio data111synthesized for each channel, and outputs the audio data111to each speaker of the multi-speaker system14as the sound signal.

In step S42, the multi-speaker system14outputs the sound to the processing target area on the basis of the sound signal supplied from the N-channel amplifying unit77.

In other words, with a series of processes in steps S38to S42, the sound in which the sound field is synthesized (wave field synthesized) is output from the multi-speaker system14.

In step S43, the image reproducing unit91accesses the storage unit34and reads the image data112of the content which starts to be reproduced in the processing target area and corresponds to the attribute serving as the attribute estimation result.

In step S44, the image reproducing unit91reproduces the image on the basis of the read image data112and outputs the image to the subtitle imposing unit93.

In step S45, the subtitle reproducing unit92accesses the storage unit34and reads the subtitle data113of the content which starts to be reproduced in the processing target area and corresponds to the attribute serving as the attribute estimation result.

In step S46, the subtitle reproducing unit92reproduces the subtitle on the basis of the read subtitle data113and outputs the subtitle to the subtitle imposing unit93.

In step S47, the subtitle imposing unit93imposes the subtitle on the reproduced image.

In step S48, the subtitle imposing unit93outputs the image obtained by imposing the subtitle to the display unit13so that the image is displayed.

In step S49, the control unit31determines whether or not there is an unprocessed area, and in a case where there is an unprocessed area, the process returns to step S37, and the processes of steps S37to S49are repeated until there is no unprocessed area. Further, in a case where it is determined in step S49that there is no unprocessed area, the process proceeds to step S50.

In step S50, it is determined whether or not an end instruction is given, and in a case where no end instruction is given, the process returns to step S31, and the subsequent processes are repeated. Then, in a case where an end instruction is given in step S50, the process ends.

With the above process, even when the pedestrian who is the listener and the group of pedestrians continuously move, the pedestrian and the group of pedestrians can continuously hear the advertisement sound at each position in accordance with the motion in a state in which the effect added to the audio data111can be appropriately recognized through the wave field synthesis in an area in which each of the pedestrian or the group of pedestrians is located.

Further, even when the pedestrian who is the viewer and the group of pedestrians continuously moves, the pedestrian and the group of pedestrians can continuously view the advertisement image displayed on the display unit13in the area in which the pedestrian or the group of pedestrians passes at an appropriate timing in accordance with the motion in an area in which each of the pedestrian or the group of pedestrians is located.

Further, since the sound and the image (including the subtitle) presented to the pedestrian who is the listener are content of the sound and the image (including the subtitle) corresponding to the attribute of the pedestrian, it is possible to cause an easily acceptable advertisement to be presented to the pedestrian who is the listener and the group of pedestrians.

Further, even when the pedestrian and the group of pedestrians having different attributes are located in the passage, since it is possible to switch and output the sound of the content corresponding to the category and the length corresponding to the time taken until the pedestrian and the group of pedestrians having different attributes are close to each other and switch and display the image, a situation in which the advertisement content is unable to be heard (viewed) in the middle is prevented even when the pedestrian and the group of pedestrians move.

In any case, as a result, it is possible to present the advertisement constituted by the sound and the image including an appropriate type of content for each area in accordance with the motion of the pedestrian in accordance with the attribute of the pedestrian or the position relation with a nearby pedestrian having a different attribute.

3. Third Embodiment

<Adjustment of Volume Corresponding to Distance to Multi-Speaker System>

The example in which the multi-speaker system14outputs the advertisement sound through the wave field synthesis in accordance with the motion estimated on the basis of the position, the moving speed, and the moving direction of the pedestrian or the group of pedestrians detected on the basis of the image captured by the sensor12has been described, but a volume curve corresponding to a distance from the sound source generated through the wave field synthesis may be set, and the volume may be adjusted in accordance with the moving distance.

In other words, as illustrated inFIG.13, in a case where there is a listener at each of positions A, B, and C from the sound source, the level of the volume changes in accordance with the distance from the sound source as indicated by curves LA, LB, and LC.

InFIG.13, a curve indicating a decrease in volume corresponding to a distance when it gets away from the sound source in a case where the listener is located at the position A at which a distance from a position of “0” on the left side in the drawing is about 10 m is indicated by the curve LA. Further, inFIG.13, a curve indicating a decrease in volume corresponding to a distance when it gets away from the sound source in a case where the listener is located at the position B at which a distance from a position of “0” is about 20 m is indicated by the curve LB. Further, inFIG.13, a curve indicating a decrease in volume corresponding to a distance when it gets away from the sound source in a case where the listener is located at the position C at which a distance from a position of “0” is about 30 m is indicated by the curve LC.

In other words, in a case where the sound source gets away from a state in which it is located at a position at which a distance from the position A indicated by a position X is about 5 m to a position O at which the distance is about 10 m when viewed from the position A, the volume at which the listener located at the position A changes from about −14 dB to about −20 dB and decreases by −6 dB, along the curve LA.

However, in a case where the sound source gets away from a state in which it is located at a position at which a distance from the position B indicated by the position X is about 15 m to a position O at which the distance is about 20 m when viewed from the position B, the volume at which the listener located at the position B changes from about −23 dB to about −26 dB and decreases by −3 dB, along the curve LB.

Further, in a case where the sound source gets away from a state in which it is located at a position at which a distance from the position C indicated by the position X is about 25 m to a position O at which the distance is about 30 m when viewed from the position C, the volume at which the listener located at the position C changes from about −28 dB to about −30 dB and decreases by −2 dB, along the curve LC.

Further, in general, if the distance is doubled, the volume decreases by −6 dB, but since it is limited to a case where an ideal condition with no sound reflection or the like is set, the decrease in the volume corresponding to the distance can be changed in accordance with a parameter setting. Therefore, the graph ofFIG.13is merely an example. Further, the example in a case where the distance between the sound source and the listener increases has been described above, but in a case where the distance decreases, the volume changes to increase.

Since a changing distance and an attenuating volume are different in accordance with the distance between the sound source and the listener as described above, the volume is attenuated on the basis of the volume curve illustrated inFIG.13, and thus it is possible to give a more appropriate sense of depth.

<Output Process in a Case where Attenuation of Volume Corresponding to Distance is Indicated>

Next, an output process in a case where the attenuation of the volume corresponding to the distance is indicated will be described with reference to a flowchart ofFIG.14. Further, processes of steps S71to S79and steps S81to S91in the flowchart ofFIG.14are similar to the processes described with reference to the flowchart ofFIG.12, and description thereof will be omitted.

In other words, in step S80, on the basis of the reproduction schedule and the audio data111in the processing target area, the sound field synthesis parameter generating unit75generates a parameter for adjusting the volume of the sound to be heard by the pedestrian which is the listener in the processing target area in accordance with the distance from the sound source in accordance with the distance to the multi-speaker system14described with reference toFIG.13and outputs the parameter to the sound field synthesizing unit76.

With this process, in step S81, the sound field synthesizing unit76synthesizes the sound field by synthesizing the audio data111for each channel through the convolutional operation on the basis of the sound field synthesis parameter and the audio data111, and outputs the sound field to the N-channel amplifying unit77. At this time, the sound field synthesizing unit76synthesizes a sound field in which the volume is adjusted by synthesizing the audio data111for each channel through the convolutional operation in consideration of a volume parameter.

In step S82, the N-channel amplifying unit77amplifies the audio data111synthesized for each channel whose volume is adjusted in accordance with the distance, and outputs the resulting audio data to each speaker of the multi-speaker system14as a sound signal.

Accordingly, even when the pedestrian who is the listener or the group of pedestrians continuously moves, the pedestrian or the group of pedestrians can hear the wave field synthesized sound appropriately in a state in which the volume is adjusted in accordance with the distance between the pedestrian and the sound source realized by the wave field synthesis in an area in which each of the pedestrian or the group of pedestrians is located.

As a result, it is possible to continuously hear the advertisement sound with a sense of depth.

Further, the example in which the volume is appropriately adjusted in accordance with the distance between the pedestrian and the sound source has been described, but the reverberations may be adjusted in accordance with the distance between the pedestrian and the sound source by setting similar curves. Since the reverberations are adjusted, it is possible to hear the advertisement sound with a sense of depth.

4. Fourth Embodiment

<Example of Giving Individual Instructions to Plurality of Listeners Simultaneously>

The example of outputting the advertisement sound and displaying the advertisement image to the pedestrian walking on the passage has been described above, but, for example, the multi-speaker system14may be installed in a dance studio or the like instead of the passage, and individual instructions may be given to a plurality of dancers dancing in the studio as listeners.

In other words, for example, a configuration in which, as illustrated inFIG.15, the multi-speaker system14is placed in a dance studio or the like, an area in front of the multi-speaker system14is divided into areas Z11to Z13, and dancer H11to H13are dancing as listeners is considered.

In such a configuration, a cooperative operation as a whole may be realized by conveying sounds with different instruction contents to the dancers H11to H13dancing in the areas Z11to Z13through the wave field synthesis.

In other words, for example, as illustrated inFIG.15, the wave field synthesis may be performed so that the dancer H11dancing in the area Z11can hear a sound of an instruction “dance like swan,” the wave field synthesis may be performed so that the dancer H12dancing in the area Z12can hear a sound of an instruction “step out right foot,” and the wave field synthesis may be performed so that the dancer H13dancing in the area Z13can hear a sound of an instruction “raise left hand.”

In the case of the example ofFIG.15, since the instructions are instructions given to the respective dancers H11to H13based on the image captured by the sensor12, for example, even in a case where there is a change from a state in which the dancer H11dances in the area Z11, and the dancer H12dances in the area Z12as illustrated in an upper portion ofFIG.16to a state in which the dancer H12dances in the area Z11, and the dancer H11dances in the area Z12as illustrated in a lower portion ofFIG.16, sounds of appropriate instructions are output to the respective dancers H11and H12.

In other words, the reproduction schedule is generated from the information of the moving speed and the moving direction of the dancers H11and H12on the basis of the image captured by the sensor12, and thus sounds of instructions to be given to the respective dancers H11and H12can be switched and output at a timing at which the areas in which the dancers H11and H12are located are switched between Z11and Z12.

Further, although the display unit13is not illustrated inFIGS.15and16, in each area of the areas Z11to Z13in the studio, the display units13which can be viewed by the dancers H11to H13dancing in the respective areas may be installed, and images corresponding to instruction contents of individual sounds output through the wave field synthesis may be displayed.

<Output Process in Case where Individual Instructions are Given to Plurality of Listeners Simultaneously>

Next, an output process in a case where individual instructions are output to a plurality of listeners simultaneously will be described with reference to a flowchart ofFIG.17. Further, in the flowchart ofFIG.17, a process in a case where the display units13that display images which can be individually viewed by the dancers dancing in the areas Z11to Z13are installed, and images corresponding to respective instruction contents are also displayed will be described.

In step S111, the sensor12captures an image of the dance studio in which the dancers are dancing and outputs the image to the audio processing unit51.

In step S112, the attribute estimating unit73of the audio processing unit51estimates, as an attribute, a correspondence between each dancer in the image and any one of the dancers H11to H13on the basis of the captured image, and outputs an attribute estimation result to the audio data acquiring unit74and the image reproducing unit91and the subtitle reproducing unit92of the image processing unit52.

In step S113, the motion estimating unit71detects the position, the moving speed, and the moving direction of each dancer on the basis of the image captured by the sensor12, estimates the position of the dancer in the future of a predetermined time as a motion, and outputs a motion estimation result to the reproduction schedule generating unit72.

In step S114, the reproduction schedule generating unit72determines whether or not the supplied motion estimation result of each dancer is different from the immediately previous motion estimation result.

In a case where it is determined in step S114that it is the first process or the difference with the immediately previous motion estimation result is larger than the predetermined threshold value, and it is different from the immediately previous motion estimation result, the process proceeds to step S115.

In step S115, the reproduction schedule generating unit72generates a schedule of causing an appropriately audible sound and an appropriately viewable image to be reproduced for each dancer on the basis of the position at each time on the estimated moving path of the dancer which is the listener on the basis of the motion estimation result. Then, the reproduction schedule generating unit72outputs the information of the generated reproduction schedule to the audio data acquiring unit74, the sound field synthesis parameter generating unit75, and the image reproducing unit91and the subtitle reproducing unit92of the image processing unit52.

In step S116, the control unit31determines whether or not it is a next content reproduction start time in the reproduction schedule.

In a case where it is determined in step S116that it is the next content reproduction start time in the reproduction schedule, the process proceeds to step S117.

In step S117, the control unit31sets any one of unprocessed dancers as the processing target person. Hereinafter, the dancer is also referred to as a target person.

In step S118, the audio data acquiring unit74accesses the storage unit34, and acquires the audio data111of the instruction content associated with the attribute of the processing target person, that is, information indicating any one of the dancers H11to H13, and outputs the audio data111to the sound field synthesis parameter generating unit75. Since the attribute includes information identifying the dancer who is the processing target person, the audio data111of the content instructed to each identified dancer is read out.

In step S119, the sound field synthesis parameter generating unit75generates the sound field synthesis parameter for synthesizing the sound filed in which the sound which can be appropriately heard by the processing target person who is the listener can be output through the wave field synthesis on the basis of the reproduction schedule of the dancer who is the processing target person, and outputs the sound field synthesis parameter to the sound field synthesizing unit76together with the audio data111.

In step S120, the sound field synthesizing unit76synthesizes the sound field by synthesizing the audio data111for each channel through a convolutional operation on the basis of the sound field synthesis parameter and the audio data111, and outputs the sound field to the N-channel amplifying unit77.

In step S121, the N-channel amplifying unit77amplifies the audio data111synthesized for each channel, and outputs the audio data111to each speaker of the multi-speaker system14as the sound signal.

In step S122, the multi-speaker system14outputs the sound to the position at which the dancer who is the processing target person is located on the basis of the sound signal supplied from the N-channel amplifying unit77.

In other words, with a series of processes from step S118to step S122, the sound in which the sound field is synthesized (wave field synthesized) with the instruction content to be given to any one of the dancers H11to H13who are the processing target persons is output from the multi-speaker system14.

In step S123, the image reproducing unit91accesses the storage unit34and reads the image data112of the instruction content associated with the attribute of the processing target person, that is, the information indicating any one of the dancers H11to H13. The image data112of the content instructed to the dancer identified as the processing target person is read out.

In step S124, the image reproducing unit91reproduces the read image data112and outputs it to the subtitle imposing unit93.

In step S125, the subtitle reproducing unit92accesses the storage unit34and reads the subtitle data113of the instruction content associated with the attribute of the processing target person, that is, the information indicating any one of the dancers H11to H13. The subtitle data113of the content instructed to the dancer identified as the processing target person is read out.

In step S126, the subtitle reproducing unit92reproduces the read subtitle data113and outputs it to the subtitle imposing unit93.

In step S127, the subtitle imposing unit93imposes the subtitle on the reproduced image.

In step S128, the subtitle imposing unit93outputs the image obtained by imposing the subtitle to the display unit13so that the image is displayed.

In step S129, the control unit31determines whether or not there is an unprocessed target person (dancer), and in a case where there is an unprocessed target person, the process returns to step S117, and the processing of steps S117to S129is repeated until there is no unprocessed target person. Further, in a case where it is determined in step S129that there is no unprocessed target person, the process proceeds to step S130.

In step S130, it is determined whether or not an end instruction is given, and in a case where no end instruction is given, the process returns to step S111, and the subsequent processes are repeated. Then, in a case where an end instruction is given in step S130, the process ends.

Further, in a case where it is determined in step S116that it is not the next content reproduction start time in the reproduction schedule, the process of steps S117to S129is skipped.

With the above process, it is possible to simultaneously output sounds of individual instructions to a plurality of dancers and display the image.

As a result, a plurality of dancer can perform a group cooperative action by receiving individual instructions and dancing, and it is possible to realize a group cohesive dance.

Further, since it is sufficient as long as a group behavior is specified, it may be used for giving an instruction in actions other than a dance, and for example, an appropriate instruction may be presented to each member by a sound and an image when a team in which a plurality of members cooperate different actions is formed.

5. Modified Example

<Example of Visually Presenting Audible Content to Listener>

The example in which the position, the moving speed, and the moving direction of the listener are recognized to estimate the motion of the listener on the basis of the image captured by the sensor12, the reproduction schedule of each previously set area or each target person is set on the basis of the motion estimation result, and the audible sound is output to the listener of each area or each target person through the wave field synthesis has been described above, but the audible content may be presented to each area to be visually recognized.

For example, a state in which the multi-speaker system14outputs, for example, a sound of Japanese to an audio reproduction area AR1, outputs, for example, a sound of English to an audio reproduction area AR2, and outputs, for example, a sound of Chinese to an audio reproduction area AR3in accordance with the reproduction schedule as illustrated inFIG.18is assumed.

Here, a projector-like area presenting device121may be installed above the multi-speaker system14so that, for example, patterns or the like which enable respective areas to be recognized as areas in which sound guidance of Japanese, English, and Chinese are output are projected onto floors of areas corresponding to the audio reproduction areas AR1to AR3.

As a result, the listener can recognize in which area the listener can currently hear the sound of what language.

Further, the area presenting device121may have other configurations as long as the listener can recognize which language is output at which position, and, for example, as illustrated inFIG.19, an LED illumination may be arranged above the multi-speaker system14in combination with the arrangement of each speaker, and light of a different color may be emitted in each of the audio reproduction areas AR1to AR3corresponding to a type in which the language output from the multi-speaker system14is output.

For example, as illustrated inFIG.19, a light of a different color may be emitted in each of areas121ato121cin which LEDs constituting the area presenting device121are arranged to which the audio reproduction areas AR1to AR3correspond so that the areas in which the sounds of different languages are output are visually recognized.

In other words, in this case, the listener can recognize a language in which the sound is output in each of the corresponding audio reproduction areas AR1to AR3since each of the areas121ato121cof the LEDs constituting the area presenting device121emits a color.

Further, in a case where the display unit13is installed in association with each area, information corresponding to a language in which the sound is output in a corresponding area may be presented in the corresponding display unit13.

Further, the operation and the control of the area presenting device121may be realized by controlling a projection position or controlling color emission in accordance with a type of audio data supplied to each of the speakers constituting the multi-speaker system14. Therefore, since the control of the area presenting device121is substantially the same as the control of the multi-speaker system14, description thereof is omitted.

6. Fifth Embodiment

<Output Process Corresponding to Attribute of Listener>

The example of presenting the advertisement including the sound and the image to the pedestrian who is the listener moving on the passage in accordance with the motion or outputting the instructions individually to a plurality of listeners at the same time has been described above, but the output may be performed in accordance with the attribute regardless of the change in the motion or the position of the pedestrian who is the listener.

Here, an output process corresponding to the attribute of the listener will be described with reference to a flowchart ofFIG.20. Further, since the processes of steps S151to S164in the flowchart ofFIG.20are the same as the processes of steps S11, S12, and S17to S28in the flowchart ofFIG.6, description of the processes is omitted.

In other words, in the output process described with reference to the flowchart ofFIG.20, in step S152, the presence/absence of a pedestrian, individual identification information of a pedestrian, a male, a female, an age, a nationality, a speaking language, one man, one woman, a group of a plurality of men, a group of a plurality of women, a group of mixed men and women, and a family, a facial expression of a pedestrian, an emotion of a pedestrian, a state of a behavior (including standing still, walking, running, a gesture, or the like), or the like is estimated as the attribute on the basis of images which are repeatedly captured.

Then, in steps S153, S158, and S160, the audio data111, the image data112, and the subtitle data113based on the attribute estimation result are read and output as the sound from the multi-speaker system14and the image from the display unit13.

However, in the process illustrated inFIG.20, since the motion of the pedestrian, that is, the estimation result related to the change in the position is unable to be obtained, the sound is output in a case where the sound source is at a specific position, and an operation of following the change in the motion or the position of the pedestrian is not performed.

However, since it is possible to output the content of the sound and the image corresponding to the attribute of the pedestrian, the sound and the image (including the subtitle) presented to the pedestrian who is the listener are the content of the sound and the image (including subtitles) corresponding to the attribute of the pedestrian, and thus it is possible to present an easily acceptable advertisement to the pedestrian who is the listener.

As a result, it is possible to present the advertisement constituted by the sound and the image including an optimum type of content to the pedestrian who is the listener in accordance with the motion of the pedestrian, and thus, it is possible to improve the advertisement effect.

7. Sixth Embodiment

<Example of Reproducing Sound and Image of Live Hall in Reproduction Hall>

The example of presenting the advertisement including the sound and the image to the pedestrian who is the listener moving on the passage in accordance with the motion or outputting the instructions individually to a plurality of listeners at the same time has been described above, but a sound and an image of a live hall may be output in other reproduction halls.

For example, as illustrated inFIG.21, a sensor12′ performs imaging (including sound recording) of a live show performed by an artist HR on a stage St11of a live hall LH, and information of the captured live image and the recorded live sound is transmitted to the information processing device11. Further, the artist HR can be regarded as a speaker that generates a sound for the listener who hears the sound.

The information processing device11acquires the live image and the live sound supplied from the sensor12′. Further, the information processing device11acquires information such as, for example, a building space size (w×d×h) or the number of gathered guests as data specific to the live hall LH, and acquires information such as, for example, a building space size (w1×d1×h1) or the number of gathered guests as data specific to the reproduction hall RH.

Then, the information processing device11generates the reproduction schedule of the live image and the live sound in the reproduction hall RH on the basis of a sensing result of the reproduction hall RH, the live image and the live sound which are the sensing result of the live hall LH, the data specific to the live hall LH, and the data specific to the reproduction hall RH, and generates the sound field synthesis parameter corresponding to the reproduction hall RH in accordance with the reproduction schedule.

More specifically, the information processing device11estimates the motion of the artist HR in the live hall LH on the basis of the captured image of the live hall LH, and converts a motion estimation result of the artist HR in live hall LH into a motion estimation result in the reproduction hall RH on the basis of the data specific to the live hall LH and the data specific to the reproduction hall RH. The information processing device11generates the reproduction schedule of the live image and the live sound in the reproduction hall RH on the basis of the motion estimation result of the artist HR in the reproduction hall RH, and generates the sound field synthesis parameter corresponding to the reproduction hall RH in accordance with the reproduction schedule.

The information processing device11performs the wave field synthesizes on the sound recorded in the live hall LH on the basis of the sound field synthesis parameter corresponding to the generated reproduction hall RH, and outputs the sound of the live hall LH as a sound suitable for the reproduction hall RH.

At this time, the information processing device11converts a size of the image data obtained in the live hall LH into a size suitable for the display unit13installed on the stage St12of the reproduction hall RH, and displays it as the artist HV.

Accordingly, even in the reproduction hall RH, the live show in the live hall LH is realized by displaying the artist HV as if the artist HR in the live hall LH is located in the reproduction hall RH through the sound and the image heard (viewed) in the live hall LH.

Further, the information processing device11may perform imaging (including sound recording) of a situation of audience in the reproduction hall RH through the sensor12and acquire an excitement (the sound of the reproduction hall or the like) as the sensing result of the reproduction hall RH.

At this time, the information processing device11may also generate the reproduction schedule in the reproduction hall RH in consideration of the motion estimation result of the audience within the reproduction hall RH.

<Hardware Configuration Example of Information Processing Device that Outputs Sound and Image of Live Hall in Reproduction Hall>

Next, a hardware configuration example of the information processing device11that outputs the sound and the image of the live hall in the reproduction hall will be described with reference toFIG.21. Further, in the information processing device11ofFIG.21, components having the same functions as those of the information processing device11ofFIG.3are denoted by the same reference numerals, and description thereof will be omitted appropriately.

In other words, the hardware configuration example of the information processing device11ofFIG.21is different from the hardware configuration example of the information processing device11ofFIG.3in that the sensor12′ installed in the live hall LH is newly added. The sensor12′ is installed in the live hall LH, captures the image in the live hall LH, records the sound, and transmits the image and the sound to the information processing device11.

In the information processing device11ofFIG.21, the control unit31controls the communication unit35such that the image and the sound supplied from the sensor12′ are acquired, and stores audio data151(FIG.22) and image data152(FIG.22) in the storage unit34.

Further, in addition to the image and the sound, the sensor12′ also acquires the subtitle data in a case where there is subtitle data or the like, for example, and transmits the subtitle data to the information processing device11. In this case, the information processing device11acquires the supplied subtitle data and stores the subtitle data in the storage unit34as subtitle data153(FIG.22). Further, other information may be acquired as long as it is information which can be sensed, and other information may be sensed, for example, if it is information by which a state of the live hall LH can be recognized such as a humidity or a temperature of the live hall LH.

<Functional Configuration Example Realized by Information Processing Device ofFIG.21>

Next, the functions realized by the information processing device11ofFIG.21will be described with reference to the functional block diagram ofFIG.22. Further, in the functional block diagram ofFIG.22, components having the same function as the functional block diagram ofFIG.4are denoted by the same reference numerals, and description thereof will be omitted appropriately.

In other words, the functional block diagram ofFIG.22is different from the functional block diagram ofFIG.4in that, instead of the audio data111, the image data112, and the subtitle data113stored in the storage unit34, the audio data151of the live sound, the image data152of the live image, and the subtitle data153presented in the live show which are supplied from the live hall LH are stored as information from the sensor12′. Further, a difference lies in that spatial information such as live hall-specific data161such as the size of live hall LH or the number of gathered guests which is acquired in advance and reproduction hall-specific data162such as the size of reproduction hall RH or the number of gathered guests is newly stored.

Further, a motion estimating unit171, a reproduction schedule generating unit172, and an image reproducing unit191are installed instead of the motion estimating unit71, the reproduction schedule generating unit72, and the image reproducing unit91.

The motion estimating unit171detects information of the position, the moving speed, and the moving direction of the artist HR on the basis of the image data152which is the image of the live hall LH captured by the sensor12′, and estimates the position of the artist HR in the hall LH in the future of a predetermined time as the motion of the artist HR.

Further, the motion estimating unit171estimates the motion of the artist HR in the live hall LH on the basis of the image data152which is the image captured in the live hall LH, converts the motion estimation result of the motion in the live hall LH into the motion estimation result in the reproduction hall RH on the basis of the data specific to the live hall LH and the data specific to the reproduction hall RH, and outputs the motion estimation result in the reproduction hall RH to the reproduction schedule generating unit172.

More specifically, as illustrated in an upper portion ofFIG.23, the motion estimating unit171estimates a position (x, y, z) and motion information (the moving speed and the moving direction) (s, t, u) of the artist HR on the stage St11of the live hall LH as the motion in the live hall LH.

The motion estimating unit171estimates the motion in the reproduction hall RH including a position (x1, y1, z1) and motion information (the moving speed and the moving direction) (s1, t1, u1) on the stage St12in the reproduction hall RH by mapping the motion in the live hall LH on the basis of the space size (w, d, h) of the live hall LH obtained from the live hall-specific data161and a space size (w1, d1, h1) of the reproduction hall RH obtained from the reproduction hall-specific data162. In other words, the motion estimating unit171estimates the motion of the artist HR in the live hall LH on the basis of the image of the live hall LH, and converts it into the motion estimated in a case where the artist HR is located in the reproduction hall RH.

The reproduction schedule generating unit172generates a reproduction schedule for reproducing the audio data151obtained by the sensor12′ on the basis of the motion estimation result in the reproduction hall RH obtained with reference toFIG.23, and outputs the reproduction schedule to the sound field synthesis parameter generating unit75.

At this time, the reproduction schedule generating unit172also decides various effects affecting the sound on the basis of the space size (w, d, h) of the live hall LH obtained from the live hall-specific data161and the space size (w1, d1, h1) of the reproduction hall RH obtained from the reproduction hall-specific data162, generates the reproduction schedule while considering the effects as well, and outputs the reproduction schedule to the sound field synthesis parameter generating unit75.

Further, since the image of the audience in the reproduction hall RH is also captured as the image by the sensor12, and the sound is also recorded, the motion estimating unit171may estimate the motion of the audience in the reproduction hall and a motion in a case where the artist HR is located in the reproduction hall RH and output the estimated motions to the reproduction schedule generating unit172.

In this case, the reproduction schedule generating unit172may generate the reproduction schedule in consideration of the motion of the audience in the reproduction hall and the motion in a case where the artist HR is located in the reproduction hall RH.

Then, the sound field synthesis parameter generating unit75generates the sound field synthesis parameter for synthesizing the sound field for outputting the sound through the wave field synthesis on the basis of the reproduction schedule generated for the reproduction hall RH generated as described above.

With a series of processes, it is possible to realize a sound having a realistic feeling similar to that in a case where it is heard in the live hall LH in the reproduction hall RH through the wave field synthesis realized in the multi-speaker system14in the reproduction hall RH through the generated sound field synthesis parameter.

Further, the image reproducing unit191has a basic function similar to that of the image reproducing unit91, but the image reproducing unit191further converts the image so that the image data152including the live image is suitable for the size of the display unit13in the reproduction hall RH on the basis of the spatial information based on the live hall-specific data161and the reproduction hall-specific data162, reproduces the image, and outputs the image to the subtitle imposing unit93.

<Output Process in Case where Sound and Image of Live Hall are Output in Re-Production Hall>

Next, an output process in a case where the sound and the image of the live hall are output in the reproduction hall will be described with reference to a flowchart ofFIG.24.

In step S231, the control unit31controls the communication unit35such that the live hall-specific data161and the reproduction hall-specific data162are acquired and stores the live hall-specific data161and the reproduction hall-specific data162in the storage unit34. Further, the live hall-specific data161and the reproduction hall-specific data162may be input in advance, for example, by the user by operating the input unit32.

In step S232, the control unit31controls the communication unit35such that the audio data151and the image data152including the live sound and the live image which are obtained by imaging (including sound recording) by the sensor12′ installed in the live hall LH and transmitted, and stores the audio data151and the image data152in the storage unit34. Further, at this time, the control unit31may control the communication unit35such that the subtitle data153is acquired and store the subtitle data153in the storage unit34.

In step S233, the sensor12performs imaging in the reproduction hall RH (including sound recording), captures the appearance of the audience, and outputs the image to the audio processing unit51.

In step S234, the attribute estimating unit73of the audio processing unit51estimates the attribute of the audience on the basis of the image of the audience in the reproduction hall RH captured by the sensor12, and outputs the attribute estimation result to the audio data acquiring unit74and the image reproducing unit191and the subtitle reproducing unit92of the image processing unit52. Further, in this case, for example, information of the attribute of audience may be acquired as information specifying a language when the subtitle is displayed. Further, for the attribute of the audience, an attribute which is largest in number may be set as the attribute of audience among the attributes of the audience instead of the individual attribute of the audience.

In step S235, the motion estimating unit171detects the position, the moving speed, and the moving direction of the artist HR on the basis of the image data152stored in the storage unit34, and estimates the position of the artist HR in the live hall LH in the future of a predetermined time as the motion.

In step S236, the motion estimating unit171converts the motion estimation result of the artist HR in the live hall LH into the motion estimation result in the reproduction hall RH on the basis of the live hall-specific data161and the reproduction hall-specific data162as described with reference toFIG.23and outputs the motion estimation result in the reproduction hall RH to the reproduction schedule generating unit72. Further, the motion estimating unit171may also obtain the motion estimation result of the audience in the reproduction hall RH captured by the sensor12if necessary and output the motion estimation result of the audience to the reproduction schedule generating unit172.

In step S237, the reproduction schedule generating unit172determines whether or not the supplied motion estimation result of the artist HR in the reproduction hall RH is different from the immediately previous motion estimation result.

In a case where it is determined in step S237that it is the first process or the difference with the immediately previous motion estimation result of the artist HR in the reproduction hall RH is larger than the predetermined threshold value, and it is different from the immediately previous motion estimation result of the artist HR in the reproduction hall RH, the process proceeds to step S238.

In step S238, the reproduction schedule generating unit172generates a schedule of causing a sound of a sound source position corresponding to the position of the artist HR to be reproduced at each time on the moving path on the basis of the motion estimation result of the artist HR in the reproduction hall RH. Further, the audio data151is regarded as, for example, an audio module constituted by a plurality of frames reproduced at the same sound source position, and information indicating a timing at which the audio module is reproduced or an effect of loading is generated as the reproduction schedule.

Then, the reproduction schedule generating unit72outputs the information of the generated reproduction schedule to the audio data acquiring unit74, the sound field synthesis parameter generating unit75, and the image reproducing unit91and the subtitle reproducing unit92of the image processing unit52.

In step S239, the control unit31determines whether or not it is a time at which the next audio module starts to be reproduced in the reproduction schedule.

In a case where it is a time at which the next audio module starts to be reproduced in the reproduction schedule in step S239, the process proceeds to step S240.

In step S240, the audio data acquiring unit74accesses the storage unit34, acquires the audio data151including the live sound of the audio module which starts to be reproduced, and outputs the audio data151to the sound field synthesis parameter generating unit75.

In step S241, the sound field synthesis parameter generating unit75generates the sound field synthesis parameter for synthesizing the sound field in which the sound for realizing the sound made by the artist HR can be output at the position in the reproduction hall RH corresponding to the live hall LH through the wave field synthesis in the reproduction hall RH through the wave field synthesis, and outputs the sound field synthesis parameter to the sound field synthesizing unit76together with the audio data151including the live sound.

In step S242, the sound field synthesizing unit76synthesizes the sound field by synthesizing the audio data151for each channel through a convolutional operation on the basis of the sound field synthesis parameter and the audio data151including the live sound, and outputs the sound field to the N-channel amplifying unit77.

In step S243, the N-channel amplifying unit77amplifies the audio data151synthesized for each channel, and outputs the audio data151to each speaker of the multi-speaker system14as the sound signal.

In step S244, the multi-speaker system14outputs the sound on the basis of the sound signal supplied from the N-channel amplifying unit77.

In other words, with a series of processes of steps S239to S244, the sound field obtained by converting the sound imaged (recorded) in the live hall LH into a state which can appropriately cope with the space of the reproduction hall RH is output from the multi-speaker system14as the synthesized (wave field synthesized) sound.

In step S245, the image reproducing unit191accesses the storage unit34and reads the image data152which starts to be reproduced.

In step S246, the image reproducing unit191converts a size of the read image data152into a size appropriate to the display unit13in the reproduction hall RH on the basis of the live hall-specific data161and the reproduction hall-specific data162, reproduces the image data152, and outputs the image data152to the subtitle imposing unit93.

In step S247, the subtitle reproducing unit92accesses the storage unit34and reads the subtitle data153corresponding to the attribute estimation result which starts to be reproduced. Further, here, as the subtitle data153, there is subtitle data corresponding to a plurality of languages, and the subtitle data153based on the information of the nationality or the speaking language classified as the attribute of the audience is read out.

In step S248, the subtitle reproducing unit92reproduces the read subtitle data153and outputs the subtitle data153to the subtitle imposing unit93.

In step S249, the subtitle imposing unit93imposes the subtitle on the reproduced image.

In step S250, the subtitle imposing unit93outputs the image obtained by imposing the subtitle to the display unit13so that the display is displayed.

In step S251, it is determined whether or not an end instruction is given, and in a case where no end instruction is given, the process returns to step S232, and the subsequent processes are repeated. Then, in a case where an end instruction is given in step S251, the process ends. Further, in the flowchart ofFIG.24, the example in which the audio data151, the image data152, and the subtitle data153acquired in the live hall LH are temporarily stored in the storage unit34to form a packet, and the reproduction schedule is generated in real time, and it is reproduced has been described, but the audio data151, the image data152, and the subtitle data153may be stored in advance in the storage unit34, and the reproduction of the live sound or image in the live hall LH may be performed at a timing different from a timing at which a live concert or the like is being held.

With the above process, even when the audience who is the listener is in the reproduction hall RH having a size different from that of the live hall LH, the reproduction schedule is generated on the basis of the motion estimation result of the artist converted corresponding to the reproduction hall RH, and the sound field synthesis parameter is generated. The wave field synthesis is performed on the audio data151including the live sound in accordance with the sound field synthesis parameter, and thus it is possible to output the live sound having a realistic feeling in the reproduction hall RH as if the live sound is heard in the live hall LH.

As a result, the audience in the reproduction hall RH can enjoy a sound having a realistic feeling as if the audience in the live hall LH hears the live sound directly.

8. Example Executed by Software

Meanwhile, a series of processes described above can be executed by hardware and can be executed by software as well. In a case where a series of processes are executed by software, a program constituting the software is installed in a computer incorporated into dedicated hardware, for example, a general-purpose computer capable of executing various types of functions in accordance with various types of installed programs, or the like from a recording medium.

FIG.25shows a configuration example of a general-purpose computer. The personal computer includes an internal central processing unit (CPU)1001. An input/output interface1005is connected to the CPU1001via a bus1004. A read only memory (ROM)1002and a random access memory (RAM)1003are connected to the bus1004.

An input unit1006including an input device such as a keyboard or a mouse used for the user to input an operation command, an output unit1007that outputs a processing operation screen or a processing result image to a display device, a storage unit1008that is constituted by a hard disk drive that stores a program or various types of data or the like, and a communication unit1009that is constituted by a local area network (LAN) adapter or the like and executes a communication process via a network represented by the Internet are connected to the input/output interface1005. Further, a drive1010that reads or writes data from or in a removable storage medium1011such as a magnetic disk (including a flexible disk), an optical disk (including a compact disc-read only memory (CD-ROM) or a digital versatile disc (DVD)), a magneto-optical disk (including a mini disc (MD)), or a semiconductor memory is connected.

The CPU1001executes various types of processes in accordance with a program which is stored in the ROM1002or a program which is read from the program removable storage medium1011such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, installed in the storage unit1008, and loaded from the storage unit1008onto the RAM1003. Data or the like necessary for the CPU1001to execute various types of processes is also stored in the RAM1003appropriately.

In the computer having the above configuration, when the CPU1001loads, for example, the program stored in the storage unit1008onto the RAM1003via the input/output interface1005and the bus1004and executes the program, a series of processes described above is performed.

For example, the program executed by computer (CPU1001) can be provided in a form in which it is recorded in the removable storage medium1011as a package medium or the like. Further, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

In the computer, as the removable storage medium1011is loaded into the drive1010, the program can be installed in the storage unit1008via the input/output interface1005. Further, the program can be received by the communication unit1009via a wired or wireless transmission medium and installed in the storage unit1008. Further, the program can be installed in the ROM1002or the storage unit1008in advance.

Further, the program executed by the computer may be a program in which the processes are chronologically performed in accordance with the order described in this specification or may be a program in which the processes are executed in parallel or a necessary timing such as when calling is performed.

Further, the CPU1001ofFIG.25realizes the functions of the control unit31inFIGS.3and21.

Further, in this specification, a system means a set of a plurality of constituent elements (devices, modules (parts), or the like), and all constituent elements need not necessarily be in the same housing. Accordingly, a plurality of devices which are housed in separate housings and connected via a network and a single device housing a plurality of modules in a single housing are both systems.

Further, the present disclosure is not limited to the above-described embodiments, and various modifications can be made within the scope not deviating from the gist of the present disclosure.

For example, the present disclosure can have a configuration of cloud computing in which one function is shared and cooperatively processed by a plurality of devices via a network.

Further, steps described in the above-described flowchart can be executed by a single device or may be shared and executed by a plurality of devices.

Further, in a case where a plurality of processes are included in one step, a plurality of processes included in one step can be executed by one device or may be shared and executed by a plurality of devices.

Further, the present disclosure can also have the following configurations.

<1> An information processing device, including:an attribute estimating unit that estimates an attribute of a person; andan audio output unit that sets content corresponding to the attribute of the person estimated by the attribute estimating unit and outputs a sound through wave field synthesis so that a sound field of the set content is spatially transmitted.

<2> The information processing device according to <1>, further including:a sound field synthesis parameter generating unit that generates a sound field synthesis parameter for synthesizing the sound field; anda sound field synthesizing unit that synthesizes the sound field by synthesizing the sound field synthesis parameter and audio data of a plurality of channels for each channel through a convolutional operation,in which the audio output unit outputs the sound through the wave field synthesis by outputting a sound with which the sound field is synthesized for each channel by the sound field synthesizing unit.

<3> The information processing device according to <2>, in which the person is a listener,the attribute estimating unit estimates an attribute of the listener,the information processing device further includes a detecting unit that detects a change in a position of the listener, andthe audio output unit sets content corresponding to the attribute of the listener detected by the attribute estimating unit, changes a position of a virtual sound source in accordance with the change in the position of the listener detected by the detecting unit, and outputs the sound through the wave field synthesis so that a spatial sound field is transmitted.

<4> The information processing device according to <3>, in which a listening area in which the sound is heard by the listener is set as a plurality of divisional areas which are divided, andthe audio output unit sets the content corresponding to the attribute of the listener estimated by the attribute estimating unit, and outputs the sound through the wave field synthesis in a divisional area in which the listener is located among the divisional areas on the basis of the change in the position detected by the detecting unit.

<5> The information processing device according to <4>, further including, a tablethat stores a wave field synthesis filter in a case where the listener is located at each position in the divisional area,in which the audio output unit sets the content corresponding to the attribute of the listener estimated by the attribute estimating unit, reads a wave field synthesis filter corresponding to the position in the divisional area from the table, and outputs the sound through the wave field synthesis.

<6> The information processing device according to <4>, further including, an area presenting unit that presents visual information corresponding to a type of content to be output from the audio output unit to the divisional area.

<7> The information processing device according to <6>, in which the area presenting unit is a projecting unit that projects a different pattern for each divisional area in accordance with the type of content to be output from the audio output unit to the divisional area or a light emitting unit that changes a color for each divisional area in accordance with the type of content to be output to the divisional area and emits light.

<8> The information processing device according to <7>, in which, in a case where there is a large number of types of attributes of the listener estimated by the attribute estimating unit, the audio output unit sets content common to a plurality of attributes, and outputs the sound through the wave field synthesis in a divisional area in which the listener is located among the divisional areas.

<9> The information processing device according to any one of <3> to <8>, further including,a display unit that reproduces and displays an image corresponding to the sound; anda subtitle reproducing unit that reproduces a subtitle corresponding to the sound, in which the subtitle reproducing unit selects and reproduces a subtitle corresponding to the attribute of the listener estimated by the attribute estimating unit.

<10> The information processing device according to any one of <3> to <8>, in which the attribute is information recognized by at least one of the image or the sound.

<11> The information processing device according to <10>, in which the information recognized by at least one of the image or the sound is information of the presence or absence of the listener.

<12> The information processing device according to any one of <3> to <8>, in which the attribute is an analysis result of at least one of the image or the sound.

<13> The information processing device according to <12>, in which the analysis result of at least one of the image or the sound includes information indicating individual identification information of the listener, a male, a female, an age, a nationality, a speaking language, one man, one woman, a group of a plurality of men, a group of a plurality of women, a group of mixed men and women, and information indicating whether or not the listener is with family and a facial expression of the listener, an emotion estimated from the facial expression of the listener, and a state of a behavior of the listener.

<14> The information processing device according to any one of <3> to <8>, in which the attribute is information used in non-verbal cues.

<15> The information processing device according to <14>, in which the information used in the non-verbal cues includes a facial expression, a facial color, a line of sight, a body gesture, a hand gesture, a body posture, how to put a physical distance with a counterpart, clothes, a hair style, breathing, a voice tone, or a voice quality.

<16> The information processing device according to any one of <3> to <8>, in which the audio output unit outputs the sound through the wave field synthesis by controlling a volume using a volume curve corresponding to a distance between the listener and a sound source in accordance with the change in the position of the listener detected by the detecting unit.

<17> An information processing method, including:an attribute estimation process of estimating an attribute of a person; anda sound output process of setting content corresponding to the attribute of the person estimated in the attribute estimation process and outputting a sound through wave field synthesis so that a sound field of the set content is spatially transmitted.

<18> A program causing a computer to function as:an attribute estimating unit that estimates an attribute of a person; andan audio output unit that sets content corresponding to the attribute of the person estimated by the attribute estimating unit and outputs a sound through wave field synthesis so that a sound field of the set content is spatially transmitted.

<19> An information processing system, including:an attribute estimating unit that estimates an attribute of a person; andan audio output unit that sets content corresponding to the attribute of the person estimated by the attribute estimating unit and outputs a sound through wave field synthesis so that a sound field of the set content is spatially transmitted.

<20> An information processing device, comprising:circuitry that:estimates an attribute of a first person located in at least one of a plurality of areas; and sets a first content corresponding to the attribute of the first person and outputs a sound through wave field synthesis so that a sound field of the set first content is spatially transmitted to at least the one of the plurality of areas via a plurality of speakers.

<21> The information processing device according to <20>, wherein the circuitry: determines an attribute of a second person in another of the plurality of areas and sets a second content corresponding to the attribute of the second person, and outputs a sound through wave field synthesis so that a sound field of the second content is spatially transmitted to the other of the plurality of areas via the plurality of speakers.

<22> The information processing device according to <21>, wherein the first content and the second content are transmitted simultaneously.

<23> The information processing device according to <21> or <22>, wherein the circuitry prioritizes transmission of the first content over transmission of the second content when the first person and the second person are in a same area of the plurality of areas.

<24> The information processing device according to <23>, wherein transmission of the first content is prioritized over transmission of the second content based on a language of the first content and a language of the second content.

<25> The information processing device according to <24>, wherein the language of the first content is English and the language of the second content is Japanese or Chinese.

<26> The information processing device according to any one of <20> to <25>, wherein the circuit:generates a sound field synthesis parameter for synthesizing the sound field, synthesizes the sound field by synthesizing the sound field synthesis parameter and audio data of a plurality of channels for each channel through a convolutional operation, andoutputs the sound through the wave field synthesis by outputting a sound with which the sound field is synthesized for each channel.

<27> The information processing device according to <26>, wherein the first person is a listener, andthe circuitry:estimates an attribute of the listener,detects a change in a position of the listener,sets content corresponding to the attribute of the listener detected, andchanges a position of a virtual sound source in accordance with the change in the position of the listener detected and outputs the sound through the wave field synthesis so that a spatial sound field is transmitted via the plurality of speakers.

<28> The information processing device according to any one of <20> to <27>, wherein the circuitry:stores a table including a wave field synthesis filter in a case where the listener is located in each of the plurality of areas,sets content corresponding to the attribute of the listener, reads a wave field synthesis filter corresponding to the position in the plurality of areas from the table, and outputs the sound through the wave field synthesis.

<29> The information processing device according to any one of <20> to <28>, wherein the circuitry presents visual information corresponding to a type of content to be output to the one of the plurality of areas.

<30> The information processing device according to <29>, wherein the circuitry projects a different pattern for each area of the plurality of areas in accordance with the type of content to be output to a corresponding area.

<31> The information processing device according to <29> or <30>, wherein the circuitry emits light to change a color of each area of the plurality of areas in accordance with the type of content to be output to a corresponding area.

<32> The information processing device according to <30> or <31>, wherein, in a case where there is a plurality of attributes estimated for the first person, the circuitry sets content common to at least a subset of the plurality of attributes, and outputs the content through wave field synthesis in an area of the plurality of areas in which the first person is located.

<33> The information processing device according to <27>, wherein the circuitry:reproduces and displays an image corresponding to the sound,reproduces a subtitle corresponding to the sound, andselects and reproduces a subtitle corresponding to the attribute of the listener.

<34> The information processing device according to <27> or <33>, wherein the attribute is information recognized by at least one of an image or the sound.

<35> The information processing device according to <34>, wherein the information recognized by at least one of the image or the sound is information of a presence or absence of the listener.

<36> The information processing device according to <27>, <33>, or <34>, wherein the attribute is an analysis result of an image.

<37> The information processing device according to <27> or any one of <33> to <36>, wherein the attribute is an analysis result of the sound.

<38> The information processing device according to <36>, wherein the analysis result of the image includes at least one of information indicating individual identification information of the listener, a male, a female, an age, a nationality, one man, one woman, a group of a plurality of men, a group of a plurality of women, a group of mixed men and women, and information indicating whether or not the listener is with family and a facial expression of the listener, an emotion estimated from the facial expression of the listener, or a state of a behavior of the listener.

<39> The information processing device according to <37>, wherein the analysis result of the sound includes a speaking language.

<40> The information processing device according to <27> or any one of <33> to <37>, wherein the attribute is information used in non-verbal cues.

<41> The information processing device according to <39>, wherein the information used in the non-verbal cues includes at least one of a facial expression, a facial color, a line of sight, a body gesture, a hand gesture, a body posture, how to put a physical distance with a counterpart, clothes, a hair style, breathing, a voice tone, or a voice quality.

<42> The information processing device according to <27> or any one of <33> to <37>, wherein the circuitry outputs the sound through the wave field synthesis by controlling a volume using a volume curve corresponding to a distance between the listener and the plurality of speakers in accordance with the change in the position of the listener.

<43> The information processing device according to <21>, wherein in a case that the first content ends and the first person continues to be located in the one of the plurality of areas, the circuitry outputs a sound through wave field synthesis so that a sound field of a third content is spatially transmitted to the one of the plurality of areas via the speakers.

<44> The information processing device according to <22>, wherein the first content and the second content are different.

<45> An information processing method, comprising:estimating, with circuitry, an attribute of a first person located in at least one of a plurality of areas;setting, with the circuitry, a first content corresponding to the attribute of the first person; andoutputting a sound through wave field synthesis so that a sound field of the set first content is spatially transmitted to at least the one of the plurality of areas via a plurality of speakers.

<46> A non-transitory computer-readable medium encoded with computer-readable instructions that, when executed by a computer, cause the computer to perform a method comprising:estimating an attribute of a first person located in at least one of a plurality of areas;setting a first content corresponding to the attribute of the first person; andoutputting a sound through wave field synthesis so that a sound field of the set first content is spatially transmitted to at least the one of the plurality of areas via a plurality of speakers.

<47> An information processing system, comprising:circuitry that:estimates an attribute of a first person located in at least one of a plurality of areas, sets a first content corresponding to the attribute of the first person, andoutputs a sound through wave field synthesis so that a sound field of the set first content is spatially transmitted to at least the one of the plurality of areas via a plurality of speakers.

REFERENCE SIGNS LIST

1Advertisement presentation system11Information processing device12,12′ Sensor13,13-1to13-3Display unit14Multi-speaker system31Control unit32Input unit33Output unit34Storage unit35Communication unit36Drive37Removable storage medium38Bus51Audio processing unit52Image processing unit71Motion estimating unit72Reproduction schedule generating unit73Attribute estimating unit74Audio data acquiring unit75Sound field synthesis parameter generating unit76Sound field synthesizing unit77N-channel amplifying unit91Image reproducing unit92Subtitle reproducing unit93Subtitle imposing unit111Audio data112Image data113Subtitle data121Area presenting device151Audio data152Image data153Subtitle data161Live hall-specific data162Reproduction hall-specific data171Motion estimating unit172Reproduction schedule generating unit