Patent Description:
There has been disclosed a method for outputting second screen information for video accompanied with sound to a mobile device enabling the input/output of sound and the display of an image (Patent Document <NUM>). According to the method disclosed in Patent Document <NUM>, a time code associated with each video scene of the video is embedded in the sound as watermark data. Then, data corresponding to second screen information is stored in advance in a mobile device so as to be associated with the time code, the sound is input to the mobile device near the video, and the second screen information is output in synchronization with the video scenes.

Patent Document <NUM> relates to a playback device that includes one or more processors and a non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the first playback device to perform functions. The functions include generating a mixed audio signal comprising one or more components of a first audio stream and one or more components of a second audio stream; and playing the mixed audio signal, where the one or more components of the second audio stream of the mixed audio signal played by the first playback device are played in synchrony with a second playback device configured to play the second audio stream.

Patent Document <NUM> is about detecting commercial advertisements associated with media presentations, mainly monitoring digital television feeds to identify transitions between television program content and commercial advertisements. This is achieved through various detection techniques, including profile change detection and random audio detection.

Patent Document <NUM> relates to the field of mobile devices and describes methods for triggering applications on these devices using an audible sonic signal.

Any embodiments or examples of the following description which are not within the scope of the appended claims are provided for illustrative purposes.

According to the method disclosed in Patent Document <NUM>, watermark data is required to be embedded in sound so as to be associated with each video scene. Therefore, a work load for manufacturing content such as a movie is high, and thus the embedding of watermark data cannot be performed for manufacturer's own convenience in some cases.

Although a method for determining the time axis of video with fingerprint that is the feature amount of sound instead of watermark data has been known, the fingerprint cannot be created in a silent situation or at a place where steady sound continues. Further, when completely the same sound is used in different time zones in one video, each of the time zones cannot be discriminated from each other with fingerprint.

In view of the above problems, the present invention has an object of providing a content reproduction program and a content reproduction device that easily and reliably determine the time axis of certain content and reproduce other content in synchronization with the reproduction of the content.

A content reproduction program according to an aspect of the present invention causes a computer to implement: a sound detection unit that detects sound of first content reproduced outside the computer; a time axis determination unit that determines, on the basis of the detected sound, a time axis of second content reproduced subsequent to the first content outside the computer; and a content reproduction unit that reproduces third content corresponding to the second content in synchronization with reproduction of the second content on the basis of the determined time axis.

A content reproduction device according to an aspect of the present invention includes: a sound detection unit that detects sound of first content reproduced outside; a time axis determination unit that determines, on the basis of the detected sound, a time axis of second content reproduced outside subsequent to the first content; and a content reproduction unit that reproduces third content corresponding to the second content in synchronization with reproduction of the second content on the basis of the determined time axis.

Note that a "unit" in the present invention does not merely represent physical means, and the present invention also includes a case in which the function of the "unit" is implemented by software. Further, the function of one "unit" or device may be implemented by two or more physical means or devices, or the functions of two or more "units" or devices may be implemented by one physical means or device.

According to the present invention, a content reproduction program and a content reproduction device that easily and reliably determine the time axis of certain content and reproduce other content in synchronization with the reproduction of the content can be provided.

A preferred embodiment of the present invention will be described with reference to the accompanying drawings. <FIG> is a diagram showing the configuration of a content reproduction system that is an embodiment of the present invention. The content reproduction system includes a user terminal <NUM> (content reproduction device), a screen <NUM>, and speakers <NUM>. The screen <NUM> and the speakers <NUM> are installed in, for example, a facility such as a movie theater and a stadium, and the user terminal <NUM> is used by a user who uses the facility. The video of content such as a movie is displayed on the screen <NUM>, and the sound of the content is output from the speakers <NUM>. Then, in the user terminal <NUM>, other content such as subtitles and an audio guide is reproduced in synchronization with the reproduction of the content.

<FIG> is a diagram showing the outline of content reproduction in the content reproduction system.

Content reproduced in a facility such as a movie theater contains first content <NUM> and second content <NUM> reproduced subsequent to the first content <NUM>. The second content <NUM> is, for example, the main part of a movie. Further, the first content <NUM> is, for example, a trailer, an advertisement, or the like (announcement) reproduced before the main part of a movie. In the first content <NUM>, watermark data <NUM> is embedded in its sound. The watermark data <NUM> is, for example, embedded for each frame of the first content <NUM> and contains data (first data) such as a time code for determining the start timing (time axis) of the second content <NUM>. Further, the watermark data <NUM> contains, for example, data (second data) indicating a facility, an area, or the like.

Third content <NUM> reproduced by the user terminal <NUM> is, for example, the text of the subtitles of a movie reproduced in synchronization with the main part of the movie. The third content <NUM> can be any content such as an audio guide, a sign-language image, and an explanatory image without being limited to subtitles so long as it is reproduced in synchronization with the second content <NUM>. In the user terminal <NUM>, the start timing (time axis) of the second content <NUM> is determined on the basis of the watermark data <NUM>, and the third content <NUM> is reproduced in synchronization with the second content <NUM>.

<FIG> is a diagram showing an example of the hardware configuration of the user terminal <NUM>. The user terminal <NUM> is, for example, a computer such as a smart phone and a tablet terminal. As shown in <FIG>, the user terminal <NUM> includes a processor <NUM>, a storage device <NUM>, a communication interface (I/F) <NUM>, a microphone <NUM>, a sound output device <NUM>, a display device <NUM>, and an input device <NUM>.

The processor <NUM> can implement various functions in the user terminal <NUM> by performing a program stored in the storage device <NUM>.

The storage device <NUM> is a storage region that stores a program or data. The storage device <NUM> can include a volatile memory that is a temporary storage region or a non-volatile memory that is a permanent storage region. The storage device <NUM> may be provided outside the user terminal <NUM>.

The communication interface <NUM> is an interface device for performing the transmission and reception of a program or data with an outside device. The communication interface <NUM> includes, for example, an interface for a mobile telephone communication network or an interface for a wireless LAN (Local Area Network). The communication interface <NUM> may include a wired interface.

The microphone <NUM> detects surrounding sound. The microphone <NUM> can detect not only sound in an audible range but also sound in a non-audible range. The microphone <NUM> may be provided outside the user terminal <NUM>.

The sound output device <NUM> is an output device for outputting sound. The sound output device <NUM> is, for example, a speaker. Further, the sound output device <NUM> may be a device for outputting sound to an earphone. When the third content <NUM> contains sound (for example, an audio guide or dubbing sound), the sound is output from the sound output device <NUM>.

The display device <NUM> is an output device for displaying an image (still image or moving image). The display device <NUM> is, for example, a liquid crystal display or an organic EL (Electroluminescence) display. The display device <NUM> may be provided outside the user terminal <NUM>. When the third content <NUM> contains an image (for example, subtitles), the image is displayed on the display device <NUM>.

The input device <NUM> is a device for receiving a user's input. The input device <NUM> may be, for example, a touch panel integrally formed with the display device <NUM>. The input device <NUM> may be provided outside the user terminal <NUM>.

<FIG> is a diagram showing an example of the function configuration of the user terminal <NUM>. The user terminal <NUM> includes a content acquisition unit <NUM>, a content storage unit <NUM>, a sound detection unit <NUM>, a time axis determination unit <NUM>, a content reproduction unit <NUM>, a reference data acquisition unit <NUM>, a reference data storage unit <NUM>, a setting unit <NUM>, and a setting data storage unit <NUM>. The respective units shown in <FIG> can be implemented, for example, when the storage region of the storage device <NUM> is used or when a program (content reproduction program) stored in the storage device <NUM> is performed by the processor <NUM>.

The content acquisition unit <NUM> acquires the third content <NUM> reproduced by the user terminal <NUM> via, for example, the Internet and stores the acquired third content <NUM> in the content storage unit <NUM>. <FIG> is a diagram showing an example of a user interface for acquiring content. A screen <NUM> is displayed, for example, when an application for reproducing the third content <NUM> is activated. The screen <NUM> includes regions <NUM> and <NUM> for selecting content types.

The screen <NUM> enables the selection of subtitles, sign language, or an audio guide in the region <NUM>. Further, the screen <NUM> enables the selection of a movie theater/theater, a theme park, a museum/art museum, a DVD/BD, or other in the region <NUM>. Then, a content list corresponding to content selected in the regions <NUM> and <NUM> is displayed in a region <NUM>. in the region <NUM>, the title of the second content <NUM> (for example, the main part) is, for example, displayed. The content acquisition unit <NUM> acquires the third content <NUM> corresponding to the second content <NUM> selected in the region <NUM>. Note that the content acquisition unit <NUM> may acquire the third content <NUM> containing an object not to be reproduced.

Further, the content acquisition unit <NUM> may acquire the third content <NUM> specified by sound (watermark data or fingerprint) detected in the first content <NUM>. For example, when watermark data embedded in the first content <NUM> contains data (second data) for specifying the third content <NUM>, the content acquisition unit <NUM> may acquire the third content <NUM> on the basis of the data.

The sound detection unit <NUM> detects the sound of content reproduced outside the user terminal <NUM>. Specifically, the sound detection unit <NUM> can detect the sound of the first content <NUM> (announcement). Further, the sound detection unit <NUM> can also detect the sound of the second content <NUM> (main part). The sound detected by the sound detection unit <NUM> is, for example, watermark data or fingerprint. The sound detection unit <NUM> can stop operating when the time axis of the second content <NUM> is determined by the time axis determination unit <NUM>.

The time axis determination unit <NUM> determines the time axis of the second content <NUM> on the basis of sound detected by the sound detection unit <NUM>. Specifically, the time axis of the second content <NUM> can be determined on the basis of, for example, data (first data) for determining the time axis of the second content <NUM>, contained in watermark data embedded in the sound of the first content <NUM>. Further, the time axis determination unit <NUM> can determine the time axis of the second content <NUM> during the reproduction of the second content <NUM> on the basis of the fingerprint of the sound of the second content <NUM>. Here, in the determination of the time axis of the second content <NUM>, the reproduction timing of the second content <NUM> is associated with time. For example, the time axis determination unit <NUM> can determine time at which the reproduction of the second content <NUM> is started (for example, actual time or relative time based on a certain point). Further, for example, the time axis determination unit <NUM> can determine which part of the second content <NUM> is being currently reproduced. The time axis determination unit <NUM> can stop operating when the time axis of the second content <NUM> is determined.

The content reproduction unit <NUM> reproduces the third content <NUM> with (in synchronization with) the reproduction of the second content <NUM> on the basis of a time axis determined by the time axis determination unit <NUM>. Further, the content reproduction unit <NUM> can output information indicating the state of determination of a time axis determined by the time axis determination unit <NUM>. <FIG> is a diagram showing an example of a screen on which the determination state of the time axis has been displayed. As shown in a screen <NUM> of <FIG>, the content reproduction unit <NUM> can display, on the display device <NUM>, a message (for example, "Preparation for displaying subtitles has been completed. Please wait for a while until the main part starts. ") indicating that the determination of a time axis has been completed.

The reference data acquisition unit <NUM> acquires reference data required when the time axis determination unit <NUM> determines a time axis on the basis of fingerprint via, for example, the Internet, and stores the acquired reference data in the reference data storage unit <NUM>. The reference data is data in which the feature amount of the sound of content is associated with the reproduction time of the content. The time axis determination unit <NUM> can determine the time axis of the second content <NUM> by comparing the feature amount of the sound of the first content <NUM> or the second content <NUM> with the reference data.

The setting unit <NUM> receives a setting for reproducing the third content <NUM> from a user and stores setting data indicating the setting in the setting data storage unit <NUM>. The content reproduction unit <NUM> can reproduce the third content <NUM> on the basis of the setting data.

<FIG> is a flowchart showing an example of content reproduction processing in the user terminal <NUM>. The processing shown in <FIG> is started, for example, when a user activates an application.

The sound detection unit <NUM> starts detecting watermark data embedded in the sound of the first content <NUM> (S701). After the watermark data is detected, the time axis determination unit <NUM> determines the time axis of the second content <NUM> on the basis of the watermark data (S702). Note that the time axis determination unit <NUM> may determine the time axis of the second content <NUM> on the basis of the fingerprint of the sound of the first content <NUM>.

Further, the sound detection unit <NUM> performs the determination (S703 to S704) of a time axis based on the sound of the second content <NUM> concurrently with the determination (S701 to S702) of the time axis based on the sound of the first content <NUM>. Specifically, the sound detection unit <NUM> starts detecting the fingerprint (feature amount) of the sound of the second content <NUM> (S703). Then, the time axis determination unit <NUM> determines the time axis of the second content <NUM> on the basis of the fingerprint (S704). The determination processing (S703 to S704) of the time axis based on the sound of the second content <NUM> is effective, for example, in a case in which the user starts watching the second content <NUM> after the reproduction of the second content <NUM> is started, a case in which the determination (S701 to S702) of the time axis based on the sound of the first content <NUM> is not normally operated, or the like.

The determination processing of the time axis is repeatedly performed unless the determination (S701 to S704) of the time axis based on the sound of the first content <NUM> or the second content <NUM> is completed (S705: N). Then, when the determination of the time axis is completed (S705: Y), the sound detection unit <NUM> and the time axis determination unit <NUM> stop operating and the content reproduction unit <NUM> reproduces the third content <NUM> in synchronization with the reproduction of the second content <NUM> on the basis of the determined time axis (S706).

The embodiment of the present invention is described above. According to the embodiment, the time axis of the second content <NUM> (for example, the main part) reproduced subsequent to the first content <NUM> can be determined on the basis of the sound of the first content <NUM> (for example, announcement). Then, the third content <NUM> (for example, subtitles or an audio guide) corresponding to the second content <NUM> can be reproduced in synchronization with the reproduction of the second content <NUM> on the basis of the determined time axis. Accordingly, even in a case in which watermark data is not embedded in the sound of the second content <NUM> or a case in which the determination of the start timing of the second content <NUM> is difficult with the fingerprint of the sound of the second content <NUM>, the time axis of the second content <NUM> can be easily and reliably determined.

Further, according to the embodiment, the time axis of the second content <NUM> can be determined on the basis of watermark data embedded in the sound of the first content <NUM>. In many cases, it is easier to embed watermark data in the first content <NUM> than in the second content <NUM>. Therefore, the time axis of the second content <NUM> can be easily and reliably determined with the watermark data embedded in the first content <NUM>.

In addition, according to the embodiment, watermark data embedded in the sound of the first content <NUM> can be sound data in a non-audible range. Thus, the time axis of the second content <NUM> can be easily and reliably determined without exerting an influence on a user who watches the first content <NUM>.

Further, according to the embodiment, the time axis of the second content <NUM> can be determined on the basis of the fingerprint (feature amount) of the sound of the second content <NUM>, besides the determination of the time axis of the second content <NUM> based on watermark data embedded in the sound of the first content <NUM>. Thus, for example, in a case in which a user starts watching the second content <NUM> after the reproduction of the second content <NUM> is started or the like, the time axis of the second content <NUM> can be easily and reliably determined.

Further, according to the embodiment, the watermark data <NUM> embedded in the sound of the first content <NUM> may contain data (for example, data indicating a facility, an area, or the like) for specifying the third content <NUM>, besides data (for example, a time code) for determining a time axis. Thus, for example, the third content <NUM> can be selected by the watermark data <NUM> embedded in the sound of the first content <NUM>.

Further, according to the embodiment, the time axis of the second content <NUM> may be determined on the basis of the feature amount of the sound of the first content <NUM>. Thus, for example, even in a case in which the watermark data <NUM> is not embedded in the first content <NUM>, the time axis of the second content <NUM> can be easily and reliably determined.

Further, according to the embodiment, the sound detection unit <NUM> and the time axis determination unit <NUM> can stop operating when the time axis of the second content <NUM> is determined. Thus, the power consumption of the user terminal <NUM> can be reduced.

Note that due to the influence of the performance or the like of the user terminal <NUM>, there could be a deviation between the time axis of the second content <NUM> retained by the user terminal <NUM> and the actual time axis of the second content <NUM> after the time axis of the second content <NUM> is determined. Therefore, after the determination of the time axis of the second content <NUM> is completed, the time axis determination unit <NUM> may correct the time axis of the second content <NUM> on the basis of the fingerprint (feature amount) of the sound of the second content <NUM> at an appropriate timing. For example, after the determination of the time axis of the second content <NUM> is completed, the time axis determination unit <NUM> may correct the time axis of the second content <NUM> on the basis of the fingerprint (feature amount) of the sound of the second content <NUM> at a prescribed time interval. Further, for example, after the determination of the time axis of the second content <NUM> is completed, the time axis determination unit <NUM> may correct the time axis of the second content <NUM> on the basis of the fingerprint (feature amount) of the sound of the second content <NUM> according to instructions from a user.

Further, according to the embodiment, information indicating the state of determination of the time axis of the second content <NUM> can be output to the display device <NUM>. Note that the output destination of the information is not limited to the display device <NUM> but may be the sound output device <NUM>. Thus, the determination state of the time axis of the second content <NUM> can be informed to a user.

Claim 1:
A content reproduction program for causing a computer to perform:
detecting watermark data embedded in sound of first content (<NUM>) reproduced outside the computer and detected by a microphone, wherein the watermark data is sound data in a non-audible range and contains first data for determining the reproduction timing of second content (<NUM>) reproduced subsequent to the first content outside the computer, and second data for specifying a third content (<NUM>) to be reproduced in synchronization with the second content (<NUM>); determining, on the basis of the first data contained in the detected watermark data, the reproduction timing of the second content (<NUM>), wherein determining the reproduction timing further includes detecting a fingerprint of sound of the second content, and comparing the fingerprint of the sound of the second content with reference data in which the fingerprint of the sound of the second content is associated with the reproduction timing of the second content; and
reproducing the third content (<NUM>) in synchronization with reproduction of the second content on the basis of the determined reproduction timing of the second content.