Synchronisation of audio and video playback

A method of playing audio content to a viewer in synchronisation with a video content. The method comprises receiving and digitising an ultrasonic signal comprising ultrasonic synchronisation signal(s) encoding a respective timecode through modulation of ultrasonic carrier signal(s), a first ultrasonic carrier signal having been amplitude modulated to encode logical “1” or “0” in data-carrying segments, each data-carrying segments being followed by non-data-carrying segments in which the amplitude of the ultrasonic carrier signal does not exceed a predetermined threshold. A digitised ultrasonic synchronisation signal is processed to identify the data-carrying segments, and the corresponding timecode is extracted from the data-carrying segments. A stored audio content is played back from a playback point determined based on the timecode.

CLAIM OF PRIORITY

This patent application claims priority from GB patent application number 1408190.5, filed on 8 May 2014, the full contents of which are incorporated herein by reference.

TECHNICAL FIELD

The present invention relates to the field of audio-visual presentation systems and, in particular, to the synchronisation of the playback by a portable device of audio content of an audio-visual presentation with video content of the audio-visual presentation being displayed to the user of the portable device on a remote screen.

BACKGROUND

Conventional audio-visual presentation systems, such as cinemas and home entertainment systems, typically display video content of an audio-visual presentation (e.g. a film, a TV show or a commercial, for example) to a viewer on a screen (e.g. a cinema or TV screen), and play audio content of the audio-visual presentation (e.g. a film soundtrack) to the viewer through a speaker system. The problem of synchronising playback of the audio and video content of an audio-visual presentation in the form of a film (or “motion picture”) was solved in the early part of the 20thcentury by placing a soundtrack encoding the audio physically onto the tape of the film reel, next to the frames of the film, in order to ensure that the right moment of audio accompanies the right moment in the video when the reel is played. Despite the many technological advances that have occurred since that time, the concept underlying this approach to synchronising picture and sound has remained widely in use.

Recently, there has been great interest in developing audio-visual presentation systems that allow each viewer of a film or other kind of audio-visual presentation to enjoy a personalised auditory experience, by listening to one of a number of alternative soundtracks via headphones or earphones, for example. The soundtrack may be selected by each viewer in accordance with their individual needs or preferences. For example, each viewer might wish to listen to one of a number of alternative language versions of a film, descriptive audio for the visually impaired, or a director's commentary. Such delivery of audio content to viewers also opens up the possibility of using binaural audio presentation to immerse each viewer in a realistic 3D sound environment that is difficult to replicate using even the most advanced surround sound speaker systems available today.

However, known approaches to delivering personalised audio content to viewers have significant drawbacks that have hindered their uptake. For example, in the case of delivering personalised audio content to viewers in a cinema via headphones that plug into headphone jacks provided in cinema seats, a significant investment in the required infrastructure and its maintenance must be made by the cinema. An alternative approach which has been proposed, namely of streaming the audio content (for example, via a WiFi™ network or a Bluetooth™ connection provided by the cinema) to a portable device, such as the cinema-goer's smart-phone or PDA, so that the cinema-goer can listen to the film's audio content via headphones or earphones connected to his/her portable device, is also costly as the cinema would need to provide an appropriate means of reliably distributing the audio content throughout the cinema auditorium during shows.

SUMMARY

The present invention provides an audio-visual presentation system for presenting audio content of an audio-visual presentation in synchronisation with video content of the audio-visual presentation to a viewer of the video content. The audio-visual presentation system comprises an ultrasonic synchronisation signal generator arranged to generate a sequence of ultrasonic synchronisation signals during display of the video content, wherein each ultrasonic synchronisation signal encodes a respective timecode through amplitude modulation of one or more ultrasonic carrier signals. The ultrasonic synchronisation signal generator is arranged to generate each ultrasonic synchronisation signal by amplitude modulating a first ultrasonic carrier signal to encode a logical “1” or a logical “0” in each of a plurality of data-carrying segments of the amplitude modulated first ultrasonic carrier signal that encode the timecode, and transmitting the data-carrying segments such that each of the data carrying segments is followed by a non-data-carrying segment in which the amplitude of the first ultrasonic carrier signal does not exceed a predetermined threshold. The audio-visual presentation system further comprises a portable device for playing the audio content of the audio-visual presentation to the viewer, the portable device comprising: a storage device for storing the audio content; a microphone module operable to receive and digitise an ultrasonic signal comprising at least one of the ultrasonic synchronisation signals; a search module arranged to identify a digitised ultrasonic synchronisation signal in the digitised ultrasonic signal; a decoding module arranged to decode the identified digitised ultrasonic synchronisation signal to determine the corresponding timecode by processing the digitised ultrasonic synchronisation signal to identify the data-carrying segments, and extracting the corresponding timecode from the identified data-carrying segments; and a playback module arranged to determine a playback point in the stored audio content based on the determined timecode and play the stored audio content to the viewer from the determined playback point such that the audio content is played in synchronisation with the displayed video content. The present invention further provides an ultrasonic synchronisation signal generator for use in an audio-visual presentation system comprising a display device for displaying video content of an audio-visual presentation and a portable device for playing audio content of the audio-visual presentation to a viewer of the video content. The ultrasonic synchronisation signal generator stores an ultrasonic synchronisation signal soundtrack which, when played by the ultrasonic synchronisation signal generator, comprises a sequence of ultrasonic synchronisation signals for synchronising playback of the audio content by the portable device with the displayed video content. Each ultrasonic synchronisation signal encodes a respective timecode through amplitude modulation of one or more ultrasonic carrier signals. The ultrasonic synchronisation signal generator is arranged to generate the timecode-carrying part of each ultrasonic synchronisation signal by amplitude modulating a first ultrasonic carrier signal to encode a logical “1” or a logical “0” in each of a plurality of data-carrying segments of the amplitude modulated first ultrasonic carrier signal that encode the timecode, and transmitting the data-carrying segments such that each of the data carrying segments is followed by a non-data-carrying segment in which the amplitude of the first ultrasonic carrier signal does not exceed a predetermined threshold.

The present invention further provides a portable device for playing audio content of an audio-visual presentation to a viewer of video content of the audio-visual presentation, wherein the portable device is arranged to synchronise playback of the audio content with the displayed video content based on at least one received ultrasonic synchronisation signal. The portable device comprises a storage device for storing the audio content, and a microphone module operable to receive and digitise an ultrasonic signal comprising the at least one ultrasonic synchronisation signal. The at least one ultrasonic synchronisation signal encodes a respective timecode through amplitude modulation of one or more ultrasonic carrier signals. The microphone module is operable to receive and digitise a first ultrasonic carrier signal which has been amplitude modulated to encode a logical “1” or a logical “0” in each of a plurality of data-carrying segments of the amplitude modulated first ultrasonic carrier signal, wherein each of the data carrying segments is followed by a non-data-carrying segment in which the amplitude of the first ultrasonic carrier signal does not exceed a predetermined threshold. The portable device further comprises a search module arranged to identify a digitised synchronisation signal in the digitised ultrasonic signal, and a decoding module arranged to decode the identified digitised ultrasonic synchronisation signal to determine the corresponding timecode by processing the digitised ultrasonic synchronisation signal to identify the data-carrying segments, and extracting the corresponding timecode from the identified data-carrying segments. The portable device also includes a playback module arranged to determine a playback point in the stored audio content based on the determined timecode and play the stored audio content to the viewer from the determined playback point such that the audio content is played in synchronisation with the displayed video content.

The present invention further provides a method of playing audio content of an audio-visual presentation to a viewer of video content of the audio-visual presentation in synchronisation with the video content, wherein playback of the audio content is synchronised with the displayed video content based on at least one ultrasonic synchronisation signal received during display of the video content. The method comprises receiving and digitising an ultrasonic signal comprising the at least one ultrasonic synchronisation signal, the at least one ultrasonic synchronisation signal encoding a respective timecode through modulation of one or more ultrasonic carrier signals, wherein a first ultrasonic carrier signal which has been amplitude modulated to encode a logical “1” or a logical “0” in each of a plurality of data-carrying segments of the amplitude modulated first ultrasonic carrier signal is received and digitised. Each of the data carrying segments is followed by a non-data-carrying segment, in which the amplitude of the first ultrasonic carrier signal does not exceed a predetermined threshold. A digitised ultrasonic synchronisation signal in the digitised ultrasonic signal is identified, and the identified digitised ultrasonic synchronisation signal is decoded to determine the corresponding timecode, by processing the digitised ultrasonic synchronisation signal to identify the data-carrying segments and extracting the corresponding timecode from the identified data-carrying segments. A playback point in the stored audio content is determined based on the determined timecode, and the stored audio content is played to the viewer from the determined playback point such that the audio content is played in synchronisation with the displayed video content.

The present invention further provides a non-transitory storage medium storing computer program instructions. When executed by a processor of a processing device, the computer program instructions cause the processing device to play audio in synchronisation with video that is displayed by a separate device. The computer program instructions cause the processing device to play the audio by: digitising a received ultrasonic signal comprising at least one ultrasonic synchronisation signal, the at least one ultrasonic synchronisation signal encoding a respective timecode through modulation of one or more ultrasonic carrier signals, wherein a first ultrasonic carrier signal which has been amplitude modulated to encode a logical “1” or a logical “0” in each of a plurality of data-carrying segments of the amplitude modulated first ultrasonic carrier signal is received and digitised, and wherein each of the data carrying segments is followed by a non-data-carrying segment in which the amplitude of the first ultrasonic carrier signal does not exceed a predetermined threshold; identifying a digitised ultrasonic synchronisation signal in the digitised ultrasonic signal; decoding the identified digitised ultrasonic synchronisation signal to determine the corresponding timecode by processing the digitised ultrasonic synchronisation signal to identify the data-carrying segments, and extracting the corresponding timecode from the identified data-carrying segments; determining a playback point in stored audio based on the determined timecode; and playing the stored audio from the determined playback point such that the audio is played in synchronisation with the displayed video.

The present invention further provides a non-transitory storage medium storing an ultrasonic synchronisation signal soundtrack for synchronising playback of audio content of an audio-visual presentation to a view of video content of the audio-visual presentation, the ultrasonic synchronisation signal soundtrack, when played, comprising a sequence of ultrasonic synchronisation signals. Each ultrasonic synchronisation signal comprises a time-code-carrying part that encodes a respective timecode through amplitude modulation of one or more ultrasonic carrier signals. The timecode-carrying part of each ultrasonic synchronisation signal comprises a first ultrasonic carrier signal amplitude-modulated to encode a logical “1” or a logical “0” in each of a plurality of data-carrying segments of the amplitude modulated first ultrasonic carrier signal that encode the timecode. Each of the data carrying segments is followed by a non-data-carrying segment, in which the amplitude of the first ultrasonic carrier signal does not exceed a predetermined threshold.

DETAILED DESCRIPTION OF EMBODIMENTS

The following describes techniques devised by the present inventors that can allow a viewer of video content of an audio-visual presentation to listen to associated audio content stored on their portable device that is played in synchronisation with the displayed video content, without the need for wired or wireless network infrastructure of the kinds mentioned above. The techniques of synchronising playback of stored audio content with displayed video content described herein rely instead on the transmission of timecodes in the form of acoustic signals that can easily be generated by many existing cinema and home entertainment sound systems, and which may be received by many smart-phones and other kinds of portable device that are in widespread use today. Moreover, these techniques employ timecodes each encoded as a burst of one or more modulated acoustic signals that are pitched above the normal limits of human hearing (hereafter referred to as “ultrasonic synchronisation signals” or “ultrasonic timecodes”), which are used to synchronise playback of a stored audio content with the displayed video content without interfering with the delivery of the audio to the viewers' ears, thus allowing many users of the audio-visual presentation system to enjoy high quality playback of their respective audio at once.

The timecode is a number that may, for example, provide a direct or indirect indication of a time that would elapse during playback of the audio (or video) content from the beginning of the audio content (or video content, as the case may be) to a point in the content associated with the timecode, or a direct or indirect indication of a time that would have yet to elapse during playback of the audio (or video) content from a point in the content associated with the timecode to the end of the audio content (or video content, as the case may be). In other words, a timecode is a numerical representation of a unique point in the audio/video content. For example, a timecode of zero may represent the beginning of the content, while a timecode of 100.0 may represent a point in the content that is 100 seconds in. In all these cases, a unique location in the audio and/or video content of the audio-visual presentation for synchronising playback of the audio content with the display of the video content is indicated by the respective timecode or may be inferred from the respective timecode, as will be explained in the following.

When developing the synchronisation techniques described herein, the present inventors have found that, owing to various effects including the reflection of pulses emitted by the sound system off walls and other objects, the ultrasonic pulses which reach the portable device tend not to end abruptly, as intended. This may lead to an intended bit value of “0” being incorrectly interpreted as a value of “1” by the portable device, which can significantly increase the error rate. The inventors have addressed this problem by interleaving data-carrying segments of the ultrasonic synchronisation signal with non-data-carrying segments, which are provided to allow any residual sound amplitude to decay to a level that is unlikely to result in a following encoding of a bit value of “0” being incorrectly decoded as a “1” by the portable device.

An audio-visual presentation system according to an embodiment may be provided in a cinema, for example; in this case, a cinema-goer may download their film soundtrack of choice onto their smart-phone (or other portable device) in the comfort of their own home, and then listen to the film soundtrack using their earphones or headphones while watching the film in the cinema, with the playback of the film soundtrack being synchronised quickly and effectively with the picture on the cinema screen by using the cinema's existing audio equipment (including amplifiers, loudspeakers etc.) to broadcast a sequence of ultrasonic synchronisation signals. As will become apparent from the following description, the synchronisation techniques described herein can ensure reliable synchronisation at any point during the film presentation and thus cater for, e.g. late-comers and viewers returning to the auditorium part way through the film presentation with refreshments, as well as interruptions to the film presentation.

In an embodiment described herein, the ultrasonic synchronisation signal generator is further arranged to generate each ultrasonic synchronisation signal by amplitude modulating a second ultrasonic carrier signal to encode a logical “1” or a logical “0” in each of a plurality of data-carrying segments of the amplitude modulated second ultrasonic carrier signal which, together with the data-carrying segments of the amplitude modulated first ultrasonic carrier signal, encode the timecode. The ultrasonic synchronisation signal generator is further arranged to transmit the data-carrying segments of the amplitude modulated first and second ultrasonic carrier signals such that: each of the data carrying segments of the amplitude modulated second ultrasonic carrier signal is followed by a non-data-carrying segment in which the amplitude of the second ultrasonic carrier signal does not exceed a predetermined threshold; the data-carrying segments of the amplitude modulated second ultrasonic carrier signal are transmitted conterminously with the non-data-carrying segments of the amplitude modulated first ultrasonic carrier signal; and the data-carrying segments of the amplitude modulated first ultrasonic carrier signal are transmitted conterminously with the non-data-carrying segments of the amplitude modulated second ultrasonic carrier signal. In this embodiment, the decoding module is arranged to decode the identified digitised ultrasonic synchronisation signal to determine the corresponding timecode by processing the digitised ultrasonic synchronisation signal to identify the data-carrying segments of the amplitude modulated first and second ultrasonic carrier signals, and extracting the corresponding timecode from the identified data-carrying segments. By offsetting the data-carrying segments of adjacent modulated ultrasonic carrier signals relative to each other such that the data-carrying segments of each modulated ultrasonic carrier signal are transmitted conterminously with the non-data-carrying segments of adjacent modulated ultrasonic carrier signal(s), the transmission of data segments and non-data-carrying segments can proceed in parallel in order to maintain a high data transmission rate, while reducing the risk of signals in one frequency channel being interfered with by the signals in adjacent frequency channels.

FIG. 1illustrates an audio-visual presentation system according to an embodiment of the present invention. By way of example, the audio-visual presentation system of the present embodiment is provided in a cinema, and serves to present to a cinema-goer audio content of a film being watched by the cinema-goer in synchronisation with the film picture being projected onto the cinema screen. However, it will be appreciated that the techniques described herein are applicable to other kinds of audio-visual presentation system, for example in home entertainment, where they may be used to make the viewing of television programs or films on a television set more enjoyable for the viewers. As another example, these techniques may also be applied to audio-visual presentation systems in educational environments, for example in museums, classrooms and lecture theatres, as well as those used in trade shows and other commercial settings.

As shown inFIG. 1, the audio-visual presentation system may, as in the present embodiment, include a display device100in the exemplary form of a cinema screen, onto which the picture of the film is projected by the cinema's projection system (not shown). In the present embodiment, the various loudspeakers and their associated signal sources, amplifiers etc. constituting the cinema's sound system are configured to function as an ultrasonic synchronisation signal generator200, which generates a sequence of synchronisation signals only at ultrasonic frequencies during the presentation of the film. These ultrasonic synchronisation signals are received by a portable device300being used by the cinema-goer, such as his/her smart-phone, which stores the audio content of the film and plays back the stored audio content via, e.g. headphones or earphones plugged into the smart-phone. Moreover, the portable device300is configured to process one or more of the received ultrasonic synchronisation signals so as to synchronise playback of the audio content with the picture being displayed on the display device100, as will be explained in the following. Accordingly, the audio-visual presentation system of the present embodiment may be regarded as a “silent cinema”, in the sense that the cinema's sound system itself produces no sounds during the film presentation that are audible to the human ear. The sound system may, however, be used for public announcements relating to advertising etc. before the film presentation, and/or for safety or security announcements during an interruption to the film presentation. However, in other embodiments, the ultrasonic synchronisation signal generator200may generate audible sounds (e.g. play background music) while it is emitting the ultrasonic synchronisation signals.

The cinema's pre-existing sound system is thus arranged to play an “ultrasonic synchronisation signal soundtrack” comprising a sequence of ultrasonic synchronisation signals during display of the video content of the film on the cinema screen, such that each of the ultrasonic synchronisation signals is an encoded representation of the timecode for a corresponding point in the film presentation. The sound system200may, as in the present embodiment, be arranged to generate the ultrasonic synchronisation signals periodically, such that the ultrasonic synchronisation signals in the sequence are emitted at regular time intervals (e.g. once every second) throughout the duration of the film. Thus, instead of the usual film soundtrack, the cinema's sound system200is configured to play the ultrasonic synchronisation signal soundtrack to the audience.

As an example, for an hour-long film, the ultrasonic synchronisation signal soundtrack may contain instructions for causing the cinema's sound system200to generate one ultrasonic synchronisation signal every second, so that 3600 different ultrasonic synchronisation signals are emitted during the course of the film presentation, where each generated ultrasonic synchronisation signal is a burst of ultrasonic pulses encoding a respective timecode value that represents the number of seconds that have elapsed since the start of the film presentation. The ultrasonic synchronisation signal soundtrack may be attached to any motion picture asset. For modern digital content, this alternative soundtrack may be interleaved with the picture data. For traditional film, the ultrasonic synchronisation signal soundtrack may replace an existing soundtrack. As those skilled in the art will be familiar with various techniques that may be used to produce such an ultrasonic synchronisation signal soundtrack, a description of these techniques will not be provided here. The ultrasonic synchronisation signal soundtrack may, as in the present embodiment, be stored in a storage medium for distribution to the cinema or other venue hosting the audio-visual presentation. By way of example, the storage medium may be a Blu-ray™ disc or a DVD disc. Preferably, the storage medium also stores the video content of the audio-visual presentation. The ultrasonic synchronisation signal soundtrack may alternatively be encoded in a signal that may be transmitted over a computer network (e.g. downloaded over the Internet) to the cinema or other venue and played through the venue's sound system while the video content is being displayed.

Further details of the structure of the ultrasonic synchronisation signals that are generated by the cinema's sound system will now be described with reference toFIGS. 2A and 2B.

FIG. 2Ais a schematic illustration of two consecutive ultrasonic synchronisation signals in the sequence of ultrasonic synchronisation signals generated by the ultrasonic synchronisation signal generator200. Each ultrasonic synchronisation signal400may, as in the present embodiment, comprise a timecode-carrying part that encodes a respective timecode through modulation of at least one ultrasonic carrier signal, as well as an ultrasonic marker signal at a different carrier frequency than the at least one ultrasonic carrier signal, wherein the timecode-carrying part and the marker signal are transmitted conterminously by the ultrasonic synchronisation signal generator200.

Although each timecode-carrying part may thus encode a respective timecode through amplitude modulation of a single ultrasonic carrier signal, the timecode-carrying part of each ultrasonic synchronisation signal preferably encodes a respective timecode through amplitude modulation of a plurality of ultrasonic carrier signals (or “channels”) having different frequencies, preferably in the range of 18 kHz to 22 kHz. In the present embodiment, the timecode-carrying part410of each ultrasonic synchronisation signal400encodes a respective timecode through amplitude modulation of eight ultrasonic carrier signals,410-1to410-8, as shown inFIG. 2A, which are transmitted simultaneously by the cinema's sound system. The frequencies of the carrier signals are preferably selected to have sufficient separation to allow the individual carrier signals to be clearly resolved by the portable device300.

More specifically, the ultrasonic synchronisation signal generator200is arranged to generate the timecode-carrying part410of the ultrasonic synchronisation signal400by amplitude-modulating each of the ultrasonic carrier signals410-1to410-8so as to encode a logical “1” or a logical “0” in each of a plurality of data-carrying segments412, and transmitting the data-carrying segments such that each of the data carrying segments412is followed by a non-data-carrying segment414. The shaded labelled segments shown inFIG. 2Aare data-carrying segments representing a bit value of “1”, while the unshaded (white) labelled segments are data-carrying segments representing a bit value of “0”. In each non-data-carrying segment414, the ultrasonic carrier signal has an amplitude which does not exceed a predetermined threshold. More particularly, the ultrasonic synchronisation signal generator200is configured to modulate the ultrasonic carrier signal to have an amplitude in each non-data-carrying segment414that is below the predetermined threshold value, which may be set such that the non-data-carrying segment414does not affect the decoding of the following data-carrying segment by the portable device300, as will be explained in the following. The ultrasonic synchronisation signal generator200may thus modulate the carrier signal by appropriately attenuating the amplitude of a generated carrier signal, or by stopping the generation of the carrier signal for the duration of the non-data-carrying segment, for example. For example, the non-data-carrying segments414may be amplitude-modulated in the same way as a logical “0” in the data-carrying segments412. As shown inFIG. 2A, the data-carrying segments412of each modulated ultrasonic carrier signal are transmitted conterminously with the non-data-carrying segments414of each adjacent ultrasonic carrier signal. In this way, the frequency channels used to carry the timecode bits alternate between two sets of frequency channels (i.e. a first set comprising channels410-2,410-4,410-6and410-8, and a second set comprising channels410-1,410-3,410-5and410-7in the present embodiment) after the transmission of each set of conterminous data-carrying segments412(e.g. Bits0-3, Bits4-7, Bits8-11, or Bits12-15shown inFIG. 2A), thereby allowing any echoes etc. to be received by the portable device300during the conterminous non-data-carrying segments414that immediately follow each set of conterminous data-carrying segments412. As will be explained in the following, the portable device300is configured to ignore the content of the non-data-carrying segments414but to extract the timecode value from the data-carrying segments412.

Each data-carrying part410of the ultrasonic synchronisation signal400shown inFIG. 2Aprovides a binary representation of the corresponding timecode value, where the amplitude of a carrier signal at a given frequency represents a logical “1” or a logical “0”. More specifically, 14 bits are used in the present embodiment to represent the timecode value, and two bits are used to provide parity information for use in a validity check, which is described below. Using 14 bits allows for 16384 unique timecode values. If one ultrasonic synchronisation signal400is transmitted every second by the sound system200, this would be sufficient for a presentation 273 minutes in length (or over 4.5 hours). As shown inFIG. 2B, the timecode-carrying part410of the ultrasonic synchronisation signal400shown inFIG. 2Aindicates a timecode value of 5301 and thus a point in the stored soundtrack that is to accompany the picture displayed on the cinema screen 5301 seconds (i.e. 1 hour, 28 minutes and 21 seconds) from the beginning of the presentation (in the example ofFIG. 2B, “Bit13” is the most significant bit while “Bit0” is the least significant bit, and “Bit14” and “Bit15” are parity bits set aside for the validity check).

Advantages that follow from interleaving non-data-carrying segments414with the data-carrying segments412will now be explained with reference toFIGS. 3A and 3B.

FIG. 3Ais a schematic illustration of four adjacent data-carrying segments in a single frequency channel, which are intended to encode bits values of “1”, “1”, “0” and “1”. However, the inventors have found that, owing to various effects including the reflection of pulses emitted by the sound system200off walls and other objects in the cinema auditorium, the ultrasonic pulses which actually reach the portable device300tend not to end abruptly, as intended, but often decay on a time scale comparable to the duration of a data-carrying segment412. This may lead to an intended bit value of “0” being incorrectly interpreted as a value of “1” by the portable device300, which can significantly increase the error rate.

The inventors have addressed this problem by: (i) interleaving the data-carrying segments412with non-data-carrying segments414carrying no data, which are provided to allow any residual sound amplitude to decay to a level that is unlikely to result in a following encoding of a bit value of “0” being incorrectly decoded as a “1” by the portable device300; and (ii) offsetting the data-carrying segments412of adjacent modulated ultrasonic carrier signals relative to each other such that the data-carrying segments412of each modulated ultrasonic carrier signal are transmitted conterminously with the non-data-carrying segments414of adjacent modulated ultrasonic carrier signal(s). This arrangement allows the transmission of data segments and non-data-carrying segments to proceed in parallel in order to maintain a high data transmission rate, while ensuring that each data segment is transmitted conterminously with non-data-carrying segments in the adjacent frequency channel(s), thereby reducing the risk of cross-talk, with signals in one frequency channel being interfered with by the signals in adjacent frequency channel(s). This may also allow a narrower channel spacing to be used.

Referring again toFIG. 2A, the ultrasonic marker signal420may, as in the present embodiment, be provided in the form of a marker pulse whose leading edge coincides with one end of the timecode-carrying part410, and whose trailing edge coincides with the other end of the timecode-carrying part410of the synchronisation signal400. The presence of the ultrasonic marker pulse420allows the portable device300to determine a timeframe for the timecode-carrying part in a digitised record of the received ultrasonic signal that is stored in the portable device300. The sound system200may, as in the present embodiment, be arranged to generate the marker pulses420so as to have a predetermined duty cycle, for example 50 percent (the duty cycle being the proportion of a repeat period of the marker pulse that is taken up by the “high” part of the marker pulse). In this case, the marker pulse420as received by the portable device300is expected to be present for 0.5 seconds in each 1-second period. The signal processing operations performed by the portable device300can make use of this property of the marker pulse420to achieve more reliable decoding of received ultrasonic synchronisation signals, as will be described in the following.

The marker pulse420may, as in the present embodiment, be provided substantially in the middle (or within a central portion) of the frequency band that spans the range of frequencies of the ultrasonic carrier signals410-1to410-8, in order to allow the portable device300to determine a value of a threshold level that is suitable for effectively decoding data-carrying segments received at all of the carrier frequencies in the frequency band, as will also be described in the following.

Functional components of the portable device300that are helpful for understanding the present invention will now be described with reference toFIG. 4.

As illustrated inFIG. 4, the portable device300of the present embodiment comprises a storage device310for storing the audio content of the audio-visual presentation, a microphone module320having a microphone322and an analog-to-digital converter (ADC)324that are arranged to receive and digitise an ultrasonic signal comprising the ultrasonic synchronisation signals400generated by the cinema's sound system, and a search module330arranged to identify a digitised ultrasonic synchronisation signal in the digitised ultrasonic signal. The search module330may, as in the present embodiment, be arranged to identify the timecode-carrying part of a digitised ultrasonic synchronisation signal in the digitised ultrasonic signal based on a received ultrasonic marker signal420. The portable device300also includes a decoding module340arranged to decode digitised ultrasonic synchronisation signal to determine the corresponding timecode, and a playback module350, which is arranged to determine a playback point in the stored audio content based on the determined timecode and play the stored audio content to the cinema-goer from the determined playback point such that the audio content is played in synchronisation with the video content. In the present embodiment, the playback module350plays the audio content through headphones which plug into the portable device300, although earphones or another kind of personal audio device may alternatively be used. The playback module350may alternatively play the audio content through speakers. The storage device310, microphone module320, search module330, decoding module340and playback module350are functionally inter-connected to communicate with one another as shown inFIG. 4.

FIG. 5provides a schematic illustration of component parts of the decoding module340, namely a sampling module342, a frequency spectrum calculation module344, a background noise estimation module346, and a frequency spectrum correction module348. These components of the decoding module340are functionally inter-connected as shown inFIG. 5to communicate with one another.

FIG. 6shows an example of programmable signal processing hardware that may form part of the portable device300and implement the functions of the component modules illustrated inFIGS. 4 and 5. The signal processing apparatus600shown inFIG. 6comprises an input/output (I/O) section610for receiving audio content of a film during download of a film soundtrack before the film presentation, and for outputting the audio content to headphones or earphones during the film presentation. In addition, the signal processing apparatus600includes a microphone322as also shown inFIG. 4. The signal processing apparatus600further comprises a processor620, a working memory630and an instruction store640storing computer-readable instructions which, when executed by the processor620, cause the processor620to perform the processing operations hereinafter described to synchronise playback of the stored audio content with displayed video content of the film. The instruction store640may comprise a ROM which is pre-loaded with the computer-readable instructions. Alternatively, the instruction store640may comprise a RAM or similar type of memory, and the computer readable instructions can be input thereto from a computer program product, such as a computer-readable storage medium650such as a CD-ROM, etc. or a computer-readable signal660carrying the computer-readable instructions.

The computer-readable instructions may, for example, take the form of a custom application (“app”), which the user can download to his/her smart-phone (e.g. from the online iTunes™ store where the smart-phone is an iPhone™, or from the Google Play™ store for smart-phones running an Android™ operating system) and use to select and download their film audio content of choice (including, for example, the spoken part, voice-overs, music and/or sound effects that is/are to accompany the video content of the film) to the portable device300, and to synchronise playback of the audio content with displayed video content of the film during the film presentation at the cinema.

In the present embodiment, the combination670of the hardware components shown inFIG. 6, comprising the processor620, the working memory630and the instruction store640, is configured to implement the functionality of the aforementioned search module330, decoding module340and playback module350shown inFIG. 4, as well as the sampling module342, frequency spectrum calculation module344, background noise estimation module346and frequency spectrum correction module348shown inFIG. 5, which will now be described in detail with reference toFIGS. 7 to 12.

FIGS. 7, 9 and 12are flow charts illustrating a process by which the portable device300synchronises playback of the stored audio content with displayed video content of the film in the present embodiment.

FIG. 7is a flow chart that provides a top-level illustration of the synchronisation process. In step S100ofFIG. 7, the microphone module320records a short amount of audio (typically a few seconds), which includes one or more of the ultrasonic synchronisation signals400generated by the ultrasonic synchronisation signal generator200, as well as ambient noise. During this process, the microphone322converts the received sound waves to analog electrical signals, which are then digitised by the ADC324to produce a digitised record of the received audio. In step S100, the microphone module320preferably records at least two consecutive ultrasonic synchronisation signals, in order to establish the playback point with a high degree of reliability. For example, in the present embodiment, recording three seconds of audio will ensure that at least two ultrasonic synchronisation signals are captured. The ADC324is arranged to sample the signal from the microphone322at a rate of 48 kHz, and record each sample value as a 16-bit integer (i.e. using a bit depth of 16). Thus, 48000 16-bit integers will represent one second of recorded audio.

In step S200, the search module330identifies a digitised ultrasonic synchronisation signal in the digitised ultrasonic signal. More particularly, the search module330may, as in the present embodiment, identify the timecode-carrying part of a digitised ultrasonic synchronisation signal in the digitised ultrasonic signal based on an ultrasonic marker signal in the received audio. In this case, the search module330searches for the presence of a tone at the carrier frequency assigned to the marker pulse420, and is able to determine a time frame in which the timecode-carrying part is located as the timecode-carrying part is conterminous with the marker pulse. In embodiments where a marker pulse is used, the marker pulse can thus allow the search module330to find the timecode-carrying part quickly, without having to resort to computationally intensive techniques such as those requiring pattern recognition to be performed in windowed portions of the digitised audio, for example.

In step S300, the decoding module340decodes the identified digitised ultrasonic synchronisation signal to determine the corresponding timecode by processing the digitised ultrasonic synchronisation signal to identify the data-carrying segments412of the timecode-carrying part410, and extracting the corresponding timecode from the identified data-carrying segments412. As will be explained further below, the decoding module340of the present embodiment processes the digitised ultrasonic synchronisation signal to identify the data-carrying segments412by making use of a marker pulse of an ultrasonic synchronisation signal that has been captured and digitised to determine a threshold level for a binarising process that is based on the identified timecode-carrying part of the digitised ultrasonic synchronisation signal. The decoding module340then decodes the identified timecode-carrying part410by performing the binarising process using the determined threshold level to generate a representation of the timecode, and identifies the data-carrying segments412in the representation of the timecode.

In step S400, the playback module350determines a playback point in the stored audio content based on the timecode determined in step S300. For example, the playback module350may, as in the present embodiment, convert the timecode value to a time that has elapsed from the start of the film, and determine the playback point using this value of the time (making any necessary allowances for the time required to process the received ultrasonic synchronisation signal and begin playback of the audio content). Additionally or alternatively, the playback module350may calculate a time offset value that allows the playback module350to determine the playback point any time after even a single timecode value has been determined, as will now be explained.

During playback of the audio content, the time elapsed from the beginning of the film presentation (herein referred to as the “movie time”) is the real (clock) time minus a time offset, which corresponds to the clock time when playback was started. For example, Table 1 below illustrates the constant time offset between the clock time and the movie time at four different points during a film presentation, which begins at 9:00 pm.

FIG. 8schematically illustrates a 3-second recording of audio generated by the microphone module320, which contains three digitised ultrasonic synchronisation signals encoding timecode values of215,216and217. Once the timecode has been determined, the playback module350of the app will have the value of the timecode (t=215, as shown at A in the example ofFIG. 8), the clock time when the recording by the microphone module320ended (as shown at point B, namely 8 h:32 m:16.3 s) and the interval in the recording from the timecode to the end of the recording (2.7 seconds). At the point labelled B inFIG. 8, both the clock time (8 h:32 m:16.3 s) and the movie time (217.7 s) are known, and the time offset can be calculated by subtracting one from the other. With this calculated value of the time offset, the clock time kept by the portable device300can be used at any point in the film presentation to determine the movie time and thus the playback point for playing the audio content in synchronisation with the video content displayed on the display device100.

Referring again toFIG. 7, in step S500, the playback module350plays the stored audio content from the determined playback point to the cinema-goer via their headphones or earphones such that the audio content is played in synchronisation with the video content being displayed on the display device100.

FIG. 9is a flow chart illustrating an example of the processes performed in step S300ofFIG. 7.

In step S310, the sampling module342of the decoding module340takes a plurality of samples of the digitised recording of the received audio that includes the ultrasonic synchronisation signals. In the present embodiment, each sample obtained by the sampling module342is a block of 1024 digitised values of the recorded sound amplitude.

In step S320, the frequency spectrum calculation module344calculates a frequency spectrum for each of the blocks obtained in step S310, for example by performing a Fast Fourier Transform (FFT) on each block. The FFT of each block of 1024 16-bit values provides an indication of which frequencies were present in the portion of the received audio signal that corresponds to the block. As the 1024 values within each block correspond to 1024/48000=0.021 seconds of recorded audio, about 140 frequency spectra can be obtained from a 3-second recording. Each of these spectra has 512 discrete “containers” corresponding to individual frequencies. Furthermore, each of the containers will contain a value indicative of the contribution of that container to the spectral content in the block of digitised audio sample values. The 3-second recording can therefore be visualised as being 140 time units wide and having 512 frequency channels, as illustrated inFIG. 10.

In order to reduce the adverse effects of background noise on the decoding process, the decoding module340may, as in the present embodiment, employ the background noise estimation module346in step S330to estimate a respective background noise component of each of the plurality of frequency spectra, and the frequency spectrum correction module348may correct each of the frequency spectra in step S340by removing therefrom the corresponding estimate of the background noise component to generate a corrected frequency spectrum of each sample.

The background noise estimation module346may estimate a respective background noise component of each of the plurality of frequency spectra in one of a number of different ways. In the present embodiment, the background noise estimation module346takes advantage of the intervals in the recording between those that contain ultrasonic synchronisation signals to obtain estimates of the background noise that are not tainted by the modulation of the carrier signals410-1to410-8. More specifically, the background noise estimation module346of the present embodiment estimates, for each frequency spectrum in the plurality of frequency spectra, a respective background noise component based on an amplitude of at least one spectral component (or “container”) of a different frequency spectrum in the plurality of frequency spectra, the different frequency spectrum having been derived from a sample of the ultrasonic signal between two adjacent ultrasonic synchronisation signals in the sequence of ultrasonic synchronisation signals. This estimate may be calculated in any suitable or desirable way, for example by averaging a plurality of FFT amplitudes in the aforementioned frequency spectrum that lies between two adjacent ultrasonic synchronisation signals, or by estimating how the background noise varies as a function of frequency in the relevant ultrasonic frequency range.

In an alternative embodiment, the background noise estimation module346may estimate, for each frequency spectrum in the plurality of frequency spectra, a respective background noise component based on an amplitude of at least one spectral component of the frequency spectrum at a frequency different from the respective frequencies of the one or more ultrasonic carrier signals410-1to410-8(and also different from the carrier frequency of the marker signal420, if such a marker signal is used). This variant is useful in cases where the background noise is expected to vary significantly on the timescale of the ultrasonic synchronisation signal (0.5 seconds in the present embodiment), where an estimate of the background noise that has been received at the same time as the ultrasonic synchronisation signal may allow more effective correction of the frequency spectra by the frequency spectrum correction module348.

An example of a grid of spectrum data obtained by removing the background noise component is illustrated inFIG. 11. In this example, the illustrated noise profile was determined by sampling and averaging a selection of non-data carrying frequency channels. More particularly, the noise profile was obtained by identifying a set of non-data carrying frequency channels in the spectrogram, and then averaging out the noise profile in those channels to create a background noise profile for the recording, which can be subtracted from each of the data-carrying frequency channels. These techniques of suppressing the adverse effects of background noise allow valid timecode data to be obtained by the portable device300in surprisingly noisy environments.

Referring again toFIG. 9, once the frequency spectra have been corrected by removal of the background noise component, the process proceeds to step S350, in which the decoding module340uses the portion of the spectrogram corresponding to the marker pulse420of the ultrasonic synchronisation signal400to determine a threshold level for binarising the corrected frequency spectra of the samples of ultrasonic signal corresponding to the timecode-carrying part410of the ultrasonic synchronisation signal400, in order to generate a representation of the timecode from which the timecode value can be obtained. The threshold level may be regarded as an “amount of signal” that is used to distinguish between a “1” and a “0”. The decoding module340thus uses the marker pulse420to determine both a timing and a sensitivity level used to decode the received ultrasonic synchronisation signals.

The decoding module340may determine the threshold level in one of a number of different ways. For example, in the present embodiment, the decoding module340calculates a respective duty cycle of the marker pulses in the periodic sequence of ultrasonic synchronisation signals for each of a plurality of candidate values of the threshold level, and selects the threshold level for the binarising process from among the candidate values of the threshold level such that the selected threshold level yields a duty cycle lying within a predetermined range of values of the duty cycle. For example, the selected threshold level may be such that it yields a calculated duty cycle that is within the range of 40 to 60 per cent, if the marker pulse is known to be transmitted with a 50 per cent duty cycle. Determining the threshold level in this adaptive manner allows the decoding module340to effectively extract timecodes from audio signals generated by a variety of sound systems and in different venues.

In step S360, the decoding module340performs the binarising process by binarising the corrected frequency spectra of the samples of the ultrasonic signal corresponding to the timecode-carrying part of the digitised ultrasonic synchronisation signal to generate a representation of the corresponding timecode. The representation of the timecode can be regarded as a black and white, two-dimensional checkerboard pattern as shown inFIG. 2, with black portions representing logical values of “1”, and white portions representing logical values of “0”, for example.

In step S370, the decoding module340determines the corresponding timecode based on the representation thereof generated in step S360. More particularly, in step S370, the decoding module340identifies the data-carrying segments412in the corrected spectral data representing the timecode (which may be visualised as a spectrogram of the kind shown inFIG. 11), for example on the basis of stored indicators of where each data-carrying segment412is located within the representation in relation to the marker pulse420, where each of the stored indicators provides, for example, an indication of the frequency channel and a time measured from the beginning of the marker pulse420at which a data-carrying segment412is located. The decoding module340then extracts the corresponding timecode from the identified data-carrying segments412, e.g. by binarising the representation of the timecode, interpreting data-carrying segments412having amplitudes above a certain level as containing a logical “1”, and data-carrying segments412having amplitudes below that level as containing a logical “0”.

Once the timecode value has been determined, the decoding module340may proceed to validate the decoded data, to reduce the risk of audio content being played back from the wrong point in the film presentation. In the present embodiment, two strategies are employed to validate the decoded data. The first is a 2-bit binary checksum for each timecode-carrying part410. For the decoded timecode-carrying part410to pass the validation, the 14 data-carrying segments should generate a 2-bit parity sum which matches the parity bits in the remainder of the timecode-carrying part410. Timecodes which fail this parity test are rejected.

The decoding module340may, as in the present embodiment, proceed to calculate a refined value of the threshold in step S380, in order to improve subsequent decoding operations. An example of a process by which such a refined value of the threshold level may be determined is illustrated in the flow chart ofFIG. 12.

In step S382ofFIG. 12, the decoding module340adjusts the determined threshold level (by incrementing it up or down), and uses the adjusted threshold level to decode a timecode-carrying part of a received first ultrasonic synchronisation signal to determine a corresponding first timecode. In step S384, the decoding module340uses the adjusted threshold level to decode a timecode-carrying part of a received second ultrasonic synchronisation signal that is different from the first ultrasonic synchronisation signal, to determine a corresponding second timecode. In step S386, the decoding module340determines whether the determined first and second timecodes have a predetermined relationship to one another. For example, where the first and second ultrasonic synchronisation signals are received one after the other, with no other ultrasonic synchronisation signal being received in the intervening period (e.g. as illustrated inFIG. 2), the decoding module340may determine whether the determined first and second timecodes correspond to integers that differ by1. If the first and second timecodes are determined not to have the relationship of the required form, steps S382to S386are repeated, with the threshold level being appropriately adjusted to a new value in each subsequent performance of step S382. However, if the first and second timecodes are determined to have the predetermined relationship to each other, the process proceeds to step S388, where the decoding module340determines the refined value of the threshold level based on a value to which the threshold level has been adjusted in the final performance of step S382.

The refined value may, for example, correspond to the value to which the threshold level has been adjusted in the final, penultimate or earlier performance of step S382, or a value derived from any of these. The decoding module340may be arranged to break out of the loop illustrated inFIG. 12if the answer in step S386is “no” after the loop has been repeated a predefined number of times, whereupon the decoding module340may proceed to acquire and process a different recording from the microphone module320.

The process illustrated inFIG. 12also provides a further validity check for the obtained timecode values. If the decoded data passes both the parity test and this further validity check, it may be concluded that the extracted timecode values are valid with a high degree of confidence. For random inputs, the likelihood of false positives is estimated to be less than 1 in 260,000.

Many modifications and variations can be made to the embodiments described above.

For example, although the portable device300storing the audio content is provided in the form of a smart-phone in the embodiments described above, the portable device300may alternatively be provided in other forms, such as a PDA, laptop computer, tablet computer, an mp3 player or other personal music player that is equipped with a microphone, for example.

In the embodiments described above, the ultrasonic synchronisation signal generator200is provided in the form of a cinema sound system. However, the ultrasonic synchronisation signals may more generally be generated by any sound reproduction system that is capable of generating ultrasonic synchronisation signals of the described forms. In this regard, it should be noted that the representation of an ultrasonic synchronisation signal in a spectrogram need not be as illustrated inFIG. 2A. For example, the ultrasonic synchronisation signal may be generated by modulating more or fewer than eight ultrasonic carrier signals, and is not limited to having 14 data-carrying segments412. Furthermore, where the data-carrying segments412of each frequency channel are interleaved with non-data-carrying segments414, the data carrying segments412of one or more adjacent channels may be transmitted conterminously.

It should also be noted that the synchronisation scheme need not make use of a marker signal420, as in the above-described embodiments, although it is advantageous to do so in many practical applications for the reasons explained above.

In the above-described embodiments, the timecode is taken, by way of example, to represent the number of seconds that have elapsed since the start of the film presentation. However, in other embodiments, where the audio-visual presentation is arranged to begin at a predetermined time, the timecode may simply indicate a clock time with which both the ultrasonic synchronisation signal generator200and the portable device300are synchronised. This clock time may be kept by the ultrasonic synchronisation signal generator200or another device (e.g. a timing server) to which both the ultrasonic synchronisation signal generator200and the portable device300can connect. In this case, the portable device300may simply subtract the known start time of the presentation from the received timecode value and thus calculate the appropriate point in the stored audio content from which to play back the audio content to the user during the presentation.

Furthermore, in embodiments in which no background noise estimation and frequency spectrum correction are performed, the decoding module340may be arranged to process the digitised ultrasonic synchronisation signal to identify the data-carrying segments412by binarising the frequency spectra of the samples of the ultrasonic signal corresponding to the digitised ultrasonic synchronisation signal to generate a representation of the corresponding timecode, and identifying the data-carrying segments412in the representation of the corresponding timecode based on indicators of locations of the data-carrying segments412in the representation.

The foregoing description of embodiments of the present invention has been presented for the purpose of illustration and description. It is not intended to be exhaustive or to limit the invention to the forms disclosed. Alterations, modifications and variations can be made without departing from the spirit and scope of the present invention.