Method for embedding and executing audio semantics

Aspects of the subject disclosure may include, for example, a device that includes a processing system having a processor and a memory that stores executable instructions that, when executed by the processing system, facilitate performance of operations, where the operations include determining parameters for adapting audio in the content to the device, wherein the device renders the content, and wherein the parameters are based on semantic metadata embedded in the content, adapting the audio in the content based on the parameters, and rendering the content, as adapted by the parameters, to represent a semantic in the semantic metadata. Other embodiments are disclosed.

FIELD OF THE DISCLOSURE

The subject disclosure relates to a method for embedding and executing audio semantics.

BACKGROUND

Static content injections (e.g., lighting within a scene or music swells) may be used to provide semantic expressions. However, a diverse set of playback environments may not convey the semantic expression intended authentically.

DETAILED DESCRIPTION

The subject disclosure describes, among other things, illustrative embodiments for rendering semantics in audio content. Other embodiments are described in the subject disclosure.

One or more aspects of the subject disclosure include a device that includes a processing system having a processor and a memory that stores executable instructions that, when executed by the processing system, facilitate performance of operations, where the operations include determining parameters for adapting audio in content to the device, wherein the device renders the content, and wherein the parameters are based on semantic metadata embedded in the content, adapting the audio in the content based on the parameters, and rendering the content, as adapted by the parameters, to represent a semantic in the semantic metadata.

One or more aspects of the subject disclosure include a machine-readable medium, comprising executable instructions that, when executed by a processing system including a processor, facilitate performance of operations, the operations comprising: determining instructions for manipulating audio in content, wherein the instructions are based on semantic metadata embedded in the content; modifying the audio in the content based on the instructions to represent a semantic in the semantic metadata, thereby creating modified content; and sending the modified content to a device for rendering.

One or more aspects of the subject disclosure include a method that includes determining, by a processing system including a processor, instructions for manipulating audio in content, wherein the instructions are determined based on semantic metadata embedded in the content, modifying, by the processing system, the audio in the content based on the instructions; and rendering, by the processing system, the content, wherein the audio modified by the instructions represents a semantic in the semantic metadata.

FIG. 2Ais a block diagram illustrating an example, non-limiting embodiment of a system for embedding and executing audio semantics within the communication network ofFIG. 1in accordance with various aspects described herein. Various elements of system200, which may be implemented on a computing platform, such as a distributed processing environment, are shown inFIG. 2Aand described below:

Content210is audio or audiovisual data that is created by a content creator.

As content210is captured (content capture 220), the creator may want an explicit feeling to be overtly expressed during creation and embedding of audio semantics, but additional modalities besides sound, like visual or haptic, can be expressed. For example, the content creator may wish to express semantics portraying a feeling of “happy,” and may achieve such expression within a home theater with lighting manipulations, low-volume low-frequency audio, and spatialized sound. However, such semantic expression may not map to a mobile device like a phone or smart watch. Instead, these devices may need additional actions like vibration, echo or delay, or visual-timed thermal activation.

Metadata221is a schema or format for device and media control, possibly extended for a user state and other manipulations. The metadata221links to a device model and a semantic database for an experience aligned to the intend effect provided by the content creator.

Alternative or mimic content222is a method for providing alternate audio or pre-analyzed content to use as an example for manipulations of content210. This alternative content222allows the creator to utilize prior work or audio semantics that are challenging to describe directly in the system.

A codec223is the system component that encodes the semantics that were specified by the content creator for a specific piece of content210. In one embodiment, the semantics are encoded as a static or constant set for an entire piece of content210(e.g., “happy”). In another embodiment, the semantics are encoded as continuous evolution of explicit semantics specified, with execution timing within the content210(e.g., “happy at 0.2 seconds, sad at 2.3 seconds”), or a weighted formulation of semantics in either of the above scenarios (e.g. “happy and excited at 0.2 seconds, happy and confused at 1.0 seconds, sad at 2.3 seconds”).

Audio analysis230is a method performed by system200for analyzing components of content from both the content creator as well as content from live playback systems. During audio analysis230, the system200can suggest audio semantics by analyzing captured content210. By using an understanding of what is being captured by cameras, microphones, or other sensors, system200can suggest semantics for encoding and more authentic playback on future systems. With a combination of time and frequency analysis tools, the system looks for similar patterns from prior ontological models while also discovering new patterns in the audio. For example, if the ontology only provides semantics for up-beat pop songs, a heavy baseline or slow and muted brass instrument score would be identified as new semantics.

All components of the audio would be analyzed and used to ascribe characteristic properties to specific semantics, such as pitch231, cadence232, performance semantics233, tempo234, key235, and spectral signature236, as defined below. While audio analysis230captures only a few examples of personal (i.e., as applied to non-melodic elements) and content (i.e., as applied to all elements of content as a whole, or in components), audio analysis230allows those skilled in the art to understand the functional intentions. It should also be noted that content may be segregated into components, both by spatial channels (stereo, left, right, front, etc.) as well as actor channels (the strings, or singers in a song, etc.).

During audio analysis230, system200analyzes the pitch231of speakers, singers, or other non-melodic sound elements of audio. In existing speech analysis art, this may include the formant (f0) of a speaker or any additional resonant harmonics (f1-fN) that may be characteristic of that non-melodic element.

Also, system200may analyze the cadence232(or pauses) and speed of both non-melodic sound elements (i.e. speakers or singers) and melodic elements (i.e. the beats of a drum) to determine the temporal patterns of these elements.

Additionally, system200analyzes performance semantics233of non-melodic sound elements, like speakers and singers. Similar to the traditional musical notation “fortisimo” to mean “very loud,” certain performance semantics may be provided or detected through audio analysis230. Specifically, these performance semantics233can describe emotional elements (happy, sad, upset, etc.), sound-based elements (guttural, nasal, etc.), and even the intensity of elements themselves (valence, arousal, dominance).

Further, system200analyzes the tempo234of individual channels or the overall sound channel. Typically, tempo234is determined by low-frequency (e.g., drum) beats, but tempo234could also be determined by repeating audio segments.

Key235of content210can be determined during audio analysis230by first performing frequency mapping of individual sounds to semitones and from a collection of semitones into a musical key signature that contains one of a set of sharps and flats. While the key235may also indicate typical performance semantics233, the two features are denoted separately because they may correspond to different melodic and non-melodic elements.

System200can also recognize the spectral signature236of content210by analyzing different granularities of spectrum analysis using time-frequency transform functions, like Fourier or wavelet family of functions. These different granularities may vary in their sampling window (i.e., 30 milliseconds, 100 milliseconds, etc.) or they may vary in the granularity of detail (e.g., the audio bandwidth).

Device240is the playback environment (e.g., cell phone, virtual reality system, home theater, etc.) that reproduces the intended semantic from the embedded metadata and may activate different modalities to approximate the intent on the content210being consumed by the end user.

Playback and execution241is a system component that executes the modulations of audio content across different modalities according to the embedded metadata. It uses decoded semantics and a possible calibration stage to faithfully execute these semantics across various playback devices240.

Calibration242is a system component that may be activated to calibrate the content manipulations needed for a new device. Calibration is learned by executing content manipulations specified by the semantics, sampling the manipulated content (either directly by monitoring the audio output digital channel or by an additional microphone on the device240), and computing the needed additional manipulations to achieve the semantic specified in the metadata221by the content creator on a particular device240. For example, calibration242would be triggered if a new combination of device and semantic is encountered after decoding the metadata221for a particular content.

Content manipulation243is the method that modifies the original audio according to the desired semantics, like pitch231, cadence232, performance semantics233, tempo234, key235, and spectral signature236of the original content. Upon reading the semantics embedded in the metadata221from the content creator, system200attempts to faithfully reproduce the characteristics of the semantics specified for a particular device. Content210is manipulated using audio processing algorithms common in the art of signal analysis. In one embodiment, some of those algorithms may directly align to the characteristics detected by audio analysis230component. In another embodiment, content manipulation243algorithms may simultaneously satisfy multiple characteristics defined by a semantic specification. For example, if the semantic is “creepy” the manipulations may lower the pitch and/or reduce the tempo or speed of the content210. In another embodiment, content manipulation243may be realized differently according to the capabilities of the device240. In one example, a home theater device240may have high-powered subwoofers and be capable of producing audio in the range of 5-100 Hz, such that a “booming” semantic can faithfully execute a low-frequency rumble of thunder. However, a second mobile device240may have speakers only capable of producing audio in the range of 2-8 kHz, but it has a physical vibration device, so the content manipulation243would utilize the vibration device in lieu of a subwoofer to faithfully execute the “booming” semantic on a low-frequency rumble of thunder.

External device models244are information stored in a database that describes the required content manipulation243methods to achieve the desired semantic for playback device240. In one embodiment, external device models244are continuously updated with new manipulation instructions from combinations of devices, semantics, and user state250information. In another embodiment, external device models244reside on a distributed or networked database instance, and are synchronized with some determined frequency, or polled as needed by the system200. In another embodiment, external device models244are computed at one time during system creation, and are distributed in a static, non-adapting form for a specific model. In yet another embodiment, content manipulation243instructions for external device models244that do not exist in the database are duplicated from records of similar device models previously stored in the database.

Normal operation of the playback and execution241component may involve the coordination of multiple sub-components, like calibration242, audio analysis230, and content manipulation243. In one embodiment, this may include the small adaptation of content manipulation243instructions from previous external device models244for a newer model device240. In another embodiment, this adaptation could be adapting the content manipulation243instructions to better accommodate the available resources on a new device240that may be underpowered for the previous content manipulation243technique. For example, if a pitch231adjustment is specified by semantics from the codec223, one device may need to use a less precise sampling technique (e.g., lower frequency) because the device has less memory. In another example, one or more underlying content manipulation243algorithms may need to be substituted for a device, such as executing a pitch231adjustment and a tempo234adjustment, by slightly slowing down the audio playback speed in the time domain instead of using advanced frequency-domain methods. In yet another embodiment, the device240may be in a previously unknown user state250that requires additional manipulation according to the semantic. For example, if analysis of the audio via on-device microphone indicates a “noisy environment” but the desired semantic is “serene” for a part of content, the system may use calibration242to first sense that content manipulation243is not effective, and then increase the volume of the content.

User state250is a system component that simplifies the user's overall context into a set of known semantics. This simplification process may be informed by sensors that are associated with the user251, the user's environment context253, or digital signals coming from a game context (i.e., game engine or application). In each of these examples, the task of the user state250component is to inform the playback system about conditions of the user wherever possible. In one embodiment, the semantics conveyed from the user state250may have intensity levels computed by the combining outputs of the user251, environment state253and game engine state252. In another embodiment, historical semantics associated with a particular user and the state of the user during time of semantic determination may be archived into a user preferences database254. The user preferences database254may be used to inform the user state250component of semantics to generate based when only partial information from a subcomponent (user251, game252, or environment253states) are available. The user preferences database254may also include user preferences augmented by signals of affinity as determined by the feedback260sub-component. In another example, the user preferences database254may contain weightings specific to user preferences that inform the semantic combination algorithm within the user state250.

User251is a component that collects and translates sensor data associated with the user into content semantics, where possible. In one embodiment, the sensor data may include biometrics about the user (heart rate, perspiration level, activity level, drowsiness, etc.). This set of biometrics is incomplete and may be further augmented by biometrics and sensor data known to those in the art. In another embodiment, user state250may also convey historical states of the user251that have occurred over time. For example, the state may indicate “serene,” “worried,” and “excited” as previous semantic states that the user251experienced in the last ten minutes.

Game252is a component that maps a digital game state into a possible set of semantics. While this component may not be realized for all content experiences, those with interactive components, such as virtual reality (VR), augmented reality (AR), etc., may have external software as a game engine that further describes the state of the user within the game. For example, if the user is in a tense logic battle with a foe the semantic may indicate “perplexed.” In another example, if the user has just succeeded at a significant game task the semantic may indicate “proud.”

Environment253is a system component that maps physical environment information into semantics. In one embodiment, environment information may be derived from location. For example, a home receives one semantic (“peaceful”) and a busy street receives another (“frantic”). In another embodiment, environment253may receive different semantics according to information about occupancy (“lonely” versus “crowded”), time of year (“solemn” versus “relaxed”), or temperature (“arid” versus “refreshing”). In one embodiment, semantics are provided by a retrieval service that indicates conditions within an environment253around the user and playback location. In another embodiment, semantics are provided by the mapping of other sensor data around the users.

Feedback260may be supplied by a user to validate the user experience. Feedback260may be collected explicitly (rating systems, affinity “like” indicators) or implicitly (alignment of user states to affirming semantics). Feedback260may also include simple signals of affinity or stronger signals indicating explicit suggestions for additional or alternate semantics. In one embodiment, the user may have an interactive slider associated with content playback and execution241that can vary the semantic of content210between one or more of the content creator's original intentions. For example, if the semantics for content210were specified as the triplet “foreboding,” “scary,” and “suspenseful,” the user may be prompted (or may explicitly choose) to provide a preference for one of these semantics. In another embodiment, the user may have controls or indicators that provide feedback260on the executed content manipulation243. In one example, the playback and execution241determines the use of content manipulation243methods to increase the volume of high-frequency components of an animal's scream to match the semantic “scary.” The user may have sensitivity to loud, high-pitched noises, so during operation, feedback may be provided to refrain from using that frequency range, which would be stored as a content manipulation243in either the user preferences database254, the external device models244database, or both databases. In yet another embodiment, the simple successful execution of a semantic during playback and execution241on a device240may be considered a passive form of feedback260and may be collected for subsequent learning stages.

Semantic learning261is a discovery of what manipulations were successful, and can be uploaded to a central (or personal) repository for effects. This sub-component complements the learning of successful content manipulations243on playback devices240. Here, feedback260can help to determine when or if a semantic was realizable (e.g., executed with some content manipulations243on devices240) and its correlation to a user state250. Semantic learning261enhances the semantics known to the system200and its outputs are most commonly utilized during explicit interactions with a user that has recently experienced playback content manipulation243. In one example, this user is also the content creator, who is fine tuning the set of semantics (and their timed execution) for their desired content. In another example, the user is a non-creator, but is interested in additional personalization of their experience with the system at a more general (not content specific) level akin to global user preferences in modern software applications.

Dashboard262is an interface for updating effects, including a generalized ontology for audio semantics. In one embodiment, dashboard262allows a user to explore when semantics were executed on the content210. In a traditional time-line view, the audio could be displayed in a frequency or waveform visualization and time indicators would explicitly indicate semantics and their intensity for execution. In another embodiment, dashboard262may be a numerical representation of a semantics and their executions (statistically aggregated by total execution time or total execution instances) across devices, users, or environments. For example, if the content was a soundtrack to a suspenseful movie, the dashboard may indicate the frequency of the semantic “surprise” or “suspenseful” in a bar chart. In a similar example, the dashboard262may indicate that the low-frequency manipulations required by the semantic “mysterious” were not executed well on mobile devices.

Adaptation263is a component that allows the system200to create or adapt a semantic using all of the properties of another semantic. In one embodiment, a content creator (here, as the user of the playback system as well) may want to create a new semantic that isn't sufficiently captured by the known set of semantic models. In one example, the creator has two sample semantics that should be used as input semantics for a final semantic. The system can analyze the characteristics of these two semantics and derive a combined (possibly with different weightings) semantic for the author to encode with the new content and test on playback. In another example, the content author prefers to describe the semantic with plain language or a series of logic instructions (e.g., “campy, which is suspenseful but more comical than scary”). In this example, the semantic learning system can map such an expression through natural language understanding (NLU) tools to create a new semantic. Afterwards, the author can associate the semantic with the new content and evaluate it on playback.

Ontological semantic models265contain semantics and their descriptions. These descriptions may include textual definitions, utility and usage information, typical content manipulation operations, effected audio characteristics (from audio analysis), and their relationships to other semantics for the system (i.e., the composition rules for the ontology). In one embodiment, the ontology265stores each of these descriptions as a different attribute (or connection) between semantics, such that the ontology265can be organized (and optionally displayed on the dashboard262) according to the user's needs. In another embodiment, the semantics can be mapped into a lexical- or language-based form such that they can be connected with the use of external information sources. In one example, a thesaurus (e.g., WordNet) may be used to make connections between semantics. In a similar example, a logical relational database (e.g., OpenCyc) may define relationships of containment (“has a”) or membership (“is a”) that can used to make connections between semantics. In another example, the text of a written manuscript (e.g., a book, a movie script, a web page, a news article) may be used to determine the connections between semantics. In yet another example, advanced natural language understanding (NLU) algorithms (e.g., word2vec) may be utilized to apply an externally learned semantic embedding to the semantics in the ontological models265. In this example, mathematical operations can be expressed in the embedding space but realized in the semantic space, like “queen minus woman equals king.”

FIG. 2Bdepicts an illustrative embodiment of methods in accordance with various aspects described herein. As shown inFIG. 2B, a method270of capturing content by a content creator is illustrated. RelatingFIG. 2Bto the discussion ofFIG. 2A, the term “adaptation” shall be synonymous used as short-hand for content manipulation243. In step271, the content creator generates new content, and the new content is matched to a semantic specification. In step272, the system determines semantics associated with adapting the semantics to the device adaptation characteristics. Next, in step273, the system fingerprints cue points in the content, which are indexed specifically for each piece of content as metadata, specifically for devices in external device models244, and in aggregation for each semantic ontology265. With these various indexes, playback and execution241can best modify content in playback and the semantic learning system261can be used for playback and execution241of semantics.

Also illustrated inFIG. 2Bis a method280of consuming content adapted to a playback device. As shown inFIG. 2B, and with reference toFIG. 2A, in step281, a user requests playback of content210. In step282, the system optionally updates the game252and environment253state. In step283, the system provides parameters for content manipulation243for the device240, to more accurately represent the semantic in the game252or environment253context.

Then, in step284, the playback system retrieves semantics relevant to the current device240from external device models244that are specific to the content210. In step285, the system provides a semantic response that includes content manipulation243to adapt the content210to the playback device240. In step286, the content manipulation243is sent to the playback system.

Next, in step287, the system plays back the original content210, as adapted by the content manipulation243. Finally, in step288, the user optionally indicates feedback260, or shares a preference for a new adaptation of the content210.

Also illustrated inFIG. 2Bis a method290of playing back content210on a new device having an unknown semantic mapping. As shown inFIG. 2B, in step291, the system discovers a new playback device. In step292, the system optionally updates the game252and environment253state. In step293, the system provides parameters for adapting the content210to the new device, to more accurately represent the semantic.

Then, in step294, the system searches for historical adaptation (playback and execution241of semantics) strategies. These strategies are required because the playback device240has no prior execution instructions for semantics, so the system attempts to start from the most similar previous execution instructions. Many techniques may be utilized, but some examples to associate similarity may be determined by product and model number of the device240, available hardware components on the device240(e.g., the speakers, the size of the body, etc.), form factors of the device240(e.g., positioning of speakers relative to hand and viewing positions).

In step295, the system queries popular recommendations for similar adaptations (successful playback and execution transactions as determined by feedback260) across external device models244, semantic ontology265, or similar users. Popular recommendations may be a fallback for more explicit (and potentially reliable) content manipulation243from an existing device model's execution instructions.

Next, in step296, the system associates the new adaptations (execution instructions) to the new device, based on a known semantic and stores these execution models either locally or in an external (and shared) index. In step297, the new device adaptation is evaluated by the user.

Next, in step298, the system optionally notifies the content creator of the new device adaptation. Finally, in step299, the system updates the device adaptation models stored in the external device database.

Referring now toFIG. 3, a block diagram300is shown illustrating an example, non-limiting embodiment of a virtualized communication network in accordance with various aspects described herein. In particular a virtualized communication network is presented that can be used to implement some or all of the subsystems and functions of communication network100, the subsystems and functions of system200, and methods270,280and290presented inFIGS. 1, 2A, 2B and 3. For example, virtualized communication network300can facilitate in whole or in part data communication paths providing the flow of data in system200illustrated inFIG. 2A.

Turning now toFIG. 4, there is illustrated a block diagram of a computing environment in accordance with various aspects described herein. In order to provide additional context for various embodiments of the embodiments described herein,FIG. 4and the following discussion are intended to provide a brief, general description of a suitable computing environment400in which the various embodiments of the subject disclosure can be implemented. In particular, computing environment400can be used in the implementation of network elements150,152,154,156, access terminal112, base station or access point122, switching device132, media terminal142, and/or VNEs330,332,334, etc. Each of these devices can be implemented via computer-executable instructions that can run on one or more computers, and/or in combination with other program modules and/or as a combination of hardware and software. For example, computing environment400can facilitate in whole or in part components of system200for analyzing and modifying audio in content.

The illustrated embodiments of the embodiments herein can be also practiced in distributed processing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed processing environment, program modules can be located in both local and remote memory storage devices.

Turning now toFIG. 6, an illustrative embodiment of a communication device600is shown. The communication device600can serve as an illustrative embodiment of devices such as data terminals114, mobile devices124, vehicle126, display devices144or other client devices for communication via either communications network125. For example, computing device600can facilitate in whole or in part components of system200illustrated inFIG. 2A.