Electrical devices control based on media-content context

A metadata server that includes circuitry is provided. The circuitry receives a first segment from a plurality of segments of first media content and determines context information associated with the first segment based on a characteristic of at least one frame of a plurality of frames included in the first segment. The circuitry generates first metadata associated with the first segment based on the context information. The first metadata includes timing information corresponding to the determined context information to control a first set of electrical devices. The circuitry further transmits the received first segment and the generated first metadata to a media device associated with the first set of electrical devices.

FIELD

Various embodiments of the disclosure relate to media content creation and rendering. More specifically, various embodiments of the disclosure relate to an apparatus and a method for electrical devices control based on media-content context.

BACKGROUND

Recent advancements in the field of media content reproduction has led to a proliferation of media devices capable of playing-back media content. In certain situations, the media device may have the capability to control one or more smart household devices (such as light, fans, blinds, or furniture). Typically, such smart household devices may be controlled by the media devices based on manual inputs or predefined conditions (for example predefined schedules). In certain situations, the control of the smart household devices may not be resonant with a context of the media content that may be played back by the media device, which may further affect user experience. Thus, there is a need of a smart and intelligent system, which may enhance the user experience, during the control of the smart household devices by the media devices.

SUMMARY

An apparatus and a method for electrical devices control based on media-content context, are provided substantially as shown in, and/or described in connection with, at least one of the figures, as set forth more completely in the claims.

DETAILED DESCRIPTION

The following described implementations may be found in the disclosed apparatus and method for automatic control of one or more electrical devices (e.g., lights, an electrical curtain, an aroma dispenser, or a smart furniture), coupled with a media device (e.g., a television, a display device, or a sound system), based on context information associated with media content played-back on the media device. The media device may control the electrical devices via a controller device that may connect the media device with the electrical devices. Exemplary aspects of the disclosure provide a metadata server for generation of metadata associated with media content. The metadata server may communicate the generated metadata to the media device which may control the one or more electrical devices based on the metadata. The metadata server may be configured to determine context information associated with each segment of a plurality of segments of media content based on a characteristic of at least one frame included in each segment of the media content. In case of the media content as an audio-video (AV) content, the determined context information may indicate whether the corresponding segment of the media content is an action scene, a comedy scene, a romantic scene, a suspense scene, a horror scene, a drama scene, a poetry scene, a party scene, or a dance scene. The metadata server may be further configured to use a machine learning model to generate metadata (i.e. associated with each of the segments) based on the determined context information associated with the corresponding segment. The generated metadata may indicate timing information (for example a time interval) at which the one or more electrical devices may be controlled during the play-back of the corresponding segment of the media content on the media device. The metadata server may be configured to transmit the plurality of segments of the media content and the metadata generated for each segment to the media device for play-back of the media content on the media device. The media device may further play-back the media content and simultaneously control the one or more electrical devices based on the generated metadata, via the controller device.

The media device may be configured to control the one or more electrical device, via the controller device, based on the received metadata (i.e. which may be generated based on the context information of the segments by the metadata server) during the playback of the media content. For example, at the time of playback of a romantic scene (as the context of the segment of the media content) by the media device, the received metadata may indicate that the controller device has to control, the electrical devices (such as dim the lights, activate aroma dispenser, or activate an electrical wind chimes) such that an emotional connection between a viewer (or listener) and the rendered media content may be established or enhanced. Thus, the control of the electrical devices based on the real-time context of the segments of the media content may further enhance rendering experience of the media content for the viewer or listener.

FIG.1is a block diagram that illustrates an exemplary network environment for control of electrical devices based on context information of media content, in accordance with an embodiment of the disclosure. With reference toFIG.1, there is shown a network environment100. The network environment100may include a metadata server102, a media device106, a first set of electrical devices108, and a controller device110. The first set of electrical devices108may include a first electrical device108A, a second electrical device108B . . . , and an Nthelectrical device108N. The metadata server102may include a machine learning (ML) model104. Alternatively, the ML model104may be implemented as a separate electronic device or a server. The metadata server102may be coupled to the media device106, via a first communication network112. As shown inFIG.1, the media device106may be coupled with the controller device110that may be further connected with the first set of electrical devices108, via a second communication network114.

The metadata server102may include suitable logic, circuitry, interfaces, and/or code that may be configured to store the ML model104. In certain embodiments, the ML model104may be separate from the metadata server102. The metadata server102may be configured to receive a first segment from a plurality of segments of first media content. The metadata server102may receive the first segment or the plurality of segments from another server (not shown inFIG.1). Alternatively, the metadata server102may receive the first segment from a memory (such as a memory206inFIG.2) of the metadata server102. The metadata server102may be further configured to determine context information associated with the first segment of the first media content based on a characteristic of at least one frame of a plurality of frames in the first segment. In addition, the metadata server102may be configured to generate first metadata associated with the first segment. The first metadata may include timing information corresponding to the determined context information to further control the first set of electrical devices108. The metadata server102may be further configured to transmit the first segment of the first media content and the generated first metadata to the media device106. Examples of the metadata server102may include, but are not limited to, an Al-based (artificial intelligence-based) server, an application server, a media server, a cloud server, a mainframe server, a database server, a file server, a web server, or other types of servers.

In one or more embodiments, the ML model104may include electronic data, which may be implemented, for example, as a software component, and may rely on code databases, libraries, external scripts, or other logic or instructions for execution of an ML algorithm by a processing device, such as a processor (such as a processor204inFIG.2) or circuitry (such as circuitry202inFIG.2) of the metadata server102. The ML model104may be implemented as code and routines configured to enable a computing device, such as the metadata server102, to perform one or more operations. The one or more operations may include classification of a characteristic of the plurality of frames in the first segment of the first media content into one of a true description or a false description associated with a parameter of the context information associated with the first segment. Additionally, or alternatively, the ML model104may be implemented using hardware including a processor, a microprocessor (e.g., to perform or control performance of the one or more operations), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). In some other instances, the ML model104may be implemented using a combination of hardware and software.

In accordance with an embodiment, the ML model104may be a neural network model. Examples of the neural network model may include, but are not limited to, a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a CNN-recurrent neural network (CNN-RNN), R-CNN, Fast R-CNN, Faster R-CNN, an artificial neural network (ANN), (You Only Look Once) YOLO network, a Long Short Term Memory (LSTM) network based RNN, CNN+ANN, LSTM+ANN, a gated recurrent unit (GRU)-based RNN, a fully connected neural network, a Connectionist Temporal Classification (CTC) based RNN, a deep Bayesian neural network, a Generative Adversarial Network (GAN), and/or a combination of such networks. In some embodiments, the neural network model may include numerical computation techniques using data flow graphs. In certain embodiments, the neural network model may be based on a hybrid architecture of multiple Deep Neural Networks (DNNs). In accordance with another embodiment, the ML model104may include, but is not limited to, a linear regression, a logistic regression, support vector machine (SVM), k-means algorithm, nearest neighborhood algorithm, meta-heuristics model, a decision tree, random forest algorithm, and principal component analysis (PCA) model.

The media device106may include suitable logic, circuitry, interfaces, and/or code that may be configured to playback the first media content. The media device106may transmit, to the metadata server102, a first request which includes at least identification information of the first media content. Further, the media device106may be configured to receive second media content that may include the requested first media content and the generated plurality of metadata for the plurality of segments in the first media content. In an alternate scenario, the media device106may receive the plurality of metadata from the metadata server102and may receive the first media content from another server (e.g., a media server, not shown inFIG.1). The media device106may be further configured to control, via the controller device110, the first set of electrical devices108based on the received metadata, in synchronization with the playback of the first media content on the media device106. In accordance with an embodiment, the controller device110may be integrated in the media device106. Alternatively, the controller device110may be implemented as a device separate from the media device106. Examples of the media device106may include, but are not limited to, a television, a mobile phone, a tablet computing device, a personal computer, a gaming console, a media player, a speaker device, a home theatre system, a digital camera, a head-mounted device, an automotive electrical device, an electrical musical instrument, or other electrical device capable of playback of media content and control one or more electrical devices through wired/wireless communication.

Each of the first set of electrical devices108may include suitable logic, circuitry, and interfaces that may be configured to perform one or more physical or sensory actions based on a capability of the corresponding electrical device. Each of the first set of electrical devices108may perform the action based on a command or a control instruction received from the controller device110. In case the controller device110is integrated with the media device106, the first set of electrical devices108may receive the command or the control instruction from the media device106. For example, the command or the control instruction may correspond to the timing information in the first metadata corresponding to the first segment that may be played-back on the media device106. Examples of the one or more physical or sensory actions may include, but are not limited to, a vibration, a sound effect, a lighting effect, a tactile output, a smell/aroma effect, a mechanical/physical action, a taste effect, or an image/video display. Examples of the first set of electrical devices108may include, but are not limited to, an aroma dispenser, an electrical furniture (e.g., smart sofa or chair), a lighting device (e.g., smart light), a sound reproduction device, an electrical curtain (e.g., blinds), an electrical toy (a smart toy, or a movie/TV series-specific toy), an electrical wind-chime, an electrical vase (e.g., smart pod or smart vase), a digital photo-frame, or an internet of things (IOT) device. In an embodiment, one or more electrical devices from the first set of electrical devices108may be integrated in the controller device110.

The controller device110may include suitable logic, circuitry, interfaces, and/or code that may be configured to control the first set of electrical devices108(i.e. coupled to the controller device110) based on context related metadata associated with media content played back on the media device106. In an example, the controller device110may receive the plurality of metadata corresponding to the first media content from the media device106. In an embodiment, the controller device110may receive subscription information from each of the first set of electrical devices108. The subscription information of an electrical device from the first set of electrical devices108may indicate a subscription of the electrical device to a certain context in the context related metadata. The controller device110may control the first set of electrical devices108based on the metadata (e.g., the plurality of metadata) and the subscription information associated with each of the first set of electrical devices108. In an embodiment, the controller device110may transmit one or more command instructions to the first set of electrical devices108to further control the first set of electrical devices108. In certain embodiments, the controller device110may generate a set of audio fingerprints from the plurality of metadata and may control the first set of electrical devices108based on the generated set of audio fingerprints. Examples of the controller device110may include, but are not limited to, a smart home device, a smart router, an IoT gateway device, a control device, or an electronic device with communication capability.

The first communication network112may include a communication medium through which the metadata server102and the media device106may communicate with each other. Examples of the first communication network112may include, but are not limited to, the Internet, a cloud network, a Wireless Fidelity (Wi-Fi) network, a Personal Area Network (PAN), a Local Area Network (LAN), or a Metropolitan Area Network (MAN). Various devices in the network environment100may be configured to connect to the first communication network112, in accordance with various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of a Transmission Control Protocol and Internet Protocol (TCP/IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), Zig Bee, EDGE, IEEE 802.11, light fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communication, wireless access point (AP), device to device communication, cellular communication protocols, and Bluetooth (BT) communication protocols.

The second communication network114may include a communication medium through which the media device106, the controller device110, and the first set of electrical devices108may communicate with each other. In some embodiment, the controller device110may be an integral part of the second communication network114. In an embodiment, the second communication network114may include a home network or a network established in a physical space where the controller device110and the first set of electrical devices108may be physically situated. Examples of the second communication network114may include, but are not limited to, a home network, a wired network, a Wireless Fidelity (Wi-Fi) network, a Personal Area Network (PAN), a Local Area Network (LAN). Various devices in the network environment100may be configured to connect to the second communication network114, in accordance with various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of a Transmission Control Protocol and Internet Protocol (TCP/IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), Zig Bee, EDGE, IEEE 802.11, light fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communication, wireless access point (AP), device to device communication, cellular communication protocols, and Bluetooth (BT) communication protocols. In accordance with an embodiment, the first communication network112and the second communication network114may be the same network or may include at least one overlapping network portion. Alternatively, the first communication network112and the second communication network114may be separate communication networks as shown inFIG.1.

In operation, the metadata server102may be configured to receive a first request from the media device106. The first request may include an identification of the first media content (for example an AV content or audio content). In an embodiment, the media device106and the first set of electrical devices108may be present in a physical space (such a room). The first media content may be stored in the memory (such as the memory206inFIG.2) of the metadata server102. Alternatively, the metadata server102may receive the first media content from another server (e.g., a media server, not shown inFIG.1). Further, the metadata server102may be configured to receive the first segment of the plurality of segments of the first media content from the memory206of the metadata server102or from the media server. The metadata server102may be further configured to determine the context information associated with the first segment of the first media content based on a characteristic of at least one frame of the plurality of frames included in the first segment. In accordance with an embodiment, the metadata server102may apply the stored ML model104on the characteristic of at least one frame of the plurality of frames in the first segment to determine the context information associated with the first segment.

In case the first media content corresponds to the audio content, the characteristic of the at least one frame in the first segment may include, but is not limited to, a loudness parameter, a pitch parameter, a tone parameter, a rate-of-speech parameter, a voice quality parameter, a phonetic parameter, an intonation parameter, an intensity of overtones, a voice modulation parameter, a pronunciation parameter, a prosody parameter, a timbre parameter, or one or more psychoacoustic parameters. In such case, the context information of the first segment of the audio content may indicate, but is not limited to, a song, a musical tone, a monologue, a dialogue, a laughter sound, a distress sound, a pleasant sound, an unpleasant sound, an ambient noise, a background sound, a loud sound, or defined sound pattern associated with a real-time object.

Further, in case the first media content corresponds to video content or audio-video (AV) content, the characteristic of the at least one frame in the first segment may include, but is not limited to, an object recognized in the at least one frame, a person recognized in the at least one frame, an emotional state of at least one object in the at least one frame, background information of the at least one frame, an ambient lighting condition in the at least one frame, motion information of at least one object in the at least one frame, a gesture associated with at least one object in the at least one frame, or genre information associated with the at least one frame. In such case, the context information of the first segment of the video content or the AV content may indicate, but is not limited to, an action scene, a comedy scene, a romantic scene, a suspense scene, a horror scene, a drama scene, a poetry scene, a party scene, or a dance scene. The details of the characteristics and the context information for the audio content and/or the AV content is described, for example, inFIGS.3and5.

The metadata server102may be further configured to generate the first metadata associated with the first segment based on the determined context information. In an embodiment, the first metadata may include timing information to control the first set of electrical devices108. The metadata server102may be further configured to transmit the first segment of the first media content and the determined first metadata to the media device106associated with the first set of electrical devices108. The first metadata and the control of the first set of electrical devices108are described, for example, inFIGS.3and6. In some embodiments, the metadata server102may be configured to generate the plurality of metadata for the plurality of segments in the first media content.

In accordance with an embodiment, the metadata server102may be configured to transmit, second media content which may include the first media content and the plurality of metadata, to the media device106. Each metadata in the plurality of metadata may be associated with a corresponding segment of the plurality of segments of the first media content. Each metadata may be generated based on the context information (associated with the corresponding segment) which may be determined based on the characteristic of the at least one frame (that may include audio and/or video content) in the corresponding segment.

FIG.2is a block diagram that illustrates an exemplary metadata server to generate metadata information to control electrical devices based on context information of media content, in accordance with an embodiment of the disclosure.FIG.2is explained in conjunction with elements fromFIG.1. With reference toFIG.2, there is shown a block diagram200of the metadata server102that may be coupled to the media device106, via the first communication network112. The metadata server102may further include circuitry202which may include one or more processors, such as a processor204. The metadata server102may further include a memory206which may be configured to store the ML model104, and a media content database208. The metadata server102may further include a network interface210. The circuitry202may be connected with the memory206and the network interface210through wired or wireless communication of the metadata server102.

The circuitry202may include suitable logic, circuitry, and/or interfaces that may be configured to execute program instructions associated with different operations to be executed by the metadata server102. For example, some of the operations may include determination of the context information associated with the first segment of the first media content. The circuitry202may be further configured to generate the first metadata associated with the first segment and transmit the generated first metadata and the first segment to the media device106. The circuitry202may include one or more specialized processing units, which may be implemented as a separate processor. In an embodiment, the one or more specialized processing units may be implemented as an integrated processor or a cluster of processors that perform the functions of the one or more specialized processing units, collectively. The circuitry202may be implemented based on a number of processor technologies known in the art. Examples of implementations of the circuitry202may be an X86-based processor, a Graphics Processing Unit (GPU), a Reduced Instruction Set Computing (RISC) processor, an Application-Specific Integrated Circuit (ASIC) processor, a Complex Instruction Set Computing (CISC) processor, a microcontroller, a central processing unit (CPU), and/or other control circuits.

The processor204may include suitable logic, circuitry, and/or interfaces that may be configured to execute a set of instructions stored in the memory206. The processor204may be configured to execute program instructions associated with different operations to be executed by the metadata server102. For example, some of the operations may include the determination of the context information, the generation of the first metadata, and the transmission of the first metadata and the first segment. The processor204may be implemented based on a number of processor technologies known in the art. Examples of the processor technologies may include, but are not limited to, a Central Processing Unit (CPU), X86-based processor, a Reduced Instruction Set Computing (RISC) processor, an Application-Specific Integrated Circuit (ASIC) processor, a Complex Instruction Set Computing (CISC) processor, a Graphical Processing Unit (GPU), and other processors.

The memory206may comprise suitable logic, circuitry, interfaces, and/or code that may be configured to store the ML model104. The memory206may be further configured to store the media content database208. The media content database208may store the first media content. In some embodiments, the memory206may be configured to store the generated first metadata or the plurality of metadata. Examples of implementation of the memory206may include, but are not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Hard Disk Drive (HDD), a Solid-State Drive (SSD), a CPU cache, and/or a Secure Digital (SD) card.

The network interface210may comprise suitable logic, circuitry, interfaces, and/or code that may be configured to facilitate communication between the metadata server102and the media device106, via the first communication network112. The network interface210may be implemented by use of various known technologies to support wired or wireless communication of the metadata server102with the first communication network112. The network interface210may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identity module (SIM) card, or a local buffer circuitry.

FIG.3is a diagram that illustrates an exemplary scenario for generation of metadata by a metadata server ofFIG.2, to control electrical devices based on context information of media content, in accordance with an embodiment of the disclosure.FIG.3is described in conjunction with elements fromFIG.1andFIG.2. With reference toFIG.3, there is shown an exemplary scenario300that may include first media content302. The first media content302may include a plurality of segments, such as a first segment302A, a second segment302B . . . and an Nth segment302N. Each segment of the plurality of segments (such as, the first segment302A shown inFIG.3) may include video content304and audio content306associated with the corresponding segment. In case of the audio content as the first media content302, each segment may include only the audio content306. Further inFIG.3, there is also shown, a first time instance308A (referred as “T1”) and a second time instance308B (referred as “T2”) which may be associated with a start time and an end time of the first segment302A within the first media content302.

InFIG.3, there is further shown the metadata server102that may include the ML model104. The ML model104may be stored in the memory206of the metadata server102. There is also shown second media content316that may be generated from the first media content302by the metadata server102using the ML model104. The second media content316may include a plurality of metadata-embedded (ME) segments, such as, a first ME segment316A (corresponding to the first segment302A), a second ME segment316B (corresponding to the second segment302B) . . . and an Nth ME segment316N (corresponding to the Nth segment302N). Each of the plurality of ME segments (such as the first ME segment316A) may include metadata310(hereinafter, referred as first metadata310for the first segment302A), as shown inFIG.3. The timing or length of the first metadata310may correspond to the time interval between the first time instance308A and the second time instance308B. In some embodiments, the timing of the first metadata310may correspond to a portion of the time interval between the first time instance308A and the second time instance308B. Further, as shown inFIG.3, the first metadata310(e.g., the first metadata associated with the first segment302A) may include timing information312and context type information314.

With reference toFIG.3, the circuitry202of the metadata server102may be configured to retrieve the first media content302from the memory206. In some embodiments, the circuitry202may receive the first media content302from a computing device (such as, the media device106, or another server not shown inFIG.1). Further, the circuitry202may store the received first media content302in the memory206for further creation of the metadata (such as the first metadata310).

In some embodiments, the circuitry202may be configured to retrieve the first media content302, based on to the receipt of the first request from the media device106. The first request may include the identification information that may uniquely identify the first media content302and/or indicate a storage location (e.g. a memory location or a Uniform Resource Locator (URL)) of the first media content302in the memory206. The first request received from the media device106may be a request to generate the plurality of metadata to further control the first set of electrical devices108based on the generated plurality of metadata. In some embodiments, the metadata server102may generate and store the plurality of metadata for the first media content302without a receipt of the first request from the media device106. In some embodiments, based on the receipt of the first request from the media device106, the circuitry202may retrieve the pre-generated metadata and transmit the retrieved and pre-generated metadata to the media device106to control the first set of electrical devices108along with the playback of the first media content302.

In certain embodiments, the circuitry202of the metadata server102may be configured to transmit, to a media server, the request which includes identification information of the first media content302. Based on the transmitted request, the circuitry202may receive the first media content302including the first segment302A or the plurality of segments from the media server. The circuitry202may store the received first media content302in the memory206and generate the first metadata310associated with the first segment302A. The circuitry202may then transmit the received first segment302A of the first media content302and the generated first metadata310to the media device106.

In accordance with an embodiment, the circuitry202may be further configured to retrieve the first segment302A from the plurality of segments (e.g., the first segment302A to the Nth segment302N) of the first media content302. In an embodiment, each of the plurality of segments, such as,302A to302N, may correspond to a portion of the first media content302. Each segment may be of a pre-defined time duration that may be defined at a time of capture or creation of the first media content302. In an embodiment, the time duration may vary from a few seconds to minutes. Each of the plurality of segments,302A to302N, may include a pre-defined number of frames. A frame may correspond to a smallest unit of capture of a video or audio. For instance, in case of a video, a frame may include a single image captured at a particular time. Further, in case of audio, a frame may include a set of audio samples captured at a particular time. The number of frames in a segment may depend on a capture rate (i.e. frame rate) of the video or audio in the segment. For instance, in case, the first segment302A includes the video content304, the time duration of the first segment302A may correspond to the time interval between the first time instance308A and the second time instance308B. The video content304of the first segment302A may include pre-defined number of video frames (say, 300 frames, in the 10 seconds segment captured at the rate of 30 frames per second).

In accordance with an embodiment, the circuitry202may be further configured to determine the context information associated with the first segment302A. The context information may be determined based on a characteristic of at least one frame of the plurality of frames included in the first segment302A. In some embodiments, the context information may be determined based on a set of frames of the plurality of frames included in the first segment302A. In accordance with an embodiment, the circuitry202may apply the ML model104on the characteristic of each of the plurality of frames in the first segment302A to determine the context information associated with the first segment302A.

For the application of the ML model104, the circuitry202may be configured to extract each of the plurality of frames of the video content304and the audio content306in the first segment302A (between the first time instance308A and the second time instance308B). In some embodiments, each of the plurality of frames may include image content (as the video content304) and the audio content306(stored in synchronization with the image content) such that both the video content304and the audio content306may be played back concurrently. For example, the combination of the video content304and the audio content306in the first segment302A may include, but is not limited to, a video clip, a movie scene, or a video shot. The circuitry202may further determine the characteristic of at least one frame of the plurality of image/video frames (of the video content304) and of at least one frame of the plurality of audio frames (of the audio content306), based on one or more image processing and audio processing techniques, respectively.

In case of the video content304, the characteristic of the at least one frame (i.e. image frame) in the first segment302A (or other segments) may include, but is not limited to, an object recognized in the at least one frame, a person recognized in the at least one frame, an emotional state of at least one object in the at least one frame, background information of the at least one frame, an ambient lighting condition in the at least one frame, motion information of at least one object in the at least one frame, a gesture associated with at least one object in the at least one frame, or genre information associated with the at least one frame. In accordance with an embodiment, the stored ML model104may be pre-trained based on different types of characteristics of the video content304to determine the context information of the first segment302A. The context information of the video content304(or image content) of the first segment302A may indicate a type of scene or shot referred by the first segment302A. Examples of the context information may include, but is not limited to, an action scene, a comedy scene, a romantic scene, a suspense scene, a horror scene, a drama scene, a poetry scene, a party scene, or a dance scene.

In case of the audio content306in the first media content302, the characteristic of the at least one frame (i.e. audio frame) in the first segment302A may include, but is not limited to, a loudness parameter, a pitch parameter, a tone parameter, a rate-of-speech parameter, a voice quality parameter, a phonetic parameter, an intonation parameter, an intensity of overtones, a voice modulation parameter, a pronunciation parameter, a prosody parameter, a timbre parameter, or one or more psychoacoustic parameters. In accordance with an embodiment, the stored ML model104may be pre-trained based on different types of characteristics of the audio content306to further determine the context information of the first segment302A. Examples of the context information of the audio content306in the first segment302A may include, but is not limited to, a song, a musical tone, a monologue, a dialogue, a laughter sound, a distress sound, a pleasant sound, an unpleasant sound, an ambient noise, a background sound, a loud sound, or defined sound pattern associated with a real-time object. In accordance with an embodiment, the circuitry202may be configured to determine the context information of the first segment302A based on the application of the pre-trained (or stored) ML model104on the determined characteristics of the video content304(image content) and the determined characteristics of the audio content306in the first segment302A.

For example, in case, the circuitry202detects an object included in the at least one frame of the first segment302as a human being, and further recognizes the human being as a celebrity (i.e. characteristic), the circuitry202may determine the context (as a comedy scene) of the first segment302A based on profile information (i.e. comedy artist) of the recognized celebrity. The circuitry202may detect the objects as human being from the plurality of frames of the first segment302A based on the image processing techniques or object detection techniques known in the art. The detailed implementation of the aforementioned technique may be known to one skilled in the art, and therefore, a detailed description for the aforementioned technique has been omitted from the disclosure for the sake of brevity

In another example, the circuitry202may determine one or more facial/emotional expressions (i.e. characteristics) of people detected in the plurality of frames of the first segment302A and may further determine the context of the video content304of the first segment302A. For instance, in case of happy, smiling or laughter emotions, the circuitry202may determine the context of the video content304as, for example, a comedy scene or a romantic scene.

In another example, the circuitry202may determine an ambient lighting condition in the plurality of frames in the first segment302A based on the image processing techniques known in the art to identify background brightness or the ambient lighting condition in the frames. In case, the circuitry202detects dim or low lighting conditions (i.e. characteristics) with frightened face expressions or emotions (i.e. characteristics) of the detected people in the frames, the circuitry202may determine the context of the video content304of the first segment302A as (for example) a suspense/thriller scene or a horror scene.

In some embodiments, the circuitry202may identify gestures (i.e. characteristics) of at least one object (such as people) detected in the plurality of frames of the first segment302A to determine the context of the video content304. For example, if a particular pattern of hand movement, body movement, or feet movement of one or more persons is detected within multiple frames of the video content304, the circuitry202may determine the context of the video content304as, for example, a dance scene.

In certain embodiments, the circuitry202may recognize a human being in the plurality of frames of the first segment302A as a celebrity (i.e. characteristics) The circuitry202may further analyze the profile information of the recognized celebrity to identify a genre (i.e., characteristics) associated with the celebrity as the genre associated with the plurality of frames of the first segment302A. For example, in case the celebrity is a drama artist, the circuitry202may determine the genre of the first segment302A as a drama genre. Examples of the genre may include, but is not limited to, at least one of an action genre, an adventure genre, an animation/anime genre, a biopic genre, a children genre, a comedy genre, a crime/detective/spy genre, a documentary genre, a drama genre, a horror genre, a family genre, a fantasy genre, a historical genre, a matured content genre, a medical genre, a musical genre, a paranormal genre, a romance genre, a sport genre, a science-fiction genre, a talk show genre, a suspense/thriller genre, a war genre, or a western genre. The circuitry202may further determine the context information of the first segment302A based on the determined genre of the people detects in the plurality of frames in the first segment302A. In another example, the circuitry202may recognize one or more objects in the plurality of frames of the video content304as animated objects. In such case, the circuitry202may determine the context information of the first segment302A as an animated scene or a kid's movie scene.

In some embodiments, the circuitry202may configured to determine the context of the first segment302A based on the combination of multiple characteristics determined from the plurality of frames in the first segment302A. For example, in case, the circuitry202detects multiple people in the plurality of frames of sports genre (for example related to cricket) as a first characteristic, and detects background information in the plurality of frames as a playground (i.e. a second characteristic), then the circuitry202may detect the context of the first segment302A as a cricket match scene (i.e. sport scene) based on the determined first characteristic and the second characteristic. In another example, in case, the circuitry202detects a vehicular object (as the first characteristics) in at least one frame and detects motion information (i.e. second characteristic) of the detected vehicle object in the plurality of plurality of frames of the first segment302A, then the circuitry202may detect the context of the first segment302A as vehicle movement, based on the determined first characteristic and the second characteristic.

In accordance with an embodiment, the circuitry may further determine the characteristics of the audio content306of the first segment302A to determine the context of the first segment302A of the first media content302. For example, in case, the circuitry202detects a high pitch sound as the loudness/pitch parameter (i.e. characteristics), the circuitry202or the ML model104may determine the context of the audio content306of the first segment302A as a loud sound or a distress sound. In another example, in case, the circuitry202detects audio beats (i.e. characteristics) at certain frequencies in the audio content306in the frames of the first segment302A, the circuitry202may determine the context of the audio content306as a musical tone, a song, a pleasant sound, or a background score. In another example, based on rate-of-speech, types of pronunciations, and voice modulation (i.e. characteristics) detected from the frames in the audio content306of the first segment302A, the circuitry202may determine the context of the audio content306as a monologue or a dialogue. In an embodiment, the circuitry202may apply the ML model104to detect if the frames of the audio content306may correspond to defined sound patterns (i.e. characteristics) associated with real-time objects, such as, but not limited to, automobile engine sounds, airplane sounds, or animal sounds. The circuitry202may identify the real-time objects based on the sound patterns and further determine the context information, for example, such as a presence of sound of the airplane in the audio frame. Furthermore, the circuitry202may detect the laughter sound (for example) based on analysis of the plurality of frames in the audio content306of the first segment302A using the ML model104. Thus, the ML model104may be pre-trained based on different image or audio characteristics to determine the context of the first segment302A. The circuitry202may be configured to determine the image or audio characteristics from the plurality of frames in the first segment302A and may apply the stored or pre-trained ML model104on the determined image or audio characteristics to determine the context information of the first segment302A.

In one or more embodiments, the circuitry202may determine the context information of the first segment302A based on a combination of the context information determined for the video content304and the context information for the audio content306. For example, if the context information of the video content304is determined as an action context and the context information of the audio content306is determined as a distress sound, the context information of the first segment302A may be determined as an action context or a fight sequence context. In case, the circuitry202determines that the audio content306corresponds to a loud sound (i.e., context information of the audio content306) and the video content304corresponds to hand and arm gestures (i.e., characteristics associated with context information of the video content304), the circuitry202may determine the context information of the first segment302A as a party or a dance context.

In another example, the circuitry202may recognize two or more human beings in the plurality of frames of the video content304of the first segment302A as celebrities, such as, actors (i.e. characteristics). The circuitry202may further categorize the two or more recognized human beings as actors in positive and negative roles respectively, based on the stored profile information of the two or more recognized human beings. Further, in case, the circuitry202detects hand gestures (i.e., characteristics) of the two or more recognized human beings from the plurality of frames and detects loud human sound (i.e. characteristics) from the audio content306, the circuitry202may determine the context information of the first segment302A as, for example, an action context.

In another example, the circuitry202may recognize a human being in the plurality of frames of the video content304of the first segment302A as a sports person (i.e., characteristics) and may determine that the audio content306corresponds to a loud sound (i.e., context information of the audio content306). For example, in case, the loud sound may correspond to cheering a crowd, then the circuitry202may determine the context information of the first segment302A as a sports context.

In an embodiment of the media device106, the media device106may be configured store or receive subscription information associated with each of the first set of electrical devices108. The subscription information may indicate a subscription of each of the first set of electrical devices108with one or more contexts (for example the determined context information of the first media content302). The media device106may control the first set of electrical devices108, via the controller device110, based on each metadata (i.e. the first metadata310that may include the determined context information) and the subscription information associated with each of the first set of electrical devices108. The control of the first set of electrical devices108based on the first metadata310and the subscription information is described, for example, inFIG.5.

For example, in case, the context information of the first segment302A corresponds to a romantic scene, and a smart light508and blinds512have subscribed for romantic scene context with the controller device110or with the media device106, then the media device106may control the smart light508and the blinds512, via the controller device110. In certain scenarios, the smart light508may operate in a dim light mode and the blinds512may operate in a closed state in case the context information of the first segment302A is a romantic scene. In an example, the media device106may enable, via the controller device110, an aroma dispenser526to dispense a pre-determined smell (such as, a fragrance of a rose) for a duration of the first segment302A associated with the romantic scene, in case the aroma dispenser526has pre-subscribed with the controller device110or with the media device106for a romantic scene context.

In another example, in case, the context information associated with the first segment302A corresponds to a motion of a vehicle that includes one or more individuals and a smart-sofa506(i.e. one of the first set of electrical devices108) has pre-subscribed with the controller device110(or with the media device106) for such context of moving objects, then the media device106may control, via the controller device110, the smart-sofa506to vibrate/shake to provide a real-time vehicular motion experience to a user504(as shown inFIG.5), to whom the first segment302A may be rendered. For example, the media device106or the controller device110may transmit a command to the smart-sofa506associated with the context of the moving object. Based on the received command, the smart-sofa506may vibrate or shake in a predefined pattern.

In yet another example, in case, the circuitry202of the metadata server102recognizes a celebrity as an action hero/super-hero (i.e. characteristics) from the first segment302A and the context information as an action scene, and further a smart celebrity toy524(one of the first set of electrical devices108) has pre-subscribed with the controller device110or the media device106for such an action scene context, then the media device106may set the smart celebrity toy524, via the controller device110, to an active or ON mode to demonstrate the action scene. The smart celebrity toy524may be configured to perform one or more pre-defined actions, such as, jump, kick, bend, walk, and the like, in the active/ON mode. In certain scenarios, the smart celebrity toy524may be configured to mimic one or more sound patterns associated with the video content304of the first segment302A.

At the time of playback of the first media content302, the control of the first set of electrical devices108may synchronize actions (physically, visually, or audibly) between the rendered first media content302and the controlled first set of electrical devices108. This may further enhance content experience of a viewer or listener (for example, the user504inFIG.5) for the first media content302as the controlled first set of electrical devices108may express or resonate the mood or emotions of the different portions of the rendered first media content302. Therefore, the viewer or listener may emotionally connect with the rendered first media content302and/or may feel as a part of the rendered first media content302.

In accordance with an embodiment, the first metadata310(i.e. generated by the circuitry202of the metadata server102for the first segment302A) may be included in the second media content316, as shown inFIG.3. The generated first metadata310may be included in the first ME segment316A of the second media content316. In some embodiments, the generated first metadata310may be time-synchronized with the first segment302A such that the timings of the first metadata310may be associated with the timing between the first time instance308A and the second time instance308B (as shown inFIG.3). Further, as shown inFIG.3, the first metadata310for the first segment302A may include timing information (for example, the timing information312) based on which the selected first set of electrical devices108may be controlled by the media device106(either directly or via the controller device110).

In accordance with an embodiment, the timing information312in the first metadata310may indicate exact time at which one or more of selected first set of electrical devices108may be controlled during the playback of the first segment302A by the media device106. In some embodiments, the timing information312may indicate the time interval between which the first set of electrical devices108may be controlled. For example, the time interval indicated by the timing information312may be same as a time interval between the first time instance308A and the second time instance308B of the first segment302A of the first media content302. In some embodiments, the timing information312may be the time interval which may be a portion (for example in certain percentage) of the time interval between the first time instance308A and the second time instance308B. In some embodiments, the time interval indicated by the timing information312may include multiple time intervals (within the first segment302A) which may vary based on the determined context information.

For example, in case multiple contexts been determined within the first segment302A (say with total time period of 20 seconds), such as a romantic context between T1and 10thsecond and a comedy context between 11thsecond to T2, the circuitry202may determine multiple metadata (for example, first sub-metadata and second sub-metadata) included in the first metadata310for the first segment302A. Examples of multiple metadata with multiple time intervals for the first segment302A is presented in Table 1:

TABLE 1Exemplary values of information in first metadata 310MetadataTiming InformationContext TypeFirst Sub metadata ofT1-10thsecondRomanticthe first metadata 310contextSecond Sub metadata of11thsecond-T2Comedythe first metadata 310context

For example, as shown in the Table 1, for the duration between T1to 10thsecond, the first sub metadata of the first metadata310may indicate a romantic context. Further, for the duration between 11thsecond to T2, the second sub metadata of the first metadata310may indicate a comedy context. In a certain scenario, in case, the smart light508is subscribed for the romantic context, while the blinds512are subscribed for the comedy context, then the media device106(either directly or via the controller device110) may control the smart light508between T1to 10thsecond of the playback of the first segment302A and control the blinds512between the 11thsecond to T2of the playback of the first segment302A. For example, between the T1to 10thsecond, the smart light508may be enabled to produce a dim light to enhance the subscribed romantic context, and between the 11thsecond to T2of the playback of the first segment302A, the blinds512may be enabled to operate in an “open” state to enhance the subscribed comedy context. Thus, an emotional timeline related to the first segment302A of the first media content302may be provided by the controlled first set of electrical devices108(for example, the smart light508and the blinds512). This further provides an enhanced user experience based on the creation of the emotional context (i.e. associated with the played-back first segment302A) by the controlled first set of electrical devices108.

In one or more embodiments, the circuitry202may be further configured to identify a type of the first segment302A or the first media content302. The type of the first segment302A may be identified based on an indicator embedded in the first media content302. Examples of the type of the first segment302A may include, but are not limited to, an audio type segment (which may only include the audio content306), a video type segment (which may only include the video content304or image content), an audio-video (AV) type segment (which may include both the audio content306and the video content304), or a gaming type segment (which may include the audio content306and the video content304with animated objects). In accordance with an embodiment, the circuitry202may generate the first metadata310associated with the first segment302A based on the identified type. In an embodiment, the type of the first segment302A may correspond to a genre of content represented by the first segment302A of the first media content302.

For example, in case the identified type of the first segment302A is an audio type, then electrical devices, such as, smart lights, blinds, and digital photo-frame may be controlled based on the subscription of such electrical devices to the context information (for example pleasant sound) associated with the first metadata310. In such example, the smart lights may operate in a “bright light” mode, the blinds may operate in an “open state” and the digital photo-frame may display pictures that may be likable by the user504based on predefined capability of the electrical devices subscribed to particular contexts.

In another example, in case the identified type of the first segment302A is an AV type, electrical devices such as, a smart-sofa506, smart lights508, blinds512, or an aroma dispenser526may be controlled based on the subscription of such electrical devices to the context information (for example vehicle racing scene) associated with the first metadata310. In such example, the smart-sofa506may vibrate in a certain pattern, the smart lights508may operate in a pulsating light mode, the blinds512may operate in a closed mode, and the aroma dispenser526may dispense a smell of motor oil or tire rubber.

In another example, consider that the circuitry202identifies that the type of the first segment302A is a gaming video or an animated video type, the electrical device, such as a robotic toy520(or a super-hero toy) may be controlled based on the subscription of such electrical device to the context information associated with the first segment302A. In such case, the robotic toy520(or the super-hero toy) may be controlled according to a character present in the gaming or animated video (for example, the robotic toy520may mimic the character or perform one or more pre-defined actions, such as jump, bend, or walk, on occurrence of a particular event in the game or the animate video).

In accordance with an embodiment, based on the generation of the first metadata310, the circuitry202may be configured to transmit the first metadata310and the first segment302A to the media device106coupled with the first set of electrical devices108. The media device106may control the first set of electrical devices108, via the controller device110, based on each received metadata (e.g., the first metadata310) and the subscription information associated with each of the first set of electrical devices108. The first metadata310may include the timing information312and the context type information314that may be used to control the first set of electrical devices108, while the first segment302A may be played-back on the media device106as described, for example, inFIG.5.

In one or more embodiments, the circuitry202may be configured to retrieve the plurality of segments (which may include segments from, the first segment302A to the Nth segment302N) from the first media content302stored in the memory206. The circuitry202may be further configured to generate a plurality of metadata, each of which corresponds to a segment, from the plurality of segments. Each metadata of the plurality of metadata may be generated based on the determination of the context information of the corresponding segment. The generation of each of the plurality of metadata may be performed in a manner similar to the generation of the first metadata310associated with the first segment302A as described above with respect toFIG.3.

In accordance with an embodiment, the circuitry202may be further configured to generate the second media content316from the first media content302based on the generated plurality of metadata. For example, the circuitry202may embed or include the generated plurality of metadata within the first media content302to generate the second media content316. The second media content316may include the plurality of ME segments (e.g., the first ME segment316A to the Nth ME segment316N), which may include corresponding embedded metadata. For instance, the first ME segment316A may include the first metadata310as shown inFIG.3. Further, the circuitry202of the metadata server102may be configured to transmit the second media content316to the media device106, for control of the first set of electrical devices108based on the generated plurality of metadata, at the time of playback of the first media content302(i.e. included in the second media content316). In some embodiments, based on the receipt of the first request from the media device106, the metadata server102may transmit the first media content302and the generated plurality of metadata separately, rather than including both in the second media content316. In some embodiments, the first request received from the media device106may only indicate a request to generate the plurality of metadata for the first media content302. In such case, the first media content302may be prestored in the media device106and the metadata server102may transmit only the generated plurality of metadata based on the received first request.

In an exemplary scenario, the media device106may receive the first media content302along with the plurality of metadata as the second media content316from a single server, such as, the metadata server102. In another scenario, the media device106may receive the first media content302from a media server (not shown inFIG.1) and may receive the plurality of metadata for the first media content302from another metadata server (such as, the metadata server102). In yet another scenario, the media device106may store the first media content302on a memory location of a computing device (such as another media server or media rendering device). The media device106may transmit the memory location within the first request to the metadata server102. The circuitry202of the metadata server102may be configured to retrieve the first media content302from the memory location, generate the plurality of metadata and transmit the second media content316to the media device106.

FIG.4is a block diagram that illustrates an exemplary media device to control electrical devices based on context information of media content, in accordance with an embodiment of the disclosure.FIG.4is explained in conjunction with elements fromFIGS.1,2, and3. With reference toFIG.4, there is shown a block diagram400of a media device (for example the media device106). The media device106may include circuitry402, which may include one or more processors, such as a processor404. The media device106may further include a memory406and an I/O device408that may further include a display device410, a speaker412, and a user-interface (UI)414. The media device106may further include an image capturing device416and a network interface418through which the media device106may be connected to the metadata server102and the first set of electrical devices108, via first communication network112and the second communication network114, respectively. The media device106may be further coupled with the controller device110to control the first set of electrical devices108, via the controller device110and second communication network114. In another embodiment, the controller device110may be integrated within the media device106(not shown inFIG.4), such that the media device106may directly control the first set of electrical devices108, via the second communication network114.

The circuitry402may comprise suitable logic, circuitry, interfaces and/or code that may be configured to playback media content (for example the first media content302). The circuitry402may communicate with the metadata server102to transmit the first request including at least the identification information of the first media content, via the first communication network112. Further, the circuitry402may be configured to receive the second media content316that may include the first media content302and the plurality of metadata as described, for example, inFIG.3. The circuitry402may be configured to control the first set of electrical devices108, via the controller device110, in synchronization with the playback of the second media content316on the I/O device408based on the timing information312and context type information314indicated in each metadata of the plurality of metadata. The circuitry402may be configured to transmit a trigger command or control instructions to the first set of electrical devices108(based on the subscription information) to control the first set of electrical devices108, via the controller device110and the second communication network114.

The processor404may comprise suitable logic, circuitry, and interfaces that may be configured to execute instructions stored in the memory406. In certain scenarios, the processor404may be configured to execute the aforementioned operations of the circuitry402. The processor404may be implemented based on a number of processor technologies known in the art. Examples of the processor404may be a Central Processing Unit (CPU), X86-based processor, a Reduced Instruction Set Computing (RISC) processor, an Application-Specific Integrated Circuit (ASIC) processor, a Complex Instruction Set Computing (CISC) processor, a Graphical Processing Unit (GPU), other processors, or a combination thereof.

The memory406may comprise suitable logic, circuitry, interfaces, and/or code that may be configured to store the first media content302or the second media content316to be played-back on the I/O device408. The memory406may be further configured to store the first metadata or the plurality of metadata. In some embodiments, the memory406may store the subscription information of each of the first set of electrical devices108which may indicate the subscription of the electrical device with one or more contexts of the media content. The first set of electrical devices108may subscribe with different contexts (for example, but not limited to, romantic, action, comedy, or action) based on their operational capability. Examples of implementation of the memory406may include, but are not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Hard Disk Drive (HDD), a Solid-State Drive (SSD), a CPU cache, and/or a Secure Digital (SD) card.

The I/O device408may comprise suitable logic, circuitry, interfaces, and/or code that may be configured to receive an input and provide an output based on the received input. The I/O device408may include various input and output devices, which may be configured to communicate with the circuitry402. Examples of the I/O device408may include, but are not limited to, a touch screen, a keyboard, a mouse, a joystick, a display device (for example, the display device410), a microphone, and a speaker (for example, the speaker412).

The display device410may comprise suitable logic, circuitry, and interfaces that may be configured to display an output of the media device106. The display device410may be utilized to render the UI414. In some embodiments, the display device410may be an external display device associated with the media device106. The display device410may be a touch screen which may enable a user to provide a user-input via the display device410. The touch screen may be at least one of a resistive touch screen, a capacitive touch screen, or a thermal touch screen. The display device410may be realized through several known technologies such as, but not limited to, at least one of a Liquid Crystal Display (LCD) display, a Light Emitting Diode (LED) display, a plasma display, or an Organic LED (OLED) display technology, or other display devices. In accordance with an embodiment, the display device410may refer to a display screen of a head mounted device (HMD), a smart-glass device, a see-through display, a projection-based display, an electro-chromic display, or a transparent display. In some embodiment, the video content304of the first media content302or the second media content316may be rendered on the display device410.

The speaker412may include suitable logic, circuitry, and interfaces that may be configured to playback an audio output. The speaker412may be configured to receive electrical audio signals from the processor404or circuitry402and convert the received electrical audio signals into the audio/sound output. In some embodiments, the speaker412may be integrated with the media device106and may be an internal component of the media device106. In some embodiments, the speaker412may be positioned outside the media device106. Examples of the speaker412may include, but are not limited to, a loudspeaker, a woofer, a sub-woofer, a tweeter, a wireless speaker, a monitor speaker, or other speakers or sound output device. In some embodiment, the audio content306of the first media content302or the second media content316may be rendered on the speaker412.

The image capturing device416may include suitable logic, circuitry, and interfaces that may be configured to capture an image/a plurality of images of a viewer/listener (e.g., a user504, as shown inFIG.5) of the media device106. Examples of the image capturing device416may include, but are not limited to, an image sensor, a wide-angle camera, an action camera, a closed-circuit television (CCTV) camera, a camcorder, a digital camera, camera phones, a time-of-flight camera (ToF camera), a night-vision camera, and/or other image capture devices. In certain embodiments, the image capturing device416may be integrated in the media device106. In some embodiments, the image capturing device may lie outside the media device502and/or may be mounted on the media device106.

The network interface418may comprise suitable logic, circuitry, interfaces, and/or that may be configured to facilitate communication between the media device106and the metadata server102, via the first communication network112. Further, the network interface418may be configured to facilitate communication between the media device106and the first set of electrical devices108, via controller device110and/or the second communication network114. In accordance with an embodiment, the network interface418may include two network interface portions, including a first interface portion to facilitate communication over the first communication network112, and a second interface portion to facilitate communication over the second communication network114. In accordance with an embodiment, the first communication network112and the second communication network114may correspond to the same network. The functions of the network interface418may be the same as the functions of the network interface210described, for example, inFIG.2. Therefore, the description of the network interface418is omitted from the disclosure for the sake of brevity. The operations of the circuitry402are described in detail, for example inFIGS.5and7.

FIG.5is a diagram that illustrates an exemplary scenario for control of electrical devices based on context information of media content by the media device, via a controller device, in accordance with an embodiment of the disclosure.FIG.5is described in conjunction with elements fromFIGS.1,2,3, and4. With reference toFIG.5, there is shown an exemplary scenario500that may include a media device502(such as a television (TV)) and a user (e.g., a user504). The user504may be a viewer/listener of the media device502. InFIG.5, there is also shown, a first set of electrical devices (such as, the first set of electrical devices108). The first set of electrical devices108shown inFIG.5, may include, but are not limited to, a smart-sofa506(e.g. smart furniture), a smart light508, a digital photo-frame510, an electrical curtain (such as blinds512), a smart wind-chime514, an electrical toy (such as a cheer-maker toy516, a robotic toy520, a smart celebrity toy524), a smart pod518(or smart speaker), an electrical vase522, or an aroma dispenser526. In some embodiments, each of the first set of electrical devices108may be an internet of things (IOT) device. In addition, inFIG.5, there is further shown the image capturing device416associated with the media device502. In certain embodiments, the image capturing device416may be integrated in the media device502. In some embodiments, the image capturing device416may lie outside the media device502and/or may be mounted on the media device502. In addition, inFIG.5, there is further shown, the controller device110as described, for example inFIG.1andFIG.4.

It may be noted that the media device502and the first set of electrical devices108shown inFIG.5are presented merely as an example of a media device and different electrical devices respectively. The present disclosure may be also applicable to other types of the media device (such as a sound system/sound bar) and other types of the electrical devices. A description of other types of the media device502and the first set of electrical devices108has been omitted from the disclosure for the sake of brevity. The number of electrical devices in the first set of electrical devices108shown inFIG.5is presented merely as an example. The first set of electrical devices108may include only one electrical device or more number of electrical devices, without a deviation from scope of the disclosure.

Further, the media device502may be communicatively coupled with the first set of electrical devices108, via the controller device110and the second communication network114(e.g., a home network, such as, a Wi-Fi network or a wired LAN network). In certain cases, the controller device110may be integrated within the media device502. Alternatively, the controller device110may be implemented separate from the media device502.

In operation, the media device502may generate the first request, which may include the identification information of the first media content302. The media device502may be configured to transmit the first request to the metadata server102, via the first communication network112. In accordance with an embodiment, the metadata server102may generate the plurality of metadata, where each metadata may be generated with respect to a context of a corresponding segment of the plurality of segments of the first media content302. The generation of the plurality of metadata based on the determined context information of the plurality of segments of the first media content302is described, for example, inFIG.3.

The media device502may be further configured to receive the plurality of metadata from the metadata server102along with the first media content302, as the second media content316. Each metadata of the plurality of metadata may include the timing information (e.g., the timing information312) and the context type information (e.g., the context type information314). In some embodiments, the first media content302and the plurality of metadata related with the first media content302may be pre-stored in the memory406as the second media content316. The circuitry402may be configured to receive a user input from the user504to playback the first media content302. In an embodiment, the first media content302may be related to a broadcast program (for example TV program) available on a predefined channel indicated in the user input received from the user504.

The media device502may playback the received first media content302(which may include the plurality of segments, including the first segment302A) and simultaneously control (via the controller device110) the first set of electrical devices108based on the received plurality of metadata. In some embodiments, to playback the first media content302, the media device502may be configured to extract each metadata from the plurality of metadata and the corresponding segment in the received first media content302. The media device502may playback the corresponding segment in synchronization with the control of the first set of electrical devices108based on the timing information312, the context type information314, and the subscription information associated with each of the first set of electrical devices108. To control the first set of electrical devices108, the media device502may transmit one or more trigger commands or control instructions to the first set of electrical devices, via the controller device110, through the second communication network114. The trigger commands or the control instructions may include the timing information312and the context type information314for a particular segment to be played or currently being played by the media device106. In some embodiments, the timing information312and the context type information314may be transmitted only to those electrical devices that may have subscribed to certain context (as indicated by the context type information314.

For example, in case, the circuitry202of the metadata server102determines the context information (i.e. context type information314) of the first segment302A as a romantic scene context, the media device502may playback the first segment302A and simultaneously control (i.e. directly or via the controller device110) the electrical devices (such as, smart light508and blinds512subscripted to the romantic context) to operate based on the corresponding metadata generated for the first segment302A, as discussed, for example inFIG.3. The media device502may similarly control the first set of electrical devices108in synchronization with the playback of the first media content302on the media device502, based on the plurality of metadata associated with the first media content302.

For example, the smart light508, the digital photo-frame510, the blinds512, and the smart wind-chime514are subscribed to operate for a romantic context (i.e. as indicated in the subscription information). In the case of determination of the romantic scene context (i.e. context type information314included in the first metadata310), the media device502during the playback of corresponding segment, may simultaneously control, via the controller device110, the smart light508, the digital photo-frame510, the blinds512, and the smart wind-chime514to operate for the romantic scene context determined from the corresponding segment (such the first segment302A). For example, during the romantic scene context, the subscribed smart light508may produce dim lights of a certain color and the digital photo-frame510may display images which may be liked or preferred by the user504(i.e. the viewer/listener of the media device502) Further, the blinds512may operate in a “closed state”, so as to reduce an ambient light in a surrounding of the user504. In addition, the smart wind-chime514may produce soothing sounds for the user504. Thus, the simultaneous play-back of a media segment and the control of the first set of electrical devices108based on the context information associated with the media segment (for example the first segment302A), may further enhance a user-experience of a viewer/listener (e.g., the user504) of the media device106during the play-back of the media segment. For example, the controlled electrical devices (such as, the smart light508, the digital photo-frame510, the blinds512, and the smart wind-chime514) may express or resonate a romantic mood or emotion and may provide an immersive experience to the user504.

In another example, in case, the first segment302A is determined to be of a sports scene context, the media device502may control (via the controller device110) the cheer-maker toy516to dance or make cheering/clapping sounds, in case the cheer-maker toy516is subscribed to the sports scene context. Similarly, in another example, the media device502may control (via the controller device110) the robotic toy520to mimic one or more characters being portrayed in the first segment302A for a particular context (for example, a comedy scene context), in case the robotic toy520is subscribed to the particular context. In another example, the media device502may control the smart pod518(i.e. subscribed to a particular context) to provide trivia related to the particular context of the first segment302A, which may be currently played-back on the media device502. In another example, in case the first segment302A or other segment is of an action or fight sequence context, the media device502may control (via the controller device110) the smart celebrity toy524to perform one or more stunt actions, in case the smart celebrity toy524is subscribed to the action or fight sequence context as indicated by the subscription information related to the smart celebrity toy524.

In accordance with an embodiment, the circuitry402of the media device502may be configured to compare the subscription information of a particular electrical device and the context type information314included in the first metadata310, and further control (or provide commands) to the electrical device subscripted to the context which may be mentioned in the context type information314included in the first metadata310. For example, in case, the context type information314included in the first metadata310indicate that the context of the first segment302A is action or fight sequence context, and the subscription information of the smart celebrity toy524indicate that the smart celebrity toy524is subscribed to the action or fight sequence context, then the media device106may provide command instruction (either directly or via the controller device110) to the smart celebrity toy524to perform one or more actions or stunt predefined in the smart celebrity toy524. In some embodiments, the provided command instruction may be to turn-on or turn-off the particular electrical device. In other embodiments, the command instruction may include information about an operational mode of the controlled electrical device, in case the electrical device may be operable in multiple operational modes. In some embodiments, the media device502may provide the context type information314(i.e. included in the received first metadata310of a particular segment, such as the first segment302A) to each of the first set of electrical devices108coupled with the media device502. Each of the first set of electrical devices108may be configured to receive the context type information314(i.e. indicate the context of the particular segment) and operate in case the corresponding electrical device is subscribed or capable to the context indicated by the context type information314. In case, a particular electrical device is not capable or subscribed to the context indicated by the context type information314, that electrical device may ignore the context information received from the media device502and may not operate based on the corresponding predefined operations.

In accordance with an embodiment, the circuitry402of the media device502may be configured to receive one or more user inputs from the user504during the playback of the first media content302. The user inputs may correspond to different operations (such as pause, play, stop, move forward, move reverse, seek) with the render of the first media content302. Based on the receipt of the user inputs, the circuitry402may be further configured to control the playback of the first media content302and, accordingly extract the metadata of the corresponding segment (to be rendered) and control the first set of electrical devices108, via the controller device110, based on the extracted metadata so that the first set of electrical devices108may be controlled in synchronization with the playback of the first media content302on a real-time basis. In case, the one or more user input indicates a change in the content channel, the circuitry402may extract the first media content302related to the changed content channel (for example TV channel) and accordingly control the first set of electrical devices108, via the controller device110, based on the plurality of metadata related with the extracted second media content316.

In accordance with an embodiment, during the playback of the first media content302and the control of the first set of electrical devices108, the media device502may control the image capturing device416to capture one or more images of one or more viewers/listeners (such as the user504) who may be present in a field-of-view of the image capturing device416. The control of the image capturing device416may be based on the control of the first set of electrical devices108in synchronization with the playback of the first media content302on the media device502. The media device502may be further configured to determine an emotional state of the one or more viewers/listeners (e.g., the user504) from the captured one or more images based on one or more facial detection techniques. The captured one or more images may be utilized to determine a facial expression of the user504. The facial expression may indicate one or more motions or positions of muscles of a face of the user504, where the facial expressions may manifest an emotion. The muscles of the face may move the skin of the user504, may create facial lines/folds, or may cause the movement of facial features, such as mouth, head, nose, eye, eyebrows of the user504. The media device502may be configured to determine the emotional state of the user504based on the determined facial expression of the user504. The plurality of categories of user emotions may include, but are not limited to, a happy emotion, a sad emotion, an angry emotion, a calm emotion, a fear emotion, a neutral emotion, an excited emotion, a confused emotion, a stressed emotion, a disgusted emotion, a surprised emotion, an excitement emotion, or a scared emotion.

The media device502may be configured to update the one or more metadata from the plurality of metadata based on the determined emotional state of the one or more viewers/listeners (i.e. the user504). In some embodiments, the media device502may be configured to send a metadata update request (which may include the determined emotional state with respect to a particular segment of the first media content302) to the metadata server102. The metadata server102may update the metadata corresponding to the particular segment based on the received request. For example, the media device502may detect that the user504has a negative emotion (e.g., a sad emotion) during the playback of the first segment302A. The media device502may send the metadata update request to the metadata server102to update the first metadata310such that the updated first metadata310may include the updated context type information314associated with the first segment302A that may be synchronous with the emotion expressed by the user504. The updated context type information314may better represent the context of the first segment302A being played back by the media device502and appropriate control the electrical device subscripted to the updated context.

In an accordance with an embodiment, a second set of electrical devices (not shown inFIG.5) may be coupled to the controller device110, in addition to the first set of electrical devices108, in real-time. The second set of electrical devices may correspond to new electrical devices that may be connected to the controller device110, via the second communication network114. Upon connection with the controller device110or directly with the media device502, each of the second set of electrical devices may transmit a request for subscription to one or more context types associated with the media content. The request may be sent to the controller device110or to the media device502. The request for subscription of an electrical device may include the subscription information that may indicate a subscription of the electrical device to a particular context type. Further, when the media device502receives the first metadata310including timing information312and the context type information314, the media device106may transmit the first metadata310to the controller device110. The controller device110may further transmit the first metadata310to the electrical devices of the first set of electrical devices108and the second set of electrical devices based on the context type subscribed by each electrical device (as per the subscription information of the electrical device) and the context type information314. Thus, a new electrical device which may be coupled to the controller device110, may receive the first metadata310if the context type subscribed by the new electrical device coincides with one of the context types in the context type information314in the first metadata310. Accordingly, the controller device110or the media device502may control the new electrical device without any changes in the first metadata310, thereby making the use of the first metadata310scalable with respect to the number and type of electrical devices to be controlled based on the first metadata310.

In accordance with an embodiment, a media service provider (associated with the metadata server102) may provide multiple versions of the second media content316(such as, a movie or game title) to a consumer (e.g., the user504who may use the media device502). Each version of the second media content316may correspond to the same first media content302, however, with a particular metadata version. In an embodiment, the metadata of various versions may be of varying precision and accuracy with respect to the timing information312and context type information314associated with the metadata. Further, the metadata may be bound to particular media content based on a digital rights management identifier (ID) or a similar unique ID for media content. Thus, the consumer may purchase a version of the second media content316that meets his/her need, based on whether he/she uses the required electrical devices that may respond to the corresponding version of the metadata. Further, the media service provider may incrementally provide additional versions of the second media content316with associated metadata with greater precision and accuracy associated with the timing information312and the context type information314. In addition, a first service provider may provide the first media content302(without the plurality of metadata), while a second service provider (e.g., a partner/vendor associated with the first service provider or a third-party) may provide metadata of a particular version (or multiple versions), related to the first media content302. Further, a third service provider may provide one or more electrical devices that may support one or more available versions of the metadata. Thus, the disclosed metadata server102, the media device502, and the first set of electrical devices108may provide an open-ecosystem for development of electrical devices and supported metadata, for continues enhancement of consumer's experience with media content playback.

FIG.6is a flowchart that illustrates a first exemplary method to generate metadata information for control of electrical devices based on context information of media content, in accordance with an embodiment of the disclosure.FIG.6is described in conjunction with elements fromFIGS.1,2,3,4, and5. With reference toFIG.6, there is shown a flowchart600. The operations of the flowchart600may be executed a computing system, such as the metadata server102or the circuitry202. The operations may start at602and proceed to604.

At604, the first segment302A may be received from the plurality of segments of the first media content302. The circuitry202may be configured to receive the first segment302A of the first media content302from the memory206or from another server (e.g., a media server not shown inFIG.1). In accordance with an embodiment, the receipt of the first segment302A may further include extraction of the plurality of frames from the first segment302A of the first media content302.

At606, the context information associated with the first segment302A of the first media content302may be determined based on a characteristic of the at least one frame of the plurality of frames included in the first segment302A. In one or more embodiments, the circuitry202may be configured to determine the context information associated with the first segment302A based on the characteristic of the at least one frame of the plurality of frames included in the first segment302A. In accordance with an embodiment, the circuitry202may determine the characteristic of the at least one frame of the plurality of frames in the first segment302A and may apply the ML model104on the determined characteristic of the at least one frame of the first segment302A to determine the context information associated with the first segment302A. The determination of the context information of the first segment302A is described, for example, inFIG.3.

At608, the first metadata310associated with the first segment302A may be generated based on the determined context information. In one or more embodiments, the circuitry202may be configured to generate the first metadata310associated with the first segment302A based on the determined context information. In accordance with an embodiment, the first metadata310may include the timing information (such as the timing information312) and the context type information (such as the context type information314) to control the first set of electrical devices108. The generation of the first metadata310is described, for example, inFIG.3.

At610, the first segment302A of the first media content302and the generated first metadata310may be transmitted to the media device106(or the media device502) associated with the first set of electrical devices108. The circuitry202may be configured to transmit the first segment302A of the first media content302and the generated first metadata310to the media device106(or the media device502). On receipt of the first segment302A and the first metadata310, the media device106(or the media device502) may playback the first segment302A and simultaneously control, via the controller device110, the first set of electrical devices108based on the generated first metadata310. The control of the first set of electrical devices108is described, for example, inFIGS.3and5. The control may pass to end.

Although the flowchart600is illustrated as discrete operations, such as602,604,606,608, and610, the disclosure is not so limited. Accordingly, in certain embodiments, such discrete operations may be further divided into additional operations, combined into fewer operations, or eliminated, depending on the particular implementation without detracting from the essence of the disclosed embodiments.

FIG.7is a flowchart that illustrates a second exemplary method for control of electrical devices based on context information of media content, in accordance with an embodiment of the disclosure.FIG.7is described in conjunction with elements fromFIGS.1,2,3,4,5and6. With reference toFIG.7, there is shown a flowchart700. The operations of the flowchart700may be executed a computing system, such as the media device106, the media device502, or the circuitry402. The operations may start at702and proceed to704.

At704, the identification information of the first media content302may be transmitted to the metadata server102. In one or more embodiments, the circuitry402of the media device106or the media device502may be configured to transmit the first request to the metadata server102. The first request may include the identification information of the first media content302. The identification information may include a file name of the first media content and/or a memory storage location of the first media content302on the memory206of the metadata server102.

At706, the second media content316may be received from the metadata server102. In one or more embodiments, the circuitry402may be configured to receive the second media content316from the metadata server102. The second media content316may include the first media content302and the plurality of metadata. Each of the plurality of metadata may be associated with a corresponding segment, from the plurality of segments of the first media content302. For instance, the plurality of metadata may include the first metadata310that is associated with the first segment302A of the first media content302. In an alternate scenario, the circuitry402may receive the plurality of metadata from the metadata server102and may receive the first media content302from another server (such as a media server; not shown inFIG.1). The generation of each metadata and the generation of the plurality of metadata are described, for example, inFIG.3.

At708, the first set of electrical devices108may be controlled, via the controller device110, in synchronization with playback of the second media content316based on the plurality of metadata in the second media content316. In one or more embodiments, the circuitry402may be configured to control the first set of electrical devices108, via the controller device110, in synchronization with the playback of the second media content316(or the first media content302) on the I/O device408, based on the plurality of metadata. Each metadata (e.g., the first metadata310) may include the timing information (e.g., the timing information312) and the corresponding context type information (e.g., the context type information314). The control of the first set of electrical devices108may be based on the timing information312and the context type information314in each metadata. The context type information314may indicate the context determined for the corresponding segment of the first media content302. The control of the first set of electrical devices108is described, for example, inFIGS.1,3, and5. The control may pass to end.

Although the flowchart700is illustrated as discrete operations, such as702,704,706, and708, the disclosure is not so limited. Accordingly, in certain embodiments, such discrete operations may be further divided into additional operations, combined into fewer operations, or eliminated, depending on the particular implementation without detracting from the essence of the disclosed embodiments.

Various embodiments of the disclosure may provide a non-transitory, computer-readable medium and/or storage medium, and/or a non-transitory machine readable medium and/or storage medium stored thereon, instructions executable by a machine and/or a computer, such as, a metadata server. At least one code section in the instructions may cause the machine and/or computer to perform operations that include receipt of a first segment from the plurality of segments of first media content. The operations may further include determination of context information associated with the first segment of the first media content based on a characteristic of at least one frame of a plurality of frames included in the first segment. Further, the operations may include generation of first metadata associated with the first segment based on the determined context information. The first metadata may include timing information corresponding to the determined context information to control a first set of electrical devices. In addition, the operations may include transmission of the received first segment of the first media content and the determined first metadata to a media device associated with the first set of electrical devices.

Exemplary aspects of the disclosure may include a metadata server (such as the metadata server102) configured to store a Machine learning (ML) model (such as the ML model104). The metadata server102may include circuitry (such as the circuitry202). The circuitry202may be configured to receive a first segment (such as the first segment302A) from a plurality of segments of a first media content (such as the first media content302). The circuitry202may be further configured to determine context information associated with the first segment302A of the first media content302based on a characteristic of at least one frame of a plurality of frames included in the first segment302A. Further, the circuitry202may be configured to generate first metadata (such as the first metadata310) associated with the first segment302A based on the determined context information. The first metadata310may include timing information (such as the timing information312) corresponding to the determined context information to control a first set of electrical devices (such as the first set of electrical devices108). The circuitry202may be further configured to transmit the first segment302A of the first media content302and the generated first metadata310to a media device (such as the media device106) associated with the first set of electrical devices108.

In certain embodiments, the circuitry202of the metadata server102may be configured to transmit, to a media server, a request which includes identification information of the first media content302. Based on the request, the circuitry202may receive the first media content302including the first segment302A. The circuitry202may further generate the first metadata310associated with the first segment302A based on the received first media content302. The circuitry202may then transmit the received first segment302A of the first media content302and the generated first metadata310to the media device106.

In one or more embodiments, the circuitry202may be further configured to apply one or more machine learning models on the characteristic of at least one frame of the plurality of frames included in the first segment302A to determine the context information associated with the first segment302A.

In an embodiment, the circuitry202may be configured to retrieve the plurality of segments from the first media content302and generate a plurality of metadata. Each of the plurality of metadata may be associated with a corresponding segment from the plurality of segments. The circuitry202may be further configured to generate second media content (such as the second media content316) from the first media content302based on the generated plurality of metadata.

In accordance with an embodiment, the first media content302may correspond to audio content. The context information of the audio content may comprise at least one of, but not limited to, a song, a musical tone, a monologue, a dialogue, a laughter sound, a distress sound, a pleasant sound, an unpleasant sound, an ambient noise, a background sound, a loud sound, or defined sound pattern associated with a real-time object. For audio content, the characteristic of at least one of the plurality of frames included in the first segment302A may include at least one of, but not limited to, a loudness parameter, a pitch parameter, a tone parameter, a rate-of-speech parameter, a voice quality parameter, a phonetic parameter, an intonation parameter, an intensity of overtones, a voice modulation parameter, a pronunciation parameter, a prosody parameter, a timbre parameter, or one or more psychoacoustic parameters.

In accordance with an embodiment, the first media content302may correspond to video content or audio-video (AV) content. The context information of the video content or the AV content comprises at least one of, but not limited to, an action scene, a comedy scene, a romantic scene, a suspense scene, a horror scene, a drama scene, a poetry scene, a party scene, or a dance scene. For video content or audio-video (AV) content, the characteristic of at least one of the plurality of frames included in the first segment302A corresponds to, but is not limited to, at least one of, an object recognized in the at least one frame, a person recognized in the at least one frame, an emotional state of at least one object in the at least one frame, background information of the at least one frame, an ambient lighting condition in the at least one frame, motion information of at least one object in the at least one frame, a gesture associated with at least one object in the at least one frame, or genre information associated with the at least one frame.

In accordance with an embodiment, the circuitry202may be further configured to identify a type of the first segment302A of the first media content302. The type of the first segment302A comprises one of an audio type segment, a video type segment, an audio-video (AV) type segment, or a gaming type segment. The circuitry202may be configured to generate the first metadata310associated with the first segment302A based on the identified type of the first segment302A.

In one or more embodiments, the first set of electrical devices108may include, but is not limited to, an aroma dispenser, an electrical furniture, a lighting device, a sound reproduction device, an electrical curtain, an electrical toy, an electrical wind-chime, an electrical vase, a digital photo-frame, or an internet of things (IOT) device.

In one or more embodiment, the first metadata310may include the timing information312and the determined context information (such as the context type information314) of the first segment302A of the first media content302.

Various embodiments of the disclosure may provide a non-transitory, computer-readable medium and/or storage medium, and/or a non-transitory machine readable medium and/or storage medium stored thereon, instructions executable by a machine and/or a computer, such as, a media device. At least one code section of the instructions may cause the machine and/or computer to perform operations that include transmission of identification information of first media content to a metadata server. The first set of electrical devices may be communicatively coupled to the media device. The operations may further include reception, from the metadata server, second media content, which may include the first media content and a plurality of metadata. Each of the plurality of metadata may be associated with a corresponding segment, from the plurality of segments of the first media content. Each metadata may correspond to context information which may be associated with the corresponding segment and determined based on a characteristic of at least one frame of a plurality of frames included in the corresponding segment. The operations may further include control of the first set of electrical devices in synchronization with playback of the first media content on the media device based on the plurality of metadata.

Exemplary aspects of the disclosure may include a media device (such as the media device106) configured to playback media content. The media device106may include a circuitry (such as the circuitry402). The circuitry402may be configured to transmit identification information of the first media content302to the metadata server102. The circuitry402may be further configured to receive second media content316, which may include the first media content302and a plurality of metadata, from the metadata server102. Each of the plurality of metadata may be associated with a corresponding segment, from the plurality of segments of the first media content302. Each metadata may correspond to context information associated with a corresponding segment and determined based on a characteristic of at least one frame of a plurality of frames included in the corresponding segment. The circuitry402may be further configured to control the first set of electrical devices108in synchronization with playback of the first media content302on the media device106based on the plurality of metadata included in the second media content316.

In accordance with an embodiment, each of the plurality of metadata (e.g., the first metadata310) may include timing information (e.g., the timing information312) and the context information (e.g., the context type information314) associated with the corresponding segment based on which the first set of electrical devices108may be controlled. The circuitry402may be further configured to control the first set of electrical devices108, via a controller device (such as the controller device110) which may be communicatively coupled to the media device106and each of the first set of electrical devices108.

The circuitry402may be configured to receive subscription information associated with each of the first set of electrical devices108. The subscription information may indicate a subscription between the corresponding electrical device with the context information included in the corresponding metadata received from the metadata server102. The circuitry402may be further configured to control the first set of electrical devices108based on each metadata and the subscription information associated with each of the first set of electrical devices108.

In one or more embodiments, the media device106may further comprise an image capturing device (such as the image capturing device416). The circuitry402may be further configured to control the image capturing device416to capture one or more images of one or more viewers/listeners in a field-of-view of the image capturing device416. The control of the image capturing device416may be based on the control of the first set of electrical devices108in synchronization with the playback of the first media content302. The circuitry402may be further configured to determine an emotional state of the one or more viewers/listeners of the media device106(or the media device502), from the captured one or more images and may update the plurality of metadata based on the determined emotional state.