Disclosed herein are system, method, and computer program product embodiments for generating protocol-based composited graphics. An embodiment operates by receiving a first set of commands configured to instruct a first application programming interface (API) to render a first graphics element. The embodiment further operates by receiving a second set of commands configured to instruct a second API to render a second graphics element. The embodiment further operates by generating a serialized set of commands comprising the first set of commands and the second set of commands. Subsequently, the embodiment operates by receiving a request to generate a frame and, in response to receiving the request to generate the frame, generating the frame based on the serialized set of commands, wherein the frame comprises the first graphics element and the second graphics element.

BACKGROUND

Field

This disclosure is generally directed to a protocol-based graphics compositor.

Background

A graphics compositor is a system that allows multiple independent layers of graphics data to be combined into a single on-screen image. Typically, existing compositors create this combined output using hardware graphics planes or composited graphics layers.

In one example for creating a composited output using traditional hardware graphics planes, an application renders a frame for each plane by calling commands in a graphics application programming interface (API). Rendering commands are turned into rendering operations carried out by hardware or software. Some commands render to the screen, and other commands render to off-screen buffers. Rendered pixels for each plane are moved from a back buffer to a front buffer by exchange or copy. Subsequently, all planes are read at the display refresh rate and blended together to output to the display.

In one example for creating a composited output using traditional composited graphics layers, an application renders a frame for each layer by calling commands in a graphics API. Rendering commands are turned into rendering operations carried out by hardware or software. Some commands render to the screen while others render to off-screen buffers. Subsequently, rendered pixels for each layer are moved from a back buffer to a front buffer by exchange or copy. An independent compositor process reads from each layer and renders the blended result to a single frame buffer. When any single layer has been updated, all layers are read and composited again. For instance, rendered pixels for the altered layer again are moved from the back buffer to the front buffer by exchange or copy. Each layer is read and the blended result is written to a single frame buffer. This single plane is read at the display refresh rate to output to the display.

However, existing compositors fall short in many areas. For example, hardware graphics planes are a limited resource allowing only a fixed number of planes with limited position and scaling capabilities. Hardware graphics planes read all pixels for all planes once per display refresh, consuming significant memory bandwidth resources even when the pixel data remains static. Composited graphics layers require two buffers containing pixel data for each layer and an additional two buffers containing the pixel data for the composited result consuming significant graphics memory resources. When any single layer changes, all layers must be read, blended, and the result written to a single frame buffer. This consumes additional graphics processing and memory bandwidth resources. As a result, both hardware graphics planes and composited graphics layers require significant memory, computational, and memory bandwidth resources to generate on-screen images.

Additionally, modern display devices, such as TVs, monitors, tablets, smartphones, and smartwatches and other wearables, etc., are manufactured in a myriad of different screen sizes and resolutions. Accordingly, modern electronic content, such as movies, TV shows, and music, typically is scaled up or down for presentation on these devices. To do so, existing scaling methods render the graphics pixels to memory and then read, filter, and write those pixels to a copy at a different size. In one example, an application renders a frame by calling commands in a graphics API. Rendering commands then are turned into rendering operations carried out by hardware or software. Some commands render to the screen, while other commands render to off-screen buffers. Subsequently, the buffer containing the image to be scaled is read, filtered, and written out at a larger size.

However, existing scaling methods still fall short in many areas. For example, if an application rendered five squares on a screen at 1280×720p resolution but 1920×1080p resolution was needed, existing scaling methods would render the five squares at 1280×720p, read the 1280×720p pixels, scale up the 1280×720p pixels, and then write the 1280×720p pixels back out at 1920×1080p. As a result, existing scaling methods are slow, require significant computational resources, and result in substantial rendering time to generate scaled graphics.

SUMMARY

Provided herein are system, apparatus, article of manufacture, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for generating protocol-based composited graphics. In contrast to existing approaches that render individual layers, the embodiments described herein solve these technological problems by utilizing a protocol-based graphics compositor that intercepts or captures graphics commands generated by applications and that would have been used by one or more application programming interface (APIs) to create a layer and then serializes the intercepted or captured graphics commands into a serialized command stream (e.g., including one or more serialized, commands, transcoded commands, protocols, any other suitable data or control signals, or any combination thereof) that can be stored and replayed in a composited manner by the one or more APIs at a later time (e.g., upon receipt of a request for a frame) to create the pixel data for the layer. In some embodiments, the protocol-based graphics compositor described herein combines a variable number of pixel planes with a variable number of protocol-based planes and recreates the final image by applying both pixel layers and replaying protocol layers until the final image is created.

In one illustrative and non-limiting example for creating a protocol-based composited graphics plane, an application can render a frame by calling commands in a graphics API. The protocol-based graphics compositor described herein can intercept (e.g., without application or API knowledge) or capture (e.g., with application or API knowledge) the commands and send a serialized version of the commands to a protocol-based graphics compositor process for storage. When any layer is altered, the protocol-based graphics compositor can replay the stored commands in order through the original API to render the graphics. The protocol-based graphics compositor can replay off-screen commands only once and replay on-screen commands when any layer is altered to generate a complete frame. Subsequently, the protocol-based graphics compositor can move rendered pixels from a back buffer to a front buffer by exchange or copy. Finally, the protocol-based graphics compositor can read a single plane at the display refresh rate to output to a display device.

Further provided herein are system, apparatus, article of manufacture, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for modifying the rendering (e.g., changing the rendered position, size, orientation, color, compression, etc.) of graphics and graphics elements by transcoding a serialized stream of graphics commands. In contrast to existing approaches, the embodiments described herein solve these technological problems by intercepting or capturing a serialized command stream of graphics commands, and then modifying the rendering commands to modify (e.g., stretch, shrink, rotate, re-color, etc.) the resulting output, causing the output to be substantially equivalent to the output generated by traditional methods but with substantially no additional writes or memory requirements.

In one illustrative and non-limiting example for scaling graphics rendering, if an application issued commands to render five squares on a screen at 1280×720p resolution but 1920×1080p resolution was needed, the embodiments described herein can be configured to intercept or capture those commands, examine them, modify them by multiplying the on-screen data (e.g., destination coordinates) included therein by 1.5, and then send the modified commands to the hardware to generate the 1920×1080p image having the five modified (e.g., scaled-up) squares.

An embodiment is directed to system, apparatus, article of manufacture, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for generating protocol-based composited graphics. In a non-limiting embodiment, the apparatus can be a media streaming device, to name just one example. The apparatus includes a memory and a processor that is communicatively coupled to the memory. In operation, in some embodiments, the processor receives a first set of commands configured to instruct a first API to render a first graphics element. In some embodiments, the processor receives a second set of commands configured to instruct a second API to render a second graphics element. In some embodiments, the processor generates a serialized set of commands comprising the first set of commands and the second set of commands. In some embodiments, the processor receives a request to generate a frame. In some embodiments, in response to receipt of the request to generate the frame, the processor generates the frame based on the serialized set of commands. In some embodiments, the frame includes the first graphics element and the second graphics element.

Another embodiment is directed to system, apparatus, article of manufacture, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for modifying graphics rendering by transcoding a serialized command stream. In a non-limiting embodiment, the apparatus can be a media streaming device, and the serialized command stream can be a serialized stream of graphics commands, to name just two examples. The apparatus includes a memory and a processor that is communicatively coupled to the memory. In operation, in some embodiments, the processor receives a command configured to instruct an API to render a graphics element. For example, in a case where the command was generated by an application and transmitted by the application to the API, the processor can receive the command by intercepting or capturing the command after transmission of the command by the application and before any processing of the command by the API. Responsive to receiving the command, the processor generates, based on the command, a transcoded command configured to instruct the API to render a modified graphics element by applying a set of modification factors to a portion of the command. Subsequently, the processor transmits the transcoded command to the API for rendering.

There are many exemplary aspects to the system, apparatus, article of manufacture, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, disclosed herein. For example, embodiments of the present disclosure provide for a protocol-based graphics compositor that (i) saves rendering time as the pixels are rendered only one time and (ii) saves graphics memory resources because there is no additional copy of the pixel data (e.g., only the pixel data for the final image needs to exist). In another example, embodiments of the present disclosure provide for intercepting or capturing graphics commands intended for a graphics API and transcoding those commands before sending them to the API and rendering them in memory. As a result of these and other embodiments described herein, embodiments of the present disclosure provide for faster graphics rendering techniques that utilize fewer computational resources (e.g., memory, CPU percentage, GPU percentage). As a further result of these and other embodiments described herein, the rendering time to generate and update on-screen images, and to generate and update modified graphics in those on-screen images, is decreased substantially based on the utilization of composited graphics planes and transcoded commands.

DETAILED DESCRIPTION

FIG. 1illustrates a block diagram of multimedia environment102, according to some embodiments. In a non-limiting example, multimedia environment102is directed to streaming media.

Multimedia environment102can include one or more media systems104and one or more content servers122, communicatively coupled via one or more communications networks120. In various embodiments, one or more communications networks120can include, without limitation, wired and/or wireless intranet, extranet, Internet, cellular, Bluetooth and/or any other short range, long range, local, regional, global communications network, as well as any combination thereof.

One or more media systems104can each include display device106, media device108, and remote control110. Display device106can be a monitor, television, computer, smart phone, tablet, and/or projector, to name just a few examples. Media device108can be a streaming media device, DVD device, audio/video playback device, cable box, and/or digital video recording device, to name just a few examples. In some embodiments, media device108can be a part of, integrated with, operatively coupled to, and/or connected to display device106. Media device108can be configured to communicate with one or more communications networks120.

One or more user devices112can interact with one or more media systems104via remote control110. Remote control110can be any component, part, apparatus or method for controlling media device108and/or display device106, such as a remote control, a tablet, laptop computer, smartphone, on-screen controls, integrated control buttons, or any combination thereof, to name just a few examples.

One or more content servers122(also called content sources) can each include databases to store content124and metadata126. Content124can include any combination of music, videos, movies, TV programs, multimedia, images, still pictures, text, graphics, gaming applications, advertisements, software, and/or any other content or data objects in electronic form. In some embodiments, metadata126includes data about content124. For example, metadata126can include associated or ancillary information indicating or related to writer, director, producer, composer, artist, actor, summary, chapters, production, history, year, trailers, alternate versions, related content, applications, and/or any other information pertaining or relating to content124. Metadata126can also or alternatively include links to any such information pertaining or relating to content124. Metadata126can also or alternatively include one or more indexes of content124, such as but not limited to a trick mode index.

FIG. 2illustrates an example block diagram of media device108, according to some embodiments. Media device108can include streaming module202, processing module204, user interface module206, and database208.

Now referring toFIGS. 1 and 2, in some embodiments, one or more user devices112can use remote control110to interact with user interface module206of media device108to select content, such as a movie, TV show, music, book, application, game, etc. Streaming module202of media device108can request the selected content from one or more content servers122over one or more communications networks120. One or more content servers122can transmit the requested content to streaming module202. Media device108can transmit the received content to display device106for presentation to one or more users of one or more user devices112. In streaming embodiments, streaming module202can transmit the content to display device106in real time or near real time as it receives such content from one or more content servers122. In non-streaming embodiments, media device108can buffer or store the content received from one or more content servers122in database208for later playback on display device106.

FIG. 3is a block diagram of example system300for generating protocol-based composited graphics, according to some embodiments. In some embodiments, example system300can include a plurality of applications (e.g., applications302A,302B, through302N), compositor303, a plurality of stored commands (e.g., stored commands305A,305B, through305N), a serialized set of commands generated based on the plurality of stored commands, a plurality of APIs (e.g., APIs306A,306B, through306N), a plurality of off-screen graphics elements (e.g., one or more off-screen graphics elements308A,308B, through308N), composited graphics plane310(e.g., including back buffer312and front buffer314), and display device316.

Each of the plurality of applications can generate one or more commands configured to instruct a respective one of the plurality of APIs to render a graphics element in a frame (e.g., a rendered frame, screen, plane, window, image, overlay (e.g., display screen overlay), or any other suitable element). For example, application302A can attempt to render a first graphics element in a frame by calling one or more graphics commands in API306A, and application302B can attempt to render a second graphics element in the frame by calling one or more graphics commands in API306B.

Compositor303can intercept or capture the commands, serialize the intercepted or captured commands, and store the serialized commands in one or more electronic storage devices. For example, at330A, compositor303can intercept (e.g., without the knowledge of either application302A or API306A) or capture (e.g., with the knowledge of one or both of application302A and API306A) the commands generated by application302A, serialize the intercepted or captured commands, and store the serialized commands in one or more electronic storage devices as stored commands305A. In another example, at330B, compositor303can intercept (e.g., without the knowledge of either application302B or API306B) or capture (e.g., with the knowledge of one or both of application302B and API306B) the commands generated by application302B, serialize the intercepted or captured commands, and store the serialized commands in one or more electronic storage devices as stored commands305B. In some embodiments, compositor303can transmit some or all of the intercepted or captured commands to a transcoder for transcoding (e.g., as described with reference toFIGS. 4 and 6).

Compositor303can serialize the plurality of stored commands to generate a serialized set of commands and store the serialized set of commands in one or more electronic storage devices. Subsequently, in response to receipt of a request to generate the frame, compositor303can transmit a respective subset of the serialized set of commands to each of the plurality of APIs for use in generation of composited graphics plane310. For example, in response to receipt of the request, compositor303can transmit stored commands305A to API306A, transmit stored commands305B to API306B, and transmit stored commands305B to API306B.

In some embodiments, when any layer is altered, example system300can replay the plurality of stored commands in order through their respective API to render composited graphics plane310. For example, to generate composited graphics plane310, example system300can replay off-screen commands only once, whereas example system300can replay on-screen commands anytime a layer is altered.

The plurality of APIs can process the stored commands to generate respective sets of on-screen graphics elements and store the generated sets of on-screen graphics elements in back buffer312of composited graphics plane310. The sets of on-screen graphics elements can include on-screen graphics elements such as on-screen squares, rectangles, cylinders, fonts, images, sequences of images, videos, overlays, any other suitable graphics elements, and any combinations thereof. For example, API306A can process stored commands305A to generate a first set of on-screen graphics elements and store the generated first set of on-screen graphics elements in back buffer312; API306B can process stored commands305B to generate a second set of on-screen graphics elements and store the generated second set of on-screen graphics elements in back buffer312; and API306N can process stored commands305N to generate an N-th set of on-screen graphics elements and store the generated N-th set of on-screen graphics elements in back buffer312.

In addition, the plurality of APIs can process the stored commands to generate respective sets of off-screen graphics elements and store the generated respective sets of off-screen graphics elements in memory. Subsequently, the plurality of APIs can generate additional sets of on-screen elements based on the respective sets of off-screen graphics elements and store the generated additional sets of off-screen graphics elements in back buffer312. For example, API306A can process stored commands305A to generate one or more off-screen graphics elements308A, generate an additional first set of on-screen elements based on one or more off-screen graphics elements308A, and store the generated additional first set of on-screen graphics elements in back buffer312; API306B can process stored commands305B to generate one or more off-screen graphics elements308B, generate an additional second set of on-screen elements based on one or more off-screen graphics elements308B, and store the generated additional second set of on-screen graphics elements in back buffer312; API306N can process stored commands305N to generate one or more off-screen graphics elements308N, generate an additional N-th set of on-screen elements based on one or more off-screen graphics elements308N, and store the generated additional N-th set of on-screen graphics elements in back buffer312. In some embodiments, example system300can move (e.g., by exchange or copy) rendered pixels from back buffer312to front buffer314.

In some embodiments, example system300can transmit composited graphics plane310(e.g., the composited result of the layers) to display device316, which can receive composited graphics plane310and read a single plane (e.g., a single layer of pixels) at the display refresh rate for output to a display implemented by display device316.

In some embodiments, example system300can serialize commands from applications into a stored command stream that can be replayed at a later time to create pixel data for composited graphics plane310. In some embodiments, example system300can combine a variable number of pixel planes with a variable number of protocol-based planes and generate composited graphics plane310by applying both pixel layers and replaying protocol layers until composited graphics plane310is created. As a result, example system300can save rendering time as the pixels are rendered only one time. Additionally, example system300can save video memory resources because there is no additional copy of the pixel data; only the pixel data for composited graphics plane310needs to exist.

FIG. 4is a block diagram of example system400for modifying graphics rendering by transcoding a serialized command stream, according to some embodiments. In some embodiments, example system400can include application402, compositor403, transcoder405, stored commands404(e.g., a serialized set of commands), API406(e.g., a graphics API), one or more off-screen graphics elements408, composited graphics plane410, and display device416.

At420, application402can generate one or more commands configured to instruct API406to render a graphics element in a frame (e.g., a rendered frame, screen, plane, window, image, overlay (e.g., display screen overlay), or any other suitable element). For example, application402can attempt to render a frame by calling one or more graphics commands in API406.

At430, compositor403can intercept (e.g., without the knowledge of either application402or API406) or capture (e.g., with the knowledge of one or both of application402and API406) the commands. At440, compositor403can serialize the intercepted or captured commands and store the serialized commands as stored commands404in the one or more electronic storage devices. Additionally or alternatively, in some embodiments, at444, compositor403can transmit one or more of the intercepted or captured commands to transcoder405. In some embodiments, at442, compositor403can transmit stored commands404to transcoder405.

In some embodiments, as part of replay, transcoder405determines whether to alter one or more of the commands in the serialized version of the intercepted or captured commands. Accordingly, transcoder405can accommodate changed circumstances such as if the reason for the transcoding changes between when the commands were stored and when they are replayed. In some embodiments, transcoder405modifies commands targeting onscreen pixels to generate transcoded commands configured to result in approximately the same pixel output as graphics that have been rendered and stored to memory prior to modification. In some embodiments, transcoder405leaves commands targeting off-screen pixels unchanged. In one illustrative and non-limiting example, transcoder405can be configured to modify an example command to generate an example rectangle in an example frame by generating scaling factors and applying the generated scaling factors to the on-screen destination data included in the command according to the following example pseudocode (which is provided for illustrative purposes and is not limiting):

Received command (e.g., intercepted or captured command):copy_rect(srcX, srcY, srcW, srcH, dstX, dstY, dstW, dstH),where srcX and srcY refer to source X and Y coordinates in the XY-plane,srcW and srcH refer to source width and height,dstX and dstY refer to destination X and Y coordinates in the XY-plane, anddstW and dstH refer to destination width and height.

As shown above, the copy_rect command includes: a source rectangle that is not modified by transcoder405because the source rectangle is off screen; and a destination rectangle that is modified by transcoder405to make the destination rectangle larger than what application402expected it to be (e.g., destination X, destination Y, destination width, and destination height are all scaled up by a subset of scaling factors).

In some embodiments, transcoder405can be configured to generate the transcoded command by accessing a database that includes a set of transcoding rules respectively associated with a set of commands (e.g., graphics rendering commands). For example, the set of rules can include a subset of copy_rect transcoding rules stored in association with a copy_rect command. The subset of copy_rect transcoding rules can include, for example, electronic instructions to: (i) not modify the source data; (ii) modify the destination data; and (iii) if the source data is on-screen source data, modify the source data as well.

In another illustrative and non-limiting example, transcoder405can receive an example command to close application402in composited graphics plane410, referred to in this example as a “delete command.” For example, application402can generate a delete command configured to slide a window associated with application402off the side of composited graphics plane410on exit. Without the knowledge of application402or API406, transcoder405can transcode the received delete command to instruct API406to delete the window in a different manner than the manner called for by application402.

In some embodiments, transcoder405can transcode the received delete command without generating new frames such that there is a one-to-one (1:1) relationship or correlation between input frames and output frames. For example, transcoder405can be configured to modify the received delete command by generating pixel alteration factors configured to fade the window associated with application402in composited graphics plane410on exit rather than sliding it off the side of composited graphics plane410as commanded by application402. In this example, application402can generate a series of commands configured to generate a series of frames each having a window with a modified X location in composited graphics plane410. Transcoder405can intercept the series of commands and modify the intercepted series of commands to fade the window in composited graphics plane410rather than modify the window's X location. In other words, transcoder405can transcode the received commands that were intended to modify the X location of the window into transcoded commands that instead modify the transparency of the window, thereby altering the exit effect without the knowledge of application402or API406. In some embodiments, when application402generates commands calling for twenty frames that move a window off screen on exit, transcoder405instead can instruct API406to generate twenty frames that fade the window off screen on exit.

In other embodiments, transcoder405can transcode the received delete command into multiple animated frames such that there is not a 1:1 relationship or correlation between input frames and output frames (e.g., by replaying all the commands to generate new frames). For example, transcoder405can be configured to modify the received delete command by generating scaling factors configured to shrink, spin, and throw away the window associated with application402in composited graphics plane410rather than sliding it off the side of composited graphics plane410as commanded by application402. In this example, transcoder405can modify the destination X and Y coordinates, the destination width and height, and the destination orientation so that the icon becomes smaller and spins as it is removed from composited graphics plane410rather than being deleted as called for by the received delete command.

In another illustrative and non-limiting example, transcoder405can be configured to modify the colors in graphics data. For example, transcoder405can receive an example command to draw pixels at a certain color in an example frame. Transcoder405can be configured to modify the received example command by generating pixel alteration factors configured to alter the color of the pixels and applying those pixel alteration factors to the pixels to generate a transcoded command configured to instruct API406to render, in composited graphics plane410, an altered-color version of the to-be-drawn pixels called for the received example command. In another example, where the received example command is a command to generate a rectangular image, transcoder405can be configured to modify the received example command by applying pixel alteration factors to the pixels of the to-be-generated rectangular image in the received example command to generate a transcoded command configured to instruct API406to render, in composited graphics plane410, a version of the rectangular image that is slightly more red than the rectangular image called for by the received example command.

In another illustrative and non-limiting example, application402may be sending compressed pixel data to API406, such as an image (e.g., a PNG image), and transcoder405can be configured to modify that image. For example, transcoder405can decompress the image, modify the decompressed pixel data, store the modified pixel data, and then, in response to generate the image in a rendered frame, transmit the stored pixel data to API406using a different command that accepts uncompressed data. As a result, the computing costs (e.g., memory, CPU, GPU) associated with recompressing the image to a new PNG image, sending it to API406, and decompressing it again are substantially eliminated.

In some embodiments, transcoder405can receive (e.g., by interception or capture) multiple commands generated by application402and generate a single command based on those multiple commands or a subset thereof. In some embodiments, transcoder405can receive a single command generated by application402and generate multiple commands based on that single command. In some embodiments, transcoder405can generate a serialized version of the intercepted or captured commands for transcoding (e.g., potential modification of one or more of the intercepted or captured commands).

At450, transcoder405transmits stored commands404to API406. Stored commands404transmitted to API406can include, for example, transcoded commands including one or more modified commands and one or more unmodified commands.

At460, API406can process the transcoded commands to generate a first set of on-screen graphics elements and store the generated first set of on-screen graphics elements in one or more buffers (e.g., which can include a front buffer and a back buffer) for use in generation of composited graphics plane410. The first set of on-screen graphics elements can include one or more unmodified on-screen graphics elements and one or more modified on-screen graphics elements. The one or more modified on-screen graphics elements can include, for example, one or more modified on-screen squares, rectangles, cylinders, fonts, images, sequences of images, videos, overlays, any other suitable graphics elements, and any combinations thereof.

At470, API406can process the transcoded commands to generate one or more off-screen graphics elements408and store one or more off-screen graphics elements408in memory. At480, example system400can generate a second set of on-screen graphics elements based on one or more off-screen graphics elements408and store the generated second set of on-screen graphics elements in one or more buffers for use in generation of composited graphics plane410. One or more off-screen graphics elements408can include one or more unmodified off-screen graphics elements (e.g., configured to remain off-screen) and one or more modified off-screen graphics elements (e.g., configured to be included on-screen in a display implemented by display device416via480and490). The one or more modified off-screen graphics elements can include, for example, one or more modified off-screen squares, rectangles, cylinders, fonts, images, sequences of images, videos, overlays, any other suitable graphics elements, and any combinations thereof.

At480, example system400can utilize the first and second sets of on-screen graphics elements stored in one or more buffers (e.g., including, but not limited to, a back buffer and a front buffer) to generate composited graphics plane410. In some embodiments, at460,470, and480, example system400can generate composited graphics plane410based on one or more transcoded commands, one or more non-transcoded commands (e.g., relating to some off-screen graphics elements), one or more off-screen graphics elements408(including, but not limited to, one or more unmodified off-screen graphics elements, one or more modified off-screen graphics elements, or a combination thereof), one or more on-screen graphics elements (including, but not limited to, one or more unmodified on-screen graphics elements, one or more modified on-screen graphics elements, or a combination thereof), any other suitable command or graphics element, or any combinations thereof. In some embodiments, the one or more modified graphics elements may not “exist” anywhere other than in composited graphics plane410. For example, a resized square may not exist as an element in regular size that then is resized. Rather, the resized square simply may have been drawn as a smaller square by example system400in composited graphics plane410based on a transcoded command received from transcoder405.

In some embodiments, in450, API406can receive one or more on-screen commands from compositor403and, in460, process the one or more on-screen commands to draw one or more on-screen graphics elements directly to composited graphics plane410. The one or more on-screen commands can include one or more modified on-screen commands (e.g., transcoded by transcoder405) and one or more unmodified on-screen commands (e.g., not transcoded by transcoder405). For example, one illustrative example of a modified on-screen command that can be processed by API406in460is where the command includes on-screen source data and on-screen destination data associated with a graphics element, such as when copying the left half of a screen to the right half of the screen.

In some embodiments, in450, API406can receive one or more off-screen commands from compositor403and, in470, process the one or more off-screen commands to draw one or more off-screen graphics elements408. The one or more off-screen commands can include one or more modified off-screen commands (e.g., transcoded by transcoder405) and one or more unmodified off-screen commands (e.g., not transcoded by transcoder405). Subsequently, in480, API406can process one or more additional commands (e.g., off-to-on-screen commands) to draw from one or more off-screen graphics elements408to composited graphics plane410. The one or more additional commands can include one or more modified additional commands (e.g., transcoded by transcoder405) and one or more unmodified additional commands (e.g., not transcoded by transcoder405). For example, the copy_rect( ) pseudo code described above is one illustrative example of a modified additional command that can be processed by API406in480.

At490, example system400can read, based on composited graphics plane410, a single plane at the display refresh rate for output to a display implemented by display device416.

FIG. 5is a flowchart for a method500for generating protocol-based composited graphics, according to an embodiment. Method500can be performed by processing logic that can include hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executing on a processing device), or a combination thereof. It is to be appreciated that not all steps may be needed to perform the disclosure provided herein. Further, some of the steps may be performed simultaneously, or in a different order than shown inFIG. 5, as will be understood by a person of ordinary skill in the art.

Method500shall be described with reference toFIG. 3. However, method500is not limited to that example embodiment.

Optionally, in501, application302A generates a first set of commands (e.g., graphics commands) configured to instruct API306A to render a first graphics element in a frame. The first graphics element can include a square, a rectangle, a cylinder, a font, an image, a sequence of images (e.g., video), an icon, any other suitable graphics element, an overlay including any of these graphics elements, or any combination thereof. In some embodiments, the first set of commands can include: first off-screen data associated with the first graphics element; and first on-screen data associated with the first graphics element. The first off-screen data can include, for example, first off-screen source data (e.g., srcX, srcY, srcW, and srcH) associated with the first graphics element; and first on-screen destination data (e.g., dstX, dstY, dstW, and dstH) associated with the first graphics element. In other embodiments, the first set of commands can include first on-screen source data and first on-screen destination data associated with the first graphics element, such as where the first set of commands are configured to instruct API306A to copy a left-hand portion of a screen (or a horizontally-flipped copy of the left-hand portion of the screen) to a right-hand portion of the screen.

In502, compositor303receives (e.g., by interception or capture) the first set of commands generated by application302A. In some embodiments, the first set of commands may have been generated by application302A and transmitted by application302A to API306A. In some embodiments, compositor303can receive the first set of commands after transmission of the first set of commands by application302A and before any processing of the first set of commands by API306A. In some embodiments, compositor303can generate stored commands305A by storing the first set of commands in one or more electronic storage devices.

In some embodiments, application302A and API306A can have no knowledge of compositor303, such as in cases where application302A and API306A are incapable of being modified (e.g., by example system300) to include knowledge of compositor303. For example, without the knowledge of either application302A or API306A, compositor303can intercept the first set of commands after transmission of the first set of commands by application302A but before any receipt of any of the first set of commands by API306A. In another example, compositor303can prevent receipt of the first set of commands by API306A by, for example, intercepting and storing the first set of commands before they are received by API306A.

In some embodiments, application302A, API306A, or both can be modified (e.g., by example system300) to include knowledge of compositor303. For example, with the knowledge of one or both of application302A and API306A, compositor303can capture the first set of commands after receipt of the first set of commands by API306A but before any processing of the first set of commands by API306A. In another example, compositor303can prevent processing of the first set of commands by API306A by, for example, capturing and storing the first set of commands after they are received by API306A but before they are processed by API306A.

Optionally, in503, application302B generates a second set of commands configured to instruct API306B to render a second graphics element in the frame. The second graphics element can include a square, a rectangle, a cylinder, a font, an image, a sequence of images (e.g., video), an icon, any other suitable graphics element, an overlay including any of these graphics elements, or any combination thereof. In some embodiments, the second set of commands can include: second off-screen data associated with the second graphics element; and second on-screen data associated with the second graphics element. The second off-screen data can include, for example, second off-screen source data associated with the second graphics element; and second on-screen destination data associated with the second graphics element. In other embodiments, the second set of commands can include second on-screen source data and second on-screen destination data associated with the second graphics element, such as where the second set of commands are configured to instruct API306B to copy an upper portion of a screen (or a vertically-flipped copy of the upper portion of the screen) to a lower portion of the screen.

In504, compositor303receives (e.g., by interception or capture) the second set of commands generated by application302B. In some embodiments, the second set of commands may have been generated by application302B and transmitted by application302B to API306B. In some embodiments, compositor303can receive the second set of commands after transmission of the second set of commands by application302B and before any processing of the second set of commands by API306B. In some embodiments, compositor303can generate stored commands305B by storing the second set of commands in one or more electronic storage devices.

In some embodiments, application302B and API306B can have no knowledge of compositor303, such as in cases where application302B and API306B are incapable of being modified (e.g., by example system300) to include knowledge of compositor303. For example, without the knowledge of either application302B or API306B, compositor303can intercept the second set of commands after transmission of the second set of commands by application302B but before any receipt of any of the second set of commands by API306B. In another example, compositor303can prevent receipt of the second set of commands by API306B by, for example, intercepting and storing the second set of commands before they are received by API306B.

In some embodiments, application302B, API306B, or both can be modified (e.g., by example system300) to include knowledge of compositor303. For example, with the knowledge of one or both of application302B and API306B, compositor303can capture the second set of commands after receipt of the second set of commands by API306B but before any processing of the second set of commands by API306B. In another example, compositor303can prevent processing of the second set of commands by API306B by, for example, capturing and storing the second set of commands after they are received by API306B but before they are processed by API306B.

In506, compositor303generates a serialized set of commands comprising the first set of commands and the second set of commands. For example, compositor303can generate the serialized set of commands based on commands included in stored commands305A and305B. In some embodiments, compositor303can receive multiple commands generated by applications302A and302B and generate the serialized set of commands based on those multiple commands or a subset thereof. In some embodiments, compositor303can receive a single command generated by application302A or302B and generate multiple commands for inclusion in the serialized set of commands based on that single command. In506, compositor303generates the serialized set of commands by serializing stored commands305A and305B into a stored command stream that can be replayed at a later time to generate pixel data (e.g., in response to receipt of a request to generate a frame).

In508, compositor303receives a request to generate a frame (e.g., composited graphics plane310). For example, compositor303can receive the request from example system300. In another example, example system300can receive a user input from one of one or more user devices112(e.g., via remote control110in response to a user input such as pressing a button, making a gesture, or speaking a vocal command), generate a request to generate or update composited graphics plane310, or a request to generate or modify an graphics element therein, and transmit the request to compositor303.

In some embodiments, compositor303can prevent processing of the serialized set of commands before receipt of the request to generate the frame. In some embodiments, compositor303can prevent processing of the first set of commands by API306A before receipt of the request to generate the frame. In some embodiments, compositor303can prevent processing of the second set of commands by API306B before receipt of the request to generate the frame.

In510, in response to receipt of the request to generate the frame, compositor303generates the frame based on the serialized set of commands. For example, in response to receipt of the request to generate the frame, compositor303can generate the frame by retrieving the serialized set of commands, transmitting a first subset of the serialized set of commands to API306A, and transmitting a second subset of the serialized set of commands to API306B.

The generated frame includes the first graphics element and the second graphics element. In some embodiments, the first graphics element comprises a first electronic overlay, and the second graphics element comprises a second electronic overlay different from the first electronic overlay. For example, the first electronic overlay comprises an image overlay, and the second electronic overlay comprises a text overlay.

In some embodiments, compositor303can render the pixel data for composited graphics plane310only once without generating an additional copy of the pixel data or a subset thereof. In one illustrative and non-limiting example, compositor303can generate the frame by combining a variable number of pixel planes with a variable number of protocol-based planes based on the serialized set of commands. Subsequently, compositor303can generate the frame by applying both pixel layers and replaying protocol layers until the frame is created.

In some embodiments, where the serialized set of commands comprises a subset of on-screen commands and a subset of off-screen commands, compositor303can generate the frame by processing the subset of on-screen commands and processing the subset of off-screen commands. After generation of the frame, compositor303can receive an electronic indication that a layer in the frame has been altered and, in response to receipt of the electronic indication that the layer in the frame has been altered, process the subset of on-screen commands and not process any of the subset of off-screen commands.

In some embodiments, compositor303can generate, based on the first set of commands and a first set of modification factors, a first subset of transcoded commands configured to instruct API306A to render a first modified graphics element. In some embodiments, compositor303can generate, based on the second set of commands and a second set of modification factors, a second subset of transcoded commands configured to instruct API306B to render a second modified graphics element. In some embodiments, the serialized set of commands can include the first subset of transcoded commands and the second subset of transcoded commands.

In some embodiments, the first set of modification factors, the second set of modification factors, or both can include a subset of scaling factors (e.g., position, size, orientation, and other suitable scaling factors), a subset of pixel alteration factors (e.g., color, hue, luminance, chrominance, and other suitable pixel alteration factors), a subset of decompression factors (e.g., decoding, dequantization, inverse transformation, and other suitable decompression factors), any other suitable modification factors, or any combination thereof. The first modified graphics element, the second modified graphics element, or both can include, for example, one or more modified squares, rectangles, cylinders, fonts, images, sequences of images, overlays, other suitable graphics elements, or any combination thereof.

In some embodiments, the first set of commands can include: first off-screen data (e.g., off-screen source data such as srcX, srcY, srcW, and srcH) associated with the first graphics element; and first on-screen data (e.g., on-screen destination data such as dstX, dstY, dstW, and dstH) associated with the first graphics element. In some embodiments, to generate the first subset of transcoded commands, compositor303can apply the first set of modification factors to the first on-screen data associated with the first graphics element and not apply the first set of modification factors to the first off-screen data associated with the first graphics element. For example, compositor303can apply a subset of scaling factors, such as scaleX and scaleY, to the on-screen destination data, resulting in scaled destination data such as dstX*scaleX, dstY*scaleY, dstW*scaleX, and dstH*scaleY. Continuing this example, compositor303can not apply the first set of modification factors to the first off-screen data associated with the first graphics element, resulting in unmodified off-screen source data such as srcX, srcY, srcW, and srcH. In such embodiments, compositor303leaves commands targeting off-screen pixels unchanged. For example, for a serialized set of commands that includes commands to draw a rectangle to off-screen memory and then copy that rectangle to the screen, compositor303can intercept and modify the command to copy the rectangle to the screen without modifying command to draw the rectangle to off-screen memory.

In some embodiments, the first off-screen data comprises first source data associated with the first graphics element, and the first on-screen data comprises first destination data associated with the first graphics element. In some embodiments, the first graphics element comprises a rectangle, and the first modified graphics element comprises a modified rectangle. In some embodiments, the first graphics element comprises a font, and the first modified graphics element comprises a modified font. In some embodiments, the first set of modification factors comprises a first subset of scaling factors.

Similarly, in some embodiments, the second set of commands can include: second on-screen data associated with the second graphics element; and second off-screen data associated with the second graphics element. In some embodiments, to generate the second subset of transcoded commands, compositor303can apply the second set of modification factors to the second on-screen data associated with the second graphics element and not apply the second set of modification factors to the second off-screen data associated with the second graphics element.

FIG. 6is a flowchart for a method600for modifying graphics rendering by transcoding a serialized command stream, according to an embodiment. Method600can be performed by processing logic that can include hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executing on a processing device), or a combination thereof. It is to be appreciated that not all steps may be needed to perform the disclosure provided herein. Further, some of the steps may be performed simultaneously, or in a different order than shown inFIG. 6, as will be understood by a person of ordinary skill in the art.

Method600shall be described with reference toFIG. 4. However, method600is not limited to that example embodiment.

Optionally, in601, application402generates one or more commands (e.g., graphics commands) configured to instruct API406to render a graphics element in a frame. The graphics element can include, for example, one or more squares, rectangles, cylinders, fonts, images, sequences of images overlays, other suitable graphics elements, or any combination thereof. In some embodiments, the one or more commands can include: off-screen data associated with the graphics element; and on-screen data associated with the graphics element. The off-screen data can include, for example, off-screen source data (e.g., srcX, srcY, srcW, and srcH) associated with the graphics element; and on-screen destination data (e.g., dstX, dstY, dstW, and dstH) associated with the graphics element. In other embodiments, the one or more commands can include on-screen source data and on-screen destination data associated with the graphics element, such as where the one or more commands are configured to instruct API406to copy the left half of a screen (or a horizontally-flipped copy of the left half of the screen) to the right half of the screen. In some embodiments, the graphics element can include a square, a rectangle, a cylinder, a font, an image, a sequence of images (e.g., a video), any other suitable graphics element, an overlay including any of these graphics elements, or any combination thereof.

In602, transcoder405receives (e.g., by interception or capture) the one or more commands generated by application402. In some embodiments, the one or more commands may have been generated by application402and transmitted by application402to API406. In some embodiments, transcoder405can receive (e.g., intercept or capture) the one or more commands after transmission of the one or more commands by application402and before any processing of the one or more commands by API406. In one example, in some embodiments, API406can receive one or more of the one or more commands or one or more portions thereof (e.g., relating to one or more off-screen graphics elements408, or one or more operations that are not to be transcoded). In another example, in some embodiments, API406can never receive the one or more commands or any portion thereof and instead receive only a transcoded versions of the one or more commands.

In some embodiments, application402and API406can have no knowledge of transcoder405, such as in cases where application402and API406are incapable of being modified (e.g., by example system400) to include knowledge of transcoder405. For example, without the knowledge of either application402or API406, transcoder405can intercept the one or more commands after transmission of the one or more commands by application402but before any receipt of the one or more commands by API406. In another example, transcoder405can prevent receipt of the one or more commands by API406by, for example, intercepting and transcoding the one or more commands before they are received by API406.

In some embodiments, application402, API406, or both can be modified (e.g., by example system400) to include knowledge of transcoder405. For example, with the knowledge of one or both of application402and API406, transcoder405can capture the one or more commands after receipt of the one or more commands by API406but before any processing of the one or more commands by API406. In another example, transcoder405can prevent processing of the one or more commands by API406by, for example, capturing and transcoding the one or more commands after they are received by API406but before they are processed by API406.

In some embodiments, transcoder405can receive multiple commands generated by application402and generate a single command for transcoding (e.g., possible modification) based on those multiple commands or a subset thereof. In some embodiments, transcoder405can receive a single command generated by application402and generate multiple commands for transcoding based on that single command. In some embodiments, transcoder405can receive a set of commands configured to instruct API406to render a set of graphics elements in a frame and generate a serialized set of commands configured to instruct API406to render (e.g., via composited graphics plane410) a set of modified graphics elements in a display implemented by display device416based on the received set of commands.

In604, transcoder405generates, based on the received one or more commands (e.g., the one or more intercepted or captured commands generated by application402), one or more transcoded commands configured to instruct API406to render one or more modified graphics elements in composited graphics plane410by applying a set of modification factors to a portion or subset of the one or more commands (e.g., on-screen data associated with the graphics element). In some embodiments, the set of modification factors can include a subset of scaling factors (e.g., position, size, orientation, and other suitable scaling factors), a subset of pixel alteration factors (e.g., color, hue, luminance, chrominance, and other suitable pixel alteration factors), a subset of decompression factors (e.g., decoding, dequantization, inverse transformation, and other suitable decompression factors), any other suitable modification factors, or any combination thereof. The modified graphics element can include, for example, one or more modified squares, rectangles, cylinders, fonts, images, sequences of images overlays, other suitable graphics elements, or any combination thereof.

In some embodiments, where the one or more commands include off-screen data (e.g., off-screen source data such as srcX, srcY, srcW, and srcH) and on-screen data (e.g., on-screen destination data such as dstX, dstY, dstW, and dstH) associated with the graphics element, the applying the set of modification factors to the portion or subset of the one or more commands can include: applying the set of modification factors to the on-screen data associated with the graphics element (e.g., applying a subset of scaling factors, such as scaleX and scaleY, to the on-screen destination data, resulting in scaled destination data such as dstX*scaleX, dstY*scaleY, dstW*scaleX, and dstH*scaleY); and not applying the set of modification factors to the off-screen data associated with the graphics element (e.g., resulting in unmodified off-screen source data such as srcX, srcY, srcW, and srcH). In such embodiments, transcoder405leaves commands targeting off-screen pixels unchanged. For example, for a serialized set of graphics commands that includes commands to draw a rectangle to off-screen memory and then copy that rectangle to the screen, transcoder405can intercept and modify the command to copy the rectangle to the screen without modifying command to draw the rectangle to off-screen memory.

In some embodiments, where the one or more commands include on-screen source data and on-screen destination data associated with the graphics element, such as when copying the left half of a screen to the right half of the screen, applying the set of modification factors to the portion or subset of the one or more commands can include: applying the set of modification factors to the on-screen data associated with the graphics element (e.g., applying a subset of scaling factors, such as scaleX and scaleY, to the on-screen source data and the on-screen destination data, resulting in scaled on-screen data such as srcX*scaleX, srcY*scaleY, srcW*scaleX, srcH*scaleY, dstX*scaleX, dstY*scaleY, dstW*scaleX, and dstH*scaleY).

In some embodiments, transcoder405can generate a serialized set of transcoded commands based on a serialized set of commands, where the serialized set of transcoded commands includes the one or more transcoded commands.

In an embodiment,604can be implemented according to the example pseudocode described with reference to transcoder405shown inFIG. 4. However, in other embodiments,604can be implemented using other code, pseudocode, algorithms, modification factors, or combinations thereof.

In606, transcoder405transmits the one or more transcoded commands to API406. In some embodiments, API406receives the one or more transcoded commands and renders one or more modified graphics elements (and, in some embodiments, one or more unmodified graphics elements) in composited graphics plane410based on the one or more transcoded commands.

Various embodiments and/or components therein can be implemented, for example, using one or more computer systems, such as computer system700shown inFIG. 7. Computer system700can be any computer or computing device capable of performing the functions described herein. For example, computer system700can be used to implement any of the embodiments described herein, as well as combinations and sub-combinations thereof.

Computer system700includes one or more processors (also called central processing units, or CPUs), such as processor704. Processor704is connected to communications infrastructure706(e.g., a bus).

In some embodiments, processor704can be a graphics processing unit (GPU). In some embodiments, a GPU may be a processor that is a specialized electronic circuit designed to process mathematically intensive applications. The GPU can have a parallel structure that is efficient for parallel processing of large blocks of data, such as mathematically intensive data common to computer graphics applications, images, videos, etc.

Computer system700also includes main memory708(e.g., a primary memory or storage device), such as random access memory (RAM). Main memory708can include one or more levels of cache. Main memory708may have stored therein control logic (i.e., computer software) and/or data.

Computer system700can also include one or more secondary storage devices or memories such as secondary memory710. Secondary memory710can include, for example, hard disk drive712, removable storage drive714(e.g., a removable storage device), or both. Removable storage drive714can be a floppy disk drive, a magnetic tape drive, a compact disk drive, an optical storage device, tape backup device, and/or any other storage device/drive.

Removable storage drive714can interact with removable storage unit718. Removable storage unit718includes a computer usable or readable storage device having stored thereon computer software (e.g., control logic) and/or data. Removable storage unit718can be a floppy disk, magnetic tape, compact disk, DVD, optical storage disk, and/any other computer data storage device. Removable storage drive714may read from and/or write to removable storage unit718.

In some embodiments, secondary memory710can include other means, devices, components, instrumentalities or other approaches for allowing computer programs and/or other instructions and/or data to be accessed by computer system700. Such means, devices, components, instrumentalities or other approaches can include, for example, removable storage unit722and interface720. Examples of removable storage unit722and interface720can include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an EPROM or PROM) and associated socket, a memory stick and USB port, a memory card and associated memory card slot, and/or any other removable storage unit and associated interface.

Computer system700can further include a communications interface724(e.g., a network interface). Communications interface724may enable computer system700to communicate and interact with any combination of external or remote devices, external or remote networks, remote entities, etc. (individually and collectively referenced by reference number728). For example, communications interface724can allow computer system700to communicate with external or remote devices728over communications path726, which can be wired, wireless, or a combination thereof, and which can include any combination of LANs, WANs, the Internet, etc. Control logic and/or data can be transmitted to and from computer system700via communications path726.

Any applicable data structures, file formats, and schemas in computer system700may be derived from standards and specifications associated with images, audio, video, streaming (e.g., adaptive bitrate (ABR) streaming, content feeds), high-dynamic-range (HDR) video, text (e.g., closed captioning, subtitles), metadata (e.g., content metadata), data interchange, data serialization, data markup, digital rights management (DRM), encryption, any other suitable function or purpose, or any combination thereof. Alternatively, proprietary data structures, formats or schemas may be used, either exclusively or in combination with another standard or specification.

Standards and specifications associated with images may include, but are not limited to, Base Index Frames (BIF), Bitmap (BMP), Graphical Interchange Format (GIF), Joint Photographic Experts Group (JPEG or JPG), Portable Network Graphics (PNG), any other suitable techniques (e.g., functionally similar representations), any predecessors, successors, and variants thereof, and any combinations thereof.

Standards and specifications associated with audio may include, but are not limited to, Advanced Audio Coding (AAC), AAC High Efficiency (AAC-HE), AAC Low Complexity (AAC-LC), Apple Lossless Audio Codec (ALAC), Audio Data Transport Stream (ADTS), Audio Interchange File Format (AIFF), Digital Theater Systems (DTS), DTS Express (DTSE), Dolby Digital (DD or AC3), Dolby Digital Plus (DD+ or Enhanced AC3 (EAC3)), Dolby AC4, Dolby Atmos, Dolby Multistream (MS12), Free Lossless Audio Codec (FLAC), Linear Pulse Code Modulation (LPCM or PCM), Matroska Audio (MKA), Moving Picture Experts Group (MPEG)-1 Part 3 and MPEG-2 Part 3 (MP3), MPEG-4 Audio (e.g., MP4A or M4A), Ogg, Ogg with Vorbis audio (Ogg Vorbis), Opus, Vorbis, Waveform Audio File Format (WAVE or WAV), Windows Media Audio (WMA), any other suitable techniques, any predecessors, successors, and variants thereof, and any combinations thereof.

Standards and specifications associated with video may include, but are not limited to, Alliance for Open Media (AOMedia) Video 1 (AV1), Audio Video Interleave (AVI), Matroska Video (MKV), MPEG-4 Part 10 Advanced Video Coding (AVC or H.264), MPEG-4 Part 14 (MP4), MPEG-4 Video (e.g., MP4V or M4V), MPEG-H Part 2 High Efficiency Video Coding (HEVC or H.265), QuickTime File Format (QTFF or MOV), VP8, VP9, WebM, Windows Media Video (WMV), any other suitable techniques, any predecessors, successors, and variants thereof, and any combinations thereof.

Standards and specifications associated with streaming may include, but are not limited to, Adaptive Streaming over HTTP, Common Media Application Format (CMAF), Direct Publisher JavaScript Object Notation (JSON), HD Adaptive Streaming, HTTP Dynamic Streaming, HTTP Live Streaming (HLS), HTTP Secure (HTTPS), Hypertext Transfer Protocol (HTTP), Internet Information Services (IIS) Smooth Streaming (SMOOTH), Media RSS (MRSS), MPEG Dynamic Adaptive Streaming over HTTP (MPEG-DASH or DASH), MPEG transport stream (MPEG-TS or TS), Protected Interoperable File Format (PIFF), Scalable HEVC (SHVC), any other suitable techniques, any predecessors, successors, and variants thereof, and any combinations thereof.

Standards and specifications associated with HDR video may include, but are not limited to, Dolby Vision, HDR10 Media Profile (HDR10), HDR10 Plus (HDR10+), Hybrid Log-Gamma (HLG), Perceptual Quantizer (PQ), SL-HDR1, any other suitable techniques, any predecessors, successors, and variants thereof, and any combinations thereof.

Standards and specifications associated with text, metadata, data interchange, data serialization, and data markup may include, but are not limited to, Internet Information Services (IIS) Smooth Streaming Manifest (ISM), IIS Smooth Streaming Text (ISMT), Matroska Subtitles (MKS), SubRip (SRT), Timed Text Markup Language (TTML), Web Video Text Tracks (WebVTT or WVTT), Comma-Separated Values (CSV), Extensible Markup Language (XML), Extensible Hypertext Markup Language (XHTML), XML User Interface Language (XUL), JSON, MessagePack, Wireless Markup Language (WML), Yet Another Markup Language (YAML), any other suitable techniques, any predecessors, successors, and variants thereof, and any combinations thereof.

Standards and specifications associated with DRM and encryption may include, but are not limited to, Advanced Encryption Standard (AES) (e.g., AES-128, AES-192, AES-256), Blowfish (BF), Cipher Block Chaining (CBC), Cipher Feedback (CFB), Counter (CTR), Data Encryption Standard (DES), Triple DES (3DES), Electronic Codebook (ECB), FairPlay, Galois Message Authentication Code (GMAC), Galois/Counter Mode (GCM), High-bandwidth Digital Content Protection (HDCP), Output Feedback (OFB), PlayReady, Propagating CBC (PCBC), Trusted Execution Environment (TEE), Verimatrix, Widevine, any other suitable techniques, any predecessors, successors, and variants thereof, and any combinations thereof, such as AES-CBC encryption (CBCS), AES-CTR encryption (CENC).

In some embodiments, a tangible, non-transitory apparatus or article of manufacture including a tangible, non-transitory computer useable or readable medium having control logic (software) stored thereon may also be referred to herein as a computer program product or program storage device. This includes, but is not limited to, computer system700, main memory708, secondary memory710, and removable storage units718and722, as well as tangible articles of manufacture embodying any combination of the foregoing. Such control logic, when executed by one or more data processing devices (such as computer system700), may cause such data processing devices to operate as described herein.

The breadth and scope of this disclosure should not be limited by any of the above-described example embodiments, but should be defined only in accordance with the following claims and their equivalents.