Method, manufacture, and apparatus for protection of media objects from the web application environment

A method, apparatus, and manufacture for content protection for protecting some objects is provided. A content protection operation or license operation is performed for at least one media object of a web application. Some or all of the objects are then marked as protected. When an operation is performed that accesses a protected object, a user agent determines whether to deny the operation, to allow the operation and mark data exported by the operation and objects storing the exported data as protected objects, or to allow the operation and not mark data exported by the operation and objects storing the exported data as protected objects.

TECHNICAL FIELD

The invention is related to computer software, and in particular, but not exclusively, to a method, manufacture, and apparatus for protecting some media objects from the web application environment.

BACKGROUND

Today, HyperText Markup Language 5 (HTML5) media tags, such as <video> and <audio> tags are available to provide content providers with a simple way to deliver their audio and video content onto the web, and have it play on any web browser or browser-based device. The underlying functionality of the HTML media tags are referred to as HTML media elements. The <video> and <audio> tags enable an ecosystem where plugins are no longer required to play this content and end users have consistent access to it across their web-enabled devices. To date, content protection mechanisms to enable use of content that its owner wants to protect from unauthorized use are not supported in the context of HTML media tags.

DETAILED DESCRIPTION

Throughout the specification and claims, the following terms take at least the meanings explicitly associated herein, unless the context dictates otherwise. The meanings identified below do not necessarily limit the terms, but merely provide illustrative examples for the terms. The meaning of “a,” “an,” and “the” includes plural reference, and the meaning of “in” includes “in” and “on.” The phrase “in one embodiment,” as used herein does not necessarily refer to the same embodiment, although it may. Similarly, the phrase “in some embodiments,” as used herein, when used multiple times, does not necessarily refer to the same embodiments, although it may. As used herein, the term “or” is an inclusive “or” operator, and is equivalent to the term “and/or,” unless the context clearly dictates otherwise. The term “based, in part, on”, “based, at least in part, on”, or “based on” is not exclusive and allows for being based on additional factors not described, unless the context clearly dictates otherwise. The term “coupled” means at least either a direct electrical connection between the items connected, or an indirect connection through one or more passive or active intermediary devices. The term “signal” means at least one current, voltage, charge, temperature, data, or other signal.

Briefly stated, the invention is related to a method, apparatus, and manufacture for content protection for protecting some objects. A content protection operation or license operation is performed for at least one media object of a web application. Some or all of the objects are then marked as protected. When an operation is performed that accesses a protected object, a user agent determines whether to deny the operation, to allow the operation and mark data exported by the operation and objects storing the exported data as protected objects, or to allow the operation and not mark data exported by the operation and objects storing the exported data as protected objects.

Illustrative Operating Environment

FIG. 1shows components of one embodiment of an environment in which the invention may be practiced. Not all the components may be required to practice the invention, and variations in the arrangement and type of the components may be made without departing from the spirit or scope of the invention. As shown, system100ofFIG. 1includes local area networks (“LANs”) wide area networks (“WANs”)-(network)111, wireless network110, client devices101-104, and server device108.

One embodiment of client devices101-104is described in more detail below in conjunction withFIG. 2. Generally, however, client devices102-104may include virtually any portable computing device capable of receiving and sending a message over a network, such as wireless network110, or the like. Client devices102-104may also be described generally as client devices that are configured to be portable. Thus, client devices102-104may include virtually any portable computing device capable of connecting to another computing device and receiving information. Such devices include portable devices such as, cellular telephones, smart phones, display pagers, radio frequency (RF) devices, infrared (IR) devices, Personal Digital Assistants (PDAs), handheld computers, laptop computers, wearable computers, tablet computers, integrated devices combining one or more of the preceding devices, and the like. As such, client devices102-104typically range widely in terms of capabilities and features. For example, a cell phone may have a numeric keypad and a few lines of monochrome LCD display on which only text may be displayed. In another example, a web-enabled mobile device may have a touch sensitive screen, a stylus, and several lines of color LCD display in which both text and graphics may be displayed.

Client device101may include virtually any computing device capable of communicating over a network to send and receive information. The set of such devices may include devices that typically connect using a wired or wireless communications medium such as personal computers, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, or the like. In one embodiment, at least some of client devices102-104may operate over wired and/or wireless network.

A web-enabled client device may include a browser application that is configured to receive and to send web pages, web-based messages, and the like. The browser application may be configured to receive and display graphics, text, multimedia, and the like, employing virtually any web-based language, including a wireless application protocol messages (WAP), and the like. In one embodiment, the browser application is enabled to employ Handheld Device Markup Language (HDML), Wireless Markup Language (WML), WMLScript, JavaScript, Standard Generalized Markup Language (SMGL), HyperText Markup Language (HTML), eXtensible Markup Language (XML), and the like, to display and send a message. In one embodiment, a user of the client device may employ the browser application to perform various activities over a network (online). However, another application may also be used to perform various online activities.

Wireless network110is configured to couple client devices102-104and its components with network111. Wireless network110contains at least one Access Point (not shown inFIG. 1). Wireless network110may include any of a variety of wireless sub-networks that may further overlay stand-alone ad-hoc networks, and the like. Such sub-networks may include mesh networks, Wireless LAN (WLAN) networks, cellular networks, and the like.

Wireless network110may further include an autonomous system of terminals, gateways, routers, and the like connected by wireless radio links, and the like. These connectors may be configured to move freely and randomly and organize themselves arbitrarily, such that the topology of wireless network110may change rapidly.

Wireless network110may further employ a plurality of access technologies including 2nd(2G), 3rd(3G), 4th(4G), 5th(5G) generation radio access for cellular systems, WLAN, Wireless Router (WR) mesh, and the like. Access technologies such as 2G, 3G, 4G, and future access networks may enable wide area coverage for mobile devices, such as client devices102-104with various degrees of mobility. For example, wireless network110may enable a radio connection through a radio network access such as Global System for Mobil communication (GSM), General Packet Radio Services (GPRS), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), and the like.

Additionally, communication media typically embodies computer-readable instructions, data structures, program modules, or other transport mechanism and includes any information delivery media. By way of example, communication media includes wired media such as twisted pair, coaxial cable, fiber optics, wave guides, and other wired media and wireless media such as acoustic, RF, infrared, and other wireless media.

Server device108is a network device. Devices that may operate as server108include various network devices, including, but not limited to personal computers, desktop computers, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, server devices, network appliances, and the like. It should be noted that while various devices are shown in a particular quantity, for example, server108is shown as one device, in various embodiments there may a different quantity of such devices, such two or more servers. There may even be two or more separate networks of servers. Thus, the invention is not to be construed as being limited to a single environment and other configurations, and architectures are also envisaged.

Illustrative Client Device

FIG. 2shows one embodiment of client device200that may be included in a system implementing the invention. Client device200may include many more or less components than those shown inFIG. 2. However, the components shown are sufficient to disclose an illustrative embodiment for practicing the present invention. Client device200may represent, for example, one embodiment of at least one of client devices101-104ofFIG. 1(and, accordingly, may include virtually any type of client device).

As shown in the figure, client device200includes a processing unit (CPU)222in communication with a mass memory230via a bus224. Client device200also includes a power supply226, one or more network interfaces250, an audio interface252, a display254, a keypad256, an illuminator258, an input/output interface260, a haptic interface262, and an optional global positioning systems (GPS) receiver264. Power supply226provides power to client device200. A rechargeable or non-rechargeable battery may be used to provide power. The power may also be provided by an external power source, such as an AC adapter or a powered docking cradle that supplements and/or recharges a battery.

Client device200may optionally communicate with a base station (not shown), or directly with another computing device. Network interface250includes circuitry for coupling client device200to one or more networks, and is constructed for use with one or more communication protocols and technologies including, but not limited to, global system for mobile communication (GSM), code division multiple access (CDMA), time division multiple access (TDMA), user datagram protocol (UDP), transmission control protocol/Internet protocol (TCP/IP), SMS, general packet radio service (GPRS), WAP, ultra wide band (UWB), IEEE 802.16 Worldwide Interoperability for Microwave Access (WiMax), SIP/RTP, or any of a variety of other wired or wireless communication protocols, including WiFi, Ethernet, and/or the like. Network interface250is sometimes known as a transceiver, transceiving device, or network interface card (NIC).

Audio interface252is arranged to produce and receive audio signals such as the sound of a human voice. For example, audio interface252may be coupled to a speaker and microphone (not shown) to enable telecommunication with others and/or generate an audio acknowledgement for some action. Display254may be a liquid crystal display (LCD), gas plasma, light emitting diode (LED), or any other type of display used with a computing device. Display254may also include a touch sensitive screen arranged to receive input from an object such as a stylus or a digit from a human hand.

Keypad256may comprise any input device arranged to receive input from a user. For example, keypad256may include a push button numeric dial, or a keyboard. Keypad256may also include command buttons that are associated with selecting and sending images. Illuminator258may provide a status indication and/or provide light. Illuminator258may remain active for specific periods of time or in response to events. For example, when illuminator258is active, it may backlight the buttons on keypad256and stay on while the client device is powered. Also, illuminator258may backlight these buttons in various patterns when particular actions are performed, such as dialing another client device. Illuminator258may also cause light sources positioned within a transparent or translucent case of the client device to illuminate in response to actions.

Client device200also comprises input/output interface260for communicating with external devices, such as a headset, or other input or output devices not shown inFIG. 2. Input/output interface260can utilize one or more communication technologies, such as USB, infrared, Bluetooth™, or the like. Haptic interface262is arranged to provide tactile feedback to a user of the client device. For example, the haptic interface may be employed to vibrate client device200in a particular way when another user of a computing device is calling.

Mass memory230includes a RAM232, a ROM234, and other storage means. Mass memory230illustrates an example of computer readable storage media (devices) for storage of information such as computer readable instructions, data structures, program modules or other data. Mass memory230stores a basic input/output system (“BIOS”)240for controlling low-level operation of client device200. The mass memory also stores an operating system241for controlling the operation of client device200. It will be appreciated that this component may include a general-purpose operating system such as a version of UNIX, or LINUX™, or a specialized client communication operating system such as Windows Mobile™, or the Symbian® operating system. The operating system may include, or interface with a Java virtual machine module that enables control of hardware components and/or operating system operations via Java application programs.

Memory230further includes one or more data storage248, which can be utilized by client device200to store, among other things, applications242and/or other data. Applications242may include computer executable instructions which, when executed by client device200, perform actions. Other examples of application programs include calendars, search programs, email clients, IM applications, SMS applications, VOIP applications, contact managers, task managers, transcoders, database programs, word processing programs, security applications, spreadsheet programs, games, search programs, and so forth. In some embodiments, applications242may include a browser, a media stack for the browser, a content decryption module (CDM) for decrypting media content for the media stack, and an application running on the browser such as a web application.

Various embodiments of client device200may include applications242to perform various functions discussed herein and/or discussed with regard to some of the steps as illustrated in the flowcharts shown below. These stored applications are processor-executable code encoded on a processor-readable storage medium, which, when executed by CPU222, enable actions to performed in accordance with the processor-executable code. In some embodiments, one or more of the applications242such as the browser may perform steps such as those illustrated in the flowcharts below.

Although the discussion of various embodiments above has described the performance of actions implemented in software, for example by means of processor-executable code encoded on a processor-readable medium, which, when executed by CPU222, enable actions to performed in accordance with the processor-executable code, in various embodiments, actions, may in some embodiments, rather than being performed only by executed software, may instead be performed based on hardware only, or by a combination of hardware and software.

Illustrative Network Device

FIG. 3shows one embodiment of a network device300, according to one embodiment of the invention. Network device300may include many more or less components than those shown. The components shown, however, are sufficient to disclose an illustrative embodiment for practicing the invention. Network device300may represent, for example, server108ofFIG. 1.

Network device300may include processing unit312, video display adapter314, and a mass memory, all in communication with each other via bus322. The mass memory generally includes RAM316, ROM332, and one or more permanent mass storage devices, such as hard disk drive328, tape drive, optical drive, and/or floppy disk drive. The mass memory may store operating system320for controlling the operation of network device300. Any general-purpose operating system may be employed. Basic input/output system (“BIOS”)318may also be provided for controlling the low-level operation of network device300. As illustrated inFIG. 3, network device300also can communicate with the Internet, or some other communications network, via network interface unit310, which may be constructed for use with various communication protocols including the TCP/IP protocol. Network interface unit310is sometimes known as a transceiver, transceiving device, or network interface card (NIC).

Data stores354may include a database, text, spreadsheet, folder, file, or the like. Data stores354may further include program code, data, algorithms, and the like, for use by a processor, such as central processing unit (CPU)312to execute and perform actions. In one embodiment, at least some of data and/or instructions stored in data stores354might also be stored on another device of network device300, including, but not limited to cd-rom/dvd-rom326, hard disk drive328, or other computer-readable storage device resident on network device300or accessible by network device300over, for example, network interface unit310.

The mass memory also stores program code and data. One or more applications350may be loaded into mass memory and run on operating system320. Examples of application programs may include transcoders, schedulers, calendars, database programs, word processing programs, Hypertext Transfer Protocol (HTTP) programs, customizable user interface programs, IPSec applications, encryption programs, authentication programs, security programs, SMS message servers, IM message servers, email servers, account managers, and so forth. In some embodiments, as discussed in greater detail below, application350may include applications for providing a content authorization service, and may provide a key and/or license to an authorized requesting client application.

Generalized Operation

The operation of certain aspects of the invention will now be described with respect toFIG. 4.

FIG. 4illustrates a flow chart of an embodiment of process470. After a start block, the process proceeds to block481, where a content protection or license operation is performed for at least media object of a web application (e.g., an HypterText Markup Language (HTML) media element). The media element is within a document object model (DOM), and the DOM includes objects including the media object(s).

The process then moves to block482, where some or all of the DOM elements are then marked. When an operation is performed that accesses a protected object, the process then advances to decision block493, where a user agent is employed to determine whether or not to allow the operation. If the determination at decision block493is negative, the process proceeds to a return block, where other processing is resumed. However, if the determination at decision block493is positive, the process proceeds to block496, where a determination is made as to whether or not to taint, that is, whether or not to mark data exported by the operation and objects storing the exported data as protected objects. The operation then advances to the return block.

The objects referred to with regard toFIG. 4may include document objects, JavaScript objects, elements, scripts, and/or the like. The content protection operation or license operation may include a license operation, a key operation, and/or the like. The user agent may include a browser and/or the like. The media object may include an HTMLVideoElement, an HTMLAudioElement, or an HTMLMediaElement, or the like, which may be represented by a <video> tag, <audio> tag, or a <media> tag, respectively. Although DOM elements are referred to above, objects other than DOM elements may also be employed, such as media objects for which no actual tag exists in the HTML, a media API such as an Audio or Video API that does not employ the DOM, data, and/or the like.

FIG. 5Aillustrates a flowchart of an embodiment of process580, which may be employed as an embodiment of process470. After a start block, the process proceeds to decision block581, where a determination is made as to whether or not a license operation or other content protection operation has been performed while protection is enabled.

If so, the process proceeds to block582, where at least some of the DOM elements (and/or other objects) are marked as protected. In some embodiments, objects are marked as protected by means of setting a Boolean flag, as discussed in greater detail below.

In some embodiments, an entire document may be marked as protected if it contains protected objects. In some embodiments, particular objects may be protected based on the license. In some embodiments, the objects to be protected are configurable.

In some embodiments, certain accesses are protected. For example, in some embodiments, <video> tags handling encrypted media are protected, and a <canvas> tag accessing the protected <video> is a protected operation.

The process then advances to block583, where protected objects are tracked. In some embodiments, protected elements in the DOM are tracked or the users agent's (e.g., the browser's) interpretation of the protected elements in the DOM are tracked, as well as the backend code that handles and acts on the behalf of the DOM elements.

In some embodiments, the process then proceeds to block584, where code that could access protected data is detected. For example, in some embodiments, extensions, add-ons, plug-ins, controls, developer tools, debuggers, and/or the like, internal or external to the user agent, that could access the protected objects are detected. In some embodiments, such detected code is disabled or certain features are disabled. In some embodiments, the decryption or rendering of protected content by the detected code is prevented. In some embodiments, the protected content is isolated from the detected code.

The process then moves to a return block, where other processing is resumed.

If at decision block581, the determination is negative, the processing advances to the return block.

FIG. 5Bshows a flowchart of an embodiment of process590, which may be employed in conjunction with an embodiment of process580ofFIG. 5A.

After a start block, the process proceeds to decision block591, where a determination is made whether or not to enable protection. In some embodiments, protection is always enabled. In some embodiments, protection is enabled only when requested by the web page, license, data/stream, and/or the like. In other embodiments, protection is enabled only when potentially adversarial behavior, scripts, or elements are detected to be present. In various embodiments, other factors that may be factors as to whether or not to enable protection may include, but are not limited to, the presence or absence of extensions, and the origin of the elements in the DOM, including whether or not the elements originate from more than one domain, as discussed in greater detail below.

If protection is enabled, the process then moves to decision block592, where a determination is made as to whether or not an operation is requested that accesses protected media. If so, the process advances to decision block593, where a determination is made as whether or not to allow the operation. If the determination is negative, the process proceeds to block594, where the operation is denied, and then processing moves to a return block, where other processing is resumed.

If, however, the determination at decision block593is positive, the operation proceeds to block595, where the operation is allowed, and then the processing moves to decision block596, where a determination is made as to whether or not to taint the operation. If the determination at decision block596is to taint the operation, the process advances to block597, where data exported by the operation and objects storing the exported data are marked as protected objects. The process then proceeds to the return block. (At decision blocks591,592, or596, if the determination is negative, the process proceeds to the return block.) For example, in some embodiments, all <canvas> reads to protected <video> are allowed but tainted, so that <canvas> can read <video> but the <canvas> cannot be exported itself.

The decision as to whether to deny, allow (i.e., fully allow, without tainting), or (allow but) taint may be made in accordance with the license, and may be made in accordance with a variety of different factors in some embodiments. In some embodiments, the determination may be made based on configurable rules in accordance with the license.

Some embodiments may allow normally prohibited access as controlled by the license, web page/application, and/or the like (sample uses include allowing <canvas>, Web Graphics Library (WebGL), and/or the like access to enable mash-ups).

In some embodiments, accesses may be limited to the same origin and/or specific accesses may be allowed using cross-origin resource sharing (CORS). In some embodiments, this may be limited to authenticated servers using technologies such as Hypertext Transfer Protocol Secure (HTTPS)/Secure Sockets Layer (SSL). With regard to cross-domain resources, in some embodiments, certain objects could get tainted by mismatched origin resources. In some embodiments, for protected video content, pixels from the protected video could be read back and cascading style sheets (CSS) applied or the data exported to an HTMLCanvasElement. In some embodiments, such operations may not be allowed if the CSS originates from another origin or the HTMLCanvasElement is created by a script in another origin.

In some embodiments, plug-ins may also be marked as protected and receive the same protections as the other protected objects.

Some embodiments of processes580and590enable valuable and protected content to be provided to the DOM, for example via the <video> and <audio> tags, without providing unlimited access to these elements. For example, some embodiments of process580and590may be used to prevent the <canvas> tag from being used to scrape frames from <video>, and/or prevent audio APIs from being used to record from <audio>.

Frames from <video> can be read by other DOM elements, such as <canvas>. This could otherwise make it easy for a web application or browser extension to copy frames and reconstruct the video later. Content owners of valuable content can use embodiments of process580and590to restrict access to protect against such an attack, and/or similar attack vectors from audio APIs for audio content.

In some embodiments, the objects to be protected, and the rules for which accesses to allow, taint, or deny, may be CDM-specific. Attack vectors may be identified and closed based on such rules. The CDM is discussed in greater detail below. In some embodiments, simple decryptions are not protected.

It is not necessarily desirable to prevent some innovative uses of protected content, and so access is not necessarily always prevented in some embodiments. The CDM and license rules may provide ways for content providers to allow it or explicitly disallow such accesses. In some embodiments, the decryption/digital rights management (DRM) module may be employed to mark a media object as protected and/or specify how it can be used/accessed.

In some embodiments, objects can be marked as protected objects by setting a Boolean flag. The user agent (e.g., browser) is configured such that, when the user agent is about to perform an operation that accesses an object, the user agent first checks the Boolean flag to see if the object is protected. If the object is protected, the user agent makes a determination as whether to allow or deny the operation, and if allowed, whether to taint, as discussed above with regard to process590.

As discussed above, in some embodiments, the protection is enabled only under certain conditions. For example, in some embodiments, no protection is applied unless there is a <canvas> tag on the page, but if there is a <canvas> tag on the page, full protection is applied. As previously discussed, every object could be checked individually in some embodiments, whereas other embodiments may apply protection globally, such as the <canvas> tag isn't allowed at all, or the <canvas> tag is always allowed but no operation on the data exported by the operation of a <canvas> tag or on objects storing data exported by the operation are allowed.

In some embodiments, user settings, infobars, and/or the like may be employed that allow the user to disable certain features (e.g., <canvas> or <canvas> ability to read a protected <video>) to enable protected content.

FIG. 6shows system600, which may be employed as an embodiment of system100ofFIG. 1. System600includes license server608, web server609, content delivery network (CDN)607, and client device601. License server608, web server609, and CDN607may each, for example, be one or more server devices such as server device108ofFIG. 1and/or network device300ofFIG. 3. Client device601may be, for example, an embodiment of client device101-104ofFIG. 1and/or client device200ofFIG. 2.

Media stack646performs decoding of media content. In some embodiments, media stack646is a set of connected media processing and rendering modules used for converting digital media data into human-perceptible representations (e.g., moving pictures and/or sound). Media stack646performs functions such as de-multiplexing, decoding, and rendering in some embodiments (in conjunction with CDM649in some embodiments). In some embodiments, media stack646has substantially no other logic, but just renders what it is fed, and can therefore be used for all types of media playback applications in an interchangeable fashion.

Application644is a media playback control application. Application644instructs media stack646to playback media content. In some embodiments, application644includes computer-executable code that implements substantially all of the logic required to play back media in a particular context, including user interface logic, subtitle control information, control of the acquisition of keys, and/or the like, but not processing and rendering. The processing and rendering is instead performed by media stack646. In some embodiments, the rendering is performed by a part of browser645other than media stack646. In other embodiments, the rendering is performed by another part (hardware or software) of the client. Together, media stack646and application644both reside in client device601and operate as a client media player. Media stack646and application644are distinct programs or binaries or libraries or scripts, etc., where media stack646can be re-used. Media stack646is part of an application framework, together with other modules10that implement user input, graphics output, network I/O etc. The application framework exposes standard abstract interfaces for these functions to application644. In some embodiments, application644may be a web application for playback of media using the HTML media elements identified in HTML5 by HTML media tags such as <media>, <video>, and <audio>, where the web application includes a scripting language (e.g., JavaScript) and HTML or a mark-up language derived from HTML (e.g., mark-up languages based on supersets/subsets/slices/extensions of HTML).

This particular separation between media stack646(as part of the application framework) and application644is useful because the media stack implementation is typically specific to the type of computer hardware platform it executes on, and therefore is implemented in a way that is strongly connected to the operating system. The same holds for user input, graphics output, network I/O, etc. In some embodiments, media stack646is specific to a particular type of content protection. Application644, however, can be the same across many different computer platforms when the underlying application framework abstracts access to user input, network, graphics, media engine etc.

In some embodiments, the media content is protected by encryption, and the license and/or key is employed to decrypt the encrypted media content. In some embodiments, the media content is protected by a license.

The decryption function in media stack646is initialized by the key/license when the key/license is provided to the media engine. A content encryption key may be generated employing any of a number of encryption/decryption symmetric mechanisms, including, but not limited to Advanced Encryption Standard (AES), RSA, RC6, IDEA, DES, RC2, RC5, Skipjack, and any other symmetric encryption algorithm. Moreover, such encryption algorithms may use, where appropriate, cipher block chaining mode, cipher feedback mode, CBC cipher text stealing (CTS), CFB, OFB, counter mode, and/or any other block mode. In some embodiments, content encryption keys are generated employing an at least 128-bit AES encryption/decryption algorithm. However, content encryption key generation is not limited to symmetric key mechanisms, and asymmetric key mechanisms may also be employed without departing from the scope of the present invention. Typically, where the content encryption key is a symmetric encryption key, such content encryption keys are also referred to as decryption keys, as the same key may be used to both encrypt and to decrypt the content.

Since the key/license acquisition occurs not in media stack646but in application644, application644has control over the error handling for the key/license acquisition, and application644can handle error conditions and the like.

CDM649performs decryption of media content in communication with media stack646. In some embodiments, CDM649may use or defer to the capabilities of platform643of client device601. In some embodiments, CDM649may be integrated with the media stack646. In other embodiments, CDM649may be integrated with the platform643.

In some embodiments, when application644acquires a license, it does so in communication with one or more servers. In some embodiments, web server608may handle requests or send directly to license server609. In some embodiments, application644may communicate directly with license server609.

In some embodiments, media stack646may acquire media content specified by application644via network stack647, where network stack647acquires the media content from CDN607.

FIG. 7illustrates a flowchart of an embodiment of process780, which may be employed as an embodiment of process580ofFIG. 5A. After a start block, the process proceeds to block771, where the client browser loads the web application. The process then moves to block772, where the client browser discovers an HTML media tag (e.g., <video> or <audio>) pointing to media content.

The process then advances to block773, where the web application instructs the media stack of the client browser to initiate playback of the media content. In some embodiments, the web application may contain an autoplay attribute, in which case the browser parses the web application, discovers the autoplay attribute, and causes the media stack to autoplay the media content. In other embodiments, there is no autoplay attribute for the media content, and the web application instructs the media stack when to play media content. The media stack is instructed to play the media content in a time and manner that is in accordance with the web application. For example, in some embodiments, the web application instructs the media stack to play the media content when a “play” button is pressed by the user.

The process then proceeds to block774, where the media stack starts to procure the media content that the media stack was instructed to play back (e.g., through file reading, or progressive or adaptive streaming).

The process then advances to decision block775, where the media stack makes a determination as to whether or not the media content is encrypted. For example, in some embodiments, the determination at decision block775may be made through out-of-band information, e.g., in the streaming manifest. In other embodiments, the determination at decision block775may be made through information in the content data itself, e.g., flags or headers. The actual mechanism of detection may be dependent on the media stack or content type.

If the content is encrypted, the process proceeds to block776, where the media stack fires an event to the web application, where the event indicates that a key and/or license is needed to begin or continue playback of the media content. The process then moves to block781, where the web application gets the key and/or license. As discussed in greater detail above and below, the web application may get the key and/or license in a variety of different ways in different embodiments. For example, in some embodiments, the application acquires the key and/or the license via communication with one or more servers. In other embodiments, the key and/or license could be stored in the application.

The process then proceeds to block782, where at least some of the DOM elements (and/or other objects) are marked as protected. The process then advances to block783, where protected objects are tracked. In some embodiments, the process then proceeds to block784, where code that could access protected data is detected. For example, in some embodiments, extensions, add-ons, plug-ins, controls, developer tools, debuggers, and/or the like, internal or external to the user agent, that could access the protected objects are detected. In some embodiments, such detected code is disabled or certain features are disabled. In some embodiments, the decryption or rendering of protected content by the detected code is prevented. In some embodiments, the protected content is isolated from the detected code.

The process then advances to block785, where the web application sends data (including the key and/or license) to the media stack. The process then proceeds to block786, where the media stack receives the data. The process then moves to a return block, where other processing is resumed.

In some embodiments, there is a separate decrypting module (e.g., CDM649ofFIG. 6) that is employed to do the decryption. In some of these embodiments, the media stack may provide the encrypted media and the key and/or license to the decryption module, and the decryption module may decrypt the media and send the decrypted media content back to the media stack or to hardware. What happens to the decrypted data varies in various embodiments. In some embodiments, the decrypted data is provided to the media stack for rendering. In some embodiments, the decrypted data is provided to a secure pipeline.

Many variations to the exact process flow illustrated inFIG. 7are within the scope and spirit of the invention. For example, in some embodiments, the web application may determine that the media is encrypted. Accordingly, rather than the web application instructing the media stack to initiate playback right away, in this case the web application can instead get the key and/or license, and after the web application gets the key and/or license, the web application can instruct the media stack to initiate playback along with sending the media stack the data that includes the key and/or license. In some embodiments, the media tags may specify that the media is encrypted.

In some embodiments, such as a key rotation scheme, the media stack can fire an event each time a new key is needed, and the application can provide the new key to the media stack.

Process780may be applied to local content, to content that is downloaded and then played, and/or to streaming content. Process780is not specific to any particular form of content protection, but may be applied to various forms of DRM and other forms of content protection in different embodiments. Process780provides a framework that is independent of and orthogonal to the particular form of content protection being used and other components of the video playback such as the content fetching mechanism, the content format, and its decoding and rendering mechanism, and supports a wide range of media containers and codecs. Process780can support a wide range of control system types, including traditional DRM systems, conditional access systems, simple key establishment without the need for DRM servers, and key distribution systems based on DRM systems using only a small subset of their typical functionality for streaming media.

In some embodiments, process780may be used for viewing commercial content, including commercial content protected by a content protection mechanism such as DRM, where the commercial content may include movies or the like. The media may be accessed via a webpage employing HTML5, or a future version of HTML that also supports media tags, where the media content is accessed in an HTML media tag such as <video> or <audio>. The media content, which is protected by some sort of content protection, such as encryption, DRM, and/or the like, can be accessed in a browser with an HTML media tag, with the media accessible, controllable, and configurable by the web application, where the media content can be streamed to the browser, downloaded, provided by the application, or local. The web application provides application level flexibility to implement the authorization exchange, and all of the HTML5 security mechanisms may be used for network transactions.

In some embodiments, one attribute (authorizationData) and one event (authorizationNeeded) is added to the HTML5 media elements, as follows. In some of these embodiments, the design adds one new attribute to media elements named authorizationData. The purpose of this attribute is to provide the content control module in the media stack with the necessary information to initialize itself and enable playback of the requested content (e.g., a decryption key). In some embodiments, authorizationData may be a method that provides the data to the media element. The attribute authorizationData and the event authorizationNeeded may have different names in different embodiments. For example, inFIG. 6, they are referred to as addKey( ) [as a called function rather than an attribute] and needkey, respectively.

The authorizationNeeded event fires when the media stack underneath the media element discovers the need for fresh authorization data to start or continue playback. This event is accompanied by one or more blobs of data generated by the media stack that is potentially opaque to the application. The format of the data may be dependent on the content control mechanism(s) in the media stack and/or content format.

The authorizationNeeded event is an embodiment of the event fired at block776of process780. In some embodiments, the authorizationNeeded event also contains supplemental data that is specific to the content control mechanism and/or content. The supplemental data can be opaque to the application—it only needs to be understood by the entity that creates the license data, which could be a network service in some embodiments.

The application uses the information mentioned above to procure the required content control data (either through an application-specific mechanism or through a content-control-specific mechanism). This could happen through in-client processing, querying (e.g. a password), retrieval from storage, or a network request to a service. The authorizationNeeded event can fire multiple times during a single playback session. e.g., for key rotation.

The acquired content control data is provided to the media element via the authorizationData attribute. Once the data is present and correctly verified by the media stack, playout starts. In order to avoid playout delays due to latencies incurred when acquiring content control data, existing preload mechanisms can be used.

In some embodiments, the authorizationNeeded event contains an array of the following tuple: type and data, where type is an identifier of content control mechanism, and data is a byte array containing contents dependent on content control mechanism to enable application to acquire authorization data. In some embodiments, the authorizationData attribute includes a byte array in which the contents are dependent on content control mechanism, which enables the media stack to play content, and/or content data format.

In one possible embodiment of an application of the authorization data acquisition mechanism, content is encrypted using a content key, and a secure key distribution system stores the key. In some embodiments, the media stack contains client code of this secure key distribution system (though a pluggable mechanism or platform capabilities in some embodiments), and the content provider runs a key distribution service as a counterpart.

In some of these embodiments, the content control system uses: a “challenge” data structure generated on the client and a “license” data structure generated on the server using the content key and the client public key. The “challenge” data structure typically contains information about the content to be played for retrieval of the correct key, and information about the client DRM module (client ID, client type, and public key for encryption of the content key for transport).

In these embodiments, the media stack detects that the content is encrypted, and can be decrypted using the secure key distribution system's client module. The content key is acquired through a network transaction. Since the application has not yet set the authorizationData attribute of the media element to a valid “license” data structure, the media stack generates a “challenge” data structure and serializes it into a byte stream.

In these embodiments, the authorizationNeeded event gets the following two attributes: (1) type=(<string or numeric identifier for the secure key distribution system>); and (2) data=<challenge bytestream>, and fires.

In these embodiments, the application catches the event using a handler for this event. In the handler, the web application generates a request to the streaming service's license server, using an existing authentication mechanism (e.g., domain cookie) to authorize the request on the server, and the opaque data generated by the media stack's content protection client module. In some embodiments, the content control mechanism may choose to invalidate the authorization data after a certain number of uses, therefore disallowing replay of previously issued authorization data, and in consequence enforcing a requirement of fresh authorization for each playback.

In these embodiments, authorization depends on the streaming service's business rules. At the minimum, licenses could be handed out to everyone. In a service like this, the content still cannot be copied, and giving out rights to play back can be stopped at any time (e.g. when the service's distribution rights for this content end). Other services may require customer authentication, and even others may impose restrictions on number of devices and concurrent usage.

In these embodiments, if authorization is successful, the license server may use the content identification data in the “challenge” data structure to retrieve the content key for the content to be played from its database, and use the client key to generate a “license” data structure, which is handed back to the application in the response. The application installs the opaque license data into the authorizationData attribute of the media element, and playout starts after successful verification of license data and extraction or derivation of the content key from the license data by the media stack. The content control client module is responsible for: the security of the decrypted content, the security of the content key, the security of the client module keys that can be used to decrypt license data, and secure disposal of the content key after playback has ended. In some embodiments, the opaque license data contains all of the necessary information to play all streams of the content in question that are authorized to the requesting user on the target device. For example, if different streams of some particular content are encrypted with different keys, the license contains all of the keys and the necessary information to associate it with the correct streams.

In some of these embodiments, the system can be extended to allow offline playback without changing the media stack implementation of the content control system, e.g. by retrieving the key and/or license from storage.

In these embodiments, a simple offline playback system allows the user to “pin” content to the client device while the device is connected. The content data (one embodiment thereof, e.g., at appropriate bitrate and quality for the target device) is downloaded from the server in its entirety and stored in local storage. The application uses a media element to initiate preload of playback for the content data in local storage, which triggers the authorization data retrieval mechanism. The authorization data is retrieved by the application as in the streaming use case, but not installed in the authorizationData attribute yet. It is cached in local storage until offline playback is initiated.

In these embodiments, if required for management of the stored authorization data, the content protection system can support the transmission of additional information on top of the opaque authorization data blob to the web application, e.g. content ID, expiration date etc., e.g. through the network protocol that also serves to transmit the authorizationData blob. This mechanism can be proprietary to the content protection system in some embodiments.

In these embodiments, when playback is initiated, the application retrieves the authorization data from its internal storage and installs it in the media element, allowing playout to proceed.

Returning toFIG. 6, as discussed above, the event authorizationNeeded and the attribute authorizationData are instead replaced by the event needkey and the function addKey( ), respectively. Application644may use Get Key to acquire a key from a server, which may in turn respond with an encrypted key via Key from the server to application644. The key may be provided from application644to media stack646via addKey( ), and from media stack646to CDM649via SetKey.

In some embodiments, canPlayType( ) is extended to provide basic information about content control systems supported by the browser.

Media stack646may provide an encrypted frame for CDM649to decrypt via Decrypt Frame. CDM649then decrypts the encrypted frame, and provides the decrypted frame back to media stack646via Frame.

In some embodiments, a generateKeyRequest( ) function may be employed from Application644to media stack646. A Key Request may be sent from media stack646to application644in response to the generateKeyRequest( ) function.