Buffer objects for web-based configurable pipeline media processing

An apparatus and method relating to buffer objects for web-based media processing are disclosed. The disclosed embodiments include a web browser implemented on a computing device. The web browser includes a web application processor for processing a web application that includes instructions to process a media stream using one or more configurable pipelines, each configurable pipeline including a plurality of components connected by data channels using buffer objects, the buffer objects including a data pointer identifying a location in a memory, the data pointer having an associated data size representing an amount of memory available at the memory location, an allocation method configured to allocate memory to the buffer object using the data pointer and the data size, and an accessor method configured to enable a component in the configurable pipeline to access data stored in the memory identified by the data pointer and the data size.

TECHNICAL FIELD

The present invention relates in general to web-based processing of media and in particular to buffer objects for web-based configurable pipeline media processing.

BACKGROUND

In the early stages of the World Wide Web (WWW), web pages generally consisted of static content (such as text and images) stored on servers. The static content was accessed and rendered by a web browser executing on a client. As the WWW has evolved, much content on the web is now dynamically generated. Such content can include web applications which that include instructions to be performed by the client web browsers. Such web applications can provide a more interactive and functional experience than earlier web pages. More recent web applications now include various forms of media, including audio and video content.

SUMMARY

Disclosed herein are embodiments of methods and apparatuses for buffer objects for web-based configurable pipeline media processing.

One aspect of the disclosed embodiments is a web browser implemented on a computing device. The web browser includes a web application processor for processing a web application that includes instructions to process a media stream using one or more configurable pipelines, each configurable pipeline including a plurality of components connected by data channels using buffer objects. At least some of the buffer objects include a data pointer identifying a location in a memory, the data pointer having an associated data size representing an amount of memory available at the memory location, an allocation method configured to allocate memory to the buffer object using the data pointer and the data size, and an accessor method configured to enable a component in the configurable pipeline to access data stored in the memory identified by the data pointer and the data size.

Another aspect of the disclosed embodiments is a buffer object configured for use in a configurable pipeline to pass data within data channels between a plurality of components in the configurable pipeline, the configurable pipeline created within a web browser and executed on a computing device. The buffer object includes a data type property, a data pointer identifying a location in a memory, the data pointer having an associated data size representing an amount of memory available at the memory location, an allocation method configured to allocate memory to the buffer object using the data pointer and the data size, and an accessor method configured to enable a component in the configurable pipeline to access data stored in the memory identified by the data pointer and the data size.

Another aspect of the disclosed embodiments is a method of transferring data with buffer objects between components in a configurable pipeline implemented using a web browser. The method includes initializing the configurable pipeline based on instructions included within a web application executed by the web browser, the initialization including creating data channels between components in the configurable pipeline, creating a plurality of buffer objects to enable the data channels to transfer data between the components of the configurable pipeline, storing data in a buffer object of the plurality of buffer objects by an output port of a first component, passing the buffer object from the first component to an input port of a second component by way of at least one of the data channels, and retrieving the data stored in the buffer object by the second component.

These and other embodiments will be described in additional detail hereafter.

DETAILED DESCRIPTION

Various solutions for including media in web applications include monolithic plugins, such as Adobe Flash and monolithic built-in browser functionality, such as the video tag included in HTML5. These solutions provide a web application high level control over a pre-defined process, such as video encoding or decoding, but do not provide customized or granular control over the process. Custom processes can require, for example, a distinct and custom programmed plugin module for the entire process. Thus, adding proprietary or other custom aspects to a process (such as Digital Rights Management (DRM), advertising overlays, video enhancements, etcetera) can be cumbersome, time-consuming, and costly. To the contrary, a modular configurable pipeline implemented in a web browser can enable a web application to configure custom processes and have granular control over those processes.

In particular, a modular configurable pipeline can be implemented using components and buffer objects. Components can be arranged within a configurable pipeline to perform processing tasks. Data channels can be established between the components to pass data through the configurable pipeline. Data can be passed through the data channels using buffer objects. Buffer objects can be utilized, for example, in a circular buffer technique or a pass through buffer technique.

FIG. 1is a diagram of a client-server system10. Server12can be, for example, a computer having an internal configuration of hardware including a processor such as a central processing unit (CPU)14and a memory16. CPU14can be a controller for controlling the operations of server12. The CPU14is connected to memory16by, for example, a memory bus. Memory16can include random access memory (RAM) or any other suitable memory device. Memory16can store data and program instructions which are used by the CPU14. Other suitable implementations of server12are possible.

The server12can be the only server or can be one of a group of servers17that includes additional servers18. The group of servers17can be implemented as a computing cluster whereby the server12and additional servers18share resources, such as storage memory, and load-balance the processing of requests to the group of servers17. The group of servers17can alternatively be a cloud computing service. For example, a cloud computing service can include hundreds or thousands of servers configured to provide scalable computing resources. In a cloud computing service, computing tasks can be performed on one or more servers or other computing devices included within the cloud computing service.

The above are only exemplary implementations of the group of servers17, and any distributed computing model can be used in their place. As used herein and in the claims, the term “server” is understood to include any combination or implementation of servers, server groups, or any other configuration of computing devices of any kind.

A network28connects the servers in the group of servers17and a client30and any additional clients40in a group of clients39. Network28is, for example, the Internet. Network28can also be a local area network (LAN), wide area network (WAN), virtual private network (VPN), or any other means of transferring data between the servers17and a group of clients39.

The client30, in one example, can be a desktop computer having an internal configuration of hardware including a processor such as a central processing unit (CPU)32and a memory34. CPU32is a controller for controlling the operations of client30. CPU32can be connected to memory34by, for example, a memory bus. Memory34may be RAM or any other suitable memory device. Memory34stores data and program instructions which are used by CPU32. Other suitable implementations of client30are possible, including handheld computing devices, laptops, or mobile telephones.

A display36configured to display a graphical user interface can be connected to client30. Display36may be implemented in various ways, including by a liquid crystal display (LCD) or a cathode-ray tube (CRT). The display36can be configured to display application windows including a web browser application window38on client30.

Other implementations of the client-server system10are possible. For example, one implementation can omit the group of servers17and additional servers18and include only a single server12. In another implementation, there may only be one client30instead of the group of clients39and additional clients40. In another implementation, additional components may be added to the encoder and decoder system10. For example, one or more peripherals, such as a video camera, can be attached to client30or some of the additional clients40.

FIG. 2is a block diagram41of a web browser42having configurable pipelines46implemented within the client-server computing scheme ofFIG. 1according to embodiments of the disclosed subject matter. Web browser42is implemented on client30. Web browser42can be in the form of computer executable instructions stored in memory34and executed by CPU32. The web browser42includes web application43, web application processor44, web application receiver45, and configurable pipelines46. However, other alternative configurations of web browser42can be utilized.

Web application43represents a web page that includes content to be executed on client30by web browser42. For example, such content can include scripting, such as JavaScript or ECMAScript. Alternatively, web application43can include other content such as Cascading Style Sheets (CSS) or other dynamic content. Web application43can be retrieved from a web server implemented on server12by way of web application receiver45. Alternatively, web application43can be cached on client30and retrieved from client30instead of from server12.

Web application processor44can be used to process instructions (such as script) included in web application43. For example, a JavaScript engine can be included within web application processor44to interpret and execute script included in web application43. Configurable pipelines46are used to process media streams by web browser42. Configurable pipelines46can be initialized, configured, and controlled based on instructions (script) included in web application43. For example, configurable pipelines46can be used to decode video and audio streams.

FIG. 2is a conceptual block diagram depicting an exemplary configuration of a web browser42on client30. Numerous alternative configurations are possible, including those that add to, remove from, and modify the blocks described above.

FIG. 3A-Dare conceptual block diagrams of various exemplary pipeline configurations according to embodiments of the disclosed subject matter. One or more configurable pipelines can be created based on these decoder pipeline configurations. These pipeline configurations are for illustrative purposes only and numerous other configurations are possible, contemplated, and expected.

Source50is a source input that accepts a media stream. Source50can obtain the media stream directly based on a Uniform Resource Locator (URL) or other location provided by, for example, the web application43. Alternatively, the media stream can be provided to source50by another component in web browser42.

Demuxer52is configured to split a media stream into two streams. For example, demuxer52can take as input a combined video and audio stream. Demuxer52can take the individual video and audio streams and output them separately. In basic configuration48a, demuxer52outputs the video stream as output53aand the audio stream as output53b.

Video decoder54is configured to decode the video stream output53a. Video decoder54can be capable of decoding video streams encoded using one or more video encoding schemes. Video renderer56is configured to take the decoded video stream produced by video decoder54and render the decoded video stream to display36via, for example, web browser application window38.

Audio decoder58is configured to decode the audio stream output53b. Audio decoder58can be capable of decoding audio streams encoded using one or more audio encoding schemes. Audio renderer60is configured to take the decoded audio stream produced by audio decoder58and render the decoded audio stream to, for example, an audio card connected to client30. The audio card can, for example, be connected to speakers or headphones to convert the rendered audio into sound waves. Alternative techniques for rendering audio to sound waves are available.

Synchronization62provides for the synchronization of rendering of the audio and video streams. In other words, synchronization62can ensure that video is rendered at the same time as the particular audio corresponding to the video so that, for example, a rendered video of a person speaking (i.e. lips moving) is synchronized with the rendered audio. Synchronization62can be accomplished using, for example, a clock component220as described inFIG. 7below.

FIG. 3Bdepicts a mixer configuration48b. The mixer configuration48bincludes the components of basic configuration48aplus video mixer64and frame buffer generator66. Video mixer64can be configured to combine video from video decoder54with images or video from frame buffer generator66. For example, frame buffer generator66can be configured to generate images containing advertisements. Such images can then be overlaid using video mixer64such that the video from video decoder54is visible in areas where there is not an advertisement. In an alternative implementation, the video mixing can be performed using alpha-blending. In this case, the streams can be overlaid with the appearance of full or partial transparency. Other techniques of video mixing can also be used.

FIG. 3Cdepicts a post-processing configuration48c. The post-processing configuration48cincludes the components of basic configuration48aplus Digital Signal Processing (DSP)68. DSP68can, for example, be configured to take an audio stream output by audio decoder58and enhance it by way of digital signal processing.

FIG. 3Ddepicts a digital rights management (DRM) configuration48d. The DRM configuration48dincludes the components of basic configuration48aplus DRM reader70. DRM reader70can be configured to accept a media stream that is encoded or encrypted with a DRM scheme. DRM schemes are used to restrict media stream access based on, among other things, user licensing and rights. A component capable of processing one or more DRM schemes, such as DRM reader70, is needed to read a media stream coded using DRM.

FIG. 4Ais a block diagram of an exemplary basic configurable pipeline100aincluding built-in components according to embodiments of the disclosed subject matter. Basic pipeline100ais controlled by web application43and includes source50, demuxer52, video decoder54, and video renderer56. Also included are control channels102a-dand data channels104a-f.

The control channels102a-dare used to control pipeline100a. For example, control channels102a-dcan access control functions of the source, components, and renderer of the pipeline to start, stop, pause, fast forward, etc., the processing of the media stream. Control channels102a-dcan include a JavaScript or other interpretive scripting language interface. For example, control channels102a-dcan be accessible by web application43via a JavaScript API. In one such implementation, control channels102a-dare exposed via objects in a document object model (DOM) accessible by the web application43. In pipeline100a, control functions are performed individually for each element in the pipeline by web application43(i.e. sending a stop control signal to video renderer56will not automatically be conveyed to video decoder54).

The data channels104a-fare used to pass data between the elements (source, components and renderer) of pipeline100a. For example, the incoming media stream is output by source50via data channel104ato web application43. Web application43then can pass the data from source50via data channel104bto demuxer52. In the pipeline configuration of pipeline100a, the web application43passes data as described above between each element of the pipeline100a.

The components shown in basic pipeline100acan be implemented by the web browser42on client30as native binary components. Native binary components are included with web browser42and execute natively on the client30on which web browser42is implemented. In other words, a binary component can be written in a computer programming language such as C++, and then compiled to machine code that is executable natively on the client that is used to execute web browser42.

FIG. 4Bis a block diagram of an exemplary configurable decoder pipeline100bincluding built-in components and an interpretive add-in component according to embodiments of the disclosed subject matter. Pipeline100bincludes the components of pipeline100a, with the addition of video enhancer110a. Video enhancer110ais an interpretive add-in component included within web application43. An interpretive add-in component is one implemented in an interpretive language, such as JavaScript instead of a native binary format. In this case, video enhancer110acan be included in web application43to be used within pipeline100b.

Web application43includes video enhancer110ain the pipeline100bby taking data from video decoder54via data channel104eand passing that data to video enhancer110a. Video enhancer110aprocesses the data, and outputs data, which is then passed to video renderer56via data channel104f. In this case, video enhancer110ais shown, which is a component configured to enhance the decoded video. However, any type of component can be used within the pipeline100b, including, for example, an advertising overlay component.

The structure of pipeline100bprovides web application43access to data input and output of each component in pipeline100bas well as control of each component. This granular access to the processing pipeline allows great flexibility for a developer of web application43to include custom processing steps in, for example, a media stream decoding process.

FIG. 4Cis a block diagram of an exemplary configurable decoder pipeline100cincluding built-in components and a binary add-in component according to embodiments of the disclosed subject matter. Pipeline100cis similar to pipeline100b, with the exception of video enhancer110b. Video enhancer110bis implemented as a binary add-in component. A binary add-in component is implemented as natively executable code, similar to native binary components. A binary add-in component can be more efficient than an interpretive component because it can execute natively on the client30.

A binary add-in component can be stored on storage medium accessible via network28, such as on server12. The web browser42can retrieve the binary add-in component in response to instructions included in web application43. The binary add-in component and interpretive add-in components can be structured in a common format and can be implemented using one or more standardized Application Programming Interfaces (APIs).

Binary and interpretive add-in components can be used by web application to dynamically change a configurable pipeline in some implementations. For example, video enhancer110aor110bcould be added into a configurable pipeline for only part of a stream being processed. In another example, video mixer64can be added into the configurable pipeline for a certain time period to display advertisements, and then later be removed when the advertisements are no longer shown.

Alternatively, components can be swapped. For example, various implementations of video decoder54can be interchanged. A first implementation of video decoder54can be configured to decode a first video encoding scheme whereas a second implementation can be configured to decode a second video encoding scheme. The first implementation and second implementation can be interchanged dynamically if the encoding scheme of the stream being decoded changes, or if a new stream with a different encoding scheme is decoded using the configurable pipeline.

FIG. 4Dis a block diagram of an exemplary configurable pipeline100dincluding a pipeline control object120for controlling the configurable pipeline100daccording to embodiments of the disclosed subject matter. The pipeline control object120, which can also be referred to as a pipeline controller, removes granular control of pipeline100dfrom the web application43. Instead, web application43can utilize pipeline control channel122to control the entire pipeline100d. The single point of control provided by pipeline control channel122can simplify the implementation of web application43.

The pipeline control channel122can be configured to take simple commands from web application43and perform more complex control operations with respect to the components of pipeline100d. In one implementation, pipeline control object120is configured with information including: a listing of all components in the configurable pipeline100dincluding, for example, the order of components and the type of component; the data passing technique(s) in use; and the functionality of the pipeline (i.e. playback, encoding, etcetera).

Based on the configuration information, pipeline control object120can be configured to construct the configurable pipeline during initialization. For example, pipeline control object120can instantiate and configure each component and configure the data channels and buffer objects. Once the configurable pipeline is initialized, pipeline control channel122can accept commands to control the configurable pipeline.

In one example, pipeline control channel122can be configured to accept, for example, “play”, “pause”, and/or “seek” commands from web application43. In the instance of “play”, pipeline control object120can be configured to set each component in the configurable pipeline to the run state. Optionally, a start or other command can be sent to one or more components that require it. In the instance of “pause”, pipeline control object120can be configured to set each component in the configurable pipeline to the paused state. In the instance of “seek”, pipeline control object120can be configured to set each component in the configurable pipeline to the pause state, send control signals for each component to flush their input/output ports (i.e. return or deallocate remaining buffer objects), change the location in the stream at the component supplying the stream, and set each component to the run state. Optionally, a start or other command can be sent to one or more components that require it.

Pipeline control channel122allows the web application43to have a simpler interface to pipeline100dwhile still allowing web application43to control and configure the components included in pipeline100d. In one implementation the pipeline control object120can be implemented using a native binary module. Such an implementation can improve the efficiency of the pipeline control object120as compared to an implementation using, for example, JavaScript.

FIG. 4Eis a block diagram of an exemplary configurable pipeline100eincluding data channels between components in the configurable pipeline100eaccording to embodiments of the disclosed subject matter. Pipeline100eis similar to pipeline100awith the exception that pipeline100eincludes provisions for passing data directly between source, components, and renderer. Data is passed via the data channels130a-c.

Direct passing of data between elements in the pipeline100cremoves the overhead of having web application43pass data from data channel to data channel as is shown in pipeline100a. Since web application43typically processes data using an interpreted scripting language, the improvement in performance by passing data directly between native binary components can be significant. Such an improvement in performance can be useful in applications such as real-time communications. However, the direct passing of data can, in some implementations, prevent the use of an interpretive add-in component. But some implementations may allow for a combination of direct passing of data between pipeline elements and also between pipeline elements and web application43.

The passing of data via data channels as described with respect toFIGS. 4A-4Ecan be accomplished using buffer objects. Buffer objects are programming constructs that provide access to memory. Buffer objects can be viewed as a type of memory pointer within web browser42. Buffer objects can be implemented so that they are accessible using JavaScript by way of the DOM of the web browser42.

Buffer objects can include some or all of the following elements: data type, data pointer, data size, properties, allocation methods, and accessor methods. The data type of a buffer object indicates what is stored in the memory controlled by the buffer object. For example, data types can include: raw data, compressed stream, uncompressed audio, uncompressed video, etcetera. In some implementations, a buffer object may have a generic data type, wherein any type of data can be stored.

The data pointer can be a memory pointer that includes a memory address of where the buffer object's data is stored. In some implementations, a buffer object may have multiple data pointers. The buffer object also includes at least one data size, which indicates an amount of memory available to the buffer object with reference to the data pointer(s). For example, the data size may be a number of bytes of available memory.

Buffer object properties can be used to describe some aspect of what is stored in its memory. For example, a property of an audio stream can include its sampling rate. In another example, a property of a video stream can include its color space (i.e. RGB, YUV, etcetera). Available properties can be preset based on the data type. Alternatively or additionally, properties can be custom defined for some buffer object implementations.

Allocation methods are used to allocate memory within the buffer object. The allocation method can be called to allocate memory when the buffer object is created, though in some implementations, it can be called at a later time. In some implementations, memory can only be allocated for a buffer object once. However, in other implementations, the memory available in a buffer object can be changed using the allocation methods over time.

Various implementations of accessor methods can be used to access data stored in a buffer object's memory. Some implementations can allow for direct access to the buffer object memory. In this case, the accessor method would return a memory pointer to allow for direct access to memory. However, such an implementation could be platform dependent (i.e. require differing implementations for various operating systems). A more generic implementation could include copying the data stored in the buffer object to a temporary memory location made accessible outside of the buffer object.

Another implementation of accessor methods can include providing access to data on a less granular basis. In other words, data can be made available on a pixel, block, line, frame, or other level depending on the type of data stored by the buffer object. Alternatively, data can be provided as return values from accessor functions of the buffer object. While such accessor functions may provide a less efficient way to access the data, they would provide a more generic and less platform dependent means of accessing the data.

FIG. 5is a block diagram of an exemplary component150according to embodiments of the disclosed subject matter. A component is the building block of the pipelines described herein. A component can include any type of processing that involves taking one or more inputs, processing those inputs, and then outputting the results of the processing to one or more outputs. Typically a component will have one input and one output as is shown by exemplary component150. However in certain situations, it can be advantageous to have multiple inputs and/or multiple outputs. In one example, a demuxer can take a combined audio and video stream, split the audio and video components apart, and output each of the split components to separate outputs. Alternatively, a component can take two video inputs and combine or overlay those inputs into a single output. Either of the above examples can be modified to accommodate combined streams including three or more streams by increasing the number of input or output ports of the component. Depending on a component's purpose, in some implementations, the component can have only inputs (and no outputs) or only outputs (and no inputs).

Exemplary component150includes an input port152. The input port152includes an input queue154, input channel156, and return channel158. In one implementation, a circular buffer technique is used to pass data into component150. The buffer technique includes passing buffer objects into the component via input channel156. Input queue154is used to retrieve the contents of the memory referenced by the buffer objects. Once the contents are retrieved by component150, the buffer object is returned via return channel158so that the memory referenced by the buffer object can be reused. Input port152can restrict the type of buffer objects accepted via input channel156. For example, a video decoder component may only accept a buffer object containing compressed video data.

Exemplary component150also includes an output port160. The output port160includes an output queue162, return channel164and output channel166. In one implementation, a circular buffer technique is used to pass data from component150. The buffer technique includes passing buffer objects out of the component via output channel166. The output queue162includes buffer objects that are used to store data output by component150. The buffer objects are sent to the next element in the pipeline by way of output channel166. Once the output data is retrieved from the buffer objects, they are returned to component150by way of return channel164so that they can be reused.

Component150can also include a command queue168and event handlers170. Command queue168can be configured to accept commands for controlling component150. For example, command queue168can include functionality to accept commands from web application43through the use of JavaScript code. Possible commands can include, for example, initializing or otherwise changing the state (described later with respect toFIG. 6) of component150. A component can have attributes that are configurable by web application43. For example, an audio encoder component can have attributes including encoding bit rate, codec type, and sampling frequency. In another example, a blurring component can have an attribute defining how much to blur the video passed into the component. Components can be controlled asynchronously or synchronously, depending on the implementation and the component.

Event handlers170can be configured to provide information to web application43of the current state of component150and any events occurring in component150. For example, event handlers170can be configured to notify web application43when the state of component150changes. In another example, a component150implementing a decoder can be configured to notify web application43upon a change in resolution, frame rate, or color of the video stream being decoded.

The core of component150is its processing module172. Processing module172is used to process data retrieved by input port152to be output by output port160. Processing module172can include any process for transforming data. For example, processing module172can include a decoder, quality enhancer, discrete cosine transform (DCT), digital rights management (DRM) decoder, color filter, resolution scaler, or any other type of processing module. While a component typically will transform the input data to produce a different output data, some components may not change the input data. For example, a component may be configured to simply inspect the data for a particular condition or occurrence.

A component can have a unique identification code (UID)174to allow the component to be identified for instantiation by web application43. UID174can be generated using a UID generation process, or may be issued by a centralized UID repository. However, in some implementations, a UID174may not be required if the component is, for example, provided by web application43.

The implementation of component150described is exemplary only and alternative implementations are possible and expected. For example, input port152and output port160may utilize different techniques of memory management to receive and send data. For example, buffer objects may be passed through the component150(i.e. instead of returning the pointer via return channel158, the pointers are used to output data via output channel166). In such a technique, once at the end of the configurable pipeline, the buffer object's memory can be deallocated, or the buffer object may be passed back to the beginning of the configurable pipeline. Alternatively, some components may include different numbers of input and/or output ports. In some implementations, processing module172can include merely storing some or all of a stream to a device or accessing some or all of a stream from a device.

FIG. 6is an exemplary state diagram190related to the operation of the exemplary component150ofFIG. 5according to embodiments of the disclosed subject matter. Uninitialized component150begins in an uninitialized state192upon execution. The state of component150transitions to initialized/stopped state194once the component is initialized. Initialization can include steps such as allocating memory to the input and output ports and initializing variables within component150. In the initialized/stopped state194, component150does not process, accept, or output data. Once in initialized/stopped state194, component150can transition to any of running state196, paused state198, or error state200.

When in running state196, component150retrieves input data from input port152, processes the data using processing module172, and outputs the processed data using output port160. When in paused state198, component150accepts input data using input port152, but does not process any of the input data. The component150can transition to error state200if there is a fatal error and the component is unable to continue to perform normally. When component150is in error state200, web application43can be notified by way of, for example, event handlers170.

The above state diagram190is illustrative of only one potential implementation of a component150. Alternative implementations are possible, including those that add, remove, and modify the states illustrated in state diagram190.

FIG. 7is a block diagram of an exemplary clock component220according to embodiments of the disclosed subject matter. Clock component220provides generic functionality for synchronizing multiple output streams for playback. For example, clock component220can be utilized to synchronize the rendering of an output audio stream and an output video stream. In other words, clock component220is capable of synchronizing the outputs of various configurable pipelines. This functionality is important to the rendered streams, since the time needed to process and render the various streams through separate configurable pipelines may differ.

In one exemplary implementation, clock component220includes updates222, control224, queries226, callback module228, and callbacks230. The stream of one configurable pipeline is designated a master stream. The master stream updates clock component220via updates222with its current media time (i.e. indicating the current location of rendering the stream) and other information that can vary from implementation to implementation. For example, in one implementation, the other information can include playback speed. In a typical implementation, the master stream passes this information to updates222on a specific periodic interval. Between the updates, clock component220can maintain synchronization using a system clock of client30. The information can be determined and passed from a renderer component, such as audio renderer60for a master audio stream.

Other streams that are to be synchronized by clock component220are slave streams output by other configurable pipelines. The slave streams can query for the current media time and other information of the master stream from queries226so that the slave streams can synchronize with the master stream. For example, a renderer component, such as video renderer56can query information for a slave video stream from queries226. Alternatively, a renderer component can set up one or more callbacks230using callbacks module228. A callback is a technique whereby a synchronized component can stop processing or “sleep” and be triggered to continue processing by the callback when a particular media time or other condition is reached.

In other implementations, clock component220can be used to synchronize streams between components in the same configurable pipeline and before rendering. For example, clock component220could be used to synchronize the streams from frame buffer generator66and video decoder54with respect to mixer configuration48b.

Clock component220can be implemented as a native binary component or as an interpretive add-in component. For example, clock component220can be included in web browser42as a native binary component to improve the efficiency and ease of use of clock component220. However, clock component220can alternatively be implemented as an interpretive add-in component. Such an implementation permits the use of clock component220, even when a particular web browser42does not natively include clock component220.

Clock component220can be accessed and controlled from web application43via control224having a JavaScript code API accessible via a DOM interface. For example, the web application43can initialize, start, stop, or update the playback speed controlled by clock component220. Clock component220can be controlled similarly to other components in the configurable pipeline(s) that it is a part of, including by web application43directly and using an intermediate pipeline control object120.

The embodiments of server12and/or client30(and the algorithms, methods, instructions etc. stored thereon and/or executed thereby) can be realized in hardware including, for example, IP cores, ASICSs, programmable logic arrays, optical processors, programmable logic controllers, microcode, firmware, microcontrollers, servers, microprocessors, digital signal processors or any other suitable circuit. In the claims, the term “processor” should be understood as encompassing any the foregoing, either singly or in combination. The terms “signal” and “data” are used interchangeably. Further, portions of server12and client30do not necessarily have to be implemented in the same manner.

Further, in one example, server12or client30can be implemented using a general purpose computer/processor with a computer program that, when executed, carries out any of the respective methods, algorithms and/or instructions described herein. In addition or alternatively, for example, a special purpose computer/processor can be utilized which can contain specialized hardware for carrying out any of the methods, algorithms, or instructions described herein.

Server12and client30can, for example, be implemented on computers in a webmail system. Client30can be implemented on a device such as a hand-held communications device (i.e. a cell phone). In this instance, server12can exchange HTTP communications with the communications device. Other suitable server12and client30implementation schemes are available. For example, client30can be a personal computer rather than a portable communications device.

Implementations or portions of implementations of the above disclosures can take the form of a computer program product accessible from, for example, a computer-usable or computer-readable medium. A computer-usable or computer-readable medium can be any tangible device that can, for example, contain, store, communicate, or transport the program for use by or in connection with any processor. The medium can be, for example, an electronic, magnetic, optical, electromagnetic, or a semiconductor device. Other suitable mediums are also available. Such computer-usable or computer-readable media can be referred to as non-transitory media, and may include RAM or other volatile memory or storage devices that may change over time.