Abstract:
A communication mechanism or facility for enabling interaction with, and control of, a remote presentation engine for presenting a rich media application, such as one written using animated graphics allows the presentation engine to be configured or instructed remotely to perform certain actions, to send requests, and/or to send information, independently of the animated graphics application being executed. The communication facility between a workstation enables a presentation engine to pass a request by the application for resource stored on a workstation—an image file or a data file, for example—thereby creating a virtual file system. Local resource references can thus be maintained in the application during development without having to download all resources to the device in connection with testing

Description:
CROSS REFERENCE TO RELATED APPLICATION 
       [0001]    This application claims the benefit of provisional application Ser. No. 60/970,438, filed Sep. 6, 2007, the disclosure of which is incorporated herein by reference. 
     
    
     TECHNICAL FIELD OF THE INVENTION 
       [0002]    The present invention relates generally to interaction with an animated graphics presentation engine on a remote, a resource-limited device. 
       BACKGROUND OF THE INVENTION 
       [0003]    “Embedded systems” are, generally speaking, special purpose computer systems designed to perform certain dedicated functions that have been embedded into a device. Examples include mobile telephones and other hand-held or mobile devices, and set top boxes for cable and satellite television. Microprocessors or other logic circuits embedded in these devices or equipment are programmed to perform specific functions for the device. Economic considerations dictate that embedded systems have limited processing capability and memory and, at least in the case of mobile devices, smaller screens. Typically, it is just enough to perform their intended functions. Resource-limited devices, such as these, are generally not intended to be independently programmable by end users. They sometimes do not permit the user to load additional applications. 
         [0004]    Computer applications are typically written using standard computational programming languages, such as C and C++. However, in order to more easily create sophisticated user interfaces and other applications providing rich media content and experiences, developers are turning to the use of graphics-oriented programming languages and platforms for developing rich media applications. These platforms reduce the burden of programming media-intensive interfaces and rich media applications by taking advantage of development tools oriented toward graphics and rich media, and presentation engines that perform much of the graphics processing. 
         [0005]    Examples of applications that can be written for set top boxes in a graphics language or platform include those that enable users to interact with advanced network services, such as video on demand (VOD) services, digital video recorder (DVR) services, and electronic program guides, as well as games and many other types of applications. Similarly, on a mobile network, a network operator may want to deploy rich media applications that can be downloaded as required to the mobile device for allowing easy interaction with services offered by a mobile network, such as, for example, mobile television, music, and podcasting, in addition to services that enable easy access to remote devices and internet-based content. 
         [0006]    In an application written using animated graphics, a series of static displays are sequentially rendered to create the illusion of animation. Each of these displays will be referred to generally as a “frame.” Examples of such development environments include the Adobe® Flash® development tools, which generate SWF files, and programs written using SVG, which is a language for describing two-dimensional graphics. These applications or files describe graphical elements, text and other elements that are to be rendered, typically using standard vector graphic techniques. They also specify, for each frame, the placement of elements on a “canvas” within the frame. Bit map images, video and audio can also be referenced and placed and displayed or played according to a time line. A “presentation engine” or “player,” reads the descriptions of the frames and the graphical elements and renders the frames according to the specified time line, and executes scripts associated with each frame and interaction with a user or the device on which the file is being executed. 
         [0007]    For example, the Flash® development environment, which is widely used for creating web-based applications, generates a SWF file that encodes descriptions of the graphical elements, a description of each frame in terms of placement of graphical elements on the canvas for the frame, and any scripts that are to executed in connection with rendering of the frame or the user&#39;s interaction with it. The frames are rendered a specified frame rate. Similarly, SVG specifies document containing one or more “pages” for display. Each page contains an ordered grouping of graphical elements. Only one grouping is displayed at a time. Animation is created by using Synchronized Multimedia Integration Language (SMIL) to specify timing for rendering each page. Scripts are used to provide interaction with the elements and navigation between the pages. 
         [0008]    A rich media application written using an animated graphics format or language does not need to be concerned with the details of rendering the graphics and coordinating video and audio, simplifying development and reducing the size of the applications. Scripts can be kept relatively simple by taking advantage of application programming interfaces (APIs) that are implemented by the presentation engines. The APIs typically provide a suite of standard functions, as well as functions useful for providing interactivity with the animated graphics and to controlling or interacting with the devices. Developers of applications are thus able to concentrate on the details of the applications, while developers of the presentation engine focus on enhancing playback performance of the presentation engine for particular devices and extending its functionality. 
         [0009]    In addition to shortened development and deployment cycles, a further benefit of using animated graphics languages and authoring tools to generate rich media applications is that it allows dividing the task of writing the applications between graphics designers, who use authoring tools to create the graphical portions of the programs, and programmers who write scripts to add functionality and interactively. 
         [0010]    One example of a presentation engine for executing a rich media application written using animated graphics on a resource limited device is the MachBlue™ presentation engine distributed by Bluestreak Networks, Inc. of Montreal, Canada. Other examples are the Flash® and Flash Lite® players of Adobe Systems, Inc. The MachBlue™ presentation engine is designed to run as middleware on embedded systems with limited processing power and memory, such as set top boxes for cable and satellite television and similar devices. The MachBlue™ presentation engine interprets and renders files encoded in SWF file format (though it does not support rendering all features of Flash® authoring environment) as well as certain extensions to the SWF file format adapted for specific devices. 
       SUMMARY 
       [0011]    Typically, a presentation engine on a resource limited device simply executes an animated graphics application when it is loaded, performing processes according to the rich media application. The invention in a preferred embodiment relates to communication mechanisms or facilities for enabling interaction with, and control of, a remote presentation engine for presenting a rich media application, such as one written using animated graphics. In the example described below, the communications mechanism allows a presentation engine to be configured or instructed remotely to perform certain actions, to send requests, and/or to send information, independently of the animated graphics application being executed. 
         [0012]    Such a communication mechanism or facility can provide several advantages and benefits. It can be utilized to configure and control the manner in which a remote presentation engine executes a rich media application on a resource-limited application. Controlling the execution would be useful in connection with, for example, debugging or profiling the application. When developing applications that will run a general purpose computer, a developer need only save the file in the necessary format and execute it. However, when developing an animated graphics application for an embedded system on a resource-limited device, the developer will typically load the application on the device and observe its behavior as it is being run. With a communications facility, instructions can be selectively sent to a presentation engine to configure the presentation engine to, for example, collect and send to a workstation traces embedded in an animated graphics application and to measure and return performance metrics on the execution of the application. 
         [0013]    According to a different aspect of a preferred embodiment of the invention, a communication facility between a workstation enables a presentation engine to pass a request by the application for resource stored on a workstation—an image file or a data file, for example—thereby creating a virtual file system. Local resource references can thus be maintained in the application during development without having to download all resources to the device in connection with testing. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0014]      FIG. 1  is a schematic diagram representing certain computer processes and files on a workstation and a remote, resource-limited device. 
           [0015]      FIG. 2  is a schematic diagram representing hardware components of the workstation and the remote, resource-limited device of  FIG. 1 . 
           [0016]      FIG. 3  is a flow diagram representing an example of a basic process of communicating between a server operating on a workstation and a presentation engine on a remote, resource-limited device. 
       
    
    
     DETAILED DESCRIPTION 
       [0017]    In the following description, like numbers refer to like elements. 
         [0018]      FIG. 1  schematically represents software-implemented processes and files on a workstation  12  and resource-limited remote device  20 .  FIG. 2  schematically represents the hardware for storing and executing software program instructions for performing the processes. 
         [0019]    Animated graphics application  10  is stored on a general purpose computer workstation  12 . The workstation could have been used to author, in whole or in part, the application, and thus may include program authoring tools that are stored and executed on the workstation. The workstation includes a processor  14 , memory  16  for temporary storage of executing programs and data, and one or more disks  18  for storage of program and data files. The workstation is coupled to a remote device  20 , such as a set top box or mobile, hand-held device with wireless communication capabilities, for example a cellular telephone or device with “Wi Fi” capabilities. The device includes a resource-limited or constrained embedded system  22  comprised of a central processing unit  24  for executing program instructions and files stored in memory  26 . The device will also have additional elements relating to the particular purpose of the device. For example, if the device is a satellite or cable set top box, it would also include a tuner and interfaces for video and audio. In the illustrated example, the device includes a display  28 , such as would be typically found on a mobile telephone. Memories  18  and  26  are intended to represent memory, in general, and are not intended to represent any particular memory structure. For example, memory in an embedded system will depend on the purpose of the system, but it typically will include some type of memory for long term storage (typically non-volatile) and working memory for use by the processor in storing program code and data. 
         [0020]    The workstation  12  and remote device  20  are coupled through a physical communications channel  30 . This communications channel is comprised of one or more links. Each link could be comprised of, for example, a wired or wireless connection. Examples of wired connections include serial, USB, and Ethernet connections. Examples of wireless connections include Bluetooth, wireless local or metropolitan area network connections (IEEE 802.11 or 802.16, for example), and cellular telephone and data. 
         [0021]    A presentation engine  32  for executing or playing animated graphics files on a resource-constrained remote device is loaded on remote device  20 . Communication server processes  34  executing on workstation  12  establish an application-level communication session  36  with presentation engine  32  over the physical communications channel  30 . The server processes are further comprised of processes for enabling exchanging information with the presentation engine, including requests that control operation of presentation engine, such as by configuring its execution, and exchanging of files. The server processes pass information to and from other applications that are running on the workstation, which want to communicate with the presentation engine. For example, authoring applications or other development tools such as debugging tools can utilize the communications facility. These applications could be implemented to include the server processes, in whole or in part, or by utilizing plug-ins that provide these services. Alternately, these processes can be implemented as an independent application. The presentation engine either includes, or is configured to utilize, software for establishing the communication session and exchange communications with the server. 
         [0022]    Resource limited devices come in many different varieties and have different capabilities for connecting to computers and networks. For example, some set top boxes support Ethernet connections and many mobile telephones do not. Mobile telephones may, on the other hand, support Bluetooth wireless connections. The processes on the workstation and the remote device for establishing and using the communication session are preferably implemented at the application level so that the communications session is independent of the nature of the medium over which communication with the presentation engine is taking place. This allows the sure communications protocol to be used over any type of communications medium independent of the processes used to establish and send information over the medium. However, the software program implementing the server processes, such as, for example, an authoring tool or debugging tool, preferably includes mechanisms or processes for configuring different types of communications mediums between a workstation and the resource limited device, over which the communications with the presentation engine may take place. 
         [0023]    A representative example of a process flow  100  utilizing the communication between the presentation engine  32  and the server  34  is illustrated by the flow chart of  FIG. 3 . 
         [0024]    Referring now also to  FIG. 3 , in addition to  FIGS. 1 and 2 , the representative process starts with selection of a communications method at step  102 . The server processes  34  preferably assist with configuring or setting up the communication medium with the remote device, depending on the type of device. For example, if it is TCP/IP connection using Ethernet interfaces at the workstation and the remote device, the application implementing the server processes can be used to enter and store IP addresses for configuring the connection. If the remote device is a cellular telephone, for example, the connection may use a Bluetooth wireless connection. The server processes could store profiles for different remote devices. Another example that can be used for Windows CE based devices is ActiveSync USB connection. Once the connection method is chosen and any configuration information not previously stored is entered, the server software initiates setting up the physical connection to the remote device at step  104  over the physical communications channel  30 . The connection depends on the software and hardware installed on workstation and remote device, to which the server software has access. Alternately, the connection may be set up manually. 
         [0025]    At step  106 , the server software sends over the established communication link a request for connection to the presentation engine  32  on the remote device. The remote presentation engine  32  confirms the request by sending back certain information that identifies the device and details of the presentation engine at step  108 . This establishes a communication session between the presentation engine and the server. The remaining steps in the flow chart are illustrative only, and can be performed out of order, depending on the purpose to which the communication facility is put. 
         [0026]    For example, the workstation at step  110  transmits a request to the presentation engine to configure itself. This is a form of controlling the presentation engine. In the example of using the communication facility to control the presentation engine for collecting debugging information and performance metrics for profiling, the request could include a list of metrics to collect. The presentation engine would then turn on these services prior to executing a rich media application. The presentation engine configures itself at step  112  and confirms the request to the server. 
         [0027]    The workstation sends at step  114  a rich media application file to the device for execution, and at step  116  the remote device  20  receives the file. The presentation engine  32  acknowledges that the file has been successfully transferred by sending a message to the server  34  at step  118 . 
         [0028]    The presentation engine then, as indicated by step  120 , launches execution of the rich media application. In this example, the application is written as a frame-based animated graphics movie comprised of a sequences of frames. The next frame, or the first frame if there was no previous frame, is executed by the presentation engine at step  122 . 
         [0029]    When executing the application, the presentation engine may determine that it needs a resource referenced in the file, for example, a bitmap image such as a JPEG file or a XML file containing data, that is not stored on the remote device but rather on the remote host. The application might only specify the name of a file, and not a URL identifying the location of the file on the host or an otherwise fully qualified path to a directory on a host where the file can be found. The presentation engine is preferably enabled to send to the workstation, at step  126 , a request for a resource file  38  on the workstation that it does not have and that is required for rendering of the frame or executing a script associated with the frame. When the request is received by the workstation server, the server assumes at step  128  that the file is located in the same directory as the rich media application  10  and transfers this file to the remote device at step  130  if it is found. If the file is not found, an error message is sent. This capability allows local resource references to be maintained in the application during development without having to download all resources to the device in connection with testing or include in the application a URL to the file on the workstation host. 
         [0030]    The presentation engine could be configured to provide information in connection with execution. In the example of the communication facility being used in connection with a debugging or profiling tool, this might be execution metrics or statistics. At step  132 , if the presentation engine has been configured to generate information or send a message when a predefined execution event has occurred, it will send to the server on the workstation at step  134  the message or information. Steps  122  to  134  are repeated until the application ends or is stopped, as represented by decision step  136 . 
         [0031]    The following is an illustrative example of a transfer protocol for implementing a preferred embodiment of communications facility for purposes of assisting with debugging and profiling of an animated graphics application being executed by a presentation engine on a remote device, and is not intended to limit the general concepts expressed above. 
         [0032]    Data sent between a workstation and a remote device is, in this example, packaged in a packet. In this example, a packet corresponds to either a “message” (no results are returned) or a request (a result is returned). (The term “message” used elsewhere in this description is not limited to this particular meaning and, instead, is intended to mean any transfer of information.) This is controlled by a flag in all packets that indicates if a result must be returned by the receiver. A packet contains two parts: a packet header and packet data or payload. The packet header includes information on the size of the data part of the packet, a validation field and a CRC field for checking data integrity. The payload depends on the type of packet. The packet data comprises what will be termed generically below as a “request,” even though the request may actually constitute a message if no result is expected. 
         [0033]    A request header includes one or more of the following types of information: an identifier of the type of packet, for example a trace, a file transfer, or a device information request; the size of the request; a unique request identifier; an application identifier that identifies a running animated graphics application file; and flags that affect the handling of a request. For example, a flag can be set to indicate that a result must be returned by the request receiver to indicate the request completion result. 
         [0034]    Following are examples of request types for implementing this example. 
         [0035]    An “Establish Connection” request is sent by the newly connected client to notify the server on the workstation that it is now waiting for requests. The servers send the request. The remote client should respond by sending a Device Information request. 
         [0036]    A “Device Info” request is sent by the client on the remote device to give information on the presentation engine and device to the newly connected client. This request is sent immediately after an Establish connection request is received. The request preferably includes, for example, the protocol version, which indicates the device supported version of the communication protocol, the device platform type, which identifies the hardware/software platform on which the presentation engine is running, and the version of the presentation engine. 
         [0037]    A “Result” request sends the result of a received request. It preferably contains a unique identifier of the received request and the requested result. 
         [0038]    A “File Transfer” request is used to send a file to the remote client. It preferably includes the name of the file, its size, and the actual file content, and a flag to indicate whether the file must be executed or launched by the presentation engine. 
         [0039]    Additional requests can be structured to configure the presentation engine and to request and receive additional information from the presentation engine, which in this example is information about the execution of the animated graphics application by the presentation engine. 
         [0040]    A “Trace” request is used by the presentation engine client to send a SWF trace. It includes an indication of the type of encoding used for the data string, for example ANSI, UCS2, UCS4, etc., the size of the data string, and the actual data string containing the trace information. 
         [0041]    A “Frame Info” request is used to transmit metrics about the execution of a frame of the animated graphics application. The metrics is packaged as a list of value-pairs. Each value-pair includes an identifier of the metric, for example the time it took to render the frame, the amount of memory used, the time it took to execute scripts associated with the frame, etc., and a data value. 
         [0042]    To configure the remote presentation engine, the server on the workstation sends a Configuration request. In the debugging/profiling example, the Configuration request specifies in its data part a list of metrics to calculate and return after execution of each frame of the animated graphics application is completed. This request is preferably sent immediately after the connection is established with a remote device. 
         [0043]    The foregoing description is of exemplary and preferred embodiments of a communications facility for controlling a presentation engine executing an animated graphics program on a remote, resource-limited device, employing at least in part certain teachings of the invention. The invention is not limited to the described examples or embodiments. Alterations and modifications to the disclosed embodiments may be made without departing from the invention. The meaning of the terms used in this specification are, unless expressly stated otherwise, intended to have ordinary and customary meaning and are not intended to be limited to the details of the illustrated structures or the disclosed embodiments. None of the foregoing description is to be read as implying that any particular element, step, or function is an essential element which must be included in the claim scope. The scope of patented subject matter is defined only by the issued claims. None of these claims are intended to invoke paragraph six of 35 USC §112 unless the exact words “means for” or “steps for” are followed by a participle.