Source: http://www.freepatentsonline.com/9075631.html
Timestamp: 2019-03-26 00:41:33
Document Index: 424697470

Matched Legal Cases: ['application No. 12189044', 'application No. 12189052', 'application No. 2', 'application No. 12189044', 'application No. 12189044', 'application No. 12189044', 'application No. 12189044', 'application No. 2', 'application No. 12189052', 'application No. 2', 'application No. 2', 'application No. 12189085', 'application No. 2', 'Application No. 2', 'Application No. 61']

Method of rendering a user interface - BlackBerry Limited
United States Patent 9075631
A user interface (UI) is presented in which a UI client engine is associated with an application, and a UI rendering engine is associated with the client engine. The UI rendering engine receives the scene graph and data items associated with elements of the scene graph, and processes a rendering thread to render a UI in accordance with the scene graph and the data items, independently of further input from the client UI engine.
Lewin, Mathias (Malmö, SE)
Andersson, Jens Ola (Malmö, SE)
Ljungkrantz, Peter (Malmö, SE)
Gurell, Christoffer (Malmö, SE)
Mowitz, Johan (Malmö, SE)
13/654914
G06F3/048; G06F9/44; G06T15/00
Download PDF 9075631 PDF help
20130055072 Multi-Threaded Graphical Display System 2013-02-28 Arnold et al. 715/240
8365144 Proxy graphical user interface generation 2013-01-29 Webb
20120144333 RENDER TRANSFORM BASED SCROLLING AND PANNING FOR SMOOTH EFFECTS 2012-06-07 Shatalin et al. 715/771
20120044251 GRAPHICS RENDERING METHODS FOR SATISFYING MINIMUM FRAME RATE REQUIREMENTS 2012-02-23 Mark et al. 345/474
20110119615 Adaptive Data Loader 2011-05-19 Cisler
20100325565 APPARATUS AND METHODS FOR GENERATING GRAPHICAL INTERFACES 2010-12-23 Skinner et al.
20100289804 SYSTEM, MECHANISM, AND APPARATUS FOR A CUSTOMIZABLE AND EXTENSIBLE DISTRIBUTED RENDERING API 2010-11-18 Jackman et al.
7703045 Method and apparatus for interacting with objects in a user interface 2010-04-20 Kawahara et al. 715/853
20090139778 User Input Using Proximity Sensing 2009-06-04 Butler et al.
7479949 Touch screen device, method, and graphical user interface for determining commands by applying heuristics 2009-01-20 Jobs et al. 345/173
7436406 Scene graph based display for desktop applications 2008-10-14 Dautelle
20080155478 Virtual interface and system for controlling a device 2008-06-26 Stross
7386856 Extension of commanding to control level 2008-06-10 Need et al.
20080005703 Apparatus, Methods and computer program products providing finger-based and hand-based gesture commands for portable electronic device applications 2008-01-03 Radivojevic et al.
7290216 Method and apparatus for implementing a scene-graph-aware user interface manager 2007-10-30 Kawahara
7290208 Executing a dialog using one or more XML components and one or more embedded scripts 2007-10-30 Saborowski et al.
7275221 Method and system for representing hierarchal structures of a user-interface 2007-09-25 McKeon 715/853
20070211022 METHOD AND DEVICE FOR THREE-DIMENSIONAL SENSING 2007-09-13 Boillot
20060244754 Intelligent caching data structure for immediate mode graphics 2006-11-02 Beda et al.
7012606 System and method for a unified composition engine in a graphics processing system 2006-03-14 Swedberg et al.
20050182844 Efficient communication in a client-server scene graph system 2005-08-18 Johnson et al. 709/230
6919891 Generic parameterization for a scene graph 2005-07-19 Schneider et al.
20050108735 Extension of commanding to control level 2005-05-19 Need et al.
20040189669 System and method for managing visual structure, timing, and animation in a graphics processing system 2004-09-30 David et al.
6791587 Method and component for navigating a hierarchical user interface representation 2004-09-14 Bamford et al.
5880729 Graphical user interfaces having animated control elements 1999-03-09 Johnston et al. 715/823
EP1566754 2005-08-24 Efficient communication in a client-server scene graph system
Extended European Search report mailed Feb. 5, 2013, in corresponding European patent application No. 12189044.6.
Extended European Search report mailed Oct. 22, 2013, in corresponding European patent application No. 12189052.9.
Office Action mailed Oct. 31, 2013, in corresponding U.S. Appl. No. 13/654,930.
Office Action mailed Apr. 22, 2014; in corresponding Canadian patent application No. 2,792,662.
Examination Report mailed Dec. 12, 2013; in corresponding European patent application No. 12189044.6.
Extended European Search Report mailed Feb. 5, 2013; in corresponding European patent application No. 12189044.6.
Response filed to the Extended European Search Report on Oct. 24, 2013; in corresponding European patent application No. 12189044.6.
Response filed to the Examination Report on Apr. 22, 2014; in corresponding European patent application No. 12189044.6.
Office Action mailed Apr. 9, 2014; in corresponding U.S. Appl. No. 13/654,930.
Office Action mailed Mar. 10, 2014; in corresponding Canadian patent application No. 2,792,900.
Response to the Extended European Search Report filed May 20, 2014; in corresponding European patent application No. 12189052.9.
Non-Final office Action mailed Oct. 6, 2014; in corresponding U.S. Appl. No. 13/654,947.
Notice of Allowance and Fee(s) Due mailed Nov. 7, 2014; in U.S. Appl. No. 13/654,930.
Response to Examiner's Requisition filed Oct. 21, 2014; in corresponding Canadian patent application No. 2,792,895.
Response to Examiner's Requisition filed Oct. 21, 2014; in corresponding Canadian patent application No. 2,792,662.
Extended European Search Report mailed Sep. 24, 2014; in corresponding European patent application No. 12189085.9.
Office Action mailed Apr. 22, 2014; in Canadian patent application No. 2,792,895.
Office Action issued in Canadian Application No. 2,792,900 on Jan. 15, 2015; 6 pages.
This application claims priority to U.S. Provisional Application No. 61/548,652, filed Oct. 18, 2011 which is fully incorporated by reference herein.
1. A method of rendering a user interface (UI), the method comprising: receiving, from a UI client engine associated with an application, a UI component tree for the application; determining a visual node tree for the UI component tree; asynchronously receiving, from the UI client engine, UI data items related to elements of the UI component tree; populating the visual node tree with the UI data items; and rendering the UI in accordance with the visual node tree independently of further input from the UI client engine.
2. The method of claim 1, further comprising: detecting a user input; and communicating the user input to the UI client engine.
3. The method of claim 2, further comprising re-rendering the UI in response to the user input independently of further input from the UI client engine.
4. The method of claim 3, wherein the user input is a button press, and the re-rendering the UI comprises re-rendering the UI to animate a button associated with the button press.
5. The method of claim 3, wherein the user input is a slider drag, and the re-rendering the UI comprises re-rendering the UI to animate a dragging of a slider associated with the slider drag.
6. The method of claim 3, wherein the user input is a list scroll, and the re-rendering the UI comprises re-rendering the UI to scroll a list associated with the list scroll.
7. The method of claim 2, further comprising: receiving from the UI client engine further data items invoking a modification, and re-rendering the UI in accordance with the further data items and an associated visual node tree, independently of further input from the UI client engine.
8. The method of claim 7, wherein the modification is an animation effect.
9. The method of claim 1, further comprising the UI rendering engine: carrying out one or more UI modification steps from a queue prior to rendering the UI; when a time remaining to render the UI is determined as sufficient to carry out a next modification step in the queue, carrying out the next modification step prior to rendering the UI; and when the time remaining to render the UI is not sufficient to carry out the next modification step, rendering the UI.
10. The method of claim 9, further comprising, when a time required to carry out one of the one or more UI modification steps and render the UI exceeds the time allowed by a frame-rate of the UI, the UI rendering engine rendering a placeholder or rendering partial results prior to completion of that UI modification step.
11. A product stored on a non-transitory medium carrying instructions which when executed by a processor of an electronic device having a display for displaying a user interface (UI), cause the electronic device to carry out a method of rendering a user interface (UI), the method comprising: receiving, from a UI client engine associated with an application, a UI component tree for the application; determining a visual node tree for the UI component tree; asynchronously receiving, from the UI client engine, UI data items related to elements of the UI component tree; populating the visual node tree with the UI data items; and rendering the UI in accordance with the visual node tree independently of further input from the UI client engine.
12. An electronic device, comprising: a display for displaying a user interface (UI); a processor; and a memory carrying instructions which when executed by the processor configure the processor to provide: a UI rendering engine configured for: receiving, from a UI client engine associated with an application, a UI component tree for the application; determining a visual node tree for the UI component tree; asynchronously receiving, from the UI client engine, UI data items related to elements of the UI component tree; populating the visual node tree with the UI data items; and rendering the UI in accordance with the visual node tree independently of further input from the UI client engine.
13. The electronic device of claim 12, the memory carrying further instructions which when executed by the processor configure the processor to provide: said UI client engine associated with an application, the UI client engine being configured for: determining said UI component tree for the application, receiving inputs from the application related to elements of the UI component tree, and determining said UI data items related to the inputs.
14. The electronic device of claim 12, wherein the UI rendering engine is further configured for: detecting a user input; and communicating the user input to the UI client engine.
15. The electronic device of claim 14, wherein the UI rendering engine is further configured for re-rendering the UI in response to the user input independently of further input from the UI client engine.
16. The electronic device of claim 15, wherein the user input is a button press, and the re-rendering the UI comprises re-rendering the UI to animate a button associated with the button press.
17. The electronic device of claim 15, wherein the user input is a slider drag, and the re-rendering the UI comprises re-rendering the UI to animate a dragging of a slider associated with the slider drag.
18. The electronic device of claim 15, wherein the user input is a list scroll, and the re-rendering the UI comprises re-rendering the UI to scroll a list associated with the list scroll.
19. The electronic device of claim 12, wherein the UI rendering engine is further configured for: receiving from the UI client engine further data items invoking the modification, and re-rendering UI in accordance with the further data items and an associated visual node tree, independently of further input from the UI client engine.
20. The electronic device of claim 19, wherein the modification is an animation effect.
21. The electronic device of claim 12, wherein the UI rendering engine is further configured to: carry out one or more UI modification steps from a queue prior to rendering the UI; when a time remaining to render the UI is determined as sufficient to carry out a next modification step in the queue, carry out the next modification step prior to rendering the UI; and when the time remaining to render the UI is not sufficient to carry out the next modification step, render the UI.
22. The electronic device of claim 21, wherein the UI rendering engine is further configured to, when a time required to carry out one of the one or more UI modification steps and render the UI exceeds the time allowed by a frame-rate of the UI, render a placeholder or render partial results prior to completion of that UI modification step.
Particle system effects are used to create a wide range of organic user interface effects such as sparkles, smoke, fire, star fields, and lava. The behavior and properties of the particles such as, direction, lifetime, number, velocity, randomness can be selected and controlled. All elements in the UI may be treated as particles. In addition, the particles can have a z-value (in addition to x- and y- values) that can be used with perspective computations to provide a three-dimensional look to the UI.
FIG. 6 shows a tree representation of a UI to which multiple applications are associated. The UI represented at FIG. 6 can have, for each of the multiple applications, a UI element or item, or several elements or items that can be rendered on the display 112 of the electronic device 100.
As in the example of FIG. 5, the tree representation of FIG. 6 is used to compose a scene to be rendered on the display by populating empty elements in the tree. As will be appreciated, conventional UI frameworks, where each application is responsible for its own UI, make it difficult to achieve a good UI, from the point of view consistency or visual appeal, when multiple applications interact with each other.
According to another aspect, and as shown in FIG. 14, the method can be implemented such that a single UI rendering engine 1402 can support multiple UI client engines 1404a, 1404b. Thus, multiple applications can coexist on the single UI rendering engine 1402. The UI client engines 1404a, 1404b are each associated with an application, or an instance of an application, while the UI rendering engine 1402 is associated with a display.
As shown in FIG. 15, the UI rendering engine 1402 receives the UI component trees from the UI client engines 1404a, 1404b (step 1502). The UI rendering engine 1402 then joins the plurality of UI component trees into a single tree structure (step 1504). To specify the parameters for joining the trees, the UI client engines 1404a, 1404b can, for example, define or indicate where in their trees other trees can be inserted. Subject to the logic implemented in the UI rendering engine 1402, the UI client engines 1404a, 1404b can indicate the location of possible tree insertions in a generic way, such as “here it is ok to insert a background effect”. The UI client engines 1404a, 1404b can also suggest, define or indicate where their tree should be inserted. This indication can also be performed in a quite general way, such as “I want to insert a particle effect in the background”. The UI rendering engine 1402 can then determine an appropriate location to insert the tree within the UI tree structure.
Different UI client engines 1404a, 1404b with different language bindings can coexist in same node/render tree, no matter what runtime limitations the language has (e.g. Python & threads). Since the individual UI component trees of the applications are combined to a single joint UI tree on the UI rendering engine 1402, the UI that is rendered by the server will, for end-users, appear as if all the applications UIs are part of the same application.
According to another aspect, the method can be implemented such that the single UI rendering engine 1402 can support multiple UI client engines 1404a, 1404b, and their associated applications, running on different devices or platforms, such as a local device and an application running on a remote device, such as in the cloud or on networked server. One example is internet TV, where several UI client engines share UI input and output with a common UI rendering engine. Another example is in the automotive context. Multiple applications, such as navigation, infotainment, etc., can use a common UI rendering engine to render their own UI elements to a display in the vehicle. As above, since the UI client engines for each application inject their trees and data items into the same tree on the UI rendering engine, all scene graph UI advantages apply. The UI rendering engine does not need to know anything about a new application, so, for example, the UI client engine for a new car radio application can be transparently injected into the common UI.
According to another aspect, and as shown in FIG. 16, the method can be implemented such that a multiple UI rendering engines 1602a, 1602b can support a single UI client engine 1604, and its associated application. For example, the single UI client engine 1604 can inject its tree, and provide data items to multiple devices, such as a desktop computer and a portable electronic device. Each device can have a separate UI rendering engines 1602a, 1602b, optimized for its particular form factor and display capabilities. Since the UI rendering engines 1602a, 1602b do their own rendering, it is possible to make a distributed UI that is responsive regardless of transport layer performance.
With reference to FIG. 17, the UI rendering engines 1602a, 1602b each receive the UI component tree from the client UI engine 1604 (step 1702), and individually determine a visual node tree for the UI component tree (step 1704). The separate UI rendering engines 1602a, 1602b asynchronously receive, from the UI client engine 1604, the UI data items related to elements of the UI component tree (step 1706), and populate the visual node tree with the UI data items (step 1708). Each UI rendering engine then renders the UI in accordance with the visual node tree independently of further input from the client UI engine (step 1710).
If a user input, such as a touch event or gesture, is detected by one of the UI rendering engines 1602a, 1602b, the input is communicated back to the UI client engine 1604, and to the other UI rendering engine. Both UI rendering engines can then re-render the UI if appropriate, while the UI client engine can provide the input to the application, or otherwise act upon it.
As a further example, the single UI client engine 1604 can use several UI rendering engines on a same device. For example, UI rendering engine 1602a could include an OpenGL renderer, while UI rendering engine 1602b could include a software rendering backend/rasterizer. The different UI rendering engines could, for example, be different versions of the rendering engine on the same device. For example, UI rendering engines 1602a, 1602b could be designed to render at different frame rates to serve different displays on a multi-display device. The UI rendering engines 1602a, 1602b could provide different power management capabilities. For example, using wallpaper as example, UI rendering engine 1602a could render wallpaper or background with less fidelity (lower resolution) to meet power management requirements.
It is also contemplated that the UI rendering engines 1602a, 1602b could form a dynamic cluster, distributing different UI elements of a client application between rendering engines to meet metrics like expected FPS, power management, and resource management. The UI rendering engines 1602a, 1602b can, for example, selectively render different elements or parts of the UI, as defined by the UI client engine 1604. The division of rendering tasks can be, for example, defined in an appropriate markup language, such as XML, or programmatically, such as in an API. Generally, the UI rendering engines 1602a, 1602b work independently to render their element(s) of the UI. However, in a standalone mode, the UI rendering engines 1602a, 1602b could exchange data to improve rendering efficiency.
<- Previous Patent (Code evaluation of f...) | Next Patent (***WITHDRAWN PATENT ...) ->