Patent Description:
<CIT> discloses systems and methods to prioritize browser tile generation from LQ tile content, <CIT> discloses predictive web page rendering using a scroll vector <CIT> discloses methods and systems for content processing.

Computing devices may be equipped with one or more high-performance graphics processing units (GPUs) providing high performance with regard to computations and graphics rendering. Computing devices may use a GPU to accelerate the rendering of graphics data for display. Examples of such computing devices may include a computer workstation, mobile phones (e.g., smartphones), embedded systems, personal computers, tablet computers, and video game consoles.

Rendering generally refers to the process of converting a three-dimensional (3D) graphics scene, which may include one or more 3D graphics objects, into two-dimensional (2D) rasterized image data. In particular, GPUs may include a 3D rendering pipeline to provide at least partial hardware acceleration for the rendering of a 3D graphics scene. The 3D graphics objects in the scene may be subdivided by a graphics application into one or more 3D graphics primitives (e.g., points, lines, triangles, patches, etc.), and the GPU may convert the 3D graphics primitives of the scene into 2D rasterized image data.

Systems and methods, the features of which are set out in the appended claims, are disclosed for displaying data on a display device using LQ tiles to reduce memory bandwidth. Users may fast scroll webpages with minimal degradation in the display quality or information content of webpages. The following summary sets out illustrative embodiments which are helpful in understanding the subject matter of the appended claims.

According to some embodiments, a method for displaying data on a display device includes computing a texture based on a difference between a high quality (HQ) tile and a corresponding low quality (LQ) tile. The method also includes storing the texture into an alpha channel of the LQ tile. The method further includes compositing the LQ tile onto the screen when an attribute of the alpha channel satisfies a threshold.

According to some embodiments, a system for displaying data on a display device includes a display device and a memory. The system also includes one or more processors coupled to the memory and display device. The one or more processors read the memory and are configured to compute a texture based on a difference between a HQ tile and a corresponding LQ tile. The processors are also configured to store the texture into an alpha channel of the LQ tile. The processors are further configured to composite the LQ tile onto the display device when an attribute of the alpha channel satisfies a threshold.

According to some embodiments, a computer-readable medium has stored thereon computer-executable instructions for performing operations including: computing a texture based on a difference between a HQ tile and a corresponding LQ tile; storing the texture into an alpha channel of the LQ tile; and compositing the LQ tile onto the display device when an attribute of the alpha channel satisfies a threshold.

According to some embodiments, an apparatus for displaying data on a display device includes means for computing a texture based on a difference between a HQ tile and a corresponding LQ tile. The system also includes means for storing the texture into an alpha channel of the LQ tile. The system further includes compositing the LQ tile onto the display device when an attribute of the alpha channel satisfies a threshold.

The accompanying drawings, which form a part of the specification, illustrate embodiments of the invention and together with the description, further serve to explain the principles of the embodiments. In the drawings, like reference numbers may indicate identical or functionally similar elements. The drawing in which an element first appears is generally indicated by the left-most digit in the corresponding reference number.

It is to be understood that the following disclosure provides many different embodiments, or examples, for implementing different features of the present disclosure. Some embodiments may be practiced without some or all of these specific details. Specific examples of components, modules, and arrangements are described below to simplify the present disclosure. These are, of course, merely examples and are not intended to be limiting.

Webpages are full of rich multimedia content that may include graphics, videos, images, text, etc. During webpage rendering, web browsers may partition webpages into tiles. The webpage content inside each tile may be rasterized into a bitmap that is then loaded into a texture for the GPU to access. Each bitmap may correspond to a tile that covers a portion of the screen. To display the webpage, the GPU composites the tiles onto the screen. As a user scrolls the webpage frame, new tiles may appear in the browser window and old tiles may disappear from the browser window.

The GPU may generate tiles having different resolutions. A low quality (LQ) tile is a lower resolution version of a corresponding high quality (HQ) tile. While HQ tiles are tiles that may have the same resolution as the screen, LQ tiles are scaled down versions of the information content overlapped by the LQ tiles. LQ tiles are relatively fast to render compared to fully rendered tiles, referred to as HQ tiles, and may be used for quickly conveying a thumbnail sketch of the webpage content overlapped by the LQ tiles.

During fast scrolling, not all of the HQ tiles of a frame may be rendered before a new frame appears in the browser window. To allow smooth scrolling of webpages in web browsers, a frame rate of about <NUM> frames per second (FPS) may be desirable. Unfortunately, this frame rate typically requires high memory bandwidth. If the user fast scrolls the webpage frame and HQ tiles of the webpage exposed on the screen have not been rendered yet, the user may see blank areas, which may be distracting and degrade overall user experience. Due to the high cost in rendering HQ tiles, corresponding LQ tiles may be generated and composited onto the screen such that a lower resolution version of the webpage can be displayed during scrolling, thus reducing the occurrence of blanking during scroll. The LQ tiles may be rendered into HQ tiles to fully display the information content.

For high resolution devices, a large amount of memory bandwidth may be required to display the entire webpage. Compositing a HQ tile onto the screen may consume a large amount of memory bandwidth and power as well as degrade performance compared to compositing a corresponding LQ tile. It may be desirable to reduce the memory bandwidth in order to improve performance and reduce power consumption. Conventional techniques that reduce the memory bandwidth include performing hardware texture compression. GPUs can perform hardware texture compression, but this technique may be undesirable because it requires hardware support and may be expensive. Alternatively, software techniques for texture compression can also be used. Software texture compression may be undesirable, however, because of the amount of central processing unit (CPU) processing required.

Techniques of the present disclosure provide solutions that overcome these disadvantages while enabling web browsers to quickly render frames of webpages with minimal degradation in display quality or information content during fast scrolling of the webpages. Systems and methods are disclosed for GPUs to composite either a HQ tile or its corresponding LQ tile onto a display device. A GPU composites the LQ tile rather than the corresponding HQ tile (without replacing the LQ tile with the HQ tile) if the LQ and HQ tiles are similar enough to not degrade the user's experience. LQ tiles are smaller and consume less memory space than their corresponding HQ tiles. By compositing the LQ tile rather than the HQ tile, the amount of memory accessed during composition by the GPU is reduced. Thus, using LQ tiles reduce the memory bandwidth required during tile composition.

In some embodiments, the GPU generates a HQ tile and a corresponding LQ tile and computes a texture based on a difference between the HQ tile and LQ tile. Each pixel in the LQ tile may have three color channels and an alpha channel. The alpha channel typically has an attribute describing the degree of opacity of an object fragment for a given pixel. Rather than store the degree of opacity, the GPU stores the texture into the alpha channel of the LQ tile. By doing so, memory space may be conserved. The texture includes a single scalar value per pixel in the LQ tile. The single scalar value corresponding to a pixel in the LQ tile is the difference between the pixel and a plurality of pixels in the corresponding HQ tile, and may be stored as the value of the attribute of the alpha channel.

The GPU composites the LQ tile onto the display device when an attribute of the alpha channel satisfies a threshold. In an example, an attribute that is below the threshold satisfies the threshold. Such an attribute may indicate that the LQ and HQ tiles are similar enough to each other such that compositing the LQ tile instead of the HQ tile onto the display device will not degrade the user's experience. Alternatively, the GPU composites the HQ tile onto the display device when the attribute does not satisfy the threshold. An attribute that is not below the threshold may indicate that the LQ and HQ tiles are not similar enough to each other to composite the LQ tile rather than the HQ tile. Accordingly, the HQ tile should be composited onto the display device instead of the corresponding LQ tile.

<FIG> is a block diagram illustrating a computing device102 that may be used to implement rendering techniques, according to some embodiments. Computing device <NUM> may include a personal computer, a desktop computer, a laptop computer, a computer workstation, a video game platform or console, a wireless communication device (e.g., a mobile telephone, a cellular telephone, a satellite telephone, and/or a mobile telephone handset), a handheld device such as a portable video game device or a personal digital assistant (PDA), a personal music player, a video player, a television, a television set-top box, a mainframe computer or any other type of device that processes and/or displays graphical data.

As illustrated in the example of <FIG>, computing device <NUM> includes a user interface <NUM>, a CPU <NUM>, a memory controller <NUM>, a system memory <NUM>, a graphics processing unit (GPU) <NUM>, a GPU cache <NUM>, a display interface <NUM>, a display device <NUM>, and bus <NUM>. User interface <NUM>, CPU <NUM>, memory controller <NUM>, GPU <NUM> and display interface <NUM> may communicate with each other using bus <NUM>. It should be noted that the specific configuration of buses and communication interfaces between the different components shown in <FIG> is merely an example, and other configurations of computing devices and/or other graphics processing systems with the same or different components may be used to implement the techniques of this disclosure.

CPU <NUM> may include a general-purpose or a special-purpose processor that controls operation of computing device <NUM>. A user may provide input to computing device <NUM> to cause CPU <NUM> to execute one or more software applications. The software applications that execute on CPU <NUM> may include, for example, an operating system, a software application <NUM> (e.g., a word processor application, an email application, a spread sheet application, a media player application, a video game application, a graphical user interface (GUI) application, or a browser), or another program. The user may provide input to computing device <NUM> via one or more input devices (not shown) such as a keyboard, a mouse, a microphone, a touch pad or another input device that is coupled to computing device <NUM> via user interface <NUM>.

Software application <NUM> may include one or more graphics rendering instructions that instruct GPU <NUM> to render graphics data to display device <NUM>. In some examples, the software instructions may conform to a graphics application programming interface (API), such as an Open Graphics Library (OpenGL. ) API, an Open Graphics Library Embedded Systems (OpenGL ES) API, a Direct3D API, an X3D API, a RenderMan API, a WebGL API, or any other public or proprietary standard graphics API. To process the graphics rendering instructions, CPU <NUM> may issue one or more graphics rendering commands to GPU <NUM> to cause it to render all or some of the graphics data. The graphics data to be rendered may include a list of graphics primitives, e.g., points, lines, triangles, quadrilaterals, triangle strips, etc..

Memory controller <NUM> facilitates the transfer of data going into and out of system memory <NUM>. For example, memory controller <NUM> may receive memory read and write commands, and service such commands with respect to memory system <NUM> in order to provide memory services for the components in computing device <NUM>. Memory controller <NUM> is communicatively coupled to system memory <NUM>. Although memory controller <NUM> is illustrated in the example computing device <NUM> of <FIG> as being a processing module that is separate from both CPU <NUM> and system memory <NUM>, in other examples, some or all of the functionality of memory controller <NUM> may be implemented on one or both of CPU <NUM> and system memory <NUM>,.

System memory <NUM> may store program modules and/or instructions that are accessible for execution by CPU <NUM> and/or data for use by the programs executing on CPU <NUM>. For example, system memory <NUM> may store user applications and graphics data associated with the applications. System memory <NUM> may additionally store information for use by and/or generated by other components of computing device <NUM>. For example, system memory <NUM> may act as a device memory for GPU <NUM> and may store data to be operated on by GPU <NUM> as well as data resulting from operations performed by GPU <NUM>. For example, system memory <NUM> may store any combination of texture buffers, depth buffers, stencil buffers, vertex buffers, frame buffers, or the like. In addition, system memory <NUM> may store command streams for processing by GPU <NUM>. System memory <NUM> may include one or more volatile or non-volatile memories or storage devices, such as, for example, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), Flash memory, a magnetic data media or an optical storage media.

GPU <NUM> may be configured to perform graphics operations to render one or more graphics primitives to display device <NUM> and to texture map an image to a pixel for display. When software application <NUM> executing on CPU <NUM> requires graphics processing, CPU <NUM> may provide graphics commands and graphics data to GPU <NUM> for rendering to display device <NUM>. The graphics commands may include draw call commands, GPU state programming commands, memory transfer commands, general-purpose computing commands, kernel execution commands, etc. In some examples, CPU <NUM> may provide the commands and graphics data to GPU <NUM> by writing the commands and graphics data to system memory <NUM>, which may be accessed by GPU <NUM>. In an example, graphics data may include a texture that is stored in system memory <NUM> and used by GPU <NUM> to determine the color for a pixel on display device <NUM>. In some examples, GPU <NUM> may be further configured to perform general-purpose computing for applications executing on CPU <NUM>.

GPU <NUM> may, in some instances, be built with a highly-parallel structure that provides more efficient processing of vector operations than CPU <NUM>. For example, GPU <NUM> may include a plurality of processing units that are configured to operate on multiple vertices, control points, pixels and/or other data in a parallel manner. The highly parallel nature of GPU <NUM> may, in some instances, allow GPU <NUM> to render graphics images (e.g., GUIs and two-dimensional (2D) and/or three-dimensional (3D) graphics scenes) onto display device <NUM> more quickly than rendering the images using CPU <NUM>. In addition, the highly parallel nature of GPU <NUM> may allow it to process certain types of vector and matrix operations for general-purposed computing applications more quickly than CPU <NUM>.

GPU <NUM> may, in some instances, be integrated into a motherboard of computing device <NUM>. In other instances, GPU <NUM> may be present on a graphics card that is installed in a port in the motherboard of computing device <NUM> or may be otherwise incorporated within a peripheral device configured to interoperate with computing device <NUM>. In further instances, GPU <NUM> may be located on the same microchip as CPU <NUM> forming a system on a chip (SoC). GPU <NUM> may include one or more processors, such as one or more microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), or other equivalent integrated or discrete logic circuitry.

GPU <NUM> may be directly coupled to GPU cache <NUM>. Thus, GPU <NUM> may read data from and write data to GPU cache <NUM> without necessarily using bus <NUM>. In other words, GPU <NUM> may process data locally using a local storage, instead of off-chip memory. This allows GPU <NUM> to operate in a more efficient manner by reducing the need of GPU <NUM> to read and write data via bus <NUM>, which may experience heavy bus traffic. GPU cache <NUM> may include one or more volatile or non-volatile memories or storage devices, such as, e.g., random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), etc. In some instances, however, GPU <NUM> may not include a separate cache, but instead use system memory <NUM> via bus <NUM>.

CPU <NUM> and/or GPU <NUM> may store rendered image data in a frame buffer that is allocated within system memory <NUM>. The software application that executes on CPU <NUM> may store the image data (e.g., texel colors, width, height, and color depth) in system memory <NUM> Display interface <NUM> may retrieve the data from the frame buffer and configure display device <NUM> to display the image represented by the rendered image data. In some examples, display interface <NUM> may include a digital-to-analog converter (DAC) that is configured to convert the digital values retrieved from the frame buffer into an analog signal consumable by display device <NUM>. In other examples, display interface <NUM> may pass the digital values directly to display device <NUM> for processing.

Display device <NUM> may include a monitor, a television, a projection device, a liquid crystal display (LCD), a plasma display panel, a light emitting diode (LED) array, a cathode ray tube (CRT) display, a surface-conduction electron-emitted display (SED), a laser television display, a nanocrystal display or another type of display unit. Display device <NUM> may be integrated within computing device <NUM>. For instance, display device <NUM> may be a screen of a mobile telephone handset or a tablet computer. Alternatively, display device <NUM> may be a stand-alone device coupled to computer device <NUM> via a wired or wireless communications link. For instance, display device <NUM> may be a computer monitor or flat panel display connected to a personal computer via a cable or wireless link.

Bus <NUM> may be implemented using any combination of bus structures and bus protocols including first, second, and third generation bus structures and protocols, shared bus structures and protocols, point-to-point bus structures and protocols, unidirectional bus structures and protocols, and bidirectional bus structures and protocols. Examples of different bus structures and protocols that may be used to implement bus <NUM> include, e.g., a HyperTransport bus, an InfiniBand bus, an Advanced Graphics Port bus, a Peripheral Component Interconnect (PCI) bus, a PCI Express bus, an Advanced Microcontroller Bus Architecture (AMBA), an Advanced High-performance Bus (AHB), an AMBA Advanced Peripheral Bus (APB), and an AMBA Advanced eXentisible Interface (AXI) bus. Other types of bus structures and protocols may also be used.

<FIG> is a block diagram illustrating CPU <NUM>, GPU <NUM>, and system memory <NUM> of computing device <NUM> in <FIG> in further detail, according to some embodiments. As shown in <FIG>, CPU <NUM> is communicatively coupled to GPU <NUM> and system memory <NUM>, and GPU <NUM> is communicatively coupled to CPU <NUM> and system memory <NUM>. GPU <NUM> may, in some examples, be integrated onto a motherboard with CPU <NUM>.

CPU <NUM> is configured to execute a software application such as a browser <NUM>, a graphics API <NUM>, a GPU driver <NUM>, and an operating system <NUM>. Browser <NUM> may include one or more instructions that cause graphics images to be displayed and/or one or more instructions that cause a non-graphics task (e.g., a general-purposed computing task) to be performed by GPU <NUM>. Browser <NUM> may include or implement a plurality of hardware components and/or software components that operate to perform various methodologies in accordance with the described embodiments.

A user may point browser <NUM> to a uniform resource locator (URL) of a webpage. Browser <NUM> may load a hypertext markup language (HTML) file referenced by the URL and render the webpage on the screen (e.g., display device <NUM> in <FIG>). Although browser <NUM> may be described as issuing instructions to GPU <NUM>, it should be understood that any software application executable in computing device <NUM> and that processes and/or displays graphical data may be used to issue the instructions. Additionally, although a webpage is described as being the renderable content, this is not intended to be limiting and the renderable/rendered content may be any text, image, or graphics on a page (that is or is not associated with a network such as the Internet).

<FIG> is an illustration of a webpage <NUM> rendered by browser <NUM>, according to some embodiments. <FIG> and <FIG> will be discussed together to better explain rendering techniques of the present disclosure. Browser <NUM> may display a frame of webpage <NUM> in a browser window.

During webpage rendering, browser <NUM> partitions webpage <NUM> into a plurality of tiles <NUM>. Webpage <NUM> is divided into tiles <NUM> of three columns and four rows for a total of <NUM> tiles. Tiles <NUM> may overlap with graphics, text <NUM>, images <NUM> and <NUM>, icons, links for videos etc. that convey information. Browser <NUM> rasterizes the webpage content inside each tile into a bitmap. Each bitmap corresponds to a tile that covers a portion of display device <NUM>.

Browser <NUM> may rasterize one or more versions of one or more tiles of plurality of tiles <NUM>. In an example, browser <NUM> rasterizes a LQ version and a HQ version of each tile. Browser <NUM> may rasterize a LQ version of tile <NUM> into an LQ bitmap <NUM>, and rasterize a corresponding HQ version of tile <NUM> into an HQ bitmap <NUM>. Browser <NUM> may rasterize the LQ version of the tile into smaller tiles than the HQ version of the tile. It may take CPU <NUM> less time to rasterize the content in the LQ version of tile <NUM> because it is smaller and contains less information than the corresponding HQ tile. As such, in some embodiments, browser <NUM> generates LQ bitmap <NUM> before HQ bitmap <NUM>. HQ bitmap <NUM> may be generated in the background while LQ bitmap <NUM> is being generated or after LQ bitmap <NUM> has been generated.

The bitmaps may be inaccessible to GPU <NUM>. To provide GPU <NUM> with access to the bitmaps, browser <NUM> may upload them into texture memory <NUM>. In an example, browser <NUM> uploads LQ bitmap <NUM> into an LQ tile <NUM>, and uploads HQ bitmap <NUM> into an HQ tile <NUM>. LQ tile <NUM> corresponds to HQ tile <NUM> and is a lower resolution version of HQ tile <NUM>. In an example, HQ tile <NUM> may map to a 512x512 texel region of display device <NUM>, and corresponding LQ tile <NUM> may map to a 32x32 texel region of display device <NUM>. A texel region includes one or more texels, and a texel is a pixel in texture memory <NUM>.

Due to the high cost in rasterizing and compositing HQ tiles, their corresponding LQ tiles may be used to render a lower resolution version of webpage <NUM> during scrolling, thus reducing the occurrence of blanking during scroll as well as the memory bandwidth. Although one LQ bitmap and HQ bitmap is illustrated in <FIG>, it should be understood that one or more LQ bitmaps and/or one or more HQ bitmaps may be stored in system memory <NUM>. Similarly, although one LQ tile and HQ tile is illustrated in <FIG>, it should be understood that one or more LQ tiles and/or one or more HQ tiles may be stored in system memory <NUM>. Additionally, a HQ tile may be described as having one corresponding lower resolution version. This is not intended to be limiting, and it should be understood that a HQ tile may have more than one corresponding lower resolution versions.

Browser <NUM> may issue instructions to graphics API <NUM>, which may translate the instructions received from browser <NUM> into a format that is consumable by GPU driver <NUM>. GPU driver <NUM> receives the instructions from browser <NUM>, via graphics API <NUM>, and controls the operation of GPU <NUM> to service the instructions. For example, GPU driver <NUM> may formulate one or more commands <NUM>, place the commands <NUM> into system memory <NUM> (e.g., in texture memory <NUM>), and instruct GPU <NUM> to execute commands <NUM>. In some examples, GPU driver <NUM> may place commands <NUM> into system memory <NUM> and communicate with GPU <NUM> via operating system <NUM>, e.g., via one or more system calls.

System memory <NUM> may store one or more commands <NUM>. Commands <NUM> may be stored in one or more command buffers (e.g., a ring buffer) and include one or more state commands and/or one or more draw call commands. A state command may instruct GPU <NUM> to change one or more of the state variables in GPU <NUM>, such as the draw color. A draw call command may instruct GPU <NUM> to render a geometry defined by a group of one or more vertices (e.g., defined in a vertex buffer) stored in system memory <NUM> or to draw content of a texture (e.g., LQ tile <NUM> or HQ tile <NUM>) onto display device <NUM>.

GPU <NUM> includes a command engine <NUM> and one or more processing units <NUM>. Command engine <NUM> retrieves and executes commands <NUM> stored in system memory <NUM>. In response to receiving a state command, command engine <NUM> may be configured to set one or more state registers in GPU <NUM> to particular values based on the state command. In response to receiving a draw call command, command engine <NUM> may be configured to cause processing units <NUM> to render the geometry represented by vertices based on primitive type data stored in system memory <NUM>. Command engine <NUM> may also receive shader program binding commands, and load particular shader programs into one or more of the programmable processing units <NUM> based on the shader program binding commands.

Processing units <NUM> may include one or more processing units, each of which may be a programmable processing unit or a fixed-function processing unit. A programmable processing unit may include, for example, a programmable shader unit that is configured to execute one or more shader programs downloaded onto GPU <NUM> from CPU <NUM>. A shader program, in some examples, may be a compiled version of a program written in a high-level shading language, such as an OpenGL Shading Language (GLSL), a High Level Shading Language (HLSL), a C for Graphics (Cg) shading language, etc..

In some examples, a programmable shader unit may include a plurality of processing units that are configured to operate in parallel, e.g., an SIMD pipeline. A programmable shader unit may have a program memory that stores shader program instructions and an execution state register, e.g., a program counter register that indicates the current instruction in the program memory being executed or the next instruction to be fetched. The programmable shader units in processing units <NUM> may include, for example, vertex shader units, pixel shader units, geometry shader units, hull shader units, domain shader units, compute shader units, and/or unified shader units. The one or more processing units <NUM> may form a 3D graphics rendering pipeline, which may include one or more shader units that are configured to execute a shader program. Browser <NUM> may send different shader programs to GPU <NUM>.

In an example, commands <NUM> include a command to render webpage <NUM>. Processing units <NUM> includes a fragment shader or pixel shader <NUM> that may during the composition stage of the rendering process, composite at most one of LQ tile <NUM> and HQ tile <NUM> onto display <NUM>. Pixel shader <NUM> may also compute and set colors for pixels covered by a texture object (e.g., texture image) displayed on display device <NUM>. The terms "fragment" and "pixel" may be used interchangeably in the disclosure.

Each pixel of display device <NUM> may have associated information. In some examples, each pixel has three color channels and an alpha channel. A color channel is a function of a specific component of that pixel, which is typically a red, green, and blue (RGB) component. Accordingly, a pixel may have a red channel, green channel, blue channel, and alpha channel. The combination of these three colors at different intensities may represent a full range of the visible spectrum for each pixel. Additionally, the alpha channel may have an attribute indicating the degree of opacity of each pixel. When the attribute is examined in a compositing program, an attribute value of one (white) represents <NUM> percent opaqueness and entirely covers the pixel's area of interest. In contrast, an attribute value of zero (black) represents <NUM> percent transparency.

In some embodiments, during the composition stage of the rendering process, GPU <NUM> may composite either LQ tile <NUM> or HQ tile <NUM> onto display device <NUM>. LQ tiles are smaller and consume less memory space than their corresponding HQ tiles. LQ tile <NUM> may include content (e.g., graphics, text, images, icons, links for videos etc.) having a lower resolution than HQ tile <NUM>. Accordingly, it may be quicker for GPU <NUM> to composite LQ tile <NUM> onto display device <NUM> instead of HQ tile <NUM> because LQ tile <NUM> contains less information than HQ tile <NUM>.

As the user scrolls the webpage frame, new tiles may appear in the browser window and old tiles may disappear from the browser window. During fast scrolling, not all of the HQ tiles of a frame may be available. LQ tile <NUM> may be available before HQ tile <NUM> is available, for example, because it may be quicker to generate LQ tile <NUM> compared to HQ tile <NUM>. Here, LQ tile <NUM> may be composited onto display device <NUM> to avoid the user seeing blank areas where the unavailable HQ tile would be displayed. Blank areas may be distracting and degrade overall user experience.

Alternatively, if both LQ tile <NUM> and HQ tile <NUM> are available, it may be desirable to composite LQ tile <NUM> onto display device <NUM> rather than HQ tile <NUM> (without replacing LQ tile <NUM> with HQ tile <NUM>) if the LQ and HQ tiles are similar enough to each other such that it is unnoticeable or not distracting to the user to see the LQ tile. GPU <NUM> may determine whether to composite HQ tile <NUM> or LQ tile <NUM> onto display device <NUM>. <FIG> is a flowchart <NUM> of a process for GPU <NUM> to composite HQ tile <NUM> or LQ tile <NUM> onto display device <NUM>, according to some embodiments, Method <NUM> is not meant to be limiting and may be used in other applications.

In an action <NUM>, a texture is computed based on a difference between HQ tile <NUM> and corresponding LQ tile <NUM>. In an example, browser <NUM> sends instructions to GPU <NUM> to compute the texture DLOW via graphics API <NUM>. GPU <NUM> compares the difference between two images (e.g., corresponding to HQ tile <NUM> and LQ tile <NUM>) having different resolutions. In an example, pixel shader <NUM> determines the degree of similarity between HQ tile <NUM> and LQ tile <NUM> by computing a texture DLOW based on a difference between the tiles.

A pixel in LQ tile <NUM> may be referred to as an LQ pixel, and a pixel in HQ tile <NUM> may be referred to as an HQ pixel. The HQ and LQ tiles have a different number of pixels. Each LQ pixel in LQ tile <NUM> may be mapped to a plurality of HQ pixels in HQ tile <NUM>. In an example, LQ tile <NUM> is a 32x32 pixel region that maps to a 512x512 pixel region in HQ tile <NUM>. For each LQ pixel in LQ tile <NUM>, the texture DLOW may include a difference value indicating the difference between the LQ pixel and its mapped plurality of HQ pixels in HQ tile <NUM>. The texture DLOW has the same resolution as LQ tile <NUM>. Each pixel in the texture DLOW may be associated with a single scalar value representing a difference between an LQ pixel and its mapped plurality of HQ pixels. GPU <NUM> may compute the texture DLOW efficiently because GPU <NUM> can process the pixels in parallel. In an example, each instance of pixel shader <NUM> may process one pixel of the browser window.

GPU <NUM> may calculate the texture DLOW in a variety of ways. In some examples, GPU. <NUM> calculates the texture DLOW in one pass. In an example, for each LQ pixel in LQ tile <NUM>, GPU <NUM> identifies a corresponding pixel region in HQ tile <NUM> (e.g., 16x16 HQ pixels). GPU <NUM> may compute an attribute of the pixel region. The attribute of the pixel region is the average of the pixel region. The intensity of the pixel values in the pixel region are averaged out. Each pixel may include an RGB color. For example, each pixel may include three scalar values, where a first scalar value corresponds to the red ("R") value, a second scalar value corresponds to the green ("G") value, and a third scalar value corresponds to the blue "B" value. The pixel intensity may be a function of the three color values. In an example, the intensity is a linear combination of the red, green, and blue values. GPU <NUM> computes a difference between the attribute of the pixel region and the LQ pixel. The difference between the attribute of the pixel region and the LQ pixel is a value in the texture DLOW. The LQ pixel corresponds to a pixel in the texture DLOW storing the difference between the attribute of the pixel region and the LQ pixel. This difference is computed for each LQ pixel in LQ tile <NUM>, and the texture DLOW includes each of these computed differences.

In another example, not falling within the scope of the claims, for each LQ pixel in LQ tile <NUM>, GPU <NUM> identifies a corresponding pixel region in HQ tile <NUM> and computes a difference between the LQ pixel and each pixel in the pixel region. GPU <NUM> may then compute an average of the one or more computed differences. The average may be a value in the texture DLOW. In an example, the LQ pixel corresponds to a pixel in the texture DLOW storing the average. This average may be computed for each LQ pixel in LQ tile <NUM>, and the texture DLOW may include each of these computed averages.

In some examples not falling within the scope of the claims, GPU <NUM> calculates the texture DLOW in more than one pass. In an example, for each LQ pixel in LQ tile <NUM>, GPU <NUM> identifies a corresponding pixel region in HQ tile <NUM> and computes a texture DHIGH based on a difference between the LQ pixel and the pixel region. The resolution of the texture DHIGH may be the same as HQ tile <NUM>'s resolution. Each pixel in the texture DHIGH may be associated with a single scalar value representing a difference in pixel intensity between HQ tile <NUM> and LQ tile <NUM>. In a separate pass, GPU <NUM> may down-sample the texture DHIGH to the resolution of LQ tile <NUM>. The texture DLOW may be the texture DHIGH down-sampled to the resolution of LQ tile <NUM>. In an example, the LQ pixel corresponds to a pixel in the texture DLOW storing the down-sampled difference between the LQ pixel and the pixel region. The texture DHIGH may be stored in one or more temporary buffers and may be discarded after the texture DLOW is computed. This down-sampled difference may be computed for each LQ pixel in LQ tile <NUM>, and the texture DLOW may include each of these down-sampled differences.

In an action <NUM>, the texture is stored into an alpha channel of the LQ tile. In an example, browser <NUM> sends instructions to GPU <NUM> to store the texture DLOW into an alpha channel of LQ tile <NUM>. Accordingly, GPU <NUM> may store the texture DLOW into an alpha channel of LQ tile <NUM>. Content of the texture DLOW, which stores a single scalar value per pixel, is written to the alpha channel of LQ tile <NUM>. In particular, each LQ pixel in LQ tile <NUM> may have an alpha channel having an attribute that describes the degree of opacity of an object fragment for the LQ pixel. The attribute of the alpha channel typically has a value of one, indicating that the tile is opaque. Because tiles are typically opaque, the attribute of the alpha channel may be used to store information different from the opaqueness of the tile. For example, the attribute may indicate a similarity (or difference) between LQ tile <NUM> and HQ tile <NUM> to determine whether compositing LQ tile <NUM> is sufficient without compositing HQ tile <NUM> onto display device <NUM>. The alpha channel is used to store the texture DLOW to save memory space.

Each value in the texture DLOW is stored in an attribute of an alpha channel of an LQ pixel in LQ tile <NUM>. The attribute is based on a difference between the LQ pixel and its mapped pixel region in HQ tile <NUM>. A difference value provides an indication of whether to composite HQ tile <NUM> or LQ tile <NUM> onto display device <NUM>. In an example, the difference value may be compared to a threshold to determine how similar LQ tile <NUM> is to HQ tile <NUM>. The threshold may depend on various factors such as the resolution of the screen and webpage. For example, the threshold may indicate a particular percentage difference in pixel values (e.g., <NUM> percent difference in pixel intensity).

In an action <NUM>, the LQ tile is composited onto the display device when an attribute of the alpha channel satisfies a threshold. Accordingly, color values from texels in LQ tile <NUM> may be used. In an example, browser <NUM> sends instructions to GPU <NUM> to read from the attribute of the alpha channel and compare the attribute to the threshold via graphics API <NUM>. GPU <NUM> may composite LQ tile <NUM> onto display device <NUM> when an attribute of the alpha channel satisfies a threshold. The attribute of the alpha channel may satisfy the threshold if the attribute is less than (or equal to) the threshold, which may indicate how similar LQ tile <NUM> is to HQ tile <NUM>. An attribute that is below the threshold may indicate that LQ tile <NUM> is similar enough to HQ tile <NUM> to not degrade the user's experience. Accordingly, LQ tile <NUM> may be composited onto display device <NUM> instead of HQ tile <NUM>.

A webpage may contain low frequency data (e.g., blanks, constant color, or slow changing gradient or images) such that if LQ tile <NUM> is composited onto display device <NUM> rather than HQ tile <NUM>, the end result is good enough for a user to view without degrading the user's experience. LQ tile <NUM> may be similar to HQ tile <NUM> if the HQ tile contains low frequency data.

GPU <NUM> may cache LQ tile <NUM> in GPU cache <NUM> for later retrieval. If GPU <NUM> accesses LQ tile <NUM>, GPU <NUM> may retrieve LQ tile <NUM> from GPU cache <NUM> rather than accessing texture memory <NUM>. Many of the browser window's pixels may (inversely) map to the same LQ tile. Accordingly, instances of pixel shader <NUM> may fetch the same LQ tile, which may be more likely to be in GPU <NUM>'s texture cache. This may result in lowered memory bandwidth for tiles containing pixels having similar color values between HQ bitmap <NUM> and LQ bitmap <NUM>, as in the case where the web page contains blank areas, slow changing gradients, etc..

Additionally, it may be unnecessary for GPU <NUM> to access HQ tile <NUM> at all. Rather, GPU <NUM> may read the alpha channel of LQ tile <NUM> to determine whether the difference between LQ tile <NUM> and HQ tile <NUM> is so small that GPU <NUM> can composite LQ tile <NUM>. If the attribute of the alpha channel satisfies the threshold, GPU <NUM> is saved from accessing HQ tile <NUM>, thus reducing memory bandwidth.

In contrast, in an action <NUM> the HQ tile is composited onto the display device when the attribute of the alpha channel does not satisfy the threshold. Accordingly, color values from texels in HQ tile <NUM> may be used. GPU <NUM> may composite HQ tile <NUM> onto display device <NUM> when the attribute of the alpha channel does not satisfy the threshold. In an example, the attribute does not satisfy the threshold if the attribute is not less than the threshold (e.g., greater than or equal to the threshold). An attribute that is not below the threshold may indicate that LQ tile <NUM> is not similar enough to HQ tile <NUM> to be displayed instead of the HQ tile. Accordingly, HQ tile <NUM> may be composited onto display device <NUM> instead of LQ tile <NUM>. A webpage may contain high frequency data such that if LQ tile <NUM> is composited onto display device <NUM> rather than HQ tile <NUM>, the end result is distracting for a user to view.

GPU <NUM> may composite HQ tile <NUM> onto display device <NUM> in a variety of ways. In an example, GPU <NUM> accesses the high-resolution tile and copies texels from HQ tile <NUM> into a frame buffer for display on display device <NUM>. In another example, LQ tile <NUM> may already be composited onto display device <NUM>. In this example, GPU <NUM> may add back the difference between HQ tile <NUM> and LQ tile <NUM> to obtain HQ tile <NUM>.

In some embodiments, actions <NUM>-<NUM> may be performed for any number of LQ tiles. It is also understood that additional actions may be performed before, during, or after actions <NUM>-<NUM> discussed above. It is also understood that one or more of the actions of method <NUM> described herein may be omitted, combined, or performed in a different sequence as desired.

Using techniques disclosed in the present disclosure, the number of texture fetches is not reduced, but may actually increase. For example, if the attribute of the alpha channel does not satisfy the threshold, GPU <NUM> accesses both LQ tile <NUM> and HQ tile <NUM>. Despite this, embodiments of the disclosure may improve performance significantly by reducing the memory bandwidth if GPU <NUM> uses texels from LQ tiles often.

In some embodiments, not falling within the scope of the claims, a processing unit (e.g., GPU <NUM> and/or CPU <NUM>) varies the threshold on-the-fly to reduce memory bandwidth. The processing unit may decrease the threshold so that LQ tiles may be used more often. In an example, during fast scroll, the user is less likely to notice the difference between LQ and HQ tiles. Accordingly, if the processing unit detects a fast scroll, the processing unit may decrease the threshold. In another example, computing device <NUM> may be in a low-battery mode (e.g., less than <NUM> percent battery left), and it may be desirable to reduce power consumption. Accordingly, if the processing unit detects that the computing device is in the low-battery mode, the processing unit may decrease the threshold.

As discussed above and further emphasized here, <FIG> are merely examples, which should not unduly limit the scope of the claims. In various embodiments of the present disclosure, execution of instruction sequences (e.g., actions <NUM>-<NUM> in <FIG>) to practice the present disclosure may be performed by computing device <NUM>. In various other embodiments of the present disclosure, a plurality of computing devices may be coupled by a communications link to a network (e.g., such as a local area network (LAN), wireless local area network (WLAN), public switched telephone network (PTSN), and/or various other wired or wireless networks, including telecommunications, mobile, and cellular phone networks) may perform instruction sequences to practice the present disclosure in coordination with one another.

In an example, instructions for compositing a LQ tile or a HQ tile may be stored in a computer readable medium of system memory <NUM><NUM>. Processors may execute the instructions to compute a texture DLow based on a difference between a HQ tile and a corresponding LQ tile and to store the texture DLow into an alpha channel of the LQ tile. Processors may also execute the instructions to composite the LQ tile onto display device <NUM> when an attribute of the alpha channel satisfies a threshold. Processors may also execute the instructions to composite the HQ tile onto display device <NUM> when the attribute of the alpha channel does not satisfy the threshold.

Where applicable, various embodiments provided by the present disclosure may be implemented using hardware, software, firmware, or combinations thereof. Also where applicable, the various hardware components, software components, and/or firmware components set forth herein may be combined into composite components including software, firmware, hardware, and/or all without departing from the scope of the appended claims. Where applicable, the various hardware components, software components, and/or firmware components set forth herein may be separated into sub-components including software, firmware, hardware, or all without departing from the scope of the appended claims. In addition, where applicable, it is contemplated that software components may be implemented as hardware components, and vice-versa. Where applicable, the ordering of various steps or actions described herein may be changed, combined into composite steps or actions, and/or separated into sub-steps or sub-actions to provide features described herein.

Claim 1:
A method (<NUM>) of displaying a webpage on a display device, the webpage partitioned into a plurality of tiles, the method comprising:
rasterising a low quality, LQ, version of each tile of the plurality of tiles into a low quality bitmap:
rasterising a high quality, HQ, version of each tile of the plurality of tiles into a high quality bitmap,
wherein each pixel of the low quality version of a tile maps to a pixel region in the high quality version of the tile; and for at least one of the plurality of tiles:
computing (<NUM>) a texture by computing, for each pixel in the LQ version of the tile, a difference between an intensity of the pixel and an average intensity of all pixels in the pixel region of the high quality version of the tile;
storing (<NUM>) the computed texture into an alpha channel of the tile, wherein each computed difference is stored in an alpha channel of a LQ pixel in the LQ version of the tile; and
during the composition stage of a webpage rendering process:
using colour values from texels in the LQ version of the tile for display on the display device when the stored texture satisfies a threshold; and
using colour values from texels in the HQ version of the tile for display on the display device when the stored texture does not satisfy the threshold.