Tiled viewport composition

A system that buffers an application image reduces bandwidth requirements for accessing memory. The application image may be logically separated into tiles. A viewport may identify a visible portion of the application image, where the visible portion is smaller than the application image. The tiles overlapped by the viewport may be buffered in a front buffer and a back buffer. The tiles not overlapped by the viewport may be buffered in the back buffer but not in the front buffer. A composition manager, with knowledge of the viewport and at least two noncontiguous tile buffers in the front buffer, may extract the visible portion of the application image directly from the noncontiguous tile buffers.

BACKGROUND

1. Technical Field

This disclosure relates to graphics and, in particular, to graphics buffers.

2. Related Art

Graphical user interfaces (GUI), Picture in Picture (PiP), and other types of graphical images may be composited from multiple independently rendered images. For example, a GUI may contain a background image and one or more rendered graphic layers on top of the background image. The graphic layers may comprise applications, task bars, or other type of graphic elements. Each of the graphic layers may be rendered in an application buffer independently of the other graphic layers by individual software applications. The graphic layers may be composited together into a single composite image in a display buffer, which may be presented on a display device, transmitted over a network, or otherwise processed. The application buffer may represent an image that is larger than the corresponding graphic layer composited into the single composite image.

DETAILED DESCRIPTION

A system for buffering an application image may include a composition manager and a memory comprising a front buffer and a back buffer. The application image may be logically divided or separated into tiles. Each of the tiles may identify a corresponding area of the application image. The composition manager may determine which of the tiles are overlapped by a viewport. The viewport may identify a visible portion of the application image. The visible portion may be smaller than the application image. For example, the visible portion may be a portion of a map that is displayed on a display device in a navigation system.

In some systems, the composition manager (or an application that renders the application image) may double buffer (or more generally, multiple buffer) the tiles that the viewport overlaps. The composition manager (or the application) may not double buffer (or multiple buffer) the tiles that the viewport does not overlap. Reducing the amount of memory that is multiple buffered may reduce bandwidth requirements for accessing the memory.

Double buffering may prevent visual tearing artifacts caused when one process writes to a buffer while another process reads from the same buffer. To decrease or avoid the possibility of causing such visual defects, the application may render the application image in the back buffer while another process, such as the composition manager, reads the application image from the front buffer. More generally, a buffer that the system writes to in order to construct a frame of the application image may be the back buffer. A buffer that the system reads the completed frame from may be the front buffer. In examples in which multiple buffering is used, the system may include and use multiple back buffers.

In response to a vertical synchronization pulse or some other event, the front buffer and the back buffer may be switched such that the back buffer becomes the front buffer, and the front buffer becomes the back buffer. The buffer switch may also be referred to as a buffer flip. After the buffer flip, the back buffer may include the contents of frame n, and the front buffer may include the contents of frame n+1. If the entire contents of the front buffer (frame n+1) is copied to the back buffer (frame n), then the back buffer may have the most recent contents (frame n+1), and the application may begin rendering a new frame, n+2 in the back buffer. The process of rendering, flipping, and displaying may be repeated.

However, in the system for buffering the application image, copying the entire contents of the application image from the front buffer to the back buffer may be avoided when the buffer flip occurs. Instead, the composition manager and/or the application may skip double buffering (or multiple buffering) of the tiles that the viewport does not overlap. In particular, the composition manager and/or the application may buffer the tiles that are overlapped by the viewport in both the front and back buffers, and buffer the tiles that are not overlapped by the viewport in the back buffer but not in the front buffer.

Each of the tiles may be buffered in a corresponding tile buffer. The tile buffers may be noncontiguously arranged in memory. Two buffers may be noncontiguous if the buffers are not adjacent to each other in the memory address space of the memory that the buffers are in.

In some systems, the composition manager may copy a buffered visible portion of the application image directly from the noncontiguous tile buffers based on the composition manager having knowledge of the viewport and the tile buffers. The application does not need to create an intermediate buffer that contains the buffered visible portion of application image in contiguous memory so that the composition manager may copy the visible portion from the contiguous memory. Avoiding the creation of the intermediate buffer reduces the bandwidth requirements for accessing memory. The composition manager may generate a composite image from the visible portion of the application image and one or more additional application images.

FIG. 1illustrates a tiled viewport composition system100. The system100may include an application102, application buffers104, a display buffer106, and a composition manager108. The tiled viewport composition system100is a system for buffering an application image110.

The application102may render the application image110. The application image110may include a window generated by a windows based operating system, such an operating system for a mobile electronic device, a desktop computer, or a server. Examples of the windows based operating system may include MICROSOFT WINDOWS®, which is a registered trademark of Microsoft Corporation of Redmond, Wash.; LINUX®, which is a registered trademark of Linus Torvalds of Finland; and ANDROID™, which is a trademark of Google, Inc. of Mountain View, Calif. Alternatively or in addition, the application image110may include a Graphical User Interface (GUI), a Picture in Picture (PiP), a task bar, a background picture, a frame in a sequence of frames that may be part of a video, or any other type of digital image. Examples of the application102may include a word processing program, a mail program, a web browser, a mobile application, or other types of software program or hardware component that may render the application image110.

The application buffers104may be one or more areas of memory that represent the application image110and/or portions thereof. The memory may include contiguous and/or noncontiguous areas of memory.

The display buffer106may comprise an area of memory that represents a composite image112assembled from individual images generated by one or more applications, such as the application102illustrated inFIG. 1. The composite image112may be presented on a display device, transmitted over a network, or locally or remotely processed.

The composition manager108may composite or direct the composition of the individual images into the composite image112. The individual images may include all or a portion of application images. Examples of the composition manager108include a hardware compositor, a window manager, or any other component that composites individual images into the composite image112.

During operation of the tiled viewport composition system100, the composition manager108may render the composite image112from a visible portion of the application image110identified by a viewport114. The visible portion identified by the viewport114may be smaller than the entire application image110. The composition manager108may render the composite image112from the visible portion of the application image110and from one or more additional images.

The application102may move the viewport114around the application image110as the visible portion of the application image110changes over time. In one example, the application image110may be a map image, and the viewport114identifies the portion of the map image that is to be displayed in a navigation system. As the location of the navigation system changes, the portion of the map image displayed in the navigation system may also change. In a second example, the application image110may be an image of a level in a video game, and the viewport114may identify the portion of an image of the level that is to be displayed on a screen. As a game character moves around the level, the portion of the image of the level displayed on the screen changes. In a third example, a webpage viewer may render more content than may be shown on the display. The composite image112may update more quickly, or produce a visibly smoother transition appearance, when the application102uses the viewport114to update the visible portion of the application image110.

The application102may logically divide or partition the application image110into tiles116. Each of the tiles116may identify a corresponding area of the application image110. The tiles116may be uniform in size and shape throughout the application image110. Alternatively, the size and the shape of the tiles116may vary throughout the application image110. The size and shape of the tiles116may be different in different examples.

The tiles116may be represented in corresponding tile buffers118. For example, the tiles116that are individually designated T1-T4inFIG. 1may be represented in the tile buffers118that are individually designated A1-A4. The tile buffers118may be included in the application buffers104.

The composition manager108may determine which of the tiles116are overlapped by the viewport114. Each of the tiles116that are overlapped or intersected by the viewport114may contain at least part120of the visible portion of the application image110. In contrast, the tiles116that are not overlapped by the viewport114may not contain any visible part120of the application image110.

The composition manager108may double buffer the tiles116that are overlapped by the viewport114, but not the remaining tiles116. A front buffer122may include the tile buffers118(A3and A4) representing the tiles116, (T3and T4) overlapped by the viewport114. Accordingly, the tile buffers118(A3and A4) in the front buffer122include a buffered visible portion124of the application image110. Conversely, a back buffer126may include the tile buffers118(A1and A2) representing the tiles116(T1and T2) not overlapped by the viewport114. In addition, the back buffer126may include the tile buffers118(B3and B4) that correspond to the tile buffers118(A3and A4) that are in the front buffer122and that represent the tiles116overlapped by the viewport114.

When generating the composite image112, the composition manager108may extract the buffered visible portion124of the application image110from the tile buffers118(A3and A4) that are in the front buffer122. The buffered visible portion124may be of frame n of the application image110. The application102may render frame n+1 of the application image110in the back buffer126by modifying the tile buffers118(A1, A2, B3, and B4) that are in the back buffer126.

When a buffer flip occurs, the composition manager108may copy the contents of a subset of the tile buffers118that are in the back buffer126to the front buffer122. The subset of the tile buffers118may be the tile buffers118(B3and B4) that correspond to the tile buffers118(A3and A4) that are in the front buffer122. The composition manager108may skip copying tile buffers118(A1and A2) representing the tiles116(T1and T2) not overlapped by the viewport114to the front buffer122.

Alternatively or in addition to copying the contents of the subset of the tile buffers118from the back buffer126to the front buffer122when the buffer flip occurs, the composition manager108may change an association between the tile buffers118and the front and back buffers122and126. The composition manager108may track, in a data structure, which of the tile buffers118are associated with the front buffer122, and which of the tile buffers118are associated with the back buffer126. The associations between the tile buffers118and the front and back buffers122and126indicate which of the tile buffers118are included in each of the front and back buffers122and126. When the buffer flip occurs, the composition manager108may disassociate the subset of the tile buffers118(B3and B4) from the back buffer126, and associate the subset of the tile buffers118(B3and B4) with the front buffer122. Correspondingly, the composition manager108may change the tile buffers118(A3and A4) that were originally associated with the front buffer122to be associated with the back buffer126. In other words, the composition manager108, in response to the buffer flip, may swap a first set of the tile buffers118associated with the front buffer122with a second set of the tile buffers118associated with the back buffer126, where the tile buffers118that are swapped represent the tiles116that the viewport114overlaps but not the tiles116that the viewport114does not overlap.

When the composition manager108swaps the tile buffers118that are associated with the front buffer122with the corresponding tile buffers118that are associated with the back buffer126, then the tile buffers118swapped into the back buffer126may represent an older frame, such as frame n. The tile buffers118swapped into the front buffer126may represent frame n+1. After the buffer swap, the application102may render a third frame, such as frame n+2, in the back buffer126.

When the viewport114moves and overlaps a new one of the tiles116, the composition manager108may double buffer the newly overlapped tile116. Conversely, when the viewport114moves and no longer overlaps one of the tiles116, then the composition manager108may stop double buffering the previously overlapped tile116.

Accordingly, when the viewport114moves, the composition manager108may adjust which of the tile buffers118are in the front buffer122so that the tile buffers118in the front buffer122correspond to the tiles116overlapped by the viewport114. Similarly, when the viewport114moves, the composition manager108may adjust which of the tile buffers118are in the back buffer126.

When the viewport114moves and overlaps a newly overlapped tile116, then the tile buffer118in the back buffer126that corresponds to the newly overlapped tile116may be included in the front buffer122. The tile buffer118added to the front buffer does not need to be completely re-rendered because the application102may have already rendered the tile buffer118in the back buffer126.

In some systems, the tile buffers118may be dynamically allocated and freed as the viewport114moves. For example, if the viewport114moves and overlaps four of the tiles116(T1-T4) illustrated inFIG. 1instead of two of the tiles116(T3and T4), then the composition manager108may dynamically allocate two tile buffers118(B1and B2, which are not shown inFIG. 1). The front buffer122may then include four of the tile buffers118(A1-A4), and the back buffer126may include the four corresponding tile buffers118(B1-B4).

As an alternative to dynamically allocating the tile buffers118, the tile buffers118may be allocated in advance. For example, upon an initialization requested by the application102, the composition manager108may allocate enough of the tile buffers118to fully represent the application image110in both the back buffer126and the front buffer122. The composition manager108may dynamically associate the pre-allocated tile buffers118to the front and back buffers122and126.

The application102and the composition manager108may have different features than those described. In one system, the composition manager108(instead of the application102) may divide or partition the application image110into tiles116. The application102may invoke an API (Application Programming Interface) to indicate to the composition manager108the size of the application image110. In response, the composition manager108may divide the application image110into the tiles116and allocate the corresponding tile buffers118. In another system, the application102(instead of the composition manager108) may double buffer the tiles116that are overlapped by the viewport114, but not the remaining tiles116. In still another system, one or more components other than the application102and the composition manager108may execute described functions of the application102and the composition manager108.

The composition manager108may render the composite image112from the visible portion of the application image110identified by the viewport114. The composition manager108, having knowledge of the tile buffers118and the viewport114, may copy the buffered visible portion124directly from the tile buffers118in the front buffer122to the display buffer106even if the buffered visible portion124is spread across multiple noncontiguous tile buffers118. Copying the buffered visible portion124from the tile buffers118in the front buffer to an intermediate buffer, and then copying from the intermediate buffer to the display buffer106may be unnecessary. The application102, for example, may provide the composition manager108with information about the viewport114and about the tile buffers118in the front buffer122. The composition manager108may use the information provided by the application102to copy the buffered visible portion124from the tile buffers118in the front buffer122to the display buffer106. Alternatively, the composition manager108may render the composite image112from an intermediate buffer. The intermediate buffer may be populated by the application102by copying from the buffered visible portion124in the front buffer122.

The frame of the application image110represented in the front buffer122may be frame n or any other frame. The frame of the application image110represented in the back buffer126may be frame n+1 or any other frame.

As described, the composition manager108, the application102, and/or some other component may double buffer the tiles116that are overlapped or lie or extend and cover part of by the viewport114. The composition manager108and/or the application102may apply multiple buffering to the tiles116that are overlapped by the viewport114, and not to the tiles116that are not overlapped by the viewport114. Multiple buffering may include double buffering, triple buffering, or other buffering processes and/or hardware that includes storing one or more frames of the application image110in one or more front buffers for extraction of the application image110, and storing one or more frames of the application image110in one or more back buffers for rendering of the application image110.

FIG. 2is a hardware diagram of the tiled viewport composition system100. The tiled viewport composition system100may include a processor204, a memory interface206, and a memory208. The tiled viewport composition system100may be referred to simply as the system100.

The processor204may be hardware that executes computer executable instructions or computer code embodied in the memory208or in other memory to perform one or more features of the tiled viewport composition system100. The processor204may include a general processor, a central processing unit, a graphics processing unit, an application specific integrated circuit (ASIC), a digital signal processor, a field programmable gate array (FPGA), a digital circuit, an analog circuit, a microcontroller, any other type of processor, or any combination thereof.

The memory interface206may manage the transportation of data going to and from the memory208, such as between the processor204and the memory208. The memory interface206may include any memory controller, such as a Memory Chip Controller (MCC), a Double Data Rate2 (DDR2) memory controller used to drive DDR2 SDRAM (double data rate synchronous dynamic random-access memory), a Peripheral Component Interconnect (PCI) bus controller, or any other type of memory controller. The memory interface206may communicate with the memory208over a bus214, such as a 64 bit DDR2 bus operating at 400 Megahertz or any other type of bus.

The memory208may comprise a device for storing and retrieving data or any combination thereof. The memory208may include non-volatile and/or volatile memory, such as a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or a flash memory. Alternatively or in addition, the memory208may include an optical, magnetic (hard-drive) or any other form of data storage device.

The memory208may store computer code, such as the composition manager108, the application102, and/or any other application. The computer code may include instructions executable with the processor204. The computer code may be written in any computer language, such as C, C++, assembly language, shader code, channel program code, and/or any combination of computer languages.

The memory208may include buffers, such as the application buffers104and the display buffer106. The application buffers104may include the tile buffers118.

All of the disclosure, regardless of the particular implementation described, is exemplary in nature, rather than limiting. The system100may include more, fewer, or different components than illustrated inFIGS. 1 and 2. For example, the system100may also include a display controller210and a display device212. The display controller210may read the composite image112from the display buffer106through the memory interface206and cause the composite image112to be displayed in the display device212.

The processor204may be in communication with the memory208via the memory interface206. The processor204may be in communication with other components, such as the display controller210and/or a network controller (not shown).

Furthermore, each one of the components of system100may include more, fewer, or different elements than is illustrated inFIGS. 1 and 2. For example, the memory208may include more, fewer, or different modules, graphics buffers, and applications. Moreover, the application102and/or the composition manager108may include various modules. Flags, data, databases, tables, entities, and other data structures may be separately stored and managed, may be incorporated into a single memory or database, may be distributed, or may be logically and physically organized in many different ways. The components may operate independently or be part of a same program or hardware. The components may be resident on separate hardware, such as separate removable circuit boards, or share common hardware, such as a same memory and processor for implementing instructions from the memory. Programs may be parts of a single program, separate programs, or distributed across several memories and processors.

In some examples, one or more of the components of the system100may be included in a System on a Chip (SOC)216. For example, the SOC216may include the processor204, the memory interface206and the display controller210. In another example, the SOC216may include additional components, such as memory.

The system100may be implemented in many processes and/or systems. For example, although some features are shown stored in computer-readable memories as logic implemented as computer-executable instructions or as data structures in memory, portions of the system100and its logic and data structures may be stored on, distributed across, or read from any other machine-readable storage media. Examples of the storage media may include memories, a cache, a buffer, RAM, a removable media, a hard drive, a hard disk, a floppy disk, a CD-ROM, or any other type of machine-readable or computer-readable storage medium. The media may include any non-transitory computer-readable storage media, such as volatile or non-volatile memory, RAM, ROM, CD-ROM, any other suitable storage device or any combination thereof. Alternatively or in addition, features and/or modules described as logic implemented as computer-executable instructions or as data structures in memory may be implemented in hardware or in a combination of hardware and software, such as in a field programmable gate array (FPGA).

The functions, acts or tasks illustrated in the figures or described may be executed in response to one or more sets of logic or instructions stored in or on computer readable media. The functions, acts or tasks are independent of the particular type of instructions set, storage media, processor or processing strategy and may be performed by software, hardware, integrated circuits, firmware, micro code and the like, operating alone or in combination. Likewise, processing strategies may include multiprocessing, multitasking, parallel processing, distributed processing, and/or any other type of processing. In one embodiment, the instructions are stored on a removable media device for reading by local or remote systems. In other embodiments, the logic or instructions are stored in a remote location for transfer through a computer network or over telephone lines. In yet other embodiments, the logic or instructions may be stored within a given computer, central processing unit (“CPU”), graphics processing unit (“GPU”), or system.

The processing capability of the system100may be distributed among multiple entities, such as among multiple processors and memories, optionally including multiple distributed processing systems. Parameters and other data structures may be separately stored and managed, may be incorporated into a single memory or database, may be logically and physically organized in many different ways, and may be implemented with different types of data structures such as linked lists, hash tables, or implicit storage mechanisms. Logic, such as programs or circuitry, may be combined or split among multiple programs, distributed across several memories and processors.

FIG. 3is a flow diagram of the logic of the tiled viewport composition system100. The application image110may be provided as logically divided into the tiles116. For example, the tile buffers118may be in the memory208and represent the tiles116.

The logic may determine (310) which of the tiles116are overlapped by the viewport114. The viewport114may identify the visible portion of the application image110, and the visible portion may be smaller than the application image110.

The tiles116that are overlapped by the viewport114may be buffered (320) in the front buffer122and the back buffer126, where the visible portion of the application image110is extracted from the front buffer122, and the application image110is modified in the back buffer126. For example, the front buffer122may buffer frame n, and the back buffer126may buffer frame n+1.

The tiles116not overlapped by the viewport114may be buffered (330) in the back buffer126in which the application image119is modified but not in the front buffer122from which the visible portion of the application image110is extracted. The logic may end, for example, by the composition manager108extracting the buffered visible portion124of the application image110from the front buffer122.

The logic may include additional, different, or fewer operations. In one such example, the logic may include dividing the application image110into the tiles116by allocating the tile buffers118. The operations may be executed in a different order than illustrated inFIG. 3.

While various embodiments have been described, it will be apparent to those of ordinary skill in the art that many more embodiments and implementations are possible within the scope of the disclosure. Accordingly, the disclosure is not to be restricted except in light of the attached claims and their equivalents.