Multiprocess GPU rendering model

Methods, systems, and apparatuses, including computer programs encoded on a computer storage medium, for rendering application content are disclosed. In one embodiment a content receiver receives application content for rendering on a display unit of a computing device. A first processing unit renders the application content onto a first frame of a plurality of frames, and a second processing unit sequentially renders the plurality of frames onto the display unit. A counter counts of a number of outstanding frames as provided by the first processing unit to the second processing unit relative to corresponding acknowledgement messages indicating that one of the outstanding frames has been rendered onto the display unit. If the count is less than a threshold, the first processing unit renders the application content onto the first frame, otherwise the first processing unit waits to render the application content until the count is less than the threshold.

BACKGROUND

In modern day computing systems, rendering content onto a screen of a computer or other device may be done using one or more processes. In a single processing environment, a rendering process renders a frame of data from an application onto a buffer, and a graphics processing unit (GPU) renders the buffered data onto a screen of a device. This process then repeats in a continuous loop, in which the rendering process is blocked from performing any additional functions until the GPU calls for the frame stored in the buffer.

In a multi-processing system, the interaction between the rendering process and the GPU also works in a continuous loop, except that the rendering process will continue to generate frames and store them in the buffer at a rate faster than the GPU can render the frames for display, until the buffer is full. When the buffer is full, the rendering process will then be blocked, and will wait until the buffer is empty to begin generating new frames of data.

Blocking the rendering process while waiting for the GPU wastes valuable processing resources on a computing device, in both single-processing and multi-processing systems. Allowing the rendering process to continually process contents until the buffer is full, often results in a choppy image or animation on the screen, because the rendering process will wait or be blocked from resuming additional frame processing until the buffer is empty.

BRIEF SUMMARY

In general, one embodiment of the subject matter described in this specification may be embodied in a system in which a content receiver receives application content for rendering on a display unit of a computing device, a first processing unit renders the application content onto a first frame of a plurality of frames, and a second processing unit sequentially renders the plurality of frames onto the display unit. A counter of the system includes a count of a number of outstanding frames as provided by the first processing unit to the second processing unit, and a workflow engine determines when the first processing unit renders the visual content onto the first frame based on the counter.

Other embodiments may include corresponding methods, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices. Further embodiments, features, and advantages, as well as the structure and operation of the various embodiments are described in detail below with reference to accompanying drawings.

DETAILED DESCRIPTION

Embodiments are described herein with reference to illustrations for particular applications. It should be understood that the embodiments are not intended to be limiting. Those skilled in the art with access to the teachings provided herein will recognize additional modifications, applications, and embodiments within the scope thereof and additional fields in which the embodiments would be of significant utility.

FIG. 1is a diagram illustrating a multi-processing system for rendering visual content, according to an embodiment. System100includes a computing device102. The computing device102includes a first processing unit (FPU)104, a second processing unit (SPU)106, a display unit108, and an application110. In one embodiment. FPU104and SPU106may work together to render content or data from application110onto display unit108of computing device102.

Computing device102may include any electronic computing device. For example, computing device102may be a computer, laptop, desktop, mobile phone, monitor, television, tablet computer, or other computing device. Computing device102may include one or more processors.

FPU104and SPU106may be processing units or processes of computing device102. FPU104and SPU106may be general purpose or specialized processes. For example, SPU106may be a graphics processing unit (GPU) while FPU104is a general processing unit that renders application content112as frames118. In one an embodiment, FPU104and SPU106may be individual threads or other processes operating on a central processing unit (CPU) or operating system of computing device102. In another embodiment, FPU104and SPU106may be separate processors executing on computing device102, which may include a multi-core processor or multiple processors.

In an example embodiment, FPU104and SPU106may processes, each operating on its own computing device102. FPU104and SPU106(operating on separate computing devices102) may then asynchronously communicate over any circuitry, wired or wireless connection enabling communication between the computing device102. For example, the computing devices102by be coupled together or may otherwise communicate over any wired or wireless network, such as the Internet or a local intranet.

Though system100is described in reference to two processing units, one skilled in the art will recognize that computing device102may include additional processing units operable within system100and that the details described herein may include more than two processing units.

Display unit108may be a monitor or a screen upon which data is displayed on computing device102. For example, display unit108may be an LCD (liquid crystal display), LED (light-emitting diode), plasma or other display unit. Computing device102may be either an integrated display unit108, such the screen of a mobile phone, or a separate display unit108, such as one or more monitors connected to a desktop computer. Other example embodiments may include wireless communications between display unit108and computing device102.

Application110may include any application operating on computing device. Application1110may provide application content112to be displayed on display unit108. Application110may receive or generate application content112. For example, application110may be a browser connected to the Internet that receives application content112from one or more web servers over a network. In other embodiments, application110may be a video game application that generates application content112, such as graphics, for display pertaining to the game being played, a word processing application that changes responsive to user input, or a multimedia playing application that receives content from a DVD or other multimedia file.

Though application110is shown as a separate unit, one skilled in the art understands that FPU104and SPU106may be processes operating within application110. For example, application110may a graphics application that processes trusted and untrusted application content110(e.g., such as content downloaded from the Internet or another outside source). As such, for security reasons, application110may include FPU104that renders frames118from application content112(as discussed in greater detail below), and SPU106that renders frames118onto display unit108. In such an embodiment, the processing described herein with regards to system100may be performed mostly or wholly within application110. In other example embodiments, application110may be a separate application that communicates with FPU104and SPU106.

Application content112may include any content from application110for rendering on display unit108. Application content112may include electronic files or data such as digital images, text, scanned images or other multi-media content. In an embodiment, application content112may include an animation, a video, a slideshow or other sequence of frames or content that is to be rendered sequentially over a period of time. For example, application content112may include a five minute video clip from a website to be rendered on display unit108, application content112including data for a sequence of steady-state frames that are to be rendered on display unit108. Or, for example, application content112may include a video game, a cartoon, or a slideshow.

In another embodiment, application content112may include updated content for rendering on display unit108, based on a determination that the currently displayed content is out-of-date or otherwise needs to be updated. For example, application110may detect the movement of a mouse or a cursor, and application content112may include new content that includes a visual indication of the new position of the mouse or cursor on display unit108.

FPU104may render or generate frame118based on the received application content112. Frame118may include a visual depiction or screen shot of application content112at a particular moment of time. Frame118may include color values for each pixel of display unit108, and alpha channel values with transparency information for the pixels. In an embodiment, FPU104may render or generate frame118based on the dimensions or specifications of display unit108. For example, the size and number of pixels and the colors used may be dependent on the capabilities of display unit108.

Buffer120may include a portion of memory used to store frames118. Buffer120may be, for example, a frame buffer used to store frame118data as provided by FPU104. Buffer120may be any size buffer. In an embodiment, buffer120is big enough to store at least two frames118. Though buffer120is shown, as residing on SPU106, other embodiments may include buffer120as residing on FPU104, or may include multiple buffers, one on each FPU104and SPU106, or may include buffer120as residing as an independent unit or part of a third processing unit not shown).

In one embodiment, SPU106may be a graphics processing unit (GPU), or other specialized process that application content112from frame118onto display unit108, SPU106may call a swap-buffers function or send a swap buffers message to retrieve a frame118from buffer120. If buffer120is a first-in, first-out (FIFO) buffer, then a swap-buffer function call may release the top-most frame118to SPU106for processing. SPU106may then render the content of frame118onto display unit108.

Display unit108may include a refresh rate122. Refresh rate122may be an al during which the screen of display unit108is refreshed with new data or a new frame118. For example, refresh rate122may be the number of times per second that SPU106renders frame118onto display unit108. During each refresh, SPU106may either re-render a currently rendered frame118on display unit108again (i.e., if no frames118are waiting in buffer120) or retrieve a new frame118from buffer120to render.

In one embodiment, SPU106may request or retrieve a new frame118from buffer120based on refresh rate122. For example, upon a refresh of display unit108, SPU106may check buffer120to see if there are any frames118available for rendering. If buffer120includes at least one frame118, then SPU106will retrieve the frame118and render it onto display unit108. Otherwise, if buffer120is empty, SPU106may render the currently rendered frame118again, or do nothing.

In another embodiment, SPU106may retrieve frames118from buffer120based upon a determination that the currently rendered frame118needs to be updated. For example, application110may determine that the current frame118needs to be updated (e.g., based on a user action or other event occurrence), and may send a signal to SPU106to update the frame118. In another example embodiment, user input, such as a mouse movement or keyboard stroke, may indicate that the current frame118is to be updated and cause SPU106to retrieve a frame118from buffer120. SPU106may, for example, retrieve and render frames118from buffer120onto display unit108at a rate limited by refresh rate122. Exceeding refresh rate122may cause a user operating computing device102not to see any frame118rendered on display unit108in between refreshes.

In conventional systems, a rendering processor generates frames and stores them on a buffer until the buffer is full. A graphics processor then retrieves the frames, one-at-a-time, and renders them on display unit108while blocking the rendering processor waits or is otherwise blocked from doing any additional processing until the buffer is empty. Once the buffer is empty, the rendering processor will begin generating additional frames and repeat the process. Such conventional systems often result in inconsistent frame rates and choppy images being rendered on a display unit. Further, such conventional systems waste resources by blocking a rendering processor from performing others actions until the buffer is empty.

System100, by contrast, rate-limits how many frames118FPU104stores in buffer120and allows FPU104to perform other processing while buffer120is at its pre-determined capacity. System100uses a counter130to count or otherwise keep track of how many frames118are stored in buffer120. Counter130maintains a count132of the number of flames118in buffer120. Counter130increments count132each time FPU104generates a frame118that is stored in buffer120, and decrements count132upon receipt of an acknowledge (ACK) message136from SPU106indicating that a frame118from buffer120has been retrieved and/or rendered on display unit108. Once count132reaches a threshold134, FPU104waits to render any additional frames118and is freed to perform other processing tasks.

Counter130counts how many outstanding frames118exist in buffer120. As just referenced, counter130may be a simple counter that increments count132up to a threshold134, and decrements count132down to zero. Count132is the number of outstanding frames118stored in buffer120. Threshold134is the maximum number of frames that are to be held in buffer120. In one embodiment, buffer120may have the capacity to hold a number of frames118greater than the number specified by threshold134. However, when count132reaches threshold134, FPU104may wait until count132drops below threshold134to continue rendering frames118.

After threshold134is reached, FPU104may perform other processing tasks until count132drops below threshold134. For example, FPU104may be a general processing unit that may render frames118, but also perform other processing tasks on computing device102. For example, such other processing tasks as may be include, for example, pre-processing application content112for rendering as a frame118or performing other functions unrelated to frame118rendering.

If upon decrementing count132, count132is less than threshold134, then FPU104may generate a new frame118from application content112. This process may repeat so long as there is application content112available. In an embodiment, if count132is less than threshold134, FPU104or counter130may request additional application content112from application110. In another embodiment, application110may push application content112as it is ready, to be received by FPU104. In another embodiment, FPU104may receive application content112from multiple applications110.

In one embodiment, counter130may have a time-out period, after which if count132has not dropped below threshold134or otherwise been decremented, then FPU104may continue rendering frames118. For example, after time-out period expires, it may be determined that ACK message136was never received but that a frame118was rendered. Then, for example, buffer120may be cleared and count132returned to zero, or count132may be decremented to account for a number of frames118that may be determined to have been processed during time-out period and FPU104may continue frame118rendering.

Threshold134may be set to a value that allows system100to operate in a smooth manner and provide a consistent frame rate by which SPU106may render frames118onto display unit108and that may coordinate with refresh rate122of display unit108. For example, display unit108may have a refresh rate122of 60 hertz, in which case, setting threshold134to two may allow for a consistent frame rate to occur when rendering application content112on system100. Other example embodiments may include display units108with divergent or variable refresh rates122and/or thresholds134.

System100provides a reliable, repeatable system by which FPU104may be rate-limited, such that buffer120does not include too many, or too few frames118and that the processing resources of FPL104are not wasted. Once threshold134is reached, FPU104is free to perform other processing tasks while SPU106renders frames118from buffer120onto display unit108. Once count132, falls below threshold134, FPU104may then continue rendering frames118rather than waiting until buffer120is empty to continue frame118rendering, thereby providing a reliable frame rate for system100. Other example embodiments may include multiple thresholds134. For example, there may be two thresholds134, one indicating a maximum threshold for a number of frames118that may be stored in buffer120and one being the number of frames118remaining in buffer120at which point FPU104continues rendering frames118. The result may be more consistent frame rates and smoother images rendered on display unit108, and freeing up FPU104to perform additional processing, rather sitting idly waiting for buffer120to empty.

FIG. 2is a flowchart of a method for rendering visual content on a multi-processing system, according to an embodiment.

At stage205, application content is received. For example, FPU104may receive application content112from application110. Application content112may include any data from application110to be rendered on display unit108. Application content112may include, for example, an animation or other steady-stream data or video content for a sequence of frames.

At stage210, a determination is made as to whether count is less than threshold. For example counter130may determine whether count132is less than threshold134. Counter130may increment count132each time FPU104renders a frame118and provides that frame118to SPU106, and may decrement count132each time SPU106returns an ACK message136indicating that an outstanding frame118has been processed displayed on display unit108). When additional application content112is received, counter130checks count132to see if count132is less than threshold134. Threshold134is the maximum number of outstanding flames118that may be stored in buffer120, waiting to be processed by SPU106.

At stage215, if the count is greater than or equal to the threshold, a determination is made as to whether an ACK message has been received. For example, SPU106may return ACK message136each time a frame118from buffer120is processed and rendered on display unit108. Counter130may track the generation or receipt of ACK messages136from SPU106. In another embodiment, another unit (not shown) may generate or transmit ACK message136after a frame118is retrieved from buffer120and/or the frame118is rendered onto display unit108.

At stage220, if no ACK message has been received, then the FPU waits until a message is received. As referenced above, counter130decrements count132each time an ACK message136is received. During this wait period220, FPU104may perform other processes that may be waiting to be performed on computing device102. In another embodiment, counter130may have a time-out period, which may indicate a maximum wait time during which to wait for the receipt of an ACK message136. If upon the expiration of the time-out period no ACK message136has been received, or if an ACK message136is received prior to the expiration of the time-out period, processing may continue at stage215. If at stage215, the time-out period had expired without receipt of an ACK message136, processing may continue to stage225. Otherwise, if the no ACK message136has been received and the time-out period has not expired, processing may continue to wait at stage220.

At stage225, the count is decremented. For example, counter130may decrement count132upon receipt or detection of ACK message136from SPU106. In another embodiment as referenced above, count132may be decremented at stage225at the expiration of a time-out period of no ACK message136is received. After count132is decremented, processing returns to stage210where count132is compared against threshold134.

At stage230, a frame is rendered on the FPU. For example, FPU104may render frame118from application content112received from application110. In one embodiment, FPU104may store application content112to be rendered locally until count132is less than threshold134, or FPU104may receive or request application content112from application110after count130is less than threshold134.

At stage240, the frame is provided to a buffer. For example, FPU104may render frame118, which is stored in buffer120. Frames118, as stored in buffer120, may then be available to SPU106for rendering on display unit108. For example, SPU106may sequentially render frames118as stored in buffer120in a first-in, first-out (FIFO) manner.

At stage240, the count is incremented. For example, after frame118is generated by FPU104and/or stored in buffer120, counter130may increment count132. Count132may track how many frames118are stored in buffer120, or are otherwise waiting to be rendered on display unit108by SPU106.

At stage250, additional video content may be requested. For example, if count132is less than threshold134, then FPU104may request additional application content112from application110is application110has not already pushed or otherwise made available addition application content112. If there is no additional application content112at that time, FPU104may perform other processing and wait until additional application content112is received. When additional application content112is received, processing may continue at stage205.

FIG. 3is a flowchart of a method for rendering visual content on a multi-processing system, according to an embodiment.

At stage310, visual content is received from an application on a first processor. For example, FPU104may receive application content112from application110. FPU104may then generate frame118corresponding to application content112.

In one embodiment, prior to rendering frame118, FPU104may check to see that count132is less than threshold134. If count130is less than threshold134, then FPU104may generate frame118, otherwise FPU104may wait to generate frame118(i.e., until count132is less than threshold134).

At stage320, a count of outstanding frames is incremented each time the first processor provides one of the rendered frames to a second processor. For example, each time FPU104provides a frame118to SPU106, or stores frame118in buffer120, counter130may increment count132by one.

At stage330, a count of outstanding frames is decremented each time an acknowledgement message is received, the acknowledgement message indicating that the second processor has rendered a received frame onto the display unit. For example, each time SPU106makes a swap-buffers call, or otherwise retrieves frame118from buffer120, SPU106may transmit ACK message136. Upon receiving ACK message136at FPU104or counter130, counter130may decrement count132by one. If count132is below threshold134after the decrement, then FPU104may generate a new frame118for buffer120.

FIG. 4illustrates an example computer system400in which embodiments of as described herein, or portions thereof, may be implemented as computer-readable code. For example, system100may be implemented in computer system400using hardware, software, firmware, tangible computer readable media having instructions stored thereon, or a combination thereof and may be implemented in one or more computer systems or other processing systems. Hardware, software, or any combination of such may embody any of the modules, procedures and components inFIGS. 1-3.

If programmable logic is used, such logic may execute on a commercially available processing platform or a special purpose device. One of ordinary skill in the art may appreciate that embodiments of the disclosed subject matter can be practiced with various computer system configurations, including multi-core multiprocessor systems, minicomputers, mainframe computers, computers linked or clustered with distributed functions, as well as pervasive or miniature computers that may be embedded into virtually any device.

For instance, a computing device having at least one processor device and a memory may be used to implement the above-described embodiments. A processor device may be a single processor, a plurality of processors, or combinations thereof. Processor devices may have one or more processor “cores.”

Processor device404may be a special purpose or a general-purpose processor device. As will be appreciated by persons skilled in the relevant art, processor device404may also be a single processor in a multi-core/multiprocessor system, such system operating alone, or in a cluster of computing devices operating in a cluster or server farm. Processor device404is connected to a communication infrastructure406, for example, a bus, message queue, network, or multi-core message-passing scheme.

Computer system400also includes a main memory408, for example, random access memory (RAM), and may also include a secondary memory410. Secondary memory410may include, for example, a hard disk drive412, removable storage drive414. Removable storage drive414may comprise a floppy disk drive, a magnetic tape drive, an optical disk drive, a flash memory, or the like. The removable storage drive414reads from and/or writes to a removable storage unit418in a well-known manner. Removable storage unit418may comprise a floppy disk, magnetic tape, optical disk, etc. which is read by and written to by removable storage drive414. As will be appreciated by persons skilled in the relevant art, removable storage unit418includes a computer usable storage medium having stored therein computer software and/or data.

In alternative embodiments, secondary memory410may include other similar means for allowing computer programs or other instructions to be loaded into computer system400. Such means may include, for example, a removable storage unit422and an interface420. Examples of such means may include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an EPROM, or PROM) and associated socket, and other removable storage units422and interfaces420which allow software and data to be transferred from the removable storage unit422to computer system400.

Computer system400may also include a communications interface424. Communications interface424allows software and data to be transferred between computer system400and external devices. Communications interface424may include a modem, a network interface (such as an Ethernet card), a communications port, a PCMCIA slot and card, or the like. Software and data transferred via communications interface424may be in the form of signals, which may be electronic, electromagnetic, optical, or other signals capable of being received by communications interface424. These signals may be provided to communications interface424via a communications path426. Communications path426carries signals and may be implemented using wire or cable, fiber optics, a phone line, a cellular phone link, an RF link or other communications channels.

In this document, the terms “computer program medium” and “computer usable medium” are used to generally refer to media such as removable storage unit418, removable storage unit422, and a hard disk installed in hard disk drive412. Computer program medium and computer usable medium may also refer to memories, such as main memory408and secondary memory410, which may be memory semiconductors (e.g. DRAMs, etc.).

Computer programs (also called computer control logic) are stored in main memory408and/or secondary memory410. Computer programs may also be received via communications interface424. Such computer programs, when executed, enable computer system400to implement the present embodiments as discussed herein, in particular, the computer programs, when executed, enable processor device404to implement the processes as illustrated by the flowcharts ofFIGS. 3 and 4discussed above. Accordingly, such computer programs represent controllers of the computer system400. Where an embodiment is implemented using software, the software may be stored in a computer program product and loaded into computer system400using removable storage drive414, interface420, and hard disk drive412, or communications interface424.

The embodiments have been described above with the aid of functional building blocks illustrating the embodiment of specified functions and relationships thereof. The boundaries of these functional building blocks have been arbitrarily defined herein for the convenience of the description. Alternate boundaries can be defined so long as the specified functions and relationships thereof are appropriately performed.

The foregoing description of the specific embodiments allows others, by applying knowledge within the skill of the art, readily modify and/or adapt for various applications such specific embodiments, without undue experimentation, without departing from the general concept of what has been described herein. Therefore, such adaptations and modifications are intended to be within the meaning and range of equivalents of the disclosed embodiments, based on the teaching and guidance presented herein. It is to be understood that the phraseology or terminology herein is for the purpose of description and not of limitation, such that the terminology or phraseology of the present specification is to be interpreted by the skilled artisan in light of the teachings and guidance.

The breadth and scope of the embodiments should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.