Patent Publication Number: US-10776895-B2

Title: GPU power and performance management

Description:
BACKGROUND 
     This disclosure relates generally to the use of graphics processing units (GPUs). More particularly, but not by way of limitation, this disclosure relates to a technique for determining the computational need of GPU-centric elements executing from within pages of another application, selecting one or more GPU&#39;s appropriate to the need, and transitioning the system to the selected GPUs. 
     In mobile and embedded devices markets, consumers typically want designs that provide both more power and lower power consumption. As a result application processors have become increasingly heterogeneous, integrating multiple components into a single System-on-Chip (SoC) design. One SoC may include a central processing unit (CPU), a GPU, an image signal processor (ISP), video encoders and decoders, and a radio processing unit. For still further processing power, one or more additional GPUs may be provided. These additional GPUs, because of their power consumption, are often not incorporated directly in the SoC and are instead included as separate functional units. Again because of their power consumption, it is important that the use of such devices is properly controlled to preserve a device&#39;s battery life. 
     SUMMARY 
     In one embodiment the disclosed concepts provide a method to select a graphics processing units (GPUs) from a collection of GPUs based on the operational characteristics of an object embedded in a page of an application (e.g., an object in a web browser window). The method may include detecting an event in a system based on an action associated with a GPU-centric object, the system having two or more GPUs, the GPU-centric object including GPU instructions and non-GPU instructions. By way of example, the GPU-centric object may be a WebGL object, a WebGPU object or a Stage3D object. The method may then continue by determining, in response to the event, a computational need of the system based on one or more rules. In one or more embodiments the rules may be heuristic in nature and can evaluate a viewable characteristic of the GPU-centric object. Viewable characteristics can include, but are not limited to, determining if: the GPU-centric object changes its operational state from active to idle (or idle to active); or the GPU-centric object changes its operational state from not-viewable to viewable (or from viewable to not-viewable). The method continues by identifying a first GPU of the two or more GPUs is a currently active GPU for the system; identifying, based on the computational need, a second GPU, wherein the second GPU is different from the first GPU; transitioning the system from the first GPU to the second GPU; and executing the GPU-centric object&#39;s GPU instructions on the second GPU. In one embodiment, the first GPU has a lower computational capability than the second GPU. In another embodiment, it is the second GPU that has the lower computational capability. In still other embodiments, the operation of transitioning from the first to the second GPU may include evaluating one or more timing restrictions associated with the first and/or second GPU. In one or more other embodiments, the various methods may be embodied in computer executable program code and stored in a non-transitory storage device. In yet another embodiment, the method may be implemented in an electronic device or system having two or more GPUs. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  shows, in flowchart form, a GPU selection operation in accordance with one or more embodiments. 
         FIG. 2  shows, in flowchart form, an event detection operation in accordance with one or more embodiments. 
         FIG. 3  shows, in flowchart form, a computational need determination operation in accordance with one or more embodiments. 
         FIG. 4  shows, in flowchart form, a graphics processing unit (GPU) selection operation in accordance with one or more embodiments. 
         FIG. 5  shows, in flowchart form, a GPU transition operation in accordance with one or more embodiments. 
         FIG. 6  shows, in block diagram form, a computer system in accordance with one or more embodiments. 
         FIG. 7  shows, in block diagram form, a multi-function electronic device in accordance with one or more embodiments. 
     
    
    
     DETAILED DESCRIPTION 
     This disclosure pertains to systems, methods, and computer readable media to improve the operation of graphics systems. In general, techniques are disclosed for determining the computational need of GPU-centric elements executing from within pages of another application, selecting one or more GPU&#39;s appropriate to the need, and transitioning the system to the selected GPUs. 
     In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the disclosed concepts. As part of this description, some of this disclosure&#39;s drawings represent structures and devices in block diagram form in order to avoid obscuring the novel aspects of the disclosed concepts. In the interest of clarity, not all features of an actual implementation may be described. Further, as part of this description, some of this disclosure&#39;s drawings are in the form of flowcharts. The boxes in any particular flowchart may be presented in a particular order. It should be understood however that the particular sequence of any given flowchart is used only to exemplify one embodiment. In one or more embodiments, one or more of the disclosed steps may be omitted, repeated, and/or performed in a different order than that described herein. In addition, other embodiments may include additional steps not depicted as part of the flowchart. Moreover, the language used in this disclosure has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter, resort to the claims being necessary to determine such inventive subject matter. Reference in this disclosure to “one embodiment” or to “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosed subject matter, and multiple references to “one embodiment” or “an embodiment” should not be understood as necessarily all referring to the same embodiment. 
     Embodiments of the GPU selection and transition techniques set forth herein can assist with improving the functionality of computing devices or systems that utilize multiple GPUs. Computer functionality can be improved by enabling such computing devices or systems to select the most appropriate GPU so as to optimize its processing capability while minimizing its power consumption. By way of example, when computational capability is not needed (such as when the device or system&#39;s display is not changing), a low(er) computationally capable GPU may be selected (as opposed to a higher performance and higher power GPU), thereby reducing overall system power consumption. Conversely, when computational capability is needed (such as when the device or system&#39;s display is changing), a high(er) computationally capable GPU may be selected (as opposed to a lower performance and lower power GPU), thereby increasing the computing device or system&#39;s performance. These actions can have the effect of improving the device or system&#39;s performance (when needed) and, for mobile devices, extending the device or system&#39;s time before its battery needs to be charged. 
     It will be appreciated that in the development of any actual implementation (as in any software and/or hardware development project), numerous decisions must be made to achieve a developers&#39; specific goals (e.g., compliance with system- and business-related constraints), and that these goals may vary from one implementation to another. It will also be appreciated that such development efforts might be complex and time-consuming, but would nevertheless be a routine undertaking for those of ordinary skill in the design and implementation of multi-GPU computing devices and systems having the benefit of this disclosure. 
     Referring to  FIG. 1 , GPU selection operation  100  in accordance with one or more embodiments may begin when one (or more) of a specified number of different events is detected (block  105 ). Illustrative events include, but are not limited to, visible portions of the display transition from having a large (or small) amount of drawing occurring to having little or no (or a large amount of) drawing occurring, the passage of a predetermined interval of time, or the current state of the battery (e.g., whether there is little battery life left—e.g., 10%—or whether the device was just plugged in to charge). An evaluation may be performed to determine what computational need exists based on the detected change or event (block  110 ). For example, if there are N GPU&#39;s available with each providing a different level of computational capability, the detected change may be mapped to one of N computational levels and the corresponding GPU identified. In another embodiment, if there are N GPUs the evaluated need may identify two or more GPUs (e.g., multiple GPUs to share the computational work load). In this latter embodiment, some of the GPUs may have the same computational capability. The currently active GPU(s) may then be identified (block  115 ), and a determination made as to whether a different GPU(s) is needed (block  120 ). If the needed GPU or computational capability does not match the level of power provided by the currently selected or active GPU(s) (the “YES” prong of block  120 ), a transition to the appropriate GPU(s) may be made (block  125 ) where after operation  100  waits for the next event (block  130 ). If the needed computational capability is in line with or matches that provided by the currently selected or active GPU(s) (the “NO” prong of block  120 ), no GPU transition is needed and operation  100  again waits for the next event (block  130 ). 
     Referring to  FIG. 2 , in one embodiment the operation of detecting an “event” in accordance with block  105  may include checking for the occurrence of any of a number of possible incidents or events (blocks  200 - 215 ). By way of example only, these events may include whether a GPU-centric object was created or destroyed (block  200 ), whether one or more GPU-centric objects became idle-after-being-active or active-after-being-idle (block  205 ), whether one or more GPU-centric objects become viewable (e.g., scrolled into view or uncovered by the movement of one or more windows) or un-viewable such as when scrolled out of site or covered (block  210 ), whether the active window is resized, whether the application itself specifies (explicitly or implicitly) that a certain GPU, type of GPU or GPU performance is needed, or whether a timed interval has expired (block  215 ). If any of these, or operationally similar, events occur (the “YES” prongs of blocks  200 - 215 ), GPU selection operation  100  may move to block  110 . If none of the prior identified events are detected (the “NO” prongs of blocks  200 - 215 ), operation  100  may return to block  105 . As used here, a “GPU-centric object” is any element that requires or calls for use of GPU resources. Illustrative GPU-centric objects include WebGL objects, Stage3D objects and WebGPU objects. 
     Referring to  FIG. 3 , in one or more embodiments a system&#39;s computational need (based on a detected event) may begin by selecting a first GPU-centric object (block  300 ) and evaluating its computational need based one a series of heuristic rules (block  305 ). Table 1 below provides a sample of illustrative heuristics. 
                     TABLE 1               Example Heuristics to Evaluate GPU-Centric Objects                                            If instance            newly idle or            newly destroyed or            newly not-visible           Then less GPU capability needed;           If instance            newly active or            newly created or            newly visible           Then more GPU capability needed;           If instance renders to significant portion            of screen           Then more GPU capability needed;           If other applications require GPU-X           Then at least GPU-X capability needed;           If instance explicitly requires GPU-Y           Then at least GPU-Y capability needed;                        
Once the current object instance has been evaluated, a check may be made to determine is additional object instances remain to be evaluated (block  310 ). If at least one object instance remains to be evaluated (the “NO” prong of block  310 ), a next object instance may be selected (block  315 ) and evaluated as described above. If all object instances have been evaluated (the “YES” prong of block  310 ), GPU selection operation  100  may continue to block  115 . In another embodiment, only a specified subset of object instance may be evaluated during performance of block  110 . For example, only object instances associated with a display window that is at least partially visible may be evaluated.
 
     Once a system&#39;s computational need has been determined, GPU selection operation  100  may determine what GPU(s) are currently active. This information may be obtained, for example, through GPU-level system APIs. Referring to  FIG. 4 , if the current computational need is greater than that provided by the currently selected GPU(s) (the “YES” prong of block  400 ), an additional or different one or more GPUs may be identified (block  405 ) before operation moves to block  125 . If the current computational need is less than that provided by the currently selected GPU(s) (the “NO” prong of block  400 ), one or more GPUs may be identified to stop or an altogether different GPU may be identified (block  410 ) before operation moves to block  125 . 
     Referring to  FIG. 5 , one embodiment of GPU transition operation  125  is shown in which a first check may be made to determine if a transition is necessary (block  500 ). For example, if the needed computational capability determined in accordance with  FIG. 4  is equal to the currently provided capability—or within some specified range of “equal” (the “NO” prong of block  500 ), the currently selected GPU(s) may be retained; that is, no transition in accordance with block  125  would be needed. (This possibility is not explicitly provided for in  FIG. 4 .) If a transition is needed (the “YES” prong of block  500 ), a check to determine if a transition would meet certain specified timing requirements may be made (block  505 ). Timing requirements may include, but are not necessarily limited to, determining if less than a specified number of transitions have been made within a specified interval; or whether less than a minimum amount of time has elapsed since the last transition. If the specified timing requirements are not met (the “NO” prong of block  505 , operation  100  continues to block  130 . If the timing requirements are not violated (the “YES” prong of block  505 ), the system may be transitioned to the identified GPUs (block  510 ). In one embodiment, a currently selected GPU (i.e., a GPU currently selected by the system to execute GPU instructions) may be transitioned from the active state to an idle state before, or coincidently with, transitioning the newly selected GPU from an idle state to an active state. In another embodiment, the newly selected GPU may be transitioned from the idle state to the active state while the system continues to use the currently selected GPU; that is, the currently selected GPU and the newly selected GPU may share the system&#39;s GPU-specific load. 
     Tables 2 through 4 provide pseudo-code that may be used to guide the implementation of the disclosed subject matter in a computational system supports the WebGL API and has two (2) GPUs (one GPU may be a lower-performance/lower-power device while the other a higher-performance/higher-power device). The Web Graphics Library (WebGL) is a JavaScript API for rendering 3D graphics within any compatible application. WebGL elements can be mixed with other embedded objects and/or HyperText Markup Language (HTML) elements and composited with other parts of a page or page background. WebGL programs consist of control code written in JavaScript and shader code that is written in the OpenGL Shading Language (GLSL) which is executed by a computer system&#39;s GPU. GLSL is a high-level shading language created by the OpenGL Architecture Review Board (ARB) to give developers more direct control of the graphics pipeline without having to use ARB assembly language or hardware-specific languages. Referring to Table 2, the “createWebGL” routine may be called every time a WebGL object is instantiated (created). As shown, the system defaults to using the lower power GPU, and thereafter sends a messages to the host controller (“host”) that a new WebGL object has been created. 
                     TABLE 2               WebGL Object Creation Pseudo-Code                                            createWebGL( )  {            // for each WebGL instance, construct an            // OpenGL context that does NOT request            // the high-power GPU            var options = { OpenGLCanUseLowPowerGPU }            global glContext =            OpenGL.createContext(options);            // tell the host/controller that something            // has changed            host.sendMessage(WebGLHasUpdated);           }                        
Table 3 shows how the host controller may respond to a message generated in accordance with the pseudo-code of Table 2 (i.e., to the creation of a WebGL object). More specifically, it checks the new object&#39;s GPU requirements.
 
                     TABLE 3               Host Controller Pseudo-Code                                            receiveMessage (message) {            if (message == WebGLHasUpdated) {             checkGPURequirements( );            }            // handle any other message types           }                        
Table 4 provides pseudo-code for how one implementation of a 2 GPU system checks for the GPU requirements a WebGL object needs. Additional details may be obtained from Table 4&#39;s internal comments.
 
     
       
         
           
               
             
               
                 TABLE 4 
               
               
                   
               
               
                 GPU Requirement Check Pseudo-Code 
               
               
                   
               
             
            
               
                   
               
            
           
           
               
               
            
               
                   
                 checkGPURequirements( )  { 
               
               
                   
                  // look at all known WebGL instances, and 
               
               
                   
                  // decide if the high-power GPU should be 
               
               
                   
                  // used 
               
               
                   
                  needsHighPowerGPU = false; 
               
               
                   
                  for (x in webglContexts)  { 
               
               
                   
                  // evaluate heuristics (see discussion 
               
               
                   
                  // above), only some of which are 
               
               
                   
                  // implemented here e.g., check the size 
               
               
                   
                  // of the WebGL object, if it is obscured, 
               
               
                   
                  // idle, etc. 
               
               
                   
                   if  (x.wantsBetterGPU( ))  { 
               
               
                   
                    needsHighPowerGPU = true; 
               
               
                   
                    break; 
               
               
                   
                    // Exit because the result is known, 
               
               
                   
                    // alternatively a minimum number of 
               
               
                   
                    // WebGL objects could be required in 
               
               
                   
                    // this mode 
               
               
                   
                   } 
               
               
                   
                  } 
               
               
                   
                  if needsHighPowerGPU { 
               
               
                   
                   triggerHighPowerGPUIfNecessary( ); 
               
               
                   
                  } 
               
               
                   
                  else { 
               
               
                   
                   releaseHighPowerGPUIfNecessary( ); 
               
               
                   
                  } 
               
               
                   
                 } 
               
               
                   
                 global highPowerGPUContext = null; 
               
               
                   
                 triggerHighPowerGPUIfNecessary( )  { 
               
               
                   
                  if (highPowerGPUContext != null)  { 
               
               
                   
                   // work already done, simply return 
               
               
                   
                   return; 
               
               
                   
                  } 
               
               
                   
                  var options = { OpenGLMustUseHighPowerGPU }; 
               
               
                   
                  // demand the system use a particular GPU 
               
               
                   
                  global highPowerGPUContext = 
               
               
                   
                   OpenGL.createContext(options); 
               
               
                   
                  // once this global object exists, the system 
               
               
                   
                  // may swap to the high-power GPU, causing all 
               
               
                   
                  // WebGL instances to migrate to the high- 
               
               
                   
                  // power GPU 
               
               
                   
                 } 
               
               
                   
                 releaseHighPowerGPUIfNecessary( )  { 
               
               
                   
                  if (highPowerGPUContext == null)  { 
               
               
                   
                   // if not forcing the GPU, nothing need be 
               
               
                   
                   // done 
               
               
                   
                   return; 
               
               
                   
                  } 
               
               
                   
                  highPowerGPUContext = null; 
               
               
                   
                  // Once this global object is erased, the 
               
               
                   
                  // system is free to swap back to the lower- 
               
               
                   
                  // power GPU 
               
               
                   
                 } 
               
               
                   
                   
               
            
           
         
       
     
     While the above detailed example is directed to a WebGL API compliant 2 GPU system, in general any computational environment that supports the embedding of objects that execute GPU directed code and has two or more GPU&#39;s may benefit from the techniques disclosed herein. The GPUs may be similar or dissimilar. For example, all of a system&#39;s GPUs could be functionally equivalent (and consume equal amounts of power). In other embodiments the GPUs could be dissimilar. One or more could be lower-performance/lower-power while others (one or more) could be higher-performance/higher-power devices. In addition, one (or more) GPUs could be combined with one or more central processing units (CPUs) on a common substrate while other GPUs could be separate (e.g., discrete) from the CPU and, possibly, from other GPUs. Some of the GPUs may have multiple cores, others may have single cores. Some of the GPUs may be programmable, others may not. 
     Referring to  FIG. 6 , the disclosed GPU power and performance management operations in accordance with this disclosure may be performed by representative computer system  600  (e.g., a general purpose computer system such as a desktop, laptop, notebook or tablet computer system, or a gaming device). Computer system  600  can be housed in single computing device or spatially distributed between two or more different locations. Computer system  600  may include one or more processors  605 , memory  610 , one or more storage devices  615 , graphics hardware  620 , device sensors  625 , image capture module  630 , communication interface  635 , user interface adapter  640  and display adapter  645 —all of which may be coupled via system bus or backplane  650 . 
     Processor module or circuit  605  may include one or more processing units each of which may include at least one central processing unit (CPU) and zero or more GPUs; each of which in turn may include one or more processing cores. Each processing unit may be based on reduced instruction-set computer (RISC) or complex instruction-set computer (CISC) architectures or any other suitable architecture. Processor module  605  may be a system-on-chip, an encapsulated collection of integrated circuits (ICs), or a collection of ICs affixed to one or more substrates. Memory  610  may include one or more different types of media (typically solid-state, but not necessarily so) used by processor  605 , graphics hardware  620 , device sensors  625 , image capture module  630 , communication interface  635 , user interface adapter  640  and display adapter  645 . For example, memory  610  may include memory cache, read-only memory (ROM), and/or random access memory (RAM). Storage  615  may include one more non-transitory storage mediums including, for example, magnetic disks (fixed, floppy, and removable) and tape, optical media such as CD-ROMs and digital video disks (DVDs), and semiconductor memory devices such as Electrically Programmable Read-Only Memory (EPROM), and Electrically Erasable Programmable Read-Only Memory (EEPROM). Memory  610  and storage  615  may be used to retain media (e.g., audio, image and video files), preference information, device profile information, computer program instructions or code organized into one or more modules and written in any desired computer programming languages (e.g., WebGL and/or the GLSL language), and any other suitable data. Graphics hardware module or circuit  620  may be special purpose computational hardware for processing graphics and/or assisting processor  605  perform computational tasks. In one embodiment, graphics hardware  620  may include one or more GPUs, and/or one or more programmable GPUs and each such unit may include one or more processing cores. When executed by processor(s)  605  and/or graphics hardware  620  computer program code may implement one or more of the methods described herein. Device sensors  625  may include, but need not be limited to, an optical activity sensor, an optical sensor array, an accelerometer, a sound sensor, a barometric sensor, a proximity sensor, an ambient light sensor, a vibration sensor, a gyroscopic sensor, a compass, a barometer, a magnetometer, a thermistor sensor, an electrostatic sensor, a temperature sensor, a heat sensor, a thermometer, a light sensor, a differential light sensor, an opacity sensor, a scattering light sensor, a diffractional sensor, a refraction sensor, a reflection sensor, a polarization sensor, a phase sensor, a florescence sensor, a phosphorescence sensor, a pixel array, a micro pixel array, a rotation sensor, a velocity sensor, an inclinometer, a pyranometer and a momentum sensor. Image capture module or circuit  630  may include one or more image sensors, one or more lens assemblies, and any other known imaging component that enables image capture operations (still or video). In one embodiment, the one or more image sensors may include a charge-coupled device (CCD) or complementary metal oxide semiconductor (CMOS) sensor. Image capture module  630  may also include an image signal processing (ISP) pipeline that is implemented as specialized hardware, software, or a combination of both. The ISP pipeline may perform one or more operations on raw images (also known as raw image files) received from image sensors and can also provide processed image data to processor  605 , memory  610 , storage  615 , graphics hardware  620 , communication interface  635  and display adapter  645 . Communication interface  635  may be used to connect computer system  600  to one or more networks. Illustrative networks include, but are not limited to, a local network such as a Universal Serial Bus (USB) network, an organization&#39;s local area network, and a wide area network such as the Internet. Communication interface  635  may use any suitable technology (e.g., wired or wireless) and protocol (e.g., Transmission Control Protocol (TCP), Internet Protocol (IP), User Datagram Protocol (UDP), Internet Control Message Protocol (ICMP), Hypertext Transfer Protocol (HTTP), Post Office Protocol (POP), File Transfer Protocol (FTP), and Internet Message Access Protocol (IMAP)). User interface adapter  640  may be used to connect microphone(s)  650 , speaker(s)  655 , pointer device(s)  660 , keyboard  665  (or other input device such as a touch-sensitive element), and a separate image capture element  670 —which may or may not avail itself of the functions provided by graphics hardware  620  or image capture module  630 . Display adapter  645  may be used to connect one or more display units  675  which may also provide touch input capability. System bus or backplane  650  may be comprised of one or more continuous (as shown) or discontinuous communication links and be formed as a bus network, a communication network, or a fabric comprised of one or more switching devices. System bus or backplane  650  may be, at least partially, embodied in a network router or hub, a wireless access point (AP) or repeater, a set-top box, or a combination thereof. 
     Referring to  FIG. 7 , a simplified functional block diagram of illustrative mobile electronic device  700  is shown according to one embodiment. Electronic device  700  could be, for example, a mobile telephone, personal media device, a notebook computer system, or a tablet computer system. As shown, electronic device  700  may include processor module or circuit  705 , display  710 , user interface module or circuit  715 , graphics hardware module or circuit  720 , device sensors  725 , microphone(s)  730 , audio codec(s)  735 , speaker(s)  740 , communications module or circuit  745 , image capture module or circuit  750 , video codec(s)  755 , memory  760 , storage  765 , and communications bus  770 . 
     Processor  705 , display  710 , user interface  715 , graphics hardware  720 , device sensors  725 , communications circuitry  745 , image capture module or circuit  750 , memory  760  and storage  765  may be of the same or similar type and serve the same or similar function as the similarly named component described above with respect to  FIG. 6 . Audio signals obtained via microphone  730  may be, at least partially, processed by audio codec(s)  735 . Data so captured may be stored in memory  760  and/or storage  765  and/or output through speakers  740 . Output from image capture module or circuit  750  may be processed, at least in part, by video codec(s)  755  and/or processor  705  and/or graphics hardware  720 . Images so captured may be stored in memory  760  and/or storage  765 . 
     It is to be understood that the above description is intended to be illustrative, and not restrictive. The material has been presented to enable any person skilled in the art to make and use the disclosed subject matter as claimed and is provided in the context of particular embodiments, variations of which will be readily apparent to those skilled in the art (e.g., some of the disclosed embodiments may be used in combination with each other). Accordingly, the specific arrangement of steps or actions shown in any of  FIGS. 1-5  should not be construed as limiting the scope of the disclosed subject matter. The scope of the invention therefore should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. In the appended claims, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein.”