Patent Publication Number: US-2020289948-A1

Title: Suspending state of cloud-based legacy applications

Description:
CLAIM OF PRIORITY 
     This application is a continuation of U.S. patent application Ser. No. 15/937,531 filed Mar. 27, 2018, the entire disclosures of which are incorporated herein by reference. U.S. patent application Ser. No. 15/937,531 is a continuation of U.S. patent application Ser. No. 13/791,379, filed Mar. 8, 2013, the entire disclosures of which are incorporated herein by references. U.S. patent application Ser. No. 13/791,379 claims the priority benefit of U.S. Provisional Patent Application No. 61/666,679 filed Jun. 29, 2012, the entire disclosures of which are incorporated herein by reference. 
     CROSS-REFERENCE TO RELATED APPLICATION 
     This application is related to commonly-assigned, co-pending provisional application Ser. No. 61/666,628, (Attorney Docket Number SCEA12004US00) filed Jun. 29, 2012, and entitled “DETERMINING TRIGGERS FOR CLOUD-BASED EMULATED GAMES”, the entire disclosures of which are incorporated herein by reference. 
     This application is related to commonly-assigned, co-pending provisional application Ser. No. 61/666,645, (Attorney Docket Number SCEA12005US00) filed Jun. 29, 2012, and entitled “HAPTIC ENHANCEMENTS FOR EMULATED VIDEO GAME NOT ORIGINALLY DESIGNED WITH HAPTIC CAPABILITIES”, the entire disclosures of which are incorporated herein by reference. 
     This application is related to commonly-assigned, co-pending provisional application Ser. No. 61/666,665, (Attorney Docket Number SCEA12006US00) filed Jun. 29, 2012, and entitled “CONVERSION OF HAPTIC EVENTS INTO SCREEN EVENTS”, the entire disclosures of which are incorporated herein by reference. 
    
    
     FIELD OF THE DISCLOSURE 
     The present disclosure is related to video game emulation. Among other things, this application describes a method and apparatus for emulating a video game that includes generating snapshots that can be used for incorporating new content into the emulated video games. 
     BACKGROUND OF THE INVENTION 
     Finding new ways to play preexisting video games can increase the longevity of older games. Instead of replaying the same level or completing the same missions repeatedly, gamers often desire new challenges when replaying legacy games. In response to this need, game designers have begun to produce mini-games. Within a mini-game, the gamer can be instructed to complete new objectives or challenge their friends for high scores in a format that was not originally designed into the legacy game. Further, since the mini-game is derived from a legacy game, the gamer already knows the characters and basic components of the game, and is therefore more likely to play the mini-game. 
     Mini-games often do not begin at traditional starting points that were used in the original game. For example, the mini-game may begin near the end of a level, just prior to facing a final opponent, or the boss of the level. A boss is an enemy-based challenge which is found in many video games. Bosses are generally seen at the climax of a particular section of the game, usually at the end of a stage or level. Due to the climactic nature of fighting a boss, mini-game designers may choose to use this section of the game as their starting point. In order to make the mini-game more challenging than the original version, the game designer may also want to limit the number of lives a player may use, or change other game parameters such as the amount of health the main character has remaining. Other game scenarios may be chosen as starting points for a min-game. For example a mini-game may begin with the game player being the batter in a baseball game where there are two outs in the bottom of the ninth inning and the batter&#39;s team is down by one run. 
     However, in order to generate mini-games that start with these specific circumstances a game designer must reverse engineer the underlying code in each game and then rewrite the code to enable the mini-game to start under these specific conditions. This process is time consuming and expensive, because it may require multiple engineers to redevelop a single legacy game. Additionally since mini-games are shorter than full-length games, there is a need to produce mini-games in larger quantities. 
     It is within this context that aspects of the present disclosure arise. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a schematic diagram of a client and an emulator communicating over a network according to an aspect of the present disclosure. 
         FIG. 2  is a flow diagram illustrating a method of producing a snapshot for an emulated game according to an aspect of the present disclosure. 
         FIG. 3A  is a schematic diagram illustrating the process of the client delivering legacy game input data to the emulator according to an aspect of the present disclosure. 
         FIG. 3B  is a schematic diagram illustrating the process of the client delivering a legacy game suspension request to the emulator, and in response the emulator delivers the snapshot to the client. 
         FIG. 4A  is a block diagram describing the instructions for how the client receives a snapshot according to an aspect of the present disclosure. 
         FIG. 4B  is a block diagram describing the instructions for how an emulator generates a snapshot according to an aspect of the present disclosure. 
     
    
    
     DETAILED DESCRIPTION OF THE DRAWINGS 
     Although the following detailed description contains many specific details for the purposes of illustration, anyone of ordinary skill in the art will appreciate that many variations and alterations to the following details are within the scope of the present disclosure. 
     Accordingly, the aspects of the present disclosure described below are set forth without any loss of generality to, and without imposing limitations upon, the claims that follow this description. 
     In order to create mini-games without reverse engineering and recoding a game designer may rely on the use of triggers and snapshots to provide information needed for the mini-games without having to dig into the code of the legacy game. Triggers are further described in commonly assigned co-pending application Ser. No. 61/666,628 (Attorney Docket Number SCEA12004US00) filed Jun. 29, 2012 and entitled “DETERMINING TRIGGERS FOR CLOUD-BASED EMULATED GAMES” and corresponding U.S. Pat. No. 9,717,989. The mini-games may be made by providing an emulator with game inputs that bring the emulated game to a certain point where the mini-game will begin. A snapshot may be generated according to aspects of the present disclosure and that point may be used as the used as the starting point in the future mini-game. Thereafter, triggers may be generated in order to provide new experiences for the game. Game designers may then develop a script and combine it with the snapshot and triggers to produce the mini-game. Therefore, in order to design mini-games in this manner, there is a need in the art for a method of quickly generating snapshots. 
     A snapshot may be a recorded description of the state of every device being emulated at a designated time during the emulation according to an aspect of the present disclosure. A snapshot may be generated by a client. First the client delivers game inputs to an emulator. The emulator receives the game inputs and proceeds to emulate the game according to the game inputs. At some point during the emulation, the client delivers a suspension request to the emulator. Once the suspension request is received, the emulator will suspend the emulated title at the next point in time at which all devices being emulated are in a steady state. As used herein, steady state means that there are no asynchronous activities occurring in the emulator. At this steady state, the emulator generates a snapshot of the emulated game by recording the current state of all devices being emulated. The snapshot is then delivered to the client. 
       FIG. 1A  is a schematic of an embodiment of the present invention. Emulator  107  may be accessed by a client  102  over a network  160 . Client  102  may access alternative emulators  107  over the network  160 . Emulators  107  may be identical to each other, or they may each be programed to emulate unique legacy game titles  106  or unique sets of legacy game titles  106 . Additionally, the client  102  may be contained within the emulator  107  and may communicate directly with the emulator over an internal system bus  150 ′ 
     Client  102  may include a central processor unit (CPU)  131 . By way of example, a CPU  131  may include one or more processors, which may be configured according to, e.g., a dual-core, quad-core, multi-core, or Cell processor architecture. Client  102  may also include a memory  132  (e.g., RAM, DRAM, ROM, and the like). The CPU  131  may execute a process-control program  133 , portions of which may be stored in the memory  132 . The client  102  may also include well-known support circuits  140 , such as input/output (I/O) circuits  141 , power supplies (P/S)  142 , a clock (CLK)  143  and cache  144 . The client  102  may optionally include a mass storage device  134  such as a disk drive, CD-ROM drive, tape drive, or the like to store programs and/or data. The client  102  may also optionally include a display unit  137  and a user interface unit  138  to facilitate interaction between the client  102  and a user who requires direct access to the client  102 . The display unit  137  may be in the form of a cathode ray tube (CRT) or flat panel screen that displays text, numerals, or graphical symbols. The user interface unit  138  may include a keyboard, mouse, joystick, light pen, or other device. A controller  145  may be connected to the client  102  through the I/O circuit  141  or it may be directly integrated into the client  102 . The controller  145  may facilitate interaction between the client  102  and a user. The controller  145  may include a keyboard, mouse, joystick, light pen, hand-held controls or other device. The controller  145  is also may be capable of generating a haptic response  146 . By way of example and not by way of limitation, the haptic response  146  may be vibrations or any other feedback corresponding to the sense of touch. The client  102  may include a network interface  139 , configured to enable the use of Wi-Fi, an Ethernet port, or other communication methods. 
     The network interface  139  may incorporate suitable hardware, software, firmware or some combination of two or more of these to facilitate communication via an electronic communications network  160 . The network interface  139  may be configured to implement wired or wireless communication over local area networks and wide area networks such as the Internet. The client  102  may send and receive data and/or requests for files via one or more data packets over the network  160 . 
     The preceding components may exchange signals with each other via an internal system bus  150 . The client  102  may be a general purpose computer that becomes a special purpose computer when running code that implements embodiments of the present invention as described herein. 
     The emulator  107  may include a central processor unit (CPU)  131 ′. By way of example, a CPU  131 ′ may include one or more processors, which may be configured according to, e.g., a dual-core, quad-core, multi-core, or Cell processor architecture. The emulator  107  may also include a memory  132 ′ (e.g., RAM, DRAM, ROM, and the like). The CPU  131 ′ may execute a process-control program  133 ′, portions of which may be stored in the memory  132 ′. The emulator  107  may also include well-known support circuits  140 ′, such as input/output (I/O) circuits  141 ′, power supplies (P/S)  142 ′, a clock (CLK)  143 ′ and cache  144 ′. The emulator  107  may optionally include a mass storage device  134 ′ such as a disk drive, CD-ROM drive, tape drive, or the like to store programs and/or data. The emulator  107  may also optionally include a display unit  137 ′ and user interface unit  138 ′ to facilitate interaction between the emulator  107  and a user who requires direct access to the emulator  107 . By way of example and not by way of limitation a client or engineer  102  may need direct access to the emulator  107  in order to program the emulator  107  to properly emulate a desired legacy game  106  or to add additional mini-game capabilities to a legacy game  106 . 
     The display unit  137 ′ may be in the form of a cathode ray tube (CRT) or flat panel screen that displays text, numerals, or graphical symbols. The user interface unit  138 ′ may include a keyboard, mouse, joystick, light pen, or other device. The emulator  107  may include a network interface  139 ′, configured to enable the use of Wi-Fi, an Ethernet port, or other communication methods. 
     The network interface  139 ′ may incorporate suitable hardware, software, firmware or some combination of two or more of these to facilitate communication via the electronic communications network  160 . The network interface  139 ′ may be configured to implement wired or wireless communication over local area networks and wide area networks such as the Internet. The emulator  107  may send and receive data and/or requests for files via one or more data packets over the network  160 . 
     The preceding components may exchange signals with each other via an internal system bus  150 ′. The emulator  107  may be a general purpose computer that becomes a special purpose computer when running code that implements embodiments of the present invention as described herein. 
     Emulator  107  may access a legacy game  106  that has been selected by the client  102  for emulation through the internal system bus  150 ′. There may be more than one legacy game  106  stored in the emulator. The legacy games may also be stored in the memory  132 ′ or in the mass storage device  134 ′. Additionally, one or more legacy games  106  may be stored at a remote location accessible to the emulator  107  over the network  160 . Each legacy game  106  contains game code  108 . When the legacy game  106  is emulated, the game code  108  produces legacy game data  109 . 
     By way of example, a legacy game  106  may be any game that is not compatible with a target platform. By way of example and not by way of limitation, the legacy game  106  may have been designed to be played on Sony Computer Entertainment&#39;s PlayStation console, but the target platform is a home computer. By way of example, the legacy game  106  may have been designed to be played on a PlayStation  2  console, but the target platform is a PlayStation  3  console. Further, by way of example and not by way of limitation, a legacy game  106  may have been designed to be played on a PlayStation console, but the target platform is a hand held console such as the PlayStation Vita from Sony Computer Entertainment. 
     Emulator  107  may be a deterministic emulator. A deterministic emulator is an emulator that may process a given set of game inputs  347  the same way every time that the same set of inputs  347  are provided to the emulator  107 . This may be accomplished by eliminating any dependencies in the code run by the emulator  107  that depend from an asynchronous activity. Asynchronous activities are events that occur independently of the main program flow. This means that actions may be executed in a non-blocking scheme in order to allow the main program flow to continue processing. Therefore, by way of example, and not by way of limitation, the emulator  107  may be deterministic when the dependencies in the code depend from basic blocks that always begin and end with synchronous activity. By way of example, basic blocks may be predetermined increments of code at which the emulator  107  checks for external events or additional game inputs  347 . The emulator  107  may also wait for anything that runs asynchronously within a system component to complete before proceeding to the next basic block. A steady state within the emulator  107  may be when all of the basic blocks are in lock step. 
     As shown in  FIG. 2 , the emulator  107  may be configured to implement a method for generating a snapshot of an emulated legacy game  106  according to an inventive method  200 . Various aspects of the method  200  may be implemented by execution of computer executable instructions running on the client  102  and/or the emulator  107  in conjunction with the actions of a client  102 . Specifically, a client  102  may be configured, e.g., by suitable programming, to implement certain client device platform instructions  270 . In addition, an emulator  107  may be configured to implement certain emulator instructions  271 . In  FIG. 2  the dashed arrows represent the flow of data between the client  102  and the emulator  107  over the network  160 . Alternatively, if the client  102  us contained within the emulator  107 , then the dashed arrows may indicate the flow of data over the internal system bus  150 ′. 
     Initially at  272 , the client  102  delivers game inputs  347  to the emulator  107  over the network  160 . By way of example, and not by way of limitation, game inputs  347  may be commands that instruct the emulator  107  where to begin in an emulation routine, or they may be commands that control the game play of a legacy game  106  that is being emulated by the emulator  107 . By way of example, and not by way of limitation, a game input  347  that instructs the emulator  107  where to begin in an emulation routine may be in the form of a previously generated snapshot  367 . By way of example, and not by way of limitation, game inputs  347  may be automatically generated by the client  102 , or they may be provided to the client  102  by an external source. By way of example, the game inputs  347  may be delivered to the emulator  107  all at the same time, or they may be delivered over a period of time. 
     By way of example, and not by way of limitation, game inputs  347  which control the game play may include commands that are generally used by a game player to advance the legacy game  106  from a first state  301  to a second state  302 . The first state  301  may be stored as a first snapshot, and the second state  302  may be the desired location for the generated snapshot. The game inputs  347  may be inputted by a controller  145 , or they may be automatically generated by the client  102 . Game inputs  347  of this nature may include, but are not limited to, inputs that cause a main character  340  in a legacy game  106  to move to a new position, swing a sword, select an item from a menu, or any other action that can take place during the game play of a legacy game  106 . Additionally, while game inputs advance the game play of the legacy game  106  from a first state  301  to a second state  302 , there may also be one or more intermediate states generated. Each of the intermediate states may optionally be recorded as a snapshot  367  as well. 
     The emulator  107  receives the game inputs  347  at  273  and then proceeds to emulate the legacy game  106  in accordance with the game inputs  347  at  274 . The emulation of the legacy game  106  causes the game to advance from a first state  301  to a second state  302 . By way of example, and not by way of limitation,  FIGS. 3A and 3B  are schematics of the emulation process according to certain aspects of the present disclosure.  FIG. 3A  depicts the client  102  delivering game inputs  347  to emulator  107 . The first state  301  of the legacy game  106  is displayed on the emulator&#39;s display unit  137 ′. By way of example, the legacy game  106  may alternatively be displayed on the client&#39;s display unit  137 . According to this example, the first state  301  includes the main character  340  standing on the left side of the mountain and the final boss  348  standing on the right side of the mountain. Additionally, the main character  340  has four lives  343  remaining and his health bar  344  is full. The time remaining to complete the level  346  is displayed on the top left corner of the display unit  137 ′ and the score  345  is displayed on the top right corner of the display unit  137 ′. 
     The emulator  107  begins emulating the legacy game  106  according to the game inputs  347  and the game is advanced to a second state  302  as shown in  FIG. 3B . In the second state  302 , the main character  340  has been moved to the right side of the mountain and is proximate to the final boss  348 . Also, the time remaining to complete the level  346  has been reduced to 1:00. Additionally, the health bar  344  indicates that the main character has limited health, and finally, there is only  1  life  343  remaining. These circumstances may be chosen by the client  102  because they present a very challenging scenario for a mini-game. As such, the client  102  may desire a snapshot to be taken at the second state  302 . 
     Once the emulator  107  has advanced the legacy game to the second state  302 , the client  102  will deliver a legacy game suspension request  357  to the emulator  107  at  275 . By way of example, and not by way of limitation, a legacy game suspension request  357  may be automatically generated by the client  102 . Alternatively, an operator of the client  102  may initiate the legacy game suspension request  357  with a keystroke from the user interface  138  or the controller  145 . The legacy game suspension request  357  is then received by the emulator at  276 . 
     The emulator  107  begins suspending the legacy game  106  in order to generate the snapshot. The emulator may suspend the legacy game  106  when every device being emulated by the emulator  107  is in a steady state. Waiting for all of the devices to come into a steady state may take as little as one frame to accomplish. Once the devices are in a steady state, the stored data may be platform independent since it does not rely on any specific set of hardware. This enables the snapshot to be used in conjunction with any target platform. The devices being emulated by the emulator  107  may be a steady state when there are no outstanding disk requests and all asynchronous activity has been synchronized. By way of example, and not by way of limitation, these conditions may be met when the emulator  107  has finished processing a basic block. Once the legacy game  106  has been suspended, the emulator  107  may proceed to record the state of every device that is being emulated at  277 . By way of example, and not by way of limitation, the emulator may record the state of the CPU, GPU, memory, register values, program counter value, programmable DMA state, buffered data for the DMA, audio chip state, and CD-ROM state. After the snapshot has been generated, the emulator  107  delivers the snapshot to the client at  278 , and the client receives the snapshot at  279 . 
     As shown in  FIG. 4A , a set of client instructions  470  may be implemented, e.g., by the client  102 . The client instructions  470  may be formed on a nontransitory computer readable medium such as the memory  132  or the mass storage device  134 . The client instructions  470  may also be part of the process control program  133 . The instructions include delivering legacy game input data  347  to the emulator at  472 . Thereafter the client  102  is instructed to deliver a legacy game suspension request to the emulator  107  at  475 . Thereafter, the client  102  is instructed to receive the snapshot  367  from the emulator  107  at  479 . 
     As shown in  FIG. 4B , a set of emulator instructions  471  may be implemented, e.g., by the emulator  107 . The emulation instructions  471  may be formed on a nontransitory computer readable medium such as the memory  132 ′ or the mass storage device  134 ′. The emulator instructions  471  may also be part of the process control program  133 ′. The instructions include receiving legacy game input data  347  from the client  102  at  473 . Thereafter the emulator  107  is instructed to begin emulating the selected legacy game  106  at  474 . While emulating the legacy game  106 , the emulator  107  is provided with instructions for receiving a legacy game suspension request  357  from the client  102  at  476 . Then at  477 , the emulator  107  is instructed to generate a snapshot  367  by recording the state of every device that is being emulated by the emulator  107 . The emulator  107  is then instructed to deliver the snapshot  367  to the client at  478 . 
     According to additional aspects of the present disclosure a replay of a mini-game may be made with only a thin input channel. Starting from a snapshot, a script could be written to only record the video and button presses that comprise the game inputs from a person playing the game. Therefore, the replay may be made by simply starting from the original snapshot and playing back the recorded game inputs. This replay method is possible with a deterministic emulator because the emulator will always go through the same sequence of operations when the same game inputs are provided to the emulator. 
     According to additional aspects of the present disclosure an emulated game may be pre-loaded in order to make the game begin as soon as a user initiates the game. Game players may access the games over a cloud. A server that hosts the titles may create a pre-build buffer so that the server is ready to go as soon as the user clicks “play”. 
     For example, a game may be loaded when a cursor is hovering over a certain game selection on a startup screen. The cloud system could base the decision to load the game on a prediction engine. The prediction engine may predict demand and initiate a sufficient number of host machines to meet the demand. The predictions may be based on what pages users are hovering over or historical data for the demand load for a given game title, e.g., certain game titles may historically have a big demand when school gets out. 
     For resuming from snapshots, the saved translated code, e.g., from the cache can be used to make the resume run faster. If this cannot be done, the code can be just-in-time (JIT) compiled. Some of this code may use information that is platform dependent. However, in such cases the platform-dependent information can be generated from platform-independent information in the snapshot data. For example, certain host-specific addresses need to be converted. For security, addresses are randomized when a new process starts. These addresses are platform-dependent, but the platform specific addresses could be generated by the host from the platform-independent snapshot data. 
     According to additional aspects of the present disclosure adaptive load balancing of software emulation of a hardware GPU may be implemented. Normally the rasterization of each line would be assigned to a different thread. However, each line could have a different number of pixels. The emulator doesn&#39;t always know how many pixels are needed for each scan line of the screen. So it&#39;s difficult to load balance the emulation of the rasterization. To overcome this, the GPU software emulation uses tile-based rasterization. In this type of rasterization small tiles of the screen can be rasterized as opposed to the entire screen. A thread can be assigned to the rasterization of each tile. Tile-based rasterization is efficient for threading because it allows for prediction of how many pixels need to be processed. The emulator can set how many pixels are in each tile and that makes it much easier to load balance since it is known a priori how many pixels are in each tile. It is easier to load balance the rasterization of the tiles than to load balance the lines. 
     According to additional aspects of the present disclosure in a cloud-based emulation of legacy games it is desirable to run threads independently while reducing latency because the games are going to be served over the cloud. It is desirable to reduce latency within each instance of a game and to reduce latencies when there are multiple instances of the same game on a given server. One source of latency is due to the fact the emulation of the GPU is not currently synchronized with emulation of everything else. Buffering a lot of commands before GPU starts can produce 15 to 30 ms of latency. In order to fix this, the emulator may save time by running things in parallel. For example, the emulator may run GPU emulation in parallel with the CPU emulator. Nominal GPU latency variance may be 20-30 ms. There is also an encoder latency, which has to start about 6 ms after GPU. However, 20 ms of latency may be saved by actually delaying the start of the GPU until after the CPU has processed the first few sets of frames. Certain types of CPU (e.g., the EE on the PS2) may finish the processing of these frames in 10-12 ms. By buffering the CPU output and waiting until it is done before sending the output to the GPU the GPU can be provide with a lot of work that can be run in parallel with the CPU&#39;s processing of the next group of frames. 
     While the above is a complete description of the preferred embodiment of the present invention, it is possible to use various alternatives, modifications and equivalents. Therefore, the scope of the present invention should be determined not with reference to the above description but should, instead, be determined with reference to the appended claims, along with their full scope of equivalents. Any feature described herein, whether preferred or not, may be combined with any other feature described herein, whether preferred or not. In the claims that follow, the indefinite article “A”, or “An” refers to a quantity of one or more of the item following the article, except where expressly stated otherwise. The appended claims are not to be interpreted as including means-plus-function limitations, unless such a limitation is explicitly recited in a given claim using the phrase “means for.”