Executing applications in remotely emulated computing devices

Disclosed are various embodiments that facilitate execution of applications through remotely emulated computing devices. An application is executed in a hosted environment. A video signal of the application is encoded into a media stream, which is sent to a client computing device. In various embodiments, access to data that is locally available to the client computing device is facilitated. In various embodiments, data related to the operation of the application is recorded.

BACKGROUND

There are many specialized forms of computing devices such as, for example, smartphones, electronic book readers, tablet computers, video game consoles, and so on. Because of the physical differences among such devices, applications may differ from device to device. Even within a device category such as smartphones, different models of smartphones may have different display and input features. Developers may have difficulties testing applications for such devices. Actual use by end users is an ideal way to identify application defects, but it is difficult to distribute applications and application updates to end users.

DETAILED DESCRIPTION

The present disclosure relates to executing applications on emulated computing devices over a network. Customers may wish to purchase applications for various computing devices over a network. In order to make an informed purchasing decision, a customer may wish to try the application before buying it. However, the customer may be using a different client device from the computing device that is intended to run the application. For example, the customer may be using a desktop computer and searching for applications for a mobile device such as a smartphone. Various embodiments of the present disclosure enable customers to use applications running remotely on emulated computing devices. Furthermore, in some embodiments, the customers may be able to buy the applications through an associated user interface. Various techniques related to executing applications remotely are described in U.S. patent application entitled “Remotely Emulating Computing Devices” filed on May 24, 2011 and assigned application Ser. No. 13/114,534, which is incorporated herein by reference in its entirety.

While using a remotely executed application, customers may wish to allow the application to access photos, documents, and/or other data stored on their local machine. Various embodiments of the present disclosure facilitate access to locally accessible data of a client by remotely executed applications. Additionally, various embodiments of the present disclosure are configured to allow a developer to deploy an application for testing by multiple users. Feedback and/or other operational data may be recorded for the developer. In the following discussion, a general description of the system and its components is provided, followed by a discussion of the operation of the same.

With reference toFIG. 1, shown is a networked environment100according to various embodiments. The networked environment100includes one or more computing devices103in data communication with one or more clients106, one or more developer clients107, and a networked storage system108by way of a network109. The network109includes, for example, the Internet, intranets, extranets, wide area networks (WANs), local area networks (LANs), wired networks, wireless networks, or other suitable networks, etc., or any combination of two or more such networks.

The computing device103may comprise, for example, a server computer or any other system providing computing capability. Alternatively, a plurality of computing devices103may be employed that are arranged, for example, in one or more server banks or computer banks or other arrangements. For example, a plurality of computing devices103together may comprise a cloud computing resource, a grid computing resource, and/or any other distributed computing arrangement. Such computing devices103may be located in a single installation or may be distributed among many different geographical locations. For purposes of convenience, the computing device103is referred to herein in the singular. Even though the computing device103is referred to in the singular, it is understood that a plurality of computing devices103may be employed in the various arrangements as described above.

Various applications and/or other functionality may be executed in the computing device103according to various embodiments. Also, various data is stored in a data store112that is accessible to the computing device103. The data store112may be representative of a plurality of data stores112as can be appreciated. The data stored in the data store112, for example, is associated with the operation of the various applications and/or functional entities described below.

The components executed on the computing device103, for example, include a emulation server application114, an electronic commerce application115, a plurality of wrappers117a. . .117N including a respective data access layer118a. . .118N, a plurality of applications119a. . .119N, a plurality of media encoders120a. . .120N, and other applications, services, processes, systems, engines, or functionality not discussed in detail herein. The emulation server application114may correspond to a type of application session server. The emulation server application114is executed to launch instances of applications119, which are executed within the wrappers117. The emulation server application114is also executed to obtain application input data122from the clients106and provide the application input data122to the respective wrapper117. In various embodiments, the application input data122may include data that is locally accessible to the client106(e.g., files, device data, etc.) which may be mirrored in the data store112for access by the application119.

The emulation server application114is also executed to send application output data123that is captured from the application119to the clients106. The emulation server application114may communicate with the client106over various protocols such as, for example, hypertext transfer protocol (HTTP), simple object access protocol (SOAP), representational state transfer (REST), real-time transport protocol (RTP), real time streaming protocol (RTSP), real time messaging protocol (RTMP), user datagram protocol (UDP), transmission control protocol (TCP), and/or other protocols for communicating data over the network109. The emulation server application114is configured to maintain state information124and input queues125associated with the executing applications119.

In various embodiments, the emulation server application114may be configured to generate a user interface using one or more network pages. The network pages may include the streaming video and/or audio generated by the emulated application119. In various embodiments, images of virtual input devices may be rendered in conjunction with the streaming video and/or audio. For example, a virtual keyboard for the emulated computing device may be included in the network page. Where the emulated computing device is a mobile computing device, an image of the mobile computing device may be included in the network page. The emulation server application114may facilitate interaction with the image of the mobile computing device in conjunction with the application119. Where the emulated computing device corresponds to a mobile computing device with a screen, the video captured from the application119may be surrounded by the image of the mobile computing device, as if the video were shown on the screen of the mobile computing device. However, in other embodiments, the video may be shown full screen, without being surrounded by the image of the mobile computing device.

The electronic commerce application115is executed in order to facilitate the online purchase of items, such as downloaded applications, from a merchant over the network109. The electronic commerce application115also performs various backend functions associated with the online presence of a merchant in order to facilitate the online purchase of items. For example, the electronic commerce application115may generate network pages or portions thereof that are provided to clients106for the purposes of selecting items for purchase, rental, download, lease, or other forms of consumption. In some embodiments, the electronic commerce application115is associated with a network site that includes an electronic marketplace in which multiple merchants participate.

The application119may correspond, for example, to a game or other types of applications. As non-limiting examples, the application119may correspond to a first-person shooter game, an action game, an adventure game, a party game, a role-playing game, a simulation game, a strategy game, a vehicle simulation game, and/or other types of games. The application119may be originally designed for execution in a general-purpose computing device or in a specialized device such as, for example, a smartphone, a video game console, a handheld game device, an arcade game device, etc. The applications119may also correspond to mobile phone applications, computer-aided design (CAD) applications, computer-aided manufacturing (CAM) applications, photo manipulation applications, video editing applications, office productivity applications, operating systems and associated applications, emulators for operating systems, architectures, and capabilities not present on a consumer device, and other applications and combinations of applications.

The application119may expect to access one or more resources of the device on which it is executed. Such resources may correspond to display devices, input devices, or other devices. In some cases, the application119may request exclusive access to one or more of the resources, whereby no other applications may have access to the particular resources.

The wrapper117corresponds to an application that provides a hosted environment for execution of the application119. In various embodiments, the wrapper117may be configured to provide a virtualized environment for the application119by virtualizing one or more of the resources that the application119expects to access. Such resources may include a keyboard, a mouse, a joystick, a video device, a sound device, a global positioning system (GPS) device, an accelerometer, a touchscreen, built-in buttons, a file system, a built-in battery, etc. In this way, the wrapper117is able to provide input commands and other data to the application119as if the wrapper117emulates a keyboard, a mouse, or another type of hardware device.

Further, the wrapper117is able to obtain a video signal and/or audio signal generated by the application119as if the wrapper117emulates a display device, an audio device, or another type of output device. Although many of the examples discussed herein relate to emulated computing devices with a display, emulated computing devices may also include, for example, set-top boxes, audio players, and/or other devices without an integrated display screen. The wrapper117is able to encode the video signal and/or audio signal by way of one or more media encoders120into a media stream. To this end, the wrapper117may include various types of media encoders120, such as, for example, Moving Pictures Experts Group (MPEG) encoders, H.264 encoders, Flash® video encoders, etc. Such media encoders120may be selected according to factors such as, for example, data reduction, encoding quality, latency, etc. In some embodiments, the wrappers117may communicate directly with the clients106to obtain the application input data122and to serve up the application output data123.

Different types of wrappers117may be provided for different applications119or classes of applications119. As non-limiting examples, different wrappers117may be provided for applications119using different application programming interfaces (APIs) such as OpenGL®, DirectX®, the Graphics Device Interface (GDI), and so on. Where the application119is configured for execution in a specialized device or another type of computing device, the wrapper117may include an emulation application that emulates the device. In some embodiments, the output of the application119may be captured by the wrapper117at a device level. For example, the application119may be executed in a physical game console, and the video output may be captured by way of a video graphics array (VGA) connection, a high-definition multimedia interface (HDMI) connection, a component video connection, a national television system committee (NTSC) television connection, and/or other connections. In other embodiments, the output of the application119may be captured by the wrapper117by intercepting system calls, such as drawing calls under a graphics API or other system calls.

The data access layer118of the wrapper117may be used to facilitate access to data that is locally available to the client106. Such data may include files such as photos, documents, spreadsheets, audio files, video files, and other types of files. Such data may also include data generated in a component of the client106such as GPS location data, accelerometer data, and/or other data. The data may be stored in a data store of the client106or in the networked storage system108.

The state information124that is maintained by the emulation server application114includes various data relating to application sessions that are currently active. For example, the state information124may track the users that are currently participating in the application session, status information associated with the users, security permissions associated with the application session (e.g., who can or cannot join), and so on. In some embodiments, some or all of the state information124may be discarded when an application session ends. The input queues125collect input commands from the application input data122for a given application119. The input commands may be reordered to a correct sequence and delays may be inserted between commands to ensure that they are interpreted correctly when presented to the corresponding application119.

The data stored in the data store112includes, for example, application code127, application feedback128, media encoders129, wrappers130, saved state data133, user data134including mirrored files135and mirrored device data136, device interfaces137, seed data138, developer data139, and potentially other data. The application code127corresponds to a library of different applications that are available to be launched as applications119. The application code127may correspond to executable code within the computing device103. Alternatively, the application code127may correspond to code that is executable within another type of device but is not executable within the computing device103. Such application code127may be referred to as “binaries,” read-only memory images (ROMs), and other terms. Application code127may be executed as multiple instances of the applications119for multiple application sessions. In one embodiment, the application code127may include source code for applications119. In some cases, the application code127may correspond to beta code, testing code, trial code, sample code, etc. that is less than a fully released production version.

The application feedback128may include operational data relating to the operation of the instances of the application119. Such operational data may include debugging information, defect information, usage information, crash information, core dumps, and/or other data that may be useful to developers, marketers, and so on. The application feedback128may also include responses to surveys by users of the applications119. Such surveys may be presented to the user in conjunction with the user interface served up by the emulation server application114.

The media encoders129correspond to the various types of media encoders120that may be employed in the computing device103. Some media encoders129may correspond to specific formats, such as, for example, H.264, MPEG-4, MPEG-2, and/or other formats. The wrappers130correspond to the executable code that implements the various types of wrappers117. The wrappers130are executable in the computing device103and may be executed as multiple instances of the wrappers117for multiple application sessions.

The saved state data133corresponds to game states that have been saved by the applications119. Because the applications119may be executed in a virtualized environment, the applications119may write state information to a virtual location, which is then mapped for storage in the data store112as the saved state data133. The saved state data133may correspond to data saved normally by the application119or may correspond to a memory image of the application119that may be resumed at any time.

The user data134includes various data related to the users of the applications119, such as, for example, types of computing devices associated with a user, security credentials, application preferences, billing information, a listing of other users that are permitted to join application sessions started by the user, and so on. In various embodiments, the user data134may also include copies of data that is locally accessible to the client106. Such data may include mirrored files135and mirrored device data136.

The mirrored files135may correspond to copies of images, videos, music, documents, etc. that are obtained from the client106or the networked storage system108on behalf of the client106. The mirrored device data136corresponds to GPS location data, accelerometer data, etc. generated in a component of the client106and maintained in the computing device103so that the applications119may have access. In one embodiment, the mirrored files135and/or the mirrored device data136may correspond to a temporary cache of the data. In one embodiment, the mirrored files135may correspond to a group of files stored in the client106(e.g., all files in a folder, all files on a drive, all files matching a filename mask, etc.), while in another embodiment, the mirrored files135may correspond to files that are accessed or predicted to be accessed by the application119. In one embodiment, the mirrored files135may be explicitly designated by the user.

The device interfaces137correspond to images, animations, code, hypertext markup language (HTML), extensible markup language (XML), cascading style sheets (CSS), and/or other data that may be used to generate a graphical representation of a virtualized computing device. It is noted that an application119that is executable on a particular computing device platform may be associated with a multitude of device interfaces137. As a non-limiting example, the Android® platform for smartphones may be supported by a multitude of different models of smartphones. Some of the models may have mini-keyboards with a touchscreen, while others may have merely a touchscreen with no physical mini-keyboard. The models may have different controls and casings. Therefore, different device interfaces137may be provided for different models of Android® smartphones.

In some embodiments, multiple different device interfaces137may be used in conjunction with one wrapper130. Where the emulated computing device does not include an integrated display, the device interface137may include a representation of an external display device showing video generated by the emulated computing device. Similarly, other non-integrated external devices that may connect to the emulated computing device (e.g., keyboards, mice, etc.) may be represented by the device interface137.

The seed data138corresponds to a library of default data that may be accessed by the application119. In some cases, the seed data138may be accessed in place of mirrored files135or mirrored device data136, for example, when the mirrored files135or mirrored device data136are unavailable. In one embodiment, the seed data138for an application119may be provided by a developer of the application119. In another embodiment, the seed data138may be generic data that may be used by many different applications119. As a non-limiting example, the seed data138may include a default GPS location that is to be used when no GPS location is available from the client106. As another non-limiting example, the seed data138may include a sample photo for use by a photo-editing application119.

The developer data139may include various information related to developers of the applications119. For example, a developer may have an account for uploading the application code127, configuring surveys and data collection for the application feedback128, uploading seed data138, and/or performing other tasks. The developer data139may include compilation instructions for source code of the application code127to be compiled into code that is executable within a particular wrapper117.

The client106is representative of a plurality of client computing devices that may be coupled to the network109. The clients106may be geographically diverse. The client106may comprise, for example, a processor-based system such as a computer system. Such a computer system may be embodied in the form of a desktop computer, a laptop computer, personal digital assistants, cellular telephones, smartphones, set-top boxes, televisions that execute applications and can access the network109, music players, web pads, tablet computer systems, game consoles, electronic book readers, or other devices with like capability.

The client106may include a display141. The display141may comprise, for example, one or more devices such as cathode ray tubes (CRTs), liquid crystal display (LCD) screens, gas plasma-based flat panel displays, LCD projectors, or other types of display devices, etc. The client106may include one or more input devices142. The input devices142may comprise, for example, devices such as keyboards, mice, joysticks, accelerometers, light guns, game controllers, touch pads, touch sticks, push buttons, optical sensors, microphones, webcams, and/or any other devices that can provide user input. Additionally, various input devices142may incorporate haptic technologies in order to provide feedback to the user.

Various data may be stored in the client106as file data143. The file data143may correspond to images, audio, video, documents, and/or other types of files. The file data143may be stored in any data store that is locally accessible to the client106. Such a data store may include, for example, hard drives, flash drives, memory sticks, floppy disks, compact discs, tape drives, network-attached storage, networked storage systems108, and/or other types of data stores.

The client106may be configured to execute various applications such as a client application145and/or other applications. The client application145is executed to allow a user to launch, join, play, or otherwise interact with an application119executed in the computing device103. To this end, the client application145is configured to capture input commands provided by the user through one or more of the input devices142and send this input over the network109to the computing device103as application input data122. The application input data122may also incorporate file data143and other data (e.g., GPS data, audio data, etc.) generated in a component of the client106for use by the application119. In various embodiments, the client application145corresponds to a browser application.

The client application145is also configured to obtain application output data123over the network109from the computing device103and render a screen148on the display141. To this end, the client application145may include one or more video and audio players to play out a media stream generated by an application119. In one embodiment, the client application145comprises a plug-in or other client-side code executed within a browser application. The client106may be configured to execute applications beyond the client application145such as, for example, browser applications, email applications, instant message applications, and/or other applications. In some embodiments, multiple clients106may be employed for one or more users to interact with the application119. As non-limiting examples, some clients106may be specialized in display output, while other clients106may be specialized in obtaining user input. It is noted that different clients106may be associated with different latency requirements which may affect a delay employed before providing input commands to the application119.

The developer client107is representative of a plurality of client computing devices, operated by developers, which may be coupled to the network109. The developer client107may comprise, for example, a processor-based system such as a computer system. Such a computer system may be embodied in the form of a desktop computer, a laptop computer, personal digital assistants, cellular telephones, smartphones, set-top boxes, music players, web pads, tablet computer systems, game consoles, electronic book readers, or other devices with like capability. The developer client107may include a display comprising, for example, one or more devices such as cathode ray tubes (CRTs), liquid crystal display (LCD) screens, gas plasma-based flat panel displays, LCD projectors, or other types of display devices, etc.

The developer client107may be configured to execute various applications such as a browser and/or other applications. The browser may be executed in a developer client107, for example, to access and render network pages, such as web pages, or other network content served up by the computing device103and/or other servers. The developer client107may be configured to execute other applications such as, for example, compilers, interpreters, software development applications, software debugging applications, email applications, instant message applications, and/or other applications.

A networked storage system108may be employed to provide access to files stored in a cloud computing resource comprising a networked plurality of computing devices. In some embodiments, the networked storage system108may correspond to a virtual file system where the actual data objects of the files are stored in a separate data storage system. A metadata service may be used to associate metadata with the files, thereby facilitating searches of the files using the metadata. By storing files in such a networked storage system108, users may access the files from any client106that has network109connectivity.

The networked storage system108maintains a file hierarchy of files and folders in networked or metadata-based file systems for users. To this end, the networked storage system108may support various file-related operations such as, for example, creating files, deleting files, modifying files, setting permissions for files, downloading files, and/or other operations. The networked storage system108may be configured to maintain a record of file activity, e.g., uploading of files, downloading of files, deletion of files, etc. The networked storage system108may be configured to serve up data addressed by URL via HTTP. Each of the users of the networked storage system108may have a separate user account with security credentials to prevent unauthorized use.

Turning now toFIG. 2A, shown is one example of a client106in the networked environment100(FIG. 1). The client106has a screen148(FIG. 1) that is rendering a user interface203generated by a client application145(FIG. 1). In one embodiment, the user interface203corresponds to a network page rendered in a browser executed in the client106. The user interface203may also be generated by a special-purpose application executed in the client106.

The user interface203corresponds to a video portion of a media stream generated by the emulation server application114(FIG. 1). The media stream is associated with an application119(FIG. 1) for photo-editing that is titled “PicSee Pro.” The client106may have various input buttons204which may generate various input commands that are sent to the emulation server application114as application input data122(FIG. 1). An audio portion of the media stream may be played out by the client application145in conjunction with the video portion of the media stream. Audio hardware controls205may be provided to control the audio playback from the client106and/or emulated computing device, e.g., increasing or decreasing the volume of the audio. In other examples, other hardware controls may be provided, e.g., record, fast forward, rewind, stop, previous track, next track, screen brightness, selected display device, etc.

The user interface203in the example ofFIG. 2Arelates to selecting a photo to be edited by the application119. A first group of options212allows a user to select default photos, for example, photos that are available to any user from the seed data138(FIG. 1). The data access layer118may make the seed data138appear as local data for the emulated computing device. A second group of options215allows a user to select photos from an account of the user in the networked storage system108(FIG. 1). Such photos, when selected, may be loaded automatically from the networked storage system108by the data access layer118with user account data obtained from the client106(e.g., security credentials and/or other data). A third group of options218allows the user to select photos that are stored in the client106as file data143(FIG. 1). Such photos, when selected, may be loaded automatically by the data access layer118from the client106over the network109(FIG. 1).

Data that is loaded remotely, either from the networked storage system108or from the client106, may be pre-loaded in advance or loaded on-demand by the data access layer118. In one example, the data access layer118may employ a complete mirror of the data in the data store112(FIG. 1). In another example, only the data that is selected in the application119is loaded over the network109. In yet another example, a user at the client106may explicitly designate to which locally accessible data the data access layer118is to have access. The user may grant or deny access for listings of files as well as the files themselves. This user selection may be performed as part of the user interface203generated by the application119or an auxiliary portion of the user interface203which is generated in part by the data access layer118or another application executed in the computing device103.

Continuing on toFIG. 2B, shown is another example of a user interface250rendered in a client application145(FIG. 1) executed in a client106(FIG. 1) in the networked environment100(FIG. 1). The user interface250shown inFIG. 2Bcorresponds to a network page generated by the electronic commerce application115(FIG. 1) overlaid with a user interface252that is a lightbox. In other examples, the user interface252may be presented in a full-screen format or another format. Portions of the user interface252may be generated by the electronic commerce application115, while other portions of the user interface252may be generated by the emulation server application114(FIG. 1).

The user interface252includes a device interface253that is a graphical representation of an emulated computing device (e.g., the client106shown inFIG. 2A), which is in this case a particular model of an Android® smartphone. In some examples, the device interface253may correspond to a generic version of the emulated computing device, e.g., a generic version of an Android® smartphone. Which model is used may be specified by the customer in the user interface250or the user interface252. In addition, a preference for a model of the emulated computing device may be stored in the user data134(FIG. 1).

The device interface253is generated by the emulation server application114from the applicable device interface137(FIG. 1). The device interface253includes virtual versions of input buttons204and/or other controls that are present in the actual device. Animations and/or images may be used to translate the physical experience into a virtual experience. Within the device interface253is a screen148(FIG. 1) showing the user interface203fromFIG. 2A, which corresponds to the video portion of a media stream captured by the wrapper117(FIG. 1) from an application119(FIG. 1) that corresponds to the selected application code127(FIG. 1). An audio portion of the media stream may be played out by the client application145in conjunction with the video portion of the media stream. Audio hardware controls205may be provided to control the audio playback from the emulated computing device, e.g., increasing or decreasing the volume of the audio. In other examples, other hardware controls may be provided, e.g., record, fast forward, rewind, stop, previous track, next track, screen brightness, selected display device, etc.

The user interface252may also include descriptive information260that may, for example, be an abbreviated version of a description in the underlying network page of the user interface250. The descriptive information260may include title, publisher, price, narrative description, and/or other information about the application code127. In this non-limiting example, the user interface252includes a timer263because the time spent testing the application code127for possible purchase is limited. In other examples, the time for use of the application code127by the customer may be unlimited.

The user interface252may include an orientation control266to rotate the orientation of the screen148. Activating such an orientation control266may cause a virtualized version of an accelerometer or other device to inform the application119that the orientation of the emulated computing device has been changed, e.g., from portrait to landscape and vice versa. The user interface252may also include one or more purchase components269for initiating a purchase of the application code127or adding the application code127to a shopping cart, wish list, gift registry, or other list of items as can be appreciated.

Next, with reference toFIGS. 1,2A, and2B, a general description of the operation of the various components of the networked environment100is provided. To begin, a customer at a client106may, for example, browse a network site served up by the electronic commerce application115. The customer may, for example, navigate a catalog taxonomy, execute a search query, select links, and/or perform other navigational functions. The customer may arrive at a detail network page such as that shown in the user interface200ofFIG. 2Aor another similar network page that features an application code127for download. The customer may specify one or more preferred models of an emulated computing device for which the customer is seeking application code127.

In addition to purchasing and/or downloading the application code127, the customer may be given an opportunity to use the application code127through a simulated interface. To this end, the customer at the client106sends a request to launch an application119to the emulation server application114, for example, by activating an emulation request tool. The emulation server application114obtains the corresponding application code127, media encoder129, wrapper130, and device interface137from the data store112. The emulation server application114then launches the application119in the corresponding wrapper117. The emulation server application114tracks the status of the application119within the state information124. In some cases, the video output of the application119may be configured to be presented full-screen or not otherwise surrounded by a simulated interface on the screen148of the client106, for example, as inFIG. 2A.

The wrapper117provides a hosted environment for execution of the application119. In some embodiments, the hosted environment may include a virtualized environment for the application119that virtualizes one or more resources of the computing device103. Such resources may include exclusive resources, i.e., resources for which the application119requests exclusive access. For example, the application119may request full screen access from a video device, which is an exclusive resource because normally only one application can have full screen access.

Furthermore, the wrapper117may virtualize input devices such as, for example, keyboards, mice, GPS devices, accelerometers, etc. which may not actually be present in the computing device103. The data generated by these components may be provided by the data access layer118. The wrapper117may also provide access to a virtual file system to the application119in connection with the data access layer118. In various embodiments, the wrapper117may correspond to a virtual machine and/or the wrapper117may be executed within a virtual machine.

The user at the client106enters input commands for the application119by use of the input devices142of the client106. As a non-limiting example, the user may depress a left mouse button. Accordingly, the client application145functions to encode the input command into a format that may be transmitted over the network109within the application input data122. The emulation server application114receives the input command, adds it to the input queue125for the application119, and ultimately passes it to the wrapper117. The wrapper117then provides a left mouse button depression to the application119by way of a virtualized mouse.

It is noted that variable latency characteristics of the network109may cause some input commands to be misinterpreted by the application119if the input commands are provided to the application119as soon as they are received. As a non-limiting example, two single clicks of a mouse button may be misinterpreted as a double click if the first single click is delayed by the network109by a greater amount than the second single click. Similarly, mouse clicks and mouse drags may be misinterpreted if the relative temporal relationship between certain input commands is not preserved.

As another non-limiting example, suppose that the application119corresponds to a game application within the fighting game genre, e.g., Mortal Kombat, Street Fighter, etc. The user at the client106may perform a series of complicated moves through a rapid sequence of input commands. If the input commands are subject to variable latency over the network109, the series of moves may be misinterpreted by the application119, thereby resulting in the character controlled by the user not performing the intended moves. For instance, two buttons may need to be pressed within a certain period of time to perform a punch move successfully. If the second input command is delayed by the network109, but the first input command is not, the move may be unsuccessful without any fault of the user. To remedy this, the emulation server application114may delay the first input command to preserve the relative temporal relationship between the first and second input commands. In so doing, the gesture, or command sequence, performed by the user is preserved for the application119.

Although the additional delay used may be predetermined, it may also be calculated based on the difference between the time period between the commands when generated in the client106and the time period between when the commands are received by the emulation server application114. The time period between the commands when generated in the client106may be determined by referring to timestamps in metadata associated with the commands. It is noted that the various fixed latencies in the system (e.g., video encoding delay, minimum network latency) might not adversely impact the functionality of the application119.

Various techniques related to adding latency and accommodating latency are described in U.S. patent application entitled “Adding Latency to Improve Perceived Performance” filed on Oct. 29, 2010 and assigned application Ser. No. 12/916,111, and in U.S. patent application entitled “Accommodating Latency in a Service-Based Application” filed on Sep. 17, 2010 and assigned application Ser. No. 12/885,296, both of which are incorporated herein by reference in their entirety.

In some cases, the delay in providing the input command to the application119may depend at least in part on a video frame region that is associated with the input command. As a non-limiting example, with an application119that is a game, it may be important to delay an input command relating to game play to preserve the meaning of the input command. However, the game screen148may also include a chat window, and the input command may be text to be sent to another player in the game by way of the chat window. If the input command relates to the chat window, the wrapper117may be configured to provide the input command to the application119without additional delay. That is, it may be preferable to send the text to the other user as soon as possible. Accordingly, the delay may depend on whether the input command is related to the region of the screen148that is the chat window. Also, it is understood that an application119may have multiple modes, where one mode is associated with a delay in providing input commands while another is not.

It may also be important to ensure that the input command is synchronized with the video frame presented to the user on the screen148when the user generated the input command in the client106. As a non-limiting example, because of the various latencies of the system, a fireball intended to be thrown by the character controlled by the user in a fighting game may be thrown later than intended. In some cases, this may not matter. For example, the user may merely want to throw the fireball, and a short delay in throwing the fireball may be perfectly acceptable to the user. However, for more precision, the application119may support an API that allows the wrapper117to associate a frame number or other temporal identifier with an input command. Therefore, the application119can know precisely when the input command was performed and react accordingly. The frame number or other temporal identifier may be sent to the emulation server application114by the client application145as metadata for the input command.

In some embodiments, different input commands may be presented to the application119from those that were generated by a client106. As a non-limiting example, if a user sends a mouse down command and the client application145loses focus, the wrapper117may be configured to send a mouse down command followed by a mouse up command. In various embodiments, the input commands may be relayed to the wrapper117as soon as possible, or the input commands may be queued by the wrapper117in the input queue125and relayed to the application119sequentially from the queue according to another approach.

Meanwhile, the graphical output of the application119is captured by the wrapper117and encoded into a media stream. Additionally, the audio output of the application119may be captured and multiplexed into the media stream. The graphical output and/or audio output of the application119may be captured by hardware devices of the computing device103in some embodiments. The media stream is transmitted by the emulation server application114to the client106over the network109as the application output data123. The client application145obtains the application output data123and renders a screen148on the display141in a user interface250. The screen148may be surrounded by a device interface253generated from the corresponding device interface137that facilitates input and output for the emulated computing device.

In some embodiments, a customer may start an application119at one client106and continue the application119at another client106. Furthermore, multiple users at diverse locations may participate in an application119. As a non-limiting example, an application119may have been developed to be executed in one device with multiple controllers. Accordingly, the wrapper117may be configured to map input commands from one client106to a first virtual controller and input commands from another client106to a second virtual controller. As another non-limiting example, an application119may have been developed to be executed in one device, where one side of the keyboard controls the first player and the other side of the keyboard controls the second player. Accordingly, the wrapper117may be configured to map input commands from one client106to keys on one side of a virtual keyboard and input commands from another client106to keys on another side of the virtual keyboard.

Various embodiments enable input generated through one type of input device142in a client106to be transformed by the wrapper117into input commands provided to the application119through an entirely different type of virtual input device. As a non-limiting example, input generated by an accelerometer in the client106may be translated by the wrapper117into input provided through a virtual mouse. Thus, completely different kinds of input devices142may be used in the application119that may not have been contemplated when the application119was implemented.

Where the input devices142incorporate haptic technologies and devices, force feedback may be provided to the input devices142within the application output data123. As a non-limiting example, a simulated automobile steering wheel may be programmed by force feedback to give the user a feel of the road. As a user makes a turn or accelerates, the steering wheel may resist the turn or slip out of control. As another non-limiting example, the temperature of the input device142may be configured to change according to force feedback. In one embodiment, force feedback generated from the application input data122of one client106may be included in the application output data123sent to another client106.

Because the client106is decoupled from the hardware requirements of the application119, the application119may be used remotely through a diverse variety of clients106that are capable of streaming video with acceptable bandwidth and latency over a network109. For example, a game application119may be played on a client106that is a smartphone. Thus, the client106need not include expensive graphics hardware to perform the complex three-dimensional rendering that may be necessary to execute the application119. By contrast, the hardware of the computing device103may be upgraded as needed to meet the hardware requirements of the latest and most computationally intensive applications119. In various embodiments, the video signal in the media stream sent by the emulation server application114may be scaled according to the bitrate and/or other characteristics of the connection between the computing device103and the client106over the network109.

Having thoroughly tested the application code127with an interface that resembles their own computing device, customers are well prepared to purchase the application code127. At any time, the customer may choose to purchase, lease, and/or download the application code127by selecting various purchase components206,265. Additionally, if a customer does not own the particular version of the emulated computing device, the customer may decide to purchase a physical version of the emulated computing device after interacting with the virtual version.

In some embodiments, the data access layer118may provide access, for the application119, to data that is ordinarily only locally available to the client106. Such data may be stored as file data143in a data store of the client106or in an account of the user in a networked storage system108. Such data may include data generated by components of the client106, including GPS devices, accelerometers, etc. The data may be accessed by the data access layer118from the client106on demand over the network109, or, periodically, the data may be mirrored in the mirrored files135and mirrored device data136. A library of seed data138may be available to provide data for use by applications119in addition to the data of the user or when the data of the user is unavailable.

As a non-limiting example, a user may be testing an application119that suggests restaurants according to the current location of the user. The application119may make a request for a location from a virtualized GPS device. The request for the location may be intercepted by the data access layer118. If the user is using a client106with a GPS device, the data access layer118may request and obtain this location from the client106over the network. In one embodiment, the GPS location may be mirrored automatically from the client106in the mirrored device data136. If the client106does not have a GPS device or if the user has not enabled access to the GPS device, the data access layer118may be configured to instead pull a default location from the seed data138.

As another non-limiting example, a user may be using a photo editor application119. When the user requests to open a photo for editing, a dialog may appear to select a file. The data access layer118may virtualize a local data store for the application119which provides access to the file data143on the client. Alternatively, the data access layer118may provide access to the mirrored files135which correspond to a cache or mirrored copy of at least a portion of the file data143. The data access layer118may obtain data such as security credentials from the client106and then access the networked storage system108to obtain files of the users. Additionally, and if the files of the user are unavailable, the data access layer118may allow access to the seed data138, which may include sample photos. Thus, a user may be able to test the photo editing functionality even when the local files are unavailable.

Various embodiments of the present disclosure may offer many features that are useful to developers of applications119. By deploying applications119through the emulation server application114, the developers may have the opportunity to test their applications119under a variety of emulated platforms by a multitude of different users. Performance of the application119may be evaluated for the different emulated devices.

A developer at a developer client107may upload a version of the application code127to the emulation server application114through a web-based interface, file transfer protocol (FTP), email, or other interface. The emulation server application114may be configured to obtain the application code127from the developer and deploy it for use in the computing device103as instances of applications119. In addition, updates to the application code127may be similarly deployed and pushed out to users through the applications119. This avoids having to wait for users to manually perform updates in the clients106.

The emulation server application114in conjunction with the various wrappers117may be configured to collect operational data about the applications119for developers. The operational data may be stored in the application feedback128and later accessed by developer clients107. The operational data may include usage analytics (e.g., a duration of use by a user of instances of the application119, etc.), when the application119has crashed and debugging information, and feedback from the users. In one embodiment, the emulation server application114may be configured to present a survey to the user through the client application145. The survey responses may be collected and stored in the application feedback128. The emulation server application114may also be configured to identify potential customers to whom to promote a beta test of an application119for a developer.

Referring next toFIGS. 3A and 3B, shown is a flowchart that provides one example of the operation of a portion of the emulation server application114according to various embodiments. It is understood that the flowchart ofFIGS. 3A and 3Bprovides merely an example of the many different types of functional arrangements that may be employed to implement the operation of the portion of the emulation server application114as described herein. As an alternative, the flowchart ofFIGS. 3A and 3Bmay be viewed as depicting an example of steps of a method implemented in the computing device103(FIG. 1) according to one or more embodiments.

Beginning with box301inFIG. 3A, the emulation server application114obtains application code127(FIG. 1) from a developer client107(FIG. 1) and deploys the application code127within the system. This may include compiling or processing the application code127for execution within multiple emulated platforms. In box302, the emulation server application114may identify potential customers who may be interested in the application119(FIG. 1) and may promote the application119to the potential customers. In one example, a developer may manually specify a set of users in the developer data139(FIG. 1). In another example, a developer may specify rules for automatically identifying users in the developer data139. Potential customers may be identified automatically, for example, according to purchase history data, browse history data, review history data, and/or other data. In one scenario, review history data is employed to identify users who are classified as good or thoughtful reviewers.

In box303, the emulation server application114obtains a request from a client106(FIG. 1) to begin an application emulation session for a particular application code127. In box306, the emulation server application114determines a preferred model of the emulated computing device either specified by the customer at the client106or otherwise associated with the customer. In box309, the emulation server application114determines whether there is no preferred model indicated. If there is no preferred model indicated, or if a device interface137(FIG. 1) for the preferred model is unavailable, the emulation server application114moves to box310and uses a generic model of the emulated computing device for executing the application code127and providing a device interface253(FIG. 2B). The emulation server application114then continues to box312. If a preferred model is indicated, the emulation server application114also continues to box312and uses the preferred model.

In box312, the emulation server application114generates the user interface252(FIG. 2B), which may include a customized device interface253, and sends the corresponding data to the client106. In box315, the emulation server application114loads the application code127and executes an instance of an application119(FIG. 1) within a wrapper117(FIG. 1) that corresponds to the emulated computing device. In box318, the emulation server application114captures the media stream from the wrapper117, which may be encoded by the media encoder120(FIG. 1) and/or other encoders. The emulation server application114sends the media stream to the client106as application output data123(FIG. 1) for rendering in the customized device interface253. In some embodiments, the device interface253may be embedded into the video signal of the media stream.

In box321, the emulation server application114obtains input commands from the client106in application input data122(FIG. 1). The emulation server application114provides the input commands to the application119executing in the wrapper117. In box322ofFIG. 3B, the emulation server application114determines whether application119is requesting access to data. If the application119is not requesting access to data, the emulation server application114proceeds to box323. Otherwise, if the application119is requesting access to data, the emulation server application114instead proceeds from box322to box324. In box324, the emulation server application114determines whether data that is locally available to the client106is used by the application119. If such data is not to be used or is unavailable, the emulation server application114in box327accesses default data for the application119in the seed data138(FIG. 1) and provides the default data to application119via the data access layer118(FIG. 1). The emulation server application114proceeds to box323.

If the emulation server application114determines instead that locally available data from the client106is to be used and is available, the emulation server application114proceeds from box324to box333and accesses the data locally available to the client106and provides the data to the application119through the data access layer118. Such data may be maintained in a local mirror or cache such as in mirrored files135(FIG. 1) or in mirrored device data136(FIG. 1). Alternatively, such data may be obtained from the client106or from a networked storage system108(FIG. 1) on behalf of the client106. The emulation server application114continues to box323.

In box323, the emulation server application114records operational data relating to the application119. Such data may be recorded in application feedback128(FIG. 1). Such data may be useful for usage analytics, marketing, and quality assurance purposes. In box336, the emulation server application114determines whether the session is over. As a non-limiting example, the session may be associated with a timer263(FIG. 2B), and the timer263may have expired. As another non-limiting example, the session may have been explicitly ended through user interaction at the client106. As yet another non-limiting example, the session may have been implicitly ended through user inaction at the client106.

If the session is not over, the emulation server application114returns to box318inFIG. 3Aand continues to capture the media stream from the application119. If the session is over, the emulation server application114proceeds to box339and ends the session and releases resources consumed by the session. In box342, the emulation server application114determines whether a survey is to be collected from the client106. If so, the emulation server application114moves to box345and collects and stores the survey response in the application feedback128. The emulation server application114proceeds to box348. If no survey is collected, the emulation server application114also moves to box348.

In box348, the emulation server application114provides the application feedback128to the developer at the developer client107. Such data may be provided in the form of an email message, bulk data upload, network page, and/or other formats. In box349, the emulation server application114determines whether the application code127is to be updated. If the application code127is to be updated, the emulation server application114proceeds to box351. In box351, the emulation server application114obtains and deploys an updated version of the application code127from the developer client107. Thereafter, the emulation server application114may return to box302ofFIG. 3Aand promote the update to current and/or potential users. Any future instance of an application119will be configured to use the updated version of the application code127, if applicable to the emulated computing device. If the emulation server application114instead determines that the application code127is not to be updated, the portion of the emulation server application114ends.

Moving on toFIG. 4, shown is a flowchart that provides one example of the operation of a portion of the electronic commerce application115according to various embodiments. It is understood that the flowchart ofFIG. 4provides merely an example of the many different types of functional arrangements that may be employed to implement the operation of the portion of the electronic commerce application115as described herein. As an alternative, the flowchart ofFIG. 4may be viewed as depicting an example of steps of a method implemented in the computing device103(FIG. 1) according to one or more embodiments.

Beginning with box403, the electronic commerce application115obtains a request from a client106(FIG. 1) for a detail network page regarding an application code127(FIG. 1). In box406, the electronic commerce application115generates the network page and sends the network page to the client106. In box409, the electronic commerce application115obtains a request from the client106to launch an application emulation session. In box412, the electronic commerce application115generates a user interface252(FIG. 2B) or a portion of the user interface252for the application emulation session in conjunction with the emulation server application114and sends the corresponding data to the client106for rendering on a display141(FIG. 1).

In box415, the electronic commerce application115obtains a request to purchase the application code127. In box418, the electronic commerce application115begins an ordering process to complete the order for the application code127by the customer, which may involve various tasks in an order pipeline. Thereafter, the portion of the electronic commerce application115ends.

With reference toFIG. 5, shown is a schematic block diagram of the computing device103according to an embodiment of the present disclosure. The computing device103includes at least one processor circuit, for example, having a processor503, a memory506, and, optionally, one or more graphics devices507, all of which are coupled to a local interface509. To this end, the computing device103may comprise, for example, at least one server computer or like device. The local interface509may comprise, for example, a data bus with an accompanying address/control bus or other bus structure as can be appreciated. The graphics devices507may correspond to high-performance graphics hardware, including one or more graphics processors512. The graphics devices507are configured to render graphics corresponding to the applications119executed in the computing device103.

Stored in the memory506are both data and several components that are executable by the processor503. In particular, stored in the memory506and executable by the processor503are the emulation server application114, the electronic commerce application115, the wrappers117, the data access layers118the applications119, the media encoders120, and potentially other applications. Also stored in the memory506may be a data store112and other data. In addition, an operating system may be stored in the memory506and executable by the processor503.

Also, the processor503may represent multiple processors503and the memory506may represent multiple memories506that operate in parallel processing circuits, respectively. In such a case, the local interface509may be an appropriate network109(FIG. 1) that facilitates communication between any two of the multiple processors503, between any processor503and any of the memories506, or between any two of the memories506, etc. The local interface509may comprise additional systems designed to coordinate this communication, including, for example, performing load balancing. The processor503may be of electrical or of some other available construction.