Patent Description:
In some instances, the application services are implemented in a streaming application environment capable of supporting one or more virtual streaming applications for use by one or more remote devices. Further, in some instances, virtual streaming application-generated content may be output in the form of rendering framebuffers that are captured, encoded and streamed to a remote device for video playback. Then, on the remote device, user-control information such as gesture or touch events and other inputs may be captured, encoded and uploaded to the streaming application environment and injected into a virtual streaming application to render its content accordingly.

Synchronizing the injection of user-control information and other inputs with the rendering of application content on a remote device, however, can be problematic even beyond the synchronization challenges presented by local applications on a remote device. For example, where a computing device such as a mobile device includes a touchscreen capable of being manipulated by a user's fingers, ensuring that user input such as a finger swipe or a scrolling gesture is tracked by rendered content can be challenging even for a locally-installed application.

As an example, many gestures directed to a touchscreen are handled using multiple events. In many instances, a gesture such as a finger swipe is generally represented by a "touch down" event that is signaled when the user first places a finger on the touchscreen, a "touch up" event that is signaled when the user lifts the finger off the touchscreen, and numerous "touch move" events that track the movement of the finger while touching the touchscreen. In many computing devices, events may be generated on a touchscreen computing device at a rate that matches a frame refresh rate for the device, i.e., the rate at which rendered content in the form of frames is updated on a touchscreen. Further, in many computing devices, frames may be updated at about <NUM> or more, and as such, a simple finger swipe may result in the generation of tens or hundreds of individual but related events. Moreover, it has been found that when some types of user input events are not appropriately synchronized with the rendered content frames during which the input events are generated, a poor user experience may result. As but one example, slowly scrolling through content by dragging a user's finger across a touchscreen can result in jerkiness when events associated with the movement are mapped to the wrong frames.

Where a computing device is interacting with a virtual streaming application in a streaming application environment over a packet-switched network such as the Internet, however, the aforementioned synchronization challenge is far greater. Due to the practical limitations of the Internet and other packet-switched networks, packets containing user input and/or rendered content may be delayed or lost entirely, and may arrive out of order. Furthermore, where network connectivity is compromised as may be the case in many mobile and/or vehicle applications, the risk of packet loss or delay is greater. While protocols such as Transmission Control Protocol (TCP) allow for packet retransmission in the event of lost packets, waiting for all packets to be received in order to ensure that all input data and rendered content is received and processed may introduce unacceptable delays and detract from the user experience. <CIT> describes techniques that provide for the transmission of a multicast stream that can resend frames that were not received using the partial state block acknowledgement mechanism. In an example embodiment, an access point reserves a channel for a transmission opportunity of a length to allow the sending of multicast data, block acknowledgement request, and receipt of a block acknowledgement. Optionally, the transmission opportunity may be of a length to enable packets that re resent to be sent during the transmission opportunity. Alternatively the access point may employ a backoff between TXOPs. The access point does not send any frames from other queues during the transmission opportunity.

In one aspect, a method is performed in a device including one or more processors and a display, which may include receiving an output display representation from a streaming application environment coupled to the device over a packet-switched network, the output display representation generated by a virtual streaming application resident in the streaming application environment, rendering a plurality of frames of the received output display representation on the display, in response to inputs generated during rendering of the plurality of frames, issuing input events for communication over the packet-switched network to the streaming application environment, and in response to a determination that no inputs are generated during a frame among the plurality of frames, reissuing a trailing event issued in a prior frame for communication over the packet-switched network to the streaming application environment. The method is further defined in Claim <NUM>.

Some implementations also include a system including one or more processors and memory operably coupled with the one or more processors, where the memory stores instructions that, in response to execution of the instructions by the one or more processors, cause the one or more processors to perform any of the above methods. Some implementations also include at least one non-transitory computer-readable medium including instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform any of the above methods.

Application streaming allows users in some instances to run applications on a computing device such as a mobile phone without having to download and install the applications on the computing device itself. In some instances, for example, a streaming application environment, e.g., as may be implemented on one or more servers in a cloud computing system that is accessible to a computing device over a network such as the Internet. Applications may be run as virtual streaming applications within the streaming application environment, e.g., by running the applications in virtual machines or in some instances as virtual computing devices that effectively emulate physical computing devices and/or the operating systems of such devices. In addition, a streaming application environment may include functionality to interface virtual streaming applications with a computing device, including an ability to receive input generated at the computing device to control the virtual streaming application as well as an ability to stream an output display representation of rendered content generated by the virtual streaming application.

<FIG>, for example, illustrates an example streaming application environment (SAE) <NUM> capable of hosting and interfacing one or more virtual streaming applications (VSAs) <NUM> with one or more remote devices <NUM> over one or more networks <NUM>. As noted above, streaming application environment <NUM> may be implemented, for example, as a service of a cloud computing system, and thus implemented using one or more hardware servers of a cloud computing system. Device(s) <NUM>, which are remote from the standpoint that they are coupled to but distinct from streaming application environment <NUM>, may be implemented using practically any type of computing device capable of accessing a cloud computing system, although the discussion hereinafter primarily focuses on mobile devices such as mobile phones, tablet computers, vehicle computers, wearable computers, etc. Communication between devices <NUM> and streaming application environment <NUM>, as noted above, may be conducted over one or more networks <NUM>, which in some implementations includes at least one packet-switched network such as a local area network (LAN), wide area network (WAN), the Internet, etc., such that communications between devices <NUM> and application streaming environment <NUM> utilize packets to communicate data therebetween.

Consistent with remote devices being mobile devices in some implementations, <FIG> illustrates an example remote device <NUM> including a touch panel <NUM> capable of receiving touch inputs from a user and a display <NUM> capable of outputting video or graphical content to a user, which in some implementations may be combined into a single touch-sensitive screen, or touchscreen. Device <NUM> may also include additional sensors and user input components <NUM> and/or additional output components <NUM>, generally depending upon the application of the device. Various sensors and components that may be used in different implementations include accelerometers, GPS sensors, gyroscopes, hard buttons, keyboards, pointing devices, microphones, imaging devices, speakers, indicators, and haptic feedback devices, among others.

Other variations and modifications to the environment illustrated in <FIG> will be appreciated, and as such, the invention is not limited to the particular environment disclosed herein.

The implementations discussed hereinafter address, in part, the synchronization of inputs with rendered visual content in a streaming application environment. For the purpose of simplifying the discussion, the hereinafter-described implementations will focus in particular on the synchronization of user gestures with rendered visual content, and in particular user input directed to a touchscreen. It will be appreciated, however, that the principles disclosed herein may be applied to other types of inputs, including other user inputs such as user manipulation of a mouse, trackball or other pointing device, user gestures captured by an imaging device or a handheld or wearable motion sensor, as well as additional inputs such as sensor inputs. Thus, the invention is also not limited to the particular inputs disclosed by way of example herein.

A common user interaction with a touchscreen device is a single or multi-finger gesture to scroll and/or zoom displayed content. On a physical touchscreen device, touchscreen hardware may trigger a continuous stream of touch events that describe the locations of the user's fingers as they move across the touchscreen. These events are generally processed and interpreted by the device to make corresponding modifications to the displayed content on the device. Swiping a finger up or down in a region of a display dedicated to scrollable content may therefore lead to a corresponding scrolling of the content to follow the motion of the finger. A seamless user experience occurs when the movement of the content precisely follows the motion of the finger, and it has been found that driving the rate (frequency) of the touch events and the rendering framerate by the same clock on a physical device obtains acceptable results. Consequently, with a nominal clock rate of <NUM>, finger movements across short intervals of <NUM>/<NUM> seconds generally are mapped to consecutive frames rendered at <NUM> fps. Undesirable results, however, can occur when the one-on-one alignment between a <NUM>/<NUM>-second movement to a rendered frame is lost, such that if two <NUM>/<NUM>-second movements are mapped to a single frame while the next frame does not see any movement, jerky scrolling can occur.

With application streaming, on the other hand, the interaction between touch events and frame rendering works similarly, except that there are multiple components and a network between a device that generates input and displays rendered content, and a virtual streaming application that processes that input to generate the rendered content displayed on the device.

In an idealistic situation where (<NUM>) the network has no packet loss and no delay jitter, i.e., the network round-trip-time (RTT) is constant, and (<NUM>) rendering framerate in the virtual streaming application, the encoding framerate in the streaming application environment, and the frame refresh rate in the remote device are the same, e.g. all at exactly <NUM>, scrolling via application streaming would be as smooth as scrolling in a local application on a physical device other than having an extra delay introduced by the network RTT and the processing time in the additional components. However, in reality some touch events can get lost in the network, and because of network jitter each touch event can take a different amount of time to travel from the remote device to the streaming application environment, and can even arrive out of order. In addition, some remote devices do not always use the same, consistent frame refresh rate (e.g., varying slightly from <NUM>, and in some instances reducing to a lower frame refresh rate when entering power-saving mode), and components in an streaming application environment may include delays that cause variations in the frame refresh rate used for a virtual streaming application. As such, there is generally an increased risk that multiple touch events can be mapped into a single frame, while no events may be mapped into adjacent frames, leading to jerky movement.

In some implementations, therefore, a number of modules or components may be implemented within a remote device <NUM> and a streaming application environment <NUM> to assist with synchronization of input with rendered content with application streaming. In particular, in some implementations an operating system <NUM> of a remote device <NUM> may include a control streamer component <NUM> and a video player component <NUM> that are respectively in communication with a control player component <NUM> and a video streaming component, or video streamer, <NUM> implemented within streaming application environment <NUM>.

Control streamer component <NUM> is used to stream both input events and frame refresh (FR) events to control player component <NUM>, the former of which are streamed to an input handler component <NUM> in control player component <NUM> and the latter of which are streamed to a frame refresh (FR) handler component <NUM> in control player component <NUM>. Input events are events generated by remote device <NUM> to control virtual streaming application <NUM>, and may include, for example, gesture events such as touch events, although it will be appreciated that events generated in remote device <NUM> in response to other inputs, e.g., other user input and/or sensor data, may also be supplied to virtual streaming application <NUM> in the form of input events. FR events are events generated in response to frame refreshes in remote device <NUM>, e.g., similar to vsync events generated in some physical computing devices in connection with rendering frames on such devices.

Input handler component <NUM>, as will become more apparent below, injects, or "plays back," input events received by control player component <NUM> into virtual streaming application <NUM>. FR handler component <NUM> generates refresh events within streaming application environment <NUM> in response to FR events received by control player component <NUM>, including virtual frame refresh (VFR) events that are provided to virtual streaming application <NUM> and to input handler component <NUM> to synchronize input events with their corresponding frames on the remote device, as well as streaming component frame refresh (SCFR) events that are provided to video streaming component <NUM> to synchronize the encoding and streaming of rendered content output by virtual streaming application <NUM>. Rendered content, in some implementations, is provided to video streaming component <NUM> in the form of virtual frames stored in a frame buffer populated by virtual streaming application <NUM>, and as such, the VFR events generated by FR handler component <NUM> are associated with virtual frames of virtual streaming application <NUM> in a similar manner to the way in which FR events generated by remote device <NUM> are associated with rendered frames in remote device <NUM>.

Video streaming component <NUM> may process the virtual frames by encoding the virtual frames and communicating the encoded virtual frames to video player component <NUM> of remote device <NUM>, which in turn decodes the encoded virtual frames and displays the rendered content on the display <NUM> of the remote device <NUM>. The rendered content may be considered in some implementations to be an output display representation, i.e., data capable of being used to generate a visual depiction of the rendered content on a remote device. It will also be appreciated that the output display representation may be encoded, unencoded, encrypted, unencrypted, or otherwise defined using various formats, so the invention is not limited to any particular format of data so long as the data may be used to generate a visual depiction of rendered content on a display. In some implementations, for example, the output display representation may be implemented as an encoded digital video stream.

In some implementations, control streamer component <NUM> may implement listener and/or callback routines to handle system events in operating system <NUM>. For example, some input events may be generated in response to events generated in response to various types of touch events detected by operating system <NUM>. Likewise, in some implementations, FR events may be generated in response to invocation of a frame rendering routine that renders a new frame for display on remote device <NUM>. FR events may be generated, for example, periodically at a vsync rate of the remote device, which may be around <NUM> in some implementations, and both FR events and input events may include associated timestamps representing remote device system time when the corresponding system event occurred. Moreover, in some implementations, each frame rendering is configured to reflect the content updates triggered by the input events, if any, with timestamps that fall within the current frame or vsync interval, i.e., between the current FR event timestamp and the previous one. In some implementations, input and FR events may also be serialized and/or encrypted prior to communication over network <NUM>.

<FIG>, for example, respectively illustrate sequences of operations that may be implemented in control streamer component <NUM> of remote device <NUM> to respectively generate input and frame refresh (FR) events for communication to streaming application environment <NUM>. As shown in <FIG>, a sequence of operations <NUM> may be used to handle input events generated within remote device <NUM>, e.g., gesture or touch events. In block <NUM>, an input event message is generated, including a remote device system time timestamp, as well as any related event parameters, e.g., touch coordinates, identifiers of related input events (e.g., for touch move or touch up events associated with a particular gesture, the timestamp of the initial touch down event that started the gesture), etc. Block <NUM> then sends the generated event message to the streaming application environment <NUM>, and sequence of operations <NUM> is complete. Similarly, as illustrated by sequence of operations <NUM> in <FIG>, for frame refresh events, an FR event message, generally including a remote device system time timestamp, may be generated in block <NUM>, and then sent to the streaming application environment <NUM> in block <NUM>. It will be appreciated that in both cases, the particular format of the messages and then manner in which the event data is encoded into packets may vary in different implementations based in part on the protocol used to communicate between remote device <NUM> and streaming application environment <NUM>.

As noted above, in some implementations streaming application environment <NUM> is configured to synchronize input from the remote device <NUM> with rendered content generated by a virtual streaming application <NUM>. In particular, in some implementations the streaming application environment <NUM> may receive frame refresh events from a remote device <NUM> that are associated with particular frames on the remote device <NUM>, receive input events from the remote device <NUM> that are associated with inputs generated during frames on the remote device <NUM>, generate virtual frames for a virtual streaming application <NUM> corresponding to the frames with which the received frame refresh events are associated, and then control controlling injection of the received input events into the virtual streaming application <NUM> to arrange the received input events within the virtual frames corresponding to the frames with which the received input events are associated.

In some implementations, for example, a frame refresh (FR) handler component <NUM> of control player component <NUM> may effectively track a frame refresh rate of the remote device <NUM> based upon FR events communicated by control streamer component <NUM> of the remote device <NUM> and use the tracked frame refresh rate to drive the injection of input events into virtual streaming application <NUM>, and in some instances, to drive the streaming of content from video streaming component <NUM>. In some implementations, for example, the FR handler component <NUM> may generate VFR events at a virtual frame refresh rate that substantially tracks the frame refresh rate of the remote device. In addition, in some implementations, an input handler component <NUM> of control player component <NUM> may implement a queue that effectively operates as a jitter buffer to synchronize the injection of input events into the virtual streaming application <NUM>.

To track the frame refresh rate, FR handler component <NUM> in some implementations may schedule VFR events in the following manner. For the purposes of this explanation, A(t) may be used to denote the streaming application environment system time when an FR event with a timestamp t is received by the streaming application environment <NUM>. Issuance of a VFR event corresponding to the received FR event may then be scheduled for a streaming application environment system time denoted as V(t) using equation (<NUM>) below: <MAT> where D denotes a maximum playback delay (e.g., set to a value such as <NUM> in some implementation) and t' is a reference timestamp selected such that A(t') - t' is not greater than any other A(t) - t for all FR events received from the remote device to date. Put another way, t' may be considered to correspond to the FR event that travelled from the remote device to the streaming application environment <NUM> with the shortest transmission time, such that A(t') - t' is effectively the relative minimum one-way network delay, which is relative because the streaming application environment system clock used by A(t') and the remote device system clock used by t' are not, and generally do not need to be, synchronized. The selection of t' may be adjusted in some implementations, particularly at the beginning of a streaming session, but will generally settle to a final minimum value. With equation (<NUM>) defined above, the streaming application environment VFR event with timestamp t' may be dispatched at V(t') = A(t') + D, i.e., with an additional playback delay of D after the arrival of the corresponding FR event from the remote device. In addition, for all other VFR events, the playback delay will generally never be greater than D. Further, in some implementations, D may be automatically adjusted over time, e.g., based upon observed network jitter.

In addition, in some implementations, extrapolation and/or interpolation may be used in connection with tracking remote device FR events. In the illustration implementation, for example, each VFR event dispatch is initially scheduled in connection with the actual dispatch or issue of the prior VFR event. For example, right after dispatching a VFR event with timestamp t<NUM> at V(t<NUM>) = A(t') + (t<NUM> - t') + D, the VFR event with timestamp t<NUM> may be scheduled to dispatch soon at V(t<NUM>) = A(t') + (t<NUM> - t') + D.

However, if the FR event with timestamp t<NUM> is significantly delayed (or even lost) in the network, it is possible that this FR event still hasn't arrived at the streaming application environment by V(t<NUM>), and therefore scheduling of the next VFR event by FR handler component <NUM> may be complicated by the fact that the timestamp t<NUM> itself is still unknown. In some implementations, therefore, t<NUM> may be extrapolated from prior FR event timestamps, e.g., using equation (<NUM>): <MAT>.

Similarly, it is possible that the FR event with timestamp t<NUM> has not arrived at the streaming application environment but a later FR event with timestamp tn has arrived. In that case, therefore, some implementations may determine V(t<NUM>) using interpolation, e.g., using equation (<NUM>): <MAT>.

Through the use of extrapolation and/or interpolation, VFR events may be periodically issued at a rate substantially matching the FR event rate from the remote device <NUM>, and generally even when some FR events are significantly delayed or lost. The VFR event schedule effectively may be recalibrated as soon as more FR events arrive in the environment.

As also noted above, FR handler component <NUM> may also drive video streaming component <NUM>. In some implementations, however, rather than issuing the VFR events directly to video streaming component a controlled delay may be applied by scheduling a delayed version of a corresponding VFR event, referred to herein as a streaming component frame refresh (SCFR) event. In some implementations, for example, an SCFR event may be scheduled for issuance about <NUM> to accommodate the time the virtual streaming application takes to render a virtual frame, such that when the video streaming component <NUM> captures a virtual frame from the virtual streaming application framebuffer, the corresponding application content is fully rendered. It will also be appreciated, however, that if a virtual streaming application is configured to push a virtual frame to video streaming component <NUM> upon completion of rendering, no separate SCFR event may be used.

<FIG>, for example, illustrates an example sequence of operations <NUM> executable by FR handler component <NUM> in some implementations for scheduling frame refresh events in streaming application environment <NUM>. The sequence of operations may be called, for example, in response to issuance of a scheduled VFR event corresponding to a current FR event. Block <NUM> first determines if a next FR event has been received, and if so, passes control to block <NUM> to schedule the next VFR event using the timestamp of the received next FR event, e.g., using equation (<NUM>) above. If the next FR event is not received, however, block <NUM> passes control to block <NUM> to determine if a later FR event has been received. If not, control passes to block <NUM> to schedule the next VFR event using extrapolation, e.g., by extrapolating from the prior two FR event timestamps using equation (<NUM>) above. If, on the other hand, a later FR event has been received, block <NUM> passes control to block <NUM> to schedule the next VFR event using interpolation, e.g., using equation (<NUM>) above.

Once the next VFR event is scheduled by block <NUM>, <NUM> or <NUM>, control passes to block <NUM> to schedule an SCFR event corresponding to the VFR event, e.g., by adding a controlled or fixed delay to the issue time for the VFR event.

Next, block <NUM> optionally adjusts one or more delays (e.g., D used in equation (<NUM>) above) based upon observed network jitter. The adjustment may be based upon tracking system responses and network-associated delays, and may be based in some instances on heuristics, and in other instances on various machine learning techniques. It will be appreciated that increasing D further delays an input event playback schedule and consequently reduces the number of late arriving input events that are effectively discarded; however, the cost of doing so is higher end-to-end latency.

<FIG> next illustrates a sequence of operations <NUM> for handling received FR events in FR handler component <NUM>. While primarily sequence of operations <NUM> queues received FR events for scheduling corresponding VFR events at the appropriate time, as noted above, in some instances FR events may be lost or delayed, and extrapolation and/or interpolation may be used to predict the timestamps of lost or delayed FR events. Thus, it may be desirable in some implementations to update or confirm predicted timestamps when delayed FR events finally arrive. In particular, as illustrated in block <NUM>, it may be desirable to determine if a VFR event corresponding to a received FR event has already been scheduled. If not, control passes to block <NUM> to queue the FR event for later scheduling of the corresponding VFR event. If so, however, block <NUM> instead passes control to block <NUM> to update the scheduled VFR event (and, if scheduled, the corresponding SCFR event) based upon the timestamp of the received FR event.

Now turning to <FIG>, this figure illustrates a sequence of operations <NUM> executable by input handler component <NUM> of control player component <NUM> to play back input to the virtual streaming application <NUM> in a synchronized fashion with the frames of the remote device. In some implementations, input handler component <NUM> may continuously receive input events sent from control streamer component <NUM> and place the received input events on a priority queue where an input event with a smaller (earlier) timestamp has a higher priority than another input event with a larger (later) timestamp. The queued input events may then be retrieved later based on a playback schedule and injected, or played back, into the virtual streaming application <NUM> as corresponding virtual input events. Input event injection or playback may be synchronized by calling sequence of operations <NUM> in response to a VFR event generated by FR handler component <NUM>. Block <NUM> initiates a FOR loop to process each input event on the priority queue. Then, for each input event, that input event is handled based upon the timestamp associated with the input event relative to the timestamp associated with the VFR event triggering sequence of operations <NUM>.

First, as illustrated in block <NUM>, in some implementations it may be desirable to send duplicate input events to account for potential packet losses in the network, and as such, if an input event is determined to be a duplicate event in block <NUM>, control may pass to block <NUM> to pop or remove the duplicate event from the priority queue, but otherwise ignore the event. The duplicate event may be detected, for example, based upon matching a timestamp of another input event that has already been injected, or based upon other identifying characteristics that indicate that the input associated with the input event has already been injected into the virtual streaming application.

If the input event is not a duplicate event, sequence of operations <NUM> then determines whether the input event is a late arrival event, an early arrival event, or an on schedule event. For the purposes of this discussion, denote the timestamp of a queued input event by t, that of the current VFR event by t<NUM>, and that of the previous VFR event by t<NUM> where t<NUM> < t<NUM>. The queued input event may be considered to be a late arrival event, an early arrival event, or an on schedule event depending on the value its timestamp t compared to the current virtual frame interval (t<NUM>, t<NUM>].

Block <NUM>, for example, determines if the input event is a late arrival event, e.g., where t ≤ t<NUM>. The input event may be considered late, as the input event should have been injected at the VFR event with timestamp t<NUM>, but arrived sometime later. Thus if the input event is a late arrival event, block <NUM> passes control to block <NUM> to pop the input event from the priority queue, and otherwise ignore the event so that it does not interfere with other on schedule events to be played back in the current virtual frame interval. However, in some implementations a late arrival event may still be useful later on, and so the last late arrival event discarded in the current virtual frame interval may also be cached future use, as will be discussed in greater detail below.

Block <NUM> determines if the input event is an on schedule event, e.g., where t<NUM> < t ≤ t<NUM>. The input event in this instance should be popped from the priority queue and injected, and as such, block <NUM> passes control to block <NUM> to pop the input event from the priority queue and inject the input event into the virtual streaming application. For each of duplicate, late arrival and on schedule events, control then returns to block <NUM> to process the next highest priority input event on the priority queue.

Returning to block <NUM>, if the input event is not an on schedule event, the input event is an early arrival event, e.g., where t > t<NUM>. The input event in such an instance should be played back in a future virtual frame interval. As such, block <NUM> may handle such an input event by passing control to block <NUM> to leave the input event on the priority queue and ignore it. In addition, because input events are processed in order on the priority queue, there is also no need to further process other input events in the priority queue in this virtual frame interval as they are all early arrival events from the perspective of the current virtual frame interval.

In some implementations, whenever all input events have been processed by the FOR loop initiated in block <NUM>, or whenever an early arrival event is processed, playback of input events for the current virtual frame interval is complete. In other implementations, and as will be discussed in greater detail below, control streamer <NUM> in remote device <NUM> may be configured to resend a trailing event, so in such implementations, blocks <NUM> and <NUM> may instead pass control to block <NUM> to determine whether a cached late arrival event has a later timestamp than a last injected input event, and if so, to block <NUM> to inject the cached event. A further discussion of this functionality is discussed in greater detail below in connection with event filling.

As noted above, in some instances, particularly with mobile devices wirelessly coupled to a streaming application environment over a packet-switching network such as the internet, packet delays and losses can have an adverse impact on application streaming performance, particularly from the perspective of injecting input events into a virtual streaming application. Packet loss can be addressed in some instances using a protocol such as Transmission Control Protocol (TCP) that automatically handles packet retransmission, congestion control and reordering, although the use of TCP packets can, particularly with lossy networks, result in unacceptably long delays. Using a faster, but less reliable protocol such as User Datagram Protocol (UDP) can avoid some of the delays associated with TCP retransmissions and congestion control; however, the lost and out-of-order packets can result in inconsistent application control.

In various implementations disclosed herein, on the other hand, a number of techniques may be individually or collectively utilized to address packet losses and delays. In some implementations, for example, resiliency against packet loss may be increased by sending input events multiples times to add redundancy to the input stream, potentially with a short delay between repeats to address bursty packet loss. In addition, in some embodiments, correction events may be synthesized within the streaming application environment to account for lost and/or delayed packets.

For example, in some implementations, control streamer component <NUM> of remote device <NUM> may be augmented to resend each input packet N times, where N may be determined based on a tradeoff between packet loss resiliency and bandwidth consumption. In addition, in some instances, to better handle bursty packet losses where consecutive packets sent in a short burst tend to get lost together, the repeated input events may be sent with a delay, e.g., about <NUM>, after a previous transmission. <FIG>, for example, illustrates a sequence of operations <NUM> for handling input events in control streamer component <NUM>, e.g., as an alternative to sequence of operations <NUM> of <FIG>. In block <NUM>, an input event message is generated, including a remote device system time timestamp, as well as any related event parameters, e.g., touch coordinates, identifiers of related input events (e.g., for touch move or touch up events associated with a particular gesture, the timestamp of the initial touch down event that started the gesture), etc. Block <NUM> then sends the generated event message to the streaming application environment <NUM> N times, and optionally including a delay between each transmission. Sequence of operations <NUM> is then complete.

It will be appreciated that even with event duplication, an input event can still get lost together with the duplicated input events. In many instances, the effect of the lost event may be transient and may be overwritten quickly by subsequent input events. However, for certain events, referred to herein as trailing events, this may not necessarily be the case. As such, in some implementations it may be desirable to additionally augment control streamer component <NUM> to incorporate event filling. A trailing event, for the purposes of this disclosure, may be considered to be the last input event in a sequence before a silent period of no further input events. A common example of a trailing event is a touch up event in a swipe gesture on a touchscreen. If the touch up event is lost together with all of its duplicate events, the streaming application environment will not be able to determine if the user has lifted his/her finger or if the finger still stays on the screen just without further movements. This unknown state is therefore no longer transient, and can persist until the user starts the next touch gesture after a long pause.

As such, in some implementations it may also be desirable to check in connection with each frame on the remote device whether any new input event was received within the frame, and if not, resend the last input event to ensure that the silent period is filled with the last trailing event repeatedly at the FR rate, and that the trailing event will eventually arrive at streaming application environment to resolve the unknown state. In some implementations, for example, such functionality may be implemented in connection with the handling of frame refresh events in control streamer component <NUM>, e.g., using as an alternative to sequence of operations <NUM> of <FIG>, sequence of operations <NUM> of <FIG>. In sequence of operations <NUM>, prior to generating an FR event message, block <NUM> may determine whether any input event has been issued during the last frame interval. If so, control passes to block <NUM> to generate an FR event message, including a remote device system time timestamp, and then to block <NUM> to send the message to the streaming application environment <NUM>. On the other hand, if it is determined that no input event has been issued during the last frame interval, block <NUM> may pass control to block <NUM> to resend the last input event as a trailing event, prior to passing control to block <NUM>. As such, during frames in which no input events are generated, a trailing event is still sent, thereby ensuring that the trailing event will eventually be received by the streaming application environment.

Next from the perspective of the control player component <NUM>, event synthesis may be used to synthesize correction events, and in some instances companion events, to address lost or delayed input events. Even synthesis may be based, for example, on inferences drawn from other events that are injected on schedule. In some implementations, for example, correction events are synthesized from an injected or played-back event if applicable, and, then for each synthesized correction event and the original played-back event, a companion event may also be synthesized if applicable.

For simplicity, the following discussion for correction events will focus on gestures such as touchscreen gestures, and in particular on single-touch gestures where a single finger is manipulated on the touchscreen. It will be appreciated that a typical gesture of such a type will begin with a gesture start event and end with a gesture stop event, and with one or more gesture move events occurring between the gesture start and stop events. In the case of touchscreen gestures, for example, a touch down event, where a user first places a finger on the touchscreen, may correspond to a gesture start event, while a touch up event, where a user lifts the finger off of the touchscreen, may correspond to a gesture stop event, and a touch move event may correspond to a gesture move event. While a single-touch gesture is described hereinafter, however, extension of the herein-disclosed techniques to other gestures, including multi-finger touch gestures, mouse, trackball or other pointing device gestures, image captured gestures, and other gestures having defined start and stop events, would be straightforward. The invention is therefore not limited to use with the particular touch events described herein.

To facilitate synthesis of events, control player component <NUM> may be configured to track the states of gesture-related input events, including synthesized events, that are injected into a virtual streaming application, and in particular may maintain a state of a gesture as being started or completed. In some implementations, the gesture state may also correspond to a state of a pointer, e.g., where a started gesture state corresponds to a pointer down state and a completed gesture state corresponds to a pointer up state. It may also be desirable in some implementations to retain the coordinates associated with the last injected or played back input event, as well as the timestamp of the last injected or played back gesture start (e.g., touch down) event, denoted by td. In addition, it may be desirable to include with each gesture-related event a field that represents the timestamp of the associated gesture start (e.g., touch down) event that initiated the current gesture. It will be appreciated that the coordinates may correspond to a two-dimensional location on a touchscreen or other two dimensional display, while in some implementations three-dimensional coordinates may be used as appropriate.

Now with reference to <FIG>, this figure illustrates a sequence of operations <NUM> executable by input handler component <NUM> of control player component <NUM> to inject gesture events into a virtual streaming application. Sequence of operations <NUM>, for example, may correspond to the injection operations referred to in blocks <NUM> and <NUM> of <FIG>. In blocks <NUM>-<NUM> of sequence of operations <NUM>, several different scenarios, which may be implemented individually or collectively in different implementations, are illustrated for synthesizing events.

Block <NUM>, for example, tests whether the event to be injected is a gesture start (e.g., touch down) event and the gesture state is started, which effectively corresponds to the loss of a prior gesture stop (e.g., touch up) event. If so, control passes to block <NUM> to generate an appropriate correction event corresponding to the lost gesture stop event. For example, in some implementations the correction event may be generated by duplicating the gesture event but changing the action from start(down) to stop(up), replacing the coordinates of the event with those of the last injected input event, and replacing the timestamp with td, the timestamp of the last injected gesture start event.

Block <NUM> tests whether the event to be injected is a gesture stop or move event, but the current state is still completed (up), which effectively corresponds to the loss of a prior gesture start (e.g., touch down) event. If so, control passes to block <NUM> to generate an appropriate correction event corresponding to the lost gesture start event. For example, in some implementations the correction event may be generated by duplicating the gesture stop or move event, but replacing the stop or move action with start.

Block <NUM> tests whether the event to be injected is a gesture stop or move event, the current state is started (down), but in the event to be injected the timestamp of the associated gesture start (e.g., touch down) event is different from td, which effectively corresponds to the event to be injected being from a new gesture and at least one prior pair of gesture stop (e.g., touch up) and gesture start (e.g., touch down) event has been lost. If so, control passes to block <NUM> to generate an appropriate pair of correction events corresponding to the lost gesture stop and gesture start events, with the synthesis of the correction gesture stop and gesture start events performed in a similar manner to that described above in connection with blocks <NUM> and <NUM>.

After correction events are synthesized, or if no such correction events are needed, control next passes to block <NUM> to inject the gesture event. In addition, in some implementations it may also be desirable to inject one or more companion events at this time. In particular, it has been observed that on a physical computing device with the touchscreen, each touch down event is always immediately followed by a touch move event, and each touch up event is always immediately preceded by a touch move event. Therefore, in some implementations it may be desirable to synthesize this behavior in block <NUM> by always preceding any gesture stop event (including a correction gesture stop event) with a companion gesture move event and following any gesture start event (including a correction gesture start event) with a companion gesture move event. A companion event may be generated, for example, by duplicating the associated gesture start or stop event and simply changing the action from start or stop to move.

Now returning once again to <FIG>, as noted above, it may be desirable in some instances to cache the last late arrival input event discarded in the current virtual frame interval (block <NUM>). After popping all the late arrival and on schedule input events in the priority queue and preserving all the early arrival input events (blocks <NUM>-<NUM>), the cached last late arrival input event may be compared against the last injected input event in the current virtual frame interval. The cached event, although being a late arrival event, may in some implementations still be useful because it may be a trailing event that is resent periodically as described above in connection with <FIG>. For this reason, the cached event may be played back as the last event in the current virtual frame interval as long as its timestamp is greater than the timestamp of the last input event injected or played back into the virtual streaming application. Testing of this condition is illustrated in block <NUM>, and injecting of the cached event is illustrated in block <NUM>. It will be appreciated that the comparison of the timestamps preserves ordering in the input events perceived by the virtual streaming application.

The herein-described techniques may be implemented in a number of different computers, computer systems, or computing devices in various implementations. <FIG>, for example, is a block diagram of an example computer <NUM> that may be used to implement a remote device and/or various computer systems within an application streaming environment. Computer <NUM> typically includes at least one processor <NUM> which communicates with a number of peripheral devices via a bus subsystem <NUM>. The input and output devices allow user interaction with computer <NUM>, and network interface subsystem <NUM> provides an interface to outside networks and is coupled to corresponding interface devices in other computers.

In general, use of the term "input device" is intended to include all possible types of devices and ways to input information into computer <NUM> or onto a communication network.

Storage subsystem <NUM> stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem <NUM> may include the logic to perform selected aspects of the aforementioned sequences of operations and/or to implement one or more components of the various devices and environments illustrated in <FIG>.

Memory <NUM> used in the storage subsystem can include a number of memories including a main random access memory (RAM) <NUM> for storage of instructions and data during program execution and a read only memory (ROM) <NUM> in which fixed instructions are stored.

Bus subsystem <NUM> provides a mechanism for allowing the various components and subsystems of computer <NUM> to communicate with each other as intended.

Computer <NUM> can be of varying types including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computer <NUM> depicted in <FIG> is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of computer <NUM> are possible having more or fewer components than the computer system depicted in <FIG>.

Moreover, particularly where computer <NUM> is used to implement a remote device such as a mobile phone, tablet computer, vehicle computer, wearable computer, etc., additional sensors <NUM> may also be included. Sensors such as accelerometers, global positioning sensors, gyroscopic sensors, health sensors, among others, may be used.

Claim 1:
A method, comprising, in a device (<NUM>) including one or more processors and a display:
receiving an output display representation from a remote streaming application environment (<NUM>) coupled to the device over a packet-switched network (<NUM>), the output display representation generated by a virtual streaming application (<NUM>) resident in the remote streaming application environment, wherein the virtual streaming application is an application run in a virtual machine or virtual computing device in the remote streaming application environment;
rendering a plurality of frames of the received output display representation on the display (<NUM>);
in response to inputs generated during rendering of the plurality of frames, issuing a plurality of input events for communication over the packet-switched network to the remote streaming application environment, wherein input events are generated by the device to control the virtual streaming application in response to user inputs; and
in response to a determination that no inputs are generated during a frame among the plurality of frames, reissuing a trailing event issued in a prior frame for communication over the packet-switched network to the remote streaming application environment, wherein each input event of the plurality of input events is associated with a corresponding timestamp, wherein the trailing event is associated with a trailing event timestamp, and wherein the trailing event timestamp corresponds to the corresponding timestamp of a respective input event for which the trailing event is reissued, and wherein the trailing event is placed in a priority queue along with the plurality of input events as a last input event of the plurality of input events based on the trailing event timestamp corresponding to the corresponding timestamp of the respective input event.