Patent Description:
Further, computing system functionality can be enhanced by a computing system's ability to be interconnected to other computing systems via network connections. Network connections may include, but are not limited to, connections via wired or wireless Ethernet, cellular connections, or even computer to computer connections through serial, parallel, USB, or other connections. The connections allow a computing system to access services at other computing systems and to quickly and efficiently receive application data from other computing systems.

Some computing systems may use network interconnections to implement a technology referred to herein as remote rendering. In particular, compact and portable systems are being created which display rich visual content to users. For example, virtual reality headsets which display virtual objects to users and augmented reality headsets which superimpose rendered, virtual objects onto real world environments and objects are now available. Typically, these headsets and other such devices are limited in size and weight, which results in limitations to the computing functionality that can be performed by the devices. Nonetheless, it is desirable that these devices be able to output complex images such as various views of 3D models.

Additionally, the images output by these devices is often dependent on user input at the devices. For example, in the context of virtual reality and augmented reality headset devices, a user may wear the headset device and then move spatially in a physical environment. The position of the user in the physical environment determines what image should be output on the device. For example, consider a case where a user is viewing a virtual 3D object. The 3D object can be placed in a stationary location. As the user moves the headset device about the 3D object, the view of the 3D object should change on the headset device. Alternatively or additionally, a user may be able to manipulate the 3D object to move the 3D object while the user remains somewhat stationary. This too will change what should be output on the headset device.

Rendering views of 3D models can require an immense amount of computing functionality. However, as noted previously, devices on which the 3D models are viewed often have limited computing functionality.

To solve this problem, remote rendering has been implemented. In a remote rendering arrangement, a more powerful computer is connected to the limited power device. The limited power device sends telemetry information, such as position information, user inputs to manipulate 3D objects, or other such information to the more powerful computer. The more powerful computer then manipulates the 3D model to create a view, which is then returned to the limited power device.

As one might imagine, it is important that the interactions and displayed images occur in a perceived seamless fashion. To accomplish this, there are certain deadlines between when images are received from the more powerful computer so that they can be displayed to the user in a seemingly seamless fashion. Similarly, there are deadlines for the more powerful computer to receive telemetry data from the device.

However, data will often be received too early or too late to be useful. This can be due to transmission jitter such as variable network delays, differences between the device clock and the more powerful computer clock, or for other reasons. For example, video data is often sent at <NUM> meaning that a new image or frame is sent in a frame packet from the more powerful computer every <NUM> milliseconds. Similarly, telemetry data is sent from the device to the more powerful computer at the same rate, i.e. <NUM> with one packet of telemetry data being sent every <NUM> milliseconds. Consider a case where there is a latency jitter in the network such that the latency for data being sent from the more powerful computer to the device changes over time. For example, at one point in time, the latency for a video packet to be sent from the more powerful computer to the device is <NUM> milliseconds. At a different time, the latency for a video packet to be sent from the more powerful computer to the device is <NUM> milliseconds. This changing latency (i.e., latency jitter) causes video packets to arrive too late to be used for rendering at the device, and/or multiple video packets will arrive close together resulting in one of the packets being discarded and the other packet used for rendering at the device. In either case, this results in dropped packets which are deleterious to the user experience at the device.

Therefore, it would be useful to implement a system that can more efficiently use streamed data packets and in particular data packets that are streamed in response to telemetry data, in the presence of network latency jitter, slight differences in system clocks, and for other reasons.

This background is only provided to illustrate one exemplary technology area where some embodiments described herein may be practiced. <CIT> describes an invention which relates to a real-time video congestion control method and device based on data driving. The method is mainly used for an end-to-end internet video transmission application scene in a wireless network. The method mainly comprises a learning-based rate control strategy and a low-delay fairness model. According to historical data in a sliding window, a linear relation between link delay change and rate change is described by means of online linear regression; And according to the linear relation, an optimal sending rate can be decided by combining with the fairness function based on the load target provided by the invention. In the whole decision process, the characteristic that the wireless network has high variability is fully considered, the longterm change of the network environment is sensed through historical data, and the sending rate is adjusted by means of the link delay to adapt to the instantaneous fluctuation of the network, so that the purposes of improving the bandwidth utilization rate and reducing the link delay are achieved, and the user experience of real-time video application is improved. <CIT> describes a method for measuring the jitter experienced by an application's network traffic. The measurement is based solely on packets sent or received by the application itself. The method does not alter the application's packets, in particular, it does not add a timestamp to the packets before they are sent. Instead, it creates a table that stores unique identifiers of the application's packets along with the time the packets are sent. On the receiving computer, a similar table is created that stores the unique packet identifiers along with the time the packets are received. Records of sent packets are associated with records of received packets so that the time a packet was sent can be compared to the time the same packet was received. The resulting data are processed to calculate network jitter and packet loss ratios.

The invention comprises a system and corresponding methods that may be practiced in a computing environment. One method includes acts for sending streamed data packets from a producer to a consumer. The method includes, at a first entity, sending consumable data packets from the first entity to a second entity at a first consumable packet rate. The method further includes receiving a first phase delta from the second entity, wherein the first phase delta is computed from transmission jitter, computed from timing information in the consumable data packets. The method further includes sending from the first entity consumable data packets at a second consumable packet rate, the second consumable packet rate being dependent on the first phase delta.

Another method that may be practiced in a computing environment includes acts for receiving streamed data packets from a producer to a consumer. The data packets are dependent on state from the consumer. The method includes at a second entity, receiving consumable data packets from a first entity at a first consumable packet rate. The method further includes computing transmission jitter based on timing information in the consumable data packets. The method further includes computing a first phase delta based on the jitter, the first phase delta defining timing differences between when consumable data packets are sent from the second entity. The method further includes providing the first phase delta to the first entity. The method further includes receiving from the first entity consumable data packets at a second consumable packet rate, the second consumable packet rate being dependent on the first phase delta.

Features of the present invention will become more fully apparent from the following description and appended claims.

Embodiments illustrated herein are directed to a remote rendering system that is able to dynamically determine dynamically changing transmission jitter over time for transmitted data packets. The remote rendering system can compute a dynamically changing "phase" delta between a transmitted signal from a producer of data transmitting a stream of data packets as compared to a received signal of the stream of the data packets received by a consumer of data. As used herein the phase delta is the time difference between when packets are sent by a producer and when they are received by a consumer. This time difference can be in terms of actual time, percent of time it takes to transmit a packet from the producer to a consumer, percent of time between the start of two adjacent packets, or consumer electronics, network PCs, minicomputers, mainframe computers, mobile other relevant time. The phase delta can be provided to the producer, which can then adjust the frequency of the transmitted signal. For example, if the transmitted signal is a signal of video packets being transmitted at <NUM> packets per second, the producer may use the received phase information to either slightly raise or lower the packet rate to <NUM> packets per second or <NUM> packets per second (or some other rate as determined by the phase delta) to cause the data packets to be received in the received signal at a rate that allows the video packets to be used efficiently. Thus, there exists a technical problem whereby extra latency is introduced to a consuming system when data packets from a consuming system arrive too soon or too late. This problem can be solved by a technical means of evaluating phase shifts and adjusting data packet transmission frequency by adjusting the rate at which data packets are sent by a producing system.

Referring now to <FIG>, an example is illustrated. <FIG> illustrates a producer <NUM> and a consumer <NUM>. The producer <NUM> produces a consumable data stream <NUM>. The consumable data stream <NUM> comprises a number of data packets. For example, in the examples illustrated herein, the producer <NUM> may be a powerful computer configured to render images from a 3D model based on telemetry data from the consumer <NUM>, and to transmit the images to the consumer <NUM>. Thus, in the illustrated example, the consumable data stream <NUM> may include a number of video packets. <FIG> also illustrates that the consumer <NUM> sends a telemetry data stream <NUM> to the producer <NUM>. The telemetry data stream <NUM> includes telemetry data produced by the consumer <NUM>.

For example, such telemetry data may include position data for the consumer <NUM>, user input at the consumer <NUM>, or other data that can be used to determine what data should be included in the consumable data stream <NUM>. For example, consider that the consumer sends a telemetry data packet <NUM>-<NUM>. The data contained in the telemetry data packet <NUM>-<NUM>, which may be for example, data indicating a user's absolute position, data indicating movement of a user, data indicating user input at a user interface, etc., is used by the producer <NUM> to generate a consumable data packet <NUM>-<NUM>, such as a video frame. Thus, in the example illustrated in <FIG>, the producer <NUM> streams consumable data packets in the consumable data stream <NUM> in response to the telemetry data packets in the telemetry data stream <NUM>.

Note that typically the packet rate of the consumable data stream <NUM> is therefore the same packet rate as the telemetry data stream <NUM>. For example, in the 3D video rendering, the consumable data stream <NUM> will typically be transmitted at <NUM>. Similarly, the telemetry data stream <NUM> will be transmitted by the consumer <NUM> to the producer <NUM> at <NUM>. As noted previously, transmission jitter may vary the rate at which producer <NUM> sends the consumable data stream <NUM> and at which the consumer <NUM> receives the consumable data stream <NUM>. For example, the consumable data stream <NUM> is sent on some form of network. That network may be a wide area network, the Internet, a local area network, a Wi-Fi connection, a Bluetooth connection, or any other network medium, or combinations thereof. That particular network may have certain latency associated with the network. That is, it takes a finite amount of time for a particular consumable data packet to travel from the producer <NUM> through the network to the consumer <NUM>. Additionally, that latency may change over time. Thus, over time, the packet rate at which the consumable data stream <NUM> is sent by the producer <NUM> will be different at different times to the packet rate perceived by the consumer <NUM>. As noted previously, this can result in dropped packets and other problems that degrade the user experience. Examples are now illustrated with references to <FIG>.

As noted previously, remote rendering is used to overcome thermal and performance limitations in power-constrained devices, such as mobile phones or un-tethered augmented reality and virtual reality headset devices. In this context, a computationally heavy rendering operation is performed in a remote machine (the server or host) periodically and is transmitted over the network to another device, the client or player, for display. This operation is repeated multiples times per second to achieve a target packet rate.

Even if the network is fast enough to send the information at the required rate, the whole process is sensitive to when exactly the received information is ready to be consumed, as shown in <FIG>.

<FIG> illustrates an ideal phase delta between the producer and the consumer. In this example, packets are created during rendering periods. For example, a packet is created during the rendering period <NUM>-<NUM> and sent from the producer to the consumer. At the consumer, a certain display period is needed to display the image created from the data in the packet at the consumer. Thus, in the example, illustrated, the packet created during the rendering period <NUM>-<NUM> arrives slightly before it is needed for processing during the display period <NUM>-<NUM> to make the deadline <NUM>-<NUM>.

<FIG> illustrates a non-ideal phase delta between the producer and the consumer. In this example, the packet created during the rendering period <NUM>-<NUM> arrives slightly after it is needed for processing by the display period <NUM>-<NUM>. Thus, it is saved for the next display period <NUM>-<NUM> introducing the extra latency <NUM>-<NUM>. Note that in the illustrated example, the rendering periods occur at a particular rate (e.g., <NUM>) and the display periods occur at the same particular rate. If the frequency of the rendering periods would have been increased slightly, the packet produced by the rendering period <NUM>-<NUM> could have been used by the display period <NUM>-<NUM>. Thus, embodiments may include functionality for identifying non-ideal phase deltas and adjusting the rate of the rendering periods, resulting in a corresponding adjustment to the packet rate for packets sent from the producer to the consumer. Note that the rate may be adjusted back down to the standard rate once deadlines are effectively being met.

<FIG> illustrates a non-ideal phase delta between the producer and the consumer where packets are arriving too early, introducing extra latency. In particular, the rendering period <NUM>-<NUM> produces and sends a packet that arrives earlier than desired at the consumer. Extra latency <NUM>-<NUM> is experienced while waiting for the display period <NUM>-<NUM> to display the image from the packet.

Note that this could be corrected by lowering slightly the frequency of the rendering periods, and therefore lowering the frequency of the packet rate of packets from the producer to the consumer. Note that the rate may be adjusted back up to the standard rate once deadlines are effectively being met.

Therefore, having an incorrect phase delta between the producer and the consumer introduces additional latency and micro-stuttering when the phase changes due to varying network conditions, or for other reasons.

Note that while the example illustrated in <FIG> shows an ideal stream, a non-ideal stream due to packets arriving too late, and a non-ideal stream due to packets arriving too early, it should be appreciated that real world examples will include mixtures of the above. For example, network latency will typically have a mean latency. If data always arrived according to the mean latency, the embodiment illustrated in <FIG> could be implemented. However, this is not the case. Latency on the network will vary such that sometimes data will be transmitted more quickly across the network and sometimes data will be transmitted less quickly across the network. Thus, over time, assuming that the rendering period rate is constant, a mixture of the various conditions illustrated in <FIG> will be experienced. However, embodiments illustrated herein attempt to vary the rendering period rate such that it is not constant, but rather varies slightly so as to attempt to achieve the example illustrated in <FIG>, except with a varying rendering period rate.

Another example is illustrated in <FIG> illustrates the producer <NUM> and the consumer <NUM>. The consumable data stream <NUM>, including a number of packets rendered by the producer <NUM>, and to be consumed by the consumer <NUM> is sent from the producer <NUM> to the consumer <NUM>. In the examples illustrated herein, the packets included in the consumable data stream may be video frame packets where each packet includes image data for a frame. The images are rendered at the producer <NUM>. The data packets received by the consumer <NUM> include image data that can be displayed at the consumer <NUM>. Note that the consumable data stream <NUM> has a packet rate which is variable. That is, the packet rate varies based on transmission latency.

Transmission latency may be caused by a number of different factors including network latency, varying clocks between the producer <NUM> and the consumer <NUM> (i.e., the timing clock at the producer <NUM> may have a slightly different frequency than the timing clock at the consumer <NUM>, as a result of clock hardware different based on slight or more pronounced manufacturing differences, aging, heating up, etc., resulting in different data rates perceived by the producer <NUM> and the consumer <NUM> even when the data rates are constant between the two), or for other reasons. <FIG> also illustrates a phase delta data stream <NUM>. The phase delta data stream <NUM> includes a number of data packets. The data packets in the phase delta data stream <NUM> include information from the perspective of the consumer <NUM> identifying the difference in phase between what the producer <NUM> perceives as the consumable data stream <NUM> and what the consumer <NUM> perceives as the consumable data stream <NUM>. For example, the producer will send the consumable data stream <NUM> with various timestamps in the packets in the consumable data stream <NUM>. The consumer <NUM> can then identify when packets were sent, which can be used to identify the frequency at which the packets are sent in the consumable data stream <NUM>. The consumer <NUM> is also able to detect when packets are actually received at the consumer <NUM>. Using this information, the consumer <NUM> can construct a phase delta describing the difference in phase between what the producer <NUM> perceives and what the consumer <NUM> perceives. Note that this phase delta will change over time due to the various factors described previously herein. Thus, the phase delta data stream <NUM> is a continuous data stream describing the changing phase deltas between what is perceived by the producer <NUM> and what is perceived by the consumer <NUM>. The producer <NUM> can use the data in the packets of the phase delta data stream <NUM> to adjust the data rate of the consumable data stream <NUM>. As noted previously, in some embodiments this may include adjusting the frequency of the rendering periods. Additional details are now illustrated.

Using a rolling window, the consumer <NUM> will keep track and/or collect statistics of the packet arrivals, in relation to the fixed deadlines. The use of a rolling window allows the system to adapt to changing transmission latency, such as changing network conditions. The optimal number of samples per window depends on the nature of the network. The larger the window, the longer it takes for statistics derived from the samples to change, and hence the longer it takes to react to a change in network conditions. One implementation uses a rolling window of <NUM> samples, meaning embodiments react within a timeframe of <NUM> (<NUM> * <NUM>) to changing network conditions.

Based on the collected information and statistics, the consumer <NUM> computes a phase delta that attempts to minimize latency (where in this context, latency is defined as the difference between arrival time of a data packet and the beginning of processing of the data packet). The computation includes computing the rate at which packets are sent by the producer <NUM> based on time stamps in the packets, the rate at which packets are received based on evaluating clock values at the consumer <NUM> when packets are received, and computing a difference between the rate that packets are sent as compared to the rate at which packets are received. On each incoming packet at the consumer, the time difference between the arrival of the video packet and the corresponding following display period is computed (i.e., the `extra latency' in <FIG>). This value is added to the window of samples. From the rolling window of these values, the median value is derived, which estimates the true phase difference between producer and consumer that one would see in the absence of network jitter. As network jitter delays packets, a temporary network jitter causes a spike in the latency sample values. These spikes are filtered out reliably by the median, which is not the case for taking a simple average. Additionally, a desired target value for the phase shift is determined. This may be hardcoded (e.g. <NUM>) or derived from the rolling window as a multiple of the standard deviation of the sample values. The producer packet rate is then adjusted to bring the median closer to this target value, speeding up the producer rate if the median is less than the target value, or slowing down the producer rate if the measured median is larger than the target value. That is, embodiments do not attempt to bring the median phase shift to <NUM>, which would mean no extra latency, but rather to a specific value that increases the probability that a packet which behaves according to the latency distribution that was sampled to arrive before the display period. In effect, on a stable network, the rolling window of samples will have a low standard deviation, so the target value can be very small; whereas on a network with high amounts of jitter, the standard deviation will be higher. In this case, a small amount extra latency is desired to improve the chances of packets arriving in time even if they are delayed over the network.

Generally, embodiments can be formulated as an "optimization problem", which is a mathematics and computer science method of finding a best solution among many feasible solutions that minimizes latency. In some embodiments, the maximum latency may be minimized. Alternatively, the average latency may be minimized. Alternatively, the median latency may be minimized. Alternatively, the k-th percentile or similar metrics of time delay can be used. For example, in some embodiments, Instead of taking the standard deviation in the description above, other statistics derived from the distribution may be used as the target value. This is what the k-th percentile may be used for. That is, embodiments choose the target value such that k% of the samples in the rolling window would arrive before the display period. The best statistics depend heavily on the nature of the distribution, although observation has found that it tends to be very one-sided for typical networks, with most samples close to the median and a few spikes, instead of e.g. a bell curve.

The computed phase delta is sent periodically in the phase delta data stream <NUM> from the consumer <NUM> to the producer <NUM> over the network. Note that in some embodiments, this communication can be merged (piggy-backed) with other packets, to minimize network traffic. For example, in some embodiments, the telemetry data stream packets may include phase delta data as well as the telemetry data.

The producer <NUM> gradually and adaptively shifts wake-up times for hardware performing rendering negatively or positively based on the requested phase-delta by the consumer. The phase delta is performed gradually to avoid the creation of jittering and/or micro-stuttering.

In some embodiments, functionality may be implemented in a bidirectional fashion. In particular, an entity may be a consumer of certain data and a producer of other data. Such an entity may need to receive data in a fashion to meet certain deadlines at the entity and to produce and provide data for a different entity in a fashion to for the different entity to meet certain deadlines.

Attention is now directed to <FIG> which illustrates a first entity <NUM> and a second entity <NUM>. The first entity <NUM> produces and sends a consumable data stream <NUM> to a second entity <NUM>. Similar to what is described above, the second entity <NUM> can provide a phase delta data stream <NUM> for the consumable data stream <NUM>. This allows the first entity <NUM> to adjust the packet rate in the consumable data stream <NUM>. This is done to ensure that packets arrive at the second entity <NUM> for the second entity to process those packets in a fashion that meets certain deadlines at the second entity <NUM>.

However, the second entity <NUM> may send a form of "consumable data" to the first entity <NUM>. In the present example, this data is illustrated as telemetry data, although it should be appreciated that other types of consumable data can be provided from the second entity <NUM> to the first entity <NUM>. Thus, <FIG> illustrates a telemetry data stream <NUM>. In the present example, the telemetry data stream <NUM> includes data packets with telemetry information about the second entity <NUM>. These packets are sent in a consumable stream, i.e. the telemetry data stream <NUM>, to the first entity <NUM> where they are consumed by the first entity <NUM>. In the context of the examples illustrated herein, the first entity <NUM> may be a 3D model manipulation system that uses the telemetry data in the telemetry data stream <NUM> to produce video frames for video packets included in the consumable data stream <NUM>.

The first entity <NUM> may send a phase delta data stream <NUM> for the telemetry data stream <NUM>. In particular, the first entity <NUM> may have certain deadlines that need to be met at the first entity <NUM>. To reduce latency, it may be desirable that those packets arrive at the first entity <NUM> in a particular fashion. As illustrated previously herein, the phase delta data stream <NUM> for the telemetry data stream <NUM> may identify a phase delta to the second entity <NUM> to allow the second entity <NUM> to adjust the packet rate of the telemetry data stream <NUM>. Using the principles illustrated previously herein, this allows the first entity <NUM> to efficiently process packets from the telemetry data stream <NUM> to meet deadlines at the first entity <NUM>.

As discussed previously, data streams and packets may be combined where appropriate. For example, phase delta data for the consumable data stream <NUM> may actually be included in the telemetry data stream <NUM>. Similarly, phase delta data for the telemetry data stream <NUM> may actually be included in the consumable data stream <NUM>. Thus, while <FIG> illustrates four separate data streams, it should be appreciated that similar functionality can be implemented with two data streams. Thus, for example, a packet in the telemetry data stream <NUM> may include telemetry data generated at the second entity <NUM> as well as phase delta data for a particular packet from the first entity <NUM> in the consumable data stream <NUM>. Similarly, a packet in the consumable data stream <NUM> may include data for a video frame as well as phase delta data for a particular packet in the telemetry data stream <NUM>. In this way, a need for having separate streams for the data shown in <NUM> and <NUM> is obviated.

Thus, in some embodiments in remote rendering, embodiments are able to decide the optimal point in time where the local client should sample the input information and send it to the rendering server. Alternatively or additionally, embodiments are able to decide the optimal point in time where the remote server will begin creating an image to be sent to the client.

Embodiments may be implemented to run continuously (and not only during communication handshake) to account for varying network conditions and for differences in the precision of the clocks in the server and client.

Having just described the various features and functionalities of some of the disclosed embodiments, attention is now directed to <FIG>, which illustrates an example computer system <NUM> that may be used to facilitate the operations described herein. It will be appreciated that, in some instances, aspects of the systems illustrated and the computer system <NUM> shown in <FIG> can be used in combination to carry out the embodiments described herein.

The computer system <NUM> may take various different forms. For example, in <FIG>, the computer system <NUM> is shown as including a head-mounted display (HMD). Although the computer system <NUM> may be, at least partially, embodied as a HMD, the computer system <NUM> may also be a distributed system that includes one or more connected computing components/devices that are in communication with the HMD. Accordingly, the computer system <NUM> may be embodied in any form and is not limited strictly to the depiction illustrated in <FIG>. By way of example, the computer system <NUM> may include a projector, desktop computer, a laptop, a tablet, a mobile phone, server, data center and/or any other computer system. Indeed, in some embodiments, the computer system <NUM> includes a cloud service/device implemented as the first entity <NUM> illustrated in <FIG>, and an HMD implemented as the second entity <NUM>. Alternatively, the computer system could include a desktop or laptop device, implemented as the first entity <NUM> illustrated in <FIG>, and an HMD implemented as the second entity <NUM>. Alternatively or additionally, the computer system <NUM> could include a specialized GPU hardware/device, a gaming console/device, or combinations thereof as the first entity <NUM> illustrated in <FIG>, and an HMD implemented as the second entity <NUM>. Alternatively, the second entity could be some other type of device other than an HMD, such as some other lower powered portable device.

In its most basic configuration, the computer system <NUM> includes various different components. For example, <FIG> shows that computer system <NUM> includes at least one hardware processing unit <NUM> (aka a "processor"), input/output (I/O) interfaces <NUM>, graphics rendering engines <NUM>, a projector <NUM>, and storage <NUM>. More detail on the hardware processing unit <NUM> will be presented momentarily. Note that the hardware can be implemented on either or both the first entity <NUM> and the second entity <NUM>. Note further that the claims herein are not necessarily directed to the entire system <NUM>, but may be directed to only portions of the system <NUM>. For example, while the first entity and the second entity work in concert to implement the system <NUM>, those different entities may be implemented as each their own systems, by each their own providers, and thus, the claimed embodiments herein may be directed to either or both portions of the system as appropriate. Thus, while individual components are described below, those components may be implemented as multiple actual components implemented in different physical forms on different portions of the system <NUM>.

The storage <NUM> may be physical system memory, which may be volatile, non-volatile, or some combination of the two. The term "memory" may also be used herein to refer to non-volatile mass storage such as physical storage media. If the computer system <NUM> is distributed, the processing, memory, and/or storage capability may be distributed as well. Thus, for example, memory may be implemented at a producer and different memory implemented at the consumer, while both memories are included in the storage <NUM>. As used herein, the term "executable module," "executable component," or even "component" can refer to software objects, routines, or methods that may be executed on the computer system <NUM>. The different components, modules, engines, and services described herein may be implemented as objects or processors that execute on the computer system <NUM> (e.g. as separate threads).

The disclosed embodiments may comprise or utilize a special-purpose or general-purpose computer including computer hardware, such as, for example, one or more processors (such as the hardware processing unit <NUM>) and system memory (such as storage <NUM>), as discussed in greater detail below. Embodiments also include physical and other computer-readable media for carrying or storing computer-executable instructions and/or data structures. Such computer-readable media can be any available media that can be accessed by a general-purpose or special-purpose computer system. Computer-readable media that store computer-executable instructions in the form of data are physical computer storage media. Thus, by way of example and not limitation, the current embodiments can comprise at least two distinctly different kinds of computer-readable media: computer storage media and transmission media.

Computer storage media are hardware storage devices, such as RAM, ROM, EEPROM, CD-ROM, solid state drives (SSDs) that are based on RAM, Flash memory, phase-change memory (PCM), or other types of memory, or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code means in the form of computer-executable instructions, data, or data structures and that can be accessed by a general-purpose or special-purpose computer.

The computer system <NUM> may also be connected (via a wired or wireless connection) to external sensors (e.g., one or more remote cameras, accelerometers, gyroscopes, acoustic sensors, magnetometers, etc.). It will be appreciated that the external sensors include sensor systems (e.g., a sensor system including a light emitter and camera), rather than solely individual sensor apparatuses. Further, the computer system <NUM> may also be connected through one or more wired or wireless networks to remote systems(s) that are configured to perform any of the processing described with regard to computer system <NUM>.

During use, a user of the computer system <NUM> is able to perceive information (e.g., a mixed-reality environment) through a display screen that is included among the I/O interface(s) <NUM> and the projector <NUM> that is visible to the user. The I/O interface(s) <NUM> may include the input elements described herein, which are linked to one or more underlying applications.

The I/O interface(s) <NUM> and sensors may also include gesture detection devices, eye trackers, and/or other movement detecting components (e.g., cameras, gyroscopes, accelerometers, magnetometers, acoustic sensors, global positioning systems ("GPS," etc.) that are able to detect positioning and movement of one or more real-world objects, such as a user's hand, a stylus, and/or any other object(s) that the user may interact with while being immersed in the scene. This information may be provided as telemetry data as described above.

The graphics rendering engine <NUM> is configured, with the hardware processing unit <NUM> and the projector <NUM>, to render one or more virtual objects within the scene. As a result, the virtual objects accurately move in response to a movement of the user and/or in response to user input as the user interacts within the virtual scene.

A "network," is defined as one or more data links and/or data switches that enable the transport of electronic data between computer systems, modules, and/or other electronic devices. When information is transferred, or provided, over a network (either hardwired, wireless, or a combination of hardwired and wireless) to a computer, the computer properly views the connection as a transmission medium. The computer system <NUM> will include one or more communication channels that are used to communicate with the network <NUM>. Transmissions media include a network that can be used to carry data or desired program code means in the form of computer-executable instructions or in the form of data structures. Further, these computer-executable instructions can be accessed by a general-purpose or special-purpose computer.

Computer-executable (or computer-interpretable) instructions comprise, for example, instructions that cause a general-purpose computer, special-purpose computer, or special-purpose processing device to perform a certain function or group of functions.

Additionally, or alternatively, the functionality described herein can be performed, at least in part, by one or more hardware logic components (e.g., the hardware processing unit <NUM>). For example, and without limitation, illustrative types of hardware logic components that can be used include Field-Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application -Specific Standard Products (ASSPs), System-On-A-Chip Systems (SOCs), Complex Programmable Logic Devices (CPLDs), Central Processing Units (CPUs), and other types of programmable hardware.

Referring now to <FIG>, a method <NUM> is illustrated. The method <NUM> may be practiced in a computing environment, and includes acts for sending streamed data packets from a producer to a consumer. The method includes, at a first entity, sending consumable data packets from the first entity to a second entity at a first consumable packet rate (act <NUM>). For example, in <FIG>, the entity <NUM> can send data packets to the second entity <NUM> in the data stream <NUM>.

The method <NUM> further includes receiving a first phase delta from the second entity, wherein the first phase delta is computed from transmission jitter, computed from timing information in the consumable data packets (act <NUM>). For example, phase delta information may be sent in the phase delta stream <NUM>.

The method <NUM> further includes sending from the first entity consumable data packets at a second consumable packet rate, the second consumable packet rate being dependent on the first phase delta (act <NUM>). For example, the first entity <NUM> may adjust the rate at which packets are sent in the stream <NUM>.

The method <NUM> may be practiced where the data in the consumable data packets is dependent on state data of the second entity provided to the first entity. For example, in some embodiments, the consumable data packets are video packets, the second entity is a consumer of video packets, the first entity is a producer of video packets, and the state data comprises telemetry data. In an alternative embodiment, the state data is received as state data packets (e.g., in the telemetry data stream <NUM>) at a first state data packet rate. In such embodiments, the method may further include sending feedback including a second phase delta to the second entity (e.g., in the phase delta stream <NUM>). The second phase delta includes timing differences between when state data packets are sent from the second entity to the first entity. The method may further include receiving the state data packets at a second state data packet rate, the second state data packet rate being dependent on the second phase delta. For example, the packet rate of the stream <NUM> may be adjusted as appropriate.

In some embodiments, the first phase delta is received in a state data packet (e.g., in a packet in the data stream <NUM>) and the second phase delta is sent in a consumable data packet (e.g., in a packet in the data stream <NUM>).

In some embodiments, the consumable data packets are telemetry packets, the second entity is a consumer of telemetry packets, and the second entity is a producer of video packets.

In some embodiments, the acts of method <NUM> are performed iteratively, such that consumable packet rates vary over time as phase deltas vary.

Referring now to <FIG>, a method <NUM> is illustrated. The method <NUM> may be practiced in a computing environment, and includes acts for receiving streamed data packets from a producer to a consumer, the data packets being dependent on state data from the consumer. The method <NUM> includes, at a second entity, receiving consumable data packets from a first entity at a first consumable packet rate (act <NUM>). For example, the entity <NUM> may receive packets in the data stream <NUM> at a particular packet rate.

The method <NUM> further includes, computing transmission jitter based on timing information in the consumable data packets (act <NUM>). For example, the second entity can use timing information in the data packets in the data stream, along with knowledge of when the data packets were received, to compute transmission jitter.

The method <NUM> further includes, computing a first phase delta based on the jitter, the first phase delta defining timing differences between when consumable data packets are sent from the second entity (act <NUM>).

The method <NUM> further includes, providing the first phase delta to the first entity (act <NUM>). For example, this phase delta information could be provided from the second entity <NUM> to the first entity <NUM> in the data stream <NUM>.

The method <NUM> further includes, receiving from the first entity consumable data packets at a second consumable packet rate, the second consumable packet rate being dependent on the first phase delta (act <NUM>). For example, the packet rate of the packets in the stream <NUM> are adjusted according to the phase delta information.

The method <NUM> may be practiced where the data in the consumable data packets is dependent on state data of the second entity provided to the first entity. For example, state data in the data stream <NUM> may be provided by the second entity <NUM> to the first entity <NUM>, and used to generate the data in the data stream <NUM>. In some such embodiments, the consumable data packets are video packets, the second entity is a consumer of video packets, the first entity is a producer of video packets, and the state data comprises telemetry data. Alternatively or additionally, in some such embodiments, the state data is sent as state data packets at a first state data packet rate. In such embodiments, the method may further include receiving feedback including a second phase delta from the first entity, the second phase delta comprising timing differences between when state data packets are sent from the second entity to the first entity; and sending the state data packets to the first entity at a second state data packet rate, the second state data packet rate being dependent on the second phase delta. Some such embodiments may be practiced where the first phase delta is sent in a state data packet (e.g., in packets in the data stream <NUM>); and the second phase delta is received in a consumable data packet (e.g., in packets in the data stream <NUM>).

In some embodiments of the method <NUM>, the consumable data packets are telemetry packets, the second entity is a consumer of telemetry packets, and the second entity is a producer of video packets.

The method <NUM> may be practiced where the acts are performed iteratively, such that consumable packet rates vary over time as phase deltas vary.

When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a computer, the computer properly views the connection as a transmission medium. Transmissions media can include a network and/or data links which can be used to carry desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.

Those skilled in the art will appreciate that the invention may be practiced in network computing environments with many types of computer system configurations, including, personal computers, desktop computers, laptop computers, message processors, hand-held devices, multi-processor systems, microprocessor-based or programmable telephones, PDAs, pagers, routers, switches, and the like.

Claim 1:
A computer system (<NUM>) enabled to send consumable data packets at variable packet rates based upon measured latency, the system comprising:
one or more processors; and
one or more computer-readable media having stored thereon instructions that are executable by the one or more processors (<NUM>) to configure the computer system (<NUM>) to send streamed data packets from a producer (<NUM>) to a consumer (<NUM>), including instructions that are executable to configure the computer system (<NUM>) to perform at least the following:
at a first entity (<NUM>, <NUM>), sending consumable data packets from the first entity to a second entity (<NUM>, <NUM>) at a first consumable packet rate (<NUM>), each consumable data packet comprising a timestamp enabling the consumer to identify when the each consumable data packet had been sent by the first entity;
at the first entity, receiving a first phase delta data stream from the second entity, wherein the first phase delta data stream is computed at least from transmission jitter, computed from timing information in the consumable data packets, including the included timestamps and information identifying when the each consumable data packet was received at the second entity using a rolling window in order to keep track and collect statistics of the packets arrivals in relation to fixed deadlines, wherein the first delta data stream defines timing differences between when consumable data packets are sent from the first entity by means of a median value derived from applying rolling windows processing of the said timing values; at the first entity,
computing a second consumable packet rate based upon the received first phase delta data stream; and
sending from the first entity consumable data packets at the second consumable packet rate, the second consumable packet rate being dependent on the first phase delta data stream.