In one example, a streaming server or a client device may measure an end-to-end time for a streaming operation in real time. The streaming server may store an input time associated with a user input and a frame presentation time associated with the frame output. The streaming server automatically may correlate the user input to the frame output generated at the streaming server in a frame generation process. The streaming server may calculate an end-to-end time based on the input time and the frame presentation time. The streaming server may adjust the frame generation process based on the end-to-end time.

BACKGROUND

A user device may connect to a streaming server to interact with a network service, such as a multiple user online game. The streaming server may generate content based on a user input. The user may provide an input to a client device, which may then be forwarded to the streaming server across a data network. For example, a user streaming a game may press a button on a client input device to make an avatar in the game jump. The client device may forward the user input to a gaming service.

The streaming server may then send media content, such as video or audio content, back across the data network for presentation to the user. The client device may present the media content as a series of discrete frames. In the previous example, the gaming service may send a series of discrete frame showing the avatar jumping.

SUMMARY

Examples discussed below relate to measuring an end-to-end time for a streaming operation in real time. A streaming server may store an input time associated with a user input and a frame presentation time associated with the frame output. The streaming server automatically may correlate the user input to the frame output generated at the streaming server in a frame generation process. The streaming server may calculate an end-to-end time based on the input time and the frame presentation time. The streaming server may adjust the frame generation process based on the end-to-end time.

DETAILED DESCRIPTION

An end-to-end time for a streaming operation describes the time from the entry of a user input to the display of the frame resulting from the user input. Previously, to measure the end-to-end time for the streaming operation, a user would enter a user input at a client device. The user would then identify when the client device displays the reaction to the user input. The variability of which frame resulted from the user input may prevent the client device or a streaming server executing the streaming operation from automatically calculating an end-to-end time for the streaming operation. By using a probability calculation, the streaming server may automatically correlate the user input to the frame output. The streaming server or the client device may then connect an input time identifying when the user input was received with a frame presentation time identifying when the frame output was displayed to the user in order to calculate the end-to-end time.

In one example, a streaming server or a client device may measure an end-to-end time for a streaming operation in real time by correlating a user input to a frame output. The streaming server may store an input time associated with a user input and a frame presentation time associated with the frame output. The streaming server automatically may correlate the user input to the frame output generated at by the streaming server in a frame generation process. The same server may perform the correlation and the frame generation process, or these activities may be distributed across multiple servers. The streaming server may calculate an end-to-end time based on the input time and the frame presentation time. The streaming server may adjust the frame generation process based on the end-to-end time.

The client device may store an input time associated with a user input and a frame presentation time associated with a frame output generated at a streaming server in a frame generation process. The client device may receive the frame output from the streaming server. The client device may identify a correlation by the streaming server of the frame output to the user input. The client device may calculate an end-to-end time based on the input time and the frame presentation time to the user. The client device may present the end-to-end time to a user.

FIG. 1illustrates, in a block diagram, one example of a streaming network100. A user may use a client device110to access a streaming server120, such as a multi-user interactive gaming service, via a data network connection130. The client device110may be a personal computer, a laptop, a tablet, a mobile phone, a game console, a smart watch, or other computing device used by the user to access the streaming server. The streaming server120may be implemented on a single server or a distributed set of servers, such as a server farm. The data network connection130may be an internet connection, a wide area network connection, a local area network connection, or other type of data network connections. The client device110may use a general use application or a dedicated streaming application, such as a gaming application, to access the streaming server120.

The client device110may implement an input capture digitizer111that creates a data representation of an input provided by the user via an input device. The client device110may implement an input processing and transmission module112to process the input data and to convert the input data into a format for transmission across the data network130. The client device110may then transmit the input data across the data network130to the streaming server120.

The streaming server120may receive the input data transmission in an input processing module121to convert the input data into a format for processing by an application processing module122, such as a game processing module. The application processing module122may use the input data to generate media data, such as audio or video data, for presentation to the user on the client device110. The streaming server120may implement an encoding module123that encodes the media data for transmission. The streaming server120may implement a packetization and transmission module124to format the media data into media packets for transmission across the data network130to the client device110.

The client device110may receive the media packets in a frame generation module113. The frame generation module113may restore a media frame from the media packet. The client device110may implement a decoding module114to decode the media frame for presentation to the user. The client device110may implement a display driver115to display the media frame to the user.

The display driver115, the application processing module122, and the input capture digitizer111may each introduce native latency into the operation of the application. The native latency is latency that is present whether the application is operated on a single device or is operated across multiple devices. The other modules may each introduce an additional network based latency. The network based latency results from distributing the operation across multiple devices on a network. The network based latency may be inherent to the vagaries of the network or may be the result of network adjacent operations, such as encoding, decoding, and packetization.

FIG. 2illustrates a block diagram of an exemplary computing device200which may act as a streaming server that operates a streaming server or a client device. The computing device200may combine one or more of hardware, software, firmware, and system-on-a-chip technology to implement a streaming server that operates a streaming server or a client device. The computing device200may include a bus210, a processor220, a memory230, a data storage240, an input device250, an output device260, and a communication interface270. The bus210, or other component interconnection, may permit communication among the components of the computing device200.

The processor220may include at least one conventional processor or microprocessor that interprets and executes a set of instructions. The processing core220may be configured to automatically correlate the user input to the frame output generated at the streaming server in a frame generation process. The processing core220may specify a causal frame number indicating a frame quantity immediately after the user input to disregard as being causal. The processing core220may choose a different causal frame number based on a latency accuracy of the end-to-end time, as determined by user feedback. The processing core220may calculate an end-to-end time based on the input time and the frame presentation time.

The processing core220may adjust the frame generation process based on the end-to-end time. On the streaming server side, the processing core220may increase or decrease an encoding complexity for the frame generation process based on the end-to-end time to a user. For example, the processing core220may adjust the frame generation process by adjusting a refresh rate for the frames based on the end-to-end time. Alternately, the processing core220may adjust the frame generation process by adjusting a rendering complexity for a frame, reducing or increasing resolution. On the client device side, the processing core220may reroute to an alternate streaming server based on the end-to-end time to a user.

The processing core220may generate a latency log storing an input identifier, the input time, a frame identifier, the frame presentation time, and the end-to-end time. The processing core220may calculate a client input processing time based on the input time identifying when the user input is received and a client transmission time identifying when the user input is transmitted from the client device. The processing core220may calculate an upstream transmission time based on a server reception time identifying when the user input is received and the client transmission time. The processing core220may calculate a server frame generation time based on a server reception time and a server transmission time identifying when the fame output is transmitted from the streaming server. The processing core220may calculate a downstream transmission time based on a server transmission time and a client reception time identifying when a frame output is received by the client device. The processing core220may calculate a client frame processing time based on the frame presentation time identifying when the frame is presented to the user and the client reception time.

The processing core220may identify an additional input time for an additional user input. The processing core220automatically may correlate the additional user input to the frame output generated at the streaming server in the frame generation process. The processing core220may calculate an additional end-to-end time based on the additional input time and the frame presentation time. The processing core220may express an aggregate input end-to-end time as at least one of a minimum end-to-end time, a maximum end-to-end time, or an average end-to-end time over multiple user inputs.

The memory230may be a random access memory (RAM) or another type of dynamic data storage that stores information and instructions for execution by the processor220. The memory230may also store temporary variables or other intermediate information used during execution of instructions by the processor220. The memory230may be configured to store an input time associated with a user input and a frame presentation time associated with a frame output. The memory230may record a server reception time upon receiving the user input. The memory230may record a server transmission time upon sending the frame output from the streaming server to a client device. The memory230may record a server reception time upon receiving the user input.

The data storage240may include a conventional ROM device or another type of static data storage that stores static information and instructions for the processor220. Data user for calculation may be stored in the memory230at runtime or in the persistent data storage240for offline processing. The data storage240may include any type of tangible machine-readable medium, such as, for example, magnetic or optical recording media, such as a digital video disk, and its corresponding drive. A tangible machine-readable medium is a physical medium storing machine-readable code or instructions, as opposed to a signal. Having instructions stored on computer-readable media as described herein is distinguishable from having instructions propagated or transmitted, as the propagation transfers the instructions, versus stores the instructions such as can occur with a computer-readable medium having instructions stored thereon. Therefore, unless otherwise noted, references to computer-readable media/medium having instructions stored thereon, in this or an analogous form, references tangible media on which data may be stored or retained. The data storage240may store a set of instructions detailing a method that when executed by one or more processors cause the one or more processors to perform the method. The data storage240may also be a database or a database interface for storing a latency log.

In a client device, the input device250may include one or more conventional mechanisms that permit a user to input information to the computing device200, such as a keyboard, a mouse, a voice recognition device, a microphone, a headset, a touch screen252, a touch pad254, a gesture recognition device256, etc. The output device260may include one or more conventional mechanisms that output information to the user, including a display screen262, a printer, one or more speakers264, a headset, a vibrator, or a medium, such as a memory, or a magnetic or optical disk and a corresponding disk drive.

The communication interface270may include any transceiver-like mechanism that enables computing device200to communicate with other devices or networks. The communication interface270may include a network interface or a transceiver interface. The communication interface270may be a wireless, wired, or optical interface. In a streaming server, the communication interface270may send the end-to-end time to a client device for presentation to a user. The communication interface270may receive an input identifier and the input time from a client device. The communication interface270may send a frame identifier for the frame output to a client device to identify the frame presentation time for the frame output. The communication interface270may receive a client transmission time from a client device identifying when the user input is transmitted from the client device to the streaming server. The communication interface270may receive from a client device a client reception time identifying when a frame output is received by the client device from the streaming server.

The computing device200may perform such functions in response to processor220executing sequences of instructions contained in a computer-readable medium, such as, for example, the memory230, a magnetic disk, or an optical disk. Such instructions may be read into the memory230from another computer-readable medium, such as the data storage240, or from a separate device via the communication interface270.

FIG. 3illustrates, in a timing diagram, one example of a streaming interaction300as a function of time T. A client device310may use a streaming server320to use an input α330to generate a frame β340. The client device310may receive one or more inputs330at the input capture digitizer312at an input time tI. The client device310may use a client network interface314to transmit an input packet at a client transmission time tCT.

The streaming server320may receive the input packet in a server network interface322at a server reception time tSR. The server network interface322may pass the input data from the input packet to an application324for processing at an application reception time tA. The application324may generate a media frame to be shown to the user, previewing the media frame in a server display326at server display time tSD. The application may pass the media frame to an encoder328of the streaming server320at frame generation time tG. The encoder328may encode the data at encoding time tEbefore sending the encoded packet to the server network interface322. The server network interface322may transmit the encoded packet back across the data network at a server transmission time tTR.

The client network interface314may receive the encoded packet at client reception time tCR. A decoder316may decode the encoded packet to generate a media frame to present to the user at decoding time tD. A client display318may present the media frame to the user at frame presentation time tFP. The client device310and the streaming server320may record each of these times to calculate and analyze various latencies in the overall process. To more efficiently analyze the process, the streaming server may correlate the user input received at application reception time tAwith a frame output generated at frame generation time tGto make a real-time end-to-end determination of the overall timing.

FIG. 4illustrates, in a block diagram, one example of a streaming server architecture400. The client device410may receive a user input420from a user. The client device410may generate an input packet412having an input identifier identifying the user input, input data describing the user input, and an input time representing the time that the user input was captured. The client device410may send the input packet412to the streaming server430. The streaming server430may deliver432the user input to an update loop440to create an updated frame. The update loop440may use the input packet412to generate442a frame represented by a frame identifier for presentation to the user. The streaming server430may correlate the input identifier to the frame identifier, indicating that the frame is a result of the user input.

The streaming server430may generate a frame packet434having a frame identifier and frame data describing a media frame for presentation to a user. The client device410may display414the media frame to the user. The client device410may record the frame presentation time that the media frame is displayed to the user. The client device410may generate a display report416having a frame acknowledgement indicating that the frame has been received, the frame identifier, and the frame presentation time. Based on the correlation between the input identifier and the frame identifier, the streaming server430or the client device410may calculate the end-to-end time to determine the latency covering from entry of the user input to presentation of the frame.

FIG. 5illustrates, in a block diagram, one example of an input packet500. The input packet500may have a server identifier510indicating the streaming server that is the target of the input packet500. The input packet500may have a client device identifier520indicating the client device sending the input packet500. The input packet500may have an input identifier530referencing the user input. The input packet500may have an input data set540describing the content of the user input. The input packet500may have an input time550indicating the time that a user input is captured by the client device.

FIG. 6illustrates, in a block diagram, one example of a frame packet600. The frame packet600may have a client device identifier610indicating the client device that is the target of the frame packet600. The frame packet600may have a streaming server identifier620indicating the streaming server sending the frame packet600. The frame packet600may have an input identifier630referencing the user input that the streaming server has correlated to the media frame so that the client device may produce a latency log describing any resulting latency timing. The frame packet600may have a frame identifier640referencing the media frame that the client device is to present. The frame packet600may have a frame data set650describing the content of the media frame.

FIG. 7illustrates, in a block diagram, one example of a display report700. The display report700may have a server identifier710indicating the streaming server that is the target of the display report700. The display report700may have a client device identifier720indicating the client device sending the display report700. The display report700may have an input identifier730referencing the user input used to calculate the latency log. The display report700may have a frame identifier740referencing the media frame that the client device presented. The input packet700may have a client transmission (TRX) time750indicating the time the input packet is sent from the client device. The display report700may have a client reception (RX) time760indicating the time the frame packet is received by the client device. The display report700may have a frame presentation time770indicating the time the frame output is presented to the user. If a latency log is generated at the client device, the display report700may have a latency log780describing operation latency to varying degrees of granularity.

FIG. 8illustrates, in a block diagram, one example of a latency log800. The latency log800may have a client device identifier802indicating the client device capturing the user input. The latency log800may have a streaming server identifier804indicating the streaming server generating the frame output. The latency log800may have an input identifier806referencing the user input that has caused the streaming server to generate a frame output. The latency log800may have an input time808describing the time that the user input was captured by the client device. The latency log800may have one or more additional input identifiers810referencing one or more additional user inputs that have also contributed to the frame output generated by the streaming server. The latency log800may have one or more additional input times812describing each time that each additional user input was captured by the client device. The latency log800may have a frame identifier814referencing the frame output sent to the client device for presentation to the user. The latency log800may have a frame presentation time816describing the time that the frame output was presented by the client device to the user.

To generate the latency log800, the client device or the streaming server may calculate a number of timing metrics to measure operational performance of the streaming server. The latency log800may have an end-to-end time818based on subtracting the input time808from the frame presentation time816. The end-to-end time818represents the length of time for a streaming operation from the capture of a user input to presentation of the frame output. The latency log800may have one or more additional end-to-end times820based on subtracting an additional input time812from the frame presentation time816. The additional end-to-end time820represents the length of time for a streaming operation from the capture of an additional user input to presentation of the frame output. The latency log800may have an aggregate input end-to-end time822representing the length of time for a streaming operation generated from multiple inputs. The aggregate input end-to-end time822may be based on a minimum end-to-end time, a maximum end-to-end time, or an average end-to-end time for each contributing user input. A contributing user input is a user input that the streaming server uses in generation of the frame output.

The client device or the streaming server may further calculate a number of additional timing metrics to measure component aspects of the operational performance of the streaming server. The latency log800may have a client input processing time824based on subtracting the input time808from the client transmission time. The client input processing time824represents the length of time for the client device to process a user input for transmission to the streaming server. The latency log800may have an upstream transmission time826based on subtracting the client transmission time from the server reception time. The upstream transmission time826represents the length of time for the data network to transfer a data packet from the client device to the streaming server. The latency log800may have a server frame generation time828based on subtracting the server reception time from the server transmission time. The server frame generation time828represents the length of time for the streaming server to generate a frame output for transmission upon receiving the user input. The latency log800may have a downstream transmission time830based on subtracting the server transmission time from the client reception time. The downstream transmission time830represents the length of time for the data network to transfer a data packet from the streaming server to the client device. The latency log800may have a client frame processing time832based on subtracting the client reception time from the frame presentation time816. The client frame processing time832represents the length of time for the client device to present the frame output to the user upon receiving the frame output from the data network.

FIG. 9illustrates, in a timing diagram, one example of an input-frame correlation900. During a given timeline910, an output device of the client device, such as a display, may present a series of frames912at a regular interval as provided by a streaming server. The client device may receive a user input914in an input device of the client device, such as a touch screen or a gesture capture device. The user input914may cause the streaming server to generate a frame912based on the user input914. The probability920that a frame912was caused by a user input914may increase the longer the time after the user input914. The streaming server may calculate the probability920that a user input is causal for a given frame based on previous time trials. The streaming server may factor into the probability calculation different generation factors, such as process type and server form factor. When the probability920for a frame912exceeds a causal threshold, the streaming server may correlate that frame912to the user input914. To more efficiently select a correlating output frame, the streaming server may specify a causal frame number916indicating a frame quantity immediately after the user input to disregard as being causal. The streaming server may adjust the causal frame number916based on a process type of the frame generation process, such as interactive video or online gaming.

FIG. 10illustrates, in a flowchart, one example of a method1000of sending a latency log from a client device. The client device may capture a user input from a user (Block1002). The client device may record the input time for the user input (Block1004). The client device may send an input packet having the input identifier, the user input, and the input time to the streaming server (Block1006). The client device may receive from the streaming server a frame packet having a frame output automatically correlated to the user input at the streaming server and a frame identifier for the frame output to identify the frame presentation time for the frame output (Block1008). The client device may display the frame output to the user (Block1010). The client device may record a frame presentation time for the frame output (Block1012). The client device may store the input time associated with the user input and the frame representation time associate with frame output generated at a streaming server in the frame generation process (Block1014).

The client device may generate a latency log storing at least one of the input identifier, the input time, the frame identifier, the frame presentation time, and the end-to-end time (Block1016). The client device may send a display report having at least one of the frame presentation time and the latency log to the streaming server (Block1018). The client device may present the end-to-end time to the user (Block1020). If the latency of the streaming operation is greater than a latency threshold (Block1022), the client device may request an adjustment to the frame generation process based on the end-to-end time (Block1024).

FIG. 11illustrates, in a flowchart, one example of a method1100of receiving a latency log in a streaming server. The streaming server may specify a causal frame number indicating a frame quantity immediately after the user input to disregard as being causal (Block1102). The streaming server may receive an input packet having a user input, an input identifier for the user input, and an input time for the user input from a client device (Block1104). The streaming server may execute a frame generation process to generate a frame output based on the user input (Block1106). The streaming server may automatically correlating the user input to the frame output generated in a frame generation process (Block1108). If the streaming server identifies an additional user input related to the frame output (Block1110), the streaming server may automatically correlate the additional user input to the frame output (Block1108).

The streaming server may send to the client device a frame packet having the correlated user input identifiers, a frame output, and a frame identifier for the frame output to identify the frame presentation time for the frame output (Block1112). The streaming server may receive from the client device a display report having an input identifier for the user input, a frame identifier for the frame output, a frame presentation time for the frame output, and a latency log (Block1114). If the latency of the streaming operation is greater than a latency threshold (Block1116), the streaming server may adjust the frame generation process based on the end-to-end time (Block1118). The streaming server may choose a different causal frame number based on a latency accuracy of the end-to-end time (Block1120).

FIG. 12illustrates, in a flowchart, one example of a method1200of receiving a latency log in a client device. The client device may capture a user input from a user (Block1202). The client device may record the input time for the user input (Block1204). The client device may send an input packet having the input identifier, the user input, and the input time to the streaming server (Block1206). The client device may record a client transmission time identifying when the user input is transmitted to the streaming server (Block1208). The client device may receive from the streaming server a frame packet having a frame output automatically correlated to the user input at the streaming server and a frame identifier for the frame output to identify the frame presentation time for the frame output (Block1210). The client device may record a client reception time identifying when a frame output is received (Block1212). The client device may display the frame output to the user (Block1214). The client device may record a frame presentation time for the frame output (Block1216). The client device may send a display report having at least one of input identifier, the frame identifier, the client transmission time, the client reception time, and the frame presentation time (Block1218). The client device may receive from the streaming server a latency log describing the end-to-end time and other performance timing metrics (Block1220). The client device may present the end-to-end time to the user (Block1222). If the latency of the streaming operation is greater than a latency threshold (Block1224), the client device may reroute to an alternate streaming server based on the end-to-end time to a user (Block1226). The client device may request an adjustment of a refresh rate based on the end-to-end time (Block1228)

FIG. 13illustrates, in a flowchart, one example of a method1300of sending a latency log from a streaming server. The streaming server may specify a causal frame number indicating a frame quantity immediately after the user input to disregard as being causal (Block1302). The streaming server may receive an input packet having a user input, an input identifier for the user input, and an input time for the user input from a client device (Block1304). The streaming server may record a server reception time upon receiving the user input (Block1306). The streaming server may execute a frame generation process to generate a frame output based on the user input (Block1308). The streaming server may automatically correlating the user input to the frame output generated in a frame generation process (Block1310). If the streaming server identifies an additional user input related to the frame output (Block1312), the streaming server may automatically correlate the additional user input to the frame output (Block1310).

The streaming server may send to the client device a frame packet having the correlated user input identifiers, a frame output, and a frame identifier for the frame output to identify the frame presentation time for the frame output (Block1314). The streaming server may record a server transmission time upon sending the frame output to a client device (Block1316). The streaming server may receive from the client device a display report having an input identifier for the user input, a frame identifier for the frame output, a frame presentation time for the frame output, a client transmission time identifying when the user input is transmitted from the client device to the streaming server, and a client reception time identifying when the frame output is received by the client device from the streaming server (Block1318).

The streaming server may generate a latency log storing at least one of the input identifier, the input time, the frame identifier, the frame presentation time, and the end-to-end time (Block1320). The streaming server may send the latency log having the end-to-end time to the client device for presentation to the user (Block1322). If the latency of the streaming operation is greater than a latency threshold (Block1324), the streaming server may adjust a rendering complexity for the frames based on the end-to-end time to a user (Block1326). The streaming server may adjust a refresh rate for the frames based on the end-to-end time (Block1328). The streaming server may choose a different causal frame number based on a latency accuracy of the end-to-end time (Block1330).

FIG. 14illustrates, in a flowchart, one example of a method of generating timing metrics, either at the client device or at the streaming server. The client device or the streaming server may identify an input time for a user input (Block1402). The client device or the streaming server may identify a frame presentation time for a frame output (Block1404). The streaming server or the client device may identify a correlation by the streaming server of the frame output to the user input (Block1406). The client device or the streaming server may calculate an end-to-end (E2E) time based on the input time and the frame presentation time (Block1408). If an additional user input is correlated to the frame output (Block1410), the client device or the streaming server may identify an additional input time for the additional user input (Block1412). The streaming server or the client device may identify an additional correlation by the streaming server of the frame output to the additional user input (Block1414). The client device or the streaming server may calculate an additional end-to-end time based on the additional input time and the frame presentation time of the frame output (Block1416). The client device or the streaming server may express an aggregate input end-to-end time based on at least one of a minimum end-to-end time, a maximum end-to-end time, and an average end-to-end time over multiple user inputs (Block1418).

The streaming server may make further calculations to determine component aspect metrics for the operation. The client device may also make these calculations, provided the proper data. The streaming server may calculate a client processing time based on the input time and a client transmission time (Block1420). The streaming server may calculate an upstream transmission time based on a client transmission time and a server reception time (Block1422). The streaming server may calculate a server frame generation time based on a server reception time and a server transmission time (Block1424). The streaming server may calculate a downstream transmission time based on a server transmission time and a client reception time (Block1426). The streaming server may calculate a client frame processing time based on a frame presentation time and a client reception time (Block1428).

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms for implementing the claims.

Examples within the scope of the present invention may also include computer-readable storage media for carrying or having computer-executable instructions or data structures stored thereon. Such computer-readable storage media may be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such computer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic data storages, or any other medium which can be used to store desired program code means in the form of computer-executable instructions or data structures, as opposed to propagating media such as a signal or carrier wave. Computer-readable storage media explicitly does not refer to such propagating media. Combinations of the above should also be included within the scope of the computer-readable storage media.

Examples may also be practiced in distributed computing environments where tasks are performed by local and remote processing devices that are linked (either by hardwired links, wireless links, or by a combination thereof) through a communications network.

Although the above description may contain specific details, they should not be construed as limiting the claims in any way. Other configurations of the described examples are part of the scope of the disclosure. For example, the principles of the disclosure may be applied to each individual user where each user may individually deploy such a system. This enables each user to utilize the benefits of the disclosure even if any one of a large number of possible applications do not use the functionality described herein. Multiple instances of electronic devices each may process the content in various possible ways. Implementations are not necessarily in one system used by all end users. Accordingly, the appended claims and their legal equivalents should only define the invention, rather than any specific examples given.