Patent ID: 12200401

DETAILED DESCRIPTION

Various embodiments and aspects will be described with reference to details discussed below, and the accompanying drawings will illustrate the various embodiments. The following description and drawings are illustrative and are not to be construed as limiting. Numerous specific details are described to provide a thorough understanding of various embodiments. However, in certain instances, well-known or conventional details are not described in order to provide a concise discussion of embodiments.

Reference in the specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in conjunction with the embodiment can be included in at least one embodiment. The appearances of the phrase “in one embodiment” in various places in the specification do not necessarily all refer to the same embodiment. The processes depicted in the figures that follow are performed by processing logic that comprises hardware (e.g. circuitry, dedicated logic, etc.), software, or a combination of both. Although the processes are described below in terms of some sequential operations, it should be appreciated that some of the operations described may be performed in a different order. Moreover, some operations may be performed in parallel rather than sequentially.

Audio-video conferencing provides for the reception and transmission of audio and/or video signals (usually as streaming content) by user devices or systems (e.g., at different locations), for communication between users in real-time. In some cases, two users may utilize audiovisual conferencing to communicate with each other in one-to-one communication at their respective devices. In other cases, multiway audiovisual conferencing may be utilized by more than two users to participate in a real-time, group conversation.

In some systems of multiway audiovisual conferencing, network degradation may be experienced by one or more of the participant systems. In the one-to-one audiovisual conferencing example, the two participating systems may both switch from a high quality/bitrate stream to a lower quality/bitrate stream, in order to accommodate for the bandwidth degradation. However, when there are more than two participants in an audiovisual conference, switching all of the participant devices to a lower quality/bitrate content stream because a single participant device has bandwidth constraints may result in a degradation of the audiovisual conference experience for all of the participants. Switching becomes more complicated when the participant systems are different types of systems, such as an iPhone executing FaceTime on a version of iOS and a smart phone executing a version of an Android operating system.

FIG.1shows an example of a conferencing environment10that can be used with one or more of the embodiments described herein. Not all of the depicted components may be used in all implementations of embodiments, however, and one or more implementations may include additional or different components than those shown in the figure. Variations in the arrangement and type of the components may be made in accordance with the embodiments described herein, and additional components, different components, or fewer components may be provided in accordance with the embodiments described herein.

The conferencing environment10includes data processing systems15,17, and19, one or more computer networks12(e.g., the Internet), and one or more AV conferencing servers14. The one or more networks12may communicatively (directly or indirectly) couple, for example, any two or more of the data processing systems15,17and19and the one or more servers14to allow for the exchange of data among the systems and the one or more servers. In one or more implementations, the one or more networks12may be an interconnected network of devices that may include, and/or may be communicatively coupled to, the Internet. For explanatory purposes, the conference environment10is illustrated inFIG.1as including systems15,17and19and one or more servers14; however, the conferencing environment10may include any number of electronic devices or data processing systems and any number of servers, often millions of devices or systems dispersed over large geographic regions. In addition, the conferencing environment10can include different types of servers that perform messaging operations (e.g., transmission of push notifications or push messages between the participant systems) and registration operations (e.g., FaceTime registration servers used to register users of the FaceTime AV conferencing system) and other operations used to set up conferencing among the participant systems, such as the data processing systems15,17, and19. The one or more servers14may be a cloud of servers that are used to facilitate AV conferencing among data processing systems such as the systems15,17, and19. In one embodiment, the one or more servers14can include the server108shown and described in U.S. Pat. No. 10,931,725 which describes a server used in a group FaceTime architecture or infrastructure; other servers (e.g., other selective forwarding servers) described in that US patent can also be used in the embodiments described in this disclosure.

The data processing systems15,17, and19may be, for example, a desktop computer, a portable computing device such as a laptop computer, a tablet computer (e.g., an iPad), a smart phone (e.g., an iPhone or an Android smart phone), a smart speaker (e.g., an Echo or Echo Show from Amazon), a peripheral device (e.g., a digital camera, headphones), a gaming device or system, a wearable device such as a headmounted display or glasses or smartwatch and the like, or any other appropriate device or consumer electronic device that includes, for example, one or more wireless interfaces, such as WLAN radios, WiFi radios, cellular radios, Bluetooth radios, Zigbee radios, near field communication (NFC) radios, and/or other wireless radios. These data processing systems can be configured to participate in audiovisual conferencing, for example, where the data processing systems15,17, and19(also referred to as participant devices or participant systems) may participate in a group conversation in which video and/or audio content streams are transmitted between the participant devices in the AV conference. In the context of the embodiments described herein, an AV conference will be understood to mean a communication where at least one of audio or video is transmitted, as streaming content, between the participant systems; normally, in one embodiment, both audio and video are transmitted (assuming at least some participant systems are equipped with a camera), but in some situations only audio may be transmitted when network bandwidth degrades to the point that only audio transmission can be supported from some or all participant systems. In one embodiment, the transmission of audio only can occur at any point during an AV conference (or even during the entirety of the AV conference). In one embodiment, at least some of the participant systems can have an AV conferencing application (e.g., the FaceTime application) installed on the participant system; the AV conferencing application on the sending device (e.g., data processing system15) can facilitate in transmitting streaming content for receipt by at least one other participant that also has the same AV conferencing application (or a version of that application) with the same media capabilities as the sending device. In one embodiment, one or more participants may not include a dedicated AV conferencing application (e.g., they do not include the FaceTime application), and they may use a web browser (or similar application) to participate in the AV conference. In this case, such participant systems can be referred to as web participants, and they may use known protocols, such as webRTC or a quick relay protocol to participate in the AV conference. In one embodiment, the audio and video content can be encrypted with end to end encryption so that the intervening servers along the path cannot decrypt the content.

A method according to one embodiment will now be described while referring toFIGS.2A,2B and2C. In this method, a new participant system (e.g., participant system107, which may be a web participant inFIG.2B) that has different media capabilities than the existing participant systems joins an ongoing AV conference (or at least an initiated AV conference) between existing participant systems, thereby causing the existing participant systems to create new streams with a codec that is common among the participant systems while the original two existing participant systems (e.g., participant systems103and109inFIG.2B) continue to send and receive content to each other using a different (e.g., better) codec. A codec is a system (e.g., software and hardware) that compresses content for transmission from a device and decompresses received content for presentation (e.g., display of images and/or playing of audio) at the device.FIG.2Ashows a method performed by one of the existing participant systems (e.g., participant system103), andFIG.2Cshows a method performed by a forwarding server (e.g., forwarding server105inFIG.2B). The method shown inFIG.2Acan begin after an initiation of the AV conference; this initiation can occur, for example, when a user at participant system103selects other participant(s) via an audiovisual conferencing application, and/or may initiate the audiovisual conference from a different mode of communication, such as group text messaging, a group voice call, etc. For example, the participants may be selected from contacts included within a contact address book stored on the participant system103. The user may initiate the audiovisual conference by selecting an appropriate user interface element provided by the audiovisual conferencing application or contact address book application or text messaging application or voice call application, thereby prompting the invited participants, at their respective devices (e.g., the participant systems107and109), to accept or decline participation in the audiovisual conference.

In operation51inFIG.2A, a first data processing system (e.g., participant system103inFIG.2B), transmits its codec capabilities (e.g., a first set of codec capabilities) for use in an AV conference; this transmission can be directed at and received by a selective forwarding server, such as selective forwarding server105inFIG.2B(or to a setup server that facilitates the AV conference). In one embodiment, the content of this transmission can then be forwarded, by the selective forwarding server, to other participant systems when they join the AV conference. The codec capabilities (transmitted in operation51) can be an exhaustive list of all available codecs (both audio and video) at the first data processing system that are available for use in the AV conference and can also include a list of all available encryption methods (for end-to-end encryption) and other available capabilities (e.g., loss recovery techniques) for use in the AV conference. In operation53, the first data processing system (e.g., the participant system103inFIG.2B) can receive a second set of codec capabilities from a second data processing system (e.g., participant system109inFIG.2B), and this received second set can be an exhaustive list of all available codecs (both audio and video) that are available for use in the AV conference and can also include a list of all available encryption methods (for end-to-end encryption) and other available capabilities (e.g., loss recovery techniques) for use in the AV conference. In operation55, the first data processing system (the participant system103inFIG.2B) can receive a request from the second data processing system (e.g., participant system109inFIG.2B) to receive a first stream of content (e.g., video) encoded with a first codec (that is one of the codecs in the set of capabilities transmitted in operation51and is one of the codecs in the second set of capabilities). In one embodiment, in response to the request received in operation55, the first data processing system can transmit, in operation57, a first stream for delivery through a forwarding server (e.g., the forwarding server105inFIG.2B) to the second data processing system. The first stream can be encoded by a high or the highest quality codec (based upon, for example, content resolution or other parameters) that is common to both the first data processing system and the second data processing system.

In the example shown inFIG.2B, the participant system103transmits, as part of operation57, the first stream115to the forwarding server105which, in turn, forwards the first stream as stream117to the participant system109as shown inFIG.2B. The first stream can be encoded, when circumstances allow, with the highest quality codec (e.g., first codec) that is common to both the first (participant system103) and the second (participant system109) systems to allow both participant systems to experience the best possible quality of video or other content. The participant system109can also transmit to the participant system103(through the forwarding server105) content that is encoded by the first codec that is common to the two systems103and109, which content is received, decompressed/decoded and presented by the participant system103.

When a third participant (e.g., participant system107inFIG.2B) joins the AV conference, the participant system103inFIG.2Breceives, in operation59inFIG.2A, a request (e.g., request or subscription121shown inFIG.2B) from the third participant for a second stream encoded with a second codec, which can be a common codec that is available at all three participants (participant systems103,107and109shown inFIG.2B) and is different than the first codec and can be a lower quality codec (in terms of one or more of picture quality or lossy behavior or resolution, etc.). The request121is forwarded by forwarding server105after the forwarding server receives the request or subscription119from the participant system107. In one embodiment, the third participant can be a web participant does not include the first codec used by participant systems103and109to send and receive the first stream115; in one embodiment, the forwarding server can attempt to verify that all participant systems include at least one common codec that is available at all participant systems in the AV conference. In response to the request in operation59, the first data processing system, in operation61, creates a second stream of content (the second stream123shown inFIG.2B) for the AV conference that is encoded by the second codec which can be the common codec. The second stream can be captured, in one embodiment, by the same camera on the first data processing system that also captures content (e.g., video) for the first stream. In operation63ofFIG.2A, the participant system103transmits the on-demand second stream123to the forwarding server105shown inFIG.2B. The second steam123is then forwarded by the forwarding server105as second stream125to the participant system107as shown inFIG.2B. The participant system103can continue to create and transmit the first stream115while transmitting the second stream123. The approach shown inFIGS.2A and2Ballows the participant systems103and109to continue to use a better codec than the common codec while permitting participant system107to join the AV conference. Thus, participant systems103and109can use the first codec to send and receive content (e.g., video) between them while sending and receiving content (e.g., video) from the participant system107using the second codec. WhileFIG.2Bshows the transmissions from participant system103, it will be appreciated that participant system103also receives content from participant systems107and109, and participant systems107and109also transmit content to the other systems so that each of the three participant systems inFIG.2Bsends content to the other participant systems and receives content from the other participant systems.

FIG.2Cshows a method that can be performed by a forwarding server (e.g., forwarding server105) when participant system107joins the AV conference between participant systems103and109. The method inFIG.2Ccan be performed while the first participant system (e.g., participant system103) performs the method shown inFIG.2A. In operation151shown inFIG.2C, the forwarding server can receive codec capabilities of the first data processing system (e.g., participant system103inFIG.2B); these codec capabilities can then be forwarded to other participant systems in the AV conference. In operation153inFIG.2C, the server can receive from a second data processing system (e.g., the participant system109inFIG.2B) a second set of codec capabilities (e.g., an exhaustive list of available codecs for the AV conference) and then transmit the second set of codec capabilities to the first data processing system. This exchange of codec capabilities allows each participant system to see what capabilities are available at each participant system to allow each participant system to select a desired stream encoded by a desired codec in the AV conference. In operation155inFIG.2C, the forwarding server can transmit to the first data processing system (e.g., participant system103inFIG.2B) a request, from the second data processing system, to receive a first stream that is encoded by a first codec on the first data processing system (where the first codec in the list of available codecs at the first data processing system). Then in operation157, the forwarding server receives the requested first stream for delivery to the second data processing system, and the forwarding server then forwards the first stream to the second data processing system. Of course, the forwarding server can also receive a request from the first data processing system to receive a stream from the second data processing system that is encoded with the first codec at the second data processing system (where the codecs used at the first and second data processing systems can be the same for the streams they send to each other), and the forwarding server can receive the stream from the second data processing system and forward that stream to the first data processing system.

In operation159, the forwarding server receives a request from a third data processing system (e.g., participant system107inFIG.2B) for a second stream encoded with a second codec that is common to the three participant systems inFIG.2B, and the forwarding server transmits the request to the first and second data processing systems (participant systems130and109). Then in operation161, the forwarding server receives the requested second stream (encoded with the second codec) and forwards the second stream to the third data processing system while the forwarding server continues to forward streams encoded with the first codec to the first and second data processing systems. This permits the participant systems103and109to continue to use a better codec (the first codec) than the common codec (second codec) while permitting participant system107to join the AV conference. In one embodiment, the first and second data processing systems can be devices that execute versions of the same operating system (e.g., versions of iOS) and share the same media capabilities and use the same AV conferencing application (e.g., FaceTime), while the third data processing system can be a device that executes a different operating system (e.g., an Android operating system) and does not include the FaceTime AV conferencing application.

In one embodiment, the participant systems that use the better codec can continue to do so while conditions (e.g., network bandwidth, battery levels, thermal status) allow this use; however, it is possible that conditions will change and require fallback approaches that switch to the use of the common codec.FIGS.3A and3Bshow one implementation of a fallback approach andFIGS.4A and4Bshow another implementation of a fallback approach.

In the method shown inFIG.3A, the participant systems are not required to provide signaling about the change in streams; rather, each participant is configured to be ready to switch to the use of the common codec in a fallback situation upon receipt of content encoded with the common codec. In the context of the environment shown inFIG.2B, at least one participant will already be using the common codec. In operation201inFIG.3Aeach participant system (e.g., participant systems210and216inFIG.3B) monitors its local state, which can include upload bandwidth, download bandwidth, thermal data about the thermal state of the system (e.g., is the device getting too hot while it has been sending multiple streams in the AV conference?), and battery state of one or more batteries (e.g., has the battery been drained below one or more thresholds?).

This monitoring can occur periodically over time during the AV conference. U.S. provisional patent application No. 63/041,549, filed Jun. 19, 2020 (by Hsien-Po Shiang, et. al. and entitled HIGH FREQUENCY PROBING FOR NETWORK BANDWIDTH ESTIMATION USING VIDEO DATA IN REAL-TIME VIDEO CONFERENCE) provides examples of how the bandwidth can be monitored during an AV conference. Thus, each participant system such as participant systems210,214and216inFIG.3Bcan monitor these states to determine whether a condition exists that requires a fallback to the common codec. For example, if network bandwidth (e.g., upload bandwidth) for the participant system degrades too much to support transmission of multiple codec streams or if the participant system is getting too hot to support transmission of multiple codec streams or the main battery has been drained too low to support transmission of the multiple codec streams, the participant system, in operation203inFIG.3A, switches from transmitting a first stream encoded with a first (better) codec to transmitting a second stream encoded with the common codec; at some point during this switch, the participant system will stop transmitting the first stream so that it can transmit only the second stream to all recipients. The participant system210in the example shown inFIG.3Bwill already be transmitting the second stream219to the participant system214(while also transmitting the first stream217to participant system216) before operation203inFIG.3A, so the switch can be performed by tagging or associating the second stream219with an identifier of the first stream217, and this will cause the forwarding server212to forward the second stream to the participant system216(which had previously subscribed to the first stream). The tagging or association of the second stream with the first stream's identifier will automatically forward the second stream219to the participant system216, and participant system210can stop transmitting the first stream217as soon as it begins tagging the second stream219with the identifier of the first stream. This tagging or association can be referred to as compound stream identifiers because the common codec stream is associated with two stream identifiers during the fallback situation. U.S. Pat. No. 10,931,725 provides further information about the use of compound stream identifiers for use in AV conferencing.

The fallback approach shown inFIGS.3A and3Ballow a participant system that is transmitting at least a common codec stream and a better codec stream to immediately switch to transmitting only the common codec stream in a fallback situation (e.g., network degradation) without requiring signaling or messages about the change. Thus, such a participant system can react very quickly to changing conditions if necessary. Once conditions improve (e.g., the network bandwidth improved) the participant system can revert back to transmission of both the common codec stream219and the first stream217again; each participant system can perform operation201after a fallback situation to determine if it can resume transmission of both streams. Reverting back to transmitting both streams requires the participant system to begin generating and transmitting the first stream (encoded with the better codec) and stop tagging the second stream with the identifier of the first stream (which is associated with the identifier of the first stream).

The fallback approach shown inFIGS.4A and4Buses messages or signaling to accomplish the fallback and does not use compound stream identifiers. In operation251inFIG.4A, each participant system monitors its local state; operation251can be similar to operation201inFIG.3A. When the monitoring shows that fallback is required, the participant system (e.g., participant system275) performs operation253by switching from transmitting both the better codec stream (the first stream) and the common codec stream to transmitting just the common codec stream. The method shown inFIG.4Acan operate in the context of the AV conference shown inFIG.4B. This switch involves sending a message (e.g., stop283message shown inFIG.4B) to the forwarding server (e.g., forwarding server277) which in turn transmits a message to the participant systems (e.g., participant system281) that have been receiving the first stream283from the participant system275) that sent the stop283message, and this message from the forwarding server includes information that first stream283will not continue to be transmitted from participant system275. Prior to operation253, the participant system had already been creating and sending the common codec stream285to participant system279through the forwarding server277, and the participant system had been transmitting the better codec stream283to the participant system281. When the fallback condition occurs, the participant system281receives a message (e.g., data based on stop283message) and then subscribes to the common codec stream by sending a subscription message (e.g., subscribe to common) to the forwarding server277. That subscription message causes the forwarding server277to begin forwarding the common codec stream to participant system281. In one embodiment, the participant system275may stop creating and transmitting the better codec stream283once it has transmitted the stop283message to the forwarding server277. The participants can continue to monitor their local states and resume transmission of the better codec stream when conditions allow this resumption. In one embodiment, resumption can involve publishing the availability of a new stream (the better codec stream) through the forwarding server which forwards the published availability to participant systems that can use the better codec stream; in turn these participant systems can subscribe to the better codec stream again.

Another aspect of this disclosure involves the use of the preferred or highest quality codec at each participant system that can take advantage of such use while using a common codec that is available for use at all participant systems in an AV conference. The common codec can be used as the primary codec for some participants that do not support the better codecs (with variations in bitrate for steams encoded with the common codec being used to deal with network bandwidth changes) and can also be used as a fallback codec for those participant systems that can support the better codecs. This aspect is shown inFIG.5A, which shows a method of an embodiment of this aspect, andFIG.5B, which shows an example of an AV conference environment that can use the method shown inFIG.5A. In operation301inFIG.5A, each participant system can exchange their list of codec capabilities (and optionally other media related capabilities) for use in the AV conference with all of the other participant systems (or at least a subset of all participant systems). For example, each participant system can publish their list of capabilities through one or more messages sent to a forwarding server, such as the forwarding server317shown inFIG.5B. The result of this exchange can provide each participant with the available codecs at other participants and allow each participant to select a codec from the other participants' list of available codecs. This enables operation303in which each participant can subscribe to a preferred or highest quality codec available at each participant (for those participants that support more than the common codec) while allowing for such participant systems to fallback to use of the common codec in fallback situations. The fallback situations can use the fallback approaches described above. In the example shown inFIG.5B, participant system310and participant system312each have the same set of codec capabilities, which include a common codec (codec X). Participant systems316and314each have the same set of codec capabilities which include the common codec. In this example, participant systems310and312can select a better codec, such as codecs A or B, when not in a fallback condition (e.g., when network bandwidth is too low) to transmit and receive streams between these systems310and312in the AV conference, and these participant systems310and312can transmit and receive common codec encoded streams to participants316and314. The participants316and314can transmit and receive the common codec streams from all other participants. The common codec stream can be used as a fallback stream for participants310and312when fallback conditions (e.g., reduced network bandwidth or thermal state too hot or battery level too low) exist.

The embodiments described herein can also use additional methods performed by server systems, such as forwarding servers, to match the different participant systems. For example, a server, in one embodiment, can manipulate, add, or remove media headers and control commands to match different participant systems, with different media capabilities, so they can join and maintain an AV conference. For example, a server can assist participant systems that are different with loss recovery, media synchronization, media attributes such as orientation of images/video, and media attributes such as audio power (e.g., volume) levels, and transport headers and additional encryption if needed.

FIG.6shows one example of a data processing system800, which may be used with one embodiment. For example, the system800may be implemented to provide a system or device that performs any one of the methods described herein. For example, the system800may be used to implement a participant system or a forwarding server. Note that whileFIG.6illustrates various components of a device, it is not intended to represent any particular architecture or manner of interconnecting the components as such details are not germane to the disclosure. It will also be appreciated that network computers and other data processing systems or other consumer electronic devices, which have fewer components or perhaps more components, may also be used with embodiments of the disclosure.

As shown inFIG.6, the device800, which is a form of a data processing system, includes a bus803which is coupled to a microprocessor(s)805and a ROM (Read Only Memory)807and volatile RAM809and a non-volatile memory811. The microprocessor(s)805may retrieve the instructions from the memories807,809,811and execute the instructions to perform operations described above. The microprocessor(s)805may contain one or more processing cores. The bus803interconnects these various components together and also interconnects these components805,807,809, and811to a display controller and display device813and to peripheral devices such as input/output (I/O) devices815which may be touchscreens, mice, keyboards, modems, network interfaces, printers and other devices which are well known in the art. Typically, the input/output devices815are coupled to the system through input/output controllers810. The volatile RAM (Random Access Memory)809is typically implemented as dynamic RAM (DRAM), which requires power continually in order to refresh or maintain the data in the memory.

The non-volatile memory811is typically a magnetic hard drive or a magnetic optical drive or an optical drive or a DVD RAM or a flash memory or other types of memory systems, which maintain data (e.g., large amounts of data) even after power is removed from the system. Typically, the non-volatile memory811will also be a random access memory although this is not required. WhileFIG.6shows that the non-volatile memory811is a local device coupled directly to the rest of the components in the data processing system, it will be appreciated that embodiments of the disclosure may utilize a non-volatile memory which is remote from the system, such as a network storage device which is coupled to the data processing system through a network interface such as a modem, an Ethernet interface or a wireless network. The bus803may include one or more buses connected to each other through various bridges, controllers and/or adapters as is well known in the art.

Portions of what was described above may be implemented with logic circuitry such as a dedicated logic circuit or with a microcontroller or other form of processing core that executes program code instructions. Thus processes taught by the discussion above may be performed with program code such as machine-executable instructions that cause a machine that executes these instructions to perform certain functions. In this context, a “machine” may be a machine that converts intermediate form (or “abstract”) instructions into processor specific instructions (e.g., an abstract execution environment such as a “virtual machine” (e.g., a Java Virtual Machine), an interpreter, a Common Language Runtime, a high-level language virtual machine, etc.), and/or electronic circuitry disposed on a semiconductor chip (e.g., “logic circuitry” implemented with transistors) designed to execute instructions such as a general-purpose processor and/or a special-purpose processor. Processes taught by the discussion above may also be performed by (in the alternative to a machine or in combination with a machine) electronic circuitry designed to perform the processes (or a portion thereof) without the execution of program code.

The disclosure also relates to an apparatus for performing the operations described herein. This apparatus may be specially constructed for the required purpose, or it may comprise a general-purpose device selectively activated or reconfigured by a computer program stored in the device. Such a computer program may be stored in a non-transitory computer readable storage medium, such as, but not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, DRAM (volatile), flash memory, read-only memories (ROMs), RAMs, EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions, and each coupled to a device bus.

A machine readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). For example, a non-transitory machine readable medium includes read only memory (“ROM”); random access memory (“RAM”); magnetic disk storage media; optical storage media; flash memory devices; etc.

An article of manufacture may be used to store program code. An article of manufacture that stores program code may be embodied as, but is not limited to, one or more non-transitory memories (e.g., one or more flash memories, random access memories (static, dynamic or other)), optical disks, CD-ROMs, DVD ROMs, EPROMs, EEPROMs, magnetic or optical cards or other type of machine-readable media suitable for storing electronic instructions. Program code may also be downloaded from a remote computer (e.g., a server) to a requesting computer (e.g., a client) by way of data signals embodied in a propagation medium (e.g., via a communication link (e.g., a network connection)) and then stored in non-transitory memory (e.g., DRAM or flash memory or both) in the client computer.

The preceding detailed descriptions are presented in terms of algorithms and symbolic representations of operations on data bits within a device memory. These algorithmic descriptions and representations are the tools used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of operations leading to a desired result. The operations are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

It should be kept in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the above discussion, it is appreciated that throughout the description, discussions utilizing terms such as “receiving,” “determining,” “sending,” “terminating,” “waiting,” “changing,” or the like, refer to the action and processes of a device, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the device's registers and memories into other data similarly represented as physical quantities within the device memories or registers or other such information storage, transmission or display devices.

The processes and displays presented herein are not inherently related to any particular device or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct a more specialized apparatus to perform the operations described. The required structure for a variety of these systems will be evident from the description below. In addition, the disclosure is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the disclosure as described herein.

In the foregoing specification, specific exemplary embodiments have been described. It will be evident that various modifications may be made to those embodiments without departing from the broader spirit and scope set forth in the following claims. The specification and drawings are, accordingly, to be regarded in an illustrative sense rather than a restrictive sense.