Patent Publication Number: US-2023138477-A1

Title: Sliding window protocol for communication among more than two participants

Description:
SUMMARY 
     In one embodiment, a method of controlling communication among a plurality of nodes in a network is provided. The method includes establishing a sender sliding window (SSW) to control sending of data from at least one node of the plurality of nodes to at least one other node of the plurality of nodes. The method also includes establishing a receiver sliding window (RSW) to control receiving of the data from the at least one node of the plurality of nodes at the at least one other node of the plurality of nodes. At least one of the SSW or the RSW is sharable among more than one node of the plurality of nodes. 
     In another embodiment, a system is provided. The system includes a plurality of nodes configured to communicate among one another. Each of the plurality of nodes includes at least one data storage container. The system also includes a SSW configured to control sending of data from at least one node of the plurality of nodes to at least one other node of the plurality of nodes. The system further includes a RSW configured to control receiving of the data from the at least one node of the plurality of nodes at the at least one other node of the plurality of nodes. At least one of the SSW or the RSW is sharable amongst more than one node of the plurality of nodes. 
     In yet another embodiment, a non-transitory computer-readable storage medium includes instructions that cause a system to establish a SSW to control sending of data from at least one node of a plurality of nodes of the system to at least one other node of the plurality of nodes. The non-transitory computer-readable storage medium also includes instructions that cause the system to establish a RSW to control receiving of the data from the at least one node of the plurality of nodes at the at least one other node of the plurality of nodes. At least one of the SSW or the RSW is sharable among more than one node of the plurality of nodes. 
     This summary is not intended to describe each disclosed embodiment or every implementation of the sliding window protocol for communication among more than two participants as described herein. Many other novel advantages, features, and relationships will become apparent as this description proceeds. The figures and the description that follow more particularly exemplify illustrative embodiments. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG.  1    is a diagrammatic illustration of a computer network in which at least some of the embodiments of the disclosure may be practiced. 
         FIG.  2    is a block diagram illustrating data transfer operations using a sender sliding window and a receiver sliding window in accordance with one embodiment. 
         FIG.  3    is a block diagram illustrating a data recovery operation in a 4-node system having 2 data containers in each node. 
         FIG.  4    is a block diagram illustrating a distributed data recovery operation using a sliding window protocol in accordance with an embodiment of the disclosure. 
     
    
    
     DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS 
     Embodiments of the disclosure relate to a sliding window protocol for communication among more than two participants. 
     Typically, sliding window protocols are employed for communication between two participants (e.g., two computers) in order to prevent overflow of the respective participant&#39;s buffers. Transmission Control Protocol (TCP) is one example of a protocol that utilizes sliding windows for data flow control between two participants. 
     In contrast with the sliding window protocols for communication between two participants, embodiments of the disclosure employ a sliding window protocol that includes a distributed parallel process that can run on more than two nodes or participants, and in which a same sliding window may be shared by, for example, multiple sending nodes that communicate with a receiving node. Prior to providing details regarding the different embodiments, a description of an illustrative operating environment is provided below. 
       FIG.  1    shows an illustrative operating environment in which certain specific embodiments disclosed herein may be incorporated. The operating environment shown in  FIG.  1    is for illustration purposes only. Embodiments of the present disclosure are not limited to any particular operating environment such as the operating environment shown in  FIG.  1   . Embodiments of the present disclosure are illustratively practiced within any number of different types of operating environments. 
     It should be noted that like reference numerals may be used in different figures for same or similar elements. It should also be understood that the terminology used herein is for the purpose of describing embodiments, and the terminology is not intended to be limiting. Unless indicated otherwise, ordinal numbers (e.g., first, second, third, etc.) are used to distinguish or identify different elements or steps in a group of elements or steps, and do not supply a serial or numerical limitation on the elements or steps of the embodiments thereof. For example, “first,” “second,” and “third” elements or steps need not necessarily appear in that order, and the embodiments thereof need not necessarily be limited to three elements or steps. It should also be understood that, unless indicated otherwise, any labels such as “left,” “right,” “front,” “back,” “top,” “bottom,” “forward,” “reverse,” “clockwise,” “counter clockwise,” “up,” “down,” or other similar terms such as “upper,” “lower,” “aft,” “fore,” “vertical,” “horizontal,” “proximal,” “distal,” “intermediate” and the like are used for convenience and are not intended to imply, for example, any particular fixed location, orientation, or direction. Instead, such labels are used to reflect, for example, relative location, orientation, or directions. It should also be understood that the singular forms of “a,” “an,” and “the” include plural references unless the context clearly dictates otherwise. 
     It will be understood that, when an element is referred to as being “connected,” “coupled,” or “attached” to another element, it can be directly connected, coupled or attached to the other element, or it can be indirectly connected, coupled, or attached to the other element where intervening or intermediate elements may be present. In contrast, if an element is referred to as being “directly connected,” “directly coupled” or “directly attached” to another element, there are no intervening elements present. Drawings illustrating direct connections, couplings or attachments between elements also include embodiments, in which the elements are indirectly connected, coupled or attached to each other. 
       FIG.  1    is a diagrammatic illustration of a computer network or system  100  in which at least some of the embodiments of the disclosure may be practiced. Computer network  100  includes multiple nodes  102  that are capable of communicating with one another using wired or wireless communication. Four nodes  102 A- 102 D (also referred to as nodes 0-3) are shown as an example. However, any suitable number of nodes  102  may be employed in different embodiments. Each node  102  may be a computing device that includes processing circuitry (not shown), communication circuitry (not shown) and data containers  104 . Data containers  104  may each include one or more data storage devices such as hard disc drives (HDDs), solid state drives (SSDs), and hybrid drives, which combine features of HDDs and SSDs in one unit. As just one example, two data containers  104  are shown per node  102 , thereby providing a total of 8 data containers denoted by  104 A- 10 H. Data sets may be distributed in multiple packets across the  8  containers  104 . The data packets are denoted by  106 A- 106 P. 
     As depicted in  FIG.  1   , the data sets  106  may be distributed across multiple nodes  102 A- 102 D (or 0-3). There are several situations that may lead to data transfers between the nodes 0-3 in such a distributed setup, e.g.,
         Storage capacity balancing for better scalability, performance and storage consumption   Handling storage device failures and recovering from the failures   Backup, etc.       

     Data transfer processes in a distributed setup of the type shown in  FIG.  1    involve various resources such as network, memory and central processing unit (CPU). Typically, the containers  104  are parts of multiple groups (e.g., each container  104  may store data belonging to different data sets with, for example, each different data set having data from a different redundant array of inexpensive disks (RAID) stripe). An expansive recovery operation across multiple groups may easily exhaust system resources (e.g., CPU, memory and network bandwidth) and effectively block regular input/output (I/O) operations, thereby leading to starvation of other tasks. Allocating resources and controlling such a recovery operation is a challenge. Embodiments of the disclosure provide a sliding window protocol that allows expansive data transfers (e.g., data transfers in the case of recovery operations) to:
         Utilize system resources such that there is no or minimal effect on regular operations such as I/O.   Provide a mechanism to report progress of distributed operations.   Handle failures and allow continuation of the distributed operation(s).       

     In embodiments of the disclosure, the sliding window protocol is configured to maintain two versions of a sliding window (e.g., a sender sliding window (SSW) and a receiver sliding window (RSW)). A sender is a node  102  that sends its contribution of the distributed data to another node  102 , while a receiver is a node  102  that receives data from one or more sender nodes  102 . A SSW is mainly used to improve communication during failure cases. Data transfer operations are mainly dictated by the RSW, which is controlled by the receiver nodes depending on their receiving bandwidth (R b ). 
       FIG.  2    is a block diagram illustrating data transfer operations using a SSW and a RSW in accordance with one embodiment. In  FIG.  2   , for simplicity, node  102 D (or node 3) is depicted as an only receiver of data packers from its peer nodes  102 A- 102 C (or nodes 0-2). As in a typical distributed store, data is divided into multiple packets  200 A- 200 L and striped across multiple nodes  102  (in this example, nodes  102 A- 102 C). In the example of  FIG.  2   , data packets  200 A- 200 F belong to a first data set or group, and are shown as shaded boxes. Data packets  200 A- 200 F are also referred to herein as data packets 0-5 of the first group and are in bold. Data packets  200 G- 200 L belong to a second data set or group, and are shown as unshaded boxes in  FIG.  2   . Data packets  200 G- 200 L are also referred to herein as data packets 0-5 of the second group and are not in bold. 
     In some embodiments, every node  102  may maintain a proxy data structure  108  (sometimes simply referred to herein as proxy) corresponding to every other node  102 . The proxy data structure  108  may be used to maintain the SSW and RSW information, and for additional purposes (e.g., for data caching). Data associated with the proxy  108  may be stored in any suitable memory within node  102 . In a particular example  110 , proxy data structure  108  may be utilized to maintain a RSW, a node status (e.g., a status indicative of whether different nodes  102  are active or inactive (e.g., due to failure)), and a sender side data cache. 
     In one embodiment, each node  102 , a sender or receiver, maintains location and size information for its stored packets  200  belonging to its respective data sets or groups. Additionally, as indicated above, each node  102  maintains proxy  108  representing its peer (sender or receiver) that caches the RSW and the SSW. The cached RSW and SSW information may include low (lo) and high (hi) identifiers for a string of data packets in a particular data set or group to be transferred using the respective RSW and SSW. Such information is generally represented by RSW (lo, hi)  and SSW (lo, hi)  in the disclosure, and examples using specific lo and hi packet identifiers are provided in  FIGS.  2  and  4   , and described further below. In the examples of  FIGS.  2  and  4   , a lo packet is identified by a solid box surrounding the data packet, and a hi packet is identified by a dashed box surrounding the data packet. It should be noted that that the lo and hi packets may be from a same date set or from different data sets. The lo and hi packets may be viewed as beginning and ending edges, respectively, of the RSW/SSW. 
     In embodiments of the disclosure, the sender node  102  (e.g., node  102 A,  102 B or  102 C) caches its contribution to the SSW and does not add it to the RSW until the receiver node  102  (e.g., node  102 D) communicates to the sender node  102  (e.g., node  102 A,  102 B or  102 C) that it has the available bandwidth to receive the cached data contribution. This avoids blocking the sender node  102  (e.g., node  102 A,  102 B or  102 C) until the receiver node  102  (e.g., node  102 D) is ready to receive data. 
     In some embodiments, the receiver node (e.g.,  102 D or node-3 in  FIG.  2   ) calculates its receiving bandwidth R b , which is a product of incoming data packets/units and a size of each data packet/unit. In the example of  FIG.  2   , R b  is the total expected incoming data packets/units from the peer nodes (e.g., nodes 0-2) multiplied by the size of each data packet/unit, which gives a total size of the expected data for node 3. The receiver (e.g., node-3 in  FIG.  2   ) also calculates the available bandwidth A b , which is a function of C b , M b , and N b , where C b , M b , N b  are CPU, memory and network bandwidths, respectively. 
     Upon calculating the receiver and available bandwidth, the receiver node  102 D may choose between receiving a subset of or all of the data units expected. This is carried out by the receiver node  102 D broadcasting its available bandwidth to all the relevant members (e.g., all of nodes  102 A- 102 C) or just to a subset of them (e.g., one or two of nodes  102 A- 102 C). Then, the sender node(s) send their respective blocks/packets of data with the help of the RSW. 
       FIG.  2    shows 3 example stages (first stage  202 , second stage  204  and third stage  206 ) of data communication among nodes  102  using the sliding window protocol of the disclosure. The separation of the inter-node communication into 3 different stages is merely for purposes of illustration. In general, inter-node communication can take place in any suitable manner without any limitation as to how, when, and in what order/number the data packets  200 A- 200 L are transferred among nodes 0-3. 
     In the first stage  202 , receiver node 3 is ready to receive data packets 0-3 of the first group, and communicates its readiness to the different sender nodes 0-2. Data packets 0-3 of the first group, which node 3 is ready to receive, are stored in different ones of sender nodes 0-2, with sender node 0 having packets 0 and 2, sender node 1 having packet 1, and sender node 2 having packet 3. Since all of sender nodes 0-2 have data to send to receiver node 3, in first stage  202 , RSW (lo, hi)  cached by each of sender nodes 0-2 in their respective Node-3 proxies in  108  is RSW (0, 3) . 
     As can be seen in first stage  202  of  FIG.  2   , sender node 0 is ready to send data packets 0 and 2 of the first group, and is also ready to send data packet 1 of the second group. Thus, SSW (lo, hi)  cached by sender node 0 is SSW (0, 1) , where the lo identifier 0 is for the first data packet of the first data set, and high identifier 1 is for the second data packet of the second data set. Since the receiver node 3 is not ready to receive any data packets belonging to the second group, node 0 sends its stored data packets 0 and 2 belonging to the first group to receiver node 3, and does not send data packet 1 of the second group. 
     In first stage  202 , sender node 1 is ready to send data packets 1 and 4 of the first group and data packets 0 and 2 of the second group. Since the receiver node 3 is not ready to receive any data packets belong to the second group, node 1 sends its stored data packet 1 belonging to the first group to receiver node 3, and does not send data packet 4 of the first group and data packets 0 and 2 of the second group. 
     Further, in first stage  202 , sender node 2 is ready to send data packets 3 and 5 of the first group. Since the receiver node 3 is not ready to receive data packet 5 of the first group, node 2 sends its stored data packet 3 belonging to the first group to receiver node 3, and does not send data packet 5 of the first group. 
     In second stage  204 , node 3 has received all the data packets (packets 0-3 of the first group) that it requested in stage  202 , and is now ready to receive remaining data packets 4 and 5 of the first group, and data packets 0 and 1 of the second group. As can be seen in  FIG.  2   , communication using SSWs and RSWs to accomplish data transfers in the second stage  204  takes place in a manner similar to that described above in connection with the first stage  202 . Therefore, in the interest of brevity, data communication details are omitted. 
     In third stage  206 , node 3 has received all the data packets (packets 4 and 5 of the first group, and packets 0 and 1 of the second group) that it requested in stage  204 , and is now ready to receive remaining data packets 2-5 of the second group. Since the transfer of data packets in third stage  206  is shown at the bottom of  FIG.  2   , and takes place in a manner to that described above, no additional details are provided. 
     The above-described sliding window protocol may be utilized in various distributed operations involving data transfers between the nodes. One such distributed operation is a data recovery operation. An example of a data recovery operation for which the sliding window protocol may be utilized is described below in connection with  FIGS.  3  and  4   . 
       FIG.  3    is a block diagram illustrating a data recovery operation in a 4-node system having 2 data containers in each node. In  FIG.  3   , D 0 -D 3  are data packets, P 0  and P 1  are parities calculated over the data blocks D 0  and D 3 , and S 0  and S 1  are spare units. Failed data is recovered in containers holding the spare units S 0  and S 1  (e.g., containers  104 F,  104 H). D 0 -D 3 , P 0 , P 1 , S 0  and S 1  that belong to a first data set or group are shown as shaded boxes, and D 0 -D 3 , P 0 , P 1 , S 0  and S 1  that belong to a second data set or group are shown as unshaded boxes. In the example shown in  FIG.  3   , both node 0 and node 1 have 1 container failure (depicted as an X in  FIG.  3   ). Lost data units D 2  and D 3  from each data set or group in containers  104 B and  104 D may be recovered in spare units S 0  and S 1  on nodes 2 and 3, respectively. 
     A group&#39;s data and parity members distributed across multiple containers and nodes are related by a set of linear equations: 
         a   00   *D   0   +a   01   *D   1   +a   02   *D   2   ±a   03   *D   3   =P   0   Equation 1
 
     where a 00 , a 01 , a 02  and a 03  are constants. 
         a   10   *D   0   +a   11   *D   1   +a   12   *D   2   ±a   13   *D   3   =P   1   Equation 2
 
     where a 10 , a 11 , a 12  and a 13  are constants. 
     Failed data is recovered by transferring the available relevant packets of data and parity to the nodes 2 and 3 hosting the spare containers. The receiving nodes 2 and 3 substitute the received data and parity pieces of the group to solve Equations 1 and 2. This process involves various resources such as network, memory and CPU. Typically, the containers are part of multiple groups. An expansive recovery operation in such a case may exhaust system resources (e.g., CPU, memory and network bandwidth) and effectively block regular I/O and lead to starvation of other tasks. Allocating resources and controlling such a recovery operation is a challenge. As indicated above, utilizing the sliding window protocol will allow such an expansive operation to:
         Use system resources such that there is no or minimal effect on regular operations such as I/O.   Provide a mechanism to report the distributed operation&#39;s progress.   Handle failures and allow continuation of the operation.       

       FIG.  4    is a block diagram illustrating a distributed data recovery operation using a sliding window protocol in accordance with an embodiment of the disclosure. As in the case of  FIG.  3    described above, in  FIG.  4   , D 0 -D 3 , P 0 , P 1 , S 0  and S 1  that belong to a first data set or group are shown as shaded boxes and are in bold, and D 0 -D 3 , P 0 , P 1 , S 0  and S 1  that belong to a second data set or group are shown as unshaded boxes and are not in bold. In the example of  FIG.  4   , blocks D 3  of each data set or group are affected due to failure of container  104 D of node 1. Data transfers take place among the 4 nodes 0-3 in order to recover lost blocks/packets D 3  of each group. For simplicity, in the example of  FIG.  4   , node 3 is depicted as an only receiver of data packets/chunks/blocks from its peer nodes 0-2, which contribute to the distributed recovery operation. 
     As noted above, every node  102  may maintain a proxy data structure (not shown in  FIG.  4    in the interest of simplification) corresponding to every other node  102 . Further, as indicated above, the proxy data structure may be used to maintain the SSW and RSW information, and for additional purposes (e.g., for data caching). Each of sender nodes 0-2 prepares to send its available block(s) of data and updates its SSW. Receiver node 3 determines its available bandwidth, and updates sender nodes 0-2 regarding the determined available bandwidth. Then, the sender nodes 0-2 send their respective blocks of data that fall into the RSW. Accordingly, in first data transfer stage  400 , receiver node 3 is ready to receive D 0 , D 1 , D 2  and P 0  of the first group and, with the help of the SSW and RSW, sender node 0 transfers D 0  and D 2  of the first group to receiver node 3, sender node 1 transfers D 1  of the first group to receiver node 3, and sender node 2 transfers P 0  of the first group to node 3. Upon receiving the blocks D 0 , D 1 , D 2  and P 0  of the first group from the nodes 0-2, node 3 recovers D 3  of the first group on S 0  of the first group by solving Equations 1 and 2 included above. This is shown on the right side of second a second data transfer stage  402  in  FIG.  4   . 
     Additionally, in the second data transfer stage  402 , receiver node 3 is ready to receive D 0 , D 1 , D 2  and P 1  of the second group and, with the help of the SSW and RSW, sender node 0 transfers D 1  and D 2  of the second group to receiver node 3, sender node 1 transfers D 0  of the second group to receiver node 3, and sender node 2 transfers P 1  of the second group to node 3. Upon receiving the blocks D 0 , D 1 , D 2  and P 1  of the second group from the nodes 0-2, node 3 recovers D 3  of the second group on S 0  of the second group by solving Equations 1 and 2 included above. This is shown on the right side of a third data transfer stage  404  in  FIG.  4   , and results in the completion of the data recovery operation. 
     Advantages of the above-described sliding window protocol include:
         Starvation prevention: The protocol helps ensure that regular operations do not starve during data recovery by automatically adapting to the available resource bandwidths.   Flow control: Controlling traffic flow through the sliding window restricts senders from flooding receivers and incidentally overloading them.   Increased resource utilization: Automatically adapting to available resource bandwidths helps increase resource utilization in the absence of other higher priority tasks.       

     As indicated above in the description of  FIG.  2   , a sender node can prepare its contribution of the data set against the requested data by the receiver node as per its available bandwidth, but waits until notified by the receiver node to send the data contribution. The sender checks with its copy of the RSW and decides to add its contribution to the RSW or to the SSW, respectively, or even discards the contribution in some cases. Specifically, the following actions may be taken:
         If D i &lt;RSW lo , where D i  is the sender&#39;s contribution and it is in the past of the RSW, in this case the receiver does not expect this contribution and the sender can discard the same.   If RSW lo &lt;D i &lt;RSW hi , then the sender can add its contribution (D i ) to the RSW.   If D i &gt;RSW hi , then the sender&#39;s contribution (D i ) is in the future, and the sender adds D i  to its SSW.       

     Reporting progress of a distributed operation can be challenging, especially in case of an expansive system where the data is divided in groups spanning multiple nodes as depicted above. The sliding window protocol provides a progress-reporting mechanism that utilizes RSW edges (e.g., RSW lo  and RSW hi  values). Progress reporting for a data group, G i , may be as follows:
         If G i &lt;RSW lo , then all the data belonging to group G i  has been processed.   If G i &gt;RSW hi , then none of the group&#39;s data has been processed yet.   If RSW lo &lt;G i &gt;RSW hi , then the data from data group G i  is being processed.
 
Such information may be useful in case the group&#39;s data is being read or modified concurrently during the distributed data operation.
       

     In some embodiments, the sliding window protocol may be adapted for different failure handling situations. Such situations may include a restart of a failed node, data missing from containers in nodes, and other node operational failure. 
     In the case of a restart of a failed node, by persisting sliding window edges (e.g., RSW [lo, hi] ), the node may continue its contribution towards data processing after the restart. There is a possibility that the other nodes can continue the data processing with respect to the distributed data operation where RSW hi  may move ahead while RSW lo  may not change for a while. Consider the following 2 cases:
         Case N&lt;D (where N is the number of nodes, and D is a number of data units distributed across those N nodes): In this case, all the N nodes participate in the data processing operation and a node failure may halt the operation until the failed node is started.   Case N&gt;D: In this case, all the N nodes need not take part in the data processing operation of all the G groups requested by the receiver(s). In such a case, the data processing operation can still continue where the RSW hi  moves ahead while RSW lo  does not as the receiver expects a contribution of a group G i  from the failed node.       

     As noted above, the sliding window protocol may also be adapted for situations involving missing data. For example, it is possible that not all the data containers spanned by a group contain data corresponding to a particular group. Such a situation may arise in case the group is sparsely populated. This may require special handling because, even though the receiver node is aware of the group&#39;s layout of data containers, it may not know whether the corresponding data container holds the data or not. Corresponding senders may not have the presence of the group at all in some such a cases. This may cause the receiver to wait endlessly as a worst case scenario. In embodiments of the disclosure, such a situation may be handled by using a SSW described above. Here, if RSW lo &lt;SSW lo , it effectively means that the sender node is in the past of the receiver node&#39;s lower edge and will not be sending the corresponding data. The sender node communicates its window to the receiver node, and the latter updates its sliding window accordingly, and ceases to wait. 
     As noted above, in embodiments of the disclosure, the sliding window protocol also supports automatic handling of operational failures occurring on participating nodes. Sender or receiver nodes report errors (e.g., read failure due to invalid or missing data) which are cached as part of the proxy mechanism. This helps the node to take relevant action such as releasing the corresponding resources, and marking the operation as failed or even continuing with it. 
     The illustrations of the embodiments described herein are intended to provide a general understanding of the structure of the various embodiments. The illustrations are not intended to serve as a complete description of all of the elements and features of apparatus and systems that utilize the structures or methods described herein. Many other embodiments may be apparent to those of skill in the art upon reviewing the disclosure. Other embodiments may be utilized and derived from the disclosure, such that structural and logical substitutions and changes may be made without departing from the scope of the disclosure. Additionally, the illustrations are merely representational and may not be drawn to scale. Certain proportions within the illustrations may be exaggerated, while other proportions may be reduced. Accordingly, the disclosure and the figures are to be regarded as illustrative rather than restrictive. 
     One or more embodiments of the disclosure may be referred to herein, individually and/or collectively, by the term “invention” merely for convenience and without intending to limit the scope of this application to any particular embodiment or inventive concept. Moreover, although specific embodiments have been illustrated and described herein, it should be appreciated that any subsequent arrangement designed to achieve the same or similar purpose may be substituted for the specific embodiments shown. This disclosure is intended to cover any and all subsequent adaptations or variations of various embodiments. Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those of skill in the art upon reviewing the description. 
     The Abstract of the Disclosure is provided to comply with 37 C.F.R. § 1.72(b) and is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, various features may be grouped together or described in a single embodiment for the purpose of streamlining the disclosure. This disclosure is not to be interpreted as reflecting an intention that the claimed embodiments include more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter may be directed to less than all of the features of any of the disclosed embodiments. 
     The above-disclosed subject matter is to be considered illustrative, and not restrictive, and the appended claims are intended to cover all such modifications, enhancements, and other embodiments, which fall within the true spirit and scope of the present disclosure. Thus, to the maximum extent allowed by law, the scope of the present disclosure is to be determined by the broadest permissible interpretation of the following claims and their equivalents, and shall not be restricted or limited by the foregoing detailed description.