Abstract:
A scheduler and method for use in ATM and packet communication systems applies a no-per-connection-timestamp discrete-rate scheduling technique which does not require the computation and storage of one timestamp per connection, and only maintains a single timestamp per supported service rate. The elimination of the per-connection timestamps has no negative effect on the delay bounds guaranteed by the scheduler. The total implementation cost of such schedulers which approximate the Generalized Processor Sharing (GPS) policy is reduced, since there is less complexity involved in maintaining and sorting the timestamps for all connections.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application is related to co-pending U.S. patent application Ser. No. 60/079,721, filed Mar. 27,1998, and entitled METHOD AND APPARATUS FOR GUARANTEEING DATA TRANSFER RATES AND DELAYS IN DATA PACKET NETWORKS USING DISCRETE DATA TRANSFER RATES. 
     This application is also related to a commonly assigned and co-pending U.S. patent application of Fabio Chiussi and Andrea Francini, entitled METHOD AND APPARATUS FOR GUARANTEEING DATA TRANSFER RATES AND DELAYS IN DATA PACKET NETWORKS USING GENERALIZED DISCRETE DATA TRANSFER RATE APPROACH, filed Feb. 9, 1999. 
    
    
     BACKGROUND OF THE INVENTION 
     The present invention relates to packet communication systems, and in particular to a method and an apparatus for scheduling packets in packet networks for guaranteeing data transfer rates to data sources and data transfer delays from data sources to destinations using a plurality of queues, each of the queues serving data connections with the same guaranteed data transfer rate, and only computing and sorting a single timestamp for each of the queues. This invention can be used in any system for data packet forwarding such as Asynchronous Transfer Mode (ATM) switches and Internet Protocol (IP) routers. 
     Per-Virtual-Connection (Per-VC) schedulers are known which aim to approximate a Generalized Processor Sharing policy, as described in A. K. Parekh and R. G. Gallager, “A Generalized Processor Sharing Approach to Flow Control in Integrated Services Networks: The Single-Node Case”, IEEE/ACM TRANSACTIONS ON NETWORKING, June 1993, pp. 344-357, which is incorporated herein by reference. As defined herein, the term “VC” is used throughout to mean “virtual connection”. It is understood that virtual connections may also include virtual circuits and Internet Protocol (IP) flows. Implementation of such Per-VC schedulers is a central issue in next-generation switching systems. In a market arena in which cost targets are precipitously dropping, an important objective is to minimize the complexity involved in Per-VC schedulers, and to minimize the cost differential with respect to switches using less sophisticated scheduling. 
     As defined herein and throughout, the term “GPS” is an abbreviation for the Generalized Processor Sharing policy, as described in A. K. Parekh et al., supra. GPS-related packet-scheduling disciplines are based on maintaining a global function, referred to by different authors either as a virtual time, such as in A. K. Parekh et al, supra, and in S. J. Golestani, “A Self-Clocked Fair Queuing Scheme for Broadband Applications”, PROCEEDINGS OF INFOCOM &#39;94, April 1994, pp. 636-646, which is incorporated herein by reference; or as a system potential, such as described in D. Stiliadis and A. Varma, “Design and Analysis of Frame-based Fair Queuing: A New Traffic Scheduling Algorithm for Packet-Switched Networks”, PROCEEDINGS OF SIGMETRICS &#39;96, May 1996, pp. 104-115; and D. Stiliadis and A. Varma, “Efficient Fair Queuing Algorithms for ATM and Packet Networks”, TECHNICAL REPORT UCSC-CRL-95-59, December 1995, with each of these references being incorporated herein by reference. 
     The global function tracks the amount of work that is done by the server to process packets in the communication system. The server uses this global function to compute, for each packet in the system, a timestamp that specifies when the packet should be transmitted relative to other packets. Packets are transmitted by increasing order of their timestamps. The specific function used as system potential determines the delay and fairness properties of each algorithm in the class. 
     The total implementation cost of these GPS-related scheduling algorithms is the combination of three factors: (i) the complexity of the function used as system potential to compute the timestamps for the packets in the system, (ii) the complexity involved in sorting the timestamps in order to select the packet with a minimum timestamp for transmission, and (iii) the cost of handling and storing the timestamps. In recent years, several scheduling algorithms which use a system-potential function of order  0 ( 1 ) complexity have been introduced. Examples of such algorithms include Self-Clocked Fair Queuing (SCFQ), as described in S. J. Golestani, supra; Frame-based Fair Queuing (FFQ), as described in D. Stiliadis et al., “Design and Analysis of Frame-based Fair Queuing . . . ”, supra; Virtual Clock, as described in L. Zhang, “Virtual Clock: A New Traffic Control Algorithm for Packet Switching”, ACM TRANSACTIONS ON COMPUTER SYSTEMS, May 1991, pp. 101-124; and Minimum-Delay Self-Clocked Fair Queuing (MD-SCFQ), described in F. M. Chiussi and A. Francini, “Minimum-Delay Self-Clocked Fair Queuing Algorithm for Packet-Switched Networks”, PROCEEDINGS OF INFOCOM &#39;98, March 1998, each of which is incorporated herein by reference. 
     In particular, among these algorithms, MD-SCFQ has both optimal delay properties and excellent fairness properties. Scheduling algorithms achieving a desired performance with a system-potential function of minimal complexity are therefore available, but the total performance cost of the scheduler is still dominated by the complexity of sorting and storing the timestamps. 
     One well-known simplification in timestamp processing by a scheduler is obtained by assigning increasing values of timestamps to consecutive packets which belong to the same session, so that for each session only the timestamp of the packet at the head of the corresponding packet queue is to be considered and processed in the packet selection process. Such a timestamp is referred to as session timestamp. The number of session timestamps which have to be sorted is therefore equal to the number of sessions V supported by the scheduler. For example, typical values of V in current ATM switches, in which sessions are referred to as VCs, are in the order of tens of thousands of sessions. The range of possible values that the timestamps can assume at any given time depends on the ratio between the maximum and minimum service rates that the scheduler is required to provide to the connections. Such a timestamp range is typically very wide. 
     In view of the complexity in sorting a large number of timestamps over a wide range of possible values at the high speeds employed in broadband digital networks, hardware implementations of packet-switching systems are only affordable by data structures and processor configurations that are specifically devised to be efficiently mapped into silicon on integrated circuits or chips. Even with such specialized structures, the implementation cost may still be too high, and techniques to further reduce complexity are necessary. Different approaches are possible for this purpose. In some cases, the specific properties of a scheduler can help in simplifying the selection process. 
     Several techniques have been proposed to reduce the cost of the sorting operation. In particular, two approaches are the Logarithmic Calendar Queue (LCQ) introduced in F. M. Chiussi, A. Francini and J. G. Kneuer, “Implementing Fair Queuing in ATM Switches—Part 2: The Logarithmic Calendar Queue”, PROCEEDINGS OF GLOBECOM &#39;97, November 1997, pp. 519-525; as well as the discrete-rate scheduler presented in J. C. R. Bennett, D. C. Stephens and H. Zhang, “High Speed, Scalable, and Accurate Implementation of Fair Queuing Algorithms in ATM Networks”, PROCEEDINGS OF ICNP &#39;97, October 1997, pp. 7-14, each of which are incorporated herein by reference. Both of these approaches are arguably the two approaches that achieve the highest reduction in the hardware complexity of a GPS-related scheduler with optimal delay properties. In addition, such approaches introduce only a very small degradation in the delay bounds of the scheduler. 
     The LCQ is an optimized calendar queue which reduces the complexity by increasing, in an optimal manner, the granularity of the bins used to sort the timestamps, so that the relative degradation in delay bounds for each connection is equalized. 
     The discrete-rate scheduler is a relatively simple structure that can be used when the guaranteed service rates that the scheduler needs to support at any given time only belong to a relatively small set of discrete values. Such operating conditions are certainly realistic in most, if not all, ATM switches. As shown in FIG. 1, the illustrated discrete-rate scheduler 10 is a per-connection-timestamp scheduler having a corresponding timestamp for each of the sessions; for example, the sessions  14 - 16  in FIG.  1 . Each of the sessions  14 - 16  has a corresponding timestamp  20 - 26 , respectively. 
     Other advantages are known for using a discrete set of rates. In this case, connections with the same service rate are grouped together in common rate First-In-First-Out (FIFO) queues, and scheduling is performed only among the connections at the head of each rate FIFO queue. Accordingly, the per-connection timestamp scheduler  10  in FIG. 1 has the plurality of registers for storing pointers as heads  12  and tails  18  for maintaining the number N of rate FIFO queues  28 , with the sessions  14 - 16  in a given queue having the same rate from among rates r 1  . . . r N . Thus, the registers  12  and  18  in Rate FIFO Queue  1  are associated with a common rate r 1 , with a first head  12  labeled HEAD( 1 ) as a head pointer pointing to a session in the queue and having an associated timestamp  20  labeled F HEAD(1) ; a first set of sessions  14 - 16  labeled VC 1,A  and VC 1,B . respectively, and having associated timestamps  22 ,  24  labeled F 1,A  and F 1,B , respectively; and a first tail TAIL( 1 )  18  as a tail pointer to a session in the queue and having an associated timestamp  26  labeled F TAIL(1) . As described above, scheduling is performed by processing the sessions pointed to at the heads of the queues, with such sessions being processed by a smallest-eligible-virtual-finishing-time-first (SEFF) selector  30  to determine a minimum eligible timestamp for service from among the sessions pointed to by the heads of the queues  28 . 
     The implemented scheduler may have certain properties for the maximum distance between timestamps of different connections having the same rate, and for the relation between system potential and timestamps, as is the case for the worst-case-fair weighted fair queuing system (WF 2 Q, or alternatively WF 2 Q+) described in J. C. R. Bennett and H. Zhang, “Hierarchical Packet Fair Queuing Algorithms”, PROCEEDINGS OF SIGCOMM &#39;96, August 1996, pp. 143-156; and other worst-case-fair schedulers, such as described in D. Stiliadis and A. Varma, “A General Methodology for Designing Efficient Traffic Scheduling and Shaping Algorithms”, PROCEEDINGS OF INFOCOM &#39;97, April 1997, with each of these articles being incorporated herein by reference. 
     By grouping together connections with the same service rate in common rate FIFO queues, the number of timestamps to be sorted is greatly reduced, for example, to be equal to the number of supported rates, and therefore the complexity of the sorting task is considerably decreased. 
     Although being important improvements in reducing the implementation complexity of GPS-related schedulers with near-optimal delay bounds, known approaches still require computing and storing a timestamp for each connection. The LCQ and other techniques presented in J. L. Rexford, A.G. Greenberg and F. G. Bonomi, “Hardware-Efficient Fair Queuing Architectures for High-Speed networks”, PROCEEDINGS OF INFOCOM &#39;96, March 1996, pp. 638-646, which is incorporated herein by reference; do not require per-connection timestamps when used to implement SCFQ, but this scheduling algorithm does not achieve near-optimal delay bounds. 
     However, because of the need of per-connection timestamps, the overhead in memory resources due to the scheduler remains significant. This is especially true in ATM, in which the size of the packet is relatively small, and the number of connections supported by the system is on the order of several thousands of connections. The number of connections is often of the same order of magnitude as the number of packets in a buffer, and that number is constantly increasing as the industry evolves to implement improved data packet networks. 
     SUMMARY OF THE INVENTION 
     It is recognized herein that it is possible to introduce approximations in the scheduling algorithms in order to simplify their implementation. In general, these approximations may negatively affect the delay and/or fairness properties of the scheduler, and the challenge is to optimize the design so that the degradations induced by the approximations are minimized. 
     It is an object of the present invention to provide a technique to further reduce the implementation cost of GPS-related schedulers with near-optimal delay bounds, using a No-Per-Connection Timestamp Discrete-Rate Scheduler. This scheduler does not require the computation and storage of a timestamp per connection, and only maintains a single timestamp per rate. The scheduler has a very simple two-level hierarchical structure, in which, at the lower level of the hierarchy, FIFO queues of connections are used with one FIFO queue per rate and with connections being queued without a timestamp. In the scheduler, a timestamp is only assigned to each rate FIFO queue, taking into account the aggregate bandwidth of all connections with that rate. 
     At the higher level of the hierarchy, the scheduler uses a worst-case-fair scheduler which schedules among the different FIFO queues. Any packet-by-packet rate-proportional server (P-RPS) as well as MD-SCFQ with a shaping mechanism can be used as a second-level scheduler. Such schedulers have been shown to constitute worst-case-fair schedulers. The shaping mechanism is obtained by adopting the SEFF packet-selection policy, which considers for service only packets whose transmission has already started in the underlying fluid system. If the worst-case-fair scheduler used is work-conserving, the resulting discrete-rate scheduler is also work-conserving. 
     The no-per-connection-timestamp scheduler has the same near-optimal delay properties of existing discrete-rate schedulers. The only disadvantage in the elimination of the timestamps per connection is some degradation in the fairness properties. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     FIG. 1 illustrates a prior-art per-connection-timestamp scheduler; 
     FIG. 2 illustrates the no-per-connection-timestamp scheduler with a hierarchical configuration; 
     FIG. 3 illustrates the no-per-connection-timestamp scheduler of FIG. 2 in greater detail; and 
     FIGS. 4A-4B illustrate a flowchart of operation of the no-per-connection-timestamp scheduler. 
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
     Referring now to FIG. 2, the present invention relates to a method and apparatus implementing a technique for reducing the implementation complexity of a GPS-related scheduler with near-optimal delay bounds. It is to be understood that, although preferred embodiments are illustratively shown and described herein to reduce the implementation complexity for a GPS-related scheduler in an ATM network, the present invention can be applied to reduce the implementation complexity for any GPS-related scheduler in any packet network. 
     The apparatus shown in FIG. 2 implements a No-Per-Connection-Timestamp Discrete-Rate Scheduler  32  and method of use which removes the need and associated cost of maintaining per-connection timestamps in the implementation of GPS-related schedulers with near-optimal delay bounds, at the nominal cost of increasing the unfairness of the scheduler in the distribution of leftover bandwidth to connections that are simultaneously backlogged. 
     As illustrated in FIG. 2, the scheduler  32  has a two-level hierarchical configuration with a lower level  34  and a higher level  36 . At the lower level  34  of the hierarchy, FIFO queues  38  are used to perform the lower-level scheduling of sessions, with one FIFO queue per rate and with connections and/or sessions  40  being queued without a timestamp. FIFO is the preferred method of queuing for optimal performance, but it is to be understood that other queuing methods may also be used, such as last-in-first-out (LIFO), which may have different performance characteristics. 
     In the preferred embodiment, sessions  40  having a first rate r, labeled ( 1 ,a  1 ,b  1 ,c  1 ,d) are queued in the first queue corresponding to the rate FIFO queue  1 , while sessions  40  having a second rate r 2  and labeled ( 2 ,a  2 ,b) are queued in the second queue corresponding to the rate FIFO queue  2 , and sessions  40  having an NT th  rate r N  and labeled (N,a N,b N,c) are queued in the N TH  queue corresponding to the rate FIFO queue N. 
     In the scheduler  32 , a timestamp is only assigned to each rate FIFO queue, taking into account the aggregate bandwidth of all connections with that rate. Such timestamps  42 - 46 , labeled F 1 , F 2  . . . F N  for the first, second, and N TH  rate, respectively, are maintained in the higher level  36 . The timestamps  42 - 46  may be associated with the heads  48 - 52 , respectively, of each of the corresponding rate FIFO queues  38 . 
     At the higher level  36  of the hierarchy, the scheduler  32  uses a worst-case-fair scheduler which schedules the selection of sessions for service from among the different FIFO queues. Any packet-by-packet rate-proportional server (P-RPS) as well as MD-SCFQ with a shaping mechanism can be used as the higher-level scheduler, since such servers have been shown to constitute worst-case-fair schedulers. The shaping mechanism is implemented in the scheduler  32  using an SEFF selector  54  to adopt a SEFF packet-selection policy from among the associated queues  38  and respective timestamps  48 - 52 . 
     The no-per-connection-timestamp scheduler  32  has the same near-optimal delay properties of existing discrete-rate schedulers, such as the scheduler  10  shown in FIG.  1 . To be precise, the no-per-connection-timestamp scheduler  32  has better performance with regard to delay. 
     The server of the scheduler  32  simultaneously supports a fixed number N of guaranteed service rates at any given time, and a rate FIFO queue of sessions is associated with each rate. Unlike existing schedulers, the scheduler  32  shown in FIG. 2 processes sessions which do not have corresponding timestamps for each session, but instead the sessions have a single commonly associated timestamp maintained for each rate FIFO queue  38 . 
     Within each of the FIFO queues  38 , sessions  40  are handled as follows: when a FIFO queue is granted a service, the session currently at its head is dequeued and, if still backlogged, queued back to the tail of the corresponding rate FIFO queue. Newly backlogged sessions are queued at the tail of the respective FIFO queues. 
     As shown in greater detail in FIG. 3, in the scheduler  32 , each of the rate FIFO queues  38  is treated as a single macro-session by the scheduler  32 , such that the queues  38  form a set of macro-sessions  56 - 60 . Accordingly, for a given rate FIFO queue such as the queue having the timestamp  42  and the head  48 , the macro-session  56  has the head  48  and tail  62  and includes the sessions  64 - 66  in the queue, with the timestamp  42  being associated with the entire queue and the entire macro-session  56 . 
     For each i TH  rate FIFO queue, the rate r 1 , allocated to macro-session I is r 1     i   =n r     i    r i , with n r     i    being the number of allocated sessions with service rate r i , and, for macro-session I, each individual session has a service rate r i . For example, the rate  68  labeled r 1     i   , is r l     1   =n r     1    r 1 which is the rate allocated to macro-session  56  labeled I 1  for rate FIFO queue  1 . 
     Macro-session I is idle when all of the allocated sessions with service rate r i  are idle, and is backlogged when at least one of those allocated sessions is backlogged. At timeslot h l   k , as the sessions are processed, after a previous session has been removed from its queue and serviced by the selector  54 , a new session reaches the head of the queue of macro-session I and the new timestamp of the macro-session I is determined to be:                F   I   k     =       max        (       F   I     k   -   1       ,     P        (     h   I   k     )         )       +     1     r   I                 (   1   )                                
     in which the global function P( ) is the system potential or virtual time. Only one session is serviced at a time by the selector  54 , and then the selection process starts over. 
     The two-level hierarchical configuration of the scheduler  32  is useful for studying the properties of the scheduler  32 . For example, the scheduler  32  distributes service in the lower level  34  to the backlogged sessions of a given service rate in a round-robin fashion. In addition, the root node at which the SEFF selector  54  operates is a GPS-related scheduler that regulates the access of the macro-sessions to the outgoing link determined by the minimum eligible timestamp. The system potential of the root server is maintained according to the evolution of the state of the N macro-sessions. 
     In order to achieve near-optimal delay bounds, the scheduler  32  uses a P-RPS or MD-SCFQ scheduler with the SEFF packet-selection policy in a SEFF selector  54  as the root node. At timeslot m, the scheduler  32  selects for the next service the connection j which is at the head of the rate queue J whose timestamp F J  is currently the minimum among the ones that satisfy the eligibility condition associated with the SEFF packet-selection policy:                F   J     =       min     1   ≤   I   ≤   N            F   I               (   2   )                                
     such that                  F   I     -     1     r   1         ≤     P        (   m   )               (   3   )                                
     Using a P-RPS or MD-SCFQ scheduler with SEFF packet-selection policy at the root node  54 , the following delay bounds hold for the no-per-connection-timestamp scheduler  32 :                D   i     ≤           b   i     +   1       r   i       +     1     4        r   i                   (   4   )                                
     in which b i  is a bucket size in a (b i , r i )-leaky-bucket-constrained session i. These delay bounds are very close to the optimal bounds for GPS-related schedulers in general. The delay bounds of the no-per-connection-timestamp scheduler  32  are better than the bounds of the discrete-rate scheduler  10  with per-connection timestamps. 
     Referring to FIGS. 4A-4B, the method of operation of the scheduler  32  is illustrated in the flowchart as performing the step of initially determining if there are any new packets to process in step  70 . If there are no new data packets, the method performs step  72 ; otherwise, the method performs step  74 . In step  72 , the method determines if there are any backlogged connections. If not, the method loops back to perform step  70 ; otherwise, if there are any backlogged connections in step  72 , the method performs step  74 , as described below. 
     Referring again to step  70 , if there are new data packets detected in step  70  for processing, the method selects one data packet in step  76 , identifies a connection for the data packet in step  78 , and stores the data packet in step  80  in a packet queue associated with the identified connection. The method then determines in step  82  if the queue length for the identified connection is equal to zero. If not, the method increments the queue length for the connection in step  84 , and proceeds to step  74 , as described below. Otherwise, if the queue length for the connection is not equal to zero in step  82 , the method increments the number of backlogged connections in step  86 , increments the queue length for the connection in step  88 , and identifies an appropriate rate queue (RQ) for the connection in step  90 . 
     The method then determines in step  92  if the rate queue for the connection is empty. If not, the method links the connection to the tail of the rate queue in step  94 , and sets the tail pointer of the rate queue in step  96  to queue the connection into the appropriate rate queue. For example, referring to FIG. 3, as the session  66  is entered into the rate FIFO queue  1  of the macro-session  56  labeled as macro-session  1 , the session  66  is linked to the tail  62 , and the tail  62  is set to point to the session  66 . 
     Referring again to FIG. 4A, in step  92 , if the rate queue appropriate to the connection is empty, the method sets the rate queue timestamp in step  98 , sets the rate queue head pointer in step  100 , and proceeds to step  96  to set the rate queue tail pointer. For example, referring to FIG. 3, prior to entry of the session  66 , when the session  64  is initially to be entered into the macro-session  56  in which the macro-session  56  is initially empty, the rate queue timestamp  42  is set in step  98 , the rate queue head pointer  48  is set to point to the session  64 , and the rate queue tail pointer  62  is set to point to the session  64 . 
     After steps  72 ,  84 , or  96  in FIG. 4A, the method proceeds in FIG. 4B to step  74  to determine if a transmitter is available. If not, the method loops back to step  70  to continue receiving new packets. Otherwise, if a transmitter is available for transmitting a packet, the method performs step  102  to determine if a serviced connection is available. If a serviced connection is not available in step  102 , the method selects a non-empty rate queue with the minimum eligible timestamp in step  104  for service by the SEFF selector  54 . The method then selects the connection at the head of the rate queue with the minimum eligible timestamp in step  106 , and sends a data packet from the packet queue associated with the selected connection to the transmitter in step  108 . The method then loops back to process new data packets in step  70 . 
     Referring back to step  102 , if a serviced connection is available, the method decrements the queue length for the serviced connection in step  110 , and the connection is removed from the corresponding rate queue in step  112 . The method then determines if the corresponding rate queue is empty in step  114 . If the corresponding rate queue is empty, the method proceeds to step  116 , as described below. Otherwise, if the rate queue is not empty in step  114 , the method updates the rate queue timestamp in step  118 , and updates the head pointer of the rate queue in step  120 . The method then proceeds to step  116 . 
     In step  116 , the method determines if the queue length for the serviced connection is equal to zero. If so, the method performs step  122 , as discussed below; otherwise, if the queue length is not equal to zero in step  116 , the method proceeds to step  124  to identify the rate queue of the serviced connection. The method then determines in step  126  if the identified rate queue is empty. If not, the method links the connection to the tail of the rate queue in step  128 , and sets the rate queue tail pointer to point to the serviced connection in step  130 . Referring back to step  126 , if the rate queue is empty, the method sets the timestamp of the rate queue in step  132 , sets the head pointer of the rate queue to point to the serviced connection in step  134 , and then proceeds to step  130  to set the rate queue tail pointer to point to the serviced connection. The system potential P( ) as used in equations (1) and (3) is then updated in step  136 , and the method proceeds to step  104 . 
     Referring back to step  116 , after it is determined that the queue length for the serviced connection is zero, the method proceeds to step  122  to decrement the number of backlogged connections. The method then determines if there are any backlogged connections in step  138 . If so, the method proceeds to step  136  and thence to step  104 . Otherwise, if the method in step  138  determines that there are no backlogged connections, the method loops back to step  70  to process new data packets. Accordingly, the method shown in FIGS. 4A-4B implements the packet and connection processing for the scheduler  32  shown in FIGS. 2-3. 
     By the foregoing a novel and unobvious scheduler and method has been disclosed by way of the preferred embodiment. However, numerous modifications and substitutions may be had without departing from the spirit of the invention. For example, while the preferred embodiment discusses an ATM scheduler, it is wholly within the purview of the invention to contemplate applying the no-per-connection-timestamps processing in the manner as set forth above to other packet network systems. Accordingly, the invention has been described by way of illustration rather than limitation.