Abstract:
A network-node bandwidth-allocation credit method includes computing credits after each completed scan of a packet-tracking queue. Such queue varies tremendously in depth, according to how much network traffic is transitioning through the involved network nodes. A bandwidth traffic-shaping manager operates to control the maximum bandwidth permitted to pass through each network node, e.g., by buffering datapackets that would exceed some service policy limit if forwarded immediately on receipt. As each network node runs less that its policy maximum, it is given a number of credits that collect in a bank account. If a datapacket presents itself that involves passage through the network node, such bank account is checked to see if sufficient bandwidth-allocation credits exist to forward the datapacket immediately. If so, an appropriate deduction of credits is made and the datapacket is forwarded toward its destination.

Description:
BACKGROUND OF THE INVENTION  
         [0001]    1. Field of the Invention  
           [0002]    The invention relates generally to computer network protocols and equipment for adjusting packet-by-packet bandwidth according to the source and/or destination IP-addresses of each such packet. More specifically, the present invention relates to methods and semiconductor devices for allocating network node bandwidth via a system of credits that are computed on a variable time basis.  
           [0003]    2. Description of the Prior Art  
           [0004]    Access bandwidth is important to Internet users. New cable, digital subscriber line (DSL), and wireless “always-on” broadband-access together are expected to eclipse dial-up Internet access in 2001. So network equipment vendors are scrambling to bring a new generation of broadband access solutions to market for their service-provider customers. These new systems support multiple high speed data, voice and streaming video Internet-protocol (IP) services, and not just over one access media, but over any media.  
           [0005]    Flat-rate access fees for broadband connections will shortly disappear, as more subscribers with better equipment are able to really use all that bandwidth and the systems&#39; overall bandwidth limits are reached. One of the major attractions of broadband technologies is that they offer a large Internet access pipe that enables a huge amount of information to be transmitted. Cable and fixed point wireless technologies have two important characteristics in common. Both are “fat pipes” that are not readily expandable, and they are designed to be shared by many subscribers.  
           [0006]    Although DSL allocates a dedicated line to each subscriber, the bandwidth becomes “shared” at a system aggregation point. In other words, while the bandwidth pipe for all three technologies is “broad,” it is always “shared” at some point and the total bandwidth is not unlimited. All broadband pipes must therefore be carefully and efficiently managed.  
           [0007]    Internet Protocol (IP) datapackets are conventionally treated as equals, and therein lies one of the major reasons for its “log jams”. When all IP-packets have equal right-of-way over the Internet, a “first come, first serve” service arrangement results. The overall response time and quality of delivery service is promised to be on a “best effort” basis only. Unfortunately all IP-packets are not equal, certain classes of IP-packets must be processed differently.  
           [0008]    In the past, such traffic congestion has caused no fatal problems, only an increasing frustration from the unpredictable and sometimes gross delays. However, new applications use the Internet to send voice and streaming video IP-packets that mix-in with the data IP-packets. These new applications cannot tolerate a classless, best efforts delivery scheme, and include IP-telephony, pay-per-view movie delivery, radio broadcasts, cable modem (CM), and cable modem termination system (CMTS) over two-way transmission hybrid fiber/coax (HFC) cable.  
           [0009]    Internet service providers (ISPs) need to be able to automatically and dynamically integrate service subscription orders and changes, e.g., for “on demand” services. Different classes of services must be offered at different price points and quality levels. Each subscriber&#39;s actual usage must be tracked so that their monthly bills can accurately track the service levels delivered. Each subscriber should be able to dynamically order any service based on time of day/week, or premier services that support merged data, voice and video over any access broadband media, and integrate them into a single point of contact for the subscriber.  
           [0010]    There is an urgent demand from service providers for network equipment vendors to provide integrated broadband-access solutions that are reliable, scalable, and easy to use. These service providers also need to be able to manage and maintain ever growing numbers of subscribers.  
           [0011]    Conventional IP-addresses, as used by the Internet, rely on four-byte hexadecimal numbers, e.g., 00H-FFH. These are typically expressed with four sets of decimal numbers that range 0-255 each, e.g., “192.55.0.1”. A single look-up table could be constructed for each of 4,294,967,296 (256 4 ) possible IP-addresses to find what bandwidth policy should attach to a particular datapacket passing through. But with only one byte to record the policy for each IP-address, that approach would require more than four gigabytes of memory. So this is impractical.  
           [0012]    There is also a very limited time available for the bandwidth classification system to classify a datapacket before the next datapacket arrives. The search routine to find which policy attaches to a particular IP-address must be finished within a finite time. And as the bandwidths get higher and higher, these search times get proportionally shorter.  
           [0013]    The straight forward way to limit-check each node in a hierarchical network is to test whether passing a just received datapacket would exceed the policy bandwidth for that node. If yes, the datapacket is queued for delay. If no, a limit-check must be made to see if the aggregate of this node and all other daughter nodes would exceed the limits of a parent node. And then a grandparent node, and so on. Such sequential limit check of hierarchical nodes was practical in software implementations hosted on high performance hardware platforms. But it is impractical in a pure hardware implementation, e.g., a semiconductor integrated circuit.  
           [0014]    The determination of whether there exists sufficient bandwidth-allocation “credit” at each network node at any one instant must be done periodically. A first approach to issuing credits involved 100-Mbps networks where updates on a twenty-five millisecond schedule was adequate. But newer, higher speed networks are contemplated that will operate at 10-Gbps and higher. The twenty-five millisecond schedule for updating bandwidth-allocation credits at each network node is far too slow.  
         SUMMARY OF THE PRESENT INVENTION  
         [0015]    It is therefore an object of the present invention to provide a method for allocating network-node bandwidth-allocation credits.  
           [0016]    It is another object of the present invention to provide a mechanism for allocating network bandwidth-allocation credits on a variable-time basis.  
           [0017]    It is a further object of the present invention to provide a method for allocating network bandwidth-allocation credits after each scan of a packet-tracking queue with dynamic size.  
           [0018]    Briefly, a network-node bandwidth-allocation credit method embodiment of the present invention includes computing credits after each completed scan of a packet-tracking queue. Such queue varies tremendously in depth, according to how much network traffic is transitioning through the involved network nodes. A bandwidth traffic-shaping manager operates to control the maximum bandwidth permitted to pass through each network node, e.g., by buffering datapackets that would exceed some service policy limit if forwarded immediately on receipt. As each network node runs less that its policy maximum, it is given a number of credits that collect in a bank account. If a datapacket presents itself that involves passage through the network node, such bank account is checked to see if sufficient bandwidth-allocation credits exist to forward the datapacket immediately. If so, an appropriate deduction of credits is made and the datapacket is forwarded toward its destination.  
           [0019]    An advantage of the present invention is a device and method are provided for allocating bandwidth to network nodes according to a policy.  
           [0020]    A still further advantage of the present invention is a semiconductor intellectual property is provided that prioritizes datapacket transfers according to service-level agreement policies in real time and at high datapacket rates.  
           [0021]    These and many other objects and advantages of the present invention will no doubt become obvious to those of ordinary skill in the art after having read the following detailed description of the preferred embodiments which are illustrated in the drawing figures. 
       
    
    
     IN THE DRAWINGS  
       [0022]    [0022]FIG. 1 is a schematic diagram of a hierarchical network embodiment of the present invention with a gateway to the Internet;  
         [0023]    [0023]FIG. 2 is a diagram of a single queue embodiment of the present invention for checking and enforcing bandwidth service level policy management in a hierarchical network;  
         [0024]    [0024]FIG. 3 is a functional block diagram of a system of interconnected semiconductor chip components that include a traffic-shaping cell and classifier, and that implements various parts of FIGS. 1 and 2; and  
         [0025]    [0025]FIG. 4 is a flowchart of a method embodiment of the present invention for allocating network bandwidth-allocation credits after each scan of a packet-tracking queue with dynamic size. 
     
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS  
       [0026]    [0026]FIG. 1 represents a hierarchical network embodiment of the present invention, and is referred to herein by the general reference numeral  100 . The network  100  has a hierarchy that is common in cable network systems. Each higher level node and each higher level network is capable of data bandwidths much greater than those below it. But if all lower level nodes and networks were running at maximum bandwidth, their aggregate bandwidth demands would exceed the higher level&#39;s capabilities.  
         [0027]    The network  100  therefore includes bandwidth management that limits the bandwidth made available to daughter nodes, e.g., according to a paid service-level policy. Higher bandwidth policies are charged higher access rates. Even so, when the demands on all the parts of a branch exceed the policy for the whole branch, the lower-level demands are trimmed back. For example, to keep one branch from dominating trunk-bandwidth to the chagrin of its peer branches.  
         [0028]    The present Assignee, Amplify.net, Inc., has filed several United States Patent Applications that describe such service-level policies and the mechanisms to implement them. Such include INTERNET USER-BANDWIDTH MANAGEMENT AND CONTROL TOOL, now U.S. Pat. No. 6,085,241, issued Mar. 14, 2000; BANDWIDTH SCALING DEVICE, Ser. No. 08/995,091, filed Dec. 19, 1997; BANDWIDTH ASSIGNMENT HIERARCHY BASED ON BOTTOM-UP DEMANDS, Ser. No. 09/718,296, filed Nov. 21, 2000; NETWORK-BANDWIDTH ALLOCATION WITH CONFLICT RESOLUTION FOR OVERRIDE, RANK, AND SPECIAL APPLICATION SUPPORT, Ser. No. 09/716,082, filed Nov. 16, 2000; GRAPHICAL USER INTERFACE FOR DYNAMIC VIEWING OF DATAPACKET EXCHANGES OVER COMPUTER NETWORKS, Ser. No. 09/729,733, filed Dec. 14, 2000; ALLOCATION OF NETWORK BANDWIDTH ACCORDING TO NETWORK APPLICATION, Ser. No. 09/718,297, filed Nov. 21, 2001; METHOD FOR ASCERTAINING NETWORK BANDWIDTH ALLOCATION POLICY ASSOCIATED WITH APPLICATION PORT NUMBERS, (Docket SS-709-07) Ser. No. 09/______, filed Aug. 2, 2001; and METHOD FOR ASCERTAINING NETWORK BANDWIDTH ALLOCATION POLICY ASSOCIATED WITH NETWORK ADDRESS, (Docket SS-709-08) Ser. No. 09/______, filed Aug. 7, 2001. All of which are incorporated herein by reference.  
         [0029]    Suppose the network  100  represents a city-wide cable network distribution system. A top trunk  102  provides a broadband gateway to the Internet and it services a top main trunk  104 , e.g., having a maximum bandwidth of 100-Mbps. At the next lower level, a set of cable modem termination systems (CMTS)  106 ,  108 , and  110 , each classifies traffic into data, voice and video  112 ,  114 , and  116 . If each of these had bandwidths of 50-Mbps, then all three running at maximum would need 150-Mbps at top main trunk  104  and top gateway  102 . A policy-enforcement mechanism is included that limits, e.g., each CMTS  106 ,  108 , and  110  to 45-Mbps and the top Internet trunk  102  to 100-Mbps. If all traffic passes through the top Internet trunk  102 , such policy-enforcement mechanism can be implemented there alone.  
         [0030]    Each CMTS supports multiple radio frequency (RF) channels  118 ,  120 ,  122 ,  124 ,  126 ,  128 ,  130 , and  132 , which are limited to a still lower bandwidth, e.g., 38-Mbps each. A group of neighborhood networks  134 ,  136 ,  138 ,  140 ,  142 , and  144 , distribute bandwidth to end users  146 - 160 , e.g., individual cable network subscribers residing along neighborhood streets. Each of these could buy 5-Mbps bandwidth service level policies, for example.  
         [0031]    The integration of class-based queues and datapacket classification mechanisms in semiconductor chips necessitates more efficient implementations, especially where bandwidths are exceedingly high and the time to classify and policy-check each datapacket is exceedingly short. Therefore, embodiments of the present invention manage every datapacket in the whole network  100  from a single queue. Rather, as in previous embodiments, than maintaining queues for each node A-Z, and AA, and checking each higher-level queue in sequence to see if a datapacket should be held or forwarded. Although this example describes a topology of four levels of aggregation hierarchy, six levels have been implemented and there is no limit of the number of levels.  
         [0032]    Each entry in the single queue includes fields for pointers to end user source and all higher level hierarchical nodes. The node data structure contains credit counts for each node. The entire credit fields of all nodes are tested in one clock cycle to see if enough credit exists at each node level to pass the datapacket along.  
         [0033]    [0033]FIG. 2 illustrates a single queue  200  and several entries  201 - 213 . A first entry  201  is associated with a datapacket sourced from or destined for subscriber node (M)  146 . If such datapacket needs to climb the hierarchy of network  100  (FIG. 1) to access the Internet, the service level policies of user nodes (M)  146 , and hierarchical nodes (E)  118 , (B)  106  and (A)  102  will all be involved in the decision whether or not to forward the datapacket or delay it. Similarly, another entry  212  is associated with a datapacket sourced from or destined for subscriber node (X)  157 . If such datapacket also needs to climb the hierarchy of network  100  (FIG. 1) to access the Internet, the service level policies of the nodes (X)  157 , (K)  130 , (D)  110  and (A)  102  will all be involved in the decision whether or not to forward such datapacket or delay it.  
         [0034]    There are many ways to implement the queue  200  and the fields included in each entry  201 - 213 . The instance of FIG. 2 is merely exemplary. A buffer-pointer field  214  points to where the actual data for the datapacket resides in a buffer memory, so that the queue  200  doesn&#39;t have to spend time and resources shuffling the whole datapacket header and payload around. A node pointer field  215 - 218  is divided into four subfields that represent the pointer to four possible levels of the hierarchy for each subscriber node  146 - 160  or nodes  126  and  128 .  
         [0035]    [0035]FIG. 3 represents a bandwidth management system  300  in an embodiment of the present invention. The bandwidth management system  300  is preferably implemented in semiconductor integrated circuits (IC&#39;s). The bandwidth management system  300  comprises a static random access memory (SRAM) bus  302  connected to an SRAM memory controller  304 . A direct memory access (DMA) engine  306  helps move blocks of memory in and out of an external SRAM array. A protocol processor  308  parses application protocol to identify the dynamically assigned TCP/UDP port number then communicates datapacket header information with a datapacket classifier  310 . Datapacket identification and pointers to the corresponding service level agreement policy are exchanged with a traffic shaping (TS) cell  312  implemented as a single chip or synthesizable semiconductor intellectual property (SIA) core. Such datapacket identification and pointers to policy are also exchanged with an output scheduler and marker  314 . A microcomputer (CPU)  316  directs the overall activity of the bandwidth management system  300 , and is connected to a CPU RAM memory controller  318  and a RAM memory bus  320 . External RAM memory is used for execution of programs and data for the CPU  316 . The external SRAM array is used to shuffle the network datapackets through according to the appropriate service level policies.  
         [0036]    The datapacket classifier  310  first identifies the end user service level policy (the policy associated with nodes  146 - 160 ). Every end user policy also has its corresponding policies associated with all parent nodes of this user node. The classifier passes an entry that contains a pointer to the datapacket itself that resides in the external SRAM and the pointers to all corresponding nodes for this datapacket, i.e. the user nodes and its parent node. Each node contains the service level agreement policies such as bandwidth limit (CR and MBR) and the current available credit for a datapacket to go through.  
         [0037]    A calculation periodically deposits credits in each four subcredit fields to indicate the availability of bandwidth, e.g., one credit for enough bandwidth to transfer one datapacket through the respective node. When a decision is made to either forward or hold a datapacket represented by each corresponding entry  201 - 213 , the credit field  217  is inspected. If all subfields indicate a credit and none are zero, then the respective datapacket is forwarded through the network  100  and the entry cleared from queue  200 . The consumption of the credit is reflected in a decrement of each involved subfield. For example, if the inspection of entry  201  resulted in the respective datapacket being forwarded, the credits for nodes M, E, B, and A would all be decremented for entries  202 - 213 . This may result in zero credits for entry  202  at the E, B, or A levels. If so, the corresponding datapacket for entry  202  would be held.  
         [0038]    The single queue  200  also prevents datapackets from-or-to particular nodes from being passed along out of order. The TCP/IP protocol allows and expects datapackets to arrive in random order, but network performance and reliability is best if datapacket order is preserved. UDP traffic used for voice and video will get in trouble if order is not preserved.  
         [0039]    The service-level policies are defined and input by a system administrator. Internal hardware and software are used to spool and despool datapacket streams through at the appropriate bandwidths. In business model implementations of the present invention, subscribers are charged various fees for different levels of service, e.g., better bandwidth and delivery time-slots.  
         [0040]    A network embodiment of the present invention comprises a local group of network workstations and clients with a set of corresponding local IP-addresses. Those local devices periodically need access to a wide area network (WAN). A class-based queue (CBQ) traffic shaper is disposed between the local group and the WAN, and provides for an enforcement of a plurality of service-level agreement (SLA) policies on individual connection sessions by limiting a maximum data throughput for each such connection. The class-based queue traffic shaper preferably distinguishes amongst voice-over-IP (voIP), streaming video, and datapackets. Any sessions involving a first type of datapacket can be limited to a different connection-bandwidth than another session-connection involving a second type of datapacket. The SLA policies are attached to each and every local IP-address, and any connection-combinations with outside IP-addresses can be ignored.  
         [0041]    A variety of network interfaces can be accommodated, either one type at a time, or many types in parallel. For example, a wide area network (WAN) media access controller (MAC)  322  presents a media independent interface (MII)  324 , e.g., 100BaseT fast Ethernet. A universal serial bus (USB) MAC  326  presents a media independent interface (MII)  328 , e.g., using a USB-2.0 core. A local area network (LAN) MAC  330  has an MII connection  332 . A second LAN MAC  334  also presents an MII connection  336 . Other protocol and interface types include home phoneline network alliance (HPNA) network, IEEE-802.11 wireless, etc. Datapackets are received on their respective networks, classified, and either sent along to their destination or stored in SRAM to effectuate bandwidth limits at various nodes, e.g., “traffic shaping”.  
         [0042]    The protocol processor  308  aids in the dynamic creation of policies associated with certain traffic flows. For example, to support video conferencing, one wants to be able to create a 300-Kbit/sec policy to support such calls whenever they start up. However, according to the H.323 protocol used in video conferencing, the actual port number associated with a particular call are negotiated during the call set up phase. The protocol processor  308 , monitors the call set up phase of the H.323 protocol, extracts the negotiated parameters, and then passes those to the micro processor so that the appropriate policy can be created.  
         [0043]    The protocol processor  308  is implemented as a table-driven state engine, with as many as two hundred and fifty-six concurrent sessions and sixty-four states. The die size for such an IC is currently estimated at 20.00 square millimeters using 0.18 micron CMOS technology.  
         [0044]    The classifier  310  preferably manages as many as two hundred and fifty-six policies using IP-address, MAC-address, port-number, and handle classification parameters. Content addressable memory (CAM) can be used in a good design implementation. The die size for such an IC is currently estimated at 3.91 square millimeters using 0.18 micron CMOS technology.  
         [0045]    The traffic shaping (TS) cell  312  preferably manages as many as two hundred and fifty-six policies using CIR, MBR, virtual-switching, and multicast-support shaping parameters. A typical TS cell  312  controls three levels of network hierarchy, e.g., as in FIG. 1. A single queue is implemented to preserve datapacket order, as in FIG. 2. Such TS cell  312  is preferably self-contained with its on chip-based memory. The die size for such an IC is currently estimated at 2.00 square millimeters using 0.18 micron CMOS technology.  
         [0046]    The traffic-shaping cell repeatedly scans the variable-depth queue to determine whether a datapacket should be forwarded through the node by checking for enough bandwidth-allocation credits, and it replenishes the bandwidth-allocation credits calculating in the variable delay caused by scanning the variable-depth queue.  
         [0047]    The output scheduler and marker  314  schedules datapackets according to DiffServ Code Points and datapacket size. The use of a single queue is preferred. Marks are inserted according to parameters supplied by the TS cell  312 , e.g., DiffServ Code Points. The die size for such an IC is currently estimated at 0.93 square millimeters using 0.18 micron CMOS technology.  
         [0048]    The CPU  316  is preferably implemented with an ARM740T core processor with 8K of cache memory. MIPS and POWER-PC are alternative choices. Cost here is a primary driver, and the performance requirements are modest. The die size for such an IC is currently estimated at 2.50 square millimeters using 0.18 micron CMOS technology. The control firmware supports four provisioning models: TFTP/Conf_file, simple network management protocol (SNMP), web-based, and dynamic. The TFTP/Conf_file provides for batch configuration and batch-usage parameter retrieval. The SNMP provides for policy provisioning and updates. User configurations can be accommodated by web-based methods. The dynamic provisioning includes auto-detection of connected devices, spoofing of current state of connected devices, and on-the-fly creation of policies.  
         [0049]    In an auto-provisioning example, when a voice over IP (VoIP) service is enabled the protocol processor  308  is set up to track SIP, or CQoS, or both. As the VoIP phone and the gateway server run the signaling protocol, the protocol processor  308  extracts the IP-source, IP-destination, port-number, and other appropriate parameters. These are then passed to CPU  316  which sets up the policy, and enables the classifier  310 , the TS cell  312 , and the scheduler  314 , to deliver the service.  
         [0050]    If the bandwidth management system  300  were implemented as an application specific programmable processor (ASPP), the die size for such an IC is currently estimated at 35.72 square millimeters, at 100% utilization, using 0.18 micron CMOS technology. About one hundred and ninety-four pins would be needed on the device package. In a business model embodiment of the present invention, such an ASPP version of the bandwidth management system  300  would be implemented and marketed as hardware description language (HDL) in semiconductor intellectual property (SIA) form, e.g., Verilog code.  
         [0051]    [0051]FIG. 4 represents a method embodiment of the present invention for allocating network bandwidth-allocation credits after each scan of a packet-tracking queue with dynamic size, and is referred to herein by the general reference numeral  400 . The method  400  comprises a step  402  which scans a variable-depth queue, e.g., queue  200  (FIG. 2). Such scan can take longer to complete, depending on the number of entries then existing in the queue. A typical scan includes a step  404  in which a decision is made whether to forward the datapacket represented by the queue entry. Enough bandwidth-allocation credits must exist at each controlled network node to afford the passing through of this datapacket, i.e., given the size in bytes of the datapacket. So a step  406  either deducts the credits from each of the accounts of the involved controlled network nodes and schedules the datapacket for forwarding through. The queued entry for this packet is removed from the queue  200  and is passed to output scheduler/marker  314 . If not enough credit is found in any of the nodes, the datapacket will remain in the queue until all the involved controlled network nodes gain sufficient credits in the later check. A step  408  determines how much time has elapsed since the last credit update. More credits will be deposited for more time having elapsed during the queue scan. A step  410  computes how many credits should be deposited in each of the accounts of the involved controlled network nodes, according to the computed time from step  408  and the bandwidth-allocation service-level policy associated with each. The process then repeats in a never-ending loop, and can be implemented therefore as a state-machine.  
         [0052]    Although the present invention has been described in terms of the presently preferred embodiments, it is to be understood that the disclosure is not to be interpreted as limiting. Various alterations and modifications will no doubt become apparent to those skilled in the art after having read the above disclosure. Accordingly, it is intended that the appended claims be interpreted as covering all alterations and modifications as fall within the true spirit and scope of the invention.