Abstract:
Method and system for hardware packet pacing using a direct memory access controller in a parallel, in one aspect, keeps track of a total number of bytes put on the network as a result of a remote get operation, using a hardware token counter. A remote get message is sent as a plurality of sub remote get packets. Each of the sub remote get packets is sent if the total number of bytes put on the network does not exceed a predetermined number.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
       [0001]    The present invention is related to the following commonly-owned, co-pending United States patent applications filed on even date herewith, the entire contents and disclosure of each of which is expressly incorporated by reference herein as if fully set forth herein. U.S. patent application Serial No. (YOR920070268US1 (21189)), for “A SHARED PERFORMANCE MONITOR IN A MULTIPROCESSOR SYSTEM”; U.S. patent application Ser. No. (YOR920070293US1 (21233)), for “OPTIMIZED COLLECTIVES USING A DMA ON A PARALLEL COMPUTER”; U.S. patent application Ser. No. (YOR920070295US1 (21232)), for “DMA SHARED BYTE COUNTERS IN A PARALLEL COMPUTER”; U.S. patent application Ser. No. (YOR920070297US1 (21208)), for “MULTIPLE NODE REMOTE MESSAGING”; U.S. patent application Ser. No. (YOR920070298US1 (21209)), for “A METHOD AND APPARATUS OF PREFETCHING STREAMS OF VARYING PREFETCH DEPTH”; U.S. patent application Ser. No. (YOR920070299US1 (21212)), for “PROGRAMMABLE PARTITIONING FOR HIGH-PERFORMANCE COHERENCE DOMAINS IN A MULTIPROCESSOR SYSTEM”; U.S. patent application Ser. No. (YOR920070300US1 (21211)), for “METHOD AND APPARATUS FOR SINGLE-STEPPING COHERENCE EVENTS IN A MULTIPROCESSOR SYSTEM UNDER SOFTWARE CONTROL”; U.S. patent application Ser. No. (YOR920070301US1 (21210)), for “INSERTION OF COHERENCE EVENTS INTO A MULTIPROCESSOR COHERENCE PROTOCOL”; U.S. patent application Ser. No. (YOR920070302US1 (21216), for “METHOD AND APPARATUS TO DEBUG AN INTEGRATED CIRCUIT CHIP VIA SYNCHRONOUS CLOCK STOP AND SCAN”; U.S. patent application Ser. No. (YOR920070303US1 (21236)), for “DMA ENGINE FOR REPEATING COMMUNICATION PATTERNS”; U.S. patent application Ser. No. (YOR920070304US1 (21239)), for “METHOD AND APPARATUS FOR A CHOOSE-TWO MULTI-QUEUE ARBITER”; U.S. patent application Ser. No. (YOR920070305US1 (21238)), for “METHOD AND APPARATUS FOR EFFICIENTLY TRACKING QUEUE ENTRIES RELATIVE TO A TIMESTAMP”; U.S. patent application Ser. No. (YOR920070307US1 (21245)), for “BAD DATA PACKET CAPTURE DEVICE”; U.S. patent application Ser. No. (YOR920070321US1 (21256)), for “EXTENDED WRITE COMBINING USING A WRITE CONTINUATION HINT FLAG”; U.S. patent application Ser. No. (YOR920070322US1 (21255)), for “A SYSTEM AND METHOD FOR PROGRAMMABLE BANK SELECTION FOR BANKED MEMORY SUBSYSTEMS”; U.S. patent application Ser. No. (YOR920070323US1 (21246)), for “AN ULTRASCALABLE PETAFLOP PARALLEL SUPERCOMPUTER”; U.S. patent application Ser. No. (YOR920070324US1 (21264)), for “SDRAM DDR DATA EYE MONITOR METHOD AND APPARATUS”; U.S. patent application Ser. No. (YOR920070337US1 (21281)), for “A CONFIGURABLE MEMORY SYSTEM AND METHOD FOR PROVIDING ATOMIC COUNTING OPERATIONS IN A MEMORY DEVICE”; U.S. patent application Ser. No. (YOR920070338US1 (21293)), for “ERROR CORRECTING CODE WITH CHIP KILL CAPABILITY AND POWER SAVING ENHANCEMENT”; U.S. patent application Serial No. (YOR920070339US1 (21292)), for “STATIC POWER REDUCTION FOR MIDPOINT-TERMINATED BUSSES”; U.S. patent application Ser. No. (YOR920070340US1 (21295)), for “COMBINED GROUP ECC PROTECTION AND SUBGROUP PARITY PROTECTION”; U.S. patent application Ser. No. (YOR920070355US1 (21299)), for “A MECHANISM TO SUPPORT GENERIC COLLECTIVE COMMUNICATION ACROSS A VARIETY OF PROGRAMMING MODELS”; U.S. patent application Ser. No. (YOR920070356US1 (21263)), for “MESSAGE PASSING WITH A LIMITED NUMBER OF DMA BYTE COUNTERS”; U.S. patent application Serial No. (YOR920070357US1 (21312)), for “ASYNCRONOUS BROADCAST FOR ORDERED DELIVERY BETWEEN COMPUTE NODES IN A PARALLEL COMPUTING SYSTEM WHERE PACKET HEADER SPACE IS LIMITED”; and U.S. patent application Ser. No. (YOR920070371US1 (21335)), for “POWER THROTTLING OF COLLECTIONS OF COMPUTING ELEMENTS”. 
     
    
     STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT 
       [0002]    This invention was made with Government support under Contract. No. B554331 awarded by Department of Energy. The Government has certain rights in this invention. 
     
    
     FIELD OF THE INVENTION 
       [0003]    The present disclosure relates generally to message passing in parallel computers, and more particularly to maintaining high performance for long messages in a parallel computing by having the DMA engine control the number of packets injected into the network, thereby preventing network buffers from filling up. 
       BACKGROUND OF THE INVENTION 
       [0004]    As is well known, throughput of networks can degrade if internal network buffers fill up. Full buffers in one part of the network can prevent other packets from passing through that part of the network. Algorithms such as TCP/IP use “pacing”, or window flow control algorithms, to limit the number of packets in the network; this can improve throughput. These algorithms use acknowledgement packets to grant a sender permission to send additional packets. However, the software overhead to implement such algorithms is excessive in a scientific parallel computing environment where high throughput and low latency are essential. 
         [0005]    Thus, it is desirable to have a pacing mechanism that can be integrated into the hardware that would eliminate software overhead. 
       BRIEF SUMMARY OF THE INVENTION 
       [0006]    Method and system for hardware packet pacing using a direct memory access controller in a parallel computer are provided. The method in one embodiment may comprise establishing a token counter on a direct memory access controller initially set to a first predetermined value and establishing maximum pacing submessage size, the maximum pacing submessage size being a value less than or equal to the first predetermined value. The method may further comprise establishing a remaining bytes count, the remaining bytes count initially set to a message length field value in an original remote get packet and setting a submessage size to the maximum pacing submessage size or the remaining bytes count, whichever is less. The method may also comprise waiting for the token counter to be greater than or equal to the submessage size, injecting a remote get packet of the submessage size to a network when the token counter is greater than or equal to the submessage size and decrementing the token counter and the remaining bytes count by the submessage size. The method may further comprise repeating the steps of setting, waiting and injecting until the remaining bytes count is zero. 
         [0007]    Still yet, the method may comprise detecting a put packet received on the direct memory access controller and incrementing the token byte counter by a number of payload bytes specified in the put packet. 
         [0008]    In another aspect, a method of hardware packet pacing using a direct memory access controller in a parallel computer may comprise detecting a remote get message descriptor in an injection fifo associated with a direct memory controller and retrieving put descriptor information from information in the remote get message descriptor. The put description information includes at least a message size, an injection offset and a reception offset. The method may also comprise setting a pacing size, setting a token counter to a predetermined value greater than the pacing size, and assembling a new remote get packet using the put description information. The new remote get packet specifies a message size that is at most the pacing size. The method may further comprise, before sending the new remote get packet, waiting until the token counter is greater than or equal to the message size specified in the new remote get packet and sending the new remote get packet. The method may still further comprise decrementing the token counter by the message size specified in the new remote get packet, and incrementing the injection offset and the reception offset by the message size specified in the new remote get packet. The method may further comprise repeating the steps of assembling, waiting, sending, decrementing and incrementing until the message size of the remote get has been processed by all the new remote get packets. 
         [0009]    Yet in another aspect, a method of hardware packet pacing using a direct memory access controller in a parallel computer may comprise dividing a remote get packet into a plurality of sub remote get packets, tracking a hardware token counter, the hardware token counter representing a total number of bytes being processed at one time as a result of sending one or more of the sub remote get packets, and controlling the total number of bytes being processed at one time using the hardware token counter. In another aspect, the step of controlling may include controlling sending of the plurality of sub remote get packets based on the hardware token counter. 
         [0010]    A system for hardware packet pacing using a direct memory access controller in a parallel computer having multiple processing nodes, in one aspect, may comprise a hardware token counter initially set to a predetermined positive value, a memory, and a direct memory access controller operable to detect a remote get packet in the memory. The direct memory access controller is further operable to assemble a plurality of sub remote get packets using information contained in the remote get packet, and to pace sending of said plurality of sub remote get packets based on the hardware token counter. The direct memory access controller may be further operable to increment the hardware token counter when a put packet is received and to decrement the hardware token when one of the said plurality of sub remote get packets are injected into the memory. 
         [0011]    Further features as well as the structure and operation of various embodiments are described in detail below with reference to the accompanying drawings. In the drawings, like reference numbers indicate identical or functionally similar elements. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0012]      FIG. 1  illustrates an architectural diagram of a node in a parallel computer system having a DMA engine. 
           [0013]      FIG. 2  illustrates DMA injection and reception counter structure in one embodiment. 
           [0014]      FIG. 3  illustrates relationship of the counter base address to the message buffer and the initial offset in a message descriptor. 
           [0015]      FIG. 4A  illustrates a plurality of fields for remote get messages used in the present disclosure in one embodiment. 
           [0016]      FIG. 4B  illustrates a plurality of fields for a put descriptor used in the present disclosure in one embodiment. 
           [0017]      FIG. 5  illustrates the steps taken by a DMA in one embodiment when it receives a put packet. 
           [0018]      FIG. 6  illustrates the DMA logic for injecting messages from a DMA injection FIFO into the network in one embodiment. 
       
    
    
     DETAILED DESCRIPTION 
       [0019]    In one embodiment of the present disclosure, a pacing mechanism is provided that can be integrated into the hardware, for example, hardware of the DMA engine. Software overhead may thus be eliminated. No acknowledgement packets are required. This in one embodiment is enabled by using remote gets. In BlueGene/P, there is a DMA engine that is integrated onto the same chip as the processors, cache memory, memory controller and network logic. This DMA engine injects packets from a message into the network. The system and method of the present disclosure in one embodiment use a DMA engine to control or limit the number of packets in the network without using acknowledgement packets and its accompanying complexity and overhead. 
         [0020]      FIG. 1  illustrates an architectural diagram of a node in a parallel computer system having a DMA engine. A parallel computer is shown in  FIG. 1  with multiple nodes  102 ,  104 ,  106  connected together by a network  108 . Each node may be based on the chip process that integrates all the functions of a computer into a single compute ASIC, enabling reduction of node size and power consumption. An ASIC (application-specific integrated circuit) is a microchip designed for a special application. In a supercomputer, this can be further leveraged to increase node density thereby decreasing the overall cost and increasing performance for the machine. Each node may function as both a compute node and an I/O (input/output) node in the system, and include multiple processing cores. The processor core may be a PowerPC450 embedded core available from IBM microelectronics, although future versions of this core may be used as technology improves. A node further may incorporate other functions into the ASIC. Besides the embedded processing core and floating point cores, a node may include embedded DRAM (dynamic random access memory), an integrated external DDR2 (double-data-rate two) memory controller, DMA (direct memory access), Gb, 10 Gb Ethernet functionality as well as all the network link cut-through routing buffers and routing control block that allow any two nodes to communicate with low latency. Each core or processor (for example,  110 ,  112 , etc.) is capable of being utilized for message handling and computation operations. 
         [0021]    A node  102  shown in  FIG. 1  includes multiple processors or cores  110  . . .  112 , a memory  114  and a DMA  116 . The memory  114  may be DRAM, SDRAM or any other memory. The DMA  116  includes a processor interface  122 , DMA logic  124 , a memory interface  126 , and a network interface  128 , Injection Counters  130 , Injection Fifo Metadata  132 , Reception Counters  134 , Reception Fifo Metadata  136  and status and control registers  138 . The Injection Fifo metadata  132  describes where in memory  114  the Injection Fifos  118  are located and the current head and tail of the Fifos  118 . The Reception Fifo metadata  136  describes where in memory  114  the Reception Fifos  120  are located and the current head and tail of the Fifos  120 . Thus, DMA has pointers to the fifos in memory  114 , for example, by means of a reception fifo metadata  136  and injection fifo metadata  132 . Injection fifos  118  in memory store message descriptors associated with message packets for injection to the network, and reception Fifos  120  in memory store packets received from the network. Memory interface  126  is used to read and write data to the memory  114  from the DMA  116 . For example, DMA logic  124  may update injection fifos  118  and reception fifos  120  via the memory interface  126 . One or more processors  110 ,  112  on the node  102  communicate with DMA  116  via a processor interface  122 . The control registers  138  are used to properly configure the DMA  116 . The status registers  138  reflect the current status, such as error conditions of the DMA  116  or which counters have hit zero. 
         [0022]      FIG. 2  shows multiple injection counters  1  to m  202  and multiple reception counters  1  to m  204 . Each injection counter ( 206 ,  208 , etc.) includes a base address  214  and a byte counter value  216 . Each reception counter ( 210 ,  212 , etc.) includes a base address  218  and a byte counter value  220 . Injection counters and reception counters are identified using counter identifiers (ids). 
         [0023]    For long messages implemented as puts, sending and receiving nodes agree on which injection counter  130  and reception  134  counter to use, and what base offset from memory location to use for a message being processed. Such agreements may be reached by sending a protocol message(s) in which a short memory Fifo message is put into reception FIFOs of a receiving node such as shown at  120 . In another embodiment, software can be constructed so that the counter ids and offsets can be agreed upon without sending protocol messages. Long message transfer may be initiated, for example, as a core processor on the sending node places a “put” message descriptor into an injection FIFO  118 , writing the injection counter  130  with base address and counter value via the processor interface  122 , and updating the injection FIFO metadata  132  for that message, for instance, advancing a tail pointer indicating the “last” message descriptor in the injection FIFO  118 . DMA logic  124  reads the injection FIFO metadata  132  and recognizes which FIFOs  118  have messages to be sent. 
         [0024]      FIG. 3  shows the relationship between a counter  302  and the message buffer  304  to be sent and/or received. The counter base address  310  is set to a value which is a lower bound  306  on the address of the message buffer  304 . The initial message offset  308  specified in the message descriptor is the starting address of the message buffer minus the base address  310  of the counter  302 . 
         [0025]    The system and method of the present disclosure in one embodiment utilizes types of messages such as remote gets and direct puts or the like. Short memory Fifo messages are used to convey control information between nodes. 
         [0026]      FIG. 4A  illustrates a plurality of fields for remote get message descriptors used in the present disclosure in one embodiment. The network packet header  402  may include network routing information, the address of the destination node, the packet type (a remote get), packet size and a packet pacing bit (set to 0), etc. The message size field  404  has the message length. The injection Fifo id  406  specifies into which injection Fifo on the destination node the payload of the resulting remote get packet corresponding to this message should be placed; the remote get payload is a put descriptor. Briefly, payload refers to data of the message packet that is not overhead information such as header or control information. The injection counter id field  408  includes an identifier for the injection counter. The injection counter offset field  410  includes a pointer or address that points to start of message payload, which in this case may be a put descriptor. The DMA uses the injection counter identifier and the offset specified in those fields to determine the location of the payload of this message. The location is the base address of the specified counter plus the offset in the descriptor. The message pacing bit  412  specifies whether or not this remote get is subject to pacing. 
         [0027]      FIG. 4B  illustrates a plurality of fields for a put descriptor used in the present disclosure in one embodiment. The network packet header  414  may include network routing information, the address of the destination node, the packet type (a put), packet size and a packet pacing bit, etc. The pacing bit should be set to 1 if the put is subject to pacing, otherwise to 0. The message size field  416  specifies the length of the message, the injection counter id field  418  specifies an identifier for the injection counter being used, and the offset field  420  specifies the offset from the memory location pointed to by the base address in the specified injection counter. Data is sent from the specified injection counter&#39;s base address plus the offset. The reception counter id field  422  specifies the identifier of a reception counter in the destination node, and the offset field  424  specifies an offset from the memory location of the base address in the reception counter. The message is to be placed in memory on the destination node starting at the specified reception counter&#39;s base address plus the offset specified in the offset field  424 . 
         [0028]    Referring to  FIG. 1 , normally without pacing, when a DMA logic detects a non-empty injection Fifo, the DMA logic causes the memory interface  126  to read the descriptor at the head of the Injection FIFO  118 . Assuming the message descriptor specifies a put message, the descriptor includes the injection ( 130 ) and reception counter ( 134 ) identifications to be used, the message length, the initial injection and reception offsets of the message, the destination node and other network routing information. The DMA  116  retrieves the message from the location in memory specified by the base address read from the injection counter  130  and offset, and assembles it into packets to be “put” on to the network  108 . DMA  116  assembles the packet including the message and the information regarding an offset from the base address specified in the reception counter  134  where the data from this packet is to be stored at the receiving node, and a count of how many payload bytes in this packet should be written. DMA  116  updates this information correctly for each packet, and puts the packets into the network interface  128 . The packet enters the network and is routed to another node, for instance, the destination compute node. 
         [0029]    After DMA  116  puts the packet in the network interface  128 , it decrements the counter valued of the specified injection counter  130  by the number of payload bytes in the packet. Upon reaching the destination, the packet is put into the network interface at that compute node (e.g.,  104  or  106 ), and the node&#39;s local DMA “recognizes” that the packet is there. Without pacing, for a put packet, the receiving node&#39;s DMA reads the reception counter identifier, offset and count from the received packet, looks up the reception counter base address, writes the appropriate number of payload bytes specified in the packet starting at the base plus packet offset, and then decrements the counter value by the payload bytes received. 
         [0030]    In one embodiment of the present disclosure, the DMA implements a remote get capability. With remote get, one node (e.g.,  104 ) can inject a short remote get packet into the network destined for another node (e.g.,  102 ) telling node at  102  to send M bytes of data back to node at  104 . Thus, if a remote get operation is used, instead of the processor on the sending node injecting a descriptor into the injection fifo  118 , the receiving node  104  sends a short get packet, which contains a put descriptor to the sender node and an injection Fifo id on the sender node. Without pacing, the DMA logic  124  at the sender node  102  puts this descriptor into the injection Fifo  118  specified in the short get message, and advances that Fifo&#39;s metadata tail pointer  132 . As described in more detail in the co-owned patent application entitled MULTIPLE NODE REMOTE MESSAGING (Attorney Docket YOR920070297US1 (21208), the payload of this remote get packet is a DMA descriptor that is deposited into an injection fifo  118  on node at  102 . This descriptor may be a “put” descriptor that contains information such as the starting address of the buffer to be sent on node at  102  and the starting address of the buffer on node  104  into which the data is to be stored, the injection and reception counter ids to use and the initial offsets from the base addresses of the counter ids. For a long message; a single remote get results in single put message, or a single memory Fifo message, which in turn injects a large number of packets into the network with no flow control. Note that a single remote get, as specified in a message descriptor, results in a single remote get packet being injected into the network. 
         [0031]    The above-described operations may be modified to implement pacing to control the packet injection rate of long messages. Pacing is implemented in one embodiment using remote gets with a message pacing option. With pacing, a single remote get for a long message is broken up into many remote gets for shorter submessages. A limit may be set on the number of outstanding bytes that may be requested from the node. This may be represented by a counter maintained in the DMA. The counter herein is referred to as a token_byte counter as an example. Any other name may be given to such counter or an element that provides the similar functionality. The shorter remote gets can only be injected into the network if the current number of outstanding bytes is low enough. When a shorter remote get is injected into the network, the token_byte counter is decremented by the submessage size. For such pacing messages, the resulting put packets contain a packet pacing bit set to 1. The token_byte counter is initialized to an arbitrary positive number, prior to a DMA activity. 
         [0032]      FIG. 5  illustrates the steps taken by a DMA in one embodiment when it receives a put packet, that is, when the DMA that sent a remote get receives a message packet in response to that remote get operation. At  502 , when a put packet arrives at a node, if the pacing bit in the packet is 1, the token_byte counter is increased by the number of payload bytes in the packet at  504 . In addition, as is done normally with a put packet, the payload of the packet is written to the correct memory location as specified by the reception counter id and the packet&#39;s put offset. That is, the data is written starting at the base address specified in the counter plus the put offset. The reception byte counter value is decremented by the number of payload bytes in the packet. 
         [0033]    Prior to any DMA activity, the token_byte counter is initialized to an arbitrary positive number, for example, token_byte max. In addition, a maximum pacing submessage size, max_size, is initialized to an arbitrary positive number less than or equal to the token_byte counter&#39;s initial value, token_byte_max. 
         [0034]      FIG. 6  illustrates the DMA logic for injecting messages from a DMA injection FIFO into the network in one embodiment. At  602 , the DMA waits until an injection Fifo is non-empty, that is, detects that there is a message to send. At  604 , the DMA starts processing the message descriptor at the beginning of the Fifo. If the descriptor message pacing bit is 0, or if the message type is not a remote get, the packets in the message are injected into the network at  630 . After all the packets of such a message are injected into the network, the injection Fifo head pointer is updated to point to the next descriptor at  632 , if any, and the logic returns to step  602 . Otherwise, at  604 , the logic proceeds to step  606  for a remote get with message descriptor pacing bit=1. At  606 , the payload of the remote get is fetched from memory. This payload is a direct put descriptor ( FIG. 4B ) that for example specifies a network packet header, a message length, initial injection and reception offsets, and injection and reception counter ids. The packet header should specify that the put is being sent back to this node. At  608 , the remaining bytes parameter, R, in the message is set to the message length just fetched, that is, the value of the message size field ( FIG. 4B ,  416 ) in the direct put descriptor. At  610 , parameter O_inj is set to the initial injection offset specified in the payload and at  612 , parameter O_rec is set to the initial reception offset specified in the payload. These values are stored in the DMA in one embodiment. O_inj is the injection offset for the put of the submessage triggered by the remote get and O_rec is the reception offset for the put of the submessage triggered by the remote get. At  614 , the submessage size S is set to the minimum of max_size and R. The parameter “max_size” represents maximum length the submessage. At  616 , the DMA waits until S&lt;=token_byte counter. This indicates that the number of outstanding bytes subject to pacing in the network is less than token_byte max, that is, maximum number of bytes allowed at one time. Eventually, this condition will be satisfied because token_byte counter gets incremented when pacing put packets return to this node. When the condition at  616  is satisfied a remote get packet is injected into the network specifying a submessage size of S, injection and reception offsets O_inj and O_rec at  618 . Otherwise the payload of the remote get is unchanged from its initial value. This remote get will cause a put message of length S to be sent back to this node at the correct initial offsets. At  620 , the token_byte counter is decremented by S. At  622 , the remaining bytes for the message is decremented by S (R=R−S). This value of R is stored in the DMA. At  624 , the injection and reception offsets are incremented by S (O_inj=O_inj+S and O_rec=O_rec+S). At  626 , if R=0, then the remote get for all submessages have been issued, so the injection Fifo head pointer is updated to point to the next descriptor at  628 , if any, and the logic returns to step  602 . Otherwise, the logic proceeds to step  614  to process the remaining bytes in the original remote get message. 
         [0035]    In one embodiment, the number of outstanding bytes of pacing put messages in the network is less than or equal to token_byte_max at all times. Since long messages typically have a fixed packet length equal to the maximum packet size, this limits the number of pacing put packets in the network. However, the number of messages (packets) not subject to pacing is not limited in this way and could grow until all the buffers in the network are full, or nearly full. In one embodiment, software in an arbitrary manner may decide which messages, if any, are subject to pacing. This may be determined by experimentation. 
         [0036]    Different implementations for hardware pacing are possible. For example, there may be multiple injection Fifos. The description above would then be modified in such a way that the DMA switches between non-empty injection Fifos. In particular, on such a switch, the method is applied to each injection Fifo, however, maintaining the condition that a remote get for a pacing submessage cannot be issued until the check in step  616  is satisfied. 
         [0037]    In another embodiment, there may be multiple token_byte counters. Suppose there are k such counters token_byte_counter(1), . . . , token_byte_counter(k) and multiple max submessage sizes max_size(1), . . . , max_size(k). In one aspect of this embodiment, there may be a control register in the DMA specifying for each injection Fifo, which token_byte_counter(i) should be used for all remote gets using that Fifo. All max_sizes, that is 1 to k may be initialized to a positive value and the max_size(i) is less than the corresponding initial value of token_byte_counter(i). In another aspect of this embodiment, the remote get message descriptor may include the index id i specifying token_byte count(i) and max_size(i) to be used. In both aspects of the embodiment, step  614  becomes S min (max_size(i), R) and step  316  becomes wait until S&lt;=token_byte_count(i). In addition, the resulting put packets may also specify the index i so that the correct token_byte counter is incremented when the put packets return to the node. 
         [0038]    Yet in another embodiment, it may be left up to the software to divide the remote get into sub remote gets and assemble the sub remote get packets. In this embodiment, hardware paces the sub remote get packets by monitoring the token_byte counter and sends a sub remote get when the token_byte counter is greater than equal to message length specified in the put descriptor of the sub remote get. The hardware increments the token_byte counter when a put packet is received and decrements the token_byte counter when a sub remote get packet is sent. 
         [0039]    The embodiments described above are illustrative examples and it should not be construed that the present invention is limited to these particular embodiments. For example, while some of the memory structure were shown and described in terms of fifo, any other queuing or structuring mechanism may be used. Thus, various changes and modifications may be effected by one skilled in the art without departing from the spirit or scope of the invention as defined in the appended claims.