Abstract:
In general, in one aspect, the disclosure describes a method that includes determining, at a first processor in a multi-processor system, that a network connection event is associated with a connection mapped to a second processor in the multi-processor system. In response, a network interface controller of the system is caused to signal an interrupt to the second processor.

Description:
REFERENCE TO RELATED APPLICATIONS  
       [0001]     This relates to U.S. patent application Ser. No. 10/815,895, entitled “ACCELERATED TCP (TRANSPORT CONTROL PROTOCOL) STACK PROCESSING”, filed on Mar. 31, 2004; this also relates to an application filed the same day as the present application entitled “DISTRIBUTING TIMERS ACROSS PROCESSORS” naming Sujoy Sen, Linden Cornett, Prafulla Deuskar, and David Mintum as inventors and having attorney/docket number 42390.P19610. 
     
    
     BACKGROUND  
       [0002]     Networks enable computers and other devices to communicate. For example, networks can carry data representing video, audio, e-mail, and so forth. Typically, data sent across a network is divided into smaller messages known as packets. By analogy, a packet is much like an envelope you drop in a mailbox. A packet typically includes “payload” and a “header”. The packet&#39;s “payload” is analogous to the letter inside the envelope. The packet&#39;s “header” is much like the information written on the envelope itself. The header can include information to help network devices handle the packet appropriately.  
         [0003]     A number of network protocols cooperate to handle the complexity of network communication. For example, a transport protocol known as Transmission Control Protocol (TCP) provides “connection” services that enable remote applications to communicate. TCP provides applications with simple commands for establishing a connection and transferring data across a network. Behind the scenes, TCP transparently handles a variety of communication issues such as data retransmission, adapting to network traffic congestion, and so forth.  
         [0004]     To provide these services, TCP operates on packets known as segments. Generally, a TCP segment travels across a network within (“encapsulated” by) a larger packet such as an Internet Protocol (IP) datagram. Frequently, an IP datagram is further encapsulated by an even larger packet such as an Ethernet frame. The payload of a TCP segment carries a portion of a stream of data sent across a network by an application. A receiver can restore the original stream of data by reassembling the received segments. To permit reassembly and acknowledgment (ACK) of received data back to the sender, TCP associates a sequence number with each payload byte.  
         [0005]     Many computer systems and other devices feature host processors (e.g., general purpose Central Processing Units (CPUs)) that handle a wide variety of computing tasks. Often these tasks include handling network traffic such as TCP/IP connections. The increases in network traffic and connection speeds have placed growing demands on host processor resources. To at least partially alleviate this burden, some have developed TCP Off-load Engines (TOEs) dedicated to off-loading TCP protocol operations from the host processor(s). 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0006]      FIGS. 1A-1E  are diagrams that illustrate use of a network interface controller interrupt to provide cross-processor signaling of a connection event.  
         [0007]      FIGS. 2 and 3  are flow-charts of processes that use a network interface controller interrupt to provide cross-processor signaling of a connection event.  
     
    
     DETAILED DESCRIPTION  
       [0008]     As described above, network connections and traffic have increased greatly in recent years. Processor speeds have also increased, partially absorbing the increased burden of packet processing operations. Unfortunately, the speed of memory has generally failed to keep pace. Each memory operation performed during packet processing represents a potential delay as a processor waits for the memory operation to complete. For example, in Transmission Control Protocol (TCP), the state of each connection is stored in a block of data known as a TCP control block (TCB). Many TCP operations require access to a connection&#39;s TCB. Frequent memory accesses to retrieve TCBs can substantially degrade system performance.  
         [0009]     To speed memory operations, many processors include caches that provide faster access to data than memory. Often, the cache and memory form a hierarchy where the cache is searched for requested data. In some caching schemes, if the cache does not store requested data (a cache “miss”), the data is loaded into the cache from memory for future use. To the extent that a connection&#39;s TCB remains cached, operations for a connection can avoid the delay associated with memory transactions.  
         [0010]     To increase the likelihood that a connection&#39;s TCB (and other connection related information) will remain cached,  FIG. 1A  depicts a multi-processor  102   a - 102   n  system that maps different connections to different processors  102   a - 102   n . As shown, the system includes multiple processors  102   a - 102   n , memory  106 , and one or more network interface controllers  100  (NICs). The NIC  100  includes circuitry that transforms the physical signals of a transmission medium into a packet, and vice versa. The NIC  100  circuitry also performs de-encapsulation, for example, to extract a TCP/IP packet from within an Ethernet frame.  
         [0011]     The processors  102   a - 102   b , memory  106 , and network interface controller(s) are interconnected by a chipset  121  (shown as a line). The chipset  121  can include a variety of components such as a controller hub that couples the processors to I/O devices such as memory  106  and the network interface controller(s)  100 .  
         [0012]     The sample scheme shown does not include a TCP off-load engine. Instead, the system distributes different TCP operations to different components. While the NIC  100  and chipset  201  may perform some TCP operations (e.g., the NIC  100  may compute a segment checksum), most are handled by processor&#39;s  102   a - 102   n.    
         [0013]     As shown, different connections may be mapped to different processors  102   a - 102   n . For example, operations on packets belonging to connections (arbitrarily labeled) “a” to “g” may be handled by processor  102   a , while operations on packets belonging to connections “h” to “n” are handled by processor  102   b . This mapping may be explicit (e.g., a table) or implicit.  
         [0014]     To illustrate operation of the system,  FIG. 1B  shows a packet  114  received by the network interface controller  100 . The network interface controller  100  can determine which processor  102   a - 102   n  is mapped to the packet&#39;s  114  connection, for example, by hashing packet data (the packet&#39;s “tuple”) identifying the connection (e.g., a TCP/IP packet&#39;s Internet Protocol source and destination address and a TCP source and destination port). In the example shown, a hash of the packet&#39;s  114  tuple indicates that the packet belongs to a connection, “c”, mapped to processor  102   a.    
         [0015]     As shown, each processor  102   a - 102   n  has a corresponding receive queue  110   a - 110   n  (RxQ) that identifies received packets to be handled by the respective processor. While the queues  110   a - 110   n  may store the actual packet data, the queues  110   a - 110   n , generally, will instead store a packet descriptor that identifies where the packet is stored in memory  106 . A descriptor may also include other information (e.g., the hash results, identification of the mapped processor, and so forth). For example, as shown, the network interface controller  100  enqueued a descriptor for received packet  114  (e.g., using Direct Memory Access (DMA)) in the queue  110   a  corresponding to processor  102   a . The processors  102   a - 102   n  consume entries from their respective queues  110   a - 110   n  and perform operations for the corresponding packet(s) such as navigating the TCP state machine for a connection, performing segment reordering and reassembly, tracking acknowledged bytes in a connection, managing connection windows, and so for (see, for example, The Internet&#39;s Engineering Task Force (IETF), Request For Comments #793).  
         [0016]     As shown, to alert the processor  102   a  of the arrival of a packet, the network interface controller  100  can signal an interrupt. Potentially, the controller  100  may use interrupt moderation which delays an interrupt for some period of time. This increases the likelihood multiple packets will have arrived before the interrupt is signaled, enabling a processor to work on a batch of packets and reducing the overall number of interrupts generated.  
         [0017]     In response to the interrupt, the processor  102   a  may dequeue and process the next entry (or entries) in its receive queue  110   a . Since the processor  102   a  only processes packets for a limited subset of connections, the likelihood that the TCB for connection “c” remains in the processor&#39;s  102   a  cache  104   a  increases.  
         [0018]     FIG. B illustrated delivery of a received packet to the processor  102   a - 102   n  mapped to the packet&#39;s connection. However, some connection-related events may originate or be received by the “wrong” processor (i.e., a processor other than the processor mapped to the connection). For example, though processor  102   a  is mapped to process packets in connection “c”, an application on processor  102   n  may initiate a transmit operation over connection “c”. Handling the event by the “wrong” processor, processor  102   n  in this case, can largely negate many of the advantages of the scheme shown in  FIG. 1B . For example, reading a connection&#39;s TCB into the “wrong” cache  104   n  may victimize a TCB of a connection mapped to the processor  102   n  from the cache  104   n . Additionally, loading a connection&#39;s TCB into the “wrong” cache  104   n  may both necessitate invalidation of the “right” cache&#39;s TCB entry  104   a  and may require a locking scheme to maintain data consistency across different processors accessing the same TCB.  
         [0019]      FIGS. 1C-1E  illustrate a scheme that transfers handling of events to the “right” processor  102   a - 102   n . To notify the “right” processor, the “wrong” processor schedules an interrupt on the network interface controller  100 . The “wrong” processor  102   n  also writes data that enables processors  102   a - 102   n  receiving the interrupt to identify its cause. For example, processor  102   n  can set a software interrupt flag in an interrupt cause register maintained by the network interface controller  100 . In response to the interrupt request, the network interface controller  100  interrupts the processors  102   a - 102   n  mapped connections. The network interface controller drivers operating on the processors  102   a - 102   n  respond to the interrupt by checking the data (e.g., flag(s)) indicating the interrupt cause. For example, the interrupt cause may indicate either a hardware interrupt (e.g., in response to one or more received packets) and/or a software generated interrupt (e.g., a transfer of event handling across processors). Based on the identified interrupt cause, the “right” processor can process the received packets and/or inter-processor event transfer.  
         [0020]     To illustrate, as shown in  FIG. 1C , processor  102   n  determines that an event  116  associated with connection “c” (e.g., a transmit operation, a connection timer, or connection start, reset, or termination) should be handled by processor  102   a . Such a determination may be made by accessing a table associating connections with processors and/or hashing the TCP/IP tuple associated with the packet&#39;s connection. As shown, processor  102   n  schedules an interrupt by network interface controller  100 .  
         [0021]     As shown in  FIG. 1D , in addition to scheduling the network interface controller  100  interrupt, processor  102   n  can also enqueue an entry for the event  116  in a processor-specific queue  112   a  and/or a connection-specific queue (not shown). The entry includes or references data (e.g., the connection, type of event, and so forth) used by the “right” processor  102  to respond to the event  116 .  
         [0022]     As shown in  FIG. 1E , the network interface controller  100  then generates the scheduled interrupt for each processor  102   a - 102   n  having a receive queue  110   a - 110   n . Alternately, the controller  100  can issue an interrupt targeted to a specific processor. After receiving an interrupt and determining that the interrupt signifies an event registered by a “wrong” processor  102   n  (e.g., by examining the interrupt cause register), the “right” processor  102   a  can retrieve the entry from the queue  112   a  and respond accordingly.  
         [0023]      FIG. 2  and  FIG. 3  illustrate processes implemented by the processors  102   a - 102   n . In  FIG. 2 , a processor  102   n  determines  152  if the connection associated with an event is mapped to a different processor  102   a . If so, the processor  102   n  can enqueue  154  an event entry and schedule  156  an interrupt to signal the event. As shown in  FIG. 3 , in response to the interrupt, the processor can determine  160  whether the interrupt was a response to an event initially handled by a different processor (e.g., by checking the interrupt cause register or other data associated with NIC  100 ). The processor can then dequeue  164  the events, if any  162 , and perform the appropriate operations  166 . This dequeueing  164  may be performed by accessing from a processor-specific queue (e.g.,  112 ) and/or by accessing different connection-specific queues of connections mapped to the processor.  
         [0024]     The scheme illustrated above can, potentially, increase the likelihood that connection specific data (e.g., the TCB) is cached in the same processor for the duration of a connection. The scheme also can eliminate or reduce the need for locks on connection-specific data. Additionally, by “piggybacking” on the network interface controller interrupt system, the scheme need not increase system complexity with an additional signaling system or burden the system with additional interrupts.  
         [0025]     Though the description above repeatedly referred to TCP as an example of a protocol that can use techniques described above, these techniques may be used with many other protocols such as protocols at different layers within the TCP/IP protocol stack and/or protocols in different protocol stacks (e.g., Asynchronous Transfer Mode (ATM)). Further, within a TCP/IP stack, the IP version can include IPv4 and/or IPv6.  
         [0026]     While  FIGS. 1A-1E  and  FIG. 4  depicted a typical multi-processor host system, a wide variety of other multi-processor architectures may be used. For example, while the systems illustrated did not feature TOEs, an implementation may nevertheless feature them.  
         [0027]     The techniques above may be implemented using a wide variety of circuitry. The term circuitry as used herein includes hardwired circuitry, digital circuitry, analog circuitry, programmable circuitry, and so forth. The programmable circuitry may operate on computer programs disposed on a computer readable medium.  
         [0028]     Other embodiments are within the scope of the following claims.