Abstract:
Some embodiments provide a system that detects whether a flow is an elephant flow; and if so, the system treats it differently than a mouse flow. The system of some embodiment detect elephants based on one or more of the following: statistics associated with a flow, packet segment size, and invoked system calls. Also, some embodiments use one or more various methods to handle elephant flows. Examples of such methods include marking each packet belonging to an elephant with a particular marking, breaking the elephants into mice, reporting the elephant to a network controller, and selectively choosing a route for each packet belonging to the elephant.

Description:
CLAIM OF BENEFIT TO PRIOR APPLICATIONS 
       [0001]    This application claims the benefit of U.S. Provisional Patent Application 61/913,899, entitled “Detecting and Handling Elephant Flows”, filed on Dec. 11, 2013. U.S. Provisional Patent Application 61/913,899 is incorporated herein by reference. 
     
    
     BACKGROUND 
       [0002]    Today, a datacenter may process different types of flows, including elephant flows and mouse flows. An elephant flow represents a long-lived flow or a continuous traffic flow that is typically associated with high volume connection. Different from an elephant flow, a mouse flow represents a short-lived flow. Mice are often associated with bursty, latency-sensitive applications, whereas elephants tend to be associated with large data transfers in which throughput is far more important than latency. 
         [0003]    A problem with elephant flows is that they tend to fill network buffers end-to-end, and this introduces non-trivial queuing delay to anything that shares these buffers. For instance, a forwarding element may be responsible for managing several queues to forward packets, and several packets belonging to a mouse flow may be stuck in the same queue behind a group of other packets belonging to an elephant flow. In a network of elephants and mice, this means that the more latency-sensitive mice are being affected. Another problem is that mice are generally very bursty, so adaptive routing techniques are not effective with them. 
       BRIEF SUMMARY 
       [0004]    Embodiments described herein provide a forwarding element that monitors network traffic to detect an elephant flow and reports the elephant flow to a network controller if it detects one. In some embodiments, the network controller is a software-defined networking (SDN) controller, which receives the report and configures, based on the report, one or more forwarding elements to handle the elephant flow. The network controller can configure the same forwarding element, and/or can configure one or more other forwarding elements that receive packets in the same elephant data flow. 
         [0005]    In some embodiments, the forwarding element that performs the monitoring and reporting is an edge forwarding element. The edge forwarding element represents a last forwarding element before one or more end machines. In some such embodiments, the edge forwarding element reports a detected elephant flow to a network controller, which in turn configures a non-edge forwarding element to handle the elephant flow. Alternatively, when a report is received from the edge forwarding element, the network controller may configure another edge forwarding element. In some embodiments, the forwarding element that performs the monitoring and reporting is a non-edge forwarding element. When the non-edge forwarding element reports an elephant flow, the network controller may configure another non-edge forwarding element or an edge forwarding element, in some embodiments. 
         [0006]    In some embodiments, the forwarding element, which performs the monitoring and reporting, is configured and managed by a first network controller (e.g., in a controller cluster) but reports to a second different network controller that configures and manages one or more other forwarding elements. For instance, the first network controller may manage a number of Layer 2 switches, while the second different network controller manages a number of Layer 3 switches. In some embodiments, the first network controller manages a number of software forwarding elements (e.g., software switches), while the second different network controller manages a number of hardware forwarding elements (e.g., Top-of-Rack (TOR) switches). 
         [0007]    The forwarding element of some embodiments monitors tunneled traffic to detect elephant flows. In some embodiments, the forwarding element is an edge forwarding element that operate in conjunction with another forwarding element to monitor and report any elephant flows, detected at either ends of the tunnel, to the network controller. As an example, in detecting elephant flows, a first forwarding element at one end of the tunnel may monitor outbound packets from a first network host to a second network host, and a second forwarding element at the other end of the tunnel may monitor outbound packets from the second network host to the first network host. When an elephant flow is detected at one end of the tunnel, the corresponding forwarding element then reports the flow to the network controller. 
         [0008]    As mentioned above, when an elephant flow is detected, the forwarding element of some embodiments identifies one or more pieces of information that can be used to identify packets belonging to an elephant flow. The forwarding element may identify tunnel information, such as the tunnel ID, the IP address of the source tunnel endpoint (e.g., the hypervisor), and the IP address of the destination tunnel endpoint. The forwarding element of some embodiments identifies the elephant flow packet&#39;s ingress port, source transport layer (e.g., UDP or TCP) port, destination transport layer port, Ethernet type, source Ethernet address, destination Ethernet address, source IP address, and/or destination IP address. 
         [0009]    In some embodiments, the network controller is registered with the forwarding element to receive an update report each time the forwarding element detects an elephant. For instance, when an elephant flow is detected, the forwarding element may store one or more pieces of information relating to the elephant flow in a storage (e.g., a database) through a database server. The network controller may be registered with the forwarding element (e.g., the forwarding element&#39;s database server) to receive an update if there are any changes to the data relating to elephant flows in the storage. Hence, if there is an update, the forwarding element&#39;s database server may send a notification (e.g., an asynchronous notification) to the network controller regarding the change to the elephant flow data stored in the database. The notification may include one or more pieces of information mentioned above. 
         [0010]    When a report regarding an elephant flow is received, the network controller of some embodiments configures one or more forwarding elements. Different embodiments configure the network controller differently. As an example, the network controller perform a Quality of Service (QOS) configuration on the forwarding element to place packets belonging to the elephant flow in a particular queue that is separate from one or more other queues with other packets, break the elephant flow into mice flows, etc. Alternatively, the network controller may configure the forwarding element to break the elephant flow into mice flows by sending packets associated with the elephant flow along different paths (e.g., equal-cost multipath routing (ECMP) legs). As another example, the forwarding element may be configured to send elephant flow traffic along a separate physical network, such as an optical network that is more suitable for slow changing, bandwidth-intensive traffic. 
         [0011]    Additional techniques for detecting and handling elephant flows are described in U.S. patent application **, entitled “Detecting and Handling Elephant Flows”, filed concurrently with this application, and having Attorney Docket No. N071.01. Furthermore, several embodiments that detect an elephant flows based on the size of a packet are described in U.S. patent application **, entitled “Detecting an Elephant Flow Based on the Size of a Packet”, filed concurrently with this application, and having Attorney Docket No. N071.02. These U.S patent applications are incorporated herein by reference. In addition, some embodiments provide a system that detects an elephant flow by examining the operations of a machine. In some embodiments, the machine is a physical machine or a virtual machine (VM). In detecting, the system identifies an initiation of a new data flow associated with the machine. The new data flow can be an outbound data flow or an inbound data flow. The system then determines, based on the amount of data being sent or received, if the data flow is an elephant flow. 
         [0012]    The preceding Summary is intended to serve as a brief introduction to some embodiments as described herein. It is not meant to be an introduction or overview of all subject matter disclosed in this document. The Detailed Description that follows and the Drawings that are referred to in the Detailed Description will further describe the embodiments described in the Summary as well as other embodiments. Accordingly, to understand all the embodiments described by this document, a full review of the Summary, Detailed Description and the Drawings is needed. Moreover, the claimed subject matters are not to be limited by the illustrative details in the Summary, Detailed Description and the Drawings, but rather are to be defined by the appended claims, because the claimed subject matters can be embodied in other specific forms without departing from the spirit of the subject matters. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0013]    The novel features of the invention are set forth in the appended claims. However, for purposes of explanation, several embodiments of the invention are set forth in the following figures. 
           [0014]      FIG. 1  illustrates an example of a forwarding element that reports an elephant flow to a network controller. 
           [0015]      FIG. 2  illustrates an example of an edge forwarding element that reports an elephant flow to a network controller, which in turn configures one or more non-edge forwarding elements. 
           [0016]      FIG. 3  conceptually illustrates a process  300  that some embodiments implement to report an elephant flow to a network controller 
           [0017]      FIG. 4  conceptually shows an example of how two forwarding elements monitors tunneled traffic between several network hosts to detect elephant flows and report them to a network controller. 
           [0018]      FIG. 5  illustrates an example of how the two edge forwarding elements implement an elephant agent to monitor tunneled traffic. 
           [0019]      FIG. 6  illustrates example operations performed by a forwarding element to report an elephant flow. 
           [0020]      FIG. 7  conceptually illustrates an example data structure to store and report data relating to each detected elephant flow. 
           [0021]      FIG. 8  provides an illustrative example of a forwarding element that performing quality of service operations on packets in a detected elephant data flow. 
           [0022]      FIG. 9  conceptually illustrates a process that some embodiments implement to perform quality of service operations on packets associated a detected elephant flow. 
           [0023]      FIG. 10  provides an illustrative example of a forwarding element that performing breaks an elephant flow into a bunch of mouse flows. 
           [0024]      FIG. 11  conceptually illustrates a process that some embodiments implement to break an elephant into mice. 
           [0025]      FIG. 12  conceptually illustrates an electronic system with which some embodiments of the invention are implemented. 
       
    
    
     DETAILED DESCRIPTION 
       [0026]    In the following detailed description of the invention, numerous details, examples, and embodiments of the invention are set forth and described. However, it will be clear and apparent to one skilled in the art that the invention is not limited to the embodiments set forth and that the invention may be practiced without some of the specific details and examples discussed. 
         [0027]    Some embodiments provide a forwarding element that monitors network traffic to detect an elephant flow and reports the elephant flow to a network controller if it detects one. In some embodiments, the network controller is a software-defined networking (SDN) controller, which receives the report and configures, based on the report, one or more forwarding elements to handle the elephant flow. The network controller can configure the same forwarding element, and/or can configure one or more other forwarding elements that receive packets in the same elephant data flow. 
         [0028]    For some embodiments of the invention,  FIG. 1  conceptually illustrates an example of a forwarding element  105  that (1) monitors network traffic to detect an elephant flow and (2) reports the elephant flow to a network controller  135  if it detects one. The figure shows a network  100 , which includes the forwarding element  105 , the network controller  135 , and several other forwarding elements  110 - 120 . Four stages  140 - 155  of the network  100  are shown in the figure. Each of these stages will be described in detail below after an introduction of the elements of the network. 
         [0029]    In the example of  FIG. 1 , each of the forwarding elements  105 - 120  can be a hardware forwarding element or a software forwarding element. The hardware forwarding element can have application-specific integrated circuits (ASICs) that are specifically designed to support in-hardware forwarding. Different from a hardware forwarding element, the software forwarding element operates on a host machine (e.g., an x86 box). The forwarding elements  105 - 120  process packets in the network. For example, each forwarding element can forward packets for different machines (e.g., the machines  125  and  130 ), drop packets, etc. 
         [0030]    The term “packet” is used here as well as throughout this application to refer to a collection of bits in a particular format sent across a network. One of ordinary skill in the art will recognize that the term “packet” may be used herein to refer to various formatted collections of bits that may be sent across a network, such as Ethernet frames, TCP segments, UDP datagrams, IP packets, etc. 
         [0031]    The forwarding elements ( 105  and  110 ) represent edge forwarding elements (EFEs), while the remaining forwarding elements ( 115  and  120 ) are non-edge forwarding elements (NEFE). The forwarding elements ( 105  and  110 ) are EFEs because they are located at or near the edge of the network  100 . In some embodiments, an EFE is the last forwarding element before one or more end machines (e.g., virtual machines (VMs), computing devices). 
         [0032]    In some embodiments, the forwarding element  105  is configured to detect elephant flows and report each detected elephant flow to the network controller  135 . The forwarding element may use one of several different methods to detect elephant flows. As an example, the forwarding element  105  might track statistics (e.g., packet count, byte count, bytes per second) associated with different data flows (e.g., flow entries). Alternatively, the forwarding element  105  may examine the sizes of different packets to detect elephant flows. The forwarding element  105  inspects the size because, in order for the packet to be of a certain size, the data flow had to already have gone through a slow start (e.g., TCP slow start) in which smaller packets are transferred and by definition be an elephant flow. 
         [0033]    In detecting elephant flows, the forwarding element  105  of some embodiments monitors tunneled traffic from one network host (e.g., the machine  125 ) to another network host (e.g., the machine  130 ). For example, the forwarding element  105  may operate in conjunction with another forwarding element, such as the forwarding element  110 , to monitor and report any elephant flows detected at either ends of the tunnel to the network controller. 
         [0034]    The network controller  135  manages and configures one or more forwarding element. In the example of  FIG. 1 , the network controller  135  manages and configures two EFEs ( 105  and  110 ). The NEFEs ( 115  and  120 ) may be managed and configured by another network controller. Also, although the figure only shows one network controller, there can be a cluster of network controllers that (e.g., collectively) configures and manages the two EFEs ( 105  and  110 ), as well as other forwarding elements. 
         [0035]    In some embodiments, the network controller  135  provides a network administrator with a unified control point in a network to simplify management, provisioning, and configuring several forwarding elements. The network controller of some embodiments allows the network administrator to perform a number of different tasks, such as defining tunnels, virtual networks, etc. The network controller cluster may push flows to the forwarding elements or push instructions that are translated to flows at the forwarding elements. The implementation of the network controller can change depending on one or more different types of forwarding elements that it supports, such as software fording element, hardware forwarding element, Layer 2 forwarding element, Layer 3 forwarding element, etc. 
         [0036]    In some embodiments, the network controller  135  receives a report regarding a detected elephant flow and configures one or more forwarding elements. The network controller  135  can configure the same forwarding element, and/or can configure one or more other forwarding elements that receive packets in the same elephant data flow. In the examples of  FIG. 1 , the network controller configures the two EFEs ( 105  and  110 ). However, the network controller may configure one or more of the two NEFEs ( 115  and  120 ). 
         [0037]    Different embodiments of the network controller configure a forwarding element differently. As an example, the network controller  135  may perform a Quality of Service (QOS) configuration on the forwarding element to place packets belonging to the elephant flow in a particular queue that is separate from one or more other queues with other packets. Alternatively, the network controller may configure the forwarding element to break the elephant flow into mice flows by sending packets associated with the elephant flow along different paths (e.g., equal-cost multipath routing (ECMP) legs). As another example, the forwarding element may be configured to send elephant flow traffic along a separate physical network, such as an optical network that is more suitable for slow changing, bandwidth-intensive traffic. 
         [0038]    Having described several elements, the operations of the network in detecting an reporting the elephant flow, and configuring one or more forwarding elements to handle the elephant flow will now be described by reference to the three stages  140 - 155  that are illustrated in  FIG. 1 . 
         [0039]    The first stage  145  shows the start of a new data flow in the network  100 . The data flow originates at the machine  125 , and terminates at the machine  130 . The data flows through the network  100  including, the forwarding elements  105 - 120 . The data also flows through a core network  160 . Here, the core network  160  represents the physical core of the network  100 . In the physical core, there can be a number of high capacity forwarding element that forward packets. Also, the data may flow through a wide area network (WAN) or the Internet. 
         [0040]    The second stage  140  shows the EFE  105  detecting an elephant flow. Specifically, the EFE  105  has monitored the flow of data from the machine  125  to the machine  130 , and determined that the data flow is an elephant flow. As mentioned above, the EFE 105  of some embodiments detects an elephant flow by tracking statistics (e.g., packet count, byte count, bytes per second) associated with the data flow. Alternatively, the EFE  105  might have examined the sizes of different packets associated with the data flow in order to detect the elephant flow. 
         [0041]    In the second stage  145 , the EFE  105  has detected an elephant flow. The third stage  150  shows that, in response to the detection, the EFE  105  reports the elephant flow to the network controller  135 . Here, in the fourth stage  155 , the network controller  135  responds to the report regarding the elephant flow from the EFE  105 . Specifically, the network controller  135  configures the two EFEs ( 105  and  110 ) to handle the elephant flow. As mentioned above, the network controller  425  may configure a forwarding element to place elephant flows into different queues from mice, choose different equal-cost multi-path routing (ECMP) paths for the elephant flows, route elephant flows along a separate physical network, etc. 
         [0042]    In the example described above, the network controller  135  configures one or more EFEs ( 105  and  110 ) to handle the elephant flow.  FIG. 2  illustrates an example of another network controller  225  that configures two NEFEs ( 115  and  120 ) when it receives a report from the EFE  105 . In some embodiments, the EFE  105  is configured and managed by the network controller  135  (e.g., shown in the previous figure) but reports to the second different network controller  225  that configures and manages one or more non-edge forwarding elements. For instance, the first network controller  135  may manage a number of software forwarding elements, while the second network controller  255  may manage a number of hardware forwarding elements. 
         [0043]    Four stages  205 - 220  of a network  200  are shown in  FIG. 2 . The first three stages  205 - 215  are identical to the previous figure. Specially, in the first stage  205 , a new data flow has been started with the machine  125  sending data to the machine  130 . In the second stage  210 , the EFE  105  has detected an elephant flow. The third stage  215  shows that, in response to the detection, the EFE  105  reports the elephant flow to the network controller  225 . However, the fourth stage  220  shows that the network controller  225  responding to the report by configuring one or more of the NEFEs ( 115  and  120 ). 
         [0044]    Having described an example of reporting an elephant flow, an example process  3  will now be described.  FIG. 3  conceptually illustrates a process  300  that some embodiments implement to report an elephant flow to a network controller. In some embodiments, the process  300  is performed by a forwarding element (e.g., a switch, a router). As shown, the process  300  begins when it monitors (at  305 ) network traffic to detect an elephant flow. In some embodiments, the process  300  examines statistics relating different flow entries. Alternatively, or conjunctively, the process  300  might detect an elephant flow by examining the size of a packet. 
         [0045]    At  310 , the process  300  determines whether an elephant flow has been detected. If an elephant flow has not been detected, the process  300  proceeds to  320 , which is described below. If an elephant flow has been detected, the process  300  reports (at  315 ) the elephant flow to a network controller. The process  300  then proceeds to  320 . 
         [0046]    The process  300  determines (at  320 ) whether to continue monitoring network traffic. Here, the forwarding element may be set to disable elephant flow detection, or the forwarding element may be reset or turned off. If the determination is made to continue monitoring, the process  300  returns to  305 , which is described above. Otherwise, the process  300  ends. Some embodiments perform variations on the process  300 . The specific operations of the process  300  may not be performed in the exact order shown and described. The specific operations may not be performed in one continuous series of operations, and different specific operations may be performed in different embodiments. 
         [0047]    Several more examples of detection and handling elephant flows will be described in detail below. Section I describes examples of how some embodiments reports elephant flows to a network controller. Specifically, Section I.A describes examples of several forwarding elements that monitors tunneled traffic between two network hosts. Section I.B describes examples of how some embodiments report elephant flows to a network controller. Section I.C then describes an example of a data structure to store and report data relating to an elephant flow. Section II then describes several different examples of how a forwarding element can be configured by a network controller to handle an elephant flow. Section III then describes an example electronic system with which some embodiments of the invention are implemented. 
       I. Example Implementation 
       [0048]    The forwarding element of some embodiments monitors tunneled traffic to detect elephant flows. In some embodiments, the forwarding element is an edge forwarding element that operate in conjunction with another forwarding element to monitor and report any elephant flows, detected at either ends of the tunnel, to the network controller. As an example, in detecting elephant flows, a first forwarding element at one end of the tunnel may monitor outbound packets from a first network host to a second network host and a second forwarding element at the other end of the tunnel may monitor outbound packets from the second network host to the first network host. When an elephant flow is detected at one end of the tunnel, the corresponding forwarding element then reports the flow to the network controller. 
         [0049]    A. Monitoring Tunneled Traffic 
         [0050]    In some embodiments, the forwarding elements communicate with a first set of network controllers to establish a tunnel, and reports elephant flows to a second set of network controllers.  FIG. 4  conceptually shows an example of how the EFEs  105  and  110  establish a tunnel through one network controller  445 , and report elephant flows to another network controller  425 . Four stages  405 - 420  of the network  400  are shown in the figure. 
         [0051]      FIG. 4  shows some of the same forwarding elements  105 ,  110 ,  115 , and  120  as shown in  FIGS. 1 and 2 . However, in  FIG. 4 , the EFEs  105  and  110  are software forwarding elements (e.g., Open Virtual Switches (OVSs)), and the NEFE  115  and  120  are top-of-rack (TOR) switches. The EFEs  105  and  110  are also managed EFEs (MEFEs) because they are managed by the network controller  445 . The NEFE  115  and  120  are also MNEFEs because they are managed by the network controller  425 . The network controller  445  and the EFEs  105  and  110  may be provided by one vendor (e.g., a company), while the network controller and the NEFEs  115  and  120  may be provided by another vendor. In such a case, the EFE  105  and  110  reports to another vendor&#39;s network controller to handle the detected elephant flows. 
         [0052]    The first stage  405  of  FIG. 4  shows the network controller configuring the EFEs  105  and  110  to establish a tunnel between machines  125  and  130 . Tunneling is encapsulating a data packet that includes a header of a first communication protocol with a header of a second communication protocol in order to transmit the data packet over a delivery network that implements the second communication protocol. A tunnel is deemed established when two network nodes of the delivery network are set up to deliver packets from a first of the two network nodes to a second of the two network nodes using the encapsulating protocol. The network nodes of the delivery network may include switches, routers, workstations, servers, virtual machines (VMs), or any devices that can transmit and/or receive data packets. 
         [0053]    In the first stage  405 , the network controller  445  sends data to establish a tunnel between the machines  125  and  130 . The second stage  410  conceptually shows the  440  the established tunnel. In some embodiments, the tunnel is established by providing various pieces of information to each of the EFEs  105  and  110 . Examples of such information include IP addresses of the machines  125  and  130 , the IP addresses of the hypervisors on which the software forwarding elements (e.g., OVSs) executes, the tunneling protocol to use, etc. 
         [0054]    The second stage  410  shows the two EFEs  105  and  110  implementing an elephant agent  435  to report any elephant flow in tunneled traffic between the machines  125  and  130 . In some embodiments, each EFEs ( 105  or  110 ) includes an elephant agent that monitors traffic on one end to detect and report elephant flows. For instance, the EFE  105  may be associated with one elephant agent module that monitor traffic destined for the machine  130  from the machine  125 , while the EFE  110  may be associated with another elephant agent module that monitor traffic destined for the machine  125  from the machine  130 . 
         [0055]    The third stage  415  shows an elephant flow being reported to the network controller  425 . In particular, when an elephant flow is detected, the elephant agent is notified of the flow. In some embodiments, the elephant agent then asynchronously notifies a network controller of the flow. In the example of  FIG. 4 , the elephant agent  435  is used to report to the elephant flow to the network controller  425  (e.g., from a different vendor). However, the elephant agent can report to the network controller  445 , in some embodiments. 
         [0056]    The fourth stage  420  shows the network controller  425  responding to the elephant flow report. Specifically, the network controller  425  configures the NEFE  115  to handle the elephant flow. The network controller  425  may configure the NEFE  115  to place elephant flows into different queues from mice, choose different equal-cost multi-path routing (ECMP) paths for the elephant flows, and/or route elephant flows along a separate physical network. 
         [0057]    In the example described above, the elephant agent  435  reports a detected elephant flow to the network controller  445 .  FIG. 5  illustrates an example of how the EFEs  105  and  110  implement such the elephant agent  435  to monitor tunneled traffic. In this example, the machine  125  is a virtual machine (VM) that operates on a hypervisor  505 . The hypervisor  505  also includes the EFE  105 . Similarly, the machine  130  is a virtual machine (VM) that operates on a hypervisor  510 . The hypervisor  510  includes the EFE  110 . 
         [0058]    In the example of  FIG. 5 , the network controller  445  configures the EFEs to establish the tunnel  440  between the VMs  125  and  130 . The tunnel is configured using the IP addresses associated with the VMs  125  and  130  and hypervisors  505  and  510 . Specifically, the VMs are using addresses from the 192.168.0.x private address space. The hypervisors are using addresses in the 17.0.0.x address space. The two VMs may also be assigned a same tunnel key or logical forwarding element identifier. The logical forwarding element identifier is used to implement a logical forwarding element from at least the EFEs  105  and  110  (e.g., the software forwarding elements). 
         [0059]    The tunnel  440  is shown as a Virtual Extensible LAN (VXLAN) tunnel because the two EFEs  105  and  110  are using the VXLAN protocol. However, the tunnel can be established using a different protocol, such as Stateless Transport Tunneling Protocol (STT), Generic Routing Encapsulation (GRE), etc. In some embodiments, the tunnel is a shared tunnel. There can be multiple connections connection. For instance, even though the tunnel may have been configured as a single tunnel, the tunnel may be shared by multiple machines (e.g., associated with different tenants). 
         [0060]    As mentioned above, when an elephant is detected, the forwarding element identifies various pieces of information that can be used to identify packets belonging to the elephant flow. The forwarding element may identify tunnel information, such as the tunnel ID, the IP address of the source tunnel endpoint (e.g., the hypervisor), and the IP address of the destination tunnel endpoint. The forwarding element of some embodiments identifies the elephant flow packet&#39;s ingress port, source transport layer (e.g., UDP or TCP) port, destination transport layer port, Ethernet type, source Ethernet address, destination Ethernet address, source IP address, and/or destination IP address. 
         [0061]    In some embodiments, the forwarding element identifies header values relating to the outer packet. The main reason for this is that these header values of the outer packet are what a subsequent forwarding element may look at. That is, the subsequent forwarding element may not even look at or perform a look up on the inner packet headers. When forwarding traffic, the subsequent forwarding element may identify the five tuples (e.g., that make up a TCP/IP connection) of the outer packet, such as source IP address, destination IP address, source port number, destination port number, and the protocol in use. As such, the forwarding element of some embodiments report such set of tuples to the network controller. 
         [0062]    In the example of  FIG. 5 , the EFEs  105  and  110  has established the tunnel  440  between the VMs  125  and  130 . Each of the EFE  105  or  110  can monitor the tunneled traffic to detect elephant flow. As mentioned above, the EFE is in a unique position to identify elephant flows. The EFE has the advance over a NEFE in that it is the last forwarding element or the tunnel endpoint before one or more end machines (e.g., VMs, computing device). Thus, the EFE can more easily monitor tunneled traffic coming from and going to an end machine than a NEFE. The EFE of some embodiments also has the advantage over the NEFE because the NEFE may not be able to detect an elephant flow based on the size of a packet. For instance, depending on how the network is configured, a NEFE may never receive large TCP packets but only receive smaller MTU-sized packets. 
         [0063]    As shown in  FIG. 5 , the EFEs  105  and  110  implement the elephant agent  435 . To implement the elephant agent  435 , each of the EFEs  105  or  110  includes an elephant agent module  515  or  520  that reports a detected elephant flow to the network controller  110 . The elephant agent module  515  reports any detected elephant flows at one end of the tunnel  440 , while the elephant agent module  520  reports any detected elephant flow at the other end of the tunnel. 
         [0064]    B. Example Mechanism for Reporting Elephant Flows 
         [0065]    In some embodiments, the network controller is registered with the forwarding element to receive an update report each time the forwarding element detects an elephant. For instance, when an elephant flow is detected, the forwarding element may identify one or more pieces of information relating to the elephant flow, and store information in a database through a database server. The network controller may be also registered with the database server to receive an update if there are any changes to the data relating to elephant flows, which are stored in the database. Hence, if there is an update, the forwarding element&#39;s database server may send a notification (e.g., an asynchronous notification) to the network controller regarding the change to the elephant flow data stored in the database. The notification may include one or more pieces of information mentioned above. 
         [0066]    In several of the example described above, a forwarding element reports any detected elephant flow to a network controller.  FIG. 6  illustrates example operations performed by a forwarding element to report such an elephant flow. 
         [0067]    Four operational stages  605 - 620  of the EFE  105  are shown in this figure. In this example, the EFE  105  includes an elephant detector  635  that detects elephant flows. The first stage  605  shows the elephant detector  625  operating on the EFE to detect an elephant flow. In the second stage  610 , the elephant detector  625  has detected an elephant flow and stores data relating to the flow in storage  630 . In some embodiments, when an elephant is detected, the system identifies various pieces of information that can be used to identify packets belonging to the elephant flow. The system may identify tunnel information, such as the tunnel ID, the IP address of the source tunnel endpoint (e.g., the hypervisor), and the IP address of the destination tunnel endpoint. The system of some embodiments identifies the elephant flow packet&#39;s ingress port, source transport layer (e.g., UDP or TCP) port, destination transport layer port, Ethernet type, source Ethernet address, destination Ethernet address, source IP address, and/or destination IP address. 
         [0068]    The third stage  615  shows that the elephant agent module  515  is notified of the changes to the data relating to elephant flows. The elephant agent module then sends the report regarding the elephant flow to the network controller  425 . In some embodiments, the storage  630  is maintained by a database server (not show) that supports bi-directional asynchronous notifications. For example, when there is an update to a database table with the elephant flow, the database server sends a notification regarding an update to the elephant agent module. The notification may include a copy of the table or a subset of the table (e.g., a record) that was updated. 
         [0069]    C. Example Data Structure 
         [0070]    As mentioned above, when an elephant flow is detected, the forwarding element of some embodiments identifies one or more pieces of information that can be used to identify packets belonging to an elephant flow.  FIG. 7  conceptually illustrates an example data structure  700  to store and/or report data relating to each detected elephant flow. As shown, the data structure includes an elephant flow identifier (ID) to (e.g., uniquely) identify the detected elephant flow and a tunnel ID to identify a tunnel with the elephant flow traffic, the IP address of the source tunnel endpoint (e.g., the hypervisor), the IP address of the destination tunnel endpoint, the packet&#39;s ingress port, source Ethernet address, destination Ethernet address, source IP address, destination IP address, Ethernet type, source transport layer (e.g., UDP or TCP) port, destination transport layer port, etc. 
         [0071]    One of ordinary skill in the art would understand that the data structure  700  is an example data structure, and the different embodiments may define one or more different data structures to identify an elephant flow and report the elephant flow to an agent that is interested in the report. For instance, the forwarding element of some embodiments may identify additional data for a detected elephant flow, or even fewer data. 
       II. Example Elephant Flow Handlers 
       [0072]    When a report regarding an elephant flow is received, the network controller of some embodiments configures one or more forwarding elements. Different embodiments configure the network controller differently. As an example, the network controller perform a Quality of Service (QOS) configuration on the forwarding element to place packets belonging to the elephant flow in a particular queue that is separate from one or more other queues with other packets, break the elephant flow into mice flows, etc. Alternatively, the network controller may configure the forwarding element to break the elephant flow into mice flows by sending packets associated with the elephant flow along different paths (e.g., equal-cost multipath routing (ECMP) legs). As another example, the forwarding element may be configured to send elephant flow traffic along a separate physical network, such as an optical network that is more suitable for slow changing, bandwidth-intensive traffic. Several such examples will now be described below by reference to  FIGS. 8-11 . 
         [0073]    A. Example QOS Operation 
         [0074]      FIG. 8  provides an illustrative example of a forwarding element  830  that performing QOS operations on packets in a detected elephant data flow. As shown, the figure shows a network that includes a forwarding element  825  that reports an elephant flow, and a network controller  805  that receives the report and configures the forwarding element  830 . To simply the description, the forwarding element  825  is also referred to below as the reporter, and the forwarding element  830  is also referred to below as the handler. 
         [0075]    The forwarding element  825  reports each detected elephant flow to the network controller. In response to a report regarding a detected elephant flow, the network controller  805  configures one or more forwarding elements. As mentioned above, the network controller can configure the same forwarding element, and/or can configure one or more other forwarding elements that receive packets in the same elephant data flow. 
         [0076]    In the example of  FIG. 8 , the network controller  805  includes an elephant agent client  810  and a configuration engine  815 . In some embodiments, the elephant agent client  810  is responsible for receiving a report regarding an elephant flow from the forwarding element  825  and storing the report in the storage  820  (e.g., a configuration database). The elephant agent client  810  may also inform the configuration engine  815  that a new report is stored in the storage  820 . In some embodiments, the elephant agent client  810  communicates with an elephant agent module that operates on the reporter  825 . An example of such a module is described above by reference to  FIG. 6 . 
         [0077]    Different from the client  810 , the configuration engine  815  receives information regarding the elephant flow stored in the storage and configures the forwarding element  830  (the handler). The configuration engine may be notified of a new report stored in the storage from the elephant agent client  810  or some other component, in some embodiments. For instance, the storage  820  may be managed by another component (e.g., database manager or server) that informs the configuration engine  815  of the changes to the data in the storage  820 . Irrespective of method of notification, the configuration engine  815  configures the handler  830 . In the example of  FIG. 8 , the configuration engine sends instructions to the handler. The forwarding element  830  then translates those instructions into a QOS configuration to perform on packets belonging to the detected elephant flow. 
         [0078]    The handler  830  is configured by the network controller to process packets belonging to the elephant flow different from other packets (e.g., in other flows, such a mouse flow or a non-detected elephant flow). As shown, the handler  830  includes (1) several ingress ports  845  to receive packets, (2) a classification engine  855  to find a matching rule to process each packet, (3) a QOS engine  860  to perform a QOS operation on each packet, (4) a number of egress queues (e.g.,  865 ,  870 , etc.), and (5) several egress ports  850  to output the packets. The handler  830  also includes a network controller interface  835 , which the network controller  805  (e.g., the configuration engine  860 ) to interface or communicate with the network controller  805  and a storage  840  to store configuration data (e.g., QOS configuration). In some embodiments, the configuration data includes one or more of elephant flow identifying information described above by reference to  FIG. 7 . 
         [0079]    Having described several components, example operations of operations of the network will now be described by reference to  FIG. 8 . As shown, the reporter  825  reports a detected elephant flow to the network controller  805 . In some embodiments, the elephant agent client  810  receives the report and stores the report in the storage  820 . The configuration engine  815  of some embodiments retrieves the report from the storage  820  and configures handler  830 . In configuration, the configuration engine  815  may communicate with the handler  830  through the interface  835 . In some endowments, the configuration data is stored in the storage  840  of the handler. Here, the storage stores QOS configuration data. 
         [0080]    The forwarding element  825 , which reported the elephant flow, also sends packets in the same elephant flow to the handler  830 . Each packet in the elephant flow is received at a particular ingress port  845 . The classification engine  855  receives the packet in the elephant flow and performs packet classification to identify a matching rule to process the packet. The packet is then received at the QOS engine  860 . The QOS engine  860  then reads the configuration data to identify the packet, and perform a QOS operation on the packet. As mentioned above, the forwarding element to place packets belonging to the elephant flow in a particular queue that is separate from one or more other queues with other packets. The particular queue may be a lower priority queue that one or more of the other queues, in some embodiments. Finally, the packet is retrieved from the particular queue and forwarded to a next hop through one of the egress ports  850 . 
         [0081]    Having described an example of performing QOS operation, an example process will now be described.  FIG. 9  conceptually illustrates a process  900  that some embodiments implement to perform QOS operations on packets associated a detected elephant flow. In some embodiments, the process  900  is performed by a forwarding element, such as the handler  830 . As shown, the process  900  receives (at  905 ) information regarding a detected elephant flow. Based on the information, the process  900  then identifies (at  910 ) a packet associated with the detected elephant flow. In some embodiments, the process identifies a packet associated with the elephant flow using one or more of pieces of information shown in  FIG. 7 . 
         [0082]    At  915 , the process  900  performs a QOS operation on the packet by placing the packet in a particular queue. The process then determines (at  920 ) whether there is another packet in the same elephant flow. If there is another packet, the process returns to  910 , which is described above. Otherwise, the process  900  ends. Some embodiments perform variations on the process  900 . The specific operations of the process  900  may not be performed in the exact order shown and described. The specific operations may not be performed in one continuous series of operations, and different specific operations may be performed in different embodiments. 
         [0083]    B. Example of Breaking an Elephant Flow into Mouse Flows 
         [0084]      FIG. 10  provides an illustrative example of a forwarding element  1030  that performing breaks an elephant flow into a bunch of mouse flows. This figure is similar to  FIG. 8 , except the handler  1030  includes a multi-path engine  1060  that reads the configuration data  1040  to send packets associated with the elephant flow along different paths. 
         [0085]    In the example of  FIG. 10 , the reporter  1025  reports a detected elephant flow to the network controller  1005 . The network controller  1005  configures the handler  1030  by storing configuration data in the storage  1040 . The forwarding element  1025 , which reported the elephant flow, also sends packets in the same elephant flow to the handler  1030 . Each packet in the elephant flow is received at a particular ingress port  1045 . The classification engine  1055  receives the packet in the elephant flow and performs packet classification to identify a matching rule to process the packet. The packet is then received at the multi-path engine  1060 . The multi-path engine  1060  then reads the configuration data to identify the packet, and perform send a packet along a particular path through one of the egress ports, such as ports  1065 ,  1070 , etc. If the multi-path engine receives another packet in the same data flow, it may send the other packet along a different path. 
         [0086]    Having described an example of breaking an elephant flow into several mouse flows, an example process will now be described.  FIG. 11  conceptually illustrates a process  1100  that some embodiments implement to break an elephant into mice. In some embodiments, the process  1100  is performed by a forwarding element, such as the handler  1030 . As shown, the process  1100  receives (at  1105 ) information regarding a detected elephant flow. Based on the information, the process  1100  then identifies (at  1110 ) a packet associated with the detected elephant flow. In some embodiments, the process identifies a packet associated with the elephant flow using one or more of pieces of information shown in  FIG. 7 . 
         [0087]    At  1115 , the process  1100  breaks an elephant flow into a mouse flow by forwarding the packet along one of several different paths. The process then determines (at  1120 ) whether there is another packet in the same elephant flow. If there is, the process returns to  1110 , which is described above. Otherwise, the process  1100  ends. Some embodiments perform variations on the process  1100 . The specific operations of the process  1100  may not be performed in the exact order shown and described. The specific operations may not be performed in one continuous series of operations, and different specific operations may be performed in different embodiments. 
       III. Electronic System 
       [0088]    Many of the above-described features and applications are implemented as software processes that are specified as a set of instructions recorded on a computer readable storage medium (also referred to as computer readable medium). When these instructions are executed by one or more computational or processing unit(s) (e.g., one or more processors, cores of processors, or other processing units), they cause the processing unit(s) to perform the actions indicated in the instructions. Examples of computer readable media include, but are not limited to, CD-ROMs, flash drives, random access memory (RAM) chips, hard drives, erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), etc. The computer readable media does not include carrier waves and electronic signals passing wirelessly or over wired connections. 
         [0089]    In this specification, the term “software” is meant to include firmware residing in read-only memory or applications stored in magnetic storage, which can be read into memory for processing by a processor. Also, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while remaining distinct software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of separate programs that together implement a software invention described here is within the scope of the invention. In some embodiments, the software programs, when installed to operate on one or more electronic systems, define one or more specific machine implementations that execute and perform the operations of the software programs. 
         [0090]      FIG. 12  conceptually illustrates an electronic system  1200  with which some embodiments of the invention are implemented. The electronic system  1200  may be a computer (e.g., a desktop computer, personal computer, tablet computer, etc.), server, dedicated switch, phone, PDA, or any other sort of electronic or computing device. Such an electronic system includes various types of computer readable media and interfaces for various other types of computer readable media. Electronic system  1200  includes a bus  1205 , processing unit(s)  1210 , a system memory  1225 , a read-only memory  1230 , a permanent storage device  1235 , input devices  1240 , and output devices  1245 . 
         [0091]    The bus  1205  collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of the electronic system  1200 . For instance, the bus  1205  communicatively connects the processing unit(s)  1210  with the read-only memory  1230 , the system memory  1225 , and the permanent storage device  1235 . 
         [0092]    From these various memory units, the processing unit(s)  1210  retrieves instructions to execute and data to process in order to execute the processes of the invention. The processing unit(s) may be a single processor or a multi-core processor in different embodiments. 
         [0093]    The read-only-memory (ROM)  1230  stores static data and instructions that are needed by the processing unit(s)  1210  and other modules of the electronic system. The permanent storage device  1235 , on the other hand, is a read-and-write memory device. This device is a non-volatile memory unit that stores instructions and data even when the electronic system  1200  is off. Some embodiments of the invention use a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive) as the permanent storage device  1235 . 
         [0094]    Other embodiments use a removable storage device (such as a floppy disk, flash memory device, etc., and its corresponding drive) as the permanent storage device. Like the permanent storage device  1235 , the system memory  1225  is a read-and-write memory device. However, unlike storage device  1235 , the system memory  1225  is a volatile read-and-write memory, such a random access memory. The system memory  1225  stores some of the instructions and data that the processor needs at runtime. In some embodiments, the invention&#39;s processes are stored in the system memory  1225 , the permanent storage device  1235 , and/or the read-only memory  1230 . From these various memory units, the processing unit(s)  1210  retrieves instructions to execute and data to process in order to execute the processes of some embodiments. 
         [0095]    The bus  1205  also connects to the input and output devices  1240  and  1245 . The input devices  1240  enable the user to communicate information and select commands to the electronic system. The input devices  1240  include alphanumeric keyboards and pointing devices (also called “cursor control devices”), cameras (e.g., webcams), microphones or similar devices for receiving voice commands, etc. The output devices  1245  display images generated by the electronic system or otherwise output data. The output devices  1245  include printers and display devices, such as cathode ray tubes (CRT) or liquid crystal displays (LCD), as well as speakers or similar audio output devices. Some embodiments include devices such as a touchscreen that function as both input and output devices. 
         [0096]    Finally, as shown in  FIG. 12 , bus  1205  also couples electronic system  1200  to a network  1265  through a network adapter (not shown). In this manner, the computer can be a part of a network of computers (such as a local area network (“LAN”), a wide area network (“WAN”), or an Intranet, or a network of networks, such as the Internet. Any or all components of electronic system  1200  may be used in conjunction with the invention. 
         [0097]    Some embodiments include electronic components, such as microprocessors, storage and memory that store computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, read-only compact discs (CD-ROM), recordable compact discs (CD-R), rewritable compact discs (CD-RW), read-only digital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a variety of recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), magnetic and/or solid state hard drives, read-only and recordable Blu-Ray® discs, ultra density optical discs, any other optical or magnetic media, and floppy disks. The computer-readable media may store a computer program that is executable by at least one processing unit and includes sets of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as is produced by a compiler, and files including higher-level code that are executed by a computer, an electronic component, or a microprocessor using an interpreter. 
         [0098]    While the above discussion primarily refers to microprocessor or multi-core processors that execute software, some embodiments are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions that are stored on the circuit itself. In addition, some embodiments execute software stored in programmable logic devices (PLDs), ROM, or RAM devices. 
         [0099]    As used in this specification and any claims of this application, the terms “computer”, “server”, “processor”, and “memory” all refer to electronic or other technological devices. These terms exclude people or groups of people. For the purposes of the specification, the terms display or displaying means displaying on an electronic device. As used in this specification and any claims of this application, the terms “computer readable medium,” “computer readable media,” and “machine readable medium” are entirely restricted to tangible, physical objects that store information in a form that is readable by a computer. These terms exclude any wireless signals, wired download signals, and any other ephemeral signals. 
         [0100]    While the invention has been described with reference to numerous specific details, one of ordinary skill in the art will recognize that the invention can be embodied in other specific forms without departing from the spirit of the invention. In addition,  FIGS. 3 ,  9 , and  11  conceptually illustrates processes. As mentioned above, the specific operations of this process may not be performed in the exact order shown and described. The specific operations may not be performed in one continuous series of operations, and different specific operations may be performed in different embodiments. Furthermore, the process could be implemented using several sub-processes, or as part of a larger macro process. Thus, one of ordinary skill in the art would understand that the invention is not to be limited by the foregoing illustrative details, but rather is to be defined by the appended claims.