Patent Publication Number: US-2023138522-A1

Title: Queue Bandwidth Estimation for Management of Shared Buffers and Allowing Visibility of Shared Buffer Status

Description:
FIELD OF THE INVENTION 
     The present invention relates generally to communication networks, and particularly to prioritized allocation of shared buffer resources to queues. 
     BACKGROUND OF THE INVENTION 
     In communication networks, streams of packets, or corresponding descriptors or pointers, are often temporarily stored in queues or buffers. 
     U.S. Pat. No. 5,014,265 describes a method of controlling congestion in a virtual circuit packet network. An initial packet buffer is assigned to each virtual circuit at each node into which incoming packets are stored and later removed for forward routing. If a larger buffer is desired for a virtual circuit to service a larger amount of data, then additional buffer space is dynamically allocated selectively to the virtual circuit on demand if each node has sufficient unallocated buffer space to fill the request. In one embodiment, the criterion for dynamic allocation is based on the amount of data buffered at the data source. In alternative embodiments, the criteria for dynamic allocation may be further based on the amount of data buffered at each node for a virtual circuit and the total amount of free buffer space at each node of a virtual circuit. 
     U.S. Pat. No. 5,541,912 discloses A dynamic threshold system and method for allocating memory among different output queues in a shared-memory ATM switch. The maximum permissible length for any individual queue at any instant of time is a function of the unused buffering in the switch. The dynamic threshold system and method deliberately reserves a small amount of buffer space, not allocating it to any currently active output queue, but attempts to equally share the remaining buffer space among the currently active output queues. The dynamic threshold system and method improve fairness and switch efficiency by guaranteeing access to the buffer space for all output queues, and by preventing any single output queue from monopolizing the memory at the expense of the others. The dynamic threshold system and method adapt to uncertain or changing load conditions. 
     SUMMARY OF THE INVENTION 
     An embodiment of the present invention that is described herein provides a network device including a memory and a memory management circuit. The memory is to store a shared buffer. The memory management circuit is to estimate respective bandwidth measures for one or more queues used in processing packets in the network device, and to allocate and deallocate segments of the shared buffer to at least one of the queues based on the bandwidth measures. 
     In some embodiments, the memory management circuit is to measure a size of data that is written to the one or more of the queues, and to estimate the respective bandwidth measures in accordance with the size of data. In some embodiments, the memory management circuit is to measure a size of data that is read from the one or more of the queues, and to estimate the respective bandwidth measures in accordance with the size of data. 
     In an embodiment, the memory management circuit is to allocate one or more segments of the shared buffer to a given queue responsively to a comparison of a bandwidth measure of the given queue and a preset maximum bandwidth limit. In another embodiment, the memory management circuit is to deallocate one or more segments of the shared buffer from a given queue responsively to a comparison of a bandwidth measure of the given queue and a preset minimum bandwidth limit. 
     In yet another embodiment, the memory management circuit is to assign a quota of segments of the shared buffer to a given queue in accordance with a bandwidth measure of the given queue, and to allocate the segments of the shared buffer to the given queue from the assigned quota. In still another embodiment, the memory management circuit is to assign a quota of segments of the shared buffer to a given queue in accordance with a bandwidth measure of the given queue, and to allocate reserved segments of the memory to the given queue from the quota. 
     In a disclosed embodiments, the bandwidth measures include one or more bandwidth histograms. In an example embodiment, the memory management circuit is to generate an interrupt responsively to a bandwidth measure of a queue. In an embodiment, the memory management circuit is to modify an allocation of segments of the shared buffer to a first queue, in response to a bandwidth measure of a second queue that shares a common resource with the first queue. In an example embodiment, the common resource is a shared-buffer pool. In an embodiment, the memory management circuit is to modify a congestion control algorithm responsively to a bandwidth measure. 
     There is additionally provided, in accordance with an embodiment that is described herein, a network device including multiple ports and a processor. The multiple ports are to communicate packets over a network. The processor is to estimate respective bandwidth measures for one or more queues used in processing the packets in the network device, and to output information indicative of the bandwidth measures. 
     There is further provided, in accordance with an embodiment that is described herein, a method including estimating respective bandwidth measures for one or more queues used in processing packets in a network device. Segments of a shared buffer of the network device are allocated and deallocated to at least one of the queues, based on the bandwidth measures. 
     There is also provided, in accordance with an embodiment that is described herein, a method including estimating respective bandwidth measures for one or more queues used in processing the packets in a network device. Information indicative of the bandwidth measures is output. 
     There is additionally provided, in accordance with an embodiment that is described herein, a method a network device. The method includes processing packets in the network device using one or more queues. Bandwidth measures are estimated for one or more of the queues. Based on the bandwidth measures, segments of a shared buffer of the network device are allocated and deallocating to at least one of the queues. 
     The present invention will be more fully understood from the following detailed description of the embodiments thereof, taken together with the drawings in which: 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG.  1    is a block diagram that schematically illustrates a network device, in accordance with an embodiment of the present invention; 
         FIG.  2 A  is a timing diagram that schematically illustrates memory accessing of a single port shared memory, in accordance with an embodiment of the present invention; 
         FIG.  2 B  is a block diagram that schematically illustrates queue bandwidth measurement in a network device, in accordance with an embodiment of the present invention; 
         FIG.  3    is a block diagram that schematically illustrates circuitry to allocate shared memory space responsively to queue bandwidth measurements, in accordance with an embodiment of the present invention; 
         FIG.  4    is a flowchart that schematically illustrates a method for allocating memory to queues responsively to queue bandwidth measurements, in accordance with an embodiment of the present invention; and 
         FIG.  5    is a block diagram that schematically illustrates a system for exposing queue bandwidth data to users, in accordance with an embodiment of the present invention. 
     
    
    
     DETAILED DESCRIPTION OF EMBODIMENTS 
     Overview 
     Network devices, such as network switches, network routers, Network Interface Controllers (NICs) , Host Channel Adapters (HCAs) and others, communicate packets over a communication network (e.g., Ethernet or InfiniBand™). 
     In a network device, the packets may be logically arranged in queues and temporarily stored in memory buffers. At least some of the memory may be shared between a plurality of queues. In high performance systems, to various concurrent queues may be managed in parallel and, hence, a plurality of queues may sometimes compete over a limited shared memory space. 
     To prioritize between queues that compete for the shared memory, various criteria can be used to allocate memory space to queues (and to deallocate memory space form other queues), including, for example, queue occupancy, the Quality of Service (QoS) of the respective flow of packets, a queue occupancy measure, various far-allocation policies, and others. 
     In accordance with such allocation criteria, network devices may allocate more space to a queue when the occupancy of the queue reaches a predefined threshold; the threshold would be set so that the queue will not overflow (e.g., exceed a maximum capacity) during the period starting when the threshold is exceeded to the time that the queue receives additional memory space (“the memory allocation response time”). In that respect, it may be beneficial to prioritize queues based on the bandwidths of the queues for better shared-buffer algorithm performance. For example, a high bandwidth queue is more likely to overflow during the memory allocation response time than a low bandwidth queue. 
     Embodiments of the present invention that are disclosed herein provide network devices and methods that use queue bandwidth as a shared-memory allocation criterion, possibly in combination with one or more other the shared-memory allocation criteria. Thus, a high bandwidth queue, which may quickly exhaust its allocated memory space, will be prioritized when competing for the allocation of additional memory resources. 
     In some embodiments, a memory management circuit (MMC) in the network device comprises circuitry to measure the bandwidth of queues. In an embodiment, the MMC comprises a memory access control circuit (MAC), which multiplexes the parallel memory access channels to bursts of data directed to the single read/write port of the shared memory; the MAC indicates the size of the data that is transferred in each burst and the ID of the corresponding queue. Bandwidth accumulators then accumulate the data sizes, independently for each queue. The accumulated data size in each time period is indicative of the bandwidth of the queue; in some embodiments the accumulation is further filtered, to achieve a smooth bandwidth vs. time relationship for the queues. 
     In embodiments, the measured queue bandwidth is used by the MMC as a criterion to increase or decrease the memory space allocated to the queue. Thus, for example, two queues having the same occupancy level but different bandwidths may be allocated different amounts of memory space (since the queue having the larger bandwidth is likely to exhaust its allocated memory space more quickly). 
     In another embodiment, the bandwidths that the MMC measures may be exposed to users, in the form of high or low bandwidth alerts, bandwidth reports, bandwidth histograms and others. 
     System Description 
     Network devices typically communicate (i.e., send and/or receive) packets, in a plurality of concurrent streams, over a communication network. In the network device, the streams may be logically handled by queues, and the network device may temporarily store packets corresponding to queues in a shared memory. 
     The shared memory is typically a limited resource, and the network processor device would typically attempt to manage the allocation of the shared memory between the queues in an optimal manner, according to a set of criteria, including queue occupancy, Quality of Service (QoS) class and others. 
     According to embodiments of the present invention, an important criterion for the allocation and deallocation of shared memory space to queues is the bandwidth of the queue (measured, for example, in Mbyte per second). Thus, a high bandwidth queue, which is likely to rapidly fill-up a large space in memory (in case the emptying of the filled data is delayed or slow), will be allocated a larger space in memory. 
       FIG.  1    is a block diagram that schematically illustrates a network device  100 , in accordance with an embodiment of the present invention. Network device  100  may be a switch, a bridge, a router, or a network interface such as a network interface controller (NIC) or a host channel adapter (HCA). The network device comprises Ports  102 , including ingress and egress ports, which communicate packets over a communication network  104  (e.g., Ethernet or InfiniBand™). Ports  102  typically comprise physical layer (PHY) and media access control (MAC) interface circuits, as are known in the art. 
     Network device  100  further comprises a shared memory  106 , a memory management circuit (MMC)  108 , a plurality of queue circuits  112  and a processor  114 , which controls the operation of the network device. Other components of network device  100  are omitted, for the sake of simplicity. 
     Network device  100  allocates packets that are communicated over the network to queue circuits  112  and may temporarily store the packets in buffers in shared memory  106 . The buffers in shared memory  106  are also referred to herein collectively as a “shared buffer”. Memory management circuit (MMC)  108 , which is coupled to the queue circuits, allocates space (e.g., segments) of the shared memory (i.e., of the shared buffer) to some queues and deallocates space from other queues, according to a set of criteria. 
     According to the example embodiment illustrated in  FIG.  1   , one of the criteria to allocate memory space (e.g., memory segments) to the queues (or deallocate memory space from the queues) is the bandwidth of the queue. Towards that end, MMC  108  comprises a shared memory allocation control circuit  116 , and a plurality of bandwidth measurement circuits  118 , which measure the respective memory access bandwidths of queues (all queues or a subset of the queues, e.g., queues with high QoS). The bandwidths of the queues are input to the shared memory allocation control, which may use the bandwidth measurements when deciding to allocate more memory to the queue, or when deciding to deallocate memory from the queue. 
     In some embodiments, the bandwidth measurement of a first queue may affect memory allocations of other queues, for example, if the first queue and the other queues share a common resource (e.g., a shared-buffer pool). Thus, according to the example embodiment illustrated in  FIG.  1    and described above, a network device measures the bandwidth of one or more of the queues and may use the measured bandwidths as a criterion for allocating shared memory to the corresponding queues. 
     The configuration of network device  100  is an example configuration that is cited by way of example; other suitable structures may be used in alternative embodiments. For example, in some embodiments the network device measures the bandwidth at the memory ports (see, for example,  FIG.  2    below); in some embodiments the shared memory is distributed among the ports. In an embodiment, processor  114  is a plurality of processors. 
     Although the description above refers to egress queues, the disclosed techniques may be applied to ingress and/or egress queues, including virtual queues. 
     Bandwidth Measurement 
     Several techniques may be used for measuring the bandwidths of the queues. We will disclose below one such technique, according to an embodiment of the present invention. 
     In some embodiments, shared memory  106  includes a single port for writing and reading data, and data flow from the various sources is multiplexed when written to the memory; similarly, data read from the memory is demultiplexed to the various data sinks. We will refer to the time-period during which a single queue writes or exchanges data with the shared memory as a timeslot. Queue bandwidth may conveniently be measured by monitoring the memory traffic related to the queue; the time-period may be measured, for example, from the time that a packet is written to a buffer to the time that the packet is read from the buffer; for another example, the time-period may be measured from the time packet processing starts to the time that the packer egresses the scheduler. 
       FIG.  2 A  is a timing diagram  200  that schematically illustrates memory accessing of a single port shared memory, in accordance with an embodiment of the present invention. Timing events are drawn along a time axis  202 , and include a first timeslot  204 , in which the transfer of data pertaining to a queue n take place, followed by a second timeslot  206 , in which the transfer of data pertaining to queue m takes place. The size of data transferred during the first timeslot is designated Sn, and the size of data in the second timeslots is designated Sm. In an embodiment, the bandwidths of queues n and m can be estimated by accumulating the corresponding Sn and Sm. 
       FIG.  2 B  is a block diagram  250  that schematically illustrates queue bandwidth measurement in a network device, in accordance with an embodiment of the present invention. MMC  108  ( FIG.  1   ) comprises a memory access control circuit (MAC)  252 , which multiplexes data to be written into memory from the various queues. For example, the MAC may transfer data pertaining to a queue n to a corresponding segment of the shared memory at a first timeslot, and data pertaining to queue m at a second timeslot. 
     According to the example embodiment illustrated in  FIG.  2 B , MAC  252  sends, after each timeslot, a queue-ID indication and a data-size indication to bandwidth accumulators  254 , which accumulate the data sizes separately for each of the queues. A timer  256  sends a transfer-and-clear trigger output to the accumulators every preset time-period (e.g., every 1 mS). The accumulators, responsively to the trigger input, send the current accumulation results for all the queues to filters  258 , and clear the accumulators, to start a new count. 
     The accumulated data sizes for each period in the time between the triggers is indicative of the bandwidth of the queues; however, in embodiments, the bandwidth-time function obtained by accumulating the data sizes at a predefined periods may not be suitable for memory allocation purpose (for example, the bandwidth-time function may include high-frequency glitches). In some embodiments, filters  258  may shape the bandwidth-time function, e.g., by applying finite-impulse-response (FIR) filtering, infinite impulse response (IIR) filtering or other suitable filtering. Filters  258  output the shaped bandwidth functions of the queues to shared memory allocation control  116  ( FIG.  1   ). As explained above, the shared memory allocation control may then use the bandwidths to determine allocation of shared memory space to the queues. 
     The configuration and the circuits used for bandwidth management, illustrated in  FIG.  2 B  and described hereinabove, are examples that are cited by way of example. Other suitable structures and circuits may be used in alternative embodiments. For example, in some embodiments, MAC  252  may transfer long bursts of data to the shared memory for some or for all the queues, and a temporal bandwidth measure for the queue may be calculated by dividing the data size by the timeslot duration. In some embodiments, bandwidth may be measured based on bytes per second, packets per second and/or fragments per second. 
     In embodiments, accumulators  254  and/or filters  256  are omitted and, instead, a processor (e.g., processor  114 ,  FIG.  1   ) receives timeslot data from the MAC, and calculates the bandwidths of the queues. 
       FIG.  3    is a block diagram  300  that schematically illustrates circuitry to allocate shared memory space responsively to queue bandwidth measurements, in accordance with an embodiment of the present invention. In an embodiment, circuitry  300  is included in MMC  108  ( FIG.  1   ). A queue bandwidth measurement circuit  302  measure the bandwidths of queues, for example, by accumulating data sizes transferred by MAC  252  ( FIG.  2 B ). 
     According to the example embodiment illustrated in  FIG.  3   , the measured bandwidth of Qn is input to a high-limit comparator  304 , which compares the bandwidth to a high threshold, and to a low-limit comparator  306 , which compares the bandwidth to a low threshold. If the bandwidth is higher than the high threshold, comparator  304  signals to shared memory allocation control  116  ( FIG.  1   ) that the memory allocation of Qn should be increased, to avoid rapid overflows that may occur as a result of the high bandwidth. If the bandwidth is lower than the low threshold, comparator  306  signals to the shared memory allocation control that the memory space allocated to Qn may be decreased (e.g., returned to a pool, and, subsequently, allocated to other queues). 
     As explained above, shared memory allocation control  116  may use additional criteria to allocate memory spaces to queues (and to deallocate memory space from queues), including, for example, queue occupancy, QoS, congestion notifications, and others. The decision to allocate additional space to a queue, or to deallocate space from a queue is taken considering the inputs from comparators  302 ,  306 , and considering other criteria. 
     Thus, according to the example embodiment illustrated in  FIG.  3    and described above, the bandwidth of a queue and, more specifically, the comparison of the bandwidth to thresholds, may be used to increase or decrease the memory space allocated to the queue. 
     The configuration of the circuits illustrated in  FIG.  3    and described above is an example that is cited for conceptual clarity. Other suitable configurations may be used in alternative embodiments. For example, in embodiments, comparators  304  and  306  are implemented in software; in some embodiments the bandwidth value is linearly weighted with other criteria when considering allocation or deallocation of memory space. In an embodiment, a programmable allocation function is implemented, with the queue bandwidth being one of the parameters of the function. 
     Types of Queues and Queue Attributes 
     The techniques described herewith may be applied to a variety of queues and queue types in network devices. Such queues may include transmit queues, receive queues and flexible queues. Separate queues may be used for each priority group and for each QoS. Some queues are defined collectively for communication flows and may include both receive and transmit queues. 
     In some embodiments, a set of attributes is defined for each queue; the attributes may include, among others, the type of the queue, a related port, a related priority group and a bandwidth attribute, which the MMC may update whenever the MMC measures a new bandwidth value. In some embodiments, queue attributes may include a queue bandwidth status, e.g., queue bandwidth is above a preset maximum, or below a preset minimum. 
       FIG.  4    is a flowchart  400  that schematically illustrates a method for allocating memory to queues responsively to queue bandwidth measurements, in accordance with an embodiment of the present invention. The flowchart is executed by the memory management circuit (MMC)  108  ( FIG.  1   ). 
     The flowchart starts at a measure-bandwidth step  402 , wherein the MMC measures the bandwidth of a queue from the set of all queues (or of a selected subset of the queues). Bandwidth measurement can be done, for example, by the circuits described with reference to  FIG.  2 B . 
     Next, at an allocate-memory step  404 , the MMC allocates, responsively to a high bandwidth value, additional shared memory space to the queue. As a high-bandwidth queue may rapidly overflow its allocated memory space, increasing the space responsively to a measured high bandwidth value may prevent loss of data. A bandwidth may be considered high responsively to a comparison of the bandwidth to a preset high threshold. In some embodiments the high threshold may dynamically change, responsively to the sum or bandwidths of all queues pertaining to the same port. (It should be noted that, in embodiments, the high bandwidth measured for the queue is one of a plurality of criteria used by the MMC to determine if the memory allocation of the queue should be increased.) 
     The MMC then enters a deallocate-memory step  406 , wherein the MMC, responsively to a low bandwidth value, deallocates memory space from the low-bandwidth queue. Typically, the MMC returns the deallocated memory space to a pool of memory segments, which the MMC may then allocate to queues that need additional memory space. 
     After step  406 , the MMC reenters step  402 , to handle bandwidth-based memory allocation and deallocation for the next queue. The loop comprising steps  402 ,  404  and  406  repeats for all queues (or, in an embodiment, for a subset of the queues). After executing the loop for all queues, the MMC may restart the loop from the first queue; in some embodiment, the loop comprising steps  402 ,  404  and  406  executes as long as the network device is active. 
     The flowchart illustrated in  FIG.  4    and described hereinabove is cited by way of example. Other suitable flowcharts may be used in alternative embodiments. For example, in an embodiment, the MMC measures bandwidth only for congested queues; in other embodiments the MMC continuously measures and registers the bandwidths of all queues but uses the registered bandwidth value only when congestion conditions occur. 
     In some embodiments, the network device may use the queue bandwidth measurements for other purposes, in addition (or alternatively) to the allocating and deallocating of memory space. For example, in an embodiment, the network device may expose the bandwidths to users. 
     Bandwidth Related Triggers 
     In some embodiments, bandwidth measurements of queues may trigger activities in the network device, in addition or instead of the memory allocation and deallocation to queues described above. In some embodiments, bandwidth related interrupts may be defined by a user (e.g., trigger a CPU interrupt if the bandwidth of any queue is beyond a preset maximum for more than a preset time). In another embodiment, queue measures may be used by a congestion control algorithm employed by the network device. In other embodiments, packet mirroring may be triggered based on the queue bandwidth, to enable visibility and/or remote analysis/telemetry. In an embodiment, the bandwidth measurement may be used by a packet processor, e.g., as a criterion to redirect or to duplicate the packet, again, to enable visibility and to allow remote analysis/telemetry. Lastly, in some embodiments, queue bandwidth measurements may be exposed to users. 
       FIG.  5    is a block diagram that schematically illustrates a system  500  for the exposition of queue bandwidth data to users, in accordance with an embodiment of the present invention. Processor  114  of Network device  100  ( FIG.  1   ) receives the bandwidth values of various communication queues from MMC  108  (the bandwidths may be measured, example, according to the technique illustrated in  FIG.  2 B  and described with reference thereof). According to the example embodiment illustrated in  FIG.  5   , the processor may run a plurality of processes, including processes that process and expose bandwidth to users (the term “user” refers, in this example embodiment, to a human user such as a network maintenance engineer; in other embodiments, however, the term “user” may also apply to non-human users, e.g., to network supervisory computer programs). 
     The processes that processor  114  executes may include a max-min bandwidth monitor  502 , which compares the bandwidth to preset extremum values and alerts the user if any of the extrema are exceeded; a queue bandwidth utilization process, which prepares and publishes (e.g., sends to a user) a report that lists the bandwidth utilization of queues; and a bandwidth histogram process  506 , which prepares and publishes histograms of the queues. 
     According to the example embodiment illustrated in  FIG.  5   , processor  114  may send the alerts, reports and/or histograms generated by processes  502 ,  504  and  506  to a local user  508  through a graphic input/output device  510  that is coupled to the network device. Alternatively, or additionally the processor may send the reports, through network  104 , to a remote graphic input output device  512 , for use by a remote user  514 . Users  508  and/or  512  may interactively control processes  502 ,  504  and/or  506 ; (e.g., defining thresholds for the alerts, adding, or removing queues from the reports and from the histogram, etc.). 
     Thus, according to the example embodiment illustrated in  FIG.  5   , a network device that measures bandwidths of queues can not only allocate memory to the queues responsively to the bandwidth, but also send bandwidth related reports, visualized views, and alerts, pertaining to the measured bandwidths. 
     It should be clarified that the configuration illustrated and  FIG.  5    and described hereinabove is an example that is cited merely for the sake of conceptual clarity. Other suitable configurations may be used in alternative embodiments. For example, in practice, the display of bandwidth related information to the user would be coupled with other visibility parameters which may be defined by the user or by a suitable monitor software. In some embodiments, the visibility data is used by a network optimization software program. 
     In some embodiments, bandwidth reports and histograms may be generated by a remote processor, which reads the bandwidth measurement results from MMC  108 . In an embodiment, a supervisory program may request the generation of bandwidth reports of queues that are coupled to a suspicious port; and in another embodiment a maintenance engineer may request, for example, that all queue bandwidth measurements during a 24-hour period should be stored in a maintenance file. 
     The configuration of network device  100 , including MMC  108 , shared-memory allocation control  116  and MAC  252 , the configuration of circuitry  300  and of bandwidth exposition system  500 , as well as flowchart  400 , are example configurations and methods that are shown purely by way of illustration. Any other suitable configurations and methods can be used in alternative embodiments. 
     In various embodiments, the bandwidth measurements circuitry, the ensuing memory allocation/deallocation circuitry, and the bandwidth-related exposition circuitry described hereinabove may be carried out by hardware, by software, or by a combination of hardware and software. 
     In various embodiments, the different elements of network device  100  seen in  FIG.  1   , including bandwidth measurement and memory allocation/deallocation elements, of MMC  108  seen in  FIG.  2 B , and of the circuitry seen in  FIGS.  3  and  5   , may be implemented using suitable hardware, such as one or more Application-Specific integrated Circuits (ASIC) or Field-Programmable Gate Arrays (FPGA), or a combination of ASIC and FPGA. 
     Processor  114  typically comprises one or more general-purpose processors, which are programmed in software to carry out at least part of the functions described hereinabove. The software may be downloaded to the processor in electronic form, over a network, for example, or it may, alternatively or additionally, be provided and/or stored on non-transitory tangible media, such as magnetic, optical, or electronic memory. 
     It will thus be appreciated that the embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and sub-combinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art. Documents incorporated by reference in the present patent application are to be considered an integral part of the application except that to the extent any terms are defined in these incorporated documents is a manner that conflicts with the definitions made explicitly or implicitly in the present specification, only the definitions in the present specification should be considered.