Abstract:
A method, computer readable medium, system and apparatus that acquires data link timing includes sequentially introducing a delay and sampling data on a link after each sequentially introduced delay. A starting edge of a valid data eye and a trailing edge of the valid data eye during the sequentially introducing the delay and the sampling of the data is determined. The sequentially introduced delay when the starting edge of the valid data eye is detected and a subsequently introduced delay when the trailing edge of the valid data eye is detected are recorded. A bit sampling time that provides the timing for the sampling of data in the valid data eye between the sequentially introduced delay and the subsequently introduced delay is determined. By way of example, an optimum bit sampling time is determined as a mean from the transition of the starting edge of the valid data eye to the trailing edge of the valid data eye. The bit sampling time for the sampling of data is applied and the link is established.

Description:
FIELD 
     The technology generally relates to computer system bus communications, and more particularly, to acquiring optimal data link timing for communicating over a system bus, such as a Hyper-Transport bus. 
     BACKGROUND 
     Communication technologies that link electronic devices are many and varied, servicing communications via both physical media and wirelessly. 
     Some communication technologies interface a pair of devices, other communication technologies interface small groups of devices, and still other communication technologies interface large groups of devices. Examples of communication technologies that couple small groups of devices include buses within computers, e.g., PCI (peripheral component interface) bus, ISA (industry standard architecture) bus, USB (universal serial bus), and SPI (system packet interface). 
     One communication technology for coupling relatively small groups of devices or components is the HyperTransport (HT) technology, previously known as the Lightning Data Transport technology (HyperTransport I/O Link Specification “HT Standard”), which is a data packet-based communication protocol for data transfer over buses. There are currently three versions of HT, with current AMD® Opteron® processors adhering to the HT 2.0b specification, for example. Generally, the HT specifications set forth definitions for a high-speed, low-latency protocol that can interface with other buses like AGP, AGP 8x, PCI, PCI-X, SPI, 1394 (Firewire), USB 2.0, Infiniband, or 1 and 10 Gb Ethernet, for example. 
     HT interconnects provide high-speed data links between coupled components. Most HT enabled components include at least a pair of HT ports so that HT enabled components may be daisy-chained. In an HT chain or fabric, each coupled component may communicate with each other coupled component using appropriate addressing and control. By way of example, HT generally is used to connect AMD® processors to other AMD® processors or bridge devices. 
     Many server computers employ HT protocol bus architectures to provide connectivity between the server host system processors and peripheral components. Currently, a bus implementing the Hyper-Transport protocol can be used to potentially transfer data at speeds anywhere from 200 MHz to 3.2 Ghz, depending on the Hyper-Transport protocol version and a number of other factors. The particular speed which is utilized is negotiated during system boot time and is limited by the speed capabilities of the connected device or peripheral. Since a server&#39;s overall performance can be limited by the bus speed, it is desirable to operate this bus at the maximum possible speed. 
     The primary challenge in utilizing the maximum or optimal speed achievable over HT buses is establishing very accurate timing for sampling of the data in the center of the data eye. The HT protocol does not provide any provisions for establishing accurate timing during negotiation of the link. As a result, achieving accurate sampling timing is very challenging at frequencies greater than about 400 MHz because of various factors, such as temperature and voltage variations. 
     SUMMARY 
     A method for acquiring data link timing includes sequentially introducing a delay and sampling data on a link after each sequentially introduced delay with an application delivery controller. The application delivery controller determines a starting edge of a valid data eye and a trailing edge of the valid data eye during the sequentially introducing the delay and the sampling of the data. The sequentially introduced delay when the starting edge of the valid data eye is detected and a subsequently introduced delay when the trailing edge of the valid data eye is detected are recorded with the application delivery controller. A bit sampling time that provides the timing for sampling of data in the valid data eye between the sequentially introduced delay and the subsequently introduced delay is determined with the application delivery controller. By way of example, an optimum bit sampling time is determined as a mean from the transition of the starting edge of the valid data eye to the trailing edge of the valid data eye. The bit sampling time is applied and the link is established with the application delivery controller. 
     A computer readable medium having stored thereon instructions for acquiring data link timing comprising machine executable code which when executed by at least one processor, causes the processor to perform steps including sequentially introducing a delay and sampling data on a link after each sequentially introduced delay. A starting edge of a valid data eye and a trailing edge of the valid data eye during the sequentially introducing the delay and the sampling of the data is determined. The sequentially introduced delay when the starting edge of a valid data eye is detected and a subsequently introduced delay when the trailing edge of the valid data eye is detected are recorded. A bit sampling time that provides the timing for the sampling of data in the valid data eye between the sequentially introduced delay and the subsequently introduced delay is determined. By way of example, an optimum bit sampling time is determined as a mean from the transition of the starting edge of the valid data eye to the trailing edge of the valid data eye. The bit sampling time for the sampling of data is applied and the link is established. 
     A data link timing acquisition system includes a data tap management system and a timing acquisition processing system in an application delivery controller. The data tap management system sequentially introduces a delay and samples data on a link after each sequentially introduced delay. The timing acquisition processing system records the sequentially introduced delay when a first unstable region, i.e. not in the valid data eye, is detected during the sampling the data and a subsequently introduced delay when a second unstable region is detected during the sampling the data. The first unstable region is separated from the second unstable region by a valid data eye. The timing acquisition processing system determines a bit sampling time that provides the timing for the sampling of data in the valid data eye between the sequentially introduced delay and the subsequently introduced delay, applies the bit sampling time, and establishes the link. By way of example, the timing acquisition system determines an optimum bit sampling time as a mean from the transition of the starting edge of the valid data eye to the trailing edge of the valid data eye. 
     An application delivery controller apparatus includes one or more host system processors, a host system memory coupled to the one or more host system processors, and a network interface unit coupled to the one or more host system processors and the host system memory via at least one bus utilizing a Hyper-Transport protocol. At least one of the network interface unit with logic configured to implement and the one or more host system processors configured to execute programmed instructions stored in the host memory system including sequentially introducing a delay and sampling data on a link after each sequentially introduced delay. A starting edge of a valid data eye and a trailing edge of the valid data eye during the sequentially introducing the delay and the sampling of the data is determined. The sequentially introduced delay when the starting edge of a valid data eye is detected and a subsequently introduced delay when the trailing edge of the valid data eye is detected are recorded. A bit sampling time that provides the timing for the sampling of data in the valid data eye between the sequentially introduced delay and the subsequently introduced delay is determined. By way of example, an optimum bit sampling time is determined as a mean from the transition of the starting edge of the valid data eye to the trailing edge of the valid data eye. The bit sampling time for the sampling of data is applied and the link is established. 
     The bus timing acquisition technology enables at or near optimal timing for sampling of the data on a bus utilizing the Hyper-Transport protocol to be achieved. Once the optimum timing is acquired and loaded, no further changes are required. Additionally, this Hyper-Transport timing acquisition technology is compatible with the existing Hyper-Transport protocol and does not require any system modifications. Further, this Hyper-Transport timing acquisition technology takes advantage of the periodic Hyper-Transport Cyclic Redundancy Check (CRC) sequence to iteratively determine the optimum sample timing to the center of the received data eye so that the Hyper-Transport bus can achieve reliable operation at much higher frequencies than previously was possible. This Hyper-Transport timing acquisition technology also enables ASICs, FPGAs and other devices which implement the Hyper-Transport bus protocol and other applicable bus protocols to operate at higher bus speeds than might otherwise be achievable utilizing prior known techniques. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is an exemplary network environment in which data link timing is acquired; 
         FIG. 2  is a block diagram of an application delivery controller in the exemplary network environment; 
         FIG. 3  is a partial block and partial flow diagram of a portion of a network interface unit in the application delivery controller; 
         FIG. 4  is a flow chart of a method for acquiring data link timing with the application delivery controller shown in  FIGS. 1-3 ; and 
         FIGS. 5-8  are timing diagrams illustrating the data link timing acquisition. 
     
    
    
     DETAILED DESCRIPTION 
     An exemplary network environment  10  in which data link timing is acquired, such as for a computer system bus employing the HyperTransport bus protocol, is illustrated in  FIG. 1 . The exemplary network environment  10  includes an application delivery controller  12 , client computing devices  14 ( 1 )- 14 ( n ), server devices  16 ( 1 )- 16 ( n ), and a communication network  18 , although the environment could include other types and numbers of systems, devices, blades, components, elements, and communication networks in other configurations. Generally, the application delivery controller  12 , utilizing the data link timing acquisition methods described herein, facilitates network communications between one or more of the client computing devices  14 ( 1 )- 14 ( n ) and one or more of the server devices  16 ( 1 )- 16 ( n ). Among other advantages, the data link timing acquisition technology enables at or near optimal timing for sampling of the data on a bus utilizing the Hyper-Transport protocol to be achieved. 
     Referring more specifically to  FIGS. 1-3 , the application delivery controller  12  includes at least one central processing unit (CPU) or processor  20 , at least one memory  22  and at least one network interface unit  24 , which are coupled together by a bus  26  or other numbers and types of links, although the controller  12  may include other components and elements in other configurations. In this example, the bus  26  utilizes the Hyper-Transport protocol and comprises a link with sixteen discrete physical lanes HT_CADIN[0]-HT_CADIN[15], along with a clock signal HT_CLKIN as shown in greater detail in  FIG. 3 , although the bus  26  could comprise other numbers and types of links and elements (e.g., HT_PWROK and HT_RESET#), and utilize other types of protocols. 
     The HT protocol implemented on bus  26  may conform to any of the HT specification versions, such as the HyperTransport™ I/O Link Specification, Revision 2.0b, Document # HTC20031125-0035-0009, published by the HyperTransport Technology Consortium, Apr. 27, 2005, and example implementations of the HT protocol are disclosed in U.S. Pat. No. 7,549,091, entitled “Hypertransport Exception Detection and Processing,” issued Jun. 16, 2009 to Rowlands, et al.; U.S. Pat. No. 7,117,308, entitled “Hypertransport Data Path Protocol,” issued Oct. 3, 2006 to Mitten, et al.; and U.S. Pat. No. 7,111,092, entitled “Buffer Management Technique for a Hypertransport Data Path Protocol,” issued Sep. 19, 2006 to Mitten, et al., each of which is incorporated herein by reference in its entirety. 
     The central processing unit (CPU) or processor  20  executes a program of stored instructions for one or more aspects of the technology as described herein, including, by way of example only, at least a portion of the methods for acquiring bus timing described herein. The memory  22  stores these programmed instructions for execution by the processor  20  to perform one or more aspects of the technology as described herein, although some or all of the programmed instructions could be stored and/or executed elsewhere. A variety of different types of memory storage devices can be used for the memory  22 , such as random access memory (RAM), read only memory (ROM) in the system, floppy disk, hard disk, CD ROM, DVD ROM or other computer readable medium read from and/or written to by a magnetic, optical, or other reading and/or writing controllers/systems coupled to the processor  20 , and combinations thereof. 
     In this example, the network interface unit  24  is a field programmable gate array that may include a local memory and be configured with logic to implement one or more aspects of the technology, including by way of example only, acquiring a bus clock signal and a sixteen bit data stream from the bus  26 , sequentially introducing delay and sampling data from the data bit stream, conducting a cyclical redundancy check or other error check on the sampled data, identifying an unstable region and a valid data eye or window in the sampled data, and determining a bit sampling time for sampling data in the data bit stream in the valid data eye, although the network interface unit  24  can comprise other types of configurable hardware and may include other components and elements configured in other manners which could implement one or more aspects of the technology. 
     Each of the client computing devices  14 ( 1 )- 14 ( n ) and server devices  16 ( 1 )- 16 ( n ) includes a central processing unit (CPU) or processor, a memory, and an interface or I/O system, which are coupled together by a bus or other link, although other numbers and types of network devices could be used. 
     The communication network  18  is a public data/communication network, such as a TCP/IP network by way of example only, which may include one or more other types and numbers of communication networks employing a variety of communications protocols, although other types of networks and protocols could be used. 
     Although an exemplary environment  10  with the application delivery controller  12 , the client computing devices  14 ( 1 )- 14 ( n ), the server devices  16 ( 1 )- 16 ( n ), and the communication network  18  are described and illustrated herein, other types and numbers of systems, devices, blades, components, and elements in other configurations can be used. It is to be understood that the systems of the examples described herein are for exemplary purposes, as many variations of the specific hardware and software used to implement the examples are possible, as will be appreciated by those skilled in the relevant art(s). 
     Furthermore, each of the systems of the examples may be conveniently implemented using one or more general purpose computer systems, microprocessors, digital signal processors, and micro-controllers, programmed according to the teachings of the examples, as described and illustrated herein, and as will be appreciated by those ordinary skill in the art. 
     In addition, two or more computing systems or devices can be substituted for any one of the systems in any example. Accordingly, principles and advantages of distributed processing, such as redundancy and replication also can be implemented, as desired, to increase the robustness and performance of the devices and systems of the examples. The examples may also be implemented on computer system or systems that extend across any suitable network using any suitable interface mechanisms and communications technologies, including by way of example only telecommunications in any suitable form (e.g., voice and modem), wireless communications media, wireless communications networks, cellular communications networks, G3 communications networks, Public Switched Telephone Network (PSTNs), Packet Data Networks (PDNs), the Internet, intranets, and combinations thereof. 
     The examples may also be embodied as a computer readable medium having instructions stored thereon for one or more aspects of the technology as described and illustrated by way of the examples herein, which when executed by a processor (or configurable hardware), cause the processor to carry out the steps necessary to implement the methods of the examples, as described and illustrated herein. 
     An exemplary method for acquiring data link timing in the application delivery controller  12  will now be described with reference to  FIGS. 1-4 . It should be noted that some steps are described in the ensuing description as being performed by certain components of the application delivery controller  12 , such as the network interface unit  24 , for exemplary purposes only. It should be understood and appreciated, however, that other components of the controller  12  could perform these steps and in other orders and sequences. 
     As such, in step  100 , the network interface unit  24  in the application delivery controller  12 , for example, initially remains in an idle state until a reset indication is received from either one of the client computing devices  14 ( 1 )- 14 ( n ), such as by an administrative user of the client devices accessing the controller  12  via a local area network connection, although other manners for initiating can be used, including the server devices  16 ( 1 )- 16 ( n ) or directly accessing an interface on the controller  12  itself to initiate the restart (e.g., restart, power or reset button). 
     In step  102 , when the reset indication is received by the network interface unit  24  in the application delivery controller  12 , for example, a bus communication link is initialized at a starting data rate, such as 200 MHz, in a sixteen bit mode between the bus  26  and the network interface unit  24 , although the link can be initialized in other manners and at other frequencies with other numbers of bits, such as with eight bits, by way of example only. As part of the initialization, the application delivery controller  12  receives and stores partner issued credits in memory  22  for later use during re-negotiation once the timing has been acquired, although the credits can be stored elsewhere. 
     In step  104 , the application delivery controller  12  initializes the cyclic redundancy check (CRC), although other manners for identifying a transmission error could be used. Additionally, the application delivery controller  12  disables issuance of the partner issued credits so the link is not established until the Hyper-Transport bus timing has been acquired as described herein. 
     In step  106 , the network interface unit  24  initializes a data tap delay to a minimum input delay, such as 75 picoseconds, although other delays and other manners for obtaining an initial or starting data tap delay can be used, such as determining an acquired data tap delay for an initial data rate and using this acquired tap delay as the starting data tap delay for acquiring another data tap delay for a second higher data rate by way of example only. When the clock signal HT_CLKIN goes high, the initialized or starting data tap delay is applied at IDELAY for each of the physical lanes HT_CDIN[0]-HT_CADIN[15] and then the data is sampled at the output of the IDELAY for each of the physical lanes HT_CDIN[0]-HT_CADIN[15]. 
     In step  108 , the network interface unit  24  conducts a CRC check on the sampled data from physical lanes HT_CDIN[0]-HT_CADIN[15], although other types of data checks could be used. If in step  108 , the network interface unit  24  determines the CRC for the sampled data from physical lanes HT_CDIN[0]-HT_CADIN[15] is either good without a prior bad CRC or is bad, then the No branch is taken to step  110 . 
     In step  110 , the network interface unit  24  advances the current data tap delay by one tap of about 75 picoseconds in this particular example, although the amount the data tap delay is advanced, or declined, can vary as needed for the particular application. Once the current data tap delay has been advanced, the network interface unit  24  samples the data on each of the physical lanes HT_CDIN[0]-HT_CADIN[15] and then the method returns to step  108  for another CRC or other error check as described above, although the data can be sampled in other manners. 
     If in step  108 , the network interface unit  24  determines the CRC is good with at least one prior bad CRC, then the Yes branch is taken to step  112 . In step  112 , the network interface unit  24  records the current data tap delay which was applied as the lower limit of a valid input delay to a valid data eye for sampling data from physical lanes HT_CDIN[0]-HT_CADIN[15], although this current data tap delay could be recorded at other locations and other manners for determining an edge or start of the valid data eye can be used. 
     In step  114 , the network interface unit  24  advances the recorded current data tap delay by one tap of about 75 picoseconds in this particular example, although the amount the data tap delay is advanced, or declined, can vary as needed for the particular application. Once the current data tap delay has been advanced, the network interface unit  24  again samples the data on each of the physical lanes HT_CDIN[0]-HT_CADIN[15] for another CRC or other error check in step  116 , although the data can be sampled in other manners. 
     In step  116 , the network interface unit  24  conducts the cyclic redundancy check (CRC) on the sampled data from physical lanes HT_CDIN[0]-HT_CADIN[15]. If in step  116 , the network interface unit  24  determines the CRC for the sampled data from physical lanes HT_CDIN[0]-HT_CADIN[15] is good, then the Yes branch is taken back to step  114  as described above. 
     If in step  116 , the network interface unit  24  determines the CRC is bad, then the No branch is taken to step  118 . In step  118 , the network interface unit  24  records the current data tap delay which was applied as the upper limit for sampling data from physical lanes HT_CDIN[0]-HT_CADIN[15], although this current data tap delay could be recorded at other locations and other manners for determining an edge or end of the valid data eye or stable region can be used. 
     In step  120 , the application delivery controller  12  determines whether the maximum data rate capability for the application delivery controller  12  has been reached, although other manners for determining whether to try a higher data rate can be used. By way of example only, if the maximum data rate capability was 400 MHz or 600 MHz, an initial determination at 200 MHz would trigger moving to the maximum data rate for the application delivery controller  12  stored in memory, although other increments above the initial data rate and below the maximum data rate could be used. As such, if in step  120 , the application delivery controller  12  determines the maximum data rate capability has not been reached, then the NO branch is taken to step  122 . If in step  120 , however, the application delivery controller determines the maximum data rate capability has been reached, then the Yes branch is taken to step  124 . 
     In step  122 , the current data rate is stepped up to the maximum data rate capability for the application delivery controller  12 , such as 400 MHz in this example, although other step values and other manners for increasing or obtaining a higher data rate can be used. Next, the process returns to step  106  where the acquired timing at the first data rate of 200 MHz is now used as the starting data tap delay at the new data rate of 400 MHz so the data initially is sampled data in a stable region. 
     In step  124 , following the network interface unit  24  determining a higher data rate should not be used, the unit  24  determines the acquired data tap delay by taking the mean of the recorded upper and lower limits for the data tap delay determined at steps  112  and  118 , for example, although another data tap delay between the recorded upper and lower limits could be used and other manners for determining the acquired data tap delay can be used. The network interface unit  24  loads the acquired data tap delay for sampling data at the current data rate. 
     In step  126 , since the bus timing has been acquired, the application delivery controller  12  loads the credits that were previously stored back at step  102 , and the process proceeds to step  128 . 
     In step  128 , the application delivery controller  12  enables sending the loaded credits and then in step  130  the data bus communication link is established. Accordingly, with the bus timing acquired at or near optimal timing for sampling of the data on the bus, utilizing the Hyper-Transport protocol can now be achieved. Additionally, once this optimum Hyper-Transport timing is acquired and loaded as illustrated in the examples herein, no further changes are required. 
     By way of example only, the timing diagram in  FIG. 5  illustrates the initial reset, the initial and sequential advancement of the data tap delay, the CRC checks and the eventual establishment of the link. 
     Referring to  FIG. 6 , an example where the data on the bus utilizing the Hyper-Transport protocol happens to align perfectly with respect to the sampling clock is illustrated. As is shown, the input is discretely swept over just greater than twice the variance of the timing uncertainty. In this example, this is approximately 800 picoseconds. Also in this example, the input tap delay resolution is 75 ps thus approximately +/−12 taps of sweeps as described in the flow chart above are used to determine the lower and upper limits and the optimum data tap delay for the sampling of the data. 
     The points marked A, B, C, and D in the timing diagram shown in  FIG. 6  represent the locations of sweep delay boundaries. Points A and D represent the maximum endpoints of the sweep delay. Points B and C frame the valid data eye or the stable region for sampling data and represent input tap delays which resulted in valid sampling during the Hyper-Transport timing acquisition. In this example, the optimum sampling point is the mean of tap delay represented by each of the points B and C, although other locations between points B and C could be used as the optimum sampling point. 
     Referring to  FIGS. 7 and 8 , the illustrated timing diagrams represent the same example as shown in  FIG. 6 , except the data on the bus utilizing the Hyper-Transport protocol is either too slow or too fast with respect to the sampling clock. Again in these other examples, the discrete input delay sweep will extract the points B and C from which the optimum sampling point again is the mean of tap delay represented by each of the points B and C, although other locations between points B and C could be used as the optimum sampling point. 
     Accordingly, as illustrated and described herein, this data bus timing acquisition method, which is applicable to buses employing the Hyper-Transport protocol, for example, enables obtaining the optimal input data sampling point allowing reliable operation of the bus at higher frequencies than previously was possible. Although these methods have been described in the context of the Hyper-Transport bus protocol, it should be appreciated that these teachings may be equally applicable to other buses and protocols without departing from the spirit and scope of these teachings. 
     Having thus described the basic concept of the invention, it will be rather apparent to those skilled in the art that the foregoing detailed disclosure is intended to be presented by way of example only, and is not limiting. Various alterations, improvements, and modifications will occur and are intended to those skilled in the art, though not expressly stated herein. These alterations, improvements, and modifications are intended to be suggested hereby, and are within the spirit and scope of the invention. Additionally, the recited order of processing elements or sequences, or the use of numbers, letters, or other designations therefore, is not intended to limit the claimed processes to any order except as may be specified in the claims. Accordingly, the invention is limited only by the following claims and equivalents thereto.