Patent Description:
Electronic devices can composed of multiple different chips which need to communicate data amongst themselves in order for the electronic device to operate. Data communications between chips may be nondeterministic. For example, data communications between chips are subject to variable latencies between the transmission time at one chip to the reception time at another chip. That is the time it takes for data to travel from one chip to another is not constant, but subject to many different sources of variance in transmission time.

<CIT> discloses methods and systems that allow <NUM>-step timestamping of messages containing time-of-day information. The <NUM>-step timestamping methods and systems may reduce the impact of non-deterministic time delays in the transmit path (e.g., encryption, expansion, inclusion of tags), and may improve the accuracy of the time-of-day information of the packets. This document discloses an inter-chip timing synchronization method comprising: for each pair of chips in a plurality of chips of a semiconductor device: determining a first one-way latency for transmissions from a first chip in the pair to a second chip in the pair of chips, and determining a second one-way latency for transmissions from the second chip in the pair to the first chip in the pair of chips; receiving, at a semiconductor device driver, the first one-way latency and the second one-way latency for the pair of chips; determining, by the semiconductor device driver and from the respective first one-way latency and the second one-way latency for each pair of chips, a loop latency between each pair of chips; and adjusting, by the semiconductor device driver and for the pair of chips, a local counter of the second chip in the at least one pair of chips based on a characteristic inter-chip latency of the semiconductor device and the first one-way latency of the at least one pair of chips For example, systems and methods may allow accurate <NUM>-step timestamping of IEEE <NUM> Precision Time Protocol packets with the uncertainty of delays from MACSec encryption or other security mechanisms. Some embodiments employ estimation non-deterministic delay of previously transmitted packets to estimate the state of the transmit path. Some embodiments include communication channels to allow circuitry in transmit path to report the state.

The various implementations provide one or more of the following advantages. For example, in some implementations, the processes described herein minimize the variance in potential data arrival times for inter-chip communications. Reducing the variance of data communications may permit the use of smaller receive data buffers in the chips of the system. In some implementations, the processes described herein make data transmission operations between chips deterministic. For example, implementations may make it possible for a program compiler to use a constant (e.g., deterministic) latency time when calculating a local counter time for a receiving chip to access data from an input buffer that was sent from a neighboring chip to the receiving chip at a particular time.

Other features and advantages will be apparent from the description, the drawings, and the claims.

In general, the disclosure relates to inter-chip time synchronization and data transmission in multi-chip systems. More specifically, the disclosure provides chip operation processes that improve the predictability of data transmission between chips, and, in some examples, around a serial-ring topology of chips. The disclosure provides example processes for synchronizing the local counters of the chips in the system and performing data transmission in a way that accounts for the inherent variability data arrival times for inter-chip data transmissions making data reception times more, and in some cases, completely deterministic.

Referring first to inter-chip time synchronization, the time synchronization includes two aspects. A first aspect , not covered by the claims, is characterization of inter-chip latency for data transmissions between respective pairs of chips on a processing system. This process provides an operational characteristic of the board (e.g., a maximum inter-chip latency) which serves as a constant for synchronizing local chip counters each time the board is booted. The second aspect covered by the claims is synchronizing the local chip counters when the board is booted (e.g., "boot-time synchronization").

More specifically, the characterization process must be completed for each redesign of a board. For example, the maximum inter-chip latency is generally a physical characteristic that is dependent on the layout of the chips on the board. The characterization process includes measuring the "round trip" loop latency for transmissions between pairs of chips (e.g., adjacent pairs of chips) on the board that will engage in direct communications with each other. Further, in implementations that include chips connected in a series-ring arrangement, the characterization process can also include measuring a round trip transmission latency around the whole ring. Data gathered from these measurements can be used to determine the maximum inter-chip latency that would be experienced between any two chips.

The boot-time synchronization is performed to synchronize the local counters of the chips each time the board is booted, reset, or both. While each chip is clocked by a local clock that is synchronized with the local clocks of the other chips (e.g., each chip's clock has the same frequency and phase), the chips operate using local counters to clock individual chip operations and, upon booting the board or the chips coming out of reset, the individual counters will generally be at different count values. Accordingly, the boot-time synchronization is used to approximately synchronize the local count values of the chips.

The boot-time synchronization process includes measuring one-way latencies for transmissions between pairs of chips on the board. A board driver determines a local counter adjustment for one chip in each pair based on the maximum inter-chip latency that has been characterized for the board and the one of the one-way latencies between the chips in the pair. For example, the driver can adjust the local counter of one of the chips in the pair by increasing the counter value by the sum of the maximum inter-chip latency and one of the one-way latencies between the chips. In some implementations, the boot-time process includes adjusting the round trip latencies between one or more chip pairs, e.g., by adjusting a FIFO buffer of one of the chips.

In some implementations, the semiconductor chips can be application-specific integrated circuits (ASIC) designed to perform machine learning operations. An ASIC is an integrated circuit (IC) that is customized for a particular use. For example, an ASIC may be designed to perform operations of machine learning models including, e.g., recognizing objects in images as part of deep neural networks, machine translation, speech recognition, or other machine learning algorithms. When used as an accelerator for a neural network, for instance, an ASIC can receive inputs to the neural network and compute a neural network inference for the inputs. Data inputs to a neural network layer, e.g., either the input to the neural network or the outputs of another layer of the neural network, can be referred to as activation inputs. The inferences can be computed in accordance with respective sets of weight inputs associated with the layers of the neural network. For example, some or all of the layers may receive a set of activation inputs and process the activation inputs in accordance with the set of weight inputs for the layer to generate outputs. Moreover, neural network operations can be performed by a system of ASICS according to explicit operations schedules. As such, deterministic and synchronized data transfers between the ASIC chips can improve the reliability of the neural network operation and simplify debugging operations.

<FIG> is a schematic that illustrates an example multi-chip system <NUM>. The multi-chip system <NUM> can be a network of integrated circuits configured to perform machine learning operations. For example, the multi-chip system <NUM> can be configured to implement a neural network architecture. The multi-chip system includes a plurality of semiconductor chips <NUM>. The chips <NUM> can be general purpose integrated circuit chips or special purpose integrated circuit chips. For example, one or more of the chips <NUM> can be ASICs, a field programmable gate array (FPGA), a graphics processing unit (GPU), or any other suitable integrated circuit chip. A clock <NUM> is coupled to each of the chips <NUM> to provide a synchronous timing signal. For example, clock <NUM> can include a crystal oscillator that provides a common timing signal (e.g., a <NUM> clock signal) to each of the chips <NUM>.

The system <NUM> also includes a system driver <NUM>. The system driver <NUM> can be, for example, an external computing system such as a laptop computer, desktop computer, or server system. The system driver <NUM> is used to perform or manage the chip synchronization processes described herein, or portions thereof. For example, the system driver <NUM> can be configured to program the chips, manage boot operations of the system <NUM>, debug the chips, or a combination thereof. The system driver can be coupled to the chips <NUM> via a communication link. The system driver <NUM> can be coupled to the chips <NUM> through a configuration status register (e.g., a low speed interface for programing and debugging the chips).

In the illustrated example, the multi-chip system <NUM> includes eight ASIC chips <NUM> and one FPGA chip <NUM> arranged in a series-ring topology. More specifically, each chip <NUM> is in data communication with two adjacent chips; one on each side, such that data is communicated from chip to a neighboring chip around the ring. The chips <NUM> and their data communication links form a closed-loop. Furthermore, the multi-chip system <NUM> includes two data paths between each pair of chips; a clockwise path <NUM> and a counterclockwise path <NUM>.

In some implementations, each ASIC chip (P0-P7) can be configured to implement a layer of a neural network. Input activation data may be received by the FPGA chip <NUM> and transmitted to P0. P0 can be configured to implement the input layer of the neural network, for example. P0 would perform computations on the activation data to generate layer output data, which would be transmitted to P1. P1 can be configured to implement the first hidden layer of the neural network and would perform computations on the output from P0, then transmit its output to the next neural network layer implemented by P2. The process may continue around the ring through each of the ASICs <NUM>, and by extension processed by each layer of the neural network. Such a process may rely on precise timing of data transfers between adjacent chips (and around the entire ring) in order for the neural network to operate reliably and accurately. Consequently, synchronization of data transfers between each ASIC may, therefore, be important to ensure proper operational coordination between the chips.

Operations internal to a single chip in a synchronous system are synchronous and deterministic, meaning that there is no variance in the timing of such internal operations. However, for inter-chip operations, such as data transmission, there is an inherent and non-deterministic variability in the timing of the operations, even in a synchronous system. One source of timing variability is a property of the physical link between two adjacent chips, which can introduce a variance of, e.g., about <NUM>-<NUM> clock cycles in the latency of data transmission between adjacent chips. A second, larger source of timing variability is the lack of synchronization between internal chip operations and a forward error correction scheme implemented by the multi-chip system. In a forward error correction scheme, error correction data is added to a data transmission between chips, but the added error correction data is not necessarily synchronized with data transmission. The introduction of non-synchronized data to a data transmission can introduce a variance of, e.g., up to <NUM> clock cycles in the latency of data transmission between adjacent chips.

When data is transmitted from one chip to another, non-adjacent chip (e.g., from P0 to P7), the variance in latency for each inter-chip transmission (e.g., from P0 to P1, from P1 to P2, etc.) accumulates into a cumulative latency at the destination chip (P7). Taking just the variance due to forward error correction as an example, the latency of a single inter-chip transmission (e.g., from P0 to P1) has a variance of ±<NUM> clock cycles. Some operations, however, may require transmission of data from one chip <NUM> to another, non-adjacent chip <NUM>, e.g., from chip P0 to chip P3, or even around the ring from the first chip P0 to the last chip P7. As discussed in more detail below, to transmit data from one chip to another, non-adjacent chip (e.g., P0 to P7), data can be transmitted through each of the intervening chips (e.g., through chips P1 to P6) using bypass operations. However, the latency variance between chips will accumulate across <NUM> chips, with the total variance in latency around the ring approaching ±<NUM> clock cycles. The processes described below improve the predictability of data transmission between chips, and, in some examples, allow inter-chip data transmission to be performed in a deterministic manner.

<FIG> depicts a flowchart of an example process <NUM> for characterizing the maximum latency in a multi-chip , the process <NUM> not being covered by the claims. Process <NUM> will be described in reference to <FIG>, <FIG>, and <FIG>. In some implementations, process <NUM>, or portions thereof, is executed or controlled by the system driver <NUM>. In some examples, process <NUM>, or portions thereof, is executed by the individual chips <NUM> of the multi-chip system <NUM>. The characterization process <NUM> is used to determine a characteristic inter-chip latency, e.g., a maximum inter-chip latency (Lmax), of a multi-chip system design. For example, process <NUM> may be performed for initial chip arrangements, and/or new, system topologies.

The first step of process <NUM> includes determining a loop latency between each pair of chips in a multichip system <NUM> (step <NUM>). For example, as shown in <FIG>, the depicted multichip system <NUM> has ten individual inter-chip communication loops (<NUM>, <NUM>) with independently measurable latencies. There are nine loops <NUM> between adjacent chips <NUM>, and one loop <NUM> around the entire ring. In the multi-chip system, absolute latency values may only be measurable in these loops <NUM>, <NUM> since there is no common time reference available. That is, although each of the chips <NUM>, <NUM> are driven by a common clock <NUM>, the local counters on each chip <NUM>, <NUM> are not necessarily synchronized to the same count value. In other words, the "local time" on each chip <NUM>, <NUM> may be different. Measuring loop latencies rather than individual one-way latencies between chips can be used to account for the differences between local counters on each chip, as described in more detail below.

The nine single chip loops <NUM> going counter-clockwise first and then clockwise have the same latency as the clockwise-first loops because these loop latencies are simply the sum of the latency in each direction. Likewise, the full system counter-clockwise loop <NUM> has the same latency as the sum of all nine single chip loops <NUM> minus the latency of the clockwise system loop <NUM>. Measuring latency differences in different directions around the loop between two chips does not provide more information as these differences can be derived from the nine small loops <NUM> and the single system loop <NUM>.

<FIG> show a series of block diagrams illustrating loop latency measurements between adjacent chips <NUM>. <FIG> depict simplified block diagrams of two adjacent chips <NUM>: Chip A and Chip B. Each chip <NUM> includes a controller <NUM> that controls the local operations of the chip, a local counter <NUM>, and communication interfaces <NUM>. For clarity of explanation, the communication interfaces <NUM> are represented as from transmitter interfaces (Tx) and to receiver interfaces (Rx). The communication interfaces <NUM> include First-In-First-Out (FIFO) buffers.

To measure the loop latency, each chip to initializes its local counter <NUM>, e.g., by booting the chip <NUM>. Each chip's local counter <NUM> represents its local time as discussed above. In some implementations, the chips <NUM> perform their individual operations (e.g., computations, reading data from input buffers, and transmitting data to other chips) at pre-scheduled counter times. The counters <NUM> do not need to be synchronized in any way for process <NUM>. For instance, in the example shown in <FIG> Chip A's local counter <NUM> is initialized to time <NUM>, and Chip B's local counter <NUM> is initialized at time <NUM>, consequently Chip A's and Chip B's local counters are out of synchronization by <NUM> clock cycles. The boot synchronization process discussed below is used to synchronize the local counters <NUM> in the chips <NUM>. It should be noted that the counter times used in <FIG> (and <FIG>) are simplified for purposes of explanation.

Referring to <FIG>, to measure the round trip latency between Chip A and Chip B, Chip A and Chip B execute a series of time stamped data transmissions, first from Chip A to Chip B, then from Chip B to Chip A. For example, first Chip A sends time stamped data 309to Chip B in order to measure the relative one-way latency for transmissions in the first direction, e.g., from Chip A to Chip B on the clockwise data path <NUM>. Chip A sends data <NUM> to Chip B that includes a time stamp with Chip A's local counter time (e.g., <NUM>) when the data <NUM> was sent. For clarity of explanation <FIG> shows only one data transmission being sent to Chip A. In practice, for example, Chip A can send a series of data transmissions <NUM> at different points in the <NUM> cycle physical coding sublayer (PCS) period, each timestamped with Chip A's local counter time at the time of transmission. Chip B receives the data <NUM> and records its own local counter time (e.g., <NUM>). The difference between Chip A's local time (e.g., <NUM>) when the data <NUM> was sent and Chip B's local time (e.g., <NUM>) when the data <NUM> was received is equal to the relative one-way relative latency from Chip A to Chip B. For example, the relative one-way latency as illustrated in <FIG> is <NUM> clock cycles.

As shown in <FIG>, Chip B performs the same process to measure the measure the relative one-way latency for transmissions in the second direction, e.g., from Chip B to Chip A on the counterclockwise data path <NUM>. Chip B sends a data <NUM> to Chip A that includes a time stamp with Chip B's local counter time (e.g., <NUM>) when the data <NUM> was sent. For clarity of explanation <FIG> shows only one data transmission being sent to Chip A. In practice, for example, Chip B can send a series of data transmissions at different points in the <NUM> cycle PCS period, each timestamped with Chip B's local counter time at the time of transmission. Chip A receives the data <NUM> and records its own local counter time (e.g., <NUM>). The difference between Chip B's local time (e.g., <NUM>) when the data <NUM> was sent and Chip A's local time (e.g., <NUM>) when the data <NUM> was received is equal to the relative one-way relative latency in the direction from Chip B to Chip A. For example, the relative one-way latency as illustrated in <FIG> is -<NUM> clock cycles. It should be noted that the relative one-way latencies can be negative due to local counter differences between two adjacent chips <NUM>.

Once a series of data transmission has been performed, each chip <NUM> (e.g., Chips A and B) calculates a relative one-way latency in one direction based on the timestamp value included in the data (e.g., data <NUM> and data <NUM>) and its own local counter time when the data was received. Each chip <NUM> can then identify the maximum relative one-way latency that it measured and send the maximum relative one-way latency to the system driver <NUM> for calculation of the respective maximum loop latency. In some implementations each chip <NUM> sends the timestamp data from each transmission in the series of transmissions along with its own associated local counter value at the time that each transmission was received to the system driver <NUM>. The system driver <NUM> then calculates, for each pair of chips, the relative one-way latencies in each direction, identifies the maximum one-way latency in each direction, and calculates the respective maximum loop latency.

The relative one-way latency values are not meaningful by themselves because the local counters on each chip <NUM> will be in unknown states. But when the two relative one-way latencies between a given pair of chips <NUM> are summed (e.g., the relative one-way latency from Chip A to Chip B with the relative one-way latency from B back to A), the local counter differences cancel leaving only the absolute latency around the loop between Chip A and Chip B. For example, the loop latency calculation can be represented by the following equations: <MAT> <MAT> and <MAT> Ra, Rb represent the local counter time that time stamped data was received on Chip A or Chip B, respectively (e.g., Ra is <NUM> and Rb is <NUM> in the present example). Sa, Sb represent the counter time when the data was sent by Chip A or Chip B, respectively (e.g., Sa is <NUM> and Sb is <NUM> in the present example). Cba the difference in counter times between Chip B's local counter time and Chip A's local counter time: Cba=Cb-Ca (this is not directly observable) (e.g., Cba is <NUM> in the present example). Lab is the max-jitter absolute latency from Chip A to Chip B (this is not directly observable). Lba is the max-jitter absolute latency from Chip B to Chip A (this is not directly observable). max(Rb - Sa) represents the maximum relative one-way latency from Chip A to Chip B. max(Rb - Sa) is the difference between Chip B's local counter time when data is received from Chip A and Chip A's local counter time when the data was sent. This is also equivalent to the actual latency (Lab) in the direction from Chip A to Chip B plus the difference between Chip B's counter and Chip A's counter (Cba). max(Ra - Sb) represents the maximum relative one-way latency from Chip B to Chip A. max(Ra - Sb) is the difference between Chip A's local counter time when data is received from Chip B and Chip B's local counter time when the data was sent. This is also equivalent to the actual latency (Lba) in the direction from Chip B to Chip A minus the difference between Chip B's counter and Chip A's counter (Cba). This relationship could also be restated as max(Ra - Sb) = Lba + Cab, where Cab is Chip A's counter value minus Chip B's counter value, e.g., the opposite of Cba. Simply put, the offset between local counters on two chips creates the appearance of "additive" latency for transmissions in one direction and "subtractive" latency for transmissions in the opposite direction. Linter-chip_loop_max represents the maximum loop latency for a given loop <NUM> between two chips.

After running several measurements of single chip to neighbor loops <NUM>, the system driver <NUM> identifies the maximum loop latency among all the chip pairs (step <NUM>). For example, the system driver <NUM> can compare the maximum measured loop latencies from each of the transmission loops <NUM> between each chip pair to identify the maximum chip-to-chip loop latency (Lloop_max).

One of the chips <NUM> or the system driver <NUM> determines the ring latency for data transmissions around the entire ring <NUM> (step <NUM>) For example, a similar technique to that described with respect to <FIG>, is used to measure and compute the latency around the full ring, except that time stamped data is transmitted around the full ring and received at the same chip <NUM> that sent the data. The maximum transmission time measured around the full ring loop <NUM> will be the maximum full ring latency (Lring_max). Consequently, local counter differences are not a concern.

The system driver <NUM> determines the characteristic inter-chip latency (Lmax) for the multi-chip system <NUM> (step <NUM>). For example, the system driver <NUM> can compare half of the maximum chip-to-chip loop latency with one-Nth of the maximum full ring latency, where N is the total number of chips <NUM> in the multi-chip system <NUM> to estimate the maximum one-way latency in the system <NUM>. The greater of these two values is the characteristic inter-chip latency (Lmax) for the multi-chip system <NUM>. The system driver <NUM> can store the characteristic inter-chip latency for use in future operations. The characteristic inter-chip latency will be a constant used in other operation such as boot-time synchronization and data transmission as discussed below. In some implementations, the characteristic inter-chip latency is also used by the compiler for generating operations schedules for each chip <NUM> to execute particular software applications e.g., a particular machine learning algorithm In the embodiment of <FIG>, characteristic inter-chip latency represents the longest time that it would take for data to be transferred from one chip to an adjacent chip. The compiler can use the characteristic inter-chip latency to schedule a receiving chip to read data from an input FIFO buffer after an adjacent chip has sent the data and be assured that all the data would have arrived by the scheduled read time.

In some implementations, the Lmax can be increased by a design factor to account for any variances that may not have been measured during the characterization process. For example, the measured Lmax may not account for the maximum possible variance in the latency for data transmission between adjacent chips. Therefore, in some implementations Lmax can be increased to ensure that the actually inter-chip latencies experienced by the multi-chip system <NUM> will not exceed the value of Lmax.

<FIG> is a flowchart of a process <NUM> for synchronizing local counters of chips in a multi-chip system <NUM> , the process <NUM> being an embodiment of the present invention. Process <NUM> will be described in reference to <FIG>, <FIG>, and <FIG>. In some implementations, process <NUM>, or portions thereof, is executed or controlled by the system driver <NUM>. In some implementations, process <NUM>, or portions thereof, is executed by the individual chips <NUM> of the multi-chip system <NUM>. The synchronization process <NUM> is used to synchronize the local counters <NUM> of the chips <NUM> in the multi-chip system <NUM>. Process <NUM> can be performed when the system <NUM> is booted, and hence is referred to as a "boot synchronization" process. However, process <NUM> can also be performed at other times as well, e.g., if the multi-chip system is reset.

For each chip pair, a first relative one-way latency for data transmission from a first chip in the pair (e.g., Chip A) to a second chip in the pair (e.g., Chip B) is determined (step 402a), and a second relative one-way latency for data transmission from the second chip in the pair (e.g., Chip B) to the first chip in the pair (e.g., Chip A) is determined (step 402b). For example, the relative one-way latency on the clockwise data path <NUM> between the two chips can be determined, and then the relative one-way latency on the counter clockwise data path <NUM> between the two chips can be determined. The first and second relative one-way latencies can be measured, for example, using the techniques described above with reference to <FIG>. The chips <NUM> send the measured relative one-way latencies back to the system driver <NUM>. In some implementations, the system driver <NUM> controls the individual chips <NUM> to perform the relative one-way latency measurements. In some implementations, the individual chips <NUM> include software (e.g., firmware) that controls the individual chips <NUM> to perform the relative one-way latency measurements upon the system being booted or reset.

The system driver <NUM> determines the loop latency between each pair of chips (step <NUM>). The system driver <NUM> determines the loop latency between a pair of chips based on the respective relative one-way latencies measured between that pair of chips. The system driver <NUM> uses the equation <MAT> to calculate a loop latency between a given pair of chips. The system driver <NUM> repeats the calculation for each loop <NUM> between respective pairs of chips in the multi-chip system <NUM>.

The system driver <NUM> confirms that each loop latency is less than or equal to the characteristic inter-chip latency of the multichip system (Lmax) (step <NUM>). For example, the system driver <NUM> can compare the calculated loop latency for each pair of chips to the stored value of the characteristic inter-chip latency. In some implementations, if any of the calculated loop latencies is greater than the characteristic inter-chip latency, system driver <NUM> re-performs the loop latency measurements. The system driver <NUM> causes steps <NUM> and <NUM> to be re-performed. In some implementations, the system driver <NUM> generates an error signal if any of the calculated loop latencies is greater than characteristic inter-chip latency.

The system driver <NUM> synchronizes the chips <NUM> by adjusting the local counter of one or more chips based on the characteristic inter-chip latency (Lmax) (step <NUM>). For example, referring to <FIG>, one chip <NUM> of the multichip system <NUM> may be selected as a reference chip. For example, the counter value of the reference chip will serve as a base for adjusting the respective local counters <NUM> of the other chips <NUM> in the multichip system <NUM> in order to synchronize the chips <NUM>. In the present example, the FPGA chip will be used as the reference chip. The system driver <NUM> adjusts local counters in a pairwise fashion starting from the reference chip. The system driver <NUM> adjusts the local counter time of one chip in each pair of adjacent chips based on Lmax and one of the measured one-way latencies between the chips. For example, starting with the FPGA and P0, the system driver <NUM> adjusts the local counter time of P0 based on Lmax and the measured one-way latency for data transmissions from the FPGA to P0 (e.g., the measured one-way latency for data transmissions along the clockwise data path <NUM>). After the local counter in P0 is adjusted, the system driver <NUM> adjusts local counter of Chip P1. For example, the system driver <NUM> adjusts local counter time of P1 based on Lmax and the measured one-way latency for data transmissions from P0 to P1. The system driver <NUM> repeats this process to adjust the local counter of each chip <NUM> around the ring until all the chips have been synchronize. However, the local counter of the FPGA (e.g., reference chip) is not adjusted.

More specifically, using the example shown in <FIG>, system driver <NUM> adjusts the local counter time of one chip in a pair of chips based on Lmax and one of the measured relative one-way latencies between the two chips. The system driver <NUM> adjusts a chip's local counter <NUM> by increasing the counter value by Lmax minus the measured relative one-way latency from one chip in the pair to the chip whose local counter is being adjusted. That is, the new counter value (Tnew) can be determined by: Tnew = Told + Lmax - (Rb - Sa), where Told is the original counter value and (Rb - Sa) represents the relative one-way latency measured by the chip whose counter is being adjusted. For example, in <FIG> the relative one-way latency from Chip A to Chip B was measured to be <NUM>. Assuming an Lmax of <NUM>, the adjustment to Chip B's counter would be Lmax - (Rb - Sa) or <NUM> - <NUM> = -<NUM>. So, the system driver <NUM> would increase Chip B's local counter by -<NUM> counts (e.g. decrease the local counter by <NUM>). Using the simplest case as an example (e.g., the counter times shown in <FIG>), the system driver <NUM> would adjust A Chip B's local counter from <NUM> to <NUM>. Although the adjusted value for Chip B's local counter is not identical to the value of Chip A's local counter (e.g., <NUM>), the two chips can be considered synchronized for the purposes of this disclosure. For example, the synchronization process does not necessarily force the local counters of two chips to be equal, but synchronizes the data transmission latency between each pair of chips across the multichip system so that the maximum relative one-way latency between each pair of chips is less than or equal to Lmax.

In some examples not covered by the claims, the FIFO buffers at the Rx communication interfaces <NUM> of the chips <NUM> can also be adjusted. For example, the system driver <NUM> can adjust the perceived latency between inter-chip links by increasing or decreasing the receive buffer size (e.g., adding or removing latency in 4ns increments, until the chip loops <NUM> all have a loop latency in the range [<NUM>max - <NUM>, <NUM>max]. Consequently, full-system loop <NUM> would then have a loop latency is in the range [NLmax - <NUM>, NLmax], where N is the number of chips in the loop. Generally, latency will only need to be added, but it is possible that latency might need to be removed from some counter-clockwise pointing data paths <NUM>, e.g., in a case where all of the two-chip loops are within their limits, but the full system clockwise loop <NUM> needs more latency. In that case, the system driver <NUM> can remove some latency on some of the counter-clockwise data paths <NUM> (e.g., by decreasing one or more of the chip's receive buffers that are couponed to a counter-clockwise data path <NUM>, and add the same amount of latency to the clockwise links (e.g., by increasing appropriate receive data buffers), thereby, preserving the latency on each two-chip loop <NUM> while adding latency on the clockwise system loop <NUM>. In some examples, latency can be adjusted by increasing or decreasing appropriate transmitter FIFO buffer sizes rather than or in addition to adjusting receiver side FIFO buffers.

Referring to <FIG>, even after the chips <NUM> have been synchronized, one or more other remaining variance in chip-to-chip and data transmissions may need to be addressed. For example, the latency variance arising due to forward error correction (FEC) operations may need to be addressed. This variance impacts transmissions from one chip to another, non-adjacent chip greater than transmissions between adjacent chips. For example, to transmit data from one chip to another, non-adjacent chip (e.g., around the ring from the first chip P0 to the last chip P7), data can be transmitted through each of the intervening chips (e.g., through chips P1 to P6) using bypass operations. The latency of each data path between adjacent chips has constant and variable components. The exact value of the variable component can be difficult or impossible to determine. When transmitting data from one chip (e.g., P0) to another, non-adjacent chip (e.g., P7), e.g., in a bypass operation, the cumulative variance in the latency is the sum of the variance in the latency for each link across which the data is transmitted. At the destination chip (e.g., P7), the cumulative variance in the latency can be significant, resulting in a large amount of variability in the arrival time of the transmitted data at the destination chip (e.g., P7). In a synchronous system, this variability in arrival time can necessitate significant buffering of data at the destination. To eliminate the variability in arrival time at the destination chip (e.g., P7) arising from the cumulative variance in latency, a delay can be imposed on the data transmission from each chip <NUM>. The introduction of a delay at each chip <NUM> negates the effect of the variance in latency, and the arrival time of the data at the destination chip becomes deterministic and compatible with a synchronous system.

In some implementations, the delay is built in to the operations of each chip by a program compiler. For example, the program compiler uses Lmax to generate program instructions as explicitly scheduled operations for each chip. As described in more detail below in reference to <FIG>, <FIG>, each chip can be pre-scheduled to re-transmit data received from an adjacent chip at a time that is the maximum inter-chip latency (e.g., Lmax) after the data was transmitted to the chip. For example, the operations of chip P0 can be pre-scheduled to transmit data to chip P1 at local counter time t. Chip P1 would then be pre-scheduled to re-transmit the data to chip P2 at local counter time t + Lmax.

One source of timing variability is a property of the physical link between two adjacent chips (e.g., PCS jitter), which can introduce a variance in the latency of data transmission between adjacent chips <NUM>. This source of variability is addressed by the system characterization and synchronization processes (<NUM> and <NUM>) described above. However, a second source of timing variability is the lack of synchronization between internal chip operations and a forward error correction scheme implemented by the multi-chip system <NUM>. In a forward error correction scheme, error correction data is added to a data transmission between chips <NUM>, but the added error correction data is not necessarily synchronized with data transmission. The introduction of non-synchronized data to a data transmission can introduce a variance of, e.g., up to <NUM> clock cycles in the latency of data transmission between adjacent chips.

When data is transmitted from one chip <NUM> to another, non-adjacent chip <NUM> (e.g., from P0 to P7), the variance in latency for each inter-chip transmission (e.g., from P0 to P1, from P1 to P2, etc.) accumulates into a cumulative latency at the destination chip (P7). Taking just the variance due to forward error correction as an example, the latency of a single inter-chip transmission (e.g., from P0 to P1) has a variance of ±<NUM> clock cycles. Some operations, however, may require transmission of data from one chip <NUM> to another, non-adjacent chip <NUM>, e.g., from chip P0 to chip P3, or even around the ring from the first chip P0 to the last chip P7. As discussed in more detail below, to transmit data from one chip to another, non-adjacent chip (e.g., from P0 to P7), data can be transmitted through each of the intervening chips (e.g., through chips P1 to P6) using bypass operations. However, the latency variance between chips will accumulate across <NUM> chips, with the total variance in latency around the ring approaching ±<NUM> clock cycles. To make this large variability in arrival time at the destination chip compatible with a synchronous system, a significant amount of buffering of data can be implemented at receiver interfaces, e.g., Rx communication interfaces <NUM> by increasing receive FIFO buffer sizes on each chip <NUM>.

Additional buffering can be avoided, however, by preventing the accumulation of variance in latency throughout a multi-chip data transmission process. To achieve this, a small amount of delay can be introduced into the data transmission operation at each chip <NUM> such that the latency for data transmission between each pair of adjacent chips <NUM> is fixed rather than variable. Specifically, the maximum inter-chip latency (Lmax) is determined as discussed above. During data transmission, when data is received at a chip <NUM> in a bypass operation, the data is stored in a receive buffer, such as a FIFO buffer, rather than being sent immediately to the next chip. The data is released from the buffer only after the maximum inter-chip latency (e.g., Lmax) has elapsed since the data transmission was initiated at the previous chip <NUM>. In controlling the timing of each bypass operation in a data transmission process, the exact amount of time the entire data transmission process will then be a known value, meaning that there is no variability in the perceived arrival time of the data at the destination chip <NUM>.

<FIG> is a flowchart of an example process <NUM>, not covered by the claims, for conducting data transmissions between chips in a multi-chip system <NUM>. Process <NUM> will be described in reference to <FIG>, <FIG>, and <FIG> show a series of block diagrams illustrating the data transmission process <NUM>. The block diagrams are similar to those in <FIG>, with the exception that internal bypass data paths <NUM> and <NUM> are labeled. For example, the chips <NUM> can include a bypass data paths in two directions that allow the chip <NUM> to directly route data to the next chip in the ring topology.

Process <NUM>, or portions thereof, is executed by the individual chips <NUM> of the multi-chip system <NUM>. The data transmission process <NUM> is used to reduce the variability in data arrival times at destination chips <NUM> in the multi-chip system <NUM> in order to make data communications between chips <NUM> more deterministic. Furthermore, the data transmission process <NUM> may reduce the data input buffer sizes required on each chip <NUM>. Process <NUM> also allows the sequence of operations performed by each chip <NUM> in the multi-chip system <NUM> to be pre-scheduled and performed at pre-scheduled local counter times.

As shown in <FIG>, data <NUM> is transmitted from a first chip (e.g., Chip A) to a second chip (e.g., Chip B) (step <NUM>). For example, the data <NUM> is bypass data that is intended for a another chip <NUM> in the system <NUM>, and not for Chip B. Chip B receives the data <NUM>, and stores the data <NUM> in a buffer (step <NUM>). For example Chip B stores the data <NUM> in a FIFO buffer. Chip B stores the data until the maximum inter-chip latency has elapsed from when Chip A send the data. In the illustrated example, Chip B receives the data <NUM> at local counter time <NUM> and the maximum inter-chip latency is assumed to be Lmax = <NUM> counter cycles. Thus, Chip B does not transmit the data <NUM> to the next chip in the system <NUM> (e.g., Chip C) until local counter time <NUM> (e.g., <NUM> = Chip A's local counter time (<NUM>) when the data <NUM> was transmitted to Chip B plus the maximum inter-chip latency (<NUM>)), for example. This represents an exemplary delay time of eight counter cycles.

After the maximum inter-chip latency (e.g., Lmax) has elapsed from when the first chip (e.g., Chip A) transmitted the data <NUM>, the second chip (e.g., Chip B) releases the stored data from the buffer (step <NUM>), and transmits the released data <NUM> (<FIG>) to a third chip (e.g., Chip C) (step <NUM>). For example, after <NUM> cycles of Chip B's counter has elapsed from when Chip A transmitted the data <NUM> to Chip B, Chip B can release the data <NUM> from its FIFO buffer, pass the data along internal bypass path <NUM>, and transmit the data (shown as <NUM> in <FIG>) to Chip C.

In some implementations, chip operations are explicitly scheduled at predetermined counter values. So, for example, the delay time for storing bypass data in a given chips buffer is accounted for in the scheduled operations. For example, in reference to the example described above, Chip A's scheduled operating instructions would instruct Chip A to transmit the data <NUM> to Chip B at Chip A's local counter time of <NUM>. Chip B's scheduled operating instructions would instruct Chip B to release the data <NUM> from its input buffer and retransmit the data to Chip C at Chip B's local counter time of <NUM>. Thus, Chip B does not need to internally calculate a delay time for retransmitting the data <NUM>.

<FIG> is a schematic that illustrates an example of special purpose logic chip (e.g., an ASIC <NUM>) that can be used as one of the chips <NUM> in the system <NUM> of <FIG>, e.g., as the ASIC chips P0-P7. The ASIC <NUM> includes multiple tiles <NUM>, in which one or more of the tiles <NUM> includes special purpose circuitry configured to perform operations, such as e.g., multiplication and addition operations. In particular, each tile <NUM> can include a computational array of cells (e.g., similar to the computational units of <FIG>), in which each cell is configured to perform mathematical operations ). In some implementations, the tiles <NUM> are arranged in a grid pattern, with tiles <NUM> arranged along a first dimension <NUM> (e.g., rows) and along a second dimension <NUM> (e.g., columns). For instance, in the example shown in <FIG>, the tiles <NUM> are divided into four different sections (710a, 710b, 710c, 710d), each section containing <NUM> tiles arranged in a grid of <NUM> tiles down by <NUM> tiles across. In some implementations, the ASIC <NUM> shown in <FIG> may be understood as including a single systolic array of cells subdivided/arranged into separate tiles, in which each tile includes a subset/subarray of cells, local memory and bus lines.

The ASIC <NUM> also includes a vector processing unit <NUM>. The vector processing unit <NUM> includes circuitry configured to receive outputs from the tiles <NUM> and compute vector computation output values based on the outputs received from the tiles <NUM>. For example, in some implementations, the vector processing unit <NUM> includes circuitry (e.g., multiply circuitry, adder circuitry, shifters, and/or memory) configured to perform accumulation operations on the outputs received from the tiles <NUM>. Alternatively, or in addition, the vector processing unit <NUM> includes circuitry configured to apply a non-linear function to the outputs of the tiles <NUM>. Alternatively, or in addition, the vector processing unit <NUM> generates normalized values, pooled values, or both. The vector computation outputs of the vector processing units can be stored in one or more tiles. For example, the vector computation outputs can be stored in memory uniquely associated with a tile <NUM>. Alternatively, or in addition, the vector computation outputs of the vector processing unit <NUM> can be transferred to a circuit external to the ASIC <NUM>, e.g., as an output of a computation.

In some implementations, the vector processing unit <NUM> is segmented, such that each segment includes circuitry configured to receive outputs from a corresponding collection of tiles <NUM> and computes vector computation outputs based on the received outputs. For instance, in the example shown in <FIG>, the vector processing unit <NUM> includes two rows spanning along the first dimension <NUM>, each of the rows including <NUM> segments <NUM> arranged in <NUM> columns. Each segment <NUM> includes circuitry (e.g., multiply circuitry, adder circuitry, shifters, and/or memory) configured to perform a vector computation, as explained herein, based on outputs (e.g., an accumulated sum) from a corresponding column of tiles <NUM>. The vector processing unit <NUM> can be positioned in the middle of the grid of tiles <NUM> as shown in <FIG>. Other positional arrangements of the vector processing unit <NUM> are also possible.

The ASIC <NUM> also includes a communication interface <NUM> (e.g., interfaces 7010A, 7010B). The communication interface <NUM> includes one or more sets of serializer/deserializer (SerDes) interfaces and a general purpose input/output (GPIO) interface. The SerDes interface is configured to receive input data for the ASIC <NUM> and to output data from the ASIC <NUM> to an external circuit. For example, the SerDes interface can be configured to transmit and receive data at a rate of <NUM> Gbps, <NUM> Gbps, or any suitable data rate over the set of SerDes interfaces included within the communications interface <NUM>. For example, the ASIC <NUM> may run a boot program when it is turned on. The GPIO interface may be used to load instructions (e.g., operation schedules) onto the ASIC <NUM> and to communicate with the system driver <NUM> to execute a boot synchronization process (e.g., process <NUM>).

The ASIC <NUM> further includes multiple controllable bus lines configured to convey data among the communications interface <NUM>, the vector processing unit <NUM>, and the multiple tiles <NUM>. Controllable bus lines include, e.g., wires that extend along both the first dimension <NUM> (e.g., rows) of the grid and the second dimension <NUM> (e.g., columns) of the grid. A first subset of the controllable bus lines extending along the first dimension <NUM> can be configured to transfer data in a first direction (e.g., to the right of <FIG>). A second subset of the controllable bus lines extending along the first dimension <NUM> can be configured to transfer data in a second direction (e.g., to the left of <FIG>). A first subset of the controllable bus lines extending along the second dimension <NUM> can be configured to transfer data in a third direction (e.g. to the top of <FIG>). A second subset of the controllable bus lines extending along the second dimension <NUM> can be configured to transfer data in a fourth direction (e.g., to the bottom of <FIG>).

Each controllable bus line includes multiple conveyer elements, such as flip-flops, that are used to convey data along the lines in accordance with a clock signal. Transferring data over a controllable bus line can include shifting, at each clock cycle, data from a first conveyer element of the controllable bus line to a second adjacent conveyer element of the controllable bus line. In some implementations, data is conveyed over the controllable bus lines upon the rising or falling edge of a clock cycle. For example, data present, at a first clock cycle, on a first conveyer element (e.g., a flip-flop) of a controllable bus line can be transferred to a second conveyor element (e.g., a flip-flop) of the controllable bus line at a second clock cycle. In some implementations, the conveyer elements can be periodically spaced apart at a fixed distance from one another. For example, in some cases, each controllable bus line includes multiple conveyer elements, with each conveyer element positioned within or proximate to a corresponding tile <NUM>.

To minimize latency associated with internal operations of the ASIC chip <NUM>, the tiles <NUM> and vector processing unit <NUM> can be positioned to reduce the distance data travels among the various components. In a particular implementation, both the tiles <NUM> and communication interface <NUM> can be segregated into multiple sections, with both the tile sections and the communication interface sections being arranged such that the maximum distance data travels between a tile and a communication interface is reduced. For instance, in some implementations, a first group of tiles <NUM> can be arranged in a first section on a first side of the communications interface <NUM>, and a second group of tiles <NUM> can be arranged in a second section on a second side of the communication interface. As a result, the distance from a communication interface to the furthest tile may be cut in half compared to a configuration in which all of the tiles <NUM> are arranged in a single section on one side of the communication interface.

Alternatively, the tiles may be arranged in a different number of sections, such as four sections. For instance, in the example shown in <FIG>, the multiple tiles <NUM> of ASIC <NUM> are arranged in multiple sections <NUM> (710a, 710b, 710c, 710d). Each section <NUM> includes a similar number of tiles <NUM> arranged in a grid pattern (e.g., each section <NUM> can include <NUM> tiles arranged in <NUM> rows and <NUM> columns). The communication interface <NUM> also is divided into multiple sections: a first communication interface 7010A and a second communication interface 7010B arranged on either side of the sections <NUM> of tiles <NUM>. The first communication interface 7010A can be coupled, through controllable bus lines, to the two tile sections 710a, 710c on the left side of the ASIC chip <NUM>. The second communication interface 7010B can be coupled, through controllable bus lines, to the two tile sections 710b, 710d on the right side of the ASIC chip <NUM>. As a result, the maximum distance data travels (and thus the latency associated with the data propagation) to and/or from a communication interface <NUM> can be halved compared to an arrangement in which only a single communication interface is available. Other coupling arrangements of the tiles <NUM> and communication interfaces <NUM> are also possible to reduce data latency. The coupling arrangement of the tiles <NUM> and communication interface <NUM> can be programmed by providing control signals to the conveyer elements and multiplexers of the controllable bus lines.

In some implementations, one or more tiles <NUM> are configured to initiate reading and writing operations with respect to controllable bus lines and/or other tiles within the ASIC <NUM> (referred to herein as "control tiles"). The remaining tiles within the ASIC <NUM> can be configured to perform computations based on the input data (e.g., to compute layer inferences). In some implementations, the control tiles include the same components and configuration as the other tiles within the ASIC <NUM>. The control tiles can be added as an extra tile or tiles, an extra row or rows, or an extra column or columns of the ASIC <NUM>. For example, for a symmetric grid of tiles <NUM>, in which each tile <NUM> is configured to perform a computation on input data, one or more additional rows of control tiles can be included to handle reading and writing operations for the tiles <NUM> performing computations on the input data. For instance, each section <NUM> includes <NUM> rows of tiles, where the last two rows of tiles may include control tiles. Providing separate control tiles increases, in some implementations, the amount of memory available in the other tiles used to perform the computations. Separate tiles dedicated to providing control as described herein are not necessary, however, and in some cases, no separate control tiles are provided. Rather, each tile may store in its local memory instructions for initiating reading and writing operations for that tile.

Furthermore, while each section <NUM> shown in <FIG> includes tiles arranged in <NUM> rows by <NUM> columns, the number of tiles <NUM> and their arrangement in a section can be different. For example, in some cases, the sections <NUM> may include an equal number of rows and columns.

Furthermore, although shown in <FIG> as divided into four sections, the tiles <NUM> can be divided into other different groupings. For example, in some implementations, the tiles <NUM> are grouped into two different sections, such as a first section above the vector processing unit <NUM> (e.g., nearer the top of the page shown in <FIG>) and a second section below the vector processing unit <NUM> (e.g., nearer to the bottom of the page shown in <FIG>). In such an arrangement, each section may contain, e.g., <NUM> tiles arranged in a grid of <NUM> tiles down (along direction <NUM>) by <NUM> tiles across (along direction <NUM>). Sections may contain other total numbers of tiles and may be arranged in different sized arrays. In some cases, the divisions between sections are delineated by hardware features of the ASIC <NUM>. For example, as shown in <FIG>, sections 710a, 710b may be separated from sections 710c, 710d by the vector processing unit <NUM>.

Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory program carrier for execution by, or to control the operation of, data processing apparatus.

The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC.

The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA, an ASIC, or a GPGPU (general purpose graphics processing unit).

Claim 1:
An inter-chip timing synchronization method comprising:
for each pair of chips (<NUM>) of a plurality of chips (<NUM>) of a semiconductor device, said plurality of chips having at least three chips:
(i) determining (402a) a first one-way latency for transmissions from a first chip in the pair to a second chip in the pair of chips, and
(ii) determining (402b) a second one-way latency for transmissions from the second chip in the pair to the first chip in the pair of chips;
(iii) receiving, at a semiconductor device driver, the first one-way latency and the second one-way latency for each pair of chips;
(iv) determining (<NUM>), by the semiconductor device driver and from the respective first one-way latency and the respective second one-way latency for each pair of chips, a loop latency between chips in each pair of chips by summing the respective first one-way latency and the respective second one-way latency;
(v) determining (<NUM>), by the semiconductor device driver, whether each loop latency is less than or equal to a characteristic inter-chip latency of the semiconductor device, the characteristic inter-chip latency representing the longest time that it would take for data to be transferred from one chip of the plurality of chips to an adjacent chip of said plurality of chips;
(vi) in response to each loop latency being less than or equal to the characteristic inter-chip latency of the semiconductor device, adjusting (<NUM>), by the semiconductor device driver and for at least one pair of chips, a local counter of the second chip in the at least one pair of chips based on the characteristic inter-chip latency of the semiconductor device and the first one-way latency of the at least one pair of chips; and
(vii) in response to at least one determined loop latency being more than the characteristic inter-chip latency, repeating operations i) to iv) or generating an error signal by the semiconductor device driver.