Semi receiver side write training for non-volatile memory system

Technology is disclosed herein for semi receiver side write training in a non-volatile memory system. The transmitting device has delay taps that control the delay between a data strobe signal and data signals sent on the communication bus. The delay taps on the transmitting device are more precise that can typically be fabricated on the receiving device (e.g., NAND memory die). However, the receiving device performs the comparisons between test data and expected data, which alleviates the need to read back the test data. After the different delays have been tested, the receiving device informs the transmitting device of the shortest and longest delays for which data was validly received. The transmitting device then sets the delay taps based on this information. Moreover, the write training can be performed in parallel on many receiving devices, which is very efficient.

BACKGROUND

In source synchronous systems, a transmitting device sends both data signals carrying data and a clock signal to a receiving device. The clock signal is sometimes referred to as a data strobe signal. The receiving device uses the clock signal to identify data values of the data carried by the data signals. In particular, the receiving device identifies levels of data pulses in response to detecting transitions of the clock signal. A deviation of the clock transitions from their optimal times is referred to as skew between the clock signal and the data signal. Too large of skew between the data signals and clock signal may cause the receiving circuit to incorrectly identify the levels of the data pulses. Increases in frequency of the data and clock signals magnify the skew problem.

A write training process may be used to calibrate delays between the clock signal and the data signals, which helps to provide a wider data valid window. The write training process typically includes trying a number of different delays between the data signals and clock signal, which can be time consuming.

Some specifications, such as the Open NAND Flash Interface (ONFI) Specification, describe procedures for write DQ training. The ONFI specification describes write DQ training at the transmitter (Tx) side, as well as optional write DQ training at the receiver (Rx) side (see Open NAND Flash Interface Specification, Revision 4.2, Feb. 12, 2020). As an example, the transmitter could include a memory controller, and the receiver could include a semiconductor die containing NAND memory cells.

For ONFI Tx side write training, the Tx side sends test data to the Rx side. Then, the test data is transferred back from the Rx to the Tx. The Tx then compares the test data that was read back to the expected data to see if further training (DQ delay) is needed. Therefore, the Tx side training can be slow due to, for example, the need to transfer test data back from the Rx to the Tx.

For ONFI Rx side write training, the Rx side may compare the test data it received from the Tx with expected data. The Rx side may adjust the DQ delays to achieve the widest data valid window. However, the circuitry on the Rx side might not be able to achieve a high resolution in the delays. This is due to typical limitations in the semiconductor process used to fabricate the Rx (e.g., NAND memory die). Therefore, Rx side write training can suffer from lack of precision. As the frequency of data transmission becomes greater, low precision in the DQ delays makes it challenging to implement Rx side write training.

DETAILED DESCRIPTION

Technology is disclosed herein for write training in a non-volatile memory system. The write training may be referred to as semi-receiver side write training. In an embodiment, the transmitting device has delay taps that control the delay between the data strobe signal and the data signals that are sent on the communication bus. The transmitting device may include a semiconductor die that contains a memory controller. Moreover, the delay taps on the transmitting device are more precise that can typically be fabricated on the receiving device (e.g., NAND memory die). However, the receiving device performs the comparisons between the test data and the expected data, which alleviates the need to read back the test data. After the different delays have been tested, the receiving device informs the transmitting device of the shortest and longest delays for which data was validly received. The transmitting device then sets the delay taps, which are on the transmitting device, based on this information. Moreover, the write training can be performed in parallel on many receiving devices, which is very efficient. Hence, semi-receiver side write training is faster than transmitting side training, and can have a finer delay resolution than receiving side write training. Having a finer delay resolution is especially important as the transmission speeds over the communication bus increase. If the delay resolution is not high enough write training may fail. Therefore, write training in which the receiver side (e.g., NAND memory die) has delay taps to control the delay may fail if the transmission speed is too high.

FIG.1shows one embodiment of a memory system100in which write training may be performed. The memory system100includes a transmitting circuit102and a number of receiving circuits104(1)-104(p). The transmitting circuit102and each respective receiving circuit104are configured to communicate with each other via a communications bus106. The following discussion will use the reference number104to refer to any of the receiving circuits. Both the transmitting circuit102and the receiver circuit104may be transceiver circuits, which can be configured to transmit and receive signals.

Additionally, in some embodiments, each of the transmitting circuit102and the receiving circuits104are integrated circuits (IC). In general, an integrated circuit (IC)—also referred to as a monolithic IC, a chip, or a microchip—is an assembly or a collection of electric circuit components (including active components, such as transistors and diodes, and passive components, such as capacitors and resistors) and their interconnections formed as a single unit, such as by being fabricated, on a substrate typically made of a semiconductor material such as silicon. For such embodiments, the transmitting circuit102and the receiving circuits104are separate integrated circuits, and the communication bus106is configured to communicate signals external to the separate transmitting circuit (IC)102and the receiving circuits (IC)104. In some embodiments, each receiving circuit104contains a memory structure having non-volatile memory cells, and the transmitting circuit102contains a memory controller. In some embodiments, each receiving circuit104contains a control circuit that is configured to connect to a memory structure that resides on a separate IC from the receiving circuit104.

The transmitting circuit102is configured to send a clock signal CLK and a plurality of data signals DQ to one or more receiving circuits104via a communications bus106. Hence, a receiving circuit104is configured to receive the clock signal CLK and a plurality data signals DQ from the transmitting circuit102via the communications bus106. The plurality of data signals DQ are shown inFIG.1as including data signals DQ(1) to DQ(N), where N is two or more. As an example, N is 8, although other integer numbers of two or more may be possible for other configurations. The clock signal may also be referred to herein as a data strobe signal.

During an embodiment of semi receiver side write training, the clock signal CLK and the data signals DQ may be sent to all of the receiving circuits104, such that write training is performed in parallel. During normal operation, the transmitting circuit102may send user (as DQ signals) and the CLK to a selected receiving circuit104, such that the user data may be stored in non-volatile memory cells.

From the perspective of the transmitting circuit102, the clock signal CLK is an output clock signal, and the data signals DQ are output data signals in that they are the clock and data signals that the transmitting circuit102outputs to the receiving circuit104. From the perspective of the receiving circuit104, the clock signal CLK is an input clock signal, and the data signals DQ are input data signals in that they are the clock and data signals that the receiving circuit104receives from the transmitting circuit102.

The communications bus106includes data lines108(1) to108(N) between the transmitting circuit102and the receiving circuit104. The receiving circuit104has data contacts112(1) to112(N), which are in physical and electrical contact with the respective data lines108(1) to108(N). The data contacts112(1) to112(N) could be pins, pads, etc. The transmitting circuit102is configured to send the data signals DQ(1) to DQ(N) simultaneously and/or in parallel over the data lines108(1) to108(N) to the receiving circuit104. Otherwise stated, the receiving circuit104is configured to receive the data signals DQ(1) to DQ(N) simultaneously and/or in parallel from over the data lines108(1) to108(N).

In addition, the communications bus106includes one or more clock lines110between the transmitting circuit102and the receiving circuit104. The clock line(s) may also be referred to herein as a data strobe line. The receiving circuits each have one or more clock input contacts114, which is/are in physical and electrical contact with the respective one or more clock lines110. The clock input contact(s) could be pins, pads, etc. The input clock signal CLK may include a single-ended clock signal or a pair of complementary clock signals (e.g., CLK and CLKB). Where the input clock signal CLK is a single-ended clock signal, the one or more clock lines110may include a single clock line. Where the input clock signal CLK is a pair of complementary clock signals CLK, CLKB, the one or more clock lines110may include two clock lines. The transmitting circuit102may be configured to transmit each clock signal CLK, CLKB of the complementary pair over a respective one of the two clock lines110. Each receiving circuit104is configured to receive the input clock signal CLK—either as a single-ended clock signal or as a pair of complementary clock signals—simultaneously and/or in parallel with the input of data signals DQ(1) to DQ(N).

The transmitting circuit102and the receiving circuits104form a source synchronous system. A source synchronous system is a system in which a transmitting (or source) circuit sends a data signal along with a clock signal to a receiving (or destination) circuit in order for the receiving circuit to use the clock signal to identify the data values of the data signal.

The transmitting circuit has a delay controller120, which is configured to control a delay between CLK and each respective data signal DQ(1)-DQ(N). Moreover, the delays can be independently controlled for each receiving circuit104(1)-104(p). In an embodiment of write training, all of the receiving circuits104(1)-104(p) are trained in parallel, which provides for efficient write training. The delay controller120scans through a number of delays during write training. That is, the delay controller120sets the delays to a certain value, and then sends test data to the receiving circuits104. Then, the delay controller120sets the delays to another value, and then again sends the test data to the receiving circuits104. This process of using different delays is repeated for a number of delays, which may be referred to herein as “scanning delay values.”

Each receiving circuit104has a data receiver130, which is configured to receive the data signals. Briefly, the data receiver130may contain on-die termination (ODT), a data receiver, and a data latch for each data path. The purpose of the data receiver130is thus to identity the data in the data signal for each respective data line.

The data compare logic140in the receiving circuits104is used during an embodiment of semi-receiver side write training. The purpose of the data comparison is to compare the data that is identified by the data receiver130with expected data during write training. Thus, the data compare logic140determines whether the data was validly received. By “validly received” it is meant that the data that is identified by the data receiver130matches the expected data.

After all of the delays have been scanned during an embodiment of write training, each receiving circuit104reports to the transmitting circuit102the delays for which data was validly received. In one embodiment, data eye information is reported.FIG.2will be referred to illustrate reporting information during write training.FIG.2shows a data eye diagram210for a data path associated with one of the data lines108. A data path includes a data line used to transmit a data signal, as well as circuitry inside of the transmitting circuit102and the receiving circuit104that process that data signal.

A data valid window220is depicted inFIG.2. The arrows230below the data eye diagram210correspond to different delays between CLK and DQ (for one data path). Arrows labeled with a “P” indicate “pass” or that data was validly received for that delay. Arrows labeled with a “F” indicate “fail” or that data was not validly received for that delay. The shortest delay is on the left, with the delays getting progressively longer moving to the right. Hence, there is a range of delays for which data was validly received. The range includes a shortest delay230afor which data was validly received, and a longest delay230bfor which data was validly received. In one embodiment, the receiving circuit104reports the shortest delay230aand the longest delay230bto the transmitting circuit102.

Returning again to the discussion ofFIG.1, the transmitting circuit102sets delay values for each data line108based on the shortest delay230aand the longest delay230b. This is done separately for each receiving circuit104(1)-104(p). In one embodiment, the transmitting circuit102sets delay taps. For example, the transmitting circuit102may set a delay tap for each data line108(1)-108(n) for each receiving circuit104(1)-104(p).

FIG.3is a block diagram of one embodiment of delay controller of the transmitting circuit102. The delay controller120has an output circuit301, which receives a number of data signals DQ(1)-DQ(N), as well as a clock signal CLK. The output circuit301includes configurable data delay circuits302(1)-302(n), which are each able to provide a configurable amount of delay for each data signal DQ(1)-DQ(N). Since different data signals DQ may have different amounts of skew relative to the clock signal CLK, the delay controller120is configured to independently control or adjust the delay of each of the data signals DQ. The output circuit301also includes a configurable clock delay circuit304which is able to provide a configurable amount of delay for the clock signal CLK. The delay controller120has data delay control circuit320, which outputs delay control signals DC_DQ(1) to DC_DQ(N) to control the delays of the configurable data delay circuits302(1)-302(n). The configurable data delay circuits302(1)-302(n) may also be referred to herein as delay taps.

The delay controller120has clock delay control circuit318, which outputs clock delay control signal DC_C to control the delay of the configurable clock delay circuit304. The delay control signals DC_DQ and DC_C may be analog signals or digital signals. For configurations in which the delay control signals DC are digital signals, the delay control signals DC may be digital codes. Each digital code may represent a p-bit binary number, where p is the number of digits of the p-bit binary number, and where each digit can be a logic 0 value or a logic 1 value. The given configurable delay circuit may respond to the digital code by delaying its respective data or clock signal by a delay amount that corresponds to the current value of the p-bit number represented by the digital code. Briefly, the delayed signals DQ(1)_d to DQ(N)_d from the output circuit301sent over the data lines108(1)-108(n). Not depicted inFIG.3are elements such as output drivers.

The delay value storage330stores delay values. In some embodiments, a delay value is stored for each configurable data delay circuit302for each receiving circuit104. For example, the delay value storage330stores separate delay values for delay circuit302(1) for each receiving circuit104. Hence, when the transmitting circuit102is sending data to a given receiver circuit104during normal operation, the delay controller120selects the appropriate delay for the selected receiving circuit104. During embodiments of semi receiver side write training, the delay values are calibrated and stored in the delay value storage330. The delay value storage330may also store delay values for the configurable clock delay circuit304. In one embodiment, a clock delay value is stored for each receiving circuit104.

The precision of the delays provided by the delay controller120may be significantly greater than would typically be possible if delay circuitry were to be implemented on the receiving circuit (e.g., NAND memory die). One reason for this is that different semiconductor fabrication techniques may be used for the semiconductor die that contains the transmitting circuit102and the semiconductor die that contains the receiving circuit104. For example, the semiconductor fabrication techniques used to form a semiconductor die that contains the receiving circuit104may be tailored to form high density memory structures, such as three-dimensional NAND memory arrays. It can be difficult to fabricate high precision delay circuitry when using such semiconductor fabrication techniques. Therefore, embodiments in which the delay controller120resides on a semiconductor die that contains, for example, a memory controller, can have higher precision in the delays. Higher precision in the delays becomes more important as data transmission across the communication bus106increases.

The delay controller120may comprise hardware, firmware (or software), or a combination of hardware and firmware (or software). For example, the delay controller120may include or be a component of an integrated circuit (IC), such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA), a circuit, a digital logic circuit, an analog circuit, a combination of discrete circuits, gates, or any other type of hardware or combination thereof. In addition, or alternatively, a delay controller120may include memory hardware that comprises instructions executable with a processor or processor circuitry to implement one or more of the features of the delay controller.

FIG.4depicts one embodiment of data receiver130and data compare logic140of a receiving circuit104. The receiving circuit104has separate data receiver130for each data path. For example, there is a separate data receiver130for each input data signal D1(1) to DQ(N). There may also be a separate data compare logic140for each data path.

The data receiver130has on-die termination (ODT)420connected to the data contact112. The ODT420includes one or more termination resistors for impedance matching to the data line108to which the ODT420is connected. In one embodiments, the ODT420includes center tap termination. In one embodiment, the ODT420includes low voltage termination logic.

The data receiver130has a data buffer402that has one input connected to the data contact112(as well as ODT420) and another input that receives a reference voltage (Vref). The data receiver130compares the data signal with Vref and outputs a result based on the comparison. For example, if the magnitude of the data signal is greater than Vref, then the data buffer402outputs a high magnitude voltage, and if the magnitude of the data signal is less than Vref, then the data buffer402outputs a low magnitude voltage. The output of the data buffer402is provided to sampling circuit404.

The sampling circuit404performs sampling actions to identify data values of data carried by the input data signal DQ. As used herein, a sampling action is an action performed to determine, identify, detect, capture, obtain, or latch onto, a level or magnitude of a signal at a given point in time. A sampling circuit may include an input terminal configured to receive the data signal. In addition, a sampling circuit may output or present the level of the input signal that it identifies. The sampling circuit may do so by generating an output signal at an output terminal of the sampling circuit at a level that indicates or corresponds to the level of the input signal. Accordingly, a sampling circuit samples an input signal, samples a level of the input signal, and outputs an output signal at a level indicating the level of the input signal.

In addition, a sampling circuit performs sampling actions in response to detecting a transitions in a clock (e.g., DQS). The clock transition may be a rising transition or a falling transition, although in some embodiments, sampling transitions may include both rising transitions and falling transitions. Each time a sampling circuit detects a clock transition, the sampling circuit samples the input signal. The input signal that a sampling circuit samples is referred to as its input data signal, and the output signal that a sampling circuit generates and outputs in response to performing sampling actions on the input signal is referred to as its output data signal.

An example sampling circuit is a flip flop, such a D flip flop for example. The sampling circuit404includes a data input terminal or node D, a data output terminal or node Q, and a clock input terminal (identified by the triangle inFIG.4). The data input terminal D is configured to receive an input data signal DIN, which the sampling circuit404is configured to sample. The clock input terminal is configured to receive a clock signal CLK of which the sampling circuit404is configured to detect sampling transitions. The data output terminal Q is configured to output an output data signal DOUT at levels and at times based on the levels of the input data signal DIN and the sampling transitions of the clock signal CLK. In particular, the sampling circuit404is configured to detect when each of the sampling transitions of the clock signal CLK occur. When the sampling circuit404detects that a sampling transition occurs, the sampling circuit404samples the level of the input data signal DIN at the data input terminal D, and generates the output data signal DOUT at the level of the input data signal DIN. The sampling circuit404maintains or holds the output data signal DOUT at the data output terminal Q at the level it identified until it detects the next sampling transition of the clock signal CLK. Upon detecting the next sampling transition of the clock signal CLK, the sampling circuit404will again identify the level of the input data signal DIN at the data input terminal D, and generate the output data signal DOUT at the level of the input data signal DIN in response to the next sampling transition. The sampling circuit404may continue to operate in this manner as it continues to receive additional data pulses of the input data signal DIN and detect sampling transitions of the clock signal CLK.

The data compare logic140will now be discussed. The data compare logic140is used during embodiments of semi receiver side write training to compare data that was identified by the sampling circuit404with expected data. The expected data may be provided ahead of time by the transmitting circuit102and stored in the pre-fixed patterns410. Thus, pre-fixed patterns410is non-transitory storage, and could include volatile memory or non-volatile memory. In some embodiments, the transmitting circuit102provides a seed pattern, from which the receiving circuit104generates the pre-fixed patterns.

The register array406is used to store data that was identified by the sampling circuit404. Hence, the register array406is non-transitory storage, and could include volatile memory or non-volatile memory. The compare logic408compares the data in the register array with the appropriate pre-fixed patterns410to determine whether the data was validly received. For example, the compare logic408determines whether the data signal is in the data valid window (seeFIG.2). That is, if the data in the register array406matches the pre-fixed patterns410, then the data was validly received. In some embodiments, the compare logic408includes an XOR logic gate circuit to identify any sampling errors. The XOR gate may compare data from the register array406with data from the pre-fixed patterns410. A sampling error refers to a case in which the sampling circuit404failed to properly identify the data in the data signal.

The address register416is used to store the delay values for which the data was validly received. Thus, in this context, an address corresponds to a delay value. For example, there might be 128 different delays used during the writing training, with each delay corresponding to a unique address. In one embodiment, the address register416is used to store the shortest delay for which the data was validly received and the longest delay for which the data was validly received (which may also be referred to as a data valid window). In one embodiment, this is implemented by storing two addresses. That is, the lowest address and the highest address for which the data was validly received may be stored in the address register416.

The divider412is used to divide the data strobe signal (DQS). The divider412provides the divided clock to a counter414. The counter414keeps track of the delays (or addresses). During the write training there will be a certain pre-determined number of DQS cycles for each delay. The divider412is configured to cause the counter414to increment once each time that the delay is changed. For example, if there are 2048 DQS cycles for each delay value, the divider412may divide DQS by 2048. In this manner, the counter414keeps track of what delay is being tested. As noted above, these different delays may also be referred to herein as addresses. In an embodiment, the receiver circuit104will report to the transmitting circuit102the lowest address and the highest address for which the data was validly received. This information may be reported for each data line108(1)-108(n).

The data receiver130and data compare logic140may each comprise hardware, firmware (or software), or a combination of hardware and firmware (or software). For example, data receiver130and data compare logic140may include or be a component of an integrated circuit (IC), such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA), a circuit, a digital logic circuit, an analog circuit, a combination of discrete circuits, gates, or any other type of hardware or combination thereof.

In general, a signal, such as the input clock signal CLK and the input data signals DQ may be at a level at a given point in time. As used herein, a level of a signal is a magnitude value, such as a voltage magnitude value or a current magnitude value. In some cases, the signal may be referred to as being at a high level or at a low level, transitioning between a high level and a low level, or transitioning between a low level and a high level. A high level of a signal may be a single high level, a level that is within a set or range of high levels, a maximum high level or a minimum high level of a set or range of high levels, or an average high level of a set or range of high levels. Similarly, a low level of a signal may be a single low level, a level that is within a set or range of low levels, a maximum low level or a minimum low level of a set or range of low levels, or an average low level of a set or range of low levels.

With reference toFIG.5A, a high level of a signal is a level that is at or above a minimum high level VH_MIN, and a low level of the signal is a level that is at or below a maximum low level VL_MAX. The minimum high level VH_MINand the maximum low level VL_MINmay be predetermined levels or values, and in particular example configurations, predetermined levels or values specified as part of a swing requirement with which the transmitting circuit102is configured to comply when transmitting the signal. A signal that transitions according to and/or in compliance with the swing requirement transitions to a high level that is at or above the minimum high level VH_MINof the swing requirement, and transitions to a low level that is at or below the maximum low level VL_MAXof the swing requirement.

In general, a signal performs transitions between its high level and its low level. A given transition of a signal may be one of two transition types, including a rising transition and a falling transition. A signal performs a rising transition when the signal transitions from its low level to its high level, and performs a falling transition when the signal transitions from its high level to its low level.

A portion of a magnitude waveform of a signal over a transition is referred to as an edge. In particular, a portion of the magnitude waveform over a rising transition is a rising edge and a portion of the magnitude waveform over a falling transition is a falling edge.

Also, a clock signal, such as the input clock signal CLK, is a signal that has repetitive cycles occurring over successive periods T. Within a cycle, one of the portions is at a high level and the other portion is at a low level. Accordingly, the portions may be defined by consecutive rising and falling transitions or edges of the clock signal. For example, a given rising edge or a given falling edge may define or mark a boundary when one portion ends and a next portion, either of the same cycle or of a next cycle, begins.

In addition, a clock signal may include clock pulses that are formed or defined by the rising and falling edges of the clock signal. In particular example configurations, the clock pulses of a clock signal correspond to the high level of the clock signal, in that each clock pulse is defined by a rising edge followed by a period where the clock signal is at its high level, and then followed by a falling edge. A pulse width of a given clock pulse is a time duration extending from a time that the magnitude of the rising edge of the clock pulse is at or rises to a predetermined level (e.g., 50% of the high level) to a time that the magnitude of the falling edge of the clock pulse is at or falls to the predetermined level. The clock pulses of the clock signal may occur according to the frequency of the clock signal.

Additionally, a data signal is a signal that carries and/or includes data. The data carried by and/or included in a data signal includes a sequence of bits, where each bit includes or has a single-bit logic value of “1” or “0”. The data signal may include a series or sequence of data pulses corresponding to a bit sequence of the data. Each data pulse may be at a level that indicates a data value, otherwise referred to as a logic level or a logic value. In addition, each data value is represented by a binary number or a binary value that includes one or more digits corresponding to and/or representing the one or more bits of the bit sequence. A duration of a data pulse is an amount of time that the level of the data pulse indicates the data value that the data pulse represents.

FIG.5Bis a schematic diagram illustrating setup time and hold time requirements of the sampling circuit404. A sampling transition of the clock signal CLK is shown as occurring at a clock event time tce. An occurrence of a sampling transition of the clock signal CLK may be referred to as a clock event. When the sampling circuit404detects a sampling transition, it detects a clock event. A time duration from a first time t1to the clock event time tce denotes the setup time tDS, and a time duration from the clock event time tce to a second time denotes the hold time tDH. In order to meet the setup and hold requirements of the sampling circuit404, the level of a data pulse of the input data signal DIN should be stable from the first time t1to the second time t2. A setup violation occurs when the level of input data signal DIN is unstable (it is still changing) after the first time t1occurs. In other words, a setup violation occurs when the actual amount of time that the level of the input data signal DIN is stable before occurrence of the sampling transition at the clock event time tce is less than the amount of the setup time tDS. In addition, a hold violation occurs when the level of the input data signal DIN is unstable (it changes) before the second time t2. In other words, a hold violation occurs when the actual amount of time that the level of the input data signal DIN is stable after occurrence of the sampling transition at the clock event time tce is less than the amount of the hold time tDH.

For a data pulse of the input data signal DIN, at least a portion of the duration that a level of the data pulse is stable—e.g., at least a portion of the duration that the data pulse is at the high level or at the low level—defines a data valid window TDVW. A data valid window TDVWis a time period or duration over which a given data pulse occurs during which a sampling circuit is to detect a sampling transition of the clock signal in order to avoid a setup violation and a hold violation. If the sampling transition occurs before the start of the data valid window TDVW, then a setup violation occurs—either because the sampling transition occurred before the starting transition of the data pulse, or because the sampling transition occurred too close to after the starting transition that the actual amount of time that the level of the data pulse is stable before occurrence of the sampling transition is less than the setup time tDS. In addition, if the sampling transition occurs after the end of the data valid window TDVW, then a hold violation occurs—either because the sampling transition occurred after the ending transition of the data pulse or occurred too close to before the ending transition that the actual amount of time that the level of the data pulse is stable after occurrence of the sampling transition is less than the hold time tDH.

Ideally, the sampling circuit404receives the clock signal CLK and the input data signal DIN relative to each other such that the sampling circuit404reliably or accurately samples the level of each data pulse in order to correctly identify the data value that each data pulse represents. Configuring the sampling circuit404to sample each data pulse in the middle or at a middle point of the duration of each pulse may maximize the chances of this ideal situation occurring. The ideal time at which to sample a data pulse is referred to as a target sampling time of the data pulse. Ideally, the sampling circuit404identifies sampling transitions in the middle of the durations of the data pulses and/or at the target sampling times of the data pulses. Accordingly, a given sampling transition is in a target sampling position when the sampling transition occurs at the target sampling time of its associated data pulse.

FIG.5Cshows a data pulse of the input data signal DIN and a pulse of the clock signal CLK, illustrating the ideal case where a sampling transition of the clock pulse is in the target sampling position. InFIG.5C, a starting transition of the data pulse occurs at a first time t1, and an ending transition of the data pulse occurs at a second time t2. A target sampling time tt of the data pulse occurs in the middle between the first time t1and the second time t2. Accordingly, a first time period T1extending from the first time t1to the target sampling time tt is the same as or equal to a second time period T2extending from the target sampling time tt to the second time t2. Additionally, the sampling transition associated with the data pulse is the rising transition of the clock pulse. The sampling transition occurs at a sampling time ts. InFIG.5B, for the ideal case, the sampling transition occurs at the target sampling time—i.e., the sampling time ts and the target sampling time tt are the same.

In actuality, when the transmitting circuit102sends the data signals DQ and the clock signal CLK to the receiving circuit104, the sampling circuitry of the receiving circuit104may not receive the clock pulses in their respective target sampling positions. For a given sampling circuit that samples data pulses of an input data signal in response to sampling transitions of a clock signal, where the sampling transitions occur at times different than the target sampling times tt, the input data signal and the clock signal have skew between them. In general, as used herein, skew between a clock signal and a data signal is a deviation of a sampling transition of the clock signal from a target sampling position to sample a data pulse of the data signal. In addition, with respect to sampling times, skew between a clock signal and a data signal is a deviation of a sampling time ts from a target sampling time tt to sample a data pulse of a data signal. For a given pair of clock and data signals, where the clock signal performs sampling transitions at sampling times ts that match or occur at the same times as the target sampling times tt, the clock and data signals do not have skew between them. Alternatively, where the clock signal performs sampling transitions at sampling times ts different than the target sampling times tt (i.e., before or after the target sampling times tt), the clock and data signals have skew between them. An amount of skew (or skew amount) may be quantified by the difference in time between the sampling time ts and the target sampling time tt.

Embodiments of semi-receiver side write training in non-volatile memory systems are disclosed herein.FIGS.6A,6B,7,8,9A and9Bdepict an example memory system in which embodiments may be practiced.FIG.6Ais a block diagram of one embodiment of a memory system600connected to a host system620. Memory system600can implement the technology proposed herein. Many different types of storage devices can be used with the technology proposed herein. One example storage device is a solid state device (SSD); however, other types of storage devices can also be used. Memory system600comprises a memory controller602, non-volatile memory604for storing data, and local memory (e.g. DRAM/ReRAM)606. In some embodiments, the memory controller602includes the transmitting circuit102and the memory packages604contain the receiving circuits104. Hence, the memory controller602may contain the delay controller120. The memory packages604may contain data receivers130and data comparison logic140. In some embodiments, the memory controller602includes a control circuit that is configured to perform transmitter side functionality during an embodiment of semi receiver side write training.

Memory controller602comprises a Front End Processor Circuit (FEP)610and one or more Back End Processor Circuits (BEP)612. In one embodiment, FEP610circuit is implemented on an ASIC. In one embodiment, each BEP circuit612is implemented on a separate ASIC. The ASICs for each of the BEP circuits612and the FEP circuit610are implemented on the same semiconductor such that the Controller602is manufactured as a System on a Chip (SoC). FEP610and BEP612both include their own processors. In one embodiment, FEP610and BEP612work as a master slave configuration where the FEP610is the master and each BEP612is a slave. For example, FEP circuit610implements a flash translation layer that performs memory management (e.g., garbage collection, wear leveling, etc.), logical to physical address translation, communication with the host, management of DRAM (local volatile memory) and management of the overall operation of the SSD (or other non-volatile storage device). The BEP circuit612manages memory operations in the memory packages/die at the request of FEP circuit110. For example, the BEP circuit612can carry out the read, erase and programming processes. Additionally, the BEP circuit612can perform buffer management, set specific voltage levels required by the FEP circuit610, perform error correction, control the Toggle Mode interfaces to the memory packages, etc. In one embodiment, each BEP circuit612is responsible for its own set of memory packages. Memory controller602is one example of a control circuit.

In one embodiment, non-volatile memory604comprises a plurality of memory packages. Each memory package includes one or more memory die. Therefore, memory controller602is connected to one or more non-volatile memory die. In one embodiment, each memory die in the memory packages604utilize NAND flash memory (including two dimensional NAND flash memory and/or three dimensional NAND flash memory). In other embodiments, the memory package can include other types of memory.

In some embodiments, controller602communicates with host system620via an interface630that implements NVM Express (NVMe) over PCI Express (PCIe). For working with memory system600, host system620includes a host processor622, host memory624, and a PCIe interface626connected to bus628. Host memory624is the host's physical memory, and can be DRAM, SRAM, non-volatile memory or another type of storage. Host system620is external to and separate from memory system600. In one embodiment, memory system600is embedded in host system620. Any combination of one or more of memory system600, and/or memory system600in combination with host system620may be referred to herein as an apparatus. In operation, when the host system620needs to read data from or write data to the non-volatile memory604, it will communicate with the memory controller602. If the host system620provides a logical address to which data is to be read/written, the controller can convert the logical address received from the host to a physical address in the non-volatile memory604.

FIG.6Bis a block diagram of one embodiment of FEP circuit610.FIG.6Bshows a PCIe interface650to communicate with host system620and a host processor652in communication with that PCIe interface. The host processor652can be any type of processor known in the art that is suitable for the implementation. Host processor652is in communication with a network-on-chip (NOC)654. A NOC is a communication subsystem on an integrated circuit, typically between cores in a SoC. NOC's can span synchronous and asynchronous clock domains or use unclocked asynchronous logic. NOC technology applies networking theory and methods to on-chip communications and brings notable improvements over conventional bus and crossbar interconnections. NOC improves the scalability of SoCs and the power efficiency of complex SoCs compared to other designs. The wires and the links of the NOC are shared by many signals. A high level of parallelism is achieved because all links in the NOC can operate simultaneously on different data packets. Therefore, as the complexity of integrated subsystems keep growing, a NOC provides enhanced performance (such as throughput) and scalability in comparison with previous communication architectures (e.g., dedicated point-to-point signal wires, shared buses, or segmented buses with bridges). Connected to and in communication with NOC654is the memory processor656, SRAM660and a DRAM controller662. The DRAM controller662is used to operate and communicate with the DRAM (e.g., DRAM606). SRAM660is local RAM memory used by memory processor656. Memory processor656is used to run the FEP circuit and perform the various memory operations. Also in communication with the NOC are two PCIe Interfaces664and666. In the embodiment ofFIG.6B, the SSD controller will include two BEP circuits612; therefore there are two PCIe Interfaces664/666. Each PCIe Interface communicates with one of the BEP circuits612. In other embodiments, there can be more or less than two BEP circuits612; therefore, there can be more than two PCIe Interfaces.

FIG.7is a block diagram of one embodiment of the BEP circuit612.FIG.7shows a PCIe Interface700for communicating with the FEP circuit610(e.g., communicating with one of PCIe Interfaces664and666ofFIG.6B). PCIe Interface700is in communication with two NOCs (Network-on-a-Chip)702and704. In one embodiment, the two NOCs can be combined to one large NOC. Each NOC (702/704) is connected to SRAM (730/760), a buffer (732/762), processor (720/750), and a data path controller (722/752) via an XOR engine (724/754) and an ECC engine (726/756). The ECC engines726/756are used to perform error correction, as known in the art. The XOR engines724/754are used to XOR the data so that data can be combined and stored in a manner that can be recovered in case there is a UECC failure. In an embodiment, XOR engines724/754form a bitwise XOR of different pages of data. The XOR result may be stored in a memory package604. In the event that an ECC engine726/756is unable to successfully correct all errors in a page of data that is read back from a memory package604, the stored XOR result may be accessed from the memory package604. The page of data may then be recovered based on the stored XOR result, along with the other pages of data that were used to form the XOR result.

Data path controller722is connected to an interface module for communicating via four channels with memory packages. Thus, the top NOC702is associated with an interface728for four channels for communicating with memory packages and the bottom NOC704is associated with an interface758for four additional channels for communicating with memory packages. Each interface728/758includes four Toggle Mode interfaces (TM Interface), four buffers and four schedulers. There is one scheduler, buffer and TM Interface for each of the channels. The processor can be any standard processor known in the art. The data path controllers722/752can be a processor, FPGA, microprocessor or other type of controller. The XOR engines724/754and ECC engines726/756are dedicated hardware circuits, known as hardware accelerators. In other embodiments, the XOR engines724/754and ECC engines726/756can be implemented in software. The scheduler, buffer, and TM Interfaces are hardware circuits.

Interfaces728/758, alone or in combination, may be referred to as a memory interface configured to be connected to non-volatile memory (e.g., memory package604). A combination of one or more of processor720/750, data path controller722/752, XOR724/754, ECC726/756may be referred to herein as a processor circuit. The buffer732/762, SRAM730/760, and/or NOCs702/704may also be considered to be a part of the processor circuit.

FIG.8is a block diagram of one embodiment of a memory package604that includes a plurality of memory die800connected to a memory bus (data lines and chip enable lines)106. The memory bus106connects to a Toggle Mode Interface796for communicating with the TM Interface of a BEP circuit612(see e.g.,FIG.7). In some embodiments, the memory package can include a small controller connected to the memory bus and the TM Interface. The memory package can have one or more memory die. In one embodiment, each memory package includes eight or 16 memory dies; however, other numbers of memory dies can also be implemented. The technology described herein is not limited to any particular number of memory dies. In some embodiments, each memory die800is a receiving circuit104that contains data receivers130and data comparison logic140. In some embodiments, write training is performed in parallel on all of the memory dies800, which provides for fast write training. In some embodiments, the TM interface796contains the delay controller120. However, some of all of the delay controller120could be located in a different part of the memory controller602.

FIG.9Ais a functional block diagram of one embodiment of a memory die800. Each of the one or more memory die800ofFIG.8can be implemented as memory die800ofFIG.9A. The components depicted inFIG.9Aare electrical circuits. In one embodiment, each memory die800includes a memory structure926, control circuitry910, and read/write circuits928, all of which are electrical circuits. Memory structure926is addressable by word lines via a row decoder924and by bit lines via a column decoder932. The read/write circuits928include multiple sense blocks950including SB1, SB2, . . . , SBp (sensing circuitry) and allow a page (or multiple pages) of data in multiple memory cells to be read or programmed in parallel. In one embodiment, each sense block include a sense amplifier and a set of latches connected to the bit line. The latches store data to be written and/or data that has been read. The sense blocks include bit line drivers.

Commands and data are transferred between the controller602and the memory die800via memory controller interface915. The memory controller interface915may also be referred to herein as a communication interface. Examples of memory controller interface915include a Toggle Mode Interface and an Open NAND Flash Interface (ONFI). Other I/O interfaces can also be used.

Control circuitry910cooperates with the read/write circuits928to perform memory operations (e.g., write, read, erase, and others) on memory structure926. In one embodiment, control circuitry910includes a state machine912, an on-chip address decoder914, a power control module916, and a memory controller interface915. State machine912provides die-level control of memory operations. In one embodiment, state machine912is programmable by software. In other embodiments, state machine912does not use software and is completely implemented in hardware (e.g., electrical circuits). In some embodiments, state machine912can be replaced by a microcontroller or microprocessor. In one embodiment, control circuitry910includes buffers such as registers, ROM fuses and other storage devices for storing default values such as base voltages and other parameters. The default values and other parameters could be stored in a region of the memory structure926.

The on-chip address decoder914provides an address interface between addresses used by controller602to the hardware address used by the decoders924and932. Power control module916controls the power and voltages supplied to the word lines and bit lines during memory operations. Power control module916may include charge pumps for creating voltages.

Memory controller interface915is an electrical interface for communicating with memory controller602. For example, memory controller interface915may implement a Toggle Mode Interface that connects to the Toggle Mode interfaces of memory interface228/258for memory controller602. In one embodiment, memory controller interface915includes a set of input and/or output (I/O) pins that connect to communication channel106(also refers to herein as a data bus). In one embodiment, communication channel106connects to the memory controller602as part of the Toggle Mode Interface. The data receiver130and data compare logic140have been discussed above.

For purposes of this document, control circuitry910, alone or in combination with read/write circuits928and decoders924/932, comprise a control circuit configured to be connected to memory structure926. This control circuit is an electrical circuit that performs at least some of the functions described below in the flow charts (such as receiver side functions of semi receiver side write training). In some embodiments, the control circuitry910and memory controller602together perform the functions described below in the flow charts. For example, control circuitry910may implement receiver side functions of semi receiver side write training, with the memory controller602implementing transmitter side functions of semi receiver side write training.

In one embodiment, memory structure926comprises a monolithic three-dimensional memory array of non-volatile memory cells in which multiple memory levels are formed above a single substrate, such as a wafer. The memory structure may comprise any type of non-volatile memory that is monolithically formed in one or more physical levels of arrays of memory cells having an active area disposed above a silicon (or other type of) substrate. In one example, the non-volatile memory cells of memory structure926comprise vertical NAND strings with charge-trapping material such as described, for example, in U.S. Pat. No. 9,721,662, incorporated herein by reference in its entirety. In another embodiment, memory structure926comprises a two dimensional memory array of non-volatile memory cells. In one example, the non-volatile memory cells are NAND flash memory cells utilizing floating gates such as described, for example, in U.S. Pat. No. 9,082,502, incorporated herein by reference in its entirety. Other types of memory cells (e.g., NOR-type flash memory) can also be used.

The exact type of memory array architecture or memory cell included in memory structure926is not limited to the examples above. Many different types of memory array architectures or memory cell technologies can be used to form memory structure926. No particular non-volatile memory technology is required for purposes of the new claimed embodiments proposed herein. Other examples of suitable technologies for memory cells of the memory structure926include ReRAM memories, magnetoresistive memory (e.g., MRAM, Spin Transfer Torque MRAM, Spin Orbit Torque MRAM), phase change memory (e.g., PCM), and the like. Examples of suitable technologies for architectures of memory structure926include two dimensional arrays, three dimensional arrays, cross-point arrays, stacked two dimensional arrays, vertical bit line arrays, and the like.

Magnetoresistive memory (MRAM) stores data by magnetic storage elements. The elements are formed from two ferromagnetic plates, each of which can hold a magnetization, separated by a thin insulating layer. One of the two plates is a permanent magnet set to a particular polarity; the other plate's magnetization can be changed to match that of an external field to store memory. A storage device is built from a grid of such memory cells. In one embodiment for programming, each memory cell lies between a pair of write lines arranged at right angles to each other, parallel to the cell, one above and one below the cell. When current is passed through them, an induced magnetic field is created.

Phase change memory (PCM) exploits the unique behavior of chalcogenide glass. One embodiment uses a Ge2Sb2Te5alloy to achieve phase changes by electrically heating the phase change material. The doses of programming are electrical pulses of different amplitude and/or length resulting in different resistance values of the phase change material.

FIG.9Bdepicts a functional block diagram of one embodiment of an integrated memory assembly904. The integrated memory assembly904may be used in a memory package604in memory system600. In one embodiment, the integrated memory assembly904includes two types of semiconductor die (or more succinctly, “die”). Memory structure die906includes include memory structure926. Memory structure926may contain non-volatile memory cells. Control die908includes control circuitry910. In some embodiments, the memory structure die906and the control die908are bonded together. The control circuitry includes state machine912, an address decoder914, a power control circuit916, memory controller interface915, data receiver130, and data comparison logic140. The control circuitry also includes read/write circuits928. In another embodiment, a portion of the read/write circuits928are located on control die908, and a portion of the read/write circuits928are located on memory structure die906.

Any subset of components in the control circuitry910can be considered a control circuit. The control circuit can include hardware only or a combination of hardware and software (including firmware). For example, a controller programmed by firmware is one example of a control circuit. The control circuit can include a processor, PGA (Programmable Gate Array, FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit), integrated circuit or other type of circuit.

Pathways952are pathways between one or more components in the control circuitry910and the memory structure on memory structure die906. A pathway may be used to provide or receive a signal (e.g., voltage, current). A pathway includes an electrically conductive path. A pathway may include one or more of, but is not limited to, a bond pad, metal interconnect, via, transistor, electrically conducting material and other material that may transfer or carry an electrical signal.

In one embodiment, integrated memory assembly904includes a set of input and/or output (I/O) pins that connect to communication channel106(also refers to herein as a data bus). In one embodiment, communication channel106connects the memory controller602directly to control die908.

FIG.10is a flowchart of one embodiment of a process1000of semi receiver side write training in a non-volatile memory system. In one embodiment, the process1000is performed by the transmitting circuit102and the receiving circuit104ofFIG.1. In one embodiment, the transmitting circuit102is included in memory controller (e.g., controller602). In one embodiment, the receiving circuit104resides on a memory die800. In one embodiment, the receiving circuit104resides on a control die908, which is configured to be connected to a memory structure die906.

Step1002includes the transmitting circuit102setting the delay taps to an initial value. In one embodiment, the data delay control circuit320in the delay controller120issues delay signals DC_DQ(1)-DC_DQ(N) to the respective configurable data delay circuits302(1)-302(N) in order to set the delay taps.

Step1004includes the transmitting circuit102selecting all receiving circuits104(1)-104(p). Step1004may also include the transmitting circuit102providing a test pattern to the receiving circuits104.FIG.11Adepicts an example of information the transmitting circuit102may send over the data lines108to implement steps1004and1006. Hence, the information inFIG.11Amay be sent on data lines108(1)-108(N). The transmitting circuit102may issue a semi receiver side writing training command1102. The example command1102is “6X” in hexadecimal format. The “X” refers to an integer. The “All Select Command”1104selects all of the receiving circuits104. In an embodiment, the All Select Command1104selects all Logical Unit Numbers (LUNs). In an embodiment, the LUNs refer to the different memory die800. In an embodiment, the LUNs refer to the different memory structure die906. In an embodiment, the LUNs refer to the different control die908. The inverse set1106, 1stpattern1108, and 2ndpattern1110are used to provide a pre-fixed pattern.

Step1006includes the transmitting circuit102writing test data on the data lines108.FIG.11Bshows an example of writing test data on the data lines. In an embodiment, the test data1120includes n+1 bits of test data for each data line108. The notation of “Address #0” indicates that this is for the initial delay value.

Step1008includes each receiving circuit104comparing the test data to expected data. With reference toFIG.4, the data compare logic140compares the data received by the sampling circuit404with the pre-fixed patterns410. The pre-fixed patterns410are based on the pre-fixed pattern inFIG.11A. The data compare logic140determines whether this a pass or a fail for this delay value. In one embodiment, all of the test data must match the expected values for a pass.

Step1010includes each receiving circuit104saving a result for this delay. With respect toFIG.4, the result is stored in the address register416. In one embodiment, a pass or fail result is stored for each delay (or for each address). Moreover, a pass/fail result may be stored for each data line108.

Step1012is a determination of whether all delays have been tested. If not, then in step1014the transmitting circuit102sets the delay taps302to the next value. In one embodiment, the data delay control circuit320in the delay controller120issues new delay signals DC_DQ(1)-DC_DQ(N) to the respective configurable data delay circuits302(1)-302(N). Then steps1006-1012are repeated.FIG.11Cdepicts test data1130again being sent on the data lines108after a “DQS timing change”. However, the address is now “address #1”, which indicates that the next delay value is being used. Each receiving circuit104may store a pass/fail result for this delay (for each data line108). After all delays have been tested (step1012is yes), step1016is performed.

Step1016includes the transmitting circuit102selecting a receiving circuit104. Step1018includes the selected receiving circuit104sending test results to the transmitting circuit102.FIG.11Dshows an example of information that may be exchanged on the data lines108in steps1016and1018. The transmitting circuit102issues a provide semi-receiver side write training results command1132. The command1132is “6X” hexadecimal in this example, where “X” is an integer. The transmitting circuit102selects one of the receiving circuits104by specifying the LUN1134. The selected receiving circuit104sends test results by sending the pass start1136and the pass end1138. Referring back toFIG.2, it is expected that normally there will be some fails, followed by a number of passes, and then some more fails. Hence, there is a pass start230aand a pass end230b.

Step1020includes the transmitting circuit setting DQ/DQS timing for this receiving circuit104. In one embodiment, the transmitting circuit102stores delay values in the delay value storage330. When the transmitting circuit102sends normal data to the receiving circuit104these delay values are used in the configurable data delay circuits302(1)-302(N). The normal data refers to, for example, user data to be written to the memory structure926as part of a program command.

Step1022includes a determination of whether there are more receiving circuits104for which the test results are needed. If so, steps1016-1020are repeated. In this manner each receiving circuit104is able to report its test results to the transmitting circuit102. Moreover, the transmitting circuit102may store separate delay values for each receiving circuit104in the delay value storage330.

In some embodiments, a reference voltage for the data buffers402is calibrated as a part of the overall semi-receiver side write training.FIG.12is a flowchart of one embodiment of a process1200for calibrating reference voltages for data buffers402in semi-receiver side write training. There are many ways in which the reference voltages for the data buffers402may be calibrated. Hence, many variations of process1200are possible.

Step1202includes all receiving circuits104setting an initial value for the reference voltage (Vref) for the data buffers402. Next, process1000is performed. Recall that process1000is an embodiment of semi receiver side write training. Hence, process1000is performed with this initial value for Vref. After performing process1000, a determination is made in step1204whether this is an additional Vref to test. If so, the value for Vref is changed in step1206. Then, process1000is performed again with this value of Vref for the data buffers402. After all value for Vref have been tested, step1208is performed.

Step1208includes each receiving circuit104setting its own value(s) for Vref. In one embodiment, a single value is used for all of the data buffers402. In one embodiment, different values of Vref can be used for different data buffers402on a receiving circuit104. Recall that in step1018of process1000, the receiving circuit104reports the test results to the transmitting circuit102. Recall that the test results may include the shortest delay for which data was validly received and the longest delay for which date is validly received. These passing delay values may be different for the different values of Vref. Hence, the receiving circuit104may inform the transmitting circuit102of the passing delay values for the Vref that is selected in step1208.

Step1210includes the transmitting circuit102setting final DQ/DQS timings for the receiving circuits104. These final DQ/DQS timing may therefore take into account the Vref that was established for each receiving circuit104in step1208.

In view of the foregoing, it can be seen that a first embodiment includes, an apparatus comprising a first semiconductor die comprising a first control circuit configured to connect to a memory structure comprising non-volatile memory cells. The apparatus comprises a communication bus comprising a plurality of data lines and a data strobe line. The apparatus comprises a second semiconductor die connected to the first semiconductor die by the communication bus. The second semiconductor die comprises a second control circuit configured to scan a data strobe signal through a set of delays while sending test data on the data lines to the first semiconductor die. The first control circuit is configured to: determine, for each of the data lines, passing delay values for which the test data is validly received; and report the passing delay values for each of the data lines to the second control circuit. The second control circuit is configured to control the delay between the data strobe signal sent on the data strobe line and user data sent on each of the data lines based on the passing delay values for each of the data lines.

In a second embodiment and in furtherance to the first embodiment, the second semiconductor die comprises a delay tap for each of the data lines. Each delay tap is configured to provide a configurable delay between the data strobe signal and user data sent on the respective data line.

In a third embodiment and in furtherance to the first or second embodiments, the first control circuit is configured to: compare the test data for each delay value to expected data to determine whether the test data is validly received; determine, for each of the data lines, a shortest delay for which the test data is validly received and a longest delay for which the test data is validly received; and report the shortest delay and the longest delay to the second control circuit.

In a fourth embodiment and in furtherance to the third embodiment, the second control circuit is configured to set the delay value for each of the data lines for the first semiconductor die based on the shortest delay and the longest delay for the respective data line.

In a fifth embodiment and in furtherance to any of the first to fourth embodiments, the apparatus further comprises additional semiconductor dies each comprising a first control circuit configured to connect to a memory structure comprising non-volatile memory cells. The second semiconductor die is connected to the additional semiconductor dies by the communication bus. The first semiconductor die and the additional semiconductor dies are a plurality of dies. The second control circuit on the second semiconductor die is configured to: select the plurality of dies for write training in parallel; and scan through the set of delays while sending the test data on the data lines to the plurality of dies.

In a sixth embodiment and in furtherance to any of the fifth embodiment, the first control circuit of each of the additional semiconductor dies is configured to determine, for each data line, a shortest delay for which the test data is valid and a longest delay for which the test data is valid.

In a seventh embodiment and in furtherance to any of the sixth embodiment, the second control circuit on the second semiconductor die is configured to: individually select respective additional dies; and control the delay between the data strobe signal sent on the data strobe line and user data sent on each of the data lines to the individually selected additional dies based on the passing delay values for each of the data lines for the selected additional die.

In an eighth embodiment and in furtherance to any of the first to seventh embodiments, the memory structure resides on the first semiconductor die.

In a ninth embodiment and in furtherance to any of the first to eighth embodiments, the apparatus further comprises a memory structure die that comprises the non-volatile memory cells. The first semiconductor die is bonded to the memory structure die.

In a tenth embodiment and in furtherance to any of the first to ninth embodiments, the memory structure comprises a three-dimensional array of NAND memory cells.

In an eleventh embodiment and in furtherance to any of the first to tenth embodiments, the second control circuit on the second semiconductor die is a memory controller that is substantially compliant with the Open NAND Flash Interface (ONFI) specification.

One embodiment includes a method of write training in a non-volatile memory system. The method comprises: a) setting delay taps on a memory controller to initial values, wherein the delay taps control a delay between a data strobe signal and data signals sent on a communication bus between the memory controller and a plurality of semiconductor dies, each semiconductor die comprising a control circuit configured to connect to non-volatile memory cells; b) sending the data strobe signal and test data on the communication bus from the memory controller to the plurality of semiconductor dies while the delay taps have the initial values; c) determining, by the control circuit on each respective semiconductor die, data eye information for each data signal; d) repeating said a) through said c) for other delay tap values; e) reporting the data eye information from each respective semiconductor die to the memory controller; and f) sending user data from the memory controller to each respective semiconductor die over the communication bus at different times, including setting the delay taps on the memory controller for each data line based on the data eye information for the respective semiconductor die when sending the user data to the respective semiconductor die.

One embodiment includes a non-volatile memory system, comprising a plurality of semiconductor dies, a communication bus comprising a plurality of data lines and a data strobe line, and a memory controller die communicatively coupled to the plurality of semiconductor dies via the plurality of data lines and the data strobe line. Each semiconductor die comprises a control circuit configured to connect to a three-dimensional memory array of non-volatile memory cells. The memory controller die comprises a delay tap for each data line. Each delay tap is configured to provide a configurable delay between the data strobe signal and a data signal associated with the delay tap. The memory controller die is configured to provide a data strobe signal on the data strobe line when providing data signals on the corresponding plurality of data lines. The memory controller die is configured to select the plurality of semiconductor dies for write training. The write training includes the memory controller die scanning through a plurality of different values for the delay taps while sending test data on the plurality of data lines to the plurality of semiconductor dies. The control circuit of each semiconductor die is configured to determine whether the test data is validly received for each data line for each of the delays. The control circuit of each semiconductor die is configured to inform the memory controller die of the shortest delay for which data is validly received for each data line and the longest delay for which data is validly received for each data line. The memory controller die is configured to set the delay taps for each data line for each semiconductor die based on the shortest delay for which data is validly received for each data line and the longest delay for which data is validly received for each data line for the respective semiconductor die.