Data processing system having lockstep operation

A data processing system and methods for operating the same are disclosed. The method includes detecting a fault by comparing output signals from a first processing core and a second processing core, entering a safe mode based upon detecting the fault, completing transactions while in the safe mode, and determining whether the fault corresponds to a hard error. Based upon the fault corresponding to a hard error, one of processing cores is identified as a faulty core. The faulty core is inhibited from executing instructions and the other processing core is allowed to execute instructions.

BACKGROUND

Field

This disclosure relates generally to data processing systems, and more specifically, to a data processing system having lockstep operation.

Related Art

As time advances, integrated circuits continue to increase in complexity. System-on-Chip (SoC) and other multiple-core integrated circuits are being developed in order to support various applications such as automotive, industrial, and medical applications, for example. Systems designed for these applications often require significant attention to safety. Accordingly, improved safety processing techniques are desired.

DETAILED DESCRIPTION

Generally, there is provided, a data processing system and method for operating in a lockstep mode. Output signals from two processing cores of a core domain are compared during a lockstep mode. When a fault is detected, the core domain enters a safe mode and a determination is made whether the fault is caused by a soft error or a hard error. While in the safe mode, the core domain is isolated from a system bus of the data processing system after outstanding transactions are completed. The fault is analyzed by performing a memory built-in self-test (MBIST) on a cache memory coupled to a processor of each processing core and a logic built-in self-test (LBIST) on the processor of each processing core. A hard error is determined when MBIST or LBIST fails and a soft error is determined when MBIST and LBIST both pass. When a hard error is determined, fault analysis also identifies which core is a faulty core.

Based upon the fault corresponding to hard error, the faulty core is inhibited and the non-faulty core is allowed to continue operation. By inhibiting the faulty core and allowing the non-faulty core to continue executing, the data processing system can continue operation in a reduced or degraded operating mode which is desirable in a safety application. Based upon the fault corresponding to a soft error, a reset of the core domain is performed while other portions of the data processing system are allowed to operate normally. Because the reset of the core domain does not impact other portion of the data processing system, significant downtime savings can be achieved which is also desirable in a safety application.

FIG. 1illustrates, in simplified block diagram form, an exemplary data processing system100in accordance with an embodiment of the present invention. In some embodiments, data processing system100may be characterized as a system-on-a-chip (SoC). Processing system100includes system bus102, core domains104and106, controller108, memory110, and other peripherals112. Core domain104includes processing cores114and116, and lockstep control block118. Lockstep control block118, controller108, memory110, and other peripherals112are each bidirectionally coupled to system bus102by way of respective communication buses. Cores114and116are each bidirectionally coupled to system bus102by way of lockstep control block118and respective communication buses.

System bus102may include any type of bus, switch fabric, network on chip (NoC), and the like for interconnecting and communicating any type of information such as data, address, instructions, interrupts, and control. System bus102provides a communication backbone for transactions such as writes to memory, data transfers, etc., as well as communication of other information. In particular, system bus102provides a communication backbone for transactions among core domains104and106, controller108, memory110, and other peripherals112.

Core domain104include cores114and116which are coupled to system bus102by way of lockstep control block118. Processing cores114and116each include a processor (120,124) and cache memory (122,126) respectively. Processor120labeled CPU1is coupled to cache memory122labeled CACHE1, and processor124labeled CPU2is coupled to cache memory126labeled CACHE2. Test circuitry such as logic built-in self-test (LBIST) is coupled to each of CPU1and CPU2, and memory built-in self-test (MBIST) is coupled to each of CACHE1and CACHE2. CPU1and CPU2may include any type of circuits for processing, computing, etc., such as state machine, microprocessor unit (MPU), microcontroller unit (MCU), digital signal processor (DSP), and other suitable types of processing units. In this embodiment, cores114and116are virtually identical to one another having like architecture and circuitry, for example.

Processing system100may include multiple core domains like core domain104(e.g., core domain106), each core domain including processing cores like cores114and116, shared memory (not shown), lockstep control circuitry like lockstep control block108, interrupt circuitry (not shown), and other periphery. For example, core domain106may be similar to core domain104including circuitry like cores114and116, lockstep control block118, and so on. Core domain106may operate independently from core domain104such as in multi-processing or multi-threaded processing systems. In some embodiments, core domain106may have different cores, memory, and other circuitry. In some embodiments, cores114and116in core domain104may each include multiple cores like cores114and116. For example, core114may include two cores having two processors like CPU1coupled to two cache memories like CACHE1, and core116may include two cores having two processors like CPU2coupled to two cache memories like CACHE2.

Cores114and116are generally configured to execute sets of instructions in order to carry out designated tasks. In the course of executing instructions, cores114and116can generate transactions such as writes to memory, data transfers, etc. Cores114and116may be configured to operate independently in a performance mode or may be configured to operate together in a lockstep mode. Cores114and116may also be configured to operate in a degraded mode where one core is disabled and the other core operates independently. In the lockstep mode, one core may shadow the other by executing the same instructions and generating the same transactions. For example, core114labeled CORE1and core116labeled CORE2may be configured to operate in a lockstep mode (e.g., as a lockstep pair) using the lockstep control block118such that CORE2shadows CORE1, allowing outputs of each core to be compared with one another for inconsistencies. By comparing outputs of the lockstep pair, a level of safety in data processing system100can be assured because hard and soft errors are detectable. In this embodiment, when CORE2shadows CORE1, outputs of CORE2are used only for comparison and are decoupled from system bus102accordingly. The term “shadows,” as used herein, refers to executing the same instructions and generating the same transactions.

Lockstep control block118includes circuitry configured to selectively route signals between CORE1and CORE2and system bus102based on control signals provided by controller108. Lockstep control block118includes inputs and outputs coupled to provide and receive signals to and from CORE1and CORE2, system bus102, and controller108. Lockstep control block118also includes circuitry configured to compare output signals from each core of CORE1and CORE2to determine whether such output signals are consistent with one another. A set of control signals are transmitted by way of control signal lines coupled between controller108and lockstep control block118.

Controller108is coupled to system bus102and lockstep control block118. Controller108includes circuitry for processing, computing, etc., such as a state machine, processing core, and the like. Controller108is generally configured to execute instructions and provide responses to received interrupt signals in order to carry out designated tasks. Controller108is configured to provide control signals to the lockstep control block118and configured to receive fault indication signals and/or interrupt signals from the lockstep control block118.

Memory110may include any type of volatile or non-volatile memory array cells, such as static random-access memory (SRAM), dynamic random-access memory (DRAM), flash memory, and the like. Processing system100may include multiple memories like memory110or a combination of different memory types. For example, processing system100may include a flash memory in addition to an SRAM110.

Other peripherals112of processing system100may include any number of other circuits and functional hardware blocks such as accelerators, timers, counters, communications, interfaces, analog-to-digital converters, digital-to-analog converters, PLLs, and the like for example. Each of the other circuits and functional hardware blocks included in other peripherals112may be coupled to system bus102by way of respective communication buses.

FIG. 2illustrates, in simplified schematic diagram form, an exemplary lockstep control block118ofFIG. 1in accordance with an embodiment of the present invention. Lockstep control block118includes inputs labeled C1IN and C2IN for receiving signals from system bus102, inputs labeled C1OUT and C2OUT for receiving output signals from CORE1and CORE2, outputs labeled C1OUTX and C2OUTX for providing output signals to system bus102, and outputs labeled C1IN and C2INX for providing input signals to CORE1and CORE2. Lockstep control block118also includes control inputs labeled SEL1, SEL2, and LS for receiving control signals from controller108, and control outputs labeled FAULT for providing one or more fault indication signals and/or interrupt signals to controller108. In this embodiment, lockstep control block118includes multiplexer circuits202-206, delay circuits208and210, and compare unit212.

Delay circuits208and210may be configured to delay a respective signal by a predetermined fixed delay amount. The fixed delay amount may be represented by a value from 0 (zero) to N, where 0 corresponds to no delay and N corresponds to N number of clock cycle delays. In some embodiments, delay circuits208and210may be configured to delay a respective signal by a programmable delay amount by way of control signal inputs (e.g., DLYI, DLYO). For example, delay circuit208may be programmed to delay a signal by a predetermined amount (e.g., two clock cycles) based on control signal values provided by controller108. The programmable delay amount may also be represented by a value from 0 (zero) to N, where 0 corresponds to no delay and N corresponds to N number of clock cycle delays.

Multiplexer circuits202-206may each be configured to select one of multiple inputs to be routed to an output based on a value of a control signal. In this embodiment, each of multiplexer circuits202-206are configured as a two-input, one-output switch circuit. Multiplexer circuits202-206may also be referred to as selector circuits. Other multiplexer circuit configurations and arrangements may be used in other embodiments.

Multiplexer circuits202-206are coupled to receive control signals LS, SEL1, and SEL2to route selected inputs to respective multiplexer outputs. Each of control signals LS, SEL1, and SEL2may have values corresponding to operating modes of the CORE1and CORE2. For example, in a lockstep mode where CORE2shadows CORE1, control signals LS, SEL1, and SEL2may each be set to a first value allowing input signals C1IN to be routed to CORE1and CORE2by way of multiplexer202, output signals C1OUT from CORE1routed to system bus102by way of multiplexer204, and output signals C2OUT from CORE2to be inhibited by way of multiplexer206. Because the same input signals are routed to both CORE1and CORE2, output signals from both CORE1and CORE2may be compared with one another in the compare unit212.

In a performance mode where CORE1and CORE2operate independently, control signals LS and SEL2may each be set to a second value (while SEL1is set to the first value) allowing C2IN input signals to be routed to CORE2and output signals C2OUT to be routed to system bus102. In a first degraded mode, CORE1is inhibited and CORE2is operable. Control signals LS, SEL1, and SEL2may each be set to the second value allowing C21N input signals to be routed to CORE2and output signals C2OUT to be routed to system bus102while CORE1output signals C1OUT are inhibited at multiplexer204. In a second degraded mode, CORE1is operable and CORE2is inhibited. Control signals SEL1and SEL2may each be set to the first value (while LS is set to the second value) allowing output signals C1OUT to be routed to system bus102while CORE2output signals C2OUT are inhibited at multiplexer206. When in the first or second degraded mode, the level of safety in data processing system100may be reduced because hard and soft errors are not as readily detectable as they were in lockstep mode.

Compare unit212includes inputs for receiving a first group of signals (e.g., C1OUTD) and a second group of signals (e.g., C2OUT), and one or more outputs for providing one or more fault indication signals (e.g., FAULT). Compare unit212includes circuitry configured to compare the first group of signals with the second group of signals and generate a fault indication when a mismatch occurs. In general, the first and second groups of signals are expected to be identical. However, a fault (also referred to as error condition) may occur in which one of the signals in the first group is different from a corresponding signal in the second group, and thus a fault indication is generated. The fault indication may be in the form of a flag, signal, and/or interrupt request by which controller108responds with further action.

FIG. 3illustrates, in simplified timing diagram form, exemplary lockstep fault signal timing300in accordance with an embodiment of the present invention. Various signals are shown on the Y axis versus time shown on the X axis. By way of example, lockstep operation corresponding toFIGS. 1 and 2is depicted in timing diagram300. CLOCK signal waveform includes numbered clock cycles for reference. In the embodiment depicted inFIG. 3, CORE1and CORE2of core domain116are operating in a lockstep mode whereby CORE2shadows CORE1. In this embodiment, input signals C1IN are routed to CORE1. C1IN signals are delayed by one clock cycle to form C1IND signals which are routed to CORE2. Thus, CORE2operates one clock cycle later in time based on the same input signals as CORE1. Accordingly, CORE1output signals C1OUT are delayed by one clock cycle to form C1OUTD signals to be compared with CORE2output signals C2OUT.

At clock cycle 3, C1OUTD waveform is depicted as a one clock cycle delay of the C1OUT waveform. For example, C1OUT signals include data value labeled D1 in clock cycle 2 and the delayed signals C1OUTD include data value D1 in clock cycle 3. Because input signals to CORE2are delayed by one clock cycle (C1IND), CORE2outputs signals C2OUT are temporally aligned with C1OUTD signals. As such, C1OUTD and C2OUT signals are compared with one another in the compare unit212. For example, in clock cycles 3 and 4, output signals C2OUT match with C1OUTD signals and do not generate a fault indication (FAULT). However, in clock cycle 5, C1OUTD signals include data value D3 while C2OUT signals include data value DX which is detected as a fault thus generating a fault indication as a logic high signal on the FAULT waveform. In turn, the rising edge of the fault indication on the FAULT waveform causes ready signals (C1RDY, C2RDY) to de-assert for both CORE1and CORE2. With both C1RDY and C2RDY signals de-asserted, core domain104can transition into a safe mode.

While in the safe mode, CORE1and CORE2are isolated from the system bus102and allowed to complete any outstanding transactions. Outstanding transactions may include a load or store requests targeting system peripherals (e.g., memory110, other peripherals112) which have not yet completed. Other core domains (e.g., core domain106) and other circuits and functional blocks (e.g., controller108, memory110, and other peripherals112) may operate normally while core domain104is in the safe mode. Waveform labeled TRANS depicts a number of outstanding transactions. For example, at clock cycle 5, three outstanding transactions are remaining when CORE1and CORE2entered the safe mode. At clock cycle 8, CORE1and CORE2have completed two transactions with one transaction remaining as shown in the TRANS waveform. At clock cycle 9, all outstanding transactions have been completed as indicated by 0 (zero) in the TRANS waveform. After outstanding transactions have been completed, C1RDY and C2RDY signals are asserted at clock cycle 10.

FIG. 4illustrates, in flow chart diagram form, a simplified lockstep fault flow in accordance with an embodiment of the present invention. Exemplary lockstep operation corresponding to the blocks and circuitry depicted inFIG. 1andFIG. 2is described in the following steps.

At step402, power-up core domain104. During power-up of core domain104, each of CORE1and CORE2, lockstep control block108, associated memories, and other core domain circuitry is reset. After power-up and reset of the core domain104, CORE1and CORE2can begin execution independently. Because core domain104may be a portion of a system-on-chip (SoC) or the like, core domain104may be powered-up during power-up of the SoC. Likewise, core domain104may be reset during a reset operation of the SoC (e.g., power-on reset).

At step404, enter lockstep mode for core domain104. Controller108provides one or more control signals to core domain104to cause CORE1and CORE2along with lockstep control block118to operate in the lockstep mode. Control signals such as LS, SEL1, and SEL2are used to configure signal flow to/from respective cores, and control signals such as DLYI and DLYO are used to configure respective signal delay amounts from 0 (zero) clock cycle delay to N number of clock cycle delays.

For example, in the lockstep mode, control signal LS is used to route signals C1IND to C2INX as CORE2input signals by way of multiplexer202. Because CORE1and CORE2receive the same input signals (e.g., instructions) in the lockstep mode, CORE1and CORE2are expected to execute the same operations and provide the same output signals unless a fault occurs. Control signal SEL1is used to route C1OUT signals to C1OUTX as CORE1output signals by way of multiplexer204, and control signal SEL2is used to inhibit C2OUT signals from being routed outside of the core domain104by way of multiplexer206. Control signal DLYI is used to select a delay amount (e.g., one clock cycle) by which delay circuit210delays C1IN signals to form C1IND signals. Likewise, control signal DLYO is used to select a delay amount (e.g., one clock cycle) by which delay circuit208delays C1OUT signals to form C1OUTD signals.

At step406, compare CORE1output signals with CORE2output signals. CORE1output signals C1OUT are delayed (e.g., one cycle) to form C1OUTD signals to temporally align CORE1with CORE2output signals. Accordingly, C1OUTD signals are compared with CORE2output signals C2OUT in compare unit212. C1OUTD and C2OUT signals may each include tens or hundreds of individual signals. Circuitry of the compare unit212is configured to individually compare each of the tens or hundreds of C1OUTD signals with respective C2OUT signals in one clock cycle. A fault may occur in which one of the signals in the C1OUTD signals is different from a corresponding signal in the C2OUT signals. The fault may correspond to a soft error or a hard error. The soft error or hard error is generally associated with CORE1or CORE2circuitry and periphery thereof (e.g., cache memory). When a fault is detected, a fault indication is generated in the form of a flag, signal, and/or interrupt request and is received at controller108.

At step408, detect fault and generate fault indication. A fault is detected by the compare unit212when one of the signals in the C1OUTD signals is different from a corresponding signal in the C2OUT signals. The fault may occur as a result of a hard error or a soft error. A hard error may also be referred to as a permanent error (e.g., electrical over-stress or latent defect). A soft error may also be referred to as a transient error (e.g., electromagnetic interference, alpha particles, or voltage spikes) that does not damage the device. In turn, a fault indication is generated in the form of a flag, signal, and/or interrupt request.

At step410, enter safe mode and complete outstanding transactions. Responsive to the fault indication, core domain104enters the safe mode. While in the safe mode, CORE1and CORE2are isolated from system bus102and any outstanding transactions are completed. Core domain106may continue processing transactions and other portions of processing system100may continue normal operation while core domain104is in the safe mode. When all outstanding transactions are completed, an interrupt request is transmitted to controller108for further action.

At step412, analyze fault by performing tests such as memory built-in self-test (MBIST), logic built-in self-test (LBIST), and the like. When the interrupt request is received at controller108, an interrupt service routine (ISR) is initiated or other software routine begins execution at controller108to analyze the fault. For example, MBIST is performed on memories (e.g., cache) coupled to each of CORE1and CORE2. When MBIST is completed, LBIST is performed on each of CORE1and CORE2. In some embodiments, other tests may be performed to determine which, if any, core or core peripheral is faulty.

At step414, determine fault type and which core, if any, is faulty. After the fault is analyzed, a fault type may be determined (e.g., soft error or hard error). When MBIST fails or LBIST fails, the fault type is characterized as a hard error. To identify which core is faulty, the hard error may correspond to an error with either of CORE1or CORE2or a memory (e.g., cache) error associated with CORE1or CORE2. For example, a cache memory error associated with CORE1may be characterized as a hard error, and because the cache memory error is associated with CORE1, CORE1is considered the faulty core. When MBIST and LBIST pass, the fault type is characterized as a soft error.

At step416, determine whether the fault type is characterized as a hard error. When the fault type is not characterized as a hard error, the fault type may be considered a soft error and flow continues at step418. When the fault type is characterized as a hard error, the flow continues at step420.

At step418, core domain104is reset. When a soft error is determined, the core domain104is reset. Core domain106and other portions of processing system100may continue normal operation while core domain104is being reset. Upon completion of core domain104reset, safe mode is exited. After core domain104is reset, flow continues by entering lockstep mode at step404.

At step420, core domain104is reset. When a hard error is determined, the core domain104is reset. Core domain106and other portions of processing system100may continue normal operation while core domain104is being reset. Upon completion of core domain104reset, safe mode is exited and flow continues by entering a degraded mode at step422.

At step422, disable faulty core and run other core in the degraded mode. Because the fault corresponded to a hard error, one of CORE1and CORE2is identified as the faulty core and the other core is configured along with the lockstep control block118to operate in a degraded mode of operation (e.g., without the safety assured when in the lockstep mode). For example, based upon the fault corresponding to a hard error and CORE2being identified as the faulty core, CORE1resumes executing instructions and CORE2is inhibited from executing instructions.

Generally, there is provided, a method of operating a data processing system including detecting a fault by comparing output signals from a first processing core and a second processing core; entering a safe mode based upon detecting the fault; completing transactions while in the safe mode; determining whether the fault corresponds to a hard error; and based upon the fault corresponding to a hard error: identifying one of processing cores as a faulty core, the faulty core associated with the hard error; and inhibiting the faulty core from executing instructions and allowing the other processing core to resume executing instructions. The method may further include operating the first processing core and the second processing core in a lockstep mode, the second processing core shadowing the first processing core. The method may further include receiving at a controller an interrupt signal corresponding to the fault, and responsive to the interrupt signal, invoking a service routine to analyze the fault. The determining whether the fault corresponds to a hard error may further include performing a memory built-in self-test (MBIST) on at least one of a first cache memory coupled to a first processor of the first processing core and a second cache memory coupled to a second processor of the second processing core; and performing a logic built-in self-test (LBIST) on at least one of the first processor and the second processor. The hard error may be determined when the MBIST or LBIST fails. The method may further include coupling a first core domain to a system bus of the data processing system, the core domain comprising the first processing core and the second processing core. The safe mode may include isolating the first processing core and the second processing core from the system bus. The method may further include based upon the fault not corresponding to a hard error, resetting the first core domain, other portions of the processing system continue normal operation during reset of the first core domain. Detecting the fault by comparing output signals may include delaying output signals from the first processing core to temporally align with output signals from the second processing core.

In another embodiment, there is provided, a processing system including a core domain coupled to a system bus, the core domain includes a first processing core comprising a first processor coupled to a first cache memory, a second processing core comprising a second processor coupled to a second cache memory, and a lockstep control circuit coupled to the first processing core and the second processing core, the lockstep control circuit configured to detect a fault by comparing output signals from the first processing core and the second processing core, enter a safe mode based upon the detected fault, and complete transactions while in the safe mode; and a controller coupled to the lockstep control circuit, the controller configured to determine whether the fault corresponds to a hard error, and responsive to the fault corresponding to the hard error, allow one of the first processing core and second processing core not associated with the hard error to continue executing instructions. The lockstep control circuit may include a compare circuit, the compare circuit configured to detect a fault when output signals from the first processing core and the second processing core do not match. The safe mode may include isolation of the first processing core and the second processing core from the system bus. The controller may be configured to receive an interrupt signal corresponding to the fault, and responsive to the interrupt signal, invokes a service routine to analyze the fault. The service routine to analyze the fault may include a memory built-in self-test (MBIST) to be executed on at least one of the first cache memory and the second cache memory, and a logic built-in self-test (LBIST) to be executed on at least one of the first processor and the second processor. Based upon the fault not corresponding to a hard error, the controller may be configured to reset first core domain and allow other portions of the processing system continue normal operation during reset of the core domain.

In yet another embodiment, there is provided, a method of operating a data processing system including operating a core domain in a lockstep mode, the core domain comprising a first processing core shadowing a second processing core; detecting a fault by comparing output signals from the first processing core and the second processing core; entering a safe mode based upon detecting the fault, the safe mode isolating the core domain from a system bus; determining whether the fault corresponds to a hard error; identifying one of processing cores as a faulty core, the faulty core associated with the hard error; based upon the fault corresponding to a hard error, inhibiting the faulty core from executing instructions and allowing the processing core not associated with the hard error to resume executing instructions. Detecting the fault by comparing output signals may include delaying output signals from the second processing core to temporally align with output signals from the first processing core. The method may further include generating an interrupt signal corresponding to the fault, and responsive to the interrupt signal at a controller, invoking a service routine to analyze the fault. Determining whether the fault corresponds to a hard error may further include performing a memory built-in self-test (MBIST) on at least one of a first cache memory coupled to a first processor of the first processing core and a second cache memory coupled to a second processor of the second processing core; and performing a logic built-in self-test (LBIST) on at least one of the first processor and the second processor; wherein the hard error is determined when the MBIST or LBIST fails. The method may further include based upon the fault not corresponding to a hard error, resetting the core domain and allowing other portions of the data processing system continue normal operation during reset of the core domain.

Therefore, by now it can be appreciated that there has been provided, a data processing system and method for operating in a lockstep mode. Output signals from two processing cores of a core domain are compared during a lockstep mode. When a fault is detected, the core domain enters a safe mode and a determination is made whether the fault is caused by a soft error or a hard error. While in the safe mode, the core domain is isolated from a system bus of the data processing system and outstanding transactions are completed. The fault is analyzed by performing a memory built-in self-test (MBIST) on a cache memory coupled to a processor of each processing core and a logic built-in self-test (LBIST) on the processor of each processing core. A hard error is determined when MBIST or LBIST fails and a soft error is determined when MBIST and LBIST both pass. When a hard error is determined, fault analysis also identifies which core is a faulty core.

Based upon the fault corresponding to hard error, the faulty core is inhibited and the non-faulty core is allowed to continue operation. By inhibiting the faulty core and allowing the non-faulty core to continue executing, the data processing system can continue operation in a reduced or degraded operating mode which is desirable in a safety application. Based upon the fault corresponding to a soft error, a reset of the core domain is performed while other portions of the data processing system are allowed to operate normally. Because the reset of the core domain does not impact other portion of the data processing system, significant downtime savings can be achieved which is also desirable in a safety application.

Also for example, in one embodiment, the illustrated elements of data processing system100are circuitry located on a single integrated circuit or within a same device. Alternatively, data processing system100may include any number of separate integrated circuits or separate devices interconnected with each other.