Mission-critical AI processor with record and replay support

Embodiments described herein provide a mission-critical artificial intelligence (AI) processor (MAIP), which includes an instruction buffer, processing circuitry, a data buffer, command circuitry, and communication circuitry. During operation, the instruction buffer stores a first hardware instruction and a second hardware instruction. The processing circuitry executes the first hardware instruction, which computes an intermediate stage of an AI model. The data buffer stores data generated from executing the first hardware instruction. The command circuitry determines that the second hardware instruction is a hardware-initiated store instruction for transferring the data from the data buffer. Based on the hardware-initiated store instruction, the communication circuitry transfers the data from the data buffer to a memory device of a computing system, which includes the mission-critical processor, via a communication interface.

BACKGROUND

Field

This disclosure is generally related to the field of artificial intelligence (AI). More specifically, this disclosure is related to a system and method for facilitating a processor capable of processing mission-critical AI applications on a real-time system.

Related Art

The exponential growth of AI applications has made them a popular medium for mission-critical systems, such as a real-time self-driving vehicle or a critical financial transaction. Such applications have brought with them an increasing demand for efficient AI processing. As a result, equipment vendors race to build larger and faster processors with versatile capabilities, such as graphics processing, to efficiently process AI-related applications. However, a graphics processor may not accommodate efficient processing of mission-critical data. The graphics processor can be limited by processing limitations and design complexity, to name a few factors.

As more mission-critical features (e.g., features dependent on fast and accurate decision-making) are being implemented in a variety of systems (e.g., automatic braking of a vehicle), an AI system is becoming progressively more important as a value proposition for system designers. Typically, the AI system uses data, AI models, and computational capabilities. Extensive use of input devices (e.g., sensors, cameras, etc.) has led to generation of large quantities of data, which is often referred to as “big data,” that an AI system uses. AI systems can use large and complex models that can infer decisions from big data. However, the efficiency of execution of large models on big data depends on the computational capabilities, which may become a bottleneck for the AI system. To address this issue, the AI system can use processors capable of handling AI models.

Therefore, it is often desirable to equip processors with AI capabilities. Typically, tensors are often used to represent data associated with AI systems, store internal representations of AI operations, and analyze and train AI models. To efficiently process tensors, some vendors have used tensor processing units (TPUs), which are processing units designed for handling tensor-based computations. TPUs can be used for running AI models and may provide high throughput for low-precision mathematical operations.

While TPUs bring many desirable features to an AI system, some issues remain unsolved for handling mission-critical scenarios.

SUMMARY

Embodiments described herein provide a mission-critical artificial intelligence (AI) processor (MAIP), which includes an instruction buffer, processing circuitry, a data buffer, command circuitry, and communication circuitry. During operation, the instruction buffer stores a first hardware instruction and a second hardware instruction. The processing circuitry executes the first hardware instruction, which computes an intermediate stage of an AI model. The data buffer stores data generated from executing the first hardware instruction. The command circuitry determines that the second hardware instruction is a hardware-initiated store instruction for transferring the data from the data buffer. Based on the hardware-initiated store instruction, the communication circuitry transfers the data from the data buffer to a memory device of a computing system, which includes the mission-critical processor, via a communication interface.

In a variation on this embodiment, the communication interface is one of: a peripheral component interconnect express (PCIe) interface; and a network interface card (NIC).

In a variation on this embodiment, the MAIP also includes encryption circuitry, which encrypts the data in the data buffer.

In a variation on this embodiment, the processing circuitry includes a plurality of processing units comprising one or more of: (i) a dataflow processing unit (DPU) comprising a scalar computing unit and a vector computing unit; and (ii) a tensor computing unit (TCU) comprising a cluster of DPUs, high-bandwidth memory devices, and input/output (I/O) devices.

In a further variation, the MAIP includes control circuitry, which operates the processing circuitry in a low-power mode by turning off one or more processing units of the plurality of processing units.

In a variation on this embodiment, the communication circuitry can store computational states of the processing circuitry in a state buffer of the computing system. This allows a second MAIP of the computing system to resume operations associated with the computational states.

In a variation on this embodiment, the communication circuitry stores the data in a storage device of a remote computing device via the communication interface using a remote memory access protocol.

In a variation on this embodiment, the instruction buffer can store the first and second hardware instructions by one of: storing the first and second hardware instructions prior to runtime; and storing the first hardware instruction prior to runtime and dynamically inserting the second hardware instruction during runtime.

In a variation on this embodiment, based on the hardware-initiated store instruction, the communication circuitry also transfers contextual information associated with the MAIP to the memory device. The contextual information includes one or more of: feature maps associated with the data in the data buffer; and information associated with one or more computational units of the MAIP. Such information includes one or more of: temperatures, working conditions, utilization, and performance statistics.

Embodiments described herein provide a system for facilitating hardware instructions to a mission-critical system. During operation, the system identifies an instruction block comprising a set of hardware instructions, which computes an intermediate stage of an artificial intelligence (AI) model. A respective instruction of the instruction block is executable on a mission-critical AI processor (MAIP). Based on a set of parameters, the system determines whether contexts associated with the instruction block should be recorded. If the contexts associated with the instruction block should be recorded, the system appends a hardware-initiated instruction to the instruction block. The hardware-initiated instruction initiates a transfer of data generated by the instruction block to outside of the MAIP. The system then provides the instruction block with the hardware-initiated instruction to the MAIP.

In a variation on this embodiment, the set of parameters includes available storage, communication bandwidth, and external inputs, which includes a policy of a datacenter, customer preferences, developer preferences, and environmental feedback.

In a variation on this embodiment, the system provides the instruction block to the MAIP prior to runtime of the MAIP.

Embodiments described herein provide a mission-critical system, which includes a system processor, a system memory device, a communication interface, and a first mission-critical artificial intelligence (AI) processor (MAIP) coupled to the communication interface, and an operating module. The first MAIP can include an instruction buffer, processing circuitry, a data buffer, command circuitry, and communication circuitry. During operation, the operating module loads a first and a second hardware instructions in the instruction buffer of the first MAIP. The processing circuitry executes the first hardware instruction, which computes an intermediate stage of an AI model. The data buffer stores data generated from executing the first hardware instruction. The command circuitry determines that the second hardware instruction is a hardware-initiated store instruction for transferring the data from the data buffer. Based on the hardware-initiated store instruction, the communication circuitry transfers the data from the data buffer to the system memory device via the communication interface.

In a variation on this embodiment, the communication interface is one of: a peripheral component interconnect express (PCIe) interface; and a network interface card (NIC).

In a variation on this embodiment, the processing circuitry of the first MAIP includes a plurality of processing units comprising one or more of: (i) a dataflow processing unit (DPU) comprising a scalar computing unit and a vector computing unit; and (ii) a tensor computing unit (TCU) comprising a cluster of DPUs, high-bandwidth memory devices, and input/output (I/O) devices.

In a variation on this embodiment, the system also includes a backup power source and power circuitry. The power circuitry can detect a power failure of the system, switch the first MAIP to the backup power source, and reduce operations of the first MAIP to save power.

In a variation on this embodiment, the system also includes a state buffer and high-availability circuitry. The state buffer stores computational states of the first MAIP. The high-availability circuitry can detect a failure of the first MAIP. The high-availability circuitry then loads the computational states of the first MAIP to a second MAIP from the state buffer and resumes operations associated with the computational states using the second MAIP.

In a variation on this embodiment, the system also includes a network interface, which can transfer the data from the memory device to a remote computing system for replaying the data.

In a variation on this embodiment, the operating module can load both the first and second hardware instructions upon powering up of the mission-critical system prior to runtime. Alternatively, the operating module can load the first hardware instruction upon powering up of the mission-critical system prior to runtime and dynamically insert the second hardware instruction during runtime.

In a variation on this embodiment, based on the hardware-initiated store instruction, the communication circuitry of the first MAIP also transfers contextual information associated with the first MAIP to the system memory device. The contextual information includes one or more of: feature maps associated with the data in the data buffer; and information associated with one or more computational units of the first MAIP. Such information includes one or more of: temperatures, working conditions, utilization, and performance statistics.

DETAILED DESCRIPTION

The following description is presented to enable any person skilled in the art to make and use the embodiments, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present disclosure. Thus, the embodiments described herein are not limited to the embodiments shown, but are to be accorded the widest scope consistent with the principles and features disclosed herein.

Overview

The embodiments described herein solve the problem of facilitating an AI processor for mission-critical systems by (i) incorporating high-availability, resource virtualization, encryption, and power adjustment capabilities into a processor; and (ii) facilitating an instruction or instruction set that allows record and replay of the process of the predictions and decisions.

Many mission-critical systems rely on AI applications to make instantaneous and accurate decisions based on the surrounding real-time environment. An AI application can use one or more AI models (e.g., a neural-network-based model) to produce a decision. Usually the system uses a number of input devices, such as sensors (e.g., sonar and laser), cameras, and radar, to obtain real-time data. Since the system can use a large number of such input devices, they may generate a large quantity of data based on which the AI applications make decisions. To process such a large quantity of data, the system can use large and complex AI models that can generate the decisions. For example, the safety features of a car, such as automatic braking and lane departure control, may use an AI model that processes real-time data from on-board input devices of the car.

With existing technologies, AI applications may run on graphics processing units (GPUs) or tensor processing units (TPUs). Typically, a GPU may have a higher processing capability among these two options (e.g., indicated by a high floating point operations per second (FLOPS) count). However, since a GPU is designed for vector and matrix manipulations, the GPU may not be suitable for all forms of tensor. In particular, since a mission-critical system may use data from a variety of input devices, the input data can be represented based on tensors with varying dimensions. As a result, the processing capabilities of the GPU may not be properly used for all AI applications.

On the other hand, a TPU may have the capability to process tensor-based computations more efficiently. However, a TPU may have a lower processing capability. Furthermore, some TPUs may only be efficiently used for applying AI models but not for training the models. Using such a TPU on a mission-critical system may limit the capability of the system to learn from a new and dynamic situation. Therefore, existing GPUs and TPUs may not be able to process large and time-sensitive data of a mission-critical system with high throughput and low latency. In addition, existing GPUs and TPUs may not be able to facilitate other important requirements of a mission-critical system, such as high availability and low-power computation for failure scenarios.

Moreover, for some AI models, such as neural-network-based models, the system provides a set of inputs, which is referred to as an input layer, to obtain a set of outputs, which is referred to as an output layer. The results from intermediate stages, which are referred to as hidden layers, are usually not presented to any entity outside of the AI model. As a result, if the predictions and decisions generated by the AI model are insufficient (i.e., the AI model fails to produce the correct result) or the system suffers a failure, there may not be sufficient information to analyze what has caused the failures.

To solve these problems, embodiments described herein provide a mission-critical AI processor (MAIP), which can be an AI processor chip, that can process tensors with varying dimensions with high throughput and low latency. Furthermore, the MAIP can provide the capability to record and replay the process of the predictions and decisions, which allows analysis of intermediate stages. A mission-critical system can use one or more MAIPs that can efficiently process the large quantity of data the input devices of the system may generate. Furthermore, an MAIP can also process training data with high efficiency. As a result, the mission-critical system can be efficiently trained for new and diverse real-time scenarios.

Furthermore, since any failure associated with the system can cause critical problems, an MAIP allows real-time handover to a standby MAIP if that MAIP suffers a failure. In some embodiments, the MAIP can maintain current computational results and operational states in a memory that can be handed over to the standby MAIP to resume operations. This feature allows the system to facilitate high availability in critical failure scenarios. The MAIP can also operate in a reduced computation mode in a power failure. If the system suffers a power failure, the MAIP can detect the failure and switch to a backup power source (e.g., a battery). The MAIP then can only use the resources (e.g., the tensor computing units or TCUs) for processing the critical operations, thereby using low power for computations.

Moreover, the MAIP facilitates hardware-assisted virtualization to AI applications. For example, the resources of the MAIP can be virtualized in such a way that the resources are efficiently divided among multiple AI applications. Each AI application may perceive that the application is using all resources of the MAIP. In addition, the MAIP is equipped with an on-board security chip (e.g., a hardware-based encryption chip) that can encrypt output data of an instruction (e.g., data resultant of a computation associated with the instruction). This prevents any rogue application from accessing on-chip data (e.g., from the registers of the MAIP).

Furthermore, the record and replay feature of the MAIP allows the system (or a user of the system) to analyze stage contexts associated with the intermediate states of an AI model and determine the cause of any failure associated with the system and/or the model. Upon detecting the cause, the system (or the user of the system) can reconfigure the system to avoid future failures. The record and replay feature can be implemented for the MAIP using a dedicated processor/hardware instruction (or instruction set) that allows the recording of the contexts of the AI model, such as intermediate stage contexts (e.g., feature maps and data generated from the intermediate stages) of the AI model. This instruction can be appended to an instruction block associated with an intermediate stage. The instruction can be preloaded (e.g., inserted prior to the execution) or inserted dynamically during runtime. The replay can be executed on a software simulator or a separate hardware system (e.g., with another MAIP).

The term “processor” refers to any hardware unit, such as an electronic circuit, that can perform an operation, such as a mathematical operation on some data or a control operation for controlling an action of a system. The processor can be an application-specific integrated circuit (ASIC) chip.

The term “application” refers to an application running on a user device, which can issue an instruction for a processor. An AI application can be an application that can issue an instruction associated with an AI model (e.g., a neural network) for the processor.

Exemplary System

FIG. 1Aillustrates an exemplary mission-critical system equipped with MAIPs supporting storage and replay, in accordance with an embodiment of the present application. In this example, a mission-critical system110operates in a real-time environment100, which can be an environment where system110may make real-time decisions. For example, environment100can be an environment commonly used by a person, such as a road system with traffic, and system110can operate in a car. Environment100can also be a virtual environment, such as a financial system, and system110can determine financial transactions. Furthermore, environment100can also be an extreme environment, such as a disaster zone, and system110can operate on a rescue device.

Mission-critical system110relies on AI applications114to make instantaneous and accurate decisions based on surrounding environment100. AI applications114can include one or more AI models113and115. System110can be equipped with one or more input devices112, such as sensors, cameras, and radar, to obtain real-time input data102. System110can apply AI model113to input data102to produce a decision104. For example, if AI model113(or115) is a neural-network-based model, input data102can represent an input layer for the model and decision104can be the corresponding output layer.

Since modern mission-critical systems can use a large number of various input devices, input devices112of system110can be diverse and large in number. Hence, input devices112may generate a large quantity of real-time input data102. As a result, to produce decision104, AI applications114need to be capable of processing a large quantity of data. Hence, AI models113and115can be large and complex AI models that can generate decision104in real time. For example, if system110facilitates the safety features of a car, such as automatic braking and lane departure control, continuous real-time monitoring of the road conditions using input devices112can generate a large quantity of input data102. AI applications114can then apply AI models113and/or115to determine decision104, which indicates whether the car should brake or has departed from its lane.

System110can include a set of system hardware116, such as a processor (e.g., a general purpose or a system processor), a memory device (e.g., a dual in-line memory module or DIMM), and a storage device (e.g., a hard disk drive or a solid-state drive (SSD)). The system software, such as the operating system and device firmware of system110, can run on system hardware116. System110can also include a set of AI hardware118. With existing technologies, AI hardware118can include a set of GPUs or TPUs. AI applications114can run on the GPUs or TPUs of AI hardware118.

However, a GPU may not be suitable for all forms of tensor. In particular, since system110may use data from a variety of input devices112, input data102can be represented based on tensors with varying dimensions. As a result, the processing capabilities of a GPU may not be properly used by AI applications114. On the other hand, a TPU may have the capability to process tensor-based computations more efficiently. However, a TPU may have a lower processing capability, and may only be efficiently used for applying AI models but not for training the models. Using such a TPU on system110may limit the capability of system110to learn from a new and dynamic situation.

Therefore, existing GPUs and TPUs may not be able to efficiently process large and time-sensitive input data102for system110. In addition, existing GPUs and TPUs may not be able to facilitate other important requirements of system110, such as high availability and low-power computation for failure scenarios. Moreover, for some AI models, contexts from intermediate hidden layers, which are derived from input data102by applying the models, are usually not presented to any entity outside of the AI model. As a result, if the AI model fails to produce a correct result or system110suffers a failure, there may not be sufficient information to analyze what has caused the failure.

To solve these problems, AI hardware118of system110can be equipped with a number of MAIPs122,124,126, and128that can efficiently process tensors with varying dimensions. These MAIPs can also process training data with high efficiency. As a result, system110can be efficiently trained for new and diverse real-time scenarios. In addition, these MAIPs are capable of recording and replaying intermediate stages associated with decision104. AI hardware118, equipped with MAIPs122,124,126, and128, thus can efficiently run AI applications114, which can apply AI models113and/or115to input data102to generate decision104with low latency. For example, with existing technologies, if a datacenter uses 100 GPUs, the datacenter may use 10 GPUs for training and 90 GPUs for inference, because 90% of GPUs are typically used for inference. However, similar levels of computational performance can be achieved using 10 MAIPs for training and 15 MAIPs for inference. This can lead to a significant cost savings for the datacenter. Therefore, in addition to mission-critical systems, an MAIP can facilitate efficient computations of AI models for datacenters as well.

An MAIP, such as MAIP128, can include a TCU cluster148formed by a number of TCUs. Each TCU, such as TCU146, can include a number of dataflow processor unit (DPU) clusters. Each DPU cluster, such as DPU cluster144, can include a number of DPUs. Each DPU, such as DPU142, can include a scalar computing unit (SCU)140and a vector computing unit (VCU)141. SCU140can include a plurality of traditional CPU cores for processing scalar data. VCU141can include a plurality of tensor cores used for processing tensor data (e.g., data represented by vectors, matrices, and/or tensors). In the same way, MAIPs122,124, and126can include one or more TCU clusters, each formed based on DPUs comprising SCUs and VCUs.

Furthermore, since any failure associated with system110can cause a critical problem, system110can be equipped with a standby MAIP120, which may not participate in active processing during normal operation (i.e., without any failure). In addition, AI hardware118can include a high availability module134, which can monitor MAIPs122,124,126, and128. During operation, MAIP128(and other MAIPs of system110) can maintain current computational results and operational states in a state buffer132, which can be a memory device. If high availability module134detects a failure of MAIP128, high availability module134initiates a real-time handover to standby MAIP120. MAIP120can obtain the computational results and operational states of MAIP128from state buffer132and resume operations. This feature allows system110to facilitate high availability in critical failure scenarios.

In some embodiments, MAIP128can also operate in a reduced computation mode in a power failure. If system110suffers a power failure, MAIP128can detect the failure and switch to a backup power source138. This power source can be part of AI hardware118or any other part of system110. MAIP128then can use the resources (e.g., the TCUs) for processing the critical operations of system110. MAIP128can turn off some TCUs, thereby using low power for computation. System110can also turn off one or more of the MAIPs of AI hardware118to save power. If the power comes back, system110can resume regular computation mode.

Moreover, MAIP128can facilitate hardware-assisted virtualization to AI applications. For example, AI hardware118can include a virtualization module136, which can be incorporated in a respective MAIP or a separate module. Virtualization module136can present the resources of MAIPs122,124,126, and128as virtualized resources130in such a way that the resources are efficiently divided among multiple AI applications. Each AI application may perceive that the application is using all resources of an MAIP and/or system110.

In addition, MAIP128can be equipped with an on-board security chip149, which can be a hardware-based encryption chip. Chip149can encrypt output data of an instruction. This data can be resultant of a computation associated with the instruction. This prevents any rogue application from accessing on-chip data stored in the registers of MAIP128. For example, if an application in AI applications114becomes compromised (e.g., by a virus), that compromised application may not access data generated by other applications in AI applications114from the registers of MAIP128.

Furthermore, the record and replay feature of MAIP128allows system110(or a user of system110) to analyze stage contexts, such as results from intermediate stages of AI model113(or115), and determine the cause of any failure associated with system110and/or AI model113. Upon detecting the cause, system110(or the user of system110) can reconfigure system110to avoid future failures. The record and replay feature can be implemented for MAIP128using a dedicated processor/hardware instruction (or instruction set) that can be executed by MAIP128. The instruction can record the contexts of AI model113, such as stage contexts (e.g., feature maps and data generated from the intermediate stages) of AI model113. This instruction can be appended to an instruction block associated with an intermediate stage. The instruction can be preloaded prior to runtime (e.g., after powering up before executing any computational instruction) or inserted dynamically during runtime. The replay can be executed on a software simulator or a separate hardware system.

FIG. 1Billustrates an exemplary system stack of a mission-critical system, in accordance with an embodiment of the present application. A system stack150of system110operates based on a TCU cluster166(e.g., in an MAIP). A scheduler164runs on cluster166that schedules the operations on TCU cluster166. Scheduler164dictates the order at which the instructions are loaded on TCU cluster166. A driver162allows different AI frameworks156to access functions of TCU cluster166. AI frameworks156can include any library (e.g., a software library) that can facilitate AI-based computations, such as deep learning. Examples of AI frameworks156can include, but are not limited to, TensorFlow, Theano, MXNet, and DMLC.

AI frameworks156can be used to provide a number of AI services154. Such services can include vision, speech, natural language processing, etc. One or more AI applications152can operate to facilitate AI services154. For example, an AI application that determines a voice command from a user can use a natural language processing service based on TensorFlow. In addition to AI frameworks156, driver162can allow commercial software158to access TCU cluster166. For example, an operating system that operates system110can access TCU cluster166using driver162.

Chip Architecture

FIG. 2Aillustrates an exemplary chip architecture of a TCU in an MAIP supporting storage and replay, in accordance with an embodiment of the present application. A DPU202can include a control flow unit (CFU)212and a data flow unit (DFU)214, which are coupled to each other via a network fabric (e.g., a crossbar) and may share a data buffer. CFU212can include a number of digital signal processing (DSP) units and a scheduler, a network fabric interconnecting them, and a memory. DFU214can include a number of tensor cores and a scheduler, a network fabric interconnecting them, and a memory. A number of DPUs202,204,206, and208, interconnected based on crossbar210, form a DPU cluster200.

A number of DPU clusters, interconnected based on a network fabric240, can form a TCU230. One such DPU cluster can be DPU cluster200. TCU230can also include memory controllers232and234, which can facilitate high-bandwidth memory, such as HBM2. TCU230can be designed based on a wafer level system integration (WLSI) platform, such as CoWoS. In addition, TCU230can include a number of input/output (I/O) interfaces236. An I/O interface of TCU230can be a serializer/deserializer (SerDes) interface that may convert data between serial data and parallel interfaces.

FIG. 2Billustrates an exemplary chip architecture of a TCU cluster in an MAIP supporting storage and replay, in accordance with an embodiment of the present application. Here, a tensor processing unit (TPU)250is formed based on a cluster of TCUs. One such TCU can be TCU230. In TPU250, the TCUs can be coupled to each other using respective peripheral component interconnect express (PCIe) interfaces or SerDes interfaces. This allows individual TCUs to communicate with each other to facilitate efficient computation of tensor-based data.

Storing Intermediate Stage Contexts

FIG. 3illustrates an exemplary hardware-initiated storage instruction of an MAIP, in accordance with an embodiment of the present application. AI model113can be based on a multi-layer decision process (e.g., a neural network). System110can provide a set of inputs, which is referred to as an input layer302, to AI model113. AI model113can process input layer302through one or more intermediate stages, which are referred to as hidden layers, to obtain a set of outputs, which is referred to as an output layer308. In this example, AI model113can have at least two intermediate stages, which are referred to as hidden layers304and306.

With existing technologies, AI model113operates based on a “black box” principle, where the computations conducted in hidden layers304and306are not available outside of AI model113. In other words, if system110(or a user of system110) wishes to analyze the intermediate computations of AI model113, system110may not gain access to those computations. As a result, if AI model113fails to produce the correct result or system110fails, data generated by the intermediate computations of hidden layers304and306may not be available to explain or analyze what has caused the failure.

This problem is solved by incorporating the capability of recording and replaying the computations of each stage of AI model113into an MAIP, such as MAIP128. This allows system110(or a user of system110) to analyze the cause of the failure and, based on the analysis, reconfigure system110to avoid future failures. MAIP128can support a processor instruction (or instruction set) that can cause MAIP128to store the data generated from an instruction block into the memory of system110. This processor instruction can be a “hardware-initiated store” instruction (e.g., a hardware dump instruction). Since this instruction is hardware initiated, it does not involve the processor cycle of the system processor of system110and can be executed in parallel to the operations of the system processor.

It should be noted that this hardware-initiated store instruction is distinct from a software-based store instruction. The software-based store instruction requires the system processor to issue an I/O interrupt, lock the current data on a buffer of MAIP128, and transfer the data to the system memory. This actually costs an instruction cycle of the system processor. Since system110can be equipped with a plurality of MAIPs, each comprising multiple TPUs running AI model113, system110can generate a significant quantity of data associated with the hidden layers. Using a software-based store instruction to store such data may cause the system processor to issue a large number of interrupts and degrade the performance of system110.

Upon executing the hardware-initiated store instruction, MAIP128can record the data stored in its buffer. Hence, this store instruction can be appended after each instruction block representing a layer of AI model113. For example, the store instruction can be appended to the instruction block that computes the transition from input layer302to hidden layer304. This would cause MAIP128to record stage context312associated with the computation. Similarly, the store instruction can also cause MAIP128to record stage context314associated with the computation of the transition from hidden layer304to hidden layer306.

Stage contexts312and314can provide contexts, such as intermediate feature maps and data generated from computations, for subsequent replay or diagnosis purposes. Information recorded in stage contexts312and314includes, but is not limited to, intermediate feature maps and processor information, such as temperatures, working conditions, utilization, and statistics. The instruction can be preloaded prior to runtime (e.g., can be inserted offline) or dynamically inserted during runtime. Stage contexts312and314can be transferred to another system, which then can be used for replaying on a software simulator or an MAIP on that system.

Store and Replay

FIG. 4Aillustrates an exemplary architecture of an MAIP supporting hardware-initiated storage instructions, in accordance with an embodiment of the present application. In this example, system hardware116of system110includes a system processor402(i.e., the central processor of system110), a memory device404(i.e., the main memory of system110), and a storage device406. Here, memory device404and storage device406are off-chip. MAIP128can include a systolic array of parallel processing engines. In some embodiments, the processing engines form a matrix multiplier unit (MXU)422. MXU422can include a number of write buffers421and423. MAIP128can also include an accumulation buffer (e.g., an accumulator)424, which can be one or more registers that can store the data generated by the computations executed by MXU422. MAIP128can also include a system control unit (SCU)426.

MAIP128can also include a dedicated unit (or units), a command sequencer (CSQ)412, to execute instructions in an on-chip instruction buffer430that control the systolic array (i.e., MXU422) for computations. A finite state machine (FSM)414of CSQ412dispatches a respective instruction in instruction buffer430. Depending on the current instruction (e.g., a fetch instruction), FSM414can also dispatch an instruction to buffer424for obtaining data stored in buffer424. In addition, upon detecting a control instruction (e.g., an instruction to switch to a low-power mode), FSM414may dispatch an instruction to SCU426.

Data generated by intermediate computations from MXU422are stored in an on-chip unified buffer416. For the example inFIG. 3, stage contexts312and314can be stored in unified buffer416. Data from unified buffer416can be input to subsequent computations. Accordingly, MXU422can retrieve data from unified buffer416for the subsequent computations. MAIP128can also include a direct memory access (DMA) controller420, which can transfer data between memory device404and unified buffer416.

MAIP128can use a communication interface418to communicate with components of system110that are external to MAIP128. Examples of interface418can include, but are not limited to, a PCIe interface and a network interface card (NIC). MAIP128may obtain instructions and input data, and provide output data and/or the recorded contexts using interface418. For example, the instructions for AI-related computations are sent from system software410(e.g., an operating system) of system110to instruction buffer430via interface418. Similarly, DMA controller420can send data in unified buffer416to memory device404via interface418.

During operation, software410provides instruction blocks corresponding to the computations associated with an AI operation. For example, software410can provide an instruction block432comprising one or more instructions to be executed on MAIP128via interface418. Instruction block432can correspond to one computational stage of an AI model (e.g., a neural network). Similarly, software410can provide another instruction block434corresponding to a subsequent computational stage of the AI model. Instruction blocks432and434are stored in instruction buffer430.

To facilitate the record and replay operations of system110, MAIP128can support a hardware-initiated store instruction, which can be an individual instruction or an instruction set, to facilitate recording of the intermediate stage contexts. This recording can be stored in on-chip or off-chip memory devices. Software410determines the execution sequence associated with instruction blocks432and434, and determines where this instruction should be inserted. For example, if instruction blocks432and434correspond to two stages of computation of the AI model, software410can insert a store instruction433after instruction block432and another store instruction435after instruction block434.

Upon completion of execution of instruction block432, data generated from the execution is stored in unified buffer416. Based on the sequence in instruction buffer430, FSM414retrieves store instruction433, and accordingly, instructs DMA controller420to initiate a transfer from unified buffer416. DMA controller420then transfers the data from unified buffer416to memory device404. This records the intermediate stage contexts, thereby providing access to the stage contexts from outside of MAIP128. Similarly, upon completion of execution of instruction block434, data generated from the execution is stored in unified buffer416. Based on store instruction435, DMA controller420transfers the data from unified buffer416to memory device404.

For a more persistent storage, data can be transferred from memory device404to storage device406. This allows retrieval of the contexts in case of a failure of system110. DMA controller420can also record the contexts directly through common communication channels (e.g., using remote DMA (RDMA)) via a network440to non-local storage on a remote storage server442. In some embodiments, storage server442can be equipped with a software simulator or another MAIP that can replay the stored results.

Software410can preload store instructions433and435prior to runtime or insert them dynamically during runtime. For example, when system110powers up, software410may load instruction blocks432and434to instruction buffer430prior to runtime. Software410can preload instructions433and435with instruction blocks432and434. This can prevent disruption of operations of MAIP128. Software410can also dynamically insert instructions433and435during runtime to accommodate a dynamic scenario. For example, if system110facilitates safety features of a car, an adverse weather condition may increase the chances of a failure. To analyze a potential failure, software410can dynamically insert store instructions upon detecting the adverse weather condition.

The timing, frequency, and locations to insert store instructions433and435can be parameterized or computed based on storage size, communication bandwidth, and external inputs, such as a policy of a datacenter, customer preferences, developer preferences, and environmental feedback, etc. For example, a user can configure how frequently the contexts are recorded. Similarly, the external environment (e.g., an adverse weather condition) can trigger the recording of contexts. If available storage in memory device404and/or storage device406becomes constrained, software410may refrain from further inserting the store instruction or even remove one or more store instructions from instruction buffer430.

FIG. 4Billustrates an exemplary incorporation of hardware-initiated storage instructions in an MAIP, in accordance with an embodiment of the present application. Software410can generate the execution sequence for the instructions in instruction buffer430, which can include the store instructions inserted by software410. In some embodiments, a compiler460inserts the store instructions during compile time. During operation, compiler460executes on a development device450to compile source code464of the AI model(s) (e.g., a piece of code based on an AI library, such as TensorFlow and Theano).

Compiler460can also obtain a set of compile parameters462, which can specify the timing, frequency, and locations of the store instructions that are to be inserted. For example, upon compiling code464, compiler460can generate an instruction set466that can be executed on an MAIP. This compilation process generates instructions supported by the MAIP. Instruction set466can include instruction blocks432and434. During compile time, compiler460determines where the store instructions should be inserted. It should be noted that the store instructions can be inserted at any of the stages of the compilation process (e.g., in any of the preprocessing, compilation, assembly, and linking stages).

Suppose that compile parameters462indicate that a store instruction should be inserted after each stage of computation of an AI model (e.g., a neural network). In the example inFIG. 4A, if instruction blocks432and434correspond to two stages of computation of an AI model, compiler460can insert store instruction433after instruction block432and store instruction435after instruction block434. On the other hand, if compile parameters462indicate that a store instruction should be inserted after an initial stage of computation of the AI model, compiler460may insert store instruction433after instruction block432and but may not insert a store instruction after instruction block434.

Upon generating instruction set466, development device450provides instruction set466to system110via network440. Software410obtains instruction set466and loads it on instruction buffer430via interface418. It should be noted that compiler460can also run on software410of system110. Under such circumstances, instruction set466can be generated in memory device404. Software410can then load instruction set466on instruction buffer430from memory device404via interface418.

The recorded contexts can reside in memory device404. Software410can periodically transfer the contexts from memory device404to storage device406. For example, if the size of the stored contexts reaches a threshold, software410can transfer the contexts from memory device404to storage device406. In some embodiments, the contexts in storage device406can be transferred to a replay device470for further analysis. Replay device470can include a software simulator472, which can simulate the operations of an MAIP. Replay device470can also include another MAIP474. The recorded contexts can be replayed on simulator472and/or MAIP474to analyze the recorded contexts. It should be noted that the same physical device can serve as development device450, replay device470, and storage server442, as described in conjunction withFIGS. 4A and 4B.

Operations

FIG. 5Apresents a flowchart500illustrating a method of a mission-critical system virtualizing available AI resources, in accordance with an embodiment of the present application. During operation, the system identifies individual MAIPs of the system and determines computational capabilities of a respective MAIP (operation502). The system then combines the computational capabilities to determine computational capabilities of the system (operation504) and generates virtualized AI resources based on the determined computational capabilities (operation506). The system presents the virtualized AI resources to the AI services running on the system (operation508).

FIG. 5Bpresents a flowchart530illustrating a method of a mission-critical system facilitating high availability, in accordance with an embodiment of the present application. During operation, the system detects a failure associated with an MAIP (operation532) and obtains current states and computational data associated with the failed MAIP (operation534) (e.g., from a state buffer of the system). The system then loads the computational data on the registers of a standby MAIP of the system (operation536) and continues processing using the standby MAIP based on the obtained states (operation538).

FIG. 5Cpresents a flowchart illustrating a method of a mission-critical system operating using low power with reduced performance, in accordance with an embodiment of the present application. During operation, the system detects a power failure associated with the system (operation552) and turns on a backup power source (operation554) (e.g., a battery). The system then determines a minimum computing requirement (operation556). The system may determine the computing requirement based on one or more of: a configuration of the system, an input from the environment of the system, and a user input.

The system then designates the MAIPs and/or computational units, such as TPUs, of a respective MAIP that can address the minimum computing requirement of the system (operation558). The system then starts operating the designated MAIPs and/or the computational units using the backup power source (operation560). The system turns off the MAIPs and/or the computational units that are not designated (operation562). In this way, the system can save power by using computational resources only for the minimum computing requirement and not for anything else. In other words, the system can save power by turning off computational resources that are not used for the minimum computing requirement.

FIG. 6Apresents a flowchart illustrating a method of a compiler incorporating hardware-initiated storage instructions into instruction blocks associated with intermediate layers, in accordance with an embodiment of the present application. During operation, the compiler obtains a set of compile parameters and the source code, and initiates a compilation process (operation602). The compiler can generate an instruction block associated with an intermediate hidden layer (operation604) and determines whether to insert a “hardware-initiated store” instruction based on the compile parameters (operation606).

If an insertion is instructed in the compile parameters (operation608), the compiler appends a “hardware-initiated store” instruction to the instruction block (operation610). On the other hand, if an insertion is not instructed (operation608), the compiler checks whether an output layer has been reached at the source code (operation612). If an output layer has been reached (operation612), the compiler can finalize the instruction sets for a mission-critical system (operation614). Otherwise, the compiler continues to generate an instruction block associated with an intermediate hidden layer (operation604).

FIG. 6Bpresents a flowchart illustrating a method of a dispatcher of an MAIP executing a hardware-initiated storage instruction, in accordance with an embodiment of the present application. In some embodiments, the dispatcher can be an FSM in a CSQ of the MAIP. During operation, the dispatcher reads a “hardware-initiated store” instruction from an instruction buffer of a CSQ (operation632). The dispatcher identifies the data associated with the preceding computations in the unified buffer of the MAIP (operation634). The dispatcher then instructs the DMA controller of the MAIP to read the identified data from the unified buffer and store the data in the system memory (operation636).

Exemplary Computer System and Apparatus

FIG. 7illustrates an exemplary computer system supporting a mission-critical system, in accordance with an embodiment of the present application. Computer system700includes a processor702, a memory device704, and a storage device708. Memory device704can include a volatile memory device (e.g., a dual in-line memory module (DIMM)). Furthermore, computer system700can be coupled to a display device710, a keyboard712, and a pointing device714. Storage device708can store an operating system716, a mission-critical system718, and data736. Mission-critical system718can facilitate the operations of one or more of: mission-critical system110, storage server442, compiler460, and replay device470. In some embodiments, computer system700can also include AI hardware706comprising one or more MAIPs, as described in conjunction withFIG. 1A.

Mission-critical system718can include instructions, which when executed by computer system700can cause computer system700to perform methods and/or processes described in this disclosure. Specifically, mission-critical system718can include instructions for a mission-critical system facilitating high-availability among the MAIPs of AI hardware706(high-availability module720). Mission-critical system718can also include instructions for the mission-critical system operating AI hardware706to address a minimum computing requirement in the event of a power failure (power module722).

Furthermore, mission-critical system718includes instructions for the mission-critical system virtualizing the resources of AI hardware706(virtualization module724). Moreover, mission-critical system718includes instructions for the mission-critical system encrypting data generated by AI hardware706(encryption module726). Mission-critical system718can also include instructions for a compiler to insert a “hardware-initiated store” instruction into the instructions that can be executed on the MAIPs of AI hardware706(instruction module728).

Mission-critical system718can further include instructions for the mission-critical system recording contexts by executing “hardware-initiated store” instructions (e.g., either in the mission-critical system or in a remote storage server) (recording module730). Mission-critical system718can also include instructions for a replay device replaying the recorded contexts (replaying module732). Mission-critical system718may further include instructions for the mission-critical system, the storage server, the compiler, and the replay device sending and receiving messages (communication module734).

Data736can include any data that can facilitate the operations of one or more of: mission-critical system110, storage server442, compiler460, and replay device470. Data736may include one or more of: the source code, instructions generated by compiling the source code, “hardware-initiated store” instructions, and the recorded contexts.

FIG. 8illustrates an exemplary apparatus that supports a mission-critical system, in accordance with an embodiment of the present application. Mission-critical apparatus800can comprise a plurality of units or apparatuses which may communicate with one another via a wired, wireless, quantum light, or electrical communication channel. Apparatus800may be realized using one or more integrated circuits, and may include fewer or more units or apparatuses than those shown inFIG. 8. Further, apparatus800may be integrated in a computer system, or realized as a separate device that is capable of communicating with other computer systems and/or devices. Specifically, apparatus800can comprise units802-816, which perform functions or operations similar to modules720-734of computer system700ofFIG. 7, including: a high-availability unit802; a power unit804; a virtualization unit806; an encryption unit808; an instruction unit810; a recording unit812; a replaying unit814; and a communication unit816.

The foregoing embodiments described herein have been presented for purposes of illustration and description only. They are not intended to be exhaustive or to limit the embodiments described herein to the forms disclosed. Accordingly, many modifications and variations will be apparent to practitioners skilled in the art. Additionally, the above disclosure is not intended to limit the embodiments described herein. The scope of the embodiments described herein is defined by the appended claims.