Patent Description:
As artificial intelligence (Al) technology progresses, there is a desire for specialized Al hardware that may perform inference and learning through operations. Various devices are being developed as hardware dedicated to the implementation of Al.

Such dedicated hardware for Al may be embodied by, for example, a central processing unit (CPU) and a graphics processing unit (GPU), or by a field-programmable gate array (FPGA) and an application-specific integrated circuit (ASIC) that may be repurposed. <CIT> discloses an intermediate value storage within a graphics processing apparatus. A tile-based graphics processor includes tile processing circuitry that has both a tile buffer and a per-pixel general purpose data store. The per-pixel general purpose data store is read accessible and write accessible by the tile processing circuitry to store intermediate values. These intermediate values are generated by the tile processing circuitry and then consumed by the tile processing circuitry to generate the output values for the tile being processed.

<NPL> discloses a GCN accelerator with hybrid architecture.

<NPL> discloses an efficient cache memory modeling in GPUs using reuse distance analysis.

<CIT> discloses technologies for dynamic multi-core network packet processing distribution.

<NPL> discloses an accelerator for state-of-the-art deep convolutional neural networks (CNNs).

The invention is what is claimed in the independent claims.

In one general aspect, an operation method of an accelerator includes receiving one or more workloads assigned by a host controller configured to control the accelerator comprising a plurality of processing elements, determining, upon the plurality of processing elements performing the workloads, reuse data of the workloads based on at least one of hardware resource information and a memory access cost of the accelerator processing elements, and providing a result of performing the workloads to the host controller.

The determining of the reuse data may include determining the reuse data with a least number of accesses by the accelerator to an external memory when the processing elements perform the workloads.

The hardware resource information of the accelerator may include at least one of usage information of a multilevel memory included in the accelerator, usage information of the processing elements, or system cache information.

The multilevel memory may include at least one of a level <NUM> memory accessible by one of the processing elements, a level <NUM> memory accessible by a portion of the processing elements, wherein the subset is a number greater than one, or a level <NUM> memory accessible by the processing elements.

The determining of the reuse data further includes dynamically determining the reuse data based on a characteristic of the workloads.

The memory access cost includes an access cost for the external memory and an access cost for the multilevel memory included in the accelerator.

The access cost for the external memory may be greater than the access cost for the multilevel memory.

The access cost for the multilevel memory may increase for a memory portion of the multilevel memory that is shared by a greater number of processing elements among the processing elements.

At least one of the hardware resource information and the memory access cost are determined through an extension offloaded to a direct memory access (DMA) configured to control data input to the multilevel memory or data output from the multilevel memory.

The determining of the reuse data may include determining a tiling method to be applied to data input for performing a workload of the workloads based on the hardware resource information of the accelerator, and determining a size of the reuse data based on the determined tiling method.

The determining of the reuse data may include determining the reuse data for at least one of the multilevel memory and to each layer of a neural network corresponding to a workload.

The reuse data may be stored in the multilevel memory in the accelerator and may not be transmitted to the external memory of the accelerator.

Each of the processing elements may include a level <NUM> memory accessible by a corresponding processing unit, a level <NUM> DMA configured to control a data input and output of the level <NUM> memory and at least one of monitor and profile data input to or output from the level <NUM> memory, a multiplier-accumulator (MAC) configured to perform an operation involved with a workload of the workloads assigned to the processing unit, and a level <NUM> controller configured to control one of the level <NUM> memory, the level <NUM> DMA, and the MAC, or a combination thereof.

The accelerator may be included in a user terminal to which data to be recognized using a neural network based on a workload is input, or in a server configured to receive the data to be recognized from the user terminal.

In another general aspect, an accelerator includes a plurality of processing elements configured to perform one or more workloads assigned by a host controller, and a multilevel memory configured to be accessible by at least one of the processing elements. When the processing elements perform the workloads, reuse data of the workloads are determined based on at least one of hardware resource information and a memory access cost of the accelerator and stored in the multilevel memory.

In still another general aspect, an accelerator system includes an accelerator including a plurality of processing elements configured to perform one or more workloads and a multilevel memory having different access costs, an external memory of the accelerator, and a host controller configured to assign the workloads to the accelerator. When the processing elements perform the workloads, the accelerator determines reuse data of the workloads based on at least one of hardware resource information and a memory access cost of the accelerator.

In still another general aspect, an operation method of an accelerator, includes receiving workloads assigned by a host controller configured to control the accelerator comprising processing elements, dynamically determining, upon sequential workloads being performed by the processing elements, reuse data of the workloads based on any two of hardware resource information, a memory access cost of the accelerator, and characteristics of the workloads, and providing a result of performing the workloads to the host controller. The reuse data is data reutilized in a subsequent operation.

The hardware resource information of the accelerator may include usage information of a multilevel memory comprised in the accelerator. The multilevel memory may include a level <NUM> memory accessible by one of the processing elements, a level <NUM> memory accessible by a subset of the processing elements, wherein the subset is a number greater than one, and a level <NUM> memory accessible by the processing elements.

The memory access cost includes an access cost for an external memory of the accelerator and an access cost for a multilevel memory comprised in the accelerator, and the access cost for the external memory may be greater than the access cost for the multilevel memory.

The access cost for the multilevel memory may increase based on a number of the processing elements having access to each of levels <NUM>, <NUM>, and <NUM> of the multilevel memory.

However, various changes or modifications of the methods, apparatuses, and/or systems described herein, and which fall within the scope of the claims, will be apparent after an understanding of the disclosure of this application.

Unless otherwise defined, all terms, including technical and scientific terms, used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains and based on an understanding of the disclosure of the present application. Terms, such as those defined in commonly used dictionaries, are to be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the disclosure of the present application and are not to be interpreted in an idealized or overly formal sense unless expressly so defined herein.

<FIG> is a diagram illustrating an example of an accelerator system.

Referring to <FIG>, an accelerator system <NUM> may include a host controller <NUM>, an accelerator <NUM>, an external memory controller <NUM>, and an external memory <NUM>. The host controller <NUM>, the accelerator <NUM>, the external memory controller <NUM>, and the external memory <NUM> may communicate with one another through a bus <NUM>.

The host controller <NUM> may be a device configured to control respective operations of components included in the accelerator system <NUM>, and may include a central processing unit (CPU), for example. In an example, the host controller <NUM> may assign one or more workloads to the accelerator <NUM>. A workload may be an instruction that instructs the accelerator <NUM> to execute a neural network for object recognition, speech recognition, pattern recognition, computer vision, and machine translation, for example. The host controller <NUM> may assign, to the accelerator <NUM>, the workloads based on one or more requested works or tasks.

The accelerator <NUM> may be an artificial intelligence (AI) accelerator configured to execute a neural network based on an assigned workload and infer data to be input. The accelerator <NUM> may be a separate processor distinguished from the host controller <NUM>. That is, the accelerator <NUM> may simultaneously perform a single or a plurality of workloads assigned by the host controller <NUM>. The accelerator <NUM> may process a workload more effectively independent of the host controller <NUM> that may be used for general computing purposes.

The neural network may include a plurality of layers. In an example, the neural network may include an input layer, a plurality of hidden layers, and an output layer. Each of the layers may include a plurality of nodes each referred to as an artificial neuron. Each of the nodes may indicate a calculation unit having at least one input and output, and the nodes may be connected to one another. A weight may be set for a connection between nodes, and be adjusted or changed. The weight may increase, decrease, or maintain a related data value to determine an influence of the data value on a final result. To each node included in the output layer, weighted inputs of nodes included in a previous layer may be input. A process in which weighted data is input from a layer to a subsequent layer of the layer may be referred to as propagation.

Operations based on the neural network may be performed in the accelerator <NUM>. To perform the operations, a plurality of processing units (or processing elements) and a multilevel memory that are included in the accelerator <NUM> may be used. The multilevel memory may be a memory accessible by, at least, one of the processing units, and may include a scratchpad memory and/or a system cache, for example. The scratchpad memory may be an on-chip memory included in the accelerator <NUM>, for example, a static random-access memory (SRAM) that is accessible through an address space. The system cache may be a last level cache (LLC), for example. The scratchpad memory may not be larger than a dynamic RAM (DRAM) in terms of memory capacity, but have a smaller access cost than the DRAM.

Based on a characteristic of the neural network described above, a result of a current operation may be used for a subsequent operation. Thus, rapid operation processing may be enabled by storing, in the multilevel memory, data, for example, an intermediate operation result, that is to be used again in a subsequent layer, and minimizing an access cost when performing an operation. However, due to a relatively small capacity of the multilevel memory, it may not be practically possible to store all sets of required data in the multilevel memory, and a portion of the data may be stored in an external memory, for example, the external memory <NUM> illustrated in <FIG>.

The external memory <NUM> may refer to a memory disposed outside the accelerator <NUM>, and be a DRAM used as a main memory of the accelerator system <NUM>, for example. The external memory <NUM> may be accessible through the external memory controller <NUM>. When a memory present inside the accelerator <NUM> is insufficient for the accelerator <NUM> to perform the workloads, the external memory <NUM> may be used.

The external memory <NUM> may have a capacity larger than that of the multilevel memory inside the accelerator <NUM>. However, memory accessing cost for from the accelerator <NUM> to the external memory <NUM> may be greater than a memory accessing cost for such an internal multilevel memory. Such memory accessing cost may indicate an amount of power and/or time that is used to access, read, or write data from or in the memory.

Thus, to improve performance of the accelerator system <NUM>, it may be desirable to minimize accesses from the accelerator <NUM> to the external memory <NUM>. That is, it may be possible to minimize accesses to the external memory <NUM> by storing, in the multilevel memory instead of the external memory <NUM>, data required for the accelerator <NUM> to perform the workloads and/or data to be used again in a subsequent layer, for example, intermediate operation result data. Such data to be stored in the multilevel memory in the accelerator <NUM> may be referred to herein as reuse data.

Hereinafter, examples will be further described in detail.

<FIG> is a diagram illustrating an example of an accelerator.

Referring to <FIG>, an accelerator <NUM> includes a plurality of processing units (or processing elements) and a multilevel memory accessible by at least one of the processing units. The multilevel memory may be a collective description of a level (LV) <NUM> memory <NUM>, an LV1 memory <NUM>, and an LV2 memory <NUM> in the accelerator <NUM>.

A processing unit <NUM>, which is one of the processing units, may include an LV0 memory <NUM>, an LV0 direct memory access (DMA) <NUM>, a multiplier-accumulator (MAC) <NUM>, and an LV0 controller <NUM>. The processing unit <NUM> may be a neural processing unit (NPU), a graphics processing unit (GPU), a CPU, a tensor processing unit (TPU), or the like.

The LV0 memory <NUM> may be a memory accessible through a corresponding processing unit <NUM>. That is, the LV0 memory <NUM> may be accessible only through the processing unit <NUM>, among the processing units, included in the accelerator <NUM>.

The LV0 DMA <NUM> may control input data and/or output data of the LV0 memory <NUM>, according to an instruction from the LV0 controller <NUM>. The LV0 DMA <NUM> may read data from the LV0 memory <NUM> or write data in the LV0 memory <NUM> based on information associated with a source, a destination, and a data size that are included in the instruction from the LV0 controller <NUM>.

In this example, data to be input to the LV0 memory <NUM> or output from the LV0 memory <NUM> may be monitored and/or profiled. The monitoring and/or profiling may be performed by the LV0 DMA <NUM> or a separate element. Through the monitoring and/or profiling, an access cost of the LV0 memory <NUM>, usage information of the LV0 memory <NUM>, and a type of data stored in the LV0 memory <NUM> may be verified. For example, the LV0 DMA <NUM> may verify a usage percentage indicated by the usage information of the LV0 memory <NUM>, and which workload is involved with the data stored in the LV0 memory <NUM>. Hereinafter, examples will be further described based on a case in which the monitoring and/or profiling is performed by the LV0 DMA <NUM> for the convenience of description.

The MAC <NUM> may perform an operation involved with a workload assigned to the processing unit <NUM>. For example, the MAC <NUM> may perform a multiply-accumulate operation on a given data. In addition, the MAC <NUM> may apply an activation function to the given data. The activation function may be sigmoid, hyperbolic tangent (tanh), or a rectified linear unit (ReLU), for example.

The LV0 controller <NUM> may be a device configured to control components included in the processing unit <NUM>. For example, the LV0 controller <NUM> may control the LV0 memory <NUM>, the LV0 DMA <NUM>, and the MAC <NUM>.

The foregoing description of the processing unit <NUM> may be applied to each of the processing units included in the accelerator <NUM>. That is, the accelerator <NUM> includes processing units that each performs operations independently.

In an example, each "n" processing units, among the processing units, may cluster together. In this example, n is a natural number greater than <NUM> and less than the number of the processing units included in the accelerator <NUM>. That is, a subset of the processing units included in the accelerator <NUM> may work together to form a cluster, for example, a processing unit cluster <NUM>.

The processing units included in the cluster <NUM> may share one LV1 memory <NUM>. That is, the LV1 memory <NUM> may be accessible by the processing units in the cluster <NUM>. For example, even though operations respectively performed in a first processing unit and a second processing unit, among the processing units, in the cluster <NUM> are different from each other, a portion of data required for both operations may be common. The common data is stored in the LV1 memory <NUM>, rather than it is stored in an LV0 memory included in each of the first processing unit and the second processing unit. Thus, the first processing unit and the second processing unit share the common data, thereby, improving the overall system operation efficiency. In the example of <FIG>, each of the processing units may access an LV1 memory adjacent to each of the processing units.

In an example, an LV1 DMA configured to control a data input and output of the LV1 memory <NUM> may monitor and/or profile data input to or output from the LV1 memory <NUM>. In addition, there may also be an LV1 controller to control the LV1 memory <NUM> and the LV1 DMA.

In addition, an entirety <NUM> of the processing units may share the LV2 memory <NUM>. That is, the LV2 memory <NUM> may be accessible by the processing units included in accelerator <NUM>. For example, there may be processing units that share a portion of data required to perform an operation, although not clustering together to form a same cluster, among the processing units included in the accelerator <NUM>. In this example, such processing units may not share the data through an LV1 memory, but effectively share the common data through the LV2 memory <NUM>, thereby increasing the overall operation efficiency. Similarly, in an example, an LV2 DMA configured to control a data input and output of the LV2 memory <NUM> may monitor and/or profile data input to or output from the LV2 memory <NUM>. In addition, there is also an LV2 controller to control the LV2 memory <NUM> and the LV2 DMA.

As described above, each of the processing units may access a corresponding LV0 memory, an LV1 memory adjacent to each of the processing units, and an LV2 memory of the accelerator <NUM>, and use these memories to perform respectively assigned workloads. The accelerator <NUM> may include the multilevel memory having hierarchical memories. In an example, each of an LV0 memory, an LV1 memory, and an LV2 memory may be a scratchpad memory and/or a system cache having a lower access cost than a DRAM, which is an external memory.

In addition, a DMA and a controller included in the accelerator <NUM> may be of a hierarchical multilevel type.

In the example of <FIG>, the processing units included in the accelerator <NUM> may simultaneously perform four workloads. For example, a workload with a relatively greater operation amount may be assigned to a greater number of processing units and processed therein, and a workload with a relatively less operation amount may be assigned to a smaller number of processing units and processed therein.

It is illustrated in <FIG> for the convenience of description that, in a non-limiting example, each eight processing units of <NUM> processing units cluster together to form a total of eight clusters, and three level memories are used to perform the four workloads (Workload <NUM>, Workload <NUM>, Workload <NUM>, and Workload <NUM>). However, various numbers of processing units, workloads, and levels may be applied without restriction.

<FIG> is a diagram illustrating an example of a multilevel memory and an example of an external memory.

In <FIG>, an LV0 memory <NUM>, an LV1 memory <NUM>, an LV2 memory <NUM>, an external memory <NUM>, a DMA <NUM> are illustrated in terms of their functionalities for the convenience of description.

The LV0 memory <NUM>, the LV1 memory <NUM>, and the LV2 memory <NUM> may be disposed as a global buffer (GLB) in an accelerator. The LV2 memory <NUM> may be a memory shared by a plurality of processing units included in the accelerator, and the LV1 memory <NUM> may be a memory shared by some of the processing units. The LV0 memory <NUM> may be included in a processing unit and not be shared with another processing unit. In the accelerator, there are the LV0 memory <NUM> provided in number corresponding to the number of the processing units included in the accelerator, the LV1 memory <NUM> provided in number corresponding to the number of clusters of the processing units, and the LV2 memory <NUM> provided as a single LV2 memory.

The external memory <NUM> may be an off-chip memory disposed outside the accelerator and include, for example, a DRAM, a three-dimensional (3D) memory such as a high bandwidth memory (HBM), and a processing in memory (PIM). The external memory <NUM> may also be referred to herein as an LV3 memory for the convenience of description.

The LV0 memory <NUM>, the LV1 memory <NUM>, the LV2 memory <NUM>, and the external memory <NUM> may be used when a workload is performed in a processing unit, and a memory access cost may differ for each level. For example, the memory access cost may increase as the level increases. That is, an access cost of the LV0 memory <NUM> may be the lowest, and an access cost of the external memory <NUM> may be the highest.

The DMA <NUM> is also illustrated in terms of its functionality. A DMA may be separately provided for each level, and used to read or write data from or in a corresponding level memory. For example, there are an LV0 DMA configured to control input data and/or output data of the LV0 memory <NUM>, an LV1 DMA configured to control input data and/or output data of the LV1 memory <NUM>, an LV2 DMA configured to control input data and/or output data of the LV2 memory <NUM>, and an LV3 DMA configured to control input data and/or output data of the external memory <NUM>, separately. The LV0 memory <NUM>, the LV1 memory <NUM>, the LV2 memory <NUM>, and the external memory <NUM> may exchange data with one another through the DMAs provided for respective levels.

<FIG> is a diagram illustrating an example of determining reuse data.

In the example <FIG>, an LV0 memory <NUM>, an LV1 memory <NUM>, an LV2 memory <NUM>, an external memory <NUM>, a DMA <NUM>, a reuse data policy maker <NUM>, and a host controller <NUM> are illustrated in terms of their functionalities for the convenience of description.

Through an extension of the DMA <NUM>, data input to a corresponding level memory or data output from the level memory are monitored and/or profiled, and a result of the monitoring and/or profiling is transmitted to the host controller <NUM>. The extension may indicate that a function of such monitoring and/or profiling of input and output data of a corresponding level memory is additionally provided, or off-loaded, in the DMA <NUM>.

The DMA <NUM> verifies usage information of a corresponding level memory. For example, an LV2 DMA may verify usage information <NUM> of the LV2 memory <NUM> for each workload. The usage information <NUM> may include, for example, information associated with a use rate of the LV2 memory <NUM> and an amount used, for each workload. In addition, an LV1 DMA may verify usage information <NUM> of the LV1 memory <NUM>. In another example of <FIG>, an LV0 DMA may verify usage information of the LV0 memory <NUM>. Such memory usage information for each level that is verified by respective level DMAs may be transmitted to the host controller <NUM>.

The host controller <NUM> assigns one or more workloads to an accelerator based on the memory usage information for each level. For example, when there is a relatively large amount of multilevel memory available because a multilevel memory of the accelerator is not used much, for example, the host controller <NUM> may assign a greater number of workloads or assign a workload having a greater operation amount. In contrast, when there is a relatively small amount of multilevel memory available because the multilevel memory of the accelerator is used much, for example, the host controller <NUM> may assign a smaller number of workloads or assign a workload having a less operation amount.

When the workloads are assigned by the host controller <NUM>, the reuse data policy maker <NUM> may determine reuse data of the workloads. The reuse data policy maker <NUM> may operate in a controller configured to control the DMA <NUM>, and be embodied as a controller that is separately included for each level as in the DMA <NUM>. For example, an LV2 controller configured to control the LV2 DMA may determine reuse data to be stored in the LV2 memory <NUM>. In addition, an LV1 controller configured to control the LV1 DMA may determine reuse data to be stored in the LV1 memory <NUM>. Similarly, an LV0 controller configured to control the LV0 DMA may determine reuse data to be stored in the LV0 memory <NUM>.

The reuse data may be determined based on hardware resource information and/or a memory access cost of the accelerator. The reuse data may refer to data that is utilized again in a subsequent operation when operations are performed in sequential order according to the workloads, and is to be stored in the multilevel memory in the accelerator. For example, data that is more frequently used in subsequent operations may be stored in a memory having a lower access cost and then used afterward.

The hardware resource information of the accelerator may include usage information of the multilevel memory included in the accelerator, and usage information of a plurality of processing units included in the accelerator. The usage information of the multilevel memory may indicate availability and/or usage information of a memory for each level. The usage information of the processing units may include usage efficiency information of each processing unit.

In an example, based on the hardware resource information of the accelerator, a tiling method that is to be applied to data input to perform a workload may be determined. The tiling method may refer to a method by which, when input data is too large to be processed at one time, the input data is divided into a plurality of tiles and each tile is processed. When dividing the input data for performing the workload into the tiles, a tile size may be determined based on the hardware resource information of the accelerator. For example, when there is a great amount of available hardware resources of the accelerator, the tile size may be determined to be large. In contrast, when there is a less amount of available hardware resources of the accelerator, the tile size may be determined to be small. According to the tiling method determined as described above, a size of the reuse data may be determined. For example, the size of the reuse data may be determined to correspond to the tile size.

The memory access cost includes an access cost for the external memory <NUM> of the accelerator and for the multilevel memory of the accelerator. For example, the access cost for the external memory <NUM> may be greater than the access cost for the multilevel memory. The access cost for the multilevel memory may increase for a memory shared by a greater number of processing units. For example, the LV2 memory <NUM> may be shared by all the processing units included in the accelerator, and thus have a greatest access cost among memories of the multilevel memory. In contrast, the LV0 memory <NUM> may not be shared by another processing unit, but be accessible only by a corresponding processing unit, and thus have a lowest access cost among the memories of the multilevel memory.

For example, data that is frequently used in a subsequent operation or work may be determined to be reuse data to be stored in a lower-level memory; however, only a smaller number of processing units may have access to the lower-level memory. Thus, based further on the number of processing units that require the data and whether the processing units are grouped together to form a same cluster, it is possible to determine whether to determine the data to be the reuse data, and determine which level memory the reuse data is to be stored.

Additionally, a characteristic of the assigned workloads considered to determine the reuse data. For example, a workload among the workloads assigned to the accelerator may need to be rapidly processed, while another workload may only need to be processed within a preset time. Thus, to enhance the overall system efficiency, reuse data for the workload that needs to be processed as fast as possible may be assigned in a higher priority at an initial step, and then reuse data for the workload that needs to be processed within a preset time may be assigned in a higher priority as the preset time approaches. As described above, a processing schedule for each workload may be additionally considered, and thus reuse data may be dynamically determined.

As described above, the reuse data may be determined for each memory of the multilevel memory. For example, reuse data may be determined for each of the LV0 memory <NUM>, the LV1 memory <NUM>, and the LV2 memory <NUM>. In addition, the reuse data may be determined for each layer of a neural network corresponding to a workload. For example, when operations are performed in sequential order along a plurality of layers included in the neural network, reuse data suitable for each of the layers may be determined. For example, data that is more frequently utilized again in subsequent layers may be determined to be reuse data having a higher priority. In this example, data associated with the operations based on the neural network may include an input feature map (IFM), an output feature map (OFM), and a weight (WGHT). In addition, when the operations are performed in sequential order along the layers, the type of data to be selected as the reuse data may change dynamically. For example, reuse data determined in frontal layers may have a higher proportion of IMF. In contrast, reuse data determined in rearward layers may have a higher proportion of WGHT. Thus, the type of data to be selected as reuse data may change dynamically based on a workload, a neural network, and/or an operation.

Based on the factors described above, reuse data that minimizes an access to an external memory of an accelerator when a plurality of processing units of the accelerator performs one or more workloads may be determined. The reuse data may be determined such that the access to the external memory is minimized for all the workloads assigned to the accelerator and for all the processing units, not for a certain workload or a portion of the processing units.

<FIG> and <FIG> are diagrams illustrating examples of an accelerator system.

Referring to <FIG>, a server <NUM> may include a host controller <NUM>, an accelerator <NUM>, and an external memory <NUM>. The host controller <NUM> may assign one or more workloads to the accelerator <NUM>. When the accelerator <NUM> performs the workloads through a plurality of processing units, the accelerator <NUM> determines at least, one set of reuse data based on hardware resource information and/or a memory access cost. The accelerator <NUM> performs the workloads using the reuse data, and provides a result of performing the workloads. In an example, the server <NUM> may be an accelerator system.

Referring to <FIG>, a user terminal <NUM> may include a host controller <NUM>, an accelerator <NUM>, and an external memory <NUM>. Each of these components may perform respective operations described herein. Although the user terminal <NUM> is illustrated as a smartphone in <FIG> for the convenience of description, the description provided above with reference to <FIG> may also be applicable to various computing devices such as a personal computer (PC), a tablet PC, and a laptop, various wearable devices such as a smart watch and smart eyeglasses, various home appliances such as a smart speaker, a smart TV, and a smart refrigerator, and other devices such as a smart vehicle, a smart kiosk, and Internet of things (IoT) devices, without restriction.

As described above, an accelerator (e.g., the accelerators <NUM> and <NUM>) may be included in a user terminal (e.g., the user terminal <NUM>) to which data to be recognized using a neural network based on a workload is input, or in a server (e.g., the server <NUM>) configured to receive the data to be recognized from the user terminal.

<FIG> is a flowchart illustrating an example of an operation method of an accelerator.

An operation method to be described hereinafter may be performed by an accelerator.

Referring to <FIG>, in operation <NUM>, the accelerator receives from a host controller, one or more workloads assigned by the host controller.

In operation <NUM>, when performing the workloads in a plurality of processing units included in the accelerator, the accelerator determines reuse data of the workloads based on hardware resource information and/or a memory access cost of the accelerator. When the processing units perform the workloads, the accelerator may determine the reuse data that minimizes an access to an external memory of the accelerator. The accelerator dynamically determines the reuse data additionally based on characteristics of the workloads. The reuse data may refer to data that is stored in an internal multilevel memory of the accelerator and may not be transmitted to the external memory of the accelerator.

The hardware resource information of the accelerator may include usage information of the multilevel memory included in the accelerator and usage information of the processing units. The memory access cost includes an access cost for the external memory and an access cost for the multilevel memory included in the accelerator.

In operation <NUM>, the accelerator provides a result of performing the workloads. The accelerator may transmit the result of performing the workloads to the host controller.

What has been described above with reference to <FIG> may be applied to the operations described above with reference to <FIG>, and thus a more detailed and repeated description will be omitted here for brevity.

The accelerator, the accelerator system <NUM>, <NUM>, accelerator <NUM>, <NUM>, <NUM>, host controller <NUM>, <NUM>, <NUM>, <NUM>, external memory controller <NUM>, external memory <NUM>, <NUM>, <NUM>, processing unit <NUM>, LV0 memory <NUM>, <NUM>, LV1 memory <NUM>, <NUM>, LV2 memory <NUM>, <NUM>, external memory <NUM>, <NUM>, DMA <NUM>, <NUM>, reuse data policy maker <NUM>, server <NUM>, and other apparatuses, units, modules, devices, and other components described herein with respect to <FIG> are implemented by hardware components. Examples of hardware components that may be used to perform the operations described in this application where appropriate include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

While this disclosure includes specific examples, it will be apparent after an understanding of the disclosure of this application that various changes in form and details may be made in these examples without departing from the scope of the claims. The examples described herein are to be considered in a descriptive sense only, and not for purposes of limitation. Descriptions of features or aspects in each example are to be considered as being applicable to similar features or aspects in other examples.

Claim 1:
A method of operating an accelerator, comprising:
receiving (<NUM>) one or more workloads assigned by a host controller configured to control the accelerator comprising a plurality of processing elements that each perform operations independently, wherein workloads of the one or more workloads with a relatively greater or lower operation amount are assigned to a greater or smaller number of processing elements, respectively, wherein operations are performed in sequential order according to the workloads;
determining (<NUM>), upon the plurality of processing elements performing the one or more workloads, reuse data of the one or more workloads based on at least one of hardware resource information and a memory access cost of the accelerator, wherein the determining of the reuse data further comprises dynamically determining the reuse data based on a characteristic of the one or more workloads, wherein the memory access cost comprises an access cost for an external memory (<NUM>) of the accelerator and an access cost for a multilevel memory (<NUM>-<NUM>) comprised in the accelerator, wherein the at least one of the hardware resource information and the memory access cost is determined through an extension offloaded to a direct memory access, DMA, (<NUM>) configured to control data input to the multilevel memory comprised in the accelerator or data output from the multilevel memory, wherein through the extension of the DMA (<NUM>), data input to a corresponding level memory or data output from the level memory are monitored and/or profiled, wherein the DMA (<NUM>) verifies usage information of a corresponding level memory, wherein the usage information is transmitted to the host controller and wherein the host controller assigns the workloads based on the usage information; and
providing (<NUM>) a result of performing the one or more workloads to the host controller, wherein the reuse data is data reutilized in a subsequent operation.