Patent Description:
In artificial intelligence (Al) technology, independent hardware dedicated to AI may perform inference and learning through predetermined operations.

A hardware accelerator may be used to efficiently process a deep neural network (DNN) due to the number of operations on complex input data. In particular, the memory bandwidth and the latency or delay time may cause a significant performance bottleneck in many process systems. <CIT> relates to a neural network device for neural network operation, method of operating same, and application processor including the neural network device. The neural network device includes a direct memory access, DMA, controller for receiving floating-point data from a memory, a data converter for converting the floating-point data received through the DMA controller to integer-type data, and a processor for performing a neural network operation based on an integer operation by using the integer-type data provided from the data converter. <CIT> relates to neural network activation compression with non-uniform mantissas. Apparatus and methods for training a neural network accelerator using quantized precision data formats are disclosed, and in particular for storing activation values from a neural network in a compressed format having lossy or non-uniform mantissas for use during forward and backward propagation training of the neural network. <CIT> relates to Fp16-s7e8 mixed-precision for deep learning and other algorithms. A processor includes fetch circuitry to fetch a compress instruction having fields to specify locations of a source vector having N single-precision formatted elements, and a compressed vector having N neural half-precision, NHP, formatted elements, decode circuitry to decode the fetched compress instruction, and an execution circuitry to respond to the decoded compress instruction.

The invention is what is claimed in the independent claims.

In one general aspect, a method of operating a storage device includes storing received input data of a first format, converting the input data into a second format for an operation to be performed on the input data of the second format using an operator included in the storage device, and re-storing the input data of the second format.

The converting may include converting the input data of the first format into the second format by applying any one or any combination of any two or more of type converting, quantization, dequantization, padding, packing, and unpacking to the input data of the first format.

The second format may have a lower memory bandwidth than the first format.

The operation to be performed on the input data may be a low precision operation performable in the second format and has a lower precision than a high precision operation performable in the first format.

The operation to be performed on the input data may be performed in the second format by the operator or an accelerator receiving the in put data of the second format from the storage device.

The operation to be performed on the input data may be one of operations that are performed by a neural network configured to infer the input data.

The method may further include converting result data of the operation performed on the input data into the first format, and outputting the result data of the first format.

The operator may be disposed adjacent to a bank configured to store data in the storage device.

The operator may include an arithmetic logic unit (ALU) configured to perform a predetermined operation.

The input data may include at least one of image data of the first format captured by an image sensor, and data of the first format processed by a host processor configured to control either one or both of the storage device and an accelerator connected to the storage device.

The storage device may be a dynamic random-access memory (DRAM) located outside an accelerator that performs the operation.

The storage device may be included in a user terminal into which data to be inferred through a neural network that performs the operation are input or a server that receives the data to be inferred from the user terminal.

The first format may be a <NUM>-bit floating point (FP32) format and the second format may be a <NUM>-bit floating point (FP16) format or an <NUM>-bit integer (INT8) format.

In another general aspect, a storage device includes a bank configured to store received input data of a first format, and an operator disposed adjacent to the bank and configured to convert the input data into a second format for an operation to be performed on the input data of the second format, wherein the input data of the second format may be re-stored in the bank.

The operator may be configured to convert the input data of the first format into the second format by applying any one or any combination of any two or more of type converting, quantization, dequantization, padding, packing, and unpacking to the input data of the first format.

The second format may be a lower memory bandwidth than the first format.

The operation to be performed on the input data may be a low precision operation performable in the second format and may have a lower precision than a high precision operation performable in the first format.

The operator may be configu red to convert result data of the operation performed on the input data into the first format, and the bank may be configured to store the result data of the first format.

An electronic device may include the storage device.

In still another general aspect, an electronic device includes a storage device configured to store received input data of a first format, convert the input data of the first format into a second format for an operation to be performed through an internal operator of the storage device, and re-store the input data of the second format, and an accelerator configured to perform the operation on the input data of the second format received from the storage device.

The storage device may include the internal operator configured to convert the input data of the first format into the second format by applying any one or any combination of any two or more of type converting, quantization, dequantization, padding, packing, and unpacking to the input data of the first format.

The accelerator may be configured to perform an inference operation on the input data of the second format received from the storage device.

Also, descriptions of features that are known after understanding of the disclosure of this application may be omitted for increased clarity and conciseness.

Spatially relative terms such as "above," "upper," "below," and "lower" may be used herein for ease of description to describe one element's relationship to another element as shown in the figures. Such spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, an element described as being "above" or "upper" relative to another element will then be "below" or "lower" relative to the other element. Thus, the term "above" encompasses both the above and below orientations depending on the spatial orientation of the device. The device may also be oriented in other ways (for example, rotated <NUM> degrees or at other orientations), and the spatially relative terms used herein are to be interpreted accordingly.

Unless otherwise defined, all terms, including technical and scientific terms, used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. Terms, such as those defined in commonly used dictionaries, are to be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art, and are not to be interpreted in an idealized or overly formal sense unless expressly so defined herein.

<FIG> illustrates an example of an electronic device.

Referring to <FIG>, an electronic device <NUM> may include a host processor <NUM>, a storage device <NUM>, a memory controller <NUM>, and an accelerator <NUM>. The host processor <NUM>, the storage device <NUM>, the memory controller <NUM>, and the accelerator <NUM> may communicate with each other through a bus.

The host processor <NUM> is a device that controls operations of components included in the electronic device <NUM>, and may include, for example, a central processing unit (CPU) and/or a graphics processing unit (GPU). The host processor <NUM> may receive a request to process a neural network in the accelerator <NUM>, and generate an instruction executable by the accelerator <NUM> in response to the request. The request is for data inference based on a neural network, and may cause the accelerator <NUM> to execute the neural network to obtain a data inference result for object recognition, pattern recognition, computer vision, speech recognition, machine translation, machine interpretation, and the like. The host processor <NUM> may transmit inference target data and parameters of the neural network to the accelerator <NUM>.

The storage device <NUM> is an off-chip memory disposed outside the accelerator <NUM> and may be, for example, a dynamic random-access memory (DRAM) utilized as a main memory of the electronic device <NUM>. The storage device <NUM> may be accessed through the memory controller <NUM>. The storage device <NUM> may store the inference target data and/or the parameters of the neural network to be executed by the accelerator <NUM>, and the stored data may be transmitted to the accelerator <NUM> for later inference. In addition, the storage device <NUM> may be utilized when the on-chip memory in the accelerator <NUM> is insufficient to execute the neural network in the accelerator <NUM>.

The accelerator <NUM> may be an AI accelerator that infer input data by executing the neural network based on instructions from the host processor <NUM>. The accelerator <NUM> may be a separate processor different from the host processor <NUM>. For example, the accelerator <NUM> may be a neural processing unit (NPU), a GPU, or a tensor processing unit (TPU).

The accelerator <NUM> may process tasks that may be more efficiently processed by a separate exclusive processor (that is, the accelerator <NUM>), rather than by the general-purpose host processor <NUM>, due to the characteristics of the operations of the neural network. In this example, the on-chip memory and one or more processing elements (PEs) included in the accelerator <NUM> may be utilized. The on-chip memory is a global buffer included in the accelerator <NUM> and may be distinguished from the storage device <NUM> disposed outside the accelerator <NUM>. For example, the on-chip memory may be a scratchpad memory, a static random-access memory (SRAM), or the like that is accessible through an address space.

The neural network may include an input layer, a plurality of hidden layers, and an output layer. Each of the layers may include a plurality of nodes, also called artificial neurons. Each node is a calculation unit having one or more inputs and an output, and the nodes may be connected to each other. A weight may be set for a connection between nodes, and the weight may be adjusted or changed. The weight amplifies, reduces, or maintains a relevant data value, thereby determining a degree of influence of the data value on a final result. Weighted inputs of nodes included in a previous layer may be input into each node included in the output layer. A process of inputting weighted data from a predetermined layer to the next layer is referred to as propagation.

The parameters of the neural network, including the weights described above, may be learned in advance. Such learning may be performed in a high-precision format (for example, a <NUM>-bit floating point) to secu re the accu racy of the neural network. The train ing operation of the neural network may be performed by an independent device other than the accelerator <NUM> that performs data inference. However, examples are not limited thereto, and the neural network may be trained by the accelerator <NUM>. Various known training techniques may be applied to the training of the neural network without limitation, and detailed descriptions thereof will be omitted.

The parameters of the trained neural network may be transmitted to the accelerator <NUM>, and the accelerator <NUM> may perform data inference based on the neural network. In such an example, the accelerator <NUM> is a dedicated hardware for executing the trained neural network to obtain a data inference result. The accelerator <NUM> may operate in a low-precision format (for example, an <NUM>-bit integer) that may reduce overhead within an allowable accuracy loss and obtain an operation efficiency to obtain the inference result quickly by analyzing a relatively large volume of data. For example, the accelerator <NUM> may perform neural network-based operations (for example, multiply and accumulate (MAC) operations) in the <NUM>-bit integer format.

To execute a high-precision trained neural network in an accelerator <NUM> that operates in a low-precision format, high-precision data parameter may be desired to be converted into a low-precision data parameter. If the format conversion is performed by the accelerator <NUM> after the high-precision data is transferred to the accelerator <NUM>, the high-precision data may be desired to be transmitted to the accelerator <NUM> using a high memory bandwidth. Thus, an inevitable memory bandwidth loss may occur, which results in software overhead. Further, an additional operation for the format conversion in the accelerator <NUM> may be desired, which may lead to an increase in the operation quantity and response time. Accordingly, it may be more efficient for the storage device <NUM> storing the data of the high-precision format to convert the data into the low-precision format and then, transmit the data of the low-precision format to the accelerator <NUM>. Herein, it is noted that use of the term 'may' with respect to an example or embodiment, e.g., as to what an example or embodiment may include or implement, means that at least one example or embodiment exists where such a feature is included or implemented while all examples and embodiments are not limited thereto.

Input data to be inferred may be data processed by the host processor <NUM>. The host processor <NUM> is general-purpose hardware for performing various processing and may operate in a high-precision format for operation accuracy. Further, the input data to be inferred may be image data of a high-precision format captured by an image sensor (for example, a high-resolution camera). In order for the accelerator <NUM> to perform inference on the input data of the high-precision format processed by the host processor <NUM> or on the image data of the high-precision format captured by the image sensor, the data may be desired to be converted into a low-precision format. Similarly, it may be more efficient, in terms of the memory bandwidth or the operation quantity for the accelerator <NUM>, for the storage device <NUM> storing the data processed by the host processor <NUM> to perform such data format conversion and then transmit the input data of the low-precision format to the accelerator <NUM>.

Hereinafter, examples will be described in more detail.

<FIG> illustrates an example of a storage device and an operation thereof.

Referring to <FIG>, a block diagram illustrating a storage device and a flowchart illustrating an operation of the storage device are shown.

In the block diagram of <FIG>, a storage device may include banks, a top bank interface, a write read input output (WRIO) interface, a decoder, a controller, one or more registers, a program register, one or more arithmetic logic units (ALUs), and a bottom bank interface. However, the storage device is not limited to the example shown in the block diagram of <FIG>, and may be applied to various combinations of components and various numbers of components without limitation.

The even and odd banks are areas configured to store data, and may be areas that are distinguished from each other by memory addresses. The top bank interface and the WRIO interface may control the input and output of data stored in the even bank. The decoder may interpret an instruction to determine what type of operation is the format conversion to be performed through the one or more ALUs, and transmit the result to the controller so that the one or more ALUs may perform the determined operation under the control of the controller. A program for the format conversion performed by the one or more ALUs may be stored in the program register, and data subject to the format conversion may be stored in the register. The one or more ALUs may convert the format of the data stored in the register according to the program stored in the program register under the control of the controller. The one or more ALUs may be operators including an adder, a multiplier, and the like to perform an operation in accordance with an instruction. The bottom bank interface may store the data of the format converted by the one or more ALUs in the odd bank.

In this way, when a bank configured to store data and one or more ALUs configured to perform format conversion are disposed adjacent to each other in the same storage device, the cost for memory access and the internal memory bandwidth may be minimized.

In the flowchart of <FIG>, in operation <NUM>, input data subject to format conversion may be written in the bank. In this example, the input data may be data received from a host processor or an external device (for example, an image sensor). In operation <NUM>, format conversion to be performed may be identified, and a corresponding program and parameters may be written in the one or more registers. The program and parameters may include information about an operation to be performed, an address value at which data subject to the operation are stored, an address value at which a result of performing the operation is to be stored, and the like. In operation <NUM>, an operation to be performed may be identified according to a fetch instruction included in the program. If the fetch instruction corresponds to any one of type converting, quantization/dequantization, padding, and packing/unpacking, operation <NUM> may follow. If the program ends, operation <NUM> may follow.

In operation <NUM>, at least one of type converting, quantization/dequantization, padding, and packing/unpacking may be performed by the one or more ALUs in the storage device. These operations correspond to pre-processing and/or post-processing for the neural network-based inference operation. Thus, when the operations are performed by the one or more ALUs in the storage device, the system throughput may effectively improve. In addition, these operations may be performed by relatively simple one or more ALUs. Thus, even when the one or more ALUs are included in the storage device, the area or size of the storage device may increase relatively less.

Type converting refers to conversion between a <NUM>-bit floating point (FP32) format and a <NUM>-bit floating point (FP16) format. When type converting is performed, data may be converted from one format to another format. Quantization refers to converting the <NUM>-bit floating point format to an <NUM>-bit integer format, and dequantization refers to converting the <NUM>-bit integer (INT8) format to the <NUM>-bit floating point format. Data padding refers to adding, to data to be processed, a predetermined bit value (for example, "<NUM>" or "<NUM>") or a predetermined bit pattern (for example, a bit pattern mirroring the last bit included in the data) so that the data have a size suitable for an operation unit of hardware, if the data are not suitable for the operation unit. Data packing refers to merging multiple data in a low-precision format to process data converted from a high-precision format (for example, FP32) to a low-precision format (for example, FP16) according to an operation unit (for example, FP32) of hardware. Data unpacking is an operation opposite to packing, and refers to dividing packed data into two or more.

In operation <NUM>, a result of performing one of type converting, quantization, dequantization, padding, packing, and unpacking by the one or more ALUs may be written in the bank. Then, operation <NUM> may be performed again.

Format conversion may be performed in a manner of performing type converting or quantization/dequantization first, followed by packing or padding. However, format conversion is not limited thereto and may be performed in various combinations.

In operation <NUM>, output data may be read out from a subsequent system after the program ends.

The format conversion described above may be applied to data pre-processing and/or post-processing, thereby minimizing software overhead and memory bandwidth, and maximizing the utilization of an accelerator of a low-precision format.

<FIG> illustrate examples of operations of storage devices.

Referring to <FIG>, an example of performing pre-processing on input data by a storage device <NUM> is illustrated. For example, when image data of a high-precision format (for example, FP32) captured by an image sensor <NUM> are input, the storage device <NUM> may first store the input data in the high-precision format. An operator <NUM> in the storage device <NUM> may perform pre-processing prior to transmitting the in put data to an accelerator <NUM> for inference. For example, the operator <NUM> may perform quantization to convert the input data of the high-precision format into a low-precision format (for example, INT8). In addition, the operator <NUM> may perform packing and/or padding on the quantized data to process the input data to correspond to an operation unit of the accelerator <NUM>. The input data pre-processed by the operator <NUM> may be stored in the storage device <NUM> and transmitted to the accelerator <NUM> for inference by the accelerator <NUM>. Since the data are transmitted from the storage device <NUM> to the accelerator <NUM> in the low-precision format, rather than the high-precision format received from the image sensor <NUM>, the memory bandwidth usage may be effectively minimized. For example, the data may be transmitted from the storage device <NUM> to the accelerator <NUM> in an INT8 format, rather than an FP32 format, whereby the memory bandwidth may be reduced to <NUM>/<NUM>. The accelerator <NUM> that operates in a low-precision format may apply an inference operation to the received input data without performing a separate format conversion. The result data may be transmitted from the accelerator <NUM> to the storage device <NUM> in the low-precision format and stored in the storage device <NUM>. In this example, the memory bandwidth usage may be effectively reduced.

Referring to <FIG>, an example of performing pre-processing and an operation on data by a storage device <NUM> is illustrated. For example, an operator <NUM> in the storage device <NUM> may perform a predetermined operation in addition to the format conversion of data. In this example, the operation performed by the operator <NUM> may be performed in a predetermined precision format (for example, FP16). In this case, input data of a high-precision format (for example, FP32) received from a host processor <NUM> may be stored in the storage device <NUM> and then converted into an operation format for the operator <NUM>. The operator <NUM> may convert the input data of the high-precision format into a predetermined precision format. In some examples, the operator <NUM> may additionally perform packing on the converted data to process the input data to correspond to an operation unit of the operator <NUM>. A predetermined operation may be performed by the operator <NUM> based on the pre-processed data. In addition, the operator <NUM> may perform dequantization and/or unpacking to convert operation result data of the predetermined precision format into a high-precision format, and store the converted data in the storage device <NUM>. The result data of the high-precision format may be transmitted back to the host processor <NUM>, and a subsequent operation may be performed thereon.

Referring to <FIG>, an example of performing pre-processing on input data by a storage device <NUM> is illustrated. For example, when data of a high-precision format (for example, FP32) processed by a host processor <NUM> are input, the storage device <NUM> may first store the input data in the high-precision format. An operator <NUM> in the storage device <NUM> may perform pre-processing on the input data prior to transmitting the input data to an accelerator <NUM> for inference. For example, the operator <NUM> may perform quantization to convert the input data of the high-precision format into a low-precision format (for example, INT8). In addition, the operator <NUM> may perform packing and/or padding on the quantized data to process the input data to correspond to an operation unit of the accelerator <NUM>. The input data pre-processed by the operator <NUM> may be stored in the storage device <NUM> and transmitted to the accelerator <NUM> for inference. Result data of the operation performed by the accelerator <NUM> in a low-precision format may be transmitted back to the storage device <NUM>. The operator <NUM> may perform dequantization to convert the result data of the low-precision format into a high-precision format so that the result data are processed by the host processor <NUM>. In addition, in some cases, the operator <NUM> may additionally perform unpacking and/or padding so that the result data may be processed to correspond to an operation unit of the host processor <NUM>. The result data post-processed by the operator <NUM> may be transmitted to the host processor <NUM>, and the host processor <NUM> may directly process the result data without performing a separate format conversion.

<FIG> illustrates an example of a method of operating a storage device.

Referring to <FIG>, an operation method performed by a processor of a storage device is illustrated.

In operation <NUM>, the storage device stores received input data of a first format. For example, the storage device may be a DRAM located outside an accelerator that performs an operation.

In operation <NUM>, the storage device converts the input data into a second format for an operation to be performed on the input data, through an operator included in the storage device. For example, the storage device may convert the input data of the first format into the second format by applying any one or any combination of type converting, quantization, dequantization, padding, packing, and unpacking to the input data of the first format. In this example, the second format may have a lower memory bandwidth than the first format.

In operation <NUM>, the storage device re-stores the input data of the second format.

The descriptions provided with reference to <FIG> may apply to the operations shown in <FIG>, and thus further detailed descriptions will be omitted.

<FIG> illustrates an example of a storage device.

Referring to <FIG>, a storage device <NUM> includes a bank <NUM> configured to store received input data of a first format, and an operator <NUM> disposed adjacent to the bank <NUM> to convert the input data into a second format for an operation to be performed on the input data. In this example, the input data of the second format are re-stored in the bank <NUM>. The bank <NUM> and the operator <NUM> may communicate with each other through a bus <NUM>.

In the storage device <NUM>, the operator <NUM> may be implemented in the form of an in-memory chip and mounted on a mobile system or a server. Alternatively, the operator <NUM> may be mounted in the form of a software development kit (SDK) provided along with an in-memory chip. Further, the storage device <NUM> may be implemented as a memory for a server system for a data center or a memory for a mobile device or a smart home appliance (for example, a smart TV) and mounted on an electronic device together with an accelerator configured to operate in an FP16 and/or INT8 format.

The descriptions provided with reference to <FIG> may apply to the elements shown in <FIG>, and thus further detailed descriptions will be omitted.

<FIG> and <FIG> illustrate examples of electronic devices.

Referring to <FIG>, an electronic device may be implemented as a user terminal <NUM>. <FIG> illustrates the user terminal <NUM> as a smart phone for ease of description. However, the description may also apply, without limitation, to various computing devices such as a tablet, a laptop and a personal computer, various wearable devices such as a smart watch and smart glasses, various home appliances such as a smart speaker, a smart TV and a smart refrigerator, a smart car, a smart kiosk, Internet of things (IoT) device, a robot, and the like. The user terminal <NUM> may obtain data to be inferred directly by using a neural network and store the data in a storage device <NUM>. The storage device <NUM> may perform pre-processing to convert input data of a high-precision format into input data of a low-precision format, and transmit the pre-processed input data to an accelerator <NUM>. The accelerator <NUM> may directly apply an inference operation to the received input data without performing a separate format conversion, and then transmit result data back to the storage device <NUM>. The storage device <NUM> may perform post-processing to convert the input data of the low-precision format into a high-precision format. The user terminal <NUM> may provide the result data of the high-precision format stored in the storage device <NUM> to a user, or may perform a subsequent operation based on the result data through a host processor.

Referring to <FIG>, an electronic device may be implemented as a server <NUM>. The server <NUM> is a separate device different from a user terminal controlled by a user, and may communicate with the user terminal through a wired and/or wireless network. Data to be inferred by using a neural network may be collected by a user terminal, transmitted to the server <NUM> through a network, and stored in the storage device <NUM>. As described above, the storage device <NUM> may perform pre-processing or post-processing on the data prior to transmitting the data to the accelerator <NUM> or after receiving an inference result. The server <NUM> may return the inference result to the user terminal, and the user terminal may simply provide the user with the inference result received from the server <NUM>, or perform a subsequent operation based on the inference result.

The electronic device <NUM>, host processor <NUM>, <NUM>, <NUM>, storage device <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, memory controller <NUM>, accelerator <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, image sensor <NUM>, operator, <NUM>,<NUM>, <NUM>, <NUM>, blank <NUM>, user terminal <NUM>, and server <NUM>, electronic device, host processor, storage device, memory controller, accelerator, image sensor, operator, blank, user terminal, and server in <FIG> that perform the operations described in this application are implemented by hardware components configured to perform the operations described in this application that are performed by the hardware components. Examples of hardware components that may be used to perform the operations described in this application where appropriate include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

Claim 1:
A method of operating a storage device, the method comprising:
storing (<NUM>) received input data of a first format in the storage device (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>);
converting (<NUM>) the input data into a second format for an operation to be performed on the input data of the second format, wherein the converting uses an operator (<NUM>, <NUM>, <NUM>, <NUM>) included in the storage device (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>); and
re-storing (<NUM>) the input data of the second format in the storage device (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>),
wherein the converting (<NUM>) comprises converting the input data of the first format into the second format by applying any one or any combination of any two or more of type converting, quantization, dequantization, padding, packing, and unpacking to the input data of the first format.