Dynamic memory mapping for neural networks

A method to map a plurality of feature maps of a neural network onto a memory hierarchy includes mapping a first feature map of the plurality of feature maps to a memory in a memory hierarchy having available memory space and providing quickest access to the first feature map. The method also includes, when the first feature map expires, removing the first feature map from the memory used to store the first feature map.

TECHNICAL FIELD

This disclosure relates generally to neural networks. More specifically, this disclosure relates to dynamic memory mapping and layer fusion for neural networks.

BACKGROUND

Deep learning or Deep Neural Networks is a revolutionary force in artificial intelligence. Computers use Deep learning to make sense of infinite amounts of data in the form of images, sound, and text. Using multiple layers of neural perceptrons, computers now have the capacity to see, learn, and react to complex situations as well as if not better than humans.

Modern deep convolutional neural networks are very deep in design and can include 10s to 100s of layers. Existing deep learning framework statically map all the feature maps onto the DRAM, which implies that the memory usage grows proportionally with the depth of the network. On the other hand, each layer has to read its input feature map from DRAM to on-chip memory and write back its output feature map to DRAM thereby requiring multiple reads and writes between on-chip memory and DRAM which is both time and energy consuming.

SUMMARY

This disclosure provides dynamic memory mapping for neural networks.

In a first embodiment, a method to map a plurality of feature maps of a neural network onto a memory hierarchy is provided. The method includes mapping a first feature map of the plurality of feature maps to a memory in a memory hierarchy having available memory space and providing quickest access to the first feature map. The method also includes, when the first feature map expires, removing the first feature map of the plurality of feature maps from the memory used to store the first feature map.

In a second embodiment, an electronic device is provided. The electronic device includes at least one memory in a memory hierarchy. The electronic device also includes at least one processor coupled to the at least one memory. The at least one processor is configured to map a first feature map of a plurality of feature maps to a memory in the memory hierarchy having available memory space and providing quickest access to the first feature map. The at least one processor is also configured to, when the first feature map expires, remove the first feature map from the memory, and unallocate the memory used to store the first feature map.

In a third embodiment, a non-transitory computer readable medium embodying a computer program is provided. The computer program includes computer readable program code that, when executed by at least one processor, causes the at least one processor to map a first feature map of a plurality of feature maps to a memory in a memory hierarchy, and, when the first feature map expires, remove the first feature map from the memory, and unallocate the memory used to store the first feature map.

In a fourth embodiment, an electronic device is provided. The electronic device includes a memory including on-chip memory and off-chip memory. The electronic device also includes at least one processor coupled to the memory. The at least one processor is configured to write data from a neural network input to on-chip memory. The at least one processor is also configured to provide the data to a fused layer of a neural network, wherein the fused layer includes two or more layers that are fused to reduce data movement between the on-chip memory and the off-chip memory, and perform a fused layer operation on the data. The at least one processor is also configured to generate an output from the fused layer operation, and write the output to off-chip memory.

DETAILED DESCRIPTION

According to embodiments of the present disclosure, various methods for improving neural networks are provided. Deep neural networks can perform various functions such as image recognition, data analysis, natural language processing, intent classification, or other functions. Neural networks can generate an output based on a weighted sum of inputs, which is then passed through an activation function. The activation function is able to determine an output after summing the inputs multiplied by the weights. It will be understood by those skilled in the art that various activation functions can be used depending on the configuration of the neural network and the result to be achieved by the neural network.

The inputs, weights, and outputs can be organized within a multilayer perceptron (MLP), wherein there is an input layer, one or more hidden layers, and an output layer. A plurality of inputs, or an input vector, make up the input layer, a plurality of hidden layer neurons reside in the hidden layer or layers, and one or more outputs can be generated for the output layer. The neural network can be a feedforward network where inputs are passed from the input layer to a hidden layer. The inputs can be processed through an activation or transfer function to provide new inputs to a next hidden layer, if the neural network has multiple hidden layers, from hidden layer to hidden layer until the final hidden layer passes the final outputs to the output layer. As a neural network is trained, the weights can be adjusted based on calculated error rates to increase the accuracy of the neural network.

Convolutional neural networks can be used for image or object recognition. A convolution layer performs convolutions between an image and a kernel (a small matrix of values) to weight sections of the image based on the kernel. Convolutions can be performed on a subset of the image at a time until the full image is weighted by a kernel. Kernels using different weights can be used for additional convolutions, creating a feature map as a result of each convolution. Each feature map can then be passed to the next layer of the neural network. The next layer of a convolutional neural network can be a batch normalization layer, or Bnorm layer. The Bnorm layer can be used to normalize the activation of each convolution layer.

A convolutional neural network can also include a rectified linear units, or ReLU, layer. The ReLU layer applies an activation function to increase the nonlinear properties of the network, such as by zeroing out negative values. A convolutional neural network can also include a pooling layer, which can partition the input image into rectangles or sub-regions. Max pooling is a common method of pooling that outputs the maximum value of the sub-region. A convolutional neural network can perform any number of convolutions, batch normalizations, ReLU calculations, and pooling operations depending on the neural network. The image can be reduced down to a vector of values and a fully connected layer then takes the vector and provides one or more outputs, such as indicating whether the image matches a particular feature or object attempting to be detected. It will be appreciated that the present disclosure is not limited to any particular type of neural network and that this disclosure can be applied to any neural network to optimize the neural network.

A neural network based application, such as an object or image classification neural network running on a specific hardware, has multiple requirements, such as accuracy, execution speed, power consumption, and the like. Designing a network that meets all the requirements on a given target hardware can be challenging. A way of approaching this problem is to design a neural network having an accuracy that meets the application's requirements, and then simplifying the neural network while running the neural network on the hardware until the speed and power consumption requirements are met. However, this approach does not take into account the target hardware characteristics when designing or when simplifying the neural network.

Certain embodiments of the present disclosure provide dynamically mapping data used in a neural network onto a memory hierarchy of a device based on network topology to save memory usage for network inference and eliminate unnecessary data exchange between on-chip and off-chip memory. Certain embodiments of the present disclosure also provide a systematic optimization that fuses multiple layers into an atomic layer to avoid unnecessary data movement between on-chip and off-chip memory and to avoid memory usage from intermediate data between layers. Fusing the multiple layers into one atomic layer allows for processing the one atomic layer in on-chip memory rather than treating each layer individually and reading and writing data for each layer between on-chip and off-chip memory. Layer fusion eliminates intermediate data memory usage and reduces memory access time and power consumption.

FIG. 1illustrates an example system100according to embodiments of this disclosure. The embodiment of the system100shown inFIG. 1is for illustration only. Other embodiments of the system100could be used without departing from the scope of this disclosure.

The system100includes a network102that facilitates communication between various components in the system100. For example, network102can communicate Internet Protocol (IP) packets, frame relay frames, Asynchronous Transfer Mode (ATM) cells, or other information between network addresses. The network102includes one or more local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), all or a portion of a global network such as the Internet, or any other communication system or systems at one or more locations.

The network102facilitates communications between various server(s)104and various client devices106-114. Server104can represent one or more servers. Each server104includes any suitable computing or processing device that can provide computing services for one or more client devices. Each server104could, for example, include one or more processing devices, one or more memories storing instructions and data, and one or more network interfaces facilitating communication over the network102.

Each client device106-114represents any suitable computing or processing device that interacts with at least one server or other computing device(s) over the network102. In this example, the client devices106-114include a desktop computer106, a mobile telephone or mobile devices108(such as a smartphone), a personal digital assistant (PDA)110, a laptop computer112, and a tablet computer114. However, any other or additional client devices could be used in the system100.

In this example, some client devices108-114communicate indirectly with the network102. For example, the client devices108and110(mobile devices108and PDA110, respectively) communicate via one or more base stations116, such as cellular base stations or eNodeBs (eNBs). Mobile devices108include both smart phones and feature phones. Smart phones represent a class of mobile devices108that are a handheld device with a mobile operating system and an integrated mobile broadband cellular network connection for voice, short message service (SMS), and internet data communication. Feature phones represent a class of mobile devices108that are a midway point between a basic phone and a smart phone. Feature phones generally have voice calling and text messaging functions in addition to basic multimedia and internet capabilities. Also, the client devices112and114(laptop computer and tablet computer, respectively) communicate via one or more wireless access points118, such as IEEE 802.11 wireless access points. Note that these are for illustration only and that each client device106-114could communicate directly with the network102or indirectly with the network102via any suitable intermediate device(s) or network(s).

In certain embodiments, the mobile device108(or any other client device106-114) can transmit information securely and efficiently to another device, such as, for example, the server104. The mobile device108(or any other client device106-114) can receive information to be processed as an input(s) into a neural network. Such information can include image data, voice/audio data, geolocation data, user information, or other data received by or stored on the mobile device108. The mobile device108(or any other client device106-114) can trigger the information transmission between itself and server104. The mobile device108(or any other client device106-114) can provide a real-time result generated by a neural network.

The processes and systems provided in this disclosure allow for a client device or a server to provide a result processed by a neural network. In certain embodiments, a client device (client device106-114) can determine the neural network result. In certain embodiments, a client device (client device106-114) receives the data to be included as inputs into a neural network and transmits the data over the network102to the server104, which determines the output(s) using the neural network.

FIGS. 2 and 3illustrate example devices in a computing system in accordance with embodiments of the present disclosure. In particular,FIG. 2illustrates an example server200, andFIG. 3illustrates an example electronic device300. The server200could represent the server104inFIG. 1, and the electronic device300could represent one or more of the client devices106-114inFIG. 1.

Server200can represent one or more local servers or one or more neural network servers for processing received inputs through a trained neural network. As shown inFIG. 2, the server200includes a bus system205that supports communication between at least one processor(s)210, at least one storage device(s)215, at least one communications interface220, and at least one input/output (I/O) unit225.

The processor210executes instructions that can be stored in a memory230. The processor210can include any suitable number(s) and type(s) of processors or other devices in any suitable arrangement. Example types of processor(s)210include microprocessors, microcontrollers, digital signal processors, field programmable gate arrays, application specific integrated circuits, and discreet circuitry.

The memory230and a persistent storage235are examples of storage devices215that represent any structure(s) capable of storing and facilitating retrieval of information (such as data, program code, neural network inputs and other data, or other suitable information on a temporary or permanent basis). The memory230can represent a random access memory or any other suitable volatile or non-volatile storage device(s). The persistent storage235can contain one or more components or devices supporting longer-term storage of data, such as a ready only memory, hard drive, Flash memory, or optical disc.

The communications interface220supports communications with other systems or devices. For example, the communications interface220could include a network interface card or a wireless transceiver facilitating communications over the network102. The communications interface220can support communications through any suitable physical or wireless communication link(s).

The I/O unit225allows for input and output of data. For example, the I/O unit225can provide a connection for user input through a keyboard, mouse, keypad, touchscreen, or other suitable input device. The I/O unit225can also send output to a display, printer, or other suitable output device.

Note that whileFIG. 2is described as representing the server104ofFIG. 1, the same or similar structure could be used in one or more of the various client devices106-114. For example, a desktop computer106or a laptop computer112could have the same or similar structure as that shown inFIG. 2.

FIG. 3illustrates an electronic device300in accordance with an embodiment of this disclosure. The embodiment of the electronic device300shown inFIG. 3is for illustration only and other embodiments could be used without departing from the scope of this disclosure. The electronic device300can come in a wide variety of configurations, andFIG. 3does not limit the scope of this disclosure to any particular implementation of an electronic device. In certain embodiments, one or more of the devices104-114ofFIG. 1can include the same or similar configuration as electronic device300.

In certain embodiments, the electronic device300is useable with data transfer applications, such as providing neural network inputs or activating a function based on a neural network result or output. For example, the electronic device300can receive information, such as voice data, transfer the data to the server200, receive a response from the server200indicating the result of processing the information through a neural network, and activate a function on the electronic device300in accordance with the result. The electronic device300can be a mobile communication device, such as, for example, a wireless terminal, a desktop computer (similar to desktop computer106ofFIG. 1), a mobile device (similar to mobile device108ofFIG. 1), a PDA (similar to PDA110ofFIG. 1), a laptop (similar to laptop computer112ofFIG. 1), a tablet (similar to tablet computer114), and the like.

As shown inFIG. 3, the electronic device300includes an antenna305, a communication unit310, a transmit (TX) processing circuitry315, a microphone320, and a receive (RX) processing circuitry325. The communication unit310can include, for example, a RF transceiver, a BLUETOOTH transceiver, a WI-FI transceiver, ZIGBEE, infrared, and the like. The electronic device300also includes a speaker330, a processor340, an input/output (I/O) interface345, an input350, a display355, a memory360, a sensor(s)365, and a biometric scanner370. The memory360includes an operating system (OS)361, applications362, and user data363.

The communication unit310receives, from the antenna305, an incoming RF signal transmitted such as a BLUETOOTH or WI-FI signal from an access point (such as a base station, Wi-Fi router, Bluetooth device) of the network102(such as a Wi-Fi, Bluetooth, cellular, 5G, LTE, LTE-A, WiMAX, or any other type of wireless network). The communication unit310can down-convert the incoming RF signal to generate an intermediate frequency or baseband signal. The intermediate frequency or baseband signal is sent to the RX processing circuitry325that generates a processed baseband signal by filtering, decoding, or digitizing the baseband or intermediate frequency signal, or a combination thereof. The RX processing circuitry325transmits the processed baseband signal to the speaker330(such as for voice data) or to the processor340for further processing (such as for web browsing data and remittance).

The TX processing circuitry315receives analog or digital voice data from the microphone320or other outgoing baseband data from the processor340. The outgoing baseband data can include web data, e-mail, or interactive video game data. The TX processing circuitry315encodes, multiplexes, digitizes, or a combination thereof, the outgoing baseband data to generate a processed baseband or intermediate frequency signal. The communication unit310receives the outgoing processed baseband or intermediate frequency signal from the TX processing circuitry315and up-converts the baseband or intermediate frequency signal to an RF signal that is transmitted via the antenna305.

The processor340can include one or more processors or other processing devices and execute the OS361stored in the memory360in order to control the overall operation of the electronic device300. For example, the processor340could control the reception of forward channel signals and the transmission of reverse channel signals by the communication unit310, the RX processing circuitry325, and the TX processing circuitry315in accordance with well-known principles. The processor340is also capable of executing other applications362resident in the memory360, such as, one or more applications for remittance, fraud detection, and the like.

The processor340can execute instructions that are stored in a memory360. The processor340can include any suitable number(s) and type(s) of processors or other devices in any suitable arrangement. For example, in some embodiments, the processor340includes at least one microprocessor or microcontroller. Example types of processor340include microprocessors, microcontrollers, digital signal processors, field programmable gate arrays, application specific integrated circuits, and discreet circuitry.

The processor340is also capable of executing other processes and programs resident in the memory360, such as operations that receive, store, and timely instruct by providing image capturing and processing. The processor340can move data into or out of the memory360as required by an executing process. In some embodiments, the processor340is configured to execute plurality of applications362based on the OS361or in response to signals received from eNBs or an operator. The processor340is also coupled to the I/O interface345that provides the electronic device300with the ability to connect to other devices, such as client devices106-114. The I/O interface345is the communication path between these accessories and the processor340.

The processor340is also coupled to the input350and the display355. The operator of the electronic device300can use the input350to enter data or inputs into the electronic device300. Input350can be a keyboard, touch screen, mouse, track ball, voice input, or other device capable of acting as a user interface to allow a user in interact with electronic device300. For example, the input350can include voice recognition processing thereby allowing a user to input a voice command via microphone320. For another example, the input350can include a touch panel, a (digital) pen sensor, a key, or an ultrasonic input device. The touch panel can recognize, for example, a touch input in at least one scheme among a capacitive scheme, a pressure sensitive scheme, an infrared scheme, or an ultrasonic scheme. Input350can be associated with sensor(s)365and/or a camera by providing additional input to processor340. The camera can be used to capture images to be processed by a convolutional neural network. Such a convolutional neural network can be an application stored on the electronic device300, or on the server200, in which case the electronic device300can transmit a captured image to the server200to be processed by the neural network.

In certain embodiments, sensor365includes inertial sensors (such as, accelerometers, gyroscope, and magnetometer), optical sensors, motion sensors, cameras, pressure sensors, heart rate sensors, altimeter, breath sensors (such as microphone320), and the like. The input350can also include a control circuit. In the capacitive scheme, the input350can recognize touch or proximity. The display355can be a liquid crystal display (LCD), light-emitting diode (LED) display, optical LED (OLED), active matrix OLED (AMOLED), or other display capable of rendering text and/or graphics, such as from websites, videos, games, images, and the like.

The memory360can include persistent storage (not shown) that represents any structure(s) capable of storing and facilitating retrieval of information (such as data, program code, and/or other suitable information on a temporary or permanent basis). The memory360can contain one or more components or devices supporting longer-term storage of data, such as a ready only memory, hard drive, Flash memory, or optical disc. The memory360also can contain user data363that includes profile data and user history data. User data363can also contain data received from sensor365. User data363can biographical and biometric data.

Electronic device300further includes one or more sensor(s)365that can meter a physical quantity or detect an activation state of the electronic device300and convert metered or detected information into an electrical signal. In certain embodiments, sensor365includes inertial sensors (such as accelerometers, gyroscopes, and magnetometers), optical sensors, motion sensors, cameras, pressure sensors, heart rate sensors, altimeter, breath sensors (such as microphone320), and the like. For example, sensor365can include one or more buttons for touch input, (such as on a headset or the electronic device300), a camera, a gesture sensor, a gyroscope or gyro sensor, an air pressure sensor, a magnetic sensor or magnetometer, an acceleration sensor or accelerometer, a grip sensor, a proximity sensor, a color sensor, a bio-physical sensor, a temperature/humidity sensor, an illumination sensor, an Ultraviolet (UV) sensor, an Electromyography (EMG) sensor, an Electroencephalogram (EEG) sensor, an Electrocardiogram (ECG) sensor, an Infrared (IR) sensor, an ultrasound sensor, an iris sensor, a fingerprint sensor, and the like. The sensor365can further include a control circuit for controlling at least one of the sensors included therein. The sensor(s)365can be used to determine an orientation and facing direction, as well as geographic location of the electronic device300. Any of these sensor(s)365can be located within the electronic device300or another electronic device in communication with the electronic device300.

AlthoughFIGS. 2 and 3illustrate examples of devices in a computing system, various changes can be made toFIGS. 2 and 3. For example, various components inFIGS. 2 and 3could be combined, further subdivided, or omitted and additional components could be added according to particular needs. As a particular example, the processor340could be divided into multiple processors, such as one or more central processing units (CPUs) and one or more graphics processing units (GPUs). In addition, as with computing and communication networks, electronic devices and servers can come in a wide variety of configurations, andFIGS. 2 and 3do not limit this disclosure to any particular electronic device or server.

FIG. 4illustrates an example electronic device400in accordance with embodiments of the present disclosure. The embodiment of the electronic device400shown inFIG. 4is for illustration only and other embodiments could be used without departing from the scope of this disclosure. The electronic device400can come in a wide variety of configurations, andFIG. 4does not limit the scope of this disclosure to any particular implementation of an electronic device. In certain embodiments, the server200or the electronic device300can include the components or configuration of electronic device400.

The electronic device400includes a processor402. The processor402as illustrated includes a core404, a level-1 (L1) memory cache406, and a level-2 (L2) memory cache408. It will be understood that there can be both an L1 memory cache for instructions and one for data. In some embodiments, the processor402includes multiple cores, and the L1 memory cache406is a dedicated memory cache for the core404, while the L2 memory cache408is shared with one or more other cores of the processor402. In such embodiments, each core accesses a dedicated L1 memory cache while sharing the L2 memory cache408with the other cores. In certain embodiments, each core has a dedicated L2 memory cache408. The electronic device400also includes a level-3 (L3) memory cache410and dynamic random-access memory (DRAM)412, located off the processor chip. The L3 memory cache410and the DRAM412are interconnected with each other and to the processor402by a bus system414. In embodiments in which the processor402includes multiple cores, the L3 memory cache410can be shared by the cores.

The L1 memory cache406, the L2 memory cache408, the L3 memory cache410, and the DRAM412form a memory hierarchy. Generally, a memory closer to the core404of the processor402is faster, but includes a smaller amount of maximum available memory. For example, the L1 memory cache406can provide the fastest data retrieval time for the core404when retrieving data stored on the L1 memory cache406. Storing data that is anticipated as being used quickly and/or frequently can be stored in the L1 memory cache406to allow the core404fast access to the data. However, the L1 memory cache406may not have as much space as the other memory caches408and410or the DRAM412.

Thus, when data needs to be accessed quickly or frequently by the core404, but the data cannot fit into the L1 memory cache406, the data can be saved in either the L2 memory cache408, L3 memory cache410, or the DRAM412based on the hierarchy. For example, if the data cannot fit into the L1 memory cache406, a determination is made as to whether the data will fit into the L2 memory cache408. If so, the data is stored in the L2 memory cache. If not, the available space of the L3 memory cache410is checked to determine if the data can be stored in the L3 memory cache410. If so, the data is stored in the L3 memory cache410. If not, the data can be stored in DRAM412. Such a procedure ensures that the data used by the core404is stored in the fastest memory available.

For neural networks, data is often stored in DRAM, and then written to on-chip memory when the data is used by the processor. While the neural network is processing all inputs and generating the final output, the original data can stay in DRAM during the entire process, and new data created by the neural network can be written to DRAM while the neural network continues to operate. For example, after each hidden layer of the network creates new data to be used in the next layer, this new data can be written to DRAM, which also can still be storing the original inputs and other data previously created by the neural network. This new data can then be read from DRAM and written to on-chip memory for use by the next layer. For deep neural networks that include a large number of layers, this can create memory usage issues, as each layer can produce more data to be stored. This can also create issues with neural network performance, as every read/write between on-chip and off-chip memory decreases the speed of the neural network and increases power consumption.

AlthoughFIG. 4illustrates one example of an electronic device400, various changes can be made toFIG. 4. For example, the electronic device400could include any number of each component in any suitable arrangement. For instance, the memory caches408and410can be either on-chip or off-chip. There can also be any number of memory caches disposed either on-chip or off-chip and that can be either dedicated to a core of the processor or shared among other processors. For example, the electronic device400could include the L3 memory cache410on-chip, and can also include another L4 memory cache off-chip. In general, computing systems, processors, and processor memory come in a wide variety of configurations, andFIG. 4does not limit the scope of this disclosure to any particular configuration. WhileFIG. 4illustrates one operational environment in which various features disclosed in this patent document can be used, these features could be used in any other suitable system.

FIG. 5illustrates a flowchart of a neural network optimization framework in accordance with embodiments of this disclosure. At a block502, a neural network is configured. This network configuration takes into account the problem(s) to be solved by the neural network, the types of data for processing by the neural network, and other factors to configure the neural network to address those issues, such as determining how many layers, that is, how deep, the neural network may be. The neural network is then trained at block504and the accuracy of the neural network is ascertained at block506. Based on the results of the training and the ascertained accuracy of the neural network, the configuration of the network can be adjusted at block508to increase the accuracy of the neural network. The adjustment performed in block508can be adjustments, such as altering the number of layers of the neural network, adjusting weights of the neural network, or other adjustments; and the network is thus reconfigured at block502. Blocks502,504,506, and508can be repeated until the accuracy of the neural network meets the application's requirements.

Typically, other neural network configuration processes can perform training, accuracy, and configuration adjustment steps and then simplify the neural network until speed and power consumption requirements are met for target hardware. However, this does not take into account the characteristics of the target hardware when designing a neural network. At block510, the target hardware configuration is determined and at step512, computations are optimized and memory mapping implemented for the target hardware. The optimized computations can be achieved by layer fusion according to embodiments of the present disclosure, and dynamic memory mapping also can be implemented according to embodiments of the present disclosure, to optimally map neural network data such as feature maps onto a given memory system. At block514, latency, speed, power, and memory usage can be evaluated. The forward pass performance metrics for the hardware target also can be evaluated. At block508, the network configuration adjustment policy takes into account not only training and accuracy of the neural network, but hardware parameters as well. The network configuration thus can be adjusted and the optimal calculations and memory mapping adjusted in order to provide a more accurate neural network that also achieves less latency, higher speeds, and lower power and memory usage.

FIG. 6Aillustrates a neural network topology in accordance with embodiments of the present disclosure. The neural network topology includes a first feature map602(FM0), which can be initially stored in memory. The first feature map602can be a feature map of an input. The first feature map602is then received by a first layer604(layer0). The first layer604processes the first feature map602, such as performing convolutions on the first feature map602. As a result, the first layer604creates a second feature map606(FM1) that is stored in memory. The neural network then passes the second feature map606to a second layer608(layer1). The second layer608can perform various operations on the second feature map606, such as additional convolutions, batch normalization operations, ReLU operations, pooling operations or other operations. As a result, the second layer608creates a third feature map610(FM2).

The neural network then passes the third feature map610to a third layer612(layer2) that can perform various operations on the second feature map606, such as additional convolutions, batch normalization operations, ReLU operations, pooling operations or other operations. As a result, the third layer612creates a fourth feature map614(FM3). The neural network then passes the fourth feature map614to a fourth layer616(layer3) that can perform various operations on the second feature map606, such as additional convolutions, batch normalization operations, ReLU operations, pooling operations or other operations. As a result, the fourth layer616creates a fifth feature map618(FM4). The fifth feature map618can be used in subsequent layers of the neural network, or can be one of a series of final weights or inputs to be used by the neural network in a fully connected layer for providing the final output.

It will be understood that the topology illustrated and described with respect toFIG. 6Ais not limited to any particular neural network type or configuration. Other types of neural networks, such as networks used for voice recognition that receive a vector of input values, are applicable to the topology ofFIG. 6Aand are applicable to the other processes described herein.

The feature maps or other data such as input vectors and the results of inputs applied to weights and sent through an activation function can all be stored in memory for use by the neural network during operation of the neural network. Neural networks often map data such as feature maps in a static manner in DRAM, such that a memory location continues to store this data during operation of the neural network while potentially adding even more data to DRAM during operation of the neural network. The memory usage for a static mapped neural network thus grows proportionally with the depth of the network. Additionally, for each layer, a static mapped neural network has to read input data to on-chip memory and write out output data to DRAM.

FIG. 6Billustrates a block diagram of an example dynamic memory mapping process for a neural network in accordance with embodiments of the present disclosure. In the example, an L1 cache620and DRAM622are depicted. The L1 cache620and the DRAM622can be connected to a processor such as that described with respect to the server200or the electronic device300. In this example, at a time T0the first feature map602is mapped to the L1 cache620by the processor. The first feature map602is ready to be fed through the first layer604to create the second feature map606. The L1 cache620has enough space for both the first feature map602and the second feature map606, and so the processor prioritizes mapping the second feature map606to the L1 cache620. Prioritizing both the first feature map602and the second feature map606allows layer604to process the first feature map602and the second feature map606directly without extra data movement between the L1 cache620and the DRAM622. At time T1, the second feature map606is ready to be processed by the second layer608. Also, by time T1, the first feature map602is no longer used and so the first feature map602expires. The processor thus removes the first feature map602from the L1 cache620and unallocates memory in the L1 cache620.

At time T1, the processor determines that the second feature map606and the third feature map610will not both fit into the L1 cache620. Therefore, the processor maps the third feature map610to DRAM622. At a time T2, the second feature map606has been processed by the second layer608and the third feature map610has been created and stored in DRAM622. Thus, by T2, the second feature map606expires and the processor removes the second feature map606from the L1 cache620. At T2, the L1 cache620is now free to store additional feature maps, if the additional feature maps fit into the L1 cache620. At T2, the processor determines that the fourth feature map614will not fit into the L1 cache620. Therefore, the processor maps the fourth feature map614to DRAM622. By time T3, the processor removes the third feature map610from DRAM622as the third feature map610has already been processed by the third layer612and the processor has stored the fourth feature map614in DRAM622in the mapped DRAM location.

At time T3, the processor determines that the fifth feature map618will fit into the L1 cache620, and the processor maps the fifth feature map618to the L1 cache620. At time T3, the fourth feature map614is still stored in DRAM622. By time T4, the processor stores the fifth feature map618in the L1 cache620, and the processor removes the fourth feature map614from DRAM622, to free up memory space in DRAM622and to allow potential subsequent feature maps to be stored in DRAM622in the space previously occupied by the fourth feature map614. The process described with respect toFIG. 6Bthus allows for memory to be freed for use during operation of a neural network, while also allowing for data to be stored in memory other than DRAM that provides faster data access to the processor(s). As illustrated inFIG. 6B, the first feature map602, the second feature map606, and the fifth feature map618can potentially have overlapping memory mappings in the L1 cache620because the life spans of these feature maps in the time axis do not overlap.

Once optimal mapping of the neural network is determined, network performance metrics can be estimated, such as forward pass speed, latency, memory usage, and power consumption, given the target hardware. In the network training process, accuracy metrics can also be obtained, which can be combined with the network performance metrics to form a comprehensive performance metric. If all the metrics meet the requirements of the target application, the optimization may be complete. Otherwise, the metrics can be provided to the network configuration adjustment policy508to determine further network configuration. The network configuration adjustment policy508takes into account both the network topology and the target hardware memory hierarchy. The adjustment policy can be prioritized to simplify the layers of the neural network that are identified as performance bottlenecks in the network according to estimated network performance. The adjustment policy508can (1) try to reduce the output channels for the bottleneck layers so that the output can fit into the on-chip memory and the speed, latency, and power requirements of the target application are met. The adjusted network can then (2) be retrained to output the accuracy metric. If the accuracy requirement is met, the network configuration adjustment is finished. Otherwise, the adjustment policy can in some embodiments add an additional layer to the neural network following the bottleneck layer and repeat steps (1) and (2) until performance requirements are met.

FIG. 7illustrates a flowchart of a dynamic memory mapping process700in accordance with embodiments of the present disclosure.FIG. 7does not limit the scope of this disclosure to any particular embodiments. While process700depicts a series of sequential steps, unless explicitly stated, no inference should be drawn from that sequence regarding specific order of performance, performance of steps or portions thereof serially rather than concurrently or in an overlapping manner, or performance of the steps depicted exclusively without the occurrence of intervening or intermediate steps. For ease of explanation, the process700is described with respect to the server200ofFIG. 2and the electronic device300ofFIG. 3and is performed by a respective processor or processing device in the server200or electronic device300. However, the process700can be used with any other suitable system.

The process700begins at block702where a feature map or other data is received for use in a neural network. At decision block704, the processor determines whether the feature map fits into an L1 cache. If so, at block706the processor maps the feature map to the L1 cache. If not, at block708, the processor maps the feature map to another memory, such as another processor cache (L2, L3, L4, and so forth), DRAM, or other memory according to the memory hierarchy. The feature map is thus mapped and stored in whichever memory provides the fastest access time for the processor and that has enough space to store the feature map.

Once the feature map is stored in either the L1 cache at block704or the next fastest available memory at block708, the processor, using a neural network application, creates a next feature map at block710. At decision block712the processor determines if the L1 cache can store the next feature map. If so, at block714, the processor maps the next feature map to the L1 cache. If not, at block716, the processor maps the next feature map to another memory, such as another processor cache (L2, L3, L4, etc.), DRAM, or other memory according to the memory hierarchy. Once the next feature map is stored in either the L1 cache at block714or the next fastest available memory at block716, at block718the processor deletes the feature map received at block702from the memory which was chosen for storing the feature map at decision block704and the processor unallocates memory for that feature map. Thus, the feature map received at block702is no longer stored in memory and memory is freed for storing other data or other feature maps.

In block720, the processor determines whether all feature maps and other data have been processed by the neural network. This can be determined by whether the neural network has provided a final output(s). If so, the process700ends at block722. If the neural network has not completed processing, the process moves back to block710where a next feature map is created or received. The process continues through decision block712, and either blocks714or716to store the next feature map, to block718where the processor deletes the feature map created when block710was last encountered to free the memory to which the previous feature map was mapped. The process700then returns to decision block720where the processor determines again if processing is complete, and, if so, the process700ends at block722.

FIG. 8illustrates a block diagram of another example dynamic memory mapping process for a neural network in accordance with embodiments of the present disclosure. There is shown an L1 cache802, an L2 cache804, and DRAM806. The L1 cache802, the L2 cache804, and the DRAM806can be connected to a processor such as that described with respect to the server200or the electronic device300. In this example, at a time T0the processor maps a first feature map808(FM0) to the L1 cache802. The first feature map808is to be fed through a first layer of a neural network to create a second feature map810(FM1). The L1 cache802in this example does not have enough space for both the first feature map808and the second feature map810, and so the processor prioritizes mapping the second feature map810to the next memory that is closest to the processor. In this example, the L2 cache804has enough unallocated memory to store the second feature map810, and so the processor maps the second feature map810to the L2 cache804. It should be noted in this example that while DRAM806also had enough free space to store the second feature map810, the L2 cache804is prioritized over DRAM806because the L2 cache804is higher in the memory hierarchy and provides quicker data access to the processor.

At time T1, the second feature map810is to be processed by a second layer of the neural network in order to create a third feature map812. Also, by time T1, the first feature map808is no longer used and so the processor removes the first feature map808from the L1 cache802and the processor unallocates memory in the L1 cache802. At time T1, after a memory to which the first feature map808was mapped is unallocated, the processor determines that the third feature map812can fit into the L1 cache802. Since the L1 cache is higher in the hierarchy than the L2 cache804or the DRAM806, the processor maps the third feature map812to the L1 cache802.

At a time T2, the second feature map810has been processed by the second layer of the neural network and the processor stores the third feature map812in the L1 cache802. Thus, by T2, the second feature map810is no longer to be used by the neural network and so the processor removes the second feature map810from the L2 cache804. At T2, the processor determines that a fourth feature map814will not fit into the L1 cache802or the L2 cache804. Therefore, the processor maps the fourth feature map814to DRAM622.

By time T3, the processor removes the third feature map812from the L1 cache802as the third feature map812has already been processed by a third layer of the neural network and the processor stores the fourth feature map814in DRAM806in the mapped DRAM location. At time T3, the processor determines that a fifth feature map816will fit into the L1 cache802, and the processor maps the fifth feature map816to the L1 cache802. At time T3, the fourth feature map814is still stored in DRAM806. By time T4, the processor stores the fifth feature map816in the L1 cache802, and the processor removes the fourth feature map814from DRAM806, to free up memory space in DRAM806and to allow potential subsequent feature maps to be stored in DRAM806in the space previously occupied by the fourth feature map814. The process described with respect toFIG. 6Bthus allows for memory to be freed for use during operation of a neural network, while also allowing for data to be stored in a memory that provides faster data access to a processor(s).

The processes described with respect toFIGS. 6A-8provide dynamic memory mapping processes that map neural networks of arbitrary topologies onto a target hardware memory hierarchy to minimize the memory usage at all levels and minimize the data movement at all levels. Since each feature map has a limited life span on the time axis based on the network topology, the mapped memory can be reclaimed when it is no longer used. Memory usage is only limited by the maximum memory to accommodate one layer's input/output feature maps. Thus, deeper networks do not necessarily use more memory when using the dynamic memory mapping described herein.

The memory hierarchy can include a prioritization scheme such as L1-L2-L3-DRAM. It will be understood that the memory hierarchy can depend on the target hardware. The memory hierarchy can reduce both the time and the power consumption used to move feature maps or other data between on-chip and off-chip memory.

FIG. 9illustrates a block diagram of one embodiment of a neural network layer fusion process900in accordance with embodiments of the present disclosure.FIG. 9shows off-chip memory902and on-chip memory904. The off-chip memory902can be DRAM or other memory located off the processor die such as some forms of L3 or L4 cache memory, or other memory types. The off-chip memory902can be connected to a processor such as that described with respect to the server200or the electronic device300. The on-chip memory904can be memory such as an L1 cache or other memory located on the processor die, such as on the processor of the processor described with respect to the server200or the electronic device300.

A neural network including a fused layer906and running on the processor associated with the on-chip memory904receives data from the on-chip memory904during operation of the neural network. The fused layer906includes operations908,910, and912. The operations908,910, and912can perform operations typically performed by separate layers, but are combined into one atomic fused layer906. Fusing the layers into an atomic layer minimizes computations, data movement, memory usage, and power consumption associated with data movement and computations. In the example illustrated inFIG. 9, the operations include a convolution/bnorm operation908, a rectified linear units (ReLU) operation910, and a pooling operation912. An input feature map914(FM0) can be initially stored in the off-chip memory902. The processor can read at least one portion or tile916(FM0) of the input feature map914from off-chip memory902and write the at least one tile916to on-chip memory904for processing by the neural network. In some neural networks, a feature map FM0can be convolved in a convolution layer to generate another feature map FM1.

The feature map FM1can then be normalized by a batch normalization layer to produce yet another feature map FM2. The convolution/bnorm operation908merges convolution with batch normalization. For example, FM1can be generated by Equation 1.
FM1=conv(FM0,K)+b(1)

Since Equation 1 and Equation 2 are linear equations, the bnorm layer parameters gamma, mean, var, and beta of Equation 2 can be absorbed into K and b and FM2can be generated from one equation: Equation 3.
FM2=conv(FM0,K′)+b′(3)

Therefore, FM0can be convolved and batch normalized within one step by the convolution/bnorm operation908. The convolution/bnorm operation908creates a temporary feature map918that has been convolved and batch normalized from the at least one tile916, and the temporary feature map918is written to on-chip memory904. It will be understood that merging convolution and bnorm steps into the convolution/bnorm operation908can be performed when configuring the neural network so that the merging is not performed during operation of the neural network so that there is no impact on the network's online inference performance.

In other neural network configurations, data granularity is managed by the cache system. In certain embodiments of the present disclosure, the fused layer906manages data granularity of the data to be processed by the neural network. The fused layer906can determine how to divide the input feature map914into the at least one tile916to be processed by the fused layer906. As shown inFIG. 9, the at least one tile916will remain in on-chip memory904throughout the process until a final output is written to off-chip memory902. The convolution tile size to be used can be based on pooling and convolution kernel sizes. For example, if the convolution kernel has size K*K and pooling kernel has size P*P, the convolution tile size is chosen by Equation 4.
T=(P+K−1)  (4)

If the input feature map914has a dimension of [H,W,C] (height, width, color), then the feature map can be decomposed into [H/(P+K−1)]*[W/(P+K−1)] tiles. Once the tile size and number of tiles is determined, the number of tiles fetched from and written to on-chip memory904to be used by the fused layer906can be determined based on the size of the on-chip memory904. For example, if K=3 and P=2, then T=(2+3−1), and so T=4. Thus, the tiles for convolving will be a size of 4*4. Convolving the 4*4 tiles with the 3*3 convolution kernels produces a temporary feature map918having a size of 2*2. After the ReLU operation910, the pooling operation912can be applied to the 2*2 temporary feature map918to produce a final output of size 1. In some embodiments, the Winograd algorithm is applied to further accelerate small kernel/tile convolutions to reduce the multiply-accumulate (MAC) operation to further increase the speed of the neural network. In the above example, the Winograd algorithm can reduce the convolution MAC from 36 to 16, for a 2.25× reduction.

The ReLU operation910applies the temporary feature map918to an activation function and then can write the result back to on-chip memory904again as another temporary feature map918. In some embodiments, the temporary feature map918before the ReLU operation910can be overwritten by the result of the ReLU operation910. In some embodiments, the activation function of f(x)=max(0,x) can be used. It will be understood that other activation functions such as a leaky ReLU function, a parametric ReLU function, an exponential linear unit function (ELU), a sigmoid function, or others can be used in the ReLU operation910.

The temporary feature map918can then be passed to the pooling operation912. The pooling operation912performs a form of pooling on the temporary feature map918, such as max pooling, to create a smaller sized feature map tile920(FM4). The tile920is then written to off-chip memory922as an output or final feature map922. In some embodiments, the fused layer906is configured such that the pooling operation912outputs just one output for each temporary feature map918received by the pooling operation912, such that the tile920is one output to be used in determining the final result of the neural network. The final feature map922in this example can be a series of values, each value one of the tiles920written to off-chip memory902. These values can then be used, for example, as votes for whether an image includes an object that the neural network is designed to recognize.

In other neural networks, separate layers such as convolutional, pooling, ReLU, and batchnorm layers are treated as atomic operations, writing an output feature map from on-chip memory to off-chip memory after each layer. For example, in other neural networks, an input layer FM0can be stored in off-chip memory. Tiles from FM0are written to on-chip memory and processed by a convolutional layer to create tiles of an intermediate feature map FM1. The tiles of feature map FM1are written from on-chip memory to off-chip memory. Tiles from feature map FM1are then written back to on-chip memory to be processed by a bnorm layer, for example, creating tiles for another intermediate feature map FM2. The tiles of feature map FM2are all written from on-chip memory to off-chip memory again until all of feature map FM2is on off-chip memory. The tiles of FM2are then written from off-chip memory back into on-chip memory to be processed by, for example, a ReLU layer to create another intermediate feature map FM3.

The tiles of intermediate feature map FM3are all written back to off-chip memory as feature map FM3. Then the tiles of feature map FM3are written back into on-chip memory again to be processed by, for example, a pooling layer to create final outputs or tiles that can be a final feature map FM4. Each of these tiles is written from on-chip memory to off-chip memory until FM4is completely stored in off-chip memory. Typically, all the feature maps FM0-FM4are too large to fit into SRAM. Because of this, memory is allocated for the three intermediate feature maps FM1, FM2, and FM3, and extra data movement between on-chip and off-chip memory is performed. Other neural networks thus perform a larger number of read/write operations between on-chip and off-chip memory (reducing speed) than neural networks of the present disclosure. Other neural networks also use more memory resources to perform the functions of the neural network, such as requiring memory to store the intermediate feature maps.

The neural network of the present disclosure provides for increased speed and more efficient use of resources. Additionally, there is less data traffic between on-chip and off-chip memory. In such an optimized network, only the input feature map and the output feature map are stored in DRAM, with all temporary/intermediate feature maps remaining in on-chip memory during operation of the fused layer.

FIG. 10illustrates a flowchart of a fused layer neural network process1000in accordance with embodiments of the present disclosure.FIG. 10does not limit the scope of this disclosure to any particular embodiments. While process1000depicts a series of sequential steps, unless explicitly stated, no inference should be drawn from that sequence regarding specific order of performance, performance of steps or portions thereof serially rather than concurrently or in an overlapping manner, or performance of the steps depicted exclusively without the occurrence of intervening or intermediate steps. For ease of explanation, the process1000is described with respect to the server200ofFIG. 2and the electronic device300ofFIG. 3and is performed by a respective processor or processing device in the server200or electronic device300. However, the process1000can be used with any other suitable system.

The process1000begins at block1002. At block1002, the processor writes a first feature map or input feature map to off-chip memory. At block1004the processor determines a tile size. The tile size can be determined as described in the present disclosure and as described with respect toFIG. 11. At block1006, the processor fetches one or more tiles from the first feature map and the processor writes the one or more tiles to on-chip memory. The number of tiles to be fetched depends on the size of the on-chip memory. Space in on-chip memory will also need to be reserved for temporary feature maps and the final outputs. At block1008, the processor feeds the one or more tiles written to on-chip memory in block1006into the fused layer and the processor performs a merged convolution/bnorm operation on the one or more tiles as described in the present disclosure. At block1010, the processor writes the result of the convolution/bnorm operation to on-chip memory as a temporary feature map. It will be understood that the convolution/bnorm operation can be performed on one of the one or more tiles at a time.

At block1012, the processor performs a ReLU operation of the fused layer on the temporary feature map. At block1014, the processor writes the result of the ReLU operation to on-chip memory as a temporary feature map. The temporary feature map written to on-chip memory as a result of the ReLU operation can overwrite the temporary feature map created as a result of the convolution/bnorm operation. At block1016, the processor performs a pooling operation of the fused layer on the temporary feature map written to on-chip memory in block1014. At block1018, the processor writes the result of the pooling operation to on-chip memory. The result can be one or more tiles of a final feature map, and can be the final outputs provided by neural network, such as one or more values indicating whether an object is recognized as appearing in an image.

At block1020, the processor writes the final feature map to off-chip memory. It will be understood that each tile of the input feature map can be processed by the fused layer at a time, and each output created from that tile as a result of the pooling operation can be written by the processor to off-chip memory one at a time until all tiles from the input feature map have been processed by the fused layer and all outputs written to off-chip memory as the final feature map. At decision block1022, the processor determines whether all tiles of the first feature map have been processed by the neural network. If not, the process1000moves back to block1006to fetch one or more additional tiles from the first feature map, depending on how much free space exists in on-chip memory. Blocks1006-1022can be repeated until at decision block1022the processor determines that all tiles of the first feature map have been processed. If so, the process1000ends at block1024.

FIG. 11illustrates a flowchart of a feature map tiling and memory allocation process1100in accordance with embodiments of the present disclosure.FIG. 11does not limit the scope of this disclosure to any particular embodiments. While process1100depicts a series of sequential steps, unless explicitly stated, no inference should be drawn from that sequence regarding specific order of performance, performance of steps or portions thereof serially rather than concurrently or in an overlapping manner, or performance of the steps depicted exclusively without the occurrence of intervening or intermediate steps. For ease of explanation, the process1100is described with respect to the server200ofFIG. 2and the electronic device300ofFIG. 3and is performed by a respective processor or processing device in the server200or electronic device300. However, the process1100can be used with any other suitable system.

The process1100begins at block1102. At block1102, the processor determines a tile size to be used in the neural network. This can be performed when initially configuring the neural network, and can be adjusted as the neural network is trained and hardware considerations are taken into account. The pooling kernel size and convolution kernel size to be used can also be determined during neural network configuration. The tile size can be determined by Equation 1, T=(P+K−1).

An input feature map can be provided as an input for the neural network. At block1104, the processor determines the dimensions of the input feature map. For example, the input feature map can be an image having 32*32 pixels. At block1106, the processor determines the number of tiles to split the feature map into. The processor can perform this determination by calculating the product of the height of the feature map divided by the tile size and the width of the feature map divided by the tile size, or [H/(P+K−1)]*[(W/(P+K−1)], or (H/T)*(W/T). Using the example of a 32*32 input image, and where T=4, the image would be split into 64 tiles. At block1108, the processor determines the number of tiles that will fit in on-chip memory. The processor can analyze the amount of free space in the on-chip memory and the size of each tile to determine how many tiles can be written to on-chip memory, taking into account space requirements for the intermediate feature maps and final outputs that will be created from each of the tiles.

At block1110, processor retrieves the number of tiles determined to fit in on-chip memory and writes the tiles to on-chip memory. At block1112, at least one of the tiles written to on-chip memory is processed by the fused layer of the neural network, producing an output that can be written to off-chip memory by the processor, freeing space in on-chip memory that the at least one tile previously occupied. At decision block1114, the processor determines whether all tiles of the input feature map have been processed by the fused layer. If not, the process1100moves to decision block1116where the processor determines if there are still tiles of the input feature map in off-chip memory that need to be written to on-chip memory for processing by the fused layer. If so, the process1100moves back to block1108where the processor determines a number of tiles of the input feature map that will fit into on-chip memory. If for some reason there is not enough free space for additional tiles to be written to on-chip memory, the processor can determine a value of zero and no additional tiles may be retrieved at block1110, allowing for another tile to finish processing by the fused layer at block1112. This can free up the space used for additional tiles when block1108is repeated after the processor determines at decision block1116that there are still tiles in off-chip memory to be processed. Blocks1108,1110,1112,1114, and1116can be repeated until all tiles are written from off-chip memory into on-chip memory.

If at decision block1116the processor determines that there are no more tiles in off-chip memory to be processed, the process1100moves back to block1112to process another at least one tile by the fused layer. Blocks1112,1114, and1116can be repeated until all tiles in on-chip memory are processed by the fused layer. Once the processor determines at decision block1114that all tiles of the input feature map have been processed by the fused layer, the process1100ends at block1118.

Layer fusion can increase the speed of the neural network. For example, a processing time for a neural network without layer fusion can be 91.8 milliseconds. Applying layer fusion for a network on the same hardware can provide a processing time of 68.8 milliseconds. As a neural network becomes deeper or more complicated, layer fusion can provide even greater improvement in processing time.