Patent Publication Number: US-2022215235-A1

Title: Memory system to train neural networks

Description:
TECHNICAL FIELD 
     The present disclosure relates generally to semiconductor memory and methods, and more particularly, to apparatuses, systems, and methods for a memory system to train neural networks. 
     BACKGROUND 
     Memory devices are typically provided as internal, semiconductor, integrated circuits in computers or other electronic systems. There are many different types of memory including volatile and non-volatile memory. Volatile memory can require power to maintain its data (e.g., host data, error data, etc.) and includes random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), synchronous dynamic random access memory (SDRAM), and thyristor random access memory (TRAM), among others. Non-volatile memory can provide persistent data by retaining stored data when not powered and can include NAND flash memory, NOR flash memory, and resistance variable memory such as phase change random access memory (PCRAM), resistive random access memory (RRAM), and magnetoresistive random access memory (MRAM), such as spin torque transfer random access memory (STT RAM), among others. 
     Memory devices may be coupled to a host (e.g., a host computing device) to store data, commands, and/or instructions for use by the host while the computer or electronic system is operating. For example, data, commands, and/or instructions can be transferred between the host and the memory device(s) during operation of a computing or other electronic system. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a functional block diagram in the form of an apparatus including a host and a memory device in accordance with a number of embodiments of the present disclosure. 
         FIG. 2A  is a functional block diagram in the form of an apparatus including a memory system in accordance with a number of embodiments of the present disclosure. 
         FIG. 2B  is another functional block diagram in the form of an apparatus including a memory system in accordance with a number of embodiments of the present disclosure. 
         FIG. 3  is a functional block diagram in the form of an apparatus including a memory system that includes a plurality of memory devices in accordance with a number of embodiments of the present disclosure. 
         FIG. 4  is a flow diagram corresponding to a memory system to train neural networks in accordance with a number of embodiments of the present disclosure. 
         FIG. 5  is a flow diagram representing an example method corresponding to a memory system to train neural networks in accordance with a number of embodiments of the present disclosure. 
     
    
    
     DETAILED DESCRIPTION 
     Methods, systems, and apparatuses related to a memory system to train neural networks are described. For example, data management and training of one or more neural networks may be accomplished within multiple memory devices of a memory system. Neural networks may thus be trained in the absence of specialized circuitry and/or in the absence of vast computing resources. A method includes performing at least a portion of a training operation for the neural network, on a first memory device, by determining one or more weights for a hidden layer of the neural network and writing the data corresponding to the neural network to a second memory device. The method further includes performing, using the data corresponding to the neural network written to the second memory device, at least a second portion of the training operation for the neural network by determining one or more weights for the hidden layer of the neural network. 
     A neural network can include a set of instructions that can be executed to recognize patterns in data. Some neural networks can be used to recognize underlying relationships in a set of data in a manner that mimics the way that a human brain operates. A neural network can adapt to varying or changing inputs such that the neural network can generate a best possible result in the absence of redesigning the output criteria. 
     A neural network can consist of multiple neurons, which can be represented by one or more equations. In the context of neural networks, a neuron can receive a quantity of numbers or vectors as inputs and, based on properties of the neural network, produce an output. For example, a neuron can receive X k  inputs, with k corresponding to an index of input. For each input, the neuron can assign a weight vector, W k , to the input. The weight vectors can, in some embodiments, make the neurons in a neural network distinct from one or more different neurons in the network. In some neural networks, respective input vectors can be multiplied by respective weight vectors to yield a value, as shown by Equation 1, which shows and example of a linear combination of the input vectors and the weight vectors. 
       ƒ( x   1   ,x   2 )= w   1   x   1   +w   2   x   2    Equation 1
 
     In some neural networks, a non-linear function (e.g., an activation function) can be applied to the value ƒ(x 1 , x 2 ) that results from Equation 1. An example of a non-linear function that can be applied to the value that results from Equation 1 is a rectified linear unit function (ReLU). Application of the ReLU function, which is shown by Equation 2, yields the value input to the function if the value is greater than zero, or zero if the value input to the function is less than zero. The ReLU function is used here merely used as an illustrative example of an activation function and is not intended to be limiting. Other non-limiting examples of activation functions that can be applied in the context of neural networks can include sigmoid functions, binary step functions, linear activation functions, hyperbolic functions, leaky ReLU functions, parametric ReLU functions, softmax functions, and/or swish functions, among others. 
       ReLU( x )=max( x, 0)   Equation 2
 
     During a process of training a neural network, the input vectors and/or the weight vectors can be altered to “tune” the network. In one example, a neural network can be initialized with random weights. Over time, the weights can be adjusted to improve the accuracy of the neural network. This can, over time yield a neural network with high accuracy. 
     Neural networks have a wide range of applications. For example, neural networks can be used for system identification and control (vehicle control, trajectory prediction, process control, natural resource management), quantum chemistry, general game playing, pattern recognition (radar systems, face identification, signal classification, 3D reconstruction, object recognition and more), sequence recognition (gesture, speech, handwritten and printed text recognition), medical diagnosis, finance (e.g. automated trading systems), data mining, visualization, machine translation, social network filtering and/or e-mail spam filtering, among others. 
     Due to the computing resources that some neural networks demand, in some approaches, neural networks are deployed in a computing system, such as a host computing system (e.g., a desktop computer, a supercomputer, etc.) or a cloud computing environment. In such approaches, data to be subjected to the neural network as part of an operation to train the neural network can be stored in a memory resource, such as a NAND storage device, and a processing resource, such as a central processing unit, can access the data and execute instructions to process the data using the neural network. Some approaches may also utilize specialized hardware such a field-programmable gate array or an application-specific integrated circuit as part of neural network training. 
     In contrast, embodiments herein are directed to data management and training of one or more neural networks within multiple memory devices. For example, embodiments herein are directed to performance of at least a portion of an operation to train a neural network in one memory device (e.g., one type of memory device) followed by performance of at least another portion of the operation to train the neural network in a different memory device (e.g., a different type of memory device). In some embodiments, the memory devices can have different characteristics (e.g., performance characteristics, bandwidth characteristics, capacity characteristics, data retention characteristics, persistence characteristics, etc.) and/or can include different types of media (e.g., media that have different memory cell structures, materials, architectures, etc.). 
     By performing different stages of neural network training while the neural network is stored in different types of memory devices, training of neural networks can be optimized in comparison to approaches in which the neural network is stored in a same type of memory device during training. For example, by leveraging characteristics of different types of memory devices, as described herein, a neural network can be trained in multiple stages that are performed while the neural network is stored in a memory device that is optimized for each stage of the training process. 
     One example of this is a neural network that is initially (e.g., partially) trained using a memory device that exhibits high capacity but low bandwidth (e.g., a NAND memory device) and then subsequently trained using a high bandwidth memory (e.g., a 3D stacked SDRAM memory device). By leveraging the capacity of a memory device that exhibits high capacity but low bandwidth, training operations involving large sets of training data can be performed to initially train the neural network. However, once initial training operations have been performed on the neural network, it may be beneficial to write the neural network to a high bandwidth memory device where further training operations can be performed more quickly than in the high capacity memory device. Accordingly, embodiments herein can optimize an amount of time, processing resources, and/or power consumed in training of neural networks by utilizing multiple memory devices during the training process. 
     In the following detailed description of the present disclosure, reference is made to the accompanying drawings that form a part hereof, and in which is shown by way of illustration how one or more embodiments of the disclosure may be practiced. These embodiments are described in sufficient detail to enable those of ordinary skill in the art to practice the embodiments of this disclosure, and it is to be understood that other embodiments may be utilized and that process, electrical, and structural changes may be made without departing from the scope of the present disclosure. 
     As used herein, designators such as “N,” “M,” etc., particularly with respect to reference numerals in the drawings, indicate that a number of the particular feature so designated can be included. It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only, and is not intended to be limiting. As used herein, the singular forms “a,” “an,” and “the” can include both singular and plural referents, unless the context clearly dictates otherwise. In addition, “a number of,” “at least one,” and “one or more” (e.g., a number of memory banks) can refer to one or more memory banks, whereas a “plurality of” is intended to refer to more than one of such things. 
     Furthermore, the words “can” and “may” are used throughout this application in a permissive sense (i.e., having the potential to, being able to), not in a mandatory sense (i.e., must). The term “include,” and derivations thereof, means “including, but not limited to.” The terms “coupled” and “coupling” mean to be directly or indirectly connected physically or for access to and movement (transmission) of commands and/or data, as appropriate to the context. The terms “data” and “data values” are used interchangeably herein and can have the same meaning, as appropriate to the context. 
     The figures herein follow a numbering convention in which the first digit or digits correspond to the figure number and the remaining digits identify an element or component in the figure. Similar elements or components between different figures may be identified by the use of similar digits. For example,  104  may reference element “ 04 ” in  FIG. 1 , and a similar element may be referenced as  204  in  FIG. 2 . A group or plurality of similar elements or components may generally be referred to herein with a single element number. For example, a plurality of reference elements  126 - 1  to  126 -N (or, in the alternative,  126 - 1 , . . . ,  126 -N) may be referred to generally as  126 . As will be appreciated, elements shown in the various embodiments herein can be added, exchanged, and/or eliminated so as to provide a number of additional embodiments of the present disclosure. In addition, the proportion and/or the relative scale of the elements provided in the figures are intended to illustrate certain embodiments of the present disclosure and should not be taken in a limiting sense. 
       FIG. 1  is a functional block diagram in the form of a computing system  100  including an apparatus including a host  102  and a memory system  104  in accordance with a number of embodiments of the present disclosure. As used herein, an “apparatus” can refer to, but is not limited to, any of a variety of structures or combinations of structures, such as a circuit or circuitry, a die or dice, a module or modules, a device or devices, or a system or systems, for example. The memory system  104  can include a number of different memory devices  126 - 1  to  126 -N, which can include one or more memory modules (e.g., single in-line memory modules, dual in-line memory modules, etc.). The memory system  104  can include volatile memory and/or non-volatile memory. In a number of embodiments, memory system  104  can include a multi-chip device. A multi-chip device can include a number of different memory devices  126 - 1  to  126 -N, which can include a number of different memory types and/or memory modules. For example, a memory system can include non-volatile or volatile memory on any type of a module. As shown in  FIG. 1 , the apparatus  100  can include control circuitry  120 , which can include logic circuitry  122  and a memory resource  124 . Each of the components (e.g., the host  102 , the control circuitry  120 , the logic circuitry  122 , the memory resource  124 , and/or the memory devices  126 - 1  to  126 -N can be separately referred to herein as an “apparatus.” The control circuitry  120  and/or the logic circuitry  122  may be referred to as a “processing device” or “processing unit” herein. 
     The memory system  104  can provide main memory for the computing system  100  or could be used as additional memory and/or storage throughout the computing system  100 . The memory system  104  can include one or more memory devices  126 - 1  to  126 -N, which can include volatile and/or non-volatile memory cells. At least one of the memory devices  126 - 1  to  126 -N can be a flash array with a NAND architecture, for example. Embodiments are not limited to a particular type of memory device. For instance, the memory system  104  can include RAM, ROM, DRAM, SDRAM, PCRAM, RRAM, and flash memory, among others. 
     In embodiments in which the memory system  104  includes non-volatile memory, the memory system  104  can include any number of memory devices  126 - 1  to  126 -N that can include flash memory devices such as NAND or NOR flash memory devices. Embodiments are not so limited, however, and the memory system  104  can include other non-volatile memory devices  126 - 1  to  126 -N such as non-volatile random-access memory devices (e.g., NVRAM, ReRAM, FeRAM, MRAM, PCM), “emerging” memory devices such as resistance variable (e.g., 3-D Crosspoint (3D XP)) memory devices, memory devices that include an array of self-selecting memory (SSM) cells, etc., or any combination thereof. 
     Resistance variable memory devices can perform bit storage based on a change of bulk resistance, in conjunction with a stackable cross-gridded data access array. Additionally, in contrast to many flash-based memories, resistance variable non-volatile memory can perform a write in-place operation, where a non-volatile memory cell can be programmed without the non-volatile memory cell being previously erased. In contrast to flash-based memories and resistance variable memories, self-selecting memory cells can include memory cells that have a single chalcogenide material that serves as both the switch and storage element for the memory cell. 
     In some embodiments, the memory devices  126 - 1  to  126 -N include different types of memory. For example, the memory device  126 - 1  can be a 3D XP memory device and the memory device  126 -N can be a volatile memory device, such as a DRAM device, or vice versa. Embodiments are not so limited, however, and the memory devices  126 - 1  to  126 -N can include any type of memory devices provided that at least two of the memory devices  126 - 1  to  126 -N include different types of memory. 
     As illustrated in  FIG. 1 , a host  102  can be coupled to the memory system  104 . In a number of embodiments, the memory system  104  can be coupled to the host  102  via one or more channels (e.g., channel  103 ). In  FIG. 1 , the memory system  104  is coupled to the host  102  via channel  103  and control circuitry  120  of the memory system  104  is coupled to the memory devices  126 - 1  to  126 -N via channel(s)  105 - 1  to  105 -N. In some embodiments, each of the memory devices  126 - 1  to  126 -N are coupled to the control circuitry  120  by one or more respective channels  105 - 1  to  105 -N such that each of the memory devices  126 - 1  to  126 -N can receive messages, commands, requests, protocols, or other signaling that is compliant with the type of memory device  126 - 1  to  126 -N coupled to the control circuitry  120 . 
     The memory devices  126 - 1  to  126 -N can include respective processing units  123 - 1  to  123 -N. The processing units can be any kind of processor and/or co-processors that are resident on the memory devices  126 - 1  to  126 -N and operable to execute instructions to cause performance of operations involving data that is stored by the memory device  126 - 1  to  126 -N on which the processing unit  123 - 1  to  123 -N is deployed. As used herein, the term “resident on” refers to something that is physically located on a particular component. For example, the processing unit  123 - 1  to  123 -N being “resident on” the memory device  126 - 1  to  126 -N refers to a condition in which a particular processing unit (e.g., the processing unit  123 - 1 ) is physically coupled to, or physically within, a particular memory device (e.g., the memory device  126 - 1 ). The term “resident on” may be used interchangeably with other terms such as “deployed on” or “located on,” herein 
     The host  102  can be a host system such as a personal laptop computer, a desktop computer, a digital camera, a smart phone, a memory card reader, and/or an internet-of-things (IoT) enabled device, among various other types of hosts. The host  102  can include a system motherboard and/or backplane and can include a memory access device, e.g., a processor (or processing device). One of ordinary skill in the art will appreciate that “a processor” can intend one or more processors, such as a parallel processing system, a number of coprocessors, etc. The system  100  can include separate integrated circuits or one or more of the host  102 , the memory system  104 , the control circuitry  120 , and/or the memory devices  126 - 1  to  126 -N can be on the same integrated circuit. The computing system  100  can be, for instance, a server system and/or a high-performance computing (HPC) system and/or a portion thereof. Although the example shown in  FIG. 1  illustrate a system having a Von Neumann architecture, embodiments of the present disclosure can be implemented in non-Von Neumann architectures, which may not include one or more components (e.g., CPU, ALU, etc.) often associated with a Von Neumann architecture. 
     The memory system  104 , which is shown in more detail in  FIG. 2 , herein, can include control circuitry  120 , which can include logic circuitry  122  and a memory resource  124 . The logic circuitry  122  can be provided in the form of an integrated circuit, such as an application-specific integrated circuit (ASIC), field programmable gate array (FPGA), reduced instruction set computing device (RISC), advanced RISC machine, system-on-a-chip, or other combination of hardware and/or circuitry that is configured to perform operations described in more detail, herein. In some embodiments, the logic circuitry  122  can comprise one or more processors (e.g., processing device(s), processing unit(s), etc.) 
     The logic circuitry  122  can perform operations to control writing of one of more neural networks within the memory devices  126 - 1  to  126 -N, as described in more detail below. In addition to, or in the alternative, the logic circuitry  122  can perform operations to control training and execution of the one or more neural networks written within the memory devices  126 - 1  to  126 -N, as described herein. 
     In a non-limiting example, the control circuitry  120  can perform operations to control writing of a neural network to a particular memory device (e.g., the memory device  126 - 1 ) and/or control performance of various operations to partially train the neural network. Continuing with this example, the control circuitry  120  can perform operations to control writing of the neural network to a different memory device (e.g., the memory device  126 -N) and/or control performance of various operations to continue training the neural network. In some embodiments, the particular memory device can be a high capacity memory device, while the different memory device can be a high bandwidth memory device, although embodiments are not so limited. As discussed in more detail herein, the control circuitry  120  can cause performance of such operations to minimize resource consumption of the computing system  100 , improve efficiency of training the neural network, and/or leverage characteristics of particular memory devices  126 - 1  to  126 -N that may improve performance in training at least certain portions of the neural network (e.g., the neural network  225  illustrated in  FIGS. 2A and 2B , herein). 
     The control circuitry  120  can further include a memory resource  124 , which can be communicatively coupled to the logic circuitry  122 . The memory resource  124  can include volatile memory resource, non-volatile memory resources, or a combination of volatile and non-volatile memory resources. In some embodiments, the memory resource can be a random-access memory (RAM) such as static random-access memory (SRAM). Embodiments are not so limited, however, and the memory resource can be a cache, one or more registers, NVRAM, ReRAM, FeRAM, MRAM, PCM), “emerging” memory devices such as resistance variable memory resources, phase change memory devices, memory devices that include arrays of self-selecting memory cells, etc., or combinations thereof. In some embodiments, the memory resource  124  can serve as a cache for the logic circuitry  122 . 
     The embodiment of  FIG. 1  can include additional circuitry that is not illustrated so as not to obscure embodiments of the present disclosure. For example, the memory system  104  can include address circuitry to latch address signals provided over I/O connections through I/O circuitry. Address signals can be received and decoded by a row decoder and a column decoder to access the memory system  104  and/or the memory devices  126 - 1  to  126 -N. It will be appreciated by those skilled in the art that the number of address input connections can depend on the density and architecture of the memory system  104  and/or the memory devices  126 - 1  to  126 -N. 
       FIG. 2A  is a functional block diagram in the form of an apparatus including a memory system  204  in accordance with a number of embodiments of the present disclosure. The control circuitry  220 , the memory devices  226 - 1  to  226 -N, and/or the neural network  225  can be referred to separately or together as an apparatus. As used herein, an “apparatus” can refer to, but is not limited to, any of a variety of structures or combinations of structures, such as a circuit or circuitry, a die or dice, a module or modules, a device or devices, or a system or systems, for example. The memory system  204  can be analogous to the memory system  104  illustrated in  FIG. 1 , while the control circuitry  220  can be analogous to the control circuitry  120  illustrated in  FIG. 1 . 
     As discussed above, the control circuitry  220  can control writing of the neural network  225  (e.g., an untrained neural network or partially trained neural network) to at least one of the memory devices  226 - 1  to  226 -N. In the example illustrated in  FIG. 2A , the control circuitry  220  can control writing of the neural network  225  to the memory device  226 - 1 . Once the neural network  225  is written to (e.g., stored in) the memory device  226 - 1 , the control circuitry  220  (e.g., the logic circuitry  222  of the control circuitry  220 ) can control performance of operations to train the neural network  225 . For example, the control circuitry  220  can perform operations to determine one or more weights for a hidden layer of the neural network  225 . 
     The neural network  225  can be a feed-forward neural network or a back-propagation neural network. Embodiments are not so limited, however, and the neural network  225  can be a perceptron neural network, a radial basis neural network, a deep feed forward neural network, a recurrent neural network, a long/short term memory neural network, a gated recurrent unit neural network, an auto encoder (AE) neural network, a variational AE neural network, a denoising AE neural network, a sparse AE neural network, a Markov chain neural network, a Hopfield neural network, a Boltzmann machine (BM) neural network, a restricted BM neural network, a deep belief neural network, a deep convolution neural network, a deconvolutional neural network, a deep convolutional inverse graphics neural network, a generative adversarial neural network, a liquid state machine neural network, an extreme learning machine neural network, an echo state neural network, a deep residual neural network, a Kohonen neural network, a support vector machine neural network, and/or a neural Turing machine neural network, among others. 
     In some embodiments, the control circuitry  220  can perform operations to determine one or more first weights for a hidden layer of the neural network  225  as part of performance of operations to at least partially train the neural network  225 . That is, in some embodiments, the control circuitry  220  can perform operations to partially but not fully train the neural network  225  while the neural network  225  is stored within the memory device  226 - 1 . In embodiments in which the control circuitry  220  performs operations to partially train the neural network  225  while the neural network  225  is stored in the memory device  226 - 1 , the control circuitry  220  can, prior to writing the neural network  225  to the memory device  226 - 1 , determine that characteristics of the memory device  226 - 1  are conducive for partially training the neural network  225 . 
     That is, in some embodiments, at least a portion of the operation to train the neural network  225  within the memory device  226 - 1  to can be performed while the neural network  225  is stored within the memory device  226 - 1  based on characteristics of the memory device  226 - 1  such as the type of media employed by the memory device  226 - 1 , the bandwidth of the memory device  226 - 1 , and/or the speed of the memory device  226 - 1 , among others. In some embodiments, once the operation(s) to train the neural network  225  have been initiated, training operations can be performed within the memory device  226 - 1  in the absence of additional commands from the control circuitry  220  and/or a host (e.g., the host  102  illustrated in  FIG. 1 , herein). 
     In some embodiments, the control circuitry  220  can control, in the absence of signaling generated by circuitry external to the memory system  204 , performance of the operations to cause the untrained or partially trained neural network  225  to be trained. By performing neural network training in the absence of signaling generated by circuitry external to the memory system  204  (e.g., by performing neural network training within the memory system  204  or “on chip”), data movement to and from the memory system  204  can be reduced in comparison to approaches that do not perform neural network training within the memory system  204 . This can allow for a reduction in power consumption in performing neural network training operations and/or a reduction in dependence on a host computing system (e.g., the host  102  illustrated in  FIG. 1 ). In addition, neural network training can be automized, which can reduce an amount of time spent in training the neural network  225 . 
     As used herein, “neural network training operations” or “operations to train the neural network,” as well as variants thereof, include operations that are performed to determine one or more hidden layers of at least one neural network. In general, a neural network can include at least one input layer, at least one hidden layer, and at least one output layer. The layers can include multiple neurons that can each receive an input and generate a weighted output. In some embodiments, the neurons of the hidden layer(s) can calculate weighted sums and/or averages of inputs received from the input layer(s) and their respective weights and pass such information to the output layer(s). 
     In some embodiments, the neural network training operations can be performed by utilizing knowledge learned by a trained neural network during their training to train an untrained neural network. This can reduce the amount of time and resources spent in training untrained neural networks by reducing retraining of information that has already been learned by a trained neural network. In addition, embodiments herein can allow for a neural network that has been trained under a particular training methodology to train an untrained neural network with a different training methodology. For example, a neural network can be trained under a Tensorflow methodology and can then train an untrained neural network under a MobileNet methodology (or vice versa). Embodiments are not limited to these specific examples, however, and other training methodologies are contemplated within the scope of the disclosure. 
     The control circuitry  220  can, in some embodiments, cause performance of operations to convert data associated with the neural network  225  (e.g., the untrained neural network and/or the partially neural network) from one data type to another data type prior to causing the untrained and/or partially trained neural network  225  to be stored in the memory devices  226 - 1  to  226 -N and/or prior to transferring the neural network  225  to circuitry external to the memory system  204 . As used herein, a “data type” generally refers to a format in which data is stored. Non-limiting examples of data types include the IEEE 754 floating-point format, the fixed-point binary format, and/or universal number (unum) formats such as Type III unums and/or posits. Accordingly, in some embodiments, the control circuitry  220  can cause performance of operations to convert data associated with the neural networks (e.g., the untrained neural network and/or the partially trained neural network) from a floating-point or fixed point binary format to a universal number or posit format prior to causing the untrained and/or partially trained neural network to be stored in the memory devices  226 - 1  to  226 -N and/or prior to transferring the neural networks to circuitry external to the memory system  204 . 
     In contrast to the IEEE 754 floating-point or fixed-point binary formats, which include a sign bit sub-set, a mantissa bit sub-set, and an exponent bit sub-set, universal number formats, such as posits include a sign bit sub-set, a regime bit sub-set, a mantissa bit sub-set, and an exponent bit sub-set. This can allow for the accuracy, precision, and/or the dynamic range of a posit to be greater than that of a float, or other numerical formats. In addition, posits can reduce or eliminate the overflow, underflow, NaN, and/or other corner cases that are associated with floats and other numerical formats. Further, the use of posits can allow for a numerical value (e.g., a number) to be represented using fewer bits in comparison to floats or other numerical formats. 
     In some embodiments, the control circuitry  220  can determine that the untrained or partially trained neural network  225  has been trained and cause the neural network  225  that has been trained to be transferred to circuitry external to the memory system  204 . Further, in some embodiments, the control circuitry  220  can determine that the untrained and/or partially trained neural network  225  has been trained and cause performance of an operation to alter a precision, a dynamic range, or both, of information (e.g., data) associated with the neural network  225  that has been trained. Embodiments are not so limited, however, and in some embodiments, the control circuitry  220  can cause performance of an operation to alter a precision, a dynamic range, or both, of information (e.g., data) associated with the untrained and/or partially trained neural network  225  prior to the untrained and/or partially trained neural network  225  being stored in the memory devices  226 - 1  to  226 -N. 
     As used herein, a “precision” refers to a quantity of bits in a bit string that are used for performing computations using the bit string. For example, if each bit in a 16-bit bit string is used in performing computations using the bit string, the bit string can be referred to as having a precision of 16 bits. However, if only 8-bits of a 16-bit bit string are used in performing computations using the bit string (e.g., if the leading 8 bits of the bit string are zeros), the bit string can be referred to as having a precision of 8-bits. As the precision of the bit string is increased, computations can be performed to a higher degree of accuracy. Conversely, as the precision of the bit string is decreased, computations can be performed using to a lower degree of accuracy. For example, an 8-bit bit string can correspond to a data range consisting of two hundred and fifty-five (256) precision steps, while a 16-bit bit string can correspond to a data range consisting of sixty-five thousand five hundred and thirty-six (63,536) precision steps. 
     As used herein, a “dynamic range” or “dynamic range of data” refers to a ratio between the largest and smallest values available for a bit string having a particular precision associated therewith. For example, the largest numerical value that can be represented by a bit string having a particular precision associated therewith can determine the dynamic range of the data format of the bit string. For a universal number (e.g., a posit) format bit string, the dynamic range can be determined by the numerical value of the exponent bit sub-set of the bit string. 
     A dynamic range and/or the precision can have a variable range threshold associated therewith. For example, the dynamic range of data can correspond to an application that uses the data and/or various computations that use the data. This may be due to the fact that the dynamic range desired for one application may be different than a dynamic range for a different application, and/or because some computations may require different dynamic ranges of data. Accordingly, embodiments herein can allow for the dynamic range of data to be altered to suit the requirements of disparate applications and/or computations. In contrast to approaches that do not allow for the dynamic range of the data to be manipulated to suit the requirements of different applications and/or computations, embodiments herein can improve resource usage and/or data precision by allowing for the dynamic range of the data to varied based on the application and/or computation for which the data will be used. 
       FIG. 2B  is another functional block diagram in the form of an apparatus including a memory system  204  in accordance with a number of embodiments of the present disclosure. The control circuitry  220 , the memory devices  226 - 1  to  226 -N, and/or the neural network  225  can be referred to separately or together as an apparatus. As used herein, an “apparatus” can refer to, but is not limited to, any of a variety of structures or combinations of structures, such as a circuit or circuitry, a die or dice, a module or modules, a device or devices, or a system or systems, for example. The memory system  204  can be analogous to the memory system  104 / 204  illustrated in  FIGS. 1 and 2A , while the control circuitry  220  can be analogous to the control circuitry  120 / 220  illustrated in  FIGS. 1 and 2A . 
     The example illustrated in  FIG. 2B  corresponds to a scenario in which the neural network has been written to the memory device  226 -N. For example, in  FIG. 2B , at least a portion of an operation to train the neural network  225  has been performed while the neural network  225  was stored in the memory device  226 - 1 , as described in connection with  FIG. 2A , and the partially trained neural network  225  has been written to the memory device  226 -N for subsequent training. 
     In some embodiments, the control circuitry  220  can determine that subsequent training of the neural network  225  can be performed more efficiently while the neural network  225  is stored in the memory device  226 -N as opposed to the memory device  226 - 1 . In such embodiments, the control circuitry  220  can cause the partially trained neural network  225  to be written to the memory device  226 -N. Once the partially trained neural network  225  has been written to the memory device  226 -N, the control circuitry  220  can control performance of operations to further train the partially trained neural network  225 . For example, the control circuitry  220  can perform operations to determine one or more weights for a hidden layer of the neural network  225  as part of a neural network training operation. 
     In some embodiments, the control circuitry  220  can perform operations to determine one or more second weights for a hidden layer of the neural network  225  as part of performance of operations to further train the neural network  225 . That is, in some embodiments, the control circuitry  220  can perform operations to finish training a partially trained neural network  225  while the neural network  225  is stored within the memory device  226 -N. In embodiments in which the control circuitry  220  performs operations to finish training the neural network  225  while the neural network  225  is stored in the memory device  226 -N, the control circuitry  220  can, prior to writing the neural network  225  to the memory device  226 -N, determine that characteristics of the memory device  226 -N are conducive to finishing training of the neural network  225 . 
     That is, in some embodiments, at least a portion of the operation to train the neural network  225  within the memory device  226 -N to can be performed while the neural network  225  is stored within the memory device  226 -N based on characteristics of the memory device  226 -N such as the type of media employed by the memory device  226 -N, the bandwidth of the memory device  226 -N, and/or the speed of the memory device  226 -N, among others. In some embodiments, once the operation(s) to train the neural network  225  have been initiated, training operations can be performed within the memory device  226 -N in the absence of additional commands from the control circuitry  220  and/or a host (e.g., the host  102  illustrated in  FIG. 1 , herein). 
     As shown in  FIG. 2B , the memory device  226 - 1  can be configured to execute a supporting application  211  as part of performance of the operation(s) to train the neural network. As used herein, the term “supporting application” generally refers to an executable computing application (e.g., a computing program) that can assist in performance of the operations described herein. For example, the supporting application  211  can coordinate performance of at least a portion of the neural network training operations described herein. Although shown as being resident on the memory device  226 - 1 , embodiments are not so limited, and a supporting application  211  can also be executed on the memory device  226 -N. 
     In a non-limiting example, an apparatus (e.g., the memory system  204 ) can include a first memory device (e.g., the memory device  226 - 1 ) and a second memory device (e.g., the memory device  226 -N). As described herein, the first memory device and the second memory device can exhibit different bandwidth, power consumption, capacity, and/or latency characteristics. A processing device (e.g., the control circuitry  220  and/or the logic circuitry  222 ) can be coupled to the memory device  226 - 1  and the memory device  226 -N. The processing device can cause performance of at least a first portion of a training operation for a neural network  225  written to the first memory device  226 - 1  by determining one or more first weights for a hidden layer of the neural network  225 . 
     The processing device can then write data corresponding to the neural network  225  to the second memory device  226 -N subsequent to performance of at least the first portion of the training operation and cause performance of at least a second portion of the training operation for the neural network  225  written to the second memory device  226 -N by determining one or more second weights for the hidden layer of the neural network  225 . The processing device can further write data corresponding to the neural network  225  to the first memory device  226 - 1  subsequent to performance of at least the second portion of the training operation and/or cause performance of at least a third portion of the training operation for the neural network  225  written to the first memory device  226 - 1  by determining one or more third weights for the hidden layer of the neural network  225 . 
     Continuing with this example, the processing device can cause performance of at least the first portion of the training operation as part of performance of a first level of training the neural network  225  and/or cause performance of at least the second portion of the training operation as part of performance of a second level of training the neural network  225 . As used herein, the terms “first level of training” and “second level of training,” as well as variants thereof, generally refer to performance of a particular quantity of iterations of a training operation that may not correspond to the neural network being fully trained. For example, a first level of training can refer to the performance of a first quantity of iterations of a neural network training operation after which the neural network is partially (e.g., not fully) trained. The second level of training can refer to the performance of a second quantity of iterations of a neural network training operation after which the neural network is either partially (e.g., not fully) or fully trained. 
     In some embodiments, the first memory device  226 - 1  can have a processing unit (e.g., the processing unit  123 - 1  illustrated in  FIG. 1 ) resident thereon and/or the second memory device  226 -N can have a processing unit (e.g., the processing unit  123 -N illustrated in  FIG. 1 ) resident thereon. In such embodiments, the processing unit (e.g., the processing unit  123 - 1 ) can cause performance of an operation to pre-process data corresponding with the neural network  225  prior to the data corresponding to the neural network  225  being written to the second memory device  226 -N. For example, the processing unit can perform an operation to normalize data (e.g., vectors) associated with the neural network  225  prior to the neural network  225  being written to the second memory device  226 -N. Embodiments are no so limited, however, and in some embodiments, the processing unit can perform image segmentation, feature extraction, and/or operations to alter, reduce, compress, or otherwise modify at least a portion of the data associated with the neural network  225  prior to the neural network  225  being written to the second memory device  226 -N. 
     The processing device can, in some embodiments, write a copy of data corresponding to a first data state associated with the neural network  225  to the first memory device  226 - 1  or the second memory device  226 -N, or both and determine that the first data state associated with the neural network written to the first memory device  226 - 1  or the second memory device  226 -N, or both, has been updated to a second data state associated with the neural network  225 . In response to such a determination, the processing device can delete the copy of the data corresponding to the first data state in response to determining that the first data state has been updated to the second data state. 
     In this manner, checkpointing operations can be implemented by the apparatus to ensure that a recoverable copy of the neural network  225  is available in the event of a failure of the apparatus or a portion thereof. For example, in some embodiments, the processing device can determine that an error involving the neural network  225  has occurred, retrieve a copy of data corresponding to the second data state from the first memory device  226 - 1  or the second memory device  226 -N, or both, and perform an operation to recover the neural network  225  using the copy of the data corresponding to the second data state. 
     In addition to, or in the alternative, the processing device can perform page swapping operations involving the first memory device  226 - 1  and/or the second memory device  226 -N to transfer the neural network  225  between the first memory device  226 - 1  and the second memory device  226 -N, or vice versa to perform different portions of training operations for the neural network  225 . This can allow for performance of training operations involving the neural network  225  to be optimized by selecting the best available memory device for each level of training the neural network  225 . 
     In another non-limiting example, a system (e.g., the computing system  100  illustrated in  FIG. 1 ) can include control circuitry  220  comprising a processing device (e.g., the logic circuitry  222 ) and a memory resource  224  configured to operate as a cache for the processing device and a plurality of memory devices  226 - 1  to  226 -N coupled to the control circuitry  220 . In this example, the control circuitry  220  can write data corresponding to a neural network  225  to a first memory device (e.g., the memory device  226 - 1 ) among the plurality of memory devices  226 - 1  to  226 -N and cause, while the neural network  225  is stored in the first memory device  226 - 1 , at least a first portion of a training operation for a neural network  225  by determining one or more first weights for a hidden layer of the neural network  225  to be performed. The control circuitry  220  can then write the data corresponding to the neural network  225  to a second memory device (e.g., the memory device  226 -N) and cause, while the neural network  225  is stored in the second memory device, at least a second portion of the training operation for the neural network  225  by determining one or more second weights for the hidden layer of the neural network  225  to be performed. 
     Continuing with this example, the control circuitry  220  can write the data corresponding to the neural network  225  to the first memory device based on a determination that at least one characteristic of the first memory device  226 - 1  meets a first set of criterion and/or write the data corresponding to the neural network  225  to the second memory device  226 -N based on a determination that at least one characteristic of the second memory device  226 -N meets a second set of criterion. The characteristics and/or the criterion for writing the neural network  225  to the first memory device  226 - 1  or the second memory device  226 -N can include a bandwidth associated with the first memory device  226 - 1  and the second memory device  226 -N, a latency associated with the first memory device  226 - 1  and the second memory device  226 -N, and/or a capacity associated with the first memory device  226 - 1  and the second memory device  226 -N, among other characteristics and/or criterion associated with the first memory device  226 - 1  and the second memory device  226 -N. 
     In some embodiments, the control circuitry  220  can write a copy of data corresponding to a first data state associated with the neural network  225  to the first memory device  226 - 1  or the second memory device  226 -N, or both. The control circuitry  220  can then determine that the first data state associated with the neural network  225  written to the first memory device  226 - 1  or the second memory device  226 -N, or both, has been updated to a second data state associated with the neural network  225 . The control circuitry  220  can then delete the copy of the data corresponding to the first data state in response to determining that the first data state has been updated to the second data state. In some embodiments, the control circuitry  220  can determine that an error involving the neural network  225 , the memory devices  226 - 1  to  226 -N and/or the memory system  204  has occurred and retrieve a copy of data corresponding to the second data state from the first memory device  226 - 1  or the second memory device  226 -N, or both. The control circuitry  220  can then perform an operation to recover the neural network using the copy of the data corresponding to the second data state. 
     Continuing with this example, in some embodiments, the control circuitry  220  can, subsequent to writing the data corresponding to the neural network  225  to the second memory device  226 -N, write observed data to the first memory device  226 - 1 , or vice versa. The control circuitry can then execute the neural network  225  on the second memory device  226 -N using the observed data written to the first memory device  226 - 1 . The observed data can, in some embodiments, include training data that is gathered from real world events and can be gathered through various sensors such as biochemical sensors, image sensors, and/or monitoring sensors, among others. 
       FIG. 3  is a functional block diagram in the form of an apparatus including a memory system  304  that includes a plurality of memory devices  326 - 1  to  326 -N in accordance with a number of embodiments of the present disclosure. The control circuitry  320 , the memory devices  326 - 1  to  326 -N, and/or the neural network  325  can be referred to separately or together as an apparatus. The memory system  304  can be analogous to the memory system  104 / 204  illustrated in  FIGS. 1 and 2A-2B , while the control circuitry  320  can be analogous to the control circuitry  120 / 220  illustrated in  FIGS. 1 and 2A-2B . 
     As shown in  FIG. 3 , training data  327  is stored by the memory device  226 - 1 . The training data  327  can be analogous to the observed data described above in connection with  FIG. 2B . That is, in some embodiments, the training data  327  can include training data  327  that is gathered from real world events and can be gathered through various sensors such as biochemical sensors, image sensors, and/or monitoring sensors, among others. Although shown as being stored in the memory device  326 - 1 , embodiments are not so limited and, in some embodiments, the training data  327  (or at least a portion thereof) can be stored in the memory device  326 -N. 
     The memory device  326 - 1  to  326 -N in which the training data  327  is stored can be based on characteristics of the memory device  326 - 1  to  326 -N in which the training data  327  is stored and at least one of the other memory devices  326 - 1  to  326 -N. For example, the training data  327  may be stored in the memory device  326 - 1  based on a determination that the memory device  326 - 1  has a higher capacity, lower bandwidth, and/or a higher latency than the memory device  326 -N. In addition to, or in the alternative, the training data may be stored in a memory device  326 - 1  to  326 -N that is not currently storing the neural network  325 , although embodiments are not so limited. 
     In some embodiments, the training data  327  can be used to at least partially train the neural network  325 . That is, the training data  327  can be included in an input layer associated with the neural network  325  and can therefore be used in connection with determining one or more hidden layers of the neural network  325 . 
       FIG. 4  is a flow diagram  430  corresponding to a memory system to train neural networks in accordance with a number of embodiments of the present disclosure. The flow  430  can be performed by processing logic that can include hardware (e.g., processing device(s), control circuitry, dedicated logic, programmable logic, microcode, hardware of a device, and/or integrated circuit(s), etc.), software (e.g., instructions run or executed on a processing device), or a combination thereof. In some embodiments, the flow  430  is performed by control circuitry (e.g., the control circuitry  120  illustrated in  FIG. 1 ). Although shown in a particular sequence or order, unless otherwise specified, the order of the processes can be modified. Thus, the illustrated embodiments should be understood only as examples, and the illustrated processes can be performed in a different order, and some processes can be performed in parallel. Additionally, one or more processes can be omitted in various embodiments. Thus, not all processes are required in every embodiment. Other process flows are possible. 
     At operation  431 , a determination can be made with respect to characteristics of multiple memory devices (e.g., the memory devices  126 - 1  to  126 -N illustrated in  FIG. 1 , herein). As described above, the characteristics can include bandwidth associated with the memory devices, latencies associated with the memory devices, capacities associated with the memory devices, media types associated with the memory devices, power consumption levels associated with the memory devices, and/or data retention characteristics associated with the memory devices, among others. 
     At operation  432 , a neural network (e.g., the neural network  225  illustrated in  FIGS. 2A and 2B ) is written to a particular memory device based on the determined characteristics. Once the neural network has been written to the particular memory device, at operation  433 , operations to partially train the neural network can be performed. As described above, in some embodiments, the operations to partially train the neural network can include operations to determine hidden layers for the neural network. 
     At operation  434 , a determination can be made as to whether a different memory device has better characteristics than the particular memory device for further training the neural network. For example, it may be determined that characteristics of a different memory device may be more suited to performing further training operations on the partially trained neural network. If it is determined that a different memory device does not have better characteristics for further training the neural network, the flow  430  can return to operation  433  and operations to partially train the neural network can commence. 
     If, however, it is determined that a different memory device has better characteristics than the particular memory device for further training the neural network, at operation  435 , the partially trained neural network can be written to the different memory device. Once the partially trained neural network has been written to the different memory device, at operation  436 , operations to further train the partially trained neural network can be performed. 
       FIG. 5  is a flow diagram representing an example method  540  corresponding to a memory system to train neural networks in accordance with a number of embodiments of the present disclosure. The method  540  can be performed by processing logic that can include hardware (e.g., processing device(s), control circuitry, dedicated logic, programmable logic, microcode, hardware of a device, and/or integrated circuit(s), etc.), software (e.g., instructions run or executed on a processing device), or a combination thereof. Although shown in a particular sequence or order, unless otherwise specified, the order of the processes can be modified. Thus, the illustrated embodiments should be understood only as examples, and the illustrated processes can be performed in a different order, and some processes can be performed in parallel. Additionally, one or more processes can be omitted in various embodiments. Thus, not all processes are required in every embodiment. Other process flows are possible. 
     At block  542 , the method  540  can include performing, using data corresponding to a neural network written to a first memory device, at least a first portion of a training operation for a neural network (e.g., the neural network  225  illustrated in  FIGS. 2A and 2B ) by determining one or more first weights for a hidden layer of the neural network. In some embodiments, the first memory device can be analogous to the memory device  126 - 1  illustrated in  FIG. 1 . 
     At block  544 , the method  540  can include writing the data corresponding to the neural network to a second memory device. In some embodiments, the second memory device can be analogous to the memory device  126 -N illustrated in  FIG. 1 . In some embodiments, the method  540  can include performing, prior to writing the data corresponding to the neural network to the second memory device, an operation to reduce a quantity of data associated with the neural network, an image segmentation operation using the neural network, or any other operation involving the neural network. 
     In some embodiments, the method  540  can include receiving, by the first memory device, training data corresponding to the neural network and/or providing the training data to an input layer of the neural network. The method can further include writing, within the first memory device or the second memory device, or both, data associated with an output (e.g., an output layer) of the neural network. 
     The method  540  can further include performing, prior to writing the data corresponding to the neural network to the second memory device, an operation to select particular vectors from the data corresponding to the neural network and/or writing the particular vectors from the data corresponding to the neural network to the second memory device. For example, a processing unit (e.g., the processing units  123 - 1  to  123 -N illustrated in  FIG. 1 ) resident on the memory devices can perform operations to, as described above, pre-process data associated with the neural network prior to transferring the neural network to the second memory device. 
     At block  546 , the method  540  can include performing, using the data corresponding to the neural network written to the second memory device, at least a second portion of the training operation for the neural network by determining one or more second weights for the hidden layer of the neural network. As described above, the first memory device or the second memory device has a higher data processing bandwidth than the other of the first memory device or the second memory device. 
     As described above, in some embodiments, the method  540  can include storing a copy of the data corresponding to a first state of the neural network in the first memory device and/or the second memory device. Determining that the first state of the neural network has been updated to a second state of the neural network. Deleting the copy of the data corresponding to the first state of the neural network in response to determining that the first state of the neural network has been updated to the second state. 
     Although specific embodiments have been illustrated and described herein, those of ordinary skill in the art will appreciate that an arrangement calculated to achieve the same results can be substituted for the specific embodiments shown. This disclosure is intended to cover adaptations or variations of one or more embodiments of the present disclosure. It is to be understood that the above description has been made in an illustrative fashion, and not a restrictive one. Combination of the above embodiments, and other embodiments not specifically described herein will be apparent to those of skill in the art upon reviewing the above description. The scope of the one or more embodiments of the present disclosure includes other applications in which the above structures and processes are used. Therefore, the scope of one or more embodiments of the present disclosure should be determined with reference to the appended claims, along with the full range of equivalents to which such claims are entitled. 
     In the foregoing Detailed Description, some features are grouped together in a single embodiment for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the disclosed embodiments of the present disclosure have to use more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separate embodiment.