Data driven mixed precision learning for neural networks

Embodiments for implementing mixed precision learning for neural networks by a processor. A neural network may be replicated into a plurality of replicated instances and each of the plurality of replicated instances differ in precision used for representing and determining parameters of the neural network. Data instances may be routed to one or more of the plurality of replicated instances for processing according to a data pre-processing operation.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates in general to computing systems, and more particularly to, various embodiments for data driven mixed precision learning for neural networks by a processor.

Description of the Related Art

In today's society, consumers, businesspersons, educators, and others communicate over a wide variety of mediums in real time, across great distances, and many times without boundaries or borders. With the increased usage of computing networks, such as the Internet, humans are currently inundated and overwhelmed with the amount of information available to them from various structured and unstructured sources. Due to the recent advancement of information technology and the growing popularity of the Internet, a wide variety of computer systems have been used in machine learning. Machine learning is a form of artificial intelligence that is employed to allow computers to evolve behaviors based on empirical data.

SUMMARY OF THE INVENTION

Various embodiments for implementing data driven mixed precision learning for neural networks by a processor, are provided. In one embodiment, by way of example only, a method for implementing mixed precision learning for neural networks for deep learning problems, again by a processor, is provided. A neural network may be replicated into a plurality of replicated instances and each of the plurality of replicated instances differ in precision used for representing and determining parameters of the neural network. Data instances may be routed to one or more of the plurality of replicated instances for processing according to a data pre-processing operation.

DETAILED DESCRIPTION OF THE DRAWINGS

Machine learning allows for an automated processing system (a “machine”), such as a computer system or specialized processing circuit, to develop generalizations about particular data sets and use the generalizations to solve associated problems by, for example, classifying new data. Once a machine learns generalizations from (or is trained using) known properties from the input or training data, it can apply the generalizations to future data to predict unknown properties.

In machine learning and cognitive science, neural networks are a family of statistical learning models inspired by the biological neural networks of animals, and in particular the brain. Neural networks can be used to estimate or approximate systems and functions that depend on a large number of inputs and are generally unknown. Neural networks use a class of algorithms based on a concept of inter-connected “neurons.” In a typical neural network, neurons have a given activation function that operates on the inputs. By determining proper connection weights (a process also referred to as “training”), a neural network achieves efficient recognition of desired patterns, such as images and characters. Oftentimes, these neurons are grouped into “layers” in order to make connections between groups more obvious and to each computation of values. Training the neural network is a computationally intense process. For example, designing machine learning (ML) models, particularly neural networks for deep learning, is a trial-and-error process, and typically the machine learning model is a black box.

Furthermore, training and using neural networks for deep learning problems is time consuming and requires extensive use of compute resources. Further, designing and tuning neural networks is an iterative process based on trial-and-error, which makes it even more imperative to speed up training.

Reduced precision computing is computing using a fewer number of digits to represent the numeric values in the computation. In one aspect, reduced precision computing may be used to increase the efficiency of training or inference using neural networks such as, for example, increasing efficiency in time, power/energy consumption, and/or memory requirements. Various computing hardware and applications, supporting multiple precisions, and libraries supporting flexible numerical formats, may enable the use of reduced precision computing in neural networks. In addition to reduced precision, mixed precision computing may also be used to increase the efficiency of training or inference using neural networks. Mixed precision computing is computing where certain operations (e.g., the multiplication operations) are performed in reduced precision using a fewer number of digits to represent numeric values in the computation, and other operations (e.g., accumulation) are performed in higher precision using more digits to represent numeric values in the computation. This is primarily done to counter the possible negative effect of reduced precision, or in other words, loss of precision leading to reduced accuracy of trained neural networks.

Said differently, reduced precision may lead to a loss in accuracy and current approaches to overcome this loss include using mixed precision (different precisions for different operations or functions in the overall neural network), finessing a design of the neural network, or reverting to computations using the original precision. Computing platforms may support multiple precisions, with lower precision computations being more efficient than higher precision computations. In one aspect, reduced or mixed precision may be used in an identical manner for processing each input data instance. However, such an approach can be either inefficient or have an adverse effect on the accuracy of the trained model because precision is not customized based on contents of the data instance and domain knowledge of the learning problem.

Accordingly, the present invention provides a solution for data driven mixed precision learning for neural networks. In one aspect, the present invention builds neural networks to exploit reduced precision computing. An original neural network may be replicated, with the neural network replica instances differing from each other in the precision used for representing and computing the parameters of the network. For example, one neural network replica instance may use 8-bits to represent each numerical value in the neural network whereas a second neural network replica instance may use 16-bits to represent each numerical value in the neural network. A pre-processing step operation may be added for the input data, where the content of the input data is analyzed. Based on the content of the data and the context of the learning problem, the pre-processing operation determines a best precision to be used for processing each data instance. This determination is then used to route input data instances to appropriate replicas of the neural network for processing.

In one aspect, the present invention provides a solution for data driven mixed precision learning for neural networks by building neural networks to exploit reduced precision computing where the original neural network is replicated, with the replicas differing from each other in the precision used for representing and computing the parameters of the network, and a data pre-processing step is used to route input data instances to appropriate replicas for processing. The number of replicas to create may be determined based on the levels of precision and numerical formats supported by the targeted computing platform. The replicas can differ in batch size or number of learners used. A data pre-processing operation can use known metrics, or a trained neural network, to determine the best level of precision for each input data instance. For example, when the learning problem uses images as input data, metrics such as smoothness or compressibility of the image may be used in the data pre-processing operation. The routing of data to appropriate neural network replicas can be determined statically based on the results of the pre-processing step, or dynamically by also taking into account instantaneous processing load at each replica. For inference, the trained neural network replicas can be combined into fewer neural networks. The combination of neural network replicas is based on a weighted combination of parameters from the replicas being merged, where the weighting may depend on the number of input data instances processed through the individual replicas. Batches can be statically or dynamically formed, and batches can be homogeneous or heterogeneous in the precision levels of data instances in the batch. For heterogeneous batches, the choice of neural network replica to use can be based on the maximum precision level for any data instance in the batch, or it can be based on the precision level that occurs most often in the batch.

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

As previously mentioned, the present invention provides for data driven mixed precision learning for neural networks. One or more data instances may be analyzed during a data pre-processing operation. A required precision level to use for processing the one or more data instances is determined during the data pre-processing operation. A neural network may be replicated into a plurality of replicated instances and each of the plurality of replicated instances differ in precision used for representing and determining parameters of the neural network. Data instances may be routed to one or more of the plurality of replicated instances for processing according to a data pre-processing operation.

It should be noted that a required precision level may be non-uniform for all instances of input data in a particular problem domain. For example, when using images as input data, images of open landscape or scenic vistas have very different properties for the purpose of learning, as compared to images of one or more specific objects. In some instances, lower precision may negatively impact accuracy, but in other instances, lower precision can facilitate increased learning. Thus, the relationship between precision used in learning and accuracy of the trained models is unpredictable. However, knowing the context of the learning problem and the contents of the input data, statistical metrics can be devised to determine the precision level that best fits the data instance for the purpose of learning from the context.

As an example, consider the learning problem of image classification. A possible metric may be a weighted combination of: 1) a measure of repetitiveness in the image (which can be approximated by the percentage compression ratio that can be achieved using a standard algorithm or compression utility), and/or 2) a measure of smoothness in the image (which can be approximated by computing, for each image point, the sum of distances (differences) of that point and all its neighbors and then taking the mean and standard deviation of the computed values for all points in the image).

A neural network can be trained to determine the precision level that best fits each input data instance. In an additional aspect, the pre-processing operation may use pre-defined metrics. The present invention works by replicating an original neural network, with the replicas differing from each other in the precision used for representing and computing the parameters of the network. The number of replicas may depend on the different precisions/numerical formats efficiently supported by the computing platform. For each replica instance, the number of learners or batch size supported by the replica may also differ depending on available hardware resources or application requirements. For each input data instance, a best level of precision to use for the input data is determined during pre-processing based on the content of the data and the context of the learning problem. The analytics of the pre-processing operation may then be used to route the input data to one or more of the neural network replicas for processing. The neural network replica that uses the same/similar level of precision may be selected/chosen, or if such a replica does not exist, then the replica supporting a next higher level of precision may be selected/chosen. Optionally, the routing of input data instances to a neural network replica can be dynamically determined, based on instantaneous processing load of each replica.

During training, each replica of the neural network may be independently trained using a subset of the training input data and the training process may be constrained to use a minimum percentage of data in the training dataset for training of each of the neural network replica instances. After training, when the neural networks are to be used for inference, the application may use the same set of replicas as in the training step. Alternatively, inference may use a computing platform that supports different precision levels, and therefore it may use only a subset of the replicas that were trained. Since the replicated copies of the neural network have the same structure, they can be combined into fewer neural networks if needed, using weighted combinations of parameters from the individual replicas being merged. The weighting can depend on the number of input data instances processed through a given replica during training.

Since training and inference may be performed on batches of input data, the present invention may determine which neural network replica to use based on properties of individual input data instances. Therefore, when forming batches, properties of each data instance within a batch may be considered. Batches can be created statically or dynamically. Further, batches may be constrained to contain data instances with the same precision level as a best fit for each instance in the batch (e.g., homogeneous batches). Alternatively, batches may contain data instances with varying precision levels for the best fit (heterogeneous batches). For heterogeneous batches, the choice of neural network replica to use can be based on the maximum precision level for any data instance in the batch, or it can be based on the precision level that occurs most often in the batch.

Turning now toFIG.4, a block diagram depicting exemplary functional components400according to various mechanisms of the illustrated embodiments is shown. In one aspect, one or more of the components, modules, services, applications, and/or functions described inFIGS.1-3may be used inFIG.4. A mixed precision learning for neural network service410is shown, incorporating processing unit (“processor”)420to perform various computational, data processing and other functionality in accordance with various aspects of the present invention. The mixed precision learning for neural network service410may be provided by the computer system/server12ofFIG.1. The processing unit420may be in communication with memory430. The mixed precision learning for neural network service410may include a data pre-processing component440, a batching component450, a routing component460, a machine learning model component470, and a replication component480.

As one of ordinary skill in the art will appreciate, the depiction of the various functional units in mixed precision learning for neural network service410is for purposes of illustration, as the functional units may be located within the mixed precision learning for neural network service410or elsewhere within and/or between distributed computing components.

In one embodiment, by way of example only, the mixed precision learning for neural network service410may modularly construct a neural network for deep learning problem. More specifically, the mixed precision learning for neural network service410, using the replication component480, may replicate a neural network into a plurality of replicated instances. Each of the plurality of replicated instances differ in precision used for representing and determining parameters of the neural network. The replication component480may determine a number of replicated instances to replicate from the neural network based on precision levels and numerical formats supported by a computing platform.

A data pre-processing component440may determine a required precision level to use for processing the one or more data instances during the data pre-processing operation. The data pre-processing component440may analyze the one or more data instances based on data content and a context of a deep learning problem during the data pre-processing operation and use one or more known metrics or a trained neural network to determine a precision level during the data pre-processing operation.

The batching component450may batch the one or more data instances according to a homogeneous precision levels or heterogeneous precision levels. Batches can be statically or dynamically formed, and batches can be homogeneous (e.g., similar) or heterogeneous (e.g., different). For heterogeneous batches, the choice/selection of one of the replicated neural network instances to use may be based on a maximum precision level for any data instance in the batch, or it may be based on a precision level that occurs most often in the batch.

The routing component460may route data instances to one or more of the plurality of replicated instances for processing according to a data pre-processing operation. That is, the routing component460may select data instances for routing to one or more of the plurality of replicated instances according to a required precision level, a dynamic load, or combination thereof.

The machine learning component470, in association with the replication component480, may independently train each replicated instance (e.g., replicated neural network instance) and may combine one or more of the replicated instances that are trained based on a weighted combination of parameters from the one or more of the plurality of replicated instances being combined.

By way of example only, the machine learning component470may determine one or more heuristics and machine learning based models using a wide variety of combinations of methods, such as supervised learning, unsupervised learning, temporal difference learning, reinforcement learning and so forth. Some non-limiting examples of supervised learning which may be used with the present technology include AODE (averaged one-dependence estimators), artificial neural networks, Bayesian statistics, naive Bayes classifier, Bayesian network, case-based reasoning, decision trees, inductive logic programming, Gaussian process regression, gene expression programming, group method of data handling (GMDH), learning automata, learning vector quantization, minimum message length (decision trees, decision graphs, etc.), lazy learning, instance-based learning, nearest neighbor algorithm, analogical modeling, probably approximately correct (PAC) learning, ripple down rules, a knowledge acquisition methodology, symbolic machine learning algorithms, sub symbolic machine learning algorithms, support vector machines, random forests, ensembles of classifiers, bootstrap aggregating (bagging), boosting (meta-algorithm), ordinal classification, regression analysis, information fuzzy networks (IFN), statistical classification, linear classifiers, fisher's linear discriminant, logistic regression, perceptron, support vector machines, quadratic classifiers, k-nearest neighbor, hidden Markov models and boosting. Some non-limiting examples of unsupervised learning which may be used with the present technology include artificial neural network, data clustering, expectation-maximization, self-organizing map, radial basis function network, vector quantization, generative topographic map, information bottleneck method, IBSEAD (distributed autonomous entity systems based interaction), association rule learning, apriori algorithm, eclat algorithm, FP-growth algorithm, hierarchical clustering, single-linkage clustering, conceptual clustering, partitional clustering, k-means algorithm, fuzzy clustering, and reinforcement learning. Some non-limiting examples of temporal difference learning may include Q-learning and learning automata. Specific details regarding any of the examples of supervised, unsupervised, temporal difference or other machine learning described in this paragraph are known and are considered to be within the scope of this disclosure.

Turning now toFIG.5, flowchart diagram500depicts an exemplary method for implementing mixed precision learning for neural networks for deep learning, again in which various aspects of the present invention may be realized. The functionality500may be implemented as a method executed as instructions on a machine, where the instructions are included on at least one computer readable medium or one non-transitory machine-readable storage medium.

One or more data instances may be received and analyzed during a data pre-processing operation, as in block510. The data pre-processing operation may include determining a best precision level using metrics and/or a machine learning model.

A batching operation may be performed upon completion of the data pre-processing operation, as in block520. That is, the data instances may be batched according to homogeneous precision levels (e.g., same/similar precision levels) or heterogeneous precision levels (e.g., different precisions levels). In one aspect, the data pre-processing operation and batching operation may be performed online and/or offline (and stored “storage”), as indicated in block530.

A neural network may be replicated into a plurality of replicated instances and each of the plurality of replicated instances differ in precision used for representing and determining parameters of the neural network and data instances may be routed to one or more of the plurality of replicated instances such as, for example replicated instance1550and/or replicated instance N560, for processing according to a data pre-processing operation, as in block540. Each data instance may be selected based on 1) a required precision level (e.g., same/similar precision level or higher), and/or 2) a dynamic load. Also, each replicated instance may differ in precision level. The replicated instances such as, for example, replicated instance1550and/or replicated instance N560, may also have a different number of learners and/or batch size depending on 1) available computing hardware/components, and/or 2) application needs.

FIG.6is an additional flowchart diagram600depicting an additional exemplary method for implementing mixed precision learning for neural networks, again in which various aspects of the present invention may be realized. The functionality600may be implemented as a method executed as instructions on a machine, where the instructions are included on at least one computer readable medium or one non-transitory machine-readable storage medium. The functionality600may start in block602.

One or more data instances may be analyzed during a data pre-processing operation, as in block604. A neural network may be replicated into a plurality of replicated instances and each of the plurality of replicated instances differ in precision used for representing and determining parameters of the neural network, as in block606. Data instances may be routed to one or more of the plurality of replicated instances for processing according to a data pre-processing operation, as in block608. The functionality600may end, as in block610.

In one aspect, in conjunction with and/or as part of at least one block ofFIG.6, the operations of method600may include each of the following. The operations of method600may determine a required precision level to use for processing the one or more data instances during the data pre-processing operation, analyze the one or more data instances based on data content and a context of a deep learning problem during the data pre-processing operation, and/or use one or more known metrics or a trained neural network to determine a precision level during the data pre-processing operation.

The operations of method600may select the data instances for routing to the one or more of the plurality of replicated instances according to a required precision level, a dynamic load, or combination thereof.

The operations of method600may batch the one or more data instances according to a homogeneous precision levels or heterogeneous precision levels. For heterogeneous batches, the choice of neural network instance to select/use can be based on a maximum or mode of precision levels in the batch. The operations of method600may determine a number of the plurality of replicated instances based on precision levels and numerical formats supported by a computing platform. Also, each of the replicated instances may be independently trained and may be combined based on a weighted combination of parameters from the one or more of the plurality of replicated instances being combined.