Cooperative execution of a genetic algorithm with an efficient training algorithm for data-driven model creation

A method includes, based on a fitness function, selecting a subset of models from a plurality of models. The plurality of models is generated based on a genetic algorithm and corresponds to a first epoch of the genetic algorithm. Each of the plurality of models includes data representative of a neural network. The method also includes performing at least one genetic operation of the genetic algorithm with respect to at least one model of the subset to generate a trainable model and sending the trainable model to an optimization trainer. The method includes adding a trained model received from the optimization trainer as input to a second epoch of the genetic algorithm that is subsequent to the first epoch.

BACKGROUND

Computers are often used to solve complex quantitative and qualitative problems. For problems that involve a large data set, a specially trained professional, known as a data scientist, is often hired. The data scientist interprets the data set and constructs models that can be processed by computers to solve the problem. However, hiring data scientists is cost prohibitive for many organizations.

For certain types of problems, advanced computing techniques, such as genetic algorithms or backpropagation, may be available to develop a model, such as a neural network, that is comparable in accuracy to a model that would be created by a data scientist. However, genetic algorithms may take a large number of iterations to converge on an acceptable neural network, and backpropagation may be slow when a large data set is being modeled or when the neural network includes a large number of nodes, connections, or layers.

Furthermore, various types of machine-learning problems exist. For example, regression problems involve evaluating a series of inputs to predict a numeric output, classification problems involve evaluating a series of inputs to predict a categorical output, and reinforcement learning involves performing actions within an environment to optimize some notion of a positive reward. Due to the differences in the various types of problems, the available mechanisms to generate and train a neural network or other machine learning solution may be problem-specific. For example, a support vector machine (SVM) may be suitable for some classification problems, logistic regression may be suitable for some regression problems, and a specialized machine learning package, such as TensorFlow, may be suitable for reinforcement learning. Thus, generating and training neural networks that meet performance requirements for each of multiple types of problems faced by an enterprise may be slow and difficult.

SUMMARY

The present application describes automated model building systems and methods that utilize a genetic algorithm and selective backpropagation to generate and train a neural network in a manner that is applicable to multiple types of machine-learning problems. To illustrate, the described automated model building techniques may enable a generalized approach to generating neural networks that can be applied for regression, classification, and reinforcement learning problems. Combining a genetic algorithm with selective backpropagation as described herein may enable generating a neural network that models a particular data set with acceptable accuracy and in less time than using genetic algorithms or backpropagation alone.

As an illustrative, non-limiting example, consider a home with four temperature sensors that periodically collect temperature readings in the living room (L), the dining room (D), the master bedroom (M), and the guest bedroom (G), respectively. In this example, a data set may include four columns, where each column corresponds to temperature readings from a particular sensor in a particular room, and where each row corresponds to a particular time at which the four sensors took a temperature reading. It may be of interest to predict the temperature in one of the rooms, e.g., L, given the temperature in the other three rooms, e.g., D, M, and G. A neural network may be suitable for such a problem, where the neural network has D, M, and/or G as input nodes and L as an output node. For example, a neural network that predicts an output value of L with 90% accuracy given input values of D, M, and/or G may be an acceptable solution.

In accordance with the described techniques, a combination of a genetic algorithm and an optimization algorithm such as backpropagation, a derivative free optimizer (DFO), an extreme learning machine (ELM) or similar optimizer may be used to generate and then train a neural network. It is to be understood that characterization of any system components of method steps as “optimizers” or “optimization trainers,” and use of such terminology herein, is not to be interpreted as requiring such components or steps to generate optimal results to the extreme (e.g., 100% prediction or classification accuracy). Rather, user of such terms is to be interpreted as indicating an attempt generate an output that is improved in some fashion relative to an input. For example, an optimization trainer that receives a trainable model as input and outputs a trained model may attempt to improve a prediction or classification accuracy of the trainable model by modifying one or more attributes of the trainable model to generate the trained model. Genetic algorithms are iterative adaptive search heuristics inspired by biological natural selection. The genetic algorithm may start with a population of random models that each define a neural network with different topology, weights and activation functions. Over the course of several epochs (also known as generations), the models may be evolved using biology-inspired reproduction operations, such as crossover (e.g., combining characteristics of two neural networks), mutation (e.g., randomly modifying a characteristic of a neural network), stagnation/extinction (e.g., removing neural networks whose accuracy has not improved in several epochs), and selection (e.g., identifying the best performing neural networks via testing). In addition, the best performing models of an epoch may be selected for reproduction to generate a trainable model. The trainable model may be trained using backpropagation to generate a trained model. When the trained model is available, the trained model may be re-inserted into the genetic algorithm for continued evolution. Training a model that is generated by breeding the best performing population members of an epoch may serve to reinforce desired “genetic traits” (e.g., neural network topology, activation functions, connection weights, etc.), and introducing the trained model back into the genetic algorithm may lead the genetic algorithm to converge to an acceptably accurate solution (e.g., neural network) faster, for example because desired “genetic traits” are available for inheritance in later epochs of the genetic algorithm.

A computer system in accordance with the present disclosure may include a memory that stores an input data set and a plurality of data structures. For example, each data structure may be a model of a neural network that models the input data set. The computer system may also include at least one processor that is configured to execute a recursive search. For example, the recursive search may be a genetic algorithm to generate a neural network that best models the input data set. During a first iteration of the recursive search, the processor may determine a fitness value for each of the data structures (e.g., neural network models) based on at least a subset of the input data set. The processor may also select a subset of data structures based on their respective fitness values and may perform at least one of a crossover operation or a mutation operation with respect to at least one data structure of the subset to generate a trainable data structure. The processor may further provide the trainable data structure to an optimization trainer that is configured to train the trainable data structure based on a portion of the input data set to generate a trained structure and to provide the trained data structure as input to a second iteration of the recursive search that is subsequent to the first iteration.

DETAILED DESCRIPTION

Referring toFIG. 1, a particular illustrative example of a system100is shown. The system100, or portions thereof, may be implemented using (e.g., executed by) one or more computing devices, such as laptop computers, desktop computers, mobile devices, servers, and Internet of Things devices and other devices utilizing embedded processors and firmware or operating systems, etc. In the illustrated example, the system100includes a genetic algorithm110and a backpropagation trainer180. The backpropagation trainer180is an example of an optimization trainer, and other examples of optimization trainers that may be used in conjunction with the described techniques include, but are not limited to, a derivative free optimizer (DFO), an extreme learning machine (ELM), etc. In particular aspects, the genetic algorithm110is executed on a different device, processor (e.g., central processor unit (CPU), graphics processing unit (GPU) or other type of processor), processor core, and/or thread (e.g., hardware or software thread) than the backpropagation trainer180. The genetic algorithm110and the backpropagation trainer180may cooperate to automatically generate a neural network model of a particular data set, such as an illustrative input data set102. The system100may provide an automated model building process that enables even inexperienced users to quickly and easily build highly accurate models based on a specified data set. Additionally, the system100simplify the neural network model to avoid overfitting and to reduce computing resources required to run the model.

The genetic algorithm110includes or is otherwise associated with a fitness function140, a stagnation criterion150, a crossover operation160, and a mutation operation170. As described above, the genetic algorithm110may represent a recursive search process. Consequently, each iteration of the search process (also called an epoch or generation of the genetic algorithm) may have an input set (or population)120and an output set (or population)130. The input set120of an initial epoch of the genetic algorithm110may be randomly or pseudo-randomly generated. After that, the output set130of one epoch may be the input set120of the next (non-initial) epoch, as further described herein.

The input set120and the output set130may each include a plurality of models, where each model includes data representative of a neural network. For example, each model may specify a neural network by at least a neural network topology, a series of activation functions, and connection weights. The topology of a neural network may include a configuration of nodes of the neural network and connections between such nodes. The models may also be specified to include other parameters, including but not limited to bias values/functions and aggregation functions.

Additional examples of neural network models are further described with reference toFIG. 2. In particular, as shown inFIG. 2, a model200may be a data structure that includes node data210and connection data220. In the illustrated example, the node data210for each node of a neural network may include at least one of an activation function, an aggregation function, or a bias (e.g., a constant bias value or a bias function). The activation function of a node may be a step function, sine function, continuous or piecewise linear function, sigmoid function, hyperbolic tangent function, or other type of mathematical function that represents a threshold at which the node is activated. The biological analog to activation of a node is the firing of a neuron. The aggregation function may be a mathematical function that combines (e.g., sum, product, etc.) input signals to the node. An output of the aggregation function may be used as input to the activation function. The bias may be a constant value or function that is used by the aggregation function and/or the activation function to make the node more or less likely to be activated.

The connection data220for each connection in a neural network may include at least one of a node pair or a connection weight. For example, if a neural network includes a connection from node N1 to node N2, then the connection data220for that connection may include the node pair <N1, N2>. The connection weight may be a numerical quantity that influences if and/or how the output of N1 is modified before being input at N2. In the example of a recurrent network, a node may have a connection to itself (e.g., the connection data220may include the node pair <N1, N1>).

The model200may also include a species identifier (ID)230and fitness data240. The species ID230may indicate which of a plurality of species the model200is classified in, as further described with reference toFIG. 3. The fitness data240may indicate how well the model200models the input data set102. For example, the fitness data240may include a fitness value that is determined based on evaluating the fitness function140with respect to the model200, as further described herein.

Returning toFIG. 1, the fitness function140may be an objective function that can be used to compare the models of the input set120. In some examples, the fitness function140is based on a frequency and/or magnitude of errors produced by testing a model on the input data set102. As a simple example, assume the input data set102includes ten rows, that the input data set102includes two columns denoted A and B, and that the models illustrated inFIG. 1represent neural networks that output a predicted a value of B given an input value of A. In this example, testing a model may include inputting each of the ten values of A from the input data set102, comparing the predicted values of B to the corresponding actual values of B from the input data set102, and determining if and/or by how much the two predicted and actual values of B differ. To illustrate, if a particular neural network correctly predicted the value of B for nine of the ten rows, then the a relatively simple fitness function140may assign the corresponding model a fitness value of 9/10=0.9. It is to be understood that the previous example is for illustration only and is not to be considered limiting. In some aspects, the fitness function140may be based on factors unrelated to error frequency or error rate, such as number of input nodes, node layers, hidden layers, connections, computational complexity, etc.

In a particular aspect, fitness evaluation of models may be performed in parallel. To illustrate, the system100may include additional devices, processors, cores, and/or threads190to those that execute the genetic algorithm110and the backpropagation trainer180. These additional devices, processors, cores, and/or threads190may test model fitness in parallel based on the input data set102and may provide the resulting fitness values to the genetic algorithm110.

In a particular aspect, the genetic algorithm110may be configured to perform speciation. For example, the genetic algorithm110may be configured to cluster the models of the input set120into species based on “genetic distance” between the models. Because each model represents a neural network, the genetic distance between two models may be based on differences in nodes, activation functions, aggregation functions, connections, connection weights, etc. of the two models. In an illustrative example, the genetic algorithm110may be configured to serialize a model into a bit string. In this example, the genetic distance between models may be represented by the number of differing bits in the bit strings corresponding to the models. The bit strings corresponding to models may be referred to as “encodings” of the models. Speciation is further described with reference toFIG. 3.

Because the genetic algorithm110is configured to mimic biological evolution and principles of natural selection, it may be possible for a species of models to become “extinct.” The stagnation criterion150may be used to determine when a species should become extinct, e.g., when the models in the species are to be removed from the genetic algorithm110. Stagnation is further described with reference toFIG. 4.

The crossover operation160and the mutation operation170is highly stochastic under certain constraints and a defined set of probabilities optimized for model building, which produces reproduction operations that can be used to generate the output set130, or at least a portion thereof, from the input set120. In a particular aspect, the genetic algorithm110utilizes intra-species reproduction but not inter-species reproduction in generating the output set130. Including intra-species reproduction and excluding inter-species reproduction may be based on the assumption that because they share more genetic traits, the models of a species are more likely to cooperate and will therefore more quickly converge on a sufficiently accurate neural network. In some examples, inter-species reproduction may be used in addition to or instead of intra-species reproduction to generate the output set130. Crossover and mutation are further described with reference toFIG. 6.

Left alone and given time to execute enough epochs, the genetic algorithm110may be capable of generating a model (and by extension, a neural network) that meets desired accuracy requirements. However, because genetic algorithms utilize randomized selection, it may be overly time-consuming for a genetic algorithm to arrive at an acceptable neural network. In accordance with the present disclosure, to “help” the genetic algorithm110arrive at a solution faster, a model may occasionally be sent from the genetic algorithm110to the backpropagation trainer180for training. This model is referred to herein as a trainable model122. In particular, the trainable model122may be based on crossing over and/or mutating the fittest models of the input set120, as further described with reference toFIG. 5. Thus, the trainable model122may not merely be a genetically “trained” file produced by the genetic algorithm110. Rather, the trainable model122may represent an advancement with respect to the fittest models of the input set120.

The backpropagation trainer180may utilize a portion, but not all of the input data set102to train the connection weights of the trainable model122, thereby generating a trained model182. For example, the portion of the input data set102may be input into the trainable model122, which may in turn generate output data. The input data set102and the output data may be used to determine an error value, and the error value may be used to modify connection weights of the model, such as by using gradient descent or another function.

The backpropagation trainer180may train using a portion rather than all of the input data set102to mitigate overfit concerns and/or to shorten training time. The backpropagation trainer180may leave aspects of the trainable model122other than connection weights (e.g., neural network topology, activation functions, etc.) unchanged. Backpropagating a portion of the input data set102through the trainable model122may serve to positively reinforce “genetic traits” of the fittest models in the input set120that were used to generate the trainable model122. Because the backpropagation trainer180may be executed on a different device, processor, core, and/or thread than the genetic algorithm110, the genetic algorithm110may continue executing additional epoch(s) while the connection weights of the trainable model122are being trained. When training is complete, the trained model182may be input back into (a subsequent epoch of) the genetic algorithm110, so that the positively reinforced “genetic traits” of the trained model182are available to be inherited by other models in the genetic algorithm110.

Operation of the system100is now described with reference toFIGS. 3-7. It is to be understood, however, that in alternative implementations certain operations may be performed in a different order than described. Moreover, operations described as sequential may be instead be performed at least partially concurrently, and operations described as being performed at least partially concurrently may instead be performed sequentially.

During a configuration stage of operation, a user may specify the input data set102and may specify a particular data field or a set of data fields in the input data set102to be modeled. The data field(s) to be modeled may correspond to output nodes of a neural network that is to be generated by the system100. For example, if a user indicates that the value of a particular data field is to be modeled (e.g., to predict the value based on other data of the data set), the model may be generated by the system100to include an output node that generates an output value corresponding to a modeled value of the particular data field. In particular implementations, the user can also configure other aspects of the model. For example, the user may provide input to indicate a particular data field of the data set that is to be included in the model or a particular data field of the data set that is to be omitted from the model. As another example, the user may provide input to constrain allowed model topologies. To illustrate, the model may be constrained to include no more than a specified number of input nodes, no more than a specified number of hidden layers, or no recurrent loops.

Further, in particular implementations, the user can configure aspects of the genetic algorithm110. For example, the user may provide input to limit a number of epochs that will be executed by the genetic algorithm110. Alternatively, the user may specify a time limit indicating an amount of time that the genetic algorithm110has to generate the model, and the genetic algorithm110may determine a number of epochs that will be executed based on the specified time limit. To illustrate, an initial epoch of the genetic algorithm110may be timed (e.g., using a hardware or software timer at the computing device executing the genetic algorithm110), and a total number of epochs that are to be executed within the specified time limit may be determined accordingly. As another example, the user may constrain a number of models evaluated in each epoch, for example by constraining the size of the input set120and/or the output set130. As yet another example, the user can define a number of trainable models122to be trained by the backpropagation trainer180and fed back into the genetic algorithm110as trained models182.

In particular aspects, configuration of the genetic algorithm110also includes performing pre-processing steps based on the input data set102. For example, the system100may determine, based on the input data set102and/or user input, whether a neural network is to be generated for a regression problem, a classification problem, a reinforcement learning problem, etc. As another example, the input data set102may be “cleaned” to remove obvious errors, fill in data “blanks,” etc. As another example, values in the input data set102may be scaled (e.g., to values between 0 and 1). As yet another example, non-numerical data (e.g., categorical classification data or Boolean data) may be converted into numerical data.

After the above-described configuration stage, the genetic algorithm110may automatically generate an initial set of models based on the input data set102, received user input indicating (or usable to determine) the type of problem to be solved, etc. (e.g., the initial set of models is data-driven). As illustrated inFIG. 2, each model may be specified by at least a neural network topology, an activation function, and link weights. The neural network topology may indicate an arrangement of nodes (e.g., neurons). For example, the neural network topology may indicate a number of input nodes, a number of hidden layers, a number of nodes per hidden layer, and a number of output nodes. The neural network topology may also indicate the interconnections (e.g., axons or links) between nodes.

The initial set of models may be input into an initial epoch of the genetic algorithm110as the input set120, and at the end of the initial epoch, the output set130generated during the initial epoch may become the input set120of the next epoch of the genetic algorithm110. In some examples, the input set120may have a specific number of models. For example, as shown in a first stage300of operation inFIG. 3, the input set may include 200 models. It is to be understood that alternative examples may include a different number of models in the input set120and/or the output set130.

For the initial epoch of the genetic algorithm110, the topologies of the models in the input set120may be randomly or pseudo-randomly generated within constraints specified by any previously input configuration settings. Accordingly, the input set120may include models with multiple distinct topologies. For example, a first model may have a first topology, including a first number of input nodes associated with a first set of data parameters, a first number of hidden layers including a first number and arrangement of hidden nodes, one or more output nodes, and a first set of interconnections between the nodes. In this example, a second model of epoch may have a second topology, including a second number of input nodes associated with a second set of data parameters, a second number of hidden layers including a second number and arrangement of hidden nodes, one or more output nodes, and a second set of interconnections between the nodes. Since the first model and the second model are both attempting to model the same data field(s), the first and second models have the same output nodes.

The genetic algorithm110may automatically assign an activation function, an aggregation function, a bias, connection weights, etc. to each model of the input set120for the initial epoch. In some aspects, the connection weights are assigned randomly or pseudo-randomly. In some implementations, a single activation function is used for each node of a particular model. For example, a sigmoid function may be used as the activation function of each node of the particular model. The single activation function may be selected based on configuration data. For example, the configuration data may indicate that a hyperbolic tangent activation function is to be used or that a sigmoid activation function is to be used. Alternatively, the activation function may be randomly or pseudo-randomly selected from a set of allowed activation functions, and different nodes of a model may have different types of activation functions. In other implementations, the activation function assigned to each node may be randomly or pseudo-randomly selected (from the set of allowed activation functions) for each node the particular model. Aggregation functions may similarly be randomly or pseudo-randomly assigned for the models in the input set120of the initial epoch. Thus, the models of the input set120of the initial epoch may have different topologies (which may include different input nodes corresponding to different input data fields if the data set includes many data fields) and different connection weights. Further, the models of the input set120of the initial epoch may include nodes having different activation functions, aggregation functions, and/or bias values/functions.

Continuing to a second stage350of operation, each model of the input set120may be tested based on the input data set102to determine model fitness. For example, the input data set102may be provided as input data to each model, which processes the input data set (according to the network topology, connection weights, activation function, etc., of the respective model) to generate output data. The output data of each model may be evaluated using the fitness function140to determine how well the model modeled the input data set102. For example, in the case of a regression problem, the output data may be evaluated by comparing a prediction value in the output data to an actual value in the input data set102. As another example, in the case of a classification problem, a classifier result indicated by the output data may be compared to a classification associated with the input data set102to determine if the classifier result matches the classification in the input data set102. As yet another example, in the case of a reinforcement learning problem, a reward may be determined (e.g., calculated) based on evaluation of an environment, which may include one or more variables, functions, etc. In a reinforcement learning problem, the fitness function140may be the same as or may be based on the reward function(s). Fitness of a model may be evaluated based on performance (e.g., accuracy) of the model, complexity (or sparsity) of the model, or a combination thereof. As a simple example, in the case of a regression problem or reinforcement learning problem, a fitness value may be assigned to a particular model based on an error value associated with the output data of that model or based on the value of the reward function, respectively. As another example, in the case of a classification problem, the fitness value may be assigned based on whether a classification determined by a particular model is a correct classification, or how many correct or incorrect classifications were determined by the model.

In a more complex example, the fitness value may be assigned to a particular model based on both prediction/classification accuracy or reward optimization as well as complexity (or sparsity) of the model. As an illustrative example, a first model may model the data set well (e.g., may generate output data or an output classification with a relatively small error, or may generate a large positive reward function value) using five input nodes (corresponding to five input data fields), whereas a second potential model may also model the data set well using two input nodes (corresponding to two input data fields). In this illustrative example, the second model may be sparser (depending on the configuration of hidden nodes of each network model) and therefore may be assigned a higher fitness value that the first model.

As shown inFIG. 3, the second stage350may include clustering the models into species based on genetic distance. In a particular aspect, the species ID230of each of the models may be set to a value corresponding to the species that the model has been clustered into.

Continuing toFIG. 4, during a third stage400and a fourth stage450of operation, a species fitness may be determined for each of the species. The species fitness of a species may be a function of the fitness of one or more of the individual models in the species. As a simple illustrative example, the species fitness of a species may be the average of the fitness of the individual models in the species. As another example, the species fitness of a species may be equal to the fitness of the fittest or least fit individual model in the species. In alternative examples, other mathematical functions may be used to determine species fitness. The genetic algorithm110may maintain a data structure that tracks the fitness of each species across multiple epochs. Based on the species fitness, the genetic algorithm110may identify the “fittest” species, shaded and denoted inFIG. 4as “elite species.” Although three elite species410,420, and430are shown inFIG. 4, it is to be understood that in alternate examples a different number of elite species may be identified.

In a particular aspect, the genetic algorithm110uses species fitness to determine if a species has become stagnant and is therefore to become extinct. As an illustrative non-limiting example, the stagnation criterion150may indicate that a species has become stagnant if the fitness of that species remains within a particular range (e.g., +/−5%) for a particular number (e.g., 5) epochs. If a species satisfies a stagnation criteria, the species and all underlying models may be removed from the genetic algorithm110. In the illustrated example, species360ofFIG. 3is removed, as shown in the third stage400through the use of broken lines.

Proceeding to the fourth stage450, the fittest models of each “elite species” may be identified. The fittest models overall may also be identified. In the illustrated example, the three fittest models of each “elite species” are denoted “elite members” and shown using a hatch pattern. Thus, model470is an “elite member” of the “elite species”420. The three fittest models overall are denoted “overall elites” and are shown using black circles. Thus, models460,462, and464are the “overall elites” in the illustrated example. As shown inFIG. 4with respect to the model460, an “overall elite” need not be an “elite member,” e.g., may come from a non-elite species. In an alternate implementation, a different number of “elite members” per species and/or a different number of “overall elites” may be identified.

Referring now toFIG. 5, during a fifth stage500of operation, the “overall elite” models460,462, and464may be genetically combined to generate the trainable model122. For example, genetically combining models may include crossover operations in which a portion of one model is added to a portion of another model, as further illustrated inFIG. 6. As another example, a random mutation may be performed on a portion of one or more of the “overall elite” models460,462,464and/or the trainable model122. The trainable model122may be sent to the backpropagation trainer180, as described with reference toFIG. 1. The backpropagation trainer180may train connection weights of the trainable model122based on a portion of the input data set102. When training is complete, the resulting trained model182may be received from the backpropagation trainer180and may be input into a subsequent epoch of the genetic algorithm110.

Continuing toFIG. 6, while the backpropagation trainer180trains the trainable model, the output set130of the epoch may be generated in a sixth stage600of operation. In the illustrated example, the output set130includes the same number of models, e.g., 200 models, as the input set120. The output set130may include each of the “overall elite” models460-464. The output set130may also include each of the “elite member” models, including the model470. Propagating the “overall elite” and “elite member” models to the next epoch may preserve the “genetic traits” resulted in caused such models being assigned high fitness values.

The rest of the output set130may be filled out by random intra-species reproduction using the crossover operation160and/or the mutation operation170. In the illustrated example, the output set130includes 10 “overall elite” and “elite member” models, so the remaining 190 models may be randomly generated based on intra-species reproduction using the crossover operation160and/or the mutation operation170. After the output set130is generated, the output set130may be provided as the input set120for the next epoch of the genetic algorithm110.

During the crossover operation160, a portion of one model may be combined with a portion of another model, where the size of the respective portions may or may not be equal. To illustrate with reference to the model “encodings” described with respect toFIG. 1, the crossover operation160may include concatenating bits0to p of one bit string with bits p+1 to q of another bit string, where p and q are integers and p+q is equal to the total size of a bit string that represents a model resulting from the crossover operation160. When decoded, the resulting bit string after the crossover operation160produces a neural network that differs from each of its “parent” neural networks in terms of topology, activation function, aggregation function, bias value/function, link weight, or any combination thereof.

Thus, the crossover operation160may be a random or pseudo-random biological operator that generates a model of the output set130by combining aspects of a first model of the input set120with aspects of one or more other models of the input set120. For example, the crossover operation160may retain a topology of hidden nodes of a first model of the input set120but connect input nodes of a second model of the input set to the hidden nodes. As another example, the crossover operation160may retain the topology of the first model of the input set120but use one or more activation functions of the second model of the input set120. In some aspects, rather than operating on models of the input set120, the crossover operation160may be performed on a model (or models) generated by mutation of one or more models of the input set120. For example, the mutation operation170may be performed on a first model of the input set120to generate an intermediate model and the crossover operation160may be performed to combine aspects of the intermediate model with aspects of a second model of the input set120to generate a model of the output set130.

During the mutation operation170, a portion of a model may be randomly modified. The frequency of mutations may be based on a mutation probability metric, which may be user-defined or randomly selected/adjusted. To illustrate with reference to the model “encodings” described with respect toFIG. 1, the mutation operation170may include randomly “flipping” one or more bits a bit string.

The mutation operation170may thus be a random or pseudo-random biological operator that generates or contributes to a model of the output set130by mutating any aspect of a model of the input set120. For example, the mutation operation170may cause the topology a particular model of the input set to be modified by addition or omission of one or more input nodes, by addition or omission of one or more connections, by addition or omission of one or more hidden nodes, or a combination thereof. As another example, the mutation operation170may cause one or more activation functions, aggregation functions, bias values/functions, and/or or connection weights to be modified. In some aspects, rather than operating on a model of the input set, the mutation operation170may be performed on a model generated by the crossover operation160. For example, the crossover operation160may combine aspects of two models of the input set120to generate an intermediate model and the mutation operation170may be performed on the intermediate model to generate a model of the output set130.

The genetic algorithm110may continue in the manner described above through multiple epochs. When the genetic algorithm110receives the trained model182, the trained model182may be provided as part of the input set120of the next epoch, as shown in a seventh stage700ofFIG. 7. For example, the trained model182may replace one of the other 200 models in the input set120or may be a 201stmodel of the input set (e.g., in some epochs, more than 200 models may be processed). During training by the backpropagation trainer180, the genetic algorithm110may have advanced one or more epochs. Thus, when the trained model182is received, the trained model182may be inserted as input into an epoch subsequent to the epoch during which the corresponding trainable model122was provided to the backpropagation trainer180. To illustrate, if the trainable model122was provided to the backpropagation trainer180during epoch N, then the trained model182may be input into epoch N+X, where X is an integer greater than zero.

In the example ofFIGS. 5 and 7, a single trainable model122is provided to the backpropagation trainer180and a single trained model182is received from the backpropagation trainer180. When the trained model182is received, the backpropagation trainer180becomes available to train another trainable model. Thus, because training takes more than one epoch, trained models182may be input into the genetic algorithm110sporadically rather than every epoch after the initial epoch. In some implementations, the backpropagation trainer180may have a queue or stack of trainable models122that are awaiting training. The genetic algorithm110may add trainable models122to the queue or stack as they are generated and the backpropagation trainer180may remove a training model122from the queue or stack at the start of a training cycle. In some implementations, the system100includes multiple backpropagation trainers180(e.g., executing on different devices, processors, cores, or threads). Each of the backpropagation trainers180may be configured to simultaneously train a different trainable model122to generate a different trained model182. In such examples, more than one trainable model122may be generated during an epoch and/or more than one trained model182may be input into an epoch.

Operation at the system100may continue iteratively until specified a termination criterion, such as a time limit, a number of epochs, or a threshold fitness value (of an overall fittest model) is satisfied. When the termination criterion is satisfied, an overall fittest model of the last executed epoch may be selected and output as representing a neural network that best models the input data set102. In some examples, the overall fittest model may undergo a final training operation (e.g., by the backpropagation trainer180) before being output.

Although various aspects are described with reference to a backpropagation training, it is to be understood that in alternate implementations different types of training may also be used in the system100. For example, models may be trained using a genetic algorithm training process. In this example, genetic operations similar to those described above are performed while all aspects of a model, except for the connection weight, are held constant.

Performing genetic operations may be less resource intensive than evaluating fitness of models and training of models using backpropagation. For example, both evaluating the fitness of a model and training a model include providing the input data set102, or at least a portion thereof, to the model, calculating results of nodes and connections of a neural network to generate output data, and comparing the output data to the input data set102to determine the presence and/or magnitude of an error. In contrast, genetic operations do not operate on the input data set102, but rather merely modify characteristics of one or more models. However, as described above, one iteration of the genetic algorithm110may include both genetic operations and evaluating the fitness of every model and species. Training trainable models generated by breeding the fittest models of an epoch may improve fitness of the trained models without requiring training of every model of an epoch. Further, the fitness of models of subsequent epochs may benefit from the improved fitness of the trained models due to genetic operations based on the trained models. Accordingly, training the fittest models enables generating a model with a particular error rate in fewer epochs than using genetic operations alone. As a result, fewer processing resources may be utilized in building highly accurate models based on a specified input data set102.

The system100ofFIG. 1may thus support cooperative, data-driven execution of a genetic algorithm and a backpropagation trainer to automatically arrive at an output neural network model of an input data set. The system ofFIG. 1may arrive at the output neural network model faster than using a genetic algorithm or backpropagation alone and with reduced cost as compared to hiring a data scientist. In some cases, the neural network model output by the system100may also be more accurate than a model that would be generated by a genetic algorithm or backpropagation alone. The system100may also provide a problem-agnostic ability to generate neural networks. For example, the system100may represent a single automated model building framework that is capable of generating neural networks for at least regression problems, classification problems, and reinforcement learning problems. Further, the system100may enable generation of a generalized neural network that demonstrates improved adaptability to never-before-seen conditions. To illustrate, the neural network may mitigate or avoid overfitting to an input data set and instead may be more universal in nature. Thus, the neural networks generated by the system100may be capable of being deployed with fewer concerns about generating incorrect predictions.

It will be appreciated that the systems and methods of the present disclosure may be applicable in various scenarios, infrastructures, and data environments. As an illustrative non-limiting example, the input data set102may include timestamped data from a large array of sensors distributed around a wind farm and may also include timestamped uptime/downtime data of individual wind turbines. The system100may generate a neural network model that is configured how likely a wind turbine is to fail. The neural network model may, in a particular example, increase failure lead time from 3-5 days to 30-40 days, which can result in reduced downtime and monetary savings for an operator of the wind farm. The system100may be capable of automatically building similar kinds of models that predict numerical values or states (e.g., failures) for internet of things (IoT), utilities, and oil/gas infrastructures.

As another illustrative non-limiting example, the input data set102may include health data and the system100may automatically build a model to predict whether a patient exhibiting certain health conditions is likely to have a particular ailment. As another illustrative non-limiting example, the input data set102may include financial data and the system100may automatically build a model to forecast market conditions. As another illustrative non-limiting example, the input data set102may include network security, network log, and/or malware data, and the system100may automatically build a model to implement firewall filtering rules, endpoint anti-malware detection, a bot/botnet detector, etc.

As another illustrative non-limiting example, the system100may generate a neural network to output aircraft auto-pilot operations (e.g. throttle, steer, flaps, etc.) based on reinforcement learning. In such an example, the reward function optimized by the neural network may involve aircraft altitude, aircraft distance traveled, etc. As yet another example, the system100may generate a neural network to predict oil/gas industry workover events (e.g., events that lead to major maintenance or remedial operations on a rig or well, which can lead to considerable production time lost and expense incurred).

Yet another example of a problem set that can be solved with neural networks generated with the system described herein is data fusion. In this case, data aggregated from a large number of sensors of various types, including multiple sensors of the same type, is collected and used to identify an object, action or phenomenon that wouldn't be entirely detectable with any one, or a small subset of sensors. For example, the detection of a submarine may be performed based on the inputs received from multiple sonar buoys which provide input to the generated neural network. Another example may be the identification of a particular type of aircraft based on both the audio signature and a visual view (which may be partially obscured, or low resolution).

FIGS. 8A and 8Bdepict a particular example of a method800of cooperative execution of a genetic algorithm and a backpropagation trainer. In an illustrative example, the method800may be performed at the system100ofFIG. 1.

The method800may start, at802, and may include generating a randomized input population of models based on an input data set, at804. Each model may include data representative of a neural network. For example, each model may include at least node data and connection data, as described with reference toFIGS. 1 and 2. Further, each of the models may be part of the input set120ofFIG. 1and may model the input data set102ofFIG. 1.

The method800may also include determining, based on a fitness function, a fitness value of each model of the input population, at806. For example, the fitness of each model of the input set120may be determined, as described with reference toFIGS. 1 and 3.

The method800may further include determining a subset of models based on their respective fitness values, at808. The subset of models may be the fittest models of the input population, e.g., “overall elites.” For example, “overall elites” may be determined as described with reference toFIGS. 1 and 4.

The method800may include performing multiple sets of operations at least partially concurrently. Continuing to826(inFIG. 8B), the method800may include performing at least one genetic operation with respect to at least one model of the subset to generate a trainable model. For example, the crossover operation160and/or the mutation operation170may be performed with respect to the “overall elites” to generate the trainable model122, as described with reference toFIGS. 1, 4, and 5.

The method800may also include sending the trainable model to a backpropagation trainer (or other optimization trainer) for training based on a portion of the input data set, at828. For example, the backpropagation trainer180ofFIG. 1may train the trainable model122based on a portion of the input data set102to generate the trained model182, as described with reference toFIGS. 1 and 5.

The genetic algorithm may continue while backpropagation training occurs. For example, the method800may include grouping the input population of models into species based on genetic distance, at810, and determining species fitness of each species, at812. To illustrate, the models of the input set120may be grouped into species and species fitness may be evaluated as described with reference toFIGS. 1, 3, and4.

Continuing to814, species that satisfy a stagnation criteria may be removed. For example, species satisfying the stagnation criterion150may be removed, as described with reference toFIGS. 1 and 4. At816, the method800may include identifying a subset of species based on their respective fitness values and identifying models of each species in the subset based on their respective model fitness values. The subset of species may be the fittest species of the input population, e.g., “elite species,” and the identified models of the “elite species” may be the fittest members of those species, e.g., “elite members.” For example, species fitness values, “elite species,” and “elite members” may be determined as described with reference toFIGS. 1 and 4.

The method800may include determining an output population that includes each “elite member,” the “overall elites,” and at least one model that is generated based on intra-species reproduction, at818. For example, the models of the output set130ofFIG. 1may be determined, where the output set130includes the overall elite models460-464, the elite members (including the elite member model470), and at least one model generated based on intra-species reproduction using the crossover operation160and/or the mutation operation170, as described with reference toFIGS. 1 and 6.

The method800may include determining whether a termination criterion is satisfied, at820. The termination criterion may include a time limit, a number of epochs, or a threshold fitness value of an overall fittest model, as illustrative non-limiting examples. If the termination criterion is not satisfied, the method800returns to806and a next epoch of the genetic algorithm is executed, where the output population determined at818is the input population of the next epoch.

As described above, while the genetic algorithm is ongoing, the backpropagation trainer may train the trainable model to generate a trained model. When training is complete, the method800may include receiving the trained model from the backpropagation trainer (or other optimization trainer), at830(inFIG. 8B). The trained model may be added to the input set of an epoch of the genetic algorithm, as shown inFIG. 8B.

When the termination criterion is satisfied, at820, the method800may include selecting and outputting a fittest model, at822, and the method800may end, at824. In some implementations, the selected model may be subjected to a final training operation, e.g., by the backpropagation trainer or by another trainer, before being output.

It is to be understood that the division and ordering of steps inFIGS. 8A and 8Bis for illustrative purposes only and is not be considered limiting. In alternative implementations, certain steps may be combined and other steps may be subdivided into multiple steps. Moreover, the ordering of steps may change. For example, the termination criterion may be evaluated after determining the “overall elites,” at808, rather than after determining the output population, at818.

In conjunction with the described aspects, a computer system may include a memory configured to store an input data set and a plurality of data structures, each of the plurality of data structures including data representative of a neural network. The system also includes a processor configured to execute a recursive search. Executing the recursive search includes, during a first iteration: determining a fitness value for each of the plurality of data structures based on at least a subset of the input data set, selecting a subset of data structures from the plurality of data structures based on the fitness values of the subset of data structures, performing at least one of a crossover operation or a mutation operation with respect to at least one data structure of the subset to generate a trainable data structure, and providing the trainable data structure to an optimization trainer. The optimization trainer is configured to train the trainable data structure based on a portion of the input data set to generate a trained data structure and to provide the trained data structure as input to a second iteration of the recursive search that is subsequent to the first iteration.

In conjunction with the described aspects, a method includes, based on a fitness function, selecting, by a processor of a computing device, a subset of models from a plurality of models. The plurality of models is generated based on a genetic algorithm and corresponds to a first epoch of the genetic algorithm. Each of the plurality of models includes data representative of a neural network. The method also includes performing at least one genetic operation of the genetic algorithm with respect to at least one model of the subset to generate a trainable model and sending the trainable model to an optimization trainer. The method includes adding a trained model received from the optimization trainer as input to a second epoch of the genetic algorithm that is subsequent to the first epoch.

In conjunction with the described aspects, a computer-readable storage device stores instructions that, when executed, cause a computer to perform operations including, based on a fitness function, selecting a subset of models from a plurality of models. The plurality of models is generated based on a genetic algorithm and corresponds to a first epoch of the genetic algorithm. Each of the plurality of models includes data representative of a neural network. The operations also include performing at least one genetic operation of the genetic algorithm with respect to at least one model of the subset to generate a trainable model and sending the trainable model to a trainer. The operations include adding a trained model received from the trainer as input to a second epoch of the genetic algorithm that is subsequent to the first epoch.

The systems and methods of the present disclosure may be embodied as a customization of an existing system, an add-on product, a processing apparatus executing upgraded software, a standalone system, a distributed system, a method, a data processing system, a device for data processing, and/or a computer program product. Accordingly, any portion of the system or a module may take the form of a processing apparatus executing code, an internet based (e.g., cloud computing) embodiment, an entirely hardware embodiment, or an embodiment combining aspects of the internet, software and hardware. Furthermore, the system may take the form of a computer program product on a computer-readable storage medium or device having computer-readable program code (e.g., instructions) embodied or stored in the storage medium or device. Any suitable computer-readable storage medium or device may be utilized, including hard disks, CD-ROM, optical storage devices, magnetic storage devices, and/or other storage media. A computer-readable storage medium or device is not a signal.

Systems and methods may be described herein with reference to screen shots, block diagrams and flowchart illustrations of methods, apparatuses (e.g., systems), and computer media according to various aspects. It will be understood that each functional block of a block diagrams and flowchart illustration, and combinations of functional blocks in block diagrams and flowchart illustrations, respectively, can be implemented by computer program instructions.

Although the disclosure may include a method, it is contemplated that it may be embodied as computer program instructions on a tangible computer-readable medium, such as a magnetic or optical memory or a magnetic or optical disk/disc. All structural, chemical, and functional equivalents to the elements of the above-described exemplary embodiments that are known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the present claims. Moreover, it is not necessary for a device or method to address each and every problem sought to be solved by the present disclosure, for it to be encompassed by the present claims. Furthermore, no element, component, or method step in the present disclosure is intended to be dedicated to the public regardless of whether the element, component, or method step is explicitly recited in the claims. As used herein, the terms “comprises”, “comprising”, or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.