FROM DEEP LEARNING TO DEEP DEDUCING: AUTOMATICALLY TRACKING DOWN NASH EQUILIBRIUM THROUGH AUTONOMOUS NEURAL AGENT, A POSSIBLE MISSING STEP TOWARD GENERAL A.I.

Contrary to most reinforcement learning studies , which emphasize on training a deep neural network to approximate its output layer to certain strategies , this paper proposes a reversed method for reinforcement learning . We call this “ Deep Deducing ” . In short , after adequately training a deep neural network according to a strategy-environment-to-payoff table , then we initialize randomized strategy input and propagate the error between the actual output and the desired output back to the initially-randomized strategy input in the “ input layer ” of the trained deep neural network gradually to perform a task similar to “ human deduction ” . And we view the final strategy input in the “ input layer ” as the fittest strategy for a neural network when confronting the observed environment input from the world outside . 1 INTRODUCTION . Contrary to most reinforcement learning studies ( as in “ Playing Atari with Deep Reinforcement Learning ( Mnih et al. , 2013 ) . ” ) , which emphasize on training a deep neural network to approximate its output layer to certain strategies and view the output layer of a deep neural network as its strategy in a certain natural environment , this paper provides a revolutionary way for reinforcement learning and a possible road toward general A.I .. Also , contrary to most Game Theory studies , which emphasize on the role of human experts to device smart mathematics algorithm ( as in “ When Machine Learning Meets AI and Game Theory ( Agrawal & Jaiswal , 2012 ) . ” ) , this paper emphasizes on the role of machine and tends to show a bridge between A.I . territory and Game Theory . The method is simple . In short , after training a deep neural network according to a strategyenvironment-to-payoff ( input-to-output ) table that records possible strategies and the environments along with the consequential payoffs , then we randomize the strategy input in the “ input layer ” and propagate the error between the actual output and the desired output back to the strategy input in the “ input layer ” of the same trained deep neural network recurrently to perform a task similar to “ human deduction ” . And we view the final strategy input in the “ input layer ” as the fittest strategy for a neural network confronting the observed environment input from the world outside . Since the information is sent back to the input layer in a reversed way like deduction in logic , we can call this “ Deep Deducing ” . By using the same trick , we can force a neural network to automatically track down the Nash Equilibrium in a Simultaneous Discrete Game ( which will be explained later ) . Since Nash Equilibrium is the optimal combination of strategies from which no player will be willing to deviate , and , for any player who deviates from NE , the player will be punished , the ability to find the Nash Equilibrium in advance in a randomized complex system indicates certain intelligence in an autonomous agent . 2 ALGORITHM : DEEP DEDUCING . To grasp the intuition behind Deep Deducing , we will first take a mosquito for example as shown in Figure 1 . Imagine the brain of a mosquito consists of a canonical deep feedforward neural network with only one input layer , one hidden layer , and one output layer . The input layer consists of strategy input and environment input . And the output layer consists of payoff out . A list of neurons in the strategy input records the strategies ( denoted is ) that the mosquito has adopted in response to certain environment information ( denoted in ) , which environment information is also simultaneously recorded by the other list of the neurons in the environment input as well . Meanwhile the neurons in the payoff output also instantaneously record the payoff ( denoted o ) to the mosquito when it adopts a certain strategy is under a certain environment in . The concatenation of is and in is denoted as i . The brain of the mosquito is trained to memorize the set of concatenations of strategy and environment ( denoted I ) as well as the set of their payoffs ( denoted O ) . That is the set of the weights of the synapse ( denoted W ) are adjusted through ( Stochastic Gradient Descent and Back Propagation ) : Wt+1 ←Wt − α ∂ ∂Wt E ( o− o ′ ) ( 1 ) where t is the current epoch , α is the learning rate , E is the error function ( the loss or cost function , here we use squared error ) , and o ′ is the estimated payoff generated by current i and Wt . This is Deep Learning . For illustration , the consequence of seeing human ’ s hand and choosing not to fly away is recorded . Of course the consequence is death , in which case the payoff to the mosquito is [ 0 ] . A cloud ( of IoT ) records the consequence ( payoff ) of every combination and trains the brain ( the neural network ) of the mosquito according to Figure 2 as below . When the brain ( the neural network ) of the mosquito is adequately trained ( by using Stochastic Gradient Descent ) according to the table in Figure 2 , upon seeing a human ’ s hand at presence , the brain ( the neural network ) of the mosquito must figure out and make deduction about what strategy it shall adopt in order to stay alive . In another word , how could the neural network figure out a strategy that , when put into the neural network along with environment input [ 1 , 0 , 0 ] , can generate the payoff [ 1 ] ? The answer that Deep Deducing gives is rather simple – Just randomize the strategy input in the input layer of the trained neural network and let the neurons that represent the strategy input to adjust itself gradually to render the neural network to reach output [ 1 ] by propagating the error between the generated output and the desired output [ 1 ] back to the neurons in the strategy input itself for fixed epochs or as long as the generated output ( of the adjusted strategy input obtained in the last epoch ) does not adhere to [ 1 ] in a reasonable deviation . That is , for a given in where 0 ≤ in ≤ 1 , a set of trained Wtrained and a pre-fixed desired payoff od , 0 ≤ od ≤ 1 , we first initialize a randomized strategy input is0 at time zero where 0 ≤ sigmoid ( i s 0 ) ≤ 1 . Then , by using Back Propagation , we let : istrategyt+1 ← i strategy t − α ∂ ∂istrategyt E ( od − o ′ t ) ( 2 ) where t is the current epoch , α is the deducing rate ( like learning rate ) , E is the error function , and o ′ t is the estimated payoff generated by current sigmoid ( i s t ) , i n and Wtrained . This is “ Deep Deducing ” . Please notice that , in , Wtrained and od are fixed , and the only thing that is changing is ist and o ′ t. And there is no Stochastic Gradient Descent since the desired payoff od is pre-fixed . By exploiting the existing trained neural network and the trick of Back Propagation , Deep Deducing forms a single , compounded and self-completed optimization problem for ist to reach the global minimum error , and therefore any kind of optimization skill , such as gradient-based particle swarm optimization , can be applied here as well . For simplicity , we will only use the most plain and modest gradient descent with only one starting point for ist to force the machine to track down Nash Equilibrium . The whole process is illustrated in Figure 3 . Finally , after fixed epochs , according to the preliminary experimental result shown in Figure 3 , the mosquito will obtain a strategy input that approximate [ 1 , 0 , 0 ] when argmaxed and one-hotted , which means the mosquito better just flies away . This final strategy information will later be sent to other parts of the mosquito to initiate the procedure of flying such as twitching wing tensors . We speculate that this simple three-layered deep neural network constitutes the very core of the mosquito ’ s decision-making mechanism . Therefore , the Deep Deducing algorithm can be dissected into two phases . The first is the training or learning phase , and the second is the deduction phase . In the training or learning phase , the mosquito learns about the consequence in response to the combination of strategy and environment . In the deduction phase , the mosquito must figure out a strategy that can make it stay alive when human is at presence . The pseudo code of Deep Deducing for the learning phase is shown in Algorithm 1 . Algorithm 1 : training phase for the mosquito example 1 : I← set of strategies and environments 2 : O← set of corresponding payoffs 3 : W0 ← initialize set of randomizedweights at time zero 4 : α← learning rate 5 : Nepochs ← number of epochs 6 : Nsamples ← number of training samples 7 : for t in [ Nepochs ] do 8 : index← randomint ( Nsamples ) 9 : i← I [ index ] 10 : o← O [ index ] 11 : o ′ ← feedforward ( i , Wt ) 12 : Wt+1 ←Wt − α ∂∂WtE ( o− o ′ ) 13 : end for 14 : Wtrained ←WNepochs As for the deduction phase , the pseudo code can be shown in Algorithm 2 . Algorithm 2 : deduction phase for the mosquito example 1 : istrategy0 ← initialize randomstrategy input at time zero 2 : inature ← environment input 3 : od ← initialize desired payoff output 4 : α← deducing rate 5 : Nepochs ← number of epochs 6 : for t in [ Nepochs ] do 7 : o ′ t ← feedforward ( concatenate ( sigmoid ( istrategyt ) , i nature ) , Wtrained ) 8 : istrategyt+1 ← i strategy t − α ∂∂istrategyt E ( o d − o′t ) 9 : end for 10 : optimal strategy ← onehot ( argmax ( sigmoid ( istrategyNepochs ) ) ) For the sake of simplicity , we will summarize the deduction phase in just one single sentence - Tuning the neurons in the strategy input to generate the desired output . 3 EXPERIMENT : TRACKING DOWN THE NASH EQUILIBRIUM IN A SIMULTANEOUS DISCRETE GAME . The first experiment here we will examine is simply the extension of the notion of the mosquito example stated above . If we take the mosquito and the hand of human as the two players A and B in a Simultaneous Discrete Game in Game Theory . We can train a neural network according to the strategies-to-payoff table for player A and B , then we can tune the neurons in the strategy input for player A to generate the desired output for player A , and vice versa . After we interchangeably and iteratively tune these two sets of neurons each represents the strategy input for player A and B , these two sets of neurons will converge to the Nash Equilibrium . For example , in Simultaneous Discrete Game , we often see the strategies-to-payoff table for player A and B as shown in Table 1 . In the same manner , we first train a neural network according to the strategies-to-payoff table for player A and B as illustrated in Figure 4 below . Second , since it is not a zero-sum game , we will tune ( by one epoch ) the neurons which represent the strategy input for player A to generate the desired payoff output for player A , namely [ 1 , * ] ( where “ * ” stands for whatever the payoff for player B already is ) , and then we will tune ( by one epoch ) the neurons which represent the strategy input for player B to generate the desired output for player B , namely [ * , 1 ] ( where “ * ” stands for whatever the payoff for player A already is ) . We will keep this procedure for fair epochs as shown in Figure 5 . After fair epochs , as the preliminary experimental result in Figure 5 , the neurons which represent the strategy input for player A will converge to [ 1 , 0 , 0 ] when argmaxed and one-hotted . And the neurons which represent the strategy of player B will also converge to [ 1 , 0 , 0 ] when argmaxed and one-hotted . This means that player A and player B will both adopt strategy [ 1 , 0 , 0 ] , which is exactly the Nash Equilibrium in the strategies-to-payoff table in Table 1 . The more formal experiment result is shown in Figure 9 . In Game Theory , this strategies-to-payoff table is Prisoner ’ s Dilemma , and strategy input [ 1 , 0 , 0 ] is the “ betrayal ” strategy . Deep Deducing automatically tracks down the Nash Equilibrium in Prisoner ’ s Dilemma which is that both players will betray each other ( as in “ Primer in Game Theory ( Gibbons , 1994 ) . ” ) . Since Nash Equilibrium is the combination of strategies that no player is willing to deviate from , and , for any player who does not choose the NE strategy , he or she will be punished for his or her deviation ( as in “ Games of Strategy , Second Edition ( Dixit & Skeath , 2004 ) . ” ) , the ability to track down NEs in a complex system in advance shows certain intelligence in a autonomous system . By this logic , since Deep Deducing , in the preliminary result , tracks down NE in the prisoner ’ s dilemma , there is certain intelligence in Deep Deducing . The pseudo code here can also be dissected into two phases , namely the training or learning phase and the deduction phase . For the training phase , we simply duplicate the training phase in the mosquito example except that this time the environment input was replace by strategy input for player B , and the original strategy by strategy input for player A .

Summary: The paper proposes a method of searching for a Nash equilibrium strategy in games where the strategy-to-payoff mapping is defined by a neural network. The idea is to perform gradient optimization of the payoff w.r.t. the strategy. Preliminary results on tic-tac-toe and variations of the prisoner’s dilemma task are presented. The paper has an interesting idea at the core. However, it is poorly written, does not properly discuss the related works and does not present a convincing method or experimental results.