Title: FROM DEEP LEARNING TO DEEP DEDUCING: AUTOMATICALLY TRACKING DOWN NASH EQUILIB-RIUM THROUGH AUTONOMOUS NEURAL AGENT, A POSSIBLE MISSING STEP TOWARD GENERAL A.I

Abstract: Contrary to most reinforcement learning studies, which emphasize on training a deep neural network to approximate its output layer to certain strategies, this paper proposes a reversed method for reinforcement learning. We call this "Deep Deducing". In short, after adequately training a deep neural network according to a strategy-environment-to-payoff table, then we initialize randomized strategy input and propagate the error between the actual output and the desired output back to the initially-randomized strategy input in the "input layer" of the trained deep neural network gradually to perform a task similar to "human deduction". And we view the final strategy input in the "input layer" as the fittest strategy for a neural network when confronting the observed environment input from the world outside.

1. INTRODUCTION:

Contrary to most reinforcement learning studies (as in "Playing Atari with Deep Reinforcement Learning (Mnih et al., 2013) ."), which emphasize on training a deep neural network to approximate its output layer to certain strategies and view the output layer of a deep neural network as its strategy in a certain natural environment, this paper provides a revolutionary way for reinforcement learning and a possible road toward general A.I.. Also, contrary to most Game Theory studies, which emphasize on the role of human experts to device smart mathematics algorithm (as in "When Machine Learning Meets AI and Game Theory (Agrawal & Jaiswal, 2012) ."), this paper emphasizes on the role of machine and tends to show a bridge between A.I. territory and Game Theory.

The method is simple. In short, after training a deep neural network according to a strategyenvironment-to-payoff (input-to-output) table that records possible strategies and the environments along with the consequential payoffs, then we randomize the strategy input in the "input layer" and propagate the error between the actual output and the desired output back to the strategy input in the "input layer" of the same trained deep neural network recurrently to perform a task similar to "human deduction". And we view the final strategy input in the "input layer" as the fittest strategy for a neural network confronting the observed environment input from the world outside. Since the information is sent back to the input layer in a reversed way like deduction in logic, we can call this "Deep Deducing". By using the same trick, we can force a neural network to automatically track down the Nash Equilibrium in a Simultaneous Discrete Game (which will be explained later). Since Nash Equilibrium is the optimal combination of strategies from which no player will be willing to deviate, and, for any player who deviates from NE, the player will be punished, the ability to find the Nash Equilibrium in advance in a randomized complex system indicates certain intelligence in an autonomous agent.

2. ALGORITHM: DEEP DEDUCING:

To grasp the intuition behind Deep Deducing, we will first take a mosquito for example as shown in Figure 1 .

Figure 1 : example of a simplified brain of a mosquito Imagine the brain of a mosquito consists of a canonical deep feedforward neural network with only one input layer, one hidden layer, and one output layer. The input layer consists of strategy input and environment input. And the output layer consists of payoff out. A list of neurons in the strategy input records the strategies (denoted i s ) that the mosquito has adopted in response to certain environment information (denoted i n ), which environment information is also simultaneously recorded by the other list of the neurons in the environment input as well. Meanwhile the neurons in the payoff output also instantaneously record the payoff (denoted o) to the mosquito when it adopts a certain strategy i s under a certain environment i n . The concatenation of i s and i n is denoted as i.

The brain of the mosquito is trained to memorize the set of concatenations of strategy and environment (denoted I) as well as the set of their payoffs (denoted O). That is the set of the weights of the synapse (denoted W) are adjusted through (Stochastic Gradient Descent and Back Propagation):

EQUATION

)

where t is the current epoch, α is the learning rate, E is the error function (the loss or cost function, here we use squared error), and o is the estimated payoff generated by current i and W t . This is Deep Learning.

For illustration, the consequence of seeing human's hand and choosing not to fly away is recorded.

Of course the consequence is death, in which case the payoff to the mosquito is [0]. A cloud (of IoT) records the consequence (payoff) of every combination and trains the brain (the neural network) of the mosquito according to Figure 2 as below.

Figure 2 : example of the learning process in a mosquito's brain

When the brain (the neural network) of the mosquito is adequately trained (by using Stochastic Gradient Descent) according to the table in Figure 2 , upon seeing a human's hand at presence, the brain (the neural network) of the mosquito must figure out and make deduction about what strategy it shall adopt in order to stay alive. In another word, how could the neural network figure out a strategy that, when put into the neural network along with environment input [1, 0, 0], can generate the payoff [1]?

The answer that Deep Deducing gives is rather simple -Just randomize the strategy input in the input layer of the trained neural network and let the neurons that represent the strategy input to adjust itself gradually to render the neural network to reach output [1] by propagating the error between the generated output and the desired output [1] back to the neurons in the strategy input itself for fixed epochs or as long as the generated output (of the adjusted strategy input obtained in the last epoch) does not adhere to [1] in a reasonable deviation.

That is, for a given i n where 0 ≤ i n ≤ 1, a set of trained W trained and a pre-fixed desired payoff o d , 0 ≤ o d ≤ 1, we first initialize a randomized strategy input i s 0 at time zero where 0 ≤ sigmoid(i s 0 ) ≤ 1. Then, by using Back Propagation, we let:

EQUATION

)

where t is the current epoch, α is the deducing rate (like learning rate), E is the error function, and o t is the estimated payoff generated by current sigmoid(i s t ), i n and W trained . This is "Deep Deducing". Please notice that, i n , W trained and o d are fixed, and the only thing that is changing is i s t and o t . And there is no Stochastic Gradient Descent since the desired payoff o d is pre-fixed. By exploiting the existing trained neural network and the trick of Back Propagation, Deep Deducing forms a single, compounded and self-completed optimization problem for i s t to reach the global minimum error, and therefore any kind of optimization skill, such as gradient-based particle swarm optimization, can be applied here as well. For simplicity, we will only use the most plain and modest gradient descent with only one starting point for i s t to force the machine to track down Nash Equilibrium.

The whole process is illustrated in Figure 3 .

Figure 3 : example of the deduction process in a mosquito's brain Finally, after fixed epochs, according to the preliminary experimental result shown in Figure 3 , the mosquito will obtain a strategy input that approximate [1, 0, 0] when argmaxed and one-hotted, which means the mosquito better just flies away. This final strategy information will later be sent to other parts of the mosquito to initiate the procedure of flying such as twitching wing tensors. We speculate that this simple three-layered deep neural network constitutes the very core of the mosquito's decision-making mechanism.

Therefore, the Deep Deducing algorithm can be dissected into two phases. The first is the training or learning phase, and the second is the deduction phase. In the training or learning phase, the mosquito learns about the consequence in response to the combination of strategy and environment. In the deduction phase, the mosquito must figure out a strategy that can make it stay alive when human is at presence. The pseudo code of Deep Deducing for the learning phase is shown in Algorithm 1. index ← randomint(N samples )

9: i ← I[index] 10: o ← O[index] 11: o ← f eedf orward(i, W t ) 12: W t+1 ← W t -α ∂ ∂Wt E(o -o ) 13: end for 14: W trained ← W N epochs

As for the deduction phase, the pseudo code can be shown in Algorithm 2.

Algorithm 2 : deduction phase for the mosquito example 1: i strategy 0 ← initialize random strategy input at time zero

2: i nature ← environment input 3: o d ← initialize desired payof f output 4: α ← deducing rate 5: N epochs ← number of epochs 6: for t in [N epochs ] do 7: o t ← f eedf orward(concatenate(sigmoid(i strategy t ), i nature ), W trained ) 8: i strategy t+1 ← i strategy t -α ∂ ∂i strategy t E(o d -o t )

9: end for 10: optimal strategy ← onehot(argmax(sigmoid(i strategy N epochs )))

For the sake of simplicity, we will summarize the deduction phase in just one single sentence -Tuning the neurons in the strategy input to generate the desired output.

3. EXPERIMENT: TRACKING DOWN THE NASH EQUILIBRIUM IN A SIMULTANEOUS DISCRETE GAME:

The first experiment here we will examine is simply the extension of the notion of the mosquito example stated above.

If we take the mosquito and the hand of human as the two players A and B in a Simultaneous Discrete Game in Game Theory. We can train a neural network according to the strategies-to-payoff table for player A and B, then we can tune the neurons in the strategy input for player A to generate the desired output for player A, and vice versa. After we interchangeably and iteratively tune these two sets of neurons each represents the strategy input for player A and B, these two sets of neurons will converge to the Nash Equilibrium.

For example, in Simultaneous Discrete Game, we often see the strategies-to-payoff table for player A and B as shown in Table 1 .

B Strategies [1, 0, 0] [0, 0, 1] A Strategies [1, 0, 0] [5, 5] [10, 0] [0, 0, 1] [0, 10] [7, 7]

Table 1 : table of prisoner's dilemma

In the same manner, we first train a neural network according to the strategies-to-payoff table for player A and B as illustrated in Figure 4 below. Second, since it is not a zero-sum game, we will tune (by one epoch) the neurons which represent the strategy input for player A to generate the desired payoff output for player A, namely [1, *] (where "*" stands for whatever the payoff for player B already is), and then we will tune (by one epoch) the neurons which represent the strategy input for player B to generate the desired output for player B, namely [*, 1] (where "*" stands for whatever the payoff for player A already is). We will keep this procedure for fair epochs as shown in Figure 5 .

After fair epochs, as the preliminary experimental result in Figure 5 , the neurons which represent the strategy input for player A will converge to [1, 0, 0] when argmaxed and one-hotted. And the neurons which represent the strategy of player B will also converge to [1, 0, 0] when argmaxed and one-hotted. This means that player A and player B will both adopt strategy [1, 0, 0], which is exactly the Nash Equilibrium in the strategies-to-payoff table in Table 1 . The more formal experiment result is shown in Figure 9 .

In Game Theory, this strategies-to-payoff table is Prisoner's Dilemma, and strategy input [1, 0, 0] is the "betrayal" strategy. Deep Deducing automatically tracks down the Nash Equilibrium in Prisoner's Dilemma which is that both players will betray each other (as in "Primer in Game Theory (Gibbons, 1994) ."). Since Nash Equilibrium is the combination of strategies that no player is willing to deviate from , and, for any player who does not choose the NE strategy, he or she will be punished for his or her deviation (as in "Games of Strategy, Second Edition (Dixit & Skeath, 2004) ."), the ability to track down NEs in a complex system in advance shows certain intelligence in a autonomous system. By this logic, since Deep Deducing, in the preliminary result, tracks down NE in the prisoner's dilemma, there is certain intelligence in Deep Deducing. The pseudo code here 

4. EXPERIMENT RESULTS:

To thoroughly testify the functionality of Deep Deducing, we will first permute the payoffs in the prisoner dilemma in  and [1, 1] , for each player. Also, the payoff for each player in each table is represented by digits as well. For example, payoff 3 is represented by [1, 1, 1, 0, 0, 0, 0, 0, 0, 0] with possible maximum payoff 10. The sequence is irrelevant and can be changed at user's will because the volume of error sent back to the input layer is all that matters. Here, we set maximum payoff as well as the optimal desired payoffs to 10 for each player. The final strategy adopted by a player depends on the closest range of its final strategy input to For the first test, the prisoner's dilemma in Table 1 is permuted into 4 different tables as below in Table 2 . As we can see in the left part of Figure 15 , regardless of the initially randomized starting points (denoted red dots) for the strategy inputs, the strategy inputs for player A and B converge to NE (denoted blue dots) in each table under Deep Deducing.

B Strategies [0] [1] A Strategies [0] [5, 5] [10, 0] [1] [0, 10] [7, 7] B Strategies [0] [1] A Strategies [0] [7, 7] [0, 10] [1] [10, 0] [5, 5] B Strategies [0] [1] A Strategies [0] [10, 0] [7, 7] [1] [5, 5] [0, 10] B Strategies [0] [1] A Strategies [0] [0, 10] [5, 5] [1] [7, 7] [10, 0]

If We further fix the strategy input for player A to [0], and tune only the strategy input for player B, we will obtain the right part of Figure 15 . As we can see in the right part of Figure 15 , the strategy inputs of player B still converge to the optimal (and conditional) strategies for player B in Table 2 (denoted blue dots), regardless of its initially randomized starting positions (denoted red dots). 2 We can see the smoothness of geometry for the strategy inputs increases as the neural network learns more about a game table and further lessens the chance of getting stuck at local minimum for gradient descent and other types of optimization skills (see Figure 9 ). This explains why Deep Deducing can be attached to Deep Learning.

However, how did the strategy inputs for player A and B automatically track down NE in the left part of Figure 15 ? This paper proposes two possible explanations. The first possible and more vivid explanation is that, imagine strategy inputs for A and B are two small ants in two different mountain sides, they both want to climb to the lowest points of their own mountains, however they overlap each other (phantom space) and wherever ant A moves, ant B will be carried to the same direction due to the this phantom effect, and vice versa. If ant A moves first and then ant B moves afterwards repeatedly, in the end, ant A and B will pull each other to a certain point (as the blue rectangles in Figure 8 ), and this point is never the lowest point of their own mountain side. This can be shown in Figure 8 . The second possible explanation is that, In Kakutani's fixed point theorem, a topology space kept mapped into itself (if closed, continuous) will converge to a point. This point is the fixed point under every mapping. By Nash theorem, if the mapping is the pursuit of the best interest of each player, this point will be the Nash Equilibrium (as in "Introduction to Topology: Pure and Applied (Adams & Franzosa, 2008) ."). In Figure 15 , we can observe that the trajectory of strategy inputs for player A and B under Deep Deducing algorithm converge to a certain point regardless of their initial different position. Therefore, by Kakutani's fixed point theorem, there is a fixed point under every mapping. Further by Nash theorem, the fixed point shall also be the Nash Equilibrium since Deep Deducing mimics the rationale of human player in the pursuit of their own interest. Therefore, Deep Deducing tracks down the fixed point, and, therefore tracks down the Nash Equilibrium. As we can see, Deep Deducing is almost 100 % accurate in tables of size 2x2, however the precision in tables of size 4x4 drops to 70%. This is because the learning process only lessens the chance of getting stuck at local minimum, we are only using plain and modest gradient descent with Back Propagation, and the strategy inputs for player A and B might get stuck at "common" local minimums in 4x4 tables, which minimum is the common minimum in the error surface of player A and B respectively (so we call it "common") in which case we cannot expect ant A and B to pull each other out of their local minimums. To solve this problem, we will apply a simplified gradient-based particle swarm optimization and transform Equation 2 into Equation 3a and 3b for player A and 4a and 4a for player B:

EQUATION

where i

A optimal t

is the optimal solution in the strategy swarm for player A at time t, and vice versa. Since Deep Deducing exploits Back Propagation as the gradient descent method,

∂ ∂i B t E(o dB -o B t

) can serve as a gradient for swarm optimization.

We initiate only seven particles in each strategy swarm A and B separately. The deducing epoch is set to be 100*100, where 100 is the iteration inside each swarm and 100 between swarm A and swarm B.

In the beginning of iteration inside a swarm, we keep the leader obtained from the last iteration and renew the rest of swarm (randomly) as well as all the momentum. In the mid of the iteration inside the swarm, the particles will learn from each other and improve their positions. And in end of the swarm iteration, the best strategy input will survive and be elected as the new leader and the rest of the swarm will be killed. This leader of swarm A will become a variable to consider with, for the iteration inside swarm B. The same process go through the strategy swarm of A and B. We can simply imagine this process as the democratic election system and international competition system. By doing so, we exploit the benefit of gradient-based particle swarm optimization to escape local minimums while keeping the swarm as resilient as possible by introducing new particles to compete with (and learn from) the old leader, killing less efficient particles in the swarm, and then raising a possible new leader to confront the enemy swarm. We will then obtain Figure 10 as the renewed accuracy for 4x4 tables: For more interesting result, please consult Appendix. From evidence above (including Appendix), we can conclude that there is certain intelligence in Deep Deducing.

5. CONCLUSIONS:

The experiments shown here are not all inclusive, but the preliminary experiments indicate that Deep Deducing can perform the similar task to human deduction by simply propagating error further back to the input layer. In this sense, the input layer of a neural network does not only serve as an information receiver (as we always envisage) but also an information sender, which information is further sent to other parts of the body such as muscles or even other neural networks.

The whole traditional deep neural network can be viewed as a gigantic memory or knowledge reservoir of strategy, environment and payoff. In a word, a simple traditional deep neural network itself is already a memory pool, ready to have its information be extracted for later usage, and Recurrent Neural Network just augmented this function by allowing variable inputs to be considered. This finding might implement the notion of "memory" as we envisaged (as in "Deep Learning (LeCun et al., 2015) .").

By back-propagating error back to the input layer, a traditional deep neural network forms an internal information exchange route similar to the internal information exchange in Hopfield Neural Network. The only difference is that the former uses Back Propagation algorithm while the latter exploits the notion of thermodynamics (Hopfield Neural Network can be viewed as a Restricted Boltzmann Machine with visible layers mapping to itself in supervised learning, while a Boltzmann Machine can be viewed as a Hopfield Neural Network with hidden layers and simulated annealing, the latter see "Connectionist architectures for Artificial Intelligence (Fahlman & Hinton, 1987) ."). This internal information from other pathways does not only train the synapses itself as we always envisage (as in "How neural networks learn from experience (Hinton, 1992) ."), but also the source (neurons in the input layer) as well.

There are pros and cons to Deep Deducing. First, Deep Deducing is highly intuitive and can be easily coded. Second, Deep Deducing can be seamlessly combined with other types of neural networks as long as the back-propagation algorithm or back-propagation-like algorithm is compatible with the latter. However, the downside is that Deep Deducing requires heavy epochs as well as training time on the training or learning phase in order to achieve high precision in the deduction phase in some application.

There is still an abundant discovery that we have made, and we are still making progress.

Figure 13 : deduction phase in Sequential Discrete Game Finally, the final strategy input for player A in layer 1 in figure Figure 13 is the optimal strategy for player A in layer 1 in the real world. After player A chooses its strategy, in the next layer of the game in the real world, player B can exploit the same technic to obtain the optimal strategy for himself or herself in layer 2. Both players will converge to the Nash Equilibrium (route) in Sequential Discrete Game in Game theory in the end. The same methodology also applies to Sequential Discrete Game with arbitrary numbers of layers.

By this method, we can teach a RNN to learn to play Tic-Tac-Toe game. The experiment result shows that Deep Deducing can handle Tic-Tac-Toe Game. The whole source code can be seen at: https://github.com/AnonymousDeepDeducingCode/DeepDeducing/ tree/master, open for inspection.

The model here we will use is simply the traditional canonical 3-layered (with one hidden layer fully interconnected) recurrent neural network with no modification. The size of the neural network is (9+9)*40*2. The squared error is used for error function (loss or cost function).

The program we set up will record historic strategies of the players and automatically generate the future legal strategies of the players to the end of the game as well as their final payoffs. The strategy of a player is represented by 9 digit corresponding to its position in the game board when flattened. The payoff for a player is simple, [1] or [0], meaning winning or not winning (might be draw).

When the recurrent neural network is trained according to Figure 12 , the recurrent neural network will take history into consideration and deep-deduce the current optimal strategy according to Figure 13 in a pre-fixed future (here we set it to be the steps toward the end of the game). The final current strategy is the argmax of the deduced current "legal" strategy corresponding to its position in the game board when flattened.

Under review as a conference paper at ICLR 2019 The order of the game is that player A will move first.

We will let Deep Deducing RNN play with the human player and let the readers judge its intelligence by its states of the play. The states are recorded as the rows in Figure 14 (where "O" represents player A assumed by human and "X" represents player B assumed by Deep Deducing):

O O X O X O O X O X O X O X O O X O X O X O X O X O X O O X O X O X O X O O X O X O O X O X O X O X O O X O X O X O X O X O X O O X O X O X O X O O X O X O O X O X O X O X O O X O X O X O X O X O O X O X O X O X

Figure 14 : the states of the play in experiment with Deep Deducing

As we can see, if adequately tuned, in some tricky first-hand advantages, Deep Deducing can avoid bad moves in order to survive.

However, the curious thing is that if we set the number of steps foreseen in the future to be odds, Deep Deducing will take a aggressive stance. In a sense, it will not defend, and, In some cases, it leads to its defeat. If we set the number of steps foreseen in the future to be evens, Deep Deducing will take a defensive stance. In some cases, it misses its chance to win, although it won't be defeated. Therefore further improvement is required.

Nonetheless, the goal of the preliminary experiment is to indicate that Deep Deducing can perform the similar task to human deduction by simply propagating error further back to the input layer.

APPENDIX II (EXPERIMENT III):: DEEP DEDUCING PERFORMING CROW REASONING, TRANSFORMING FEED-FORWARD NEURAL NETWORK INTO RECURRENT NEURAL NETWORK Imagine a crow in the video: https://youtu.be/cbSu2PXOTOc was trained to solve different puzzles such as dropping stones into bottle to get food and picking up a key to unlock the cage to get food. The crow is familiar with each different puzzle and it was trained to solve each of the puzzles separately and non-sequentially. However, when faced with a consecutive puzzle that must be solved by dropping stones into bottle to get a key and then use the key to unlock the cage to get food, can the crow can still solve the puzzle? The answer is yes. The crow, though not knowing how to speak human language, can perform the reasoning task of an intelligent specie.

In this experiment, we will try to let Deep Deducing mimic the rationale of the crow. Since Deep Deducing exploits Back Propagation and propagates error back to the input layer, and the input layer of a neural network can serve as the output layer of another, for a single trained deep feed-forward neural network, we can stack the deep feed-forward neural network into a recurrent neural network and make a crow-like reasoning to solve a consecutive puzzle. This single trained deep feed-forward neural network resembles the brain of the crow after it being trained to solve each puzzle separately and non-sequentially. The DFNN-transformed recurrent neural network resembles the brain of the crow when faced with a consecutive puzzle, and the recurrent neural network must figure out a sequence of strategies to meet the goal, such as dropping stones into water to get the key and then using the key to get the food. The whole process can be shown in Figure 15 : To testify the functionality of this method, we will start from teaching a machine to move points A to points B in a 5x5 grid. We first train a deep feed-forward neural network to learn what position in the grid the points will achieve when it pushes the present points to left, up, right, down or remains the points still. Then we will permute the deep feed-forward neural network into a recurrent neural network by Figure 15 , making the hidden as well as the output layers as the states of the grid and the input layer as the direction where it moves the points. Then we set the position of points B as the goal where the machine is expected to push points A to match the status of points B. By deep deducing, the machine starts a crow-like reasoning and find the shortest route to push points

:

O X O X O O X O X O X O X O O X O X O O X O X O X O X O O X O X O X O X O X O X O O X O X O X O X O O X O X O O X O X O X O X O O X O X O X O X O X O X O O X O X O X O X O O X O X O O X O X O X O X O O X O X O X O X O X

Figures and Tables:

Figure 4: example of the learning process for prisoners

Figure 5: example of the deduction process for prisoners

Since a canonical deep neural network use sigmoid as activation function, when the training of the neural network is sufficient, a little deviation in the input layer does not generally affect the output of the neural network as a whole, which renders the optimization geometry for the strategy inputs rather smooth and further lessens the chance of getting stuck at local minimum for gradient descent and other types of optimization skills. If we take sum of the squared errors (o dBo B ) 2 as axis z, we can obtain Figure 17.

Figure 7: the learning or training process to smooth the optimization geometry for player B under lower left tablein Table 2

Figure 8: the trajectories of strategy inputs under different error surface for player A and B under lower left table inTable 2

the second test to track down NEs in 50 randomly generated game tables, we instead use a 3-layered deep feedforward neural network with size (1+1)*15*(10+10) for table 2x2 and (2+2)*15*(10+10) for table 4x4 with learning and deducing rate at 0.1 and deducing epochs at 5000. The success rate to track down NEs in each table type (2x2 and 4x4) varies with learning or training epochs. The final result is shown in Figure 9.

Figure 9: accuracy for tracking down NEs in different learning epochs

Figure 10: accuracy for tracking down NEs in tables 4x4 in different learning epochs coupled with simplified gradient-based swarm optimization

Figure 15: transforming Feed-forward Neural Network into Recurrent Neural Network to perform crow reasoning by Deep Deducing

Table 1 to check if Deep Deducing with a set of trained synapses can track down Nash Equilibriums (NEs) in different locations with arbitrarily randomized starting points for strategy inputs for player A and B in each permuted table. Then we will generate 50 game tables with the size of 2x2 and 4x4, with randomly generated payoffs, to check if Deep Deducing with a set of trained synapses can track down the sole and the only NE in each table with different size. The program we set up guarantees there is only one NE in each table. Therefore, at a random guess, the chance to spot NE is 25 % in table 2x2 and 6.25 % in table 4x4.The model here we will use is simply the traditional canonical 3-layered deep feedforward neural network with no modification. The squared error is used for error function (loss or cost function). To avoid noise in Back Propagation in Deep Deducing, the strategies are represented in digits. For example, strategies in table 2x2 can be represented by 1 digit, [0] or [1], since 1 digit is sufficient to represent all possible strategies for a player. Strategies in table 4x4 can be represented by 2 digits,

Table content: <table><tr><td>[0, 0], [1, 0], [0, 1]</td></tr></table>

[0]  or[1]. For example, if the strategy input for player A arrives at [0.13] in the end, player A will adopt strategy[0]. If the strategy input for player B arrives at [0.653] in the end, player B will adopt strategy[1].The size of the 3-layered neural network is set to be n*h*m, where n is the sum of the size of the strategy inputs for A and B in each table, h is the size of hidden layer, and m is the sum of the size of the neurons that represent possible maximum payoffs for player A and B in each table. Therefore,

Table content: <table/>



Table content: <table/>

Peer review (iclr2019):
 The paper proposes a method of searching for a Nash equilibrium strategy in games where the strategy-to-payoff mapping is defined by a neural network. The idea is to perform gradient optimization of the payoff w.r.t. the strategy. Preliminary results on tic-tac-toe and variations of the prisoner’s dilemma task are presented. The paper has an interesting idea at the core. However, it is poorly written, does not properly discuss the related works and does not present a convincing method or experimental results. Pros: * The paper considers an interesting question of exploring the applications of neural networks to the game theory problems. * The idea of the paper is reasonable. It makes sense to me to perform gradient-based search over the strategies (assuming that the payoff is differentiable). Cons: * Writing - The paper is over the mandatory length limit of 10 pages. - The paper makes a grandiose claim: “this paper provides a revolutionary way for reinforcement learning and a possible road toward general A.I.” However, there are arguably no revolutionary ideas, and certainly no reinforcement learning experiments! - Despite the claim, the novelty of the paper is limited. There is no discussion of the related work: optimization of the neural networks w.r.t. the inputs [1]; related RL ideas such as model-based learning [2,3] and Monte-Carlo Tree Search [4]. - The problem being solved is never formally stated. As far as I understand, Nash equilibrium is usually defined (1) in mixed strategies, while the paper seems to consider pure strategies; (2) in the scenario where every player attempts to maximize their payoff, while in the paper the players attempt to achieve some pre-fixed value of the payoff. - The flow of the paper is generally poor. Instead of presenting a general solution and then showcasing its applications, the paper iterates on similar ideas multiple times. For example, all four algorithms are just gradient-based optimization of either the weights or the inputs to a model. - The paper provides extremely misleading analogies and explanations. I am quite sure that a mosquito brain is not a one hidden layer fully-connected neural network! Also, the example of avoiding a moving hand is poor: since the outcome is life or death, the learning should happen via evolution, not during the lifetime of a single insect. The claim that the neural networks with sigmoid activation functions are less prone to local optima is questionable as well. * Method and experiments - The proposed method is essentially a greedy gradient-based planning procedure. For this to work, we need to have a very good environment model. This is a strong assumption that is not discussed. - The experiments are performed on very simple synthetic problems: matrix games and tic-tac-toe. They do not suggest that the method is general and can work on harder problems, say, Sokoban [2]. - The experiments do not present any baselines, so it is unclear how well the method performs compared to the alternatives. One obvious candidate is gradient-free optimization, such as Nelder-Mead, and gradient descent with momentum, which can be less prone to local optima. [1] Brandon Amos, Lei Xu, J. Zico Kolter “Input Convex Neural Networks”, ICML 2017 [2] Racanière et al. “Imagination-Augmented Agents for Deep Reinforcement Learning”, NIPS 2017 [3] David Ha, Jürgen Schmidhuber “Recurrent World Models Facilitate Policy Evolution”, NIPS 2018 [4] Thomas Anthony, Zheng Tian, David Barber “Thinking Fast and Slow with Deep Learning and Tree Search”, NIPS 2017