Optimistic and Pessimistic Neurons
By: Ido Ben Artzi
Tags: Dopamine, Optimism, AI, Neuroscience, Reinforcement Learning

Do neurons watch Monty python?

The concept of Machine Learning being influenced by Neuroscience is very much familiar. However, in a recent experiment published in Nature by google’s AI research department DeepMind, the process went the other way around. DeepMind’s researchers saw that its Reinforcement Learning algorithms (more on that in a minute) tend to use “optimistic and pessimistic” methods to efficiently solve problems. Thus, they were curious to look for those kinds of procedures also in the brain.

Before we delve more deeply into the experiment, we need to have a more basic understanding of what is Reinforcement Learning.

The chicken is reinforced to pick pink by the food it is given.

Other than chickens acquiring behavior, we humans also use reinforcement learning to obtain knowledge about the world. Consider the following example from the breakfast I have been eating for the last two years. When I make my eggs, I shall decide at each moment what action would maximize the reward (taste) I’ll be getting. Unfortunately, I am not being rewarded immediately after flipping the eggs but only at the end of the process. As a result, I should have a long-term evaluation knowing to connect an over-cooked egg with a bad taste, and a crispy one with good taste.

I update the values about the process using the reward I get when eating the egg

First, there are some basic concepts we need to get familiar with. In the egg-making process, I am the “agent” but generally, the agent could be any algorithm or organism that tries to learn the environment it is in. Another concept is a “state” which is basically the situation the “agent” is currently facing. In each such “state” the agent should decide on an “action”. Each action leads the agent to a new state, for example in the “state” in the picture above I could decide to flip or not to flip. At the end of the process, the agent will receive a “reward” which is the signal from the environment which it is trying to maximize. After discovering the reward, the “agent” will update its “value function” which refers to the long-term evaluation of every state in the process. Finally, the agent will be able to conduct a “policy” which is a mapping of what action is to be made in each state (Sutton&Burto, 1998).

When the state is cooked — the best action is to serve.

After having all this sorted out we need to learn about “prediction error”. We use our previously described value function to create predictions about the world. For example, I might predict using my value function that the egg on the picture would be tasty 4.3/5. However, it could be that when I taste it and actually get the reward, I would be surprised for the better or the worst. This discrepancy between our prediction and the actual reward is the prediction error. We will use the prediction error to update our value function, and thus our knowledge about the world (Schultz, 2016).

We use the prediction and update it only if the reward is not equal to our prediction

The prediction error is integrated into our value function using the learning rate (alpha). If we take my egg-making example, then on each day when I taste my egg, I update my value function to be a factor of this day’s reward and all of my past experiences. The higher my learning rate will be, the more weight I will give to this day’s reward on account of my former experiences (Hertz & Sahar, RL workshop 2020).

My predictions for Thursday is (1-alpha) times my prediction on Wednesday+ alpha times the actual reward from Wednesday

Dopamine neurons are the ones calculating the above-mentioned prediction error. The firing rate (a measure of neurons activation) of dopamine neurons changes according to the prediction error. There are 3 axioms about the relationship between prediction error and dopamine firing rate:

Winning 5$ will cause more firing than losing 5$. If you are 100% certain of an outcome — no change in firing shall occur When winning, certainty will decrease firing, when losing, certainty will increase firing.

Those points help us understand that dopamine neurons change their activation rates according to errors in prediction, and not just according to value (Caplin & Dean, 2008).

So now the introduction is finally over, and we are ready to see what is deepmind’s new hypothesis. The classical learning process described until now (called temporal-difference learning) assumes that all dopamine neurons have the same learning rate towards positive and negative prediction errors. However, the new hypothesis called “Distributional temporal-difference learning” assumes that some neurons learn faster from either positive or negative surprises. When the prediction error (delta) is positive, we will have a specific learning rate for positive events (alpha_plus) and vice versa.

We learn differently for positive and negative events, differentiating neurons according to their learning rates.

So what does it matter that neurons might have separate learning rates for positive and negative events? According to the model, this separation would cause neurons to learn different expected values. The “optimistic” neurons will react much more to positive surprises and thus learn from them. The “pessimistic” neurons will react much more towards negative events and thus learn from those.

The higher your alpha_plus comparing to your alpha_minus — the higher value you learn.

The difference between optimistic and pessimistic neurons is realized by their reversal points. The reversal point is the reward magnitude the neurons are expecting. A higher reward than the reversal point will cause an increase in firing rate, and a lower reward than that will cause a decrease in firing rate.

The classical neuron is purple in the middle

After assembling their hypothesis, the researchers conducted a physiological experiment to test whether the neurons actually react the way they thought they will. In the experiment, the change in the firing of neurons was recorded while mice were given 7 different amounts of sugar-water.

The mice could be getting tasty sugar doses ranging from 0.1 to 20 microliter.

According to the model’s predictions, some optimistic neurons will expect a high reward so they will decrease their firing rate for most of the sugar doses. On the other hand, some pessimistic neurons will expect a low reward and thus will increase their firing rates to most of the rewards. Conversely, according to classical TD learning, all neurons should react more or less the same.

The computational models of the two hypotheses, in which each line represents a neuron. On the left, all neurons are roughly the same. On the right, Optimistic neurons are on the top lines and Pessimistic ones on the bottom. RPE=Reward Prediction Error

The neural data from the electrodes in the mice brain was in line with DeepMind’s hypothesis. The data looks as if the neurons' population is divided according to their expected values.

Neural data confirming the distributional learning hypothesis. Optimistic neurons on the top (decrease firing rate for almost every amount of sugar). Pessimistic neurons on the bottom (increase firing rate for almost every amount of sugar).

So what can we learn from this experiment? I think that first, it raises questions regarding the importance of being realistic. Being overly optimistic may actually cause us to expect impossible rewards and have quite a frustrating life as a result. On a similar note, if this learning mechanism is indeed common in the brain, then why is it evolutionary adaptive? It could be that having mini-agents that are over-optimistic and vice versa could help our decision making as individuals and as teams. Furthermore, it may be interesting to study the connection of those neural differences to individual differences in life perspective.