SciPost Physics Lecture Notes

Submission

Normative approaches to neural coding and behavior
Ann M. Hermundstad1⋆
1 Janelia Research Campus, Howard Hughes Medical Institute⋆ hermundstada@janelia.hhmi.org

Abstract
These are a brief set of lectures notes for lectures given at the Les Houches summer
course in Theoretical Biological Physics in July 2023. In these notes, I provide an introduction to some of the theoretical frameworks that are used to understand how the brain
makes sense of incoming signals from the environment to ultimately guide effective behavior. I then discuss how we can apply these frameworks to understand the structureand function of real brains.
Copyright attribution to authors.
This work is a submission to SciPost Physics Lecture Notes.
License information to appear upon publication.
Publication information to appear upon publication.

Received Date
Accepted Date
Published Date

1

2

Contents
1 Introduction
1.1 The normative approach22

2 Frameworks that bridge systems
2.1 Sensory coding
2.2 Inference
2.3 Action selection
2.4 Summary

4
4
7
12
16

20

3 Closing the loop in a single system
3.1 Using sensory stimuli to build accurate internal representations
3.1.1 Maintaining persistent internal representations in the absence of input
3.1.2 Accurately updating internal representations via internal input
3.1.3 Reliably tethering internal representations to the external world
3.2 Using internal representations to guide effective behavior
3.2.1 Inferring a generative model of behavior
3.2.2 Learning the parameters of a generative model based on experience
3.2.3 Building a structured behavioral policy
3.3 Coupling sensory representations and behavior in closed loop
3.4 Summary

17
17
18
23
24
26
27
29
31
33
35

21

4 Outlook35

22

A Decomposing mutual information

37

23

B Maximum entropy distributions

37

3
4

5
6
7
8
9

10
11
12
13
14
15
16
17
18
19

1

SciPost Physics Lecture Notes

Submission

24

C Bayesian ideal observer model in a switching environment

38

25

D Primer on reinforcement learning

39

26

E Derivation of policy gradient algorithm

42

27

F

44

28

References

Structuring a motor drive about a goal heading

45

29
30

31

1

Introduction

56

The brain is responsible for a myriad of different computations that enable animals to behave
effectively in an ever-changing world. The brain must sense signals in the environment, makeinferences and predictions about the causes and consequences of those signals, and must planand execute appropriate behavior based on those inferences and predictions.
One way that we try to test our understanding of such computations is by building artificial systems that can mimic the computations and behaviors of real animals. To date, our best
artificial systems pale in comparison to the capabilities of real brains (but large language models provide an interesting counterpoint); real brains continue to outperform our best artificial
systems. And yet, real brains have many limitations that artificial systems do not—limitationsin speed and precision, and in energy use and storage capacity, for example. In order to overcome these limitations, it is thought that brains have evolved smart strategies to exploit thelawfulness of the world in order to achieve “good enough” performance. This can happen
over many difference timescales: over the course of evolution, development, learning, andadaptation. This idea necessarily implies that there are better and worse solutions for exploiting the lawfulness of the world to secure a performance advantage. One goal of theoreticalneuroscience is to understand the space of such solutions—how many there might be, theiradvantages and disadvantages, and, ultimately, why biology might have persisted with one
solution over another. This involves understanding the possible functions of, and constraints
faced by, different biological systems.
In the following notes, I broadly organize these computations beginning with sensory input
and moving toward motor output. In between, the brain has to transform sensory input into
appropriate inferences, decisions, and plans of action. In the first section, I begin by discussing
some theoretical frameworks that are used to understand computation across animals andbrain regions. In the second section, I discuss how we can use these frameworks to try to close
the loop in a single system.

57

1.1

32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

58
59
60
61
62
63
64
65

The normative approach

Before diving in, it’s worth briefly discussing the philosophical and methodological approachthat we will take throughout these discussions. Much of theoretical and computational neuroscience concerns itself with how the brain works: if I can give you a mathematical modelthat accurately reproduces an experimental finding and makes testable predictions for new
experiments, I have understood something about how a certain biological process might unfold. A smaller swath of theoretical neuroscience concerns itself with understanding not just
how a process might unfold, by why it unfolds in this particular way. Why, of all of the possible
strategies that the brain could have adopted to solve a particular problem, is this the strategy
2

SciPost Physics Lecture Notes

66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115

Submission

it adopted? There are many possible answers to this question, some of which could be ‘this isjust what evolution stumbled upon’. But in many cases, the putative answer to this question
helps us understand how a given strategy might be advantageous with respect to performinga particular function, in a particular setting, and subject to a particular set of biological or
physical constraints.
To give a more concrete example: consider your own visual perception of the world. This
perception is something that can be measured and tested by asking you to discriminate different visual stimuli. By designing those stimuli in a way that isolates different statisticalproperties of the world, it is possible to examine which properties of the visual world are easy
for you to see, and which are more difficult. This is the purview of the field of psychophysics.
For example, if you are presented with visual patterns hidden in a noisy background, you
could be asked to locate the pattern in the noise. The more accurately you can do this, thebetter you are able to discriminate these patterns from random noise. It is possible to design
these visual patterns with certain statistical structure by enforcing certain multi-point correlations in light intensity between nearby pixels [1, 2]. Now, if we pick a particular order ofcorrelation—say, 4th order—it turns out that people can easily discriminate some 4th orderpatterns, but struggle to discriminate others, even though they have the same amount of statistical structure [3]. This is an empirical observation, and we could use this to build a modelthat can reproduce this finding. But why it is that we should be better able to see some patterns
over others? Why those particular patterns?
The answer is that the patterns that we can easily discriminate are those patterns that are
most informative about our natural visual world [3, 4]. Were our visual world organized insome different way, this would imply that we would be good at seeing different sets of patterns.
Just by knowing the statistical structure of the visual world, we can predict how sensitivea person will be to different types of visual patterns in the world, and we can do this in aparameter-free manner. This same idea—that we should be most sensitive to sensory signalsthat are most informative about our surroundings—has been used to explain a wide range ofneural and behavioral responses across different species and different sensory systems.
Implicit in this statement, however, is the fact that our sensory systems are constrained:
we have to prioritize some sensory signals, some visual patterns, over others. If we had infinite
resources, we would not need to prioritize—we would in principle be able to fully discriminate
any visual pattern from any other. So a more precise version of the previous statement isthat our sensory systems should use minimal resources to maximize the information that they
convey about the natural world in which we operate.
This is an example of a ‘normative’ statement: it postulates a goal or a function that asystem is trying to achieve (potentially at the expense of other goals). The mere fact that wecan propose and test normative statements reflects that fact that the world around us obeys
predictable laws that can be exploited by biological systems, and that they systems themselves
are limited in their ability to exploit these laws. If biological systems had infinite memory of all
past states of the world and could perfectly predict and access all future states, there would be
no need to prioritize some goals over others. Similarly, if the world were completely random
and unstructured, there would be nothing to prioritize. In this way, normative theories relate
the lawfulness of the world to the performance advantage that can be achieved by exploitingthat lawfulness. In formulating a normative theory, we postulate (i) a function to be performed,(ii) a context in which that function will be performed, and (iii) a set of constraints on thesystem that performs the given function in the given context. We then derive the optimal
solution for achieving this particular function in this context and subject to these constraints.
In the example above about our own visual sensitivity to different patterns, (i) the function
was to maximize information about incoming visual signals, (ii) the context was the naturalvisual world, and (iii) the constraints were limitations in bandwidth (i.e., how many signals

3

SciPost Physics Lecture Notes

116
117
118
119

Submission

could be reliably transmitted downstream).
In what follows, we will use this normative approach to consider how sensory systems
might optimally encode and infer features of the external environment, and how downstream
brain regions can in turn guide optimal behavior based on these inferences (Figure 1).

Figure 1: The brain has to use its limited resources to make sense of signals in theenvironment and guide appropriate actions.

120

2

Frameworks that bridge systems

124

In this first section, we will briefly discuss three different normative theoretical frameworks—efficientcoding (Section 2.1), Bayesian inference (Section 2.2), and reinforcement learning (Section2.3)—that have been used to understand the computational processes involved in sensory
coding, inference, and action selection.

125

2.1

121
122
123

126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149

Sensory coding

The world around us is full of different signals—patterns of light, sounds, smells—that we use
to make sense of our surroundings. At the very earliest stages of processing, the brain must
encode these external signals in internal ones. These internal signals are then used to build
our perceptions of the world, and make decisions based on them. It is remarkable that our
entire understanding of the world is created internally and is built from signals that are firsttransduced in our peripheral nervous system.
From the perspective of the brain, external signals in the world are not directly accessible,and can at best be disambiguated from the responses of neurons. This places limitations on theaccuracy with which the brain can “know” the actual state of the environment. To examine theprocess of disambiguation, let’s take a brief detour to consider the following example (takenfrom David MacKay’s book [5]), which might already be familiar to many readers: you are
given 12 balls that look identical, but one ball is heavier or lighter than the rest. You are given
one scale that can compare any two sets of balls. How do you determine which ball is the odd
ball, and whether it is heavier or lighter, in as few uses of the scale as possible? If you haven’t
already solved this problem before, it’s worth sketching out your solution before moving on.
Stated another way, the goal is to design a set of measurements to disambiguate a set of
hypotheses about the world as efficiently as possible (Figure 2). Here, there are 24 hypotheses
in total: there are 12 balls, and each ball could be heavier or lighter, for a total of 24 possibilities. Each measurement corresponds to a usage of the scale, and there are 3 possible outcomes
per measurement: the lefthand side of the scale is heavier, lighter, or equal to the righthandside. Thus, given 2 measurements, it is possible to disambiguate 32 = 9 hypotheses; given 3
measurements, it is possible to disambiguate 33 = 27 hypotheses. From this, we know that it
is possible to disambiguate all 24 of our hypotheses with a total of 3 measurements. The next
step is to design those measurements.
4

SciPost Physics Lecture Notes

150
151
152
153
154
155
156
157
158
159
160
161

162
163

164
165

166
167

Submission

The most efficient way to design a measurement is to maximize the entropy over the set of
possible outcomes for that measurement. Here, by “most efficient”, we mean that the measurement will reduce our uncertainty as much as possible, given the resolution of the measurement
device. This is an example of a normative statement: it posits an optimal solution (maximizing
the entropy over possible measurement outcomes) for a particular problem (disambiguatinga fixed set of hypotheses as quickly as possible) and subject to a particular set of constraints(subject to the resolution of the measurement device).
For example, if we begin with 24 hypotheses and have 3 possible outcomes, we want toassign 24/3 = 8 hypotheses to each outcome. One way to do this would be to divide our 12
of balls into 3 sets: balls 1-4 (set 1), balls 5-8 (set 2), and balls 9-12 (set 3). We can thencompare any two sets (e.g. sets 1 and 2). There are then three possible outcomes that evenly
break apart our hypothesis space:
1. Set 1 is heavier. There are then 8 remaining hypotheses: set 1 has the heavier ball (4hypotheses), or set 2 has the lighter ball (4 hypotheses)
2. Set 1 is lighter. There are again 8 remaining hypotheses: set 1 has the lighter ball, or
set 2 has the heavier ball
3. Both sets are equal. Again, 8 remaining hypotheses: set 3 has the heavier ball, or set 3
has the lighter ballFigure 2: Designing measurements to disambiguate hypotheses (adapted from [5]).
168
169
170
171
172
173
174
175
176

I will leave it up to the reader to carry this logic through to design the remaining two sets ofmeasurements. When applying this measurement technique in succession, we can identify theodd ball, and determine whether it is heavier or lighter than the rest, in the fewest possible
measurements (it’s easy to verify that a different measurement scheme would require morethan 3 measurements on average). This is precisely because we reduce as much as possible
our uncertainty about the unknown input—and gain as much information as possible aboutthat input—with the outcome of each successive measurement. However, had we been given
only a single measurement, we would not have been able to fully disambiguate the identity
and relative weight of the odd ball. For example, if we performed the single measurement
5

SciPost Physics Lecture Notes

177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204

Submission

shown in Figure 2 and observed that the left side of the scale was heavier, we would at best
be able to say that one of balls 1-4 was heavier or one of balls 5-8 was lighter. This highlightsthat our ability to disambiguate is limited by the resolution of our measurement device.
We can apply a similar perspective to the process of disambiguating signals in the brain.
For example, consider a neuron that acts as a device to measure incoming light signals. Given
some knowledge about the distribution of light signals that could be present in the environment, one can ask how to design the response function of the neuron to best disambiguatethese signals subject to the limitations of the device itself. If the neuron can produce a binary
output (e.g., it can spike or not spike), then this optimal response function should partition theincoming distribution of light signals into two equal probability chunks, and assign one output
to each chunk. This would, for example, lead to a scenario in which the neuron spikes if theincoming light signal is greater than the median of the distribution, and is silent if the incoming
signal is less than the median. By designing a response function that partitions the incoming
distribution into equal probability chunks, we are guaranteed that the entropy of the neural
response will be maximized given the distribution of input signals. This is often referred to as‘histogram equalization’, because the histogram of neural responses will be flat. However, it’s
important to note that there are many ways to partition the distribution into equal probability
chunks, and thus we often have to invoke other constraints (such as continuity of the response
function) before comparing to biology.
This idea forms the basis of one of the most influential normative frameworks in neuroscience: efficient coding. Efficient coding posits that sensory systems maximize the information that they convey to downstream brain regions about incoming sensory signals, and indoing so exploit the statistics of the environment in which an organism must function [6].
This hypothesis was first formulated by Fred Attneave [7] and Horace Barlow [8]. Given an
input message (here, our stimulus S) and an output message (here, our neural response R),
Barlow hypothesized that the goal of the early nervous system is to maximize the informationthat the output message conveys about the input message. The mutual information between
the input and output is given by:I(R; S) = H(R) − H(R|S)

205
206
207
208
209

where H(R) is the entropy of the response, and H(R|S) is the conditional entropy of the response given the stimulus (see Appendix A for derivation). In the limit of low input noise(i.e., the stimulus is reliable), H(R|S) is close to zero, and maximizing mutual information isequivalent to maximizing the entropy of the neural response. Barlow used this idea to define
a quantity called the ‘redundancy’, R, to be minimized:

R = (1 − H(R))/C
210
211
212
213
214
215
216
217
218
219
220
221
222

(1)

(2)

where C is the capacity of the channel along which the messages are sent. It’s worth notingthat entropy maximization, or information maximization more broadly, does not concern itself
with how this information will be used downstream. In fact, Attneave discussed this as an oneof the advantages of the efficient coding framework—that it could be used to understand thefirst stages of sensory processing, without having to know the putative relevance of different
sensory signals [7].
More broadly, this idea can be used to derive the entropy-maximizing set of neural responses subject to different constraints on the response distribution (Appendix B). For example, given a constraint on the total number N of discriminable responses that a neuron can
produce, the entropy-maximizing response distribution is flat (i.e., P(r ) = 1/N; this is our histogram equalization). Given a constraint on the mean firing rate µ of the neuron, the response
distribution is exponential: P(r ) = exp(−r /µ)/µ. And given a constraint on the varianceinpfiring rate σ 2 , the response distribution is Gaussian: P(r ) = exp(−(r − µ)2 /2σ 2 )/ 2πσ 2 .
6

SciPost Physics Lecture Notes

Submission

Figure 3: Maximizing entropy about the distribution of incoming stimuli (adaptedfrom [9]).

242Barlow’s hypothesis was first tested experimentally in 1981 by Simon Laughlin [9], whoshowed that he could predict the responses of motion-sensitive neurons in the visual system of
the blowfly using only the distribution of light signals found in the natural environment (Figure3). Since then, there have been numerous studies that have sought to understand different
aspects of sensory processing in terms of maximizing information about the distribution of
inputs that a sensory system encounters in its environment, or in terms of minimizing error
in the reconstruction of those inputs [6]. The optimal sensory coding scheme depends on thespecific statistics of the environment, the resource limitations and sources of noise within theencoding scheme, and the specific objective function used to optimize the encoding scheme.
As a result, there is no single characterization of an efficient code. For example, in the limit of
low input noise, when incoming sensory signals are reliable, optimal coding strategies tend todecorrelate these signals (consistent with Barlow’s redundancy reduction hypothesis). In thelimit of high input noise, optimal coding strategies tend to average incoming signals in order
to combat the adversarial effects of noise. These different coding regimes emerge from thesame underlying framework but under different assumptions, and are predicted at the level ofsingle cells [10–13], population codes [14, 15], and behavior [4].
A key assumption underlying all of this work is that sensory systems have evolved to exploit the particular statistics of the environment in which an organism must function. If thatenvironment were to change, we would expect sensory systems to change as well—a pointthat we will examine in the next section.

243

2.2

244

This section covers material from Młynarski & Hermundstad (2018) and (2021).

245

In the previous section, we discussed how sensory systems could have evolved to exploit the

223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241

Inference

7

SciPost Physics Lecture Notes

246
247
248

Submission

statistics of the environment. This implies that if the environment were to change, sensory
systems should change with it (Figure 4). In this way, efficient coding has provided a normative
perspective on sensory adaptation [16, 17].

Figure 4: If the statistics of incoming sensory signals change over time, neural responses should adapt to the change.
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276

However, the statistics that are most relevant for adapting to a change in the environment
are not necessarily the same as those that should be optimally encoded in steady state [18].
Moreover, the different sensory signals carry different relevance for downstream computations,which can necessitate different coding strategies [19]. In this section, we treat both of thesequestions.
In the previous section, we considered how to best design a neural response function inorder to efficiently encode signals s drawn from a distribution P(s ). We will now assume
that this distribution can be parameterized by a latent variable θ , such that we can specify
P(s |θ ) (we will also refer to θ as the “state” of the environment). For example, for a Gaussianstimulus distribution, θ can paramterize either the mean or the variance of the distribution. We
can then compare a neural response function that maximizes the information about incoming
sensory stimuli (as discussed in the previous section) with a response function that maximizes
information about the underlying latent parameters of the distribution. In other words, when
designing the optimal response function, we can concern ourselves with disambiguating thecurrent sensory stimulus s or the underlying latent state θ .
The parameter θ is not observable, and thus its value must be inferred from incoming
sensory signals. Such a computations falls within the purview of Bayesian inference, which
specifies the statistically-optimal computation for inferring a latent variable from a set of measurements. We will consider a simple case described above, in which θ specifies either themean or the variance of a Gaussian distribution. We will also assume that θ takes on one of
two values—a high value or a low value. Finally, we will assume that the environment candynamically switch between these two different values with a small but fixed probability hper timestep (Figure 5). At each timestep t , a signal s t ∼ p(·|θ t ) is sampled from a distribution parameterized by θ t . This setup mimics one that is widely used in sensory neuroscienceto study the dynamics of adaptation to changes in the mean and variance of sensory stimuli [16,17,20–23]. The goal of the inference, then, is to use the history of past sensory signalssτ≤t determine whether θ t is high (θ t = θ H ) or low (θ t = θ L ) at a particular timestep.
To illustrate the inference process, we will derive an “ideal observer” that has certain knowl8

SciPost Physics Lecture Notes

Submission

Figure 5: Example nonstationary environment in which the mean (or variance) of a
Gaussian stimulus distribution switches between a low and high value over time.

277
278
279
280
281
282
283
284
285
286
287

edge about the environment, and can use this knowledge to optimally perform the inference.
In this setting, we will assume that the observer knows (1) that the environment exists in oneof two states (high or low); (2) the identity of the two states (i.e., the values θ H and θ L ); (3)the switching probability h; and (4) the values of all other parameters of the stimulus distribution (i.e., if θ parameterizes the mean of the Gaussian distribution, we assume that the ideal
observer knows the variance of that distribution; if θ parameterizes the variance, we assume
that the ideal observer knows the mean). In other words, the observer is armed with a great
deal of prior knowledge; the only thing that the observer does not know is the current valueof θ t , and the specific signal that will be sampled from p(s |θ t ). To infer θ t , the ideal observer
can use the history of observed signals sτ≤t to construct the posterior distribution P(θ t |sτ≤t )(also called the “posterior belief”, or “belief distribution”):
P(θ t |sτ≤t ) =

288

1
Ω

P(s t |θ t )

290
291
292
293
294

where

¨

296
297
298
299
300
301

(3)

(1 − h)θ t = θ t −1h

θ t ̸= θ t −1

(4)

specifies the probability that the environment switched states at time t , P(s t |θ t ) is the stimulus distribution, and Ω is a normalization constant (see Appendix C for a derivation of this
distribution). For an environment that consists of only two states (θ L and θ H ), the posterior distribution is defined over two values, and can thus be specified by a single number
P tL ≡ P(θ t = θ L |sτ≤t ) that specifies the probability that the environment is in the low state(and we can use this to compute P tH = (1 − P tL )). We can use this to rewrite the posterior as:
P tL =

295

P(θ t |θ t −1 )P(θ t −1 |sτ≤t −1 ),θ t −1

P(θ t |θ t −1 ) =
289

X

1



P(s t |θ t = θ L ) (1 − h)P tL−1 + h(1 − P tL−1 )(5)ΩFrom this expression, one can see that the posterior belief that the environment was in the low
state at time t depends on the prior belief at time t − 1, weighted by the probability that theenvironment stayed in the low state, and by the likelihood of sampling the observed signal s t
in the low state.
We can use this belief distribution to construct a point estimate θ̂ t of the current stateθ t , for example by computing the mean of the posterior (note that this is the optimal point
estimate that minimizes the mean-squared error between θ t and θ̂ t [24]):θ̂ t = 〈P(θ t |sτ≤t )〉θt
= θ L P tL + θ H (1 − P tL )9(6)

SciPost Physics Lecture Notes

302
303
304
305
306
307

Submission

Standard Bayesian inference assumes that the sensory signal s t is directly used to update
the posterior belief P(θ t |sτ≤t ). However, as discussed in the previous section, any incoming
sensory signals must be encoded in neural responses before they are used to perform any
downstream computations. Because neurons have finite precision and bandwidth constraints,
there is necessarily loss in this encoding step. As a result, the choice of encoding schemes will
impact any downstream inferences (Figure 6).

Figure 6: Different architectures of efficient coding and Bayesian inference.
308
309
310

To mitigate the negative impact of the encoding, we can design an encoding code schemethat preserves information about the incoming stimuli that is relevant for updating the posterior belief. For example, consider an encoder with a response function of the form:r t (s t ; k, s0 ) =

311
312
313
314
315
316
317
318
319
320
321
322
323
324
325

1
1 + exp (−k(s − s0 ))(7)

where the parameters k and s0 respectively control the slope and offset of this sigmoidal
function, and η is additive noise (alternatively, we can discretize the output r t into a set of
N discriminable response levels, analogous to the encoding scheme discussed in Section 2.1).
This response function has finite resolution that it can devote to incoming signals; because of
the saturating nature of this response function, signals that are sufficiently large or small will
not be distinguishable from one another. The output of this response function, r t , can then be
decoded to construct an estimate ŝ t of the incoming sensory signal. For the purposes of this
discussion, we will assume that we can construct and optimize a simple linear decoder to get
the estimate ŝ t . The Bayesian observer must then construct a posterior belief P(θ t |ŝτ≤t ) built
from the history of past signal estimates, rather than directly using the history of true signal
values.
One advantage of choosing a parameterized encoder is that we can directly optimize theparameters k and s0 of the encoder (as well as the parameters of the linear decoder). For
the sake of illustration, we will consider two different objective functions for optimizing theseparameters:(8)
argmink,s0 (ŝ − s )2 P(s |θ̂ )
t

326

+η

and
argmink,s0θ̂ (s ) − θ̂ (ŝ )
10

2

t

P(s t |θ̂ t ).

(9)

SciPost Physics Lecture Notes

327
328
329
330
331
332
333
334
335
336
337
338
339
340
341

Submission

The first of these objective functions prioritizes an accurate estimate of incoming sensory signals by minimizing the “reconstruction error”; the second of these objectives prioritizes anaccurate estimate of the latent state of the distribution from which the signals were generated
by minimizing the “inference error”. In contrast to typical formulations of efficient coding,which optimize the encoder based on the true distribution of incoming stimuli P(s t |θ t ), bothof these objective functions optimize the encoder based on the current belief about the incoming stimulus distribution, P(s t |θ̂ t ) [18, 19]. As a result, any inaccuracies in this belief will
shape the current encoding scheme, and any inaccuracies in encoding will in turn shape theevolution of the posterior belief. This can be viewed as a form of adaptive coding, where theoutput of the encoder is used to update the posterior belief, and the posterior belief is used,in turn, to adapt the encoding on the next timestep. Moreover, rather than using the current
estimate θ̂ t to update the encoder, a better strategy is to use a prediction θ⃗t +1 about the stateof the environment at the next timestep, and optimize the encoder based on the distribution
P(s t |θ⃗t +1 ). The point prediction θ⃗t +1 can be obtained analogously to θ̂ t , using the predicted
posterior:
X
P(θ t +1 |sτ≤t ) =P(θ t +1 |θ t )P(θ t |sτ≤t ) .
(10)θ t −1

342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373

As should be expected, the two objective functions in Eqs. 8-9 lead to different optimal parameters depending on the current prediction θ⃗t +1 . Since this prediction is varying in time, theoptimal encoding parameters will also vary in time (Figure 7). This can easily be seen throughnumerical optimization. For example, the first objective function leads to an encoder thattracks the current prediction of the incoming stimulus distribution: the optimal offset tracks
the predicted mean of the stimulus distribution, s0∗ = µ⃗ t +1 , and the optimal slope tracks theinverse of the predicted standard deviation of the stimulus distribution, k ∗ = 1/σ⃗ t +1 . As aresult, these parameters vary continuously with the current posterior belief (blue curves in
Figure 7). In the limit that the observer is certain about the current state of the environment
and maintains a correct estimate of that state, the encoder is aligned with the true incoming
stimulus distribution, and prioritizes accurate encoding of stimuli from that distribution, analogous to classic efficient coding. However, when the state of the environment changes, thecurrent encoder is misaligned with the new distribution, and can thus be slow to detect andadapt to a change. As a result, when averaged over time, this strategy often suffers from higher
reconstruction error than a strategy that is optimized for inference. This highlights the factthat an encoding that performs optimal reconstruction of sensory signals does not necessarily
support optimal adaptation to changes in the distribution of those signals.
In contrast, the second objective function is optimized for inference and leads to an encoder whose parameters vary discontinuously with the current prediction of the incoming
stimulus distribution (green curves in Figure 7). The resulting encoder performs a form of
uncertainty-dependent change detection: when the observer is certain about the current stateof the environment (and maintains a correct estimate of that state), the encoder shifts its finite resolution away from the true stimulus distribution, in anticipation of changes that mightoccur in the future. When the state of the environment changes and the observer becomes
more uncertain, the encoder takes a form that can best discriminate that change: for mean
estimation, the optimal encoding function is sharp and centered between the two candidate
distributions; for variance estimation, the optimal encoding function is shifted toward the tailof the high variance distribution (if more certain of the high variance state) or centered about
the low variance distribution (if more certain of the low variance state). As a result, this encoder is less accurate in steady state, but more sensitive and faster to adapt to changes, and canachieve lower average reconstruction error than a strategy that is optimized for reconstruction.
As highlighted in the previous discussion, both of these strategies are optimized greedily
11

SciPost Physics Lecture Notes

Submission

Figure 7: Optimal tuning curves (far left schematic) adapt based on uncertainty,
shown for mean adaptation (top row) and variance adaptation (bottom row).
Adapted from [18].

374
375
376
377

based on the observer’s current knowledge of the environment, and exhibit different tradeoffs
in performance. One might therefore wonder whether it is possible to leverage the advantagesof both strategies. To explore this question, one can form composite codes that balance these
two objectives:
argmink,s0 α (ŝ − s )2 P(s |θ̂ ) + (1 − α)ttθ̂ (s ) − θ̂ (ŝ )2P(s t |θ̂ t )

,

(11)

392

where α ∈ [0, 1] is a parameter that balances reconstruction and inference errors. For manydifferent assumptions about the stimulus distributions and their dynamics, the optimal encoder
balances these two sources of errors, with an optimal value of α that is between 0 and 1(see [18] for specific examples). This finding suggests that optimally reconstructing signals
in nonstationary environments requires devoting some bandwidth to detecting changes in theunderlying distribution of those signals.
This also provides a normative perspective on the temporal dynamics of sensory adaptation(Figure 8). Sensory codes that are optimized for reconstruction versus inference show different temporal dynamics that mimic those observed in so-called adapting and sensitizing retinal
ganglion cells in the salamander, mouse, and primate [22, 25, 26]. Moreover, this framework
predicts that slower environmental dynamics (smaller values of h) will lead to slower adaptation, because the ideal observer requires more stimulus samples to be convinced of a change.
As a result, the predicted timescale of adaptation scales with the periodicity of changes in theenvironment, consistent with experimental observations in the fly, mouse, rat, and electric
fish [17, 21, 27, 28].

393

2.3

394

This section covers material from Ma & Hermundstad (2022)

378
379
380
381
382
383
384
385
386
387
388
389
390
391

Action selection

12

SciPost Physics Lecture Notes

Submission

Figure 8: Optimal neural dynamics exhibit qualitative features of “adapting” (left)
and “sensitizing” (right) retinal ganglion cells in response to switches in stimulus
variance (adapted from [18]).

395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422

In the previous sections, we saw how compressed sensory signals (conveyed through a resourceconstrained encoder) could impact the inference of latent environmental states, and how thisinference, in turn, could be used to dynamically adapt the encoder over time. We formulated
this problem in a passive setting, where we did not consider how the inference process impacted the downstream selection of actions. We will now consider an active setting, whereinferences guide the selection of actions, which in turn impacts which sensory signals will begathered in the future.
We will again consider a scenario in which the environment can take on one of two different
states, θ t ∈ {θ+ , θ− }, that can change with a small but fixed probability h per timestep, as given
in Eq. 4 (note that we have changed our notation from R, L to +, − for convenience). However,rather than using this state to specify the parameters of a stimulus distribution, we will take thisstate to specify the probability of rewards at two different ports that can be sampled by an agent(Figure 9). This is an example of a “nonstationary two-armed bandit” task (more generally
referred to as a dynamic foraging task) in which an agent is faced with two different levers, or“bandits”, that deliver rewards with different probabilities. In a nonstationary setting, theseprobabilities change over time, and thus the agent is best served by using its past observations
to infer which of the two levers is more rewarding at a particular time. In the simplest versionof this task, the observations o ∈ {o+ , o− } are binary: with each lever pull, the agent either
receives a reward (o = o+ ) or receives nothing (o = o− ). This type of task has been used
to study decision making in many different species, including humans [29, 30], non-humanprimates [31, 32], rodents [30, 33–36], and flies [37].
At each timestep, the agent has the option of selecting one of two actions, a ∈ {a+ , a− },that correspond to sampling one of the two levers. When the environment is in state θ+ , the ‘+’lever is more rewarding, and thus the agent should select action a+ ; when the environmentis in state θ− , the ‘−’ lever is more rewarding, and the agent should select action a− . We
assume that the more rewarding port delivers reward with probability phigh , and that the lessrewarding port delivers reward with probability plow . The reward probability can then be
written as:p(o | θ , a) =423

1 + o (θ a∆p + ∆p)2(12)where ∆p = phigh − plow , p = (phigh + plow )/2, and ∆p̄ = 2p̄ − 1, and where we make use of13

SciPost Physics Lecture Notes

Submission

Figure 9: In a example of a nonstationary two-armed bandit task, an agent can sample one of two reward ports (‘+’ and ‘−’) whose reward probabilities are determinedby a latent state θ that can switch over time.
424

the following:a ∈ {a− , a+ } = {−1, 1}o ∈ {o− , o+ } = {−1, 1}(13)θ ∈ {θ− , θ+ } = {−1, 1}
425
426
427
428
429
430
431
432
433
434
435
436

This task is an example of a partially observable Markov decision process, or POMDP. The
optimal actions in this task can be derived via two separable steps: (1) using Bayesian inference
to derive the ideal observer that optimally infers a belief u about the current environmental
state, and (2) using reinforcement learning to derive the optimal behavioral policy π∗ (a|u),which specifies the optimal actions given the current belief [38–40] (see Appendix D for a brief
primer on reinforcement learning).
As in the previous section, we can derive the ideal observer model that optimally infers
the current environmental state based on its prior knowledge. We assume that the observer
knows (1) that the environment exists in one of two states (+ or −); (2) the values of phigh andplow ; and (3) the switching probability h. The goal of the inference is then to use the historyof past observations oτ≤t and actions aτ≤t to determine the current state θ t . Following thederivations in Appendix C, we can write:p(θ t | aτ≤t , oτ≤t ) =
=
=

437
438

1
Ω
1
Ω

, 
)p(θ t |ap(o t |θ t , a t , 
aτ<t
oτ<t
t , aτ<t , oτ<t )Xp(o t |θ t , a t )p(θ t |θ t −1 )p(θ t −1 |aτ<t , oτ<t )θ t −1

1X
Ωθ

t −1

p(θ |θ ) p(o |θ , a ) p(θ |a −1 , oτ≤t −1 )| t{z t −1} | t {zt t } | t −1 τ≤t{z}

world dynamics reward delivery

prior at time t −1

To simplify the final expression of the belief, we define u t to be the difference in belief values
between the two states:u t ≡ P(θ t = θ+ |oτ≤t , aτ≤t ) − P(θ t = θ− |oτ≤t , aτ≤t ).

439
440

(14)

(15)

Using this, together with the results from Eq. 12 for p(o t |θ t , a t , o t ) and Eq. 4 for p(θ t |θ t −1 ),we can write the belief as:
u t = (1 − 2h) ·a t o t ∆p + (1 + o t ∆p) u t −1
a t o t ∆p u t −1 + (1 + o t ∆p)
14

.

(16)

SciPost Physics Lecture Notes

441
442
443
444
445

Submission

Given the belief u, we can then determine the optimal value function v∗ (u) that specifies how‘good’ it is (in terms of future accumulated rewards) to maintain a particular belief u (seeAppendix D for a discussion of value functions). We do this using a technique called value
iteration [40] that iteratively updates the value function over time based on the outcomes ofdifferent actions (note that we first must discretize u before performing value iteration):Xv t (u) = maxp(u ′ , o | u, a) o + (t − 1)v t −1 (u ′ ) /t ,au ′ ,o(17)|{z}q (u,a)

446
447
448
449
450

Note that this is modified from standard value iteration to include a running average of expected reward [41]; see Eq. D.11 for the standard form. Analogously to the value functionv (u), the action-value function q(u, a) quantifies the value of taking a particular action agiven a particular belief u. These value functions stabilize to their optima relatively quickly;
we can then use them to specify the optimal policy:π∗ (a|u) = argmaxa q∗ (u, a) = sgn u

451

(18)

Note that this corresponds to a purely greedy policy given the current belief:
agreedy ≡ argmax〈r (o | u, a)〉a= argmax p(o = o+ | u, a)a= argmax

1 + a ∆p u + ∆p

a

(19)

2

= sgn u
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476

This illustrates how we can use an optimal belief, derived through Bayesian inference, to determine the optimal actions conditioned on that belief. In contrast to the previous section,
here we assume that the agent can precisely measure observations in the environment (a reasonable assumption if, as in this case, the observations are binary). However, this also assumes
that agent can precisely track its current belief. Just as we explored how the compression ofincoming sensory signals impacts inference, so too can we consider how the compression of
an internal belief can impact the action selection process. However, as soon as we attempt
to compress the belief u, there is no guarantee that the optimal (compressed) policy can bederived via the two separable steps of deriving the ideal observer model and deriving the optimal policy conditioned on the observer model. In fact, there are many compressed policies
that achieve near optimal performance [41] (see [42] for a general treatment of the tradeoff
between optimal performance and computational complexity).
To see this, consider discretizing the belief update in Eq. 16 under the optimal policy in Eq.
18 (Figure 10). This update can be transformed into a Markov chain that consists of n ∈ Nstates corresponding to each of the discretized belief values. These states can be labeled according to the actions that they specify; all states with u(n) > 0 are labeled +, and all states
will u(n) < 0 are labeled −. There are 2N transitions between states, corresponding to thetwo observations that can be obtained from taking the action specified by each state; thesetransitions can be determined from Eq. 16. In the limit as N → ∞, this Markov chain willapproximate the optimal belief update and action selection. However, in the limit that N is
finite, the Markov chain can deviate significantly from the optimum, depending on its architecture. For N small, we can directly enumerate all possible Markov chains and compare theirperformance. For N states, there are at most 2 N N 2N possible Markov chains (corresponding
to 2 labels for each of N states, times N target states for each of 2N possible transitions). For
N = 5 states, this amounts to a whopping 312, 500, 000 possibilities. Luckily, 99.9% of these15

SciPost Physics Lecture Notes

Submission

Figure 10: Top row: we can create a compact agent by transforming an optimal
Bayesian agent into a discrete Markov chain that specifies actions conditioned on
past outcomes. Bottom row: by varying the elements of the Markov chain, there
are many ways to build compact agents that achieve good performance (adaptedfrom [41]).

483

are repeats that generate identical behavior under the symmetric task that we’ve considered;a total of 263, 428 are unique. This is still a staggering number; again, luckily for us, only afraction of these—on the order of 5, 000—exhibit high performance. Of these, only a small
fraction exhibit behavior that can be interpreted analogously to the belief update of a Bayesian
agent [41]. Thus, by considering compact systems that have limited resources for performing inferences and guiding actions, we can discover a myriad of solutions that achieve good
performance.

484

2.4

477
478
479
480
481
482

485
486
487
488
489
490
491
492
493
494
495
496
497

Summary

In this section, we have introduced some key normative frameworks—including efficient coding (Section 2.1), Bayesian inference (Section 2.2), and reinforcement learning (Section 2.3)—
that have been used to understand optimal sensory coding, inference, and action selection.
These frameworks allow us to understand the nature of computations that generalize acrossspecies, brain regions, and modalities. We have also seen examples of how these frameworks
inform and constrain one another: for example, in Section 2.2, we saw how different sensory
coding schemes impact the process of inference, and how inference, in turn, can be used to
guide adaptive coding schemes; in Section 2.3, we saw how optimal inferences can be used to
select actions in uncertain and changing environments, and how compact representations of
those inferences can yield a diversity of different strategies for guiding effective behavior.
In the next section, we will use these approaches to understand neural dynamics and behavior in a specific system, and we will follow a set of computations from the encoding of
sensory stimuli to the selection of actions (and back).

16

SciPost Physics Lecture Notes

498

3

Submission

Closing the loop in a single system514

We now consider a specific system—visually-guided navigation and learning in the fruit fly—where we can, in principle, follow an entire stream of computations, from the encoding of
sensory stimuli, to their use in building and modifying internal representations of stored variables, to the selection of actions based on these internal representations. Flies have a reasonably rich behavioral repertoire that allows us to study how behavior changes over time inresponse to new experience. Moreover, there is a massive genetic toolkit that enables us tomonitor and manipulate specific cell types during behavior. Finally, with the recent release ofa synaptic-level connectome, it is now possible to relate the physiology of these cell types totheir morphology and connectivity.
For the purposes of this discussion, we will consider a class behaviors in which flies associate rewards and punishments with stimuli that signal different locations in the environment,
and use these associations to modify their behavior over time, for example to navigate to good
locations or away from bad ones. As an agent in such an environment, this type of navigationis enabled by knowing (A) where you are, (B) where you want to go, and (C) how to get from
A to B. In what follows, we will consider how these quantities are represented in the brain andupdated based on experience.

515

3.1

499
500
501
502
503
504
505
506
507
508
509
510
511
512
513

516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544

Using sensory stimuli to build accurate internal representations

This section covers material from Noorman, Hulse, Romani, Jayaraman, & Hermundstad (2022)
and Kim, Hermundstad, Romani, Abbott, & Jayaraman (2019); see also: Fisher, Marquis, D’Alessandro,& Wilson (2019).
We begin by asking how an animal knows where it is—i.e., how the brain keeps track of wherean animal is relative to its surroundings. This is something that many animals do effortlessly.
For example, if I asked you to close your eyes, and with your left index finger, point to the door
that you used to enter the room you’re currently in—you would be able to do this without any
external sensory signals, and without knowing in advance that this is something you’d haveto remember upon entering the room. Our brains use sensory signals to build maps of oursurroundings, and we can use these maps ‘offline’ when we no longer have direct access to
those signals.
For several decades now, we have had a theoretical framework that explains how a network of neurons might build and maintain an internal sense of direction [43–51] (see [52–55]
for recent reviews). The theory of ‘attractor networks’ posits that a population of neurons can
encode and update internal representations of a variable (such as which direction you’re facing) through their recurrent activity. To give some intuition for how this could work, consider
building a neural compass, made from a population of neurons, that keeps track of your current direction. There are a few key requirements that we might have for this compass: (1)we want one single compass needle (no more, no fewer), (2) we want the needle to movewhen we turn, and (3) we want the needle to stop when we stop turning. We can build such a
compass with a population of neurons that respond to different orientations; i.e., when you’refacing north, one group of neurons responds, and when you’re facing east, another group responds. If we arrange these neurons along a ring based on this tuning, then we can figureout how to connect these neurons together to meet all of the requirements listed above. To
build a single compass needle, we can connect nearby neurons on the ring with local excitatory connections...but if all neurons locally excite their neighbors, then any activity in one
position along the ring will spread to fill the entire ring, and we will lose our compass needle. To prevent this from happening, we can have neurons broadly inhibit distant neurons
around the ring; this will keep any activity localized in one region of the ring. With these two
17

SciPost Physics Lecture Notes

545
546

Submission

ingredients—local excitation and broad inhibition—we can create a single compass needle in
the form of a localized bump of neural activity (Figure 11).

Figure 11: A ring attractor network can maintain a localized bump of neural activity,and can shift the bump by integrating self-motion input.

565

To move this bump of activity, we need to invoke additional mechanisms: we can do this byincluding additional inputs to the ring that are tuned to angular velocity, and that effectively‘push’ the bump around the ring when turning right or left. And to keep the bump still whenthose inputs are removed, we need to ensure that the activity pattern that holds the bumpis stable, and can persist at the same orientation along the ring without any external inputs.
Theoretically, this typically requires that we use an infinitely large population of neurons tobuild the compass (see [44, 49, 50, 56] for studies that use large networks to approximate
the precision of infinite networks, and [57–59] for studies that highlight failure modes ofsmall networks). However, biology seems to be able to construct a neural compass from only
a handful of neurons. The fruit fly, for example, maintains an internal sense of direction
in a donut-shaped brain structure called the Ellipsoid Body (EB). A population of “compass
neurons” maintains a persistent bump of activity that encodes the fly’s current heading [60],and that can be updated by integrating the fly’s angular velocity [58, 61] (Figure 12). The
connectivity [62, 63] and dynamics [64, 65] of this network are consistent with theoreticalaccounts of a continuous attractor network, with one notable exception: the fly network is
composed of a very small number of computational units (on the order of 8 units), rather thanthe extremely large network that we’d expect to need. In what follows, we will explore how
it is possible to accurately build, maintain, and update an internal sense of direction in such asmall network.

566

3.1.1

547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564

567
568
569

Maintaining persistent internal representations in the absence of input

To see how we go about building such a network, let’s begin with a population of neurons,
indexed i ∈ [1, ..., N], that have neural activities r1 , ..., r N . Let’s consider the case where wecan describe these activity with a linear system of equations:ṙ = −r + W r + C0

570
571
572

(20)

where W is a connectivity matrix that specifies the strength of excitatory or inhibitory connections between all pairs of neurons, and C0 is a constant that specifies feedforward activity
that is injected into the network. To build some intuition, consider the following connectivity

18

SciPost Physics Lecture Notes

Submission

Figure 12: The fly maintains an internal sense of direction in a neural compass. Left:schematic of the fly brain. Middle: schematic of neurons that innervate a donutshaped brain region called the ellipsoid body (EB). Right: it is possible to image
calcium activity of compass neurons in the EB while the fly is tethered under a microscope and walking on an air-lifted ball. The activity of the compass neurons forms
a localized bump that moves around the donut and tracks the movements of the fly
relative to the world.

573

574
575

576

577
578
579
580
581
582
583

matrix:



λ1 0
W=
,
0 λ2

(21)

where λ2 < 0. This defines a linearly-independent system of equations (adapted from Steven
Strogatz’s book [66]):
 
  
r1
ṙ1
λ1 − 10=+ C0 ,(22)ṙ20
λ2 − 1 r2
whose solutions are:r1 (t ) ∝ A1 + B1 exp (λ1 − 1)t

r2 (t ) ∝ A2 + B2 exp (λ2 − 1)t



(23)

where A1 , A2 , B1 , B2 are constants. The entries of W determine the dynamics of this system. If
λ1 < 1, there is one stable fixed point in the network (also called a ‘point attractor’); the valueof λ1 (and specifically whether it is less than or greater than λ2 ), determines the relative rate
at which r1 and r2 approach the fixed point (Figure 13). Alternatively, if λ1 > 1, there is asingle saddle point; the dynamics are stable along the r2 axis, and unstable along the r1 axis.
In the special case that λ1 = 1, there is a continuum of stable fixed points along the r1 axis
(also called a ‘line attractor’).

Figure 13: Fixed points of the linear system in Eq. 22, with λ2 < 0 (adapted from[66]).
584
585

In the case where the system of equations is not linearly independent, the same logic
applies to the system of equations. The eigenvectors ξ of W specify the axes of flow, and the19

SciPost Physics Lecture Notes

586
587
588
589
590
591
592
593
594
595
596

Submission

sign and magnitude of the eigenvalues λ specify the direction and rate of flow, respectively.
If the system has a single unity eigenvalue and all other eigenvalues less that one, then there
is a continuum of stable fixed points. This continuum can be used to stably encode a single
linear variable via appropriate combinations of r1 and r2 . This idea was used to propose how
a network of as few as two neurons, with activities r1 and r2 , could stably encode a linear
variable like the position of an animal’s eyes [43].
However, this formulation cannot encode a circular variable like orientation. Instead, one
can build a ring of stable fixed points by stitching together multiple line attractors over fixedintervals (Figure 14). For this to work, there must be a precise handoff between line attractors,
which requires a nonlinearity in the network (and some additional fine tuning, which we will
come to). We can consider a simple nonlinear network of the form:ṙ = −r + W φ(r) + C0

597
598
599
600
601
602
603
604
605
606

(24)

where φ(·) is a nonlinear function. For illustrative purposes, we will take φ(·) to be a thresholdlinear function. This form of nonlinearity ensures that only a subset of neurons in the network
is active at any given time; importantly, the dynamics of these active neurons are governedby a set of linear equations via an active submatrix Wact of the full connectivity matrix W. As
before, the eigenvalues and eigenvectors of this active submatrix determine the fixed point
structure of these dynamics. Thus, by appropriately choosing a connectivity matrix W, it ispossible to ensure that all active submatrices of W have a single unity eigenvalue, and allother eigenvalues less than one. Each active subsystem can thus be constructed to encode aline attractor over a finite interval, and these line attractors can then be stitched together toform a more complex attractor structure [65].

Figure 14: Left: a linear system can encode a continuum of marginally-stable fixed
points. Middle, right: in a nonlinear system, a ring attractor can be constructed bystitching together line attractors [65].
607608

To construct a ring attractor, we want a rotationally invariant set of line attractors. Thiscan be achieved by choosing W to be a circulant matrix. A common choice is:
W=

609
610
611

1
N


JI + JE cos(θi − θ j ) ,(25)

where JI controls the broad inhibition in the network, and JE control local excitation between
neurons with preferred headings θi and θ j . With appropriate choices of JE , the active submatrices of W are identical, and each has a single unity eigenvalue. For example, given four20

SciPost Physics Lecture Notes

612
613
614
615

Submission

neurons with activities r1 , r2 , r3 , and r4 , a ring attractor can be constructed by stitching together four line attractors spanned by ordered pairs of units: (1) r1 and r2 , (2) r2 and r3 , (3)r3 and r4 , and (4) r4 and r1 . In this case, the preferred headings θi and θi±1 are all separated
by 90◦ , and the full connectivity matrix is:
1
0 −1 0JI
JE  0
1
0 −1
W=
+
.
−1 0
1
0
N
N
0 −1 0
1


616

617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633

The 2 × 2 active submatrix has a connectivity structure:
1 JI + JE
JIWact =.
JI
JI + JE
N

(26)

(27)

If we diagonalize this active submatrix, the eigenvectors are ξ1 = [−1, 1] and ξ2 = [1, 1],with corresponding eigenvalues λ1 = JE /N and λ2 = 2JI /N + JE /N (note that here, we assume JI < 0, in which case λ1 ≥ λ2 ; more generally, it is possible to derive the maximal values
of JI for which the network will generate a stable bump of activity [65]). With N = 4 neurons,
the leading eigenvalue is 1 if JE = 4. This solution guarantees that the bump of activity canpersist anywhere along a continuum of orientations between θi and θi±1 (and one can showthat this is a solution for all even-sized networks of size N ≥ 4; see Figure 15 and [65]). The
corresponding eigenvector describes a situation in which any increases in the activity of one
neuron are offset by decreases in activity of the other, and the resulting population activity
pattern is marginally stable. This type of formulation can be used to construct a ring attractor
that can encode a continuum of values on a circle with a localized bump of activity. For a network of N = 4, there is one single setting of the local excitation that can generate a continuumof fixed points with line attractors spanning 2 neurons. For a network size N = 6, there are 3
ways to generate this continuum from line attractors spanning either 2, 3 or 4 neurons. Moregenerally, for a network of N neurons, there are N − 3 ways to generate a ring attractor fromline attractors spanning n = [2, ..., N − 2] neurons. Thus, the larger the network, the easier it
is to find a parameter setting that will generate a continuum of fixed points around a ring.

Figure 15: Larger networks have more ways to tune the local excitation in order togenerate a ring attractor (adapted from [65]). Circles and squares denote optimal
solutions for networks with even and odd numbers of neurons, respectively.

21

SciPost Physics Lecture Notes

634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650

Submission

This construction assumes that the local excitation, JE , can be chosen optimally. If JE is notchosen optimally, the full connectivity matrix W will have two different active submatrices thatdiffer in size (Figure 16). The larger of these two active submatrices, with a bump spanned
by Nact = n + 1 active neurons, has a leading eigenvalue greater than one, and thus generates
an unstable fixed point; the smaller of these active submatrices, with a bump spanned by
Nact = n active neurons, has a leading eigenvalue less than one, and generates a single stable
fixed point. The bump of activity will then be pushed away from the unstable fixed point
and pulled toward the stable fixed point. As it does so, the bump transitions between thetwo different linear regimes that maintain the bump of activity with n + 1 versus n active
neurons. The angular span of these two different regimes is closely related to the drift rate
within each regime—i.e., how quickly the bump is pushed from or pulled toward a fixed point.
As a result, the dynamics of the bump are governed by three factors: the orientations of thestable and unstable fixed points, the rate at which the bump is pulled toward or pushed fromthese fixed points, and the angular span of each stable and unstable regime. In the limit that
JE approaches an optimal value, the drift rate in one regime tends to zero as the angular spanof that regime grows to fill the entire ring. Thus, a ring attractor emerges in the limit that thebump drifts infinitely slowly over an increasingly large fraction of the ring.

Figure 16: Left, middle: optimal values of local excitation generate a continuum of
marginally-stable fixed points on a ring. Right: non-optimal values of local excitation
generate a ring of stable and unstable fixed points.
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666

Up until now, we have built intuition about this ring attractor solution by patching together linear dynamical systems. An alternative formulation of this solution can be derived
by computing the “energy landscape” (a Lyapunov function) of the dynamical system (Figure17). This energy landscape describes the states to which the system will evolve in the longtime limit. In the limit of an infinitely large network with connectivity given by Eq. 25, theenergy landscape will be flat as a function of orientation. For a finite network with naivelychosen JE , the energy will be hilly, with discrete basins (corresponding to stable fixed points)
separated by energy barriers (corresponding to unstable fixed points). The optimal values of
JE will flatten this energy landscape. To derive these values, one can first perform a discrete
Fourier transform on the system of nonlinear equations in Eqs. 24-25 to transform a system
of N equations that describe the dynamics of individual neurons into a system of 3 equations
that describe the dynamics of the amplitude a c , width wc , and orientation θc of the compass
bump (because of the choice of connectivity, only the DC mode and first modes survive theFourier transform, after initial transients). This set of equations can then be used to construct
the energy of the system. One can then derive the Hessian matrix, which captures the local
curvature of the landscape. When computed at the orientations of the stable fixed points,

22

SciPost Physics Lecture Notes

667
668
669
670
671
672

Submission

the Hessian separates into a block-diagonal matrix, with a single eigenvector aligned with theorientation θc . The corresponding eigenvalue, ∂ 2 E/∂ θc2 , depends only on JE , N, and Nact .
Thus, for a given network size, it is possible to find the values of JE that will locally flatten theenergy as a function of orientation. One can further show that these same values of JE agreewith those derived from the linear systems perspective, and they guarantee that the energy is
globally flat as a function of orientation (see the SI of [65] for more details).

Figure 17: Non-optimal values of local excitation lead to a hilly energy landscape(left); optimal values flatten this landscape (right).

673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688

3.1.2

Accurately updating internal representations via internal input

In the previous subsection, we showed how it was possible to maintain a persistent bump ofactivity at any of a continuum of orientations along a ring. For this bump of activity to function
as a compass needle, it must be able to be updated based on changes in orientation. This canbe done by injecting self-motion input into the network; the network then has to integrate this
input to appropriately shift the bump of activity to the correct location along the ring (notethat this self-motion input is thought to be computed from a combination of motor and optic
flow inputs [67], which raises an interesting question about whether these inputs are optimallycombined to estimate self-motion; see [68] for a perspective on this issue). A simple way to
do this is to add two additional “side rings” to the network that are gated by clockwise orcounterclockwise angular velocity [69]; these side rings inherit the bump of activity from theprimary, or “center” ring, and project back to the center ring with shifted connections. Thus,
clockwise turns will push the bump in one direction around the ring, and counterclockwise
turns will push the bump in the opposite direction. In the limit that the time constants of theseside rings are very fast, we can approximate their dynamics with a set of velocity-dependent
inputs into the network:ṙ j = −r j +

689

1 X
N

sym

asym

W j k + vin W j kφ (rk ) + C0 ,

sym

1
N


JI + JE cos(θ j − θk ) ,1
asym
W j k = sin(θ j − θk ),
N
691
692
693
694
695
696
697

(28)

where vin is the input velocity, and where
Wjk =

690

j = 1, . . . , N,

k

(29)

consistent with Eq. 25 (note that we typically include an integration time constant τ, which
we have taken to 1). If the excitation is chosen optimally, this network can perfectly integrate
its inputs. However, if the excitation is not optimally chosen, the movement of the bump will
feel the effects of the stable and unstable fixed points in the network: the bump will movefaster than the input velocity as it accelerates away from an unstable fixed point, and thebump will move slower than the input velocity as it decelerates toward a stable fixed point.
These dynamics are again determined by the angular orientations of the unstable and stable
fixed points, the angular span of the linear regimes about each fixed point, and the drift rate at23

SciPost Physics Lecture Notes

Submission

718

which the bump is pushed away from or pulled toward these fixed points. In the limit of small
velocity inputs, the drift rates and angular spans of each regime will remain unchanged, but
the orientations of the fixed points will shift; the stable fixed points will shift in the directionof the input velocity, and the unstable fixed points will shift against the direction of the input
velocity. So long as the stable and unstable fixed points remain within their respective regimes,
the bump will evolve toward and persist at the stable fixed point, and the network will fail tocontinuously integrate its inputs. However, above a particular threshold velocity, the stable
fixed point will move into the unstable regime, and the unstable fixed point will move into thestable regime. As a result, when in the stable regime, the bump will be pulled toward a stable
fixed point, but can never reach it; instead, it will transition into the unstable regime, where
it will be pushed away from an unstable fixed point. This push and pull will cause the bumpto speed up and slow down as it transitions between regimes.
In the view of the energy landscape, the drift rates determine the local curvature of theenergy landscape within each linear regime. As the velocity increases, the fixed points will
move, but the local curvature about those fixed points will remain approximately unchanged.
This leads to a tipping of the energy landscape in the direction of the input velocity. If the input
velocity is very small, the degree of tipping will be small, and the bump will still get stuck at
the stable fixed points in the network. As the input velocity increases, the tipping will be more
severe. Again, above a given threshold velocity, the bump will move continuously down theenergy landscape without getting stuck, but will speed up and slow down as it feels the effectof the curvature of the landscape.

719

3.1.3

698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717

720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746

Reliably tethering internal representations to the external world

The previous subsections highlighted how we could build a network that maintains and updates an internal sense of direction, even in the absence of external sensory cues. However, tofunction in an external environment, this internal representation must be tethered to cues inthat environment. In the fly compass network, these external sensory cues reach the compass
through another set of “ring neurons” [63]. There are many different classes of ring neurons
that bring in information about polarized light [70–72], wind direction [73], and local visualfeatures [74–77]. This again raises an interesting question as to whether and how these different inputs are combined to (optimally?) infer an estimate of orientation, something we will not
discuss in detail here. For the sake of illustration, we will focus one modality that is encodedby visual ring neurons that respond to local spatiotemporal features in a scene [74, 76]. The
response properties of these neurons are well captured by Gabor-like spatial filters combinedwith biphasic temporal filters, a structure that is thought to efficiently exploit the structure of
spatiotemporal correlations in natural scenes (as discussed in Section 2.1; [78, 79]). At thesimplest level, this means that a given ring neuron will be active whenever a particular featurein the visual scene is present at a particular location relative to the fly. The presence of multiple
inputs to the compass—here, external sensory inputs and internal self-motion inputs—raises
the question as to how the compass network can reliably keep these sets of inputs in registerto accurately update a single, self-consistent internal representation of heading.
This is thought to be achieved through plasticity between ring neurons and compass neurons [80–83]. Each ring neuron makes all-to-all synapses onto compass neurons [63], andthese synapses are plastic and can be modified over time based on experience [82–84]. As aresult, plasticity is thought to “map” the external world onto the compass, and several lines ofongoing work are focused on how this map remains self-consistent [82–85]. One set of ideas
stems from Kohonen’s ‘self-organizing map’, which is an unsupervised competitive learningalgorithm that tries to iteratively map a high-dimensional input space into a low-dimensional‘map space’ [86] (Figure 18). Given an input space spanned by input patterns {p} and a map
space spanned by output patterns {q }, this is achieved through plasticity in weights Wi j that24

SciPost Physics Lecture Notes

Submission

Figure 18: Illustration of self-organizing map developing over time (adapted from
Wikipedia, user:Mcld).

747

link the two:
∆Wi j = αθ ( j )(pi − Wi j )

748
749
750
751
752
753
754

For a given input pattern p, this rule updates the weights in the local neighborhood θ ( j )around the most strongly responding output unit q j (also referred to as the ‘best matching
unit’). The parameter α is a learning rate that determines how quickly the weights will update
over time.
If we define the input patterns to be the responses of the ring neurons, and the output
patterns to be the responses of the compass neurons, then the ring attractor enforces a naturalneighborhood function defined by the bump of compass activity. The rule then becomes:∆Wi j = αr jc (rir − Wi j )

755
756
757
758
759
760
761
762
763

765
766
767
768
769
770
771
772
773
774

(31)

where rir and r jc define the activity of the ring and compass neurons, respectively. This rule issimilar to a classic Hebbian learning rule in which neurons that “fire together, wire together”—
in other words, coactivity between neurons i and j will lead to a strengthening of the weightsthat link them.
In the case of the fly’s neural compass, the ring neurons are thought to be inhibitory [87,88], and so the appropriate learning rule is “anti-Hebbian” (i.e., coactivity between neurons iand j will weaken, rather than strengthen, the weights that link them). Moreover, evidence
suggests that the learning rate α scales with the fly’s angular velocity v [82, 84], and thus has
a form similar to:∆Wi j = −v 2 r jc (rir − Wi j )

764

(30)

(32)

This velocity-dependent learning rate ensures that the map is only updated when the fly turns,
and does not update when the fly persists at the same orientation for longs periods of time.
As the fly explores a new visual scene, this plasticity rule will generate a self-consistentmapping of the visual world onto the compass, such that turns that drive the bump of activity
are matched by weak inhibition from ring neurons that respond to the corresponding featuresin the visual scene (Figure 19). For example, consider a visual scene that consists of a single
visual feature, like a vertical bar. And consider that a given ring neuron rir , which responds
to features at an orientation φi , is co-active with a given compass neuron r jc , which responds
when the bump is at an orientation θ j . If the fly turns, the visual scene will change by anorientation ∆φ, such that a ring neuron i + ∆φ is now active. At the same time, self motion
inputs will drive the bump to change orientations by some angle ∆θ , such that a compass
25

SciPost Physics Lecture Notes

775
776
777
778

Submission

neuron j + ∆θ is now active. The plasticity rule above will ensure that ring neuron i + ∆φwill have the weakest weight, and thus the smallest inhibition, onto compass neuron j + ∆θ .
As a result, the compass bump will move in lock step with the movements of the fly and the
movements of the visual scene.

Figure 19: Plasticity creates a self-consistent mapping between visual and self-motion
inputs that drive the compass (adapted from [82]; see also [83]). Left: visual ringneurons have receptive fields that tile visual space. These neurons make all-to-all
inhibitory synapses onto compass neurons that maintain the compass bump of activity. Middle, right: as the fly turns, the compass bump moves around the EB, excitinga different compass neuron (upper row). At the same time, the visual scene moves
relative to the fly, and excites a different ring neuron (middle row). The coactivity
between ring and compass neurons weakens the synapses between them. Over time,
this creates a self-consistent mapping between the visual and self-motion inputs thatdrive the compass.

786

By this same logic, one can easily construct a scenario that might alter this mapping. For
example, consider a visual scene with a two-fold symmetry, such as two identical bars separated by 180◦ . This scene will induce identical responses in two ring neurons that respond toorientations separated by 180◦ . However, the plasticity rule introduced above will eventually
weaken synapses at only one location in the compass network, leading the bump to favor oneof the two orientations that are associated with this symmetry. I will leave it to the reader to
speculate as to how this might impact the dynamics of the compass bump; we will return tothis idea in Section 3.3.

787

3.2

788

This section covers material from Dan, Hulse, Kappagantula, Jayaraman, & Hermundstad (2021).

779
780
781
782
783
784
785

789
790
791
792
793
794
795
796

Using internal representations to guide effective behavior

We can now consider how this internal sense of direction is used to guide the fly’s behavior.
For this, we will start by analyzing the behavior itself, and those changes in behavior thatarise during learning. We will then explore how these changes in behavior could be mediatedby internal representations in the brain, such as the sense of direction that we have been
discussing.
For this, we will consider a variant of a older learning paradigm [89] in which flies navigate
in a one-dimensional virtual environment (Figure 20; physicist readers might find it interesting to know that these experiments were pioneered by Martin Heisenberg, the son Werner
26

SciPost Physics Lecture Notes

797
798
799
800
801
802
803
804
805
806

Submission

Heisenberg, prompting some to ask whether one can formulate an analogous ‘fliesenberg uncertainty principle’ [90]). This environment consists of an LED screen than spans 330◦ anddisplays different visual patterns [91, 92]; this screen surrounds the fly, which is tethered inplace but able to move its wings. The orientation of the patterns on the screen is then coupled
to the fly’s movements; when the fly tries to turn left, the patterns on the screen are rotatedto the right, and vice versa. In this way, the fly has closed-loop control of the movements ofthe patterns on the screen. By tethering the fly, it is then possible to image from the brain
while the fly is navigating. By further pairing orientations of the visual scene with rewards
and punishments, it is possible to study how the fly’s neural activity and behavior change in
response to different types of feedback.

Figure 20: Flies can learn to avoid punishment associated with different visual patterns (adapted from [93]). Left: a visual scene has repeating sets of visual patterns;
one set of patterns is paired with an aversive heat punishment. Right: in a virtual
reality setup, the fly orients relative to the visual scene, and is pinged in the abdomenwith an infrared laser whenever it orients toward one of the punished visual patterns.

807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822

3.2.1

Inferring a generative model of behavior

Flies exhibit different modes of patterned behavior. In flight, they exhibit periods of straightflight, or ‘fixations’, punctuated by abrupt turns, or body ‘saccades’ [91, 93–95] (Figure 21).
We can segment the behavior into these two different modes, measure how the propertiesof these modes vary over time and across flies, and use this to infer a generative model ofbehavior. This generative model can be phrased in terms of a behavioral policy π(a|θ ) thatspecifies the probability of taking an action a given a particular state θ of the environment [40](analogously to our discussion of behavioral policies in Section 2.3).
We begin by assuming that behavior can be structured as a sequence of saccades and fixations, such that the termination of one mode initiates the other mode, and vice versa. In thelanguage of reinforcement learning, each action then comes in two different types: a ∈ {F, S}.
Each mode can be specified by an angular velocity, ω, and a duration conditioned on thatangular velocity, ∆t |ω.
Fixations tend to have near-zero angular velocity, so we can approximate this as a delta
function: P(ω) = δ(ω). The distribution of fixation durations—whether measured acrosstime or flies—is well fit by an inverse Gaussian distribution:P(∆t ; µ, λ) =

823
824

v
t

λ



2π∆t 3exp −λ(∆t − µ)22µ2 ∆x(33)This distribution is appealing because we can specify a process that generates it. Consider thefollowing stochastic process:X t ∼ νt + σW t
27

(34)

SciPost Physics Lecture Notes

Submission

Figure 21: Left: flies fixate (F) and saccade (S). These flight patterns vary in angularvelocity ω and duration ∆t . Right: we can use the statistics of these flight patterns
to infer a generative model of behavior (adapted from [93]).

825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853

where W t is standard Brownian motion, and ν is a drift term (this process is often called a‘drift diffusion’ process). The first passage time of this process, given a fixed threshold ρ > 0,follows an inverse Gaussian distribution, with µ = ρ/ν and λ = (ρ/σ)2 [96–98]. Thus, we
can capture a single fixational event by sampling from this process; the fixation terminates
when the process crosses a threshold ρ.
This process requires specifying three parameters: the threshold ρ, the drift rate ν, andthe diffusion spread σ. In principle, any or all of these parameters could vary over time basedon experience, and the variation in each of these parameters would manifest in predictablechanges in the mean and variance of the inverse Gaussian distribution. To see this, note thatthe mean mean = µ and variance var = µ3 /λ are related by: log(var) = 3 log(mean) − λ. If,
for example, the variability in behavior is explained by changes in the drift rate ν alone (withfixed threshold ρ and spread σ), we would expect to see the relationship between the varianceand mean of fixation durations to follow a line with a slope of 3 and offset of −λ = −ρ 2 /σ 2 . By
fitting this distribution across time and across flies (or, alternatively, by measuring the mean
and variance empirically), we can show that the behavior is indeed most consistent with amodel in which the drift rate is changing over time. Thus, we can model fixations as eventswith near-zero angular velocity, and with durations generated by a drift diffusion process withan adaptable drift rate.
Saccades follow a similar structure, but on a different time scale. In contrast to fixations,these events are thought to be ballistic, such that they cannot be interrupted or modified during their execution. Using the same types of analyses, we can show that the angular velocity ofsaccades is well fit with two log-normal distributions (corresponding to clockwise and counterclockwise saccades), but with an adaptable bias that captures the tendency to initiate saccades
in one direction versus another. We find that the duration of saccades, analogously to fixations,also follows an inverse Gaussian distribution. However, rather than having a flexible drift rate,
the distribution of saccades is consistent with an angular-velocity-dependent drift rate. Thus,
we can model saccades as events with nonzero angular velocity drawn from a lognormal distribution with an adaptable bias, and with a duration generated by a drift diffusion process withan angular-velocity-dependent drift rate. The similarity in the statistical structure between

28

SciPost Physics Lecture Notes

Submission

857

fixations and saccades suggests that, in both cases, a drift diffusion process can be used tokeep time and thereby specify the duration of different behavioral modes. The parameters ofthat process can then be used to adjust these durations based on other kinematic properties,
or based on experience.

858

3.2.2

854
855
856

859
860
861
862
863

Learning the parameters of a generative model based on experience⃗ we can now
Given a parameterized generative model of behavior, or behavioral policy π(a; ψ),⃗
ask how the parameters ψ of that policy should be modified over time based on experience.
To this end, we can construct an agent that comes embodied with this generative model, andthe study how the behavior of that agent changes as we update the generative model basedon experience (Figure 22).

Figure 22: We can use reinforcement learning methods to learn an optimal behavioral
policy, and compare it to the behavior of real flies (adapted from [93]).
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883

In problems of spatial navigation, a common approach is to discretize the environment into
⃗ θ ) that specifies the optimal parameter settingsstates θ , and construct a lookup table q (ψ,for each state. In our setting, these states correspond to different orientations of the agent.
Moreover, since we are modeling an agent that uses a neural compass to maintain an internal
representation of its own orientation, we will take these states to correspond to the compass
heading θc (see Section 3.1 for a discussion of the compass heading). The agent then uses
the current parameter settings, together with its own compass heading, to select actions, andthen iteratively updates the parameters based on the outcomes of those actions in the current
environment. This process can be quite slow, because there is typically no a priori structure
built into the lookup table, and thus the agent has to experience the consequences of eachparameter setting at each compass heading in order to make adjustments to these parameters(see Appendix D for more details).
To speed up this process, the agent can instead learn a continuous function that specifies
the parameter settings over space. For example, rather than specifying parameter settings
at each of a set of discretized compass headings, one can use a discrete set of continuous
basis functions to tile the space of headings, and learn a discrete set of weights on those basis
functions (Figure 22). This can speed up the learning process because any parameter changes
that are learned at one heading are immediately used to update parameters at nearby headings(if the basis functions are localized in space).
We can use this to learn a policy with the following structure:⃗ ν ) = fν ψ
⃗ ν · g⃗ (θc )ν(θc ; ψ(35)⃗ b ) = fb ψ
⃗ b · g⃗ (θc )b(θc ; ψ(36)
29

SciPost Physics Lecture Notes

884
885
886
887
888
889
890

Submission

where ν and b specify the drift rate of fixations and turn bias of saccades derived in the previous
section, and g⃗ (θc ) specifies a set of basis functions that tile compass headings θc . Informedby what we know about the fly’s internal compass, we choose a set of 8 von Mises functions;the shape of this function closely matches the shape of the compass bump, and the number
mimics the known discretization of the compass (see Section 3.1). We can then use a common
policy-gradient algorithm to update the policy parameters based on experience (see Appendix
E for a derivation of this algorithm):sample action from policy⃗a ∼ π(·|θc ; ψ)take action, gather reward
R = r (a, θc )(37)

update parameters⃗ = αR∇ ⃗ log π(a|θc ; ψ)
⃗
∆ψ
ψ



update state θc
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910

where r (a, θc ) is a reward function that we will discuss shortly. Using this algorithm, thisagent can be trained through experience to reliably gather rewards or avoid punishments by
iteratively updating the parameters of its behavioral policy. We can then compare the outputof this trained agent to the behavior of real flies after they have gone through the learning
paradigm schematized in Figure 20.
In this paradigm, different orientations of the visual scene are paired with rewards orpunishments. The visual scene itself consists of a set of 4 visual patterns, each separated by90◦ . The scene was chosen to have a two-fold symmetry, such there are two distinct visual
patterns, A and B, each repeated twice (i.e., ABAB). One set of patterns (either A or B) is
paired with an aversive heat punishment that is delivered by pinging the fly in the abdomenwith an infrared laser. Thus, if pattern A is punished, the fly would experience heat wheneverit orients within ±45◦ of the center of pattern A (we refer to this region as the “danger zone”,
and the unpunished region as the “safe zone”). Because we do not know whether learning isdriven by the negative effects of punishment or the positive effects of relief from punishment,
we treat the former as generating a negative reward (R = −1), and the latter as generating
a positive reward (R = +1; note that one can use techniques from inverse reinforcement
learning to try to deduce this reward function from observed behavior [99]).
When we train an artificial agent in this same paradigm, the agent learns to generate⃗ ν ) and turn biases b(θc ; ψ⃗ b ) vary approximately
fixations and saccades who drift rates ν(θc ; ψsinusoidally with the compass heading θc (Figure 22):⃗ ν ) ∝ (1 − cos(θc − θsafe ))/2
ν(θc ; ψ⃗ b ) ∝ (1 − sin(θc − θsafe ))/2b(θc ; ψ

911
912
913
914
915
916
917
918
919
920

(38)

where θsafe denotes the center of the safe zone. Low drift rates lead to long fixations at thecenter of the safe zone; a high turn bias to the left of the safe zone generates directed turns
toward safety.
When we compare this learned behavior to the behavior of real flies, we see that flies
exhibit similar structure to their behavior (to see this, we must align the fixations and saccadesof individual flies to heading at which these actions were initiated relative to the center of thesafe zone, and then average across flies; signatures of this structure can also been seen on
an individual fly basis). However, this behavioral structure is apparent in flies even before
they experience any laser punishment, provided that we align the behavior to the preferredheading of each fly prior to averaging. The fact that both naive and trained flies exhibit the30

SciPost Physics Lecture Notes

Submission

936

same behavioral structure suggests that this is not something that needs to be learned in a newenvironment; rather, this behavioral structure seems to innately guide how flies sample theirsurroundings. If this is the case, learning need only shift and scale this behavioral structure,
rather than build it up from scratch.
Such a formulation—in which the behavioral policy has a fixed functional form, but flexible
parameters—dramatically changes the learning process. Rather than building an agent thatlearns individual associations between policy parameters and locations, this suggests that theagent should come pre-equipped with a policy that specifies functional relationships between
policy parameters at all locations in the environment. By building this sort of function into
the agent, any changes that are learned at one location can be used to update behavior at
all locations (rather than being restricted to the immediate vicinity of the agent). This cansignificantly speed up the learning process, but comes at the cost of restricting the space of
behavioral policies that the agent can learn. Moreover, for this to work in real brains, neuralcircuits must be able to enforce this behavioral structure while flexibly shifting and scaling thisstructure to direct movements to different parts of the environment. In the next section, we
discuss how the brain might achieve this.

937

3.2.3

921
922
923
924
925
926
927
928
929
930
931
932
933
934
935

938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957

Building a structured behavioral policy

To construct the policy given in Eq. 38, we use simple operations that could feasibly be performed by neural circuits (Figure 23). We assume that the fly maintains encodes its current
compass heading in a sinusoidal activity profile fc (θ ; θc ) = cos(θ − θc ), where θc denotes thecurrent compass heading, and θ describes an anatomical axis that encodes orientation. Notethat this profile takes on a von Mises shape in the brain region that first encodes the compass heading, but is reformatted into a sinusoidal shape before propagating to downstream
brain regions [62]. This sinusoidal shape is thought to support a range of different vector
computations [100].
We assume that the fly maintains a goal activity profile fg (θ ) that can take on an arbitrary
shape. When we convolve these two activity profiles— fc (θ ; θc ) and fg (θ )—the output issinusoidal and peaks when the compass heading is aligned with the circular mean of the goal
activity profile (Figure 23; see Appendix F for brief derivation). We can thus define the goal
heading, θg , to be the circular mean of the goal activity profile fg (θ ).
If we use the output of this convolution as a motor drive, this operation will guaranteethat the motor drive is strongest when the compass heading is aligned with the goal heading.
However, the profiles given in Eq. 38 require motor drives that are peaked 90◦ and 180◦ awayfrom the goal heading. This can be achieved by phase-shifting the compass heading (and thus
the compass activity profile) before convolving it with the goal activity profile. For a phaseshift of φ, the resulting motor drive will be peaked when the compass heading is aligned withθg − φ (Appendix F). To create the profiles in Eq. 38, we need phase shifts:φν = 180◦ ,φ b = 90◦ .

958
959

This formulation guarantees that the motor drive will be sinusoidally structured, regardless ofthe shape of the goal activity profile, and leads to a policy of the form:⃗ ν ) ∝ Ag cos(θc − θg + φν ) + Bνν(θc ; ψ⃗ b ) ∝ Ag cos(θc − θg + φ b ) + B bb(θc ; ψ

960
961

(39)

(40)

where Ag and θg are the strength and orientation of the circular mean of fg (θ ), respectively,and (Bν , B b ) are offsets. This formulation means that the goal activity profile can be updated
31

SciPost Physics Lecture Notes

Submission

Figure 23: Convolving a sinusoidal compass heading with an arbitrary goal profile
leads to a structured motor output (adapted from [93]; see also [101]).
962
963
964

over time based on experience, without disrupting the motor drive profile. We can thus use
simple Hebbian plasticity rules (analogous to those presented in Section 3.1.3) to update thegoal profile over time:∆ fg (θ , θc ; fg ) = αg ∆g

965

where
 




+
f
(θ
;
θ
)
−
f
(θ
)
Θ(1
−
f
)
−
f
(θ
)
−
f
(θ
;
θ
)
Θ( fg )
c
c
g
g
g
c
c
 
+
+


∆g = − fc (θ ; θc ) − fg (θ ) + Θ( fg ) + fg (θ ) − fc (θ ; θc ) + Θ(1 − fg )

0

966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984

(41)

R(θ ) > 0
R(θ ) < 0(42)

R(θA ) = 0

Here, [·]+ denotes rectification, and Θ(·) is the heaviside function. This plasticity rule will
strengthen the goal profile at the current compass heading when the fly is being rewarded(R > 0), and will weaken the goal profile at the current compass heading when the fly is being
punished (R < 0).
The fly central brain has circuit motifs that could plausibly implement this set of computations [63,93] (Figure 24). As discussed in Section 3.1, the compass heading is maintained by aring attractor network in a region of the brain called the ellipsoid body (EB). From the EB, thecompass activity profile is reformatted from a von-Mises-like shape to a sinusoidal shape [62]before traveling to a downstream region called the fan-shaped body (FB) that is thought tocarry and combine internal representations about the fly’s goals and future actions [63]. There
are two populations of neurons—so-called PFL2 and PFL3 populations—that are thought toimplement the phase shifts in Eq. 39, respectively, and that project either bilaterally or unilaterally to pre-motor regions that are involved in initiating rightward and leftward turns.
The PFL2 population implements a 180◦ phase shift and projects bilaterally to both premotor
regions, and is thus well-positioned to control straight flight. The two PFL3 populations implement ±90◦ phase shifts and project unilaterally to one or the other premotor regions, andare thus well-positioned to control directed turns. Finally, there are dopaminergic neurons
that synapse onto both PFL2 and PFL3 populations, and that themselves receive inputs thatare shaped by the fly’s current compass heading; thus, these neurons are well-positioned to32

SciPost Physics Lecture Notes

Submission

Figure 24: Left: Neurons in the fly central brain implement an anatomical reference
frame tethered to the fly’s current heading. Center: additional populations of neurons implement anatomical phase shifts in this reference frame. Right: putative goal
neurons could store and update goal headings (adapted from [63] and [93]).

986

update the strength of goal synapses in a heading-dependent manner, which could in turn be
used to drive behavior through the PFL populations.

987

3.3

985

988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009

Coupling sensory representations and behavior in closed loop

In Section 3.1, we saw how small neural circuits could maintain and accurately update aninternal representation of heading, and could tether this representation to sensory cues inthe environment through a form of unsupervised learning. In Section 3.2, we saw how thisinternal representation of heading could be used to update an internal representation of goals
in the environment through a form of reinforcement learning. These two systems—learningwhere you are in your environment, and learning goals in that environment—are coupledthrough the same internal representation of heading, and through the actions that are selectedby the behavioral policy that is tethered to this representation (Figure 25). As a result, any
inaccuracies in this representation will impact the formation of goals, the selection of actions,and the subsequent sensory stimuli that are used to update this representation. As a result,the coupling of these systems can shape individual variability in behavior and learning.
This becomes apparent if we revisit the ideas from Section 3.1.3, about how the internal
representation of heading becomes tethered to the outside world. At the end of that section, we
introduced the idea that symmetries in the visual scene can lead to a scenario in which plasticity
tethers the bump to one of two symmetric location in the environment. If two ring neurons areactive for a particular view of a symmetric scene, but one strongly inhibits the bump and theother only weakly inhibits the bump, this induces competition between these two ring neurons,
and the bump will tend to jump to the location of weakest inhibition. In other words, thecompass will develop a two-to-one mapping, in which two different symmetric orientations of
the scene will be mapped onto a single compass heading. This type of confusion is analogous
to the confusion you might expect to experience inside a room with two identical doors on
opposite sides of the room. When you initially enter the room, you might remember which
33

SciPost Physics Lecture Notes

Submission

Figure 25: Two learning systems interact to guide behavior. One learning system(pink) maps visual inputs onto an internal compass heading; a second learning system (green) uses the compass heading to learn a goal heading and guide actions via
a behavioral policy. Plasticity acts at specific locations (white circles) to allow these
representations to change over time based on experience.

1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038

of the two doors you used to enter, but after some time, you might get confused about which
door is which. We can see this confusion in the dynamics of the compass bump, and we canwatch the compass bump jump between different orientations that correspond to symmetricviews of the visual world [93].
This highlights the important difference between the absolute properties of the visual environment and the relative nature of the internal constructs that we build to represent thatenvironment. In many cases, these are closely matched, but in some cases, our internal perceptions differ markedly from the outside world. This has interesting implications for any
downstream behavior that is tethered to these representations. For example, if the fly’s actions are tethered to this internal representation, as evidence suggests, then its actions will
follow this representation, even as that representation is jumping over time. This can be advantageous in an environment where rewards and punishments are coupled to these same
symmetries of the environment. For example, given a policy that is tethered to the difference
between the fly’s current and goal headings (Eq. 40), a jumping heading bump will serve tocopy over this policy at the arena headings that correspond to symmetric views of the visual
scene. The means that the fly need only learn one goal heading; its neural circuits will efficiently structure behavior about that goal heading, and its jumping compass bump will copythat structure at multiple locations that share the same visual patterns and reward structures.
For this to work effectively, the mapping of the visual world onto the compass must be
stabilized, which takes time in a new environment. As a result, stronger initial goal headings,
which lead to more exploitative behavior about the goal heading, can help to quickly anchor
the mapping of the visual world onto the compass, particular in the presence of visual symmetries. This, in turn, can help the fly more quickly update its goal heading in the face of
rewards and punishments. Weak goal headings, on the other hand, can lead to more diffusive,
exploratory behavior, can slow the development of the visual map in the presence of symmetries, and can lead to misalignment between the learning of the visual map and the learningof goals tethered to that visual map. This, in turn, can slow the overall learning process. As aresult, individual variability in learning can arise purely because of the dynamics that couple
these multiple internal representations. See [93] for further discussion.

34

SciPost Physics Lecture Notes

1039

3.4

Submission

Summary1057

In this section, we explored some of the challenges of building compact internal representations of the outside world, bootstrapping these representations by using one evolving representation to update another, and ultimately tethering efficient behavior to these internal
representations. One of the major insights from this line of work is that pre-motor circuits inthe brain seems to employ hardwired anatomical motifs for exploiting structure in an animal’s
predictable relationship to its surroundings, just as early sensory circuits exploit predictablestructure in incoming sensory signals. In the field of machine learning, these architectural
constraints are often called ‘structural priors’ or ‘inductive biases’, and ongoing work in thefield seeks to understand what sorts of inductive biases can lead to more flexible and generalizable architectures [102, 103]. Here, we saw how inductive biases in the fly brain—in theform of anatomical motifs that efficiently structure behavior about a single goal heading—cansignificantly speed learning by reducing the number of associations that an animal has to experience. This suggests an exciting direction for future research, in terms of understanding
how the brain should best balance the speed enabled by hardwired circuits, and the flexibility
afforded by plasticity within these circuits. More broadly, this prompts the question as to what
additional features of an animal’s relationship to its environment are predictable enough towarrant hard-wired solutions—something that will be aided by quantitative analyses of natural
behavior in the context of ethologically-relevant tasks.

1058

4

1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056

Outlook1076

In these notes, we have touched upon different theoretical frameworks—including efficientcoding, Bayesian inference, and reinforcement learning—that are used to understand sensory
coding, inference, and action selection. We then used aspects of these frameworks in thecontext of a specific example—navigational learning in the fruit fly—to understand how real
brains make sense of incoming sensory stimuli, build internal representations of their relationship to the external world, and use these internal representations to guide behavior. In both ofthese contexts, we discussed how these frameworks provide a perspective on not only how thebrain operates, but also why it might operate in one manner versus another. Looking ahead,one major opportunity in theoretical neuroscience is to understand how these different frameworks relate to one another, and how they inform and constrain one another. We saw how
resource constraints, as considered in the framework of efficient coding, can impact the speed
and accuracy of Bayesian inference, and how constraints on the precision of inference canin turn impact action selection. We also saw how behavioral policies can exploit the reliable
statistics of movement to speed learning, just as sensory codes can exploit the reliable statistics
of sensory stimuli to speed inference. Finally, we saw that it is becoming possible to test thepredictions of these interconnected frameworks in real brains, where the fruit fly—with its
targeted genetic tools, near-complete connectome, and behavioral repertoire—provides one
exciting test bed to link multiple successive computations in closed-loop.

1077

Acknowledgements

1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075

1078
1079
1080
1081

Wiktor Młynarski and Marcella Noorman provided helpful feedback on these notes. As indicated in the respective section headings, some of the work and perspectives presented in
Section 2 derived from collaborations with Wiktor Młynarski and Tzuhsuan (Maz) Ma; similarly, some of the work and perspectives presented in Section 3 derived from collaborations

35

SciPost Physics Lecture Notes

Submission

1085with Marcella Noorman, Brad Hulse, Vivek Jayaraman, Sung Soo Kim, Larry Abbott, Sandro
Romani, and Chuntao Dan. Some perspectives presented in the introduction derived fromcollaborations with Gašper Tkačik, Jonathan Victor, Mary Conte, John Briguglio, and Vijay
Balasubramanian.

1086

Funding information

1082
1083
1084

This work was supported by the Howard Hughes Medical Institute.

36

SciPost Physics Lecture Notes

1087

1088

A

Submission

Decomposing mutual information

The mutual information between X and Y is defined as:X
I(X ; Y ) =P(x , y) log

x ∈X,y∈Y
1089

(A.1)

x ∈X

Noting that the conditional entropy of X given Y is defined as:XX
H(X |Y ) =P(y) −
P(x |y) log P(x |y) ,

(A.3)

x ∈X

y∈Y
1091

P(x )P(y)We can write P(x , y) = P(x |y)P(y), which allows us to re-express mutual information:X
P(x |y)P(y)P(x |y)P(y) log
I(X ; Y ) =P(x )P(y)x ∈X,y∈Y(A.2)
X
X
P(x |y) [log P(x |y) − log P(x )]
P(y)
=
y∈Y

1090



P(x , y)we can use this to rewrite the mutual information as:
X
I(X ; Y ) = −H(X |Y ) −P(x , y) log P(x )x ∈X,y∈YX= −H(X |Y ) −P(y|x )P(x ) log P(x )(A.4)

x ∈X,y∈Y= −H(X |Y ) −X

P(x ) log P(x )

x ∈X

1092

1093

1094
1095

X

P(y|x )y∈Y

= −H(X|Y ) + H(X )P
P
where we have used the fact that H(X ) = − x ∈X P(x ) log P(x ), and y∈Y P(y|x ) = 1.

B

Maximum entropy distributions

Consider a neuron that can produce a discrete set of discriminable responses. The entropy of
the response distribution is:XH =−pi log(pi )
(B.1)
i

1096
1097
1098

subject to the constraint i pi = 1.
To find the entropy-maximizing distribution of responses, we can define the Lagrangian Lwith Lagrange multiplier λ0 :XXL(p, λ0 ) = −pi log(pi ) + λ0
pi − 1
(B.2)
P

i
1099

i

Maximizing this with respect to pi gives:∂ L(p, λ0 )∂p

= 0 =⇒ − log(pi ) − 1 + λ0=⇒ pi = exp(λ0 − 1)
37

(B.3)

SciPost Physics Lecture Notes

1100

Imposing the constraintP

Submissioni pi = 1 means:Xexp(λ0 − 1) = 1=⇒ N exp(λ0 − 1) = 1(B.4)=⇒ pi = 1/N
1101
1102
1103
1104

Intuitively, given a constraint on the number of discriminable response levels, the maximum
entropy distribution is flat (and corresponds to ‘histogram equalization’).
If we now consider a continuous distribution of responses but impose a constraint on themean firing rate mu, this gives:
Z
Z

Z


L(p, λ0 , λ1 ) = −1105

p(r ) log(p(r ))d r + λ0p(r )d r − 1 + λ1

r p(r )d r − µ

(B.5)

and
∂L
∂p

= − log(pi ) − 1 + λ0 + λ1 r = 0(B.6)=⇒ p(r ) = exp(λ0 − 1) exp(λ1 r ).
1106

Plugging this back into the equations for our constraints, we have:
λ1 = −1/µ;exp(λ0 − 1) = 1/µ1107

which givesp(r ) =

1
µ


exp −

r



µ

(B.7)

1110

Thus, given a constraint on the mean firing rate, the distribution of neural responses is exponential. It is straightforward to use the same logic to show that an additional constraint on
the variance of neural responses leads to a Gaussian distribution.

1111

C

1108
1109

1112
1113
1114

Bayesian ideal observer model in a switching environment

Here, we derive the ideal observer model for the scenario described in Section 2.2, which must
infer the current state θ t of the environment given the history of past sensory signals s τ ≤ t .
Given random variables A and B, Bayes Rule states:
P(A|B) =

1115

P(B)

(C.1)

For three variables A, B, C , this becomes:
P(A|B, C ) =1116

P(B|A)P(A)P(B|A, C )P(A|C )P(B|C )

(C.2)

We can now make the substitutions: A = θ t , B = s t , and C = sτ<t :P(θ t |s t , sτ<t ) =P(s t |θ t , sτ<t )P(θ t |sτ<t )P(s t |sτ<t )
38

(C.3)

SciPost Physics Lecture Notes

1117

Submission

This distribution must be normalized, which gives:X

P(θ t |s t , sτ<t ) = 1 =

1118

t

t

τ<t )P(θ t |sτ<t )P(s t |sτ<t )

θt

θt

1119

X P(s |θ , s(C.4)

Since the distribution in the denominator does not depend on θ t , we can pull it out of the sum
and define this to be the normalization constant Ω:XP(s t |θ t , sτ<t )P(θ t |sτ<t )(C.5)Ω ≡ P(s t |sτ<t ) =
θt

1120

We now have:P(θ t |s t , sτ<t ) =

1121

1

P(s t |θ t , sτ<t )P(θ t |sτ<t )Ω
The second distribution can be expanded and simplified as follows:X
P(θ t |sτ<t ) =P(θ t |θ t −1 , sτ<t )P(θ t −1 |sτ<t )

(C.6)

(C.7)

θ t −1=X

P(θ t |θ t −1 )P(θ t −1 |s t −1 , sτ<t −1 )(C.8)

θ t −1
1122

Putting this together, we have:P(θ t |s t , sτ<t ) =1Ω

P(θ t |s t )X

P(θ t |θ t −1 )P(θ t −1 |s t −1 , sτ<t −1 )(C.9)

θ t −11126

For the specific case of a two-state environment, we can define P tL ≡ P(θ t = θ L |s t , sτ<t ),and P tH = (1 − P tL ). The posterior distribution is thus fully specified by a single number: theprobability that the environment is in the low state, given the history of past observations. In
matrix notation, we now have:





 1 − ps
P tL−1
1
P tL
ps
L
H
P(s t |θ t = θ ) P(s t |θ t = θ )=(C.10)
1 − P tLps
1 − p s 1 − P tL−1Ω1127

Or equivalently,

1123
1124
1125

P tL =

1128

1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140

D

1
Ω

P(s t |θ t = θ L )[(1 − p s )P tL−1 + p s (1 − P tL−1 )](C.11)

Primer on reinforcement learningThis is meant to provide a quick-and-dirty tutorial on reinforcement learning; for a more thorough overview and background, please refer to Sutton & Barto [40], which provided the source
for these notes.
Reinforcement learning considers the process of learning to select appropriate actions a inparticular states θ of the environment, based on feedback from the environment in the formof rewards r . Note that it is typical to denote states with the variable s, instead of θ ; here weuse θ to highlight that these states are typically hidden from the agent must be inferred fromsensory signals, which we denote s . In Section 2.2, we used such latent states to parameterize
distributions of sensory stimuli; in Section 2.3, the latent state specified the reward probabilityof different levers that could be sampled by an agent. And in Section 3, the latent state specified
the animal’s heading in the environment, which must be inferred from visual and self-motion
signals.
39

SciPost Physics Lecture Notes

1141
1142
1143

Submission

The goal of a reinforcement learning agent is to maximize long-term accumulated reward,
which is called the return G t and is defined as the total sum of all rewards received at future
times t ′ > t :G t ≡ R t +1 + R t +2 + R t +3 + ...

1144
1145
1146

(D.1)

As written, this expression assumes that all future rewards are equally important. To accountfor the fact the immediate rewards might be more valuable than rewards in the distant future,
we can “discount” future rewards with a discount factor γ ∈ [0, 1]:
G t = R t +1 + γR t +2 + γ2 R t +3 + ...
∞X′=
γ t −1 R t ′(D.2)

t ′ =t +1= R t +1 + γG t +1
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161

where γ = 1 defines a far-sighted agent that values all rewards equally, and γ = 0 defines apurely myopic agent that only values immediate rewards.
The return G t represents the sum of actual future rewards, which is not knowable to any
agent. Instead, an agent can compute the expected future rewards, starting in state θ andfollowing a policy π(a|θ ). The policy specifies the probability of selecting a given action afrom a state θ . If it is helpful to have a concrete example in mind, consider a scenario in whichan agent navigates in 2D environment to gather rewards that are given when the agent reaches
specific locations in the environment. A typical way to treat this scenario is to discretize theenvironment into a grid of locations; these locations then serve as our states (so, in a 10 × 10
grid, there are 100 states). At any grid location, it is often assumed that there are at most 5
available actions: stay, move 1 step up, move 1 step down, move 1 step left, or move 1 step
right. A policy can then be specified as the probability of taking each of the 5 actions fromeach of the 100 states in the environment.
With this example in mind, we can start by computing the expected future rewards thatwill be obtained one timestep in the future:
Eπ [R t +1 |θ t = θ ] =X

π(a|θ )

1163
1164
1165
1166

p(θ ′ |θ , a)r (θ , a, θ ′ )

θ′

a
1162

X

where p(θ ′ |θ , a) governs the dynamics of the environment and specifies the probability thatthe agent transitions to a state θ ′ when beginning in state θ and taking action a; the reward
function r (θ ′ , a, θ ) specifies the reward received under the same transition. We can use Eq.
D.3 to define a state-value function vπ (θ ) that defines how ‘good’ it is (in terms of expected
return) to begin in state θ and follow the policy π:vπ (θ ) ≡ Eπ G t θ t = θ

1167

(D.3)

(D.4)

Using the result from Eqs. D.2-D.3, we can write this as:vπ (θ ) = Eπ R t +1 + γG t +1 θ t = θ

X
X


=
π(a|θ )p(θ ′ |θ , a) r (θ , a, θ ′ ) + γEπ G t +1 θ t +1 = θ ′
=

a

θ′

X

X

a

π(a|θ )

p(θ ′ |θ , a) r (θ , a, θ ′ ) + γvπ (θ ′ )

θ′

40



(D.5)

SciPost Physics Lecture Notes

1168
1169
1170

Submission

This is the recursive Bellman equation for the state-value function [104]. We can also use thisto define a state-action value function qπ (θ , a) that defines the value of starting in a state θ ,taking an action a, and following the policy π from then onward:


X
′
′
′
qπ (θ , a) ≡p(θ |θ , a) r (θ , a, θ ) + γvπ (θ )
(D.6)
θ′

1171

for which there is an analogous Bellman equation:XXqπ (θ , a) =p(θ ′ |θ , a) r (θ , a, θ ′ ) + γπ(a′ |θ ′ )qπ (θ ′ , a′ )
θ′

1172

The optimal value function satisfies the Bellman optimality equation:v∗ (θ ) = max Eπ R t +1 + γv ∗ (θ t +1 ) θ t = θ , a t = a
a
X


= max
p(θ ′ |θ , a) r (θ , a, θ ′ ) + γv∗ (θ ′ )
a

1173
1174
1175
1176

1178
1179
1180
1181
1182
1183
1184
1185

and analogously for q∗ (θ , a).
Note that one can use a value function vπ (θ ) or action-value function qπ (θ , a) to definea policy. For example, a purely greedy policy would select actions that maximize the current
value:

1186

1188
1189
1190
1191
1192
1193
1194

1196

(D.10)

where β → ∞ drives purely exploitative actions that maximize q , and β → 0 drives purely
random actions.
A goal of reinforcement learning is to use value functions to determine the optimal policies that maximize the return G. Historically, there have been three main sets of methods
for learning value functions. The first are dynamic programming methods, which assume amodel of the environment in the form of p(θ ′ |θ , a) and r (θ , a, θ ′ ), and learn v∗ , q∗ , andπ∗ via bootstrapping (i.e., updating an estimate of one quantity based on an estimate of another quantity). Value iteration, which we introduced in Section 2.3, is one such method thatconverts Eq. D.8 into an update rule for the value function:


X
′
′
′
v t +1 (θ ) = max
p(θ |θ , a) r (θ , a, θ ) + γv t (θ )(D.11)
a

1195

(D.9)

If q = q∗ is the optimal action-value function, then Eq. D.9 defines the optimal policyπ∗ (a|θ ) = argmaxa (q∗ (θ , a)). However, if q is not optimal, it can be advantageous to usea policy that balances exploitative actions (which maximize the current estimate of q ) with
exploratory actions (which have lower expected value but could result in higher long-term
payoffs). A simple version of such a policy is called an ‘epsilon-greedy’ policy, in which theagent chooses the greedy (exploitative) action with probability (1 − ε), and chooses a random(exploratory) action with probability ε, where ε ∈ [0, 1] is typically chosen to be small. Alternatively, one can tie the degree of exploration to the value function by choosing a ‘softmax’
policy:π(a|θ ) ∝ exp(βq (θ , a))

1187

(D.8)

θ′

A = argmaxa q (θ , a)
1177

(D.7)

a′

θ ′ ,r

The second set of methods are Monte Carlo methods, which do not assume a model of theenvironment, and instead directly estimate v ∗ , q ∗ , and π∗ via sampling (i.e., from the outcome
41

SciPost Physics Lecture Notes

1197

Submissionof simulated experiences). An example of a Monte Carlo update is the following:v t +1 (θ ) = v t (θ ) + α[G t − v t (θ )]

1198
1199
1200
1201
1202

(D.12)

where the return G t must be computed from the outcome of simulated experiences. G t is thenused as a target to update v t .
A third class of methods, temporal difference (TD) methods, combine aspects of dynamic
programming and Monte Carlo methods to learn from experience via bootstrapping [40]. The
simplest TD method replaces the return G t in Eq. D.12 with the estimate G t ≊ R t +1 +γv t (θ t +1 ):
v t +1 (θ t ) = v t (θ t ) + α R t +1 + γv t (θ t +1 ) − v t (θ t )
q t +1 (a t , θ t ) = q t (a, θ t ) + α R t +1 + γq t (a t +1 , θ t +1 ) − q t (a t , θ t )

1203
1204
1205
1206
1207
1208
1209
1210
1211

(D.14)

Here, δ t = R t +1 + γv t (θ t +1 ) − v t (θ t ) (and similarly δ t = R t +1 + γq t (a t +1 , θ t +1 ) − q t (a t , θ t ))is often referred as the TD error. Eq. D.14 forms the basis of the so-called SARSA learningalgorithm. Note that the updating of the value functions depends on the actions taken andrewards received, which in turn depend on the policy used to select those actions. Thus, whenusing one of the policies defined in Eqs. D.9-D.10, the actions selected via the policy willchange over time as the value functions are updated via Eqs. D.13-D.14.
With an algorithm like SARSA, the agent can only update the value of states as these states
are visited. To update states that were visited in the past, we can augment this update rulewith an eligibility trace Zt (θ ) that allows previously-visited to be eligible for updating:v t +1 (θ t ) = v t (θ t ) + α R t +1 + γv t (θ t +1 ) − v t (θ t ) Zt (θ t )1212(D.13)(D.15)

where Zt (θ ) follows its own update rule:¨
Zt (θ ) =λγZt −1 (θ )θ ̸= θ t

1 + λγZt −1 (θ )

θ = θt(D.16)

1216

Here, λ ∈ [0, 1] is a trace-decay parameter that specifies how quickly the eligibility of eachstate will decay over time. When λ = 0, only the current state can be updated; when λ = 1,
the eligibility of an unvisited state falls by γ with each timestep. Larger values of λ cause theagent to associate previously-visited states with current rewards.

1217

E

1213
1214
1215

1218
1219

Derivation of policy gradient algorithm

This follows the derivation given in Sutton & Barto [40]. We begin by defining a performance⃗ that we want to maximize, given a policy π(a|θ ; ψ)⃗ parameterized by ψ:⃗measure J(ψ)⃗ ≡ vπ (θ )
J(ψ)
⃗
ψ

1220
1221

(E.1)

⃗ starting in state θ .
where vπψ⃗ (θ ) is the value for a policy π, parameterized by ψ,
We want to find the parameter update that increases performance over time:X∇ψ⃗ vπ (θ ) = ∇ψ⃗=Xaπ(a|θ )qπ (θ , a)aqπ (θ , a)∇ψ⃗ π(a|θ ) + π(a|θ )∇ψ⃗ qπ (θ , a)

42



(E.2)

SciPost Physics Lecture Notes

1222

Submission

The second term in this sum, *, can be written as:Xp(θ ′ , r |θ , a)(r + vπ (θ ′ ))∗ = π(a|θ )∇ψ⃗
θ ′ ,r= π(a|θ )∇ψ⃗
1223

θ′

(E.3)

p(θ ′ |θ , a)∇ψ⃗ vπ (θ ′ )We can now note that∇ψ⃗ vπ (θ ) =

1224

X

X



a

qπ (θ , a)∇ψ⃗ π(a|θ ) + π(a|θ )

X
θ′


′

′

p(θ |θ , a)∇ψ⃗ vπ (θ )and in a similar manner, ∇ψ⃗ vπ (θ ′ ) can be ‘rolled out’ and expressed in terms of ∇ψ⃗ vπ (θ ′′ ):∇ψ⃗ vπ (θ ) =Xaqπ (θ , a)∇ψ⃗ π(a|θ ) + π(a|θ )Xp(θ ′ |θ , a)

θ′

X
X
qπ (θ ′ , a′ )∇ψ⃗ π(a′ |θ ′ ) + π(a′ |θ ′ )p(θ ′′ |θ ′ , a′ )∇ψ⃗ vπ (θ ′′ )
1225

1227



.

Note that the first two rows of the above equation are identical in form, but are weighted
by the probability of transitioning between states under policy π. Replacing θ with its value
sampled at time t , we can write:⃗ = Eπ∇ψ⃗ J(ψ)

X
a⃗qπ (θ t , a)∇ψ⃗ π(a|θ t ; ψ)
= Eπ 

X⃗ π (θ t , a)π(a|θ t ; ψ)q

a

= Eπ

X
a

1228
1229



⃗
∇ψ⃗ π(a|θ t ; ψ)⃗π(a|θ t ; ψ)

⃗ π (θ t , a)∇ ⃗ log π(a|θ t ; ψ)⃗π(a|θ t ; ψ)q
ψ

1230

1232
1233

1234



(E.7)

where G t is the return at time t , and where we have written qπ (θ t , a t ) = Eπ [G t |θ t , a t ]. The
argument of this expectation can be sampled on every timestep, and the expectation of this⃗ [105]. This gives the update rule that we used inquantity is equal to the gradient ∇ψ⃗ J(ψ)
Eq. 37:⃗ t +1 = γ⃗t + αR t ∇ ⃗ log π(a t |θ t ; ψ)
⃗
ψ
ψ

1235

(E.6)

⃗ in the second line, and written ∇x /x = ∇ log x
where we have multiplied and divided by π(a|θ t , ψ)in the third line. If we now replace a by a sampled action a t ∼ π(·|θ t ), we have:⃗ = Eπ G t ∇ ⃗ log π(a t |θ t ; ψ)⃗∇ψ⃗ J(ψ)
ψ

1231

(E.5)

θ ′′

a′

1226

(E.4)

(E.8)

where we replaced the full return G t with the instantaneous reward R t (i.e., we considered apurely myopic agent).

43

SciPost Physics Lecture Notes

1236

1237

F

Submission

Structuring a motor drive about a goal heading

We consider a motor drive m of the form:
Z 2πm=dθ cos(θ − θc ) fg (θ ),

(F.1)

0
1238
1239
1240

where cos(θ − θc ) is an activity profile that encodes the current compass heading θc , andwhere fg (θ ) is an arbitrary activity profile that encodes the goal heading.
Expanding the cos, we have:

m=

Z 2π
dθ

1

2
Z
2π
−iθ


e i(θ −θc ) + e i(θ −θc ) fg (θ )

0

=
1241

e

c

2

dθ e

iθ

fg (θ ) +

0

e iθc
2

(F.2)

Z 2π
dθ e

−iθ

fg (θ ),

0

where
Z 2π

dθ e ±iθ fg (θ ) ≡ rg e ±iθg

(F.3)

0
1242
1243
1244

defines the circular mean of the function fg (θ ), and rg and θg are the modulus and orientationof the circular mean, respectively.
This allows us to write:m = m(θc , θg ) =rge −iθc e iθg + e iθc e −iθg2= rg cos(θc − θg ).

1245
1246
1247
1248

(F.4)

Thus, when integrated over a full period, the product of the goal and compass activity profiles
will be largest when the compass heading is aligned with the circular mean of the goal activity
profile. We can then use this circular mean to define the goal heading.
If we now add a phase shift φ to the compass heading, this becomes:m(θc , θg , φ) =Z 2πdθ cos θ − (θc + φ) fg (θ )

0

(F.5)

= rg cos(θc + φ − θg ),
1249
1250
1251
1252
1253

which peaks at θc = θg − φ. Different phase shifts can then be used to generate motor drives
that peak at different compass headings relative to the goal heading. Importantly, the motor
drive will retain a sinusoidal profile regardless of the shape of fg (θ ), and thus regardless ofthe goal heading θg . As a result, we can use any method we like for updating the goal activity
profile, and it will not disrupt the structure of the motor drive.

44

SciPost Physics Lecture Notes

1254

1255
1256

1257
1258

1259
1260
1261

1262
1263

1264
1265

1266
1267

1268
1269

1270
1271

1272
1273

1274
1275
1276

1277
1278

1279
1280

1281
1282

1283
1284
1285

1286
1287

1288
1289

1290
1291

Submission

References[1] B. Julesz, E. N. Gilbert and J. D. Victor, Visual discrimination of textures with identical
third-order statistics, Biological cybernetics 31, 137 (1978).
[2] J. D. Victor, C. Chubb and M. M. Conte, Interaction of luminance and higher-order statistics in texture discrimination, Vision research 45(3), 311 (2005).
[3] G. Tkačik, J. S. Prentice, J. D. Victor and V. Balasubramanian, Local statistics in naturalscenes predict the saliency of synthetic textures, Proceedings of the National Academy of
Sciences 107(42), 18149 (2010).
[4] A. M. Hermundstad, J. J. Briguglio, M. M. Conte, J. D. Victor, V. Balasubramanian andG. Tkačik, Variance predicts salience in central sensory processing, eLife 3, e03722 (2014).
[5] D. J. MacKay, Information theory, inference and learning algorithms, Cambridge university press (2003).
[6] E. P. Simoncelli and B. A. Olshausen, Natural image statistics and neural representation,
Annual review of neuroscience 24(1), 1193 (2001).
[7] F. Attneave, Some informational aspects of visual perception., Psychological review 61(3),183 (1954).
[8] H. B. Barlow et al., Possible principles underlying the transformation of sensory messages,
Sensory communication 1, 217 (1961).
[9] S. Laughlin, A simple coding procedure enhances a neuron’s information capacity,
Zeitschrift für Naturforschung c 36(9-10), 910 (1981).
[10] M. V. Srinivasan, S. B. Laughlin and A. Dubs, Predictive coding: a fresh view of inhibitionin the retina, Proceedings of the Royal Society of London. Series B. Biological Sciences
216(1205), 427 (1982).
[11] J. J. Atick and A. N. Redlich, Towards a theory of early visual processing, Neural computation 2(3), 308 (1990).
[12] J. J. Atick and A. N. Redlich, What does the retina know about natural scenes?, Neuralcomputation 4(2), 196 (1992).
[13] J. H. van Hateren, A theory of maximizing sensory information, Biological cybernetics
68(1), 23 (1992).
[14] G. Tkačik, J. S. Prentice, V. Balasubramanian and E. Schneidman, Optimal population coding by noisy spiking neurons, Proceedings of the National Academy of Sciences
107(32), 14419 (2010).
[15] E. Doi and M. S. Lewicki, A simple model of optimal population coding for sensory systems,
PLoS computational biology 10(8), e1003761 (2014).
[16] N. Brenner, W. Bialek and R. d. R. Van Steveninck, Adaptive rescaling maximizes information transmission, Neuron 26(3), 695 (2000).
[17] A. L. Fairhall, G. D. Lewen, W. Bialek and R. R. d. R. van Steveninck, Efficiency andambiguity in an adaptive neural code, Nature 412(6849), 787 (2001).

45

SciPost Physics Lecture Notes

1292
1293

1294
1295

1296
1297

1298
1299

1300
1301

1302
1303

1304
1305

1306
1307

1308
1309

1310
1311

1312
1313

1314
1315
1316

1317
1318
1319
1320

1321
1322

1323
1324

1325
1326
1327

1328
1329

1330
1331

Submission[18] W. F. Młynarski and A. M. Hermundstad, Efficient and adaptive sensory codes, Nature
Neuroscience 24(7), 998 (2021).
[19] W. F. Młynarski and A. M. Hermundstad, Adaptive coding for dynamic sensory inference,
Elife 7, e32055 (2018).
[20] I. Dean, N. S. Harper and D. McAlpine, Neural population coding of sound level adapts
to stimulus statistics, Nature neuroscience 8(12), 1684 (2005).
[21] B. Wark, A. Fairhall and F. Rieke, Timescales of inference in visual adaptation, Neuron
61(5), 750 (2009).
[22] D. B. Kastner and S. A. Baccus, Coordinated dynamic encoding in the retina using opposing
forms of plasticity, Nature neuroscience 14(10), 1317 (2011).
[23] I. Nemenman, 4. information theory and adaptation, Quantitative Biology: From Molecular to Cellular Systems p. 73 (2012).
[24] C. Robert, The Bayesian choice: from decision-theoretic foundations to computational
implementation, Springer Science & Business Media (2007).
[25] D. B. Kastner and S. A. Baccus, Spatial segregation of adaptation and predictive sensitization in retinal ganglion cells, Neuron 79(3), 541 (2013).
[26] T. R. Appleby and M. B. Manookin, Neural sensitization improves encoding fidelity in the
primate retina, Nature communications 10(4017), 1 (2019).
[27] B. N. Lundstrom, M. H. Higgs, W. J. Spain and A. L. Fairhall, Fractional differentiation
by neocortical pyramidal neurons, Nature neuroscience 11(11), 1335 (2008).
[28] S. E. Clarke, A. Longtin and L. Maler, Contrast coding in the electrosensory system: parallels with visual computation, Nature Reviews Neuroscience 16(12), 733 (2015).
[29] J. O’Doherty, M. L. Kringelbach, E. T. Rolls, J. Hornak and C. Andrews, Abstract rewardand punishment representations in the human orbitofrontal cortex, Nature neuroscience4(1), 95 (2001).
[30] P. Vertechi, E. Lottem, D. Sarra, B. Godinho, I. Treves, T. Quendera, M. N. Oude
Lohuis and Z. F. Mainen, Inference-based decisions in a hidden state foraging task:
Differential contributions of prefrontal cortical areas, Neuron 106(1), 166 (2020),
doi:https://doi.org/10.1016/j.neuron.2020.01.017.
[31] L. P. Sugrue, G. S. Corrado and W. T. Newsome, Matching behavior and the representationof value in the parietal cortex, Science 304(5678), 1782 (2004).
[32] V. D. Costa, V. L. Tran, J. Turchi and B. B. Averbeck, Reversal learning and dopamine: abayesian perspective, Journal of Neuroscience 35(6), 2407 (2015).
[33] B. A. Bari, C. D. Grossman, E. E. Lubin, A. E. Rajagopalan, J. I. Cressy and J. Y. Cohen,
Stable representations of decision variables for flexible behavior, Neuron 103(5), 922(2019), doi:https://doi.org/10.1016/j.neuron.2019.06.001.
[34] M. P. Karlsson, D. G. Tervo and A. Y. Karpova, Network resets in medial prefrontal cortex
mark the onset of behavioral uncertainty, Science 338(6103), 135 (2012).
[35] Y. Liu, Y. Xin and N.-l. Xu, A cortical circuit mechanism for structural knowledge-based
flexible sensorimotor decision-making, Neuron 109(12), 2009 (2021).
46

SciPost Physics Lecture Notes

1332
1333
1334

1335
1336
1337

1338
1339

1340
1341

1342
1343

1344
1345

1346
1347

1348
1349

1350
1351
1352

1353
1354
1355

1356
1357

1358
1359
1360

1361
1362

1363
1364
1365

1366
1367

1368
1369

1370
1371
1372

Submission[36] C. C. Beron, S. Q. Neufeld, S. W. Linderman and B. L. Sabatini, Mice exhibit stochastic
and efficient action switching during probabilistic decision making, Proceedings of the
National Academy of Sciences 119(15), e2113961119 (2022).
[37] A. Rajagopalan, R. Darshan, J. E. Fitzgerald and G. C. Turner,
Expectationbased learning rules underlie dynamic foraging in drosophila,
bioRxiv (2022),
doi:10.1101/2022.05.24.493252.
[38] K. J. Astrom, Optimal control of markov decision processes with incomplete state estimation, J. Math. Anal. Applic. 10, 174 (1965).
[39] L. P. Kaelbling, M. L. Littman and A. R. Cassandra, Planning and acting in partiallyobservable stochastic domains, Artificial intelligence 101(1-2), 99 (1998).
[40] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, The MIT Press,
second edn. (2018).
[41] T. Ma and A. M. Hermundstad, A vast space of compact strategies for highly efficientdecisions, bioRxiv pp. 2022–08 (2022).
[42] N. Tishby and D. Polani, Information theory of decisions and actions, In Perception-action
cycle: Models, architectures, and hardware, pp. 601–636. Springer (2010).
[43] H. S. Seung, How the brain keeps the eyes still, Proceedings of the National Academy of
Sciences 93(23), 13339 (1996), doi:10.1073/pnas.93.23.13339.
[44] M. Camperi and X.-J. Wang, A model of visuospatial working memory in prefrontal cortex:
recurrent network and cellular bistability, Journal of computational neuroscience 5(4),383 (1998).
[45] D. Hansel and H. Sompolinsky, Modeling feature selectivity in local cortical circuits, In
C. Koch and I. Segev, eds., Methods in neuronal modeling: from synapses to networks,
chap. Chap 13. Cambridge, MA, 2 edn. (1998).
[46] M. Tsodyks, Attractor neural network models of spatial maps in hippocampus, Hippocampus 9(4), 481 (1999).
[47] A. Compte, N. Brunel, P. S. Goldman-Rakic and X.-J. Wang, Synaptic mechanisms andnetwork dynamics underlying spatial working memory in a cortical network model, Cerebral cortex 10(9), 910 (2000).
[48] A. Samsonovich and B. L. McNaughton, Path integration and cognitive mapping in a continuous attractor neural network model, Journal of Neuroscience 17(15), 5900 (1997).
[49] S. Stringer, T. Trappenberg, E. Rolls and I. de Araujo, Self-organizing continuous attractor
networks and path integration: one-dimensional models of head direction cells., Network(Bristol, England) 13(2), 217 (2002).
[50] Y. Burak and I. R. Fiete, Accurate path integration in continuous attractor network models
of grid cells, PLoS computational biology 5(2), e1000291 (2009).
[51] H. S. Seung, Continuous attractors and oculomotor control, Neural Networks 11(7-8),1253 (1998).
[52] M. Goldman, A. Compte and X.-J. Wang, Neural integrator models, In L. R. Squire, ed.,Encyclopedia of Neuroscience, pp. 165–178. Academic Press, Oxford, ISBN 978-0-08045046-9, doi:https://doi.org/10.1016/B978-008045046-9.01434-0 (2009).
47

SciPost Physics Lecture Notes

1373
1374
1375

1376
1377

1378
1379

1380
1381
1382

1383
1384

1385
1386
1387

1388
1389
1390

1391
1392

1393
1394
1395

1396
1397
1398
1399

1400
1401
1402
1403

1404
1405
1406

1407
1408

1409
1410

1411
1412
1413

Submission[53] J. J. Knierim and K. Zhang, Attractor dynamics of spatially correlated neural activity in the limbic system, Annual Review of Neuroscience 35(1), 267 (2012),
doi:10.1146/annurev-neuro-062111-150351, PMID: 22462545.
[54] R. Chaudhuri and I. Fiete, Computational principles of memory, Nature neuroscience
19(3), 394 (2016).
[55] B. K. Hulse and V. Jayaraman, Mechanisms underlying the neural computation of headdirection, Annual review of neuroscience 43, 31 (2020).
[56] K. Wimmer, D. Q. Nykamp, C. Constantinidis and A. Compte, Bump attractor dynamics
in prefrontal cortex explains behavioral precision in spatial working memory, Natureneuroscience 17(3), 431 (2014).
[57] I. Pisokas, S. Heinze and B. Webb, The head direction circuit of two insect species, eLife9, e53985 (2020), doi:10.7554/eLife.53985.
[58] D. B. Turner-Evans, S. Wegener, H. Rouault, R. Franconville, T. Wolff, J. D. Seelig,
S. Druckmann and V. Jayaraman, Angular velocity integration in a fly heading circuit,
eLife 6, e23496 (2017), doi:https://doi.org/10.7554/eLife.23496.
[59] K. S. Kakaria and B. L. de Bivort, Ring attractor dynamics emerge from a spiking modelof the entire protocerebral bridge, Frontiers in Behavioral Neuroscience 11 (2017),
doi:10.3389/fnbeh.2017.00008.
[60] J. D. Seelig and V. Jayaraman, Neural dynamics for landmark orientation and angularpath integration, Nature 521(7551), 186 (2015), doi:10.1038/nature14446.
[61] J. Green, A. Adachi, K. K. Shaw, J. D. Hirokawa, P. S. Magani and G. Maimon, A neural circuit architecture for angular integration in drosophila, Nature 546, 101 (2017),
doi:https://doi.org/10.1038/nature22343.
[62] D. B. Turner-Evans, K. T. Jensen, S. Ali, T. Paterson, A. Sheridan, R. P. Ray, T. Wolff,
J. S. Lauritzen, G. M. Rubin, D. D. Bock and V. Jayaraman, The neuroanatomical ultrastructure and function of a biological ring attractor, Neuron 108(1), 145 (2020),
doi:https://doi.org/10.1016/j.neuron.2020.08.006.
[63] B. K. Hulse, H. Haberkern, R. Franconville, D. B. Turner-Evans, S.-y. Takemura, T. Wolff,
M. Noorman, M. Dreher, C. Dan, R. Parekh et al., A connectome of the drosophila central complex reveals network motifs suitable for flexible navigation and context-dependentaction selection, ELife 10, e66039 (2021).
[64] S. S. Kim, H. Rouault, S. Druckmann and V. Jayaraman,
Ring attractor
dynamics in the drosophila central brain,
Science 356(6340), 849 (2017),
doi:10.1126/science.aal4835.
[65] M. Noorman, B. K. Hulse, V. Jayaraman, S. Romani and A. M. Hermundstad, Accurate
angular integration with only a handful of neurons, bioRxiv pp. 2022–05 (2022).
[66] S. H. Strogatz, Nonlinear dynamics and chaos with student solutions manual: With applications to physics, biology, chemistry, and engineering, CRC press (2018).
[67] B. K. Hulse, A. Stanoev, D. B. Turner-Evans, J. Seelig and V. Jayaraman, A rotational
velocity estimate constructed through visuomotor competition updates the fly’s neural compass, bioRxiv pp. 2023–09 (2023).

48

SciPost Physics Lecture Notes

1414
1415

1416
1417
1418

1419
1420
1421

1422
1423
1424

1425
1426
1427

1428
1429

1430
1431

1432
1433
1434
1435

1436
1437
1438
1439

1440
1441
1442

1443
1444
1445

1446
1447

1448
1449
1450

1451
1452
1453

Submission[68] J. Laurens and D. E. Angelaki, The brain compass: a perspective on how self-motion
updates the head direction cell attractor, Neuron 97(2), 275 (2018).
[69] K. Zhang, Representation of spatial orientation by the intrinsic dynamics of the headdirection cell ensemble: a theory, Journal of Neuroscience 16(6), 2112 (1996),
doi:10.1523/JNEUROSCI.16-06-02112.1996.
[70] H. B.j., J. Omoto, P. Kandimalla, B. Nguyen, M. Keleş, N. Boyd, V. Hartenstein and
M. Frye, A visual pathway for skylight polarization processing in drosophila, eLife 10,e63225 (2021), doi:10.7554/eLife.63225.
[71] S. Heinze and S. M. Reppert,
Sun compass integration of skylightcues in migratory monarch butterflies,
Neuron 69(2), 345 (2011),
doi:https://doi.org/10.1016/j.neuron.2010.12.025.
[72] H. Vitzthum, M. Müller and U. Homberg, Neurons of the central complex of the locust
schistocerca gregaria are sensitive to polarized light, Journal of Neuroscience 22(3), 1114
(2002), doi:10.1523/JNEUROSCI.22-03-01114.2002.
[73] T. Okubo, P. Patella, I. D’Alessandro and R. Wilson, A neural network for wind-guided
compass navigation, Neuron 107, 924 (2020), doi:10.1016/j.neuron.2020.06.022.
[74] J. Seelig and V. Jayaraman, Feature detection and orientation tuning in the drosophila central complex, Nature 503, 262–266 (2013), doi:https://doi.org/10.1038/nature12601.
[75] J. J. Omoto, M. F. Keleş, B.-C. M. Nguyen, C. Bolanos, J. K. Lovick, M. A. Fryeand V. Hartenstein, Visual input to the drosophila central complex by developmentallyand functionally distinct neuronal populations, Current Biology 27(8), 1098 (2017),
doi:https://doi.org/10.1016/j.cub.2017.02.063.
[76] Y. Sun, A. Nern, R. Franconville, H. Dana, E. R. Schreiter, L. L. Looger, K. Svoboda, D. S. Kim, A. M. Hermundstad and V. Jayaraman, Neural signatures ofdynamic stimulus selection in drosophila, Nat Neurosci 20, 1104–1113 (2017),
doi:https://doi.org/10.1038/nn.4581.
[77] H. Shiozaki and H. Kazama, Parallel encoding of recent visual experience and selfmotion during navigation in drosophila,
Nat Neurosci 20, 1395–1403 (2017),
doi:https://doi.org/10.1038/nn.4628.
[78] D. W. Dong and J. J. Atick, Temporal decorrelation: a theory of lagged and nonlagged
responses in the lateral geniculate nucleus, Network: Computation in neural systems
6(2), 159 (1995).
[79] B. A. Olshausen and D. J. Field, Emergence of simple-cell receptive field properties bylearning a sparse code for natural images, Nature 381(6583), 607 (1996).
[80] A. J. Cope, C. Sabo, E. Vasilaki, A. B. Barron and J. A. R. Marshall, A computationalmodel of the integration of landmarks and motion in the insect central complex, PLOS
ONE 12, 1 (2017), doi:10.1371/journal.pone.0172325.
[81] J. Green and G. Maimon, Building a heading signal from anatomically defined neuron types in the drosophila central complex, Current Opinion in Neurobiology 52, 156(2018), doi:https://doi.org/10.1016/j.conb.2018.06.010, Systems Neuroscience.

49

SciPost Physics Lecture Notes

1454
1455
1456

1457
1458
1459

1460
1461

Submission[82] S. S. Kim, A. M. Hermundstad, S. Romani, L. F. Abbott and V. Jayaraman, Generation of
stable heading representations in diverse visual scenes, Nature 576(7785), 126 (2019),
doi:10.1038/s41586-019-1767-1.
[83] Y. E. Fisher, J. Lu, I. D’Alessandro and R. I. Wilson,
Sensorimotor experience
remaps visual input to a heading-direction network, Nature 576(7785), 121 (2019),
doi:10.1038/s41586-019-1772-4.
[84] Y. E. Fisher, M. Marquis, I. D’Alessandro and R. I. Wilson, Dopamine promotes headdirection plasticity during orienting movements, Nature 612(7939), 316 (2022).

1464[85] H. Haberkern, S. S. Chitnis, P. M. Hubbard, T. Goulet, A. M. Hermundstad and V. Jayaraman, Maintaining a stable head direction representation in naturalistic visual environments, bioRxiv (2022), doi:10.1101/2022.05.17.492284.

1465[86] T. Kohonen, The self-organizing map, Proceedings of the IEEE 78(9), 1464 (1990).

1462
1463

1466
1467
1468

1469
1470
1471
1472

[87] U. Hanesch, K.-F. Fischbach and M. Heisenberg, Neuronal architecture of the centralcomplex in drosophila melanogaster, Cell and Tissue Research 257(2), 343 (1989),
doi:10.1007/BF00261838.
[88] J. Isaacman-Beck, K. C. Paik, C. F. R. Wienecke, H. H. Yang, Y. E. Fisher, I. E. Wang,
I. G. Ishida, G. Maimon, R. I. Wilson and T. R. Clandinin, Sparc enables genetic manipulation of precise proportions of cells, Nature Neuroscience 23(9), 1168 (2020),
doi:10.1038/s41593-020-0668-9.

1475[89] R. Wolf and M. Heisenberg, Basic organization of operant behavior as revealed in
drosophila flight orientation, Journal of Comparative Physiology A 169(6), 699 (1991),
doi:10.1007/BF00194898.

1476[90] P. W. Tillberg, personal communication.

1473
1474

1477
1478
1479

1480
1481
1482

1483
1484

1485
1486
1487

1488
1489
1490

1491
1492

1493
1494
1495

[91] K. G. Götz, Course-control, metabolism and wing interference during ultralong tethered
flight in drosophila melanogaster, Journal of Experimental Biology 128(1), 35 (1987),
doi:10.1242/jeb.128.1.35.
[92] M. B. Reiser and M. H. Dickinson,
A modular display system for insect behavioral neuroscience,
Journal of Neuroscience Methods 167(2), 127 (2008),
doi:https://doi.org/10.1016/j.jneumeth.2007.07.019.
[93] C. Dan, B. K. Hulse, V. Jayaraman and A. M. Hermundstad, A neural circuit architecture
for rapid behavioral flexibility in goal-directed navigation, bioRxiv (2024).
[94] R. Wolf and M. Heisenberg, On the fine structure of yaw torque in visual flightorientation of drosophila melanogaster, J. Comp. Physiol. A 140, 69–80 (1980),
doi:https://doi.org/10.1007/BF00613749.
[95] F. T. Muijres, M. J. Elzinga, N. A. Iwasaki and M. H. Dickinson, Body saccades ofdrosophila consist of stereotyped banked turns, Journal of Experimental Biology 218,864 (2015), doi:doi: 10.1242/jeb.114280.
[96] E. Schrödinger, Zur theorie der fall-und steigversuche an teilchen mit brownscher bewegung, Physikalische Zeitschrift 16, 289 (1915).
[97] M. Smoluchowski, Notiz über die berechnung der brown’schen molekularbewegung bei derehrenhaft-millikan’schen versuchsanordnung, Pisma Mariana Smoluchowskiego 1(2),477 (1927).
50

SciPost Physics Lecture Notes

1496
1497
1498

1499
1500

1501
1502

1503
1504
1505

1506
1507

Submission[98] J. L. Folks and R. S. Chhikara, The inverse gaussian distribution and its statistical application—a review, Journal of the Royal Statistical Society Series B: Statistical Methodology
40(3), 263 (1978).
[99] A. Y. Ng, S. Russell et al., Algorithms for inverse reinforcement learning., In Icml, vol. 1,
p. 2 (2000).
[100] D. S. Touretzky, A. D. Redish and H. S. Wan, Neural representation of space Using sinusoidal arrays, Neural Computation 5(6), 869 (1993), doi:10.1162/neco.1993.5.6.869.
[101] R. Goulard, C. Buehlmann, J. E. Niven, P. Graham and B. Webb, A unified mechanism
for innate and learned visual landmark guidance in the insect central complex, PLoS
computational biology 17(9), e1009383 (2021).
[102] C. M. Bishop, Pattern Recognition and Machine Learning, Springer-Verlag, second edn.
(2006).

1510[103] M. Botvinick, S. Ritter, J. X. Wang, Z. Kurth-Nelson, C. Blundell and D. Hassabis,
Reinforcement learning, fast and slow, Trends in Cognitive Sciences 23(5) (2019),
doi:https://doi.org/10.1016/j.tics.2019.02.006.

1511[104] R. E. Bellman, Dynamic Programming, Princeton University Press (1957).

1508
1509

1512
1513

[105] R. J. Williams, Simple statistical gradient-following algorithms for connectionist reinforcement learning, Machine learning 8, 229 (1992).

51