Learning low-dimensional generalizable natural features from retina using a U-net
Authors: Siwei Wang
Benjamin Hoshal
Elizabeth A de Laittre
Olivier Marre
Michael Berry
Stephanie Palmer
Venue: NeurIPS 2022 Accept (2022)
Keywords: retina
efficient coding
convolutional neural network
synergy
natural scene

Abstract: Much of sensory neuroscience focuses on sensory features that are chosen by the experimenter because they are thought to be behaviorally relevant to the organism. However, it is not generally known what these features are in complex, natural scenes. This work focuses on using the retinal encoding of natural movies to determine the presumably behaviorally-relevant features that the brain represents. It is prohibitive to parameterize a natural movie and its respective retinal encoding fully. We use time within a natural movie as a proxy for the whole suite of features evolving across the scene. We then use a task-agnostic deep architecture, an encoder-decoder, to model the retinal encoding process and characterize its representation of ``time in the natural scene'' in a compressed latent space. In our end-to-end training, an encoder learns a compressed latent representation from a large population of salamander retinal ganglion cells responding to natural movies, while a decoder samples from this compressed latent space to generate the appropriate movie frame. By comparing latent representations of retinal activity from three movies, we find that the retina performs transfer learning to encode time: the precise, low-dimensional representation of time learned from one movie can be used to represent time in a different movie, with up to 17ms resolution. We then show that static textures and velocity features of a natural movie are synergistic. The retina simultaneously encodes both to establishes a generalizable, low-dimensional representation of time in the natural scene.

Learning low-dimensional generalizable natural features from retina using a U-net
Siwei Wang', Benjamin Hoshal', Elizabeth A de Laittre?, Olivier Marre", Michael J Berry II*, and Stephanie E Palmer'
Department of Organismal Biology and Anatomy, University of Chicago
CCommittee on Computational Neuroscience, University Chicago
Sorbonne Université, INSERM, CNRS, Institut de la Vision PPiinceton Neuroscience Institute, Princeton University
Abstract
Much of sensory neuroscience focuses on presenting stimuli that are chosen by the experimenter because they are parametric and easy sample and thought be behaviorally relevant to the organism. However, it is not generally known what these relevant features are in complex, natural scenes. This work focuses using the retinal encoding of natural movies to determine the presumably behaviorally- relevant features that the brain represents. is prohibitive to parameterize natural movie and its respective retinal encoding fully. We use time within a natural movie proxy for the whole suite of features evolving across the scene. We then use a task-agnostic deep architecture, an encoder-decoder, to model the retinal encoding process and characterize its representation of "time in the natural scene" in compressed latent space. In our end-to-end training, an encoder learns a compressed latent representation from large population of salamander retinal ganglion cells responding to natural movies, while decoder samples from this compressed latent space to generate the appropriate future movie frame. By comparing latent representations of retinal activity from three movies, find that the retina has a generalizable encoding for time in the natural scene: the precise, low-dimensional representation of time learned from one movie can be used to represent time in a different movie, with up to 17 ms resolution. that static textures and velocity features of natural movie synergistic. The retina simultancously encodes both to establishes a generalizable, low-dimensional representation of time in the natural scene.
then show
Introduction
The flexibility and computational power of convolutional neural networks (CNNs) has helped sensory
neuroscience model the neural code for natural stimuli with rich feature repertoires. It has been
shown that CNNs carryout encoding computations similar to those observed in the retina [1 [2] A
CNN also makes the inverse problem decoding complex stimuli from the retinal response 3 more
tractable. Understanding decoding has its own significant merit because neural systems downstream of the retina can only 'sce' the world through the retinal code. For any visual information to be used to guide behavior, it must first be decoded from retinal responses. Historically, the decoding problem for natural stimuli has been challenging because both the retinal response and the stimuli high-dimensional. Although deep neural networks capture the high dimensionality of neural inputs and responses [4), they do so by projecting the neural code into another high-dimensional parameter space that is also hard interpret. While these tools tell that we can decode, ability to understand the features of neural activity relevant for that decoding is limited. In this work, we
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

propose novel artificial neural architecture that can decode complex natural scenes from retinal responses with high fidelity while providing a -iw-deensional latent space that is interpretable. Using this architecture, we obtain unique insights into what features are important for reading out the retina's 8 population code, and why encoding these features might enable an animal to navigate a complex, dynamic natural environment.
We anchor our results on unique dataset from the salamander retina. Nearly one hundred output ganglion cells were recorded simultancously while several different natural movies were projected onto the photoreceptor layer. The relatively long (20 s) movie clips were played many times, in a pseudo-random order. During the lifespan of salamander, it goes through a transition from being aquatic to terrestrial. The sampled movies attempt to span these different motion environments. A movie of small fish in an aquarium with live plants was set up to match what a larval tiger salamander might see underwater while it hunts for food. A movie of leaves blowing the wind resembles the scene a salamander may live in after it undergoes metamorphosis. Does a salamander retina re-use how it encodes features during the aquatic larval phase to represent features in a terrestrial scene? This motivates us to investigate whether the encoding of natural features from one particular movie is generalizable to a novel movie.
Natural movies contain complex spatio-temporal features on multiple scales. This makes enumerating all possible stimulus states natural movies intractable. is more feasible to investigate how the retina encodes time in the natural scene, as has been done in other studies [5]. The salamander retina elicits precisely timed spikes 6 The idea that the retina may encode how features change over time to discriminate between frames has been explored before 77. It has an intricate connection to (imullu-ddepedent representational drift in sensory systems [8. Previous work [5] reported that a low dimensional compressed representation of activity from mouse V1 population can be used to discriminate frames that are Is apart. Furthermore, the authors showed that if such an encoding of time in the natural scenes exists, it is likely to be low-dimensional [5]. We train our deep neural network (an encoder-decoder in machine learning parlance) (Fig [ and Section 2 for decoding. It reconstructs movie frames from retinal responses. This is different from previous works 2 2 that use CNNs as "in-silico retinas" " to understand how specific stimuli drive the retina. During training, the encoder part of the deep neural network learns continuous, compressed latent representation of the retinal responses from which the decoder part samples to reconstruct target movie frames. We enforce structural constraints (9, 10 12 [3] to obtain meaningful, continuous latent space. We find that using this compressed representation, we can decode time in the natural scene up to fingle-famme resolution (17ms) (Fig This learned, compressed representation allows for precise decoding in a novel set of natural scenes. We show that the retina is responding to spatio-temporal features that change over time, rather than having clock. We divided these features into static (texture) and dynamic (optic flow-a vector field describing the motion between subsequent frames [14) motifs. This allows us construct two distributions which reflect how static features and dynamic features cluster to discriminate between frames. We find that these features are synergistic with respect the encoding. By simultaneously encoding static and dynamic features, the retina establishes a generalizable, low-dimensional representation of time in natural scenes.
2 Data and encoder-decoder architecture
Data: Our dataset contains retinal recordings of 93 cells responding to repeated, 20 second pre- sentations of three natural movies at 60 frames per second. There are 85-90 presentations of each movie interleaved pseudo-random order. Spikes are binned at 17ms, to align with the movie frame duration. We compute each neuron's firing rate as a function of time within each movie, (the peri-stimulus time histogram, or PSTH) by averaging spikes across trials in these bins]
Encoder-decoder architecture: We use a U-net [15 as the backbone architecture. The U-net supplements an encoder of contracting layers with nearly symmetric decoder of expansive layers (hence the U-shape, see Supplementary Information) [6] 177. is successful in domain-conversion problems, i.e., text-to-speech [8] We modify its skip connections to supply the decoder noisy features from the encoder. Because the intermediate features from the feedforward encoder have different resolutions at their convolutional layers, these skip connections enable the decoder to form a multi-scale and multi-level feature representation of the input. Our particular network has an
You can find our Pytorch implementation at htthtttp:///tttti wtt.//ooo/////eeee comm/spaammerr/U-nnet
2

---

A
response
training
encoder
Z
decoder
past
trannd encoder
response
4.0
generalization
M
(Z nos))
dimensions
0.5
0.0
(D(S)TH)
MAA
(Z water)
2.5
(Z leaf)
2.0
8
latent dimensions
Figure A) Encoder-decoder network trained to predict movie frame 100ms the future from a 500ms window of retinal activity in the past (the aggregated receptive field of the retina population is shown in the heat map from the left). The network learns low dimensional variational representation, where and characterize the posterior distribution p(z]2). The samples drawn from this latent space referred
Z in subsequent sections. All skip connections are trained obtain separate latent space for Z. Using a highly expressive encoder [22), ie., ResNet18, we empirically observe that the latent space learned for all skip connections are similar. B) We train the network on one movie (fish, for example), then the encoder weights are frozen. We obtain the Z's for retinal activity responding to different movies by passing test samples from all three movies through the trained encoder. C) The intrinsic dimensionality of the retinal activity (dashed line) and latent dimensions that intrinsic dimension measures the complexity of the retinal activity. We use it as a lower bound to constrain dim Z and add additional latent dimensions to improve reconstruction of target movie frames. We stop at dim Z 10 because observe highly accurate reconstruction with dim Z = 10 empirically.
latent space with varying dim Z. Note
encoder with the same architecture as ResNet18. We also initialize with the ResNet18 19 weights pre-trained by ImageNet [20). Our decoder mirrors the feedforward architecture of the encoder (see Supplementary Information). In addition, we turn these skip connections into variational sampling layers and use them as compressed representations of the input retinal activity. The set of activation values in this latent space in response to the neural activity (input) from a given stimulus movie is referred to as moovie. where 'movie' is either 'fish', 'water', or 'leaf"). We constrain all variational sampling layers to have the same dimensionality, to simplify the training.
Knowing that the decoder reconstructs the movie frame from features across multiple spatial and temporal scales, we use the perceptual loss (21) as the objective function for the encoder-decoder. This loss function compares a reconstructed frame with its target frame with respect learned features from a pretrained VGG, as opposed to their raw pixels. Throughout the paper, we use features from the pretrained VGG19 represent movie frames.
Results
3.1 Retinal activity has low intrinsic dimension
Fig TA shows how we train an encoder-decoder for a specific movie (in this example the fish movie). The input is the retinal PSTHs from the 45 most reliably spiking cells in the population (see Supplementary Information). We take trial-averaged firing rates, and ignore "noise" correlations between cells, to make an initial pass at this high-dimensional problem. Considering that many of the observed retinal computations happen on a timescale below 400ms (23], we restrict ourselves to 500 ms long snippets of the PSTHs. To encourage the model to learn temporal structure within the movie, we ask the decoder to reconstruct a movie frame 100ms in the future, after the end of the 500ms snippet of neural response. We choose this particular A/ based on the timescale of predictive information in retinal populations (2 We train one predictive encoder-decoder for specific movie using 40,000 training samples (see Supplementary Information for details) with 90%/10% training/ valiia//aaliatiion split. The reconstruction is from additional held-out test set of 10,000
3

---

samples (100 frames. 100 PSTH patterns in each frame, see Supplementary Information). We also train a static encoder-decoder that learns to reconstruct a frame centered within the 500ms window of neural activity (using 250ms before and after the target frame as the input, similar to R. The predictive encoder-decoder achieves a reconstruction performance similar to the static one. We focus on the predictive encoder-decoder in our subsequent analysis because it may capture both static and temporal structures of a natural movie by design.
The latent space is a compressed representation, Z. of the mean firing rate patterns from the retinal population. We estimate the intrinsic dimension [25] 26 of the retinal activity and use it to guide the selection of the dimensionality of the latent space. This intrinsic dimension is the number of
variables needed to describe a data distribution [27]. It is also a complexity measure of data because it determines the number of samples needed to characterize a data manifold [28 29]. We find that
our estimate of the intrinsic dimensionality for all retinal activity in response to the three movies yields the same result as the estimate of the intrinsic dimensionality from the activity for each movie separately, i.e., including retinal activity from a different movie does not add complexity to the latent space representation of the retinal response. This suggests that features in natural movies may be encoded by the retina in a generalizable way across movies. We use the intrinsic dimension of the retinal activity lower bound to determine latent dimension, i.e., dim Z in the encoder-decoder. We then add additional latent dimensions to help encode factors that may result from combinations of different intrinsic dimensions. We observe that the increase in the estimated intrinsic dimension of the latent space decreases after dim Z= > 4. Mcanwhile, we empirically observe good reconstruction performance (the pixel MSE is about 0.02 averaged over 100 test frames of size 64X64 with pixel intensity E [0, 255)) when we use latent dimension equal to 10 to reconstruct the held-out movie segments (see Supplementary Information). Thus, for further analyses, we use dim Z = 10 unless otherwise stated. To address questions about whether features learned from retinal activity responding to one movie are generalizable another novel movie, we generate "mismatched" Z's. These "mismatched" Z's are compressed representations of the retinal responses to Zwater and Zieaf movies by passing those inputs through the encoder trained for the Zissh movie (Fig IB). The results presented below are similar regardless of which movie (water, leaf, fish) we train on and which other two movies are used to generate mismatched Z's.
3.2 The retina encodes complex, but interpretable spatio-temporal features in natural movies
The goal in this section is qualitatively assess latent space stimulus features to show that the encoder-decodee contains features that are plausibly encoded by the retina. We visualize highly activated features from decoding layers of multiple spatial scales (threshold by top 1% activation intensity, see Supplementary Information for more visualization). We find both features that resemble background motion and features that resemble object motion (Fig 2) in a specific decoding layer. In particular, the object motion feature closely traces the movement of the fish in the target movie segment. The average activation of this feature along the x-axis shows that this feature may be responsible to decode both position and velocity of the fish movement in the target movie frames (top row). Because removing the latent activations silences both background and motion features, we also determined that this specific variational sampling layer generates these disentangled features. This observation agrees with previous experiments that the retina can encode features that evolve across time and space [30]. This motivates to investigate whether can decode the "time in the natural movic", i.c., discriminate between different frames from this specific decoding layer.
3.3 Low dimensional, generalizable representation of time in multiple movies
In this section, we investigate whether we can discriminate frames in natural movics and thus decode time in natural scenes. We can also ask whether the feature space of the retinal population used for one movie can be used for the other two natural movies in our three-movie dataset. If this is the case, it would suggest that there is general representation of spatio-temporal features in retinal activity that supports this decoding of time.
Inspired by previous work that the visual cortex may contain a low-dimensions representation of complex stimulus features evolving over time (5, we decode time in the natural scene with the particular latent space that corresponds to the decoding layer shown in Fig 2 We obtain the compressed representation of retinal activity on the held-out test frames for the fish movie, i.e., fish. as well as test frames for two other novel movies (c.g., leaf and water). This use of an encoder-

---

Frames
Background
fish/motion
2
X Position
Velocity
100 ms
Figure 2: Features from decoder layer separately decode background and object motion. 1st (from top bottom): target movie frames that are 100ms apart; 2nd row: an example feature inside the trained decoder and its activation for decoding frames in the top row. 3rd row: the spatial, temporal activation. 4th row: X-T receptive field (aggregated x-axis activation over time).
decoder to generate a compressed representation a data distribution has been investigated in detail in representation learning (9]. For each fishtteaf. water separately, we linearly decode the 1D frame label of held-out test frames from the corresponding Z. Fig 3A shows all decoding performance as a function of the number of dimensions allowed in Z (we trained series of encoder-decoders with different dim Z). Because all encoder-decoder models are trained with the fish movie, the decoding for the fish movie outperforms the other two. However, we observe that dim Z is sufficient for the incoder-decoder to build a latent space Z which can decode >80% frames in all three movies. This is also the dimensionality needed for decoding in the natural scene reported in mouse visual cortex (5). Since these frames are 17ms apart, this decoding performance shows that salamander retina establishes a low dimensional representation of "time in the natural scene" with fine temporal resolution.
The high decoding performance from the mismatched Z's suggests that retinal activity may establish a general encoding for "time in the natural scene". How is this possible? The retina is presumably encoding general space-time features that are predictive of the future space-time features in natural scenes. These could be complex, but they seem to generalize across movies. We use information theory directly evaluate the information that all three Zs contain that is relevant for decoding time. We first calculate the mutual information between latent representations of retinal activity responding different movies Z show the mutual information of fish vs. leaf [2 Zieas) and fish vs. water, [[2 Zwater)), respectively). We also observed that fiiji Zieaf lime) fish fish; Zteaf) ), i.e., including time does not add additional mutual information. This tells us the latent representation obtained from retinal activity for one movie encodes the generalizable "time in natural scenes" for a different movie, up to the full entropy of time itself. To show this, we use the chain rule of mutual information and subtract mutual information that is independent of time, i.e., I (Zyiihhi fish Ziea [time) ((shi Zrea f, time)- [Zieaa tême)= = I f(Zhh Zieaf) )- T( I(Zteaf; time), in Fig(4A The cyan bar shows the MI between fish and mismatched Z that about time, TiZ Zieaf) - [(Z Zieaf [time). It nearly saturates H (time). We also observe that the mutual information between different movies is mostly about time. The difference between the pink and cyan bar very small. This estimate confirms that the retina has generalizable, precise representation of time in natural scenes that can discriminate consecutive frames that are only 17ms apart. Table[[ that latent representations obtained from retinal responses to any one movie can be used to decode time for all three movies.
the learned compressed retinal responses to the fish movie, we
5

---

A
B
100
Hitime)
performanece (%)
75
Mutual info (bits)
50
movie
25
fish leaf
2
8
dim(Z)
leaf
water
Figure 3: A) Decoding performance of latent representations on test movie segments for all three movies. The encoder-decoder is trained in fish movie. The latent representations are obtained from retinal activity of all three movies (fish, leaf, water). B) Mutual information between movies mutual information with respect time. (time) = 6.6 bits for the test movie clip of 100 frames long. Because we use the encoder-decoder trained with fish movie here, we show [ (Z Zieaf) and I (Z Zwater). (See the supplementary information)
Water(5d) 78.2%
Water( (kd)
Leaf(5d)
Leaf 10d)
Fish
96.9%
72.2%
98.0%
Leaf
79.5%
97.8%
84.4%
99.1% 1
Water
84.0%
97.4%
71.7%
97.9%
Table I: Latent representations trained on any one movie can decode time in all three movies. Here we show dim Z = 5 and dim Z = 10, respectively.
In the supplementary information, we also included simple visualizations and decoders to demonstrate that decoding "time in the natural scene" is challenging. First, "time in the natural scene" cannot be observed as simple visual trend using the mean PSTH, pairwise frame-to- to-frame distance, or the 2D latent space trace. Second, we also included linear decoders trained on instantaneous PSTH'ss raw PSTH's, shuffled PSTH's and three other dimensionality reductions (Isomap, 10D-PCA and 50D-PCA). We found that shuffled PSTH shows inferior performance compared raw PSTH's. This suggests that decoding time does not come from trivial gross changes in spiking statistics. We also observed that linear decoders fall short of the 10-D latent representation Iearned by the U-net in terms of generalizable performance, even when using the 50D-PCA. To learn low dimensional feature space that can be applied to all three natural movies, our variational U-net carries out significant nonlinear transformation.
one trained on
A
B
4D (acc 68.3%)
464ms
944ms
Decode
True
Figure 4: A) The seatter plot of decoding performance with Zycah of dim 4. Correctly decoded samples appear along diagonal line of true vs. pred (decode), incorrectly decoded samples show up in off-diagonal regions. With dim Z > 4, the decoding performance increases to ~ 95% (shown in Fig B) Two examples of decoding errors (within the boxes shown in A). The two frames in the red box contain similar optic flow (dynamic) whereas the two frames in the orange box contain similar static textures.

---

3.4 Synergistic features for encoding "time in the natural scene"
Fig shows that the retina has a low-dimensional, generalizable representation for time in the natural scene. We next ask what features the retina uses for this low-dimensional, generalizable representation.
Although there are retinal circuits that encode object motion, most decoding work only uses natural images as their input stimuli motion might interact. In Fig we show two example decoding errors (B)) one where the predicted and true frame have similar static features and one other where the predicted and true frames with similar optic flow (dynamic features). This indicates that our observed latent representation of "time in the natural scene" " can be confused when either the static or dynamic structure between frames is similar. These examples are not sufficient to exclude the possibility that static textures may also be used discriminate between different dynamic frames. To investigate this, we perform two parallel hierarchical clusterings: one on the frames themselves (static) and one the optic flow frames (dynamic) of the test movie segment. If textures govern the discrimination between frames in both dynamic and static settings, then these clusterings should produce similar clusters. Before clustering, we convert all static frames and their corresponding optic flow frames to features that are the activation from the last ReLu layer in pre-traind VGG19 network [32]. These activations
Our stimulus set allows us to explore how natural texture and
believed to mimic features used in human perception of generic natural stimuli [21). We observe that hierarchical clusterings on dynamic versus static motifs of all three movies yield different results (see
Supplementary Information for details).
The difference between clusters of frames based static versus dynamic features enables us to construct three distributions of "time in the natural scene" (Fig 5A). One uses the clustering based on the static features Y Ystatic. another using the dynamic features Y dynammic and third combining both sets of features Yooint- By construction, we would like the joint distribution Yjoin to have an entropy as close as possible to the full entropy of time H (time) = log 26100)=6.6 bits. This construction narrows down our search of coding schemes to how the retina combines dynamic and static structures within natural movies to encode the "time in the natural scene".
Previous work showed that the retina performs efficient coding redundancy should be minimized among different features of interest. Therefore, while we would like the joint distribution ((joiin)) to contain most of the entropy of time, we also want to mínimize the mutual information between the two components Ystatie and Ydgnamic (minimize their redundancy). We varied the threshold on the clustering hicrarchies to coarse grain the distributions Ystatic and Y Ydynomies and also to create the joint distribution Yjoint. Because we do not know a priori their relative contributions, we coarse-gral -rain both clusterings with thresholds such that the coarse grained E(ddynamic)) In Fig 4A, show
R3 Efficient coding predicts that
entropy of the two components the coarse- coarse-grained joint distribution we use for subsequent analysis. It contains a small amount of redundancy, i.e., I(Y dynamies Y stattie) 0.8 bits, while Y, Joint includes most of the entropy for time (H (Ynnine) = 5.05 bits = 76% of (lime) = 6.6 bits). These two components are dominated by either static or dynamic features, respectively. In Supplementary Information, we discuss in detail how we find these distributions. A different threshold may either introduce a significantly higher redundancy or sacrifice too much information from H (time).
comparable, i.e., E(Ystatic)
Using these distributions, now we can ask how the neural population encodes time through encoding the both the static and dynamic features of the natural scene. The latent activation of retinal inputs are compressed representations, so we frame this encoding problem using the information bottleneck method (35). The information bottleneck method identifies whether compressed representation retains as much information as possible about the relevant variable Y while compressing away irrelevant components of the input X. In this context, the information bottleneck shows how much information a compressed representation needs, in order to encode a specific amount of information about the features of interest, Y.
The information bottleneck method minimizes the following objective function:
(p((x)) = T(X;T) - TT;
To make meaningful measurement of T(X;T) and 7(Y; 7), we first ensure that we have meaning- ful latent representation Z. The challenge is to prevent the so-called "posterior collapse" [[]. This is phenomenon previously reported in eecodersdecoders with highly expressive architectures (like the
7

---

ResNet18 network that we use here) [22]. These expressive architectures are capable of decoding complex features, e.g., our movie frames, without using Z. This results in the latent code Z only containing noise, i.e. T( T(X; 2) O. Here, we use simple heuristic to circumvent this scenario. As discussed in [36 12 37 we can obtain a meaningful Z by making the posterior have a small but nonzero noise. To be specific, we have o² > 0 for p(X 2). This is also dubbed 'committed rate' for the encoder. our case, we choose log o² = 1 1.0 to further ensure numerical stability of the mutual information estimator that we use [38].
The mutual information estimator also requires the prior of p(Z) to have factorized marginal (each marginal is an independent Gaussian), .N (0, I) (they are independent Gaussians). This is a typical constraint introduced in the original variational autoencoder [13]. Combining this constraint on p(Z) and the above constraint on p(X Z), we can approximate I [Xx; Z) with the following estimator,
I(X: H() (Z
V log Σ
exp( 1 1 - 2, 1
D
log + log 2x)
p
j 2
0²
2
P is the number of test samples (P- 10000 in our case) and 2i, Bj are the latent activations for the in or jih sample.
Because P(Y) has uniform distribution (100 retinal inputs per frame and there are 100 held-out frames), we can use the following estimator for 1(Y; 2) (we also validated this estimation with widely used non- can-aaaametic estimator [39], the difference is less than 0.1 in all our calculations):
2) = H - H (
Σ
1 log
Σ
1 ||
-
2
< p
exp(
p
2
g²
Σ Pt
log
1
Σ
exp(
1
1|
23
+YY!=! P
2
J.Y,=
Note that pr is the number of test samples for the 4n frame. pl = 100 in all of the test distributions.
Using Z)) with factorized Gaussian marginal brings an additional benefit: it is the sufficient
and necessary condition for the latent space to exhibit orthogonal symmetry [40 41 422. This
Harmois-Skitovin characterization was first introduced to identify unique factors for independent
component analysis 44). Recent work also showed that this factorized marginal Z is necessary
to capture "ground truth factors of variation" [10 in the latent representation Z
45 12].
In Fig 5B we show the information plots for time, dynamic features, static features and the joint ((Y dynamie: Ystatic)) features. X is the retinal activity. Y are features of interest, which are time, ddmomiee Ystatic and Yoonn. shown in different colors. Z are the latent activations of series of encoder-decoders with different dim Z. Because the encoding of time reaches its full entropy of H (time) (specifically, we observe this saturation at dim Z 5), this shows that the compressed representations Z from the pncoder-decoded are near-optimal. In the information bottleneck technique, the best any representation can do is encode the full entropy of the relevant variable Y, i.e., here H(Y) H (Lime). This is similar to the near-optimality shown in previous work in the variational information bottleneck [46 38 In addition, we use many Z's s spanning a range of dimensions to construct these information curves. We observe that all of these compressed representations encode comparable amounts of information about both dynamic and static features. This suggests that the dynamic features are prominent as the static features in the retinal population's 's internal representation of complex natural scenes.
We highlight the benefit of simultaneous encoding of dynamic and static features in Fig SC By comparing the information about Yjoint = {Ystaties Y Ydymamie) } and the sum of the information about these two components, we observe that there is synergy = [Y joim By construction, the presence of synergistic information corresponds to latent representations with lower dimensions. This suggests why the retina can compress its encoding of both static and dynamic features. When encodes both features simultaneously, the synergy between these features helps the retina to represent the full entropy of time itself in fewer latent dimensions.
(Yayuamiic)) [Yfoamm Z)-(I(Yssui)) sumii +)+ >>+I(Y dynamic):; ); Z) )
8

---

A
B
Hitime)
8
Dynamic+Slalic
H(jint)
2008
3:06
Dynamic
Synaro
2,00
Static
2.02
Static
3.00
5.0
7.5
10.0
25
5.0
10.0
12.5
(0K:2)/bt))
10K:2/bits)
Figure 5: A: Joint distribution of static and dynamic features. This joint distribution includes 76% % of the entropy of time. Note that the mutual information between coarse-grained static and dynamic distributions is about 0.8 bits. Given that the entropy of statie/dynamic features is around 2.8-2.9 bits, the amount of mutual information between them is relatively small. B: the information plane for fish data. Dark blue: the information curve for encoding time; Light blue: the information curve for encoding the joint distribution combining static and dynamic features; Red/Green: information curves for separated static (red) and dynamic (green) features. See Supplementary Information for information curves of the other two movies C: Blue: the information curve for encoding the joint distribution, the same B; Black: the sum of information curves from (edemiiceee+-SSttttrrreen There is synergistic region between the information curve for the joint and the sum.
4 Discussion
This work uses a U-net-based deep learning architecture to reverse engineer a retinal encoding process for complex natural movies. Using the PSTHs of a large salamander retinal population, we identify stereotypical features that are generalizable across multiple natural movies. We find that the retina uses transferable, low dimensional representation encode rich of natural space-time features. The encoding obtained from one movie can be used to decode "time in the natural scene" for different movie. despite differences their particular spatio-tempore structures. We also discover that the retina encodes time through synergistic coding of both dynamic and static features.
Here, only observed synergy within the feature space (using mean firing rates of retinal activity, we assume all cells are independent). We also decoded time in its simplest form by asking how well discriminate between different frames. In future work, we would like extend
analysis to temporal structure with proper predictive constraints, i.e., predicting future at longer A/ should be more challenging than predicting smaller AL [47 244] We also aware that the synergy here is different from what can be observed between cells in the neural data. The synergy in the neural code may combine synergy in the feature space with synergy in the population code, itself [48 49
Our work is most similar to [50 51 when compared to other methods that also identify a latent representation between brain activity and external stimuli. They used multilayer perceptron (MLP), a highly expressive feedforward encoder. MLP is fully-connected, so that its learned latent representation corresponds
single global scale. Our U-net architecture, in contrast to the MLP, employs a ResNet as the encoder. The ResNet encoder attains the same performance as the MLP, but by cascading Resblocks from coarse-to-fine scales. This makes it possible for the U-net architecture to simultaneously learn compressed latent representation at various scales. Although we did not specifically explore this feature, it might be relevant for future research on understanding brain dynamics in flexible natural environments. For example, there is a hierarchy of timescales both in natural scenes and output natural behaviors, ranging from hundreds of milliseconds to minutes (whisking to walking to making action plans [52] (3 With additional constraints [54], These variational sampling layers may learn hicrarchically distinct latent representations for each timescale individually and comprehend how they might be coupled to create complicated behavioral outputs. Outside of neuroscience, This U-net is compatible learn latent representations between other temporal sequences (e.g., text) and complex apatio-temporal signals (speech or video). Text-to- speech and video summarization are two possible applications. Combining latent representation at multiple scales may also reveal semantic relationships between complex features in general object recognition, e.g, how does a model combine local features (nose, eye) with global shape (e.g., body size) to discriminate between cats and dogs.

---

Our work shows that the retina leverages feature representations that are common across natural movies. This knowledge transfer differs from what is referred as "transfer learning" in computer vision and machine learning. In computer vision, transfer learning refers to training a model with a much more complicated dataset (e.g. ImageNet with 1000 classes) and performing inference on novel, but much smaller dataset (e.g. CIFAR 1A/1000000001 or CelebA). Transfer learning presupposes that models trained on complex datasets contain sufficient variation to allow the learned features to be reused on new datasets. For the retina, evolutionary timescales underlie the "training from complex dataset" stage. The retina shaped in such a way that behaviorally significant components of all natural imputs in organism's 's ecological niche are selectively encoded. This enables our training on one movie/retinal response dataset to reveal features transferable to another movie of a similar complexity scale. Future studies may enable to determine if such generalizable feature representation is innate (sculpted only by evolution) or whether visual experience within a lifetime may refine it. This would depend our ability track changes in visual processing beyond the retina (e.g. cortex) over the course of an animal's life (similar to fine tuning in the transfer learning domain).
Acknowledgments and Disclosure of Funding
This work was supported by the National Institutes of Health BRAIN-ROI EB026943, the National Science Foundation (through the Center for the Physics of Biological Function PHY-1734030 and Clustering of Neural Activity: Design Principle for Population Codes PHY-180699 1806932) and An ERC CoG grant (grant agreement 101045253) and Grants from the ANR (DECORE, ShootingStar). This work was also based upon work supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. (DGE--776045)) 146045). Any opinions, findings, and conclusions or recommendations expressed in this material are those the author(s) and do not necessarily reflect the views of the National Science Foundation.
References
[1] Lane McIntosh, Niru Maheswaranathan, Aran Nayebi, Surya Ganguli, and Stephen Baccus. Deep learning models of the retinal response natural scenes. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
[2] Hidenori Tanaka, Aran Nayebi, Niru Maheswaranathan, Lane McIntosh, Stephen Baccus, and Surya Ganguli. From deep learning mechanistic understanding in neuroscience: the structure of retinal prediction. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, Buc, F. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[3] Vicente Botella-Soler, Stéphane Deny, Georg Martius, Olivier Marre, and Gašper Tkačik. Nonlinear decoding of complex movie from the mammalian retina. PLOS Computational Biology, 1405)=1106055 may 2018
[4] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521:436 May 2015.
[5] Xia, Tyler D. Marks, Michael J. Goard, and Ralf Wessel. Stable representation of a naturalistic movie emerges from episodic activity with gain variability. Nature Communications, 12(1), aug 2021.
[6] Michael J. Berry, David K. Warland, and Markus Meister. The structure and precision of retinal spike trains. Proceedings of the National Academy of Sciences, (1100))511-54161 may 1997.
[7] Thierry Mora Olivier Marre David Schwab, Stephamie Palmer: Decoding and encoding retinal ganglion cell responses with deep neural networks. NeurIPS2016 workshop: Brains and Bits.
[8] Tyler D. Marks and Michael Goard. Stimulus-dependent representational drift in primary visual cortex. Nature communicationss, 12:5169, August 2021.
10

---

[9] Geoffrey E Hinton and Richard Zemel. Autoencoders, minimum description length and helmholtz free energy. In J. Cowan, G. Tesauro, and Alspector, editors, Advances in Neural Information Processing Systems, volume 6. Korgan-Kaufmann, 1993.
[10] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35:1798- 15:1788-1828, August 2013.
[11] Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from continuous space. SIGNLL Conference on Computational Natural Language Learning (CONLL)), 2016, November 2015.
[12] Abhishek Kumar and Ben Poole. htpps///rrivvorr//a////////// 1.1259,, January 2020.
On implicit regularization in -vaes.
[13] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. December 2013.
[14] Berthold K.P. Horn and Brian G. Schunck. Determining optical flow. Artificial Intelligence, 17(1-3):185-203 aug 1981.
[15] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. May 2015.
[16] Linjing Fang, Fred Monroe, Sammy Weiser Novak, Lyndsey Kirk, Cara R. Schiavon, Scungy- oon Yu, Tong Zhang, Melissa Wu, Kyle Kastner, Alaa Abdel Latif, Zijun Lin, Andrew Shaw, Yoshiyuki Kubota, John Mendenhall, Zhao Zhang, Gulcin Pekkurnaz, Kristen Harris, Jeremy Howard, and Uri Manor. Deep learning- carninggbaseed based point-scanning super-resolution imaging. Nature methods, 18:406-416, April 2021.
[17] Jeremy Howard al. fastai. https: fttps://githuut com/fastai/fastai 2018.
[18] Rui Li, Dong Pu, Minnie Huang, and Bill Huang. Unet-tts: Improving unseen speaker and style transfer in one-shot voice cloning. ICASSP2022, Septemher 2021.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, jun 2016.
[20] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25, Curran Associates, Inc., 2012.
[21] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. European Conference Computer Vision (ECCV), page 694 711, March 2016.
[22] Hongzhou Lin and Stefanie Jegelka. Resnet with one-neuron hidden layers is universal approximator. June 2018
[23] Stephen A. Baccus and Markus Meister. Fast and slow contrast adaptation in retinal circuitry. Neuron, 36:909-919 919,, December 2002.
[24] Stephanie E. Palmer, Olivier Marre, Michael J. Berry, and William Bialek. Predictive informa- tion in sensory population. Proceedings of the National Academy of Sciences, 112(22):608- 6913, may 2015.
[25] Phillip Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact on learning. April 2021.
[26] David J.C. MacKay and Zoubin Ghahramani. Comments 'maximum like- lihood estimation of intrinsic dimension' by e. levina and p. bickel (2004).

---

[27] Elizaveta Levina and Peter Bickel. Maximum likelihood estimation of intrinsic dimension. In L. Saul, Y. Weiss, and L. Bottou, editors, Advances in Neural Information Processing Systems, volume 17. MIT Press, 2004.
[28]
Hariharan Narayanan and Partha Niyogi. On the sample complexity of learning smooth cuts on a manifold. COLT, 2009.
[29] Hariharan Narayanan and Sanjoy Mitter. Sample complexity of testing the manifold hypothesis.
In J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta, editors, Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc., 2010.
[30] Norma Krystyna Kühn and Tim Gollisch. Joint encoding of object motion and motion direction in the salamander retina. The Journal of neuroscience: the official journal of the Society for Neuroscience, 36: 12203-12216, November 2016.
[31] Nora Brackbill, Colleen Rhoades, Alexandra Kling, Nishal Shah, Alexander Sher, Alan M Litke, and FJ Chichilnisky. Reconstruction of natural images from responses of primate retinal ganglion cells. el.ife, 9, nov 2020.
[32] Matthias Kummerer, Thomas S.A. Wallis, Leon A. Gatys, and Matthias Bethge. Understanding low- and high-level contributions to fixation prediction. In 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, oct 2017.
[33] Joseph J. Atick and A. Norman Redlich. What does the retina know about natural scenes? 4:196-210, 1992.
[34] H. B. Barlow. Possible principles underlying the transformations of sensory messages. MIT Press, 1961.
[35] William Bialek and Naftali Tishby. Predictive information. February 1999.
[36] Ali Razavi, Aäron van den Oord, Ben Poole, and Oriol Vinyals. Preventing posterior collapse with delta-vaes. January 2019.
[37] Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem. Disentangling factors of variation using few labels. Eighth International Conference on Learning Representations ICLR 2020, May 2019.
[38] Artemy Kolchinsky, Brendan D. Tracey, and David H. Wolpert. Nonlinear information bottle- neck. May 2017.
[39] Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical review. E, Statistical, nonlinear, and soft matter physics, 69:066138, June 2004.
[40]
V.Skitovitch. On property of the normal distribution. DAN SSSR, 89:217-219, 1953.
[41] G. Darmois. Analyse générale des liaisons stochastiques: etude particulière de l'analyse factorielle linéaire. Revue de l'Institut International de Statistique Review of the International Statistical Institute, (1/2)): 1953.
[42] Eugene Lukacs and Edgar King. A property of the normal distribution. The Annals of Mathe- matical Statistics, 25(2)3389 -944, 1954.
[43] Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural Networks, 12(3):429 439, apr 1999.
[44] Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of Causal Inference: Foundations and Learning Algorithms. Adaptive Computation and Machine Learning. MIT Press, Cambridge, MA, 2017.
[45] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a con- strained variational framework. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Truck Proceedings. OpenReview.net, 2017.
12

---

[46] Alexander A. Alemi, lan Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational infor- mation bottleneck. Proceedings of the International Conference on Learning Representations (ICLR) 2017, December 2016.
[47] Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method.
April 2000.
[48] Elad Schneidman, William Bialek, and Michael J. Berry. Synergy, redundancy, and indepen- dence in population codes. The Journal of neuroscience the official journal of the Society for Neuroscience, 23: 15:11539-1155: December 2003.
[49] Peter Latham and Sheila Nirenberg. Synergy, redundancy, and independence in popula- tion codes, revisited. The Journal of neuroscience : the official journal of the Society for Neuroscience, 25:5195-5206, May 2005.
[50] Ran Liu, Mehdi Azabou, Max Dabagia, Chi-Heng Lin, Mohammad Gheshlaghi Azar, Keith B. Hengen, Michal Valko, and Eva L. Dyer. Drop, swap, and generate: A self-supervised approach for generating neural activity. Annual Conference on Neural Information Processing Systems (NeurIPS), November 2021.
[51] Ding Zhou and Xue-Xin Wei. Learning identifiable and interpretable latent models of high- dimensional neural activity using pi-vae. NeurlPS 2020, November 2020.
[52] Stefano Recanatesi, Ulises Pereira-Obilinove Obilinovie, Masayoshi Murakami, Zachary Mainen, and Luca Mazzucato. Metastable attractors explain the variable timing of stable behavioral action sequences. Neuron, 110:139-153.e9 January 2022.
[53] Merav Stern, Nicolae Istrate, and Luca Mazzucato. A reservoir of timescales in random neural networks, oct 2021.
[54] Ilyes Khemakhem, Diederik P. Kingma, Ricardo Pio Monti, and Aapo Hyvärinen. Variational autoencoders and nonlinear ica: A unifying framework. Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, pages 2207-2217, year 2020, July 2019.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] see Discussion
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (cither in the supplemental material or as a URL)? [No] The data will be published in an experimental paper in the near term. we will release our code/data by then (or by the camera ready deadline if our submission is successful)
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] see Supplementary Information
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] see Supplementary Information
4. If you using existing assets (c.g., code, data, models) or curating/releasing assets...
13

---

(a) If your work uses existing assets, did you cite the creators? [Yes] We cite the creators in the Result section 3.1
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're
using/curating? [N/A] People who created the data are included co-authors.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals. if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "W4thjAcks8Z", "reviewer": "Reviewer_VKvw", "summary": "In their paper \"Learning low-dimensional generalizable natural features from retina using a U-net\", the authors modify a U-net with a ResNet encoder and variational sampling layers for skip connections to identify salamander retinal population response space-time features. They do this by reconstructing movie frames from neural responses. Moreover, they analyze the latent representations and find that static features and dynamic features are encoded synergistically.", "questions": "Do the authors see other potential applications of their U-net modifications outside of neuroscience?\n\nFigure 2 caption: Bottom row appears twice. The authors should properly refer to all four rows.\n\nFigure 3: The authors trained on the fish movie and tested on the other two movies. Would the results be similar if training on the leaf or water movie and testing on the other two movies?\n\nLine 248: Network misspelled.\n\nEq. 2: Please add brackets for clarity. Moreover, there is no explanation that this approximation comes from an independent Gaussian assumption. Please clarify the text.\n\nLine 270: Subscript \"joint\" broken.", "limitations": "The authors do not discuss any negative societal impacts of their work. In my opinion, this is adequate in this case.\n\nThere is an independence assumption, which is very briefly discussed in the Discussion. The authors also discuss the lack of proper predictive constraints and the particular type of synergy that they looked at. Overall, limitations have been appropriately discussed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The modifications of U-net in terms of the ResNet encoder and variational sampling layers for skip connections is an interesting innovation that might have broader applications and could be of interest to the wider machine learning community. However, the authors do not discuss other potential applications of their methodology and instead just mention that the architecture can learn many compressed latent representations simultaneously.\n\nTo the best of my knowledge, the finding that static features and dynamic features are encoded synergistically in the retina is novel. Given that this finding helps to elucidate how the retina processes visual stimuli, this is a significant contribution. Therefore, the paper should be of interest to the wider neuroscience community.\n\nThe paper is written in a clear and compact way except for a couple of minor points, which I mention in Questions. Figures are informative and are well integrated into the text.\n\nI could not find any conceptual or technical mistakes. The authors clearly state their assumptions and analyze their data rigorously.\n\nEDIT: The authors addressed my minor concerns. I increased my rating to 7.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JhSIYt1dZi", "reviewer": "Reviewer_YUEv", "summary": "In this manuscript the authors try to apply a U-net architecture to predict future movie frames from salamander retinal cell responses to the previous frames. The main finding is a signal for the time since the start of the movie in the latent representation.\n", "questions": "Unfortunately, I don't think my concerns could be removed with anything the authors could do within the confines of a rebuttal phase.\n\nIn general, find a sufficiently large dataset, find an input format for the retinal responses that has some spatial correspondence and keeps which neuron (type) responded where, and try to include some interpretability of the output to find out what can be decoded.", "limitations": "---", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nUnfortunately, I don’t think any of the methods or results in this paper are useable. The plan to train a large DNN architecture on the reconstruction of a few seconds of movie is doomed from the start. A U-Net architecture can probably reconstruct the whole movie clip from a single number for the frame as input without any retinal data. This also means that finding a signal for which frame to construct is not unexpected.\nMoreover, I fail to understand what the underlying aim of training this U-Net was in the first place. A deep neural network trained to predict future video frames is a technique for unsupervised learning, i.e. explicitly to learn complex features that are not easily available in the original inputs and filter out quickly changing information. I don’t see how this would be helpful for understanding the encoding of retinal cells, even if it worked.\n\nBesides these fundamental problems, there are some odd choices in the model design and sometimes it was not entirely clear what the authors did exactly:\n1) The input is a stack of “Gramian angular field” images constructed from the PSTHs for individual neurons. This transformation has no spatial correspondence to the image. Thus, the whole U-net architecture makes little to no sense with this input.\n2) Similarly the weights of a resnet pretrained on images are not sensible for this kind of input.\n3) The authors add noise to the architecture at various points without much justification except a hope for more interpretable representations. \n4) At various places, I lost track which analysis were based on the U-Net and which ones were based on the additional VGG.\n5) The training section in the supplementary material suggests that different samples were generated by dropping 2 cells randomly, but there is no explanation how the network could handle input with permuted cell identities sensibly. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5TOOH64_w5L", "reviewer": "Reviewer_vaQu", "summary": "\nIn this manuscript the authors describe the use of a deep-learning pipeline to understand the nature of encoding in the salamander retina. Namely, they collect PSTHs from neurons in response to short natural movies, then the PSTHs are converted to images and used as the input to a U-Net, which is trained to reconstruct the movie. Curiously, the latent representation can be used to accurately decode time in the movie, even for completely novel movies. This suggests that the model learns a low-dimensional representation of elapsed time. Lastly, the authors use information theoretic analysis to show that encoding of time requires jointly encoding of static and dynamic features.\n", "questions": "1. Did the authors attempt to use simpler models to decode time?\n1. Can you predict time in a held-out movie using visual features from another? Might be a remote possibility, but I want to know if there are any \"simple\" visual features that predict time. If you take frames from the fish movie and compute the pairwise distance to all frames in another movie, I assume there is no correlation with time?\n1. Did the authors try simple autoencoder architectures with a single latent layer rather than a U-Net? I am curious what benefit is conferred by the multi-resolution architecture, especially given that they are using a GAF representation of spike trains.", "limitations": "I found the discussion of limitations adequate. No concerns with respect to societal impact.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1)\tI found the pipeline presented by the authors to be fairly novel and well-motivated. I am not aware of the use of Gramian angular fields and autoencoders to understand population coding. This has the potential to be useful in other fields of neuroscience more generally.\n\n2)\tTheir result – that the salamander retina may have a movie-invariant code for time – is genuinely interesting.\n\n3)\tFor the most part the paper was well-written.\n\nWeaknesses\n\n1)\tLack of controls and comparisons with other methods. I do not know how likely this is, but one could imagine that time could be encoded by fairly trivial aspects of the data – e.g. a change in the aggregate statistics of spiking. Perhaps neurons fire more to the onset of the movie and slowly decay to 0 by the end. Since simple controls aren’t presented, it is hard to know if this sort of triviality explains their results. \n\n    1.\tFirst, the authors could simply shuffle the identity of neurons in the test phase. This would account for any trivial gross changes in spiking statistics accounting for decoding accuracy. \n\n    1.\tSecond, a much simpler model should be presented as a baseline. For instance, train a linear regression model to predict the time in the movie from the PSTHs (or a dimensionally reduced version of them), in addition to summary statistics computed across neurons (e.g. the mean and variance). How does the accuracy of this baseline model compare to the U-Net? The outcome of this comparison would provide a much better motivation for the authors’ pipeline. While it is novel, it is important to know how much extra information it provides.\n\n    1.\tThird, show more raw results. Visualize how the latent space evolves over time for each of the three movies along with the PSTHs. It would be nice to have a better intuition for how the model is transforming the spikes, and what features in the spike trains are relevant for encoding time. Perhaps there are no obvious features one can spot by eye, which provides even better motivation for the authors’ method.\n\n\nMinor\n\n1)\tThe authors mention domain-conversion as an application of U-Nets. This could be my own misunderstanding, but I am not sure what is meant here since one of the examples given is super-resolution imaging.\n\n2)\tFigure 2 caption has bottom row twice. I also don’t know how the authors computed this receptive field. More detail would be nice here.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1o4dwV7PsM2", "reviewer": "Reviewer_PGRz", "summary": "The authors adopted a task-agnostic U-net to model the retinal encoding process and characterize the representation of \"time in the natural scene\" in a low-dimensional compressed latent space. The U-net takes PSTHs of the salamander retinal population as input and the reconstructed movie frames as output, and learns with error propagation. They found that the retina has a generalizable encoding for time in the natural scene, which learned from one movie can be used to represent time in another movie. They also showed that static textures and velocity features of a natural movie are synergistic. ", "questions": "Q1: since neural responses are delayed (when an input stimulus is presented), how to align the input (neural responses) and the output (movie frames) to make a proper input-output pairs? Is it necessary to shift few miniseconds to find a good alignment? \n\nQ2: beside the background and motino seperation showed in Fig2, is there any other disentangled feature found in the decoder layer? Is this feature disentanglement due to the utilization of the variational sampling layers? Any other tricks to find these disentangled feature?\n\nQ3: does the decoding performance reported in Fig3 means to decode the temporal order of N movie frames? ", "limitations": "Yes, the authors addressed the limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is clearly organized and the logic flow is good. Discovering the low-dimensional feature representation in retinal neuronal firing (the time feature) with the combination of a ML based method is interesting. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept