Representing Latent Dimensions Using Compressed Number Lines
Authors: Sahaj Singh Maini
James Mochizuki-Freeman
Chirag Shankar Indi
Brandon G Jacques
Per B Sederberg
Marc Howard
Zoran Tiganj
Venue: Submitted to ICLR 2023 (2023)

Abstract: Humans use log-compressed number lines to represent different quantities, including elapsed time, traveled distance, numerosity, sound frequency, etc. Inspired by recent cognitive science and computational neuroscience work, we developed a neural network that learns to construct log-compressed number lines. The network computes a discrete approximation of a real-domain Laplace transform using an RNN with analytically derived weights giving rise to a log-compressed timeline of the past. The network learns to extract latent variables from the input and uses them for global modulation of the recurrent weights turning a timeline into a number line over relevant dimensions. The number line representation greatly simplifies learning on a set of problems that require learning associations in different spaces - problems that humans can typically solve easily. This approach illustrates how combining deep learning with cognitive models can result in systems that learn to represent latent variables in a brain-like manner and exhibit human-like behavior manifested through Weber-Fechner law.

Under review as a conference paper at ICLR 2023
REPRESENTING LATENT DIMENSIONS USING COM- PRESSED NUMBER LINES
Anonymous authors Paper under double-blind review
ABSTRACT
Humans use log-compressed number lines to represent different quantities, includ- ing elapsed time, traveled distance, numerosity, sound frequency, etc. Inspired by recent cognitive science and computational neuroscience work, we developed a neural network that learns to construct log-compressed number lines. The net- work computes a discrete approximation of real-domain Laplace transform using RNN with analytically derived weights giving rise to a log-compressed time- line of the past. The network learns to extract latent variables from the input and uses them for global modulation of the recurrent weights turning timeline into a number line over relevant dimensions. The number line representation greatly simplifies learning a of problems that require learning associations in dif- ferent spaces problems that humans can typically solve easily. This approach illustrates how combining deep learning with cognitive models can result in sys- tems that learn to represent latent variables in brain-like manner and exhibit human-like behavior manifested through Weber-Fechner law.
INTRODUCTION
The human ability to map sensory inputs onto number lines is critical for rapid learning, reasoning, and generalizing. Recordings of activity from individual neurons in mammalian brains suggest particular form of representation that could give rise to mental number lines over different variables. For instance, the presentation of salient stimulus to an animal triggers sequential activation of neurons called time cells which are characterized by temporally tuned unimodal basis functions (MaeDonald et al. 201 Tiganj et al.. 2017 Eichenbaum, 2014). Each time cell reaches its peak activity at a particular lar time after the onset of some salient stimulus. Together, a population of time timeline of the stimulus history (Howard et al., 2015] Tiganj et al., 2018). Similarly, as animals navigate spatial environments neurons called place cells exhibit spatially tuned unimodal basis functions (Moser et al. 2008). A population of place cells constitutes a spatial number line that be used for navigation (Bures al. 1997 Banino al. 2018). The same computational strategy seems to be used to represent other variables as well, including numerosity (Nieder & Miller 2003), integrated evidence (Morcos & Harvey 2016), pitch of tones (Aronov et al. (007)). and conjunctions of these variables (Nich et al. (021). Critically, many of these "neural number lines" appear to be log-compressed (Cao al. 2022 Nieder Miller 2003), providing natural account of the Weber-Fechner law observed in psychophysics (Chater & Brown 2008 Fechner 1860/1912). Here we present method by which deep neural networks can construct continuous, oogcooppeeeed number lines of latent task-relevant dimensions.
cells constitutes temporal number line
Modern deep neural networks are excellent function approximators that learn in a distributed man- ner: weights are adjusted individually for each neuron. Neural activity in the brain suggests a representation where a population of neurons together encodes a distribution over latent variable in the form of a number line. In other words, latent variable is not represented as scalar (e.g., count of objects could be encoded with single neuron with a firing rate proportional to the count), but as function supported by population of neurons, each tuned to particular magnitude of the latent variable. To build deep neural networks with this property, we use global modulation such that recurrent weights of population of cells are adjusted simultaneously. We show that this gives rise to the comprrssd number lines and can greatly facilitate associative learning in the latent space.

---

Under review as a conference paper at ICLR 2023
Inspired by experiments animals, conduct experiments where neural network learns number lines for spatial distance and count of objects appearing over time. We designed an experimental setup where the network needs to predict when target event will happen. In experiments, time to the target event depends either on the elapsed time, traveled spatial distance, or the count of how many times some object appears in the input. Critically, just like in the experiments with animals, these variables are not directly observable from the inputs - they are hidden and have to be learned from the spatiotemporal dynamics of the input. For example, if the target event will happen after some object appears a certain number of times, the network needs to learn to identify the object and learn the correct number of appearances (Experiment 3; see also an illustration in Fig. (3). Similarly,
people can estimate distance when riding a bicycle using their motor outputs and sensory inputs
we can learn non-linear mapping of motor outputs and sensory inputs onto velocity and integrate velocity to estimate distance (this concept is an inspiration for Experiments 2a and 2b; see also an illustration in Fig.
Building models from computational and cognitive neuroscience (Shankar & Howard 2012 Howard et al. 2014, 2018), we propose a neural network architecture that gives rise to a number line supported by unimodal basis functions. The network is composed of three layers. The input is fed into a recurrent layer with the weights analytically computed to approximate the real-domain Laplace transform of the input. Critically, we use properties of the Laplace domain and apply global modulation to the recurrent weights to convert functions of time into functions of other variables, such as distance or count. The output of the Laplace layer is mapped through a linear layer with analytically computed weights implementing the inverse Laplace transform. The inverse gives rise log-oompeessed number line supported with unimodal basis functions. Depending on the mod- ulatory signal, this population can resemble, for instance, time cells, place cells or count cells. The output is then mapped through trainable dense layer with a sigmoid activation function to the net- work's output. This approach augments the capabilities of cognitive models, which typically rely on handcrafted features enabling them to learn latent variables. At the same time, the structure and properties of the cognitive model are preserved, allowing the resulting system to have strong explanatory power of neural activity in the brain and behavioral data.
To the best of our knowledge, this work is the first time that the Laplace transform and its inverse have been implemented as a part of a neural network trainable with error backpropagation - in our experiments, the gradient flows through the Laplace and inverse Laplace transform. Global modu- lation of analytically computed recurrent weights is also a novel approach to learning in Recurrent Neural Networks (RNNs). The Laplace transform uses a diagonal weight matrix which makes it robust to problems of exploding and vanishing gradients related to backpropagation through time. These problems have been somewhat reduced with gated networks, such as Long Short- Mem- ory (LSTM) (Greff et al. [016 Hochreiter & Schmidhuber 1997) and Gated Recurrent Units (GRU) Chung et al. (014)) Recent approaches bounded the gradients, such as Coupled Oscillatory RNN (coRNN) (Rusch & Mishral 2020) or used formalism that does not require learning of the recur- rent weights, such as Lagrange Memory Unit (LMU) (Voelker et al., 2019). Echo state networks (Jaeger 2001) used RNNs with non-trainable, fixed recurrent weights but without global modula- tion. Similarly, the multiscale temporal structure approach used fixed weights with a spectrum of time constants (Mozer 1992).
2 METHODS
We first describe the construction of a log- compresse timeline using an approximation of the real- domain Laplace and the inverse Laplace transform. Then describe how the timeline can be turned into a more general number line via global weight-modulation in the Laplace domain.
2.1 CONTINUOUS-TINE -IME FORMULATION OF LAPLACE AND INVERSE LAPLACE TRANSFORM
Given one mie-dimensional input signal f(t), define modified version of the Laplace transform F(s;
F(s; t) - f'e-<(t-))(()))
(1)
This modified version differs from the standard Laplace transform only in variable s: instead of s being complex value composed of the real and imaginary part, we use real and positive s. This
2

---

Under review as a conference paper at ICLR 2023
modification simplifies the neural network implementation while still giving the computational benefits the standard Laplace transform, as we will illustrate below. Note that is also a function of time t. This implies that at every moment construct the Laplace transform of the input function up to time t: f(0 - t' < t) & F(s,t).
To construct the timeline need invert the Laplace transform. The inverse which we denote as f(F; () can be computed using Post inversion formula (Post 1930):
t(::t)= = L = (-1)* ++1 de
d P(:
(2)
where F k/s and l 20.
2.2 GENERALIZING TIMELINE INTO A NUMBER LINE
We can use the Laplace domain to convert a timeline into a number line if we learn a time derivative of a variable that we want to represent using the number line. To achieve this, instead computing the Laplace transform for function of time ((), we compute the transform for a function of some variable (x(())
We first rewrite Eq. in differential form:
dsF(s;t) (s; f(t).
The impulse response (response to input ftt 8(0)) of ts; decays exponentially function
of time t with decay rate s: e (Fig. la).
We define time derivative of (t) a(t)
ds and modify Eq. equation follows:
dF(ss -a(t)sF(sst + f(t),
(4)
dt
such that att) modulates the decay rate s (Fig. [b).
By reorganizing terms in Eq. equation and applying the chain rule we obtain the Laplace transform of (x(()))
dF((s;)) = sF(s;x)=
f(z).
(5)
dr
Note that Eq. equation 3 and Eq. equation 5 are the same, except that in Eq. equation Eq. equation This allows us to use the same calculation as in Eq. equation to obtain the inverse Laplace transform of (();)))
3 is in
2.3 NEURAL NETWORKS IMPLEMENTATION OF THE LAPLACE AND INVERSE LAPLACE TRANSFORM WITH GLOBAL WEIGHT MODULATION
We implement approximation of the modified Laplace and inverse Laplace transform two- layer neural network with analytically computed weights (Fig. [b.. The first layer implements the modified Laplace transform through RNN with recurrent weights. The second layer implements the inverse Laplace transform as a dense layer with weights analytically computed to implement k - th order derivative with respect s.
While in the Laplace domain 8 is a continuous variable, here we redefine as N elements long vector. We can now write a discrete-time approximation of Eq. equation 4as an RNN with a diagonal connectivity matrix and linear activation function:
Fst Wt, F Sc.
where Wr, is recurrent weight matrix W, e at where W,, is N by N matrix and S a diagonal matrix with diagonal elements composed of values. At every time step f, Fez is N elements long vector. For brevity of the notations, assume that the duration of discrete-ting step At = Importantly, since the values of vector s are fixed, W,, is also fixed and not learned during training.
3

---

Under review as a conference paper at ICLR 2023
a
b
fa
Modulatoryy Inputs
Trainable dense
F
F
mb
Fes
Trainable dense
Prediction
Figure 1: The activity of neurons in CNL with and without modulatory input. Impulse response of the Laplace transform approximation decays exponentially with decay rate s. The impulse response of the inverse Laplace transform approximation has unimodal form such that each curve peaks at 7. Note that if time was shown on the log-scale, the unimodal curves would be equally wide and equidistant. b. modulates the decay rate of Fs.. When = O. Fs;: is constant since its time derivative is O. When a = 0.5 the decay of Fsz is slowed down by a factor of 2. c. Schematic of the CNL network with input fa and modulatory inputs. Ja feeds into the recurrent layer that implements an approximation of the Laplace transform using analytically computed recurrent weights. Modulatory inputs are passed through a trainable dense layer and modulate the recurrent weights. Approximation of the inverse Laplace transform fr-- is computed by applying a linear operator LE on Esgt- Afterward, J=.. fed into trainable dense layer with sigmoid activation function. The network has single prediction output which is trained to predict the activity of target fb.
Following Eg. equation 2 discrete approximation of the inverse Laplace transform, fr can be implemented as a dense layer on top of F...- The connectivity matrix of the dense layer is L (see Appendix Sec. A.1 for the derivation of the exact matrix form of L
To interpret f= and to select values in an informed way, compute the impulse response of fr
For input f(t) 6(0), the activity of f4 is:
kk+1 u(l) k!
u(t)
where u(t tx) = Ei= (t.). a( S t tw)) = 1, then u(t = tn) = tn- To investigate properties of this approximation and motivate our choice for 7 and consequently the case where a(O < ftttt)= 1.
analyze
The impulse responses of units in f; is a set of unimodal basis functions (Fig. 1 and Fig. 2 2). To better characterize its properties we first find the peak time by taking partial derivative with respect to t, equate with 0 and solve for t: Of; /0t
+ t = 7. Therefore each unit in Fr peaks +. Note that if we computed the exact continuous-time Laplace and inverse Laplace transform (which would require infinitely many neurons, since s and would be continuous variables), the impulse response would be 8(f). This would provide a perfect reconstruction of the input function f(0 < < t) rather than its approximation.
To further characterize our approximation, we express the width of the unimodal basis functions of the impulse response of fr., through the coefficient of variation (see Appendix Sec. A.2 for the
derivation c): c = 1/vk+I Importantly, e does not depend on t and T, implying that the width of the unimodal basis functions increases linearly with their peak time. Therefore when observed as a function of log(t). the width of the unimodal basis functions is constant. Note that this property is critical for tog-compression.
4

---

Under review as a conference paper at ICLR 2023
We choose values of 7 as log-spaced between some minimum and maximum (see next section for the values used in the experiments). Because of the log-spacing and because с does not depend on and T, when analyzed wide, providing uniform support over log(!) axis. This result is analogous to the scale-invariance observed in human timing and perception, formalized is beneficial since the more recent past carries more predictive power than the more distant past. Hence, our approximation of function f(0 t' t) will be better for values closer to than to O.
function of log(t), the unimodal basis functions are equidistant and equally
the Weber-Fechner law. Intuitively, this
Note that fixing the values of f and choosing & also fixes values of since = k/f so s is not trainable parameter.
To convert a log-compressed timeline into log-compressed number line, we implemented global modulation of recurrent weights as in Eq. equation [ We assume that a is not known and that it needs to be learned from the input data. axt) is computed through feedforward projection which receives a subset of inputs designated as modulatory inputs (e.g. inputs from which a latent variable, such as velocity count, can be computed). Details of the feedforward projection are stated in the description of each experiment since they varied across experiments due to variability in the input dimensionality. We feed fz,. into a trainable dense layer with a single output and sigmoid activation function. Due to its log-compression of the input signal and gating of the recurrent weights, we call our approach Compressed Number Line or CNL.
RESULTS
We tested the proposed network on four event prediction experiments inspired by psychophysies experiments in humans and other animals. In our experiments, event a predicted event b such that the time between them depends on either the actual amount of elapsed time (Experiment 1), traveled distance with velocity as a latent variable which can be learned as a non-linear combination of the inputs (Experiments 2a and 2b) or the number of occurrences of a specific pattern (Experiment 3). Our goal was to demonstrate that the proposed network can identify latent variables by learning an appropriate mapping from inputs onto a global modulatory signal a. The network then constructs a number line representation over integral of the latent variable (e.g., if velocity is latent variable, then traveled distance is its integral across time), making the learning more accurate, faster, and more robust than with other approaches.
CNL parameters were kept the same for all the experiments. We set k = 8 and we used 64 neurons in F and 50 neurons in f with input fa- The number of neurons in f was smaller than in F order to avoid edge effects when taking derivative with respect to 8 (the number of units in f is 2k less than the number of units in F; Appendix Sec. Sec.[A.1 for details on the computation of the discrete approximation of the derivative). 7 was composed of 50 log-spaced values between 5 and 20000. These lower and upper limits were chosen to cover the entire range of temporal relationships in the input signal. The output of f layer was fed through trainable weights into single output neuron with a sigmoid activation function.
To evaluate how commonly used RNNs perform on this task we compared CNL with a simple RNN, GRU, LSTM, LMU, and CoRNN followed by fully connected layer. For LMU we experimented with hyperparameters and got the best performance with: di 1. 0 5000 and d 128. For coRNN we set the hyperparameters to the same values as in Rusch & Mishra (020): dt = 0.016, y = 94.5 and 9.5. The hidden size was 64 in all cases. As with CNL, the output layer had single neuron with sigmoid activation function. While still relatively simple, these networks had many more parameters than CNL (e.g., Table because most of the CNL parameters were computed analytically.
The results were obtained as an average of three runs with 95% confidence intervals. Each of the networks was trained for 1000 epochs using the learning rates 0.001, 0.01, 0.1, and 1.0 with ADAM optimization on an Nvidia Tesla P100 12GB GPU. We used sample weighting to train the networks using binary cross- cross-entropy as supervising loss and report binary cross-entropy test loss. We also report the average distance between the actual timestamp of the target event and the timestamp that received the highest probability estimate by a particular network.
5

---

Under review as a conference paper at ICLR 2023
3.1 EXPERIMENT 1: PREDICTING INTERVAL DURATION AT A WIDE RANGE OF TEMPORAL SCALES
To test the baseline performance of different approaches, we start with simple experiment with no latent variables. We designed a setup with input [a(() 8(x)) and target (label) fa(() 5(x + x') (Fig. 2 We used three values for x': x' E [50, 500, 5000]. Length of (t()) and [u(t) was 4x', 200, 2000 and 20000. We created only 3 training examples and 12 validation and 35 test examples, every time with a random x sampled uniformly from a range between 1 and 3x'.
Results are shown in Table [ (cross-entropy loss) and Table 5 (average distance). At the shortest timescale with x' = 50 and medium timescale with x' = 500. LMU did perfect job of predicting fü. At the largest time scale (x' = 5000), CNL performed better than the other approaches, which effectively failed to provide a useful prediction (Fig. likely due to a large number of parameters and problems with gradient backpropagation in long temporal sequences.
Table indicates that for CNL the error in the distance grows roughly linearly with x'. This gradual drop in performance is due the log-spacing of the recurrent weights. Fig. folshows how the standard deviation of the bell-shaped prediction curve scales with the temporal distance. These properties are consistent with temporal estimates in humans and the Weber-Fechner law.
To investigate the importance of the inverse Laplace transform, we also compared CNL (which is composed of the Laplace and inverse Laplace transform) with F(s) (only the Laplace transform, without the inverse) combined with a trainable dense layer of 50 neurons. We label the later model as CNL-F. CNL performed better than CNL-F all scales. We attribute this to the fact that the in- verse Laplace transform constructs a timeline, which is a representation that makes learning tempo- ral associations particularly easy. Timeline representation means that neurons activate sequentially following event a. This means that some neurons are active only when event b happens, making it easy for the network to learn the temporal relationships. This is confirmed by observing the learned weights of the dense layer in CNL (Fig. 3 Since the Laplace transform is linear, the output weights are interpretable. The weights are largest around T = x' and have a Mexican hat connectivity pattern which helps to sharpen the prediction peak (Fig. 6)
x'=50
x'=500
x'=5000
Params
CNL-F
0.399 1 0.031
0.433 + 0.008
0.639 + 0.014
51
CNL
0.212 + 0.003
0.227 + 0.004
0.255 + 0.004
51
RNN
0.521 + 0.317
0.446 + 0.281
0.561 + 0.184
4353
LSTM
0,347 0.500
0.669 - 0.077
0.647 1 0.000
17217
GRU
0.103 1 0.111
0.326 1 0.559
0.646 1 0.001
12929
CORNN
0.264 + 0.255
0.408
= 0.304
0.647 = 0.000
8385
LMU
0.005 = 0.008
0.208 = 0.265
0.732 - 0.103
12610
Table 1: Experiment 1: Binary cross-entropy loss across different scales.
3.2 EXPERIMENT 2A: PREDICTING DISTANCE IN THE PRESENCE OF TWO MODULATORY INPUTS
In this experiment input fa predicts input foo but the time between them was modulated by other inputs, Je and fa (see Fig. 4 for an illustration of the experiment). This means that the effective duration of each time step was a weighted sum of Se and fa as described in Alg. 1. For the CNL network, this meant that it had to learn that a wife fef + wyfd. where wi and W2 were randomly generated weights between 0 and and fa and fa had a magnitude ranging from 0 to 5. The network only observed fa and fa and not the weights w/ and W2, so it had to learn the relationship between and fe and fa. We can view this problem as one of learning spatial distance between event a and event b such that velocity (in this case (t)) is not directly observable but has to be learned from the modulatory inputs fe and fa. Therefore, a does not predict b after particular amount of time has elapsed. but instead after traveling a particular distance at a velocity that has to be learned from the network inputs. Similar to Experiment 1, in the absence of modulatory inputs fa(t) 5(xz) and fa(t) = 6(x + x') with x' E [50, 500, 5000]. Modulatory inputs fa and fa were stepwise functions broken into 15 pieces with amplitude of each piece between 0 and 10. The two modulatory inputs

---

Under review as a conference paper at ICLR 2023
passed through set of weights Wa (one weight for each modulatory input), adding more parameters to CNL.
Algorithm Generating an interval duration o/n with modulatory inputs fo and fa. If fo and fa
equal to 1, F'm = x' 100.
i -
r' =0
while S x do i=it!
I'n = x'm + wi Sala + + w2sa(w + +i)
end while
We constructed 6 training, 22 validation, and 22 test examples. The error was again quantified binary cross-entropy loss in predicting ft( Results from Experiment 2 are given in Table 2 (cross-entropy loss) and Table 6 (average distance). Despite having much fewer parameters, CNL performed better than the other approaches at all timescales. We attribute this to the ability of CNL to learn a from the inputs and cast the problem to be similar to Experiment by constructing a number line and then simply learning which component on the number line corresponds to the target, which is simple associative task. From observing plots in Fig. 7 provide useful prediction.
we see that other approaches failed to
x'=50
x'=500
x'=5000
Params
CNL-F
0.537 1 0.171
0.585 1 0.006
0.633 + 0.047
53
CNL
0.275 = 0.007
0.277 = 0.004
0.336 0.018
53
RNN
0.578 = 0.489
0.574 = 0.316
0.575 = 0.157
4481
LSTM
0.637 + 0.349
0.677 + 0.179
0.617 + 0.179
17729
GRU
0.653 0.098
0.659 0.131
0.594 0.096
13313
CoRNN
0.611 + 0.093
0.593 - 0.071
0.509 + 0.019
8513
LMU
0.645 1 0.161
0.592 + 0.258
0.521 + 0.350
12740
Table 2: Experiment 2a: Binary cross-entropy loss across different scales.
a
I6c
200
206
Distance
Distace
Distarice
Count
-50
16
:50
203
150
100
150
200
Time
Time
Time
Time
2-
21
50
10
150
300
-50
50
10
300
200
(10)
150
100
150
200
Time
Tire
Time
Time
-30
5D
ICC
200
-00
to
163
30
200
-30
50
100
100
200
50
100
20
Time
-50
10
100
Time
Time
Distance
Distance
Distance
Count
Figure 2: Temporal relationships in the presence modulatory inputs. Delta pulse in fa predicts delta pulse in fo- a No temporal modulation (e.g. 1). b and e When the modulatory signal corresponds to velocity (Experiment 2), the network represents traveled distance as manifested by units that activate sequentially function of distance (bottom row). d Temporal modulation with delta pulses (Experiment 3). The network represents the total count of presented pulses.
7

---

Under review as a conference paper at ICLR 2023
X x'=50
X x'=500
Params
Loss
Distance
Loss
Distance
CNL.F
0.601
0.220
17.4 7.6
0.498 0.004
229.7 0.0
1352
CNL
0.2031 + 0.019
7.4 0.6
0.2601 + 0.020
52.6 = 8.8
1352
RNN
0.658 + 0.162
70.4 = 15.9
0.676 + 0.229
746,6 + 96.0
7553
LSTM
0.627 0.064
67.3 3.7
0.657 - 0.170
675.4 1: 224.0
30017
GRU
0.578 + 0.134
57.1 = 16.6
0.640 = 0.075
573.9 - 22.2
22529
CORNN
0.627 + 0.196
54.8 = 21.9
0.627 + 0.043
733.1 + 112.4
11585
LMU
0.614 + 0.062
104.3
0.639 0.120
561.3 + 239.0
15860
Table 3: Experiment 2b: Binary cross-entropy loss across different scales and the average distance between the actual timestamp of the target event and the timestamp that received the highest proba- bility estimate.
count=10
count=200
Params
Loss
Distance
Loss
Distance
CNL-F
0.469 + 0,249
24.2 + 53.1
0.085 + 0.001
1005.0 1 0.0
196
CNL
0.201 + 0.036
1.3 + 0.4
0.077 = 0.004
75.8 = 21.5
196
RNN
0.323 + 0.076
69.8 + 24.5
0.108 = 0.023
1365.1 421.1
5377
LSTM
0.303 + 0.140
62.0 + 14.2
0.096 = 0.010
1264.4 + 941.1
21313
GRU
0.324 0.119
28.8 + 1.7
0.116 + 0.072
629.0
1006.0
16001
CORNN
0.636 + 0.117
60.8 + 25.3
0.093 = 0.000
1619.3 +199.:000 0.0
9409
LMU
0.402 1 0,297
65.5 1 87.8
0.146 + 0.135
597.7 + 781.4
13650
Table 4: Experiment 3: Binary cross-entropy loss across different scales and the average distance between the actual timestamp of the target event and the timestamp that received the highest proba- bility estimate.
3.3 EXPERIMENT 2B: PREDICTING DISTANCE IN THE PRESENCE OF HIFTY MODULATORY INPUTS
This experiment is more complex version of Experiment 2a with fifty modulatory inputs instead of just two. In this
EE wafs. where fi were the modulatory inputs and weights w, for 1 i I 6 were chosen randomly between 0 and and weights w; for 7 S S 50 were The between 0 and 5. Thus the network had to learn to
range of values for the magnitude of ignore 44 of the inputs (which were effectively noise) and add together weighted sum of 6 inputs. If the network correctly learns a, then the distance between fa and fú becomes fixed number line. The fifty modulatory inputs were passed through a set of weights W.. which was two-layer feedforward network.
We constructed 60 training, 20 validation, and 20 test examples. We conducted the experiment at two temporal scales with x' 50 and x'' 500. Since this experiment was more demanding in terms of resources, we omitted the timescale x' = 5000. Results from Experiment 2b are shown in Table 3 CNL again performed better than the other approaches at timescales. Results shown in Fig. suggest that CNL. was again the only approach that provided a useful estimate.
3.4 EXPERIMENT 3: COUNTING
In Experiment 3, the time between and was modulated by the number of specific patterns pre- sented as modulatory inputs (Fig. 5). At each time step, pattern composed of 16 elements with binary values was presented. One of the patterns, which we refer to as the target pattern, was re- peated multiple times and the network had to learn that the time between and b depends the number target patterns. The patterns were generated randomly.
To solve this task, the network had to learn to recognize the target pattern and count its repetitions. In this case, a had to be learned as a temporal derivative of the count. Whenever the target pattern appears, the count changes by one, therefore 1. Whenever the pattern was not the target pattern,
8

---

Under review as a conference paper at ICLR 2023
the counts stay the same and 0. Fig. 2 shows an illustration of this, assuming that the network has learned appropriate a.
We conducted the experiment two different scales, with 10 target counts and one with 200 target counts. In the first case, the input signal had 200 time steps, and in the second case, it had 2000 time steps. The results are shown in Table 4 At both scales, CNL performed better than the other approaches demonstrating that it learned the temporal derivative and constructed a number line representing the count of the target pattern. Fig. 9 shows prediction results for each of the seven networks.
4 DISCUSSION
The mammalian brain seems to commit to representing many physical and abstract dimensions as number lines. Computational neuroscience and cognitive science work developed a framework for the real-value Laplace transform and inverse Laplace transform that gives rise to such representa- tion. Our approach uses that framework as a part of a deep neural network that can extract relevant latent dimensions. Because of commitment to the number line representation, the network can learn various associative relationships with very few training examples and a small number of weights. Existing widely-used networks, such as LSTMs and GRUs fail at most learning tasks used here. While, in principle, neural networks with multiple layers, many training examples and long training time could potentially perform well, the proposed approach provides much simpler alternative.
Cognitive models provide immense utility in understanding neural computations in the brain, but they are usually limited to handcrafted features and associative learning. Here we have shown that incorporating cognitive models into neural networks can expand the utility and explanatory power of these models. CNL take advantage of deep learning and learn latent features while at the same time utilizing the benefits of the cognitive model and structured representation of knowledge which allows easy associative learning. The resulting performance resembled human performance in psychophysics tasks, commonly characterized by the Weber-Feehneg law. This is manifested by a roughly linear increase of error with distance Table and Table the standard deviation of the bell-shaped prediction curve in Fig. 6 as a function of distance.
[ as well as linear increase of
This work introduces trainable neural network implementation of real-domain Laplace transform and inverse Laplace transform. The Laplace domain enables access to number of useful operations. For instance, translation, convolution and correloorll can be efficiently implemented in the Laplace domain. In this work, the network constructs the real-domain Laplace transform of the position of as a function of the latent dimension. In the experiments used here, this is simply a delta function. Convolution of two delta functions at two real numbers produces delta function at their sum 6(z) * d(y) = 6(x + y); cross-correlation is similarly understandable as subtraction. In this way, abstract dimensions could be used with sparse representations as part of number system for symbolic computation (Howard et al., 2015 Howard & Hasselmo 2020). Because the same representational scheme can be generated for many diffferent latent dimensions, the same operations could be reused for many different kinds of information. Supplementary material contains code and detailed instructions for reproducing the results and integrating the Laplace and inverse Laplace transform into deep neural networks.
Activity patterns of neurons in the proposed network resemble the activity of neurons recorded in mammalian brains. Fig. 2 shows time cells, place cells (in ID environment) and cells tuned to particular number of objects. Each of these cell types has been recorded mammalian brains (Bures et al. 1997; Monad et al., 2011 Morcos & Harvey| 2016). The experiments used here require the network to learn a variety of complicated one-dimensional latent dimensions. Neurons in the mammalian hippocampus and prefrontal cortex have been shown to form conjunctive representations of multiple variables (Rigotti et al. 2013; Fusi et al. 2016 Nich et al., (021). We have yet to scale to multiple latent dimensions. Nonetheless, the current
show that the proposed approach results indicate that is possible to automatically learn continuous supported latent dimensions using standard techniques for training artificial neural networks.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Dmitriy Aronov, Rhino Nevers, and David W Tank. Mapping of non-spatial dimension by the hippocampal--ntorhinall circuit. Nature, 543(7647::71-772,, 2017. doi: 10.10388nature216622
Andrea Banino. Caswell Barry, Benigno Uria, Charles Blundell, Timothy Lillicrap. Piotr Mirowski, Alexander Pritzel, Martin Chadwick, Thomas Degris, Joseph Modayil, et al. Vector-based navigation using grid-like representations in artificial agents. Nature, 557(7705):42944333 2018.
J Bures, AA Fenton, Kaminsky, and L Zinyuk. Place cells and place navigation. Proceedings of the National Academy of Sciences, 94(1):343-3500 1997.
Rui Cao, John H Bladon, Stephen J Charezynski, Michael E Hasselmo, and Marc W Howard. Inter- nally generated time in the rodent hippocampus is logarithmically compressed. bioRxiv, 2021.
Nick Chater and Gordon D A Brown. From universal laws of cognition to specific cognitive models. Cognitive Science, 32(1):36-67, 2008. doi: 10. 108000364021000188199911
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: (41.v554 2014.
Howard Eichenbaum. Time cells in the hippocampus: a new dimension for mapping memories. Nature Reviews Neuroscience, 15(11):732 44, 2014. doi: 10.1038/mm3827
Gustav Fechner. Elements of Psychophysics. Vol. I. Houghton Mifflin, 1860/1912.
Stefano Fusi, Earl K Miller, and Mattia Rigotti. Why neurons mix: high dimensionality for higher cognition. Current opinion neurobiology, 37:66-74, 2016.
Klaus Greff, Rupesh Srivastava, Jan Koutník, Bas Steumebrink. and Jürgen Schmidhuber. Lstm: A search space odyssey. IEEE transactions on neural networks and learning systems, 28(10): 2222-211 2232, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
M. W. Howard, K. H. Shankar, W. Aue, and A. H. Criss. A distributed representation of internal time. Psychological Review, 122(1):24-53, 2015.
Mare W Howard and Michael E Hasselmo. Cognitive computation using neural representations of time and space in the laplace domain. arXiv preprint arXiv:2003.11663 2020.
Mare W Howard, Christopher MacDonald, Zoran Tiganj. Karthik Shankar, Qian Du Michael Hasselmo, and Howard Eichenbaum. A unified mathematical framework for coding time, space, and sequences in the hippocampal region. Journal of Neuroscience, 34(13):4692-70)) 2014. doi: 10. 10.1122/JNEUROSII..S.
12.2014.
Mare W Howard, Andre Luzardo, and Zoran Tiganj. Evidence accumulation in laplace domain decision space. Computational brain behavior, 1(3):237-251, 2018.
Herbert Jaeger. The "echo state" approach to analysing and training recurrent neural networks-with erratum note. Bonn, Germany: German National Research Center for Information Technology GMD Technical Report, 148(34): 2001.
C. J. MacDonald, K. Q. Lepage, U. T. Eden, and H. Eichenbaum. Hippocampal "time cells" bridge the gap in memory for discontiguous events. Neuron, 71(4):73--74, 2011.
Ari S Morcos and Christopher Harvey. History-dependent variability in population dynamics during evidence accumulation in cortex. Nature Neuroscience, 19(12):1672-1681 2016.
Edvard I Moser. Emilio Kropff, and May-Britt Moser. Place cells, grid cells, and the brain's 'ss spatial representation system. Annu. Rev. Neurosci., 31:69-89, 2008
Michael C Mozer.
Induction of multiscale temporal structure. In Advances neural information processing systems, pp. 275-282, 1992.
10

---

Under review as a conference paper at ICLR 2023
Andreas Nieder and Earl K Miller. Coding of cognitive magnitude: compressed scaling of numerical information in the primate prefrontal cortex. Neuron, 37(1):149-577 2003.
Edward H Nich, Manuel Schottdorf, Nicolas W Freeman, Ryan J Low, Sam Lewallen, Sue Ann Koay, Lucas Pinto, Jeffrey L Gauthier, Carlos D Brody, and David W Tank. Geometry of abstract learned knowledge in the hippocampus. Nature, pp. 1-5, 2021.
E. Post. Generalized differentiation. Transactions of the American Mathematical Society, 32:723- 781, 1930.
Mattia Rigotti, Omri Barak, Melissa Warden, Xiao-Jing Wang, Nathaniel Daw, Earl K Miller, and Stefano Fusi. The importance of mixed selectivity in complex cognitive tasks. Nature, 497 (7451):585-90) 90, 2013. doi: 10. 100800/ature120000 12160.
T Konstantin Rusch and Siddhartha Mishra. Coupled oscillatory recurrent neural network (cornn): An accurate and (gradient) stable architecture for learning long time dependencies. arXiv preprint arXiv:2010.00951, 2020.
K. H. Shankar and M. W. Howard. A scale-invariant internal representation of time. Neural Com- putation, 24(1): 134-193, 2012.
Z. Tiganj. J. Kim, M. W. Jung, and M. W. Howard. Sequential firing codes for time in rodent mPFC. Cerebral Cortex, 27:5663-56711 2017.
Zoran Tiganj, Jason A Cromer, Jefferson E Roy, Earl K Miller, and Mare W Howard. Compressed timeline of recent experience in monkey IPFC. Journal of Cognitive Neuroscience, 30:935-950,, 2018.
Aaron Voelker, Ivana Kajić, and Chris Eliasmith. Legendre memory units: Continuous-time repre- sentation in recurrent neural networks. Advances in neural information processing systems, 32, 2019.
11

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 DERIVATION OF THE CONNECTIVITY MATRIX FOR THE INVERSE LAPLACE TRANSFORM
As described in Eq. the inverse Laplace transform is performed using the Post inversion formula Post 1930):
f(f:t)= = L; = (-1)*
g+1 dt
F(s:t)
k! ds
where 7 = k/s. To implement this equation in a neural network we construct a discrete approx- d² First we compute N by N linear operator D which approximates the first order
imation for Z5
derivative: (Alg. 2) and then raise D to power k to implement k-th order derivative: D d'
dak
Algorithm Constructing N by N linear operator D) which approximates the first order discrete derivative.
D zeros(N) (N, N) for 4 to N do
Dit, - 1] f- [5 s
+ 1
IJ}
-ل+1|--0d -6-11
D[i, <- 2ul
2011
00
a|++1 -
Dit, + 1] (s(1 (((=1))=(((((+))) -|
-
end for
A.2 COMPUTING COEFFICIENT OF VARIATION OF
has unimodal impulse response with peak at F. The coefficient of variation, is ratio of standard deviation and mean. The mean of fi:
(f;st)dt
1
1++11
k+1
-ki
h!
k+1
k+1
F dt
k!
+
The standard deviation of f is:
Jo (1-1)3f(s;t))
h++1
t A+1 k
(t (t-w)²
e dt
k!
Vk +1
k
Finally, the coefficient of variation is then:
VK+I
The coefficient of variation depends only on k. Therefore the variance grows with the peak time. In other words, variance is constant function of the logarithm of the peak time.
12

---

Under review as a conference paper at ICLR 2023
A.3
VISUALIZAION OF WEIGHTS IN CNL
a
530
5001
Weggt
Weigft
Weght
200
-000
540
-250
-300
150
900
1860
1300
2000
Figure 3: CNL weights for the output dense layer when x' 50 (a), x' 500 (b) and x' 5000 (c). Since the Laplace transform is linear, the output weights are interpretable. The weights are largest around 7 50 (a), + 500 (b) and 5000 (c) and have Mexican hat connectivity pattern which helps to sharpen the prediction.
A.4 ILLUSTRATIONS OF EXPERIMENT 2 A AND
b
fedt
f vdt
J vdt
fadt
f dt
Figure 4: Illustration of predicting distance by learning latent modulatory inputs, as highlighted in Experiment 2. a. Let us assume that we want to estimate traveled distance from location A to location B. If we integrate time steps while traveling from be accurate (broad distribution shown in blue) since our velocity changes during the trip. However, if we integrate velocity (y axis) then the estimate would be more accurate (narrow distribution shown in blue) and subject only to error in the velocity estimate itself. b. If velocity is not directly observable, needs to be estimated as a combination of observable inputs, fo and fa as in Example 2a (for example, perhaps
(x axis) then the estimate would not
riding bicycle and only observe motor outputs and sensory inputs those would correspond to fo and fa here). we use only fc or fa, our estimate of traveled distance will not be accurate (broad distributions shown in blue). However if find correct combination of fe and fa (in this simple illustration that is fe + fa) then the velocity estimate more accurate (narrow distribution shown in blue) and subject again only error in the velocity estimate itself.
13

---

Under review as a conference paper at ICLR 2023
Target Pattern
888818888118BB88111
Input Data
T
8808111681811881111
8
1118818018010011BB0
8
8181 1181188888888881111
11811111008880111
81111811111111111
1111800881881800111
11188111110811111
181098081 18888181188
11
B1 111111111118811111
81811181 10111001100001111
8181 181 11111111111111111
100111100111111111
018811811111001111
Figure 5: Example output from the Counting dataset used in Experiment (duration and size of the pattern are shortened for illustrative purposes). Input Data columns represent modulatory inputs, and rows represent time steps. this example, the five rows highlighted in blue contain the target pattern, with the red portions being the target pattern itself. The target number of pattern occurrences 3, corresponding to the number patterns between the signal to start counting "S" (fed into the network as fo) and the hidden target "
A.5
AVERAGE DISTANCE METRIC EXPERIMENTS AND 2A
x'=50
x'=500
x'=5000
Params
CNL-F
8.0 + 0.0
141.0 0.0
2658.6 6.1
51
CNL
2.0 = 0.0
19.0
249.3 = 1.4
51
RNN
83.1
107.0
641.4
612.8
7513.0 + 10853.5
4353
LSTM
11.1
41.4
763.5
302.9
12560.0 + 0.0
17217
GRU
0.8 + 1.7
102.8
405.9
12405.7 = 154.8
12929
CORNN
4.0 + 3.7
438.3
594.5
12539.0 +:
8385
LMU
0.0 + 0.0
0.0
0.0
5001.3
1.0..:550 5724.4
12610
Table 5: Experiment The average distance between the actual timestamp of the target event and the timestamp that received the highest probability estimate.
x'=50
x'=500
x'=5000
Params
CNL-F
29.7 + 14.5
354.0 0.0
8546.1 + 9514.7
53
CNL
2.4 0.7
35.2 5.2
433.4 + 118.1
53
RNN
57.7 + 22.1
735.2 = 139.3
6783.5 + 3448.6
4481
LSTM
55.9 - 8.7
607.1 50.6
8200.2 1 3628.6
17729
GRU
62.2 + 13.7
592.8 + 90.9
5293.3 + 1597.3
13313
CoRNN
49.9 10.7
568.1 = 57.4
5553.4 + 913.9
8513
LMU
47.9 31.4
778.3 = 128.8
7884.5 + 2141.8
12740
Table 6: Experiment 2a: The average distance between the actual timestamp of the target event and the timestamp that received the highest probability estimate.
14

---

Under review as a conference paper at ICLR 2023
A.6
VISUALIZATION REPRESENTATIVE EXAMPLES IN EACH EXPERIMENT
a
b
c
CNL-F
CNL
RNN
LSTM
GRU
CoRNN
LMU
Figure 6: Experiment I: representative examples of prediction (orange lines) for different models. Input fa is shown blue and target fo is in green. Each row corresponds to a different time scale. Top row: x' = 50, middle row: x. 500 and bottom row: x' 5000.
a
b
d
CNL-F
CNL
RNN
LSTM
GRU
coRNN
LMU
Figure 7: Experiment 2a: Odd rows: representative examples of prediction (orange lines) for differ- ent models. Input fa is shown in blue and target fo. is in green. Even rows: Modulatory inputs fa and Sa- Plots shown for three temporal scales. Top two rows: x' 50, third and fourth row: x' 500 and bottom two rows: x' = 5000.
15

---

Under review as a conference paper at ICLR 2023
a
b
c
d
CNL-F
CNL
RNN
LSTM
GRU
CORNN
LMU
Figure 8: Experiment 2b: Odd rows: representative examples of prediction (orange lines) for differ- ent models. Input fo shown in blue and target fo is in green. Even rows: Modulatory inputs. Plots shown for two temporal scales. Top two rows: x' 50, third and fourth row: x' = 500.
a
b
c
d
CNL-F
CNL
RNN
LSTM
GRU
coRNN
LMU
Figure 9: Experiment 3: Top row: representative examples of prediction (orange lines) for different models. Input fa is shown in blue and target fo green. Plots are shown for two temporal seales. Top row: fo = 1 after 10 counts, bottom row: fo = 1 after 200 counts.
16


Reviews:
[{"review_id": "kpNimCweCi", "reviewer": "Reviewer_p2xG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper first summarizes many areas of cognitive neuroscience which have shown evidence for log-compression of quantities, such as time, distance, and numerosity. This is described in psychophysics as following the Weber-Fechner law. The authors then present a method to construct a deep neural network that can learn temporal relationships between parts of a sequence by encoding continuous, log-compressed \"number line\" representations of latent dimensions relevant to the task. This method relies on (1) a fixed-weight recurrent matrix which implements a modified Laplace transform, (2) some learnable terms $\\alpha$ which modulate the recurrent weights, (3) a dense layer that performs an approximate inverse Laplace transform, and finally (4) a trainable dense output layer. The authors then tested their proposed network on event prediction experiments that are modeled after the cognitive neuroscience experiments described at the beginning of the work. They compare the performance of their Compressed Number Line (CNL) model to other similar alternatives, such as various RNNs and the Legendre Memory Unit model. They find that their model, which has far fewer free parameters than the competing models, often fares the best.", "summary_of_the_review": "In the current state, this work is borderline. While the paper has novel contributions that would merit a spot at the conference, I do have a couple of key concerns about the experiments that I would like for the authors to address. I am not convinced that the competing models were evaluated fairly, and this has led me to lower my correctness score and rate this paper as a marginal reject. This score can be raised with some additional details provided by the authors and/or further data/experiments to illustrate fair evaluation of the models.", "strength_and_weaknesses": "Strengths\n- The technical contribution of recasting a Laplace transform as an RNN and the inverse Laplace transform as a dense NN is novel, to my knowledge, and can be used in other applications beyond the ones tested\n- Experiment setup is novel for ML and provides challenging tasks for existing methods like LSTMs/GRUs\n    - Experiment 2 is well-constructed and insightful, supporting their motivating choices for designing the CNL model. It reveals that learning temporal relationships that depend on sorting out many latent variables is fairly difficult for other networks\n- The choices of alternative models to evaluate is sensible and provides good context with the existing literature on learning temporal relationships in data\n\nWeaknesses\n- I found the methods section difficult to understand from a first read. I found it easier to interpret when it laid out which variables were kept fixed and which were learnable (i.e. the last 2 paragraphs of section 2). My recommendation is to introduce this first, and then run through how you set up the NN to approximate the Laplace transform / inverse Laplace. Otherwise too many intermediate variables are introduced which do not end up being key to the final proposal.\n    - for example, here's a place where this became confusing. The text says that $c$ does not depend on $\\overset{\\ast}{\\tau}$, but based on the definitions of $c$ and the relationship between $\\overset{\\ast}{\\tau}$ and $k$ stated after Eq. 2, it can be rewritten as $c = 1 / \\sqrt{\\overset{\\ast}{\\tau}s + 1}$. The independence from $\\overset{\\ast}{\\tau}$ only makes sense because the experimenters are fixing $k$, $\\overset{\\ast}{\\tau}$, and $s$.\n- Some experimental choices are not clear and might have contributed to misleading results\n    - Why were there different sizes of training/validation/test sets for the different experiments? No justification is given.\n    - Was there any early stopping criterion? Training for 1000 epochs on so little data seems like it would result in overfitting the competing models like the LSTM.\n\nFurther suggestions\n- I appreciated Tables 5 & 6, which provide support for Weber law-like behavior for the CNL on the tasks presented here. I'm curious whether this also applies to the counting task given the literature on numerosity (e.g. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7870758/). I see that the authors only tested 10 vs. 200, so testing some additional values along a log-axis may be helpful here.\n- While I understand the motivation to capture existing results from the human/animal literature, this paper would be strengthened by a discussion of areas of ML that would benefit from the types of log-compressed relationships learned by the CNL. For example, do the authors envision that human-like representations of time/space as learned by the CNL would be useful in robotics or RL? Is there any utility in using CNL for learning temporal relationships in real-world time series data, especially very long temporal sequences?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "cpq5WKuAm_", "reviewer": "Reviewer_8ytJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors presented Compressed Number Line (CNL), a deep neural network model that can extract time-dependent latent variables from sensory inputs, and use them to make predictions through Laplace transform-based timeline representation. This method essentially generalizes the timeline model to number lines, and showed the Weber-Fechner law in the network’s prediction.", "summary_of_the_review": "I’m leaning towards rejecting this paper for ICLR. The advances made here seem incremental when compared to Liu et al. 2018. The results are also not very strong.", "strength_and_weaknesses": "Strengths:\n- The paper is clearly written with well-defined mathematical variables. I also quite like the figure of the model (Figure 1c) which is simple and clear. The paper showed the emergence of hippocampal representations such as time cells and place cells (Figure 2) while also including many experiments beyond using only time and place.\n\nMajor weaknesses:\n- The CNL model doesn’t seem to deviate much from the time cell model introduced in [Liu et al., 2018](https://onlinelibrary.wiley.com/doi/10.1002/hipo.22994), except the part where they included the trainable dense network to compute  $\\alpha$, the latent variable that can modulate the recurrent weights $W_L$ in the Laplace layer, from modulatory inputs. The baseline models that CNL is compared to, including RNN, LSTM, GRU, coRNN, LMU, are most often used as time cell models (wherein sequential activities can be observed in the hidden state activities), which would be fair comparisons to the “core” layers in CNL which perform Laplace and inverse Laplace transform since Liu et al showed that these core layers alone can produce time cells. In experiments 2a, 2b, and 3 (in which the latent variable $\\alpha$ is involved), authors show that CNL outperforms the baseline models, which suggests the possibility that CNL relies on the feedforward layers that translate modulatory inputs to $\\alpha$ to perform these tasks. What would happen if you augment these layers to the baseline models, with $\\alpha$ modulating the parameters in the baseline models?\n- Relatedly, since CNL performs worse than some of the baseline models in the $x’$=50 and 500 settings in experiment 1 where time is the only variable, does this mean that the feedforward layers hinder the timeline representation, at least in the $x’$=50 and 500 settings?\n- In the model from Liu et al. there is a layer II that ensures Dale’s Law that this paper doesn’t include.What would happen if you consider Dale’s Law in CNL? (Also, please cite and discuss Liu et al. 2018 as this current work is largely based on that paper.)\n- Comparing Table 1 and Figure 6, it seems that two models with similar binary cross-entropy loss can have very different prediction patterns (specifically I’m talking about CNL and LMU in the $x’$=500 setting). This hints at the possibility that the choice of loss function cannot capture the desired prediction pattern (a bell-shaped curve that follows Weber-Fechner law). Have you tried using other loss functions?\n \n\nMinor weaknesses:\n- This work only applies to latent variables that are a function of time, i.e. x(t). However, in the introduction section, the author cited a few example number line representations that are independent of time, such as numerosity in Nieder & Miller 2003 and pitch of tones in Aronov et al., 2017. Can these experimental findings involving time-independent variables be explained by CNL?\n\nQuestions:\n- Section 3.3: “The 50 modulatory inputs were passed through a set of weights $W_\\alpha$, which was a two layer feedforward network”. Is the feedforward network two-layer for all experiments or only this one with 50 inputs? Does the depth of the feedforward network affect CNL’s performance?\n- Figure 7b first row: Why is the prediction up to the time when $f_a$ comes on a non-zero constant? This can be observed in more than one successful model (i.e. model that beat other comparison models) (another eg. would be Figure 6g second row) \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YxcbzqR9_xu", "reviewer": "Reviewer_ZzkM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method that learns to map input sequences into log-compressed number lines. The proposed method is evaluated on event prediction using synthetic data in 3 different settings and shown to outperform RNNs, LSTMs and several other baselines (despite having fewer parameters). \n\n\n", "summary_of_the_review": "I think this is a well written paper with convincing results, but have not evaluated its novelty with respect to prior work. I think the paper would benefit with stronger links to the machine learning / sequence modeling literature (i.e.on what tasks would the proposed approach excel in, and on what tasks would it not)", "strength_and_weaknesses": "**Strengths:** This paper is very clearly written. The suite of experiments is extensive and evaluates various aspects of the proposed method, the results of which are well discussed. The proposed approach also requires fewer parameters compared to standard sequence models such as RNNs and LSTMs.\n\n**Weaknesses:* My main concern is the lack of a discussion on what areas of machine learning the proposed method can be applied to with significant performance gains. The experiments suggest some directions (eg. needing fewer parameters), but a concrete, consolidated discussion of on which ML tasks this method might succeed or fail is missing. It may be worth mentioning some of these tasks in the introduction too. As framed currently, the paper is largely focused on adapting neural networks to mimic a particular cognitive phenomenon, and less on arguing when and why such mimicry might be useful in practice.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "RBGZ8KM7Fw", "reviewer": "Reviewer_GVDk", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors introduce a neural network model that learn to map input sequences into a logarithmically compressed temporal representation. The model is based on the analytical computation of recurrent weight matrices to construct a discrete approximation of the real part of the Laplace transform. The proposed method is tested on a set of synthetic problems, showing better performance compared to several benchmark methods.", "summary_of_the_review": "This paper could stimulate further research work, though it seems incremental in nature. Moreover, I believe the general framework should be described and presented more clearly, and its performance should be assessed on other challenging prediction tasks and benchmarked against state-of-the-art models.", "strength_and_weaknesses": "One strength of the current approach is that it requires a much smaller number of parameters compared to other methods, since most of the model’s parameters are derived analytically. However, it would have been useful to test it more directly against similarly inspired work (DeepSITH by Jacques et al., 2021) and/or other state-of-the-art methods for time series analysis (e.g., LogSparse Transformer by Li et al., 2019), and to demonstrate its performance on realistic datasets, rather than just focusing on synthetic data.\n\nMoreover, in the counting experiment, rather than just considering two extreme cases (10 vs. 200) it would be interesting to systematically study how the encoding accuracy is modulated by the number of events to be counted, to verify whether model behavior adheres to Weber’s law (see discussion in Testolin and McClelland, 2021).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject