Simpler neural networks prefer subregular languages
Authors: Charles John Torres
Richard Futrell
Venue: EMNLP 2023 Findings (2023)
Keywords: subregularity
formal language theory
minimum description length
inductive biases
TL;DR: LSTMs subject to $L_0$ regularization show a pronounced preference to learn subregular instead of regular patterns. LSTMs which learn subregular languages attain higher accuracy with fewer parameters.

Abstract: We apply a continuous relaxation of $L_0$ regularization (Louizos et al., 2017), which induces sparsity, to study the inductive biases of LSTMs. In particular, we are interested in the patterns of formal languages which are readily learned and expressed by LSTMs. Across a wide range of tests we find sparse LSTMs prefer subregular languages over regular languages and the strength of this preference increases as we increase the pressure for sparsity. Furthermore LSTMs which are trained on subregular languages have fewer non-zero parameters. We conjecture that this subregular bias in LSTMs is related to the cognitive bias for subregular language observed in human phonology which are both downstream of a simplicity bias in a suitable description language.

Simpler neural networks prefer subregular languages
Charles Torres University of California, Irvine
Richard Futrell
University of California, Irvine
Language Science charlt4@uci. edu
Language Science
rfutrell@uci. edu
Abstract
as well as patterns within tree structures character- izing syntax (Graf, 2022). The particular subregu- lar languages theorized to be human Icarnable are strictly piecewise (SP), strictly local (SL), and mul- tiple tier-based strictly local (MTSL). Artificial lan- guage learning tasks have shown these languages to be easier to acquire for humans (Lai, 2015; Aveu and Hestvik, 2020; McMullin and Hansson, 2019).
We apply a continuous relaxation of Lo reg- ularization (Louizos et al., 2018), which in- duces spansity, to study the inductive biases of LSTMs. particular, we are interested the patterns of formal languages which are readily learned and expressed by LSTMs. Across wide range of tests find sparse LSTMs pre- fer subregular languages over regular languages and the strength this preference increases as we increase the pressure for sparsity. Further- more LSTMs which are trained on subregular languages have fewer non-zero parameters, and subregular patterns have fewer non-zero parameters than those trained on unattested patterns. We con- jecture that this subregular bias in LSTMs is related to the cognitive bias for subregular lan- guage observed in human phonology, in that both are downstream of a bias for simplicity in a suitable computational description language.
Here we take up the question of inductive bi- ases for subregular languages in LSTMs (Hochre- iter and Schmidhuber, 1997), using relaxation of Lo regularization to study networks that are constrained to be simple in the sense of network sparsity. We train these networks on several sets of toy languages which contrast subregular lan- guages against regular languages in controlled manner. This technique not only allows to study the inductive biases of sparse networks, but also to address an outstanding question in the literature on subregular languages: whether there exists a computational description language under which subregular languages are simpler than other regular languages (Heinz and Idsardi, 2013).
LSTMs trained
1 Introduction
Despite the wide application of neural networks for sequence modeling, their inductive biases remain incompletely understood: for example, whether RNNs have an inductive bias favoring the kind of hierarchical structures used in sentence formation in natural language (Weiss et al., 2018; Ravfogel et al., 2019; Hewitt al., 2020; McCoy et al., 2020; Edelman et al.. 2022). It is important to study the inductive biases of neural networks not only from a scientific perspective, but also a practical one: if neural networks can be developed that have in- ductive biases toward the same kinds of formal language structures that characterize human lan- guage, then these neural networks should be able to learn human language more casily.
To preview results, we find that simple LSTMs favor subregular languages: such lan- guages can be represented by LSTMs with fewer parameters, and sparse LSTMs favor subregular languages in generalization.
The remainder of the paper is structured as fol- lows: Section 2 gives relevant background on sub- regular languages, Section 3 gives our general methods for training sparse neural networks under Lo regularization, Section 4 presents experi- ments on matched pairs of regular and subregular languages, and Section concludes.
Background
Many patterns in natural language are character- ized by a particular subset of the regular languages, called subregular languages. These languages have been hypothesized to form a bound on learn- able human phonological patterns (Heinz, 2018),
is known that phonological constraints should be regular given the most common formalisms in linguistics (Kaplan and Kay, 1994). However, the regular language class too permissive: many pat-

---

Language type Example language
In language
Not in language
SL
G = {ac}
be, ade, cad, bc, adb, cad, G {ac}, T ba, bc, cad, adbc....
ac, bac, acd, ac, adc, abc, ... ac, adc, adde,..
SP
G = {ac}
TSL
Table I: Example subregular languages on the alphabet x = a, b, c, d. Both SP and SL languages are defined by their alphabet and set of forbidden substrings or subsequences (G) in their definition. The SL language above forbids the substring ac. The SP language above forbids the subsequence o...., in other words, e cannot follow in string. TSL languages require a second set (T) specifying which characters are preserved in the tier projection. A tier projection function Ji maps strings strings, preserving only characters in the set T' For example, if a, b, then fr(adca)) = aca. The restrictions G' in a TSL languages are forbidden substrings in the projected strings.
terns not attested in natural language are members of it. The subregular hypothesis is an attempt im- prove this bound, lowering to certain subclasses the regular languages (Heinz, 2018). There are two versions of the hypothesis the weak version and the strong version- version--hppothesiiine hypothesizing different subregular languages as the boundary for human learnable patterns. The strong subregular hypothe- sis asserts that phonological constraints are within the strictly local (SL) and strictly piecewise (SP) subclasses, whereas the weak hypothesis raises this bound to the tier-based strictly local (TSL) lan- guages (the union of multiple constraints being mul- tiple tier-based strictly local or MTSL). These lan- guages are not the only subregular languages, but they are the most important subregular languages for the hypothesis.
stimulated some work on the learnability of sub- regular patterns by RNNs. Results in this area have been mixed. LSTMs and RNNs both show difficulty when trained to predict membership in subregular languages (Aveu et al., 2017). Never- theless, other work has still shown that the learning of subregular patterns is easier for sequence to se- quence RNNs than the learning of other patterns, echoing human-like preferences (Prickett, 2021). Work in this area is still limited.
What can explain these preferences LSTMs or humans? Previous work has attempted to tie pref- erence for simplicity to biases in natural language, for example by using the minimum description length (MDL) principle, a formulation of Occam's 's razor favoring simple hypotheses that account for given data, where hypothesis is simple if can be described using small number of bits in some de- scription language (Grünwald, 2000). Can the sub- regular preference be similarly explained? Heinz and Idsardi (2013) levelled a challenge against ac- counts of human language learning based on this method in explaining this preference: It is easy to find examples where subregular and regular language both have the same description length in some computational formalism (for example finite-state automaton, FSA). While they left open the possibility that a computational formalism may be found where description length can distinguish these examples, they expressed doubt that such a thing could be found. A similar challenge was raised in Rawski et al. (2017) where MDL accounts were challenged to explain learning biases.
What the SL, SP, and TSL languages all share in common is that they are defined by prohibiting features in the strings of that language. The SI. and SP languages (corresponding to the strong subreg- ular hypothesis) are defined by sets of prohibited substrings and subsequences respectively. These prohibitions are meant to capture phonological con- straints associated with local restrictions (SL) or harmony-like patterns (SP). The TSL languages slightly more complex than the other two, forbid- ding substrings after characters have been removed from a string (the retained characters are defined by a second set, usually labeled T for tier). To demonstrate these prohibitions in action examples of these languages are given in Table 1, however curious readers can look to Heinz (2018), Rogers and Pullum 1), and Heinz et al. (2011) for rigorous treatment of these languages as well as definitions for other subregular languages which are not part of the subregular hypothesis.
Is there computational representation that, when given access data consistent with a sub- regular and regular language, would learn only the subregular pattern by using an MDL method? We attempt to address these challenges. We will show that when controlling for description length (num-
The recent interest in subregular languages and their ability to capture phonological patterns has

---

ber of nonzero parameters) in our case studies, neu- ral networks prefer subregular patterns over regular ones on the basis simplicity, where FSAs not. That this is true LSTMs leaves open the possi- bility that MDL accounts are capable of explaining this bias in other systems, as in human language learning.
3 General Methods
A crucial part of our method requires finding opti- mal tradeoffss between accuracy and network com- plexity. We operationalize the complexity of neu- ral network as its number of nonzero parameters, reflecting the size of the subnetwork required to achieve given accuracy, As such, would like use the Tog norm, which counts the number of nonzero entries the parameter vector 0:
Figure 1: A simple depiction of Lo regularization, LSTM neurons in green,. Right: an unregularized net- work with similar architecture in the harmony test Left: after Le regularization has pruned connections.
parameters:
[0]
|0
1 0g #0
(1)
N
R(0,
E x~(((=))
Lppg(a)). M1) + (3)
N
Ideally, the Lo norm would form a part of our ob- jective function like so:
((2 > 0
N R(O) N Σ
where the expectation over mask vectors 2 drawn conditional on the vector masking parameters æ. The regularization term now reflects the expected number of parameters whose value is not masked. We use Equation 3 in all our experiments below. with different loss functions I. required. When L is form of cross-entropy this term represents the cost of encoding the data using the distribution specified by the neural network. Under these con- ditions, the objective becomes a form of minimum description length as two part code: with the regularization term encoding the distribution, and I encoding the data (Grünwald, 2000). Though our approach focuses the MDL interpretation, this formulation also has Bayesian interpretation, optimizing the negative log likelihood with the reg- ularization term as a prior and the loss term as a posterior as we show Appendix A.
((pola)). vi) + Al0loo, (2)
where [(po (mr), yi is the log likelihood of the l'th label given input z; in the training data under the model with parameters 6. However, the Lo norm is not differentiable, and cannot serve as part of our objective function during training.
We get around this limitation using relaxation of Lo regularization developed by Louizos al. (2018). For the j'th parameter in our model fJ. we assign a masking parameter и, > O. This parame- terizes distribution with cumulative distribution function Q-1IT)), from which we can sample mask value 2j E [0, 1], which critically can take the values of true 0 or 1. The vector of values generated in this way is then used mask to filter the parameters 0, generating vector of masked parameters 7 where 8, 2,65.
Experiments
4.1 Subregular Battery
This parameterization lends us new objective function where the labels y/ are predicted from the inputs z; using the masked parameters 0, and the model is penalized for having too few masked
The first experiment measures the complexity of LSTMs that learn subregular and regular languages. We call this the subregular battery: an experiment

---

run on three sets of three formal languages, where the description length of the generating FSAs are controlled within each set. Each set of languages contains one strictly local and one strictly piece- wise (which are both subregular) and one regular language. One set of languages was used in Heinz and Idsardi (2013)'s argument that minimum de- scription length fails to separate subregular lan- guages from other regular ones; offer two addi- tional sets. The languages are described in Table 2.
the development over all 400 epochs. We repeat this procedure five times for various regularization penalties to get data on performance at different levels of network complexity.
Evaluation Once data is collected, we evaluate the trained networks in terms of the trade-off of network complexity-measteee nonzero parameters- and error, defined as the av- erage per-character KL divergence of the LSTMs from the target distribution (the pFSAs from which the languages were sampled). For a target lan- guage with distribution QG and model Po. this divergence is
as the number of
4.1.1 Methods
The method can be split into 3 steps: data genera- tion, network training, and evaluation.
Data generation For each language, we sample 2000 words from probabilistic finite state au- tomaton (pFSA) designed to generate words from that language following geometric distribution on word length. The geometric distribution on word length important for two reasons: (1) it implies a constant stopping probability in all states, mean- ing that the distribution over string lengths does not depend on the structure of the strings in a lan- guage, and (2) it is memoryless, meaning that the network need not keep track of the total characters observed so far to attain high accuracy. Maintain- ing geometric distribution on string lengths is the simplest method for attaining the same distri- bution over string lengths across languages while sampling from pFSAs.
DiQc [Po] H[Qc, Pol H[Qc].
(4)
Thus, using the KL divergence here instead of cross entropy allows us control for the entropy H[Qo] of the target language G. We measure an approx- imate per-character KL divergence using the N sample strings in the evaluation set for language G:
T
Qa(af aist) Po 22
= N Σ
DiQcl
In
(5)
where Ti is the length of the i'th sample string in the evaluation set, x² is the t'th character of the i'th sample string, and xt is the prefix of characters up to index t in the n ''th sample string.
We use area under the curve (AUC) as met- ric to evaluate complexity. This is defined as the area under the convex hull of (complexity, KL ap- proximation) points. A small area under the curve indicates that it possible to achieve low error with simple network; a large area under the curve means that error is high even with more com- plex network. We determine if there is a significant difference between languages by performing per- mutation test. This is done by randomly sampling points from the union of two languages and com- paring the AUC statistic in the permuted samples.
This data is then split into training, development, and testing sets at an 80/10/10 ratio. Sets are con- structed such that no duplicate words existed be- tween sets. This was done by grouping words by form, and then randomly assigning them to the training, testing, or development sets before reshuf- fling them within the group.
Network training Each LSTM has the same ar- chitecture: a 3-dimensional embedding layer, a 5 unit LSTM layer, and 2-layer perceptron decoder with hidden width of 5 units. We train LSTMs for up to 400 epochs on the training set. For the subregular battery, we use cross entropy loss with the aforementioned To regularization technique as our objective function. After each epoch, we eval- uate development set performance (including the approximate Ta penalty), and if improvement on it has not increased for 50 epochs, training is halted. Otherwise we use the best performing network on
4.1.2 Results
Figure 2 shows the trade-off of number of param- eters and KL divergence for the three sets of lan- guages. The areas under the curve are shaded, and the points which closely mimic the target distri- bution shown in color. For a given level of complexity, the areas under the curve for the reg- ular languages exceed both the piecewise testable and strictly local languages. The specific AUC numbers are shown in Table 3. In every case the
This high patience is necessary due to the stochastic na- ture of the Lo regularization technique.

---

Set
SL variant
SP variant
Regular variant
*0ac
No string aac
No sequence .........
No if the number of a (mod 3)
+ab, * ba
No string ab / ba
No sequence ....b/ .
No (a) after an odd number of a (b)
*0C
No string
No sequence ....c
No after odd number of a
Table 2: Language sets composing the subregular battery. Each language set consists of three languages matched for FSA complexity: strictly local language (SL), strictly piecewise language (SP), and regular language. All languages are subsets of {a, b, c)" The table gives short description of each language. The last set is from Heinz and Idsardi (2013).
Strictly Local
Strictly Piecewise
Regular
0.4
200
Babback-Liebbee Liebler Divergence (bits)
0.2
0.1-
0.5
0.3
0.2
Kullback-1nn
0.0
0.4
392
0.2
0.1
20
40
60
20 40
60
20
40
60
Number of parameters (expected value)
Figure 2: A trade-off of accuracy (KL divergence from the target language to the modell and complexity (number of nonzero parameters) of LSTMs trained on subregular and regular languages matched description length in an FSA representation. Points within 1/100 of a bit or less from the true distribution area is the area under the curve (AUC) used to compare the languages in our statistical analysis in Table 3. The subregular languages show consistently lower AUC than the matched regular languages.
shown in color. The shaded
differences between both subregular languages and regular languages are statistically significant and in the expected direction as determined by permuta- tion test.
#CC
*0ac
*ab, *ha
Reg.-St SL
.354**
2.70**
2.688*#
We augment this test with an additional post- hoc analysis of the most accurately performing net- works (which we define as having KI. divergence of less than .01 bits) to determine if the size of the minimal LSTMs also significantly different. This analysis also done with a permutation test, These networks (Table 4) attain lower complexity score for the subregular languages, significant difference all cases except #ac.
Reg. Reg.-SP SP
*620**
1.727*#
2.260**
SP-SI. SI.
2666
.970**
.428
Table 3: Differences in AUC between languages within the subregular battery, as shown in Figure 2. The first row shows the AUC difference for regular languages minus SI. languages, etc. Comparisons marked are significant at p < 0.01 in permutation test. The AUC differences between regular and subregular languages are always significant. Comparing SP and SL languages. SP language has significantly higher AUC for the *aac languages, and the difference is otherwise insignif-
4.1.3 Discusssion
This experiment's s results demonstrate that, least in the languages chosen, strictly piccewise and strictly local languages better favor tradeoff be-
icant.

---

tween simplicity and complexity as demonstrated in our AUC analysis. We also see that they can be generated by functions implemented with simpler LSTMs in all but one case. In our post-hoe analy- sis we find that all but one subregular language is significantly simpler to accurately mimic in com- parison with the regular language. These results also demonstrate that checking for the presence of two substrings is more complicated than one, and checking for longer substring is more compli- cated than a shorter one. But if compared within a similar strictly regular pattern (here represented by controlling for generating FSA size) the preference is clearly for subregular languages.
ters in-between. These examples are a simplistic representation of patterns that are attested in human language (standard harmony) versus not attested (first-last harmony).
In the experiment below, we apply a similar test and evaluation method as in the subregular battery to demonstrate that the functions which generate SH patterns (which are MTSL) are simpler than functions which generate FL patterns (which are LT).
SH
FL
bab
bab
cac
cac
babbab beabeh
#GC
*aac
+ab, *ba
cacacae
cebabe
Reg.-LL .981
44.7**
23.8**
bbbbabb
beacabb
Reg.-SP
5.74**
43.7**
22.9**
SP-SI. -
.963
.972
Table 5:
Examples of legal words in standard hhamony (SH) and first-last harmony (FL.) patterns.
Table 4: Differences in complexity between languages within the subregular battery, as shown in Figure 2. The first row shows the complexity difference for regular lan- guages minus SL languages, etc. Comparisons marked are significant at p < 0.01 in a permutation test. Complexity differences are significant between subreg- ular and regular except for $ac.
4.2.1 Methods
We use LSTMs with with the same architecture as in section 4.1. The only difference that the objective for these networks is to predict the cor- rect continuation set: the set of possible charac- ters that can grammatically follow after a prefix, an objective introduced in Gers and Schmidhuber (2001) and used elsewhere (Suzgun al., 2019). The network is trained to predict, for each possible character continuing prefix, whether that charac- ter is licit or not-that is, whether that character is a member of the continuation set C. Given a continuation set C' and an LSTM equipped with a decoder that reads in prefix x of characters and produces vector of real values E R|O,, the loss for the network is
4.2 Standard Harmony and First-Last Harmony
In the second test we investigate whether prefer- ence exists between two subregular languages. One language is MTSL, a class of subregular languages believed to be learnable in the subregular hypoth- esis. The other is a locally testable (LT) language, which is believed to not be human learnable. This test is inspired by previous experimental works (Lai, 2015; Aveu and Hestvik, 2020) which were investigations of standard vowel harmony (attested in human languages) and first-last harmony (not at- tested). These two languages are not controlled for FSA size, but are languages that serve as important evidence for the subregular hypothesis.
J Σ ic|C
(p= C)
(og(po())))) (6)
[
(1 (1-C)) (x):
This breaks with the prior work using the contin- uation set objective, as we use the binary cross- entropy loss instead of mean squared error.
The difference between standard vowel harmony (SH) and first-last (FL. harmony languages is shown in Table 5. In an SH language we require languages to begin and end with an a b, and we forbid both a and b from occurring in the same string. In an FL harmony pattern, we require a word to begin and end with a or b, and to begin and end with the same character, but allow any charac-
Using this loss function is necessary to control for differences in entropy between the SH and FL languages when conducting comparisons. Above, we control for differences in entropy by using an approximate KL divergence as an accuracy metric and we control for length-relate effects on LSTM

---

4.2.2 Results
0.
Our results are shown in Figure Again, we see a similar pattern as in the subregular battery, but this time showing the human-learmable subregu- lar language is preferred over the unlearnable one. Standard harmony performance deterioriates at the 40 parameter mark, whereas first-last harmony de- teriorates at the 75 parameter mark. The areas approximated by the frontier points (shaded) main significantly different under permutation test, with p .009.
Avarage Bamoutt las (per charadar)
0.0
0.-
0.4
4.2.3 Discussion
0.2
These results continue the pattern seen in the sub- regular battery, but demonstrate not just a subreg- ular preference, but a human-learnable subregular preference. While prior studies have shown pref- erence and higher performance for SH over FL with sequence to sequence networks, this experi- ment demonstrates that beyond preference, these functions more easily represented in simpler LSTMs.
0.0
20 parameters (expected value)
Figure 3: Graphs for LSTMs trained on first last har- mony versus standard harmony, with description lengths the x-axis, and test set performance the y-axis (lower is better).
complexity by making the lengths of strings geo- metrically distributed. This is not possible when comparing the SH and FL languages, because they cannot be generated by pFSAs in a way that yiclds geometric distribution on string length.
4.3 Generalization
Above, found that subregular languages, and particularly human-learnable ones, were repre- sented by simpler LSTMs. The question remains: does this simplicity constraint affect generaliza- tion?
Data generation To accommodate this new task, we sampled words uniformly given length. For SH data, words were generated such that they began and ended with the same character (either or e) with a geometrically sampled number of charac- ters in-between (p - .05) consistent with the SH pattern (see table 5). For FL languages the same procedure followed except that the center char- acters were consistent with the FL pattern.
Here, we test whether pressure for sparsity in LSTMs yields a bias towards learning subregular language when the network is trained on data that is consistent with both a subregular and regular language, an approach suggested by Rawski et al. (2017). More precisely, the test as follows. Take two formal languages Gsub and Greg, one subregu- lar and one regular. Now, using their intersection Grub SsubnGrp Greg as training data, what does the induced grammar correspond to, Gsub or Greg?
Network training Training was performed simi- larly to the subregular battery. This was for a maxi- mum of 200 epochs, potentially terminating earlier if development set performance did not improve for 50 epochs.
4.3.1 Methods
As our pair of langauges, we use languages from the subregular battery in Section 4.1. We take Goub the language and Greg the regular lan- guage which forbids b after an odd number of as. This language is the test described in Rawski et al. (2017).
Evaluation At the end of training, (complexity, continuation-sel loss) pairs were calculated from the best network's performance on the heldout test set. As in Section 4.1 we use AUC as the perfor- mance metric, the only difference being the points are now (complexity, continuation-se loss) instead (complexity, KL approximation)
Data generation For each network, we generate data from Gsub O Grog. dividing these into train- ing set and a development set with approximately 1600 words in the training set and 200 words

---

Gzub Preference
Gouo Prefemed
kg Prefened
-0.2
50
100
150
200
Number of Parameters (expected value)
Figure 4: Model preference for subregular language Gsub over regular language Greg. where preference defined according to Equation 7, function of the number of model parameters. The preference for Gzub peaks in the shaded region occupied by simpler networks.
the testing set. We generate two additional testing datasets for Gsub and Greg individually, checking the sets such that no words in the testing set ap- peared in the training or development data of 200 words each. This matches the size of the dataset to that of Section 4.1.
4.3.2 Results
The results our generalization experiment show clear preference for subregular language re- gardless of complexity (Figure 4). However, unlike prior results, we also see a clear increase in prefer- ences as network complexity is constrained. This difference is especially pronounced in the high- lighted region. In the lowest complexity range, there collapse in performance where the net- work can do nothing but predict uniform distribu- tion on all characters all times.
Network training With the data generated, our LSTMs are trained according to the per character cross entropy loss, regularized in Equation with procedure identical to that used in Section 2.
Evaluation After training, performance on Gsub and Greg is evaluated using the model which per- formed best on the development set over the course of training. As in the subregular analysis, we calcu- late the approximate KL divergence by subtracting the true per-character entropy as determined by the generating PFSA from the cross-entropy in Equation 5.
Examining the KI. divergences for the individual languages, as in Figure 5, we find that this result is driven by high accuracy in modeling the subregular language Gsub using small number of parameters. Accuracy in modeling the regular language Greg is mostly poor, while accuracy modeling subreg- ular language Gsub is highest when the expected number of parameters between 25 and 75.
Our analysis, unlike the previous ones, is qual- itative. We will investigate the subregular bias, operationalized as the difference:
4.3.3 Discussion
While our prior results in the complexity evalua- tions have shown that subregular languages are gen- erated with simpler networks, these results demon- strate more clear inductive bias. LSTMs with constrained complexity generalize better to the sub- regular language than the regular language. This indicates that, not only the subregular function simpler, but that simpler networks prefer subregular languages.
PrefG.. = DiQG..g Pa] DIQG,.. (7)
where the approximate divergences D calcu- lated over the evaluation sets for the two languages Gsub and Greg. in Eq. 5. The value Prefa... is positive when the model's distribution over strings more closely resembles the one defined by the PFSA for language Gaib-

---

mean that this generalization pattern does not hold. Why do LSTMs favor subregular languages? We argue that our results may explain this prefer- ence via the lottery ticket hypothesis (Frankle and Carbin, 2019). This hypothesis states that neural networks learn by tuning "winning tickets"- are subnetworks that perform well the task initialization
Kullback-Liett L--ibeerr Divergence (bits)
0.6-
explains why larger neural networks perform well (they "buy more tickets"). The existence of smaller networks (as we define them) for subregular languages means that any neu- ral network will contain more tickets for subregular languages at initialization, and thus have an induc- tive bias toward such languages. If this line of reasoning is true, Lo regularization does not intro- duce anything qualitatively new in LSTM training, but shifts the probability mass to favor tickets with smaller subnetworks.
0.2
0.0
100
150
200
Number of Parameters (expected value)
Figure 5: KL. divergences from the target languages to the model distributions as function of the num- ber of model parameters. Performance subregular languages (black) dips in the shaded region whereas per- formance on regular languages is relatively poor (red).
Perhaps more controversially, we also believe these results may be of interest to human cognition. While the human brain is not LSTM, results indicate that subregular bias can be downstream of a bias for simplicity within an appropriate com- putational formalism, showing that rich (and often puzzling) constraints can downstream from sim- ple rules. A bias like this can be construed as a universal grammar- a bound on learnable formal languages (Nowak et al., 2002).
But we also see that this inductive bias exists even in low regularization regimes. This result is consistent with previous findings, and may be re- lated to other phenomena like previously observed locality biases (Ravfogel et al., 2019) which are basic constraint in all RNN functioning (Battaglia al., 2018).
More needs be done, of course. This work is far from proof of the simplicity of subregular languages in LSTMs. Likewise, more subregular languages ought to be investigated. We may find, for example, that the true inductive bias for LSTMs is not exactly subregular, or that only certain as- pects of subregularity are simple to implement. But find the current results exciting and intriguing, for its relationship network complexity and func- tion implementation, its potential to explain LSTM inductive biases, and the demonstration of subregu- lar biases resulting from computational complexity constraints.
5 Conclusion
In complexity measurement experiments observe that subregular languages are represented by consistently simpler networks than similar regu- lar languages, and that human-learnable subregular patterns are represented by simpler networks than unlearnable ones. This result is surprising, given that the use of a description length criterion to dis- tinguish these languages had been deemed unlikely (Heinz and Idsardi, 2013).
Furthermore, we found that in the generalization experiment, data consistent with a subregular and regular language leads to preferential generaliza- tion towards learning the subregular distribution, and that this preference increases the network be- comes simpler. This supports the idea that change in computational representation systems may favor subregular over regular languages, with the caveat that comparing subregular languages with radically different complexities in FSA representation may
Limitations
The largest limitation of this work is that it is ex- perimental, and not a proof. We try to show several demonstrations of the simplicity of subregularity, and believe further work should be done, but are aware of the limitations of this kind of work in addressing matters of mathematical curiosity.
We also understand that, because this work ad- dresses these issues in an experimental way, there

---

is no certainty that our object of study (subregular languages and their patterns) are what LSTMs are truly biased towards.
Benjamin L Edelman, Surbhi Goel, Sham Kakade, and Cyril Zhang. 2022. Inductive biases and variable cre- ation in self-attention mechanisms. In Proceedings of the 39th International Conference on Machine Learn- ing, volume 162 of Proceedings of Machine Learning Research, pages 5793-5831, PMLR.
Our work uses LSTMs, rather than the Trans- former architectures which underlie recently influ- ential large language models (Brown et al., 2020). Although LSTMs are not currently the most popu- lar architecture, recurrent models are recently see- ing revival (Peng et al., 2023) due to their inherent advantages such as being able to model indefinitely long sequences. In contrast to LSTMs, Transform- ers are known to be highly constrained in their strict formal expressivity (Hahn, 2020); however, their formal inductive biases in practice are not yet thoroughly explored.
Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations.
Felix A Gers and Schmidhuber. 2001. LSTM recur- rent networks learn simple context-free and context- sensitive languages. IEEE Transactions on Neural Networks, 1333:13333
Thomas Graf. 2022. Subregular linguistics: Bridging theoretical linguistics and formal grammar. Theoreti- cal Linguistics, 48: 1884
Ethical considerations
Peter Grünwald. 2000. Model selection based min- imum description length. Journal of Mathematical Psychology, 44(1):133-152.
We foresce no ethical issues arising from this work.
Acknowledgements
Michael Hahn. 2020. Theoretical limitations of self- attention in neural sequence models. Transactions of the Association for Computational Linguistics, 8:156- 171.
This work was supported by NSF grant #1947307 to R.F. We thank Connor Mayer, Jon Rawski, and Aniello de Santo for helpful discussion.
Jeffrey Heinz. 2018 The computational nature of phonological generalizations. Phonological typol- ogy. phonetics and phonology. pages 126-195.
References
Enes Aveu and Arild Hestvik. 2020. Unlcarnable phono- tactics. Glossa: a journal of general linguisties, 5(1).
Jeffrey Heinz and William Idsardi. 2013. What com-
plexity differences reveal about domains in language. Topics in Cognitive Science, 5(1):111-133
Enes Aveu, Chihiro Shibata, and Jeffrey Heinz. 2017.
Subregular complexity and deep learning. CLASP Papers in Computational Linguistics, page 20.
Jeffrey Heinz, Chetan Rawal, and Herbert G. Tanner. 2011. Tier-based strictly local constraints for phonol- ogy. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Hu- man Language Technologies, pages 88-64, Portland, Oregon. USA. Association for Computational Lin- guistics.
Peter W. Battaglia, Jessica B. Hamrick, Vietor Bapst, Alvaro Sanchez-Gonzalez, Vinícius Flores 7am- baldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Çaglar Gülçehre, H. Francis Song, Andrew J. Ballard, Justin Gilmer, George Dahl, Ashish Vaswani. Kelsey Allen, Charles Nash, Victoria Langston. Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matthew M. Botvinick, Oriol Vinyals. Yujia Li, and Razvan Pascanu. 2018 Relational inductive biases, deep learning, and graph networks. Computing Re- search Repository, arxiv: 100.:/8000000 01261, v3.
John Hewitt, Michael Hahn, Surya Ganguli, Percy Liang, and Christopher D. Manning, 2020. RNNs can generate bounded hierarchical languages with optimal memory. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages sociation for Computational Linguistics.
Online. As-
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. 33:1877- 1901.
Sepp Hochreiter and Jüürenn Schmidhuber. 1997. Long short-term memory. Neural Computation, 987: 1780.
Ronald M. Kaplan and Martin Kay. 1994. Regular models phonological rule systems. Computational Linguistics, 20(3):331-377
Regine Lai. 2015. Learnable vs. Unlearnable Harmony Patterns. Linguistic Inquiry, 46(3):425-451

---

Christos Louizos, Max Welling, and Diederik Kingma. 2018. Learning sparse neural networks through Lo regularization.
number of non-zero parameters in the model. Con- sider the standard MAP estimation problem:
MMP - arg max f (())0))((),
(8)
Thomas McCoy, Robert Frank, and Tal Linzen. 2020. Does syntax need to grow on trees? Sources of hier- archical inductive bias in sequence-to--squencc net- works. Transactions of the Association for Computa- tional Linguistics, 8:125-140.
where f is distribution conditional on parameters and g is a prior distribution on A. Now consider the expected value of non-zero parameters 0'. Equation 3, A' is the regularization term, as shown
Kevin McMullin and Gunmar Ólafur Hansson. 2019. Inductive learning of locality relations in segmental phonology. Laboratory Phonology, 10(1).
below:
101
Martin A Nowak, Natalia Komarova, and Partha Niyogi. 2002. Computational and evolutionary as- pects of language. Nature, 417(6889):61 1-617.
Oes mz)
(9)
Bo Peng, Bric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, et al. 2023. RWKV: Reinventing RNNs for the transformer era. Computing Research Repository, arXiv:2305.1304.
((2) Ty) (2) > 02
(10)
101
/ q(z TT) 12z 0) d2
Σ
(11)
j=1
Brandon Prickett. 2021. Modelling subregular bias in phonological learning with recurrent neural networks. Journal of Language Modelling, 9.
[0]
f q(< T)
(2j > 0) dz
(12)
j=1
Shauli Ravfogel, Yoav Goldberg, and Tal Linzen. 2019, Studying the inductive biases of RNNs with synthetic variations of natural languages. the 2019 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pages 3532-3542. Association for Computational Linguistics.
|0|
Proceedings of
E x-(())
Σ
(zg > 0)
(13)
= =0,
(14)
with probability density function g(- | अ) corre- sponding to cumulative distribution function Q( T), and indicator function 1(-). If we consider an exponential prior on we would have:
Jon Rawski, Aniello De Santo, and Jeffrey Heinz. 2017. Reconciling minimum description length with chaamarrindepeenee complexity measures. MIT Workshop on Simplicity in Grammar Learning.
q(0') de A0
(15)
James Rogers and Geoffrey K Pullum. 2011. Aural pattern recognition experiments and the subregular hierarchy. Journal of Logic, Language and Informa- tion, 34:22.3342.
Then by Eq. and the monotonicity of the log function, the MAP estimation problem is equiva- lent to finding
Mirac Suzgun, Yonatan Belinkov, Stuart Shieber, and Sebastian Gehrmann. 2019. LSTM networks can perform dynamic counting. pages 54-
ÓMAP arg min log
(16)
which is equal to our objective in Eq. 3 up to an additive constant log À that does not depend on the parameters to be optimized.
Gail Weiss, Yoav Goldberg, and Eran Yahav. 2018. On the practical computational power of finite precision RNNs for language recognition. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 740-745, Melbourne, Australia. Association for Computational Linguistics.
A Lo regularization as maximum a posteriori (MAP) estimation
We show that Lo regularization is equivalent to MAP estimation with an exponential prior on the


Reviews:
[{"review_id": "LzIWkcZFxX", "reviewer": "Reviewer_Vqqg", "summary": "", "questions": "- What happens as you scale the network architecture? Would a much larger network still exhibit the same behavior? Would the sparsity patterns still be seen with the L0 regularizer?\n- Along similar lines, what would happen if you sampled more than 2,000 words from the pFSA? How does this change the analysis?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and your time. We try to address your questions below.\n\n>*What happens as you scale the network architecture? Would a much larger network still exhibit the same behavior? Would the sparsity patterns still be seen with the L0 regularizer? Along similar lines, what would happen if you sampled more than 2,000 words from the pFSA? How does this change the analysis?*\n\nBoth of these are good questions. Unfortunately, the method we used (the Louizos (2017) method) is computationally costly. That, paired with having to run the training many times at different regularization penalties means we were forced to work with smaller networks and datasets. Presently, we are experimenting with faster means of approximating the L0 norm, however we have not been able to run the experiment at a larger size. At least on wider networks, any difference in results would be due to priors imposed by the gradient descent algorithm, as the objectives we used were based on an unnormalized number of parameters.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- interesting insights into formal language theory in relation to neural networks\n- very well-written. Clear and concise for a difficult topic", "reasons_to_reject": "- the LSTM model is quite small. \"a 3-dimensional embedding layer, a 5 unit LSTM layer, and a 2-layer perceptron decoder with a hidden width of 5 units\"\n", "questions_to_authors": "- What happens as you scale the network architecture? Would a much larger network still exhibit the same behavior? Would the sparsity patterns still be seen with the L0 regularizer?\n- Along similar lines, what would happen if you sampled more than 2,000 words from the pFSA? How does this change the analysis?", "paper_topic_and_main_contributions": "This paper looks at sparsity patterns as a means of studying inductive biases in LSTMs for formal languages. In general, this is a very well-written paper that covers formal language theory and gives interesting insights into how subregular languages are represented by simpler networks. The paper motivates why LSTMs are still an interesting problem and why that can relate to more popular neural methods such as the lottery ticket hypothesis. Overall, I would be happy to see this paper at EMNLP as it gives some interesting insights into formal language theory, neural networks, and demonstrates some empirically insightful results. In addition, I think it will have longer lasting power than many of the recent papers in the field that maximize a metric without any significant insights.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "v5ngVTU9qp", "reviewer": "Reviewer_ABqy", "summary": "", "questions": "1- I am not sure if I understand the connection between networks that are biased towards sub-regular languages  and sparsity, e.g. as you claim on lines 053 and 054 on page 1. Why in order to study simplicity we need to work with sparsity? Can you please clarify? \n\n-- a little while down, you do tell us why, that you would like ot use the L0 norm, and this is a reason. But can you now tell us why \n   1.1. why would you like to use L0 norm?\n    1.2. why the choice of a norm is related to the simplicity of a neural network?\n\n2- On page 1, line 092, can you please tell us what a minimum description language is?\n\n3- After equation (3) on page 2, you discuss the different interpretation for L, when it is cross entropy Firstly, do other cost functions also have different interpretations? If not, why cross entropy is special? Secondly, please can you tell us where do these two interpretations come from and how do they relate to your main concept, namely MDL?\n\n4- On page 4, line 187, you talk about  the importance of using a geometric distribution on word length when generating your data. But do not tell us why this is important and why a geometric distribution is used? Is it, for instance, related to Zipf's law etc? Why not use a Gaussian for instance, which is the more standard one?\n\n4- On page 4, lines 262 - 264, you conclude  simpler LSTM's can generate  strictly piecewise and strictly local languages, but the table of results only shows that the AUC is better for these languages in comparison with regular languages. Can be more precise in your statement?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and your time. We address your questions below, and will add clarifications to the paper or to make the introductions of these topics better.\n\n>*1- I am not sure if I understand the connection between networks that are biased towards sub-regular languages and sparsity, e.g. as you claim on lines 053 and 054 on page 1. Why in order to study simplicity we need to work with sparsity? Can you please clarify? -- a little while down, you do tell us why, that you would like to use the L0 norm, and this is a reason. But can you now tell us why 1.1. why would you like to use the L0 norm? 1.2. why the choice of this norm is related to the simplicity of a neural network?*\n\nThe L0 norm is desirable for two reasons. First, it results in networks that are smaller in the sense of having fewer parameters. This corresponds directly to a notion of “description length” for a network and so connects to MDL methods. Second, the L0 norm may reveal inductive biases of neural networks via the lottery ticket hypothesis. This hypothesis states that neural networks tune “winning tickets”–subnetworks that perform well on the task at initialization–which explains why larger neural networks perform well (they “buy more tickets”). While this is not the main thrust of this paper, the existence of smaller networks (as we define them) for subregular languages can explain already observed inductive biases: any neural network will contain more “tickets” for subregular languages at initialization. We will add text explaining these points to the conclusion.\n\n\n>*2- On page 1, line 092, can you please tell us what a minimum description language is?*\n\nThe Minimum Description Length method is a way of rigorously employing the notion of simplicity in learning. It is a mathematically specified version of Occam’s razor: that the best hypothesis to account for a given dataset is the simplest one. This is done using a two part code, one part encoding the distribution and another part encoding the observations, which is a form of MDL which is flexible and applicable to diverse model classes (including neural networks, Grunwald (2000)). This is the source of our objective function, and the interpretation consistent with the MDL literature. We will add text on this to Section 2.\n\n>*3- After equation (3) on page 2, you discuss the different interpretation for L, when it is cross entropy Firstly, do other cost functions also have different interpretations? If not, why cross entropy is special? Secondly, please can you tell us where do these two interpretations come from and how do they relate to your main concept, namely MDL?*\n\nThe cross entropy of a distribution P (for example, the true distribution in a training set) and Q (for example, a model distribution) can be interpreted as the cost of encoding data from distribution P under distribution Q, in terms of the number of bits required. For this reason, minimizing cross entropy is equivalent to finding a minimal-length description of P, and also equivalent to maximizing the likelihood of the data. In our case, we are adding a regularization term, which means we are searching for a minimal-length encoding of both the model and the data. We will add text explaining this point to Section 3. \n\n>*4- On page 4, line 187, you talk about the importance of using a geometric distribution on word length when generating your data. But do not tell us why this is important and why a geometric distribution is used? Is it, for instance, related to Zipf's law etc? Why not use a Gaussian for instance, which is the more standard one?*\n\nWe will add text explaining the importance of the geometric distribution on lengths: “The geometric distribution on word length is important for two reasons: (1) it implies a constant stopping probability in all states, meaning that the distribution over string lengths does not depend on the structure of the strings in a language, and (2) it is memoryless, meaning that the network need not keep track of the number of characters observed so far to attain high accuracy. Maintaining a geometric distribution on string lengths is the simplest method for attaining the same distribution over string lengths across languages while sampling from pFSAs.”\n\n>*5 - On page 4, lines 262 - 264, you conclude simpler LSTM's can generate strictly piecewise and strictly local languages, but the table of results only shows that the AUC is better for these languages in comparison with regular languages. Can be more precise in your statement?*\nIn accordance with this comment, we performed an additional statistical analysis on LSTMs which achieved high accuracy (within .01 bits of 0). The results are below:\n\n\n|   |  \\*ac | \\*aac| \\*ab, \\*ba |\n|---|---|---|---|\n| Reg - SL | .981 | 44.7\\*\\*|23.8\\*\\*| \n| Reg - SP| 5.74\\*\\*|43.7\\*\\*|22.9\\*\\*|\n| SP-SL| -4.75\\*|.963|.972|\n\nWe will add this data in and qualify our statements in accordance with this analysis, as results were significant and in the expected direction between all pairings except for Reg-SL on *ab, which was not statistically significant. We will also amend the language in those results to reflect instead the favorable accuracy/complexity tradeoff that the LSTMs showed for the subregular languages.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea is original and interesting, it would be great if other tests could be done and other people could regenerate the results so we were sure of the results, that LSTM's indeed learn the sub-regular human aligned languages better. The paper is well written and explains the ideas and results well. ", "reasons_to_reject": "Something that bothered me all along was the connection between the main idea, that LSTM's learn sub-regular better than regular, and sparsity. Why are these two related?\n\nThe other issue, for me, was the main objective function and its relationship to cross entropy and other cost functions. This has to be mapped out better, see my questions below. ", "questions_to_authors": "1- I am not sure if I understand the connection between networks that are biased towards sub-regular languages  and sparsity, e.g. as you claim on lines 053 and 054 on page 1. Why in order to study simplicity we need to work with sparsity? Can you please clarify? \n\n-- a little while down, you do tell us why, that you would like ot use the L0 norm, and this is a reason. But can you now tell us why \n   1.1. why would you like to use L0 norm?\n    1.2. why the choice of a norm is related to the simplicity of a neural network?\n\n2- On page 1, line 092, can you please tell us what a minimum description language is?\n\n3- After equation (3) on page 2, you discuss the different interpretation for L, when it is cross entropy Firstly, do other cost functions also have different interpretations? If not, why cross entropy is special? Secondly, please can you tell us where do these two interpretations come from and how do they relate to your main concept, namely MDL?\n\n4- On page 4, line 187, you talk about  the importance of using a geometric distribution on word length when generating your data. But do not tell us why this is important and why a geometric distribution is used? Is it, for instance, related to Zipf's law etc? Why not use a Gaussian for instance, which is the more standard one?\n\n4- On page 4, lines 262 - 264, you conclude  simpler LSTM's can generate  strictly piecewise and strictly local languages, but the table of results only shows that the AUC is better for these languages in comparison with regular languages. Can be more precise in your statement?", "paper_topic_and_main_contributions": "The authors test whether LSTM's have a  bias for -- thus can better learn -- sub-regular languages than regular ones. Within the answer set, they check for different types of sub-regularity, namely minimal description versus local and strict descriptions. They also check for sub-regular languages that were better aligned to human taste. In each occasions the answer is yes. LSTM's seem to prefer human-related sub-regular language to non human-related ones, to regular ones. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA"}, "extra_text": {}}, {"review_id": "DjzoQn12Xk", "reviewer": "Reviewer_eZgg", "summary": "", "questions": "A) lines 134--137: This sentence is confusing. Is Q supported on {0, 1}? Does sampling z from Q produce either 0 or 1? Or can z take any value in the interval [0, 1] ?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and your time. We will address what we can in the paper. As space is limited, we may make a point of addressing readers to the respective sources after giving a brief introduction to them.\n\n>*Personally, I lacked more pedagogical coverage of subregular languages in Section 2. The work seems to imply that the reader is already familiar with this area, but this is not entirely true for many potential readers of the article.*\n\nThis is a fair point. We will devote some space in the final paper to introducing the notion of subregularity and the subregular classes under investigation for the sake of future readers.\n\n>*A) lines 134--137: This sentence is confusing. Is Q supported on {0, 1}? Does sampling z from Q produce either 0 or 1? Or can z take any value in the interval [0, 1] ?*\n\nQ is supported on the interval $[0, 1]$, however it is defined such that the distribution has a non-zero probability of sampling the values of 0 and 1 in particular. We did not want to spend too much space on this topic, as we are just reintroducing the method from Louizos (2017). The distribution they (and we) use is defined in the following way.\n\nFirst, take a binary concrete random variable S parameterized by location $log(\\alpha)$ and temperature $\\beta$. This variable has support on the open interval $(0, 1)$, and can also be fit using the reparameterization trick. Then, “stretch” this distribution $\\bar{s} = s(\\zeta - \\gamma) + \\gamma$. Finally, clip this distribution  so it is again within the interval specified $min(1, max(0, \\bar{s}))$. This procedure results in obtaining samples from the closed interval $[0, 1]$ with non-zero probabilities for obtaining 0 or 1. This is explained in more detail in section 2.2 of Louizos (2017).\n\nWe will make the supported interval clearer in the text, although we want to avoid devoting too much space to explaining the method, as it is not our own contribution.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The work explores the bias towards simplicity, which is the basis for the successful training of modern learning algorithms. Unlike other works in this area, the focus here is on the nature of human language and its relationship to subregular languages.\n", "reasons_to_reject": "- Personally, I lacked more pedagogical coverage of subregular languages in Section 2. The work seems to imply that the reader is already familiar with this area, but this is not entirely true for many potential readers of the article.", "questions_to_authors": "A) lines 134--137: This sentence is confusing. Is Q supported on {0, 1}? Does sampling z from Q produce either 0 or 1? Or can z take any value in the interval [0, 1] ?", "paper_topic_and_main_contributions": "The paper shows that learning a sparse LSTM in a subregular language is more successful than in a regular language. In a sense, this shows a bias towards the simplicity of the LSTM _trained by the gradient-based method_. The paper compares the performance of LSTM 1) in subregular versus regular languages, 2) between two subregular languages, 3) as well as its generalizing ability in subregular versus regular languages. The analysis was carried out empirically, but at the same time thoroughly.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The methodology, results, and presentation are sound. The relevance to linguistics and human cognition are slimmer.
Decision: Accept-Findings