Explanatory Learning: Beyond Empiricism in Neural Networks
Authors: Antonio Norelli
Giorgio Mariani
Luca Moschella
Andrea Santilli
Giambattista Parascandolo
Simone Melzi
Emanuele Rodolà
Venue: ICLR 2022 Submitted (2022)
Keywords: explainability
rationalism
deep learning

Abstract: We introduce Explanatory Learning (EL), an explanation-driven machine learning framework to use existing knowledge buried in symbolic sequences expressed in an unknown language. In EL, the burden of interpreting explanations is not left to humans or human-coded compilers, as done in Program Synthesis. Rather, EL calls for a learned interpreter, built upon existing explanations paired with observations of several phenomena. This interpreter can then be used to make predictions on novel phenomena, and even find an explanation for them. We formulate the EL problem as a simple binary classification task, so that common end-to-end approaches aligned with the dominant empiricist view of machine learning could, in principle, solve it. To these models, we oppose Critical Rationalist Networks (CRNs), which instead embrace a rationalist view on the acquisition of knowledge. CRNs express several desired properties by construction, they are truly explainable, can adjust their processing at test-time for harder inferences, and can offer strong confidence guarantees on their predictions.

Under review as a conference paper at ICLR 2022
EXPLANATORY LEARNING: BEYOND EMPIRICISM IN NEURAL NETWORKS
Anonymous authors Paper under double-bling review
ABSTRACT
We introduce Explanatory Learning (EL), an explanation-drive -drivee machine learning framework to use existing knowledge buried in symbolic sequences expressed in an unknown language. In EL, the burden of interpreting explanations is not left to humans or human-coded compilers, as done in Program Synthesis. Rather, EL calls for a learned interpreter, built upon existing explanations paired with obser- vations of several phenomena. This interpreter can then be used to make predic- tions on novel phenomena, and even find an explanation for them. We formulate the EL problem approaches aligned with the dominant empiricist view of machine learning could, in principle, solve it. To these models, we oppose Critical Rationalist Networks (CRNs), which instead embrace a rationalist view on the acquisition of knowl- edge. CRNs express several desired properties by construction, they truly explainable, can adjust their processing at test-time for harder inferences, and can offer strong confidence guarantees on their predictions. As a final contribution. we introduce Odeen, basic EL. environment that simulates small flatland-style universe full of phenomena to explain. Using Odeen as testbed, we show how CRNs outperform empiricist end-to-end approaches of similar size and architec- ture (Transformers) for discovering new explamations unexplained phenomena.
simple binary classification task, so that common end-to-end
INTRODUCTION
Making accurate predictions about the future is a key ability to survive and thrive in a habitat. Living beings have evolved many systems this end, such memory (McConnell] 1962), and several can predict the course of complex phenomena (Taylor et al. (002)). However, no animal comes even close to the prediction ability of humans, which stems from unique-in-nature system.
At the core of this system lies an object called explanation, formed by the proposition of language, which has remarkable property: can be installed with ease into another human speaking the same language, allowing to make predictions on new phenomena without ever having experienced them. When the installation is successful, we say that the human has understood the explanation. This process is key to the success of human beings. An individual is not required to go through a painful discovery process for every phenomenon of interest, but only needs an operating system mastering language and someone who communicates explanations; then, the individual can focus on unexplained phenomena. When an explanation is found for them, is added to the existing shared collection, which call knowledge.
How can we make machines take part in this orchestra? With this work, we try to shed new light on this problem. Specifically, propose a learning paradigm to allow machines (i) to understand existing explanations, in the sense described above, and (ii) create new explanations for unexplained phenomena, much like human scientists do.
Our contribution in this sense is threefold:
i) We formulate the challenge of creating machine that masters a language learning problem by treating explanations as simple strings, while keeping their role well separated from the role of the data. This results in the Explanatory Learning (EI.) framework described in Sec.

---

Under review as a conference paper at ICLR 2022
Sketches of Jupiter moons made by Golleo
Observations of phenomenon in Odeen
Explanation:
Explanation:
Four wandering staus laving their period around a principal star
Exactly one blue surrounded by triangles
The Odeen universe. A convenient environment study and test the process of knowl- edge discovery in machines. Like the night sky was for humans *Galileo did sketch negative examples.
Figure 1:
ii) We present Odeen, a basic environment to test EL approaches, which draws inspiration from the board game Zendo (Heath) 200T). Odeen simulates the work of scientist in a small universe of We present it in Sec. 3 and will release it with this work.
simple geometric figures, see Figure 1
iii) We argue that the dominating empiricist ML approaches are not suitable for EL problems. We propose Critical Rationalist Networks (CRNs), family of models designed according to the epis- temological philosophy pushed forward by Popper 1935). Although CRN is implemented using two neural networks, its working hypothesis does not coincide with the model parameters. Instead, the working hypothesis of CRNs is a language proposition that can only be accepted or refused in toto. We will present CRNs in Sec. 4 and test their performance Odeen in Sec. 5
2 EXPLANATORY LEARNING
Humans do not master a language from birth. A baby can not use the message "this soap stings" to predict the burning sensation caused by contact with the substance. Instead, the baby gradually learns interpret such messages and make predictions for entire universe of phenomena (Schulz ict al. 2007). We refer to this state of affairs as mastering a language, and we aim to replicate it in machine as the result of analogous learning process. In particular, we propose to do so without explicitly prescribing a grammar, using instead an incomplete set of example explanations in the form of plain sequences of symbols.
Our focus is on learning an interpreter purely from observations. This allows not only to make predictions about phenomena for which already have explanation, but also discover planation for unexplained phenomena. We first describe the problem setup in the sequel, comparing existing ML problems; detail our approach in Sec.
Problem setup. Formally, let phenomena P1, P3, Pa.... be subsets of a universe U, which is a large set with no special structure (i.e., the entire data space U (#1...., ... x.))). U. one can define a language L as a pair (E,, It), where Σg is a finite collection of short strings over some alphabet A, with |Σ,| > |A], and Ir a binary function Iz : x Σ, - {0, 1}. which we call interpreter. We say that a phenomenon P; is explainable in language L if there exists string e E Σ, such that, for any æ E U, occurs I, (x, e) = 1p, (x), where 1 (xx) is the indicator function of P.. We call the string
Over a universe
explanation, in the language L, for the phenomenon P..
Our first contribution the introduction of a new class of machine learning problems, which we refer to as Explanatory Learning (EL).
Consider the general problem of making new prediction for a phenomenon Po C U. In our setting, this is phrased as a binary classification task: given sample z' U, establish whether x' E Po or not. We are interested two instances of this problem, with different underlying assumptions:
The communication problem: we have an explanation. We are given an explana- tion eo for Po. in an unknown language L. This means that we do not have access
'The explanation Four wandering stars having their period around principal star is adapted from the English translation of the Sidereus Nuncius Galilei 2016 page 9). First sketch the left is compatible with the rule since the fourth moon can he hidden by of the other moons by Jupyter itself.
2

---

Under review as a conference paper at ICLR 2022
to interpreter Iz; it looks like Japanese to non-Japanese speaker. Instead, also given other explanations {e1,..., ,,,,, in the same language, for other phenomena PL. PI....,PP Pn. as well as observations of them, ie., datasets {Di....,Dyi, D,} in the form Dj = ((11, 1 (z1))..... (xm, 1 (fm))), with m < |U]- Intuitively, here we expect the learner to use the explanations paired with the observations to build an approximated interpreter Σz, and then use it make the proper prediction for x' by evaluating (xx, eo).
The scientist problem: we do not have an explanation. We are given explanations fei,..., en) in an unknown language L for other phenomena Pi..... .. Pn, and observa- tions of them (D1.-. ....... D. However, we do not have an explanation for Po; instead, we are given just small set of observations Do = {Z),, 1 (I) .)..).................. (Iz, 1 (2x)) and two guarantees, namely that Po is explainable in I., and that Do is representative for Po in L. That is, for every phenomenon P * Po explainable in J. there should exist at least a z; E Do such that 1 (z,)) + 11(z)). Again, we expect the learner to build the interpreter Σr. which should first guide the search for the missing explanation eg based on the clues
Do. and then provide the final prediction through (z(,'e0).
Several existing works fall within the formalization above. The seminal work of Angluin (1987) on learning regular sets is an instance of the scientist problem, where finite automata take the role of explanations, while regular sets are the phenomena. More recently, CLEVR (Johnson et al. 2017) posed communication problem in universe of images of simple solids, where explanations are textual and read like "There is a sphere with the same size as the metal cube". Another recent ex- ample is CLIP (Radford et al. (02I), where 400,000,000 captioned internet images are arranged in a communication problem to train an interpreter, thereby elevating captions to the status of explana- tions rather than treating them simple l With EL., we aim offer unified perspective on these works, making explicit the core problem of learning an interpreter purely from observations.
Relationship with other ML problems. We briefly discuss the relationship between EL and other problems in ML, pointing to Sec. 6 for additional discussion on the related work.
EL can be framed in the general meta-learning framework, where the learner gains experience over multiple tasks to improve its general learning algorithm, thus requiring less data and computation on new tasks. Yet, differently from current meta-learning approaches (Hospedales et al., 2020), we are not optimizing an explicit meta-objective. Instead, we expect the sought generality to be a con- sequence of learning a language from examples, rather than a goal to optimize for. EL also shares some aspects with multitask learning and few-shot learning. The core difference here resides in the additional assumption of a common language between tasks and the centrality of the interpreter learning problem. The comparison between CRN and EMP-R in the experiments shows that this additional assumption grants a disruptive increase in generalization performance over a naive mul- titask learning approach. Also the problem of cross-task generalization in NLP shares many aspects with the communication and scientist problem of EL, such as the common language in which the different tasks are framed, see for instance (Ye et al. (02I). Yet, it exhibits also a crucial difference: in NLP this problem is always in a text-to-text format, i.e. in NLP the problem is framed in a single textual domain where explanations and data share the same role.
To many, the concept of explanation may sound close to the concept of program; similarly. the sci- entist problem may seem a rephrasing of the fundamental problem of Inductive Logic Programming (ILP) (Shapiro) 1981) or Program Synthesis (PS) (Balog et al. 2017). This is not the case. IIP has the analogous goal of producing a hypothesis from positive/negative examples accompanied by background knowledge. Yet, ILP requires data expressed as logic formulas by human expert; the ILP solver outputs a logic proposition, which the expert in turn interprets5] With EL, data can be fed as-is without being translated into logic propositions, and an interpreter plays the expert's 's role. PS also admits raw data as input, it yields a program as output, and replaces the expert with hand- crafted interpreter; this way, the sequence of symbols produced by a PS system only makes sense to a human, not to the system itself. Instead, in EL,, the interpreter is learned from scratch rather than hardcoded. An empirical comparison demonstrating the benefits of EL. over PS is given in Sec. 5
Next we introduce Odeen, an environment and benchmark to experiment with the EL paradigm.
Thhis shift greatly improved the performance of their model, as discussed in Radforr et al. 2021
Sec. 2.3).
In this sense, II. suffers the same problem of Explanation- Daoaanation-Basedd Learning, see Minton (1990)
3

---

Under review as a conference paper at ICLR 2022
3 ODEEN: A PUZZLE GAME AS EXPLANATORY LEARNING ENVIRONMENT
Single game. The inset shows a typical situation in a game of Odeen. The player looks at a set of structures made of simple geometric figures. Each structure is tagged red or green according to a secret rule, and the player's goal is to guess this rule. In the example, the rule can not possibly be "A structure must contain at least one red square" since the fifth structure the left does not contain a red square, but respects the rule (green tag). Once the player figures out the rule, she uses to tag a large number of new structures simplified interactive version of Odeen https//bit.ll///HeHHH
We made a
Odeen challenge. Our challenge goes beyond attempting to solve single game, which already difficult problem per se. Specifically, consider the point of view of some- one who does not speak the language in which the rules are written; an example of this is in the inset, where the secret explanations are given in hieroglyphics rather than English. Such a player would not be able to tag any structure accord- ing to the secret rule, even the latter is given. However, assume the player has been watching several games together with their secret rules. Reasonably, the player will grow an idea of what those strange symbols mean. If the player then wins several Odeen games, it would be strong evidence of mastering the Odeen language.
Zeno-toci- trinnglen pointing
An-lonat--ww rols worrchinnrg blee
At-l aquaeerre wat-ea------mt--eeeee
Problem formulation. Each game of Odeen is a different phenomenon P. of a universe U whose elements are sequences of simple geometric figures. In this universe, players scientists who try to explain the new phenomenon; see Figure [ The specific task is to make correct predictions for new phenomenon Po (a new game) given: (i) few observations Do of Po (tagged structures), in conjunction with (ii) explanations (el...., } and observations (Di..... Dr. of other phenomena (other games and their secret rules); see Figure (A and B). More formally:
Let us be given s unexplained phenomena with & observations each, and n explained phenom- ena with m observations each; let the n phenomena be explained in an unknown language, i.e., e1,... en are plain strings without any interpreter. The task is to make & correct predictions for each of the unexplained phenomena.
We consider / = 1176 (1% of structures); 8= = 1132; k= 32; m- = 10K, 1K, 100; n = 1438 or 500.
Why not explicitly ask for the rule? Instead of requiring the player to reveal the secret explanation explicitly, we follow the principle of zero-knowledge proofs (Blum et al. 1988). In our setting, this done by asking the player correctly tag many unseen structures according to the discovered rule. This makes it possible for any binary classification method to fit our EL environment without generating text. A winning condition is then defined by counting the correct predictions, instead of a textual similarity between predicted and correct explanation, which would require the player to guess word-by-word the secret rule. In fact, different phrasings with the same meaning should grant a victory, e.g., "at least one pyramid pointing up and at most one pyramid pointing up" is winning guess for the secret rule "exactly one pyramid pointing up"" brute-force enumeration of all equivalent phrasings, in turn, would not allow solutions like "exactly one one pyramid pointing up", where " "one" is mistakenly repeated twice; intuitively, we want to accept this as correct and dismiss the grammatical error. Similarly, a solution like "exactly one pointing up", where "pyramid" is omitted, should be accepted in a universe where only pyramids point up. We will reencounter these examples in Sec. 5 when we discuss the key properties of our approach.
Dataset generation. Odeen structures are sequences of up to six elements including spaces, blues or reds, squares or pyramids, the latter pointing up or down. The size of the universe is [0]
'The solution of the inset game is at the end of this footnote. Odeen is inspired by the board game Zendo, where players must explicitly guess the rule, known only to master. In Zendo, players can also experiment by submitting new structures to the master. Solution: At least one square at the right of a red pyramid.
4

---

Under review as a conference paper at ICLR 2022
In R
Interpreter
AA
1,178
End-to-end
Conjecture Generator
(A)
(B)
(C)
Odeen Explanatory Learning problem. Given observations and explanations in an unknown language for some phenomena (A), plus few observations of new phenomenon, explain the latter and prove this knowledge by correctly tagging large set of new samples (B). An empiricist approach attempts to extract this knowledge from data (C, left); a rationalist one conceives data as theory-laden observations, used to find the true explanation among set of conjectures (C, right),
Figure 2:
7 = 117, 649 possible structures. We further created a small language with objects, attributes, quantifiers, logical conjunctions, and interactions (e.g.. "touching". Appendix A). The grammar generates 2/5k valid rules in total. Each of the [U] structures is tagged according to all the rules. The tagging is done by interpreter implemented via regular expressions.
Metrics. As described above, the task to tag new structures for each of s unexplained games. An EL algo- rithm addressing this task encodes the predicted rule as an E-dimensional binary vector per game (predicted vector).
1.1
Predicted vector
where = 1 means that the i-th structure satisfies the pre- dicted rule, and 2 ground-truth vector, obtained by tagging the l structures ac- cording to the correct secret rule. Then, the Hamming distance (v, w") measures the number of wrong tags assigned by the EL algorithm; if dæ (v, w") < du (v, w:), where w; L w" ranges over all the possible 25k rules, then the predicted rule v made by the algorithm is deemed correct.
2WED pyramid
otherwise (see inset). Let w" be the at.lcaat 1 zod block 1
According to this, the Nearest Rule Score (NRS) is the number of correctly predicted rules over a total of 8 games. second score, the Tagging Accuracy (T-Acc), directly counts the number of correct tags averaged over games; this is more permissive in the following sense. Consider two different rules A and B sharing 99% of the taggings, and let A be the correct one; if an EL model tags all the structures according to the wrong rule B, it still reaches T-Ace of 99%, but the NRS would be O. An EL algorithm with these scores would be good at making predictions, but would be based on wrong explanation.
CRITICAL RATIONALIST NETWORKS
In principle, an EL problem like Odeen can be approached by training an end-to-end neural network
to predict ÿ = 1p (a'), given as input a set of observations D, and a single sample x' (see Figure C). Such a model would assume that all the information needed to solve the task is embedded in the data, ignoring the explanations; we may call it "radical empiricist" approach (Pearl 2021). A variant that includes the explanations in the pipeline can done by adding a textual head to the network. This way, expect performance improve because predicting the cxplanation string can aid the classification task. As we show in the experiments, the latter approach (called "conscious empiricist") indeed improves upon the former; yet, it treats the explanations more than mute strings to match, in a Chinese room fashion (Searle 1980 Bender & Koller 2020).
data, nothing
In the following,
introduce a "rationalist" approach to solve EL problems. This approach recog- nizes the given explanations as existing knowledge, and focuses on interpreting them. Here theory comes first, while the data become theory- observations.
5

---

Under review as a conference paper at ICLR 2022
Input: dataset Do. sample x'
Output: explanation cg- prediction j
C - 0
I
Transtormer Rule Encoder
Transtormer Label Decoder
while |C| < do
Do < rundom puirs C Do
generate
conjectures
C C (Do)
end
pox
[CL9
[s < 0} for 1....t
for
j
l,.... k
(x.)) v) random E Dn
for ,,. do
verify conjectures
Transtormer Decooder
Transformer Board Encoder
Transformer Label Decodar
g <- [e: E C,
if = then
end
[RU
é arg mrg (a) CC 2((o.x))
CG
EMP-R
Figure 3: Left: Test-time algorithm of CRNs. Right: CRNs are implemented using encoder- decoder transformers blocks, details of the parameters in Appendix Right-top: denotes the interpreter model (rule encoder and label decoder). Right-bottom: The conjecture generator CG is composed by blue blocks. The "radical empiricist" (EMP-R) is composed by orange blocks. The "conseious empiricist" (EMP-C) baseline model consists of all the transformer blocks in the right- bottom figure, board encoder with rule and label decoders (all the blue and orange blocks).
Learning model. Our Critical Rationalist Networks (CRNs) tackle the EL. scientist problem intro- duced in Sec. 2 to find y Pr, (a') given x', Do. {Di...., Dn}. {Ei...., en They are formed by two independently trained models:
(i) A stochastic Conjecture Generator
CG ((x, (r())) He,
taking h < DDol pairs (x, 1r(x)) E D, as input, and returning an explanation string E E as output. CG is trained to maximize the probability that CG((:) = e, for all i = -,....; n, where
D, C D; is random sampling of D,, and [D -0i==k k.
(ii) learned Interpreter
I: (c, + 0, which takes input a string Σ and sample U, to output a prediction (0, I is trained to maximize the probability that I(e,, = 1p, (z), with 1,..., n and pp E D,
At test time, we given a trained CG and trained I, and must predict whether some x' & Do belongs to Po or not. The idea is to first generate / conjectures by applying CG times to the dataset Da; then, each conjecture is verified by counting how many times the interpreter I outputs a correct prediction over Do. The conjecture with the highest hit rate is our candidate explanation én for Po- Finally, we obtain the prediction g' I(êo, x'). See Figure (left) for a step-by-step pseudo code.
Remarks. The interpreter I is a crucial component of our approach. A poor I may fail to identify eg among the generated conjectures, or yield wrong prediction y' when given the correct ep. On the other hand, we can work with a CG of any quality and safely return as output an unknown token, rather than a wrong prediction, whenever e, does not appear among the generated conjectures. The role of CG is to trade-off performance for computational cost, and is controlled by the parameter I. exhaustive search taken
Larger values for imply more generated conjectures, corresponding to the limit (as done, e.g., in Radford et al. (2021)). This potential asymmetry in quality between CG and I is tolerated, since the learning problem solved by CG is generally harder.
Secondly, although CRN implemented using neural networks, as we shall see shortly, its working hypothesis does not coincide with a snapshot of the countless network's parameters; rather, the working hypothesis is but small conjecture analyzed hypothesis is detached from the model and can only be accepted or refused in its entirety, rather than being slightly adjusted at each new data sample (Figure C, the hypothesis is in orange).
given moment. This way, the CRN

---

Under review as a conference paper at ICLR 2022
Implementation. Figure (right) illustrates the architecture of CRNs, which implement using encoder-decodee transformers (Vaswani et al. 2017). The figure also shows the architecture of the baseline methods EMP-R R and EMP-C, corresponding to the end-to-end NN model and its variant with a textual head, respectively. We refer to the Appendix for further details.
EXPERIMENTS
We extensively compared CRNs to the radical (EMP-R) and conscious (EMP-C) empiricist models over the Odeen challenge, and analyzed several fundamental aspects.
Generalization power. The Odeen challenge directly addresses the generalization capability of given algorithm, by asking for explanations to unexplained phenomena. This is evaluated over 8 = 1132 new games, where each game is given with = 32 tagged structures (guaranteed to satisfy unique, yet unknown rule) and requires correctly tag 1176 unseen structures according 1438 games with ground-truth explanations and does include any rule equivalent to the
to the unknown rule. The training set are m 1000 tagged structures per game. The training rules. One important example is the bigram "exactly two", which appears in the test set, but was deliberately excluded from training; the training rules only contain "at least/most two" and "exactly one". The CRN guessed 40% of the 72 test rules with "exactly two", while the empiricist models (EMP-C, EMP-R) scored 4% and 0% respectively. The table below reports the full results.
The NRS of 77.7% denotes that the CRN discovered the correct explanation for 880 out of 1132 new phenomena. Using the same data and a similar number of learnable pa- rameters, the empiricist models score 22.5% at most. Some example games can be found in Appendix
MODEL
NRS
T-Acc
R-Acc
CRN
0.777
0.980
0.737
EMP-C
0.225
0.905
0.035
EMP-R
0.156
0.898
In the table, R-Ace measures how frequently an output explanation is equivalent to the correct one; two rules A and B are equivalent if the tags assigned by the hard-coded interpreter to all the ~ 117k structures in U are the same for A and B.
As expected, the explanation predicted by the conscious empiricist model is rarely correct (R-Acc 3.5%), even when tags some structures properly (NRS 22.5%); indeed, EMP-C gives no guar- antee for the predicted explanation consistently provides the correct explanation when it is able to properly tag the new structures (NRS 77.7%, R-Acc 73. 73).%). The 4% gap between the two scores is clarified in the next paragraph.
be consistent with the tags prediction. Conversely, the CRN
Handling ambiguity and contradiction. One may reasonably expect that a CRN equipped with the ground-trute interpreter used to generate the dataset, would perform better than a CRN with leamed interpreter. Remarkably, this is not always the case, as reported in Table
Table I: Explanatory Learning vs Program Synthesis paradigm. Performance comparison of data-driven vs ground- truth interpreter CRN. The last column shows the tag prediction accuracy of the learned Z, when provided with the correct rule.
NRS FULLY-LEARNED CRN
T-AcC LEARNED Z
TRAIN DATA
HARDCODED I CRN
10K STRUCT.
1438 RULES
0.813
0.801
0.997
IK STRUCT.
1438 RULES
0.777
0.754
1.000
100 STRUCT.
1438 RULES
0.402
0.406
0.987
10K STRUCT.
500 RULES
0,354
0.377
0.923
IK STRUCT.
500 RULES
0.319
0.336
0.924
100 STRUCT.
500 RULES
0.109
0.101
0.920
The better performance of the fully learned interpreter over the ground-truth one is due to its ability to process ill-forned conjectures generated by the CG. The conjecture "at least pointing up" makes the hard-coded interpreter fail, since "pointing up" must always follow the word "pyramid" by the grammar. Yet, in Odeen, pyramids are the only objects that point, and the learned Z interprets
7

---

Under review as a conference paper at ICLR 2022
the conjecture correctly. Other examples include: "exactly one red block touching pyramid blue" ("pyramid" and "blue" are swapped), or the contradictory "at least one two pyramid pointing up and exactly one red pyramid",, which was interpreted correctly by ignoring the first "one". When the learned interpreter is not very accurate, the negative effect of errors in tagging prevails.
Making sense out of ambiguous contradictory message is a crucial difference between learned interpreter vs a hardcoded one. As Rotal (1991) reminds us, a concept does not need to be precisely defined in order to be meaningful. Our everyday reasoning is not precise, yet is effective. "After the small tower, turn right"; we will probably reach our destination, even when our best attempts at defining "tower", as found, e.g.. in the Cambridge dictionary, begin with "a tall, narrow structure..."
Explainability. The predictions of a CRN are directly caused by human-underssandddbe expla- nation that is available in the output; this makes CRNs explainable by construction. Further, CRNs allow counterfactuals; one may deliberately change the output explanation with a new one obtain a new prediction. The bank ML algorithm spoke: "Loan denied"; explanation: "Two not paid loan in the past and resident in district with a high rate of insolvents". With CRN, we can easily discard this explanation and compute a new prediction for just "Two not paid loan in the past". Importantly, by choosing a training set, we control the language used for explanations. This allows CRN to ignore undesirable patterns in the data (c.g., skin color) if these can not be expressed in the chosen language. If the Odeen training set had no rule with "pointing up/down", the learned interpreter would see all equal pyramids, even with unbalanced training data where 90% of pyramids point up.
On the contrary, current explainability approaches for NNs (end-to-end empiricist models) either require some form of reverse engineering, e.g., by making sense out of neuron activations (Goh) al. 202T), or introduce an ad-hoc block to generate an explanation given a prediction, without establishing cause-effect link between the two (Hendricks et al. 2016 Hind et al. 2019)
Adjustable thinking time. End-to-end models do not exhibit a parameter adjust their processing the com- plexity of the incoming prediction. By contrast, CRNs have a test-time parameter t, corresponding to the number of generated conjectures, which trades off computational cost for performance. In the inset, we plot the cumulative R-Ace score (y axis) against the number t of generated conjectures (I axis). The curves show that >> 60% of cor- rect explanations are found within the first 50 candidates, and > 80% are within the first 300. As a reference, a brute
-.. 100 Struet 500 Pates
Struct 1433 Rules
struct. 200 Rules
14. Struct 1438 furss
10K Struct 500 fulls
10K Struct 1438 Rules
00
05-
02-
50
100
130
200
300
force exhaustive search would reach 100% over search space of 24, 794 possible explanations.
Prediction confidence. As explained in Sec. 4. at test time the CRN selects the conjecture with the highest hit rate among the ones generated by the CG. Alternatively, one may keep only the conjectures coherent with all the structures in the table, returning an "unknown explanation" signal no such conjectures are found. If the interpreter is sufficiently accurate, this stricter condition barely deteriorates the CRN performance (1-2% gap on average), and it will never return a prediction based on a possibly wrong explanation. For example, tested in a setting with n = 1438, m = 1000 (same as the second row of Table D). this stricter CRN discovers the correct explanation for 861 out of I 1132 new phenomena (76%), and admits its ignorance on the other 271. Conversely, evaluating the confidence of an end-to-end neural network remains open problem (Meinke & Hein 2019).
RELATED WORK
Epistemology. The deep learning model we propose in this work, CRNs, is designed according to the epistemological theory of critical rationalism advanced by PPoper (935),, where knowledge derives primarily from conjectures, criticized at a later stage using data. Deutsch (201I) remarks that to make this critique effective, conjectures should not be adjustable but can only be kept or rejected at each new data sample, as done in CRNs at test time. Only in this way we can discover explanations with "reach", namely that maintain predictive power in novel situations.
TThis is of seven essential abilities for intelligence found in GEB (Hofstadter 1979 Introduction).
8

---

Under review as a conference paper at ICLR 2022
Machine learning. Explanatory Learning enriches the fundamental problem of modern program synthesis (c.g.. Balog et al., [017 Ellis et al., 2020) by including the interpretation step among what should be learned. As seen with few examples (sce Handling ambiguity in Sec. [5) CRN with learned I can exploit the ambiguity of language to impose new meaning on arbitrary substrates, which (Santoro et al. (202T) recognize as fundamental trait of symbolic behavior. Recent literature finds few yet remarkable approaches that fit our EL paradigm, such as CLIP (Radford et al.) 2021) in the vision area, and Generate & Rank (Shen al. 2021) for Math Word Problems in NLP.
The Odeen challenge continues the tradition of Al benchmarks set in idealized domains (Mitchel] 2021]. Unlike CLEVR (Johnson et al. 2017) and ShapeWorle (Kuhnle & Copestake 2017). Odeen focuses on abduction rather than deduction. Unlike ARC (Chollet (019), Odeen is a closed en- vironment providing all takes to learn the language needed to solve it. Unlike the ShapeWorld World adaptation of Andreas et al. (2017), its score is measured in terms of discovered explanations rather than sparse guessed predictions; further, the test and training set do not share any phenomenon.
Learning theory. Finally, we point out that the expression Explanatory Learning was previously used by Aaronson (203)] Sec. 7) to argue about the necessity of a learning theory that models "predictions about phenomena different in kind from anything observed". The author pointed to the work of Angluin [1877). who generalized the PAC model (Valiant) (984) by moving the goal from successful predictions to comprehensive explanations. This setup was discussed by Valiant himself in Robust Logics (Valiant 2000), and pushed forward in (Michael & Valiant 2008 Michael 2014), which present a dual inference scheme similar to the CRN, and in Možina et al. (2007). where data labels are motivated by explanations (arguments). In recent work Michael (2019) introduces a variant of the typical PAC definition that conceives the dual nature of explanations that emerges in this work, as propositions of a language to learn, and as explanation of a whole phenomenon.
Finally, we want to remark that Explanatory Learning should not be confused with Explainable Learning, a branch of XAI. While the goal of EL is to achieve generality by means of language explanations, XAI focuses on the explanation itself, with the ultimate goal of making Al predictions more digestible for humans (Hind et al. 2019
Teso & Kersting 2019).
7 CONCLUSIONS
Recently, the attention on the epistemological foundations of deep learning has been growing. The century-old debate between empiricists and rationalists about the source of knowledge persists, with two Turing prizes on opposite sides; LeCun (2019) argues that empiricism still offers a fruitful re- search agenda for deep learning, Pharl (021) supports a rationalist steering to embrace model- based science principles. This new debate relevant, since as Pearl notes, today we can submit the balance between empiricism and innateness to experimental evaluation on digital machines.
Limitations and future directions. EL models the essential part of the knowledge acquisition pro- cess, namely the interval that turns a mute sequence of symbols into an explanation with reach. However, our modeling assumes a representative structures of the new phenomenon). A more comprehensive explanatory model would allow the player to do without these observations, and instead include an interaction phase with the environ- ment where the Do itself is actively discovered. We see this as an exciting direction for follow-ups.
of observations Do to be given (the k 32
Odeen has potential as parametric environment to experiment with EL approaches. The structure length (6 in this paper), the number of shapes (3), attributes (2), and the grammar specifications (see Appendix A) can be easily tweaked obtain either simpler or significantly more complex environ- ments. The design choices of this paper provide good starting point; the resulting benchmark is far from being saturated, but experimenting with different variants a possibility for the future.
Finally, we expect CRNs to be more resilient than end-to-end models to adversarial attacks. For a given data point x' E Po classified correctly by an empiricist model, a small adversarial change on Do can flip the prediction for x' while remaining unnoticed. Conversely, suppose that CRN made the prediction for x', and assume that the correct explanation was ranked as the 5th most likely by the CG. The same attack on Do will have the effect of moving the correct explanation lower in the ranking; however, as long as it stays within the first conjectures (300 in this paper), it will always be found by the interpreter as the correct solution.

---

Under review as a conference paper at ICLR 2022
7.1 ETHICS STATEMENT
The Explainability paragraph in Section brieffy touches upon the topic of fairness in AI, pointing possible way address data debiasing- To this end, our proposed CRN model can be beneficial. In particular, CRNs can be potentially used to reduce the data bias, which is often identified as one possible cause of algorithmic discrimination in automated decision processes. While addressing these aspects is out of scope for this paper and it certainly deserves deeper investigation, we do not foresee any potentially harmful or inappropriate application of our methodology.
7.2 REPRODUCIBILLIT STATEMENT
Training: The proposed CRN model is composed of two parts, learnable interpreter I and conjecture generator CG. Their architecture is described in Figure (right) and in Appendix B. The training procedure, including the choice of hyperparameters, is also described in Appendix B. Testing: The algorithm used test time is fully described in Figure (left).....t Data: One of the main contributions of the paper is the introduction of new dataset and benchmark, called Odeen. Its full description is given in Section 3
(Problem formulation and Dataset generation paragraphs) and in Appendix A. The latter section also includes formal definition of the Odeen grammar. The metrics used for evaluation in the Odeen benchmark are defined in Section 3 (Metrics paragraph). A simple interactive version of the Odeen game, help the readers familiarize with the concept, is available at
Full code and data of the entire benchmark will be made publicly available
upon acceptance.
REFERENCES
Scott Aaronson. Why philosophers should care about computational complexity. Turing. Gödel, Church, and Beyond, 261:327, 2013.
Computability:
Jacob Andreas, Dan Klein, and Sergey Levine. Learning with latent language. arXiv preprint arXiv:1711.000 2017.
Dana Angluin Learning regular sets from queries and counterexamples. Inf. Comput., 75(2):87-106. November 1987. ISSN 0890-5401. doi: 10. 00001//08900500187790055557 (87) (00068709900-0 90052-6
URLhttps:://da
M Balog, AL Gaunt, M Brockschmidt, Nowozin, and D Tarlow. Deepcoder: Learning to write programs. In 5th International Conference on Learning Representations, ICLR 2017-Conference Track Proceedings, 2017.
Emily M. Bender and Alexander Koller. Climbing towards NLU: On meaning. form, and under- standing in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5185-5198, Online, July 2020. Association for Computational Linguistics. doi: 2020. aaa.mmaa........mmm.mm..a.mamama 463.
URL htcpp::/
org/
Manuel Blum, Paul Feldman, and Silvio Micali. Non-interactive zero-knowledge and its applica- tions. In Proceedings of the Twentieth Annual ACM Symposium on Theory of Computing, STOC '88, pp. 103-112, New York. NY, USA. 1988. Association for Computing Machinery. ISBN 0897912640. doi: 10.1145/62212.62222 62222
URL https:
François Chollet. On the measure of intelligence. arXiv preprint 2019.,
2019.
David Deutsch. The beginning of infinity: Explanations that transform the world. Penguin UK, 2011.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sable-Meyer, Luc Cary, Lucas Morales, Luke Hewitt, Armando Solar-Lezama and Joshua B Tenenbaum. Dreamcoder: Growing gen- eralizable, interpretable knowledge with wake-sleep bayesian program learning. arXiv preprint arXiv:2006.088 08381, 2020.
10

---

Under review as a conference paper at ICLR 2022
Galileo Galilei. Sidereus Nuncius, Or The Sidereal Messenger. University of Chicago Press. 2016. URL
edu/ www.e/wwwwwng//eeeeeeeeeeeeeeeee http:/iiinnngn/aaaaaaaai555551
SideriusNuncius. pde.
Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 2021. doi:
00.23915/distill00000 http://diiilllbbb2222///mtmtaaad
Kory Heath. Zendo, 2001. URL http:://www.orr//ett....oom zendo/
Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell. Generating visual explanations. In European conference on computer vision, pp. 3-19. Springer, 2016.
Michael Hind, Dennis Wei, Murray Campbell, Noel CF Codella, Amit Dhurandhar, Aleksandra Mojsilovic, Karthikeyan Natesan Ramamurthy, and Kush R Varshney. Ted: Teaching ai to explain its decisions. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp- 123-129, 2019.
Douglas Hofstadter. Güdel, Excher, Bach: eternal golden braid, volume 13, Basic books New York, 1979.
T. M. Hospedales, A. Antoniou, Micaelli, and A. J. Storkey. Meta-learning neural networks: A survey, IEEE Transactions Pattern Analysis & Machine Intelligence, may 2020. ISSN 1939-3539. doi: 10.1199/PAML22001.3779999
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Vi- sual Reasoning. Proc. CVPR, pp. 1988-1999 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Alexander Kuhnle and Ann Copestake. Shapeworld a new test methodology for multimodal lan- guage understanding. 2017.
Yann LeCun. The epistemology of deep learning. Institute for Advanced Studies https::/wwwtt es/ http/vvvvvoo///o/ee/ liteoo//ecun-oe ias 20190222. pdÉ; https:
las. /as.ecti/fittes//ffaall//// http://ttttteeeeeeeeeeoe/e/e/ee/
//youtu.be//e5NNceeet
2019, Accessed: 2021-10-04.
James McConnell, Memory transfer through cannibalism planarians. J. Neuropsychiat, 3:542-
548, 1962.
Alexander Meinke and Matthias Hein. Towards neural networks that provably know when they don't t know. arXiv preprint arXiv:1909, 12180, 2019.
Loizos Michael. Simultaneous learning and prediction. Fourteenth International Conference on the Principles of Knwwledge Representation and Reasoning, 2014.
Loizos Michael. Machine coaching. In Proceedings of the IJCAI 2019 Workshop on Explainable Artificial Intelligence (XAI), pp. 80-86, 2019.
Loizos Michael and Leslie G Valiant. first experimental demonstration of massive knowledge infusion. In KR, pp. 378-389, 2008.
Steven Minton. Quantitative results concerning the utility of explanation-based learning. Artificial Intelligence, 42(2-3):363-399 1990.
arXiv preprint Melanie Mitchell. Abstraction and analogy-making artificial intelligence. 10717 2021.
Martin Možina, Jure Žabkar, and Ivan Bratko. Argument based machine learning. Artificial Intelli- gence, (1010155)922-- 2007.
Judea Pearl. Radical empiricism and machine learning research. Journal of Causal Inference, 9(1): 78-82, 2021.
11

---

Under review as a conference paper at ICLR 2022
Karl Popper. The Logic of Scientific Discovery. Julius Springer, Hutchinson & Co, 1935.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proc. ICML, 2021.
Gian-Carlo Rota. The pernicious influence of mathematics upon philosophy. Synthese, 88(2):165- 178, 1991. doi: (0.1007/BF00567744
Adam Santoro, Andrew Lampinen, Kory Mathewson, Timothy Lillicrap, and David Raposo. Sym- bolic behaviour in artificial intelligence. arXiv preprint arXiv:2102.034000 2021.
Laura E Schulz, Alison Gopnik, and Clark Glymour. Preschool children learn about causal structure from conditional interventions. Developmental science, 10(3):322-333 2007.
John R. Searle. Minds, brains, and programs. Behavioral and Brain Sciences, 3(3):417-424, 1980. doi: 10.101775014555500007777
Ehud Y Shapiro. Inductive inference of theories from facts. Yale University, Department of Com-
puter Science, 1981.
Jianhao Shen, Yichun Yin, Lin Li, Lifeng Shang, Xin Jiang, Ming Zhang, and Qun Liu. Generate & rank: multi-task framework for math word problems. arXiv preprint arXiv:2109.03034 2021.
Alex H. Taylor, Rachael Miller, and Russell D. Gray. New caledonian crows reason about hidden causal agents. PNAS, 109(40): 16389-1611,,, 2012.
Stefano Teso and Kristian Kersting. Explanatory interactive machine learning. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp- 239-245, 2019.
Leslie G Valiant. A theory of the learnable. Communications of the ACM, 27(11): 1984.
Leslie G Valiant. Robust logics. Artificial Intelligence, 117(2):231-2533 2000.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st Inter- national Conference on Neural Information Processing Systems, NIPS' 17, pp. 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc ISBN 9781 110669944
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. Crossfit: A few-shot learning challenge for cross-task generalization in nip. arXiv preprint 1rXiv:2104.08833 2021.
A FURTHER DETAILS ON THE ODEEN DATASET
Training set. The total number of rules produced by the Odeen grammar 24,794. We consider training sets varying from 500 to 1438 rules. We choose these rules such that each token and each syntactic construct appears at least once; then, we uniformly select the others from the distribution. We removed from the training set any rule containing the bigram exactly 2, well as any rule of the form at_least 2 X and at_most 2 X, equivalent to exactly 2 X. Each rule associated with a set of 32 labelled structures that unambiguously identify a rule equivalence class. The first 10 structures are chosen by searching pairs of similar structures with different labels, following common human strategy in Zendo. The remaining 22 structures are selected to ensure the lack of ambiguity on the board.
Test set. We generate the 1,132 games that compose the test set the same way, with the additional constraint of excluding the rules belonging to an equivalence class that is already in the training set. In the test set, 72 rules contain the bigram exact 2.
12

---

Under review as a conference paper at ICLR 2022
Formal definition of the Odeen grammar. The context-free grammar in Figure 4 defines all the acceptable rules in Odeen. This grammar only formalizes which rules are syntactically correct. Token names (e.g. red, 1 touching) do not imply any rule meaning.
The hard-coded interpreter formalizes how to interpret the rules. Similarly to compilers, tokenizes and transforms the rule into abstract syntax tree (AST). The interpreter then adds semantic infor- mation to the AST, establishing the truth value of each node based on the truth value of its children and the structure under evaluation.
The Odeen binary semantic representations. By simulating the process of scientific discovery, Odeen offers convenient simulation of world described by language. Besides the computa- tional tractability, the simplicity and adjustable size of the Odeen world allows us explicit the whole semantics of its language.
These semantics can be encoded in a binary semantic matrix S with the 24,794 rules e; on the rows and the 1 117,649 structures zy on the columns. The S;j element of this matrix is equal structure Iy complies with the rule and O otherwise, see inset in Section 3 Metrics paragraph. Si.. the 117,64 d..4eiiddieesiinalll binary vector coinciding with the i-th row of 5, fully represents the meaning of rule in the Odeen world. Similarly, each structure zz is represented by the 24,794- dimensional binary vector coinciding with the column S.g of S.
the
In Figure 5 we analyze the distribution of the Hamming weights (i.e., the number of ones) in {
5b). We observe asymmetry between the rule and struc- ture distributions. On one hand, the semantic representation of rule can be quite unbalanced, with populated extremes of rules evaluating all structures with (or 0) shown in Figure [a On the other hand, F shows that the semantic representations of structures are very balanced; most of them have around half zeros and half ones, with no structure with less than 10k more than 14k ones.
3- 117.69 5a and (
This balanced trend, along with the well separable PCA of [S.,) (Figure 6a suggests that the chosen semantics produce representations that are effective in separating structures. Conversely, the PCA of {S 117,669 is much less homogeneous (Figure 6b). Here we can recognize two poles, corresponding respectively rules with all ones and all zeros. We believe that this analysis of the binary semantic representations is only partial, and we leave further exploration for follow-up work.
B IMPLEMENTATION DETAILS
In this paragraph, we give the implementation details of the models proposed and depicted in Figure (right). All the models are based on Transformer block composed of layers and 8 heads. We used hidden dimension of 256 for all the models except for the interpreter, where we used a hidden dimension of 128. The models differ primarily by the type of transformer block used (encoder/decoder), inputs and embeddings. In detail:
RULEE) + (PROP S)) (PROP) (PROP.S) (CONI) (PROP_S)
(PROP) + (QTY) (OBJ) (REL) (OBJ)
(PROP_S) () F (QTY) (OBJ)
(OBJ) F (COL) | (SHAPE) (COL) (SHAPE)
(QTY) } at.least (NUM) exactly (NUM) at.most (NUM) zero
(SHAPE) pyramid (ORIEN) pyramid block
(REL) touching surrounded.by at.theeright.off
(ORIEN) pointing.up pointing.down
(NUM) + 1 | 2
(CONJ) } and or
(COL) + red blue
Figure 4: Grammar productions for the Odeen Language.
13

---

Under review as a conference paper at ICLR 2022
120000
100000
20003
50000
13000
Smuztree
00000
10000
10000
15000
20000
25000
20000
40000
60000
80000
100000
123000
Structures
(a)
Figure 5: Hamming weight of the binary semantic representation of each rule (a) and each structure (b). We sort them in descending order for visualization purposes.
(a)
(b)
Figure 6: PCA applied the binary semantic representation of structures (a) and rules (b). We high- light in red the structures that have two touching pyramids pointing down. To ease the visualization, pair every structure with different string of characters. Each character replaces Odeen according to well-defined mapping. The rules distribution reflect what can be observed in Figure 5a
element of
TRANSFORMER LABEL DECODER. This is transformer block used to predict label given a structure. The imput structure is a sequence of six learned embeddings, one per piece. We add a sinusoidal positional encoding to each embedding as in the original trans- former implementation. The embedding size is 128 the I and 256 the Empiricist models (EMP-C, EMP-R). We used the standard transformer encoder block and added special token [CLS] ] at the beginning of the structure like Devlin et al. (2019) to perform the classification task.
TRANSFORMER RULE DECODER. This is a transformer decoder block with embedding size of 128 and sinusoidal positional encoding. This decoder block used to generate the rule by the EMP-C and CG models.
TRANSFORMER BOARD ENCODER. This is transformer encoder block used to encode the (structure, label) pairs. The input is encoded sequence of 32 learned embeddings, per structure-label pair. The size of each embedding is 256. We did not add positional
14

---

Under review as a conference paper at ICLR 2022
encodings, since the specific position of structure-label pairs among the 32 is not relevant. This block used in all the models.
TRANSFORMER RULE ENCODER. This transformer encoder block is used in Z to encode the rule. Its implementation is analogous to the TRANSFORMER RULE DECODER, with the only difference that does not use causal attention since it is an encoder layer.
Table Number of training epochs for each training regimen.
TRAINING REGIMEN
NUMBER OF EPOCHS
10K STRUCT.
1438 RULES
IK STRUCT.
1438 RULES
20
100 STRUCT.
1438 RULES
200
10K STRUCT.
500 RULES
STRUCT.
500 RULES
100 STRUCT.
500 RULES
576
Training Procedure. All the models are trained with a learning rate of 3 10-4 using Adam (Kingma & Ba 2017). batch size of 512 and early-stop and dropout We train all the models on randomly sampled sets of 32 (structure, label) pairs to prevent overfitting on specific boards. Table 2 describes the number of epochs for each training regimen. Models are trained to: predict the label of structure give the board (EMP-R); predict the label of structure given the 32 pairs (structure, label) and the associated rule (EMP-C); predict the rule given the 32 pairs (CG); predict the label of a structure given rule (Z).
0.1 to prevent overfitting.
C EFFICIENCY
Data efficiency In the Odeen challenge, CRNs require less training data to match the performance of empiricist models. For instance, in the case of 1438 rules at training, we see in Table 33 that the CRN trained 100 structures per rule (NRS= 40, 2%) still overcomes the performance of empiricist models trained on a dataset 100 times bigger (NRS- = 35.2% on 10k structures per rule).
Computational cost. In this section, we discuss the computational cost at test time of the ratio- nalist and empiricists approaches. We discuss the results reported in Tables a respectiveell for tagging 8 new structures
textual rule from the board. We evaluate the cost per rule in two ways: i) by counting the number of calls of each trained neural network and ii) by measuring the absolute time in seconds of each method with the same hardware configuration.
for explicitly predicting
We refer to the first quantity as the Computational Cost and parametrize it in terms of the main blocks of the models. This value is independent of the batch size and the hardware adopted. As an example, the cost of tagging the new structures for CRN using 300 conjectures is given by:
Table 3: T-Acc and NRS for different training regimens.
TRAIN DATA
MODEL
NRS
T-Acc
TRAIN DATA
MODEL
NRS
T-AcC
CRN
0.813
0.984
CRN
0.354
0.932
10K STRUCT. 1438 RULES
10K STRUCT.
EMP-C
0.352
0.930
EMP-C
0.095
0.869
500 RULES
EMP-R
0.179
0.895
EMP-R
0.068
0.863
CRN
0.777
0.980
CRN
0.319
0,930
IK STRUCT. 1438 RULES
IK STRUCT. S00 RULES
EMP-C
0.225
0.905
EMP-C
0.088
0.874
EMP-R
0.156
0.898
EMP-R
0.084
0.876
CRN
0.402
0.939
CRN
0.109
0.883
100 STRUCT. 1438 RULES
100 STRUCT.
EMP-C
0.125
0.865
EMP-C
0.057
0.823
500 RULES
EMP-R
0.163
0.896
EMP-R
0.117
0.872
15

---

Under review as a conference paper at ICLR 2022
300 CG + 300 I + I.
Where 300 CG stands for the 300 beams used get 300 conjectures from the conjecture generator CG. Each conjecture (300) is then tested on all the board structures (b) by the interpreter I. Finally I called to apply the chosen conjecture on each new structure (8). As an upper bound, an exhaustive search algorithm (Exv src) uses no conjecture generator, and thus has to evaluate all admissible rules (r) on each structure on the board with Z. Conversely, the empiricist approach provide label predictions through single end-to-end model which is simply called times. Concerning the problem of inferring explicitly the textual rule, using more beams in the empiricists models does not provide any increase in performance, ie. the true rule is not more probable proposition accessible through a larger beam search.
We measured also the absolute time in seconds with the following hardware configuration for all the experiments: single core hyper threaded Xeon CPU Processor with 2.2 Ghz, 2 threads; 12.7 GiB. of RAM; Tesla T4 GPU, with 320 Turing Tensor Core, 2,560 NVIDIA CUDA cores, and 15.7 GDDR6 GiB of VRAM.
Table 4: Computational cost of models at test time to tag structures. In Odeen r=24,794, 1=32, 1=,,16 Notice how CRNs offer good balance between computational efficiency and per- formance, this trade-off is regulated by a single parameter, the number of beams.
MODEL.
COMPUTATIONAL COST
T (S)
NRS
SRC
y. --I 8-
47.9
0.99
CRN [300a]
300 CG + 300 b I
0.79
0.81
CRN [10m]
10 CG + 10 b. I + I
0.43
0.35
EMP
8. EMP-R
0.15
0.35
Table 5: Computational cost of our models at test time to produce the textual rule in output
MODEL
COMPUTATIONAL COST
T (S)
R-ACC
EXV SRC
b. 300 - CG - 300 b.
47.8
0.99
CRN [300a]
0.72
0.77
CRN [108]
10 CG - 10 b.
0.35
0.35
EMP [3008]
300- EMP-C
0.41
0.07
EMP [108]
10- EMP-C
0.10
EMP [1B]
1. EMP-C
0.10
0.07
D ODEEN EXAMPLE GAMES
In this section we propose a collection of qualitative results showing series of Odeen games from the test set and how they are solved by the proposed models. For each model, report the pre- dicted rule (if output by the model), the accuracy on the structures labeling (T-acc), and mark that indicates whether the nearest rule is the correct (NRS). All the models are trained 10000 structures with 1438 rules.
16

---

Under review as a conference paper at ICLR 2022
X
IAAAA
X
Board
Golden Rule: "at least 2 pyramid pointing down" CRN: "at_least pyramid coantingddowni T-ace 1.0 / EMP-C: let_l 1 pyramid touching touching"; T-acc: 0.76 X
EMP-R: T-ace 0.72 X
Board 04
Golden Rule: "at_most blue pyramid pointing-upi
CRN: "zero blue at-most 1 blue pyramid pointing-up uu::: T-acc 1.0 v
EMP-C: "zero blue touching or or": T-ace: 0.89 x
EMP-R: R: T-ace 0.92 v
Board 09
Golden rule: "exactly 1 pyramid pointing up touching red pyramid pointing down"
CRN: "exactly red pyramid pointing down touching pyramid pointing up"" T-ace 0.95 x EMP-C: "exactly 1 red at_the_right_of and red",, T-ace: 0.80 x
EMP-R: T-ace 0.79. x
17

---

Under review as a conference paper at ICLR 2022
MIVA
X
Board 25
Golden rule: "at least 2 red touching blue pyramid pointing.doe down" CRN: "at_least red touching blue pyramid pointing.down", down", T-acc 1.0 /
EMP-C: let_t 2 red touching blue pyramid", T-acc: 0.87 x
EMP-R: T-ace 0.69 X
X
X
A.
Board 30
Golden rule: "exactly blue pyramid touching blue block"
CRN: "exactly blue pyramid touching blue block", T-ace 1.0 V EMP-C: "exactly 1 blue pyramid touching block block", T-ace: 0.97
EMP-R: R: T-ace 0.79 x
X
Board 75
Golden rule: "zero blue touching red pyramid"
CRN: "zero blue touching red pyramid"', T-ace 1.0 /
EMP-C: "zero blue touching red", T-ace: 0.85 X
EMP-R: T-ace 0.91 /
18

---

Under review as a conference paper at ICLR 2022
Board 97
Golden rule: "at_most red block touching red" CRN: "at_most I red block touching red", T-ace 1.0 / EMP-C: "at most red touching at the right of red", T-ace: 0.98 v
EMP-R: T-ace 0.93 x
Board 103
Golden rule: "at_most blue pyramid pointing touching red" CRN: "at_most I blue pyramid pointing_dow touching red"", T-acc 1.0 / EMP-C: "at most blue pyramid pointing down touching red", T-ace: 0.98
EMP-R: T-ace 0.85 x
19


Reviews:
[{"review_id": "tqDYuVHUrFS", "reviewer": "Reviewer_VRSC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: 1. The proposed framework forces us to think about explanations and generalization as first class citizens. \n\nWeaknesses: \n1. The idea sounds very familiar to explanation based learning. It will be good to contrast the two.\n2. My main criticism is that I am not sure we need an entirely new framework for targeting explanation based solutions. Many recent work in NLP try to frame problems in English to get cross task generalization. I would prefer this paper to be positioned as an improvement to such existing approaches, as this is tackling a harder problem class. \n3. The experimental results are all on the new game like dataset. It will be good to see performance of proposed methods on real datasets, or already existing synthetic datasets. The baselines are also weak, so its not clear if the proposed techniques will perform better than other cross task generalization methods.  ", "summary_of_the_paper": "The paper proposes a new framework for studying explanation driven machine learning problems called Explanatory Learning. The goal is to learn an interpreter model from explanations paired with observations for a particular phenomenon. The explanations might be in an unknown language, but the explanations paired with observations can be used to learn a good interpreter. Once learnt, the interpreter should be able to follow new explanation for an unseen phenomenon. They refer to this problem as the communication problem. They also define the scientist problem where explanations for the unseen phenomenon is not available. The paper also introduces an Odeen dataset to facilitate experiments with Explanatory Learning. The authors propose a neural network architecture as a solution to the scientist problem. It has two components: a conjecture generator which generates a set of candidate English explanations, and a second interpreter model. Experimental results show better generalization compared to end to end neural systems.  ", "summary_of_the_review": "Although I am all for explanation based learning solutions, I don't think the components introduced in this paper warrants introduction of a new learning framework. I think it will be better to place it as an improvement to existing approaches for cross task generalization. I also found the experimental results to be weak. It was not clear if the proposed techniques will perform better than other cross task generalization methods.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qs3d7g_Lg-R", "reviewer": "Reviewer_PMcp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- Acknowledges the bilateral nature of explanations in building explainable AI/ML.\n- Offers a new benchmark problem.\n\nWeaknesses:\n\n- Does not fully explicate the assumptions that it is making in terms of the explanations.\n- Does not properly position itself with respect to relevant work in the literature.\n- The somewhat over-the-top philosophical discussion distracts from the essential point.\n\nGeneral remarks:\n\nThe problem acknowledges the need to have bilateral explainability; i.e., build machines that can explain, by first explaining *to* the machines. This is discussed, for example, by Michael, \"Machine Coaching\", IJCAI Workshop on XAI 2019, which seems to be rather relevant to the current paper, especially given that the paper makes an effort to connect to learning theory. Another line of work that seems to be relevant is Mozina et al., “Argument Based Machine Learning”, AIJ 2007, where data points are accompanied by an argument (i.e., an explanation) on why they are labeled as they are, as is the case in the current paper. Ignoring the obvious difference from the current work that these two works assume that explanations are in logic, the underlying theme of the cited papers and the current paper seems considerably close to ignore.  \n\nI found the philosophical positioning of the paper as a study in empiricism vs rationalism to be somewhat far-fetched. I believe it is instructive to take a step back and seek what are the underlying assumptions of the paper, and whether those bring something new to the table. \n\nIt is clear that the label of a data point is a function of the data point and the phenomenon, and not only of the data point. And since the phenomenon is essentially associated with an explanation, then the label is a function of the data point and the explanation. What the authors propose, then, as the pipeline for predicting the label is a form of chaining of learned pieces of knowledge, where one predicts the explanation, and then using that previous prediction one proceeds to predict the label at a second cycle of inferencing. Of relevance here is the work by Valiant, \"Robust Logics\", AIJ 2000, and Michael, \"Simultaneous Learning and Prediction\", KR 2014, as well as some follow-up papers by the authors, that establish the benefits of chaining.\n\nFrom a formal point of view, there is nothing in the explanations that makes them explanations. As the paper says, they are simply strings. Is it really the case that they are arbitrary strings, or should they be strings that are learnable as a function of the data points (or more properly, sets of data points)? That is to say, if each type of explanation for a phenomenon P_j was simply replaced by the number j, would then this still be considered an arbitrary language of explanations, despite having no structure and not being learnable? If indeed, explanations cannot be arbitrary strings, then one needs to carefully state what are the underlying assumptions on the language of explanations. If, on the other hand, explanations can be arbitrary (e.g., explanation j for phenomenon P_j) then this makes the term \"explanation\" rather mood; it is simply another signal in the data (at a meta-level, see my comment below). \n\nAnother aspect of the paper is the two levels of learning problems that exist, as mentioned in the last two paragraphs: the object-level learning problem of learning from a data point and its explanation the label of the data point; and the meta-level learning problem of learning from sets of data points to predict the phenomenon/explanation. Both of these two problems seem to follow the empiricist view, in the sense of the paper, which, as I said above, makes it unclear what the philosophical discussion on empiricism vs rationalism really add to the picture. The conjecture generator seems to be simply a meta-level classifier (albeit a stochastic one). Which brings me back again to the question on whether the explanations need to have some learnable structure.\n\nAdditional points:\n\nThe metric of NRS seems to include in its definition the identification of a nearest neighbor. Why? Shouldn't the goal be to predict the actual explanation? If the algorithm that makes the prediction wishes to use the nearest neighbor to reach that decision, then that would be fine. But the use of the nearest neighbor seems more natural to be part of the algorithm that attempts to solve the problem, not part of the evaluation metric for measuring success.\n\nThe proposed approach to solving the problem seems not to be accompanied by any formal guarantees on its performance. This would typically be compensated by an extensive experimental section, which is not the case here. \n\nI would have found a different narrative for this paper to be more convincing and impactful: the introduction of Odeen as a benchmark problem, and a deeper discussion of its features and parameters, and then present the particular approach, properly placed in the context of relevant work, as a suggested direction for what type of systems would presumably be useful in tackling the Odeen benchmark. \n\n------- After the Author Rebuttal -------\n\nI acknowledge that the authors have made an effort to engage with the points that I raised, and I have increased my score. I believe that the connections with learning theory are much more deep than the brief remarks offered in the revised version of the paper, and I hope that the authors will consider exploring them further in their future work as a formal underpinning of their empirical work. \n", "summary_of_the_paper": "The paper considers the problem of classification learning where each data point is accompanied by some explanation. The explanation is in some arbitrary language. The paper proposes a way to increase the performance of the classification task by learning to predict the explanation associated with a given data point, and then using that explanation along with the data point to proceed to make a prediction about the data point's label. The paper implements this pipeline using neural networks, and presents empirical results on a new benchmark dataset to demonstrate its performance.  \n", "summary_of_the_review": "I find Odeen to be a useful contribution, and one that would raise awareness on the need of certain underused techniques in the machine learning literature. But, the rest of the paper needs to be more clearly and properly placed in the context of existing work in the literature.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JuWRoCKfuG2", "reviewer": "Reviewer_ZgwB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The paper is a pleasure to read.\n\n\n- The choice of illustrative examples is excellent.\n\n\n- The contribution is conceptual in nature, which is good.  Alas, the motivation for pursuing this research direction is unclear.  Specifically:\n\n1) It is not clear why the machine should estimate *both* the link between sentences and data and between sentences and sentences *jointly*.\n\n2) It is not clear why the lanugage would be unknown and unstructured.\n\n3) It is not clear what (new?) real-world applications this setup is meant to capture.\n\n(Please note that this setup does not solve the \"semantic gap\" problem, because the semantics of the sentences are implicitly supplied by whoever designs the data set.)\n\nI would advise the authors to ground their motivation in concrete conceptual problems and/or applications.\n\nGiven that the learning framework is *the* key contribution of the paper (the value of the two other major contributions depends on whether the learning setting makes sense), the fact that motivation is lacking/unclear is a serious concern.\n\nThis makes it hard to evaluate the significance of the paper.\n\n\n- The terminology used in the paper feels somewhat misleading/inappropriate.\n\nAs far as I can see, \"phenomena\" are simply *concepts* and the \"explanations\" are intensional *descriptions* thereof, except in a language unknown to the machine (but known to the annotator).  I do not understand what is gained by using this terminology.  Indeed, I find the latter confusing and I expect other readers to be confused by it too.\n\nIn particular, I do not understand the link between \"description\" and \"explanation\", especially considering the causal connotations of the latter term (which are becoming more and more clear as work on causality is being merged into AI and ML.)\n\n\n- The work seems to rely on a rather strict assumption.  In particular, the fact that D_0 should be discriminative for P_0 in L is (as far as I can see) unlikely to hold in practice -- especially considering that D_0 is supposed to contain a \"small set of observations\" -- and it simplifies the learning problem considerably.  In what applications is it reasonable for this asumption to hold?  Is it necessary?  What happens if it doesn't hold in practice?  How costly is it to acquire a D_0 that explicitly satisfies this assumption?  Given that such a D_0 would not be IID, how would this impact statistical learning of P_0?  These issues are touched upon in the conclusions, but they deserves an actual discussion.\n\n\n- The proposed architecture is reasonable but surprisingly involved and the details are hidden in the appendix.  It would be more straightforward to explain in detail the various pieces that make up the architecture (how many, what they take as input and what they spit out) from the get-go, rather than relying on Figure 3, which lacks mathematical precision.\n\n\n- Inferring whether an instance x satisfies a concept P is surprisingly involved as it requires to generate a (presumably large) number of candidate descriptions for P and then counting, for each description, whether x satisfies it.  Presumably this scales poorly with language complexity.  Is this efficient at all?\n\n\n- The experiments are limited to an interesting but entirely synthetic (actually, quite toy) setting.  For instance, as far as I can tell no sub-symbolic inputs are present.  Experiments with real data would have been useful to evaluate the efficacy of the proposed pipeline.  CRNs are compared only against two simpler baselines.  I realize that the focus of the paper is in its conceputal contribution, but a more varied selection of experiments would have been welcome.\n\n\nMinor issues\n------------\n\n- Wouldn't it be more natural to define the interpreter as a map from descriptions to classifiers (indicator functions)?\n\n- It would be good to disambiguate the term \"explanatory learning\" from previous uses of the same term, see \"Explanatory interactive machine learning\" AIES 2019.\n\n- The \"communication problem\" shares some aspects with multitask learning, especially so if the language is compositional and the tasks form a hierarchically (or can be related logically to each other, e.g., P0 is the negation of P1 etc.).  It also shares aspects with few-shot learning.  It may be worth highlighting the connection.", "summary_of_the_paper": "The authors introduce a novel learning framework that revolves around learning a map between concepts and their description, where the latter are expressed in a language unknown to the learner.  Two learning problems are defined: (i) learning a new concept from a description of that concepts as well as examples and descriptions of other concepts, (ii) learn a new concept from examples of that concept as well as examples and descriptions of other concepts.  The underlying assumption is that the map (interpreter) from sentences to sets of examples is shared.  The authors propose an environment to evaluate these tasks and a neural architecture to tackle them.  ", "summary_of_the_review": "Potentially great paper with unclear motivation/applications", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "C-6LEcN_gZ0", "reviewer": "Reviewer_H4ra", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Questions: \n- In Figure 1, why the first left sequence is correct according to the rule? \n- When tagging unseen structures based on the secret rule discovered by the user, how many unseen structures are required to completely rule out other rules that are marginally different from the secret one. Is this number always 1176 in the current setting? \n- Since some figures don’t have caption, it’s hard to refer to them… In the figure on the lest side of “Metrics.” How the predicted vector is derived based on various rules? \n- As the authors point out the goal here is analogous to that of IPL. Do I understand this correctly that the main advantage of EL to IPL is skipping the translation of the data to logic? If that’s the case, one can learn the translation itself and eliminate the difficulty. Can you elaborate on the advantage of EL over IPL?\n- Can’t T-Acc be changes to something auc based instead that takes care of the permissively problem? \n\n\nOther remarks: \n- This is a language specific problem, where order of the sequence does not necessarily matters. The current title does not  reflect this. Perhaps something like Beyond empiricism in neural language learning or something alike would be a better suit. \n- Some approaches to explanatory AI that aim at generating explanations as well as predictions such as “TED: Teaching AI to Explain its Decisions” are missed here and should be commended on. ", "summary_of_the_paper": "The paper proposes approaching learning a language as a learning problem that is grammar, alphabet, etc. agnostic. Such formulation has resulted in the so called Explanatory Learning (EL) framework that is paired up with an environment to test it. The starting point for learning a language is to extract an interpreter based on a given set of observation and their explanation and then use the interpreter to determine whether an observation belongs to the language, in a binary classification setting. ", "summary_of_the_review": "- Overall, I quite like the idea of the paper due to the generality of the approach, namely learning an interpreter from observation. \n\n- “proving” the discovered rule is indeed the secret one by examining it in practice (i.e., being used to tag unseen structure) as opposed to revealing the rule directly in text is also a very neat idea.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an explanation-based learning approach that learns from both observations (examples) and explanations paired with examples. It proposes to learn an interpreter that can map from natural language sentences to examples. The authors also develop an evaluation environment and protocols for the tasks.

Strengths:
- The proposed idea is intuitive and seems general
- The benchmark dataset is useful resource for future research

Weakness:
- The motivation of the present problem setup needs more justification
- The close connection to the line of work on explanation-based learning (especially recent ones in modeling natural language explanations) are not thoroughly discussed and compared.
- Experiments beyond the game-like datasets will help validate the claims better and justifies that the problem setup has real-world applications
Decision: Reject