Hidden Schema Networks
Year: 2023

Abstract:
Large, pretrained language models infer powerful representations that encode rich semantic and syntactic content, albeit implicitly. In this work we introduce a novel neural language model that enforces, via inductive biases, explicit relational structures which allow for compositionality onto the output representations of pretrained language models. Specifically, the model encodes sentences into sequences of symbols (composed representations), which correspond to the nodes visited by biased random walkers on a global latent graph, and infers the posterior distribution of the latter. We first demonstrate that the model is able to uncover ground-truth graphs from artificially generated datasets of random token sequences. Next, we leverage pretrained BERT and GPT-2 language models as encoder and decoder, respectively, to infer networks of symbols (schemata) from natural language datasets. Our experiments show that (i) the inferred symbols can be interpreted as encoding different aspects of language, as e.g. topics or sentiments, and that (ii) GPT-like models can effectively be conditioned on symbolic representations. Finally, we explore training autoregressive, random walk “reasoning" models on schema networks inferred from commonsense knowledge databases, and using the sampled paths to enhance the performance of pretrained language models on commonsense If-Then reasoning tasks.

heading: 1 Introduction
text: Much of the developmental and causal theories of human cognition are predicated on relational structures of knowledge that naturally exhibit compositionality. Semantic content is intrinsically relational, as one is only able to explain a given unit of knowledge – such as a concept, word or perception – insofar as there are other units of knowledge which relate to it (Block, 1986; Margolis and Laurence, 1999). Thus we can partially construe a concept through its relationships to other concepts
(like when we say “a dog is an animal that barks”), just as we can partially construe it through its relationships to our perceptions (when we say “that is a dog”, whilst pointing to a dog on the street) or the words we use (when we use the word dog to refer to the concept dog). Likewise, we can partially construe words not only through their relationships to concepts or percepts, but also through their relationships to other words, as words that occur in the same context tend to have similar meanings (Harris, 1954; Firth, 1957). Note that is precisely this contextual semantic word content what we explicit have access to, when processing our raw text datasets. On the other hand, generalization, reasoning and understanding seem to be inevitably tied to the compositional nature of knowledge. Indeed, the ability to compose a set of knowledge units (and their relations) into new, more complex relational units, which can be deployed to understand and reason about unseen data – a feature usually referred to as combinatorial generalization – is regarded as key to human-level intelligence (Fodor and Pylyshyn, 1988; Fodor and Lepore, 2002; Lake et al., 2017; Battaglia et al., 2018). Relational structures allowing for compositionality thus seem to comprise not a sufficient, but a necessary attribute of any representation scheme that strives for the generalization power of human cognition.
From the computational side, if one is to inform any modern language model with such structural characteristics, one will initially encounter the problem of finding suitable primitives or data structures. Distributed continuous word representations (Bengio et al., 2003), for example, are routinely leveraged in many different downstream tasks. These representations are trained to encode average contextual semantics – precisely the kind of semantic content typical of word co-occurrence relations we mentioned above – into a semantic space, which allows meaning to change continuously within it (Mikolov et al., 2013). Yet, de-
4764
spite earlier attempts (Mitchell and Lapata, 2008), it is unclear whether such representations can be meaningfully composed into representations of, say, unseen sentences and thus mimic the compositional character of natural language. More recently, contextualized continuous word representations inferred by deep learning architectures have shown spectacular results in many NLP tasks (Radford et al., 2018; Devlin et al., 2018; Radford et al., 2019; Brown et al., 2020). Their success stems from those models’ ability to infer flexible representations through, inter alia, raw, massive datasets, data-scalable attention mechanisms and minimal inductive biases (Vaswani et al., 2017). These representations are known to not only contain rich contextual word semantics, but also consistently encode sentence-level grammar (Hewitt and Manning, 2019), and the models from which they are obtained have been shown to implement different notions of compositionality too (Hupkes et al., 2020; Wei et al., 2022). Nevertheless, it is still unclear whether such representations can be composed into representations of novel sentences (Yu and Ettinger, 2020; Bhathena et al., 2020). In fact, most of their syntactic properties are implicit and therefore inferred only a posteriori, typically through probes which neither guarantee their presence, nor establish how they were obtained in the first place (Liu et al., 2019; Rogers et al., 2020; Ravichander et al., 2021).
Roughly following the program outlined by Tenenbaum et al. (2011), we develop a novel, neural language model – the Hidden Schema Network model (HSN) – that enforces, via inductive biases, a discrete, relational structure for sentence representation which allows for compositionality1 onto the output representations of large, pretrained language models (LPLM). Using a variational autoencoder (VAE) framework (Kingma and Welling, 2013; Rezende et al., 2014), HSN leverages LPLMs to encode natural language sentences into sequences of symbols, which correspond to the nodes visited by biased random walkers on a global latent graph, while inferring the posterior distribution of the latter.
In practice, translating the implicit knowledge encoded by LPLMs into explicit relational struc-
1Note that throughout the paper we refer only to compositionality of representations and not to the compositional functions that can be implemented by the models we use. The latter, functional compositionality, is studied by e.g. Hupkes et al. (2020).
tures has some naturally appealing properties. For example, HSN can support symbolic reasoning via inference of missing semantic connections, through high-order paths along the inferred graphs (Lao et al., 2011; Das et al., 2018; Chen et al., 2020). Likewise secondary, autoregressive “reasoning” models can be trained on the symbol sequences inferred by HSN from task-specific natural language sequences, like e.g. question-answer pairs. The reasoning paths sampled from such autoregressive models could then be used to inform GPT-like models and improve their performance on natural language understanding tasks, with which they are known to struggle (Brown et al., 2020; Dunietz et al., 2020; Talmor et al., 2021). Below we demonstrate that (i) HSN is able to uncover ground-truth graphs from artificially generated datasets of random token sequences, and that (ii) using pretrained BERT and GPT-2 language models as encoder and decoder, respectively, HSN is able to infer schema networks from natural language datasets, whose symbols encode different aspects of language (like e.g. topics or sentiments). Finally, we also explore training secondary, autoregressive models on the schema networks inferred from commonsense knowledge databases, and using the sampled paths to enhance the performance of LPLM on commonsense If-Then reasoning tasks.

heading: 2 Related Work
text: In cognitive psychology, a schema is (roughly) defined as a large, complex unit of knowledge representing what is typical of a group of instances (Bartlett, 1932; Piaget, 1948; Rumelhart, 2017). Marvin Minsky’s frames (Minsky, 1974, 1975) are similar in function to a schema, but perhaps more easily characterized in terms of data structures. We use these terms in a loose fashion, however. Our aim being only to be suggestive of the general problem of knowledge representation (Thagard, 1984). We are in fact concerned with representation schemes for natural language processing. Within the context of linguistics, Jackendoff (1978) argues that there must be a level of representation – the so-called conceptual structures – at which information conveyed by language must be compatible with information coming from sensory systems. Conceptual structures must, he goes on, be able to represent all the conceptual distinctions made by natural language, and provide some degree of compositionality. Earlier computational models
implementing (some kind of) conceptual structure rely on either hand-coded (semantic) network representations (Quillan, 1966; Collins and Quillian, 1969; Brachman, 1977) or hand-coded databases (McClelland and Rogers, 2003). Other works focus instead on learning semantic representations from raw text data directly, via topic models (Griffiths et al., 2007b), and even infer latent concept graphs through nonparametric priors (Chambers et al., 2010). In sharp contrast with these works, (a line of) modern, neuro-symbolic reasoning approaches in NLP seek to combine the implicit knowledge of LPLM with external information, structured as knowledge graphs, in order to solve commonsense question-answering tasks (Lin et al., 2019; Ma et al., 2019; Yasunaga et al., 2021; Verga et al., 2021; Kapanipathi et al., 2021; Hu et al., 2022b). Note that the knowledge graphs here can also be interpreted as (some class of) human-authored conceptual structure (see however West et al. (2021), who shows that LPLM can generate commonsense knowledge graphs too). Our work runs perpendicular to these approaches, for it leverages LPLMs to automatically discover, in an unsupervised fashion, knowledge graphs in representation space, and introduces reasoning via random walk processes conditioned on the discovered graphs. This problem necessarily involves the (neural) inference of discrete variables, which has also been successfully treated in the past (van den Oord et al., 2017; Hu
et al., 2017; Zhao et al., 2018; Kaiser and Bengio, 2018; Kaiser et al., 2018; Stratos and Wiseman, 2020).

heading: 3 Hidden Schema Networks
text: We address the problem of learning the joint probability distribution over sequences of words, while inferring symbolic representations capturing their semantics. Neural autoregressive language models approximate such distributions with a product over conditional probabilities, such that
p(x1:T ) = T∏
i=1
pθ(xi|x<i), (1)
where x1:T = (x1,x2, . . . ,xT ) labels the sequence of words in question, and each conditional is given by (the pdf of) a categorical distribution over some vocabulary of size V . The class probabilities of these conditionals are generally computed as πi = softmax(W ·hθ(x<i)), with W ∈ RV×D trainable, andD the output dimension of hθ, a deep neural network model with parameter set θ (Bengio et al., 2003). Models of this form allow for tractable estimation of and sampling from either the joint distribution, or any product of the conditionals in Eq. 1. Indeed, their recent implementation in terms of large-capacity, self-attention architectures, the likes of GPT-3 (Brown et al., 2020), has been shown to generate syntactically correct, diverse and
fluent text. In what follows we attempt to condition the joint distribution of Eq. 1 on an additional latent, discrete representation which can, at least in principle, capture the relational and compositional features of semantic content.
Let us assume there is a set of K symbols, E = {e1, e2, . . . , eK}, each of which encode some high-level, abstract semantic content of natural language. Let this set be the set of nodes of a hidden (semantic) graph G, with adjacency matrix A, so that adjacent (connected) symbols are semantically related. These symbols can generically be defined as learnable, dense vectors in RS , for some dimension S. Without loss of generality, however, we opt below for simple indicator (“one-hot”) vectors of dimensionK instead. We define a schema ej1:jL as a sequence of L≪ K symbols (ej1 , ej2 , . . . , ejL), where the indices j1, . . . , jL label a subset of connected nodes in G. Accordingly, we refer to G as a schema network. The symbols composing the schemata are chosen through a L-step stochastic process conditioned on G. Partially motivated by research on random walks and human memory search (Griffiths et al., 2007a; Abbott et al., 2012), as well as by the simplicity of their inference, we choose to compose the schemata via biased random walk processes on G, and leave exploring different schema processes for future work. Finally, let us assume that natural language sentences are generated conditioned on these schema representations. We can then define a generative language model as follows.

heading: 3.1 Generative Model
text: Let the joint probability over a sequence x1:T of T words, together with the hidden graph G, be
pθ(x1:T ,A) =∑
z1:L
pθ(x1:T |ej1:jL)p(z1:L|A)p(A), (2)
where z1:L labels the sequence z1, . . . , zL of Kdimensional, one-hot vectors representing the node labels j1, . . . , jL visited by a random walker on G, and θ denotes the trainable model parameters. Note that we introduced the one-hot representation of ji for notational convenience, as shall become evident below2. Next, we specify the different components of Eq. 2.
2Explicitly, ji denotes the index of the non-zero component of zi, i.e. ji = {k ∈ [1,K] : zki = 1}, with the superindex k denoting the components of zi.
Prior over (global) graph. A prior on the adjacency matrix p(A) allows us to control the topological properties of G. One can choose, for example, random graph models whose degree distributions asymptotically follow a power law (Barabási and Albert, 1999), or unbiased, maximum entropy graph models, with respect to some given constrains (Park and Newman, 2004). Here we choose a Bernoulli (Erdös-Rényi) random graph model (Solomonoff and Rapoport, 1951; Erdös and Rényi, 1959), for which each link is defined via an independent Bernoulli variable with some fixed, global probability p ∈ [0, 1], which is a hyperparameter of our model.
Prior over random walks. The probability p(z1:L|A) of a random walk over the nodes of G can generally be written as
p(z1:L|A) = p(z1) L∏
i=2
p(zi|zi−1,A)
=
( K∏
m=1
ρ zm1 m
) L∏
i=2
  K∏
j=1
K∏
k=1
P zki z j i−1
k,j
  , (3)
where p(z1) labels the probability of selecting j1 as the starting point of the walk, and it is given by (the pdf of) a categorical distribution over the nodes of G, with class probabilities {ρi}Ki=1. Similarly p(zi|zi−1,A) labels the conditional probability of jumping from ji−1 to ji, which we define in terms of a K ×K transition probability matrix P. Now, to allow for biased random walks, let each node k on G be given a positive weight fk, so that the probability of jumping from j to k is proportional to fk Akj . We then write the transition probability matrix as
Pk,j = fk Akj∑K i=1 fiAij , (4)
so that the motion of the random walker is biased according to the node weights fk. These weights should be understood as encoding aspects of the diffusion dynamics that are independent of the topology of the graph (Gómez-Gardenes and Latora, 2008; Lambiotte et al., 2011). Three comments are in order: first, note that one can also train the prior over walks by making the vectors ρ and f learnable. Second, setting the node weights f = I and the class probabilities ρ = 1K I, with I the K-dimensional vector of ones, yields a uniform random walk over G, i.e. a process in which the walker has equal probability of jumping to any of
its neighbors. Third, one can also allow for inhomogeneous random walks in which the probability matrix changes at each step of the random walk. Such processes can be parameterized with a sequence of weights f [1], f [2], . . . , f [L−1].
Decoder and likelihood. Just as in Eq. 1 we define the joint probability over word sequences as a product of conditional probabilities, each of which is now conditioned on the schema ej1:jL too. Accordingly, the class probabilities of the ith conditional are now given by πi = softmax(W · hdecθ (x<i, ej1:jL)), with W ∈ RV×D trainable and hdecθ a deep neural network model. We let h dec θ be a pretrained GPT-2 language model, and modify it to also process the schema ej1:jL , but remark that any other model for sequence processing (as e.g. a recurrent neural net) could be used instead. A bit more in detail, to condition GPT-2 on ej1:jL , without perturbing its optimized weights too much, we use the pseudo-self -attention (PSA) mechanism introduced by (Ziegler et al., 2019). In a nutshell, this mechanism augments the key and value matrices of GPT-2 in their first L rows with projections of ej1:jL . Figure 1 shows an illustration of the complete decoder model, including the PSA mechanism. Please check Appendix A for the explicit equations of the latter.

heading: 3.2 Inference Model
text: The generative model we presented above is hierarchical. The random graph is shared across all sentences and thus constitutes a global latent object. The random walks, in contrast, are local random variables. Our task is to infer the schema and graph posterior distributions that best describe the collection of word sequences in our datasets. To do this, we approximate the true posterior distribution of these variables with a variational posterior of the form
qϕ(z1:L,A|x1:T ) = qϕ(z1:L|x1:T ,A)qϕ(A), (5)
where ϕ labels the set of trainable parameters. We model qϕ(A), the posterior over the graph, assigning again Bernoulli variables to its links, but we let the probability of observing each link depend on the global symbols. That is, we model the link probabilities as pϕ(ei, ej) = sigmoid(gϕ(ei, ej)), for all ei ∈ E , with gϕ : E × E → R a deep neural network.
Likewise we model qϕ(z1:L|x1:T ,A), the posterior probability over random walks on the graph,
analog to Eq. 3. This time, however, instead of having a single transition probability matrix, we have a sequence of them, thereby allowing the posterior to capture inhomogeneous random walks. We therefore model the class probabilities over the starting point of the random walks as ρ(x1:T , ϕ) = softmax(henc1 ), whereas the sequence of transition matrices is given by
Q [i] k,j(x1:T ,A, ϕ) = f [i] k (x1:T , ϕ)Akj∑
m f [i] m (x1:T , ϕ)Amj
, (6)
with the sequence of node weights f [1], f [2] . . . , f [L−1] = exp(henc2:L). Now, the set h enc 1 ,h enc 2 , . . . ,hencL ∈ RK is the sequence of outputs of a deep neural network model hencϕ (x1:T ) processing the input sequence of T words. The model hencϕ (x1:T ) must then map a sequence of T vectors to a sequence of L vectors. We define hencϕ by a pretrained BERT model (Devlin et al., 2018), followed by a single Transformer block, randomly initialized. The Transformer block processes the T (D-dimensional) outputs from BERT as keys and values, together with a set of L learnable vectors q1:L as queries. The right-hand side of Figure 1 illustrates the complete encoder architecture. For completeness, we also give the explicit and complete expressions of both posterior graph and random walk models in Appendix C.

heading: 3.3 Training Objective
text: To optimize the parameter sets {θ, ϕ} of our latent variable model we would, as usual, maximize a variational lower bound on the logarithm of the marginal likelihood pθ(x1:T ) (Bishop, 2006). It is, however, well known that VAE models tend to encounter problems learning representations encoding information about the data – the so-called posterior collapse problem – especially when dealing with natural language (Bowman et al., 2015). To solve this issue practitioners resort to maximizing the variational lower bound, together with the mutual information between data and representations (Zhao et al., 2018; Fang et al., 2019; Zhao et al., 2019). We follow this same route here and refer the reader to Appendix D for the derivation of our objective function. The training algorithm is presented in Appendix B and training times in Appendix I. Source code to reproduce all experiments is available online.3
3Source code: https://github.com/ramsesjsf/ HiddenSchemaNetworks

heading: 4 Inferring Ground-Truth Graphs
text: Before testing the behavior of our methodology on natural language data, we evaluate the ability of the model to infer hidden graph structures from sequential data in a controlled experiment. To this end, we define a synthetic language model with an underlying, ground-truth graph G∗ as follows: Given a graph G∗ with K nodes, and a vocabulary of random tokens V of size V , we assign one random bag of tokens (i.e. one pdf over V) to each node of the graph. Let the K random bags be the set E of K symbols {e1, e2, . . . , eK} of the synthetic language model. We then sample N uniform random walks of length L over G∗, and sample one random token from each symbol (i.e. from each random bag) along the walks. The result is a set of random token sequences of the same length as that of the random walks.
A simple proof-of-concept. We consider a problem in which the set of symbols E is known, so that the ground-truth graph G∗ has a fixed labeling. This setting will allow for simple comparison between G∗ and our inferred graphs. Following the procedure above we generate two datasets from two ground-truth graphs with different topologies. One sampled from the Barabási-Albert model (Barabási and Albert, 1999), the other from the Erdös-Rényi model (Erdös and Rényi, 1959). We set both graphs to haveK = 100 symbols, and the token sequences to have length L = 10. Given these sets of random token sequences, the task is to infer the hidden, ground-truth graphs G∗. To solve it we first replace BERT in Fig. 1 with a 2-block Transformer encoder (Vaswani et al., 2017) and note that, since the symbols are known, the likelihood of the model is simply given by ∏L i=1 eji where, as before, ji denotes the index of the non-zero component of zi. We then train this simplified model by maximizing the HSN objective function. Further details about the graph model parameters, the dataset generation procedure and statistics, training procedure, and model sizes and hyperparameters can be found in Appendix E. The synthetic datasets are available in the source code too.
Results. Table 1 shows our results for our two synthetic datasets. Specifically, we compute the Area Under the Receiver Operating Characteristic Curve (ROC AUC) of our model qϕ(A) with respect to G∗, and the Frobenius norm between qϕ and two graphs: the ground-truth one G∗, and a second random graph Grand sampled from the same random graph model as G∗. We train ten (10) models in total and display the mean and standard deviation of our results. We also use a different Grand for each calculation run. The first metric shows that qϕ(A) correctly predicts the edges of G∗, whereas the other two metrics show that G ∼ qϕ(A) is closer to G∗, than to any other random graph sampled from the same distribution. The last two columns in Table 1 show however that qϕ(A) tends to generate denser graphs as compared to the target. This overproduction of edges can be understood as being caused by the prior graph model, which is chosen to be an Erdös-Rényi model (see Section 3.1). The expected number of edges of this model is ( K 2 ) p, where p is the prior edge probability. In our experiments we have K = 100, and choose p = 0.2 (0.6) when training the model on the Barabasi (Erdös) dataset (see Appendix E). The average number of edges of the prior graph model is therefore 990 (2970) for the Barabasi (Erdös) dataset. During training, the posterior model tries to fit the data, while simultaneously being close (in distribution) to the prior, which explains the overproduction of edges. Using different priors, like e.g. a sparse graph model, would reduce the average number of edges of the posterior graph distribution. See also Table 5 in Appendix E, where we report results from HSN with smaller prior edge probabilities.

heading: 5 Natural Language HSN
text: In the previous section we demonstrated that HSN can indeed infer hidden graph structure from sequential data in a simple setting. We now move on to our main problem: representation learning from natural language. We leverage pretrained BERT and GPT-2 language models as encoder
and decoder, respectively, which we finetune to discover schema networks encoding natural language datasets. We consider three widely used public datasets, namely the Penn Treebank (PTB) (Marcus et al., 1993), Yahoo and Yelp (Yang et al., 2017) corpora, and evaluate the quality of the inferred representations via the mutual information (MI) between schemata and data, and the perplexity (PPL) of the model. The latter is estimated through Monte Carlo samples. We compare HSN against a pretrained GPT-2, fine-tuned during a single epoch. We also compare against two VAE language models: iVAEMI (Fang et al., 2019) and Optimus (Li et al., 2020). The former implements both encoder and decoder as one-layer LSTMs (Hochreiter and Schmidhuber, 1997). The latter does it via pretrained BERT and GPT-2, just as HSN. Further details on the definition and computation of the evaluation metrics, as well as on hyperparameters and training procedures can be found in Appendix F. In what follows we explore HSNs with K = {50, 100} symbols and hidden random walks of L = {5, 20} steps. Let us label these configurations as HSN(K,L).
Results. As shown in Table 2, HSN achieves a much better performance than all baselines under both metrics, which implies that (i) HSN successfully encodes sentences into symbol sequences, and that (ii) our modified GPT-2 can effectively interpret symbolic representations. Longer random walks clearly have more capacity and, accordingly, encode more information. In contrast, shorter random walks perform better wrt. PPL, which illustrates the usual trade-off between language modeling and representation learning, typical of VAE (Li et al., 2020). We speculate the reason why the
structured representations of HSN help with language modeling might be, that they allow for a sequential composition of the sentences’ semantic components via reusable units. This feature might yield more efficient encodings than dense representations do. HSN also allows to infer (graph) connections not directly seen during training, that account for unseen symbol combinations, and hence (might account for) unseen sentences. We additionally report in Appendix F the averaged KL and likelihood values from five HSN, trained with different random initializations, against the baselines. Now we investigate the structure and semantics of the inferred schema networks.
Structure of hidden schema networks. We characterize the structure of the schema networks we inferred in terms of five statistics, namely their diameter, average distance, clustering coefficient, number of connected components and degree distribution. We define all these and report their computed values, for all HSN configurations and all datasets, in Appendix F. One observation we can immediately make about these results is that the schema networks discovered in each corpora tend to have smaller average distances, and much larger clustering coefficients, than any random graph of the same size, sampled from a maximumentropy (p = 0.5) Erdös-Rényi model. Interestingly enough, the combination of these two features defines the so-called small-world structure (Watts and Strogatz, 1998). Now, intuitively speaking, a larger clustering coefficient implies that a random walker starting from a given node “k” will have a larger number of paths bringing it back to “k”. In such an scenario, random walkers tend to cluster in neighborhoods around their starting point – a
property that could help encode different semantic aspects in different regions of G. Similarly, one could also expect schemata composed of repeated symbols. We close this subsection with Figure 2, which displays the difference between the degree distributions of the schema networks and that of purely random graphs.
Schemata and semantics. Taking advantage of the labels available to both Yahoo and Yelp, we display in Figure 3 the schema distributions over the schema networks for 4 subsets of these datasets. Similar plots for all subsets of both corpora, extracted with all HSN configurations, can be found in Appendix F. Note how the “hot” symbols per category reside on different regions of the graphs – as suspected already from the large clustering coefficient of G – and yet, the “Science & Math” schemata (both nodes and edges) of Yahoo are closer to the “Education & Reference” schemata than to the “Sport” schemata, where closer nodes in the figure indicate well-connected nodes in the underlying graph G. These findings (qualitatively) indicate that the schemata indeed encode semantic notions of their corpora. A similar picture holds for Yelp. We also introduce and explore “schema interpolations” in Appendix F.3, and observe that the translation procedure we employ successfully interpolates between categories of both Yelp and Yahoo. These experiments hint too at the different semantic notions encoded by HSN.
To get a more quantitative evaluation of the semantics encoded in the schema networks, one can also investigate how the distribution of attention weights, in the HSN decoder, and wrt. the symbols changes, as the decoder processes specific words. We exemplify this idea in Appendix G.

heading: 6 Commonsense Reasoning Generation
text: In the previous section we empirically showed that HSN is able to infer schema networks from natural language datasets, whose symbols encode different notions of semantics. Next, we explore training “reasoning” models on the schema networks inferred from commonsense knowledge databases, and using these secondary models to sample novel (reasoning) paths and enhance the performance of LPLM on commonsense If-Then reasoning tasks. Let us start by defining this task.
It has been recently proposed that LPLMs finetuned on (natural language) knowledge graph (KG) tuples, can express their encoded knowledge through language generation, thereby providing commonsense knowledge on demand (Bosselut et al., 2019; Hwang et al., 2020). Consider a KG of natural language tuples of the form (s, r, o), where s = (xs1, . . . ,x s |s|) labels the phrase subject of the tuple, r = xr is the relation token and o = (xo1, . . . ,x o |o|) is the phrase object of the tuple. In this setting, commonsense reasoning consists in
generating novel objects o, given s and r. The COMET framework of Bosselut et al. (2019) finetunes GPT-like models by maximizing the likelihood of the object, conditioned on the sequence (s, r) = (xs1, . . . ,x s |s|,x
r). We instead approach the problem of commonsense knowledge generation with a three-step process: (I) we auto-encode the KG tuples using HSN so that the first L2 symbols of the schemata encode (s, r) only, while their last L2 symbols encode the complete (s, r, o) tuple; (II) we train an autoregressive (AR) “reasoning” model, that takes (s, r) and z1:L
2 (the first L2
schema symbols inferred in step I) as inputs, and learns to generate the second half of the schema zL
2 +1:L (the half that encodes the object); (III) given unseen (s, r) pairs and their associated first half-schemata z1:L
2 , we sample novel, second half-
schemata zL 2 +1:L from the trained AR model, and use them to condition the HSN decoder and generate novel objects.
To do step I we write the random walk posterior distribution qϕ(z1:L|s, r,o) of HSN in the product form qϕ(z1:L
2 |s, r)qϕ(zL 2 +1:L|zL 2 , s, r,o),
where we omitted the explicit dependence on A, in all terms, for clarity. Both terms in the product are modeled with the same HSN encoder architecture as before (see Fig. 1), but share a single, pretrained BERT model. To do step II we introduce an AR “reasoning” model pθ(zL
2 +1:L|z1:L 2 , s, r), that pro-
cesses the first half-schema z1:L 2 via a recurrent neural net, and the pair (s, r) via a second, pretrained BERT model. This AR model is trained to generate the second half-schema on samples from qϕ(zL
2 +1:L|zL 2 , s, r,o). Finally, we do step III by
concatenating the first half-schema, sampled from the posterior model, with the second half-schema, sampled from the AR model. The novel reasoning path is used to condition the HSN decoder to generate novel objects. See Appendix H for additional details.
For this preliminary study we focus on the ATOMIC dataset (Sap et al., 2019), evaluate the quality of the generated objects with both, BLEU-2 (Papineni et al., 2002) and BERT Score (Zhang*
et al., 2020) metrics, and compare against GPT2, GPT-2-XL and BART, all trained within the COMET framework (Hwang et al., 2020). We also fix the number of symbols K = 100, and the random walk length L = 20.
Results. We find that our best AR models only learn about 58% of the second half-schema. That is, about half of the symbols encoding the complete (s, r, o) tuple. Table 3 shows the scores we obtain when conditioning the decoder on the schemata sampled from the posterior (which sees the target object), and from the AR model (which does not sees the object): HSN(50, 20) and HSNAR(50, 20), respectively. That HSN(50, 20) outperforms all baselines is expected, for HSN can successfully encode its inputs, as we empirically showed in Section 5 above. We find interesting that HSNAR(50, 20) is comparable to COMET(GPT2) in BERT Score, despite leveraging an imperfect AR model, which only reproduces about 58% of the correct encoding. The scores achieved by HSN(50, 20) imply that improving upon the AR model should increase the performance of HSNAR(50, 20), even beyond that of COMET(BART). We speculate that the AR “reasoning” model struggles to infer the schema distribution because of the complexity of the latter. This complexity is direct consequence of the object-encoding scheme we use (step I above), for which e.g. missing one single symbol can lead to random walks completely off. We shall investigate different encoding schemes in future work.

heading: 7 Conclusion
text: We have introduced a novel representation learning algorithm for natural language that (i) enforces, via inductive biases, relational structures which allow for compositionality onto the output representations of LPLM, and (ii) allows for symbolic reasoning via random walk processes on the discovered graphs. Future work shall investigate replacing both encoding and decoding models with more expressive LPLM, and explore training different “reasoning” models and encoding schemes, on natural language understanding tasks.

heading: Limitations
text: Although our methodology is agnostic to the specific sequence-processing models used as encoder and decoder, its main purpose is to be used together with LPLM. This clearly limits its use to institutions and users with access to computing facilities able to handle such models. Indeed, although the encoder of HSN can successfully be trained while keeping the weights of BERT (or any other LPLM) frozen, a feature which reduces training time, the GPT-based decoder does need to be fine-tuned to learn how to interpret the inferred symbols. This last points further limits the use of our methodology to “not-so-large” decoder models. That being said, fine-tuning HSN decoders via LoRA (Hu et al., 2022a) is an exciting option that can possibly solve (or help with) this last issue. We shall explore using LoRA with HSN in the near future.

heading: Ethics Statement
text: We do not find any ethical concerns either with the model or the datasets we used (which are publicly available).

heading: Acknowledgements
text: This research has been funded by the Federal Ministry of Education and Research of Germany and the state of North-Rhine Westphalia as part of the Lamarr-Institute for Machine Learning and Artificial Intelligence, LAMARR22B. Part of this work has been funded by the Vienna Science and Technology Fund (WWTF) project ICT22059. César Ojeda has been partially supported by Deutsche Forschungsgemeinschaft (DFG) ProjectID 318763901 - SFB1294 as well as ProjectID 235221301 - SFB 1114. Part of the work was also funded by the BIFOLD-Berlin Institute for the Foundations of Learning and Data (ref. 01IS18025A and ref 01IS18037A). C.O. would like to thank Manfred Opper for inspiring discussions, and Sarah Elena Müller for relevant literature.

heading: Appendix

heading: A Pseudo-Self Attention Mechanism Revisited
text: The attention mechanism of the original Transformers (Vaswani et al., 2017) is defined as
Attention(Q,K,V) =
softmax ( D− 1 2 Q ·KT ) ·V, (7)
where Q, K and V ∈ RT×D are sets of queries, keys and values, respectively, given by a sequence of T , D-dimensional vectors, packed into matrices. In practice, these queries, keys and values are projected many times with different learnable, linear maps. The Attention(·) operation (Eq. 7) is performed on these different projections in parallel, whose outputs are then concatenated and projected once more with a final, linear map. The complete operation is known as Multi-head Attention (Vaswani et al., 2017), and we use this notation in Fig. 1 of the main text.
Now, the question is how to condition GPT-2 on the schema ej1:jL . Given a sequence of input representations u1:T , the self -attention mechanism in GPT-2 is obtained by choosing Q = u1:T ·WQ,K = u1:T ·WK and V = u1:T ·WV , all in RT×D, with WQ,WK and WV ∈ RD×D pretrained matrices. We leverage a pseudo-self attention (PSA) mechanism (Ziegler et al., 2019) that augments the key and value matrices in their first L rows, with projections of ej1:jL so that
K̃ =
( ej1:jL ·WeK + penc
K
) ,
Ṽ =
( ej1:jL ·WeV + penc
V
) , (8)
both in R(L+T )×D, where penc is a positional encoding, just as the one used in the original Transformer implementation (Vaswani et al., 2017). The latter informs GPT-2 about the ordering of the symbols in the schema, as selected by the random walk process. PSA is then simply given by Eq. 7 with the keys and values replaced with the augmented ones, K̃ and Ṽ. The WeK ,W e V here are randomly initialized, learnable parameters mapping the schemata onto the decoder self-attention, Ddimensional space, and we have as many of them as layers in GPT-2. Therefore this mechanism allows GPT-2 to attend to the projected schema at
each of its layers, with a minimal addition of untrained parameters (Ziegler et al., 2019).

heading: B Hidden Schema Networks Algorithm
text: Algorithm 1: HSN Training (ϕ, ψ) foreach minibatch x1:T ∼ p(D) do
(1) Sample schema network from posterior graph model:
A ∼ qϕ(A),
(2) Compute parameters of posterior random walk model:
henc1 ,h enc 2 . . . ,h enc L = h enc ϕ (x1:T ),
ρ(ϕ) = softmax(henc1 ),
Q [i] k,j(ϕ) = f [i] k (ϕ)Akj∑
m f [i] m (ϕ)Amj
,
with f [1], . . . , f [L−1] = exp(henc2:L)
(3) Compute parameters of prior random walk model:
Pk,j = fk Akj∑K i=1 fiAij
(4) Sample random walks from posterior distribution:
z1:L ∼ qϕ(z1:L|x1:T ,A)
(5) Decode sentence:
for i = 0 to T − 1 do xi ∼ pθ(xi|x<i, ej1:jL), πi = softmax(W · hdecθ (x<i, ej1:jL)) end
(6) Compute loss and back-propagate:
L[θ, ϕ] = 1 N
N∑
n=1
E qϕ(z1:L|x (n) 1:T ,A)qϕ(A)
{ log pθ(x (n) 1:T |z1:L) }
− Eqϕ(A)KL [ q∗ϕ(z1:L|A); p(z1:L|A) ]
− KL[qϕ(A); p(A)]
end
C Inference model: Full Equations

heading: C.1 Posterior over (global) graph
text: We model the posterior over the graph assigning Bernoulli variables to its links, but we let the probability of observing each link depend on the global symbols
qϕ(A) = ∏
i,j
pϕ(ei, ej) aij (1− pϕ(ei, ej))1−aij
where pϕ(ei, ej) = sigmoid(gϕ(ei, ej)),
with gϕ : E × E → R a deep neural network, and pϕ(ei, ej) ∈ [0, 1], for all ei ∈ E , the link probabilities. Our reasoning here is that the network gϕ should infer graphs connecting symbols which are semantically related via the encoded sentences.
C.2 Posterior over random walks (encoder model)
We model the posterior probability over random walks on G as
qϕ(z1:L|x1:T ,A) = ( K∏
i=1
ρi(x1:T , ϕ) zi1
)
· L∏
i=2
  K∏
j=1
K∏
k=1
( Q
[i−1] k,j (x1:T ,A, ϕ)
)zki zji−1   ,
(9)
where instead of having a single transition probability matrix, we have a sequence of them, thereby allowing the posterior to capture inhomogeneous random walks. Note that we could have also chosen a mean-field decomposition along the steps of the random walk, simply by either ignoring the dependency on the graph, or making the graph fully connected (see Appendix D.5). Going back to Eq. 9, we model the probabilities over the starting point of the random walks and the transition matrices as follows
ρ(x1:T , ϕ) = softmax(henc1 ),
Q [i] k,j(x1:T ,A, ϕ) = f [i] k (x1:T , ϕ)Akj∑
m f [i] m (x1:T , ϕ)Amj
,
with f [1], . . . , f [L−1] = exp(henc2:L),
where henc1 ,h enc 2 , . . . ,h enc L ∈ RD is the sequence of outputs of a deep neural network model hencϕ (x1:T ) processing the input sequence of T words. The model hencϕ (x1:T ) must then map a sequence of T
vectors to a sequence of L vectors. We define hencϕ by a pretrained BERT model (Devlin et al., 2018), followed by a single Transformer block, randomly initialized. The Transformer block processes the T (D-dimensional) outputs from BERT as keys and values, together with a set of L learnable vectors q1:L as queries. The right hand side of Figure 1 illustrates the complete encoder architecture.

heading: D On the HSN Training Objective
text: Following standard methods (Bishop, 2006) one readily can show that the Evidence Lower Bound (ELBO) of the Hidden Schema Network model is given by
L[θ, ϕ] = 1 N
N∑
n=1
{
E qϕ(z1:L|x (n) 1:T ,A)qϕ(A)
[ log pθ(x (n) 1:T |z1:L) ]
− Eqϕ(A)KL [ qϕ(z1:L|x(n)1:T ,A); p(z1:L|A) ]}
− KL[qϕ(A); p(A)], (10)
where KL[·] denotes the Kullback-Leibler (KL) divergence.
Note that this is not the training objective of the main text. There we maximize the ELBO together with the mutual information between sentences and schemata. We give details about this modified objective in subsections D.3 and D.4 below. Before getting into that, let us first calculate the explicit expressions for the two divergences above.

heading: D.1 Kullback-Leibler Divergence Between Random Walks
text: For notational convenience we will not write the explicit dependence on the graph A in what follows. Using the explicit product form of the probabilities over walks leads to
KL[qϕ(z1:T |x(n)1:T ); p(z1:T )] = L∑
i=2
E q̂ϕ(zi−1|x(n)1:T )qϕ(zi|zi−1x (n) 1:T )
{ log qϕ(zi|zi−1,x(n)1:T ) p(zi|zi−1) } + KL[qϕ(z1); p(z1)],
(11)
where q̂ϕ(zi|x(n)1:T ) is the aggregated probability over all walks until step i. Since the random walks
are Markovian, q̂ can be explicitly written as
q̂ϕ(zi|x(n)1:T ) =∏
1≤j<i Q[j](x (n) 1:T , ϕ) · ρ(x (n) 1:T , ϕ), (12)
where the (posterior) transition matrices Q[i] are defined in Eq. 6 of the main text. Using the generic random walk definition in Eqs. 3 we can readily write the argument of the expectation value in Eq. 11 above as
log qϕ(zi|zi−1,x(n)1:T )
p(zi|zi−1) =
∑
k,j
zki z j i−1 log
Q [i−1] k,j (x (n) 1:T , ϕ)
Pk,j , (13)
which means we only need to compute the expectation of the product zki z j i−1. This one can easily be shown to be
E q̂ϕ(zi−1|x(n)1:T )qϕ(zi|zi−1x (n) 1:T )
[ zki z j i−1 ] =
Q [i−1] k,j (x (n) 1:T , ϕ) ρ̂ [i−1] j (x (n) 1:T , ϕ), (14)
where ρ̂[i]j (x (n) 1:T , ϕ) is the jth class probability of q̂ϕ(zi|x(n)1:T ), defined in Eq. 12. Finally, the second KL term in Eq. 11 can be directly evaluated
KL[qϕ(z1); p(z1)] = K∑
j=1
ρj(x (n) 1:T , ϕ) log
ρj(x (n) 1:T , ϕ)
ρj , (15)
where ρj(x (n) 1:T , ϕ) and ρj are, respectively, the posterior and prior class probabilities for the random walks’ starting points.
Putting all together we write
KL[qϕ(z1:T |x(n)1:T ); p(z1:T )] = L∑
i=2
K∑
k,j=1
Q [i−1] k,j (x (n) 1:T , ϕ) ρ̂ [i−1] j (x (n) 1:T , ϕ)
× log Q
[i−1] k,j (x (n) 1:T , ϕ)
Pk,j
+ K∑
j=1
ρj(x (n) 1:T , ϕ) log
ρj(x (n) 1:T , ϕ)
ρj (16)

heading: D.2 Kullback-Leibler Divergence Between Random Graph Models
text: Since both prior and posterior graph models treat each edge in G as a Bernoulli random variable, we can write directly
KL[q(A); p(A)] = ∑
ij
{ pϕ(ei, ej) log ( pϕ(ei, ej)
p
)
+(1− pϕ(ei, ej)) log ( 1− pϕ(ei, ej)
1− p
)} ,
(17)
where pϕ(ei, ej) is the posterior link probability, which is conditioned on the symbols connected by the link, and p is the global prior probability over all links.

heading: D.3 Maximizing Mutual Information
text: We would like to maximize the mutual information between the word sequences in our dataset and the schema representations. We have argued that the training objective in the main text already includes such a mutual information term. To see this is indeed the case we need to workout some identities.
Let us, for simplicity of notation, consider two discrete variables z and x, the last of which follows an unknown distribution pD(x). What follow are identities
−EpD(x)KL[q(z|x); p(z)] =EpD(x)Eq(z|x) { log p(z)− log(z|x) } =Hq(z|x) + ∑
x
pD(x) ∑
z
q(z|x) { log p(z)
+ log q∗(z)− log q∗(z) }
=Hq(z|x)−Hq∗(z) + ∑
z
q∗(z) { log p(z)− log q∗(z) }
=− I(z;x)− KL[q∗(z); p(z)], (18)
where
Hq(z|x) = − ∑
x
pD(x) ∑
z
q(z|x) log q(z|x), (19)
is the conditional entropy with respect to distribution q (see e.g. page 17 in Cover and Thomas (1991)) and
Hq∗(z) = − ∑
z
q∗(z) log q∗(z), (20)
is the entropy of distribution q∗(z), which we define as the marginal (data-aggregated) distribution
q∗(z) = ∑
x
pD(x)q(z|x). (21)
Finally, we used the definition of mutual information
I(x; z) = Hq∗(z)−Hq(z|x). (22)
See e.g. page 20 in Cover and Thomas (1991). It follows from Eq. 18 that maximizing the ELBO (Eq. 10), together with the mutual information between word sequences and schemata, simply amounts to replacing the KL between the approximate posterior and prior random walk distributions, with the KL between the aggregated posterior and prior random walk distributions. To wit
I(z1:L;x1:T |A)− 1
N
N∑
n=1
Eqϕ(A) {
KL [ qϕ(z1:L|x(n)1:T ,A); p(z1:T |A) ]}
= −Eqϕ(A)KL [ q∗ϕ(z1:L|A); p(z1:T |A) ] , (23)
where we introduced the aggregated posterior over random walks wrt the word sequence
q∗ϕ(z1:L|A) = Ep(x1:T ) [ qϕ(z1:L|x1:T ,A) ]
≈ 1 N
N∑
n=1
qϕ(z1:L|x(n)1:T ,A). (24)
In practice we approximate this quantity with
q∗ϕ(z1:L|A) ≈ q∗ϕ(z1) L∏
i=2
q∗ϕ(zi|zi−1,A), (25)
where q∗ϕ(z1) is a categorical distribution whose class probabilities ρ∗j (ϕ) are the average of those from our approximate posterior, that is
ρ∗j (ϕ) = 1
N
N∑
n=1
ρj(x (n) 1:T , ϕ), (26)
and the transition probabilities q∗ϕ(zi|zi−1,A) have transition probability matrices
Q ∗ [i] k,j (A, ϕ) = 1
N
N∑
n=1
Q [i] k,j(x (n) 1:T ,A, ϕ). (27)

heading: D.4 Training Objective
text: Putting everything together, the training objective for the HSN model reads
L[θ, ϕ] = 1 N
N∑
n=1
{
E qϕ(z1:L|x (n) 1:T ,A)qϕ(A) log pθ(x (n) 1:T |z1:L)
}
− Eqϕ(A)KL [ q∗ϕ(z1:L|A); p(z1:L|A) ]
− KL[qϕ(A); p(A)]. (28)

heading: D.5 Mean-Field Solution
text: Instead of modeling the posterior over random walks in the usual way, we could consider a meanfield decomposition along the time component, by ignoring the dependency on the graph G
qϕ(z1:L|x1:T ) = L∏
i=1
qϕ(zi|x1:T ), (29)
where at each step of the walk we have a stepdependent categorical distribution
qϕ(zi|x1:T ) = K∏
j=1
( ρ [i] j (x1:T , ϕ) )zji , (30)
whose class probabilities live in the K-simplex. We could model the latter via
ρ[1], . . . ,ρ[L] = softmax(henc1 , . . . ,h enc L ) (31)
where henc1 , . . . ,h enc L are the outputs of our encoder neural network model, shown in Figure 1 of the main text.
Replacing the mean-field approximation of
Eq. 29 into Eq. 10 yields
KLL[qϕ(z1:T |x(n)1:T ); p(z1:T |A)
= L∑
i=2
{ E qϕ(zi|x(n)1:T ) log qϕ(zi|x(n)1:T )
− E qϕ(zi|x(n)1:T )qϕ(zi−1|x (n) 1:T )
log p(zi|zi−1) }
+ KL[qϕ(z1); p(z1)],
= L∑
i=1
K∑
j
ρ [i] j (x1:T , ϕ) log
ρ [i] j (x1:T , ϕ)
ρj
− L∑
i=2
K∑
k,j
E qϕ(zi|x(n)1:T )qϕ(zi−1|x (n) 1:T )
[ zki z j i−1 ]
× logPk,j
=
L∑
i=1
K∑
j
ρ [i] j (x1:T , ϕ) log
ρ [i] j (x1:T , ϕ)
ρj
− L∑
i=2
K∑
k,j
ρ [i] k (x1:T , ϕ)ρ [i−1] j (x1:T , ϕ)
× logPk,j . (32)

heading: D.6 Fully Connected Graph
text: We can replace the adjacency matrix A in the definition of the transition probability matrix of our posterior Q(x1:T ,A, ϕ), with that of a fully connected graph. The aggregated posterior over all walks up to step i (Eq. 12 above) reduces in this case to
ρ̂ [i] k (x1:T , ϕ) =
K∑
j
( f [i−1] k (x1:T , ϕ)Ak,j∑
m f [i−1] m (x1:T , ϕ)Am,j
) ρ̂ [i−i] j (x1:T , ϕ)
= ( f [i−1] k (x1:T , ϕ)∑
m f [i−1] m (x1:T , ϕ)
)  K∑
j
ρ̂ [i−i] j (x1:T , ϕ)
 
= f [i−1] k (x1:T , ϕ)∑
m f [i−1] m (x1:T , ϕ)
, (33)
which is equivalent to that of the mean-field approximation of section D.5 with ρ̂[i]k = ρ [i] k .

heading: E On Synthetic Dataset Experiments
text: In this section we give additional details of and results from our proof-of-concept experiments.

heading: E.1 Synthetic Language Model
text: We generate our synthetic dataset as follows: first, we sample a single, fixed graph G∗ with K nodes from a predefined random graph model. Second, we define a set of random tokens V , of size V , to be our vocabulary. We create each token as a random 3-tuple from the Latin alphabet, and choose to have at least one order of magnitude more tokens than nodes in G (that is, V ≫ K). Third, we assign a random bag of tokens to each node in G∗. These random bags can simply be understood as probability distributions over V , and can be represented as V -dimensional vectors whose components live on the simplex. Note in particular that, by construction, tokens can be shared among the different nodes of G∗. Finally, let us identify the K random bags with the K symbols {e1, e2, . . . , eK} of the synthetic language model.
To generate synthetic sentences we sample uniform, L-step random walks on G∗, whose transition matrix is given by Eq. 4 in the main text, with f = I. Having obtained a set of random walks on G∗, we sample one random token from each of the symbols (i.e. from each random bag) along the walks.

heading: E.2 Experimental Settings
text: Here we give additional details for reproducibility
Datasets
• Following the procedure above we generated two datasets from two random graphs with different topologies. One sampled from the Barabási-Albert model (Barabási and Albert, 1999), the other from the Erdös-Rényi model (Erdös and Rényi, 1959). We generate these graphs using NetworkX, a Python language software package for network structures (Hagberg et al., 2008). Specifically, we generate Barabási-Albert graphs by attaching 3 edges from each new node to old ones, and ErdösRényi graphs with an edge probability of 0.5. We set both graphs to have K = 100 symbols.
• We define each random bag of tokens in G∗ to have two tokens only (each with equal probability).
• We use a vocabulary of 1000 random tokens.
• Once the graph is fixed, we set the token sequence length to L = 10 (L = 11) for the Erdös (Barabási) datasets and generate a total of N = 100000 token sequences from each random graph.
Hidden Schema Network (HSN) settings
• We train randomly initialized embeddings of dimension 256, one for each token. We sample these from a normal distribution with zero mean and a standard deviation of 0.01.
• The posterior graph model is defined via a single feed-forward neural network with 256 hidden units.
• The prior graph model has the edge probability p as hyperparameter. We crossvalidate it from the set p = {0.1, 0.2, 0.5, 0.6, 0.8} and found that HSN could fit the Barabási dataset only with small values {0.1, 0.2}. HSN could fit the Erdös dataset with larger values {0.5, 0.6}
• The posterior random walk model is defined by replacing BERT with a 2-block Transformer encoder (Vaswani et al., 2017), each with 2 heads, 256 hidden units and dropout probability of 0.2.
• The prior random walk model was set to a uniform random walk.
Training details
• We use a batch size of 256 and train with Adam (Kingma and Ba, 2014), with a learning rate of 0.0001, in all experiments.
• To sample both graph and random walk posterior models with use the Gumbel-Softmax trick (Jang et al., 2016), with a constant temperature of 0.75
• We train the models for 200 epochs

heading: E.3 Additional Results
text: Table 5 displays the mean and standard deviation of some additional results on our proof-of-concept experiments. We trained ten models in total.
We first trained a simple LSTM Network to infer the correct symbol order in each random token sequence. We noticed that a network with 256 hidden units was enough to solve this task perfectly. Indeed, the negative log-likelihood (NLL) of these models corresponds to choosing the 2-token random bag sequence (i.e. the schema) that yields the correct token sequence without errors. The HSN performs equally well on the Barabási dataset, and slightly worst on the Erdös dataset. In fact, we have noticed the Erdös dataset proved to be more challenging to learn with the HSN in all regards. See, for example, the AUC scores or the Frobenious norms of HSN in this dataset, as compared to the Barabási case. We think this might be due to the fact that Barabási graphs have more structure, simply because of their sparsity, which arguably make them easier to infer with our inductive bias.
Note also how increasing the prior edge probability p affects the average number of edges of the inferred graphs.

heading: F On Language Modelling Experiments
text: In this section we give additional details of and results from our language modelling and representation learning experiments.

heading: F.1 Experimental Settings
text: Here we give additional details for reproducibility Datasets
• We consider three widely used public datasets, namely the Penn Treebank (PTB) (Marcus et al., 1993), Yahoo and Yelp (Yang et al., 2017) corpora.
• PTB training set has a total of 38219 sentences. The average length of which is of about 22 words. The validation and test set have 5527 and 5462 sentences, respectively. The minimum (maximum) sentence length in PTB is of 2 (78) words.
• Yahoo training set has a total of 100000 sentences. The average length of which is of about 80 words. The validation and test sets have 10000 sentences each. The minimum (maximum) sentence length in Yahoo is of 21 (201) words. The Vocabulary size is of 200000 words.
• Yelp training set has a total of 100000 sentences. The average length of which is of about 97 words. The validation and test sets have 10000 sentences each. The minimum (maximum) sentence length in Yelp is of 21 (201) words. The Vocabulary size is of 90000 words.
HSN settings
• In all experiments we leveraged pretrained BERT and GPT-2 models, both with 12 layers, 768 hidden dimensions (D) and 12 attention heads. We used the public HuggingFace implementation of both these models (Wolf et al., 2020).
• The posterior graph model is set to a 2-layer feed forward network, each with hidden dimension 512.
• We crossvalidated the prior edge probability over the set of values p = {0.1, 0.2, 0.5, 0.6} and found p = 0.5 (a maximum entropy prior) to yield the best results. All results we report correspond to this (p = 0.5) case.
• We also train an inhomogeneous random walk prior model by making ρ and the sequence of weights f [1], f [2], . . . , f [L−1] trainable. We initialized them by sampling from a normal distribution with zero mean and standard deviation of 0.01.
• We experimented with HSN of K = {50, 100} symbols and random walks of length L = {5, 20}.
Training details
• We used a batch size of 32 and train with Adam (Kingma and Ba, 2014), with a learning rate of 0.00001, in all experiments.
• To sample both graph and random walk posterior models with used the Gumbel-Softmax trick (Jang et al., 2016), with a constant temperature of 1.0.
• We used a cyclical schedule to anneal both KL terms in our training objective from zero to one (Fu et al., 2019). When the annealing weight (usually called β in the literature) is finite, we used a KL threshold scheme (Li et al., 2019), with a threshold value of 0.1.
• We trained the models for 100 epochs, although they usually needed about 60 epochs only to converge (in the NLL).
• We applied word dropout to the input of the decoder model with probability 0.3 in the following cases: (i) for all models trained on PTB; (ii) and all models with L = 50 trained on all datasets.

heading: F.2 Evaluation Metrics

heading: Monte Carlo Perplexity Estimation
text: We compute the perplexity of HSN models via the importance weighted Monte Carlo estimator of the marginal log-likelihood,
log pθ(x1:T ) = log 1
R
R∑
r
1
S
S∑
s
pθ(x1:T |z(r)1:L)p(z (r) 1:L|A(s))p(A(s))
qϕ(z (r) 1:L|A(s))qϕ(A(s))
, (34)
where R and S are the number of random walk samples and graph samples, respectively. We used R = 100 and S = 10 per each sentence in the test set to estimate the perplexity.
Mutual Information Another metric that we use to evaluate our models is the mutual information between the representations (random walks) and the word sequences
I(z;x) =
EpD(x)KL[q(z|x); p(z)]− KL[q∗(z); p(z)]. (35)
Full derivation of this expression can be found in Appendix D.3 above.

heading: F.3 Additional Results
text: Here we report results complementing the conclusions of the main text.
Language modelling. Table 6 displays mutual information, reconstruction loss, KL of the random walks and KL of the graph. We report their mean and standard deviation obtained when repeating the experiments with the HSN model five times, with different initializations.
The conclusion of the main text, viz. that our results outperform all baselines, remains unaltered, even within error bars.
Graph statistics. We characterize the structure of G in terms of five statistics: (i) the diameter D, which measures the maximum path length over all node pairs in G; (ii) the average distance l, which instead measures the average shortest path length between all node pairs; (iii) the clustering coefficient C, which represents the probability that two neighbors of a randomly chosen node are themselves neighbors; (iv) the number of connected components CC; and (v) the degree distribution
P (k), which represents the probability that a randomly chosen node will have k neighbors.
Table 4 reports the statistics of our inferred graphs for all datasets, and all model configurations.
We can see that increasing the random walk length from 5 to 20 increases the number of connected components of the graphs. As a consequence, subsets of word sequences are map onto smaller subgraphs, the larger of which is about 50 symbols. One could argue that, since longer random walk lengths imply a larger set of possible schema configurations, the number of symbols required to describe our three corpora can simply decrease. In other words, less symbols are needed by long schemata. Similarly, directly increasing the symbols number leads too to a larger number of connected components. Indeed, even the short schemata in Yelp and Yahoo do not use all available symbols to model the corpora.
Finally, Figure 11 displays the difference between the degree distributions of the schema networks, and that of an Erdös-Rényi graph for all HSN configurations.
Schema distributions on inferred networks. We can get a graphical picture of the features we just discussed above in Figures 4–6 below. Importantly, we see that the schema distribution is different for each category of each corpora in all model configurations. In other words, we do not observe any kind of mode collapse.
Schema interpolations. Finally, we have also explored “schema interpolations”: given two schemata ej1:jL and em1:mL , we find the shortest path (of length l) on G connecting the end of ej1:jL with the beginning of em1:mL . Our interpolation steps are the schemata {ej1+i:jL+i : ∀ 0 ≤ i ≤ l + L along the path}. Tables 9–10 show interpolations of random instances from all datasets. Note how the model successfully interpolates between categories in both Yelp and Yahoo.

heading: G Which Symbols do Words Attend to? A Preliminary Study on Yelp Reviews
text: In this section we investigate how symbols are used by HSN when generating text. We do this by exploring the decoder attention matrix between the symbols and the generated tokens. Reading the attention weights, we can examine which symbols are most important for the generation of any given token, i.e. which symbols are attended to more
strongly. A bit more in detail we select, for a given token in a given sentence, the symbol with the highest attention value. We can then compute the distribution of most attended symbols when generating that token for the complete dataset.
Thus, for a model trained on the Yelp dataset, we examine to which symbols does the decoder of HSN attend to, when processing the words good, great and bad. Figure 7 shows the most attended symbol distribution for layers 1 (first), 5 (middle), 12 (last), when averaging the attention matrices over all attention heads. Figures 8, 9, 10 show
these distributions for each head separately. Note how, for a fixed token, the distribution of attention changes as one moves between heads and layers, although there are some repeating patterns too.
We can quantify these features by computing the Kullback-Leibler (KL) divergence between these distributions. The KL values are shown in Table 7.
Interestingly enough, the distribution of symbols that are attended to when processing the word great is closer to the distributions of symbols attended by the word good, than to the distributions of symbols attended by the word bad.

heading: H On Commonsense Reasoning Generation
text: In this section we expatiate on the details of our approach to commonsense reasoning generation.
First, we modify the encoder component of HSN to process the tuples (s, r,o) as
qϕ(z1:L|[s, r,o],A) = qϕ(z1:L
2 |[s, r],A)qϕ(zL 2 +1:L|[s, r,o], zL 2 ,A),
so that the first half of the schema depends on subject and relation only, whereas the second half depends on the entire 3-tuple. As it will become evident below, this decoupling is necessary for the inference of novel objects.
Each of the posterior distributions above is modelled with the same architecture, as shown in Fig. 1, but sharing a single pretrained BERT model. That is, we have two copies of all pink-shaded blocks in the Fig. 1, one for qϕ(z1:L
2 |[s, r],A), the other
for qϕ(zL 2 +1:L|[s, r,o], zL 2 ,A), and a single pretrained BERT model. Using such a 2-component encoder model we are able to successfully infer schema representations for the KG tuples, as shown in Table 3. The task is however to infer new objects, given only subject-relation pairs. We thus need a way to infer schema representations without relying on the phrase object.
The classical solution to this inference problem is to replace, à la Kalman Filter, qϕ(zL
2 +1:L|[s, r,o], zL 2 ,A) with a local, trainable
prior model of the form pθ(zL 2 +1:L|[s, r], zL 2 ,A) – where zL 2 is sampled from qϕ(z1:L 2 |[s, r],A) – and train the prior via the KL term in Eq. 28. However, and as shown in Section D, maximizing the mutual information between data and representations averages out all local information in the KL term, and thus hinders the learning of the prior.
An alternative is to train, in the spirit of knowledge distillation (Hinton et al., 2015), a thirdparty model on the inferred schemata, to predict zL
2 +1:L conditioned on z1:L 2 . Indeed, given the inferred schemata from the training KG, we consider a sequence-to-sequence model which inputs z1:L
2 , together with the subject-relation pair, and outputs zL 2 +1:L. That is, a model of the form pθ(zL 2 +1:L|z1:L 2 , [s, r]). Specifically we use (i) a bidirectional LSTM network with hidden dimension of 512 to encode the first half of the schemata, (ii) a pretrained BERT model to encode the subject-relation pair, and (iii) a LSTM network of dimension 512 as an autoregressive decoder model. The initial (hidden) states of the latter are determined by an MLP which inputs the representations from the LSTM and BERT encoder models. The model is trained on samples from qϕ(zL
2 +1:L|[s, r,o], zL 2 ,A).
Our preliminary results, HSNAR(50, 20) in Table 3, are comparable to a stand-alone COMET(GPT-2) model, even when the thrid-party model only learns
about 58% of the second half of the schemata.

heading: H.1 Atomic Dataset
text: For this preliminary study we focus only on the ATOMIC dataset of Sap et al. (2019). It contains 877K (s, r,o) tuples covering a variety of social commonsense knowledge around specific If-Then events. A bit more in detail, ATOMIC splits its commonsense knowledge into nine categories, covering the event’s causes, its effects on the agent, and its effect on other direct (or implied) participants. We use the training splits from Sap et al. (2019), resulting in 710K training, 80K validation, and 87K test tuples respectively.

heading: I Training Time and Resource Consumption
text: In Table 8 we report the training times of our HSN models. The times are similar for all HSN configurations, so we report their average training time for each dataset.

heading: K = 100, L = 20 [LCC]
text: ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitation section
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3, F
3 B1. Did you cite the creators of artifacts you used? 1, 3, 4, 5, F
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? All datasets and artifacts we used are publicly available. Our code (attached as sup. material) shall also be make publicly available once the paper is accepted.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The datasets we used are publicly available and widely used in research.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? The datasets we used are publicly available and widely used in research. Documentation about our own artifacts (models) can be found in the Appendix
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. E, F
C 3 Did you run computational experiments? 4, 5, 6
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? E, F, I
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? E, F
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? E, F
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? F and source code
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We shall explore using LoRA with HSN in the near future