Hidden Schema Networks
Authors: Ramses J Sanchez
Lukas Alexander Conrads
Pascal Welke
Kostadin Cvejoski
Cesar Ojeda
Venue: Submitted to ICLR 2023 (2023)
Keywords: Discrete representation learning
Unsupervised knowledge graph learning
Relational inductive biases
Semantic representation
Pretrained language models
Discrete VAE
Neuro-symbolic AI
Language modelling

Abstract: Most modern language models infer representations that, albeit powerful, lack both compositionality and semantic interpretability. Starting from the assumption that a large proportion of semantic content is necessarily relational, we introduce a neural language model that discovers networks of symbols (schemata) from text datasets. Using a variational autoencoder (VAE) framework, our model encodes sentences into sequences of symbols (composed representation), which correspond to the nodes visited by biased random walkers on a global latent graph. We first demonstrate that the model is able to uncover ground-truth graphs from artificially generated datasets of random token sequences. Next we leverage pretrained BERT and GPT-2 language models as encoder and decoder, respectively, to train our model on language modelling and commonsense knowledge generation tasks. Qualitatively, the model is able to infer schema networks whose nodes (symbols) can be interpreted as encoding different aspects of natural language (as e.g. topics, sentiments). Quantitatively, our results show that the model successfully interprets the encoded symbol sequences, as it achieves state-of-the-art scores on VAE language modeling benchmarks. Source code to reproduce all experiments is provided with the supplementary material.

Under review as a conference paper at ICLR 2023
HIDDEN SCHEMA NETWORKS
Anonymous authors
Paper under double-blind review
ABSTRACT
Starting from the assumption that a large proportion of semantic content is neces- sarily relational, introduce neural language model that discovers networks of symbols (schemata) from text datasets. Using a variational autoencoder (VAE) framework, our model encodes sentences into sequences of symbols (composed representation), which correspond to the nodes visited by biased random walkers on a global latent graph. We first demonstrate that the model is able to uncover ground-truth graphs from artificially generated datasets of random token sequences. Next, we leverage pretrained BERT and GPT-2 language models as encoder and decoder, respectively, to train our model on language modelling and commonsense knowledge generation tasks. Qualitatively, the model is able to infer schema net- works whose nodes (symbols) can be interpreted encoding different aspects of natural language (as e.g. topics or sentiments). Quantitatively, our results show that the model successfully interprets the inferred symbol sequences, it achieves state-of-the-an scores on language modeling benchmarks. Source code to reproduce all experiments is provided with the supplementary material.
INTRODUCTION
Much of the developmental and causal theories of human cognition are predicated on relational structures of knowledge that naturally exhibit compositionality Semantic content is intrinsically relational, one is only able to explain a given unit of knowledge such as concept, word or perception insofar as there are other units of knowledge which relate to it (Block 1986). Thus we can partially construe a concept through its relationships to other concepts (like when we say "a dog is an animal that barks"), just as we can partially construe it through its relationships to our perceptions (when we say "that a dog". whilst pointing to dog on the street) or the words we use (when we use the word dog to refer to the concept dog). Likewise, we can partially construe words not only through their relationships to concepts or percepts, but also through their relationships to other words, words that occur in the same context tend to have similar meanings (Harris 1954; Firth 1957). Note that is precisely this contextual semantic content of words what we have explicit access to, when processing our raw text datasets. On the other hand, generalization, reasoning and understanding seem to be inevitably tied to the compositional nature of knowledge. Indeed, the ability compose set of knowledge units (and their relations) into new, more complex relational units, which can be deployed to understand and reason about unseen data a feature usually referred as combinatorial generalization is regarded as key human-level intelligence (Fodor Pylyshyn 1988 [odor & Lepore, 2002 Lake et al. 2017: Battaglia et al. 2018). Relational structures allowing for compositionality thus seem to comprise not a sufficient, but necessary attribute of any representation scheme that strives for the generalization power of human cognition.
From the computational side, if one is inform any modern machine learning model with such structural characteristics, one will initially encounter the problem of finding suitable primitives or data structures. In natural language processing (NLP), for example, it has become common place to leverage distributed continuous representations of words (Bengio et al. 2003) for different downstream tasks. Such representations are trained to encode average contextual semantics - precisely the kind of semantic content typical of word co-occurrence relations we mentioned above into a semantic space, which allows meaning to change continuously within it (Mikolov et al. 2013). Yet, despite earlier attempts (Mitcelll & Lapata| 2008), it is unclear whether such representations can be meaningfully composed into representations of, say. unseen sentences and thus mimic the compositional character of natural language. More recently, contextualized continuous word

---

Under review as a conference paper at ICLR 2023
R (A)
X2, Xg..... XI
26
Lincar Softnax
Norn Linear
Feed Forward
XM
Foed Forward
Norm
Nora
Z2
Masked
Multi-head
Z1
attentoo
Q
K
V
Poskizd cooudes
K
Livz
po (
X1, X2 XT
X1, X2 ......
XT-1
Cj Oz.......
Gj
92 X1 Xg.
XT
Figure 1: Left: Diagram of Hidden Schema Network model. Center: Decoder architecture modified GPT-2 model see the Appendix for details. The "c" operations labels concatenation. Right: Encoder architecture as BERT model, followed by single Transformer block. In both center and right figure purple shaded blocks represent submodules with pretrained parameters. Pink shaded blocks represent submodules with randomly initialized parameters.
layers, with pseudo-selffattention mechanism attend to the schema ejuis- Please
representations inferred by deep learning architectures have shown spectacular results in many NLP tasks (Radford et al.] 2018; Devlin et al.] 2018) Radford et al. 2019 Brown et al. 2020). Their success stems from those models' ability to infer flexible representations through, inter alia, raw, massive datasets, data-scalable attention mechanisms and minimal inductive biases (Vaswani et al. 2017). These representations are known to not only contain rich contextual word semantics, but also consistently encode sentence-level grammar (Hewitt & Manning. 2019), and the models from which they are obtained seem to implement some notions of compositionality too (Hupkes et al. 2020 Wei et al., 2022). Nevertheless, it is still unclear whether such representations into representations of novel sentences (Yu & Ettinger [020) Bhathena et al. (020). In fact, most of their syntactic properties implicit and therefore inferred only a posteriori, typically through probes which neither guarantee their presence, nor establish how they were obtained in the first place (Rogers al. 2020).
be composed
In this work we use VAE framework (Kingma & Welling 2013 Rezende et al. 2014 to develop language model the Hidden Schema Network model (HSN) that enforces, via inductive biases, discrete, relational structure for sentence representation which allows for compositionalin/ while exploiting the well-known advantages of attention models and contextualized, pretrained representations. We first demonstrate that the model is able to uncover ground-truth graphs from artificially generated datasets of random token sequences. Next, we leverage our methodology to translate the implicit lexical and grammatical aspects of language encoded by pretrained BERT and GPT-2 language models into explicit relational structures, and apply the latter on language modelling and commonsense knowledge generation tasks.
Our main contribution then an exploration of novel way to integrate discrete (symbols), relational (graphs) and continuous (neural representations) machine learning components into end-to differentiable representation learning algorithm for natural language modelling. Our aim is thus to try connect the modern NLP paradigm with classical notions of linguisties, and begin answer the recent calls for neuro-symbolic integration (Garcez. & Lamb 2020 Cartuyvels et al. 2021).
Note that throughout the paper we refer only to compositionality of representations and not to the compo- sitional functions that can be implemented by the models we use. The latter, functional compositionality, is studied by e.g. Hupkes al. 2022))
2

---

Under review as a conference paper at ICLR 2023
2 RELATED WORK
In cognitive psychology, a schema is (roughly) defined as a large, complex unit of knowledge representing what is typical of a group of instances (Bartlen 1932, Piaget 1948 Rumelhaat (017). Marvin Minsky's frames (Minsky 1974 1975) are similar in function to a schema, but perhaps more easily characterized in terms of data structures. We use these terms in a loose fashion, however. Our aim being only to be suggestive of the general problem of knowledge representation (Thagard 1984). We are fact concerned with representation schemes for natural language processing. Within the context of linguistics, [Jackndofff (1978) argues that there must be a level of representation the so-called conceptual structures which information conveyed by language must be compatible with information coming from sensory systems. Conceptual structures must, he goes on, be able to represent all the conceptual distinctions made by natural language, and provide some degree of compositionality. Earlier computational models implementing (some kind of) conceptual structure rely on either hand-coded (semantic) network representations (Quillan 1966 Collins & Quillian
1969 Brachman 1977) or hand-coded databases (MeClelland & Rogers, 2003 Other works focus
instead on learning semantic representations directly from text data via topic models (Griffiths et al. 2007b), and even infer latent concept graphs through nonparametric priors (Chambers et al.) 2010).
In sharp contrast with these works, modern, neural-based language models incorporate no explicit linguistic notions, and leverage massive datasets and attention mechanisms, in the form of large pre-trained language models and contextualized, continuous word representations. We build on top of these ideas, while trying to connect back with models of conceptual structure, which necessarily involve discrete representations (van den Oord et al. 2017 Hu et al. 2017 Zhao et al. 2018 Kaiser Bengio 2018 Kaiser et al. 2018).
HIDDEN SCHEMA NETWORKS
We address the problem of learning the joint probability distribution over sequences of words, while inferring interpretable representations capturing their semantics. Neural autoregressive language models approximate such distributions with a product over conditional probabilitics, such that
T II pa (xt),
p(XI.)
where X1:7 = (x1, _2, X.. XT) labels the sequence of words in question, and each conditional is given by (the pdf of) categorical distribution over some vocabulary of size V. The class probabilities of these conditionals are generally computed as π.; = softmax (W. ho(x<s), with W E V trainable, and D the output dimension of hg, a deep neural network model with parameter set (Bengio al. 2003). Models of this form allow for tractable estimation of and sampling from either the joint distribution, or any product the conditionals Eq, Indeed, their recent implementation in terms of large-capacity, self-attention architectures such GPT-2 (Radford et al. 2019) has been shown to generate syntactically correct, diverse and fluent text. Yet. most of the linguistic structure encoded by the output representations of these models is implicit and difficult to interpret (Rogers) et al. 2020). In what follows we shall condition the joint distribution of Eq. [on an additional latent, discrete representation which can, at least in principle, capture the relational and compositional features of semantic content.
Let us assume there is & (ei, .,.......,,, of K symbols that encode some high-level, abstract semantic content of natural language. Let this set be the set of nodes of a hidden (semantic) graph G. with adjacency matrix A, so that adjacent (connected) symbols are semantically related. These symbols can generically be defined as learnable, dense vectors in IRS for some dimension S. Without loss of generality, however, we opt below for simple indicator ("one-hot") vectors of dimension K instead. We define a schema Ojrsi, as sequence of L < K symbols (ojz Cja..... Cj₁ ). where the indices ,i,...,jr. label a subset of connected nodes in G. Accordingly, we refer to G as a schema network. The symbols composing the schemata are chosen through a L-step stochastic process conditioned on G. Partially motivated by research on random walks and human memory search (Griffiths et al.) a (007aa Abbott et al. 20122), as well by the simplicity of their inference, we choose to compose the schemata via biased random walk processes on G, and leave exploring different schema processes for future work.. Let us now specify the generative model in detail.
3

---

Under review as a conference paper at ICLR 2023
3.1 GENERATIVE MODEL
We write the joint probability over sequence X1.T of T words, together with the hidden graph G. as
Po (xI:T A) = Σ
Po (e.xxrrlessii (t2.:///.///pp....
(2)
R1:L.
where 21.7 labels the sequence 21...... 2r of K-dimensional, one-hot vectors representing the node labels j1, -......J1. visited by random walker on G, and 0 denotes the trainable model parameters. Note that we introduced the one-hot representation of j; for notational convenience, as shall become evident below Next, specify the different components of Eq.
Prior over (global) graph. A prior on the adjacency matrix p(A) allows us to control the topological properties of G. One can choose, for example, random graph models whose degree distribution asymptotically follow a power law (Barabási & Albert 1999), or unbiased, maximum entropy graph models, with respect some given constrains (Park & Newman 2004). For the sake of simplicity we choose a Bernoulli (Erdös-Rény)) random graph model (Solomonoff & Rapoport 1951 Erdös & Rényi 1959, for which each link dzj is defined via independent Bernoulli variable with some fixed, global probability p E [0, 1]. so that
K
(AA)=
II f==1
p 00 (1
(1-p)
1-a3
The probability p will be a hyperparameter of our model.
Prior over random walks. The probability p(z).I|A) of random walk over the nodes of G can generally be written as
L p(ZiL|A) pp(z) II p(z 1
K
L
K K
x/=
II
Pm
RJ
(4)
j=1k=1
where p(z1) labels the probability of selecting ji the starting point of the walk, and it is given by (the pdf of) categorical distribution over the nodes of G, with class probabilities {p)} Similarly p(z, 1, A) labels the conditional probability of jumping from ji-1 to j.. which we define in terms of K x K transition probability matrix P. Now, to allow for biased random walks, let each node k on be given a positive weight fs. so that the probability of jumping from j is proportional to fa Akj. We then write the transition probability matrix as
fa Akj K
PEJ
(5)
F.
so that the motion of the random walker is biased according to the node weights fk. These weights should be understood as encoding aspects of the diffusion dynamics that are independent of the topology of the graph (Gómez-Gardens are in order: first, note that one can also train the prior over walks by making the vectors p and learnable. Second, setting the node weights f and the class probabilities p = I with I the K --dimmnsional vector of ones, yields uniform random walk over G. i.c. process in which the walker has equal probability of jumping to any of its neighbors. Third, one can also allow for inhomogeneous random walks in which the probability matrix changes at each step of the random walk. Such processes can be parameterized with sequence of weights f(]], f(2) f(2)..... 1]
& Latora 2008: Lambiotte et al.) (01T). Three comments
Decoder and likelihood. Just as in Eq. D we define the joint probability over word sequences as a product of conditional probabilities, this time conditioned on the schema @jiii too, that is
Po (x(xi:1e) = II pa (x, Sjnje), n = softmax SaaimaxxWWWaa W hhde (x Sjvcja)),
(6)
with the class probabilities of the ith conditional, W E R VDD trainable, and h deep neural network model. We let hd be a pretrained GPT-2 language model, and modify it to also process the
Explicitly. ji denotes the index of the non-zero component of z, i.c. ji {I [1, K] of 1}, with the superindex i denoting the components of z,.
4

---

Under review as a conference paper at ICLR 2023
schema Cjuih but remark that any other model for sequence processing (as e.g. recurrent neural net) could be used instead. A bit more in detail, to condition GPT-2 on ji without perturbing its optimized weights too much,
the psendo-self -elf-attention (PSA) mechanism introduced by Ziegler et al. 2019). In a nutshell, this mechanism augments the key and value matrices of GPT-2 in their first I. rows with projections of e Figure shows illustration of the complete decoder model, including the PSA mechanism. Please check Appendix A for the explicit equations of the latter.
3.2 INFERENCE MODEL
The generative model we presented above is hierarchical. The random graph is shared across all sentences and thus constitutes a global latent object. The random walks, in contrast, are local random variables. Our task is to infer the schema and graph posterior distributions that best describe the collection of word sequences in our dataset. To do this, we approximate the true posterior distribution of these variables with a variational posterior of the form
4 (. = (Z1. (.r.A))444.. (A),
where & labels the set trainable parameters. Let us specify each its components.
Posterior over (global) graph. We model the posterior over the graph assigning again Bernoulli variables to its links, but we let the probability of observing each link depend on the global symbols
g0 (A) Pole, e,) e. (1 Po(e,, e,))) where po(e,,e,) sigmoid(g e,)), (8)
1
with 9o E IR a deep neural network, and (e(e,, e,) E [0, 1]. for all e; E E, the link probabilities. Our reasoning here is that the network ge should infer graphs connecting symbols which are semantically related via the encoded sentences.
Posterior random walks (encoder model). Analog to Eq. 4 model the posterior probability over random walks on G as
II I (xIT, zt (44))
M o)
(III A)
(9)
where instead of having a single transition probability matrix, we have a sequence of them, thereby allowing the posterior capture inhomogeneous random walks. Note that could have also chosen a mean-field decomposition along the steps of the random walk, simply by either ignoring the dependency on the graph, making the graph fully connected (see Appendix [.4). Going back to Eq. 9 matrices follows
we model the probabilities over the starting point of the random walks and the transition
p(XIT,0) softmax(h$"1
(10)
[|( (XI.T, ②) A
XL.T A, σ) =
with r!)..... pll.-1] exp(hs.t)
(11)
Σn 1 (XITT σ) Amj
where h h h processing the input sequence of T words. The model ho (x1:r) must then map a sequence of T vectors to sequence of L vectors. We define hge by pretrained BERT model (Devlin et al. 2018 followed by single Transformer block, randomly initialized. The Transformer block processes the T (D-dimensional) outputs from BERT as keys and values, together with set of L learnable vectors 71fb queries. The right hand side of Figure iillustrates the complete encoder architecture.
L E RDD is the sequence of outputs of a deep neural network model hece (x1:r)
3.3 TRAINING OBJECTIVE
To optimize the parameter sets {8, Φ} of our latent variable model we would, as usual, maximize a variational lower bound on the logarithm of the marginal likelihood pe px(x):r) (Bishop) 2006). It is. however, well known that VAE models tend to encounter problems learning representations encoding information about the data the so-called posterior collapse problem especially when dealing with
5

---

Under review as a conference paper at ICLR 2023
Graph g"
ROC AUC
| Gll
[|gmml
G/l
N. edges(G)
N. edges((")
Barabasi Erdos
0.989 + 0.001
17 + 2
26 + 1
1360 + 104
291
0.94 + 0.06
36.8 + 0.8
44 +
3131 + 156
2092
Table I: Inference of ground-truth random graphs
natural language (Bowman et al. 2015). To solve this issue practitioners resort to maximizing the variational lower bound, together with the mutual information between data and representations (Zhao) et al. 2018 Fang et al. 2019 Zhao et al. 2019). We follow this same route and show (in Appendix BB3p that maximizing the lower bound and the mutual information corresponds to maximizing the objective
N Σ
[
ta 1. (x ()a((A) log Po logro
(o R1.L)
= N
E8 (A) KL (Z A A); A))
KL KL ( P)):(AA)
(12)
where KL labels the Kullback-Leibler divergence (Kullback & Leibler 1951) between prior and posterior distributions, and (,(21:1 (A) is the aggregated posterior distribution over random walks.
The latter is defined as ())(()):)) [ga (2.... XI:T, A) and is in general intractable. In practice, we approximate it with expression identical to Eq. | but with the class probabilities and transition matrices (Eqs. [0] and [1) replaced with their data-averaged counterparts. We refer the reader to Appendix B for details on this, well for the explicit, closed-form expressions of the Kullback-
Leibler terms in Eq. 12 The full training algorithm is presented in Appendix
4 PROOF OF CONCEPT: INFERRING GROUND-TRUTH RANDOM GRAPHS
Before testing the behaviour of our methodology on natural language data, we evaluate the ability of the model to infer hidden graph structures from sequential data in controlled experiment. To this end, we define a synthetic language model with an underlying, ground-truth graph G* as follows: Given a graph G* with K nodes, and vocabulary of random tokens of size V, we assign one random bag of tokens (i.e. one pdf over V) to each node of the graph. Let the K random bags be the K symbols {ei, .......er ek} of the synthetic language model. We then sample N uniform random walks of length L over G*, and sample one random token from each symbol (i.c. from each random bag) along the walks. The result a of random token sequences of the same length as that of the random walks. Appendix D contains more detailed description of this generation procedure.
Given this set of random token sequences, the task infer the hidden ground-trun -toun graph G*
Experimental settings. Following the procedure above we generated two datasets from two random graphs with different topologies. One sampled from the Barabási-All Albert model (Barabási & Albert 1999), the other from the Erdös-Réén model (Erdös & Rényi 1959). We set both graphs to have K 100 symbols, and the token sequences to have length = 10. Each dataset has a total of N 100000 token sequences. Further details about the random graph model parameters and the
dataset statistics can be found in Appendix D The synthetic datasets are available in the source code.
A simple proof-of-coneee
We consider problem in which the set of symbols (random bags) E known, so that the ground-truth graph G" has fixed labelling. This setting will allow for simple comparison between G and our inferred graphs. To infer G used simplified version of HSN, namely: we (i) replace BERT in Fig. with a 2-block Transformer encoder (Vaswani et al.) 2017): (ii) set the graph model 9, (Eq. 8 single-layer, feed forward network; and (iii) note that, since the symbols are known, the likelihood of the model is simply given by II'_, e,, where, as before, ja denotes the index of the non-zero component of z, We train this model by maximizing Eq. and refer to Appendix D for details on hyperparameters, training procedure and model sizes.
Results. Table T shows results for our synthetic datasets. Specifically, we compute the Area Under the Receiver Operating Characteristic Curve (ROC AUC) of our model qu (A) with respect to 9", and the Frobenious norm between qa and two graphs: the ground-truth one 9", and a second random graph grand sampled from the same random graph model as G*. We train ten (10) models

---

Under review as a conference paper at ICLR 2023
Method
PTR
YAHOO
YELP
ptb
GPT2 (one epoch)
24.23
22.00
23.40
yelp
GPT2 line tuned]
19.14
20.64
19.77
Node Count
yahoo
iVAEM
53.44
47.93
36.88
Optimus
23.58
22.34
21.99
HSN (100.
17.72
20.28
19.18
HSN (100.5)
17.79
20.10
19.05
HSN (oo, 20)
16.88
19.59
19.01
20 Node Degree
HSN (50.5)
17.41
20.06
18.95
Table Left: Perplexity per word (lower is bet- ter) on three datasets. GPT2 [one epoch] and Op- timus results were extracted from [i ct al. (2020). iVAEMI was taken from Fang al. (2019). GPT2 [fine-tuned] was computed by us. End-of-sequence tokens are kept during evaluation.
Figure Empirical degree distributions of ferred graphs from each corpora. Results corre- spond to HSN with L = 5, K = 50. We also show the distribution for random graphs with p 0.5. The graphs are sampled 500 times.
in total and display the mean and standard deviation of our results. We also different Grand for each calculation run. The first metric shows that 9ø correctly predicts the edges of g", whereas the other two metrics show that G g is closer to G* than to any other random graph sampled from the same distribution. The last two columns Table show however that qo (A) tends to generate denser graphs compared to the target.
Having demonstrated that HSN can indeed infer hidden graph structure from sequential data in a simple setting we now move main problem: language modelling.
LANGUAGE MODELLING AND REPRESENTATION LEARNING
Natural language modelling deals with the prediction of the word in sentence or document, given a sequence of previously observed words. A natural evaluation metric is therefore the perplexity per word of the model, which defined as the exponential of the data-averaged. negative log- likelihood of the model, divided by the number of words in the sequence. One complication with this is that latent variable models can only approximately estimate likelihood function. One can readily see, however, that Eq. 12 is also lower bound on log Po (Bishop 2006) and so, we estimate the perplexity of our models with cxp(-L/T).
Datasets and baselines. We consider three widely used public datasets, namely the Penn Treebank (PTB) (Mareus et al., 1993), Yahoo and Yelp (Yang et al., 2017) corpora. For completeness we include statistics of these datasets in Appendix [ We compare HSN against pretrained GPT-2, fine-tuned both during a single epoch and until its objective function platcaus. We also compare againts two VAE language models: iVAEMI (Fang et al. 2019) and Optimus (Li et al., 2020). The former implements both encoder and decoder as one-layer LSTMs (Hochreiter & Schmidhuberi 1997). The latter uses pretrained BERT and GPT-2 as encoder and decoder, respectively.
Experimental settings. In all experiments we leverage pretrained BERT and GPT-2 models, both with 12 layers, 768 hidden dimensions (D) and 12 attention heads. Note that Optimus shares these settings. We use the public HuggingFace implementation The graph model is set to 2-layer feed forward network, each with hidden dimension 512, and also train an inhomogencous random walk prior model (Eq. 4 by making P and the sequence of weights f(1], f!] f!!..... f(1-1] trainable. Furthermore, we explore HSNs with K = {50, 100) symbols HSN(K, L).
both these models (Wolf et al. 2020).
and hidden random walks of (5, 20) steps. Let us label these configurations Additional details on hyperparameters and training procedures can be found in Appendix E
Results. Table shows the perplexity of our model, together with the baselines, evaluated on the test set of the three corpora. HSN achieves much better performance than all baselines under this metric, which implies it successfully interprets the symbol sequences uses to encode the sentences. Note in particular that HSNs with 50 symbols perform consistently better than their 100-symbol counterparts.
We could, of course, now study the harder problem for which the symbols are unknown. However, the learned graphs model would not he aligned with making the graph comparison non-trivial.
7

---

Under review as a conference paper at ICLR 2023
Science & Mathematics
Education & Reference
Sports
Politics & Government
1.0
1.0
0.8
0.8
Edge Vists
0.0
0.0
Symbol
0.4
-0.4
0.2
0.2
0.0
0.0
Figure 3: Schema distributions inferred by HSN(50, 5) for four labels of the Yahoo corpora. The node positions in the figure consistent among labels and were computed using force-directed embedding of the global graph G.
As we discuss in Appendix E. 100-symbol HSNs tend to infer networks with many disconnected subgraphs, the largest of which has usually about 50 symbols. It appears then that (about) 50 symbols enough to encode these corpora. We have repeated these experiments five (5) times, each with different random initialization of the pink shaded blocks in Fig. We find our mean perplexities to be better than all baselines even within error bars. The reader find these results in Appendix H To get a deeper insight into the features of these representations we now explore the structure of the learned global graphs G. as well as the semantic content of the schemata.
Structure of hidden schema networks. We characterize the structure of G in terms of five statisties: its (i) diameter D. (ii) average distance I, (iii) clustering coefficient C, (iv) number of connected components CC and (v) degree distribution P(k) (see Appendix E for the definition of these). We report our results in Table for HSN(50, 5). Results for the other HSN configurations can be found in the Appendix, from which we mainly find that longer random walks and larger symbol number generically favor larger CC. Going back to Table we observe that the schema networks from each corpora tend to have smaller average distances 4, and much larger clustering coefficients C, than any random graphs (with p = 0.5) of the same size where random graphs with p = 0.5 correspond to our prior model. Let remark that the combinations of these two features defines the so-called world structure (Watts Strogatz 1998). Intuitively, a larger C implies that a random walker starting from given node k will have larger number of paths bringing it back k. In such scenario, random walkers tend to cluster in neighborhoods around their starting point a property that could help encode different semantic aspects in different regions of g. Another consequence is that one could expect schemata composed of repeated symbols. Figure showws the degree distributions of HSN(50, 5). Here we see another aspect on which the schema networks differ from a purely random graph. In particular, the former are more densely connected than the latter.
Schemata and semantics. To qualitatively grasp the semantic content of the learned schemata we take advantage of the labels available to both Yahoo and Yelp corpora. For example, Figure 3 displays the random walk distributions over the schema networks for four (4) subsets of Yahoo, as inferred with HSN(50, 5). Similar plots for all subsets (labels) of both corpora, extracted with all our HSN configurations can be found in Appendix E Note how the "hot" symbols per category reside different regions of the graphs of G and yet, the "Science & Math" schemata (both nodes and edges) of Yahoo are closer to the "Education & Reference" schemata than to the "Sport" schemata, where closer nodes in the figure indicate well-connected nodes in the underlying graph g. We can understand these findings indicating that the schemata indeed encode semantic notions of their corpora. A similar picture
suspected already from the large clustering coefficient
Dataset
n. edges
D
CC
largest CC
PTB
694.26 9.47
2.00 = 0.00
1.43 + 0.01
0.83 0.01
1.00
0.00
50.00 0.00
YAHOO
892.67 8.22
2.00 = 0.00
1.24 0.01
0.84 0.00
2.00
0.04
49.00 0.04
YELP
891.06 6.50
2.73 0.46
1.24 + 0.03
0.84 0.01
2.24
0.85
48.76 0.85
Random
611.69 = 17.61
2.00 0.00
1.50 0.01
0.50 0.02
1.00 = 0.00
50.00 0.00
Table 3: Statistics of Schema Networks per corpora with K 50 and L 5. Random denotes Erdös-Rea Rénös-Rényi model with p = 0.5 for the corresponding K.
8

---

Under review as a conference paper at ICLR 2023
COMET(GPTXXI) COMETGPTa
COMET (BART)
HSN
HSN[pooo
HSN(KD]
BLEU-2
0.225
0.300
0.330
0.332
0.067
0.125
BERT Score
0.486
0.638
0.650
0.782
0.435
0.561
Table 4:
Metrics of object generation quality for ATOMIC dataset. COMET and COMET (BART) results were extracted from Hwaag et al. (2020). COMET computed by us. The HSN models have K- = 50, 20. All models greedy decoding for all text prefixes in the dataset.
holds for Yelp. Finally, we have also defined and explored "schema interpolations" (Appendix F and have investigated how the schemata attended to by the model (Appendix F). These experiments (qualitatively) show too that the schemata encode different semantic notions natural language.
COMMONSENSE REASONING GENERATION
It has been proposed recently that large, pretrained language models fine-tuned on (natural language) knowledge graph (KG) tuples, can express their encoded knowledge through language generation, thereby providing commonsense knowledge on demand (Bosselut et al. 2019 Hwang et al. 2020). These commonsense KGs live however in data (i.e. text) space the nodes and edges are represented by either single words sequences of them. This observation led to investigate whether one could use the COMET framework of Bosselut et al. (2019), together with the inductive biases of HSN, to translate the implicit knowledge of pre-trained models into KGs in representation space. Arguably so abstract a KG could encompass larger commonsense KGs in data space. With this intuition in mind, let us revisit the COMET framework.
Task, datasets and baselines. Consider a training KG of natural language tuples of the form (s, o), where (xi-- ...................... x labels the phrase subject of the tuple, r x" is the relation token and -xp, (xx....,,x") x is the phrase object of the tuple. The task is to generate the object o, given and r. In other words, to infer the distribution p(o r)). COMET finetunes its pretrained models by maximizing the likelihood of the object, conditioned on the sequence [s. r] = (xf. ..... x(a] x")
(Bosselut et al. 2019). In contrast, HSN is trained to auto-encode the complete sequence [s. o) =
(Xi...... x x xy...., ) and is evaluated on object generation tasks, conditioned not only on [s, but also on the schema ejuie For this preliminary study we focus on the ATOMIC dataset (Sap
al., 2019a), evaluate the quality of the generated objects with both, BLEU-2 (Papineni al. 2002) and BERT Score (Zhang* et al. (200)) metrics, and compare against GPT-2, GPT-2-XL and BART, all trained within the COMET framework (Hwang al. 2020).
Results. Table 4 shows HSN outperforms all baselines which entails it successfully infers and interprets schemata encoding the KG tuples. These schemata, however, are inferred via a posterior of the form go (2 4 [[s, r), A 40 (z +1:z [s. o). 2. A) see Appendix for details. Yet, in practice, one does not have access to any object during inference. The classical solution, la Kalman Filter, is to replace q 11:1.11 [s, r, o), z A) with a local prior model of the form po + lis. [s. 2 A),
and train the latter via the KI. term in Eq. 12 Maximizing the mutual information, however, averages out all local information from the prior and hinders its learning see e.g. HSN in Table 4 An alternative is to train, in the spirit of knowledge distillation (Hinton et al. 2015). third-party model on the inferred schemata, to predict z L conditioned on 2 Our preliminary results, reported as HSN in Table improve upon HSN and even outperform COMET the BERT Score.
CONCLUSION
We introduced a novel representation learning algorithm for natural language modelling that infers discrete, relational representations which allow for compositionality. Experiments show our model learns representations encoding high-level semantics of natural sentences, thereby adding some novel layers of interpretability to large, pretrained language models.
Note, in particular, that BART Lewis al. 2020 has 400M parameter, whereas HSN has 250M.
9

---

Under review as a conference paper at ICLR 2023
8 REPRODUCIBILITY STATEMENT
We provide source code to reproduce our results as supplementary material. The README.rs file within it contains instructions to install and run the corresponding libraries. The synthetic datasets of section aree also provided within the source code file, in the data directory. All other datasets we used are available online and automatically downloaded by our training scripts.
We additionally provide explicit derivation and/or details for all mathematical expression within the main text in the Appendix. Details hyper-parameter selection and training also be found in the Appendix.
REFERENCES
Joshua T Abbott, Joseph Austerweil, and Thomas L Griffiths. Human memory search as a random walk in semantic network. In NIPS, pp. 3050-3058, 2012.
Albert-László Barabási and Réka Albert. Emergence of scaling in random networks. science, 286 (5439):509-35 1999.
Frederic Bartlett. Remembering: A Study in Experimental and Social Psychology. Cambridge University Press, 1932.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Matcusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 1xXiv:186.012211 2018.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. neural probabilistic language model. Journal of machine learning research, ((Feb)::13771111:
2003. 2003.
Hanoz Bhathena, Angelica Willis, and Nathan Dass. Evaluating compositionality of sentence representation models. In Proceedings of the Sth Workshop on Representation Learning for NLP, pp. 185-193, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v11 1/ 2020.rep14nlp- 1.22. URL htLhttps::///
Christopher M Bishop. Pattern recognition and machine learning. springer, 2006.
Ned Block. Advertisement for semanties for psychology. Midwest studies in philosophy. 10: 615-678, 1986.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. Comet: Commonsense transformers for automatic knowledge graph construction. arXiv preprint arXiv:1906.05317, 2019.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from continuous space. arXiv preprint 1rXiv:1511.06344 2015.
Ronald J Brachman. What's in concept: structural foundations for semantic networks, International journal of man-machine studies, 9(2):127-152, 1977.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. arXiv preprint arXiv:2005. 2020.
Ruben Cartuyvels, Graham Spinks, and Marie-Francine Moens. Discrete and continuous represen- tations and processing in deep learning: Looking forward. Al Open, 2:143-159, 2021. ISSN 2666-6510.
America Chambers, Padhraic Smyth, and Mark Steyvers. Learning concept graphs from text with stick-breaking priors. In J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta (eds.), Advances in Neural Information Processing Systems, volume 23, Curran Associates, Inc., 2010.
Allan M Collins and M Ross Quillian. Retrieval time from semantic memory. Journal of verbal learning and verbal behavior, 8(2):240-247, 1969.
10

---

Under review as a conference paper at ICLR 2023
Thomas M Cover and Joy A Thomas. Information theory and statistics. Elements of information theory, 1(1):279-335, 1991.
Jacob Devlin, Ming-Wat Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
bidirectional transformers for language understanding. arXiv preprint arXiv:1810.048055 2018.
Paul Erdös and Alfréd Rényi. On random graphs i. Publicationes Mathematicae Debrecen, 6: 290-297, 1959.
Le Fang, Chunyuan Li, Jianfeng Gao, Wen Dong, and Changyou Chen. Implicit deep latent variable models for text generation. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNIP-IJCNLP)), pp. 1946-3956. Association for Computational Linguistics, 2019.
JR Firth. A synopsis of linguistic theory, 1930 55. Studies in Linguistic Analysis. London: Black, 1957.
Jerry A Fodor and Ernest Lepore. The compositionality papers. Oxford University Press, 2002.
Jerry Fodor and Zenon Pylyshyn. Connectionism and cognitive architecture: critical analysis.
Cognition, 28( (8(1-2)::3 112):3-77, 1988.
Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, and Lawrence Carin. Cyclical annealing schedule: A simple approach to mitigating KL vanishing. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers). Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
Artur d' Avila Garcez and Luis C Lamb. Neurosymbolic ai: the 3rd wave. arXiv preprint arXiv:2012.00586., 2020.
Jesús Gommez-Gardenes and Vito Latora. Entropy rate of diffusion processes on complex networks. Physical Review E, 78(6):065102 102 2008
Thomas L Griffiths, Mark Steyvers, and Alana Firl. Google and the mind: Predicting fluency with pagerank. Psychological science, 1069--0666 2007a.
Thomas L Griffiths, Mark Steyvers, and Joshua B Tenenbaum. Topics in semantic representation. Psychological review, 14(())211, 2007b.
Aric A. Hagberg, Daniel A. Schult, and Pieter J. Swart. Exploring network structure, dynamics, and function using networkx. Gaël Varoquaux, Travis Vaught, and Jarrod Millman (eds.), Proceedings of the 7th Python in Science Conference, pp. 11 15, Pasadena, CA USA, 2008.
Zellig S Harris. Distributional structure. Word, 10(2-3):1461166 1954.
John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4129 4138, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015. URL http:://t/ orq/abs hlg/abs/// 1503. 1.003..00 02531
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8): 7735-17807 1997.
Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama. Learning discrete representations via information maximizing self-augmented training. In Proceedings of the 34th International Conference Machine Learning Volume 70, ICML'17, pp. 1558-1567 JMLR. 2017.
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: how do neural networks generalise? Journal of Artificial Intelligence Research, 67:757-795, 2020.
11

---

Under review as a conference paper at ICLR 2023
Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejn Choi. Comet-atomic 2020: On symbolic and neural commonsense knowledge graphs. arXiv preprint arXhv:2010.05553,, 2020.
Ray Jackendoff. An argument about the composition of conceptual structure. American Journal of Computational Linguistics, pp. 69-73, 1978.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144. 2016.
Lukasz Kaiser and Samy Bengio. Discrete autoencoders for sequence models. arXiv preprint 1XXiv:801100997, 2018.
Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. Fast decoding in sequence models using discrete latent variables. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pp. 2390-2399. PMLR, 10-15 Jul 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. 1rXiv: 14, 2013.
arXiv preprint
S. Kullback and R. A. Leibler. On information and sufficiency. The Annals of Mathematical Statistics, 22(1):79-86, 1951.
Brenden M Lake, Tomer D Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Renaud Lambiotte, Roberta Sinatra, J-C Delvenne, Tim S Evans, Mauricio Barahona, and Vito Latora. Flow graphs: Interweaving dynamics and structure. Physical Review E, 84(1):017102, 2011.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequeeec pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, Online, July 2020. Association for Computational Linguistics.
Bohan Li, Junxian He, Graham Neubig, Taylor Berg- Kirkpatrick, and Yiming Yang. A surprisingly effective fix for deep latent variable modeling of text. arXiv preprint 1909v 1909.00888, 2019.
Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng, Xiujun Li, Yizhe Zhang, and Jianfeng Gao. Optimus: Organizing sentences via pre-trained modeling of a latent space. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4678 4699, Online, November 2020. Association for Computational Linguistics.
Mitchell Marcus, Beatrice Santorini, and Mary Ann Mareinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19, 1993.
James I. McClelland and Timothy T Rogers. The parallel distributed processing approach to semantic cognition. Nature reviews neuroscience, ((4):310-222, 2003.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation word representa- tions in vector space. arXiv preprint 17X.1:1301.3781, 2013.
Marvin Minsky. framework for representing knowledge. MIT-AI Laboratory Memo 306, 1974.
Marvin Minsky. Minsky's frame system theory. In TINLAP, volume 75, pp. 104- 14--116,11111 1975.
Jeff Mitchell and Mirella Lapata. Vector-based models of semantic composition. In proceedings of ACL-088 HLT, pp. 236-244, 2008.
12

---

Under review as a conference paper at ICLR 2023
Kishore Papineni, Salim Roukos, Todd Ward. and Wei-Jing Zhu. Bleu: method for automatic the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics.
evaluation of machine translation. In Proceedings of the 40th Anmual Meeting the
Juyong Park and Mark EJ Newman. Statistical mechanies of networks. Physical Review 70(6): 0661 17, 2004.
Jean Piaget. Le langage et la pensée chez l'enfant: Études sur la logique de I' l'enfant. Delachaud & Niestlé, 1948.
M Ross Quillan. Semantic memory. Technical report, BOLT BERANEK AND NEWMAN INC CAMBRIDGE MA, 1966.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference machine learning, pp. 11278-1286 PMLR, 2014.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. primer in BERTology: What we know about how BERT works. Transactions the Association for Computational Lingussicss,, 8:842-866. 2020. doi: 10.1162/tcc_ a a_00049. URL https: https::/aclantollooo//. 220.://oooooo..oog22222...a....
David E Rumelhart. Schemata: The building blocks of cognition. In Theoretical issues in reading comprehension, pp. 33-58. Routledge, 2017.
Maarten Sap. Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. Atomic: An atlas of machine common- sense for if-then reasoning. 809-1.
11111///AA11////AA 19. AAAI Press, 2019a. ISBN 178-1-57735--
Maarten Sap. Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. Atomic: An atlas of machine common- sense for if-then reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3027-3035, 2019b.
Ray Solomonoff and Anatol Rapoport. Connectivity of random nets. The bulletin of mathematical hiophysics, 13 1951.
Paul Thagard. Frames, knowledge, and inference. Synthese, 61(2):233-22 1984.
Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu, Neural discrete representation learning. In I. Guyon, U. Von Luxburg, Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008. 2017.
Duncan J Watts and Steven H Strogatz. Collective dynamics nettoos..rdddnttto..s nature, 393 (6684):440-443) 1998.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 11903, 2022.
13

---

Under review as a conference paper at ICLR 2023
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing: System Demonstrations, Online, October 2020. Association for Computational Linguistics.
Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg- Kingg-rrkattickk Improved variational autoencoders for text modeling using dilated convolutions. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3881-3890, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR.
Lang Yu and Allyson Ettinger. Assessing phrasal representation and composition in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 4896-4907, Online, November 2020. Association for Computational Linguis- tics. doi: 10. 10..180000 0665:////2020..mmmmmmmmiii//
URL https: pttps::/aaclattthoooggy. www.eogg.....g/ooo 2020
emnlp-main. 397
Tianyi Zhang*, Varsha Kishore*, Felix Wu", Kilian Q. Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL htttp:::p///
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Infovae:
Balancing learning and inference in variational autoencoders. Proceedings of the AAA/ Conference on Artificial Intelligenne, 33(01): 5885-5892, Jul. 2019. URL hatps:: view/4538
aaai wtt.aaii..../ieeeeeeeeeeeeeeeeee phpp/AAI/articl....
Tiancheng Zhao, Kyusong Lee, and Maxine Eskenazi. Unsupervised discrete sentence representation learning for interpretable neural dialog generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1098-1107. Association for Computational Linguistics, 2018.
Zachary M Ziegler, Luke Helas-Kyriazi, Sebastian Gehrmann, and Alexander M Rush. Encoder- agnostic adaptation for conditional language generation. arXiv preprint arXiv: /908. 06938, 2019.
14

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A PSEUDO-SELF ATTENTION MECHANISM REVISITED
The attention mechanism of the original Transformers (Vaswani et al. 2017) is defined as
Attention Q, K, V) softmax (D- Q K? V.
(13)
where Q, K and V RT'x are sets queries, keys and values, respectively, given by a sequence of T. D-dimensional vectors, packed into matrices. In practice, these queries, keys and values are projected many times with different learnable, linear maps. The Attention operation (Eq. [3) is performed on these different projections in parallel, whose outputs then concatenated and projected once more with a final, linear map. The complete operation is known as Multi-head Attention (Vaswani al. 2017), and
this notation in Fig. [of the main text.
Now, the question is how to condition GPT-2 on the schema Cjr Given a sequence of input representations uI:T, the self-attention mechanism in GPT-2 is obtained by choosing Q u1.7 Wo, K = u1:7 Wx and V u1:T Wv, all in RT'D, with We, W and Wy E IR DxD pretrained matrices. We leverage a pseudo-self attention (PSA) mechanism (Ziegler et al. 2019) that augments the key and value matrices in their first L rows, with projections of e/13z so that
K @jisi W K
K + Penc
V
We
Pene
E B((++))
(14)
where Pene is positional encoding, just as the one used in the original Transformer implementation (Vaswani et al. 2017). The latter informs GPT-2 about the ordering selected by the random walk process. PSA is then simply given by Eq. 13 with the keys and values replaced with the augmented ones, K and V. The Wi: K We here are randomly initialized, learnable parameters mapping the schemata onto the decoder self-attention, D-dimensional space, and we have many of them as layers in GPT-2. Therefore this mechanism allows GPT-2 to attend the projected schema at each of its layers, with a minimal addition untrained parameters (Ziegler et al. 2019)
symbols in the schema, as
B TRAINING OBJECTIVE
The Evidence Lower Bound (ELBO) of the Hidden Schema Network model reads
N
Σ
E
(x()) A)a(AA log (x (n) (x z1.z)
(x
N
E
KL
gdl (Z1:L
x x
A) A): p(zz.)
(L()
(15)
where KL[-] denotes the Kullback-Leibler (KL) divergence.
Note that this is not the training objective with the mutual information between sentences and schemata. We give details about this modified objective in subsection belllo... Before getting into that, let us first calculate the explicit expressions for the two divergences above.
main text. There we maximize the ELBO together
B. KULLBACK- BETWEEN RANDOM WALKS
For notational convenience we will not write the explicit dependence on the graph A in what follows. Using the explicit product form of the probabilities walks leads
(2.-: xeot)a log (0 (2 1, (1))
(1) p( KL [
1177
pln, (2,-1)
KL p(((2))
(16)
15

---

Under review as a conference paper at ICLR 2023
Graph C
Model
NLT.
KI.
KT- G
ADC
10"- D/F
(0 -FF
N. odgest))
$3.071:
Barabasi
IIS (0.1)
53.08
0.01
0.10 1 0.06
9
0.977
0.003
17 2
1090 1: 143
HS (0.2)
53.07 0.02
0.09 + 0.06
4.8 4: 0.5
0.989
0.001
17:22
1360 + 104
LSTM
48,24
0.02
Erdos
HS (0.5)
50.9 0.8
12
4
0.95
0.06
34.8 0.9
2812 + 344
HS
1.10.11
1
36.8
3.
3131 156
Table 5: Inference on ground-truth random graphs. Here we use the notation HS(p) to denote Hidden Schema Network models with prior graph distributions whose edge probability is set p.
where qo(z,) (z(2)(x),7)
(n)) is the aggregated probability over all walks until step Since the random walks are Markovian, can be explicitly written as
(z z
II ISi
QU (n) Ø) p(x o),
(17)
where the (posterior) class probabilities the walks' starting points p. and the transition matrices
Q are defined in Eqs. 10 and 11 of the main text. Using the definitions in Eqs. 4 and we can write the argument the expectation value in Eq. [6 above as
go(z, 11 (n)
1] (x 1x1 Φ)
(n)
k.j
log
log
(18)
plz Zi-1)
Pk.j
which means we only need to compute the expectation of the product shown to be
2/ --- This one can easily be
- -/ (x(") ②) of (x d).
-i
E (2. Jas
(19)
where 2/51 (x/1) Φ) is the jth class probability of q (x defined in Eq. Finally, the second KL term in Eq. ca be directly evaluated
17
K Σ
Ps (x ②) log Py
p=(x
(n)
(L))
(20)
Pj
1=1
(n) where p5 (x Φ) and Pj are, respectively, the posterior and prior class probabilities for the random walks' starting points.
Method
PTB
YAHOO
YELP
GPT2 [one epoch)
24.23
22.00
23.40
GPT2 [fine-emed]]
19.14
20.64
19.77
iVAExM
53.44
47.93
36.88
Optimus
23.58
22.34
21.99
HSN (100.
17.61
19.68
18.99
HSN (100,5) 5)
17.69
19.84
19.00
HSN
16.88
19.59
19.01
HSN (60, 5)
17.41
20.06
18.95
HSS (100,
17.6 -: 0.1
20.1 +: 0.2
19.12 1 0.08
HSN (100, 5)
17.79 + 0.09
20.0 0.1
19.07 + 0.08
HSN (80, 20)
17.0 + 0.1
19.8 0.2
19.3 0.5
HSN (60, 5)
17.44 + 0.07
20.09 + 0.05
18.9 + 0.1
Table 6: Perplexity per word (lower is better). The last four (4) rows show the mean and standard deviation obtained after training five (5) HSN.
16

---

Under review as a conference paper at ICLR 2023
YAROO
Model
9
KL
KL 9
KL
KL
IIS I 20]
2.0 1 0.3
0.004 1. 0.002
1.3 1 0.6
0.005
1 0.003
1.9 0.7
0.011 0.002
HS [L 5]
1.33 1. 0.05
0.0013 1. 0.0009
1.54 1 0.05
0.0006 1 0.0006
1.50 1: 0.09
0.002 1: 0.002
Table 7: Kullback-leibler divergence for 100-symbol HSN models (trained 5 times) in all datasets
Putting all together write
Qk5
(x
(x (
KL(ga(Z1:1 p:
k.
[x 1:
Φ) p
(x Φ) log
Pko
im kjj=1
K
Pj P (n)
Py 1:7: Φ) log
(21)
Pj
B.2 KULLBACK-LEIBLER BETWEEN RANDOM GRAPH MODELS
Since both prior and posterior graph models treat each edge in & as a Bernoulli random variable, we can write directly
(Polo,,0)) 03)
(L(g(A)lp(A)) Σ
polo;: log
p
- pa(e; ej)
+(1 - (p 0;)) log
(22)
p
where (e,, e,) is the posterior link probability, which is conditioned on the symbols connected by the link, and p is the global prior probability over all links, as defined Eq.3lof the main text.
B.3 MAXIMIZING MUTUAL INFORMATION
We would like to maximize the mutual information between the word sequences in our dataset and the schema representations. We have argued that the training objective in the main already includes such a mutual information term. To see this is indeed the case we need to workout some identities.
Let us. for simplicity of notation, consider two discrete variables z and x, the last of which follows an unknown distribution (x)).. What follow are identities
E -xp (l.(g(2/x)::p(2) = (xp(x) Ry(aix) logg(z) log( 10g(2/x))
(z/() + Σ pD(x) Σ ((z1x)) { logp(z) + logg" (z) - logg (2)
=H,(z|x) H1g (z) + Σ (z) logp(z) - logg (2))
x); Eyy(o) { log gz(z)
p(z)
= -1( - KLLg (z);
(23)
where
Hz(zlx) = Σ pD(x) Σ
q(z log (z)xx)
(24)
the conditional entropy with respect to distribution q (see e.g. page 17 in (Cover Thomas and
[991
17

---

Under review as a conference paper at ICLR 2023
Dataset
edges
1
CC
Iargest CC
PTH
694.26 =: 9.47
2,00
1: 0.00
144 0.01
0.33 1 0.01
1,00 = 0.00
50.00 1 0.00
HSN(50, 5)
YARGO
YELP
891.06 = 6.50
2.73 = 0.46
1.24 0.03
0.84 = 0.0
2.24 = 0.85
48.76 + 0.85
Random
611.69
17.61
2,00 = 0.00
1.50 0.01
0,50
+ 0.02
1,00
0.00
50.00
0.00
218
764.28
0.01
0.77
0.77
FS, 20)
YAHCO
356.35 1: 7.76
3.17 1: 0.37
1.57 1 0.05
0.58 1 0.02
12.04 1: 1.49
38.96 1. 1.49
YRT.P
Random
611.69 + 17.61
2,00 = 0.00
1.50 0.01
0.50 0.02
1,00 = 0.00
50.00 + 0.00
PTR
1198.18 = 16.44
2.56 + 0.50
1.76 0.01
0.83 0.01
1,19 +
99.81 +: 0.40
HLN(100,
1239.21 12.19
0,51 0001
35,93 1.41
65.07 1.41
YELP
1295.68 = 12.93
3,36 = 0.48
1.55 + 0.03
0.57 = 0.01
27.38 = 1.69
73.62 + 1.69
Random
2474.92 = 36.58
+
1.50
0,50 0.01
1,00 0.00
PTP
892.53 10.04
3.04 1 0.24
141 0.04
0.45 0.00
46.04 1.53
54.96 1.54
YAHCO
261. = 7.14
2,18 + 0.38
1.95 0.00
0.91 0.01
1.01 0.10
99.99 0.10
HLN(100, 20)
515.84
0.38
45.27 2.58
55.67 2.58
Random
2474.92 = 36.58
2,00 = 0.00
1.50 + 0.01
0.50 + 0.01
1.00 = 0.00
100.00 + 0.00
Table 8: Statistic of inferred graphs for all datasets
Hx) (z)= -x)====- (x))
(25)
the entropy of distribution (z), which we define as the marginal (data-aggregated) distribution
('(z) = Σ (D(x)
(26)
Finally, we used the definition mutual information
T(x: z) Hg- (z) - (2(2)x).
(27)
See e.g. page 20 in (Cover & Thomas 1991).
It follows from Eq. 23 that maximizing the ELBO (Eq. 15 together with the mutual information between word sequences and schemata, simply amounts to replacing the KI. between the approximate posterior and prior random walk distributions, with the KL between the aggregated posterior and prior random walk distributions. To wit
Σ (o) KL gol (z1.) x(1) A); A): p(Z)r))) IZI1L: x A)
(A)
(A) KL (1
(28)
where we introduced the aggregated posterior over random walks wrt the word sequence
N
Ls (21.L) ( (20(211))x1)r)]
Σ
go(2).1 ( x
(29)
N
In practice we approximate this quantity with (z(z)) (z))
II t=2
do (z A),,
(30)
where 9 (z1) is categorical distribution whose class probabilities (0) are the average of those from our approximate posterior (Eq. [0 in the main text)
(}(0) Σ py (x(x Φ)
(31)
and the transition probabilities
2 (2 12, A) have transition probability matrices
N
16 (A, 6)
2
Σ
Φ). n=l
(32)
N
18

---

Under review as a conference paper at ICLR 2023
B.4 MEAN-FIELD SOLUTION
Instead of modeling the posterior over random walks with Eq. of the main text, we could consider mean-field decomposition along the time component, by ignoring the dependency on the graph
ao (z1.L) (1L)) = II do (Z(())))r))
(33)
where at each step of the walk we have a step-dependent categorical distribution
K II j=1
P (xI:, x))
of
(z X1:T)
(34)
whose class probabilities live in the K K-simplex. We could model the latter via
softmax ..... h2²")
(35)
where h h...,,h, main text.
he are the outputs of our encoder neural network model, shown in Figure T of the
Replacing the mean-field approximation of 33 into 15 yields
E
log (z: (x)
to (x.1)))))
E
log (ggp(zi/2,-1) + (2)gg(2)):
qo (2.
(xI:T, log Pj (XI.T, Φ)
j
E
2 2/
log Pk.j
(2: (x (0s
im2 k,j
K
Pg 1 (x1:T, Φ)
(XI.T, Φ) log
(36)
Pj
[5] Pk (x1T,5)) 1 (x1:T, x) log Pa.s
(37)
2 RJ
B.5 FULLY CONNECTED GRAPH
We replace the adjacency matrix A in the definition of the transition probability matrix of our posterior Q(x1TT, A, о). with that of a fully connected graph. The aggregated posterior over all walks up step (Eq. above) reduces in this case
Aii (XITT Aku Σ pli (XIIT Am.)
pl (xI:1,
Σ
D) (x Φ)
(x1:T)
11-11 (xI,T, ②)
Σ (
(38)
Σn pii (xITT
Em pii (xI:T, +)
B.4 with 2//1 =p 11]
which is equivalent that of the mean-field approximation of section
19

---

Under review as a conference paper at ICLR 2023
C HIDDEN SCHEMA NETWORKS ALGORITHM
Algorithm I: HSN Training (Ø, 1/)
foreach minibatch X1:T ((D) do
(1) Sample schema network from posterior graph model:
A
Ls (A),
(2) Compute parameters of posterior random walk model:
h
cne
ene h2
h,.
h h (x1:T)
(()) = softmax(h3 (h?""
11 Auj
k.j (Ø)
with f(4)
f(h-1] exp(h)%))
Σ.. sld (o) Amj
(3) Compute parameters of prior random walk model:
Aks
PRj E fiAj
(4) Sample random walks from posterior distribution:
21:L qo (0(Z1:L X1.T, A)
(5) Decode sentence:
for
0 do
X; px (nsssi)
n softmax(W (W. (xcxx B)oj)))
end
(6) Compute loss and back-propagate:
C10,0] Σ
log log (n) 21-1)
Vola, v A)aa(A)
(A) KL (el(annlAA):s(())])) (L)
end
D ON SYNTHETIC DATASET EXPERIMENTS
In this section give additional details of and results from proof-of-concept experiments.
D.1 SYNTHETIC LANGUAGE MODEI.
We generate our synthetic dataset as follows: first, we sample single, fixed graph G with K nodes from a predefined random graph model. Second, we define set of random tokens V, of size V, to be our vocabulary. We create each token as random 3-tuple from the Latin alphabet, and choose to have at least one order magnitude more tokens than nodes G (that is, V >> K). Third, we assign a random bag of tokens each node in G* These random bags can simply be understood as probability distributions over V, and can be represented as d-dimensional vectors whose components live the simplex. Note in particular that, by construction, tokens can be shared among the different nodes of G*. Finally, let us identify the random bags with the symbols {e) e ex } of the synthetic language model.
20

---

Under review as a conference paper at ICLR 2023
To generate synthetic sentences sample uniform, L-step random walks G", whose transition matrix is given by Eq. in the main text, with f = L. Having obtained a set of random walks on G", sample one random token from each of the symbols (i.e. from each random bag) along the walks.
D.2 EXPERIMENTAL SETTINGS
Here we give additional details for reproducibility
Datasets
Following the procedure above generated two datasets from two random graphs with dif- ferent topologies. One sampled from the Barabási-Alber model (Barabási & Albert 1999), the other from the Erdös-Rény model (Erdös & Kényi 1959). We generate these graphs using NetworkX, a Python language software package for network structures (Hagberg et al.) generate Barabási-Albert graphs by attaching 3 edges from each new node to old ones, and Erdös-Rényi graphs with an edge probability of 0.5. We set both graphs have K 100 symbols.
2008). Specifically,
We define each random bag of tokens in G to have two tokens only (each with equal probability).
We use a vocabulary of 1000 random tokens.
Once the graph is fixed, we the token sequence length L 10 (L 11) for the Erdös (Barabási) datasets and generate total of N = 100000 token sequences from each random graph.
Hidden Schema Network (HSN) settings
We train randomly initialized embeddings of dimension 256, one for each token. We sample these from normal distribution with zero mean and a standard deviation 0.01.
The posterior graph model is defined via single feed-forward neural network with 256 hidden units.
The prior graph model has the edge probability p as hyperparameter. We crossvalidate from the set p {0.1, 0.2, 0.5, 0.6, 0.8} and found that HSN could fit the Barabási dataset only with small values (0.1, 0.2}. HSN could the Erdös dataset with larger values (0.5, 0.6]
The posterior random walk model defined by replacing BERT with a 2-block Transformer encoder (Vaswani et al. 2017), each with 2 heads, 256 hidden units and dropout probability of 0.2.
The prior random walk model was set uniform random walk.
Training details
We use batch size of 256 and train with Adam (Kingma & Ba 2014). with learning rate 0.0001, in all experiments.
To sample both graph and random walk posterior models with use the Gumbel-Softmal trick (Jang et al. (016)), with a constant temperature of 0.75
We train the models for 200 epochs
D.3 ADDITIONAL RESULTS
T displays the mean and standard deviation of some additional results on our proof-of-concept experiments. We trained ten models in total.
We first trained a simple LSTM Network to infer the correct symbol order in each random token sequence. We noticed that a network with 256 hidden units was enough solve this task perfectly. Indeed, the negative log-likelihood (NLL) of these models corresponds to choosing the 2-token random bag sequence (i.e. the schema) that yields the correct token sequence without errors. The HSN performs equally well on the Barabási dataset, and slightly worst on the Erdös dataset. In fact, have noticed the Erdös dataset proved be more challenging learn with the HSN in all regards.
21

---

Under review as a conference paper at ICLR 2023
See, for example, the AUC scores the Frobenious norms of HSN in this dataset, as compared to the Barabási case. We think this might be due to the fact that Barabási graphs have more structure, simply because of their sparsity, which arguably make them easier to infer with inductive bias.
Note also how increasing the prior edge probability affects the average number of edges of the inferred graphs.
E ON LANGUAGE MODELLING EXPERIMENTS
In this section we give additional details of and results from our language modelling and representation learning experiments.
E.
EXPERIMENTAL SETTINGS
Here give additional details for reproducibility
Datasets
We consider three widely used public datasets, namely the Penn Treebank (PTB) (Marcus et al. 1993), Yahoo and Yelp (Yang et al. (017) corpora.
PTB training set has total of 38219 sentences. The average length of which of about 22 words. The validation and test set have 5527 and 5462 sentences, respectively. The minimum (maximum) sentence length in PTB is (78) words.
Yahoo training set has a total of 100000 sentences. The average length of which is of about 80 words. The validation and test sets have 10000 sentences each. The minimum (maximum) sentence length in Yahoo is of 21 (201) words. The Vocabulary size is of 200000 words.
Yelp training set has a total of 100000 sentences. The average length of which is of about 97 words. The validation and test sets have 10000 sentences each. The minimum (maximum) sentence length in Yelp is of 21 (201) words. The Vocabulary size is of 90000 words.
HSN settings
In all experiments we leveraged pretrained BERT and GPT-2 models, both with 12 layers, 768 hidden dimensions (D) and 12 attention heads. We used the public HuggingFace implementation of both these models (Wolf et al. 2020)..
The posterior graph model is set 2-layer feed forward network, each with hidden dimension 512.
We crossvalidated the prior edge probability over the set of values (0.1, 0.2, 0.5, 0.6) and found p 0.5 (a maximum entropy prior) to yield the best results, All results we report correspond to this (p- 0.5) case.
We also train inhomogeneous random walk prior model by making and the sequence of weights f(1l, 2 ... f trainable. We initialized them by sampling from normal distribution with zero mean and standard deviation of 0.01.
We experimented with HSN of K = (50, 100} symbols and random walks of length L (5, 20}.
Training details
We used a batch size of 32 and train with Adam (Kingma & Ba 2014). with a learning rate of 0.00001, in all experiments.
To sample both graph and random walk posterior models with used the Gumbel-Softmal trick (Jang et al. 2016). with a constant temperature of 1.0.
We used cyclical schedule to anneal both KL terms in our training objective from zero to one (Fu et al. (2019). When the annealing weight (usually called in the literature) is finite, used KT. threshold scheme (Li et al. 2019), with threshold value of O.
22

---

Under review as a conference paper at ICLR 2023
We trained the models for 100 epochs, although they usually needed about 60 epochs only to converge (in the NLL).
We applied word dropout to the input of decoder model with probability 0.3 in the following cases: (i) for all models trained on PTB; (ii) and all models with L 50 trained on all datasets.
E.2 ADDITIONAL RESULTS
Here report results complementing the conclusions of the main text.
Language modelling. Table 6 displays our perplexity results on all datasets, just as the main additionally report the mean and standard deviation obtained
text. In the last four when repeating the experiments with the HSN model five times, with different initializations. The conclusion of the main text, viz. that our results outperform all baselines, remains unaltered, even within error bars. We additionally report in Table 7 the mean values of the KL for five 100-symbol HSN runs.
Graph statisties. We characterize the structure of G in terms of five statistics: (i) the diameter D, which measures the maximum path length over all node pairs in 9; (ii) the average distance 1, which instead measures the average shortest path length between all node pairs; (iii) the clustering coefficient C, which represents the probability that two neighbors of a randomly chosen node are themselves neighbors; (iv) the number of connected components CC; and (v) the degree distribution P(k). which represents the probability that a randomly chosen node will have neighbors.
Table 8 reports the statistics of our inferred graphs for all datasets, and all model configurations.
We can see that increasing the random walk length from 20 increases the number of connected components of the graphs. As consequence, subsets of word sequences are map onto smaller subgraphs, the larger of which is about 50 symbols. One could argue that, since longer random walk lengths imply larger of possible schema configurations, the number of symbols required to describe our three corpora can simply decrease. other words, less symbols are needed by long schemata. Similarly, directly increasing the symbols number leads connected components. Indeed, even the short schemata in Yelp and Yahoo do not use all available symbols model the corpora.
larger number of
Representation learning. We can get graphical picture of the features we just discussed above in Figures 11 below. Very importantly, we see that the schema distribution different for each category of each corpora in all model contigurations. In other words, we do not observe any kind of mode collapse.
Finally, we have also explored "schema interpolations": given two schemata Gjrj, and em may. we find the shortest path (of length 1) on G connecting the end of ej with the beginning of 6m Our interpolation steps are the schemata {en tijt +i A 0 < i S 1 I, along the path). Tables 10 12 show interpolations of random instances from all datasets. Note how the model successfully interpolate between categories in both Yelp and Yahoo.
23

---

Under review as a conference paper at ICLR 2023
layer
KL(good, bad)
KL(good, great)
KLggreat, bad)
0.807
0.336
1.227
5
0.738
0.177
1.245
12
0.635
0.224
0.957
Table 9: Kuibeeck-Leiberr divergence between the distributions of most attended symbols. when generating the tokens good, bad and great. Results are computed with HSN(100, 5) trained on Yelp. The KI. values are computed for each head separately and then averaged.
F WHICH SYMBOLS DO WORDS ATTEND TO? PRELIMINARY STUDY ON YELP REVIEWS
In this section we investigate how symbols are used by HSN when generating text. We do this by exploring the decoder attention matrix between the symbols and the generated tokens. Reading the attention wights, we can examine which symbols are most important for the generation of any given token, i.e. which symbols are attended to more strongly. bit more detail we select, for given token in given sentence, the symbol with the highest attention value. We can then compute the distribution of most attended symbols when generating that token for the complete dataset.
Thus, for a model trained the Yelp dataset, examine to which symbols does the decoder of HSN attend to, when processing the words good, great and bad. Figure4sshoe the most attended symbol distribution for layers (first). 5 (middle), 12 (last), when averaging the attention matrices over all attention heads. Figures 6 show these distributions for each head separately. Note how, moves between heads and layers, albeit
for fixed token, the distribution of attention changes there are too some repeating patterns.
We quantify these features by computing the Kullback-Leibler (KL) divergence between these distributions. The KL. values are shown in Table
Interestingly enough, the distribution of symbols that attended to when processing the word great is closer to the distributions of symbols attended by the word good, than to the distributions of symbols attended by the word bad.
24

---

Under review as a conference paper at ICLR 2023
0.05
good bed Steea
0.03
0.02
0.02
0.02
symbol
(a) Layer 1
great
0.03
0.02
0.02
0.00
(b) Layer 5
0.04
0.02
0.00
(c) Layer 12
Figure Distribution of most attended symbols when generating tokens good, bad, great for HSN(100, 5) trained on the Yelp data set. The decoder attention matrices between symbols and output are averaged over all attention heads for layers 5 and 12.
25

---

Under review as a conference paper at ICLR 2023
2.24
4.10
head5
0.20
teadS:
Teed
0.22
Figure 5: Distribution of most attended symbols when generating tokens good, bad, great for HSN(100, 5) trained on the Yelp data set. The distribution is computed from the decoder attention matrices between symbols and output for each attention head for layer 1.
26

---

Under review as a conference paper at ICLR 2023
2.25
0.04
0.2
010
2.34
01.4
headS1
Teed 32
0.30
Figure 6: Distribution of most attended symbols when generating tokens good, bad, great for HSN(100, 5) trained on the Yelp data set. The distribution is computed from the decoder attention matrices between symbols and output for each attention head for layer 5.
27

---

Under review as a conference paper at ICLR 2023
011
012
bead
headS1
Teed 32
Figure 7: Distribution of most attended symbols when generating tokens good, bad, great for HSN(100, 5) trained on the Yelp data set. The distribution is computed from the decoder attention matrices between symbols and output for each attention head for layer 12.
28

---

Under review as a conference paper at ICLR 2023
G ON COMMONSENSE REASONING GENERATION
In this section we expatiate on the details of our approach to commonsense reasoning generation.
First, we modify the encoder component of HSN process the tuples (s. r,
(0(21.1||s r. o). A) 1ol 1: : [[s. r), A (z ( [s,
".. A);
(39)
so that the first half of the schema depends on subject and relation only, whereas the second half depends on the entire 3-tuple. As it will become evident below, this decoupling is necessary for the inference of novel objects.
Each of the posterior distributions above is modelled with the same architecture, as shown in Fig. but sharing single pretrained BERT model. That is, we have two copies of all pink-shaded blocks in the Fig, one for go (2.44 [[s, r], A), the other for ( 4+1:L [s, r, o), 2 A), and single pretrained BERT model.
Using such 2-component encoder model we are able to successfully infer schema representations for the KG tuples, shown in Table The task is however to infer objects, given only subject- relation pairs. We thus need a way to infer schema representations without relying on the phrase object
The classical solution to this inference problem is to replace, à la Kalman Filter, 1s(2 2 +1:2||s, r, o), 2g A) with local, trainable prior model of the form po (2 +1:1 [[s. r), 1,2%. A) where z 1 is sampled from qo(Z). r). and train the prior via the KL term in Eq. 12
As shown Section maximizing the mutual information between data and representations averages all local information in the KL term, and thus hinders the learning of the prior see e.g. HSN[poo in Table 4 the samples from the prior are not close enough to those of the posterior, hence the significant drop in performance of the model.
An alternative is to train, in the spirit of knowledge distillation (Hinton et al. 2015). a third-party model on the inferred schemata, to predict z 4 +1:1 conditioned on 1
Indeed, given the inferred schemata from the training KG, we consider a sequence-to-sequeee model which inputs 1 together with the subject-relation pair, and outputs Z 4411. That is, model of the form pe(zg 4 +11 11 [s. rl). Specifically we use (i) a bidirectional LSTM network with hidden dimension of 512 to encode the first half of the schemata, (ii) a pretrained BERT model to encode the subject-relation pair, and (iii) LSTM network of dimension 512 as an autoregressive decoder model. The initial (hidden) states of the latter are determined by an MIP which inputs the representations from the LSTM and BERT encoder models. The model is trained samples from (z(zz $ +1: r, o), z A).
Our preliminary results, HSNIKD in Table 4 show that this approach improves upon the untrained prior model, and outperforms the stand-alone COMET(GPT-22) model.
G.1 ATOMIC DATASET
For this preliminary study we focus only on the ATOMIC dataset Sap et al. (2019b). It contains 877K (s, r, o) tuples covering a variety social commonsense knowledge around specific If-Then events. A bit more in detail, ATOMIC splits its commonsense knowledge into nine categories, covering the event's causes, its effects on the agent, and its effect on other direct (or implied) participants. We use the training splits from Sap et al. (2019b), resulting in 710K training, 80K validation, and 87K test tuples respectively.
29

---

Under review as a conference paper at ICLR 2023
Society & Culture
Science & Mathematics
Health
Education & Reference
Computers & Internet
Sports
Business & Finance
Entertainment Music
Family Relationships
Politics & Government
(a) HSN(50, 5)
Society Culture
Science Mathematics
Health
Education Reference
Computers & Internet
o
Sports
Business & Finance
Entertalnment Music
Family Relationships
Politics & Government
(b) HSN(50, 20)
Figure 8: Schema distributions inferred from each category of the Yahoo dataset, for HSN(50, L) with L, (5, 20). The node positions in the figure consistent among labels and computed using a force-directed embedding of the global graph G.
with considerable irony the case also shows how completely japan has turned the tables
on u.s. business
(1) in brief the chancellor of the exchequer nigel lawson's 's decisions were justified by their intended political and financial convenience and credit
(2) analysts said they expect the federal authority to be totally revamped giving japanese manufacturers more clear way to measure their exports.
(3) but others say inco commission has been inadequate
(4) in 1970 banco exterior an agency run by banco exterior <unk> de <unk> <unk> attempting
raise the sale certain works by the division the amended filings also point out that under a new agreement <unk> has an <unk> obligation to sell farmers to axa upon acquisition of b.a.t
to reduce liabilities
Table 10: Interpolation between two random instances from the PTB dataset
30

---

Under review as a conference paper at ICLR 2023
Society Culture
Science Mathematics
Health
Education & Reference
Computers Internet
Sports
Business & Finance
Entertainment Music
Family Relationships
Politics & Government
B
(a) HSN(100,
Society Culture
Science & Mathematics
Health
Education & Reference
Computers & Internet
Orocoo
Sports
Business & Finance
Entertainment Music
Family Relationships
Politics & Government
(b) HSN(100, 20)
Figure 9: Schema distributions inferred from each category of the Yahoo dataset, for HSN(100, L)
with L = {5. 20). The node positions in the figure are consistent among labels and were computed using a force-directed embedding of the global graph g.
31

---

Under review as a conference paper at ICLR 2023
very bad
bad
mediocre
good
very good
(a) HSN(50, 5)
very bad
mediocre
good
very good
DOO
oc
CO
o co
Goo0 co e
(b) HSN(50, 20)
very bad
bad
mediocre
good
very good
HSN(100,
very bad
mediocre
good
very good
Seeeaaa doods 800g
(d) HSO((10,, 20)
Figure 10: Schema distributions inferred from each category of the Yelp dataset. The node positions in the figure are consistent among labels and were computed using a force-directed embedding of the global graph G.
32

---

Under review as a conference paper at ICLR 2023
k=50,
50
20
ptb
yelp
yelp erdos
6-
erdos
yahoo
yahoo
10
5-
10
20
30
50
10
20
30
50
Node Degres
Node Degree
K 100,
K 100, 20
ptb
40
ptb
yelp
yelp
erdes
erdos
yahoo
yahoo
Node
20
10
0
0-
20
40
60
80
100
Node Degrea
Node Degree
100,
100, (LCC)
ptb
ptb
yelp
15.0
erdes
12.5
erdos
Courtt
yahoo
6
10.0
Rode
7.5
40
60
100
20
40
100
Node Degree
Degree
Figure 11: Degree distributions of inferred graphs from all corpora, compared to Erdos-Renyi graphs for p 0.5. The upper four plots show results for full inferred graphs, the lowest two show the degree distributions of the largest connected component of the models for K = 100.
33

---

Under review as a conference paper at ICLR 2023
Interpolate Society & Culture Science & Mathematics
steady eye contact good? when i am communicating with someone, fend to give very steady, long .UNK eye contact. so tried to _UNK it as young naive girl, and now it's a habit can't losc... it just depends on the person you are having the conversation with...
(I) i am 14 years old. their is this girl again who speaks very much of me and talks 2 me, the idea of me her and never gives any suggestion verbal UNK for my 2nd. listen, therapy!
(2) what do you do when you think your best friend told you shes bisexual? when she says that,
or you might have believed your friend said it is. they' re inevitably sharing that they don't 't share...
(3) how do you change liquid in an ice cube into liquid form? paste, mix and freeze _UNK a gallon of co2
into UUK and then JUNK in some ice to form a coating.
(4) what kind of rules does gravity apply? if certain weight placed container, the force
applied on it hits the water surface and the right weight will turn into gravity
how does a photovoltaic system that feeds back into the power grid get on the same phase angle? or? should say does it need to be the same the UNK's?
Interpolate: Business & Finance Family Relationships
at 35, am i too old to go to college to become a psychiatrist? i'm 37, and just started my second semester in 2 year college... you need to be prepare for the financial aspects, but the social ones are no problem...
(1) what would be a good title for LUN UUK? have UNK UUK and there are no real courses done for it but do love the job and i've already done my freshman year. i am currently teaching placement
at UNK and need the same as the average undergraduate student...
(2) has anyone here applied in the past months is it better get try out _UNK slightly better long term career _UNK-.. ...
(3) lately im having trouble with my fiancee, how do bring him back? it obvious at this point that you can't 't hring us together ". try playing games. (4) could i still go out with this guy and still be friends and respect him.? i don't 't want just fell in love with the guy that i with. want be with friend's girl and still be friends...
how do know if my man, is inlove with me? well... some questions, how old are UNK?
are you wealthy?, is he wealthy?. how long have you been together...
Table 1: Interpolation between four random instances from the Yahoo dataset
34

---

Under review as a conference paper at ICLR 2023
Interpolate: Very bad - bad
do not use this company!! they told within one hour, then called again they said the driver have 90 mins. 90 minutes later, they said the driver is in traffic and wait for 15 minutes, checked google map no accident, all green on freeway...
(1) i ordered for pick up as my daughter hadn't 't been told that or even ordered online. when i spoke to the young lady, who was _UNK, she carried
conversation with not a manager. it's bad customer service and wouldn't even bother with this place...
(2) place was clean... when called to let them know "d get something else, the person that answering the phone wouldn't 't understand me... really? gave this restaurant a + for the cleanliness of the food and the friendliness of the staff
(3) i had the quesadilla and the carnitas tacos. felt every bite of these were so rubbery and the potatoes were off. feel like the service and the quality of food can do much better. (4) somewhat disappointed.
it once and loved but today, today's 's water bitter and salty... and the mint and cherry blossom UNK' flavors just taste that way.
the food quality doesn't 't match the place at all. think it's ok for pub but this place is supposed to be nice place for professional lunches. had the chicken flatbread and the chicken
like subway chicken! with so many options around that area won't pick this place for lunch.
Interpolate: Very bad - Very Good
skip it... there are much better options there! the " hot food was hot, and the flavor was only mediocre at most.
(1) indifferent to locals. the kids size pizzas were a billion times worse than a pizza hut. the quality of food was just awful. i wouldn't 't recommend this to significant other for what it is.
(2) this new mexican spot is ok, bordering childish. i went with friends and ordered a carne asada burro... it wasn't off the hook: what made this place great were the chips salsa sucked. yuck! ... (3) wow. JUN you give so much frosting!! we were a groupon special for a cupcake for the princess of chocolate,
and we were pretty stoked. they were _UK and creative. they even suggested we try the coconut... we 'Il definitely be back soon.
(4) went for the first time during recent trip vegas. our server jeff made special recommendations for friends and it was fantastic most of the food was light and fresh... would highly recommend this place! had dinner at republic kitchen tonight for the first time and was very impressed with the service, the decor, the menu, and the food quality... i am going back sunday for their brunch and jazz!
Table 12: Interpolation between four random instances from the Yelp dataset
35


Reviews:
[{"review_id": "tjbZ88Ty3-", "reviewer": "Reviewer_nMiy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a graphical model parameterized by BERT and GPT2 to infer the hidden schema network within the language data. Experiments show that this model is effective in recovering the underlying graphs (to an extend) and effective in language modeling. ", "summary_of_the_review": "At the first sight of this paper, in the introduction and related work, I was surprised by the critics in this paper made towards language models (”lack both compositionally and semantic interpretability”; “difficult to manipulate”) which is simply wrong (at least from my understanding of the field and years as an NLP researcher). Yet when I continue to read the modeling part, the inference method, and the experiment setting, I find these content seem to be OK (from a modeling perspective). I would refrain myself from rejecting a paper because lack of commonsense of the area (which in this case is somehow independent of its tech aspects). Yet I would also refrain myself from accepting a paper that has a wrong understanding of its own filed. \n\nOverall, I think the critics in the paper (abstract, introduction, and related work) requires substantial revision to show, understand and acknowledge the development of language models. Yet apart from that, the modeling seems to be OK but requires stronger results, more than language modeling. Combining these aspects, I rate this paper as a reject.", "strength_and_weaknesses": "**Strength**\n\n**Interesting generative model.** The generative model proposed in this paper, which discovers a latent schema network within the contextualized representations, is interesting and worth attention. \n\n**Experiments validate the effectiveness of the method.** The experiments is performed in a principled way, starting from a synthetic setting then the real language. The language modeling experiments (though the field have moved from language modeling) shows the advantages of the method\n\n**Weakness**\n\n**Biased critics on language models.** Unfortunately, critics made in this paper towards language models is very biased, or even wrong sometimes. Specifically:\n\n- “lack both compositionality” (abstract and introduction): recent developments in language models, especially when language models are large, show extraordinary ability in generating novel and meaningful sentences compositionally (for example: Brown et. al. 2020, Wei et. al. 2022). The general belief of the community is that the content generated by large language models are quite novel and impressive. The authors can also validate this themselves simply by trying to let GPT3 to generate and play with it (registration is free).\n- “lack … semantic interpretability” (abstract and introduction): there is the whole “Bertology” field studying the interpretability, either lexically, syntactically, or semantically (Rogers et. al. 2020, Liu et. al. 2019, Hewitt and Manning 2019). The general belief of the community is that the representations produced by the language models contain very rich semantic interpretations.\n- “semantic content is intrinsically relational” (first paragraph in introduction): this statement is wrong. The semantics of certain words may not need to be relational, for example:\n    - Proper nouns, like name of countries (the United States). They have semantic meaning on their own, and one does not necessarily need their relations to other words to better understand their meaning.\n    - New words that emerges as the society develops. For example, when “electricity” was first discovered. The understanding of these words require grounding to the physical, actual world, but may be hard to understand from their relations to other words.\n- “difficult to manipulate” (abstract and introduction): there are multiple works controlling the generation from language models, for example Miao et. al. 2019, Qin et. al. 2022.\n\n**Limited impact of the result:** the field is now moved from language modeling, as it does not necessarily lead to improvements on the actual tasks. So emphasizing on language modeling results may not be a significant enough contribution. \n\nReferences:\n\nBrown et. al. 2020. Language Models are Few-Shot Learners\n\nWei et. al. 2022. Chain of Thought Prompting Elicits Reasoning in Large Language Models\n\nRogers et. al. 2020. A Primer in BERTology: What we know about how BERT works\n\nLiu et. al. 2019. Linguistic Knowledge and Transferability of Contextual Representations\n\nHewitt and Manning 2019. A Structural Probe for Finding Syntax in Word Representations\n\nMiao et. al. 2019. CGMH: Constrained Sentence Generation by Metropolis-Hastings Sampling\n\nQin et. al. 2022. COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ic1Aj5v7CmM", "reviewer": "Reviewer_5yft", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes building a generative LM (HSN) using a VAE framework to encode sentences into symbols which encode different aspects of language (like topics and sentiments). Using these discrete symbols, the authors claim that we can achieve compositionality of sentence encodings.\n\nThe proof of concept is demonstrated by using HSN to generate a sequence of symbols which correspond to a known ground truth latent graph (nodes corresponding to symbols). A significant overlap is seen between the ground truth graph and the generated graph, implying that the model is successful in uncovering the latent graph.\n\nThe authors then show the performance of HSN while using BERT and GPT2 to train the model for LM and Commonsense Generation task. The performance (perplexity) of HSN is better than other baselines which is impressive. Different topics of text sentences are represented by different \"hot\" symbols in the latent graph.", "summary_of_the_review": "The paper proposes a novel approach for representation learning which allows for compositionality. This approach leads to significant improvements in LM performance over other predominant models like GPT. The paper is mostly thorough in its analysis, but some of the claims regarding compositionality should be better justified using existing mathematical metrics comparing different representations. The paper could however explain qualitative aspects in a better manner. Apart from a few minor things, it is well written.", "strength_and_weaknesses": "**Strengths:**\n1. Authors present a generative model which is better than GPT2 on the Language modelling task.\n2. Presents a scientifically sound proof of concept which shows that the network indeed learns the latent graph representation, and that the results are not simply due to randomness.\n3. Impactful problem addressed: Capturing relational and compositional features of semantic contents means that these representations can prove to be better for use in common NLP tasks which right now rely on contextual encodings (like BERT encodings).\n\n**Weaknesses:**\n1. Authors claim that the contextual sentence embeddings lack semantic interpretability. Authors claim is that since different sets of \"hot\" symbols represent different topics (e.g. Science and Math), HSN provides semantic interpretability. However, can't we say similar things for continuous contextual features? It's highly possible that in a similar way \"Science And Math\" topic has high value for some specific continuous features, whereas \"Health\" has high values for some other set of features. In this way, the significance of the work (of using discrete symbols) is not clear.\n2. The authors make claims about compositionality, but they do not compare the compositionality using existing metrics which have been studied in the literature. https://aclanthology.org/2020.repl4nlp-1.22.pdf. Since compositionality is a major part of the paper, readers would be interested in how the compositionality compares to BERT.\n3. It is not clear why enforcing a semantic understanding for the (hidden representation) symbols significantly helps with the language modelling performance. At the end of the day, the hidden representation is an internal detail. More text is needed on this aspect.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RPUODCj0vg", "reviewer": "Reviewer_tUks", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method for learning discrete sequential representations of sequence data. The model demonstrates surprisingly strong performance in language modeling, and preliminary results on commonsense reasoning seem promising. Analysis of the schema networks also demonstrates interpretability.", "summary_of_the_review": "I recommend acceptance. The paper delivers on the surprising claim of a state-of-the-art sentence VAE with discrete latent representations.\n\nEdit on 12/5: After further discussion with the authors, the perplexity comparisons do not use a valid bound. Without updated and fair numbers, I am changing my score to reject until the evaluation numbers are finalized. I think this is interesting work and encourage the authors to (re-)submit with updated numbers and claims.\n\nEdit on 12/13: After receiving the updated evaluation metrics, I am changing my score back to accept.", "strength_and_weaknesses": "*Strengths*\n\nThe proposed method has the potential to be more interpretable and controllable than single-vector representations and achieves strong performance on ELBO evaluations. The perplexity bound evaluations in this paper are standard in the related sentence VAE literature [1,2].\nBefore reading this paper, my prior belief was that such a latent variable model would not result in perplexity improvements. Those beliefs have now changed.\n\n*Weaknesses*\n\nThe paper is convincing but could be improved with further experiments on controllable text generation (following Li et. al 2020) and further development of the commonsense reasoning experiment.\n\nEdit: I also agree with reviewer 5yft's position that compositionality claims are not supported. Reducing the claim to a state-of-the-art sentence VAE with discrete representations is one solution.\n\n*Questions and comments*\n* To check my understanding, is the following alternative interpretation of the model correct: the model is a (discrete, first-order) hidden Markov model with an autoregressive emission model (GPT-2) and global sparsity distribution over the transition matrix (schema network)?\n* What is the reason for using LSTMs in the commonsense reasoning schema completion model (Appendix E)? BART or GPT2 might give better results.\n\n[1] Li, Chunyuan et. al. “Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space.” EMNLP (2020).\n\n[2] Kim, Yoon et al. “Semi-Amortized Variational Autoencoders.” ICML (2018).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "FDpylNZlgK", "reviewer": "Reviewer_Dats", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a variational autoencoder (VAE) framework for encoding natural language a natural language sentence into a sequence of nodes in a global latent graph. The author claims two major contributions:\n1. The model enhances the interpretability of traditional pretrained language models by interpreting nodes in the latent graph.\n2. The model reaches SOTA performance in language modeling.", "summary_of_the_review": "The work researches VAE in NLP. It is an open problem of major significance. It proposes HSN to auto-encode text sentences into a trajectory in a latent graph. However, the author claims that HSN outperforms traditional LMs in language modeling task, but I consider this claim questionable because concerns about the soundness of evaluation (see weakness 1 for details). Therefore, I propose to reject this paper unless my concerns are addressed.", "strength_and_weaknesses": "## Strengths\n\n1. This work studies VAE for natural language. This is an important open problem in NLP and VAE community.\n\n2. They conduct experiments using modern pretrained language models such as BERT and GPT-2, and compared with these strong baselines in experiments.\n\n3. This work presents an interesting analysis on the structure of the latent graph (Figure 3 and Table 3), showing possibility of interpreting latent representations of big language models.\n\n## Weaknesses\n\n1. There is a major soundness problem in the LM evaluation. According to Section 3 and Figure 1, the proposed method first encodes the *full* input sentence using a modified Transformer encoder and get $\\mathbf{h}^\\text{enc}$, which is used to compute $\\mathbf{q}$ and then the node sequence $\\mathbf{e}$. In the decoding stage, the modified Transformer decoder model has access to $\\mathbf{e}$ via attention. As a result, although indirectly, the **decoder has access to the full input sentence**. An ordinary Transformer encoder-decoder model having access to the full input sentence can easily achieve a perplexity as low as 1 if trained properly. Therefore, the empirical result that the proposed method outperforms GPT in language modeling makes little sense in showing the strength of the proposed method.\n\n2. The formulation of the algorithm in training and generation is still not very clear. It would be better if there are some pseudo-codes to explain the method step by step.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject