Conversational Semantic Parsing using Dynamic Context Graphs
Authors: Parag Jain
Mirella Lapata
Venue: EMNLP 2023 Main (2023)
Keywords: semantic parsing
SPARQL
Knowledge Graphs
Conversational Semantic Parsing
TL;DR: We propose dynamic graphs based method for conversational semantic parsing over general purpose large-scale knowledge graphs.

Abstract: In this paper we consider the task of conversational semantic parsing over general purpose knowledge graphs (KGs) with millions of entities, and thousands of relation-types. We focus on models which are capable of interactively mapping user utterances into executable logical forms (e.g., Sparql) in the context of the conversational history. Our key idea is to represent information about an utterance and its context via a subgraph which is created dynamically, i.e., the number of nodes varies per utterance.  Rather than treating the subgraph as a sequence, we exploit its underlying structure and encode it with a graph neural network which further allows us to represent a large number of (unseen) nodes. Experimental results show that dynamic context modeling is superior to static approaches, delivering performance improvements across the board (i.e., for simple and complex questions). Our results further confirm that modeling the structure of context is better at processing discourse information, (i.e., at handling ellipsis and resolving coreference) and longer interactions.

Conversational Semantic Parsing using Dynamic Context Graphs
Parag Jain
Mirella Lapata
Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB parag. jain@ed. ac. uk mlap@inf.. ed. ac uk
Abstract
Who starred in Mathias Kneissl ? SELECT WHERE { wd:03298576 wdt:P161 ?x
In this paper we consider the task of conversa- tional semantic parsing over general purpose knowledge graphs (KGs) with millions of enti- ties, and thousands of relation-types. We focus on models which are capable of interactively mapping user utterances into executable logical forms (e.g.. SPARQL) in the context of the con- versational history. Our key idea to represent information about an utterance and its context via subgraph which is created dynamically, Le., the number of nodes varies per utterance. Rather than treating the subgraph as sequence, we exploit its underlying structure and encode it with a graph neural network which further allows to represent large number of (un- seen) nodes. Experimental results show that dynamic context modeling superior to static approaches, delivering performance improve- ments across the board (i.e., for simple and complex questions). Our results further con- firm that modeling the structure of context is better at processing discourse information, (i.c., handling ellipsis and resolving coreference) and longer interactions.
:d1 wdl: Q582855229555 .}
Rainer Werner Fassbinder, Volker Schlöndorff, Hanna Schygulla
2. Who was the director of that work of art
SELECT ?x WHERE { wdl: (3298529 P57 ?x
?x
wdt:P31 wd:Q502895
Reinhard Hauff
3. Does Dubashi have that person actor wdl: Q24887888
ASK wd: wd:076625 wdt:P161 No
4. Which works of art are Rainer Werner Fassbinder Laura Esquivel screenwriter of
SELECT ?x WHERE ?x wdt:P58 wd:044426
?x
wdt:P31 wd:Q838948
UNION ?x wd8: wd:0230586
?x wdt:P31
wd:Q838948 ]
The American Soldier, Lili Marleen, Love Is Colder
Than Death
Q3298576: Mathias Kneissl, Q76825: Reinhard Hauff, Q24807818: Dubashi, Q44426: Rainer Werner Fassbinder Q238586: Laura Esquivel, Q838948: work of art, Q502895: common name, P161: cast member, P31: instance of, P57: director, P58: screenwriter
Figure 1: Example interaction from SPICE dataset (Perez-Beltrachini et al., 2023) with utterances, corre- sponding SPARQI. queries, and answers returned after executing the queries on the Wikidata graph engine. The bottom block shows the KG elements (i.e., graph nodes) involved in this interaction.
1 Introduction
General purpose knowledge graphs (KG), like Wikidata (Vrandečić and Krötzsch, 2014) struc- ture information in semantic network of entities, attributes, and relationships, allowing machines to tap into vast knowledge base of facts. Knowl- edge base question answering (KBQA) is the task retrieving answers from a KG, given natural lan- guage questions. A popular approach to KBQA (see Gu et al. 2022 and the references therein) is based on semantic parsing which maps questions to logical form queries (e.g., in SPARQL) that return answer once executed against the KG.
semantic parsing. Specifically, our interest lies in building systems capable of interactively mapping user utterances to executable logical forms the contexi of previous utterances. Figure I shows an example of a user-system interaction, taken from SPICE (Perez-Beltrachin et al., 2023), a recently released conversational semantic parsing dataset. Each interaction consists of series of utterances that form a coherent discourse and are translated to executable semantic parses (in this case SPARQL queries). Interpreting each utterance, and mapping to the correct parse needs be situated in a particular context as the exchange proceeds. To an- swer the question in utterance 2, the system needs
Existing work (e.g.. Bogin al. 2019; Ravis- hankar al. 2021; Yin et al. 2021) has mostly fo- cused on answering questions in isolation, whereas we consider the less studied task of conversational

---

recall that Mathias Kneissl is still the subject of the conversation, however, the user is no longer interested in who starred in the film but in who directed it. It is also natural for users to omit previ- ously mentioned information (e.g., through ellipsis or coreference), which would have to be resolved to obtain complete semantic parse.
through a pretrained model (Devlin et al., 2019). In addition, we introduce context context-dependent type link- ing, based on the entity and its surrounding context which further helps with type disambiguation.
Experimental evaluation on the SPICE dataset PPerez-Beltracie et al., 2023) demonstrates that modeling context dynamically is superior to static approaches, improving performance across the board (i.e., for simple and complex questions re- quiring comparative or quantitative reasoning). Our results further confirm that modeling the structure of context is better at processing discourse informa- tion, (i.e., at handling ellipsis and resolving coref- erence) and longer interactions with multiple turns.
In addition challenges arising from process- ing contextual information, the semantic parsing task itself involves linking entities, types, and pred- icates to elements in the KG (e.g., Mathias Kneissl to Q3298576) whose topology is often complex with a large number of nodes. Moreover, unlike relational databases, the schema of an entity is not static but dynamically instantiated (Gu et al., 2022). For example, the entity type person can have hun- dreds of relations but only a fraction of these will be relevant for a specific utterance. Therefore, to gen- erate faithful queries, we cannot rely on memoriza- tion and should instead make use of local schema instantiation. In general, narrowing down the set of entities and relations is critical to parsing utterances requiring complex reasoning (i.e., where numerical and logical operators apply over sets of entities).
2 Related Work
Previous work on semantic parsing for KBQA (Gu et al., 2022) has focused on mapping stand-alone utterances to logical form queries. Various ap- proaches have been proposed to this effect which broadly follow three modeling paradigms. Rank- ing methods first enumerate candidate queries from the KB and then select the query most similar to the utterance the semantic parse (Ravishankar et al., 2021; Hu al., 2021; Bhutani et al., 2019). Coarse-to-fine methods (Dong and Lapata, 2018; Ding et al.. 2019; Ravishankar et al., 2021) perform semantic parsing in two stages, by first predicting a query sketch, and then filling in missing details. Finally, generation methods (Yin et al., 2021) first rank candidate parses and then predict the final parse by conditioning on the utterance and best retrieved logical forms.
Existing work (Perez- Beltrachinn et al., 2023) handles the aforementioned challenges by adopting various simplifications and shortcuts. For instance, since it is not feasible to encode the entire KG, only subgraph relevant to the current utterance is extracted and subsequently linearized and treated sequence. Entity type information that is not directly accessible via neighboring subgraphs is obtained through a global lookup (essentially a re- verse index of all types in the KG). This solution is computationally expensive, as the lookup is per- formed practically for every user utterance, and does not scale well (the index would have to be recreated every time the KG changed).
Our task most related to conversational text- to-SQL parsing, as manifested in datasets like SParC (Yu et al., 2019b), CoSQL (Yu et al., 2019a), and ATIS (Dahl et al.. 1994; Suhr et al., 2018). SParC and CoSQL. cover multiple domains and include multi-turn user and system interactions. These datasets are challenging in requiring gener- alization to unseen databases, but the conversation length is fairly short and the databases relatively small-scale. ATIS contains utterances paired with SQL queries pertaining to a US flight booking task; it exemplifies several long-range discourse phenom- ena (Jain and Lapata, 1), however, it covers a single domain with simple database schema.
In this paper we propose modeling approach conversational semantic parsing which relies on dynamic context graphs. Our key idea to represent information about an utterance and its context through a dynamically generated subgraph, wherein the number of nodes varies for each ut- terance. Moreover, rather than treating the sub- graph sequence, we exploit its underlying structure and encode it with graph neural network (Scarselli et al., 2009; Gori et al., 2005). To im- prove generalization, we learn implicit node embed- dings by aggregating information from neighboring nodes whose embeddings are in turned initialized
Graph-based methods have been previously em- ployed semantic parsing primarily to encode the database schema, so as to enable the parser to glob- ally reason about the structure of the output query

---

(Bogin et al., 2019). Other work (Hui et al., 2022) uses relational graph networks to jointly represent the database schema and syntactic dependency in- formation from the questions. In the context of con- versational semantic parsing, Cai and Wan (2020) use a graph encoder to model how the elements of the database schema interact with information in preceding context. Along similar lines, Hui et al. (2021), use graph neural network with dynamic memory decay mechanism to model the interac- tion of the database schema and the utterances in context as the conversation proceeds. All these ap- proaches encode the schema of relational databases which are significantly smaller in size (e.g.. num- ber of entities and types of relations) compared to large-scale KGs, where encoding the entire graph in memory is not feasible.
(e.g., Q3298576). Answer A, at time is the result of executing Y, against G. Given the interaction history I[: - 1] at turn l and current utterance X,, our goal is to generate Y, This involves under- standing X in the context of X 1. A, and G, and learning to generate Y, based on encoded con- textual information.
Model
Our modeling approach combines three compo- nents. We first ground named entities in the user utterance to KG entities, and use these linked enti- ties to extract a subgraph that functions as context (Section 4. 4.1). The second component is responsi- ble for type linking in the context of current and previously mentioned named entities (Section 4.2). And finally, our semantic parser learns to map user utterances into SPARQL queries (Section 4.3).
Closest to our work are methods which cast conversational KGQA as a semantic parsing task (Kacupaj et al., 2021; Marion et al., 2021). grammars that
4.1 Entity Grounding and Disambiguation
These approaches build are not directly executable to a KG engine. Further- more, they assume the KG can be fully encoded in memory which may not be feasible in real-world settings.
We are interested in grounding user utterance X, to graph G. Since encoding the entire KG is not feasible, we extract a subgraph from G' which is relevant to the current turn. To achieve this, we first perform named entity recognition with an off-the- shelf NER system, in our experiments we use Al- lenNLP (Gardner et al., 2018). We perform named entity linking through efficient string matching' (Aho and Corasick, 1975) unlike Perez-Beltrachni et al. (2023) who deploy an ElasticSearch server for querying inverted index.
et al. (2023) develop a parser which executable with a real KG engine (c.g., Blazegraph) but simplify the task by consid- ering only limited conversation context.
3 Problem Formulation
Given a general purpose knowledge graph, such Wikidata, our task is to map user utterances to formal executable queries, SPARQL in our case. We further assume an interactive setting where users converse with the system natural language and the system responds while taking into account what has already been said (see Figure 1). The system's 's response is obtained upon executing the query against graph query engine.
A string can be ambiguous, i.e., link to multiple entities. For example, Rainer Werner Fassbinder can be linked to filmmaker (Q44426) and movie (Q33561976) To deal with ambiguity and to in- et al. (2023) do not
crease recall, ever commit to single entity but instead include the top- K matching ones; however, this introduces noise and increased computational cost. Instead, disambiguate entities based on their popularity in the training set (Shen al., 2015) and compare the two approaches in Section 6.
Let G denote the underlying KG and I a sin- gle interaction. I consists of a sequence of turns where each turn is represented by (X,, A,, Yi) de- noting an utterance-answer--uuee triple at time f (see blocks in Figure 1). A user utterance X; is a se- quence of tokens T2,, 7 xxa)) where Xg] is the length of the sequence and each ,,, E [1, [X,]] is a natural language token. Query string Y: is sequence of tokens (y1, ... v where |Yr] is the length of the sequence and each ys, i E [1, [V]]] is a either a token from the SPARQL syntax vocab- ulary (e.g., SELECT, WHERE) or a KG element e G
Following BeettrcBllta et al. (2023), for each identified KG entity e, we extract triples r, and (s, where s and denote the subject or object of relation r. For instance, en- tity Dubashi would have triple (Dubashi, country of origin, India). Subjects and objects are further mapped to their types in place of actual entities
'Our implementation is based on pyahocorasick https:
//pypi

---

Previous graph context state St-1
Current graph context state St
Merged graph context state
SELECT Px WHERE wd: 23298576 wdt:
vdt: wd ud:0552895
BERT
GATv2
Transformer Decoder
Extract
Extract Sub-graph
BERT
Sub-graph
Who starred in Mathias Kneissl [SEP] Rainer Werner Fassbinder.., (SEP) Who was the director of that work of art
Figure 2: Model architecture. The previous and current utterance are concatenated and their subgraphs are merged and encoded in graph neural network. The subgraphs represent the entity neighborhood and type linking.
(i.e., (e, r2 Otype) and (Stype: r, e)). In our exam- ple, the triple for Dubashi then becomes (Dubashi, country of origin, country), where country is type Q6256. We denote the set of typed triples Grml
A--1. and subgraph G.-1. We use G2 to repre- sent merged context subgraphs G1 and G:-1 such that G. Ge U G.-1- We encode the context subgraph G, with graph neural network (GNN, Scarselli al. 2009; Gori et 2005) and user utter- ances and their discourse context with BERT (De- vlin et al., 2019). Our decoder is a transformer network (Vaswani et al., 2017) that conditions on the user utterance encoding and the corresponding graph representations.
4.2 Context-Dependent Type Linking
Entities in SPARQI. queries have types, for example, in Figure 1, KG element Q502895 is placeholder for the type "common name". Type instances are often present the one-hop entity neighbor- hood G but can also be more hops away. Perez- Beltrachini et al. (2023) index all KG types and perform a global lookup which is computationally expensive, and solely applicable to the KG they are working with. Instead, we perform type linking based on the entities mentioned in the current con- text. We expand the grounded entities to extract triples with type information.? 2 Since considering multi-hop neighborhoods would lead to extremely large subgraphs and would not be memory efficient, we prune these based on their string overlap with the user utterance, significantly reducing the num- ber of triples. The pruned graph Geaee is merged with the previously obtained entity graph Gont, such that, G G U GAMe..
Utterance Encoder We use BERT (Devlin et al., 2019) to represent the concatenation of previous utterance X,-1, previous answer At-1- and current utterance X, (see Figure 2). To distinguish between current and past context we use the [SEP] token. More formally, let X, = [CLS] X [SEP] A,-1 [SEP) X, denote the input to BERT, where X2 = (œ1, 22 ... .Ixx)) A, = (01,a2...4A.... AA-.)) and X, = (I), x2 2Ix,_1) are sequences of natural lan- guage tokens. We obtain latent representations Z, as Z, BERT(X;).
Graph Encoder We represent the KG sub- graph Ge at time as directed graph G (V,E) (hereafter, we simplify notation and drop time t), where V
4.3 Dynamic Context Graph Model
{1i, U2...., ux} are nodes such that v; E {entitics, relations, types) and E x V. Each node vs consists of a se- quence of natural language tokens, such that Ys (Ua U22.... ... Rijea)) Our KG has a large num- ber of distinct nodes, but we cannot possibly attest all of them during training. To handle unseen nodes at test time, obtain generic node representa-
Figure shows the overall architecture of our dy- namic context graph model (which we abbrevi- ate to DCG). DCG takes as input a tuple form (C,, X1, G.), where X₃ is a user utterance at time t and Gt is the corresponding subgraph. Ce de- notes the previous context information that includes the previous utterance X,-1, the previous answer
Foor entity ent we query select 2r1 ?nl ?t1 ?r2 ?n2 ?t2 where { wd:ent 2r1 7nl 2n1 P31 ?t1 OPTIONAL [?nl ?r2 2n2 2n2 P31 ?t2) ).
We emplay BERT for fair comparison with prior work. Nonetheless, our model does not have any inherent restrictions that would prevent the use of other pretrained models.

---

tion he for node vs, where NC (e:) In other words, we compute encoding hę by tak- ing the average of the individual token encodings obtained from BERT. We do not create a separate embedding matrix but directly update the BERT representations during learning, which allows us to scale to large number of (unseen) nodes.
decoder state s; at the 8th generation step. the prob- ability of generating yi is calculated as:
p(u yes. H 8,) = Po yess H;, + ps(sy Veir 8:) (4)
where Pg and ps are the probability of generat- ing graph node and syntax symbol, respectively. such that
A graph neural network (GNN) learns node rep- resentations by aggregating information from its neighboring nodes. Each GNN layer I, takes as input node representations {h
We calculate Ps = softmax(W1ss) W1 E R//d and [Va] is the SPARQI. syntax vo- cabulary size. We calculate Pş using node embed- dings H1, PG softmax(H(ss.)
E n]
and edges E. The output of each layer is an up-
dated set of node representations {hξ i E n.).
We use Graph Attention Network v2 (GATv2, Brody et al. 2022) for updating node representa- tions which replaces the static attention mechanism of GAT (Velickovic et al., 2018) with dynamic and more expressive variant. Let {uy E V (j.a) i) E E} denote the neighbors of node e and a;j the attention score between node hy and hy. We calculate attention as a single-layer feedforward neural network, parametrized by weight vector and weight matrix W:
Training Our model is trained end-to-end by op- timizing the cross-entropy loss. Given training in- stance (Ct, X1, Yt. G1), where Yú is sequence of gold output tokens (y1, M2. ................................ ./yaa) we minimize the token-level cross-entropy as:
(())) = -logp(n Xt, Gr Ct)
(5)
where i denotes the predicted output token at de- coder step i.
Experimental Setup
exp( 1)
Oij = ELEN ( (1 n)
(1)
Dataset We performed experiments on SPICE (Perez-Beltrachini al., 2023), a recently released large-scale dataset for conversational semantic parsing built on top of the CSQA benchmark (Saha et al., 2018). SIPICE consists of user-system in- teractions where natural language questions are paired with SPARQL parses and answers provided by the system correspond SPARQL execution results (see Figure 1). We present summary statis- tics of the dataset in Table As can be seen, it contains a large number of training instances, the conversations are relatively long (the average turn length is 9.5), and the underlying KG is sizeable (12.8M entities). SPICE has simple factual ques- tions but also more complicated ones requiring rea- soning sets of triples; it also exemplifies vari- ous riscourse-related phenomena such corefer- ence and ellipsis. We provide examples of the types of questions attested in SPICE in Appendix C.
The scoring function ab is computed as follows:
a" LeakyReL.U(W [M-- 1)
(2)
where ' represents transposition and is the con- catenation operation. Attention coefficients corre- sponding each node are then used to compute linear combination of the features corresponding neighhoring nodes as:
1/4 = Σ LasWhn JENS
Decoder Our decoder is a transformer net- work (Vaswani et al., 2017). Let H! = (Nier Mi...,. 14,)) denote the sequence of node representations from the last layer of the graph net- work (recall t here represents an interaction turn). Our decoder models the probability of generating SPARQL parse conditioned on the graph and input context representations, i.c., p(Y, HI Zz). Gener- ating the SPARQI. parse requires generating syntax symbols (such as SELECT, WHERE) and KG elc- ments (i.e., entities, types, and relations). Given
Metrics Following previous work (Perez-Beltrachin et al., 2023), we report match accuracy and Fl (or accuracy depending on question type). Exact match is the percentage of predicted SPARQL queries that string match with the corresponding gold SPARQL. Fl (or
Evaluation

---

instances
197K
ing with an ElasticSearch? inverted index. As mentioned earlier, they do not explicitly perform named entity disambiguation (they consider the K = 5 best matching entities and their neighbor- hood graphs as part of the vocabulary) and use a global lookup for type linking. As the size of the linearized subgraph often exceeds BERT's s maxi- mum number of input tokens (which is 512), they adopt workaround where the graph is chunked into several subsequences, and encoded separately. We refer to their Semantic Parser as BertSP where GC is shorthand for Global Lookup.
entities
12.8M
relations
2,738
types
3,064
Avg. turn length
9.5
Avg. entities per conversation
7.6
Avg. types per conversations
6.5
Avg. neighborhood per turn
181.4 triples
Table 1: Statistics of the SPICE dataset.
answer accuracy) is calculated between execution results of predicted queries and gold queries. For Verification queries and queries involving Quantitative and Comparative Reasoning, we calculate execution answer accuracy. For other types of questions, Fl scores are calculated by treating the results
In addition to our full dynamic context graph model which performs Context-dependent Type Linking (DCGgz), we also build simpler variant (DCG) which only relies the entity neighbor- hood subgraph for type information. Moreover, we create two variants of our model, one which disam- biguates entities, and another one which does not (similar to Perez- Beltrachini et al. 2023).
Model Configuration Our model is imple- mented using PyTorch (Paszke et al., 2019) and trained with the AdamW (Loshchilov and Hutter, 2019) optimizer: Model selection was based on ex- act match accuracy on the validation set. We used two decoder layers and two GATv2 layers for all experiments. We used HuggingFace'ss pretrained BERT embeddings (Wolf et al., 2020), specifically the uncased base version. Our GATv2 implemen- tation is based on Py Torch Geometric (Fey and Lenssen, 2019) with two attentions heads. We use adjacency matrices stacking as method of cre- ating mini-batches for our GNN across different examples. We identify named entities using the AllenNLP named entity recognition (NER) system (Gardner et al., 2018). Our execution results are based on the Wikidata subgraph provided by Perez- Beltrachini et al. (2023). Our SPARQL server is deployed using Blazegraph. 6 See Appendix B for more implementation details.
Results
In this section, evaluate the performance of our semantic parser on the SPICE test set. We report results on individual question types and overall. We also analyze our system's ability to handle different discourse phenomena like ellipsis and coreference as well as interactions of varying length.
The Effect of Dynamic Context Table 2 sum- marizes our results. We first concentrate on model variants without entity disambiguation for fair comparison with Frrez-Beltrachin et al. (2023).
We compare DCGGL, a version of our model which adopts a global lookup for type liking simi- lar to BertSP and differs only in how contextual information is encoded. As we can see, our graph- based model performs better, reaching an Fl score of 72.3% compared to 59% obtained by BertSP which is limited by the way it encodes contextual in- formation. Recall, that BertSP linearizes the sub- graph context, splits into subsequences and feeds it to the model chunk-by-chunk. Our model alleviates this problem by efficiently encoding the KG infor- mation with a graph neural network, preserving dependencies captured in its structure. As result, DCGG2 performs better on most question types, including simple questions, and reasoning-style questions. We further compare DCGgc against variant which uses context-dependent type linking
As described in Section 4.3, each utter- ance, our model encodes the previous subgraphs. Larger context is informative but can also introduce noise. We treat t as a hyperparameter and optimize its value on the development set. We report results with t = 5 (see Appendix A for details).
Comparison Models We compare against the semantic parser of Perez-Beltrachiní et al. (2023).
Their model is based on BERT (Devlin et al., 2019), it relies on AllenNLP (Gardner et al., 2018) for named entity recognition, and performs entity link-
*Our code can be downloaded from https://github. com/ homm/parajain/dyaaicccmmm hhttps://blazegraaphh com/
hhttps://www clastic.co/

---

Without Disambiguation
With Disambiguation
DCGGs
DCGge
BertSP GE
DCGcc
DCG
Question Type
F1
EM
EM
F1
EM
EM
Clarification
78.60
73.63
80.42
69.47
83.91
76.58
82.01
74.82
82.03
72.10
Logical Reasoning
64.12
49.51
51.14
31.54
22.74
28.61
93.95
79.52
93.33
78.19
Quantitative Reasoning
55.66
26.29
93.25
76.88
76.20
59.01
59.83
31.17
56.66
28.66
Comparative Reasoning
76.06
35.59
80.59
47.28
69.56
39.37
90.91
62.46
90.09
61.11
Simple Question (Coref)
86.36
72.03
84.92
67.15
76,51
58.83
88.49
79.90
87.41
79.18
Simple Question (Direct)
87.29
71.10
83,24
65.89
71.43
58.71
88.27
62.25
85,60
61.44
Simple Question (Ellipsis)
65.22
56.08
52.84
47.48
58,14
50.90
79.08
83.87
84.35
82,45
AC
EM
AC
FM
AC
FM
AC
EM
AC
EM
Veriicationn (Boolean)
78.30
36.97
69.07
21.00
37.16
24.90
87.41
66.32
86.75
63.66
Quantitative Reasoning (Count)
61.10
56.94
66.59
62.70
50.86
48.44
75.20
70.84
72.96
69.02
Comparative Reasoning (Count)
42.79
30.40
60.91
44.68
43.48
40.67
67.70
57.34
66.76
56.60
Overall
69.55
50.85
72.30
53.41
59.00
48.60
81.28
66.85
80.59
65.24
Table 2: Results on SPICE dataset (test set). Be Beltraciel al., 2023) uses NER based on AllenNLP) and global look-up (subscript çc) for type linking. DCGez uses context-dependent type linking (subscript ec) and AllenNLP. DCG no type linking. measure Fl, Accuracy (AC), and Exact Match (EM).
instead (DCGcc). We find that mentext-depeneent type linking is slightly worse than global lookup which is expected given that it does not have access the full list of KG types.
BerSS (-D)
DCGGe
Exaee Match Accuracy (%)
DCGez (-D)
DCGGED (+D)
In general, we observe that results with exact match (EM) are lower than Fl or Accuracy. EM is stricter metrie, it does not allow for any deviation from the goldstandard SPARQI.. However, it is possible for two queries have different syntax but equivalent meaning, and for partially vell-formed queries to evaluate to partially accurate results. In contrast to EM, and Accuracy give partial credit and thus obtain higher scores.
0.6
0.4
10
16
Turn Position
Figure Exact Match Accuracy averaged across ques- tion types at different turn positions. +/-D denotes the presence/absence entity disambiguation.
The Effect Entity Disambiguation We now present results with a variant of our model which operates over disambiguated entities (compare sec- ond and fifth blocks in Table 2, with heading DCGez). We observe that disambiguation has significant effect model performance, leading Fl increase of more than 1 11%. We further assess the utility of context-dependent linking by comparing DCGç2 to variant which does not have access to the type graph GAMe neither during training nor during evaluation (see column DCG, With Disambiguation). This type-deficient model performs overall worse both in terms of Fl and EM, but is still superior to BertSP even though the latter has access to more information via the top-K entity lookup and global type linking. This points to the importance of encoding context in a targeted manner rather than brute force. In Ap- pendix A we discuss the effect of context length on the availability of type information.
The Effect of Conversation Length We next examine the benefits of modeling context dynam- ically. Ideally, a model should produce an ac- curate semantic parse matter the conversation length. Figure 3 plots exact match accuracy (aver- aged across question types) against different turn positions. general, we observe that utterances occurring later the conversation are more diffi- cult to parse. As the dialogue progresses, subse- quent turns become more challenging for the model which is expected to leverage the common ground established so far. This involves maintaining the subgraph context based the conversation history in addition to handling linguistic phenomena such as coreference and ellipsis. Overall, DCGcz (with and without entity disambiguation) is superior BertSP and the gap between the two models is more pronounced for later turns.

---

Without Disambiguation
With Disanbiguation
The Nature of Parsing Errors Overall, we find that our model able to predict syntactically valid
DCGes
DCGg2
Be
DCGec
DCG
Coref= 58,25
51.10
49.39
74.23
72.22
SPARQL queries. Errors are mostly due to mis-
Coref<
23.52
12.42
00.00
33.64
32.40
Ellipsis
39.30
39,90
26.39
62.26
61.10
interpretations of the question's intent given the graph context and previous questions or missing information. Our model also has difficulty parsing Clarification and Quantitative Reasoning questions. For Clarification questions, it is not able to select the right entity after clarification. For example, in the following conversation:
MEntities
43.71
53.46
41.64
61.59
58.90
Table 3: Average Exact Match accuracy. Coref=-1 are utterances with referring expressions resolved in the previous turn. Coref< - utierances with referring expressions resolved in the wider discourse context, be- yond the previous turn. MEntities abbreviates multiple entities and refers utterances with plural mentions.
Answer:
Van Winkle, Arthur Boreman, William E. Stevenson
Utterance:
Which language is that person capable of writing
Modeling Discourse Phenomena Discourse phenomena, such as ellipsis and coreference, are prevalent in conversations. Ellipsis refers to gram- matical omissions from an utterance that can be recovered from context. In the interaction:
Clarification: Did you mean Arthur Boreman
Answer: No, meant Peter G. Van Winkle, Could
you tell me the answer that?
selects Arthur I. Boreman (Q709961) instead of Peter G. Van Winkle (Q1 404201) leading to an in- correct SQL parse. In this case, the broader context overrides useful information in the immediately preceding turn. Determining relevant context based on specific question intents would be helpful, how- ever, we leave this to future work.
(Q1)
What does Andrei Neagoe do for living?
(Q2) And how about Wilhelm Dictrichson?
the phrase do for a living is elided from (Q2) but can be understood the context (QI). Corefer- ence on the other hand, occurs between utterances that refer to the same entity. For example, between utterances 2 and 3 in Figure 1.
Failures in type linking are a major cause of errors for Quantitative Reasoning questions which typically have no or very limited context (e.g., "Which railway stations were managed by exactly social group ?"). However, our model relies on the availability of types in the entity neigh- borhood, as performs type linking in context dependent manner. We observe that it becomes better at parsing such questions when given access to all KG types (see Table 2, DCGGc vs. DCGçc).
In Table 3, using exact match, we assess how different models handle ellipsis and coreference across question types. Coref=-1 refers to cases where coreference can be resolved in the immediate context, i.c., the previous turn. Coref< - involves utterances that require access to wider conversation context, beyond the previous turn. In the setting that does not disambiguate entities, we observe that models which exploit discourse context (variants DCGGc and DCGç2) are better resolving co- referring and elliptical expressions compared to BertSPcc We also see that entity disambiguation is very helpful, leading to substantial improvements for DCGeg across discourse-relater phenomena.
Conclusions
In this paper, we present a semantic parser for KBQA which interactively maps user utterances into executable logical forms, in the context of pre- vious utterances. Our model represents information about utterances and their context as KG subgraphs which are created dynamically and encoded us- ing graph neural network. We further propose a context-dependent approach to type linking which is efficient and scalable.
Similar to Beltr-bellaii et al. (2023), we also evaluate model performance on utterances with plural mentions; these are typically linked to multiple entities which the semantic parser must enumerate in order to build correct parse (MEn- tities in Table 3). DCGç2 with disambiguation is overall best, while DCG (without type linking) is worse. This is not surprising, utterances with mul- tiple entities generally have complex parses, with multiple sub-queries and entity types, which DCG does not have access to.
Our experiments reveal that better modeling of contextual information improves performance, in terms of entity and type linking, resolving coref- erence and ellipsis, and keeping track of the in- teraction history as the conversation evolves. Di- rections for future work are many and varied.

---

experiments, we use an off-the shelf NER system, however, jointly learning a semantic parser and an entity linker would be mutually beneficial, avoiding error propagation. Given that it is prohibitive to en- code the entire KG, we encode relevant subgraphs on the fly. We could further explicitly model the relationship between KG entities and question to- kens which previous work has shown is promising (Wang et al., 2020). Finally, it would be interest- ing to adapt our model so as to handle non-i.i.d generalization settings.
to-SQL parsing. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-NCCNL)), pages 3659-3664, Hong Kong, China. Association for Computational Linguistics.
Shaked Brody, Uri Alon, and Eran Yahav. 2022. How attentive graph attention networks? In The Tenth International Conference on Learning Representa- rioms, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Yitao Cai and Xiaojun Wan. 2020. IGSQL: Database schema interaction graph based neural model for context-dependent text-to-SQI. generation. Pro- ceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP). pages 6903-6912, Online, Association for Computa- tional Linguistics.
8 Limitations
Our model relies pre-trained NER module for entity linking. As this module is trained and evaluated on specific datasets, its performance may not generalize on unseen domains within Wikidata. Moreover, we did not explicitly consider relations. We assume that the correct information will be available which may not always be the case, We focus on encoding KG structural information and pass the learning of the interactions between the KG and the linguistic utterances to the decoder. As shown in previous work (Zhang al., 2022), effec- tively combining KG information with a language model can be mutually beneficial in the context of question answering. However, it requires an exten- sive study in itself to determine the task specific parametrization (Wang et al., 2022).
Deborah A. Dahl, Madeleine Bates, Michael Brown, William Fisher, Kate Suith, David Pallett. Christine Pao, Alexander Rudnicky, and Elizabeth Shriberg. 1994, Expanding the scope of the ATIS task: The ATIS-3 corpus. In Human Language Tech- nology: Proceedings of Workshop held Plains- boro, New Jersey, March 8-11, 1994.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Acknowledgements
Jiwei Ding, Wei Hu, Qixin Xu, and Yuzhong Qu. 2019. Leveraging frequent query substructures to generate formal queries for complex question answering. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLA) pages 261 2614 2622, Hong Kong, China. Association for Computa- tional Linguistics.
This work supported in part by Huawei and the UKRI Centre for Doctoral Training in Natural Lan- guage Processing (grant EP/S022481/I) Lapata gratefully acknowledges the support the UK Engi- neering and Physical Sciences Research Council (grant EP/W002876/1).
Li Dong and Mirella Lapata. 2018. Coarse-to-fine de- coding for neural semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 731-742 Melbourne, Australia. Association for Computational Linguistics.
References
Alfred V. Aho and Margaret J. Corasick. 1975, Effi- cient string matching: An aid to bibliographic search. Commun. ACM, 33363333
Nikita Bhutani, Xinyi Zheng, and H V Jagadish. 2019. Learning to answer complex questions over knowl- edge bases with query composition. In Proceedings of the 28th ACM International Conference on In- formation and Knowledge Management, CIKM '19, page 739-748, New York, NY, USA. Association for Computing Machinery.
Matthias Fey and Jan E. Lenssen. 2019. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F, Liu, Matthew Pe- ters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: deep semantic natural language pro- cessing platform. In Proceedings of Workshop for
Ben Bogin, Matt Gardner, and Jonathan Berant. 2019. Global reasoning over database structures for text-

---

NLP Open Source Software (NLP-OSS), pages 1-6. Melbourne, Australia. Association for Computational Linguistics.
Antiga, Alban Desmaison, Andreas Kopf. Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, library. In Advances in Neural Information Process- ing Systems, volume 32, pages 8026-8037. Curran Associates, Inc.
M. Gori, G. Monfardini, and F. Scarselli. 2005. A new model for learning in graph domains. In Proceedings. 2005 IEÉE International Joint Conference on Neural Networks, 2005., volume 2, pages 729-734 vol. 2.
deep learning
Yu Gu, Vardaan Pahuja, Gong Cheng, and Yu Su. 2022. Knowledge base question answering: A semantic parsing perspective.
Laura Perez- Beltrachtanc, Parag Jain, Emilio Monti, and Mirella Lapata. 2023. Semantic parsing for conver- sational question answering over knowledge graphs. In Proceedings of the 17th Conference of the Euro- pean Chapter of the Association for Computational Linguisties, pages 2499-2514, Dubrovnik, Croatia. Association for Computational Linguistics.
Xixin Hu, Yiheng Shu, Xiang Huang, and Yuzhong Qu. 2021. Edg-based question decomposition for complex question answering over knowledge bases. In The Semantic Web- ISWC 2021, pages 128-145,
Cham. Springer International Publishing,
Srinivas Ravishankar, June Thai. Ibrahim Abdelaziz. Nandana Mihindukulasooriya,, Tahira Nascem, Pa- van Kapanipathi, Gactano Rossillo, and Achille Fok- oue. 2021. A two-stage approach towards generaliza- tion knowledge base question answering. CoRR, abs/21 11 1.05225.
Binyuan Hui, Ruiying Geng, Qiyu Ren, Binhua Li, Yongbin Li. Jian Sun, Fei Huang, Luo Si. Pengfei Zhu, and Xiaodan Zhu. 2021. Dynamic hy- brid relation exploration network for cross-domain context-dependent semantic parsing. Proceedings of the AAAI Conference on Artificial Intelligence, 35(14);:3 13116-33144
Amrita Saha, Vardaan Pahuja, Mitesh Khapra, Karthik Sankaranarayanan, and Sarath Chandar. 2018, Com- plex sequential question answering: Towards learn- ing to converse over linked question answer pains with a knowledge graph. In Proceedings of the Thirty- Second AAAI Conference on Artificial Intelligence (AAAI-18), pages 705-713, New Orleans, Louisiana, USA. AAAI Press.
Binyuan Hui, Ruiying Geng, Lihan Wang, Bowen Qin, Yanyang Li, Bowen Li, Jian Sun, and Yongbin Li. 2022. SOLQ Injecting syntax to question-scheme interaction graph encoder for text-to-SQL. parsers.
In Findings of the Association for Computational Linguistics: ACL 2022. pages 1254-1262, Dublin, Ireland. Association for Computational Linguistics.
Franco Scarselli, Marco Gori, Ab Chung Tsoi, Markus Hagenbuchner. and Gabriele Monfardini. 2009. The graph neural network model. JEEE Transactions on Neural Networks, 20( 1):61-80.
Parag Jain and Mirella Lapata. 2021. Memory-Basse Semantic Parsing. Transactions of the Association for Computational Linguistics, 9:1197-1212.
Wei Shen, Jianyong Wang, and Jiawei Han. 2015. Entity linking with a knowledge base: Issues, techniques, and solutions. IEEE Transactions on Knowledge and Data Engineering, 27(2):443-460.
Endri Kacupaj. Joan Plepi, Kuldeep Singh, Harsh Thakkar, Jens Lehmann, and Maria Maleshkova. 2021. Conversational question answering over knowledge graphs with transformer and graph atten- tion networks. In Proceedings the 16th Conference of the European Chapter of the Association for Com- putational Linguistics: Main Volume, pages 850-862, Online. Association for Computational Linguistics.
Alane Suhr, Srinivasan Iyer, and Yoav Artzi. 2018. Learning to map context-dependent sentences to exc- cutable formal queries. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Pupers), pages 2238-2249, New Orleans, Louisiana. Associa- tion for Computational Linguistics.
Ilya Loshchilov and Frank Hutter. 2019, Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019. New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Pierre Marion, Pawel Nowak, and Francesco Piccinno. 2021. Structured context and high-coverage grammar for conversational question answering over knowl- edge graphs. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 8813-8829, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph attention networks. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3. 2018, Conference Track Proceedings. OpenReview.net.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca

---

Denny Vrandečié and Markus Krötzsch. 2014, Wiki- data: A free collaborative knowledgebase. Commu- nications of the ACM, 57(10)778-85.
Exact Match Accuracy (%)
Bailin Wang. Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. RAT-SQL: Relation-aware schema encoding and linking for text- to-SQL parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 7567-7578, Online. Association for Computational Linguistics.
DCGee (+D)
10
Kuan Wang. Yuyu Zhang, Diyi Yang, Le Song, and Tao Qin. 2022. GNN is a counter? revisiting GNN for question answering, In The Tenth International Conference on Learning Representations, ICIR 2022, Viruall Event, April 25-29, 2022. OpenReview.net.
Turn Position
Figure 4: Exact Match Accuracy for different context lengths averaged across question types (validation set; + D:: with entity disambiguation).
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Seao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: Sfate-of-the-arr natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
and Jure Leskovec. 2022. GreaseLM: Graph REA- Soning enhanced language models. In The Tenth International Conference on Learning Representa- tions, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
A The Effect of Context Length
Figure 4 plots the performance of DCGęz (disam- biguation setting) against progressively increasing context length E [0]. We observe that access to wider context is beneficial up to a point. Perfor- mance deteriorates with very long contexts (beyond turn position 5). We stipulate two reasons for this. Firstly, longer interactions might be long because users ask about more than one entity topic, in which case local context might be sufficient to pro- vide an answer. And secondly, longer interactions might be genuinely confusing and noisy for anno- tators to create, let alone models.
Xiaoyu Yin, Dagmar Gromann, and Sebastian Rudolph. 2021. Neural machine translating from natural lan- guage to SPARQL. Future Generation Computer Systems, 117:510-51 519.
Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li. Luyao Chen, Yuwen Zhang, Shreya Dixit, Vin- cent Zhang, Caiming Xiong, Richard Socher, Walter Lasecki, and Dragomir Radev. 2019a. CoSQL:: conversational text-to-SLL challenge towards cross- domain natural language interfaces databases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Lan- pages 1962-
We further assess how context length interacts with the availability of type information. Table 4 shows the difference in performance with and with- out explicit type linking at context lengths I and 5. As described in Section 5, DCG does not have explicit type linking while DCGez uses context- dependent linking, while both models upply entity disambiguation. AFLI is the absolute difference in Fl score between DCGez and DCG for context length 1. Similarly, AF1, denotes the difference for context length 5.
guage Processing 1979, Hong Kong, China. Association for Computa- tional Linguistics.
Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Heyang Er, Irene Li, Bo Pang, Tao Chen, Emily Ji, Shreya Dixit, David Proctor, Sungrok Shim, Jonathan Kraft, Vin- cent Zhang. Caiming Xiong, Richard Socher, and Dragomir Radev. 2019b. SParC: Cross-domain se- mantic parsing in context. In Proceedings of the 57th Anmual Meeting of the Association for Computa- tional Linguistics, pages 4511 1-4533, Florence, Italy. Association for Computational Linguistics.
Overall, we find a significant drop in perfor- mance for context length 1 compared to context length 5. This indicates that more type information becomes available with increased context length. However, performance varies with question types. Specifically, the exact match difference is lot bigger for Clarification questions compared to Quantita-
Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D. Manning,

---

Context Length
Context Length
Dift' due to type linking
DCGec
DCG
DCGpc
DCG
Question Type
EM
F1
LM
F1
EM
F1
LM
AFI,
ARM.
Arig
AEM₁
Clarification
75.66 68.61
52.22
53.87
82.01
74.82
82.03
72.1
23.44
14.74
2.72
Logical Reasoning
92.59 77.07
86.9
67.94
93.95
79.52
93.33
78.19
5.69
9.13
0.62
1.33
Quantitarive Reasoning
36.1 13.74
30.91
11.04
59.83
31.17
56.66
28.66
5.19
2.7
3.17
2.51
Comparative Reasoning
76.72 39.75
74.77
37.79
90.91
62.46
90.09
61.11
1.95
1.96
0.82
1.35
Simple Question (Coref)
88.18 79.83
83.04
76.27
88.49
79.9
87.41
79.18
5.14
3.56
1.08
0.72
Simple Question (Direct)
87.56 61.59
79.74
58.21
88.27
62.25
85.6 61.44
7.82
3.38
2.67
0.81
Simple Question (Ellipsis)
81.75 80.38
74.2
76.84
83.87 79.08
82.45 84.35
6.18
4.91
5.27
1.42
AC
EM
AC
EM
AC
EM
AC
FM
Aac
AgM:
AcC
A AgMs
Verification (Boolean)
88.02 61.45
82.15 48.24
87.41 66.32
86.75
63.66
5.87
13.21
0.66
2.66
Quantitative Reasoning (Count)
69.41 65.34
65.16 60.58
75.2 70.84
72.96 69.02
4.25
4.76
2.24
1.82
Comparative Reasoning (Count)
42.81 30.5
39.74
28.69
67.7
57.34
66.76
56.6
3.07
1.81
0.94
0.74
Overall
73.74 57.96
66.88 51.95
81.28
66.85
80.59 65.24
6.86
6.01
0.69
1.61
Table 4: Interaction of context length and type linking. Ar1, is the absolute difference in Fl score between DCGec and DCG for context length 1. AF1, is the absolute F1 difference for context length
tive Reasoning questions which seem to require access to larger KB subgraphs.
are diagonally stacked (Fey and Lenssen, 2019). This creates combined graph that holds multiple isolated subgraphs as shown below:
B Model Details
A
Our model implemented using Py- Torch (Paszke et al., 2019) and trained with the AdamW (Loshchilov and Hutter, 2019) optimizer. It was trained with an A100 GPU with batch size of 64 and initial learning rate of 0.001. AdamW coefficients 31 and B2 (used for computing running averages of gradient and its square) were set 0.9 and 0.999, respectively. W The weight decay coefficient was set to 0.01 for all experiments. Hyperparameters were set based on initial experiments using a manually selected grid. We did not tune learning rate parameters. We choose the number of GATv2 and decoder layers from [1, 4] and found 2 to work best. Our SPARQL query server
A
An.
where is the batch-size number of graphs. Node input H and target features are simply concate- nated in the node dimension as follows:
H
H
H
H
deployed using Blazegraph. which uses only CPU-based resources and has access 100G of RAM.
We use two attention heads with GATv2. Specif- ically, lct K denote the attention head as com- puted (Velickovic et al., 2018) in Equation (3). The output of each head is concatenated follows:
Σ Caty w JEN
where || represents concatenation. at are normal- ized attention coefficients computed by the k-th attention mechanism as in Equation (1).
Our graph is represented adjacency ma- trix. To create a mini-batch, adjacency matrices
com/

---

C
The SPICE Dataset: Question Types
Utterance: Who created the design for Samus Aran?
Answer: Hiroji Kiyotake
Simple Question (Ellipsis)
Utterance: And how about The Dreamland Chronicles: Freedom Ridge?
Answer: Julian Gollop
Utterance: Who starred in Mathias Kneissl ?
Simple Question (Direct)
Answer: Rainer Werner Fassbinder, Volker Schlöndorff, Hanna Schygulla
Utterance: Who was the director of work of
Simple Question (Coreferenced)
Answer: Reinhard Hauff
Utterance: Does Dubashi have that person actor
Verification (Boolean)
Answer: No
Utterance: Which works of art are Rainer Werner Fassbinder or Laura Esquivel a screenwriter of ?
Logical Reasoning (All)
Answer: The American Soldier, Lili Marleen, Love Is Colder Than Death...
Utterance: How many administrative territories or political territories
did that work of art originate ?
Clarification
Did you mean Lili Marleen ?
No, meant Querelle. Could you tell me the answer for that?
Answer:
Utterance: How many still waters situated nearby Norway
Quantitative Reasoning (Count)
or Austria- justria-Hungary? ?
Answer:
Utterance: How many watercourses are more number landscapes located than Kafue River ?
Comparative Reasoning (Count)
Answer:
Utterance: Which political territories are located nearby lesser number of watercourses or bodies of water than Bareyo ?
Comparative Reasoning (All)
Answer:
Table 5: Examples of question types attested in SPICE (Perez-Beltrachini al., 2023). The utterance in gray is provided for ease of interpretation.


Reviews:
[{"review_id": "UIQCR3zvwv", "reviewer": "Reviewer_Qheq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We discuss the Lasagne model (Kacupaj et al., 2021) in the Related Work Section (line 176 onwards). However, their work is not directly comparable to ours because they assume a smaller, fixed ontology. This means that their model cannot be used (without modification) in a large-scale dynamic environment, like the one we are addressing in this paper. We will explain this in more detail. *Although the results are not directly comparable, we present below the performance of our model and Lasagne (as reported in Perez et al., 2023).*\n\nOur work outperforms Lasagne:\n| Model | F1 | EM\n|---|---|---|\n|Lasagne | 79.50 | 66.32 |\n| Ours | 81.28 | 66.85 |\n\nWe are not aware of any previous work that integrates graph structural information with conversational semantic parsing. We are happy to accommodate any references the reviewer has in mind. As we discuss in Related Work, most previous approaches consider a non-conversational setting, while for those that are convesational the graphs are essentially table schemas which are significantly smaller and static. In contrast to Lasagne, our approach is executable and our knowledge graphs are created dynamically, i.e., the number of nodes varies per utterance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- An interesting approach which builds upon previous work and introduces dynamic context graph for semantic parsing\n- Extended comparison and analysis with BertSP", "reasons_to_reject": "- Analysis and comparison with similar model like LasagneSP is missing. \n-  The model lacks novel design", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses the problem of semantic parsing by utilizing dynamic subgraphs. These subgraphs are built by utilizing the extracted entities during training. The paper runs their experiments on SPICE dataset and compares with one of the models that is introduced in the dataset paper. They provide an extended comparison between different variants of the models, and provide an analysis of different components. However, I believe the paper is missing a comparison with LasagneSP (Kacupaj et. al), which is the most similar with their work, and the main different is the global graph versus a dynamic graph that is introduced as novelty in this work. The results from LasagneSP also computed in SPICE paper (Perez-Beltrachini et al.) are not included in the results tables.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eLHyS0FaQ0", "reviewer": "Reviewer_bYQ1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "SPICE is the only dataset we are aware of that can serve as a testbed for conversational semantic parsing over KGs. We agree that evaluation of our method on other datasets with new entities is an avenue for future work, in particular since our results show that entity disambiguation plays a significant role for this task.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "•\tThe strengths of the paper are in the good description of the model, detailed analysis of the results, reasonable experiment structure and baselines.\n•\tResults are presented well with a detailed ablation study; The ablation analysis and case studies provide additional insight into the effectiveness of the approach.\n•\tCode and data are going to be released\n•\tIt presents a novel approach with strong performance, and clear motivation. \n•\tThis is a very well written paper, coherent and easy to understand.\n•\tArchitecture of the model is well-defined in figure 2. \n•\tOverall, this is an effective paper with strong results", "reasons_to_reject": "•Most of the improvement is coming from disambiguation based on popularity of an entity in training data.  More experiments are needed to see the generalizability of this approach on new dataset and unknown entities", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper investigates conversational semantic parsing over general purpose KG’s. They use dynamic context modeling, shows promising result in handling conversational features such as ellipses and co-references.  The area of open domain conversational Q&A using dynamic graphs is an under researched area, and the paper explores that.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2haOejygW1", "reviewer": "Reviewer_4Uzo", "summary": "", "questions": "See Reasons To Reject", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Our work focuses on conversational semantic parsing over large-scale knowledge graphs. It is rather challenging to evaluate ChatGPT in this setting, for several reasons: \n\na) data leakage, we cannot know which portions of our training/test data have been seen by ChatGPT; \n\nb) it is not clear how to encode the KG as context for ChatGPT; \n\nc) it is not obvious how to engineer prompts that can perform the reasoning required for the task and incorporate discourse context; \n\nd) evaluating ChatGPT's answers against our goldstandard would also require special treatment as ChatGPT's answers are typically a lot longer and exact match would in most cases fail. \n\nSuch a study is not a simple baseline and but would rather merit a separate investigation which we leave to future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) This paper is easy to understand\n(2) The experiment results are strong", "reasons_to_reject": "In the era of foundation model, it is beneficial to check the performance of ChatGPT.", "questions_to_authors": "See Reasons To Reject", "paper_topic_and_main_contributions": "This paper proposes a semantic parser for KBQA. The model represents information about utterances and their context as KG subgraphs which are created dynamically and encoded using a graph neural network. Furthermore, the propose a context-dependent approach to type linking which is efficient and scalable.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an approach to conversational semantic parsing over large knowledge graphs. The paper has unfortunately received rather superficial reviewing which could not be improved during the discussion period. In what follows, I summarize my own view (AC), departing from reviewers' evaluations.

The paper addresses an important task of conversational QA. This is a very relevant contribution -- and the findings are evaluated on a very recent (and already established) benchmark. The paper follows a principled approach, by introducing dynamic graph modeling ("dynamic context graph"). 

The paper presents a thorough and detailed evaluation. It shows a detailed comparison against SOTA and a very insightful analysis of the system's performance, going beyond simple numbers. The results suggest an improvement for both simple and complex questions.

Minor suggestion for the authors: Figure 1 is very instructive and does a good job at making your task clear -- however, the queries are impossible to understand and rather confusing. Would it be possible to either incorporate node names into the figure directly or provide some explanation on them a bit earlier than line 067?
Decision: Accept-Main