Multilingual Generation and Answering of Questions from Texts and Knowledge Graphs
Authors: Kelvin Han
Claire Gardent
Venue: EMNLP 2023 Findings (2023)
Keywords: question generation
question answering
multilinguality
multi-modality
knowledge bases
consistency
TL;DR: We bring multilinguality (Brazilian Portuguese and Russian) to multimodal (Knowledge Graphs and Natural Language) QG-QA.

Abstract: The ability to bridge Question Generation (QG) and Question Answering (QA) across structured and unstructured modalities has the potential for aiding different NLP applications. One key application is in QA-based methods that have recently been shown to be useful for automatically evaluating Natural Language (NL) texts generated from Knowledge Graphs (KG). While methods have been proposed for QG-QA across these modalities, these efforts have been in English only; in this work, we bring multilinguality (Brazilian Portuguese and Russian) to multimodal (KG and NL) QG-QA. Using synthetic data generation and machine translation to produce QG-QA data that is aligned between graph and text, we are able to train multimodal, multi-task models that can perform multimodal QG and QA in Portuguese and Russian. We show that our approach outperforms a baseline which is derived from previous work on English and adapted to handle these two languages.

Multilingual Generation and Answering of Questions from Texts and Knowledge Graphs
Kelvin Han and Claire Gardent CNRS/LORIA and Université de Lorraine {huiyuan. han, claire gardent fr
Abstract
and Question Answering (QA) can be used for as- sessing the semantic consistency between an input KG graph and a generated English text, thereby providing reference-less metric to measure the quality of KG verbalisers (RDF-to-Texi models), The intuition is that, for match between an input and its output, the answers that are extracted for given set of questions from the input should be consistent with the answers that are extracted from the output (for the same set of questions).
The ability to bridge Question Generation (QG) and Question Answering (QA) across struc- tured and unstructured modalities has the poten- tial for aiding different NLP applications. One key application is in QA-based methods that have recently been shown to be useful for au- tomatically evaluating Natural Language (NL) texts generated from Knowledge Graphs (KG). While methods have been proposed QG-QA across these modalities, these efforts have been in English only; in this work, we bring multi- linguality (Brazilian Portuguese and Russian) multimodal (KG and NI.) QG-QA. Using synthetic data generation and machine transla- tion to produce QG-QA data that aligned be- tween graph and text, we are able to train mul- timodal, multi-task models that can perform multimodal QG and QA in Portuguese and Russian. We show that our approach outper- forms baseline which is derived from previous work on English and adapted to handle these two languages. Our code, data and models are available at https://gtt tlab. inria. fr/ hankelvin/mull http//mulltnnnnalleeee text_agqa.
One key challenge to building multimodal QG- QA however, is the lack of annotated QG-QA data that aligned between modalities that is available for training and evaluation of such models. Fur- thermore, to be useful for RDF-to-Text output eval- uation, a multimodal QG-QA approach has to be robust to surface variations since the answer to a question likely has different surface forms in the input graph and the output text. Its QG and QA also have to be cross-lingual that questions from multilingual texts be generated and answered from English-centric KGs. Additionally, the of questions generated for a given (graph, text) pair should sufficiently large and the model(s) should also perform QA with accuracy and consistency, so as to be able to provide a fair assessment of the semantic consistency between graph and text.
1 Introduction
The ability to generate and answer questions from both Knowledge Graphs (KG) and Natural Lan- guage (NL) text is useful in number of ways. It permits easing users' access to the knowledge con- tained in KGs without having to master complex query languages, since an NL query from a user may be directly applied to KG information to de- rive the answer. It also allows for questions to be generated and answered from both the open do- main information contained in NL text and the fac- tual knowledge contained in KGs/knowledge bases,
In this work, we investigate the generation and answering of questions from graph and from text for multiple languages besides English. We present models enabling this for Russian and in Brazilian Portuguese. In addition, we apply our approach to WEBNI.G, dataset of (graph, text) pairs that is used to train RDF-to-Text models, and we show that our approach extends to (Rebuffel al., 2021)'s reference-less, semantic adequacy metric for RDF- to-Text models for these two languages.
thereby widening the pool of information that is interrogable. And it is useful also for verifying the consistency of information between modalities. In particular, (Rebuffel et al., 2021) recently showed that multimodal KG/NL Question Generation (QG)
We make the following contributions. First, we create training data where questions are aligned with both their graph and their text answers, allow- ing the training of models that show cross-modal

---

consistency: i.c. for a given question, the answers obtained from a graph and from its semantically equivalent text are consistent.
machine translate this data to create training data for Brazilian Portuguese and Russian. Finally, we use this automatically translated data to train multi- modal (KG/NL), multi-task (QG-QA) models for these two languages. The following outlines our approach and details are provided in the subsequent sections.
Second, we derive silver training data from En- glish using Machine Translation (MT) and we demonstrate that the resulting multimodal, Por- tuguese and Russian QG-QA models trained on them have high internal consistency: in most cases, applying Russian or Portuguese question gener- ated from graph the same graph returns an answer that matches the graph entity that the ques- settiionnnll-gnneratee from. Third, we show that our approach has much larger question coverage than a baseline adapting (Rebuffel et al., 2021); 1)'s approach to our target lan- guages. Finally, to overcome the lack of any gold QA/QG data that is aligned between text and graph in these two languages, tion suite to include multiple internal, cross-modal, cross-approach and external checks. These checks also demonstrate that our approach significantly improves on the baseline.
Creating Question/Answen Data for pairs of English Texts and Wikidata Knowledge Graphs. We first create a large synthetic dataset (Section 3) of QA pairs for graphs and texts in English as leveraging existing resources (datasets and mod- els) already developed for that language allows to obtain QA pairs at scale. We do this by apply- ing off-the shelf QG and QA models texts from KELM (Agarwal et al., 2021), a large dataset of (Wikidata graph, English text) pairs. We call this data Q-KELM
tion was originally
designed our evalua-
Learning Controllable QG Models for En- glish Texts and Wikidata Knowledge Graphs. Using the Q-KELM dataset for training, we learn two controllable graph- and text-based QG mod- els which allows for multiple, varied questions, of different graph sizes and question types, to be generated from the same (graph, text) pairs. This is essential as it is through controllable generation with these models that we can significantly increase the QA/QG training data coverage (Section 7.2).
1.1
Terminology and notations
En PiBr
English
Brazilian Portuguese
Russian
g
(or KG grapht/gripks) subgraph' Wikidata KG (Vrandečić and Krötzsch, 2014); comprised of set triples (also called facts) of the form (subject, predicate, ohject) in English. (or NL texts/iexts).
Creating Question & Answer Data for the WebNLG Dataset (from both English Texts and Graphs). By applying our controllable QG models to the (graph, text) pairs of the WERNLGG (Gardent et al., 2017) dataset, (question, answer) pairs from texts and graphs that can be used to train models used in verifying the semantic match between WEBNLGG graphs and texts generated from these graphs.
"
text En-
sish/Poottguuss/Rssiiiii
"X"
The context of question one modal." ity (text or graph)
X
Semantically equivalent X in the other modality
create large dataset of
q
question
A collection questións" An answer in X (a graph answer (ag) is either a subject or an object entity in g: a text answer (a:) is a span in t) A subgraph of corresponding to and its answer
ax
d'
Learning Multimodal, Multi-task QG-QA Models for Brazilian Portuguese and for Rus- sian. By further using MT, heuristics and qual- ity filters (Section 5), we obtain silver-aligned, in- domain QA pairs that enable us to train QG-QA models for WEBNLG graphs and their texts in Por- tuguese or Russian.
nf
The number of facts related to given q (i.c. the size of its corresponding subgraph Ig'])
pine
A text in Portuguese (the superscript distinguishes languages)
2 Approach
Testing on WEBNLG and with an indepen- dent QA model. We apply our Portuguese and Russian, multimodal QG-QA models to WERNLG Portuguese and Russian evaluation data and com- pute correlation (Section 7.4) with human judge-
Since there is no cross-modal QG/QA data for Brazilian Portuguese and Russian, we approach the task by first creating the data necessary for training multimodal QG-QA models for English. We then

---

ment of semantic adequacy (i.e. Does a generated text match the semantic content of its input graph?). We also apply the generated questions to a retrieval- based multilingual QA model (Section 7.5) to fur- ther verify the quality of the QG (i.e. How well can the generated questions be answered by an external model?).
4 Creating QA/QG Data for English
Our first phase generates synthetic (question, an- swer) pairs for WEBNI.G graphs and English texts in three main steps as follows.
Step 1: Synthetic QA data In the first step, we derive (t, ae: 9) triplcts from KELM texts² using off-the-shelff models for textual QG and QA. 3 We first use the QG model to generate questions from texts. Next we use the QA model to generate an- swers from the resulting (text, question) pairs. Us- ing set of heuristics, we then align (i) each text answer az to an entity in g giving the graph an- swer ag and (ii) each question to the subgraph g' of 9 matching its content, which in turn allows us obtain the question size, rtf.. The set of generated questions then filtered to ensure the quality of the silver data; this includes posing each question to a second deepset textual QA model for English and only accepting questions which both QA models found answerable and whose answers are the same or overlap. Questions whose text answer could not be matched to a graph answer were also removed. This gives us an initial synthetic QA/QG dataset of (g, ag. t, at, nff..f... q.g") tuples, which we call Q- *
We show that our approach brings substantial improvements over the baseline, to question cov- erage, QA consistency, correlations with human judgments of semantic adequacy as well as answer- ability by a retrieval-basee multilingual QA model.
3 Data
WERNLG has 38,872 (graph, English texts) pairs where each graph a Wikidata graph and each English text was crowdsourced to match the input graph.'
W (Shimorina et al., 2019) created a Russian version of WEBNLG by machine translat- ing 16.522 English texts from the original WEBNLG dataset, followed by crowdsourced post-editing. This dataset used in the 2020 WebNLG Chal- lenge (Castro Ferreira et al., 2020).
WEBNLG Similarly, (Almeida Costa et al., 2020), created Brazilian Portuguese version of the WEBNLG test set by using MT and a pool of native Portuguese speakers for post-editing.
Step 2: English Text/KG QG models Using O-KELM#", we train two QG models that can con- trollably generate simple (one KG graph fact) and complex (>1 KG graph fact) questions from either text or graph. The controls we use are: the size of the question in terms of KG triples (nf), and plau- sible question types (e.g., who, what, when, where, which, how (to)) in the case of simple questions generation. These controllable QG models enable our generation of wide-coverage in-domain QG- QA data that also allows us to generate multiple questions per input (see next point and Section 6).
KELM consists of 15M (Wikidata, English Texts) pairs. The English texts were synthetically generated from Wikidata graphs using a T5 pre- trained model that was itself fine-tuned on TekGen, large dataset of (graph, text) pairs that was created using distant supervision.
We reserve the test sets of WEBNLG PoBr and *F - 1,606 (g, PtBrr and 1,102 (g, thby pairs respectively - for evaluation as the parallel (g, l) in these test sets means that if question can be answered by a text (or graph), can also be an- swered by the graph (or text) that is paired with this allows us to test cross-modal QG and QA for consistency (Section 7.3). We use the training por- tion of WEBNLG to produce our QA/QG datasets, which is done in two phases and which we describe in the following sections and illustrate in Figure 1.
"We used version of KELM filtered for (g. t) pairs where g has between 2 and 5 triples (as larger sizes lead to unnatural questions). We also filtered out the KELM (g. O) pairs (i) whose properties in were not found in the Wikidata SPARQL endpoint have functional nature, i.c. containing terms such 'image of", 'instance of human'. 'list of 'disambiguation' which gives rise to superfluous in KELM; and (ii) that do not have a high fidelity with their g, using a similarity measure (Scao and Gardent, 2023) trained via contrastive loss RDF graph-tex tettt-tettttttttt pairs.
ht tps: hpps://hugaa http///ggggggga//eee wtwp:inngaace.cc/aaaalll.. co/valhalla/ t5- 15-base-e2en ag. a T5-base QG model fine-tuned on SQUAD 1.0 data, as well as and https: co/ beappsstt/robert-bbaeeee / //huggingfar acee co/ teteeseet/debertaaaee sesa--bae--quuas squad2 which are CoBFRTTw/DBBRRTaabbba QA models that are hoth fine-tuned on SQuAD 2.0.
II WEBNLG, the graphs are from the DBpedia KG. Here we use version where some of the DBPedia graphs have been mapped to Wikidata (Han et al., 2022), or else removed of underscores and camelcase to align with the Wikidata format.

---

SQUAD
Synth Synth-QA/QG QA/QG'm
SQUAD"
Synth-QA/QG"
(g. 17)
(g. a 4")
WebNLG
WebNLG
TextQG
(g) 1m)
TextQ6
Baseline
Our approach: Phase
Our approach: Phase
KELM WoDNLG
Q-KELM
Q-WebNLG
TextQG
Q-WebNLG
Q-WebNLG"
(g. )m)
(9.g R. 0,
(a. a tm, o 8's nf. 9m)
cQ-Gen
MT
Leerese aped
Triningi isu
SQ-Gen
MT
Dinoeet
100 Gghemati
PIR
WIKIDATA
Figure 1: Comparing our approach against the baseline. Using KELM and controllable QG increases coverage. Aligning questions with contexts and answers across modalities improves consistency. Machine translation provides multilingual training data.
Step In-domain training data Finally, these two models are applied to the training portion of WERNLGG By cycling through the set of possible graph and text answers, and leveraging the con- trols we introduced in Step 2 (question sizes and question types), we ensure wide coverage of the resulting set of questions.
form Q- D-WERNLG into Portuguese and Russian versions.
We describe the process for Portuguese first. We translate both the English texts of WEBNLGG training data and the set of questions from Q-WEBNLG using T5 Porrish-Poruuguese translation model We filter these translations using checks (details Appendix A.2) that include back-translation and keeping only those whose automatic scores are above cut-off. Since we machine translate the questions from English, the semantics of the ques- tions might be affected; such for each translated question y, then textual QA model trained on Portuguese SQuAD data obtain the answer to that question from the (automatically translated) Portuguese text (a. PtBr) and we use heuristics (Ap- pendix 1.2.1) to align this text answer to an entity 0g in the matching graph We discard any QA pair for which a PtBr is different from the graph answer a associated with the original English question (the question q is translation of).
Answerability, consistency and alignment We add wwweerabblity+ccoossseeee filter the questions generated from either graph or text of WERNLG, by posing them the two deepset tex- tual QA models - we then keep only questions where both QA models return an answer which (i) has a confidence score > 0.7, (ii) shares at least token overlap with the other model's answer, and (iii) has token overlap with the answer used to condition QG (i.c. text answer for questions from text; graph answer for questions from graph). In this way, for all questions that are generated from graph, we ensure that they have a matching graph and text answer. We call the resulting dataset of graph- and text-based (question, answer) pairs, Q Re
We do the same for Russian, except that we do not translate WEBNLG training data to Russian as it is already available (Shimorina et al., 2019). To translate the WEBNLG questions into Rus-
5 Creating QA/QG Data for Russian and Portuguese
translation-en-pt-
the second phase, we leverage MT and multi- lingual KG entity labels from Wikidata to trans-
*Recall that in WERNI,G. N.N,, each text is paired with match- ing graph with semantically equivalent content.

---

nf
Q-K Ri
Q-W
Q-WPeRr
Q-
our QG training instance can then contain multi- ple questions, enabling the generation of multiple questions from a given (X, n.f) input.
44,464
19,467
7,557
3,250
341,082
61,346
15,463
8,736
234,170
25,170
5,446
3,612
For QA, we associate each (q, ax) pair from a given X to other contexts in the data that encom- 2,004
22,607
13,918
2,411
7,031
TOTAL
1,149.354
119.901
30,877
17,602
pass to give (X, ax ; 7) every (X, ax 9) triplet in this set is then created as a QA training instance, thereby allowing QA coverage to be increased.
Table I: Data Statistics. Number of questions the QA datasets; rff: the size of the question (no. of facts)
Handling Unanswerable Questions. To allow our model to abstain from an answer if the ques- tion cannot be answered from the context, we use two strategies (details in Appendix B.1) to obtain negative unanswerable (g. -X) pairs.
sian, we use NLLB-200-1.3B" (NLLB Team et al.. 2022), an MT model achieving state-of-the-ar for 200 languages.
We call the resulting QA datasets, Q- Table
WEB and Q- Q-WEBNLGR*, summarises their statistics and example instances of the data can be seen in Table 7 in Appendix A.
Evaluation and results
7.1 Baseline: single-task models
6 Multimodal multi-task QG-QA model
We train monolingual models (i.e. one each for Portuguese and Russian) to limit the effect of cross- lingual transfer during training (i.e. only English- Portuguese or Russsihn when generating and answering with graph). Each language-specinn model - which is fine-tuned from the public check- point of the 400M-parameds mT5-small (Xue et al., 2021) for 10 epochs (to be comparable with the baseline) is trained in multimodal (graph and text), multi-task (QG and QA) manner, where each training batch contains a mix of all the tasks, i.e. Text QG, Text QA, KG QG and KG QA. This gives us a single unified model that can generate and answer questions from text and from graph.
Dataset
Portuguese
Russian
SQuAD
118,678
50,364
WebNLG
125,957
62,007
TOTAL
244,635
112,371
Table 2: Baseline training data. Number of (t, or, triplets in the obtained datasets for each language.
The baseline we compare against comprises four different models for each language and is similar the approach described for the Pata-QuestEvall met- ric (Rebuffel et al., 2021) for English, which they have shown to be useful for reference-less evalua- tion of English RDF-to-Text models. First, textual QG and QA models are trained on SQuAD data for Portuguese and Russian." The textual QG mod- els are then applied to the Portuguese and Russian texts of the WEBNIGG training data; whereby the entities from the graph that is paired with each text are used to condition QG. This provides (g. 0g ?) triplets that can be used for training the KG QG and KG QA models. Table 2 contains statistics about the training data for the baseline. To remain in the same model paradigm as (Rebuffel et al., 2021), ev- ery model here is also fine-tuned from mT5-small for 10 epochs.
It takes approximately 20 hours to fine-tune of our language-specific multimodal multi-task models using a single Nvidia A40 GPU.
Maximising Coverage To maximise coverage, we train the QG model to generate multiple ques- tions from the same input and we extend the set of possible sources for a (question, answer) pair thereby facilitating question answering.
For QG, gather into set the questions gen- erated from each X (t or g) in Q-WEBNLG **, and add to questions that were gen- erated from other contexts whose semantic content is contained in X. QG coverage is then maximised by gathering all n sized questions that shared the same answer in this set - by doing so, each of
FFor Portuguese we use a version of SQuaD v1.0 (Rajpurkar et al., 2016) we understand produced with MT and post-edited by native speakers, avail- able at htttps:/huuggii squad sqqad_vv__pt____ _r_____br;bb for Russian we use the SherQuad (Etimov et al., 2020) dataset.
co/ aoo/daattts///rhuraaaii
'https: //wgggnggggga.cee...mmmm co/ co/facebook/ nl1b-200- 2ist000ddllleddd11.11

---

We evaluate the multimodal QG-QA models by computing three consistency metrics that consider the various answers that question q can be asso- ciated with: ax, the ground truth; åx, the answer generated from source X (e.g., text); and åx', the answer generated from the other modality X' (e.g., a graph). Insernal Same-mod (GT) compares åx with the ground truth answer ax, indicating the approach's 's self-consistency. Internal X-mod (GT) compares ax with åx' the answer derived from the other modality. Internal X-mod (Gen Ans) com- pares âx and åx, . the answers from each modality, 7.2 Evaluating Question Coverage
Modality reference
Portuguese
Russian
We use two measures to assess question cover- age. We first compare the number of unique questions generated by cach approach on the "WeNLo test sets. We further use BERTScore (BSc) (Zhang et al., 2020) to assess their semantic overlap by taking one approach's question for given X as prediction and the other's generated questions for the same as multi-references. The intuition is that if approach A scores higher with approach B's questions as references than vice-versa, A's questions are "con- tained" in B' s and conversely, B has wider semantic coverage.
Text
Baseline
83.1
81.3
Ours
85.0
81.3
Graph
Baseline
80.7
75.9
Ours
84.6
80.1
Table 3: Avg BSc, where questions generated by Sys- tem A for given or are scored against the set of generated questions by System for the same g or
ground truth (GT). d'x denotes generated answer from context X. We superscripts distinguish outputs from different models for e.g., a is the answer derived from input context X by model A.
Figure 2: Portuguese: Comparative QG coverage Baseline vs. Our Approach. Number of generated questions much higher for our approach across both modality and input size.
By also posing the questions generated by one approach to the other approach's QA in a cross approach manner, we gain an external indication of their QG-QA capabilities. For this, we ex- amine the two answers that approach can gen- crate (a and a when given 9 a question generated by A. We do this on three levels: (i) X-Appr Same-mod (GT) compares a against ax on the same modality that ox came from; (ii) X-Appr X-mod (GT) compares B's generated an- swer with the GT answer across modalities (i.e. a vs a and (iii) X-Appr X-mod (Gen Ans) where a is compared against á A graphical overview of these comparisons in Figure 4.
Puorage get pelty giite TINATOO
quar woited ERom
14.7% 8110
Figure 3: Russian: Comparative QG coverage Baseline vs. Our Approach. Number of generated questions is also much higher for our approach across both modality and input size.
Wider q coverage Our approach has a sub- stantially higher question coverage (up to 4x more for Portuguese and nearly 7x more for Russian) than the baseline (Figure 2-3). In terms of seman- tic coverage (Table 3), we outperform the baseline in all modalities for Portuguese; also outper- form in graph for Russian, and on par in the text modality there.
Both approaches usually generate different number of questions for given X; therefore. to ensure fair evaluation, when Approach A gen- erates more questions for X, we randomly sample from its set as many questions that Approach generates for X. We also added a self-consistency filter removing those questions that are unanswer- able from their own context. Settings for these are in Appendix B.3.
7.3 Evaluating QA Consistency
what follows, the answer ax that is used to condition the generation of qx is referred to as the
As the token Fl metric commonly used extrac- tive textual QA cannot account for lexical variation

---

Portuguese
Approach
Internal
X-Appr
X
X
QG
Baseline Baseline
Ours Ours
Baseline Ours
Ours Baseline
d"a
QA
Self (GT)
X-Appr
(Gen, [
X-Appr X-modd (GT)
T
(_.0)
94.7 (20.41)
73.6/-12.7)
(0.16-34.
60.1
(50.25)
(GTT)
Same-mod mod
G
_5.4(_).03)
96.8 (00.5)
58.2
(-27.2)
61.0
61.0
(-35.8)
(20.12)
(10.53)
x
X-mod (GT)
a
G T
43.7 (10015) 65.6
47.0
(10.11
+3.3)
47.8
47.8
(10.30)
Internnd
[L,m
(GT)
T - G
40.0(10.u8)
62.9 (2.
45.7
(20.12)
+5.7)
37.81-25.11 (-2511
37.8
(40.35)
X-mod (Gen Ans)
Internal X-mod
G T
39.9 67.1 (=0.34)
42.5
(10.09)
-2.6)
47.7
(-
(10.30)
X
(Gen, Ans)
(1000)) 65.0 (20.77) 44.3% +4.3)
(10.11)
39.0
(10.34)
Approach
Russian
Figure 4: QA Accuracy. Bold lines denote QA com- parisons within/between modalities and/or approaches. Dotted arrows indicate the context X or X' that the question (gx) is posed against obtain answers. Leross; cross- lonss-lingual answer comparison (e.g. PtBr/Ru against En); 1 same denote comparison in the same lan- guage.
Internal
X-Appr
QG QA
Baseline Baseline
Ours Ours
Baseline Ours
Ours
Baseline
Self (GT)
T - T
84.6(_0.02) 0.02)
89.8 (9.111
57.8 57.8 (-26.8)
(#0.10)
(10.11))8)
79.4(10.03)
(6.2()
48.94-30.5
(10.10)
57.1
(-391)
(10.53)
X-mod (GT)
present in cross-modal QA, we follow (Rebuffel al., 2021)'s use of BERTScore (BSc) for eval- uation; however, we also computed token FI and exact match scores for verification, and these are provided in Tables 8-9 in Appendix C.I.
G 20.8(_0.vs) (2
24.5(
+3.7)
20.6
-9.1)
(10.06)
(10.211
T G 20.2(_0.00) 30.0(_0.0s)
17.3 17.3
(20.03)
-2.9)
21.5
(40.06)
-8.0)
X-mod (Gen Ans)
18.6/20.04) (1.1
22.8 +4.2)
20.3 10.08)
(10.06)
(10.23)
Table 4 show the results of the QA consistency tests for Portuguese and Russian. We observe simi- lar trends for both languages, though the BSes for Russian are noticeably lower under the modsl (and cross-lingual too, since 0g is English) tings, This is likely due to (i) the smaller size the training data for Russian (Table 1), (ii) more fre- quent transliteration foreign names into Cyrillic, and (iii) potentially, the freer word order in Rus- sian which relies case instead of S- V-V order to mark the subject and object (as it is for English and Portuguese).
T G G 18.4220.07) (0.10)
15.0
(20.03)
-3.4)
22.2 -8.6) (10.066)
Table Consistency Results. Average of BSes be- tween answers. In subscripts are std. dev. across 5 random runs; superscript are the difference between X-Appr and Internal, the differences provides meaningful compari- son between the baseline and our approach since each of their QA performances are different. Whenever our QA is used, the drop in performance is reduced.
over the baseline in Internal X-mod (GT). This is particularly pertinent for the Data-QuestEval metric, as the metric computed by com- paring the generated answer ax' against the GT answer ax- Our approach also always leads to gains in Internal X-mod (Gen Ans) over Internal X-mod (GT), which does not happen for the baseline. This means that our QG-QA gener- ates answers that more consistent between both modalities. This improvement could have come the expense of the QG-QA' performance vis-a-vis the GT answer (i.e. Internal X-mod (GT)), but this is not the case for our approach, which further val- idates our multinodal multi task training to give
More self-consistent QA Our approach con- sistenily leads to higher scores compared to the baseline in the Internal comparisons for all lan- guages, showing that multimodal, multi-task train- ing using silver aligned data (even at much smaller scale nearly 4 times less for Portuguese and more than 3 times less for Russian; cf. Table I and 2), improves QA self-consistency.
More consistent cross-modal QA We also improve by between 8.9 (29.7 over 20.8, Rus- sian) to 22.9 (65.6 over 43.7, Portuguese) BSc

---

more consistent cross-modal QG-QA.
Better correlations with human judgments Using our approach to compute the Data- QuestEval metric leads to a gain of more than 12 points in its correlation with human judgments (Table 5). Together with the more consistent inter- nal and cross-modal QA above, this shows that our approach leads to more robust QG-QA. Ii also indicates that QA-based reference-free evalua- tion methods like Data- Sata-QuestEval can be improved upon with wider QG coverage.
QQ-QA externally validated Whenever the baseline's 's QA is used to answer questions gener- ated by our approach, QA accuracy drops signif- icantly less (in some cases, there a gain) than vice-versa (Table 4). This could be predominantly the result of either (i) better QG by our approach: or (ii) better QA by the baseline - however, our ap- proach's 's stronger performance in all the Internal, X-mod and X-Appr comparisons suggests that the former is the likely factor. This gives indication of the quality of the questions generated by our approach over the baseline.
Baseline
Our Approach
16.4 (4.78n-06)
28.7 (4.21x-16)
Table 5: ments. The baseline vs our approach used compute the Data-QuestEval metric. All (p-values) < 0.001.
Correlations (Pearson's r) with human judg-
QA consistent over q qiiee Finer-graineds anal- ysis (Table 10 and 11 in Appendix C.2) shows that our approach's QA performance is consistent across nff, i.e. it is also generating and answering questions of consistent quality across questions of different "sizes" (i.e. simple and complex).
7.5 Multilingual Retrieval-based QA
For a further verification of the QG abilities of the baseline and our approach, we also posed the questions generated by each (the same ones as in Table 4) to mGEN (Asai et al., 2021). This is mul- tilingual retrieval-based QA model that fine- tuned from m'T5bbase (Xue al., 2021) check- point to generate the answer question given a collection of retrieved input contexts."
7.4 Data- vuestevall metric
The QA consistency tests above reflect "gold" setting where both modalities (texts and graphs) are semantically aligned. To evaluate whether our approach brings improvements settings where this does not hold (i.e. Can it generate and answer questions across modalities where one modality has missing, or additional, information vis-a-vis the other?), we compared it against the baseline when used to compute the Data- Suta-QuestFval metric.
Portuguese
External Retrieval-QA
QG QA
Baseline mGEN
Ours mGEN
(GT)
We do this by comparing the metric's result- ing correlation with human judgments of semantic quality when using each approach. For this, we used the sampled set of system submissions to the Russian RDF-to-Text task in the 2020 WebNLG Challenge. together with their human ratings", com- prising 660 generated texts from six submissions." Since the texts here were machine-gennerate given a WEBNLG graph input, these texts may contain errors or are ill-formed, therefore the approach giv- ing a higher correlation with the human ratings indicates that it is better able to answer questions that cannot be answered by the other modality (i.e. detect difference in information content).
T-T G
71.4(20.07) 57.7(20.18)
78.0 (0.58)
72.8, (3..8)
Russian
External Retrieval-QA
QG QA
Baseline mGEN
mGEN
Self (GT)
T T
(5.9 19:
68.5 (3... 18)
G - G
66.9, 16.9)10.18)
76.0 76.0
Table 6: QA consistency (BSc) comparing QG of Baseline and Ours, using mGEN for QA.
Better answerable questions Compared to the baseline, approach' questions were answered
"https: //gi thub. wwt..cco//eeeeeeeeeeeeeeeeo WebN/lebGGGo Www.eeeehhaaallaaaaeeeeeeeaoaaaa /tt.g//hhhhaeeeeeeeeee
"We use the ratings of Data Coverage. Relevance and Cor- rectness (summing their normalised scores). Since the out- puts for the challenge's 's human evaluations were sampled in a random stratified manner, we computed correlation using Pearson's 's
"We use the version of the code and weights that was releused as part of the MIA-2022 Shared Task (Asai et al., 2022). See https: /ts:://ithul wom/mia-wwrkssee MIA-Shared- Tatee-Taask-2e2rreeeeeeeee 2022/tree/main

---

better by mGEN we see (Table 6) an increase of between 2.6 BSc for TexQQG (Russian) and 15.1 BSc (Portgguese) in answer accuracy, providing a further independent verification of our approach's 's QG capabilities.
Synthetic data Other work has investigated the use of synthetically-generrii data with round trip filtering techniques and shown improved textual QA performance (Alberti et al., 2019; Puri et al., 2020; Kwiatkowski et al., 2019). (Riabi et al., 2021; Agrawal et al., 2023) have also examined multilin- gual synthetic QA/QG data generation. Similarly, we used data augmentation and round trip filter- ing to improve generalisation; however, these other work are aimed at improving textual QA only; un- like ours which aims to improve multilingual QG and QA jointly, which is also cross-moos moosl for text and graph while also ensuring wide QG coverage.
8 Related work
Joint QG & QA and data Existing work on joint QG & QA (Wang et al., 2017: Duan et al., 2017; Lyu et al., 2021; Luo et al., 2022) have mostly focused on the text modality and in English only. Where more than one modality or multiple lan- guages are considered, these have usually been for QA alone. This is partly because available anno- tated multilingual QA/QG datasets (Lewis et al.. 2020; Artetxe et al., 2020) are limited in size and primarily reserved for testing, well as built for one modality only. While there exists moderate- sized KG QA datasets, including the long-running QALD (Usbeck al., 2023) and LC-QuAD (Dubey et al., 2019) KG QA challenges, these are still of a limited size (and coverage). Furthermore, none of these QA/QG datasets have alignment of QA pairs across modalities, as well as across languages. Notably, since open KGs such as Wikidata (though large and with wide and growing fact coverage) remain English-centric, the setting examine has involve cross- lonss-lingulliyy when carrying out QG and QA the graph modality.
QA-based evaluation of texts Our work is most similar to (Rebuffel et al., 2021) who proach like our baseline to show that QG and QA can be useful for assessing the semantic adequacy of generated texts in Text tasks in English. Contemporaneous work by (Cohen et al., 2023) also similarly leverage consistency measures to ver- ify the factuality of a model's 's generated answers. Nonetheless, these works remain in one modality and language only (English texts).
ap-
Conclusion
In this work, we examine the task of multimodal QG and QA from text and from graph in multiple languages which has application in QA- eval- uation of text generated from KG. By generating synthetic QA/QG data in English that has questions and answers aligned between text and graph, and using MT. heuristics and quality filters, we obtain "silver" data for Brazilian Portuguese and Russian that enables the training of multimodal multi-task models. Our models provide wider QG coverage, is cross-lingual for QG-QA from KG graphs, and achieves greater internal and cross-modal QA con- sistency over baseline that is derived from work in English (Data-QuestFval) (Rebuffel et al., 2021) recently shown to be useful as metric for eval- uating the semantic consistency of RDF-to Text generations. In fact, using our approach leads to > 12 points gain in the metric's correlation with human judgments for Russian. We also see strong performance in QA accuracy of up to 15.5 BSc when our approach's questions are posed to mul- tilingual retrieval- based QA model. Holistically, our approach's consistently better performance in coverage and QA consistency, demonstrates the improvements that it brings over the baseline.
Bridging modalities Early efforts to bridge QA between modalities focused on supplementing lim- ited KG coverage by extracting relational infor- mation from more abundant texts. For instance, (Fader et al., 2014; Das et al., 2017) leveraged both structured (KB, tables, lists etc) and unstructured (text) information and used information extraction methods such OpenlE (Banko et al., 2007) and UniversalSchema (Yao et al., 2012) so to employ semantic parsing- or mules-basdd KG QA methods. More recent work instead casts structured infor- mation as text to access their knowledge through textual QA methods. (Agarwal et al., 2021) con- structed KELM as verbalisation of a large KG (Wikidata) to add to retrieval LM corpus, obtain- ing performance improvements on benchmark QA datasets. (Oguz et al., 2022) obtain improvements by adding Wikipedia tables and lists to the data
mix. In similar vein as (Agarwal et al., 2021). (Zhang et al., 2023) verbalise a multilingual KG QA dataset to utilise textual QA methods.

---

10 Acknowledgments
entity. This results in there being more textual QA instances than KG QA. For balance, may be pos- sible to using beam search (or variants of it such as diverse beam search (Vijayakumar et al., 2016) and constrained beam search (Post and Vilar, 2018)) to increase the generation of complex questions from graph using the Q- O general purpose QG model. Finally, although we used the agreement of two state-of-the-an QA models when checking for QG acceptability, we cannot be cer- tain that questions rejected by the QA models are not actually valid questions - in such cases, it means that the coverage of QA pairs in our datasets is constrained.
We thank the anonymous reviewers for their feed- back. This research was supported by ANR Project We
QUANTUM -arpicc--AN-----E22---000000 gratefully acknowledge the support of the French National Research Ageney (Gardent; award ANR- 20-CHIA-0003, XNLG "Multi-lingual, Multi- Source Text Generation") and of Facebook Al Re- search (FAIR) Paris. Experiments presented in this paper were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS. RENATER and sev- eral Universities as well as other organizations (see fr).
12 Ethics Statement
11 Limitations
As advancements in generative technologies ac- celerate in terms of capabilities, scale and public access, so too must the need for the the ability to understand if such machine- seaiineggeerrtedd information are reliable.
Our data generation process relies on the KELM
dataset (Agarwal et al., 2021), which was gener- ated using a single pretrained language model (T5- large). As such, KELM - and hence the texts in our generated data (Q-KEI.M too - reflects only the set of factual or linguistics characteris- ties in this model. For instance, there is certainly more than one way KG graph (or set of facts) can be lexicalised in text, however KELM only contains one lexicalisation for each of the KG graph within it. In addition, we only use a sin- gle QG model for generating our initial sets of synthetic QA pairs Q-KELM Although by using O-KELM to train our two general QG models, we may have introduced new varieties of questions into Q- WEBNLG (and therefore Q- WEBNIGG PtB and Q- WEBNLG it is unlikely we have obtained the full range of questions possible for a given context-answer pair. This limitation of KELM may however be alleviated by for example, ensembling the KELM dataset with generations from different LMs on the same subgraphs used to generate the KELM sentenceesppasaages.
We believe that our multilingual KG/NL- QA/QG data creation method and multimodal multi-tas multi-task QG-QA modeling has the potential to contribute positively in the following man- ner: (i) a cross-lingual (English-Brazilien Por- tuguese/Russian) QG-QA model with cross-modal consistency can aid in tasks that includes but are not limited to automated fact verification, KG-to- text/text-to- KG quality estimation and KG comple- tion; and (ii) the ability to generate multilingual in- domain QA data can help improve downstream QA performance and dialogue systems to aid human- machine interactions with KGs in English as well as other languages. On the other hand, a direct ap- plication of our method and model for a task such as fact verification could lead to a failure to cap- ture misinformation. which have the potential for substantive socictal harm. This risk arises because KELM is based on snapshot of the Wikidata KG from circa 2019. Additionally, the T5 pre- pre-trained language models used in producing KELM, and also the mT5 pretrained checkpoints used in all our models, were trained with data up to 2020/2021. These constrain the extent and validity of factual knowledge in our model up to these points in time.
When setting up the KG QA training instances from Q- W-WENNIGG*15//0- =FRr complex questions, we chose quality over quantity and only used those originating from KG graphs (i.e. we ex- cluded complex questions originated from texts)." !I This is because answers for questions generated from text may not be constrained to a single KG
1I Note that, without this step, our KG QA model would be steered towards generating Text QA-like answers. This would then bias the cross-modal comparisons (i.e. X-mod (GT) and X-mod (Gen Ans) and will also likely negatively affect the reliability of the Data- FuestFyalll metric to accurately assess the presence of KG facts mentioned in a given text.
References
Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. 2021. Knowledge graph based synthetic corpus generation for knowledge-enhancee language

---

model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3554-3565, Online. Association for Computational Linguistics.
on Natural Language Generation from the Semantic Web (WebNLG+), pages 55-76, Dublin, Ireland (Vir- tual). Association for Computational Linguistics.
Roi Cohen, May Hamri, Mor Geva, and Amir Glober- son. 2023. Lm vs Im: Detecting factual errors via cross examination.
Priyanka Agrawal, Chris Alberti, Fantine Huot, Joshua Maynez, Ji Ma, Sebastian Ruder, Kuzman Ganchev, Dipanjan Das, and Mirella Lapata. 2023, Qameleon: Multilingual qa with only 5 examples.
Rajarshi Das, Manzil Zaheer, Siva Reddy, and Andrew McCallum. 2017. Question answering on knowl- edge bases and text using universal schema and mem- ory networks. In Proceedings of the 55th Annual Meeting the Association for Computational Linguistics (Volume 2: Short Papers), pages 358- 365, Vancouver, Canada. Association for Computa- tional Linguisties.
Chris Alberti, Daniel Andor, Emily Pitler, Jacob De- vlin, and Michael Collins. 2019. Synthetic QA corpora generation with roundtrip consistency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6168 6173, Florence, Italy. Association for Com- putational Linguistics.
Nan Duan, Duyu Tang, Peng Chen, and Ming Zhou. 2017. Question generation for question answer- ing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. pages 866-874, Copenhagen, Denmark. Association for Computational Linguistics.
Felipe Almeida Costa, Thiago Castro Ferreira, Adri- ana Pagano, and Wagner Meira. 2020. Build- ing the first English-Braziin Brazilian Portuguese corpus for automatic post-editing. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6063 6069, Barcelona, Spain (On- line). International Committee Computational Lin- guistics.
Mohnish Dubey, Debayan Banerjee, Abdelrahman Ab- delkawi, and Jens Lehmann. 2019. Lc-quad 2.0: A large dataset for complex question answering over wikidata and dbpedia. In The Semantic Web- ISWC 2019, pages 69-78. Springer International Publish- ing.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 4623-4637, On-
Pavel Efimov, Andrey Chertok, Leonid Boytsov, and Pavel Braslavski. 2020. Sberquad russian read- ing comprehension dataset: Description and anal- ysis. In Experimental IR Meets Multilinguality. Multimodality, and Interaction, pages 3-15. Springer International Publishing.
line. Association for Computational Linguistics.
Akari Asai, Shayne Longpre, Jungo Kasai, Chia- Hswan Lee, Rui Zhang, Junjie Hu, Ikuya Yamada, Jonathan H. Clark, and Eunsol Choi. 2022. MIA 2022 shared task: Evaluating cross-lingual open- retrieval question answering for 16 diverse languages. In Proceedings of the Workshop Multilingual Information Access (MIA), pages 108-120, Seattle, USA. Association for Computational Linguistics.
Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 2014. Open question answering over curated and extracted knowledge bases. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '14, page 1156-1155, New York, NY, USA. Association for Computing Machinery.
Akari Asai, Xinyan Yu. Jungo Kasai, and Hanna Hajishirzi. 2021. One question answering model for many languages with cross-lingual dense pas- sage retrieval. In Advances in Neural Information Processing Systems, volume 34, pages 7547-7560. Curran Associates, Inc.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachine 2017. Creating training corpora for NLG of the 55th Annual Meeting of the Association for Computational I. Linguisties (Volume I: Long Papers), pages 179-188, Vancouver, Canada. Associ- ation for Computational Linguistics.
In Proceedings
Michele Banko, Michael J. Cafarella, Stephen Soder- land, Matt Broadhead, and Oren Etzioni. 2007. Open information extraction from the web. In Proceedings of the 20th International Joint Conference on Artifical Intelligence, IJCAI'07 page 2670-2676, San Francisco, CA, USA. Morgan Kaufmann Pub- lishers Inc.
Kelvin Han, Thiago Castro Ferreira, and Claire Gardent. 2022. Generating questions from Wikidata triples. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 277-290, Mar- seille, France. European Language Resources As- sociation.
Thiago Castro Ferreira, Claire Gardent, Nikolai Ilinykh, Chris van der Lee. Simon Mille, Diego Mous- sallem, and Anastasia Shimorina. 2020. The 2020 bilingual, bi-directional WebNLG+ shared task: Overview and evaluation results (WebNLG+ 2020). In Praceedings of the 3rd International Workshop
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew

---

Keleey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. 2019. Nat- ural questions: A benchmark for question answer- ing research. Transactions of the Association for Computational Linguistics, 7:452-466 466..
Papers), pages 1314-1324, New Orleans, Louisiana. Association for Computational Linguistics.
Raul Puri, Ryan Spring, Mohammad Shoeybi, Mostofa Patwary, and Bryan Catanzaro. 2020. Training ques- tion answering models from synthetic data. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5811-5826, Online, Association for Computa- tional Linguisties.
Patrick Lewis, Barlas Oguz, Ruty Rinott, Sehastian Riedel, and Holger Schwenk. 2020. MLQA: Eval- uating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7315-7330, Online. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pages 2393-2392, Austin, Texas. Association for Computational Lin- guistics.
Hongyin Luo, Shang-Wen Li, Mingye Gao, Se- unghak Yu, and James Glass. 2022. Coopera- tive self-training of machine reading comprehen- sion. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 244-257, Seattle, United States. Association for Computational Linguistics.
Clement Rebuffel, Thomas Scialom, Laure Soulier, Ben- jamin Piwowarski, Sylvain Lamprier, Jacopo Staiano, Geoffrey Scoutheeten, and Patrick Gallinari. 2021. Data-QuestEval: referenceless metric for data- to-text semantic evaluation. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8029- 8036, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Chenyang Lyu, Lifeng Shang, Yvette Graham, Jennifer Foster, Xin Jiang, and Qun Liu. 2021. Improving unsupervised question answering via summarization- informed question generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4134-4148, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Arij Riabi, Thomas Scialom, Rachel Keraron, Benoît Sagot, Djamé Seddah, and Jacopo Staiano. 2021. Synthetic data augmentation for zero-shot cross- lingual question answering. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pages 7016-7030, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Hef- fernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun. Skyler Wang. Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Bar- rault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov. Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Kochn, Alexandre Mourachko, Christophe Ropers. Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human- centered machine translation.
Teven Scao and Claire Gardent. 2023. Joint repre- sentations of text and knowledge graphs for retrieval and evaluation. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2023, Online and Bali, Indonesia. Association for Com- putational Linguistics.
Anastasia Shimorina, Elena Khasanova, and Claire Gar- dent. 2019. Creating a corpus for Russian data-to- text generation using neural machine translation and post-editing. In Proceedings the 7th Workshop on Balto- Natural Language Processing, pages 44 49, Florence. Italy. Association for Computa- tional Linguistics.
Barlas Oguz, Xilun Chen, Vladimir Karpukhin. Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Scott Yih. 2022. UniK-QA: Unified representations of structured and unstructured knowledge for open-domain question answering. In Findings of Association for Computational Linguistics: NAACL 2022, pages Seattle, United States. Association for Computational Linguistics.
Ricardo Usbeck, Xi Yan, Aleksandr Perevalov, Longquan Jiang, Julius Schulz, Angelie Kraft. Cedric Möller, Junbo Huang, Jan Reineke, Axel- Cyrille Ngonga Ngomo, et al. 2023, Qald-10 10th challenge on question answering over linked data.
Matt Post and David Vilar. 2018. Fast lexically con- strained decoding with dynamic beam allocation for neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long
Ashwin K Vijayakumar, Michael Cogswell, Ram- prasath R. Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2016. Diverse beam search: Decoding diverse solutions from neural se- quence models.

---

Denny Vrandečié and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledge base. Communications of the ACM, 57(10):78-85.
A
Data details
A.1 Synthetic QA data in English
Tong Wang, Xingdi (Eric) Yuan, and Adam Trischler. 2017. A joint model for question answering and question generation. In Learning to generate natural language workshop. ICML, 2017.
We used subset of KELM filtered for (g. t) pairs where g has between 2 and 5 triples; this is because larger sizes typically lead to unnatural questions. KEIMM (Agarwal et al., 2021) has 15M (g. t) pairs of which we used a subset of about 2M pairs to seed the generation of synthetic QA data on; about 1.1M texts remain in Q-KELM after our QA pair filtering steps. The distribution of Q-KELM Bn's questions by n.f can be found in Table 1.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws. Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young. Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's 's neural machine translation system: Bridging the gap between human and machine trans- lation.
A.2 Synthetic QA data in Portuguese and Russian
Quality filters for machine translation
Verifying MT on Portuguese texts To en- sure the quality of the machine translations of the texts from English to Portuguese, we backtrans- lated them and used two automatic scores: (i) BERTScore (BSc) (Zhang al., 2020) and (ii) Google BLEU (GLEU) (Wu et al., 2016) to assess the semantic and lexical similarity of the transla- tions and the original text - we used cut-offs of 00.7 for BERTScore and 0 for GLEU. We also used a regular expression to check for the presence of 10 or more consecutively repeated words which is typical error in MT, indicating issues with the quality of the translation.
Linting Xue, Noah Constant, Adam Roberts, Mi- hir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: mas- sively multilingual pre-trained text-to-text trans- former. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 483-498, Online. Association for Computational Linguistics.
Limin Yao, Sehastian Riedel, and Andrew McCallum. 2012. Probabilistic databases of universal schema. In Proceedings of the Joint Workshop Automatic Knowledge Base Construction and Web-scale Knowledge Extraction (AKBC-WEKE)), pages 116-121, Montréal, Canada. Association for Compu- tational Linguistics.
Verifying MT on questions For questions, we also use the same BSc and GLEU cutoffs above (=0.7 and 00.3) on the backtranslations of the questions to filter out poor quality translations.
Chen Zhang, Yuxuan Lai, Yansong Feng, Xingyu Shen, Haowei Du, and Dongyan Zhao. 2023. Cross-lingual question answering over knowledge base as read- ing comprehension. Findings the Association for Computational Linguistics: EACL 2023, pages 2439-28 2452, Dubrovnik, Croatia. Association for Computational Linguistics.
A.2.1 Aligning text answer to graph entity
The process here involves mapping to Por- tuguese/Russian using Wikidata multilingual labels for entity names. If multilingual label for the language cannot be found, the name is translated from English using Google Translate API; here, we found that a commercial-grade MT system can be better suited to translate entity names. These multilingual versions of g are only used to aid the mapping of a Ia to graph entity and are set aside thereafter (i.e. we do not use them in training)
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In Internatoonll Conference on Learning Representations.
B Training and evaluation details
B.1 Negative sampling for QA
For the textual QA and KG QA tasks, negative examples were created so as to allow the model to

---

Data
Greph
CAki Masecen of Art, Doet couns, 3
Miseam Art opening date, 2013 >
Nakaderi
(D)
< Alta Musserr of Art flor araa 3746 0716666600000 (sge)
Text
[EM] Akila
Art 142 Nakoskeri
Myneil
[Eneg] Museu
Nakoddei
with lotel area of 1746 ob aquare mnstre and vas insggamooo
vrroes ofroall maperaaaas 3746,66 eoupurmus sorpas fuue
com sina dnca seal 1745 As mettre gustmates
28th Septemtber
eoagui ceutralpe 2013 ros..
integataao scteniloo de
Q-W*
X- graph
[B], [C)
Museum
(Whei mnccem upeand 100 name that -93- Nikoskei7)
KKanoll sp orspauaiee ccurnifqu reas Henaagee, Hanasupee)
(O musce Nakaskof", alria 201 2013-09-326 Nakadcelty
(le whut your was the Akita Stseam of opensdf) Which year
18 masse roay Bas etapae Xpaseceruuuaa agnell Auarut, emgger Nyanescmenmaa seyvell Aorrar))
(Eex que uno iheno Maseu de Aute Akial, O Maseu de ugarad sigan anel))
spensoff)
X lost
[B9]
[EN)
[BNe]
The Akina Maseum of
(Mysak suyyeeu Asara)
(Maacs de Aste do Akina 142 Nakalori)
senall,
odittro
ponazi
ana 3740 ds square netres?)
mooaas 3766,60 ancopernna serpoaf)
3746,65 netus qualinlos))
[10)
Av opan
(2013)
(28 corratge 2013 mos)
(2013)
(What the Akiaa Mascum Art insggariidd Wlich year was the monea (rangenned))
Gas esapert Xpacnceraouuaa Amaral,
[O Musca Arts Akinu inagaando há slgun anot, Qual ano fut isaggrrdoo museet )
sances may Out omagen sepeer))
Table 7: Q- @-WEBNLGen instances derived from WebNLG Data and translated into Portuguese and Russian to give D-WEBNLG"MRE and Q-WEBNLG** R* Enclosed letters refer to the triple/text above.
recognise questions that are unanswerable given the context. These samples are created using random (i.. 50-50) assignment to one of these two strategies:
is rejected, new (Xother's L' aXgener)) is drawn from M using another X. After 10 tries or when all X has been exhausted, simple nega- tive is created instead.
simple negatives for a given (X, 5, ax) instance are created by picking another (Xother, g', (xx)) instance in the QA training set where X and Xother do not share any common entities/values between them. If X is text, we use the graph that it is paired with in WEBNLG to check for common entities. A new instance (X, q', "unanswerable") is then created in the training data,
* Strategy 1:
Baseline: negative QA instances For training the basclinc's textual QA and KG QA models, we create unanswerable instances in a 2: ratio by ran- domly replacing the context for question with another in the data.
B.2 Upsampling
We carry out upsampling on two levels when preparing Q- WEBNLG and Q- O-WEBNLG*t (the training data for our language specific multimodal multi- QG-QA models).
Strategy 2: hard negatives are created in the following manner: mapping M is first created ahead of time where every entity that can be found in the training data is associated with the set of all the (X, q, ax) instances where is mentioned in X (e E X). If X text, we use the graph is paired with in WEBNLG when creating M.
Between modalities for QG subtasks and be- tween modalities for the same task For QG this is done between complex textual and complex KG QG, simple textual and simple KG QG to ensure that the QG subtasks are balanced. It is also done between modalities for the QA tasks (e.g. textual QA and KG QA) to ensure that the tasks are bal- anced between modalities. The negative samples for the QA tasks are also upsampled in the same way.
For a given (X, 9. (x) instance, another instance, i.e. (Xother, d', ax is randomly chosen using M and a random e E X. If a single token from the answer a.eaeeer overlaps with (i) any of the tokens for any e (an entity or value) found in X or (ii) any of the tokens of X, then this (Xother> d. ).aaeer) candidate is rejected. Otherwise, a new instance (X,d. "unanswerable"" is created in the train- ing data using the instance and the process for (X, g, ax) terminates. When a candidate instance
For instance, my and m2 are the sets of samples for modality I and modality 2 respectively, and suppose m, m2 |.

---

If m2 // m! | 0.5, we randomly sample mi 1-1 m2 from m2 balance them.
mGEN for KG QA. The question in the input is in Portuguese/Russian and the context in English; the output answer is in English.
however m2 mi < 0.5, we upsample my up to at most 1/3- my This is to ensure that we do not overrepresent my in the data and overfit on it during training.
Globally for certain tasks This was done for simple QG as a whole (i.e. after simple textual and simple KG QG have been consolidated as one). This is because they are significantly less instanti- ated samples of this task in the data than the rest, The number of samples for these tasks were tripled.
B.3 Evaluation settings
BERTScore We the same settings, except the following for clearer analysis: (i) lower- casing, (ii) "unanswerable" strings were to an empty string to avoid non-zero BSc for these and "over-counting" them, and (iii) rescaling BSes against a baseline (computed following the official method for a wider spread.
ceff-consistency filter Since the models are trained on machine-translated synthetic data, some generated questions may be ill-formed and pose an impact on QA. We therefore filter from both our model and the baseline, the questions: (i) which cannot be answered from their source context; or (ii) whose generated answer åx has a BSc 0.7 when compared against the reference ax. We fo- analysis for QA Consistency (Section 4), the Data- QuestEval (Section 7.4) and mGEN (Sec- tion 7.5) evaluations on the results after filtering as this the upper bound of the approaches' per- formance. For congruence with our QG Coverage analysis, if the filtering will leave a given approach A - and therefore B as well with no QA pairs (i.c. coverage), we keep one QA pair for A.
Settings for mGEN experiments Since mGEN was trained with language tokens to produce an- swers in different languages for question in lan- guage I., we leverage these tokens to replicate the same language setting for the inputs and outputs experimental set-up: (i) when the answer- ing modality is text, the input (both question and context) are in Portuguese/Russianand the output answer is in the same language. (ii) When the answering modality a graph, we use the lineari- sation scheme from (Oguz et al., 2022) to utilise
12 naster/jounaa//recaaeebbbaallli

---

Detailed results
C.2
Finer-grained QA consistency tests
C.1 QA consistency: Token and Exact
Portuguese
Match
Num Facts
2
3
Portuguese
Same-mod (GT)
Internal
X-Appr
T- T
93.6
%6.0
95.5
95.4
QG QA
Baseline
QTT
Baseline QTT
QTT
G G 95.7
98.1
97.8
98.5
Baseline
QTT
Baseline
Cross-mod (GT)
Same-mod (GT)
T T- G
65.6
62.5
58.2
54.6
T-
(2. 94.9
25)
77.6 12.3)
=0.00)
67.9 27.0)
(=0.6)
G G T
67.1
66.0
61.0
57.0
G
G (0.00) 30)4
56.6
23.0)
52.2 -2.21
T Cross-mod (Gen Ans)
(20.11)
(20.8)
59.3
55.7
G T
69.2 67.0
61.1
57.4
Russian
Table 10: Fine-grained analysis of QTT's QA perfor- mance (BSc) for Portuguese. Num Facts denote the number of facts (nzf) the set of QA-pairs relate to (i.e. denotes an SQ of fact, 2 denotes a CQ of 2 facts etc.... The nf sets are mutually exclusive.
Internal
X-Appr
QG QA
Baseline
QTT
Baseline QTT
QTT Baseline
Baseline
QTT
Same-mod (GT)
T T-T T (00 (4.7(40.14) (-33.)
42.5
42.2)
(40.06)
10.11)
G G 72.6(20.05) (5.2 61.2 (+0.15)
-114)
48.9
46.3)
(#0.58)
Russian
Table 8: Consistency Results. Average of Token Fl between answers. In the first brackets () the standard deviations across 5 random runs; for the right column, in the second brackets() are the difference between X- Appr and Internal.
Num Facts
2
3
4
Same-mod (GT)
T-1 T 90.2
90.6
90.6
86.0
G G 93.8
97.6
98.0
96.8
Cross-mod (GT)
T- G
29.8
30.1
30.8
29.5
Portuguese
G T
29.6
29.3
30.0
31.0
Internal
X-Appr
Cross-mod (Gen Ans)
T G
30.8
30.3
30.9
31.6
QG
Baseline
QTT
Baseline QTT
QTT
G
32.8
29.5
30.7
30.9
QA
Baseline
QTT
Baseline
Same-mod (GT)
Table 11: Fine-grained analysis of QTT's QA per- formance (BSe) for Russian. Num Facts denote the number of facts (nf)) the set of QA-pairs relate to (i.e. denotes an SQ of fact, 2 denotes a CQ of 2 facts etc..... The nff sets mutually exclusive.
(2.5() (7.5
(8.1(-14.4)
58.1 (-14.4
43.2 (3.2(-4.35 44.3)
(40.16)
(0.21)
G - G (3.2 87.2
48.9 48.9-244.3
24433 24.3)
(=0.21)
41.9
(60.26)
-46.3
Russian
Internal
X-Appr
QG QA
Baseline
QTT
Baseline QTT
QTT
Baseline
QTT
Bascline
Same-mod (GT)
T
58. (8.. 66.0
33.8
33.gf-24 24.5)
20.12)
36.6
(9.44) 29.4)
-0.121
G - G
10) 92.9(10.52)
39.3
-
(40.14)
21.8)
33,9
(=0.74)
59.00
Table Consistency Results. Average Exact Match between answers. In the first brackets () are the standard deviations across random runs; for the right column, in the second brackets() are the difference between X- Appr and Internal

---

C.3
Multilingual Retrieval-based QA
Portuguese
External Retrieval-QA
QG QA
Baseline mGEN
Ours mGEN
Self (GT)
T- T
(3.102))01)
75.8,
G
(1.3
70.0
Russian
External Retrieval-QA
QG
Baseline mGEN
Ours mGEN
QA
Self (GT)
T T
(10. 19.4(10.17)
17.
G-G
(7.2
Table 12: QA consistency (Token F1) comparing the QG of Baseline and Ours, using mGEN for QA.
Portuguese
External Retrieval-QA
QG
Baseline mGEN
Ours
QA
mGEN
Self (GT)
T
12. 45.2
(3.5
G-G
59.3(20.74)
Russian
External Retrieval-QA
QG QA
Baseline mGEN
Ours
mGEN
Self (GT)
T
36.4(10.18)
10.
G G
45.8 45.8 (0.
58.8(11.05)
Table 13: QA consistency (Exact Match) comparing the QG of Baseline and Ours, using mGEN for QA.


Reviews:
[{"review_id": "53LS2nf4hU", "reviewer": "Reviewer_zbSx", "summary": "", "questions": "A: Figures 2 and 3, do the authors check the quality of these generated questions?\n\nB: Table 4 and the explanation doesn't seem to align.  Line 431: \"... however, our approach’s stronger performance in all the Internal, X-mod, and X-Appr comparisons suggest that the former is the likely factor.\" but in Table 4 X-appr that doesn't seem to be the case. Is there perhaps a problem with the heading order?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and engaging with us to discuss our work. \n\n__Novelty of our approach__\n\nHowever, we respectfully disagree with your view that our work is not novel for the following reasons: \n\n- Our work provides an approach for generating synthetic data in two languages where such QA/QG training data that is aligned between text and graph modality does not exist. One of the languages is Brazilian Portuguese which is under-resourced. \n\n- This synthetic data enables the joint multi-modal multi-task training we propose, one that gives a single multi-modal multi-task QG+QA model (Section 6, page 4) that is significantly different from the approach in Rebuffel et al 2021 (that requires multiple models, i.e. one for each modality and task) which we derive the baseline from. \n\n- Importantly (for reasons we note in Line 077, page 1), our work – which is independent of theirs – departs significantly from the Rebuffel approach by also being able to maximise coverage (Line 275, page 4 and Line 319, page 5) via training to generate multiple questions from the same input in a single step. \n\n- Finally, to overcome the lack of any gold QA/QG data that is aligned between text and graph modality in these two languages, we designed our evaluation suite to include multiple internal, cross-modal and cross-approach checks (Line 341, page 6) to validate the performance of each approach. Note that we also show (Line 441, page 7) that these gains translate to improved correlations with human judgements over the baseline when used to compute the Data-QuestEval metric. \n\n__Question A: Quality of generated questions__ \n\nNone of the authors are speakers of these two languages, however, we wish to point out that our comparisons in Section 7.3 (particularly the cross-approach ones; page 6) give an indication of the quality of the questions generated by our approach over the baseline – see our discussion in Line 425 (page 7). \n\nFurthermore, we have also included additional experimental results in our response to Reviewer HqCQ; in summary (for details, please refer to the tables and the _“Additional experimental validation of our approach”_ section in our response there), we use another SOTA model for QA on the questions generated by baseline and our approach. The results there show that questions generated by our approach consistently lead to better QA accuracy compared to those generated by the baseline.\n\n__Question B: Understanding X-Appr results__\n\nThe Table 4 (page 7) __X-Appr__ headings are correct. To interpret the X-Appr performance between baseline and our approach, we have to take the difference of the __X-Appr__ BSC score with the __Internal__ BSC score. This will give a fairer comparison since the QA performance is different between baseline and our approach. \n\nFor instance, in the first row of Table 4 (page 7), the 73.6 BSC from QG (Baseline)+QA (Ours); T→T has to be compared against the QG (Baseline)+QA (Baseline); T→T setting on the left, giving the difference of -12.7 (in superscript brackets and in blue font).\n\nAs seen from the table, whenever our QA is used (no matter the modality combination, or GT/Gen. Ans), the drop in QA accuracy is lower (in fact in half of the cases, there is even a gain). \n\nBased on your feedback, we agree that the table layout for the __X-Appr__ column can be improved and will modify it for the camera-ready to show the score difference in larger font instead of the raw BSC scores. \n___\n\nWe hope that these clarifications address your questions and concerns under Reasons to Reject, and you will consider raising your score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is the first one to attempt to do so in a multilingual setting.\n\nIt is shown to outperform the baseline technique (Rebuffel et al.'s) in some metrics.", "reasons_to_reject": "The paper is an improvement to Rebuffel et al.'s work, but the approach is not novel and it comprises reused components from previous work.\n\nSome results are confusing to follow. Some need further discussion to clarify the impact.", "questions_to_authors": "A: Figures 2 and 3, do the authors check the quality of these generated questions?\n\nB: Table 4 and the explanation doesn't seem to align.  Line 431: \"... however, our approach’s stronger performance in all the Internal, X-mod, and X-Appr comparisons suggest that the former is the likely factor.\" but in Table 4 X-appr that doesn't seem to be the case. Is there perhaps a problem with the heading order?", "paper_topic_and_main_contributions": "The paper bridges QA and QG with multimodalities (text and graph) in a multilingual setting (Brazilian Portuguese and Russian).", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "`Figure 2` and `3`: be consistent either `Baseline` or `Rebuffel`. Fix the colors as well so the legend and the graph match; there seems to be some transparency issue."}, "extra_text": {}}, {"review_id": "abPAdottV4", "reviewer": "Reviewer_URPa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback on our work. We wish the provide the following response to your reasons to reject: \n\n__Experimental results and necessity of various modules__ \n\nWe take your (and Reviewer HqCQ’s) feedback into consideration and include additional experimental results. These include (i) an additional external validation of both baseline and our approach’s QG performance by using the mGEN reader from the CORA (Asai et al, 2021) state-of-the-art cross-lingual retrieval QA system, as well as (ii) additional token F1 and Exact Match scores. These results are consistent with the reported results in our submission and further show the gains from our approach’s over the baseline. For these results, please refer to the tables we provide in our response to Reviewer HqCQ. \n\nSince there is no cross-modal QG/QA data for Brazilian Portuguese and Russian, all of the modules are necessary for producing and ensuring the quality of the synthetic QG/QA data in these languages. The main modules (represented by the rectangle blocks in Figure 1, page 4) and their purpose are as follows: \n\n- _Off-the-shelf QG and QA models_: these are necessary to obtain an initial set of synthetic questions and answers (Line 173, page 3), as well as provide a check on answerability and consistency before using the initial data to train the intermediate controllable QG models (Line 209, page 3 and also see next point).\n\n- _Two controllable QG models that we trained (Line 194, page 3)_: these are also necessary because it is through controllable generation using them that we can significantly increase the QA/QG training data coverage (Line 207, page 3). \n\n- _Brazilian Portuguese and Russian machine translation and off-the-shelf QA (Line 226, page 3)_: these are necessary in order to obtain the questions as well as answers (and texts for Portuguese) in these languages. Note that since we are machine translating the questions from English, the semantics of the questions might be affected, which is why we posed the translated question to the PtBr/Ru TextQA, and verify the generated answer alignment to the original graph answer (Line 238, page 4). \n\n__Main research problem__\n \nOur main research problem is to be able to generate and answer questions across multiple modalities, from graph and from text, and to be able to do so for multiple languages besides English  (Line 044, page 1). To be useful, the approach has to be robust to surface variations, perform QA with accuracy and consistency, as well as generate wide QG coverage (Lines 056, 065 and 077, page 1). Note also that our work is on languages for which there is no available QA/QG training data that is aligned between the modalities. As we state in Introduction (Line 029, page 1), being able to do so can be particularly useful for verifying the consistency of information between modalities. As we also state in our Ethics Statement (Line 616, page 9), the areas where such verification can assist in include automatic fact verification (e.g. against a curated KG) as well as quality estimation of machine generated text from structured information (KG information), which are increasingly important as the prevalence of machine-generated is expected to increase. With the additional page available for the camera-ready, we can state these points more clearly in the abstract and Introduction. \n\n___\nWe hope that these clarifications address your concerns under Reasons to Reject, and that you will take them into consideration and raise our score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper proposes a framework for training multilingual, multimodal (natural language and graph), multi-task (QA and QG) models. This is important as it provides a framework for languages with less resource.\n- A well-thought framework that incorporates various modules of different purposes.", "reasons_to_reject": "- Experimental results are a bit lacking. For example, are all the modules necessary?\n- The presentation of the paper could be improved a lot. See comments below.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a framework for training multilingual, multimodal (natural language and graph), multi-task (QA and QG) models. The authors leverage synthetic data generation and machine translation to produce graph-text aligned QG-QA data in Portuguese and Russian. Experimental results show that the proposed method improves question coverage, QA consistency, and correlation to human judgements.   ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "### Comments\n\n- Line 55, I would suggest ending this sentence with a colon without starting a new paragraph.\n- The abstract, Section 1 and 2 are pretty hard to follow. What is the main research problem? Why is it important? I think the authors jump into the details too early and miss the opportunity to present the big picture of the paper.\n- Since the framework consists of multiple tasks and modality, providing some examples would help readers understand better.\n\n### Proofreading\n\n- Line 552, are"}}, {"review_id": "1mK4qJg5Pv", "reviewer": "Reviewer_HqCQ", "summary": "", "questions": "1. In the process of data generation, it is mentioned that the proposed model only contains one lexical representation of each knowledge graph context. This may result in multiple lexical representations occurring in the text being ignored. For future research, is it possible to consider integrating the KELM dataset with the generation results of the same subgraph generated by other language models to expand the range of lexical representations covered?\n\n2. When constructing KG QA training samples, how do the authors choose quality or quantity? It is recommended to give comprehensive examples for showing the new datasets.\n\n\n3. Are there more methods for comparison？as the baseline comparison results given in the article are very limited. So, adding comparison methods can improve the persuasiveness of the article.\n  4. In the experimental setting, the author chose quality instead of quantity, which is equivalent to biasing the experiment to their favorable situation, so when the number of experiments increases, I am curious whether the proposed method is scalable and robust.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and comments, we appreciate the opportunity to engage with you to discuss our work. To address your comments and questions:\n\n__Additional experimental validation of our approach__\n\nRegarding our evaluation approach, we wish to highlight that the results have to be seen holistically (i.e. (i) question coverage, (ii) self, cross-modal and cross-approach QA consistency, as well as (iii) downstream usage for the Data-QuestEval metric). Holistically, our approach’s consistently better performance provides strong support for the improvements that it brings over the baseline (one that we independently replicated from a recent approach with modifications for our multilingual setting). \n\nNonetheless, we appreciate your (and Reviewer URPa’s) feedback on this aspect and have run further evaluation to provide additional validation for our approach. We do so in two ways:  \n\n- Firstly, we extend our Section 7.3 (page 6) analysis by posing the questions generated by baseline and our approach to mGEN, a textQA model that is part of the CORA (Asai et al, 2021) state-of-the-art cross-lingual retrieval QA system (see below for details of experimental setup). This set of experiments can be seen as a second external validation (Line 425, page 7) of the QG quality (i.e. besides the cross-model X-Appr results in our submission) of baseline and our approach.\n\n- Secondly, we also add F1 and EM scores (see Tables 2 and 3 below) for the __Internal: Self (GT)__ comparison the same set of questions and answers as Table 4 of our submission (page 7), i.e. these new results will be placed next to the BSC figures we already reported. Since the __X-mod__ comparisons are cross-lingual and the F1 and EM would not be informative, we left these out. \n\nThe additional mGEN experiments confirms our findings in the submission (Line 425, page 7); we observe meaningful increases (at least 2.6 and up to 15.1 BSC) in answer accuracy of our questions compared to baseline: \n\n> Portuguese: \n> - __+6.6__   __BSC__ \tfor QGT -> QAT  (72.8 vs 57.7)\n> - __+15.1__ __BSC__ \tfor QGG -> QAG (78.0 vs 71.4)  \n\n> Russian:\n> - __+2.6__   __BSC__ \tfor QGT -> QAT  (68.5 vs 65.9)\n> - __+9.1__   __BSC__ \tfor QGG -> QAG (76.0 vs 66.9)\n\nFurthermore, our approach similarly outperforms baseline on (i) EM for all comparisons; and (ii) almost all cases in Token F1 (except for T -> T in Russian). These consistently strong findings with a second (alternative) QA approach should suffice in further demonstrating our approach’s substantial strength over the baseline. \n\n__Limitations and practical implications/applications__ \n\nWe wish to highlight that we do discuss the most pressing limitations and potential biases of the data in the Limitations section (Section 10, e.g. Line 555) and Ethics Statement (Section 11, e.g. Line 625, where we include an example of potential risk). We also discuss the relevance and applicability of our work in our Introduction and Ethics Statement (please see Lines 021 to 038, and 612). With the additional page for the camera-ready, we can raise these more prominently in the main body of the paper. \n\n__Responses to Questions__\n\n__Q1__: This will likely increase the lexical variety available in our synthetic QA/QG training data, but will however require additional resources to obtain, for e.g. some fine-tuning as was done in the original KELM paper, or possibly with few-shot in-context prompting on recent instruction-tuned models.\n\n__Q2__: We wish to highlight that we did provide example instances from the Q-WebNLG dataset, please see Table 6 in the appendix (page 13). To give an illustration of the rationale for this step, for e.g. a piece of text may be “The movie Z stars X and Y.” and its KG graph would be [ < X, stars in , Z >, < Y, stars in, Z > ]. In our Step 3 (Line 202), the set of possible answers for text are the noun phrases and named entities found in a given text, therefore one of the answers in this case could be “X and Y”, and a complex question that is generated from this answer and text might be “Who are the stars of Z?”, whose answer does not map to a single KG graph. Although we seek to maximise coverage available for training (Line 275), if we had simply used the questions generated from text (and their answers) as training instances for our KG QA, our model – which is generative – would learn to wrongly produce such answers for KG QA that would also “unfairly” bring them closer to the Text QA answers. \n\n__Q3__: As discussed above, we take your feedback into consideration and have included additional experimental findings above. \n\n__Q4__: We respectfully disagree with your comment that our prioritisation of “quality over quantity” (Line 587) for our KG QA training data unfairly biases the experimental set-up in our favour. To clarify:\n\nAs our illustration for the answer to your Q2 shows, skipping this step would have the opposite effect, i.e. by steering our KG QA model towards generating Text QA-like answers for those cases, it will bias the cross-modal comparisons (i.e. X-mod (GT) and X-mod (Gen Ans) rows in Table 4, Section 7.3). Importantly, this will also likely negatively affect the reliability of the Data-QuestEval metric to accurately assess the presence of KG facts mentioned in a given text. \n\nIn fact, we view our taking of this step as a potential limitation of our current work because the number of training instances for KG QA is less compared to the training instances available for Text QA and we had to rely on upsampling to balance the data (Line 975; needed since we are training in a multi-task manner where all tasks are present in a single training batch). It is possible that with better balance in KG QA data in the first place, it could possibly lead to further gains on the KG QA performance.   \n\n\n___\nWe hope that these clarifications address your questions and concerns under Reasons to Reject, and that you will consider raising our score. \n___ \n___\n\\ \\\n\n\n__Tables and references__\n\n| | |  |  |  |  |  |\n|---|:---:|:---:|:---:|:---:|:---:|:---:|\n| Portuguese | External QA |  |  |  |  |  |\n| QG | Baseline |  |  | Ours |  |  |\n| QA | mGEN |  |  | mGEN |  |  |\n| Modality | BSc | F1 | EM | BSc | F1 | EM |\n|  | Self (GT) |  |  |  |  |  |\n| T -> T | 71.4 (0.07) | 73.1 (0.04) | 52.4 (0.13) | 78.0 (0.58) | 75.8 (0.60) | 63.5 (0.79) |\n| G -> G | 57.7 (0.13) | 56.3 (0.10) | 45.2 (0.15) | 72.8 (0.48) | 70.0 (0.39) | 59.3 (0.74) |\n|---|:---:|:---:|:---:|:---:|:---:|:---:|\n| Russian | External QA |  |  |  |  |  |\n| QG | Baseline |  |  | Ours |  |  |\n| QA | mGEN |  |  | mGEN |  |  |\n| Modality | BSc | F1 | EM | BSc | F1 | EM |\n|  | Self (GT) |  |  |  |  |  |\n| T -> T | 65.9 (0.13) | 61.7 (0.19) | 36.4 (0.13) | 68.5 (0.18) | 60.6 (0.17) | 42.2 (0.14) |\n| G -> G | 66.9 (0.15) | 59.4 (0.17) | 45.8 (0.20) | 76.0 (0.77) | 67.2 (0.82) | 58.8 (1.08) |\n\n_Table 1: Downstream QA consistency results using mGEN as QA on questions generated by baseline and our approach. Average of automatic scores (BERTScore, Token F1 and Exact Match); in subscripts are the standard deviations across 5 random runs._ \n\n| | | | | |\n|---|:---:|:---:|:---:|:---:|\n| Portuguese | Internal |  | X-Appr |  |\n| QG | Baseline | QTT | Baseline | QTT |\n| QA | Baseline | QTT | QTT | Baseline |\n| T -> T | 89.9 (0.02) | 94.9 (0.25) | 77.6 (0.09)(-12.3) | 67.9 (0.26)(-27.0) |\n| G -> G | 80.5 (0.04) | 90.4 (0.31) | 56.6 (0.11)(-23.9) | 52.2 (0.26)(-38.2) |\n| Russian | Internal |  | X-Appr |  |\n| QG | Baseline | QTT | Baseline | QTT |\n| QA | Baseline | QTT | QTT | Baseline |\n| T -> T | 86.7 (0.10) | 84.7 (0.14) | 53.1 (0.06)(-33.6) | 42.5 (0.11)(-42.2) |\n| G -> G | 72.6 (0.05) | 95.2 (0.27) | 61.2 (0.15)(-11.4) | 48.9 (0.58)(-46.3) |\n\n_Table 2: Consistency Results. Average of __Token F1__ between answers. In the first brackets () are the standard deviations across 5 random runs; for the right column, in the second brackets() are the difference between X-Appr and Internal._\n\n| | | | | |\n|---|:---:|:---:|:---:|:---:|\n| Portuguese | Internal |  | X-Appr |  |\n| QG | Baseline | QTT | Baseline | QTT |\n| QA | Baseline | QTT | QTT | Baseline |\n| T -> T | 72.5 (0.08) | 87.5 (0.61) | 58.1 (0.16)(-14.4) | 43.2 (0.21)(-44.3) |\n| G -> G | 73.2 (0.10) | 87.2 (0.36) | 48.9 (0.21)(-24.3) | 41.9 (0.26)(-45.3) |\n| Russian | Internal |  | X-Appr |  |\n| QG | Baseline | QTT | Baseline | QTT |\n| QA | Baseline | QTT | QTT | Baseline |\n| T -> T | 58.3 (0.23) | 66.0 (0.33) | 33.8 (0.12)(-24.5) | 36.6 (0.12)(-29.4) |\n| G -> G | 60.9 (0.16) | 92.9 (0.52) | 39.3 (0.14)(-21.6) | 33.9 (0.74)(-59.0) |\n\n_Table 3: Consistency Results. Average of __Exact Match__ between answers. In the first brackets () are the standard deviations across 5 random runs; for the right column, in the second brackets() are the difference between X-Appr and Internal._\n\n__Details for mGEN experiments__\n\n- mGEN was fine-tuned from a mT5-base checkpoint to generate the answer to a question given a collection of retrieved input contexts. We use the version (code and weights) that was released as part of the MIA-2022 Shared Task (Asai et al, 2022) https://github.com/mia-workshop/MIA-Shared-Task-2022/tree/main\n- The sets of questions posed to mGEN are the same as those for the results in our submission’s Table 4 (page 7), i.e. direct comparisons can be made between the additional results here and __X-Appr: Self (GT)__ results there.\n- Since mGEN was trained with language tokens to produce answers in different languages for a question in language L, we leverage these tokens to replicate the same language setting for the inputs and outputs as our experimental set-up:\n  \n   - When the answering modality is a text, the input (both question and context) are in PtBr/Ru and the output answer is in the same language. \n\n   - When the answering modality is a graph (see Line 391 of our submission), we use the linearisation scheme from (Oguz et al., 2022) to utilise mGEN for KBQA. The question in the input is in PtBr/Ru and the context is in English; the output answer is in English. \n\n__References__\n1. Asai et al, 2021: One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval\n\n2. Asai et al, 2022: MIA 2022 Shared Task: Evaluating Cross-lingual Open-Retrieval Question Answering for 16 Diverse Languages\n\n3. Oguz et al, 2022: UniK-QA: Unified Representations of Structured and Unstructured Knowledge for Open-Domain Question Answering", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses an important problem in the field by extending QG-QA research to multilingual and multimodal scenarios.\n\n2. The proposed method shows improvements over the baseline, suggesting its effectiveness in generating questions and providing answers in different languages and modalities.\n\n3. The use of synthetic data generation and machine translation techniques to align graph-text data is innovative and contributes to the development of QG-QA models.", "reasons_to_reject": "1. The evaluation of the proposed method could be further strengthened. It is recommended to provide more extensive experimental results, including additional metrics and comparisons against alternative approaches.\n\n2. The limitations and potential biases of using synthetic data generation and machine translation are less discussed in the current version. Further analysis of their impact and potential challenges is needed to ensure the reliability of the proposed method.\n\n3. The practical implications and applications of the proposed method for real-world scenarios should be discussed. Providing insights into its potential in the industry or specific domains would enhance the quality of the paper.", "questions_to_authors": "1. In the process of data generation, it is mentioned that the proposed model only contains one lexical representation of each knowledge graph context. This may result in multiple lexical representations occurring in the text being ignored. For future research, is it possible to consider integrating the KELM dataset with the generation results of the same subgraph generated by other language models to expand the range of lexical representations covered?\n\n2. When constructing KG QA training samples, how do the authors choose quality or quantity? It is recommended to give comprehensive examples for showing the new datasets.\n\n\n3. Are there more methods for comparison？as the baseline comparison results given in the article are very limited. So, adding comparison methods can improve the persuasiveness of the article.\n  4. In the experimental setting, the author chose quality instead of quantity, which is equivalent to biasing the experiment to their favorable situation, so when the number of experiments increases, I am curious whether the proposed method is scalable and robust.", "paper_topic_and_main_contributions": "This paper introduces a multilingual (Brazilian Portuguese and Russian) and multimodal (knowledge graph and natural language) approach in the fields of Question Generation (QG) and Question Answering (QA), which is an important advance in research in this field.\n\nUsing synthetic data generation and machine translation methods, the authors generated QG-QA data aligned with graphs and texts to train multi-modal, multi-task models, and realized the translation of knowledge graphs and natural language texts in Portuguese and Russian. QG and QA. This approach provides an effective solution for QA evaluation using knowledge graphs to generate multilingual texts.\n\nExperiments demonstrate that the proposed method outperforms a baseline approach derived from previous research on English and adapted to handle Portuguese and Russian.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "Well written, except for few minor errors."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an approach for multilingual question generation and question answering, comprising knowledge graphs and textual inputs. The paper is well positioned against the state of the art and can be seen as a (substantial, non-incremental) extension of Rebuffel et al.

The paper is solid and interesting. It addresses an important multilingual task and the model yields excellent results. Section 7 (Results) merits a special mention, as it discusses results from multiple different perspectives and offers a very good understanding of the system's behaviour. All the reviewers have appreciated the paper's "excitement".

However, the paper is extremely hard to read. It addresses three phenomena (multilinguality, kg/nlp and qg-qa) and thus offers a rather complex architecture with multiple rationales/subtasks/modules/.. I (AC) have some difficulty understanding individual parts -- and all the reviewers, I believe, have encountered the same: all the three reviewers have asked multiple clarification questions (addressed thoroughly by the authors in the rebuttal). I believe therefore that the presentation issue should be addressed, following very constructive comments: put more emphasis on the bigger picture (reviewer URPa), highlight use-case/impact (reviewer HqCQ), have a more principled discussion of modules and their roles (reviewers zbSx, URPa) etc.
Decision: Accept-Findings