Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.
Digital Object Identifier 10.1109/ACCESS.2022.0092316

Content Order-Controllable MR-to-text
KEISUKE TOYAMA (Graduate Student Member, IEEE), KATSUHITO SUDOH, and SATOSHINAKAMURA (Fellow, IEEE)
Nara Institute of Science and Technology

Corresponding author: Keisuke Toyama (e-mail: toyama.keisuke.tb5@is.naist.jp).
Part of this work was supported by JSPS KAKENHI Grant Number JP21H05054.

ABSTRACT Content order is critical in natural language generation (NLG) for emphasizing the focus of agenerated text passage. In this paper, we propose a novel MR (meaning representation)-to-text method that
controls the order of the MR values in a generated text passage based on the given order constraints. We
use an MR-text dataset with additional value order annotations to train our order-controllable MR-to-text
model. We also use it to train a text-to-MR model to check whether the generated text passage correctlyreflects the original MR. Furthermore, we augment the dataset with synthetic MR-text pairs to mitigate thediscrepancy in the number of non-empty attributes between the training and test conditions and use it to train
another order-controllable MR-to-text model. Our proposed methods demonstrate better NLG performance
than the baseline methods without order constraints in automatic and subjective evaluations. In particular,
the augmented dataset effectively reduces the number of deletion, insertion, and substitution errors in thegenerated text passages.
INDEX TERMS Controllable Text Generation, Data Augmentation, Data-to-Text, Meaning Representation, Natural Language Generation

I. INTRODUCTION

ATA-to-text tasks generate a text passage from such(semi-)structured data as concepts, tables, graphs, etc.
Such input data usually include multiple elements that will
be mentioned in the output passage (Table 1), meaning that
we can control the text generation’s structure, length, andword order by content planning. This work focuses on theorder of the contents, i.e., the data values, since the order iscrucial for data-to-text that emphasizes important words orphrases. Here we explain two examples of the importance ofcontrolling the content order.

TABLE 1. Example of E2E dataset

D(1) Preserving the order to avoid changing the focus
of a text passage: Suppose we replace the value “Italian”
of the food with “French” in Table 1. In this case, theideal text passage becomes “Wildwood is a restaurant thatserves French food located near Raja Indian Cuisine in thearea of riverside. Unfortunately, it is not kid friendly.” In
this passage, the word “Italian” is changed to “French,”
although the content order is unchanged. In the reference
text passage and the ideal text passage, the “no” of thefamilyFriendly attribute is emphasized by placing an
independent sentence at the end of the text passage: “Unfortunately, it is not kid friendly.” However, if we cannot control
the order of a data-to-text system, it may generate a text
VOLUME 10, 2022

MR

Text

Attribute
Value
name
Wildwood
eatTyperestaurant
food
ItalianpriceRange(empty)customer rating (empty)areariversidefamilyFriendlynonear
Raja Indian Cuisine
Wildwood is a restaurant that serves Italian food located near
Raja Indian Cuisine in the area of riverside. Unfortunately, it is
not kid friendly.

passage with an unintended order: “Wildwood is a restaurantthat is not family friendly. It serves French food located near
Raja Indian Cuisine in the riverside.” This text passage does
not emphasize the “no” of the familyFriendly attributebecause of the location of the phrase “that is not family
friendly” in the middle of the sentence.
(2) Controlling the order to change the focus of a text
passage: Suppose we emphasize the value “riverside” ofthe area attribute and “Raja Indian Cuisine” of the near
attribute in Table 1. In this case, these values should appear at
the beginning of the passage: “In the riverside area near Raja
1Toyama et al.: Content Order-Controllable MR-to-text

Indian Cuisine, you can visit a restaurant called Wildwood
where Italian food is served. It is not family-friendly.”Concerning such a content order problem in data-to-text,
Kasner et al. [1] proposed a method that generates a text
passage from resource description framework (RDF) triples
in a zero-shot setting. RDF triples are converted using atemplate to a set of sentences, which are ordered to maximize their coherency by sentence ordering. However, thismethod automatically determines the sentence order without directly controlling the order using explicit constraints.
Leng et al. [2] proposed a data-to-text method that controls
sentence splits, the entity order, and the text length. They
controlled the entity order by aligning the input data with thereference text. However, their method’s performance in suchstandard automatic measures as BLEU [3], ROUGE_L [4],and METEOR [5] was lower than their baselines. Su etal. [6] proposed a data-to-text method that first predicts thesuitable order of attributes from the data input and thengenerates output sentences from both the data input andthe predicted order. Their method explicitly outputs ordering
information as a content plan so that the order can be edited
arbitrarily. Their human evaluation showed that the generated
sentences accurately reflected the ordering information if theorder was unedited. However, when the order was randomly
shuffled, the accuracy was worse than without shuffling.
Luo et al. [7] proposed a few-shot table-to-text generation
method using a pre-trained language generation model. To
generate sentences, the prompt “summarize the following
table:” followed by the order of attributes that appear in thegenerated sentence and the input data are given to a pretrained generation model. However, their human evaluation
showed that the word order’s accuracy was 0.84, which is
insufficient.
In this work, we propose an MR-to-text (MR2T) methodthat controls the order of the MR values in generated text
passages with additional value order annotations using an
MR-text dataset called the E2E refined dataset [8]. First,
we train an order-constrained MR2T model with a textto-MR (T2MR) model. Then we augment the MR-text
dataset to obtain a better-balanced distribution in terms ofthe number of non-empty attributes by synthesizing theattribute-value pairs that do not appear in the dataset. We
investigate the performance of an MR2T model trainedusing the augmented dataset by automatic and subjective
evaluations and show that it can control the order of theMR values with high accuracy. Our code is available athttps://github.com/KSKTYM/content_order-controllable_mrto-text.
II. DATASETThe E2E dataset [9] used in the E2E NLG Challenge [10] iswidely utilized in MR2T studies. It consists of a set of pairsof a British English text passage and a corresponding MR
with the following eight attributes in a restaurant recommendation domain: name, eatType, food, priceRange,customer rating, area, familyFriendly, and
2

TABLE 2. Example of E2E refined dataset

Attribute

Value
Order
NAME
1
(WILDWOOD)
eatTyperestaurant2
food
Italian3
priceRange(empty)0
customer rating (empty)
0
areariverside5
familyFriendly
no
6
NEAR
near
(RANA INDIAN
4
CUISINE)
NAME(WILDWOOD) is a restaurant that serves Italian foodlocated near NEAR(RAJA INDIAN CUISINE) in the area ofriverside. Unfortunately, it is not kid friendly.
name

MR

Text

TABLE 3. Number of training data in terms of non-empty attributes: NEA
denotes non-empty attributes.

# NEA
1
2
3
4
5
6
7
8
Total

Original data
55
393
2,910
8,119
12,442
10,0585,393
1,190
40,560

Augmented data
0
7
926
4,323
0
2,384
7,049
11,25225,941

Merged data
55
400
3,836
12,442
12,442
12,442
12,442
12,442
66,501

near. However, some of its MR-text pairs suffer from thefollowing errors:deletion: some attribute-value pairs are not mentioned
in the text;• insertion: some empty attributes are mentioned incorrectly with unintended values;• substitution: some MR values are replaced by wrong
ones.
•

Such errors must be fixed to properly control a text passage’s content by MR2T. Although updates have rectified
some of these errors [11] [12], the updated datasets still
include such errors. We refined the E2E dataset with additional error fixes by manual annotations of the MR values
and made it public as the E2E refined dataset1 [8]. It also
includes the order of the MR values in the corresponding text
passage (Table 2). Here all of the name and near values aredelexicalized, because they appear as-is in the text passages.
We used the E2E refined dataset in this work to train the
MR2T models that control the order of the MR values.
III. DATA AUGMENTATIONThe number of training instances with non-empty attributes
is shown in the second column of Table 3. For example, thenumber of training instances with eight non-empty attributes
is only 1,190; that with five non-empty attributes is 12,442.
Such an imbalanced training data distribution causes poor
1 https://github.com/KSKTYM/E2E-refined-dataset (distributed under theCC 4.0-BY-SA license)
VOLUME 10, 2022

Toyama et al.: Content Order-Controllable MR-to-text

TABLE 4. All variations of MR values in E2E refined dataset

Attributename# variations
1

eatType

4

food

11

priceRange

7

customer rating

7

area
familyFriendly
near

3
3
2

MR values (delexicalized)
NAME(empty), coffee shop, pub,
restaurant(empty), American, Canadian,Chinese, English, fast food,French,Indian,Italian,Japanese, Thai
(empty), £20-25, cheap, expensive, less than £20, moderate, more than £30(empty), 1 out of 5, 3 out of 5,
5 out of 5, average, high, low(empty), city centre, riverside(empty), no, yes(empty), NEARTABLE 5. Number of every possible combination of MR values and MR
orders and obtained samples: NEA denotes non-empty attributes.

# NEA
1
2
3
4
5
6
7
8
Total

MR
value
1
30
355
2,142
7,096
12,912
11,952
4,320
38,808

MR
order
1
2
6
24
120
720
5,040
40,320-MR
value & order
1
60
2,130
51,408
851,520
9,296,640
60,238,080
174,182,400
244,622,239

Obtained
samples
0
11
1,167
30,949112,139
206,353
191,21269,120610,955

performance for instances with many non-empty attributes,
as shown later in the experimental results.
Data augmentation is a promising approach for mitigatingsuch data imbalance. Existing studies on data augmentation for NLG use text generation and text analysis models.
Kedzie et al. [13] used noise injection sampling. First, they
synthesized the under-represented MR in the training data.
Second, they converted the MR to a text passage using their
MR2T model by injecting Gaussian noise into the decoder
hidden states. Then they obtained an MR using their MR
parser. Finally, a pair of the obtained MR and the generated
text passage was accepted as augmented data. Unfortunately,noise injection sampling made insertion and deletion errors
in the generated text passage.
Chai et al. [14] proposed a feedback-aware self-training
method for their conditional text generation. First, they generated a text passage whose condition is different from theoriginal. Then a classifier predicted the condition from thegenerated text passage. A condition-passage pair was used
as augmented data if the input and predicted conditions
matched.
We applied an idea that resembles Chai’s approach toaugment the training data with synthetic examples generated
in the following steps.
Step 1

Generate every possible combination of MR values and orders. The number of MR value patterns is calculated fromVOLUME 10, 2022

the variation of the MR values (Table 4). The number ofMR order patterns is a factorial of their non-empty attributes.
Since the number of generated combinations is enormous(244.6 million), we randomly sampled them to 16 patterns
for an MR order, removed the MRs included in the original
dataset to avoid data leakage, and obtained 610,955 MR
combinations (Table 5).
Step 2

Convert the MRs obtained in Step 1 to text passages by the
MR2T model trained using the original training data. The
model’s details are explained in Section IV-D.
Step 3

Convert the text passages obtained in Step 2 to MRs using
the T2MR model trained using the original training data. The
T2MR model is also explained in Section IV-D.
Step 4

Augment the training data with the pairs of an MR and a text
passage as synthetic MR-text pairs when the result of Step 3
matches the MR generated in Step 1. Here our motivation
is to balance the data distribution. We sampled the pairs
for a total of 12,442 (the maximum number in the original
training data with five non-empty attributes) for each nonempty attribute (Table 3). Finally, we obtained 66,501 MRtext pairs as the augmented training data.
IV. METHOD

Next we explain the MR2T methods used in this work:
TGEN [15], JUG [16], Transformer-based MR2T, and theT2MR methods based on Transformer.
A. TGENAs our baseline, we use TGEN2 [15], based on an LSTMbased sequence-to-sequence model with an attention mechanism. TGEN is also used as the baseline for the E2E NLG
Challenge [10]. Note that TGEN does not constrain the orderof the MR values in its text generation.
B. JUGAs another baseline, we use JUG3 [16], based on an LSTMbased generative model for joint natural language understanding (NLU) and generation, which couples NLU andNLG through a shared latent variable. According to [16], the
BLEU score for the original E2E dataset was better than thatof TGEN. This method does not constrain the order of theMR values in its text generation either.
C. TRANSFORMER WITHOUT ORDER CONSTRAINTS

We use Transformer [17] for MR2T, configured as shownin Fig. 1. The Transformer-based MR2T model takes a sequence of MR values as its input tokens and generates output
2 https://github.com/UFAL-DSG/tgen (distributed under the Apache License 2.0)
3 https://github.com/andy194673/Joint-NLU-NLG

3

Toyama et al.: Content Order-Controllable MR-to-text

D. TRANSFORMER WITH ORDER CONSTRAINTS
output token

We propose another Transformer-based MR2T modelthat takes the content order constraints. For example, ittakes [“<sos>”, “NAME”, “restaurant”, “Italian”, “NEAR”,“riverside”, “no”, and “<eos>”] as input, where MR-value
tokens appear in the corresponding order with their mentionsin the text passage. The order-constrained T2MR model also
has identical Transformer architecture, although it is trained
to predict the MR values in the order of their mentions in
the input text. We train these models with the original andaugmented training datasets.

linear
add & norm
feed forward

add & norm

add & norm

feed forward

multi-headattention

add & norm

add & norm

multi-headattention

multi-headattention

N layer

V. EXPERIMENTS
N layerpositionembeddingtoken value
embeddingtoken value
embeddingpositionembeddingpositionvaluesinput tokens

outputtokenspositionvaluesFIGURE 1. Transformer model

tokens one by one. For example, it takes [“<sos>”, “NAME”,“restaurant”, “Italian”, “riverside”, “no”, “NEAR”, and“<eos>”] as the input tokens when the MR shown in Table 2
is used. Here each MR value is treated as one token without
further tokenization into subwords. “<sos>” and “<eos>”are special symbol tokens that express a sequence’s start andits end. The MR-value tokens are given in a fixed order fromname to near, and empty values are excluded from the
input. Note that since all the MR values are unique (Table 4),
the input tokens do not need to include MR attributes. For the
positional values, we use [0, 1, ..., n + 1] (where n equals thenumber of non-empty attributes) as the input vector. These
values are embedded with a trainable embedding.
For T2MR, we use Transformer with the same structure
as that for MR2T. It takes a sequence of text tokens as
input and predicts the MR values one by one in the fixedorder of the attributes. We use the word_tokenizer
module in the Python NLTK library for the text tokenization. For example, it takes [“<sos>”, “NAME”, “is”, “a”,
“restaurant”, ..., “kid”, “friendly”, “.”, and “<eos>”] as input
to induce the corresponding MR value sequence as output: [“<sos>”, “NAME”, “restaurant”, “Italian”, “riverside”,“no”, “NEAR”, and “<eos>”].
For the inferences, MR2T runs greedily to generate a text
passage, and T2MR predicts its MR values. If the T2MR
result matches the original MR, the generated text passage
is deemed reliable. If the MRs are not identical, MR2T
generates text passages using beam search (width = 5) andapplies T2MR to check whether the result is reliable andchooses the reliable one with the best score.

4

We investigated the effect of the order constraints in MR2T
by the following experiments.
A. DATAWe used the E2E refined dataset with 40,560 MR-text pairs
for training and 4,555 pairs for evaluation. We also used theaugmented training data (66,501 MR-text pairs) to train theTransformer with an order constraints model. To investigate
the MR2T performance with different MR orders, we augmented the test data by reordering the MR values and usedthem as inputs. We sampled four different orders for thetest data with three or more non-empty attributes and thealternative MR value order for those with two non-empty
attributes and obtained 21,140 additional test instances. We
then used four random seeds for the test data augmentationand obtained four sets of augmented test data (21,140×4sets). Hereafter, we call them the reordered test data.
B. MODEL CONFIGURATIONWe set the embedding vector size to 256, the feed-forward
network vector size to 512, the head number to 8, andthe layer number to 3. For training, we used the following
settings: a batch size of 128, a learning rate of 0.0005,
100 epochs, a dropout rate of 0.1, and a clip norm of 1.0.
The model was optimized using Adam [18]. We used one
NVIDIA GeForce RTX 3090 GPU. It took about 33 minutes
to train the MR2T model with the original training data and53 minutes to train it with the augmented training data. We
chose the best model that resulted in minimum loss on thevalidation data in the E2E refined dataset among those 100
trained models at the end of the training epochs. The loss was
saturated in ten epochs by the MR2T model without orderconstraints, nine epochs by that with the order constraints
using the original training data, and ten epochs by that using
the augmented training data. For TGEN and JUG, we usedthe distributed programs “as-is”.
C. METRICS

We evaluated the MR2T performance by BLEU [3],NIST [19], METEOR [5], ROUGE_L [4], and CIDEr [20],all of which can be obtained using the E2E challenge metrics
VOLUME 10, 2022

Toyama et al.: Content Order-Controllable MR-to-text

TABLE 6. Accuracy of T2MR models. Bold indicates best result.

MethodTransformer w/o order
Transformer w/ order
Transformer w/ order(augmented training data)MRvalue & order
n/a
98.62

MR
value
98.18
98.95

MR
order
n/a
98.84

98.79

99.25

99.06

script4 . To calculate these metrics, we prepared the following
two types of references for the test instances:
Order-independent references: those corresponding to
the given MR without order constraints (the average
number of reference text passages: 6.67);• Order-dependent references: those corresponding to thegiven MR with order constraints (the average number ofreference text passages: 1.21).

•

We also evaluated the performance by checking whether
the original MR to MR2T and the predicted MR from T2MR
are identical. We named this method MRcheck. Since theT2MR performance is highly accurate, as shown in Table 6,
we believe it can be used to evaluate MR2T. We used theorder-constrained T2MR model trained with the augmented
training data for MRcheck.
D. RESULTS

Table 7 shows the scores. The Transformer MR2T model
without order constraints outperformed TGEN and JUG,
indicating Transformer’s effectiveness in this task. Our proposed order-constrained MR2T model clearly outperformed
the model without order constraints. The proposed modeltrained using the augmented data also outperformed themodel trained with the original data. Although an advantage
was observed even with the order-independent references, it
was much larger with the order-dependent references.
Table 8 shows the MRcheck results. The order-constrained
Transformer preserved the MR order very accurately(99.58%) for the original test data, although the baselines
failed to do so (6.74%, 5.93%, and 7.42%). Even though theaccuracy of the reordered test data (88.07%) was worse thanthat for the original test data (99.58%), the performance ofthe proposed model trained using the augmented training data
was almost perfect (99.95%).
Furthermore, we found more MRcheck errors in the test
instances with a larger number of non-empty attributes (Table9): for example, 12 and 1,055.8 errors for the instances witheight non-empty attributes for the original and reordered test
data. However, the amount of proposed model training using
the augmented data significantly improved from 1,055.8 to
5.5. The numbers of deletion/insertion/substitution errors
were also reduced, from 583.5 to 2.8 for deletion errors in
the MR values, from 1.8 to 0.0 for insertion errors in the MR
values, from 109.8 to 2.5 for substitution errors in the MR
values, and from 1,842.5 to 4.8 for substitution errors in the
4 https://github.com/tuetschek/e2e-metrics
VOLUME 10, 2022

MR order. These results suggest the effectiveness of our data
augmentation method.
VI. HUMAN EVALUATIONWe also conducted a human evaluation on the MR2T results.
Native English-speaking crowdworkers5 rated the naturalness, the adequacy [21], and the focus of the generated
text packages of 150 selected examples from the original
test data. The 150 examples were randomly selected after
conditioning the attribute distribution that appeared first in
the text package to be approximately uniform, as shownin Table 10. Three workers evaluated each text passage.
To avoid any misunderstanding that the values of attributes
name and near are emphasized because they are shown inuppercase, we capitalized only the first letter of each word of
those attributes. We gave the instructions shown in Figs. 2, 3,
and 4 and showed a pair of the MR and the text passages fromeach of the 150 examples (Figs. 5, 6, and 7) to the evaluators
who evaluated them.
a: Naturalness

The evaluators gave scores on a 6-point Likert scale (higher
is better) on the following four questions:
1) Is the sentence natural?
2) Is the sentence grammatical?
3) Is the sentence comprehensible?
4) Is the sentence acceptable as English, even if it is notnatural/grammatical/comprehensible?
b: Adequacy

The evaluators answered the following two questions witheither “yes” or “no”:
1) Does the generated sentence meet all the MR values?
2) Does the generated sentence meet all the MR values
and the MR orders?
c: Focus

In this experiment, we assumed that the emphasized attributevalue pair appears first in the text passage [22]. The evaluators answered the following question with either “yes” or“no”:• Is [(attribute) value] the focused attribute-pair inthe sentence?
Here “[(attribute) value]” is the attribute-value pairwhich appears first in each text passage.
The results are shown in Table 11. The naturalness scoresof the three baselines are comparable: TGEN, JUG, andTransformer w/o order. On the other hand, the scores forour proposed methods with order constraints and reference
are slightly lower than those of the baselines, because itis natural for restaurant recommendation sentences to startwith a name attribute, and sentences starting with otherattributes lose some naturalness. Table 10 shows that the
5 We utilized Prolific (https://www.prolific.co/) and paid each worker
thirteen pounds per hour.

5Toyama et al.: Content Order-Controllable MR-to-text

TABLE 7. Results of automatic evaluation: Bold indicates best result for order-independent reference, and Italic indicates best result for order-dependent reference

MethodTGEN
JUG
Transformer w/o order
Transformer w/ order
Transformer w/ order(augmented training data)

Referenceorder-independent
order-dependent
order-independent
order-dependent
order-independent
order-dependent
order-independent
order-dependent
order-independent
order-dependent

BLEU(↑)
0.56260.33390.57330.35050.58400.36000.62800.4836
0.63350.4914NIST(↑)
7.89075.68157.68965.62427.82275.71518.6083
7.09478.61987.0941

METEOR(↑)
0.42780.37620.43370.38710.4384
0.39100.45950.4356
0.46240.4393

ROUGE_L(↑)
0.6614
0.52620.64880.52010.66590.53440.75510.74220.75750.7453CIDEr(↑)
2.40662.08852.39722.05802.51412.17532.77833.30622.7855
3.3383

TABLE 8. MRcheck results of accuracy: Bold indicates best result for original test data, and Italic indicates best result for reordered test data)

MethodTGEN
JUG
Transformer w/o order
Transformer w/ order
Transformer w/ order(augmented training data)

Test data
original
original
original
original
reorderedoriginal
reordered

MR value & order
6.74
5.93
7.42
99.58
88.03
100.0
99.95

MR value
99.41
98.24
100.0
99.96
96.74
100.0
99.98

MR order
6.74
5.93
7.42
99.58
88.07
100.0
99.95

TABLE 9. MRcheck errors (vd: deletion errors in MR values, vi: insertion error in MR values, vs: substitution error in MR values, os: substitution error in MR order).
NEA denotes non-empty attributes.

# NEA
1
2
3
4
5
6
7
8
Total

# data
0
1
92
311
631
1,114
1,422
984
4,555

Original test data
# errors
Transformer w/ order
Transformer w/ order(augmented training data)
0
0
0
0
0
0
2
0
0
0
0
0
5
0
12
0
19
0
vd: 2
vd: 0
vi: 0
vi: 0
vs: 0
vs: 0
os: 17
os: 0# data
0
0
4
546
2,990
5,570
7,110
4,920
21,140

Reordered test data
# errors
Transformer w/ order
Transformer w/ order(augmented training data)
0.0
0.0
0.0
0.0
0.0
0.0
30.0
1.0
203.5
0.5
427.8
2.0
814.0
1.0
1,055.8
5.5
2,531.0
10.0
vd: 583.5
vd: 2.8
vi: 1.8
vi: 0.0
vs: 109.8
vs: 2.5
os: 1,842.5
os: 4.8

TABLE 10. Number of samples in terms of attributes that first appear in text passage

First attribute
nameeatTypefood
priceRange
customer ratingarea
familyFriendly
near
Total

6

Training data

Validation data

Test data

25,783
1,608
2,573
2,426
1,389
2,773
2,000
2,008
40,560

2,787
260
137
165
167
407
258
308
4,489

2,727
299
197
230
83
363
267
389
4,555

Selected data forhuman evaluation
19
19
18
19
18
19
19
19
150

VOLUME 10, 2022

Toyama et al.: Content Order-Controllable MR-to-text

text passages in more than half of the E2E refined dataset
start with the name attribute. This distribution causes thevalue of the name attribute to always appears first in thegenerated text passages of the baseline methods. Table 12
also lists the five examples with the lowest naturalness scores,
and the generated text passages with lower scores in termsof naturalness tend to start with attributes other than name.
However, looking at the acceptable scores in Table 11, there
is no significant difference between the baseline methods
and our proposed methods, and although the order control
has slightly lost some naturalness, our proposed methods
can generate good English without any problems. Table 13
shows an example of the generated text passages for an MR.
The reference text’s style is rather free, whereas the others
resemble templates. This situation might explain why thereference scores are worse than the others.
For adequacy, we found that the generated text passages ofthe proposed methods appropriately met almost all the MR
values and orders. A comparison of the adequacy results withthose of MRcheck (Table 8) identifies a clear correlation,suggesting that MRcheck works as an effective automatic
measure of the reliability of synthetic data.
For focus, the scores of our proposed methods significantly
outperformed those of the baselines. A comparison of theadequacy and focus results shows a clear correlation. Thismeans that correct control of the content order also allows
for correct control of the emphasized attribute-value pair in
the generated sentences.
VII. CONCLUSION

We proposed an MR-to-text method that controls the orderof the MR values in generated text passages using MR
order constraints. Our proposed method worked effectively
and precisely controlled the content order in automatic evaluations. Data augmentation also effectively improved theperformance using the MR2T and T2MR models to balance
the data distribution in terms of non-empty attributes. The
human evaluation results suggest that our proposed methods can focus attribute-value pairs for correct emphasis bycontrolling the content order. Even though the proposed
methods suffered slightly less naturalness compared with thebaselines, they generated proper English sentences without
any problem. Future work will control such other aspects
as the text structure and the output length and apply suchmethods to other MR-to-text datasets.
APPENDIX A EXAMPLES

Some examples are shown in Tables 14, 15, and 16. Comparing Tables 14 and 15, the Transformer models with orderconstraints accurately reflected the MR values and the MRorder in the generated text passages, although TGEN, JUG,
and the Transformer models without order constraints did
not. Comparing Tables 14 and 16, the Transformer models
with order constraints properly preserved the MR order inthe generated text passages, although TGEN, JUG, and theTransformer models without order constraints did not.
VOLUME 10, 2022

APPENDIX B INSTRUCTIONS FOR HUMANEVALUATION

The instructions for those participating in the human evaluation are shown in Fig. 2. The instructions for adequacy andfocus are shown in Figs. 3 and 4. Screenshots of the humanevaluation are shown in Figs. 5, 6, and 7. These systems were
designed using Google Forms.
REFERENCES[1] Z. Kasner and O. Dušek, “Neural Pipepline for Zero-Shot Data-to-Text
Generation,” in Proc. the 60th Annu. Meeting of the Assoc. for Computational Linguistics, Volume 1, 2022, pp. 3914–3932.
[2] Y. Leng, F. Portet, C. Labbé, and R. Qader, “Controllable Neural Natural
Language Generation: comparison of state-of-the-art control strategies,”in Proc. the 3rd Int. Workshop on Natural Language Generation from the
Semantic Web, 2020, pp. 34–39.
[3] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: a Method forAutomatic Evaluation of Machine Translation,” in Proc. the 40th Annu.
Meeting of the Assoc. for Computational Linguistics, 2002, pp. 311–318.
[4] C.-Y. Lin, “ROUGE: A Package for Automatic Evaluation of Summaries,”in Proc. the Workshop on Text Summarization Branches Out,” 2004, pp.
74–81.
[5] M. Denkowski and A. Lavie, “Meteor Universal: Language Specific Translation Evaluation for Any Target Language,” in Proc. the 9th Workshop on
Statistical Machine Translation, 2014, pp. 376–380.
[6] Y. Su, D. Vandyke, S. Wang, Y. Fang, and N. Collier, “Plan-then-Generate:
Controlled Data-to-Text Generation via Planning,” in Findings of the
Assoc. for Computational Linguistics: EMNLP, 2021, pp. 895–909.
[7] Y. Luo, M. Lu, G. Liu, and S. Wang, “Few-Shot Table-to-Text Generation
with Prefix-Controlled Generator,” in Proc. the 29th Int. Conf. on Computational Linguistics, 2022, pp. 6493–6504.
[8] K. Toyama, K. Sudoh, and S. Nakamura, “E2E Refined Dataset,” 2022,
arXiv:2211.00513. [Online]. Available: https://arxiv.org/abs/2211.00513[9] J. Novikova, O. Dušek, and V. Rieser, “The E2E Dataset: New Challenges
For End-to-End Generation,” in Proc. the 18th Annu. SIGdial Meeting on
Discourse and Dialogue, 2017, pp. 201–206.
[10] O. Dušek, J. Novikova, and V. Rieser, “Findings of the E2E NLG Challenge,” in Proc. the 11th Int. Conf. on Natural Language Generation, 2018,
pp. 322–328.
[11] O. Dušek, D. M. Howcroft, and V. Rieser, “Semantic Noise Matters for
Neural Natural Language Generation,” in Proc. the 12th Int. Conf. on
Natural Language Generation, 2019, pp. 421–426.
[12] T. C. Ferreira, H. Vaz, B. Davis, and A. Pagano, “Enriching the E2E
dataset,” in Proc. the 14th Int. Conf. on Natural Language Generation,
2021, pp. 177–183.
[13] C. Kedzie and K. McKeown, “A Good Sample is Hard to Find: Noise
Injection Sampling and Self-Training for Neural Language Generation
Models,” in Proc. the 12th Int. Conf. on Natural Language Generation,
2019, pp. 584–593.
[14] J. Chai, R. Pryzant, V. Y. Dong, K. Golobokov, C. Zhu, and Y.
Liu, “FAST: Improving Controllability for Text Generation with Feedback Aware Self-Training,” 2022, arXiv:2210.03167. [Online]. Available:
https://arxiv.org/abs/2210.03167[15] O. Dušek and F. Jurčíček, “Sequence-to-Sequence Generation for Spoken
Dialogue via Deep Syntax Trees and Strings,” in Proc. the 54th Annu.
Meeting of the Assoc. for Computational Linguistics, 2016, pp. 45–51.
[16] B.-H. Tseng, J. Cheng, Y. Fang, and D. Vandyke, “A Generative Modelfor Joint Natural Language Understanding and Generation,” in Proc. the58th Annu. Meeting of the Assoc. for Computational Linguistics, 2020,
pp. 1795–1807.
[17] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,
Ł. Kaiser, and I. Polosukhin, “Attention Is All You Need,” in Proc. the 31st
Conf. on Neural Information Processing Systems, 2017, pp. 6000–6010.
[18] D. P. Kingma and J. L. Ba, “ADAM: A Method for Stochastic Optimization,” in Proc. the 3rd Int. Conf. on Learning Representations, 2015.
[19] G. Doddington, “Automatic Evaluation of Machine Translation Quality
using N-gram Co-Occurrence Statistics,” in Proc. the 2nd Int. Conf. on
Human Language Technology Research, 2002, pp. 138–145.
[20] R. Vedantam, C. L. Zitnick, and D. Parikh, “CIDEr: Consensus-based
Image Description Evaluation,” in Proc. 2015 IEEE Conf. on Computer
Vision and Pattern Recognition, 2015.
7

Toyama et al.: Content Order-Controllable MR-to-text

TABLE 11. Results for human evaluation in naturalness with 95% confidence interval, adequacy, and focus. Bold indicates best result.

MethodReference
TGEN
JUG
Transformer w/o order
Transformer w/ order
Transformer w/ order(augmented training data)

Grammatical
4.12±0.13
4.76±0.10
4.78±0.10
4.80±0.10
4.44±0.12
4.47±0.12

Naturalness
Comprehensible
Natural4.96±0.10
3.91±0.13
5.24±0.08
4.86±0.09
5.28±0.07
4.77±0.10
5.27±0.07
4.86±0.09
5.10±0.09
4.36±0.12
5.08±0.09

4.36±0.12

Acceptable
5.15±0.09
5.35±0.08
5.36±0.07
5.36±0.08
5.26±0.09

MR value
89.11
92.22
94.44
95.56
94.67

Adequacy
MR value & order
88.89
4.22
4.00
5.56
92.89

5.27±0.09

94.89

92.44

Focus
86.22
56.44
53.78
55.33
91.33
90.67

TABLE 12. Examples with five lowest scores for human evaluations (G: Grammatical; C: Comprehensible; N: Natural; A: Acceptable).

Metrics

G

C

N

A

Score
1.33

MethodTGEN

1.33

Reference

1.67

Reference

1.67

Transformer w/ order

1.67

Transformer w/ order

3.00

Transformer w/ order

3.00

Reference

3.33

Reference

3.33

TGEN

3.33

Reference

1.33

TGEN

1.33

Reference

1.67

Reference

2.00

Reference

2.00

Reference

3.67

Reference

3.67

Transformer w/o order
Transformer w/ order(augmented training data)

3.67
3.67

TGEN

3.67

Reference

Text
NAME is a children friendly pub in the city centre near NEAR. It is in the high price range.
5 out of 5 for this pub, although no facilities for children. It is close to NEAR, called NAME near to theriverside has a price list of more than £30 serves Japanese cuisine.
There is a children friendly English restaurant in the riverside area. It is high price range. It is called
NAME, and is located near NEAR.
Moderately priced NAME is a non kid friendly restaurant located in the city centre, near NEAR. It
serves Chinese food.
There is a 5 out of 5 pub that is not children friendly near NEAR called NAME in the riverside area.
The price range is more than £30 and Japanese food.
Near NEAR is a restaurant that is family friendly in the riverside area called NAME. It serves Italianfood and is cheap.
A kids friendly Japanese pub along the riverside is called NAME and is next to NEAR.
Near NEAR their is a restaurant that is family friendly along the riverside named NAME which serves
Italian food on a cheap price range.
NAME is a children friendly pub in the city centre near NEAR. It is in the high price range.
Located near NEAR in the riverside area is an eat type pub called NAME is a children friendly thatserves Japanese food and has a price range more than £30.
NAME is a children friendly pub in the city centre near NEAR. It is in the high price range.
5 out of 5 for this pub, although no facilities for children. It is close to NEAR, called NAME near to theriverside has a price list of more than £30 serves Japanese cuisine.
3 out of 5 star restaurant style restaurant NAME offers child friendly atmosphere near NEAR.
If you are looking for a high quality, family-friendly dining experience in the heart of city centre, NAME
is for you. This pub I near NEAR and serves fast-food like it’s gourmet.
Near NEAR by riverside is a pub that is yes family friendly with a low customer rating called NAME
and the prices are less than £20.
Located near NEAR in the riverside area is an eat type pub called NAME is a children friendly thatserves Japanese food and has a price range more than £30.
NAME is a Japanese pub in riverside near NEAR. It is children friendly and has a high customer rating.
Moderately priced NAME is a non kid friendly restaurant located in the city centre. It serves Chinese
food and is near NEAR.
NAME is a children friendly Japanese pub in the city centre near NEAR with a high price range and acustomer rating of 5 out of 5.
5 out of 5 rated child friendly NAME, serves Japanese food in a pub at higher than normal prices.
Located in the city centre near NEAR.

TABLE 13. One example for an MR(name: NAME (order=3), eatType: restaurant (order=2), customer rating: 3 out of 5 (order=1), familyFriendly: yes(order=4), and near: NEAR (order=5)) where TGEN and JUG have good scores and reference has bad scores for human evaluation (G: grammatical; C:
comprehensible; N: natural; A: acceptable).

MethodReference
TGEN
JUG
Transformer w/o order
Transformer w/ order
Transformer w/ order(augmented training data)

8

Text
3 out of 5 star restaurant style restaurant NAME offers child friendly atmosphere near
NEAR.
NAME is a three star family friendly restaurant located near NEAR.
NAME is a family friendly restaurant with a customer rating of 3 out of 5. It is located near
NEAR.
NAME is a family friendly restaurant located near NEAR. It has a customer rating of 3 outof 5.
There is a 3 star restaurant NAME that is family friendly located near NEAR.
There is a three star restaurant NAME that is family friendly located near NEAR.

Score
G

C

N

A

1.67

4.67

1.67

5.33

6.00

5.67

6.00

6.00

6.00

5.67

6.00

6.00

6.00

5.67

5.67

6.00

4.00

5.33

4.33

5.67

5.00

5.67

4.00

6.00

VOLUME 10, 2022

Toyama et al.: Content Order-Controllable MR-to-text

TABLE 14. One example of MR value, MR order, and generated text passages

MR

Reference
TGEN
Text

JUG
Transformer w/o order
Transformer w/ order
Transformer w/ order(augmented training data)

Attribute
Value
Order
name
NAME (THE WATERMAN)
1
eatTypepub
2
food
Italian5
priceRange
less than £20
6customer rating (empty)
0
areacity centre
3
familyFriendly
no
7
near
NEAR (RAJA INDIAN CUISINE)
4NAME pub is located in the city centre area near NEAR. It has Italian food in the £20 or less price range and is notfamily-friendly.
NAME is an Italian pub located in the city centre near NEAR. It is not family-friendly and has a price range of lessthan £20.
NAME is a pub in the city centre near NEAR. It serves Italian food for less than £20. It is not family-friendly.
NAME is a pub that serves Italian food. It is located in the city centre near NEAR. It is not family-friendly and hasa price range of less than £20.
NAME is a pub in the city centre near NEAR. It serves Italian food for less than £20 and is not family-friendly.
NAME is a pub in the city centre near NEAR. It serves Italian food for less than £20 and is not family-friendly.

TABLE 15. Another example of generated text passages: MR value is identical to Table 14, although MR order is different.

MR

Reference
TGEN
Text

JUG
Transformer w/o order
Transformer w/ order
Transformer w/ order(augmented training data)

Attribute
Value
Order
name
NAME (THE WATERMAN)
1
eatTypepub
3
food
Italian4
priceRange
less than £20
5customer rating (empty)
0
areacity centre
7
familyFriendly
no
2
near
NEAR (RAJA INDIAN CUISINE)
6NAME is a non family-friendly pub that serves Italian food for less than £20. It is located near NEAR in the city
centre area.
NAME is an Italian pub located in the city centre near NEAR. It is not family-friendly and has a price range of lessthan £20.
NAME is a pub that serves Italian food for less than £20. It is located in the city centre near NEAR and is notfamily-friendly.
NAME is a pub that serves Italian food. It is located in the city centre near NEAR. It is not family-friendly and hasa price range of less than £20.
NAME is a non family-friendly pub serving Italian food for less than £20 near NEAR in the city centre.
NAME is a non family-friendly pub that serves Italian food for less than £20. It is located near NEAR in the city
center.

TABLE 16. Another example of generated text passages: Except the MR value of eatType, all MR values and MR orders are identical to Table 14.

MR

Reference
TGEN
JUG
Text

Transformer w/o order
Transformer w/ order
Transformer w/ order(augmented training data)

VOLUME 10, 2022

Attribute
Value
Order
name
NAME (THE WATERMAN)
1
eatTyperestaurant2
food
Italian5
priceRange
less than £20
6customer rating (empty)
0
areacity centre
3
familyFriendly
no
7
near
NEAR (RAJA INDIAN CUISINE)
4NAME restaurant is located in the city centre area near NEAR. It has Italian food in the £20 or less price range andis not family-friendly.
NAME is a non family-friendly Italian restaurant in the city centre near NEAR with a price range of less than £20.
NAME is a restaurant that serves Italian food for less than £20. It is located in the city centre near NEAR and is notfamily-friendly.
NAME is a non family-friendly Italian restaurant located in the city centre near NEAR. It has a price range of lessthan £20.
NAME is a restaurant in the city centre near NEAR. It serves Italian food for less than £20 and is not familyfriendly.
NAME is a restaurant located in the city centre near NEAR. It serves Italian food for less than £20. It is not familyfriendly.

9

Toyama et al.: Content Order-Controllable MR-to-text

FIGURE 2. Instructions for Human Evaluation[21] J. Amidei, P. Piwek, and A. Willis, “The use of rating and Likert scales
in Natural Language Generation human evaluation tasks: A review andsome recommendations,” in Proc. the 12th Int. Conf. on Natural Language
Generation, 2019, pp. 397–402.
[22] N. Suzuki, S. Nakamura, “Representing ‘how you say’ with ‘what you
say’: English corpus of focused speech and text reflecting corresponding
implications,” in Proc. Interspeech, 2022, pp. 4980–4984.

10

VOLUME 10, 2022

Toyama et al.: Content Order-Controllable MR-to-text

FIGURE 3. Instructions for Human Evaluation for Adequacy

VOLUME 10, 202211

Toyama et al.: Content Order-Controllable MR-to-text

FIGURE 4. Instructions for Human Evaluation for Focus

12

VOLUME 10, 2022

Toyama et al.: Content Order-Controllable MR-to-text

FIGURE 5. Screenshot of Human Evaluation for Naturalness and Focus (1)FIGURE 6. Screenshot of Human Evaluation for Naturalness and Focus (2)VOLUME 10, 2022

13

Toyama et al.: Content Order-Controllable MR-to-text

FIGURE 7. Screenshot of Human Evaluation for Adequacy

14

VOLUME 10, 2022

Toyama et al.: Content Order-Controllable MR-to-text

KEISUKE TOYAMA (GS’07–M’08–GS’19) received the B.E. degree in engineering from
Nagoya University, Japan, in 1996, the M.E. degree in engineering from Nara Institute of Scienceand Technology, Japan, in 1998, and the MSc
degree in electronic engineering by research with
distinction from Queen Mary University of London, UK, in 2008. He is currently a Ph.D. candidate at Nara Institute of Science and Technology.
Since 1998, he has been with Sony Corporation(renamed Sony Group Corporation in 2021) as a research engineer in audio
and music signal processing field.
Mr. Toyama is a member of the Acoustical Society of Japan.

SATOSHI NAKAMURA (Fellow, IEEE) received
the B.S. degree from the Kyoto Institute of Technology in 1981 and the Ph.D. degree from Kyoto
University in 1992.
From 1994 to 2000, he was an associate professor with the Graduate School of Information
Science, Nara Institute of Science and Technology,
Japan. He was the director of the ATR Spoken
Language Communication Research Laboratories
from 2000 and 2008 and the vice president of
ATR from 2007 to 2008. He was the director general of the Keihanna
Research Laboratories and the executive director of the Knowledge Creating
Communication Research Center, National Institute of Information andCommunications Technology, Japan from 2009 to 2010. He is currently aprofessor with the Graduate School of the Science and Technology, Nara
Institute of Science and Technology, a project leader of the Tourism Information Analytics Team, RIKEN, Center for Advanced Intelligence Project
AIP, an honorary professor with the Karlsruhe Institute of Technology,
Germany, and an ATR fellow. He is also a director of the Augmented Human
Communication Laboratory and a full professor with the Graduate School ofInformation Science, Nara Institute of Science and Technology. His researchinterests include modeling and systems of speech-to-speech translation andspeech recognition.
Prof. Nakamura is one of the leaders of speech-to-speech translation
research and has been serving for many years on various worldwide speechto-speech translation research projects, including C-STAR, IWSLT, and ASTAR. He has been an elected board member of the International SpeechCommunication Association (ISCA) since June 2011, an IEEE Signal
Processing Magazine Editorial Board Member since April 2012, and an
IEEE SPS Speech and Language Technical Committee Member since 2013.
He has received the following awards: the Yamashita Research Award,
the Kiyasu Award from the Information Processing Society of Japan, theTelecom System Award, the AAMT Nagao Award, the Docomo Mobile
Science Award in 2007, and the ASJ Award for Distinguished Achievements
in Acoustics. He received a Commendation for Science and Technology
from the Minister of Education, Science and Technology and the Commendation for Science and Technology by the Minister of Internal Affair andCommunications. He also received the LREC Antonio Zampolli Award in
2012.

KATSUHITO SUDOH received the B.S. degree
in engineering and M.S. and Ph.D. degrees in
informatics from Kyoto University in 2000, 2002,
and 2015.
He was with NTT Communication Science Laboratories from 2002 to 2017 and is currently an
associate professor with Nara Institute of Scienceand Technology. His research interests include
machine translation and natural language processing.
Dr. Sudoh is a member of the Association for Computational Linguistics(ACL) and the International Speech Communication Association (ISCA).
VOLUME 10, 202215