A Character-Level Length-Control Algorithm for Non-Autoregressive Sentence Summarization
Authors: Puyuan Liu
Xiang Zhang
Lili Mou
Venue: NeurIPS 2022 Accept (2022)
Keywords: summarization
non-autoregressive generation
length control

Abstract: Sentence summarization aims at compressing a long sentence into a short one that keeps the main gist, and has extensive real-world applications such as headline generation. In previous work, researchers have developed various approaches to improve the ROUGE score, which is the main evaluation metric for summarization, whereas controlling the summary length has not drawn much attention. In our work, we address a new problem of explicit character-level length control for summarization, and propose a dynamic programming algorithm based on the Connectionist Temporal Classification (CTC) model. Results show that our approach not only achieves higher ROUGE scores but also yields more complete sentences.

A Character-Level Length-Control Algorithm for Non-Autoregressive Sentence Summarization
Puyuan Liu, Xiang Zhang, Lili Mou"
Dept. Computing Science, Alberta Machine Intelligence Institute (Amii)
University of Alberta, Canada "Canada CIFAR AI Chair, Amii
(puyuan, xzhang23/ @ua bberrta.... ca, wwwbblepwer..ooufeeee com
Abstract
Sentence summarization aims at compressing long sentence into a short one that keeps the main gist, and has extensive real- applications such as headline generation. In previous work, researchers have developed various approaches to improve the ROUGE score, which is the main evaluation metric for summarization, whereas controlling
summary length has not drawn much attention. In our work, we address new problem of explicit character-level length control for summariza- tion, and propose a dynamic programming algorithm based on the Connectionist Temporal Classification (CTC) model. Results show that our approach not only achieves higher ROUGE scores but also yields more complete sentences
Introduction
Sentence summarization aims at compressing a long sentence into short one that conveys the main idea of the input. is an established setting of summarization, and has extensive real-world applications such as generating news headlines [26] 30 g and being a key component of document summarization 44 21. Previous research has focused on improving the ROUGE score, a widely [1] while less attention has been paid to controlling the
used evaluation metric for summarization length of the summary.
Recently, researchers argue that length control is the key to summarization, since it is typically required by real-world applications (20). Moreover, the ROUGE score is found be sensitive the summary length [33 (32, and summarization systems can achieve higher scores by simply generating longer output. Previous work mainly addresses length control in the word level, i.c., restricting the summary length by pre-defined number of words 33 18
In this paper, address explicit length control in the character level for summarization, which is different setting from previous work. In other words, we restrict the summary length by the number of characters, such letters, punctuation marks, and whitespaces. We observe that this is a more realistic setting in real-world applications than word-level length control. For example, the headline shown in mobile app or web page is constrained by the screen width (roughly speaking, the number of characters), rather than the number of words. Likewise, a commercial LED display allows certain number of characters; ideally, the text shown should fit the character-level budget, or otherwise will be scrolling, making it difficult to read. However, the character-level constraint is unlikely to be satisfied if we only perform word-level control, despite the positive correlation between word and character lengths. Moreover, character- length control is a common evaluation setting for summarization systems. For example, a sub-task in the DUC evaluation requires the maximum target length to be 75 bytes (27). Such an important setting, unfortunately, is not adequately addressed
Our code, model, and output released http::://tthbbbbcco///aat
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

in previous summarization studies. Our work largely bridges the gap between the methodological summarization research and its evaluation.
We further observe that controlling the summary length by characters cannot be easily addressed by previous approaches. For example, truncating is able explicitly control the length, but the resulting summary is incomplete; Takase and Okazaki [36] feed length embeddings into the model as input, but such an approach cannot control the summary length in an explicit manner; and Schumann et al. B3 perform constrained discrete optimization by selecting a certain number of words from the source text as the output, but their generated summaries may vary to large extent in terms of the number of characters.
To this end, propose NACC, Non- Autoregressive summarization model with Character- length Control. We adopt a non-autoregrressive approach because it generates all tokens in parallel and is much faster than autoregressive models. More importantly, we observe that non-autoregressive models predict the output words independently. Such predicted probabilities are thus local, which provides us with the unique opportunity to design a dynamic programming algorithm to constrain the summary length. Specifically, we formulate length control as a knapsack-like problem, where the weight is the number of characters in a token and the value the predicted probability of the token. In this way, we are able to explicitly control the summary length at the character level, while retaining the completeness of the output text.
We evaluated our NACC model on the Gigaword headline generation [10] and DUC2004 [ datasets in two settings: supervised and unsupervised. In the latter setting, NACC learns from the pseudo-reference given by an unsupervised worddextraccton method based on discrete search 33 Experiments show that NACC establishes the mate-of-theear performance of non-autoregresssve summarization under various target lengths in both settings; NACC even outperforms autoregressive Transformers (37) in the unsupervised setting, where the input and output have stronger correspon- dence. These all confirm the effectiveness of our length-control algorithm. Regarding inference efficiency, we show that non-autoregressive models without length control are 10 times faster than autoregressive ones; even with our length- coggtol dynamic programming, NACC is still several times more efficient. Further, our NACC capable of trath-r generation, i.e., generating summaries of different lengths from the training targets.
Related Work
Non-Autoregressive Generation. aon-autoregressive (NAR) models [2] predict target tokens in parallel and thus enjoy much higher inference efficiency than autoregressive (AR) ones, which generate output tokens sequentially. NAR generation is more difficult than AR, because there lacks dependencies among the outputs. Previous work addresses the dependency issue by iterative or structured decoding with the conditional random field (CRF)
refinement [6 22
K
The Connectionist Temporal Classification (CTC) algorithm [I]] addresses a common problem in NAR generation, namely, token repetition, by merging consecutive identical tokens (unless separated by an empty token). We follow our previous work [18] and adopt CTC for NAR summarization, allowing empty tokens scattering over the entire sentence to generate a short summary.
NAR models are traditionally thought to generate worse-quality text than AR models. However, have the unique insight that NAR also brings new opportunities for designing length-control algorithms: since the model predicts output independently, the decoding problem can be divided into shared sub-problems for dynamic programming.
Text Summarization. tive and abstractive. Extractive methods output summary by extracting important sentences or clauses from the source text [5] 23 5
Summarization systems can be generally categorized into two types: extrac-
while abstractive methods generate summaries with new
expressions 255 28 D..
Depending on the availability of training data, we may categorize summarization systems into supervised and unsupervised ones. Supervised methods typically follow a eqquence-to-sequees (seq2seq) paradigm [3 19] Recently, unsupervised summarization is drawing increasing attention because it does not require parallel training data. Yang et al. 42 use the lead approach (selecting the first several words or sentences) as pseudo-grounding consistency is adopted as the training objective for unsupervised summarizaiton [38] 2]. Schumann et
for seq2seq training. Alternatively, cycle
2

---

al. [3] generate summary by maximizing a heuristically defined scoring function (involving fluency and semantic similarity) with word-level extraction. We consider both supervised and unsupervised settings in our work.
Controlling the output length is crucial for deploying summarization systems real-world ap- plications. In early extractive summarization research, truncating is adopted for fair compari- son 23 24 15), but the resulting summary may not be complete sentences. As pointed out by [32 33], however, the length-control problem is not adequately addressed for abstractive summarization in the neural era, probably because researchers do not want to hurt the completeness. For example, length information [20 36 and soft penalty [44] are adopted to encourage short summaries, but they cannot control the length in an explicit way. Very recently, Schumann et al. [33] address the length-contrr problem by extracting a certain number of source tokens as the summary. Our previous work [[8] designs CTC-based algorithm that controls the number of output words explicitly. However, these approaches cannot be applied to the character level. Our paper proposes character-leve length- control algorithm that can explicitly constrain the number of characters within a given budget, which is different from all previous work.
3 Approach
In this section, we first introduce the CTC-trained NAR model for summarization ($3.1) Then, we propose a dynamic programming algorithm explicitly control the number of characters ($3.2)
3.1 A Non-Autoregressive Model for Summarization
In the summarization task, the goal is to compress a source text (x1, X2, ..,,, xs)))...)))).....).)))))... into a shorter text y = (y1.y2, yT), while preserving the key information.
Encoder-Only Architecture. Our work follows recent non -on-autoregressive summarization meth- ods [34 18]. utilizing the encoder-only Transformer [37] as the base model (Figure [[)). It is essentially stack of Transformer layers, the last of which predicts all words simultancously. Thus, has a much higher inference efficiency than autoregressive models.
Formally, let g Rxd be the contextual representation of the input x the ith Transformer layer, where S is the number of tokens in x and is the dimension. Specially, E is the word
and positional embeddings of input tokens. Each layer is a Transformer block 37 based on its predecessor, which can be denoted by g00 = Layer (g(G))) for i 2 1.
At the last layer E(T,,, a softmax function applied to predict the probability at every prediction slot independently, given by P&(-Ix) = softmax(We." ( for slots = 1.... where e!²) is column vector transposing the sth row of matrix E(L), and W is weight matrix.
Such encoder-only architecture is able to capture the strong correspondence between the input and output in the summarization task and outperforms encoder-decoder non-autoregressive models, shown by our previous work [18].
CTC Training. Training the above encoder-only architecture requires padding the target with empty tokens c; otherwise, the output will have the same length as the input, and cannot be summary. Gu et al. 12 pad es at the end of the target, but this hinders the input-output correspondence.
Instead, we follow our previous work [18 and train our model with the Connectionist Temporal Classification (CTC) [.] allowing es to scatter over the entire target summary as appropriate. During inference, the CTC approach removes to generate shorter text as the summary. In addition, CTC merges consecutive identical tokens that are not separated by e, because non-autoregressive generation suffers from the word-repetition problem 12 31. We denote the CTC operation as I', for example, I' (aaeeaabbe) = aab
For training, CTC maximizes the marginal likelihood of token sequences (including e) that can be reduced the target text y:
P(y x) = Σ P(w[x)= = Σ II
P.(w,lx)
(1)
w:
w:ww)=yy a=1
3

---

Model architecture
(b) Decoding algorithm
Generation slots
Wi:1
W1.2
W13
W1:4
W1.
6,-0.3
-0.2
6E,
66C,
CEEE,
(8)
Ws Wa
Manufeeturinggo .0.
[1.5]
-0.1
US, -0.2
Length buckets
budinesn 4.4
[6, 10]
Factory, -0.2
maboor 02.2
US maker, -0.3
Transformer
[1]
N/A
Ass 0
Factory has, -0.3
-0.2
has, -0.5
Factory
a. 0.2
Factory has -0.7
Embedding
:
:
:
Input: Factory orders for manufactured goods rose 2.3 percent in september, the commerce department said here thursday.
[41,45]
N/A
has orders september
[46,50]
Figure I: Overview of our NACC approach. Dashed yellow arrows refer to transitions that do not increase the summary length, while solid blue arrows refer to the increase of length. Thick arrows and blocks refer to the selected path by CTC. Due the space limit, is omitted in the predicted sentence, and use e(8) to denote sequence of S-many es. The number demonstrates the value (i.e., (og-probability) word.
where P(w[x) is the probability of generating token sequence = W1,".. ws. and given by the non-autoregressive model. Although a brute force summation in Eqn. (T is intractable, can be efficiently computed by dynamic programming. We refer interested readers to Alex et al. [I] for the details of the above marginalization in CTC training.
is
3.2 The Proposed Algorithm for Character-Level Length-Contro Decoding
One of our main contributions is to propose a dynamic programming (DP) algorithm for character- level length control. As mentioned in ş controlling the number of characters in the output is important for summarization, since real-world applications usually involve character-level length constraints.
Our insight is that a non-autoregressive model predicts probabilities independently, so the length- control problem can be divided into shared sub-problems, making DP algorithms feasible. We formulate character-level length control as a knapsack-like problem. We treat the number of characters word (plus one) the weight denoted by u(w) for the word w, and the predicted bog-prbbbbility as the value Us (w) = log P. (w|xx x for the prediction slot s. Our goal of character-level length-contrg summarization can be formulated as
maximize
Σ Dv((s)) Wi WW
subject to
Σ
u(y) <
where U is the total length budget. Here, the value is the sum of the log-probability of every generation slot including e, whereas the length is said in terms of the words of the CTC-reduced sequence y [(w1, ws).
We observe that handling every possible integer weight (i.e., length) in standard knapsack algorithm may slow down the inference. Thus, we divide the lengths into buckets for efficient characters. where a is hyperparameter controlling the bucket size. We denote by d = . probable s-token sequence that is reduced to summary in the Ith length bucket. Specially, we let d=00 mean that the reduced summary has zero words.
inference. Formally, let the /th bucket cover the length ranging from a (1 (1-11 1) +
... ......... the most
The initialization of ! fills in the DP table for = 0 and s 1.
For the purposes of research, we assume every word is appended with another character, namely, a whitespace. In real applications, our algorithm can be applied to any measure of length, such as the display width of word in some font.

---

For 1 = O, we must have d = c...c (s-many), because I = 0 means non-c word has been generated. (First row of Figure [p.)
For = 1, we have
if 1 (w(w). if >> 0
d
(3)
argmax
Here, = the same as the previous bullet item. For / O, we select the most probable word for each length bucket according the value 21 (1)),, i..,, the predicted log-pret log-probability of the first generation slot. (First column of Figure [
The DP recursion is to compute d based on newly predicted token wa. assuming its top-left sub-table is filled. This involves three scenarios:
Case I: W = c. In this case, the new word c. Thus, the index for generation slots increases from 1 to s, but the summary length does not change. We denote ger as the set containing the candidate sequence, given by
= Φe}
(4)
where @ denotes string concatenation. (See yellow dash arrows in Figure(b..
Case 2: W. * but w, = d=- In other words, the candidate non-c word w. for the sth slot is identical to the last token of d= Since repeated tokens are merged during CTC decoding, the output length index is unchanged. We form this sequence as a set:
25 {d--ld 1, @ df=
(Also see yellow dash arrows in Figure
Case W, * and w, # d==l"" for some 1' < I. That is, w, is neither nor repetition, and thus the summary length will be increased from bucket l' We denote this candidate set by
20d { d=-1d' w. (u(w.)+ +
Σ u(d)) E [a (1 1) + a,0.1 [1,
(6)
dEd=-1.'
# e,
# d=
11" and l' <
(See blue arrows in Figure p.)
Then, our DP finds the most probable sequence at each recursion step:
d
argmax dE 1E UOO
C. (de)
where d, is the sth token of sequence d from the three candidate sets above.
Theorem (I) the bucket size and consecutive repetitions are merged, then d is the most probable sentence T T characters given by the S prediction slots. (2) If a & or repeating tokens are merged, our algorithm may not be exact. (See Appendix Afor the proof.)
Discussion. Our DP algorithm inspired by the standard 0-1 knapsack problem [ but also differs in several significant ways. First, merge consecutive tokens during CTC decoding; this establishes some dependencies among different generation slots, and thus exact inference with DP is not possible. it changes over time. We require that every slot
Second, our value function is non-stationary, should select a token, either e or word. In both cases, the token's 's value added to the total value. Therefore, algorithm is compatible with negative values, namely, probabbiailsssiss in application, because only the relative difference matters for the value function.
Experiments
4.1 Setup
Datasets.
Our model is evaluated on the Gigaword headline generation 30 and DUC2004 datascts (27). The Gigaword dataset comprises pairs of news articles and titles; we follow the
5

---

standard setting [3 33 18 and adopt the first sentence of each news article as the input and consider
the news title as the summary, In total, the dataset contains 3.8M, 198K, and 1951 samples for training, validation, and test, respectively. On the other hand, the DUC2004 dataset has 500 paired samples and is designed for test only; its performance obtained by models trained on Gigaword.
Metries. We follow previous work 33 30 34] and evaluate the output by ROUGE scores [17] ROUGE-n evaluates n-gram overalpping, whereas ROUGE-L evaluates the longest common se- quence. We follow the convention and adopt ROUGE F1 for the Gigaword dataset 38 33
and
ROUGE Recall for the DUC2004 dataset 6 39.
Implementation Details. We use a Transformer encoder as the base model, which has layers and 8 attention heads for each layer. following the settings in [37]. The dimensions are 512 and 2048 for the attention and feed-forward modules, respectively. Each training batch contains samples amounting to 4K tokens. The learning rate is chosen from (1e-4, 5e-4) by validation, and we ran 100K gradient updates for the unsupervised setting, but 400K updates for the supervised setting. Note that, in the unsupervised setting, our model learns from an extractive search-based method [33], and thus its training is relatively easy. By contrast, the supervised setting takes human-written sentences as the training targets, which are more abstractive and difficult to train with. All experiments were run on an i9-9940X CPU and an RTX6000 GPU.
For our length-contre algorithm, we adopt a bucket size of 4, and only consider the most probable 20 words for every generation slot (ef. w, in Equ. (6 due to efficiency concerns.
4.2 Results and Analyses
Results on Gigaword Headline Generation. Table presents the performance on the Gigaword test set in both supervised and unsupervised settings. For the supervised setting, we train with the groundtruth references, which contain 51.29 characters on average. the unsupervised setting, the training targets of machine learning models (Rows 9-13) are the output summaries given by an unsupervised search- based method [33]; specifically, we adopt the standard 8- 8-word setting, which counts to 48.75 characters average. Based on these, set our length constraint to 50 characters. We observe that our proposed algorithm is the only machine learning- approach that can perform explicit character-level length control. For fair comparison, we truncate the output of other methods to satisfy the length constraint.
As seen, NACC even with truncating outperforms all other non-autoregressivee (NAR) models 34
29
41 in both settings. Specifically, Su et al. 34 emit multiple end-of-sequence tokens at the end of
the output sequence to generate shorter summary than the source text (Rows & 9); Qi et al. [29] propose to pretrain a summarization system in an autoregressive manner, and gradually adapt it to the NAR setting (Rows 5 & 10); and Yang et al. [41 propose two-step strategy of autoregressive part-of-speech (POS) prediction and aon-autoregressive summary generation (Rows 6 & 11). Our NACC trained by CTC is able to surpass all these methods, demonstrating the effeectiveness of CTC training for summarization. Besides, NACC outperforms its search-based teacher model 33 showing that learning can smooth out the search noise.
Equipped with the length-control algorithm, our NACC model has a significant boost in terms of ROUGE scores. In fair comparison with the same base architecture, the length-control algorithm alone improves NACC (truncate) by -3 points. Our full model achieves an improvement of more than 4 total ROUGE points compared with previous state-of-the-art NAR methods.
Regarding the inference efficiency, NACC with truncating is faster than previous NAR models. Even with the length-contry algorithm, our NACC is still in the same ballpark, being -500x faster than the bearch-based method.
Additionally, we design a variant of the unsupervised summarization method based on [33], where we directly impose the character-leve length constraint during each search step (Row 8). We find this approach outperforms truncating word-constrained search (Row 7), but is much worse than our machine learning-based NACC with the length-control algorithm.
YYan et al. 41 only provided execution commands in their GitHub repo, but no training code. We emailed the authors, but have not obtained the code either. The reported results based our best replication.

---

ROUGE FI
Setting
Approach
Time
R-1
R-2
R-L
AR
Su et
34
(truncate)
38.43
32.28
14.21
30.56
0.016
2
Qi
29
(truncate)
27.98
31.69
12.52
30.05
-2.79
0.019
Supervised
3
NAR
Yang et al.
41
(trnncate)
35.37
28.85
6.45
27.00
-14.75
NACC (truncate)
34.15
33.12
13.93
31,34
1.34
0.011
5
NACC (length control)
34.40
33.66
13.73
31.79
4.74
0.017
Baseline
Lead-50 chars
49.03
20.66
7.08
19.30
-9.23
Schumann et al. B3 (truncate)
45.45
24.98
9.08
23.18
0.97
9.573
Search
8
Char-constrained search
44.05
25.30
9.25
23.43
1.71
17.324
Unsupervised
9
Su ct al. [4]
(truncate)
43.24
24.65
8.64
22.98
0
0.017
10
Qi et al. 22 (truncate)
44.54
24.31
7.66
22.48
-1.82
0.019
NAR
Yang et al.
(truncate)
49.37
21.70
4.60
20.13
-9.84
12
NACC (truncate)
47.77
25.79
8.94
23.75
2,21
0.012
13
NACC (length control)
47.03
27.45
8.87
25.14
5.19
0.025
Table 1: Performance on the Gigaword headline generation test set, where NAR stands for non- autoregressive. Len: Average number of characters in the predicted summaries. R-1, R-2, R-L:
ROUGE-I, ROUGE-2, ROUGE-L. AR: The difference of total ROUGE (sum of R-1, R-2, and R-L) in comparison with the (previous) state-of-the-ar NAR summarization system 3 Time: Average inference time in seconds for one sample on an 19-9940X CPU and an RTX6000 GPU.
ROUGE
Recall
Approach
Time
R-1
R-2
R-L
Baseline
Lead-75 chars
22.52
6.50
19.74
-4.97
Schumann al. [R] (Iruncate)
26.09
8,03
22.86
3.25
30.362
Scarch
3
Char-constrainee search
26.30
7.95
22.78
3.30
31.540
Su et al. 34 (truncate)
24.67
7.25
21.81
0.017
Qi ct al. [29 (truncate)
22.79
5.91
20.05
-4.98
0.018
NAR
NACC (truncate)
26.43
7.86
22.66
3.22
0.012
NACC (length control)
28.37
7.74
24.30
6.68
0.030
Table Results on DUC 2004 dataset.
Results on DUC2004. Table the performance of NACC on DUC2004, where we constrain the
summary length to be at most 75 characters, following previous work [39 2 33 Since the Gigaword training summaries have very different lengths from DUC2004 summaries, it is not straightforward to evaluate NACC in the supervised setting, as this involves length transfer. Therefore, we consider the unsupervised setting for DUC2004, where use 13-word summaries (-80 characters) from 33 as our training targets. (We will have length-transfer experiments in later analysis.)
Experiments show that NACC with length control again largely outperforms previous NAR models, while retaining high inference efficiency. Results are consistent with the Gigaword experiment.
Human Evaluation. We additionally conducted human evaluations to investigate the effectiveness of our approach. Due the limit of time and resources, we mainly considered the overall quality and completeness/flunnyy as they are the key of our work but may not be adequately captured by automatic metrics such as ROUGE scores. For controlled comparison, we consider NACC with truncating and NACC with the proposed length-control algorithm.
We invited three human annotators to compare the output summaries each of the 150 randomly selected samples from the Gigaword test set. We adopted the unsupervised setting in Table [ where models were trained with 8-word summaries generated by 33
The two systems' outputs for
Decoding
Wins
Ties
Loses
p-value
Truncate
18%
44%
38%
Overall quality
0.0001
Length control
38%
44%
18%
Completeness
Truncate
22%
36%
42%
0.0002
& fluency
Length control
42%
36%
22%
Table 3: Human evaluation comparing truncating and length-control decoding of our NACC approach on 150 samples selected from the Gigaword headline generation dataset in the unsupervised setting. The p-value is given by a two-sided binomial test.

---

ROUGE FI
Setting
Approach
Time
R-1
R-2
R-L
Transformer (truncate)
40.89
35.12
16.61
32.55
0.093
AR
Supervised
Transformer (length control)
39.73
34.10
15.65
31.64
0.095
NACC (truncate)
34.15
33.12
13.93
31.34
0.011
NAR
NACC (length control)
34.40
33.66
13.73
31.79
0.017
Transformer (truncate)
46.62
26.31
9.33
24.29
0.092
AR
Unsupervised
Transformer (length control)
45.23
25.33
9.03
23.44
0.095
NACC (truncate)
25.79
23.75
0.012
NAR
NACC (length control)
47.03
27.45
8.87
25.14
0.025
Table 4: Comparing autoregressive (AR) and non-autoregressivy (NAR) models on the Gigaword headline generation test set. Our length-contro algorithm requires the predicted probabilities to be independent, and thus is not compatible with AR models.
ROUGE 244
Average ROUGE Score
Average ROUGE Score
Interence Time (S)
0.030
NACC
22
0.028
20
20
188
18
0.026
16
16
0.024
14
14
3
30
40
50
60
70
80
90
Size
Length Budget (chars)
Figure 2: Performance of NACC with differ-
Figure 3: Length-transfer performance of NACC and Su et al. (34]
bucket sizes.
sample were blindly given to annotators random order; annotators then voted for the better summary tic, regarding the overall quality and completeness/ttunnyy separately. We counted the votes for each decoding method on the two evaluation criteria, and show the percentage of wins/ties/loses in Table
As seen, our length-control decoding has dominant advantage over the truncating method in terms of both the overall quality and completeness/ftunnyy Further, this result is statistically significant based on a two-sided binomial test (ignoring ties), verifying that our length-control algorithm indeed improves the quality and completeness of the predicted summaries.
Comparison with Autoregressive Models. We curious about how non-autoregressive NACC is compared with autoregressive (AR) methods. Thus, we train a standard AR Transformer with truncating and length-control decodings, and show results in Table
As seen, our length-control algorithm is not compatible with the AR Transformer and hurts the ROUGE scores (Rows 2 & 6). This is because algorithm is based dynamic programming and requires model outputs to be local, so that the length-control problem can be divided into shared sub- problems; however, the predicted probabilities of the AR Transformer depend partial generation at previous time steps. Note that this is not a disadvantage of our approach, but shows that NAR generation provides unique opportunities for length control.
Admittedly, NACC has slightly lower ROUGE scores than AR Transformers in the supervised setting (Rows 1-4), because human- written summaries flexible, causing difficulties for NAR training. Nevertheless, our NACC achieves much higher inference efficiency, and outperforms all previous NAR systems (recall Table
In the unsupervised setting, our approach achieves higher performance than the AR Transformer, which is very strong result. This is because we learn from search- method that extracts source words as the summary (33], and our CTC training-winn blank tokens scattering over whole output sentence as appropriate- can capture such strong correspondence between input and output. Moreover, the proposed length-control algorithm is able to maintain the summary completeness given the length constraint, achieving better ROUGE scores than NACC with truncating.

---

Input: singapore airline and delta air lines announced two differing stralegies to upgrade their ongg inghht
service for business travelers.
Reference: business travel: competing strategies; crowded skies
Supervised Setting:
AR Transformer (no control): delta airlines upgrade service for business travel (50 characters)
NACC (no control):
delta to upgrade long-haul in-flight (36 characters)
NACC (length control):
delta air to upgrade long-hate haul in fngghthh service (48 characters)
Unsupervised Setting:
AR Transformer (no control): delta air lines differing strategies upgrade their in-Olght service for business travelers
NACC (no control):
delta air lines differing strategies upgrade long-haul in in-flight service for business travelers
NACC (length control);
delta air lines upgrade service husiness travelers
Table Example summaries for Gigaword headline generation, where gray words are truncated for fair comparison.
Analysis of the Length Bucket. Our dynamic programming is an approximate algorithm with a-sized length bucket (see Figure [p and $3.2). Here, we investigate the effect of the bucket size in terms of ROUGE scores (the arithmetic mean of R-1, R-2, and R-L) and inference efficiency in the unsupervised setting where training targets are 8-word summaries from (3), following Table []
As seen in Figure 2, the ROUGE score continuously decreases with larger bucket size (thick orange curve). This not only confirms the inexactness of our algorithm, but also shows that a small bucket size does not hurt the performance much. On the other hand, the inference time decreases drastically at the beginning (thin blue curve) because we have fewer dynamic programming steps; as the bucket size increases, the inference time convergences to NACC without length control. Based on this analysis, we set the bucket size to be 4 in our experiments.
Length- Length-Transfer Generation. Our NACC model is capable of length-transfer generation, that is, generating summaries of different lengths from the training targets. Such generation is important to real-world applications where summaries of various lengths are needed for the same input, e.g., fitting different screen widths. Although generating a short enough summary may satisfy all possible length constraints, a longer summary that better utilizes the length budget can preserve more information; this is also reflected by ROUGE scores, which prefer longer summaries, as argued by 33
Figure 3 compares the performance of NACC with Su et al. 34 when learning from 8-word summaries in the unsupervised setting. When the inference length budget is less than training (æ-axis < 50), the ROUGE score of NACC decreases almost linearly with a decreasing length budget, but Su et al. 34 is approach degrades faster than ours. For z-axis > 50, we find the soft penalty in 34 is unable to generate longer summaries than trained (shown by the dashed orange line), whereas our approach is able to utilize the increased length budget and achieve higher ROUGE scores.
Case Study. Table shows example summaries generated by NACC and the AR Transformer on the Gigaword test set.
As seen, a model without length control may generate a summary that happens to have the desired
length (AR Transformer in the supervised setting), shorter summary (NACC in the supervised setting), or a longer summary (both AR Transformer and NACC in the unsupervised setting). A shorter summary fails to utilize the entire length budget, leading less information conveyed, whereas longer summary requires truncating for explicit length control. Both cases are undesired.
By contrast, our proposed algorithm is able to generate summary whose length is close to but less than the length budget. The resulting summary is more complete than truncating, and better keeps the key information.
5 Conclusion
In this work, we propose a Non-Autoregreesivve Summariation model with Character-level length Control (NACC), which can explicitly control the number of characters the predicted summaries. Experiments show that our NACC approach not only achieves the state-of-the-art non-autoregressive (NAR) performance on Gigaword and DUC2004 datasets under length constraints, but several times faster than autoregressive (AR) models and even outperforms an AR Transformer in the unsupervised

---

setting. Moreover, our approach is able to perform length trans--ttanferrrr generation, that is, generating summaries of different lengths from the training set.
Limitation and Future Work. Our proposed length-control algorithm only works with non- autoregressive models, which is limitation (but not disadvantage) of our work. We have clearly stated this throughout the paper.
Our NACC approach-althouse largely outperforming previous NAR models and retaining high inference efficiency-achiewe lower ROUGE scores than AR models the supervised setting. This is the limitation of all NAR approaches in general, and can be addressed by future work. We nevertheless outperform AR models in the unsupervised setting.
Appendices
The full paper, including appendices, is available at attttsss/aar.....00///////
Acknowledgments
We thank all reviewers for their valuable comments. The research is supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) under grant No. RGPIN2020-04466 the Amii Fellow Program, the Canada CIFAR Al Chair Program, a UAHJIC project, a donation from DeepMind, and the Digital Research Alliance of Canada (alliancecan.ca).
References
[1] Rumen Andonov, Vincent Poirriez, and Sanjay Rajopadhye. Unbounded knapsack problem: Dynamic programming revisited. Europeun Journal of Operational Research, 123(2):394-40) 2000.
[2] Christos Baziotis, Ion Androutsopoulos, Ioannis Konstas, and Alexandros Potamianos. SEQ3: Differentiable
autoencoder for unsupervised abstractive sentence compression. In NAACL-HLT, pages 673-681, 2019.
[3] Ethan A. Chi, Julian Salazar, and Katrin Kirchhoff. Align-refine: Non-autoreerressvve speech recognition via iterative realignment. NAACL-ILLT, pages 1920-1927, 2021.
[4] Yuntian Deng and Alexander Rush. Cascaded text generation with Markov Transformers. In NeurlPS, pages 170-181, 2020.
[5] Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. Ban- ditSum: Extractive summarization as contextual bandit. In EMNLP, pages 3739-3748. 2018.
[6] Bonnie Dorr, David Zajic, and Richard Schwartz. Hedge trimmer: A parse-and-trim approach to headline generation. In Proc. NAACL-HILT Text Summarization Workshop, pages 1-8, 2003.
[7] J.-B. Durand, P. Goncalves, and Guedon. Computational methods for hidden Markov tree models-an application to wavelet trees. IEEE Transactions Signal Processing, 52(9):2551- 2560, 2004.
[8] Katja Filippova, Enrique Alfonseca, Carlos A. Colmenares, Lukasz Kaiser, and Oriol Vinyals. Sentence compression by deletion with LSTMs. In EMNLP, pages 360-368, 2015.
[9] Sebastian Gehrmann, Yuntian Deng, and Alexander M Rush. Bottom-up abstractive summariza- tion. EMNLP, pages 4098-4109, 2018
[10] David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. English Gigaword. Linguistic Data Consortium, 2003.
[11] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In ICML, page 369-376, 2006.
10

---

[12] Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. Non- autoregressive neural machine translation. In ICLR, 2018.
[13] Chenyang Huang, Hao Zhou, Osmar R Zaïane, Lili Mou, and Lei Li. Non-autoregressive translation with layer-wise prediction and deep supervision. In AAAI, pages 10776-10784, 2022.
[14] Frederick Jelinek. Statistical Methods for Speech Recognition. MIT Press, 1998.
[15] Mikael Kägebäck, Olof Mogren, Nina Tahmasebi, and Devdatt Dubhashi. Extractive summa- rization using continuous vector space models. In Proc. Workshop on Continuous Vector Space Models and their Compositionality, pages 31-39, 2014.
[16] Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic non aan-aoorrrgrssivee neural sequence modeling by iterative refinement. In EMNLP, pages 1173-1182 2018.
[17] Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, 2004.
[18] Puyuan Liu, Chenyang Huang, and Lili Mou. Learning non-autoregressive models from search for unsupervised sentence summarization. In ACL., pages 7916-7929, 2022.
[19] Yixin Liu, Zi-Yi Dou, and Pengfei Liu. RefSum: Refactoring neural summarization. In NAACL, pages 1437-1448, 2021.
[20] Yizhu Liu, Zhiyi Luo, and Kenny Zhu. Controlling length in abstractive summarization using a convolutional neural network. In EMNLP, pages 4110-4119, 2018
[21] Afonso Mendes, Shashi Narayan, Sebastião Miranda, Zita Marinho, André T. Martins, and Shay B. Cohen. Jointly extracting and compressing documents with summary state representa- tions. In NATCL-HLT, pages 3955 -366,, 2019.
[22] Ning Miao, Hao Zhou, Lili Mou, Rui Yan, and Lei Li. CGMH: Constrained sentence generation by Eestropolis-Hastinge sampling. In AAAI, pages 6834-688 2019.
[23] Rada Mihalcea and Paul Tarau. TextRank: Bringing order into text. In EMNLP, pages 404-411, 2004.
[24] Gabriel Murray, Steve Renals, and Jean Carletta. Extractive summarization of meeting record- ings. In European Conference Speech Communication and Technology, pages 593-596, 2005.
[25] Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. Ab- stractive text summarization using sequence-to-sequeee RNNs and beyond. In CoNLL, pages 280-290, 2016.
[26] Ani Nenkova, Kathleen McKeown, et al. Automatic summarization. Foundations and Trends in Information Retrieval, 5(2-3):103-233, 2011.
[27] Paul Over and James Yen. An introduction DUC-2004: Intrinsic evaluation of generic news text summarization systems. In Proc. Document Understanding Conference, 2004.
[28] Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. In ICLR, 2018.
[29] Weizhen Qi. Yeyun Gong, Jian Jiao, Yu Yan, Weizhu Chen, Dayiheng Liu, Kewen Tang, Houqiang Li, Jiusheng Chen, Ruofei Zhang, Ming Zhou, and Nan Duan. Bang: Bridging autoregressive and auo-euttreeressivve generation with large scale pretraining. In ICML, pages 2021.
[30] Alexander M. Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. In EMNLP, pages 379-389, 2015.
[31] Chitwan Saharia, William Chan, Saurabh Saxena, and Mohammad Norouzi. Non-autoregressive machine translation with latent alignments. In EMNLP, pages 1098-1108, 2020.

---

[32] Itsumi Saito, Kyosuke Nishida, Kosuke Nishida, Atsushi Otsuka. Hisako Asano, Junji Tomita, Hiroyuki Shindo, and Yuji Matsumoto. Length-controllabe abstractive summarization by guiding with summary prototype. arXiv preprint urXiv:2001.0733// 2020.
[33] Raphael Schumann, Lili Mou, Yao Lu, Olga Vechtomova, and Katja Markert. Discrete opti- mization for unsupervised sentence summarization with word-level extraction. In ACL, pages 5032-5042 2020.
[34] Yixuan Su, Deng Cai, Yan Wang, David Vandyke, Simon Baker, Piji Li, and Nigel Collier. Non-autoreerresivve text generation with pre-trained language models. In EACL, pages 234-243, 2021.
[35] Zhiqing Sun, Zhuohan Li, Haoqing Wang, Di He, Zi Lin, and Zhihong Deng, Fast structured decoding for sequence models. In Newr/PS, pages 3011-3020, 2019.
[36] Sho Takase and Naoaki Okazaki, Positional encoding to control output sequence length. In NAACL, pages 4999-4004, 2019.
[37] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, pages 2017.
[38] Yaushian Wang and Hung-Yi Yi Lee. Learning to encode text as human-readable summaries using generative adversarial networks. In EMNLP, pages 4187-4195, 2018.
[39] Peter West, Ari Holtzman, Jan Buys, and Yejin Choi. BottleSum: Unsupervised and self- supervised sentence summarization using the information bottleneck principle. In EMNLP. IJCNLP, pages 7762-3761, 2019.
[40] Jiacheng Xu and Greg Durrett. Neural extractive text summarization with syntactic compression.
In EMNLP-IJCNLP, pages 3292-3303, 2019.
[41] Kexin Yang, Wenqiang Lei, Dayiheng Liu, Weizhen Qi, and Jiancheng Lv. POS-constrained parallel decoding for non-autoregressive generation. In ACL-IJCNLP, pages 5990-6000, 2021.
[42] Ziyi Yang, Chenguang Zhu, Robert Gmyr, Michael Zeng, Xuedong Huang, and Eric Darve. Ted: A pretrained unsupervised summarization model with theme modeling and denoising. In EMNLP, pages 1865-1874, 2020.
[43] Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. ICML, pages 1322-11139,, 2020.
[44] Jinming Zhao, Ming Liu, Longxiang Gao, Yuan Jin, Lan Du, He Zhao, He Zhang, and Gho- lamreza Haffari. SummPip: Unsupervised multi-document summarization with sentence graph compression. In SIGIR, pages 1949-1952, 2020.
12


Reviews:
[{"review_id": "vb1p5MlOIt", "reviewer": "Reviewer_dLJR", "summary": "The authors proposed a method to control the summary length by characters in non-autoregressive sentence summarization models. The proposed method utilizes dynamic programming-based decoding to follow the length limitation as an imposed constraint. Since considering all possible states in the decoding is intractable, the authors separately consider the length states by relying on buckets. Experimental results showed that the proposed method could control the summary length by keeping ROUGE scores. Furthermore, human evaluation results also showed that the proposed method outperformed the truncation-based length controlling.", "questions": "- In experiments, you prepared four buckets to reduce computational time in decoding. However, the inference time of a single bucket shown in Figure 2 seems practical. How did you decide on the bucket size in the experiments?\n- Comparison with the autoregressive models in the unsupervised setting showed that NACC achieved comparable ROUGE scores with the AR methods. It indicates that NACC is effective in terms of informativeness. On the other hand, the current setting lacks the comparison between the proposed and AR methods in terms of fluency. Is there any information about this point?", "limitations": "The proposed method NACC is on the basis of non autoregressive decoding, which may be inferior to autoregressive decoding in fluency.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- Exact length control based on dynamic programming.\n- Length can be controlled while maintaining summary performance.\n\nWeakness:\n- Lack of exact solution due to the use of buckets for the proposed method to run in practical time.\n- Lack of human evaluation for the proposed method and conventional AR methods.\n- Controlling summary length by a non-autoregressive model has already been conducted in the deletion-based sentence compression task. [1]\n\n[1] Wang, Liangguo, et al. \"Can Syntax Help? Improving an LSTM-based Sentence Compression Model for New Domains.\" Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2017.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZsJV0iMxB_m", "reviewer": "Reviewer_aypC", "summary": "The paper proposes a character-length-constrained sentence summarization model by extending the length constraint from word-level to character-level in inference. The major technical part follows NAUS [Liu et al, ACL’22] which is a BERT-encoder with a CTC decoder. The main contribution is extending the NAUS’s word length constraint from word level to character level, which is formalized as a standard 0-1 knapsack problem with a DP solver. The experiments on Gigaword headline generation and DUC2004 datasets demonstrate the performance over several truncation models in terms of ROUGE scores. ", "questions": "Please see Limitations.", "limitations": "1. The bucket grouping trick (lines 142-147, page 4) has some limits: (1) it would not reduce the theoretical complexity, i.e., quadric remaining as the 2D programming; (2) it may suffer from heavy collisions, as the skewed long-trail distributed character counting makes most words share the same bucket. \n\n2. The idea and the technical part seem quite similar to NAUS (both using DP solution), it would be better to clearly state what the main novelty is compared with word-level constraints (not only with 0-1 knapsack problem) and in what points make methodologically is challenging.\n\n3. Besides the truncation model, the method should be compared with the non-truncation model, i.e., NAUS. \n\nMinor issue:\nIn Theorem 1. “See the appendix for the proof.” -> no appendix found.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\nThe write-up is good, and the paper is easy to follow\n\nCons:\n1. The character-level constraint for summarization is not well-motivated. In common sense,  character counts are positively correlated with character counts, especially considering most words share similar character counts as the long-tail distribution. \n\n2. Limited technical novelty. The model heavily follows NAUS’s ([Liu et al, ACL’22]) DP-based word-length-constraint method. The idea of extending from word-level to character-level is trivial. \n\n3. Lacks comparison with the non-truncation model, i.e., NAUS.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XGPXDrjKVS8", "reviewer": "Reviewer_5WDy", "summary": "This paper proposed a dynamic programming algorithm based on the Connectionist Temporal Classification (CTC) model for controlling the  summary length in text summarisation tasks. This is important in many tasks such as headline generation. Experiments on DUC2004 and Gigaword showed the effectiveness of this method. This paper might be more appropriate for NLP conferences. \n", "questions": "For unsupervised text summarisation, there is also a length control mechanism, e.g. do clustering for sentences and choose the centroid or do text compression for each cluster, so the number of clusters is proportional to the summary length. See [1]. Apart from the truncate methods, it would be better to add such baseline methods. \n\n\n[1] Zhao, J., Liu, M., Gao, L., Jin, Y., Du, L., Zhao, H., Zhang, H. and Haffari, G., 2020, July. Summpip: Unsupervised multi-document summarization with sentence graph compression. In Proceedings of the 43rd international acm sigir conference on research and development in information retrieval (pp. 1949-1952).", "limitations": "* The domain of the datasets are limited to news.\n* Some experiments with large pre-trained models (e.g. T5, BART) and CTC may be considered. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Leverage the CTC method for controlling the summary length in text summarization. \n\nWeakness:\n* For evaluation ROUGE is considered limited, some other semantic focused metrics like BertScore may also be considered.\n* DUC-2004 and Gigaword are small news datasets, the summaries are more lying in the first few sentences, it would be better if the authors can try larger datasets from other domains, e.g. arXiv scientific paper title generation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "N3XGjiv0JX", "reviewer": "Reviewer_BRgS", "summary": "This paper investigates the problem of length-controlled sentence summarization and proposes NACC, a Non-Autoregressive summarization model with Character-level length Control based on a knapsack-like algorithm. The authors conduct an evaluation of the Gigaword headline generation and DUC2004 datasets in both supervised and unsupervised settings and the experimental results show that the proposed framework can obtain comparable results to the baseline methods.\n", "questions": "na", "limitations": "The authors  addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. Length-controlled text summarization is a useful task in practice especially meets the headline generation tasks.\n2. The motivation and assumption are reasonable. A knapsack-like problem setting and non-autoregressive framework are interesting. The proposed method is straightforward and easy to be implemented. \n3. The experimental comparisons and analysis are sufficient and human evaluation is also conducted.\n\nWeakness:\n1. This work only conducts experiments on Gigawords and DUC2004, more experiments on more datasets are more concrete.\n2. From the results tables we could observe that the results are much worse than some traditional auto-regressive methods. For example, on the test set of Gigawords, previous works can obtain some better results: 36, 17, and 33 for R-1, R-2, and R-L respevtively. Therefore I am wondering about the real quality of the generated sentences, though the length is a restricted condiction, the quality is more crucial.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept