CASR: Generating Complex Sequences with Autoregressive Self-Boost Refinement
Authors: Hongwei Han
Mengyu Zhou
Shi Han
Xiu Li
Dongmei Zhang
Venue: ICLR 2023 poster (2023)
Keywords: self-boost refinement
complex answers
autoregressive generation

Abstract: There are sequence generation tasks where the best order to generate the target sequence is not left-to-right. For example, an answer to the Sudoku game, a structured code like s-expression, and even a logical natural language answer where the analysis may be generated after the decision. We define the target sequences of those tasks as complex sequences. Obviously, a complex sequence should be constructed with multiple logical steps, and has dependencies among each part of itself (e.g. decisions depend on analyses). It's a great challenge for the classic left-to-right autoregressive generation system to generate complex sequences. Current approaches improve one-pass left-to-right generation on NLG tasks by generating different heuristic intermediate sequences in multiple stages. However, for complex sequences, the heuristic rules to break down them may hurt performance, and increase additional exposure bias. To tackle these challenges, we propose a PLM-friendly autoregressive self-boost refinement framework, CASR. When training, CASR inputs the predictions generated by the model itself at the previous refinement step (instead of those produced by heuristic rules). To find an optimal design, we also discuss model architecture, parameter efficiency and initialization strategy. By evaluating CASR on Sudoku, WebQSP, MTOP and KVRET through controlled experiments and empirical studies, we find that CASR produces high-quality outputs. CASR also improves Accuracy on Sudoku (70.93% --> 97.28%) and achieves state-of-the-art performance on KVRET with Micro F1 score (67.88% --> 70.00%).

Published as a conference paper at ICLR 2023
CASR: GENERATING COMPLEX SEQUENCES WITH AUTOREGRESSIVE SELF-BOOST REFINEMENT
Hongwei Han'
Mengyu Zhou?t Shi Han" Xiu Li Dongmei Zhang
'Tsinghua Shenzhen International Graduate School, Tsinghua University
2 Microsofft Research
hhw200mai testtingiii tsinguua... edu en, 11 xi xiulsz. wisssz.tsinghua. edu. mezho shihan, dongmei @mmi www.orr.tccoooooooooooooooo com
ABSTRACT
There are sequence generation tasks where the best order generate the target sequence is not left-to-right. For example, an answer to the Sudoku game, a struc- tured code like s-expression, and even logical natural language answer where the analysis may be generated after the decision. We define the target sequences of those tasks as complex sequences. Obviously, complex sequence should be constructed with multiple logical steps, and has dependencies among each part of itself (e.g. decisions depend on analyses). It's great challenge for the classic left- to-right autoregressive generation system to generate complex sequences. Current approaches improve one-pass left-to-right generation on NLG tasks by generating different heuristic intermediate sequences in multiple stages. However, for com- plex sequences, the heuristic rules to break down them may hurt performance, and increase additional exposure bias. To tackle these challenges, friendly autoregressive self-boost refinement framework, CASR. When training, CASR inputs the predictions generated by the model itself at the previous re- finement step (instead of those produced by heuristic rules). To find an optimal design, we also discuss model architecture, parameter efficiency and initializa- tion strategy. By evaluating CASR on Sudoku, WebQSP, MTOP and KVRET through controlled experiments and empirical studies, we find that CASR pro- duces high-quality outputs. CASR also improves Accuracy Sudoku (70.93% + 97.28%) and achieves state-of-the-arr performance on KVRET with Micro F1 score 67% 70.00%).
propose PLMM
INTRODUCTION
Design
Model Architecture
Encoder
Encoder
Encoder
SepEnc, UniEne
Parameter
Decoder
Decoder
Decoder
Efficiency
0
Finetune, Adapter
X
Y
X
Y
(X,
82)
Y
(X, X2
Initialization
Castep 1=0
Castep 1=1
Castep 1=
Strategy
Restart, Continue
Tasks with complex answers, like Sudoku, WebQSP, MTOP, and KVRET
Figure The Overview of CASR Framework. X, Y and Ý denote the input, ground truth and prediction, respectively. The blue arrows show how we iteratively added back the previous-step prediction :t-1 to the input for generating refined output ÿt!
The contributions by Hongwei Han have heen conducted and completed during their internships at Microsoft Research Asia, Beijing. China. Corresponding authors.

---

Published as a conference paper at ICLR 2023
Sequence generation models are widely used in tasks related to natural, domain- and pro- gramming languages E.g., question answering (Pandya & Bhatt 202[). neural machine transla- tion (Yang et al. 2020), speech recognition (Malik et al. (2021). automatic data analysis (Zhou et al. 2020). drug discovery (Kim et al., 2021), document summarization (Ma et al. 2020), code search 2021). e/c.
and generation (Lee et al.
To achieve better performance on these tasks, recent works often adopt autoregressive (AR) mod- els (Wu et al. (2016), especially the ones with one-pass L2R (left-to-right) token-by-token gener- ation / decoding order. Many CoTA-pperfrmmace generative PLMs (pre-trained language models) one-pass
are one- pass L2R models, such as GPT (Radford et al. 2018). T5 (Raffel et al., 2020j. Bart (Lewis) let al., (020), etc. Different from -oo--utoreeressive (NAR) models (Gu et al. 2017) which as- sume independence among tokens, 1.2R models assume the conditional probability in the form of X) = II: P(uil Ves) which better captures the left-side dependencies that exist in most generation tasks. More variations of generation models are discussed 2.
However, for many sequence generation tasks, beyond the left-side dependencies. there are right-side dependencies in the answer se- quence, which together lead to multi-hop de- pendency chains, making left-to-right not the best order for generation. We call these tasks as complex tasks, and the answer sequences of these tasks as complex sequences. Com- plex tasks (see more details in $2.22 including Sudoku (PARK). WebQSP (Yih et al. 2016). MTOP (Li al. 2021). and KVRET (Eric et al.) 2017). etc..) require better generation mechanism beyond one-pass 1.2R generation, since complex sequences are usually long, dif- ficult, structured, or logical, which should be constructed with multiple logical steps.
6 9
2
4 8
7
3
5
3d
1
4b
2
1 8
3
5
2
4
6
4
2c
3
1a
5 7
4
3
6
9
2
8 2 4 1 3 7
4 6
7
3
9
2g
4
1
3
3 2 9
8
6
1
7
1
3
2e
4
7 5 8
2
4
9
3
6
(h) A
4x4
9 1
5
6
7
4
2 4
8 1
Sudoku Exam- ple. The game solving order of a human is from "a" to "g", rather than row-by-row.
8 3
6
9
2
5
2 4
1
8
6
5
(a) 9x9 Sudoku Example. White cells denote blanks, and the green numbers in them de- note the ground truth.
Human beings solve a complex problem with respect to its intrinsic order. For example, the order to write hierarchical answers (such as s- Expression SQL code) is usually bottom-up or top-down following the dependencies between components as discussed by Sun et al. (2020). The order to give an NL response is first analyses then decisions discussed by CoT (Wei et al. (022). The order to solve puzzle (such as the example 4x4 Sudoku game in Figure 2 is usually from easy parts to hard parts, because the hard parts become easier when the easy parts are correctly solved. (That is also verified in $5.T where our CASR model learns to solve easy parts before hard ones.) Obviously, people give answers to different tasks in various orders with respect all kinds of dependencies.
Figure 2: Examples of Sudoku.
Mimicking human behavior, some existing works design specific intermediate sequences to solve the dependency order challenge. E.g., templates (Hua & Wang 2020) or heuristic rules (Zhang et al., 2018 Tan et al.] 2021) are applied in autoregressive NI. generation, allowing models to generate some parts (intermediate sequences) before the other parts in an answer via iterative refinement (rather than one-pass decoding). However, 1) it's really hard to design the best heuristic order and easy to miss intrinsic dependencies for some tasks, and we need expert knowledge or manual efforts to design specific heuristic orders for all different tasks. Besides, 2) when using teacher forcing strategy to parallel train all refinement iterations, additional exposure bias occurs.
In this paper, CASR (Generating Complex Sequences with Auto Aego-segressive Self Boost Refinement) framework is proposed by us to: 1) decide intermediate sequences of complex answers for different tasks in data-driven way, 2) avoid additional exposure bias. As shown in Figure [ and will be discussed in $3 in CASR we design a model architecture ($3.2) to not only take in the original input X, but also the previous prediction p--11 in both training and inference. A special process ($3.1 is designed to train refinement models A/ for each step t == 0, 1, ..... - To enhance the performance on downstream tasks, CASR models could be initialized with pre-trained language models ($3.3) such as T5 (Raffel et al. 2020), and even trained in "Continue" way ($3.4) by initializing M/
2

---

Published as a conference paper at ICLR 2023
with M For CASR models to be more efficient, we also explore the parameter-efftiien model designs (SepEnc vs. UniEne in $3.2). and tuning with parallel adapters 3.3
We evaluate CASR (and several baselines) WebQSP, MTOP, KVRET, and Sudoku in $4 achiev- ing SOTA performance on KVRET. In detail, CASR improves Fl on WebQSP from 70.81 to 74.81, 1, EM on MTOP from 78.64 to 81.92, Micro F1 KVRET from 67.40 70.0d and accuracy on Sudoku from 70.93 to 97.28. We find the optimal CASR design (44.) is "Fine-tuning" "SSeEnn" with "Continue" strategy. We also do empirical studies on complexity ($5.1), attention map ($5.2), and visualize cases (5..) Then we find that, CASR benefits hard sequences more than easy ones, and CASR can indeed correct the wrong part of the previous prediction according other parts depends on (as shown in Fig
In summary, our major contributions are: First, we point out the challenge to generate com- plex sequences due to the existence of multi-hop dependency chains and conduct a comprehen- sive review on existing iterative refinement methods. Second, we propose an autoregressive self- boost refinement framework, CASR, to decide intermediate sequences of complex answers in data-driven way. The code of CASR framework open sourced in the repository at hhttps 7/github. com/ com//alll RRRlppHaaan CASR Third, we conduct experiments and empirical studies on four complex tasks to show CASR works and interpret how it works.
2
RELATED WORK
Table 1: The Objectives of Sequence Generation Methods.
Table 2: The Number of Samples in Train, Dev, and Test Splits of Web- QSP. MTOP, KVRET,
Method
Ohjective
NAT (Gu et al.
2017)
II P(y]
INAT (Lee al 2018
Levensiten Gu
2019
Initate an expert policy to delete and insert
and Sudoku.
1.2R (Wu et al. 2016
Il P(yl (11
XLNNc Yang ci al. 2019
11 Plys, 3.g.... May)
Task
Train
Test
Bidirectional (Zhang et
2018
II. P(yd X,8 11 Bn) II, P(gi X. tr 01,
WebQSP
2673 309
1639
Progressive
Tan et al.
II. P(yf y1-) o/-
MTOP
15667
2235 4386
KVRET
6291
777
808
Ours
11 P6/1X
Sudoku
800K
100K
100K
2.1 SEQUENCE GENERATION METHODS
In recent years, many studies have been made seq2seq generation. For input X and target Y sequences, the ultimate objective is to maximize P(Y As shown in Table [ different method formulates the objective differently. These methods be divided into non-autoregressive (row 1-3) and autoregressive (row 4-7) ones. Highly related ones are run as baselines in Table
NAT (Gu et al., 2017) assumes that each token in sequence Y is mutually independent from each other, thus formulating XX) as II, P(yi(X) To reduce the modeling bias in NAT, iterative refinement is applied
answers in INAT (Lee al., 2018). Levenshtein Transformer (Gu et al., 2019) further breaks down each refinement step into deletion, insertion, and classification so as to allow the model to edit the generation in a non-autoregressive way. Despite their efficiency, non- autoregressive models perform relatively poor comparing to autoregressive ones (see $4.4).
L2R (Wu al. 2016) generation tries decode from left to right, which also adopted by GPT RRdffrd et al., 2018), basic transformer decoder (Vaswani et al.] 2017). VAE decoder (Yu al. 2020), and RNN decoder (Xu et al. 2015 Xia et 2017). XLNNet YYang et al. 2019) allows the model to generate with any given order Z. but cannot decide by itself the best order. Bidirec- tional decoder (Zhang et al. (2018) generates backward (R2L) then forward (L2R), which handles at most 2-step logic. It trains additional R2L decoder from scratch, without leveraging generation capabilities of PLMs as in CASR (see $3.3)
PAIR (Hua& & Wang 2020) is used in controlled text generation with template (constructed from a set of provided key phrases and placeholders) as input. It fills and refines the placeholders in an auto-regressive way. The template, the sequence length and position of placeholders remain
77000 achieves the SOTA for micro fl on KVRET, and the previous SOTA 67.88 (ie al. 2022
3

---

Published as a conference paper at ICLR 2023
unchanged during refinement. Similarly, chain of thought prompting (Wei et al. 2022) constructs the intermediate process through manually provided templates. These two methods are not chosen as baselines in $ because they both require expert designs for each specific task.
Progressive generation (Tan et al. 202T) is proposed to remove the requirement of templates and the limitation of fixed length during refinement. It breaks down the vocabulary into multiple stages according to word importance (average of tf-idf). Important words are generated first during early refinement steps, while other words (both important and unimportant) are generated later. Note that in Table y (row 7) denotes the intermediate sequence produced by the heuristics of progressive generation, but y-t-1 (row 8) is the previous prediction generated by our CASR model (see
2.2 TASKS WITH COMPLEX ANSWERS
The four complex-answer tasks referred in will be further introduced in this section.
WebQSP Yih et al. 2016) is a classic dataset for KBQA(Knowledge Base Question Answering). The input consists of a knowledge graph and NL query, and the output is s-Expression which can be executed on the knowledge graph. The SOTA method of WebQSP (F1=83.6%) is ranking with bootstrapping negative samples (Ye al. 2022).
MTОP (Li et al. 2021) is benchmark for comprehensive multilingual task-oriented semantic pars- ing. The input consists of list of API calls and NL query, and the output is tree-based TOP Representation that can be executed.
KVRET (Eric al. 2017) is benchmark for table conversation. The input consists of a table and an NL query, and the output is an NL response corresponding to the dialog. The SOTA method of MTOP (EM=86. 78%) and KVRET (Micro F1=67.88%) multi-task (20+ tasks) prefix tuning with T5-3B as the backbone (Xie et al. 2022).
Sudoku (PARK) an open dataset on Kaggle. Its game target is fill the blanks correctly with the constraint that any two numbers in the same row, column, and house shouldn's 't have the same value. We choose Sudoku as an intuitive toy task for better demonstration of ideas in this paper.
Examples of WebQSP (Figure 5). MTOP (Figure 6) and KVRET (Figure 7 are shown in Appendix The number of samples each split of the four tasks is shown Table 2
AA
3
CASR FRAMEWORK
As shown in Figure I and introduced in 1 and 2.1 the key idea of CASR framework is to take the previous- step prediction ý-t-1 part of the current-step (t) input for generating refined output Y The iterative inference process in CASR follows the idea: As shown in Algorithm [ the prediction eastep is Y (0 < t < T) generated by the corresponding CASR model M where the input M is the original input X, and the input to M > 0) is (X, yt-1). Following the notions in 2.11
X, Y, and denote the input, the ground truth and the prediction, respectively. The iterative refinement process in CASR will take at most T casteps (CASR steps).
The undetermined M' models in the CASR inference process lead to more questions: 1) How to design the training process and objectives that match the inference process? 2) How design CASR model architecture that leverages existing PLMs and takes the extra ýtt-1 as input? When leveraging a large PLM, can we train and save CASR models M with less parameters? 4) Can the training and inference process be more effective by exploiting the relationships among CASR models M of each castep? In the following, we will discuss these problems one by one.
3.1 TRAINING PROCESS
As formulated in Algorithm 2 the training process is also iterative.
At the 0-th castep, we train the model M normally on train set with the objective to maximize P(Y X) (as discussed in 12.1). When the training is done, we use A to generate prediction ÿ-o for each X in train and dev sets with beam searching (Ney et al. 1987).
4

---

Published as a conference paper at ICLR 2023
At the t-th castep (0 T), train M' to refine the last prediction y The objective becomes maximizing (07] train set. After training, generate y all train and dev sets compose the samples (X, Y, YY)) for the next castep 1+1.
After the training process, T versions of CASR model M (0 S 1 < T) are available for the inference process. Because the distribution of yt-1 could change with t, and different casteps may learn different refinement patterns, by default all the versions are saved.
In the following, we will discuss more details of the algorithms on M' model architecture ($ 3.2 tuning ($.3), and initialization (3.4). More discussions on result selections are Appendix
3.2 MODEL ARCHITECTURE
SepEnc+Fine-tume y
UnlEnc+Fing-tun Y
X
X
E1
E2
E
D
D
[-0
1-,,,...,TT T-1
SepEnc+ Adapter
AdiEneeAAdapte
Restart
PL.M
Train and Predict
Train and Predict
Truin Predict
You
X
Y
Scratch
AEI
E
AE2
E
AE
E
Continue
PLM
Train and Predict
Train and Predict
Train and Prediet
Scratch
D
D
The arrows begin at the source of initial weights of M
AE and AD denote adapters of encoder and decoder. and gray segments are the parameters to be trained and frozen, respectively.
Figure 4: Initialization Strategies.
Figure Model Architectures and Parame- ter Efficiency Choices.
The next problem after the training process is how to design model architecture which could take (X, yt-1) input and generate Yt. Starting from any Transformer encoder-decoder (Vaswani ct al. 2017) based model (such as T5, or an untrained Transformer), CASR we provide two modification approaches to take y-t-1
As shown in Figure SepEnc has two encoders to encode X and ÿ separately (encode be- fore concat), and UniEne adopts only encoder for both inputs (concat before encode). The output sequence length of SepEnc and UniEnne are both len(X) len(Ý"(-1). Let H denote the encoder output. For SepEne, Il Concat (X), Encodery ((hhoder(((tt-1)) and for UniEnc, H E.w.EeeneCCCCac (Conoaee fr--1))
Both approaches have their own advantages: SepEnc naturally handles the distribution differences between X and rr- while UniEnc requires fewer model parameters. UniEnc forces one encoder to handle two kinds sequences with potentially different types of contents and lengths. As trade- off, SepEne brings two times larger encoding part. And the decoder generates answers from H, just like the classic transformer decoder (Vaswani et al. 2017).
3.3 PARAMETER EFFICIENCY
Both SepEnc and UniEne approaches in $3.2 could take a PLM (pre-trained language model, such as T5) as starting point for model parameter initialization (more details in $3.4). In this way, CASR could leverage the existing knowledge from the PL.M to enhance performances on downstream tasks (e.g., the ones in 2.2)
However, it's costly to fine-tune all the parameters from a large PLM and save the parameters of all the 7' CASR models M Thus, besides the fine-tuning approach, in CASR we also try parameter- efficient tuning. As shown in Figure 3 parallel adapters (He et al. (021) could be added to every encoder and decoder. By adopting adapters, the parameters from the PLM frozen, and only the parameters of the adapters are trained. In other words, one only needs to save the adapter (rather than the whole model) parameters for times.
5

---

Published as a conference paper at ICLR 2023
3.4 INITIALIZATION STRATEGY
In Algorithm 2 at the beginning of each castep, we initialize the parameter weights of M with chosen strategy from (Restart, Continue]. Illustrations of each strategy are shown in Figure 4
"Restart" means initializing each M1 with the same PLM weights (or from scratch). For SepEnc architecture, Both Encoder, and Encoder2 can be initialized with the parameter weights of the PLM encoder. "Continue" means initializing with the best checkpoint from the previous castep. In this way, M inherits the knowledge from M to avoid the potential cold start issue at castep
In the "Restart" strategy, when comparing the late castep with the early castep, the improvement only brought by refinement, rather than the extra continuous training steps (which exist in the "Continue" strategy). Note that, we design "Restart" mainly to decouple the effects of refinement from training steps, which is necessary for controlled experiments $4
4 EXPERIMENTS
In this section, we run controlled experiments to evaluate CASR designs ($3) on WebQSP Yih et al. 2016), MTOP (Li et al. 2021), KVRET (Eric al., (017) and Sudoku (PARK) 2.2). and compare CASR with high-related baselines 2
4.1 EXPERIMENT SETUP
In this section, we explain how we design the controlled experiments and reproduce the high-related baselines. For more details about hyperparameter and resource consumption (number parameters, training/inference time, etc.) please see Appendix $C
4.1.1 CONTROLLED EXPERIMENTS
In 3 apply CASR (with max castep T' = 3)
Table 3: Controlled Experiments of CASR Framework WebQSP,
MTOP, and KVRET.
on the 24-layer generative PLM T5- base (Raffel et al.] 2020 Controlled exper- iments A~E are run to compare the variations of Parameter Efficiency ($3.3) Initialization Strat-
CASR Variations
WebQSP
MТОP
KVRET
Param.E.
Arch.
Acc.
EM
BLEU
Micro AVG
A0
Fine-tune
Restart SepEnc
B0
Fine-tune Continue Sepline
70.81
82.49
78.64
18.33
67.40
63.53
CO
Fine-tune Restart
UniEne
DO
Fine-tune Continue
UniEne
50
Adapter Restart
SepEne
70.20
77.34
73.10
16.38
65.59
60.52
A1
Fine-tune Restart SepEnc
73.03
82.92
78.93
18.17
67.65
64.14
egy ($3.4) and Model Architecture ($3.2). The evaluation metrics (on test sets) are F1 score for WebQSP, template accuracy and exact match for МТОP, and BLEU and Micro Fl scores for KVRET. For each row, the average (AVG) of all metrics (on test set) is calculated to provide overall merged score.
B1
Fine-tune Continue SepEnc
74.61
85.07
81.19
18.55
70.00*
65.88
CD
Fine-tune
Restart UniEne
73.14
82.83
78.80
18.80
67.88
64.29
D1
Fine-tune Continue UniEne
74.16
84.86
81.24
18.06
69.12
65.49
EC
Adapter Restart SepEne
67.92
77.43
73.21
16.60
65.86
60.20
A2
Finc-tune Restart SepEnne
73.09
82.92
78.93
18.26
67.70
64.18
B2
Fine-tune Continue Seplice
74.81
85.61
81.69
19.12
69.85
66.22
Fine-tune Restart UniEne
73.14
82.83
78.77
18.78
68.06
64.32
D2
Fine-tune Continue UniEne
74.32
85.66
81.92
18.34
68.80
65.81
E2
Adapter
Restart SepEne 68.13
77.45
73.23
16.63
65.66
60.22
The three row blocks (AO~EO, Al ADDEE and (A2 E2 correspond to M .M' and M²2 models
Castep = 0, 1, respectively. By definitions in $ row (A0)(D0) share the same result because initialization and architecture choices do not influence Mo models.
Since CASR converges on Sudoku with larger T, we evaluate separately. In Table 4 We train 12-layer encoder-decoder transformers from scratch with max castep T 5. The met-
Table 4: Sudoku Testing Results.
Restart Continue Continue w/o Pf-I
70.93
ric is accuracy (% correctly filled blanks). For a 9 x board,
77.00
85.71
84.28
the input and output sequences of a Sudoku game are both 81-
2 78.69
92.48
89.16
number serialized sequences. During training, we apply su-
79.11
95.66
91.13
4
79.21
97.28
92.09
pervision only blanks and apply constrained generation to

---

Published as a conference paper at ICLR 2023
make sure the model only predicts blanks. And we fill the predictions back to the blanks of the input as the combination of X and Y, at the beginning of each castep & > 0. In other words, use "Restart" and "Continue" correspond the same choices as C and D in Table 3 "Continue w/o yr-1" denotes that the previous predictions are not put back to the blanks of the input at each castep, which equivalent to training the origin model T times steps as usual.
special UniEnc encode the combination, with H -Enodeer (ombine(XX
4.1.2 BASELINE COMPARISONS
In Table 5 run some high-related baselines as introduced in 22.1 We run INAT and Levenshtein with the official code pro- vided by fairsed where self-attention layers of the encoder and the decoder are initialized with bert, leaving cross-attention lay- ers of the decoder trained from scratch. To decouple the effect from PLMs and make fairer comparison between NAR and AR methods, we add row Xc,, where the decoder of CASR is not initialized with T5 but from scratch (random weights), marked as "/Dec". In row L2) "CASR-L" denotes changing the backbone of row (B2) from TS-base to T5-large. For fair comparisons, the original backbones in Bidirectional Decoder (GRU) and Progressive Generation (Bart) implementations are changed to T5- ba-e (same as CASR), and set T = 3 (same as CASR) for Progressive Generation.
Table 5: Comparisons between CASR and Baselines.
"NAR" and "AR" denote con-autoregressive and autoregressive refinement.
0_01-" denotes the metrics are lower than 0.01.
WebQSP
MTOP
KVRET
Methods
F1
Acc.
EM
BLEU Micro FT
AVG
X6 XB
INAT (Lee et al
2018
01
0.00
NAR
Levenshtein (Cu et al: 2019
22.54
01
4.51
X6
CASR 8 Dec
4,08
63.02
12.93
7.18
10.17
19.48
Xd
Bidirectional Zhang et 2018
57.84
8.98
54.77
X6
AR
Progressive Tun
72.05
80,80
77.04
15.49
64.54
61.98
B0
Finetune
70.81
82.49
78.64
18.33
67.40
63.53
B2 M
CASR
74.81
85.61
81.69
19.12
69.85
66,22
CASR-I.
77.99
87.73
84.54
18.14
68.80
67.44
4.2 BEST CASR DESIGNS
We achieve the SOTA performance on Micro Fl of KVRET, which is 70.00. The previous SOTA is 67.88 (iee et al.) 2022) based on T5-3B, and we beat it with smaller T5-base.
For the parameter initialization choices, "Continue" strategy brings better performance than "Restart". This is observed by comparing A with B rows, C with ①) rows in Table 3 and com- paring "Restart" and "Continue" columns in Table
For the model architectures, by comparing A with rows, B with D rows, we find that there is no great performance gap between SepEnc and UniEnc, and SepEne is slightly better than UniEnc when combined with "Continue".
For the parameter efficiency, by comparing L with A rows, we can find that adapter-tuning perform worse than fine-tuning, As the cost of freezing PLM parameters for efficiency, performance drops as expected when bringing adapter-tuning.
4.3 ANALYSIS ON SELF-BOOST REFINEMENT
As introduced in $3.4 we design "Restart" to decouple the effects of refinement from fitting the training set longer than /=0, The variations with "Restart" strategy (A) C and P in Table 3 "Restart" column in Table 4 demonstrates how self-boost refinement could improve the quality of
generated sequences. Here the improvement is only brought by refinement since CASR models at all casteps are initialized with the same PLM/Scratch version and take the same training steps.
In Table comparing "Continue" with "Continue w/o ş-t-1" we find that self-boostly feeding yt- the model is indeed helpful, which improves the accuracy from 92.09% 97. 97.28%.
Interestingly, the performance gap between castep 0 and is much larger than the gap between t to + 1 (t > 0). One possible explanation is that later refinements have reached the upper bound.
https: wtt.:h/cww...cccooooeeeeeeeeeeee

---

Published as a conference paper at ICLR 2023
As in Table 4 Sudoku requires more refinement steps (5 in case) for the gap to converge. This shows there are more logical steps required and more dependencies exist in the generated Sudoku answers.
4.4 COMPARISON WITH BASELINES
find the results of NAR methods are not par with AR methods. With case study (Figure 10 we find the ability of NAR methods to learn syntax poor. Comparing row (h)) and (Xc) find the poorness of NAR methods doesn't come from the initialization of the decoder. Comparing row (xd. (Xe), and (BO, we find that those AR refinement methods using heuristic intermediate sequences are indeed harmful (worse than directly fine-tuning) for complex sequence generation. Compare row (B2) and (L2) we find larger backbone (T5-base + TS-large)
In the NAR block of Table
overall helpful (UnifiedSGG (Xie et 2022 observes the same phenomenon that T5-large worse than T5-bau base on KVRET).
In summary, following conclusions can be drawn for CASR:
Self-boost refinement leads to better outputs than vanilla AR (one-pass I.2R) generation, NAR refinement generation, and heuristic-nule-basel AR refinement generation.
In general, the best combination is "Fine-tuning" "SepEne" with "Continue" strategy.
5 EMPIRICAL STUDIES
Table 6: Result on Different Answer Length.
Produced by (D in Table (Finetun=+Contine SepEn)).
Table 7: The average diffi-
D. denotes the improvement from castep 1
culty and ratio (sum to 1) of each correct-solving castep, produced by the setting of col- umn "Continue" in Table
Task
WebQSP
MTOP
KVRET
Metric
F1
Ace
Match
Bleu
Micro Fl
AVG
A
Short
78.80
85.23
83.65
14.05
65.91
65.53
Middle
83.07
84.83
80.51
21.93
71.16
68.30
AVG Difficulty
Ratio
Long
50.51
77.31
71.62
15.08
65.69
56.04
82.80
63.54%
Short
80.12
87.21
85.50
12.17
65.93
66.19
86.95
18.48%
Middle
86.41
87.19
83.01
19.81
71.51
69.59 1.29
88.60
8.50%
Long
57.52
80.71
74.95
16.33
69.48
59.80 3.76
89.83
4.25%
Short
80.12
87.96
86.25
12.32
64.52
66.23
90.86
2.51%
Middle
86.41
87.53
83.34
19.77
71.79
69.77
0.18
S (failed)
92.21
2.72%
Long
58.13
81.26
75.36
17.00
69.18
60.19 0.39
In this section, we conduct empirical studies interpret how CASR works.
In $5.1 we verify that CASR brings more improvement on more complex sequence. Also, we find CASR model may implicitly learns dependency because it solves easy parts first during refinements.
In 5.2 the cross-attention map between ÿr and ÿ implies the informativeness and correctness of Meanwhile, from a microscopic perspective, each row the cross-attention map implies the foundation of changing component, where the hot-attended parts can be considered as the cause of changes.
In 5.33 through case studies we show the intermediate steps generated by CASR models.
5.1 ANALYSIS ON COMPLEXITY
CASR helps more problems with more complexity. For WebQSP, MTOP, and KVRET, assume that the complexity problem (X, Y) positively correlated with the length of its an- swer. Thus, divide each test dataset into "Short", "Middle" and "Long" equal splits according to len(Y), the length of ground truth answer. As we can see in Table 6 CASR brings the more improvement the problems with "Long" answers than "Middle" and "Short" ones.
Take Sudoku as example, we find that CASR solves easy blanks before difficult blanks during refinements. In late casteps, the generation benefits more from refinement by gradually handling
8

---

Published as a conference paper at ICLR 2023
complex dependencies. These shown in Table 7 As castep t increases, the average difficulty of remaining blanks increase. Here we formulate the difficulty of blank cell as: Dif ficulty = (r-1)- (c- 1) 1hh-1 1) where r, C, and h denote the number of blanks in the same row, column, and house of the blank. Also, we can define the correct-solved castep for a blank: If a blank is correctly filled at castep t and remains the same for the rest casteps, then the blank is solved castep & and t is the correct-solved castep for the blank. Otherwise, by default the correct-solved castep is set to T. The "Ratio" column in Table 7 means the percentage of blanks is correctly solved at castep
5.2 ANALYSIS ON ATTENTION
To measure to what extend does ÿ attend to X or ÿ in cross-attention, we define the density. The cross attention map, A, is tensor that displays how the tokens in Y attend to H. Therefore, the size of A is [num-layers, num-heads, len(Y"', len( (en(())+en((+-)))) We define the density to X and to - as: Dx A!.... (on(X)).mmant)) and Dyr- A[...,le((X) (en(X) :).mean()) which are the average attention weight among each layer, head, and token, for and for pt-1
Similar to Table 6 We sort the samples according to their Dy. 1. and divide test dataset into sparse, middle, and dense splits, each holding 1/3 of the total test set. We evaluate the metries of yt-1 on the three splits. We find the performance compute the average Dx and Dy over all samples in the test set, and find that D Dr.- grows with Therefore, f-1 becomes more and more informative thus be attended to by Y'. Please refer to Table and Table 11 in Appendix for more details.
ÿt-1 increases with De.-.. Besides, we
Meanwhile, we draw top-5 positions in cross-attention map (filter out non-blank cells and self) for each changed cell in Figure 8 As see, the corrected cell usually attends to cells in the same row, column, or house.
5.3
CASE STUDIES
As examples, the intermediate CASR prediction of each castep are shown in Figure of of SD We can find that as castep t increases, there is more green (keywords) and less red (mistakes) parts, which means ÿt becomes closer to Y. In the WebQSP case, the wrong part (marked red) of ÿd is deleted by late casteps. In the MTOP case, we find that CASR correctly generates the right part ("ISL:RECIPES INCLUDED INGR EDIENT dairyl")) of the answer before the left part ("[SL:MET HOD RECIPES recipel"), ) which one-pass L2R decoding cannot achieve. For Sudoku, an interesting phenomenon oceurs. Although the wrong predictions (marked red) become fewer with the growth of castep, some previous correct predictions are modified to wrong ones, and then modified back. This can be considered a kind of exploration.
Appendix
We also present the predictions from other experiments (baselines and CASR-L, as discussed in Table 5 in Figure 10 and we find it a challenge for NAR methods to generate complex sequences.
CONCLUSION
In this paper, CASR framework is proposed by us to generate Complex sequences with Autoregressive Self-Boost Refinement, which can decide intermediate sequences of complex an- swers in data-driven way with no need for expert knowledge or manual efforts. Through controlled experiments, we find the optimal design CASR is Fine-tuning SepEnc with Continue strategy. Also, the interpretability of CASR is enhanced via empirical studies on problem complexity and attention dependencies. CASR sheds more light on the sequence generation methods, especially on complex tasks (e.g. automatic data analysis and drug discovery), for future research.
'Difficulty: According to this definition, when blank is the only blank in a row, column, or house, the difficulty is 0, which makes sense because the blank can be directly solved. The larger and h are, the more difficult it solve the blank.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
Datasets This work collects the public dataset for research purposes. We believe there no pri- vacy issue, because Sudoku, WebQSP, MТOP, and KVRET accessible the public.
Models This work leverages T5-base and T5-large, which are pre-trained on C4, a clean corpus. Therefore, make sure that CASR will not produce discriminatory answers.
Computational Resources We train on 4 Tesla V100 GPUs, and the training and inference time consumption is show in g
which is acceptable considering the performance gain.
REPRODUCIBILITY STATEMENT
We describe experiment setup in 4.1 and provide the complete details such hyperparameter (in {C.J) and resource consumption (in CC.2). please see $C Meanwhile, we make our code open
source in this repository https: /tt///gttuuue wiith.b.coom com Ra Rwwp//aaaaaaaaaaaaaeaaaaaaaaaaaa CASR These resources should
be sufficient to reproduce results of the paper.
ACKNOWLEDGMENTS
This research was partly (WDZC3208832055500),
supported by
Shenzhen
Stable
Supporting
Program
REFERENCES
Mihail Eric, Lakshmi Krishnan, François Charette, and Christopher D. Manning. Key-value trieval networks for task-oriented dialogue. In Kristiina Jokinen, Manfred Stede, David Vault and Annie Louis (eds.), Proceedings of the 18th Annual SIGdial Meeting Discourse and Di- alogue, Saarbrücken, Germany, August 15-17, 2017, pp. 37-49. Association for Computational Linguistics, 2017. doi: 10.11863/vl/w// 17-5655 URL https:///oii http:/o.o...1111..11888...6331 w17-5506
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. Non- autoregressive neural machine translation. CoRR, abs/1711 10.021,,,, 2017. URL. http://
Brxiv.org/abs/11:
02281
Jiatao Gu, Changhan Wang, and Junbo Zhao. Levenshtein transformer. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Achéé-Bue, Emily B. Fox, and Roman Garnett Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14. 2019, Vancouver, BC, wtw.eiinnnnn...uuggemm www.nnnurip... www.rrpps.commmmm cttpcccc/pp////////////
(eds.), Advances in Neural Information Processing Systems 32: Annual Conference
Canada, pp. 11189-11189, 2019. URL
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards unified view of parameter-effficientt transfer learning. CoRR, abs/2110.04366, 2021. URL
httts::/aaxxt org/ ab// wttpg//bs/211..... 04366
Xinyu Hua and Lu Wang. PAIR: planning and iterative refinement in pre-trained transformers for long text generation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 781-793. Association for Computational Linguistics, 2020. doi: 10. 10.18653/ www.2200.emmlpmaaai mainlppmain.57. URL /ttt::///oo..ogg001118553333///
emm 1mm.lp-a main.
Jintae Kim, Sera Park. Dongbo Min. and Wankyu Kim. Comprehensive survey of recent drug discovery using deep learning. International Journal of Molecular Sciences, 22(18);9983, 2021.
Celine Lee, Justin Gottschlich, and Dan Roth. Toward code generation: A survey and lessons from semantic parsing. CoRR, abs/2105.00 105.03317, 2021. URL lhtttttt///// arxiv. httpxiv..org/b///10055 abs/2155.
0
10

---

Published as a conference paper at ICLR 2023
Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic non-autoregressive neural quence modeling by iterative refinement. In Ellen Riloff, David Chiang. Julia Hockenmaier, and Jun iichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pp. 1173- 1182. Association for Computational Linguistics, 2018. doi: 10.18653/v1/118-11444 URL
1149
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-fo--sqquence pre- training for natural language generation, translation, and comprehension. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.). Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online. July 5-10, 2020, pp. 7871-7880. Association for Computational Linguistics, 2020. doi: 10.11663/11/2222accc 703.. URL ac ma 703
Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. MTOP:
A comprehensive multilingual task-oriented semantic parsing benchmark. In Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chap- fer of the Association for Computational Linguistics: Main Volume, EACI 2021, Online, April 19 Association for Computational Linguistics, 2021. doi: 10. 10.18653/v1/ eacl main. 257
- 23, 2021, pp. 2021 mael-emm-mmi 257.
Congbo Ma. Wei Emma Zhang, Mingyu Guo, Hu Wang, and Quan Sheng. Multi-document summarization via deep learning techniques: A survey. CoRR, abs/2011.04845 1.04843, 2020. URL
https://arxx org/ abs /ws//0111...........22.a11t11att 04843
Mishaim Malik, Muhammad Kamran Malik, Khawar Mehmood, and Imran Makhdoom. Auto- matic speech recognition: a survey. Multim. Tools Appl., 80(6))9411 9457, 2021, doi: 10.1007/ sl 1042--000- URL http://///....ooo./..../////0/00
Hermann Ney, Dieter Mergel, Andreas Noll, and Annedore Paeseler. data-driven organization of the dynamic programming beam search for continuous speech recognition. In IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP '87, Dallas, Texas, USA, April 6-9, 1987, pp. 833 836. IEEE, 1987. doi: 10.1 11111//ASSP...9...111999 doi. org 11199/1011.11/AAA10 IIA.II 1987 1169844
URL https://
Hariom A. Pandya and Brijesh S. Bhatt. Question answering survey: Directions, challenges, datasets, evaluation matrices. CoRR, abs/21 2021. URL httts::/azxiv..rrr
abs/2112. 03572
KYUBYONG PARK.
million sudoku games. Accessed: 2022-05-20.
www.kkaaglee..co//ddteeeesst
bryanpark saudokk
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to- text transformer. J. Mach, Learn. Res., 21:140:1-140:67, 2020. URL httphttpp//// /ttp://mmmr.. org/ papers/v21/20-077 html
Zeyu Sun, Qihao Zhu, Yingfei Xiong, Yican Sun, Lili Mou, and Lu Zhang. Treegen: A tree-based transformer architecture for code generation. In The Thirty-Fourtt AAAI Conference on Artifi- cial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances Artificial Intelli- gence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 8984 8991. AAAI 8991 Press, 2020. URL hwt.hhttss/ccccmm
aaai, wwwporgggnnnn.....e.e index. /iddee/////Aaa///tiii///////////
Bowen Tan. Zichao Yang, Maruan Al-Shedivat, Eric P. Xing, and Zhiting Hu. Progressive gener- ation of long text with pretrained language models. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tüir, Iz. Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy
11

---

Published as a conference paper at ICLR 2023
Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computaaional Linguistics: Human Language Technologies,
NAT 2021, Online, June 6-11, 2021, pp. 4313-4324 Association for Computational Linguistics, 2021. doi: 2218553/11222211 maa.:6///1000....aammamm....... nnaccl-main.341 URL htttps://dde org/10. 1
18653/v1/202 nwacl ma i 341
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 6008, 2017. URL httppstspppoccceesdiiitie neur neur ips. cc http//aaer2211/////// (eeeee2245577ee911lleee 4at.e//ee1e011111.01aa4a4aa0aaa0
html
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201.1 11903, 2022. URL /tttp::/arxiv..rrrt
11903
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey. Jeff Klingner, Apurva Shah. Melvin John- son, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.00 61900819011100 44, 2016. URL pattp://arxla org/ abs 1 1609.666600 08144
Yingce Xia, Fei Tian, Lijun Wu, Jianxin Lin, Tao Qin, Nenghai Yu, and Tie-Yan Liu. Delibera- tion networks: Sequence generation beyond one- decoding. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 1784-1794, 2017. URL https: http:::::/pppcocpeeepepe neuri www.rrppp........ ps. wc/paper/20000 peaaper/217/haaae 2017/hash/ c60 36 6bbe29bb2llc (1222266664999997769999 01 2e0 Abst
html
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien- Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang. Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir R. Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to- text language models. CoRR, abs/2201.05966. 2022.
URL httthhtts::/// /xaxxiiiiiie www.aaivvviio www.iivorrooooo org. abs/ 2201 05966
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, vol- ume 37 of JMLR Workshop and Conference Proceedings, pp. 2048-2057. JMLR.org, 2015. URL http: httt:://proceedings. mmlringggs.mmr.......mmmmmrm http:/eeddiigg......eeeeeee///es wwwps/vws11111 http:///33xxcc1... html
Shuoheng Yang, Yuxin Wang, and Xiaowen Chu. A survey of deep learning techniques for neu- wwwpiiiorrg/bbb.. /bs/
ral machine translation. CoRR, abs/2002.07526, 2020. URL
2002. 07526)
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'Achh-Buu,, Emily B. Fox, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 5764, 2019.
Xi Ye, Semih Yavuz, Kazuma Hashimoto, Yingbo Zhou, and Caiming Xiong. RNG-KBQA: generation augmented iterative ranking for knowledge base question answering. In Smaranda
12

---

Published as a conference paper at ICLR 2023
Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Anmual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. URL tics, 2022. doi: 10. 11118/6///1222222/////a 2022. long. 417.
Association for Computational Linguis- 18653/1163
Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming-Wei Chang, and Jina Suh. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Anmual Meeting the Association for Computational Linguistics. ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 2: Short Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/v1/p16-2000 URL /doi 116.://0...1.3/111111116616-16
Meng-Hsuan Yu, Juntao Li, Danyang Liu, Bo Tang, Haisong Zhang, Dongyan Zhao, and Rui Yan. Draft and edit: Automatic storytelling through multi-pass hierarchical conditional vari-
ational autoencoder. In The Thirty-Fortth AAAI Conference Artificial Intelligence, AAAI 2020, The Thirty-Secome
Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium Educational Advances in Artificial Intelligence, FAAI
2020, New York, NY, USA, February 7-12, 2020, pp- 1741-1748. AAAI Press, 2020. URL
https:/ //tp://jjj...... aaaii.aaa......................a org/ index. php/ AAAT/artice//veew............eet 5538
Xiangwen Zhang, Jinsong Su, Yue Qin, Yang Liu, Rongrong Ji, and Hongji Wang. Asyn- chronous bidirectional decoding for neural machine translation. Sheila A. Mcllraith and Kilian Q Weinberger (eds.), Proceedings of the Phirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 5698-5705, AAAI Press, 2018, URL https: / www.aaai... otg/ ocs/ Lnndx..php BAA./AAAIAAAA /ttp://AAAAAAA1/ppeeeeereeee/e//
Mengyu Zhou, Wang Tao, Pengxin Ji, Han Shi, and Dongmei Zhang. Table2analysis: Modeling and recommendation of common analysis patterns for multi-dimensional data. In The Thirty- Artificial Intelligence, AAAI 2020, The Thirty-Sconnd Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7- 12, 2020, pp. 320-328, AAAI Press, 2020. URL httph:tt////o aatp::::///saaaa..ooooa.aaaaaa/a awai. indexxiexxx/////// php/ BAAL/articee/view//
Fourth AAAI Conference
A EXAMPLES
UnifiedSKG (Xie et al. 2022) did great job to formulate 21 structured knowledge grounding tasks into a unified seq2seq form, and we apply their input-output form of WebQSP, MTOP, and KVRET in CASR. And for Sudoku, we apply their original serialization method, which flattens the 9x9 game table row by row into an 81 81-dimensional number list. Examples of WebQSP (Figure MTOP (Figure 6) and KVRET (Figure 7 are shown.
B BEST RESULT SELECTION
The default strategy is to run all T casteps and take the Last output ÿT the final prediction. However, one could argue that for a specific input X, the best prediction may occur early and more refinements may hurt. Following this we try to pick the ÿ (0 < t < T) with the highest prediction probability (given by M9): Ý = Slank where lpick = argmmar,Po P( 1t- (t.1,MM). However, that doesn't always work and we leave it as furture work.
C EXPERIMENT DETAILS
C.1 HYPERPARAMETERS
On WebQSP, MTOP and KVRET tasks, we initialize CASR models with T5-base (Raffel et al. 2020), 24-leyer generative PL.. We set casteps T = 3. Max epoch F. is set 2K, 4K, and
13

---

Published as a conference paper at ICLR 2023
Errest Rutherfa
dlym
m.tafeeaal m
Request: what school did ernest rutherford go Answer: (JDIN education education institution) (JOIN (R people.person.eduee eesiccttoutaooonn m. m.0227rl))
iduucationeddcation
Figure 5: An Example of WebQSP.
PENGAOLL Saat Aed WhLP CENDOW yoe Zur Neniviee sue maA.et uncorn.
para my heart
pans metturve
CONSICL OITIOEL IO
MRSURATT unndedd 311 cROuP
botel
Fot Stop
arool
INSIN MERINE
IAA UNNAARIL AANAAAA SWWA MAoI C.V
infoney
unceclo
milo
brlff.
round Lible
picz TESUMMAE
313 art
voko
heary traffs
modertte trffic
strutt
Request: recipe dairy?
Request:
grocery,
Answer: [INIS TRUE RECIPES [SLIEETHOD RECIPES recipe]
Response: are 4 miles away from whole foods and from safeway: you prefer?
[SLRECIPES, INGREDIENT dairy]]
Figure 6: An Example of MTOP.
Figure 7: An Example of KVRET.
4K steps for fine-tuning WebQSP, MTOP and KVRET, respectively. For adapter-tuning, we double the epoch number, which is 4K, 8K, and 8K steps. For the three tasks, we the batch-size 128, learning-rate to 2e-5, max-ing maxx-nnut-length to 1024, geegraiiooon-e to 128, beam-size to 4, and evaluate every 2K steps for checkpoint selection.
For Sudoku, we train a 12-layer encoder-decoder transformer from scratch, with d-model=512. ffn- dim=2048, num-heads=8. We set max castep T 5 and max epoch 10K steps. We set the batch-size to 1024, learning-rate to 2e-5, beam-size to 2, and evaluate every 2K steps for checkpoint selection.
C.2 RESOURCE CONSUMPTION
Table 8: CASR Variations and Their Number of Parameters.
Table Training and Inference Time Con-
sumption (take WebQSP as example)
A, and de-
Traininghh) Inferenes(mnn)
note #parameters of adapter, encoder and decoder.
Xa
INAT (Log 2018
Tuning
Init.
Arch.
#Parameters
XB
2019
X2
CASR Dec
Fine-tune
Restart
SepEne
T (2E
Xd
Ridirectional /hang et 2018
B
Fine tune
Continue
Sepline
T (2E +
Ne
Progressive Tan
16
Fine-tune
Restart
UniFne
T 17)
80
Fine
Continue
Unitine
T
B2
CASR
Adapter
Restart
SepEioc
T A+
Q
CASR-L
28
141
Table 8 shows the total parameters of each controlled experiment to help the audience understand our methods. In row (., the total parameters is T *+H++D) rather than T*A+(2)+)), because Encoder and Encoder, initialized with the same PI.M and frozen all the time.
14

---

Published as a conference paper at ICLR 2023
Table shows the training and inference time consumption of CASR and baselines. Note that C runs slowly because we leverage deepspeed to avoid OOM.
We train 4 Tesla V100 GPUs. It takes 10, 22, and 19 hours to fine-tune WebQSP, MTOP and KVRET, respectively, and 12, 25, and 21 hours to adapter-tune them. Adapter-tuning is intrinsically hard it takes more training steps tune fewer parameters comparing fine-tuning. For Sudoku, takes 32 hours to train from scratch.
D DETAILED RESULTS OF EMPIRICAL STUDIES
Table 10: Performance of last-step prediction Y when castep 2=2, produced by the setting of row
Table 11:
The average density to input X and previous prediction yt-1 of the setting of row
B in Table 8 grouped by DÿI, the density to
B in Table g Note that, when castep t=0, the
density to the previous prediction is 0, thus not listed.
Task
WebQSP
МТОP
KVRET
Task
WehQSP
MTOP
KVRET
AVG
Metric
F1
Acc
Match
Bleu
Micro F1
AVG
Input 0.10%
0.11%
0,40%
0.20%
Sparse
51.88
78.52
72.72 16.70
64.17
56.80
Previous Prediction
0.34%
0.33%
0.84%
0.50%
Middle
85.84
87.53
83.17
20.18
75.09
70.36
Input 0.09%
0.10%
0.39%
0.19%
Dense
85.78
89.08
87.63 20.26
85.80
73.71
Previous Prediction
0.61%f
0.43%
1.01% f
0.68%
6 9
1 3
5
4
8
3
6
1
3
1 8
3
4 6
3
5
9
2
4
1
2
7
4 2 9
5
7
4
6
9
2 8
7
7
4
3
1
6
8
5 7
3
1
6
9
8
4 6
2
5 9
4 6
1
2
5
3879
4 6
7
5
3
8
3 5
9
8
6
2 4
2 9
8 6
1574
3 2
9
8 6
1
5
4 6 1
7 2
5 1 &
7
7 8
7 4 9 6
7 5
9
3
9
7
2
5 6
9 1
4
8 3
5
6 2
17
5
4
2 4
7 3 5
4
2 3
8
6
5
1-1 Figure 8: The Dependencies between Casteps Discovered by CASR. Shown purple (!=1 attends to t=0) and blue (t=2 attends to 1=1) arrows.
t=
t-2
D.1 AN EXAMPLE COMPUTE DIFFICULTY OF BLANKS
In the game table Figure 2 for the top-left blank (the ground truth is 6 in green). r=5 (the ground truths of the row blanks are 6, 9, 4, 8, and 3 respectively), c=6 (the ground truths of the column blanks are 6, 4, 3, 9, 8, and 2). and h=4 (the ground truths of the house blanks are 6, 9, 7, and 4). Then, the difficulty to fill the top-left blank is (5-1)=66-))((411)=66.
15

---

Published as a conference paper at ICLR 2023
Task&Input X
CASR Predictions Y
Ground Truth Y
WebQSP
input:task: webqsp: what
(AND (JOIN
(JOIN
(JOIN
(JOIN
school did sir ernest
common.topic.net
education.educatio education.educatio
education.education. institution) (JOIN
rutherford to? structured
(JOIN (R
ble types m.01nf)
n.institution)
n.institution)
knowledge: Ernest Rutherford:
education.educati n.institution) (JOIN (R
(JOIN (R
(JOIN (R
(R
m.02m7r| m.068kst
people.person.cduc coople.pesson.due
people.person.edue peuuu tion) m.02m7r)
common.topic.notabe types
m.01m9 (omitted to save
people.person.cdue ation) m.02m7r))
ation) m.02m7r))
ation) m.02m7r))
space) m..0180w basc.act
MTOP
El
F2
input:task: mtop: does this
[N:ISTTRUE TRUE RECI [IN:IS TRUETRU___RR RECI [IN:IS TRUE RECI PES
[N:AS TRUE RECIP
recipe have duiry: structured
PES
PES
ES
knowledge: IN:GET:
SL:RECIPES INCL
[SL:METHOD REC [SL:METHOD REC
[SL:METHHDD RECI
MESSAGE, WEATHER....
UDED INGREDIEN IPES recipe
IPES recipe
recipe
(omitted to save space)
T dairy
ISL:ECIPES INCL [SL:RECIPES INCL
[SL:RECIPES INCL
IN:FOLOOW: MUSIC
UDED INGREDIEN UDED INGREDIEN
UDED INGREDIENT dairy
T dairy I1
T dairy I]
KVRET
[=1
F
imput:task: kyret: : where is the
whole foods is
the closest
the closest
we are 4 miles away
closest grocery_store store;
miles away at
grocery_ste sttrrsttteeeee is whole foods 819 alma st.
grocery_stur sttrrestteeeee is
from whole foods and from safeway: which do you prefer?
structured knowledge: col poi
819 alma
whole foods which is miles away 819 alma st.
type address distance
traffic info row pizza my heart pizza restaurant (omitted to save space) 3
miles moderate traffie;
context:
Sudoku
2.3,4
7
6 2
3
6 2 35
3
09248
2
3
6
1
31
16
27
16339
3
69
57
6
2
7
43 169
2
67 69
8743 1 2
6
5
1
33697
4
2
3 7
40175388
10177361
6
6
529861078
329
6
4 493
7 9
7 7
6
403
7
2
9 42
9 8156
4
9
01567428
9
9
0360
836925
656925413
Figure 9: Case study for CASR (row B on WebQSP, МТOP, KVRET, and Sudoku at different casteps. The first column shows how the input is serialized from Figure Figure 6 and Figure 7 For WebQSP, MTOP, and KVRET, keywords are highlighted with green color, and red color denotes wrong predictions. For Sudoku, the white fields denote the blanks be filled, the red numbers in them denote that the predictions are wrong, and the green ones denote the correct predictions. It's a coincidence that the Sudoku prediction for this sample remains the same for t=2,3,4, so we draw them together.
16

---

Published as a conference paper at ICLR 2023
Method
WebQSP
MTOP
KVRET
(JOIN (R
error iserror theerror at is error
(AND
and IN: the_ JACT [] the
INAT
(JOIN
dduccatio..duccationddeeeee
for: IIN for
(JOIN eduucatinneducation.d).)v)w m.cducationmaO2datt)
would would issis____ error is
Levenshtein
(JOIN (R
eduucttin.eduuaaiionnsstttttut (JOIN
what is for you.
(R peopleperrson..ducattionn) m.03xsv3))
the nearest
(JOIN (R
[IN:GET RECIPES [SL:RECIPES DISH
grocery store is whole foods alma st. would you like directions there?
CASR /eec
education.educcaa cnccatio..duuaatio.nnnsttttoooo (JOIN
(R peopleperrson..ducatioonn) m.02mjmr))chheee
]]
(JOIN (R
[IN:GET RECIPES
Bidirectional
(JOIN
[SL:EEIPEES INCLUDE
safeway is miles away.
(R people.person..cuccttionn) m.02m7r))
D DJINGREDIENT dairy 1]
(AND (JOIN
Nommon.tooic.notabl. t_ppes_t_pe_______p_eeeeee m.01nf)
IN:AS RECIPES
the closest store
Progressive
(JOIN (R
[SL:RECPPES INCLUDE
safeway.
inncatiioncducaationnnntttttiiii (JOIN
INGREDIENT dairy II
(R peoplepeerson..euccttion))) m.02m7r))
(JOIN (R
[NNIS TRUE RECIPES
the closest grocery store whole_ foods at 819 alma
CASR-L
(JOIN
[SL:RECIPES INCLUDE
(R peopleperrson..duaatiionn) m.02m7r))
INGREDIENT dairy II
Figure 10: Case study for other experiments (baselines and CASR-L).
17

---

Published as a conference paper at ICLR 2023
Algorithm CASR Training Process ($3.1)
Input: max castep T; max epoch E; all input X and ground truth Y in train (S,) and dev (Sa) sets
Output: CASR models M (0 S < T); for in 0, 0,.,
I:
do < parameter initialization ($3.4)
2:
3:
for in 0, 1, .... E 1 do
4:
f argmaz by updating using
Algorithm CASR Inference Process.
train $/ i then
Input: max castep T; input X (from test set); well-trained CASR models M (0 1tst<T
5:
orgmaze
Output: the best prediction Y for X
Σ (X,)ES P( else
1: for in 0, 1 --...,-111 do 1 argmnar by beam searching #/
7:
2:
argmaze
3:
if' = 0 then
ExXYIES, [(Y|X,Yt-1,8) end if M 40 end for
4:
Y t argmary M(Y
5:
else
Y1
10:
6:
argmary
P(Y X, p1-1 MD
11:
12:
M t- checkpoint selection from M
7:
end if
(0 S <e< on Ss
8: end for
9: return Y
13:
1* predict with beam searching AX e
S. U.S. using Mr
14:
ift 0 then
15:
y + argmary P(Y ((YXXMM)) M')
16:
else
17:
+ P .M
argmary
18:
end if
19:
end for
20:
return M (0<t<T)
18


Reviews:
[{"review_id": "Gdg7rD-qDR7", "reviewer": "Reviewer_rnaL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper deals with tasks where left-to-right generation is not the best strategy. The authors propose a refinement framework named CASR – which models p(i-th-timestep output token at t-th refinement step | input to task, output of model at the (t-1)-th refinement step, the prefix/history of output tokens at t-th refinement step). So when training CASR, the input contains the models’ predictions at the previous refinement step as well as the input to the task, and the output is the reference answer. At test-time, iterative refinement is applied. \n\nExperiments are done on WebQSP, MTOP, KVRET, Sudoku (see Section 2.2 for details). The authors have also done ablation studies on architecture choices (e.g., how to encode the input to the CASR model, how to initialize weights of the model at each refinement step, fine-tuning vs. using adapters, etc.). CASR is effective, and achieves SoTA results on KVRET. \n", "summary_of_the_review": "The idea is interesting. Results on the chosen tasks (though quite specific) are great. Ablation studies are comprehensive and inspiring. I would want to see two more baselines -- labeled (*) above; or have the authors justify why the two baselines are not necessary. Currently I'm leaning toward weak accept. ", "strength_and_weaknesses": "The comparison of the authors’ approach vs. iterative refinement and progressive generation is quite clear in Table 1. The table helps understanding greatly. \n\nAblation studies are quite comprehensive and inspiring. The analysis on attention is interesting. \n\nReproducibility is great, given that the code is provided. \n\n\n\nI’m not convinced at why using the previous refinement round’s output as input to the CASR model is necessary. Experiments on iterative refinement using a regular refinement model would be helpful. Regular refinement: train regular refinement model (inputs: generated output from original model; outputs: reference output); then, do one-round refinement after the original model and use these outputs as the input to the refinement model; then, do one-round refinement using this updated model (after the original model), and use these outputs as input to the refinement model; etc. Please correct me if I’m missing important details. (*)\n\nThe second paragraph on page 2 (starting with “however”) is lacking. The authors believe that if there are right-side dependencies “which together lead to multi-hop dependency chains,” then “left-to-right is not the best order for generation.” There needs to be citation or empirical proof. First, left-to-right generation with multiple refinement steps could lead to good results. Second, if we have a perfect model that scores each sequence, and a good decoding strategy (e.g., sampling many candidates and then reranking), then left-to-right generation can solve the tasks like Sudoku and WebQSP. The issue may be that our model is not perfect at scoring sequences, so we may need to rely on other strategies. Anyhow, the authors’ writing that “because there are multi-hop dependency chains, left-to-right is not the best order” should be more nuanced and in depth. \n\nRelated to the above: Have the authors attempted better decoding strategies like sampling 100 candidates and rerank, or beam search with large beam size? It’s not clear whether the authors have definite proof that left-to-right decoding doesn’t work and refinement is necessary. (*)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CGByGpCDtoD", "reviewer": "Reviewer_Nx7n", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel approach [CASR] for sequence generation where multiple language models are connected in an iterative manner and predictions of a previous timestep are taken as an input to the model at future timestep [different from vanilla RNN]. The rationale is that sequence generation problems only specify “difficult” FINAL sequences which need to be solved, and the intermediate order to solve “simpler sequences” is defined by heuristics [Tan et Al]. \n\n", "summary_of_the_review": "This work makes a SINGLE minor change to the mathematical formulation of Tan et Al [Figure 1], but still improves empirical results on only 1 out of 3 datasets studied. [Sudoku is a toy dataset from Kaggle which authors don’t show other methods against]. Due to lack of novelty in technical contribution, but still beating SOTA on 1 dataset, this work has incremental contributions. ", "strength_and_weaknesses": "**Strengths**\n\n- The authors demonstrate that a network can itself learn the simpler sequences it needs to solve, and that there is a natural order of progression from solving “simpler” sequences to complex ones. Furthermore, they empirically reveal that this progression is different from the simple left-to-right solving order that sequence generation usually takes. \n\n- The manuscript is well crafted, with convincing experiments and detailed ablations. Authors claim SOTA [70%] on KVRET F1 metric. \n\n**Weaknesses**\n\n- The basic training process [Algo 2] has a limitation that the language model at each timestep needs to be trained fully [line 3] before moving on to the next timestep. This prevents the network to be E2E trainable, leaving CASR prone to parameter-effficient tuning. [3.3].\n\n- Authors define the notion of density measuring the attention magnitude at each sequence token [5.2]. Increasing density at each recurrent step, seems to show that the prediction made by the model becomes more informative. I am curious as to whether the density becomes maximum when the network converges to optimal solution. Please clarify this.\n\n- On sudoku, the restart strategy lags behind Continue strategy by a huge difference ~20% (table 4). But we don’t observe this trend in other three datasets, well recognized in the community (Table 3).  Could the authors please mention their intuitions on these observations?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4eFpY4oo8Fb", "reviewer": "Reviewer_B3Ea", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes CASR: a framework that generates complex sequences by iteratively editing previously generated sequences. The problem is well-motivated: in complex scenarios, we might need multiple rounds of editing to generate correct sequences. CASR achieves this goal by feeding a transformer model with input X and prediction from last iteration. The model achieves good performance over both conventional AR and NAR baselines. Compared to template-based methods, CSAR is more flexible and avoids additional exposure bias ", "summary_of_the_review": "Given those discussed in Weakness and Novelty, I think this submission is not ready for acceptance yet, but I am happy to raise my score if my concerns are addressed.  ", "strength_and_weaknesses": "Strength:\n\n1. Well-motivated problem and clear presentation.\n2. The proposed CSAR is effective compared to vanilla AR and NAR models.\n3. The evaluation and analysis are somewhat comprehensive. \n\nWeakness:\n\nThe major concern I have regarding this submission is that some critical related works are missing.\n\nIn particular, this line of related work, according to my knowledge, started from Deliberation Networks 2016[1], which is similar to CSAR except that it only focuses on the decoder (the previous prediction will go through the decoder multiple times to be refined). Another recent model that I found similar to CSAR is Draft and Edit 2020 [2], which is basically a CASR with T=2, and uses VAE as the backbone. Many other similar models can be found in Deliberation Networks' citation. \nIn short, I believe this line of research is highly correlated with this submission and has to be discussed. I can see that there are still differences between CSAR and these works, but those have to be thoroughly discussed in the paper. \n\nFurthermore, although the existing evaluation is already quite comprehensive, compared to related works above is necessary. Such that we can perceive what is the advantage of CSAR over other iterative models. \n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster