A Framework for Bidirectional Decoding: Case Study in Morphological Inflection
Authors: Marc Canby
Julia Hockenmaier
Venue: EMNLP 2023 Findings (2023)
Keywords: morphology
decoding
inflection
transformers
TL;DR: We present a framework for bidirectional decoding and show strong experimental results on a morphological inflection task.

Abstract: Transformer-based encoder-decoder models that generate outputs in a left-to-right fashion have become standard for sequence-to-sequence tasks. In this paper, we propose a framework for decoding that produces sequences from the "outside-in": at each step, the model chooses to generate a token on the left, on the right, or join the left and right sequences. We argue that this is more principled than prior bidirectional decoders. Our proposal supports a variety of model architectures and includes several training methods, such as a dynamic programming algorithm that marginalizes out the latent ordering variable. Our model sets state-of-the-art (SOTA) on the 2022 and 2023 shared tasks, beating the next best systems by over 4.7 and 2.7 points in average accuracy respectively. The model performs particularly well on long sequences, can implicitly learn the split point of words composed of stem and affix, and performs better relative to the baseline on datasets that have fewer unique lemmas.

A Framework for Bidirectional Decoding: Case Study in Morphological Inflection
Mare E. Canby and Julia Hockenmaier
University of Illinois at Tubana-Champall {marcec2, jul j
Abstract
We explore this issue for the task of morphologi- cal inflection, where the goal learn a mapping from a word's lexeme (c.g. the lemma walk) to a particular form (e.g. walked) specified by a set of morphosyntactic tags (e.g. V.V.PTCP:PST), This has been the focus of recent shared tasks (Cotterell et al., 2016, 2017, 2018; McCarthy et al., 2019; Vylomova al., 2020; Pimentel al., 2021; 1; Kod- ner et al., 2022: Goldman et al., 2023). Most ap- proaches use neural encoder-decoder architectures, e.g recurrent neural networks (RNNs) (Aharoni and Goldberg, 2017: Wu and Cotterell, 2019) or transformers (Wu et al., 2021).2 To our knowledge, Canby et al. (2020) is the only model that uses bidirectional decoding for inflection; it decodes the sequence in both directions simultancously and returns the one with higher probability.
Transformer- based encoder-decoder models that generate outputs a left-to-right fash- ion have become standard for sequence-to- sequence tasks. In this paper, we propose a framework decoding that produces se- quences from the "outside-in": each step, the model chooses to generate token on the left, on the right, join the left and right sequences. We argue that this is more principled than prior bidirectional decoders. Our proposal supports variety of model architectures and includes several training methods, such as a dynamic programming algorithm that marginalizes out the latent ordering variable. Our model sets state-of-the-ar (SOTA) on the 2022 and 2023 shared tasks, beating the next best systems by over 4.7 and 2.7 points in average accuracy re- spectively. The model performs particularly well long sequenees, can implicitly learn the split point of words composed of stem and af- fix, and performs better relative to the baseline datasets that have fewer unique lemmas (but more examples per lemma)."
this paper, we propose novel framework for bidirectional decoding that supports a variety model architectures. Unlike previous work ($2), at each step the model chooses to generate a token on the left, generate a token on the right, or join the left and right sequences.
1 Introduction
This proposal is appealing for several reasons. As a general framework, this approach supports a wide variety of model architectures that may be task-specific. Further, it generalizes L2R and R2L decoders, as the model can choose to generate se- quences in a purely unidirectional fashion. Finally, the model is able to decide which generation order is best for each sequence, and can even produce parts of a sequence from each direction. This is particularly appropriate for a task like inflection. where many words are naturally split into stem and affix. For example, when producing the form walked, the model may chose to generate the stem
Transformer-based encoder-decoder architectures (Bahdanau al., 2014; Vaswani al., 2017) that decode sequences from left to right have become dominant for sequence-to-se to-s-quenseee tasks. While this approach quite straightforward and intuitive, some research has shown that models suffer from this arbitrary constraint. For example, models that decode left-to-right are often more likely to miss tokens near the end of the sequence, while right- to-left models are more prone to making mistakes near the beginning (Zhang et al., 2019; Zhou et al., 2019a). This a result of the "snowballing" effect, whereby the model's s use of own incorrect pre- dictions can lead future predictions to be incorrect (Bengio et al., 2015; Liu et al., 2016).
OOtthooonal to the concerns in this paper, various data augmentation schemes such as heuristic alignment or rule- based methods (Kann and Schütze. 2017: Anastasopoulos and Neubig, 2019) or the use of multilingual data (Bergmanis et al., 2017; 7; McCarthy et al., 2019) have heen proposed to improve these standard architectures.
Our cnde is available at https: http://gtttuubccco com/ marccanby/bidi_co http://bddiieeedddnnngeeeeee/e/e

---

walk from the left and the suffix ed from the right. We explore several methods for training models under this framework, and find that they are highly effective on the 2023 SIGMORPHON shared task on inflection (Goldman et al., 2023). Our method improves by over 4 points in average accuracy over a typical L2R model, and one of our loss func- tions is particularly adept at learning split points for words with clear affix. We also set SOTA on both the 2022 and 2023 shared tasks (Kodner et al., 2022), which have very different data distributions.
with a start-of-sentecce token; each step a token chosen based on those preceding, and the process halts once an end-of-sentence token is predicted. In contrast, our bidirectional scheme starts with an empty prefix $ and suffix #. At each timestep. the model chooses to generate the next token of either the prefix or the suffix, and then whether or not to join the prefix and suffix. If join is predicted, then generation is complete.
We define an ordering sequence of left and right decisions: that is, ol') E { RR. We use yl to refer to the token generated at time I under a particular ordering, and yj (St) and T (St) to refer to the prefix and suffix generated up to (and including) time t.? An example derivation of the word walked is shown below:
o.) as a
2 Prior Bidirectional Decoders
Various bidirectional decoding approaches have been proposed for tasks such as machine transla- tion and abstractive summarization, including ones that use some form of regularization encourage the outputs from both directions to agree (Liu et al., 2016; Zhang et al., 2019; Shan et al., 2019), or al- gorithms where the model first decodes the entire sequence in the R2L direction and then conditions on that sequence when decoding in the L2R direc- tion (Zhang al., 2018; Al-Sabahi et al., 2018). Still more methods utilize synchronous decoding, where the model decodes both directions at the same time and either meet in the center (Zhou et al., 2019b; Imamura and Sumita, 2020) or proceed un- til each direction's hypothesis is complete (Zhou et al., 2019a; Xu and Yvon, 2021). Lawrence et al. (2019) allows the model to look into the future by filling placcholder tokens at each timestep.
y
Fish
y (50
$
#
(1) 1
y())
Sw
0 L
yO = a
Swa
#
( R
y d
Swa
d#
4 L
y(4)
Swal
d#l
5 R
y e
Swal
ed#
6 L
y k
Swalk
ed#
Dropping the dependence on æ for notational convenience, we define the joint probability of out- put sequence and ordering o as
[vl p(o giso (((t)).
Ply.
3 A Bidirectional Decoding Framework
Ply
0
3 (<0)
y ((t)) Q1)
(3)
The following sections present a general framework for training and decoding models with bidirectional decoding that is irrespective of model architecture, subject to the constraints discussed in $3.3.
where (1(1) is the probability of joining (or not joining) the prefix and suffix:
Q
P(join 7
T
if if lvl otherwise
1 1- -(joonnn (SA)
(St)
3.1 Probability Factorization
For unidirectional models, the probability of an L2R sequence ÿ y1 ... Yn or an R2L sequence y V..... y1 given an input æ is defined as
3.2 Likelihood and MAP Inference
To compute the likelihood of a particular sequence y. we need marginalize over all orderings: P(y]=) Σ. P(y,o|z) Since we cannot enu- merate all 2 orderings, we have developed an exact O dynamic programming algorithm, reminiscent of the forward algorithm for HMMs. To simplify notation, let PL V. y <i: " <j) (or PR 9 ; z) <s. y <))) be the probability
II
<i,
P( (m)
P m)
where V. - y/ y yn-j+1 is the ith or jth token in particular direction. Generation begins
'We use superscripts to refer to timesteps, and subscripts for sequence positions. Note that if, at particular timestop i, we have prefix 7 si and suftix T S then i + j = t.

---

generating the ith token from the left (or the jth token from the right), conditioned on z <i and y <j. the prefix and suffix generated thus far:
3.3 Why does dynamic programming work? Dynamic programming (DP) only works for this problem if the local probabilities (i.e. the token, join, and order probabilities) used to compute f(ij]] depend only on the prefix and suffix cor- responding to that cell, but not a particular dering that produced the prefix and suffix. This is similar to the how the Viterbi algorithm relies on the fact that HMM emission probabilities depend only on the hidden state and not on the path taken. To satisfy this requirement, the model's architec- ture should be chosen carefully. Any model that simply takes a prefix and suffix as input and returns the corresponding local probabilities is sufficient. However, one must be careful if designing model where the hidden representation is shared or reused across timesteps. This particularly problematic if hidden states computed from both the prefix and suffix are reused. In this case, the internal represen- tations will differ depending on the order in which the prefix and suffix were generated, which would cause DP cell to rely on all possible paths to that cell thus breaking the polynomial nature of DP.
V = Des. <)) P(T Vec T Pn(TT,Ie> W...e.. P(R Fe ag) P( 3W.) Pa
Let Qú be the join probability for y Si and y sy:
P(join Je: T s P(join gin S>)
ii+tj=l lyv otherwise
Qu
Finally, denote the joint probability of prefix 7 <i and suffix y y by fli. j].
We set the probability of an empty prefix and
suffix (the base case) to 1:
f[0.0] -
The probability of a non-empty prefix 3 si and empty suffix can be computed by multiplying fli-1,0) 0] (the probability of prefix 3 <i and empty suffix <) by PL(V D <v, <) (the probability of generating 5/ 1) and the join probability Qi::
fli.0] = fli 1. (T 3 c) Qi0
Analogously, we define
<j) Quj f[o,j] = f - 1] Pu s/c, y
3.4 Training
Finally, [[ represents the case where both prefix z si and suffix マı are non-empty. This prefix-suffix pair can be produced either by append- ing V. to the prefix Ji and leaving the suffix unchanged, or by appending TV to the suffix & <j and leaving the prefix unchanged. The sum of the probabilities of these cases gives the recurrence:
We propose two different loss functions to train bidirectional model. Based our probability factorization, we must learn the token, join, and order probabilities at each timestep.
Our first loss function (#) trains each of these probabilities separately using cross-entropy loss. However, since ordering is a latent vari- able, it cannot be trained with explicit supervision. Hence, we fix the order probability to be 0.5 at each timestep, making all orderings equi-probable. We then define S to contain the indices of all valid prefix-suffix pairs in given sequence y:
f[i,j] = fli - 1,j] Pr
Qit+ <i, V
Pn( P Si, マソ Qij
After filling out the dynamic programming table f. the marginal probability P(y) can be computed by summing all entries fli, where i + j = lyl:
P(v)
> I(i+j (lw)=EIdh 25
by) f[i.j]
S = (ii,)) | S, ly + S |
Hence, S has 0(1y elements.
If all local probabilities can be calculated in con- stant time, the runtime of this algorithm is O())/?). As aside, the MAP probability, or the proba- bility of the best ordering for a given sequence, can be calculated by replacing each sum with a max:
Finally, we define a simple loss L2H (0) that av- erages the cross-entropy loss for the token probabil- ities (based on the next token in z or 'y and join probabilitics (based on whether the given prefix and suffix complete y):
fix,]=max (f[i 1, Pr <s. V -
(on(9) _ ((Z(0) Z(() L (ljoin) (8)
[(b - 1] Pa V su 7e)) Qu)
3
max P 0) max (I(t + lw|) Jlb
Z(00- = |s|
1
Σ log P(V. V ci T 23
The best ordering itself can be found with a back- tracking procedure similar to Viterbi for HMM's.
i((0) |s|
1
Σ log P(T, v <b 4;

---

PÚjoin vz.
P'order 5
P(y+i Myo
po P ولم
Join Linear Softmax
L2R Linear Suitmax
RZL Linear + Softmax
Linear Softmax
Encoder
el. "
Decoder
SJ S0-
$128 $12t>
W
CJ
Co
CLzz
Gazz.
d
V 1st Sg PST
Yzi
Yzi
Figure 1: Architecture for bidirectional decoding model. Depicts the token inputs for the verb walked at timestep
== 3 with y s2 = $wa and y d# All inputs are surrounded by rectangle.
( (0)
log Qu (1)))
The formula for P(y, o o|z) is given by Equation 3. Each hypothesis prefix- suffix pair. We start with single hypothesis: an empty prefix and suffix, represented by start- and end-of-sentence tokens. At a given timestep, each hypothesis is expanded by considering the distribution over possible actions: adding token the left, adding token on the right, or joining. The k best continuations are kept based on their (joint) probabilities. Generation stops once all hypotheses are complete (i.e. the prefix and suffix are joined).
[S]
where Qij is defined in Equation 4.
Due to the size of S, this loss takes O( time to train.* 4 Given that a typical unidirectional model takes O(Iyl) time to train, we also propose an Olyl)) approach that involves sampling from S; this is presented in Appendix F.
An alternative is to train with Maximum Marginal Likelihood (MML) (Guu et al., 2017; Min al.. 2019), which learns the order probabil- ities via marginalization. This is more principled because it directly optimizes P(y x). the quan- tity of interest. The loss given by LM (6) =
Model Architecture
Our architecture (Figure 1) is based on the character-level transformer (Wu et al., 2021), which has proven useful for morphological inflec- tion. First, the input sequence æ is encoded with typical Transformer encoder; for the inflection task, this consists of the lemma (tokenized by character) concatenated with a separator token and of tags. Given prefix si and suffix 'y şj (as well as the encoder output), the decoder must produce each direction's 's token probabilities, the join probability, and the order probability. We construct the input to the decoder by concatenating the prefix and suffix tokens with some special classification tokens:
log P zz; 0), which is calculated with the dy- namic programming algorithm described in $3.2.5 Learning the order probabilities enables the model assign higher probability mass to orderings it prefers and ignore paths it finds unhelpful.
This loss also requires (2)) time to train.
3.5 Decoding
The goal of decoding to find y such that y argmax, P(y]x). Unfortunately, it is not com- putationally feasible to use the likelihood algo- rithm in $3.2 to find the best sequence y, even with heuristic like beam search. Instead, we use beam search to heuristically identify the sequence y and ordering that maximize the joint probabil- ity P(y. o))):
(CJ, co. V1..., .... Vi CL2R, CR2L, V,..., '
The tokens CJ,, Co, CL2R; and CR21 special clas- sification tokens that serve a purpose similar to the CLS embedding in BERT (Devlin et al., 2019). We feed this input to a Transformer decoder as follows:
y argmaxy..o ol
s,, 80, 8 LLR; 8 Decoder((:..)
"This assumes that local probabilities take O(1) time to compute, which is not the case for most neural architectures; however. this section is about the runtime the training algo- rithms without regard to model architecture.
These vectors are fed through their own linear lay- ers and softmax, giving the desired probabilities:
AAppendix C. descrihes how to train this loss in practice.
P(join
V Sin y (s) = Softmax(so)

---

# Langs
Langs (y - 0.05)
Avg
BL2
BL2
BL2
BL2
L2R
80.26
R2L
79.65
90%
B1.2
82.59
Accuracy
xH
84.25
19/27
12/27
12/27
3/27
MML
81.43
9/27
5/27
11/27
11/27
80%
xH-Rerank
84.38
18/27
12/27
13/27
2/27
MML-Rerank
81,50
9/27
5/27
12/27
10/27
70%
BL.2xH BL2-MMI.
84.00
24/27
12/27
15/27
83.54
18/27
7/27
17/27
3/27
Goldman et (2023)
81.6
60%
02
22
88
Output Form Length
Table 1: Accuracies of Methods. Accuracy averaged over all languages in the SIGMORPHON 2023 shared task, and number of languages whose accuracy equals exceeds (2) the best baseline BL2. The entry Goldman et al. (2023) shows accuracy the next best system submitted the shared task. Also shows number of languages with a statistically significant improvement (>) or degradation (<), or no statistically significant change (-), in accuracy compared with BI.2 using gaired-permutatio test (Zmigrod et al., 2022) with a = 0.05. The best entry in each column bold. See Table 9 in Appendix D for results by language.
BL2
xhi Rerank
Figure 2: Accuracies of xH--Rrrank and B1.2 by Word Length. Average accuracies of BL2 and xH-Rerank models over languages, grouped by length (number of characters) of the output form.
Model hyperparameters. Our models are im- plemented in fairseq (Ott al., 2019). We exper- iment with small, medium, and large model sizes (ranging from ~440 to ~~7.3M parameters). For each language, we select a model size based on the L2R and R2L unidirectional accuracies; this procedure is detailed in Appendix A.
P(order P(
Tsi. y <j) = Softmax(sJU)
V si. 5ز Softmax(s. (s) P(V;
The only additional parameters in our bidirec- tional model come from the embeddings for the classification tokens (described in 84); hence, our unidirectional and bidirectional models have roughly the same number of parameters.
V si y <s) Softmax(s,
Since this architecture does have cross-attention between the prefix and suffix, the decoder hidden states for each prefix-suffix pair must be recom- puted each timestep to allow for DP (see $3.3).
Training. We use a batch size of 800, an Adam optimizer (31 = 0.9, 33 0.98), dropout of 0.3, and an inverse square root scheduler with initial learning rate 1e-07. Training is halted if validation accuracy does not improve for 7,500 steps. All validation accuracies are reported in Appepndix A. Inference. Decoding maximizes joint probabil- ity Pyy o (a)) using the beam search algorithm of $3.5 with width 5. In some experiments, we rerank the best candidates according to their marginal probability p(yl x). which can be calculated with dynamic programming ($3.2).
5 Experimental Setup
Datasets. We experiment with inflection datasets for all 27 languages (spanning families) from the SIGMORPHON 2023 shared task (Goldman al., 2023). Each language has 10,000 training and 1,000 validation and test examples, and no lemma occurs more than one of these partitions. We also show results on the 20 "large" languages from the SIGMORPHON 2022 shared task (Kod- ner et al., 2022), which has a very different sam- pling of examples in the train and test sets. A list of all languages can be found in Appendix A. Tokenization. Both the lemma and output form split by character; the tags are split by semi- colon. For the 2023 shared task, where the tags are "layered" (Guriel al., 2022), we also treat each open and closed parenthesis as token. Appendix B describes the treatment of unknown characters.
Models. We experiment with the following mod- els (see Appendices D and for more variants):
L2R & R2L: Standard unidirectional trans- former baselines, trained with the loss given in Equations and 2.
BI.2: naive "bidirectional" baseline that returns either the best L2R or R2L hypothesis based on which has a higher probability.

---

xHH-Rerank
BL2-XH
Acc(xH-Rerank) AAc(BL2)
10%
Acc(BL2-xHH) Acc(BL22)
10%
6%
6%
24
0%
0%
2%
-
1%
-4%
90%
4PH
2.10
quie
nep
Bus
DAU
Its
neeu 5r6
uny
UU
nap
JM
eNS
Jra
Guo
pqu
GRN
now
dal
ayp
ous
umu
Language
Language
Figure 3: Accuracy Improvement by Language. Difference in accuracy between our best models (xH-Rerank and BL2-xH) and our best baseline BL2.
xH & MML: Our bidirectional transformer ($4) trained under the cross-entropy or MML loss of $3.4, and decoded under Pyy, o/
or R2L hypothesis based on marginal probability under xH or MML is very effective. Both of these methods improve over BL2, which chooses be- tween the same options based on unidirectional probability. BL22xH stands out by not having sta- tistically significant degradation on any language. Comparison with Prior SOTA. Goldman et al. (2023) presents the results of seven other systems submitted to the task; these, five are from other universities and two are baselines provided by the organizers. The best of these systems is the neu- ral baseline (a unidirectional transformer), which achieves average accuracy of 81.6 points. Our best system, XH-Rerank, has an accuracy of 84.38 points, achieving an improvement of 2.7 points.
-H-Rerank & MML-Rerank: These vari- ants rerank the candidates returned by beam search of the xH and MML models according to their marginal probability P(y|zz)
BL2-xH & BL2-MML: These methods se- lect the best L2R or R2L candidate, based on which has higher marginal probability under the xH or MML model.
6 Empirical Results
6.1 Comparison of Methods
Accuracies averaged over languages are shown in Table 1; results by language are in Appendix D. Baselines. BL2, which selects the higher prob- ability among the L2R and R2L hypotheses, im- proves by more than 2.3 points in average accu- racy over the best unidirectional model. This sim- ple scheme serves as an improved baseline against which to compare our fully bidirectional models.
6.2 Improvement by Language
Table shows that the best methods are xH-Rerank (by average accuracy) and BL2-xH (improves upon BL2 on the most languages). Figure 3 illustrates this by showing the difference in accuracy between each of these methods and the best baseline BI.2. The plots show that accuracy difference with BL2 has higher range for xH-Rerank (-2.6% 8.7%) than for BL2-xH (-0.5% to 5.8%). This is because xH-Rerank has the ability to generate new hypotheses, whereas BL2-xH simply discriminates between the same two hypotheses as BL2.
xH & MML. Our bidirectional xH model is clearly more effective than all baselines, having statistically significant degradation in accuracy only 3 languages. The MML method is far less effective, beating L2R and R2L but BL2. MML may suffer from discrepancy between training and inference, since inference optimizes joint prob- ability while training optimizes likelihood.
7 Analysis of Results
7.1 Length Output Forms
xH- & MML-Rerank. Reranking according to marginal probability generally improves both bidi- rectional models. xH-Rerani is the best method overall, beating BI.2 by over 1.75 points in aver- age accuracy. MML-Rerank is better than either unidirectional model but still underperforms BL2. BL2-xH & BL. Selecting the best L,R
Figure 2 shows the accuracies by output form length for BL2 and our best method xH-Rerank. xH-Rerank outperforms the baseline every length (except 10), but especially excels for longer outputs (2 16 characters). This may be due to the bidirec- tional model's decreased risk of "snowballing": it can delay the prediction an uncertain token by

---

efb
afb
amh
amh
amh
arz
arz
be
bel
dan
deu
deu
eng
eng
fin
fro
fra
gro
gro
heb
heb
heb
hebu
hebu
hun
hun
hye
nye
hye
Ita
jap
jap
jap
kat
keb
kin
mkd
mkd
nav
rus
rus
san
sme
sme
spa
sql
swe
swa
tur
tur
0%
50%
100%
0%
50%
100%
0%
50%
100%
Correct;
Wrong
Figure 4: Morphology of words in test set. Percentage forms are suffix-only, prefix-only, neither in the test set for each language.
Figure 5: Analysis for prefix- and suffix-only words. Percentage of forms for each training method that (1) are correct and whose ordering agrees with the form's morphology: (2) are correct but whose ordering does not agree with the form's 's morphology; and (3) are incorrect.
generating the opposite side first, property not shared with unidirectional models.
90%
Accurscy
7.2 How does generation order compare with the morphology of a word?
80%
75%
In this section we consider only forms that can be classified morphologically prefix-only (e.g. will (walk) or suffix-only (e.g. walkled), because these words have obvious split point. Ideally, the bidirectional model will exhibit the desired split point by decoding the left and right sides of the form from their respective directions.
Suffix
Prefix
Nelther
BL2
XH
MML
Figure 6: Accuracy of models by word type. Accuracy of words that are suffix- prefix-only or neither.
We first classify all inflected forms in the test set as suffix-only, prefix-only, or neither. We do this by aligning each lemma-form pair using Levenshtein distance and considering the longest common sub- string that has length of at least be the stem. 6 If the inflected form only has an affix attached the stem, then it is classified as prefix-only or suffix- only; otherwise, it is considered neither."
Figure 4 shows the percentage of words that are prefix-only, suffix-onl,, or neither for each lan- guage. Most languages favor suffix-only inflec- tions, although Swahili strongly prefers prefixes and several other languages have a high proportion of words without a clear affix.
Finally, Figure 5 shows the percentage of words with clear affix on which each bidirectional model has the correct analysis. A correct analysis occurs when the model joins the left and right sequences at the correct split point and returns the correct word.
'For Japanese, we allow the stem to be length due the prevalence Kanji characters in the dataset.
"This heuristic approach is likely to work well on examples without infixes or phonetic alternations that obscure the stem. A potential drawbuck is that it is based on individual lemma- form pairs; a probabilistic method that collects evidence from other examples may be beneficial. However, we feel that the interpretability of this approach as well as qualitative analy supporting its efficacy makes it sufficient for our study.
It is immediately obvious that the MML models tend exhibit the correct analysis, while the xH models generally have the wrong analysis. This make sense because MML learns the latent order-

---

2022
2023
Train
Test
Overall
Unseen
Overall/Unseen
2022
2023
2022
2023
L2R
73.20
74.99
80.26
Unique Lemma
3636.4
753,4
1492.0
94.1
R2L
74.48
75.70
79.65
Unseen Lemma
619.0
94.1
B1.2
75.96
77.23
82.59
Forms per Lemma
2.5
19.3
1.4
15.4
Lemmas per Tagset
100.9
209.9
15.4
22.1
xH
72.76
74.85
84.25
xH-Rerank
72.91
74.72
84.38
B1.2-XH
76.03
78.02
84.00
Table 3:
Dataset Statistics. Number unique lemmas, unseen lemmas, average number of forms per lemma, and average number of lemmas per tagset averaged over all languages for the 2022 and 2023 datasets. 2022 numbers are scaled to the 2023 size (10k train, ~1k test examples) to allow for direct comparison.
Yang et al. (2022)
71,26
74.96
Table 2: Comparison of 2022 and 2023 results. Macro-averaged accuracies over all languages in the SIGMORPHON 2022 and 2023 shared tasks. Accura- cies on test lemmas that unseen in the training data are also reported (for 2023, all test lemmas are unseen in the training data). The average accuracies of the best system (Yang et al., 2022) submitted to the 2022 shared task are also reported.
as much from unidirectional decoding. Since it was trained to treat all orderings equally, would ex- pect to do reasonably well on any given ordering. Nonetheless, it still drops by about points for L2R decoding and about 13 points for R2L decoding. This shows that the full bidirectional generation procedure is crucial to the success of this model.
ing variable, unlike cross-entropy. Despite MML's success at learning this morphology. it tends to have lower accuracy than xH; we explore this by breaking down accuracy by word type in Figure Learning the ordering seems to be harmful when there is no obvious affix: compared with BL2, MML barely drops in accuracy on prefix- and suffix-only forms but degrades greatly when there is no clear split. The xH model, which does not learn ordering, improves in all categories.
7.4 Results on 2022 Shared Task
We also train bidirectional cross-entropy model on the 2022 SIGMORPHON inflection task (Kod- ner et al., 2022), which, unlike the 2023 data, does have lemmas that occur in both the train and test sets. The results are shown in Table 2. All of our methods (including the baselines) outperform the best submitted system (Yang et al., 2022) on the 2022 data; our best method BL2-xH improves by 4.7 points in average accuracy. However, only BL2-xH outperforms the baseline BL2 (barely), which is in stark contrast the 2023 task, where cross-entropy-basee methods beat the baseline considerably. To make the comparison between the years more fair, we evaluate the 2022 models only on lemmas in the test set that did not occur in training. Again, only BL2-xH outperforms the baseline, this time by wider margin; xH and xH-Rerank still underperform.
We conclude that MML models better reflect the stem-affix split than cross-entropy models but have lower accuracy. Improving the performance of MML models while maintaining their linguistic awareness is a promising direction for future work.
7.3 Ablation Study: Does bidirectional decoding help?
In this section, we analyze to what extent the bidi- rectional models' improvement is due to their abil- ity to produce tokens from both sides and meet at any position. To this end, force our trained xH and MML models to decode in a fully L2R R2L manner by setting the log probabilities of tokens in the opposite direction to oo at inference time. The results are shown in Table
We posit that this diserepancy is likely due to the considerably different properties of the 2022 and 2023 datasets, which are shown Table 3. The 2023 languages have far fewer unique lemmas and have many more forms per lemma. Hence, it seems that our bidirectional model improves much more compared with the bascline when there fewer but more "complete" paradigms.
The bidirectional models perform poorly when not permitted to decode from both sides. This is par- ticularly detrimental for the MML model, which is expected as the marginalized training loss enables the model to assign low probabilities to some or- derings. Clearly, our MML model does favor unidirectional orderings.
This investigation shows that the performance of inflection models depends substantially on the data sampling, which is not always controlled for. Kod- ner et al. (2023) makes progress on this matter, but
The xH model, on the other hand, does not suffer

---

Forced L2R
Forced
inflection task, which is fairly straightforward in some ways: there is typically only one output for a given input (unlike translation, for example), and a large part of the output is copied from the input. It would be informative to test the efficacy of our bidi- rectional framework on more diverse generation tasks, such as translation or question-answeriigg From a practical standpoint, the most serious limitation is that, in order to use dynamic program- ming, the model architecture cannot be trained with a causal mask: all hidden states must be recom- puted at each timestep. Further, our xH and MML schemes are quadratic in sequence length. These two properties cause the training time of our bidi- rectional method to be O(Ivl*) in runtime rather than O ?) (like the standard transformer). Alle- viating these constraints would enable a wider vari- ety of experiments on tasks with longer sequences.
Bidi
Bidi-2
R2L
Uni
80.26
79.65
82.59
xH
84.25
71.05
77.31
78.42
MML
81.43
4.68
0.07
2.33
Ablation study on 2023 dataset. Macro- averaged accuracies for bidirectional models decoded using the method of $3.5 (Bidi), or when forced to de- code in an L2R or R2L manner. Bidi-2 indicates the outcome when selecting between the forced unidirec- tional decodings based on which has a higher probability. The unidirectional models (Uni) indicate the accuracies of standard unidirectional transformers and BL2.
Table 4:
does not explicitly examine paradigm "complete- ness", which should be a focus in future studies.
8
Conclusion
We have proposed novel framework for bidirec- tional decoding that allows a model to choose the generation order for each sequence, a major dif- ference from previous work. Further, our method enables an efficient dynamic programming algo- rithm for training, which arises due to an inde- pendence assumption that can be built into waanformer-based architecture. We also present a simple seauhh algorithm for decoding, the outputs of which can optionally be reranked using the likelihood calculation. Our model beats SOTA on both the 2022 and 2023 shared tasks without resorting to data augmentation. Further investiga- tions show that our model is especially effective on longer output words and can implicitly learn the morpheme boundaries of output sequences.
Acknowledgements
This work utilizes resources supported by the Na- tional Science Foundation's 's Major Research In- strumentation program, grant #1725729, as well as the University of Illinois at Urbana-Champanl In particular, we made significant use of the HAL computer system (Kindratenko et al., 2020). We would also like to acknowledge Weights & Biases (Biewald, 2020), which we utilized manage our experiments.
References
Roce Aharoni and Yoav Goldberg, 2017. Morphologi- cal inflection generation with hard monotonic atten- tion. In Proceedings of the 55th Annual Meeting of the Association for Computtaionaa Linguistics ( (Vol- une I: Long Papers), pages 2004-2015, Vancouver, Canada. Association for Computational Linguistics.
There are several avenues for future research. One open question is the extent to which data aug- mentation can improve accuracy. We also leave open the opportunity to explore our bidirectional framework on other sequence tasks, such as ma- chine translation, sion, and named-entity transliteration. Various other architectures could also be investigated. such as the bidirectional attention mechanism of Zhou et al. (2019b) or non-transformer based approaches. Finally, given the effectiveness of MML rerank- ing, it could be worthwhile to explore efficient ap- proaches to decode using marginal probability.
Kamal Al-Sabahi, Zhang Zuping, and Yang Kang. 2018. Bidirectional attentional encoder-decoder model and bidirectional beam search for abstractive summariza- tion. arXiv preprint arXiv: 1809.0666..
wiwpeemme-----hnemmee conver-
Antonios Anastasopoulos and Graham Neubig. 2019. Pushing the limits of low-resoure, morphological in- flection. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP- -JCNLPP, pages 984-996, Hong Kong, China. Association for Com- putational Linguistics.
9 Limitations
We acknowledge several limitations to our work. For one, we only demonstrate experiments on the
*Our xH- Rand method in Appendix F has runtime O( and is also quite effective on the inflection task.

---

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint UrXiv:1409.044
deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In Ad- vances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
Omer Goldman, Khuyagbaatar Batsuren, Khalifa Salam, Aryaman Arora, Garrett Nicolai, Reyt Tsarfaty, and Ekaterina Vylomova. 2023. SIGMORPHON- UniMorph 2023 shared task Typologically di- verse morphological inflection. In Proceedings of the 20th SIGMORPHON Workshop on Computational Research in Phoneties, Phonology, and Morphology. Toronto, Canada. Association for Computational Lin- guistics.
Toms Bergmanis, Katharina Kann, Hinrich Schütze, and Sharon Goldwater. 2017. Training data augmentation for low-resource morphological inflection. In Pro- ceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection, pages 31-39, Vancouver. Association for Computational Linguistics.
David Guriel, Omer Goldman, and Reut Tsarfaty. 2022. Morphological reinflection with multiple arguments: An extended annotation schema and Georgian case study. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Vol- ume 2: Short Papers), pages 196-202, Dublin, Ire- land. Association for Computational Linguistics.
Lukas Biewald. 2020.
Experiment tracking with weights and biases. Software available from wandb.com.
Marc Canby, Aidana Karipbayeva, Bryan Lunt, Sahand Mozaffari, Charlotte Yoder, and Julia Hockenmaier. 2020. University of Illinois submission to the SIG- MORPHON 2020 shared task O: Typologically di- verse morphological inflection. In Proceedings of the 17th SIGMORPHON Workshop Computational Research in Phonetics, Phonology, and Morphology, pages 137-145, Online. Association for Computa- tional Linguisties.
Kelvin Guu, Panupong Pasupat, Evan Liu, and Percy Liang. 2017. From language to programs: Bridging reinforcement learning and maximum marginal like- lihood. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Vol- une 1: Long Papers), pages 1051-1062, Vancouver, Canada. Association for Computational Linguistics.
Ryan Cotterell, Christo Kirov, John Sylak- Mlak-Glasmman, Géraldine Walther, Ekaterina Vylomova, Arya D. Mc- Carthy, Katharina Kann, Sabrina J. Mielke, Garrett Nicolai, Miikka Silfverberg, David Yarowsky, Ja- son Eisner, and Mans Hulden. 2018. The CoNLL- SIGMORPHON 2018 shared task: Universal mor- phological reinflection. In Proceedings of the CONLL-SIGMORPPEER 2018 Shared Task: Univer- sal Morphological Reinflection. pages 1-27, Brussels. Association for Computational Linguistics.
Kenji Imamura and Eiichiro Sumita. 2020. Transformer- based double-token bidirectional autoregressive de- coding in neural machine translation. In Proceedings of the 7th Workshop on Asian Translation, pages 50- 57, Suzhou, China. Association for Computational Linguistics.
Katharina Kann and Hinrich Schütze. 2017. The LMU system for the GGNIJ-SIGMORPHON 2017 shared task on universal morphological reinflection. In Pro- ceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflcction, pages 40 48, Vancouver. Association for Computational Linguistics.
Ryan Cotterell, Christo Kirov, John
Géraldine Walther, Ekaterina Vylomova. Patrick Xia. Manaal Faruqui, Sandra Kübler, David Yarowsky, Jason Eisner, and Mans Hulden. 2017. C. SIGMORPHON 2017 shared task: Universal mor- phological reinflection in 52 languages. In Proceed- ings of the CONLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection, pages 1-30, Vancouver. Association for Computational I. Linguis- tics.
Volodymyr Kindratenko, Dawei Mu, Yan Zhan, John Maloney, Sayed Hadi Hashemi, Benjamin Rabe, Ke Xu, Roy Campbell, Jian Peng, and William Gropp. 2020. Hal: Computer system for scalable deep learn- ing. In Practice and Experience in Advanced Re- search Computing, PEARC '20, page 4 41-48, New York, NY, USA. Association for Computing Machin- ery.
Ryan Cotterell, Christo Kirov, John Sylak-Glassme David Yarowsky, Jason Eisner, and Mans Hulden. 2016. The SIGMORPHON 2016 shared Task- Morphological reinflection. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology. pages 10-22, Berlin, Germany. Association for Com- putational Linguistics.
Jordan Kodner, Salam Khalifa, Khuyagbaatar Bat- suren, Hossep Dolatian, Ryan Cotterell, Faruk Akkus, Antonios Anastasopoulos, Taras Andrushko, Arya- man Arora, Nona Atanalov, Gábor Bella, Elena Budianskaya, Yustinus Ghanggo Ate, Omer Gold- man, David Guriel, Simon Guriel, Silvia Guriel- Agiashvili, Witold Kieras, Andrew Krizhanovsky.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of

---

Natalia Krizhanovsky, Igor Marchenko, Magdalena Markowska, Polina Mashkovtseva, Maria Nepomni- ashchaya, Daria Rodionova, Karina Scheifer, Alexan- dra Sorova, Anastasia Yemelina, Jeremiah Young, and Ekaterina Vylomova. 2022. SIGMORPHON- UniMorph 2022 shared task O: Generalization and typologically diverse morphological inflection. In Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology. and Morphology, pages 176-203, Seattle, Washing- ton. Association for Computational Linguistics.
Yong Shan, Yang Feng, Jinchao Zhang, Fandong Meng. and Wen Zhang. 2019. Improving bidirectional de- coding with dynamic target semanties in neural ma- chine translation. arXiv preprint 1rXiv:191..01590
Ashish Vaswani. Neam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30, Curran Associates, Inc.
Fakaterina Vylomova, Jennifer White, Elizaheth Salesky, Sabrina J. Mielke, Shijie Wu, Edoardo Maria Ponti, Rowan Hall Maudslay, Ran Zmigrod, Josef Valvoda, Svetlana Toldova, Francis Tyers, Elena Klyachko, Ilya Yegorov, Natalia Krizhanovsky, Paula Czarnowska, Irene Nikkarinen, Andrew Krizhanovsky. Tiago Pimentel, Lucas Torroba Henni- gen, Christo Kirov, Garrett Nicolai, Adina Williams, Antonios Anastasopoulos, Hilaria Cruz, Elcanor Chodroff, Ryan Cotterell. Miikka Silfverberg, and Mans Hulden. 2020. SIGMORPHON 2020 shared task O: Typologically diverse morphological inflec- tion. In Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 1-39, Online, Association for Computational Linguisties.
Jordan Kodner, Sarah Payne, Salam Khalifa, and Zoey Liu. 2023. Morphological inflection: A reality check.
Carolin Lawrence, Bhushan Kotnis, and Mathias Niepert. 2019. Attending to future tokens for bidirectional sequence generation. arXiv preprint arXiv:1908.059/5.
Lemao Liu. Andrew Finch, Masao Utiyama, and Eiichiro Sumita. 2016. Agreement on target- bidirectional Istms for sequence-to-sequence learn- ing. In Thirtieth AAAI Conference on Artificial Intel- ligence.
Arya D. McCarthy, Ekaterina Vylomova, Shijie Wu, Chaitanya Malaviya, Lawrence Wolf-Sonkin, Garrett Nicolai, Christo Kirov, Mïikka Silfverberg, Sabrina J. Mielke, Jeffrey Heinz, Ryan Cotterell, and Mans Hulden. 2019. The SIGMORPHON 2019 shared task: Morphological analysis in context and cross- lingual transfer for inflection. In Proceedings of the 16th Workshop on Computational Research in Pho- netics, Phonology, and Morphology, pages 229 244, Florence, Italy. Association for Computational Lin- guistics.
Shijie Wu and Ryan Cotterell. 2019. Exact hard mono- tonic attention for character-level transduction. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1530- 1537, Florence, Italy. Association for Computational Linguistics.
Shijie Wu. Ryan Cotterell, and Mans Hulden. 2021. Ap- plying the transformer to character-level transduction. In Proceedings of the 16th Conference of the Furo- pean Chapter of the Association for Computational Linguistics: Main Volume, pages 1901-1907, Online. Association for Computational Linguisties.
Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019, A discrete hard EM ap- proach for weakly supervised question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP)) pages 2851- 2864, Hong Kong, China. Association for Computa- tional Linguistics.
Jitao Xu and François Yvon. 2021. One source, two targets: Challenges and rewards of dual decoding. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 8533 8546, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Myle Oit Sergey Fdunnv, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Con- ference of North American Chapter of the Associa- tion for Computational Linguistics (Demonstrations) pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
Changbing Yang, Ruixin (Ray) Yang. Garrett Nicolai, and Miika Silfverberg. 2022. Generalizing mor- phological inflection systems to unseen lemmas. In Proceedings of the 19th SIGMORPHION Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 226-235, Seattle, Washing- ton. Association for Computational Linguistics.
Xiangwen Zhang, Jinsong Su, Yue Qin, Yang Liu, Ron- grong Ji, and Hongji Wang. 2018. Asynchronous bidirectional decoding for neural machine translation. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
Tiago Pimentel, Maria Ryskina, Sabrina Mielke, Shijie Wu, Elcanor Chodroff, Brian Leonard, Garrett Nico- lai, Yustinus Ghanggo Ate. Salam Khalifa, Nizar Habash, et al. 2021. Sigmorphon 2021 shared task on morphological reinflection: Generalization across languages. In Proceedings of the 18th SIGMOR- PHON Workshop on Computational Research in Pho- netics Phonology, and Morphology. pages 229-259.
Zhirui Zhang, Shuangzhi Wu, Shujie Liu, Mu Li, Ming Zhou, and Tong Xu. 2019. Regularizing neural ma- chine translation by barget-bidrrettional agreement.

---

Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 450.
B Handling Unknown Characters
If an unknown character is encountered in lemma at test time, then special UNK character is used; however, this character is not explicitly trained. If an UNK character is predicted by the model, then we replace it with the first (leftmost) unknown charac- ter in the lemma; if no such character exists then it is ignored.
Long Zhou, Jiajun Zhang. and Chengqing Zong. 2019a. Synchronous bidirectional neural machine translation. Transactions of the Association for Computational Linguistics, 7:91-105.
Long Zhou, Jiajun Zhang, Chengqing Zong, and Heng Yu. 2019b. Sequence generation: From both sides to the middle. arXiv preprint arXiv:/906.0000
We adopt a special scheme for Japanese, which has very high number of unknown characters. All characters that occur fewer than 100 times in the training
Ran Zmigrod, Tim Vieira, and Ryan Cotterell. 2022. permutation testing for structured test statistics. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, pages 4894 4902, Seattle, United States. Association for Computational Linguistics.
Exact pawred-pppnaeeeeee
considered "unknown". If lemma has unknown tokens, then these are replaced with UNK1 UNK,; the corresponding tokens in the inflected form are replaced as well. this way, the model can learn to copy rare or unknown characters to their appropriate locations in the output. At test time, each predicted unknown token is replaced with its corresponding character in the lemma.
A Datasets, Hyperparameter Tuning, & Validation Accuracies
The languages in the SIGMORPHON 2022 and 2023 datasets are listed in Tables 7 and 8. We experiment with small, medium, and large model sizes for each language, whose configurations and approximate number of parameters can be found in Table 5:
C Tempering the Order Distribution at Train Time
Initial empirical results showed that training with MML loss caused the model to quickly reach "de- generate" state, where every sequence was decoded in the same direction. To encourage the model to explore different orderings at early stage, we temper the order probabilities over warmup pe- riod. The temperature is degraded from initial tem- perature To over period of W steps follows:
S
M
Embed dim
64
128
256
FFN dim
256
512
1024
Num. layers
Num, heads
Learning rate
0.005
0.001
0.001
TO W
(W n) a +1
Tn
Num. params 240k
1-144
~
Table 5: Hyperparameters. Hyperparameters for small, medium, and large models.
The parameter a controls how fast the shift occurs. and n corresponds to the training step. This temper- ature is applied to the softmax of order probabilities for the first W steps of training.
For each language, we train L2R and R2L models (with random initialization) for each hyperparam- eter size (a total of 6 models per language), and select size based the average of the L2R and R2L validation accuracies. The model sizes cho- sen for each language, along with each language's validation accuracies, are reported Tables and 14.
In our experiments, we set 4, 000, T 50 and a =
D All Results
The accuracies for all languages in our study are shown in Table 9 (2023 data) and Table 10 (2022 data). These tables also display L2R- Rerank (which reranks the 5 candidates from the L2R model's beam search under the cross-entropy or MML model), R2L-Rerank, and (L2R+R2L)- Rerank (which reranks the 10 candidates returned from the L2R and R2L's beam search under the cross-entropy or MML model).
Note that the number of parameters vary slightly among languages due different vocabulary sizes (i.e. number of unique characters in the training set), and the bidirectional models also have a small number of cxtra parameters due to the additional classification tokens described $4.

---

100%
3000
Number Test Examples
2500
90%
Accuracy
1500
1000
70%
3 5 7
13 17 27
60%
02
22
22
492
Output Form Length
Output Form Length
Figure 8: Number of Test Examples by Length. Num-
BL2
BL2-XH
test examples across languages by number characters in (correct) output form.
Figure 7: Accuracies of BL22xH and BL2 by Word Length. Average accuracies of BL2 and XL2-xH mod- els over all languages, grouped by length (number of characters) of the output form.
G Additional Results
G.1 Accuracy by Length
E Oracle Scores
Figure in $7.1 compares the accuracy of bidi- rectional method xH-Rerank with that of the base- line BL2 by the length of the output form. Figure 7 shows a similar comparison for BL2-xH (our other best method) with BL2; consistent with the analysis of $6.2, there is less of difference between these methods, but BL2-xH does equal or outperform BL2 all lengths.
Table 11 shows the oracle score for each method; this gives an upper bound for choosing among set of hypotheses. We see that both xH-Rerank and BL2-Rerank approach their respective bounds: the average accuracy for xH-Rerank is within I point of its oracle score, and the average accuracy for BL2-xH is within 2 points of its oracle score.
Figure shows the distribution of output form length across all languages.
F Cross-entropy with Random Path (xH-Rand)
G.2 Accuracy by Part-of-Speech
The cross-entropy loss presented in $3.4 requires enumerating ((gg(²) prefix-suffix pairs. Here, we propose an O( variant in which the join loss is averaged over a random set of suffix-suffix pairs for each word. Specifically, the set S de- fined such that there is only one (i,j) pair for each 1 < k < lyl where i + loss (OH--Ron((O)) is the same the cross-entropy loss of $3.4. Since this loss has O(wy) runtime, it has the same complexity as a standard unidirec- tional loss (assuming all local probabilities take constant time to compute).
Figures 10 and 9 compare the accuracies of xH- Rerank and BL2-xH (our best bidirectional meth- ods) with the accuracy of BL2 by part-of-specch, We see that xH- xH-Rerank maintains or improves ac- curacy over BL2 in all categories except V.MSDR (masdars), and BL2-xH maintains improves accuracy in all categories except V.MSDR and V.PTCP (participles). These categories make up a small fraction the data; this can be seen in Figure 11, which shows the distribution part-of-speech categories across all languages.
k. Otherwise. this
G.3 What orderings does each method prefer?
Table 12 compares the accuracies of this model with the other bidirectional variants discussed in $6. Reranking xH-Rand is slightly better than not reranking, and this performs well: its average accu- racy is almost percentage point higher than BL2 and it improves on 15/27 languages. xH-Rand is better than MML but not as good xH. Nonethe- less, its faster runtime and competitive performance makes this a useful method.
In this section, we investigate the ordering pref- erences for each method: does model prefer to decode words entirely in the L2R R2L direction. or partially in each direction? These results can be seen for each language in Figure 12.
Both the xH and MML methods have a strong tendency to decode words partially in each di-

---

15k
Number of Test Examples
10k
Accuracy
80%
5k
70%
60%
AD] MPTOP
V.CVB
V.MSDR
Speech
BL2
BL2-10
of Speech
Figure 9: Accuracies of BL2-xH and BL2 by Part Speech. Accuracies of BL2 and BL2-xH models averaged over all languages, grouped by part of specch.
Figure 11: Number of Test Examples by Part-of- speech. Number of test examples across all languages by part-of-speech.
100%
Time per examples (8)
L2R
0.276
R2L
0.277
90%
0.724
Accuracy
xH-Rand MML
0.738
80%
0.772
Table 6: Inference times. Average time to perform ference 50 test examples averaged over all languages on the 2023 dataset on NVIDIA V100 GPU's.
70%
60%
N
AD3
V.PTCP
V.VBB
V.MSDR
Speech
down inference by factor of about 3. However, in practice, we barely notice the difference on this task, the test sets have only 1,000 exam- ples each. Given the strong outperformance of the bidirectional methods over the unidirectional base- lines (and even over the naive bidirectional baseline BL2), one must therefore make a tradcoff between time and performance.
Figure 10: Accuracies of xH-Rerank and BL.2 by Part Speech. Accuracies of BL2 and xH- models averaged over all languages, grouped by part of speech.
rection; however, MML models clearly have a higher proportion of words decoded from both di- rections than their xH counterparts. Out of the words decoded entirely in one direction, the xH model shows slight preference for R2L genera- tions, though most languages have words decoded from both directions. On the other hand, for the MML model, no language shows a preference for R2L generations over L2R generations; fact, R2L generations are extremely rare for the MML. models.
G.4 Empirical Inference Times
Given that our bidirectional model must recompute previous hidden states at each timestep during in- ference (see $4), we wish to compare the empirical slowdown in decoding for our bidirectional models compared with unidirectional models. The average number of seconds taken to decode 50 examples is shown in Table 6.
Recomputing hidden states at each step slows

---

XH
MML
afb
amh
arz
arz
bel
bel
dan
dan
deu
deu
eng
eng
fin
fin
fra
fra
grcl
grc
heb
heb
hebu
hun
hun
hye
hye
ita
ita
jap
jap
kat
kat
kir
kir
mkd
mkd
rus
san
san
sme
sme
spa
spa
sqil
sqi
swa
swa
tur
tur
0%
50%
100%
0%
50%
100%
Fully L2R
Neither
Fully RZL
Figure 12: Ordering choices. Percentage of examples for each language and training method that decoded fully L2R, fully R2L, or partially from each direction.

---

Table 7: 2023 Dataset Information Information on each language in 2023 datasct, including language family and genus, baseline accuracies on set, and model size chosen (based on validation accuracies).
Hebrew
Geyptian& Arabic
Ancient Greek
Macedonian
Unvocalized)
Gulf Arabic
Turkish
Swahili
Albanian
Spanish
Sanskrit
Russian
Navajo
Khaling
Georgian
Japanese
Italian
Armenian
Hungarian
Hebrew
French
Finnish
English
German
Danish
Relarusian
Amharic
Language
Sami
Turkic
Nigor-C Niger-Conge Congo
Indo- Indo-Europee European
Indo- Indo-Europed
Uralic
Indo- Indo-Europer European
Indo- Indo-Eurgge
Na- Na-Dené Dené
Indo European
Sino Tibetan
Kartvelian
Japonic
Indo- European
Indo- European
Uralic
Afro-Asialt Asiatic
Afro- Asiatic
Indo European
Indo European
Uralic
Indo- Indo-Europen European
Indo- Ind-EEurpee European
Indo- European
Indo- European
Afro- Afro-Asiatic Asiatic
Afro- Asiatic
Afro Asiatic
Family
European
European
Linguistic Information
Oghuz
Bantu
Alhunian
Romance
Finnic
Indic
Slavic
Southern Athabaskan
Slavic
Kiranti
Karto -Za
Romance
Armenian
Ugric
Semíic
Semitie
Hellenie
Romance
Finnic
Germanic
Germanic
Germunia
Slavic
Semilic
Semitic
Semitic
Genus
hebu/ heb unvoc
Language Code
tur
spu
sme
san
rus
nav
mkd
kir
kat
jap
ita
hye
hun
heb
grc
eng
dou
arz
88.
89.00
82.50
90.40
64.10
61.50
82.10
53.70
90.90
98.30
79.70
93.10
89,30
82.00
77.70
79.40
87.41
49,40
63.70
81 81.60
94.60
80.80
87.70
71.70
87.20
86.20
75.20
Small
L2R Test Accuracies
88.80
92.70
85.000
90.30
63.40
$2.40
83.20
$3.40
89.70
99.40
76.30
93.80
82.90
86.00
77.40
78.50
9325
$3.20
65.10
78.30
94.50
75.70
88.40
73.00
89.60
84,40
74,70
Medium
89.10
86. 90
80,00
88.90
56.50
41.50
84.00
45.60
89,70
95.80
76.60
91.90
91.40
84.00
73.30
76.50
91.14
50.80
57.20
76.90
94.80
73.10
87.10
70.30
87.80
83.50
75.10
Large
87. 97.40
92.90
38.30
93.50
61.10
60.70
84.90
48.90
89.40
95.60
79.50
89.60
93.90
86.40
66.10
71.40
88.22
47.50
67.70
72.60
93.80
76.50
86.20
68.80
88.10
82.80
78.
Small
R2L Test Accuracies
83.
92. 92.90
84, 84,40
91 91.20
65.
44.10
86.00
53.30
92.00
98.30
82.00
91.00
78.60
76.90
71.10
74.10
90.33
39,10
59.50
74.00
94.00
77.10
74,80
87 87.30
89 30
76.10
Medium
80.00
92.40
75.00
87.50
56.40
44.00
80.10
52.60
88.60
98.30
69.00
0 92.60
79.60
65.70
70.40
73.40
87.41
34
50.10
72.60
93.20
78.00
86.50
70.10
86.70
83.20
75.50
Large
small
medium
medium
medium
medium
small
small
small
mediuu
medium
small
medium
small
small
small
medium
large
medium
small
medium
medium
large
medium
large
small
medium
small
Chosen
Model Size

---

Table 8: 2022 Dataset Information. Information on each language in the 2022 datasel, including language family and genus, buseline accuracies on test set, and model size chosen (based on validation accuracies). Only large datasets (7 7,000 train examples) are used.
Khalkha
Old English
Turkish
Slovak
Pomak
Polish
Old Norse
Ludic
Karelian
Korean
Mongolian
Kazakh
Georgian
Armenian
Hungarian
Hebrew
Gothic
Evenki
Assamese
Arahic
Language
Veps
Uraali
Turkic
Indo- European
Indo European
Indo European
Indo- European
Uralic
Uralic
Koreanic
Mongolic
Turkic
Kartvelian
Indo- European
Uralic
Afro-A Asi-Aiiiic
Indo- Indo-Euron
Tungusic
Indo- Indo-Euroge European
Afro- Afro-Asiatic Asiatic
Indo- Indo-Europent
Family
European
Europeun
Linguistic Information
Finnic
Oghuz
Slavic
Slavie
Slavic
Germanic
Finnic
Finnic
Kipchak
Karto-Zai
Armenian
Ugric
Semitie
Germanic
Northem Tungusic
Indic
Semitic
Germanic
Genus
Language Code
vep
tur
slk
poma
pol
non
lud
krl
kor
khk
kuz
Kat
hye
hun
heb
got
cvn
asm
aru
ang
60.91
94.15
91 10
66.98
90.10
82.92
63.16
65.18
57.13
46.26
64.14
84.15
89.90
68. 68.20
47.90
72.52
56.11
81.46
77.74
56.78
Small
L2R Test Accuracies
Medium
61.26
93.85
92.55
65.68
90.40
84.43
74.14
71.49
57.18
48.59
64.34
86.10
91.90
78.40
49.50
73.47
56.05
80.75
76.94
59.93
63.37
93.70
93.25
67.83
88.95
84.73
68:27
67.44
57.54
48.69
62.99
84.50
91.15
75.80
49.35
73.62
54.68
81.11
77.49
62.06
Large
60.56
93.65
93.25
66 88
89 .0.30
82,92
72.32
70.79
57.23
46.97
62.54
83885
90.15
72.65
52.70
70.
57.37
74.42
77.84
60.49
Small
R2L Test Accuracies
Medium
60.41
95.40
94.10
64.23
89.15
82.32
78.34
70.89
58.76
48.59
69.11
88.15
92.90
76.40
50.55
68.51
54.79
84.87
77.59
63.03
62.57
92.90
93.55
67.88
89 89.90
85.89
56.98
72.39
57.64
48.43
68.15
88.95
92.70
75.80
53.10
74.77
52.09
83.57
77.24
64.55
Large
Model Size
large
medium
large
large
medium
large
medium
medium
large
medium
medium
large
medium
medium
large
large-
small
medium
small
large
Chosen

---

Number (p S 0.05)
Number (p & 0.05)
Number (p < 0.05)
tur
swa
sqi medium
spa medium
sme small
sam
BaY
midd
kör
kat
jap
ita
hye
hum
hebbu
grc medium
fra small
fin
eng medium
bel large
arz small
amhh medium
aft small
Table 9: All Accuracies (2023 data), A numberi is starred (*) if it shows statistically significant difference with the best baseline BL.2; a number is colored in green if it improves over BL2 (rogardless of significance) using a paired permutation test (Zmigrod et 2022): and number is bold if it is the best for the language.
Number M
small
medium
medium
small
small
medium
medium
small
medium
small
smm small
small
medium
large
medium
large
medium
Model Size
Average
80.26
88. 88.80 87.40
92.
85.00
90.30 91.20 90.90
63.40
61.50
82 84.90 87.40
53.70 48.90 54.00
89.70
99.40
79.70
93.80
89.30
82.00
77.70 66.10 76.30
78.50 74.10 77.30
91.14 8741 92.95
53.20 39.10 90
63.70 67.70
78. 78.30 74.00 79.20
94.50 94.00 95.60
73.10 78.00 79.70
88 40.40. 87.80
70.30 70.10 73.50
84 89.30
75.20 78. 80.70
L2R
27.20- 88. 89 20.20
Baselines
79.65 82.59
92.90
84.40
65.30 69.90
60.70 63.30
92.00 91.90
98.30
79.50
91.00 92.80
93.90 95.80
86.40 88.40
R2L BL2
90.90
93.10
87.60
99.40
84.10
19.00
88.80
90
3/27
12/27
12/27
19/77
84.25
94.00
96.60
91.00'
93.20'
67.40
67.70'
84.20'
55.10
92.10
99.40
81.30'
94,90
94.40
94.20'
84.30'
83,70"
92.45
56.00'
71.70
83.60"
95.70
80.20
86.50'
72.90
89.10
88.90
84,10'
xH
Standalone
11/27
11/27
5/27
9/27
81.43
89.90
90.50'
82.50'
93.40
75.60'
$9.60'
82.10'
57.10'
91.40
99.40
81 $1.40*
92.30
92.70'
86.50 50
75.00*
84.0 84.09'
53. $3.50'
71.60
77.70
95.80
81.10
83.60'
72.80
87.50
83.40 33.40'
78. 70"
MML
30"
2/27
13/27
12/27
18727
84.38
94.20"
96.60'
91.90
93.30'
67.30
65.90
85.50
55.60
92.40
99.40
82.90
94,90'
93.70'
94.30'
85.00'
83.60'
92.55
56.00'
72.90'
82.70'
95.70
79.70
87.50
72.90
88.7 88.70
88.60
84.60'
xH-Rerank
3/27
12/27
12/27
18/27
84
93. 93.40'
95.60'
89.90'
93.30'
69.00
66.40"
85.40"
55.00
92.40
99.40
82.60
94.
92.10"
94. 20
85.10'
83.60"
92.25
55. 55.70"
73.50"
83,90
96.20
80.8 80,80
86. 30
73.2 73.20
88 90
87.9 67.90
84.90"
MML
7/27
12/27
8/27
17/27
82,90)
93.00
91.40
82.80'
93.30
70.00
66.10
84.70
$7.00
93.20
99.40
83.50
93.50
95.50
88 40
82 00
81 515
86, 86.10'
$3.60
74.90 74.90
77.70
96.00
80.8 80.80
85 35.00"
74.00
87 97
85 90"
81 81.50
xH
MML-Rerank Rerank
MML
10/27
12/27
5/27
9/27
81 81.50
89.90
900.000
82.40"
93.50
75.20"
60.60
83.30'
$7.00"
91.50
99.40
81. 81.10"
92.10
92.70"
86.20
80. 80.10"
75.00"
84.09
$3.30"
71.50
77.90
95.80
81.00
83.60
72.90
87, $7.40"
83.40"
79.20
MML
0/27
15/27
12/27
24/27
84.00
91.00
93.10
88.60'
91.50
71 71.80"
69.10'
87.30
55.10
92.40
99.40
84.70
94.20'
97.20'
91.40"
79. 79.80*
79. 79.30'
93.55
53.70*
74.70'
78.80
95.90
79.70
89.50
74.70
88.70
90.60'
82.20'
XH
BL Discriminator
3/27
17/27
7/27
18/27
83.54
90.30
93.00
85.90"
91.00
70.90
68.60"
87.30
55.60"
91.90
99.40
84.60
93.40
96.40
91.10"
81.20'
78.20
91. 91.04
55.10"
74.40"
78.50
95.80
81.00'
89.60
74.40
89.10
87.30*
80,30
MML
7/27
11/27
9/27
16/27
82.64
92.40"
93.10
89,30
91.50
66
67.10'
83.30
$4.90
93.80"
99.40
82.70'
90.10' 94.60"
85.10' 81 51.50"
81 $1.70"
79 79.30*
91.74 90.53
54 44.60*
72-00* 70. 70.70
81.20 $1"20"
95. 95. 95.60
74.80 74.80'
89 99.30
71. 11.00
89.20 89.20
85.80 85.80'
80 20.20
XH
1,2R- Rerank
'4.80' 93.00
9/27
15/27
3/27
16/27
81.85
91.60
92.80
85.80'
91.20
70.00
65.50
81.20'
56.40
92.00
99.40
84.
79. 10
77.50
55. 80
81.0 81.00
75. 75.70'
85. 85.70'
73.60
88. 88.50
82 20'
78.00'
MML
6/27
18/27
3/27
1
81
88.10
93.30
87.80
91.80
67.90
61.90
86,30
$4,30
93.20
99,10
$1.40"
94.40"
93.90'
88
75 75.20
79 40
88 62-
41 115
72.70"
78.10
94.95
80. 80.30
86 60'
71.60 71.60
88 30
89 99.30
80.00
xH
ReI-Rerank
10/27
15/27
2/27
8/27
81
86.10"
92.90
85.30"
91 91.70
71.10
60.50
86.00
$5.60
92.70
99.00
81.10"
92.20
95 95.40
87.60
76.10
74.20*
84.99*
43.10'
74.00"
78.70
94.80
81 81.50"
85,70'
73.80
89,20
84,70'
76.90*
MML
1/27
14/27
12/27
19/27
84.11
93.30'
93.50
90.30"
92.30'
69.90
66.60'
86.30
36.00
92,60
99.40
8:3 $3.60
94.80"
94.30'
92.70"
81 $1.40*
82.40"
91.64
56.20"
74,80*
81 $1,20"
95.50
80,20
72.10
89.10
89.10
82,90'
XH
(1.2R+R2L)
88.30
4/27
16/27
7/27
14/27
83.00
91.30
93.00
86.60
92.00
75.00"
65.20
84,30"
57.10"
91.90
99.40
83.70
92.40
94.70
88.00
81.10"
76.20
86.20'
59.30*
75.00
79.80
95.80
81.90
85.40"
74.10
88.90
83,60"
79,20
MML
RRerank

---

Table 10: Accuracis (2022 data). number colored in green if improves over BL.2, and number bold it best the language.
poma large
vep large
non large
lud medium
krl
kor
khk medium
kaz medium
Kat
hye
han
heb
got
evn small
asm medium
ara small
ang
Number
Average
medium
large
medium
medium
large
large
medium
medium
large
lurge
large
Size
Model
73.20
63.37
93.85
93.25
67.83
90.40
84.73
74.14
71.49
57.54
48.59
64.34
84.50
91.90
78.40
49.35
73.62
56.11
80.75
77.74
62.06
L2R
Baselines R2L
74.48
62.57
95.40
93.45
67.88
89.15
85,89
78.34
70.89
$7.64
48.59
69.11
88.95
92.90
76.40
53.10
74.77
57.37
84.87
77.84
64.55
75.96
65.43
95.75
94 20
69.75
91.95
87.95
82.19
73.40
59.11
94
70. 36
95
93.60
78.15
53.95
75.33
60.70
85.73
79.45
65.25
BL2
2/20
72.76
63.67
95. 60
94.05
67. 88
89.40
84.88
64.37
64.38
57.64
48.94
65,70
92.00
90.30
77.
49.95
72.17
56.86
83.42
76.19
60.89
xH
Fidirectional
xH Rerank
2/20
72.91
63.32
95.60
94.00
67.93
89,
85.03
63,82
65.33
$7.69
48 48.99
66. 66,70
91.50
90.85
77.00)
49.95
72.62
56.86
83.22
77.09
61.05
BL2-xH
13/20
76.03
65.48
94,90
70.14
90.
88.05
80.62
72.34
59,93
48.99
70.96
89.75
93.90
77.
52.00
75.83
61.10
87.44
79.20
65.77
60
L2R Rerank xH
0V20
73.49
64.33
95.20
93.55
69.73
89. .0
84.83
70.34
69.59
57.59
48.79
63.19
87.70
92.10
77.60
49.35
74.32
56.51
83.87
78.70
62.77
Baselines Rerank
R2L R2L-Rerank Rerank-HH (L2R+R2L) Rerank xXH
3/20
74.40
63.87
95.80
93.10
68.88 63.83
89 89.40
85.84
77.13
68. 68.99
58.81
48 48.94
69.46
90.30
92.60
76.60)
50.80
74.02
58.00
83.67
77.59
64.20
3/20
74.30
64.53
95.75
94.15
69.03
89.70
85.64
71.26
68.49
58.81
48.99
69.01
91.70
92.05
77.05
50.35
74.07
58.35
85.18
78.45
63.53

---

Model
Baselines
Bidirectional xH-Rand
Size
12R
R2I.
B1.2
B1.10
xH
MMI.
afb
small
89.00
89.50
85.10
93.40
86.20
83.80
84,90
amh
medium
89.30
96.40
92.00
96.70
89.00
89.00
89.20
arz
small
94.90
94.70
90.70
96.30
89.20
88.50
89.60
bel large
82.60
82.40
77.40
87,30
75.10
74.70
78.50
dan
medium
95.00
92.40
92.50
97.00
88.50
89.10
87.80
deu
large
81.60
86.50
82.30
90.30
81.20
81.30
83.80
eng
medium
98.20
97.10
96.70
98.70
96.40
96.70
96.90
fin
medium
89.50
83.60
81.40
91.00
84,50
86.40
80.40
fra
small
86.00
89.60
79.40
94,90
74.40
73.20
80,90
gre
medium
63.00
48.50
55.80
69.90
56.00
49.10
55.50
heb large
95.07
90.43
94.36
96.68
92.45
89.43
86.30
hebu medium
86.80
83,50
82.40
90.70
85 60
86.4
88.80
hun small
88.30
83.60
83.60
91.30
85 $5.30
84.50
84.80
hye
small
87.00
90.60
92.00
95.60
94.30
91.90
88.70
ita small
92.60
97.60
97.90
98.40
94.80
95.10
95.80
medium
97.00
94.50
94.70
97.00
94.90
93.60
93.60
kat
small
85.80
85.20
85.80
88.80
83.20
80.90
84,90
klr
medium
100.00
99.60
99.40
100.00
99.40
99.30
100.00
mkd
medium
96.10
96.60
93.60
98.20
92.60
93.20
94,20
small
63.70
65.30
58.30
72.40
56.40
53.60
59.40
small
89.70
93.50
90.20
94.70
86.10
87.80
87.30
small
81.30
72.90
72.40
84.30
68.20
67.30
75.60
medium
78.00
77.20
73.80
86.00
70.80
70.80
77.50
spa
medium
93.90
93.30
92.30
95.10
93.30
93.00
93.70
sqi
medium
95.20
90.80
90.00
97.50
92.90
89.40
82.90
swa
medium
96.40
97.40
93.10
97.70
97.20
97.70
91.40
small
94.70
90.20
91.10
95.80
94.30
90.20
94.50
Average
88.54
87.52
85.86
92.43
85.27
84.29
85.44
Table Oracle Accuracies (2023 data). Accuracies of each method if an oracle were used select among the hypotheses returned from beam search. In the case of BL,00 oracle chooses out of the 10 candidates returned from L2R and R2L's beam search; in the case BL2, an oracle chooses between the best L2R and best R2L hypothesis.

---

Model
Baselines R2L
Standalone xH-Rand
Reranker xH-Rand
Size
L2R
BL2
xH
MML
xH
MML
afb small
75,20
78.10
80.70
84.10
81.00
78.70
84.60
82.40
79.20
amh medium
84.40
89,30
88.90
88.90
88.50
83.40
88.60
88.40
83.40
arz small
87.20
88. 10
89.20
89.10
87.80
87.50
88. 70
87.50
87.40
bel large
70.30
70.10
73.50
72.90
70.80
72.80
72.90
72.30
72.90
dan medium
88
87.80
88,80
86.50
87.70
83.60
87.50
87.40
83.60
deu large
73.10
78.00
79.70
80.20
80.90
81.10
79.70
80.50
81.00
eng
medium
94,50
94.00
95.60
95.70
95.80
95.80
95.70
95.90
95.80
fin medium
78.30
74.00
79.20
83,60
85.10
77.70
82.70
85,90
77.90
fra small
63.70
67.70
69.30
71.70
72.10
71.60
72.90
72.30
71.50
medium
53,20
39.10
48.90
56,00
48.90
53.50
56.00
49.00
53.30
heb large
91.14
87.41
92.95
92.45
89.12
84.09
92.45
89.22
84.09
hehu medium
78.50
74.10
77.30
83.70
86.10
75.00
83.60
86,20
75.00
hun small
77.70
66.10
76.30
84.30
83.80
79.30
85.00
84.30
80.10
hye small
82.00
86.40
88.40
94.20
90.60
86.50
94.30
91.30
86. 86.20
ita small
89.30
93.90
95.80
94.40
94.30
92.70
93.70
94.70
92.70
medium
93.80
91.00
92.80
94.90
92.70
92.30
94.90
93.60
92.
kat small
79.70
79.50
84.10
81.30
79.80
81.40
82.90
80.80
81.10
kir
medium
99.40
98.30
99.40
99.40
99.20
99.40
99.40
99.20
99.40
mkd
medium
89.70
92.00
91.90
92.10
92.80
91.40
92.40
93.20
91.50
small
53.70
48.90
54.00
55.10
50.40
57.10
55.60
$2.10
57,00
small
82,10
84.90
87.40
84.20
85,30
82,10
85.50
86,80
83.30
small
61.50
60.70
63.30
67.70
65.50
59.60
65.90
66.80
60.60
medium
63.40
65.30
69.90
67.40
67.30
75.60
67.30
67.30
75.20
spa
medium
90.30
91.20
90.90
93.20
93.40
93.30
93.00
93.50
sqi
medium
85.00
84.40
87.60
91.00
88.30
82.50
91.90
87,90
82.40
swa
medium
92.70
92.90
93.10
96.90
90.50
97.30
90.50
tur
small
87.40
90.90
94.00
89,50
94.20
89,30
89.90
Average Number
80.26
79.65
82.59
84.25
83.08
81.43
84.38
83.50
81.50
19/27
14/27
9/27
18/27
15/27
9/27
Table 12: Random Cross-Entropp Accuracies (2023 data), number is colored green it improves over BL.2, and number is bold if it is the best for the language.

---

Model Size
L2R Val. Accuracies
R2L
Val. Accuracies
Bidirectional Val. Accuracies
Language
Chosen
M
S
M
L
xHI-Rand
MML
afb
small
74.5
77.4
74.8
79.9
76.6
74.4
83.7
81.1
81.6
amh
medium
81.1
84.8
83.0
81.8
83,9
82.1
88.7
90.1
86.1
small
87.9
89.3
88.3
89.0
87.4
87.9
89.6
89.4
88.2
bel large
73.1
73.3
74.5
73.2
75.5
74.8
76.0
77.1
76.4
dan medium
88.9
90.0
89.8
89.4
89.9
88.5
87.3
90.3
87.2
deu large
79.3
80.1
78.6
76.7
76.9
79.6
80.5
81.1
77.8
eng medium
94.8
95.9
94.8
94.9
94.6
93.8
95.2
95.9
95.0
fin medium
93.7
96.8
92.4
92.6
90.8
91.2
98.1
96.3
94,8
fra small
76.3
80.7
73.5
79.7
74.9
72.4
83.2
82.5
80.0
gre medium
57.1
62.5
57.4
56.5
57.5
54.7
67.2
63.3
66.6
hebu medium
91.3
92.7
90.7
92.8
92.4
91.6
94.3
93.9
93.5
heb large
89.9
90.7
85.4
86.6
89.9
93.3
93.0
90.4
small
85.9
81.7
74.7
79.5
hye small
86.5
84.3
84.6
75.4
70.1
95.1
92.8
94.4
ita small
95.8
94.7
96.8
jap medium
89.6
88.6
89.0
92.9
92.6
small
81.6
79.4
74.7
71.3
80.0
79.1
kir medium
99.6
99.4
98.8
98.7
99.4
99.8
99.8
99.8
mkd medium
93.2
94.0
92.6
91.3
93.6
90.6
96.0
95.7
93.9
nav small
59.3
53.5
50.1
54.4
$7.6
$9.0
rus small
88.3
87.5
86.3
87.9
87.3
85.7
88.6
87.5
86.5
sme medium
72.2
74.7
72.0
70.2
70.3
62.5
76.0
79.0
81.4
spa medium
95.2
96.0
91.3
94.5
94.0
94.1
98.5
97.5
95.9
sqi medium
89.2
89.7
79.5
90.4
90.6
74.8
93.9
93.4
89.5
swa
medium
97.1
96.5
92.4
96.8
97.6
95.5
97.6
97.6
97.3
tur small
92.1
92.1
87.5
90.7
82.4
70.7
97.1
94.6
90.0
san small
67.5
63.2
57.6
63.1
46.2
47.9
76.2
75.6
71.4
Table 13: Validation Accuracies (2023 data). Validation accuracies for each language in the 2023 dataset. Validation accuracies on unidirectional models are used for hyperparameter selection. The bidirectional validation accuracies (xH, xH-Rand, MML.) reported for the chosen model size for each language.

---

Model Size
L2R
Val. Accuracies
R2L
Val. Accuracies
Language
xH
Chosen
S
M
L
S
M
L
ang
large
59.5
64.1
63.9
62.7
64.5
ara small
75.0
74.5
75.3
75.9
75.7
74.9
asm medium
83.6
84.4
85.0
76.9
88.0
85.1
evn small
52.1
50.9
53.4
49.3
got large
81.0
81.7
81.8
80.0
79.0
82.1
78.3
heb large
26.1
27.7
31.2
31.3
31.7
hun medium
67.1
75.0
73.4
70.6
75.9
75.0
76.5
hye
medium
91.2
93.4
92.1
91.0
93.9
93.4
91.7
kat large
86.0
88.7
87.7
86.0
89.2
90.4
92.6
kaz medium
65.6
67.7
67.3
54.0
61.2
60.3
57.7
khk medium
38.0
39.8
39.2
39.2
39.7
39.2
39.2
kor
large
56.9
57.6
58.6
57.8
59.6
58.9
56.5
krl
medium
64.6
66.8
64.7
65.5
65.7
67.0
62.7
lud
medium
59.8
71.5
65.2
71.5
76.6
55.3
60.5
non large
85.7
88.9
86.9
87.4
86.5
88.8
89.1 1
pol medium
90.3
91.9
90.3
90.5
90.5
91.0
91.0
poma large
49.4
52.7
53.6
49.3
55.4
$7.5
$2.9
slk large
90.7
91.4
92.4
92.8
92.7
92.9
92.7
tur
medium
96.0
96.5
96.4
95.7
97.2
96.0
97.3
vep large
63.0
61.6
63.5
59.7
59.6
60.9
64.1
Table 14: Validation Accuracies (2022 data). Validation accuracies for each language in the 2022
dataset. Validation accuracies on unidirectional models are used for hyperparameter selection. The xH validation accuracies are reported for the chosen model size for each language.


Reviews:
[{"review_id": "r1J5tbilHe", "reviewer": "Reviewer_HEuK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reproducibility**\n\nWe’d first like to point out a minor oversight of the reviewer regarding reproducibility: the reviewer writes “[The paper] doesn't provide any code or mention the possibility of future release of the code.” To the contrary, the footnote to our abstract on page 1 states “Our code will be made available on Github for the camera-ready version.” We commit to doing exactly this; we will publish the code as an extension to the fairseq library, so that interested researchers can easily use (and extend or modify) our method within a well-known framework on a variety of datasets.\n\n**Runtime**\n\nWe do acknowledge the increase in training runtime (from $O(|y|^2$) to $O(|y|^4)$), which, as the reviewer says, could limit the applicability of the method to other tasks with longer sequences; we try to be up front about this point in the paper, both in the relevant section of the main paper (3.4) and in the Limitations section. Addressing this issue would be a key area of future work. We would however like to make a few comments on this topic:\n* In Appendix F, we actually provide an $O(|y|^3)$ method called xH-Rand that improves over the L2R baseline by an average of 3.24 points (compared with 4.12 for the $O(|y|^4)$ method). This shows that more efficient methods under the bidirectional framework are possible and still yield significant improvement over the baselines.\n* We designed our framework in such a way that it could be used to train an architecture with inner state preservation under the xH-Rand loss in $O(|y|^2)$ time, the same complexity as a standard transformer. Note that a model that uses inner state preservation *and* attention across the prefix and suffix would render DP impossible (see section 3.3); however, the probability factorization and generation procedure of decoding such a model from the outside-in would remain the same.\n* Such efficiency improvements are fertile ground for future research; this paper’s primary goal is to provide a starting point for such work and demonstrate the feasibility of a flexible bidirectional decoding framework.\n\n**Does the complexity of the model make the performance boost worth it?**\n\nThe reviewer makes a practical point that training the more complicated bidirectional model xH-Rerank may not be worth the hassle compared with training BL2 (our simple bidirectional baseline), since its improvement is a “modest” 2.3 points. We would like to make the following points about this:\n* In the time since we submitted this paper, the official shared task paper (Goldman et al., 2023) was released. A total of three teams submitted, and only our system beats the (neural) baseline, which had an average accuracy of 81.6 compared with ours of 84.3. The best submissions from the other teams were 76.9 (TUB) and 72.4 (AZ) respectively. Hence, it seems that the magnitude of our system’s improvement is actually in line with the differences between other systems’ accuracies. We intend to include these results as a comparison in the final version of our paper.\n* As we show in the paper, our model sets SOTA on the 2022 shared task by 4.7 points. Therefore, our system is SOTA on both the 2022 and 2023 datasets. Further, our significance tests on the 2023 dataset show that xH-Rerank has a statistically significant degradation on only 2/27 languages, suggesting that our model’s improvements are real. Therefore, we believe that a practitioner who primarily cares about performance should favor our model over BL2.\n* We believe the larger question of whether it is worth training the full bidirectional model for this improvement over the simpler BL2 should be left for downstream researchers, whose use cases and goals may vary greatly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Improvement Over Baselines: The paper introduces a bidirectional model that consistently outperforms baselines, including unidirectional models and a baseline (BL2) that selects higher probabilities between L2R and R2L hypotheses. \n2. Effective Reranking Strategies: The paper explores different reranking strategies based on marginal probabilities, both under the proposed xH model and the Maximum Marginal Likelihood (MML) approach. These strategies show enhancement in performance compared to the baselines, showcasing the potential of these techniques for improving bidirectional decoding.\n3. Language and Output-Length Agnostic Improvement: The results show improvement across various languages and different output form lengths. This demonstrates the robustness of the proposed bidirectional approach across different linguistic contexts.\n4. Decoding Strategies: The paper introduces effective decoding strategies, such as beam search, for optimizing joint probabilities. While direct likelihood-based decoding might not be computationally feasible, the paper adopts practical strategies for achieving accurate decoding.\n6. Clear Presentation of Results: The paper presents a comprehensive analysis based on the results, and it uses many plots, making it easy for readers to follow the research methodology, results, and implications.", "reasons_to_reject": "1. ~~Reproducibility: The paper does not provide information on code availability, making it difficult to assess the reproducibility of the results. It also doesn't provide any code or mention the possibility of future release of the code. Implementing the model and the some of the hyperparameter schedules (see their appendix) seems far from trivial, so it would be hard for the model to be adopted by the research community without some well-documented code.~~\n\n2. The model is clearly useful for morphological inflection; it seems to be able to both inflect and be used for segmentation (despite only being trained for one of the tasks). However, it's suitability for other sequence generation tasks seems limited, and so I'm skeptical of the generality of the framework. I'm hard pressed to think of tasks outside of word-level ones where there is a natural split between two components like there is between stem and affix. As mentioned in the limitations section, this framework incurs significant computational burden, going from |y|^2 to |y|^4. Thus, the framework exacerbates rather than alleviates one of the most poignant critique of transformer-based models which is their quadratic time and space complexity, and is thus (in it's current form) unlikely to be adopted outside of word-level tasks like those regularly presented in SIGMORPHON shared task. Overall, I believe that the authors claim that this is a generally practical bidirectional framework is a strong overreach. \n\n3. For all the complexity in implementing and training this model to convergence, the improvement over the naive bidirectional baseline (returning the best L2R or R2L hypothesis) is only modest (2.3 points).", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors have introduced a new approach for bidirectional decoding, where a model can decide the (L2R or R2L) order of generation for each subsequence. They've incorporated this into a transformer-based architecture, enabling efficient training through a dynamic programming algorithm. It seems to be able to implicitly learn the morpheme structure of words.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eKkMqEr1fN", "reviewer": "Reviewer_4BJv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comparison with other bidirectional approaches**\n\nThe only iteration of the SIGMORPHON shared task where a team submitted a bidirectional approach was 2020, where one team (Canby et al., 2020) used the method of Zhou et al., 2019a. In our initial experiments, we did train xH models using our framework on three of the datasets from that year, and all three of them beat the approach from Canby et al., 2020: Old English (79.69 vs. 78.40), Turkmen (85.78 vs. 85.60), and Sanskrit (93.81 vs. 93.40). However, we ultimately decided to use more recent shared task datasets (2022 and 2023) for this paper, and so we could not directly compare with the results from 2020 in a systematic way.\n\nAs the reviewer writes, we chose not to re-implement previous bidirectional decoders for this task. We chose instead to spend our energy training and fine-tuning high-quality L2R and R2L baselines. These are standard unidirectional transformers and actually beat previous SOTA on the 2022 datasets: the L2R baseline beats SOTA by about 1.9 points and the R2L baseline beats it by about 3.2 points, as shown in Table 2 of the paper. We also opt to focus our comparisons on SOTA rather than previous bidirectional decoders, although we do also provide a simple bidirectional baseline BL2 as a benchmark. In the end, even though it may be interesting to compare with other bidirectional decoders, we feel that it is sufficient to compare with well-trained baselines, BL2, and SOTA on the years in question.\n\n**Ablation Studies**\n\nThe reviewer asks “how do we know empirical improvement comes from being able to predict the decoding direction and being able to meet at any position predicted by the model?” This is a very good question, and in our rebuttal we provide two simple insights into this as follows; we plan to include these in the final version of our paper.\n\n*Study 1*\n\nFirst, we simply compute the “confusion matrix” for our bidirectional model xH and our best baseline BL2 over all languages: this shows the number of examples that both models get correct, that one model is correct and the other is incorrect, and that both models get incorrect:\n\n| |xH✔︎ | xH✗ |\n| -------- | --------| ------- |\n|BL2✔︎|21210|1082|\t\n|BL2✗|1530|3171|\n\nThe xH model clearly has a net improvement of 448 (1530 – 1082) examples over the BL2 baseline. The question is whether the 1,530 examples that the xH model gets right and BL2 gets wrong are due to xH’s ability to meet at any position in the output string. We thus compute the percentage of examples in each of these cells for which the xH model chooses to generate from both directions (rather than in a purely L2R or R2L direction, which is what BL2 does):\n\n|\t|xH✔︎|\txH✗|\n| -------- | --------| ------- |\n|BL2✔︎\t|76.5%|94.2%|\n|BL2✗|92.2%|87.6%|\n\nWe see that in all cells, the xH model generally favors decoding words from both sides; however, it strongly favors this in the two diagonal cells, where one model is correct and the other is wrong. Most importantly, 92.2% of examples that are corrected by xH (bottom left cell) are decoded from both sides, compared with 76.5% of examples that both models get right. This suggests that the improvement over BL2 does indeed have to do with decoding order. As a side note, about 94.2% of examples that the xH model gets incorrect and BL2 gets correct (top right cell) are decoded from both sides; hence, in some cases decoding from both sides hurts accuracy. However, the confusion matrix shows that xH has a net improvement of 448, so, when looked at in its entirety, the new decoding method helps.\n\n*Study 2*\n\nHere, we perform a more traditional ablation study: we take our bidirectional models (xH and MML) and force decoding to be either fully L2R or fully R2L. We do this by setting the log probabilities of the opposite direction to $-\\infty$ so that the beam search algorithm cannot choose them. The results are shown below in a format similar to Table 1 of the paper (averaged over all languages):\n\n| Model | Avg Acc. |\n| -------- | ------- |\n|L2R|80.26|\n|R2L|79.65|\n|BL2|82.59|\n|xH|84.25|\n|MML|81.43|\n|xH-forced L2R decoding|71.05|\n|xH-forced R2L decoding|77.31|\n|xH2|78.42|\n|MML-forced L2R decoding|4.68|\n|MML-forced R2L decoding|0.07|\n|MML2\t|2.33|\n\nHere, xH2 and MML2 are analogous to BL2 in that they pick the best option from the L2R-decoded sequence and the R2L-decoded sequence.\n\nClearly, the bidirectional models perform poorly when they are not allowed to be decoded as bidirectional models. This is particularly clear for MML, and it makes sense this would be detrimental for it: since it was trained with marginal likelihood, it can learn the order probabilities and thus assign low probability to some orderings (see Section 3.4 of the paper). Clearly MML does not favor unidirectional orders (also explored in Appendix G.3). Thus, the bidirectional nature of the MML model is crucial to its success.\n\nThe xH model, on the other hand, does not suffer detrimental accuracy drops when forced to decode in the L2R or R2L direction; this is because it was trained to treat all orderings equally, so it should do reasonably well on any given ordering. However, the xH model with unidirectional decoding does suffer a drop between about 5 and 13 points compared with the xH model with full bidirectional decoding. This shows that following the generation procedure of Section 3.5 is critical to the success of the xH model, which is the model that sets SOTA.\n\n**Note about SOTA on 2023 data**\n\nFinally, we would like to make the reviewer aware that the official shared task paper (Goldman et al., 2023) was released in the time since we submitted this paper. It turns out that a total of three teams submitted, and only our system beats the (neural) baseline, which had an average accuracy of 81.6 compared with ours of 84.3. The best submissions from the other teams were 76.9 (TUB) and 72.4 (AZ) respectively. We will include this comparison in the final report, since it shows that we set SOTA on the 2023 shared task (as well as the 2022 shared task, which is already included in our paper).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposes a novel bidirectional decoding framework that chooses to generate a token on the left, generate a token on the right, or join the left and right sequences.\n\nThe proposed approach demonstrates significant improvements in accuracy compared to conventional left-to-right models, particularly in the context of morphological inflection tasks.\n\nThe paper sets a new state-of-the-art (SOTA) result for the 2022 shared task, showcasing the practical value and relevance of the proposed framework for real-world NLP applications.", "reasons_to_reject": "Lack of comparison systems that use bidirectional approaches: In (Zhou et al., 2019b; Imamura and Sumita, 2020), the model decodes interactively from both directions at the same time and meets in the center. In contrast, this model decodes interactively from both directions and can meet at any position predicted by the model itself, with the ability to predict the decoding direction at each step. However, the comparison is made with just a naive bidirectional baseline that returns either the L2R or R2L hypothesis.\n\nLack of ablation studies: The proposed model can predict the decoding direction at each step, allowing it to decode interactively from both directions and now can meet at any position predicted by the model itself, rather than just meeting in the center (Zhou et al. 2019b). But how do we know empirical improvement comes from being able to predict the decoding direction and being able to meet at any position predicted by the model?", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper is a work on using bidirectional decoding to address unidirectional decoding issues. This paper proposes a bidirectional decoding method that chooses to generate a token on the left, generate a token on the right, or join the left and right sequences. The proposed method is tested on morphological inflection task.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TcFbEFkv7k", "reviewer": "Reviewer_qVTT", "summary": "", "questions": "Question A: could you elaborate on how the dynamic programming is achieved on the attention mechanism?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dynamic programming**\n\nThe reviewer writes “the complexity of attention mechanism in transformer is not polynomial” - this is incorrect: the standard transformer’s attention mechanism has $O(|y|^2)$ complexity, which is polynomial. For our bidirectional method, the DP table $F$ is of size $|y|\\times |y|$, and each row represents a token in the L2R direction and each column represents a token in the R2L direction (the top left corner is empty prefix-empty suffix). DP will work as long as $F[i,j]$ depends only on a fixed number of previously computed cells and not on all possible paths to that cell (which would be exponential). In our case, $F[i,j]$ depends only on $F[i,j-1]$ and $F[i-1,j]$, and the local probabilities involved with predicting the $i$th L2R letter and the $j$th R2L letter. Crucially, these local probabilities cannot rely on previous hidden states, since these would be based on the path through the table up to this cell; instead, we recompute all hidden states at each cell $F[i,j]$, which incurs an additional $O(|y|)$ in complexity. Making sure that the local probabilities are path-independent is what enables DP to work on this task.\n\n**Inference Times**\n\nIt turns out that the increase in runtime only has a minor impact on empirical decoding speed. As part of this rebuttal, we timed inference on the trained models for the baselines (L2R and R2L) and the bidirectional variants. The following table reports the average time it takes to decode 50 examples across all languages for each model type (on 4 NVIDIA V100 GPU’s). In the final version of our paper, we plan to present this information with box plots or a similar figure.\n\n|  Model  | Time per 50 examples (s) |\n| -------- | ------- |\n| L2R | 0.276   |\n| R2L | 0.277     |\n|xH|0.724|\n| xH-Rand    | 0.738   |\n|MML\t|\t0.772|\n\nClearly, the need to recompute hidden states at each step slows inference down by about 3x. However, in practice we barely notice the difference on this task, as the test sets have only 1,000 examples. Given the strong outperformance of the bidirectional methods over the unidirectional baselines (and even over the naive bidirectional baseline BL2), one must therefore make a tradeoff between time and performance. Nonetheless, a major thrust for future work in this area is efficiency: how we can maintain the appealing aspects of the bidirectional decoding framework (e.g. decoding from the outside-in and the strong performance gains) while speeding up the model.\n\n**Note about SOTA on 2023 data**\n\nFinally, we would like to make the reviewer aware that the official shared task paper (Goldman et al., 2023) was released in the time since we submitted this paper. It turns out that a total of three teams submitted, and only our system beats the (neural) baseline, which had an average accuracy of 81.6 compared with ours of 84.3. The best submissions from the other teams were 76.9 (TUB) and 72.4 (AZ) respectively. We will include this comparison in the final report, since it shows that we set SOTA on the 2023 shared task (as well as the 2022 shared task, which is already included in our paper).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. I think the application of bidirectional decoding to morphological inflection is new and reasonable.  The model is able to decide which generation order is best for each sequence, and can produce part of a sequence from each direction. So the search space is greatly expanded comparing with pure L2R decoding. \n\n2. The experiment is well designed and related work comparison is sufficient, achieving SOTA on SIGMORPHON 2022 shared task.\n", "reasons_to_reject": "1. I am a little confused by the application of dynamic programming. Since the complexity of attention mechanism in transformer is not polynomial, so it is not feasible for dynamic programming.  As the author mentioned at the end of Sec4, \"since this architecture does have cross-attention ...  the decoder hidden states must be recomputed at each time step to allow for DP\".  I think it would be very time consuming for this computation.  So I suggest the author to list out the decoding speed results comparison to show a full picture of this work.\n", "questions_to_authors": "Question A: could you elaborate on how the dynamic programming is achieved on the attention mechanism?", "paper_topic_and_main_contributions": "This paper introduce a bidirectional decoding framework for morphological inflection. Different from previous works, this work generates sequences from both sides to the middle and utilize an efficient dynamic programming algorithm for training, combing the merits of both R2L and L2R decoders.  \nExperimental results show that the model performs particularly well on long sequences and achieve good results on SIGMORPHON inflection tasks.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel framework for decoding in the morphological inflection task. Instead of standard left-to-right decoding, the model chooses at each step whether to generate a token on the left, on the right, or merge the left and right sequences. The approach leads to improvements over baselines and SoTA models in the 2022 and 2023 shared tasks on morphological inflection. The reviewers agree that the model is novel, reasonable, scalable to other languages, and yields better results than (more traditional) left-to-right decoders. Some reviewers note that the paper lacks ablation studies, comparison to other bidirectional models, and the model complexity is higher, but the authors carefully, neatly, and in a very detailed way addressed all of them.
Overall, the paper provides a strong model for morphological inflection that might become a new SoTA. The paper should gain sufficient interest in the community of NLP/computational morphology.
Also, it's especially great to see that the paper authors care about time complexity and use dynamic programming!
Decision: Accept-Findings