Accelerating Transformer Inference for Translation via Parallel Decoding
Year: 2023

Abstract:
Autoregressive decoding limits the efficiency of transformers for Machine Translation (MT). The community proposed specific network architectures and learning-based methods to solve this issue, which are expensive and require changes to the MT model, trading inference speed at the cost of the translation quality. In this paper, we propose to address the problem from the point of view of decoding algorithms, as a less explored but rather compelling direction. We propose to reframe the standard greedy autoregressive decoding of MT with a parallel formulation leveraging Jacobi and Gauss-Seidel fixed-point iteration methods for fast inference. This formulation allows to speed up existing models without training or modifications while retaining translation quality. We present three parallel decoding algorithms and test them on different languages and models showing how the parallelization introduces a speedup up to 38% w.r.t. the standard autoregressive decoding and nearly 2x when scaling the method on parallel resources. Finally, we introduce a decoding dependency graph visualizer (DDGviz) that let us see how the model has learned the conditional dependence between tokens and inspect the decoding procedure.

heading: 1 Introduction
text: In recent years there have been dramatic improvements in Machine Translation (MT) (Edunov et al., 2018; Liu et al., 2020) thanks to the transition to neural models and the advent of the Transformer architecture (Vaswani et al., 2017). These models can produce high-quality translations while being extremely parallelizable during training. However, Transformers are used sequentially at inference time, generating one token per time (i.e., sending each token as input for the next autoregressive iteration). This process of autoregressive inference hampers the efficiency of neural machine translation systems in terms of latency, limiting applications and portability. Considering that these systems are
extensively used in production multiple times to produce new translations (e.g., Google Translate1, DeepL Translator2), even a minor speedup would be beneficial in the long run, especially if the translation is done on embedded devices.
To address this issue, the community proposed ad-hoc trained models specific for parallel machine translation under the umbrella term of NonAutoregressive Machine Translation models (NAT) (Gu et al., 2018). These models produce the translation in parallel but require (i) a complete reengineering of the MT system, (ii) extensive training resources and (iii) complex design choices like distillation from larger autoregressive models. These requirements are quite demanding and not easily satisfiable. For example, production systems are heavily optimized for hardware and software and even introducing a minimal modification requires non-trivial human effort (Wu et al., 2016; Kim et al., 2019). Furthermore, training a new model from scratch is not always possible due to non-released training data or low-resource languages having few or lacking parallel corpora.
In this paper, we propose to address the problem of parallel machine translation with an orthogonal approach consisting in novel decoding algorithms that work in parallel and can be used on top of existing autoregressive models for MT. We overcome previous limitations with a flexible and generic method that does not require any modification to the model or costly retraining. Specifically, inspired by previous successes in speeding up feedforward computation for image generation (Song et al., 2021b), we reframe the greedy autoregressive decoding for MT as a system of nonlinear equations solvable in parallel. This simple formulation speeds up the decoding procedure by using fixed-point iteration methods like Jacobi and GaussSeidel while having mathematical guarantees on
1https://translate.google.com/ 2https://www.deepl.com/
12336
the quality of the translation. A high-level description of the method is available in (Fig. 1). Our contributions can be summarized as the following:
• We reframe the standard greedy autoregressive decoding procedure in MT with a parallel formulation, introducing three parallel decoding algorithms (PJ, PGJ, HGJ) and a stopping condition that preserves translation quality.
• We perform extensive experiments with different transformer sizes (base and large) and datasets, showing speedups up to 38% in time, obtaining a nearly 2× speedup when scaling the model on parallel resources while preserving quality. To the best of our knowledge, this is one of the first studies to introduce a speedup in multilingual machine translation.
• We introduce a decoding dependency graph visualizer (DDGviz) to inspect the learned tokens’ conditional dependence and when parallel decoding is effective.
All the code is publicly released3.

heading: 2 Related Work
text: Gu et al. (2018) first introduced NonAutoregressive Translation models (NAT) as ad-hoc trained models capable of producing the translation all at once in parallel. With NATs, it is possible to consistently reduce the latency and speed up the translation at the expense of a slightly worse translation quality due to the multimodality problem (i.e., we lose the dependency between tokens in the target output). Finding a tradeoff between translation quality and speed is an active
3https://github.com/teelinsan/ parallel-decoding
research direction, with current methods trying to fill the gap in terms of translation quality (Geng et al., 2021; Savinov et al., 2022). Nevertheless, all proposed NAT models are learning-based and require different tricks to reach the quality of autoregressive models (Gu and Kong, 2021). The most common is the sequence-level knowledge distillation of large autoregressive models into parallel models (Kim and Rush, 2016). Other approaches include defining alternative training objectives (Ghazvininejad et al., 2020a; Saharia et al., 2020; Du et al., 2021; Huang et al., 2021), architectures that model dependencies between output sentence tokens (Ghazvininejad et al., 2019; Qian et al., 2021; Song et al., 2021a; Gu and Kong, 2021; Song et al., 2022) or multi-iteration methods (Ghazvininejad et al., 2020b; Kasai et al., 2020; Hao et al., 2021; Geng et al., 2021; Savinov et al., 2022; Huang et al., 2022; Xia et al., 2022) that apply iterative refinements to a translation, trading some speed for greater quality. In our approach, we also employ iterative refinements of solutions to non-linear equations, but we do not perform any training or modification to the model. Other works that require retraining or modifications to the model add additional decoding heads (Stern et al., 2018) or use shallow decoders (Kasai et al., 2021). We refer the reader to Xiao et al. (2022) for a thorough survey on NAT methods. Further orthogonal approaches use specialized hardware (TPU) with low-precision calculations (Wu et al., 2016) or software optimizations (Kim et al., 2019). In the context of Grammatical Error Correction, Sun et al. (2021) recently proposed aggressive parallel decoding, assuming that the model output is similar to the input. More recently, inspiring our work, Song et al. (2021b) showed that it is possible to parallelize feedforward computations
by thinking of them as a system of non-linear equations. They parallelized the backpropagation of RNNs, feedforward layers and autoregressive generative models on images. We extend the approach defined on dense pixel prediction to the discrete conditional token generation in MT. While this work was under submission and anonymity period, Leviathan et al. (2022), Chen et al. (2023) and Kim et al. (2023) concurrently proposed decoding approaches that speed up inference of a large transformer model by using another smaller model to draft tokens. Compared to these approaches our method requires just an existing autoregressive model (no matter the size) and mathematically guarantees the output quality. In the next Section we describe the method.

heading: 3 Method
text: In this Section, we introduce notations, develop the theory behind Parallel Decoding, present three algorithms (Fig. 2), and discuss the initialization and stopping conditions for the proposed approaches.

heading: 3.1 Notation
text: The goal of MT is to translate a sentence x in a source language (e.g., Italian) with its translation y in the target language (e.g., English). Source and target sentences are generally tokenized in words or subwords (Kudo and Richardson, 2018; Schuster and Nakajima, 2012; Sennrich et al., 2016; Kudo, 2018); here, we use the subfix notation x = (x1, . . . , xn) and y = (y1, . . . , ym) to indicate specific tokens in the sequence. We also use the notation x1:n to indicate a slice of a sequence as a shorthand of x = (x1, . . . , xn). From a probabilistic perspective, an MT model estimates pθ(y | x). Once an MT model has been trained, the inference phase is traditionally performed by sampling tokens from the model probability conditioned on the input sequence x and previously generated tokens (y1, . . . , yi−1):
pθ (yi | y1, . . . , yi−1,x) . (1) Different sampling strategies are employed (e.g., Greedy, Top-K, Top-p (Kool et al., 2020; Holtzman
et al., 2020)) alongside search strategies that estimate the total conditional probability (e.g., Greedy search, Beam search (Reddy, 1977)). The most straightforward strategy, Greedy Search, selects the element yi of a sequence with:
yi = argmax pθ(yi | y1:i−1,x). (2)
Given the formalization above, a standard autoregressive setting runs m inference steps sequentially to generate an output sequence of m elements.
Parallel Decoding. Given Equation (2), it is possible to write the greedy decoding procedure on all tokens as:
   y1 = argmax pθ(y1 | x) y2 = argmax pθ(y2 | y1,x) ... ym = argmax pθ(ym | y1:m−1,x)
(3)
Defining f(yi,y1:i−1,x) = yi − argmax pθ(yi | y1:i−1,x) , we can rewrite the system of Equations (3) as:
   f(y1,x) = 0 f(y2, y1,x) = 0 ... f(ym,y1:m−1,x) = 0
(4)
This system has m non-linear equations (each equation employ a neural network) with m variables.

heading: 3.2 Parallel Decoding Algorithms
text: The autoregressive decoding implicitly solves the system of Equations (4) by substitution, i.e., given the [BOS] token and the input sentence x, it solves equations from first to last, progressively replacing the resolved variables. In this paper, we rely on Jacobi and Gauss-Seidel (GS) fixed-point iteration methods (Ortega and Rheinboldt, 1970) to solve in parallel system (4) until a stopping condition is reached. This formulation is particularly flexible and has several advantages: Firstly, it is completely agnostic to the underlying MT model used; Secondly, it can be analyzed with analytical tools and
has guarantees of convergence to the exact solution for system (4); Thirdly, it can be potentially extended by drawing from the numerical methods literature for non-linear equations solving methods (Saad, 2003). We see that, with the proper stopping condition, it is possible to have quality guarantees over the output. We present here three algorithms (PJ, PGJ, HGJ) that leverage these fixed-point iteration methods to speedup decoding in MT.
Parallel Jacobi (PJ) Decoding. First, we propose Algorithm 1. This algorithm works by initializing a draft translation for the whole target sentence and then iteratively translating the whole sentence in parallel until the stopping condition is triggered. This is equivalent to solving system (4) with Jacobi, hence the name of the method.
Parallel GS-Jacobi (PGJ) Decoding. Decoding the whole target sentence in parallel may introduce difficulties in inferring long dependencies between tokens since the underlying model is trained to model the conditional distribution of a token given the previous tokens. In general, we observed that shorter dependencies are easily predicted since decoding happens at the sub-word level, and the model can decode sub-word unities in parallel rather than the whole sentence. To this end, we propose Algorithm 2, called GS-Jacobi, that splits the sentence into contiguous b-dimensional blocks. Starting from the first one, it decodes in parallel all its elements. Once a block is finished or the stopping condition within the block is triggered, the algorithm performs a sequential (Gauss-Seidel) step and proceeds with (Jacobi) decoding on the next one.
Hybrid GS-Jacobi (HGJ) Decoding. Algorithms 1 and 2 assume to know beforehand the number of equations m (i.e., the target length). This is not usually the case for MT, where the model dynamically controls the length through the emission of a special end-of-sentence token [EOS]. To overcome this issue, we propose a flexible Hybrid Algorithm 3 that mixes PGJ computations with standard autoregressive decoding. This algorithm performs parallel GS-Jacobi decoding up to a certain prefixed length h. If the [EOS] token is generated within a block, then the algorithm stops, returning the translation up to [EOS]. Otherwise, the algorithm concludes the translation by reaching the [EOS] token with standard autoregressive decoding. In this case, the length h regulates the trade-off between
Algorithm 1 Parallel Jacobi Decoding Input: x = (x1, . . . , xn), pθ Output: y = (y1, . . . , ym) 1: y← INITT(x) 2: m← len(y) 3: for i = 1 to m do 4: o← copy(y1:m) 5: y1:m ← argmax(pθ(y1:m|y1:m,x)) 6: stop← STOPC(o,y1:m) 7: if stop then 8: break 9: end if 10: end for 11: return y
parallel and sequential computation, limiting the waste of resources beyond [EOS].

heading: 3.3 Initialization and Stopping
text: Our algorithms share two components: the initialization procedure and the stopping condition.
Initialization INITT(x). The initialization procedure is a function that inputs the source sentence and produces an initial draft translation as output. In this paper we experimented with a simple initialization procedure that initialize the translation with all [PAD] tokens. This choice is fast and doesn’t depend on the underlying MT model. We leave as future work the research of different initialization procedures to further speedup the decoding.
Stopping Condition STOPC(yk−1, yk). The stopping condition is a function that takes as input the previous-iteration sentence yk−1 and the current-iteration sentence yk and decides whether to stop the algorithm or not. This function is crucial since it regulates the trade-off between speedup and translation quality. In this paper we introduce as stopping condition for MT:
yk−1 − yk = 0 (5)
i.e., the sentence from the previous step has not changed. This stop condition allows for preserving quality and quickening translations simultaneously.

heading: 3.4 Quality Guarantees
text: Compared to NAT methods which do not have any quality guarantee since a novel parallel model is trained from scratch, our formulation guarantees to have the same quality of using autoregressive decoding with the same MT model. System (4) is known in literature as a triangular system of m equations with m variables, this characterization allows to state an important property.
Proposition 1. Algorithms 1, 2, 3 converge and yield the same results of greedy autoregressive decoding in at most m parallel iterations, for any initialization and providing stopping condition (5).
We refer the reader to Song et al. (2021b) for a formal proof. Intuitively, with m steps the algorithm used the same number of iterations of autoregressive, hence the final solution is the same regardless the initialization. In this worst case, the wall-clock time is the same but in general the algorithm reach the stopping condition earlier with a lower wall-clock time and overall speedup.
3.5 DDGviz
Equation 1 models the dependency between tokens in the decoding phase. In the classical autoregressive mode, each token depends on all the previous ones for the generation. However, it is possible to show that this dependency is actually relaxed (i.e., not all tokens depends on all the previous ones), thus it would be interesting to visualize the actual distribution pθ (yi | ·,x) learned by an existing MT model. To this end, we build the Decoding Dependency Graph visualizer (DGGviz) to visualize the dependency graph of tokens in the decoding phase. In the standard autoregressive decoding this graph is a fully-connected chain where the i-th token is
connected to all the previous tokens, starting from the encoding x: to decode yi you need to decode first y1, . . . , yi−1. Instead we show that there are skipping connections between independent tokens that can be visualized with DGGviz. We detail DGGviz with an example in section 4.3.

heading: 4 Experiments

heading: 4.1 Experimental Settings
text: Datasets. We evaluate our approach using standard evaluation datasets proposed for parallel MT (Gu et al., 2018): WMT14 English-German [EnDe], WMT16 English-Romanian [En-Ro] (Bojar et al., 2014, 2016). Additionally, we tested our method on different language pairs with varying (low-medium) resources: IWSLT15 (EnglishVietnamese [En-Vi]) (Tran et al., 2015), IITB (English-Hindi [En-Hi]) (Kunchukuttan et al., 2018), WMT17 (English-Finnish [En-Fi]) (Bojar et al., 2017), FLORES-101 (English-Italian [En-It]; English-French [En-Fr]) (Goyal et al., 2022). All the datasets are evaluated in both directions.
Evaluation. All the evaluations are performed using the official test split for each dataset, downloaded using Huggingface dataset library (Lhoest et al., 2021). No training or hyperparameters tun-
ing is performed. We use SacreBLEU to evaluate the translation quality (Papineni et al., 2002; Post, 2018). We measure speedup in wall-clock time and iterations w.r.t. the same autoregressive model. GPU times are calculated after calling torch.cuda.synchronize(). All the experiments were performed by caching the past Keys and Values of the transformer to further speed up the computation (Ramachandran et al., 2017) and in the online inference setting with batch size equal to 1. For the Jacobi and GS-Jacobi algorithms, we assume to know beforehand the length m of the target and measure the speedup in the ideal condition. For the Hybrid GS-Jacobi algorithm, we set h equal to the maximum (i.e., the stopping condition is triggered within a parallel block) to decouple the effective speedup regardless of the length produced by the initialization function (see Section 3.2). We remark that HGJ does not assume to know beforehand the target length and is applicable to real MT translation scenarios.
Model Configuration. We tested transformer models in the two standard configurations: base (512 model dimension, 6 attention layers for both encoder and decoder) and big (1024 model dimension, 12 attention layers for both encoder and decoder). We used pretrained models of Opus (Tiedemann and Thottingal, 2020) for the former and MBart50 (Tang et al., 2020) for the latter. Opus is a transformer base model (74M parameters) trained on language pairs from the homonymous dataset (Zhang et al., 2020). MBart50 is a large multilingual transformer model fine-tuned for translation on 50 languages (610M parameters). We tested the models on CPU since this is the default environment for MT models in production, except for the model MBart50 which runs on GPU. We run the experiments on a standard 16-core machine, except for the scaling experiments. Additional specifications are available in Appendix B

heading: 4.2 Algorithms Comparison
text: In Table 1 we compare the proposed parallel decoding algorithms with the standard sequential autoregressive decoding baselines. As we can observe, the fastest algorithms are PGJ Decoding (b=3) and HGJ Decoding (b=3) which are up to 34% and 38% times faster on Opus and up to 5% and 8% faster on MBart50, depending on the language pair. We note also that results empirically show that all the parallel decoding algorithms guarantee the same quality of greedy autoregressive decoding, as evidenced by the unchanged BLEU scores. This is an experimental verification of the formal Proposition 1. The table also shows that the Beam Search algorithm with a beam size of 5 generally performs better in terms of BLEU score, although at a cost of speed. This difference in terms of BLEU is expected, as beam search is a heuristic search strategy, while our method is a decoding algorithm. We discussed better this aspect in the "Beam Search" paragraph. Nevertheless, beam search is ∼30% slower than greedy autoregressive and 63% to 68% slower than PGJ, depending on the model and language pair. This means that the proposed parallel algorithms allow trading a little translation quality (e.g., on en→ro the difference between beam search and parallel decoding algorithms in BLEU is just 0.20 points) for greater decoding speed.
Another aspect to note is that the algorithms PJ and PGJ (b=5) are sometimes slower than greedy autoregressive. There are several factors that can influence the actual wall-clock time like how the underlying hardware schedule and execute the various operations, which might vary according to the architecture and the workload. In particular, longer sequences (e.g., the whole sentence in PJ or blocks of 5 tokens in PGJ) may require more memory to store, and the CPU/GPU may have to perform more memory accesses, which can slow down the computation (although theoretically it should happen in parallel). In the end, these computational
overheads slow down the actual execution. This is also the case for the difference in speedups between MBart50 and Opus. We better investigated this aspect in the section "Computational Scaling" and report in the appendix results on a different architecture, with also results in terms of iterations speedups which are architecture agnostic.

heading: 4.3 Analysis and Validation
text: Cross Languages. In order to demonstrate the robustness of our decoding algorithms with respect to the translation languages, we leveraged the multilingual capabilities of the MBart50 model and selected a diverse range of language pairs for evaluation. The results, presented in Table 2, show that both PGJ and HGJ achieve a consistent speedup in comparison to the autoregressive decoding method, with an improvement ranging from 2-7% for PGJ and 1-5% for HGJ, regardless of the language pair used. Additionally, we observed a speedup in terms of iterations of 7-11% for PGJ and 5-7% for HGJ. These findings indicate that our algorithms have the potential to match or surpass the speedup in terms of wall-clock time by fully exploiting this saving in terms of iterations. We note that, similar to the previous experiment, PJ suffers from an overhead problem. To the best of our knowledge, this is one of the first studies that have achieved a speedup in multilingual machine translation, concurrent with the work of Song et al. (2022), while this latter is significantly different in spirit and requirements (NAT model). We leave BLEU scores in the Appendix D for space constraints together with qualitative results in different languages.
Computational Scaling. In Figure 3, we present an analysis of the scalability of our proposed methods in relation to increasing computational resources. Starting with 8 cores, our methods demonstrate a slight improvement in terms of wall-clock time for PGJ and HGJ, with speedups of 1.11 and 1.09 respectively. On the other hand, this amount of resources is too restricting for PJ which needs to fit the whole sentence and thus achieve a score of 0.46 due to the aforementioned overhead problem. As the resources are increased, our method demonstrates the ability to effectively leverage hardware and significantly reduce decoding time, while the autoregressive baseline is constrained by sequential processing. With 122 cores, a substantial speedup of 1.98× and 1.99× is achieved for PGJ and HGJ respectively, while the autoregressive baseline is
bounded by sequential processing at 1.00×. It is important to note that this experiment does not simulate a real production system, but rather it is meant to show what results can be achieved when the underlying computation is properly optimized to run in parallel. In our case, we simulated this setting with increasing cores, nevertheless similar results can be achieved with additional software optimizations to further reduce latency and overheads (Ahmed et al., 2022; Kim et al., 2019) and increase the speed gain with parallel-optimized computations. Overall this experiment serves as a proof of concept for the capabilities of parallel decoding in contexts with limited overhead and shows a promising direction for further improvements.
Comparison with NATs. Table 3 reports the comparison of our parallel decoding algorithm with a selection of NAT methods for parallel MT. Following prior works, we report for each method the speedup relative to the autoregressive transformer base baseline from their original paper (Xiao et al., 2022). It is worth noting that, although these methods can achieve higher speedups, they are very demanding in terms of computational resources which must be accounted for in a fair comparison. To estimate quantitatively this cost, we evaluated the number of floating point operations (FLOPs) required for training and inference on WMT14.
Results show that our method HGJ uses the least number of computational resources, even considering the additional cost at inference time. Relating the speedup obtained with the used resources (FLOPs/speed), our method still achieves the best
cost-benefit ratio. Furthermore, NATs generally degrade the translation quality if compared to their autoregressive baseline. On the contrary, our method mathematically guarantees the same quality of autoregressive decoding, which is higher than standard NAT models.
SUNDAE achieves BLEU of 28.46, but requires more resources than training RoBERTa (Liu et al., 2019) on 16 TPUs (see Appendix C). Other methods require further elaborate techniques like profound architectural changes, additional losses to force parallel translation and sequence-level distillation from large autoregressive transformers (Gu and Kong, 2021). Our approach is a decoding method that does not involve any training or modification to the model and can be used to speed up existing models on standard desktop hardware.
Speedup Analysis. We provide here a preliminary analysis of the factors responsible for the observed speedup in our method. We first distinguish between two types of speedup: wall-clock speedup and iterations speedup. The former is primarily driven by the parallelization capability of our method, as demonstrated in the "Computational Scaling" section. With parallel decoding, underlying operations can be optimized and fused to be executed fastly. Compared to Sheng et al. (2023), our method allows parallelizing sequence operations ("row-by-row" setting). The latter instead may vary consequently to several factors (e.g., model/vocabulary size, training data, language, etc). For this reason, we experimented with several variations of these factors (models Transformer Base vs. Big, vocabularies 58K Marian vs. 250K MBart50, languages, and hardware). While it is challenging to decouple different elements, our analysis point out several interesting insights. For example, we observed that iteration results on MBart50 are generally higher compared to Marian (Tables 2-6), possibly due to the finer-grained tokenization of MBart50. We also hypothesize that language and linguistic features, such as inflectionally rich or agglutinative/gendered languages, may influence iteration speedups. To facilitate this type of analysis, we developed DDGviz, which we believe will be useful for research in this area.
Visualizing Parallel Decoding. In previous experiments, we demonstrated that parallel decoding is feasible. This suggests that the dependency learned by the model between certain tokens is relaxed, as some tokens can be decoded in parallel.
versus women"→"Cât de satisfacuti sunt cuplurile romanes, ti: bărbat,ii împotriva femeilor". (Highlighted tokens decoded in parallel). On top: the Decoding Dependency Graph, omitting redundant edges on non-parallel tokens to ease visualization. On bottom: DDGviz shows at each Parallel Jacobi iteration (vertical axis) which tokens have been generated in parallel (horizontal axis) with the corresponding probability (cell number).
Analyzing and understanding when this happens allows shedding light on the behavior of existing models and a separate study focused on this issue would be needed. In this work, we lay the ground for a such study introducing the necessary inspection tools. While we have already introduced DDGviz in Section 3.5, in this experiment we show how it works and how it can be used with a practical example. In summary, the DDGviz visualizer allows to show the real decoding distribution pθ (yi | ·,x) learned by a MT model. This decoding distribution is plotted as a graph, where a connection indicates the dependency pθ(yi | ·), by using Parallel Jacobi decoding. At each PJ decoding iteration (vertical axis of Figure 4), DDGviz keeps track of which tokens have been correctly decoded w.r.t. the gold autoregressive reference of the model, showing the tokens correctly decoded and the probability of each one (horizontal axis). Figure 4 shows DDGviz applied on an example. The example shows that for y4 = _sa it is possible to decode more than one token in parallel y5 = tis, y6 = fa, hence here the decoding of y6 does not depend on the decoding of y5 - pθ (y6 | y1:4,x). We observed this phenomenon frequently, explaining the speedups in the previous experiments. The
example also shows that the model is able to decode five tokens in parallel after y7 = _cu. This is a peculiar case since the model, given "How satisfi_", is generating all at once "_ed are the Romanian couples" (proposed here in English for better readability, original version in Romanian is available in Figure). This example indeed shows how DDGviz can be used to highlight possible biases encoded in the model as it is not clear how the model can be so confident (see cell probability) that after "satisfied" the most straightforward tokens to decode are "Romanian couples" (Chang et al., 2019; Savoldi et al., 2021). We leave other use cases for future works and show in Appendix D several visualizations with equally interesting phenomena.

heading: 5 Conclusions
text: In this paper, we showed that is possible to speed up existing machine translation models by simply changing the decoding algorithm with a parallel formulation. We introduced three parallel decoding methods which achieve consistent speedups without requiring any training, modifications, or quality loss. Our solution is orthogonal to previous approaches proposed in literature which often entail demanding requirements in terms of data, computational resources, and engineering effort. Although our method is not without shortcomings, it is a first valuable step toward integrating parallel decoding algorithms into any model. This is particularly relevant in limited-resource scenarios where NATs are not a viable option and to speed up any transformer model, especially fine-grained or character-level models (Edman et al., 2023). We believe that further advancements in this area, including the exploration of optimal initialization procedures and stopping conditions, as well as the use of alternative parallel solvers for non-linear equations, will close the gap with learning-based techniques and continue to improve the efficiency and effectiveness of parallel decoding algorithms.

heading: Acknowledgements
text: We would like to thank Sébastien Bratières for his throughout feedback provided on this project. This work is supported by Translated with an Imminent Research Grant, ERC Starting Grant No. 802554 (SPECGEO), and PRIN 2020 project n.2020TA3K9N "LEGO.AI". Riccardo Marin is also supported by an Alexander von Humboldt Foundation Research Fellowship.
Limitations
The proposed algorithms allow to speed up an existing model out-of-the-box, without any modification or retraining. However, there are some considerations to bear in mind when using parallel decoding in order to have a speedup in terms of wall-clock time. Firstly, as the name implies, the method executes the decoding phase in parallel. Therefore, to appreciate the speedup one should be able to run computations in parallel. Using parallel decoding without parallel resources or parallel-optimized software may increase wall-clock time due to overheads, leading to a waste of computation. This is further discussed in Section 4.3 "Computational Scaling". The reported wall-clock time results are thus to be considered within the scope of the experimental setup proposed in this paper and they may vary depending on the underlying hardware and software. Secondly, the method allows speedup of the decoding by scaling on parallel resources. This implies an additional computational cost during the inference phase to achieve a speedup. While using parallel decoding, one should consider a trade-off between the desired acceleration and the utilization of computational resources. Thirdly, since our method performs the decoding in parallel, as for NAT systems, it is difficult to combine it with Beam Search. Beam Search is inherently a dynamic programming algorithm and it is not possible to efficiently maximize the joint probability of the large search space without using sequential intermediate computations. We better explain this aspect in the next paragraph.
Beam Search. Beam search is widely employed to enhance the translation quality in MT (Sutskever et al., 2014; Bahdanau et al., 2015) as well as in other domains such as audio (Reddy, 1977; Postolache et al., 2023). However, it is an inherently sequential procedure that stores partial joint probabilities of the entire sequence (beams) while progressing with autoregressive decoding. Determining the maximal joint probability of all sequences in parallel is a challenging task, equivalent to a full maximum a posteriori (MAP) estimation. This is an open research problem and it is also an issue for NAT methods. NAT methods patch up this limitation with sequence-level KD which has the advantage of "not requiring any beam search at test-time" (Kim and Rush, 2016) thanks to learning and distillation from large models. Since our
method is a decoding algorithm, we cannot use the same approach without learning. Nevertheless, the quality guarantee allows our methods to have performance on par with greedy autoregressive and generally better than a NAT model. We think of our method, not as a replacement for beam search, but rather as a way to obtain a speedup at inference time that is a middle ground between autoregressive greedy decoding (high quality, no requirements, no speed) and NATs (quality compromises, increasing requirements with increasing speed). Future works might address the quality gap with beam search by combining parallel decoding with alternative techniques like Minimum Bayes Risk (Eikema and Aziz, 2020).
Ethics Statement
Increasing the inference speed of MT can positively impact society by giving people a fast and good translation. This will enable people from different language backgrounds to communicate with each other and help remove cultural and trade barriers. As demonstrated by comparing the number of FLOPs in Table 3, our method uses fewer resources compared to alternatives and thus has a smaller carbon footprint, making it a more sustainable choice (Strubell et al., 2019). Furthermore, since our method does not involve training procedures or change the quality of results, we do not introduce any societal bias (e.g. racism, sexism, homophobia) into the translations. The latter, however, can be introduced through data in the training of the backbone autoregressive models and NATs. It is the task of those who train these models to mitigate this problem. DDGviz can also help investigate and visualize some potential harmful biases encoded in the model like in Figure 4.

heading: A Algorithms details
text: We propose here the pseudocode of Algorithms 2 and 3 due to space limitations in the main body of the paper.
The function copy(yi:i+b) creates a copy of the tensor in input detached from the source. This is done in practice to avoid the overwriting of pointers to the same memory location. Function CHECKEOS(yi:i+b) returns the index of the token EOS in the block if present, else −1. Function CHECKEOS(yi) returns True if the tokes in exactly the token EOS, else False. The function argmax selects from the model distribution over the vocabulary the index (token) with maximum probability. This procedure is done for all the tokens in parallel, in the case of parallel decoding, or for just a single token in the case of autoregressive decoding. Generally, the output is the prediction for the next token; hence it should be shifted left before the reassignment to a variable. We omitted this implementation detail for clarity.

heading: B Additional implementation details
text: We run Opus experiments in table 1 on an AMD EPYC Milan with 16 cores at 2.45 GHz and 64GB of RAM (accessible on Google Cloud - c2d-standard-16). For the scalability experiment in figure 3, we also used Google Cloud instances with an increasing number of cores (referred to as c2d-standard-XX, where XX is the number of used cores). Experiments with MBart50 on table 1, 2 and 6 are performed on a Desktop machine with Ubuntu 20.04.4 LTS, AMD
Ryzen 9 3900X 12-Core Processor, 32GB of RAM, and a Palit Nvidia 3090 GPU. Additional experiments with Opus in table 6 are also performed on this machine. Models are implemented in Pytorch 1.11.0 (Paszke et al., 2019) and the Huggingface Transformer library (Wolf et al., 2020). We used python 3.8 and NVIDIA-SMI Drivers 510.73.05 with CUDA version 11.6. For OPUS we used Huggingface models available on the hub under the tag Helsinki-NLP/opus-mt-{src}-{tgt} except for the language pair RoEn where we used the model Helsinki-NLP/opus-mt-roa-en and the pair En-De where we used the checkpoint opus-2021-02-22 4. For the model MBart50, we used the facebook pre-trained model available on the hub with the tag mbart-large-50-many-to-many-mmt. Since this is a multilingual model, we prepend the source and target language tag corresponding properly to the language pair to be translated. We report results for a single run over the test dataset since we found low variance in estimates with multiple runs which can be calculated by simply varying the corresponding parameter in the config.yaml file. For each dataset, we used the official test split via the Huggingface dataset library (Lhoest et al., 2021). Datasets statistics are reported in table 4.

heading: C FLOPs calculation details
text: We measured computational complexity using floating point operations (FLOPs), which, as the name imply, counts the number of floating point operation performed by a model. This is a standard metric used in literature to measure hardware-agnostic complexity. This means that hardware and software optimizations are not counted in the score (Wu et al., 2016; Kim et al., 2019). We used the
4https://object.pouta.csc.fi/Tatoeba-MT-models/engdeu/opus-2021-02-22.zip
Algorithm 3 Hybrid GS-Jacobi Decoding Input: x = (x1, . . . , xn), pθ, b Output: y = (y1, . . . , ym)
1: y← INITT(x) 2: h← len(y) 3: i← 1 4: eos_cond← False 5: while i ⩽ h do 6: o← copy(yi:i+b) 7: yi:i+b ← argmax(pθ(yi:i+b|y1:i+b,x)) 8: stop← STOPC(o,yi:i+b) 9: eos_ind← CHECKEOS(yi:i+b)
10: if stop and eos_ind > −1 then 11: y← y1:eos_ind 12: eos_cond← True 13: break 14: end if 15: if stop then 16: i← i+ b 17: break 18: end if 19: end while 20: while eos_cond ! = True do 21: yi ← argmax(pθ(yi|yi−1,x)) 22: i← i+ 1 23: eos_cond← ISEOS(yi) 24: end while 25: return y
ELECTRA flops calculator5 inserting the number of parameters and the number of training step performed for each model analyzed in table 3 according to the training specification in each paper. For inference FLOPs, we computed the decoding cost of each sentence in the testset of WMT14 En-De for each model. For a scale reference, we report in here Table 5 training flops of other well-known architecture. The code package contains the scripts to replicate all the experiments.

heading: D Additional results
text: We propose here additional results to the experiments in the paper that were omitted due to limitations constraints. Table 6 shows the same experiments of Table 1 in the main paper, proposed here on a standard desktop CPU with also the speedup in terms of iterations. It is possible to observe that in the case of MBart50 and PGJ there is a speedup
5https://github.com/googleresearch/electra/blob/master/flops_computation.py
of 8 − 11% in terms of iterations compare to a time speedup of 3 − 8%. This means that there is room for improvement for our algorithm. Furthermore, results show that the time speedups are consistent also with standard desktop hardware. Table 7 shows the BLEU scores for the cross-lingual experiment. It is possible to observe that parallel decoding algorithms guarantee quality compared to greedy autoregressive and are not so distant from beam search. We show also here in table 5 some qualitative results for the experiments in table 2. Finally, we propose additional visualizations using DGGviz in Figure 6.
WMT17 IITB IWSLT15 FLORES En-Fi En-Hi En-Vi En-It En-Fr
Dec. Algorithm ← → ← → ← → ← → ← → Autoregressive 17.55 25.34 16.50 24.70 31.92 33.94 22.78 26.38 39.51 38.90 Beam Search 18.39 26.04 16.87 25.24 32.14 34.59 23.52 26.80 39.59 39.21 PJ 17.54 25.35 16.50 24.69 31.92 33.94 22.78 26.38 39.50 38.90 PGJ (b=3) 17.55 25.35 16.50 24.70 31.93 33.94 22.78 26.38 39.51 38.90 HGJ (b=3) 17.55 25.35 16.50 24.70 31.93 33.94 22.78 26.38 39.51 38.90
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations section

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethics Statements

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract

heading: 3 A4. Have you used AI writing assistants when working on this paper?
text: We used ChatGPT to rephrase some sentences in the final camera-ready version in sections 4.3 and 5.
B 3 Did you use or create scientific artifacts? Code to reproduce the experiments (zip)

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4 and Appendix B

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: License file in the code repository

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? License file in the code repository
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. No data was collected
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Data is automatically downloaded with standard train/test/dev splits via the Huggingface datasets library. Additional statistics in Appendix B
C 3 Did you run computational experiments? Section 4 - Experiments

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4.1
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4.1

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4.1 and Appendix B
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Future works might address the quality gap with beam search by combining parallel decoding with alternative techniques like Minimum Bayes Risk (Eikema and Aziz, 2020)