SlowGuess/NAACL_2022_OCR huggingface dataset:
We observe a significant contribution to Hourglass' performance with increasing the number of vanilla layers. One reason is that we perform more computations as in vanilla layers we process the sequence in token-level - no shortening is applied. We also see that the distribution of vanilla layers before shortening and after shortening does impact the training (see Tab. 6), and equal distribution leads to the best perplexity.
Table 5: Impact of increasing the number of shortened layers on perplexity. Vanilla layers: (1,1) for CIFAR-10 and (2,2) for enwik8, shorten factor 3 used in both.
Vanilla layers	enwik8	CIFAR-10
(0, 0)	1.460	3.429
(0, 2)	1.176	3.108
(2, 0)	1.189	3.035
(1, 1)	1.171	3.012
(2, 2)	1.128	2.966
Table 6: Impact of the distribution of vanilla layers on enwik8 (BPC) and CIFAR-10 score (BPD). We see that equal distribution of layers before and after shortening leads to better results on both datasets.
4.4 Upsampling method
In Table 7 we investigate different possibilities of choosing the upsampling method. For attention-free methods, linear upsampling performs better on images, while repeat upsampling works well for text. Attention upsampling works well regardless of the function  $U$  and has the lowest perplexity.
Upsampling method	enwik8	CIFAR-10
Repeat	1.148	3.062
Linear	1.163	3.020
U(x,x') = x	1.145	2.967
U(x,x') = x + Linear(x')	1.132	3.012
4.5 Pooling method
Table 8 presents impact of pooling method on both enwik8 (BPC) and CIFAR-10 (BPD). Attention pooling reaches the lowest perplexity for both datasets. Average pooling performs well on text among attention-free methods, while linear pooling works better for images. Both of these methods perform significantly worse for the other modality. Attention pooling demonstrates small differences with respect to chosen shortening function  $S$  (Section 2.1), still preserving the preference towards linear pooling on images and average pooling on text.
Table 7: Upsampling method ablation - baseline configurations are (2@1, 24@4, 2@1) and (1@1, 8@3, 1@1) for enwik8 and CIFAR-10, respectively.
Pooling method	enwik8	CIFAR-10
AvgPool	1.129	3.116
Attention, S = AvgPool	1.124	3.012
Attention, S = LinearPool	1.142	2.998
LinearPool	1.159	2.998
Table 8: Ablation of pooling methods. Attention pooling achieves the best perplexity on both datasets.
4.6 Shortening strategies
While the analysis above gives a clear indication of what methods to choose for shortening and upsampling, we are still left with the question of which shorten factors to use and whether to do single-stage or multi-stage shortening.
Consistently, it is beneficial to do at least one shortening and by a factor of at least 3, while keeping 2-3 vanilla layers. Beyond that, a number of different configurations can yield similar results. In Table 1 we present the different hierarchical configurations that we tested on enwik8 and plotted in Figure 1. It can be seen that configurations with similar computation costs perform similarly. The sequence length used in these experiments is 2048 – we hypothesise that more hierarchy may be beneficial with even longer sequences.
Shortening in Transformers Shortening in our work is inspired by Funnel-Transformer (Dai et al., 2020). The key difference is that they train an encoder model for text classification, where our work is entirely focused on language modeling, which provides additional challenges we had to solve regarding shortening in the autoregressive setup (Section 2.2). Another difference is that they use repeat upsampling method while we use attention. There are also a few works related to character-level modeling which use shortening, namely (Clark et al., 2021) and (Tay et al., 2021). However, the authors of these works focused mainly on shortening sequence in encoder part of the transformer, whereas we focused on applying shortening in decoder.
The idea of shortening is also discussed in (Subramanian et al., 2020). However, proposed architectures either focus on downsampling or upsampling, while Hourglass is a U-Net-like architecture and is symmetric in these terms. Their models use transformer layers on the finest scales when post-processing final representations. We do these also, in the beginning, to preprocess tokens on the finest scale, and we have found it essential to the score (Section 4.3). Our attention upsampling method is similar to their aggregation layer in the bottom-up model, however we use one upsampling for each scale change while they combine different scales to create one global upsampling.
Relative positional encoding Our work is primarily built on the backbone of Transformer-XL (Dai et al., 2019) - we use the same relative attention parametrization. Instead of the segment-level recurrence mechanism, we use shortening to make our model more efficient and feasible to train on longer sequences. Another relative attention parametrization is RoFormer (Su et al., 2021) where rotary positional embeddings are introduced.
We find this work particularly relevant because the method is compatible with any attention type, including efficient attention, and can be combined with our model (Section 3.4.2).
Sparse Attention A well-known approach addressing the memory bottleneck is utilizing sparsity patterns in the attention matrix - Routing (Roy et al., 2020) and Sparse Transformer (Child et al., 2019) are examples of such methods. Our solution is different in the sense that it uses full attention - just with shortened sequence length. Combiner (Ren et al., 2021) makes a step further and provides full attention capabilities with similar computational complexity to Routing and Sparse transformers by leveraging structured factorization. This work, similarly to papers mentioned above on efficient transformers, concentrates on speeding up the attention component, while the most important feature of the Hourglass architecture is that it can use any attention module as a drop-in.
Image generation on downsampled ImageNet VDM (Kingma et al., 2021) and DenseFlow (Grcic et al., 2021) are recently proposed state-of-the-art methods for density estimation on this dataset. The difference between these methods and Transformer-based methods (Parmar et al., 2018; Ho et al., 2019) including this work is that the former, unlike Transformers, are non-autoregressive.
In this paper, we show how hierarchy can improve the efficiency of Transformers in a language modeling setup. Our proposed architecture, Hourglass, significantly outperforms the baseline both in terms of perplexity reached at a given computation cost (Figure 1) and empirical metrics like running memory (Figure 6). Hourglass achieves state-of-the-art results among autoregressive models on the ImageNet32 generation task and competitive results on other image generation and language modeling tasks.
Hourglass can be used with any attention type, which opens many directions for future research related to Transformers capable of processing longer sequences. Another line of future work might be related to advances in the shortening mechanism itself, for example, involving a dynamic pooling operation that could explicitly handle the problem of fixed-size groups in multi-stage shortening. We also leave open the problem of choosing the best hi
erarchy for a task. We conjecture that experiments with much longer contexts will provide better guidance for this choice and will benefit even more from the Hourglass architecture.
Some experiments were performed using the Entropy cluster funded by NVIDIA, Intel, the Polish National Science Center grant UMO-2017/26/E/ST6/00622, and ERC Starting Grant TOTAL. The work of Henryk Michalewski was supported by the Polish National Science Center grant UMO-2018/29/B/ST6/02959. The authors would like to thank Marek Cygan and Kamil Wilczek for their help with cluster setup, and Grzegorz Grudzinski, Dawid Jamka and Sebastian Jaszczur for helpful discussions. This article describes a Team Programming Project completed at the University of Warsaw in the academic year 20/21. We are grateful to Janusz Jabłonowski, the head of Team Programming Projects, for his support and open-mindedness.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. 2020. Generative pretraining from pixels. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1691-1703. PMLR.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen
Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating large language models trained on code.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. 2021. Rethinking attention with performers.
Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. 2021. Canine: Pre-training an efficient tokenization-free encoder for language representation.
Zihang Dai, Guokun Lai, Yiming Yang, and Quoc V. Le. 2020. Funnel-transformer: Filtering out sequential redundancy for efficient language processing.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding.
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. 2020. Jukebox: A generative model for music.
Angela Fan, Thibaut Lavril, Edouard Grave, Armand Joulin, and Sainbayar Sukhbaatar. 2021. Addressing some limitations of transformers with feedback memory.
Matej Grcic, Ivan Grubišić, and Siniša Šegvic. 2021. Densely connected normalizing flows.
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. 2019. Axial attention in multidimensional transformers.
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M. Dai, Matthew D. Hoffman, Monica Dinulescu, and Douglas Eck. 2018. Music transformer.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020. Transformers are rnns: Fast autoregressive transformers with linear attention.
Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. 2021. Variational diffusion models.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer.
Alex Krizhevsky. 2009. Learning multiple layers of features from tiny images.
Matt Mahoney. 2011. Large text compression benchmark.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. 2018. Image transformer.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-shot text-to-image generation.
Hongyu Ren, Hanjun Dai, Zihang Dai, Mengjiao Yang, Jure Leskovec, Dale Schuurmans, and Bo Dai. 2021. Combiner: Full attention transformer with sparse computation cost.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation.
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2020. Efficient content-based sparse attention with routing transformers.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. 2021. Roformer: Enhanced transformer with rotary position embedding.
Sandeep Subramanian, Ronan Collobert, Marc'Aurelio Ranzato, and Y-Lan Boureau. 2020. Multi-scale transformer language models.
Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. 2019. Adaptive attention span in transformers.
Sainbayar Sukhbaatar, Da Ju, Spencer Poff, Stephen Roller, Arthur Szlam, Jason Weston, and Angela Fan. 2021. Not all memories are created equal: Learning to forget by expiring.
Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. 2021. Charformer: Fast character transformers via gradient-based subword tokenization.
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. 2016a. Pixel recurrent neural networks. CoRR, abs/1601.06759.
Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. 2016b. Conditional image generation with pixelCNN decoders.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.
Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. 2021. Long-short transformer: Efficient transformers for language and vision.
A Autoregressive shortening
In Section 2.2 we address two problems of shortening in an autoregressive setup: information leaks and reduced expressivity. Here we study these issues in more detail.
A.1 Motivation behind using vanilla layers
At first sight, it may be tempting to create hierarchical models that directly shorten the input to maximize the efficiency gains. In this section, we explain why vanilla layers are crucial for modeling at least some sequences, especially due to autoregressivity.
Consider a sequence modeling task where the input is a random sequence with repeats, such as A#AC#CD#DB#B. The sequence consists of chunks L#L where L is a random uniform letter and # is a special symbol. A vanilla Transformer language model can achieve  $66\%$  sequence accuracy on this task - it cannot predict the token at the beginning of the chunk, but it can predict the last token of the chunk by simply copying the token at 2 positions earlier, which is possible using a vanilla self-attention layer.
It is however not easy to learn this task in a shortening setup when there are no vanilla layers operating on the finest scale - this is the situation defined in Reduced expressivity subsection of Section 2.2. Assume shorten factor is  $k = 3$  and the input is A#AB#BC#C. To avoid information leak, we shift the input sequence right by 1, and then by  $k - 1 = 2$  directly before shortening. Then the sequence is 000A#AB#B. Our shortened embeddings are as follows:  $e_0 = S(emb_0, emb_0, emb_0), e_1 = S(emb_A, emb_\#, emb_A)$  where  $emb$  is input embedding matrix and  $S$  is a shortening function.
Shortened embeddings	[000]	[A#A]	[B#B]
Shifted input embeddings	0A#	AB#	BC#
Target sequence	A#A	B#B	C#C
Positions	123	456	789
Table 9: Example input sequence which is difficult to model without vanilla layers. The model can use only input embeddings shifted by one from the residual and shortened embeddings (shorten factor is 3) to predict the target sequence. Note that it is impossible to predict tokens at positions divisible by 3 using only that information.
Because no vanilla layers are used, for prediction we can use only shortened embeddings and input token embeddings shifted right by 1 from
the residual connection. Note that to predict the A token at position 3 we can use only embedding of  $emb_{\#}$  and  $e_0$  - both of these contain no information so we are unable to predict this token better than randomly (see Table 9). An analogous situation occurs for prediction of any tokens at positions divisible by 3, which makes the model unable to achieve more than  $50\%$  accuracy when the task has vocabulary size of at least 2.
This issue can be solved by adding at least one vanilla layer to the model, so that it can attend within the neighborhood of  $k$  previous tokens. For this particular problem, it is sufficient to use local attention with context size  $k$  in vanilla layers which is significantly more efficient than full attention.
A.2 Information leaks - analysis
A.2.1 Definition of autoregressive model
Formally, given a target sequence,  $x = x_{1},\ldots ,x_{n}$  an autoregressive model (e.g. transformer decoder) models the sequence as  $P(x) = \prod_{i = 1}^{n}P(x_i|x_1,\dots,x_{i - 1})$  and
$$
\forall_ {i} P (x _ {i} | x _ {1}, \dots , x _ {n}) = P (x _ {i} | x _ {1}, \dots , x _ {i - 1})
$$
namely  $x_{i}$  token depends only on previous tokens, never on itself nor next ones.
A.2.2 Definition of information leak
We say that a leak was caused by function  $F_{n} \colon A^{n} \longrightarrow A^{n}$  transforming sequence of input tokens  $(x_{1}, x_{2}, \ldots, x_{n})$  into another sequence  $(u_{1}, \ldots, u_{n}) = F((x_{1}, \ldots, x_{n}))$  when  $\exists_{i < j < n} P(x_{i} | x_{1}, \ldots, x_{i-1}, x_{j}) \neq P(x_{i} | x_{1}, \ldots, x_{i-1})$ , namely there exists  $j \geq i$  that token  $x_{i}$  depends on  $x_{j}$  which violates the autoregressive property.
A.2.3 Model representation
Let  $R_k \colon A^n \longrightarrow A^n$  be a shift right function which reindexes tokens by shifting each of them right by  $k$  positions:
$$
R _ {k} ((x _ {1}, x _ {2}, \ldots x _ {n})) = (\underbrace {0 , \ldots , 0} _ {k}, x _ {1}, \ldots , x _ {n - k})
$$
$S_{k}:A^{*}\longrightarrow A^{*}$  shortening function with factor  $k$  which takes on input  $x_{1},\ldots ,x_{n}$  sequence and returns  $s_1,\ldots ,s_m$  where  $m = \frac{n}{k}$ $U_{k}$  upsampling function which works in similar way but upsamples  $U_{k}((u_{1},\dots,u_{m})) = u_{1},\dots,u_{n}$
Between them there is also applied  $D$  decoder function,  $D = D_{1}\circ \dots \circ D_{l}$ , where each  $D_{i}$  is a
function representing decoder block. Due to causal attention masking in the decoder block, there is no risk of information leak caused by function  $D$ .
A.2.4 Leak description
Because of mentioned attention mask, we will omit the flow of information between tokens caused by the influence of attention mechanism because this mask keeps the autoregressive property.
Now, let  $(x_{1},\ldots ,x_{n})$  be an input sequence and  $(u_{1},\dots,u_{n}) = U(D(S_{k}(T_{s}((x_{1},\dots,x_{n})))) = F.$  In order to preserve autoregressive property, it is obligatory that no leak occurs.
We will show that shift by any value  $0 < s < k - 1$  where  $k$  is the shorten factor will cause a leak.
To start with, consider input sequence  $(x_{1},\dots,x_{n})$  and perform operation  $F$ .  $R_{s}((x_{1},\dots,x_{n})) = (0,\dots,0,x_{1},\dots,x_{n - s}) = r$ . Assuming that  $n$  is divisible by  $s$ , we have  $S_{k}(r) = (v_{1},\dots,v_{\frac{n}{k}}) = v$  where each  $v_{i}$  consists of information obtained in  $(r_{(i - 1)\cdot k + 1},\dots,r_{ik})$ . Now let see that operation  $D$  preserves autoregressive property, let  $d = D(t)$ . Now,  $U(d) = (u_{1},\dots,u_{n})$  and each  $u_{i}$  depends on  $d\left\lfloor \frac{i - 1}{k}\right\rfloor +1$ .
Now consider  $s \leq k - 2$  and let  $(u_{1},\dots,u_{n}) = F((x_{1},\dots,x_{n}))$  will be a result of our Transformer part. Let take  $u_{1}$  which depends on  $d_{1}$  and  $d_{1}$  depends on  $(r_1,\ldots ,r_k) = (0,\ldots ,0,x_1,\ldots ,x_{k - s})$ . For that reason  $d_{1}$  depends on  $x_{1},x_{2},\ldots ,x_{k - s}$ , so we have
$$
P \left(x _ {1} \mid x _ {k - s}\right) \neq P \left(x _ {1}\right)
$$
which violates the autoregressive property.
B Experimental setup
B.1 Common parameters
Here we list common hyperparameters used for all experiments mentioned in the paper. We use Adam optimizer with  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.98$  and  $\epsilon = 1\mathrm{e} - 9$ . Weight decay and gradient clipping is not used.
In terms of model details, we decided to use a Pre-Norm architecture and FastGelu activation in feed-forward layers.
B.2 Enwik8
We use  $d_{model} = 512$ ,  $d_{ff} = 2048$  and 8 attention heads. Models in ablation study are trained for 200k steps with cosine learning rate schedule, setting cycle length for 200k steps and linear warmup of 4000 steps.
For the main result achieving 0.98 bpc with  $4@1,8@3,4@1$  hierarchy, we set  $d_{model} = 768$ ,  $d_{ff} = 3072$  and  $n_{heads} = 8$  which results in 146M parameters. It is trained for a total number of 350k steps with one cycle of cosine schedule. Linear warmup of 20k steps is used.
At the beginning of our work on this paper, we have performed grid search over following hyperparameters for enwik8:
- batch size:  $\{8, 16, 32\}$ , finally chosen 8
- dropout:  $\{0.05, 0.1, 0.15, 0.20\}$ , finally chosen 0.15
learning rate:  $\{1\mathrm{e} - 4,2\mathrm{e} - 4,3\mathrm{e} - 4,4\mathrm{e} - 4,5\mathrm{e} - 4\}$  finally chosen  $4\mathrm{e} - 4$
All next experiments were conducted using these parameters without additional searching.
B.3 Downsampled ImageNet - common parameters
For ImageNet32 and ImageNet64 experiments we use inverse square root learning rate decay from (Vaswani et al., 2017), setting warmup steps to 8000 in both experiments. Total batch size is 64.
B.4 ImageNet32
In this dataset, we operate on input sequence length of 3072. We use  $d_{\mathrm{model}} = 512$ ,  $d_{\mathrm{ff}} = 2048$ , 8 attention heads and 0.01 dropout rate. We perform 400k training steps with linear warmup and inverse square root decay and then we train for additional 70k steps with cosine learning rate decay, starting from the learning rate from the previous schedule at 400k and decreasing it to 0 at 470k steps.
B.5 ImageNet64
As an input we receive a sequence of 12288 tokens representing  $64 \times 64 \times 3$  images. We set  $d_{\mathrm{model}} = 768$ ,  $d_{\mathrm{ff}} = 3072$ , 8 attention heads and dropout equal to 0. We perform 300k steps with linear warmup and inverse square root decay.
B.6 CIFAR-10
All the ablation studies are run for 100k training steps, unless otherwise specified. Input sequence has length 3072 and model parameters are as follows:  $d_{\mathrm{model}} = 512$ ,  $d_{\mathrm{ff}} = 2048$ , 8 attention heads and dropout equal to 0. Total batch size is 8. Cosine learning rate decay with linear warmup of 5000 steps and 100k cycle length is used.
C Environment setup
Experiments are conducted on several setups.
- Ablation Study and short training sessions were computed on nodes consisting of 4x Titan V with 12GB memory each, 64GB RAM, Intel Xeon E5-2660 v4 CPU
- longer trainings were completed on 8x RTX 2080 Ti with 11GB memory each, 128GB RAM and Intel Xeon E5-2660 v4 CPU.
- Few longest trainings were conducted on  $8 \times 8$  TPUv3 units, each with 16GB memory.
C.2 Software