[1] R. Y. Aminabadi et al., "Deepspeed inference: Enabling efficient inference of transformer models at unprecedented scale," arXiv preprint arXiv:2207.00032, 2022.
[2] T. Brown et al., "Language models are few-shot learners," Advances in neural information processing systems, vol. 33, pp. 1877-1901, 2020.
[3] Z. Cheng, J. Kasai, and T. Yu, "Batch prompting: Efficient inference with large language model apis," arXiv preprint arXiv:2301.08721, 2023.
[4] W.-L. Chiang et al., Vicuna: An open-source chatbot impressing gpt-4 with  $90\%$  * chatgpt quality, Mar. 2023. [Online]. Available: https://lmsys.org/blog/2023-03-30-vicuna/.
[5] J. Choi, H. Li, B. Kim, S. Hwang, and J. H. Ahn, "Accelerating transformer networks through recomposing softmax layers," in 2022 IEEE International Symposium on Workload Characterization (IISWC), IEEE, 2022, pp. 92-103.
[6] A. Chowdhery et al., “Palm: Scaling language modeling with pathways,” arXiv preprint arXiv:2204.02311, 2022.
[7] T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré, "Flashattention: Fast and memory-efficient exact attention with io-awareness," Advances in Neural Information Processing Systems, vol. 35, pp. 16344-16359, 2022.
[8] T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer, "Llm.int8(): 8-bit matrix multiplication for transformers at scale," arXiv preprint arXiv:2208.07339, 2022.
[9] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
[10] J. Fang, Y. Yu, C. Zhao, and J. Zhou, “Turbot Transformers: An efficientgpu serving system for transformer models,” in Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, 2021, pp. 389–402.
[11] E. Frantar, S. Ashkboos, T. Hoefer, and D. Alistarh, “GPTQ: Accurate post-training compression for generative pretrained transformers,” arXiv preprint arXiv:2210.17323, 2022.
[12] M. Ghazvininejad, O. Levy, Y. Liu, and L. Zettlemoyer, "Mask-predict: Parallel decoding of conditional masked language models," arXiv preprint arXiv:1904.09324, 2019.
[13] J. Gu, J. Bradbury, C. Xiong, V. O. Li, and R. Socher, “Non-autoregressive neural machine translation,” arXiv preprint arXiv:1711.02281, 2017.
[14] J. Gu and X. Tan, “Non-autoregressive sequence generation,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, 2022, pp. 21–27.
[15] J. Hoffmann et al., “Training compute-optimal large language models,” arXiv preprint arXiv:2203.15556, 2022.
[16] E. J. Hu et al., “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685, 2021.
[17] Y. Huang et al., "Gpipe: Efficient training of giant neural networks using pipeline parallelism," Advances in neural information processing systems, vol. 32, 2019.
[18] S. Kim et al., “Full stack optimization of transformer inference: A survey,” arXiv preprint arXiv:2302.14017, 2023.
[19] A. Koubaa, “Gpt-4 vs. gpt-3.5: A concise showdown,” 2023.
[20] J. Lee, E. Mansimov, and K. Cho, "Deterministic non-autoregressive neural sequence modeling by iterative refinement," arXiv preprint arXiv:1802.06901, 2018.
[21] J. Lee, R. Shu, and K. Cho, "Iterative refinement in the continuous space for non-autoregressive neural machine translation," arXiv preprint arXiv:2009.07177, 2020.