Small-scale proxies for large-scale Transformer training instabilities LLM AI(7)

LLM(Large Language Model) Advent Calendar 2024
https://qiita.com/advent-calendar/2024/llm
2日目投稿予定の記事です。

Small-scale proxies for large-scale Transformer training instabilities
Mitchell Wortsman, Peter J. Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D. Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-dickstein, Kelvin Xu, Jaehoon Lee, Justin Gilmer, Simon Kornblith
https://arxiv.org/abs/2309.14322v2

<この項は書きかけです。順次追記します。>
This article is not completed. I will add some words and/or centences in order.
### References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
[2] Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, Antoine Dedieu, Claudio Fantacci, Jonathan God- win, Chris Jones, Ross Hemsley, Tom Hennigan, Matteo Hessel, Shaobo Hou, Steven Kapturowski, Thomas Keck, Iurii Kemaev, Michael King, Markus Kunesch, Lena Martens, Hamza Merzic, Vladimir Mikulik, Tamara Norman, George Papamakarios, John Quan, Roman Ring, Francisco Ruiz, Alvaro Sanchez, Laurent Sartran, Rosalia Schneider, Eren Sezener, Stephen Spencer, Srivatsan Srinivasan, Miloˇs Stanojevi ́c, Wojciech Stokowiec, Luyu Wang, Guangyao Zhou, and Fabio Viola. The DeepMind JAX Ecosystem, 2020. URL http://github.com/ deepmind.
[3] Blake Bordelon and Cengiz Pehlevan. Dynamics of finite width kernel and prediction fluctuations in mean field neural networks. arXiv preprint arXiv:2304.03408, 2023.
[4] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake Van- derPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/ jax.
[5] X. Chen, S. Xie, and K. He.
study of training self-supervised vision transform- ers. In 2021 IEEE/CVF International Confer- ence on Computer Vision (ICCV), pages 9620– 9629, Los Alamitos, CA, USA, oct 2021. IEEE Computer Society. doi: 10.1109/ICCV48922.2021. 00950. URL https://doi.ieeecomputersociety. org/10.1109/ICCV48922.2021.00950.
[6] Aakanksha Chowdhery, Sharan Narang, Jacob De- vlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sut- ton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
[7] Jeremy M Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on An empirical neural networks typically occurs at the edge of sta- bility. arXiv preprint arXiv:2103.00065, 2021.
[8] Jeremy M Cohen, Behrooz Ghorbani, Shankar Kr- ishnan, Naman Agarwal, Sourabh Medapati, Michal Badura, Daniel Suo, David Cardoze, Zachary Nado, George E Dahl, et al. Adaptive gradient methods at the edge of stability. arXiv preprint arXiv:2207.14484, 2022.
[9] Alex Damian, Eshaan Nichani, and Jason D Lee. Self- stabilization: The implicit bias of gradient descent at the edge of stability. arXiv preprint arXiv:2209.15594, 2022.
[10] Aaron Defazio and Konstantin Mishchenko. Learning- rate-free learning by d-adaptation. arXiv preprint arXiv:2301.07733, 2023.
[11] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision trans- formers to 22 billion parameters. arXiv preprint arXiv:2302.05442, 2023.
[12] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 (): 8-bit matrix mul- tiplication for transformers at scale. arXiv preprint arXiv:2208.07339, 2022.
[13] Emily Dinan, Sho Yaida, and Susan Zhang. Effective theory of transformers at initialization. arXiv preprint arXiv:2304.02034, 2023.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representa- tions (ICLR), 2021. https://arxiv.org/abs/2010. 11929.
[15] Colin Gaffney, Dinghua Li, Ruoxin Sang, Ayush Jain, and Haitang Hu. Orbax, 2023. URL http://github. com/google/orbax.
[16] Justin Gilmer, Andrea Schioppa, and Jeremy Co- hen. Intriguing properties of transformer training instabilities. To appear.
[17] Justin Gilmer, Behrooz Ghorbani, Ankush Garg, Sneha Kudugunta, Behnam Neyshabur, David Car- doze, George Dahl, Zachary Nado, and Orhan Firat. A loss curvature perspective on training instability in deep learning. arXiv preprint arXiv:2110.04369, 2021.
[18] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international confer- ence on artificial intelligence and statistics, pages 249–256. JMLR Workshop and Conference Proceedings, 2010.
[19] Google. Grain - feeding jax models, 2023. URL http://github.com/google/grain.

[20] Jonathan Heek, Anselm Levskaya, Avital Oliver, Mar- vin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. Flax: A neural network li- brary and ecosystem for JAX, 2023. URL http: //github.com/google/flax.
[21] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.
[22] Jordan Hoffmann, Sebastian Borgeaud, Arthur Men- sch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Jo- hannes Welbl, Aidan Clark, et al. Training compute- optimal large language models. arXiv preprint arXiv:2203.15556, 2022.
[23] Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le. Transformer quality in linear time. In International Conference on Machine Learning, pages 9099–9117. PMLR, 2022.
[24] Maor Ivgi, Oliver Hinder, and Yair Carmon. Dog is sgd’s best friend: A parameter-free dynamic step size schedule. arXiv preprint arXiv:2302.12022, 2023.
[25] Arthur Jacot, Franck Gabriel, and Cl ́ement Hon- gler. Neural tangent kernel: Convergence and gen- eralization in neural networks. In Advances in Neu- ral Information Processing Systems (NeurIPS), 2018. https://arxiv.org/abs/1806.07572.
[26] Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of a tensor pro- cessing unit. In Proceedings of the 44th annual inter- national symposium on computer architecture, pages 1–12, 2017.
[27] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
[28]Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226, 2018.
[29] Jaehoon Lee. A random walk model of transformer parameter growth, 2023.
[30] Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv:1908.03265, 2019.
[31] Peter J. Liu*, Mohammad Saleh*, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. In International Conference on Learning Representations, 2018. URL https://openreview. net/forum?id=Hyg0vbWC-.
[32] Ilya Loshchilov and Frank Hutter. Sgdr: Stochas- tic gradient descent with warm restarts. In In- ternational Conference on Learning Representations
(ICLR), 2016. https://arxiv.org/abs/1608.03983.
[33] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019. https: //openreview.net/forum?id=Bkg6RiCqY7.
[34] William Merrill, Vivek Ramanujan, Yoav Goldberg, Roy Schwartz, and Noah Smith. Effects of param- eter norm growth during transformer training: In- ductive bias from gradient descent. arXiv preprint arXiv:2010.09697, 2020.
[35] Igor Molybog, Peter Albert, Moya Chen, Zachary DeVito, David Esiobu, Naman Goyal, Punit Singh Koura, Sharan Narang, Andrew Poulton, Ruan Silva, et al. A theory on adam instability in large-scale machine learning. arXiv preprint arXiv:2304.09871, 2023.
[36] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), 2019. https://arxiv.org/abs/1912.01703.
[37] Ofir Press and Lior Wolf. Using the output embed- ding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Vol- ume 2, Short Papers, pages 157–163, Valencia, Spain, April 2017. Association for Computational Linguistics. URL https://aclanthology.org/E17-2025.
[38] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners, 2019. https: //openai.com/blog/better-language-models/.
[39] Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text trans- former. Journal of Machine Learning Research, 2020. http://jmlr.org/papers/v21/20-074.html.
[40] Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of trans- fer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020. URL http://jmlr.org/papers/v21/20-074. html.
[41] Jie Ren, Samyam Rajbhandari, Reza Yazdani Am- inabadi, Olatunji Ruwase, Shuangyan Yang, Min- jia Zhang, Dong Li, and Yuxiong He. {ZeRO- Offload}: Democratizing {Billion-Scale} model train- ing. In 2021 USENIX Annual Technical Conference
(USENIX ATC 21), pages 551–564, 2021.
[42] Noam Shazeer and Mitchell Stern. Adafactor: Adap- tive learning rates with sublinear memory cost. In International Conference on Machine Learning, pages 4596–4604. PMLR, 2018.
[43] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864, 2021.
[44] Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, and Joshua Susskind. The sling- shot mechanism: An empirical study of adaptive opti- mizers and the grokking phenomenon. arXiv preprint arXiv:2206.04817, 2022.
[45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing sys- tems, 30, 2017.
[46] Mitchell Wortsman, Jaehoon Lee, Justin Gilmer, and Simon Kornblith. Replacing softmax with relu in vision transformers.
[47] MitchellWortsman,TimDettmers,LukeZettlemoyer, Ari Morcos, Ali Farhadi, and Ludwig Schmidt. Sta- ble and low-precision training for large-scale vision- language models. arXiv preprint arXiv:2304.13013, 2023.
[48] Sho Yaida. Meta-principled family of hyperparameter scaling strategies. arXiv preprint arXiv:2210.04909, 2022.
[49] Greg Yang and Edward J Hu. Tensor programs iv: Feature learning in infinite-width neural networks. In International Conference on Machine Learning, pages 11727–11737. PMLR, 2021.
[50] Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Ten- sor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466, 2022.
[51] Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, and Josh Susskind. Stabilizing trans- former training by preventing attention entropy col- lapse. arXiv preprint arXiv:2303.06296, 2023.
[52] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. arXiv preprint arXiv:2303.15343, 2023.
[53] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.

### term list
|no.|term|count|
|:----|:----|:----|
|1|the|471|
|2|e|231|
|3|and|225|
|4|of|204|
|5|in|180|
|6|to|151|
|7|we|142|
|8|lr|129|
|9|learning|117|
|10|for|115|
|11|a|110|
|12|loss|106|
|13|is|101|
|14|n|99|
|15|rate|88|
|16|this|88|
|17|with|83|
|18|figure|75|
|19|that|68|
|20|at|67|
|21|sensitivity|67|
|22|as|64|
|23|scaling|64|
|24|model|61|
|25|arxiv|59|
|26|on|59|
|27|when|57|
|28|attention|55|
|29|by|54|
|30|layernorm|53|
|31|training|48|
|32|rms|47|
|33|scale|45|
|34|instability|43|
|35|models|43|
|36|al|42|
|37|et|41|
|38|qk|40|
|39|are|39|
|40|which|39|
|41|not|38|
|42|logit|37|
|43|section|37|
|44|decay|36|
|45|use|35|
|46|final|34|
|47|parameters|32|
|48|growth|31|
|49|param|31|
|50|from|30|
|51|layers|30|
|52|transformer|30|
|53|large|29|
|54|gradient|28|
|55|number|28|
|56|an|27|
|57|preprint|27|
|58|weight|25|
|59|eval|24|
|60|max|24|
|61|step|24|
|62|without|24|
|63|instabilities|23|
|64|z|23|
|65|be|22|
|66|default|22|
|67|small|22|
|68|output|21|
|69|effect|20|
|70|it|20|
|71|layer|20|
|72|or|20|
|73|our|20|
|74|have|19|
|75|logits|19|
|76|test|19|
|77|up|19|
|78|vs|19|
|79|appendix|18|
|80|dim|18|
|81|rates|18|
|82|where|18|
|83|across|17|
|84|also|17|
|85|head|17|
|86|num|17|
|87|size|17|
|88|b|16|
|89|can|16|
|90|grad|16|
|91|neural|16|
|92|parameter|16|
|93|width|16|
|94|adamw|15|
|95|depth|15|
|96|divergence|15|
|97|first|15|
|98|has|15|
|99|more|15|
|100|params|15|
|101|scales|15|
|102|steps|15|
|103|update|15|
|104|x|15|
|105|curves|14|
|106|does|14|
|107|experiment|14|
|108|no|14|
|109|other|14|
|110|stability|14|
|111|transformers|14|
|112|block|13|
|113|dh|13|
|114|optimal|13|
|115|p|13|
|116|two|13|
|117|useful|13|
|118|warm|13|
|119|batch|12|
|120|change|12|
|121|conference|12|
|122|high|12|
|123|i|12|
|124|interventions|12|
|125|j|12|
|126|largest|12|
|127|m|12|
|128|optimizer|12|
|129|refer|12|
|130|characteristics|11|
|131|fan|11|
|132|h|11|
|133|hyperparameter|11|
|134|illustrated|11|
|135|independent|11|
|136|liu|11|
|137|magnitude|11|
|138|mean|11|
|139|root|11|
|140|self|11|
|141|simple|11|
|142|will|11|
|143|work|11|
|144|adaptive|10|
|145|additional|10|
|146|before|10|
|147|changes|10|
|148|d|10|
|149|different|10|
|150|increases|10|
|151|international|10|
|152|language|10|
|153|occurs|10|
|154|results|10|
|155|spikes|10|
|156|study|10|
|157|their|10|
|158|there|10|
|159|these|10|
|160|trends|10|
|161|using|10|
|162|behavior|9|
|163|dimension|9|
|164|do|9|
|165|https|9|
|166|if|9|
|167|increasing|9|
|168|information|9|
|169|larger|9|
|170|mlp|9|
|171|norm|9|
|172|predict|9|
|173|so|9|
|174|square|9|
|175|url|9|
|176|value|9|
|177|was|9|
|178|while|9|
|179|yang|9|
|180|adam|8|
|181|between|8|
|182|but|8|
|183|data|8|
|184|dehghani|8|
|185|examine|8|
|186|experiments|8|
|187|gilmer|8|
|188|however|8|
|189|known|8|
|190|lee|8|
|191|networks|8|
|192|observed|8|
|193|only|8|
|194|org|8|
|195|pages|8|
|196|peter|8|
|197|range|8|
|198|such|8|
|199|then|8|
|200|via|8|
|201|warmup|8|
|202|because|7|
|203|both|7|
|204|c|7|
|205|cases|7|
|206|cohen|7|
|207|david|7|
|208|deviation|7|
|209|edge|7|
|210|find|7|
|211|flax|7|
|212|g|7|
|213|google|7|
|214|how|7|
|215|http|7|
|216|increase|7|
|217|infrastructure|7|
|218|instead|7|
|219|jax|7|
|220|justin|7|
|221|let|7|
|222|linear|7|
|223|measuring|7|
|224|mitchell|7|
|225|mitigation|7|
|226|need|7|
|227|network|7|
|228|orders|7|
|229|paper|7|
|230|per|7|
|231|previously|7|
|232|reduces|7|
|233|related|7|
|234|reported|7|
|235|than|7|
|236|three|7|
|237|trained|7|
|238|v|7|
|239|values|7|
|240|weights|7|
|241|whether|7|
|242|zhai|7|
|243|base|6|
|244|believe|6|
|245|changing|6|
|246|chen|6|
|247|com|6|
|248|contributed|6|
|249|denote|6|
|250|descent|6|
|251|described|6|
|252|diverged|6|
|253|during|6|
|254|embedding|6|
|255|hutter|6|
|256|ii|6|
|257|instance|6|
|258|issue|6|
|259|li|6|
|260|log|6|
|261|loshchilov|6|
|262|machine|6|
|263|meaning|6|
|264|new|6|
|265|norms|6|
|266|now|6|
|267|over|6|
|268|predicted|6|
|269|prediction|6|
|270|previous|6|
|271|process|6|
|272|progressive|6|
|273|question|6|
|274|relationship|6|
|275|required|6|
|276|research|6|
|277|result|6|
|278|set|6|
|279|sharpening|6|
|280|shazeer|6|
|281|shown|6|
|282|similar|6|
|283|softmax|6|
|284|stabilization|6|
|285|standard|6|
|286|text|6|
|287|they|6|
|288|too|6|
|289|train|6|
|290|zero|6|
|291|become|5|
|292|becomes|5|
|293|been|5|
|294|best|5|
|295|chowdhery|5|
|296|coefficient|5|
|297|collapse|5|
|298|compute|5|
|299|curvature|5|
|300|details|5|
|301|ei|5|
|302|empirical|5|
|303|experimental|5|
|304|factor|5|
|305|figures|5|
|306|full|5|
|307|george|5|
|308|github|5|
|309|independently|5|
|310|initialization|5|
|311|intervention|5|
|312|jaehoon|5|
|313|key|5|
|314|keys|5|
|315|longer|5|
|316|lower|5|
|317|may|5|
|318|measure|5|
|319|metric|5|
|320|noam|5|
|321|performance|5|
|322|plot|5|
|323|probabilities|5|
|324|processing|5|
|325|proposed|5|
|326|query|5|
|327|recommended|5|
|328|s|5|
|329|schedule|5|
|330|sequence|5|
|331|shrink|5|
|332|smaller|5|
|333|stable|5|
|334|studied|5|
|335|technical|5|
|336|unscaled|5|
|337|used|5|
|338|variant|5|
|339|vision|5|
|340|widthscaling|5|
|341|writing|5|
|342|zhang|5|
|343|abs|4|
|344|alex|4|
|345|all|4|
|346|analysis|4|
|347|appear|4|
|348|based|4|
|349|better|4|
|350|case|4|
|351|constant|4|
|352|deepmind|4|
|353|discussed|4|
|354|due|4|
|355|each|4|
|356|effective|4|
|357|employed|4|
|358|examining|4|
|359|example|4|
|360|fast|4|
|361|finally|4|
|362|fit|4|
|363|focus|4|
|364|function|4|
|365|hu|4|
|366|hyperparameters|4|
|367|iii|4|
|368|image|4|
|369|important|4|
|370|initialize|4|
|371|input|4|
|372|james|4|
|373|jeffrey|4|
|374|john|4|
|375|led|4|
|376|length|4|
|377|methods|4|
|378|minimum|4|
|379|moreover|4|
|380|narang|4|
|381|one|4|
|382|practice|4|
|383|pre|4|
|384|predicting|4|
|385|primarily|4|
|386|proceedings|4|
|387|project|4|
|388|queries|4|
|389|regime|4|
|390|representations|4|
|391|reproduce|4|
|392|same|4|
|393|scaled|4|
|394|see|4|
|395|shape|4|
|396|sharan|4|
|397|shift|4|
|398|show|4|
|399|simon|4|
|400|sizes|4|
|401|stabilizing|4|
|402|techniques|4|
|403|throughout|4|
|404|tokens|4|
|405|trueqk|4|
|406|unstable|4|
|407|validation|4|
|408|wortsman|4|
|409|would|4|
|410|xj|4|
|411|zk|4|
|412|above|3|
|413|access|3|
|414|activation|3|
|415|add|3|
|416|advances|3|
|417|affect|3|
|418|aidan|3|
|419|aims|3|
|420|alleviate|3|
|421|although|3|
|422|appears|3|
|423|applying|3|
|424|around|3|
|425|averaged|3|
|426|ben|3|
|427|bias|3|
|428|billion|3|
|429|causes|3|
|430|colin|3|
|431|common|3|
|432|computer|3|
|433|computing|3|
|434|conducted|3|
|435|connections|3|
|436|consistent|3|
|437|contains|3|
|438|contrast|3|
|439|cosine|3|
|440|could|3|
|441|dahl|3|
|442|decrease|3|
|443|decreasing|3|
|444|deep|3|
|445|depthdim|3|
|446|dettmers|3|
|447|direction|3|
|448|discussion|3|
|449|dynamics|3|
|450|element|3|
|451|end|3|
|452|exceeds|3|
|453|experimentation|3|
|454|far|3|
|455|faster|3|
|456|feature|3|
|457|features|3|
|458|following|3|
|459|framing|3|
|460|free|3|
|461|gradually|3|
|462|grain|3|
|463|hand|3|
|464|he|3|
|465|here|3|
|466|iccv|3|
|467|iclr|3|
|468|igor|3|
|469|ilya|3|
|470|improve|3|
|471|improvement|3|
|472|indicate|3|
|473|intermediate|3|
|474|into|3|
|475|inverse|3|
|476|investigate|3|
|477|investigation|3|
|478|issues|3|
|479|its|3|
|480|jeremy|3|
|481|jmlr|3|
|482|jonathan|3|
|483|k|3|
|484|kernel|3|
|485|kornblith|3|
|486|last|3|
|487|lead|3|
|488|left|3|
|489|long|3|
|490|lrs|3|
|491|maximum|3|
|492|meaningfully|3|
|493|measures|3|
|494|michael|3|
|495|most|3|
|496|naman|3|
|497|next|3|
|498|nm|3|
|499|noah|3|
|500|normalization|3|
|501|observation|3|
|502|observe|3|
|503|optimizers|3|
|504|orbax|3|
|505|papers|3|
|506|particular|3|
|507|phenomenon|3|
|508|plots|3|
|509|pmlr|3|
|510|points|3|
|511|pools|3|
|512|presents|3|
|513|programs|3|
|514|projection|3|
|515|provided|3|
|516|pytorch|3|
|517|quadratic|3|
|518|raffel|3|
|519|recommend|3|
|520|reduce|3|
|521|reducing|3|
|522|regularization|3|
|523|remainder|3|
|524|remains|3|
|525|reproduced|3|
|526|researchers|3|
|527|resource|3|
|528|respectively|3|
|529|right|3|
|530|roberts|3|
|531|roman|3|
|532|run|3|
|533|sequences|3|
|534|shifting|3|
|535|slightly|3|
|536|slow|3|
|537|st|3|
|538|studying|3|
|539|substantially|3|
|540|successful|3|
|541|summarize|3|
|542|systems|3|
|543|tends|3|
|544|tensor|3|
|545|therefore|3|
|546|though|3|
|547|threshold|3|
|548|tom|3|
|549|top|3|
|550|total|3|
|551|transfer|3|
|552|trevor|3|
|553|typically|3|
|554|u|3|
|555|understanding|3|
|556|variation|3|
|557|vaswani|3|
|558|w|3|
|559|xw|3|
|560|y|3|
|561|zachary|3|
|562|zhou|3|
|563|zj|3|
|564|aaron|2|
|565|abhishek|2|
|566|achieve|2|
|567|adafactor|2|
|568|adlam|2|
|569|advice|2|
|570|aeos|2|
|571|alec|2|
|572|alemi|2|
|573|alexander|2|
|574|allows|2|
|575|amodei|2|
|576|amount|2|
|577|andreas|2|
|578|annual|2|
|579|another|2|
|580|answer|2|
|581|any|2|
|582|applies|2|
|583|architecture|2|
|584|areas|2|
|585|arthur|2|
|586|association|2|
|587|auxiliary|2|
|588|babuschkin|2|
|589|baseline|2|
|590|basil|2|
|591|behrooz|2|
|592|below|2|
|593|beyer|2|
|594|biases|2|
|595|big|2|
|596|blocks|2|
|597|bottom|2|
|598|bradbury|2|
|599|cai|2|
|600|cause|2|
|601|characterized|2|
|602|child|2|
|603|chris|2|
|604|clark|2|
|605|clear|2|
|606|closely|2|
|607|co|2|
|608|combine|2|
|609|comparing|2|
|610|computational|2|
|611|conclude|2|
|612|confirm|2|
|613|consider|2|
|614|consistently|2|
|615|context|2|
|616|corresponding|2|
|617|cossim|2|
|618|currently|2|
|619|curve|2|
|620|damian|2|
|621|dan|2|
|622|daniel|2|
|623|dario|2|
|624|decoupled|2|
|625|decreases|2|
|626|defined|2|
|627|demonstrating|2|
|628|depends|2|
|629|detailed|2|
|630|dickstein|2|
|631|displays|2|
|632|diverge|2|
|633|documented|2|
|634|doi|2|
|635|ecosystem|2|
|636|edward|2|
|637|emerge|2|
|638|emerges|2|
|639|enable|2|
|640|enables|2|
|641|entropy|2|
|642|eos|2|
|643|eps|2|
|644|epsilon|2|
|645|equally|2|
|646|etai|2|
|647|everett|2|
|648|evidence|2|
|649|examines|2|
|650|exhibit|2|
|651|explain|2|
|652|explanation|2|
|653|exploring|2|
|654|extrapolating|2|
|655|fails|2|
|656|falsen|2|
|657|finding|2|
|658|fixed|2|
|659|focuses|2|
|660|followed|2|
|661|forum|2|
|662|found|2|
|663|francisco|2|
|664|frank|2|
|665|further|2|
|666|gao|2|
|667|gaurav|2|
|668|ghorbani|2|
|669|goyal|2|
|670|gradients|2|
|671|greg|2|
|672|grow|2|
|673|gur|2|
|674|had|2|
|675|heads|2|
|676|heek|2|
|677|helpful|2|
|678|highlight|2|
|679|html|2|
|680|id|2|
|681|identify|2|
|682|ieee|2|
|683|impact|2|
|684|implementations|2|
|685|improves|2|
|686|index|2|
|687|individually|2|
|688|initial|2|
|689|insight|2|
|690|insights|2|
|691|interaction|2|
|692|interesting|2|
|693|invariant|2|
|694|investigations|2|
|695|isolation|2|
|696|iv|2|
|697|ivgi|2|
|698|izzeddin|2|
|699|jake|2|
|700|jakob|2|
|701|jascha|2|
|702|jason|2|
|703|jianfeng|2|
|704|joint|2|
|705|jointly|2|
|706|jones|2|
|707|journal|2|
|708|just|2|
|709|kaiser|2|
|710|kaplan|2|
|711|katherine|2|
|712|katie|2|
|713|kelvin|2|
|714|kevin|2|
|715|kolesnikov|2|
|716|kumar|2|
|717|kxk|2|
|718|latter|2|
|719|leads|2|
|720|lechao|2|
|721|less|2|
|722|library|2|
|723|limits|2|
|724|lin|2|
|725|linguistics|2|
|726|littwin|2|
|727|loading|2|
|728|losseps|2|
|729|lossweight|2|
|730|low|2|
|731|lucas|2|
|732|lukasz|2|
|733|luke|2|
|734|main|2|
|735|mainly|2|
|736|many|2|
|737|matena|2|
|738|matrix|2|
|739|meaningful|2|
|740|measurable|2|
|741|mechanism|2|
|742|merrill|2|
|743|method|2|
|744|middle|2|
|745|min|2|
|746|mitigates|2|
|747|mitigations|2|
|748|modification|2|
|749|mostafa|2|
|750|moya|2|
|751|much|2|
|752|multiple|2|
|753|muparam|2|
|754|mustafa|2|
|755|nado|2|
|756|nanodo|2|
|757|net|2|
|758|neurips|2|
|759|norman|2|
|760|note|2|
|761|novak|2|
|762|obtained|2|
|763|occur|2|
|764|oct|2|
|765|offers|2|
|766|often|2|
|767|oliver|2|
|768|openreview|2|
|769|opportunities|2|
|770|optax|2|
|771|order|2|
|772|otherwise|2|
|773|overall|2|
|774|packed|2|
|775|padding|2|
|776|parameterizations|2|
|777|parameterizing|2|
|778|paszke|2|
|779|pennington|2|
|780|performs|2|
|781|periodic|2|
|782|perspective|2|
|783|pi|2|
|784|play|2|
|785|pointwise|2|
|786|possible|2|
|787|precision|2|
|788|preconditioned|2|
|789|predicts|2|
|790|produces|2|
|791|provides|2|
|792|qi|2|
|793|qklayernorm|2|
|794|quadratically|2|
|795|radford|2|
|796|raises|2|
|797|random|2|
|798|recall|2|
|799|received|2|
|800|regardless|2|
|801|reliable|2|
|802|relu|2|
|803|repeats|2|
|804|report|2|
|805|reporting|2|
|806|residual|2|
|807|resolve|2|
|808|resolves|2|
|809|resources|2|
|810|rewon|2|
|811|reyes|2|
|812|rmsn|2|
|813|role|2|
|814|rotary|2|
|815|row|2|
|816|roy|2|
|817|runs|2|
|818|ryan|2|
|819|sam|2|
|820|scientific|2|
|821|sebastian|2|
|822|sections|2|
|823|sensitivityscaling|2|
|824|sensitivitystandardmuparam|2|
|825|sentencepiece|2|
|826|setting|2|
|827|sgd|2|
|828|sharpness|2|
|829|sho|2|
|830|shows|2|
|831|shuangfei|2|
|832|singh|2|
|833|smallest|2|
|834|sohl|2|
|835|some|2|
|836|sometimes|2|
|837|sources|2|
|838|specify|2|
|839|standardmuparam|2|
|840|state|2|
|841|steiner|2|
|842|stephen|2|
|843|stern|2|
|844|still|2|
|845|succeeds|2|
|846|summary|2|
|847|support|2|
|848|susan|2|
|849|susskind|2|
|850|sweep|2|
|851|th|2|
|852|thank|2|
|853|theory|2|
|854|thilak|2|
|855|thomas|2|
|856|through|2|
|857|tim|2|
|858|tokenizer|2|
|859|tool|2|
|860|towards|2|
|861|tpus|2|
|862|unified|2|
|863|unit|2|
|864|unless|2|
|865|usenix|2|
|866|uszkoreit|2|
|867|varying|2|
|868|very|2|
|869|ways|2|
|870|wei|2|
|871|weizhu|2|
|872|were|2|
|873|works|2|
|874|wu|2|
|875|xi|2|
|876|xiao|2|
|877|xiaodong|2|
|878|xiaohua|2|
|879|xu|2|
|880|yaida|2|
|881|yanqi|2|
|882|zeros|2|
|883|zettlemoyer|2|
|884|TRUE|2|
|合計|1,882|	9,477|

合計は出現数１の単語を含みます。