Title: Stack More Layers Differently: High-Rank Training Through Low-Rank Updates
Venue: NeurIPS 2023
Accepted: no
Keywords: efficient training

scaling laws

language models

low-rank training

parameter-efficient fine-tuning
Abstract: Despite the dominance and effectiveness of scaling, resulting in large networks with hundreds of billions of parameters,
the necessity to train overparametrized models remains poorly understood, and alternative approaches do not necessarily make it cheaper to train high-performance models.
In this paper, we explore low-rank training techniques as an alternative approach to training large neural networks. We introduce a novel method called ReLoRA, which utilizes low-rank updates to train high-rank networks.
We apply ReLoRA to pre-training transformer language models with up to 350M parameters and demonstrate comparable performance to regular neural network training. Furthermore, we observe that the efficiency of ReLoRA increases with model size, making it a promising approach for training multi-billion-parameter networks efficiently. Our findings shed light on the potential of low-rank training techniques and their implications for model scaling.

1

Abstract

Despite the dominance and effectiveness of scaling, resulting in large networks with hundreds of billions of parameters, the necessity to train overparametrized models remains poorly understood, and alternative approaches do not necessarily make it cheaper to train high-performance models. In this paper, we explore low-rank training techniques as an alternative approach to training large neural networks. We introduce a novel method called ReLoRA, which utilizes low-rank updates to train high-rank networks. We apply ReLoRA to pre-training transformer language models with up to 350M parameters, and demonstrate comparable performance to regular neural network training. Furthermore, we observe that the efficiency of ReLoRA increases with model size, making it a promising approach for training multi-billion-parameter networks efficiently. Our findings shed light on the potential of low-rank training techniques and their implications for scaling laws. 1

1 Introduction

Over the past decade, the machine learning field has been dominated by the trend of training increasingly overparametrized networks or adopting the "stack more layers" approach [32, 21, 27]. The definition of a large network has evolved from models with 100 million [46, 39] to hundreds of billions [8, 12] of parameters, which has made computational costs associated with training of such networks prohibitive to most of the research groups. Despite this, the necessity to train models which can have orders of magnitude more parameters than the training examples [8, 12, 16], is poorly understood theoretically [25, 4, 60].

Alternative approaches to scaling, such as more compute-efficient scaling optima [22], retrievalaugmented models [28, 7], and the simple approach of training smaller models for longer [50], have offered new interesting trade-offs. However, they do not bring us closer to understanding why we need overparametrized models and rarely democratize the training of these models. For example, training RETRO [7] requires a complex training setup and infrastructure capable of quickly searching over trillions of tokens, while training LLaMA-6B [50] still requires hundreds of GPUs.

In contrast, approaches like zero-redundancy optimizers [43], 16-bit training [37], 8-bit inference [14], and parameter-efficient fine-tuning (PEFT) [33] have played a crucial role in making large models more accessible. Specifically, PEFT methods have enabled fine-tuning of billion-scale language or diffusion models on consumer hardware. This raises the question: Can these approaches also benefit pre-training?

On one hand, pre-training is exactly the step that allows for small modifications to the network to 32 adapt it to new tasks. Aghajanyan et al. [1] demonstrated that the rank of the changes required 33

0

2500

5000

7500

10000

12500

15000

17500

20000

3.0

3.5

4.0

4.5

5.0

Loss

250M

250M ReLoRA

(99M trainable)

99M

0

2500

5000

7500

10000

12500

15000

17500

20000

Step

0

100

250

Trainable Params

to learn a task decreases the more you pre-train the network. On the other hand, multiple studies 34 have demonstrated the simplicity of features extracted and utilized by language and vision models, 35 along with their low intrinsic dimensionality [31, 17, 42, 47]. For instance, attention patterns in 36 transformers [51] often exhibit a small rank, which has been successfully leveraged to develop more 37 efficient variants of attention [52, 11]. Moreover, overparametrization is also not necessary for 38 training. The Lottery Ticket Hypothesis [17] empirically demonstrates that during initialization (or 39 early in training [18]), there exist sub-networks - winning tickets - that when trained in isolation 40 reach the performance of the full network. 41

In this study, we focus on low-rank training techniques and introduce ReLoRA that uses low-rank 42 updates to train a high-rank network. We empirically demonstrate that ReLoRA performs a high-rank 43 update and achieves performance similar to regular neural network training. The components of 44 ReLoRA include initial full-rank training of the neural network (similar to Frankle et al. [18]), LoRA 45 training, restarts, a jagged learning rate schedule, and partial optimizer resets. We evaluate ReLoRA 46 on transformer language models up to 350M parameters. We chose to focus on autoregressive 47 language modeling, as this approach has demonstrated its universality in most of the applications of 48 neural networks [41, 56, 3, 35, 10]. Finally, we observe that the efficiency of ReLoRA increases with 49 model size, making it a viable option for efficient training of multi-billion-parameter networks. 50

52

2 Related work

Scaling versus Efficiency The relationship between overparametrization and neural network 53 trainability and generalization has been extensively studied [59, 5, 17, 38, 47], yet it remains a 54 mystery [60]. Moreover, scaling laws [27, 19, 22, 30, 2] demonstrate a simple and strong power-law 55 dependence between network size and its performance across a variety of modalities. This finding 56 not only supports overparametrization but also encourages the training of extraordinarily resource57 intensive neural networks [8, 12, 16]. Nonetheless, the Lottery Ticket Hypothesis [17, 18] suggests 58 that overparametrization could, in principle, be minimized. Specifically, it shows that early in training, 59 subnetworks exist that can be trained to achieve the performance of the full network (winning tickets) . 60

Parameter-efficient fine-tuning Aghajanyan et al. [1] found that pre-training reduces the amount 61 of change to the network, or its intrinsic dimensionality, to learn a new task through fine-tuning. I.e., 62 larger networks or networks pre-trained on more data require smaller modifications in terms of the 63 rank of the range to learn a new task. This explains the success of parameter-efficient fine-tuning 64 methods [33] and has also motivated the development of low-rank fine-tuning methods such as LoRA 65 [23] and Compacter [36]. 66

67

68

69

70

71

72

73

74

75

76

77

78

0

2000

4000

6000

8000

10000

Step

0.0

0.2

0.4

0.6

0.8

1.0

Learning Rate

Low-rank neural network training Training low-rank representations has been explored in the context of CNN compression, regularization, and efficient training [24, 26, 49, 44, 34, 57]. However, most of these methods are either specific to CNNs, do not scale well, or have not been evaluated on large transformers [51] with hundreds of millions of parameters, which can benefit greatly from efficient training. While transformers have been shown to have a low-rank internal dimensionality and representations [1, 52], the study by Bhojanapalli et al. [6] demonstrated that the low rank of key and query projections in multi-head attention bottlenecks the performance of transformers. Our own experiments (Section 3) also demonstrate that low-rank transformers perform significantly worse compared to the full-rank baseline and ReLoRA.

3 Method

Let's start by revisiting linear algebra-101. In particular, we are interested in the rank of the sum of two matrices:

This bound on the rank of the sum is tight: for a matrix A , rank ( A ) < dim ( A ) , there exists B , 79 rank ( B ) < dim ( B ) such that sum of the matrices has a higher rank than either A or B . We want to 80 exploit this property to make a flexible parameter-efficient training method. We start with LoRA [23] 81 which is a parameter-efficient fine-tuning method based on the idea of low-rank updates. LoRA can 82 be applied to any linear operation parametrized through W ∈ R m × n . Specifically, LoRA decomposes 83 the weight update δW into a low-rank product W A W B as shown in Equation 2, where s ∈ R is a 84 fixed scaling factor usually equal to 1 r . 85

In practice, LoRA is usually implemented by adding new trainable parameters W A and W B , which 86 could be merged back into the original parameters after training. Thus, even though Equation 1 87 allows the total update over training time ∑ t δW t to have a higher rank than any of the individual 88 matrices, LoRA implementations are restricted by the rank r = max W A ,W B rank ( W A W B ) . 89

If we could restart LoRA, meaning we merge W A and W B during training and reset the values of 90 these matrices, we could increase the total rank of the update. Doing this multiple times brings the 91 total neural network update to 92

where the sums are independent enough, meaning that rank ( W i A W i B ) + rank ( W j A W j B ) ≥ r . 93

However, implementing restarts is not trivial in practice and requires certain modifications to the 94 optimization procedure. Naïve implementation causes the model to diverge right after the restart. 95 Unlike plain stochastic gradient descent, which solely relies on the value of the gradient at the current 96 optimization timestep, Adam [29] update is guided mainly by the first and second moments of the 97 gradient accumulated over the previous steps. In practice, gradient moment smoothing parameters β 1 98 and β 2 are usually very high 0 . 9 -0 . 999 . Let's assume that at the reinitialization boundary W 1 A and 99 the corresponding gradient moments m A and v A , are full-rank ( r ). Then, after the merge-and-reinit, 100 continuing to use old gradient moments for W 2 A will guide it in the same direction as W 1 A and 101 optimize the same subspace. 102

103

104

105

106

107

To resolve this issue, we propose ReLoRA. ReLoRA performs a partial reset of the optimizer state during merge-and-reinit and sets the learning rate to 0 with a subsequent warmup. Specifically, we set 99% of low-magnitude optimizer state values to zero and use a jagged-cosine learning rate schedule (Figure 2). Our ablation studies (Section 3) show that both of these modifications are required to improve the performance over vanilla LoRA.

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

To reiterate, ReLoRA is a low-rank training method inspired by LoRA that uses restarts to increase the effective rank of the update, uses partial optimizer reset, and a jagged scheduler to stabilize training and warm starts. All of this allows ReLoRA to achieve performance comparable to full-rank training, especially in large transformer networks, by only training a small set of parameters at a time. ReLoRA is described in Algorithm 1.

Enhancing computational efficiency Unlike other low-rank training techniques [44, 49], ReLoRA follows the LoRA approach by maintaining the frozen weights of the original network and adding new trainable parameters. At first glance, this may appear computationally inefficient; however, the differentiation between frozen and trainable parameters plays a crucial role in parameter-efficient fine-tuning [33].

These methods achieve significant improvements in training time and memory efficiency by reducing the size of the gradients and the optimizer states. Notably, Adam states consume twice as much memory as the model weights. Moreover, it is common practice to maintain gradient accumulation buffers in 32-bit precision for large networks, thereby adding significant overhead to the memory consumption of gradients.

By substantially reducing the number of trainable parameters, ReLoRA enables the utilization of larger batch sizes, maximizing hardware efficiency. Additionally, it reduces the bandwidth requirements in distributed setups, which are often the limiting factor in large-scale training.

Furthermore, since the frozen parameters are not being updated between restarts, they can be kept in a low-precision quantized format, further reducing their memory and computational impact. This additional optimization contributes to overall improved efficiency in terms of memory utilization and computational resources of ReLoRA and increases at scale.

4 Experiments

To evaluate the effectiveness of ReLoRA, we apply it to train a transformer language model on the C4 dataset [41] using various model sizes: 60M, 130M, 250M, and 350M. Language modeling has been shown to be a fundamental task in machine learning [40], it enables text and image classification [56], translation [8], programming [9], in-context learning, step-by-step reasoning [54], and many other emergent abilities [53]. Given its significance, we focus solely on language modeling for the purposes of this paper.

Architecture and training hyperparameters Our architecture is based on transformer [51] and closely resembles LLaMA [50]. Namely, we use pre-normalization, RMSNorm [58], SwiGLU activations [45], 8 3 h fully-connected hidden state size [50], and rotary embeddings [48]. All hyperparameters are presented in Table 1.

We use bfloat16 for all floating point operations and Flash attention [13] for effective attention 141 computation. Compared to attention in LLaMA, which uses float32 for softmax computation, this 142 increased training throughput by 50-100% without any training stability issues. 143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

Algorithm 1 ReLoRA. θ is model parameters, ˆ θ is model parameters with linear layers replaced with ReLoRA, M and V are Adam optimizer states, η is learning rate scheduled according to a jagged scheduler, and finally, q is the reinit frequency.

Most of our models were trained on 8 RTX 4090 for one day or less. Due to computational constraints, we train much smaller models than LLaMA, with the largest model having 350M parameters, the same as BERT Large [15]. We select the number of pre-training tokens based on the Chinchilla scaling laws [22] for all models, except for the largest one, which we train for 6.8B tokens while 9.5B tokens are Chinchilla-optimal.

ReLoRA and baselines setup In our low-rank training experiments, ReLoRA replaces all attention and fully-connected network parameters, while keeping the embeddings full-rank. The RMSNorm parametrization remains unchanged. Since ReLoRA-wrapped models have fewer trainable parameters than full-rank training, we include a Control baseline, which is a full-rank transformer with the same number of trainable parameters as ReLoRA.

We initialize ReLoRA from a checkpoint of full-rank training at 5,000 update steps and reset it every 5,000 steps thereafter, 3 times in total. After each reset, 99% of the optimizer state is pruned based on magnitude, and the loss is warmed up for the next 100 iterations. ReLoRA parameters are reinitialized following LoRA best practices, Kaiming initialization [20] for A -matrix, and zeros for B -matrix. In case of not using the restarts, the B -matrix also uses Kaiming initialization to avoid gradient-symmetry issues.

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

0.0

0.5

1.0

1.5

2.0

Singular Value

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Frequency

Q Projections

ReLoRA

LoRA

Full-rank

training

0.0

0.5

1.0

1.5

2.0

Singular Value

0.0

0.5

1.0

1.5

2.0

2.5

3.0

K Projections

ReLoRA

LoRA

Full-rank

training

0.0

0.5

1.0

1.5

2.0

Singular Value

0.0

0.5

1.0

1.5

2.0

2.5

3.0

V Projections

ReLoRA

LoRA

Full-rank

training

0.0

0.5

1.0

1.5

2.0

Singular Value

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Down Projections

ReLoRA

LoRA

Full-rank

training

5 Results

Parameter-efficient pre-training Our main results are resented in Table 2. ReLoRA significantly outperforms low-rank LoRA training demonstrating the effectiveness of our proposed modifications (ablated in Section 3). Furthermore, ReLoRA achieves similar performance to full-rank training, and the performance gap diminishes as network size increases.

Interestingly, the only model in which ReLoRA couldn't surpass the Control baseline was our smallest model with 60M parameters. This observation suggests that ReLoRA is particularly effective in improving the training of large networks, which aligns with our goal of developing a method that improves large-network training.

High-rank training through low-rank updates To determine whether ReLoRA performs a higher rank update than LoRA we plot the singular value spectrum of the difference between warm-start weights and the final weights for ReLoRA, LoRA, and full-rank training. Figure 3 illustrates significant qualitative differences between LoRA and ReLoRA for the singular values of W Q , W K , W V , and W down .

While most of the singular values for LoRA are zero (Figure 4) with a noticeable number of exceptionally high values above 1.5, ReLoRA exhibits a higher distribution mass between 0.1 and 1.0, reminiscent of full-rank training. This observation emphasizes the significance of high-rank updates and demonstrates the qualitative efficacy of ReLoRA, which accomplishes a high-rank update by performing multiple low-rank updates.

5.1 Ablation studies

We conduct ablation studies on all four crucial components of ReLoRA: restarts, jagged schedule, 180 optimizer resets, and warm starts, utilizing the 130M-sized model. The results are presented in 181 Table 3. In this section, we will focus on and analyze certain combinations of these components. 182

WQ

WK

WV

Wup

Wdown

0

200

400

600

Full-rank

Training

ReLoRA

LoRA

LoRA ReLoRA, without the aforementioned components, is essentially equivalent to training 183 a low-rank network parameterized by LoRA. This approach yields remarkably high perplexity, 184 indicating that a simple matrix decomposition has significantly different training dynamics from 185 full-rank training. 186

187

188

189

190

191

192

193

Adding restarts and optimizer resets ReLoRA, without a jagged schedule and optimizer reset, performs similarly to LoRA because old optimizer states force the newly initialized parameters into the same subspace as the prior weights, limiting the model's capacity. However, doing a naive optimizer reset with ReLoRA causes the model to diverge. A jagged schedule helps to stabilize training and has a positive impact on the mixture. In our initial experiments, we also observed that a combination of partial optimizer reset and jagged scheduler allows for a quicker warm-up, as low as 50 steps, instead of hundreds of steps required when the optimizer is initialized from scratch.

194

195

196

197

198

199

200

Warm start The warm start shows the most significant improvement, dropping perplexity by almost 10 points. To investigate whether post-warmup training contributes to the loss, we measured the perplexity of the warmed-up network, which equals 27 . 03 . It outperforms all low-rank methods except for our final ReLoRA recipe but still demonstrates a significant difference from the final network. This demonstrates the importance of early training, similar to the concept of the lottery ticket hypothesis with rewinding [18].

6 Conclusion

In this paper, we investigated low-rank training techniques for large transformer language models. 201 We first examined the limitations of a simple low-rank matrix factorization (LoRA) approach and 202 observed that it struggles to effectively train high-performing transformer models. To address this 203 issue, we proposed a novel method called ReLoRA, which leverages the rank of sum property to train 204 a high-rank network through multiple low-rank updates. Similar to the lottery ticket hypothesis with 205 rewinding, ReLoRA employs a full-rank training warm start before transitioning to ReLoRA. Addi206 tionally, ReLoRA introduces a merge-and-reinit (restart) strategy, a jagged learning rate scheduler, 207 and partial optimizer resets, which collectively enhance the efficiency of ReLoRA and bring it closer 208 to full-rank training, particularly in large networks . ReLoRA efficiency increases with the network 209 size making it a viable candidate for multi-billion-scale training. 210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

We firmly believe that the development of low-rank training methods holds great promise for improving the efficiency of training large language models and neural networks in general. Furthermore, low-rank training has the potential to provide valuable insights for the advancement of deep learning theories, aiding our understanding of neural network trainability through gradient descent and their exceptional generalization capabilities in the overparametrized regime.

7 Limitations and Future Work

Scaling beyond 350M Due to limited computational resources, our experiments were constrained to training language models with up to 350M parameters. Nonetheless, ReLoRA already demonstrates promising results at this scale. However, we anticipate its true potential will be realized in the 1B+ parameter region. Additionally, while the 350M model outperforms the Control baseline, it does not continue the trend of narrowing the gap between ReLoRA and full-rank training. We attribute this to suboptimal hyperparameter choice, which requires further investigation.

Furthermore, in 60-350M experiments, even though ReLoRA significantly reduces the number of trainable parameters, we did not observe substantial improvements in memory and computation for the networks of this size. To evaluate the efficiency of our current implementation at a larger scale, we trained the 1.3B-parameter model for a small number of iterations to estimate memory and compute improvements of ReLoRA. At this scale, we observe 30% memory consumption reduction and 52% training throughput increase . We expect to observe even bigger improvements over the full-training baseline for larger networks since the number of trainable parameters for ReLoRA, similar to LoRA, increases at a much slower rate compared to the number of frozen parameters. ReLoRA implementation could be further improved by effectively utilizing gradient checkpointing for ReLoRA layers, custom backward functions, and converting frozen model weights to int8 or int4 quantized format [14].

Comparison to other low-rank training methods A number of approaches to low-rank training have been explored with other model architectures in earlier work [44, 49, 55]. Two aspects set our work apart from these earlier efforts. First, the approach we propose performs high-rank updates through low-rank training. Second, our work demonstrates competitiveness of the low-rank training methods in large-scale transformer language models with 100M+ parameters.

References

A. Aghajanyan, S. Gupta, and L. Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 7319-7328, Online, Aug. 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.568. URL https://aclanthology.org/2021.acl-long.568 .

A. Aghajanyan, L. Yu, A. Conneau, W.-N. Hsu, K. Hambardzumyan, S. Zhang, S. Roller, N. Goyal, O. Levy, and L. Zettlemoyer. Scaling laws for generative mixed-modal language models, 2023.

J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems , 35:23716-23736, 2022.

Z. Allen-Zhu, Y. Li, and Z. Song. A convergence theory for deep learning via over-parameterization. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 242-252. PMLR, 09-15 Jun 2019. URL https://proceedings.mlr.press/v97/allen-zhu19a.html .

M. Belkin, D. J. Hsu, S. Ma, and S. Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences , 116:15849 - 15854, 2018.

S. Bhojanapalli, C. Yun, A. S. Rawat, S. Reddi, and S. Kumar. Low-rank bottleneck in multi-head attention models. In International Conference on Machine Learning , pages 864-873. PMLR, 2020.

S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. Van Den Driessche, J.-B. Lespiau, B. Damoc, A. Clark, D. De Las Casas, A. Guy, J. Menick, R. Ring, T. Hennigan, S. Huang,

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

Maggiore, C. Jones, A. Cassirer, A. Brock, M. Paganini, G. Irving, O. Vinyals, S. Osindero, K. Simonyan,

Rae, E. Elsen, and L. Sifre. Improving language models by retrieving from trillions of tokens. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 2206-2240. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/ borgeaud22a.html .

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 1877-1901. Curran Associates, Inc., 2020. URL https://proceedings. neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf .

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. 2021.

J. Cho, J. Lei, H. Tan, and M. Bansal. Unifying vision-and-language tasks via text generation. In International Conference on Machine Learning , pages 1931-1942. PMLR, 2021.

K. M. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlós, P. Hawkins, J. Q. Davis, A. Mohiuddin, L. Kaiser, D. B. Belanger, L. J. Colwell, and A. Weller. Rethinking attention with performers. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021. URL https://openreview.net/forum?id=Ua6zuk0WRH .

A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. M. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. C. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-Ari, P. Yin, T. Duke, A. Levskaya, S. Ghemawat, S. Dev, H. Michalewski, X. García, V. Misra, K. Robinson, L. Fedus, D. Zhou, D. Ippolito, D. Luan, H. Lim, B. Zoph, A. Spiridonov, R. Sepassi, D. Dohan, S. Agrawal, M. Omernick, A. M. Dai, T. S. Pillai, M. Pellat, A. Lewkowycz, E. Moreira, R. Child, O. Polozov, K. Lee, Z. Zhou, X. Wang, B. Saeta, M. Díaz, O. Firat, M. Catasta, J. Wei, K. S. Meier-Hellstern, D. Eck, J. Dean, S. Petrov, and N. Fiedel. Palm: Scaling language modeling with pathways. ArXiv , abs/2204.02311, 2022.

T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Re. Flashattention: Fast and memory-efficient exact attention with IO-awareness. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=H4DqfPSibmx .

T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer. GPT3.int8(): 8-bit matrix multiplication for transformers at scale. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=dXiGWqBoxaD .

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423 .

W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res. , 23(1), jan 2022. ISSN 1532-4435.

J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum? id=rJl-b3RcF7 .

J. Frankle, G. Karolina Dziugaite, D. M. Roy, and M. Carbin. Stabilizing the lottery ticket hypothesis. arXiv e-prints , pages arXiv-1903, 2019.

B. Ghorbani, O. Firat, M. Freitag, A. Bapna, M. Krikun, X. García, C. Chelba, and C. Cherry. Scaling laws for neural machine translation. ArXiv , abs/2109.07740, 2021.

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. CoRR , abs/1502.01852, 2015. URL http://arxiv.org/abs/1502.01852 .

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2016.

J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, O. Vinyals, J. W. Rae, and L. Sifre. An empirical analysis of compute-optimal large language model training. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/ forum?id=iBBcRUlOAPR .

E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9 .

Y. Idelbayev and M. A. Carreira-Perpinan. Low-rank compression of neural nets: Learning the rank of each layer. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8046-8056, 2020. doi: 10.1109/CVPR42600.2020.00807.

A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems , NIPS'18, page 8580-8589, Red Hook, NY, USA, 2018. Curran Associates Inc.

M. Jaderberg, A. Vedaldi, and A. Zisserman. Speeding up convolutional neural networks with low rank expansions. In Proceedings of the British Machine Vision Conference . BMVA Press, 2014. doi: http://dx.doi.org/10.5244/C.28.88.

J. Kaplan, S. McCandlish, T. J. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. ArXiv , abs/2001.08361, 2020.

U. Khandelwal, O. Levy, D. Jurafsky, L. Zettlemoyer, and M. Lewis. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=HklBjCEKvH .

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. CoRR , abs/1412.6980, 2015.

T. Klug and R. Heckel. Scaling laws for deep learning based image reconstruction. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum? id=op-ceGueqc4 .

O. Kovaleva, A. Romanov, A. Rogers, and A. Rumshisky. Revealing the dark secrets of bert. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 4365-4374, 2019.

A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. Burges, L. Bottou, and K. Weinberger, editors, Advances in Neural Information Processing Systems , volume 25. Curran Associates, Inc., 2012. URL https://proceedings.neurips. cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf .

V. Lialin, V. Deshpande, and A. Rumshisky. Scaling down to scale up: A guide to parameter-efficient fine-tuning, 2023.

R. Lin, C.-Y. Ko, Z. He, C. Chen, Y. Cheng, H. Yu, G. Chesi, and N. Wong. Hotcake: Higher order tucker articulated kernels for deeper cnn compression. In 2020 IEEE 15th International Conference on Solid-State &Integrated Circuit Technology (ICSICT) , pages 1-4, 2020. doi: 10.1109/ICSICT49897.2020.9278257.

J. Lu, C. Clark, R. Zellers, R. Mottaghi, and A. Kembhavi. UNIFIED-IO: A unified model for vision, language, and multi-modal tasks. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=E01k9048soZ .

R. K. mahabadi, J. Henderson, and S. Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , 2021. URL https://openreview.net/forum?id=bqGK5PyI6-N .

P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu. Mixed precision training. In International Conference on Learning Representations , 2018. URL https://openreview.net/forum?id=r1gs9JgRZ .

P. Nakkiran, G. Kaplun, Y. Bansal, T. Yang, B. Barak, and I. Sutskever. Deep double descent: where bigger 368 models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment , 2021, 2019. 369

370

371

A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding by generative pre-training. 2018.

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2019.

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21(140):1-67, 2020. URL http://jmlr.org/papers/v21/20-074.html .

A. Raganato, Y. Scherrer, and J. Tiedemann. Fixed encoder self-attention patterns in transformer-based machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 556-568, 2020.

S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis , pages 1-16, 2020. doi: 10.1109/SC41405.2020.00024.

S. Schotthöfer, E. Zangrando, J. Kusch, G. Ceruti, and F. Tudisco. Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 20051-20063. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_ files/paper/2022/file/7e98b00eeafcdaeb0c5661fb9355be3a-Paper-Conference.pdf .

N. Shazeer. Glu variants improve transformer, 2020.

K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arxiv.org/ abs/1409.1556 .

S. P. Singh, G. Bachmann, and T. Hofmann. Analytic insights into structure and rank of neural network hessian maps. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , 2021. URL https://openreview.net/forum?id=otDgw7LM7Nn .

J. Su, Y . Lu, S. Pan, B. Wen, and Y . Liu. Roformer: Enhanced transformer with rotary position embedding. ArXiv , abs/2104.09864, 2021.

Y. Sui, M. Yin, W. Yang, Y. Gong, J. Xiao, H. Phan, D. Ding, X. Xu, S. Liu, Z. Chen, and B. Yuan. ELRT: Towards efficient low-rank training for compact neural networks, 2023. URL https://openreview. net/forum?id=TC39w69m8bB .

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768 , 2020.

J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research , 2022. ISSN 2835-8856. URL https://openreview.net/forum?id=yzkSU5zdwD . Survey Certification.

J. Wei, X. Wang, D. Schuurmans, M. Bosma, brian ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=_VjQlMeSB_J .

H. Yang, M. Tang, W. Wen, F. Yan, D. Hu, A. Li, H. H. Li, and Y. Chen. Learning low-rank deep neural networks via singular vector orthogonality regularization and singular value sparsification. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pages 2899-2908, 2020.

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research , 2022. ISSN 2835-8856. URL https://openreview.net/forum?id=Ee277P3AYC .

X. Yuan, P. H. P. Savarese, and M. Maire. Growing efficient deep networks by structured continuous sparsification. In International Conference on Learning Representations , 2021. URL https: //openreview.net/forum?id=wb3wxCObbRT .

B. Zhang and R. Sennrich. Root mean square layer normalization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_ files/paper/2019/file/1e8a19426224ca89e83cef47f1e7f53b-Paper.pdf .

C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations , 2017. URL https://openreview.net/forum?id=Sy8gdB9xx .

C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64:107 - 115, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a low-rank way of training LLMs which allows more flexibility than most if not all popular low-rank training methods. It's a parameter-efficient way which only consumes limited gpu usage so it has the potential to be applied to larger models.\n\nSTRENGTHS:\n1. The rationale behind the idea is reasonable. We'd like to have a higher rank models to provide capacity and in the meantime keeping # trainable parameters small.\n\n2. The Introduction reads well.\n\nWEAKNESSES:\nOverall, I like the idea but somehow there are few concerns that I think needs to be improved in order to accept it.\n\n1. It's not well written as many there are many notation unexplained (L85 r), linkage error (e.g., L106 sec 3), and the algorithm is not well explained in a line by line fashion. Many details are skipped, which leads to the problem that reading the code is the only way to fully capture the proposed method.\n\n2. The motivation is not convincing. Rank(A + B) < Rank (A) + Rank(B) doesn't guarantee Ranks(A + B) will be larger than min( Rank(A) + Rank(B)). Without any further constraints, there is no guarantee L93 is true. We just know the upper bound is higher but to claim it, we need to bound lower-bound, which is not discussed at all. The idea itself is interesting but rationale is wrong. \n\n3. Experiments are limited, as shown in Question sections.\n\nQUESTIONS:\n1. A fundamental problem with the setup is that LoRA is used for fine-tune a pre-trained LLM \"for downstream tasks\". Thus, comparing PEFT methods on pre-training performance seems to be meaningless. Authors should compare ReLORA for both pre-train + fine-tuning vs full train + LoRA fine-tune and analyze the ranks or so to make readers understand what's going under the hood. Without this type of analysis (no matter ReLORA performs better or worse), the insights are limited.\n\n2. Following 1, I'd like to see some analysis/experiments on downstream tasks like GLUE/super-GLUE tasks.\n\n3. Jagged Schedule is rather arbitrary. Not sure what's going on if I switch to another dataset, switch models and how it should be combined with fine-tuning. There is no theoretical analysis of the stability, nor experimental experiments to demonstrate it. \n\n4. Another baseline which should be included is training using plain SGD with Jagged Schedule. The above mentioned problems with Adam might totally be redundant given such a complex schedule designed. \n\n5. What's the formal definition of Control? It's not clear how it's performed and frankly speaking, the performance of control is fairly comparable to ReLoRAin Table 2. \n\n6. Selection of r needs to be discussed.\n\nLIMITATIONS:\nIt's discussed in the paper and it reads well.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on low-rank training techniques and introduce ReLoRA that uses low-rank updates to train a high-rank network (<=350M parameters). The main idea is to employ LoRA during training and \"restart\" it in order to artificially increase the rank, which is a nice idea. The difficulty remains in the optimization process due to the gradient after the \"reset\". ReLoRA performs a partial reset of the optimizer state during merge-and-reinit and sets the learning rate to 0 with a subsequent warmup.\n\nThe proposed method is elegant and novel. It is a nice trick to increase artificially the rank of LoRA.\n\nOverall, it is unclear to me what we mean by \"pre-training\". From the text, it seems we are talking about fine-tuning large models with adapters. In the experiment section, the authors talk about training on C4 with a model similar to LLaMA, which indicates pre-training. However, ReLoRA is initialized from a full-rank training at 5k steps. My guess would be that the authors initialize randomly a model (e.g., BERT) and pre-train using an adapter approach.\n\nI am disappointed with the experiment section. While I understand the limitation in resource computation, training models <= 350MB is disappointing when playing in the league of training large neural networks (as suggested by the title) or comparing with LLaMA. Nowadays, 8  GPU days of compute allows to fine-tune a LLaMA or similar model (in 1 day), not to pre-train.\nIn Table 2, it is missing the training time for each model to understand the benefits of ReLoRA over the control, full training, and LoRA. If the method is 2 times slower than Control, it does not necessarily makes sense to use it since the performance at 350M is less than 1ppl (which I expect to be even smaller with larger models).\n\nEvaluating LMs only on perplexity is not sufficient. Please compare pre-trained models on GLUE/etc (as commonly done for adapters/prompting papers) using fine-tuning and PEFT approaches. This will show that:\n- The base model after pre-training with ReLoRA is (maybe?) better than full-rank pre-training or standard pre-training\n- Higher performance can be obtained with ReLoRA during fine-tuning.\n\nmissing references:\n[1] Wang et al. 2023, LEARNING TO GROW PRETRAINED MODELS FOR EFFICIENT TRANSFORMER TRAINING (ICLR)\n\nSTRENGTHS:\n- Nice trick to increase artificially the rank of LoRA\n- The proposed method is sound\n\nWEAKNESSES:\n- the paper writing and clarity must be improved\n- the experiment section is insufficient\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach, ReLoRA, for training large-scale neural networks. Recognizing the limitations of conventional low-rank matrix factorization (LoRA) in training high-performing transformer models, the authors propose ReLoRA that employs a high-rank network training through multiple low-rank updates. This new method uses a full-rank training warm start followed by a merge-and-reinit (restart) strategy, jagged learning rate scheduler, and partial optimizer resets, making it efficient particularly for large networks. The research finds that the efficiency of ReLoRA increases with the network size, positioning it as a potential candidate for efficient training of multi-billion-parameter networks. The paper's results suggest that low-rank training methods can potentially improve the efficiency of training large language models and provide valuable insights for deep learning theories. These insights could further our understanding of neural network trainability and their exceptional generalization capabilities in the overparameterized regime.\n\nSTRENGTHS:\n1. Important work of applying LoRA in pre-train\n2. Good reproducibility: Code is released for readers; Hyperparameter settings are available. But still some parameters are missing, see below.\n3. Sophisticated design of ablation study\n\nWEAKNESSES:\n1. What's the number of trainable parameters in the experiments? Rank r of LoRA is also not reported (or hard to find). I assume 60M, 130M, 250M, and 350M are the total number of parameters.\n2. The perplexity reported for Control with 250M parameters appears to be an outlier, greater than that of 130M parameters. Please double-check. If it’s real, then the fluctuation range of perplexity could be too large for us to draw reliable conclusions with the results.\n3. One important missing baseline is LoRA with warm-start, which is reported in the ablation study, but not in the main results of Table 2, as well as in Figure 3 and 4. As the gain of the main techs developed in the work for ReLoRA can only be seen when compared to the proper baseline of LoRA, unfair to compare with LoRA with no warm-start.\n\nQUESTIONS:\n1. Can authors also report the absolute amount of training resource saved (e.g. total GPU memory, total training wall time for each setting, instead of just 30% memory reduction and 52% training throughput)? Hard to see efficiency of low rank training without these values. Also curious to see how much more cost compared to LoRA with warm start, if there is any.\n\nLIMITATIONS:\n1. Initial warm start is important/indispensable to reach a performance comparable to the full training. If one start from scratch, with no warm started checkpoint, they may still be limited by the computing resource to warm start. The proposed method may just save some time later after warm start, which is, however, not shown in the results.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper proposes an extension LoRA. The main insight in the paper is  LoRa ca be initialization multiple times during training layers and this in the end will produce a high rank update. The authors show that it is quite challenging to re-intialize the layers mostly due to the internal initialization state of Adam.

The authors then propose a pruning based technique to overcome this limitation when using Adam. The also propose a learning rate schedule which helps to avoid the divergence of the network. The authors show that this leads to better performance than Lora on the C4 datasets.

STRENGTHS:
The paper proposes an interesting technique which might be useful. 

The experiments performed by the authors are reasonable and even with limited compute resources they clearly show a clear picture. 

The authors have used the C4 dataset. Which is quite reasonable for a start

WEAKNESSES:
* The methods proposed is more of set of approaches to avoid divergence due to Adam.

* The techniques used a bit of trial and error, is there more principal.

* Would have been better to see what is the downstream performance of ReLoRa transform. 

* Can you provide a better understanding why the second to last row in Table 3 performs very similar to No attemto

QUESTIONS:
Please look at the weakness section.

I am happy to bump up my score to a weak accept if you can do the following two things - 

1. Perform evaluation on downstream task atleast one or two datasets

2. Why is performance so close in Table 3.

LIMITATIONS:
I think the paper is well written. I understand the lack of compute to perform large experiments.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-28