Title: Mamba State-Space Models Can Be Strong Downstream Learners
Venue: NeurIPS 2024
Accepted: no
Keywords: Mamba

state-space models

in-context learning

large language models

LLMs

SSMs
Abstract: Mamba state-space models (SSMs) have recently outperformed state-of-the-art (SOTA) Transformer large language models (LLMs) in various tasks and been widely adapted. However, Mamba’s downstream learning capabilities remain either unexplored–e.g., mixed-precision (MPFT) and parameter-efficient fine-tuning (PEFT)–or under-evaluated–e.g., in-context learning (ICL). For the latter, recent works reported Mamba’s ICL rivals SOTA Transformer LLMs using non-standard benchmarks. In contrast, we show that on standard benchmarks, pretrained Mamba models achieve only 38% of the ICL performance improvements (over zero-shot) of comparable Transformers.

Enabling MPFT and PEFT in Mamba architectures is challenging due to recurrent dynamics and highly customized CUDA kernels, respectively. However, we prove that Mamba’s recurrent dynamics are robust to small input changes using dynamical systems theory. Empirically, we show that performance changes in Mamba’s inference and fine-tuning due to mixed-precision align with Transformer LLMs. Furthermore, we prove that targeting key memory buffers in Mamba’s customized CUDA kernels for low-rank adaptation regularizes SSM parameters, thus achieving parameter efficiency while retaining speedups. We show that combining MPFT and PEFT enables up to 2.15 times more tokens-per-second and 65.5% reduced memory-per-token compared to full Mamba fine-tuning, while achieving up to 81.5% of the ICL performance improvements (over zero-shot) of comparably fine-tuned Transformers.

Abstract

Mamba [22] state-space models (SSMs) have recently outperformed state-of-theart (SOTA) Transformer large language models (LLMs) in various tasks and been widely adapted. However, Mamba's downstream learning capabilities remain either unexplored-e.g., mixed-precision (MPFT) and parameter-efficient fine-tuning (PEFT)-or under-evaluated-e.g., in-context learning (ICL). For the latter, recent works [45, 19] reported Mamba's ICL rivals SOTA Transformer LLMs using nonstandard benchmarks. In contrast, we show that on standard benchmarks, pretrained Mamba models achieve only 38% of the ICL performance improvements (over zero-shot) of comparable Transformers.

Enabling MPFT and PEFT in Mamba architectures is challenging due to recurrent dynamics and highly customized CUDA kernels, respectively. However, we prove that Mamba's recurrent dynamics are robust to small input changes using dynamical systems theory. Empirically, we show that performance changes in Mamba's inference and fine-tuning due to mixed-precision align with Transformer LLMs. Furthermore, we show that targeting key memory buffers in Mamba's customized CUDA kernels for low-rank adaptation regularizes SSM parameters, thus achieving parameter efficiency while retaining speedups. We show that combining MPFT and PEFT enables up to 2.15 times more tokens-per-second and 65.5% reduced pertoken-memory compared to full Mamba fine-tuning, while achieving up to 81.5% of the ICL performance improvements (over zero-shot) of comparably fine-tuned Transformers.

1 Introduction

Innovating on previous state-space models (SSMs) [23, 11], Mamba [22] has been recently proposed as an accurate, sub-quadratic alternative to Transformer large language models (LLMs). Mamba was initially shown to greatly outperform comparable Transformer LLMs [5] across a large number of standard natural language benchmarks. Subsequently, pretrained Mamba models have been widely adapted across different data modalities [42, 65, 36, 46, 37], tasks [60, 62, 48, 63, 57, 37, 2], and architectures [1, 45, 40].

However, despite such rapid and widespread adaptation, evaluation of Mamba's ability to perform standard downstream learning abilities exhibited by Transformer-based LLMs have either not been extensively conducted on standard natural benchmarks or are completely lacking. For instance, while recent works [45, 19, 30] have evaluated Mamba's ability to perform in-context learning (ICL), such studies focused extensively on either non-natural tasks [30, 17] or non-standard benchmarks [25].

Furthermore, evaluation of Mamba's mixed-precision fine-tuning (MPFT) and performance efficient 34 fine-tuning (PEFT) capabilities are currently lacking. For the former, MPFT (and, by extension, 35 mixed-precision inference) are made difficult due to potential sensitivities of Mamba's recurrent 36 dynamics, where [21, 29] suggest full precision ( FP32 ) is required to perform stable training. For 37 the latter, PEFT via standard low-rank adaptation (LoRA) [28] is made difficult within Mamba's 38 SSM layer (referred to herein as the MambaBlock ) due highly customized SSM CUDA kernels which 39 provide competitive performance to attention-based speedups [10] at the cost of standard adapter 40 support. However, PEFT and MPFT are arguably two of the most widely utilized techniques for LLM 41 alignment [53] and customization [55], and are typically combined to drastically decrease hardware 42 demands needed to fine-tune modern LLMs [12]. 43

44

45

46

47

48

49

50

51

52

53

Herein, we extensively explore Mamba's downstream learning capabilities across standard natural benchmarks. For ICL, we show that, in contrast to recent non-standard studies showing Mamba models rival state-of-the-art (SOTA) LLMs of similar parameter counts, the pretrained benefits of Mamba few-shot learning are significantly less than comparable Transformer LLMs across standard natural benchmarks ; averaged across the benchmarks and parameter counts in Table 1, Mamba models only achieve 38% of the performance improvements (relative to zero-shot) of comparable Transformer models from the Pythia suite [5]. However, we show in the sequel that Mamba models can more than halve this gap through efficient fine-tuning , achieving as much as 81.5% of the average few-shot learning improvement (relative to zero-shot) of comparable Transformers.

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

For MPFT, we leverage theory from dynamical systems to show that small input changes in a MambaBlock do not lead to exponentially deviating outputs. Empirically, we validate this theoretical result; compared to full-precision, deviations due to mixed-precision for Mamba inference and fine-tuning are on par with those demonstrated by Transformer LLMs (Section 6). For PEFT, we show that by targeting the largest memory buffer exploited by Mamba's highly customized CUDA kernels, LoRA may be used for extremely efficient fine-tuning, while simultaneously regularizing the majority of Mamba's SSM parameters via weight tying. We show that this leads to extremely efficient PEFT, resulting in up to 2.15 times faster training and 65.5% reduced memory compared to the largest evaluated Mamba model without MPFT or PEFT.

2 Background

Downstream learning for LLMs . Since the release of the Transformer architecture [54], attentionbased LLMs have exhibited several downstream learning abilities-in particular, PEFT, MPFT, and ICL-which allow the rapid adaptation of foundation models towards specific applications. PEFT using adapters [24] allows a large pretrained model to be efficiently adapted for a particular downstream task by freezing the full model and training only a small number of extra parameters. Arguably the most widely used such PEFT method is LoRA [28], which injects trainable low-rank matrices into Transformer layers to approximate weight updates.

To further decrease the computational demands necessary for LLM fine-tuning and inference, MPFT via mixed-precision (i.e., FP16 or BF16 ) [31, 43] and quantized low-precision [12] have proven effective strategies to reduce GPU memory and runtime requirements without deleterious effects on downstream performance [12, 59]. Additionally, mixed-precision approaches have paved the way for hardware-aware optimizations within the self-attention module [10], greatly mitigating the quadratic complexity of Transformer LLMs. Together, PEFT and MPFT have created a rich ecosystem with which varying combinations of these approaches may be used to meet the computational constraints of a given training system. We note that post-fine-tuning quantization approaches [13] may be further used to decrease Transformer LLM computational demands, but such approaches are not considered in this work.

ICL provides an adaptable alternative to fine-tuning. Rather than fine-tune the LLM directly, ICL 81 augments a prompt with n relevant examples (called shots ) preceding the query of interest. Given 82 sufficiently large models and pretraining data [8, 58], Transformer LLMs have proven adept at 83 learning new concepts on the fly provided such few-shot prompting. However, it is worth noting 84 that ICL inference time increases dramatically as the number of shots grows (due to self-attention's 85 quadratic complexity) and PEFT (when possible) is known to produce more accurate downstream 86 learning results [8, 41]. 87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

State-space Models . Structured state-space sequence (S4) models [23, 14] are SSMs which leverage linear time-invariant (LTI) systems to combine the computational advantages of Transformers-i.e., highly parallelizable training-and recurrent neural networks (RNNs)-i.e., subquadratic autoregressive inference using recurrency. Within the S4 layer, an input signal is discretized and LTI parameters representing the input's latent dynamics are learned. Owing to the S4 block's latent dynamics being LTI, the S4 block's output may be thus compactly represented as a single convolution between the input and an SSM convolution kernel (a matrix whose entries are products of LTI learnable parameters resulting from unrolling the state-space equations). However, despite hardware efficiency and long-dependency-modeling improvements, LTI-based S4 models remained inferior to Transformers of comparable parameter-sizes for natural language tasks, even when augmenting S4 layers with attention-layers for hybrid architectures [22].

Innovating on these previous S4 approaches, Mamba utilizes timevarying parameters to model latent dynamics, thus broadening the ability to capture nuanced changes evolving in discrete-time. Without LTI dynamics, however, the input-output representation via the SSM convolution kernel is no longer applicable, thus voiding previous hardware-aware S4 optimizations [14]. To enable hardware efficiency with time-varying SSM parameters, [22] thus introduced extensively customized CUDA kernels which implement highly parallelized prefix sums to compute recurrent states.

3 Mamba state-space models

For model dimension d and maximum input sequence length T , the MambaBlock defines state-space parameters A , B t , C t , ∆ t P R d ˆ d for t P t 1 , . . . , T u . The matrix ∆ t controls the discrete stepsize. Given an input sequence u 1 , . . . , u T P R d , the following linear mapping through latent states x 1 , . . . , x T P R d is used to produce the output y 1 , . . . , y T P R d :

where ¯ ∆ t ' softplus p Linear p ∆ t qq P R d ˆ d , ¯ A t ' exp p ¯ ∆ t A q and ¯ B t ' A ´ 1 p ¯ A ´ I q B t . In practice, A , B t , C t and ∆ t are diagonal matrices.

Hardware-aware optimizations . As matrices B t , C t and ∆ t are time-varying, S4 optimizations via the SSM convolution kernel [11] are no longer applicable. However, by diagonality, each dimension may be computed in parallel. Furthermore, the recurrence along every dimension is a prefix sum (also called a scan ), which is highly parallelizable [7]. [15] thus capitalizes on this through extensively customized CUDA kernels wherein the majority of temporal variables are carefully laid out in a large buffer of GPU memory and manipulated. Instantiated as a PyTorch linear layer's weight matrix, this memory buffer W P R n ˆ 3 d is used to store and access the diagonal elements of B t , C t and ∆ t for all t P t 1 , . . . , T u , such that

The customized Mamba prefix scan kernel heavily relies on this memory layout to optimize the access pattern of W in Equations 5 and 6.We note that, rather than adjusting Mamba's low-level CUDA kernels themselves to integrate LoRA within the highly optimized prefix scan, we can instead directly target W . Doing so, we have the following, where the proof is available in Appendix A.

Theorem 1. Consider the weight matrix W of a MambaBlock from Equation 3. Targeting W for LoRA during fine-tuning ties adaptation weights across B t , C t and ∆ t .

4 Stable dynamics in the MambaBlock

The Mamba foundation models were pretrained in full FP32 precision. Consequently, official Mamba implementations have cautioned against fine-tuning or training in reduced precision [21, 29], with potential sensitivities of MambaBlock recurrent dynamics remaining an open question. We answer the latter using theory from dynamical systems. For Mamba's discrete dynamic system in Equations 5 and 6, define

where θ denotes the time-varying parameters described in Section 3. For input sequence u 1 , . . . , u T 133 and initial latent state vector x 0 , we thus write 134

The rate of divergence between two scalar ε -close inputs to a discrete dynamical system is bounded by the system's maximal Lyapunov exponent λ max [44]. Given λ max and two initial values p x 0 , u 1 q and p x 0 ` ε, u 1 ` ε q , the maximum deviation between these points grows as [33, 50]:

Thus, when λ max ą 0 , nearby trajectories exponentially separate and, when λ max ď 0 , nearby trajectories ultimately converge to the same fixed point or periodic cycles.

The maximal Lyapunov exponent is defined as

where }} 2 denotes the spectral norm for matrices. For an arbitrary MambaBlock , we prove the following:

Theorem 2. Let p x t ´ 1 , u t q be the latent state and input at an arbitrary time t P t 1 , . . . , T u within a MambaBlock . Then small changes p x t ´ 1 ` ε, u t ` ε q produce deviations which are exponentially nonincreasing over discrete-time. That is, max | F N θ p x t ´ 1 , u t q´ F N θ p x t ´ 1 ` ε, u t ` ε q| P O p ε exp p Nζ qq , for some scalar ζ ď 0 .

The proof of Theorem 2 is available in Appendix B, where the maximal Lyapunov exponent for an arbitrary MambaBlock is first proven to be non-positive. The main result subsequently follows.

Consequences for automatic mixed-precision . During a forward pass, automatic mixed-precision (AMP) saves time and memory by computing forward activations in half-precision ( FP16 or BF16 ). During a backward pass, AMP computes gradients in half-precision and up-casts to full-precision prior to updating. In contrast to full-precision fine-tuning, MPFT within the MambaBlock thus results in small differences to the inputs u 1 , . . . , u T fed into the SSM scan (which are passed through a SwiGLU ), ¯ ∆ t (which is passed through a softplus ), and the gradients calculated during training.

For a discrete dynamical system with λ max ą 0 , changes due to AMP compound after repeated expansion of the recurrent state, thus leading to exponential deviations between quantities calculated using mixed- versus full-precision. We note that Transformers are not recurrent, and thus not susceptible to such issues. Yet, just as differences introduced by quantization/mixed-precision produce output differences in Transformer results, differences are expected in Mamba results using different precision strategies. However, by Theorem 2, such differences do not exponentially compound over discrete-time within the MambaBlock .

5 Related Work

Several recent works [45, 19, 30, 40] have studied Mamba's ability to perform ICL. However, none of these have extensively studied Mamba's ICL capabilities either on standard NLP benchmarks or on pure MambaBlock foundation models. In particular, foundational Mamba models' ICL abilities were tested in [45] to learn simple function classes (e.g., logistic regression and decision trees [17]) and in [19] to learn non-standard NLP benchmarks (i.e., task vectors [25]). While [45, 19] report Mamba's ICL abilities rival SOTA Transformers, their utilized benchmarks were proposed as supplemental ICL studies after Transformer LLMs' success on standard NLP benchmarks [8]. Indeed, direct evaluation of Mamba foundation models on standard NLP benchmarks does not lead to higher gains over zero-shot performance relative to comparable Transformer LLMs (demonstrated in Table 1).

Lyapunov exponents have previously been considered for classic RNN structures (e.g., vanilla RNNs, LSTMs, GRUs, PLRNNs, etc.) [44, 56], to determine when such models exhibit chaotic dynamics and the impact on the exploding/vanishing gradient phenomena * . For more recent S4 neural

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

models, [18] used Hurwitz matrices to characterize the numerical stability of linear time-invariant 175 (LTI) S4 models. However, such analysis is not applicable to time-varying models, such as Mamba, 176 nor does it characterize the effects of sensitive dependence on initial conditions (e.g., divergence of 177 two ε close inputs). To the best of our knowledge, no previous works have used Lyapunov exponents 178 to explore the effects of mixed-precision on recurrent neural models or Mamba architectures. 179

180

181

182

183

184

185

186

187

188

As in [22], the majority of subsequent Mamba works have focused on pretraining MambaBlock s using full precision [65, 62, 1, 40]. Notably, the official implementation of Jamba [40], the TransformerMamba hybrid, supports mixed- and 8-bit precision, but avoids MambaBlock s when applying such quantization [32]. Similarly, the official Mamba sources advise using full precision within the MambaBlock [29, 21], cautioning against using mixed-precision due to potential recurrent sensitivities. To the best of our knowledge, no existing works have either theoretically explored the effects small input changes (e.g., due to mixed-precision) have on Mamba's recurrent dynamics, empirically explored such effects downstream impact on fine-tuning and inference, or explored pure Mamba networks fine-tuning abilities relative to Transformer LLMs.

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

6 Experiments

To demonstrate the implications of Theorem 2, we explore the performance difference between running inference with full-precision pretrained weights and using mixed-precision ( FP16 and BF16 ) weights. Model performance is measured as percent accuracy using the MMLU [26] dataset. The difference in model performance is reported as the mean divergence (i.e., absolute difference) between the original full-precision and respective mixed-precision model, averaged over {0, 1, 3, 5}-shot percent accuracy. Thus, a divergence greater than one denotes an average difference greater than one entire percentage of accuracy.

Mamba pretrained checkpoints are compared to pretrained Transformer models of similar parameter counts and no more than ' 300B total pretraining tokens (Pythia [5], OLMo [20] 336B-token checkpoint, and Phi 1.5 [39]). We note that Pythia and Mamba models were both pretrained using the same corpus [15], allowing the fairest comparison between SSMs and Transformers. To limit extraneous numerical effects within experiments (e.g., due to parameter aggregation across multiple GPUs), all models were run using a single GPU (Nvidia A10G, 24 GB total memory). All models were evaluated using the LM evaluation harness from Eleuther AI [16]. Further experimental details are available in Appendix C. The results are available in Table 2.

From Table 2, inferencing in Pythia using FP16 and BF16 result in an average 0.13 and 0.41 fullprecision divergence, respectively. Mamba displays similar averages in comparison: inferencing in Mamba using FP16 and BF16 result in an average 0.10 and 0.48 divergence, respectively. Interestingly, both SSM and Transformer architectures exhibit large divergence spikes -i.e., mean divergence greater than a percentage point-when using BF16 , which occurs once for Mamba and Phi 1.5 models and twice for Pythia models. In the following, we show that such spikes may be mitigated for Mamba SSMs by combining mixed-precision with parameter-efficient adapters during fine-tuning.

Non-divergent Mamba fine-tuning. We next explore the implications of Theorem 2 on fine-tuning, wherein mixed-precision is especially critical; MPFT combined with PEFT adapters have been shown to drastically reduce Transformer fine-tuning times [12]. We are thus interested in the divergence between Mamba models fully fine-tuned (i.e., no adapters, all model weights are trained) in fullprecision and models fine-tuned using mixed-precision and/or PEFT adapters. We focus on utilizing LoRA [28], which is arguably the most widely used PEFT framework for LLMs.

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

Mamba 130M

Mamba 370M

Pythia 410M

Mamba 790M

Pythia 1B

Model

0.0

0.5

1.0

1.5

Mean Per-shot FP32 Divergence

FP16 Full

BF16 Full

FP16 ALL LoRA

BF16 ALL LoRA

FP16 SLL LoRA

BF16 SLL LoRA

Using the Alpaca dataset [51], Mamba 160M , 410M , and 790M models are fine-tuned for three epochs with a maximum sequence length of 512. We denote the targeting of all linear layers (ALL) for LoRA as ALL LoRA , the targeting of a subset of linear layers (SLL) for LoRA as SLL LoRA , and no adapters as Full (i.e., full fine-tuning). Both ALL and SLL LoRA adapt the large memory buffer described in Theorem 1.

Each fine-tuning run occurred on a single A10G GPU. To further limit extraneous numerical effects, the same batch size is used for all FP32 , FP16 , and BF16 experiments for a given model size. While this leads to hardware underutilization (i.e., non-saturated GPU memory for mixed-precision and LoRA experiments), this is necessary to guarantee no divergence is due to differences in parameter update schedules. For comparison, Pythia 160M , 410M , and 1B models are fine-tuned using the same experimental setup. The training recipe for all models was adapted from [53], with the AdamW_torch optimizer and a cosine annealing schedule. Further experimental details are available in Appendix C.

For each Mamba and Pythia model, Figure 1 shows the mean divergence calculated between the respective FP32 Full and mixed-precision ALL / SLL LoRA fine-tuned models, averaged over {0, 1, 3, 5}-shot MMLU accuracy. Across mixed-precisions and adapter settings, Mamba displays comparable divergences to Pythia models. E.g., for FP16 , Mamba demonstrates an average divergence of 0.1, compared to 0.14 for Pythia . Similarly, for BF16 , Mamba demonstrates an average divergence of 0.18, compared to 0.28 for Pythia . Importantly, Mamba models do not exhibit large deviation spikes after fine-tuning (in contrast to Pythia models).

Hardware throughput and memory-utilization improvements . With comparable divergences to Transformers and stable dynamics, we show that MPFT and PEFT may be used to significantly increase GPU-training throughput for Mamba SSMs. To demonstrate such improvements, we utilize the previous fine-tuning settings for the Alpaca dataset. However, we now adjust the batch size to maximize throughput per MPFT and PEFT configuration.

For each MPFT and PEFT configuration, the average tokens-per-second (ATPS) is calculated as the total tokens used for fine-tuning divided by total training time, and the maximum memory-per-token (MMPT) is calculated as the maximum GPU memory utilization incurred (over the entire fine-tuning run) divided by the total number of tokens in each mini-batch. Results are plotted in Figure 6.

Both throughput and memory utilization improve as the number of Mamba parameters increases in Figure 6. Compared to the full-precision full fine-tuning of Mamba 790M (the largest model supported by an A10G 's memory capacity), evaluated MPFT and PEFT combinations result in an average 2.15 times more training tokens-per-second while reducing per-token memory utilization by an average 62.7% . Across all model sizes, evaluated MPFT and PEFT combinations result in an average 1.74 times more training tokens-per-second while reducing per-token memory utilization by an average 47.2% compared to respective full-precision fine-tuned runs.

6.1 Fine-tuning narrows the ICL gap between Mamba and Transformers

We next explore how MPFT and PEFT affect Mamba ICL performance. All Mamba pretrained models are instruction fine-tuned using ALL LoRA and the OpenHermes dataset [52] (which consists of 242,000 supervised samples). We use the training recipe of [53], which includes BF16 utilization.

261

262

263

264

265

266

267

268

269

Figure 3 displays AIPSS for pretrained and instruction fine-tuned Mamba and Pythia models. As previously noted, pretrained Mamba models do not display similar ICL ability as comparable Pythia models on the evaluated standard NLP benchmarks. In particular, Mamba 2.8B , the largest pretrained Mamba model, displays inconsistent zero-shot improvements as the number of shots increase. However, after fine-tuning, all Mamba models larger than Mamba 130M consistently improve in ICL performance as the number of shots increase. Compared to Mamba pretrained models, which are only capable of 38% of the AIPSS compared to similar pretrained Pythia models, fine-tuned ALL LoRA Mamba models are capable of 81.5% of the AIPSS compared to similarly fine-tuned Pythia models.

Fine-tuning robustness . We show that Mamba is robust to the choice of PEFT hyperparemters. We 270 conduct an extensive hyperparameter search across the learning rate, LoRA dimension, and number of 271 warmup steps. From the Cartesian-product of these three parameters, 150 hyperparameter configura272 tions were sampled and used to fine-tune Mamba 370M over the Openhermes dataset. For comparison, 273 Pythia 410M is similarly fine-tuned using the same set of 150 hyperparameter configurations. 274

Mamba 130m

Mamba 370m

Mamba 790m

Model

0

1

2

3

Tokens/second

1e4

fp32 Full

fp16 ALL LoRA

bf16 ALL LoRA

fp16 SLL LoRA

bf16 SLL LoRA

(a) Average tokens-per-second

Ò

.

Mamba 130m

Mamba 370m

Mamba 790m

Model

0

2

4

6

8

fp32 Full

fp16 ALL LoRA

bf16 ALL LoRA

fp16 SLL LoRA

bf16 SLL LoRA

(b) Maximum memory-per-token

Ó

.

Performance is evaluated using the datasets from Table 1-HellaSwag [64], PIQA [6], Arc-E [9], 258 Arc-C [9], and WinoGrande [49]-and report the average improvement percentage of {1, 3, 5}shot 259 versus 0shot (AIPSS). For comparison, Pythia pretrained models are instruction fine-tuned using the 260 same training recipe and ALL LoRA (i.e., all Pythia linear layers are adapted).

0-Shot

1-Shot

3-Shot

5-Shot

0

2

4

% 0-shot ICL Improvement

2.8B

1.4B

790M

370M

130M

(a) Mamba Pretrained

0-Shot

1-Shot

3-Shot

5-Shot

0

1

2

3

4

% 0-shot ICL Improvement

2.8B

1.4B

1B

410M

160M

(b) Pythia Pretrained

0-Shot

1-Shot

3-Shot

5-Shot

0

2

4

% 0-shot ICL Improvement

2.8B

1.4B

790M

370M

130M

(c) Mamba

ALL

LoRA

0-Shot

1-Shot

3-Shot

5-Shot

0

2

4

% 0-shot ICL Improvement

2.8B

1.4B

1B

410M

160M

(d) Pythia

ALL

LoRA

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

10

7

10

6

10

5

10

4

10

3

Learning Rate

0.24

0.26

MMLU 5-Shot Acc.

16

32

64

128

256

0

1000

2000

(a)

Mamba

370M

10

7

10

6

10

5

10

4

10

3

Learning Rate

0.24

0.26

MMLU 5-Shot Acc

16

32

64

128

256

0

1000

2000

(b)

Pythia

410M

The MMLU 5-shot performance for each of the 150 Mamba and Pythia fine-tuned models is displayed in 6.1. Pythia 410M is capable of higher performance than Mamba 370M , where the average accuracy for the former and the latter are 26.5% and 24.8%, respectively. However, Mamba 370M is much more robust to the choice of hyperparameters, with a difference of 1.5% between the minimum (23.3%) and maximum (24.8%). In contrast, Pythia 410M fine-tuned models display a large performance difference of 4.7% between the minimum (22.9%) and maximum (27.6%).

7 Discussion

We've extensively explored Mamba's downstream learning capabilities. Using dynamical systems theory, we've shown that Mamba's recurrent dynamics are robust to small input perturbations (contrary to the current understanding of Mamba's recurrent sensitivities). We've extensively confirmed this result, showing that: a) Mamba inference is robust to changes due to mixed-precision, (b) Mamba inference differences due to mixed-precision align with Transformers, (c) Mamba fine-tuning is robust to changes due to mixed-precision and PEFT, and (d) differences in downstream performance for Mamba due to MPFT and PEFT can be more robust than Transformers. Using both MPFT and PEFT, we've shown that instruction fine-tuning Mamba SSMs greatly narrows the previously observed ICL gap, going from only 38% (post pretraining) up to 81.5% (post fine-tuning) of the ICL abilities of similar Transformers. Furthermore, we've shown that combining MPFT and PEFT can more than halve training time and nearly triple memory efficiency for Mamba models.

There are significant avenues for future work. In particular, adapting Mamba's CUDA kernels to support more aggressive low-precision PEFT methods [12] would further decrease the hardware needed to train Mamba models, while providing additional speedups. Furthermore, while the largest pure Mamba model contains 2.8B parameters, the training speedups and improved memory utilization described herein may be applied to more efficiently pretrain larger pure Mamba SSMs (e.g., 7B parameters and greater), where Mamba models may better manifest emergent abilities previously displayed by Transformers (or even manifest previously unobserved abilities).

Limitations. While we explored the use of LoRA for Mamba models, many other PEFT adapters exist [41, 38, 27, 35]. Furthermore, while mixed-precision using FP16 and BF16 were explored, lower-precision methods exist [12] (which may be enabled by adapting Mamba's highly customized CUDA kernels). Both are interesting directions for future work. Finally, our timing and memory usage experiments using Alpaca did not consider the largest two Mamba models (1.4B and 2.8B) due to their exceeding A10G memory capacity for FP32 full fine-tuning.

Broader Impact. The Mamba models considered are all LLMs, and thus have the same potential positive and negative societal impacts as other LLMs (e.g., hallucinations). Furthermore, fine-tuning is known to possibly erode existing LLM guardrails, and thus our methods may be adapted for this fine-tuning use case (as is the case for all PEFT and MPFT methods). However, our work improves the quality of Mamba models for downstream applications, which may be adapted for all positive LLM applications in society (e.g., personal assistants, task automation, code completion, etc.). Finally, our work decreases the computational constraints required to train and inference Mamba SSMs, which has implications for green ML (e.g., decreased CO2 emissions, positive climate change impact, etc.). 410 GPU days were used to produce the results for this paper.

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

References

Quentin Anthony, Yury Tokpanov, Paolo Glorioso, and Beren Millidge. Blackmamba: Mixture of experts for state-space models. arXiv preprint arXiv:2402.01771 , 2024.

Ali Behrouz and Farnoosh Hashemi. Graph mamba: Towards learning on graphs with state space models. arXiv preprint arXiv:2402.08678 , 2024.

Nils Bertschinger and Thomas Natschläger. Real-time computation at the edge of chaos in recurrent neural networks. Neural computation , 16(7):1413-1436, 2004.

Nils Bertschinger, Thomas Natschläger, and Robert Legenstein. At the edge of chaos: Real-time computations and self-organized criticality in recurrent neural networks. Advances in neural information processing systems , 17, 2004.

Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning (ICML) , pages 2397-2430. PMLR, 2023.

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 7432-7439, 2020.

Guy E Blelloch. Prefix sums and their applications. 1990.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 , 2018.

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems , 35:16344-16359, 2022.

Tri Dao, Daniel Y Fu, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. In Proceedings of the 11th International Conference on Learning Representations (ICLR) , 2023.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems , 36, 2024.

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. 2023.

Daniel Y Fu, Tri Dao, Khaled Kamal Saab, Armin W Thomas, Atri Rudra, and Christopher Re. Hungry hungry hippos: Towards language modeling with state space models. In International Conference on Learning Representations (ICLR) , 2023.

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 , 2020.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023.

Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn incontext? a case study of simple function classes. Advances in Neural Information Processing Systems (NeurIPS) , 35:30583-30598, 2022.

Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré. It's raw! audio generation with state-space models. In International Conference on Machine Learning , pages 7616-7633. PMLR, 2022.

Riccardo Grazzi, Julien Siems, Simon Schrodi, Thomas Brox, and Frank Hutter. Is mamba capable of 365 in-context learning? arXiv preprint arXiv:2402.03170 , 2024. 366

Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh 367 Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language 368 models. arXiv preprint arXiv:2402.00838 , 2024. 369

370

371

Albert Gu and Tri Dao. Mamba Precision Guidance . " https://github.com/state-spaces/mamba# precision ", 2023. "Accessed: 2024-04-25".

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint 372 arXiv:2312.00752 , 2023. 373

Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state 374 spaces. In International Conference on Learning Representations (ICLR) , 2022. 375

Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified 376 view of parameter-efficient transfer learning. In International Conference on Learning Representations 377 (ICLR) , 2021. 378

379

380

Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. In The 2023 Conference on Empirical Methods in Natural Language Processing , 2023.

381

382

383

384

385

386

387

388

389

390

391

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations , 2020.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning , pages 2790-2799. PMLR, 2019.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

Huggingface. Mamba PEFT . " https://huggingface.co/docs/transformers/en/model_doc/ mamba#peft-finetuning ", 2024. "Accessed: 2024-04-25".

Samy Jelassi, David Brandfonbrener, Sham M Kakade, and Eran Malach. Repeat after me: Transformers 392 are better than state space models at copying. arXiv preprint arXiv:2402.01032 , 2024. 393

Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth 394 Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, et al. A study of 395 bfloat16 for deep learning training. arXiv preprint arXiv:1905.12322 , 2019. 396

AI 21 Labs. Jamba PEFT . " https://huggingface.co/ai21labs/Jamba-v0.1 ", 2024. "Accessed: 397 2024-04-25". 398

Tanguy Laffargue, Khanh-Dang Nguyen Thu Lam, Jorge Kurchan, and Julien Tailleur. Large deviations of 399 lyapunov exponents. Journal of Physics A: Mathematical and Theoretical , 46(25):254002, 2013. 400

Thomas Laurent and James von Brecht. A recurrent neural network without chaos. In Proceedings of the 401 11th International Conference on Learning Representations (ICLR) , 2017. 402

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. 403 arXiv preprint arXiv:2104.08691 , 2021. 404

Kai Li and Guo Chen. Spmamba: State-space model is all you need in speech separation. arXiv preprint 405 arXiv:2404.02063 , 2024. 406

Lincan Li, Hanchen Wang, Wenjie Zhang, and Adelle Coster. Stg-mamba: Spatial-temporal graph learning 407 via selective state space model. arXiv preprint arXiv:2403.12418 , 2024. 408

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv 409 preprint arXiv:2101.00190 , 2021. 410

Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 411

Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463 , 2023. 412

Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked 413 Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer-mamba language 414 model. arXiv preprint arXiv:2403.19887 , 2024. 415

416

417

418

Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems , 35:1950-1965, 2022.

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint arXiv:2401.10166 , 2024.

Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. In International Conference on Learning Representations (ICLR) , 2018.

Jonas Mikhaeil, Zahra Monfared, and Daniel Durstewitz. On the difficulty of learning chaotic dynamics with rnns. Advances in Neural Information Processing Systems , 35:11297-11312, 2022.

Jongho Park, Jaeseung Park, Zheyang Xiong, Nayoung Lee, Jaewoong Cho, Samet Oymak, Kangwook Lee, and Dimitris Papailiopoulos. Can mamba learn how to learn? a comparative study on in-context learning tasks. International Conference on Machine Learning (ICML) , 2024.

Changsheng Quan and Xiaofei Li. Multichannel long-term streaming neural speech enhancement for static and moving speakers. arXiv preprint arXiv:2403.07675 , 2024.

Antônio H Ribeiro, Koen Tiels, Luis A Aguirre, and Thomas Schön. Beyond exploding and vanishing gradients: analysing rnn training using attractors and smoothness. In International conference on artificial intelligence and statistics (AISTATS) , pages 2370-2380. PMLR, 2020.

Jiacheng Ruan and Suncheng Xiang. Vm-unet: Vision mamba unet for medical image segmentation. arXiv preprint arXiv:2402.02491 , 2024.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM , 64(9):99-106, 2021.

Hiroki Sayama. Introduction to the modeling and analysis of complex systems . Open SUNY Textbooks, 2015.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github. com/tatsu-lab/stanford_alpaca , 2023.

Teknium. Openhermes . " https://huggingface.co/datasets/teknium/openhermes ", 2024. "Accessed: 2024-04-25".

Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944 , 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Kushala VM, Harikrishna Warrier, Yogesh Gupta, et al. Fine tuning llm for enterprise: Practical guidelines and recommendations. arXiv preprint arXiv:2404.10779 , 2024.

Ryan Vogt, Maximilian Puelma Touzel, Eli Shlizerman, and Guillaume Lajoie. On lyapunov exponents for rnns: Understanding information propagation using dynamical systems tools. Frontiers in Applied Mathematics and Statistics , 8:818799, 2022.

Chloe Wang, Oleksii Tsepa, Jun Ma, and Bo Wang. Graph-mamba: Towards long-range graph sequence modeling with selective state spaces. arXiv preprint arXiv:2402.00789 , 2024.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research , 2022.

Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev, and Paulius Micikevicius. Integer quantization for 461 deep learning inference: Principles and empirical evaluation. arXiv preprint arXiv:2004.09602 , 2020. 462

Jianhao Xie, Ruofan Liao, Ziang Zhang, Sida Yi, Yuesheng Zhu, and Guibo Luo. Promamba: Prompt463 mamba for polyp segmentation. arXiv preprint arXiv:2403.13660 , 2024. 464

Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning 465 as implicit bayesian inference. In International Conference on Learning Representations (ICLR) , 2021. 466

467

468

Zhaohu Xing, Tian Ye, Yijun Yang, Guang Liu, and Lei Zhu. Segmamba: Long-range sequential modeling mamba for 3d medical image segmentation. arXiv preprint arXiv:2401.13560 , 2024.

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

Recall the form of W :

where W r 0 , : d s ' diag p ∆ 1 q , W r n ´ 1 , d : 2 d s ' diag p B T q , and so on. For index r t ´ 1 , j s , we thus have

Thus, the weights U r t ´ 1 , : s are tied for any parameter ˜ W r t ´ 1 , j s , j P t 1 , . . . , 3 d u , which are used 493 to adapt parameters ∆ 1 , B t , and C t . 494

495

Yijun Yang, Zhaohu Xing, and Lei Zhu. Vivim: a video vision mamba for medical video object segmentation. arXiv preprint arXiv:2401.14168 , 2024.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 4791-4800, 2019.

Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. International Conference on Machine Learning (ICML) , 2024.

A Proof of weight-tying using LoRA in the MambaBlock

Due to the low-level nature of Mamba's prefix scan optimizations (discussed in Section 3), standard use of LoRA adapters is made difficult within Mamba's SSM-layer. E.g., while B t , C t and ∆ t are conceptually PyTorch linear layers, their bundling in a contiguous memory block and careful manipulation makes appending a LoRA adapter on any of these invidiual matrices non-trivial (particularly, while respecting the highly specialized layout of each LoRA adapters targeted layer). However, we note that the overall design of the MambaBlock 's hardware optimizations may be leveraged to both efficiently learn the parameter-space for the majority of time-varying parameters (thus achieving PEFT) and regularize parameters during training (thus improving fine-tuning generalization).

Theorem 1. Consider the weight matrix W of a MambaBlock from Equation 3. Targeting W for LoRA during fine-tuning ties adaptation weights across B t , C t and ∆ t .

Proof. Let r be the specified LoRA dimension. Targeting this matrix for LoRA results in the adapter

where U P R n ˆ r , V P R r ˆ 3 d , and W is frozen during fine-tuning. Thus, for index r i, j s ,

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

B Mamba stable dynamics proof

Recall the state-space parameters and equations for the MambaBlock ; A , B t , C t , ∆ t P R d ˆ d for t P t 1 , . . . , n u ' r n s . Given an input sequence u 1 , . . . , u n P R d , the following linear mapping through latent states x 1 , . . . , x n P R d is used to produce the output y 1 , . . . , y n P R d :

where ¯ ∆ t ' softplus p Linear p ∆ t qq P d ˆ d , ¯ A t ' exp p ¯ ∆ t A q , ¯ B t ' A ´ 1 p ¯ A ´ I q B t , and is the set of non-negative real numbers. In practice, A , B t , C t and ∆ t are diagonal matrices.

Furthermore, recall the following definitions:

where θ denotes the aforementioned time-varying parameters. For input sequence u t , . . . , u T and initial latent state value x 0 , we thus write

We first prove that, given two scalar ε -close inputs to a MambaBlock , their deviations do not grow exponentially as the number of recurrences increases (Lemma 1). The main result in the paper is subsequently proved.

Lemma 1. For input p x 0 , u 1 q to a MambaBlock , small changes p x 0 ` ε, u 1 ` ε q produce deviations which are exponentially non-increasing over discrete-time. That is, max | F N θ p x 0 , u 1 q ´ F N θ p x 0 ` ε, u 1 ` ε q| P O p ε exp p Nζ qq , for some scalar ζ ď 0 .

Proof. Firstly, we note that within the MambaBlock , A is stored in log-space followed by a negative exponentiation prior to use. Thus, A P d ˆ d , where is the set of non-positive real numbers.

Recall that for the maximum deviation, we have:

where the maximal Lyapunov exponent λ max is defined as:

and }} 2 denotes the spectral norm for matrices.

Thus, to complete the proof, it suffices to show that λ max ď 0 . Recall that A and ¯ ∆ t are diagonal. From Equation 5, we thus have

Let i be the dimension which corresponds to the output of the spectral norm, i.e., i ' 518 argmax j ' 1 ,...,d t exp ř T t ' 0 p ¯ ∆ t r j, j s A r j, j squ . We thus have 519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

A r i, i s is non-positive and lim T Ñ8 1 T ř T t ' 0 ¯ ∆ t r i, i s ě 0 , since ¯ ∆ t r i, i s P @ t . Thus, λ max ď 0 .

Theorem 2. Let p x t ´ 1 , u t q be the latent state and input at an arbitrary time t P r 1 , T s within a MambaBlock . Then small changes p x t ´ 1 ` ε, u t ` ε q produce deviations which are exponentially decreasing over discrete-time, i.e., max | F N θ p x 0 , u 1 q ´ F N θ p x 0 ` ε, u 1 ` ε q| P O p ε exp p Nζ qq , for some scalar ζ ď 0 .

Proof. Let τ p t q be a function that maps time values such that τ p t q P r 1 , T ´ t s and τ p t q ' 1 , τ p t ` 1 q ' 2 , . . . , τ p t ` T q ' T ´ t . Then B τ p t q , C τ p t q , ∆ τ p t q define a new MambaBlock with inputs u τ p t q , . . . , u τ p t ` T q and subsequent recurrent states x τ p t q , . . . , x τ p t ` T q . Applying Lemma 1 to this MambaBlock with p x τ p t q´ 1 , u τ p t q q completes the proof.

C Experimental Details

All model checkpoints were evaluated on all benchmarks and few-shot settings using the LM evaluation harness from Eleuther AI [16], version 0.4.2 . Pythia and Mamba Huggingface checkpoints were used for all inference and fine-tuning experiments, e.g., EleutherAI/pythia-160m and state-spaces/mamba-130m-hf for the smallest respective models. All fine-tuning experiments were run using package versions Transformers 4.40.0.dev0 , Accelerate 0.28.0 , TRL 0.8.1 , PyTorch 2.2.1+cu121 , and PEFT 0.10.0 .

For MPFT, Flash Attention 2.0 [10] via flash_attn 2.5.7 was used for Pythia models. For FP16 and BF16 inference results, Flash Attention 2.0 was used for both Pythia and OLMo models. For OLMo results, the 336B-token checkpoint was used by specifying revision=step80000-tokens336B .

Outside of the OpenHermes hyperparameter search, all Alpaca and OpenHermes fine-tuning experiments used the following training recipe (adapted from [53]): AdamW_torch optimizer, cosine annealing schedule, no gradient accumulation, maximum norm of 1.0 for gradient clipping, and no warmup steps. Training epochs used for all Alpaca and OpenHermes experiments were three and one, respectively. For both Pythia and Mamba models, the learning rate and LoRA dimension r were scaled to improve performance of smaller models (per-model values listed in Table 3).

For SLL LoRA , targeted Mamba layers were {x_proj, embeddings, in_proj, out_proj} ; x_proj is the large MambaBlock memory buffer which, when targeted by LoRA, regularizes the majority of SSM parameters during fine-tuning through weight tying (Theorem 1). Pythia targeted SLL LoRA layers were {dense, embed_in, query_key_value, dense_h_to_4h,dense_4h_to_h} , chosen to balance performance across model sizes.

All experiments in Tables 1 and 2, Figures 1 and 6 were run using a signle-GPU Nvidia A10G (24 GB total memory). For Pythia and Mamba ALL LoRA experiments in Figure 3, all experiments were run on an A10G, except for Mamba 2.8B , which exceeded A10G memory capacity and was run on an Nvidia H100 (80 GB total memory).

For the hyperparameter search results in Figure 6.1, all experiments were run using 8 H100 GPUs. SLL LoRA was used for Mamba and Pythia models. The range of hyperparameter values was as follows:

learning rate P t 1 e ´ 7 , 2 e ´ 7 , 5 e ´ 7 , 1 e ´ 6 , 2 e ´ 6 , 5 e ´ 6 , 1 e ´ 5 , 2 e ´ 5 , 5 e ´ 5 , 1 e ´ 4 , 2 e ´ 4 , 5 e ´ 4 , 1 e ´ 3 , 2 e ´ 3 , 5 e ´ 3 u

561

562

563

564

565

566

567

LoRA dimension r P t 16 , 32 , 64 , 128 , 256 u

warmup steps P t 0 , 1000 , 2000 u

All other hyperparameters followed previous experiments.

The Alpaca dataset is freely available for download at ttps://huggingface.co/datasets/ tatsu-lab/alpaca under open-source license CC-by-NC 4.0 . The OpenHermes dataset is freely available for download at https://huggingface.co/datasets/teknium/OpenHermes-2.5 under open-source license MIT, Apache 2.0, CC .

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All claims made in the abstract and introduction are directly derived from theoretical and experimental results presented in the main paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations of experimental results are described in the limitations section, under Discussion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All theoretical results list any underlying assumptions in the main text and full proofs are available in the supplementary.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Relevant experimental results are detailed in the main text, with extensive details for all experiments further elaborated upon in the supplementary.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: While we currently answer no, and provide enough detail to reproduce our experiments, we are actively working towards packaging our code for release. All datasets are already open source, with licenses listed in the supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All datasets are open source, and all experimental hyperparameters are specified in the paper. All results are fully reproducible with these details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The paper does not report statistical significance.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper details (at length) the hardware requirements necessary to run each experiment. Environmental requirements are available as experimental details both in the main text and supplementary.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The work detailed in the paper conforms to all aspect of the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

773

Justification: The societal impact of this work is addressed in the Discussion section.

Guidelines: 774

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [No]

Justification: The work does not aim to release pretrained models or datasets.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Extensive lengths were made to cite all original authors for any and all utilized code/data/work.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper currently does not release source code. However, as previously mentioned, we are actively working to remedy this.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

878

879

880

881

882

883

884

885

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the capabilities of Mamba state-space models (SSMs) in comparison to Transformer large language models (LLMs) in various downstream learning tasks. Despite Mamba's success in some areas, the paper identifies challenges and limitations in achieving performance parity with Transformers on standard benchmarks, particularly in in-context learning (ICL), mixed-precision fine-tuning (MPFT), and parameter-efficient fine-tuning (PEFT). The study demonstrates that while Mamba models have robust recurrent dynamics and can achieve significant speed and memory efficiency gains through fine-tuning techniques, their downstream learning improvements still lag behind those of Transformers.\n\nSTRENGTHS:\n1. This paper is well-written and easy to read.\n\n2. The study shows that Mamba’s recurrent dynamics are robust to small input changes, which is validated both theoretically and empirically. This robustness ensures stability in training and fine-tuning processes.\n\n3. Despite initial shortcomings in ICL performance, extensive experiments demonstrate that mamba models exhibit strong potential for improvement through efficient fine-tuning. The models can achieve up to 81.5% of the ICL performance improvement, highlighting their adaptability with appropriate tuning methods.\n\nWEAKNESSES:\nI appreciate the authors for providing a theoretical analysis to demonstrate the controllability of implementing AMP on Mamba blocks, and the experiments indicate that PEFT is also suitable for Mamba. However, I have several concerns:\n\n1. The authors define the Mamba process as a generalized operation: $x_t=F_{\\theta}(x_{t-1},u_t)$, but the actual output of Mamba is $y_t = \\bar{C_t}x_t$.Therefore, the theoretical analysis provided in the paper pertains to the stability of the hidden state under small perturbations. Is it possible to extend this analysis directly to the output $y_t$? Since the stability of the hidden state does not necessarily imply the stability of the output.\n2. Theorem 1 ensures the feasibility of implementing LoRA on Mamba blocks but focuses on the $W$ matrix, neglecting the consideration of the most crucial transition matrix $\\bar{A_t}$ in Mamba. Does this mean that the $\\bar{A_t}$ matrix was not subjected to LoRA during fine-tuning? If so, is it possible to consider applying PEFT to the $\\bar{A_t}$ matrix as well?\n3. As an empirically-driven paper, would it be possible to include more backbones for comparison in future versions? Currently, the only baseline for comparison is Pythia.\n\nQUESTIONS:\nPlease see weakness.\n\nLIMITATIONS:\nThe authors raise some limitaions, for example, the lower-precision method is not explored in this paper, but the authors claime to solve them in the future.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores Mamba's downstream learning capabilities through two primary aspects: (i) fine-tuning and (ii) in-context learning. Specifically, it examines the training stability and robustness of fine-tuning when mixed precision is applied, as well as Mamba's ability to perform in-context learning. The contributions of this paper include:\n\n- Theoretical analysis of the stable dynamics of Mamba.\n- The theoretical analysis is corroborated by the experiments.\n- Experimental demonstration of Mamba's limitations on real datasets in terms of ICL.\n- ICL performance improvement of Mamba through fine-tuning.\n\nSTRENGTHS:\n- The theoretical analysis section, although I did not verify the proofs, offers valuable insights and paves the way for more complex analyses in future work.\n- This paper is well-motivated.\n\nWEAKNESSES:\nThe weakness is majorly from the ICL part. \n\nIn fact, the authors show that pretrained Mamba cannot learn well via ICL, but can learn well after fine-tuning. This fact indicates that this limitation does not come from Mamba architecture itself, which is also consistent with the observation of other works such as [1]. Therefore, the limitations observed in this paper is just a general limitation caused by training recipes, which is not Mamba-specific and has been studied in many works. The solution is also standard, and the improvements are also expected since once trains well, Mamba should be able to perform in-context learning as shown in [1]. \n\nMoreover, many questions are still unclear. For instance, why does Mamba suffer from such limitations? \n\nTherefore, the study in terms of the ICL part is lack of depth, novelty, and technical contribution. \n\nIn terms of the mixed precision part, there are also many places that are unclear to me. I suggest the authors to use more space in discussing the speciality of Mamba compared to Transformers, and what structure of Mamba caused this problem. If recurrence is the main cause of the problem, having more experiments of similar models such as GLA, linear attention, etc would also help readers to understand more about the phenomenon. \n\n---\n\nReferences\n\n[1] Park, Jongho, et al. \"Can mamba learn how to learn? a comparative study on in-context learning tasks.\" ICML 2024.\n\nQUESTIONS:\n- Is mean divergence the main metric in measuring the performance difference of a model in terms of different precision?\n- Where does the numbers in L267 and L269 comes from? Based on Figure 3, it looks like the percentage of improvements should be less than 5% by observing the y limit of the figures.\n\nLIMITATIONS:\nThe work is well-motivated, but the study lacks depth.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper looks at improving start space models or Mamba by enabling mixed precision handling to improve inference and fine-tuning. The results show similar performance with a significantly reduced memory requirement

STRENGTHS:
There are extensive results compared to full-precision models
The authors provide a proof of the theorem to back up their claim
The change to the mamba block is clear and easy to implement by others

WEAKNESSES:
The actual change is relatively minor in quantity but does deliver the author's required memory reductions.
The works don't use the larger models available due to limitations on memory requirements still

QUESTIONS:
nothing further

LIMITATIONS:
the limitations are well discussed at the end of the paper and generally relate to LLM or transformers in general too
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-15