Table 2: Comparison with different methods on GLUE. The first and second best results are in **bold** and **underlined**, respectively. All baseline results for BERTbase and RoBERTlarge are from [49] and [17], respectively. We report Spearman's Correlation for STS-B and matched accuracy for MNLI on BERTbase. The hyper-parameters after each backbone are used for measuring the memory footprint.  $r = 8$  for all MEFTs. All models are trained in FP16 if not specified with FP32.
Method	#Param. (%)	Memory (GB)	RTE	MRPC	STS-B	CoLA	SST-2	QNLI	QQP	MNLI	Avg.
Peak	Act.
BERTbase (batch size = 32, sequence length = 512)
Full FT	100	16.67	14.98	71.11.5	85.71.8	89.00.5	59.30.6	92.60.2	91.50.1	91.50.0	84.40.2	83.2
Prefix-Tuning	0.17	13.58	13.00	70.50.5	85.90.9	88.80.2	58.91.2	91.90.5	90.80.1	89.10.1	82.80.2	82.3
LoRA	0.27	13.45	13.02	65.91.5	84.51.0	88.70.1	57.60.8	92.10.4	90.60.2	89.40.0	83.01.1	81.5
MAM	6.97	14.21	13.41	69.11.8	87.20.7	89.00.5	47.92.4	83.917.	90.90.2	90.80.1	83.30.2	80.3
AutoPEFT	1.40	-	-	72.40.9	87.50.9	89.20.0	60.91.5	92.10.3	91.10.1	90.60.1	84.00.1	83.5
vanilla gradient
MEFT1	0.27	13.64	13.21	74.21.4	86.70.2	89.00.0	62.10.2	92.90.2	91.60.1	89.90.1	83.80.4	83.8
MEFT2	0.27	13.73	13.31	74.70.3	86.60.5	89.40.1	61.80.7	93.00.1	91.60.1	90.20.1	84.50.1	84.0
MEFT3	0.27	13.64	13.21	76.10.8	87.40.3	88.90.1	62.30.5	93.20.2	91.50.1	90.10.1	84.20.2	84.2
reversible gradient
MEFT1(FP32)	0.27	2.75	2.33	73.90.5	86.50.2	88.80.1	60.30.6	92.70.4	91.40.0	88.80.1	83.40.1	83.2
MEFT2(FP32)	0.27	3.53	3.11	74.00.6	86.30.4	88.60.1	60.71.5	92.80.2	91.50.1	88.90.1	83.10.1	83.2
MEFT3(FP32)	0.27	2.99	2.57	70.80.6	84.60.5	88.20.3	53.91.0	92.20.4	90.40.2	86.90.3	81.50.1	81.1
RoBERTlarge (batch size = 32, sequence length = 128)
Full FT	100	11.47	6.05	86.6	90.9	92.4	68.0	96.4	94.7	92.2	90.2	88.9
AdapterH	0.23	6.05	4.66	72.93.0	87.71.7	91.50.5	66.32.0	96.30.5	94.70.2	91.50.1	90.30.3	86.4
AdapterH	1.69	6.18	4.71	83.41.1	88.72.9	91.01.7	66.54.4	96.20.3	94.70.2	92.10.1	89.90.5	87.8
AdapterP	0.23	6.16	4.77	80.12.9	89.71.2	91.90.4	67.82.5	96.60.2	94.80.3	91.70.2	90.50.3	87.9
AdapterP	0.85	6.21	4.78	83.82.9	90.20.7	92.10.7	68.31.0	96.10.3	94.80.2	91.90.1	90.20.3	88.4
LoRA	0.23	6.11	4.72	85.21.1	90.21.0	92.30.5	68.21.9	96.20.5	94.80.3	91.60.2	90.60.2	88.6
vanilla gradient
MEFT1	0.23	6.19	4.81	89.50.8	91.50.2	92.30.1	69.90.7	96.80.1	94.90.1	91.50.1	90.30.2	89.6
MEFT2	0.23	6.20	4.82	88.60.6	91.30.4	92.20.1	68.80.7	96.80.1	94.80.1	91.40.1	90.60.0	89.3
reversible gradient
MEFT1	0.23	3.11	1.73	87.60.3	90.50.6	91.60.1	63.31.7	95.90.1	94.30.2	90.10.1	89.20.7	87.8
MEFT1(FP32)	0.23	3.63	2.25	90.00.5	91.20.2	92.40.1	66.10.7	96.70.3	94.80.1	90.20.5	90.10.1	88.9
MEFT2(FP32)	0.23	3.75	2.37	88.20.5	90.50.4	92.10.0	64.40.6	95.90.2	94.30.1	89.40.1	88.40.5	87.9
BARTlarge (batch size = 100, sequence length = 128)
Full FT [20]	100	12.75	9.62	77.6	89.2	-	59.3	95.8	94.3	89.5	90.8	85.2
J'-Tuning [20]	7.7	-	-	62.8	79.2	-	44.4	94.4	88.2	85.5	82.3	76.7
LST(FP32) [21]	2.6	7.05	6.14	69.7	87.3	-	55.5	94.7	91.9	89.5	86.1	82.1
reversible gradient
MEFT1	0.20	2.54	1.75	72.21.3	88.11.3	-	51.01.8	95.10.2	92.40.1	87.50.0	87.00.2	81.9
MEFT1(FP32)	0.20	2.54	1.75	74.30.7	88.40.5	-	57.42.2	95.40.1	93.90.1	89.30.1	88.30.1	83.9