MNLI	QQP	QNLI	SST-2	CoLA	STS-B	MRPC	RTE	WNLI	Avg
Low-data-100
Baseline	33.893.02	30.650.38	58.784.81	56.013.68	5.204.84	40.009.64	74.800.0	49.392.86	55.213.01	44.87
AA	33.642.66	30.880.39	59.616.19	51.282.52	-0.551.87	45.1814.17	74.800.0	50.113.44	55.482.46	44.49
AdapterDropAA	33.722.84	30.620.40	57.505.78	54.012.59	4.107.95	36.538.93	74.800.0	49.392.86	56.061.38	44.08
AdapterDrop13	33.712.76	30.610.4	58.394.27	53.442.56	3.917.6	36.238.68	74.800.0	49.462.81	55.761.91	44.04
AA-focusedspec	35.282.06	44.3716.31	63.754.39	52.944.64	5.6810.91	62.793.34	74.800.01	51.482.72	54.084.51	49.47
AA-focuseduni	36.362.61	44.3716.31	63.364.86	55.874.42	4.754.9	59.376.78	74.940.2	51.123.45	51.834.12	49.11
AA-focusedsim	34.773.18	45.7814.40	63.134.30	61.5810.95	17.5411.19	59.897.70	74.770.07	52.202.93	51.835.52	51.28
|Baseline|	24	24	24	24	24	24	24	24	24
|AA|	13.21.7	15.03.0	13.62.2	14.64.0	15.82.1	16.42.7	13.01.8	11.21.8	12.35.7
|AdapterDropAA|	14	13	15	16	16	14	15	13	16
|AdapterDrop13|	13	13	13	13	13	13	13	13	13
|AA-focusedspec|	14	13	15	16	16	14	15	13	16
|AA-focuseduni|	13	13	13	13	13	13	13	13	13
|AA-focusedsim|	13	13	13	13	13	13	13	13	13
Low-data-300
Baseline	36.554.76	61.508.66	69.621.24	79.8614.15	30.405.48	78.242.81	76.551.31	51.623.21	45.924.33	58.91
AA	37.142.49	66.070.38	71.331.82	72.5216.59	26.058.74	82.081.6	74.032.21	51.832.84	47.044.76	58.68
AdapterDropAA	38.865.93	62.984.85	66.712.91	79.2914.17	16.8912.06	78.51.99	75.740.67	51.193.35	46.764.02	57.44
AdapterDrop13	37.955.56	63.724.84	66.712.91	80.014.47	16.312.05	77.522.08	76.030.92	51.333.4	46.484.27	57.34
AA-focusedspec	44.624.11	66.831.06	73.721.09	85.872.94	34.518.3	81.162.04	76.721.06	54.584.72	46.203.92	62.69
AA-focuseduni	46.694.29	69.251.33	74.162.95	87.570.72	35.653.26	81.712.64	75.971.55	56.895.56	52.397.26	64.48
AA-focusedsim	45.972.08	68.361.36	73.982.68	86.831.90	37.433.10	78.813.58	76.661.30	55.962.81	48.445.53	63.61
|AA|	17.01.3	16.21.0	14.81.8	12.83.2	16.82.2	18.61.9	16.01.1	12.41.2	12.42.1
|AA-focusedspec|	18	16	13	9	17	16	16	13	13
Low-data-500
Baseline	44.356.08	69.491.12	73.481.89	88.261.53	37.984.42	82.070.99	78.331.11	59.281.76	49.866.08	64.79
AA	47.335.11	67.522.99	75.02.3	84.933.06	39.964.87	84.560.87	78.381.0	59.283.18	50.135.16	65.23
AdapterDropAA	42.667.02	69.521.03	74.152.19	89.010.49	38.444.51	82.051.05	78.191.04	59.282.6	49.3636	64.73
AdapterDrop13	43.056.41	69.120.88	72.821.83	88.970.6	36.895.03	80.771.32	77.860.8	58.562.44	49.016.57	64.12
AA-focusedspec	54.962.66	69.521.14	77.301.27	87.941.10	39.513.47	84.300.69	78.921.70	59.202.58	48.736.27	66.71
AA-focuseduni	56.131.88	69.322.29	76.852.37	87.891.47	41.753.83	83.481.25	78.000.35	60.421.75	50.425.07	67.14
AA-focusedsim	55.852.62	69.862.56	77.301.93	87.571.69	39.791.42	83.231.61	78.751.26	60.071.62	49.586.75	66.89
|AA|	12.86.0	16.81.3	16.42.6	14.62.1	10.68.3	19.61.4	16.62.4	14.36.8	12.63.2
|AA-focusedspec|	14	17	18	15	17	18	14	16	14
Full Data
Baseline	85.08	88.68	91.95	93.00	58.28	89.75	83.12	70.39	56.34	79.62
AA	84.73	88.38	91.01	92.55	57.60	90.11	82.36	63.18	53.52	78.16
AdapterDropAA	84.96	88.75	91.38	93.35	58.63	89.85	82.84	66.06	56.34	79.12
AdapterDrop13	84.73	87.15	90.92	92.78	57.42	88.84	83.34	64.25	56.34	78.42
AA-focusedspec	84.77	88.46	91.38	92.32	56.79	89.74	83.42	64.98	57.75	78.84
AA-focuseduni	85.41	88.61	91.51	92.66	54.62	89.34	84.88	67.15	56.34	78.94
AA-focusedsim	85.32	88.41	91.85	91.4	57.96	89.38	84.42	67.86	57.75	79.37
|AA|	14	18	17	18	20	20	18	16	15
|AA-focusedspec|	14	18	17	18	20	20	18	16	15
Table 2: Comparing the results of (a) the standard adapter model that includes an adapter layer on all the 24 BERT-large layers (Baseline), (b) adaptable adapter (AA), (c) AdapterDrop, and (d) AA-focused adapters, in which the architecture of the adapter is selected based on the selected layers by AA. The architecture of  $AA$ -focused $spec$  is selected based on the selected layers by AA for the corresponding task and data setting when the random seed is 42. The architecture of  $AA$ -focused $uni$  is selected based on the selected layers by AA for the task of  $QQP$  on the Low-data-100 setting and for random seed 42.  $AA$ -focused $sim$  only contains an adapter layer with a rational activation function at the last 13 layers of BERT-large, i.e., the total number of adapter layers in  $AA$ -focused $uni$ . The number of layers at the inference time for the AdapterDrop $AA$  experiments are selected based on the number of layers in the corresponding  $AA$ -focused $spec$  experiments. The number of inference time layers for AdapterDrop $13$  equals 13. Except for Full Data, the reported results are averaged over five random seeds. The subscript reports the corresponding standard deviation. The Full Data results are reported for one random seed. The  $|AA|$  rows report the average number of selected adapter layers by AA using different random seeds.  $|AA$ -focused $^*$ | rows report the number of added adapter layers in the corresponding  $AA$ -focused $^*$  experiments.  $|AA$ -focused $uni$ | and  $|AA$ -focused $sim$ | are the same for all data settings.  $AdapterDrop^*$ | rows report the number of included adapter layers for the corresponding AdapterDrop experiment at the inference time.  $AdapterDrop^{AA}$ | is always the same as the corresponding  $AA$ -focused $spec$ , and  $AdapterDrop^{13}$ | is always the same as  $AA$ -focused $sim$ . The test data is the same for all the experiments. The Avg column reports the average score across all datasets. The highest performances for each dataset and each data setting are boldfaced.