SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In our experiments, we train large stacked LSTM language models on the PTB dataset at different sparsity levels. The model architecture follows the same design as in [63], comprising an embedding layer, two stacked LSTM layers, and a softmax layer. The vocabulary size for the model is 10000, the embedding layer size is 1500, and the hidden layer size is 1500 resulting in a total of 66 million parameters. We compare our method with AGP at different sparsity levels, including  $80\%$ ,  $85\%$ ,  $90\%$ ,  $95\%$ , and  $97.5\%$  as in [68]. The results are summarized in Table 11, and our method achieves better results consistently. For the AGP, numbers are taken directly from the original paper, and since they only provided one experimental result for each sparsity level, no standard deviation is reported. For our method, we run three independent trials and provide both the mean and standard deviation for each sparsity level. During the initial training stage, we follow the same training procedure as in [63]. The details of the prior annealing and fine-tuning stages of our method for different sparsity levels are provided below.
During the initial training stage, we follow the same training procedure as in [63] for all sparsity, and hence  $T_{1} = 55$ .
During the prior annealing stage, we train a total of 60 epochs using SGHMC. For all levels of sparsity considered, we fix  $\sigma_{1,n}^2 = 0.5$ ,  $\lambda_n = 1e - 6$ , momentum  $1 - \alpha = 0.9$ , minibatch size  $= 20$ , and we fix  $T_2 = 5 + T_1 = 60$ ,  $T_3 = T_2 + 20 = 80$ . We set the initial temperature  $\tau = 0.01$  for  $t \leq T_3$  and gradually decrease  $\tau$  by  $\tau = \frac{0.01}{t - T_3}$  for  $t > T_3$ .
During the fine tune stage, we apply a similar training procedure as the initial training stage, i.e., we use SGD with gradient clipping and we decrease the learning rate by a constant factor after certain epochs. The minibatch size is set to be 20, and we apply early stopping based on the validation perplexity with a maximum of 30 epochs.
Table 12 gives all hyperparameters (not specified above) for different sparsity levels.
Note that, the mixture Gaussian prior by nature is a regularization method, so we lower the dropout ratio during the prior annealing and fine tune stage for models with relatively high sparsity.
Table 11: Test set perplexity for different methods and sparsity levels.
Method	Sparsity	Test Perplexity
baseline	0%	78.40
PA	80.87% ± 0.09%	77.122 ± 0.147
PA	85.83% ± 0.05%	77.431 ± 0.109
PA	90.53% ± 0.05%	78.823 ± 0.118
PA	95.40% ± 0.00%	84.525 ± 0.112
PA	97.78% ± 0.05%	93.268 ± 0.136
AGP	80%	77.52
AGP	85%	78.31
AGP	90%	80.24
AGP	95%	87.83
AGP	97.5%	103.20
Table 12: Word Language Model Compression: Hyperparameters for our method during the prior annealing and fine tune stage. We denote the learning rate by LR, the prior annealing stage by PA, and the fine tune stage by FT.
Hyperparameters/Sparsity	80%	85%	90%	95%	97.5%
Dropout ratio	0.65	0.65	0.65	0.4	0.4
(σinit0,n)2	0.0005	0.0007	0.00097	0.00174	0.00248
(σend0,n)2	1e-7	1e-7	1e-7	1e-7	1e-6
LR PA	0.004	0.004	0.008	0.008	0.01
LR FT	0.008	0.008	0.008	0.1	0.2
LR decay factor FT	1/0.95	1/0.95	1/0.95	1/0.95	1/0.95
LR decay epoch FT	5	5	5	5	5
Table 13: Training details for the additional autoregressive order selection experiment.
yi = (0.8 - 1.1 exp{-50yi-1}) yi-1 + ηi	PA-RNN 1,3,5,7,10,15
Learning rate	1e-4
Iterations	2000
Optimizer	SGHMC
Batch size	36
Subsample size per iteration	50
Predicton horizon	1
σ21,n	0.05
(σinit0,n)2	1e-5
(σend0,n)2	1e-6
λn	1e-6
Temperature	0.1
T1	500
T2	1000
T3	1500