SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The parameters of LSTM, GRU and RNN are:  $M = 16$ ,  $N = 100$ ,  $K = 4$ . The time constant of neurons in the RNN is  $\tau = 2$ . The parameters for data augmentation are:  $\Delta T_{max} = 60$ , the noise strength inserted between neighboring items is  $\sigma = 0.01$ . The recurrent connection weights  $\mathbf{W}^{rec}$  are initialized by sampling from an independent Gaussian distribution with zero mean and SD equal to  $g / \sqrt{N}$ , with  $g$  representing the 'gain' of the network,  $g = 0.2$ . The connections of LSTM and GRU adopt Glorot are initialized by an uniform distribution. For the design of target functions, see Sec. A. The batch size is  $B = 16$ , and the total number of training epochs is 200.
[1] Wang, X. J. (2008). Decision making in recurrent neuronal circuits. Neuron, 60(2), 215-234.
[2] Okazawa, G., & Kiani, R. (2023). Neural Mechanisms that Make Perceptual Decisions Flexible. Annual Review of Physiology, 85, 191-215.
[3] Shadlen, M. N., & Newsome, W. T. (1996). Motion perception: seeing and deciding. Proceedings of the national academy of sciences, 93(2), 628-633.
[4] Sussillo, D., & Barak, O. (2013). Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks. Neural computation, 25(3), 626-649.
[5] Maheswaranathan, N., Williams, A., Golub, M., Ganguli, S., & Sussillo, D. (2019). Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics. Advances in neural information processing systems, 32.
[6] Garofolo, J. S., Lamel, L. F., Fisher, W. M., Fiscus, J. G., Pallett, D. S. & Dahlgren, N. L. (1993). DARPA TIMIT Acoustic Phonetic Continuous Speech Corpus CDROM NIST
[7] Carandini, M., & Heeger, D. J. (2012). Normalization as a canonical neural computation. Nature Reviews Neuroscience, 13(1), 51-62.
[8] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., ... & Chintala, S. (2019). Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
[9] Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780.
[10] Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555.