SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We present a sparse Hopfield model with a memory-retrieval dynamics that corresponds to the sparse-structured attention mechanism. This model is capable of merging into deep learning architectures with data-dependent sparsity. Theoretically, we introduce a principled construction for modern Hopfield models, based on the convex conjugate of different entropy regularizers. It allows us to easily recover the dense modern Hopfield model [Ramsauer et al., 2021] using Gibbs entropy. Moreover, we introduce the sparse Hopfield model using the Gini entropic regularizer, and ex
plore its theoretical advantages, delineating conditions that favor its use. Empirically, we demonstrate our theoretical results and methodology to be effective on various synthetic and realistic settings. This work extends the correspondence between artificial and biological neural networks to sparse domain, potentially paving the way for future Hopfield-based methodologies and bio-inspired computing systems.
Table 2: Results for MIL benchmark datasets in terms of AUC score. The baselines are Path encoding [Küçukası and Baydoğan, 2018], MInD [Cheplygina et al., 2015], MILES [Chen et al., 2006], APR [Dietterich et al., 1997], Citation-KNN [Wang and Zucker, 2000] and DD [Maron and Lozano-Pérez, 1997]. Results for baselines are taken from [Ramsauer et al., 2021]. The results suggest the proposed model achieves near-optimal performance even when the data is not sparse.
Method	Tiger	Fox	Elephant	UCSB
Dense Hopfield	0.878 ± 0.028	0.600 ± 0.011	0.907 ± 0.022	0.880 ± 0.013
Sparse Hopfield	0.892 ± 0.021	0.611 ± 0.010	0.912 ± 0.016	0.877 ± 0.009
Path encoding	0.910 ± 0.010	0.712 ± 0.014	0.944 ± 0.007	0.880 ± 0.022
MInD	0.853 ± 0.011	0.704 ± 0.016	0.936 ± 0.009	0.831 ± 0.027
MILES	0.872 ± 0.017	0.738 ± 0.016	0.927 ± 0.007	0.833 ± 0.026
APR	0.778 ± 0.007	0.541 ± 0.009	0.550 ± 0.010	-
Citation-kNN	0.855 ± 0.009	0.635 ± 0.015	0.896 ± 0.009	0.706 ± 0.032
DD	0.841	0.631	0.907	-
Post-Acceptance Note [November 28, 2023]. After the completion of this work, the authors learn of two upcoming works — [Anonymous, 2023] at ICLR'24 and [Martins et al., 2023] in the Associative Memory & Hopfield Networks Workshop at NeurIPS'23 — both addressing similar topics. Both of these works explore theoretical generalizations of this work. In addition, [Anonymous, 2023] further presents a (sparse) Hopfield-based deep learning model for multivariate time series prediction. We thank the authors of [Martins et al., 2023] for enlightening discussions and for sharing their preliminary manuscript.
JH would like to thank Tim Tsz-Kit Lau, Robin Luo and Andrew Chen for enlightening discussions on related topics, and Jiayi Wang for invaluable support in facilitating experimental deployments. The authors would like to thank the anonymous reviewers and program chairs for constructive comments.
JH is partially supported by the Walter P. Murphy Fellowship. HL is partially supported by NIH R01LM1372201, NSF CAREER1841569, DOE DE-AC02-07CH11359, DOE LAB 20-2261 and a NSF TRIPODS1740735. This research was supported in part through the computational resources and staff contributions provided for the Quest high performance computing facility at Northwestern University which is jointly supported by the Office of the Provost, the Office for Research, and Northwestern University Information Technology. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies.
Anonymous. STanhop: Sparse tandem hopfield model for memory-enhanced time series prediction. In Submitted to The Twelfth International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=6iwg437CZs. under review.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems, 32, 2019. URL https://arxiv.org/abs/1909.01377.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020. URL https://arxiv.org/abs/2004.05150.
Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, et al. Findings of the 2017 conference on machine translation (wmt17). Association for Computational Linguistics, 2017.
Johannes Brandstetter. Blog post: Hopfield networks is all you need, 2021. URL https://ml-jku.github.io/hopfield-layers/. Accessed: April 4, 2023.
Johann S Brauchart, Alexander B Reznikov, Edward B Saff, Ian H Sloan, Yu Guang Wang, and Robert S Womersley. Random point sets on the sphere—hole radii, covering, and separation. Experimental Mathematics, 27(1):62-81, 2018. URL https://arxiv.org/abs/1512.07470.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. URL https://arxiv.org/abs/1512.07470.
T Tony Cai and Tiefeng Jiang. Phase transition in limiting distributions of coherence of high-dimensional random matrices. Journal of Multivariate Analysis, 107:24-39, 2012. URL https://arxiv.org/abs/1102.2926.
Marc-André Carbonneau, Veronika Cheplygina, Eric Granger, and Ghyslain Gagnon. Multiple instance learning: A survey of problem characteristics and applications. Pattern Recognition, 77: 329-353, 2018. URL https://arxiv.org/abs/1612.03365.
Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 12270-12280, 2021. URL https://arxiv.org/abs/2107.00651.
Yongyi Yang, Zengfeng Huang, and David Wipf. Transformers from an optimization perspective. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=VT0Y4PlV2m0.
Alan L Yuille and Anand Rangarajan. The concave-convex procedure (cccp). Advances in neural information processing systems, 14, 2001. URL https://proceedings.neurips.cc/paper_files/paper/2001/file/a012869311d64a44b5a0d567cd20de04-Paper.pdf.
Alan L Yuille and Anand Rangarajan. The concave-convex procedure. Neural computation, 15(4): 915-936, 2003.
Willard I Zangwill. Nonlinear programming: a unified approach, volume 52. Prentice-Hall Englewood Cliffs, NJ, 1969.
Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The Eleventh International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=vSVLM2j9eie.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106-11115, 2021. URL https://arxiv.org/abs/2012.07436.
Tian Zhou, Ziqing Ma, Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin, Rong Jin, et al. Film: Frequency improved legendre memory model for long-term time series forecasting. Advances in Neural Information Processing Systems, 35:12677-12690, 2022. URL https://arxiv.org/abs/2205.08897.