SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We propose an exponentially convergent algorithm for nonconvex SMF training using new lifting techniques. Our analysis demonstrates strong convergence and estimation guarantee. We compare the robustness of filter-based and feature-based SMF, finding that the former is computationally more robust while the latter is statistically more robust. The algorithm's exponential convergence is numerically verified. In cancer classification using microarray data analysis, our algorithm successfully identifies discriminative gene groups for various cancers and shows potential for identifying important gene groups as protein complexes or pathways in biomedical research. Our analysis framework can be extended to more complex classification models, such as combining a feed-forward deep neural network with a matrix factorization objective. While our convergence analysis holds in certain parameter regimes, we discuss them in detail. We have tested our method and convergence analysis on various real-world datasets but recommend further numerical verification on a wider range of datasets.
[1] Real / fake job posting prediction. https://www.kaggle.com/datasets/shivamb/real-or-fake-fake-jobposting-prediction. Accessed: 2023-05-16.
[2] Hervé Abdi and Lynne J Williams. Principal component analysis. Wiley interdisciplinary reviews: computational statistics, 2(4):433-459, 2010.
[3] Alekh Agarwal, Sahand Negahban, and Martin J Wainwright. Fast global convergence rates of gradient methods for high-dimensional statistical recovery. Advances in Neural Information Processing Systems, 23, 2010.
[4] Sanae Amani and Christos Thrampoulidis. Ucb-based algorithms for multinomial logistic regression bandits. Advances in Neural Information Processing Systems, 34, 2021.
[5] Woody Austin, Dylan Anderson, and Joydeep Ghosh. Fully supervised non-negative matrix factorization for feature extraction. In IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium, pages 5772-5775. IEEE, 2018.
[6] Shivam Bansa. fake job postings dataset. https://www.kaggle.com/datasets/shivamb/real-or-fake-fake-jobposting-prediction, 2017.
[7] Amir Beck. First-order methods in optimization. SIAM, 2017.
[8] Christopher M Bishop et al. Neural networks for pattern recognition. Oxford university press, 1995.
[48] Angelika Rohde and Alexandre B Tsybakov. Estimation of high-dimensional low-rank matrices. 2011.
[49] Evelina Sjostedt, Wen Zhong, Linn Fagerberg, Max Karlsson, Nicholas Mitsios, Csaba Adori, Per Oksvold, Fredrik Edfors, Agnieszka Limiszewska, Feria Hikmet, et al. An atlas of the protein-coding genes in the human, pig, and mouse brain. Science, 367(6482):eaay5947, 2020.
[50] Gui-Xian Tian and Ting-Zhu Huang. Inequalities for the minimum eigenvalue of m-matrices. The Electronic Journal of Linear Algebra, 20:291-302, 2010.
[51] Tian Tong, Cong Ma, and Yuejie Chi. Accelerating ill-conditioned low-rank matrix estimation via scaled gradient descent. Journal of Machine Learning Research, 22(150):1-63, 2021.
[52] Joel A Tropp et al. An introduction to matrix concentration inequalities. Foundations and Trends® in Machine Learning, 8(1-2):1-230, 2015.
[53] Stephen Tu, Ross Boczar, Max Simchowitz, Mahdi Soltanolkotabi, and Ben Recht. Low-rank solutions of linear matrix equations via procrustes flow. In International Conference on Machine Learning, pages 964-973. PMLR, 2016.
[54] Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
[55] Michael E Wall, Andreas Rechtsteiner, and Luis M Rocha. Singular value decomposition and principal component analysis. In A practical approach to microarray data analysis, pages 91-109. Springer, 2003.
[56] Lingxiao Wang, Xiao Zhang, and Quanquan Gu. A unified computational and statistical framework for nonconvex low-rank matrix estimation. In Artificial Intelligence and Statistics, pages 981-990. PMLR, 2017.
[57] Stephen J Wright. Coordinate descent algorithms. Mathematical Programming, 151(1):3-34, 2015.
[58] Yangyang Xu and Wotao Yin. A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor factorization and completion. SIAM Journal on imaging sciences, 6(3):1758-1789, 2013.
[59] Yael Yankelevsky and Michael Elad. Structure-aware classification using supervised dictionary learning. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4421-4425. IEEE, 2017.
[60] Pavel Yaskov. Controlling the least eigenvalue of a random gram matrix. Linear Algebra and its Applications, 504:108-123, 2016.
[61] Qiang Zhang and Baoxin Li. Discriminative k-svd for dictionary learning in face recognition. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 2691-2698. IEEE, 2010.
[62] Shijie Zhao, Junwei Han, Jinglei Lv, Xi Jiang, Xintao Hu, Yu Zhao, Bao Ge, Lei Guo, and Tianming Liu. Supervised dictionary learning for inferring concurrent brain networks. IEEE transactions on medical imaging, 34(10):2036-2045, 2015.
[63] Tuo Zhao, Zhaoran Wang, and Han Liu. A nonconvex optimization framework for low rank matrix estimation. Advances in Neural Information Processing Systems, 28:559, 2015.
[64] Qinqing Zheng and John Lafferty. A convergent gradient descent algorithm for rank minimization and semidefinite programming from random linear measurements. arXiv preprint arXiv:1506.06081, 2015.