Package: pygranso
PyGRANSOPyGRANSO: A PyTorch-enabled port of GRANSO with auto-differentiationPlease checkhttps://ncvx.orgfor detailed instructions (introduction, installation, settings, examples...).PyGRANSO is AGPL v3.0 licensed, but it also contains a small portion of GPL code.  Please see the LICENSE_INFO folder for more information.Brief IntroductionOptimizing nonconvex (NCVX) problems, especially nonsmooth and constrained ones, is an essential part of machine learning. However, it can be hard to reliably solve such problems without optimization expertise. Existing general-purpose NCVX optimization packages are powerful but typically cannot handle nonsmoothness. GRANSO is among the first optimization solvers targeting general nonsmooth NCVX problems with nonsmooth constraints, but, as it is implemented in MATLAB and requires the user to provide analytical gradients, GRANSO is often not a convenient choice in machine learning (especially deep learning) applications. To greatly lower the technical barrier, we introduce a new software package called NCVX, whose initial release contains the solver PyGRANSO, a PyTorch-enabled port of GRANSO incorporating auto-differentiation, GPU acceleration, tensor input, and support for new QP solvers. NCVX is built on freely available and widely used open-source frameworks, and as a highlight, can solve general constrained deep learning problems, the first of its kind.InstallationInstalling PyGRANSO is simple. Here is a step-by-step instruction:InstallPython >= 3.9Get the most recent PyGRANSO package (including examples and requirements file):git clone https://github.com/sun-umn/PyGRANSO.git
 cd PyGRANSOInstall PyGRANSO solver from PyPI:pip install git+https://github.com/sun-umn/PyGRANSO.gitInstall Dependencies from PyPI:OS:LinuxORWindows; Compute Platform:CUDA:pip install -r requirements.txt -f https://download.pytorch.org/whl/cu111/torch_stable.htmlOS:Linux; Compute Platform:CPU:pip install -r requirements_linux_cpu.txt -f https://download.pytorch.org/whl/cpu/torch_stable.htmlOS:MacORWindows; Compute Platform:CPU:pip install -r requirements_cpu.txt(CUDA) Run test to make sure the dependency installation is correct:python test_cuda.py(CPU) Run test to make sure the dependency installation is correct:python test_cpu.pyCheck theexample folderin the source code orexample sectionon the documentation website to get started.DependenciesPython-3.9.7

numpy-1.20.3

scipy-1.7.1

pytorch-1.9.0

osqp-0.6.2

Jupyter Notebook-6.4.5

gurobiChange LogsPlease checkCHANGELOG.mdin the main folder.Notes on DocumentationPyGRANSO is a PyTorch-enabled port of GRANSO with auto-differentiation, and some of its documentation uses MATLAB conventions. In the PyGRANSO source code docstrings, please note that:structrefers topygransoStruct, which is a simple class that users need to use for specifying their problems and options.Vector and matrix refer to PyTorch tensor with(n,1)and(m,n)dimension, respectively.AcknowledgementsWe would like to thankFrank E. CurtisandMichael L. Overtonfor their involvement in creating the BFGS-SQP algorithm that is
implemented in the software packageGRANSO. This work was supported by UMII Seed Grant Program and NSF CMMI 2038403.CitationIf you publish work that uses or refers to PyGRANSO, please cite the following two papers,
which respectively introduced PyGRANSO and GRANSO:[1] Buyun Liang, Tim Mitchell, and Ju Sun,
NCVX: A User-Friendly and Scalable Package for Nonconvex
Optimization in Machine Learning, arXiv preprint arXiv:2111.13984 (2021).Available athttps://arxiv.org/abs/2111.13984[2] Frank E. Curtis, Tim Mitchell, and Michael L. Overton,
A BFGS-SQP method for nonsmooth, nonconvex, constrained
optimization and its evaluation using relative minimization
profiles, Optimization Methods and Software, 32(1):148-181, 2017.Available athttps://dx.doi.org/10.1080/10556788.2016.1208749BibTex:@article{liang2021ncvx,
    title={{NCVX}: {A} User-Friendly and Scalable Package for Nonconvex 
    Optimization in Machine Learning}, 
    author={Buyun Liang, Tim Mitchell, and Ju Sun},
    year={2021},
    eprint={2111.13984},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

@article{curtis2017bfgssqp,
    title={A {BFGS-SQP} method for nonsmooth, nonconvex, constrained optimization and its evaluation using relative minimization profiles},
    author={Frank E. Curtis, Tim Mitchell, and Michael L. Overton},
    journal={Optimization Methods and Software},
    volume={32},
    number={1},
    pages={148--181},
    year={2017},
    publisher={Taylor \& Francis}
}ContactFor questions or bug reports, please either:raise issues in the PyGRANSO repository (https://github.com/sun-umn/PyGRANSO/) orsend an email to:Buyun Liang(liang664 an_at_symbol umn a_dot_symbol edu)Tim Mitchell(tim an_at_symbol timmitchell a_dot_symbol com)Ju Sun(jusun an_at_symbol umn a_dot_symbol edu)Thanks to other contributors and bug reporters:Hengyue Liang: Tested PyGRANSO on adversarial robustness problems. Tested PyGRANSO across multiple platforms. Debugged several functions.Ying Cui: Advised the adversarial robustness problems.Chen Jiang: Tested perceptual attack example (ex6). Tested PyGRANSO on Win10. Debugged updatePenaltyParameter function.