Package: neuralg
neuralgNeural network approximators of linear algebra operations on GPU with PyTorchView Example notebookTable of ContentsAboutBuilt WithGetting StartedPrerequisitesInstallationUsageSmall exampleSupport and training distributionsAvailable modelsGoalsContributingLicenseAboutTheneuralgmodule is a neural-network-based collection of approximators for common numerical linear algebra operations such as eigenvalue or singular value computation. It allows utilizing GPUs for differentiable and efficient computation withPyTorch. This also supports maintaining stable gradient flows.The software is free to use and is designed for the machine learning community in general, and users focusing on topics involving numerical linear algebra in particular.Built WithThis project is built with the following packages:PyTorch, implying full differentiability and can be used for machine learningGetting StartedBelow is a short guide for setting up neuralg on your device.PrerequisitesUsers who want to use the GPU needs to set up CUDA including installing acudatoolkit.InstallationTo install neuralg via pip, runpipinstallneuralgNote thatpipwillnotset up PyTorch with CUDA and GPU support.GPU UtilizationTo set up the GPU version of PyTorch, please refer to installation procedures atPyTorch DocumentationUsageSmall exampleThe neuralg module is designed to resemble existing, commonly used numerical linear algebra libraries. Below is a small example showing how neuralg can be used to approximate the eigenvalues of a batch of random matrices. For a more elaborate and interactive example, please refer to the jupyter notebookexample problem.importtorchimportneuralgfromneuralgimporteigvals# Enable GPU support if availableneuralg.set_up_torch(torch_enable_cuda=True)# Sample a batch of matrices with uniform iid coefficients# Note that neuralg only supports input of tensor typebatch_size,matrix_size=10000,5matrix_batch=torch.rand(batch_size,matrix_size,matrix_size)# Call neuralg to approximate eigenvalueseigvals=eigvals(matrix_batch)# outputs are torch.complexfloat dtype# Construct symmetric matricessymmetric_matrix_batch=torch.triu(matrix_batch,0)+torch.transpose(torch.triu(matrix_batch,1),1,2)# Symmetric matrices have specialized appproximatorseigvals_sym=eigvals(symmetric_matrix_batch,symmetric=True)# output are torch.float dtypeSupport and training distributionsBuilt with PyTorch and targeting GPU utilization, neuralg only supports input of typetorch.Tensor. The current version of neuralg supports real valued input matrices. Current available models have been trained and evaluated on random square matrices. For details, see specifications for corresponding operations and matrix types.function callsupported dtypessupported sizestraining matrix distributioneig(x,symmetric = True)float32 (in / out), float64 (in / out)[2,20]Real valued symmetric with i.i.d. centered normally distributed eigenvalues with variance 100/3 and eigenvectors uniformly distributed on the unit sphere.eig(x,real = True)float32 (in / out), float64 (in / out)[2,10]Real valued asymmetric with i.i.d. centered normally distributed eigenvalues with variance 100/3.eig(x)float32 (in), float64 (in), complexfloat64 (out), complexfloat128(out)[2,5]Real valued with i.i.d. uniformly distributed elements on [-10,10].svd(x)float32 (in / out), float64 (in / out)[2,20]Real valued with i.i.d. uniformly distributed elements on [-10,10].Available modelsAll current available models are listed below. The nerf type models are implementations of architecture fromNeRF: Representing Scenes as Neural Radiance Fields for View Synthesismodel namemodel typehyper parametersAdditional informationEigNERFnerf8 hidden layers, 200 neurons, skips at layers [2,4,6]CEigNERFcomplex nerf, based onnerf8 hidden layers, 200 neurons, skips at layers [2,4,6]Outputs are complex valuedCustomizing and adding modelsAdditionally, neuralg supports training models from scratch or re-training and fine tuning existing models, depending on specific user applications. It is also possible to introduce new model architectures to expand the module. Please refer to thetutorialfor a thorough how-to guide.GoalsProof of concept: Recent years of rapid advances in machine learning aside, neural network models are yet to reach competitive results in the field of numerical linear algebra. Some attention has been paid to the subject, e.g. withparameter rich transformers. neuralg serves as a demonstration of a competitive small-scale approach, with the goal of mitigating issues with memory and time complexity related to larger models.Supporting science: Linear algebra problems serve as fundamental computational components in countless science and engineering applications. Eigenvalue and singular value decompositions, solving linear system of equations and matrix inversion appear as essential parts of solutions in  optimization, dynamical systems, signal processing etc. Ultimately, neuralg aims to provide useful tools to researchers within these fields, with a focus on parallell computation.Addressing efficient vectorization: The nature of classical numerical algorithms for linear algebra operations are often iterative and difficult to vectorize efficiently on GPUs and specialized machine learning hardware. Existing built-in libraries often synchronize with CPU, which can severly slow down computation. To this end, neuralg aims to allow users to exploit the computational benefits from GPU parallelization on targeted hardware.ContributingThe project is open to community contributions. Feel free to open anissueor write us an email if you would like to discuss a problem or idea first.If you want to contribute, please proceed according to:Fork the project onGitHub.Get the most up-to-date code by installing neuralg from source:Getminicondaor similarClone the repogitclonehttps://github.com/esa/neuralg.gitSetup the environment. This will create a conda environment calledneuralgcondaenvcreate-fenvironment.yml
condaactivateneuralgAfter cloning the repository, developers can check the functionality ofneuralgby running the following command in the root directory:pytestNow you are all set to contribute. PRs should be created from and into thedevelopbranch. For each release the develop branch is merged into main.Create your Feature Branch (git checkout -b feature/NewFeature)Commit your Changes (git commit -m 'Add some NewFeature')Push to the Branch (git push origin feature/NewFeature)Open a Pull Request on thedevelopbranch,notmainand we will have a look at your contribution as soon as we can.Furthermore, please make sure that your PR passes all automated tests, after which the contribution will be revire. Only PRs created on thedevelopbranch with all tests passing will be considered.LicenseDistributed under the GPL-3.0 License. SeeLICENSEfor more information.