Package: l4casadi
*Due to a bug, Hessians for functions with non-scalar outputs were wrong prior to version 1.3.0Learning 4 CasADi FrameworkL4CasADi enables the use of PyTorch models and functions in a CasADi graph while supporting CasADi code generation
capabilities. The only requirement on the PyTorch model is to be traceable and differentiable.Projects using L4CasADiInstallationExamplesReal-time L4CasADiIf you use this framework please cite our paper@article{salzmann2023neural,
  title={Real-time Neural-MPC: Deep Learning Model Predictive Control for Quadrotors and Agile Robotic Platforms},
  author={Salzmann, Tim and Kaufmann, Elia and Arrizabalaga, Jon and Pavone, Marco and Scaramuzza, Davide and Ryll, Markus},
  journal={IEEE Robotics and Automation Letters},
  doi={10.1109/LRA.2023.3246839},
  year={2023}
}Projects using L4CasADiReal-time Neural-MPC: Deep Learning Model Predictive Control for Quadrotors and Agile Robotic PlatformsPaper|CodeNeural Potential Field for Obstacle-Aware Local Motion PlanningPaper|Video|CodeIf your project is using L4CasADi and you would like to be featured here, please reach out.InstallationPrerequisitesIndependently if you install from source or via pip you will need to meet the following requirements:Working build system: CMake compatible C++ compiler.PyTorch (>=2.0) installation in your python environment.python -c "import torch; print(torch.__version__)"Pip Install (CPU Only)Ensure all build dependencies are installedsetuptools>=68.1
scikit-build>=0.17
cmake>=3.27
ninja>=1.11
torch>=2.0Runpip install l4casadi --no-build-isolationFrom Source (CPU Only)Clone the repositorygit clone https://github.com/Tim-Salzmann/l4casadi.gitAll build dependencies installed viapip install -r requirements_build.txtBuild from sourcepip install . --no-build-isolationThe--no-build-isolationflag is required for L4CasADi to find and link against the installed PyTorch.GPU (CUDA)CUDA installation requires nvcc to be installed which is part of the CUDA toolkit and can be installed on Linux viasudo apt-get -y install cuda-toolkit-XX-X(whereXX-Xis your installed Cuda version - e.g.12-3).
Once the CUDA toolkit is installed nvcc is commonly found at/usr/local/cuda/bin/nvcc.Install L4CasADi viaCUDACXX=<PATH_TO_NVCC> pip install l4casadi --no-build-isolationorCUDACXX=<PATH_TO_NVCC> pip install . --no-build-isolationto build from source.Exampleshttps://github.com/Tim-Salzmann/l4casadi/blob/23e07380e214f70b8932578317aa373d2216b57e/examples/readme.py#L28-L40Please note that onlycasadi.MXsymbolic variables are supported as input.Multi-input multi-output functions can be realized by concatenating the symbolic inputs when passing to the model and
splitting them inside the PyTorch function.To use GPU (CUDA) simply passdevice="cuda"to theL4CasADiconstructor.Further examples:Simple nonlinear programming with L4CasADi model as objective and constraints:examples/simple_nlp.pyL4CasADi in pure C(++) projects:examples/cpp_executableUse PyTorch L4CasADi Model in Matlab:examples/matlabNaive L4CasADiFor small models the overhead of context switches between PyTorch and CasADi can be significant even in pure C++.
Thus, L4CasADi provides aNaiveL4CasADiModulewhich will directly re-create the PyTorch computational graph in C++
and copies the weights. This, however is limited to a small subset of PyTorch operations - onlyMultiLayerPerceptronmodels and CPU inference are supported.https://github.com/Tim-Salzmann/l4casadi/blob/59876b190941c8d43286b6eb3d710add6f14a1d2/examples/naive/readme.py#L5-L9Batch DimensionIf your PyTorch model expects a batch dimension as first dimension (which most models do) you should passmodel_expects_batch_dim=Trueto theL4CasADiconstructor. TheMXinput to the L4CasADi component is then expected
to be a vector of shape[X, 1]. L4CasADi will add a batch dimension of1automatically such that the input to the
underlying PyTorch model is of shape[1, X].Acados IntegrationTo use this framework with Acados:Follow theinstallation instructions.Install thePython Interface.Ensure thatLD_LIBRARY_PATHis set correctly (DYLD_LIBRARY_PATHon MacOS).Ensure thatACADOS_SOURCE_DIRis set correctly.An example of how a PyTorch model can be used as dynamics model in the Acados framework for Model Predictive Control
can be found inexamples/acados.pyTo use L4CasADi with Acados you will have to setmodel_external_shared_lib_dirandmodel_external_shared_lib_namein theAcadosOcp.solver_optionsaccordingly.ocp.solver_options.model_external_shared_lib_dir = l4c_model.shared_lib_dir
ocp.solver_options.model_external_shared_lib_name = l4c_model.namehttps://github.com/Tim-Salzmann/l4casadi/blob/421de6ef408267eed0fd2519248b2152b610d2cc/examples/acados.py#L156-L160Real-time L4CasADiReal-time L4Casadi (formerApproximatedapproach inML-CasADi) is the underlying framework poweringReal-time Neural-MPC. It replaces complex models with local Taylor approximations.
For certain optimization procedures (such as MPC with multiple shooting nodes) this can lead to improved optimization times.
However,Real-time L4Casadi, comes with many restrictions (only Python, no C(++) code generation, ...) and is therefore not
a one-to-one replacement forL4Casadi. Rather it is a complementary framework for certain special use cases.More informationhere.https://github.com/Tim-Salzmann/l4casadi/blob/62219f61375af4c4133167f1d8b138d90f678c32/l4casadi/realtime/examples/readme.py#L32-L43Warm UpNote that PyTorch builds the graph on first execution. Thus, the first call(s) to the CasADi function will be slow.
You can warm up to the execution graph by calling the generated CasADi function one or multiple times before using it.RoadmapFurther development of this framework will be prioritized by popular demand. If a feature is important to your work
please get in contact or create a pull request.Possible upcoming features include:- Explicit multi input, multi output functions.