Minimizing a function in c++

username: Omroth
cooked: <p>Hi all.</p>
<p>I have a function f(X) which takes a 1d tensor and returns a float.  I would like to attempt to find an X which minimizes f(X) using the pytorch c++ api.</p>
<p>I’ve seen a couple of mentions here of modelling the function with pytorch modules, but I’m having difficulty getting my head around the concept - can anyone help?</p>
<p>Thanks,<br>
Ian</p>

username: Omroth
cooked: <p>Here is my first attempt, however the SGD optimizer doesn’t like taking a tensor (where the tensor is the values I’m trying to optimise), failing with “no matching function for call to ‘torch::optim::SGD::SGD(at::Tensor&amp;, double)”</p>
<pre><code class="lang-auto">torch::Tensor dynamic_parameters = torch::zeros({1}, torch::dtype(torch::kFloat64).requires_grad(true));

{
    torch::NoGradGuard guard;
    dynamic_parameters[0] = initial_guess;
}

torch::optim::SGD optimizer(dynamic_parameters, /*lr=*/0.01);

#define N_STEPS 20

for (int s = 0; s &lt; N_STEPS; s ++)
{
    torch::Tensor result = torch::zeros({1}, torch::dtype(torch::kFloat64).requires_grad(true));

    torch::Tensor magnitudes = torch::zeros({N_V}, torch::dtype(torch::kFloat64).requires_grad(false));

    magnitudes += gState.magnitudes_ones_tensor * dynamic_parameters[0];

    torch::Tensor synth = gState.dots * magnitudes;

    torch::Tensor differences = torch::abs(real - synth);

    result[0] = torch::sum(differences);

    torch::Tensor loss = torch::nll_loss(result, prediction);

    loss.backward();

    optimizer.step();
}
</code></pre>

username: tom
cooked: <p>Neither do the Python optimizers take a single tensor.<br>
You can take pass in a <a href="https://pytorch.org/cppdocs/api/classtorch_1_1optim_1_1_s_g_d.html#_CPPv4N5torch5optim3SGD3SGDENSt6vectorI6TensorEE10SGDOptions" rel="noopener nofollow ugc">vector of tensors</a>, tough (<code>{dynamic_parameters}</code> might do the trick).</p>
<p>Best regards</p>
<p>Thomas</p>

username: Omroth
cooked: <p>Thank you, that compiles now.</p>
<p>I’m compiling a python module using pybind11, and now when I <em>import</em> my module (which imported ok before I started trying this pytorch stuff) I get:</p>
<p>importerror: undefined symbol: _ZTVN5torch5optim9OptimizerE</p>
<p>I’ll try to debug in c++, but I was wondering if you’d seen anything like that before?</p>

username: Omroth
cooked: <p>I include pytorch in the .so with this:</p>
<p><span class="hashtag">#include</span> &lt;torch/torch.h&gt;</p>
<p>But the .so that is created is 10 megabytes… I’m unclear whether it is actually including my built version of PyTorch?  It seems unlikely.</p>

username: tom
cooked: <p>No you need to link to it. The easiest is to use cmake.</p>

username: Omroth
cooked: <p>Thanks for your help Tom.  This is my distutils script - is there an easy line to link it in to Python’s satisfaction?</p>
<pre><code class="lang-auto">from distutils.core import setup, Extension


def configuration(parent_package='', top_path=None):
      import numpy
      from numpy.distutils.misc_util import Configuration
      from numpy.distutils.misc_util import get_info

      #Necessary for the half-float d-type.
      info = get_info('npymath')

      config = Configuration('',
                             parent_package,
                             top_path)
      config.add_extension('reson8',
                           ['reson8.cpp'],
                           extra_info=info,
                           include_dirs=["/home/ian/anaconda3/lib/python3.7/site-packages/pybind11/include",
                                          "/home/ian/anaconda3/lib/python3.8/site-packages/pybind11/include",
                                          "/home/ian/dev/hedgey/Engine/lib/libtorch/include",
                                          "/home/ian/dev/hedgey/Engine/lib/libtorch/include/torch/csrc/api/include"])

      return config


if __name__ == "__main__":
      from numpy.distutils.core import setup
      setup(configuration=configuration)
</code></pre>

username: tom
cooked: <p>Probably using the <a href="https://pytorch.org/docs/stable/cpp_extension.html#torch.utils.cpp_extension.CppExtension" rel="noopener nofollow ugc">CppExtension/CUDAExtension</a> helps set that up.</p>

username: Omroth
cooked: <p>I’ve changed my setup.py to use the CppExtension:</p>
<pre><code class="lang-auto">from distutils.core import setup, Extension
from torch.utils.cpp_extension import BuildExtension, CppExtension

def configuration(parent_package='', top_path=None):
      import numpy
      from numpy.distutils.misc_util import Configuration
      from numpy.distutils.misc_util import get_info

      #Necessary for the half-float d-type.
      info = get_info('npymath')

      config = Configuration('',
                             parent_package,
                             top_path)

      config.ext_modules.append(CppExtension(
                name='reson8',
                sources=['reson8.cpp'],
                extra_info=info,
                extra_compile_args=['-g'],
                include_dirs=["/home/ian/anaconda3/lib/python3.7/site-packages/pybind11/include",
                                          "/home/ian/anaconda3/lib/python3.8/site-packages/pybind11/include",
                                          "/home/ian/dev/hedgey/Engine/lib/libtorch/include",
                                          "/home/ian/dev/hedgey/Engine/lib/libtorch/include/torch/csrc/api/include"]
                                ))

      return config


if __name__ == "__main__":
      from numpy.distutils.core import setup
      setup(configuration=configuration)
</code></pre>
<p>But I get a similar error:</p>
<pre><code class="lang-auto">ImportError: reson8.so: undefined symbol: _ZTIN3c1021AutogradMetaInterfaceE

</code></pre>
<p>So there’s still a build problem…</p>

username: Omroth
cooked: <p>Well, it builds fine, it just isn’t including something it needs.</p>
<p>I also added the pytorch include directory to <code>LD_LIBRARY_PATH</code>, but that didn’t help.</p>

username: tom
cooked: <p>If you get the error on import, you might have a version mismatch somewhere from having several PyTorch versions lying around?</p>

username: Omroth
cooked: <p>I suppose… but I don’t know how to fix that easily.  The .so should be using the version of pytorch from the c++ import, not the current PIP version right.</p>

username: tom
cooked: <p>My standard approach is to use the libtorch copy that comes with the Python packages for compilation, but I only have self-built PyTorch…</p>

username: Omroth
cooked: <p>I have self-built pytorch - can you tell me exactly what you do?</p>

username: Omroth
cooked: <p>I managed to fix this in the end - I posted the answer here:</p>
<p><a href="https://stackoverflow.com/questions/65774829/how-do-i-link-in-all-of-pytorch-with-a-pybind11-so/65828949#65828949" rel="noopener nofollow ugc">https://stackoverflow.com/questions/65774829/how-do-i-link-in-all-of-pytorch-with-a-pybind11-so/65828949#65828949</a></p>