How can we use autograd when using the function implemented in c

username: geunseok_choi
cooked: <p>Hi, I want to use the function implemented by the c language</p>
<p>First the overall flow is this<br>
input --&gt; NN --&gt; param --&gt; [function(C language) --&gt; output] --&gt; loss(output - reference)<br>
[] is done with c.</p>
<p>It’s too time-consuming to implement the function implemented by c into python.<br>
Also, the calculation of the function is specialized in c because of intel MKL.</p>
<p>So I tried to use ctypes to send arrays from python to c.<br>
But in this case, I have to detach tensor and autograd.</p>
<p>After calculation in c language, I send arrays to python and make it be tensor type.<br>
But when I tried to train NN, I can’t update the weight.<br>
This is of course because the autograd is not existed.</p>
<p>Here comes the problem.<br>
First, I don’t know how to make autograd of function, c part.<br>
Second, I don’t know how to update the last of the part, the NN part without using loss.backward.</p>
<p>how can I do it?<br>
here is the code</p>
<pre><code>for epoch in range(num_epoch):
	scheduler.step
	running_loss = 0

	# train
	for i, (inputs, raws_in, targets, raws_out) in enumerate(train_loader):
		inputs = inputs.to(device)
		targets = targets.to(device)			

		# Polynomial model - python version
		# params = model(inputs)
		# inputs = torch.stack([inputs[:, 0] * inputs[:, 0], inputs[:, 0], torch.FloatTensor([1, 1, 1, 1, 1, 1]).to(device), inputs[:, 1]])
		# outputs = params.matmul(inputs)
		# outputs = torch.diag(outputs).view([-1, 1])

		# sc3d code input - c
		params = model(inputs)
		grad = params.grad

		print(inputs)
		print(params)
		print('----------------------------------------------')

		params_c = params.to(torch.device('cpu'))
		params_c = params_c.detach().numpy()

		inputs_c = torch.stack([inputs[:, 0] * inputs[:, 0], inputs[:, 0], torch.FloatTensor([1, 1, 1, 1, 1, 1]).to(device), inputs[:, 1]])
		inputs_c = inputs_c.to(torch.device('cpu'))
		inputs_c = inputs_c.detach().numpy()

		libname = os.path.abspath(os.path.join(os.path.dirname(__file__), "sc3d.so"))
		LIBC = ctypes.CDLL(libname)

		_double_pp = np.ctypeslib.ndpointer(dtype=np.uintp, ndim=1)
		# _pp = np.ctypeslib.ndpointer(dtype=ctypes.c_float, ndim=1)

		LIBC.polyno.argtypes = [ctypes.c_int, ctypes.c_int, _double_pp, _double_pp, _double_pp] 
		LIBC.polyno.restype = None
		
		params_pp = (params_c.__array_interface__['data'][0] + np.arange(params_c.shape[0])*params_c.strides[0]).astype(np.uintp) 
		inputs_pp = (inputs_c.__array_interface__['data'][0] + np.arange(inputs_c.shape[0])*inputs_c.strides[0]).astype(np.uintp) 
		
		m = ctypes.c_int(params_pp.shape[0])
		n = ctypes.c_int(inputs_pp.shape[0]) 

		# outputs = np.zeros_like(m, m) # param(6*4) * input(4*6)
		outputs = np.zeros((6, 1)) # param(6*4) * input(4*6)
		outputs_pp = (outputs.__array_interface__['data'][0] + np.arange(outputs.shape[0])*outputs.strides[0]).astype(np.uintp)

		LIBC.polyno(m, n, params_pp, inputs_pp, outputs_pp)
		outputs = torch.FloatTensor(outputs).to(device)
		# outputs.requires_grad_(True)

		print(inputs_c)
		print(params_c)
		print(outputs)
		print(targets)

		# Forward pass
		loss = criterion(outputs, targets)
		# loss = torch.sqrt(criterion(outputs, targets))

		# Backward and optimize
		optimizer.zero_grad()
		loss.backward(gradient=params.grad)
		
		optimizer.step()
		running_loss += float(losses(outputs, targets))
		if (i + 1) % train_step == 0:
			print('Epoch [{}/{}], Step [{}/{}], Train Loss: {}'
						.format(epoch + 1, num_epoch, i + 1, train_step, running_loss/train_step), end='')
	train_losses.append(running_loss / train_step)</code></pre>

username: Nikronic
cooked: <p>Hi,</p>
<p>In general, I am afraid that you cannot do that. Only operations within torch will be tracked by autograd which enables you to do the backward and update params.<br>
But there is one exception, which is the fact that you can extend <code>autograd.Function</code> and use C/C++ or python` codes easily in it to enable autograd tracking for computations that pytorch do not support internally. For instance, you cannot use numpy and still track grads, but if you can obtain derivitives manually, then you can simply create a box that does that and let you to do the backward.</p>
<p>Here are the links that help you throughout the process and I think documentation is clear enough for a start:</p>
<ol>
<li>Learn about autograd:</li>
</ol>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://blog.paperspace.com/favicon.png" class="site-icon" width="15" height="15">
      <a href="https://blog.paperspace.com/pytorch-101-understanding-graphs-and-automatic-differentiation/" target="_blank" rel="noopener nofollow ugc" title="08:03PM - 29 May 2019">Paperspace Blog – 29 May 19</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:128/82;"><img src="https://blog.paperspace.com/content/images/2019/05/full_graph.png" class="thumbnail" width="128" height="82"></div>

<h3><a href="https://blog.paperspace.com/pytorch-101-understanding-graphs-and-automatic-differentiation/" target="_blank" rel="noopener nofollow ugc">PyTorch Basics: Understanding Autograd and Computation Graphs</a></h3>

<p>In this article, we dive into how PyTorch's Autograd engine performs automatic differentiation.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<ol start="2">
<li>extending pytorch (autograd, nn.module, etc):</li>
</ol>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/docs/stable/notes/extending.html#extending-torch-autograd" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="15" height="15">

<h3><a href="https://pytorch.org/docs/stable/notes/extending.html#extending-torch-autograd" target="_blank" rel="noopener nofollow ugc">Extending PyTorch — PyTorch 1.7.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<ol start="3">
<li>Tutorial on a custom layer/network with manual autograd</li>
</ol>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="15" height="15">

<h3><a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html" target="_blank" rel="noopener nofollow ugc">PyTorch: Defining New autograd Functions — PyTorch Tutorials 1.7.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<ol start="4">
<li>Extending pytorch by C++/cuda and creating binding directly from C++ for pytorch:</li>
</ol>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="15" height="15">

<h3><a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" target="_blank" rel="noopener nofollow ugc">Custom C++ and CUDA Extensions — PyTorch Tutorials 1.7.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/docs/stable/cpp_extension.html" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="15" height="15">

<h3><a href="https://pytorch.org/docs/stable/cpp_extension.html" target="_blank" rel="noopener nofollow ugc">torch.utils.cpp_extension — PyTorch 1.7.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Bests</p>

username: geunseok_choi
cooked: <p>Hi,<br>
Thank you for your kindness.</p>
<p>But still, there is one problem.<br>
I don’t know how to make the backward part.<br>
The backward part consisted of the gradient of the function is made with the eigenvalue problem.<br>
So, I can’t exactly describe what the gradient is.</p>
<p>In the end, how can I make the gradient of the black box in terms of autograd<br>
Here is the code</p>
<p>class sc3d(torch.autograd.Function):</p>
<pre><code>@staticmethod
def forward(ctx, input, param):

	ctx.save_for_backward(input, param)

	param_c = param.to(torch.device('cpu'))
	param_c = param_c.detach().numpy()
	
	input_c = torch.stack([input[:, 0] * input[:, 0], input[:, 0], torch.FloatTensor([1, 1, 1, 1, 1, 1]).to(device), input[:, 1]])
	input_c = input_c.to(torch.device('cpu'))
	input_c = input_c.detach().numpy()
	
	libname = os.path.abspath(os.path.join(os.path.dirname(__file__), "sc3d.so"))
	LIBC = ctypes.CDLL(libname)

	_double_pp = np.ctypeslib.ndpointer(dtype=np.uintp, ndim=1)
	# _pp = np.ctypeslib.ndpointer(dtype=ctypes.c_float, ndim=1)
	
	LIBC.polyno.argtypes = [ctypes.c_int, ctypes.c_int, _double_pp, _double_pp, _double_pp] 
	LIBC.polyno.restype = None
	
	param_pp = (param_c.__array_interface__['data'][0] + np.arange(param_c.shape[0])*param_c.strides[0]).astype(np.uintp) 
	input_pp = (input_c.__array_interface__['data'][0] + np.arange(input_c.shape[0])*input_c.strides[0]).astype(np.uintp) 
	
	m = ctypes.c_int(param_pp.shape[0])
	n = ctypes.c_int(input_pp.shape[0]) 
	
	# outputs = np.zeros_like(m, m) # param(6*4) * input(4*6)
	output = np.zeros((6, 1)) # param(6*4) * input(4*6)
	output_pp = (output.__array_interface__['data'][0] + np.arange(output.shape[0])*output.strides[0]).astype(np.uintp)
	
	LIBC.polyno(m, n, param_pp, input_pp, output_pp)
	output = torch.FloatTensor(output).to(device)
	# outputs.requires_grad_(True)

	return output

@staticmethod
def backward(ctx, grad_output):

	input, param = ctx.saved_tensors
	grad_input = grad_param = None

	# calculate gradient part
	# grad_param = grad_output.clone()
	
	return grad_input, grad_param
</code></pre>
<p>Best regards,</p>
<p>Choi</p>

username: KFrank
cooked: <p>Hi Choi!</p>
<aside class="quote no-group" data-username="geunseok_choi" data-post="4" data-topic="104176" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/geunseok_choi/48/31544_2.png" class="avatar"> geunseok_choi:</div>
<blockquote>
<p>I don’t know how to make the backward part.<br>
The backward part consisted of the gradient of the function is made with the eigenvalue problem.</p>
</blockquote>
</aside>
<p>If your main roadblock really is just backpropagating through the<br>
“eigenvalue problem,” then you could perform that part of the<br>
calculation using pytorch’s <code>torch.eig()</code> (or <code>torch.symeig()</code>),<br>
and use autograd to backpropagate through the eigenvalue piece.</p>
<p>If, furthermore, the rest of your loss function can also be written with<br>
pytorch tensor functions, you could use autograd to backpropagate<br>
through the entire loss function.</p>
<p>But even if not, you could:</p>
<pre><code class="lang-plaintext">loss = (tensor functions) --&gt; c-function-A --&gt; torch.eig() --&gt; c-function-B --&gt; (more tensor functions)
</code></pre>
<p>If you can figure out the gradient for <code>c-function-A</code> and <code>c-function-B</code><br>
(even if you can’t for the eigenvalue problem), you can write your<br>
custom c functions with both <code>.forward()</code> and <code>.backward()</code> functions,<br>
and use autograd to backpropagate through the whole thing.  This way,<br>
you’re letting pytorch’s autograd do the “hard part” (namely, calculating<br>
the gradient of the eigenvalue problem) for you, while you write the<br>
gradients (the <code>.backward()</code> functions) for <code>c-function-A</code> and<br>
<code>c-function-B</code>, which you can’t, or don’t want to, write using pytorch<br>
tensor functions.</p>
<p>Best.</p>
<p>K. Frank</p>

username: geunseok_choi
cooked: <p>Hi<br>
Thanks for your answer.</p>
<p>But there is one more question.<br>
As you said, if I use c-function-A, then I have to calculate the gradient of c-function-A with the analytic form of c-function-A?</p>
<p>Is there any way to use the numerical form of c-function-A<br>
Like for example, can I make [ c-function-A(x-delta x) / x-delta x ] as a backward part?<br>
If I can, is it too slower than the analytic way?</p>
<p>Best regards,<br>
Choi</p>

username: KFrank
cooked: <p>Hi Choi!</p>
<aside class="quote no-group" data-username="geunseok_choi" data-post="6" data-topic="104176" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/geunseok_choi/48/31544_2.png" class="avatar"> geunseok_choi:</div>
<blockquote>
<p>Is there any way to use the numerical form of c-function-A<br>
Like for example, can I make [ c-function-A(x-delta x) / x-delta x ] as a backward part?</p>
</blockquote>
</aside>
<p>I’ve never tried this approach – using <em>numerical differentiation</em> to<br>
calculate <code>.backward()</code> – but I have thought about it.</p>
<p>Advantages:  Numerical differentiation is a very general-purpose tool,<br>
so you can apply it to lots of use cases.</p>
<p>Disadvantages:  A sledgehammer is also a general-purpose tool that<br>
can be “applied” to almost anything.  But it is a crude tool, so you<br>
may not get the results you want …</p>
<p>More concretely, numerical differentiation can be numerically tricky.<br>
The core issue is how do you choose your step size, <code>delta_x</code> so<br>
that you get a good estimate of the local derivative, but you don’t<br>
get too much round-off error when you take the finite differences?<br>
(Consider numerically differentiating <code>abs (x)</code> when your step size<br>
crosses <code>x = 0</code>.)</p>
<p>(This is a well-studied problem:  Many scientific subroutine libraries<br>
include numerical-differentiation algorithms, but if you look at them<br>
you will see that they are much more involved than the simple<br>
approach you outlined above.)</p>
<aside class="quote no-group">
<blockquote>
<p>If I can, is it too slower than the analytic way?</p>
</blockquote>
</aside>
<p>It is likely to be slower than a numerical implementation of the analytic<br>
derivative (by possibly a lot).</p>
<p>If you can calculate the derivative analytically – even if doing so is a<br>
fair amount of work – it will be worth doing it that way.  If it were me, I<br>
would only calculate <code>.backward()</code> numerically as a quick-and-dirty<br>
hack to test something or if I were really struggling with the analytic<br>
derivative.</p>
<p>Best.</p>
<p>K. Frank</p>

username: geunseok_choi
cooked: <p>Hi,<br>
thanks for your answer.</p>
<p>From your words, I decided to use python code only.<br>
But there is one more problem here.</p>
<p>there is no way to use eigenvalue solver with the complex problem right now</p><aside class="quote quote-modified" data-post="1" data-topic="71983">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ksteensig/48/21251_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-use-complex-eigenvalues/71983">How to use complex eigenvalues?</a> 
  </div>
  <blockquote>
    I have a nonsymmetric real matrix that I want to calculate the eigenvalues and matrices for. I use the torch.eig() method which works. 
But my eigenvalues are complex and I am unsure of how to handle this. 
I am using the following to go from eigenvalues/vectors and back. 
X = torch.rand(20,20)
D,T = torch.eig(X, eigenvectors=True)
T.mm(torch.diag(D)).mm(T.t())

But because D consists of complex eigenvalues, and in PyTorch a complex value is a vector of two values, I am unable to calculate the o…
  </blockquote>
</aside>

<p>Therefore, I’m considering to use an external library.</p><aside class="onebox githubfolder" data-onebox-src="https://github.com/williamFalcon/pytorch-complex-tensor/tree/master/pytorch_complex_tensor">
  <header class="source">

      <a href="https://github.com/williamFalcon/pytorch-complex-tensor/tree/master/pytorch_complex_tensor" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h3><a href="https://github.com/williamFalcon/pytorch-complex-tensor/tree/master/pytorch_complex_tensor" target="_blank" rel="noopener nofollow ugc"></a></h3>

  <p><a href="https://github.com/williamFalcon/pytorch-complex-tensor/tree/master/pytorch_complex_tensor" target="_blank" rel="noopener nofollow ugc">//github.com/williamFalcon/pytorch-complex-tensor/tree/master/pytorch_complex_tensor</a></p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Did you know any other way to use autograd with the complex eigenvalue problem?<br>
If not, is it possible in tensorflow?</p>
<p>Best regards,<br>
Choi</p>