Overwriting from_blob tensor data

username: tmaric
cooked: <p>What do words “exposes” and “without taking ownership” in <code>from_blob</code></p>
<blockquote>
<p>Exposes the given <code>data</code> as a <code>Tensor</code> <strong>without taking ownership of the original data</strong>.</p>
</blockquote>
<p><a href="https://pytorch.org/cppdocs/api/function_namespacetorch_1ac009244049812a3efdf4605d19c5e79b.html?highlight=from_blob" rel="noopener nofollow ugc">https://pytorch.org/cppdocs/api/function_namespacetorch_1ac009244049812a3efdf4605d19c5e79b.html?highlight=from_blob</a></p>
<p>actually mean? Are internally copy perations done on <code>data</code> by</p>
<p><code>at::[Tensor](https://pytorch.org/cppdocs/api/classat_1_1_tensor.html#_CPPv4N2at6TensorE) </code>torch::<code> </code>from_blob<code> (void **data* , at::IntArrayRef *sizes* , at::IntArrayRef *strides* , *const* at::TensorOptions &amp;*options* = at::TensorOptions())</code></p>
<p>?</p>
<p>Is it possible to have a huge C array in memory and use <code>from_blob</code> to “view” it as a <code>Tensor</code> and also <em>modify the arrays data</em>? Example:</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;iostream&gt;
#include &lt;array&gt; 
    
int main ()
{
    float array[] = {1.23, 2.34, 3.45, 4.56, 5.67};
    auto options = torch::TensorOptions().dtype(torch::kFloat32);
    torch::Tensor t1 = torch::from_blob(array, {5}, options);
    std::cout &lt;&lt; t1 &lt;&lt; std::endl;

    t1 = torch::ones_like(t1); 

    std::cout &lt;&lt; t1 &lt;&lt; std::endl;

    for (auto el : array)
        std::cout &lt;&lt; el &lt;&lt; std::endl;
    return 0;
}
</code></pre>
<p>I expected this code to overwrite the array elements, but the output shows they are not overwritten,</p>
<pre><code class="lang-auto"> ./example-app 
 1.2300
 2.3400
 3.4500
 4.5600
 5.6700
[ CPUFloatType{5} ]
 1
 1
 1
 1
 1
[ CPUFloatType{5} ]
1.23
2.34
3.45
4.56
5.67
</code></pre>
<p>Are all <code>Tensor</code> constructors deep-copy operations, or is there one that imposes a “write” view on an array?</p>
<p>If it is not possible to construct a <code>Tensor</code> without copying <code>data</code>, what would be the the best way to assign values of <code>t1</code> (in this simple example, the <code>ones_like</code>) to <code>array</code>? Is there a way to get a pointer to the data from <code>Tensor</code>?</p>

username: ptrblck
cooked: <p><code>from_blob</code> should create a “view” to the underlying object and inplace operations should thus also change the original array.<br>
However, <code>torch.ones_like</code> will not create another view but a new tensor and will override the <code>t1</code> variable.</p>

username: tmaric
cooked: <p>OK, I just find that kind of strange: yes, <code>ones_like</code> generates another tensor, but shouldn’t the assignment operator of <code>Tensor</code> simply assign whatever <code>ones_like</code> creates to <code>t1</code> without altering its view? Something like</p>
<pre><code class="lang-auto">void Tensor::operator=(Tensor const&amp; rhs){
  this.data = rhs.data // in a sense
}
</code></pre>
<p>or</p>
<pre><code class="lang-auto">void Tensor::operator=(Tensor&amp;&amp; rhs){
  this.data = std::move(rhs.data) // in a sense
}
</code></pre>
<p>My real scenario looks something like</p>
<pre><code class="lang-auto">void* huge_data = complexObject.pointer(); 
Tensor nn_forward = from_blob(huge_data) 

// train network 

nn_forward = nn-&gt;forward()
</code></pre>
<p>So I am guessing now after seeing your answer that <code>nn-&gt;forward()</code> also creates a new Tensor and overrides <code>nn_forward</code>, and removes the non-const view of <code>huge_data</code>. This means I need to do a copy-assign from the new <code>nn_forward</code> to <code>huge_data</code> and it also means that I am storing 2 <code>huge_data</code> variables in memory - one in <code>complexObject</code> that I want to modify, and another one genreated by <code>forward()</code>.</p>
<p>Is there a way to circumvent this and make forward write to <code>huge_data</code> via the non-const view to <code>huge_data</code> available in <code>nn_forward</code> somehow?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="tmaric" data-post="3" data-topic="153031">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/f19dbf/48.png" class="avatar"> tmaric:</div>
<blockquote>
<p>Is there a way to circumvent this and make forward write to <code>huge_data</code> via the non-const view to <code>huge_data</code> available in <code>nn_forward</code> somehow?</p>
</blockquote>
</aside>
<p>If you want to write directly to <code>huge_data</code>, you could check if the last operation in your model supports the <code>out</code> argument so that you could pass this tensor to it (assuming the shape etc. is correct).</p>

username: tmaric
cooked: <p>I’m using Sequential, and <a href="https://pytorch.org/cppdocs/api/classtorch_1_1nn_1_1_sequential_impl.html#classtorch_1_1nn_1_1_sequential_impl" rel="noopener nofollow ugc">(using Python as pseudocode)</a></p>
<pre><code class="lang-auto">def forward(sequential, input):
  for module in sequential:
    input = module(input)
  return input
</code></pre>
<p>This looks like <a href="https://pytorch.org/cppdocs/api/classtorch_1_1nn_1_1_sequential_impl.html#_CPPv4I0DpEN5torch2nn14SequentialImpl7forwardE10ReturnTypeDpRR10InputTypes" rel="noopener nofollow ugc">Sequential does not offer the output variable as an argument to forward</a>, is this true?</p>

username: tmaric
cooked: <p>I’ve solved it with a for loop (<code>forAll</code> is custom OpenFOAM’s macro for the standard C++ for loop)</p>
<pre><code class="lang-auto">    // Evaluate the best NN. 
    //  - Reinterpret OpenFOAM's output volScalarField as scalar* array 
    volScalarField::pointer vf_nn_data = vf_nn.ref().data();
    //  - Use the scalar* (volScalarField::pointer) to view 
    //    the volScalarField as torch::Tensor without copying data. 
    torch::Tensor vf_nn_tensor = torch::from_blob(vf_nn_data, {vf.size()});
    //  - Evaluate the volumeScalarField vf_nn using the best NN model.
    vf_nn_tensor = nn_best-&gt;forward(cc_tensor);
    //  - FIXME: 2022-06-01, the C++ PyTorch API does not overwrite the blob object.
    //           If a Model is coded by inheritance, maybe forward(input, output) is
    //           available, that overwrites the data in vf_nn by acting on the 
    //           non-const view of the data given by vf_nn_tensor. TM.
    forAll(vf_nn, cellI)
    {
        vf_nn[cellI] = vf_nn_tensor[cellI].item&lt;double&gt;();
    }
    //  - Evaluate the vf_nn boundary conditions. 
    vf_nn.correctBoundaryConditions()
</code></pre>
<p>in case someone else requires this, but I am not sure this is fast and it is definitely not clean… Also, it is relatively simple for a scalar data transfer to the source of <code>huge_data</code>, for vectors and other tensors it becomes cumbersome. I’ve checked the assignment operators in the C++ API, the documentation doesn’t say anything about data ownership transfer…and I didn’t find a deep copy option, std::copy fails because missing begin() end() mfunctions in Tensor.</p>

username: Gonen_Raveh
cooked: <p>Hi tmaric and ptrblck. Does this solution means that libtorch is <strong>filling</strong> the Module’s output directly to the given GPU/CUDA buffer?</p>
<p>I have a code like this:<br>
// wrap the user-provided GPU Pointer as an output PyTorch Tensor.<br>
at::Tensor predTensor = torch::from_blob((void*)output_ptr, c10::IntArrayRef(pytorchTensorShapeRefPtr), tensor_options);<br>
std::move(predTensor) = mModule.forward(inputs).toTensor();</p>
<p>it works but I still see a DeviceToDevice Data copy from GPU to GPU. So does it mean that it actually did not work? I am trying to <strong>avoid any d2d gpu copy</strong>.</p>