"Convolve" a custom module/model on a tensor (forward a module on segments of array with striding)

username: mikolchon
cooked: <p>Say I have array <code>1,2,3,4,5,6,7,8,9,0</code>. I wish to “convolve” a custom module on the array. In other words, I want to forward a custom module on segments of the array with certain stride. For example, I may have a neural network and want to forward the above array taking input segments of length 3 and stride 2: on <code>1,2,3</code>, then <code>3,4,5</code>, then <code>5,6,7</code>, … then the final output would be the concatenation of the per-segment outputs.</p>
<p>I know one can easily implement this manually, but I want to precisely avoid the concatenation as this creates a copy. I was hoping there is a functionality in Pytorch where I can pass a custom module (instead of a linear filter) and perform the “convolution”, returning the already contiguous array.</p>

username: ptrblck
cooked: <p>You could use <code>tensor.unfold</code> to create these patches, but you would still need to allocate the memory for the output.<br>
If you want to return a contiguous tensor, it has to be created at some point, even in some internal PyTorch functions.<br>
Let me know, if I misunderstood your question.</p>

username: mikolchon
cooked: <p>Thank you very much. I think this is what I was looking for. But I am a bit confused by the provided example:</p>
<pre><code class="lang-python">&gt;&gt;&gt; # Convolution is equivalent with Unfold + Matrix Multiplication + Fold (or view to output shape)
&gt;&gt;&gt; inp = torch.randn(1, 3, 10, 12)
&gt;&gt;&gt; w = torch.randn(2, 3, 4, 5)
&gt;&gt;&gt; inp_unf = torch.nn.functional.unfold(inp, (4, 5))
&gt;&gt;&gt; out_unf = inp_unf.transpose(1, 2).matmul(w.view(w.size(0), -1).t()).transpose(1, 2)
&gt;&gt;&gt; out = torch.nn.functional.fold(out_unf, (7, 8), (1, 1))
&gt;&gt;&gt; # or equivalently (and avoiding a copy),
&gt;&gt;&gt; # out = out_unf.view(1, 2, 7, 8)
&gt;&gt;&gt; (torch.nn.functional.conv2d(inp, w) - out).abs().max()
tensor(1.9073e-06)
</code></pre>
<p>In this example, I’m not sure I understand why <code>torch.nn.functional.fold</code> is equivalent to using <code>view</code>. In this example <code>out_unf</code> is not contiguous (<code>out_unf.is_contiguous()</code> is <code>False</code>) because of the last transpose. My questions are:</p>
<p>a) In what cases are they equivalent?<br>
b) Shouldn’t <code>.view(...)</code> raise an error since <code>out_unf</code> is not contiguous?: <a href="https://discuss.pytorch.org/t/runtimeerror-input-is-not-contiguous/930/2" class="inline-onebox">RuntimeError: input is not contiguous</a></p>

username: ptrblck
cooked: <p>Generally, a dimension cannot span across two contiguous subspaces.<br>
<code>.view</code> will not necessarily raise an error, if the desired shape does not violate the first assumption.<br>
Also, note that <code>out</code> is not contiguous in this case either.</p>

username: mikolchon
cooked: <p>Thanks for the clarification.</p>
<p>Does <code>unfold</code> create a copy of the original array? If so I don’t think I will be able to use it due to my limited memory. My aim is to evaluate some module/model on an input array in a sliding window fashion. I could do this with a loop, then concatenate at the end. For example:</p>
<pre><code class="lang-python">suboutputs = []
for i in range(x_length - window_length):
  suboutputs.append(model(x[i:i+window_length])
output = torch.cat(suboutputs)  # creates a copy
</code></pre>
<p>Is there a way to avoid the concatenation?</p>