Conv2d Argument Error For Custom Dataset

kavi14: <p>My task is to create a CNN model which takes in an (1)image, (2)robot path and (3)robot action and output a (4)reward value.</p>
<p>(1), (2), (3) are all 2D binary matrices which are output from a simulator I made using Python.</p>
<p>E.g. of a data point containing (1), (2), (3) and (4):</p>
<p>[ [ [image], [robot_path], [robot_action] ] , reward_value ]</p>
<p><strong>image</strong> is a list of 3 binary matrices = torch.Size[(5, 5, 3)]<br>
<strong>robot_path</strong> is a single binary matrix = torch.Size[(5, 5)]<br>
<strong>robot_actions</strong> is a list of 3 binary matrices = torch.Size[(5, 5, 3)]<br>
<strong>reward_value</strong> is an integer.</p>
<p>I created a custom Dataset class and also the DataLoader.<br>
Then I followed the code given here:<a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Training a Classifier — PyTorch Tutorials 1.9.0+cu102 documentation</a></p>
<p>This is the output when I print a single data point from my Dataset object:</p>
<pre><code class="lang-auto">[[[tensor([[1, 1, 1, 1, 1],
        [0, 1, 0, 0, 0],
        [0, 0, 1, 0, 0],
        [0, 0, 0, 1, 0],
        [1, 1, 1, 1, 1]]), tensor([[1, 1, 1, 1, 1],
        [0, 1, 0, 0, 0],
        [0, 0, 1, 0, 0],
        [0, 0, 0, 1, 0],
        [1, 1, 1, 1, 1]]), tensor([[1, 1, 1, 1, 1],
        [0, 1, 0, 0, 0],
        [0, 0, 1, 0, 0],
        [0, 0, 0, 1, 0],
        [1, 1, 1, 1, 1]])], tensor([[1, 1, 1, 1, 1],
        [0, 1, 0, 0, 0],
        [0, 0, 1, 0, 0],
        [0, 0, 0, 1, 0],
        [1, 1, 1, 1, 1]]), [tensor([[1, 1, 1, 1, 1],
        [0, 1, 0, 0, 0],
        [0, 0, 1, 0, 0],
        [0, 0, 0, 1, 0],
        [1, 1, 1, 1, 1]]), tensor([[1, 1, 1, 1, 1],
        [0, 1, 0, 0, 0],
        [0, 0, 1, 0, 0],
        [0, 0, 0, 1, 0],
        [1, 1, 1, 1, 1]]), tensor([[1, 1, 1, 1, 1],
        [0, 1, 0, 0, 0],
        [0, 0, 1, 0, 0],
        [0, 0, 0, 1, 0],
        [1, 1, 1, 1, 1]])]], 7]
</code></pre>
<p>However, I get the following error when running the code:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_78598/322194925.py in &lt;module&gt;
     12 
     13         # forward + backward + optimize
---&gt; 14         outputs = net(inputs)
     15 #         loss = criterion(outputs, labels)
     16 #         loss.backward()

~/.local/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

/tmp/ipykernel_78598/3992303050.py in forward(self, x)
     13 
     14     def forward(self, x):
---&gt; 15         x = self.pool(F.relu(self.conv1(x)))
     16         x = self.pool(F.relu(self.conv2(x)))
     17         x = torch.flatten(x, 1) # flatten all dimensions except batch

~/.local/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

~/.local/lib/python3.8/site-packages/torch/nn/modules/conv.py in forward(self, input)
    441 
    442     def forward(self, input: Tensor) -&gt; Tensor:
--&gt; 443         return self._conv_forward(input, self.weight, self.bias)
    444 
    445 class Conv3d(_ConvNd):

~/.local/lib/python3.8/site-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight, bias)
    437                             weight, bias, self.stride,
    438                             _pair(0), self.dilation, self.groups)
--&gt; 439         return F.conv2d(input, weight, bias, self.stride,
    440                         self.padding, self.dilation, self.groups)
    441 

TypeError: conv2d() received an invalid combination of arguments - got (list, Parameter, Parameter, tuple, tuple, tuple, int), but expected one of:
 * (Tensor input, Tensor weight, Tensor bias, tuple of ints stride, tuple of ints padding, tuple of ints dilation, int groups)
      didn't match because some of the arguments have invalid types: (!list!, !Parameter!, !Parameter!, !tuple!, !tuple!, !tuple!, int)
 * (Tensor input, Tensor weight, Tensor bias, tuple of ints stride, str padding, tuple of ints dilation, int groups)
      didn't match because some of the arguments have invalid types: (!list!, !Parameter!, !Parameter!, !tuple!, !tuple!, !tuple!, int)
</code></pre>
<p>I did realize that the conv2d takes in images of 3 channels and I think mine has about 7 channels, but this still did not work.</p>
<p>What can I do about this?</p>

KFrank: <p>Hi Kavi!</p>
<aside class="quote no-group quote-modified" data-username="kavi14" data-post="1" data-topic="128558" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/34f0e0/48.png" class="avatar"> kavi14:</div>
<blockquote>
<p>[ [ [image], [robot_path], [robot_action] ] , reward_value ]</p>
<p><strong>image</strong> is a list of 3 binary matrices = torch.Size[(5, 5, 3)]<br>
<strong>robot_path</strong> is a single binary matrix = torch.Size[(5, 5)]<br>
<strong>robot_actions</strong> is a list of 3 binary matrices = torch.Size[(5, 5, 3)]<br>
<strong>reward_value</strong> is an integer.</p>
<p>…</p>
<p>I did realize that the conv2d takes in images of 3 channels and I think mine has about 7 channels, but this still did not work.</p>
</blockquote>
</aside>
<p>I don’t know whether your specific use case makes sense, but to<br>
answer your immediate question:</p>
<p>Let <code>image</code> be a three-channel image (say RGB) of shape<br>
<code>[height = 5, width = 5, nChannel = 3]</code>;<br>
let <code>robot_path</code> be a single-channel image (with no explicit <code>nChannel</code><br>
dimension) of shape <code>[height = 5, width = 5]</code>;<br>
and let <code>robot_actions</code> be a three-channel image of shape<br>
<code>[height = 5, width = 5, nChannel = 3]</code>.</p>
<p>We will <code>unsqueeze()</code> <code>robot_path</code> to give it an explicit <code>nChannel</code><br>
dimension of size <code>nChannel = 1</code>, and then <code>cat()</code> the three images<br>
together to produce a single seven-channel image, <code>input</code>, of shape<br>
<code>[height = 5, width = 5, nChannel = 7]</code>.</p>
<p>We then permute <code>input</code>’s dimensions so that <code>nChannel</code> comes<br>
before <code>height</code> and <code>width</code> and <code>unsqueeze()</code> it so that it has a leading<br>
<code>nBatch</code> dimension of size <code>nBatch = 1</code>.  We do this because <code>Conv2d</code><br>
requires an <code>nBatch</code> dimension and also requires that the <code>nChannel</code><br>
dimension precede the <code>height</code> and <code>width</code> dimensions.</p>
<p>You would now want your convolution neural network (CNN) to start<br>
with a <code>Conv2d</code> layer that has <code>in_channels = 7</code>.</p>
<p>Here is an example that shows how to <code>cat()</code> your input images<br>
together and pass them to an appropriate <code>Conv2d</code>.  (The rest of<br>
the CNN is left as an exercise for the reader.)</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; image = torch.randn (5, 5, 3)
&gt;&gt;&gt; robot_path = torch.randn (5, 5)
&gt;&gt;&gt; robot_actions = torch.randn (5, 5, 3)
&gt;&gt;&gt; image.shape
torch.Size([5, 5, 3])
&gt;&gt;&gt; robot_path.shape
torch.Size([5, 5])
&gt;&gt;&gt; robot_actions.shape
torch.Size([5, 5, 3])
&gt;&gt;&gt; input = torch.cat ([image, robot_path.unsqueeze (-1), robot_actions], dim = -1)
&gt;&gt;&gt; input.shape
torch.Size([5, 5, 7])
&gt;&gt;&gt; input = input.permute (2, 0, 1).unsqueeze (0)
&gt;&gt;&gt; input.shape
torch.Size([1, 7, 5, 5])
&gt;&gt;&gt; conv = torch.nn.Conv2d (in_channels = 7, out_channels = 2, kernel_size = 3)
&gt;&gt;&gt; conv (input).shape
torch.Size([1, 2, 3, 3])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

kavi14: <p>Thanks Frank, I solved this!</p>

Kappa: <p>How did you solve having a similiar problem</p>