RuntimeError: Expected tensor to have size 256 at dimension 1, but got size 3 for argument #2 'batch2' (while checking arguments for bmm)

username: CVAIDL
cooked: <p>When I execute the program mapping_mat = torch.matmul(frame, scaling_mat)，there is a RuntimeError: Expected tensor to have size 256 at dimension 1, but got size 3 for argument <span class="hashtag">#2</span> ‘batch2’ (while checking arguments for bmm).<br>
frame is torch.Size([1, 3, 256, 256])<br>
scaling_mat is torch.Size([1, 3, 3])</p>

username: Kushaj
cooked: <p><code>scaling_mat</code> should be of size (1, 256, 3) for <code>matmul</code> to work.</p>
<p>Check the pytorch <a href="https://pytorch.org/docs/master/generated/torch.matmul.html" rel="nofollow noopener">doc</a> for input dimension instructions.</p>

username: CVAIDL
cooked: <p>@ Kushaj   Thanks for your reply. frame is an input image, and templateimgw is obtained by warping the template with a homography.<br>
I have wrote the programm to compute the IOU between two images, but there are some errors in these codes below.<br>
ZOOM_OUT_SCALE = 4<br>
batch_size = frame.shape[0]<br>
fake_template = torch.ones([batch_size, 1, 74 * ZOOM_OUT_SCALE, 115 * ZOOM_OUT_SCALE], device=frame.device)<br>
scaling_mat = torch.eye(3, device=frame.device).repeat(batch_size, 1, 1)<br>
scaling_mat[:, 0, 0] = scaling_mat[:, 1, 1] = ZOOM_OUT_SCALE<br>
target_mask = warp.warp_image(fake_template, scaling_mat, out_shape=fake_template.shape[-2:])<br>
mapping_mat = torch.matmul(frame, scaling_mat)<br>
mapping_mat = torch.matmul(templateimgw.inverse(), mapping_mat)output_mask = warp.warp_image(fake_template, mapping_mat, out_shape=fake_template.shape[-2:])<br>
output_mask = (output_mask &gt;= 0.5).float()<br>
target_mask = (target_mask &gt;= 0.5).float()</p>

username: Kushaj
cooked: <p>What is the error? Show the complete log for easy debugging.</p>

username: CVAIDL
cooked: <p>I want to compute the IOU between two images “frame” and “templateimgw”<br>
The code is as follow.</p>
<p>ZOOM_OUT_SCALE = 4<br>
batch_size = frame.shape[0]<br>
fake_template = torch.ones([batch_size, 1, 74 * ZOOM_OUT_SCALE, 115 * ZOOM_OUT_SCALE], device=frame.device)<br>
scaling_mat = torch.eye(3, device=frame.device).repeat(batch_size, 1, 1)<br>
scaling_mat[:, 0, 0] = scaling_mat[:, 1, 1] = ZOOM_OUT_SCALE<br>
target_mask = warp.warp_image(fake_template, scaling_mat, out_shape=fake_template.shape[-2:])<br>
mapping_mat = torch.matmul(frame, scaling_mat)<br>
mapping_mat = torch.matmul(templateimgw.inverse(), mapping_mat)<br>
output_mask = warp.warp_image(fake_template, mapping_mat, out_shape = fake_template.shape[-2:])<br>
output_mask = (output_mask &gt;= 0.5).float()<br>
target_mask = (target_mask &gt;= 0.5).float()</p>
<p>The error is as follow.</p>
<p>Traceback (most recent call last):<br>
File “ioucompute.py”, line 199, in mapping_mat = torch.matmul(frame, scaling_mat)<br>
RuntimeError: Expected tensor to have size 256 at dimension 1, but got size 3 for argument <span class="hashtag">#2</span> ‘batch2’ (while checking arguments for bmm)</p>

username: Kushaj
cooked: <p>I haven’t checked the logic of your code. But your code to work, you have to change this line</p>
<pre><code class="lang-python">scaling_mat = torch.eye(3, device=frame.device).repeat(batch_size, 1, 1)
</code></pre>
<p>to</p>
<pre><code class="lang-python">scaling_mat = torch.eye(256, device=frame.device).repeat(batch_size, 1, 1)
</code></pre>

username: CVAIDL
cooked: <p>I have changed this line</p>
<pre><code class="lang-auto">scaling_mat = torch.eye(3, device=frame.device).repeat(batch_size, 1, 1)
</code></pre>
<p>to</p>
<pre><code class="lang-auto">scaling_mat = torch.eye(256, device=frame.device).repeat(batch_size, 1, 1)
</code></pre>
<p>but there is a error  as follow.</p>
<p>Traceback (most recent call last):<br>
File “ioucompute.py”, line 197, in <br>
target_mask = warp.warp_image(fake_template, scaling_mat, out_shape=fake_template.shape[-2:])<br>
File “warp_image.py”, line 46, in warp_image<br>
xy_warped = torch.matmul(H, xy)  # H.bmm(xy)<br>
RuntimeError: Expected tensor to have size 256 at dimension 1, but got size 3 for argument <span class="hashtag">#2</span> ‘batch2’ (while checking arguments for bmm)</p>
<p>The “warp_image” function is as follow.<br>
def warp_image(img, H, out_shape=None, input_grid=None):<br>
if out_shape is None:<br>
out_shape = img.shape[-2:]<br>
if len(img.shape) &lt; 4:<br>
img = img[None]<br>
if len(H.shape) &lt; 3:<br>
H = H[None]<br>
assert img.shape[0] == H.shape[0], ‘batch size of images do not match the batch size of homographies’<br>
batchsize = img.shape[0]<br>
# create grid for interpolation (in frame coordinates)<br>
if input_grid is None:<br>
y, x = torch.meshgrid([<br>
torch.linspace(-utils.BASE_RANGE, utils.BASE_RANGE,<br>
steps=out_shape[-2]),<br>
torch.linspace(-utils.BASE_RANGE, utils.BASE_RANGE,<br>
steps=out_shape[-1])<br>
])<br>
x = x.to(img.device)<br>
y = y.to(img.device)<br>
else:<br>
x, y = input_grid<br>
x, y = x.flatten(), y.flatten()</p>
<pre><code># append ones for homogeneous coordinates
xy = torch.stack([x, y, torch.ones_like(x)])
xy = xy.repeat([batchsize, 1, 1])  # shape: (B, 3, N)
# warp points to model coordinates
xy_warped = torch.matmul(H, xy)  # H.bmm(xy)
xy_warped, z_warped = xy_warped.split(2, dim=1)

# we multiply by 2, since our homographies map to
# coordinates in the range [-0.5, 0.5]
xy_warped = 2.0 * xy_warped / (z_warped + 1e-8)
x_warped, y_warped = torch.unbind(xy_warped, dim=1)
# build grid
grid = torch.stack([
    x_warped.view(batchsize, *out_shape[-2:]),
    y_warped.view(batchsize, *out_shape[-2:])
],
    dim=-1)

warped_img = torch.nn.functional.grid_sample(
    img, grid, mode='bilinear', padding_mode='zeros')
batchsize0 = img.shape[0]
batchsize1 = img.shape[1]
batchsize2 = img.shape[2]
if utils.hasnan(warped_img):
    print('nan value in warped image! set to zeros')
    warped_img[utils.isnan(warped_img)] = 0
return warped_img</code></pre>

username: Kushaj
cooked: <p>You have to check the logic of your code again. The warp function takes <code>scaling_mat</code> as input. <code>scaling_mat</code> was earlier of shape <code>(3,3)</code> but now is <code>(256,256)</code>.</p>

username: CVAIDL
cooked: <p>scaling_mat is the homography matrix with shape [B, 3, 3] or [3, 3], since a matrix with shape [256, 256] is not a valid transformation matrix.</p>