Concatenating images

Niki: <p>I want to write a code in by Pytorch that concatenate two images  (32 by 32), in the way the output image becomes (64 by 32), how should I do that?</p>
<p>Thank you~</p>

ptrblck: <p><code>torch.cat</code> should do the job:</p>
<pre><code class="lang-python">a = torch.randn(3, 32, 32)
b = torch.randn(3, 32, 32)
c = torch.cat((a, b), 1)
print(c.shape)
&gt; torch.Size([3, 64, 32])
</code></pre>

Niki: <p>Thank you for your reply. I saw the torch.cat but don’t know how I can apply it here:</p>
<p>import torch<br>
import torchvision<br>
import torchvision.transforms as transforms</p>
<p>transform = transforms.Compose(<br>
[transforms.ToTensor(),<br>
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])</p>
<p>trainset = torchvision.datasets.CIFAR10(root=’./data’, train=True,<br>
download=True, transform=transform)<br>
trainloader = torch.utils.data.DataLoader(trainset, batch_size=5,<br>
shuffle=True, num_workers=1)</p>
<p>classes = (‘plane’, ‘car’, ‘bird’, ‘cat’,‘deer’, ‘dog’, ‘frog’, ‘horse’, ‘ship’, ‘truck’)</p>
<p>for epoch in range(3):<br>
for i, data in enumerate(trainloader, 0):<br>
inputs, labels = data<br>
#Final_input=image 64 by 32</p>

ptrblck: <p>You are currently using a batch size of 5, which won’t work if you would like to concatenate two images.<br>
If you use an even batch size, you could concatenate the images using this code:</p>
<pre><code class="lang-auto">inputs = torch.cat((inputs[::2], inputs[1::2]), 2)
</code></pre>
<p>Since you are using <code>shuffle=True</code>, I assume the pairs used to create the larger tensors do not matter. Is this correct or would you like to concatenate specific pairs of image tensors?</p>

Niki: <p>Thank you very much! It’s working. No, I don’t consider specific pairs.<br>
If I want to concatenate labels at the same time, this is correct?</p>
<p>labels = torch.cat((labels, labels))</p>

ptrblck: <p>Not really, as your samples not contain more than a single class and you are now dealing with a multi-label classification use case.<br>
In this setup, you could create multi-hot targets and change the criterion to e.g. <code>nn.BCEWithLogitsLoss</code>:</p>
<pre><code class="lang-python">one_hot_labels = torch.nn.functional.one_hot(labels)
one_hot_labels = one_hot_labels[::2] | one_hot_labels[1::2]
one_hot_labels = one_hot_labels.float()
</code></pre>

Niki: <p>I tried that, it’s getting me this error:</p>
<p>AttributeError: module ‘torch.nn.functional’ has no attribute ‘one_hot’</p>
<p>the torch version is 1.0.1.post2</p>

ptrblck: <p>I’m using <code>1.0.0.dev20190312</code>. You could use the nightly build or if you prefer to stay in <code>1.0.1post2</code>, you could use this code to create the one-hot encoded targets:</p>
<pre><code class="lang-python">one_hot_labels = torch.zeros(labels.size(0), nb_classes, dtype=torch.long).scatter_(1, labels.unsqueeze(1), 1)
</code></pre>

Niki: <p>Thank you! That’s work, but I realized the “one_hot_labels” at the end of the loop has a size [2,10], it’s possible that has size [2,10] inside the loop?<br>
If I want to concatenate more than two images, which parts of inputs and one_hot_labels should be changed?</p>

ptrblck: <p>A more general approach could use <code>torch.split</code> and a better <code>.scatter_</code> call:</p>
<pre><code class="lang-python">nb_cat = 3  # Number of images to concatenate

for i, data in enumerate(trainloader, 0):
    inputs, labels = data
    batch_size = inputs.size(0)
    split_size = batch_size // nb_cat
    inputs = torch.cat(inputs.split(split_size), 2)

    labels = torch.stack(labels.split(nb_cat))
    one_hot_labels = torch.zeros(inputs.size(0), nb_classes, dtype=torch.long).scatter_(1, labels, 1)
    one_hot_labels = one_hot_labels.float()
</code></pre>
<p>EDIT: Always assuming the batch size if divisible by <code>nb_cat</code> without a remainder.<br>
I haven’t tested edge cases!</p>

Niki: <p>nb_cat = 3  # Number of images to concatenate<br>
nb_classes=10</p>
<p>for epoch in range(2):<br>
for i, data in enumerate(trainloader, 0):<br>
inputs, labels = data<br>
batch_size = inputs.size(0)<br>
split_size = batch_size // nb_cat<br>
inputs = torch.cat(inputs.split(split_size), 2)<br>
labels = torch.stack(labels.split(nb_cat))<br>
one_hot_labels = torch.zeros(inputs.size(0), nb_classes, dtype=torch.long).scatter_(1, labels, 1)<br>
one_hot_labels = one_hot_labels.float()</p>
<p>this gives me this error:<br>
RuntimeError: split_size can only be 0 if dimension size is 0, but got dimension size of 2</p>

ptrblck: <p>It seems your batch size is smaller than <code>nb_cat</code>, thus the <code>split_size</code> is 0.<br>
This might be the case if you set a smaller <code>batch_size</code> in your <code>DataLoader</code> or alternatively the last batch in the training loop might be smaller, if the number of samples is not divisible by the batch size without a remainder. You could avoid this issue if you set <code>drop_last=True</code> in your <code>DataLoader</code> (or handle the smaller batch in another way).</p>

Niki: <p>I added drop_last=True and it’s working now, for batch_size=6, nb_cat=2, nb_classes=10, epoch=3:<br>
print(inputs.shape): torch.Size([3, 3, 64, 32])<br>
print(one_hot_labels.shape): torch.Size([3, 10])</p>
<p>here (in size) the first element (3) is split_size, am I right? Is there any ways that be dependent on batch_size, like Size([6, 3, 64, 32]) or Size([6, 10])?</p>

Niki: <p>when I use these inputs and one_hot_labels in my code, in the calculating loss part, shows me this error:</p>
<p>RuntimeError: Expected object of scalar type Long but got scalar type Float for argument <span class="hashtag">#2</span> ‘target’</p>
<p>this is because of float in one_hot_labels? how can I solve it?</p>

ptrblck: <p>If you concatenate the images, you’ll get “less” samples, so I’m not sure how you would like to keep the batch size as 6.<br>
Could you explain your use case a bit?</p>
<p>Since you are now dealing with multi-hot encoded targets (i.e. multi-label classification), you could use <code>nn.BCELoss</code> or <code>nn.BCEWithLogitsLoss</code> instead.</p>

Niki: <p>I think you are right regarding batch size.<br>
Thank you very much for your help, by using this new loss it’s working now.</p>

Mohamad_Hadi: <p>Hello, how can convert 16 images with size 128 by 128 into a 512 by 512 image?<br>
That is, 4 in width and 4 in length</p>

ptrblck: <p>You could use <code>view</code> or <code>F.fold</code> to create the larger output tensor (depending how the current tensor is shaped one or the other might be easier).</p>

Mohamad_Hadi: <p>thanks for your response<br>
If in this case, i want to enter 2 images with size of 512 by 512 in network each time and each image has a different label. Should give batch_size = 32 ?<br>
I am a beginner. excuse me</p>

ptrblck: <p>If you convert the 16 images into one large image, I would assume that this large image corresponds to a single target. Could you explain your use case a bit, i.e. how the targets are defined and why you would like to reshape the images?</p>