Concat image datasets with different size and number of channels

RonMcKay: <p>Good evening everyone,</p>
<p>I got a problem with loading multiple datasets and did not find a solution so far.<br>
For loading multiple datasets into one dataloader I can use the ConcatDataset class, but how do I concatenate e.g. CIFAR10 and MNIST. For this I would have CIFAR10 to resize and convert to grayscale without applying the transformations to MNIST.</p>
<p>Any ideas on how to do that?</p>

ptrblck: <p>You could specify the transformations for each dataset.<br>
This code should work:</p>
<pre><code class="lang-python">mnist = datasets.MNIST(
    root='/home/ptrblck/python/data',
    download=False,
    transform=transforms.Compose([
        transforms.ToTensor()
    ])
)

cifar = datasets.CIFAR10(
    root='/home/ptrblck/python/data',
    download=False,
    transform=transforms.Compose([
        transforms.Grayscale(),
        transforms.Resize(28),
        transforms.ToTensor()
    ])
)

dataset = ConcatDataset((mnist, cifar))
</code></pre>
<p>As you can see <code>transforms.Resize</code> and <code>transforms.Grayscale</code> was only applied for the CIFAR10 dataset.</p>

RonMcKay: <p>Oh yes of course, I am really a blockhead. <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=6" title=":grin:" class="emoji" alt=":grin:"><br>
Thanks!</p>

adwaykanhere: <p>Can you concatenate the targets using this approach? If not, how do we go about it?</p>

ptrblck: <p><code>ConcatDataset</code> will concatenate all passed <code>Dataset</code>s keeping their samples and targets.</p>

Brando_Miranda: <aside class="quote no-group" data-username="ptrblck" data-post="5" data-topic="36362">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>keeping their samples and targets.</p>
</blockquote>
</aside>
<p>may I ask what this means? I’d lake to take the union for the labels and being relabeled from from scratch from <code>0</code> to <code>len_dataset1+len_dataset2</code>.</p>
<hr>
<p>easiest solution to what I want is to do use this: <a href="https://discuss.pytorch.org/t/does-concatenate-datasets-preserve-class-labels-and-indices/62611/12" class="inline-onebox">Does Concatenate Datasets preserve class labels and indices - #12 by Brando_Miranda</a> by using learn2learn’s union of data sets.</p>

ptrblck: <p><code>ConcatDataset</code> will just iterate both datasets as seen <a href="https://github.com/pytorch/pytorch/blob/cf5699f2fc3e322efe2bb2446949dc30f1d36b4b/torch/utils/data/dataset.py#L199-L246">here</a>.<br>
Here is a small example:</p>
<pre><code class="lang-python">dataset1 = TensorDataset(torch.arange(5).float().view(-1, 1))
dataset2 = TensorDataset(torch.arange(5).float().view(-1, 1) + 0.1)

dataset = torch.utils.data.ConcatDataset((dataset1, dataset2))

for d in dataset:
    print(d)

# (tensor([0.]),)
# (tensor([1.]),)
# (tensor([2.]),)
# (tensor([3.]),)
# (tensor([4.]),)
# (tensor([0.1000]),)
# (tensor([1.1000]),)
# (tensor([2.1000]),)
# (tensor([3.1000]),)
# (tensor([4.1000]),)

loader = DataLoader(dataset, batch_size=2)
for d in loader:
    print(d)
    
# [tensor([[0.],
#         [1.]])]
# [tensor([[2.],
#         [3.]])]
# [tensor([[4.0000],
#         [0.1000]])]
# [tensor([[1.1000],
#         [2.1000]])]
# [tensor([[3.1000],
#         [4.1000]])]

loader = DataLoader(dataset, batch_size=2, shuffle=True)
for d in loader:
    print(d)

# [tensor([[0.1000],
#         [1.1000]])]
# [tensor([[3.1000],
#         [2.0000]])]
# [tensor([[1.],
#         [0.]])]
# [tensor([[4.],
#         [3.]])]
# [tensor([[4.1000],
#         [2.1000]])]
</code></pre>
<p>Is this what you are referring to as “union” or are samples duplicated in both datasets which you want to remove?</p>

Brando_Miranda: <p>no, I just mean concatenate and re-index the class. Basically assumes mutually exclusive labels.</p>

ptrblck: <p>In that case my code should work and shows this behavior.</p>

Brando_Miranda: <p>but you still need to produce the right labels. It’s not hard I suppose but if the data loader is random we want the classes to be consistently &amp; correct labeled.</p>

ptrblck: <p>The datasets are indexed separately as seen in the linked code, so they cannot be mixed up.<br>
You can use my code snippet to add targets and would see that the correspondence is not broken (it would otherwise make this class quite useless):</p>
<pre><code class="lang-python">dataset1 = TensorDataset(torch.arange(5).float().view(-1, 1), torch.zeros(5, 1))
dataset2 = TensorDataset(torch.arange(5).float().view(-1, 1) + 0.1, torch.ones(5, 1))

dataset = torch.utils.data.ConcatDataset((dataset1, dataset2))

loader = DataLoader(dataset, batch_size=2, shuffle=True)
for d in loader:
    print(d)
</code></pre>

Brando_Miranda: <p>apologies if I am not being clear. But I think this might be what I’m looking for, where the labeling is re-done assuming mutually exclusive labels for each data set:</p>
<pre><code class="lang-auto">class ConcatDataset(Dataset):
    """

    ref: https://discuss.pytorch.org/t/concat-image-datasets-with-different-size-and-number-of-channels/36362/12
    """

    def __init__(self, datasets: list[Dataset]):
        """
        """
        # I think concat is better than passing data to a self.data = x obj since concat likely using the getitem method of the passed dataset and thus if the passed dataset doesnt put all the data in memory concat won't either
        self.concat_datasets = torch.utils.data.ConcatDataset(datasets)
        # maps a class label to a list of sample indices with that label.
        self.labels_to_indices = defaultdict(list)
        # maps a sample index to its corresponding class label.
        self.indices_to_labels = defaultdict(None)
        # - do the relabeling
        offset: int = 0
        new_idx: int = 0
        for dataset_idx, dataset in enumerate(datasets):
            assert len(dataset) == len(self.concat_datasets.datasets[dataset_idx])
            assert dataset == self.concat_datasets.datasets[dataset_idx]
            for x, y in dataset:
                y = int(y)
                new_label = y + offset
                self.indices_to_labels[new_idx] = new_label
                _x, _y = self.concat_datasets[new_idx]
                _y = int(_y)
                assert y == _y
                assert torch.equal(x, _x)
                self.labels_to_indices[new_label] = new_idx
            num_labels_for_current_dataset: int = max([y for _, y in dataset])
            offset += num_labels_for_current_dataset
            new_idx += 1
        assert len(self.indices_to_labels.keys()) == len(self.concat_datasets)
        # contains the list of labels from 0 - total num labels after concat
        self.labels = range(offset)
        self.target_transform = lambda data: torch.tensor(data, dtype=torch.int)

    def __len__(self):
        return len(self.concat_datasets)

    def __getitem__(self, idx: int) -&gt; tuple[Tensor, Tensor]:
        x = self.concat_datasets[idx]
        y = self.indices_to_labels[idx]
        if self.target_transform is not None:
            y = self.target_transform(y)
        return x, y
</code></pre>
<p>does this look right to you? Need to think how to test it.</p>
<hr>
<p>assertion fails:</p>
<pre><code class="lang-auto">    assert torch.equal(x, _x)
AssertionError
python-BaseException
</code></pre>
<p>code</p>
<pre><code class="lang-auto">def check_xs_align_cifar100():
    from pathlib import Path

    root = Path("~/data/").expanduser()
    # root = Path(".").expanduser()
    train = torchvision.datasets.CIFAR100(root=root, train=True, download=True)
    test = torchvision.datasets.CIFAR100(root=root, train=False, download=True)

    concat = ConcatDataset([train, test])
    print(f'{len(concat)=}')
    print(f'{len(concat.labels)=}')
</code></pre>
<p>cifar100 doesn’t test the relabling of disjoint union of data sets but the x’s should’ve aligned…but they didnt</p>
<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/73913522/why-dont-the-images-align-when-concatenating-two-data-sets-in-pytorch-using-tor">
  <header class="source">

      <a href="https://stackoverflow.com/questions/73913522/why-dont-the-images-align-when-concatenating-two-data-sets-in-pytorch-using-tor" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/1601580/charlie-parker" target="_blank" rel="noopener nofollow ugc">
    <img alt="Charlie Parker" src="https://i.stack.imgur.com/2B6rV.png?s=256&amp;g=1" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/73913522/why-dont-the-images-align-when-concatenating-two-data-sets-in-pytorch-using-tor" target="_blank" rel="noopener nofollow ugc">Why don't the images align when concatenating two data sets in pytorch using torch.utils.data.ConcatDataset?</a>
</h4>

<div class="tags">
  <strong>python, machine-learning, pytorch, dataset</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/1601580/charlie-parker" target="_blank" rel="noopener nofollow ugc">
    Charlie Parker
  </a>
  on <a href="https://stackoverflow.com/questions/73913522/why-dont-the-images-align-when-concatenating-two-data-sets-in-pytorch-using-tor" target="_blank" rel="noopener nofollow ugc">08:07PM - 30 Sep 22 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<hr>
<p>after converting images to tensor comparison still fails <a class="mention" href="/u/ptrblck">@ptrblck</a> :</p>
<pre><code class="lang-auto">    img2tensor: Callable = torchvision.transforms.ToTensor()
    x, _x = img2tensor(x), img2tensor(_x)
    assert torch.equal(x, _x), f'Error for some reason, got: {data_idx=}, {x.norm()=}, {_x.norm()=}, {x=}, {_x=}'
</code></pre>

Brando_Miranda: <p>answer here I think: <a href="https://stackoverflow.com/a/74038903/1601580" class="inline-onebox" rel="noopener nofollow ugc">python - Why don't the images align when concatenating two data sets in pytorch using torch.utils.data.ConcatDataset? - Stack Overflow</a></p>