Accessing ImageFolder() imgs after using ConcatDataset()

bluAsterisk: <p>Hello,</p>
<p>I have a question about how I can access the images from the call to ImageFolder() after feeding it into ConcatDataset(). This does seem to execute fine, but the ConcatDataset object doesn’t have an attribute for imgs that ImageFolder() has.</p>
<p>Here is the code I am using to setup the training data, the commented line is what I originally used. Now I am trying to take two datasets of images.</p>
<pre><code class="lang-auto">size = (224, 224)
data_transform = ImageTransform(size)
# train_data = torchvision.datasets.ImageFolder(root=train_dir, transform=data_transform.train_transform)

train_dataA = torchvision.datasets.ImageFolder(root=train_dirA, transform=data_transform.train_transform)
train_dataB = torchvision.datasets.ImageFolder(root=train_dirB, transform=data_transform.train_transform)
train_data = ConcatDataset([train_dataA, train_dataB])

val_data = torchvision.datasets.ImageFolder(root=val_dir, transform=data_transform.val_transform)
train_loader = torch.utils.data.DataLoader(train_data,
                                          batch_size=batch_size,
                                          shuffle=True)
val_loader = torch.utils.data.DataLoader(val_data,
                                          batch_size=batch_size,
                                          shuffle=True)
dataloaders_dict = {'train': train_loader, 'val': val_loader}
</code></pre>
<p>In another block I try and access the train_data.imgs, but it gives me an error of</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
&lt;ipython-input-30-e1395bae7815&gt; in &lt;module&gt;()
----&gt; 1 weights = make_weights_for_balanced_classes(train_data.imgs, carClasses) #len(train_data.classes)
      2 weights[1] /= 2
      3 weights = torch.FloatTensor(weights)
      4 print(weights)

AttributeError: 'ConcatDataset' object has no attribute 'imgs'
</code></pre>
<p>Thanks in advance!</p>

ptrblck: <p>You would have to access the internal attribute through the <code>.datasets</code> attribute as seen here:</p>
<pre><code class="lang-python"># setup
datasetA = TensorDataset(torch.randn(10, 1), torch.randn(10, 1))
datasetB = TensorDataset(torch.randn(10, 1), torch.randn(10, 1))

# access internal attribute
datasetA.tensors[0].shape

# concat
dataset = ConcatDataset((datasetA, datasetB))

# access attribute through the internal datasets
dataset.datasets[0].tensors[0].shape
</code></pre>

bluAsterisk: <p>Thanks for the reply!<br>
I tried your approach like so</p>
<pre><code class="lang-auto"># setup
train_dataA = torch.utils.data.TensorDataset(torch.randn(10, 1), torch.randn(10, 1))
train_dataB = torch.utils.data.TensorDataset(torch.randn(10, 1), torch.randn(10, 1))

# access internal attribute
train_dataA.tensors[0].shape

# concat
train_data = torch.utils.data.ConcatDataset((train_dataA, train_dataB))

# access attribute through the internal datasets
train_data.datasets[0].tensors[0].shape
</code></pre>
<p>But it seems to be having issues in the defined function here</p>
<pre><code class="lang-auto">def make_weights_for_balanced_classes(images, nclasses):                        
    count = [0] * nclasses                                                      
    for item in images:                                                         
        count[item[1]] += 1                                                     
    weight_per_class = [0.] * nclasses                                      
    N = float(np.max(count))                                                   
    for i in range(nclasses):                                                   
        weight_per_class[i] = N/float(count[i])                                 
#     weight = [0] * len(images)                                              
#     for idx, val in enumerate(images):                                          
#         weight[idx] = weight_per_class[val[1]]                                  
    return weight_per_class
</code></pre>
<pre><code class="lang-auto">---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
&lt;ipython-input-49-93f1a7a4ed45&gt; in &lt;module&gt;()
----&gt; 1 weights = make_weights_for_balanced_classes(train_data.datasets[0].tensors[0].shape, carClasses) #len(train_data.classes)
      2 weights[1] /= 2
      3 weights = torch.FloatTensor(weights)
      4 print(weights)

&lt;ipython-input-42-7fa267b3c6b9&gt; in make_weights_for_balanced_classes(images, nclasses)
      2     count = [0] * nclasses
      3     for item in images:
----&gt; 4         count[item[1]] += 1
      5     weight_per_class = [0.] * nclasses
      6     N = float(np.max(count))

TypeError: 'int' object is not subscriptable
</code></pre>
<p>Sorry if I misinterpreted your code, was there something that I missed?<br>
I’m still quite new to PyTorch, I need to get familiar with all the library functions.</p>

ptrblck: <p>My example code was accessing a random attribute to show that you would need to use the <code>.datasets[index]</code> approach to use it after wrapping the datasets into the <code>ConcatDataset</code>.<br>
In your current code snippet you are passing the <code>shape</code> of the underlying tensors to the method, which sounds wrong so just manipulate your code from:</p>
<pre><code class="lang-python">train_data.imgs
</code></pre>
<p>to</p>
<pre><code class="lang-python">train_data.datasets[index].imgs
</code></pre>

enterthevoidf22: <p>i have a similar situation and i want to be able to access an attribute programatically whether it is in ConcatDataset or not.</p>
<p>is it possible?</p>
<p>i was thinking about defining a parent class and modifying it’s get attribute function such that it checks whether the child class is a ConcatDataset object or Dataset object and modifies the get attribute class in accordance.</p>
<p>how would you go about doing that?</p>