Transformation error while training dataset

mchowdhu: <p>I am facing the transformation error while calling the train function in CNN model<br>
Here is my code that i used for dataloader</p>
<pre><code class="lang-auto">#make a class for data
class OAIDataset(Dataset):
    def __init__(self, csv_file,root_dir, transform=None):
       # self.data = pd.read_csv(csv_file, header=None)
       self.data = csv_file
       self.root_dir = root_dir
       self.transform = transform

    def __len__(self):
       return len(self.data)

    def __getitem__(self,idx):
        img_name = os.path.join(self.root_dir, str(self.data['ID'].iloc[idx])+'.npy')

        patches, p_id = np.load(img_name, allow_pickle= True)

        img_class = int(self.data.iloc[idx,2])

        side = self.data.iloc[idx,1]

        if side ==1:
            image = np.array(patches['R'].astype('uint8'), 'L')                                              #[image['R':side]|image['L':side]]
        else:
            image = np.array(patches['L'].astype('uint8'), 'L')

        if self.transform is not None:
             image = self.transform(image)

        sample = {'image': image, 'grade':img_class}

        return sample

# Define our data transforms
#
trans = transforms.Compose([ transforms.ToTensor()])
        # Call the dataset
#trans =transforms.ToTensor()
train_data =OAIDataset(train_df,root_dirA,transform=trans )

test_data =OAIDataset(test_df,root_dirA,transform=trans)
from sklearn.model_selection import GroupKFold # import sklearn module

# creating kfold and make indices
kfold = GroupKFold(n_splits=5)


snapshots = []
#iterating over folds
for train_index, val_index in kfold.split(X=train_df, y=grades_dev, groups=groups_dev):
        print("\nTRAIN:", train_index, "TEST:", val_index)

        #Subset of a dataset at saccording to indices

        train_set = torch.utils.data.Subset(train_data, train_index)

        val_set = torch.utils.data.Subset(train_data, val_index)
        #load data(train and validation)
        train_loader = torch.utils.data.DataLoader(train_set,batch_size=50,shuffle=True,num_workers=0)

        val_loader = torch.utils.data.DataLoader(val_set,batch_size=50,shuffle=False,num_workers=0)
model = ConvNet()

def train_val_model(model, epoch):
    model.train() #set the model to training mode
    for epoch in range(epoch):
        losses= 0
        closs = 0

        for i,batch in enumerate(train_loader):
            image, grade = batch['image'], batch['grade']

            #image=image.unsqueeze(1).type(torch.FloatTensor)
            #image = torch.from_numpy(image).float()
            optimizer.zero_grad()

            prediction = model(image)
            loss = costFunction(prediction,grade)
            closs += loss.item()
            loss.backward()
            optimizer.step()
            # storing the loss
            losses += loss.data[0]
            # losses.append(loss.item())
            # num_times = num_times + 1
            print('epoch',epoch,'losses',closs)
</code></pre>
<p>Here is my error massages</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/fatema/Downloads/assignment3/DATAfile.py", line 288, in &lt;module&gt;
    train_val_model(model,1)
  File "/home/fatema/Downloads/assignment3/DATAfile.py", line 154, in train_val_model
    for i,batch in enumerate(train_loader):
  File "/home/fatema/miniconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 560, in __next__
    batch = self.collate_fn([self.dataset[i] for i in indices])
  File "/home/fatema/miniconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 560, in &lt;listcomp&gt;
    batch = self.collate_fn([self.dataset[i] for i in indices])
  File "/home/fatema/miniconda3/lib/python3.6/site-packages/torch/utils/data/dataset.py", line 107, in __getitem__
    return self.dataset[self.indices[idx]]
  File "/home/fatema/Downloads/assignment3/DATAfile.py", line 62, in __getitem__
    image = self.transform(image)
  File "/home/fatema/miniconda3/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 49, in __call__
    img = t(img)
  File "/home/fatema/miniconda3/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 76, in __call__
    return F.to_tensor(pic)
  File "/home/fatema/miniconda3/lib/python3.6/site-packages/torchvision/transforms/functional.py", line 48, in to_tensor
    img = torch.from_numpy(pic.transpose((2, 0, 1)))
ValueError: axes don't match array

Process finished with exit code 1

</code></pre>
<p>I tried to implement sever transformation but could not solved.</p>

ptrblck: <p>The raised <code>ValueError</code> is actually raised from the numpy array <code>pic</code>, which apparently does not have three dimensions:</p>
<pre><code class="lang-python">x = np.random.randn(10, 10)
x.transpose((2, 0, 1))
&gt; ValueError: axes don't match array
</code></pre>
<p>Could you print the shape of <code>image</code> before passing it to the transformation?</p>

mchowdhu: <p>In getitem I have put ’ <code>img_class = int(self.data.iloc[idx,2])</code>’ in that it showed error like <code>Traceback (most recent call last): File "/home/fatema/Downloads/numpy data.py", line 107, in &lt;module&gt; print(i, sample['image'].shape, sample['grade'].shape) AttributeError: 'int' object has no attribute 'shape'</code><br>
When i removed 'int ’ from imag_class,i am abled to print image shape but i have not got any information about img_class.<br>
Here is my data set image shape.</p>
<pre><code class="lang-auto">print(sample['image'].shape)
print(sample['grade'].shape)
Traceback (most recent call last):
  File "/home/fatema/miniconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3325, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "&lt;ipython-input-35-b3e87aaa979e&gt;", line 2, in &lt;module&gt;
    print(sample['grade'].shape)
AttributeError: 'int' object has no attribute 'shape'
(64, 64)
</code></pre>
<p>The shape of the image is(64,64)<br>
I am wondering that why i am not getting shape of ‘‘grade’’?</p>

ptrblck: <p>Is the <code>AttributeError</code> thrown for <code>sample['image']</code> or <code>sample['grade']</code>?<br>
Are you able to get some sample before the original error is raised?<br>
If so, could you add the print statement regarding the shape into <code>__getitem__</code> directly before calling <code>self.transform</code> and post the output here?</p>

mchowdhu: <aside class="quote no-group quote-modified" data-username="mchowdhu" data-post="1" data-topic="52049">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/4af34b/48.png" class="avatar"> mchowdhu:</div>
<blockquote>
<p>torch.from_numpy(pic.transpose((2, 0, 1))) ValueError: axes don’t match array</p>
</blockquote>
</aside>
<p><code>AttributeError</code> seems comes from <code>sample['grade']</code>whenever i call '‘print(sample[‘grade’])’ it gives output*<br>
(64, 64)<br>
2.0<br>
I am not quit sure what do you mean by print <code>__getitem__</code>? Now i called the ‘print’ function  like this <code>print(sample['image']) print(sample['grade'])</code><br>
And got the result like this<code>print(sample['image']) print(sample['grade']) [[175 177 176 ...  81  79  68]  [175 175 174 ...  79  82  79]  [173 172 173 ...  76  75  78]  ...  [122 120 121 ...  45  45  46]  [121 121 120 ...  45  45  44]  [122 122 123 ...  44  44  43]] 2 </code></p>