How to solve problem of memory error in multi label classification?

Nisucuk: <p>I have 5529 labels,<br>
and each image has variable number of labels.<br>
In my final result, i can have upto 100 labels.<br>
I made my custom dataset following this code. <a href="https://www.kaggle.com/mratsim/starting-kit-for-pytorch-deep-learning" rel="nofollow noopener">https://www.kaggle.com/mratsim/starting-kit-for-pytorch-deep-learning</a><br>
class myCustomDataset(Dataset):<br>
“”“my dataset.”""</p>
<pre><code class="lang-auto">def __init__(self, csv_file, root_dir,img_ext,transform=None):
    """
    Args:
        csv_file (string): Path to the csv file with annotations.
        root_dir (string): Directory with all the images.
        transform (callable, optional): Optional transform to be applied
            on a sample.
    """
    #.iloc[:,0]=[[0]]
     #
    tmp_df = pd.read_csv(csv_file,sep=';',header=None)
    assert tmp_df.iloc[:,0].apply(lambda x: os.path.isfile(root_dir + x + img_ext)).all(), \
</code></pre>
<p>“Some images referenced in the CSV file were not found”</p>
<pre><code class="lang-auto">    self.mlb = MultiLabelBinarizer()
    self.root_dir = root_dir
    self.img_ext = img_ext
    self.transform = transform

    self.X_train = tmp_df.iloc[:,0]
    self.y_train = self.mlb.fit_transform(tmp_df.iloc[:,0].str.split()).astype(np.float32)

        
        
        

def __len__(self):
    return len(self.X_train.index)

def __getitem__(self, index):
    
    img = Image.open(self.root_dir + self.X_train[index] + self.img_ext)
    img = img.convert('RGB')
    if self.transform is not None:
        img = self.transform(img)
    
    label = torch.from_numpy(self.y_train[index])
    
    return img, label
</code></pre>
<p>now,i give my validationset to the dataset and create dataloader.</p>
<p>transformedvalid_dataset= myCustomDataset(csv_file=’/home/nis/Downloads/trialdata/Validation-Concepts.csv’,<br>
root_dir=’/home/nis/Downloads/trialdata/validation-set/’,<br>
img_ext=’.jpg’,<br>
transform=transforms.Compose([transforms.RandomSizedCrop(224),<br>
transforms.RandomHorizontalFlip(),<br>
transforms.ToTensor(),<br>
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))<br>
])<br>
)<br>
I am able to create my custom dataset ,and make validation loader.<br>
print(len(transformedvalid_dataset))<br>
14157</p>
<p>trainloader = torch.utils.data.DataLoader(transformedvalid_dataset, batch_size=32, shuffle=True)<br>
dataiter = iter(trainloader)<br>
images, labels = dataiter.next()<br>
print(type(images))<br>
print(images.shape)<br>
print(labels.shape)</p>
<p>&lt;class ‘torch.Tensor’&gt;<br>
torch.Size([32, 3, 224, 224])<br>
torch.Size([32, 14157])</p>
<p>But when i do same for train images(56638 images), it gives memory error when i provide my data set. I am far from making data loader work in case of train images as it gives memory error while giving images to dataset. Data loader would be the next step and batch size is provided at data loader.<br>
i could not figure out or find any links to correct the error.</p>
<p>how can i solve it? Also,my labels shape is equal to numbers of images in validation set,is it correct?<br>
Also, how to know how data looks at<br>
self.y_train = self.mlb.fit_transform(tmp_df.iloc[:,0].str.split()).astype(np.float32)<br>
and<br>
label = torch.from_numpy(self.y_train[index])</p>

ptrblck: <p>What kind of memory error do you get?<br>
Could you post the stack trace so that we can have a look?</p>

Nisucuk: <p>I solved the problem. I   had to put comma inside split function.</p>

ricslator: <p>Memory errors happens a lot with python when using the 32bit Windows version . This is because 32bit processes only gets 2GB of memory to play with by default.</p>
<p>The solution for this error is that pandas.<a href="http://net-informations.com/ds/pda/csv.htm" rel="noopener nofollow ugc">read_csv()</a> function takes an option called dtype. This lets pandas know what types exist inside your csv data.</p>
<p>For example: by specifying dtype={‘age’:int} as an option to the .read_csv() will let pandas know that age should be interpreted as a number. This saves you lots of memory.</p>
<p><code>pd.read_csv('data.csv',dtype={'age':int})</code></p>
<p>Or try the solution below:</p>
<p><code>pd.read_csv('data.csv',sep='\t',low_memory=False)</code></p>