Dataset&Dataloader

MR.Hossein19: <p>Hi this is my dataset and codes.<br>
In training code has tensor error and after transform has to pilimage error.<br>
this is my code:</p>
<p>import pandas as pd<br>
import torchvision.transforms as trs<br>
from torch.utils.data import Dataset, DataLoader<br>
from ReadData import mean, std</p>
<p><span class="hashtag-raw">#Dataset</span><br>
class welldata(Dataset):<br>
def <strong>init</strong>(self, press_file=‘press.csv’, label_file=‘label.csv’):<br>
self.press = pd.read_csv(press_file)<br>
self.label = pd.read_csv(label_file)<br>
self.tr = trs.Compose([<br>
trs.Normalize((mean), (std))<br>
trs.ToTensor()<br>
])</p>
<pre><code>def __len__(self):
    return len(self.press)

def __getitem__(self, idx):
    press = self.tr(self.press.iloc[idx])
    label = self.tr(self.label.iloc[idx])
    return press, label
</code></pre>
<p>well_dataloader = DataLoader(dataset= welldata(), batch_size=5, shuffle=True)</p>
<p><span class="hashtag-raw">#Creat</span> model<br>
class model(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(model, self).<strong>init</strong>()<br>
self.lazy1 = nn.LazyLinear(1)<br>
self.rnn1 = nn.RNN(2, 2, 20)<br>
self.relu1 = nn.ReLU()<br>
self.bifc1 = nn.Bilinear(2, 5, 5)<br>
self.relu2 = nn.Hardtanh()<br>
self.lstm1 = nn.LSTM(5, 5)<br>
self.relu3 = nn.ReLU()</p>
<pre><code>def forward(self, z1, z2):
    y = self.lazy1(z1)
    y = torch.reshape(y, (z1.size(0), 2))
    y, _ = self.rnn1(y)
    y = self.relu1(y)
    y = self.bifc1(y, z2)
    y = self.relu2(y)
    y, _ = self.lstm1(y)
    y = self.relu3(y)
    return y
</code></pre>
<h1><a name="optimizer-1" class="anchor" href="#optimizer-1"></a>Optimizer</h1>
<p>model = model()<br>
optimizer = torch.optim.AdamW(model.parameters(), lr=0.1)</p>
<h1><a name="loss-2" class="anchor" href="#loss-2"></a>Loss</h1>
<p>loss = nn.SoftMarginLoss()</p>
<p><span class="hashtag-raw">#Training</span><br>
for x1, x2, yt in well_dataloader:<br>
optimizer.zero_grad()<br>
yp = model(x1, x2)<br>
loss_value = loss(yp, yt)<br>
loss_value.backward()<br>
optimizer.step()</p>
<p>and this is my data<br>
after debuging I’ll created data</p>
<p>column 0 and 1 is list and has many num float<br>
column 0 and 1 are input data in lazylinear<br>
and other column are input in RNN<br>
and they are input in bilinear Network<br>
this data are label and network must be answer</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/1/31b8ab39f6a7814ea2da79ac1caa8ec6dc1cdad4.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/31b8ab39f6a7814ea2da79ac1caa8ec6dc1cdad4" title="Data"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/31b8ab39f6a7814ea2da79ac1caa8ec6dc1cdad4_2_476x500.jpeg" alt="Data" data-base62-sha1="75R2VBRVYylD8qM8sacu8wAb03O" width="476" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/31b8ab39f6a7814ea2da79ac1caa8ec6dc1cdad4_2_476x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/31b8ab39f6a7814ea2da79ac1caa8ec6dc1cdad4_2_714x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/31b8ab39f6a7814ea2da79ac1caa8ec6dc1cdad4_2_952x1000.jpeg 2x" data-dominant-color="EDF0EC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Data</span><span class="informations">1920×2016 358 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>what’s the problem?</p>

ptrblck: <p>It’s unclear from the code and screenshots what the actual error is (at least to me). Could you describe the issue in more detail and, if possible, post a minimal, executable, and formatted code snippet to reproduce the issue?</p>

MR.Hossein19: <pre><code class="lang-auto">import torch
import torch.nn as nn
import pandas as pd
import torchvision.transforms as trs
from torch.utils.data import Dataset, DataLoader
from ReadData import mean, std

class welldata(Dataset):
    def __init__(self, press_file='press.csv', label_file='label.csv'):
        self.press = pd.read_csv(press_file)
        self.label = pd.read_csv(label_file)
        self.tr = trs.Compose([
                               trs.ToTensor(),
                               trs.Normalize((mean), (std))
                              ])

    def __len__(self):
        return len(self.press)

    def __getitem__(self, idx):
        press = self.press.iloc[idx]
        # press = self.tr(self.press.iloc[idx])
        label = self.label.iloc[idx]
        # label = self.tr(self.label.iloc[idx])
        return press, label

well_dataloader = DataLoader(dataset= welldata(), batch_size=5, shuffle=True)

class model(nn.Module):
    def __init__(self):
        super(model, self).__init__()
        self.lazy1 = nn.LazyLinear(1)
        self.rnn1 = nn.RNN(2, 2, 20)
        self.relu1 = nn.ReLU()
        self.bifc1 = nn.Bilinear(2, 5, 5)
        self.relu2 = nn.Hardtanh()
        self.lstm1 = nn.LSTM(5, 5)
        self.relu3 = nn.ReLU()

    def forward(self, z1, z2):
        y = self.lazy1(z1)
        y = torch.reshape(y, (z1.size(0), 2))
        y, _ = self.rnn1(y)
        y = self.relu1(y)
        y = self.bifc1(y, z2)
        y = self.relu2(y)
        y, _ = self.lstm1(y)
        y = self.relu3(y)
        return y

# Optimizer
model = model()
optimizer = torch.optim.AdamW(model.parameters(), lr=0.1)

# Loss
loss = nn.SoftMarginLoss()

for x1, x2, yt in well_dataloader:
    optimizer.zero_grad()
    yp = model(x1, x2)
    loss_value = loss(yp, yt)
    loss_value.backward()
    optimizer.step()
</code></pre>
<blockquote>
<p>raise TypeError(f"pic should be PIL Image or ndarray. Got {type(pic)}")<br>
TypeError: pic should be PIL Image or ndarray. Got &lt;class ‘pandas.core.series.Series’&gt;</p>
</blockquote>
<p>this is error after using transforms<br>
and my data type is not image</p>
<blockquote>
<p>raise TypeError(default_collate_err_msg_format.format(elem_type))<br>
TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class ‘pandas.core.series.Series’&gt;</p>
</blockquote>
<p>and this error without using transforms</p>

ptrblck: <p>You would need to transform the <code>pandas.Series</code> to an image before trying to apply the transformation on it.</p>

MR.Hossein19: <p>Is there a problem with my data set?<br>
I feel that I have not written anything in the data set</p>

ptrblck: <p>I don’t know if your dataset has any other issues, but right now the code is failing since a <code>pandas.Series</code> object is not supported in the used transformations as it does not represent an image type.<br>
Assuming the internal <code>numpy</code> array represents an image, you could create a <code>PIL.Image</code> via:</p>
<pre data-code-wrap="python"><code class="lang-python">image = PIL.Image.fromarray(series.iloc[idx].to_numpy())
</code></pre>
<p>but would of course need to check the shape and <code>dtype</code> making sure it is indeed a valid image type.</p>