Loading input/output folders with .npy extension

Dulat: <p>Hi, I am new in using pytorch and in ML in general.</p>
<p>I am building encoder-decoder architecture, similar to autoencoder, but input and output are different numpy arrays.<br>
I have two folders “input” and “output”, where input and output pairs are store in .npy format.<br>
At the moment I called each pair of input and output with the same name.<br>
So, for example, input/output files of pair 1 has same name, but stored in different folders.</p>
<p>I am not sure how to use how to use DatasetFolder, so my dataset will become X,Y which I will be able to split in train/test and feed to the model.</p>
<p>Thanks</p>

ptrblck: <p>Based on your description I think implementing a custom <code>Dataset</code> would be the best approach. In its <code>__init__</code> method you could store the folder paths and load each pair in the <code>__getitem__</code> using the corresponding index.</p>

Dulat: <p>Thanks prtblck!</p>
<p>I did as you have advised, I am getting error, which I could not figure out by googling, so let me ask in this post.</p>
<p>I have my numpy arrays in the shape (1, 42, 58), as mentioned in two folders with identical names for each pair.<br>
I am getting following error now:<br>
linear(): argument ‘input’ (position 1) must be Tensor, not list<br>
Could you please advise what causing it?</p>
<pre><code class="lang-auto">class En_De_coder_dataset(Dataset):
    def __init__(self, in_dir, out_dir):
        self.in_dir = in_dir
        self.out_dir = out_dir
        
    def __len__(self):
        return len(os.listdir(self.in_dir))

    def __getitem__(self, index):
        input = torch.from_numpy(np.load(self.in_dir + '/grid_aug_' + str(index) + '.npy')),float()
        output = torch.from_numpy(np.load(self.out_dir + '/grid_aug_' + str(index) + '.npy')),float()

        return (input, output)

dataset = En_De_coder_dataset(in_dir = 'augmented_inputs', out_dir = 'augmented_outputs')

train_set, test_set = torch.utils.data.random_split(dataset, [100, 44])

train_loader = DataLoader(dataset=train_set, batch_size=64, shuffle=True)
test_loader = DataLoader(dataset=test_set, batch_size=64, shuffle=True)

class En_De_coder(nn.Module):
    def __init__(self):
        super().__init__()        
        self.encoder = nn.Sequential(
            nn.Linear(28 * 28, 128), # (N, 784) -&gt; (N, 128)
            nn.ReLU(),
            nn.Linear(128, 64),
            nn.ReLU(),
            nn.Linear(64, 12),
            nn.ReLU(),
            nn.Linear(12, 3) # -&gt; N, 3
        )
        
        self.decoder = nn.Sequential(
            nn.Linear(3, 12),
            nn.ReLU(),
            nn.Linear(12, 64),
            nn.ReLU(),
            nn.Linear(64, 128),
            nn.ReLU(),
            nn.Linear(128, 28 * 28),
            nn.Sigmoid()
        )

    def forward(self, x):
        encoded = self.encoder(x)
        decoded = self.decoder(encoded)
        return decoded

model = En_De_coder()

criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(),
                             lr=1e-3, 
                             weight_decay=1e-5)

num_epochs = 10
outputs = []
for epoch in range(num_epochs):
    for batch_idx, (data, targets) in enumerate(train_loader):
        recon = model(data)
        loss = criterion(recon, targets)
        
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

print(f'Epoch:{epoch+1}, Loss:{loss.item():.4f}')
outputs.append((epoch, targets, recon))
</code></pre>
<p>my data looks like this<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/e/fe46b6521c5f61e82d83f53551d1579162d9d5c2.png" alt="image" data-base62-sha1="AhqMdkWiqdT06l6CBmCKPx6U2TE" width="373" height="451"></p>

ptrblck: <p>Your <code>__getitem__</code> method seems to have a typo as I guess you want to transform the tensors to <code>float32</code> and not create a <code>tuple</code> with a <code>0.0</code> as the second item:</p>
<pre><code class="lang-python">    def __getitem__(self, index):
        input = torch.from_numpy(np.load(self.in_dir + '/grid_aug_' + str(index) + '.npy')),float()
        output = torch.from_numpy(np.load(self.out_dir + '/grid_aug_' + str(index) + '.npy')),float()

        return (input, output)
</code></pre>
<p>Replace the comma with a dot and it should work I guess.</p>

Dulat: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> , you are awesome!!!<br>
It works now!!!</p>