Multi Input Images, with MultiOutput Regression Label

username: Henseljahja
cooked: <p>Dear experts, so i just moved from tensorflow to Pytorch and i have a list of images in directories as such</p>
<pre><code class="lang-auto">root/
    ImageA/
            ImageA001.jpg
            ImageA002.jpg 
            ,,,,,,,,
    ImageB/
            ImageB001.jpg
            ImageB002.jpg 
            ,,,,,,,,
</code></pre>
<p>and labels of regression value like so</p>
<pre><code class="lang-auto">ImageA,A_Value,ImageB,B_Value
root/ImageA/ImageA001.jpg,32,root/ImageA/ImageB001.jpg,66
root/ImageA/ImageA001.jpg,43,root/ImageA/ImageB001.jpg,32
</code></pre>
<p>My Questions is how to load it to Models, creating multi input and multi output single models possibly with transfer learning. I Have used tensorflow in the past, and i cant wrap my mind on how to implemented it in Pytorch.</p>
<p>All of the tutorials only tells me how to do it only on classification and only in one output models, and i have modified Classes, to get label for regression such for only one (ImageA) folders only.</p>
<pre><code class="lang-auto">class CustomImageDataset(torch.utils.data.Dataset):
    def __init__(self, annotations_file, img_dir, transform=None, target_transform=None):
        self.img_labels = pd.read_csv(annotations_file)
        self.img_dir = img_dir
        self.transform = transform
        self.target_transform = target_transform

    def __len__(self):
        return len(self.img_labels)

    def __getitem__(self, idx):
        img_path = os.path.join(self.img_dir, self.img_labels.iloc[idx, 0])
        image = read_image(img_path)
        label = self.img_labels.iloc[idx, 1]
        if self.transform:
            image = self.transform(image)
        if self.target_transform:
            label = self.target_transform(label)
        return image, label

dataset = CustomImageDataset(
    annotation_file="&lt;Path&gt;.csv",
    root_dir = "&lt;Root&gt;",
    transform = my_transform
    )
train_set, validation_set = torch.utils.data.random_split(pepe, [TRAIN_LENGTH, VAL_LENGTH])

train_loader = DataLoader(dataset=train_set, shuffle=shuffle, batch_size=batch_size,num_workers=num_workers,pin_memory=pin_memory)
validation_loader = DataLoader(dataset=validation_set, shuffle=shuffle, batch_size=batch_size,num_workers=num_workers, pin_memory=pin_memory)
</code></pre>
<p>and for the models</p>
<pre><code class="lang-auto">class CNN(nn.Module):
    def __init__(self, train_CNN=False, num_classes=1):
        super(CNN, self).__init__()
        self.train_CNN = train_CNN
        self.inception = models.inception_v3(pretrained=True, aux_logits=False)
        self.inception.fc = nn.Linear(self.inception.fc.in_features, num_classes)
        self.relu = nn.ReLU()
        self.dropout = nn.Dropout(0.5)
        self.linear = nn.linear()

    def forward(self, images):
        features = self.inception(images)
        return self.linear(self.dropout(self.relu(features))).squeeze(1)
</code></pre>
<p>and the training file</p>
<pre><code class="lang-auto">model = CNN().to(device)

criterion = nn.BCELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

for name, param in model.inception.named_parameters():
    if "fc.weight" in name or "fc.bias" in name:
        param.requires_grad = True
    else:
        param.requires_grad = train_CNN

def check_accuracy(loader, model):
    if loader == train_loader:
        print("Checking accuracy on training data")
    else:
        print("Checking accuracy on validation data")

    num_correct = 0
    num_samples = 0
    model.eval()

    with torch.no_grad():
        for x, y in loader:
            x = x.to(device=device)
            y = y.to(device=device)

            scores = model(x)
            # predictions = torch.tensor([1.0 if i &gt;= 0.5 else 0.0 for i in scores]).to(device)
            num_correct += (predictions == y).sum()
            num_samples += predictions.size(0)
    return f"{float(num_correct)/float(num_samples)*100:.2f}"
    print(
            f"Got {num_correct} / {num_samples} with accuracy {float(num_correct)/float(num_samples)*100:.2f}"
        )
    model.train()

def train():
    model.train()
    for epoch in range(num_epochs):
        loop = tqdm(train_loader, total = len(train_loader), leave = True)
        if epoch % 2 == 0:
            loop.set_postfix(val_acc = check_accuracy(validation_loader, model))
        for imgs, labels in loop:
            imgs = imgs.to(device)
            labels = labels.to(device)
            outputs = model(imgs)
            loss = criterion(outputs, labels)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            loop.set_description(f"Epoch [{epoch}/{num_epochs}]")
            loop.set_postfix(loss = loss.item())

if __name__ == "__main__":
    train()
</code></pre>
<p>and still i get the errors like so</p>
<pre><code class="lang-auto">TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class 'PIL.Image.Image'&gt;
</code></pre>
<p>Is theres any false implementations in my code? and For my other questions, i have only tried to do it only in one Folder Images (ImageA), is theres other implementation to do it for all folder images, with each label regression like so, Or do i just loop over all the folders? Thanks in advance</p>

username: ptrblck
cooked: <p>Based on the error message it seems you are not transforming the <code>PIL.Image</code> to a tensor, so I guess in your code the <code>transform</code> argument might not be set properly. Note that your posted code snippet uses <code>root_dir</code> instead of <code>img_dir</code>, soI assume your real code is not using the same posted code.<br>
To further isolate the issue, check the <code>type(image)</code> in <code>__getitem__</code> before returning it and make sure it’s a tensor.</p>
<p>Multiple inputs and outputs can be implemented directly in the <code>forward</code> via:</p>
<pre><code class="lang-python">def forward(self, input1, input2):
    ...
    out1 = self.layers(input1)
    ...
    out2 = self.layers(input2)
    return out1, out2
</code></pre>
<p>I’m unsure, if you need to iterate all folders, as it seems you are already using a mapping where the image paths and the corresponding label is given. In this case, I would just reuse the provided mapping.</p>

username: Henseljahja
cooked: <p>Thankyou sir, you’re literally a live safer. The snipped that you just posted, worked fine, i just needed add another transfer learning layer. Thankyou so much!</p>