Multiple input model architecture

username: jdrouet
cooked: <p>Hi,</p>
<p>I’m really new in machine learning and I’d like to have some advices.<br>
My problem is the following:<br>
I’ve 2 images (1st is 256x256 and the second 64x64) and some data (list of 10 floats) as an input and I’d like to classify the data in 4 classes (for now).</p>
<p>Is there a way to organize my data to fit it in a standard model with one input?<br>
Is there a way to architecture my model to fit in each item separately?</p>
<p>Thank you</p>

username: ptrblck
cooked: <p>The main question for the first approach would be, how to stack your different inputs together?<br>
The images might be resized to a common size and stacked together, but how would you handle the list data?</p>
<p>You could easily create a model architecture, where you have different paths for each input and concatenate them together at some point. Assuming your images have one single channel, here is a small example:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.features1 = nn.Sequential(
            nn.Conv2d(1, 3, 3, 1, 1),
            nn.MaxPool2d(2),
            nn.ReLU(),
        )
        self.features2 = nn.Sequential(
            nn.Conv2d(1, 3, 3, 1, 1),
            nn.MaxPool2d(2),
            nn.ReLU(),
        )
        self.features3 = nn.Sequential(
            nn.Linear(10, 5),
            nn.ReLU(),
        )

        self.classifier = nn.Linear(128*128*3 + 32*32*3 + 5, 4)
        
    def forward(self, x1, x2, x3):
        x1 = self.features1(x1)
        x2 = self.features2(x2)
        x3 = self.features3(x3)

        x1 = x1.view(x1.size(0), -1)
        x2 = x2.view(x2.size(0), -1)
        x3 = x3.view(x3.size(0), -1)
        
        x = torch.cat((x1, x2, x3), dim=1)
        x = self.classifier(x)
        return x

model = MyModel()
batch_size = 1
x1 = torch.randn(batch_size, 1, 256, 256)
x2 = torch.randn(batch_size, 1, 64, 64)
x3 = torch.randn(batch_size, 10)

output = model(x1, x2, x3)
</code></pre>

username: shzgong
cooked: <aside class="quote no-group quote-modified" data-username="ptrblck" data-post="2" data-topic="19754">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>self.features3 = nn.Sequential( nn.Linear(10, 5), nn.ReLU(), ) self.classifier = nn.Linear(128<em>128</em>3 + 32<em>32</em>3 + 5, 4) def forward(self, x1, x2, x3):</p>
</blockquote>
</aside>
<p>Hi, I have a simple question.<br>
After I implement such a multi-input model, with 3 inputs and 1 output.<br>
How should I do the loss.backward() operation?<br>
Is it the same as usual if all the parameters’ requires_grad = True?</p>

username: ptrblck
cooked: <p>Since you are only dealing with a single output, you can calculate the loss as usual and just call <code>loss.backward()</code> to compute all gradients.</p>

username: shzgong
cooked: <p>Thank you, that sounds good. I will try that ~</p>

username: styx97
cooked: <p>Hi, I’m having a similar situation over here where I’m feeding three images into a network where they take different paths but get ultimately merged before the final layer. My question is what changes do I need to make in the dataloader and the scheduler to accommodate for this ? (Feeding three images into the network at once)</p>

username: asere
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="19754">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>You could easily create a model architecture, where you have different paths for each input and concatenate them together at some point.</p>
</blockquote>
</aside>
<p>Hello, I’m a bit confused as to how I could do this in a segmentation problem. At which point in the segmentation network can I concatenate the scalar list?</p>

username: ptrblck
cooked: <p>It’s hard to give a general answer, as it’s unclear what would work the best.<br>
Could you explain your use case a bit and what the scalar input represents?</p>

username: asere
cooked: <p>I have a 3D image and scalars that i want to give as input to the network. The image is a medical scan and the list of scalars represent clinical values. I would like to input both the image and the scalars to my segmentation network (UNet-like).</p>
<p>My concern is where exactly do I concatenate both inputs.</p>
<p>Should I concatenate the scalars to the image in the bottleneck of the network by flattening the image and then proceding to the decoder part via a linear layer? Is there a better approach?</p>
<p>Thank you in advance</p>

username: ptrblck
cooked: <p>I’m really unsure, what might work the best.<br>
Your bottleneck approach sounds reasonable and I would give it a try.</p>

username: Hemlata
cooked: <p>Hi,  Thanks for this explanation.</p>
<p>I have one question.<br>
Can we take maximum of all three features X1,X2 and X3 Instead of concatenate them together. If yes than how can we find maximum of all three features<br>
x=torch.max(x1,x2,x3)</p>
<p>If it is not correct way to take maximum please let me know the correct way to take maximum of all three features.</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>Would you like to get the elementwise maximum of these features, i.e. the result might contain different features from all input tensors or would you like to somehow use a scalar value and select only a single input tensor?</p>
<p>The first use case should work using <code>torch.max</code> two times, as it will only accept a pair of tensors.</p>

username: whanafy
cooked: <p>Hello, I have a related question. I have a network that takes two inputs. But there I am some cases that (only in evaluation) that I need to only pass one of them. The idea is that i don’t want to pass random/zeros to save the processing time?</p>

username: ptrblck
cooked: <p>If you are not using the second input during evaluation, you could pass a <code>None</code> value to them.<br>
Since these additional inputs are not used, you shouldn’t get any errors.</p>

username: sleepyo
cooked: <p>Hi thanks for this discussion, I managed to make a multi-input one.<br>
Now the question is, Instead of concatenating it with torch.cat, how about if I want to add some weight/importance like input 1 is more important the input 2 (75:25)?</p>