An autoencoder with multiple inputs

username: javad_t
cooked: <p>Hi everybody,<br>
I’m new to pytorch and trying to implement a multimodal deep autoencoder(means: autoencoder with multiple inputs)<br>
At the first all inputs encode with same encoder architecture, after that, all outputs concatenates together and the output goes into the another encoding and deoding layers:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5ff61ead95bca4d6d37876e67abb6e0ebf6ff204.jpeg" alt="SharedScreenshot" data-base62-sha1="dGUFtFLmt5D4UXLr6UoePXmsAYc" width="539" height="301"></p>
<p>At the end, last decoder layer must reconstruct the inputs as multiple outputs.</p>
<p>Now I have between one and 9 inputs depending on the user’s choice and each input is a 1215x1519 matrix.</p>
<p>I’m rally stuck in first and last layers of this autoencoder.</p>
<p>Can anyone help me in this case?<br>
Thanks.</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: ptrblck
cooked: <p>You could implement the posted model architecture using <code>nn.ModuleList</code>s as seen here:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoders = nn.ModuleList()
        for _ in range(9):
            self.encoders.append(nn.Linear(4, 3))
        
        self.encoder = nn.Sequential(
            nn.Linear(9*3, 4),
            nn.ReLU(),
            nn.Linear(4, 3)
        )
        self.decoder = nn.Sequential(
            nn.Linear(3, 4),
            nn.ReLU(),
            nn.Linear(4, 9*3)
        )
        self.decoders = nn.ModuleList()
        for _ in range(9):
            self.decoders.append(nn.Linear(3, 4))
        
    def forward(self, inputs):
        out = []
        for idx, enc in enumerate(self.encoders):
            out.append(enc(inputs[idx]))
        out = torch.cat(out, dim=1)
        z = self.encoder(out)
        out = self.decoder(z)
        out = torch.split(out, 3, dim=1)
        outs = []
        for idx, dec in enumerate(self.decoders):
            outs.append(dec(out[idx]))
        return outs
            
model = MyModel()
inputs = [torch.randn(1, 4) for _ in range(9)]
outs = model(inputs)
</code></pre>
<p>I don’t know which activation functions should be used etc. so you could use this code snippet as a base implementation and adapt it to your use case.</p>

username: anantguptadbl
cooked: <p>Do you mean that you want to use the same model in case there are 5 inputs( each input being 1215x1519) or 1 input or 9 inputs?</p>

username: javad_t
cooked: <p>Thank you for your help, it was very helpful <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=10" title=":smile:" class="emoji" alt=":smile:"></p>
<p>I have another problem:<br>
Each input is a computed embedding of a graph and as I said before, each input is 1215x1519 matrix.<br>
On the other hand, since we do not have any labels for our data, the original  9x1215x1519 (9 is number of inputs) data considered as label and then considered a noisy version of original data with same shape for model input, in this way we’re trying to reconstruct input according to the labels.</p>
<p>In another implementation of this case with tensorflow and keras, developer fit the model with keras fit() function:</p>
<h5>
<a name="for-fitting-the-model-1" class="anchor" href="#for-fitting-the-model-1"></a>For fitting the model</h5>
<pre><code>history = model.fit(X_train_noisy, X_train, epochs=epochs, batch_size=batch_size, shuffle=True,
                    validation_data=(X_test_noisy, X_test),
                    callbacks=[EarlyStopping(monitor='val_loss', min_delta=0.0001, patience=5)])
</code></pre>
<p>Both X_train_noisy and X_train are 9x1215x1519.</p>
<p>Now I’m really confused about how to do this with pytorch!<br>
Thank you again</p>
<p><a class="mention" href="/u/anantguptadbl">@anantguptadbl</a></p>

username: ptrblck
cooked: <p>I don’t know what exactly the Keras model is doing, as the <code>fit</code> method doesn’t show any information about the loss function etc.<br>
You could thus check its internal implementation and use the same approach in PyTorch. E.g. if it’s some form of mse loss, use <code>nn.MSELoss</code> in PyTorch to calculate the loss.<br>
I also don’t know what the shape represents in Keras (Is dim0 the batch size? If so, the input shape looks wrong, but I’m also not deeply familiar with Keras) so you should check how each dimension is used inside the model.</p>

username: javad_t
cooked: <p>thank you again for you help</p>

username: Padmaksha_Roy
cooked: <p>hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , in my case, I have a single encoder (for all domains/classes)and multiple decoders. So is there no way to implement one encoder for multiple inputs or do I have to do it like this case - one encoder for each domain , concatenate/stack them, and pass through the decoder. Then separate the decoder outputs and minimize the reconstruction error? My forward function will look something like this:</p>
<p>Kindly give some insights here. Thanks!</p>

username: Padmaksha_Roy
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/7/a77d25a47bf8c735f614c21d61dc5972367d859b.png" alt="Multi_AE" data-base62-sha1="nTFRyGXzkDcF3a1HLkec9TqnGUr" width="605" height="315"><br>
puts[idx]))</p>
<blockquote>
<p>def forward(self, inputs):<br>
out = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for idx, enc in enumerate(self.encoders):<br>
out.append(enc(in</p>
</blockquote>
<blockquote>
<pre><code>    z = torch.cat(out, dim=1)
    #z = self.encoder(out)
    #out = self.decoder(z)
    z = torch.split(z, 3, dim=1)
    outs = []
    for idx, dec in enumerate(self.decoders):
        outs.append(dec(out[idx]))
    return outs
</code></pre>
</blockquote>

username: ptrblck
cooked: <p>Your approach looks alright and I’m not sure why it wouldn’t be possible.<br>
Do you see any errors or are you stuck at one point?</p>

username: Padmaksha_Roy
cooked: <p>hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , Thank you for the reply.</p>
<p>One more question is, whatever I told you above - is it implementing the same model that I pasted you above? My only doubt is that in that model there is a single encoder and multiple domain specific decoders whereas the model in this post has multiple encoders to collect inputs.</p>

username: ptrblck
cooked: <p>I honestly don’t know which approach would work the best for your use case (i.e. using multiple or a single shared encoder). Technically, i.e. from the point of view of your PyTorch code, the code looks correct and should work.</p>

username: Padmaksha_Roy
cooked: <p>Can you please tell me how do I change the code for a single shared encoder? Exactly which lines do I need to change? I want to change the code for a single shared encoder. Thank you!</p>

username: ptrblck
cooked: <p>Your code already indicates the usage of a single encoder.<br>
In the <code>__init__</code> method you would initialize a single <code>encoder</code> via:</p>
<pre><code class="lang-python">self.encoder = Encoder() # or whatever module you are using
</code></pre>
<p>and in the forward you would then iterate the outputs:</p>
<pre><code class="lang-python">out = []
for x_ in x: # assuming x is a list of tensors
    out.append(self.encoder(x_))
</code></pre>
<p>and could use <code>out</code> afterwards.</p>

username: Padmaksha_Roy
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , there is a small issue here. The code that you provided returns list of tensors with grads. Now, when I try to calculate MSE loss, it gives the following error. What should be the return type here and how do I convert it? to_tensor method don’t work here.</p>
<blockquote>
<p>criterion = nn.MSELoss()<br>
loss = criterion(outs,inputs)</p>
</blockquote>
<p>AttributeError: ‘list’ object has no attribute ‘size’</p>
<p>Kindly suggest!</p>

username: ptrblck
cooked: <p>You could <code>torch.cat</code> or <code>torch.stack</code> the list to a single tensor. Make sure it has the same shape as the target.</p>

username: Padmaksha_Roy
cooked: <p>Thank you! Solved this problem. I have another issue with the loss and I will try to find a post related to it. Thanks again!</p>

username: Padmaksha_Roy
cooked: <p>hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , can you please clarify one thing in the sample code that you provided - does the “9” represent the number of classes? The input that you provided is 9 * 1 * 4, so is it like you are considering 1 sample from each of the 9 classes and doing SGD? Also, why did you split the decoder out matrix into 3 parts?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Padmaksha_Roy" data-post="20" data-topic="139138">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/padmaksha_roy/48/47934_2.png" class="avatar"> Padmaksha_Roy:</div>
<blockquote>
<p>does the “9” represent the number of classes? The input that you provided is 9 * 1 * 4, so is it like you are considering 1 sample from each of the 9 classes and doing SGD?</p>
</blockquote>
</aside>
<p>No, the <code>9</code> represents the number of ecoders as explained in the original question:</p>
<blockquote>
<p>Now I have between one and 9 inputs depending on the user’s choice…</p>
</blockquote>
<aside class="quote no-group" data-username="Padmaksha_Roy" data-post="20" data-topic="139138">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/padmaksha_roy/48/47934_2.png" class="avatar"> Padmaksha_Roy:</div>
<blockquote>
<p>Also, why did you split the decoder out matrix into 3 parts?</p>
</blockquote>
</aside>
<p>I didn’t since <code>out = torch.split(out, 3, dim=1)</code> will create 9 outputs each with a size of <code>3</code>.<br>
You can just copy/paste my code and add print statements to the <code>forward</code> method to check the shape and content of each tensor if in doubt.</p>

username: Padmaksha_Roy
cooked: <p>hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , thank you so much! I did the prints and all but was unclear about the numbers. Now, it is much clear to me. Each domain is encoded with one encoder and 3 can be a batch size as inputs.</p>