Initialize a DNN with the weights matrix of an autoencoder

username: lima
cooked: <p>I’m trying to implement <a href="https://reader.elsevier.com/reader/sd/pii/S0264127518308682?token=A6FBC3ABBEE0E3345FE54A4639B69D88C7FFE7D02DA26181D34571DD447EBF5708998467594191BFAB549C6C0C532E42&amp;originRegion=eu-west-1&amp;originCreation=20220318095520" rel="noopener nofollow ugc">this paper</a> following <a href="https://github.com/sf254/SC/blob/master/DNN4layers.m" rel="noopener nofollow ugc">this matlab code</a>, but I’m new to pytorch, so it’s a little over my head.<br>
The idea is to use the optimized weights of the autoencoder to initialize the DNN.<br>
My questions are given the below autoencoder and DNN:<br>
1- How to make the autoencoder return the parameters (weight+bias)?<br>
2- How to initialize the layers of the DNN with those parameters?</p>
<p>Stacked AE:</p>
<pre><code>class AutoEncoder(nn.Module):

    def __init__(self, input_size, output_size):
        super(AutoEncoder, self).__init__()

        self.encoder = nn.Sequential(
            nn.Linear(input_size, output_size),
            nn.ReLU(),
        )
        self.decoder = nn.Sequential(
            nn.Linear(output_size, input_size), 
            nn.ReLU(),
        )

        self.criterion = nn.MSELoss()
        self.optimizer = torch.optim.SGD(self.parameters(), lr=0.1)

    def forward(self, x):
        x = x.detach()
        y = self.encoder(x)

        if self.training:
            x_reconstruct = self.decoder(y)
            loss = self.criterion(x_reconstruct, Variable(x.data, requires_grad=False))
            self.optimizer.zero_grad()
            loss.backward()
            self.optimizer.step()
            
        return y.detach()

    def reconstruct(self, x):
        return self.decoder(x)


class StackedAutoEncoder(nn.Module):
    r"""
    A stacked autoencoder made from the autoencoders above.
    Each autoencoder is trained independently and at the same time.
    """

    def __init__(self):
        super(StackedAutoEncoder, self).__init__()

        self.ae1 = AutoEncoder(54, 20)
        self.ae2 = AutoEncoder(20, 8)
        self.ae3 = AutoEncoder(8, 3)
        self.ae4 = AutoEncoder(3, 1)


    def forward(self, x):
        a1 = self.ae1(x)
        a2 = self.ae2(a1)
        a3 = self.ae3(a2)
        a4 = self.ae4(a3)
        return a1, a2, a3,a4
</code></pre>
<p>and the DNN:</p>
<pre><code>class DNN(nn.Module):

    n_classes = 1

    def __init__(self, d_in, n_layers, units, dropout, activation=nn.GELU, final_act=nn.Sigmoid):

        super().__init__()

        assert len(units) == n_layers

        self.d_in = d_in
        self.n_layers = n_layers
        self.units = units
        self.activation = activation
        self.final_act = final_act

        if dropout is None:
            self.dropout = None
        else:
            self.dropout = nn.Dropout(p=dropout)

        if dropout is None:
            layers = [nn.Linear(d_in, units[0]), self.activation()]
            if n_layers &gt; 1:
                for i, u in enumerate(units):
                    if i &gt; 0:
                        layers.extend([nn.Linear(units[i-1], units[i]),
                                       self.activation()])
        else:
            layers = [nn.Linear(d_in, units[0]), self.activation(),
                      self.dropout]
            if n_layers &gt; 1:
                for i, u in enumerate(units):
                    if i &gt; 0:
                        layers.extend([nn.Linear(units[i-1], units[i]),
                                       self.activation(), self.dropout])

        self.final_layer = [nn.Linear(units[-1], self.n_classes), self.final_act()]
        layers.extend(self.final_layer)

        self.layers = nn.Sequential(*layers)

    def forward(self, x):

        return self.layers(x)</code></pre>

username: wanchaol
cooked: <p>Hi <a class="mention" href="/u/lima">@lima</a>, thanks for posting.</p>
<p>If you want the autoencoder to return the parameters, you can simply call <code>model.parameters()</code>, <code>model.named_parameters()</code>, or <code>model.state_dict()</code> (the latter one involves buffers as well), which will give you all the parameters of the encoder, then in your DDN, you can manually copy the parameters over or use <code>model.load_state_dict()</code> if the model architecture is the same as your saved state_dict. For details you can refer to this <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" rel="noopener nofollow ugc">note</a></p>

username: lima
cooked: <p>Thank you <a class="mention" href="/u/wanchaol">@wanchaol</a> for your input, but what if I want to have the output of the first layer (or any particular layer) and use that as the input of another model?</p>

username: wanchaol
cooked: <p>couldn’t you just store the output of the first layer somewhere after the first layer execution, and save it if you want, then use it as the input of another model? what’s the blocker here?</p>