Can anyone share the code of stack auto-encoder? Please

Muhammad_Furqan_Rafi: <p>Can anyone share the code of stack auto-encoder?</p>

michaelklachko: <p>Click the first link: <a href="https://www.google.com/search?q=pytorch+autoencoder" rel="nofollow noopener">https://www.google.com/search?q=pytorch+autoencoder</a></p>

Muhammad_Furqan_Rafi: <p>Thanks but its a simple auto-encoder and I want the code for stack autoencoder</p>

michaelklachko: <p>What is “stack autoencoder”?</p>

Muhammad_Furqan_Rafi: <p>Trying to implement this in pytorch</p>
<p><a href="https://www.mathworks.com/help/deeplearning/examples/train-stacked-autoencoders-for-image-classification.html" rel="nofollow noopener">https://www.mathworks.com/help/deeplearning/examples/train-stacked-autoencoders-for-image-classification.html</a></p>

michaelklachko: <p>It’s called unsupervised pretraining, and they do it one layer at the time. No one does it like that anymore, because you get the same or better accuracy by training the same network from scratch (random initialization).</p>
<p>If you still insist on doing it like that (why?), then the code from the link I posted above is sufficient. Just remove upper layers, then add them back one by one.</p>
<p>Seems like you’re a beginner, so I suggest you take an online course first (fast.ai is pretty good).</p>

Muhammad_Furqan_Rafi: <p>Of course, I’m a newcomer to this world <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=6" title=":blush:" class="emoji" alt=":blush:">.  But I have to do it for some reasons, here is the code you posted, Could you please tell me which layers I have to remove and then add them back one by one.</p>
<p>import os</p>
<p>import torch<br>
import torchvision<br>
from torch import nn<br>
from torch.autograd import Variable<br>
from torch.utils.data import DataLoader<br>
from torchvision import transforms<br>
from torchvision.datasets import MNIST<br>
from torchvision.utils import save_image</p>
<p>if not os.path.exists(’./mlp_img’):<br>
os.mkdir(’./mlp_img’)</p>
<p>def to_img(x):<br>
x = 0.5 * (x + 1)<br>
x = x.clamp(0, 1)<br>
x = x.view(x.size(0), 1, 28, 28)<br>
return x</p>
<p>num_epochs = 100<br>
batch_size = 128<br>
learning_rate = 1e-3</p>
<p>img_transform = transforms.Compose([<br>
transforms.ToTensor(),<br>
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))<br>
])</p>
<p>dataset = MNIST(’./data’, transform=img_transform)<br>
dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=True)</p>
<p>class autoencoder(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(autoencoder, self).<strong>init</strong>()<br>
self.encoder = nn.Sequential(<br>
nn.Linear(28 * 28, 128),<br>
nn.ReLU(True),<br>
nn.Linear(128, 64),<br>
nn.ReLU(True), nn.Linear(64, 12), nn.ReLU(True), nn.Linear(12, 3))<br>
self.decoder = nn.Sequential(<br>
nn.Linear(3, 12),<br>
nn.ReLU(True),<br>
nn.Linear(12, 64),<br>
nn.ReLU(True),<br>
nn.Linear(64, 128),<br>
nn.ReLU(True), nn.Linear(128, 28 * 28), nn.Tanh())</p>
<pre><code>def forward(self, x):
    x = self.encoder(x)
    x = self.decoder(x)
    return x
</code></pre>
<p>model = autoencoder().cuda()<br>
criterion = nn.MSELoss()<br>
optimizer = torch.optim.Adam(<br>
model.parameters(), lr=learning_rate, weight_decay=1e-5)</p>
<p>for epoch in range(num_epochs):<br>
for data in dataloader:<br>
img, _ = data<br>
img = img.view(img.size(0), -1)<br>
img = Variable(img).cuda()<br>
# ===================forward=====================<br>
output = model(img)<br>
loss = criterion(output, img)<br>
# ===================backward====================<br>
optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step()<br>
# ===================log========================<br>
print(‘epoch [{}/{}], loss:{:.4f}’<br>
.format(epoch + 1, num_epochs, loss.data[0]))<br>
if epoch % 10 == 0:<br>
pic = to_img(output.cpu().data)<br>
save_image(pic, ‘./mlp_img/image_{}.png’.format(epoch))</p>
<p>torch.save(model.state_dict(), ‘./sim_autoencoder.pth’)</p>

osofr: <p>But why would you ever <strong>not</strong> opt for reducing the number of weights by 1/2 in your model? This is what old-school stacked AE acomplishes. Can you at the very least point to the experiments / papers that demonstrate the superiority or non-inferiority of back-to-back training (without shared parameters), besides the statement that “all cool kids do it these days”?</p>

michaelklachko: <p>What are you talking about? How do you reduce weights with stacked AEs?</p>

osofr: <p>Actually, I was really wrong on this one. My apologies.</p>
<p>Early instances of (denoising) AE use exactly the same (transposed) weights for each decoder/encoder layer (but different biases). Basically described in all DL textbooks, happy to send the references. Sharing the transposed weights allows you to reduce the number of parameters by 1/2 (training each decoder/ encoder one layer at a time).</p>
<p>My point of reference was an unsupervised clustering paper from facebook (that used denoising AEs as a start): <a href="https://arxiv.org/abs/1511.06335" rel="nofollow noopener">https://arxiv.org/abs/1511.06335</a></p>
<p>Even though they trained the denoising (stacked) AE one layer at a time, they used separate parameters for the decoder and the encoder layers. So I was clearly wrong.</p>
<p>However, my question remains the same: where is the blog post, experiment, paper that shows that weight-sharing (b/ween encoding/decodin layers) in a stacked denoising AE gives absolutely no benefit? If you have a question about what is a stacked AE just read the referenced paper.</p>

michaelklachko: <p>I’ve never actually said anything about weight sharing. I was merely pointing out that unsupervised pre-training for classification is unnecessary, and that AEs can be trained end to end, rather than one layer at a time. This has nothing to do with weight sharing.</p>
<p>Regarding weight sharing, it might act as a regularizer, which could help when the model is overparametrized. For small models it could lead to underfitting. I don’t have an intuition how it affects the training process.</p>
<p>More importantly, I’m not sure if you can share weights when you use convolutional layers in encoder and decoder: could the same filters be used for both convolution and “deconvolution” purposes? Do you know any papers that did it?</p>

osofr: <aside class="quote no-group" data-username="michaelklachko" data-post="11" data-topic="37350">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michaelklachko/48/541_2.png" class="avatar"> michaelklachko:</div>
<blockquote>
<p>I’ve never actually said anything about weight sharing. I was merely pointing out that unsupervised pre-training for classification is unnecessary, and that AEs can be trained end to end, rather than one layer at a time. This has nothing to do with weight sharing.</p>
</blockquote>
</aside>
<p>Got it! Makes sense. Any chance you are aware of any experiments that show this? Would be curious to read, no worries if you don’t have these handy.</p>
<aside class="quote no-group" data-username="michaelklachko" data-post="11" data-topic="37350">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michaelklachko/48/541_2.png" class="avatar"> michaelklachko:</div>
<blockquote>
<p>More importantly, I’m not sure if you can share weights when you use convolutional layers in encoder and decoder: could the same filters be used for both convolution and “deconvolution” purposes? Do you know any papers that did it?</p>
</blockquote>
</aside>
<p>Weight sharing is not possible at all in convolution-deconvolution scenario, as far as I understand it. Which is probably the reason why weight sharing has fallen out of favor in feedforward AEs as well. Even though I could not find any single references for this, I’ve noticed that at some point the feedforward AEs implementations moved away from weight sharing.</p>