Plot the latent space of an autoencoder

username: pedram1
cooked: <p>Hi all</p>
<p>I am trying to build an auto encoder. Actually I did it, however I was wondered to see if there is anyway of plotting latent space ? (my network shrink to two numbers in the latent space) I assumed that no PCA is required then.<br>
below u can find my codes:</p>
<p><span class="hashtag">#Autoencoder</span> and Autodecoder conv layer…<br>
class Autoencoder(nn.Module):</p>
<p>def <strong>init</strong>(self):<br>
super(Autoencoder,self).<strong>init</strong>()</p>
<pre><code>    self.encoder = nn.Sequential(
        nn.Conv1d(1,5,kernel_size=5, stride=2),
        nn.MaxPool1d(3, stride=1),
        nn.ReLU(True),
</code></pre>
<p>nn.Conv1d(5,10,kernel_size=5, stride = 2),<br>
nn.MaxPool1d(3,stride=1),<br>
nn.ReLU(True),</p>
<pre><code>        nn.Conv1d(10,15,kernel_size=5, stride=2),
        nn.MaxPool1d(3,stride = 1),
        nn.ReLU(True),

        nn.Conv1d(15,20,kernel_size = 4, stride = 1),
        nn.ReLU(True))

     self.decoder = nn.Sequential(          
        nn.ConvTranspose1d(20,15,kernel_size = 1, stride = 4),
        nn.ReLU(True),
        

        nn.ConvTranspose1d(15,10,kernel_size = 2,stride = 4),
        nn.ReLU(True),
        
        

        nn.ConvTranspose1d(10,5,kernel_size= 9,stride = 2),
        nn.ReLU(True),

        nn.ConvTranspose1d(5,1,kernel_size = 10, stride = 2),
        nn.ReLU (True))
</code></pre>
<p>def forward(self,x):<br>
x = self.encoder(x)<br>
x = self.decoder(x)<br>
return x</p>
<p>model = Autoencoder().cuda()</p>
<p>criterion = nn.MSELoss()</p>
<p>optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate,</p>
<p>weight_decay=1e-5)</p>
<p><span class="hashtag">#training</span> loop<br>
loss = 0<br>
epoch_num = 0<br>
error = []<br>
for epoch in range(num_epochs):<br>
for data in train_loader:<br>
img = data<br>
img = Variable(img).cuda()</p>
<pre><code>output = model (img)
loss = criterion (output,img)

optimizer.zero_grad()
loss.backward()
optimizer.step()
error.append(loss.item())


if epoch%10 == 9:
  epoch_num += epoch_num

  print ('\r Train Epoch : {}/{} \tLoss : {:.4f}'.format (epoch+1,num_epochs,loss/32))
</code></pre>
<p>model_save_name = ‘autoencoder2.pt’<br>
path = F"/content/gdrive/My Drive/{model_save_name}"<br>
torch.save(model.state_dict(), path)<br>
plt.plot(error)</p>
<p>/////////////////<br>
thank you</p>

username: sdv4
cooked: <p>The last line of your encoder reads:</p>
<aside class="quote no-group" data-username="pedram1" data-post="1" data-topic="63045">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pedram1/48/21032_2.png" class="avatar"> pedram1:</div>
<blockquote>
<p>nn.Conv1d(15,20,kernel_size = 4, stride = 1), nn.ReLU(True))</p>
</blockquote>
</aside>
<p>So where does the two element hidden state come in?</p>

username: pedram1
cooked: <p>15 is previous layer channel number, 20 is last layer channel number, and finally by applying a kernel size = 4 and stride = 1 , the length of the signal that was 94 shrink to just 2…<br>
I would like to plot that.</p>
<p>ps:my data size is (65535,94) that 65535 is my data point number and 94 is length of each data point (I am working with 1d signal)</p>

username: sdv4
cooked: <p>If your encoder shrinks the input down to an element of length 2, you would still have 20 elements of length 2 coming out of the last layer of your encoder, right? So the output of the encoder is a tensor with shape (batch_size, 20,2,1).</p>
<p>If this is correct, then your could plot each of the (20,2,1) elements of your data set by running PCA on the first dimension and specifying a single principal component. This would essentially be picking the most important filter and would give you elements of size (1,2,1) which you could then plot in 2D.</p>

username: pedram1
cooked: <p>I see,<br>
Just sth is slightly confusing for me. in (20,2,1) what that 1 means?</p>
<p>a lot thanks</p>

username: sdv4
cooked: <aside class="quote no-group" data-username="pedram1" data-post="5" data-topic="63045">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pedram1/48/21032_2.png" class="avatar"> pedram1:</div>
<blockquote>
<p>in (20,2,1) what that 1 means?</p>
</blockquote>
</aside>
<p>It just means that the tensor has 1 column. So (20, 2, 1) means the tensor contains 20 items with 2 rows and 1 column.</p>

username: pedram1
cooked: <p>and 20 means number of channels that I made them by applying  kernels, right?<br>
and I have no idea how to apply a PCA on that? Is there any module for that?</p>
<p>Thanks a lot</p>

username: sdv4
cooked: <aside class="quote no-group" data-username="pedram1" data-post="7" data-topic="63045">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pedram1/48/21032_2.png" class="avatar"> pedram1:</div>
<blockquote>
<p>and 20 means number of channels that I made them by applying kernels, right?</p>
</blockquote>
</aside>
<p>Yes, that’s right.</p>
<aside class="quote no-group" data-username="pedram1" data-post="7" data-topic="63045">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pedram1/48/21032_2.png" class="avatar"> pedram1:</div>
<blockquote>
<p>I have no idea how to apply a PCA on that? Is there any module for that?</p>
</blockquote>
</aside>
<p>Sci-kit learn has a nice PCA implementation. I would use that.</p>