How to steer latent representation generation with an encoder?

username: sebt
cooked: <p>Hi there! I think using an autoencoder is the right path for me, but I am wondering what might be the best way of achieving the desired result.</p>
<p>I have some features (of length 279) for objects which are created by a deep model for ~10 mio objects. as these features are created in a self-supervised way, they hold some redundant and undesired information. The end goal is to further model these features for downstream tasks which would benefit from having fewer features. Therefore I find autoencoders enticing, for simply reducing the number of features (not for generative purposes) but my main question is this: Would there be a way of steering the latent representation towards discarding the unwanted information?</p>
<p>if an autoencoder is trained to perfectly reconstruct its inputs, it will of course include the unwanted information unless its told otherwise. Therefore, would it be possible for me to label the data such that samples that are somewhat different in the 279-feature representation get the same label and this could guide the autoencoder to represent them together? Off the top of my head, having a model that encodes a reduced latent representation and uses it to predict labelled classes could be promising.</p>
<p>Below is a UMAP of ~10K objects.<br>
The three large clusters in the top right are mostly desired to be modelled as the same.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/2/42a82fe895b77c0ed3d23416e28119b2e50927e8.png" alt="UMAP" data-base62-sha1="9vFQQoJufD64FLQsVr07MQKjS7m" width="288" height="277"></p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="sebt" data-post="1" data-topic="143753">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/e79b87/48.png" class="avatar"> sebt:</div>
<blockquote>
<p>Off the top of my head, having a model that encodes a reduced latent representation and uses it to predict labelled classes could be promising.</p>
</blockquote>
</aside>
<p>That sounds generally like a valid idea and I would imagine you could use the latent tensor and feed it into a classifier to train the generator using both paths, the classifier as well as the decoder.</p>

username: sebt
cooked: <p>Thank you for your input and time ptrblck! I’ll give it a go.</p>

username: sebt
cooked: <p>Okay so I have trained the first autoencoder only with reconstruction as objective for now. Its based on 50K samples of original size 588 which is mapped to 18 and then reconstructed to 588. I have across 3 epochs achieved between 0.01 and 0.04 MSE on the trainingset. But I am experiencing some odd behavior when I look at the UMAPs.</p>
<p>Below is the UMAP of the original data:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/9/69cf91fca2da4f7a8ca077b7cdd65bdcbcea28f3.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/69cf91fca2da4f7a8ca077b7cdd65bdcbcea28f3" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/69cf91fca2da4f7a8ca077b7cdd65bdcbcea28f3_2_308x250.jpeg" alt="image" data-base62-sha1="f62QNFrk9NQZ2QaoKTa90umuhLt" width="308" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/69cf91fca2da4f7a8ca077b7cdd65bdcbcea28f3_2_308x250.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/69cf91fca2da4f7a8ca077b7cdd65bdcbcea28f3_2_462x375.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/69cf91fca2da4f7a8ca077b7cdd65bdcbcea28f3_2_616x500.jpeg 2x" data-dominant-color="CEE1EE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1268×1029 138 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here is the UMAP of the size 18 encoded variables:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2ebda007705965ab7bb1dd5e194880691992331a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/2ebda007705965ab7bb1dd5e194880691992331a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2ebda007705965ab7bb1dd5e194880691992331a_2_345x225.jpeg" alt="image" data-base62-sha1="6FuetQTXTD6zD5w2znKuF3wkxdM" width="345" height="225" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2ebda007705965ab7bb1dd5e194880691992331a_2_345x225.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2ebda007705965ab7bb1dd5e194880691992331a_2_517x337.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2ebda007705965ab7bb1dd5e194880691992331a_2_690x450.jpeg 2x" data-dominant-color="DEEBF3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1415×926 139 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here is the UMAP of the reconstructed (size 588 again):<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/3/a3a13322df0b430aeffecf94a13157c6050394c5.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a3a13322df0b430aeffecf94a13157c6050394c5" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a3a13322df0b430aeffecf94a13157c6050394c5_2_345x226.jpeg" alt="image" data-base62-sha1="nlxcbyVFJHyihhkDt44us3JxuER" width="345" height="226" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a3a13322df0b430aeffecf94a13157c6050394c5_2_345x226.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a3a13322df0b430aeffecf94a13157c6050394c5_2_517x339.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a3a13322df0b430aeffecf94a13157c6050394c5_2_690x452.jpeg 2x" data-dominant-color="DDEAF2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1411×925 140 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The “original data” is exactly what the network is given. all of the UMAPs are computed with the same settings. All data have the same scale/range (0-1), data type (float64) etc. I noticed that my original data has more decimals, but rounding it to same decimals made no difference.</p>
<p>also as a sanity check, heres  a scatter of original data vs reconstructed for a single sample:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/5/455a75c04cf47403aa593a4cefff69b1cd39663a.png" alt="image" data-base62-sha1="9TwKOnXcLfvpjeH1RyUbqO9Ly3w" width="268" height="174"><br>
(generally the reconstructed overestimates the values slightly but there is a great correlation coefficient)</p>
<p>Does anyone have any idea what might be going on?</p>

username: sebt
cooked: <p>EDIT,EDIT: I cant reproduce what I’ve written below, so probably normalization wasnt the key. At the moment I guess that while the network is producing a low MSE, it still introduces some dynamic which UMAP picks up on and maps instead of the topology of the original data.</p>
<p>Allow me to answer the question myself… I remember one of my students showed me such a UMAP wrt normalization of data so I tried to add in LayerNorm into the encoder and decoder.</p>
<p>Here is a UMAP of the reconstructed samples where the model includes LayerNorm between the Linear and ReLU:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/b/2bfe0b2a96855b04c577f163614af2628fc40691.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/2bfe0b2a96855b04c577f163614af2628fc40691" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2bfe0b2a96855b04c577f163614af2628fc40691_2_345x225.jpeg" alt="image" data-base62-sha1="6haOFcWRGrGrvmA2XHNQxZnmuvT" width="345" height="225" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2bfe0b2a96855b04c577f163614af2628fc40691_2_345x225.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2bfe0b2a96855b04c577f163614af2628fc40691_2_517x337.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2bfe0b2a96855b04c577f163614af2628fc40691_2_690x450.jpeg 2x" data-dominant-color="CEE1ED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1405×920 173 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I get the same MSE across the trainingset though.</p>