Denoising Autoencoder for Multiclass Classification

Kirty_Vedula: <p>This is a follow up to the question I asked <a href="https://discuss.pytorch.org/t/denoising-autoencoder-testing-mode-for-multiclass-classification/70789">previously</a> a week ago. Thanks to <a class="mention" href="/u/ptrblck">@ptrblck</a>, I followed his advice on following Approach 2 in my question and I am getting better results. However, there still seems to be a few issues.</p>
<p>Here’s just a quick intro: I am training an autoencoder for a multiclass classification problem where I transmit 16 equiprobable messages and send them through a denoising autoencoder to receive them. I am trying to implement the result (modification of Fig. 3b) in this paper, to be specific: Please refer to Fig. 2 in <a href="https://arxiv.org/pdf/1702.00832.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1702.00832.pdf</a> for the model.</p>
<p>Please have a look at my code here: <a href="https://gist.github.com/kirtyvedula/9698a7a728d1484b3bffb0394c1191d7" rel="nofollow noopener">https://gist.github.com/kirtyvedula/9698a7a728d1484b3bffb0394c1191d7</a></p>
<p>I have the following questions.The code runs fine, but it is very sensitive to the number of training and test examples I provide, batch size and learning rate. And I am also having problems with running it on a GPU (NVIDIA 1060 Ti). I have a Keras implementation which works perfectly. (I can upload it if necessary). PyTorch gives me weird issues. For example, it gives me “better” results or overcompensates something. Foe example, the errors go down way too low, which is not the scenario at all.</p>
<pre><code class="lang-auto">Eb/N0: 0.0 | test BLER: 0.1104
Eb/N0: 0.5 | test BLER: 0.0932
Eb/N0: 1.0 | test BLER: 0.0752
Eb/N0: 1.5 | test BLER: 0.0611
Eb/N0: 2.0 | test BLER: 0.0504
Eb/N0: 2.5 | test BLER: 0.0389
Eb/N0: 3.0 | test BLER: 0.0312
Eb/N0: 3.5 | test BLER: 0.0222
Eb/N0: 4.0 | test BLER: 0.0162
Eb/N0: 4.5 | test BLER: 0.0121
Eb/N0: 5.0 | test BLER: 0.0082
Eb/N0: 5.5 | test BLER: 0.0058
Eb/N0: 6.0 | test BLER: 0.0031
Eb/N0: 6.5 | test BLER: 0.0023
Eb/N0: 7.0 | test BLER: 0.0012
Eb/N0: 7.5 | test BLER: 0.0007
Eb/N0: 8.0 | test BLER: 0.0002
Eb/N0: 8.5 | test BLER: 0.0001
Eb/N0: 9.0 | test BLER: 0.0000
Eb/N0: 9.5 | test BLER: 0.0000
Eb/N0: 10.0 | test BLER: 0.0000
Eb/N0: 10.5 | test BLER: 0.0000
</code></pre>
<p>Sorry if these are too many details. Those are just boilerplate digital communications stuff. Please have a look at the general flow of the program and see if I am doing something wrong here.</p>
<ol>
<li>Do the training and testing loop make sense?</li>
<li>Do the choices of batch size, number of training and test examples</li>
<li>Am I printing the right losses? I checked <a class="mention" href="/u/ptrblck">@ptrblck</a>’s suggestion last time and it was not an overflow problem.</li>
</ol>
<p>Any ideas would be most welcome. Thank you!</p>

ptrblck: <ol>
<li>
<p>The training loss looks alright. I used the provided shapes to check the input and target and think <code>y = (y.long()).view(-1)</code> should be unnecessary. Could you verify it?<br>
The test loss might be alright. I’m not too familiar with your use case, but note that you are not applying the <a href="https://gist.github.com/kirtyvedula/9698a7a728d1484b3bffb0394c1191d7#file-awgn_pytorch_v1-py-L43-L45">normalization</a>. You’ve added the noise addition inside your loop, which should be fine, but I’m not sure, if you need the normalization in your test loop. If so, consider writing it as another class method you could call similar to <code>model.transmitter</code>.</p>
</li>
<li>
<p>Not sure about it and it should be use case dependent.</p>
</li>
<li>
<p>Also looks alright.</p>
</li>
</ol>
<p>What kind of issues are you seeing using PyTorch?<br>
Is this regarding using the GPU or some other errors?</p>

Kirty_Vedula: <p>Thanks a lot <a class="mention" href="/u/ptrblck">@ptrblck</a>! I think normalization might be it. That seems to be the cause for inconsistency.</p>
<p>One other inconsistency is that I am unable to run it on a GPU. I did all the usual checks and made sure my device (Windows, GTX GeForce 1060) is running standard codes. There must be something wrong in here where I am changing the device back to CPU or something like that. I am unable to get it run on a GPU. Could you please have a look at my code and check if I am doing something wrong in terms on <code>.to(device)</code> commands</p>

ptrblck: <p>Your code runs fine on my GPU without any changes.<br>
Do you get any error message or is the utilization at 0%?</p>

Kirty_Vedula: <p>Oh, that’s good to hear. It does not give me any errors, but it looks like it isn’t utilizing my GPU to a full extent. It even shows up in <code>nvidia-smi</code>. Am I reading it wrong or does my GPU seem to work? I am concerned because when my Keras implementation of the same algorithm runs on the GPU, it behaves differently.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c0b400bfdaf131369d4d26e6005d2b78178a7da.png" data-download-href="https://discuss.pytorch.org/uploads/default/9c0b400bfdaf131369d4d26e6005d2b78178a7da" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c0b400bfdaf131369d4d26e6005d2b78178a7da_2_475x375.png" alt="image" data-base62-sha1="mgqAvkxZSnQMv2z6zVD6yKsqjbQ" width="475" height="375" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c0b400bfdaf131369d4d26e6005d2b78178a7da_2_475x375.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c0b400bfdaf131369d4d26e6005d2b78178a7da_2_712x562.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c0b400bfdaf131369d4d26e6005d2b78178a7da.png 2x" data-dominant-color="F7F8F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">908×715 110 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/4/04d0eb95a91f6385b99cf60b6ef55c50206d7e12.png" data-download-href="https://discuss.pytorch.org/uploads/default/04d0eb95a91f6385b99cf60b6ef55c50206d7e12" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/4/04d0eb95a91f6385b99cf60b6ef55c50206d7e12.png" alt="image" data-base62-sha1="GBwcNqJwN1rpkNlAp3oH2eCeI2" width="517" height="259" data-dominant-color="0E170E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">837×420 12.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Based on the output of <code>nvidia-smi</code> it seems your GPU is working.</p>
<p>Regarding the task manager output: there should be a “compute” tab you could activate somehow to see the actual compute workload. I’m not that familiar with Windows, but we had similar issues here in the forum, where the Windows task manager didn’t show the actual GPU usage.</p>
<p>PS: You could speed up your code by:</p>
<ul>
<li>Precompute what seems to be fixed in your model. E.g. the <code>sqrt</code> calculation seems to be fixed to <code>sqrt(7)</code>, so you might instead register it as a buffer in your <code>__init__</code> via <code>self.register_buffer('sqrt', torch.sqrt(torch.tensor(7.)))</code>.</li>
<li>Use local variables instead of global: <code>self.k</code> instead of <code>k</code> (same for <code>n_channel</code>).</li>
<li>Use multiple workers (<code>num_workers &gt; 0</code>) in your <code>DataLoader</code>.</li>
</ul>
<p>I haven’t profiled the code, but these changes seems to yield a speedup just looking at the outputs.</p>

Kirty_Vedula: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="71994">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<ul>
<li>Use local variables instead of global: <code>self.k</code> instead of <code>k</code> (same for <code>n_channel</code> ).</li>
<li>Use multiple workers ( <code>num_workers &gt; 0</code> ) in your <code>DataLoader</code> .</li>
</ul>
</blockquote>
</aside>
<p>These worked, in addition to putting my code in a dedicated main function and calling it. Thank you so much for your help. However, I am still not able to see the performance on compute_0 and compute_1 tabs on Task Manager. That’s okay for now, I think, since I do some improvement in speed.</p>