Thoughts on Proposed Faster Training Plugin

username: Jdw136
cooked: <p>Hi,</p>
<p>I am a university student that is currently developing a Pytorch add-on. The add-on would aim to make models train 70%-90% faster.</p>
<p>However, this speed boost while training comes at the expense of model accuracy (approximately 3%-5%).</p>
<p>I was hoping to open a discussion about people’s thoughts on this proposed framework. More specifically, what people think about the speed vs accuracy trade off.</p>

username: ptrblck
cooked: <p>A model with a 2x throughput for an accuracy loss of 5% sounds like a really interesting use case.<br>
Depending, how your method works, would it be possible to “restore the better accuracy” by disabling your method towards the end of the training?</p>

username: Jdw136
cooked: <p>Unfortunately not. However you can easily build much larger networks to make up for the lost accuracy.</p>

username: ptrblck
cooked: <p>Wouldn’t this slow down the training again and we have the original model back? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Anyway, I still think it’s an interesting idea and some quantization approaches use exactly this strategy (losing a bit of model accuracy to gain speed).</p>

username: Jdw136
cooked: <p>I suppose you’re right. I didn’t think of that!</p>
<p>But do you think this loss of accuracy would generally be worth the speed boost?</p>

username: ptrblck
cooked: <p>It might depend a bit on your use case and how exactly the method is applied.<br>
E.g. if your method would also speedup the inference (and even on e.g. the mobile platform), this might be a valid “deployment” mode (a lower model accuracy <em>might</em> be acceptable on mobile platforms).</p>
<p>For general training you would have to check how much work implementing your method would be, as even small accuracy drops might scare people from trying out your method. E.g. the <a href="https://pytorch.org/blog/introduction-to-quantization-on-pytorch/#accuracy-results">quantization accuracy results</a> show a smaller accuracy drop for a <a href="https://pytorch.org/blog/introduction-to-quantization-on-pytorch/#performance-results">similar or higher performance gain</a>.</p>
<p>But again, it might depend on the overall complexity of your method and in which situation you are working on it. E.g. if it’s a new research direction, which might be part of your studies (or PhD), small steps might be good enough to create a foundation and fix some issues in the next steps.</p>

username: Jdw136
cooked: <p>Hello again <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20">! I took your suggestions regarding the potentially concerning drop in accuracy seriously. I entirely agree with your points. Not only will the drop in performance possibly scare some users off, but a 5% drop in accuracy could potentially make the plugin useless in some applications (e.g. Medical Procedures, Self-driving cars, other high-risk tasks).</p>
<p>After redesigning the algorithm, I was able to remove the performance gap! Now the plugin is able to train a model approximately 50% faster without any noticeable drop in performance.</p>
<p>Below I have included a screenshot of my training to demonstrate the plugin’s performance. The same model was trained on the MNIST Digits dataset using a batch size of 32 and 10 epochs. The left panel on the image is training using the plugin and the right panel is training without the plugin.</p>
<p>I apologize for censoring some parts of the image (it was necessary to protect my privacy), and for using the Tensorflow framework instead of PyTorch.</p>
<p>If you have any comments please get back to me. Thank you!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/2/42871a430f6882f094fce21a925ab07c09ec3324.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/42871a430f6882f094fce21a925ab07c09ec3324" title="MNIST_Plugin_Test"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/2/42871a430f6882f094fce21a925ab07c09ec3324_2_690x361.jpeg" alt="MNIST_Plugin_Test" data-base62-sha1="9uwY4QWwNfjjRo2GHO6uhQ0w912" width="690" height="361" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/2/42871a430f6882f094fce21a925ab07c09ec3324_2_690x361.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/2/42871a430f6882f094fce21a925ab07c09ec3324_2_1035x541.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/2/42871a430f6882f094fce21a925ab07c09ec3324_2_1380x722.jpeg 2x" data-dominant-color="371D31"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">MNIST_Plugin_Test</span><span class="informations">1851×971 305 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Jdw136" data-post="7" data-topic="93761">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/e95f7d/48.png" class="avatar"> Jdw136:</div>
<blockquote>
<p>After redesigning the algorithm, I was able to remove the performance gap! Now the plugin is able to train a model approximately 50% faster without any noticeable drop in performance.</p>
</blockquote>
</aside>
<p>That sounds awesome, congrats! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
Looking forward to read about your method in a paper, so please post it once you finished your experiments.</p>