Implement AGC optimizer helper class

BongHoe_Koo: <p>C++ Optimizer header link<br>
<span class="hashtag-raw">#-----------------------------------------------------------------------------------------------------</span>---------------------</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/main/torch/csrc/api/include/torch/optim/optimizer.h">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/main/torch/csrc/api/include/torch/optim/optimizer.h" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/main/torch/csrc/api/include/torch/optim/optimizer.h" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/main/torch/csrc/api/include/torch/optim/optimizer.h</a></h4>


      <pre><code class="lang-h">#pragma once

#include &lt;ATen/Tensor.h&gt;
#include &lt;c10/util/Exception.h&gt;
#include &lt;c10/util/flat_hash_map.h&gt;

#include &lt;torch/arg.h&gt;
#include &lt;torch/csrc/Export.h&gt;

#include &lt;algorithm&gt;
#include &lt;functional&gt;
#include &lt;iterator&gt;
#include &lt;memory&gt;
#include &lt;string&gt;
#include &lt;vector&gt;

// Forward declarations confuse Doxygen
#ifndef DOXYGEN_SHOULD_SKIP_THIS
namespace at {
class Tensor;
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/main/torch/csrc/api/include/torch/optim/optimizer.h" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p><span class="hashtag-raw">#-----------------------------------------------------------------------------------------------------</span>---------------------</p>
<p>Hi pytorch team</p>
<p>I really want to train NFNet model(Nomalization Free series and my custom model) in C++<br>
but the problem is that here is no official AGC optimizer in libtorch (Adaptive Gradient Clipping)</p>
<p>reference link : <a href="https://arxiv.org/abs/2102.06171" class="inline-onebox" rel="noopener nofollow ugc">[2102.06171] High-Performance Large-Scale Image Recognition Without Normalization</a><br>
reference pytorch code link : <a href="https://github.com/gellston/DeepLearningStudy/blob/main/torch/util/nf_helper.py" class="inline-onebox" rel="noopener nofollow ugc">DeepLearningStudy/torch/util/nf_helper.py at main · gellston/DeepLearningStudy · GitHub</a> (target AGC python class)</p>
<p>That is why i made some question list for AGC implementation<br>
(FYI : i am c++ developer and i know basic c++ container like std::vector, std::map , blabla )</p>
<ol>
<li>
<p>Is there simple optimizer inheritance tutorial code for libtorch C++?</p>
</li>
<li>
<p>how to unpack params of optimizer argument and pack again in C++? (for skipping specific parameter)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/4/b4a57df940f739a4965c3b890cf840cdbc5ff569.png" data-download-href="https://discuss.pytorch.org/uploads/default/b4a57df940f739a4965c3b890cf840cdbc5ff569" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/4/b4a57df940f739a4965c3b890cf840cdbc5ff569.png" alt="image" data-base62-sha1="pM4vBOnkpbUOUAUlSSZfIVfSD4R" width="690" height="288" data-dominant-color="131517"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">937×392 18.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
(<a href="https://github.com/gellston/DeepLearningStudy/blob/e51b11565e924becc39355bf712c398902bc323a/torch/util/nf_helper.py#L54" class="inline-onebox" rel="noopener nofollow ugc">DeepLearningStudy/torch/util/nf_helper.py at e51b11565e924becc39355bf712c398902bc323a · gellston/DeepLearningStudy · GitHub</a>)</p>
</li>
<li>
<p>how can i get p, p.grad in libtorch<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3f3e1af4e822ee19880c91fc8b5a22c6652fed0.png" data-download-href="https://discuss.pytorch.org/uploads/default/d3f3e1af4e822ee19880c91fc8b5a22c6652fed0" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3f3e1af4e822ee19880c91fc8b5a22c6652fed0.png" alt="image" data-base62-sha1="uf1g7ZhX3e2l7puIV7s3CTzKGpW" width="690" height="363" data-dominant-color="131417"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">817×430 15.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
(<a href="https://github.com/gellston/DeepLearningStudy/blob/e51b11565e924becc39355bf712c398902bc323a/torch/util/nf_helper.py#L93" class="inline-onebox" rel="noopener nofollow ugc">DeepLearningStudy/torch/util/nf_helper.py at e51b11565e924becc39355bf712c398902bc323a · gellston/DeepLearningStudy · GitHub</a>)</p>
</li>
<li>
<p>how to calculate total model loss in step function C++ (like closure)<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/7/470b6cce26c6a0ecf14c7685faa03a5c1fa499c2.png" alt="image" data-base62-sha1="a8unsyhC4rhIIJ7eyhMi9vHBiLM" width="686" height="309"><br>
<a href="https://github.com/gellston/DeepLearningStudy/blob/e51b11565e924becc39355bf712c398902bc323a/torch/util/nf_helper.py#L89" class="inline-onebox" rel="noopener nofollow ugc">DeepLearningStudy/torch/util/nf_helper.py at e51b11565e924becc39355bf712c398902bc323a · gellston/DeepLearningStudy · GitHub</a></p>
</li>
<li>
<p>if there is easy way to implement AGC (without inheritance optimizer) please share</p>
</li>
</ol>