Questions about Back Propagation and Network Setup: porting Encoder network from Tensorflow 1.12

Gears_Gears: <p>Dear Community,</p>
<p>I’m trying to port this Encoder network <a href="https://github.com/haebeom-lee/l2b/blob/master/encoder.py" class="inline-onebox" rel="noopener nofollow ugc">l2b/encoder.py at master · haebeom-lee/l2b · GitHub</a> and I’m getting confused. Could someone please give me some guidance on the following?</p>
<p>1 Should I have <strong>backward propagation</strong> for this network? Its loss is combined with the loss from the ‘main’ Model-Agnostic Meta-Learning network and optimized by the optimizer. (Tensorflow’s optimizer just takes the loss, not the parameters, so I’m not sure if this Encoder network’s parameters should be part of my argument when I initialize my torch.optim.Adam)<br>
2 Should I make this network by the ‘conventional’ method (inheriting from <strong>nn.Module</strong>) or the dictionary (<strong>dict[str, torch.Tensor]</strong>) method? I tried the former but found myself setting up a bunch of sub-networks and nn.LazyLinear which I don’t think is correct.<br>
The Tensorflow implementation is using functions like dense() defined in <a href="https://github.com/haebeom-lee/l2b/blob/master/layers.py" class="inline-onebox" rel="noopener nofollow ugc">l2b/layers.py at master · haebeom-lee/l2b · GitHub</a> is creating tensor variables, so it’s like using the dictionary method and doing the torch.nn.Functional.layer to manually calculate stuff rather than using torch.nn.that_same_layer’s forward method? I’m baffled! <img src="https://discuss.pytorch.org/images/emoji/apple/cry.png?v=12" title=":cry:" class="emoji" alt=":cry:" loading="lazy" width="20" height="20"></p>
<p>Thanks so much!<br>
P.S. I read the paper and, the Encoder is illustrated as:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/2/821f830caf5ccabcfe8b1f0a0e1ac26eb4895dfd.png" data-download-href="https://discuss.pytorch.org/uploads/default/821f830caf5ccabcfe8b1f0a0e1ac26eb4895dfd" title="encoder"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/821f830caf5ccabcfe8b1f0a0e1ac26eb4895dfd_2_690x188.png" alt="encoder" data-base62-sha1="iz7AhcgNtzmUHzqrxRI6GJDWSzH" width="690" height="188" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/821f830caf5ccabcfe8b1f0a0e1ac26eb4895dfd_2_690x188.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/821f830caf5ccabcfe8b1f0a0e1ac26eb4895dfd_2_1035x282.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/821f830caf5ccabcfe8b1f0a0e1ac26eb4895dfd_2_1380x376.png 2x" data-dominant-color="E9E9E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">encoder</span><span class="informations">1820×498 86.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Honestly, this hasn’t been much help to me. What I could make out from the paper is:<br>
info 1 The 3x3 Cones and FC is the main network (model-agnostic meta-learning, which is using the dict[str, torch.Tensor] method–probably because it gets backward propagation twice?)<br>
info 2 There are two layers of these NNs, which I think can be nn.Module? We get mean, variance, and cardinality from them, repeat the process, and send the statistics to the red, green, blue circles, which are the ‘balancing’ variables to improve MAML’s performance. (It seems like this network is just doing some complicated things to get basic statistics like mean and variance?–I can’t put my finger on it!)<br>
info 3 The <strong>x</strong> is a input. There are K of them in each class, and there are C classes total.<br>
Source: <a href="https://github.com/haebeom-lee/l2b/blob/master/images/encoder.png" class="inline-onebox" rel="noopener nofollow ugc">l2b/images/encoder.png at master · haebeom-lee/l2b · GitHub</a></p>