How to construct the conditional decoder for each input class?

MinsooKim: <p>Hello, everyone.</p>
<p><strong>I have a question about, how to construct decoder for different class, and how to train this architecture.</strong></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/e/fe827f4dea5b2385f54c3ff3ba5540ab3ee74225.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/fe827f4dea5b2385f54c3ff3ba5540ab3ee74225" title="File (3)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/fe827f4dea5b2385f54c3ff3ba5540ab3ee74225_2_448x500.jpeg" alt="File (3)" data-base62-sha1="AjuRJkPBPi6JDejvQRZbozjkgG9" width="448" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/fe827f4dea5b2385f54c3ff3ba5540ab3ee74225_2_448x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/fe827f4dea5b2385f54c3ff3ba5540ab3ee74225_2_672x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/fe827f4dea5b2385f54c3ff3ba5540ab3ee74225_2_896x1000.jpeg 2x" data-dominant-color="E6E6D5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">File (3)</span><span class="informations">1345×1500 173 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here is my desired network architecture (the first of the image). The purpose of this architecture is to generate better prediction of each class, using this conditional structure.</p>
<p>Input consists of sequential data, and it get through FC layer.<br>
And then, by input class (Input can has three classes, {<strong>A</strong>, <strong>B</strong> or <strong>C</strong>}), the output of FC layer gets into a decoder. (If input class is <strong>A</strong>, then it goes through <strong>decoder A</strong>).<br>
<strong>Should I make each nn.Module for each encoder and decoder, separately?</strong></p>
<p>Currently, I am trying to construct this architecture in <strong>one single nn.Module</strong> like the second of the image. Mini-batch consists of samples with various classes.<br>
After FC layer, split batch for each class.<br>
Then, get into each decoder, then concatenate all output of decoders as final output.<br>
Using this final output, calculate loss and back-propagation.<br>
Is it a correct way to train them?</p>
<p>I am wondering that total loss is back propagated each decoder, which means <strong>A decoder</strong> can be back-propagated by <strong>B decoder prediction loss</strong>.</p>

ptrblck: <aside class="quote no-group" data-username="MinsooKim" data-post="1" data-topic="95180">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/c67d28/48.png" class="avatar"> MinsooKim:</div>
<blockquote>
<p>Currently, I am trying to construct this architecture in <strong>one single nn.Module</strong> like the second of the image. Mini-batch consists of samples with various classes.<br>
After FC layer, split batch for each class.<br>
Then, get into each decoder, then concatenate all output of decoders as final output.<br>
Using this final output, calculate loss and back-propagation.<br>
Is it a correct way to train them?</p>
</blockquote>
</aside>
<p>This approach looks correct for training.<br>
However, how would this method work for inference (or validation and testing), if no labels are available?</p>