How to avoid RNNs/LSTMS from producing none-sense tokens like SOS and PADD

pinocchio: <p>I have an LSTM I am having it return one-hot vectors. I just use the vocabulary size. But now its randomly generating SOS tokens and PADD tokens when I don’t want it (like in the middle of a sentence or something like that).</p>
<p>How are LSTM models fixed to avoid this?</p>
<p>Note that just removing them from the indices list causes the problem that now the first token I pass my LSTM cannot be SOS <img src="https://discuss.pytorch.org/images/emoji/apple/cry.png?v=12" title=":cry:" class="emoji" alt=":cry:" loading="lazy" width="20" height="20"></p>
<hr>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://www.quora.com/unanswered/How-do-I-avoid-RNNs-LSTMS-from-producing-nonesense-tokens-like-SOS-and-PADD">
  <header class="source">
      <img src="https://qsf.cf2.quoracdn.net/-4-images.favicon-new.ico-26-07ecf7cd341b6919.ico" class="site-icon" width="" height="">

      <a href="https://www.quora.com/unanswered/How-do-I-avoid-RNNs-LSTMS-from-producing-nonesense-tokens-like-SOS-and-PADD" target="_blank" rel="noopener nofollow ugc">Quora</a>
  </header>

  <article class="onebox-body">
    <img width="" height="" src="https://qph.cf2.quoracdn.net/main-custom-t-980-600x315-sqxxospfciejqkjfytssmfbdcvzbbxla.jpeg" class="thumbnail">

<h3><a href="https://www.quora.com/unanswered/How-do-I-avoid-RNNs-LSTMS-from-producing-nonesense-tokens-like-SOS-and-PADD" target="_blank" rel="noopener nofollow ugc">How do I avoid RNNs/LSTMS from producing nonesense tokens like SOS and PADD?</a></h3>

  <p>1 person wants answers to this question. Be the first to answer.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<aside class="onebox discoursetopic" data-onebox-src="https://forums.fast.ai/t/how-to-avoid-rnns-lstms-from-producing-none-sense-tokens-like-sos-and-padd/53617">
  <header class="source">
      <img src="https://forums.fast.ai/uploads/default/optimized/3X/b/3/b395de7a2ba00b82865031c97a8cadf3d80e71e5_2_32x32.png" class="site-icon" width="" height="">

      <a href="https://forums.fast.ai/t/how-to-avoid-rnns-lstms-from-producing-none-sense-tokens-like-sos-and-padd/53617" target="_blank" rel="noopener nofollow ugc" title="08:26PM - 30 August 2019">fast.ai Course Forums – 30 Aug 19</a>
  </header>

  <article class="onebox-body">
    <img src="https://forums.fast.ai/uploads/default/original/3X/2/4/24b0b4833ba73768625a1f6c7f8543dfa9c39c5e.png" class="thumbnail onebox-avatar" width="32" height="32">

<div class="title-wrapper">
  <h3><a href="https://forums.fast.ai/t/how-to-avoid-rnns-lstms-from-producing-none-sense-tokens-like-sos-and-padd/53617" target="_blank" rel="noopener nofollow ugc">How to avoid RNNs/LSTMS from producing none-sense tokens like SOS and PADD?</a></h3>
  <div class="topic-category">
      <span class="badge-wrapper bullet">
        <span class="badge-category-bg" style="background-color: #AB9364;"></span>
        <span class="badge-category clear-badge">
          <span class="category-name">Deep Learning</span>
        </span>
      </span>
  </div>
</div>

  <p>I have an LSTM I am having it return one-hot vectors. I just use the vocabulary size. But now its randomly generating SOS tokens and PADD tokens when I don’t want it (like in the middle of a sentence or something like that).  How are LSTM models...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


Mazhar_Shaikh: <p>Hi pinocchio,<br>
That sounds like your LSTM may not be trained well. You may want to run your training longer to let LSTM learn how to make better sequences.<br>
Another thing you could do if perform beam search in your sequence generation. This would enable you to explore what the sequence would look like if you feel the SOS/PAD token has popped up unnaturally.</p>

pinocchio: <p>Oh ok. So its normal to allow the network to output random SOS and PADD tokens? It just feels strange to me to leave them in the vocabulary, knowing that they are non-sense!</p>

pinocchio: <p>If it outputs SOS is the network usually just punished with the Cross Entropy loss?</p>