Attention for sequence classification using a LSTM

NLPsimon: <p>Hello,</p>
<p>I am using a LSTM with word2vec features to classify sentences. In order to improve performance, I’d like to try the attention mechanism. However, I can only find resources on how to implement attention for sequence-to-sequence models and not for sequence-to-fixed-output models.</p>
<p>Thus, I have a few questions:</p>
<ol>
<li>Is it even possible / helpful to use attention for simple classifications?</li>
<li>Is there a small working example on how to combine a simple LSTM with attention? I could not find something helpful. All the code I found is very complicated, uncommented and also for seq2seq.</li>
</ol>
<p>Best,<br>
Simon</p>

wizardk: <p>Sure, you can use attention mechanism for the seq-2-one.</p>
<p>You can just imagine the seq-2-one is a special case in seq-2-seq. Attention mechanism just adjust the weights to the input features of decoder by the features, last output and last hidden of RNN (not necessary if decoder is not a RNN). This mechanism itself even don’t know you are doing seq-2-one or seq-2-seq task. It’s all up to you.</p>
<p>Did you see these examples? You can see them as the introductory tutorial.<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://assets-cdn.github.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/AuCson/PyTorch-Batch-Attention-Seq2seq" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/17193366?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/AuCson/PyTorch-Batch-Attention-Seq2seq" target="_blank" rel="nofollow noopener">AuCson/PyTorch-Batch-Attention-Seq2seq</a></h3>

<p>PyTorch implementation of batched bi-RNN encoder and attention-decoder. - AuCson/PyTorch-Batch-Attention-Seq2seq</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://assets-cdn.github.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/spro/practical-pytorch/tree/master/seq2seq-translation" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/211613?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/spro/practical-pytorch/tree/master/seq2seq-translation" target="_blank" rel="nofollow noopener">spro/practical-pytorch</a></h3>

<p>PyTorch tutorials demonstrating modern techniques with readable code - spro/practical-pytorch</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://assets-cdn.github.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/keon/seq2seq" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/10793962?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/keon/seq2seq" target="_blank" rel="nofollow noopener">keon/seq2seq</a></h3>

<p>Minimal Seq2Seq model with Attention for Neural Machine Translation in PyTorch - keon/seq2seq</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

Milos_Milunovic: <p>Do you have any instructions on how to change any of these implementations for seq2one?</p>

vdw: <p>Here you go: <a href="https://github.com/chrisvdweth/ml-toolkit/blob/master/pytorch/models/text/classifier/rnn.py" rel="noopener nofollow ugc">https://github.com/chrisvdweth/ml-toolkit/blob/master/pytorch/models/text/classifier/rnn.py</a></p>
<p>I hope that helps. As the others suggested, it’s just the Seq2Seq example simplified.</p>