Need help with recreating paper

username: hadaev8
cooked: <p><aside class="onebox pdf">
  <header class="source">
      <a href="https://arxiv.org/pdf/1906.10859.pdf" target="_blank" rel="nofollow noopener">arxiv.org</a>
  </header>
  <article class="onebox-body">
    <a href="https://arxiv.org/pdf/1906.10859.pdf" target="_blank" rel="nofollow noopener"><span class="pdf-onebox-logo"></span></a>
<h3><a href="https://arxiv.org/pdf/1906.10859.pdf" target="_blank" rel="nofollow noopener">1906.10859.pdf</a></h3>

<p class="filesize">328.58 KB</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
In this paper they modify tacotron2 gst module to train it in semi supervised way with small amount of labeled data.<br>
Vanilla gst from mozilla<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/mozilla/TTS/blob/97c7a8ff139455b243a8d479c90f346a069863fa/layers/gst_layers.py#L119" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/mozilla/TTS/blob/97c7a8ff139455b243a8d479c90f346a069863fa/layers/gst_layers.py#L119" target="_blank" rel="nofollow noopener">mozilla/TTS/blob/97c7a8ff139455b243a8d479c90f346a069863fa/layers/gst_layers.py#L119</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="109" style="counter-reset: li-counter 108 ;">
<li>        # prosody_encoding: 3D tensor [batch_size, 1, encoding_size==128]</li>
<li>        tokens = torch.tanh(self.style_tokens) \</li>
<li>            .unsqueeze(0) \</li>
<li>            .expand(batch_size, -1, -1)</li>
<li>        # tokens: 3D tensor [batch_size, num tokens, token embedding size]</li>
<li>        style_embed = self.attention(prosody_encoding, tokens)</li>
<li>
</li>
<li>        return style_embed</li>
<li>
</li>
<li>
</li>
<li class="selected">class MultiHeadAttention(nn.Module):</li>
<li>    '''</li>
<li>    input:</li>
<li>        query --- [N, T_q, query_dim]</li>
<li>        key --- [N, T_k, key_dim]</li>
<li>    output:</li>
<li>        out --- [N, T_q, num_units]</li>
<li>    '''</li>
<li>
</li>
<li>    def __init__(self, query_dim, key_dim, num_units, num_heads):</li>
<li>
</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
So, they make attention one head and use attention weights as outputs for target class.</p>
<p>This is my code:</p>
<pre><code>class MultiHeadAttention(nn.Module):
	def __init__(self, query_dim, key_dim, num_units):
		super().__init__()
		self.num_units = num_units
		self.key_dim = key_dim
		self.W_query = LinearNorm(query_dim, num_units, bias=False)
		self.W_key = LinearNorm(key_dim, num_units, bias=False)
		self.W_value = LinearNorm(key_dim, num_units, bias=False)

	def forward(self, query, key):
		querys = self.W_query(query)  # [N, T_q, num_units]
		keys = self.W_key(key)  # [N, T_k, num_units]
		values = self.W_value(key)

		# score = softmax(QK^T / (d_k ** 0.5))
		scores = torch.matmul(querys, keys.transpose(1, 2))
		scores = scores / (self.key_dim ** 0.5)
		scores_before_softmax = scores
		scores = F.softmax(scores, dim=-1)

		out = torch.matmul(scores, values)  # [h, N, T_q, num_units/h]

		return out, scores_before_softmax.squeeze(1)
</code></pre>
<p>But i am not sure if i get everything right.</p>
<p>why this line<br>
scores = scores / (self.key_dim ** 0.5<br>
?</p>
<p>Another question, how should i handle attention weights, is it ok to return weights before softmax and apply cross entropy loss on it?</p>
<p>Do ignore index in cross entropy work as i think: set -100 to all unlabeled data?</p>
<p>Cross entropy loss struggles around 2 and do not decrease.</p>