Tensorflow get_variable into PyTorch

Ahmed_Abdelaziz: <p>Hi all,</p>
<p>I am trying to convert this tensorflow code into pytorch. For example, I converted the below tensorflow code</p>
<pre><code class="lang-auto">tf.get_variable("char_embeddings", [len(data.char_dict), data.char_embedding_size]),  char_index)  # [num_sentences, max_sentence_length, max_word_length, emb]

</code></pre>
<p>into</p>
<pre><code class="lang-auto">class CharEmbeddings(nn.Module):
    def __init__(self, config, data):
          ....
          self.embeddings = nn.init.xavier_uniform_(torch.empty(len(data.char_dict), data.char_embedding_size))
   

    def forward(self, char_index):
        # [num_sentences, max_sentence_length, max_word_length, emb]
        char_emb = self.embeddings[char_index]       
</code></pre>
<p>From what I read, if no initializer is passed to the <code>get_variable</code> <a href="https://www.tensorflow.org/api_docs/python/tf/compat/v1/get_variable" rel="nofollow noopener">here</a>, the <code>glorot_uniform_initializer </code> will be used which is I think it equivalent to <code>xavier_uniform_</code></p>
<p>Two questions here:</p>
<ol>
<li>Is that conversion valid?</li>
<li>Should I expect the original embeddings <code>self.embeddings</code> to backpropagate and update its values? Is that the expected behavior from the tensorflow version as well? Should I add <code>requires_grad</code> to the embeddings tensor?</li>
</ol>

ptrblck: <ol>
<li>
<p>Probably yes, but you should compare the default arguments to both methods, as each framework might use other defaults.</p>
</li>
<li>
<p><code>self.embeddings</code> is not created as an <code>nn.Parameter</code>, so Autograd won’t calculate the gradients for this tensor. You could use:</p>
</li>
</ol>
<pre><code class="lang-python">emb = torch.empty(len(...))
nn.init.xqavier_uniform_(emb)
self.embeddings = nn.Parameter(emb)
</code></pre>
<p>instead to create a parameter, which will get gradients.</p>

Ahmed_Abdelaziz: <p>Thanks for your response. I don’t get the difference here. Why a tensor with <code>requires_grad</code> will not get gradients while the Parameter will do? I am not sure if I understand <code>required_grad</code> well though. Any explanation?</p>
<p>Thanks!</p>

ptrblck: <aside class="quote no-group" data-username="Ahmed_Abdelaziz" data-post="3" data-topic="83456">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/958977/48.png" class="avatar"> Ahmed_Abdelaziz:</div>
<blockquote>
<p>Why a tensor with <code>requires_grad</code> will not get gradients while the Parameter will do?</p>
</blockquote>
</aside>
<p>Both will get gradients, but in your initial post you didn’t set <code>requires_grad=True</code> in the tensor creation, so <code>self.embeddings</code> wouldn’t get any gradients.<br>
Additionally to that, a registered <code>nn.Parameter</code> will be automatically pushed to the device, if you call <code>model.to()</code> and will also be returned in <code>model.parameters()</code>, while a tensor will not.</p>

Ahmed_Abdelaziz: <p>I got it, thank you!</p>

Ahmed_Abdelaziz: <p>Following up on this question, I see that the model I am migrating from TensorFlow uses get_variable to intiazliae network biases</p>
<p><code>hidden_bias = tf.get_variable("hidden_bias_{}".format(i), [hidden_size])</code></p>
<p>So I assumed I can use <code>torch.nn.init.xavier_uniform_</code> to init network bias in Torch too</p>
<p><code>torch.nn.init.xavier_uniform_(self.input.bias)</code> but I got that error</p>
<p><code>Fan in and fan out can not be computed for tensor with less than 2 dimensions</code></p>
<p>Any idea how to mimic the same init as TensorFlow for biases</p>

ptrblck: <p>I don’t know how TF initializes the bias, but as the error message claims, <code>xavier_uniform</code> cannot be used on parameters with less than 2 dimensions (which is the case for your <code>bias</code> parameter).</p>

Ahmed_Abdelaziz: <p>Yes true. However I did investigate the histograms for the bias from TF and it seems they pass Fan-in = Fan-out</p>

ptrblck: <p><code>fan_in</code> and <code>fan_out</code> are calculated as:</p>
<pre><code class="lang-python">def _calculate_fan_in_and_fan_out(tensor):
    dimensions = tensor.dim()
    if dimensions &lt; 2:
        raise ValueError("Fan in and fan out can not be computed for tensor with fewer than 2 dimensions")

    num_input_fmaps = tensor.size(1)
    num_output_fmaps = tensor.size(0)
    receptive_field_size = 1
    if tensor.dim() &gt; 2:
        receptive_field_size = tensor[0][0].numel()
    fan_in = num_input_fmaps * receptive_field_size
    fan_out = num_output_fmaps * receptive_field_size

    return fan_in, fan_out
</code></pre>
<p>Would you then set both values to <code>bias.size(0)</code>?</p>
<p>If that’s the case, you could manually apply it to the <code>xavier_uniform</code> method, which is defined as:</p>
<pre><code class="lang-python">    fan_in, fan_out = _calculate_fan_in_and_fan_out(tensor)
    std = gain * math.sqrt(2.0 / float(fan_in + fan_out))
    a = math.sqrt(3.0) * std  # Calculate uniform bounds from standard deviation
</code></pre>
<p>and replace <code>_calculate_fan_in_and_fan_out</code> with <code>tensor.size(0)</code> for both values.</p>