Change model from bert into electra - FloatTensor changes to tuple?

werdas34: <p>Hello,</p>
<p>I would like to add electra to the <a href="https://github.com/vdobrovolskii/wl-coref" rel="noopener nofollow ugc">wl-coref model</a>.<br>
In doing so, I received a type change in the variable and thus the whole thing does not work.</p>
<p>If the model is bert, then the variable “out” has the type torch.FloatTensor and the dtype torch.float32.<br>
With electra “out” has the type tuple.<br>
The error message that appears: “TypeError: only integer tensors of a single element can be converted to an index”.<br>
If I try to change the type with “out = torch.tensor(out, dtype=torch.float32, device=self.config.device)” I only get the error message “ValueError: only one element tensors can be converted to Python scalars”.</p>
<p><a href="https://github.com/vdobrovolskii/wl-coref/blob/master/coref/coref_model.py#L332" rel="noopener nofollow ugc">_bertify Method</a></p>
<pre><code class="lang-auto">def _bertify(self, doc: Doc) -&gt; torch.Tensor:
    subwords_batches = bert.get_subwords_batches(doc, self.config,
                                                    self.tokenizer)

    special_tokens = np.array([self.tokenizer.cls_token_id,
                                self.tokenizer.sep_token_id,
                                self.tokenizer.pad_token_id])
    subword_mask = ~(np.isin(subwords_batches, special_tokens))

    subwords_batches_tensor = torch.tensor(subwords_batches,
                                            device=self.config.device,
                                            dtype=torch.long)
    subword_mask_tensor = torch.tensor(subword_mask,
                                        device=self.config.device)

    # Obtain bert output for selected batches only
    attention_mask = (subwords_batches != self.tokenizer.pad_token_id)
    out = ""
    if "bert" in self.config.bert_model:
        out, _ = self.bert(
            subwords_batches_tensor,
            attention_mask=torch.tensor(
                attention_mask, device=self.config.device))
        del _
    elif "electra" in self.config.bert_model:
        out = self.bert(
            subwords_batches_tensor,
            attention_mask=torch.tensor(
                attention_mask, device=self.config.device))
        out = torch.tensor(out, dtype=torch.float32, device=self.config.device)
    
    # [n_subwords, bert_emb]
    return out[subword_mask_tensor]
</code></pre>
<p>How do I fix this?</p>
<p>Thanks</p>

ptrblck: <aside class="quote no-group" data-username="werdas34" data-post="1" data-topic="161119">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/c37758/48.png" class="avatar"> werdas34:</div>
<blockquote>
<p>In doing so, I received a type change in the variable and thus the whole thing does not work.</p>
</blockquote>
</aside>
<p>I’m not sure if the <code>type</code> really changes.<br>
Assuming the <code>"bert"</code> config is the default one while <code>"electra"</code> is the new one, note that the output of <code>self.bert</code> returns a tuple while only the first element is used:</p>
<pre data-code-wrap="python"><code class="lang-python">    if "bert" in self.config.bert_model:
        out, _ = self.bert(
            subwords_batches_tensor,
            attention_mask=torch.tensor(
                attention_mask, device=self.config.device))
        del _
</code></pre>
<p>Maybe you want to use the same approach in your <code>"electra"</code> branch.</p>

werdas34: <p>Thank you for your reply. Through this I have found the solution.</p>
<p>bert returns a tuple with two values (out, _) and electra returns a tuple with one value (out).<br>
So with electra the tuple is not resolved to the variables.</p>
<p>Solution: Adress the first value of the tuple. out[0]</p>
<pre><code class="lang-auto">elif "electra" in self.config.bert_model:
    out = self.bert(
        subwords_batches_tensor,
        attention_mask=torch.tensor(
            attention_mask, device=self.config.device))[0]
</code></pre>