BertAbs decoder_input_ids
Author: ohmeow
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

What should the `decoder_input_ids` look like if we are fine-tuning the model on our own dataset?

I tried `[unused0] [unused2] summary_sent_toks [unused2] summary_sent_toks2 [unused1]` (looking at the paper) ... but I get shape errors because of line 150 in `modeling_bertabs.py`:

```
decoder_input_ids[:, :-1], encoder_hidden_states, dec_state
```

The `decoder_input_ids` shape I'm passing in in `(8, 512)` ... but the code above chops off the last column.


Comments:
Could you please post the full stack trace as well as the part of the code you use for fine-tuning?

See here:  https://gist.github.com/ohmeow/f2cc6ea0a9d0e4a5fa227942edcfa723

I think it has something to do with how I'm preparing the target tokens but
I'm not sure what the appropriate fix is.  Looked at the BertSum source
code on github but it was confusing.  Either way, the shape of the
decoder_ids is (batch size, max_seq_len) ... but the model chops off the
last column before passing the ids off to the decoder.  My gut feeling is
that this is to account for the need to shift the ids right by 1 for the
gold labels but not sure ... and that means the input should be
(batch_size, max_seq_len+1).

Any thoughts on what I should do or what I'm missing?

Thanks

On Thu, Dec 12, 2019 at 3:15 AM Rémi Louf <notifications@github.com> wrote:

> Could you please post the full stack trace as well as the part of the code
> you use for fine-tuning?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/2153?email_source=notifications&email_token=AAADNMBPVGGALZOJ3SCV543QYIMN3A5CNFSM4JZZWIK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGWKWLY#issuecomment-564964143>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAADNMCRVCPLMOPCLI2L4S3QYIMN3ANCNFSM4JZZWIKQ>
> .
>


I think I may have solved the `decoder_input_ids` issue with the fix to my code:

```
def fit_to_block_size(sequence, block_size, pad_token_id, sep_token_id, is_summary:bool=False):
    """ Adapt the source and target sequences' lengths to the block size.
    If the sequence is shorter than the block size we pad it with -1 ids
    which correspond to padding tokens.
    """
    
    if len(sequence) > block_size:
        if (is_summary):
            sequence = sequence[:block_size]+ [symbols['EOS']]
        else:
            # ensure inclusion of whole sentences if possible
            sent_sep_idxs = [ idx for idx, t in enumerate(sequence)  if t == sep_token_id and idx < block_size ]
        
            last_sent_sep_idx = min(max(sent_sep_idxs)+1 if (len(sent_sep_idxs) > 0) else block_size, block_size)
        
            sequence = sequence[:last_sent_sep_idx]
        
    if len(sequence) < block_size:
        sequence.extend([pad_token_id] * (block_size - len(sequence)))
        
        if (is_summary): 
            sequence += [pad_token_id]
        
    return sequence
```
However, I'm now running into an error when the "context" attention is calculated in the `TransformerDecoderLayer` ...

```
~/development/_training/ml/nlp-playground/tritonlyticsai/text/modeling_bertabs.py in forward(self, key, value, query, mask, layer_cache, type, predefined_graph_1)
    601 
    602         if mask is not None:
--> 603             mask = mask.unsqueeze(1).expand_as(scores)
    604             scores = scores.masked_fill(mask, -1e18)
    605 

RuntimeError: The expanded size of the tensor (1) must match the existing size (512) at non-singleton dimension 3.  Target sizes: [512, 8, 8, 1].  Tensor sizes: [8, 1, 512, 512]
```
The passed in mask is built by the model code based on the dimensions of the source and target input ids ... which look right to me.

@ohmeow Have you been able to fine-tune the BertAbs on your dataset? I would appreciate if your can share you experience.

This is still a work in progress  ... but the below should help you get
started on fine-tuning the pretrained model.

Look here:  https://gist.github.com/ohmeow/7aa294e2959c1315fe7dfdf8091f2d87

You'll notice that I also copied a few of the HF .py files into my own
package (ohmeow.text).  I did this two be able to step through the code,
troubleshoot, and also because a modification has to be made to
modeling_bertabs.py.

#pdb.set_trace()
encoder_hidden_states = encoder_output #encoder_output[0] --WTG--
dec_state = self.decoder.init_decoder_state(
encoder_input_ids, encoder_hidden_states
)
decoder_outputs, _ = self.decoder(
decoder_input_ids[:, :-1], encoder_hidden_states, dec_state
)
#return decoder_outputs #--WTG--
return self.generator(decoder_outputs)


the commented out sections are what was originally there in the HF code.

On Mon, Dec 30, 2019 at 6:17 PM Ehsan Hemmati <notifications@github.com>
wrote:

> @ohmeow <https://github.com/ohmeow> Have you been able to fine-tune the
> BertAbs on your dataset? I would appreciate if your can share you
> experience.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/2153?email_source=notifications&email_token=AAADNMD6OHZO67VQURDKBJLQ3KTSRA5CNFSM4JZZWIK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH3TI6Q#issuecomment-569848954>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAADNMGJ6MSCUTEYGHNB4BTQ3KTSRANCNFSM4JZZWIKQ>
> .
>


@ohmeow Thanks for sharing this.
Just what is the HF files you mentioned?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.