Training for sentence vectors in niche domain

Question:
Hi everyone,
I have been inspired to create a semantic text search engine for a niche domain, and I am wondering how I should proceed. The basic approach will be to use a transformer model to embed potential results into vectors, use the same model to embed search queries, and then use cosine similarity to compare the query vector with the result vectors.
The main issue I see right now is that it is hard to have good embeddings for a niche domain.  From what I can gather, training a model on an NLI task (textual entailment) is best for having good sentence embeddings, but NLI is a supervised task that requires labeled data. The next closest task would be NSP, which can be done without a labeled dataset, but RoBERTa showed that NSP isn’t a good way of training a model. What I’ve noticed other people do for Covid semantic searches is to take SciBERT or BioBERT, train it more on PubMed articles or Cord-19 articles doing MLM, and then finally end with training on an NLI task. I think the NLI task was unrelated to Covid or biology because I don’t know of any in-domain NLI tasks like that.
I have seen joeddav’s blogpost and the recent ZSL pipeline work, and while ZSL is cool and has it’s purposes, it would be ineffective for comparing a search query against thousands or even just hundreds of results in real time.
I have one main question: How should I train a model to generate good sentence vectors in a niche domain?
My current plan is to take a pretrained model, fine-tune it using MLM on in-domain texts, and then do NLI training using SNLI. I am worried that it will be hard to gauge when to stop the NLI training because it seems like the longer it trains, the better it gets at producing sentence-level vectors, but the more it forgets about in-domain information.
Moreover, I’m worried that the fine-tuning using MLM won’t go great because I have tens of thousands of 2-4 sentence chunks rather than long documents.

Context:
Hi @nbroad , interesting question.
What kind of Niche domain do you consider?
Since nowadays we have several hundreds (if not thousands) of NLP datasets, is it possible to find similar datasets for pre-MLM before final-MLM by your own data ?
I don’t have direct experience on sentence similarity training, but I once trained a classifier on multi-langauges Toxic-comment domain (maybe a bit niche) where finetuning with MLM did improve the performance compared to non-MLM.


Question:
My request was posted as an issue 3.
Environment info

transformers version: 4.2.0
Platform: Linux-4.15.0-123-generic-x86_64-with-glibc2.10
Python version: 3.8.5
PyTorch version (GPU?): 1.7.1 (True)
Tensorflow version (GPU?): not installed (NA)
Using GPU in script?: Yes
Using distributed or parallel set-up in script?: No

Information
Model I am using script to initialize Longformer starting from HerBERT.
The problem arises when using:

[ ] the official example scripts: (give details below)
[x] my own modified scripts: (give details below)

The tasks I am working on is:

[ ] an official GLUE/SQUaD task: (give the name)
[x] my own task or dataset: (give details below)

To reproduce
Steps to reproduce the behavior:

Install dependencies: python3 -m pip install -r requirements.txt.
Install apex according to official documentation.
Run command CUDA_VISIBLE_DEVICES=0 python3 convert_model_to_longformer.py --finetune_dataset conllu.

We are using dataset in .jsonl format, each line contains 1 CoNLLu entry. It is converted using custom LineByLineTextDataset class to LineByLine format from current version of transformers. I’ve added this class to be able to use it in older version (v3.0.2).
Using suggested by author on allenai/longformer I’ve used transformers in version 3.0.2 and it works fine. But I would like to use recent models to convert them to Long* version and I can’t make conversion script work.
Result
As a result of running command above with transformers in version 4.2.0 I’ve got:
Traceback (most recent call last):
  File "convert_model_to_longformer.py", line 277, in <module>
    pretrain_and_evaluate(
  File "convert_model_to_longformer.py", line 165, in pretrain_and_evaluate
    eval_loss = trainer.evaluate()
  File "/server/server_1/user/miniconda3/envs/longformer_summary/lib/python3.8/site-packages/transformers/trainer.py", line 1442, in evaluate
    output = self.prediction_loop(
  File "/server/server_1/user/miniconda3/envs/longformer_summary/lib/python3.8/site-packages/transformers/trainer.py", line 1566, in prediction_loop
    loss, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
  File "/server/server_1/user/miniconda3/envs/longformer_summary/lib/python3.8/site-packages/transformers/trainer.py", line 1670, in prediction_step
    outputs = model(**inputs)
  File "/server/server_1/user/miniconda3/envs/longformer_summary/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/server/server_1/user/miniconda3/envs/longformer_summary/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 1032, in forward
    outputs = self.roberta(
  File "/server/server_1/user/miniconda3/envs/longformer_summary/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/server/server_1/user/miniconda3/envs/longformer_summary/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 798, in forward
    encoder_outputs = self.encoder(
  File "/server/server_1/user/miniconda3/envs/longformer_summary/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/server/server_1/user/miniconda3/envs/longformer_summary/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 498, in forward
    layer_outputs = layer_module(
  File "/server/server_1/user/miniconda3/envs/longformer_summary/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/server/server_1/user/miniconda3/envs/longformer_summary/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 393, in forward
    self_attention_outputs = self.attention(
  File "/server/server_1/user/miniconda3/envs/longformer_summary/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/server/server_1/user/miniconda3/envs/longformer_summary/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 321, in forward
    self_outputs = self.self(
  File "/server/server_1/user/miniconda3/envs/longformer_summary/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "convert_model_to_longformer.py", line 63, in forward
    return super().forward(hidden_states, attention_mask=attention_mask, output_attentions=output_attentions) # v4.2.0
  File "/server/server_1/user/miniconda3/envs/longformer_summary/lib/python3.8/site-packages/transformers/models/longformer/modeling_longformer.py", line 600, in forward
    diagonal_mask = self._sliding_chunks_query_key_matmul(
  File "/server/server_1/user/miniconda3/envs/longformer_summary/lib/python3.8/site-packages/transformers/models/longformer/modeling_longformer.py", line 789, in _sliding_chunks_query_key_matmul
    batch_size, seq_len, num_heads, head_dim = query.size()
ValueError: too many values to unpack (expected 4)

I’ve changed function /server/server_1/user/miniconda3/envs/longformer_summary/lib/python3.8/site-packages/transformers/models/longformer/modeling_longformer.py up to line 789:
    def forward(
        self,
        hidden_states,
        attention_mask=None,
        is_index_masked=None,
        is_index_global_attn=None,
        is_global_attn=None,
        output_attentions=False,
    ):
        """
        :class:`LongformerSelfAttention` expects `len(hidden_states)` to be multiple of `attention_window`. Padding to
        `attention_window` happens in :meth:`LongformerModel.forward` to avoid redoing the padding on each layer.

        The `attention_mask` is changed in :meth:`LongformerModel.forward` from 0, 1, 2 to:

            * -10000: no attention
            * 0: local attention
            * +10000: global attention
        """
        hidden_states = hidden_states.transpose(0, 1)

        # project hidden states
        query_vectors = self.query(hidden_states)
        key_vectors = self.key(hidden_states)
        value_vectors = self.value(hidden_states)
        print(f"query_vectors: {query_vectors.shape}")
        print(f"key_vectors: {key_vectors.shape}")
        print(f"value_vectors: {value_vectors.shape}")
        print(f"attention_mask: {attention_mask.shape}")

        seq_len, batch_size, embed_dim = hidden_states.size()
        assert (
            embed_dim == self.embed_dim
        ), f"hidden_states should have embed_dim = {self.embed_dim}, but has {embed_dim}"

        # normalize query
        query_vectors /= math.sqrt(self.head_dim)

        query_vectors = query_vectors.view(seq_len, batch_size, self.num_heads, self.head_dim).transpose(0, 1)
        key_vectors = key_vectors.view(seq_len, batch_size, self.num_heads, self.head_dim).transpose(0, 1)

        attn_scores = self._sliding_chunks_query_key_matmul(
            query_vectors, key_vectors, self.one_sided_attn_window_size
        )

        # values to pad for attention probs
        remove_from_windowed_attention_mask = (attention_mask != 0)[:, :, None, None]

        # cast to fp32/fp16 then replace 1's with -inf
        float_mask = remove_from_windowed_attention_mask.type_as(query_vectors).masked_fill(
            remove_from_windowed_attention_mask, -10000.0
        )

        print(f"attn_scores: {attn_scores.shape}")
        print(f"remove_from_windowed_attention_mask: {remove_from_windowed_attention_mask.shape}")
        print(f"float_mask: {float_mask.shape}")

        # diagonal mask with zeros everywhere and -inf inplace of padding
        diagonal_mask = self._sliding_chunks_query_key_matmul(
            float_mask.new_ones(size=float_mask.size()), float_mask, self.one_sided_attn_window_size
        )

And as a result I’ve got:
attention_mask: torch.Size([2, 1, 1, 1024])
query_vectors: torch.Size([1024, 2, 768])
key_vectors: torch.Size([1024, 2, 768])
value_vectors: torch.Size([1024, 2, 768])
attn_scores: torch.Size([2, 1024, 12, 513])
remove_from_windowed_attention_mask: torch.Size([2, 1, 1, 1, 1, 1024])
float_mask: torch.Size([2, 1, 1, 1, 1, 1024])

And after changing version to 3.0.2 and adding print statements I’ve got:
attention_mask: torch.Size([2, 1024])
query_vectors: torch.Size([1024, 2, 768])
key_vectors: torch.Size([1024, 2, 768])
value_vectors: torch.Size([1024, 2, 768])
attn_scores: torch.Size([2, 1024, 12, 513])
remove_from_windowed_attention_mask: torch.Size([2, 1024, 1, 1])
float_mask: torch.Size([2, 1024, 1, 1])

So maybe it’s problem with _sliding_chunks_query_key_matmul function?
Files:
convert_model_to_longformer.py, based on allenai/longformer/scripts/convert_model_to_long.ipynb 7:
import logging
import os
import math
import copy
import torch
import argparse
from dataclasses import dataclass, field
from transformers import RobertaForMaskedLM, XLMTokenizer, TextDataset, DataCollatorForLanguageModeling, Trainer, XLMTokenizer, PreTrainedTokenizer
from transformers import TrainingArguments, HfArgumentParser, XLMTokenizer, RobertaModel, XLMTokenizer
from transformers import LongformerSelfAttention # v4.2.0
# from transformers.modeling_longformer import LongformerSelfAttention # v3.0.2
from conllu import load_conllu_dataset, save_conllu_dataset_in_linebyline_format
from torch.utils.data.dataset import Dataset

logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO)

class LineByLineTextDataset(Dataset):
    """
    This will be superseded by a framework-agnostic approach
    soon.
    """

    def __init__(self, tokenizer: PreTrainedTokenizer, file_path: str, block_size: int):
        assert os.path.isfile(file_path)
        # Here, we do not cache the features, operating under the assumption
        # that we will soon use fast multithreaded tokenizers from the
        # `tokenizers` repo everywhere =)
        logger.info("Creating features from dataset file at %s", file_path)

        with open(file_path, encoding="utf-8") as f:
            lines = [line for line in f.read().splitlines() if (len(line) > 0 and not line.isspace())]

        batch_encoding = tokenizer(
            lines,
            add_special_tokens=True,
            truncation=True,
            padding="max_length",
            max_length=block_size,
            pad_to_multiple_of=512)
        self.examples = batch_encoding["input_ids"]

    def __len__(self):
        return len(self.examples)

    def __getitem__(self, i) -> torch.Tensor:
        return torch.tensor(self.examples[i], dtype=torch.long)

class RobertaLongSelfAttention(LongformerSelfAttention):
    def forward(
        self,
        hidden_states,
        attention_mask=None,
        head_mask=None,
        encoder_hidden_states=None,
        encoder_attention_mask=None,
        past_key_value=None,
        output_attentions=False,
    ):
        return super().forward(hidden_states, attention_mask=attention_mask, output_attentions=output_attentions)

class RobertaLongForMaskedLM(RobertaForMaskedLM):
    def __init__(self, config):
        super().__init__(config)
        for i, layer in enumerate(self.roberta.encoder.layer):
            # replace the `modeling_bert.BertSelfAttention` object with `LongformerSelfAttention`
            layer.attention.self = RobertaLongSelfAttention(config, layer_id=i)

class RobertaLongModel(RobertaModel):
    def __init__(self, config):
        super().__init__(config)
        for i, layer in enumerate(self.encoder.layer):
            # replace the `modeling_bert.BertSelfAttention` object with `LongformerSelfAttention`
            layer.attention.self = RobertaLongSelfAttention(config, layer_id=i)

def create_long_model(initialization_model, initialization_tokenizer, save_model_to, attention_window, max_pos):
    model = RobertaForMaskedLM.from_pretrained(initialization_model)
    tokenizer = XLMTokenizer.from_pretrained(initialization_tokenizer, model_max_length=max_pos)
    config = model.config

    # extend position embeddings
    tokenizer.model_max_length = max_pos
    tokenizer.init_kwargs['model_max_length'] = max_pos
    current_max_pos, embed_size = model.roberta.embeddings.position_embeddings.weight.shape
    max_pos += 2  # NOTE: RoBERTa has positions 0,1 reserved, so embedding size is max position + 2
    config.max_position_embeddings = max_pos
    assert max_pos > current_max_pos

    # allocate a larger position embedding matrix
    new_pos_embed = model.roberta.embeddings.position_embeddings.weight.new_empty(max_pos, embed_size)

    # copy position embeddings over and over to initialize the new position embeddings
    k = 2
    step = current_max_pos - 2
    while k < max_pos - 1:
        new_pos_embed[k:(k + step)] = model.roberta.embeddings.position_embeddings.weight[2:]
        k += step
    model.roberta.embeddings.position_embeddings.weight.data = new_pos_embed
    model.roberta.embeddings.position_ids.data = torch.tensor([i for i in range(max_pos)]).reshape(1, max_pos) # v4.2.0
    # model.roberta.embeddings.position_ids = torch.tensor([i for i in range(max_pos)]).reshape(1, max_pos) # v3.0.2

    # replace the `modeling_bert.BertSelfAttention` object with `LongformerSelfAttention`
    config.attention_window = [attention_window] * config.num_hidden_layers
    for i, layer in enumerate(model.roberta.encoder.layer):
        longformer_self_attn = LongformerSelfAttention(config, layer_id=i)
        longformer_self_attn.query = copy.deepcopy(layer.attention.self.query)
        longformer_self_attn.key = copy.deepcopy(layer.attention.self.key)
        longformer_self_attn.value = copy.deepcopy(layer.attention.self.value)

        longformer_self_attn.query_global = copy.deepcopy(layer.attention.self.query)
        longformer_self_attn.key_global = copy.deepcopy(layer.attention.self.key)
        longformer_self_attn.value_global = copy.deepcopy(layer.attention.self.value)

        layer.attention.self = longformer_self_attn

    logger.info(f'saving model to {save_model_to}')
    model.save_pretrained(save_model_to)
    tokenizer.save_pretrained(save_model_to)
    return model, tokenizer

def copy_proj_layers(model):
    for i, layer in enumerate(model.roberta.encoder.layer):
        layer.attention.self.query_global = copy.deepcopy(layer.attention.self.query)
        layer.attention.self.key_global = copy.deepcopy(layer.attention.self.key)
        layer.attention.self.value_global = copy.deepcopy(layer.attention.self.value)
    return model

def pretrain_and_evaluate(args, model, tokenizer, eval_only, model_path, max_size):
    val_dataset = LineByLineTextDataset(
        tokenizer=tokenizer,
        file_path=args.val_datapath,
        block_size=max_size,
    )
    if eval_only:
        train_dataset = val_dataset
    else:
        logger.info(f'Loading and tokenizing training data is usually slow: {args.train_datapath}')
        train_dataset = LineByLineTextDataset(
            tokenizer=tokenizer,
            file_path=args.train_datapath,
            block_size=max_size,
        )

    data_collator = DataCollatorForLanguageModeling(
        tokenizer=tokenizer,
        mlm=True,
        mlm_probability=0.15,
    )
    trainer = Trainer(
        model=model,
        args=args,
        data_collator=data_collator,
        train_dataset=train_dataset,
        eval_dataset=val_dataset,
        # prediction_loss_only=True,
    )

    eval_loss = trainer.evaluate()
    eval_loss = eval_loss['eval_loss']
    logger.info(f'Initial eval bpc: {eval_loss/math.log(2)}')

    if not eval_only:
        trainer = Trainer(
            model=model,
            args=args,
            data_collator=data_collator,
            train_dataset=train_dataset,
            eval_dataset=val_dataset,
            prediction_loss_only=False,
        )

        trainer.train(model_path=model_path)
        trainer.save_model()

        eval_loss = trainer.evaluate()
        eval_loss = eval_loss['eval_loss']
        logger.info(f'Eval bpc after pretraining: {eval_loss/math.log(2)}')

@dataclass
class ModelArgs:
    attention_window: int = field(default=512, metadata={"help": "Size of attention window"})
    max_pos: int = field(default=1024, metadata={"help": "Maximum position"})

def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument("--finetune_dataset", required=True, choices=["conllu"], help="Name of dataset to finetune")
    return parser.parse_args()

if __name__ == "__main__":
    parser = HfArgumentParser((TrainingArguments, ModelArgs,))
    args = parse_args()

    training_args, model_args = parser.parse_args_into_dataclasses(look_for_args_file=False, args=[
        '--output_dir', 'tmp_4.2.0',
        '--warmup_steps', '500',
        '--learning_rate', '0.00003',
        '--weight_decay', '0.01',
        '--adam_epsilon', '1e-6',
        '--max_steps', '3000',
        '--logging_steps', '500',
        '--save_steps', '500',
        '--max_grad_norm', '5.0',
        '--per_device_eval_batch_size', '2',
        '--per_device_train_batch_size', '2',
        '--gradient_accumulation_steps', '4',
        # '--evaluate_during_training',
        '--do_train',
        '--do_eval',
        '--fp16',
        '--fp16_opt_level', 'O2',
    ])

    if args.finetune_dataset == "conllu":
        saved_dataset = '/server/server_1/user/longformer_summary/conllu/'
        if not os.path.exists(saved_dataset):
            os.makedirs(saved_dataset)
            dataset = load_conllu_dataset('/server/server_1/user/conllu_dataset/')
            save_conllu_dataset_in_linebyline_format(dataset, saved_dataset)

    training_args.val_datapath = os.path.join(saved_dataset, 'validation.txt')
    training_args.train_datapath = os.path.join(saved_dataset, 'train.txt')

    initialization_model = 'allegro/herbert-klej-cased-v1'
    initialization_tokenizer = 'allegro/herbert-klej-cased-tokenizer-v1'

    roberta_base = RobertaForMaskedLM.from_pretrained(initialization_model)
    roberta_base_tokenizer = XLMTokenizer.from_pretrained(initialization_tokenizer, model_max_length=512)

    model_path = f'{training_args.output_dir}/{initialization_model}-{model_args.max_pos}'
    if not os.path.exists(model_path):
        os.makedirs(model_path)

    logger.info(f'Converting roberta-base into {initialization_model}-{model_args.max_pos}')
    model, tokenizer = create_long_model(
        initialization_model=initialization_model,
        initialization_tokenizer=initialization_tokenizer,
        save_model_to=model_path,
        attention_window=model_args.attention_window,
        max_pos=model_args.max_pos,
    )

    logger.info(f'Loading the model from {model_path}')
    tokenizer = XLMTokenizer.from_pretrained(model_path)
    model = RobertaLongForMaskedLM.from_pretrained(model_path)

    logger.info(f'Pretraining {initialization_model}-{model_args.max_pos} ... ')

    pretrain_and_evaluate(
        training_args,
        model,
        tokenizer,
        eval_only=False,
        model_path=training_args.output_dir,
        max_size=model_args.max_pos,
    )

    logger.info(f'Copying local projection layers into global projection layers... ')
    model = copy_proj_layers(model)
    logger.info(f'Saving model to {model_path}')
    model.save_pretrained(model_path)

    logger.info(f'Loading the model from {model_path}')
    tokenizer = XLMTokenizer.from_pretrained(model_path)
    model = RobertaLongModel.from_pretrained(model_path)

conllu.py
import re
import glob
import torch
from torch.utils.data import Dataset
import time
import os
import json
from xml.etree.ElementTree import ParseError
import xml.etree.ElementTree as ET
from typing import List, Dict
from sklearn.model_selection import train_test_split

def load_conllu_jsonl(
    path: str,
) -> List[Dict[str, str]]:
    dataset: List[Dict[str, str]] = list()
    with open(path, 'r') as f:
        for jsonl in f.readlines():
            json_file = json.loads(jsonl)
            conllu = json_file['conllu'].split('\n')
            doc_text: str = ""
            utterance: Dict[str, str] = dict()
            for line in conllu:
                try:
                    if line[0].isdigit():
                        if utterance:
                            masked_text = utterance["text"]
                            doc_text = f"{doc_text} {masked_text}.".strip()
                            utterance = dict()
                    elif line[0] == '#':
                        text = line[1:].strip()
                        key = text.split('=')[0].strip()
                        value = text.split('=')[1].strip()
                        utterance[key] = value
                except IndexError:
                    pass

            dataset.append({"text": doc_text})

    return dataset

def load_conllu_dataset(
    path: str,
    train_test_val_ratio: float = 0.1,
) -> Dict[str, List[Dict[str, str]]]:
    dataset: Dict[str, List[Dict[str, str]]] = dict()
    data_dict: Dict[str, List[str]] = dict()

    filepath_list = glob.glob(os.path.join(path, '*.jsonl'))

    train = filepath_list[:int(len(filepath_list)*0.8)]
    test = filepath_list[int(len(filepath_list)*0.8):int(len(filepath_list)*0.9)]
    val = filepath_list[int(len(filepath_list)*0.9):]

    data_dict["test"] = test
    data_dict["train"] = train
    data_dict["validation"] = val

    for key, value in data_dict.items():
        dataset_list: List[Dict[str, str]] = list()
        for filepath in value:
            data = load_conllu_jsonl(path=filepath)
            if data:
                dataset_list.extend(data)

        dataset[key] = dataset_list

    return dataset

def save_conllu_dataset_in_linebyline_format(
    dataset: Dict[str, List[Dict[str, str]]],
    save_dir: str,
) -> None:
    for key, value in dataset.items():
        with open(os.path.join(save_dir, f'{key}.txt'), 'w') as f:
            for line in value:
                # print(line["full"])
                f.write(f'{line["text"]}\n')

requirements.txt:
apex @ file:///server/server_1/user/apex
certifi==2020.12.5
chardet==4.0.0
click==7.1.2
datasets==1.2.0
dill==0.3.3
filelock==3.0.12
idna==2.10
joblib==1.0.0
multiprocess==0.70.11.1
numpy==1.19.4
packaging==20.8
pandas==1.2.0
pyarrow==2.0.0
pyparsing==2.4.7
python-dateutil==2.8.1
pytz==2020.5
regex==2020.11.13
requests==2.25.1
sacremoses==0.0.43
sentencepiece==0.1.94
six==1.15.0
tokenizers==0.8.1rc1
torch==1.7.1
tqdm==4.49.0
transformers==3.0.2
typing-extensions==3.7.4.3
urllib3==1.26.2
xxhash==2.0.0

Expected behavior
Model should be converted, saved and loaded. After that it should be properly fine-tuned and saved on disk.
Comparing codebase of version 3.0.2 and 4.2.0 I have noticed that forward function differs. I have added deleted lines right at the beginning of the function:
    def forward(
        self,
        hidden_states,
        attention_mask=None,
        is_index_masked=None,
        is_index_global_attn=None,
        is_global_attn=None,
        output_attentions=False,
    ):
        """
        :class:`LongformerSelfAttention` expects `len(hidden_states)` to be multiple of `attention_window`. Padding to
        `attention_window` happens in :meth:`LongformerModel.forward` to avoid redoing the padding on each layer.

        The `attention_mask` is changed in :meth:`LongformerModel.forward` from 0, 1, 2 to:

            * -10000: no attention
            * 0: local attention
            * +10000: global attention
        """
        attention_mask = attention_mask.squeeze(dim=2).squeeze(dim=1)

        # is index masked or global attention
        is_index_masked = attention_mask < 0
        is_index_global_attn = attention_mask > 0
        is_global_attn = any(is_index_global_attn.flatten())

and now model seems to be working, but returns:
{'eval_loss': nan, 'eval_runtime': 20.6319, 'eval_samples_per_second': 1.939}

Below You can find results of consecutive steps in forward function. Can You see something wrong here?
diagonal_mask: tensor([[[[-inf, -inf, -inf,  ..., 0., 0., 0.]],

         [[-inf, -inf, -inf,  ..., 0., 0., 0.]],

         [[-inf, -inf, -inf,  ..., 0., 0., 0.]],

         ...,

         [[0., 0., 0.,  ..., -inf, -inf, -inf]],

         [[0., 0., 0.,  ..., -inf, -inf, -inf]],

         [[0., 0., 0.,  ..., -inf, -inf, -inf]]],

        [[[-inf, -inf, -inf,  ..., 0., 0., 0.]],

         [[-inf, -inf, -inf,  ..., 0., 0., 0.]],

         [[-inf, -inf, -inf,  ..., 0., 0., 0.]],

         ...,

         [[0., 0., 0.,  ..., -inf, -inf, -inf]],

         [[0., 0., 0.,  ..., -inf, -inf, -inf]],

         [[0., 0., 0.,  ..., -inf, -inf, -inf]]]], device='cuda:0',
       dtype=torch.float16)
attn_scores: tensor([[[[   -inf,    -inf,    -inf,  ...,  0.5771,  0.2065, -1.0449],
          [   -inf,    -inf,    -inf,  ..., -1.3174, -1.5547, -0.6240],
          [   -inf,    -inf,    -inf,  ..., -1.3691, -1.3555, -0.3799],
          ...,
          [   -inf,    -inf,    -inf,  ...,  1.7402,  1.6152,  0.8242],
          [   -inf,    -inf,    -inf,  ...,  0.5122,  1.0342,  0.2091],
          [   -inf,    -inf,    -inf,  ...,  1.7568, -0.1534,  0.7505]],

         [[   -inf,    -inf,    -inf,  ..., -0.8066, -1.7480, -2.5527],
          [   -inf,    -inf,    -inf,  ..., -3.3652,  0.1046, -0.5811],
          [   -inf,    -inf,    -inf,  ..., -0.0958, -1.0957, -0.2377],
          ...,
          [   -inf,    -inf,    -inf,  ..., -0.4148, -0.9497, -0.1229],
          [   -inf,    -inf,    -inf,  ..., -1.9443, -1.3467, -1.5342],
          [   -inf,    -inf,    -inf,  ...,  0.1263, -0.4407,  0.1486]],

         [[   -inf,    -inf,    -inf,  ..., -0.9077, -0.1603, -0.5762],
          [   -inf,    -inf,    -inf,  ..., -0.2454,  0.1932, -0.5034],
          [   -inf,    -inf,    -inf,  ..., -1.4375, -1.2793, -1.0488],
          ...,
          [   -inf,    -inf,    -inf,  ..., -0.3452,  0.1405,  1.3643],
          [   -inf,    -inf,    -inf,  ..., -0.2168, -1.0000, -0.9956],
          [   -inf,    -inf,    -inf,  ..., -1.7451,  0.1410, -0.6221]],

         ...,

         [[-1.3965,  0.7798,  0.4707,  ...,    -inf,    -inf,    -inf],
          [ 0.6260, -0.4146,  0.9180,  ...,    -inf,    -inf,    -inf],
          [ 0.4807, -1.0742,  1.2803,  ...,    -inf,    -inf,    -inf],
          ...,
          [ 0.0909,  0.8022, -0.4170,  ...,    -inf,    -inf,    -inf],
          [-2.6035, -1.2988,  0.5586,  ...,    -inf,    -inf,    -inf],
          [-0.6953, -0.8232,  0.0436,  ...,    -inf,    -inf,    -inf]],

         [[ 1.0889, -0.2776, -0.0632,  ...,    -inf,    -inf,    -inf],
          [-0.4128,  0.4834, -0.3848,  ...,    -inf,    -inf,    -inf],
          [-0.8794,  0.9150, -1.5107,  ...,    -inf,    -inf,    -inf],
          ...,
          [ 0.8867, -0.4731,  0.3389,  ...,    -inf,    -inf,    -inf],
          [-0.1365,  0.4905, -2.0000,  ...,    -inf,    -inf,    -inf],
          [-0.0205, -0.5464, -0.6851,  ...,    -inf,    -inf,    -inf]],

         [[    nan,     nan,     nan,  ...,    -inf,    -inf,    -inf],
          [    nan,     nan,     nan,  ...,    -inf,    -inf,    -inf],
          [    nan,     nan,     nan,  ...,    -inf,    -inf,    -inf],
          ...,
          [    nan,     nan,     nan,  ...,    -inf,    -inf,    -inf],
          [    nan,     nan,     nan,  ...,    -inf,    -inf,    -inf],
          [    nan,     nan,     nan,  ...,    -inf,    -inf,    -inf]]],

        [[[   -inf,    -inf,    -inf,  ..., -4.0469, -2.6270, -5.4805],
          [   -inf,    -inf,    -inf,  ..., -0.9312, -0.6743, -1.9688],
          [   -inf,    -inf,    -inf,  ..., -0.0593, -0.9507, -0.6392],
          ...,
          [   -inf,    -inf,    -inf,  ...,  0.3105,  2.3926,  1.0664],
          [   -inf,    -inf,    -inf,  ..., -0.0166,  2.2754,  1.0449],
          [   -inf,    -inf,    -inf,  ..., -0.4224,  1.7686, -0.2603]],

         [[   -inf,    -inf,    -inf,  ..., -0.5088, -1.2666, -0.4363],
          [   -inf,    -inf,    -inf,  ..., -0.3823, -1.7998, -0.4504],
          [   -inf,    -inf,    -inf,  ..., -0.1525,  0.1614, -0.0267],
          ...,
          [   -inf,    -inf,    -inf,  ...,  0.0225, -0.5737,  0.2318],
          [   -inf,    -inf,    -inf,  ...,  0.7139,  0.6099,  0.3767],
          [   -inf,    -inf,    -inf,  ...,  0.2008, -0.6714,  0.5869]],

         [[   -inf,    -inf,    -inf,  ..., -0.9302, -1.5303, -2.7637],
          [   -inf,    -inf,    -inf,  ..., -0.1124, -0.5850,  0.0818],
          [   -inf,    -inf,    -inf,  ..., -1.5176, -1.7822, -0.9111],
          ...,
          [   -inf,    -inf,    -inf,  ..., -0.3618,  0.3486,  0.4368],
          [   -inf,    -inf,    -inf,  ..., -0.4158, -1.1660, -0.9106],
          [   -inf,    -inf,    -inf,  ..., -0.4636, -0.7012, -0.9570]],

         ...,

         [[-1.0137, -1.2324, -0.2091,  ...,    -inf,    -inf,    -inf],
          [ 0.0793,  0.1862, -0.6162,  ...,    -inf,    -inf,    -inf],
          [ 0.2406,  0.1237, -1.0420,  ...,    -inf,    -inf,    -inf],
          ...,
          [ 0.5308,  0.3862,  0.9731,  ...,    -inf,    -inf,    -inf],
          [-0.5752, -0.8174,  0.4766,  ...,    -inf,    -inf,    -inf],
          [-0.4299, -0.7031, -0.6240,  ...,    -inf,    -inf,    -inf]],

         [[-2.9512, -1.0410,  0.9194,  ...,    -inf,    -inf,    -inf],
          [-0.0306, -0.8579,  0.1930,  ...,    -inf,    -inf,    -inf],
          [ 0.2927, -1.4600, -1.6787,  ...,    -inf,    -inf,    -inf],
          ...,
          [ 0.6128, -0.8921,  1.2861,  ...,    -inf,    -inf,    -inf],
          [-0.7778, -0.8564,  2.3457,  ...,    -inf,    -inf,    -inf],
          [-0.8877, -1.4834,  0.7783,  ...,    -inf,    -inf,    -inf]],

         [[    nan,     nan,     nan,  ...,    -inf,    -inf,    -inf],
          [    nan,     nan,     nan,  ...,    -inf,    -inf,    -inf],
          [    nan,     nan,     nan,  ...,    -inf,    -inf,    -inf],
          ...,
          [    nan,     nan,     nan,  ...,    -inf,    -inf,    -inf],
          [    nan,     nan,     nan,  ...,    -inf,    -inf,    -inf],
          [    nan,     nan,     nan,  ...,    -inf,    -inf,    -inf]]]],
       device='cuda:0', dtype=torch.float16)

Context:
Hey @adamwawrzynski
The implem of Lonformer is in 4.2.0 is different from 3.0.2, you might need to modify the convert script for the new version.


Generating sentence embeddings from pretrained transformers model

Question:
Hi, I have a pretrained BERT based model hosted on huggingface.

      huggingface.co

microsoft/SportsBERT · Hugging Face 2

How do I generate sentence vectors using this model? I have explored sentence bert but it doesn’t allow you to use custom trained models. I have also seen Bert as a client. It works but for my current scenario, I was wondering if there’s something which could be done without running a server for converting to vectors.

Context:
I think the best method is to go with sentence-bert. Indeed, you can use your own model, just try reproducing what they do in the paper 11.
You add a pooling layer at the output of your model, from the paper:

We experiment with three pooling strategies: Using the output of the CLS-token,computing the mean of all output vectors (MEAN-strategy),  and computing a max-over-time of the output vectors (MAX-strategy). The default configuration is MEAN.

Finally, you might want to fine-tune your model for comparison, too:

In order to fine-tune BERT / RoBERTa (your model), we create siamese and triplet networks  (Schroff  et  al.,2015) to update the weights such that the produced sentence embeddings are semantically meaningful and can be compared with cosine-similarity.

Hope this help!


Converting Word-level labels to WordPiece-level for Token Classification

Question:
Hi all,
I am building a BertForTokenClassification model but I am having trouble figuring out how to format my dataset. I have already labeled my dataset with span labeling. So for example:
sequence = “Hugging Face Inc. is a company based in New York City. Its headquarters are in DUMBO, therefore very close to the Manhattan Bridge.”
In my dataset, I would have this labeled by hand as:
[(Hugging, B-org), (Face, I-org), (Inc., L-org), (is, O), (a, O), (company, O), … (Bridge, L-org)]
However, when I pass this through my BertTokenizer, I get the following tokens:
[[CLS], Hu, ##gging, Face, Inc., ., is, a, company, …, Bridge, [SEP]]
My question is, how do I handle the Hu, ##gging <-> Hugging label mismatch issue? I have Hugging labeled as B-org, and if I zip these tokens with my labels my labels will be offset by one:
[(Hu, B-org), (##gging, I-org), (Face, L-org), (Inc., O), (is, O), (a, O), (company, O), … (Bridge, OUT_OF_LABELS)]
Has anybody been able to handle this problem before?

Context:
Hi @altozachmo,
you can “extend” the labels list, adding as many labels as the number of token splits.
So, for example:
sequence = “Hugging Face"
labels = [(Hugging, B-org),(Face, I-org)]
tokenized_sentence = [[CLS], Hu, ##gging, Face, [SEP]]
tokenized_labels =  [(Hu, B-org),(##gging, B-org),(Face, I-org)]
(the tokenized_labels should include also the labels for the CLS and SEP tokens but I omitted them)
To do this, you can check this tutorial 45 and look for the “tokenize_and_preserve_labels” function


how to convert text to word embeddings using bert’s pretrained model ‘faster’?

Question:
I’m trying to get word embeddings for clinical data using microsoft/pubmedbert. I have 3.6 million text rows. Converting texts to vectors for 10k rows takes around 30 minutes. So for 3.6 million rows, it would take around - 180 hours(8days approx).

Is there any method where I can speed up the process?

My code -
from transformers import AutoTokenizer
from transformers import pipeline
model_name = "microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext"
tokenizer = AutoTokenizer.from_pretrained(model_name)
classifier = pipeline('feature-extraction',model=model_name, tokenizer=tokenizer)

def lambda_func(row):
    tokens = tokenizer(row['notetext'])
    if len(tokens['input_ids'])>512:
        tokens = re.split(r'\b', row['notetext'])
        tokens= [t for t in tokens if len(t) > 0 ]
        row['notetext'] = ''.join(tokens[:512])
    row['vectors'] = classifier(row['notetext'])[0][0]
    return row

def process(progress_notes):
    progress_notes = progress_notes.apply(lambda_func, axis=1)
    return progress_notes

progress_notes = process(progress_notes)
vectors_breadth = 768
vectors_length = len(progress_notes)
vectors_2d = np.reshape(progress_notes['vectors'].to_list(), (vectors_length, vectors_breadth))
vectors_df = pd.DataFrame(vectors_2d)

My progress_notes dataframe looks like -
progress_notes = pd.DataFrame({'id':[1,2,3],'progressnotetype':['Nursing Note', 'Nursing Note', 'Administration Note'], 'notetext': ['Patient\'s skin is grossly intact with exception of skin tear to r inner elbow and r lateral lower leg','Patient with history of Afib with RVR. Patient is incontinent of bowel and bladder.','Give 2 tablet by mouth every 4 hours as needed for Mild to moderate Pain Not to exceed 3 grams in 24 hours']})

Note - 1) I’m running the code on aws ec2 instance r5.8x large(32 CPUs) - I tried using multiprocessing but the code goes into a deadlock because bert takes all my cpu cores.

Context:
hi @madhuryadav,
you could try to use onnxruntime   for this to get some speed-up. Here’s a notebook 45 which shows, how to use onnxruntime for bert.
This could also help

      github.com

patil-suraj/onnx_transformers 17

  Accelerated NLP pipelines for fast inference on CPU. Built with Transformers and ONNX runtime.


Question:
I’m trying to work on a translation yo-en using MarianMt since I found a pretrained bilingual for my need however I checked the link here, but the source txt language was encoded differently, a lot of the characters changed. I need help in proceeding, I think it would affect performance. Thanks🤍
https://object.pouta.csc.fi/OPUS-MT-models/yo-en/opus-2020-01-16.test.txt 4

Context:
You are probably on Windows, right? That text file contains UTF-8 characters, but windows (still) defaults to cp1252 or something like that. That means it does not correctly display those characters by default. That does not mean that the text is incorrect: in byte format, it is correct but your computer is just showing it incorrectly. You can check this by downloading the file and opening it in your favourite editor with an UTF-8 encoding. So if you open this file in Python, for instance, then you have to use something like
with open(yourfile, encoding="utf-8") as fh:
   ...

That should help you. However, this is a very general issue and has nothing at all to do with transformers or any other HF libraries. So please use some other forums for general questions like this, like Stack Overflow.


Token classification on custom BERT and data

Question:
I am trying to follow the instructions from here 15 but using my own fine-tuned BERT model. I had to make a few adaptations but I think I got the data in the same format as the example. However, when I try to run the Trainer, I get the following error:
RuntimeError: The size of tensor a (8160) must match the size of tensor b (16) at non-singleton dimension 0
I had defined the max_length=510 on the call to tokenizer function, to match what I used to train the language model. I also manually pad the labels to match this length.
The code I’m using is the following:
def main():
    train_texts, train_tags = read_data('./data/ner_train.pkl')
    val_texts, val_tags = read_data('./data/ner_test.pkl')

    unique_tags = set(tag for doc in train_tags for tag in doc)
    tag2id = {tag: id for id, tag in enumerate(unique_tags)}
    id2tag = {id: tag for tag, id in tag2id.items()}

    tokenizer = AutoTokenizer.from_pretrained("C:\\Users\\Rogerio\\Documents\\bert-beaver-language")

    train_encodings = tokenizer(train_texts, is_split_into_words=True, return_offsets_mapping=True, padding=True,
                                truncation=True, max_length=510)
    val_encodings = tokenizer(val_texts, is_split_into_words=True, return_offsets_mapping=True, padding=True,
                              truncation=True, max_length=510)

    labels = [[tag2id[tag] for tag in doc] for doc in train_tags]
    train_labels = []
    for doc_labels, doc_offset in zip(labels, train_encodings.offset_mapping):
        # pad labels if necessary
        if len(doc_labels) < len(doc_offset):
            doc_labels += [-100] * (len(doc_offset) - len(doc_labels))

        for label, offset in zip(doc_labels, doc_offset):
            if offset[0] != 0 or offset == (0, 0):
                label = -100
            train_labels.append(label)

    labels = [[tag2id[tag] for tag in doc] for doc in val_tags]
    val_labels = []
    for doc_labels, doc_offset in zip(labels, val_encodings.offset_mapping):
        # pad labels if necessary
        if len(doc_labels) < len(doc_offset):
            doc_labels += [-100] * (len(doc_offset) - len(doc_labels))

        for label, offset in zip(doc_labels, doc_offset):
            if offset[0] != 0 or offset == (0, 0):
                label = -100
            val_labels.append(label)

    train_encodings.pop("offset_mapping")  # we don't want to pass this to the model
    val_encodings.pop("offset_mapping")
    train_dataset = NERDataset(train_encodings, train_labels)
    val_dataset = NERDataset(val_encodings, val_labels)

    model = AutoModelForTokenClassification.from_pretrained("C:\\Users\\Rogerio\\Documents\\bert-beaver-language",
                                                            num_labels=len(unique_tags))

    training_args = TrainingArguments(
        output_dir="C:\\Users\\Rogerio\\Documents\\bert-beaver-ner\\ner_output",
        logging_dir="C:\\Users\\Rogerio\\Documents\\bert-beaver-ner\\ner_logs",
        num_train_epochs=8,  # total number of training epochs
        per_device_train_batch_size=16,  # batch size per device during training
        per_device_eval_batch_size=64,  # batch size for evaluation
        warmup_steps=500,  # number of warmup steps for learning rate scheduler
        weight_decay=0.01,  # strength of weight decay
        logging_steps=10,
        save_total_limit=5,
        overwrite_output_dir=True,
        save_steps=750,
        do_eval=True,
        do_train=True,
        do_predict=True
    )

    trainer = Trainer(
        model=model,  # the instantiated 🤗 Transformers model to be trained
        args=training_args,  # training arguments, defined above
        train_dataset=train_dataset,  # training dataset
        eval_dataset=val_dataset  # evaluation dataset
    )

    trainer.train()

The full stack trace is below:
 0%|          | 0/1797240 [00:00<?, ?it/s]Traceback (most recent call last):
  File "C:/Users/Rogerio/Documents/Python_Projects/beaver-model-training/ner_bert.py", line 558, in <module>
    main()
  File "C:/Users/Rogerio/Documents/Python_Projects/beaver-model-training/ner_bert.py", line 496, in main
    trainer.train()
  File "C:\Users\Rogerio\python-virtual-envs\beaver-model-training\lib\site-packages\transformers\trainer.py", line 747, in train
    tr_loss += self.training_step(model, inputs)
  File "C:\Users\Rogerio\python-virtual-envs\beaver-model-training\lib\site-packages\transformers\trainer.py", line 1075, in training_step
    loss = self.compute_loss(model, inputs)
  File "C:\Users\Rogerio\python-virtual-envs\beaver-model-training\lib\site-packages\transformers\trainer.py", line 1099, in compute_loss
    outputs = model(**inputs)
  File "C:\Users\Rogerio\python-virtual-envs\beaver-model-training\lib\site-packages\torch\nn\modules\module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\Rogerio\python-virtual-envs\beaver-model-training\lib\site-packages\transformers\models\bert\modeling_bert.py", line 1541, in forward
    active_loss, labels.view(-1), torch.tensor(loss_fct.ignore_index).type_as(labels)
RuntimeError: The size of tensor a (8160) must match the size of tensor b (16) at non-singleton dimension 0

Any ideas?

Context: This error seems to mean your outputs and labels have mismatched shapes. I would double-check the shapes by grabbing a batch in your training dataloader and looking at the inputs and labels.


MRPC Reproducibility with transformers-4.1.0

Question:
I always get lower precision following the MRPC example, what’s the reason?
python run_glue.py \
  --model_name_or_path bert-base-cased \
  --task_name $TASK_NAME \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3.0 \
  --output_dir /tmp/$TASK_NAME/

and get
12/18/2020 17:16:38 - INFO - __main__ -   ***** Eval results mrpc *****
12/18/2020 17:16:38 - INFO - __main__ -     eval_loss = 0.5318707227706909
12/18/2020 17:16:38 - INFO - __main__ -     eval_accuracy = 0.7622549019607843
12/18/2020 17:16:38 - INFO - __main__ -     eval_f1 = 0.8417618270799347
12/18/2020 17:16:38 - INFO - __main__ -     eval_combined_score = 0.8020083645203595
12/18/2020 17:16:38 - INFO - __main__ -     epoch = 3.0

12/18/2020 16:45:29 - INFO - __main__ -   ***** Eval results mrpc *****
12/18/2020 16:45:29 - INFO - __main__ -     eval_loss = 0.47723284363746643
12/18/2020 16:45:29 - INFO - __main__ -     eval_accuracy = 0.8063725490196079
12/18/2020 16:45:29 - INFO - __main__ -     eval_f1 = 0.868988391376451
12/18/2020 16:45:29 - INFO - __main__ -     eval_combined_score = 0.8376804701980294
12/18/2020 16:45:29 - INFO - __main__ -     epoch = 3.0

12/18/2020 16:34:37 - INFO - __main__ -   ***** Eval results mrpc *****
12/18/2020 16:34:37 - INFO - __main__ -     eval_loss = 0.571368932723999
12/18/2020 16:34:37 - INFO - __main__ -     eval_accuracy = 0.6838235294117647
12/18/2020 16:34:37 - INFO - __main__ -     eval_f1 = 0.8122270742358079
12/18/2020 16:34:37 - INFO - __main__ -     eval_combined_score = 0.7480253018237863
12/18/2020 16:34:37 - INFO - __main__ -     epoch = 3.0

GPU: GTX 1080
transformers: 4.1.0
Torch: 1.6.0
python: 3.8
Server: Ubuntu 18.04

Context: Please don’t use two different account to post the exact same message in two different categories.


Based on HF documentation, unnaswerable questions from Squad 2.0 don’t make it into train/val data

Question:
Hi, I wanted to finetune Electra on my own Squad 2.0-style dataset so I looked at the following documentation to figure out what the data format should be.

      huggingface.co

Fine-tuning with custom datasets — transformers 3.5.0 documentation 5

It seems in the walkthrough that only answerable questions actually make it into the training/validation datasets. In the JSON files, if a question cannot be answered, the “answers” array is empty. However in the walkthrough, this is how a (context, question, answer) triplet gets added to the data:
    for answer in qa['answers']:
        contexts.append(context)
        questions.append(question)
        answers.append(answer)

Because it’s iterating through the “answers” array, if i’m not mistaken, the questions that are unanswerable will never get added to the data.

Context: @valhalla @sgugger Not sure if you two are the right people to tag but thought I’d start somewhere!


Specify attention masks for some heads in multi-head attention

Question:
Suppose I have 16-head Transformer layers in a standard BERT model. I want to constrain the first head of all the transformer layers to attend to tokens only in the same sentence, while the other 15 heads can attend to all the (non-padding) tokens (which is the default).
I looked at head_mask, but that merely specifies which heads to deactivate (0/1).
Looked at attention_mask, but that does not provide a way to specify different masks for different heads.
Any suggestions would be awesome!
EDIT - example
The input is a number of sentences, let’s say:

Drums, drums in the deep. We cannot get out. They are coming.

I want the first head of multi-head attention to just attend to tokens/words in the same sentence. So, when calculating the dot-product attention for “We”, the only words considered are “We cannot get out”. The other sentences are ignored. This can be specified by getting a num_words x num_words mask for the first head, and for each row, placing 1’s for other words in the same sentence, and 0s for words not in the same sentence.
However, there doesn’t seem to be a clean way of specifying per-head attention masks. I want to make sure that I am not missing some obvious way of doing this using the huggingface methods.

Context:
Hi arunirc,
(I’m not sure I understand what you want to do.   Are you planning to do a Next Sentence Prediction type of task, where you input pairs of texts having up to 512 tokens each?   Or, when you say “same sentence” do you mean a set of words separated by full stops?)
Within each layer, after the attention heads, the output from each head needs to be concatenated before the feed-forward network.   For that concatenation to work, I suspect that the matrices need to be the same size.   If so, then I don’t think you could have one head only attending to a few tokens, because its matrix would be smaller.
Have you considered using two completely separate BERT models, one of which has only 1 head per layer and takes as input only the first text,  and the other of which has 15 heads per layer and takes as input the pairs of texts, and combining the outputs later.


Question:
First off, the zero-shot module is amazing. It wraps up a lot of boiler-plate that I’ve been using into a nice succinct interface. With that however, I’m having trouble getting the gradients of intermediate layers. Let’s take an example:
from transformers import pipeline
import torch

model_name = 'facebook/bart-large-mnli'
nlp = pipeline("zero-shot-classification", model=model_name)

responses = ["I'm having a great day!!"]
hypothesis_template = 'This person feels {}'
candidate_labels = ['happy', 'sad']
nlp(responses, candidate_labels, hypothesis_template=hypothesis_template)

This works well! The output is:
{'sequence': "I'm having a great day!!",
 'labels': ['happy', 'sad'],
 'scores': [0.9989933371543884, 0.0010066736722365022]}

What I’d like to do however, is look at the gradients of the input tokens to see which tokens are important. This is in contrast to looking at the attention heads (which is also another viable tactic). Trying to rip apart the internals of the module, I can get the logics and embedding layers:
inputs = nlp._parse_and_tokenize(responses, candidate_labels, hypothesis_template)
predictions = nlp.model(**inputs, return_dict=True, output_hidden_states=True)
predictions['logits']

tensor([[-3.1864, -0.0714,  3.2625],
        [ 4.5919, -1.9473, -3.6376]], grad_fn=<AddmmBackward>)

This is expected, as the label for “happy” is index 0 and the entailment index for this model is 2, so the value of 3.2625 is an extremely strong signal. The label for “sad” is 1 and the contradiction index is 0, so the value of 4.5919 is also the correct answer.
Great! Now I should be able to look at the first embedding layer and check out the gradient with respect to the happy entailment scalar:
layer = predictions['encoder_hidden_states'][0]
layer.retain_grad()
predictions['logits'][0][2].backward(retain_graph=True)

Unfortunately, layer.grad is None. I’ve tried almost everything I can think of, and now I’m a bit stuck. Thanks for the help!

Context: I’ve reproduced this but not sure if I have a good answer – looks like more of a Bart/PyTorch question rather than something specific to the zero shot pipeline. Maybe @patrickvonplaten would have an idea?


TokenizerFast with various units (e.g., BPE, wordpiece, word, character, unigram)

Question:
I am using BartTokenizerFast to encode & decode my dataset.
I can find the documentation for using BPE unit for TokenizerFast
(including
https://colab.research.google.com/github/huggingface/transformers/blob/master/notebooks/01-training-tokenizers.ipynb 2)
Is there any useful documentation for using other units (e.g., wordpiece, word, character) for TokenizerFast?
Thank you !

Context:
Yes all the doc is here: https://huggingface.co/docs/tokenizers/python/latest/ 7
cc @anthony @Narsil


Question: Hi all - just a quick clarification about causal masks in GPT2 and BertLMHeadModel.  Its clear that GPT2 automatically adds causal masks (lines 115 and 151 in modeling_gpt2.py).  I believe that this should be the case for BertLMHeadModel as well - my understanding is that it is mainly meant to be used as a decoder in the EncoderDecoder construct (please correct me if this is not the case).  However, I am having a hard time finding where this occurs in the source.  Are causal masks automatically added? I suspect I am just missing where this happens in the code as the attention_mask input (meant for masking padding tokens) wouldn’t take care of this.  Thanks!

Context:
Hi @dilawn, have you figured this out yet by any chance? I have just posted a similar question on not finding causal masks in RobertaForCausalLM and BertGeneration classes. I am just wondering if I misunderstand the concept or missing the line of code that applies these causal masks.
Link to my post 8.


Positional Encoding error, Protein Bert Model

Question:
Hi Guys,
This seems very obivious but I can’t seem to find an answer anywhere. I’m trying to build a very basic roberta protein model similar to ProTrans. It’s just Roberta but I need to use a very long positional encodings of 40_000, because protein seqeunces are about 40,000 amino acids long. But anytime I change the max postional embeddings to 40k I keep getting an CUDA error: device-side assert triggered error.
(additidonal question I save the linebylinetext to a variable. From what I understand linebyline splits the sentence into chunks, but how can I index in to see each chunk?)
(also other than a lambda function is there any more efficient way I can preprocess and tokenise this dataset?)
Steps

Preprocess a Uniref50 sequence into a single space text document. Treating each Amino acid as a word and each protein as a sentence.
Tokenise, I can load in the tokens from ProTrans model, tokenises fine.
Use line by line text dataset to split each into 40_0000 blocks,
Step up the config, important changes to roberta vocab size 30, max_positional_embeddings= 40_000
Run through datacollator, model setup, train setup, get error.

Context:
Hi @donal I guess it’s near impossible to put 40 000 tokens in any transformer model that uses full attention. Maybe a better choice is Reformer 1 or LongFormer(I am not sure that LongFormer can handle 40 000 tokens too). The main problem of BERT architecture is that it’s memory and computational complexity is O(L^2) where L is max_seq_len, so I guess about 2048 tokens is upper limit.
Or maybe consider using something like sliding window or some kind of memory banks


What could be causing ” line 51, in write_predictions_to_file if not preds_list[example_id]: IndexError: list index out of range” in token-classification?

Question:
I am running token classification on my data, and it runs just fine except for write_predictions_to_file in https://github.com/huggingface/transformers/blob/master/examples/token-classification/tasks.py#L51 1
I believe its and issue with my data because I run the script over other data sources without any issues.
It seems that it runs the predictions just fine, and the issue is with writing the predictions to file; there seems to be some sort of index mismatch. The full error is
Traceback (most recent call last):
  File "run_ner.py", line 308, in <module>
    main()
  File "run_ner.py", line 297, in main
    token_classification_task.write_predictions_to_file(writer, f, preds_list)
  File "/content/transformers/examples/token-classification/tasks.py", line 51, in write_predictions_to_file
    if not preds_list[example_id]:
IndexError: list index out of range

Specifically, the mismatch seems to be between pred_list and test_input_reader .
I’ve been looking at the difference between the data that has caused the error, and data that runs just fine, but I can’t seem to pick anything out. I was thinking maybe it was caused by several new lines in a row, or some other lining issue, but I haven’t seen it yet.
Anyone have an idea.
For convenience, I recreated the issue in this colab notebook.

      colab.research.google.com

Google Colaboratory 4

Context: Hi @reSearch2vec, I will have a look on it and report back here


Loading models sometimes maxes DISK%, then crashes

Question:
Got an issue where quite often, somewhere in the chain my disk goes 100% read (500mb/s for 10-20m) then crash. I’ve put loggers everywhere to see what’s causing it, and the last logger is usually after loading the summarization model (code here 2, model used). It could be anything; that’s the most common model used on the service. All I know is it’s transformers, as it’s always that file/module that triggers it.
My models are all persisted, so it’s not re-downloading. Dev’ing Docker on Windows (WSL2 with nvidia-docker / dev-channel). I know that’s the smoking gun, but it happens on my Ubuntu server-server too. pytorch=1.6.0 cuda=10.1 cudnn=7 transformers=3.3.1 python=3.8 (github/lefnire/dockerfiles, forum not letting me post >2 links). I saw github/huggingface/transformers/issues/5001 which had me wondering if it’s a pytorch<->cuda<->transformers version bad-match (the ticket’s very old; cuda 9.2, etc). But is there a recommended/common version-combo of Pytorch, CUDA, Python for transformers?
Could it be something with the .lock files?

Context: I looked into Huggface Dockerfiles, and we’re using the same setup except Python version (theirs Ubuntu 18.04 default Python 3.6). They also install mkl, an Intel optimizer, but not sure if used anywhere; I installed it too check, no cigar. Blast, it’s nearly every time I load facebook/bart-cnn-large.


Fine tuning bert on next sentence prediction task

Question:
I am trying to fine-tune Bert using the Huggingface library on next sentence prediction task. I looked at the tutorial and I am trying to use  DataCollatorForNextSentencePrediction  and  TextDatasetForNextSentencePrediction  . When I am using that I get the following error(use the pastebin link to see the error)https://pastebin.pl/view/bde2c3d4 50. I have provided my code bellow.
============Code================
def train(bert_model,bert_tokenizer,path,eval_path=None):
out_dir = “/content/drive/My Drive/next_sentence/”
training_args = TrainingArguments(
output_dir=out_dir,
overwrite_output_dir=True,
num_train_epochs=1,
per_device_train_batch_size=30,
save_steps=10000,
save_total_limit=2,
)
data_collator = DataCollatorForNextSentencePrediction(
tokenizer=bert_tokenizer,mlm=False,block_size=512,nsp_probability =0.5
)
  dataset = TextDatasetForNextSentencePrediction(
    tokenizer = bert_tokenizer,
    file_path=path,
    block_size=512,
  )

  trainer = Trainer(
      model=bert_model,
      args=training_args,
      train_dataset=dataset,
      data_collator=data_collator,

  )
  trainer.train()
  trainer.save_model(out_dir)
def main():
  print("Running main")
  bert_tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
  bert_model = BertForNextSentencePrediction.from_pretrained("bert-base-cased")
  train_data_set_path = "/content/drive/My Drive/next_sentence/line_data_set_file.txt"
  train(bert_model,bert_tokenizer,train_data_set_path)
  #prepare_data_set(bert_tokenizer)
main()

Context: Can you fix the formatting in your post? It would make it easier to read


Trying to understand XForSequenceClassification heads

Question:
I’m interested in 1-sentence and 2-sentence text classification, so I’ve been looking at the classification heads for BERT, GPT2, XLNet, and RoBERTa. I have a few questions:
1. I see that there are dedicated classification classes BertForSequenceClassification 1, XLNetForSequenceClassification, and  RobertaForSequenceClassification. However, there is no XForSequenceClassification class for GPT2. Is there any documentation to help us write our own?
2. When I look at the classification heads for BERT, XLNet, and RoBERTa, the layer structure for producing the logits appears to be different for each one. I would think that the final few layers would be exactly the same.
For example, here is the code for the BERT classification head:
class BertForSequenceClassification(BertPreTrainedModel):
    def __init__(self, config):
        self.bert = BertModel(config)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)
        self.classifier = nn.Linear(config.hidden_size, config.num_labels)

    def forward( ... ):
        outputs = self.bert( ... )
        pooled_output = outputs[1]
        pooled_output = self.dropout(pooled_output)
        logits = self.classifier(pooled_output)

Here is the code for the XLNet classification head:
class XLNetForSequenceClassification(XLNetPreTrainedModel):

    def __init__(self, config):
        self.transformer = XLNetModel(config)
        self.sequence_summary = SequenceSummary(config)
        self.logits_proj = nn.Linear(config.d_model, config.num_labels)

    def forward( ... ):
        transformer_outputs = self.transformer( ... )
        output = transformer_outputs[0]
        output = self.sequence_summary(output)
        logits = self.logits_proj(output)

Here is the code for the RoBERTa classification head:
class RobertaForSequenceClassification(BertPreTrainedModel):
    config_class = RobertaConfig
    base_model_prefix = "roberta"

    def __init__(self, config):
        super().__init__(config)
        self.num_labels = config.num_labels

        self.roberta = RobertaModel(config)
        self.classifier = RobertaClassificationHead(config)

    def forward( ... ):
        outputs = self.roberta( ... )
        sequence_output = outputs[0]
        logits = self.classifier(sequence_output)

class RobertaClassificationHead(nn.Module):
    """Head for sentence-level classification tasks."""

    def __init__(self, config):
        super().__init__()
        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)
        self.out_proj = nn.Linear(config.hidden_size, config.num_labels)

    def forward(self, features, **kwargs):
        x = features[:, 0, :]  # take <s> token (equiv. to [CLS])
        x = self.dropout(x)
        x = self.dense(x)
        x = torch.tanh(x)
        x = self.dropout(x)
        x = self.out_proj(x)
        return x

From the above code, producing the output logits involves:

BERT: dropout, linear
XLNet: linear
RoBERTa: dropout, linear, tanh, dropout, linear

Why does each model implement the final layers differently? Are the implementations taken from the original papers? Wouldn’t it be better to make the classification layers exactly the same for each model so that classification relative performance would be a result of the models’ internal architecture rather than the classification layer?
Thank you for any help.

Context: Any help would be greatly appreciated.


Question:
Hello,
I occasionally run into bugs (rare, minor, of course) in the Huggingface codebase, which I fix in my own pulls. Does Huggingface accept pull requests from the general user base (with an appropriate issue, of course)?
Thanks,
Andreas

Context: Yes! PRs are  welcome   (I’m a contributor ),  Here’s the contributing guide 7


Question:
First of all, sorry if I am missing something because this is my first post on the forum.
I have thought that users / teams that train and share BERT like models could share the formatted corpus in order to other users to be able to use it in new models such as Electra easily saving time and computation (as the corpus format is the same).
Maybe HF could store and share formatted corpus or just the users/team can provide it.
The idea comes because I have seen several BERT like models in the hub for different languages and I would like to bring it to Electra as it needs less resources for pretraining and I (and maybe many users) can deal with it.

Context: HuggingFace already provides ~100 NLP datasets in their nlp 14 repository. I think these are mostly evaluation datasets. However, you can add your own dataset 8, too!


Question:
Hi,
Due to recent code changes by @sshleifer, I am trying to understand what is desired for BART’s input for training and generation, and whether the codebase is reflecting it properly as I’ve encountered some inconsistencies.
I am assuming both src_ids and tgt_ids are encoded with a BART tokenizer, and therefore have the format of [bos, token1, token2, …, eos].
Looking at transformers/examples/seq2seq/finetune.py#L151 10
decoder_input_ids = shift_tokens_right(tgt_ids) means that eos will be the first token and bos will be the second token.
This has an effect on generation:

We need decoder_start_token_id=eos_token_id.
The first actually generated token (i.e. after decoder_start_token_id) will be bos.

Questions:

The default value for decoder_start_token_id is missing from facebook/bart-base and facebook/bart-large-mnli, which means it falls back to bos. The other BART models have eos as their decoder_start_token_id. Why is the difference? Looks to me that using finetune.py with bart-base/bart-large-mnli will not have generation as intended.

In fairseq’s implementation the equivalent for decoder_start_token_id is set to bos: fairseq/models/bart/hub_interface.py#L123. Can you please explain why did you decide to use the format of [eos, bos, token1, token2, ...] for decoder_input_ids instead of [bos, token1, token2, ...]?

Is there still need for force_bos_token_to_be_generated?
It was introduced in transformers/pull/6526 (new user, can’t add another link), when the first token of decoder_input_ids was bos and the second was the first regular token of the target sequence (transformers/examples/seq2seq/finetune.py#L144). Using it now shouldn’t have any effect, if I understand correctly (because a trained model will easily learn to always output bos in this position anyway).

Thanks!

Context:
This is a great question and super confusing.
You can get a good snapshot of my understanding here 67. I don’t think that will clear everything up, but if you could read that and let me know what you still don’t understand it would be helpful.
If I encounter empirical evidence that I should change decoder_start_token_id for any model I will do so.


Properly loading a fine tuned model from directory

Question:
Hi everybody,
After reading the docs I still don’t really understand how should I load my saved model properly.
I fine tuned a CamembertForSequenceClassification.from_pretrained model with some data, the results was good so I saved it using save_pretrained(model_path). Now I would like to use this model to do inference… I use these lines of code:
config = AutoConfig.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForSequenceClassification.from_config(config)
I now have several interogation:

If I tokenize a new sequence with tokenizer.encode_plus(new_seq), it doesn’t tokenize the sentence the way it did when I previously saved the model. I must precise all the params (max_length…) to retrieve what I want.  Is that normal?
If I load my saved model without using AutoModelForSequenceClassification but with AutoModel the model do not load a classification layer however I saved my model using this last layer. Is that also normal?

So I’m wondering how can I be sure that I load the exact same model with all the weights and params that I saved previously.
Thanks for everything!

Context:
Yes, you should pass the same params to tokenizer that you passed during training.

AutoModel won’t add the heads (classfication layer, LM head etc). Use proper AutoModel classes to get same weights and params i.e AutoModelForSequenceClassification for classification model, AutoModelForCausalLM for causal LM’s like GPT-2 etc

You can load the saved model using just AutoModelForSequenceClassification.from_pretrained, it’ll load the config itself. You should pass config explicitly when you want to change something in it.


Question: Hi, I am working on training a domain-specific ‘extractive Q&A’ model but I was wondering if the model could say “I dont know” when the answer is not in the context.  Any comments?

Context:
I think SQuAD 2.0 introduced this kind of “unanswerable” questions:

      arXiv.org

Know What You Don't Know: Unanswerable Questions for SQuAD 13

Extractive reading comprehension systems can often locate the correct answer
to a question in a context document, but they also tend to make unreliable
guesses on questions for which the correct answer is not stated in the context.
Existing datasets...

Demo on the model hub that uses a fine-tuned model:
https://huggingface.co/deepset/bert-base-cased-squad2?text=Where+lives+Wolfgang%3F&context=My+name+is+Wolfgang+and+I+live+in+Berlin 18
Outputs “Berlin” correctly. Now an unanswerable question (because it is not given in the context):
https://huggingface.co/deepset/bert-base-cased-squad2?text=Where+was+Obama+born%3F&context=My+name+is+Wolfgang+and+I+live+in+Berlin 17


RoBERTa from scratch with different vocab vs. fine-tuning

Question:
I have a question about training custom RoBERTa model. My corpus consists of 100% english text, but the structure of the text I have is totally different than well formed english books / wikipedia sentences. As the overall nomenclature of my dataset is very different form books / wikipedia I wanted to train new LM from scratch using a new tokenizer trained on my dataset, to capture this corpus-specific nomenclature.
I would like to hear from experts - which of the following approaches is the best one for my case?

Train custom tokenizer and train RoBERTa from scratch
Just fine tune pretrained RoBERTa and rely on the existing BPE tokenizer
Use pretrained RoBERTa and somehow adjust the vocab (if it’s even possible and if so, then how?)

Context: Bump, anyone?


Question:
Hi folks,
Problem: fine-tuned model adopts peculiar behaviour with beam search.
Specifically, beam search outputs include 2 bos tokens and exclude the first word token. I have double checked my data feed and the inputs look fine, with one bos and the first word token is present. Even more strange, there are no such problems with nucleus sampling, which suggests that this is particular to beam search.
for example, here are two outputs from the same fine-tuned model;
beam search output: <s><s>rek likes to play football. </s>
nuleus sampling output: <s> Derek likes to go to the football pitch. </s>

Any clues?

Context: Hi,how did you prepare labels and decoder_input_ids ?


Performing Back Translation with T5 network

Question:
Hi, current trying to reproduce the paper “Unsupervised Translation of Programming Languages 6” (TransCoder) from Facebook research, but using the T5 network as the seq2seq model. Right now I am stuck on the back translation part from the approach:
def back_translate(self, batch: List[torch.Tensor]) -> Dict[str, torch.Tensor]:
        device = batch[0]['input_ids'].device
        print(device)
        input_ids = torch.stack([example['input_ids'] for example in batch])
        target_ids = torch.stack([example['target_ids'] for example in batch])
        attention_mask = torch.stack([example['attention_mask'] for example in batch])
        batch_langs = [self.tokenizer.decode([ids[0].detach()]) for ids in target_ids]
        lang = random.choice(self.langs)
        self.model.eval()
        cpu_model = self.model.to('cpu')
        outputs = cpu_model.generate(
            input_ids = input_ids, attention_mask = attention_mask,
            decoder_start_token_id = self.tokenizer.encode(f'<{lang}>')[0], max_length = 256
        )
        self.model.train()

        inputs = [self.tokenizer.decode(ids).replace('complete: ', '') for ids in input_ids]
        outputs = [self.tokenizer.decode(ids[1:]) for ids in outputs] # remove lang token
        examples = []
        for inpt, outpt, l in zip(outputs, inputs, batch_langs):
            inpt = f'complete: {inpt}  </s>'
            outpt = f'<{l}>{outpt}'
            input_encodings = self.tokenizer.encode_plus(inpt, pad_to_max_length = True, max_length = 256, truncation = True)
            target_encodings = self.tokenizer.encode_plus(outpt, pad_to_max_length = True, max_length = 256, truncation = True)
            encodings = {
                'input_ids': torch.tensor(input_encodings['input_ids'], dtype=torch.long, device = xm.xla_device()),
                'attention_mask': torch.tensor(input_encodings['attention_mask'], dtype=torch.long, device = xm.xla_device()),
                'target_ids': torch.tensor(target_encodings['input_ids'], dtype=torch.long, device = xm.xla_device()),
                'target_attention_mask': torch.tensor(target_encodings['attention_mask'], dtype=torch.long, device = xm.xla_device())
            }

            examples.append(encodings)

        input_ids = torch.stack([example['input_ids'] for example in examples])
        input_ids, _ = self.masked_data_collator.mask_tokens(input_ids)
        lm_labels = torch.stack([example['target_ids'] for example in examples])
        lm_labels[lm_labels[:, :] == 0] = -100
        attention_mask = torch.stack([example['attention_mask'] for example in examples])
        decoder_attention_mask = torch.stack([example['target_attention_mask'] for example in examples])
        print(input_ids.device, xm.xla_device())
        return {
            'input_ids': input_ids,
            'attention_mask': attention_mask,
            'lm_labels': lm_labels,
            'decoder_attention_mask': decoder_attention_mask
        }

I am training this in Google colab using TPUs, but even though I am explicitly putting the tensors onto the TPU device, it is giving me an error saying: Input tensor is not an XLA tensor: torch.FloatTensor
Here is a link to the full colab notebook: https://colab.research.google.com/drive/1nRGkCdei7D6v6njKWPVZZWtGgPAedkVQ?usp=sharing 6
Any help or advice would be greatly appreciated!

Context:
Can you post the full error trace? Just to be safe, you can do
        return {
            'input_ids': input_ids.to(xm.xla_device()),
            'attention_mask': attention_mask.to(xm.xla_device()),
            'lm_labels': lm_labels.to(xm.xla_device()),
            'decoder_attention_mask': decoder_attention_mask.to(xm.xla_device())
        }


Question:
Do one time:
wget https://github.com/stas00/git-tools/blob/master/how-to-make-pr/github-make-pr-branch
chmod u+x github-make-pr-branch
sudo cp github-make-pr-branch /usr/bin/

(or put it in your local bin path)
And then any time you want to work on a new huggingface transformers feature, just run:
github-make-pr-branch ssh GITHUB-USERNAME huggingface transformers new-feature

It will:

Fork the project if needed
Set up this fork to track the upstream
Sync your fork to upstream master
Create ‘new-feature’ branch
Set its upstream for an easy push

Now you just need to:
cd transformers-new-feature

do your magic and when ready:
git commit -a
git push

and the PR is ready (just need to go to github and click on PR suggestion to submit)
It, of course, works for any github project.
If this is useful I can make a PR to add instructions to CONTRIBUTING.pm.
This could also be converted in an even simpler script invocation specific to transformers, so it’ll just need:
transformers-pr new-feature

currently it uses an ssh access.
There is also a detailed guide to  How to Make a Pull Request (PR) 1.

Context:
Cool!
Another way is to directly use the github cli 1.
My own process:

Clone repo
git clone repo_url

Create new branch
git checkout -b feat-add_cool_stuff

Regular commits (I do them through VS Code because I like the interface)
Send PR with github cli
gh pr create


Calculate Impact of Input Tokens on BERT Output Probability

Question:
Say I’ve trained a BERT model for classification. I’d like to calculate the proportional impact each input token is having on the predicted output.
For example - and this is very general - if I have a model that labels input text as {‘about dogs’ : 0, ‘about cats’ : 1}, the following input sentence:
s = 'this is a sentence about a cat'
should output very close to:
 1
HOWEVER, what I’d like is to calculate each input’s impact on that final prediction, e.g. (assuming we’re tokenizing on the level of words - which is not how it would be done in practice, I know):
{this : .01,     is: .005,     a : .02,     sentence : .0003,     about : [some other low prob],     a: [another low prob],     cat : 0.999999}
Intuitively I’d think this means running a forward pass with the input sentence, then looking at the backprop values? But I’m not quite sure how you’d do that. Thoughts?

Context:
Hi @matthew,
I’ve looked into this a bit in the past and your intuitions are nearly spot on! What you have described is something like a saliency map. Here are some references that might be useful:

https://arxiv.org/pdf/1312.6034.pdf 22
https://arxiv.org/pdf/1703.01365.pdf 13
https://arxiv.org/pdf/1706.03825.pdf 12

Taking the simplest case, what is effectively done is a forward pass, then a backward pass all the way to the input layer. The intuition is that will give you a sense of which tokens have the largest impact on the output if you were to change them by a small amount. In other words, if one of the input tokens from a sentence about cats is changed in such a way that the output of the model becomes 0 (i.e. a sentence about dogs), then that particular token would have a high saliency score.
Some of the papers above still were a bit unclear to me, so here are some supplementary references to hopefully get you on your way:

https://medium.com/@thelastalias/saliency-maps-for-deep-learning-part-1-vanilla-gradient-1d0665de3284 36
https://glassboxmedicine.com/2019/06/21/cnn-heat-maps-saliency-backpropagation/ 12
https://blog.qure.ai/notes/visualizing_deep_learning 17


Preprocessing required for fine-tuning RoBERTa

Question:
I am finetuning a QA model in Hindi using a trained Roberta LM. I need to preprocess the dataset for Roberta. What are the steps that I need to take before I feed the input to the model? One script for English is given here 9. I am not sure if other languages behave in a similar way. e.g The linked notebook adds an extra " " character before the start token. Is this necessary for Roberta? What are other nuances that have to be taken care of?
I am only concerned about spans at token-level and not character-level as explained in the link mentioned above.
Thank you.

Context:
Hi @kushalj001,
Maybe you can consider XLM-Roberta (XLM-R) instead of Roberta?
XLM-R supports 100 languages out-of-the-box, including its tokenizer which can tokenize most language in the world.
This kaggle notebook show how to finetune 3 languages simultaneously on XLM-R with TF2+TPU , which is extremely efficient (10x faster than P100)
https://www.kaggle.com/riblidezso/finetune-xlm-roberta-on-jigsaw-test-data-with-mlm 30


Question:
Show us what you’ve created using  transformers, nlp ! It could be a blog post, a jupyter notebook, a colab, a picture, a github repo, a web app, or anything else. Some tips:

Probably the easiest way to blog is using FastPages 5. You can easily convert your notebooks and .md files into blog posts using FastPages.
The easiest way to share a notebook on github is to install the gist it extension  1. This will only be possible if you use a platform that supports jupyter extensions, such as GCP. Otherwise, you can create a notebook gist by clicking File->Download to get your notebook to your computer, and then follow the steps from this SO post :

Go to https://gist.github.com/YOUR-GITHUB-USERNAME/

Click ‘New Gist’ on the upper right corner
Open the folder in a Finder/Explorer window on your local computer
Drag the file into the text box (the ‘code space’). This should fill the space with JSON looking text for the
framework of the notebook content.
Copy/Paste the full file name (e.g., mynotebook.ipynb) into the filename box, and give a description above.
Create the Gist!

If you want to have folks on the forum look at a draft and give feedback without sharing it more widely, just mention that in your post
You can also just use a reply to this topic to describe what you did - preferably pasting in a picture or two!

Context:
To start this,
Happy to announce it here first, I’ve been working on Question Generation using  transformers for past two months, and today releasing the first set of experiments here.
Question generation is the task of automatically generating questions from a text paragraph. This project is aimed as an open source study on question generation with pre-trained transformers (specifically seq-2-seq models) using straight-forward end-to-end methods without much complicated pipelines. The goal is to provide simplified data processing and training scripts and easy to use pipelines for inference.
Specifically, I trained T5 model for

answer aware question generation
multitask qa and qq
end-to-end question generation (without answer supervision)

Here’s a sneak peek
qa-qg-ss2048×1780 319 KB
Everything is built using  HuggingFace libraries

Dataset:  nlp library
model and training:  transformers

models hosted on:  model hub

For more details here’s the repo 13
Do share your feedback, specifically regarding the quality of questions, the mistakes and any ethical biases that you observe. Happy to discuss more details here. Cheers !
All models 1 are available on hub with configured inference API. You can search using question-generation tag.
Here’s a colab 9 if anyone wants to play more with it.


Question: Use this category to ask any question related to the course 87 or organize study groups.

Context:
Hey everyone, can we form a study group similar to how MLT 21 does it?
I really suck at keeping up with a course alone and would love to set up a weekly meeting kind of thing where we all can actually go through some parts of the course on our own and then discuss the same. We can then leave the entire week to practice it out and then discuss it briefly at the start of the next week session before repeating.


Question: Use this topic for any question about Chapter 7 5 of the course.

Context:
Hi - this is a relatively simple question but i’m totally new to HuggingFace so apologies in advance but on section 3 you discuss domain adaption.
I’m just experimenting with the task at the end of the section i.e. “To quantify the benefits of domain adaptation, fine-tune a classifier on the IMDb labels for both the pretrained and fine-tuned MiniLM checkpoints…”
Can you use the ‘Fill-Mask’ domain-adapted checkpoint you generated in the course (huggingface-course/distilbert-base-uncased-finetuned-imdb) for a classification task? Or do you have to adapt the original distilbert-base-uncased to the domain specifically for classification?


Question:
Please read the topic category description 2 to understand what this is all about

Description
For many online applications it is not known in advance what language an end-user will communicate in. The goal of this project is to build a system that can automatically predict the language a text is written in.

Model(s)
There are a few popular multilingual models that you can start with:

xlm-roberta-base 3
bert-base-multilingual-uncased 2

Datasets
There are quite a few multilingual datasets 6 available on the Hub. Many of these have a “language” field that could be used as a target for the model to predict.

Challenges
This project will likely require you to combine several datasets together to gain enough coverage of many languages.

Desired project outcomes

Create a Streamlit or Gradio app on  Spaces 1 that can predict the language of a piece of text provided by an end-user
Don’t forget to push all your models and datasets to the Hub so others can build on them!

Additional resources
A good baseline to compare your model against is the Python langid library 5

Discord channel
To chat and organise with other people interested in this project, head over to our Discord 4 and:

Follow the instructions on the #join-course channel

Join the #language-detection channel

Just make sure you comment here to indicate that you’ll be contributing to this project

Context:
@lewtun
Hi, I would like to work on this project.


Question: Use this topic for any question about Chapter 5  9 of the course.

Context:
Please correct me if I’m wrong.
A DataSet object can be thought of as some tabular data, whose rows and columns are examples and features respectively.  The length of a DataSet object is the number of examples, which equals to the number of its rows. Each column corresponds to one feature.
Given this understanding of DataSet, I found the following descriptions in chapter 5 are confusing (in other words incorrect?).

… here those 1,000 examples gave 1,463 new features, resulting in a shape error.

1,463 is the number of rows (i.e. examples) of the newly added columns (i.e. features) such as  attention_mask, input_ids, etc.

We can check that our new dataset has many more features than the original dataset by comparing the lengths:
len(tokenized_dataset[“train”]), len(drug_dataset[“train”])
(206772, 138514)

Obviously the above two numbers are the numbers of rows, i.e. the numbers of examples, not the number of columns (features). The number of features in this case is 4. Specifically, these 4 features are
attention_mask
input_ids
overflow_to_sample_mapping
and
token_type_ids


Question: Use this topic for any question about Chapter 6 1 of the course.

Context:
Hi,
In the section “Fast tokenizers’ special powers” (Tensorflow tutorial) executing this part of the code triggers an error:

Screenshot from 2021-11-16 11-05-532072×1026 232 KB


Question:
Hi,
I have been looking through one of the part 2 course tutorials for Inside the Token classification pipeline (PyTorch) illustrated by @sgugger. I have tried to use the example in (https://colab.research.google.com/github/huggingface/notebooks/blob/master/course/videos/token_pipeline_pt.ipynb) and used sshleifer/tiny-dbmdz-bert-large-cased-finetuned-conll03-english as a model-checkpoint. However, I had the following error,
AttributeError: ‘list’ object has no attribute ‘argmax’ in `import torch
probabilities = torch.nn.functional.softmax(outputs.logits, dim=-1)[0].tolist()
predictions = probabilities.argmax(dim=-1)[0].tolist()
print(predictions)`
Am I missing something?
Best,
Ghadeer

Context: Hey @ghadeermobasher there are several strategies that you can use to merge the entities and my suggestion would be to inspect the implementation in the token-classification pipeline 1. As you can see in the docstring 2 there are quite some subtleties associated with merging entities correctly (depends on the language, tokenizer etc)!


Question:
Hi there,
I have been following the tensorflow track of the HF course and got an http 404 error when running the below:
from transformers import TFAutoModelForSeq2SeqLM

model = TFAutoModelForSeq2SeqLM.from_pretrained(model_checkpoint)

error message:
404 Client Error: Not Found for url: https://huggingface.co/Helsinki-NLP/opus-mt-en-fr/resolve/main/tf_model.h5
I went to the model card and could not find the tf_model.h5 file. Is there something that I am missing or does the model only work for Torch?
thanks

Context:
Hi @nickmuchi, thanks for the bug report!
Indeed, you’re right that this model only has weights for PyTorch. However, you can load it in TensorFlow using the from_pt argument as follows:
from transformers import TFAutoModelForSeq2SeqLM

model = TFAutoModelForSeq2SeqLM.from_pretrained(model_checkpoint, from_pt=True)

In the meantime, I’ll post a fix on the website - thanks!


Question: Use this topic for any question about Chapter 3 27 of the course.

Context: Hi @sgugger, thank you for informative course. Do you have any documents or tutorial for fine-tuning a pretrain model for translation task? Thank you very much.


Question:
Please read the topic category description to understand what this is all about

Description
Many countries have populations that speak and write in more than one language. Building NLP applications in these conditions can be challenging, especially if the languages differ significantly from each other. The goal of this project is to explore the effectiveness of multilingual Transformer models by training a classifier that can analyze texts in multiple languages at once.

Model(s)
There are a few popular multilingual models that you can start with:

xlm-roberta-base 3
bert-base-multilingual-uncased 1

Datasets
There are several multilingual datasets on the Hub that you can use to get started:

amazon_reviews_multi 4

Even better would be to create a multilingual dataset in your own languages!

Challenges

The current multilingual models are typically limited to 100 languages or so. Check out the corresponding papers to see if your language is supported.

Desired project outcomes

Create a Streamlit or Gradio app on  Spaces that can automatically classify text in multiple languages.
Don’t forget to push all your models and datasets to the Hub so others can build on them!

Additional resources

This project has some overlap with the summarization section of Chapter 7 in the course.

Context:
Hi Lewtun,
is there any notebook with a solution of this task?
Kind regards


Question:
Chapter 3 of the course is called “fine-tuning”. Is this use of the term “fine-tuning” consistent with this article (Transfer learning & fine-tuning 1), which differentiates between transfer learning and fine tuning?
i.e. When we fine-tune a model as in Chapter 3, are we training the whole network, or are we only training the new layer(s) attached to the body of the pre-trained network?

Context: Fine-tuning is the act of re-training a pretrained model on a new dataset/task, it has nothing to do with freezing part of the network or not (at lest in the wat we use it in the course, which is the general way it is used in the published literature AFAIK). Freezing part of the network might help in some situations (like computer vision) but it doesn’t really help for Transformer models, usually it gives worse results.


Question:
Hi, I was using amazon datasets 1 for trying to build a small language detector. but bumped up the numpy.str_ error during training phase. You can view my colab notebook here: Google Colab 1 .
I was using review body field as text and language field as label. and dropped the other fields.
I found that the ‘language’ data field is type of datasets.Value, not the datasets.ClassLabel. I guessed this causing the numpy.str_ error during training.
Question:  how do I convert  datasets.Value to datasets.ClassLabel ? One way I can think of is doing  str2int inside preprocess_function/tokenize method but curious that is there any existing conversion method to do that.
Thanks

Context:
Hey @ivanlau I think your idea to apply ClassLabel.str2int() is the simplest approach, e.g.
from datasets import load_dataset, ClassLabel, Features

dset = load_dataset("amazon_reviews_multi", "all_languages", split="test")
# Create ClassLabel feature
langs = dset.unique("language")
lang_feature = ClassLabel(names=langs)
# Update default features
features = dset.features
features["language"] = lang_feature
# Update dataset
dset_with_classlabel = dset.map(lambda x : {"language": lang_feature.str2int(x["language"])}, features=features)

dset_with_classlabel.features
# {'language': ClassLabel(num_classes=6, names=['de', 'en', 'es', 'fr', 'ja', 'zh'], names_file=None, id=None),
#  'product_category': Value(dtype='string', id=None),
#  'product_id': Value(dtype='string', id=None),
#  'review_body': Value(dtype='string', id=None),
#  'review_id': Value(dtype='string', id=None),
#  'review_title': Value(dtype='string', id=None),
#  'reviewer_id': Value(dtype='string', id=None),
#  'stars': Value(dtype='int32', id=None)}

Alternatively, you can provide the features dictionary when you load the dataset with load_dataset. Hope that helps!


Question:
Description
A common data science task for many business is to be able to condense the news about their products or services into short summaries. The goal of this task is to fine-tune a model to automatically summarise news articles, ideally in a domain that is of interest to you!

Model(s)
There are various summarisation models on the Hub that have been fine-tuned on the famous CNN/Dailymail dataset. These provide a good starting point for performing domain adaptation:

google/pegasus-cnn_dailymail 4
google/roberta2roberta_L-24_cnn_daily_mail 4

There are also other summarization models that are worth investigating:

facebook/bart-large-cnn 3

Datasets
Using the summarization filter on the Hugging Face Hub 3 gives a good list of datasets to start from.

Challenges
[Explain whether the task is feasible with a single T4 GPU (what we get from AWS), does the data need a lot of preprocessing, are there ethical considerations etc]

Desired project outcomes

Create a Streamlit or Gradio app on  Spaces that can summarize news articles, either from their text or from a given URL.
Don’t forget to push all your models and datasets to the Hub so others can build on them!

Additional resources
Here are some existing spaces as inspiration:

https://huggingface.co/spaces/chinhon/News_Summarizer 11
https://huggingface.co/spaces/benthecoder/news-summarizer 5

Discord channel
To chat and organise with other people interested in this project, head over to our Discord 3 and:

Follow the instructions on the #join-course channel

Join the #new-summarizer channel

Just make sure you comment here to indicate that you’ll be contributing to this project

Context: Hi Lewis, I am interested in working on a text summarizer as my course project. How can I join this project? - Ali


Build a question answering system in your own language

Question:
Please read the topic category description to understand what this is all about

Description
One of the major challenges with NLP today is the lack of systems for the thousands of non-English languages in the world. In this project, the goal is to build a question answering system in your own language. There are two main approaches you can take:

Find a SQuAD-like dataset in your language (these tend to only exist for a few languages unfortunately)
Find a dataset of question / answer pairs and build a search engine that returns the most likely answers to a given question

Model(s)
For the SQuAD-like task, any BERT-like model in your language would be a good starting point. If such a model doesn’t exist, consider one of the multilingual Transformers like mBERT or XLM-RoBERTa.
For the search-like task, check out one of the many sentence-transformers models on the Hub: sentence-transformers (Sentence Transformers)

Datasets

squad 2

squad_it 1 (for inspiration)
mqa

Challenges
This is a somewhat complex project because it involves both training multilingual models on potentially low-resource languages.

Desired project outcomes

Create a Streamlit or Gradio app on  Spaces 1 that allows users to obtain answers from a snippet of text in your own language, or returns the top-N documents the might contain the answer.
Don’t forget to push all your models and datasets to the Hub so others can build on them!

Additional resources

GitHub - crux82/squad-it: A large scale dataset for Question Answering in Italian 1

Discord channel
To chat and organise with other people interested in this project, head over to our Discord and:

Follow the instructions on the #join-course channel

Join the #question-answering-de-fr channel

Just make sure you comment here to indicate that you’ll be contributing to this project

Context: Hi @lewtun would be glad to work on that project  Thanks


Question:
Description
Sometimes students spend more time working on their cover letters, customizing them for each company, than working on their portfolio. And the preliminary screening steps that are based only on cover letters are quite unfair. The goal of this project is to build a model that can automatically generate cover letters from a targeted job and a person’s skills / previous experiences / resume…

Model(s)
I’m open for suggestions but I believe these are good places to start with:

t5-base 2

Datasets
This is the hardest part. I already started working on scraping a couple of example resumes from a couple of websites (Indeed…), but we still need to fill the person’s information accordingly.

Challenges
Scrape scarce data for a real-world problem. Fine-tune a state-of-the-art language model.

Desired project outcomes

Create a Streamlit or Gradio app on  Spaces  that can generate a cover letter from a targeted job and a person’s resume
Don’t forget to push all your models and datasets to the Hub so others can build on them!

Additional resources
A good baseline to compare our model with gpt2 cover letter generator 4

Discord channel
To chat and organise with other people interested in this project, head over to our Discord and:

Follow the instructions on the #join-course channel

Join the #cover-letter-generator channel

Just make sure you comment here to indicate that you’ll be contributing to this project

Context: This looks like a super cool project @nouamanetazi - thanks for suggesting it! Let me know if you want a Discord channel and I’ll create it