Why my simple Bert model for text classification could not learn anything?

Question:
Hello, I try  transformers.BertModel to deal with a simple text classification, but the result makes me puzzled.
the code is simple,I implement the model with pytorch.
they are…
# a Dataset class for BertModel
class BertDataset(Dataset):
    def __init__(self, train_file, tokenizer):
        super(BertDataset, self).__init__()
        self.train_file = train_file
        self.data = []
        self.label2id = {}
        self.id2label = {}
        self.tokenizer = tokenizer
        self.init()

    def init(self):
        with open(self.train_file, 'r', encoding='utf-8') as f:
            for line in f:
                blocks = line.strip().split('\t')
                if blocks[1] not in self.label2id:
                    self.label2id[blocks[1]] = len(self.label2id)
                    self.id2label[len(self.id2label)] = blocks[1]
                self.data.append({'token': self.tokenizer(blocks[0], add_special_tokens=True, max_length=100,
                                                          padding='max_length', return_tensors='pt',
                                                          truncation=True),
                                  'label': self.label2id[blocks[1]]})

    def __getitem__(self, item):
        return self.data[item]

    def __len__(self):
        return len(self.data)

# a collate function for torch.utils.data.DataLoader
def bert_collate_fn(batch_data):
    input_ids, token_type_ids, attention_mask, labels = [], [], [], []
    for instance in copy.deepcopy(batch_data):
        input_ids.append(instance['token']['input_ids'][0].squeeze(0))
        token_type_ids.append(instance['token']['token_type_ids'][0].squeeze(0))
        attention_mask.append(instance['token']['attention_mask'][0].squeeze(0))
        labels.append(instance['label'])
    return torch.stack(input_ids), torch.stack(token_type_ids), \
           torch.stack(attention_mask), torch.tensor(labels)

# Model
class PTModel(nn.Module):
    def __init__(self, model, n_class):
        super(PTModel, self).__init__()
        self.n_class = n_class
        self.model = model
        self.linear = nn.Linear(768, self.n_class)
        self.softmax = nn.Softmax(dim=-1)

    def forward(self, input_ids, token_type_ids=None, attention_mask=None):
        cls_emb = self.model(input_ids=input_ids, token_type_ids=token_type_ids, attention_mask=attention_mask)
        cls_emb = cls_emb[0][:, 0, :].squeeze(1)
        logits = self.linear(cls_emb)
        # logits = self.softmax(logits)
        return logits

# train code
def train1():
    # data
    batch_size = 16
    tokenizer = BertTokenizer.from_pretrained(pretrained_path)
    dataset = BertDataset('../data/dataset/data.txt', tokenizer)
    train_len = int(len(dataset)*0.8)
    train_dataset, dev_dataset = random_split(dataset=dataset, lengths=[train_len, len(dataset)-train_len])
    train_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True, collate_fn=bert_collate_fn)
    dev_dataloader = DataLoader(dev_dataset, batch_size=batch_size, shuffle=True, collate_fn=bert_collate_fn)
    # model
    device = torch.device('cuda:{}'.format(args.cuda))
    bert_model = BertModel.from_pretrained(pretrained_path)
    model = PTModel(model=bert_model, n_class=len(dataset.label2id)).to(device)
    optimizer = torch.optim.Adam(params=model.parameters(), lr=args.lr)
    scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer=optimizer, milestones=[30, 40], gamma=0.1)
    loss_func = torch.nn.CrossEntropyLoss()
    # train
    for i in range(args.epoch):
        model.train()
        train_loss, dev_loss, f1_train, f1_dev = [], [], [], []
        dev_pred_list, dev_gold_list = [], []
        for input_ids, token_type_ids, attention_mask, label in tqdm(train_dataloader):
            input_ids, token_type_ids, attention_mask, label = input_ids.to(device), token_type_ids.to(device), \
                                                               attention_mask.to(device), label.to(device),
            outputs = model(input_ids=input_ids, token_type_ids=token_type_ids, attention_mask=attention_mask)
            array_outputs = np.array(outputs.cuda().data.cpu())
            optimizer.zero_grad()
            loss = loss_func(outputs, label)
            results = outputs.cuda().data.cpu().argmax(dim=1)
            score = f1_score(label.cuda().data.cpu(), results, average='micro')
            train_loss.append(loss.item())
            f1_train.append(score)
            # optim
            loss.backward()
            optimizer.step()
        scheduler.step()
        print('epoch {}'.format(i))
        print('train_loss:{}'.format(np.mean(train_loss)))
        print('train_f1:{}'.format(np.mean(f1_train)))

The train log is following(only 10 epoches). And the result was already clear: The model could not learn anything!!!
PS: the learning rate  was 1e-3.
100%|█████████████████████████████████████████| 250/250 [00:43<00:00,  5.72it/s]
epoch 0
train_loss:4.217772917747498
train_f1:0.081
100%|███████████████████████████████████████████| 63/63 [00:03<00:00, 19.52it/s]
dev_f1:0.08928571428571429
dev_loss:4.111690880760314
100%|█████████████████████████████████████████| 250/250 [00:43<00:00,  5.71it/s]
epoch 1
train_loss:4.094675525665283
train_f1:0.084
100%|███████████████████████████████████████████| 63/63 [00:03<00:00, 19.16it/s]
dev_f1:0.0882936507936508
dev_loss:4.1316274839734275
100%|█████████████████████████████████████████| 250/250 [00:43<00:00,  5.71it/s]
epoch 2
train_loss:4.084259546279907
train_f1:0.08525
100%|███████████████████████████████████████████| 63/63 [00:03<00:00, 19.37it/s]
dev_f1:0.08928571428571429
dev_loss:4.108004717599778
100%|█████████████████████████████████████████| 250/250 [00:44<00:00,  5.62it/s]
epoch 3
train_loss:4.0770455904006955
train_f1:0.09425
100%|███████████████████████████████████████████| 63/63 [00:03<00:00, 19.07it/s]
dev_f1:0.08928571428571429
dev_loss:4.1077501395392035
100%|█████████████████████████████████████████| 250/250 [00:45<00:00,  5.54it/s]
epoch 4
train_loss:4.070150758743286
train_f1:0.086
100%|███████████████████████████████████████████| 63/63 [00:03<00:00, 19.41it/s]
dev_f1:0.09027777777777778
dev_loss:4.103204295748756
100%|█████████████████████████████████████████| 250/250 [00:45<00:00,  5.52it/s]
epoch 5
train_loss:4.064209712982178
train_f1:0.0895
100%|███████████████████████████████████████████| 63/63 [00:03<00:00, 19.31it/s]
dev_f1:0.08928571428571429
dev_loss:4.117827377622089
100%|█████████████████████████████████████████| 250/250 [00:43<00:00,  5.70it/s]
epoch 6
train_loss:4.065111406326294
train_f1:0.08425
100%|███████████████████████████████████████████| 63/63 [00:03<00:00, 19.34it/s]
dev_f1:0.0882936507936508
dev_loss:4.099656305615864
100%|█████████████████████████████████████████| 250/250 [00:44<00:00,  5.58it/s]
epoch 7
train_loss:4.0547873935699466
train_f1:0.09175
100%|███████████████████████████████████████████| 63/63 [00:03<00:00, 19.30it/s]
dev_f1:0.08928571428571429
dev_loss:4.105985126798115
100%|█████████████████████████████████████████| 250/250 [00:43<00:00,  5.76it/s]
epoch 8
train_loss:4.0595885887145995
train_f1:0.08875
100%|███████████████████████████████████████████| 63/63 [00:03<00:00, 19.26it/s]
dev_f1:0.09027777777777778
dev_loss:4.121003010916332
100%|█████████████████████████████████████████| 250/250 [00:45<00:00,  5.46it/s]
epoch 9
train_loss:4.054850312232971
train_f1:0.08825
100%|███████████████████████████████████████████| 63/63 [00:03<00:00, 18.86it/s]
dev_f1:0.08928571428571429
dev_loss:4.12501887669639
100%|█████████████████████████████████████████| 250/250 [00:45<00:00,  5.46it/s]
epoch 10
train_loss:4.0566882238388065
train_f1:0.08525
100%|███████████████████████████████████████████| 63/63 [00:03<00:00, 18.85it/s]
dev_f1:0.09126984126984126
dev_loss:4.103033436669244

Before this BertModel, I have tried LSTM, and the LSTM worked well. the dev f1 reached 0.96.
# LSTM
class SimpleModel(nn.Module):
    def __init__(self, **kwargs):
        super(SimpleModel, self).__init__()
        self.embedding = nn.Embedding.from_pretrained(kwargs['pretrained_embedding'], freeze=False)
        self.lstm = nn.LSTM(kwargs['pretrained_embedding'].shape[1],
                            kwargs['hidden_size'],
                            batch_first=True,
                            bidirectional=True)
        self.linear = nn.Linear(kwargs['hidden_size']*2, kwargs['n_class'])

    def forward(self, inputs, lens):
        inputs = self.embedding(inputs)
        _, (h, _) = self.lstm(pack_padded_sequence(inputs, lens, batch_first=True, enforce_sorted=False))
        h = h.permute(1, 0, 2).contiguous().view(h.shape[1], -1)
        logits = self.linear(h)
        logits = logits.softmax(dim=-1)
        return logits

Could  any good man tell me why this code can’t work.
Is there something wrong with my writing?
I have been confused  for days…
Thank you very much!

Context: I think the problem might be that you call optimizer.zero_grad() after outputs are calculated, and it zeros out the gradients from the forward pass. Try putting that line before the line where outputs are calculated.


How to train from scratch with run_mlm.py, .txt file?

Question:
Hello! Essentially what I want to do is: point the code at a .txt file, and get a trained model out. How can I use run_mlm.py to do this? I’d be satisfied if someone could help me figure out how to even just recreate the EsperBERTo tutorial.
I’m getting bogged down in flags, trying to load tokenizers, errors, etc.
What I’ve done so far: I managed to run through the EsperBERTo tutorial (here 20), and now I’m trying to do the same thing with run_mlm.py.
I went to the examples at transformers/examples/pytorch/language-modeling at master · huggingface/transformers · GitHub 26 first, and I’ve been attempting to adapt those.

First of all, the section at transformers/examples/pytorch/language-modeling at master · huggingface/transformers · GitHub 18 seems the closest to EsperBERTo. So I started with the example for “your own training and validation files”
I wanted to train from scratch, so I edited it to use --model_type

I wanted to just use the oscar.eo.txt, so I omitted --validation_file and --do_eval

I set --max_seq_length 512 to match the tutorial.

I keep running into problems with the tokenizer.

run_mlm.py doesn’t let you not specify a tokenizer when you are training from scratch, apparently.
if you give it --model_type roberta and --tokenizer_name <path to the vocab.json and mergest.txt> it complains about not being able to find a config.json
if you then put a config.json for the Model, not the tokenizer in that folder, the error goes away. But why is it looking in the tokenizer folder for model config? And why does it need a model config at all? I told it model type?

What finally I did was:

made a folder called EsperBERTo. In there I put the vocab.json and merges.txt from the Colab tutorial, and a config.json that I found for RobertaForMaskedLM here 9

ran the following command:

python run_mlm.py \
    --model_type roberta \
    --tokenizer_name /path/to/EsperBERTo/ \
    --train_file /path/to/oscar.eo.txt \
    --max_seq_length 512 \
    --do_train \
    --output_dir ./output/test-mlm

But now I’m getting “IndexError: index out of range in self”. Some Googling lead me to believe this might be to do with the vocab size? I edited config.json to match the tokenizer (52000 vocab size), but no dice.
Here’s config.json
{
  "architectures": [
    "RobertaForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "eos_token_id": 2,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-05,
  "max_position_embeddings": 207,
  "model_type": "roberta",
  "num_attention_heads": 6,
  "num_hidden_layers": 3,
  "pad_token_id": 1,
  "type_vocab_size": 1,
  "vocab_size": 52000
}

At this point I’m thoroughly confused as to how to proceed, and I’m not even sure I’m going down the right path.
Can anyone give guidance on the proper process to go from .txt file to trained model in the command line? I’m not having much success transferring my knowledge (and my trained tokenizer) from Colab.

Context: You seem to be on the correct path, could you tell us more about the index error you encountered? What did the stack trace look like? Also could you try briefly with another model than roberta (like bert for instance) and report if the error disappears?


KeyError: ‘loss’ during Fine Tuning bert-base-italian-cased for QA

Question:
I was finetuning bert-base-italian-cased on SQuAD-it dateset with the following arguments
args = TrainingArguments(
    f"test-squad_it",
    evaluation_strategy = "epoch",
    learning_rate=2e-5,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    num_train_epochs=3,
    weight_decay=0.01,
    label_names = ["start_positions", "end_positions"]
)

trainer = Trainer(
    model,
    args,
    train_dataset=tokenized_datasets["train"],
    eval_dataset=tokenized_datasets["validation"],
    data_collator=data_collator,
    tokenizer=tokenizer,
)

Trying to do trainer.train() it suddenly throws this error:
KeyError                                  Traceback (most recent call last)
<ipython-input-168-4e078f57a6ea> in <module>()
      1 #We can now finetune our model by just calling the train method:
----> 2 trainer.train()

3 frames
/usr/local/lib/python3.7/dist-packages/transformers/trainer.py in train(self, resume_from_checkpoint, trial, **kwargs)
   1270                         tr_loss += self.training_step(model, inputs)
   1271                 else:
-> 1272                     tr_loss += self.training_step(model, inputs)
   1273                 self.current_flos += float(self.floating_point_ops(inputs))
   1274

/usr/local/lib/python3.7/dist-packages/transformers/trainer.py in training_step(self, model, inputs)
   1732                 loss = self.compute_loss(model, inputs)
   1733         else:
-> 1734             loss = self.compute_loss(model, inputs)
   1735
   1736         if self.args.n_gpu > 1:

/usr/local/lib/python3.7/dist-packages/transformers/trainer.py in compute_loss(self, model, inputs, return_outputs)
   1774         else:
   1775             # We don't use .loss here since the model may return tuples instead of ModelOutput.
-> 1776             loss = outputs["loss"] if isinstance(outputs, dict) else outputs[0]
   1777
   1778         return (loss, outputs) if return_outputs else loss

/usr/local/lib/python3.7/dist-packages/transformers/file_utils.py in __getitem__(self, k)
   1736         if isinstance(k, str):
   1737             inner_dict = {k: v for (k, v) in self.items()}
-> 1738             return inner_dict[k]
   1739         else:
   1740             return self.to_tuple()[k]

KeyError: 'loss'

I’ve already had a look on similar questions about this same issue, but nothing seems to work and I’m really desperate. Any suggestions?
EDIT
My data are taken from SQuAD-it. I’ve created this dataDict:
DatasetDict({
    train: Dataset({
        features: ['answer_text', 'answer_start', 'title', 'context', 'question', 'answers', 'id'],
        num_rows: 48328
    })
    validation: Dataset({
        features: ['answer_text', 'answer_start', 'title', 'context', 'question', 'answers', 'id'],
        num_rows: 5831
    })
})

And then I’ve preprocessed the data as follows (from a tutorial with little adjustments):
def prepare_train_features(examples):
    # Tokenize our examples with truncation and padding, but keep the overflows using a stride. This results
    # in one example possible giving several features when a context is long, each of those features having a
    # context that overlaps a bit the context of the previous feature.
    tokenized_examples = tokenizer(
        examples["question" if pad_on_right else "context"],
        examples["context" if pad_on_right else "question"],
        truncation="only_second" if pad_on_right else "only_first",
        max_length=max_length,
        stride=doc_stride,
        return_overflowing_tokens=True,
        return_offsets_mapping=True,
        padding="max_length",
    )

    # Since one example might give us several features if it has a long context, we need a map from a feature to
    # its corresponding example. This key gives us just that.
    sample_mapping = tokenized_examples.pop("overflow_to_sample_mapping")
    # The offset mappings will give us a map from token to character position in the original context. This will
    # help us compute the start_positions and end_positions.
    offset_mapping = tokenized_examples.pop("offset_mapping")

    # Let's label those examples!
    tokenized_examples["start_positions"] = []
    tokenized_examples["end_positions"] = []

    for i, offsets in enumerate(offset_mapping):
        # We will label impossible answers with the index of the CLS token.
        input_ids = tokenized_examples["input_ids"][i]
        cls_index = input_ids.index(tokenizer.cls_token_id)

        # Grab the sequence corresponding to that example (to know what is the context and what is the question).
        sequence_ids = tokenized_examples.sequence_ids(i)

        # One example can give several spans, this is the index of the example containing this span of text.
        sample_index = sample_mapping[i]
        answers = examples["answers"][sample_index]
        # If no answers are given, set the cls_index as answer.
        if answers["answer_start"] == 0:
            tokenized_examples["start_positions"].append(cls_index)
            tokenized_examples["end_positions"].append(cls_index)
        else:
            # Start/end character index of the answer in the text.
            start_char = answers["answer_start"]
            end_char = start_char + len(answers["text"])

            # Start token index of the current span in the text.
            token_start_index = 0
            while sequence_ids[token_start_index] != (1 if pad_on_right else 0):
                token_start_index += 1

            # End token index of the current span in the text.
            token_end_index = len(input_ids) - 1
            while sequence_ids[token_end_index] != (1 if pad_on_right else 0):
                token_end_index -= 1

            # Detect if the answer is out of the span (in which case this feature is labeled with the CLS index).
            if not (offsets[token_start_index][0] <= start_char and offsets[token_end_index][1] >= end_char):
                tokenized_examples["start_positions"].append(cls_index)
                tokenized_examples["end_positions"].append(cls_index)
            else:
                # Otherwise move the token_start_index and token_end_index to the two ends of the answer.
                # Note: we could go after the last offset if the answer is the last word (edge case).
                while token_start_index < len(offsets) and offsets[token_start_index][0] <= start_char:
                    token_start_index += 1
                tokenized_examples["start_positions"].append(token_start_index - 1)
                while offsets[token_end_index][1] >= end_char:
                    token_end_index -= 1
                tokenized_examples["end_positions"].append(token_end_index + 1)

    return tokenized_examples

tokenized_datasets = train_dataset.map(prepare_train_features, batched=True, remove_columns=train_dataset["train"].column_names)

I’m quite new in coding and maybe I’m relying too much in other people’s work without understanding every detail.

Context: How is your model created? How is your data processed? It’s hard to help debug the root of the error without seeing those.


Evaluating QA model on single SQuAD file

Question:
Hello guys
I would like to evaluate a model from the HF repo (‘mrm8488/bert-italian-finedtuned-squadv1-it-alfa’) on a SQuAD file I compiled, just to have a rough estimation of what could be the metrics.
This is the code i wrote:
from transformers import AutoTokenizer, AutoModelForQuestionAnswering, Trainer, TrainingArguments
import torch
from transformers import default_data_collator
import json

# Model from HuggingFace
model_checkpoint = 'mrm8488/bert-italian-finedtuned-squadv1-it-alfa'

# Import tokenizer
my_tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)

# Import model
my_model = AutoModelForQuestionAnswering.from_pretrained(model_checkpoint)

# Dataset for evaluation
eval_data_path = '/content/drive/MyDrive/BERT/SQuAD_files/result.json'

with open(eval_data_path) as json_file:
  data = json.load(json_file)

data_collator = default_data_collator
trainer = Trainer(
    my_model,
    data_collator=data_collator,
    tokenizer=my_tokenizer
)

trainer.evaluate(data)

Unfortunately, this does not work and I can’t understand why.
This is the error:
KeyError                                  Traceback (most recent call last)
<ipython-input-31-54109037e744> in <module>()
----> 1 trainer.evaluate(data)

5 frames
/usr/local/lib/python3.7/dist-packages/transformers/trainer.py in evaluate(self, eval_dataset, ignore_keys, metric_key_prefix)
   2006             prediction_loss_only=True if self.compute_metrics is None else None,
   2007             ignore_keys=ignore_keys,
-> 2008             metric_key_prefix=metric_key_prefix,
   2009         )
   2010

/usr/local/lib/python3.7/dist-packages/transformers/trainer.py in evaluation_loop(self, dataloader, description, prediction_loss_only, ignore_keys, metric_key_prefix)
   2145         observed_num_examples = 0
   2146         # Main evaluation loop
-> 2147         for step, inputs in enumerate(dataloader):
   2148             # Update the observed num examples
   2149             observed_batch_size = find_batch_size(inputs)

/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py in __next__(self)
    515             if self._sampler_iter is None:
    516                 self._reset()
--> 517             data = self._next_data()
    518             self._num_yielded += 1
    519             if self._dataset_kind == _DatasetKind.Iterable and \

/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py in _next_data(self)
    555     def _next_data(self):
    556         index = self._next_index()  # may raise StopIteration
--> 557         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    558         if self._pin_memory:
    559             data = _utils.pin_memory.pin_memory(data)

/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py in <listcomp>(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

KeyError: 0

A glimpse of my SQuAD formatted file:
{
  "data": [
    {
      "paragraphs": [
        {
          "qas": [
            {
              "question": "Qual è l’età?",
              "id": 78079,
              "answers": [
                {
                  "answer_id": 89658,
                  "document_id": 84480,
                  "question_id": 78079,
                  "text": "02/01/1966",
                  "answer_start": 113,
                  "answer_category": "SHORT"
                }
              ],
              "is_impossible": false
            },
            {
              "question": "Qual è il titolo di studio?",
              "id": 78082,
              "answers": [
                {
                  "answer_id": 89661,
                  "document_id": 84480,
                  "question_id": 78082,
                  "text": "media superiore",
                  "answer_start": 1157,
                  "answer_category": "SHORT"
                }
              ],
              "is_impossible": false
            },
    ...]
    "context" = "..."
    "document_id" = "..."

What am I doing wrong?
PS: is there a tool from the HF libraries to annotate files for question answering in a smart way?
Thanks a lot

Context: The Trainer requires processed data. Have a look at the run_qa examples to see how it can be done,


I got ‘ValueError: You have to specify either input_ids or inputs_embeds’ when I am training GPT2 using huggingface Trainer

Question:
Below is the error code generated
ValueError                                Traceback (most recent call last)
<ipython-input-38-29d47e6260b2> in <module>()
----> 1 trainer.train( )

4 frames
/usr/local/lib/python3.7/dist-packages/transformers/models/gpt2/modeling_gpt2.py in forward(self, input_ids, past_key_values, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, use_cache, output_attentions, output_hidden_states, return_dict)
    674             batch_size = inputs_embeds.shape[0]
    675         else:
--> 676             raise ValueError("You have to specify either input_ids or inputs_embeds")
    677
    678         device = input_ids.device if input_ids is not None else inputs_embeds.device

and here is my code
from transformers import GPT2Tokenizer, GPT2Model, Trainer, trainer_utils, TrainingArguments
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2Model.from_pretrained('gpt2')

from datasets import Dataset
dataset = Dataset.from_text('/content/chatbot.txt')

trainArgs = TrainingArguments( output_dir = os.path.join( os.getcwd(), 'customGPT2' )
                              , overwrite_output_dir = True
                              , do_train=True
                              , do_eval=True
                              , evaluation_strategy='steps'
                              , per_device_train_batch_size=4
                              , per_device_eval_batch_size =4
                              , gradient_accumulation_steps=1
                              , eval_accumulation_steps=1
                              , weight_decay=0
                              , adam_epsilon= 1e-08
                              , max_grad_norm = 1.0
                              , num_train_epochs =3.0
                              , max_steps = -1
                              , lr_scheduler_type = trainer_utils.SchedulerType('linear')
                              , logging_dir = os.path.join( os.getcwd(), 'log' )
                              , logging_steps = 2000
                              , logging_strategy = 'steps'
                              , save_steps = 2000
                              , save_strategy = 'steps'
                              , seed = 66
                              , fp16 = False
                              , fp16_opt_level = 'O1')

trainer=Trainer( model, args = trainArgs,train_dataset=dataset)

As I am quite new using trainer I actually try to follow as closely to docs as possible and only change few thing such as dataset because I need to use local dataset but I make sure to import it to datasets.Dataset() to be the same as the format that docs require and changing some training arguments. Thank you

Context: You haven’t processed your dataset: it only contains the raw texts and not the input IDs the model expects. Have a look at the training tutorial to see how you can tokenize it!


Question:
Hi, may I get some help resetting my API key? I might have leaked mine. Thanks! @julien-c @pierric
I followed a previous post: How can I renew my API key 6

Context: @r3dhummingbird Your api token was successfully renewed.


Question:
I tried to run notebook facebook/wav2vec2-large-xlsr-53 · Hugging Face
But when I run
model = Wav2Vec2ForCTC.from_pretrained(
    "facebook/wav2vec2-base-960h",
    attention_dropout=0.1,
    hidden_dropout=0.1,
    feat_proj_dropout=0.0,
    mask_time_prob=0.05,
    layerdrop=0.1,
    gradient_checkpointing=True,
    ctc_loss_reduction="mean",
    pad_token_id=processor.tokenizer.pad_token_id,
    vocab_size=len(processor.tokenizer)
)

I get this error
RuntimeError: version_ <= kMaxSupportedFileFormatVersion INTERNAL ASSERT FAILED at ../caffe2/serialize/inline_container.cc:132, please report a bug to PyTorch. Attempted to read a PyTorch file with version 3, but the maximum supported version for reading is 2. Your PyTorch installation may be too old. (init at ../caffe2/serialize/inline_container.cc:132)
frame #0: c10::Error::Error(c10::SourceLocation, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) + 135 (0x14e2f6787 in libc10.dylib)
frame #1: caffe2::serialize::PyTorchStreamReader::init() + 2350 (0x13795414e in libtorch.dylib)
frame #2: caffe2::serialize::PyTorchStreamReader::PyTorchStreamReader(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) + 143 (0x13795379f in libtorch.dylib)
frame #3: void pybind11::cpp_function::initialize<void pybind11::detail::initimpl::constructor<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >::execute<pybind11::class_<caffe2::serialize::PyTorchStreamReader>, 0>(pybind11::class_<caffe2::serialize::PyTorchStreamReader>&)::'lambda'(pybind11::detail::value_and_holder&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >), void, pybind11::detail::value_and_holder&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor>(pybind11::class_<caffe2::serialize::PyTorchStreamReader>&&,  (*)(0...), void pybind11::detail::initimpl::constructor<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >::execute<pybind11::class_<caffe2::serialize::PyTorchStreamReader>, 0>(pybind11::class_<caffe2::serialize::PyTorchStreamReader>&)::'lambda'(pybind11::detail::value_and_holder&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) const&...)::'lambda'(pybind11::detail::function_call&)::operator()(pybind11::detail::function_call&) const + 147 (0x1346277c3 in libtorch_python.dylib)
frame #4: pybind11::cpp_function::dispatcher(_object*, _object*, _object*) + 3382 (0x13401fe66 in libtorch_python.dylib)
<omitting python frames>

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
<ipython-input-108-5337759b7278> in <module>
      1 from transformers import Wav2Vec2ForCTC
      2
----> 3 model = Wav2Vec2ForCTC.from_pretrained(
      4     #"facebook/wav2vec2-base-960h",
      5     "facebook/wav2vec2-large-xlsr-53",

/opt/anaconda3/lib/python3.8/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   1062                 state_dict = torch.load(resolved_archive_file, map_location="cpu")
   1063             except Exception:
-> 1064                 raise OSError(
   1065                     f"Unable to load weights from pytorch checkpoint file for '{pretrained_model_name_or_path}' "
   1066                     f"at '{resolved_archive_file}'"

OSError: Unable to load weights from pytorch checkpoint file for 'facebook/wav2vec2-large-xlsr-53' at '/Users/elizavetatrusova/.cache/huggingface/transformers/5d2a20b45a1689a376ec4a6282b9d9be42f931cdf8daf07c3668ba1070a059d9.db2a69eb44bf7b1efcfff155d4cc22155230bd8c0941701b064e9c17429a623d'If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.

I run it in Jupyter, what can I do with it?

Context: Perhaps it’s a permissions error?


How to see BERT,BART… output dimensions?

Question: how can i see the output dimensions for BERT Large,BART Large,RoBERTa Large, BART Large CNN, XLM RoBERTa Large?

Context: really no one knows?!


Not able to import MBart50TokenizerFast from transformers

Question:
Hi I am not able to import MBart50TokenizerFast from transformers. Below is the error that it is giving
In [6]: from transformers import MBartForConditionalGeneration, MBart50TokenizerFast
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-6-6bac02268080> in <module>
----> 1 from transformers import MBartForConditionalGeneration, MBart50TokenizerFast

ImportError: cannot import name 'MBart50TokenizerFast' from 'transformers' (unknown location)

Below are the transformers and torch versions
 $ pip freeze | grep transformers
transformers==4.3.2
$ pip freeze | grep torch
torch==1.7.1

Context: AFAIK this model+tokenizer is not part of a full release yet. You’ll have to install from the master branch to use it.


Question: I understand that there are a few different tokenizers; e.g., DistilBertTokenizerFast, etc. However, I don’t understand the concept of “Auto” in tokenizer selection. Using IMDB text as an example. What do I get if I use AutoTokenizer to tokenize the text?

Context: The AutoTokenizer will work on any checkpoint and pick the proper architecture for you (whereas DistilBNertTokenizerFast will only work for distilbert checkpoints).


Question:
Hi,
I’m trying to apply wave2vec2 models on long audiofiles (~1h) for speech to text.
However processing the entire audio file at once is not feasible because it requires more than 16GB. How can I import a sound file as audio stream into the wave2vec models?

Context:
Here is one way to do this with librosa.stream:

      github.com/huggingface/transformers

      can't allocate memory error with wav2vec2 106

        opened

        Feb 24, 2021

          closed

        Apr 23, 2021

          kleekaai

    I am trying out the wav2vec2 model for ASR from the huggingface library. Here, I… am passing a 7 min(~15 MB file) long wav file having a conversation(english) to the wav2vec2 model. I am getting "can't allocate memory" error. I found that the model uses all 64 GB of the available RAM. Can anyone help with this.

- `transformers` version: 4.3.2
- Platform: Linux-3.10.0-1127.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.3
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: (NA)
- Using distributed or parallel set-up in script?: (NA)

Code
```
import os
import librosa
import soundfile as sf
from pydub import AudioSegment

def convert_audio_segment(fp, upload_dir_path):
    """Convert audio file"""

    USER_UPLOAD_DIR = upload_dir_path
    formats_to_convert = ['.m4a']
    dirpath = os.path.abspath(USER_UPLOAD_DIR)

    if fp.endswith(tuple(formats_to_convert)):

        (path, file_extension) = os.path.splitext(fp)
        file_extension_final = file_extension.replace('.', '')
        file_handle = ''

        try:
            track = AudioSegment.from_file(fp,
                    file_extension_final)
            print("track", track)
            wav_path = fp.replace(file_extension_final, 'wav')
            file_handle = track.export(wav_path, format='wav')
        except Exception:
            print("ERROR CONVERTING " + str(fp))
        return file_handle
    else:
        print("No file format conversion required " + str(fp))
        return fp

def load_wav2vec_100h_model():
    tokenizer = Wav2Vec2Tokenizer.from_pretrained("facebook/wav2vec2-base-100h")
    model = Wav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-base-100h")
    return tokenizer, model

def correct_sentence(input_text):
    sentences = nltk.sent_tokenize(input_text)
    return (' '.join([s.replace(s[0],s[0].capitalize(),1) for s in sentences]))

def asr_transcript(tokenizer, model, input_file):

    speech, fs = sf.read(input_file)

    if len(speech.shape) > 1:
        speech = speech[:,0] + speech[:,1]

    if fs != 16000:
        speech = librosa.resample(speech, fs, 16000)

    input_values = tokenizer(speech, return_tensors="pt").input_values
    logits = model(input_values).logits

    predicted_ids = torch.argmax(logits, dim=-1)

    transcription = tokenizer.decode(predicted_ids[0])

    return correct_sentence(transcription.lower())

if __name__ == "__main__":

    tokenizer_100h, model_100h = load_wav2vec_100h_model()
    wav_input = 'Recording_biweu.wav'
    fp = wav_input

    processed_file = convert_audio_segment(str(fp), str(data_dir))
    text = asr_transcript(tokenizer_100h,model_100h,processed_file)
    print(text)
```
I am adding more details about my wav file here
```
General
Complete name                            : Recording_biweu.wav
Format                                   : Wave
File size                                : 13.8 MiB
Duration                                 : 7 min 30 s
Overall bit rate mode                    : Constant
Overall bit rate                         : 256 kb/s
Track name                               : Recording_biweu
Recorded date                            : 2021
Writing application                      : Lavf57.83.100

Audio
Format                                   : PCM
Format settings                          : Little / Signed
Codec ID                                 : 1
Duration                                 : 7 min 30 s
Bit rate mode                            : Constant
Bit rate                                 : 256 kb/s
Channel(s)                               : 1 channel
Sampling rate                            : 16.0 kHz
Bit depth                                : 16 bits
Stream size                              : 13.8 MiB (100%)
```

Error
```
Some weights of the model checkpoint at facebook/wav2vec2-base-100h were not used when initializing Wav2Vec2ForCTC: ['wav2vec2.mask_time_emb_vector']
- This IS expected if you are initializing Wav2Vec2ForCTC from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing Wav2Vec2ForCTC from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Traceback (most recent call last):
  File "asr_wav2vec2.py", line 130, in <module>
    text = asr_transcript(tokenizer_100h,model_100h,processed_file)
  File "asr_wav2vec2.py", line 96, in asr_transcript
    logits = model(input_values).logits
  File "/home/joel/pyvenv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/joel/pyvenv/lib/python3.8/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 795, in forward
    outputs = self.wav2vec2(
  File "/home/joel/pyvenv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/joel/pyvenv/lib/python3.8/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 646, in forward
    encoder_outputs = self.encoder(
  File "/home/joel/pyvenv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/joel/pyvenv/lib/python3.8/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 457, in forward
    hidden_states, attn_weights = layer(hidden_states, output_attentions=output_attentions)
  File "/home/joel/pyvenv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/joel/pyvenv/lib/python3.8/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 392, in forward
    hidden_states, attn_weights, _ = self.attention(hidden_states, output_attentions=output_attentions)
  File "/home/joel/pyvenv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/joel/pyvenv/lib/python3.8/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 286, in forward
    attn_weights = torch.bmm(query_states, key_states.transpose(1, 2))
RuntimeError: [enforce fail at CPUAllocator.cpp:65] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 24373495488 bytes. Error code 12 (Cannot allocate memory)
```


Question:
Hello to all,
Looking at tutorials and examples I see that many do fine tuning with run_lm_finetuning.py. However when I install the latest version of transformers this file doesn’t exist, I haven’t found it in the github repository either.
So my question is if this file was discontinued, and, if so, what is the actual .py file that should be used for fine tuning of a language model?

Context:
Hi,
It’s now here: https://github.com/huggingface/transformers/tree/master/examples/language-modeling 960


Bug Report: Mask token mismatch with the model on hosted inference API of Model Hub

Question:
In my model card, I used to be able to run the hosted inference successfully, but recently it prompted an error: "<mask>" must be present in your input.
My model uses RoBERTa MLM and BERT Tokenizer. So the mask token is actually “[MASK]”. I have already set it in tokenizer_confg.json but the inference API still mismatches with that.
In the past it is OK but recently it turns to prompt an error. Seems like the front-end start to double-check the mask token. How can I set the mask token in an appropriate way? Is it documented to set mask token in inference API?
Thanks!

To reproduce
Steps to reproduce the behavior:

Go to ethanyt/guwenbert-base · Hugging Face 2

Run an example with “[MASK]”

Expected behavior
In the past it was OK. See snapshot in  guwenbert/README_EN.md at main · Ethan-yt/guwenbert · GitHub 1

Context: Should be resolved in Mask token mismatch with the model on hosted inference API of Model Hub · Issue #11884 · huggingface/transformers · GitHub 8, but if possible do not open duplicate issues/forum posts. Thanks!


Application of a transformer model without fine tuning for NER task

Question:
I am interested in using pre-trained models from Huggingface for named entity recognition (NER) tasks without any further training or testing of the model. The only information on Huggingface for the model are to use the following lines:
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")
model = AutoModel.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")

I tried the following code, but I am getting a tensor output instead of class labels for each named entity.
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")
model = AutoModel.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")

text = "my text for named entity recognition here."

input_ids = torch.tensor(tokenizer.encode(text, padding=True, truncation=True,max_length=50, add_special_tokens = True)).unsqueeze(0)

with torch.no_grad():
  output = model(input_ids, output_attentions=True)

Can anyone suggest what am I doing wrong here? It will be good to have a short tutorial on how to use a pre-trained model for NER (without any fine tuning).

Context:
hey @shoaibb, my suggestion would be to use the ner pipeline (see docs 6) to extract the named entities because this is quite tricky to get right.
also, emilyalsentzer/Bio_ClinicalBERT appears to be a pretrained language model so you’ll need to either fine-tune it on a labelled NER dataset or see if you can find an existing model on the Hub that suits your needs. you can find an example on fine-tuning NER models here: Google Colaboratory 5


Question:
Hello,
I’m trying to implement  NER with BioBERT.
from transformers import AutoTokenizer, AutoModelForTokenClassification, pipeline

tokenizer = AutoTokenizer.from_pretrained("dmis-lab/biobert-v1.1")
model = AutoModelForTokenClassification.from_pretrained("dmis-lab/biobert-v1.1")

nlp = pipeline("ner", model=model, tokenizer=tokenizer)
sentence = "This expression of NT-3 in supporting cells in embryos and neonates may even preserve in Brn3c null mutants the numerous spiral sensory neurons in the apex of 8-day old animals."

result = nlp(sentence)
print(result)

But the result isn’t what I’m expecting.
Some weights of BertForTokenClassification were not initialized from the model checkpoint at dmis-lab/biobert-v1.1 and are newly initialized: ['classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
[{'word': 'This', 'score': 0.5616263747215271, 'entity': 'LABEL_1', 'index': 1, 'start': 0, 'end': 4}, {'word': 'expression', 'score': 0.6285454630851746, 'entity': 'LABEL_1', 'index': 2,

The output is pretty clear : I need to train the model.
But, I’m not sure if with a trained model, I will manage to get rid off the ‘entity’: ‘LABEL_1’ issue.
My desired output would be something like:
https://bern.korea.ac.kr/ 14
With a complete response such as:
{
    "project": "BERN",
    "sourcedb": "",
    "sourceid": "43c1bfdebd3ccb8c9a42d10a22a3be3e8b2fe9ae7601b244b6318d71-Thread-18603546",
    "text": "This expression of NT-3 in supporting cells in embryos and neonates may even preserve in Brn3c null mutants the numerous spiral sensory neurons in the apex of 8-day old animals.",
    "denotations": [
        {
            "id": [
                "HGNC:8020",
                "BERN:324182202"
            ],
            "span": {
                "begin": 19,
                "end": 23
            },
            "obj": "gene"
        },
        {
            "id": [
                "MIM:602460",
                "HGNC:9220",
                "Ensembl:ENSG00000091010",
                "BERN:324351702"
            ],
            "span": {
                "begin": 89,
                "end": 94
            },
            "obj": "gene"
        }
    ],
    "timestamp": "Thu May 27 08:22:14 +0000 2021",
    "logits": {
        "disease": [],
        "gene": [
            [
                {
                    "start": 19,
                    "end": 23,
                    "id": "HGNC:8020\tBERN:324182202"
                },
                0.9999972581863403
            ],
            [
                {
                    "start": 89,
                    "end": 94,
                    "id": "MIM:602460\tHGNC:9220\tEnsembl:ENSG00000091010\tBERN:324351702"
                },
                0.9999972581863403
            ]
        ],
        "drug": [],
        "species": []
    }
}

Am I in the right path to achieve that?
Any help/suggestion is more than welcome!
Cheers,
Vivian

Context: @Vivian Did you get any success with this issue? I am also in a similar situation.


Question:
I am using bert for next-sentence prediction on a cpu. I want to call the model twice in a row to select a sentence from a list of sentence pairs. I am using a batch size of 128. My code looks something like this:
def bert_batch_compare(self, prompt1, prompt2):
    encoding = self.tokenizer(prompt1, prompt2, return_tensors='pt', padding=True, truncation=True, add_special_tokens=True)
    target = torch.ones((1,len(prompt1)), dtype=torch.long)
    outputs = self.model(**encoding, next_sentence_label=target)
    logits = outputs.logits.detach()
    return logits

def call_bert(self):
    batch_pattern = []
    batch_template = []
    batch_input = []
    ## make batch_pattern, batch_input, patch_template here!
    si = self.bert_batch_compare(batch_pattern, batch_input)
    ## second call to this fn is killed because of memory limitations
    sj = self.bert_batch_compare(batch_input, batch_template)

If I lower the batch size to something like 24 it runs, but I’d like to use a larger batch size. I am not doing any training right now. I’m using ‘bert-base-uncased’. During the second call to ‘bert_batch_compare()’ the memory usage increases to 100% and the program crashes. I have 16G to work with. Until that time the code only uses 1.8Gig. I am using linux and python 3.6, along with pytorch 1.8.

Context:
Might not be a memory leak but a case of larger batch padding.
If the longest sequence in the first batch is 80 tokens, then that batch will (likely) be padded to 80 items and that may fit into memory. But if the longest sequence in the next batch then contains 250 tokens, then the whole batch is padded to 250 and that might not fit into memory. So verify the length of each individual sample to be sure.


Enhance a MarianMT pretrained model from HuggingFace with more training data

Question:
I am using a pretrained MarianMT machine translation model from English to German 5. I also have a large set of high quality English-to-German sentence pairs that I would like to use to enhance the performance of the model, which is trained on the OPUS corpus, but without making the model forget the OPUS training data. Is there a way to do that? Thanks.
Also on StackOverflow 17

Context:
You could further fine-tune it on your own corpus, and I think if you have a high quality dataset then it should improve the results after fine-tuning.
You can use the finetune.py script from here 129 for fine-tuning marian


Question: Hello. I have a question about a general understanding of how transformers work. The tokenizer feeds the model text, which is usually normalized, cleared of punctuation, and so on. But as I assume the transformer is trained on a raw corpus - I made this conclusion after seeing the single characters in the vocabulary. Hence the question - should I normalize and do other preprocessing of the sentences for which I want to get embbedings? If the model was trained on the raw corpus, how correct will the preprocessing described above be for the text under study?

Context:
No, you should not preprocess the dataset. Depending on the tokenizer scheme, you may want to tokenize words beforehand (which will then still be tokenised into subword units). For something like sentencepiece that is not needed (in fact it is recommended to not pretokenise in sentencepiece because a space is a regular character there).
It is best if the data is “clean” though, in the sense that it should not contain HTML/XML tags or other longer sequences of strange characters. But that should then also be true of inference data.


Use wav2vec2 models with a microphone easily

Question:
Hello folks,
I wrote a little lib to be able to use any wav2vec2 model from the model hub with a microphone. Since wav2vec2 does not support streaming mode, I used voice activity detection to create audio chunks that I can feed into the model.
Here is a little example, you can find the code on github 109.
from live_asr import LiveWav2Vec2

german_model = "maxidl/wav2vec2-large-xlsr-german"
asr = LiveWav2Vec2(german_model,device_name="default")
asr.start()

try:
    while True:
        text,sample_length,inference_time = asr.get_last_text()
        print(f"{sample_length:.3f}s"
        +f"\t{inference_time:.3f}s"
        +f"\t{text}")

except KeyboardInterrupt:
    asr.stop()

If you have any questions or feedback feel free to write me.

Context:
Pretty cool! would you consider making an implementation for Google Colab / notebooks? Similar to this:

      github.com

    voidful/huggingface_notebook/blob/main/xlsr_gpt.ipynb 23

    {
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "xlsr tw.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZXXYTItj9pDQ"

  This file has been truncated. show original

But with the VAD to get a near real-time transcription.


Question:
Loving this model, and if we can use the API we’re happy to do so.  Using Pegasus summarization for some testing.
human-centered-summarization/financial-summarization-pegasus · Hugging Face 3
However, there seem to be no options to pass via POST to determine a minimum and maximum length, which renders this endpoint useless.
Am I missing something here?  Thanks, happy to be a new subscriber.

Context:
Hi @induveca ,
Why would you say the endpoint is useless without being able to specify min_length and max_length ? Models are conditioned to summarize, so the output should work.
Anyway, you can still specify min_length and max_length even if these arguments are not documented (they are not documented which means behavior is subject to change in the future).  They are specified in tokens length, which is really hard to gauge if you are only handling text.
If you are using those, expect that the output might be cut mid sentence as you are basically forcing the model to stop outputting tokens even if it had still tokens to say.
Cheers,
Nicolas


Which model to use for suggesting article to the user based on details provided?

Question:
Hello Members,
I have a use case in which the user enters the details of the problem his computer is facing and the system will return the most relevant article from the list of articles that potentially might solve the issue.
I tried cosine similarity and BM25 but not getting decent results. Can someone please suggest which pre-trained model i can use for this kind of data and use-case.
Sorry I am new to transformers.

Context:
hey @gladmortal your use case sounds like a good match for dense retrieval, where you use a transformer like BERT to embed your documents as dense vectors and then measure their similarity to a query vector.
you can find an example on how to do this with FAISS and the datasets library here: Adding a FAISS or Elastic Search index to a Dataset — datasets 1.6.2 documentation 4
there is also a nice example from sentence-transformers here if your articles are not too long: Semantic Search — Sentence-Transformers documentation 3
alternatively, if you have a corpus of (question, article) tuples, you could try doing a similarity comparison of new questions against the existing ones and using the matches to return the most relevant article. there’s a tutorial of this from a nice library called haystack which is built on transformers here: https://haystack.deepset.ai/docs/latest/tutorial4md 2


Extra Dimension with DataCollatorFor LanguageModeling into BertForMaskedLM?

Question:
Hi all,
EDIT: I forgot to state that I am on transformers 4.6.1 and python 3.7
On Colab, I am trying to pre-train a BertforMaskedLM using a random subset of half of Wikitext-103. I am using a simple custom dataset class and the DataCollatorForLanguageModeling as follows.
import torch
import torchtext
from torch.utils.data import Dataset, DataLoader
import torch.nn as nn
import torch.optim as optim
import torch.nn as nn
import torch.optim as optim
import re
import random

from transformers import BertForMaskedLM, BertModel, BertConfig, BertTokenizer
from transformers import DataCollatorForLanguageModeling
from transformers import Trainer, TrainingArguments
from transformers import PreTrainedTokenizer

wiki_train, wiki_valid, wiki_test = torchtext.datasets.WikiText103(root='data',
                                                                   split=('train','valid','test'))
def scrub_titles_get_lines(dataset):
    pattern = " =+.+ =+"
    pattern = re.compile(pattern)
    title_scrubbed = []
    for example in dataset:
        if not example.isspace() and not bool(pattern.match(example)):
            title_scrubbed.append(example)
    return title_scrubbed

class LineByLineBertDataset(torch.utils.data.Dataset):
    def __init__(self, data, tokenizer: PreTrainedTokenizer,  max_len=512):
        self.examples = data
        self.tokenizer = tokenizer
        self.max_length = max_len

    def __len__(self):
        return len(self.examples)

    def __getitem__(self, i):
        result = self.tokenizer(self.examples[i],
                                add_special_tokens=True,
                                truncation=True,
                                return_special_tokens_mask=True,
                                padding='max_length',
                                max_length=self.max_length,
                                return_tensors='pt')
        return result

configuration = BertConfig()
model = BertForMaskedLM(configuration)

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
wiki_train = random.sample(wiki_train, len(wiki_train)//2)  # list of strings
train_set = LineByLineBertDataset(wiki_train, tokenizer)

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=True, mlm_probability=0.15
)

training_args = TrainingArguments(
    output_dir="/content/drive/MyDrive/BERT_TEST",
    overwrite_output_dir=True,
    num_train_epochs=1,
    save_steps=10_000,
    save_total_limit=2,
    prediction_loss_only=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_set,
)

trainer.train()

However, I get an error in the forward() method of the model:
/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)
     923         elif input_ids is not None:
      924             input_shape = input_ids.size()
--> 925             batch_size, seq_length = input_shape
      926         elif inputs_embeds is not None:
      927             input_shape = inputs_embeds.size()[:-1]

ValueError: too many values to unpack (expected 2)
Each of the tensors in the batch encoding are of shape (8,512)
In the DataCollatorForMaskedLM I know that at some point another dimension gets added. If I do:
res = tokenizer(wiki_train[:8],
        add_special_tokens=True,
        return_special_tokens_mask=True,
        truncation=True,
        padding='max_length',
        max_length=512,
        return_tensors='pt')

collated = data_collator([res])
collated['input_ids'].size()

Output: torch.Size([1, 8, 512])
So it seems that maybe this first dimension needs to be squeezed out. However, I am not sure what parameter I can tweak to ensure that the correct tensor is being seen by the model after collation.
Any thoughts?

Context:
No the dimension was added by you when you passed
collated = data_collator([res])

res was already a list with 8 elements here, by putting in a new list you add the 1.


Zero-Shot Classification Pipeline - Truncating

Question:
Hi,
Is it possible to specify arguments for truncating and padding the text input to a certain length when using the transformers pipeline for zero-shot classification?
For instance, if I am using the following:
classifier = pipeline(“zero-shot-classification”, device=0)
Do I need to first specify those arguments such as truncation=True, padding=‘max_length’, max_length=256, etc in the tokenizer / config, and then pass it to the pipeline?
Thank you in advance

Context:
hey @valkyrie the pipelines in transformers call a _parse_and_tokenize function that automatically takes care of padding and truncation - see here 10 for the zero-shot example.
so the short answer is that you shouldn’t need to provide these arguments when using the pipeline. do you have a special reason to want to do so?


Correct interpretation of the model embbedings output

Question:
Hello. Using DeepPavlov transformer I was surprised to get different embbedings for the same word ‘шагать’. This is a fictional example showing the essence of the question.
MODEL_NAME = 'DeepPavlov/rubert-base-cased-sentence'
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModel.from_pretrained(MODEL_NAME)

model(**tokenizer('шагать шагать', return_tensors='pt', truncation=True, max_length=512)).last_hidden_state.detach().squeeze()

As I can see the tokenizer splits word ‘шагать’ on two tokens: ‘шага’ and ‘##ть’
Output for embbedings is:
tensor([[-0.5780,  0.0937, -0.3210,  …, -0.3401,  0.0203,  0.4830],

   [-0.6516,  0.0278, -0.3610,  ..., -0.4095,  0.0527,  0.5094],*

   [-0.6018,  0.1147, -0.2739,  ..., -0.4194,  0.0580,  0.4853],*

   [-0.6632,  0.0110, -0.3995,  ..., -0.3953,  0.0823,  0.4497],*

   [-0.6711,  0.1017, -0.2829,  ..., -0.3797,  0.0994,  0.4285],*

   [-0.6337,  0.0572, -0.3519,  ..., -0.3553,  0.0126,  0.4479]])*

I have expected that vector 1([-0.6516,  0.0278, -0.3610,  …, -0.4095,  0.0527,  0.5094] - I guess it corresponds to ‘шага’) will be equal to vector 3 - but I see other values. Same is true and for pair vectors 2,4 (’##ть’).
I guess it is result of my mismatching of  how model works. Please explain me wht is wrong in my undestanding…

Context:
hey @Roman, the reason why you don’t get the same embedding for the same word in a sequence is because the transformers like BERT produce context-sensitive representations, i.e. they depend on the context in which they appear in a sequence.
the advantage of such representations is that you can deal with tricky examples like “Time flies like an arrow; fruit flies like a banana”, where the word “flies” has two very different meanings
you can find a nice description of contextual embeddings here: The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar – Visualizing machine learning one concept at a time. 1


Inconsistent Bleu score between test_metrics[‘test_bleu’] and written-to-file test_metric.predictions

Question:
I got a bleu score at about 11 and would like to do some error analysis, so I saved the predictions to file. When I read the predictions, I felt that the bleu score should be much lower than 11 because most  tokens in the references are missing in the predictions. Therefore, I directly calculated the bleu score by giving the predictions file and references file to sacrebleu (which is the package used as metric in the training program) and the bleu score is about 2. The predictions and references files are both formatted one sentence a line. Each predicted sentence has only one reference.
Relevant code snippets are attached below:

import sacrebleu
metric = load_metric("sacrebleu")

#----------------------------------------------------------#
# Define compute_metrics for trainer
#----------------------------------------------------------#

def postprocess_text(preds, labels):
    preds = [pred.strip() for pred in preds]
    labels = [[label.strip()] for label in labels]

    return preds, labels

def compute_metrics(eval_preds):
    preds, labels = eval_preds
    if isinstance(preds, tuple):
        preds = preds[0]
    decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)

    # Replace -100 in the labels as we can't decode them.
    labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
    decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

    # Some simple post-processing
    decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)

    result = metric.compute(predictions=decoded_preds, references=decoded_labels)
    result = {"bleu": result["score"]}

    prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds]
    result["gen_len"] = np.mean(prediction_lens)
    result = {k: round(v, 4) for k, v in result.items()}
    return result

#----------------------------------------------------------#
# Calculate metric for test dataset, get bleu score, and save predictions to file
#----------------------------------------------------------#
test_metric = trainer.predict(test_dataset = tokenized_datasets['test'], metric_key_prefix = 'test', num_beams=6)
print(test_metric.metrics['test_bleu']) # get about 11

detokenized_predictions = tokenizer.batch_decode(test_metric.predictions, skip_special_tokens=True)
with open(path_predictions_file, 'w') as outfile:
    s = '\n'.join(detokenized_predictions)
    outfile.write(s)

#----------------------------------------------------------#
# load previously-saved predictions files and references file to calculate bleu score
#----------------------------------------------------------#
predictions = []
with open(path_predictions_file) as prediction_infile:
    for sentence in prediction_infile:
        predictions.append(sentence.strip())
references = []
with open(path_references_file) as reference_infile:
    for sentence in reference_infile:
        references.append(sentence.strip())

bleu = sacrebleu.corpus_bleu(predictions, [references])
print('{}'.format(bleu.format(score_only=True))) #get about 2

Thank you very much for the reading! Really appreciate any suggestions

Context: The post-processing in the first example is not applied to the predictions written to disk.


Question:
hey.
do you know what this model is trained on and what task is it fine tuned for? (presume aspect sentiment analysis)
maybe @patrickvonplaten will know the answer…?
thanks!!

      huggingface.co

allenai/reviews_roberta_base · Hugging Face

Context:
ok, from this related paper

      arxiv.org

2004.10964.pdf

  1717.18 KB

it seems that it’s pretrained on amazon reviews dataset


Text Generation, adding random words, weird linebreaks & symbols at random

Question:
Here’s the code I’m using to generate text.
tokenizer = GPT2Tokenizer.from_pretrained("gpt2-medium")
model = GPT2LMHeadModel.from_pretrained("gpt2-medium" , pad_token_id = tokenizer.eos_token_id)

sentence= tokenizer.encode(kw, return_tensors="pt") output = model.generate(sentence, max_length = 500, no_repeat_ngram_size = 2, do_sample=False) text.append(tokenizer.decode(output[0], skip_special_tokens = True))

The issue is that the output often comes like this:
"What are the benefits of using collagen?
,
,
,
,
, __________________, __________
The skin that has collagen has a higher level of hydrophilic (water-loving) proteins. `
or like this:
Yes, collagen is a natural skin-repairing substance. It is also a powerful anti-inflammatory and antiaging agent.  , and, are the most common types of collagen found in skin.
As you can see, at the start it wrote “, and,” at random and it happens EXTREMELY often, nearly in every single text generation I did.
I don’t know if it’s related to my settings or not but I’d appreciate all the help you guys can give. I want to get my text to be as human-readable as possible & up to 100-500 words each input.

Context: It might help if you give more information about the model and tokenizer that you use.


Question:
Hi,
When first I did
from transformers import BertModel
model = BertModel.from_pretrained('bert-base-cased')

Then it’s fine.
But after doing the above, when I do:
from transformers import BertForSequenceClassification
m = BertForSequenceClassification.from_pretrained('bert-base-cased')

I get warning messages:
Some weights of the model checkpoint at bert-base-cased were not used when
initializing BertForSequenceClassification: ['cls.predictions.bias',
'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias',
'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias',
'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']

- This IS expected if you are initializing BertForSequenceClassification from the
checkpoint of a model trained on another task or with another architecture (e.g.
initializing a BertForSequenceClassification model from a BertForPreTraining model).

Some weights of BertForSequenceClassification were not initialized from the model
checkpoint at bert-base-cased and are newly initialized: ['classifier.weight',
'classifier.bias']

There is another topic regarding the same issue in the forum here 1.
What I have understood is that, due to the first code which I ran, the weights of the pre-trained bare   bert-base-cased model got downloaded, and when I ran the second code for sequence classification, the weights regarding the sequence classification didn’t get downloaded because it is grabbing its checkpoint from the first code which I ran.
The same is also given in the last paragraph of the warning message.
So, what’s the solution to download the pre-trained weights for sequence classification tasks or in general other tasks?
Thanks.

Context:
Hi,
Can anyone help me regarding this?
Thanks.


A universal granular method to breakdown text for modeling

Question:
My gut feeling is one thing but,
Is there a way to breakdown text into the most granular form that can be applied to any model? (For instance IOB, tagging, tokens etc?)

Context:
hey @Joanna if i understand correctly, your question is fundamentally about the manner in which we tokenize raw text into tokens right? for example, BERT uses a WordPiece tokenizer that can be used for all downstream tasks, provided some alignment between the tokens and labels is given for tasks like NER.
there’s a nice walkthrough the various tokenization strategies that are employed in transformers here: Summary of the tokenizers — transformers 4.5.0.dev0 documentation 2


Question:
hi everyone,
are there any paid/unpaid tutorials on usage of this awesome library? or any github pages with code walkthroughs?
any help is appreciated!!
thanks,
Kishor

Context:
huggingface.co

Transformers 31

  State-of-the-art Natural Language Processing for Jax, Pytorch and TensorFlow 🤗 Transformers (formerly known as pytorch-transformers and pytorch-pretrained-be...


Value Error when instantiating MultiClass Classifier

Question:
Here is the code I am using:
model = ClassificationModel('roberta', 'roberta-base', num_labels=37, args={'learning_rate':1e-5, 'num_train_epochs':10, 'reprocess_input_data':True})
Here is the error it generates:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-16-affadfcfb25e> in <module>
      1 #Need to figure out how many labels in os data
----> 2 model = ClassificationModel('roberta', 'roberta-base', num_labels=37, args={'learning_rate':1e-5, 'num_train_epochs':10, 'reprocess_input_data':True})

~\Anaconda3\lib\site-packages\simpletransformers\classification\classification_model.py in __init__(self, model_type, model_name, tokenizer_type, tokenizer_name, num_labels, weight, args, use_cuda, cuda_device, onnx_execution_provider, **kwargs)
    334
    335         if num_labels:
--> 336             self.config = config_class.from_pretrained(
    337                 model_name, num_labels=num_labels, **self.args.config
    338             )

~\Anaconda3\lib\site-packages\transformers\configuration_utils.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    425
    426         """
--> 427         config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
    428         if "model_type" in config_dict and hasattr(cls, "model_type") and config_dict["model_type"] != cls.model_type:
    429             logger.warn(

~\Anaconda3\lib\site-packages\transformers\configuration_utils.py in get_config_dict(cls, pretrained_model_name_or_path, **kwargs)
    482         try:
    483             # Load from URL or cache if already cached
--> 484             resolved_config_file = cached_path(
    485                 config_file,
    486                 cache_dir=cache_dir,

~\Anaconda3\lib\site-packages\transformers\file_utils.py in cached_path(url_or_filename, cache_dir, force_download, proxies, resume_download, user_agent, extract_compressed_file, force_extract, use_auth_token, local_files_only)
   1269     if is_remote_url(url_or_filename):
   1270         # URL, so get it from the cache (downloading if necessary)
-> 1271         output_path = get_from_cache(
   1272             url_or_filename,
   1273             cache_dir=cache_dir,

~\Anaconda3\lib\site-packages\transformers\file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, use_auth_token, local_files_only)
   1492                     )
   1493                 else:
-> 1494                     raise ValueError(
   1495                         "Connection error, and we cannot find the requested files in the cached path."
   1496                         " Please try again or make sure your Internet connection is on."

ValueError: Connection error, and we cannot find the requested files in the cached path. Please try again or make sure your Internet connection is on.

What can I do to resolve this issue?

Context: hey @parkz your issue seems to be about the simpletransformers library (not transformers) so i suggest you post you question on their repo to see if someone can help: GitHub - ThilinaRajapakse/simpletransformers: Transformers for Classification, NER, QA, Language Modelling, Language Generation, T5, Multi-Modal, and Conversational AI 1


Encountering issues using camemBERT with rasa

Question:
in the Rasa forum there is some developer who succeeded using CamemBERT
here is the example RASA and camemBERT - #3 by Zoukero - Rasa Open Source - Rasa Community Forum 6
unfortunately while trying their solution I get this error

ImportError: cannot import name ‘modeling_tf_camembert’ from ‘transformers’ (unknown location)

any help/advice will be much appreciated
Thanks

Context: This must have been with a Transformers version < 4.0.0. You should just do from transformers.models.camembert import modeling_tf_camembert now.


How to pad tokens to a fixed length on a single sentence?

Question:
>>> from transformers import BartTokenizerFast
>>> tokenizer = BartTokenizerFast.from_pretrained("facebook/bart-large")
>>> str = "How are you?"
>>> tokenizer(str, return_tensors="pt")
{'input_ids': tensor([[   0, 6179,   32,   47,  116,    2]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1]])}
>>> tokenizer(str, padding=True, max_length=10, return_tensors="pt")
{'input_ids': tensor([[   0, 6179,   32,   47,  116,    2]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1]])}

Why didn’t it show something like
{'input_ids': tensor([[   0, 6179,   32,   47,  116,    2, 1, 1, 1, 1]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 0, 0, 0, 0]])}

and how can I do it?

Context:
hey @zuujhyt, you can activate the desired padding by specifying padding="max_length" in your tokenizer as follows:
tokenizer(str, return_tensors="pt", padding="max_length", max_length=10)

when padding=True the tokenizer will pad to the longest sequence in the batch (or no padding for the single sentence case)


Tokenization of Sequencepair for Pipeline

Question:
Hi,
for my model I tokenize my input like this:
encoded_dict = xlmr_tokenizer(term1, term2, max_length=max_len, padding='max_length',truncation=True, return_tensors='pt')
so that it gets 2 different strings as input (term1, term2), that it separates with the special separator token.
How can I use the huggingface pipeline for input like this?
If I load the pipeline like this:
tokenizer_xlmr = XLMRobertaTokenizerFast.from_pretrained("xlm-roberta-base")
my_pipeline = pipeline("sentiment-analysis", model=my_model, tokenizer=tokenizer_xlmr)
my_pipeline([term1,term2])

I get two separate predictions instead of one. Thanks in advance for any type of help!

Context:
Further information:
calling
my_pipeline(term1+"</s></s>"+term2)
gives nearly the same output as using the trained model directly on the data from the encoded_dict as defined above. The labels are the same, but the probability differ from the 5th/6th decimal place onwards. Why is that?


BertForNextSentencePrediction with larger batch size

Question:
Is there a way to use BertForNextSentencePrediction in inference mode with a batch size larger than 1? I have some code.
encoding = self.tokenizer(prompt1, prompt2, return_tensors='pt', padding=True, truncation=True, add_special_tokens=True)
print(encoding)
outputs = self.model(**encoding, next_sentence_label=torch.LongTensor([1]), target_batch_size=10)
logits = outputs.logits
#print(logits)

Here prompt1 and prompt2 are lists of sentences. The list is 10 sentences long. I get an error like this:

ValueError: Expected input batch_size (10) to match target batch_size (1).

Context:
hey @DLiebman i think the problem is that you’re passing a batch of 10 examples, but only a single label in the next_sentence_label argument. changing your code to the following works for me:
outputs = model(**encoding, next_sentence_label=torch.ones((10,1), dtype=torch.long), target_batch_size=10)


Difference between tokenizer and tokenizerfast

Question:
Hi,
I have searched for the answer for my question, but still can’t get the clear answer.
Some issues in the github/forum also report that the result of tokenizer and tokenizerfast is a little bit different.
I want to know what is the difference between them (in terms of mechanism)?
If they should output the same result, then why we need both of them?

Context:
hey @ad26kr can you provide a few links on the reported differences between the two types of tokenizers?
cc @anthony who is the tokenizer expert


GPT2 working perfectly in local system, but doesn’t generate text (stuck) when deployed in server

Question:
I have built a simple web app 8 where the user inputs a few words and the length of the text to generate, and the model takes a minute to produce the results locally. But when I deploy it in Heroku, its taking forever (not displaying any results even after a couple of hours). You can check it out here 4.
Is it because the server’s CPU is too slow/weak? If yes, how do I use a faster CPU in Heroku, or can you suggest some other service instead of Heroku that would be better to deploy GPT2 based web apps? If not, what’s the issue and how do I fix it?

Context:
hey @kristada673, looking at your code it seems that you load the model every time a user provides a prompt. a better approach would be to load the model once when the server spins up and then call it in a dedicated endpoint for prompting.
depending on your use case, a simpler alternative to heroku would be streamlit - you can find many examples online using GPT-2 with it (e.g. here 6)


Help resolving error (“TextInputSequence must be str”)

Question:
Hi,
I’m very new to HuggingFace, I’ve come around this error “TextInputSequence must be str” on a notebook which is helping me a lot to do some practice on various hugging face models. The boilerplate code on the notebook is throwing this error (I guess) due to some changes in huggingface’s API or something. So I was wondering if someone could help me and suggest some changes that I can make to the code to resolve the error.
The error can easily be reproduced by just running all the cells of the notebook.
Link: Colab Notebook 9

11865×678 267 KB

Context: 21802×593 330 KB


Question:
Hi,
I was wondering why the length of the output_scores is always +1 longer than the max_length in the output of generate()? This seems to be not consistent with the documentation https://huggingface.co/transformers/internal/generation_utils.html#transformers.generation_utils.BeamSampleEncoderDecoderOutput 28
I found the scores from the output of the generate() function when setting output_scores to be True is (max_length+1,)-shaped tensors or shorter due to the early eos_token_id with each element of shape (batch_size*num_beams, config.vocab_size). The shape of the output.sequence is (batch_size, max_length).

Context:
Hey @Kylie,
good observation! So output_scores should max_length - 1. The reason is that the first token, the decoder_start_token_id is not generated, meaning that no scores can be calculated.
Here an example:
#!/usr/bin/env python3
from transformers import AutoModelForSeq2SeqLM
import torch

model = AutoModelForSeq2SeqLM.from_pretrained('facebook/bart-large')

out = model.generate(torch.tensor([10 * [1]]), return_dict_in_generate=True, output_scores=True, max_length = 10)

print("len scores:", len(out.scores))  # should give 9

Would you be interested in correcting the documentation in a PR for Transformers?


Wav2Vec2ForCTC.from_pretrained for already trained Models?

Question:
Hi Guys,
how do i further train a Model someone else already fine-tuned? I wanted to load a already Fine-Tuned Modell like this:
    model = Wav2Vec2ForCTC.from_pretrained(
        model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
        activation_dropout=model_args.activation_dropout,
        attention_dropout=model_args.attention_dropout,
        hidden_dropout=model_args.hidden_dropout,
        feat_proj_dropout=model_args.feat_proj_dropout,
        mask_time_prob=model_args.mask_time_prob,
        gradient_checkpointing=model_args.gradient_checkpointing,
        layerdrop=model_args.layerdrop,
        ctc_loss_reduction="mean",
        pad_token_id=processor.tokenizer.pad_token_id,
        vocab_size=len(processor.tokenizer),
    )

But im getting:
RuntimeError: Error(s) in loading state_dict for Wav2Vec2ForCTC:
	size mismatch for lm_head.weight: copying a param with shape torch.Size([174, 1024]) from checkpoint, the shape in current model is torch.Size([35, 1024]).
	size mismatch for lm_head.bias: copying a param with shape torch.Size([174]) from checkpoint, the shape in current model is torch.Size([35]).

How would i do this right?
Something like
    model = Wav2Vec2ForCTC.from_pretrained(model_args.model_name_or_path)
    model.to("cuda")
    model.freeze_feature_extractor()
    processor = Wav2Vec2Processor.from_pretrained(model_args.model_name_or_path)

But im not sure if this is the way to go.
Ty in advanced

Context:
It seems like your vocabulary size does not match the vocab size from the original model. You can remove the old LM head manually and initialise a new one:
state_dict = torch.load(f"{model_args.model_name_or_path}/pytorch_model.bin", map_location='cpu')
state_dict.pop('lm_head.weight')
state_dict.pop('lm_head.bias')

model = Wav2Vec2ForCTC.from_pretrained(
        model_args.model_name_or_path,
        state_dict=state_dict,
        ...
    )


Question:
I use ubuntu 20.04, and it seems that after I’m using some models, the free space is just disappear. (250 GB)
where does the models cache are saved to?
is there some command to clean them?
thanks!

Context: Models are cached in the ~/.cache/huggingface/transformers directory (from your home directory)


Question:
I think my model would benefit from the inclusion of character encoding similar to ELMO, i.e. as discussed in CharacterBERT (https://www.aclweb.org/anthology/2020.coling-main.609.pdf ) since I have loads of acronyms so I think added characters will help.
Is there a layer in the huggingface library I could use to replicate this configuration? I don’t see one, nor a hugginface ELMO but I could well have missed it. Otherwise I’ll try and use AllenNLP.

Context:
hey @david-waterworth, looking at the paper’s github repo (link), it seems that they just changed the embedding layer of the BERT implementation in transformers to accept character embeddings: character-bert/character_bert.py at 0519a6e22a2912bf0fdbbd49bd562cc2e5410bc7 · helboukkouri/character-bert · GitHub 1
you could try the same approach and see if that works!


Missing content in task specific pipeline docs

Question: Looks like missing doc content in the pipeline page here Pipelines — transformers 4.5.0.dev0 documentation

Context: Thanks for flagging! Investigating why this is the case.


Question:
I’m looking to use wav2vec2 english for own project use-case for the purpose of transcription, it is possible that I can feed large audio files (upto 1hr of duration) for the task?
Thanks in Advance!

Context: Yes, you should be able to. Here 5 are some references on fine-tuning wav2vec on common voice dataset. With a csv file having one column for location of audio clips and the other column the label, you can fine-tune it for your own audio files.


How to load weights which was trained older version of PyTorch?

Question:
Hi,
I trained a RoBERTa model on PyTorch 1.6 and tried to load the weights on different server which had PyTorch 1.4.
But I could not load the weights and got an error which said,
OSError: Unable to load weights from pytorch checkpoint file. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.

I can upgrade PyTorch which is not a problem but I’m curious how can I load weights trained on older version of PyTorch?
Environments of both servers:
ubuntu: 18.04 ( both servers )
transformers: 3.1.0 ( both servers )
Python 3.7.9 and Python 3.7.3
Thanks in advance.

Context: AFAIK PyTorch versions shouldn’t make a difference. Check your path, check that the file is not empty. Also post the code that you use to save and load the model.


In the “Write With Transformer” page, how do different suggestions are selected?

Question: When using Write With Transformer, after hitting tab, there are multiple suggestions. Sometimes one word, sometimes a partial sentence. How are these selected? In particular, for one word, does it take the top k probabilities from the output? When does it decide to continue to a sentence?

Context:
No answer from anyone?
Maybe link to the source code of the suggestion generation?


Ideas to correct Wav2Vec2 transcription results

Question:
I’m tinkering with the preprocessing (segmentation) to achieve the best Wav2Vec2 transcriptions I can, and am fairly impressed with the results (compared to others like Silero, and previous experience with Sphinx, or alternatives like ELAN).
(I’m finding I probably need to reduce the maximum segment times below 60s and haven’t quite got that down perfectly yet, but that’s beside the point)
However, there are some pretty glaring phonetic transcription mistakes and I’m wondering if there are standard approaches to adjust these (in an automated way, before resorting to manual adjustment).
For instance, I’m seeing “Boric Johnson” (for Boris Johnson, the UK Prime Minister), “ennay chess” (for NHS, the UK’s health service), and “social medeor” (social media).
These are all clearly phonetic ‘guesses’ and would be identifiable as outside the vocabulary of a standard text model: I’m curious if there’s a standard post-processing step that attempts to ‘realign’ such outputs with possible alternatives (which I could perhaps explore in an interface to resolve ambiguous parts).
I’m wondering if anyone knows of the usual approach (or if this is not a standard step, could suggest an innovative approach) using language models — even just the proper terms for what I’m trying to do here would help me research my next steps.
I’d think it was a similar-ish problem to spelling mistakes (for which there’s ‘T5 Sentence Doctor 6’ for example, but tests aren’t too encouraging). I may be missing a more appropriate alternative I don’t know about so I thought I’d ask the community here.
Thanks, first question on the forum so please let me know if this is off topic, I’ve used the new Wav2Vec2 960h model and may try the other versions next but expect this will apply to all of them.

Context:
Hi,
I think you need a language model trained on a large corpora. You can have a look on this  issue 45.
Best,
Omar


KeyError: ‘loss’ even after appending labels while Fine Tuning Transformer XL

Question:
I am trying to do Casual Language Modeling Task by fine tuning Transformer XL (transfo-xl-wt103) model on my custom data. I have data in which max input words in a line are approx 50,000 and average number of words are 1000. As I have read that Transformer XL can take unlimited length of input. So, I think its the best option for my task. Moreover, I don’t want to concatenate and make blocks of length 128 as it would defeat the actual purpose. I can add padding/truncation to make all sentence of equal length. I want to know is my task achievable. Also, what memory & GPU would I require for this task.
Currently I am doing the same task with dummy data but getting error. Please check the code and help me with this error.
Here is my complete code:
from datasets import load_dataset
import pandas as pd
from datasets import Dataset
from transformers import AutoTokenizer
from transformers import AutoModelForCausalLM
from transformers import Trainer, TrainingArguments

data = """
A laptop, laptop computer, or notebook computer is a small, portable personal computer (PC).
Laptops are folded shut for transportation, and thus are suitable for mobile use.
Its name comes from lap, as it was deemed practical to be placed on a person's lap when being used.
Today, laptops are the used in a variety of settings, such as at work, in education, web browsing, and general home computer use.
Design elements, form factor and construction can also vary significantly between models depending on intended use.
Examples of specialized models of laptops include rugged notebooks for use in construction or military applications.
"""

dataList = data.strip().split('.')
dataset = []
for line in dataList[0: -2]:
  dataset.append([line])
dataFrame = pd.DataFrame(dataset, columns = ['data'])
valDataFrame = pd.DataFrame(dataset[0:2], columns = ['data'])
print(dataFrame.shape)
print(valDataFrame.shape)
dataset = Dataset.from_pandas(dataFrame)
print(dataset)

modelCheckpoint = 'transfo-xl-wt103'
tokenizer = AutoTokenizer.from_pretrained(modelCheckpoint)
tokenizer.pad_token = tokenizer.eos_token

def tokenizeFunction(examples):
  return tokenizer(examples["data"],  add_special_tokens = True, padding = True, pad_to_max_length = True, max_length = 10, truncation = True)

dataset = dataset.map(tokenizeFunction, remove_columns=["data"])
valDataset = valDataset.map(tokenizeFunction, remove_columns=["data"])
print(dataset[0], end = " ")

def appendLabels(examples):
  examples["labels"] = examples["input_ids"].copy()
  return examples

dataset = dataset.map(appendLabels)
valDataset = valDataset.map(appendLabels)
print(dataset[0], end = " ")

model = AutoModelForCausalLM.from_pretrained(modelCheckpoint)

trainingArgs = TrainingArguments(
    "test-clm",
    evaluation_strategy = "epoch",
    learning_rate = 2e-5,
    weight_decay = 0.01,
    label_names = ["labels"]
)

trainer = Trainer(
    model = model,
    args = trainingArgs,
    train_dataset = dataset,
    eval_dataset = valDataset
)
trainer.train()

Error:
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-37-0d4f1e0acd39> in <module>()
     15     eval_dataset = valDataset
     16 )
---> 17 trainer.train()

3 frames
/usr/local/lib/python3.7/dist-packages/transformers/file_utils.py in __getitem__(self, k)
   1614         if isinstance(k, str):
   1615             inner_dict = {k: v for (k, v) in self.items()}
-> 1616             return inner_dict[k]
   1617         else:
   1618             return self.to_tuple()[k]

KeyError: 'loss'

Version & Details:
- `transformers` version: 4.5.1
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1+cu101 (False)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

Context: Yes, transformer XL is the only model of the library incompatible with Trainer because it return losses (not averaged) instead of loss.


Question: can I have an unlimited usage of the models for 9$ or is too low??

Context:
Hi @micole66 ,
No the 9$ does not provide unlimited usage of the models.
You are enabled to use 300k input characters with your subscription currently. We want to make the API usage something pay-as-you go so we can make it affordable for everyone.
Cheers,
Nicolas


Question:
Hi,
Is there a JavaScript example for using inference API - 🤗 Accelerated Inference API — Api inference documentation 25

Context:
Hi @hgarg ,
Currently we don’t provide documentation for JS (or TS).
You could probably use simple fetch:
const HF_API_TOKEN = "api_xxxx";
const model = "XXX"

const data = {inputs:"Something here"};

const response = await fetch(`https://api-inference.huggingface.co/models/${model}`, {headers: {"Authorization": `Bearer ${HF_API_TOKEN}`}, method: "POST", data:JSON.stringify(data)});
const data = await response.json()

should work.
Cheers,
Nicolas


Question:
I would like to train some models to multiple GPUs.
Let suppose that I use model from HF library, but I am  using my own trainers,dataloader,collators etc.
Where I should focus to implement multiple GPU training? I need to make changes only in the Trainer class? If yes, can you give me a brief description?
Thank you in avance.

Context: The Trainer class automatically handles multi-GPU training, you don’t have to do anything special.


Question:
Hi  I’m trying to fine-tune model with Trainer in transformers,
Well, I want to use a specific number of GPU in my server.
My server has two GPUs,(index 0, index 1) and I want to train my model with GPU index 1.
I’ve read the Trainer and TrainingArguments documents, and I’ve tried the CUDA_VISIBLE_DEVICES thing already. but it didn’t worked for me.
import os
os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"] = "1"

(I did it within Jupyter, before I import all libraries)
It gave me a runtime error when the trainer tries to initiate self.model = model.to(args.device) line.
and the error says like RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable.
trainer = Trainer(
 model=model,
args=training_args,
data_collator=data_collator,
train_dataset=train_dataset)

123944×820 127 KB
I’ve also tried torch.cuda.set_device(1), it also didn’t work.
I don’t know how to set it up. It seems like I don’t have any options in argument of class
Please help me to handle this problem.
Thank you.

Context: If torch.cuda_set_device(1) doesn’t work, the problem is in your install. Does the command nvidia-smi show up two GPUs?