Customized BertForTokenClassification Model
Author: lixin4ever
Pull request: no
Labels: name: wontfix

I try to customize BertForTokenClassification model by myself to perform sequence tagging and strictly follow the [original implementation](https://huggingface.co/pytorch-transformers/_modules/pytorch_transformers/modeling_bert.html#BertForTokenClassification). However, I cannot obtain the same results (lower scores) with those produced by BertForTokenClassification when I simply set the top-most tagging component/model as an Linear layer (i.e., the current model is identical to BertForTokenClassification). My code is below:

```python
class BertTagger(BertPreTrainedModel):

    def __init__(self, bert_config):
        super(BertTagger, self).__init__(bert_config)
        self.num_labels = bert_config.num_labels
        #self.tagger_config = TaggerConfig()
        self.bert = BertModel(bert_config)
        self.bert_dropout = nn.Dropout(bert_config.hidden_dropout_prob)
        self.classifier = nn.Linear(bert_config.hidden_size, bert_config.num_labels)
        self.apply(self.init_weights)

    def forward(self, input_ids, token_type_ids=None, attention_mask=None, labels=None,
                position_ids=None, head_mask=None):
        outputs = self.bert(input_ids, position_ids=position_ids, token_type_ids=token_type_ids,
                            attention_mask=attention_mask, head_mask=head_mask)
        # the hidden states of the last Bert Layer, shape: (bsz, seq_len, hsz)
        tagger_input = outputs[0]
        tagger_input = self.bert_dropout(tagger_input)
        logits = self.classifier(tagger_input)
        outputs = (logits,) + outputs[2:]
        if labels is not None:
            loss_fct = CrossEntropyLoss()
            # Only keep active parts of the loss
            if attention_mask is not None:
                active_loss = attention_mask.view(-1) == 1
                active_logits = logits.view(-1, self.num_labels)[active_loss]
                active_labels = labels.view(-1)[active_loss]
                loss = loss_fct(active_logits, active_labels)
            else:
                loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
            outputs = (loss,) + outputs
        return outputs
```

Has anyone encountered the same issue?

Comments:
@lixin4ever  I have the same question. Do you solve it?

I have solve it. Thank you!

@searchlink How do you solve this problem?

For reference, the updated resource link mentioned in the original post can be now found [here](https://huggingface.co/transformers/_modules/transformers/modeling_bert.html#BertForTokenClassification); It was affected by the renaming into `transformers`, too.

But I can't see any difference between "pytorch-transformers" and "transformers" except the line initializing BERT parameters.  Have you met the same problem? @dennlinger 

I was just pointing to the up-to-date reference. I'm currently looking into token classification using BERT (or in my case, I would prefer RoBERTa or other iterations of BERT, but unfortunately they seem not available yet).

So Token classification using BERT does not work?


As you can see below, `BertForTokenClassification` works as expected with **PyTorch 1.3.1** and **Transformers 2.2.2** installed with `pip install transformers`.
```
Python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import transformers
>>> from transformers import BertForTokenClassification
>>> from transformers import BertTokenizer
>>> tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
>>> model = BertForTokenClassification.from_pretrained('bert-base-uncased')
>>> text='Hello, my dog is cute'
>>> import torch
>>> input_ids = torch.tensor(tokenizer.encode(text, add_special_tokens=True)).unsqueeze(0)  # Batch size 1
>>> input_ids
tensor([[  101,  7592,  1010,  2026,  3899,  2003, 10140,   102]])
>>> labels = torch.tensor([1] * input_ids.size(1)).unsqueeze(0)  # Batch size 1
>>> labels
tensor([[1, 1, 1, 1, 1, 1, 1, 1]])
>>> outputs = model(input_ids, labels=labels)
>>> outputs
(tensor(0.7529, grad_fn=<NllLossBackward>), tensor([[[ 0.5078,  0.1628],
         [-0.0593,  0.0163],
         [ 0.0308, -0.2312],
         [ 0.0863, -0.1000],
         [-0.2833, -0.2656],
         [-0.2014, -0.5225],
         [-0.2912, -0.1220],
         [-0.2781, -0.2919]]], grad_fn=<AddBackward0>))
>>> len(outputs)
2
>>> loss=outputs[0]
>>> scores=outputs[1]
>>> loss
tensor(0.7529, grad_fn=<NllLossBackward>)
>>> scores
tensor([[[ 0.5078,  0.1628],
         [-0.0593,  0.0163],
         [ 0.0308, -0.2312],
         [ 0.0863, -0.1000],
         [-0.2833, -0.2656],
         [-0.2014, -0.5225],
         [-0.2912, -0.1220],
         [-0.2781, -0.2919]]], grad_fn=<AddBackward0>)
>>>
```

It's working with **TensorFlow 2.0.0** and **Transformers 2.2.2** too!
```
Python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
/home/vidiemme/anaconda3/envs/huggingface/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/vidiemme/anaconda3/envs/huggingface/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/vidiemme/anaconda3/envs/huggingface/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/vidiemme/anaconda3/envs/huggingface/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/vidiemme/anaconda3/envs/huggingface/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/vidiemme/anaconda3/envs/huggingface/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
>>> import transformers
>>> from transformers import BertTokenizer
>>> from transformers import TFBertForTokenClassification
>>> tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
>>> model = TFBertForTokenClassification.from_pretrained('bert-base-uncased')
2019-12-17 12:54:37.120123: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-12-17 12:54:37.320081: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3408000000 Hz
2019-12-17 12:54:37.320815: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55a42296edd0 executing computations on platform Host. Devices:
2019-12-17 12:54:37.320841: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
>>> text='Hello, my dog is cute'
>>> input_ids = tf.constant(tokenizer.encode(text))[None, :]  # Batch size 1
>>> input_ids
<tf.Tensor: id=6056, shape=(1, 8), dtype=int32, numpy=
array([[  101,  7592,  1010,  2026,  3899,  2003, 10140,   102]],
      dtype=int32)>
>>> outputs = model(input_ids)
>>> len(outputs)
1
>>> outputs
(<tf.Tensor: id=7961, shape=(1, 8, 2), dtype=float32, numpy=
array([[[ 0.09657212, -0.51087016],
        [ 0.28020248, -0.25160134],
        [-0.09995201,  0.0843759 ],
        [-0.12110823,  0.20886022],
        [-0.03617962,  0.00401567],
        [-0.03330922,  0.01042   ],
        [-0.21674895, -0.1601235 ],
        [ 0.1076538 ,  0.19144017]]], dtype=float32)>,)
>>> scores = outputs[0]
>>> scores
<tf.Tensor: id=7961, shape=(1, 8, 2), dtype=float32, numpy=
array([[[ 0.09657212, -0.51087016],
        [ 0.28020248, -0.25160134],
        [-0.09995201,  0.0843759 ],
        [-0.12110823,  0.20886022],
        [-0.03617962,  0.00401567],
        [-0.03330922,  0.01042   ],
        [-0.21674895, -0.1601235 ],
        [ 0.1076538 ,  0.19144017]]], dtype=float32)>
```

> So Token classification using BERT does not work?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.