XLMForTokenClassification
Author: avacaondata
Pull request: no
Labels: name: wontfix

# 🌟New model addition

## Model description
As the XLM version for token classification (NER tasks) was not implemented, I followed a similar path as for BertForTokenClassification, and it seems to work. The reason for this is that BERT multilingual works horribly for WikiNER in spanish, achieving only 55% F1 with much fine-tuning, against the 87% of Spacy. Therefore, I'm trying to improve this metric and for that purpose I decided to use XLM, which is trained only on 15 different languages, not on more than 100. There's still one think that my model implementation lacks, and that is the fact that model dimension has to be set manually. I've been trying to add d_model to XLMConfig and then pass this config to my class, but it says XLMModel has no attribute d_model. If anyone can help me out with that I'd appreciate that. 
<!-- Important information -->

The code: 

class XLMForTokenClassification(XLMModel):
    
    def __init__(self, config, d_model=1024):
        super(XLMForTokenClassification, self).__init__(config)
        self.num_labels = config.num_labels
        self.xlm = XLMModel(config)
        self.dropout = nn.Dropout(config.dropout)
        self.classifier = nn.Linear(d_model, config.num_labels)
        self.init_weights()
    
    def forward(self, input_ids=None, attention_mask=None, langs=None, token_type_ids=None,
               position_ids=None, head_mask=None, labels=None):
        
        outputs = self.xlm(input_ids,
                          attention_mask=attention_mask,
                          langs=langs,
                          token_type_ids=position_ids,
                          head_mask=head_mask)
        
        sequence_output = self.dropout(outputs[0])
        logits = self.classifier(sequence_output)
        
        outputs = (logits, ) + outputs[2:] #add hidden states and attention if they are here
        
        if labels is not None:
            
            loss_fct = nn.CrossEntropyLoss()
            #only keep active parts of the loss
            if attention_mask is not None:
                
                active_loss = attention_mask.view(-1) == 1
                active_logits = logits.view(-1, self.num_labels)[active_loss]
                active_labels = labels.view(-1)[active_loss]
                loss = loss_fct(active_logits, active_labels)
                
            else:
                loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
            
            outputs = (loss, ) + outputs
        
        return outputs


## Open Source status

* [x] the model implementation is available: (give details)
It is uploaded above.
* [ ] the model weights are available: (give details)
* [x] who are the authors: (mention them)
Alejandro Vaca Serrano

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
Hi @alexvaca0 ,

Try to remove the `d_model` parameter in the constructor. Use `config.emb_dim` (`emb_dim` is specified in the [xlm config](https://s3.amazonaws.com/models.huggingface.co/bert/xlm-mlm-en-2048-config.json)), so this should work:

```python
self.classifier = nn.Linear(config.emb_dim, config.num_labels)
```

:)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.