Using HuggingFace TransfoXLLMHeadModel() with custom Torchtext vocabulary
Author: h56cho
Pull request: no
Labels: name: wontfix

Hello,
I am trying to use the HuggingFace TransfoXLLMHeadModel on WikiText2 dataset under a customized TransfoXLConfig with different vocabulary, and it causing an error. I am not sure how to fix it. Below are my code:

```js

# Import packages 
import torch
import torch.nn as nn
import torch.nn.functional as F
from transformers import TransfoXLConfig, TransfoXLTokenizer, TransfoXLModel, TransfoXLLMHeadModel, TFTransfoXLModel, TFTransfoXLLMHeadModel
import spacy
import torchtext
from torchtext.data.utils import get_tokenizer
from torchtext.data import Field, BPTTIterator, TabularDataset 
import math
import random
import numpy as np
import pandas as pd 
import time

# define the English text field
TEXT = Field(tokenize = 'spacy',
             init_token='<sos>',
             eos_token='<eos>',
             tokenizer_language='en',
             lower=True)

# load WikiText-2 dataset and split it into train and test set
train_Wiki2, val_Wiki2, test_Wiki2 = torchtext.datasets.WikiText2.splits(TEXT)

# build vocabulary based on the field that we just defined.
TEXT.build_vocab(train_Wiki2, val_Wiki2, test_Wiki2)

# get number of tokens
ntokens = len(TEXT.vocab.stoi) # ntokens =  28871

# define transformer-XL configuration.
transfoXLconfig = TransfoXLConfig(vocab_size_or_config_json_file = ntokens,
                                  cutoffs = [20000, 40000, 200000], 
                                  d_model = 64, 
                                  d_embed = 64, 
                                  n_head = 16, 
                                  d_head = 64,
                                  n_layer = 5,
                                  attn_type = 0,
                                  dropout = 0.1, 
                                  output_hidden_states = True,
                                  output_attentions = True)

# define the transformer-XL model based on the specified configuration.
model = TransfoXLLMHeadModel(transfoXLconfig) # this line is causing an error.
"""
Error message: 
Traceback (most recent call last):

  File "<ipython-input-14-fa91df67f439>", line 1, in <module>
    model = TransfoXLLMHeadModel(transfoXLconfig)

  File "/Users/jin-dominique/anaconda3/lib/python3.7/site-packages/transformers/modeling_transfo_xl.py", line 818, in __init__
    self.transformer = TransfoXLModel(config)

  File "/Users/jin-dominique/anaconda3/lib/python3.7/site-packages/transformers/modeling_transfo_xl.py", line 599, in __init__
    div_val=config.div_val)

  File "/Users/jin-dominique/anaconda3/lib/python3.7/site-packages/transformers/modeling_transfo_xl.py", line 421, in __init__
    self.emb_layers.append(nn.Embedding(r_idx-l_idx, d_emb_i))

  File "/Users/jin-dominique/anaconda3/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 97, in __init__
    self.weight = Parameter(torch.Tensor(num_embeddings, embedding_dim))

RuntimeError: Trying to create tensor with negative dimension -171129: [-171129, 1]
model = TransfoXLLMHeadModel(transfoXLconfig)
"""
```
How can I use HuggingFace TransfoXLLMHeadModel( ) with a custom vocabulary of different size?

Thank you,



Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I faced the same problem:I solved it by passing the size of your vocabulary (from your custom tokenizer) as a parameter.
I proceeded as follows:
`vocabulary_size = tokenizer.vocab_size`
```
configuration = tf.TransfoXLConfig(vocab_size_or_config_json_file=vocabulary_size, cutoffs=cutoffs,
                                       d_model=512, d_embed=512, n_head=8, d_head=64, n_layer=12, d_inner=2048)
```
I hope that helped :)

P.s. What do you pass as inputs and labels?
For now, I create a batch as follows:
"The quick brown fox jumps over the lazy dog"
If I have batch_size=2, and sequence length=4:
["The quick brown fox",
"jumps over the lazy"]

What do you feed to the Transformer-XL as input?