How to use custom built Torchtext vocabulary with HuggingFace TransfoXLLMHeadModel?
Author: h56cho
Pull request: no
Labels: name: wontfix

Hello,

I am trying to use my custom built vocabulary which I defined using Torchtext functions with the HuggingFace TransfoXLLMHeadModel, and I am having some troubles with it.
I defined my text field as below:
```js

# Import packages 
import torch
import torch.nn as nn
import torch.nn.functional as F
from transformers import TransfoXLConfig, TransfoXLTokenizer, TransfoXLLMHeadModel
from transformers import AdamW, WarmupLinearSchedule
import spacy
import torchtext
from torchtext.data.utils import get_tokenizer
from torchtext.data import Field, BPTTIterator, TabularDataset 
import tensorflow as tf
#import lineflow as lf
#import lineflow.datasets as lfds
import math
import random
import numpy as np
import pandas as pd 
import time

# define tokenizer
en = spacy.load('en')

def Sp_Tokenizer(text): 
    return [tok.text for tok in en.tokenizer(text)]

# define the English text field
TEXT = Field(tokenize = Sp_Tokenizer,
             init_token='< sos >',
             eos_token='< eos >',
             unk_token='< unk >',
             tokenizer_language='en',
             lower=True)

# load WikiText-2 dataset and split it into train and test set
train_Wiki2, val_Wiki2, test_Wiki2 = torchtext.datasets.WikiText2.splits(TEXT)
train_Wiki103, val_Wiki103, test_Wiki103 = torchtext.datasets.WikiText103.splits(TEXT)
train_Penn, val_Penn, test_Penn = torchtext.datasets.PennTreebank.splits(TEXT)

# build custom vocabulary based on the field that we just defined.
TEXT.build_vocab(train_Wiki2, val_Wiki2, test_Wiki2, 
                 train_Wiki103, val_Wiki103, test_Wiki103,
                 train_Penn, val_Penn, test_Penn)
```
and then I defined the HuggingFace transformer's configuration as below:
```js

# set hyperparameter ntokens
ntokens = len(TEXT.vocab.stoi)

# define transformer-XL configuration.
transfoXLconfig = TransfoXLConfig(vocab_size_or_config_json_file = ntokens,
                                  cutoffs = [20000, 40000, 200000], 
                                  d_model = 64, 
                                  d_embed = 64, 
                                  n_head = 16, 
                                  d_head = 64,
                                  n_layer = 5,
                                  attn_type = 0,
                                  dropout = 0.1, 
                                  output_hidden_states = True,
                                  output_attentions = True)

# define the transformer-XL model based on the specified configuration.
model = TransfoXLLMHeadModel(transfoXLconfig)

# add new tokens to the embeddings of our model
model.resize_token_embeddings(ntokens)
```
and then I want to somehow specify that I want to use my `TEXT.vocab` that I defined earlier via Torchtext for my vocabulary along with the TransfoXLLMHeadModel, but I am not sure how to do this. Can someone help me on this? Thank you!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.