Package: transformer-embedder
Transformer EmbedderA Word Level Transformer layer based on PyTorch and 🤗 Transformers.How to useInstall the library fromPyPI:pipinstalltransformer-embedderIt offers a PyTorch layer and a tokenizer that support almost every pretrained model from Huggingface🤗Transformerslibrary. Here is a quick example:importtransformer_embedderastretokenizer=tre.Tokenizer("bert-base-cased")model=tre.TransformerEmbedder("bert-base-cased",subtoken_pooling="mean",output_layer="sum")example="This is a sample sentence"inputs=tokenizer(example,return_tensors=True){
   'input_ids': tensor([[ 101, 1188, 1110,  170, 6876, 5650,  102]]),
   'offsets': tensor([[0, 0], [1, 1], [2, 2], [3, 3], [4, 4], [5, 5], [6, 6]]]),
   'attention_mask': tensor([[True, True, True, True, True, True, True]]),
   'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0]])
   'sentence_length': 7  # with special tokens included
}outputs=model(**inputs)# outputs.shape[1:-1]       # remove [CLS] and [SEP]
torch.Size([1, 5, 768])
# len(example)
5InfoOne of the annoyance of using transfomer-based models is that it is not trivial to compute word embeddings from the sub-token embeddings they output. With this library it's as easy as using 🤗Transformers API to get word-level embeddings from theoretically every transformer model it supports.ModelTheTransformerEmbedderoffer 4 ways to retrieve the word embeddings, defined bysubtoken_poolingparameter:first: uses only the embedding of the first sub-token of each wordlast: uses only the embedding of the last sub-token of each wordmean: computes the mean of the embeddings of the sub-tokens of each wordnone: returns the raw output of the transformer model without sub-token poolingThere are also multiple type of outputs you can get usingoutput_layerparameter:last: returns the last hidden state of the transformer modelconcat: returns the concatenation of the last four hidden states of the transformer modelsum: returns the sum of the last four hidden states of the transformer modelpooled: returns the output of the pooling layerIf you also want all the outputs from the HuggingFace model, you can setreturn_all=Trueto get them.classTransformerEmbedder(torch.nn.Module):def__init__(self,model:Union[str,tr.PreTrainedModel],subtoken_pooling:str="first",output_layer:str="last",fine_tune:bool=True,return_all:bool=False,)TokenizerTheTokenizerclass provides thetokenizemethod to preprocess the input for theTransformerEmbedderlayer. You
can pass raw sentences, pre-tokenized sentences and sentences in batch. It will preprocess them returning a dictionary
with the inputs for the model. By passingreturn_tensors=Trueit will return the inputs astorch.Tensor.By default, if you pass text (or batch) as strings, it splits them on spacestext="This is a sample sentence"tokenizer(text)text=["This is a sample sentence","This is another sample sentence"]tokenizer(text)You can also use SpaCy to pre-tokenize the inputs into words first, usinguse_spacy=Truetext="This is a sample sentence"tokenizer(text,use_spacy=True)text=["This is a sample sentence","This is another sample sentence"]tokenizer(text,use_spacy=True)or you can pass an pre-tokenized sentence (or batch of sentences) by settingis_split_into_words=Truetext=["This","is","a","sample","sentence"]tokenizer(text,is_split_into_words=True)text=[["This","is","a","sample","sentence","1"],["This","is","sample","sentence","2"],]tokenizer(text,is_split_into_words=True)# here is_split_into_words is redundantExamplesFirst, initialize the tokenizerimporttransformer_embedderastretokenizer=tre.Tokenizer("bert-base-cased")You can pass a single sentence as a string:text="This is a sample sentence"tokenizer(text){
  'input_ids': [101, 1188, 1110, 170, 6876, 5650, 102],
  'offsets': [(0, 0), (1, 1), (2, 2), (3, 3), (4, 4), (5, 5), (6, 6)],
  'attention_mask': [True, True, True, True, True, True, True],
  'token_type_ids': [0, 0, 0, 0, 0, 0, 0],
  'sentence_length': 7
}A sentence pairtext="This is a sample sentence A"text_pair="This is a sample sentence B"tokenizer(text,text_pair){
  'input_ids': [101, 1188, 1110, 170, 6876, 5650, 138, 102, 1188, 1110, 170, 6876, 5650, 139, 102],
  'offsets': [(0, 0), (1, 1), (2, 2), (3, 3), (4, 4), (5, 5), (6, 6), (7, 7), (8, 8), (9, 9), (10, 10), (11, 11), (12, 12), (13, 13), (14, 14)],
  'attention_mask': [True, True, True, True, True, True, True, True, True, True, True, True, True, True, True],
  'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1],
  'sentence_length': 15
}A batch of sentences or sentence pairs. Usingpadding=Trueandreturn_tensors=True, the tokenizer returns the text ready for the modelbatch=[["This","is","a","sample","sentence","1"],["This","is","sample","sentence","2"],["This","is","a","sample","sentence","3"],# ...["This","is","a","sample","sentence","n","for","batch"],]tokenizer(batch,padding=True,return_tensors=True)batch_pair=[["This","is","a","sample","sentence","pair","1"],["This","is","sample","sentence","pair","2"],["This","is","a","sample","sentence","pair","3"],# ...["This","is","a","sample","sentence","pair","n","for","batch"],]tokenizer(batch,batch_pair,padding=True,return_tensors=True)Custom fieldsIt is possible to add custom fields to the model input and tell thetokenizerhow to pad them usingadd_padding_ops.
Start by simply tokenizing the input (without padding or tensor mapping)importtransformer_embedderastretokenizer=tre.Tokenizer("bert-base-cased")text=[["This","is","a","sample","sentence"],["This","is","another","example","sentence","just","make","it","longer"]]inputs=tokenizer(text)Then add the custom fileds to the resultcustom_fields={"custom_filed_1":[[0,0,0,0,1,0,0],[0,0,0,0,1,0,0,0,0,1,0]]}inputs.update(custom_fields)Now we can add the padding logic for our custom fieldcustom_filed_1.add_padding_opsmethod takes in inputkey: name of the field in the tokenzer inputvalue: value to use for paddinglength: length to pad. It can be anint, or two string value,subtokenin which the element is padded to the batch max length relative to the sub-tokens length, andwordwhere the element is padded to the batch max length relative to the original word lengthtokenizer.add_padding_ops("custom_filed_1",0,"word")Finally, pad the input and convert it to a tensor:# manual processinginputs=tokenizer.pad_batch(inputs)inputs=tokenizer.to_tensor(inputs)The inputs are ready for the model, including the custom filed.>>> inputs

{
   "input_ids": tensor(
       [
           [101, 1188, 1110, 170, 6876, 5650, 102, 0, 0, 0, 0],
           [101, 1188, 1110, 1330, 1859, 5650, 1198, 1294, 1122, 2039, 102],
       ]
   ),
   "offsets": tensor(
       [
           [[0, 0], [1, 1], [2, 2], [3, 3], [4, 4], [5, 5], [6, 6], [7, 7], [-1, -1], [-1, -1], [-1, -1]],
           [[0, 0], [1, 1], [2, 2], [3, 3], [4, 4], [5, 5], [6, 6], [7, 7], [8, 8], [9, 9], [10, 10]],
       ]
   ),
   "attention_mask": tensor(
       [
           [True, True, True, True, True, True, True, False, False, False, False],
           [True, True, True, True, True, True, True, True, True, True, True],
       ]
   ),
   "word_mask": tensor(
       [
           [True, True, True, True, True, True, True, False, False, False, False],
           [True, True, True, True, True, True, True, True, True, True, True],
       ]
   ),
   "token_type_ids": tensor(
       [[0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]
   ),
   "sentence_length": tensor([7, 11]),
   "custom_filed_1": tensor(
       [[0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0], [0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0]]
   ),
}SpaCy TokenizerBy default, it uses themultilingual modelxx_sent_ud_sm. You can change
it with thelanguageparameter during the tokenizer initialization. For example, if you prefer an English tokenizer:tokenizer=tre.Tokenizer("bert-base-cased",language="en_core_web_sm")For a complete list of languages and models, you can gohere.To-DoFuture developmentsAdd an optional word tokenizer, maybe using SpaCyAddadd_special_tokenswrapperMakepad_batchfunction more generalAdd logic (like how to pad, etc) for custom fieldsDocumentationInclude all model outputsDocumentationA TensorFlow version (improbable)AcknowledgementsMost of the code in theTransformerEmbedderclass is taken from theAllenNLPlibrary. The pretrained models and the core of the tokenizer is from🤗 Transformers.