3 Types of Contextualized Word Embeddings Using BERT | by Arushi Prakash | Medium | Towards Data Science

Since Google launched the BERT model in 2018, the model and its capabilities have captured the imagination of data scientists in many areas. The model has been adapted to different domains, like SciBERT for scientific texts, bioBERT for biomedical texts, and clinicalBERT for clinical texts. The lofty model, with 110 million parameters, has also been compressed for easier use as ALBERT (90% compression) and DistillBERT (40% compression). The original BERT model and its adaptations have been used for improving the performance of search engines, content moderation, sentiment analysis, named entity recognition, and more.
In this article, I will demonstrate show three ways to get contextualized word embeddings from BERT using python, pytorch, and transformers.
The article is split into these sections:
What is transfer learning?
How have BERT embeddings been used for transfer learning?
Setting up PyTorch to get BERT embeddings
Extracting word embeddings (“Context-free” pre-trained embedding, “Context-based” pre-trained embedding, “Context-averaged” pre-trained embedding)
Conclusion
In transfer learning, knowledge embedded in a pre-trained machine learning model is used as a starting point to build models for a different task. Transfer learning applications have exploded in the fields of computer vision and natural language processing because it requires significantly lesser data and computational resources to develop useful models. It has been termed as the next frontier in machine learning.
BERT has been used for transfer learning in several natural language processing applications. Recent examples include detecting hate speech, classify health-related tweets, and sentiment analysis in the Bengali language.
Check out my Jupyter notebook for the full code
# Importing the relevant modulesfrom transformers import BertTokenizer, BertModelimport pandas as pdimport numpy as npimport torch# Loading the pre-trained BERT model#################################### Embeddings will be derived from# the outputs of this modelmodel = BertModel.from_pretrained(‘bert-base-uncased’,           output_hidden_states = True,)# Setting up the tokenizer#################################### This is the same tokenizer that# was used in the model to generate# embeddings to ensure consistencytokenizer = BertTokenizer.from_pretrained(‘bert-base-uncased’)
We also need some functions to massage the input into the right form
def bert_text_preparation(text, tokenizer):    """Preparing the input for BERT        Takes a string argument and performs    pre-processing like adding special tokens,    tokenization, tokens to ids, and tokens to    segment ids. All tokens are mapped to seg-    ment id = 1.        Args:        text (str): Text to be converted        tokenizer (obj): Tokenizer object            to convert text into BERT-re-            adable tokens and ids            Returns:        list: List of BERT-readable tokens        obj: Torch tensor with token ids        obj: Torch tensor segment ids            """    marked_text = "[CLS] " + text + " [SEP]"    tokenized_text = tokenizer.tokenize(marked_text)    indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)    segments_ids = [1]*len(indexed_tokens)    # Convert inputs to PyTorch tensors    tokens_tensor = torch.tensor([indexed_tokens])    segments_tensors = torch.tensor([segments_ids])    return tokenized_text, tokens_tensor, segments_tensors
And another function to convert the input into embeddings
def get_bert_embeddings(tokens_tensor, segments_tensors, model):    """Get embeddings from an embedding model        Args:        tokens_tensor (obj): Torch tensor size [n_tokens]            with token ids for each token in text        segments_tensors (obj): Torch tensor size [n_tokens]            with segment ids for each token in text        model (obj): Embedding model to generate embeddings            from token and segment ids        Returns:        list: List of list of floats of size            [n_tokens, n_embedding_dimensions]            containing embeddings for each token        """        # Gradient calculation id disabled    # Model is in inference mode    with torch.no_grad():        outputs = model(tokens_tensor, segments_tensors)        # Removing the first hidden state        # The first state is the input state        hidden_states = outputs[2][1:]    # Getting embeddings from the final BERT layer    token_embeddings = hidden_states[-1]    # Collapsing the tensor into 1-dimension    token_embeddings = torch.squeeze(token_embeddings, dim=0)    # Converting torchtensors to lists    list_token_embeddings = [token_embed.tolist() for token_embed in token_embeddings]    return list_token_embeddings
We are going to generate embeddings for the following texts
# Text corpus############### These sentences show the different# forms of the word 'bank' to show the# value of contextualized embeddingstexts = ["bank",         "The river bank was flooded.",         "The bank vault was robust.",         "He had to bank on her for support.",         "The bank was out of money.",         "The bank teller was a man."]
Embeddings are generated in the following manner
# Getting embeddings for the target# word in all given contextstarget_word_embeddings = []for text in texts:    tokenized_text, tokens_tensor, segments_tensors = bert_text_preparation(text, tokenizer)    list_token_embeddings = get_bert_embeddings(tokens_tensor, segments_tensors, model)        # Find the position 'bank' in list of tokens    word_index = tokenized_text.index('bank')    # Get the embedding for bank    word_embedding = list_token_embeddings[word_index]    target_word_embeddings.append(word_embedding)
Finally, distances between the embeddings for the word bank in different contexts are calculated using this code
from scipy.spatial.distance import cosine# Calculating the distance between the# embeddings of 'bank' in all the# given contexts of the wordlist_of_distances = []for text1, embed1 in zip(texts, target_word_embeddings):    for text2, embed2 in zip(texts, target_word_embeddings):        cos_dist = 1 - cosine(embed1, embed2)        list_of_distances.append([text1, text2, cos_dist])distances_df = pd.DataFrame(list_of_distances, columns=['text1', 'text2', 'distance'])
We create a Pandas DataFrame to store all the distances.
Check out my Jupyter notebook for the full code
The first text (“bank”) generates a context-free text embedding. This is context-free since there are no accompanying words to provide context to the meaning of “bank”. In a way, this is the average across all embeddings of the word “bank”.
Understandably, this context-free embedding does not look like one usage of the word “bank”. This is evident in the cosine distance between the context-free embedding and all other versions of the word.
Embeddings generated for the word “bank” from each sentence with the word create a context-based embedding. These embeddings are the most common form of transfer learning and show the true power of the method.
In this example, the embeddings for the word “bank” when it means a financial institution are far from the embeddings for it when it means a riverbank or the verb form of the word.
When all the embeddings are averaged together, they create a context-averaged embedding. This style of embedding might be useful in some applications where one needs to get the average meaning of the word.
Surprisingly, the context-free and context-averaged versions of the word are not the same as shown by the cosine distance of 0.65 between them.
Transfer learning methods can bring value to natural language processing projects. In this article, I demonstrated a version of transfer learning by generating contextualized BERT embeddings for the word “bank” in varying contexts. I also showed how to extract three types of word embeddings — context-free, context-based, and context-averaged. It is important to understand the distinction between these embeddings and use the right one for your application.
Check out my Jupyter notebook for the full code