How to do AVERAGE and MAX word embedding for LONG sentences? | by Jia Yi Chan | Towards Data Science

In the case of the BERT model we used in this experiment, we used the ESG_BERT model from https://github.com/mukut03/ESG-BERT. In this BERT model, max sequence the model can handle is only 512 tokens. Hence if your sequence is longer than 512 tokens, you require the technique called Chunking. We should also note that for the model we used, the word embedding vector size is trained at 768, so our embeddings output will also show a size of vector of 768 per token.
Suppose we have a sequence of 978 tokens, we can chunk them into 2 chunks because 978//512 = 1 (we take one chunk more than the floor division). For illustration, we assume that the embedding vector size of each word is 3.
In the orange box be the Average Sentence Embedding(AvgSE) and the green box represent the Max Sentence Embedding(MaxSE).
To calculate the AvgSE, we require to
Chunk the long sequence upWithin each chunk:
Chunk the long sequence up
Within each chunk:
Find word embedding of each word in the chunk
Find average of ALL the word embeddings in the chunk (sum all the vectors together, divided by the number of words in the chunk)
3. Average again the by adding all the word embeddings(in bullet 2), divided by the number of chunks
To calculate the MaxSE, we require to
Chunk the long sequence upWithin each chunk:
Chunk the long sequence up
Within each chunk:
Find word embedding of each word in the chunk
Find max of ALL the word embeddings in the chunk
3. Max again the word embeddings across the chunks (in bullet 2)
from typing import List, Dict, Tuple, Any, Callable
import numpy as np
import pandas as pd
import spacy_universal_sentence_encoder
import spacy
from transformers import AutoTokenizer, AutoModel
import torch
from helper_for_word_embedding import get_word_idx,get_hidden_states,chunking

CHUNK_SIZE = 300
_verbose = 0
_batch_size = 1
bert_path = "../../apps/bert_model/"
# Init word embedding models once
tokenizer = AutoTokenizer.from_pretrained(bert_path)
model = AutoModel.from_pretrained(bert_path, output_hidden_states=True)

print("tokenized tokens:", tokenizer.tokenize("i am jokingly angry"))

tokenized tokens: ['i', 'am', 'joking', '##ly', 'angry']

sent= "i am a girl in the house"

def chunking(max_len, sent):
    tokenized_text = sent.lower().split(" ")
    # using list comprehension
    final = [tokenized_text[i * max_len:(i + 1) *max_len] for i in range((len(tokenized_text) + max_len - 1) // max_len)] 
    
    # join back to sentences for each of the chunks
    sent_chunk = []
    for item in final:
        sent_chunk.append(' '.join(item))
    return(sent_chunk)

print(len(chunking(2,sent)))
chunking(2,sent)

4

['i am', 'a girl', 'in the', 'house']
 

Under Tokenization, we see that the original text of 4 words in “i am jokingly angry” is now tokenized to [‘i’, ‘am’, ‘joking’, ‘##ly’, ‘angry’]. This tokenization is prevalent in BERT models, and depending on which type of BERT model you choose, the words will be split up differently as well.
Under Chunking, here is a flavor of how our chunking code works, with chunking size of 2. Our sentence is “i am a girl in the house”. After Chunking, we have the sentences split into many parts, each of size 2 (except the last chunk): [‘i am’, ‘a girl’, ‘in the’, ‘house’]
The reason why we bring in these 2 concepts is because they will be used in the main code chunks under section Main Code : Average Word Embedding for Long Sentences and Main Code : Max Word Embedding for Long Sentences below.
# core code for finding average word embedding per sentence
def main_avg(sent: str, layers=None, chunk_size=300):
    """Gives the average word embedding per sentence

    Args:
        sent (str): The input sentence

    Returns:
        torch tensor: word embedding per sentence, dim = 768
    """
    # change all standard form numbers to decimal
    np.set_printoptions(formatter={"float_kind": "{:f}".format})

    # Use last four layers by default
    layers = [-4, -3, -2, -1] if layers is None else layers
    global tokenizer
    global model

    # chunking
    chunked_tokens = chunking(chunk_size, sent)  # helper func 3

    # initialise a outside chunk
    word_embedding_avg_collective = []
    # for each chunked token, we embed them separately
    for item in chunked_tokens:
        # adding tensors
        word_embedding_torch = get_hidden_states(
            item, tokenizer, model, layers
        )  # helper fun 2

        # convert torch tensor to numpy array
        word_embedding_avg_np = word_embedding_torch.cpu().detach().numpy()
        word_embedding_avg_chunks = np.mean(word_embedding_avg_np, axis=0)
        word_embedding_avg_collective.append(word_embedding_avg_chunks)
    word_embedding_avg = np.mean(word_embedding_avg_collective, axis=0)
    assert len(word_embedding_avg) == 768
    return word_embedding_avg

main_avg("AnfaGrow is committed to identifying and mitigating climate-related risks that have the potential to impact our operations, supply chain and distribution network through our risk management process. We also have a process for identifying and acting on opportunities to address healthcare needs that are emerging as a result of climate change and to increase operating efficiencies by reducing our climate-related impacts through the management of our emissions, water and waste streams. Through these processes, we have determined that climate-related risks and opportunities exist at site and regional levels throughout our value chain, however, Abbott is not exposed to substantive climate related risks or opportunities at a global level. Risks The risks that we analyze include physical and transitional risks that result from emerging regulation and new expectations of our businesses, and risk exposure through our suppliers and customers. We assess and manage climate-related risks through our integrated, company-wide risk management process, which identifies opportunities to build resilience in both our operations and our business model. Abbott regularly updates risk-management policies, standards and programs to align with global best practices and regulatory requirements and aims to anticipate emerging risks and upcoming regulatory changes. Our enterprise risk management (ERM) process identifies and evaluates the most critical risks for our business and provides guidance to our Board of Directors and management team. For more information on our ERM approach, go to the Sustainability Overview section in our 2019 Global Sustainability Report on, pg. XX10. Our EHS governance team monitors emerging climate-related trends and regulations to analyze the potential impacts on our business, understand our risk exposures and develop appropriate management strategies. Our EHS, Economics and Business Continuity organizations use scenario sensitivity risk-modeling to understand the financial implications of climate-related risks. Recently, we have analyzed potential risks resulting from the COP21 agreements on climate change, carbon taxes, water scarcity and the impact of climate change on agricultural supply chains. Abbott's Business Continuity and Crisis Management organizations implements measures to ensure business continuity and minimize the financial impacts of physical climate-related risks and other disruptions. These physical risks fall into two categories: acute and chronic. The acute physical risks associated with climate change include unforeseen extreme weather events. We cannot develop a strategy to prevent such events. However, we have developed a strategy for mitigating and responding to them. Our Business Continuity and Crisis Management organizations work with our EHS, Engineering and Supply Chain groups to identify and implement measures that will strengthen business resilience in the face of extreme weather events. In addition, our Engineering and EHS policies and management standards consider chronic physical risks, such as water scarcity, and require sites to conduct regular risk and opportunity evaluations and implement mitigation strategies. Abbott's Supply Chain Council (SCC) oversees the development of our global supply chain strategy, meeting quarterly and reporting to the Global Operations Council as needed. The SCC strives to achieve a sustainable and resilient supply chain and works to embed a consistent approach to identify and manage sustainability risks. Abbott's SCC and Business Continuity group use a real-time risk intelligence and supplier mapping tool to track the geopolitical, security, sustainability, environmental and infrastructure risks that could affect Abbott's supply chain. Through this system, we have identified several suppliers which could be at high risk of losing manufacturing capacity due to natural disasters, and all Abbott businesses have prepared contingency plans for such catastrophic events. We also use the World Resources Institute Aqueduct Tool(tm) to determine which suppliers have the greatest risk of water supply interruptions. The SCC has established multiple initiatives to embed sustainability as a key element of our supplier partnerships. These initiatives enable us to manage risks systematically and to identify opportunities to enhance the sustainability of our business. Our analyses of climate-related risks conclude that Abbott is not exposed to climate-related risks that could generate a substantive change in business operations, revenue or expenditure at a corporate level. Limited physical risks exist at site and regional operation levels, and throughout our supply chain. However, our diverse geographical distribution significantly mitigates the potential for substantive impact on our business from climate-related risks. Our most significant climate-related risks are transition risks that relate to emerging expectations and regulations around businesses managing their GHG emissions. These risks include carbon limits and taxes, enhanced emissions-reporting obligations, costs to transition to lower emissions technologies, and increased costs of goods and services. We have identified the global expectation that businesses should manage and mitigate their environmental impacts as an enterprise risk for Abbott. In response to this, Abbott's business strategy includes reducing the energy and carbon footprint of our operations and engage our value chain in strategic sourcing categories. Opportunities As part of our product research and development and climate risk management processes, we consider opportunities that might result from climate change. These fall into two main categories: increased operating efficiencies as a result of achieving carbon reduction targets, and the opportunity to advance our mission to help people live their best lives by meeting changing healthcare and nutrition needs. Compared to many industries, Abbott's carbon footprint is relatively small, and our progress in meeting our 2020 carbon reduction targets mean that we have fewer opportunities to realize significant operating efficiencies and cost savings in the future. Although further opportunities in this area exist, they are unlikely to have a substantive financial impact on our business. Changes to the climate have the potential to influence the disease burden and result in increased need for the pharmaceutical, diagnostic, medical device and nutrition products that Abbott makes. However, we do not anticipate that this will have a substantive financial or strategic impact on our business. We will continue to respond to increased humanitarian needs as a result of severe weather events and changes in the spread of disease in line with our caring values, and primarily through our philanthropic organization and product donations. ", chunk_size = 300)

array([-2.069005, -1.297163, 1.897537, -0.665995, 0.215819, -1.238932,
       -0.999157, -2.848238, -1.761889, -0.469389, 1.362041, -2.107933,
       1.356710, -0.067856, 2.261845, 0.048849, 0.386831, -0.160173,
       -1.735833, 3.672765, -1.414307, 1.424178, -1.822046, -0.061529,
       0.623235, 0.206411, 1.620223, 2.997343, -1.630599, -1.730808,
       0.114404, -3.037097, 0.290654, -1.588261, 0.338137, 2.589040,
       2.080817, -3.027153, 2.825588, -0.232242, -1.011434, 1.281097,
       -1.854541, -0.444737, 0.266219, 4.217216, 1.950694, 0.563303,
       0.542498, 1.628319, 0.773733, 0.366540, -2.902859, 0.287877,
       -0.301168, -2.505357, -1.090314, -0.509069, 1.364079, 1.693157,
       -2.438950, -1.296722, 0.521895, 0.925156, 2.336379, 1.796914,
       0.550823, -1.007046, 0.725143, 0.394005, -0.203064, -1.975158,
       2.703486, -1.300529, 1.583893, 1.269207, 0.284467, 3.265515,
       -0.190064, -3.448970, 1.260846, -0.863609, -1.235532, -0.758569,
       -1.308591, -2.970235, -2.651636, 0.744518, -0.415437, -0.829547,
       -1.385304, 0.239445, -0.251578, 0.277904, 0.851987, -2.487554,
       -2.076280, 0.327125, -0.321944, -1.598023, -0.432868, 2.055221,
       -0.785362, -0.947012, 1.188459, 2.012056, -0.635362, 4.557046,
       2.394783, -3.550794, 0.203139, -0.759144, -1.395220, -2.396142,
       3.586489, -0.007602, 0.722289, -3.357730, -2.507507, 0.880027,
       0.482132, -0.430051, -1.860358, 0.348181, 2.567036, -1.817246,
       -0.776126, 3.613238, 0.233643, 0.760583, 1.806730, -1.326855,
       -0.457524, -4.550256, -0.367592, 2.309610, 2.461022, -1.380205,
       0.392285, 1.736651, 2.548849, -2.003599, 0.515821, 4.238533,
       -0.754656, -0.666439, -0.797021, 0.339927, -1.409508, 1.866766,
       -1.513095, 0.142373, 1.340692, -1.320463, 2.835776, 1.564724,
       -0.820446, -2.372817, -0.170672, 0.795012, 0.312633, 1.567399,
       1.021786, 0.600795, 1.333246, -0.685193, -1.100175, 5.094521,
       0.402451, 0.481791, -1.190105, -1.983706, 0.747961, 1.988846,
       -0.128537, -2.466100, 0.580541, 0.290726, 0.088725, 0.345864,
       0.334719, 3.761334, -1.691322, 3.115104, 2.861830, 1.005932,
       -2.443249, 0.757257, -2.688693, -2.031623, -0.161037, -0.448210,
       -0.350518, -0.666195, -2.575506, -2.318582, 0.292609, -7.959647,
       -2.886156, -0.650922, -1.620792, 0.924887, 1.365774, -0.135699,
       3.450950, -3.989872, 2.725019, -0.047472, 0.498871, 0.481255,
       0.173682, -0.307382, 2.346156, -1.808317, 0.584965, 0.210533,
       -2.529187, 1.804229, 1.870696, 1.645651, 0.571733, 0.809525,
       -1.475625, -1.436652, -3.185799, -0.345234, -1.044051, -2.628528,
       -0.918885, -0.049260, 1.038188, -1.176445, -1.899982, 0.064409,
       -0.379223, -3.667358, 1.095597, 0.267653, -1.672376, 0.215453,
       -0.371529, -0.316430, -3.101951, -1.943470, -1.070211, -3.104949,
       2.773751, -0.620517, 1.545586, 1.962791, 1.686239, -0.436328,
       3.077949, 0.104622, 2.061519, 1.058273, -0.588779, -1.157057,
       0.427531, 0.867689, 3.583284, 2.010793, 2.349355, -2.801258,
       -2.065550, 2.587725, 2.115925, -0.521326, 1.736003, -2.004060,
       5.184472, 0.844518, -1.429248, 1.665160, 3.833745, -2.302060,
       3.280442, 0.615262, 0.681306, 0.391300, -0.002604, 1.934099,
       1.491943, 2.808527, -0.704851, 1.573778, -1.139348, -1.414605,
       0.213634, -2.269306, 2.220178, 4.204537, -0.942772, -2.444116,
       -1.013936, -1.648879, 0.564564, -2.690284, -2.970220, -0.177509,
       -0.179317, 4.487469, 1.832992, -1.014761, -1.714701, 0.821974,
       0.888527, 1.030442, -1.444443, 2.014353, -0.791166, -0.006726,
       -1.593742, 1.743300, 1.183587, -1.651197, 2.127725, -1.631748,
       -0.495837, 1.745529, -0.272165, -1.946047, 0.447692, -0.059874,
       2.515031, 0.054065, -2.281726, -1.561723, 0.485854, -0.786024,
       -1.929316, 2.341942, 2.672265, -0.840231, 1.952047, -1.037010,
       1.177897, 1.605331, 0.860934, 2.921018, 1.187772, 0.267826,
       -3.725641, -1.408114, -1.479047, -0.270847, 0.620572, -1.538835,
       0.983706, -1.765637, -0.780493, 0.177010, -1.076528, 2.151928,
       0.785024, 0.313175, -1.400213, -1.645296, 0.475628, 2.944529,
       -0.853154, -1.869930, 0.006834, -0.974114, 1.117081, 1.777257,
       -0.562039, -3.460936, 1.671339, 2.371669, -0.097315, 1.901112,
       -4.219271, 1.878993, -1.902072, 1.240675, -0.908544, 2.959465,
       0.775912, -2.523776, 1.571106, 1.816986, -0.338277, -1.477727,
       3.557108, -1.095492, 1.652628, -0.663556, 1.288636, -0.974580,
       2.178181, 1.691398, 1.718664, 0.912274, -0.533265, -0.515311,
       0.263380, 2.008967, 0.134322, 0.597025, -0.629317, 0.526631,
       -4.652263, -0.000153, 0.177407, 2.026528, 1.908481, 0.442277,
       -1.333624, -1.062187, 0.651182, -1.927337, 0.767219, 0.442638,
       -1.025325, -1.917132, 2.909675, 0.779187, 2.179921, 1.743582,
       2.541965, -1.247596, 0.683323, -2.468955, 13.741922, 0.994043,
       0.591783, 0.095969, 1.944384, 1.311726, -0.146098, 0.673918,
       3.255137, -1.523252, 0.438895, 3.686114, -3.276689, -1.295597,
       0.344556, -1.022516, -0.727196, 1.285344, -2.086396, 1.698545,
       0.539153, -2.891305, 0.273943, 0.540950, 0.897944, -0.719528,
       -1.898919, 0.710078, 1.759045, -3.394097, 0.832057, -1.126649,
       0.619468, 1.614887, 1.009596, 0.658375, 0.667145, -0.440513,
       2.644860, 4.376307, -1.868778, -0.073416, 3.739684, 2.730416,
       1.686562, 2.766210, 1.131445, 0.188842, -0.020838, 0.826908,
       -3.838250, 0.976980, -1.591110, -3.479329, -0.207301, 2.005702,
       -2.753670, 0.220943, -3.397778, -1.830705, -2.317495, -0.271421,
       -2.866215, 0.905667, 3.118837, -0.239333, -2.592990, -0.579240,
       3.386519, -0.387755, 1.659416, -1.062404, 0.316701, 0.402465,
       0.017823, -1.303568, 0.100284, -0.927823, -1.501460, -0.180206,
       0.394587, 0.342294, 3.511187, 2.194754, -1.653516, 0.493785,
       -0.154764, 0.711610, 1.622528, 3.551189, -0.606257, 0.583629,
       -2.075105, 0.828311, 0.490017, -0.242381, 0.157428, 1.983957,
       -0.916390, 1.280473, -0.248426, -0.294099, -3.890427, -3.039372,
       -1.897835, 0.668367, 2.314362, -1.061765, -3.239893, -3.059426,
       0.750577, -1.425263, 2.725688, 3.173386, -0.875738, 0.509152,
       2.292987, 0.717704, 2.612330, 2.104307, 0.809621, 1.893720,
       -3.241573, -0.584649, -2.078237, 0.095293, 0.455616, 0.876588,
       -0.694242, 1.920480, 3.844588, 0.671297, -0.546143, 1.247792,
       1.140373, 2.001224, -0.280107, -0.301192, -2.428901, 0.317335,
       -0.915918, 2.041941, 0.303396, 0.898906, 3.914610, -0.215909,
       1.314461, 0.590366, -1.244806, -1.782811, 0.924122, -2.094736,
       3.718682, 1.484810, -0.891566, -3.178143, -0.091641, -0.381061,
       -0.606722, -3.654816, -1.575541, -0.135408, 1.038194, 0.061292,
       -1.190889, 1.320670, 1.765615, -0.687588, 3.432351, -0.602367,
       2.631968, 0.069879, 3.240031, 1.993984, -2.576908, -1.206025,
       1.138778, 0.281970, 0.564534, 1.061519, 0.881408, -0.029467,
       -2.600369, 0.211318, 3.493402, 0.651741, -0.237523, -0.073185,
       0.962477, 1.953131, -0.196566, 0.273639, -1.838220, -0.718419,
       -2.832070, 2.028375, 0.730685, 1.083714, 2.637357, 2.220489,
       0.692680, 0.767908, -0.003996, 3.317253, -0.597663, -1.570525,
       1.016961, -0.420814, 2.685907, -1.169058, -0.967749, -2.022867,
       -0.882033, 0.328596, 0.917013, -0.376423, 1.400107, 2.684333,
       1.160401, 1.298169, 2.544685, 2.309297, 2.290207, -2.092325,
       0.480673, 0.883918, 2.835300, 2.810617, -1.591112, -0.354609,
       -3.252123, 0.228087, -1.432474, -0.250098, -0.263726, -0.538434,
       0.408937, -2.967703, 1.646640, 1.858239, -1.274054, -2.258950,
       0.020468, 1.028286, 0.556833, -0.787763, 1.857360, 2.093471,
       0.026893, 1.356672, 1.541057, -3.274049, 0.884621, 3.906209,
       2.133838, -1.554275, 0.155315, -1.096956, -0.234297, 1.507194,
       0.603697, 0.695325, -2.161526, 3.551227, -2.653571, 1.743917,
       1.869280, 1.807632, 0.400632, 0.869061, -1.350228, 1.132255,
       0.911014, 1.937392, -2.885293, 1.726256, 0.224319, -4.401726,
       -1.517258, -0.703586, 0.186629, 1.275138, 1.743060, -1.976776,
       -3.978956, -0.765919, -0.282317, -1.532714, 1.087273, 4.616387,
       -5.265229, -1.123801, 3.258021, 4.042605, -2.139258, 0.074779,
       1.418722, 1.080052, -0.934217, -1.811041, 1.130777, -3.233244,
       -0.139647, -2.362241, -0.200611, 3.119538, -0.986597, 1.201791,
       1.385224, -0.332644, -0.690881, 0.504689, -1.698751, 2.526792,
       -1.814546, 0.759662, 0.907874, -2.027709, -2.131575, 2.287480,
       -3.240891, -0.088914, 2.539668, -2.114273, 0.780708, 1.237171,
       -0.536391, -1.768716, -0.910431, -0.603116, -0.510919, -2.319856,
       -0.549336, 0.452244, 0.796902, 0.673577, 1.388029, -0.551370,
       1.831863, -1.895629, -2.641852, 2.489860, 1.070573, 2.935132],
      dtype=float32)
 

# core code for finding max word embedding per sentence
def main_max(sent, layers=None, chunk_size=300):
    """Gives the max word embedding per sentence

    Args:
        sent (str): The input sentence

    Returns:
        torch tensor: word embedding per sentence, dim = 768
    """
    # change all standard form numbers to decimal
    np.set_printoptions(formatter={"float_kind": "{:f}".format})
    # Use last four layers by default
    layers = [-4, -3, -2, -1] if layers is None else layers
    # tokenizer = AutoTokenizer.from_pretrained(esg_bert_path)
    # model = AutoModel.from_pretrained(esg_bert_path, output_hidden_states=True)
    global tokenizer
    global model

    # chunking
    chunked_tokens = chunking(chunk_size, sent)  # helper func 3
    # # make sure the len(chunked_tokens) > 1 or else some of the embeddings will appear as len 1 instead of 768. 
    # initialise a outside chunk
    word_embedding_max_collective = []
    # for each chunked token, we embed them separately
    for item in chunked_tokens:
        # adding tensors
        word_embedding_torch = get_hidden_states(
            item, tokenizer, model, layers
        )  # helper func 2

        # convert torch tensor to numpy array
        word_embedding_max_np = word_embedding_torch.cpu().detach().numpy()
        word_embedding_max_chunks = np.amax(word_embedding_max_np, axis=0)
        word_embedding_max_collective.append(word_embedding_max_chunks)
    word_embedding_max = np.amax(word_embedding_max_collective, axis=0)
    assert len(word_embedding_max) == 768
    return word_embedding_max

main_max("AnfaGrow is committed to identifying and mitigating climate-related risks that have the potential to impact our operations, supply chain and distribution network through our risk management process. We also have a process for identifying and acting on opportunities to address healthcare needs that are emerging as a result of climate change and to increase operating efficiencies by reducing our climate-related impacts through the management of our emissions, water and waste streams. Through these processes, we have determined that climate-related risks and opportunities exist at site and regional levels throughout our value chain, however, Abbott is not exposed to substantive climate related risks or opportunities at a global level. Risks The risks that we analyze include physical and transitional risks that result from emerging regulation and new expectations of our businesses, and risk exposure through our suppliers and customers. We assess and manage climate-related risks through our integrated, company-wide risk management process, which identifies opportunities to build resilience in both our operations and our business model. Abbott regularly updates risk-management policies, standards and programs to align with global best practices and regulatory requirements and aims to anticipate emerging risks and upcoming regulatory changes. Our enterprise risk management (ERM) process identifies and evaluates the most critical risks for our business and provides guidance to our Board of Directors and management team. For more information on our ERM approach, go to the Sustainability Overview section in our 2019 Global Sustainability Report on, pg. XX10. Our EHS governance team monitors emerging climate-related trends and regulations to analyze the potential impacts on our business, understand our risk exposures and develop appropriate management strategies. Our EHS, Economics and Business Continuity organizations use scenario sensitivity risk-modeling to understand the financial implications of climate-related risks. Recently, we have analyzed potential risks resulting from the COP21 agreements on climate change, carbon taxes, water scarcity and the impact of climate change on agricultural supply chains. Abbott's Business Continuity and Crisis Management organizations implements measures to ensure business continuity and minimize the financial impacts of physical climate-related risks and other disruptions. These physical risks fall into two categories: acute and chronic. The acute physical risks associated with climate change include unforeseen extreme weather events. We cannot develop a strategy to prevent such events. However, we have developed a strategy for mitigating and responding to them. Our Business Continuity and Crisis Management organizations work with our EHS, Engineering and Supply Chain groups to identify and implement measures that will strengthen business resilience in the face of extreme weather events. In addition, our Engineering and EHS policies and management standards consider chronic physical risks, such as water scarcity, and require sites to conduct regular risk and opportunity evaluations and implement mitigation strategies. Abbott's Supply Chain Council (SCC) oversees the development of our global supply chain strategy, meeting quarterly and reporting to the Global Operations Council as needed. The SCC strives to achieve a sustainable and resilient supply chain and works to embed a consistent approach to identify and manage sustainability risks. Abbott's SCC and Business Continuity group use a real-time risk intelligence and supplier mapping tool to track the geopolitical, security, sustainability, environmental and infrastructure risks that could affect Abbott's supply chain. Through this system, we have identified several suppliers which could be at high risk of losing manufacturing capacity due to natural disasters, and all Abbott businesses have prepared contingency plans for such catastrophic events. We also use the World Resources Institute Aqueduct Tool(tm) to determine which suppliers have the greatest risk of water supply interruptions. The SCC has established multiple initiatives to embed sustainability as a key element of our supplier partnerships. These initiatives enable us to manage risks systematically and to identify opportunities to enhance the sustainability of our business. Our analyses of climate-related risks conclude that Abbott is not exposed to climate-related risks that could generate a substantive change in business operations, revenue or expenditure at a corporate level. Limited physical risks exist at site and regional operation levels, and throughout our supply chain. However, our diverse geographical distribution significantly mitigates the potential for substantive impact on our business from climate-related risks. Our most significant climate-related risks are transition risks that relate to emerging expectations and regulations around businesses managing their GHG emissions. These risks include carbon limits and taxes, enhanced emissions-reporting obligations, costs to transition to lower emissions technologies, and increased costs of goods and services. We have identified the global expectation that businesses should manage and mitigate their environmental impacts as an enterprise risk for Abbott. In response to this, Abbott's business strategy includes reducing the energy and carbon footprint of our operations and engage our value chain in strategic sourcing categories. Opportunities As part of our product research and development and climate risk management processes, we consider opportunities that might result from climate change. These fall into two main categories: increased operating efficiencies as a result of achieving carbon reduction targets, and the opportunity to advance our mission to help people live their best lives by meeting changing healthcare and nutrition needs. Compared to many industries, Abbott's carbon footprint is relatively small, and our progress in meeting our 2020 carbon reduction targets mean that we have fewer opportunities to realize significant operating efficiencies and cost savings in the future. Although further opportunities in this area exist, they are unlikely to have a substantive financial impact on our business. Changes to the climate have the potential to influence the disease burden and result in increased need for the pharmaceutical, diagnostic, medical device and nutrition products that Abbott makes. However, we do not anticipate that this will have a substantive financial or strategic impact on our business. We will continue to respond to increased humanitarian needs as a result of severe weather events and changes in the spread of disease in line with our caring values, and primarily through our philanthropic organization and product donations. ", chunk_size = 300)

array([6.805729, 4.375031, 8.499806, 4.177418, 7.693119, 5.701173,
       5.759755, 2.837549, 4.798483, 4.237041, 7.684998, 4.353240,
       7.267815, 6.053005, 9.581602, 5.607996, 6.679993, 6.245728,
       5.893171, 10.088954, 5.040035, 8.295496, 3.458732, 7.600258,
       7.293615, 6.809679, 6.654215, 8.309592, 2.088390, 4.805450,
       7.344614, 3.813967, 7.236437, 4.843852, 5.681024, 7.938098,
       7.224466, 4.748833, 10.679108, 5.508364, 5.299173, 7.419199,
       3.833404, 5.834118, 6.500022, 10.382873, 7.653900, 5.775349,
       6.764424, 6.309328, 6.895959, 4.687091, 4.291939, 6.160853,
       7.253861, 2.953460, 5.401671, 5.019510, 9.307771, 7.323975,
       2.089552, 3.630132, 5.970394, 7.337933, 9.680780, 7.439181,
       6.601732, 4.498766, 6.963731, 4.683110, 4.251820, 3.030715,
       9.179097, 6.470676, 8.088331, 7.020998, 6.626089, 10.865126,
       5.770435, 2.637362, 5.836210, 5.401630, 3.545075, 4.714892,
       3.428622, 3.691884, 3.075176, 5.585213, 6.426898, 5.540763,
       5.580419, 6.395550, 5.979355, 7.106596, 8.906451, 3.227191,
       5.802490, 5.183506, 5.376815, 3.875749, 5.168766, 8.254588,
       5.673604, 3.662191, 6.089957, 7.365190, 6.295435, 11.111140,
       7.112841, 3.332601, 6.150575, 5.596840, 4.675416, 2.858708,
       9.215329, 6.099337, 6.379060, 4.148998, 4.122796, 6.290338,
       7.184327, 5.727400, 3.888063, 7.060883, 7.980391, 5.235572,
       3.656922, 9.060629, 6.921495, 7.413661, 7.879916, 4.478697,
       3.912730, 0.552235, 4.446080, 8.466642, 8.916345, 5.135750,
       6.022637, 8.585208, 9.375729, 5.106371, 5.547313, 9.557014,
       5.775074, 5.713921, 5.857843, 7.367391, 3.234677, 8.363438,
       3.941983, 5.846553, 7.585428, 5.481585, 9.992062, 7.533081,
       6.051783, 4.063644, 5.017175, 7.098746, 6.415111, 5.696528,
       6.054219, 5.832032, 7.412758, 6.475196, 3.846331, 12.894347,
       6.995192, 7.470239, 3.178046, 4.415548, 6.390093, 9.079231,
       5.689376, 2.426248, 8.353623, 6.165197, 6.171763, 5.958351,
       7.100211, 10.389503, 9.461224, 10.796867, 8.868233, 8.157881,
       4.526734, 5.352566, 2.892432, 4.582036, 5.564053, 4.780064,
       6.816301, 4.004356, 3.032656, 2.637586, 5.728351, 1.026220,
       4.909800, 4.118415, 3.467223, 7.797954, 7.754595, 4.838362,
       8.988747, 1.152650, 7.820503, 5.239417, 5.757623, 6.117602,
       5.643327, 4.177190, 8.146049, 5.146054, 7.309677, 7.016794,
       5.650390, 7.236885, 8.965467, 8.028029, 5.587788, 6.494536,
       5.150053, 5.450854, 5.070943, 4.730285, 5.828648, 1.485036,
       4.567852, 6.316761, 7.183798, 2.937900, 4.750881, 5.106349,
       5.999448, 2.051567, 7.127003, 6.910133, 3.605593, 7.360428,
       4.130997, 5.102177, 4.089386, 3.832532, 3.615499, 1.657634,
       9.053363, 4.561336, 8.644236, 8.032099, 6.327707, 3.200377,
       9.146029, 8.562849, 9.459332, 9.052984, 6.188026, 6.385575,
       6.301089, 6.903450, 10.085500, 6.795608, 7.047089, 3.486167,
       4.859134, 7.766998, 8.343951, 5.674760, 7.365826, 4.766391,
       15.006124, 7.172134, 6.723362, 6.469397, 9.553068, 2.340155,
       9.262548, 5.140223, 6.883518, 5.824446, 5.628864, 8.473395,
       7.071916, 9.562339, 6.400918, 7.307479, 3.749036, 4.175467,
       6.500440, 6.475127, 9.826133, 9.049512, 5.468864, 4.091249,
       8.175467, 4.666560, 8.040548, 4.196706, 3.070106, 5.884329,
       4.571704, 11.202284, 8.328909, 6.450729, 4.792257, 5.494537,
       7.930249, 5.915768, 2.870098, 8.418445, 4.845572, 6.503769,
       2.733168, 6.955701, 6.519325, 3.269755, 6.789036, 3.157291,
       4.585633, 7.521475, 6.293416, 3.227533, 7.457714, 6.743287,
       9.444475, 7.705032, 4.500249, 4.958095, 6.090357, 4.394763,
       2.924061, 7.130053, 8.833180, 5.279778, 7.348455, 5.984032,
       5.934505, 7.695699, 5.507162, 7.657325, 6.087687, 7.831409,
       1.911556, 3.260344, 3.947514, 6.945468, 6.330166, 2.987940,
       7.734982, 0.182710, 5.220557, 5.619095, 5.072118, 7.647855,
       8.119056, 5.433576, 4.063998, 4.087398, 6.151192, 7.773042,
       3.780213, 4.423661, 6.548451, 6.078121, 6.701793, 7.130221,
       4.275297, 6.344665, 6.957475, 10.174275, 5.934914, 7.070611,
       2.025634, 7.238626, 3.172354, 7.098652, 4.232665, 8.146763,
       5.380277, 2.300678, 7.858373, 9.842705, 6.569454, 6.777760,
       10.568830, 4.292348, 7.191491, 4.701573, 7.716341, 5.952081,
       6.451053, 7.079156, 8.560876, 5.653956, 4.720266, 5.908130,
       6.149865, 7.089313, 5.232348, 5.848696, 5.590966, 7.477598,
       4.163276, 7.376535, 4.625238, 7.979105, 7.250296, 5.290232,
       4.713013, 3.737339, 5.247944, 5.262582, 7.221301, 6.886269,
       4.813965, 5.928740, 8.215522, 7.607764, 8.691781, 7.860248,
       9.430499, 4.279301, 7.011847, 3.972684, 32.300091, 2.865953,
       5.031526, 6.522912, 8.147434, 6.665096, 4.293168, 6.163336,
       10.416369, 5.503635, 5.361992, 9.289345, 4.719818, 3.249290,
       6.780697, 4.554296, 7.221393, 8.098908, 3.996507, 7.343800,
       6.690678, 4.636441, 5.651355, 7.601608, 7.564597, 6.703537,
       3.494119, 8.005252, 7.775165, 3.662652, 9.142111, 2.804011,
       5.672795, 7.476822, 7.724471, 6.644853, 5.233309, 5.412231,
       7.691191, 8.917459, 4.478935, 7.309610, 11.818733, 9.749537,
       7.696684, 7.882431, 7.774083, 5.964764, 5.990480, 7.217134,
       4.597292, 5.867262, 3.982222, 2.412991, 4.917072, 7.577150,
       3.669922, 4.394014, 2.582626, 6.417785, 3.475491, 6.748754,
       3.031434, 5.628732, 9.112783, 6.335259, 1.996946, 3.821673,
       10.063855, 4.939135, 7.363873, 5.625710, 5.825156, 6.724142,
       4.690587, 5.713294, 5.413768, 6.135306, 4.844501, 6.006926,
       7.357086, 5.527079, 10.448018, 7.634327, 5.560106, 6.333005,
       7.590524, 7.962666, 6.939282, 9.490266, 4.328447, 6.461566,
       2.730730, 6.845242, 8.313006, 6.625759, 7.447044, 8.547960,
       4.010792, 5.927190, 5.564487, 5.163180, 3.277546, 2.685732,
       3.975101, 6.546746, 8.117289, 6.080804, 2.456064, 2.684086,
       5.535557, 4.072181, 9.164190, 7.395768, 5.749873, 5.853021,
       7.265991, 6.674250, 8.307633, 8.752364, 6.309198, 8.316397,
       3.095292, 7.256077, 3.592412, 4.300582, 6.738859, 7.472331,
       4.722396, 7.689555, 11.490110, 5.941499, 5.851166, 6.881021,
       6.373092, 8.272969, 4.891885, 5.160688, 4.903255, 4.980679,
       5.566558, 9.127219, 6.083465, 7.551468, 9.129075, 6.195861,
       6.292424, 6.850673, 4.893875, 3.284906, 8.260836, 2.866710,
       10.516780, 7.312702, 5.245462, 1.828476, 6.239605, 4.518696,
       5.732191, 2.357109, 5.149320, 5.657053, 6.577658, 5.744939,
       3.792763, 8.132065, 7.683682, 5.053378, 10.952696, 4.332111,
       9.554989, 6.372986, 8.683263, 8.160653, 3.606551, 4.544869,
       8.267568, 6.115463, 6.919366, 7.949358, 5.922500, 6.529612,
       4.584568, 5.173990, 9.414206, 4.843125, 5.128804, 5.705178,
       6.912664, 8.732327, 5.604402, 7.313241, 5.538630, 7.026895,
       3.158371, 8.041910, 6.217119, 6.549814, 10.392530, 7.389754,
       6.669248, 6.936916, 5.996890, 8.194631, 5.582872, 5.543167,
       6.238813, 4.840163, 8.118135, 4.046813, 4.874819, 4.681991,
       8.245940, 5.343390, 7.156136, 5.574592, 6.897003, 9.771425,
       6.981195, 7.415888, 9.609451, 7.410915, 7.572303, 3.177071,
       6.570842, 7.253745, 8.597585, 10.470009, 5.051155, 4.755615,
       3.082440, 8.632995, 5.222728, 5.451725, 6.135768, 5.451841,
       4.627967, 1.898759, 9.465576, 8.560442, 3.892127, 3.536751,
       4.631108, 8.156595, 5.920186, 5.704526, 8.252375, 8.076779,
       4.937462, 5.866892, 6.098383, 2.611325, 5.595641, 9.153108,
       8.461097, 3.781582, 6.006515, 6.231102, 4.817896, 7.453861,
       7.450366, 7.510406, 5.073956, 10.756120, -0.197574, 6.595834,
       7.707531, 8.029993, 5.597455, 7.535989, 4.943944, 7.106375,
       8.384113, 7.115477, 4.723933, 7.093085, 5.010959, 1.528501,
       6.512274, 6.686188, 5.430101, 12.571142, 6.973195, 4.258281,
       5.268918, 4.470957, 5.517951, 3.711493, 5.930163, 10.046264,
       0.903439, 7.659471, 9.644814, 9.140053, 3.121001, 5.704132,
       7.377662, 4.873896, 5.747383, 3.702909, 8.485018, 4.704550,
       5.952401, 3.781699, 4.562886, 9.798097, 4.356740, 5.817946,
       6.443051, 4.240712, 5.443954, 7.371186, 3.724655, 8.130217,
       3.101558, 5.572737, 5.885617, 4.477139, 3.491349, 6.526273,
       3.003878, 6.064904, 9.570217, 3.721537, 7.470549, 6.516582,
       4.123355, 4.088974, 6.077446, 4.823407, 5.552065, 6.523480,
       6.629506, 7.912446, 6.348028, 6.374221, 7.070266, 4.981774,
       7.481637, 3.770473, 3.833290, 12.463604, 6.095844, 8.423599],
      dtype=float32)
 

First take away: This is a simple method to overcome the restrictions in the sequence length when doing word embedding.
Second take away: This tutorial talks about how to find average and max word embedding of the whole sentence/sequence.
Hope you found it helpful!
Disclaimer: Content is generated by the author and not affiliated to any organization