Multi-node multi-gpu for basic BERT tokens

username: vr25
cooked: <p>Hi,</p>
<p>I am following this popular <a href="https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/" rel="nofollow noopener">post</a> on understanding BERT.</p>
<p>I have posted my issue on GitHub <a href="https://github.com/huggingface/transformers/issues/2230" rel="nofollow noopener">here</a></p>
<p>I have access to a cluster with multiple nodes each having 4 GPUs but I am not sure how to use all for my task below.</p>
<p>I want to use say, 10 nodes x 4 GPUs = 40 GPUs. I read up on DP and DDP but I think I need to manually chunk my long document into chunks of several sentences and then assign each chunk to GPU.</p>
<p>Following is the code for the above task:</p>
<p>Blockquote</p>
<p>import torch</p>
<p>from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForMaskedLM</p>
<p>import logging<br>
<span class="hashtag">#logging</span>.basicConfig(level=logging.INFO)<br>
import pickle<br>
import nltk<br>
from nltk import word_tokenize, sent_tokenize<br>
import time<br>
import os<br>
import sys</p>
<p>start = time.time()</p>
<p>tokenizer = BertTokenizer.from_pretrained(‘bert-base-uncased’)</p>
<p>f = open(‘sample.txt’, ‘r’).read()<br>
<span class="hashtag">#print</span>("f: ", f)<br>
text = sent_tokenize(f)</p>
<p><span class="hashtag">#text</span> = [“After stealing money”, “The bank robber was seen”] # from the bank vault, the bank robber was seen " <br>
#“fishing on the Mississippi river bank.”<br>
<span class="hashtag">#marked_text</span> = “[CLS] " + text + " [SEP]”</p>
<p>i=1</p>
<p>for marked_text in text:<br>
<span class="hashtag">#marked_text</span> = text</p>
<pre><code># Tokenize our sentence with the BERT tokenizer.
tokenized_text = tokenizer.tokenize(marked_text)

# Print out the tokens.
print (tokenized_text)

# Add the special tokens.
#marked_text = "[CLS] " + text + " [SEP]"

# Split the sentence into tokens.
tokenized_text = tokenizer.tokenize(marked_text)

# Map the token strings to their vocabulary indeces.
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)

# Display the words with their indeces.
for tup in zip(tokenized_text, indexed_tokens):
    print('{:&lt;12} {:&gt;6,}'.format(tup[0], tup[1]))

# Mark each of the 22 tokens as belonging to sentence "1".
segments_ids = [1] * len(tokenized_text)

print (segments_ids) 

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Convert inputs to PyTorch tensors
tokens_tensor = torch.tensor([indexed_tokens]).to(device)
segments_tensors = torch.tensor([segments_ids]).to(device)


# Load pre-trained model (weights)
model = BertModel.from_pretrained('bert-base-uncased')

if torch.cuda.device_count() &gt; 1:
    model = torch.nn.DataParallel(model)

# Put the model in "evaluation" mode, meaning feed-forward operation.
model.to(device)#cuda()
model.eval()

# Predict hidden states features for each layer
with torch.no_grad():
    encoded_layers, _ = model(tokens_tensor, segments_tensors)

print ("Number of layers:", len(encoded_layers))
layer_i = 0

print ("Number of batches:", len(encoded_layers[layer_i]))
batch_i = 0

print ("Number of tokens:", len(encoded_layers[layer_i][batch_i]))
token_i = 0

print ("Number of hidden units:", len(encoded_layers[layer_i][batch_i][token_i]))

# `encoded_layers` is a Python list.
print('     Type of encoded_layers: ', type(encoded_layers))

# Each layer in the list is a torch tensor.
print('Tensor shape for each layer: ', encoded_layers[0].size())

token_embeddings = torch.stack(encoded_layers, dim=0)

print("token_embeddings_size: ", token_embeddings.size())

# Remove dimension 1, the "batches".
token_embeddings = torch.squeeze(token_embeddings, dim=1)

print("After removing batches dimension: ", token_embeddings.size())
print("First layer: ", token_embeddings)
print("Second layer: ", token_embeddings[1])
print("Third layer: ", token_embeddings[2])
print("Twelfth layer: ", token_embeddings[11])

torch.save(token_embeddings, "cuda_" + str(i) + '_hid_weights_tensor.pt')

i = i + 1
</code></pre>
<p>print("Total execution time: “, time.time() - start, " sec”)</p>
<p>Can you please give me some suggestions on how to use all GPUs for a single document?</p>
<p>I have a set of around 10K documents. So I think it would be more efficient to chunk the documents and then give each a GPU ID.</p>
<p>Thanks!</p>