Multi Threads Transformer Model

username: gnadaf
cooked: <p>Can we train Transformer model using 2 threads?<br>
first thread on GPU-1<br>
second threads on GPU-2?</p>
<p>Is it possible?</p>

username: BramVanroy
cooked: <p>What you are describing sounds like a typical distributed data parallel set up, which works with transformer models indeed. You can look it up for more information.</p>

username: gnadaf
cooked: <aside class="quote no-group" data-username="BramVanroy" data-post="2" data-topic="97744">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bramvanroy/48/28628_2.png" class="avatar"> BramVanroy:</div>
<blockquote>
<p>u can look it up for more information</p>
</blockquote>
</aside>
<p>BramVanroy,</p>
<p>I experimented with nn.DataParallel(model),  For example, if I want to use 2 GPU<br>
but in this method, the entire model copy is copied on both GPUs</p>
<p>I would like to do, half data should be trained on one GPU and another half data on another GPU, eventually I can complete training in less time.</p>
<p>I wonder that is it possible to nn.DataParallel(model) ?</p>

username: BramVanroy
cooked: <p>Not DataParallel, but DistributedDataParallel, which does what you require.</p>

username: gnadaf
cooked: <p>ok, I will try with DistributedDataParallel</p>
<p>Thanks BramVanroy</p>

username: gnadaf
cooked: <aside class="quote no-group" data-username="gnadaf" data-post="5" data-topic="97744">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/cdc98d/48.png" class="avatar"> gnadaf:</div>
<blockquote>
<p>DistributedDataParallel</p>
</blockquote>
</aside>
<p>Since I am a beginner, understanding these frameoworks.</p>
<p>How to wrap the below code with DistributedDataParallel?</p>
<p><strong>class TransformerModel(nn.Module):</strong><br>
def <strong>init</strong>(self, ntoken, ninp, nhead, nhid, nlayers, dropout=0.5):<br>
super(TransformerModel, self).<strong>init</strong>()<br>
from torch.nn import TransformerEncoder, TransformerEncoderLayer<br>
self.model_type = ‘Transformer’<br>
self.src_mask = None<br>
self.pos_encoder = PositionalEncoding(ninp, dropout)<br>
encoder_layers = TransformerEncoderLayer(ninp, nhead, nhid, dropout)<br>
self.transformer_encoder = TransformerEncoder(encoder_layers, nlayers)<br>
self.encoder = nn.Embedding(ntoken, ninp)<br>
self.ninp = ninp<br>
self.decoder = nn.Linear(ninp, ntoken)<br>
self.init_weights()<br>
def <em>generate_square_subsequent_mask(self, sz):<br>
mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1)<br>
mask = mask.float().masked_fill(mask == 0, float(‘-inf’)).masked_fill(mask == 1, float(0.0))<br>
return mask<br>
def init_weights(self):<br>
initrange = 0.1<br>
self.encoder.weight.data.uniform</em>(-initrange, initrange)<br>
self.decoder.bias.data.zero_()<br>
self.decoder.weight.data.uniform_(-initrange, initrange)<br>
def forward(self, src):<br>
if self.src_mask is None or self.src_mask.size(0) != len(src):<br>
device = src.device<br>
mask = self._generate_square_subsequent_mask(len(src)).to(device)<br>
self.src_mask = mask<br>
src = self.encoder(src) * math.sqrt(self.ninp)<br>
src = self.pos_encoder(src)<br>
output = self.transformer_encoder(src, self.src_mask)<br>
output = self.decoder(output)<br>
return F.log_softmax(output, dim=-1)</p>
<p><strong>class PositionalEncoding(nn.Module):</strong><br>
def <strong>init</strong>(self, d_model, dropout=0.1, max_len=5000):<br>
super(PositionalEncoding, self).<strong>init</strong>()<br>
self.dropout = nn.Dropout(p=dropout)<br>
pe = torch.zeros(max_len, d_model)<br>
position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)<br>
div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))<br>
pe[:, 0::2] = torch.sin(position * div_term)<br>
pe[:, 1::2] = torch.cos(position * div_term)<br>
pe = pe.unsqueeze(0).transpose(0, 1)<br>
self.register_buffer(‘pe’, pe)<br>
def forward(self, x):<br>
x = x + self.pe[:x.size(0), :]<br>
return self.dropout(x)</p>

username: gnadaf
cooked: <p>I believe that DistributedDataParallel does not work with Transformer model,</p>