Bert got error of device side assert tirggerred

username: Sangwon_Jake
cooked: <p>Train the bert with class number=5</p>
<p>import torch<br>
from kobert.pytorch_kobert import get_pytorch_kobert_model</p>
<p>import torch<br>
from torch import nn<br>
import torch.nn.functional as F<br>
import torch.optim as optim<br>
from torch.utils.data import Dataset, DataLoader<br>
import gluonnlp as nlp<br>
import numpy as np<br>
from tqdm import tqdm, tqdm_notebook</p>
<p>from kobert.utils import get_tokenizer<br>
from kobert.pytorch_kobert import get_pytorch_kobert_model</p>
<p>from transformers import AdamW<br>
from transformers.optimization import WarmupLinearSchedule</p>
<h2>Setting parameters</h2>
<p>max_len = 64<br>
batch_size = 64<br>
warmup_ratio = 0.1<br>
num_epochs = 1<br>
max_grad_norm = 1<br>
log_interval = 200<br>
learning_rate =  5e-5</p>
<p>input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]])<br>
input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]])<br>
token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]])</p>
<p>model, vocab  = get_pytorch_kobert_model()</p>
<p>all_encoder_layers, pooled_output = model(input_ids, token_type_ids, input_mask)<br>
pooled_output.shape</p>
<p>tokenizer = get_tokenizer()<br>
tok = nlp.data.BERTSPTokenizer(tokenizer, vocab, lower=False)</p>
<p>class BERTDataset(Dataset):<br>
def <strong>init</strong>(self, dataset, sent_idx, label_idx, bert_tokenizer, max_len,<br>
pad, pair):<br>
transform = nlp.data.BERTSentenceTransform(<br>
bert_tokenizer, max_seq_length=max_len, pad=pad, pair=pair)</p>
<pre><code>    self.sentences = [transform([i[sent_idx]]) for i in dataset]
    self.labels = [np.int32(i[label_idx]) for i in dataset]

def __getitem__(self, i):
    return (self.sentences[i] + (self.labels[i], ))

def __len__(self):
    return (len(self.labels))
</code></pre>
<p>data_train = BERTDataset(train_temp[0:2000], 1, 0, tok, max_len, True, False)<br>
data_test = BERTDataset(train_temp[2000:], 1, 0, tok, max_len, True, False)<br>
train_dataloader = torch.utils.data.DataLoader(data_train, batch_size=batch_size, num_workers=5)<br>
test_dataloader = torch.utils.data.DataLoader(data_test, batch_size=batch_size, num_workers=5)</p>
<p>class BERTClassifier(nn.Module):<br>
def <strong>init</strong>(self,<br>
bert,<br>
hidden_size = 768,<br>
num_classes=5,<br>
dr_rate=None,<br>
params=None):<br>
super(BERTClassifier, self).<strong>init</strong>()<br>
self.bert = bert<br>
self.dr_rate = dr_rate</p>
<pre><code>    self.classifier = nn.Linear(hidden_size , num_classes)
    if dr_rate:
        self.dropout = nn.Dropout(p=dr_rate)

def gen_attention_mask(self, token_ids, valid_length):
    attention_mask = torch.zeros_like(token_ids)
    for i, v in enumerate(valid_length):
        attention_mask[i][:v] = 1
    return attention_mask.float()

def forward(self, token_ids, valid_length, segment_ids):
    attention_mask = self.gen_attention_mask(token_ids, valid_length)
    
    _, pooler = self.bert(input_ids = token_ids, token_type_ids = segment_ids.long(), attention_mask = attention_mask.float().to(token_ids.device))
    if self.dr_rate:
        out = self.dropout(pooler)
    return self.classifier(out)
</code></pre>
<p>model = BERTClassifier(bertmodel,  dr_rate=0.5).to(device)</p>
<h1>Prepare optimizer and schedule (linear warmup and decay)</h1>
<p>no_decay = [‘bias’, ‘LayerNorm.weight’]<br>
optimizer_grouped_parameters = [<br>
{‘params’: [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)], ‘weight_decay’: 0.01},<br>
{‘params’: [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)], ‘weight_decay’: 0.0}<br>
]</p>
<p>optimizer = AdamW(optimizer_grouped_parameters, lr=learning_rate)<br>
loss_fn = nn.CrossEntropyLoss()</p>
<p>t_total = len(train_dataloader) * num_epochs<br>
warmup_step = int(t_total * warmup_ratio)</p>
<p>scheduler = WarmupLinearSchedule(optimizer, warmup_steps=warmup_step, t_total=t_total)</p>
<p>def calc_accuracy(X,Y):<br>
max_vals, max_indices = torch.max(X, 1)<br>
train_acc = (max_indices == Y).sum().data.cpu().numpy()/max_indices.size()[0]<br>
return train_acc</p>
<p>%%time<br>
for e in range(num_epochs):<br>
train_acc = 0.0<br>
test_acc = 0.0<br>
model.train()<br>
for batch_id, (token_ids, valid_length, segment_ids, label) in enumerate(tqdm_notebook(train_dataloader)):<br>
optimizer.zero_grad()<br>
token_ids = token_ids.long().to(device)<br>
segment_ids = segment_ids.long().to(device)<br>
valid_length= valid_length<br>
label = label.long().to(device)<br>
out = model(token_ids, valid_length, segment_ids)<br>
loss = loss_fn(out, label)<br>
loss.backward()<br>
torch.nn.utils.clip_grad_norm_(model.parameters(), max_grad_norm)<br>
optimizer.step()<br>
scheduler.step()  # Update learning rate schedule<br>
train_acc += calc_accuracy(out, label)<br>
if batch_id % log_interval == 0:<br>
print(“epoch {} batch id {} loss {} train acc {}”.format(e+1, batch_id+1, loss.data.cpu().numpy(), train_acc / (batch_id+1)))<br>
print(“epoch {} train acc {}”.format(e+1, train_acc / (batch_id+1)))<br>
model.eval()<br>
for batch_id, (token_ids, valid_length, segment_ids, label) in enumerate(tqdm_notebook(test_dataloader)):<br>
token_ids = token_ids.long().to(device)<br>
segment_ids = segment_ids.long().to(device)<br>
valid_length= valid_length<br>
label = label.long().to(device)<br>
out = model(token_ids, valid_length, segment_ids)<br>
test_acc += calc_accuracy(out, label)<br>
<span class="hashtag">#torch</span>.cuda.reset_max_memory_cached()<br>
print(“epoch {} test acc {}”.format(e+1, test_acc / (batch_id+1)))</p>
<hr>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in </p>
<p>~/venv/lib/python3.7/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)<br>
164                 products. Defaults to <code>False</code>.<br>
165         “”"<br>
–&gt; 166         torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
167<br>
168     def register_hook(self, hook):</p>
<p>~/venv/lib/python3.7/site-packages/torch/autograd/<strong>init</strong>.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)<br>
97     Variable._execution_engine.run_backward(<br>
98         tensors, grad_tensors, retain_graph, create_graph,<br>
—&gt; 99         allow_unreachable=True)  # allow_unreachable flag<br>
100<br>
101</p>
<p>RuntimeError: transform: failed to synchronize: cudaErrorAssert: device-side assert triggered</p>

username: ptrblck
cooked: <p>Is the model running fine on the CPU?<br>
If so, could you rerun it via <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> and post the stack trace here please?</p>

username: X_Zhao
cooked: <p>Have you solve the problem now? I meet similar problem with you.</p>

username: XuxinLi
cooked: <p>Similar Problem, always around 7 epoch then throw a error</p>
<p><strong>RuntimeError: reduce failed to synchronize: cudaErrorAssert: device-side assert triggered</strong></p>
<p>the loss function is BCE , the optimize is Adam.<br>
Have you solve the problem?</p>