Issue 1:
My aim is to make a five-category text classification

I am running transformers fine tuning bert with cnnbase model but my program stops at loss.backward() without any prompt in cmd.

I debug find that the program stop at the loss.backward line without any error prompt

My program executed successfully in rnn base such as lstm and rcnn.

But when I am running some cnnbase model the strange bug appears.

My cnn model code:

import torch
import torch.nn as nn
import torch.nn.functional as F
from transformers.modeling_bert import BertPreTrainedModel, BertModel
n_filters = 200
filter_sizes = [2,3,4]
class BertCNN(BertPreTrainedModel):
def __init__(self, config):
super(BertPreTrainedModel, self).__init__(config)
self.num_filters = n_filters
self.filter_sizes = filter_sizes
self.bert = BertModel(config)
for param in self.bert.parameters():
param.requires_grad = True
self.convs = nn.ModuleList(
[nn.Conv2d(1, self.num_filters, (k, config.hidden_size))
for k in self.filter_sizes])
self.dropout = nn.Dropout(config.hidden_dropout_prob)
self.fc_cnn = nn.Linear(self.num_filters *
len(self.filter_sizes), config.num_labels)

def conv_and_pool(self, x, conv):
x = F.relu(conv(x)).squeeze(3)
x = F.max_pool1d(x, x.size(2)).squeeze(2)
return x

def forward(self, input_ids,
attention_mask=None, token_type_ids=None, head_mask=None):
outputs = self.bert(input_ids,
attention_mask=attention_mask,
token_type_ids=token_type_ids,
head_mask=head_mask)
encoder_out, text_cls = outputs
out = encoder_out.unsqueeze(1)
out = torch.cat([self.conv_and_pool(out, conv)
for conv in self.convs], 1)
out = self.dropout(out)
out = self.fc_cnn(out)
return out

My train code:

for step, batch in enumerate(data):
self.model.train()
batch = tuple(t.to(self.device) for t in batch)
input_ids, input_mask, segment_ids, label_ids = batch
print("input_ids, input_mask, segment_ids, label_ids SIZE: \n")
print(input_ids.size(), input_mask.size(),segment_ids.size(), label_ids.size())
# torch.Size([2, 80]) torch.Size([2, 80]) torch.Size([2, 80]) torch.Size([2])
logits = self.model(input_ids, segment_ids, input_mask)
print("logits and label ids size: ",logits.size(), label_ids.size())
# torch.Size([2, 5]) torch.Size([2])
loss = self.criterion(output=logits, target=label_ids) #loss function:CrossEntropyLoss()
if len(self.n_gpu) >= 2:
loss = loss.mean()
if self.gradient_accumulation_steps > 1:
loss = loss / self.gradient_accumulation_steps
if self.fp16:
with amp.scale_loss(loss, self.optimizer) as scaled_loss:
scaled_loss.backward()
clip_grad_norm_(amp.master_params(self.optimizer), self.grad_clip)
else:
loss.backward() # I debug find that the program stop at this line without any error prompt

HELP~！~ 、

I posted my questions on various community platforms，stackoverflow、other github repositories.

No one replied to me.

Issue 2:
Whenever backward() is called to compute gradients the python script never terminates. The backward function is not per se blocking and all lines after it are still executed however the script just does not terminate. This may be related with the operation system which is Windows 7 or with the use of conda. At least there is another (post on Stack Overflow) describing the same problem in the same environment.

To Reproduce

Steps to reproduce the behavior:

install PyTorch on windows 7 via conda (conda create --name grad_test pytorch -c pytorch)

run example code given below taken from the post on Stack Overflow

import torch

x = torch.randn(3, requires_grad=True)
y = x * 2
print(y)

gradients = torch.tensor([0.1, 1.0, 0.0001], dtype=torch.float)
y.backward(gradients)
print(x.grad)

print("all done")

Expected behavior

Script runs and exits after printing "all done".

Environment

PyTorch version: 1.3.0

Is debug build: No

CUDA used to build PyTorch: 10.1

OS: Microsoft Windows 7 Enterprise

CMake version: version 3.9.6

Python version: 3.7

Is CUDA available: Yes

CUDA runtime version: 10.1.243

GPU models and configuration: GPU 0: GeForce GTX 1080

Nvidia driver version: 441.12

cuDNN version: Could not collect

Versions of relevant libraries:

[pip] numpy==1.17.3

[pip] torch==1.3.0

[pip] torchvision==0.4.1

[conda] blas 1.0 mkl anaconda

[conda] libblas 3.8.0 14_mkl conda-forge

[conda] libcblas 3.8.0 14_mkl conda-forge

[conda] liblapack 3.8.0 14_mkl conda-forge

[conda] mkl 2019.4 245

[conda] mkl-service 2.3.0 py37hfa6e2cd_0 conda-forge

[conda] pytorch 1.3.0 py3.7_cuda101_cudnn7_0 pytorch

[conda] torchvision 0.4.1 py37_cu101 pytorch

cc @ezyang @ssnl @albanD @zou3519 @gqchen @peterjc123