Issue 1:
🐛 Bug

Training with nn.DataParallel in v1.1(v1.2 or v1.3) is much slower than v1.0.1.

Such as run_squad.py in transformers library as its README

https://github.com/huggingface/transformers/blob/master/examples/run_squad.py

To Reproduce

export SQUAD_DIR=/path/to/SQUAD

python run_squad.py

--model_type bert

--model_name_or_path bert-base-cased

--do_train

--do_eval

--do_lower_case

--train_file $SQUAD_DIR/train-v1.1.json

--predict_file $SQUAD_DIR/dev-v1.1.json

--per_gpu_train_batch_size 5

--learning_rate 3e-5

--num_train_epochs 2.0

--max_seq_length 384

--doc_stride 128

--output_dir /tmp/debug_squad/

The smaller batch size the speed is more distinct.

Expected behavior

They should have almost the same speed.

Environment

For v1.0.1

PyTorch version: 1.0.1

Is debug build: No

CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 18.04.3 LTS

GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0

CMake version: version 3.10.2

Python version: 3.7

Is CUDA available: Yes

CUDA runtime version: Could not collect

GPU models and configuration:

GPU 0: GeForce GTX 1080 Ti

GPU 1: GeForce GTX 1080 Ti

GPU 2: GeForce GTX 1080 Ti

GPU 3: GeForce GTX 1080 Ti

GPU 4: GeForce GTX 1080 Ti

GPU 5: GeForce GTX 1080 Ti

GPU 6: GeForce GTX 1080 Ti

GPU 7: GeForce GTX 1080 Ti

Nvidia driver version: 410.104

cuDNN version: Could not collect

Versions of relevant libraries:

[pip] numpy==1.16.2

[pip] numpydoc==0.9.1

[pip] torch==1.0.1

[conda] blas 1.0 mkl defaults

[conda] mkl 2019.4 243 defaults

[conda] mkl-service 2.0.2 py37h7b6447c_0 defaults

[conda] mkl_fft 1.0.14 py37ha843d7b_0 defaults

[conda] mkl_random 1.0.2 py37hd81dba3_0 defaults

[conda] pytorch 1.0.1 cuda100py37he554f03_0 defaults

For v1.3

PyTorch version: 1.3.0

Is debug build: No

CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 18.04.3 LTS

GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0

CMake version: version 3.10.2

Python version: 3.7

Is CUDA available: Yes

CUDA runtime version: Could not collect

GPU models and configuration:

GPU 0: GeForce GTX 1080 Ti

GPU 1: GeForce GTX 1080 Ti

GPU 2: GeForce GTX 1080 Ti

GPU 3: GeForce GTX 1080 Ti

GPU 4: GeForce GTX 1080 Ti

GPU 5: GeForce GTX 1080 Ti

GPU 6: GeForce GTX 1080 Ti

GPU 7: GeForce GTX 1080 Ti

Nvidia driver version: 410.104

cuDNN version: Could not collect

Versions of relevant libraries:

[pip] numpy==1.16.2

[pip] numpydoc==0.9.1

[pip] torch==1.3.0

[conda] blas 1.0 mkl defaults

[conda] mkl 2019.4 243 defaults

[conda] mkl-service 2.3.0 py37he904b0f_0 defaults

[conda] mkl_fft 1.0.14 py37ha843d7b_0 defaults

[conda] mkl_random 1.1.0 py37hd6b4f25_0 defaults

[conda] pytorch 1.3.0 py3.7_cuda10.0.130_cudnn7.6.3_0 pytorch

Issue 2:
Hi,

I'm working with Pytorch 0.4, I got the FileNotFoundError Exception in functional.py when it invokes the binary_cross_entropy loss. Here is the stack trace:

loss += torch.mean(F.binary_cross_entropy(F.sigmoid(out0), all0) +

File "/home/local/.../anaconda2/envs/py27/lib/python2.7/site-packages/torch/nn/functional.py", line 1483, in binary_cross_entropy

return torch._C._nn.binary_cross_entropy(input, target, weight, size_average, reduce)

RuntimeError: cudaEventSynchronize in future::wait: device-side assert triggered

I'm aware of this type of issues like #6932. I added the fix in the locations suggested by the users in threads, however, the exception still occurs. I'd be happy for any help.

Best,

S.