Issue 1:
Documentation issue

In the documentation of PyTorch 1.0.1. It is mentioned that:

batch_first – If True, then the input and output tensors are provided as (batch, seq, feature). Default: False

So I would expect the ouput tensor h_n of shape: (batch, num_layers * num_directions, hidden_size). But it is not the case. The batch is the second dimension, which does not match with the above statement from the documentation.

To Reproduce

Steps to reproduce the behavior:

input = torch.rand([60, 8, 256]) #batch size is 60, sequence size is 8, features are 256
lstm = torch.nn.LSTM(256, 512, batch_first=True)
_, (finalHiddenState, _) = lstm(input)
print(finalHiddenState.shape)
Ouput: torch.Size([1, 60, 512])

Batch (60) is now the second dimension here

Issue 2:
🐛 Bug

When using MPI to launch processes, and mpi4py to assign ranks and get world size, DDP fails to initialize.

To Reproduce

Steps to reproduce the behavior:

Here is a complete reproducer script:

import torch
import random

# This calls mpi.Init()
from mpi4py import MPI

# What rank? World size?
rank = MPI.COMM_WORLD.Get_rank()
world_size = MPI.COMM_WORLD.Get_size()

# Pytorch data parallel:
import torch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP
import torch.multiprocessing as mp

# Create a 1-layer network, put it on GPU:
n = torch.nn.Linear(1, 128).cuda()

# May need to change init method to not be 'cadams' home dir ...
init_method = f'file:///home/cadams/.torch_ddp_init'
dist.init_process_group("nccl", init_method=init_method,rank=rank, world_size=world_size)

n = DDP(n)

print("SUCCESS")

I launch this with mpirun -n 2 python reproducer.py

For each rank, I see this:

Traceback (most recent call last):
File "pytorch_mpi4py_ddp_crash.py", line 25, in <module>
n = DDP(n)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 389, in __init__
self.broadcast_bucket_size)
File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 606, in _distributed_broadcast_coalesced
dist._broadcast_coalesced(self.process_group, tensors, buffer_size)
RuntimeError: NCCL error in: ../torch/lib/c10d/ProcessGroupNCCL.cpp:558, invalid usage, NCCL version 2.7.8

Is it possible that the mpi4py call to MPI_Init is being duplicated in pytorch? I know this was a bug in Summer 2020 but not sure if it persists.

Expected behavior

I expect DDP to successfully init and wrap the networks, one per rank.

Environment

Collecting environment information...

PyTorch version: 1.7.0a0+8deb4fe

Is debug build: False

CUDA used to build PyTorch: 11.0

ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)

GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0

Clang version: Could not collect

CMake version: version 3.14.0

Python version: 3.6 (64-bit runtime)

Is CUDA available: True

CUDA runtime version: Could not collect

GPU models and configuration:

GPU 0: A100-SXM4-40GB

GPU 1: A100-SXM4-40GB

GPU 2: A100-SXM4-40GB

GPU 3: A100-SXM4-40GB

GPU 4: A100-SXM4-40GB

GPU 5: A100-SXM4-40GB

GPU 6: A100-SXM4-40GB

GPU 7: A100-SXM4-40GB

Nvidia driver version: 450.51.05

cuDNN version: Probably one of the following:

/usr/lib/x86_64-linux-gnu/libcudnn.so.8.0.4

/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.0.4

/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.0.4

/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.0.4

/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.0.4

/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.0.4

/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.0.4

HIP runtime version: N/A

MIOpen runtime version: N/A

Versions of relevant libraries:

[pip3] numpy==1.19.1

[pip3] pytorch-transformers==1.1.0

[pip3] torch==1.7.0a0+8deb4fe

[pip3] torch-cluster==1.5.7

[pip3] torch-geometric==1.6.1

[pip3] torch-scatter==2.0.5

[pip3] torch-sparse==0.6.7

[pip3] torchtext==0.6.0

[pip3] torchvision==0.8.0a0

[conda] magma-cuda110 2.5.2 5 local

[conda] mkl 2019.1 144

[conda] mkl-include 2019.1 144

[conda] nomkl 3.0 0

[conda] numpy 1.19.1 py36h30dfecb_0

[conda] numpy-base 1.19.1 py36h75fe3a5_0

[conda] pytorch-transformers 1.1.0 pypi_0 pypi

[conda] torch 1.7.0a0+8deb4fe pypi_0 pypi

[conda] torchtext 0.6.0 pypi_0 pypi

[conda] torchvision 0.8.0a0 pypi_0 pypi

Additional context

I am running this on 8xA100 DGX systems, using nvidia's container: docker pull nvcr.io/nvidia/pytorch:20.09-py3