Issue 1:
Description

I've tried finding resources regarding to this but they seem outdated.

The problem is that I am trying to build in visual studio with the binaries I downloaded from

https://download.pytorch.org/libtorch/cu102/libtorch-win-shared-with-deps-debug-1.5.0.zip

I have set the following properties in Visual Studio 2019

Additional include directories : libtorch\include ; libtorch\include\torch\csrc\api\include\

Additional Library Directory for linker : \libtorch\lib\

Conformance mode to No

Linker input : torch.lib;c10.lib

However the resources are also suggesting libs like caffe2_nvrtc.lib and caffe2.lib but they are not present in \libtorch\lib\

I tried to run the following program

#include<torch/torch.h>
#include<iostream>
int main()
{
torch::Tensor tensor = torch::eye(3);
std::cout << tensor << std::endl;
}

I got series of errors and warnings :

Operating System : Windows 10

Issue 2:
🐛 Bug

Running TestDistBackend.test_DistributedDataParallel from test_distributed suite

with a WORLD_SIZE <= nGPUs/2 on the machine

results in an error "Model replicas must have an equal number of parameters".

To Reproduce

Steps to reproduce the behavior:

Use a machine which has at least 4 GPUs

Build pytorch from source for python3.6 OR use one of the available docker images.

Run the following command: BACKEND=nccl WORLD_SIZE=2 TEMP_DIR=/tmp python3.6 test_distributed.py --verbose TestDistBackend.test_DistributedDataParallel

On a ROCm build of PyTorch, I get the below stack trace (although this issue isn't ROCm-specific):

Process process 0:
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/opt/conda/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "distributed/test_distributed.py", line 2126, in _run
getattr(self, self.id().split(".")[2])()
File "distributed/test_distributed.py", line 2052, in wrapper
fn(self)
File "distributed/test_distributed.py", line 119, in wrapper
return func(*args, **kwargs)
File "distributed/test_distributed.py", line 135, in wrapper
return func(*args, **kwargs)
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 102, in wrapper
return func(*args, **kwargs)
File "distributed/test_distributed.py", line 1878, in test_DistributedDataParallel
self._test_DistributedDataParallel(gpu_subset=gpus, rank=rank)
File "distributed/test_distributed.py", line 1812, in _test_DistributedDataParallel
model_DDP, device_ids=gpu_subset
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 305, in __init__
self._ddp_init_helper()
File "/var/lib/jenkins/.local/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 387, in _ddp_init_helper
expect_sparse_gradient)
RuntimeError: Model replicas must have an equal number of parameters.

If we limit the number of devices as follows:

HIP_VISIBLE_DEVICES=0,1 BACKEND=nccl WORLD_SIZE=2 TEMP_DIR=/tmp python3.6 test_distributed.py --verbose TestDistBackend.test_DistributedDataParallel

the test passes.

Expected behavior

Test should pass.

Environment

Collecting environment information...
PyTorch version: 1.5.0a0+f5e5569
Is debug build: No
CUDA used to build PyTorch: Could not collect

OS: Ubuntu 16.04.6 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
CMake version: version 3.6.3

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect
ROCm version: 3.1

Versions of relevant libraries:
[pip] numpy==1.18.1
[pip] torch==1.5.0a0+f5e5569
[pip] torchvision==0.5.0a0+44a5bae
[conda] blas 1.0 mkl
[conda] mkl 2020.0 166
[conda] mkl-include 2020.0 166
[conda] mkl-service 2.3.0 py36he904b0f_0
[conda] mkl_fft 1.0.15 py36ha843d7b_0
[conda] mkl_random 1.1.0 py36hd6b4f25_0

cc @ezyang @gchanan @zou3519 @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @xush6528 @osalpekar