Issue 1:
🐛 Bug

For some reason, I want to divide all training processes into several subset, and do communication in each of them. As the division may change during Traing, after some research, I want to use dist.new_group(). Howerver, I keep getting error like ConnectionRefused or just stuck.

To Reproduce

import torch
import torch.distributed as dist
import torch.multiprocessing as mp

import os
import time
import argparse
from datetime import timedelta

def main():
os.environ["MASTER_ADDR"] = '127.0.0.1'
os.environ["MASTER_PORT"] = '25500'

mp.spawn(main_worker, nprocs=4)

def main_worker(rank):
print(f'Started process {rank}')
torch.cuda.set_device(rank)
dist.init_process_group('gloo', world_size=4, rank=rank)
for _ in range(10):
test_group(rank)

def test_group(rank):
if rank == 0:
perm = torch.randperm(4)
else:
perm = torch.zeros(4, dtype=torch.int64)
dist.broadcast(perm, 0)
print(f'rank: {rank} get broadcast {perm}')
start = 0
while rank not in perm[start: start + 2]:
start += 2
group = perm.tolist()[start: start + 2]
pg = dist.new_group(group, timeout=timedelta(seconds=30))
print(f'Rank: {rank} Group: {group}')
tensor = torch.randn(4, 5).cuda()
dist.all_reduce(tensor, group=pg)
print(f'Rank:{rank}\tGroup:{group}\tTensor: {tensor}\n')
dist.destroy_process_group(pg)
print(f'Rank: {rank} process group destroyed')

if __name__ == '__main__':
main()

Steps to reproduce the behavior:

run the script above witch Gloo

At most time, I will get a connection refused error

Started process 2
Started process 0
Started process 1
Started process 3
rank: 2 get broadcast tensor([3, 1, 0, 2])
rank: 1 get broadcast tensor([3, 1, 0, 2])
rank: 0 get broadcast tensor([3, 1, 0, 2])
rank: 3 get broadcast tensor([3, 1, 0, 2])
Rank: 2 Group: [0, 2]
Rank: 0 Group: [0, 2]
Traceback (most recent call last):
File "group_test.py", line 44, in <module>
main()
File "group_test.py", line 14, in main
mp.spawn(main_worker, nprocs=4)
File "/data/users/yangx/miniconda3/envs/multiRing/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
while not spawn_context.join():
File "/data/users/yangx/miniconda3/envs/multiRing/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 118, in join
raise Exception(msg)
Exception:

-- Process 3 terminated with the following error:
Traceback (most recent call last):
File "/data/users/<user>/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
fn(i, *args)
File "/data/users/<user>/group_test.py", line 21, in main_worker
test_group(rank)
File "/data/users/<user>/group_test.py", line 28, in test_group
pg = dist.new_group(group, timeout=timedelta(seconds=30))
File "/data/users/<user>/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1434, in new_group
timeout=timeout)
File "/data/users/<user>/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 484, in _new_process_group_helper
timeout=timeout)
RuntimeError: [/opt/conda/conda-bld/pytorch_1565272271120/work/third_party/gloo/gloo/transport/tcp/pair.cc:761] connect [127.0.1.1]:23170: Connection refused

But sometimes, I can get some result at first(but NOT correct!), then the code just stuck, and I have to Ctrl-C

The really strange things is that Rank 1 and 3 should in different groups, but they act like in the same(the all-reduce result is the same).

Started process 0
Started process 1
Started process 3
Started process 2
rank: 2 get broadcast tensor([0, 3, 2, 1])
rank: 1 get broadcast tensor([0, 3, 2, 1])
rank: 3 get broadcast tensor([0, 3, 2, 1])
rank: 0 get broadcast tensor([0, 3, 2, 1])
Rank: 3 Group: [0, 3]
Rank: 1 Group: [2, 1]
Rank:1	Group:[2, 1]	Tensor: tensor([[-0.3007, -1.8406, 0.1597, -0.9097, 1.7759],
[-0.7950, 1.5906, 0.4027, 0.8949, 0.2737],
[ 1.4671, 1.3343, 0.2170, 1.2460, 0.6926],
[-0.1553, -1.6191, 0.9762, 1.9962, -1.2016]], device='cuda:1')

Rank: 1 process group destroyed
Rank:3	Group:[0, 3]	Tensor: tensor([[-0.3007, -1.8406, 0.1597, -0.9097, 1.7759],
[-0.7950, 1.5906, 0.4027, 0.8949, 0.2737],
[ 1.4671, 1.3343, 0.2170, 1.2460, 0.6926],
[-0.1553, -1.6191, 0.9762, 1.9962, -1.2016]], device='cuda:3')

Rank: 3 process group destroyed

Expected behavior

The group can be established and destroyed during running, and the tensor can be correct sent in each subset.

Environment

Please copy and paste the output from our

environment collection script

(or fill out the checklist below manually).

Collecting environment information...

PyTorch version: 1.2.0

Is debug build: No

CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 16.04.6 LTS

GCC version: (Ubuntu 4.9.4-2ubuntu1~14.04.1) 4.9.4

CMake version: version 3.5.1

Python version: 3.7

Is CUDA available: Yes

CUDA runtime version: 10.1.168

GPU models and configuration:

GPU 0: Tesla K80

GPU 1: Tesla K80

GPU 2: Tesla K80

GPU 3: Tesla K80

GPU 4: Tesla K80

GPU 5: Tesla K80

GPU 6: Tesla K80

GPU 7: Tesla K80

Nvidia driver version: 418.67

cuDNN version: Probably one of the following:

/usr/local/cuda-9.0/targets/x86_64-linux/lib/libcudnn.so.7.4.1

/usr/local/lib/libcudnn.so.6

Versions of relevant libraries:

[pip3] numpy==1.16.3

[pip3] torch==1.1.0

[pip3] torchvision==0.3.0

[conda] _tflow_select 2.3.0 mkl https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main

[conda] blas 1.0 mkl https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main

[conda] mkl 2018.0.3 1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main

[conda] mkl_fft 1.0.6 py37h7dd41cf_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main

[conda] mkl_random 1.0.1 py37h4414c95_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main

[conda] pytorch 1.2.0 py3.7_cuda10.0.130_cudnn7.6.2_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch

[conda] pytorch-transformers 1.0.0 pypi_0 pypi

[conda] tensorflow 1.13.1 mkl_py37h54b294f_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main

[conda] tensorflow-base 1.13.1 mkl_py37h7ce6ba3_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main

[conda] torchtext 0.4.0 pyhb384e40_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch

[conda] torchvision 0.4.0 py37_cu100 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch

You can get the script and run it with:

wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py

Additional context

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera

Issue 2:
🐛 Bug

Seems that CUDA10.0 could build pytorch objects, but during linkage phase. it failed. could you help to check reason, or give me some hints for solving with it? now my mac with nvidia GPU 1080i can't use torch with GPU.

To Reproduce

Steps to reproduce the behavior:

Set USE_DISTRIBUTED=OFF in CMakeLists.txt

MACOSX_DEPLOYMENT_TARGET=10.13 CC=clang CXX=clang++ python setup.py bdist_wheel

Expected behavior

torch wheel should be generated with GPU supported (0.4 torch wheel could be generated in the same way), but failed with following error

error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/cudnn_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/cudnn_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/blob_gpu_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/blob_gpu_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/context_gpu_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/context_gpu_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/event_gpu_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/event_gpu_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/net_gpu_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/net_gpu_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/operator_gpu_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/operator_gpu_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/math_gpu_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/math_gpu_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/conv_op_cache_cudnn_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/conv_op_cache_cudnn_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/batch_matmul_op_gpu_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/batch_matmul_op_gpu_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/elementwise_op_gpu_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/elementwise_op_gpu_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/operator_fallback_gpu_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/operator_fallback_gpu_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/reshape_op_gpu_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/reshape_op_gpu_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/roi_align_op_gpu_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/roi_align_op_gpu_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/utility_ops_gpu_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/test/utility_ops_gpu_test (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/lib/python3.6/site-packages/caffe2/python/caffe2_pybind11_state_gpu.cpython-36m-darwin.so (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/lib/libtorch.1.dylib (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/lib/libtorch.1.dylib (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/lib/libcaffe2_detectron_ops_gpu.dylib (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Volumes/llv24macOS/git_projects/ml/pytorch/torch/lib/tmp_install/lib/libcaffe2_detectron_ops_gpu.dylib (for architecture x86_64) option "-add_rpath /usr/local/cuda/lib" would duplicate path, file already has LC_RPATH for: /usr/local/cuda/lib
+ [[ ON == \O\N ]]
++ pwd
++ cd ..
++ pwd
+ echo 'Copying Caffe2 proto files from /Volumes/llv24macOS/git_projects/ml/pytorch/build/caffe2/proto to /Volumes/llv24macOS/git_projects/ml/pytorch/caffe2/proto'
Copying Caffe2 proto files from /Volumes/llv24macOS/git_projects/ml/pytorch/build/caffe2/proto to /Volumes/llv24macOS/git_projects/ml/pytorch/caffe2/proto
++ find caffe2/proto
+ echo 'All the files in caffe2/proto are caffe2/proto
caffe2/proto/CMakeFiles
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/flags.make
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/progress.make
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/hsm.pb.cc.o
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/cmake_clean.cmake
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/torch.pb.cc.o
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/build.make
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/DependInfo.cmake
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/depend.make
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/CXX.includecache
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/depend.internal
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/caffe2.pb.cc.o
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/prof_dag.pb.cc.o
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/metanet.pb.cc.o
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/predictor_consts.pb.cc.o
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/caffe2_legacy.pb.cc.o
caffe2/proto/CMakeFiles/CMakeDirectoryInformation.cmake
caffe2/proto/CMakeFiles/progress.marks
caffe2/proto/__init__.py
caffe2/proto/prof_dag.pb.cc
caffe2/proto/Makefile
caffe2/proto/hsm.pb.cc
caffe2/proto/cmake_install.cmake
caffe2/proto/CTestTestfile.cmake
caffe2/proto/prof_dag.pb.h
caffe2/proto/hsm.pb.h
caffe2/proto/caffe2_legacy.pb.cc
caffe2/proto/predictor_consts.pb.cc
caffe2/proto/caffe2_legacy.pb.h
caffe2/proto/predictor_consts.pb.h
caffe2/proto/torch.pb.cc
caffe2/proto/torch.pb.h
caffe2/proto/metanet.pb.cc
caffe2/proto/metanet.pb.h
caffe2/proto/caffe2.pb.cc
caffe2/proto/caffe2.pb.h
caffe2/proto/prof_dag_pb2.py
caffe2/proto/hsm_pb2.py
caffe2/proto/metanet_pb2.py
caffe2/proto/caffe2_legacy_pb2.py
caffe2/proto/predictor_consts_pb2.py
caffe2/proto/torch_pb2.py
caffe2/proto/caffe2_pb2.py'
All the files in caffe2/proto are caffe2/proto
caffe2/proto/CMakeFiles
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/flags.make
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/progress.make
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/hsm.pb.cc.o
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/cmake_clean.cmake
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/torch.pb.cc.o
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/build.make
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/DependInfo.cmake
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/depend.make
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/CXX.includecache
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/depend.internal
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/caffe2.pb.cc.o
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/prof_dag.pb.cc.o
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/metanet.pb.cc.o
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/predictor_consts.pb.cc.o
caffe2/proto/CMakeFiles/Caffe2_PROTO.dir/caffe2_legacy.pb.cc.o
caffe2/proto/CMakeFiles/CMakeDirectoryInformation.cmake
caffe2/proto/CMakeFiles/progress.marks
caffe2/proto/__init__.py
caffe2/proto/prof_dag.pb.cc
caffe2/proto/Makefile
caffe2/proto/hsm.pb.cc
caffe2/proto/cmake_install.cmake
caffe2/proto/CTestTestfile.cmake
caffe2/proto/prof_dag.pb.h
caffe2/proto/hsm.pb.h
caffe2/proto/caffe2_legacy.pb.cc
caffe2/proto/predictor_consts.pb.cc
caffe2/proto/caffe2_legacy.pb.h
caffe2/proto/predictor_consts.pb.h
caffe2/proto/torch.pb.cc
caffe2/proto/torch.pb.h
caffe2/proto/metanet.pb.cc
caffe2/proto/metanet.pb.h
caffe2/proto/caffe2.pb.cc
caffe2/proto/caffe2.pb.h
caffe2/proto/prof_dag_pb2.py
caffe2/proto/hsm_pb2.py
caffe2/proto/metanet_pb2.py
caffe2/proto/caffe2_legacy_pb2.py
caffe2/proto/predictor_consts_pb2.py
caffe2/proto/torch_pb2.py
caffe2/proto/caffe2_pb2.py
++ pwd
+ for proto_file in '$(pwd)/caffe2/proto/*.py'
++ pwd
+ cp /Volumes/llv24macOS/git_projects/ml/pytorch/build/caffe2/proto/__init__.py /Volumes/llv24macOS/git_projects/ml/pytorch/build/../caffe2/proto/
cp: /Volumes/llv24macOS/git_projects/ml/pytorch/build/../caffe2/proto/__init__.py and /Volumes/llv24macOS/git_projects/ml/pytorch/build/caffe2/proto/__init__.py are identical (not copied).
Failed to run 'bash ../tools/build_pytorch_libs.sh --use-cuda --use-nnpack caffe2 libshm THD'

Environment

$ python ~/Downloads/collect_env.py

Collecting environment information...

PyTorch version: 1.0.0.dev20181016

Is debug build: No

CUDA used to build PyTorch: Could not collect

OS: Mac OSX 10.13.6

GCC version: Could not collect

CMake version: version 3.10.1

Python version: 3.6

Is CUDA available: No

CUDA runtime version: 10.0.130

GPU models and configuration: Could not collect

Nvidia driver version: Could not collect

cuDNN version: Probably one of the following:

/usr/local/cuda/lib/libcudnn.7.dylib

/usr/local/cuda/lib/libcudnn.dylib

/usr/local/cuda/lib/libcudnn_static.a

Versions of relevant libraries:

[pip] Could not collect

[conda] pytorch-nightly 1.0.0.dev20181016 py3.6_0 pytorch

cc @malfet @seemethere @walterddr