Issue 1:
🐛 Bug

I am using both PyTorch 1.5 (stable) and nightly build (1.6.0.dev20200611).

Sometimes I need to run code in v1.5 and use files saved by torch.save from the nightly build.

However, PyTorch 1.5 fails to torch.load such files.

To Reproduce

Steps to reproduce the behavior:

PyTorch nightly

import torch
a = torch.randn(1)
torch.save(a, 'a.pt')

PyTorch 1.5

import torch
torch.load('a.pt')

Then it shows the following error message

Traceback (most recent call last):

File "", line 1, in

File "/home/seungjun/local/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/serialization.py", line 586, in load

with _open_zipfile_reader(f) as opened_zipfile:

File "/home/seungjun/local/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/serialization.py", line 246, in init

super(_open_zipfile_reader, self).init(torch.C.PyTorchFileReader(name_or_buffer))

RuntimeError: version <= kMaxSupportedFileFormatVersion INTERNAL ASSERT FAILED at /opt/conda/conda-bld/pytorch_1587428266983/work/caffe2/serialize/inline_container.cc:132, please report a bug to PyTorch. Attempted to read a PyTorch file with version 4, but the maximum supported version for reading is 3. Your PyTorch installation may be too old.

Expected behavior

I expect PyTorch 1.5 to load files without error.

Current behavior could be due to the update in PyTorchFileReader but I assume 1.5 and 1.6 should be compatible as their version number is close to each other.

Environment

Loading:

PyTorch Version (e.g., 1.0): 1.5.0

OS (e.g., Linux): Ubuntu 18.04.4 LTS

How you installed PyTorch (conda, pip, source): conda

Build command you used (if compiling from source):

conda install pytorch torchvision cudatoolkit=10.2 -c pytorch

Python version: 3.7

CUDA/cuDNN version: 10.2.89, 7.6.5

Saving:

PyTorch Version (e.g., 1.0): 1.6.0.dev20200611 for saving

OS (e.g., Linux): Ubuntu 18.04.4 LTS

How you installed PyTorch (conda, pip, source): conda

Build command you used (if compiling from source):

conda install pytorch torchvision cudatoolkit=10.2 -c pytorch-nightly

Python version: 3.8

CUDA/cuDNN version: 10.2.89, 7.6.5

cc @ezyang @gchanan @zou3519

Issue 2:
🐛 Bug

This issus is same as #40425, but 40425 can not slove my question, so i reopen it again ,and add some NOTE.

I am using cusolverDn in c++ and import it to python with pybind11.

After use cusolverDnCreate() and cusolverDnDestroy() with torch,Segment Fault appear when program destruction.

NOTE:

1: When the demo run with Torch1.2,it run success, the SegFault disappear, does it mean Torch 1.5 has some bug?

2: This SegFault only appear with "thread_local", Does it provide some information for this bug?

3: Please see test.py below, if we "import torch" after "import example", SegFault disappear.

4: Torch have not link libcusolver.so, so we just link it once.

@ezyang say that upgarde cuda may fix this.But i can not upgrade in production environment。

cc please @ezyang @gchanan @zou3519 @yf225 @glaringlee @ngimel @mcarilli

To Reproduce

Steps to reproduce the behavior:

1.paste the c++ code, name bind_cudevice.cc

2.compile bind_cudevice.cc with my g++ compile code

3.paste my python code named test.py

4. just run test.py and see

bind_cudevice.cc

#include <iostream>

#include <cublas_v2.h>
#include <cuda.h>
#include <cuda_runtime_api.h>

#include <cusolverDn.h>
#include <cusolverSp.h>
#include <cusolverRf.h>

#include <pybind11/pybind11.h>

class CuDevice {

public:
CuDevice() {
initialized_ = true;

cudaSetDevice(0);

cusolverDnCreate(&cusolverdn_handle_);
cusolverDnSetStream(cusolverdn_handle_, cudaStreamPerThread);
}

~CuDevice() {
std::cout << "in ~CuDevice" << std::endl;
if (cusolverdn_handle_) {
std::cout << "in ~CuDevice and cusolverdn_handle_" << std::endl;
cusolverDnDestroy(cusolverdn_handle_);
std::cout << "Destory success" << std::endl;
}
}

private:
bool initialized_;
cusolverDnHandle_t cusolverdn_handle_;

};

int init_cudevice(){
thread_local CuDevice a = CuDevice();
std::cout << "hello world" << std::endl;
return 0;
}

PYBIND11_MODULE(example, m) {
m.def("init_cudevide",
&init_cudevice,
"select gpu id");
}

g++ compile code

g++ -O3 -Wall -shared -std=c++11 -fPIC -I/usr/include/python3.6m -I./third-party/pybind11-2.2.4/include -I/usr/local/cuda/include -Wl,-rpath=/usr/local/cuda/lib64 -L/usr/local/cuda/lib64 -lcudart -lcusolver `python3 -m pybind11 --includes` binding_cudevice.cc -o example`python3-config --extension-suffix`

test.py

import torch
import example
example.init_cudevide()

Core Dump Info

(gdb) bt
#0 0x00007f78b6a391de in ?? () from /usr/local/cuda/lib64/libcusolver.so.10.0
#1 0x00007f78b6a5b4ca in ?? () from /usr/local/cuda/lib64/libcusolver.so.10.0
#2 0x00007f78b5db6e49 in ?? () from /usr/local/cuda/lib64/libcusolver.so.10.0
#3 0x00007f78b5b854c1 in cusolverDnDestroy () from /usr/local/cuda/lib64/libcusolver.so.10.0
#4 0x00007f78be143952 in kaldi::CuDevice::~CuDevice (this=0x2f18a38, __in_chrg=<optimized out>) at cu-device.cc:683
#5 0x00007f79880d7ad1 in (anonymous namespace)::run (p=<optimized out>) at /home/nwani/m3/conda-bld/compilers_linux-64_1560109574129/work/.build/x86_64-conda_cos6-linux-gnu/src/gcc/libstdc++-v3/libsupc++/atexit_thread.cc:75
#6 0x00007f79b752cc99 in __run_exit_handlers () from /usr/lib64/libc.so.6
#7 0x00007f79b752cce7 in exit () from /usr/lib64/libc.so.6
#8 0x00007f79b751550c in __libc_start_main () from /usr/lib64/libc.so.6
#9 0x0000000000400c20 in _start ()

Expected behavior

No error happen

Environment

PyTorch Version (e.g., 1.5):1.5

OS (e.g., Linux):Centos7

How you installed PyTorch (conda, pip, source):source

Build command you used (if compiling from source):

Python version:3.6.8

CUDA/cuDNN version:10.0

GPU models and configuration:K40

Any other relevant information:

Additional context

cuSolver is base cublas and cusparse, reference about cuSolver：

https://docs.nvidia.com/cuda/archive/10.0/cusolver/index.html#cuSolverDNhandle