Repository: pytorch/pytorch

Issue #60930: error for using faster rcnn on c++ & GPU
Created: 2021-06-29T05:49:09Z
I use pytorch to train a faster rcnn model to locate a hand in a image. After converting the model to torchscript by using tracing mothed. and I am very sure that the model and input are all placed on GPU, there is still always an erreor that "Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!". And its log is listed as follow. and can anyone help me to solve this tough issue?
terminate called after throwing an instance of 'std::runtime_error'
what():  The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript, serialized code (most recent call last):
File "code/torch.py", line 8, in 
def forward(self: torch.TraceWrapper,
argument_1: Tensor) -> Tuple[Tensor, Tensor, Tensor]:
_0, _1, _2, = (self.model).forward(argument_1, )
~~~~~~~~~~~~~~~~~~~ <--- HERE
return (_0, _1, _2)
File "code/torch/torchvision/models/detection/faster_rcnn.py", line 18, in forward
s = ops.prim.NumToTensor(torch.size(img, 1))
s0 = ops.prim.NumToTensor(torch.size(img, 2))
_4, _5, _6, = (_3).forward(argument_1, )
~~~~~~~~~~~ <--- HERE
_7, _8, _9, _10, _11, _12, _13, _14, _15, _16, = (_2).forward(_4, )
_17 = (_1).forward(_7, _8, _9, _10, _11, _12, _13, _14, _15, _4, _5, _6, )
File "code/torch/torchvision/models/detection/transform.py", line 12, in forward
_0 = torch.slice(mean, 0, 0, 9223372036854775807, 1)
_1 = torch.unsqueeze(torch.unsqueeze(_0, 1), 2)
_2 = torch.sub(image, _1, alpha=1)
~~~~~~~~~ <--- HERE
_3 = torch.slice(std, 0, 0, 9223372036854775807, 1)
_4 = torch.unsqueeze(torch.unsqueeze(_3, 1), 2)
Traceback of TorchScript, original code (most recent call last):
/home/kevin/anaconda3/envs/script/lib/python3.6/site-packages/torchvision/models/detection/transform.py(124): normalize
/home/kevin/anaconda3/envs/script/lib/python3.6/site-packages/torchvision/models/detection/transform.py(104): forward
/home/kevin/anaconda3/envs/script/lib/python3.6/site-packages/torch/nn/modules/module.py(704): _slow_forward
/home/kevin/anaconda3/envs/script/lib/python3.6/site-packages/torch/nn/modules/module.py(720): _call_impl
/home/kevin/anaconda3/envs/script/lib/python3.6/site-packages/torchvision/models/detection/generalized_rcnn.py(79): forward
/home/kevin/anaconda3/envs/script/lib/python3.6/site-packages/torch/nn/modules/module.py(704): _slow_forward
/home/kevin/anaconda3/envs/script/lib/python3.6/site-packages/torch/nn/modules/module.py(720): _call_impl
faster_rcnn_script.py(34): forward
/home/kevin/anaconda3/envs/script/lib/python3.6/site-packages/torch/nn/modules/module.py(704): _slow_forward
/home/kevin/anaconda3/envs/script/lib/python3.6/site-packages/torch/nn/modules/module.py(720): _call_impl
/home/kevin/anaconda3/envs/script/lib/python3.6/site-packages/torch/jit/init.py(1109): trace_module
/home/kevin/anaconda3/envs/script/lib/python3.6/site-packages/torch/jit/init.py(955): trace
faster_rcnn_script.py(17): do_trace
faster_rcnn_script.py(64): save_jit_model
faster_rcnn_script.py(89): main
faster_rcnn_script.py(93): 
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
my code is:
cv::Mat pred_img = m_predict_img.clone();
#ifdef RGB
pred_img.convertTo(*m_img_float, CV_32FC3, 1.0 / 255.0);
#else
pred_img.convertTo(*m_img_float, CV_32FC1, 1.0 / 255.0);
#endif
#ifdef TRACE
torch::Tensor tensor_image = torch::from_blob(m_img_float->data, {1, m_img_float->rows,
m_img_float->cols, m_img_float->channels()}, torch::kF32);
tensor_image = tensor_image.permute({0, 3, 1, 2}); //trace
#else
torch::Tensor tensor_image = torch::from_blob(m_img_float->data, {m_img_float->rows,
m_img_float->cols, m_img_float->channels()}, torch::kF32);
tensor_image = tensor_image.permute({2, 0, 1}); //script
#endif
auto img_var = torch::autograd::make_variable(tensor_image, false);

std::vector<torch::jit::IValue> inputs;
torch::jit::IValue output;
if (check_gpu_available()) {

#ifdef TRACE
inputs.push_back(img_var.to(torch::kCUDA));
//inputs.push_back(img_var.to(torch::kCPU));
#else
inputs.push_back(c10::Listtorch::Tensor(img_var.to(torch::kCUDA)));
//inputs.push_back(c10::Listat::Tensor(img_var.to(torch::kCPU)));
#endif
std::cout << "before prediction!" << std::endl;
output = m_crop_module.forward(inputs);
std::cout << "after prediction!" << std::endl;
} else {
inputs.push_back(c10::Listat::Tensor({img_var.to(torch::kCPU)}));
output = m_crop_module.forward(inputs);
}
inputs.pop_back();
auto out = output.toTuple()->elements();

My environment is:
ubuntu20.04+libtorch1.6.0+torchvision0.7.0

Duplicate issue #60929: faster rcnn prediction failure
Created: 2021-06-29T05:39:24Z
🐛 Bug
I use pytorch trained a faster rcnn model to locate the hand in a image. it is ok at python environment. But when I transfer its to torchscript model and then deploy it on c++. For the CPU it is still ok to predict. but it always a problem on GPU. Then I use gdb to debug and its log listed as follow. can anyone tell me why it is?
(gdb) bt
#0  0x00007fff9c55db70 in  () at /lib/x86_64-linux-gnu/libcuda.so.1
#1  0x00007fff9c2fab02 in  () at /lib/x86_64-linux-gnu/libcuda.so.1
#2  0x00007fff9c3a13a3 in cuLaunchKernel () at /lib/x86_64-linux-gnu/libcuda.so.1
#3  0x00007fffa7465f40 in torch::jit::fuser::cuda::FusedKernelCUDA::FusedKernelCUDA(short, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::vector<torch::jit::fuser::TensorDesc, std::allocatortorch::jit::fuser::TensorDesc >, std::vector<torch::jit::fuser::TensorDesc, std::allocatortorch::jit::fuser::TensorDesc >, std::vector<torch::jit::fuser::PartitionDesc, std::allocatortorch::jit::fuser::PartitionDesc >, std::vector<torch::jit::fuser::PartitionDesc, std::allocatortorch::jit::fuser::PartitionDesc >, bool) () at /usr/local/libtorch/lib/libtorch_cuda.so
#4  0x00007fffa746a50d in torch::jit::fuser::cuda::createFusionKernel(short, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::vector<torch::jit::fuser::TensorDesc, std::allocatortorch::jit::fuser::TensorDesc >, std::vector<torch::jit::fuser::TensorDesc, std::allocatortorch::jit::fuser::TensorDesc >, std::vector<torch::jit::fuser::PartitionDesc, std::allocatortorch::jit::fuser::PartitionDesc >, std::vector<torch::jit::fuser::PartitionDesc, std::allocatortorch::jit::fuser::PartitionDesc >, bool) () at /usr/local/libtorch/lib/libtorch_cuda.so
#5  0x00007fffa746cf84 in std::_Function_handler<std::shared_ptrtorch::jit::fuser::FusedKernel (short, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::vector<torch::jit::fuser::TensorDesc, std::allocatortorch::jit::fuser::TensorDesc >, std::vector<torch::jit::fuser::TensorDesc, std::allocatortorch::jit::fuser::TensorDesc >, std::vector<torch::jit::fuser::PartitionDesc, std::allocatortorch::jit::fuser::PartitionDesc >, std::vector<torch::jit::fuser::PartitionDesc, std::allocatortorch::jit::fuser::PartitionDesc >, bool), std::shared_ptrtorch::jit::fuser::FusedKernel ()(short, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::vector<torch::jit::fuser::TensorDesc, std::allocatortorch::jit::fuser::TensorDesc >, std::vector<torch::jit::fuser::TensorDesc, std::allocatortorch::jit::fuser::TensorDesc >, std::vector<torch::jit::fuser::PartitionDesc, std::allocatortorch::jit::fuser::PartitionDesc >, std::vector<torch::jit::fuser::PartitionDesc, std::allocatortorch::jit::fuser::PartitionDesc >, bool)>::_M_invoke(std::_Any_data const&, short&&, std::__cxx11::basic_string<char, std::char_traits, std::allocator >&&, std::__cxx11::basic_string<char, std::char_traits, std::allocator >&&, std::vector<torch::jit::fuser::TensorDesc, std::allocatortorch::jit::fuser::TensorDesc >&&, std::vector<torch::jit::fuser::TensorDesc, std::allocatortorch::jit::fuser::TensorDesc >&&, std::vector<torch::jit::fuser::PartitionDesc, std::allocatortorch::jit::fuser::PartitionDesc >&&, std::vector<torch::jit::fuser::PartitionDesc, std::allocatortorch::jit::fuser::PartitionDesc >&&, bool&&) () at /usr/local/libtorch/lib/libtorch_cuda.so
#6  0x00007fffe6343cd6 in torch::jit::fuser::compileKernel(torch::jit::fuser::KernelSpec const&, torch::jit::fuser::ArgSpec const&, std::vector<long, std::allocator > const&, c10::Device) () at /usr/local/libtorch/lib/libtorch_cpu.so
#7  0x00007fffe6353520 in torch::jit::fuser::runFusion(long, std::vector<c10::IValue, std::allocatorc10::IValue >&, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) () at /usr/local/libtorch/lib/libtorch_cpu.so
#8  0x00007fffe635b4d3 in torch::jit::runFusion(long, std::vector<c10::IValue, std::allocatorc10::IValue >&) () at /usr/local/libtorch/lib/libtorch_cpu.so
#9  0x00007fffe6742f40 in std::_Function_handler<int (std::vector<c10::IValue, std::allocatorc10::IValue >&), torch::jit::(anonymous namespace)::{lambda(torch::jit::Node const*)#2}::operator()(torch::jit::Node const*) const::{lambda(std::vector<c10::IValue, std::allocatorc10::IValue >&)#1}>::_M_invoke(std::_Any_data const&, std::vector<c10::IValue, std::allocatorc10::IValue >&) () at /usr/local/libtorch/lib/libtorch_cpu.so
#10 0x00007fffe65dfa42 in torch::jit::InterpreterStateImpl::runImpl(std::vector<c10::IValue, std::allocatorc10::IValue >&) () at /usr/local/libtorch/lib/libtorch_cpu.so
#11 0x00007fffe65d1051 in torch::jit::InterpreterState::run(std::vector<c10::IValue, std::allocatorc10::IValue >&) () at /usr/local/libtorch/lib/libtorch_cpu.so
#12 0x00007fffe65c9043 in torch::jit::(anonymous namespace)::DifferentiableGraphOp::operator()(std::vector<c10::IValue, std::allocatorc10::IValue >&) const ()
at /usr/local/libtorch/lib/libtorch_cpu.so
#13 0x00007fffe65dfa42 in torch::jit::InterpreterStateImpl::runImpl(std::vector<c10::IValue, std::allocatorc10::IValue >&) () at /usr/local/libtorch/lib/libtorch_cpu.so
#14 0x00007fffe65d1051 in torch::jit::InterpreterState::run(std::vector<c10::IValue, std::allocatorc10::IValue >&) () at /usr/local/libtorch/lib/libtorch_cpu.so
#15 0x00007fffe65c70e1 in torch::jit::GraphExecutorImplBase::run(std::vector<c10::IValue, std::allocatorc10::IValue >&) () at /usr/local/libtorch/lib/libtorch_cpu.so
#16 0x00007fffe631b18a in torch::jit::GraphFunction::operator()(std::vector<c10::IValue, std::allocatorc10::IValue >, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, c10::IValue, std::hash<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, c10::IValue> > > const&) () at /usr/local/libtorch/lib/libtorch_cpu.so
#17 0x00007fffe632c32d in torch::jit::Method::operator()(std::vector<c10::IValue, std::allocatorc10::IValue >, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, c10::IValue, std::hash<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::equal_to<std::cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::cxx11::basic_string<char, std::char_traits, std::allocator > const, c10::IValue> > > const&) () at /usr/local/libtorch/lib/libtorch_cpu.so
#18 0x00000000004437e6 in torch::jit::Module::forward(std::vector<c10::IValue, std::allocatorc10::IValue >) (this=0x1ce1820, inputs=...)
at /usr/local/libtorch/include/torch/csrc/jit/api/module.h:114
#19 0x000000000043d0ba in deepLearning::crop_hand(std::vector<std::vector<cv::Point, std::allocator<cv::Point > >, std::allocator<std::vector<cv::Point, std::allocator<cv::Point > > > >&) (this=0x1ce14b0, hand_loc=std::vector of length 0, capacity 0) at ../Matching/deeplearning.cpp:118
#20 0x000000000043e2e5 in deepLearning::run() (this=0x1ce14b0) at ../Matching/deeplearning.cpp:193
#21 0x00007fffa1965845 in QThreadPrivate::start(void*) (arg=0x1ce14b0) at thread/qthread_unix.cpp:329
--Type  for more, q to quit, c to continue without paging--
#22 0x00007fffa1810609 in start_thread (arg=) at pthread_create.c:477
#23 0x00007fffa13e9293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)
(gdb)
(gdb) frame 18
#18 0x00000000004437e6 in torch::jit::Module::forward (this=0x1ce1820, inputs=...) at /usr/local/libtorch/include/torch/csrc/jit/api/module.h:114
114	    return get_method("forward")(i);
(gdb) print i
$1 = std::vector of length 1, capacity 1 = {{payload = {as_int = 140732926823168, as_double = 6.9531304382017387e-310, as_bool = false, as_intrusive_ptr = 0x7ffef01c8b00,
as_device = {type = -29952, index = -4068}}, tag = c10::IValue::Tag::GenericList, is_intrusive_ptr = true}}
(gdb)
my code can be checked as follow:
cv::Mat pred_img = m_predict_img.clone();
#ifdef RGB
pred_img.convertTo(*m_img_float, CV_32FC3, 1.0 / 255.0);
#else
pred_img.convertTo(*m_img_float, CV_32FC1, 1.0 / 255.0);
#endif
#ifdef TRACE
torch::Tensor tensor_image = torch::from_blob(m_img_float->data, {1, m_img_float->rows,
m_img_float->cols, m_img_float->channels()}, torch::kF32);
tensor_image = tensor_image.permute({0, 3, 1, 2}); //trace
#else
torch::Tensor tensor_image = torch::from_blob(m_img_float->data, {m_img_float->rows,
m_img_float->cols, m_img_float->channels()}, torch::kF32);
tensor_image = tensor_image.permute({2, 0, 1}); //script
#endif
auto img_var = torch::autograd::make_variable(tensor_image, false);

std::vector<torch::jit::IValue> inputs;
torch::jit::IValue output;
if (check_gpu_available()) {

#ifdef TRACE
//inputs.push_back(img_var.to(torch::kCUDA));
inputs.push_back(img_var.to(torch::kCPU));
#else
inputs.push_back(c10::Listtorch::Tensor(img_var.to(torch::kCUDA)));
//inputs.push_back(c10::Listat::Tensor(img_var.to(torch::kCPU)));
#endif
std::cout << "before prediction!" << std::endl;
output = m_crop_module.forward(inputs);
std::cout << "after prediction!" << std::endl;
} else {
inputs.push_back(c10::Listat::Tensor({img_var.to(torch::kCPU)}));
output = m_crop_module.forward(inputs);
}
inputs.pop_back();
auto out = output.toTuple()->elements();

my environment: ubuntu20.04+libtorch1.6.0+torchvision0.7.0

Closing comment (2021-06-29T06:19:33Z): Closing as duplicate of #60930.