Issue 1:
I’m trying to convert my model to ONNX format for further deployment in TensorRT. Here is a sample code to illustrate my problem in layer_norm here.

import torch
from torch import nn

class ExportModel(nn.Module):
def __init__(self):
super().__init__()

def forward(self, x):
# n, c, h, w = x.shape
# y = nn.functional.layer_norm(x, [c, h, w]) # not working
# y = nn.functional.layer_norm(x, x.size()[1:]) # not working
y = nn.functional.layer_norm(x, [16, 32, 128])

return y

def main():
model = ExportModel()

dummy_input = torch.randn(64, 16, 32, 128)
input_names = [ "input" ]
output_names = [ "output" ]

with torch.no_grad():
torch.onnx.export(
model, dummy_input, "sample.onnx", verbose=True,
input_names=input_names, output_names=output_names
)
return

if __name__ == '__main__':
main()

It could only work when the parameter of layer_norm is constant number. If not, the following error will occur.

Traceback (most recent call last):
File "sample.py", line 31, in <module>
main()
File "sample.py", line 26, in main
verbose=True, input_names=input_names, output_names=output_names
File "/opt/conda/lib/python3.6/site-packages/torch/onnx/__init__.py", line 148, in export
strip_doc_string, dynamic_axes, keep_initializers_as_inputs)
File "/opt/conda/lib/python3.6/site-packages/torch/onnx/utils.py", line 66, in export
dynamic_axes=dynamic_axes, keep_initializers_as_inputs=keep_initializers_as_inputs)
File "/opt/conda/lib/python3.6/site-packages/torch/onnx/utils.py", line 409, in _export
fixed_batch_size=fixed_batch_size)
File "/opt/conda/lib/python3.6/site-packages/torch/onnx/utils.py", line 289, in _model_to_graph
fixed_batch_size=fixed_batch_size)
File "/opt/conda/lib/python3.6/site-packages/torch/onnx/utils.py", line 132, in _optimize_graph
graph = torch._C._jit_pass_onnx(graph, operator_export_type)
File "/opt/conda/lib/python3.6/site-packages/torch/onnx/__init__.py", line 179, in _run_symbolic_function
return utils._run_symbolic_function(*args, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/torch/onnx/utils.py", line 647, in _run_symbolic_function
return op_fn(g, *inputs, **attrs)
File "/opt/conda/lib/python3.6/site-packages/torch/onnx/symbolic_helper.py", line 128, in wrapper
args = [_parse_arg(arg, arg_desc) for arg, arg_desc in zip(args, arg_descriptors)]
File "/opt/conda/lib/python3.6/site-packages/torch/onnx/symbolic_helper.py", line 128, in <listcomp>
args = [_parse_arg(arg, arg_desc) for arg, arg_desc in zip(args, arg_descriptors)]
File "/opt/conda/lib/python3.6/site-packages/torch/onnx/symbolic_helper.py", line 81, in _parse_arg
"', since it's not constant, please try to make "
RuntimeError: Failed to export an ONNX attribute 'onnx::Gather', since it's not constant, please try to make things (e.g., kernel size) static if possible

I have few code blocks in my model have layer_norm op. It would turn into some ugly code if I explicitly mark all parameters constant number. Is there any “best practice” of how to use dynamic shape for this kind of use case?

Also, I have posted the same issue on forum. I'm not sure where is the better place for this kind of quesion, so I duplicate the issue here.

Thanks in advance.

cc @houseroad @spandantiwari @lara-hdr @BowenBao @neginraoof

Issue 2:
🐛 Bug

I have encountered the following error in the backward pass of a simple 2 convolution layer model using nn.Conv3d.

torch.Size([1, 1, 256, 256, 256])
torch.Size([1, 1, 256, 256, 256])
Forward pass successful
Loss computed successfully
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-1-04985c43e191> in <module>
28 loss = criterion(output, y)
29 optimizer.zero_grad()
---> 30 loss.backward()

/home/software/miniconda3/envs/torch_gpu/lib/python3.7/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
164 products. Defaults to ``False``.
165 """
--> 166 torch.autograd.backward(self, gradient, retain_graph, create_graph)
167
168 def register_hook(self, hook):

/home/software/miniconda3/envs/torch_gpu/lib/python3.7/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
97 Variable._execution_engine.run_backward(
98 tensors, grad_tensors, retain_graph, create_graph,
---> 99 allow_unreachable=True) # allow_unreachable flag
100
101

RuntimeError: cuDNN error: CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input.

To Reproduce

This error is encountered when executing the following code for a simple 2 layer 3D convolutional neural network. An input tensor of size [1, 1, 256, 256, 256] is passed into the network. The first layer performs a 3D convolution with a 3x3x3 filter and padding=1, and the second layer performs a dilated 3D convolution with a 3x3x3 filter, dilation=2 and padding=2. The output tensors of the two layers are also of size [1, 1, 256, 256, 256].

import torch
from torch import nn

class Simple2Conv(nn.Module):
def __init__(self):
super(Simple2Conv, self).__init__()
self.layer_1 = nn.Conv3d(1, 1, 3, padding=1)
self.layer_2 = nn.Conv3d(1, 1, 3, padding=2, dilation=2)

def forward(self, x):
layer_1 = self.layer_1(x)
layer_2 = self.layer_2(layer_1)

print(layer_1.shape)
print(layer_2.shape)
return layer_2

model = Simple2Conv().cuda()

isize = 256
osize = isize
x = torch.FloatTensor(1,1,isize,isize,isize).cuda()
y = torch.LongTensor(1,osize ,osize ,osize).cuda()
criterion = nn.CrossEntropyLoss().cuda()
optimizer = torch.optim.Adam(model.parameters(), lr=0.1)

output = model(x)
print('Forward pass successful')

loss = criterion(output, y)
print('Loss computed successfully')
optimizer.zero_grad()
loss.backward()
print('Backward pass successful')

Expected behavior

The expected output of the code snippet without the error is:

torch.Size([1, 1, 256, 256, 256])
torch.Size([1, 1, 256, 256, 256])
Forward pass successful
Loss computed successfully
Backward pass successful

Any of the following changes will resolve the error:

replace the dilated convolution layer with a regular convolution layer (i.e. duplicate the first layer)

swap the order of the two convolution layers (i.e. place the dilated convolution layer first)

Having only a single (dilated) convolution layer (i.e. remove either convolution layer)

for the dilated convolution layer, change padding=2 to padding=1 (note: this requires modification of osize = isize to osize=isize-2)

use CPU instead of GPU (by removing .cuda() throughout)

reduce isize to any number <=255

Environment

(from running python collect_env.py)

PyTorch version: 1.3.1

Is debug build: No

CUDA used to build PyTorch: 10.1.243

OS: CentOS Linux 7 (Core)

GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)

CMake version: Could not collect (manually added from cmake --version: 3.14.6)

Python version: 3.7

Is CUDA available: Yes

CUDA runtime version: Could not collect (manually added from nvidia-smi: 10.1)

GPU models and configuration:

GPU 0: Tesla V100-SXM2-32GB

Nvidia driver version: 418.87.01

cuDNN version: Could not collect (manually added from numba -s | grep cudnn: 7.6.3)

Versions of relevant libraries:

[pip3] numpy==1.16.5

[conda] _pytorch_select 0.2 gpu_0

[conda] blas 1.0 mkl

[conda] mkl 2019.4 243

[conda] mkl-service 2.3.0 py37he904b0f_0

[conda] mkl_fft 1.0.15 py37ha843d7b_0

[conda] mkl_random 1.1.0 py37hd6b4f25_0

[conda] pytorch 1.3.1 py3.7_cuda10.1.243_cudnn7.6.3_0 pytorch

[conda] pytorch-gpu 1.3.1 0

[conda] torchsummary 1.5.1 pypi_0 pypi

[conda] torchvision 0.4.2 py37_cu101 pytorch

Additional context

Note in relation to issue #22496, tensor sizes in this example are only 256^3 = 2^24, i.e. < 2^31, thus do not violate the cudnn limit of 2^31. This error has been replicated on other systems with different versions of CUDA and PyTorch.