Issue 1:
🐛 Bug

This bug report is created as instructed by the fail message below:

Traceback (most recent call last):
File "qat_residual.py", line 60, in <module>
TRAINING_DEVICE, print_freq=500)
File "/home/lijun/repo/quant/utils.py", line 73, in train_one_epoch
output = model(image)
File "/home/lijun/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/home/lijun/repo/quant/residual_model.py", line 312, in forward
return self._forward_impl(x)
File "/home/lijun/repo/quant/residual_model.py", line 295, in _forward_impl
x = self.conv1(x)
File "/home/lijun/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/home/lijun/conda/lib/python3.7/site-packages/torch/nn/qat/modules/conv.py", line 36, in forward
self._conv_forward(input, self.weight_fake_quant(self.weight)))
File "/home/lijun/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
result = self.forward(*input, **kwargs)
File "/home/lijun/conda/lib/python3.7/site-packages/torch/quantization/fake_quantize.py", line 86, in forward
self.ch_axis, self.quant_min, self.quant_max)
RuntimeError: iter.device(arg).is_cuda() INTERNAL ASSERT FAILED at /opt/conda/conda-bld/pytorch_1587428398394/work/aten/src/ATen/native/cuda/Loops.cuh:56, please report a bug to PyTorch.

To Reproduce

Steps to reproduce the behavior:

A float-point number ResNet was trained

The pre-trained ResNet was prepared for QAT

When QAT ran on the first epoch, it raised the message above.

Expected behavior

The model should be able to move from system memory to GPU memory.

Environment

Please copy and paste the output from our

environment collection script

(or fill out the checklist below manually).

You can get the script and run it with:

wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py

PyTorch Version (e.g., 1.0):

OS (e.g., Linux):

How you installed PyTorch (conda, pip, source):

Build command you used (if compiling from source):

Python version:

CUDA/cuDNN version:

GPU models and configuration:

Any other relevant information:

Additional context

cc @jerryzh168 @jianyuh @dzhulgakov @raghuramank100 @jamesr66a

Issue 2:
🐛 Bug

I'm following this tutorial : https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html

This tutorial uses a CPU, it works fine with a CPU, but to speed it up I want to move it to the GPU and I get an error. It works with Pytorch version 1.4 but not with Pytorch nightly (1.5.0.dev20200413).

To Reproduce

Steps to reproduce the behavior:

1.Follow the following tutorial:

https://pytorch.org/tutorials/advanced/static_quantization_tutorial.html

In the "5. Quantization-aware training" section

Instead of doing

for nepoch in range(8):
train_one_epoch(qat_model, criterion, optimizer, data_loader, torch.device('cpu'), num_train_batches)

do :

device = torch.device("cuda:0")
for nepoch in range(8):
device = torch.device("cuda:0")
train_one_epoch(qat_model, criterion, optimizer, train_loader, torch.device(device), num_train_batches)

also changed in the section "2. Helper functions":

def load_model(model_file):
model = MobileNetV2()
state_dict = torch.load(model_file)
model.load_state_dict(state_dict)
model.to(device) #device instead of 'cpu'
return model

3.Get the error

F:\Anaconda3\envs\pytorch-nightly\lib\site-packages\torch\quantization\fake_quantize.py in forward(self, X)
84 if self.qscheme == torch.per_channel_symmetric or self.qscheme == torch.per_channel_affine:
85 X = torch.fake_quantize_per_channel_affine(X, self.scale, self.zero_point,
---> 86 self.ch_axis, self.quant_min, self.quant_max)
87 else:
88 X = torch.fake_quantize_per_tensor_affine(X, float(self.scale),

RuntimeError: iter.device(arg).is_cuda() INTERNAL ASSERT FAILED at C:/cb/pytorch_1000000000000/work/aten/src\ATen/native/cuda/Loops.cuh:60, please report a bug to PyTorch.

Expected behavior

No error, run smoothly. QAT is supposed to support CUDA.

Quote from the documentation

"Quantization-aware training (through FakeQuantize) supports both CPU and CUDA."

However, it works with PyTorch Version : 1.4.0.

Environment

PyTorch Version : 1.5.0.dev20200413 (pytorch nightly)

OS :Windows 10

How you installed PyTorch :conda (version 4.7.12)

Build command you used (if compiling from source):

conda install pytorch torchvision cudatoolkit=10.1 -c pytorch-nightly -c defaults -c conda-forge

Python version: 3.7.7

CUDA/cuDNN version: 10.1

GPU models and configuration: GPU: Nvidia Geforce RTX 2060 Super (8Go GDDR6)

Versions of relevant libraries:

[pip] numpy==1.18.1

[pip] torch==1.5.0.dev20200413

[pip] torchvision==0.7.0.dev20200415

[conda] blas 1.0 mkl

[conda] cudatoolkit 10.1.243 h74a9793_0

[conda] mkl 2020.0 166

[conda] mkl-service 2.3.0 py37hb782905_0

[conda] mkl_fft 1.0.15 py37h14836fe_0

[conda] mkl_random 1.1.0 py37h675688f_0

[conda] numpy 1.18.1 py37h93ca92e_0

[conda] numpy-base 1.18.1 py37hc3f5095_1

[conda] pytorch 1.5.0.dev20200413 py3.7_cuda101_cudnn7_0 pytorch-nightly

[conda] torchvision 0.7.0.dev20200415 py37_cu101 pytorch-nightly

cc @jerryzh168 @jianyuh @dzhulgakov @raghuramank100 @jamesr66a