Issue 1:
We define 4D tensor as stored in channels last memory format, when dimensions order is NCHW and C-strides < W-strides < H-strides < N-strides (If size of any dimension is equal to 1, this dimension strides value is not taken into account).

Channels last contiguous tensor is channel last tensor which occupies contiguous memory block. So x.is_contiguous(memory_format=torch.channels_last) checks if tensor is channels last contiguous.

The goal of the experiment is to use channels last memory format in all Resnet (https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py) model's operators and to measure performance gains on the Volta devices with CudNN library available.

This experiment requires:

Update operators kernels to follow the next rule: if one of the operator's inputs is channel last tensor, all outputs should also be in the channel last memory format.

For better performance gain, update DataLoader to output channel last tensors.

To avoid changing the model itself and more importantly, introduce this optimization to the existing saved models. We need to introduce the next changes:

to operator should preserve memory format. copy_device_to_device should be memory format aware. (#23899)

empty_like operator should preserve memory format by default. (#23899)

resize_as_ operator should be memory format aware. (#23899)

clone operator should preserve memory format. (#23899)

scatter and gather functions should be memory format aware. (#24121)

TensorIterator based point-wise operators should preserve memory format (#24038).

adaptive_avg_pool2d_cuda and adaptive_avg_pool2d_backward_cuda should have channel last optimized kernels. (#24396)

max_pool2d_with_indices_cuda and max_pool2d_with_indices_backward_cuda should have channel last optimized kernels (#24872).

cudnn_batch_norm and cudnn_batch_norm_backward should support channels last memory format. (#23861)

cudnn_convolution_forward and cudnn_convolution_backward should support channels last memory format. (#23861)

Writing memory format aware operators require special functions introduced in #23391

auto memory_format = input_tensor.suggest_memory_format();
auto output_tensor = at::empty(output_shape, memory_format);

switch (memory_format) {
case MemoryFormat::ChannelsLast: {
input_cl_contiguous = input_tensor.contiguous(
MemoryFormat::ChannelsLast); // if kernel requires memory contiguous
// tensor
// .... kernel code
break;
}
case MemoryFormat::Contiguous: {
// .... standard kernel
break;
}
default:
TORCH_CHECK(
false,
"Unsupported memory format. Supports only ChannelsLast, Contiguous");
}

Notes

Resnet calls x = x.reshape(x.size(0), -1) before linear layers, we are going to update reshape and view code and convert tensor's memory format to torch.contiguous_format at this step.

Making old models/libraries to work faster also requires BC sacrifices, such as

empty_like ( and all _like operators ) will return channels last tensor if input is channels last, similar will apply to to, clone, resize_as. We are thinking about the ability to control suggest_memory_format behaviors by the global variable.

Issue 2:
If you have a question or would like help and support, please ask at our

forums.

If you are submitting a feature request, please preface the title with [feature request].

If you are submitting a bug report, please fill in the following details.

Issue description

Unable to do git clone 0.4.0 of pytorch

Provide a short description.

Code example

Cloning into 'pytorch-src'...

remote: Counting objects: 101473, done.

remote: Compressing objects: 100% (62/62), done.

remote: Total 101473 (delta 24), reused 20 (delta 10), pack-reused 101401

Receiving objects: 100% (101473/101473), 54.60 MiB | 49.09 MiB/s, done.

Resolving deltas: 100% (75514/75514), done.

~/pytorch/pytorch-src ~/pytorch

Note: checking out 'v0.4.0'.

You are in 'detached HEAD' state. You can look around, make experimental

changes and commit them, and you can discard any commits you make in this

state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may

do so (now or later) by using -b with the checkout command again. Example:

git checkout -b

.....

Cloning into '/home/ec2-user/pytorch/pytorch-src/third_party/eigen'...

Username for 'https://github.com': gautamkmr

Password for 'https://gautamkmr@github.com':

remote: Repository not found.

fatal: repository 'https://github.com/RLovelett/eigen.git/' not found

fatal: clone of 'https://github.com/RLovelett/eigen.git' into submodule path '/home/ec2-user/pytorch/pytorch-src/third_party/eigen' failed

Failed to clone 'third_party/eigen' a second time, aborting

[ec2-user@ip-172-31-17-149 pytorch]$

System Info

Please copy and paste the output from our

environment collection script

(or fill out the checklist below manually).

You can get the script and run it with:

wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py

PyTorch or Caffe2:

How you installed PyTorch (conda, pip, source):

Build command you used (if compiling from source):

OS:

PyTorch version:

Python version:

CUDA/cuDNN version:

GPU models and configuration:

GCC version (if compiling from source):

CMake version:

Versions of any other relevant libraries: