Empty_cpu doesn't set tensor's memory_format

tgsmdww: <pre><code class="lang-auto">Tensor empty_cpu(IntArrayRef size, const TensorOptions&amp; options_, c10::optional&lt;c10::MemoryFormat&gt; optional_memory_format) {
  ......
  auto memory_format = options.memory_format_opt().value_or(MemoryFormat::Contiguous);
  tensor.unsafeGetTensorImpl()-&gt;empty_tensor_restride(memory_format);

  return tensor;
}
</code></pre>
<p>Here tensor.options().has_memory_format is false. When I want to copy tensor to CUDA, which will call “to_impl”, it will come into</p>
<pre><code class="lang-auto">if (memory_format == MemoryFormat::Preserve) {
    if (self.is_non_overlapping_and_dense()) {
      // Copy all strides
      auto r = at::empty_strided(self.sizes(), self.strides(), options.memory_format(c10::nullopt));
      r.copy_(self, non_blocking);
      return r;
    } else {
      memory_format = self.suggest_memory_format();
    }
  }
</code></pre>
<p>Must I use <code>tensor.to("CUDA", memory_format=torch.contiguous_format)</code> to set it’s memory_format to Contiguous?</p>

ptrblck: <p>The <code>memory_format</code> seems to be respected in this code snippet:</p>
<pre><code class="lang-python">x = torch.empty(2, 3, 4, 5, memory_format=torch.contiguous_format)
print(x.is_contiguous(memory_format=torch.contiguous_format))
print(x.is_contiguous(memory_format=torch.channels_last))

y = x.to('cuda')
print(y.is_contiguous(memory_format=torch.contiguous_format))
print(y.is_contiguous(memory_format=torch.channels_last))


x = torch.empty(2, 3, 4, 5, memory_format=torch.channels_last)
print(x.is_contiguous(memory_format=torch.contiguous_format))
print(x.is_contiguous(memory_format=torch.channels_last))

y = x.to('cuda')
print(y.is_contiguous(memory_format=torch.contiguous_format))
print(y.is_contiguous(memory_format=torch.channels_last))
</code></pre>
<p>Could you post a code snippet, which would break it, please?</p>

tgsmdww: <p>I mean tensor doesn’t have default memory_format unless we explicitly specify his memory_format.</p>

ptrblck: <p>The default format is <code>torch.contiguous_format</code>, so you wouldn’t have to specify it.</p>

tgsmdww: <aside class="quote no-group" data-username="tgsmdww" data-post="1" data-topic="84701">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tgsmdww/48/25107_2.png" class="avatar"> tgsmdww:</div>
<blockquote>
<p>tensor.options().has_memory_format</p>
</blockquote>
</aside>
<p>I don’t understand why tensor.options().has_memory_format() is always false no matter I set it’s memory_format or not.</p>

ptrblck: <p>I’m not sure I understand the use case correctly.<br>
Based on your first code snippet it seems you are trying to create a contiguous tensor, so you wouldn’t need to pass a specific memory format to the initialization of the tensor.<br>
Let me know, if I misunderstand the use case.</p>

tgsmdww: <p>I only want to know how to make <code>tensor.options().has_memory_format()</code> return true.</p>

ptrblck: <p>This flag would be <code>true</code>, if you explicitly create a <code>TensorOptions</code> object and set the memory format as shown in this example:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn
from torch.utils import cpp_extension

cuda_source = """
std::vector&lt;torch::Tensor&gt; my_fun(void)
{
    auto options = at::TensorOptions(at::MemoryFormat::ChannelsLast);
    auto out = torch::ones({2, 3, 4, 5}, options);
    std::cout &lt;&lt; std::boolalpha &lt;&lt; options.has_memory_format() &lt;&lt; std::endl;
    return {out};
}
"""

cpp_source = """
    std::vector&lt;torch::Tensor&gt; my_fun(void);
"""

module = torch.utils.cpp_extension.load_inline(
    name="cuda_test_extension",
    cpp_sources=cpp_source,
    cuda_sources=cuda_source,
    functions="my_fun",
    verbose=True,
)

out = module.my_fun()
</code></pre>