Difference between tensor.t() and tensor.T?

f3ba: <p>Given a matrix <code>A</code>, say:</p>
<pre><code class="lang-auto">A = torch.randn(5,5)
</code></pre>
<p>What is the difference between <code>A.T</code> and <code>A.t()</code>?</p>

ptrblck: <p>From the docs:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.t.html#torch.t"><code>tensor.t</code></a>:</p>
<blockquote>
<p>Expects <code>input</code> to be &lt;= 2-D tensor and transposes dimensions 0 and 1.<br>
0-D and 1-D tensors are returned as is. When input is a 2-D tensor this is equivalent to <code>transpose(input, 0, 1)</code>.</p>
</blockquote>
<p><a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.T"><code>tensor.T</code></a>:</p>
<blockquote>
<p>Returns a view of this tensor with its dimensions reversed.<br>
If <code>n</code> is the number of dimensions in <code>x</code>, <code>x.T</code> is equivalent to <code>x.permute(n-1, n-2, ..., 0)</code>.</p>
</blockquote>
<p>In your use case both will yield the same result.</p>

maxiuw: <p>I think also it is valuable ti add that <code>t()</code> won’t work for dim &gt;= 2, but <code>transpose</code> will. so  so then it is necessary to use <code>transpose(input, 0, 1).</code> as ptrblck kinda wrote above</p>

MinSnz: <p>By testing on my GPU machine, for 2-D Tensors <code>x.T</code> is consistently, slightly faster than <code>x.t()</code>.</p>
<p><code>x.T</code> is also syntactically shorter and more intuitive so it should be preferred.</p>

ptrblck: <aside class="quote no-group" data-username="MinSnz" data-post="4" data-topic="165640">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/minsnz/48/76345_2.png" class="avatar"> MinSnz:</div>
<blockquote>
<p>for 2-D Tensors <code>x.T</code> is consistently, slightly faster than <code>x.t()</code>.</p>
</blockquote>
</aside>
<p>Could you show your profiling code to measure this overhead?</p>

MinSnz: <p>Yes, here:</p>
<pre data-code-wrap="python"><code class="lang-python">import time

import pandas as pd
import torch


def benchmark_transpose(dims_list, num_iterations=100000):
    results = []

    for dims in dims_list:
        x = torch.randn(*dims, device="cuda")
        # Warmup
        for _ in range(100):
            _ = x.T
            _ = x.t()

        # Benchmark x.T
        torch.cuda.synchronize()
        start_time = time.perf_counter()
        for _ in range(num_iterations):
            _ = x.T
            torch.cuda.synchronize()
        end_time = time.perf_counter()
        t_property_time = (end_time - start_time) / num_iterations

        # Benchmark x.t()
        torch.cuda.synchronize()
        start_time = time.perf_counter()
        for _ in range(num_iterations):
            _ = x.t()
            torch.cuda.synchronize()
        end_time = time.perf_counter()
        t_method_time = (end_time - start_time) / num_iterations

        results.append(
            {
                "dimensions": dims,
                "tensor.T (ms)": t_property_time * 1000,
                "tensor.t() (ms)": t_method_time * 1000,
                "size": x.nelement() * x.element_size(),
            }
        )

    return pd.DataFrame(results)


if __name__ == "__main__":
    assert torch.cuda.is_available()
    dims_to_test = [
        (5, 5),
        (55, 55),
        (555, 555),
        (5555, 5555),
        (25555, 25555),
    ]
    results_df = benchmark_transpose(dims_to_test)

    # Format and display results
    pd.set_option("display.float_format", lambda x: "{:.4f}".format(x))
    results_df["size_mb"] = results_df["size"] / (1024 * 1024)
    results_df["T/t() ratio"] = (
        results_df["tensor.T (ms)"] / results_df["tensor.t() (ms)"]
    )

    print("\nTranspose Operations Benchmark Results:")
    print("======================================")
    print(
        results_df[
            ["dimensions", "tensor.T (ms)", "tensor.t() (ms)", "size_mb", "T/t() ratio"]
        ]
    )
</code></pre>
<p>Local results:</p>
<pre data-code-wrap="python"><code class="lang-python">Transpose Operations Benchmark Results:
======================================
       dimensions  tensor.T (ms)  tensor.t() (ms)   size_mb  T/t() ratio
0          (5, 5)         0.0080           0.0083    0.0001       0.9664
1        (55, 55)         0.0081           0.0083    0.0115       0.9656
2      (555, 555)         0.0080           0.0083    1.1750       0.9675
3    (5555, 5555)         0.0079           0.0082  117.7140       0.9619
4  (25555, 25555)         0.0080           0.0083 2491.2187       0.9648
</code></pre>

ptrblck: <p>I don’t think your benchmark shows a real difference and would claim the noise is to large when you are measuring <code>us</code> intervals with host timers.<br>
E.g. if I move the syncs outside of the for loop right before stopping the host timers I get the opposite results:</p>
<pre data-code-wrap="python"><code class="lang-python"># Transpose Operations Benchmark Results:
# ======================================
#        dimensions  tensor.T (ms)  tensor.t() (ms)   size_mb  T/t() ratio
# 0          (5, 5)         0.0007           0.0007    0.0001       1.0178
# 1        (55, 55)         0.0007           0.0007    0.0115       1.0174
# 2      (555, 555)         0.0007           0.0007    1.1750       1.0191
# 3    (5555, 5555)         0.0007           0.0007  117.7140       1.0181
# 4  (25555, 25555)         0.0007           0.0007 2491.2187       1.0254
</code></pre>

MinSnz: <aside class="quote no-group" data-username="ptrblck" data-post="7" data-topic="165640">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>E.g. if I move the syncs outside of the for loop right before stopping the host timers I get the opposite results</p>
</blockquote>
</aside>
<p>Interesting when I do the same on different architectures. On my Tesla T4, I still get similar results:</p>
<pre data-code-wrap="python"><code class="lang-python">Transpose Operations Benchmark Results:
======================================
       dimensions  tensor.T (ms)  tensor.t() (ms)   size_mb  T/t() ratio
0          (5, 5)         0.0013           0.0014    0.0001       0.9411
1        (55, 55)         0.0013           0.0014    0.0115       0.9168
2      (555, 555)         0.0013           0.0014    1.1750       0.9211
3    (5555, 5555)         0.0013           0.0014  117.7140       0.9319
4  (25555, 25555)         0.0013           0.0014 2491.2187       0.9386
</code></pre>
<p>But on an A100, results are similar to yours:</p>
<pre data-code-wrap="python"><code class="lang-python">Transpose Operations Benchmark Results:
======================================
       dimensions  tensor.T (ms)  tensor.t() (ms)   size_mb  T/t() ratio
0          (5, 5)         0.0008           0.0007    0.0001       1.0309
1        (55, 55)         0.0007           0.0007    0.0115       1.0317
2      (555, 555)         0.0008           0.0007    1.1750       1.0375
3    (5555, 5555)         0.0007           0.0007  117.7140       1.0349
4  (25555, 25555)         0.0007           0.0007 2491.2187       1.0163
</code></pre>
<p>Anyway, then I tried the <a href="https://discuss.pytorch.org/t/how-to-measure-time-in-pytorch/26964/2">recommended</a> <code>torch.cuda.Event</code> for timing, and finally got results that drew the same conclusion on both architectures: <code>x.T</code> is indeed <strong>(slightly) faster</strong> than <code>x.t()</code>; the difference is yet less significant on newer architectures.</p>
<pre data-code-wrap="python"><code class="lang-python"># T4:
Transpose Operations Benchmark Results:
======================================
       dimensions  tensor.T (ms)  tensor.t() (ms)   size_mb  T/t() ratio
0          (5, 5)         1.2723           1.3940    0.0001       0.9127
1        (55, 55)         1.2873           1.3875    0.0115       0.9278
2      (555, 555)         1.2917           1.3887    1.1750       0.9302
3    (5555, 5555)         1.2803           1.3813  117.7140       0.9269
4  (25555, 25555)         1.1278           1.3752 2491.2187       0.8201

# A100:
Transpose Operations Benchmark Results:
======================================
       dimensions  tensor.T (ms)  tensor.t() (ms)   size_mb  T/t() ratio
0          (5, 5)         0.6749           0.6923    0.0001       0.9748
1        (55, 55)         0.6860           0.6945    0.0115       0.9877
2      (555, 555)         0.6919           0.6969    1.1750       0.9929
3    (5555, 5555)         0.6956           0.6996  117.7140       0.9943
4  (25555, 25555)         0.6814           0.6970 2491.2187       0.9776
</code></pre>
<p>Full code:</p>
<pre data-code-wrap="python"><code class="lang-python">import pandas as pd
import torch


def benchmark_transpose(dims_list, num_iterations=100000):
    results = []

    for dims in dims_list:
        x = torch.randn(*dims, device="cuda")
        # Warmup
        for _ in range(100):
            _ = x.T
            _ = x.t()
        start_event = torch.cuda.Event(enable_timing=True)
        end_event = torch.cuda.Event(enable_timing=True)

        # Benchmark x.T
        start_event.record()
        for _ in range(num_iterations):
            _ = x.T
        end_event.record()
        torch.cuda.synchronize()
        t_property_time = start_event.elapsed_time(end_event) / num_iterations

        # Benchmark x.t()
        start_event.record()
        for _ in range(num_iterations):
            _ = x.t()
        end_event.record()
        torch.cuda.synchronize()
        t_method_time = start_event.elapsed_time(end_event) / num_iterations

        results.append(
            {
                "dimensions": dims,
                "tensor.T (ms)": t_property_time * 1000,
                "tensor.t() (ms)": t_method_time * 1000,
                "size": x.nelement() * x.element_size(),
            }
        )

    return pd.DataFrame(results)


if __name__ == "__main__":
    assert torch.cuda.is_available()
    dims_to_test = [
        (5, 5),
        (55, 55),
        (555, 555),
        (5555, 5555),
        (25555, 25555),
    ]
    results_df = benchmark_transpose(dims_to_test)

    # Format and display results
    pd.set_option("display.float_format", lambda x: "{:.4f}".format(x))
    results_df["size_mb"] = results_df["size"] / (1024 * 1024)
    results_df["T/t() ratio"] = (
        results_df["tensor.T (ms)"] / results_df["tensor.t() (ms)"]
    )

    print("\nTranspose Operations Benchmark Results:")
    print("======================================")
    print(
        results_df[
            ["dimensions", "tensor.T (ms)", "tensor.t() (ms)", "size_mb", "T/t() ratio"]
        ]
    )
</code></pre>
<p><em>Edit: typo slower → faster</em> <a class="mention" href="/u/ptrblck">@ptrblck</a></p>