akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Gymnasium FrozenLake - why one-hot encoding for state is required?

impostor: <p>Hello, complete beginner here. I’m trying to implement Deep Q-Learning on the FrozenLake environment.</p>
<p>The state space is <code>Discrete(16)</code>, and the action space is <code>Discrete(4)</code>, so I initially designed a simple model:</p>
<pre data-code-wrap="python"><code class="lang-python">nn.Sequential(
    nn.Linear(1, 128),
    nn.Sigmoid(),
    nn.Linear(128, 4)
)
</code></pre>
<p>However, it failed. I spent nearly three days on this without success. Core of my DQL algorithm is 100% correct (I even tried copying code from working solutions).</p>
<p>After reviewing other implementations, I noticed that everyone used one-hot encoding for state. So instead of directly feeding 1-16 to model they feed its one-hot encoding representation. When I applied it, my model suddenly started working and successfully solved the environment.</p>
<p>But I don’t understand <em>why</em> one-hot encoding is necessary. I found this comment:</p>
<p><a href="https://www.reddit.com/r/reinforcementlearning/comments/1bwmwt3/comment/ky9a2hb/?utm_source=share&amp;utm_medium=web3x&amp;utm_name=web3xcss&amp;utm_term=1" class="onebox" target="_blank" rel="noopener nofollow ugc">https://www.reddit.com/r/reinforcementlearning/comments/1bwmwt3/comment/ky9a2hb/?utm_source=share&amp;utm_medium=web3x&amp;utm_name=web3xcss&amp;utm_term=1</a></p>
<p>I kinda understand it but, I still don’t get why the model fails to “memorize”/“bruteforce” the correct action for each state without one-hot encoding. I also tried adding more hidden layers to help the network learn a better representation of the discrete states, but that didn’t work either.</p>
<p>Can someone explain why one-hot encoding is essential in this case?<br>
Is learning “gridworld” problem without one-hot encoding even possible?</p>
Help/suggestion with MeshGraphNets

Malaika: <p>I’m working with meshgraphnets, i tried predicting 3 outputs with a single head, error reduced for one output only, so i tried weighted loss , multi-component loss weighting and Gradnorm at the end, when i tried GradNorm i logged the gradient norms for loss components which now had different output heads, they approached 0 after a few train steps, also i logged gradients and saw same thing happened, so i tried multi head outputs without gradnorm, but i see gradients getting 0 after few starting steps, i was normalizing my inputs and outputs with z_score, i didn’t use activation at the decoder heads, what can be the possible reason for this, one of my output has a mean close to 0, but other 2 are skewed , they don’t have long tails but are multi-modal at either side of 0, has someone explored a similar problem, i’d appreciate any suggestions or recommendations.</p>

singh20anurag: <blockquote>
<p>i didn’t use activation at the decoder heads</p>
</blockquote>
<p>Why? No activation on the decoder can cause unbounded outputs. Have you tried adding one? E.g:<br>
<code>self.head = nn.Sequential(nn.Linear(...), nn.Tanh())</code></p>
<p>Also consider using min-max normalization as z-score can compress the signal if your distribution is skewed.</p>
Compiling earlier PyTorch versions on Blackwell

asiron: <p>It seems that if you have a 5000 series GPU (Blackwell) with Compute Capability 12.0, you are forced to use CUDA 12.8. Is it possible to compile older PyTorch versions (specifically 1.13 or 2.0) using CUDA 12.8 ? Unfortunately, I have some software that strictly requires 1.13 and Python 3.8.</p>
<p>I published this as an issue <a href="https://github.com/pytorch/pytorch/issues/150034#event-16997289003" class="inline-onebox" rel="noopener nofollow ugc">Compile earlier PyTorch versions on Blackwell · Issue #150034 · pytorch/pytorch · GitHub</a>, but it is unfortunately not planned. Therefore, I am looking for some guidance in the community. Has anyone succeeded ?</p>

ptrblck: <p>Already released binaries won’t be rebuilt again as was already mentioned in the GitHub issue. I would probably not bother trying to backport all needed changes into ancient PyTorch branches but would instead check why your repositories require these old versions and update them.</p>

asiron: <p>Thanks for the quick answer, unfortunately it’s Vitis-AI which is stuck at 1.13 or 2.0.0 and Python 3.8 and I dont believe I can do much, but I will try, perhaps I can meet somewhere in the middle</p>
Running GPU's on a PI or other edge device - See MyTorch.net

Vet_Ranger: <p>Hi Everyone,</p>
<p>I just wanted to share something with you all, just because I thought it was pretty cool.</p>
<p>There’s this team here in Colorado and they have this new pytorch add-on tool that I was asked to try. It allows anything that can run pytorch to connect and use GPUs with no code changes.</p>
<p>For example, I tried to run pytorch on a raspberry pi, and with this tool it ran on the pi but all the Llama model loading and GPU processing ran on some Nvidia V100 but it looked like it was on the PI. It was pretty awesome.</p>
<p>But the thing was that there were Zero code changes to make this happen. Just a simple pip install of an add-on package.</p>
<p>These guys are just in alpha mode, but it might be worth checking out.</p>
<p>see: <a href="https://mytorch.net/" rel="noopener nofollow ugc">https://mytorch.net</a></p>

Prasad_Raghavendra: <p>Nice idea. So, does it just do <code>device</code> automatically? Is that all what this “magic” is about? [like list <code>device</code> and then, pick all of them?]</p>
<p>What is the crux of the package?</p>

esch: <p>Hi everyone - I’m one of the guys from this mytorch team.</p>
<p>MyTorch currently selects from a pool of GPUs on the backend. For now, it’s our pool, but it could be a private or other public pool.  As a client system makes a GPU request via standard pytorch, that request is translated and transferred to the GPU, executed and the result of the call is then sent back.</p>
<p>Remember that this in Alpha at the moment and has limited flexibility.   Beta will let you add optional env vars or values to specify a specific GPU type.  Additionally we are only supporting a few Llama models for Alpha, though that is moving fast as models are not a limitation, the code does not care what model you would like to use.</p>
<p>The code is largely GPU agnostic, the Alpha pool will have MI100’s or V100’s, and we’re just about to enable Neuron devices from AWS and soon after that TPU or GPUs from Google.  The higher end devices will eventually be pay per time or token model but at a discounted price as we are working with several of these cloud vendors to get discounts.   But for now, everything is totally free on the lower end devices.  I think there will always be a free option, but can’t say that for sure.</p>
<p>We are currently taking on a few dozen Alpha users just to hear what folks think.  Total free, no credit card or anything like that.  Just provide an email address, we send you a token.  On the site is a short video, its a super simple install, it’s just a pip package.</p>
<p>Where is this going?  It’s an edge device ML option,  or “Internet of Intelligent things” (IoIT) - Further down the road we will see some very special sauce, but we are not there yet. - For now  we just want to hear what others think.</p>
<p>Thanks to “Vet_Ranger” for mentioning us here.</p>
<p>See: <a href="https://mytorch.net" rel="noopener nofollow ugc">https://mytorch.net</a></p>
Errors in Pytorch Profiler Example

ravikamtalwar: <p>Hi,</p>
<p>I’m trying to run example of pytorch profiler from <a href="https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Profiler — PyTorch Tutorials 2.6.0+cu124 documentation</a>.<br>
But there are errors as below:<br>
AttributeError: partially initialized module ‘torchvision.models’ has no attribute ‘resnet18’ (most likely due to a circular import)<br>
torch version - 2.6.0<br>
torchvision version - 0.21.0</p>
<p>Does this example require some specific versions?</p>

ptrblck: <p>Do you have a local file called <code>models.py</code> in your <code>wdir</code> or did you <code>cd</code> into the <code>vision</code> folder?</p>
Help me understand how to fix the error: Process finished with exit code -1073741819 (0xC0000005)

Behruz_Sherov: <pre><code>#include &lt;torch/torch.h&gt;
#include &lt;torch/script.h&gt;
#include &lt;iostream&gt;

// Определяем архитектуру сети
struct Net : torch::nn::Module {
    torch::nn::Linear fc1{nullptr}, fc2{nullptr}, fc3{nullptr};

    Net() {
        fc1 = register_module("fc1", torch::nn::Linear(784, 128));
        fc2 = register_module("fc2", torch::nn::Linear(128, 64));
        fc3 = register_module("fc3", torch::nn::Linear(64, 10));
    }

    torch::Tensor forward(torch::Tensor x) {
        x = x.view({-1, 784}); // Преобразуем входные данные в вектор
        x = torch::relu(fc1-&gt;forward(x));
        x = torch::relu(fc2-&gt;forward(x));
        x = fc3-&gt;forward(x);
        return torch::log_softmax(x, 1); // Выход с log_softmax
    }
};

int main() {
    // Указываем устройство CPU
    torch::Device device(torch::kCPU);

    // Создаем модель
    auto net = std::make_shared&lt;Net&gt;();
    net-&gt;to(device);

    auto data = torch::randn({32, 784}); // 32 изображения
    auto target = torch::randint(0, 10, {32}); // Метки

    torch::optim::SGD optimizer(net-&gt;parameters(), torch::optim::SGDOptions(0.01));

    for (int epoch = 0; epoch &lt; 10; epoch++) {
        optimizer.zero_grad();
        torch::Tensor output = net-&gt;forward(data);
        torch::Tensor loss = torch::nll_loss(output, target);
        loss.backward();
        optimizer.step();
        std::cout &lt;&lt; "Epoch: " &lt;&lt; epoch &lt;&lt; ", Loss: " &lt;&lt; loss.item&lt;float&gt;() &lt;&lt; std::endl;
    }

    std::string model = "neut_model.pth";
    torch::save(net, model);

    return 0;
}
</code></pre>
`.to(torch::kUInt8)` position caused weird issues

L_R: <p>I’m translating some python codes into C++.<br>
my original codes is like this</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">  pred_masks = pred_masks.ge(0.5).to(torch::kUInt8);
...
  for (auto i = 0; i &lt; pred_bboxes.sizes()[0]; i++) {
        auto res = func(pred_bboxes[i], pred_masks[i]);
  }
...
Segmentation func(
    const torch::Tensor &amp;pred_bbox, const torch::Tensor &amp;pred_mask) const {
...
  torch::Tensor const img =
      pred_mask.index({torch::indexing::Slice(
                           static_cast&lt;int&gt;(pred_bbox[1].item&lt;double&gt;()),
                           static_cast&lt;int&gt;(pred_bbox[3].item&lt;double&gt;()) + 1),
                       torch::indexing::Slice(
                           static_cast&lt;int&gt;(pred_bbox[0].item&lt;double&gt;()),
                           static_cast&lt;int&gt;(pred_bbox[2].item&lt;double&gt;()) + 1)});
...
</code></pre>
<p>but the masks are very wrong, I changed my codes to the following</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">  // pred_masks = pred_masks.ge(0.5).to(torch::kUInt8);
  pred_masks = pred_masks &gt;= 0.5; // dtype: bool
  pred_masks = pred_masks.to(torch::kBool);
  std::vector&lt;torch::Tensor&gt; mask_vec;
  for (int i = 0; i &lt; pred_masks.sizes()[0]; ++i) {
    mask_vec.push_back(pred_masks[i]);
  }
  pred_masks = torch::cat({mask_vec}, 0).squeeze(1);
  ...
  for (auto i = 0; i &lt; pred_bboxes.sizes()[0]; i++) {
        auto res = func(pred_bboxes[i], pred_masks[i]);
  }
  ...

Segmentation func(
    const torch::Tensor &amp;pred_bbox, const torch::Tensor &amp;pred_mask) const {
...
  torch::Tensor const img =
      pred_mask.index({torch::indexing::Slice(
                           static_cast&lt;int&gt;(pred_bbox[1].item&lt;double&gt;()),
                           static_cast&lt;int&gt;(pred_bbox[3].item&lt;double&gt;()) + 1),
                       torch::indexing::Slice(
                           static_cast&lt;int&gt;(pred_bbox[0].item&lt;double&gt;()),
                           static_cast&lt;int&gt;(pred_bbox[2].item&lt;double&gt;()) + 1)})
                      .to(torch::kUInt8);
  cv::Mat const mat(static_cast&lt;int&gt;(img.sizes()[0]),
                    static_cast&lt;int&gt;(img.sizes()[1]), CV_8UC1,
                    img.data_ptr&lt;uint8_t&gt;());
  mask.mask = std::move(
      *cv_bridge::CvImage(std_msgs::msg::Header(), "mono8", mat).toImageMsg());
...
</code></pre>
<p>and now the masks are normal. it is weird why the previous version could not give correct masks.<br>
The only difference is when <code>.to(torch::kUInt8)</code> happens (I found that using between <code>.ge(0.5)</code> or <code>&gt;=0.5</code> does not matter as well as long as I have put <code>.to(torch::kUInt8)</code> inside the function).<br>
and I think <code>pred_mask</code> should be <code>torch::Tensor</code>, but it is actually <code>at::Tensor</code> inside the function, which is weird</p>
Alternative to load training data from PostgreSQL DB

evilroach: <p>My current trainning data are stored in a PostgreSQL DB.<br>
Since the volume of the data is huge, it’s impossible to preload all the data into memory before the trainning.</p>
<p>I implemented my own Dataset. When it being iterated, it loads features from the DB one by one.<br>
And then I use a torch.utils.data.DataLoader to iterate it when trainning.</p>
<p>When I pass a low value(1~32) of the parameter <code>num_workers</code> to the DataLoader, the trainning process goes normally but the loading throughput is low, so that the usage of the GPU is very low.</p>
<p>When <code>num_workers</code> hit or exceed 64, the loading rate stop to increase, and I found that the main process is using 100% CPU of a core.</p>
<p>I guess the reason perhaps is that in the main process, DataLoader has a lot of communicating or synchronizing works with sub-processes to do. The more sub-processes, the more sync overhead, so that the overhead of receiving data from the sub-processes become a bottleneck of the main process.</p>
<p>I don’t know whether above analysis is right.<br>
If so, would you pls tech me a correct way to load data from a SQL DB?<br>
Maybe I should use a memory DB such as mongodb/redis instead? With that type of DB, I don’t need to use so many workers to load data?<br>
Thanks!</p>
v2.Compose([v2.ToImage(), v2.ToDtype(torch.float32, scale=True)]) return Image instead of Tensor?

Marcus_Wong: <p>[<a href="https://pytorch.org/vision/main/generated/torchvision.transforms.v2.ToTensor.html#:~:text=v2.ToTensor%20is%20deprecated%20and%20will%20be%20removed%20in,scale%3DTrue%29%5D%29.%20Output%20is%20equivalent%20up%20to%20float%20precision." class="inline-onebox" rel="noopener nofollow ugc">ToTensor — Torchvision main documentation</a>]([v2.ToTensor()] [DEPRECATED] Use v2.Compose([v2.ToImage(), v2.ToDtype(torch.float32, scale=True)]) instead.)</p>
<p>v2.Compose([v2.ToTensor()]) is return a Tensor<br>
But v2.Compose([v2.ToImage(), v2.ToDtype(torch.float32, scale=True)]) is return a Image</p>
<p>Are there steps missing?</p>

ptrblck: <p>The second transformation will return a <a href="https://pytorch.org/vision/main/generated/torchvision.tv_tensors.Image.html#torchvision.tv_tensors.Image"><code>torchvision.tv_tensors._image.Image</code></a> as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">transform = v2.Compose([v2.ToImage(), v2.ToDtype(torch.float32, scale=True)])
image = torchvision.transforms.ToPILImage()(torch.randn(3, 224, 224))
out = transform(image)
print(type(out))
# &lt;class 'torchvision.tv_tensors._image.Image'&gt;
</code></pre>
<p>If you want to access the internal tensor use the <code>.data</code> attribute as shown in the docs.</p>
PyTorch compatible GPU rental?

Geremia: <p>What are some cloud-based, GPU rental services that support PyTorch?</p>

ptrblck: <p>All NVIDIA GPUs between Maxwell (sm_50) and Blackwell (sm_120) are supported in current PyTorch binary builds.</p>

Geremia: <aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/get-started/cloud-partners/">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/get-started/cloud-partners/" target="_blank" rel="noopener nofollow ugc">PyTorch</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/362;"><img src="https://pytorch.org/assets/images/social-share.jpg" class="thumbnail" width="690" height="362"></div>

<h3><a href="https://pytorch.org/get-started/cloud-partners/" target="_blank" rel="noopener nofollow ugc">Start via Cloud Partners</a></h3>

  <p>Start via Cloud Partners</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
Lightning.ai gives “35 free GPU hours”.</p>
Using Symmetric Memory One Shot All Reduce

wdykas: <p>I am attempting to use symmetric memory for all reduces, specifically the one shot kernel but I am hitting errors. There doesn’t seem to be much documentation on these features and I was wondering if there are any examples I can take a look at to use. Below is a small reproducer of the errors that I am seeing:</p>
<pre><code class="lang-auto">import os
import torch
import torch.distributed as torch_dist
import torch.distributed._symmetric_memory as symm_mem

def main():
    # Initialize the distributed environment
    local_rank = int(os.environ.get("LOCAL_RANK", 0))
    rank = int(os.environ.get("RANK", 0))
    world_size = int(os.environ.get("WORLD_SIZE", 1))
    
    # Set the device
    device = torch.device(f"cuda:{local_rank}")
    torch.cuda.set_device(device)
    
    # Initialize the process group
    torch_dist.init_process_group("nccl")
    
    # Create an explicit process group with all processes
    ranks = list(range(world_size))
    explicit_group = torch_dist.new_group(ranks=ranks)
    
    # Get the group name from the explicit group
    group_name = explicit_group.group_name
    
    if rank == 0:
        print(f"Created group with name: {group_name}")
    
    # Create a simple tensor with values equal to the rank
    tensor_size = 1024
    tensor_regular = torch.ones(tensor_size, device=device) * rank
    tensor_copy = tensor_regular.clone()
    
    # Regular all-reduce using the explicit group
    torch_dist.all_reduce(tensor_regular, op=torch_dist.ReduceOp.SUM, group=explicit_group)
    
    try:
        if rank == 0:
            print(f"Using group name for symmetric memory: {group_name}")
        
        # Create symmetric memory tensor 
        msg = symm_mem.empty(
            tensor_size,
            dtype=torch.float32,
            device=device,
        )
        
        # Perform rendezvous with the explicit group's name
        symm_mem.rendezvous(msg, group_name)
        
        # Copy input data
        msg.copy_(tensor_copy)
        
        # Perform one-shot all-reduce with the explicit group's name
        torch.ops.symm_mem.one_shot_all_reduce(
            msg,
            "sum",
            group_name
        )
        
    except Exception as e:
        print(f"[Rank {rank}] Error: {e}")
        import traceback
        traceback.print_exc()
    
    # Clean up
    torch_dist.destroy_process_group()

if __name__ == "__main__":
    main()
</code></pre>
<p>This is giving me the error even though I am passing group names.</p>
<pre><code class="lang-auto">RuntimeError: CUDASymmetricMemory::rendezvous: `group_name` is neither specified during allocation nor passed to rendezvous().

</code></pre>
Need help debugging torch.compile issue

vivekvpandya: <p>I have build an experimental torch.compile inductor backend for a custom accelerator.<br>
Its mostly based on cpu codegen only but I have done some monkey patching which will be removed later.<br>
It works fine for tests but if I try to use it with cpu inductor backend in same test I see following error:</p>
<pre><code class="lang-auto">    args = (CppCSEVariable(name: tmp0, bounds: VR[-oo, oo], is_vec: False, dtype: torch.float32, dependent_itervars: {x0}),)
kwargs = {}

    def inner(*args, **kwargs):
        bounds = CSEProxy._bound_variable(name, *args, **kwargs)

&gt;       value = getattr(parent_handler, name)(*args, **kwargs)  # type: ignore[has-type]
E       torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
E       TypeError: CppOverrides.sigmoid() takes 1 positional argument but 2 were given
E
E       Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
E
E
E       You can suppress this exception and fall back to eager by setting:
E           import torch._dynamo
E           torch._dynamo.config.suppress_errors = True
</code></pre>
<p>Could someone please guide me on how to debug this further?</p>
<p>Here is my test for reference:</p>
<pre><code class="lang-auto">@pytest.mark.parametrize("dtype", [torch.float, torch.float16])
def test_sigmoid_mean_mul(dtype):
    def sigmoid(x):
        return torch.sigmoid(x)

    def sigmoid_with_mean_reduction(x):
        sig = sigmoid(x)
        mean_sig = torch.mean(sig)
        return mean_sig

    def pointwise_computation(x, mean_sig):
        return x * mean_sig

    def fn(x):
        mean_sig = sigmoid_with_mean_reduction(x)
        result = pointwise_computation(x, mean_sig)
        return result

    x = torch.randn((10, 10000), device="my", dtype=dtype)
    fn_compile = torch.compile(
        fn,
        backend="my-inductor",
        fullgraph=True,
    )
    out_compile = fn_compile(x)
    fn_compile_cpu = torch.compile(
        fn,
        backend="inductor",
        fullgraph=True,
    )
    out = fn_compile_cpu(x.cpu())
    assert torch.allclose(out.cpu(), out_compile.cpu(), rtol=0.01, atol=0.01)
</code></pre>
Why native float const is faster than cuda tensor const

valjean: <p>When doing large scale tensor operation on cuda including constant, I expect that pre-allocate constant on cuda can accelerate computation at least slightly. But the result is almost opposite: cuda tensor const is always slightly slow than native float constant. Below is my code:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import time

scale = 0.5
scale_cuda = torch.tensor(0.5, device="cuda")

def test_native_float(x_cuda):
    # use native float
    for _ in range(1000):
        x_cuda = x_cuda * scale 

def test_cuda_tensor(x_cuda):
    # use CUDA tensor
    for _ in range(1000):
        x_cuda = x_cuda * scale_cuda

x = torch.randn(10000, 10000, device="cuda")

# warm up
test_native_float(x)
test_cuda_tensor(x)
torch.cuda.synchronize()

# test time
start = time.time()
test_native_float(x)
torch.cuda.synchronize()
print("Native float time:", time.time() - start)

start = time.time()
test_cuda_tensor(x)
torch.cuda.synchronize()
print("CUDA Tensor time: ", time.time() - start)
</code></pre>
<p>Native float time: 1.0922577381134033<br>
CUDA Tensor time:  1.1039364337921143</p>
<p>OS: windows 10<br>
GPU: 3080TI<br>
cuda version: 12.6</p>
To integrate postgresql to project created with libtorch

Ahmet: <p>I created a project  using  libtorch and CMake. Here is the CMakeLists.txt ```  cmake_minimum_required(VERSION 3.18 FATAL_ERROR)<br>
project(torch2)</p>
<p>find_package(Torch REQUIRED)<br>
set(CMAKE_CXX_FLAGS “${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}”)</p>
<h1><a name="p-468009-specify-the-path-where-libpqxx-is-built-1" class="anchor" href="#p-468009-specify-the-path-where-libpqxx-is-built-1"></a>Specify the path where libpqxx is built</h1>
<p>set(LIBPQXX_DIR “/usr/local/cmake1/libpqxx/build/src”)</p>
<h1><a name="p-468009-create-an-alias-for-linking-2" class="anchor" href="#p-468009-create-an-alias-for-linking-2"></a>Create an alias for linking</h1>
<p>add_library(pqxx STATIC IMPORTED)<br>
set_target_properties(pqxx PROPERTIES IMPORTED_LOCATION “${LIBPQXX_DIR}/libpqxx.a”)</p>
<h1><a name="p-468009-also-link-against-libpq-postgresql-client-library-3" class="anchor" href="#p-468009-also-link-against-libpq-postgresql-client-library-3"></a>Also link against libpq (PostgreSQL client library)</h1>
<p>find_library(PQ_LIB pq)</p>
<p>add_executable(torch2 main.cpp CSVToTensorDataset.cpp)</p>
<h1><a name="p-468009-link-torch-libpqxx-and-libpq-4" class="anchor" href="#p-468009-link-torch-libpqxx-and-libpq-4"></a>Link Torch, libpqxx, and libpq</h1>
<p>target_link_libraries(torch2 “${TORCH_LIBRARIES}” pqxx ${PQ_LIB})</p>
<p>set_property(TARGET torch2 PROPERTY CXX_STANDARD 17)</p>
<pre><code class="lang-auto">
Kind regards</code></pre>
Why does not my code cause the error 'element 0 of tensors does not require grad and does not have a grad_fn'

liao_zhang: <p>Given the code fragment below, I freeze the weights of the neural network using <code>model.fc1.requires_grad_(False)</code> and <code>model.fc2.requires_grad_(False)</code>.<br>
If I use <code>loss = criterion(outputs, y_batch)</code> to calculate the loss, the training is fine. Why does it not cause the error <code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</code>?<br>
I think it should cause the error since all layers of the neural network have been frozen.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset

# Define a simple neural network
class SimpleNet(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(SimpleNet, self).__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.fc2 = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        x = self.fc1(x)
        x = self.fc2(x)
        return x

input_size = 3
hidden_size = 3
output_size = 3
batch_size = 16
num_epochs = 1
learning_rate = 0.01
num_samples = 1000 
X_train = torch.randn(num_samples, input_size, requires_grad = True)  
y_train = torch.ones((num_samples, output_size))
dataset = TensorDataset(X_train, y_train)
dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=True)

model = SimpleNet(input_size, hidden_size, output_size)


criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=learning_rate)

for epoch in range(num_epochs):
    total_loss = 0
    for batch_idx, (X_batch, y_batch) in enumerate(dataloader):
        optimizer.zero_grad()
        outputs = model(X_batch)
        loss = criterion(outputs, y_batch)
        print(loss._grad_fn)
        loss.backward() 
        optimizer.step() 
                
        if (batch_idx + 1) % 10 == 0:
            print(f'Batch [{batch_idx + 1}/{len(dataloader)}], Loss: {loss.item():.4f}')
            model.fc1.requires_grad_(False)
            model.fc2.requires_grad_(False)
                        
print("Training complete!")
</code></pre>

ptrblck: <p>Your input still requires gradients, so <code>X_train.grad</code> will receive valid gradients.<br>
Use <code>X_train = torch.randn(num_samples, input_size)</code> and it will fail with the expected error message.</p>
RuntimeError: Unsupported value kind: Tensor

ettu: <p>Hi, I’ve been trying to convert a Python model to Torch Script for a while now, and have been debugging this error: “RuntimeError: Unsupported value kind: Tensor”. This forum post (<a href="https://discuss.pytorch.org/t/runtime-error-unsupported-value-kind-tensor/190276/3" class="inline-onebox">Runtime error - Unsupported value kind: Tensor - #3 by nivtorch</a>) covers the same issue, but the answer is pretty vague, saying that somewhere for them they needed to add another argument.</p>
<p>I followed one user’s suggestion there to print from the jit’s source file, using “print(‘—\n’, concrete_type, method_stubs, property_stubs)”, and the issue seems to maybe occur during the model’s forward() method. Whenever I try the common suggestions to use the “Optional” type keyword for my parameters of the method, a bunch of new errors pop up about type mismatches.</p>
<p>Another forum post suggested that there can be a version mismatch between Pytorch and libtorch, but my versions match up between the two.</p>
<p>Has anyone had experience working past this error? Thanks</p>
Nightly pytorch download with xpu support, '2.8.0.dev20250325+xpu', reports "AssertionError: Torch not compiled with XPU enabled"

KFrank: <p>Hi All!</p>
<p>I used:</p>
<pre><code class="lang-plaintext">pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/xpu
</code></pre>
<p>into a freshly-created python=3.13 conda environment.  No installation errors were reported<br>
and the pytorch version appears to be as expected, but xpu support appears to be absent:</p>
<pre><code class="lang-plaintext">(2_8_0_xpu_nightly) xxxxx@xxxxxxxxxxx:~/Documents/devl$ python
Python 3.13.2 | packaged by Anaconda, Inc. | (main, Feb  6 2025, 18:56:02) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.8.0.dev20250325+xpu'
&gt;&gt;&gt; tt = torch.randn (10, device = 'xpu')
Traceback (most recent call last):
  File "&lt;python-input-2&gt;", line 1, in &lt;module&gt;
    tt = torch.randn (10, device = 'xpu')
  File "/home/xxxxx/miniconda3/envs/2_8_0_xpu_nightly/lib/python3.13/site-packages/torch/xpu/__init__.py", line 118, in _lazy_init
    raise AssertionError("Torch not compiled with XPU enabled")
AssertionError: Torch not compiled with XPU enabled
</code></pre>
<p>(I get the same result in a python=3.12 environment, while the release version, ‘2.6.0+xpu’<br>
works,)</p>
<p>Is there any way I can fix this, or should I just wait until, say, a 2.7 xpu-enabled version is<br>
released?</p>
<p>Thanks.</p>
<p>K. Frank</p>
Does it make sense to mix the labels in each batch?

laro: <p>When using binary classification model:</p>
<ul>
<li>When training a deep model, at each training step, the model receives a batch (i.e batch of size 32 samples).</li>
<li>Let’s assume that in each training batch there are always 16 samples with label ‘0’ and 16 samples with label ‘1’.</li>
</ul>
<ol>
<li>Does it matter how those samples are arranged in the batch?</li>
<li>Is there a difference if the 16 labels with “0” appear first and then with “1” <strong>vs</strong> mix of all 32 samples?</li>
</ol>

ptrblck: <p>No and no. The sample order inside the batch won’t matter unless you treat it as a sequence inside your model (and thus as a sequence dimension, not a batch dimension). The overall shuffling of the dataset still matters, i.e. the batches in each epoch should be created via random sampling.</p>
Unexpected Behavior of torch.median()

nishantbadhautiya: <p>I noticed an issue while using <code>torch.median()</code> on a PyTorch tensor. It’s not returning the correct median of all elements. Here’s an example:</p>
<pre><code class="lang-auto">import torch

e = torch.tensor([[5., 8., 0.],
                  [8., 9., 6.]])

print(torch.median(e))  # Output: tensor(6.)

</code></pre>
<p>However, when I sort and manually compute the median:</p>
<pre><code class="lang-auto">sorted_e = torch.sort(e.flatten()).values
median_value = (sorted_e[len(sorted_e) // 2 - 1] + sorted_e[len(sorted_e) // 2]) / 2
print(median_value)  # Output: 7.0

</code></pre>
<p>The expected median should be <strong>7.0</strong>, but <code>torch.median(e)</code> gives <strong>6.0</strong>.</p>

ptrblck: <p>From the <a href="https://pytorch.org/docs/stable/generated/torch.median.html">docs</a>:</p>
<blockquote>
<p>The median is not unique for input tensors with an even number of elements. In this case the lower of the two medians is returned. To compute the mean of both medians, use torch.quantile() with q=0.5 instead.</p>
</blockquote>
How to convert adjacency matrix to edge_index format?

Haziq: <p>I have an adjaceny tensor</p>
<pre><code class="lang-auto">tensor([[0., 1., 1., 1.],
        [1., 0., 1., 1.],
        [1., 1., 0., 1.],
        [1., 1., 1., 0.]])
</code></pre>
<p>How can I convert it to the <code>edge_index</code> form used in pytorch geometric? <a href="https://pytorch-geometric.readthedocs.io/en/latest/notes/introduction.html" class="inline-onebox" rel="noopener nofollow ugc">Introduction by Example — pytorch_geometric 2.0.4 documentation</a> The actual adjacency matrix is far larger so it’s not something I want to create manually.</p>

Andre_Amaral_IST: <p>I found a way to do it:</p>
<aside class="quote" data-post="1" data-topic="148343">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andre_amaral_ist/48/48306_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/adjacency-matrix-to-edge-index-solution/148343">Adjacency Matrix to Edge Index (Solution)</a> <a class="badge-category__wrapper " href="/c/data/37"><span data-category-id="37" data-drop-close="true" class="badge-category " title="Topics related to DataLoader, Dataset, torch.utils.data, pytorch/data, and TorchArrow."><span class="badge-category__name">data</span></span></a>
  </div>
  <blockquote>
    Hey, 
For those who have this question, here you have a way to solve it! 
adj_t = torch.tensor([[0,1,0,0],[1,0,0,0], [1,0,0,1], [0,0,1,0]])
edge_index = adj_t.nonzero().t().contiguous()

&gt;&gt;&gt; tensor([[0, 1, 2, 2, 3],
            [1, 0, 0, 3, 2]])

Thanks for your attention, 
André Amaral
  </blockquote>
</aside>

<p>Hope it helps!</p>

Han_Xiao: <p>Check out <a href="https://pytorch-geometric.readthedocs.io/en/2.5.2/modules/utils.html#torch_geometric.utils.to_edge_index" rel="noopener nofollow ugc">to_edge_index</a></p>
Modifying Pytorch + Increasing GPU Memory

sfaz: <p>Hello Everyone<br>
I am working on a simulation using M3GNet, a machine learning potential that employs deep learning techniques. It specifically utilizes PyTorch as the framework for building and training its models. To accelerate the simulation, I am aiming to leverage GPU resources. I have already installed the necessary GPU drivers on my system.<br>
the information of CUDA binary is that<br>
nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2023 NVIDIA Corporation<br>
Built on Tue_Aug_15_22:02:13_PDT_2023<br>
Cuda compilation tools, release 12.2, V12.2.140<br>
Build cuda_12.2.r12.2/compiler.33191640_0</p>
<p>I follow these pathes for installing lammps conda-forge, m3gnet, matgl, dgl, and pytorch are:<br>
conda create -n lammps_m3gnet python=3.11<br>
conda activate lammps_m3gnet<br>
conda install -c conda-forge lammps<br>
pip install m3gnet<br>
pip install matgl</p>
<p>Install cudatoolkit v11.8.0:<br>
conda install -c conda-forge cudatoolkit<br>
Install cudnn v8.9.7:<br>
conda install -c conda-forge cudnn<br>
Install Pytorch:<br>
pip install torch==2.2.0 torchvision torchaudio --index-url <a href="https://download.pytorch.org/whl/cu118" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu118</a><br>
Install dgl:<br>
pip install dgl -f <a href="https://data.dgl.ai/wheels/cu118/repo.html" rel="noopener nofollow ugc">https://data.dgl.ai/wheels/cu118/repo.html</a><br>
pip install dglgo -f <a href="https://data.dgl.ai/wheels-test/repo.html" rel="noopener nofollow ugc">https://data.dgl.ai/wheels-test/repo.html</a></p>
<p>after installing all requirments, when I run a job with the binary of lmp, I got this one</p>
<pre><code>PID    USER DEV     TYPE  GPU        GPU MEM    CPU  HOST MEM Command
</code></pre>
<p>3923427 ss   0  Compute   0%    386MiB   2%    100%   6742MiB lmp -in lmp-m<br>
3928976 ss   0  Compute   0%    386MiB   2%    100%   4725MiB lmp -in lmp-m</p>
<p>As you see, just 2% of mem of GPU is implemented for runnig each job. the question is that how to increase the %mem of GPU for each job? Is it neccessary to modify Pytorch for controlling the %mem of GPU?</p>
<p>Thanks in advanced</p>
YolovV7 inference with libtorch c++

MohammadShokouhiGol: <p>I am using libtorch to perform inference with a TorchScripted YOLOv7 model. The issue I’m facing is that during the second inference call (not the first one), the inference takes much longer (around 20 seconds). After that, the inference time drops to about 20 milliseconds.</p>
<p>Upon profiling, I discovered that during the 20-second delay, there are numerous <code>cuModuleDataLoad</code> API calls. This suggests that Torch is trying to load and compile the kernels.</p>
<p>When I set <code>torch::jit::setGraphExecutorOptimize(false)</code>, the warm-up time reduced to 400 milliseconds, but the remaining inference time increased to 40 milliseconds. (Inference times of less than 30 milliseconds are crucial in my case.)</p>
<p>I understand that enabling <code>GraphExecutorOptimize</code> fuses certain kernels and creates input-size-specific kernels to speed up inference. This is why the inference time is doubled when the optimization is disabled.</p>
<p>My question is: do you know how to store the fused and optimized kernels to avoid excessive warm-up time?</p>
<p>I am aware that I can add fake inferences with random inputs to warm up the model. However, my goal is to perform inference with my actual inputs as quickly as possible, without waiting for a warm-up with fake data. In other words, I cannot afford to wait for the warm-up process to complete, as it results in the same delay as waiting for the first or second real frame.</p>

singh20anurag: <pre><code class="lang-auto">My question is: do you know how to store the fused and optimized kernels to avoid excessive warm-up time?
</code></pre>
<p>If your input shape is static you could consider <a href="https://pytorch.org/blog/accelerating-pytorch-with-cuda-graphs/" rel="noopener nofollow ugc">CUDA graphs</a> for AOT compilation.</p>
Building v2.5.1 from source

NQHuy1905: <p>Hi i am building torch v2.5.1</p>
<p>When I git check out and check with git branch it return v2.5.1<br>
But when i run setup.py it print out that “Building wheel torch-2.5.0a0+gita8d6afb”<br>
Why is that and how can i build torch v2.5.1 with source repo</p>
TORCH_WARN("Both fused kernels do not support non-zero dropout.");

githubsgi: <p>Couple of questions.</p>
<ol>
<li>
<p>What kernels are referred to in  the following line<br>
TORCH_WARN(“Both fused kernels do not support non-zero dropout.”);<br>
from<br>
<a href="https://github.com/pytorch/pytorch/blob/51f91e3428be9201e9c1199ec704bc132f979955/aten/src/ATen/native/transformers/sdp_utils_cpp.h#L240" class="inline-onebox" rel="noopener nofollow ugc">pytorch/aten/src/ATen/native/transformers/sdp_utils_cpp.h at 51f91e3428be9201e9c1199ec704bc132f979955 · pytorch/pytorch · GitHub</a></p>
</li>
<li>
<p>Is there a way to see what attention kernel torch picked up for running ?</p>
</li>
</ol>
How to check grads in each step of model?

elinliu0823: <p>Hi there:<br>
I’ve implement a Pytorch version of <a href="https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI" rel="noopener nofollow ugc">Retrieval-based-Voice-Conversion(RVC for short)</a> at <a href="https://github.com/ElinLiu0/RVCTorch/blob/master/POC_Torch.ipynb" rel="noopener nofollow ugc">here</a>.<br>
The question is,when i wanna export my implementation pipeline into ONNX using below code:</p>
<pre data-code-wrap="python"><code class="lang-python">with torch.inference_mode(), torch.cuda.amp.autocast(enabled=False):
 torch.onnx.export(
     pipeline, 
     (audio.cuda(),),
     "pipeline.onnx",
     input_names=["input"],
     output_names=["output"],
     opset_version=14
 )
</code></pre>
<p>It rasing below error:</p>
<pre data-code-wrap="python"><code class="lang-python">RuntimeError: Cannot insert a Tensor that requires grad as a constant. Consider making it a parameter or input, or detaching the gradient
Tensor:
 0.6670
[ torch.cuda.HalfTensor{1} ]
</code></pre>
<p>Typically rasing with an <code>nn.BatchNorm2d</code> cell called at <a href="https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/blob/main/infer/lib/rmvpe.py" rel="noopener nofollow ugc">rmvpe.py</a> at line 244.</p>
<p>So how could i fix this error,since this implementation finally will deploy on C# or model serving platform like NVIDIA Triton.</p>

MyCenturaHealth: <p>The error occurs because PyTorch’s torch.onnx.export() method attempts to export a tensor that requires gradients, which is not allowed in ONNX. To fix this, detach the tensor from the computation graph using audio.detach() before passing it to torch.onnx.export(). This prevents the tensor from requiring gradients and allows the export to work smoothly. Additionally, ensure your model is in evaluation mode (model.eval()) before exporting, as this affects the behavior of layers like BatchNorm2d<a href="https://animeslayer.me/" rel="noopener nofollow ugc">.</a></p>
Machine Learning Interview in Big Companies

Tornike: <p>Hi, I don’t know if its the right place to ask that question. i researched a little bit about that but could not find a proper answer.</p>
<p>Do Big companies ask about Math formulas of DL/ML in the ML interviews? I’m not talking about Data science interviews.</p>
<p>Or do they ask to derive some formula or write formulas of some loss/optimization functions? Or write Backpropagation Mathematically?</p>
<p>Or do they need you to know just geometric Undestanding of different Linear Algebra/Calculus Topics? How does it work, why is it working like this and how would u use that in real-world scenarios?</p>

Tornike: <p>Or Do they ask to compute some equation with formulas.</p>

josephazizi: <p>Hi <a class="mention" href="/u/tornike">@Tornike</a>,</p>
<p>Yes, big companies do ask about math in ML interviews, especially for deep learning roles. You might need to derive key formulas, explain loss functions, and understand backpropagation mathematically.</p>
<p>While memorizing formulas isn’t the focus, strong intuition in linear algebra, calculus, and optimization is crucial. They often test your ability to apply mathematical concepts to real-world ML problems.</p>
<p>For a detailed guide on ML interview questions, check out this resource: <a href="https://www.igmguru.com/blog/machine-learning-interview-questions" rel="noopener nofollow ugc">Machine Learning Interview Questions</a>.</p>
<p>Thanks</p>
What does dataloader.dataset and dataloader.dataset.targets do?

make1234: <p>Hello,</p>
<p>I am experimenting with pytorch and did not find a doc description of what the mentioned attributes do/constitute for. I printed the output so I know the first returns an object that encompasses all the data and the second somehow returns all labels. Does anyone know what <em>exactly</em> they do? Thanks!</p>

ptrblck: <p><code>loader.dataset</code> returns the <code>Dataset</code> object you passed to the <code>Dataloader</code>. <code>loader.dataset.target</code> then accesses the internal <code>target</code> attribute of your dataset. Check how your dataset is defined and how <code>target</code> is created to understand its content as this attribute won’t be available in all datasets.</p>
Request for assistance: Building PyTorch 2.5.1 (libtorch + main PyTorch) with CUDA 12.6.3 and cuDNN 9.5.1

kksspoi: <p>Hello PyTorch Team and Community,</p>
<p>I would like to request guidance on the correct method to build PyTorch 2.5.1 (both libtorch and the main PyTorch) from source, ensuring compatibility with CUDA 12.6.3 and cuDNN 9.5.1.</p>
<p>Machine Specifications</p>
<pre><code>•	OS: Ubuntu 22.04.5 LTS
•	CPU: Intel i7-9700K
•	Memory: 64 GB
•	GPU: RTX 4060 Ti (16 GB VRAM)
•	CUDA/cuDNN: CUDA 12.6.3 and cuDNN 9.5.1, both installed system-wide (not through Miniconda or other package managers).
•	Python Environment: Miniconda3 with Python 3.11.10 (virtual environment).
</code></pre>
<p>My goal is to build PyTorch from source for use in image generation AI workflows (e.g., ComfyUI).</p>
<p>Context and Current Process</p>
<p>I want to confirm the correct procedure for building PyTorch (including libtorch) with cmake from source, targeting CUDA 12.6.3 and cuDNN 9.5.1, and subsequently packaging the build as a .whl file for installation in my virtual environment.</p>
<p>Previously, I followed this process:<br>
1.	Set the environment variable export BUILD_LIBTORCH_WHL=1 and built libtorch (TORCH_NO_PYTHON) with cmake, configuring it for CUDA 12.6.2 and cuDNN 9.5.1.<br>
2.	Used the following command to build and create a wheel file:</p>
<p>python -m build --wheel -Ccmake.define.BUILD_CMAKE_FROM_SOURCE=ON</p>
<pre><code>3.	Deleted the build directory using rm -rf build and cleaned up with python setup.py clean.
4.	Then, I used this built version of libtorch as a dependency for building the main PyTorch. Before doing so, I set export BUILD_PYTHON_ONLY=1, configured CUDA 12.6.2 and cuDNN 9.5.1 with cmake again (using the same settings as the libtorch build), and attempted to build with the same command:
</code></pre>
<p>python -m build --wheel -Ccmake.define.BUILD_CMAKE_FROM_SOURCE=ON</p>
<p>However, this approach often results in errors during the main PyTorch build phase, such as torch_python not being found. To work around this, I have removed the BUILD_PYTHON_ONLY flag and rebuilt PyTorch using the following command instead:</p>
<p>python -m build</p>
<p>While this allows me to build a package compatible with CUDA 12.6.2 and cuDNN 9.5.1 for my virtual environment, I suspect this approach is incorrect, especially for integrating libtorch properly into the main PyTorch build.</p>
<p>Questions</p>
<pre><code>1.	What is the correct and complete method to build PyTorch 2.5.1, including both libtorch and the main PyTorch, from source, targeting CUDA 12.6.3 and cuDNN 9.5.1?
2.	Is it correct to assume that libtorch must be included as part of the main PyTorch build to consider the package complete?
3.	Are there additional dependencies, environment variables, or specific cmake options required for this process?
</code></pre>
<p>If further information is needed regarding my build environment, configuration, or errors encountered, please let me know, and I will provide additional details.</p>
<p>Thank you for your assistance!</p>

kksspoi: <p>I was able to build pytorch with CUDA12.8.1+cuDNN9.8.0. This problem was solved.</p>
Simple regression with integer output

torch_newbie: <p>Hi, I have been studying and doing some pytorch lately in our project and I came up with this problem. I am trying to create a model with number of users (integer) as output and with inputs of workplace area, day of the year and time of the day. As of now i am doing great with floats as output but I have no idea how to implement integer outputs. My model is written below.</p>
<pre><code class="lang-auto">import torch
from torch import nn


class TestModel(nn.Module):
    def __init__(
            self, input_size: int = 3,
            output_size: int = 1,
            hidden_size: int = 8,
            n_hidden_layers: int = 4
        ):
        super().__init__()

        module_list = [nn.Linear(input_size, hidden_size), nn.ReLU()]
        for _ in range(n_hidden_layers):
            module_list.extend([nn.Linear(hidden_size, hidden_size), nn.ReLU()])
        module_list.append(nn.Linear(hidden_size, output_size))

        self.sequential = nn.Sequential(*module_list)


    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        return self.sequential(x)
</code></pre>
<p>Any help on the model modification and how to train the model is much appreciated. Thank you.</p>

vdw: <p>I don’t think this easily works with integers. Why not just train “pretending” your integers are floats for training, and then for inferencing round the predicted float values?</p>

torch_newbie: <p>Thank you for your response. Actually, that’s what I did when I presented the my output to my team. SInce Im new to neural networks and pytorch, I thought there is a better way to do this because I thought my code was just a hack. Thank you very much. You directed my to the right direction.</p>
Can I (as a hack) use optimiser as a learning rate scheduler?

Prasad_Raghavendra: <p>I have a new optimiser that I have designed. But, the thing is that I need comparison and say, “hey, see these optimisers, loss, learning rates etc. and the one that I designed is better”. I have been unable to get any proper learning rate out of optimiser. But, I am seeing that the learning rate can be gotten out of learning rate scheduler though. So, can I use optimiser as a learning rate scheduler?</p>

Prasad_Raghavendra: <p>Also, if I happen to get some time, I am actually thinking about contributing to this nice library. If something like <code>LR_view</code> of optimiser is needed, I can probably work on it. I am also thinking about integrating my own optimiser that I have been building into PyTorch in some (and I really hope I do that). So, I may follow-up using GitHub if you think that is better.</p>