Question:
Hi guys,
I’m having a strange error out of a sudden (see below). I don’t know, what I changed in my system that caused this error (I changed almost nothing). Moreover, I can’t debug it at all. Because actually in doesn’t occur if I start the training in the PyCharm debug mode. Does somebody have an idea, how I could debug this? (I tried launch-blocking)
The full error output looks like this btw.:
Training:   0%|          | 1/10000 [09:51<1642:14:45, 591.27s/it]CUBLAS error: out of memory (3) in magma_ssyevd_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/ssyevd_gpu.cpp:226
CUBLAS error: not initialized (1) in magma_ssyevd_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/ssyevd_gpu.cpp:226
CUBLAS error: out of memory (3) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:137
CUBLAS error: not initialized (1) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:137
CUBLAS error: out of memory (3) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:138
CUBLAS error: not initialized (1) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:138
CUBLAS error: out of memory (3) in magma_ssyevd_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/ssyevd_gpu.cpp:226
CUBLAS error: not initialized (1) in magma_ssyevd_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/ssyevd_gpu.cpp:226
CUBLAS error: out of memory (3) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:137
CUBLAS error: not initialized (1) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:137
CUBLAS error: out of memory (3) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:138
CUBLAS error: not initialized (1) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:138
CUBLAS error: out of memory (3) in magma_ssyevd_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/ssyevd_gpu.cpp:226
CUBLAS error: not initialized (1) in magma_ssyevd_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/ssyevd_gpu.cpp:226
CUBLAS error: out of memory (3) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:137
CUBLAS error: not initialized (1) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:137
CUBLAS error: out of memory (3) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:138
CUBLAS error: not initialized (1) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:138
CUBLAS error: out of memory (3) in magma_ssyevd_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/ssyevd_gpu.cpp:226
CUBLAS error: not initialized (1) in magma_ssyevd_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/ssyevd_gpu.cpp:226
CUBLAS error: out of memory (3) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:137
CUBLAS error: not initialized (1) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:137
CUBLAS error: out of memory (3) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:138
CUBLAS error: not initialized (1) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:138
CUBLAS error: out of memory (3) in magma_ssyevd_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/ssyevd_gpu.cpp:226
CUBLAS error: not initialized (1) in magma_ssyevd_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/ssyevd_gpu.cpp:226
CUBLAS error: out of memory (3) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:137
CUBLAS error: not initialized (1) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:137
CUBLAS error: out of memory (3) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:138
CUBLAS error: not initialized (1) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:138
CUBLAS error: out of memory (3) in magma_ssyevd_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/ssyevd_gpu.cpp:226
CUBLAS error: not initialized (1) in magma_ssyevd_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/ssyevd_gpu.cpp:226
CUBLAS error: out of memory (3) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:137
CUBLAS error: not initialized (1) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:137
CUBLAS error: out of memory (3) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:138
CUBLAS error: not initialized (1) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:138
CUBLAS error: out of memory (3) in magma_ssyevd_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/ssyevd_gpu.cpp:226
CUBLAS error: not initialized (1) in magma_ssyevd_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/ssyevd_gpu.cpp:226
CUBLAS error: out of memory (3) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:137
CUBLAS error: not initialized (1) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:137
CUBLAS error: out of memory (3) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:138
CUBLAS error: not initialized (1) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:138
CUBLAS error: out of memory (3) in magma_ssyevd_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/ssyevd_gpu.cpp:226
CUBLAS error: not initialized (1) in magma_ssyevd_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/ssyevd_gpu.cpp:226
CUBLAS error: out of memory (3) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:137
CUBLAS error: not initialized (1) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:137
CUBLAS error: out of memory (3) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:138
CUBLAS error: not initialized (1) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:138
CUBLAS error: out of memory (3) in magma_ssyevd_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/ssyevd_gpu.cpp:226
CUBLAS error: not initialized (1) in magma_ssyevd_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/ssyevd_gpu.cpp:226
CUBLAS error: out of memory (3) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:137
CUBLAS error: not initialized (1) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:137
CUBLAS error: out of memory (3) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:138
CUBLAS error: not initialized (1) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:138
CUBLAS error: out of memory (3) in magma_ssyevd_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/ssyevd_gpu.cpp:226
CUBLAS error: not initialized (1) in magma_ssyevd_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/ssyevd_gpu.cpp:226
CUBLAS error: out of memory (3) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:137
CUBLAS error: not initialized (1) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:137
CUBLAS error: out of memory (3) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:138
CUBLAS error: not initialized (1) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:138
CUBLAS error: out of memory (3) in magma_ssyevd_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/ssyevd_gpu.cpp:226
CUBLAS error: not initialized (1) in magma_ssyevd_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/ssyevd_gpu.cpp:226
CUBLAS error: out of memory (3) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:137
CUBLAS error: not initialized (1) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:137
CUBLAS error: out of memory (3) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:138
CUBLAS error: not initialized (1) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:138
CUBLAS error: out of memory (3) in magma_ssyevd_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/ssyevd_gpu.cpp:226
CUBLAS error: not initialized (1) in magma_ssyevd_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/ssyevd_gpu.cpp:226
CUBLAS error: out of memory (3) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:137
CUBLAS error: not initialized (1) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:137
CUBLAS error: out of memory (3) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:138
CUBLAS error: not initialized (1) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:138
python: /opt/conda/conda-bld/magma-cuda102_1583546904148/work/interface_cuda/interface.cpp:810: void magma_queue_create_internal(magma_device_t, magma_queue**, const char*, const char*, int): Assertion `queue->dCarray__ != __null’ failed.

Context:
ok, by manually inserting tons of logging messages all over my code, I finally found the place where the error occurs. It is the following line
    self.mvn = MultivariateNormal(loc=torch.zeros(d, device=data.device), covariance_matrix=cov)

When this line is called with d=1, cov=tensor([[0.1121]]) and data.device = “cuda:0” I receive the error message:
CUBLAS error: out of memory (3) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:138
CUBLAS error: not initialized (1) in magma_spotrf_LL_expert_gpu at /opt/conda/conda-bld/magma-cuda102_1583546904148/work/src/spotrf_gpu.cpp:138
The training loop continues then until it comes again to this point where it produces the same error again. After a few iteration the script crashes with the error:
python: /opt/conda/conda-bld/magma-cuda102_1583546904148/work/interface_cuda/interface.cpp:810: void magma_queue_create_internal(magma_device_t, magma_queue**, const char*, const char*, int): Assertion `queue->dCarray__ != __null’ failed.
Aborted (core dumped)
I actually can’t imagine what could be so problematic about a 1-dim gaussian variable. Does anybody have a clue?


Question:
I encountered this problem first.
>>> from setuptools import setup, Extension
>>> from torch.utils import cpp_extension
INFO: Could not find files for the given pattern(s).

And when I’m trying to use code here 24 to compile a simple extension, I encountered this issue, which I suspect is related to the problem above. But I’m not sure what is it exactly.
UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specified

I googled this issue but didn’t find a satisfactory answer.

Context: Seems related to this issue 334. And apparently the solution was to upgrade python 305.


torch::jit::script::Module hang on forever on loading or inference pass

Question:
Hi everyone, I’m not sure if this is the right place to post this issue or if it’s more CUDA-related.
I have a super simple script which should load a torch::jit::script::Module  and run some forward passes on a dummy tensor.
However, depending on the model I use, it gets stuck either when it tries to load the model (model1) or when it runs the first forward pass (model2.pt), and I have to manually kill the process.
I’m working on a Windows 10 laptop with a NVIDIA GeForce RTX 3080 and:

CUDA Toolkit 10.2.
LibTorch 1.10.1 for CUDA 10.2 (both Release and Debug).
Visual Studio 2019

The output of nvidia-smi is:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 466.81       Driver Version: 466.81       CUDA Version: 11.3     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ... WDDM  | 00000000:01:00.0 Off |                  N/A |
| N/A    0C    P5    22W /  N/A |    183MiB / 16384MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

I built a VS project with this template. The code is the following:
#include <iostream>

#include <ATen/Context.h>
#include <torch/torch.h>
#include <torch/script.h>

int main() {
	const torch::Device device = torch::Device(torch::kCUDA, 0);
	torch::jit::script::Module model;
	std::string MODEL_PATH = "path/to/model1.pt";
        // std::string MODEL_PATH = "path/to/model2.pt"
	// Load Model
        std::cout << "Trying to load model..." << std::endl;
	try {
		model = torch::jit::load(MODEL_PATH, device);
		model.eval();
		std::cout << "AI model loaded successfully." << std::endl;
	}
	catch (const c10::Error& e) {
		std::cerr << e.what() << std::endl;

	}

	std::cout << "Warming up model..." << std::endl;

	auto dummy = torch::zeros({ 1, 3, 512, 512}).to(device);
	torch::Tensor output;
	std::vector<torch::jit::IValue> inputs;

	inputs.clear();
	inputs.emplace_back(dummy);

	std::cout << "\tIteration: ";

	for (int i = 0; i < 20; i++) {
		std::cout << i << ", ";
		output = model.forward(inputs).toTensor();
		torch::cuda::synchronize();
	}
}

Output for model1

Trying to load model...

Output for model2

Trying to load model...
AI model loaded successfully.
Warming up model...
        Iteration: 0,

I can’t figure out what could be the problem or what could I do to further investigate it.

Context:
Follow-up: On the RTX30 Series you MUST use at least CUDA 11.1 since it’s the first CUDA version which offers support for NVIDIA Ampere architecture (source: CUDA 11.1 Release Notes chapter 1.2).
After installing CUDA Toolkit 11.3 and LibTorch for CUDA 11.3 the model started working properly.


Gdb debugging access index({}) or size()

Question:
Hi,
I was playing with libtorch. I’ve configured and built with cmake  a really simple example.
cppFrontEnd.cpp
#include <torch/torch.h>
#include <iostream>

int main() {
  torch::Tensor tensor1 = torch::ones(5);
  std::cout << tensor1 << std::endl;
  tensor1.index_put_({2},5);
  std::cout << tensor1 << std::endl;
  std::cout<< tensor1.index({2});
  // std::cout<<tensor1.size(0)<<std::endl;

  return 0;
}

CMakeLists.txt
cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(cppFrontEnd)
set(CMAKE_PREFIX_PATH "/path/to/cppFrontEnd/libtorch")
set(CMAKE_BUILD_TYPE "Debug")
find_package(Torch REQUIRED)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS} -O0")
set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} ${TORCH_CXX_FLAGS_DEBUG} -O0")
add_executable(cppFrontEnd cppFrontEnd.cpp)
target_link_libraries(cppFrontEnd "${TORCH_LIBRARIES}")
set_property(TARGET cppFrontEnd PROPERTY CXX_STANDARD 14)

Then with gdb when I tried to access one element of the tensor
with
p tensor1.index({1})
I got
Cannot evaluate function -- may be inlined
and with
p tensor1.size(0)
Cannot resolve method at::Tensor::size to any overloaded instance

Otherwise
p tensor.sizes()[0] works if I uncomment the line std::cout<<tensor1.size(0)<<std::endl;
I assume the inline problem is generated by the compiler as if you don’t use or explicitly instantiate some template method it is not generated in resulting binary. then gdb cannot see it and that is why sizes()[] start working when I call tensor1.size() in the source code, am I right?
Is there a way to get a complete debug of the methods and attributes of the objects define in libtorch? Sorry if this is a basic question or if this is not the place to do it.
Thank you

Context:
You could try to build your application with debug symbols via cmake --build . --config Debug.
I haven’t checked this approach and would guess that you would also need a debug libtorch build.


Question:
What is the right way to cross-compile libtorch for aarch64 target on an amd64 Linux host?
I’m currently using tools/build_libtorch.py, however, when I try to use this library to build my C++ code using, cmake -DCMAKE_PREFIX_PATH=/path/to/build_libtorch .., I get errors regarding missing header files and libcaffe2_gpu.so.
Is there a script or a tutorial that would help organize the libtorch library similar to the structure released on the downloads page?
libtorch/
  bin/
  include/
  lib/
  share/

Context:
You might try to follow the route android takes, providing a CMAKE_TOOLCHAIN_FILE and you might need a host protobuf, see scripts/build_android.sh for inspiration.
I will admit that I did not try but build natively on aarch64 for the time being…
Best regards
Thomas


Question:
Hello!
I’m creating PyTorch C++ extensions and building them with bazel.
The documentation (source 40) explains how to build your extensions with either setuptools or JIT. However, Bazel likes to take the building into its own hands.
I’d like to get some advise on how to proceed. I currently have two working solutions:

Hacky solution
Per extension, create a bazel genrule, that just invokes a python setup.py build, and set the resulting .so file as the output artifact. This can then be loaded in the code. This leverages all the nice abstractions and build arguments that are set through the torch utilities (torch.utils.cpp_extension.$)

Proper solution
Create the library through bazel’s cc_library. This is nice, but everything (arguments, flags, includes, directories, etc) need to be set manually.

Is anyone using pytorch extensions with bazel already, or does anyone have any general advice here.

Context:
Just to help out some other people, here is the gist of it. The solution does prerequires having setup for bazel the (1) python headers and (2) pip requirements and (3) cuda.
Create a .bzl file containing something like
load("@local_config_cuda//cuda:build_defs.bzl", "if_cuda")
load("@local_config_cuda//cuda:build_defs.bzl", "cuda_default_copts")

load("@pip_deps//:requirements.bzl", "requirement")

def pytorch_cpp_extension(name, srcs=[], gpu_srcs=[], deps=[], copts=[], defines=[],
                          binary=True, linkopts=[]):
    """Create a pytorch cpp extension as a cpp and importable python library.

    All options defined below should stay close to the official torch cpp extension options as
    defined in https://github.com/pytorch/pytorch/blob/master/torch/utils/cpp_extension.py.
    """
    name_so = name + ".so"
    torch_deps = [
        requirement("torch", target = "cpp"),
    ]

    cuda_deps = [
        "@local_config_cuda//cuda:cudart_static",
        "@local_config_cuda//cuda:cuda_headers",
    ]

    copts = copts +[
        "-fPIC",
        "-D_GLIBCXX_USE_CXX11_ABI=0",
        "-DTORCH_API_INCLUDE_EXTENSION_H",
        "-fno-strict-aliasing",
        "-fopenmp",
        "-fstack-protector-strong",
        "-fwrapv",
        "-O2",
        "-std=c++14",
        "-DTORCH_EXTENSION_NAME=" + name
    ]

    if gpu_srcs:
        native.cc_library(
            name = name_so + "_gpu",
            srcs = gpu_srcs,
            deps = deps + torch_deps + if_cuda(cuda_deps),
            copts = copts + cuda_default_copts(),
            defines = defines,
            linkopts = linkopts,
        )
        cuda_deps.extend([":" + name_so + "_gpu"])

    if binary:
        native.cc_binary(
            name = name_so,
            srcs = srcs,
            deps = deps + torch_deps + if_cuda(cuda_deps),
            linkshared = 1,
            copts = copts,
            defines = defines,
            linkopts = linkopts,
        )
    else:
        native.cc_library(
            name = name_so,
            srcs = srcs,
            deps = deps + torch_deps + if_cuda(cuda_deps),
            copts = copts,
            defines = defines,
            linkopts = linkopts,
        )

    native.py_library(
        name = name,
        data = [":" + name_so],
    )

And be sure you can actually require torch as a cpp target library like like so;
genrule_directory(
    name = "include",
    srcs = [":extracted"],
    cmd = "mkdir -p $@ && cp -a $</torch/lib/include/. $@",
)

# NOTE: Make sure this yields the same includes as `include_paths()`:
# See https://github.com/pytorch/pytorch/blob/master/torch/utils/cpp_extension.py#L494

cc_library(
    name = "cpp",
    hdrs = [":include"],
    visibility = ["//visibility:public"],
    includes = [
        "include",
        "include/torch/csrc/api/include",
        "include/TH",
        "include/THC",
    ],
    deps = [
        "@//third_party/python:headers",
    ]
)


Pytorch Version 1.6.0 Broke LBFGS Optimizer

Question:
Hello!
I am building a Libtorch implementation of Neural Style Transfer based on GitHub - andrew98450/Style-Transfer: Style Transfer using libtorch C++ Frontend
This was written before Pytorch 1.6.0, and it works fine with Pytorch 1.4.0, 1.5.0, or 1.5.1. But if I try to use it with version 1.6.0 and above, the program throws an exception right after the lambda function returns the loss (and hence, right as the optimizer takes a step). At line 13 in WrapFunctionIntoFunctor.h: “c10::Error at memory location 0x00000029AECFCF10.”
For some reason, the error is also times at WrapFunctionIntoRuntimeFunctor.h memory error on line 18 and in lbfgs.cpp, even in (seemingly) the same conditions.
It works fine with the Adam optimizer in all versions.
Any hints would be appreciated. I couldn’t find any commits on the 1.6.0 release that seemed relevant, but it must be there.
I am on Windows and have tried this with both CUDA and CPU (I modified the code to use CPU). I am happy to provide any other potentially helpful information.

Context: Would you mind creating an issue on GitHub 1 with this description and a reproducible code snippet, if possible, please?


Question:
I want custom a cuda matrix multiplication using tensor cores in PyTorch. But it doesn’t work when compling the operator. The source code was refered to the sample code provided by NVIDIA which act normally on my machine.
The error message is mainly
FAILED: /home/hemeng/Desktop/extension-cpp-master/cuda/build/temp.linux-x86_64-3.9/matmul_cuda_kernel.o
/usr/local/cuda-11.3/bin/nvcc  -I/home/hemeng/.local/lib/python3.9/site-packages/torch/include -I/home/hemeng/.local/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/home/hemeng/.local/lib/python3.9/site-packages/torch/include/TH -I/home/hemeng/.local/lib/python3.9/site-packages/torch/include/THC -I/usr/local/cuda-11.3/include -I/usr/include/python3.9 -c -c /home/hemeng/Desktop/extension-cpp-master/cuda/matmul_cuda_kernel.cu -o /home/hemeng/Desktop/extension-cpp-master/cuda/build/temp.linux-x86_64-3.9/matmul_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=matmul_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++14
/home/hemeng/Desktop/extension-cpp-master/cuda/matmul_cuda_kernel.cu(127): error: incomplete type is not allowed

/home/hemeng/Desktop/extension-cpp-master/cuda/matmul_cuda_kernel.cu(128): error: incomplete type is not allowed

2 errors detected in the compilation of "/home/hemeng/Desktop/extension-cpp-master/cuda/matmul_cuda_kernel.cu".

and the kernel implementation is mainly (haven’t use scalar_t in order to locate the problem)
using namespace nvcuda;
// template <typename scalar_t>
__global__ void simple_wmma_gemm(float *a, float *b, float *c, int m_ld,
                                 int n_ld, int k_ld, int num_bh)
{
  // Leading dimensions. Packed with no transpositions.
  int lda = k_ld;
  int ldb = k_ld;
  int ldc = n_ld;

  // Tile using a 2D grid
  int warpM = (blockIdx.x * blockDim.x + threadIdx.x) / warpSize;
  int warpN = (blockIdx.y * blockDim.y + threadIdx.y);

  // Declare the fragments
  wmma::fragment<wmma::matrix_a, WMMA_M, WMMA_N, WMMA_K, float, wmma::row_major> a_frag;
  wmma::fragment<wmma::matrix_b, WMMA_M, WMMA_N, WMMA_K, float, wmma::col_major> b_frag;
  wmma::fragment<wmma::accumulator, WMMA_M, WMMA_N, WMMA_K, float> c_frag;

  wmma::fill_fragment(c_frag, 0.0f);

  // Loop over k
  for (int i = 0; i < k_ld; i += WMMA_K)
  {
    int aCol = i;
    int aRow = warpM * WMMA_M;
    int bCol = warpN * N;
    int bRow = i;

    // Bounds checking
    if (aRow < m_ld && aCol < k_ld && bRow < k_ld && bCol < n_ld)
    {
      // Load the inputs
      wmma::load_matrix_sync(a_frag, a + aCol + aRow * lda, lda);
      wmma::load_matrix_sync(b_frag, b + bRow + bCol * ldb, ldb);

      // Perform the matrix multiplication
      wmma::mma_sync(c_frag, a_frag, b_frag, c_frag);
    }
  }

  // Load in the current value of c, scale it by beta, and add this our result
  // scaled by alpha
  int cCol = warpN * WMMA_N;
  int cRow = warpM * WMMA_M;

  if (cRow < m_ld && cCol < n_ld)
  {
    // Store the output
    wmma::store_matrix_sync(c + cCol + cRow * ldc, c_frag, ldc,
                            wmma::mem_row_major);
  }
}

May be the problem is the compiler could not recognize the “wmma::fragment<wmma::matrix_a, WMMA_M, WMMA_N, WMMA_K, float, wmma::row_major>” type ?
The full code is here
#include <torch/extension.h>

#include <cuda.h>
#include <cuda_runtime.h>

#include <mma.h>

#include <vector>

#define TILE_DIM 64

#define WMMA_M 16
#define WMMA_N 16
#define WMMA_K 16

#define M 16
#define N 16
#define K 16

// namespace
// {
template <typename scalar_t>
__global__ void matmul_cuda_1d_kernel(
    torch::PackedTensorAccessor<scalar_t, 1, torch::RestrictPtrTraits, size_t> mat_a,
    torch::PackedTensorAccessor<scalar_t, 1, torch::RestrictPtrTraits, size_t> mat_b,
    torch::PackedTensorAccessor<scalar_t, 1, torch::RestrictPtrTraits, size_t> mat_c,
    int m)
{
  int index = blockIdx.x * blockDim.x + threadIdx.x;
  if (index < m)
  {
    mat_c[index] = mat_a[index] * mat_b[index];
  }
}

template <typename scalar_t>
__global__ void matmul_cuda_2d_kernel(
    torch::PackedTensorAccessor<scalar_t, 2, torch::RestrictPtrTraits, size_t> mat_a,
    torch::PackedTensorAccessor<scalar_t, 2, torch::RestrictPtrTraits, size_t> mat_b,
    torch::PackedTensorAccessor<scalar_t, 2, torch::RestrictPtrTraits, size_t> mat_c,
    int m, int n, int k)
{
  int row = blockIdx.y * blockDim.y + threadIdx.y;
  int col = blockIdx.x * blockDim.x + threadIdx.x;
  if (row < m && col < k)
  {
    float Pvalue = 0;
    for (int i = 0; i < n; i++)
    {
      Pvalue += mat_a[row][i] * mat_b[i][col];
    }
    mat_c[row][col] = Pvalue;
  }
}

template <typename scalar_t>
__global__ void matmul_cuda_4d_kernel(
    torch::PackedTensorAccessor<scalar_t, 4, torch::RestrictPtrTraits, size_t> mat_a,
    torch::PackedTensorAccessor<scalar_t, 4, torch::RestrictPtrTraits, size_t> mat_b,
    torch::PackedTensorAccessor<scalar_t, 4, torch::RestrictPtrTraits, size_t> mat_c,
    int m, int n, int k)
{
  int row = blockIdx.y * blockDim.y + threadIdx.y;
  int col = blockIdx.x * blockDim.x + threadIdx.x;
  for (size_t batch = 0; batch < mat_a.size(0); batch++)
  {
    for (size_t head = 0; head < mat_a.size(1); head++)
    {
      if (row < m && col < k)
      {
        float Pvalue = 0;
        for (int i = 0; i < n; i++)
        {
          Pvalue += mat_a[batch][head][row][i] * mat_b[batch][head][i][col];
        }
        mat_c[batch][head][row][col] = Pvalue;
      }
    }
  }
}

template <typename scalar_t>
__global__ void shared_matmul_cuda_4d_kernel(
    torch::PackedTensorAccessor<scalar_t, 4, torch::RestrictPtrTraits, size_t> mat_a,
    torch::PackedTensorAccessor<scalar_t, 4, torch::RestrictPtrTraits, size_t> mat_b,
    torch::PackedTensorAccessor<scalar_t, 4, torch::RestrictPtrTraits, size_t> mat_c,
    int m, int n, int k)
{
  int row = blockIdx.y * blockDim.y + threadIdx.y;
  int col = blockIdx.x * blockDim.x + threadIdx.x;
  __shared__ float aTile[TILE_DIM][TILE_DIM];
  __shared__ float bTile[TILE_DIM][TILE_DIM];
  for (size_t batch = 0; batch < mat_a.size(0); batch++)
  {
    for (size_t head = 0; head < mat_a.size(1); head++)
    {
      if (row < m && col < k)
      {
        aTile[threadIdx.y][threadIdx.x] = mat_a[batch][head][row][threadIdx.x];
        bTile[threadIdx.y][threadIdx.x] = mat_b[batch][head][threadIdx.y][col];
        __syncwarp();
        float Pvalue = 0;
        for (int i = 0; i < n; i++)
        {
          Pvalue += aTile[threadIdx.y][i] * bTile[i][threadIdx.x];
        }
        mat_c[batch][head][row][col] = Pvalue;
      }
    }
  }
}
using namespace nvcuda;
// template <typename scalar_t>
__global__ void simple_wmma_gemm(float *a, float *b, float *c, int m_ld,
                                 int n_ld, int k_ld, int num_bh)
{
  // Leading dimensions. Packed with no transpositions.
  int lda = k_ld;
  int ldb = k_ld;
  int ldc = n_ld;

  // Tile using a 2D grid
  int warpM = (blockIdx.x * blockDim.x + threadIdx.x) / warpSize;
  int warpN = (blockIdx.y * blockDim.y + threadIdx.y);

  // Declare the fragments
  wmma::fragment<wmma::matrix_a, WMMA_M, WMMA_N, WMMA_K, float, wmma::row_major> a_frag;
  wmma::fragment<wmma::matrix_b, WMMA_M, WMMA_N, WMMA_K, float, wmma::col_major> b_frag;
  wmma::fragment<wmma::accumulator, WMMA_M, WMMA_N, WMMA_K, float> c_frag;

  wmma::fill_fragment(c_frag, 0.0f);

  // Loop over k
  for (int i = 0; i < k_ld; i += WMMA_K)
  {
    int aCol = i;
    int aRow = warpM * WMMA_M;
    int bCol = warpN * N;
    int bRow = i;

    // Bounds checking
    if (aRow < m_ld && aCol < k_ld && bRow < k_ld && bCol < n_ld)
    {
      // Load the inputs
      wmma::load_matrix_sync(a_frag, a + aCol + aRow * lda, lda);
      wmma::load_matrix_sync(b_frag, b + bRow + bCol * ldb, ldb);

      // Perform the matrix multiplication
      wmma::mma_sync(c_frag, a_frag, b_frag, c_frag);
    }
  }

  // Load in the current value of c, scale it by beta, and add this our result
  // scaled by alpha
  int cCol = warpN * WMMA_N;
  int cRow = warpM * WMMA_M;

  if (cRow < m_ld && cCol < n_ld)
  {
    // Store the output
    wmma::store_matrix_sync(c + cCol + cRow * ldc, c_frag, ldc,
                            wmma::mem_row_major);
  }
}
// } // namespace

torch::Tensor matmul_cuda_1d(
    torch::Tensor mat_a,
    torch::Tensor mat_b)
{
  auto mat_c = torch::zeros_like(mat_a);
  int m = mat_a.size(0);

  int threads = 1024;
  int blocks = (m + threads - 1) / threads;

  AT_DISPATCH_FLOATING_TYPES(mat_a.type(), "matmul_cuda_1d", ([&]
                                                              { matmul_cuda_1d_kernel<scalar_t><<<blocks, threads>>>(
                                                                    mat_a.packed_accessor<scalar_t, 1, torch::RestrictPtrTraits, size_t>(),
                                                                    mat_b.packed_accessor<scalar_t, 1, torch::RestrictPtrTraits, size_t>(),
                                                                    mat_c.packed_accessor<scalar_t, 1, torch::RestrictPtrTraits, size_t>(),
                                                                    m); }));
  return mat_c;
}

torch::Tensor matmul_cuda_2d(
    torch::Tensor mat_a,
    torch::Tensor mat_b)
{
  int m = mat_a.size(0);
  int n = mat_a.size(1);
  int k = mat_b.size(1);
  auto mat_c = torch::zeros({m, k});

  int dimx = 32;
  int dimy = 32;
  dim3 block(dimx, dimy);
  dim3 grid((m + block.x - 1) / block.x, (k + block.y - 1) / block.y);

  AT_DISPATCH_FLOATING_TYPES(mat_a.type(), "matmul_cuda_2d", ([&]
                                                              { matmul_cuda_2d_kernel<scalar_t><<<grid, block>>>(
                                                                    mat_a.packed_accessor<scalar_t, 2, torch::RestrictPtrTraits, size_t>(),
                                                                    mat_b.packed_accessor<scalar_t, 2, torch::RestrictPtrTraits, size_t>(),
                                                                    mat_c.packed_accessor<scalar_t, 2, torch::RestrictPtrTraits, size_t>(),
                                                                    m, n, k); }));
  return mat_c;
}

torch::Tensor matmul_cuda_4d(
    torch::Tensor mat_a,
    torch::Tensor mat_b)
{
  // A: m*n    B: n*k
  int m = mat_a.size(2);
  int n = mat_a.size(3);
  int k = mat_b.size(3);
  int batchs = mat_a.size(0);
  int heads = mat_a.size(1);

  auto options = torch::TensorOptions().device(torch::kCUDA, 0);

  auto mat_c = torch::zeros({batchs, heads, m, k}, options);

  // auto ac_mat_a = mat_a.packed_accessor<scalar_t, 4, torch::RestrictPtrTraits, size_t>();
  // auto ac_mat_b = mat_b.packed_accessor<scalar_t, 4, torch::RestrictPtrTraits, size_t>();
  // auto ac_mat_c = mat_c.packed_accessor<scalar_t, 4, torch::RestrictPtrTraits, size_t>();

  int dimx = 32;
  int dimy = 32;
  dim3 block(dimx, dimy);
  dim3 grid((m + block.x - 1) / block.x, (k + block.y - 1) / block.y);

  AT_DISPATCH_FLOATING_TYPES(mat_a.type(), "matmul_cuda_4d", ([&]
                                                              { matmul_cuda_4d_kernel<scalar_t><<<grid, block>>>(
                                                                    mat_a.packed_accessor<scalar_t, 4, torch::RestrictPtrTraits, size_t>(),
                                                                    mat_b.packed_accessor<scalar_t, 4, torch::RestrictPtrTraits, size_t>(),
                                                                    mat_c.packed_accessor<scalar_t, 4, torch::RestrictPtrTraits, size_t>(),
                                                                    m, n, k); }));

  return mat_c;
}

torch::Tensor opt_matmul_cuda_4d(
    torch::Tensor mat_a,
    torch::Tensor mat_b)
{
  // A: m*n    B: n*k
  int m = mat_a.size(2);
  int n = mat_a.size(3);
  int k = mat_b.size(3);
  int batchs = mat_a.size(0);
  int heads = mat_a.size(1);

  auto options = torch::TensorOptions().device(torch::kCUDA, 0);

  auto mat_c = torch::zeros({batchs, heads, m, k}, options);

  // auto ac_mat_a = mat_a.packed_accessor<scalar_t, 4, torch::RestrictPtrTraits, size_t>();
  // auto ac_mat_b = mat_b.packed_accessor<scalar_t, 4, torch::RestrictPtrTraits, size_t>();
  // auto ac_mat_c = mat_c.packed_accessor<scalar_t, 4, torch::RestrictPtrTraits, size_t>();

  int dimx = 32;
  int dimy = 32;
  dim3 block(dimx, dimy);
  dim3 grid((m + block.x - 1) / block.x, (k + block.y - 1) / block.y);

  AT_DISPATCH_FLOATING_TYPES(mat_a.type(), "matmul_cuda_4d", ([&]
                                                              { shared_matmul_cuda_4d_kernel<scalar_t><<<grid, block>>>(
                                                                    mat_a.packed_accessor<scalar_t, 4, torch::RestrictPtrTraits, size_t>(),
                                                                    mat_b.packed_accessor<scalar_t, 4, torch::RestrictPtrTraits, size_t>(),
                                                                    mat_c.packed_accessor<scalar_t, 4, torch::RestrictPtrTraits, size_t>(),
                                                                    m, n, k); }));

  return mat_c;
}

torch::Tensor tensor_core_matmul_cuda_4d(
    torch::Tensor mat_a,
    torch::Tensor mat_b)
{
  // A: m*n    B: n*k
  int m = mat_a.size(2);
  int n = mat_a.size(3);
  int k = mat_b.size(3);
  int batchs = mat_a.size(0);
  int heads = mat_a.size(1);

  auto options = torch::TensorOptions().device(torch::kCUDA, 0);

  auto mat_c = torch::zeros({batchs, heads, m, k}, options);

  // auto ac_mat_a = mat_a.packed_accessor<scalar_t, 4, torch::RestrictPtrTraits, size_t>();
  // auto ac_mat_b = mat_b.packed_accessor<scalar_t, 4, torch::RestrictPtrTraits, size_t>();
  // auto ac_mat_c = mat_c.packed_accessor<scalar_t, 4, torch::RestrictPtrTraits, size_t>();

  int dimx = 32;
  int dimy = 32;
  dim3 block(dimx, dimy);
  dim3 grid((m + block.x - 1) / block.x, (k + block.y - 1) / block.y);

  // AT_DISPATCH_FLOATING_TYPES(mat_a.type(), "tensor_core_matmul_cuda_4d", ([&]
  //                                                                         { simple_wmma_gemm<float><<<grid, block>>>(
  //                                                                               mat_a.data<float>(),
  //                                                                               mat_b.data<float>(),
  //                                                                               mat_c.data<float>(),
  //                                                                               m, n, k, 0); }));
  simple_wmma_gemm<<<grid, block>>>(
      mat_a.data<float>(),
      mat_b.data<float>(),
      mat_c.data<float>(),
      m, n, k, 0);
  // AT_DISPATCH_FLOATING_TYPES(mat_a.type(), "tensor_core_matmul_cuda_4d", ([&]
  //                                                                         { simple_wmma_gemm<scalar_t><<<grid, block>>>(
  //                                                                               mat_a.data<scalar_t>(),
  //                                                                               mat_b.data<scalar_t>(),
  //                                                                               mat_c.data<scalar_t>(),
  //                                                                               m, n, k, 0); }));
  // AT_DISPATCH_FLOATING_TYPES(mat_a.type(), "matmul_cuda_4d", ([&]
  //                                                             { shared_matmul_cuda_4d_kernel<scalar_t><<<grid, block>>>(
  //                                                                   mat_a.packed_accessor<scalar_t, 4, torch::RestrictPtrTraits, size_t>(),
  //                                                                   mat_b.packed_accessor<scalar_t, 4, torch::RestrictPtrTraits, size_t>(),
  //                                                                   mat_c.packed_accessor<scalar_t, 4, torch::RestrictPtrTraits, size_t>(),
  //                                                                   m, n, k); }));

  return mat_c;
}

I just learned how to develop custom cuda operator in PyTorch and it’s just SO FRUSTRATED to stuck in the BUG.
Any help will be appreciated!!!

Context:
OH MY GOSH!!! I solve it.
The problem is still my inexperience.
The definition of wmma::fragment is
  //
  // Fragment template
  //
  template<typename Use, int m, int n, int k, typename T, typename Layout=void> class fragment;

  //
  // Fragments for 16x16x16
  //
  template<> class fragment<matrix_a, 16, 16, 16, __half, row_major> : public __frag_base<__half, 16> {};
  template<> class fragment<matrix_a, 16, 16, 16, __half, col_major> : public __frag_base<__half, 16> {};
  template<> class fragment<matrix_b, 16, 16, 16, __half, row_major> : public __frag_base<__half, 16> {};
  template<> class fragment<matrix_b, 16, 16, 16, __half, col_major> : public __frag_base<__half, 16> {};
  template<> class fragment<accumulator, 16, 16, 16, __half> : public __frag_base<__half, 8> {};
  template<> class fragment<accumulator, 16, 16, 16, float> : public __frag_base<float, 8> {};

#ifdef __CUDA_IMMA__
  template<> class fragment<matrix_a, 16, 16, 16, signed char, row_major> : public __frag_base<signed char, 8> {};
  template<> class fragment<matrix_a, 16, 16, 16, signed char, col_major> : public __frag_base<signed char, 8> {};
  template<> class fragment<matrix_a, 16, 16, 16, unsigned char, row_major> : public __frag_base<unsigned char, 8> {};
  template<> class fragment<matrix_a, 16, 16, 16, unsigned char, col_major> : public __frag_base<unsigned char, 8> {};
  template<> class fragment<matrix_b, 16, 16, 16, signed char, row_major> : public __frag_base<signed char, 8> {};
  template<> class fragment<matrix_b, 16, 16, 16, signed char, col_major> : public __frag_base<signed char, 8> {};
  template<> class fragment<matrix_b, 16, 16, 16, unsigned char, row_major> : public __frag_base<unsigned char, 8> {};
  template<> class fragment<matrix_b, 16, 16, 16, unsigned char, col_major> : public __frag_base<unsigned char, 8> {};
  template<> class fragment<accumulator, 16, 16, 16, int> : public __frag_base<int, 8> {};
#endif  /* __CUDA_IMMA__ */

#ifdef __CUDA_AMPERE_MMA__
  template<> class fragment<matrix_a, 16, 16, 16, __nv_bfloat16, row_major> : public __frag_base<__nv_bfloat16, 8> {};
  template<> class fragment<matrix_a, 16, 16, 16, __nv_bfloat16, col_major> : public __frag_base<__nv_bfloat16, 8> {};
  template<> class fragment<matrix_b, 16, 16, 16, __nv_bfloat16, row_major> : public __frag_base<__nv_bfloat16, 8> {};
  template<> class fragment<matrix_b, 16, 16, 16, __nv_bfloat16, col_major> : public __frag_base<__nv_bfloat16, 8> {};
#endif  /* __CUDA_AMPERE_MMA__ */

  //
  // Fragments for 32x8x16
  //
  template<> class fragment<matrix_a, 32, 8, 16, __half, row_major> : public __frag_base<__half, 16> {};
  template<> class fragment<matrix_a, 32, 8, 16, __half, col_major> : public __frag_base<__half, 16> {};
  template<> class fragment<matrix_b, 32, 8, 16, __half, row_major> : public __frag_base<__half, 16> {};
  template<> class fragment<matrix_b, 32, 8, 16, __half, col_major> : public __frag_base<__half, 16> {};
  template<> class fragment<accumulator, 32, 8, 16, __half> : public __frag_base<__half, 8> {};
  template<> class fragment<accumulator, 32, 8, 16, float> : public __frag_base<float, 8> {};

#ifdef __CUDA_IMMA__
  template<> class fragment<matrix_a, 32, 8, 16, signed char, row_major> : public __frag_base<signed char, 16> {};
  template<> class fragment<matrix_a, 32, 8, 16, signed char, col_major> : public __frag_base<signed char, 16> {};
  template<> class fragment<matrix_a, 32, 8, 16, unsigned char, row_major> : public __frag_base<unsigned char, 16> {};
  template<> class fragment<matrix_a, 32, 8, 16, unsigned char, col_major> : public __frag_base<unsigned char, 16> {};
  template<> class fragment<matrix_b, 32, 8, 16, signed char, row_major> : public __frag_base<signed char, 4> {};
  template<> class fragment<matrix_b, 32, 8, 16, signed char, col_major> : public __frag_base<signed char, 4> {};
  template<> class fragment<matrix_b, 32, 8, 16, unsigned char, row_major> : public __frag_base<unsigned char, 4> {};
  template<> class fragment<matrix_b, 32, 8, 16, unsigned char, col_major> : public __frag_base<unsigned char, 4> {};
  template<> class fragment<accumulator, 32, 8, 16, int> : public __frag_base<int, 8> {};
#endif  /* __CUDA_IMMA__ */

#ifdef __CUDA_AMPERE_MMA__
  template<> class fragment<matrix_a, 32, 8, 16, __nv_bfloat16, row_major> : public __frag_base<__nv_bfloat16, 16> {};
  template<> class fragment<matrix_a, 32, 8, 16, __nv_bfloat16, col_major> : public __frag_base<__nv_bfloat16, 16> {};
  template<> class fragment<matrix_b, 32, 8, 16, __nv_bfloat16, row_major> : public __frag_base<__nv_bfloat16, 4> {};
  template<> class fragment<matrix_b, 32, 8, 16, __nv_bfloat16, col_major> : public __frag_base<__nv_bfloat16, 4> {};
#endif  /* __CUDA_AMPERE_MMA__ */

  //
  // Fragments for 8x32x16
  //
  template<> class fragment<matrix_a, 8, 32, 16, __half, row_major> : public __frag_base<__half, 16> {};
  template<> class fragment<matrix_a, 8, 32, 16, __half, col_major> : public __frag_base<__half, 16> {};
  template<> class fragment<matrix_b, 8, 32, 16, __half, row_major> : public __frag_base<__half, 16> {};
  template<> class fragment<matrix_b, 8, 32, 16, __half, col_major> : public __frag_base<__half, 16> {};
  template<> class fragment<accumulator, 8, 32, 16, __half> : public __frag_base<__half, 8> {};
  template<> class fragment<accumulator, 8, 32, 16, float> : public __frag_base<float, 8> {};

#ifdef __CUDA_IMMA__
  template<> class fragment<matrix_a, 8, 32, 16, signed char, row_major> : public __frag_base<signed char, 4> {};
  template<> class fragment<matrix_a, 8, 32, 16, signed char, col_major> : public __frag_base<signed char, 4> {};
  template<> class fragment<matrix_a, 8, 32, 16, unsigned char, row_major> : public __frag_base<unsigned char, 4> {};
  template<> class fragment<matrix_a, 8, 32, 16, unsigned char, col_major> : public __frag_base<unsigned char, 4> {};
  template<> class fragment<matrix_b, 8, 32, 16, signed char, row_major> : public __frag_base<signed char, 16> {};
  template<> class fragment<matrix_b, 8, 32, 16, signed char, col_major> : public __frag_base<signed char, 16> {};
  template<> class fragment<matrix_b, 8, 32, 16, unsigned char, row_major> : public __frag_base<unsigned char, 16> {};
  template<> class fragment<matrix_b, 8, 32, 16, unsigned char, col_major> : public __frag_base<unsigned char, 16> {};
  template<> class fragment<accumulator, 8, 32, 16, int> : public __frag_base<int, 8> {};
#endif  /* __CUDA_IMMA__ */

#ifdef __CUDA_AMPERE_MMA__
  template<> class fragment<matrix_a, 8, 32, 16, __nv_bfloat16, row_major> : public __frag_base<__nv_bfloat16, 4> {};
  template<> class fragment<matrix_a, 8, 32, 16, __nv_bfloat16, col_major> : public __frag_base<__nv_bfloat16, 4> {};
  template<> class fragment<matrix_b, 8, 32, 16, __nv_bfloat16, row_major> : public __frag_base<__nv_bfloat16, 16> {};
  template<> class fragment<matrix_b, 8, 32, 16, __nv_bfloat16, col_major> : public __frag_base<__nv_bfloat16, 16> {};
#endif  /* __CUDA_AMPERE_MMA__ */

#ifdef __CUDA_SUBBYTE_IMMA__
  //
  // Fragments for 8x8x32
  //
  template<> class fragment<matrix_a, 8, 8, 32, experimental::precision::u4, row_major> : public __frag_base<experimental::precision::u4, 8, 1> {};
  template<> class fragment<matrix_a, 8, 8, 32, experimental::precision::s4, row_major> : public __frag_base<experimental::precision::s4, 8, 1> {};
  template<> class fragment<matrix_b, 8, 8, 32, experimental::precision::u4, col_major> : public __frag_base<experimental::precision::u4, 8, 1> {};
  template<> class fragment<matrix_b, 8, 8, 32, experimental::precision::s4, col_major> : public __frag_base<experimental::precision::s4, 8, 1> {};
  template<> class fragment<accumulator, 8, 8, 32, int> : public __frag_base<int, 2> {};

  //
  // Fragments for 8x8x128
  //
  template<> class fragment<matrix_a, 8, 8, 128, experimental::precision::b1, row_major> : public __frag_base<experimental::precision::b1, 32, 1> {};
  template<> class fragment<matrix_b, 8, 8, 128, experimental::precision::b1, col_major> : public __frag_base<experimental::precision::b1, 32, 1> {};
  template<> class fragment<accumulator, 8, 8, 128, int> : public __frag_base<int, 2> {};
#endif  /* __CUDA_SUBBYTE_IMMA__ */

#ifdef __CUDA_AMPERE_MMA__
  //
  // Fragments for 16x16x8
  //
  template<> class fragment<matrix_a, 16, 16, 8, precision::tf32, row_major> : public __frag_base<precision::tf32, 4> {};
  template<> class fragment<matrix_a, 16, 16, 8, precision::tf32, col_major> : public __frag_base<precision::tf32, 4> {};
  template<> class fragment<matrix_b, 16, 16, 8, precision::tf32, row_major> : public __frag_base<precision::tf32, 4> {};
  template<> class fragment<matrix_b, 16, 16, 8, precision::tf32, col_major> : public __frag_base<precision::tf32, 4> {};
  template<> class fragment<accumulator, 16, 16, 8, float> : public __frag_base<float, 8> {};

  //
  // Fragments for 8x8x4
  //
  template<> class fragment<matrix_a, 8, 8, 4, double, row_major> : public __frag_base<double, 1> {};
  template<> class fragment<matrix_a, 8, 8, 4, double, col_major> : public __frag_base<double, 1> {};
  template<> class fragment<matrix_b, 8, 8, 4, double, row_major> : public __frag_base<double, 1> {};
  template<> class fragment<matrix_b, 8, 8, 4, double, col_major> : public __frag_base<double, 1> {};
  template<> class fragment<accumulator, 8, 8, 4, double> : public __frag_base<double, 2> {};
#endif  /* __CUDA_AMPERE_MMA__ */

So wmma::fragment<wmma::matrix_a, 16, 16, 16, float, wmma::row_major> a_frag; should not be exist in the code.
After re-set the size and precision of wmma::fragment, everything is perfect!


Can I get the CUDA tensor pointer in the python pytorch

Question: Hello I have device tensors defined in the pyton script - can I get the pointer to the begining of the array

Context:
// get the pointer of variable anytensor (cast to float type, this is up to your data type)
auto c_pointer = anytensor.data_ptr<float>();


Pytorch build from source stuck on multiple MKL_THREADING = OMP and quit configure without the result

Question:
OS: Ubuntu 20.04
Cuda 10.2
Tesla K10
driver: nvidia 470.82.01
GCC: 8
Anaconda ver: 2021.11
Cmake: 3.19.6
Before build were installed:
conda install -c numba numba
conda install astunparse numpy ninja pyyaml mkl mkl-include setuptools cmake cffi typing_extensions future six requests dataclasses
conda install -c pytorch magma-cuda102
The output of configure stage ends with stuck and quit with no result, no build

Building wheel torch-1.11.0a0+git08d8f81
– Building version 1.11.0a0+git08d8f81
cmake -GNinja -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/home/dig/pytorch/torch -DCMAKE_PREFIX_PATH=/home/dig/anaconda3/lib/python3.9/site-packages -DNUMPY_INCLUDE_DIR=/home/dig/anaconda3/lib/python3.9/site-packages/numpy/core/include -DPYTHON_EXECUTABLE=/home/dig/anaconda3/bin/python -DPYTHON_INCLUDE_DIR=/home/dig/anaconda3/include/python3.9 -DPYTHON_LIBRARY=/home/dig/anaconda3/lib/libpython3.9.a -DTORCH_BUILD_VERSION=1.11.0a0+git08d8f81 -DUSE_CUDA=1 -DUSE_CUDNN=1 -DUSE_MKLDNN=1 -DUSE_NUMPY=True /home/dig/pytorch
– The CXX compiler identification is Clang 10.0.0
– The C compiler identification is GNU 8.4.0
– Detecting CXX compiler ABI info
– Detecting CXX compiler ABI info - done
– Check for working CXX compiler: /usr/bin/c++ - skipped
– Detecting CXX compile features
– Detecting CXX compile features - done
– Detecting C compiler ABI info
– Detecting C compiler ABI info - done
– Check for working C compiler: /usr/bin/cc - skipped
– Detecting C compile features
– Detecting C compile features - done
– Not forcing any particular BLAS to be found
– Could not find ccache. Consider installing ccache to speed up compilation.
– Performing Test COMPILER_WORKS
– Performing Test COMPILER_WORKS - Success
– Performing Test SUPPORT_GLIBCXX_USE_C99
– Performing Test SUPPORT_GLIBCXX_USE_C99 - Success
– Performing Test CAFFE2_EXCEPTION_PTR_SUPPORTED
– Performing Test CAFFE2_EXCEPTION_PTR_SUPPORTED - Success
– std::exception_ptr is supported.
– Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING
– Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING - Failed
– Turning off deprecation warning due to glog.
– Performing Test C_HAS_AVX_1
– Performing Test C_HAS_AVX_1 - Failed
– Performing Test C_HAS_AVX_2
– Performing Test C_HAS_AVX_2 - Success
– Performing Test C_HAS_AVX2_1
– Performing Test C_HAS_AVX2_1 - Failed
– Performing Test C_HAS_AVX2_2
– Performing Test C_HAS_AVX2_2 - Success
– Performing Test C_HAS_AVX512_1
– Performing Test C_HAS_AVX512_1 - Failed
– Performing Test C_HAS_AVX512_2
– Performing Test C_HAS_AVX512_2 - Failed
– Performing Test C_HAS_AVX512_3
– Performing Test C_HAS_AVX512_3 - Failed
– Performing Test CXX_HAS_AVX_1
– Performing Test CXX_HAS_AVX_1 - Failed
– Performing Test CXX_HAS_AVX_2
– Performing Test CXX_HAS_AVX_2 - Success
– Performing Test CXX_HAS_AVX2_1
– Performing Test CXX_HAS_AVX2_1 - Failed
– Performing Test CXX_HAS_AVX2_2
– Performing Test CXX_HAS_AVX2_2 - Success
– Performing Test CXX_HAS_AVX512_1
– Performing Test CXX_HAS_AVX512_1 - Failed
– Performing Test CXX_HAS_AVX512_2
– Performing Test CXX_HAS_AVX512_2 - Failed
– Performing Test CXX_HAS_AVX512_3
– Performing Test CXX_HAS_AVX512_3 - Failed
– Current compiler supports avx2 extension. Will build perfkernels.
– Performing Test CAFFE2_COMPILER_SUPPORTS_AVX512_EXTENSIONS
– Performing Test CAFFE2_COMPILER_SUPPORTS_AVX512_EXTENSIONS - Success
– Current compiler supports avx512f extension. Will build fbgemm.
– Performing Test COMPILER_SUPPORTS_HIDDEN_VISIBILITY
– Performing Test COMPILER_SUPPORTS_HIDDEN_VISIBILITY - Success
– Performing Test COMPILER_SUPPORTS_HIDDEN_INLINE_VISIBILITY
– Performing Test COMPILER_SUPPORTS_HIDDEN_INLINE_VISIBILITY - Success
– Performing Test COMPILER_SUPPORTS_RDYNAMIC
– Performing Test COMPILER_SUPPORTS_RDYNAMIC - Success
– Found CUDA: /usr/local/cuda-10.2 (found version “10.2”)
– The CUDA compiler identification is NVIDIA 10.2.89
– Detecting CUDA compiler ABI info
– Detecting CUDA compiler ABI info - done
– Check for working CUDA compiler: /usr/local/cuda-10.2/bin/nvcc - skipped
– Detecting CUDA compile features
– Detecting CUDA compile features - done
– Caffe2: CUDA detected: 10.2
– Caffe2: CUDA nvcc is: /usr/local/cuda-10.2/bin/nvcc
– Caffe2: CUDA toolkit directory: /usr/local/cuda-10.2
– Caffe2: Header version is: 10.2
– Found CUDNN: /usr/lib/cuda/lib64/libcudnn.so
– Found cuDNN: v8.3.0  (include: /usr/local/cuda-10.2/include, library: /usr/lib/cuda/lib64/libcudnn.so)
– /usr/local/cuda-10.2/lib64/libnvrtc.so shorthash is 08c4863f
– Autodetected CUDA architecture(s):  3.0 3.0 3.0
– Added CUDA NVCC flags for: -gencode;arch=compute_30,code=sm_30
– Building using own protobuf under third_party per request.
– Use custom protobuf build.
– 3.13.0.0
– Looking for pthread.h
– Looking for pthread.h - found
– Performing Test CMAKE_HAVE_LIBC_PTHREAD
– Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
– Check if compiler accepts -pthread
– Check if compiler accepts -pthread - yes
– Found Threads: TRUE
– Performing Test protobuf_HAVE_BUILTIN_ATOMICS
– Performing Test protobuf_HAVE_BUILTIN_ATOMICS - Success
– Caffe2 protobuf include directory: $<BUILD_INTERFACE:/home/dig/pytorch/third_party/protobuf/src>$<INSTALL_INTERFACE:include>
– Trying to find preferred BLAS backend of choice: MKL
– MKL_THREADING = OMP
– Looking for sys/types.h
– Looking for sys/types.h - found
– Looking for stdint.h
– Looking for stdint.h - found
– Looking for stddef.h
– Looking for stddef.h - found
– Check size of void*
– Check size of void* - done
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
– MKL_THREADING = OMP
I try to change the Conda ver from 2021.4 up to 2021.11 - no result

Context: I also try to build the pytorch 1.10 - also the same problem is


Low performance of transferring tensor to CUDA

Question:
Hello Team, i using Libtorch with CUDA support. I get lowest performance on transferring memory to CUDA operation. It cost about 2.60s.
tensor_image = tensor_image.to(torch::kCUDA);
How can i improve this operation, check it or  profile?
  clock_t tROIset = clock();
  cv::Rect myROI(30, 10, 400, 400);
  printf("ROI set.\n Time taken: %.2fs\n", (double)(clock() - tROIset)/CLOCKS_PER_SEC);

  clock_t tLoadOpenCV = clock();
  cv::Mat img = cv::imread("test_data/image.png");     // 600x900
  printf("loader of Tensor.\n Time taken: %.2fs\n", (double)(clock() - tLoadOpenCV)/CLOCKS_PER_SEC);

  clock_t tTester = clock();
  tensor_image = tensor_image.to(torch::kCUDA);
  printf("CPU-GPU transfer.\n Time taken: %.2fs\n", (double)(clock() - tTester)/CLOCKS_PER_SEC);

Result of execution:
ROI set in Tensor .               Time taken: 0.00s
loader of Tensor.                  Time taken: 0.01s
CPU-GPU Tensor transfer.  Time taken: 2.60s

PyTorch version: 1.10.0
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89

Context:
How can i use non blocking memory transfering here ?
non_blocking=True, or something like this…


Using torch::from_blob(…) with const data

Question:
Hello,
due to infrastructural limitations, I need to use a const input to from_blob to create a torch::tensor. Minimal reproducible example:
#include <iostream>
#include <torch/torch.h>

int main ()
{
    const float array[] = {1.23, 2.34, 3.45, 4.56, 5.67};
    auto options = torch::TensorOptions().dtype(torch::kFloat32);
    torch::Tensor t1 = torch::from_blob(array, {5}, options);
    std::cout << t1 << std::endl;
    return 0;
}

This raises however a compilation error: invalid conversion from ‘const void*’ to ‘void*’ [-fpermissive]. Of course I can cast away the const-ness of the array, but is there a way to avoid this? Can I somehow use from_blob (or similar)  with const data input?
As always, help is highly appreciated…
Thank you & Best regards,
RB

Context:
You can use std::copy_n as below:
const float array[] = {1.23, 2.34, 3.45, 4.56, 5.67};
auto options = torch::TensorOptions().dtype(torch::kFloat32);
torch::Tensor t1 = torch::empty({5}, options);
std::copy_n(array, 5, t1.contiguous().data_ptr<float>());


Libtorch C++ *.so files truncated error when ldconfig

Question:
I am trying to add libtorch C++ shared library to system library path, as shown below
root@:/etc#cat ld.so.conf.d/local.conf
/usr/local/torch_lib
root@:/etc#
root@:/etc#
root@:/etc#cat ld.so.conf
include ld.so.conf.d/*.conf

It gives lots of truncated errors. How shall this be fixed? I tried both the officially provided libtorch C++ library and my own built libtorch C++
ldconfig: file /usr/local/torch_lib/libc10_cuda.so is truncated

ldconfig: file /usr/local/torch_lib/libcaffe2_detectron_ops_gpu.so is truncated

ldconfig: file /usr/local/torch_lib/libthnvrtc.so is truncated

ldconfig: file /usr/local/torch_lib/libcudart-f7fdd8d7.so.9.0 is truncated

ldconfig: file /usr/local/torch_lib/libnvrtc-builtins.so is truncated

ldconfig: file /usr/local/torch_lib/libnvToolsExt-3965bdd0.so.1 is truncated

ldconfig: file /usr/local/torch_lib/libnvrtc-007d19c9.so.9.0 is truncated

ldconfig: file /usr/local/torch_lib/libgomp-4f651535.so.1 is truncated

Context: I encountered the same problem. Were you able to figure out a way to fix this?


Question:
I need to create an instance of AdamParamState. I looked through the adam.cpp code as an example, and accordingly copied the following code from there. But, with the provided headers, it still does not recognize AdamParamState.
I appreciate any help or comment on this matter.
#include <torch/optim/adam.h>
#include <torch/csrc/autograd/variable.h>
#include <torch/nn/module.h>
#include <torch/serialize/archive.h>
#include <torch/utils.h>

#include <ATen/ATen.h>

void get_state(torch::optim::Optimizer *optimizer){
    for (auto& group : optimizer->param_groups()) {
        for (auto &p : group.params()) {
            if (!p.grad().defined()) {
                continue;
            }

            auto grad = p.grad();
            TORCH_CHECK(!grad.is_sparse(),
                        "Adam does not support sparse gradients"/*, please consider SparseAdam instead*/);
            ska::flat_hash_map<std::string, std::unique_ptr<torch::optim::OptimizerParamState>>& state_ = optimizer->state();
            auto param_state = state_.find(c10::guts::to_string(p.unsafeGetTensorImpl()));
            auto tmp_ = p.dim();
            int tmp_0;
            int tmp_1;
            if (tmp_ > 0)
                tmp_0  = p.size(0);
            if (tmp_ > 1)
                tmp_1  = p.size(1);

            std::cout << tmp_ << tmp_0 << tmp_1 << std::endl;
//                auto& options = static_cast<AdamOptions&>(group.options());
            auto& state = static_cast<AdamParamState&>(*state_[c10::guts::to_string(p.unsafeGetTensorImpl())]);
        }
    }

}

Context:
I found that this works:
auto& state = static_cast<torch::optim::AdamParamState&>(*state_[c10::guts::to_string(p.unsafeGetTensorImpl())]);
very simple and juicy!


I can’t build for LibTorch on aarch64. Cannot find libcublas.so

Question:
I’m trying to install LibTorch on a Jetson AGX Xavier.
I do a recursive git clone of this pytorch repo and run 'python3 tools/build_libtorch.py.
The build fails with the following error
FAILED: lib/libtorch_global_deps.so
: && /usr/bin/cc -fPIC -fopenmp -DNDEBUG -O3 -DNDEBUG -DNDEBUG -Wl,–no-as-needed -rdynamic -shared -Wl,-soname,libtorch_global_deps.so -o lib/libtorch_global_deps.so caffe2/CMakeFiles/torch_global_deps.dir/__/torch/csrc/empty.c.o -Wl,-rpath,/usr/lib/aarch64-linux-gnu/openmpi/lib:/usr/local/cuda/lib64:::::::: /usr/lib/aarch64-linux-gnu/openmpi/lib/libmpi_cxx.so /usr/lib/aarch64-linux-gnu/openmpi/lib/libmpi.so /usr/local/cuda/lib64/libcurand.so /usr/local/cuda/lib64/libcufft.so -lCUDA_cublas_LIBRARY-NOTFOUND /usr/lib/aarch64-linux-gnu/libcudnn.so /usr/local/cuda/lib64/libcudart.so /usr/local/cuda/lib64/libnvToolsExt.so && :
/usr/bin/ld: cannot find -lCUDA_cublas_LIBRARY-NOTFOUND
collect2: error: ld returned 1 exit status
[2357/3927] Building CXX object c10/test/CMakeFiles/c10_intrusive_ptr_test.dir/util/intrusive_ptr_test.cpp.o
ninja: build stopped: subcommand failed.

Versions
JetPack 4.6
Nvidia Jetson AGX Xavier
The current pytorch repo available on GitHub.

Context: libcublas.so won’t be built, but ships with the CUDA toolkit. Make sure this library was installed in the CUDA toolkit and set the CUDA path if needed. The install log should show which CUDA version was detected and where it’s located.


Torch model parameters by dictionary type

Question:
Hi! guys!
When I coded fully connected layer model  into like below:
(The number of fully connected layer is dependent on “net_info” variable)

class Model(nn.Module):

def __init__(self,net_info):
   super(Model,self).__init__()
       self.net_info = net_info
       self.w,self.b = {},{}
     for idx,hidden_size in enumerate(self.net_info):
         if idx == 0:
             self.w[idx] = nn.Parameter(torch.rand(1,hidden_size),requires_grad=True)
            self.b[idx] = nn.Parameter(torch.rand(hidden_size),requires_grad=True)
        else:
           self.w[idx] = nn.Parameter(torch.rand(net_info[idx-1],hidden_size),requires_grad=True)
          self.b[idx] = nn.Parameter(torch.rand(hidden_size),requires_grad=True)
  self.w[idx+1] = nn.Parameter(torch.rand(hidden_size,1),requires_grad=True)
  self.b[idx+1] = nn.Parameter(torch.rand(1),requires_grad=True)

def forward(self,x):

  for idx,hidden_size in enumerate(self.net_info):
      x = F.elu(torch.matmul(x,self.w[idx])+self.b[idx])
   x = torch.matmul(x,self.w[idx+1])+self.b[idx+1]

return x

The problem of above torch model is that does not create “model.parmeters()”

net_info = [50,60,70,50]
model = Model(net_info)
for p in model.parameters():

print p

Nothing happen, No trainable parameters! Do you have any idea?

Context: Using a Python dict will not properly register the parameters so use nn.ParameterDict instead.


Question:
Hello,
I want to take the output of the forward pass and detach it to generate a matrix (that is similar to the input) using an external function (not a pytorch function), then create a loss function based on this generated matrix and the input matrix.
does this cause a problem for the gradient?
Thank you

Context:
Yes, using other libraries or explicitly detaching the tensors won’t allow Autograd to track these operations and will thus detach the tensor from the computation graph (parameters used in previous operations also won’t gradients).
To use other libraries you could write a custom autograd.Function as described here 1.


How to get predict interval using Monte Calro Dropout. RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

Question:
Hi. I have the problem here.
Now, I am trying to get predict interval with using Monte Calro dropout.
However, I got following errors when I call loss.backward() after the running of following code
I do not kown why, any reply will be appreciated.
thanks.
class Net(nn.Module):
    def __init__(self, x1, x2, p, m, mode=0):
        super().__init__()
        input_dim = p*m
        hidden_dim = 128
        self.dropout = nn.Dropout(p=0.1)
        if mode == 1:
            self.embed = Encoder1(x1, x2, p)
        elif mode == 2:
            self.embed = Encoder2(x1, x2, p)
        else:
            self.embed = Encoder(x1, x2, p)

        self.share_block = nn.Sequential(
            ・・・・・
        )
        self.head = nn.Sequential(
            nn.BatchNorm1d(hidden_dim),
            nn.utils.weight_norm(nn.Linear(hidden_dim, 1))
        )

    def forward(self, x):
        x = self.share_block(x)
        _out = self.head(x)
        _out = self.dropout(_out)

        return _out

    def predict(self, x1, x2, alpha=0.95, samples=100):
        x1 = x1.to(device)
        x2 = x2.to(device)
        x = self.embed(x1, x2)
        z_c = norm.ppf(alpha)
        samples_ = [self.forward(x).detach()
                    for _ in range(samples)]
        pred_sample = torch.cat(samples_, dim=1)
        pred_mean = torch.mean(pred_sample, dim=1)
        pred_std = z_c * torch.std(pred_sample, dim=1)

        return pred_mean, pred_std

def train_and_test_gpu(model, num_epochs, dataset):
    train_dataset = utils.data.Subset(dataset, train_idx)
    train_loader = utils.data.DataLoader(
        train_dataset, batch_size=64, worker_init_fn=seed_worker, generator=g, shuffle=True)

    test_dataset = utils.data.Subset(dataset, test_idx)
    test_loader = utils.data.DataLoader(
        test_dataset, batch_size=64, shuffle=False)

    optimizer = optim.Adam(model.parameters(), lr=1e-6)
    criterion = nn.MSELoss()

    scheduler = optim.lr_scheduler.OneCycleLR(
        optimizer,
        max_lr=1e-3,
        steps_per_epoch=len(train_loader),
        epochs=num_epochs,
    )

    model.train()
    for epoch in tqdm(range(num_epochs)):
        for x1, x2, y in train_loader:

            x1 = x1.to(device)
            x2 = x2.to(device)
            y = torch.Tensor(y).to(device)

            optimizer.zero_grad()
            y_pred, _ = model.predict(x1, x2)

            loss = criterion(y_pred, y)
            loss.backward()  <ーーーーーー error occured here
            optimizer.step()
            scheduler.step()

    model.eval()
    with torch.no_grad():
        y_true_list = []
        y_pred_list = []
        for x1, x2, y_true in test_loader:
            x1 = torch.Tensor(x1).to(device)
            x2 = torch.Tensor(x2).to(device)

            y_pred, _ = model.predict(x1, x2)
            y_pred_list.extend(y_pred.detach().flatten().cpu().tolist())
            y_true_list.extend(y_true.detach().cpu().tolist())
    return y_true_list, y_pred_list

Context:
In your predict method you are explicitly detaching the tensors from the computation graph in:
samples_ = [self.forward(x).detach()
            for _ in range(samples)]

so that the samples_ doesn’t have a valid grad_fn anymore (it’s not attached to any computation graph) and thus the backward call fails.


Question:
hi, i have a problem here, i got a sequence of Variables which are the outputs of the bi-directional RNN, and i stacked them into a matrix xs_h whose dimension is (seq_length, batch_size, hidden_size), them i want to update the matrix xs_h by convoluting on two slices in xs_h, some codes are as follows:
new_xs_h = xs_h.clone()
vp, vc = xs_h[idx_0, bidx], xs_h[idx_1, bidx]
x = tc.stack([self.f1(vp), self.f2(vc)], dim=1)[None, :, :]
new_xs_h[idx_1, bidx] = self.tanh(self.l_f2(self.conv(x).squeeze()))
actually, i want to update the Variable xs_h and then let the new updated matrix new_xs_h get into my computation graph again. However, i got following errors when i call backward() after the running of above code:
RuntimeError: element 0 of variables does not require grad and does not have a grad_fn
i do not kown why, any reply will be appreciated.
thanks.

Context: It sounds like the problem is that your xs_h don’t have requires_grad=True. Have you tried creating Variables with requires_grad=True?


Implementing Truncated Backpropagation Through Time

Question:
Hello,
I’m implementing a recursive network that is going to be trained with very long sequences. I had memory problems when training because of that excessive length and I decided to use a truncated-BPTT algorithm to train it as described here 472, that is,
every k1 steps backpropagate taking k2 back steps
checking some examples 354  I could easily write the case when k1 = k2. However, I haven’t been able to implement the general case yet.
My first idea was to freeze the  gradient graph after the first k2 steps and keep changing the Variables referenced there. Later, I saw that the graph doesn’t directly reference earlier Variables, instead it directly incorporates the gradient graphs from that Variables. So I thought about looking for that copied subgraphs and substituting them by a Variable reference, but I found out that the gradient graph is unwritable and immutable.
The only idea I have left is to recompute the calculations in the overlaps between backpropagations. That’s going to work, but I’d really love to avoid recomputing things.
Any idea on how to implement this efficiently?

Context:
Here is an implementation that will work for any k1 and k2 and will reduce memory usage as much as possible.
If k2 is not huge and the one_step_module is relatively big, the slowdown of doing multiple backward should be negligible.
The code is not super clean and has been tested only against current master branch (where Variable and Tensor are merged) so you might need slight modifications if you use 0.3.
Hope this helps.

class TBPTT():
    def __init__(self, one_step_module, loss_module, k1, k2, optimizer):
        self.one_step_module = one_step_module
        self.loss_module = loss_module
        self.k1 = k1
        self.k2 = k2
        self.retain_graph = k1 < k2
        # You can also remove all the optimizer code here, and the
        # train function will just accumulate all the gradients in
        # one_step_module parameters
        self.optimizer = optimizer

    def train(self, input_sequence, init_state):
        states = [(None, init_state)]
        for j, (inp, target) in enumerate(input_sequence):

            state = states[-1][1].detach()
            state.requires_grad=True
            output, new_state = self.one_step_module(inp, state)
            states.append((state, new_state))

            while len(states) > self.k2:
                # Delete stuff that is too old
                del states[0]

            if (j+1)%self.k1 == 0:
                loss = self.loss_module(output, target)

                optimizer.zero_grad()
                # backprop last module (keep graph only if they ever overlap)
                start = time.time()
                loss.backward(retain_graph=self.retain_graph)
                for i in range(self.k2-1):
                    # if we get all the way back to the "init_state", stop
                    if states[-i-2][0] is None:
                        break
                    curr_grad = states[-i-1][0].grad
                    states[-i-2][1].backward(curr_grad, retain_graph=self.retain_graph)
                print("bw: {}".format(time.time()-start))
                optimizer.step()

seq_len = 20
layer_size = 50

idx = 0

class MyMod(nn.Module):
    def __init__(self):
        super(MyMod, self).__init__()
        self.lin = nn.Linear(2*layer_size, 2*layer_size)

    def forward(self, inp, state):
        global idx
        full_out = self.lin(torch.cat([inp, state], 1))
        # out, new_state = full_out.chunk(2, dim=1)
        out = full_out.narrow(1, 0, layer_size)
        new_state = full_out.narrow(1, layer_size, layer_size)
        def get_pr(idx_val):
            def pr(*args):
                print("doing backward {}".format(idx_val))
            return pr
        new_state.register_hook(get_pr(idx))
        out.register_hook(get_pr(idx))
        print("doing fw {}".format(idx))
        idx += 1
        return out, new_state

one_step_module = MyMod()
loss_module = nn.MSELoss()
input_sequence = [(torch.rand(200, layer_size), torch.rand(200, layer_size))] * seq_len

optimizer = torch.optim.SGD(one_step_module.parameters(), lr=1e-3)

runner = TBPTT(one_step_module, loss_module, 5, 7, optimizer)

runner.train(input_sequence, torch.zeros(200, layer_size))
print("done")


Gradient flow through torch.count_nonzero

Question:
As per below code the, the backward function is called after run. But once i uncomment the line “output=torch.count_nonzero(output)” it shows the error “RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn”. It is necessary to use “torch.count_nonzero” function and wants to pass the “grad_input” 1 as it is non differentiable.
from torch.autograd import Function
import torch
import torch.nn as nn

class custom_fun(Function):

    @staticmethod
    def forward(cxt, input):
        cxt.save_for_backward(input)
        output=torch.add(input,2)
        print("calling forward",output)
        # output=torch.count_nonzero(output)
        return output

    @staticmethod
    def backward(cxt, grad_output):
        input, = cxt.saved_tensors
        print("Calling backward function ")
        grad_input = grad_output.clone()
        grad_input = torch.ones(1)
        return grad_input
class my_module(nn.Module):

 def __init__(self):
      super(my_module,self).__init__()
      self.fc1 = nn.Linear(1,1)
      self.my_act_fn=custom_fun()
  def forward(self, x):
      out1=self.fc1(x)
      out=self.my_act_fn.apply(out1)
      return out

mod = my_module()
x=torch.tensor([1.1])
target=torch.tensor([2.5])
out_m=mod(x)
criterion=torch.nn.L1Loss()
loss=criterion(out_m,target)
loss.backward()

Context:
Hi,
The problem is that the output of count_nonzero is a Tensor of type int64. So it cannot require gradients (only continuous types can).
So you want to add a output = output.float() after the count_nonzero to make sure the output of your custom Function can require gradients.


Question:
Hi,
I got the following error:
“RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
RuntimeError: tensor does not have a device”
Some of my modules inherit jit.ScriptModule. Torch version:1.10.1
Thanks
Best regards

Context: Could you post a minimal, executable code snippet to reproduce this issue, please?


Building a custom operator using two pytorch ops

Question:
I have the following code in my nn.Module.
x = torch.cdist(a,b)
y = torch.softmax(x)
These two are differentiable, but due to the size of x, x/y need a lot of GPU memory, causing a OOM during backprop. Since I’m not sure if the checkpoint can help this (getting NaN with ddp) and x/y are very sparse (many near zero values), I was thinking of a custom operator in the following fashion, but cannot find a doc on how to call backward of existing pytorch constructs in a custom operator. Can anyone point me out some examples on how to complete this?
class MyFunc(torch.autograd.Function):
    @staticmethod
    def forward(ctx, a, b):
        x = torch.cdist(a,b)
        ctx.save_for_backward(a,b, sparse(x))
        return torch.softmax(x)

    @staticmethod
    def backward(ctx, g):
     ??

Context:
Have you looked into saved tensor hooks? Saved tensor hooks let you interpose behavior when the tensor is “packed” to be saved for backward and when it is “unpacked” to be used during the backward pass. Since GPU memory is your  issue, you can consider using the context manager torch.autograd.graph.save_on_cpu (which utilizes saved tensor hooks in its implementation). In this context all intermediary results are moved to cpu, and copied back to the original device when it is used during the backward pass.
https://pytorch.org/docs/stable/autograd.html?highlight=saved%20tensor%20hooks#torch.autograd.graph.saved_tensors_hook 1
To directly answer your question though, I believe some of the backwards are available in the torch.ops.aten namespace, e.g., torch.ops.aten._softmax_backward_data, torch.ops.aten._cdist_backward. The full list is available in the Pytorch source at aten/src/ATen/native/native_functions.yaml. This is NOT intended to be public api though, and is subject to change.


Bug of pytorch 1.10 for NVIDIA RTX A6000

Question:
Hi there,
I ran my code below on RTX A6000 with 2 GPUs or 4 GPUs. However, the CE loss becomes nan after just a few iterations. Then I check, it because the learnable parameters become nan after almost the first backpropagation. Then I test my code on other GPUs such as TITAN RTX, TITAN V, and Tesla V100 (32G). They work well on other GPUs except for RTX A6000. I am wondering if there is a bug on PyTorch for A6000. Could you guys please re-test my code on A6000 again to see if there is a PyTorch bug on A6000 or not?
The command I ran my code is
python -m torch.distributed.launch --master_port=6396 --nproc_per_node=2 debug_train_dist.py --ngpu 2 --reduction 8 -lr 0.001 -epoch 80 -nb_worker 8 -bs 50
The project code is below which helps you to reproduce. By the way, my PyTorch version on RTX A6000 is 1.10.0.dev20210831. The PyTorch version I test on other GPUs is 1.6.0.
import numpy as np
from random import randrange
import torch
import torch.nn as nn
from torch.utils import data
from tqdm import tqdm
import argparse
import json

def get_args():
    parser = argparse.ArgumentParser()
    parser.add_argument('-bs', type=int, default=100)
    parser.add_argument('-lr', type=float, default=0.001)
    parser.add_argument('-epoch', type=int, default=80)
    parser.add_argument('-nb_worker', type=int, default=8)
    parser.add_argument('-seed', type=int, default=1234)
    parser.add_argument('-model_params', type=json.loads, default=
    '{"first_conv":3, "in_channels":1, "filts":[128, [128,128], [128,256], [256,256]],' \
    '"blocks":[2,4], "nb_fc_att_node":[1], "nb_fc_node":1024, "gru_node":1024, "nb_gru_layer":1}')
    parser.add_argument('--local_rank', type=int, default=0)
    parser.add_argument('--ngpu', type=int, default=2)
    parser.add_argument('--reduction', type=int, help='reduction rate')
    args = parser.parse_args()
    return args

def keras_lr_decay(step, decay = 0.0001):
    return 1./(1. + decay * step)

class Residual_block_imgs(nn.Module):
    def __init__(self, nb_filts, shift, reduction=8, first = False):
        super(Residual_block_imgs, self).__init__()
        self.shift = shift
        self.lrelu = nn.LeakyReLU()
        self.lrelu_keras = nn.LeakyReLU(negative_slope=0.3)
        self.conv1 = nn.Conv1d(in_channels=nb_filts[0],
                               out_channels=nb_filts[1],
                               kernel_size=3,
                               padding=1,
                               stride=1)
        self.bn2 = nn.BatchNorm1d(num_features=nb_filts[1])
        self.conv2 = nn.Conv1d(in_channels=nb_filts[1],
                               out_channels=nb_filts[1],
                               padding=1,
                               kernel_size=3,
                               stride=1)

        if nb_filts[0] != nb_filts[1]:
            self.downsample = True
            self.conv_downsample = nn.Conv1d(in_channels=nb_filts[0],
                                             out_channels=nb_filts[1],
                                             padding=0,
                                             kernel_size=1,
                                             stride=1)
        else:
            self.downsample = False
        self.mp = nn.MaxPool1d(3)
        channel = nb_filts[0]
        self.avg_pool_t1 = nn.AdaptiveAvgPool1d(1)
        self.avg_pool_t2 = nn.AdaptiveAvgPool1d(1)
        self.down_t1 = nn.Sequential(
            nn.Linear(channel, channel // reduction, bias=False),
            nn.ReLU(inplace=True)
        )
        self.up_diff = nn.Sequential(
            nn.Linear(channel // reduction, channel, bias=False),
            nn.Sigmoid()
        )
        self.down_t2 = nn.Sequential(
            nn.Linear(channel, channel // reduction, bias=False),
            nn.ReLU(inplace=True)
        )

    def forward(self, x):
        identity = x
        x = self._middle(x)
        out = self.conv1(x)
        out = self.bn2(out)
        out = self.lrelu_keras(out)
        out = self.conv2(out)
        if self.downsample:
            identity = self.conv_downsample(identity)
        out += identity
        out = self.mp(out)
        return out

    def _middle(self, x):
        b, c, _ = x.size()
        length = x[:, :, self.shift:].shape[-1]
        x_t2 = x[:,:,self.shift:]
        x_t1 = x[:,:,:length]
        b_t2, c_t2, _ = x_t2.size()
        y_t2 = self.avg_pool_t2(x_t2).view(b_t2, c_t2)
        b_t1, c_t1, _ = x_t1.size()
        y_t1 = self.avg_pool_t1(x_t1).view(b_t1, c_t1)
        y_t2 = self.down_t2(y_t2)
        y_t1 = self.down_t1(y_t1)
        y = y_t2 - y_t1
        y = self.up_diff(y).view(b, c, 1)
        x = x * y.expand_as(x)
        return x

class Model_imgs(nn.Module):
    def __init__(self, d_args, reduction=8):
        super(Model_imgs, self).__init__()
        self.reduction = reduction
        self.first_conv = nn.Conv1d(
            in_channels=d_args['in_channels'],  # 1
            out_channels=d_args['filts'][0],  # 128
            kernel_size=d_args['first_conv'],  # 3
            stride=d_args['first_conv'])  # 3

        self.first_bn = nn.BatchNorm1d(
            num_features=d_args['filts'][0])  # 128
        self.lrelu_keras = nn.LeakyReLU(
            negative_slope=0.3)

        self.block0 = self._make_layer(
            nb_blocks=d_args['blocks'][0],  # 2
            nb_filts=d_args['filts'][1],  # 128
            shifts=[2667, 889],
            first=True)
        self.block1 = self._make_layer(
            nb_blocks=d_args['blocks'][1],  # 4
            nb_filts=d_args['filts'][2],  # 256
            shifts=[296, 99, 33, 11])
        self.bn_before_gru = nn.BatchNorm1d(
            num_features=d_args['filts'][2][-1])  # 256
        self.gru = nn.GRU(
            input_size=d_args['filts'][2][-1],  # 256
            hidden_size=d_args['gru_node'],  # 1024
            num_layers=d_args['nb_gru_layer'],  # 1
            batch_first=True)
        self.fc1_gru = nn.Linear(
            in_features=d_args['gru_node'],  # 1024
            out_features=d_args['nb_fc_node'])  # 256
        self.fc2_gru = nn.Linear(
            in_features=d_args['nb_fc_node'],  # 256
            out_features=d_args['nb_classes'],  # 6112
            bias=True)

    def forward(self, x):
        x = x.unsqueeze(1)
        x = self.first_conv(x)
        x = self.first_bn(x)
        x = self.lrelu_keras(x)
        x = self.block0(x)
        x = self.block1(x)
        x = self.bn_before_gru(x)
        x = self.lrelu_keras(x)
        x = x.permute(0, 2, 1)
        self.gru.flatten_parameters()
        x, _ = self.gru(x)
        x = x[:, -1, :]
        code = self.fc1_gru(x)
        code_norm = code.norm(p=2, dim=1, keepdim=True) / 10.
        code = torch.div(code, code_norm)
        out = self.fc2_gru(code)
        return out

    def _make_layer(self, nb_blocks, nb_filts, shifts, first = False):
        layers = []
        for i in range(nb_blocks):
            first = first if i == 0 else False
            shift = shifts[i]
            layers.append(Residual_block_imgs(nb_filts=nb_filts, shift=shift, reduction=self.reduction, first=first))
            if i == 0: nb_filts[0] = nb_filts[1]
        return nn.Sequential(*layers)

class Dataset_imgs(data.Dataset):
    def __init__(self, ):
        a = 1

    def __len__(self):
        return 10000

    def __getitem__(self, index):
        y = randrange(6112)
        X = np.random.random_sample((59049,)).astype(np.float32)
        return X, y

def train_model(model, db_gen, optimizer, epoch, args, device, lr_scheduler, criterion):
    model.train()
    if args.local_rank == 0:
        pbar = tqdm(total=len(db_gen))
    for idx_ct, (m_batch, m_label) in enumerate(db_gen):
        m_batch, m_label = m_batch.to(device), m_label.to(device)
        output = model(m_batch)
        loss = criterion(output, m_label)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        if args.local_rank == 0:
            pbar.set_description('epoch: %d, cce:%.3f'%(epoch, loss))
            pbar.update(1)
        lr_scheduler.step()
    if args.local_rank == 0:
        pbar.close()

def main():
    args = get_args()
    args.model_params['nb_classes'] = 6112

    world_size = args.ngpu
    torch.distributed.init_process_group(
        'nccl',
        init_method='env://',
        world_size=world_size,
        rank=args.local_rank,
    )

    torch.manual_seed(args.seed)
    np.random.seed(args.seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

    cuda = torch.cuda.is_available()
    device = torch.device('cuda' if cuda else 'cpu')

    trainset = Dataset_imgs()

    sampler_devset = torch.utils.data.distributed.DistributedSampler(
        trainset,
        num_replicas=args.ngpu,
        rank=args.local_rank)

    trainset_gen = data.DataLoader(trainset,
                                 batch_size=args.bs,
                                 drop_last=True,
                                 pin_memory=True,
                                 sampler=sampler_devset,
                                 num_workers=args.nb_worker)

    torch.cuda.set_device(args.local_rank)
    model = Model_imgs(args.model_params, args.reduction)
    model.cuda()

    model = torch.nn.SyncBatchNorm.convert_sync_batchnorm(model)
    model = torch.nn.parallel.DistributedDataParallel(
        model,
        device_ids=[args.local_rank],
        output_device=args.local_rank,
        find_unused_parameters=False)

    criterion= nn.CrossEntropyLoss()

    params = [
        {
            'params': [
                param for name, param in model.named_parameters()
                if 'bn' not in name
            ]
        },
        {
            'params': [
                param for name, param in model.named_parameters()
                if 'bn' in name
            ],
            'weight_decay':
                0
        },
    ]
    optimizer = torch.optim.Adam(params,
                                 lr=args.lr,
                                 weight_decay=0.0001,
                                 amsgrad=True)
    lr_scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda=lambda step: keras_lr_decay(step))

    # Train
    for epoch in range(args.epoch):
        sampler_devset.set_epoch(epoch)
        if args.local_rank == 0:
            print('training epoch:', epoch + 1)
        train_model(model=model,
                    db_gen=trainset_gen,
                    args=args,
                    optimizer=optimizer,
                    lr_scheduler=lr_scheduler,
                    criterion=criterion,
                    device=device,
                    epoch=epoch)

if __name__ == '__main__':
    main()

If you need other information, please do not hesitate to leave a message. Thanks in advance!

Context: Are you using the binaries with CUDA11.3 or 11.5? I assume you are not seeing any issues running the same code with a nightly binary on the other GPUs or are you stuck to 1.6.0 in the other setups (this would help identifying if the issue could be in the framework itself or the libraries)?


Question:
I have currently the following network with two losses:

I have a single optimizer and found that multiplying by a scalar one of the losses leads to better results such that:
L_total = lambda L1 + L2
I have the hypothesis that the lambda hyperparameter should only affect the shared representation. Therefore, I would like to “apply” it after both separated networks backpropagation. My questions are:

How to apply the lambda to the loss after each of the single networks’ backpropagation so that only the shared representation is affected by (lambda L1 + L2). Is multiplying the gradients that come from L1 by lambda enough?
Does that make any sense?

Thank you.

Context: @deepfailure Multiplying with a scalar will not affect the overall loss. The only way it could have affected is if you have a backpropagation on the total loss and the lambda scalar is a negative number


Understanding gradient descent in pytorch

Question:
Hi,
I have a doubt regarding gradient descent. The below function tires to approximate the value of w and b. After backpropagating the value for w.grad(w is initialized with 1.0) should have been a negative number but I am receiving high positive number.
A negative grad value indicates that w should be increased while a positive number number indicates that w should be reduced.
Am I missing something? Please help me resolve this query. Thank you.
import torch
x = torch.rand([20, 1], requires_grad=True)
y = 3*x - 2
w = torch.tensor([1.], requires_grad = True)
b = torch.tensor([1.], requires_grad = True)

y_hat = w*x + b

loss = torch.sum((y_hat - y)**2)
print(loss) # tensor(79.0468, grad_fn=<SumBackward0>)

loss.backward()
print(w.grad, b.grad) # tensor([35.0812]) tensor([76.0854])

Context:
If you drive mathematic for grad, you can see that nothing is wrong with the result.
grad_w = (2*(y_hat-y)*(x)).sum()
grad_b = (2*(y_hat-y)*(1)).sum()

import torch
x = torch.rand([20, 1], requires_grad=True)
y = 3*x - 2
w = torch.tensor([1.], requires_grad = True)
b = torch.tensor([1.], requires_grad = True)

y_hat = w*x + b

loss = torch.sum((y_hat - y)**2)
print(loss)

loss.backward()
print(w.grad, b.grad)

grad_w = (2*(y_hat-y)*(x)).sum()
grad_b = (2*(y_hat-y)*(1)).sum()

print(grad_w,grad_b)

output:
tensor(71.8559, grad_fn=)
tensor([37.8953]) tensor([73.1675])
tensor(37.8953, grad_fn=) tensor(73.1675, grad_fn=)


How to confirm parameters of frozen part of network are not being updated?

Question:
I am new to PyTorch. I set the requires_grade for the features extraction layers of vgg16 to false (as I want to freeze these layers for fine tuneing the model) using following code:
for name, param in model.named_parameters():
if param.requires_grad and ‘features’ in name:
param.requires_grad = False
Is it the correct way of freezing some of the network’s layers? Also is there any way to confirm that the weights and biases are not being updated in the frozen part of the network?

Context: You could check the .grad attribute of the frozen parameters after the backward() call and make sure it’s set to None. Additionally, you could also print the parameters before and after the optimizer.step() call to make sure they are not updated.


Torch.sigmoid function gradient issue after first epoch (Trying to backward through the graph a second time)

Question:
Hi, I’m running the following code for an optimization problem. (The loss function here is just a simplified example). The loss depends on the values of the weights tensor, which is passed through a sigmoid layer and division to make sure: (1) Each entry is between 0, 1 (2) The vector sums to 1.
def test_weights(epochs):
  weights = torch.rand(64, requires_grad= True)
  optimizer = torch.optim.SGD([weights], lr=1e-2, momentum=0.9)

  for e in range(epochs):
    optimizer.zero_grad()

    weights = torch.sigmoid(weights.clone())
    weights = (weights / weights.sum()).clone()

    error = weights[1] + weights[2]
    error.backward()
    optimizer.step()

    print(weights[0:5])
  return weights

However, after printing the first epoch, I got the error:

Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.

Is there any way to solve this issue?

Context: Try modifying your loss.backward() call with loss.backward(retain_graph=True)


Question:
I am new for pytorch and I had a few confusions so bear with me. I  know we don’t do back propagation during the evaluation stage and for that we can use torch.no_grad(). However,I was just wondering if it’s a must to use torch.no_grad()? That is, is it okay not to use torch.no_grad() as long as we don’t do loss.backward() and optimizer.step() in evaluation step like the following:
criterion = torch.nn.CrossEntropyLoss()
for epoch in range(n_epochs):
model.train()
for x,target in data:
prediction=model(x)
optimizer.zero_grad()
loss=criterion(prediction,target)
loss.backward()
optimzer.step()

as can see from above, we do back propagation followed by step.
model.eval()
for x,target in data:
prediction = model(x)
loss=criterion(prediction,target)
VS
model.eval()
for x, target in data:
with torch.no_grad():
prediction=model(x)
loss=criterion(prediction,target)
Since we aren’t doing back propagation and updating each parameter of the model, do we necessarily have to write with torch.no_grad() and if so why is it so? thanks for your help in advance.

Context: No, you don’t need to use the no_grad() wrapper during the evaluation phase. While it would reduce the memory usage by avoiding to store intermediate tensors needed to calculate the gradients, it’s a performance improvement and won’t change the functionality otherwise.