Not able to run jit.script() on spatial pyramid pooling model

Question:
I tried spatial pyramid pooling (SPP) in one of my research work, which required to save the model in lite version so that it can be used by android app.
Jit Script:
from torch.utils.mobile_optimizer import optimize_for_mobile
traced_script_module = torch.jit.script(model)
traced_script_module_optimized = optimize_for_mobile(traced_script_module)
traced_script_module_optimized._save_for_lite_interpreter ("Model_PyTorchCNN_forMobileV10_Spatial.ptl")

Ref of SPP: https://github.com/revidee/pytorch-pyramid-pooling
But I am getting the following error:
RuntimeError:

_pad(Tensor input, int[] pad, str mode="constant", float value=0.) -> (Tensor):
Expected a value of type 'float' for argument 'value' but instead found type 'int'.
:
  File "<ipython-input-9-030eeb1a6a5a>", line 47
            assert w_pad1 + w_pad2 == (w_kernel * levels[i] - previous_conv_size[1]) and h_pad1 + h_pad2 == (h_kernel * levels[i] - previous_conv_size[0])

            padded_input = F.pad(input=previous_conv, pad=[w_pad1, w_pad2, h_pad1, h_pad2],
                           ~~~~~ <--- HERE
                                 mode='constant', value=0)
            if mode == "max":
'spatial_pyramid_pool' is being compiled since it was called from 'SpatialPyramidPooling.forward'
  File "<ipython-input-10-52240195b8a3>", line 18
    def forward(self, x):
        return self.spatial_pyramid_pool(x, self.levels, self.mode)
               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE

On the following block of code:
def spatial_pyramid_pool(previous_conv, levels, mode):
        """
        Static Spatial Pyramid Pooling method, which divides the input Tensor vertically and horizontally
        (last 2 dimensions) according to each level in the given levels and pools its value according to the given mode.
        :param previous_conv input tensor of the previous convolutional layer
        :param levels defines the different divisions to be made in the width and height dimension
        :param mode defines the underlying pooling mode to be used, can either be "max" or "avg"
        :returns a tensor vector with shape [batch x 1 x n],
                                            where n: sum(filter_amount*level*level) for each level in levels
                                            which is the concentration of multi-level pooling
        """
        num_sample = previous_conv.size(0)
        previous_conv_size = [int(previous_conv.size(2)), int(previous_conv.size(3))]
        for i in range(len(levels)):
            h_kernel = int(math.ceil(previous_conv_size[0] / levels[i]))
            w_kernel = int(math.ceil(previous_conv_size[1] / levels[i]))
            w_pad1 = int(math.floor((w_kernel * levels[i] - previous_conv_size[1]) / 2))
            w_pad2 = int(math.ceil((w_kernel * levels[i] - previous_conv_size[1]) / 2))
            h_pad1 = int(math.floor((h_kernel * levels[i] - previous_conv_size[0]) / 2))
            h_pad2 = int(math.ceil((h_kernel * levels[i] - previous_conv_size[0]) / 2))
            assert w_pad1 + w_pad2 == (w_kernel * levels[i] - previous_conv_size[1]) and h_pad1 + h_pad2 == (h_kernel * levels[i] - previous_conv_size[0])

            padded_input = F.pad(input=previous_conv, pad=[w_pad1, w_pad2, h_pad1, h_pad2],
                                 mode='constant', value=0.0)
            if mode == "max":
                pool = nn.MaxPool2d((h_kernel, w_kernel), stride=(h_kernel, w_kernel), padding=(0, 0))
            elif mode == "avg":
                pool = nn.AvgPool2d((h_kernel, w_kernel), stride=(h_kernel, w_kernel), padding=(0, 0))
            else:
                raise RuntimeError("Unknown pooling type: %s, please use \"max\" or \"avg\".")
            x = pool(padded_input)
            if i == 0:
                spp = x.view(num_sample, -1)
            else:
                spp = torch.cat((spp, x.view(num_sample, -1)), 1)

        return spp

If I change the value of “value = 0” to “value = 0.0”, I am getting the following error:
RuntimeError:
Arguments for call are not valid.
The following variants are available:

  aten::eq.Tensor(Tensor self, Tensor other) -> (Tensor):
  Expected a value of type 'Tensor' for argument 'other' but instead found type 'str'.

Context:
@ptrblck rblc
Any suggestion?


Can we resume training with only torchscript model?

Question:
I saw some transfer learning Libtorch codes using pretrained torchscript models and all of them is only training additional layers added in Libtorch codes.
(e.g) Transfer-Learning-Dogs-Cats-Libtorch/main.cpp at master · krshrimali/Transfer-Learning-Dogs-Cats-Libtorch · GitHub)
My question is, can we train the torchscript model without defining network model in Libtorch?
I think torchscript model has already model’s weights and structure is similar to torch::nn::Module(actually it is instance of torch::jit::Module object) so I assume it’s not impossible…but still can’t find the way.
Best Regards

Context:
Solved from this issue.
How to train a torch::jit::script::Module? · Issue #28478 · pytorch/pytorch (github.com)


RuntimeError during compilation on different computer

Question:
I have a module that is running fine on one computer, but on another computer it throws this error
  File "/home/guillefix/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
RuntimeError: __nv_nvrtc_builtin_header.h(78048): error: function "operator delete(void *, size_t)" has already been defined

__nv_nvrtc_builtin_header.h(78049): error: function "operator delete[](void *, size_t)" has already been defined

2 errors detected in the compilation of "default_program".

nvrtc compilation failed:

#define NAN __int_as_float(0x7fffffff)
#define POS_INFINITY __int_as_float(0x7f800000)
#define NEG_INFINITY __int_as_float(0xff800000)

template<typename T>
__device__ T maximum(T a, T b) {
  return isnan(a) ? a : (a > b ? a : b);
}

template<typename T>
__device__ T minimum(T a, T b) {
  return isnan(a) ? a : (a < b ? a : b);
}

extern "C" __global__
void fused_tanh_mul_add__14203776399538843293(float* tv_, float* tb0_2, float* tv__, float* tv___, float* tv____, float* aten_log, float* aten_cat) {
{
if ((long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)<8ll ? 1 : 0) {
    aten_cat[(long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)] = (((long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)<4ll ? 1 : 0) ? (__ldg(tv__ + (long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x))) / (1.f / (1.f + (expf(0.f - ((__ldg(tv___ + (long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x))) * (tanhf(__ldg(tv____ + (long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)))) + 2.f)))) + 0.1192029193043709f) - (__ldg(tb0_2 + (long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x))) : __ldg(tv_ + ((long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)) - 4ll));
  }if ((long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)<4ll ? 1 : 0) {
    float v = __ldg(tv___ + (long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x));
    float v_1 = __ldg(tv____ + (long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x));
    aten_log[(long long)(threadIdx.x) + 512ll * (long long)(blockIdx.x)] = logf(1.f / (1.f + (expf(0.f - (v * (tanhf(v_1)) + 2.f)))) + 0.1192029193043709f);
  }}
}

The computer in which it works has cuda 11.4, and torch 1.10.0a0+git36449ea, and the comptuer in which it doesnt work has cuda 11.5 and torch 1.10.1+cu102 (though I also tried with 1.10.0+cu102). So it seems like quite a similar set up. Any idea about why it’s failing?

Context: It worked after updating the Nvidia Toolkit to version 11.6!


Question:
I use gdb to debug pytorch, but cann’t step in func and get detail func body.  And backtrace is discontinuous.

image1812×718 168 KB

Context: To get more information in gdb rebuild PyTorch with debug symbols via DEBUG=1 python setup.py install.


How to correctly use the “pad” function in TorchScript?

Question:
The following code works fine in regular Pytorch
def pad_audio(self,tensor,n_max=50000):

    diff_pad = n_max - len(tensor)

    tensor = F.pad(tensor,(int(diff_pad/2),diff_pad - int(diff_pad/2)),'constant',0)

But when using JIT script, I get the following error:

_pad(Tensor input, int[] pad, str mode="constant", float value=0.) -> (Tensor):
Expected a value of type 'List[int]' for argument 'pad' but instead found type 'Tuple[int, Tensor]'.
:
        diff_pad = n_max - len(tensor)

        tensor = F.pad(tensor,(int(diff_pad/2),diff_pad - int(diff_pad/2)),'constant',0)
                 ~~~~~ <--- HERE

What is the correct way of using pad in TorchScript?

Context:
Make sure to use List[int] as described in the error message. Afterwards, you would run into another type mismatch, since the value is expected as a float and lastly you would have to annotate the inputs since n_max would be expected as a Tensor otherwise.
This should work:
@torch.jit.script
def pad_audio(tensor, n_max=50000):
    # type: (Tensor, int)  -> Tensor
    diff_pad = int(n_max - len(tensor))
    tensor = F.pad(tensor, [int(diff_pad/2), diff_pad - int(diff_pad/2)], 'constant', 0.)
    return tensor


RuntimeError second time running loaded module

Question:
Hello,
I’ve saved a model usint torch.jit.trace, and then loaded it using torch.jit.load.
The first time I run the model it outputs correctly, but the second and subsequent times, it throws this error
RuntimeError                              Traceback (most recent call last)
<ipython-input-25-946d12987c4d> in <module>
----> 1 out = model(inputs)

~/.local/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1100         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1101                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1102             return forward_call(*input, **kwargs)
   1103         # Do not call functions when jit is used
   1104         full_backward_hooks, non_full_backward_hooks = [], []

RuntimeError: _Map_base::at

Any idea what could be causing it?
Checked with torch 1.8.0 and torch 1.10.0. Happens both on CPU and GPU

Context: As the second pass apparently does the optimization, is there a way to disable this optimization, as that’s where it seems to be failing?


Memory leak while exporting torchscript using torch.jit.trace

Question:
I have found there is obvious memory leak when I try to export torchscript multiple times.
Here is the piece of code to reproduce this fault. The process may occupy over 20 GB RAM after exporting resnet50 40 times.
import torch
import torchvision.models as vision_models
from torchvision.models import resnet

resnet50 = vision_models.resnet50()

for i in range(1000):
    jit_resnet50 = torch.jit.trace(resnet50, torch.randn(16, 3, 448, 448))
    print('{} times of jit trace done!'.format(i + 1))
    jit_resnet50 = None

Pytorch version: 1.10.0

Platform: Ubuntu 18.04, with or without CUDA

Context:
Oh oh. I think this is half-expected because of how TorchScript’s caching works. It isn’t a good thing, though, and at least there should be a way to reset things.
Best regards
Thomas


No graph attribute when use torch.jit._overload_method

Question:
PyTorch version: 1.6.0.dev20200407+cu101
When use torch.jit._overload_method to overload functions, it seems that there is no graph attribute in the output torchscript.
class MyModule(nn.Module):
    def __init__(self):
        super().__init__()

    @torch.jit._overload_method  # noqa: F811
    def forward(self, x: int) -> int:  # noqa: F811
        pass

    @torch.jit._overload_method  # noqa: F811
    def forward(self, x: torch.Tensor) -> torch.Tensor:  # noqa: F811
        pass

    def forward(self, x):  # noqa: F811
        original = x

        if isinstance(original, int):
            return original + 2
        else:
            return original.sum()

model = MyModule()
s = torch.jit.script(model)
torch._C._jit_pass_inline(s.graph)  # error happens here

error message:
torch.nn.modules.module.ModuleAttributeError: 'RecursiveScriptModule' object has no attribute 'graph'

Context:
Hi, this is is because there is no single “graph”, there are two separate graphs, one for each overload. I could maybe add an api for something like model.forwards.graphs, or model.forward.graph_for_types(1) to get the graph for a specific overload.
Generally, overloads should be usable and work as you expect when they are contained in other modules but you may run into a little bit of difficulty if you are interacting with them as a top level module. They’re still an internal feature but will be cleaned up and released probably in the next release or the one after.


Type torch::jit::Module converted to torch._C.ScriptModule

Question:
Hi,
I’m trying to develop some torch.jit modules with c++，and then use them in python with pybind.
I found that the orginal method of torch._C.ScriptModule._create_method_from_trace
was created with arg types torch::jit::Module in c++.  But the corresponding method arg type was torch._c.ScriptModule in python.
How dose this conversion happens?
I try to define a method in c++ with arg type torch::jit::Module. When exported to python, it’s arg type still torch::jit::Module.
What’s missed when building the program?

Context:
Build with these cxx flags can fix:

 cmake_cxx_flags: List[str] = []
    for name in ["COMPILER_TYPE", "STDLIB", "BUILD_ABI"]:
        val = getattr(torch._C, f"_PYBIND11_{name}")
        if val is not None:
            cmake_cxx_flags += [f'-DPYBIND11_{name}=\\"{val}\\"']
        print("val: ", val)
        print("cmake_cxx_flags: ", cmake_cxx_flags)


Question:
Hi, guys. I wonder how to call scripts written in cython when creating jit scripts?
Traditionally, I can call cython class directly in python file after compiling those cython modules. And part of my model inference requires those cython scripts for acceleration. I wonder if it is possible to call the functions and classes written in cython when creating jit.script for my model?
I tried directly calling them (just as how I did in the original python script), but it says that there’s missing a .py file (I thought this is as expected because all cython modules are ended with xxx.pyx and xxx.pyd, and they are compiled to xxx.cytree.cpython-38-x86_64-linux-gnu.so on my machine):
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/torch/_utils_internal.py", line 49, in get_source_lines_and_file
    sourcelines, file_lineno = inspect.getsourcelines(obj)
  File "/opt/conda/lib/python3.8/inspect.py", line 967, in getsourcelines
    lines, lnum = findsource(object)
  File "/opt/conda/lib/python3.8/inspect.py", line 790, in findsource
    raise OSError('source code not available')
OSError: source code not available

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "new_convert.py", line 410, in <module>
    tmp=torch.jit.script(LSTMNet(hanabi_config,inverse_transform))
  File "/opt/conda/lib/python3.8/site-packages/torch/jit/_script.py", line 897, in script
    return torch.jit._recursive.create_script_module(
  File "/opt/conda/lib/python3.8/site-packages/torch/jit/_recursive.py", line 352, in create_script_module
    return create_script_module_impl(nn_module, concrete_type, stubs_fn)
  File "/opt/conda/lib/python3.8/site-packages/torch/jit/_recursive.py", line 410, in create_script_module_impl
    create_methods_and_properties_from_stubs(concrete_type, method_stubs, property_stubs)
  File "/opt/conda/lib/python3.8/site-packages/torch/jit/_recursive.py", line 304, in create_methods_and_properties_from_stubs
    concrete_type._create_methods_and_properties(property_defs, property_rcbs, method_defs, method_rcbs, method_defaults)
  File "/opt/conda/lib/python3.8/site-packages/torch/jit/annotations.py", line 76, in get_signature
    source = dedent(''.join(get_source_lines_and_file(fn)[0]))
  File "/opt/conda/lib/python3.8/site-packages/torch/_utils_internal.py", line 56, in get_source_lines_and_file
    raise OSError(msg) from e
OSError: Can't get source for <class 'cytree.Roots'>. TorchScript requires source access in order to carry out compilation, make sure original .py files are available.

I wonder how do you guys integrate cython modules to a jit script? Is re-write the cython module to the traditional python script the only solution?
Thanks!

Context:
You can add @jit.ignore decorated forwarders.
Alternatively, C++ extension ops work with JIT, even supporting gradients, but I don’t know how exactly to register them from python. In C++ it is like:
static auto registry =
torch::RegisterOperators()
.op("namespace::func", &func)

with that you can call torch.ops.namespace.func(…)


Question:
I am using  d2go.
when I convert a mask_rcnn_fbnetv3g_fpn based trained model,got the following error:
CPUAllocator.cpp:76] data. DefaultCPUAllocator: not enough memory: you tried to allocate   xxxx
my convert code is:
patch_d2_meta_arch()
@contextlib.contextmanager
def create_fake_detection_data_loader(height, width, is_train):
with make_temp_directory(“detectron2go_tmp_dataset”) as dataset_dir:  #创建一个临时目录 dataset_dir=detectron2go_tmp_dataset
runner = create_runner(“d2go.runner.GeneralizedRCNNRunner”) 。
cfg = runner.get_default_cfg()
cfg.DATASETS.TRAIN = [“default_dataset_train”]
cfg.DATASETS.TEST = [“default_dataset_test”]
    with make_temp_directory("detectron2go_tmp_dataset") as dataset_dir: #再次创建相同的目录?
        image_dir = os.path.join(dataset_dir, "images")
        os.makedirs(image_dir)
        image_generator = LocalImageGenerator(image_dir, width=width, height=height)

        if is_train:
            with _register_toy_dataset(
                "default_dataset_train", image_generator, num_images=3
            ):
                train_loader = runner.build_detection_train_loader(cfg)
                yield train_loader
        else:
            with _register_toy_dataset(
                "default_dataset_test", image_generator, num_images=3
            ):
                test_loader = runner.build_detection_test_loader(
                    cfg, dataset_name="default_dataset_test"
                )
                yield test_loader

def test_export_torchvision_format():
runner = GeneralizedRCNNRunner()
cfg = runner.get_default_cfg()
cfg.merge_from_file(model_zoo.get_config_file(“mask_rcnn_fbnetv3a_dsmask_C4.yaml”))
cfg.MODEL.WEIGHTS = os.path.join("./output_mask_rcnn_fbnetv3a_dsmask_C4_20211225", “model_0009999.pth”)
cfg.MODEL_EMA.ENABLED = False
cfg.MODEL.DEVICE=“cpu”
cfg.DATASETS.TRAIN = (“infusion_train”,)
cfg.DATASETS.TEST = (“infusion_val”,)
cfg.DATALOADER.NUM_WORKERS = 1
#cfg.INPUT.MAX_SIZE_TEST = 1920
#cfg.INPUT.MAX_SIZE_TRAIN = 1920
#cfg.INPUT.MIN_SIZE_TEST = 1920
#cfg.INPUT.MIN_SIZE_TRAIN = (1920,)
cfg.SOLVER.IMS_PER_BATCH = 1
cfg.SOLVER.STEPS = []  # do not decay learning rate
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 1
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 25

pytorch_model = runner.build_model(cfg, eval_only=True)
pytorch_model.cpu()
#pytorch_model.eval()

from typing import List, Dict
class Wrapper(torch.nn.Module):
    def __init__(self, model):
        super().__init__()
        self.model = model
        coco_idx_list = [0,1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24,25]

        self.coco_idx = torch.tensor(coco_idx_list)

    def forward(self, inputs: List[torch.Tensor]):
        x = inputs[0].unsqueeze(0) * 255
        scale = 320.0 / min(x.shape[-2], x.shape[-1])
        x = torch.nn.functional.interpolate(x, scale_factor=scale, mode="bilinear", align_corners=True, recompute_scale_factor=True)
        out = self.model(x[0])
        res : Dict[str, torch.Tensor] = {}
        res["boxes"] = out[0] / scale
        res["labels"] = torch.index_select(self.coco_idx, 0, out[1])
        res["scores"] = out[2]
        #print("return",inputs,[res])
        return inputs, [res]

size_divisibility = max(pytorch_model.backbone.size_divisibility, 10)
h, w = size_divisibility, size_divisibility * 2
with create_fake_detection_data_loader(h, w, is_train=False) as data_loader:
    predictor_path = convert_and_export_predictor(
        copy.deepcopy(cfg),
        copy.deepcopy(pytorch_model),
        "torchscript_int8@tracing",
        './',
        data_loader,
    )

    orig_model = torch.jit.load(os.path.join(predictor_path, "model.jit"))
    wrapped_model = Wrapper(orig_model)
    # optionally do a forward
    wrapped_model([torch.rand(3, 1920, 1080)])
    scripted_model = torch.jit.script(wrapped_model)
    scripted_model.save("d2go.pt")

if name == ‘main’:
test_export_torchvision_format()

Context: The error message points out that you are running out of RAM. How large is the reported allocation? Does the values fit your expectation? If so, you would need to reduce the memory requirements by e.g. lowering the batch size if possible.


How to script the model and lazily load parameters at the same time?

Question:
Hi, I am working on a tool to make model only load parameters when needed to reduce peak memory (link 1). And I would like to take advantage of the torch.jit optimization but it failed. Here is a reproducible code:
import functools
import torch
import torch.nn as nn
import torch.nn.functional as F  # for torch.jit.script

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(32, 32)
        self.fc2 = nn.Linear(32, 32)

    def forward(self, input):
        out = self.fc1(input)
        out = self.fc2(out)
        return out

def release_weights(module, _input=None, _output=None):
    module.to('cpu')

def load_weights(module, _input=None):
    module.to('cuda')

def lazy_loading(func, module):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        load_weights(module)
        res = func(*args, **kwargs)
        release_weights(module)
        return res
    return wrapper

def main():
    model = Net().eval()
    data = torch.rand(1, 32)

    for module in [model.fc1, model.fc2]:
        # method 1: decorator
        module.forward = lazy_loading(module.forward, module)

        # method 2: hook
        # module.register_forward_hook(release_weights)
        # module.register_forward_pre_hook(load_weights)

    data = data.cuda()

    with torch.no_grad():
        model = torch.jit.script(model)
        output = model(data)

main()

I’ve tried:

For adding decorators (before/after torch.jit.script), the decorator seems to be ignored.
For adding hooks before torch.jit.script, it comes  Type mismatch with hooks.
For adding hooks after torch.jit.script, it raises RuntimeError: register_forward_hook is not supported on ScriptModules.

I wonder is it possible to script the model and lazily load parameters at the same time? (without directly modifying model internal code)
Any feedback would be greatly appreciated. Thanks!
torch version: 1.10.1

Context:
With method 2 + type hints (link 1),
...
def release_weights(module, _input: Tuple[torch.Tensor], _output):
    module.weight.data = module.weight.to('cpu')
...

it comes:
RuntimeError:
Tried to set an attribute: data on a non-class: Tensor:
  File "/home/siahuat0727/test.py", line 22
def release_weights(module, _input: Tuple[torch.Tensor], _output):
    module.weight.data = module.weight.to('cpu')
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE

So the main issue would be: Is it possible to move ScriptModule parameters across devices?
Here’s a similar issue Problem with jit TorchScript while copying data between GRUs · Issue #28267 · pytorch/pytorch (github.com).


Type mismatch with hooks input when using JIT

Question:
Hi,
I’m trying to register hooks in order to get the layers’ activation values in my model.
It does work with normal python runtime (like in this example 1).
However I cannot make it work in JIT:
As questioned here 3 the type of “input” in the hook function is a tuple. And the Jit compiler does not like it:
 Traceback (most recent call last):
  File "main.py", line 22, in <module>
    script_net = torch.jit.script(net)
  File "/home/zodiac/.venv/ia/lib64/python3.7/site-packages/torch/jit/_script.py", line 1258, in script
    obj, torch.jit._recursive.infer_methods_to_compile
  File "/home/zodiac/.venv/ia/lib64/python3.7/site-packages/torch/jit/_recursive.py", line 451, in create_script_module
    return create_script_module_impl(nn_module, concrete_type, stubs_fn)
  File "/home/zodiac/.venv/ia/lib64/python3.7/site-packages/torch/jit/_recursive.py", line 513, in create_script_module_impl
    script_module = torch.jit.RecursiveScriptModule._construct(cpp_module, init_fn)
  File "/home/zodiac/.venv/ia/lib64/python3.7/site-packages/torch/jit/_script.py", line 587, in _construct
    init_fn(script_module)
  File "/home/zodiac/.venv/ia/lib64/python3.7/site-packages/torch/jit/_recursive.py", line 491, in init_fn
    scripted = create_script_module_impl(orig_value, sub_concrete_type, stubs_fn)
  File "/home/zodiac/.venv/ia/lib64/python3.7/site-packages/torch/jit/_recursive.py", line 520, in create_script_module_impl
    create_hooks_from_stubs(concrete_type, hook_stubs, pre_hook_stubs)
  File "/home/zodiac/.venv/ia/lib64/python3.7/site-packages/torch/jit/_recursive.py", line 377, in create_hooks_from_stubs
    concrete_type._create_hooks(hook_defs, hook_rcbs, pre_hook_defs, pre_hook_rcbs)

RuntimeError: Hook 'hook' on module 'Linear' expected the input argument to be typed as a Tuple but found type: 'Tensor' instead.

This error occured while scripting the forward hook 'hook' on module Linear. If you did not want to script this hook remove it from the original NN module before scripting.
This hook was expected to have the following signature: hook(self, input: Tuple[Tensor], output: Tensor).
The type of the output arg is the returned type from either the forward method or the previous hook if it exists.
Note that hooks can return anything, but if the hook is on a submodule the outer module is expecting the same return type as the submodule's forward.

Here’s the minimum code needed to reproduce this issue:
import torch

class NN(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.l1 = torch.nn.Linear(1, 12)

    def forward(self, x):
        return self.l1(x)

def hook(model, input, output):
    pass

net = NN()
net.l1.register_forward_hook(hook)
script_net = torch.jit.script(net)

Any ideas?
I’m on fedora 33, using Python 3.7.12 and Torch 1.10.0
Have a good day!

Context: Same on macos 12.0.1 using python 3.9 and torch 1.10 on rosetta!


How does torch.jit trace submodules with “unsupported” input type

Question:
Hi, I have a model with two submodules A and B.
A should not be traced (control flow depends on the input tensor) and B contains “unsupported” input types.

Tracing is ideal for code that operates only on Tensor s and lists, dictionaries, and tuples of Tensor s. torch.jit.script — PyTorch 1.10.1 documentation

How can I trace only on B? (or trace the whole model but skip submodule A? torch.jit.ignore does not work for this case)
I think it’s technically possible since torch.jit.trace can correctly trace submodules with “unsupported” input types. (see comment1 below)
import torch
import torch.nn as nn

class Foo(nn.Module):
    def __init__(self):
        super().__init__()
        self.identity = nn.Identity()

    def forward(self, x1, x2):
        out = self.identity(x1)
        if x2 is not None:
            out = out + self.identity(x2)
        return out

class Bar(nn.Module):
    def __init__(self):
        super().__init__()
        self.foo1 = Foo()
        self.foo2 = Foo()

    def forward(self, x):
        out = self.foo1(x, x)
        out = out + self.foo2(x, None)
        return out

bar = Bar()
data = torch.ones(42)

# comment1: This is ok
# bar = torch.jit.trace(bar, data)

# comment2: This is ok too
# bar.foo1 = torch.jit.trace(bar.foo1, (data, data))

# comment3: Fails
bar.foo2 = torch.jit.trace(bar.foo2, (data, None))

RuntimeError: Type 'Tuple[Tensor, NoneType]' cannot be traced. Only Tensors and (possibly nested) Lists, Dicts, and Tuples of Tensors can be traced

Any feedback would be greatly appreciated!
torch version: 1.10.1

Context:
Well, I found a solution that worked for me.
We can first torch.jit.script submodule A (maybe combine with torch.jit.ignore to make it scriptable, just to ensure that it will be skipped by torch.jit.trace), and then torch.jit.trace the whole model.
Still looking for a “solution” for the title.


Question:
I want to export LSTM based pytorch model to TorchScript script but I failed because of such error:
RuntimeError:
Arguments for call are not valid.
The following variants are available:

  forward__0(__torch__.torch.nn.modules.rnn.LSTM self, Tensor input, (Tensor, Tensor)? hx=None) -> ((Tensor, (Tensor, Tensor))):
  Expected a value of type 'Tensor' for argument 'input' but instead found type '__torch__.torch.nn.utils.rnn.PackedSequence'.

  forward__1(__torch__.torch.nn.modules.rnn.LSTM self, __torch__.torch.nn.utils.rnn.PackedSequence input, (Tensor, Tensor)? hx=None) -> ((__torch__.torch.nn.utils.rnn.PackedSequence, (Tensor, Tensor))):
  Expected a value of type 'Optional[Tuple[Tensor, Tensor]]' for argument 'hx' but instead found type 'Tensor (inferred)'.
  Inferred the value for argument 'hx' to be of type 'Tensor' because it was not annotated with an explicit type.

The original call is:
  File "", line 24
                                 enforce_sorted=False)

        output, feat = self.lstm(x, feat)
                       ~~~~~~~~~ <--- HERE
        output = pad_packed_sequence(output,
                                     batch_first=True,

Script to reproduce:
import torch
from torch import nn
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence

class LSTMBlock(nn.Module):
    def __init__(self, input_size, hidden_size):
        super().__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size

        self.lstm = nn.LSTM(input_size,
                            hidden_size,
                            batch_first=True,
                            bidirectional=False)

    def forward(self, x, input_lengths, feat=None):
        total_length = x.shape[1]
        x = pack_padded_sequence(x,
                                 input_lengths,
                                 batch_first=True,
                                 enforce_sorted=False)

        output, feat = self.lstm(x, feat)
        output = pad_packed_sequence(output,
                                     batch_first=True,
                                     total_length=total_length)[0]

        return output, input_lengths, feat

model = LSTMBlock(100, 100)
script = torch.jit.script(model)

Is there a way how to handle models trained with packed_sequence?

Context: Did you solve this issue? I met a very similar issue when I was trying to convert LSTM pytorch model to TorchScript.


How can i get access to first and second Tensor from Tuple, returned from forward() method in Libtorch C\C++

Question:
Hello Team, it is my code.
I’m using Jit generated model with Libtorch in C\C++ in Ubuntu 18.04, with CUDA 10.2
I get correct result form my first call     result  = mE.forward(x);
    torch::Tensor tensor_image = ...
    torch::IValue result;   // it is standard return of forward() in JIT traced model

    std::vector<torch::jit::IValue> x;
    x.push_back(tensor_image);
    result  = mE.forward(x);
    std::cout << sp_and_gl_output_CUDA << std::endl;
    std::vector<torch::IValue> in_to_G;
    in_to_G.push_back(result);
    img_tensor = mG.forward(in_to_G);

On the image print of result  variable, returned by forward() method.
  std::cout << sp_and_gl_output_CUDA << std::endl;
The result contained, two Tensors, but i can’t to pass it in next forward(), in right form.

изображение1757×1174 164 KB

The result of execution:
mG.forward load spatial and global:

forward() Expected a value of type 'Tensor' for argument 'v' but instead found type 'Tuple[Tensor, Tensor]'.
Position: 1
Declaration: forward(__torch__.models.networks.generator.StyleGAN2ResnetGenerator self, Tensor v, Tensor v0) -> (Tensor)
Exception raised from checkArg at ../aten/src/ATen/core/function_schema_inl.h:184 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f1c5d0290db in /home/interceptor/Документы/Git_Medium_repo/Binary_search_engine_CUDA/torch_to_cpp/libtorch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f1c5d024d2e in /home/interceptor/Документы/Git_Medium_repo/Binary_search_engine_CUDA/torch_to_cpp/libtorch/lib/libc10.so)
frame #2: <unknown function> + 0x128d542 (0x7f1c473af542 in /home/interceptor/Документы/Git_Medium_repo/Binary_search_engine_CUDA/torch_to_cpp/libtorch/lib/libtorch_cpu.so)
frame #3: <unknown function> + 0x12918c1 (0x7f1c473b38c1 in /home/interceptor/Документы/Git_Medium_repo/Binary_search_engine_CUDA/torch_to_cpp/libtorch/lib/libtorch_cpu.so)
frame #4: torch::jit::GraphFunction::operator()(std::vector<c10::IValue, std::allocator<c10::IValue> >, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, c10::IValue, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, c10::IValue> > > const&) + 0x2d (0x7f1c49c97b2d in /home/interceptor/Документы/Git_Medium_repo/Binary_search_engine_CUDA/torch_to_cpp/libtorch/lib/libtorch_cpu.so)
frame #5: torch::jit::Method::operator()(std::vector<c10::IValue, std::allocator<c10::IValue> >, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, c10::IValue, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, c10::IValue> > > const&) const + 0x175 (0x7f1c49ca7235 in /home/interceptor/Документы/Git_Medium_repo/Binary_search_engine_CUDA/torch_to_cpp/libtorch/lib/libtorch_cpu.so)
frame #6: <unknown function> + 0x6266 (0x564143038266 in ./particle_cuda_test)
frame #7: <unknown function> + 0x47af (0x5641430367af in ./particle_cuda_test)
frame #8: __libc_start_main + 0xe7 (0x7f1bf9682bf7 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: <unknown function> + 0x4daa (0x564143036daa in ./particle_cuda_test)

Main module:  torch_and_cuda_call.cpp  completed.
ok!

Context:
Seems it WORKED
    auto output1_sp = sp_and_gl_output_CUDA.toTuple()->elements()[0].toTensor();
    auto output2_gl = sp_and_gl_output_CUDA.toTuple()->elements()[1].toTensor();
    //-------------------------------------------------------------------------------------------------------------------
    in_to_G.push_back(output1_sp.to(at::kCUDA));
    in_to_G.push_back(output2_gl.to(at::kCUDA));
    //-------------------------------------------------------------------------------------------------------------------
    img_tensor = mG.forward(in_to_G);


Question:
Hello,
While trying to export a ppm_deepsup model, I got this error:
RuntimeError: Expected 4-dimensional input for 4-dimensional weight [512, 2048, 1, 1], but got 3-dimensional input of size [2048, 1, 1] instead.
The PPM_DEEPSUP is defined as follwos:
class PPMDeepsup(nn.Module):
def init(self, num_class=150, fc_dim=4096,
use_softmax=False, pool_scales=(1, 2, 3, 6)):
super(PPMDeepsup, self).init()
self.use_softmax = use_softmax
self.ppm = []
for scale in pool_scales:
self.ppm.append(nn.Sequential(
nn.AdaptiveAvgPool2d(scale),
nn.Conv2d(fc_dim, 512, kernel_size=1, bias=False),
BatchNorm2d(512),
nn.ReLU(inplace=True)
))
self.ppm = nn.ModuleList(self.ppm)
self.cbr_deepsup = conv3x3_bn_relu(fc_dim // 2, fc_dim // 4, 1)
self.conv_last = nn.Sequential(
nn.Conv2d(fc_dim+len(pool_scales)*512, 512,
kernel_size=3, padding=1, bias=False),
BatchNorm2d(512),
nn.ReLU(inplace=True),
nn.Dropout2d(0.1),
nn.Conv2d(512, num_class, kernel_size=1)
)
self.conv_last_deepsup = nn.Conv2d(fc_dim // 4, num_class, 1, 1, 0)
self.dropout_deepsup = nn.Dropout2d(0.1)
def forward(self, conv_out, segSize=None):
conv5 = conv_out[-1]
input_size = conv5.size()
ppm_out = [conv5]
for pool_scale in self.ppm:
ppm_out.append(nn.functional.interpolate(
pool_scale(conv5),
(input_size[2], input_size[3]),
mode=‘bilinear’, align_corners=False))
ppm_out = torch.cat(ppm_out, 1)
x = self.conv_last(ppm_out)
if self.use_softmax:  # is True during inference
x = nn.functional.interpolate(
x, size=segSize, mode=‘bilinear’, align_corners=False)
x = nn.functional.softmax(x, dim=1)
return x
# deep sup
conv4 = conv_out[-2]
_ = self.cbr_deepsup(conv4)
_ = self.dropout_deepsup()
_ = self.conv_last_deepsup()
x = nn.functional.log_softmax(x, dim=1)
_ = nn.functional.log_softmax(_, dim=1)
return (x, _)
and the export script is as follows :
model_decoder = ModelBuilder.build_decoder(“ppm_deepsup”, weights=checkpoint_decoder, use_softmax=True, num_class=150, fc_dim=2048)
model_decoder.load_state_dict(model_decoder_checkpoints, strict=False)
model_decoder.to(device)
model_decoder.eval()
example_enc = torch.zeros(1, 3, 450, 450).to(device)# torch.zeros(1, 3, 2048, 224,224).to(device)
example_dec = torch.zeros(1, 3, 224,224).to(device)
traced_encoder = torch.jit.trace(model_encoder, example_enc, strict=False)
traced_decoder_script = torch.jit.trace(model_decoder, traced_encoder(example_enc), example_dec, strict=True)#, example
Any help please to solve this issue ?
Thanks

Context: your example tensors are 4d, it is inconsistent with 3d inputs to the model (no batch dim perhaps?)


Failed JIT Trace for Torch Mobile iOS (Size not specified in nn.interpolate?) (Segmentation)

Question:
I’m attempting to JIT trace a semantic segmentation model based on a Resnet50 dilated encoder and an accompanying PPMDeepsup decoder to use in iOS/torch mobile.
When running the following to JIT trace my segmentation module with a random image input of 224x224:
input_tensor = torch.rand(1, 3, 224, 224)
model.eval()
script_model = torch.jit.trace(model, input_tensor)

I get the  the following trace. Apparently when my Forward function calls nn.interpolate() it claims that the size variable (final image segmentation size of 224x224) is not provided (found None) even though as seen in the trace below I’ve even tried hard coding the necessary tuple of (224, 224). Why am I still receiving this exception?
Trace
~/thd-visual-ai/segmentation/semantic-segmentation-pytorch/models/models.py in forward(self, conv_out, segSize)
    481         print(segSize)
    482         if self.use_softmax:  # is True during inference
--> 483             x = nn.functional.interpolate(x, size=(224, 224), mode='bilinear', align_corners=False)
    484             x = nn.functional.softmax(x, dim=1)
    485             return x

~/Desktop/Projects/thdEnv/lib/python3.6/site-packages/torch/nn/functional.py in interpolate(input, size, scale_factor, mode, align_corners)
   2516         raise NotImplementedError("Got 4D input, but linear mode needs 3D input")
   2517     elif input.dim() == 4 and mode == 'bilinear':
-> 2518         return torch._C._nn.upsample_bilinear2d(input, _output_size(2), align_corners)
   2519     elif input.dim() == 4 and mode == 'trilinear':
   2520         raise NotImplementedError("Got 4D input, but trilinear mode needs 5D input")

~/Desktop/Projects/thdEnv/lib/python3.6/site-packages/torch/nn/functional.py in _output_size(dim)
   2470
   2471     def _output_size(dim):
-> 2472         _check_size_scale_factor(dim)
   2473         if size is not None:
   2474             return size

~/Desktop/Projects/thdEnv/lib/python3.6/site-packages/torch/nn/functional.py in _check_size_scale_factor(dim)
   2461     def _check_size_scale_factor(dim):
   2462         if size is None and scale_factor is None:
-> 2463             raise ValueError('either size or scale_factor should be defined')
   2464         if size is not None and scale_factor is not None:
   2465             raise ValueError('only one of size or scale_factor should be defined')

ValueError: either size or scale_factor should be defined

I’ve defined size at line 483 hardcoded to try to fix this and it’s still giving none. I will attach the model.summary() here of my semantic segmentation model to help.
Sequential(
  (0): ResnetDilated(
    (conv1): Conv2d(3, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
    (bn1): SynchronizedBatchNorm2d(64, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
    (relu1): ReLU(inplace=True)
    (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (bn2): SynchronizedBatchNorm2d(64, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
    (relu2): ReLU(inplace=True)
    (conv3): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (bn3): SynchronizedBatchNorm2d(128, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
    (relu3): ReLU(inplace=True)
    (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
    (layer1): Sequential(
      (0): Bottleneck(
        (conv1): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): SynchronizedBatchNorm2d(64, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): SynchronizedBatchNorm2d(64, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (downsample): Sequential(
          (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (1): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        )
      )
      (1): Bottleneck(
        (conv1): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): SynchronizedBatchNorm2d(64, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): SynchronizedBatchNorm2d(64, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (2): Bottleneck(
        (conv1): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): SynchronizedBatchNorm2d(64, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): SynchronizedBatchNorm2d(64, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
    )
    (layer2): Sequential(
      (0): Bottleneck(
        (conv1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): SynchronizedBatchNorm2d(128, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (bn2): SynchronizedBatchNorm2d(128, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): SynchronizedBatchNorm2d(512, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (downsample): Sequential(
          (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): SynchronizedBatchNorm2d(512, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        )
      )
      (1): Bottleneck(
        (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): SynchronizedBatchNorm2d(128, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): SynchronizedBatchNorm2d(128, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): SynchronizedBatchNorm2d(512, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (2): Bottleneck(
        (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): SynchronizedBatchNorm2d(128, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): SynchronizedBatchNorm2d(128, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): SynchronizedBatchNorm2d(512, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (3): Bottleneck(
        (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): SynchronizedBatchNorm2d(128, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): SynchronizedBatchNorm2d(128, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): SynchronizedBatchNorm2d(512, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
    )
    (layer3): Sequential(
      (0): Bottleneck(
        (conv1): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): SynchronizedBatchNorm2d(1024, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (downsample): Sequential(
          (0): Conv2d(512, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (1): SynchronizedBatchNorm2d(1024, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        )
      )
      (1): Bottleneck(
        (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
        (bn2): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): SynchronizedBatchNorm2d(1024, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (2): Bottleneck(
        (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
        (bn2): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): SynchronizedBatchNorm2d(1024, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (3): Bottleneck(
        (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
        (bn2): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): SynchronizedBatchNorm2d(1024, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (4): Bottleneck(
        (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
        (bn2): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): SynchronizedBatchNorm2d(1024, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (5): Bottleneck(
        (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
        (bn2): SynchronizedBatchNorm2d(256, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): SynchronizedBatchNorm2d(1024, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
    )
    (layer4): Sequential(
      (0): Bottleneck(
        (conv1): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): SynchronizedBatchNorm2d(512, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
        (bn2): SynchronizedBatchNorm2d(512, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): SynchronizedBatchNorm2d(2048, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (downsample): Sequential(
          (0): Conv2d(1024, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (1): SynchronizedBatchNorm2d(2048, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        )
      )
      (1): Bottleneck(
        (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): SynchronizedBatchNorm2d(512, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(4, 4), dilation=(4, 4), bias=False)
        (bn2): SynchronizedBatchNorm2d(512, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): SynchronizedBatchNorm2d(2048, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (2): Bottleneck(
        (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): SynchronizedBatchNorm2d(512, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(4, 4), dilation=(4, 4), bias=False)
        (bn2): SynchronizedBatchNorm2d(512, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): SynchronizedBatchNorm2d(2048, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
    )
  )
  (1): PPMDeepsup(
    (ppm): ModuleList(
      (0): Sequential(
        (0): AdaptiveAvgPool2d(output_size=1)
        (1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (2): SynchronizedBatchNorm2d(512, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (3): ReLU(inplace=True)
      )
      (1): Sequential(
        (0): AdaptiveAvgPool2d(output_size=2)
        (1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (2): SynchronizedBatchNorm2d(512, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (3): ReLU(inplace=True)
      )
      (2): Sequential(
        (0): AdaptiveAvgPool2d(output_size=3)
        (1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (2): SynchronizedBatchNorm2d(512, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (3): ReLU(inplace=True)
      )
      (3): Sequential(
        (0): AdaptiveAvgPool2d(output_size=6)
        (1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (2): SynchronizedBatchNorm2d(512, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
        (3): ReLU(inplace=True)
      )
    )
    (cbr_deepsup): Sequential(
      (0): Conv2d(1024, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (1): SynchronizedBatchNorm2d(512, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
      (2): ReLU(inplace=True)
    )
    (conv_last): Sequential(
      (0): Conv2d(4096, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (1): SynchronizedBatchNorm2d(512, eps=1e-05, momentum=0.001, affine=True, track_running_stats=True)
      (2): ReLU(inplace=True)
      (3): Dropout2d(p=0.1, inplace=False)
      (4): Conv2d(512, 150, kernel_size=(1, 1), stride=(1, 1))
    )
    (conv_last_deepsup): Conv2d(512, 150, kernel_size=(1, 1), stride=(1, 1))
    (dropout_deepsup): Dropout2d(p=0.1, inplace=False)
  )
  (2): NLLLoss()
)

I appreciate any help!

Context:
Can you share the model itself? A simple test around interpolate (using the nightly version of PyTorch) seems to indicate that it works
def test_interpolate(x):
    return nn.functional.interpolate(x, size=(224, 224), mode='bilinear', align_corners=False)

# this is successful
x = torch.jit.trace(test_interpolate, (torch.randn(1, 3, 224, 224)))
print(x.graph)

If the bug persists, you might have better luck using torch.jit.script around the area that calls interpolate.


[TorchScript] fail to convert list to tuple

Question:
Hi, in my torch.jit.script I would like to convert a list to a tuple, but it seems not to be the easy way.
The size of the list is fixed - although not in a way that jit prefers:  which seems to be the problem
On Python 3.8 with PyTorch 1.9.1
result = []
val = torch.full((1,1),1.)
result.append(val) # size() = 1
return tuple(result)

… fails with Runtimeerror: cannot statically infer the expected size of a list in this context:
same as
val = torch.full((1,1),1.)
result:List[Tensor] = val * 1
return tuple(result)

…fails with Runtimeerror: cannot statically infer the expected size of a list in this context:
Can someone please tell me how to do this in torch.jit.script (not trace!)
Thx

Context:
as error suggests, tuple sizes are static(hardcoded) when “compiled”, so you have to do something like
return (result[0],result[1])
and there may be a need to annotate this as Tuple[Tensor,Tensor] in some contexts


About the reinforcement-learning category

Question: A section to discuss RL implementations, research, problems

Context: The reinforce.py and actor-critic.py examples do not really converge for a running_reward > 200 for me. Did anyone get it to work? I found that the running reward reached 199 quite often and then the rewards start to decrease. Does anyone have a similar experience?


Inverting Gradients - Gradient of critic network output wrt action

Question:
Hello there,
I want to implement the inverting gradients as shown in the paper " Deep Reinforcement Learning in Parametrized Action Space". But firstly I need to have gradient of the critic network’s output ( Q(s,a)  ) with respect to the action ( dQ(s,a) / da ) which is (6) equation in the paper.
I can write the loss as,
loss = value_net(state, policy_net(state))
But how can I differentiate this network with only respect to the action instead state ?

Context: @frknayk I am trying to implement this paper too and having trouble implement its gradient inversion. Have you figured it out?


[resolved] Actor Critic with a large amount of possible actions

Question:
I’m working on a hexapod (18x servos) that teaches itself how to walk. I’ve gotten pretty familiar with actor critic implementations and how it works, but just about everything I’ve seen applies to a situation with a small set of possible actions (ie. left, right, forward). In my situation, there are 18 servos with at least 100 different possible values for each, so that would be a massive output on the network (10^36).
My ideal output would be just one vector of 18 servo values. I know that the probability distribution of the actions is crucial, but what if I were to remove it? What if I took out the softmax from the last layer and just use the 18x output values? Would this work? If not, what other solutions could I try? Should I move to a different algo?

Context: could you first assign say 50 specific types movements for the hexapod then use that as possible actions


Torch Parallel Layers Visualization in wandb

Question:
Hi everyone,
I created a dynamic actor-critic module deriving from nn.Module. The module is made up of 3 submodules: a shared network made up of a number of convolutional layers and 2 independent parts made up of fc layers that receive the flattened output of the previous module as input. The code that I used is the following:
class Conv3DModelFree(nn.Module):
    def __init__(self, in_shape, num_actions, **kwargs):
        fc_layers = kwargs["fc_layers"]
        super(Conv3DModelFree, self).__init__(
            num_actions, features_out=fc_layers[-1])
        # super().__init__()
        conv_layers = kwargs["conv_layers"]

        self.shared_layers = []
        self.num_actions = num_actions
        self.in_shape = in_shape
        self.num_channels = in_shape[0]
        self.num_frames = kwargs["num_frames"]
        next_inp = None
        # =============================================================================
        # FEATURE EXTRACTOR SUBMODULE
        # =============================================================================
        for i, cnn in enumerate(conv_layers):
            if i == 0:
                self.shared_layers.append(nn.Conv3d(
                    self.num_channels, cnn[0], kernel_size=cnn[1], stride=cnn[2]))
                self.shared_layers.append(nn.LeakyReLU())
            else:
                self.shared_layers.append(nn.Conv3d(
                    next_inp, cnn[0], kernel_size=cnn[1], stride=cnn[2]))
                self.shared_layers.append(nn.LeakyReLU())
            next_inp = cnn[0]

        for i, layer in enumerate(self.shared_layers):
            if i == 0:
                fake_inp = torch.zeros(
                    [1, self.num_channels, self.num_frames, *self.in_shape[1:]])
                fake_inp = self.shared_layers[i](fake_inp)
            else:
                fake_inp = self.shared_layers[i](fake_inp)
        next_inp = fake_inp.view(1, -1).size(1)

        # =============================================================================
        # ACTOR AND CRITIC SUBMOODULES
        # =============================================================================
        self.actor_subnet = []
        self.critic_subnet = []
        for i, fc in enumerate(fc_layers):
            if i == 0:
                self.shared_layers.append(nn.Linear(next_inp, fc))
                self.shared_layers.append(nn.LeakyReLU())
            else:
                # Separate submodules for the actor and the critic
                self.actor_subnet.append(nn.Linear(next_inp, fc))
                self.critic_subnet.append(nn.Linear(next_inp, fc))
                self.actor_subnet.append(nn.LeakyReLU())
                self.critic_subnet.append(nn.LeakyReLU())
            next_inp = fc
def forward(self, input)
        for i, layer in enumerate(self.shared_layers[:-2]):
            if i == 0:
                x = self.shared_layers[i](input)
            else:
                x = self.shared_layers[i](x)
        # last 2 shared layers requires a reshape of the input
        x = x.view(x.shape[0], -1)
        x = self.shared_layers[-2](x)
        x = self.shared_layers[-1](x)

        action_logits = None
        for i, layer in enumerate(self.actor_subnet):
            if i == 0:
                action_logits = self.actor_subnet[i](x)
            else:
                action_logits = self.actor_subnet[i](action_logits)
        action_logits = self.actor(action_logits)

        value = None
        for i, layer in enumerate(self.critic_subnet):
            if i == 0:
                value = self.critic_subnet[i](x)
            else:
                value = self.critic_subnet[i](value)
        value = self.critic(value)

        return action_logits, value

When I use wandb.watch on this class, the graphs do not appear to me probably because wandb cannot identifies the submodules of the network that are included in a list. I tested a different solution, trying to wrap the various pieces in some nn.Sequential modules as follows:
class Conv3DModelFree(OnPolicy):
    def __init__(self, in_shape, num_actions, **kwargs):
        fc_layers = kwargs["fc_layers"]
        super(Conv3DModelFree, self).__init__(
            num_actions, features_out=fc_layers[-1])
        # super().__init__()
        conv_layers = kwargs["conv_layers"]

        shared_layers = OrderedDict()
        self.num_actions = num_actions
        self.in_shape = in_shape
        self.num_channels = in_shape[0]
        self.num_frames = kwargs["num_frames"]
        next_inp = None
        # =============================================================================
        # FEATURE EXTRACTOR SUBMODULE
        # =============================================================================
        for i, cnn in enumerate(conv_layers):
            if i == 0:
                shared_layers["conv_0"] = nn.Conv3d(
                    self.num_channels, cnn[0], kernel_size=cnn[1], stride=cnn[2])
                shared_layers["activ_0"] = nn.LeakyReLU()
            else:
                shared_layers["conv_"+str(i)] = nn.Conv3d(
                    next_inp, cnn[0], kernel_size=cnn[1], stride=cnn[2])
                shared_layers["activ_" + str(i)] = nn.LeakyReLU()
            next_inp = cnn[0]

        # flatten the output starting from dim=1 by default
        shared_layers["flatten"] = nn.Flatten()

        for i, layer in enumerate(shared_layers):
            if i == 0:
                fake_inp = torch.zeros(
                    [1, self.num_channels, self.num_frames, *self.in_shape[1:]])
                fake_inp = shared_layers[layer](fake_inp)
            else:
                fake_inp = shared_layers[layer](fake_inp)
        next_inp = fake_inp.view(1, -1).size(1)

        # =============================================================================
        # ACTOR AND CRITIC SUBMODULES
        # =============================================================================
        actor_subnet = OrderedDict()
        critic_subnet = OrderedDict()
        for i, fc in enumerate(fc_layers):
            if i == 0:
                shared_layers["fc_0"] = nn.Linear(next_inp, fc)
                shared_layers["fc_activ"] = nn.LeakyReLU()
            else:
                # Separate submodules for the actor and the critic
                actor_subnet["actor_fc_"+str(i)] = nn.Linear(next_inp, fc)
                critic_subnet["critic_fc_" +
                              str(i)] = nn.Linear(next_inp, fc)
                actor_subnet["actor_activ_"+str(i)] = nn.LeakyReLU()
                critic_subnet["critic_activ_"+str(i)] = nn.LeakyReLU()
            next_inp = fc
        actor_subnet["actor_out"] = nn.Linear(next_inp, self.num_actions)
        critic_subnet["critic_out"] = nn.Linear(next_inp, 1)

        self.shared_network = nn.Sequential(shared_layers)
        self.actor = nn.Sequential(actor_subnet)
        self.critic = nn.Sequential(critic_subnet)

    def forward(self, input):

        shared_net_output = self.shared_network(input)
        action_logits = self.actor(shared_net_output)
        value = self.critic(shared_net_output)

        return action_logits, value

Using this different implementation, I can visualize the graphs but I get completely different results and extremely high loss values. Does anyone know a correct way to define this architecture so that it is viewable on wandb?

Context:
First code
The graph cannot be visualized because the layers are not in nn container.
You should put them into a container like below,
self.nn_shared_layers = nn.ModuleList(*self.shared_layers)
self.nn_actor_subnet = nn.ModuleList(*self.actor_subnet)
self.nn_critic_subnet = nn.ModuleList(*self.critic_subnet)

I recommend you to use nn.Sequential instead of nn.ModuleList.
Except the shared_layers, your forward function is designed in a serial manner. In short,
You can rewrite your forward function like
# In __init__
self.nn_actor_subnet = nn.Sequential(*self.actor_subnet)
# In forward
action_logits = self.nn_actor_subnet(x)

Second Code
I guess your code is well modified to visualize the graph but the reason that your result seems weird comes from not using activation functions.
I don’t know how small your learning rate and which optimizer you use. But if the learning rate is not that big such as 1, the problem is usually mis-designed network.

class Conv3DModelFree(nn.Module):
    def __init__(self, in_shape, num_actions, **kwargs):
        fc_layers = kwargs["fc_layers"]
        super(Conv3DModelFree, self).__init__(
            num_actions, features_out=fc_layers[-1])
        # super().__init__()
        conv_layers = kwargs["conv_layers"]

        self.shared_layers = []
        self.num_actions = num_actions
        self.in_shape = in_shape
        self.num_channels = in_shape[0]
        self.num_frames = kwargs["num_frames"]
        next_inp = None
        # =============================================================================
        # FEATURE EXTRACTOR SUBMODULE
        # =============================================================================
        for i, cnn in enumerate(conv_layers):
            if i == 0:
                self.shared_layers.append(nn.Conv3d(
                    self.num_channels, cnn[0], kernel_size=cnn[1], stride=cnn[2]))
                self.shared_layers.append(nn.LeakyReLU())
            else:
                self.shared_layers.append(nn.Conv3d(
                    next_inp, cnn[0], kernel_size=cnn[1], stride=cnn[2]))
                self.shared_layers.append(nn.LeakyReLU())

            if i == len(conv_layers) - 3:
                self.shared_layers.append(nn.Flatten(start_dim=1))

            next_inp = cnn[0]

        for i, layer in enumerate(self.shared_layers):
            if i == 0:
                fake_inp = torch.zeros(
                    [1, self.num_channels, self.num_frames, *self.in_shape[1:]])
                fake_inp = self.shared_layers[i](fake_inp)
            else:
                fake_inp = self.shared_layers[i](fake_inp)
        next_inp = fake_inp.view(1, -1).size(1)

        # =============================================================================
        # ACTOR AND CRITIC SUBMOODULES
        # =============================================================================
        self.actor_subnet = []
        self.critic_subnet = []
        for i, fc in enumerate(fc_layers):
            if i == 0:
                self.shared_layers.append(nn.Linear(next_inp, fc))
                self.shared_layers.append(nn.LeakyReLU())
            else:
                # Separate submodules for the actor and the critic
                self.actor_subnet.append(nn.Linear(next_inp, fc))
                self.critic_subnet.append(nn.Linear(next_inp, fc))
                self.actor_subnet.append(nn.LeakyReLU())
                self.critic_subnet.append(nn.LeakyReLU())
            next_inp = fc

        self.nn_shared_layers = nn.Sequential(*self.shared_layers)
        self.nn_actor_subnet = nn.Sequential(*self.actor_subnet)
        self.nn_critic_subnet = nn.Sequential(*self.critic_subnet)

    def forward(self, input)
        x = self.nn_shared_layers(input)
        action_logits = self.nn_actor_subnet(x)
        value = self.nn_critic_subnet(x)

        return action_logits, value


Question:
Hi,
Iam new to this forum and to pytorch.
Could someone help me with that error
RuntimeError: invalid argument 4: Index tensor must have same dimensions as input tensor at
def learn(self, batch, gamma):
“”"Update value parameters using given batch of experience tuples.
    Params
    ======
        experiences (Tuple[torch.Variable]): tuple of (s, a, r, s', done) tuples
        gamma (float): discount factor
    """

    #states, actions, rewards, next_states, dones = experiences
    states = np.array([each[0][0] for each in batch], ndmin=3)
    actions = np.array([each[0][1] for each in batch])
    rewards = np.array([each[0][2] for each in batch])
    next_states = np.array([each[0][3] for each in batch], ndmin=3)

    dones = np.array([each[0][4] for each in batch])
    #dones  = np.array([map(lambda x: 1 if x else 0, dones)],dtype=np.float16)
    dones = dones.astype(np.int16)
    states = torch.from_numpy(states).float().cuda()
    next_states = torch.from_numpy(next_states).float().cuda()
    rewards = torch.from_numpy(rewards).float().cuda()
    actions = torch.from_numpy(actions).long().cuda()
    dones = torch.from_numpy(dones).float().cuda()
    #next_states = torch.from_numpy(next_states).float().unsqueeze(0).cuda()
    #print(states.shape)
    states = states.view(32,8)
    #print(states.shape)
    next_states = next_states.view(32,8)
    ## TODO: compute and minimize the loss
    "*** YOUR CODE HERE ***"
    Q_targets_next = self.qnetwork_target(next_states).detach().max(1)[0].unsqueeze(1)
    print(rewards.shape)
    print(Q_targets_next)
    Q_targets =  rewards + (gamma * Q_targets_next * (1-dones))

    Q_expected = self.qnetwork_local(states).gather(1, actions)

    loss = F.mse_loss(Q_expected, Q_targets)
    self.optimizer.zero_grad()
    loss.backward()
    self.optimizer.step()

    # ------------------- update target network ------------------- #
    self.soft_update(self.qnetwork_local, self.qnetwork_target, self.tau)

I don’t know if this gather is correct here found the code online and it worked
now I try to modify to my problem
Thank you very much for your help
If you need more information let me know
Best Regards
Chris

Context:
Which line throws this error?
Could you post the shapes and if necessary the values of all used tensors in this operation?


Using backward of function for multiple networks

Question:
Hi, I have implemented an actor-critic method with linear networks on both actor and critic also I am using a custom function for loss calculation which is loss function used in PPO. The reward function of environment is different from the loss function of networks.
This is the loss calculation:
 dist = self.actor(states)
critic_value = self.critic(states)
critic_value = torch.squeeze(critic_value)

new_probs = dist.log_prob(actions)
prob_ratio = new_probs.exp() / old_probs.exp()
 weighted_probs = advantage[batch.item()] * prob_ratio
weighted_clipped_probs = torch.clamp(prob_ratio, 1-self.policy_clip,
                                                   1+self.policy_clip)*advantage[batch.item()]
actor_loss = -torch.min(weighted_probs,
                                weighted_clipped_probs).mean()

returns = advantage[batch.item()] + values[batch.item()]
critic_loss = (returns-critic_value)**2
critic_loss = critic_loss.mean()

total_loss = actor_loss + 0.5*critic_loss

and then model omtimization:
self.actor.optimizer.zero_grad()
 self.critic.optimizer.zero_grad()
total_loss.backward()
self.actor.optimizer.step()
self.critic.optimizer.step()

Now my questions:

I want to know is it correct to use backward in this way?
Is the backward now returning grad for both actor and critic networks and the both networks are optimizing?
If I add a feature extractor network before inputting the features to network, and input the extracted features to the actor and critic, will this backward works for feature extractor network?

Context:
Yes, calling total_loss.backward() looks correct. I don’t know if the implemented training logic fits your use case, as I’m not familiar with it.

Yes, as long as no tensors are (accidentally) detached. To verify it, check the .grad attributes of the parameters of both models before and after the first backward() call. Before the call, they should be set to None, afterwards to a valid gradient tensor.

Yes and the same applies as in 2. If you are not detaching the tensors, the feature extractor would also get gradients.


Question:
Hi
I encountered this problem and haven’t been able to sort it out for a long while now. Everything works fine until the backward propagation.
it gives me the error message:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 6]], which is output 0 of SliceBackward, is at version 30; expected version 27 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
Heres the code:
#running [1,6] vector inputs to 3 NN models
torch.autograd.set_detect_anomaly(True)
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)
#training
k=0
TrainData0=Mynodein[0,0:1,:].clone().detach().requires_grad_(True)
TrainData1=Mynodein[0,1:2,:].clone().detach().requires_grad_(True)
TrainData2=Mynodein[0,2:3,:].clone().detach().requires_grad_(True)

for epoch in range(50):

    epoch_lossN0 = 0
    epoch_lossN1 = 0
    epoch_lossN2 = 0
    lgcount=0
    #for i in range(k*Tbatch,k*Tbatch+Tbatch):
    for i in range(1):
        N0_weight = model.T0_forward(TrainData0)
        N1_weight = model.T1_forward(TrainData1)
        N2_weight = model.T2_forward(TrainData2)

        diff0=t[:,i]+TimeOfFlight_Matrix[0,:]-t[0,i]
        #diff0=t[:,time_idx].clone()-t[0,time_idx].clone()  #without propagation delay
        diff0_cat=torch.cat([diff0[0:0], diff0[0+1:]])
        prod0=torch.sum(diff0_cat* N0_weight)
        t0=t[0,i]+Tnode[0] + prod0  #evaluates the next timestamp for node 0, i.e., t[0,i+1]
        t[0,i+1]=t0

        diff1=t[:,i]+TimeOfFlight_Matrix[1,:]-t[1,i]
        #diff1=t[:,time_idx].clone()-t[1,time_idx].clone()  #without propagation delay
        diff1_cat=torch.cat([diff1[0:1], diff1[1+1:]])
        prod1=torch.sum(diff1_cat* N1_weight)
        t1=t[1,i]+Tnode[1] + prod1  #evaluates the next timestamp for node 1, i.e., t[1,i+1]
        t[1,i+1]=t1

        diff2=t[:,i]+TimeOfFlight_Matrix[2,:]-t[2,i]
        #diff2=t[:,time_idx].clone()-t[e,time_idx].clone()  #without propagation delay
        diff2_cat=torch.cat([diff2[0:2], diff2[2+1:]])
        prod2=torch.sum(diff2_cat* N2_weight)
        t2=t[2,i]+Tnode[2] + prod2  #evaluates the next timestamp for node 2, i.e., t[2,i+1]
        t[2,i+1]=t2

        timevec=[t0,t1,t2]

        lgtensor=torch.tensor(2+lgcount)

        local_lossN0= torch.log(lgtensor)*lossfn0(timevec)
        local_lossN1= torch.log(lgtensor)*lossfn1(timevec)
        local_lossN2= torch.log(lgtensor)*lossfn2(timevec)

    optimizer.zero_grad()
    local_lossN0.backward(retain_graph=True)
    local_lossN1.backward(retain_graph=True)
    local_lossN2.backward(retain_graph=True)

    optimizer.step()

I don’t know what I am doing wrong

Context:
I don’t know where tensor t is coming from but based on your code I would guess that these inplace operations are causing the issue:
t[0,i+1]=t0
t[1,i+1]=t1
t[2,i+1]=t2

Replace these calls with:
t = torch.stack((t0, t1, t2))

(or torch.cat depending on the shape of the tensors) and it might work.
Also, recreating a new tensor will detach it from the computation graph, so replace:
lgtensor=torch.tensor(2+lgcount)

with:
lgtensor = 2+lgcount

assuming lgcount is a tensor attached to a computation graph.


What are the alternatives for torch.save, torch.Tensor in tensorflow?

Question:
Does anyone know alternative for torch.save, torch.Tensor([]) in tensorflow? I assume alternative for torch.save() could be tensorflow.saved_models.save() and alternative for torch.Tensor([]) could be tensorflow.Tensor(numpy.empty()), but I am not really sure. Can anyone clear out my doubt?
especially, I am not relly sure what would be correct function in tensorflow for
 myVars = torch.Tensor([]).to(self.device)

Context: I would recommend to post TF-specific question in their discussion board, as you’ll most likely find the TF experts there.


RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [3, 1]], which is output 0 of TanhBackward, is at version 1; expected version 0 instead

Question:
Hey,
I’m getting this error but i don’t understand why it has a problem with ‘action_value = torch.tanh(self.action_values(x))’ when I use relu or hardtanh there is no problem.
class NAF(nn.Module):
    def __init__(self, state_size, action_size,layer_size, n_step, seed):
        super(NAF, self).__init__()
        self.seed = torch.manual_seed(seed)
        self.input_shape = state_size
        self.action_size = action_size

        self.head_1 = nn.Linear(self.input_shape, layer_size)
        self.ff_1 = nn.Linear(layer_size, layer_size)
        self.action_values = nn.Linear(layer_size, action_size)
        self.value = nn.Linear(layer_size, 1)
        self.matrix_entries = nn.Linear(layer_size, int(self.action_size*(self.action_size+1)/2))

    def forward(self, input_):
        """

        """

        x = torch.relu(self.head_1(input_))
        x = torch.relu(self.ff_1(x))
        action_value = torch.tanh(self.action_values(x))
        entries = torch.tanh(self.matrix_entries(x))
        V = self.value(x)

        # create lower-triangular matrix
        L = torch.zeros((input_.shape[0], self.action_size, self.action_size)).to(device)

        # get lower triagular indices
        tril_indices = torch.tril_indices(row=self.action_size, col=self.action_size, offset=0)

        # fill matrix with entries
        L[:, tril_indices[0], tril_indices[1]] = entries#.clone().detach()
        L.diagonal(dim1=1,dim2=2).exp_()

        # calculate state-dependent, positive-definite square matrix
        P = L*L.transpose(1, 2)

        # calculate action:
        dist = Normal(action_value, 1) #torch.inverse(P)
        action = dist.sample()
        action = torch.clamp(action, min=-1, max=1).unsqueeze(-1)
        action_value.unsqueeze_(-1)

        # calculate Advantage:
        A = (-0.5 * (action - action_value).transpose(1, 2) * P * (action - action_value)).squeeze(2)

        Q = A + V

        return action, Q, V

the problem only appears when trying to backprop
    def learn(self, experiences):
        """Update value parameters using given batch of experience tuples.
        Params
        ======
            experiences (Tuple[torch.Tensor]): tuple of (s, a, r, s', done) tuples
        """
        self.optimizer.zero_grad()
        states, actions, rewards, next_states, dones = experiences

        # get the Value for the next state from target model
        with torch.no_grad():
            _, _, V_ = self.qnetwork_target(next_states)

        # Compute Q targets for current states
        V_targets = rewards + (self.GAMMA * V_ * (1 - dones))

        # Get expected Q values from local model
        _, Q, _ = self.qnetwork_local(states)

        # Compute loss
        loss = F.mse_loss(Q, V_targets)

        # Minimize the loss
        loss.backward()
        #clip_grad_norm_(self.qnetwork_local.parameters(),1)
        self.optimizer.step()

        # ------------------- update target network ------------------- #
        self.soft_update(self.qnetwork_local, self.qnetwork_target)
        return loss.detach().cpu().numpy()

hope you guys can help

Context:
The issue is created by the inplace unsqueeze_ call on action_value, but is raised in the tanh.
If you use action_value = action_value.unsqueeze(-1) instead, your code should work.


I am getting this error when I try to run the pretrained hindsight-experience-replay

Question:
Traceback (most recent call last):
File “demo.py”, line 21, in
o_mean, o_std, g_mean, g_std, model = torch.load(model_path, map_location=lambda storage, loc: storage)
File “C:\Users\Student\anaconda3\envs\myenv\lib\site-packages\torch\serialization.py”, line 365, in load
f = open(f, ‘rb’)
FileNotFoundError: [Errno 2] No such file or directory: “saved_models/‘FetchReach-v1’/model.pt”
The link of source is GitHub - TianhongDai/hindsight-experience-replay: This is the pytorch implementation of Hindsight Experience Replay (HER) - Experiment on all fetch robotic environments. 2

Context: As the error message explains, it seems the specified file is not found and I guess you might have skipped this step 2, which asks you to download the files.


UserWarning: This overload of add_ , addcmul_, addcdiv_ is deprecated: errors while implementing SharedAdam

Question:
Hi,
I am new to pytorch and implementing A3C.
For this I have got a code for SharedAdam optimizer which is of pytorch version 0.4 which is producing above error in pytorch 1.6.0
Can someone please help to let me know what should be correct implementations of the functionality of above 3 functions in pytorch 1.6.0
Below is the code for pytorch 0.4

class SharedAdam(optim.Adam):
def __init__(self, params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8,
             weight_decay=0):
    super(SharedAdam, self).__init__(params, lr, betas, eps, weight_decay)

    for group in self.param_groups:
        for p in group['params']:
            state = self.state[p]
            state['step'] = torch.zeros(1)
            state['exp_avg'] = p.data.new().resize_as_(p.data).zero_()
            state['exp_avg_sq'] = p.data.new().resize_as_(p.data).zero_()

def share_memory(self):
    for group in self.param_groups:
        for p in group['params']:
            state = self.state[p]
            state['step'].share_memory_()
            state['exp_avg'].share_memory_()
            state['exp_avg_sq'].share_memory_()

def step(self, closure=None):
    """Performs a single optimization step.
    Arguments:
        closure (callable, optional): A closure that reevaluates the model
            and returns the loss.
    """
    loss = None
    if closure is not None:
        loss = closure()

    for group in self.param_groups:
        for p in group['params']:
            if p.grad is None:
                continue
            grad = p.grad.data
            state = self.state[p]

            exp_avg, exp_avg_sq = state['exp_avg'], state['exp_avg_sq']
            beta1, beta2 = group['betas']

            state['step'] += 1

            if group['weight_decay'] != 0:
                grad = grad.add(group['weight_decay'], p.data)

            # Decay the first and second moment running average coefficient
            exp_avg.mul_(beta1).add_(1 - beta1, grad)
            exp_avg_sq.mul_(beta2).addcmul_(1 - beta2, grad, grad)

            denom = exp_avg_sq.sqrt().add_(group['eps'])

            bias_correction1 = 1 - beta1 ** state['step'][0]
            bias_correction2 = 1 - beta2 ** state['step'][0]
            step_size = group['lr'] * math.sqrt(bias_correction2) / bias_correction1

            p.data.addcdiv_(-step_size, exp_avg, denom)

    return loss

Is this the correct implementation-
            exp_avg.mul_(beta1).add_(grad, alpha = 1 - beta1)
            exp_avg_sq.mul_(beta2).addcmul_(grad, grad, value = 1 - beta2)

            p.data.addcdiv_(exp_avg, denom, value = -step_size)

Context:
It seems you are calling the mentioned method as:
add_(Number alpha, Tensor other)

which is deprecated and should be changed to:
add_(Tensor other, *, Number alpha)

so in your case, you could use
exp_avg.mul_(beta1).add_(grad, alpha=1 - beta1)


Question:
Thanks I’m implementing DDPG where the actor and critic have a shared module. I’m running into an issue and I was wondering if I could get some feedback. I have the following:
INPUT_DIM = 100
BOTTLENECK_DIMS = 10
class SharedModule(nn.Module):
  def __init__(self):
    self.shared = nn.Linear(INPUT_DIM, BOTTLENECK_DIMS)
  def forward(self, x):
    return self.shared(x)

class ActorCritic(nn.Module):

  def __init__(self, n_actions, shared: SharedModule):
    self.shared = shared
    self.n_actions = n_actions

    # Critic definition
    self.action_value = nn.Linear(self.n_actions, BOTTLENECK_DIMS)
    self.q = nn.Linear(BOTTLENECK_DIMS, 1)

    # Actor Definition
    self.mu = nn.Linear(BOTTLENECK_DIMS, self.n_actions)

    self.optimizer = optim.Adam(self.parameters(), lr=self.lr)

  def forward(self, state, optional_action=None):
    if optional_action:
        return self._wo_action_fwd(state)
    return self._w_action_fwd(state, optional_action)

  def _wo_action_fwd(self, state):
    shared_output = self.shared(state)

    # Computing the actions
    mu_val = self.mu(F.relu(shared_output))
    actions = T.tanh(mu_val)

    # Computing the Q-vals
    action_value = F.relu(self.action_value(actions))
    state_action_value = self.q(
      F.relu(T.add(compressed, action_value))
    )
    return actions, state_action_value

  def _w_action_forward(self, state, action):
    shared_output = self.shared(state)
    action_value = F.relu(self.action_value(actions))
    state_action_value = self.q(
      F.relu(T.add(compressed, action_value))
    )
    return actions, state_action_value

My training process is then
shared_module = SharedModule()
actor_critic = ActorCritic(n_actions=3, shared_module)

shared_module = SharedModule()
T_actor_critic = ActorCritic(n_actions=3, shared_module)

s_batch, a_batch, r_batch, s_next_batch, d_batch = memory.sample(batch_size)

########################################
# Generate labels
########################################
# Get our critic target
_, y_critic = T_actor_critic(s_next_batch)
target = T.unsqueeze(
    r_batch + (gamma * d_batch * T.squeeze(y_critic)),
    dim=-1
)

########################################
# Critic Train
########################################
actor_critic.optimizer.zero_grad()
_, y_hat_critic = actor_critic(s_batch, a_batch)
critic_loss = F.mse_loss(target, y_hat_critic)
critic_loss.backward()
actor_critic.optimizer.step()

########################################
# Actor train
########################################
actor_critic.optimizer.zero_grad()
_, y_hat_policy = actor_critic(s_batch)
policy_loss = T.mean(-y_hat_policy)
policy_loss.backward()
actor_critic.optimizer.step()

Issues / doubts

Looking at OpenAI DDPG formula discussion, I’ve done step 12 and step 13 correctly (as far as I can tell). However, I don’t know how to do step 14.

The issue is that although I can calculate the entire Q-value, I don’t know how to take the derivative only with regards to theta. How should I go about doing this? I tried using
  def _wo_action_fwd(self, state):
    shared_output = self.shared(state)

    # Computing the actions
    mu_val = self.mu(F.relu(shared_output))
    actions = T.tanh(mu_val)

    # Computing the Q-vals
    with T.no_grad():
      action_value = F.relu(self.action_value(actions))
      state_action_value = self.q(
        F.relu(T.add(compressed, action_value))
      )
    return actions, state_action_value

but unsurprisingly I got an error of has no gradients which is unsurprising since I dropped off the gradients.

This is more of a DDPG question as opposed to a pytorch one, but is my translation of the algorithm correct? I do a step for the critic and then one for the actor?  I’ve seen

loss = torch.stack(policy_losses).sum() + torch.stack(value_losses).sum()

Is there a way to train it so that the shared module is stable? I imagine that being trained on two separate losses (I’m optimizing over 2 steps) might make convergence of that shared module wonky.

Context:
In response to #2, I found that line of code from examples/actor_critic.py at master · pytorch/examples · GitHub which I found from Actor Critic implementation problem 1
Since I’m new I can’t post more than 2 links in my post womp womp


Using Tensor Instead of numpy array in reinforcement learning tasks

Question:
Hi, I am trying to make an RL agent and environment, in this process I am using numpy array basically for action space which is a shape of [3]. So the main input and output of the neural net agent class which extends the torch.nn module, is a numpy ndarray. And the reward and other primary variables are numpy array too. But, the input of agent should be torch tensor. For example:
action, prediction = agent.act(state)
next_state, reward, done = env.step(action)
states.append(np.expand_dims(state, axis=0))
next_states.append(np.expand_dims(next_state, axis=0))
action_onehot = np.zeros(3)

In the code shown that is a function is based on numpy array like expand. And the challeng is transforming the numpy array to torch tensor and vice versa.
After all, I came up with three options and now I want to know which one is better to use(or any other option you may give):
1.Use the from_numpy function and convert the array to tensor before inputting it to the agent and then again transform it to numpy after prediction?
2.Use the transformers(from numpy and to numpy) in the forward function of the agent class?
3.Use torch tensor instead of numpy array in the whole process(which may not possible)?
And I believe keeping the grad of torch tensor is crucial, So please consider this.
Please explain how should I do the task.
Your answer much appreciated in regard.

Context: Would you please help @ptrblck