akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Using results of torch.concat in Forward Pass

a1o: <p>Hey all,<br>
I am working on a Model in which a layer takes two inputs.<br>
The first input is part of the input data, and the second input are embeddings (of other input data.)</p>
<p>I’ve tried to combine the two using torch.concat(), which returns a tensor.<br>
That tensor however cannot be used downstream without modifications;<br>
I receive errors like</p>
<pre><code class="lang-auto">TypeError: cannot assign 'torch.FloatTensor' as child module 'l1' (torch.nn.Module or None expected)

</code></pre>
<p>I’ve tried to cast the tensor as a Parameter using <code> nn.Parameter()</code> leading to the same error message.</p>
<p>The next option would be to create a custom Module just to cast the tensor from <code>torch.concat()</code> to the type <code>nn.Module</code>, but there is probably a better way.</p>
<p>How would you solve this issue?</p>

ptrblck: <aside class="quote no-group" data-username="a1o" data-post="1" data-topic="158698">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3bc359/48.png" class="avatar"> a1o:</div>
<blockquote>
<p>The next option would be to create a custom Module just to cast the tensor from <code>torch.concat()</code> to the type <code>nn.Module</code>, but there is probably a better way.</p>
</blockquote>
</aside>
<p>I don’t fully understand the use case as a tensor cannot be used or transformed to an <code>nn.Module</code>.<br>
Could you describe the use case a bit more and how the tensor should be used?</p>

a1o: <p>I try to use the result of a <code>torch.concat()</code> further in my forward pass, which looks like this:</p>
<pre><code class="lang-auto">  def forward(self, A, B):
        self.flat_embeddings = self.flatten(self.embeddings(A)) 
        self.combined = torch.cat((self.flat_embeddings, B), 1)
        self.l1 = self.relu(self.combined)  &lt;-- error here
        ...
</code></pre>
<p>So should I either use something different than <code>torch.concat()</code>  or is feeding a single layer with 2 inputs not possible in PyTorch?</p>

ptrblck: <p><code>self.combined</code> is a single tensor which is passed to an <code>nn.ReLU</code> and should work.<br>
However, I guess you are trying to assign the output to <code>self.l1</code> which seems to be an <code>nn.Module</code> and which is wrong.</p>

a1o: <p>thanks, I overlooked that one should pass the result tensor (<code>res</code>) of the layers above to the next layer using <code>self.layer(res)</code></p>
Two sets of dataloader different

Hsu_Chih-Shen: <p>I want to use two different transform methods to create two sets of dataloader in the same group of pictures,</p>
<pre><code class="lang-auto">train_set_1 = DatasetFolder("/content/food-11/training", 
              loader=lambda x: Image.open(x), extensions="jpg", 
              transform=train_tfm_1, )
train_set_2 = DatasetFolder("/content/food-11/training", 
            loader=lambda x: Image.open(x), extensions="jpg", 
            transform=train_tfm_2)

train_loader_1 = DataLoader(train_set_1, batch_size=batch_size, shuffle=True, pin_memory=True, worker_init_fn=worker_init_fn)
train_loader_2 = DataLoader(train_set_2, batch_size=batch_size, shuffle=True, pin_memory=True, worker_init_fn=worker_init_fn)

</code></pre>
<p>the current problem is that the order of the two sets of dataloader will be different,</p>
<pre><code class="lang-auto">dataiter = iter(train_loader_1)
images1, labels1 = dataiter.next()

dataiter2 = iter(train_loader_2)
images2, labels2 = dataiter2.next()type or paste code here
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/9/b9073646301a900f33fc672b695ff9c561d4c758.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b9073646301a900f33fc672b695ff9c561d4c758" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b9073646301a900f33fc672b695ff9c561d4c758_2_690x487.jpeg" alt="image" data-base62-sha1="qoPMAvgypeGuNtP2P4Kym80cA6Q" width="690" height="487" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b9073646301a900f33fc672b695ff9c561d4c758_2_690x487.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/b/9/b9073646301a900f33fc672b695ff9c561d4c758.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/9/b9073646301a900f33fc672b695ff9c561d4c758.jpeg 2x" data-dominant-color="716E6C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">765×540 93.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I want to please ask <strong>how to use the same folder to create two sets of dataloader with the same order, thank you!</strong></p>

Hsu_Chih-Shen: <p>I finally solved it as follows:</p>
<pre><code class="lang-auto">

g = torch.Generator()
g.manual_seed(0)

h = torch.Generator()
h.manual_seed(0)

train_loader_1 = DataLoader(train_set_1, batch_size=batch_size, shuffle=True, pin_memory=True, generator=g)
train_loader_2 = DataLoader(train_set_2, batch_size=batch_size, shuffle=True, pin_memory=True, generator=h)

</code></pre>
Proper way of setting h,c in bidirectional RNN Layers

lard2: <p>Hello everybody,</p>
<p>I’m having an issue understanding bidirectional RNN layers when non-zero hidden variable initialization is concerned (i.e. when h and c come from e.g. other layers). For example in this case</p>
<pre><code class="lang-python">
from torch import nn

input_size = 10
hidden_size =20
n_layers = 4
batch_size = 5
seq_len = 40

b_lstm = nn.LSTM(
    input_size,
    hidden_size,
    n_layers,
    batch_first=True,
    bidirectional=True,
)

# variables
# input
input = torch.randn(batch_size, seq_len, input_size)

# forward input hidden variables
h0 = ... # torch.randn(n_layers, batch_size, hidden_size)
c0 = ... # torch.randn(n_layers, batch_size, hidden_size)

# backwards input hidden variables
h1 = ... # torch.randn(n_layers, batch_size, hidden_size)
c1 = ... # torch.randn(n_layers, batch_size, hidden_size)
</code></pre>
<p>The way that seems more practical is simply stack them along the furthest dimension</p>
<pre><code class="lang-python"># first way of stacking
h = torch.cat([h0, h1], dim=0)
c = torch.cat([c0, c1], dim=0)

# first stacking execution
output, (hn, cn) = b_lstm(input, (h, c))
</code></pre>
<p>However, when comparing hn and cn for simple geometries, it appears that hn is staggered in the following way <code>hn = [h00, h01, h10, h11, h20, h21, ... hn0, hn1]</code> and not, as I would expect  <code>hn = [h00, h10, h20, ...hn0, h01, h11, h21, ..., hn1]</code>, hence this other way of combining variables:</p>
<pre><code class="lang-python">
# second way of stacking
h_ = torch.cat([h0.reshape(-1, hidden_size * batch_size),
                h1.reshape(-1, hidden_size * batch_size)],
               dim=1).reshape(-1, batch_size, hidden_size)
c_ = torch.cat([c0.reshape(-1, hidden_size * batch_size),
                c1.reshape(-1, hidden_size * batch_size)],
               dim=1).reshape(-1, batch_size, hidden_size)

# second stacking exection
output_, (hn_, cn_) = rnn(input, (h_, c_))
</code></pre>
<p>Does anybody have experience with this matter?</p>
RuntimeError: register_forward_hook is not support on ScriptModules

MartinZhang: <p>How to deal : register_forward_hook is not support on ScriptModules</p>
<p>Or whether there are alternative solutions， Thanks.</p>

ptrblck: <p>That’s a known limitation and the feature request is tracked <a href="https://github.com/pytorch/pytorch/issues/34329">here</a> so feel free to add your use case to the linked issue.</p>
A parameter about aten::scile

MartinZhang: <p>Today , I get aten::scile from TorchScriptModule， but I find <code>[0, 9223372036854775807, 0, 1]</code> in this <code>aten::scile</code> args.</p>
<p>What does this number <code>9223372036854775807</code> mean？</p>
<p>It’s <code>[...]</code>?</p>
<pre><code class="lang-auto">
prim::Constant[value=9223372036854775807]
</code></pre>

ptrblck: <p>I don’t know what <code>aten::scile</code> is and where it’s defined, but the value looks like an uninitialized value.</p>

MartinZhang: <p>Thank you for your reply <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=12" title=":blush:" class="emoji" alt=":blush:" loading="lazy" width="20" height="20">. I found it from ： <a href="https://pytorch.org/cppdocs/api/function_namespaceat_1ae6bdafd2514da71bc5a656a649bcb575.html" rel="noopener nofollow ugc">Function at::slice — PyTorch master documentation</a></p>

ptrblck: <p>Ah, it’s <code>slice</code>. Sorry, I was thinking if it’s a typo, but couldn’t really map it.<br>
Still, I think you are slicing an uninitialized tensor or are reading the underlying values in a wrong <code>dtype</code>. How did you create the tensor and how are you slicing it?</p>

MartinZhang: <p>Oh，Sorry, I make a typo. <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=12" title=":joy:" class="emoji" alt=":joy:" loading="lazy" width="20" height="20"></p>
<p>I get a TorchScriptModule (The network has slice operation)</p>
<p>Then I need to get parameters from slice operation</p>
<p>like this:</p>
<pre><code class="lang-auto">for node in model.inlined_graph.nodes():
    if node.kind() == 'aten::slice':
        print(list(node.inputs()))
</code></pre>
<p>Strangely， the inputs have a parameter is <code>prim::Constant[value=9223372036854775807]</code></p>
Converting Tensorflow Code to PyTorch by Removing Placeholders

PyTorch2603: <p>Hi. I know that placeholders in Tensorflow are used to define a variable which will have data assigned to it later on, but they cannot be directly evaluated.<br>
I was trying to convert the following code to PyTorch: <a href="https://github.com/jaechanglim/CVAE/blob/master/model.py" class="inline-onebox" rel="noopener nofollow ugc">CVAE/model.py at master · jaechanglim/CVAE · GitHub</a><br>
In the code they define placeholders like this:<br>
self.X = tf.compat.v1.placeholder(tf.int32, [self.batch_size, None])<br>
self.Y = tf.compat.v1.placeholder(tf.int32, [self.batch_size, None])<br>
self.C = tf.compat.v1.placeholder(tf.float32, [self.batch_size, self.num_prop])<br>
self.L = tf.compat.v1.placeholder(tf.int32, [self.batch_size])<br>
These placeholders are used before data is assigned to them such as here:<br>
weights = tf.sequence_mask(self.L, tf.shape(self.X)[1])<br>
Then data is finally assigned to them such as here:<br>
X = tf.nn.embedding_lookup(self.embedding_encode, self.X)<br>
I have read that you pass tensors directly to modules in PyTorch and placeholders are not used, but what exactly does this mean? Also, what changes would I make to this to convert it into PyTorch so that I don’t declare a placeholder for self.X, but am able to still use it? I would appreciate any clarification and assistance on how I could do this. Thanks!</p>

ptrblck: <p>You are right that PyTorch does not use placeholder variables and allows you to directly use tensors.<br>
I.e. instead of using this logic:</p>
<pre><code class="lang-python"># initialize placeholders with shape etc.
var1 = placeholder(...)
var2 = placeholder(...)

# set values to place holder
var1.load_data(...)
...

# execute model graph
...
</code></pre>
<p>you can directly use the tensors as in e.g. numpy but will get the benefit of Autograd and can execute your workload on different devices such as the GPU:</p>
<pre><code class="lang-python"># initialize input with random values
x = torch.randn(shape)
# or load as an image
img = PIL.Image.open(...)
x = torch.from_numpy(img)
# or load from a numpy array
arr = np.load(...)
x = torch.from_numpy(arr)
...

# execute model
out = model(x)

# now you can directly print the output
print(out)
</code></pre>

PyTorch2603: <p>Thanks for the reply. I have one question: In Tensorflow, the placeholders sometimes use None, meaning that the size of that dimension could be any number. Would I be able to do anything like that in PyTorch or would it not be necessary?</p>

ptrblck: <p>You don’t need to define “dynamic” and “static” axes and can just create the tensors in your desired shape.</p>
TorchServe GPU memory sharing when loading big models

bubble_tee: <p>I’m loading a very big model into GPU and I have multiple workers, and to save memory I want them to share GPU memory since it will be immutable.</p>
<p>With TorchServe or Gunicorn, if I’m using CPU I can fork() those workers and on Linux it will be Copy On Write by default. As long as I keep them immutable the memory will be shared among workers.</p>
<p>I wonder what happens when I’m using GPU memory? There are some tips on PyTorch multiprocessing and sharing CUDA memory (<a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" class="inline-onebox" rel="noopener nofollow ugc">Multiprocessing best practices — PyTorch 1.12 documentation</a>), which talks about using spawn() instead of fork(), but I wonder if this is/how it is implemented in TorchServe (it obviously won’t work with Gunicorn any more)?</p>
<p>Thanks!</p>

marksaroufim: <p>Unfortunately TorchServe does not yet support sharing GPU memory. It is however a top priority for the team so please follow updates on <a href="http://github.com/pytorch/serve" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/serve: Serve, optimize and scale PyTorch models in production</a> release notes when we release this</p>

bubble_tee: <p>Any way I can contribute? I’m a machine learning engineer with quite some experience on backend development. I have some personal interest in this and spare time.</p>
<p>Did some quick research and this seems quite doable if we load the model as immutable (to avoid trouble of locks) and use CUDA IPC.</p>
<p>Would be interesting to make Torch GPU multiprocessing work with Gunicorn too, as it’s very popular for ML Serving with Python.<br>
(Although this may require more work as Gunicorn uses fork() internally. On Linux systems this shares memory with workers as long as it’s immutable so can be used for sharing the IPC handle. Win32 a different story though).</p>

marksaroufim: <p>Sure yeah I’m always happy to see community contributions. Feel free to open a Github issue on pytorch/serve and we can discuss some concrete next steps</p>
Bias in MobileNet V2 int8 model

Xinyu_Wu: <p>I tried to understand the computation flow of pytorch MobileNet V2 int8 model and want to know how the bias, scale and zero-point are applied to a fused convolution layer. For instance as following, this layer has 4 params in state_dict: weight, bias, scale, zero-point. The weight is quantized from FP32 to INT8 with its own scale 0.106 and zero point; and the scale 0.0693 is supposed to convert accumulated result from FP32 to INT8 for next layer. But how to apply the bias? Does the bias applied to accumulated results after multiplication? These bias looks pretty small number comparing to accumulated results.</p>
<p>(‘features.1.conv.0.0.weight’,<br>
tensor([[[[ -0.1069,  -0.1069,  -0.1069],<br>
[ -0.1069,   0.0000,   0.8550],<br>
[ -0.1069,  -0.1069,   0.1069]]],<br>
…<br>
[[[  0.9619,  -0.4275,  -0.7482],<br>
[  4.3820,  -0.3206,  -3.9545],<br>
[  0.9619,  -0.2138,  -0.5344]]]], size=(32, 1, 3, 3),<br>
dtype=torch.qint8, quantization_scheme=torch.per_tensor_affine,<br>
scale=0.10687889158725739, zero_point=0)),<br>
(‘features.1.conv.0.0.bias’,<br>
tensor([-1.1895e-02,  8.7035e-01, -6.8617e-02,  3.8501e-01,  3.2915e-01,<br>
…<br>
8.4619e-01, -1.9708e-01], requires_grad=True)),<br>
(‘features.1.conv.0.0.scale’, tensor(0.0693)),<br>
(‘features.1.conv.0.0.zero_point’, tensor(0)),<br>
(‘features.1.conv.1.weight’,</p>

supriyar: <p>For Conv and Linear operations the bias stored in the state_dict is in FP32.<br>
For FBGEMM the bias is not quantized and we add the bias to the result of the final quantized matrix multiplication.<br>
For QNNPACK the bias is quantized to int32 (internally in the operator) and then added to the intermediate quantized output.</p>

aimen123: <p><a class="mention" href="/u/supriyar">@supriyar</a> hello，I want to set bias to int8, what i need to do ? thanks</p>
Negative label in CrossEntropyLoss

yzxhd: <pre><code class="lang-auto">loss = nn.CrossEntropyLoss()
input = torch.randn(3, 5, requires_grad=True)
target = torch.empty(3, dtype=torch.long).random_(5).long()
target[1] = -100
output = loss(input, target)
</code></pre>
<p>The target contains negative values but no error is raised. But if I change <code>target[1]</code> to other negative values, e.g. <code>-1</code>, <code>-10</code>, then the error can be raised correctly. Why is <code>-100</code> so magic?</p>
<p>System: Ubuntu 18.04<br>
Pytorch version: 1.10.0</p>

KFrank: <p>Hi yzx!</p>
<aside class="quote no-group" data-username="yzxhd" data-post="1" data-topic="158873" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/y/9f8e36/48.png" class="avatar"> yzxhd:</div>
<blockquote>
<p>Why is <code>-100</code> so magic?</p>
</blockquote>
</aside>
<p>From the <a href="https://pytorch.org/docs/1.12/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">documentation for CrossEntropyLoss:</a></p>
<blockquote>
<p><em>ignore_index=- 100</em></p>
</blockquote>
<p>and</p>
<blockquote>
<p><strong>ignore_index</strong> (<a href="https://docs.python.org/3/library/functions.html#int" rel="noopener nofollow ugc"><em>int</em></a><em>,</em> <em>optional</em>) – Specifies a target value that is ignored<br>
and does not contribute to the input gradient. When <code>size_average</code><br>
is <code>True</code>, the loss is averaged over non-ignored targets. Note that<br>
<code>ignore_index</code> is only applicable when the target contains class indices.</p>
</blockquote>
<p>That is, <code>-100</code> is the “magic value” that <code>ignore_index</code> is set to by default, so<br>
those targets are ignored (including the fact that the label would otherwise<br>
be an invalid negative value).</p>
<p>Best.</p>
<p>K. Frank</p>
How to "mount" an volume (directory that contains image data) to run with Torchserve?

nhthanh0809: <p>Hi all,<br>
I’m deploying Torchserve as service on local server. To optimize time processing in sending image data to Torchserve (my image size is about 2000x2000), I send request (REST API) to my Torchserve by sending image path. However, the Torchserve can not read the input image path. The Torchserve doesn’t know where is the image path, even I put image file to “model-store” directory that contains model in .mar format.<br>
So my question is how to setup Torchserve can read image file in expected directory?<br>
If it is possible, I also want to save the output as image file to other directory.<br>
Thank you!<br>
For more detail, please refer to my source code of Torchserve handler in below:</p>
<pre><code>class VTO(BaseHandler):
  def initialize(self, context):
      """Initialize function loads the model.pt file and initialized the model object.
     First try to load torchscript else load eager mode state_dict based model.
      Args:
          context (context): It is a JSON Object containing information
          pertaining to the model artifacts parameters.
      Raises:
          RuntimeError: Raises the Runtime error when the model.py is missing
      """
      properties = context.system_properties

      self.manifest = context.manifest

      self.map_location = "cuda" if torch.cuda.is_available() else "cpu"

      self.device = torch.device(
          # self.map_location + ":" + str(properties.get("gpu_id"))
          self.map_location + ":0"
          if torch.cuda.is_available()
          else self.map_location
      )

      self.transform = MLSDeformation()

      self.initialized = True

  def preprocess(self, requests):
      preprocessed_data = []
      for req in requests:
          data = req.get("data") or req.get("body")

          ### Final ###
          ## data_str = data.decode("utf-8")
          data_obj = json.loads(data)

          ### Get image path ###
          img_path = data_obj['image']
          print(img_path)
          file_exists = os.path.exists(img_path)
          input_image = cv2.imread(img_path,1)
          preprocessed_data.append((input_image))
      return preprocessed_data
  def inference(self, model_inputs):

     ........................................................
</code></pre>
<p>My request code as below:</p>
<pre><code>dict = {"image": './test.jpeg',
          "p_list": p_data,
          "q_list": q_data}
  data_obj = json.dumps(dict)
  response = requests.post(url_0, json=data_obj)
</code></pre>

marksaroufim: <p>This was resolved by lxning here <a href="https://github.com/pytorch/serve/issues/1778" class="inline-onebox" rel="noopener nofollow ugc">Read data from local directory of Torchserve server side · Issue #1778 · pytorch/serve · GitHub</a></p>
Torchvision loads cifar10 data slowly

Yuhang_Li: <p>Hi, I used <code> ds = torchvision.datasets.CIFAR10(root='./CIFARdata', train=False, download=True, transform=transform)</code> to load cifar10 dataset, and I found the training was slow and the max cuda utilization was about 50%. Later, I changed to custom data loader below, and found the training was faster and cuda utilization could be ~70%.</p>
<pre><code class="lang-auto">def image_preprocess(img, config):
    img = img.type(torch.float) / 255
    img_amp = img.view(1, 3, img.shape[1], img.shape[2])
    img_amp = F.interpolate(img_amp, size=(config.x_num,
                            config.y_num), mode='bilinear', align_corners=True)

    img_padx = (config.total_x_num - config.x_num) // 2
    img_pady = (config.total_y_num - config.y_num) // 2

    img_amp = F.pad(img_amp, (img_pady, img_pady, img_padx, img_padx))

    return img_amp.view(3, config.total_x_num, config.total_y_num)

class Cifar10Dataset(data.Dataset):
    def __init__(self, config, is_training=True):
        super(Cifar10Dataset, self).__init__()
        '''
        Args:
            is_training: network in training phase or not
        '''
        self.config = config
        self.image_dir = config.image_data_path
        self.image_transform = config.image_transform
        self.is_training = is_training  # training set or test set

        self.image_data, self.targets = self._load_data()
        print(1)
    def _load_data(self):
        image_data = []
        targets = []
        if self.is_training:
            image_file = ['data_batch_1', 'data_batch_2', 'data_batch_3', 'data_batch_4', 'data_batch_5']
        else:
            image_file = ['test_batch']
        for file_name in image_file:
            file_path = os.path.join(self.image_dir, file_name)
            with open(file_path, 'rb') as f:
                entry = pickle.load(f, encoding='latin1')
                image_data.append(entry['data'])
                if 'labels' in entry:
                    targets.extend(entry['labels'])
                else:
                    targets.extend(entry['fine_labels'])
        image_data = np.vstack(image_data).reshape(-1, 3, 32, 32)
        return torch.tensor(image_data).cuda(), torch.tensor(targets).cuda()

    def _transformation(self, input_img):
        data_transform = transforms.Compose([
            transforms.RandomHorizontalFlip(),
            transforms.RandomRotation(8)
        ])
   
        input_img = torch.squeeze(data_transform(torch.unsqueeze(input_img, dim=0)))
        return input_img

    def __getitem__(self, index):
        input_img, target = self.image_data[index], self.targets[index]  # Tensor

        if self.is_training and self.image_transform is not None:
            input_img = self._transformation(input_img)

        prop_img = image_preprocess(input_img, self.config)

        return prop_img, target

    def __len__(self) -&gt; int:
        return len(self.image_data)
</code></pre>
<p>Currently, I suspect it’s due to the <code>transform</code> that <code>torchvision.datasets.CIFAR10</code>  uses CPU and custom one uses GPU.<br>
Anyone has some ideas? Thank you!</p>
An increase in GPU consumption in the gradients

v-moayman: <p>Hi all,</p>
<p>I am raising this issue with the following example:</p>
<pre><code class="lang-auto">for inner_step in range(self.inner_steps):
  d_losses, _, bert_outputs = model.forward_with_params(
                          inputs_embeds=self.inputs_embeds,
                          labels=self.labels,
                          weights=weights,
                          output_attentions=True,
                      )
  grads = torch.autograd.grad(
                          d_loss, weights.values(), create_graph=True, allow_unused=True
                      )
   weights = OrderedDict(
                          (name, param - grad) if grad is not None else (name, param)
                          for ((name, param), grad) in zip(weights.items(), grads)
                      )
</code></pre>
<p>Now, when I am watching the GPU consumption, I have noticed a huge increase in such a consumption. I have tried to make the updates of the weights to be no_grad(), but it did not work as well.<br>
Your help is appreciated!</p>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/self-scaler-step-self-d-optimizer-assertionerror-no-inf-checks-were-recorded-for-this-optimizer/158800">here</a>.</p>
Python to C++ axis swap

soulslicer: <p>In Python i have</p>
<pre><code class="lang-auto">&gt; permute = [2, 1, 0]
&gt; t[:, permute]
</code></pre>
<p>How do i do this in C++?</p>
<p>I tried doing:</p>
<pre><code class="lang-auto">    std::vector&lt;long&gt; index_vector = {0,1,2};
    auto index_tensor = torch::from_blob((long*) (index_vector.data()),
                       {3}, torch::kLong);
    std::vector&lt;at::indexing::TensorIndex&gt; index;
    index = {"...", "...", index_tensor};
    img_tensor = img_tensor.index(index);
</code></pre>
<p>But this does not work</p>
What is the Difference between asssigning layer and assigning weight of layer?

muhammedcanpirincci: <p>i have two code example:</p>
<pre><code class="lang-auto">model.embedding = new_emb
</code></pre>
<p>and</p>
<pre><code class="lang-auto">model.shared.weight = new_emb.weight
</code></pre>
<p>What would be the difference between those two?</p>
<p>Thanks.</p>
Passing negative samples data into FasterRCNN

BobPupkin: <p>Hello everyone<br>
I am using FaterRCNN network and provide [0.0, 0.0, 0.0, 0.0] boxes with label=0, as a negative example.<br>
I get the error <code>ValueError: All bounding boxes should have positive height and width. Found invalid box [0.0, 0.0, 0.0, 0.0] for target at index 1.</code><br>
<strong>Additional info:</strong></p>
<ul>
<li>I don’t have IDs for images;</li>
<li>I tried passing area to a <code>targets {}</code> dictionary;</li>
<li>This did NOT work for me as well: <a href="https://github.com/pytorch/vision/pull/1911" class="inline-onebox" rel="noopener nofollow ugc">Train Faster R-CNN with negative samples by mnc537 · Pull Request #1911 · pytorch/vision · GitHub</a>
</li>
</ul>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> tagging you, because you are some kind of guardian angel of this forum)</p>
Return indices instead of value

Aarya: <p>The following script returns <strong>tuple of sorted nodes</strong>(“text”, bounding box dimensions) i.e</p>
<blockquote>
<p>Tuple[Tuple[graphanno.Node,…],…],<br>
where</p>
</blockquote>
<blockquote>
<p>Tuple[graphanno.Node,…]<br>
represents a line.</p>
</blockquote>
<p>But, i want to return the tuple of <strong>indices of sorted nodes</strong> i.e</p>
<blockquote>
<p>Tuple[Tuple[int,…],…],</p>
</blockquote>
<p>Following is the script that sorts nodes from left to right and returns tuple of sorted nodes.</p>
<pre><code class="lang-auto">def sort_nodes_into_lines(
        nodes: Iterable[graphanno.Node],
    ) -&gt; Tuple[Tuple[graphanno.Node, ...], ...]:
        """
        Merge nodes that belong to one line.
    
        The algorithm does the following:
        1. Sort nodes by x-coordinate.
        2. Iterate through them and merge those that are close in x- and have
           overlap in y-direction to lines.
        3. If the current node does not belong to a previously seen line,
           start a new line from the node.
    
        Parameters
        ----------
        nodes
            The nodes that shall be sorted.
    
        Returns
        -------
        Tuple[Tuple[graphanno.Node,...],...]
            A tuple of the nodes sorted.
        """
        nodes_sorted = sorted(nodes, key=lambda node: node.box.origin_x)
        reference, *remainder = nodes_sorted
        lines: Dict[graphanno.Node, List[graphanno.Node]] = {reference: [reference]}
        for node_idx, node in enumerate(remainder, start=1):
            is_matched = False
            for reference, line in lines.items():
                if _has_sufficient_y_overlap(reference, node):
                    *_, last_in_row = line
                    if _in_x_proximity(last_in_row, node):
                        line.append(node)
                        is_matched = True
                        break
            if not is_matched:
                lines[node] = [node]
        return tuple(
            sorted(
                (tuple(line) for line in lines.values()),
                key=_get_line_origin_x,
            )
        )
    
    
    def _get_line_origin_x(nodes: Sequence[graphanno.Node]) -&gt; int:
        """Find min value of y coordinate of the origin of the bounding box."""
        return min(node.box.origin_y for node in nodes)
</code></pre>
<p>Any help would be appreciated!</p>
<p>Thank you!</p>
How to know for sure that DDP is sharing the gradients as it is supposed to do?

PaulNaduri: <p>I wrapped my model with DDP. However in the past I had issues where due to the model architecture DDP was not hooking to all the tensors and therefore not sharing the gradients as it was supposed to.</p>
<p>Now with a different model and architecture, I am concerned and want to know for sure that DDP is working correctly. Is there a simple way to test this?</p>
<p>I also created a torchviz graph diagram of my model, but I am not sure what I should be looking at. There do seem to be some scatter’s through in there but I do not know for sure if I can tell from the diagram if all is working fine or not.</p>
<p>Torchviz diagram: <a href="https://i.imgur.com/htq9TCb.png" rel="noopener nofollow ugc">https://i.imgur.com/htq9TCb.png</a></p>
<p>So to rephrase? How do I test that DDP is sharing the gradients as it should?</p>
Compute block diagonal of multiplication of two tensors efficiently

seewoo5: <p>Assume that we have two 100 by 30 matrices A and B. I want to compute ten 10 by 10 block diagonals of the matrix AB^T, and transform it as a 10 by 100 matrix by concatenating them. Is there an efficient way to do this?<br>
Currently, I chunked the matrices A and B as ten 10 by 30 matrices and compute A_{i} * B_{i}^T for each chunk 0 &lt;= i &lt; 10, and concatenate it, which uses for loop. I want to do this without using any loops.<br>
Thanks in advance.</p>

Nikronic: <p>Hi,</p>
<p>First step which is the extracting 10x30 matrices can be done in the same way as we extract patches out of matrices. The proper function for doing it is <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Unfold" rel="nofollow noopener">torch.unfold</a>.</p>
<p>After this, we need to compute <code>ab.T</code> operation over batches which <a href="https://discuss.pytorch.org/t/dot-product-batch-wise/9746">torch.bmm</a> enables us to do the dot product patch-wise.</p>
<p>Finally it is just a reshape to convert each 10x10 matrix to a 100 vector.</p>
<pre><code class="lang-auto"># initalize section - testing purposes
a = torch.ones((100, 30))
b = torch.ones((100, 30))
for i in range(10):
    a[i*10:(i+1)*10, :]=i
    b[i*10:(i+1)*10, :]=-i

# main code
a = a.unfold(0, 10, 10).reshape(10, -1, 30)  # torch.Size([10, 10, 30])
b = b.unfold(0, 10, 10)  # torch.Size([10, 30, 10])
result = torch.bmm(a, b)  # torch.Size([10, 10, 10])
result = result.reshape(10, -1)  # torch.Size([10, 100])
</code></pre>
<p>bests,<br>
Nik</p>
<p>edit: if you have any trouble with <code>fold</code> and <code>unfold</code>, <a href="https://discuss.pytorch.org/t/how-to-extract-smaller-image-patches-3d/16837/25">this discussion</a> may help.</p>

seewoo5: <p>Thanks! I understand what unfold does.<br>
But for <code>a</code>, I think we need to use <code>transpose(-1, -2)</code> instead of <code>reshape(10, -1, 30)</code>. Am I right? It seems that the last <code>reshape</code> for <code>result</code> also doesn’t give a right answer.</p>

Nikronic: <p>Yes! you are right about <code>transpose(-1, -2)</code> for <code>a</code>. As I used same constant value for all cells of each patch, the result of transpose and reshape was identical.</p>
<p>Concerning the last <code>reshape</code>, it just works like <code>flatten</code> as we want to merge <code>dim=1</code> and <code>dim=2</code> into 1D tensor with size=100. In <code>result</code> we are not transposing any dim with any other and as you mentioned in your question, it is more like concatenating them in the same way they already are. What do you think?</p>

111319: <p>I have the same requirements. But in my problem, I need to split the matrices into different sizes. For example, I have two 10xC matrices A &amp; B, and I need to compute the block diagonals with sizes of 3x3, 3x3, and 4x4 for matrix AB^T. Is there an efficient way to avoid adopting for loops?</p>
Pytorch distributed master node

SandPhoenixX: <p>Hello everyone,</p>
<p>I am trying to use pytorch distributed for training and inference but I hate the fact that there is a master node, I would like all the nodes to keep trying to work even if the master node does not exist.<br>
Is there a way to get rib of the master node or to maybe put multiple nodes as master? any hint or help will be apprecited</p>
<p>Thanks in advance</p>
Where is the torch.nn.init

Shawn1993: <p>I just update the newest version of pytorch. I want to use the init module, but got some error as following:</p>
<pre><code>In [1]: torch.__version__
Out[1]: '0.1.11_5'

In [2]: torch.nn.init
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
&lt;ipython-input-6-a03bb0a8e5dc&gt; in &lt;module&gt;()
----&gt; 1 torch.nn.init

AttributeError: module 'torch.nn' has no attribute 'init'</code></pre>

wangg12: <p>Have you tried <code>from torch.nn import init</code>?</p>

Shawn1993: <p>Thank you ~ It does really work! Can you explain why?</p>

smth: <p><a class="mention" href="/u/shawn1993">@Shawn1993</a> it is standard python packages and how they behave. Without importing something it wont be available.</p>

Shawn1993: <p>I can do something like:</p>
<pre><code>import torch.nn.functional as F
</code></pre>
<p>But I can’t do the same operation with init package.</p>

vabh: <p>This should work: <code>import torch.nn.init as weight_init</code></p>

Shawn1993: <p>Yes, you are right!  It just can’t be autocomplete with Ipython . It seems a silly question now.</p>

Adel_Saleh: <p>No it is not silly question look this</p>
<p>import torch.nn as nn<br>
import torch.nn.init<br>
Traceback (most recent call last):<br>
File “/usr/lib/python2.7/dist-packages/IPython/core/interactiveshell.py”, line 2883, in run_code<br>
exec(code_obj, self.user_global_ns, self.user_ns)<br>
File “”, line 1, in <br>
import torch.nn.init<br>
File “/home/adel/Desktop/pycharm-community-2016.3.3/helpers/pydev/_pydev_bundle/pydev_import_hook.py”, line 21, in do_import<br>
module = self._system_import(name, *args, **kwargs)<br>
ImportError: No module named init</p>

aneeka_azmat: <p>it is not working for me <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
Gaussian Loss and variance resize

Berny: <p>Hello, I’ m using the Gaussian loss this way:</p>
<pre><code>        varTrain = torch.ones(224, 1, requires_grad=True)
        varTrain = torch.reshape(varTrain, (32, 7))
        loss = loss_mse(recon_batch, XTrain, mu, logvar) + loss_gauss(predictions, YTrain.view(-1, 1), 
                                    varTrain)
</code></pre>
<p>where predictions is [32, 7], YTrain represents the true labels (0,1,2,…), and varTrain should have the same size of predictions. The code only works at the first epoch, then I get the following error:</p>
<pre><code>        ValueError: var is of incorrect size
</code></pre>
<p>How can I fix it? Thank you in advance.</p>

ptrblck: <p>From the <a href="https://pytorch.org/docs/stable/generated/torch.nn.GaussianNLLLoss.html">docs</a>:</p>
<blockquote>
<p><code>Var</code>: <code>(N, *)</code> or <code>(*)</code>, same shape as the input, or same shape as the input but with one dimension equal to 1, or same shape as the input but with one fewer dimension (to allow for broadcasting)</p>
</blockquote>
<p>Check the shape of <code>varTrain</code> and make sure it meets these requirements.</p>

Berny: <p>Actually, the dimension of varTrain is [32, 7] as the dimension of predictions, indeed, the code seems to work until the first epoch. After that I get the error and I cannot understand the reason. Any suggestion?</p>

ptrblck: <p>The shape seems to change after the first epoch, so check why that’s the case.</p>

Berny: <p>Yes, I noticed it but really don’t know why. I think to have done as the example in the doc does.</p>
The difference between torch.nn.utils.parametrizations.spectral and torch.nn.utils.spectral

wei_liu1: <p>When I tried to use  torch.nn.utils.parametrizations.spectral_norm in GRUs, I found it raises the error “Trying to backward through the graph a second time” in the second mini-batch iteration.  While this problem does not arise when I put the model on CPU or just use torch.nn.utils.spectral. It is really strange.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/3/c34879bf7535b12bb53cc17bcb6361f0cdd8a127.png" data-download-href="https://discuss.pytorch.org/uploads/default/c34879bf7535b12bb53cc17bcb6361f0cdd8a127" title="QQ截图20220810161007"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c34879bf7535b12bb53cc17bcb6361f0cdd8a127_2_690x133.png" alt="QQ截图20220810161007" data-base62-sha1="rRynRJxXOhEh9in6KMRUpd3faQf" width="690" height="133" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c34879bf7535b12bb53cc17bcb6361f0cdd8a127_2_690x133.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c34879bf7535b12bb53cc17bcb6361f0cdd8a127_2_1035x199.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c34879bf7535b12bb53cc17bcb6361f0cdd8a127_2_1380x266.png 2x" data-dominant-color="303235"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">QQ截图20220810161007</span><span class="informations">1633×317 49.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The codes are as follows:</p>
<p>import torch<br>
import torch.nn as nn<br>
from torch.nn.utils.parametrizations import spectral_norm</p>
<p>torch.manual_seed(111)<br>
torch.cuda.manual_seed(111)<br>
class KKK(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(KKK, self).<strong>init</strong>()<br>
self.encoder =spectral_norm(nn.GRU(input_size=100,<br>
hidden_size=100,<br>
num_layers=1,<br>
batch_first=True,<br>
bidirectional=True),name=‘weight_ih_l0’)<br>
self.dropout = nn.Dropout(0.2)<br>
self.fc = spectral_norm(nn.Linear(200, 2))</p>
<pre><code>def forward(self, inputs):
    outputs, _ = self.encoder(inputs)
    outputs = outputs[:,0,:]
    logits = self.fc(self.dropout(outputs))
    return logits
</code></pre>
<p>loss_func=nn.CrossEntropyLoss()<br>
input = torch.randn(3, 10,100, requires_grad=True).to(‘cuda:1’)<br>
target = torch.empty(3, dtype=torch.long).random_(2).to(‘cuda:1’)</p>
<p>model=KKK().to(“cuda:1”)<br>
opt=torch.optim.Adam(model.parameters(),lr=0.001)<br>
model.train()<br>
for i in range(10):<br>
opt.zero_grad()<br>
lg=model(input)<br>
loss=loss_func(lg,target)<br>
loss.backward()<br>
print(‘yes’)<br>
opt.step()</p>
Spectral_norm used in RNN causes "parameter types mismatch" in GPU

oh-y: <h2>Issue description</h2>
<p>spectral_norm used in <code>nn.Linear</code> is okay. But when it’s used in <code>nn.RNN</code>,there will be a RuntimeError while running <code>model = network().cuda()</code></p>
<p>Traceback (most recent call last):<br>
File “/DATA/119/yhe/soft/anaconda3/envs/env1/lib/python3.6/site-packages/IPython/core/interactiveshell.py”, line 3296, in run_code<br>
exec(code_obj, self.user_global_ns, self.user_ns)<br>
File “”, line 1, in <br>
runfile(’/DATA/119/yhe/code/crit/mnist/example.py’, wdir=’/DATA/119/yhe/code/crit/mnist’)<br>
File “/home/yhe/.pycharm_helpers/pydev/_pydev_bundle/pydev_umd.py”, line 197, in runfile<br>
pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script<br>
File “/home/yhe/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py”, line 18, in execfile<br>
exec(compile(contents+"\n", file, ‘exec’), glob, loc)<br>
File “/DATA/119/yhe/code/crit/mnist/example.py”, line 17, in <br>
model = network().cuda()<br>
File “/DATA/119/yhe/soft/anaconda3/envs/env1/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 265, in cuda<br>
return self._apply(lambda t: t.cuda(device))<br>
File “/DATA/119/yhe/soft/anaconda3/envs/env1/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 193, in _apply<br>
module._apply(fn)<br>
File “/DATA/119/yhe/soft/anaconda3/envs/env1/lib/python3.6/site-packages/torch/nn/modules/rnn.py”, line 127, in _apply<br>
self.flatten_parameters()<br>
File “/DATA/119/yhe/soft/anaconda3/envs/env1/lib/python3.6/site-packages/torch/nn/modules/rnn.py”, line 123, in flatten_parameters<br>
self.batch_first, bool(self.bidirectional))<br>
RuntimeError: param_from.type() == param_to.type() ASSERT FAILED at /pytorch/aten/src/ATen/native/cudnn/RNN.cpp:541, please report a bug to PyTorch. parameter types mismatch</p>

<h2>Code</h2>

<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

class network (nn.Module):
    def __init__(self):
        super(network, self).__init__()
        self.sn_rnn = nn.utils.spectral_norm(nn.RNN(20,10,1),name='weight_hh_l0')
        # self.fc = nn.utils.spectral_norm(nn.Linear(20,10), name='weight')
    def forward (self,x):
        # x = F.tanh(self.fc(x))
        out,_ = self.sn_rnn(x)
        return F.log_softmax(x, dim=1)

if __name__ == '__main__':
    x = torch.randn(2, 10, 20).cuda()
    model = network().cuda()
    out= model(x)
    print('end')
</code></pre>
<h2>Environment</h2>
<ul>
<li>PyTorch Version (e.g., 1.0): 1.1.0</li>
<li>Python version: 3.6.8</li>
<li>CUDA/cuDNN version:  cuda9.0</li>
</ul>

wei_liu1: <p>Will this problem occur when you use CPU?</p>
How to use Spectral Normalization in RNN or Transformer?

Sine: <p>Hi everyone, I am trying to train a GAN model with RNN and Transformer. Some paper show that using the spectral normalization is beneficial to training. However, The examples of  <code>torch.nn.utils.spectral_norm</code>  I saw are all used on CNN and Linear.</p>
<p>Can I use spectral normalization in RNN or Transformer? How to use?<br>
Thanks.</p>

wei_liu1: <p>Hello! I have the same question. Have your question been addressed?</p>
Load a model and use multiple GPUs at inference

FiReTiTi: <p>I have a model that I train on multiple GPUs, and then use it for inference. If I do training and inference all at once, it works just fine, but if I save the model and try to use it later for inference using multiple GPUs, then it fails with this error:</p>
<pre><code class="lang-auto">RuntimeError: Expected tensor for argument #1 'input' to have the same device as tensor for argument #2 'weight'; but device 1 does not equal 0 (while checking arguments for cudnn_convolution)
</code></pre>
<p>So, this works <strong>fine</strong>:</p>
<ul>
<li>Create model</li>
<li>Train model on multiple GPUs</li>
<li>Use model for inference using multiple GPUs</li>
</ul>
<p>This <strong>fails</strong> at inference:</p>
<ul>
<li>Create model</li>
<li>Train model on multiple GPUs</li>
<li>Save model using <code>torch.save(model, path)</code>
</li>
<li>Load model using <code>torch.load("path", map_location=torch.device("cpu"))</code>, also fails if I don’t use <code>map_location</code>
</li>
<li>Use model for inference using multiple GPUs (I don’t forget to use <code>to(device)</code>)</li>
</ul>
<p>What am I doing wrong?</p>

ptrblck: <p>Could you explain how multiple GPUs are used in the working cases?<br>
Are you using <code>DistributedDataParallel</code> or any manual model sharding?</p>

FiReTiTi: <p>Thank you for your answer.<br>
No, I use <code>if torch.cuda.device_count() &gt; 1: model = nn.DataParallel(model)</code> for the multiple GPUs part in both scenarios. In fact the code is exactly the same, it’s just that if the model is not already trained, then I train and saved it.<br>
I should precise that I work on a cluster made of nodes containing 4 GPUs each. I did the test requesting and using 2 rtx2080.</p>

ptrblck: <p>OK, strange. Could you post a minimal, executable code snippet showing the failure, please?<br>
The model definition as well as some random input tensors might be sufficient to reproduce the issue.</p>

FiReTiTi: <p>So, I fixed the issue, or at least I did what was “recommended”.</p>
<p>When saving the model, I was saving the entire model <code>torch.save(model, path)</code>, not the state dict <code>torch.save(model.state_dict(), path)</code>. And now I can use multiple GPUs or not for training, save the model, load it, and use multiple GPUs of not for inference.</p>
<p>However, it seems that saving the entire model is bugged.</p>

FiReTiTi: <p>I had to train another model, completely different architecture. The exact same thing happened, with the same solution: if I save the entire model, I cannot use multiple GPUs after, but if I save the state dict, then everything is fine.</p>
Torchscript - is it better to script the model in the same environment?

may1: <p>I converted a pytorch model into torchscript in one server (using torch.jit.trace)<br>
and checked that torchscript model is faster than a torch model.</p>
<p>I copied the weight path to the other server (computer specs are different but virtual environment is same) and loaded it.<br>
However, inference speed of torchscript model was slower than the original torch model in the server.<br>
Do I have to script the model in the given server?<br>
Does scripting work differently with different gpu?</p>

ptrblck: <p>I don’t know what exactly is currently saved from a scripted model as the backend moves quite fast at the moment. Could you run another test and see if the speed would change if you script the model again on the target system?</p>
Newbie: Is there a tutorial for this scenario

eformx: <p>I wish the tuts on PyTorch started off with a busiiess use case but they never seem to do that which makes it extremely difficult to find the correct tutorial.</p>
<p>I have a dataset with two columns, one contains a paragraph of text and the other a label (intent).  The dataset contains five possible labels.  All the data is pre-tagged.  I would like a model to take the inputs as is (the labels) and have it learn the contextual similarities based on the paragraphs and their pretagged label.</p>
<p>I created an NLP model to support this data structure in Apache OpenNLP many years agao but want to convert it into a PyTorch/Python model.  Is there a tutorial someone can recommend that walks a newbie through this type of scenario?</p>

FlankCoder: <p>Yeah you can go with the official pytorch tutorial that is collabrated with microsoft learn that one is amazing</p>