akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Replacing the for loop with Pytorch's tensor operation(s)

melike: <p>Hi all, I have a tensor with integers in the range [0, N-1] and I need to create N masks, one for each integer value, i.e. in the 0’s mask all values of 0 should be True and the rest False, and so on. So far I could come up with a for loop on the range, but that drastically increases the runtime. I’d be grateful if someone could suggest equivalent Pytorch tensor operation(s) instead.</p>

KFrank: <p>Hi Melike!</p>
<aside class="quote no-group" data-username="melike" data-post="1" data-topic="132936" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/7ea924/48.png" class="avatar"> melike:</div>
<blockquote>
<p>suggest equivalent Pytorch tensor operation(s) instead.</p>
</blockquote>
</aside>
<p>Try:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; _ = torch.manual_seed (2021)
&gt;&gt;&gt; N = 10
&gt;&gt;&gt; t = torch.randint (N, (2, 3, 5))
&gt;&gt;&gt; i = torch.ones ([N] + list (t.shape)).cumsum (0) - 1
&gt;&gt;&gt; masks = t == i
&gt;&gt;&gt; t
tensor([[[4, 9, 3, 2, 5],
         [6, 2, 0, 5, 0],
         [1, 9, 7, 7, 4]],

        [[8, 0, 0, 2, 4],
         [1, 7, 5, 7, 0],
         [1, 6, 9, 8, 6]]])
&gt;&gt;&gt; masks[0]
tensor([[[False, False, False, False, False],
         [False, False,  True, False,  True],
         [False, False, False, False, False]],

        [[False,  True,  True, False, False],
         [False, False, False, False,  True],
         [False, False, False, False, False]]])
&gt;&gt;&gt; masks[4]
tensor([[[ True, False, False, False, False],
         [False, False, False, False, False],
         [False, False, False, False,  True]],

        [[False, False, False, False,  True],
         [False, False, False, False, False],
         [False, False, False, False, False]]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Splitting and padding a tensor

pepsbn: <p>Hi all, is there any way to change A to B using pytorch’s built-in functions?</p>
<p>A=[1,2,0,4,5,6,7,0,8,9,10]<br>
B = [[1,2,0,0,0],[4,5,6,7,0][8,9,10,0,0]]</p>
<p>Basically I’m trying to split A by value (0 in this case) and pad results.</p>
<p>A -&gt; split(0) -&gt; [1,2], [4,5,6,7], [8,9,10] -&gt; pad -&gt; [1,2,0,0,0],[4,5,6,7,0], [8,9,10,0,0]</p>
<p>Thanks</p>

Shaked_G: <p>did you manage  to find such a function?</p>

YutongZheng: <p>You can check out [torch.nn.utils.rnn.pad_sequence] (<a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.rnn.pad_sequence.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.utils.rnn.pad_sequence — PyTorch 1.9.1 documentation</a>)</p>
Access and overwrite ALL the weights (automatically, in a loop)

andrea.migliorati: <p>Hi, I’m looking for a way of accessing all the weights in a model in an automatic way (i.e. without manually resorting to the name of each layer) so that I can overwrite them.</p>
<p>I understand that I can see them all by doing:</p>
<p>for param in model.parameters():<br>
…</p>
<p>but what I actually need to do is take that <strong>param</strong> and change it so that the <strong>param</strong> will have the desired value also inside the model (hence allowing me to see at inference time how the change is affecting accuracy). Thanks in advance</p>

shivammehta007: <p>If I understand correctly you can use</p>
<pre><code class="lang-python">for name, param in model.named_parameters():
   # Just an example
   if 'weight' in name:
         param.data == torch.zeros_like(param)
   elif 'bias' in name:
         param.data == torch.zeros_like(param) 
</code></pre>
<p>If you do this the param will reflect the changes back without adding a step to the gradient tape.</p>

andrea.migliorati: <p>Ok thank you very much, yes this immediately works. I had found this other solution first <a href="https://discuss.pytorch.org/t/overwrite-parameters-of-model-with-new-values/92952" class="inline-onebox">Overwrite parameters of model with new values</a> but it’s nice you can avoid using <strong>state_dict()</strong>.</p>
Need Guidance regarding End to End Multilingual Scene Text Detection and Recognition Using Pytorch

aatiibutt: <p>I am a bit newbie in this research area i.e. text detection and recognition. I worked with EAST for text detection. Now, I wanna start working with multi-lingual text detection and recognition. In this regard, I need a kind of expert opinion/guidance about it. As mentioned in title, I am familiar with PyTorch. So, I shall be grateful, if you share some useful resources as well.</p>

Ivan_A: <p>Good morning, for text detection and recognition, I’m working with  MMOCR  and you can check detectron2 as well. cheers.</p>
Question about loss.backward

Jaeyoung_Shin: <p>I have an RNN network model that is connected to a linear layer. My question is that if I use the softmax function outside the model(like outside the forward function) to calculate loss with cross-entropy loss, will it be included in the loss.backward() calculation?</p>

shivammehta007: <p>Yes, it will be included. The backpropagation is not linked with the forward function. As much as I understand <code>forward</code> is just useful to make sure all the hooks and other essential <code>nn.Module</code> functionalities are maintained before the <code>__call__() </code> of the class.</p>
Rewriting Tf to Pytorch DepthwiseConv2d functor opperation

moefke: <p>Hello, In my research I have to rewrite the following function to Pytorch. The project involves looping over the layers in the network. Does anyone know how to rewrite the following:</p>
<pre><code class="lang-auto">depthwise = DepthWiseConv2d(in_channels = inp.shape[2], out_channels = out.shape[2], kernel_size = kernel, padding=padding, stride=stride, dilation = dilation_rate)
functor = K.function([inp, K.learning_phase()], [depthwise])
depthwise_conv2d_vals = functor([test_ex, 0])[0]
</code></pre>

ptrblck: <p>I don’t know what a <code>functor</code> is in TF/Keras, but would assume that you could just create the <code>nn.Conv2d</code> layer in PyTorch and pass the input directly to it.<br>
If that’s not the case, could you explain what this code is doing?</p>

moefke: <p>Thank you for the reply, it is already fixed. So this topic could be removed.</p>
How BiLSTM works with padding/pack_padded_sequence

XezXey: <p>I understand how padding and pack_padded_sequence work, but I have a question about how it’s applied to Bidirectional.</p>
<ol>
<li>
<p>Does the BiLSTM (from nn.LSTM) automatically applied the inverse of the sequence (also in case of using the pad_packed_sequence)?</p>
</li>
<li>
<p>If yes, so does the padding affect the first-last timestep?</p>
<pre><code>for example : seq1=[a, b, c, d, e], seq2=[x, y, z]
and after padding : seq1=[a, b, c, d, e], seq2=[x, y, z, 0, 0]
If we input the seq2, this means that BiLSTM
      - take input of [x, y, z, 0, 0] and [0, 0, z, y, z]
  or  - take input of [x, y, z, 0, 0] and [z, y, x, 0, 0]
</code></pre>
</li>
</ol>
<p>Would you please help me clarify these points? Thank you very much</p>

huahuanZ: <p>As far as I know, case 2 is the correct way.<br>
Edit: I remove the misleading expression, please refer below discussion.</p>

XezXey: <p>Is it because of lengths parameters that we use while packing to make the padding mask not be included in a network input?</p>

huahuanZ: <p>I think the length information is even not required.<br>
Note that we pad the short sequences with zeros. RNN with input <code>hidden=None</code> (the first frame) would actually creates <code>hidden=torch.zeros(...)</code>. In this case, <code>[z, y, x, 0, 0]</code> and <code>[0, 0, z, y, x]</code> just produce same results.<br>
But for theoretically, an inverse (<code>[x, y, z, 0, 0] -&gt; [0, 0, z, y, x]</code>) and a shift (<code>[0, 0, z, y, x] -&gt; [z, y, x, 0, 0]</code>) are required in case paddings are not zeros.</p>
How to export split to ONNX with dynamic split_size?

Wwwwei: <p>I need to implement dynamic tensor split op in work. But when I want to export this split op to ONNX with dynamic split_size, it seems not work.</p>
<p>I am new to ONNX. Anyone can help me? Thanks a lot.</p>
<h2>
<a name="to-reproduce-1" class="anchor" href="#to-reproduce-1"></a>To Reproduce</h2>
<pre><code class="lang-python">import torch

dummy_input = (torch.tensor([1, 4, 2, 7, 3]), torch.tensor([1, 2, 2]))

class Split(torch.nn.Module):
    def forward(self, x, l):
        return x.split(l.cpu().numpy().tolist(), dim=-1)
    
model = Split()

with torch.no_grad():
    torch.onnx.export(
        model, dummy_input, 'split.onnx', verbose=False, opset_version=13,
        input_names=['a', 'b'],
        output_names=['c'],
        dynamic_axes={'a': [0], 'b': [0], 'c': [0]}
    )
</code></pre>
<p>when I use the onnx model, it seems not to work. I get this error:<br>
[ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Invalid Feed Input Name:b</p>
<pre><code class="lang-python">import onnxruntime as ort

model_path = './split.onnx'
sess = ort.InferenceSession(model_path)

a = torch.tensor([4, 2, 3, 4])
b = torch.tensor([1, 3])
sess.run(['c'], {'a':a.numpy(), 'b':b.numpy()})
</code></pre>
<p>Tensor b seems can not be used as an input, but I do need a parameter to represent the dynamic split_size.</p>
<h2>
<a name="environment-2" class="anchor" href="#environment-2"></a>Environment</h2>
<ul>
<li>PyTorch Version (e.g., 1.0): 1.8.1+cu111</li>
<li>Python version: 3.8</li>
</ul>
How to use output of one model as input of other model?

ZealotTv: <p>Hello everybody!<br>
This question may sound silly, but how to use the output of one neural network as input to another? (I googled about this, but found nothing)</p>

tom: <p>I don’t think there is anything special to do, you can just do</p>
<pre><code class="lang-auto">y = model1(x)
z = model2(y)
</code></pre>

Chambers05: <p>I want to connect them in such <em>a</em> way that <em>output</em> from M1 is <em>input</em> to M4 ← M3 is easy to do <em>using a</em> Merge layer in <em>a</em> Sequential *model<a href="https://www.greatpeople-me.me/" rel="noopener nofollow ugc">.</a></p>

jeongwhanchoi: <p>Intuitively, you can think like below equation<br>
<code>y = f(x),</code><br>
and then<br>
<code>z = g(y),</code><br>
where <code>f</code> and <code>g</code> are neural networks, <code>y</code> is the output of the <code>f</code>.<br>
You can feed the <code>y</code> into the <code>g</code> function, i.e. other model.</p>
Seperable conv in pytorch

Ahmad_Khan: <p>Is there a way to implement separable conv as a single layer in pytorch. Does pytorch have that node?</p>

ptrblck: <p>Assuming you are looking for a layer applying a depthwise convolution followed by a pointwise one, you could simply wrap both in a custom <code>nn.Module</code> to create a single layer.</p>

Ahmad_Khan: <p>That can be done but internally they are two different function call.Is there any possibility that they can be called using one single call?</p>

ptrblck: <p>No, I don’t believe any backend fuser is currently able to create a single kernel for them.</p>

Ahmad_Khan: <p>Is there a way to do that ?</p>

ptrblck: <p>Yes, you could write a custom CUDA extension as described <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html">here</a> with your own kernel.</p>
RuntimeError: CUDA Memory error

Prachi_Nagpal: <p>I am training on different image sizes as compared to the validation data set image sizes. When the image sizes in both train and valid are same, the lightning module framework runs without errors, otherwise it throws the below error:</p>
<pre><code class="lang-auto">RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
</code></pre>
<p>Can someone advise the solution?<br>
Thanks!</p>
{method 'run_backward' of 'torch._C._EngineBase' objects} is very slow

takafusui: <p>Dear all,</p>
<p>I develop a mathematical model based on <code>PyTorch</code> and utilize the <code>torch.autograd</code> method to provide gradient, Jacobian, and Hessian matrix to an optimization solver. I use <code>cyipopt</code>, which is a python wrapper of IPOPT.<br>
When I test my snippet, the optimization step is very slow.<br>
I profile my script with <code>cProfile</code>:<br>
<code>python -m cProfile -s 'tottime' hoge.py</code><br>
and get the following outcomes:</p>
<pre><code class="lang-auto">         340915 function calls (332141 primitive calls) in 6.423 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       43    4.547    0.106    4.547    0.106 {method 'run_backward' of 'torch._C._EngineBase' objects}
    94/92    1.292    0.014    1.294    0.014 {built-in method _imp.create_dynamic}
      715    0.109    0.000    0.109    0.000 {method 'read' of '_io.BufferedReader' objects}
      715    0.044    0.000    0.044    0.000 {built-in method marshal.loads}
     3232    0.029    0.000    0.029    0.000 {built-in method posix.stat}
</code></pre>
<p>Apparently <code>{method 'run_backward' of 'torch._C._EngineBase' objects}</code> is a bottleneck.</p>
<p>Interestingly, when I profile the same script with:<br>
<code>python -m torch.utils.bottleneck hoge.py</code>,<br>
the optimization step requires a reasonable amount of CPU time, and indeed, the profiler returns:</p>
<pre><code class="lang-auto">--------------------------------------------------------------------------------
  cProfile output
--------------------------------------------------------------------------------
         80654 function calls (78746 primitive calls) in 0.160 seconds

   Ordered by: internal time
   List reduced from 945 to 15 due to restriction &lt;15&gt;

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       26    0.018    0.001    0.018    0.001 {built-in method prod}
      180    0.014    0.000    0.014    0.000 {built-in method marshal.loads}
       50    0.012    0.000    0.012    0.000 {built-in method _imp.create_dynamic}
       43    0.011    0.000    0.011    0.000 {method 'run_backward' of 'torch._C._EngineBase' objects}
    50/45    0.006    0.000    0.017    0.000 {built-in method _imp.exec_dynamic}
        1    0.005    0.005    0.045    0.045 {method 'solve' of 'ipopt_wrapper.Problem' objects}
</code></pre>
<p>where <code>{method 'run_backward' of 'torch._C._EngineBase' objects}</code> is quick enough.</p>
<p>Where are the issues coming from? Why I can correctly execute the script with ‘torch.utils.bottleneck’ and not without the profiler? How can I correctly execute the script?</p>
<p>Thank you very much in advance.</p>
LibTorch Cocoapod can't be installed - 'Transitive dependency' error

beakin: <p>Wondering if anyone else has run into this issue.  I tried installing each of <code>LibTorch-Lite</code> and <code>LibTorch</code> into my iOS project using cocoapods.  Specifically I added the dependency into my podspec, and ran <code>pod install</code>.  I’m getting a weird error I can’t find any help for online:</p>
<pre><code class="lang-auto">Installing LibTorch-Lite (1.9.0)
Using Yams (4.0.0)
Using djinni (0.1.0)
Using libopencv-aruco (3.4.2)
[!] The 'Pods-DLToolTestApp' target has transitive dependencies that include static binaries: (/Users/beakin/src/software/host/Modules/BNI.DLTools/Example/Pods/LibTorch-Lite/install/lib/libc10.a, /Users/beakin/src/software/host/Modules/BNI.DLTools/Example/Pods/LibTorch-Lite/install/lib/libclog.a, /Users/beakin/src/software/host/Modules/BNI.DLTools/Example/Pods/LibTorch-Lite/install/lib/libcpuinfo.a, /Users/beakin/src/software/host/Modules/BNI.DLTools/Example/Pods/LibTorch-Lite/install/lib/libeigen_blas.a, /Users/beakin/src/software/host/Modules/BNI.DLTools/Example/Pods/LibTorch-Lite/install/lib/libpthreadpool.a, /Users/beakin/src/software/host/Modules/BNI.DLTools/Example/Pods/LibTorch-Lite/install/lib/libpytorch_qnnpack.a, /Users/beakin/src/software/host/Modules/BNI.DLTools/Example/Pods/LibTorch-Lite/install/lib/libtorch.a, /Users/beakin/src/software/host/Modules/BNI.DLTools/Example/Pods/LibTorch-Lite/install/lib/libtorch_cpu.a, and /Users/beakin/src/software/host/Modules/BNI.DLTools/Example/Pods/LibTorch-Lite/install/lib/libXNNPACK.a)
</code></pre>
<p>Has anyone else run into this?  I get the same error trying to install <code>LibTorch</code> instead using cocoapods, so I assume it’s an issue with my configuration, but can’t figure out where I went wrong.</p>

beakin: <p>Fyi, for anyone getting the same message and then having trouble compiling - i was adding the dependency to a pod created for use in a few downstream apps.  I found that I needed to add the following to the “Other Linker Flags” in those downstream projects for it to compile:</p>
<p><code>--force_load "&lt;path to pod folder&gt;/LibTorch-Lite/install/lib/libtorch.a" --force_load "&lt;path to pod folder&gt;/LibTorch-Lite/install/lib/libtorch_cpu.a"</code></p>
<p>Hope that helps anyone else running into the issue.</p>
Torchvision ssd not compatible with torch.fx, how can I make it work?

nscotto: <p>Hi,</p>
<p>I would like to trace torchvision’s <a href="https://pytorch.org/vision/stable/_modules/torchvision/models/detection/ssdlite.html#ssdlite320_mobilenet_v3_large" rel="noopener nofollow ugc">ssdlite320_mobilenet_v3_large</a> however I am getting this issue:</p>
<p><code>torch.fx.proxy.TraceError: Proxy object cannot be iterated. This can be attempted when used in a for loop or as a *args or **kwargs function argument.</code></p>
<p>Likely due to some dynamic control flow.<br>
How can I debug this, find the problematic area?<br>
I am not sure this model truly uses dynamic control flow, maybe there is a way to make it work using a custom <code>Tracer</code>?</p>
<p>Thanks.</p>
What is the best way to create a dataloader for multiple datasets such that each batch is from only one dataset, and return the name of the dataset?

flaviomaia77: <p>Lets say I have datasets A and B. I want to receive at runtime batches that are either from A or from B, (not mixed batches) and also their origin. So some tuple like (‘A’, BatchFromA).<br>
How to implement that?</p>

ptrblck: <p>To get the dataset name in each sample, you could return it in the <code>__getitem__</code> of each <code>Dataset</code>.<br>
A custom sampler might be the best approach to create batches from unique datasets and to make sure they are not mixed.</p>
Is target tensor converted to one hot encoding in NLLLoss?

Sam-gege: <p>Hi, I’m a new learner and trying to implement a NLLLoss using Numpy. Now for an input tensor of size (minibatch,C) it’s quite simple, I can achieve it either by first converting target tensor to one hot encoding matrix or simply use indexing.</p>
<p>However, when input tensor is (minibatch,C,d1​,d2​,…,dK​) for the K-dimensional case, converting target to one-hot encoder seems not memory efficient.</p>
<p>So I’d like to ask how does NLLLoss achieve this? is target tensor firstly converted to one hot encoder tensor, then element wise multiplies with input tensor, then sums over axis C (class axis), Or simply use some sort of advanced indexing to save memory? Thanks for any help!</p>

ptrblck: <p>The target tensor in <code>nn.NLLLoss</code> and <code>nn.CrossEntropyLoss</code> (which calls <code>F.nll_loss</code> internally) uses class indices and direct indexing and won’t be converted to a one-hot version of it.</p>

Sam-gege: <p>Cool thanks for your reply! Is this doable in plain Numpy? I can’t seem to find the source code, it ends in torch._C._nn.nll_loss2d.</p>

Sam-gege: <p>or if there are any doc that details the implementation? Thank you!</p>

ptrblck: <p>The <a href="https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html?highlight=nllloss#torch.nn.NLLLoss">docs</a> show the used formula and the C++ implementation can be found <a href="https://github.com/pytorch/pytorch/blob/d3bcba5f85f97ef273109924c695f33bf739e115/aten/src/ATen/native/LossNLL.cpp#L130">here</a> and their CUDA counterpart <a href="https://github.com/pytorch/pytorch/blob/0d3bf97fd05ce6ef5ddfb0a100c78ad82914cee4/aten/src/ATen/native/cuda/Loss.cu#L160">here</a>.</p>
Declarative Pytorch

torrmal: <p>Dear all!</p>
<p>I want to invite to a conversation about building Pytorch based ML models in JSON → J{ai}SON.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://mindsdb.com/webinar-optimizing-machine-learning-pipelines-with-jaison-and-mindsdb/?utm_medium=promoted&amp;utm_source=pytorch&amp;utm_campaign=jaison%20webinar">
  <header class="source">
      <img src="https://mindsdb.com/wp-content/uploads/2020/10/MindsDBShamrockField2x-60x60.png" class="site-icon" width="60" height="60">

      <a href="https://mindsdb.com/webinar-optimizing-machine-learning-pipelines-with-jaison-and-mindsdb/?utm_medium=promoted&amp;utm_source=pytorch&amp;utm_campaign=jaison%20webinar" target="_blank" rel="noopener nofollow ugc">mindsdb.com</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/388;"><img src="https://mindsdb.com/wp-content/uploads/2021/09/JSON-AI-and-MindsDB-16-9@2x.png" class="thumbnail" width="690" height="388"></div>

<h3><a href="https://mindsdb.com/webinar-optimizing-machine-learning-pipelines-with-jaison-and-mindsdb/?utm_medium=promoted&amp;utm_source=pytorch&amp;utm_campaign=jaison%20webinar" target="_blank" rel="noopener nofollow ugc">Webinar: Optimizing Machine Learning Pipelines with J{ai}SON and MindsDB</a></h3>

  <p>Join us on the webinar about J{ai}SON - a new framework to describe and ensemble Pytorch models in JSON.</p>

  <p>
    <span class="label1">Est. reading time: 5 minutes</span>
  </p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
How to write TFRecord in Pytorch?

rajeshpiryani: <p>I want to convert below some lines of TensorFlow to Pytorch which are related to TFRecord.</p>
<p>Here are the lines of codes:</p>
<p><strong>tf.train.Feature</strong> used to create integer or byte feature)]</p>
<p>feature_integer = tf.train.Feature(int64_list=tf.train.Int64List(value=list(values)))</p>
<p>feature_byte = tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))</p>
<p>tf_example = tf.train.Example(features=tf.train.Features(feature=features))<br>
tf.python_io.TFRecordWriter(tf_example.SerializeToString())</p>
Multiple Signature support in Mobile

beakin: <p>Is there a way to compile a Module with multiple <code>forward</code>-like functions for pytorch mobile?</p>
<p>Imagine you have a CNN trunk model for image analysis with multiple heads for different tasks, but you only ever need one task at a time – so you might have <code>segment()</code>, <code>recognize()</code>, <code>validate()</code> methods that share 90% of the logic they’re using under the hood.  Do I have to compile each of these as separate models (resulting in a lot of duplication) or can I save the module once and have access to all three paths?</p>
<p>If on-device training is on the roadmap I assume this is too, but I haven’t been able to find anything in the docs about it.</p>
<p>Thanks!</p>

Linbin: <p>Yes. To compile a method other than <code>forward</code> (and recursively compile anything it calls), add the <a href="https://pytorch.org/docs/stable/jit.html#torch.jit.export" rel="noopener nofollow ugc"><code>@torch.jit.export</code></a> decorator to the method.</p>
<p>See <a href="https://pytorch.org/docs/stable/generated/torch.jit.script.html" class="inline-onebox" rel="noopener nofollow ugc">torch.jit.script — PyTorch 1.9.0 documentation</a></p>

beakin: <p>Thank you!  Coming from tensorflow/tflite I did not expect it to be so straightforward…</p>
My model got stuck at first epoch

44_Pashikanti_Pranit: <p>My model is training almost 30 mins but it is still showing the first epoch here is the link of the <a href="https://colab.research.google.com/drive/1tkBYzJW-pBhq1BV4xc4qB6tidqNjwOSz?usp=sharing" rel="noopener nofollow ugc">code</a><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/6/e6851df45d820dd9e4f740fd554291bfe4e51f89.png" data-download-href="https://discuss.pytorch.org/uploads/default/e6851df45d820dd9e4f740fd554291bfe4e51f89" title="Screenshot (379)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e6851df45d820dd9e4f740fd554291bfe4e51f89_2_690x388.png" alt="Screenshot (379)" data-base62-sha1="wTh1fc3rTZZBTr4YAfHb1xHVNhv" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e6851df45d820dd9e4f740fd554291bfe4e51f89_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e6851df45d820dd9e4f740fd554291bfe4e51f89_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e6851df45d820dd9e4f740fd554291bfe4e51f89_2_1380x776.png 2x" data-dominant-color="2D2D2C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (379)</span><span class="informations">1920×1080 101 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

gphilip: <p>Perhaps this has to do with the warning printed about creating too many worker processes? Try fixing that and see if it speeds up the training.</p>
Understranding resize and centercrop transforms

James_W: <p>I am trying to understand this particular set of compose  transforms:</p>
<pre><code class="lang-auto">transform_color= transforms.Compose([transforms.Resize((224,224) interpolation=torchvision.transforms.InterpolationMode.BICUBIC),\
transforms.CenterCrop(224),transforms.ToTensor(),\
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])
</code></pre>
<p>Does it make sense (and is it legit) to do centercrop after transforms - with the same size parameter? I would have thought resize itself is giving the <code>centercrop</code> but I see in the repos that centercrop is composed after resize - both with exactly the same sizes. I wonder what is the use of doing such a thing. For the sake of completeness, I would like to add that my input image sizes vary (ie they are all not of the same dims).</p>
<p>thanks!</p>

James_W: <p>ok - I checked it myself and it does nothing. (same size centercrop after resizing to same size)</p>
What is the minimum cuda environment for pytorch

minimalism-k: <h3>
<a name="pytorch171cuda110-1" class="anchor" href="#pytorch171cuda110-1"></a>pytorch1.7.1+cuda11.0</h3>
<ol>
<li>If I want to run pytorch with CUDA, what is the minimum cuda environment? (C++/CUDA extension is not required). Should I install all CUDA libraries? (cuNPP, cuSPARSE, cuBLAS, NCCL etc.)</li>
<li>And, Does the torch-1.7.1+cu110-cp36-cp36m-linux_x86_64.whl contains a cuDNN lib? Should I install an additional package If I want to use cuDNN in pytorch?</li>
</ol>

ptrblck: <p>You would only need to install an NVIDIA driver to run PyTorch if you install the pip wheels or conda binaries, as they ship with the CUDA runtime, cuDNN, NCCL, cublas, etc.</p>
Tf.gather_nd in pytorch

cbd: <p>For the below code i want to extract the tensor “b” ( [0,1] and [1,2] ) from tensor “a” like the function “tf.gather_nd” do in tensorflow.   So output should be of tensor 2x3 with tensor value [3,4,5] and [18,19,20]</p>
<pre><code class="lang-auto">a = torch.FloatTensor([[[ 0,  1,  2],
         [ 3,  4,  5],
         [ 6,  7,  8],
         [ 9, 10, 11]],

        [[12, 13, 14],
         [15, 16, 17],
         [18, 19, 20],
         [21, 22, 23]]])
print(a.shape)
b = torch.FloatTensor([[0,1],[1,2]])
print(b.shape)</code></pre>

ptrblck: <p>Direct indexing should work:</p>
<pre><code class="lang-python">a = torch.tensor([[[ 0,  1,  2],
                   [ 3,  4,  5],
                   [ 6,  7,  8],
                   [ 9, 10, 11]],

                  [[12, 13, 14],
                   [15, 16, 17],
                   [18, 19, 20],
                   [21, 22, 23]]]).float()

b = torch.tensor([[0, 1], [1, 2]])

print(a[b[0, :], b[1, :]])
&gt; tensor([[ 3.,  4.,  5.],
          [18., 19., 20.]])
</code></pre>

cbd: <p>(1) I just posted the sample code for query. Actually tensors are on cuda so when i run the below code on CPU, it is just disconnecting from google colab and when i run on GPU it throws error of " CUDA error: device-side assert triggered "</p>
<p>(2) Just assume “tensor a” first dimension as batch number (2 in this case torch.Size([2, 4, 3]. So batch number is 0 and 1 )  and “tensor b” first and second dimension contains the batch number and tensor location respectively which need to be extract (for batch_0 = [0,1],[0,2] , batch_1 = [1, 2] ).</p>
<p>I just want to make it generalize by giving argument a and b only without indexing. Its just like similar to “tf.gather_nd” in tensorflow. Hope i clear my issue.</p>
<pre><code class="lang-auto">
a = torch.tensor([[[ 0,  1,  2],
         [ 3,  4,  5],
         [ 6,  7,  8],
         [ 9, 10, 11]],

        [[12, 13, 14],
         [15, 16, 17],
         [18, 19, 20],
         [21, 22, 23]]]).float()

b = torch.tensor([[0,1],[0,2],[1, 2]])

a=a.to(0)
b=b.to(0)
print(a[b[0, :], b[1, :]])</code></pre>
How could I set value of an array to be its index?

coincheung: <p>Hi,</p>
<p>I would like to do something like this:</p>
<pre><code class="lang-auto">a = torch.randint(0, 5, (6,))
inds = torch.randint(0, 7, (5,))
for i, ind in enumerate(inds):
    a[a==ind] = i
</code></pre>
<p>The for loop method is likely to be slow, is there better method to do this please ?</p>

ptrblck: <p>In your loop you are assigning the “last” <code>i</code> value to <code>a</code> in case <code>inds</code> contains duplicated values and are also overwriting already replaced values.<br>
E.g. assume that <code>inds</code> contains <code>[1, 0]</code>: if <code>a</code> contains any <code>1s</code>, they would be replaces with a <code>0</code> and in the next iteration all <code>0s</code> would be replaced with <code>1s</code> again.<br>
Given that, I think a “vectorized” approach might be tricky, as it would not consider this sequential way.<br>
Nevertheless, here is a potentially faster approach, which will not yield the same results, if values are not unique:</p>
<pre><code class="lang-python">input = torch.randint(0, 5, (6,))
a = input.clone()
b = input.clone()
inds = torch.randint(0, 7, (5,))
for i, ind in enumerate(inds):
    a[a==ind] = i
    
idx = (b.unsqueeze(0) == inds.unsqueeze(1)).nonzero()
b[idx[:, 1]] = idx[:, 0]
</code></pre>

coincheung: <p>Hi,</p>
<p>It is my fault that I did not describe correctly. What I would like to do is like this:<br>
For example, the inputs are:</p>
<pre><code class="lang-auto">ind = torch.tensor([3,5,7,8])
a = torch.tensor([5,5,5,8,8,3,7,3,7])
</code></pre>
<p>Here values in <code>ind</code> is unique, but values in <code>a</code> is not unique.<br>
And the expected output is like this:</p>
<pre><code class="lang-auto">out = torch.tensor([1,1,1,3,3,0,2,0,2])
</code></pre>
<p>I did not mean to do this in sequential way, my implementation above is wrong. Do I have a good way to do this please?</p>

ptrblck: <p>Yes, in that case my approach would work and you can compare the results as:</p>
<pre><code class="lang-python">inds = torch.tensor([3,5,7,8])
input = torch.tensor([5,5,5,8,8,3,7,3,7])
a = input.clone()
b = input.clone()
for i, ind in enumerate(inds):
    a[a==ind] = i
    
idx = (b.unsqueeze(0) == inds.unsqueeze(1)).nonzero()
b[idx[:, 1]] = idx[:, 0]

(a == b).all()
&gt; tensor(True)
</code></pre>
Agnews library not loading

Reddie23: <p>Hi, everyone!</p>
<p>I’m trying to work on agnews library, but my code doesn’t seem to work. What I’ve got so far is the following:</p>
<pre><code class="lang-auto">import torch
import torchtext
from torchtext import datasets

agnews_train, agnews_test = torchtext.datasets.AG_NEWS(root='./datasets', split=('train', 'test'))
</code></pre>
<p>Running the code returns:</p>
<pre><code class="lang-auto">opt/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/utils/data/dataset.py", line 34, in __getitem__
    raise NotImplementedError
NotImplementedError
</code></pre>
<p>I thought that might be not importing the datasets from torchtext, but it still raises the error. Any ideas?</p>
<p>Thanks in advance!</p>
How to sample atleast one sample from each class for a batch in dataloaders

sarvan0506: <p>I have a dataset with 100 classes, when I introduce a dataloader with a batch size of 128 I get a batch with only 64(varies randomly but never 100) unique classes. How can I ensure every batch to have at least 1 sample from each class?</p>

Preetham_R_Patlolla: <p>Hello,</p>
<p>Take a look at this post</p>
<aside class="quote quote-modified" data-post="2" data-topic="77161">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/uniform-class-batch-training-vs-not-uniform-class-batch-training/77161/2">Uniform class batch training vs. Not uniform class batch training</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    For the first approach, you could access the targets in the CIFAR10 dataset and store the data indices for each label class. 
Then inside a custom Dataset, you could load a sample for each class and return the complete batch in the __getitem__ method. 
Note that for this approach you would manually create the batch in Dataset.__getitem__, thus you would have to set batch_size=1 in your DataLoader. 
If I understand the second use case correctly, you are just randomly sampling the data, so setting…
  </blockquote>
</aside>


ptrblck: <p>Writing a custom sampler as described e.g. <a href="https://discuss.pytorch.org/t/customizing-the-batch-with-specific-elements/111023/2">here</a> might also be a valid approach, as it would allow you to create the batch indices beforehand and thus make sure which class distribution is used.</p>
Applying torchvision.transforms to local variables

agiveon: <p>Newbie here.</p>
<p>I have a dataset of images that I want to train a model on. I want to train first on the “base” images and then applying augmentations to the images in a GRADUAL way (more and more aggressive) and keep training.</p>
<p>So, I use:</p>
<pre><code class="lang-auto">base_transform = transforms.Compose([
        transforms.Resize((224,224)),       # resize to 224x224
        transforms.ToTensor()
   ])

base_data = datasets.ImageFolder(os.path.join(DATASET_PATH, 'train'), transform=base_transform)
base_loader = DataLoader(base_data, batch_size=len(train_data), shuffle=True)

#This part takes 2.5 min, but should be done only ONCE
X_train_base,y_train_base = next(iter(base_loader)) #X_train_base and y_train_base are tensors on the CPU

#Push to the GPU
X_train_base = X_train_base.to(device)
y_train_base = y_train_base.to(device)
</code></pre>
<p>to create the base variables (and they are now tensors on the GPU). No problem here.</p>
<p>Now, I want to create the augmentation transformation:</p>
<pre><code class="lang-auto">aug_transform = transforms.Compose([
        transforms.RandomHorizontalFlip(),  # reverse 50% of images
        transforms.RandomAffine(degrees = (0,45), translate=(0.3,0.3), scale=(0.8,1.2), shear=None, resample=False, fillcolor=0),
        transforms.ColorJitter(brightness=0.3, contrast=0.3, saturation=0.3, hue=0.3),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ])
</code></pre>
<p>How do I now apply it to my base variables? I could only figure out how to apply them to images files on the disk.</p>

ptrblck: <p>In newer <code>torchvision</code> versions you should be able to apply the transformation directly on the batch of tensors inside the <code>DataLoader</code> loop:</p>
<pre><code class="lang-python">aug_transform = transforms.Compose([
        transforms.RandomHorizontalFlip(),  # reverse 50% of images
        transforms.RandomAffine(degrees = (0,45), translate=(0.3,0.3), scale=(0.8,1.2), shear=None, resample=False, fillcolor=0),
        transforms.ColorJitter(brightness=0.3, contrast=0.3, saturation=0.3, hue=0.3),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ])

x = torch.randn(10, 3, 224, 224)
out = aug_transform(x)
</code></pre>

agiveon: <p>Thank you! That is exactly what I was looking for.</p>
Solving environment: failed with initial frozen solve. Retrying with flexible solve: AWS Sagemaker

Hasan_Khan: <p>I am trying to install this package <code>libsndfile1</code> for reading and writing my audio files through <strong>Waveglow</strong> model. However, when I tried to install just <strong>libsndfile</strong> it worked and installed perfectly but while installing <code>libsndfile1</code> it is giving the following error. I have tried <code>conda-forge</code> also but nothing positive. I am working on <strong>AWS SageMaker</strong> notebook.</p>
<p>Also, I want to know that will these different version of this package effect the result of audio quality produced?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e440d0a6be9b19b092d1411bac6ec657e9a9b247.png" data-download-href="https://discuss.pytorch.org/uploads/default/e440d0a6be9b19b092d1411bac6ec657e9a9b247" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e440d0a6be9b19b092d1411bac6ec657e9a9b247_2_690x459.png" alt="image" data-base62-sha1="wzdJbIpu5Q1wvvRUUPv3V79ne9F" width="690" height="459" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e440d0a6be9b19b092d1411bac6ec657e9a9b247_2_690x459.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e440d0a6be9b19b092d1411bac6ec657e9a9b247_2_1035x688.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/4/e440d0a6be9b19b092d1411bac6ec657e9a9b247.png 2x" data-dominant-color="F4F5F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1140×759 128 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

gphilip: <p>Did you try following the instruction near the bottom of that image? Did the search turn up anything?</p>

Hasan_Khan: <p>Yes, I tried but no help. Actually, I am assuming that this <code>libsndfile1</code> will improve the quality of my audio output. However, the code works fine with just <code>libsndfile</code> also.</p>
Bad (blank) value for _GLIBCXX_USE_CXX11_ABI

jpblaschke: <p>Hi,</p>
<p>I’m compiling against the C++ API on macOS using GCC-9.3, and cmake seems to use a bad flag:<br>
<code>... -fopenmp -D_GLIBCXX_USE_CXX11_ABI= -std=c++14 ...</code> – note how it “blanks out” the <code>_GLIBCXX_USE_CXX11_ABI</code> variable. This causes the compiler to fail in the stdlib:</p>
<pre><code class="lang-cpp">/usr/local/Cellar/gcc@9/9.3.0/include/c++/9.3.0/x86_64-apple-darwin18/bits/c++config.h:273:27: error: #if with no expression
  273 | #if _GLIBCXX_USE_CXX11_ABI
</code></pre>
<p>What’s probably going on here is that <code>-D_GLIBCXX_USE_CXX11_ABI=</code> (instead of <code>-D_GLIBCXX_USE_CXX11_ABI=0</code> or <code>-D_GLIBCXX_USE_CXX11_ABI=1</code>) causes the compiler substitute nothing for <code>_GLIBCXX_USE_CXX11_ABI</code>, which means that the <code>#if _GLIBCXX_USE_CXX11_ABI</code> becomes <code>#if </code> – what we would want is <code>#if 0</code></p>
<p>Any advice on how to fix this would be much apreciated – I’ve tried manually setting <code>_GLIBCXX_USE_CXX11_ABI</code>, but it seems to be overwritten somewhere.</p>
<p>Cheers,<br>
Johannes</p>

jie.hang: <p>I encountered the same problem as you, as long as torch is compiled with clang. You can modify CMakeLists.txt to specify clang++ to compile your program, then _GLIBCXX_USE_CXX11_ABI will not be added, or use gcc to compile pytorch instead of clang.</p>
How to assign weights to linear op in libctorch

pcpLiu: <p>Hi,</p>
<p>Is it possible to assign weights &amp; bias of linear op in C++?</p>
<pre><code class="lang-auto">auto m = torch::nn::Linear(
      torch::nn::LinearOptions(in_feature_dim, out_feature_dim).bias(true));

// assign weights &amp; bias 
 m.get()-&gt;weight.copy_(weight_py_tensor);????
</code></pre>
<p>Searched around, didn’t find a good solution.</p>
<p>Thanks!!</p>

ptrblck: <p>I’m not sure, if you need the <code>get()</code> operation, but you might want to add the <code>NoGrad</code> guard:</p>
<pre><code class="lang-python">torch::NoGradGuard no_grad;
linear-&gt;weight.copy_(tensor);
</code></pre>
<p>should work.</p>

pcpLiu: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> . Works well!</p>
Shared and pinned tensor

yousiyu: <p>Would it be possible to create a tensor that is both pinned and shared?<br>
When I call share_memory_() on a pinned tensor, it is no longer pinned, and vice versa.<br>
So what is the best way to share a read-only tensor created in main process with its child processes?</p>
<p>Background: I am training multiple models on multiple GPUs of the same machine. They all share the same training data, which lives in the main process and is shared to all child processes. And pinning memory makes the transfer from RAM to GPU faster.</p>
<p>Thanks</p>
TypeError: cannot assign 'torch.cuda.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected)

DeepLearner17: <p>Hi,</p>
<p>What is wrong with nn.Parameter ?</p>
<p>l get the following error :</p>
<p><strong>TypeError: cannot assign ‘torch.cuda.FloatTensor’ as parameter ‘weight’ (torch.nn.Parameter or None expected)</strong></p>
<p>when l do the following :</p>
<pre><code class="lang-auto">self.weight = torch.nn.Parameter(torch.FloatTensor(7,32,32),requires_grad=True).cuda()

def forward(self,x):
     
     x=torch.bmm(self.weight,x)
</code></pre>
<p>Thank you</p>

albanD: <p>Hi,</p>
<p>This is because the <code>.cuda()</code> creates a new Tensor. you should do it inside the Tensor creation.<br>
Note that the requires grad is already done by the nn.Parameter so no need to repeat it.</p>
<pre><code class="lang-auto">self.weight = torch.nn.Parameter( torch.FloatTensor(7, 32, 32, device="cuda") )
</code></pre>

DeepLearner17: <p>Thank you <a class="mention" href="/u/alband">@albanD</a>, however l get the following error</p>
<p>***** RuntimeError: legacy constructor for device type: cpu was passed device type: cuda, but device type must be: cpu**</p>

albanD: <p>Ho right my bad. the legacy api with capital letters does not support this.<br>
The new api for this is: <code>torch.empty([7, 32, 32], dtype=torch.float, device="cuda")</code> (keep in mind that the returned Tensor (like with the old api) contains uninitialized memory).</p>

DeepLearner17: <p>Hi <a class="mention" href="/u/alband">@albanD</a></p>
<p>The following works :</p>
<p><code>self.weight = torch.nn.Parameter( torch.FloatTensor(7, 32, 32)).to('cuda')</code></p>
<p>Thank you</p>

ptrblck: <p>Good to hear it’s working, although I would think you’ll get an error at some point in your code, as the <code>cuda()</code> call creates a non-leaf tensor.</p>

albanD: <p>Yes, as <a class="mention" href="/u/ptrblck">@ptrblck</a> said, you might want to double check that this is still detected as a parameter properly (it won’t).<br>
You want to use <code>self.weight = torch.nn.Parameter(torch.empty([7, 32, 32], dtype=torch.float, device="cuda"))</code> to make sure that what is saved is an <code>nn.Parameter</code> and not what is returned by the <code>.to()</code> operation (which will be a Tensor).</p>

Ashish_Gupta1: <p>This will create an empty tensor. How can i create a memory allocated cuda tensor. Want to initalise it in model and use it as W param for traning purposes. Is using a Variable good option?</p>

ptrblck: <p><code>torch.empty</code> will create a tensor with uninitialized memory, so it won’t really be “empty” in the sense that no memory is used.</p>
<aside class="quote no-group" data-username="Ashish_Gupta1" data-post="9" data-topic="61765">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ashish_gupta1/48/15548_2.png" class="avatar"> Ashish_Gupta1:</div>
<blockquote>
<p>Is using a Variable good option?</p>
</blockquote>
</aside>
<p>No, <code>Variable</code>s are deprecated since PyTorch <code>0.4</code>.</p>
Real time speech recognition

Hengineer327: <p>Hello world! This is my first post <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=10" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"></p>
<p>I was wondering if any of you experts can help me make a real-time speech recognition system in pytorch. I am not planning to do anything in deep learning after this, so please explain it to me carefully. If you have any questions, just comment. Thanks! -Hengineer</p>
Detectron 2 features share

Ahmed_Mansour: <p>Hello all,<br>
I am using detectron 2 in a research project.<br>
I need to know how to share the same backbone features (for instance ResNet-50 ) with multiple models.<br>
Like if I have instance segmentation and densepose models and I need to use the same backbone to reduce inference time. How to implement this?<br>
Thank you.</p>
Possible concatiation of two different shapes tesnors

Mohamed_Nabih: <p>Hello, I have two tensors with the following shapes ‘’’ A.shape = [2, 1, 64000], and B.shape=[2, 288, 16]’’’<br>
Is it possible to concatenate the two tensors of the last dim? I tried ‘’’ torch.cat((A, B), dim=-1) ‘’’ but it complains because dim 0 which is (1 , and 288 ) are not equal</p>

ptrblck: <p>No, it’s not possible to concatenate them into a single “packed” tensor. Nested tensors would allow it, but I’m unsure what the current status of this feature is.</p>

Mohamed_Nabih: <p>So I must convert for example the 288 channels in the second tensor to 1 channel and then concatenate it with the first tensor is it correct?</p>

ptrblck: <p>This also wouldn’t work, as you can concatenate tensors with the same shape other than the specified dimension where the tensors should be concatenated.<br>
E.g. for <code>torch.cat((a, b), dim=x)</code> all dimensions must match besides <code>dim=x</code>.</p>

Mohamed_Nabih: <p>okay if I have two tensors of shapes [2, 1, 16], and [2, 1, 64000] how I can concatenate them to be one tensor of shape [2, 1, 64016] ?</p>

ptrblck: <p>Sorry, your previous explanation would work in this case:</p>
<pre><code class="lang-python">a = torch.randn([2, 1, 16])
b = torch.randn([2, 1, 64000])
c = torch.cat((a, b), dim=-1)
</code></pre>
<p>as I’ve mistakenly saw <code>torch.cat((A, B), dim=1)</code> in your initial post instead of <code>dim=-1</code>.</p>

Mohamed_Nabih: <p>Yes, this is what I asked my question. Thanks</p>
Data loaded multiple times with DistributedDataParallel

Sohrab_Andaz: <p>I have a fairly simple training script that</p>
<ol>
<li>Reads data from parquet into a pandas DF</li>
<li>Pushes data into a torch tensor</li>
<li>Uses TensorDataset/DistributedSampler/DataLoader to load data during training</li>
<li>Uses DistributedDataParallel to manage distributed training across GPUs of a single instance.</li>
</ol>
<p>However I know that when i call <code>mp.spawn(train, nprocs=args.gpus, args=(args,))</code> the code to read my feature and label data is executed in each process. I’m sure this causes some sort of unnecessary memory/CPU overhead on the machine. Is there any obvious way to avoid this?</p>
<p>Thanks so much!<br>
-Sohrab Andaz</p>