akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Data loader disk and RAM access

Sumit_Diware: <p>Hello all,</p>
<p>Consider a MNIST dataloader with batch size 128:<br>
train_loader = data.DataLoader( datasets.MNIST(’./data’, batch_size=128, shuffle=True, train=True, transform=transforms.ToTensor())</p>
<p>When a batch of 128 images is processed during training, will this data loader always need to go to the disk for fetching the next batch of 128 images into the RAM?</p>
<p>In case it has to go to the disk every time, how can we fetch it all at once (as MNIST is a small dataset where most of it could fit directly into RAM) and keep it in the RAM until all training iterations are complete – assuming that the training still needs batch size 128?</p>
<p>Thanks and Regards,<br>
Sumit</p>

ptrblck: <p>The <code>data</code> and <code>target</code> tensors in MNIST will be directly stored into the RAM, as they are quite small, as seen in <a href="https://github.com/pytorch/vision/blob/f5843099d895e72c27ffa9d29cc91dd8df7f3832/torchvision/datasets/mnist.py#L82-L93">these lines of code</a>.</p>

Sumit_Diware: <p>Thank you very much for the answer and the code reference.</p>
<p>Just to follow up, how is the data and target storage in RAM managed by the dataloder when the datasets become bigger e.g. CIFAR-100 (medium scale) to Imagenet (large scale). I mean what logic is used by the dataloader to figure out how much of it is to be kept in RAM, considering available memory and batch size? I somtimes need to use custom datasets (some small scale, some large), so such conceptual clarification and reference to some code parts will help me in creating an efficient dataloader for them based on their size.</p>

ptrblck: <p>It depends on the implementation of the corresponding dataset and you could check it in the source code (as done for MNIST using my link).<br>
E.g. the often used <code>ImageFolder</code> dataset uses lazy loading, to save memory, while the CIFAR datasets also load the data and target into the RAM as seen <a href="https://github.com/pytorch/vision/blob/f5843099d895e72c27ffa9d29cc91dd8df7f3832/torchvision/datasets/cifar.py#L79-L91">here</a>.</p>
<p>Custom <code>Dataset</code> implementations can be written in either way, since you are defining how the data is loaded.</p>

Sumit_Diware: <p>Thanks a lot for your fast responses. That was really helpful. Cheers!</p>
Padding other value (i.e., 255 instead of 0) in Pytorch grid_sample

Yongjie_Shi: <p>Hi, I wonder how to pad other values in the grid_sample function instead of 0, could you give me some advice?</p>

Yongjie_Shi: <p>I think I get a solution. We can firstly map the tensor through y = x-255, when we apply the grid_sample for y, we can use  z = y+255 to get the padding effect of 255</p>
Why is pytorch's result faster?

111480: <p>I compared Pytorch to Pytorch JIT<br>
Before inference, I did warm-up 4 times.</p>
<p>Pytorch load:</p>
<pre><code class="lang-auto">models.resnet152(pretrained=True).cuda().eval()
</code></pre>
<p>Pytorch JIT load :</p>
<pre><code class="lang-auto">torch.jit.load('./resnet152_model.pt').cuda().eval()
</code></pre>
<p>then both case are faster.<br>
The reason of result of pytorch JIT is that the optimized graphs are cached along with the bytecode by JIT compiler.<br>
But why is pytorch’s result faster?</p>
OCR CNN with RNN model can't torch.jit.script with dynamic batch size inputs?

luvwinnie: <p>I’m creating a Pytorch OCR model with a Resnet based feature extractor with both ctc classifier and attention based classifer outputs.<br>
It includes a BiLSTM layer that input from the resnet so I can recognize the text image. I’m trying to use the torchserve which the model need to be torch.jit.trace or torch.jit.script. I’m able to use the torch.jit.trace to create a traced model, however it shows the following errors when the input’s batch size not matching the batch size that being traced.</p>
<pre><code class="lang-auto">RuntimeError: self dim 0 must match batch2 dim 0
</code></pre>
<p>Then I try to use the torch.jit.script it shows the following error.</p>
<pre><code class="lang-auto">    276         method = getattr(self, 'build_' + node.__class__.__name__, None)
    277         if method is None:
--&gt; 278             raise UnsupportedNodeError(ctx, node)
    279         return method(ctx, node)
    280 

UnsupportedNodeError: GeneratorExp aren't supported:
  File "/home/usr/venv/lib/python3.7/site-packages/torch/nn/modules/rnn.py", line 158
        # alias would break the assumptions of the uniqueness check in
        # Module.named_parameters().
        unique_data_ptrs = set(p.data_ptr() for p in self._flat_weights)
                               ~ &lt;--- HERE
        if len(unique_data_ptrs) != len(self._flat_weights):
            return
</code></pre>
<p>How can I solve this problem?</p>

Kushaj: <p>This might be due to an unsupported operand in torchserve. There is an open <a href="https://github.com/pytorch/pytorch/issues/46375" rel="noopener nofollow ugc">PR</a> that might be related to your problem.</p>

Darden403: <p>Dynamic Programming in Hidden Markov Models.11.5 Minibatch Stochastic Gradient Descent . In this book, we will teach most concepts just in time. 3.x you would download the bash script with strings “Miniconda3” and other considerations) the type, size, and the number of inputs and outputs.</p>

Darden403: <aside class="quote no-group quote-modified" data-username="Darden403" data-post="3" data-topic="122016" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/58f4c7/48.png" class="avatar"> Darden403:</div>
<blockquote>
<p>Dynamic Programming in Hidden Markov Models.11.5 Minibatch Stochastic Gradient Descent . In this book, we will teach most concepts just in time. 3.x you would download the bash script with strings “Miniconda3” and other considerations) the type, <a href="https://www.walgreenslistens.live/" rel="noopener nofollow ugc">walgreenslistens</a> and the number of inputs and outputs.</p>
</blockquote>
</aside>
<p>Thanks for the information</p>
How to create a PadNode

bucky: <p>/// Creates a PadNode with the given \p name and output type \p outTy which<br>
/// pads the given \p input with the explicit pads \p pads according to the<br>
/// padding mode \p mode and with the given value \p value. The padding mode<br>
/// \p mode is one of enumeration values from \ref PaddingMode. For an input<br>
/// with N dimensions (rank N) the \p pads must be a vector with 2*N values<br>
/// with the following format:<br>
/// pads = [pad_before(D1), pad_before(D2), …, pad_before(DN),<br>
///         pad_after (D1), pad_after (D2), …, pad_after (DN)].<br>
/// The mode PaddingMode::CONSTANT pads the input using the constant value<br>
/// \p value and currently is the only mode supported.</p>
<pre><code class="lang-auto">  PadNode *createPad(llvm::StringRef name, NodeValue input, TypeRef outTy,
                     unsigned_t mode, llvm::ArrayRef&lt;int&gt; pads, float value);
</code></pre>
<p>I am trying to pad a placeholder node that is of dimensions (NHWC) 16X5X5X1 to 16X5X5X4</p>
<p>I am able to create a PadNode and add it to the graph(I crosschecked it by dumping the DAG), but I am hitting a segmentation fault while generating IR.</p>
<p>Am I passing the correct values to /p pads parameter?</p>
<pre><code class="lang-auto">                                int pads[] = {0, 0, 0, 0, 0, 0, 0, 3};

                                auto filterOutTy = F-&gt;getParent()-&gt;uniqueType(ElemKind::FloatTy, filterOutPadDims);
                                auto inputOutTy = F-&gt;getParent()-&gt;uniqueType(ElemKind::FloatTy, inputOutPadDims);

                                auto *INPUTPAD = F-&gt;createPad("pad", nodeValue0, inputOutTy, PaddingMode::CONSTANT, pads, 0.f);
</code></pre>

jfix: <p>I think it looks reasonable, though I don’t see what <code>inputOutPadDims</code> is. Have you tried running in assert mode, to see if it hits an assertion instead of seg faulting? Or have you tried running in a debugger to see where the seg fault is?</p>

bucky: <blockquote>
<p>Unhandled node; perhaps the node should have been lowered, or the backend should have specified an IRGen case for this node to a backend-specific Instr.</p>
</blockquote>
<p>Hey <a class="mention" href="/u/jfix">@jfix</a> ! Thanks for the reply. so I tried compiling with g++ instead of clang, then I hit the above assertion instead of a segfault. <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>
<p>I was using a mock backend, and tried to add the padNode in “transformpostlowering” looks like the node should be lowered. Thanks for the Help.</p>
How can I install all the package of pytorch for my neural network model?

ziva: <p>I am new to PyTorch. I want to install all the packages in the spyder. Can someone help me to give me all the commands or the link from where I can install everything?</p>

Kushaj: <ol>
<li>Download anaconda <a href="https://www.anaconda.com/products/individual#Downloads" rel="nofollow noopener">link</a>
</li>
<li>Go to anaconda-navigator, and install spyder.</li>
<li>Install pytorch in conda environment <a href="https://pytorch.org/" rel="nofollow noopener">link</a>
</li>
</ol>

ziva: <p>Thankyou so much <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Can you explain me the details on CUDA?</p>

Kushaj: <p>On pytorch install page, select cuda and the install command will do everything for you.</p>

ziva: <p>ImportError: cannot import name ‘USE_GLOBAL_DEPS’ from ‘torch._utils_internal’ (C:\ProgramData\Anaconda3\lib\site-packages\torch_utils_internal.py)</p>
<p>I am facing this error</p>

Kushaj: <p>Try changing directory. You have to provide the complete process that got you to the error, it is difficult to tell without that.</p>

Antony_Joy: <hr>
<p>ImportError                               Traceback (most recent call last)<br>
 in <br>
----&gt; 1 import torch</p>
<p>~\anaconda3\lib\site-packages\torch_<em>init</em>_.py in <br>
21<br>
22 from ._utils import _import_dotted_name<br>
—&gt; 23 from ._utils_internal import get_file_path, prepare_multiprocessing_environment, <br>
24     USE_RTLD_GLOBAL_WITH_LIBTORCH, USE_GLOBAL_DEPS<br>
25 # TODO(torch_deploy) figure out how to freeze version.py in fbcode build</p>
<p>ImportError: cannot import name ‘USE_GLOBAL_DEPS’ from ‘torch._utils_internal’ (C:\Users\ANTHONY\anaconda3\lib\site-packages\torch_utils_internal.py)</p>
<p>I am having this error. I reinstalled it but, same error persists</p>
How to train a subset of model weights using masks while using AMP in PyTorch (using NVIDIA DL/Resnet implementation)?

Rio1210: <p>Hello all, I am trying to train a Resnet50 model on imagenet. My requirement is that I want to only train a subset of weights using a mask. I have successfully done this with general resnet models on cifar10 using the following sequence of code:</p>
<pre><code class="lang-auto">        optimizer.zero_grad()
        loss.backward()
        for name, p in model.named_parameters():
            if 'weight' in name:
                p.grad.data = p.grad.data * masks[name]
        optimizer.step()
</code></pre>
<p>Where <code>masks</code> is a dictionary containing layerwise masks with values of 1s and 0s.</p>
<p>Now, currently I am trying to incorporate the same idea for distributed imagenet training. I am using nVidia’s Resnet repo which utilizes AMP. I tried with including the following codeblock in nVidia’s implementation in this function in line 209 <a href="https://github.com/NVIDIA/DeepLearningExamples/blob/8d8c524df634e4dfa0cfbf77a904ce2ede85e2ec/PyTorch/Classification/ConvNets/image_classification/training.py#L184" rel="noopener nofollow ugc">here</a>:</p>
<pre><code class="lang-auto">        scaler.scale(loss).backward()

        for name, p in model_and_loss.named_parameters():
            if 'weight' in name:
                p.grad.data = p.grad.data * masks[name]
</code></pre>
<p>However, with the above code, still all of the weights are getting trained after gradient step. The optimizer step is done with (I think) <a href="https://github.com/NVIDIA/DeepLearningExamples/blob/8d8c524df634e4dfa0cfbf77a904ce2ede85e2ec/PyTorch/Classification/ConvNets/image_classification/training.py#L284" rel="noopener nofollow ugc">here</a>:</p>
<pre><code class="lang-auto">        optimizer_step = ((i + 1) % batch_size_multiplier) == 0
        loss = step(input, target, optimizer_step=optimizer_step)
        if ema is not None:
            ema(model_and_loss, epoch*steps_per_epoch+i)
</code></pre>
<p>I am unsure, how to achieve this in this nvidia implementation, also unsure how the gradient step is being taken.</p>
<p>My goal was to zero-out a subset of the gradients before taking the optimizer.step(), however with AMP it does not seem to be straightforward, specially in nvidia’s repo.</p>
<p>My question is, is there a different way to accomplish this in the training code <a href="https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/Classification/ConvNets/image_classification/training.py#L184" rel="noopener nofollow ugc">here</a>.</p>
<ol start="2">
<li>Moreover, during AMP gradient update, is the same gradients from parameters.grad are used or am I missing something?</li>
</ol>
<p>Thanks a lot for reading this. Please let me know if I need to clarify something here, I hope to be quick to response. I appreciate the help.</p>
How to use multiple models to perform inference on same data in parallel?

JimW: <p>I hope to perform the ensemble inference on a same validation data on multiple GPUs (i.e. 4 GPUS).<br>
Originally, there was some data parallellism in this framework, and if I just used one single model for inference, it worked well with utility for all 4 GPUs  above 85%.</p>
<p>But if I tried to use 2 models to do the inference, it got much slower and both GPU and CPU utility dropped to 25%.  I think it must be caused by my not using the correct method to parallel it (I am using a for loop here):</p>
<pre><code class="lang-auto">##### The is for the evaluation ######
pretrained_models = ['model1', 'model2'] 
pool = []
for i, cur_model in enumerate(pretrained_models):
    prediction = prediction_dict[cur_model]
    pool.append(prediction.unsqueeze(0))
    if(i==len(pretrained_models)-1):
         tmp = torch.cat(pool)
         ensemble_pred = tmp.mode(dim=0).value
         my_metric_save(ensemble_pred);
</code></pre>
<p>The basic idea is, assuming we already have the prediction vectors obtained from both pretrained models,  I am using a for-loop to extract them one after another, and finally combined them together as a new prediction vector “ensemble_pred”.  I don’t know how to profile the runtime, but this probably destroyed the original parallel flow, so it slowed down the validation dramatically.</p>
<p>Could someone provide some guidance on what is the efficient way to do ensemble inference (mulitple pretrained models to evaluate one same data)?</p>

eqy: <p>Depending on the relative computational cost of the models, it maybe difficult to parallelize them across multiple GPUs “simultaneously” and synchronize the models across each batch. Since this is for pretrained models, can you do the predictions in a more “offline” way, where the first model processes all the data followed by the second model with the predictions being aggregated after both models are done?</p>

JimW: <p>Thanks for the reponse. I will try to do that in an “offline” manner.</p>

wanchaol: <p>Yeah the original model might occupies multiple GPUs, in this case if you try to parallel to models to do the inference on the set of GPUs, there might be more synchronizations happens to ensure two model computation not interfering each other, which might reduce the GPU utility in general. You can try what <a class="mention" href="/u/eqy">@eqy</a> suggested. Also, if you like to know how your two models use GPU separately, you can do a profling for the two models separately with pytorch profiler, and see if there’re already operations occuping multiple GPUs.</p>
Downloading ImageNet

julianolm: <p>Hello, I am developing a model to apply on FMD (<a href="https://people.csail.mit.edu/lavanya/fmd.html" class="inline-onebox" rel="noopener nofollow ugc">Flickr Material Database</a>), but training on that same database just lead to 30% accuracy. Now I’m gonna pre-train the model on ImageNet, but don’t know how to do it. I haven’t yet even discovered how to download it in a simple way. How should I do it?</p>
<p>Also, since don’t have GPUs I am using Colab, wich has a small storage (64GB) in comparison to ImageNet’s size (think 150GB zipped). So I think my way is to upload the dataset to my edu drive account, wich has unlimited storage (as for now).</p>
<p>Thank you!</p>

eqy: <p>If you are just prototyping you might be able to just use a pretrained torchvision model:<br>
<a href="https://pytorch.org/vision/stable/models.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/stable/models.html</a></p>
<p>Downloading from the <a href="https://image-net.org" rel="noopener nofollow ugc">original website</a> is often tricky, but the <a href="https://academictorrents.com/collection/imagenet" rel="noopener nofollow ugc">academic torrents</a> version works well if you have a way to mount or access &gt; 150GB of storage.</p>
Torch::nn::functional::conv2d memory leak?

zheng_wang: <p>Hello, I encountered a memory leak when writing a convolutional layer using libtorch1.7GPU version. When I execute the code of torch::nn::functional::conv2d in a loop, the memory keeps growing. Excuse me, why is this?</p>
<pre><code class="lang-auto">torch::Tensor gaussian_filter(torch::Tensor x, int channels)
{
	/*
	*Parameters:
	* tensor x [C,H,W]  int channels 
	*Return tensor x [C,H,W]
	*/

	namespace F = torch::nn::functional;
	torch::Tensor kernel = 
		torch::tensor({ {0.00078633, 0.00655965, 0.01330373, 0.00655965, 0.00078633},
						{0.00655965, 0.05472157, 0.11098164, 0.05472157, 0.00655965},
						{0.01330373, 0.11098164, 0.22508352, 0.11098164, 0.01330373},
						{0.00655965, 0.05472157, 0.11098164, 0.05472157, 0.00655965},
						{0.00078633, 0.00655965, 0.01330373, 0.00655965, 0.00078633} });


	kernel = kernel.unsqueeze(0).unsqueeze(0);
	kernel = torch::repeat_interleave(kernel, channels, 0);
	torch::Tensor out_put = F::conv2d(x.unsqueeze(0), kernel.to(x.device()), F::Conv2dFuncOptions().stride(1).padding(2).groups(channels));
	return out_put.squeeze(0);

}
</code></pre>

ptrblck: <p>I assume you are running <code>gaussian_filter</code> in a loop?<br>
If so, are you storing the output tensor in each iteration?</p>

zheng_wang: <p>Execute <code>gaussian_filter</code> in a loop. I don’t want to store <code>torch::Tensor out_put</code>, I just want to get the return value and assign it to other tensors. Then destroy the memory of <code>torch::Tensor out_put</code>. But I found that the memory has been growing.Please tell me what should I do？</p>

ptrblck: <p>I’m still unsure if you are (accidentally) storing the tensors somehow or if they are attached to a computation graph, which could be stored in the assignment, so could you post an executable code snippet, which would reproduce this issue?</p>

zheng_wang: <p>Thank you very much, I solved this problem by canceling the gradient of the variables in the convolution operation. It may have been because of the calculation gradient that took up memory.</p>
Getting the gradients of global_model with .grad

Samiur_Rahman: <p>Hello, i am training a CNN model in federated learning settings. i update my <code>local_models</code> and then i average the <code>weights</code> i get from the <code>local_model</code> update to get the weight of my <code>global_model</code>. Since i use <code>loss.backward()</code> during my <code>local_model</code> training, i can get the gradients by using :</p>
<pre><code class="lang-auto">updates_list = []
for item in model.parameters():
    updates_list.append(copy.deepcopy(item.grad))
print(updates_list)

</code></pre>
<p>But i can not use <code>.grad</code> for my <code>global_model</code>. My question is, how do i get the updated gradients of my <code>global_model</code>?</p>

ptrblck: <p>I’m not sure, if I understand the use case correctly, but in case you would like to use the gradients in <code>updates_list</code>, you could assign these values to the <code>.grad</code> attributes of the parameters of the <code>global_model</code> and call <code>optimizer.step()</code> afterwards.</p>
Control maximum size of shared memory tensor

GuillaumeLeclerc: <p>Hello,</p>
<p>While trying to debug what seemed to be a deadlock I realized that I could not share a tensor to sub processes if it was more than 128kb.</p>
<pre><code class="lang-auto"># Works fine
buffer = ch.zeros(32768).share_memory_()

# sub-processes hang when try to read/write the tensor (even protected by a lock)
# (Main process can read it fine)
buffer = ch.zeros(32769).share_memory_()
</code></pre>
<p>Is there a configuration option to allow me to allocate more shared memory in a single tensor ?</p>
<p>Thank you for your help</p>

GuillaumeLeclerc: <p>It seems to be a bug since it was working in version 1.4.0. Filed a bug report here: <a href="https://github.com/pytorch/pytorch/issues/58962" class="inline-onebox" rel="noopener nofollow ugc">Operating on more than 128kb of shared memory hangs · Issue #58962 · pytorch/pytorch · GitHub</a></p>
Single-Process Multi-GPU is not the recommended mode for DDP

Mars_Su: <h3>
<a name="problem-1" class="anchor" href="#problem-1"></a>Problem</h3>
<p>Hi, Everyone. I have encountered some problem about pytorch ddp on single node multiple gpus.<br>
My setting is follow as:</p>
<pre><code class="lang-python">os.environ["MASTER_PORT"] = "9999" 
os.environ["CUDA_VISIBLE_DEVICES"] = "0,1,2,3"
.....
distributed_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
torch_dataloader = torch.utils.data.DataLoader(dataset,
                                                           batch_size=64,
                                                           pin_memory=True,
                                                           num_workers=4,
                                                           sampler=distributed_sampler)

model.cuda()
model = torch.nn.parallel.DistributedDataParallel(model)
</code></pre>
<p>But this setting is slower than <code>DataParallel</code>, and get some message.</p>
<h3>
<a name="error-message-2" class="anchor" href="#error-message-2"></a>Error Message</h3>
<pre><code class="lang-auto">UserWarning: Single-Process Multi-GPU is not the recommended mode for DDP. 
In this mode, each DDP instance operates on multiple devices and creates multiple module replicas within one process. 
The overhead of scatter/gather and GIL contention in every forward pass can slow down training.
Please consider using one DDP instance per device or per module replica by explicitly setting device_ids or CUDA_VISIBLE_DEVICES. 
</code></pre>
<h3>
<a name="environment-3" class="anchor" href="#environment-3"></a>Environment</h3>
<p>python: 3.7<br>
pytorch: 1.7<br>
GCP ml-engine image_uri: <a href="http://gcr.io/cloud-ml-public/training/pytorch-gpu.1-7" rel="noopener nofollow ugc">gcr.io/cloud-ml-public/training/pytorch-gpu.1-7</a><br>
gpu_type: complex_model_m_p100 (p100x4 on single node)</p>
<p>Hope someone can answer my problem. I will appreciate.</p>

wanchaol: <p>As you can see from the error message, it’s better to use multiple process for multiple GPU training even on a single node, you can use <code>torch.multiprocessing.spawn(train_fn, args=(world_size,), nprocs=world_size)</code> to initialize the training in multiple process</p>
CMake Generate step failed while compling from source code

mareikedaubert: <p>Hi all,<br>
my computer has a GeForceGT710 graphics card (OS Linux Mint). My understanding is that I need to install pytorch from source code to avoid the “CUDA error: no kernel image is available for execution on this device” warning.<br>
I have so far followed the instructions specified on github (<a href="https://github.com/pytorch/pytorch#from-source" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration</a>). However, I am getting an error while compiling:</p>
<blockquote>
<p>CMake Warning at modules/observers/CMakeLists.txt:12 (add_library):<br>
Cannot generate a safe runtime search path for target caffe2_observers<br>
because files in some directories may conflict with libraries in implicit<br>
directories:</p>
<pre><code>runtime library [libcudart.so.10.1] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /home/mareike/miniconda3/envs/ginjinn/lib
runtime library [libnvToolsExt.so.1] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /home/mareike/miniconda3/envs/ginjinn/lib
runtime library [libcufft.so.10] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /home/mareike/miniconda3/envs/ginjinn/lib
runtime library [libcurand.so.10] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /home/mareike/miniconda3/envs/ginjinn/lib
runtime library [libcublas.so.10] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
  /home/mareike/miniconda3/envs/ginjinn/lib
</code></pre>
<p>Some of these libraries may not be found correctly.</p>
<p>CMake Error at modules/observers/CMakeLists.txt:12 (add_library):<br>
The install of the caffe2_observers target requires changing an RPATH from<br>
the build tree, but this is not supported with the Ninja generator unless<br>
on an ELF-based platform.  The CMAKE_BUILD_WITH_INSTALL_RPATH variable may<br>
be set to avoid this relinking step.</p>
<p>CMake Error at modules/observers/CMakeLists.txt:12 (add_library):<br>
The install of the caffe2_observers target requires changing an RPATH from<br>
the build tree, but this is not supported with the Ninja generator unless<br>
on an ELF-based platform.  The CMAKE_BUILD_WITH_INSTALL_RPATH variable may<br>
be set to avoid this relinking step.</p>
<p>– Generating done<br>
CMake Generate step failed.  Build files cannot be regenerated correctly.</p>
</blockquote>
<p>My best idea so far is to set CMAKE_BUILD_WITH_INSTALL_RPATH to TRUE. However, I am not too knowledgeable about programming, let alone cmake.<br>
Other information that may be of use: I run gcc 9.3.0, g++ 9.3.0, cudatoolkit 10.1.243,  and cudnn 7.6.5. I have everything including dependencies (numpy,… again as specified on github) installed in a conda environment. However, the error also occurs when I install everything globally.<br>
Can someone help me to resolve this?<br>
Best,<br>
Mareike</p>
How to re-initialize individual neurons during training

Joost_van_der_Burgt: <p>I would like to test the impact of randomly resetting (re-initializing) certain neurons during training, in a Pytorch-based model with a number of linear layers like this:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.linear1 = nn.Linear(D, M)
        self.linear2 = nn.Linear(M, M)
        self.linear3 = nn.Linear(M, M)
        self.linear4 = nn.Linear(M, K)

    def forward(self, x):
        x = F.relu(self.linear1(x))
        x = F.relu(self.linear2(x))
        x = F.relu(self.linear3(x))
        x = self.linear4(x)
        x = F.log_softmax(x, dim=1)
               
        return x
</code></pre>
<p>Ideally, I would like to implement a function/code snippet that resets the weights of 1 or more random neurons in each layer at the start of each epoch. The idea is that each neuron has a small probability of being re-initialized each epoch, to see what the effects are on performance in general, as well as on common issues such as dying relu’s and vanishing gradients for sigm and tanh.</p>
<p>In other words: instead of a single epoch dropout, you’d have a permanent replacement of an existing neuron with a ‘fresh’ new trainable neuron.</p>
<p>I tried implementing this via calls to parameters or state_dict, but so far I only get permission and anomaly errors when trying this. I’m also not sure whether this should be implemented in the module itself or in the training part.</p>
<p>Any suggestions would be highly welcome!</p>

pascal_notsawo: <p>There are several ways to initialize the parameters of a model in deep learning : <a href="https://pytorch.org/docs/stable/nn.init.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.init — PyTorch 1.8.1 documentation</a></p>
<p>So based on the code you proposed above, I suggest something like :</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.linears = nn.ModuleList([
	        nn.Linear(D, M)
	        nn.Linear(M, M)
	        nn.Linear(M, M)
	        nn.Linear(M, K)
        ])  

        # ...
    # forward
    # ...

def init_linear(linear):
    with torch.no_grad():
        # Choose your approach here, I just suggest some
        nn.init.normal_(linear.weight, mean=0, std=1) 
        #nn.init.xavier_uniform_(linear.weight)
        nn.init.constant_(linear.bias, 0.)

# Or when you want to initialize the whole model
def init_model(model):
    with torch.no_grad():
        for name, param in model.named_parameters():
            # Choose your approach here, I just suggest some
            if 'weight' in name:
                torch.nn.init.xavier_uniform_(param.data)
            elif 'bias' in name:
                param.data.fill_(0)
            # elif :
            # ...
</code></pre>
<p>And during training:</p>
<pre><code class="lang-auto">for _ in range(MAX_EPOCH) :
	to_init = you choose randomly the indices of the layers you want to reinitialize
	for i in to_init :
		init_linear(model.linears[i])
	# ...
	# training
</code></pre>
Stacked RNN with different hidden size at each layer?

Finspire13: <p>Can I build a multi-layer RNN with different hidden size per layer using PyTorch?</p>
<p>For example, a 3-layer RNN with feature size of 512, 256, 128 at each layer respectivey?</p>

chenyuntc: <p>Yes, but you need to figure out the input and output of RNN/LSTM/GRU.</p>

Finspire13: <aside class="quote no-group" data-username="chenyuntc" data-post="2" data-topic="8184" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chenyuntc/48/299_2.png" class="avatar"> chenyuntc:</div>
<blockquote>
<p>Yes, but you need to figure out the input and output of RNN/LSTM/GRU.</p>
</blockquote>
</aside>
<p>By ‘layer’ I mean the layers of a stacked RNN. PyTorch RNN module only takes a single parameter ‘hidden_size’ and all stacked layers are of exactly the same hidden size. But is it possible to make layers have different hidden size?</p>

chenyuntc: <p>How about this:</p>
<pre><code class="lang-auto">import torch 
from torch import nn
from torch.autograd import Variable

# layer1
# input_dim=10, output_dim=20
rnn1 = nn.LSTM(10, 20, 1)
input = Variable(torch.randn(5, 3, 10))
output1, hn = rnn1(input)

# layer2
# input_dim=20 output_dim=30
rnn2 = nn.LSTM(20, 30, 1)
output2, hn2 = rnn2(output1) 
</code></pre>

Finspire13: <p>It works well. Thanks!</p>

gt_tugsuu: <p>Is it okey to discard hidden weights <code>hn</code> from first stack?</p>

gt_tugsuu: <p>It seems ok.<br>
                    <a href="https://i.stack.imgur.com/TtfMs.jpg" target="_blank" class="onebox" rel="nofollow noopener">
            <img src="https://i.stack.imgur.com/TtfMs.jpg" width="665" height="500">
          </a>

</p>

StannisLaw: <p>Could you please elaborate on that: I do not understand why we have to discard <code>hn</code> in the second layer while scheme says, that h2_t depends on both h2_t-1 and h1_t? Is there a way to handle both states in pytorch?</p>

wzm: <p>h2_t-1 is included in the same layer and h1_t is included in the output of the previous layer. You don’t need <code>hn</code> to make your network work.</p>
Better predictions from model at earlier epochs

iamneerav: <p>I am doing a pixel-wise regression using VNet architecture. The loss decreases continuously over epochs, as expected and the loss function has a form of <code>ln(L1 x L2)</code> where L1 and L2 are lagrangian and eulerian MSE Losses. The very weird thing is that the predictions of the model after epoch 1 is wayyyyy better than the one from epoch 30 or 50 etc. I am training on 51200 samples, each of shape (3,64,64,64) and testing is performed on 5120 samples of the same shape.</p>
<p>The loss function looks as follows:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/b/0baaeb327560f711ed75411c06717c5e81536048.png" data-download-href="https://discuss.pytorch.org/uploads/default/0baaeb327560f711ed75411c06717c5e81536048" title="plots"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0baaeb327560f711ed75411c06717c5e81536048_2_690x459.png" alt="plots" data-base62-sha1="1FdrwhUVE7rTZsA5CNnUkXLfs7e" width="690" height="459" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0baaeb327560f711ed75411c06717c5e81536048_2_690x459.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0baaeb327560f711ed75411c06717c5e81536048_2_1035x688.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/b/0baaeb327560f711ed75411c06717c5e81536048.png 2x" data-dominant-color="FDFDFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">plots</span><span class="informations">1200×800 37.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The predictions (one of the 4 statistics we use) is as follows:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/c/5c38da85a94331930e4bf54aee38d9fce404e2d2.png" data-download-href="https://discuss.pytorch.org/uploads/default/5c38da85a94331930e4bf54aee38d9fce404e2d2" title="1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5c38da85a94331930e4bf54aee38d9fce404e2d2_2_686x500.png" alt="1" data-base62-sha1="d9PJBuYUhM9XXUqKVEKTFYfROiC" width="686" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5c38da85a94331930e4bf54aee38d9fce404e2d2_2_686x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5c38da85a94331930e4bf54aee38d9fce404e2d2_2_1029x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/c/5c38da85a94331930e4bf54aee38d9fce404e2d2_2_1372x1000.png 2x"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">6933×5052 767 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What might be wrong?</p>
ONNX inference fails for a simple model structure with conditional statements

rammy_vadlamudi: <p>Find below my model, which includes conditional statements in forward block</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear( 1, 3 )
        self.fc2 = nn.Linear( 3, 10 )
        self.fc3 = nn.Linear( 10, 2 )
        
    def forward(self,x):
        
        if x.shape[0] ==1 :
            x = self.fc2( self.fc1(x)  )
            return x 
        else:
            x = self.fc3( self.fc2( self.fc1(x)  ) )
            return x 
</code></pre>
<p>To handle the conditional flow statements I have converted the model to torchscript</p>
<pre><code class="lang-auto">model = Net()
script_model = torch.jit.script( model )
</code></pre>
<p>Pass through dummy data</p>
<pre><code class="lang-auto">data = torch.randn((1,1))
outputs = script_model(data) 

</code></pre>
<p>ONNX model export</p>
<pre><code class="lang-auto">onnx_model_path = "saved_models/model.onnx"
torch.onnx.export(script_model, data, onnx_model_path, opset_version=11, input_names=["input"] ,  example_outputs= outputs,
                  dynamic_axes={ "input":{0:"batch_size"} }, output_names=["output"])
</code></pre>
<p>ONNX checker throws no error, However when I do inference I am having a crash with below error</p>
<pre><code class="lang-auto">onnx_session(onnx_model_path)

Fail: [ONNXRuntimeError] : 1 : FAIL : Load model from saved_models/model.onnx failed:Node (If_5) Op (If) [TypeInferenceError] Graph attribute inferencing failed: Node (If_10) Op (If) [TypeInferenceError] Graph attribute inferencing failed: This is an invalid model. Type Error: Type 'tensor(int64)' of input parameter (25) of operator (If) in node (If_14) is invalid.
</code></pre>
<p>The JIT converted model is working fine, Is there any limitation of using tensor.size or shape  functions in conditional statements when converting to ONNX ?</p>
<p>Please help me figure out :)) Thanks</p>

mlizhardy: <p>Hi there,</p>
<p>for torch.onnx you want to pass the torch model directly, not the jit model.  this works fine for me:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import onnx


class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear( 1, 3 )
        self.fc2 = nn.Linear( 3, 10 )
        self.fc3 = nn.Linear( 10, 2 )

    def forward(self,x):

        if x.shape[0] ==1 :
            x = self.fc2( self.fc1(x)  )
            return x
        else:
            x = self.fc3( self.fc2( self.fc1(x)  ) )
            return x

model = Net()

data = torch.randn((1,1))
outputs = model(data)
onnx_model_path = "model.onnx"
torch.onnx.export(model, data, onnx_model_path, opset_version=11, input_names=["input"] ,  example_outputs= outputs,
                  dynamic_axes={ "input":{0:"batch_size"} }, output_names=["output"])
</code></pre>
Doubt if should call .item()

andrejankas: <p>I am doing an unittest, and was wondering wether to use <code>.item()</code> or not. Both give the same result, so my question is it correct If I do it the 1-way? (more clean to me)</p>
<pre><code class="lang-auto">import unittest
import torch


class TestTensor(unittest.TestCase):
    def test_values(self):
        x = torch.arange(1, 10) &gt; 8
        self.assertTrue(x.any())  # 1-way
        self.assertTrue(x.any().item())  # 2-way

</code></pre>
The parameters of a CNN model is returning None

Samiur_Rahman: <p>Hello, i create a CNN model like this <code>global_model = CNNMnist(args=args)</code>. Then i send it to device, set it to train. Then i train my local models, collect the local_weights and the average them to get updated global_model.<br>
Now i am trying to get the items from the <code>.parameters()</code> function but all i get is <code>None</code> as <code>item.grad</code>. When i do the same thing for the local_models, i get the desired output. What am i doing wrong?</p>
<pre><code class="lang-auto">global_model.to(device)
global_model.train()
...................
global_weights = average_weights(local_weights)
global_model.load_state_dict(global_weights)
last_update = []
for item in global_model.parameters():
    last_update.append(copy.deepcopy(item.grad))
    print(item.grad)
</code></pre>
<p><code>Output: None None None None None None None None</code></p>
<p>Any help would be appreciated.</p>

Rohit_R: <p>item.grad stores the gradients of the parameters wrt the loss function and not the weights.</p>
<p>What you want is -</p>
<p>weights = [c.detach().numpy() for c in global_model.parameters()]</p>

Samiur_Rahman: <p>Hi, thanks for your reply. could you please elaborate a little? where exactly should i insert the line and how do i get the desired outcome?<br>
Thanks</p>

Rohit_R: <p>Hi. You can try the following.</p>
<pre><code class="lang-auto">for param in global_model.parameters():
    last_update.append(param.clone().data.cpu().numpy())
</code></pre>
Jointly training two deep neural networks

Mohamed_Nabih: <p>Hello, I have two deep neural networks, one for speech enhancement to remove noise from the speech signals and the other network is for intent classification. I am trying to train them jointly i.e. the output of the first network is passed directly to the second network as follows:</p>
<p>def _train_epoch(self, epoch=100):</p>
<pre><code>    for i, (mixture, clean, name) in enumerate(self.train_data_loader):
        mixture = mixture.to(self.device, dtype=torch.float)
        clean = clean.to(self.device, dtype=torch.float)
        self.optimizer.zero_grad()
        enhanced = self.model(mixture).to(self.device)
        loss = self.loss_function(clean, enhanced)
        loss.backward()
        self.optimizer.step()
        
        for i, d in enumerate(train_set_generator):
            enhanced, l = d
            model_back.train()
            y = model_back(enhanced.float().to(device2))
            loss_back = loss_func_back(y[0], l[0].to(device2))
            print("Iteration %d in epoch%d--&gt; loss = %f" % (i, epoch, loss_back.item()), end='\r')
            loss_back.backward()
            optimizer_back.step()
            optimizer_back.zero_grad()
            if i % 100 == 0:
              model_back.eval()
              correct = []
              for j, ev in enumerate(valid_set_generator):
                enhanced, label = ev
                y_eval = model_back(enhanced.float().to(device2))
                pred = torch.argmax(y_eval[0].detach().cpu(), dim=1)
                intent_pred = pred
                correct.append((intent_pred == label[0]).float())
                if j &gt; 100:
                    break
              acc = np.mean(np.hstack(correct))
              intent_acc = acc
              iter_acc = '\n iteration %d epoch %d --&gt;' %(i, epoch)
              print(iter_acc, acc, best_accuracy)
              if intent_acc &gt; best_accuracy:
                improved_accuracy = 'Current accuracy {}, {}'.format(intent_acc, best_accuracy)
                print(improved_accuracy)
                torch.save(model_back.state_dict(), 'bestmodel.pkl')
</code></pre>
<p>Where the loss of the first DNN is the MSE loss and the loss of the second DNN is the Cross entropy loss. The problem is that after many iterations in the first epoch the second DNN starts to iterate on the first loop again not goes to the second loop. Any help will be appreciated.</p>
Plotting Cross Entropy for Multi-class and its weighted version

shakeel608: <p>Is there any tutorial , to plot CEntropy loss for a particular class and its weighted version to see how the weights affect the class imbalance problem</p>
<p>like below</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8faf8fc35ab278a9e1849a68139e599135da93f5.png" data-download-href="https://discuss.pytorch.org/uploads/default/8faf8fc35ab278a9e1849a68139e599135da93f5" title="FL_v_CE"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8faf8fc35ab278a9e1849a68139e599135da93f5_2_690x417.png" alt="FL_v_CE" data-base62-sha1="kv6pRLyZXwDsH6IUW2j3kbzG93n" width="690" height="417" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8faf8fc35ab278a9e1849a68139e599135da93f5_2_690x417.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8faf8fc35ab278a9e1849a68139e599135da93f5_2_1035x625.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/f/8faf8fc35ab278a9e1849a68139e599135da93f5.png 2x" data-dominant-color="F3F3F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">FL_v_CE</span><span class="informations">1052×636 148 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Method to create one-hot encoding in data.utils.dataset

banikr: <p>Hello All,</p>
<p>I am trying to create dataset for multi-class segmentation. Image sizes are <code>48,64,64</code> and segmentation masks have 9 different labels.</p>
<pre><code class="lang-auto">.
.
.
imgnp = imgnp[None, ...]
segD = np.zeros((num_labels, 48, 64, 64))
for i in range(0, num_labels):  # this loop starts from label 1
    seg_one = segnp == labels[i]
    segD[i, :, :, :] = seg_one[0:segnp.shape[0], 0:segnp.shape[1], 0:segnp.shape[2]]
imgD = imgnp.astype('float32')
segD = segD.astype('float32')
return imgD, segD
</code></pre>
<p>The output is image of shape <code>1,48,64,64</code> and one-hot encoded binary segmentation mask with 9 channels for 9 tissue labels: <code>9,48,64,64</code>.</p>
<p>My ques: is that an efficient way to create the <code>dataset</code> or there are any mistakes or other better ways to do so.</p>
<p>*N.B.: the chunk of the code is from <code>def __getitem__</code><br>
Thanks in advance.</p>

sh0416: <p>Use <code>scatter</code></p>
<pre><code class="lang-auto">def create_one_hot(x: Tensor, n_class: int):
  """
  :param x: [B, D1, D2, D3] 0 &lt;= x.min() &amp;&amp; x.max() &lt; C
  :return: [B, C, D1, D2, D3]
  """
  B, D1, D2, D3 = x.shape
  out = torch.zeros((B, n_class, D1, D2, D3), dtype=torch.float, device=x.device)
  x = x[:, None, :, :, :]
  out.scatter_(dim=1, index=x, src=torch.ones_like(x, dtype=out.dtype))
  return out
</code></pre>

I-Love-U: <p>For creating a one-hot tensor, I have implemented five functions…</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/lartpang/PyLoss/blob/main/losspy/utils/convert_onehot.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/lartpang/PyLoss/blob/main/losspy/utils/convert_onehot.py" target="_blank" rel="noopener nofollow ugc">lartpang/PyLoss/blob/main/losspy/utils/convert_onehot.py</a></h4>
<pre><code class="lang-py"># -*- coding: utf-8 -*-
# @Time    : 2020/12/13
# @Author  : Lart Pang
# @GitHub  : https://github.com/lartpang

import time

import torch
from torch.nn.functional import one_hot


def bhw_to_onehot_by_for(bhw_tensor: torch.Tensor, num_classes: int):
    """
    Args:
        bhw_tensor: b,h,w
        num_classes:

    Returns: b,h,w,num_classes
    """
    assert bhw_tensor.ndim == 3, bhw_tensor.shape
</code></pre>

  This file has been truncated. <a href="https://github.com/lartpang/PyLoss/blob/main/losspy/utils/convert_onehot.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Hope it works for you.</p>