akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to calculate PSNR correctly?

ZdsAlpha: <p>I am having trouble computing PSNR on entire dataset. My results are not matching. What is the correct way to compute PSNR?</p>
<ol>
<li>PSNR = 20 * log10(255.0 / Average_RMSE)</li>
<li>PSNR = MEAN( PSNR_OF_EACH_IMAGE )</li>
<li>Compute total Suqare Error of dataset and then average it by dividing total pixels in dataset. Then compute PSNR = 20 * log10( 255.0 / RMSE)</li>
</ol>
<p>I tried all three methods but I get slightly more PSNR. For third method, I compared bilinear PSNR of Set14 and got 27.6 while its mentioned as 26.55 in different papers.</p>

wayne1226: <p>I have same question among this and here is my analysis about 1 and 2, 3 should be same as 1.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0c02abaf81a789d540e75b8bc74d554ea83bfaf1.png" data-download-href="https://discuss.pytorch.org/uploads/default/0c02abaf81a789d540e75b8bc74d554ea83bfaf1" title="MSEANDPSNR"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c02abaf81a789d540e75b8bc74d554ea83bfaf1_2_353x500.png" alt="MSEANDPSNR" data-base62-sha1="1IfrZufGDYU1Ez6ix182TCxRp8R" width="353" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c02abaf81a789d540e75b8bc74d554ea83bfaf1_2_353x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c02abaf81a789d540e75b8bc74d554ea83bfaf1_2_529x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c02abaf81a789d540e75b8bc74d554ea83bfaf1_2_706x1000.png 2x" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">MSEANDPSNR</span><span class="informations">1654×2339 46.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
What is the correct way to calculate mean PSNR among a dataset?

wayne1226: <p>I am working on a image quality assessment tasks and am wondering can we compute the average among a dataset? If we can, what is the correct way to compute it? In my analysis, the <em><strong>method2</strong></em> should be the correcte way, but I don’t know whether this analysis is correct or is there any other method to compute PSNR among a dataset<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0c02abaf81a789d540e75b8bc74d554ea83bfaf1.png" data-download-href="https://discuss.pytorch.org/uploads/default/0c02abaf81a789d540e75b8bc74d554ea83bfaf1" title="MSEANDPSNR"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c02abaf81a789d540e75b8bc74d554ea83bfaf1_2_353x500.png" alt="MSEANDPSNR" data-base62-sha1="1IfrZufGDYU1Ez6ix182TCxRp8R" width="353" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c02abaf81a789d540e75b8bc74d554ea83bfaf1_2_353x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c02abaf81a789d540e75b8bc74d554ea83bfaf1_2_529x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0c02abaf81a789d540e75b8bc74d554ea83bfaf1_2_706x1000.png 2x" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">MSEANDPSNR</span><span class="informations">1654×2339 46.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
CUDA error: out of memory,why?

chunchun: <p>I’m testing a object tracker,during running the code,I found that when test the fifth vedio in the dataset,there was an error:CUDA error: out of memory.I can’t know why.<br>
Could you please tell me some methoda to solve this problem?</p>

albanD: <p>Hi,</p>
<p>This can happen for several reasons. If it happens after a few iteration, this is most likely that you are increasing your GPU memory needs from one iteration to the other, for example if you append to a list whole frames of the video at every iteration of the loop.<br>
Can you give more details on what you’re doing?</p>

chunchun: <p>Thanks a lot for your attention.<br>
I use VOT2016 dataset which has many videos as subdatasets to test the tracker finetuning SiamRPN that I add a new feature extraction part to it.And this part will store each frame information just like LSTM.To testify my code of this new part is correct,I just test the whole code without train.Although it works every time,but when the tracker read-in the fifth vedio,there will an error:CUDA error: out of memory.<br>
I don’t know whether it is because the the code I have added.Could you please tell me how to deal with this?Thanks a lot.</p>

albanD: <p>Hi,</p>
<p>You can easily check the previous code by running it I guess.<br>
For the new code, you want to make sure you don’t store too much Tensors that would fill up your memory.<br>
Also be careful to use <code>with torch.no_grad():</code> when evaluating your model to reduce memory usage.</p>

chunchun: <p>That’s cool.<br>
My problem has been solved.<br>
Thank you very much.</p>

chunchun: <p>That’s cool.<br>
My problem has been solved.<br>
Thank you very much.</p>

saran: <p>how you solved the problem</p>
How to use correctly multilabel classification in pytorch

TAF: <p>Hi guys, I have some csv file with structure</p>
<pre><code class="lang-python">  img |class0|class1|class2|class3|class4|class5...
|-----|------|------|------|------|------|------
|path1|0     |0     |1     |0     |0     |1     
|path2|0     |1     |1     |0     |1     |0     
|path3|0     |0     |0     |1     |0     |0     
.................................................
.................................................
|pathn|1     |1     |0     |1     |0     |0     
|-----|------|------|------|------|------|------
class mydataset(Dataset):
...
def __getitem__(self,idx):
...
labels=np.array(self.labels.iloc[idx,1:])    
    return {'img':img,'labels':labels}
</code></pre>
<p>I am trying to use</p>
<pre><code class="lang-python">n_classes=df.shape[0]-1
model = EfficientNet.from_pretrained('efficientnet-b1',num_classes=n_classes).to(device) 
</code></pre>
<p>with</p>
<pre><code class="lang-python">loss = torch.nn.BCELoss()
</code></pre>
<p>but it doesn’t work, how to implement in my case multilabel classification?</p>

eqy: <p>What is the exact error you are getting in this case?</p>
<p>You might want to check that the datatype of labels is a torch tensor as expected.</p>
<p>The documentation shows a simple of example of input being passed to the loss function: <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html" class="inline-onebox" rel="noopener nofollow ugc">BCELoss — PyTorch 1.9.0 documentation</a><br>
although in your case you will likely also have a batch dimension.</p>
Convert tensor([4,1]) to tensor([4,2]) without changing elements

Aarya: <p>Hello,</p>
<p>I want to convert<code> tensor([4]) to tensor([4,2])</code> without changing elements in it.</p>
<p>I have tried using torch.reshape, but it does not work, and gives some error.</p>
<p>I do not want to change the dimension using squeeze or unsqeeze.</p>
<p>My aim is to use BInary Cross Entropy,<br>
And I have t<code>arget shape tensor([4,1]) and output shape tensor([4,2])</code></p>
<p>But it says that BCE Loss inputs must be of same shape.</p>
<p>Any help is most appreciated!</p>
<p>Thank you in advanvce. <a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/eqy">@eqy</a></p>

JuanFMontesinos: <p>If you have a tensor which is 4,1 you can’t magically get a tensor which is 4,2.<br>
One is 4 elements and the other 8.<br>
You should think in the logics behind that. You may want to use random numbers, to tile the tensor…</p>
<p>Reshapes just reorder elements but doesn’t create the. Unsqueeze creates new dimensions but not new elements.</p>
How to optimize code when switching from CPU to GPU?

youdar: <p>Hi,</p>
<p>I am trying to improve training performance of a Transformers type, not very large, neural network, by using a single T4 gpu (AWS g4dn.xlarge instance).</p>
<p>I’ve changed my code, replacing all the numpy functions with torch functions, following advice from <a href="https://towardsdatascience.com/7-tips-for-squeezing-maximum-performance-from-pytorch-ca4a40951259" rel="noopener nofollow ugc"> 7 Tips To Maximize PyTorch Performance</a></p>
<p>but I see only about 17% improvement in performance (on a 1000 training epochs)</p>
<p>Running profiler<br>
36% of time is spent on <code>method 'run_backward' of 'torch._C._EngineBase</code><br>
17% on <code>built-in method tensor</code></p>
<p>Any suggestions on code optimization for GPU ?<br>
is there even value in using GPUs if the NN is not huge ?</p>
<p>If there is no way to make the GPU significantly faster, it might be cheaper to get more CPU and parallel training on CPUs, wouldn’t it ?</p>
nn.Transformer src and tgt related question for sequence to sequence prediction

sajidmahmud14: <p>I am trying to train a Transformer model using (<a href="https://pytorch.org/docs/stable/generated/torch.nn.Transformer.html" class="inline-onebox" rel="noopener nofollow ugc">Transformer — PyTorch 1.9.0 documentation</a>) which will take in an input of shape (batch_size, 73, 640) and output a sequence of shape (batch_size, 2, 640). I have initialized the transformer with d_model to 640 and kept the rest at default. Now while training, the model requires two inputs: src and tgt. I can safely assume src is the input that I have (batch_size, 73, 640) and the tgt would be the ground truths that I have (batch_size, 2, 640). However, while in the validation/test phase of the model, I may not have ground truths for prediction. What should be done in this case? Or am I understanding the tgt incorrectly?</p>
<p>Any help would be greatly appreciated.</p>

JuanFMontesinos: <p>Well, I would recommend you to look for hands on code to understand the concept.<br>
Let me first to pose the problem (I think ) you are talking about, which is predicting sequences given a context.<br>
Imagine you have N elements. When you train, you assume that <strong>for each element</strong> the network knows n_i and all the previous elements.<br>
For example: to predict n3 you know n2 n1 and n0.<br>
To predict n4 you know n3 n2 n1 and n0.<br>
You achieve this by using masks and all this happens in parallel (thanks to the masks too).<br>
On contrary, with lstms you need to predict element by element.</p>
<p>However, for testing the problem is different. Instead of doing all the predictions in parallel thanks to the masks, you will have to run the transformer element by element.</p>
<p>This is, you 1st predict n0. Then feed n0 to get n1. Then feed n0 and n1 and so on.</p>

sajidmahmud14: <p>Thank you. I am getting the gist of the idea even though I still have some questions. So what I understand, I should be generating masks for src and tgt both so that it would hide the elements ahead (not yet predicted).</p>

JuanFMontesinos: <p>Well it depends on the field and the task. You have to think in the real pipeline.<br>
For example, generating speech from text.<br>
Do you have access to the whole sentence? Yes, thus you don’t need to mask that.<br>
Do you have access to the whole speech? No, because you are predicting it. Thus you have to mask it.<br>
The key is you (usually) don’t need masks at inference, as you predict one element and feed that and all the previous.</p>
<p>But the best you can do is looking for code <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"> that always help. No matter it is not using pytorch’s transformer, the key idea is the same.</p>
Distributed training. Synchronized loss calculation in eval mode before and after training on a minibatch

anakuz: <p>For my project I would like to calculate the prediction gain on a batch before and after training. To do so I need to get losses in eval mode before and after training on a batch. See pseudo code below.</p>
<pre><code class="lang-auto">1. loss_1 = model(batch) (in .eval() mode)
2. model(batch) (train mode)
3. loss_2 = model(batch) (in .eval() mode)
</code></pre>
<p>What will be a right way to maintain the synchronization for loss calculation across &gt; 1 GPUs while using the distributed training with multiprocess?</p>
<p>Currently I am using <code>torch.distributed.all_reduce()</code> to collect the losses across the gpus. The problem is when I am logging the losses to the text file, I get two different losses (if 2 gpus are used) during the same iteration:</p>
<pre><code class="lang-auto">{"losses": "(1057.0538330078125, 1056.30419921875)", "iepoch": 1, "iiter": 6}
{"losses": "(1082.687744140625, 1082.0916748046875)", "iepoch": 1, "iiter": 6}
</code></pre>
<p>where the <code>losses</code> contain <code>loss_1</code> and <code>loss_2</code>.</p>
<p>Any suggestions why would it happen after doing <code>.all_reduce()</code>? If the two losses are returned I would expect them to be the same.</p>
<p>Thank you!</p>
Libtorch1.9 Can't use gpu in windows10 visio studio 2019

zheng_wang: <p>After I updated libtorch1.7 to version 1.9, the GPU cannot be used.<br>
Environmental information<br>
windows10<br>
visio studio 2019<br>
Before libtorch 1.6 version, I added it in the linker → command line → other options</p>
<pre><code class="lang-auto">/INCLUDE:?searchsorted_cuda@native@at@@YA?AVTensor@2@AEBV32@0_N1@Z
/INCLUDE:?warp_size@cuda@at@@YAHXZ 
</code></pre>
<p>But now in version 1.9 of libtorch, this method is invalid. Is there any other way to use libtorch1.9 version for windows GPU?</p>

adaber: <p>Hey, zheng_wang</p>
<p>I ended up using only /INCLUDE:?searchsorted_cuda@native@at@<span class="mention">@YA</span>?AVTensor@2@AEBV32@0_N1@Z to force the linker to link against torch_cuda_cu.lib.</p>
<p>So, you also have to add torch_cuda_cu.lib to Linker-&gt;Input and copy torch_cuda_cu.dll to the folder where your executable is located.</p>
<p>Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Unfold and Conv1d

IvanZ: <p>Hello everyone!</p>
<p>I need assistance with the effective calculation of conv1d over unfolded tensor.</p>
<p>So I have a tensor of size  <code>[b, 2, L_1]</code> - batch size, channels, sequence length. Then I unfold it to have <code>[b, 2, n_frames, L_2]</code> - n_frames is the resulting number of frames after unfolding with kernel size L_2 and some stride. After that, I want to apply Conv1d to this tensor, and here is the problem.</p>
<ol>
<li>I can reshape it to <code>[b * n_frames, 2, L_2]</code> but this results in copying of the tensor data and a GPU’s memory overflow. <em>DOES NOT WORK</em>
</li>
<li>I can iterate in a loop over <code>b</code> (samples of my batch) and treat <code>n_frames</code> is a new batch dimension, feeding tensors of <code>[n_frames, 2, L_2]</code> into my model and that stacking the result after the loop. <em>WORKS BUT SLOW</em>
</li>
</ol>
<p>All that is to be done because of <code>L_2 &lt; L_1</code>, i.e. my inner model takes inputs of length <code>L_2</code> but an outer one works with sequences of length <code>L_1</code>, so I cut my longer input with overlaps and feed into the inner model.</p>
<p>Thank you!</p>
How do I train a pretrained JIT model?

zswartz: <p>I am trying to perform transfer learning using one of the silerio STT models located in the torch hub. The model loads as a JIT model which I am entirely unfamiliar with.</p>
<p>Does anyone know how I can work with this model as I would a traditional pytorch module? I want to be able to freeze / unfreeze layers, remove layers, etc, as well as use it in a training loop.</p>
<p>Thank you for your help!</p>
Python code to convert 1D tensor to 2D tensor

Aarya: <p>Hello,</p>
<p>I am trying to use Binary Cross Entropy Loss (BCE loss) for Simese network.<br>
I have two inputs for BCE loss function:</p>
<ol>
<li>output <code>(input_dy)</code> →  tensor of size [4] , output of neural network</li>
<li>true_labels <code>(y_true)</code>  →  tensor of size [4],  target (true value)</li>
</ol>
<p>For BCE loss, the input parameters must be of the dimension:</p>
<ol>
<li>output <code>(input_dy)</code>  → [Batch_size, no. of classes]</li>
<li>true_labels <code>(y_true)</code> → [Batch_size]</li>
</ol>
<p>The following diagram explains the query:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/1/514eebf884a387635525637129060081c02819e4.png" data-download-href="https://discuss.pytorch.org/uploads/default/514eebf884a387635525637129060081c02819e4" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/1/514eebf884a387635525637129060081c02819e4.png" alt="Capture" data-base62-sha1="bBhLnA39U2HCRX1kw0KPLzMXN0o" width="690" height="211" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">784×240 3.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I need a function in python using pytorch to convert the <code>dy</code> matrix to a 2D matrix with the output probabilities that sum to 1. [To note: <code>dy</code> should be iterated through length of it, as it is the output of the network for every input ]<br>
Further a 2D array must be represented into one hot encoding, which will be true_labels (that will represent Binary classes with 0 &amp; 1)</p>
<p>I need both output matrix and true_labels matrix for BCE Loss with following dimensions:</p>
<ol>
<li>output dimension → [4, 2]</li>
<li>true_labels → [4]</li>
</ol>
<p>Any help is most appreciated!<br>
Thank you in advance. <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

eqy: <p>There are many ways to do this, but here is a simple and straightforward version:</p>
<pre><code class="lang-auto">input_dy = torch.empty(dy.size(0), 2)
input_dy[:,0] = dy
input_dy[:,1] = 1-dy

y_true = torch.round(input_dy)
</code></pre>
<p>I don’t understand the second part of the question; is the “one-hot” encoding supposed to have a shape of [4] as well? If it was truly “one-hot” I would expect the shape to be [4, 2] as well. The posted code snippet implements a true-false encoding rather than a one-hot.</p>

Aarya: <p>Yes,<code> y_true</code> should have the dimension [4].<br>
<code>y_true</code> matrix should contain either 0 or 1, for e.g. <code>y_true</code> first element is 1 because it is the index of<br>
<code>input_dy</code> maximum value, 0.6607 is greater than 0.3393, and index of 0.6607 is 1, so <code>y_true</code> is 1.<br>
<a class="mention" href="/u/eqy">@eqy</a></p>
What is the best way to implement a Tree Convolution?

Brando_Miranda: <p>I was reading the paper Tree Gen (<a href="https://arxiv.org/pdf/1911.09983.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1911.09983.pdf</a>) and wanted to implement the TreeConvolution:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/a/5a7791ca2a9afd835ebd535b1ea664a516416371.png" alt="Screen Shot 2021-07-27 at 9.16.54 AM" data-base62-sha1="cUj9dXFsCNlBYgDu90XpDPy6byV" width="581" height="277"></p>
<p>I was wondering if this was possible to be done with PyTorch Geometric or DGL or something like that or if the best way was to simply generate the adjacency matrix, multiply my vectors with it and then do a normal convolution?</p>
<p>cross-posted:</p>
<ul>
<li><a href="https://discuss.pytorch.org/t/what-is-the-best-way-to-implement-a-tree-convolution/127851" class="inline-onebox">What is the best way to implement a Tree Convolution?</a></li>
<li><a href="https://discuss.dgl.ai/t/what-is-the-best-way-to-implement-a-tree-convolution/2189" class="inline-onebox" rel="noopener nofollow ugc">What is the best way to implement a Tree Convolution? - Questions - Deep Graph Library</a></li>
<li><a href="https://stackoverflow.com/questions/68553212/what-is-the-best-way-to-implement-a-tree-convolution-in-pytorch" rel="noopener nofollow ugc">https://stackoverflow.com/questions/68553212/what-is-the-best-way-to-implement-a-tree-convolution-in-pytorch</a></li>
</ul>

Brando_Miranda: <p>Hi <a class="mention" href="/u/alband">@albanD</a>, sorry for the direct post but I wasn’t sure who else to tag. Is there a way to “insert a (any) GraphNN layer” in a feedforward model I already have in pytorch (e.g. pytorch geometric)?</p>
<p>e.g.</p>
<p>imagine I wanted to implement a sequential model with a GCN:</p>
<pre><code class="lang-auto">f = nn.Sequential(OrderedDict([
    ('f1', nn.Linear(Din, Dout)),
    ('out', nn.ReLU()),
    ('gcn1', torch_geometric.GCN(*params))
]))
y: Tensor =  f(x)
</code></pre>
<p>Something like that. Is it possible?</p>

albanD: <p>I am not sure to follow what would prevent you from doing that?<br>
You will need to make sure that input/output shape match. But that’s it no?</p>
How can I disable GeneralizedRCNNTransform from MaskRCNN torchvision model?

WaterKnight: <p>Torchvision has a MaskRCNN mode, when you look in the layers it has a layer called <code>GeneralizedRCNNTransform</code>.</p>
<p>I would like to remove it. Is it a good idea? How can i achieve that?</p>

ptrblck: <p>MaskRCNN doesn’t seem to use it, but FasterRCNN?<br>
You could disable it by writing a custom class, deriving from <code>FasterRCNN</code> and removing <a href="https://github.com/pytorch/vision/blob/fed843dacb9e7d15ae6d833fc23dfd53a121d1dd/torchvision/models/detection/faster_rcnn.py#L231">this line of code</a>.<br>
However, note that this class just normalizes and reshapes the data as seen <a href="https://github.com/pytorch/vision/blob/d6ee8757eca7b74b98e5f0d434a565eb7b1c410b/torchvision/models/detection/transform.py#L60">here</a>.<br>
It might be thus easier to set the mean as zeros and the std as ones to disable the normalization.</p>

WaterKnight: <p>However, I would like to use the pretrained weights for resnet50.</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a>  Is there any way of slicing the network after having loaded the wieghts?</p>

ptrblck: <p><code>MaskRCNN</code> uses a pretrained renset50 as its backbone, no?<br>
The <a href="https://pytorch.org/docs/stable/torchvision/models.html#torchvision.models.detection.maskrcnn_resnet50_fpn">docs to maskrcnn_fpn_resnet50</a> show that a resnet is used internally.</p>

WaterKnight: <p>Yes, I know.</p>
<p>However, when Pretrained=True, weights for all layers gets downloaded not just the ones from resnet50.</p>

gkalliatakis: <p>I am facing the same issue, I want to disable it altogether because I’m working with single channel images and the transformation gives me 3 channel images even when my input is explicitly given as single channel (model architecture changed prior to that as well). Is there a way to do this without losing the pretrained weights?</p>

gkalliatakis: <p>For anyone interested I’ve found a workaround taken from this answer here <a href="http://5.9.10.113/64957944/error-when-trying-to-train-fasterrcnn-with-custom-backbone-on-grayscale-images" rel="noopener nofollow ugc">http://5.9.10.113/64957944/error-when-trying-to-train-fasterrcnn-with-custom-backbone-on-grayscale-images</a></p>
<p>Basically, we need to redefine the <code>GeneralizedRCNNTransform</code> and attach it to your model</p>
<pre><code class="lang-auto">grcnn = torchvision.models.detection.transform.GeneralizedRCNNTransform(min_size=800, max_size=1333, image_mean=[0.485], image_std=[0.229])
model.transform = grcnn
model.to(device)
</code></pre>
Using libtorch in OpenMP gets wrong results

lymhust: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/6/9655a49353cfbdaad59b776b0c453ec62055a950.png" data-download-href="https://discuss.pytorch.org/uploads/default/9655a49353cfbdaad59b776b0c453ec62055a950" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/6/9655a49353cfbdaad59b776b0c453ec62055a950.png" alt="image" data-base62-sha1="lrV6NdaDDoIaMU0UB723ilJ0oEg" width="690" height="426" data-dominant-color="2F312C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">835×516 42.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/a/9a2badc3023bd990e63f87384137259bd3663911.png" data-download-href="https://discuss.pytorch.org/uploads/default/9a2badc3023bd990e63f87384137259bd3663911" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/a/9a2badc3023bd990e63f87384137259bd3663911.png" alt="image" data-base62-sha1="lZR6VA8e5z4aBa5BNFYUk2dNPGh" width="187" height="500" data-dominant-color="2E2C2C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">357×953 14.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I want to use libtorch functions in OpenMP. However, there are some problems:</p>
<ol>
<li>Only the last thread returns correct results.</li>
<li>Other threads get wrong results.</li>
</ol>
<p>Has anyone encountered the same problem? Thanks!</p>

lymhust: <p>Finally, I found that maybe it is caused by set_num_interop_threads.<br>
For earlier libtorch (e.g. libtorch 1.0.0, libtorch 1.1.0), there are no functions like at::set_num_interop_threads or torch::set_num_interop_threads, and only one thread works when calling libtorch functions in OpenMP.<br>
From libtorch 1.2.0, we can find at::set_num_interop_threads in &lt;ATen/Parallel.h&gt; or  torch::set_num_interop_threads in &lt;torch/utils.h&gt; in later versions, and the final results are correct.</p>

xym2009: <p>I’ve also used libtorch (GPU) functions in OpenMP, and I notice the computation time is not linearly reduced as CPU functions. For example, a function need 1000ms in one core, and 100ms using OpenMP with 10 cores; however, for GPU, a function may need 100ms, while using OpenMP with 10 cores, the time may be 80ms (the GPU memory is enough), do you have any idea about this problem? Thanks a lot!</p>
What is the best practice to share a massive CPU tensor over multiple processes (read-only + single machine + DDP)?

siahuat0727: <p>Hi everyone, what is the best practice to share a massive CPU tensor over multiple processes (read-only + single machine + DDP)?</p>
<p>I think <a href="https://pytorch.org/docs/stable/storage.html?highlight=from_file#torch.DoubleStorage.from_file" class="inline-onebox" rel="noopener nofollow ugc">torch.Storage — PyTorch 1.9.0 documentation</a> with <code>share=True</code> suited my needs, but I can’t find a way to save storage and read it as a tensor. (same <a href="https://github.com/pytorch/pytorch/issues/28794" rel="noopener nofollow ugc">open issue</a> on Oct 29, 2019 )</p>
<p>I also tried to copy training data to <code>/dev/shm</code> (<a href="https://pytorch-lightning.readthedocs.io/en/stable/benchmarking/performance.html#preload-data-into-ram" rel="noopener nofollow ugc">reference</a>) and run DDP with 8 GPUs, but nothing is different. The memory usage when running with 8 GPUs is the same as before, but I tested with a single process, loading the dataset may occupy about 1 GB of memory. Am I missing something here?</p>

wanchaol: <p>Thanks for posting the question <a class="mention" href="/u/siahuat0727">@siahuat0727</a>, did you try <code>torch.multiprocessing.Queue</code> to pass the tensor objects between the processes? you can take a look at the `torch.multiprocessing <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" rel="noopener nofollow ugc">doc</a> and see if this works for you.</p>

wanchaol: <p>you can also use <code>tensor.shared_memory()</code> for sharing a big tensor across processes.</p>

siahuat0727: <p>Hi <a class="mention" href="/u/wanchaol">@wanchaol</a>, thank you very much for your reply!<br>
I think <code>tensor.shared_momory</code> should work well in pure multiprocess program.<br>
I will look at how to pass the reference in pytorch-lightning DDP mode.</p>

siahuat0727: <p>Finally, I found a way to utilize <code>torch.Storage.from_file</code>.<br>
For detail, see <a href="https://discuss.pytorch.org/t/how-to-save-a-storage-file-that-can-be-loaded-using-torch-storage-from-file/127737/2">here</a>.</p>
Implementing dropout

Zeeyuu: <p>Hi, I wonder if I want to implement dropout by myself, is something like the following sufficient (taken from <a href="https://stackoverflow.com/questions/54109617/implementing-dropout-from-scratch" class="inline-onebox" rel="noopener nofollow ugc">machine learning - Implementing dropout from scratch - Stack Overflow</a>):</p>
<pre><code class="lang-auto">class MyDropout(nn.Module):
    def __init__(self, p: float = 0.5):
        super(MyDropout, self).__init__()
        if p &lt; 0 or p &gt; 1:
            raise ValueError("dropout probability has to be between 0 and 1, " "but got {}".format(p))
        self.p = p

    def forward(self, X):
        if self.training:
            binomial = torch.distributions.binomial.Binomial(probs=1-self.p)
            return X * binomial.sample(X.size()) * (1.0/(1-self.p))
        return X
</code></pre>
<p>My concern is even if the unwanted weights are masked out (either through this way or by using a mask tensor), there can still be gradient flow through the 0 weights (<a href="https://discuss.pytorch.org/t/custom-connections-in-neural-network-layers/3027/9" class="inline-onebox">Custom connections in neural network layers - #9 by Kaixhin</a>). Is my concern valid?</p>

Varal7: <p>This looks like a fine implementation.</p>
<p>In the example you link,<br>
<code>y = w * x</code>, so <code>dy/dw = x</code>.</p>
<p>However, in your context, you’d have something like<br>
<code>y = dropout(w * x + b) + ...</code><br>
if dropout returns 0, the contribution of <code>w</code> is 0</p>

Zeeyuu: <p>Thanks for your reply. So can I just do something like this:<br>
<code>relu(self.fc(x)*self.mask)</code></p>
<p>not sure if masking happens before or after activation.</p>
How to plot error vs training size learning curves

Cow_woC: <p>Hi,</p>
<p>I’m seeing a lot of people plotting the training and validation error value vs epoch but both <a href="https://www.dataquest.io/blog/learning-curves-machine-learning/" rel="noopener nofollow ugc">https://www.dataquest.io/blog/learning-curves-machine-learning/</a> and Andrew Ng talk about plotting the error vs the size of the training set.</p>
<p>scikit-learn provides a nice function called <code>learning_curve()</code> which plots the error curves vs training set size. How can we do the same in pytorch? Is there a convenience class/function that does this?</p>
<p>Thanks,<br>
Gili</p>
Taking autograd of model.parameters()

ccook: <p>I am trying to get the hessian after a model has completed estimating (to then use to calculate standard errors). Given a model object, my thought was that something like the following should work:</p>
<pre><code class="lang-auto">d1 = torch.autograd.grad(loss, model.parameters(), create_graph=True)[0]
d2 = torch.autograd.grad(loss, d1, create_graph=True)[0]
</code></pre>
<p>However, this gives the error “One of the differentiated Tensors appears to not have been used in the graph.” Could someone point me in the right direction?</p>
<p>The model class, for what it’s worth, is the following:</p>
<pre><code class="lang-auto">class MultinomialLogitWithEmbeddings(torch.nn.Module):

    def __init__(self, m, n_fe):
        super(MultinomialLogitWithEmbeddings,self).__init__()
        self.n_features = m
        self.n_fe_cats = n_fe        
        self.emb_dim = 1      
        
        # Layers 
        self.emb = torch.nn.Embedding(self.n_fe_cats, self.emb_dim) # Store FEs as an embeddings lookup 
        self.linear = torch.nn.Linear(self.n_features + self.emb_dim, 1, bias=False) # Linear utility layer
        self.lsmax = torch.nn.LogSoftmax(1) # Log soft max activation
        
        # Initialize the layer weights 
        nn.init.kaiming_normal_(self.linear.weight.data)
        
    def forward(self, x_cont, x_cat):    
        x = torch.cat([x_cont.float(), self.emb(x_cat).squeeze(dim=-1).float()], 2)        
        y_pred=self.linear(x)
        y_pred=self.lsmax(y_pred)
        return y_pred.squeeze()
</code></pre>

Varal7: <p>How about using <a href="https://pytorch.org/docs/stable/generated/torch.autograd.functional.hessian.html" class="inline-onebox" rel="noopener nofollow ugc">torch.autograd.functional.hessian — PyTorch 1.9.0 documentation</a>?</p>
<p>The error comes from the fact that <code>loss</code> does not depend on <code>d1</code>.<br>
You probably want to do something like <code>d2 = torch.autograd.grad(d1, model.parameters(), create_graph=True)[0]</code> , but I haven’t tested whether that works.</p>

ccook: <p>Ah, sorry, I should have clarified that it’s the first line that creates an error.</p>
<p>I’ve gotten a bit closer by doing:</p>
<pre><code class="lang-auto">d1 = torch.autograd.grad(loss, model.linear.weight, create_graph=True)
</code></pre>
<p>Although currently giving me another error: “one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [13, 1]], which is output 0 of TBackward, is at version 151; expected version 150 instead.” (Trying to sort out what’s gone awry now)</p>

Varal7: <p>Here’s a solution that’s a bit hacky but should give you what you want:</p>
<pre><code class="lang-auto">def f(weight):
    m = MultinomialLogitWithEmbeddings(1, 1)
    m.linear.weight.data = weight
    x_cont = torch.ones(1, 1, 1)
    x_cat = torch.LongTensor([[[0]]])
    return m(x_cont, x_cat)

hessian = torch.autograd.functional.hessian(f, model.linear.weight)
</code></pre>
Gradient of output wrt specific inputs

tvdw: <p>Hi there,</p>
<p>I am trying to retrieve the gradients of the output variables wrt the input variables of a Neural Network model (loaded from a .pt file). Note that the neural network describes a dynamical system in which a history of states is taken into account. The network is too complicated to have an intuition of the range in which the gradients should lie.</p>
<p>I am currently tackling it the following way:</p>
<pre><code>    x = torch.zeros(1,1500) # 1500 inputs to model
    x[0][:] = torch.from_numpy(array_of_inputs)  # Give all input and state data we have to the NN
    x.requires_grad = True  # Make sure gradients can be extracted

    # Compute outputs
    out = model(x)
    out = out[0]

    alphadot = out[1]  # Get gradient of variable alphadot (index in output array is 1)
    gradient = torch.autograd.grad(outputs=alphadot, inputs=x, grad_outputs=torch.ones_like(alphadot),
                                   retain_graph=True)
    gradient = gradient[0].tolist()
    gradient = gradient[0]
</code></pre>
<p>The variable gradients contains the gradients of output with index 1 (alphadot) with respect to all 1500 input variables in x. Does the code snippet above behave as I think it does in my description? If not, what am I doing wrong?</p>
<p>Thanks a lot for your help!</p>

albanD: <p>Hi,</p>
<p>Yes this code does what you think <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

tvdw: <p>Thanks for the confirmation!</p>

peymanhk20: <aside class="quote no-group" data-username="tvdw" data-post="1" data-topic="58585">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tvdw/48/12334_2.png" class="avatar"> tvdw:</div>
<blockquote>
<pre><code class="lang-auto">    gradient = gradient[0].tolist()
    gradient = gradient[0]
</code></pre>
</blockquote>
</aside>
<p>What if we want to store the gradient of output w.r.t input for every epoch? Does this code work too? why in this code there is no loss.backward()?</p>

Varal7: <p>There is no notion of epoch nor loss is OP’s example.<br>
Checkout <a href="https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html#sphx-glr-beginner-blitz-neural-networks-tutorial-py" class="inline-onebox" rel="noopener nofollow ugc">Neural Networks — PyTorch Tutorials 1.9.0+cu102 documentation</a> for an intro to how neural networks with PyTorch.</p>
Question about Oversampling

PeterCheng: <p>Hi All,</p>
<p>For an imbalanced dataset, can we first use oversampling to balance the two classes, and then use Opacus for DP training over the artificially balanced dataset?</p>

ffuuugor: <p>Hi <a class="mention" href="/u/petercheng">@PeterCheng</a> and thanks for your question.</p>
<p>You can certainly do that, but it would affect how you would interpret privacy for the trained model.<br>
Opacus provides privacy guarantees with respect to each individual sample in the dataset. We also use privacy amplification by subsampling - i.e. the epsilon guarantees directly tied to the probability of being sampled for a minibatch.</p>
<p>When you oversample, you essentially increase sampling rate for that particular data record.<br>
To get an eps estimate for oversampled instances, you need to modify <code>PrivacyEngine.get_privacy_spent</code> method and use adjusted sample rate</p>

PeterCheng: <p>Thank you for the detailed explanation. I have a better understanding now.</p>
DP regression with Opacus

AliSh: <p>Hello,</p>
<p>Is there any example for DP regression in Opacus please?</p>

karthikprasad: <p>Hi <a class="mention" href="/u/alish">@AliSh</a>, thank you for your questions. I’m presuming you are looking for an example of linear regression task. While we do not have an example in the Opacus repo at the moment, it should still work out of the box just and the workflow looks the same as that of other examples. (DP-SGD clips and noises gradients rather than the outputs).</p>
<p>Please let us know if you are running into any issues there.</p>
<p>P.S.: We do welcome and appreciate pull-request if you’d like to contribute an example <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
How to multiply multiple masks with rgb image?

seyeeet: <p>I have a mask tensor which is in shape of <code>BxCxHxW</code> where <code>C</code> represents the number of masks (<code>C=10</code>). I also have an <code>input</code> tensor that is in shape of <code>Bx3xHxW</code>.<br>
I wonder if I can do the following without loop to be faster…</p>
<p>how I can multiply my input with the mask so that I have output of shape <code>C*Bx3xHxW</code>.<br>
where <code>output[:C,...]</code> is the output of <code>input[0,...]</code> times <code>mask[0,...]</code>, and output[C:2C] is equal to <code>input[1,...]</code> times <code>mask[1,...]</code>, and so on…</p>
<p>In the loop form it would be like:</p>
<pre><code class="lang-auto">output = torch.zeros(B*C,3,H,W)
for b in range(B):
   for i in range(C):
      output[i + b*C] = torch.mul(input[b].unsqueeze(0), 
                                     mask[b,i].unsqueeze(0).unsqueeze(0))
</code></pre>

eqy: <p>I think this is a natural place to do an einsum as it is effectively a variation of an outer-product style computation:</p>
<pre><code class="lang-auto">import torch
import time

B = 16
C = 64
H = 32
W = 32

input = torch.randn(B, 3, H, W)
mask = torch.randn(B, C, H, W)

output = torch.zeros(B*C,3,H,W)
t1 = time.time()
for b in range(B):
   for i in range(C):
      output[i + b*C] = torch.mul(input[b].unsqueeze(0),
                                    mask[b,i].unsqueeze(0).unsqueeze(0))
t2 = time.time()
output2 = torch.einsum('blhw,bchw-&gt;bclhw', input, mask)
output2 = output2.reshape(B*C, 3, H, W)
t3 = time.time()
print(t2-t1, t3-t2)
print(torch.allclose(output, output2))
</code></pre>
<pre><code class="lang-auto">0.0157625675201416 0.0012059211730957031
True
</code></pre>
Iterating model parameters in a coarser granularity

wasabi: <p>Say I have a resnet that looks like the following (truncated):</p>
<pre><code class="lang-auto">ResNet(
  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (relu): ReLU(inplace=True)
  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
  (layer1): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (1): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )
)
</code></pre>
<p>The normal iterating way of</p>
<blockquote>
<p>for name, W in model.named_parameters():<br>
print(name)<br>
will give me a list that looks like</p>
</blockquote>
<pre><code class="lang-auto">conv1.weight
bn1.weight
bn1.bias
layer1.0.conv1.weight
layer1.0.bn1.weight
layer1.0.bn1.bias
layer1.0.conv2.weight
layer1.0.bn2.weight
layer1.0.bn2.bias
layer1.1.conv1.weight
layer1.1.bn1.weight
layer1.1.bn1.bias
layer1.1.conv2.weight
layer1.1.bn2.weight
layer1.1.bn2.bias
</code></pre>
<p>However, is it possible to iterate the model such that I can get</p>
<pre><code class="lang-auto">conv1
bn1
layer1.0.conv1
layer1.0.bn1
layer1.0.conv2
...
</code></pre>
<p>that is, the finest granularity that I iterate is the predefined torch modules? Thanks.</p>

wasabi: <p>Answering my own question is anyone else runs into this. We can simply iterating using model.modules() or model.children() given the need.</p>
How to load saved Adam optimizer to a new Adam optimizer with different learning rate?

TH_M_Bui_Dinh: <p>Hi, I have trained my model with Adam optimizer (lr=0.0001) and successfully saved the optimizer’s state_dict. Now I want to continue training with with a different learning rate (say lr=0.00005). What is the proper way to load the saved optimizer of learning rate 0.0001 to a new Adam optimizer instance with a different learning rate?</p>

safakkbilici: <p>You can set all learning rates by doing</p>
<pre><code class="lang-python">for group in optim.param_groups:
    group["lr"] = 0.001
</code></pre>
<p>Also using a learning rate scheduler might be helpful as well.<br>
I hope these solutions would be helpful.</p>

TH_M_Bui_Dinh: <p>Maybe you misunderstood what I was asking.<br>
Say that I have trained the model for 30 epochs with:</p>
<blockquote>
<p>optimizer = torch.optim.Adam(model.parameters(), lr=0.0001)</p>
</blockquote>
<p>and saved the optimizer’s state_dict:</p>
<blockquote>
<p>torch.save(optimizer.state_dict(), PATH)</p>
</blockquote>
<p>Now I want to continue the training with a new optimizer with different learning rate:</p>
<blockquote>
<p>optimizer = torch.optim.Adam(model.parameters(), lr=0.00005)</p>
</blockquote>
<p>How should I load the state_dict of the previous optimizer to the new one?<br>
Should I load it at all or I just need to use the new optimizer without any loading?</p>
Applying skip connection to seq2seq model

hw-kim-ivan: <p>Hello! I’m looking for help with applying skip connection in Seq2Seq model:)<br>
I got each 3 LSTM layers for Encoder(encoder 1, 2 and 3) and Decoder(decoder 1, 2, and 3) like below.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/8/d8e6c29fb39930d631321e21f5854dcddb55a5cb.png" data-download-href="https://discuss.pytorch.org/uploads/default/d8e6c29fb39930d631321e21f5854dcddb55a5cb" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/8/d8e6c29fb39930d631321e21f5854dcddb55a5cb.png" alt="image" data-base62-sha1="uWNxbcfEHiYckHJnR6bSn2jKhu3" width="605" height="500" data-dominant-color="242424"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">661×546 6.16 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And I want to connect (encoder 1 with decoder 3), (encoder 2 with decoder 2) and (encoder 3 with decoder 1) like C-shape.</p>
<p>Do you have any idea with connecting those LSTM layers?</p>
Efficient Gradient Penalty

LeanderK: <p>I want optimise the following loss function wrt to the parameters of f efficiently:</p>
<pre><code class="lang-auto">loss(x,y) = criterion(f(x),y) + lambda*max_i ||grad_(x_i) f(x_i)||_2
</code></pre>
<p>so I want to optimise the criterion regularised by the max L2 Norm of the gradient of f(x_i) with respect to (x_i), so the input. With (x_i) i mean a single element of the batch x. How can I formulate this? I struggle to construct a single computational graph, x and y is a batch of data.</p>
<p>This is not in the context of a GAN, f is a usual classifier</p>
How do I train on variable length audio?

Superklez: <p>How do I allow my neural network to train on an audio stream? In particular, what I want to achieve is to take in, for example, an audio that is 4 seconds long and pass a sliding analysis window wherein I classify the audio in every window and return the class with the highest average probability in all the windows. I’m not sure how to do this in PyTorch. Does this have something to do with the data loaders? Or is it more on the design of the network?</p>

eqy: <p>A classic solution in this case would be to simply pad the audio to the longest length possible.</p>

shivammehta007: <p>As I mentioned in another answer, <span class="mention">@egy</span>’s answer is correct to pad to the longest length possible and you can take the example of DataLoaders from Nvidia’s Tacotron2 source code <a href="https://github.com/NVIDIA/tacotron2/blob/master/data_utils.py" rel="noopener nofollow ugc">here</a>.</p>
Batch of audio with unequal length and transformations

Superklez: <p>I understand that when dealing with audio with unequal length, one could define a collate function that would pad all audio to the same  length, i.e. the length of the longest audio. However, if I were to do some transformations in my custom dataset’s <code>__getitem__</code>, like taking the log-Mel spectrogram, how do I pad audio of unequal length? My guess is to still use a collate function and pad along the last dimension of the batch of audio transformed to a log-Mel spectrogram, but I want to know what the best practices are with regards to this matter. Thanks!</p>

shivammehta007: <p>What you mentioned is the best practice as per my knowledge and referring to the Nvidia’s Tacotron2 source code <a href="https://github.com/NVIDIA/tacotron2/blob/master/data_utils.py" rel="noopener nofollow ugc">Tacotron2/data_utils.py</a> the same is done here.</p>