akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Seed effect on model training

Ahmed_Jimy: <p>Hello, I am training a binary classification model and I want my results to be reproducible. So, I am using this function before every training session begins:</p>
<blockquote>
<p>def seed_everything(seed_value, use_cuda):<br>
if use_cuda:<br>
torch.cuda.manual_seed(seed_value)<br>
torch.cuda.manual_seed_all(seed_value)<br>
torch.backends.cudnn.deterministic = True<br>
torch.backends.cudnn.benchmark = False</p>
</blockquote>
<p>What I am asking is, should I seed_everything even if I loaded a model using last model.state_dict() from a previous session?</p>
<p>I am asking this because for example if I trained for 30 epochs and saved model weights at epoch 20, then I start from epoch 20 again, the results of epoch 20 to 30 are not the same as the last training session.</p>

ptrblck: <aside class="quote no-group" data-username="Ahmed_Jimy" data-post="1" data-topic="90004">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ahmed_jimy/48/6742_2.png" class="avatar"> Ahmed_Jimy:</div>
<blockquote>
<p>the results of epoch 20 to 30 are not the same as the last training session.</p>
</blockquote>
</aside>
<p>This might be caused by the shuffling of the data, if you haven’t restored the state of epoch 20 for the data loading before continuing the training.<br>
If you are shuffling the data, you could try to add 20 “empty” epoch iterations using the <code>DataLoader</code> and check the final performance again.</p>
How to use kDouble type input for Linear layer in C++

kila_suelika: <p>We use</p>
<pre><code class="lang-auto">linear3(register_module("linear3",torch::nn::Linear(100,out_sz)))
</code></pre>
<p>to create a Linear layer. However it looks like that the default type of it is <code>kFloat</code>. If <code>x</code> is <code>kDouble</code> type, then <code>linear3-&gt;forward(x)</code> will generate an error.</p>
<p>Then how do we use double type data as input?</p>

kila_suelika: <p>I found that one can use</p>
<pre><code class="lang-auto">model.to(torch::kDouble);
</code></pre>
<p>to cast parameters of the whole model to double type.</p>
Torch.jit.script error: Tensor cannot be used as a tuple

kevpan: <p>Here is my forward function.<br>
def forward(self, inputs):<br>
if self.first_time:<br>
p3, p4, p5 = inputs</p>
<pre><code>        p3_in = self.f3(p3)
        p4_in = self.f4(p4)
        p5_in = self.f5(p5)

    else:
        p3_in, p4_in, p5_in = inputs
</code></pre>
<p>error shows:<br>
RuntimeError:<br>
Tensor cannot be used as a tuple:<br>
if self.first_time:<br>
p3, p4, p5 = inputs<br>
~~~~~~ &lt;— HERE</p>
<p>is there anyway to resolve this error?</p>

111365: <p>I got the same problem. Did you solve it ? THX</p>
How to fix "Unknown IValue type for pickling: Device" in PyTorch 1.3?

kaituoxu: <p>I got a model trained with PyTorch 1.4. If I script and save this model with PyTorch 1.4, it will save successfully, but I need to script and save this model with PyTorch 1.3. When I save model with 1.3, I got error message:</p>
<pre><code class="lang-auto">RuntimeError: Unknown IValue type for pickling: Device (pushIValueImpl at /pytorch/torch/csrc/jit/pickler.cpp:125)
</code></pre>
<p>I want to know how to fix this Device issue. Thx.</p>

Michael_Suo: <p>It implies that somewhere in your model, you’re storing a device type and trying to serialize it (like <code>self.foo = my_device</code>). This is supported in 1.4 but not 1.3; if you want your model to work with 1.3, you’ll have to avoid storing references to devices.</p>

kaituoxu: <p>Thx.<br>
In my class, I have a</p>
<pre><code class="lang-auto">@property 
def device():...
</code></pre>
<p>After I delete this part, I can convert model successfully.</p>
How to perform optimization with Multiple Optimizer?

SKYHOWIE25: <p>Hi</p>
<p>Suppose I have three optimizers and I want to optimize a graph three times within one batch. Which one is the right order to call those functions?</p>
<ol>
<li></li>
</ol>
<blockquote>
<pre><code>optimizer1.zero_grad()
optimizer2.zero_grad()
optimizer3.zero_grad()

loss1.backward(retain_graph=True)
loss2.backward(retain_graph=True)
loss3.backward()

optimizer1.step()
optimizer2.step()
optimizer3.step()
</code></pre>
</blockquote>
<ol start="2">
<li></li>
</ol>
<blockquote>
<pre><code>optimizer1.zero_grad()
loss1.backward(retain_graph=True)
optimizer1.step()

optimizer2.zero_grad()
loss2.backward(retain_graph=True)
optimizer2.step()

optimizer3.zero_grad()
loss3.backward()
optimizer3.step()
</code></pre>
</blockquote>
<p>Thanks</p>

ptrblck: <p>I don’t know which order is correct, but I’m curious why you would like to use 3 different optimizers sequentially.<br>
Note, that after a weight update your model moved already on the loss surface and the current loss has to be calculated for your current weights. Applying the “old” gradients seems to be wrong in my opinion, so what is your approach? <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

SimonW: <p>First option: each optimizer will see sum of gradients from three losses. In fact, you can do <code>(loss1 + loss2 + loss3).backward()</code>, which is more efficient.<br>
Second option: each optimizer will see gradients only from the specific loss.</p>

SKYHOWIE25: <p>Hi</p>
<p>The three losses and optimizers have their own purpose. For example, one is for the normal backpropagation to update the parameters of the whole model. Another is used to update some certain layers with specific gradient.</p>

SKYHOWIE25: <p>Hi Simon</p>
<p>One optimizer is used to update the whole graph and the others are only used to update part of it. So i guess the second one is the right choice for me?</p>

SimonW: <p>It still depends on what gradient you want your optimizers to see <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Satish1901: <p>Hello</p>
<p>Were you able to do it. I am trying something similar but getting issue</p>
<blockquote>
<p><code>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [16]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</code></p>
</blockquote>
Anyone implemented bi-lstm wrapper that takes customized cell and able to handle packed_sequence

meng_lin: <p>I’m trying to reproduce the result of a paper which uses layernorm lstm cells. The author uses tensorflow to implement it so it’s fairly simple. I searched for bi-lstm with layer norm online but I only found those works that does not support different lengths. I implemented a bi-lstm that handles different lengths before but it’s slow and it doesn’t take packed sequence as the official one does.<br>
I’m not familiar with Jitter also.<br>
I wonder if a good wrapper exists.</p>
Can anyone suggest a best pyTorch book for deep learning?

jyotirmoykarjee: <p>Can any one tell a good (best) book to follow-up which provides a overall concepts of deep learning using pytorch ?</p>

Nikronic: <p>Hi,</p>
<p>Here is the PyTorch book:<br>
</p><aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">
      <a href="https://pytorch.org/deep-learning-with-pytorch" target="_blank" rel="nofollow noopener">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="https://pytorch.org/assets/images/pytorch-logo.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://pytorch.org/deep-learning-with-pytorch" target="_blank" rel="nofollow noopener">PyTorch</a></h3>

<p>An open source deep learning platform that provides a seamless path from research prototyping to production deployment.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>You can get it for free AFAIK.</p>
<p>Also, there are many great resources out there, so I won’t stop just here.</p>
<p>Bests</p>
Obtain the class and class index

Ganga: <p>how do i print the class and class indices values,where i am using torchvision.datasets.ImageFolder</p>

ptrblck: <p>You could print the <code>dataset.classes</code> and <code>dataset.class_to_idx</code> attributes.</p>

Ganga: <blockquote>
<p>print(dataset.classes)</p>
</blockquote>
<p>File “main.py”, line 193, in main<br>
print(dataset.classes)<br>
NameError: global name ‘dataset’ is not defined</p>

ptrblck: <p><code>dataset</code> is a placeholder name for the <code>Dataset</code> object variable in your script, so you would need to change it based on your variable naming.</p>
FlatMapDataset - transforming a map-style dataset into an iterable-style dataset

pkubik: <p>I sketched the concept of flat-map operation for PyTorch datasets - <code>FlatMapDataset</code>. You provide it with a regular map-style dataset and a transformation function that yields a sequence. When iterated over, it fetches an element of the source <code>Dataset</code> and pass it to your transformation function. All results of the function are being continously concatenated into stream of examples (i.e. <code>IterableDataset</code>). It can be passed to a <code>DataLoader</code> to form the training batches.</p>
<p>Here is an example:</p>
<pre><code class="lang-auto">source_dataset = SourceDataset()
source_train, source_valid = torch.utils.data.random_split(
        source_dataset, [9000, 1000])

def crop_boxes(example):
    image, boxes = example
    for box in boxes:
        yield crop_image(image, box)

train_dataset = FlatMapDataset(source_train, crop_boxes)
data_loader = DataLoader(train_dataset)
</code></pre>
<p>It is supposed to be used in situations where one file have to be split into an unknown number of actual training examples, e.g. when you’d like to extract multiple crops of a single training image to train an autoencoder / GAN. Sometimes a large pre-processing is not suitable in such a setting.</p>
<p><a href="https://yrwehere.hashnode.dev/flatmapdataset-flat-map-data-transform-for-pytorch-ckcxod0s2006qeks13iic89an" class="onebox" target="_blank" rel="nofollow noopener">https://yrwehere.hashnode.dev/flatmapdataset-flat-map-data-transform-for-pytorch-ckcxod0s2006qeks13iic89an</a></p>
Building images dataset and storing on disk

agent04: <p>I have 500,000 images of size [224,224,3]. How should I proceed to store these images in disk such that it can be later used in vision model ? Will be helpful if someone can bifurcate these findings by IO operations time. (Writing time to disk and reading time using data loader from disk).</p>
<p>I tried to use HDF5, but iterating on data loader takes much time.</p>
Pytorch distributed concurrent queue/buffer

ryuxin: <p>Hi everyone,</p>
<p>I am wondering whether there is anyway for pytorch distributed to build one concurrent queue(or buffer) between parameter server and workers.</p>
<p>So that, every worker can work as a producer to send the msg to the concurrent queue.</p>
<p>And the parameter server can work as consumer to consume msg from concurrent queue.</p>
<p>Besides, parameter server can detect the length of the concurrent queue.</p>
<p>Thank you!</p>

mrshenli: <p>Hey <a class="mention" href="/u/ryuxin">@ryuxin</a>, can this be implemented as a wrapper on top of the <a href="https://pytorch.org/docs/master/rpc.html" rel="nofollow noopener">RPC API</a>? For example, can you implement the queuing logic as an RPC target function? Some related tutorials:</p>
<ol>
<li><a href="https://pytorch.org/tutorials/intermediate/rpc_param_server_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/rpc_param_server_tutorial.html</a></li>
<li><a href="https://github.com/pytorch/tutorials/blob/release/1.6/intermediate_source/rpc_async_execution.rst" rel="nofollow noopener">https://github.com/pytorch/tutorials/blob/release/1.6/intermediate_source/rpc_async_execution.rst</a></li>
</ol>

ryuxin: <p>Thanks for the hint!</p>
Do the same transformations on a list of PIL Image

fgauthier: <p>I have a dataset of around 200 images. However, the format of my image is not standard because it has 6 dimensions.</p>
<p>One image would look like this (before being cast to tensor and being stack in a 3d Tensor):</p>
<p>[img0, img1, img2, img3, img4, img5]. (They are all grayscale PIL Image )</p>
<p>However my dataset is quite small and I want to do upsampling.  I wanted to use torchvisions.transforms to do that. My problem is that I need to perform the exact same list of transformation (like RandomHorizontalFlip, Random Rotation, etc…) for all the dimension of my image and but I do not want to do the same transformation for every image. Is there a way to do so?</p>

ptrblck: <p>You could use the functional API of <code>torchvision.transforms</code> as given in <a href="https://discuss.pytorch.org/t/torchvision-transfors-how-to-perform-identical-transform-on-both-image-and-target/10606/7">this example</a>.<br>
This would make sure to apply the same “random” transformation on all image slices.<br>
Let me know, if this would work for you.</p>

fgauthier: <p>Yes I found your example. It worked thank you very much.</p>
How to separate classes in a imagefolder

Muhammad_Ahmed: <p>I have my dataset stored in 2 folders named test and train. I have read both of them under a single imagefolder as it was giving error when i specifically gave path for train or test folder.<br>
Now I want to separate the imagefolder on the basis of its classes. Test and train.<br>
How do i separate data on the basis of its class.<br>
Please help<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>I’m not sure I understand the use case completely.<br>
Are <code>test</code> and <code>train</code> the two classes you would like to split or are they corresponding to the training and testing datasets and contain subfolders for each class inside of them?</p>
<p>In the latter case passing <code>root=path_to_train_or_test</code> to <code>ImageFolder</code> should work.<br>
What kind of error are you seeing?</p>

Muhammad_Ahmed: <p>My data has 2 folders<br>
Test and train<br>
Inside each folder, we don’t have separate classes, it’s plain images.<br>
To use image folder, we need to have classes.<br>
I found a solution, I used image folder and then used subset call to separate them.<br>
Thanks.</p>
Reduce training time with large sample size and small batch size

KENG: <p>I have a data set of about 15 million samples, and I noticed that the neural network I am training converges the best with a relatively small batch size of 128. The problem is, if I use this small batch size, it takes really long time to go through the training sample, while it only takes about one fourth of my GPU memory, and the GPU utitily is also about 30%. Is it possible to reduce the training time through some sort of parallelism? The common data parallelism seems to split a large batch sample to multiple GPUs, which is not what I am looking for. Many thanks!</p>
Problems with Squeezenet

GastonLagaffe2013: <p>Salut,<br>
based on the article “Finetuning Torchvision Models” (<a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html</a>) I wrote a python script that trains all mentioned networks.<br>
For all networks like alexnet, ResNet, Densnet, Vgg and resNeXt, the training goes well with the expected reduction in the loss value and the accuracy of the network classifying the images. ROC values are good as well.<br>
Except for SqueezeNet: loss stays at a high value and after some epochs, the net classifies all images the same way. This is the same for single-label as well as multi-label.<br>
As I use the same code to run the training and only change the model definition, I think I am doing something wrong with initializing Sqeezenet:</p>
<pre><code class="lang-auto">model = models.squeezenet1_1(pretrained=usePreTrain)
model.classifier[1] = nn.Conv2d(512, nnClassCount, kernel_size=(1,1), stride=(1,1))
set_parameter_requires_grad(model, feature_extract)
model.num_classes = nnClassCount
criterion = nn.BCELoss(size_average = True)
</code></pre>
<p>I uploaded the full code to github (<a href="https://github.com/GastonLagaffe2013/PyTorchModels" rel="nofollow noopener">https://github.com/GastonLagaffe2013/PyTorchModels</a>)</p>
<p>Any help is appreciated, Mathias</p>
Which activation function to use depending on your data?

dllacer: <p>Hi everyone,</p>
<p>This is a general doubt I have about the relationship between the activation function and the data.</p>
<p>Imagine our data (images), after normalization, is centered at 0 and take values between -1 and 1. It means our network will try to output images which values will be also between -1 and 1.<br>
So, for example, if we use ReLU as activation function after each layer, we are removing negative values during training, so the network will never be able to correctly predict our targets.</p>
<p>Is that right?<br>
If I use ReLU, should I normalize my data between 0 and 1 insted of between -1 and 1?</p>
<p>PD: I ask this because I saw cases where they normalize between -1 and 1 and they use ReLU successfully.</p>
<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>

KFrank: <p>Hi Deep!</p>
<aside class="quote no-group" data-username="dllacer" data-post="1" data-topic="90127" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/50afbb/48.png" class="avatar"> dllacer:</div>
<blockquote>
<p>if we use ReLU as activation function after each layer, we are removing negative values during training, so the network will never be able to correctly predict our targets.</p>
</blockquote>
</aside>
<p>No, this way of looking at it isn’t correct.</p>
<p>The reason is that the weights in your layers can be negative and<br>
the biases are not constrained.  So a negative input value can be<br>
multiplied by a negative weight or have a positive bias added to it,<br>
and therefore become positive, before it gets to the <code>ReLU</code> activation.</p>
<p>Best.</p>
<p>K. Frank</p>
Understanding GPU Utilisation during Inference

badesairam: <p>Im trying to see the effect of batchsize on time taken for inference on BERT model. I used different batchsizes to perform inference , I get inference speeding up until certain batchsizes and increasing slowly after a certain batchsize. I am attaching the GPU RAM and time taken for inference in the below table for different batch sizes<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/5/554fc2e88f3f9f029c91f77635165cc6376ac165.png" data-download-href="https://discuss.pytorch.org/uploads/default/554fc2e88f3f9f029c91f77635165cc6376ac165" title="Screenshot 2020-07-22 at 7.51.17 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/554fc2e88f3f9f029c91f77635165cc6376ac165_2_690x276.png" alt="Screenshot 2020-07-22 at 7.51.17 PM" data-base62-sha1="caHtjT9WJPXRKWPgTIPu0RwUd2l" width="690" height="276" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/554fc2e88f3f9f029c91f77635165cc6376ac165_2_690x276.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/554fc2e88f3f9f029c91f77635165cc6376ac165_2_1035x414.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/5/554fc2e88f3f9f029c91f77635165cc6376ac165.png 2x" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2020-07-22 at 7.51.17 PM</span><span class="informations">1144×458 14.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>GPU used is <code>GeForce GTX 1080</code></p>
<p>GPU utilisation from nvidia-smi command is always 100% . I understand  this utilisation does not correspond to the GPU resource utilisation.Im wondering if all the GPU resources are utilised at batchsize 32 without consuming all the RAM. Is there a way to monitor the actual GPU resource utilization?</p>
<p>Adding the code that is profiled</p>
<pre><code class="lang-auto">start_time = time.time()

    for batch_num,(input_ids, input_mask, segment_ids) in enumerate(tqdm(pred_dataloader, desc="Predicting")):
        percent_done = float(batch_num)*100/num_batches
        input_ids = input_ids.to(device)
        input_mask = input_mask.to(device)
        segment_ids = segment_ids.to(device)

        with torch.no_grad():
            logits, _ = model(input_ids, segment_ids, input_mask)

        logit_max = torch.max(F.softmax(logits, dim=1), dim=1)
        logits_soft = F.softmax(logits, dim=1).cpu().numpy()
        all_preds = np.concatenate((all_preds, logits_soft))
        logits = torch.argmax(F.log_softmax(logits, dim=1), dim=1)
        logits = logits.detach().cpu().numpy()
        logit_max = logit_max.values.detach().cpu().numpy()

        pred_vals.extend(logit_max)
        preds.extend(logits)
    all_preds = all_preds[1:]
     logger.info("TIme for batch loop %f" % (time.time() - start_time))
</code></pre>
Save and load Inline and Lowered Graphs as ScriptModules

vdantu: <p>Is it possible to save a inline’d and lowered graph as a ScriptModule and load it using torch.jit.load? My use case is to resolve the ClassType inputs into actual accessed tensor parameters.</p>

eellison: <p>Yes, this should be possible. What did you try that didn’t work ?</p>

vdantu: <p><a class="mention" href="/u/eellison">@eellison</a>:  Thanks for your response. I need some help/suggestion in how to proceed. Lowering a graph (which belongs to a scriptmodule’s method) returns a lowered graph and parameters. This pass replaces the input class object (for example <code>"self"</code>) with actual parameters belonging to <code>"self"</code> which are accessed throughout the graph. Now, creating a method with this lowered graph and adding this method to a module will seem like adding a <code>method</code> which takes original inputs along with the parameter tensors. The question I had is, how do I package these parameters and this lowered graph such that when we run the method, I would only need to provide the inputs to original ScriptModule and not these parameters? Not sure if I am explaining the question well. Please do let me know if I am not thinking straight.</p>

vdantu: <p>Could we do the following?</p>
<ol>
<li>Run LowerGraph pass: This pass will return a new Graph and a list of tensors.</li>
<li>Add these Tensors as constants in the graph and remove the corresponding Graph input from the graph. Replace all the input’s references with the constants references.</li>
<li>Create a module from this graph.</li>
</ol>
<p>Is this a valid approach?</p>
<p>This will create a graph whose inputs are only the original model’s inputs.</p>
Pytorch model inference in Java

mahesh_sivan: <p>I have a pytorch trained model with BERT (pytorch_model.bin) that i trained using python. Is there a way i can load and infer that model in Java ?</p>
<p>I used this to train <a href="https://github.com/kamalkraj/BERT-NER" rel="nofollow noopener">https://github.com/kamalkraj/BERT-NER</a></p>
Merge consistent convolutional layers

watermelon: <p>Hi there!</p>
<p>Suppose I have three consistent convolutional layers like (256, 144, 1)-&gt; (144, 144, 3) -&gt; (144, 256, 1). In fact, the first and the last one are matrices. Is there any way to ‘merge’ these three layers in a one preserving the trained information? Since there is no non-linearity between them. It’d be similar to a linear layers, for example (we can replace 2 of them with a one).</p>
<p>I found smth similar here <a href="https://stackoverflow.com/questions/58357815/how-do-i-merge-2d-convolutions-in-pytorch" rel="nofollow noopener">https://stackoverflow.com/questions/58357815/how-do-i-merge-2d-convolutions-in-pytorch</a></p>
Infinite Random Sampler

beneyal: <p>Does PyTorch have a sampler that continually gives me a batch of random samples, without reaching a <code>StopIteration</code> error when it’s done? I want the ability to sample infinitely. If not, how could I implement one?</p>
<p>Thanks!</p>

albanD: <p>Hi,</p>
<p>I am not sure this would work. You would need to subclass <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.Sampler"><code>Sampler</code></a> and give an instance of your custom sampler to the dataloader you’re creating.<br>
The problem is that the length of the sampler cannot be infinite as python does not have infinite integer. So you may have to use a very large number there like <code>int(1e100)</code>.<br>
Then the iterator is simply an iterator that return a random valid index every time. You can do that with a function that yeild such a number inside an infinite loop for example.</p>

beneyal: <p>I’ll give it a try <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

gt_tugsuu: <p>I do this by setting the length of dataset to 1e100.</p>
<pre><code class="lang-auto">...
def __len__(self):
        return 1e100
...
</code></pre>

bananacode: <p>You can do this with the RandomSampler by setting replacement=True and num_samples to a very high number.</p>
<pre><code class="lang-auto">sampler = RandomSampler(dataset, replacement=True, num_samples=1e10)
</code></pre>
DataParallel gives "arguments are located on different GPUs" when you assign to `forward()`

drevicko: <p>I met an interesting (and breaking) behaviour with DataParallel when I override <code>forward()</code> by assignment. Something like:</p>
<pre><code class="lang-auto">class MyEmbedding(nn.Embedding):
    def __init__(self, swap_fwd=True, *args, **kwargs):
        super().__init__(*args, **kwargs)
        if swap_fwd:
            self.FC = nn.Linear(self.embedding_dim, self.embedding_dim)
            self.forward = self.forward_
        else:
            self.FC = None

    def forward_(self, inputs):
        embeds = super().forward(inputs)
        return self.FC(embeds)
</code></pre>
<p>This works fine on a single gpu, but with multiple, I get “arguments are located on different GPUs”. It seems that when <code>_forward()</code> is executing, <code>self</code> is not the appropriate replica (on the correct gpu), but is the original object (typically on gpu 0).</p>
<p>If instead I override <code>forward()</code> in the normal way, everything works fine. Something like:</p>
<pre><code class="lang-auto">...
    def forward(self, inputs):
        embeds = super().forward(inputs)
        if self.FC is not None:
            return self.FC(embeds)
        return embeds
</code></pre>
<p>I’d be interested to know what’s going on?</p>

ptrblck: <p>I assume that you are calling <code>model.forward_()</code> in your code instead of the model directly?<br>
If so, then this breaking behavior is expected, as utility methods such as hooks and data parallel rely on the “standard” usage of calling the model directly.<br>
For <code>nn.DataParallel</code>, the <code>model.__call__</code> method would call into <code>DataParallel.forward</code>, which takes care of the parallelization as seen <a href="https://github.com/pytorch/pytorch/blob/5c9918e757b019564e74ae6f676cacfe70a87afd/torch/nn/parallel/data_parallel.py#L147-L162">here</a>.</p>

drevicko: <p>No, I use the usual <code>__call__</code> interface (ie: treat the module instance as a function). It actually calls my forward_() method, but the <code>self</code> in that call is the original module instance, not the replica created by DataParallel. That means it’s parameters are not on  the same gpu as the <code>inputs</code> passed to forward.</p>
<p>I suspect it’s something related to the way python dynamically populates the <code>self</code> variable - when I do <code>self.forward = self.forward_</code>, the current self (ie: the original one that executes <code>__init__</code>) is associated with <code>forward_</code>, and this association isn’t changed when DataParallel clones the module. I looked at it with a debugger, and <code>id(self)</code> within the <code>forward_</code> method is different to <code>id(self)</code> in <code>Module.__call__()</code> immediately beforehand (at least for threads working with a different gpu).</p>
<p>I put this here partly for others to find, should they encounter this behaviour, and partly because I don’t understand how that can happen, how the mechanism works to cause this behaviour.</p>
Different behaivour of exported model in TorchScript

eusebioaguilera: <p>Hi there,</p>
<p>I am trying to export and use a model defined in pytorch by using the “torch.jit.script” method. When I try to use the exported model the result obtained is awful if you compare with the original results. The model loads pretrained weigths and after all of these I use the method “torch.jit.script” and save it to the file. I have found some related <a href="https://github.com/pytorch/pytorch/issues/30754" rel="nofollow noopener">issue in github</a>, however I have not managed to solve the problem.</p>
<p>Any idea of the problem? Thank you in advance.</p>

Michael_Suo: <p>Do you have some code to reproduce the issue? It’s hard to say what’s going on without more information</p>

eusebioaguilera: <p>Thank you for your response. Finally, I have managed to get similar results using the exported model. The problem was that the tensor was transposed by using a custom ToTensor transform instead of using the standard ToTensor transform.</p>
Specifying memory destination when copy tensor to cpu, just like numpy `out=`

JiaRu2016: <p>is there any way to specify memory location when copy a cuda tensor to cpu? In numpy we can specify <code>out=destination</code> for some functions, is there similar functionality for pytorch, maybe like:</p>
<pre><code class="lang-python">destination = np.empty(shape=(...very_large_n...)
cuda_tesnor.cpu(out=destination[some_slice])
</code></pre>
<p>I’v tried</p>
<pre><code class="lang-python">destination[some_slice] = cuda_tensor.cpu().data.numpy()
</code></pre>
<p>found that is used twice time as purely <code>tensor.cpu().data.numpy()</code>, so I suggest it copied two times.</p>
Gradients of neural network for 3 Dim input - 1 Dim output regression

beeginer: <p>I want to get the gradients of model(neural network).<br>
In tensorflow, there is a ‘tf.gradient’<br>
so the following way is possible.</p>
<pre><code class="lang-auto">x = tf.placeholder(shape=[None,1],dtype=tf.float32)
y = tf.placeholder(shape=[None,1],dtype=tf.float32)
inp = tf.concat([x,y],axis=-1)

def network(x):
	x = tf.layers.dense(x,128,activation=tf.nn.relu)
	x = tf.layers.dense(x,128,activation=tf.nn.relu)
	x = tf.layers.dense(x,128,activation=tf.nn.relu)
	x = tf.layers.dense(x,128,activation=tf.nn.relu)
	x = tf.layers.dense(x,128,activation=tf.nn.relu)
	x = tf.layers.dense(x,1)
	return x

pred = network(inp)
dfdx,dfdy = tf.gradients(pred,[x,y])
</code></pre>
<p>But how to I get similar results with above code using pytorch?</p>
<pre><code class="lang-auto">class net(torch.nn.Module):
  def __init__(self):
    super(net,self).__init__()
    self.linear={}
    self.wei={}
    self.bias={}
    self.para=[]
    self.linear[0] = torch.nn.Linear(3,128)
    self.linear[1] = torch.nn.Linear(128,128)
    self.linear[2] = torch.nn.Linear(128,128)
    self.linear[3] = torch.nn.Linear(128,128)
    self.linear[4] = torch.nn.Linear(128,1)
  def forward(self,x):
    for i in range(4):
      x=F.elu(self.linear[i](x))
    x=self.linear[4](x)
    return x

M=net()

for i in range(5):
  M.para+=list(M.linear[i].parameters())


L=nn.MSELoss()
opt=torch.optim.Adam(M.para,0.00035)

######################################

X=torch.Tensor(data[:,0:3])   ### 3 dim input
X.requires_grad_(True)
Y=torch.Tensor(data[:,3])
Y=Y.reshape(-1,1)

for epoch in range(10000):
    opt.zero_grad()
    idx=torch.randint(0,X.shape[0],(24,))
    M_dev=torch.autograd.grad(M.forward(X),X,torch.ones_like(M.forward(X)))   ????

</code></pre>
<p>I tried ‘torch.autograd.grad(M.forward(X),X,torch.ones_like(M.forward(X)))’<br>
but it gives wrong answer…</p>
<p>How I can get like ‘tf.gradient( M.forward(X), X)’  ???</p>
How to get unique sequences

danliu2: <p>Hi, I want to get unique rows from a matrix, which means from</p>
<blockquote>
<p>torch.LongTensor([[2,2,3],[1,2,3],[1,2,3]])<br>
I want to get [[2,2,3],[1,2,3]] and their position [0,1]. and remove all redundancy rows</p>
</blockquote>

smth: <p>we dont have any in-built function to do this.</p>

RevoGen: <p>You can use <a href="https://pytorch.org/docs/master/generated/torch.unique.html" rel="nofollow noopener">torch.unique</a></p>
PIL.UnidentifiedImageError: cannot identify image file

venuv: <p>I’m getting an unidentified image error in running a dataloader on a PNG file (which should be covered by PIL image readers I thought. Further the reader seems to append a leading "._’ to the file name (error and actual file name below). Any suggestions appreciated.</p>
<p>The error:<br>
PIL.UnidentifiedImageError: cannot identify image file &lt;_io.BufferedReader name=’…/DiaperImagesMoire/parameter_screen_20200324/gray_img/._J153_brickGrid_5.15_9.5_4.76_8.5_ellipse_2.24_2.0_1.2_0.6_300ppi_gray.png’&gt;</p>
<p>Actual file name (no leading characters before J153):<br>
J153_brickGrid_5.15_9.5_4.76_8.5_ellipse_2.24_2.0_1.2_0.6_300ppi_gray.png**</p>
<p>The code:<br>
dataset = datasets.ImageFolder(root=’…/DiaperImagesMoire/parameter_screen_20200324/’, transform=transforms.Compose([<br>
transforms.Grayscale(),<br>
transforms.Resize(64),<br>
transforms.ToTensor(),<br>
transforms.Normalize(mean=[0.5], std=[0.5])<br>
]))</p>
<p>train_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, shuffle=True, num_workers=2)<br>
train_iter = iter(train_loader)<br>
images, labels = train_iter.next()</p>

yash1994: <p>Hi,</p>
<p>There seems to be a problem with the <code>ImageFolder</code> root path, there are three dots (<code>…/DiaperImagesMoire/parameter_screen_20200324/</code>) in the path instead of two. I would also suggest using an absolute path for the <code>ImageFolder</code> root.</p>

Easquel_Huang: <p>Actually I got the same issue. Check if there is a hidden file “._J153_brickGrid_5.15_9.5_4.76_8.5_ellipse_2.24_2.0_1.2_0.6_300ppi_gray.png” under your directory.</p>
Is the output reparameterize in Variational Autoencoder must be tensor?

Baileyswu: <p>As written in <a href="https://github.com/pytorch/examples/tree/master/vae" rel="nofollow noopener">pytorch-example</a>, the reparameterize function will output sample in hidden space. Finally, it will be a part of loss_funtion to be updated. The final loss will be 105. I have two questions below:</p>
<ol>
<li>the reparameterize function doesn’t contain any tensor parameters to be updated. But when I change the output of it, namely z, into z.detach() before input into decode, the loss will be 200 and nearly few descent.So why?</li>
<li>The loss is consists of two parts: BCE(the reconstruction loss between x and new x) and KLD (the divergence between posterior and prior). So the BCE will update the params of decoder, while the KLD will update the params of encoder. In my option, z.detach() will not infleunce the update of decoder params. I don’t know if it is true.</li>
</ol>
<p>Dear pytorchians, I hope you can write down your ideas~ thanks~</p>
Finding the mean per class in Pytorch

Debasmit_Das: <p>I am naively iterating over each and every sample of the dataset. Is there anyway to calculate the mean efficiently ?</p>
<pre><code class="lang-auto">my_root = '/mini_imagenet_full_size/train/'
Clmean = []
dir_list = os.listdir(my_root)
print(len(dir_list))
miniImagenet_dataset = datasets.ImageFolder(root=my_root, transform=data_transform)
Clmean=torch.zeros([64,3,224,224])
for t,c in miniImagenet_dataset:
   print(c)
   Clmean[c,:,:,:]+=t
   
print(Clmean)
</code></pre>
PyTorch Build from Source in the ARM Cortex A9

xifanlover: <p>Operating on a virtual machine，My Steps：<br>
1、git clone --recursive <a href="https://github.com/pytorch/pytorch" rel="nofollow noopener">https://github.com/pytorch/pytorch</a><br>
2、cd pytorch<br>
3、git checkout v1.0.0<br>
4、export NO_CUDA=1<br>
export NO_TEST=1<br>
export NO_DISTRIBUTED=1<br>
5、<br>
export CC="/opt/poky/1.7/sysroots/x86_64-pokysdk-linux/usr/bin/arm-poky-linux-gnueabi/arm-poky-linux-gnueabi-gcc"<br>
export CXX="/opt/poky/1.7/sysroots/x86_64-pokysdk-linux/usr/bin/arm-poky-linux-gnueabi/arm-poky-linux-gnueabi-g++"<br>
6、export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/…/"}<br>
7、python setup.py install<br>
However，when install ，error：arm-poky-linux-gnueabi-gcc can not found。<br>
Anyone can tell me：can PyTorch be build from Source in the ARM Cortex A9？</p>

xifanlover: <p>I use sudo python setup.py install,It report:<br>
c++: internal compiler error: Killed (program cc1plus)<br>
Please submit a full bug report,<br>
with preprocessed source if appropriate.<br>
See &lt;file:///usr/share/doc/gcc-7/README.Bugs&gt; for instructions.<br>
caffe2/CMakeFiles/caffe2.dir/build.make:1083: recipe for target 'caffe2/CMakeFiles/caffe2.dir/<strong>/aten/src/ATen/core/register_symbols.cpp.o’ failed<br>
make[2]: *** [caffe2/CMakeFiles/caffe2.dir/</strong>/aten/src/ATen/core/register_symbols.cpp.o] Error 4<br>
CMakeFiles/Makefile2:3724: recipe for target ‘caffe2/CMakeFiles/caffe2.dir/all’ failed<br>
make[1]: *** [caffe2/CMakeFiles/caffe2.dir/all] Error 2<br>
Makefile:140: recipe for target ‘all’ failed<br>
make: *** [all] Error 2<br>
Failed to run ‘bash …/tools/build_pytorch_libs.sh --use-nnpack --use-mkldnn --use-qnnpack caffe2’</p>

xifanlover: <p>I create 4G  swap，complie can going on。wait for my new message。</p>
How to add a derivable patch into an image？

0809zheng: <p>I want to put a derivable patch (e.g. 60×60) into a size 500×500 image, then put it into a 608×608 input network (YOLOv4).<br>
I want to get patch.grad, how can I get it?<br>
I realize using transforms to do resize is seemly failed.</p>

ptrblck: <p>You could create the initial <code>patch</code> with <code>requires_grad=True</code> and copy it into the bigger tensors using your desired indices (or apply <code>F.interpolate</code> etc. on it):</p>
<pre><code class="lang-python">patch = torch.randn(1, 1, 60, 60, requires_grad=True)

x = torch.zeros(1, 1, 500, 500)
x[:, :, :60, :60] = patch
y = torch.zeros(1, 1, 608, 608)
y[:, :, :500, :500] = x

conv = nn.Conv2d(1, 1, 3, 1, 1)
out = conv(y)
out.mean().backward()

print(patch.grad)
</code></pre>
Convert a pytorch generator to tensorflow

Giovanni_Cena: <p>Hi guys, i’m trying for a group project to convert this code into tensorflow but it didn’t go as planned, and while the net works good on pytorch it takes a very long time to train on tf, and so we thought some of you could help us. This is the model i want to convert:</p>
<pre><code class="lang-auto">class generator_model(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()
        self.main = nn.Sequential( 
            nn.ConvTranspose2d(100, 160 * 16, 4, bias=False),
            nn.BatchNorm2d(160* 16),
            nn.ReLU(True),
            nn.ConvTranspose2d(160* 16, 160* 8, 4, 2, 1, bias=False),
            nn.BatchNorm2d(160* 8),
            nn.ReLU(True),
            nn.ConvTranspose2d(160* 8, 160* 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(160* 4),
            nn.ReLU(True),
            nn.ConvTranspose2d(160* 4, 160* 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(160* 2),
            nn.ReLU(True),
            nn.ConvTranspose2d(160* 2, 160, 4, 2, 1, bias=False),
            nn.BatchNorm2d(160),
            nn.ReLU(True),
            nn.ConvTranspose2d(160, 3, 4, 2, 1, bias=False),
            nn.Tanh()
        )

    def forward(self, input):
        return self.main(input)
</code></pre>
<p>And we can’t get why the first layer of pytorch conv2transpose usually is changed by a dense layer in tf on similar implementations.</p>
<pre><code class="lang-auto">def make_generator_model():
    model = tf.keras.Sequential()
    model.add(layers.Dense(160*16, use_bias=False, input_shape=(100,)))
    model.add(layers.Reshape((4, 4, 160)))
    model.add(layers.Conv2DTranspose(160*16, (4, 4), strides=(2, 2), padding='same', use_bias=False))
    model.add(layers.BatchNormalization())
    model.add(layers.ReLU())
    model.add(layers.Conv2DTranspose(160*8, (4, 4), strides=(2, 2), padding='same', use_bias=False))
    model.add(layers.BatchNormalization())
    model.add(layers.ReLU())
    model.add(layers.Conv2DTranspose(160*4, (4, 4), strides=(2, 2), padding='same', use_bias=False))
    model.add(layers.BatchNormalization())
    model.add(layers.ReLU())
    model.add(layers.Conv2DTranspose(160*2, (4, 4), strides=(2, 2), padding='same', use_bias=False))
    model.add(layers.BatchNormalization())
    model.add(layers.ReLU())
    model.add(layers.Conv2DTranspose(3, (4, 4), strides=(2, 2), padding='same', use_bias=False, activation='tanh'))
    return model
</code></pre>
<p>How much are different those two implementations? How can be used the ConvTranspose2d on tf with the same (16,100) input?</p>

ptrblck: <p>I think you might get a better and faster answer on StackOverflow for TensorFlow-specific questions, as there might be more TF users than here. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Any similar function?

blessy_hebzeba: <p>Is there any function in Pytorch similar to history in Tensorflow?</p>

ptrblck: <p>What does this method do?<br>
I could only find a Keras History function, which seems to be some kind of callback.</p>

bfeeny: <p>Yes I am pretty sure thats what they meant.  It records loss, lr, score, etc.  So you have everything you need at the end of a fold for example to make a nice graph.  Pytorch doesn’t have anything like Keras history does it?</p>

ptrblck: <p>PyTorch doesn’t ship with these utilities (the <code>ImageNet</code> example uses some metric objects, but these are probably not what you are looking for).<br>
However, higher-level wrappers such as <a href="https://pytorch.org/ignite/">Ignite</a> or <a href="https://github.com/PyTorchLightning/pytorch-lightning">Lightning</a> provide such callbacks.</p>
Mixed training precision leads to nan of loss

ML0401: <p>Hello,</p>
<p>to save memory I am trying to train a resnet18 model in half precision. Therefore, I converted my inputs and my model. I also changed the BachNorm2d layers back to normal floating tensors with the following lines:</p>
<pre><code class="lang-auto">resnet18.half()
for layer in resnet18.modules():
    if isinstance(layer, nn.BatchNorm2d):
        layer.float()
</code></pre>
<p>Nevertheless, I receive nan values as loss after the second batch is trained. Since, everything works find without half precision and the chance of instabilities during applying the BatchNorm2d, I was wondering if I am missing something or if there is another way?</p>

ptrblck: <p>Calling <code>model.half()</code> might work, but can easily create overflows and thus NaNs.<br>
We recommend to use the <a href="https://pytorch.org/docs/master/amp.html">automatic mixed-precision training</a>, which takes care of these issues and also stabilizes the training using loss scaling.<br>
You can use it in the nightly binaries or by building PyTorch from source.</p>
What to do for gradients when rsample is not implemented in a distribution?

amitfishy: <p>Hello,</p>
<p>I’m currently looking at the <a href="https://pytorch.org/docs/stable/distributions.html#mixturesamefamily" rel="nofollow noopener">MixtureSameFamily</a> distribution class. I’m trying to have a Gaussian Mixture Model that I can sample (using <em>rsample</em>), use the samples to calculate a loss, and then do a backward pass based on that loss. <em>rsample</em>, however is currently not implemented for MixtureSameFamily.</p>
<p>I have no trouble getting it to work for the case where the distribution is just <a href="https://pytorch.org/docs/stable/distributions.html#multivariatenormal" rel="nofollow noopener">MultiVariateNormal</a> since this does have a <em>rsample</em> method.</p>
<p>I understand that for some reason it is problematic to implement for the mixture class (I also saw the same problem in tensorflow’s mixture class). That being said, is there any other way within torch or outside it that I can collect samples from my GMM and have a loss defined by them which I can make a backward call over?</p>
<p>Any suggestions would be highly appreciated! Thank You!</p>
Index a tensor using tensor does not work in Libtorch

Carol_Ye_Liu: <p>In python, I can easily index a tensor using another tensor like the example below.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/9/19be3d04759efc713b82669b4026022fe53b16f1.png" alt="image" data-base62-sha1="3FJwgYppk6ATEEZXlL7ddQoyt7r" width="563" height="150"></p>
<p>But in C++, when I executed the following code, I got an error.<br>
auto category = torch::tensor({1,2,7,9});<br>
auto sample =  torch::tensor({1,0,2,2});<br>
std::cout&lt;&lt;category[sample]&lt;&lt;std::endl;<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/1/a10f628f6d59cc2292eecbf3f27c69054f174e70.png" alt="image" data-base62-sha1="mYNPLz9dL3nUKjzYRIPyeM3zYVq" width="661" height="51"></p>
<p>What would be the right way to do this? Thanks!</p>

albanD: <p>Hi,</p>
<p>Yes we couldn’t overload the <code>[]</code> operator to do  be the same as in python. You can check the <a href="https://pytorch.org/cppdocs/notes/tensor_indexing.html?highlight=index">doc</a> for the corresponding c++ API. I think <code>category.index({sample});</code> in this case.</p>
Loss does not decrease after modifying prediction with torch.floor()

Mainul: <p>Hello,<br>
For the following code, the loss decreases.</p>
<pre><code class="lang-auto">loss_function=nn.MSELoss()
loss=loss_function(pred,label)
</code></pre>
<p>But, the loss remains completely unchanged if I change the <code>pred</code> by <code>floor</code> function. I checked the parameters after <code>opt.step()</code>, they are not changing.</p>
<pre><code class="lang-auto">loss_function=nn.MSELoss()
loss=loss_function(torch.floor(pred),label)
</code></pre>
<p>Why this might happen? Does it break computation graph?</p>

mfernezir: <p>If you plot the floor function, you will see that the derivative is either zero or undefined. That’s a problem to optimize with gradient descent. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<p>Try replacing it with <code>torch.floor(pred).detach() + pred - pred.detach()</code>.</p>