akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Build errors when compiling PyTorch on Windows 10 with CUDA support

arpan96: <p>I have been trying to compile PyTorch from source on Windows 10. I am following the steps as suggested on <a href="https://github.com/pytorch/pytorch#from-source" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration</a>.</p>
<p>I have followed these steps to compile it with Ninja after having problems trying to compile it with the default steps mentioned.</p>
<p>These are the steps I have followed after cloning pytorch.</p>
<p>1.set CMAKE_GENERATOR=Ninja<br>
2.set USE_NINJA=ON</p>
<p>3.set CMAKE_GENERATOR_TOOLSET_VERSION=14.11<br>
4.set DISTUTILS_USE_SDK=1</p>
<p>5.for /f “usebackq tokens=*” %i in (<code>"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -version [15^,16^) -products * -latest -property installationPath</code>) do call “D:\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvarsall.bat” x64 -vcvars_ver=%CMAKE_GENERATOR_TOOLSET_VERSION%</p>
<p>6.set CUDAHOSTCXX=D:\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.11.25503\bin\HostX64\x64\cl.exe</p>
<p>7.python setup.py install</p>
<p>and after taking hours to compile I received this traceback from the anaconda console.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/ac89d64ec223a32b671921d1381e5fcad880f4fd.png" data-download-href="https://discuss.pytorch.org/uploads/default/ac89d64ec223a32b671921d1381e5fcad880f4fd" title="error"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/a/ac89d64ec223a32b671921d1381e5fcad880f4fd_2_690x371.png" alt="error" data-base62-sha1="oClrfVdf9korQZleflg4fOgKkDH" width="690" height="371" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/a/ac89d64ec223a32b671921d1381e5fcad880f4fd_2_690x371.png, https://discuss.pytorch.org/uploads/default/optimized/2X/a/ac89d64ec223a32b671921d1381e5fcad880f4fd_2_1035x556.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/a/ac89d64ec223a32b671921d1381e5fcad880f4fd_2_1380x742.png 2x" data-dominant-color="1B1B1B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">error</span><span class="informations">3840×2065 353 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have uploaded the Image from the traceback and I would really get some help on how to solve this as I am trying to go through the fastai course using my own gpu but it says that old gpus are not supported. I have a GTX 960M GPU and was asked to compile it from source to make it work.</p>
<p>Thanks.</p>

arpan96: <p>So I tried fixing the problem and found out that I had not downloaded and install cudnn.<br>
I installed it and reran python setup.py install and it worked.</p>
Training multiple objects

IAmIronMan: <p>I have kitti dataset for training my Yolo model. The training dataset of kiiti dataset contains images with multiple objects. How to train that ?<br>
Intuitively, I thought to use single object images like to train the model for car and pedestrian one by one with different images, but in Kitti dataset, all the objects are in single images</p>
<p>Anyone who has worked with Kitti dataset ?</p>

ptrblck: <p>Have a look at the <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="nofollow noopener">Detection tutorial</a>, where the target is explained for this kind of task.<br>
In the <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#defining-the-dataset" rel="nofollow noopener">Defining a Dataset</a> section you can see that multiple boxes are provided for each image. While this tutorial uses another dataset, you might need to adapt this work flow to your dataset.</p>

IAmIronMan: <p>Thanks. I got it. I was little confused over the terminologies of anchor boxes and bounding boxes given in online blogs. So, I was not sure to train multiple objects in an image.</p>
Pytorch get stuck at .to('cuda') or .cuda()

DeliciousGarfield: <p>I use a docker environment to do some pytorch training work, but frequently get stuck at .to(‘cuda’) or .cuda() calls.  Not know why.<br>
Some system behaviors I notice: The training process has a cpu usage of 100%, and almost 97% are sys time as shown by top 1. I used strace to debug the training process, and also used the /proc file system, found the process keep getting stuck on a poll syscall to a pipe, but I don’t know where the pipe is used in pytorch.</p>
Hook function that demands epoch

FruitVinegar: <p>Hi. I have a problem with Pytorch Hook function,<br>
as I mentioned in title, my Hook demands current epoch for recording some statistics that changed by every epoch.<br>
Is there any some tricky method for solving this problem?</p>
<p>For example,</p>
<pre><code class="lang-auto">my_model = some_net()
my_model.specific_layer.register_forward_hook(tricky_hook_for_recording_epoch)

for epoch in range(epoch_size):
    for i, (data, label) in enumerate(dataloader):
        output = my_model(data)
</code></pre>
<p>I’m using TensorBoard for Pytorch and this is why I need to record epoch.<br>
Thanks for any answer!</p>

spanev: <p>Hi <a class="mention" href="/u/fruitvinegar">@FruitVinegar</a></p>
<p><a href="https://github.com/pytorch/ignite/" rel="nofollow noopener">pytorch/ignite</a> might be help you to achieve that easily.<br>
Please look <a href="https://pytorch.org/ignite/concepts.html#events-and-handlers" rel="nofollow noopener">here</a> for a description of how it works.</p>

spanev: <p>But if you don’t want to bother with <code>ignite</code> I guess you could also do this:</p>
<pre><code class="lang-auto">global epoch, prev_epoch

def tricky_hook_for_recording_epoch(module, input, output):
    if epoch &gt; prev_epoch:
        # TensorBoard stats, logging, etc
        prev_epoch = epoch

my_model = some_net()
my_model.specific_layer.register_forward_hook(tricky_hook_for_recording_epoch)
prev_epoch = 0
for epoch in range(epoch_size):
    for i, (data, label) in enumerate(dataloader):
        output = my_model(data)
</code></pre>

FruitVinegar: <p>I was refactoring my spaghetti code with many parameters and libraries, so that Ignite looks helpful for me. Thank you for introducing that!</p>
Run time error with backward() after upgrading to pytorch 1.2.0

kikiloverafa: <p>The following error occurs after I upgrade to Pytorch 1.2.0. No errors when running in 1.1.0</p>
<blockquote>
<p>one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [3]], which is output 0 of AddBackward0, is at version 1; expected version 0 instead.</p>
</blockquote>
<p>Codes:</p>
<blockquote>
<p>import torch</p>
<p>b0 = torch.tensor(5.0, requires_grad = True)<br>
a = torch.Tensor([1,0,0])<br>
b = torch.zeros(3, requires_grad = True)<br>
c = torch.ones(3, requires_grad = True)</p>
<p>bb = b+1</p>
<p>bb[a!=0] = torch.min(bb[a!=0], c[a!=0])</p>
<p>d = torch.sum(bb)<br>
d.backward()</p>
</blockquote>
<p>Is there a way to make it work in 1.2.0? I tried adding clone() at a couple of locations, but still didn’t work.</p>

kikiloverafa: <p>It’s being followed up here.<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/24853" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <a href="https://github.com/kikiloverafa" rel="nofollow noopener">
<img src="https://avatars1.githubusercontent.com/u/9371073?v=2&amp;s=96" class="thumbnail onebox-avatar" width="60" height="60">
</a>

<h4><a href="https://github.com/pytorch/pytorch/issues/24853" target="_blank" rel="nofollow noopener">Issue: Run time error with backward()  after upgrading to pytorch 1.2.0</a></h4>

<div class="date" style="margin-top:10px;">
	<div class="user" style="margin-top:10px;">
	opened by <a href="https://github.com/kikiloverafa" target="_blank" rel="nofollow noopener">kikiloverafa</a>
	on <a href="https://github.com/pytorch/pytorch/issues/24853" target="_blank" rel="nofollow noopener">2019-08-19</a>
	</div>
	<div class="user">
	</div>
</div>

<pre class="content" style="white-space: pre-wrap;">🐛 Bug
The following error occurs after I upgrade to Pytorch 1.2.0.
one of the variables needed for gradient computation has been modified...</pre>

<div class="labels">
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">high priority</span>
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: autograd</span>
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">needs discussion</span>
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
Recommended pytorch way for cross validation with dataloaders and subsets

kirk86: <p>What’s the recommended way in pytorch to preform cross validation. So far I’ve seen two ways, one is to use the <code>RandomSubsetSampler</code> to make indices and pass those along to the <code>DataLoader</code> and the other one is to create your own method which makes indices for each fold, make your custom <code>DataSet</code> which returns train/valid loader and pass that to <code>DataLoader</code>.</p>
<p>I guess both are valid approaches but what I’m asking is what’s the native pytorch recommended way to avoid errors and be consistent with best practices. (side note: I’ve read pretty much all posts I could find relative to the subject but couldn’t find a clarification on this topic)</p>

kirk86: <p>I’ll try to answer my own question in case someone else finds it useful. One way to achieve this (althought I’m not 100% if this is the correct pytorch way) is to use <code>Subset</code> class to retrieve the corresponding data and targets according to a set of indices.</p>
<p>For instance:</p>
<pre><code class="lang-auto">for tr_idx, te_idx in kfold(...):
     train_subset = torch.utils.data.Subset(train_dataset, tr_idx)
     train_loader = torch.utils.data.DataLoader(train_subset, batch_size=64)
     for x, y in train_loader:
          model.train()
          y_hat = model(x)
</code></pre>
Where to use the requires_grad=True?

Jose615243: <p>In a simple linear network:<br>
4(input)—&gt;100<br>
100—&gt;3(output)<br>
With a tensor X for the input and Y for output. Where should I use requires_grad=True in X,Y or the nn.Linear creates weights tensors that have the requires_grad=True automatically set?</p>

ptrblck: <p>Usually you don’t need to set this attribute manually.<br>
The parameters inside the layers will be created as <code>nn.Parameter</code>, which will automatically have <code>requires_grad=True</code>.<br>
A use case for setting this attribute manually, would be e.g. if you use some extra parameters in your model (for your fancy operations) or e.g. if you need gradients in the input (adversarial training).</p>
Is it possible to slice a model at an arbitrary layer?

mgpoirot: <p>Hi all,</p>
<p>I’m looking to chop an exciting network up in parts so I could later chain different parts together, eg.</p>
<pre><code class="lang-auto">model = densenet121()
first_half = densenet121[:6]
second half = densenet121[6:]
</code></pre>
<p>However, this has not been so easy, I’ve tried splitting the model using model.parameters, model.modules and model.features, but I’ve not been able retain functionality when tied back together.</p>
<p>What is the proper way to copy or disregard one half of a model?</p>

ptrblck: <p>I would recommend to derive a custom class using the Densenet121 implementation as the parent class. Inside the <code>__init__</code>, you could try to split the <code>self.features</code> sequential module and create your own <code>forward</code> method using these splits.</p>
<p>Your code snippet would work for very simple models, which can be defined in an <code>nn.Sequential</code> module alone.</p>
Uneven splitting in using data parallel

barakb: <p>Hi, I’m trying to to run a big model (resnext-wsl 32x48d) on 8 GPUs with 12GB each, but I’m getting a memory error:“RuntimeError: CUDA out of memory. Tried to allocate 42.00 MiB (GPU 0; 11.75 GiB total capacity; 10.28 GiB already allocated; 2.94 MiB free; 334.31 MiB cached)”</p>
<p>It seems that the data-parallel is splitting the memory in un even way, as can be shown in the memory layout printed a moment before crashing (before optimizer.step):<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/8/8e80a4ed869a1ae7b6a5822e560353c574130850.png" alt="image" data-base62-sha1="kkDq1QSDXINV9T0RY1UEGbhJ61O" width="176" height="184"></p>
<p>As can be seen, GPU0 is overloaded while the rest still have some space.<br>
Is there anyway to solve this problem? is it possible that this model cannot be trained on gpus with “only” 12GB?</p>
<p>Thanks.</p>

ptrblck: <p>The imbalanced memory usage and possible workarounds are described in <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255" rel="nofollow noopener">this blog post</a> by <a class="mention" href="/u/thomas_wolf">@Thomas_Wolf</a>.</p>
Equivalent of  `tf.dynamic_partition`

DimitrisNightmare: <p>I need to use an equivalent of  <code>tf.dynamic_partition</code>  in PyTorch. Is there anything with similar functionality in PyTorch or other library or is there a simple and clever way to code it for PyTorch and work fast? The same for <code>tf.dynamic_stitch</code>. Thanks!</p>

LeviViana: <p>The following implementation works as equivalent of Vector Partitions in the <a href="https://www.tensorflow.org/api_docs/python/tf/dynamic_partition" rel="nofollow noopener">tf documentation</a>.</p>
<pre><code class="lang-python">import torch

def dynamic_partition(data, partitions, num_partitions):
  res = []
  for i in range(num_partitions):
    res += [data[(partitions == i).nonzero().squeeze(1)]]
  return res

data = torch.Tensor([10, 20, 30, 40, 50])
partitions = torch.Tensor([0, 0, 1, 1, 0])
dynamic_partition(data, partitions, 2)
</code></pre>
<p>There are ways of implementing this more efficiently, specially for big CUDA tensors. But maybe this is good enough for your application.</p>
<p>Regarding the <code>tf.dynamic_stitch</code>, the following snippet works as well (it matches the input-&gt;output from the <a href="https://www.tensorflow.org/api_docs/python/tf/dynamic_stitch" rel="nofollow noopener">tf documentation</a>).</p>
<pre><code class="lang-python">indices = [None] * 3

indices[0] = 6
indices[1] = [4, 1]
indices[2] = [[5, 2], [0, 3]]

data = [None] * 3

data[0] = [61, 62]
data[1] = [[41, 42], [11, 12]]
data[2] = [[[51, 52], [21, 22]], [[1, 2], [31, 32]]]

data = [torch.tensor(d) for d in data]
indices = [torch.tensor(idx) for idx in indices]

def dynamic_stitch(indices, data):
  n = sum(idx.numel() for idx in indices)
  res  = [None] * n
  for i, data_ in enumerate(data):
    idx = indices[i].view(-1)
    d = data_.view(idx.numel(), -1)
    k = 0
    for idx_ in idx: res[idx_] = d[k]; k += 1
  return res

dynamic_stitch(indices, data)
</code></pre>
<p>If these implementations don’t perform well enough for you application, consider the possibility of implementing an extension.</p>

DimitrisNightmare: <p>Great! Thanks for the answer!</p>
Does creating multiple model instances resets the weights as well?

kirk86: <p>I was wondering if instantiating a model inside a loop would reset the params in each instance of the model.<br>
For example if we had something like the following:</p>
<pre><code class="lang-auto">for iter in range(5):
     model = Model()
     optim = torch.optim.Adam(model.parameters())
</code></pre>
<p>Some initial experimentation indicates that the above is true but just wanted to verify this from the devs or community experts.</p>
<p>Thanks!</p>

kirk86: <p>Never mind folks found the answer <a href="https://discuss.pytorch.org/t/cross-validation-model-reset/21176/2">here</a></p>
Remove for loop filling Tensor

Roulbac: <p>Hi</p>
<p>I have the following snippet</p>
<pre><code class="lang-auto">329         for i in range(n):
330             foo[i, :] = functor(x - i)
331             bar[i, :] = functor(y - i)
</code></pre>
<p>Clearly these operations are parallelizable and I’m looking into something like numpy apply_along_axis or any operation that would make my stuff parallel.</p>
<p>Functor uses PyTorch operations.</p>
<p>Any ideas?</p>

Roulbac: <p>Solved using broadcasting where functor is an elementwise op and x,y are 1D vectors</p>
<pre><code class="lang-auto">326         range_tensor = torch.arange(n, out=torch.FloatTensor()).reshape(-1, 1)
327         foo = functor(x.reshape(1, -1) - range_tensor)
328         bar = functor(y.reshape(1, -1) - range_tensor)
</code></pre>
Get appropriate model in/output type programmatically

mgpoirot: <p>Hi all,</p>
<p>Of course, the datatype of the data you’re trying to propagate through your network matters.<br>
Feeding your model input with wrong input type might give you:<br>
<code>RuntimeError: Input type (torch.LongTensor) and weight type (torch.FloatTensor) should be the same</code><br>
And feeding model output of wrong type might yield something along the lines of:<br>
<code>RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'target'</code></p>
<p>To build a more flexible system I was wondering: <strong>How can one find the data type that will fit a model programmatically?</strong></p>
<p>For output I’ve tried <code>list(model.parameters())[-1]</code> which might be a bit naive.</p>

spanev: <p>Might not be the best way, but for the inputs you could do something like:</p>
<pre><code class="lang-auto">current_children = model
while list(current_children.children()) != []: 
   current_children = list(current_children.children())[0]

for p_name, p in current_children.named_parameters():
    print('{}: {}'.format(p_name, p.dtype))     
</code></pre>
How to train mobilenet and mnasnet with my n-pair-angular loss?

Tomas_Batrla: <p>Hello,<br>
I am trying to train torchvisions (0.4.0) mobilenet and mnasnet with n-pair-angular loss function. But I am getting error which is probably related to <strong>inplace=True</strong></p>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: <strong>[torch.cuda.FloatTensor [20, 1280, 7, 7]]</strong>, which is output 0 of <strong>HardtanhBackward1</strong>, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient.</p>
<p>Error at : result = torch._C.<em>nn.hardtanh</em>(input, min_val, max_val)</p>
<p>Training works for other models from torchvision models, but I am stuck with those two models.<br>
Is it possible to make it work? Does changing inplace=True to False change the function of the model itself?</p>
Can pytorch 1.2 support cuda 9.0?

ano: <p>I try to install pytorch 1.2.0 with anaconda, but found that  <code>torch.cuda.is_available()</code>  always returns False. Turns out our server environment can only support CUDA 9.0, but I do not have administrator right to upgrade the nvidia driver.<br>
Give that pytorch 1.2.0 introduced Bool type, and I have already updated my code for corresponding change of API, I do not want to undo this efforts.<br>
Will pytorch 1.2.0 support CUDA 9.0 backend in the future?</p>

ptrblck: <p>You could build PyTorch from source as described <a href="https://github.com/pytorch/pytorch#from-source" rel="nofollow noopener">here</a> and use your local CUDA installation to build it.<br>
The binaries currently ship with CUDA9.2 and CUDA10.0.</p>
Alias specs in signatures in PyTorch 1.2

leowalkling: <p>With the PyTorch 1.2, I have noticed that any alias specification when registering an operator would cause a crash (tried on Win64).<br>
So I had to change</p>
<pre><code class="lang-auto">"some_op(Tensor(a) t) -&gt; Tensor(a)"
</code></pre>
<p>to</p>
<pre><code class="lang-auto">"some_op(Tensor t) -&gt; Tensor"
</code></pre>
<p>to make it work again.</p>
<p>Is there an alternative?</p>

Sebastian_Messmer: <p>Do you have a minimal code example reproducing the crash? And what is the error message you’re seeing exactly?</p>

leowalkling: <p>The error message is simply:<br>
OSError: [WinError 1114] A dynamic link library (DLL) initialization routine failed<br>
I have created a repo for it:<br>
<a href="https://github.com/leowalkling/pytorch_register_op_minimal" rel="nofollow noopener">https://github.com/leowalkling/pytorch_register_op_minimal</a><br>
I’m using scikit-build to easily generate a solution file but the compilation flags are taken from <code>torch.utils.cpp_extension</code>.</p>

leowalkling: <p>I found out, that in PyTorch 1.2, the default behavior is not to allow alias specifications, leading to a segfault.<br>
This can be changed, but alias specifications are not allowed for third-party extensions as stated in the error message:</p>
<pre><code class="lang-auto">RuntimeError: node-&gt;kind().is_prim() || node-&gt;kind().is_aten() INTERNAL ASSERT FAILED at ..\torch\csrc\jit\passes\alias_analysis.cpp:392, please report a bug to PyTorch. The current code base should only have AliasAnalysisKind::FROM_SCHEMA for aten:: and prim:: ops but we found it for tp::broadcast_tensors. We want to open this up though. (analyzeImpl at ..\torch\csrc\jit\passes\alias_analysis.cpp:392)
(no backtrace available)
</code></pre>
<p>To get this error message, I passed an Options struct as a third (optional) argument to RegisterOperators.op(), i.e.:</p>
<pre><code class="lang-auto">torch::RegisterOperators()
.op(
"some_op(Tensor(a) t) -&gt; Tensor(a)",
&amp;some_op,
torch::RegisterOperators::options().aliasAnalysis(c10::AliasAnalysisKind::FROM_SCHEMA))
</code></pre>
How I can convert row of vector back to words

Aiman_Mutasem-bellh: <p>Dear friends,</p>
<p>I use BucketIterator to generate vector  as below:</p>
<blockquote>
<p>train_iterator, valid_iterator, test_iterator = BucketIterator.splits(<br>
(train_data, valid_data, test_data), sort_key=lambda x: len(x.src),<br>
batch_size = BATCH_SIZE,<br>
device = device)</p>
</blockquote>
<p>successfully, I get ( train_iterator, valid_iterator, test_iterator), and I can access each vector as :</p>
<blockquote>
<p>Get_vector_row = next(iter(train_iterator))</p>
</blockquote>
<blockquote>
<p>vars(Get_vector_row).keys()</p>
</blockquote>
<p>dict_keys([‘batch_size’, ‘dataset’, ‘fields’, ‘input_fields’, ‘target_fields’, ‘src’, ‘trg’])</p>
<blockquote>
<p>Get_vector_row.src[0:]</p>
</blockquote>
<blockquote>
<p>tensor([[   2,  293, 1674,  …,    1,    1,    1],<br>
[   2,  720, 2373,  …,    1,    1,    1],<br>
[   2,  114, 6271,  …,    1,    1,    1],<br>
…,<br>
[   2,  135, 5987,  …, 2345,    4,    3],<br>
[   2,   26, 1306,  …,    1,    1,    1],<br>
[   2, 1186, 1780,  …,    1,    1,    1]], device=‘cuda:0’)</p>
</blockquote>
<p>My question is how to convert the tensor in the previous line back to words?</p>
Obtaining the MLP model from its adjacency matrix

Arvind_Subramaniam: <p>I have treated my Multi-Layer Perceptron (MLP) as a graph and obtained its adjacency matrix (adj_matrix). I have modified the adjacency matrix (adj_modified) and would like to get back the MLP model with the modified weights (from adj_modified). Is there any way to treat the adj_modified matrix as a ‘.pt’ file, so that I can directly load the new weights onto the model?</p>
<p>I am stuck and dont know how to load the new weights. Any other methods to do this would be welcome. Please let me know if the question is unclear in any way</p>
Easy question about mean() and dim

Yolkandwhite: <p>I have no idea how torch.mean(dim=0) is working.</p>
<p>for example when i run the code below</p>
<p><span class="hashtag">#------------------------------------------------</span><br>
import torch<br>
t = torch.FloatTensor([[1, 2], [3, 4]])<br>
print(t.mean(dim=0)<br>
<span class="hashtag">#------------------------------------------------</span></p>
<p>The output is</p>
<p>tensor([2., 3.])</p>
<p>I don’t know why the output comes like this.<br>
Anyone with the insight please tell me how it is working like this.</p>

spanev: <p>It means: give me the mean of the elements iterated of the axis <code>dim</code>.<br>
In your example what it does is:</p>
<pre><code class="lang-auto">(t[0] + t[1]) / num_elements = ([1, 2]+ [3, 4]) / 2 = [2., 3.]
</code></pre>
<p>Similarly if you specified <code>dim1</code>, it would do:</p>
<pre><code class="lang-auto">(t[:,0] + t[:,1]) / 2 = tensor([ 7., 13.])
</code></pre>
<p>same result as</p>
<pre><code class="lang-auto">&gt;&gt;&gt; t.mean(dim=1)
tensor([ 7., 13.])
</code></pre>

Mazhar_Shaikh: <p>The dim parameter defines over which dimension of the tensor you are taking a mean. In your case, dim=0</p>
<pre><code class="lang-auto">t = [[1,2],
     [3,4]]
t.mean(dim=0) = [(1+3)/2, (2+4)/2] = [2,3]
</code></pre>
<p>For dim=1,</p>
<pre><code class="lang-auto">t = [[1,2],
     [3,4]]
t.mean(dim=1) = [(1+2)/2, (3+4)/2] = [1.5,3.5]
</code></pre>
<p>Hope this helps!<br>
Regards</p>

Yolkandwhite: <p>Helped me alot<br>
thank you!</p>
What does the ProcessGroupNCCL.cpp syncStreams function do?

qiuyang: <p>I do not understand what does the syncStreams function do in pytorch.Can the function synchronize different nodes cuda streams？<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/d2eb7e7d925ec44c2a693974e99ec2445f5f086b.png" data-download-href="https://discuss.pytorch.org/uploads/default/d2eb7e7d925ec44c2a693974e99ec2445f5f086b" title="23"><img src="https://discuss.pytorch.org/uploads/default/original/2X/d/d2eb7e7d925ec44c2a693974e99ec2445f5f086b.png" alt="23" data-base62-sha1="u5SOnzAhhYMD5I87MvueUf6nMEP" width="690" height="192" data-dominant-color="373933"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">23</span><span class="informations">836×233 32.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Load training data from one single huge file

flyaway: <p>Hi, all,<br>
I have problems regarding reading large training data from disk.<br>
The situation is I have one single huge file (roughly 100G), each line is a training example.<br>
Clearly I can not load them all into memory.<br>
When I searched for the solution.<br>
One possible solution is to split this huge file into small files and use DataSet and DataLoader like mentioned in here: <a href="https://discuss.pytorch.org/t/loading-huge-data-functionality/346">Loading huge data functionality</a></p>
<p>However, in this way, the total number(the return value of <code>__len__</code> in <code>DataSet</code>) of training example becomes the number of small files not the true number of my training examples. DataLoader sees each small file as one example which is weird.<br>
Can any one provide an elegant solution for this situation? Perhaps producer/consumer style?</p>

jetcai1900: <p>Are there any solutions for this? Thanks.</p>

marchss: <p>Same issue here. Any ideas?</p>

flyaway: <p><a class="mention" href="/u/marchss">@marchss</a> <a class="mention" href="/u/jetcai1900">@jetcai1900</a> Not yet. Finally, I implemented the producer/consumer loader by myself.</p>

marchss: <p>Are you willing to share a short example with us? Thanks</p>
Confused about how a completed SGD is implemented in PyTorch?

feiyuhuahuo: <p>I am dazzled about all the pieces of tutorials from any kind of websites. How SGD was implemented in PyTorch?<br>
lr: learning rate<br>
w: weights<br>
dw: the grad of weights</p>
<p>Is this equation <code>w'= momentum * w - lr * (dw + weight_decay * w)</code> right?<br>
Or this <code>v=momentum * v(t-1) + (dw + weight_decay * w)</code>, then <code>w = w - lr * v</code>, here <code>v(t-1)</code> means the last time <code>v</code>.</p>
Libtorch compilation problem visual studio 2015

zaynabhabibi: <p>Hi,<br>
I use cmake to generate a visual studio solution, the configuration an generation are done. But when I compile the example in the tutorial : “<a href="https://pytorch.org/tutorials/advanced/cpp_export.html" class="inline-onebox" rel="noopener nofollow ugc">Loading a TorchScript Model in C++ — PyTorch Tutorials 2.2.0+cu121 documentation</a>” I have many errors as in the attached image.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/367247b5fd21df44edce656808f416513d3935f9.png" data-download-href="https://discuss.pytorch.org/uploads/default/367247b5fd21df44edce656808f416513d3935f9" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/3/367247b5fd21df44edce656808f416513d3935f9.png" alt="image" data-base62-sha1="7LECTQZBIpCak448Pr1b7f0WdC1" width="690" height="464" data-dominant-color="2C2C2D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1222×822 31.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Did anybody face the same problem?<br>
Windows 7<br>
visual studio 2015</p>
<p>Thank you</p>

yf225: <p>Do you mind translating the error messages for us?</p>
C++ result differs from Python results

xiaofanwei: <p>I trained Dense Net in Pytorch, set model.eval() and moved the model to C++ following the tutorial. But in C++ when i do the inference, the result is different from the result in Pytorch(In Python), and I already do the same normalization.</p>

Oleg_Mikulchenko: <p>It is scary. For me, having the assumption of using python for the training/studying and C++ for the inference with same results and easy model sharing was one of the main points to choose PyTorch over other frameworks.</p>

yf225: <p><a class="mention" href="/u/xiaofanwei">@xiaofanwei</a> Could you share with us the steps you take to reproduce this problem? I can help debug this.</p>
Mildnet for pytorch?

Tomas_Batrla: <p>Hello, I am new in Pytorch and would need a little bit of help.<br>
I am trying to use Mildnet  architecture for image retrieval task. I have the network defined in<br>
Tensorflow estimator api and would like to use it as a pytoch model.<br>
Paper is at : <a href="https://arxiv.org/abs/1903.00905" rel="nofollow noopener">https://arxiv.org/abs/1903.00905</a><br>
Code for Keras is at : <a href="https://github.com/gofynd/mildnet" rel="nofollow noopener">https://github.com/gofynd/mildnet</a></p>
<p>My code is for Tensorflow Keras API</p>
<pre><code>module = hub.Module(https://tfhub.dev/google/imagenet/mobilenet_v1_100_224/feature_vector/1, trainable=True)
out = module(images, signature="image_feature_vector", as_dict=True)


out1 = out['MobilenetV1/Conv2d_1_depthwise']
out2 = out['MobilenetV1/Conv2d_2_depthwise']
out4 = out['MobilenetV1/Conv2d_4_depthwise']
out6 = out['MobilenetV1/Conv2d_6_depthwise']
out12 = out['MobilenetV1/Conv2d_12_depthwise']
out13 = out['MobilenetV1/Conv2d_13_depthwise']

intermediate_layer_outputs = [out1, out2, out4, out6, out12]
convnet_output = layers.GlobalAveragePooling2D()(out13)
for output in intermediate_layer_outputs:
  output = layers.GlobalAveragePooling2D()(output)
  convnet_output = layers.concatenate([convnet_output, output])

convnet_output = layers.Dense(1024, activation='relu')(convnet_output)
convnet_output = layers.Dropout(0.5)(convnet_output)
convnet_output = layers.Dense(128, activation='relu')(convnet_output)
#embeddings = layers.Lambda(lambda  x: K.l2_normalize(x,axis=1))(convnet_output)
if params.l2_normalize:
    convnet_output = tf.nn.l2_normalize(convnet_output, axis=1)

return convnet_output
</code></pre>
<p>Would you help me to make it work? Thanks a lot</p>
How does transforms.ToTensor() work and computation of mean and std values

varghese_alex: <p>Hi</p>
<ol>
<li>
<p>I am currently using the transforms.ToTensor(). As per the document it converts data in the range 0-255 to 0-1.<br>
However, the transform work on data whose values ranges between negative to positive values? Any ideas how this transform work. And the transformed values no longer strictly positive.</p>
</li>
<li>
<p>In most tutorials regarding the finetuning using pretrained models, the data is normalized with [0.485, 0.456, 0.406], [0.229, 0.224, 0.225]). I would like to know how these values are computed? Are they computed by attained by using mean &amp; std  value of each channel from the entire training data?</p>
</li>
</ol>

WERush: <p><code>ToTensor()</code> works for the image, whose elements are in range 0 to 255. You can write your custom Transforms to suit your needs.</p>
<p>[0.485, 0.456, 0.406] is the normalized mean value of ImageNet, and [0.229, 0.224, 0.225] denotes the std of ImageNet.</p>
<p>Yes, it is computed per channels.</p>

varghese_alex: <p>Oki I got it.</p>
<p><a href="http://pytorch.org/docs/0.2.0/_modules/torchvision/transforms.html#ToTensor" rel="nofollow noopener">http://pytorch.org/docs/0.2.0/_modules/torchvision/transforms.html#ToTensor</a> explains. The entire array is converted to torch tensor and then divided by 255. This is how it is forces the network to be between 0 and 1.</p>

LvJC: <p>Actually, I found it is different from <code>*1.0/255</code> in C++. The results between <code>transforms.ToTensor()</code> in Python and <code>*1.0/255</code> in C++ are not same.</p>
<p>I am wondering why…</p>

neonb88: <p>My gut instinct was <code>* 1.0 / 256</code> <em>instead of</em> <code>* 1.0 / 255</code>.  Does this fix the problem, <a class="mention" href="/u/lvjc">@LvJC</a>?</p>
Unable to do autograd way back to the input

harishkool: <p>Hi,<br>
I am trying to perform autograd in the following example, but unable to calculate gradients way back to the theta variable.</p>
<pre><code class="lang-auto">import torch
from torch.autograd import Variable
    theta = Variable(torch.randn(1)*3.1414)
    theta.requires_grad_(True)
    cosval = Variable(torch.cos(theta))
    cosval.requires_grad_(True)
    mat = Variable(torch.Tensor([1,cosval,cosval]))
    mat.requires_grad_(True)
    inpt = torch.Tensor([3,3,3])
    y = torch.dot(mat,inpt)
    y.backward(torch.ones(3))
</code></pre>
<p>when I ran the above code I am able to see the gradients of ‘<strong>mat</strong>’ but not for ‘<strong>cosval</strong>’ and ‘<strong>theta</strong>’. Here gradients for cosval and theta are none.</p>

spanev: <p>Hi <a class="mention" href="/u/harishkool">@harishkool</a>,</p>
<p>Here the graph built by autograd will represent only " y = mat . inpt".</p>
<p>That’s because <code>mat = Variable(torch.Tensor([1,cosval,cosval]))</code> is a leaf node (think that <code>mat</code> creation torch.Tensor([1,cosval,cosval] is not a differentiable operation).</p>
<p>Also <code>Variable</code> <a href="https://pytorch.org/docs/stable/autograd.html#variable-deprecated" rel="nofollow noopener">is deprecated</a> and you should use <code>Tensor</code> instead.<br>
If you replace them by Tensor and use <code>torch.cat</code> to create <code>mat</code>:</p>
<pre><code class="lang-auto">mat = torch.cat((torch.Tensor([1]), cosval, cosval))
</code></pre>
<p>it should do.</p>
<p>You will also need to call <code>retain_grad</code> on <code>Tensor</code>s you want to access the gradient after the backward pass. So you should end up with something like:</p>
<pre><code class="lang-auto">import torch

theta = torch.randn(1, requires_grad=True)*3.1414
theta.retain_grad()

cosval = torch.cos(theta)
cosval.retain_grad()

mat = torch.cat((torch.Tensor([1]), cosval, cosval))
mat.retain_grad()

inpt = torch.Tensor([3,3,3])
y = torch.dot(mat,inpt)
y.backward(torch.ones(3))
</code></pre>

harishkool: <p>Thank you. Your solution solved my problem. But I am just wondering how come torch.Tensor([1,cosval,cosval]) is not differentiable and torch.cat((torch.Tensor([1]),cosval,cosval)) is differentiable?. Is it because torch.Tensor() is a leaf node where as torch.cat() is a torch op and makes it as a part of the graph?</p>

spanev: <p>Np, happy to read that! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
That’s correct, <code>torch.Tensor()</code> is not an op but will be and will be a leaf node in your graph, but <a href="https://pytorch.org/docs/stable/torch.html#torch.cat" rel="nofollow noopener">torch.cat</a> is one.</p>

harishkool: <p>Got it. Thank you Serge.!!</p>
Dataset/Dataloader minibatch class distribution

kammcm: <p>I am trying to find out if there is any way to force the distribution of classes in each batch that is produced when using the pytorch Dataset and Dataloader functionality.  For example, I am doing binary classification and (because my class sizes are imbalanced) during training I would like each batch to be 50% positive examples and 50% negative.  Is there any way to achieve this with Dataloader?<br>
Thanks!</p>

alex.veuthey: <p>I think the proper way to have such a functionnality is to subclass <code>torch.utils.data.Sampler</code> with a custom one that samples equiprobably from positive and negative samples. Then, you pass an instance of this custom sampler to the <code>DataLoader</code> as the <code>sampler</code> argument.</p>
<p>In any case, however, you will need to decide if you want to repeat the smaller class (so that you visit every element of the larger class once per epoch) or to trim the larger class (so that you visit every element of the smaller class once per epoch). In the first case, data augmentation is recommended.</p>

kammcm: <p>Thank you, that’s exactly what i was looking for <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=6" title=":grin:" class="emoji" alt=":grin:"></p>

atriantafy: <p>Personally, I use the <code>WeightedRandomSampler</code> class (<a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.WeightedRandomSampler" rel="nofollow noopener">https://pytorch.org/docs/stable/data.html#torch.utils.data.WeightedRandomSampler</a>) and give each sample a weight of inverse the frequency of each class, thus resulting in an equal representation of the classes in the mini-batch (assuming a big enough batch size I guess)</p>

Griffintaur: <p>I want to clarify one point regarding the WeightedRandomSampler.<br>
While it is oversampling the minority class it is also undersampling the majority class .<br>
Lets say i have 100 images of classA and 900 images of classB<br>
Then dataloader length will be 1000. and when we will iterate in minibatches it will ensure equal distribution thus approx 500 images of class A and 500 images of classB will be used for training.<br>
Can’t we say it is oversampling the minority but undersampling the majority in dataset?</p>

ptrblck: <p>Answered <a href="https://discuss.pytorch.org/t/balanced-sampling-between-classes-with-torchvision-dataloader/2703/30">here</a>.</p>
TD(lambda) backward view

Swakshar_Deb: <p>Hi ,<br>
i am trying to apply backward view algorithm but i can’t able to update my  weight manually.<br>
here is the formula for updating weight:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/6/62b879f0ea33ef937e1ab6c46b686222f3606f32.png" data-download-href="https://discuss.pytorch.org/uploads/default/62b879f0ea33ef937e1ab6c46b686222f3606f32" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/6/62b879f0ea33ef937e1ab6c46b686222f3606f32_2_690x166.png" alt="Capture" data-base62-sha1="e5k1Uu1EnXLD63eYi01bFqnMPsK" width="690" height="166" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/6/62b879f0ea33ef937e1ab6c46b686222f3606f32_2_690x166.png, https://discuss.pytorch.org/uploads/default/original/2X/6/62b879f0ea33ef937e1ab6c46b686222f3606f32.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/6/62b879f0ea33ef937e1ab6c46b686222f3606f32.png 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">889×215 18.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I try to update weight like this:</p>
<pre><code class="lang-auto">def learn(self, batch_state, batch_next_state, batch_reward, batch_action):
        self.optimizer.zero_grad()
        outputs = self.dqn(batch_state).gather(1, batch_action.unsqueeze(1)).squeeze(1)
        next_outputs = self.dqn(batch_next_state).detach().max(1)[0]
        target = self.gamma*next_outputs + batch_reward
        loss = F.mse_loss(outputs, target)
        loss.backward()
        #self.optimizer.step()
        Weight =[]
        with torch.no_grad():
            for p in self.dqn.parameters():
                Weight.append(p)
                Weight = self.backward_view(outputs, target, batch_reward, p, loss, p.grad)
                p.copy_(Weight)
                
    def backward_view(self, outputs, target, batch_reward, p, loss, grad):
        Weight = []
        alpha, lambd = 0.0001, 0.5
        for bs in range(32):
            i = 0
            for parameter in p:
                self.E[i] = torch.Tensor(lambd*self.gamma*self.E[i]) + grad[bs]
                parameter -= alpha * self.E[i]*(target[bs] - outputs[bs]) 
                Weight.append(parameter)
                i += 1
        return Weight

but got following error:
TypeError: new(): data must be a sequence (got float)

</code></pre>
<p>can anyone tell me how to update weight according to this formula using pytorch, please?</p>