akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to build an artificial layer of neural network?

Seation_Lee: <p>Hi, I’ve read the codes of PyTorch, the layers are mainly about “nn.linear()”, “nn.sequential()” and so on, but I want to build a layer that is completely artiifcial by myself, I’ve read the documents of PyTorch, but I havn’t an idea, how could it be solved?<br>
Please give an idea, thanks.</p>

ptrblck: <p>Could you explain a bit, what you mean by “artificial”?</p>

Seation_Lee: <p>Hi, the “artificial” means that can edit the layers optionally, for example, can consuct a kind of novel neural network just like ResNet before, that is, could PyTorch construct a network optionally? And could PyTorch edit the neurons of the network? Thanks.</p>

ptrblck: <p>If I understand your question correctly, you would like to use something like a neural architecture search?<br>
If so, have a look at e.g. <a href="https://github.com/quark0/darts" rel="nofollow noopener">this DARTS implementation</a>.</p>

Seation_Lee: <p>Thanks very much, I’ll have a study on the codes you present.</p>
Pytorch 1.2 on Win10 with Cuda10

manuelz: <p>I’m trying to install Pytorch 1.2 on a laptop with Windows 10 Home (GTX 1050) in a new conda environment with Python 3.7, following the instructions from:</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">
      <a href="https://pytorch.org/get-started/locally/" target="_blank" rel="nofollow noopener">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="https://pytorch.org/assets/images/pytorch-logo.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://pytorch.org/get-started/locally/" target="_blank" rel="nofollow noopener">PyTorch</a></h3>

<p>An open source deep learning platform that provides a seamless path from research prototyping to production deployment.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>conda install pytorch torchvision cudatoolkit=10.0 -c pytorch</p>
<p>I can only get a “false” when testing my cuda capabilities with torch.cuda.is_available()</p>
<p>What can I do please?</p>
<p>EDIT: It works using the “Cuda 9.2” option from the above-referred url:</p>
<p>conda install pytorch torchvision cudatoolkit=9.2 -c pytorch -c defaults -c numba/label/dev</p>

peterjc123: <p>Have you tried to update your GPU driver?</p>
Why there is no show-case of model of pytorch users?

111137: <p>Hi,</p>
<p>↑ that is all.</p>
<p>As a first penguin of the topic, my model is here;</p>
<pre><code class="lang-http">https://github.com/IAMAl/QNet
</code></pre>
<p>My model is experiment to replace GRU cell with my proposal cell called Q-cell.<br>
I use MNIST as an experiment application.</p>
Sample from two torch.tensors based on certain weights or probabilities

debowin: <p>Hello,<br>
I’m curious if its possible to do something like <code>np.random.choice([a, b], p=[p_a, p_b])</code><br>
using PyTorch where <code>a</code> and <code>b</code> are 1-d tensors of length L and <code>p_a</code> and <code>p_b</code> are probabilities used to sample elements from either tensor <code>a</code> or <code>b</code> into the resultant 1-d tensor.</p>
<p>I think <code>torch.multinomial([p_a, p_b], L)</code> might be useful here - it returns a 1-d tensor length <code>L</code> of 0s and 1s based on the probabilities I give it but I’m drawing a blank of how to utilize this tensor to sample from my tensors <code>a</code> and <code>b</code>. Appreciate any help!</p>

ptrblck: <p>I’m not sure to completely understand the use case, but this code snippet should work, if you would like to sample either the element from <code>a</code> or <code>b</code> based on the probabilities:</p>
<pre><code class="lang-python">L = 10
a = torch.arange(L)
b = torch.arange(L, 2*L)

p_a = 0.2
probs = torch.empty(1, L).uniform_()
idx = (probs &gt;= p_a).to(torch.long)
c = torch.stack((a, b))
result = torch.gather(c, 0, idx)

print(c)
&gt; tensor([[ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9],
          [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]])
print(probs)
&gt; tensor([[0.8268, 0.0361, 0.2744, 0.4381, 0.7244, 0.8509, 0.5047, 0.0549, 0.2767,           0.3203]])
print(idx)
&gt; tensor([[1, 0, 1, 1, 1, 1, 1, 0, 1, 1]])
print(result)
&gt; tensor([[10,  1, 12, 13, 14, 15, 16,  7, 18, 19]])
</code></pre>

debowin: <p>Thank you - this is great! I ended up using the gather function(which I was previously unaware of) to sample from my a and b tensors after stacking them in dim=0 with the indices I got from torch.multinomial()</p>
Neural net object detection on Roulette wheel/ball and Prediction! help!

XxaemaethxX: <p>Hi, My senior design team for Computer Engineering our doing a project where we use a Neural network to object detect a roulette ball and the 0fret on the Roulette wheel and try to have the network generate a prediction of where the ball will land via some initial conditions.</p>
<p>Has anyone here done a similar project? Can anyone offer any recommendations of what type of camera/webcam we could use to detect the ball that would provide high enough specifications so the neural net is able to identify the ball (good image quality etc…). We’re not sure what FPS we need, or resolution, global shutter, etc…</p>
<p>Anyone help would be appreciated!!!<br>
git hub repo for the project is here:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/alpizano/Deep-Learning-Neural-Network-for-Chaotic-Systems-" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/39110859?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/alpizano/Deep-Learning-Neural-Network-for-Chaotic-Systems-" target="_blank" rel="nofollow noopener">alpizano/Deep-Learning-Neural-Network-for-Chaotic-Systems-</a></h3>

<p>Senior design neural network using yolov3, darknet, Python, PyTorch, and OpenCV to predict outcomes of the chaotic system - Roulette. - alpizano/Deep-Learning-Neural-Network-for-Chaotic-Systems-</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
Computing the Hessian matrix -- correctness and performance

mengxixi: <p>Hello,</p>
<p>I’ve been trying to compute the Hessian of my loss function with respect to the parameters of my network. The code I use below is inspired from <a href="https://discuss.pytorch.org/t/compute-the-hessian-matrix-of-a-network/15270">this discussion</a>,</p>
<pre><code>def compute_hessian(grads, params):
    H = []
    for i, (grad, p) in enumerate(zip(grads, params)):
        grad = grad.reshape(-1)
        d = len(grad)
        dg = torch.zeros((d, d))

        for j, g in enumerate(grad):
            g2 = autograd.grad(g, p, create_graph=True)[0].view(-1)
            dg[j] = g2

        H.append(dg)

    return H
</code></pre>
<p>where <code>params</code> is the list of parameters of my network and <code>grads</code> is computed manually rather than using <code>backward</code>, i.e.</p>
<pre><code>grads = autograd.grad(loss, self.params, create_graph=True)
</code></pre>
<p>My first question is, by iterating through the list of parameters, I am essentially computing a sub-Hessian for each parameter, and therefore my overall Hessian is block diagonal rather than being the full Hessian. While I do realize that I can concatenate all the parameters of my model into one giant vector then compute the full Hessian, when I update the parameters I need to break them apart and re-distribute to the list of parameters which seems to be a hassle. Am I making a mistake somewhere or is this how it’s suppose to work?</p>
<p>My second question is, the for-loop above used to take the grad of each gradient seem to be slow. Is there any way to speed it up or add more parallelization to it?</p>
<p>Thanks,</p>

Yaroslav_Bulatov: <p>Yes, this is how it’s supposed to work. However, for any useful network, full-Hessian is going to be too large to fit in memory. Block-diagonal is a common approximation because off-diagonal blocks may approach rank-1 matrices for deep network, hence not too useful.</p>
<p>You can save some time by computing block Hessians for all layers from same set of backward values, here’s an example – <a href="https://gist.github.com/yaroslavvb/6b08eb8f683b646785d39e97679fdb2e" rel="nofollow noopener">https://gist.github.com/yaroslavvb/6b08eb8f683b646785d39e97679fdb2e</a></p>
Equivalent of x.sum().item() in torch 0.3

fbrah: <p>Hi,</p>
<p>I am working on someone’s code in torch 0.3.<br>
I want to count number of padding so I wrote:</p>
<pre><code class="lang-auto">            ys = valid_decoder_inputs[1:,:].view(-1) #true target 
            not_padding = ys.ne(PAD_IDX)
            no_not_padding = not_padding.sum().item()
</code></pre>
<p>However torch 0.3 doesn’t support item(). Can you help of how to write this line of code in torch 0.3?</p>

ptrblck: <p>You could replace it with <code>x.sum().data[0]</code>.<br>
The probably better approach would be to port the code to the latest PyTorch version, as you might encounter already solved bugs etc.</p>
Initialize Model vs Transfer Learning

arjung: <p>I have the following code:</p>
<pre><code class="lang-auto">for i in range(num_models):

    model = ConvNet()
    model.cuda()
    model = nn.DataParallel(model, device_ids=[0])

    ...

</code></pre>
<p>I suspect that these lines don’t actually reinitialize the model with random weights. I believe this because I observed better results for models which are trained in later iterations of the for-loop than if the same models were trained individually without any for-loop, leading me to think that some sort of transfer learning is taking place.</p>
<p>Is this a possibility? How could I reinitialize the weights of the model to be random with every iteration of the for-loop?</p>

ptrblck: <p>The model is randomly initialized in each loop. You could run a quick test and compare (some) parameters of the new model to the last trained one (use <code>.clone()</code> to get the real value instead of a reference).</p>
Using an trained model to finetune, and error occured when loading dict

huihuiM: <p>Two model：Net_D，Net_C<br>
Net_D</p>
<pre><code class="lang-auto">class Net_D(nn.Module):
    def __init__(self):
        super(Net_D, self).__init__()

        model = models.resnet50(pretrained=False)
        model.fc = nn.Linear(hidden_dim, num_D_class)
        model = remove_last(model)
        self.resnet = model
........
</code></pre>
<p>class Net_C(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Net_C, self).<strong>init</strong>()</p>
<pre><code>    model = self.load_state_dict(torch.load("best.mdl")
    model.fc = nn.Linear(hidden_dim, num_C_class)
    model = remove_last(model)

    self.resnet = model
</code></pre>
<p>…</p>
<pre><code class="lang-auto">
Two training code: train_D.py, train_C.py
part code for train_D.py
 torch.save(model.state_dict(), 'best.mdl')

part code for train_C.py
    model = Net_C()
    model.to(device)
    print('Model is built ...')
。。。。。。

I run the train_D.py successfuly and I got the best.mdl file.
But when I try to run train_C.py to pretrain the best model, I got the following error info:
RuntimeError: Error(s) in loading state_dict for Fusion_Net_C:
	Unexpected key(s) in state_dict: "resnet.0.weight", "resnet.1.weight", "resnet.1.bias",......
But I found there are actually such keys above.
Any help, thanks.</code></pre>

ptrblck: <p>Your <code>Net_C</code> doesn’t seem to initialize the resnet50 model, which was used in <code>Net_D</code>.<br>
Could you explain your use case a bit, e.g. why you need a new model architecture?</p>

huihuiM: <p>Thank you very much for your reply.<br>
My work is to train two tasks, which are related but different. In fact,I can say the relation between the two tasks is hierarchical. I planed to train taskone( which is task D) firstly and save the model, and then do some modification to the model D to train task two(which is task C).</p>
<p>and the code<br>
“self.resnet = model” is what I try to init the resnet.</p>

ptrblck: <p>The <code>state_dict</code> will contain all parameters and buffers of the model.<br>
In the case of <code>Net_D</code> it will contain the parameters/buffers of the internal <code>resnet50</code> as well as your custom linear layer. I’m not sure, what <code>remove_last</code> does, but if it removes the last linear layer, your custom one will most likely be gone.</p>
<p>In <code>Net_C</code> you are calling <code>self.load_state_dict</code> immediately in the <code>__init__</code> method, which won’t work, as your model does not have any parameters or buffers yet.<br>
If you just want to initialize the linear layer and load its <code>state_dict</code>, you could use <a href="https://discuss.pytorch.org/t/loading-a-specific-layer-from-checkpoint/52725/2">this approach</a>.</p>
TypeError: cannot assign 'torch.FloatTensor' object to parameter '0' (torch.nn.Parameter or None required)

111137: <p>Hi,</p>
<p>I try to make custom model which includes;</p>
<pre><code class="lang-python">class Model(nn.Module):
  def __init__(self):
    super(Model, self).__init__()
        self.w = [torch.randn((NUM_INPUT), requires_grad=True) for _ in range(NUM_HIDDEN)]
        for index in range(NUM_HIDDEN):
          self.parameters = nn.ParameterList([self.w[index]])
</code></pre>
<p>I did the “nn.ParameterList()” because of error “no parameter” is occurred.<br>
But, running makes an error of;</p>
<pre><code class="lang-python">TypeError: cannot assign 'torch.FloatTensor' object to parameter '0' (torch.nn.Parameter or None required)
</code></pre>
<p>Why this error is occurred, and then how to solve it?</p>

111137: <p>I did;</p>
<pre><code class="lang-python">        for index in range(NUM_HIDDEN):
          self.parameters = nn.Parameter(self.w[index])
</code></pre>
<p>Then it worked.</p>
After building optimizer, moving model to other device and move back, does optimizer still work?

realmongoose: <p>as this<a href="https://github.com/pytorch/pytorch/issues/2830" rel="nofollow noopener">issue</a> say that: every time you move model to different device, you should build optimizer again, so</p>
<pre><code class="lang-python">model = Model()
model.cuda()
optimizer = optim.Adam(model.parameters())

for d, gt in trn_dataloader:
    # train
    ... 
    optimizer.step()
    model.cpu() # move to cpu
    # eval or do other things
    ...
    model.cuda()  # but finnally, move back
</code></pre>
<p>does optimizer run as expected?</p>
Use module with **kwargs in SummaryWriter.add_graph

BenjaminBossan: <p>I would like to add a module graph to tensorboard using <code>SummaryWriter.add_graph</code>. My module’s <code>forward</code> method requires some <code>*args</code> and some <code>**kwargs</code>.</p>
<p>Walking down the stack, it looks like <code>add_graph</code> calls <code>torch.jit.get_trace_graph(model, args)</code>, i.e. not passing any kwargs, even though <code>torch.jit.get_trace_graph</code> would support it.</p>
<p>Is there a way to pass kwargs? The only solution I see right now is <code>inspect</code>ing the <code>foward</code> parameters and rearranging the arguments so that they are all converted to args, which I would really like to avoid.</p>
How to save model in C++ with network structure?

beichen2012: <p>I wrote a simple self-define module：</p>
<pre><code class="lang-auto">struct BasicConvOptions
{
	BasicConvOptions(int ch_in, int ch_out, int ksize) :
		ch_in_(ch_in), ch_out_(ch_out), ksize_(ksize) {}
	TORCH_ARG(int, ch_in);
	TORCH_ARG(int, ch_out);
	TORCH_ARG(int, ksize);
	TORCH_ARG(int, stride) = 1;
	TORCH_ARG(int, padding) = 0;
};

class BasicConvImpl : public torch::nn::Cloneable&lt;BasicConvImpl&gt;
{
public:
	explicit BasicConvImpl(BasicConvOptions options);

	torch::Tensor forward(const torch::Tensor&amp; input);

	void reset() override;

	/// Pretty prints the `BatchNorm` module into the given `stream`.
	void pretty_print(std::ostream&amp; stream) const override;

protected:
	//torch::nn::Sequential seq = nullptr;
	torch::nn::Conv2d conv = nullptr;
	torch::nn::BatchNorm bn = nullptr;
};

TORCH_MODULE(BasicConv);

</code></pre>
<p>There would don’t save the network structure when using torch::save；<br>
actually, I want to load the module torch::jit::load in the future.</p>
<p>can any one help me？</p>
How to drop specific filters in some CNN layers

Piggy: <p>I have read the 2019 CVPR paper:<em>RePr: Improved Training of Convolutional Filters</em> and want to implement its method. First, we need to train a CNN several iterations. Then we can use a metric algorithm to rank all filters and drop the 30% least important filters to continue to train a few iterations just like pruning. After that, re-initialize the filters  that are dropped before and continue to train the networks. We modify the training process by cyclically removing redundant ﬁlters, retraining the network, re-initializing the removed ﬁlters, and repeating.<br>
In my opinion, drop the filter  is not really removing it. It just make the weight of that filters don’t work and don’t update. I want to achieve this goal by resetting the weight of dropped filters to 0. For example, I want drop the 23rd filter in first layer,<br>
<code>params = list(model.named_parameters()) params[0][1][22] = 0</code><br>
But it has an error when loss.backward():RuntimeError:leaf variable has been moved into graph interior.<br>
Does anyone know an official method or some other methods to drop specific filters in some layers? Thanks very much!</p>

Daniel_Dagnino: <p>One option can be something like this:</p>
<pre><code class="lang-auto">for key, value in dict(model.named_parameters()).items():
    if 'the_name_of_the_conv_you_want_to_zero_out' in key:
        value.data[0, 1, 22] = 0
</code></pre>
<p>or, another option:<br>
<code>model.the_name_of_the_conv_you_want_to_zero_out.weight.data[0, 1, 22] = 0</code></p>

find1dream: <p>However, if you want to prune that network, you have to choose another way.<br>
Here is the post about how to replace network in running time: <a href="https://find1dream.github.io/en/How_to_change_CNN_structure_in_running_time_with_pytorch" rel="nofollow noopener">https://find1dream.github.io/en/How_to_change_CNN_structure_in_running_time_with_pytorch</a></p>
Strange behavior of tensor.floor()

anxu: <p>Hi, I am using the floor function and found this strange behavior:</p>
<p>a=torch.tensor(1-5.9605e-8)<br>
a.floor()<br>
=&gt;tensor(0.)<br>
(a+1).floor()<br>
=&gt;tensor(2.)</p>
<p>How is floor computed and why does not (a+1).floor() output 1.0?</p>

KFrank: <p>Hello An!</p>
<aside class="quote no-group" data-username="anxu" data-post="1" data-topic="54223" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/eb8c5e/48.png" class="avatar"> anxu:</div>
<blockquote>
<p>Hi, I am using the floor function and found this strange behavior:</p>
<p>a=torch.tensor(1-5.9605e-8)<br>
a.floor()<br>
=&gt;tensor(0.)<br>
(a+1).floor()<br>
=&gt;tensor(2.)</p>
<p>How is floor computed and why does not (a+1).floor() output 1.0?</p>
</blockquote>
</aside>
<p>I haven’t checked the arithmetic precisely, but I believe the following<br>
is going on:</p>
<p>A <code>torch.tensor</code> by default uses single-precision (32-bit) floating<br>
point numbers.  These have approximately 7 decimal places of<br>
precision.  Your small number, 5.9e-8, is (relative to 1) just on<br>
the edge of this precision.</p>
<p>So it turns out that <code>1 - 5.9e-8</code>, when represented by a 32-bit<br>
floating-point number, is, indeed, not equal to (and a little bit less<br>
than) 1, so <code>floor()</code> takes it down to 0.  But <code>(1 - 5.9e-8) + 1</code>,<br>
when represented by a 32-bit floating-point number, is equal<br>
to 2, so <code>floor()</code> leaves it at 2.</p>
<p>Redo your experiment with</p>
<pre data-code-wrap="python"><code class="lang-python">a = torch.tensor (1 - 5.9605e-8, dtype = torch.float64)
</code></pre>
<p>and you should be your expected results.  (That is, when<br>
represented by a 64-bit floating-point number, <code>(1 - 5.9e-8) + 1</code><br>
will, indeed, not be equal to (and will be a little bit less than) 2,<br>
so <code>floor()</code> will take it down to 1, rather than leaving it at 2.)</p>
<p>Again, I haven’t done the arithmetic exactly, but if you do the<br>
<code>float64</code> experiment, but try various values of your small number around 1.e-16, you should be able to find a value where your<br>
“strange behavior” shows up again – just pushed down to smaller<br>
numbers by the increased precision of <code>float64</code> relative to <code>float32</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

anxu: <p>Thank you! It works now.</p>
How to implement the follow structure of network

Laine_J: <p>Hi, I wantto implement the following network, each layer receives actication of the lower layer along with speaker code, and using back-propagation to update the speaker code. but I don’t know how to implement it.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/e/e08b5a642b015e9d7d9e82e271775b8bc67a0219.png" alt="%E5%BE%AE%E4%BF%A1%E6%88%AA%E5%9B%BE_20190824191901" data-base62-sha1="w2pvZvtN1DNAr6hJH6yWukaeCKJ" width="225" height="274"></p>

andreaskoepf: <p>In your desired architecture the inputs of the layers are a combination of speaker code and feature previous layer outputs or the feature vector input. I guess you want to condition on the speaker code.</p>
<p>You could pass the speaker code as additional input, e.g. by concatenating it to the regular layer input. Another very simple method is to use addition to mix in the speaker code. From your diagram I cannot see what kind of network you want, e.g. if the red layers are fully-connected layers or conv layers. The first thing you could do is to write down all the tensor dimensionalities and then think about bringing them into a ‘compatible form’, e.g. if the dimensionality does not match you could try use a linear layer to adapt it.</p>

Laine_J: <p>I tried to implement this structure, but I‘am not sure that’s right. The following code just a example.</p>
<pre><code class="lang-auto">class TestDNN(nn.Module):
    def __init__(self):
        super(TestDNN, self).__init__()
        self.speaker_code = nn.Parameter(torch.randn(1))
        self.fc1 = nn.Linear(2, 2, bias=False)      
        self.fc2 = nn.Linear(3, 1, bias=False)
    
    def forward(self, input):
        x = torch.cat((self.speaker_code.repeat(input.shape[0],1), input), 1)                  
        x = self.fc1(x)
        x = torch.cat((self.speaker_code.repeat(input.shape[0],1), x),1)                                
        return self.fc2(x)
</code></pre>
Pre-compiled tests failing

mboedigh: <p>Sorry I did not see this posted before and no mention in the installation instructions.</p>
<p>In both the release and debug distributions of the libtorch libraries there are test directories with pre-compiled executable programs.</p>
<ol>
<li>Is it expected that a fair number of the test programs are not passing?</li>
<li>is it expected that ones that fail differ between the release and debug versions of the libtorch<br>
library?</li>
<li>some issue debug assertions (e.g. “list iterator not decrementable” in GraphTest.exe) pointing to programming errors rather than test failures</li>
</ol>
<p>My system has Windows 10, CUDA 10.0, cuDNN v7.6.2, libtorch 1.2 and a Nvidia 1080 graphics card.</p>
<p>I’m asking because I have general issues getting anything downstream to work, and I’m trying to troubleshoot each component separately. So, if anyone has the tests working, particularly on a similar system, please let me know.</p>
<p>A feature request is adding a “tests passing|failing” icon generated by CI to the git repository so we know what to expect</p>

peterjc123: <p>Yes, they are currently missing in the Windows tests. Would you please create a ticket at <a href="https://github.com/pytorch/pytorch/issues" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues</a>. I tried running the tests locally and most of the tests passed. The only one failed is c10_metaprogramming_test.exe. I didn’t try the debug versions.<br>
About the icon, there is already one in <a href="https://github.com/pytorch/pytorch" rel="nofollow noopener">https://github.com/pytorch/pytorch</a> or <a href="https://ci.pytorch.org/jenkins/job/pytorch-builds/job/pytorch-win-ws2016-cuda9-cudnn7-py3-trigger" rel="nofollow noopener">https://ci.pytorch.org/jenkins/job/pytorch-builds/job/pytorch-win-ws2016-cuda9-cudnn7-py3-trigger</a>.</p>

mboedigh: <p>Thanks,  this was very helpful. The solution to getting these running is the same solution you provided that fixed my other issues. the libtorch-release/lib/*.dlls need to be copied into the same directory as the exe files (or the environment variable PATH needs to have a path to the dlls).  I didn’t check all of the tests, but I tested some that were failing before.</p>
PyTorch Get Started Page Gives Wrong Install Instruction

xian_kgx: <p>Not sure where do I report this but as per the topic suggest, the <a href="https://pytorch.org/get-started/locally/" rel="nofollow noopener">Start Locally | PyTorch</a> gives the wrong install instruction <strong>if used without user interaction (default)</strong>.</p>
<p>As of now, without user interaction (default), the install instruction is:</p>
<p><code>conda install pytorch torchvision cudatoolkit=9.2 -c pytorch -c defaults -c numba/label/dev</code></p>
<p>for <strong>CUDA 10.0</strong>. However, when the user clicks on CUDA 10.0, it then proceeds to give the correct install instruction, so this is just some initialization problem.</p>

ptrblck: <p>Thanks for letting us know!<br>
The install command indeed points to the CDUA9.2 install while CUDA10.0 is selected.<br>
As you explained, the right command is shown after reselecting CUDA10.0.</p>

smth: <p>Thanks for reporting. I fixed it via <a href="https://github.com/pytorch/pytorch.github.io/commit/0c87f939997aa964ff42248d7f15f939f4c54ef2" rel="nofollow noopener">https://github.com/pytorch/pytorch.github.io/commit/0c87f939997aa964ff42248d7f15f939f4c54ef2</a> and the fix should be live in 5 minutes.</p>
A linear transformation

slavavs: <p>This is a clipping from an article: “Here instead of using the embedding, I simply used a linear transformation to transform the 11-dimensional data into an n-dimensional space. This is similar to the embedding with words.”<br>
I can’t understand how I can do such a linear transformation. Can someone explain to me with a simple example?</p>

ptrblck: <p>Maybe the author is just using a simple linear layer: <code>nn.Linear(11, n_dim)</code>.</p>

slavavs: <p>The vector that we presented in n-dimensional space (similar to ebedding), should its n-dimensional value be constant, or is it a trained parameter?</p>

andreaskoepf: <p>The n-dimensional outputs of the linear layer will not be constant (for reasonable weight values). They will be different for each input vector and the projection of a given input vector changes over the course of the training as the weights of the linear layer are adjusted.</p>
Concatenate two different dimensions tnesor

Laine_J: <p>Hi, I want to concate two different dimensions tensor, like x = [[1,2], [3, 4], [5, 6]] , y =[ [1, 1], [2,2]], and the result of concatenate should be z = [[1, 1, 1, 2], [1, 1, 3, 4], [2, 2, 5, 6]].</p>
Issue with pytorch & ODL integration

Subho: <p>I ran into this issue while working on CT image reconstruction. I have written a small script (attached) which will reproduce the issue.</p>
<p>script: <a href="https://colab.research.google.com/drive/1k6QTHNMnA1OJfuRWfXfOil1ZP5yxsJrz" rel="nofollow noopener">https://colab.research.google.com/drive/1k6QTHNMnA1OJfuRWfXfOil1ZP5yxsJrz</a></p>
<p>What I am doing in this script is the following: I am loading the MNIST images and applying the ray transform operator on them. Let’s denote the ground-truth images by x_0 and their CT measurement as y = A(x_0). I am creating the Radon transform op. A and then wrapping it as a PyTorch module using ODL (<a href="https://github.com/odlgroup/odl" rel="nofollow noopener">https://github.com/odlgroup/odl</a>). Subsequently, I am computing an estimate of x_0 by minimizing J(x) = || y - A(x) ||_2^2 w.r.t. x (basically, running Landweber-type iterations, but using Adam instead of SGD). Surprisingly, the gradient of J(x) w.r.t x goes to 0 just after 1 iteration, regardless of the initialization (which is random Gaussian, btw). The problem, apparently, is not with the optimizer setting in Pytorch, because, I just tested the minimization of J_1(x) = || x_0 - x ||_2^2 for a sanity check and it works perfectly fine. I tested with a different forward operator A’ (namely, the forward operator that removes a set of pixels from the image and retains the rest), and there it works perfectly fine as well.I would really appreciate any help in fixing this particularly strange issue. Thanks in advance.</p>
<p>P.S. It is surprising that although the gradient vanishes, the cost keeps changing.</p>
Torch.as_tensor vs register_buffer in nn.Module

dashesy: <p>What is the benefit of using <code>register_buffer</code> compared to declaring a normal tensor in <code>__init__</code>?</p>
<pre><code class="lang-python">class MyModule(nn.Module):
    def __init__(self, child):
        self.child = torch.as_tensor(child).int()
        # vs
        self.register_buffer('child', torch.from_numpy(np.array(child, dtype=np.int32)))
</code></pre>
<ol>
<li>The buffer is serialized along with the module, but if we initialize it in <code>__init__</code> then that would not matter.</li>
<li>Also buffer will be pushed to cuda if the model is pushed to cuda, but we could ourselves check the device and push accordingly</li>
</ol>
<p>So what is the special case that only buffer can do?<br>
It would be nice to have a <code>.register_buffer</code> that does not serialize the tensor, just push to cuda with model.cuda</p>

ptrblck: <ol>
<li>
<p>Since buffers are serialized, they can also be restored using <code>model.load_state_dict</code>. Otherwise you would end up with a newly initialized buffer.</p>
</li>
<li>
<p>That’s possible, but not convenient, e.g. if you are using <code>nn.DataParallel</code>, as each model will be replicated on the specified devices. Hard-coding a device inside your model won’t work, so you would end up using some utility functions are pushing the tensor inside <code>forward</code> to the right device.</p>
</li>
</ol>

dashesy: <p>Would be nice as a feature to have a <code>register_buffer</code> option that avoid the serializing part. I have some intermediate tensors that are constant, so there is no point in serializing them for example. But still will benefit from automatic <code>.cuda()</code> pushing to the correct device for example.</p>
Type Error while Scripting Learned Positional Embedding

sukuya: <p>I am trying to script LearnedPositionalEmbedding from Fairseq:<br>
<code>CODE</code></p>
<pre><code class="lang-auto">import torch.nn as nn
import torch
import torch.jit
from fairseq import utils

class LearnedPositionalEmbedding(nn.Embedding):
    def __init__(self, num_embeddings, embedding_dim, padding_idx, left_pad):
        super().__init__(num_embeddings, embedding_dim, padding_idx)
        self.left_pad = left_pad
        # self.register_buffer('padding_idx_', padding_idx)

    def forward(self, input_, incremental_state=None):
        """Input is expected to be of size [bsz x seqlen]."""
        if incremental_state is not None:
            positions = input_.data.new(1, 1).fill_(self.padding_idx_ + input_.size(1))
        else:
            positions = utils.make_positions(input_.data, self.padding_idx_, self.left_pad)
        return super().forward(positions)

    def max_positions(self):
        """Maximum number of supported positions."""
        return self.num_embeddings - self.padding_idx_ - 1

padding_idx = torch.tensor([[1]], dtype=torch.long)
model = LearnedPositionalEmbedding(4,5, padding_idx, False)
model_scripted = torch.jit.script(model)
</code></pre>
<p>And I get following error:</p>
<pre><code class="lang-auto">TypeError: 
'Tensor' object for attribute 'padding_idx' is not a valid constant.
Valid constants are:
  1. a nn.ModuleList
  2. a value of type {bool, float, int, str, NoneType, function, device, layout, dtype}
  3. a list or tuple of (2)
</code></pre>
<p>Even issue <a href="https://github.com/pytorch/pytorch/issues/16284" rel="nofollow noopener">#16284</a> is not of help.</p>

driazati: <p>In the <code>nn.Embedding</code> provided by PyTorch, <code>padding_idx</code> is an <code>int</code> that doesn’t change, for that reason we have added it to <code>__constants__</code> in <code>nn.Embedding</code> (<a href="https://github.com/pytorch/pytorch/blob/10c4b98ade8349d841518d22f19a653a939e260c/torch/nn/modules/sparse.py#L77-L78" rel="nofollow noopener">code here</a>). Since <code>LearnedPositionalEmbedding</code> does not override <code>__constants__</code>, it gets the one from <code>nn.Embedding</code>.</p>
<p>If you want <code>padding_idx</code> to be a Tensor (<a href="https://pytorch.org/docs/master/jit.html#python-defined-constants" rel="nofollow noopener">which is not a supported constant type</a>), you’ll have to provide your own <code>__constants__</code> that removes <code>padding_idx</code> but keeps the other stuff around, something like:</p>
<pre><code class="lang-python">from fairseq import utils

class LearnedPositionalEmbedding(nn.Embedding):
    __constants__ = ['num_embeddings', 'embedding_dim', 'max_norm',
                     'norm_type', 'scale_grad_by_freq', 'sparse']
    def __init__(self, num_embeddings, embedding_dim, padding_idx, left_pad):
        super().__init__(num_embeddings, embedding_dim, padding_idx)
        self.left_pad = left_pad</code></pre>
What is the equivalent of torch.no_grad() and model.eval() in torch 0.3?

fbrah: <p>Hi,</p>
<p>I am working on someone’s code which is written in torch 0.3. I realized torch 0.3 doesn’t support torch.no_grad() or model.eval() in this regard, I wonder how I can make sure that no back propagation is being done while doing the validation on dev or test set?</p>
<p>Thanks</p>

SimonW: <p><code>Variable(..., volatile=True)</code> for inputs</p>
Conv layers not being properly frozen?

glefundes: <p>Hello,<br>
I’m having some trouble when trying to freeze layers of a pretrained network in order to learn  a new head. My forward pass code is as follows:</p>
<pre><code class="lang-auto">def forward(self, x):
        x = self.conv1(x)
        x = self.blocks(x)
        x = self.conv2(x)

        a = self.linear7(x)
        a = self.linear1(a)
        a = a.view(a.size(0), -1)

        if (self.transfer):
            x = self.conv3(x)
            x = self.conv4(x)
            x = x.view(x.size(0), -1)
            
            x1 = torch.softmax(self.fc1(x), dim=1)
            x2 = torch.softmax(self.fc2(x), dim=1)

            return a, x1, x2

        return a
</code></pre>
<p>What I want to do is, learn the conv3, conv4, fc1 and fc2 layers for a new task, using the features extracted using previously trained weights for the other layers.</p>
<p>Relevant parts of the training code are:</p>
<pre><code class="lang-auto">if args.resume:
	to_train = ['fc1.weight', 'fc1.bias', 'fc2.weight', 'fc2.bias', 'conv3.weight', 'conv3.bias', 'conv4.weight', 'conv4.bias']
	model.load_state_dict(torch.load(args.resume)['backbone_net_list'], strict=False)
	# Freeze layers
	for name, param in model.named_parameters():
		if(name not in to_train):
			param.requires_grad = False
for batch_idx, (img, labels) in enumerate(train_loader):
		
		img = img.to(model.device)
		gt1 = labels['1'].to(model.device)
		gt2 = labels['2'].to(model.device)
		#Forward parse_args
		a, x1, x2 = model(img.float())

		# # Cost Function
		cost = utils.cost_fn(gt1, x1, gt2, x2)
		# Backprop
		optimizer.zero_grad()
		cost.backward()
		# Update parameters
		optimizer.step()
		torch.save(model.state_dict(), os.path.join(args.output, 'model-{}.pth'.format(epoch)))
</code></pre>
<p>(I’m saving a model after just one backprop in order to investigate the updating of the weights.)<br>
My problem is, even after just one iteration, if I load the saved model and compare it’s output for the ‘a’ variable against that of the original model, I get different results. If my reasoning is correct, I should get the same output given that I had the weights frozen, and that output never passes through the new head, right? What am I doing and/or thinking wrong here?</p>
Exporting ONNX v5 from torch onnx export

bhushans23: <p>I am using PyTorch 1.2 and exporting hugging face GPT-2 model into ONNX.<br>
But, exported model is of ONNX v4 file format and is based on ONNX op version 9.<br>
I am expecting and would like to export to v5 file format and op version 10 (i.e. ONNX 1.5.0)<br>
How do we control exported ONNX version?<br>
As per PyTorch 1.2 release notes (<a href="https://github.com/pytorch/pytorch/releases" rel="nofollow noopener">https://github.com/pytorch/pytorch/releases</a>) this support exists.</p>
How to get the values of 'torch._C.Value'

richardhahahaha: <p>I am working to export a trained model to other format. Now I can get the graph and nodes from onnx export function but I cannot read the parameters of some operations(or layers like convolution) of network. these values sames be stored in the instance of ‘torch._C.Value’.  Anyone know how to read the parameters?</p>
<p>Thanks.</p>

ivory_yang: <p>I have the same problem，so have you fix it?</p>
LSTMCell "Batch" Dimension

whoab: <p><a href="https://pytorch.org/docs/stable/nn.html#lstmcell" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#lstmcell</a></p>
<ol>
<li>Where can we set the “Depth” of the LSTMCell found in Stacked LSTMs?</li>
<li>Is the batch dimension here like sentence length? How do we setup variable length ‘batch’ then?</li>
</ol>
<p>The data is (N, nL, nE) where N is number of sentences, nL is a sentence length (Variable) and nE is the dimension of each word.</p>
<p>How do i setup the LSTM  cell for this?</p>
Backpropagate through input data

Krish: <p>I want to backpropagate through input to generate adversarial data. How do I train the model to do this task such that during training time, the image is not altered?</p>

benoriol: <p>I guess you mean you have a first model that outputs what is going to be the input of a second model?</p>
<p>If you want to propagate an error from, let’s say, the output of the second model all the way to the first model, I think PyTorch should be doing that for you automatically,along. Of course that is if you are using PyTorch tensors all along and unless you have at some point in between some sort of detachment, or you have something like  <code>with torch.no_grad()</code></p>
<p>I’m not sure if I understood your problem, I hope it helps.</p>

Krish: <p>I am trying to train a classifier on the CIFAR Dataset first and then use this trained model to generate adversarial examples.<br>
After training, I want to freeze the weights and pass a new image to the model with a target label and then optimise the input such that the probability of the target class become maximum.</p>

Krish: <p>I am trying to train a classifier on the CIFAR Dataset first and then use this trained model to generate adversarial examples.<br>
After training, I want to freeze the weights and pass a new image to the model with a target label and then optimise the input such that the probability of the target class become maximum.</p>

benoriol: <p>When you do the forward pass, all the gradients are computed. Not only the gradients for the model weights (which you will not need if you freeze de model), but all the gradients of the tensors that have been involved in computing the loss function (including the input image). So what you have to do is to declare an optimizer (Such as adam, SDG,…) , passing it the input image instead of the model’s weights as parameters to optimize.<br>
Then when you do ‘optimizer.step()’, you are optimizing the input image to a version that minimizes the loss function you defined.</p>
<p>However, I still didn’t understand what you meant by “How do I train the model to do this task such that during training time, the image is not altered?”. The way I explained it, the input image is going to be altered. Isn’t that the whole point of this kind of training?</p>
<p>Sorry for the delay in the answer.</p>

Krish: <p>Thank you for the answer. That was exactly what I am trying to do.<br>
Just one question, how do I pass the image only to the optimizer? Should I make it a parameter or can I just pass the image tensor with <code>requires_grad = True</code> to the optimizer?</p>

benoriol: <p>Instead of doing somthing like:<br>
<code>optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)</code><br>
try<br>
<code>optimizer = torch.optim.Adam(inputTensor, lr=learning_rate)</code></p>
<p>And maybe, as you said, you should make sure requires_grad=True for the input tensor.</p>

Krish: <p>Thanks a lot for the help.</p>
A strange problem with torch.optim: the variable changes, but the difference between the old and the updated variable is shown to be zero

Subho: <p>I stumbled upon a rather weird issue while debugging a larger code in PyTorch. A simple instance to reproduce the issue is here: <a href="https://colab.research.google.com/drive/11kBaxMOxN9i0X1vtaX47Yz7rkkHz7Kpu" rel="nofollow noopener">https://colab.research.google.com/drive/11kBaxMOxN9i0X1vtaX47Yz7rkkHz7Kpu</a></p>
<p>As you can see, the variable x gets updated in every epoch (see that their norms are different), however, the norm of their difference (rel_diff) turns out to be zero. Any help is deeply appreciated.</p>

ptrblck: <p>Use <code>x_old = x.detach().clone()</code>, since <code>.data</code> will be a reference and point to the same location in <code>x_old</code> and <code>x_new</code>.</p>
How to choose specific parameter for specific data

a53153fb407732ef019a: <p>Suppose that I have a group of data to classify, each data is represented as (x,topic,y). x is the input of my model and y is the label. Topic is some relevant information of x and there are 3 classes of topics. I want different topic data share part of the parameters while other parameters are different. For example</p>
<pre><code class="lang-python">class Net(nn.Module):
    def __init__(self,input_dim,hidden_dim,num_classes):
        self.W = nn.Linear(input_dim,hidden_dim)
        self.W1 = nn.Linear(hidden_dim,hidden_dim)
        self.W2 = nn.Linear(hidden_dim,hidden_dim)
        self.W3 = nn.Linear(hidden_dim,hidden_dim)

    def forward(self,x,topic):
        x = F.relu(self.W(x))
        if topic == 0:
            x = self.W1(x)
        elif topic == 1:
            x = self.W2(x)
        elif topic == 2:
            x = self.W3(x)
        return x
</code></pre>
<p>Since we would pack a group of data into a batch, above code only work when batch_size == 1. And I find it hard to extend to a bigger batch_size. I found two ways to solve this problem. One is to use loop in <code>forward</code> which seems unefficiently, the other is make a huge matrix Parameter according to topic tensor and do batch matrix multiplication in <code>forward</code> which is unefficient either. I am looking forward someone to help me solve this problem efficiently. I would appreciate it a lot!</p>
Softmax VS. Negative Sampling

tridelt: <p>Dear Community,</p>
<p>eventually Negative Sampling was introduced as substitute for the expensive Softmax Function as part of the word2vec model (<a href="https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf" rel="nofollow noopener">https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf</a>).</p>
<p>One questions though, if we had unlimited processing power would be still use Negative Sampling?<br>
Is it better to only update a few weights (NS) compared to all the weights (Softmax)?</p>
<p>Would love to hear your thoughts.</p>
<p>Yours,<br>
Luca</p>
Sequential inputs affecting model grad values

Yupjun: <p>I made a simple toy experiment. I want to get feedback about this… so let’s get started<br>
Last part, I made a simple question about this situation.</p>
<ol>
<li>Hypothesis<br>
I made a simple training code lines like this.<br>
I wanted to know whether the previous input value is contaminating model grad values.<br>
If the previously input value contaminate model, the grad values would be changed based on the previous value.<br>
So I made some experiment like this.<br>
if the mean of single_input_history is almost the same as multi_input_history:<br>
it means that previous input values are considered independent on the current input.</li>
</ol>
<pre><code class="lang-auto">class TestNet(nn.Module):
    def __init__(self):
        super(TestNet, self).__init__()
        self.layer = nn.Linear(1, 1)
    def forward(self, x):
        out = self.layer(x)
        return out
model = TestNet()
l1 = nn.L1Loss()

single_input_history = []
multi_input_history =[]

for _ in range(10000):
    model = TestNet()
    x = torch.tensor([1.0], requires_grad = True)
    y = torch.tensor([2.0], requires_grad = True)
#     z = torch.tensor([4.0], requires_grad = True)
    temp = model(x)
    loss = l1(temp,z)
    loss.backward()
    single_input_history.append(x.grad.data.item())


for _ in range(10000):
    model = TestNet()
    x = torch.tensor([1.0], requires_grad = True)
    y = torch.tensor([2.0], requires_grad = True)
    z = torch.tensor([4.0], requires_grad = True)
    temp = model(y)
    temp = model(x)
    loss = l1(temp,z)
    loss.backward()
    multi_input_history.append(x.grad.data.item())


print(sum(single_input_history)/len(single_input_history))
print(sum(multi_input_history)/len(multi_input_history))
</code></pre>
<p>is my experiment method got right? any suggestions or evaluation about this outcome would be very helpful to me.</p>
BiLSTM training slower on GPU than CPU

JamesGu14: <p>I’m new to Pytorch and I created a typical BiLSTM+CRF sequence labeling model on a typical NER task.</p>
<p>Embedding layer - 100 dim word2vec (Chinese Character).<br>
Hidden dim - 100<br>
LSTMCell - GRU (bidirectional)<br>
dropout - 0.2<br>
optimizer - AdamW<br>
batch size - 128</p>
<p>I’ve tried to train this model on 1080Ti, 2080Ti and my Mac CPU(8 cores). And it turns out Mac CPU training is much faster than than on 1080Ti and 2080Ti. I’m now confused about this, has anyone ever encountered same issue at all?</p>
<p>Speed:<br>
|   | CPU  | 1080Ti / 2080Ti |<br>
|1 epoch | 3’20"  |  18’20" |<br>
| evaluate | 20"  | 200"  |</p>
[SOLVED]What "pil_loader" means?

Petro_Key: <p>Thank you always for your help.<br>
I cannot understand what the following code does.<br>
What is the purpose of this definition?<br>
Thank you in advance.</p>
<p>def pil_loader(path):<br>
with open(path, ‘rb’) as f:<br>
with Image.open(f) as img:<br>
return img.convert(‘RGB’)</p>

alex.veuthey: <p>That looks like a helper function to load an image using the <code>PIL</code> library, and return it as a RGB image. Note that you will need <code>from PIL import Image</code> somewhere above this code for the function to work.</p>

Petro_Key: <p>Thank you for your reply, <a class="mention" href="/u/alex.veuthey">@alex.veuthey</a>,<br>
Your reply was quite understandable and helpful for me!</p>