akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
ReLU activation outputting all negatives

username: aj-ai
cooked: <p>I’ve been having some trouble with my neural net, as it’s outputting all negatives for all 200k test set data points. I know that this isn’t possible with ReLU activation, which is why I’m so concerned.</p>
<p>Here’s my neural net class:</p>
<pre><code>class Net(nn.Module):
   def __init__(self, *, dims:dict):
       super(Net, self).__init__()

       # Layer 1
       self.fc_1 = nn.Linear(in_features = dims['input_dim'], out_features=dims['layer_1'])
       self.actv_1 = nn.ReLU()

       # Layer 2
       self.fc_2 = nn.Linear(in_features = dims['layer_1'], out_features = dims['layer_2'])
       self.actv_2 = nn.ReLU()
       self.bn_2 = nn.BatchNorm1d(num_features = dims['layer_2'])

       # Layer 3
       self.fc_3 = nn.Linear(in_features = dims['layer_2'], out_features = dims['layer_3'])
       self.actv_3 = nn.ReLU()
       self.bn_3 = nn.BatchNorm1d(num_features=dims['layer_3'])

       # initialize
       xavier_normal_(self.fc_1.weight)
       xavier_normal_(self.fc_2.weight)
       xavier_normal_(self.fc_3.weight)

       self.fc_output = nn.Linear(in_features = dims['layer_3'], out_features = dims['output_dim'])

   def forward(self, x):
       x = self.actv_1(self.fc_1(x))
       x = self.actv_2(self.bn_2(self.fc_2(x)))
       x = self.actv_3(self.bn_3(self.fc_3(x)))
       return self.fc_output(x)
</code></pre>
<p>Is this something wrong with my weight initialization? If so, what other options should I use?</p>
<p>Thanks in advance!</p>

username: ptrblck
cooked: <p>Since the last layer (<code>self.fc_output</code>) doesn’t use any activation function, its output values are unbound and can take negative as well as positive values.</p>
<p>If you would like to clip the output to a certain range, you could apply relu again on the last layer or use <code>.clamp(min=0.0)</code>.</p>

username: aj-ai
cooked: <p>Thanks for your response!</p>
<p>I’m also having another issue though - after applying ReLU to the output layer, I’m still getting all 0 values (as expected) as all the predictions from nn.Linear have been negative. Thus my confusion matrix has exclusively true negatives and false negatives, and no real predictions.</p>
<p>Is there something wrong with my model? It doesn’t look like there’s anything wrong but I’m not sure.</p>
<p>Thanks for your time!</p>
How to use smapler

username: n0obcoder
cooked: <p>i know how to make and use a custom dataset. but i am not sure how to make and sue a custom sampler.<br>
can someone pls tell me how a sampler works</p>
<p>for example when i make a custom dataset, i simple write custom_dataset[index] and it outputs the data present at that index. ,and further when i pass this custom_dataset into a batch loader it creates batches of those data. its clear and simple</p>
<p>but in case of a custom sample, how to i get a sample data out of the sampler ?</p>
<p>it would be great if you could me the working of a sampler. Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <p>The sampler returns the indices used to index your <code>Dataset</code>.<br>
Have a look at <a href="https://github.com/pytorch/pytorch/blob/b8c8977be797cb16354aa97c3fcdc910d3e9e5c5/torch/utils/data/sampler.py#L103" rel="nofollow noopener">these implementations</a> or the implemented samplers.<br>
As you can see, the <code>__iter__</code> method returns an iterator or a generator.</p>
<p>How should your custom sampler generate the indices?</p>

username: n0obcoder
cooked: <p>i was going through <a href="https://github.com/adambielski/siamese-triplet" rel="nofollow noopener">https://github.com/adambielski/siamese-triplet</a> and i was not able to figure out whats the use of a batch sampler that has been written in datasets.py</p>
<p>So i wanted to know how and where are these batch samplers are used.</p>
<p>I am still clueless</p>
RuntimeError: Error(s) in loading state_dict for DenseNet:

username: aidlml
cooked: <p>RuntimeError: Error(s) in loading state_dict for DenseNet:<br>
size mismatch for classifier.weight: copying a param with shape torch.Size([1000, 2208]) from checkpoint, the shape in current model is torch.Size([102, 2208]).<br>
size mismatch for classifier.bias: copying a param with shape torch.Size([1000]) from checkpoint, the shape in current model is torch.Size([102]).</p>

username: ptrblck
cooked: <p>The <code>classifier</code> layer was apparently changed before loading the <code>state_dict</code>.<br>
If you want to fine tune the model, load the <code>state_dict</code> before changing the <code>classifier</code>.</p>
Output Binary Array

username: skub
cooked: <p>I am just starting with PyTorch and I can’t seem to figure out how to output an array of 6 binary values or probabilities that do not sum to 1. All of my training data has labels such as [0, 1, 1, 0, 1, 1] for a single entry representing the presence of certain chemical compounds.</p>
<p>I’m just trying to get an output value that is similar to my labels or a collection of probabilities such as [0.0, 0,9, 0.7, 0.1, 0,8, 0.9]. I don’t know what that would be called and I’m sorry I can’t phrase my question well.</p>

username: ptrblck
cooked: <p>It looks like you are dealing with a multi-label calssification, i.e. each sample might have more than a single ground truth class.<br>
If that’s the case, you could use <code>nn.BCEWithLogitsLoss</code> and use an output layer as:</p>
<pre><code class="lang-python">nn.Linear(in_features, nb_classes)
</code></pre>
<p>Each unit in the output layer will be treated as giving the probability for the corresponding class independently.<br>
The target should be a <code>FloatTensor</code> having the same shape as your model output in this case, e.g. <code>[batch_size, nb_classes]</code>.</p>
<p>Let us know, if you need more information or get stuck somewhere.</p>
BN1d vs. BN2d when to use?

username: blackbirdbarber
cooked: <p>Can someone explain when to use <code>BatchNorm1d</code> and when <code>BatchNorm2d</code>?<br>
From <a href="https://pytorch.org/docs/stable/nn.html#batchnorm1d" rel="noopener nofollow ugc">here</a> <strong>BN1d</strong> is called <em>temporal</em>:</p>
<blockquote>
<p>Because the Batch Normalization is done over the C dimension, computing statistics on (N, L) slices, it’s common terminology to call this Temporal Batch Normalization.</p>
</blockquote>
<p>From <a href="https://pytorch.org/docs/stable/nn.html#batchnorm2d" rel="noopener nofollow ugc">here</a> <strong>BN2d</strong> is called <em>spatial</em>:</p>
<blockquote>
<p>Because the Batch Normalization is done over the C dimension, computing statistics on (N, H, W) slices, it’s common terminology to call this Spatial Batch Normalization.</p>
</blockquote>
How to load the prebuilt Resnet models weight ?(or any other prebuilt models)

username: 21fa417e3fb06e56040c
cooked: <p>Can you please let me know how to load the pretrained models using c++ frontend? I am trying to load<br>
the resnet model using C++ front end as follows.</p>
<pre><code>	// replacement for the last layer
	torch::nn::Linear lin(2048, 2); // the last layer of resnet, which you want to replace, has dimensions 512x1000

	std::shared_ptr&lt;torch::jit::script::Module&gt; resnet;
	resnet = torch::jit::load("../../PretrainModel/resnet101.pt");

	torch::optim::Adam optimizer(lin-&gt;parameters(), torch::optim::AdamOptions(starting_lr).weight_decay(1e-4));
	resnet-&gt;to(device);
	lin-&gt;to(device);

	for (size_t epoch = 1; epoch &lt;= kNumberOfEpochs; ++epoch)
	{
		double running_loss = 0;
		double running_corrects = 0;

		lin-&gt;train();

		clock_t s = (int)std::clock();

		for (const auto&amp; batch : *train_loader)
		{
			std::vector&lt;torch::jit::IValue&gt; input;
			auto data = batch.data.to(device), targets = batch.target.to(device);
			input.push_back(data);
			// some training loop
			torch::Tensor out = resnet-&gt;forward(input).toTensor().squeeze();
			out = lin(out);
			auto pred = out.argmax(1);
	
			torch::Tensor loss = torch::nll_loss(out.log_softmax(1), targets);
			optimizer.zero_grad();
			loss.backward();
			optimizer.step();

			running_loss += loss.template item&lt;float&gt;();
			running_corrects += pred.eq(targets).sum().template item&lt;int64_t&gt;();
		}

		clock_t e = (int)std::clock();

		std::printf
		(
			"\rTrain Epoch: %ld [%5d] Loss: %.4f acc: %.4f processing time : %0.3f s",
			epoch,
			train_dataset_size, running_loss / train_dataset_size, running_corrects / train_dataset_size, (float)(e - s) / CLOCKS_PER_SEC);
</code></pre>
<p>}</p>
<p>But I want to create a new model using the values ​​of all the parameters.<br>
How to do it ?</p>
<p>Thank you.</p>

username: yf225
cooked: <p><a class="mention" href="/u/21fa417e3fb06e56040c">@21fa417e3fb06e56040c</a> For your new model, is it defined with TorchScript or with C++ frontend? Or do you just want to clone the existing Resnet model?</p>

username: 21fa417e3fb06e56040c
cooked: <p>This problem has been solved.</p>
PyTorch c++ binaryop runtime type error

username: ycliu
cooked: <p>I was writing pytorch custom op with c++. Two at::Tensor with shape (1, 3) cannot be multiply or add because of the following error.</p>
<pre><code class="lang-auto">RuntimeError: expected type CPUFloatType but got Variable[CPUFloatType] (compute_types at /pytorch/aten/src/ATen/
native/TensorIterator.cpp:134)
frame #0: std::function&lt;std::string ()&gt;::operator()() const + 0x11 (0x7f8fcf5d3021 in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x2a (0x7f8fcf5d28ea in /opt/conda/lib/pyt
hon3.6/site-packages/torch/lib/libc10.so)
frame #2: at::TensorIterator::compute_types() + 0x36b (0x7f8fd016c89b in /opt/conda/lib/python3.6/site-packages/t
orch/lib/libcaffe2.so)
frame #3: at::TensorIterator::Builder::build() + 0x46 (0x7f8fd016e4f6 in /opt/conda/lib/python3.6/site-packages/t
orch/lib/libcaffe2.so)
frame #4: at::TensorIterator::binary_op(at::Tensor&amp;, at::Tensor const&amp;, at::Tensor const&amp;) + 0x2c4 (0x7f8fd016ee6
4 in /opt/conda/lib/python3.6/site-packages/torch/lib/libcaffe2.so)
frame #5: at::native::add_out(at::Tensor&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::Scalar) + 0x71 (0x7f8fd004f
0d1 in /opt/conda/lib/python3.6/site-packages/torch/lib/libcaffe2.so)
frame #6: at::native::add(at::Tensor const&amp;, at::Tensor const&amp;, c10::Scalar) + 0x32 (0x7f8fd004f542 in /opt/conda
/lib/python3.6/site-packages/torch/lib/libcaffe2.so)
</code></pre>
<p>I print out the two tensor via cout and the result is</p>
<pre><code class="lang-auto"> 0
 1
 0
[ CPUFloatType{3} ]
 0
 0
 0
[ Variable[CPUFloatType]{3} ]
</code></pre>
<p>I don’t get the reason why they have different type. If so, how can I turn the type Variable[CPUFloatType]{3} to CPUFloatType?</p>

username: yf225
cooked: <p><a class="mention" href="/u/ycliu">@ycliu</a> Did you use <code>torch::</code> instead of <code>at::</code> for tensor factory functions? Could you share the custom op code?</p>
Making an android app

username: Intel_Novel
cooked: <p>Should I use TensorFlow or Caffe2 to create an app for android. I said let’s use caffe2, since TensorFlow is longer name and I like coffee too.</p>
<p>But when tried the <a href="https://caffe2.ai/docs/mobile-integration.html" rel="nofollow noopener">tutorials</a> I believe  someone is trying to help you to create the apps as hard as possible.</p>
<p>Do I really have to use <a href="https://caffe2.ai/docs/AI-Camera-demo-android.html" rel="nofollow noopener">Android Studio</a> to create an app. Can I make APK in Python?</p>

username: Vandalko
cooked: <p>It’s not a problem to integrate Python into your app, but adding PyTorch or Tensorflow or any other framework is completely different problem.<br>
Mainly because those libraries usually rely on native bindings which are not available for Android.</p>
<p>Speaking about performance, your best choice would be TensorFlow(lite) or any other framework that supports Android Neural Networks API.</p>
<p>Otherwise you would rely on quality of ported code by 3rd-parties. For example, <a href="https://caffe2.ai/docs/mobile-integration.html" rel="noopener nofollow ugc">tutorial</a> you mentioned contains no black magic - they just provide you native (C++) library and it’s up to you to implement all required integrations with your client code.</p>
<p>Also, check this <a href="https://discuss.pytorch.org/t/pytorch-model-running-in-android/27238/4" class="inline-onebox">Pytorch model running in Android - #4 by ptrblck</a></p>
<blockquote>
<p>Do I really have to use <a href="https://caffe2.ai/docs/AI-Camera-demo-android.html" rel="noopener nofollow ugc">Android Studio</a> to create an app. Can I make APK in Python?</p>
</blockquote>
<p>No, you are not required to use Android Studio. Even Android SDK is optional (you may write your own implementation of existing tools - they are open source). But you have to satisfy APK format in one or another way: compile code → convert to DEX → archive everything into APK (which is just ZIP with extra meta info) → zipalign &amp; sign</p>

username: Intel_Novel
cooked: <p>You made an excellent feedback but I will need few days to process that @ <a href="https://discuss.pytorch.org/u/Vandalko">Vandalko</a></p>
Manually add utils.h? (for torch::NoGradGuard)

username: Shern
cooked: <p>Hi,<br>
Is it just me, or do we have to manually add<br>
<code>#include &lt;torch/csrc/api/include/torch/utils.h&gt;</code><br>
within the <code>[path]/libtorch/include/torch/scripts.h</code> file to use <code>torch::NoGradGuard</code> ?</p>
<p>Also, it would be helpful if it was included in the Getting-started code.</p>
<p>Thanks!</p>

username: tom
cooked: <p>I think it might well be possible that you have to add it.</p>
<p>Can you link the getting started code do you have in mind to add it to? Then we can open an issue.</p>
<p>It currently is a bit in the flux with <a href="https://github.com/pytorch/pytorch/issues/13638" rel="nofollow noopener">the Tensor&lt;-&gt;Variable merge</a>.</p>
<p>Best regards</p>
<p>Thomas</p>

username: yf225
cooked: <p>This is fixed on master, and <code>#include &lt;torch/torch.h&gt;</code> is sufficient for using <code>torch::NoGradGuard</code>.</p>
Training deserialization in C++

username: Anthair
cooked: <p>Hi all,</p>
<p>I am training a LNN with pytorch 1.1.0 and using the training to perform an inference on C++.<br>
Due to various compatibility issues with CERN ROOT (long story) I cannot use the C++ torch libraries, so I re-implemented the entire forward process in C++ using matrix algebra.</p>
<p>In order to do so, I wrote a python script to convert the training .pt file in three separate csv files (weights, biases, norm) to be fed to the C++ script. This works fine, but is obviously very annoying.</p>
<p>I want to directly load and deserialize the pytorch .pt model in C++ without using any torch-related library. I had a look at the source code for the load() function and it looks it is based on the pickle library, but I failed in loading the .pt file using pickle.</p>
<p>Could anyone help me understand how the deserialization process is performed in pytorch?</p>

username: tom
cooked: <p>The root of the annoyance is not using libtorch. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I’m assuming here you want state dicts (rather than, say, JIT models).<br>
But so PyTorch’s save function is implemented in <a href="https://github.com/pytorch/pytorch/blob/master/torch/serialization.py" rel="nofollow noopener"><code>torch/serialization.py</code></a> and for storage (Tensor doesn’t store it’s own memory, but delegates to a TensorStorage), these will be deserialized by the code in <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/generic/serialization.cpp" rel="nofollow noopener">torch/csrc/generic/serialization.cpp</a>.</p>
<p>Maybe the quickest way is is to look beyond pure PyTorch and store in some more common format, e.g. form numpy (don’t know if there is good C++ support for those) or hdf5.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Anthair
cooked: <p>Hi <a class="mention" href="/u/tom">@tom</a>, thanks for the reply.</p>
<p>Storing info using another format is more or less my current solution, but I wanted to avoid having two copies of each training file.<br>
I’ll have a look at the code you posted, maybe I can come up with a solution.</p>
<p>Best,<br>
Gabriele</p>

username: yf225
cooked: <p><a class="mention" href="/u/anthair">@Anthair</a> Do you mind sharing the compatibility issues you encountered with CERN ROOT? We might be able to find a solution for it.</p>
How to group bunch of layers together?

username: John_Deterious
cooked: <p>I have a model, it is a bit complicated, and I want parts of it to be grouped under one name and other part, ditto. The reason why I want this is so that I can train the parts individually, specifically, so that I can pass the parameters of that part alone to the optimizer. How can I do that?</p>
<p>To be concrete, here’s a snippet of my code:</p>
<pre><code class="lang-auto">        self.feat1 = Cconv1d(in_channels=30, out_channels=60, kernel_size=2, stride=1, padding=2)
        self.feat2 = Cconv1d(in_channels=60, out_channels=120, kernel_size=3, stride=2, padding=0)
        # self.feat3 = nn.MaxPool1d(kernel_size=4)
        self.featl1 = Clinear(6360, 500)
        self.featl2 = Clinear(500, 31)
        self.featl3 = nn.Softmax(dim=1)
</code></pre>
<p>I want all of those to be under model.part1<br>
PS: I know the Sequntial thing, but I don’t want it because I have special forward between my layers. Any other solution? or  a hacky way to pass parameters of those layers alone to the optimizer?</p>

username: ptrblck
cooked: <p>You could create a custom submodule inside your main module:</p>
<pre><code class="lang-python">class MySubmodule(nn.Module):
    def __init__(self):
        super(MySubmodule, self).__init__()
        self.feat1 = nn.Conv2d(1, 1, 3, 1, 1)
        # ...
        
    def forward(self, x):
        x = self.feat1(x)
        return x
    
class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.part1 = MySubmodule()
        
    def forward(self, x):
        x = self.part1(x)
        return x
</code></pre>
<p>Alternatively, you could just create lists with the necessary parameters and pass them to the corresponding optimizer:</p>
<pre><code class="lang-python">params = list(model.feat1.parameters()) + list(model...)
optimizer = torch.optim.SGD(params, lr=1e-3)
</code></pre>

username: John_Deterious
cooked: <p>I <img src="https://discuss.pytorch.org/images/emoji/apple/heart.png?v=9" title=":heart:" class="emoji" alt=":heart:"> modularity. Thanks.</p>
Initializing a Tensor from a Tensor C++

username: pablovela5620
cooked: <p>I have no idea if this is best practice or not, but I am using Pytorch as an optimization library rather than a Neural Network Library. I have a function in python where I take a previously defined pytorch tensor</p>
<pre><code class="lang-python">R_wrist = torch.eye(3)
</code></pre>
<p>and I need to make a new tensor from it to keep track of its gradients, so in a separate function I create a new tensor based on the one I built</p>
<pre><code class="lang-python">R = torch.tensor(R, requires_grad=True)
</code></pre>
<p>This works for what I need, but now I am trying to convert this code to C++, unfortunatley I cannot convert a Tensor into a Tensor there either through <code>torch::tensor()</code> or <code>torch::from_blob()</code></p>
<p>How would I go about solving this problem. Should I even be creating a new tensor that requires gradient? should I initialize the original tensor to require gradient and avoid it all together? If not should i be converting the first tensor into an array and passing that array into <code>torch::from_blob()</code>?</p>

username: yf225
cooked: <p>In the Python frontend, using <code>torch.tensor()</code> to copy-construct a Tensor from a Tensor is actually deprecated, and the recommended way is <code>sourceTensor.clone().detach().requires_grad_()</code>.</p>
<p>In C++ frontend, currently the best way to achieve the same functionality is:</p>
<pre><code class="lang-cpp">#include &lt;torch/csrc/autograd/variable.h&gt;

torch::Tensor a = torch::eye(3);
torch::Tensor b = torch::autograd::Variable(a.clone()).detach().set_requires_grad(true);
</code></pre>
<p>We are actively working on improving the API to make <code>a.clone().detach().requires_grad_()</code> work soon.</p>
Time per epoch same for pretrained model and training from scratch

username: lorenzo_fabbri
cooked: <p>I’m using ResNet18 for a classification task.<br>
I tried using the pretrained model and training it from scratch: one thing that bothers me is that the time per epoch is more or less the same for both. On my GPU (GTX 1060), it takes roughly 19 minutes if I use the pretrained model, and around 23 minutes if I train it from scratch. For the pretrained model, I just re-initialize the first convolutional layer and the FC layer.<br>
Is this expected?</p>

username: ptrblck
cooked: <p>If you train all layers, it’s obviously expected, since you won’t save any computations.<br>
However, if you freeze some layers, you might save the backward computation, since you don’t need the gradients before a certain layer.</p>
<p>In your use case, it seems you are retraining the input layer, thus the backward pass has to go through the complete model up to the input layer, even if the intermediate layers do not require gradients.</p>

username: lorenzo_fabbri
cooked: <p>Well, yes that makes sense.<br>
Unfortunately my images have 6 channels so the only way I found is to replace the input layer to accept 6 channels. I don’t think it would make sense the initialize it with the weights of ResNet since they are not RGB images.<br>
Thank you.</p>
torch.nn.Conv2d custom filters

username: Naoufel_EL_ABBASSI
cooked: <p>How torch.nn.Conv2d compute the convolution matrix using its default filter.<br>
I have a small problem to know how the calculation is performed and how to use my own filter (mask vector), and why we use unsqueeze from the model.<br>
My code:</p>
<blockquote>
<p>inputconv = torch.randn(3, 4, 4)<br>
inputconv.unsqueeze_(0)<br>
mm = torch.nn.Conv2d(3, 3, kernel_size=2, padding=0, stride=(2, 1))<br>
print(inputconv)<br>
print('Conv2d: ')<br>
print(mm(inputconv))</p>
</blockquote>
<p>output:</p>
<blockquote>
<p>tensor([[[[ 0.8845,  0.2022, -0.8536, -0.5750],<br>
[-0.6650,  0.1512,  1.4356,  0.4598],<br>
[ 0.1666,  1.8639,  0.2500, -0.3754],<br>
[-1.1593,  0.1265,  1.1665, -0.6877]],</p>
<pre><code>     [[-0.3083, -0.7364,  1.6745, -1.6611],
      [ 0.7673, -0.9379,  0.0095, -0.4120],
      [ 0.8867,  0.0865,  0.1563, -0.0828],
      [ 0.8034, -0.6904, -0.4510, -0.6925]],

     [[-0.4779, -0.7025, -0.1098,  0.6809],
      [ 0.3011,  0.3318, -0.7675, -0.8880],
      [ 0.9923, -1.5812, -0.1318,  0.2460],
      [-0.0436, -0.1883, -0.1694,  0.3716]]]])
</code></pre>
<p>Conv2d:<br>
tensor([[[[-0.1268,  0.0031, -0.3460],<br>
[-0.0473, -0.3350, -0.1950]],</p>
<pre><code>     [[ 0.1417, -0.8431, -0.4891],
      [ 0.9959, -0.2300,  0.1318]],

     [[ 0.0118, -0.4037, -0.5443],
      [ 0.3780, -0.5303, -0.3546]]]])
</code></pre>
</blockquote>
<p>My question is how to custom filters using tensor variables: example tf = torch.Tensor([[1, 0, -1], [1, 0, -1], [1, 0, -1]])</p>
<p>Thank you all in advance!</p>

username: ptrblck
cooked: <p>You need to use <code>.unsqueeze_(0)</code> in your example, since the batch dimension is missing for <code>inputconv</code>.<br>
<code>nn.Conv2d</code> expects an input of the shape <code>[batch_size, channels, height, width]</code>.<br>
<code>.unsqueeze(0)</code> adds an additional dimension at position 0, i.e. <code>inputconv</code> will have shape <code>[1, 3, 4, 4]</code>.</p>
<p>You can create a custom filter kernel and apply it using the functional API.<br>
The shape of the filter kernel should be <code>[number_of_filters, input_channels, height, width]</code>.<br>
Here is a small example:</p>
<pre><code class="lang-auto">conv_filter = torch.randn(1, 3, 5, 5)
x = torch.randn(1, 3, 24, 24)
output = F.conv2d(x, conv_filter, padding=2)
</code></pre>

username: Naoufel_EL_ABBASSI
cooked: <p>I tried with small matrix and custom conv_filter, it works fine.</p>
<blockquote>
<p>conv_filter = torch.Tensor([[[[1, 0, 0]], [[0, 1, 0]], [[0, 0, 1]]]])<br>
x = torch.randn(1, 3, 5, 4)<br>
output = F.conv2d(x, conv_filter, padding=0)</p>
</blockquote>
<p>Than you for your help</p>

username: Shisho_Sama
cooked: <p>What if instead of a random initial weigh, the filter is created using a custom function?<br>
suppose there are two parameters(variables) that are responsible for producing an entry in the weight matrix (2 for each entry for example). Something like <a href="https://discuss.pytorch.org/t/why-doesnt-these-variables-get-updated/51187">this</a> .<br>
how should we go about such cases?</p>
How to limit the cpu kernel usage?

username: coincheung
cooked: <p>Hi,</p>
<p>Our server has 56 cpu cores, but when I use the dataloader with num_workers=0, it took all the cpu cores. From htop, I see that all cpu cores works with workload of 100%. What is the cause of this, and how could I confine the cpu usage to a few cpu cores?</p>
<p>Thanks,<br>
CoinCheung</p>

username: Tony-Y
cooked: <aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/16894" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <a href="https://github.com/mahmoodn" rel="nofollow noopener">
<img src="https://avatars1.githubusercontent.com/u/11626212?v=2&amp;s=96" class="thumbnail onebox-avatar" width="60" height="60">
</a>

<h4><a href="https://github.com/pytorch/pytorch/issues/16894" target="_blank" rel="nofollow noopener">Issue: Number of CPU threads for the python process</a></h4>

<div class="date" style="margin-top:10px;">
	<div class="user" style="margin-top:10px;">
	opened by <a href="https://github.com/mahmoodn" target="_blank" rel="nofollow noopener">mahmoodn</a>
	on <a href="https://github.com/pytorch/pytorch/issues/16894" target="_blank" rel="nofollow noopener">2019-02-08</a>
	</div>
	<div class="user">
	</div>
</div>

<pre class="content" style="white-space: pre-wrap;">For a test, I didn't use --cuda in order to run a cpu version. While the CPU has 8 physical cores...</pre>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>This might be related to your problem.</p>

username: coincheung
cooked: <p>Thanks, but there is one thing that I cannot explain. I have set my dataloader worker number to be 0, and other operations are done with gpu. Why will there still be many threads if no constraints are set? What are these threads doing?</p>

username: Tony-Y
cooked: <p>Did you check all operations working on gpu?</p>
How can torchvison.models deal with image whose size is not 224*224?

username: 681408ed5ec9ed11af98
cooked: <p>Hi!<br>
RestNet, DenseNet and other model in torchvison.model are all designed for image of size 224x224?<br>
How can I use it to other image tasks with image size not 224x224?.<br>
These days I’m trying to use torchvison.models for cifar10 classificaion task, and I already try some methods to overcome the image size mismatch problem：</p>
<ul>
<li>resize the image size to 224x224. but i found that it will cost more times for training than using a image size of 32x32.</li>
<li>In ResNet model, Ichange the avgpool to adaptiveavgpool, like that <code>model.avgpool = t.nn.AdaptiveAvgPool2d(1).</code> But I found that it does’t obtain a good performance, the best validation acc is just 84%. I found somebody can reach 92%. So whats’s the problem with this method?</li>
</ul>
<p>So, I want to know other better methods to deal with the problem! I’ll be really grateful for any advice or help.</p>

username: ptrblck
cooked: <p>A lot of pretrained models will use the image size of <code>224x224</code>, so you might want to use a custom architecture for smaller images (or resize them as you’ve already done).</p>
<p>The <a href="https://github.com/pytorch/vision/blob/c187c2b12d86c3909e59a40dbe49555d85b98703/torchvision/models/resnet.py#L151" rel="nofollow noopener">current resnet implementation</a> already uses an adaptive pooling layer, so your change shouldn’t make a difference regarding the performance.</p>

username: 681408ed5ec9ed11af98
cooked: <p>Thank you for you advice!<br>
In addition，I found that by resizing image size and then use the pretrained resnet18 it can achieve about 94% accuracy in cifar10. But using a adaptive pooling layer the best performance is only 84%.  So why they are so different?</p>

username: justusschock
cooked: <p>Simply spoken:</p>
<p>The difference here is, that the kernels have been trained to a specific image size (224) and thus the features they usually extract are usually of a certain size.</p>
<p>If you use smaller images, the kernels might not be able to extract the features with the usual size, since they are smaller (ore larger), which may result in a difference in performance.</p>
<p>If your input size varies to much from 224x224 pixels the results may even become much worse.</p>

username: 681408ed5ec9ed11af98
cooked: <p>Thank you very much! I understand what u said.</p>
Unable to finetune pretrained inception_v3 in multi-gpu training

username: 111104
cooked: <p>PyTorch Version:  1.1.0<br>
Torchvision Version:  0.3.0</p>
<p>I’m trying to finetune inception_v3 these days but meet a bug:</p>
<blockquote>
<p>Blockquote<br>
Traceback (most recent call last):<br>
File “train.py”, line 133, in <br>
outputs, aux = model(images)<br>
File “/home/yjh/.conda/envs/phone/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 493, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/home/yjh/.conda/envs/phone/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py”, line 153, in forward<br>
return self.gather(outputs, self.output_device)<br>
File “/home/yjh/.conda/envs/phone/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py”, line 165, in gather<br>
return gather(outputs, output_device, dim=self.dim)<br>
File “/home/yjh/.conda/envs/phone/lib/python3.7/site-packages/torch/nn/parallel/scatter_gather.py”, line 67, in gather<br>
return gather_map(outputs)<br>
File “/home/yjh/.conda/envs/phone/lib/python3.7/site-packages/torch/nn/parallel/scatter_gather.py”, line 62, in gather_map<br>
return type(out)(map(gather_map, zip(*outputs)))<br>
TypeError: <strong>new</strong>() missing 1 required positional argument: ‘aux_logits’</p>
</blockquote>
<p>Finally, I figure out that there may be something wrong when the model is distributed to multi-gpu by model = nn.DataParallel(model), after commenting this line, everything seems ok and it began training.<br>
However, I could not utilize the other gpu, which increase my time cost.</p>
<p>Is there bugs or I didn’t use it correctly?</p>
Problem in an ensemble of subnets

username: t.ouyang
cooked: <p>Hi, I am a beginner with Pytorch, and the following code confuses me a lot. <img src="https://discuss.pytorch.org/images/emoji/apple/cry.png?v=9" title=":cry:" class="emoji" alt=":cry:"></p>
<p>Instead of using two seperate sub-models with one optimizer/scheduler each, I want to use  an ensemble of these two sub-models, <em>so that</em> I only need to write <code>mymodel.train()/eval()</code>, <code>mymodel.cuda()/cpu()</code> or <code>torch.save(mymodel, 'mymodel.pt')</code> rather than two copies of those for two sub-models. Besides, I only need one optimizer/scheduler. Here is the demo code:</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self, subnet1, subnet2):
        super(MyModel, self).__init__()
        self.subnet1 = subnet1
        self.subnet2 = subnet2
    def loss(self, sample):
        # calculating the loss using self.subnet1 and self.subnet2
        ......
        return loss_tensor

mymodel = MyModel(net1, net2)
optimizer = torch.optim.Adam(mymodel.parameters(), lr=LR)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=step_size, gamma=gamma)
</code></pre>
<p>There is no forward method in <code>MyModel</code> class since <code>subnet1</code> and <code>subnet2</code> are <code>nn.Module</code> with a forward method, And I use the <code>loss.backward()</code> and optimizer/scheduler to train my ensemble.</p>
<p>BUT, I found the accuracy is ~8% lower than the seperate two-sub-model version on the same ground (the latter one with two optimizers/schedulers). Is there sth I’ve missed? or just the bad luck for the degradation. <img src="https://discuss.pytorch.org/images/emoji/apple/cry.png?v=9" title=":cry:" class="emoji" alt=":cry:"></p>
<p>Furthermore, Is there any available ways to integrate my sub-models? <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>

username: tom
cooked: <p>I think the point of ensemble models is to train them separately, otherwise you’ll (at least morally) get a single “twice as wide network with little interaction between two halves”  model.</p>
<p>Best regards</p>
<p>Thomas</p>

username: t.ouyang
cooked: <p>Thank you very much, Thomas. <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>
Efficient detached vector Jacobian product

username: sidms
cooked: <p>I am trying to compute input gradients of a network in such a way that effectively the Jacobian of the network is detached before its multiplied with the grad_outputs. The below code shows what I’m trying to achieve but is horribly inefficient due to the loop. Here <code>instance_losses</code> is a vector that contains the loss per instance in the batch.</p>
<pre><code class="lang-auto">params = self.get_params(model)
instance_loss_grads = torch.autograd.grad(loss, [instance_losses], create_graph=True)[0]  # dims: [batch]

# EXPENSIVE OPERATION: Compute Jacobian wrt training instances.
instance_param_grads = [torch.autograd.grad(instance_loss, params, retain_graph=True) for instance_loss in instance_losses]  # type: list(list(params))

param_grads = []
for i in range(len(params)):  # for each param group
    # Create Jacobian of param group wrt training instances
    # Note the `gs[i].detach()`. This is needed and is what makes the problem complex.
    group_jacobian = torch.stack([gs[i].detach() for gs in instance_param_grads], dim=0)  # dims: [batch, ...]

    # Multiply differentiable `instance_loss_grads` and constant Jacobian. 
    param_grads.append(torch.einsum('i,ij...-&gt;j...', instance_loss_grads, group_jacobian))

</code></pre>
<p>I see two potential options:</p>
<ul>
<li>Find a way to tell <code>torch.autograd.grad</code> that all operations during backprop must be performed after detaching.</li>
<li>Find a way to perform efficient computation of Jacobian. This has been discussed many times over PyTorch forums with no general solution.</li>
</ul>
<p>Any suggestions on how this can be optimized?</p>

username: Chi_Yu
cooked: <p>do you use wechat?<br>
my wechat is:appleyuchi.<br>
I also work on how to compute hessian of neural network<br>
which is similar to yours.</p>
Custom loss error [grad can be implicitly created only for scalar outputs]

username: gt_tugsuu
cooked: <p>Hey, guys.<br>
I could not understand why this code is giving an error. Please anyone help?<br>
It is not giving error when i use <code>nn.MSELoss()</code>.</p>
<pre><code class="lang-auto">for epoch in range(1):
    for i, (A,B,labelA,labelB) in enumerate(training_generator):
        A = Variable(A.unsqueeze(0).float())
        B = Variable(B.unsqueeze(0).float())
        label = Variable((labelA==labelB).float())
        A.requires_grad=True
        B.requires_grad=True
        label.requires_grad=True
        
        
        pha, phb = model_lstm((A,B))
        pha = pha.squeeze()
        phb = phb.squeeze()
        
        out_warper = model_warper(torch.cat((pha,phb),1))
        
        Sab = torch.abs(pha-phb).mean(1).unsqueeze(1)
        Sab = torch.exp(-Sab*out_warper)
        
        
        loss = label*nn.Sigmoid()(Sab) + (1-label)*nn.Sigmoid()(1-Sab)
        
        
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
</code></pre>
<p>Error:</p>
<blockquote>
<p>RuntimeError: grad can be implicitly created only for scalar outputs</p>
</blockquote>

username: tom
cooked: <p>Try using sum or mean to get a single number. (Print the loss, too)</p>

username: gt_tugsuu
cooked: <p>Thanks, somehow it worked using loss = nn.BCELoss since Sab is already single number.</p>
Layer Reshape Issue

username: shivangi
cooked: <p>I have dynamically created my network, but I am facing an issue when I try to Reshape my layer from Linear back to Convolutional.</p>
<p>I am trying to work with MNIST dataset.</p>
<p>This is my network:<br>
(0): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(1): ReLU()<br>
(2): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=1, ceil_mode=False)<br>
(3): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(4): ReLU()<br>
(5): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=1, ceil_mode=False)<br>
(6): Flatten()<br>
(7): Linear(in_features=3136, out_features=32, bias=True)<br>
(8): ReLU()<br>
(9): Linear(in_features=32, out_features=3136, bias=True)<br>
(10): ReLU()<br>
(11): ReshapeLayer()<br>
(12): MaxUnpool2d(kernel_size=(2, 2), stride=(2, 2), padding=(0, 0))<br>
(13): ConvTranspose2d(64, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(14): ReLU()<br>
(15): MaxUnpool2d(kernel_size=(2, 2), stride=(2, 2), padding=(0, 0))<br>
(16): ConvTranspose2d(32, 1, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(17): ReLU()</p>
<p>I tried to print the ouput shapes.<br>
And this is the error that I am getting:<br>
torch.Size([50, 32, 28, 28])<br>
torch.Size([50, 32, 28, 28])<br>
torch.Size([50, 32, 14, 14])<br>
torch.Size([50, 64, 14, 14])<br>
torch.Size([50, 64, 14, 14])<br>
torch.Size([50, 64, 7, 7])<br>
torch.Size([50, 3136])<br>
torch.Size([50, 32])<br>
torch.Size([50, 32])<br>
torch.Size([50, 3136])<br>
torch.Size([50, 3136])<br>
torch.Size([50, 64, 7, 7])<br>
x = layer(x)<br>
File “SomePath/python3.6/site-packages/torch/nn/modules/module.py”, line 491, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
TypeError: forward() missing 1 required positional argument: ‘indices’</p>
<p>And this is how my Custom Layers look like:<br>
class ReshapeLayer(nn.Module):</p>
<pre><code>def __init__(self,batch_size, n_channels, height, width):
    super(ReshapeLayer, self).__init__()
    self.shape = [batch_size, n_channels, height, width]

def forward(self, x):
    a = x.view(self.shape)
    return a
</code></pre>
<p>class Flatten(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Flatten, self).<strong>init</strong>()</p>
<pre><code>def forward(self, x):
    x = x.view(x.size()[0], -1)
    return x</code></pre>

username: royboy
cooked: <p>MaxUnpool2d requires an <code>indices</code> argument, you can get this from your MaxPool2d layers by specifying <code>return_indices=True</code>.</p>

username: shivangi
cooked: <p>How exactly can you retrieve these indices in MaxUnpooling layer?<br>
Can you explain with example</p>

username: royboy
cooked: <p>Here’s a minimal example of pool and unpool:</p>
<pre><code class="lang-auto">import torch
from torch import nn

x = torch.rand(1, 3, 540, 960)
pool = nn.MaxPool2d(2, 2, return_indices=True)
unpool = nn.MaxUnpool2d(2, 2)

out, indices = pool(x)
out = unpool(out, indices)
</code></pre>

username: TaihuLight
cooked: <p>How to print the output shapes of each layer for your net? Cloud you share you code?</p>
Use multiprocessing to parallel models

username: souxun
cooked: <p>I have 4 gpus in my machine. I want to use multiprocessing to creat 4 processing to train 4 models separately. Each processing will train 1 model in 1 GPU. The code is as follows:</p>
<pre><code class="lang-auto">    import multiprocessing as mp
    p_random = mp.Pool(processes=4) 
    process = []
    for epoch in range(1, args.epochs):
        sp = epoch % 4 # assign gpu
        t = p_random.apply_async(train, (epoch, net_cluster['model{}'.format(sp)]))
        process.append(t)
</code></pre>
<p>But when I moniter the information of gpus, each gpu creat 4 processes. Technically, I think each process use one specific gpu, so each gpu should have 1 process.<br>
I am confusing about that.  Is there anything wrong in my code?</p>

username: Tony-Y
cooked: <p><a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/multiprocessing.html</a></p>
<p>This document might help you.</p>

username: souxun
cooked: <p>Thanks for that. I have checked this page. But I didn’t find the answer.</p>
<p>When I use the pool to create 4 processes, and I assign each process to one gpu, but finally I get 4 processes in each gpu. I am confusing about that.</p>

username: Tony-Y
cooked: <aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/pytorch/examples/tree/master/mnist_hogwild" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/21003710?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/pytorch/examples/tree/master/mnist_hogwild" target="_blank" rel="nofollow noopener">pytorch/examples</a></h3>

<p>A set of examples around pytorch in Vision, Text, Reinforcement Learning, etc. - pytorch/examples</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Could this example code work on your computer as expected?</p>
Sequential results

username: Jaideep_Valani
cooked: <p>Hi I two models<br>
One dense net and one Efficient net…</p>
<p>when i do sequential on both<br>
body =nn.Sequential(*list(model.children())[:-1])<br>
in this dict keys are preserved like conv0.,Relu0 etc<br>
but in below dict keys get lost and converted into the 0 ,1,2  why is difference…<br>
body1=nn.Sequential(*list(md_ef.children())[:-1])<br>
Sequential(<br>
(0): Conv2dStaticSamePadding(<br>
3, 48, kernel_size=(3, 3), stride=(2, 2), bias=False<br>
(static_padding): ZeroPad2d(padding=(0, 1, 0, 1), value=0.0)<br>
)<br>
(1): BatchNorm2d(48, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)<br>
(2): ModuleList(</p>

username: tom
cooked: <p>I don’t think that that is actually the case, could it be that the outermost sequential (which is the only bit you are changing) is just short for the first?<br>
You would need to show more of what you are doing to get help, preferably enclosing the code in triple backticks (```).</p>
<p>Best regards</p>
<p>Thomas</p>
How to train a generator and two discriminators simultaneously - How to relate second discriminator?

username: Nikronic
cooked: <p>Hi everyone,</p>
<p>I want to train a GAN which has a generator and two discriminators and the discriminators have different structures but the same loss function. But I am a little confused how to deal with the second discriminator.</p>
<p>Here is what I have done briefly:</p>
<pre><code class="lang-python">x,y = # input and ground truth

details_net.train() # generator
disc_one.train()
disc_two.train()

# train generator
details_optim.zero_grad()
gen_imgs = details_net(x)
g_loss = criterion(disc_one(gen_imgs), valid)
g_loss.backward(retain_graph=True)
details_optim.step()


# train discriminator one
disc_one_optim.zero_grad()
real_loss = criterion(disc_one(y), valid)
fake_loss = criterion(disc_one(gen_imgs), fake)
disc_one_loss = (real_loss + fake_loss) / 2
disc_one_loss.backward(retain_graph=True)
disc_one_optim.step()


# train discriminator two
disc_two_optim.zero_grad()
real_loss = criterion(disc_two(y, valid)
fake_loss = criterion(gen_imgs, fake)
disc_two_loss = (real_loss + fake_loss) / 2
disc_two_loss.backward()
disc_two_optim.step()
</code></pre>
<p>The above code is a the <code>DataLoader</code> loop.<br>
I am using <code>nn.MSE</code> loss function for both discriminators and generators.</p>
<p>And my question is, am I using <code>.backward</code> and <code>criterion</code> correctly?</p>
<p>Thank you for your help</p>

username: tom
cooked: <ol>
<li>You could do<code>for p in XXX.parameters(): p.requires_grad_(False)</code> for the bits you are not training and set them to True for the bits you are training. Most GAN examples do that, it saves getting and storing unneeded gradients.</li>
<li>You could have g_loss1 and g_loss2 for each discriminator and <code>g_loss = alpha * g_loss1 + beta * g_loss2</code>. To mix both criteria.</li>
<li>You likely don’t need <code>retain_graph=True</code> in the g_loss.backward if you just detach (after training the generator) or regenerate the fake image. A while ago, most GANs had  several disc steps per generator step. I would not know if that changed. So it would be re-generating mostly.</li>
<li>Instead of <code>retain_graph=True</code>  in the disc1 training, you might just add disc_one_loss and disc_two loss and do a single backward.</li>
<li>You didn’t ask about that, but it is very unlikely that you want <code>disc_..._loss = real_loss + fake_loss</code>.<br>
If the generator minimizes criterion, then you probably want it to be <code>real_loss - fake_loss</code>, so that the discriminators push up the criterion for fake images.</li>
</ol>
<p>Best regards</p>
<p>Thomas</p>
How to interpolate 2D image with zero value?

username: 591ece3a2a04495180b7
cooked: <p>Hi, I want to interpolate 2D image with some of missing values.</p>
<p>e.g)<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/9/96f5c001a4868c6f9d916eae31025fce0a0de5b9.png" alt="image" data-base62-sha1="lxs8wOfSZnZAYwLp9TCW2NLAGg1" width="529" height="434"></p>
<p>and I have to replace the value ‘0’ with the proper one, which should be computed by bilinear interpolation using 4 neighbor pixels.</p>
<p>I found the function which can do this in scipy.interpolate but I coudln’t find it in Pytorch.</p>
<p>Can I do this job on GPU using pytorch function?</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>Here is the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.functional.interpolate" rel="nofollow noopener">interpolate</a> function you are looking for. This function is available on CUDA.</p>

username: 591ece3a2a04495180b7
cooked: <p>I knew it but I think that function is only available for up/downsampling, not replacing the zero value. Isn’t it?</p>