akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to convert neural network for solvers like Microsoft Z3 or any SAT solvers?

eso: <p>I have seen many papers where they verify neural networks. Either they try to check adversarial robustness or finds an input that the model predicts incorrectly. They try to convert a layer to a mathematical equation for eg a boolean expression. And then solves it by available solvers (Microsoft Z3, Gurobi). But I could not find any tutorial or starting code.</p>
<p>Can you help me start the process? For example, share a code to convert one fully connected layer with ReLu activation for MNIST data. Then I can build on top of that for a more complex layer.</p>
<p>Thanks</p>
Can pytorch and libtorch share identical name libraries in same machine?

Alex_Luya: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e943c537f5c370c1d1dd3f8eadcacff6bdbfa90d.png" data-download-href="https://discuss.pytorch.org/uploads/default/e943c537f5c370c1d1dd3f8eadcacff6bdbfa90d" title="Workspace 1_999(005)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e943c537f5c370c1d1dd3f8eadcacff6bdbfa90d_2_690x250.png" alt="Workspace 1_999(005)" data-base62-sha1="xhyrM5MACk7Xgs2vvU3PXUu1SgZ" width="690" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e943c537f5c370c1d1dd3f8eadcacff6bdbfa90d_2_690x250.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e943c537f5c370c1d1dd3f8eadcacff6bdbfa90d_2_1035x375.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/9/e943c537f5c370c1d1dd3f8eadcacff6bdbfa90d.png 2x" data-dominant-color="EFEAEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Workspace 1_999(005)</span><span class="informations">1114×404 78.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Hello,as you can see that they both have two large identical name .so:</p>
<p>libtorch_cuda.so<br>
libtorch_cpu.so</p>
<p>Can I delete one side’s .so,and make a soft link from another side?</p>

albanD: <p>Hey,</p>
<p>I think this is one of the goal of splitting these but I’m not sure this is possible yet. <code>libtorch_python.so</code> should contain the python specific stuff.<br>
Note that you will need the exact same versions for this to work though.</p>
<p>You can try and see if it works. But this is not guaranteed I’m afraid <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
Despite convergence of the cost, predictions are far from result and seemingly noisy. Why?

Capo_Mestre: <p>Hello! I am trying to train a DenseNet, which I defined <a href="https://discuss.pytorch.org/t/runtime-error-does-not-make-sense/91997/3">here</a><br>
The cost functions seem to converge for both train and dev sets as shown in this snapshot (cost function for dev set is not normalized):<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/0/308155837a5b190012bf3c1ccbb1a219e5d0d98d.png" alt="Capture" data-base62-sha1="6V60Xv2gUJA5rCstWbdJJy04ps1" width="450" height="271"><br>
However, when I plot together the actual label (which is a vector of 128 numbers) and the predicted label, the result looks like this:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f03afe3e9fb508b8f8590d507ad0bbaae22d87fd.png" alt="Capture1" data-base62-sha1="yhaZ4QZ71Y1cH7cs2JrX6bgQt6l" width="339" height="240"><br>
Would somebody have had a similar experience and/or have some ideas what to check and what to try?</p>
Time Overhead in torchvision.datasets.ImageFolder

Prateek_Varshney: <p>Hi. I was experimenting with torchvision datasets and model architectures. I seem to be incurring a very large time overhead when running torchvision.models.Resnet18 for one epoch on ImageNet Dataset:</p>
<pre><code class="lang-auto">Dataloader_time:  [6.656000018119811e-05]
TrainD_time:  [7.61386392974854]
train_stat_time:  [46.41145703125]
stat_time1:  [0.24797286987304687]
stat_time2:  [0.0009789440035820008]
validation_time:  [27.89709765625]
epoch_time:  [164.072703125]
</code></pre>
<p>Here</p>
<ol>
<li>Train_stat time is running the model on entire training dataset after each epoch to compute performance metrics on training_dataset (such as accuracy/loss)</li>
<li>stat_time1 is for computing class wise metrics on Training Dataset and stat_time 2 is for computing class wise metrics on Validation Dataset.</li>
<li>For debugging, I had downsampled the Training set to 12.8K and Validation set to 6.4K samples.</li>
</ol>
<p>There are 2 major issues here:</p>
<ol>
<li>Individual Times are not adding up to the epoch time</li>
<li>The Validation/Train_Stat Time are much more than Training Time</li>
</ol>
<p>For comparison, running the same code on the entire torchvision.datasets.CIFAR10 gives the following results:</p>
<pre><code class="lang-auto">Dataloader_time:  [0.00016486400365829467]
TrainD_time:  [17.981506515502918]
train_stat_time:  [7.0972744140625]
stat_time1:  [0.0033064959049224855]
stat_time2:  [0.000319487988948822]
validation_time:  [0.4533729248046875]
epoch_time:  [25.934220703125]
</code></pre>
<p>As evident, here Training Time is the major component and the numbers seem to add up.</p>
<p>The only difference between the two instances is that the previous one uses <code>datasets.ImageFolder</code> to create the training/validation datasets and the latter uses <code>datasets.CIFAR10</code>.</p>
<p>I have verified that the time issue does not stem from image size differences in the two datasets by using the same image size crop (32 x 32) on both. (For reference, ImageNet has 256 x 256 as compared to 32 x 32 in CIFAR10/100 after standard transformations)</p>
<p>Any help would be deeply appreciated!!</p>
torch.jit.SciptedFunction handle in C++

fatvlad: <p>Hello!<br>
There are many tutorials on saving models into <code>.pt</code> file and loading them in C++. However I haven’t found any hint in documentation/tutorials on how to pass traced model through pybind (probably it is not a coincidence <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">). However I can see that traced/scripted model class <code>torch.jit.ScriptedModel</code> has pybind11’s metaclass, so there’s probably a bound class in C++ which will allow to write function taking object of that type as an input. I haven’t found it yet, maybe somebody already knows?</p>

fatvlad: <p>If anyone else is interested, C++ class that is bound to <code>torch.jit.ScriptedModel</code> is <code>torch::jit::StrongFunctionPtr</code>. But using binding directly is not advised because you will need ABI-compatible version of pybind11 and there’s a chance that some binding make use of private symbols.</p>
Why do white holes appear when learning images end-to-end?

yhkwon: <p>Hi,</p>
<p>I am learning image reconstruction end-to-end</p>
<p>However, if you look at the validation result during the learning process, as shown in the figure below, holes often occur in the image and appear as artifacts even when learning progresses.<br>
<a href="/uploads/short-url/4HNigoWLjjh6yvYintPCMo6NUee.png">1|138x201</a><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/0/20fc0ce85a3d92506e9a31d93334ac14f535292a.png" alt="1" data-base62-sha1="4HNigoWLjjh6yvYintPCMo6NUee" width="138" height="201"></p>
<p>I’m using cnn and instance normalization and lrelu, but I’m not sure what the problem is.<br>
L1-loss, perceptual loss, and adversarial loss are being used, and even if the weight of the l1 loss is greatly increased, the same result is obtained.</p>
<p>Have you experienced similar experiences and overcame any examples?</p>
<p>Thanks!</p>

Can_Keles: <p>Hey, I have encountered a similar problem when I worked on an image colorization project. Some parts of the image were black and other parts seemed fine. I solved the problem by experimenting with different network architectures. The network on pix2pix paper worked fine if I remember correctly. Best of luck!</p>

yhkwon: <p>Thanks for the reply!!<br>
Network changes may be one solution, but I am wondering what is causing the problem.</p>

yhkwon: <p>I solved the above problem by changing the transpose convolution normalization method.</p>
<p>If anyone faced a similar problem with me, it would be better to eliminate normalization or use layer norm.</p>
Fine Tuning Network with New Dataset

eprox: <p>Hi,<br>
I am training a neural network on task A using a set of word embeddings, I would like to re-use the model for task B, using a different set of word embeddings.</p>
<p>I have a neural network trained for task A using a set of word embeddings, for which I save the model state dict for and optimizer state dict:</p>
<pre><code class="lang-auto">...
torch.save(self.best_opt_wts)
torch.save(self.best_model_wts)
...
</code></pre>
<p>Next, I want to reload the model to train for task B (transfer learning), with a new set of input embeddings.<br>
I first set up the model using the set of new embeddings:</p>
<pre><code class="lang-auto">class RNN(nn.Module):
	def __init__(self, options):
		...
		self.embedding = nn.Embedding.from_pretrained(torch.load(options['embeddings_loc']))
</code></pre>
<p>Then later in my training script, I attempt to load the state dict into the model:</p>
<pre><code class="lang-auto">self.model.load_state_dict(torch.load(self.args.ft_model))
</code></pre>
<p>However, I get this error:</p>
<pre><code class="lang-auto">size mismatch for input_encoder.embedding.weight: copying a param with shape torch.Size([75323, 300]) from checkpoint, the shape in current model is torch.Size([16694, 300])
</code></pre>
<p>It looks like since the encoder needs to be reshaped somehow to be used with the new embeddings.<br>
What is the best way to do this to preserve the previous model?<br>
Is this a valid strategy for fine-tuning a model of Task A for Task B?</p>
How to create reusable framework to visualize internal model calculations

jvhoffbauer: <p>We want to visualize internal model calculations, e.g. attention maps, in a research project with multiple different models. The goal is to create a common framework for these visualizations with the following constraints:</p>
<ul>
<li>they should only be calculated and stored during validation to save computation time during training</li>
<li>the interface should be common for all models (i.e. PyTorch modules with a forward function)</li>
<li>it should be possible to automatically visualize the generated images in tensorboard</li>
<li>it should work (e.g. correct naming of generated images) with modules calling other submodules</li>
</ul>
<p>I am wondering what is the best way to implement this (i.e. the most usable and generic way). Basically I had the idea of passing a dictionary to the module into which the module can write, depending on it’s current mode.</p>
<p>Did you already encounter such a problem? If yes how did you implement this?</p>
<p>Thanks!<br>
Jan</p>

ptrblck: <p>I don’t know, if this would exactly fit your use case, but have a look at <a href="https://captum.ai/">Captum</a>, which provides a variety or methods for model interpretability.</p>
<p>CC <a class="mention" href="/u/narine_kokhlikyan">@Narine_Kokhlikyan</a></p>

jvhoffbauer: <p>Thank you for the reply!</p>
<p>I was more asking for general device on how to design such a framework. I would be very happy, if you could provide some ideas on how you wold design this?</p>
<p>Thanks, Jan</p>
Accessing weight corresponding to a single element in a batch

Eugene_Lee: <p>Hi there,</p>
<p>I’ve not dig deep into the PyTorch codebase but I assume that in order to perform batch inference/training, weights are duplicated and loaded into the GPU to perform parallel inference depending on the batch size. Let’s say I’d like to access the weights corresponding to every element in a batch, i.e. my model is an adaptive one and I’d like to modify the duplicated weights independently w.r.t. to the current input. What’s the best way to do this?</p>
<p>Thanks!</p>

ptrblck: <p>The parameters are not duplicated and batched operations are used instead, i.e. each input sample will use the same parameters and the output will be a batch again.</p>

Eugene_Lee: <p>I see, thanks! So it’s impossible to have multiple set of duplicated weights for a given batch of inputs on the current version of PyTorch.<br>
I assume it’s technically feasible, since weights will be loaded into separate CUDA cores prior to concurrent processing of different inputs. Am I right on this part?</p>

ptrblck: <p>I don’t fully understand why you would like to duplicate the weights.<br>
It would surely be possible to clone the weights and apply them one-by-one for each input sample, but this is usually not necessary, so could you explain your use case a bit more?</p>

Eugene_Lee: <p>It’s a new idea that I’m trying to implement. It consist of the adaptation of weights w.r.t. the inputs. Since a batch could consist of inputs of different nature, i.e. different class in the case of classification, the adaptation of weights would results in the multiple set of weights for a single batch. It’s a novel method that I’m working on. A workaround is to restrict a batch to contain inputs that are closely distributed.</p>

ptrblck: <p>Thanks for the explanation.<br>
In that case you could probably either clone the parameter or initialize multiple layers for the current input e.g. via <code>nn.ModuleList</code> and select the desired one.</p>

Eugene_Lee: <p>Thanks for the reply, this is one way to do it, but it will result in linear growth in memory according to the batch size, e.g. batch size of 128 will result in 128 different layers/modules stored. In the meantime, I’ll think of a more general approach that doesn’t require the replication of weights.</p>
Sparse Matrix status

tueboesen: <p>Hi</p>
<p>I’m trying to find any up to date information about the status on Pytorch’s sparse matrix capabilities, but I can’t seem to find anything recent.</p>
<p>Does anyone have any estimated timeline for when the sparse module would be operational?</p>
<p>More concretely for my particular issue:<br>
I’m doing research in machine learning, and we are using pytorch, but unfortunately the lack of sparse matrix module is a huge hindrance for the networks we are producing.<br>
Thus far I have managed to hack my own sparse matrix multiplication together when one of the matrices is a diagonal matrix. (It isn’t fast and neither is it pretty, but it does work), and combined with the matrix multiplication of (sparse,dense) I can actually compute all the operations I need for the network I’m making. However when I run my network and do .backward() I encounter “RuntimeError: calculating the gradient of a sparse Tensor argument to mm is not supported.”</p>

zcajiayin: <p>With pytorch 1.6, now you can call <code>torch.sparse.mm(A, B)</code> where <code>A</code> is the sparse matrix and <code>B</code> is a dense matrix (I think it also work with sparse <code>B</code>).</p>
Model produce nothing

Aryan_Asadian: <p>Hi. I have created a multi-module model by using add_module method.<br>
First I defined an empty class model:</p>
<pre><code class="lang-auto">class EmptyModel(nn.Module):
    def __init__(self):
        super(EmptyModel, self).__init__()

    def forward(self, x):
        return x

empty_model = EmptyModel()
</code></pre>
<p>Then I add modules from another previously defined model:</p>
<pre><code class="lang-auto">empty_model.add_module("stage_1",vgg_8_sub_student)
empty_model.stage_1.eval()
empty_model.add_module("stage_2",nn.Sequential(*list(vgg_8_model.features)[7:]))
empty_model.add_module("classifier",nn.Sequential(vgg_8_model.classifier))
</code></pre>
<p>As you can see, the first module is already trained so I set it to eval().<br>
When I use torchsummary to visualize each of these modules, everything works fine. They work properly, But when I try to visualize the whole model, I get an error:</p>
<pre><code class="lang-auto">  summary(student_partially_trained_from_stage_1,input_size=(3,224,224),device="cpu")
  File "/home/aasadian/.local/lib/python3.6/site-packages/torchsummary/torchsummary.py", line 102, in summary
    total_params_size = abs(total_params.numpy() * 4. / (1024 ** 2.))
AttributeError: 'int' object has no attribute 'numpy'
</code></pre>
<p>Also, I found that when I pass some data points, the whole model does not produce output, i.e., if I feed my model with a batch of data (32,3,224,224), the output is the same size. any help is appreciated.</p>

ptrblck: <aside class="quote no-group" data-username="Aryan_Asadian" data-post="1" data-topic="95284">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aryan_asadian/48/27084_2.png" class="avatar"> Aryan_Asadian:</div>
<blockquote>
<p>Also, I found that when I pass some data points, the whole model does not produce output, i.e., if I feed my model with a batch of data (32,3,224,224), the output is the same size. any help is appreciated.</p>
</blockquote>
</aside>
<p>This is expected, since the <code>forward</code> method was never changed.<br>
Even though you were adding valid modules via <code>add_module</code> to <code>empty_model</code>, they will never be used in:</p>
<pre data-code-wrap="python"><code class="lang-python">    def forward(self, x):
        return x
</code></pre>

Aryan_Asadian: <p>Thank you. It makes sense.</p>
Debugging backward errors

googlebot: <p>I have a big composite module, that jit compiles and executes forward() ok, but fails in backward(). The big issue is that there is no error with jit disabled, and set_detect_anomaly() is not too helpful in jit mode.</p>
<p>I’m 90% sure the error itself is related to how jit incorrectly enables requires_grad in multiple scenarios. But are there any techniques to localize it?</p>
<p>For reference, here is exception text:</p>
<pre><code class="lang-auto">The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
  File "&lt;string&gt;", line 138, in &lt;backward op&gt;
                   dim: int):
            def backward(grad_outputs: List[Tensor]):
                grad_self = torch.stack(grad_outputs, dim)
                            ~~~~~~~~~~~ &lt;--- HERE
                return grad_self, None
RuntimeError: sizes() called on undefined Tensor
</code></pre>
<p>So, some generated code, look like for unbind() operation, where outputs have inconsistent requires_grad?</p>
<p>Also note how “dim:int” line cutoff does a bad service.</p>
<p>And console:</p>
<p>[W …\torch\csrc\autograd\python_anomaly_mode.cpp:104] Warning: Error detected in struct torch::jit::`anonymous namespace’::DifferentiableGraphBackward. Traceback of forward call that caused the error:<br>
…<br>
has traceback that stops at jit module</p>

tom: <p>To me the error looks like you have an undefined Tensor (aka None) showing up unexpectedly.<br>
I would try to cut down the model (or just the JITed part) a bit to zoom into where it happens. (You can see a method I use to grab a submodule <a href="https://lernapparat.de/transformers-pytorch-tvm/" rel="nofollow noopener">when searching for DebugWrap in my blog Post on PyTorch and TVM</a>).<br>
Naturally, we would be most grateful if you found a reproducing snippet that you can share to fix this in PyTorch.</p>
<p>Best regards</p>
<p>Thomas</p>

googlebot: <p>I couldn’t easily divide that module (issue with mutable objects in mixed mode). What helped somewhat was running backward() in profiler context, exporting chrome trace and looking at last successful ops, but that’s not a good or reliable solution. Luckily, I identified unbind from the above message, and indeed unbind was the problem.</p>
<p>Now, for some reason I failed to reproduce the failure in a small script, however I think it is somehow related to “<a href="https://github.com/pytorch/pytorch/issues/36608" rel="nofollow noopener">Backward through view of unbind output</a>” issue and seems to only fail in “legacy” jit mode (which is 1.6 default), so perhaps this will be handled in next release.</p>
Change Depth of Torchvision Wide Resnet 50 Model

Kale-ab_Tessera: <p>I am using the <code>wide_resnet50_2</code> model from <code>torchvision.models</code>. I want to change the depth of the model to 28, as the paper mentions various depths and performance levels per depth.</p>
<p>How would I do this using the <code>wide_resnet50_2</code> model? Is there a simple configuration of the base <code>_resnet</code> class that is equivalent to wide resnet 50, of depth 28 from the original <a href="https://arxiv.org/pdf/1605.07146.pdf" rel="nofollow noopener">paper</a>.</p>

Kale-ab_Tessera: <p>So I have seen this <a href="https://github.com/meliketoy/wide-resnet.pytorch" rel="nofollow noopener">implementation</a> of wideresnet in pytorch that has depth 28, but the architecture seems quite different to the one provided by torchvision.</p>
nn.Linear as forward function complex neural network

Schalky: <p>Hey guys!</p>
<p>I am running into a problem when using the complex library. I am trying to write a little training program in which I am using torch.nn.Linear as a forward function. The input data I am working with, however, can be complex.<br>
Therefore in torch.nn.Linear I end up trying to do matmul with complex valued tensors, which is not allowed (under the release 1.6.0). The reason that I end up in a situation in which I want to perform complex valued tensors is that my cost function is complex which leads to complex gradients and hence a possibly complex weights.<br>
Can someone tell me if there is perhaps another forward function I could use with which I would not run into these problems? Or can someone give me some insight in another way in which I could write little neural network with a complex cost function?</p>

tom: <p>If you don’t mind me asking, what does it mean to have a complex cost function (i.e. how do you “minimize”)?<br>
One obvious way could be to feed the real and imaginary into the network separately, it would seem PyTorch still needs a bit before supporting complex for everything. Matmul should work with the preview nightly but not Linear, so you could define your own linear if you want.</p>
<p>Best regards</p>
<p>Thomas</p>

Schalky: <p>So I am minimising over the difference in probality of making a certain measurement (in the learnt and optimal case). This probability is determined by both the imaginary and real part of the amplitude. And the values that I am optimising over influence the final state of the system in a way determined by complex functions. So there is no way (to my knowledge) to rewrite the system in such a way that I can get rid of these complex values here without loosing a lot of information.</p>
<p>I will have a look at the preview nightly and perhaps define my own linear.</p>
<p>Thanks for your help,<br>
Merel</p>
Convert Conv2d to Linear

Ameen_Ali: <p>Hello<br>
saying that I have a conv2d layer, I know that its a special case of a Linear layer, how can I convert a Conv2d Layer to Linear?<br>
for example :</p>
<pre><code class="lang-auto">Conv2d(96, 1000, kernel_size=torch.Size([10, 10]), stride=(1, 1))
</code></pre>

ptrblck: <p>You could <code>unfold</code> the input to apply the <code>im2col</code> transformation and reshape the kernels to the kernel matrix as explained in <a href="https://petewarden.com/2015/04/20/why-gemm-is-at-the-heart-of-deep-learning/">this blog post</a> by Pete Warden.</p>
1j * cos( theta), error?

Schalky: <p>Hey guys!</p>
<p>So I am trying to create the following rather simple function:</p>
<pre><code class="lang-auto"># Make sure to use pytorch version 1.6.0 as I am using type torch.cfloat
from torch import cos, tensor
def func():
	return tensor([[1.0, - 1j*cos(tensor(1.0))],[1.0,0.0]], dtype=torch.cfloat)
print(func())
</code></pre>
<p>I get an error stating that: “RuntimeError: value cannot be converted to type double without overflow: (0,-0.848705)”.<br>
I have played around with different options for a while and it seems like this error is induced by the fact that I am doing 1j*cos(tensor(1.0)). Can someone explain what is going wrong when I multiply this cosine of tensor(1.0) by this complex amplitude?<br>
I</p>

tom: <p>With a slightly newer PyTorch (self-compiled from the dev branch, you could also use a nightly):</p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.__version__
'1.7.0a0+98ad5ff'

&gt;&gt;&gt; torch.tensor([[1.0, - 1j*torch.cos(torch.tensor(1.0))],[1.0,0.0]], dtype=torch.cfloat)
tensor([[1.+0.0000j, 0.-0.5403j],
        [1.+0.0000j, 0.+0.0000j]])
</code></pre>
<p>so maybe something got fixed (complex support is rather new and there probably are rough edges).</p>
<p>Best regards</p>
<p>Thomas</p>

Schalky: <p>Thanks for the tip! I will try this with the newer version of torch.</p>
Using DataLoader with num_workers>0 causes re-run of script

Alon: <p>I tried to make a minimal representation of the code:</p>
<pre><code class="lang-auto">from torch.utils.data import DataLoader
from torch.utils.data.sampler import Sampler
from torch.utils.data import Dataset
import numpy as np

print('running this script')

class Basic_Sampler(Sampler):
    def __init__(self, samples_list):
        self.sample_list = samples_list
        self.epoch_size = len(samples_list)
    def __iter__(self):
        return iter(self.sample_list)
    def __len__(self):
        return self.epoch_size

class fake_dataset(Dataset):
    def __init__(self):
            pass
    def __getitem__(self, item):
        return np.zeros(5)

sampler = Basic_Sampler([1,2,3])
dl = DataLoader(fake_dataset(), batch_size=2, shuffle=False, sampler=sampler, num_workers=0)

iter(dl).next()
</code></pre>
<p>If you run this with num_workers=0 it will print <code>running this script</code> and return as expected.<br>
If you run this with num_workers=1 it will run <strong>twice</strong> for some reason, it will print:</p>
<pre><code class="lang-auto">running this script
running this script
</code></pre>
<p>and then it will crash with <code>RuntimeError: DataLoader worker (pid(s) 18984) exited unexpectedly</code></p>
<p>I’m running:</p>
<ul>
<li>win10 64bit</li>
<li>Python 3.8.5</li>
<li>Torch <a href="https://download.pytorch.org/whl/cu102/torch-1.6.0-cp38-cp38-win_amd64.whl" rel="nofollow noopener">https://download.pytorch.org/whl/cu102/torch-1.6.0-cp38-cp38-win_amd64.whl</a>
</li>
</ul>
<p>Am I missing something here?</p>
<p>Edit:<br>
on Colab it works without issue: <a href="https://colab.research.google.com" rel="nofollow noopener">https://colab.research.google.com</a> Does that mean it is a Windows problem?</p>

ptrblck: <p>In Windows you would need to use the if-clause protection as explained in the <a href="https://pytorch.org/docs/stable/notes/windows.html#multiprocessing-error-without-if-clause-protection">Windows FAQ</a> to avoid this behavior.</p>
Get different output after changging inplace to False in pretrained models in ReLU

CharlieTao: <p>I am trying to load a pretrained model and finetuning it. the original ReLU layer has a “inplace=True” option. Since this inplace option is causing some problems in autograd, I change the option to “inplace=False”. What confuses me is that the output of the network is totally different with these two different inplace options. Since ReLU has no trainable parameters, why would this happen?</p>
<p>Below is the module where the problem happened.</p>
<pre><code class="lang-auto">

class ResidualConvUnit(nn.Module):

    def __init__(self, features):
        super().__init__()

        self.conv1 = nn.Conv2d(
            features, features, kernel_size=3, stride=1, padding=1, bias=True
        )
        self.conv2 = nn.Conv2d(
            features, features, kernel_size=3, stride=1, padding=1, bias=True
        )
        self.relu = nn.ReLU(inplace=False)

    def forward(self, x):
        out = self.relu(x)
        out = self.conv1(out)
        out = self.relu(out)
        out = self.conv2(out)
        return out + x
</code></pre>

Henry_Chibueze: <p>What kind of data is the network working with here?<br>
From my experience Setting the relu inplace to true modifies the input features inplace rather than allocate new memory to a copy of the input and also autograd doesn’t really work well with relu Inplace = True so I don’t see any issues here can u pls upload ur training process too</p>

ptrblck: <p>As <a class="mention" href="/u/henry_chibueze">@Henry_Chibueze</a> pointed out, the input <code>x</code> would be modified inplace, which would thus yield different results in <code>return out + x</code>.<br>
You could either <code>clone()</code> <code>x</code> before passing it to the first <code>relu</code>:</p>
<pre><code class="lang-python">out = self.relu(x.clone())
</code></pre>
<p>or just don’t use the inplace version.</p>
Reflection padding for 3D

CS.Enthu: <p>How to implement a relfelction padding in 3D? I want to implement nn.ReflectionPad(2D) in 3D.</p>

ptrblck: <p>According to <a href="https://github.com/pytorch/pytorch/issues/27655">this issue</a> it’s not implemented yet and contributions are welcome. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Tensorboard in PyTorch 1.1.0 -- SummaryWriter doesn't have attribute 'flush'

savourylie: <p>I’m using an older version of PyTorch 1.1.0, and I realize that the SummaryWriter doesn’t have the flush method. Is there a workaround for me to update all the metrics in tensorboard while training?</p>

ptrblck: <p>You could try to add the <a href="https://github.com/pytorch/pytorch/blob/c84f78470b0a9ffa18fc54483ea5b08e50dc575d/torch/utils/tensorboard/writer.py#L132-L137"><code>flush</code> method</a> manually or update PyTorch to the latest version.<br>
That being said, it seems that <code>flush</code> was added in <a href="https://github.com/pytorch/pytorch/pull/16196">this PR</a>, which should be available in <code>1.1.0</code>.</p>
How data in a batch is processed by different network

zcajiayin: <p>I have a batch of data:</p>
<pre><code class="lang-auto">[[0.1, 0.3],
 [0.5, 0.2],
 [0.7, 0.8],
 [0.6, 0.6],
 [0.2, 0.9]]
</code></pre>
<p>I have two networks, <code>mlp1()</code> and <code>mlp2()</code></p>
<p>How can I process the data <code>[0.7, 0.8]</code> and <code>[0.2, 0.9]</code> with <code>mlp1()</code>, and the rest with <code>mlp2()</code> to get:</p>
<pre><code class="lang-auto">[mlp2([0.1, 0.3]),
 mlp2([0.5, 0.2]),
 mlp1([0.7, 0.8]),
 mlp2([0.6, 0.6]),
 mlp1([0.2, 0.9])]
</code></pre>
<p>Any suggestions are appreciated!</p>

ptrblck: <p>I’m not sure, if I understand the use case correctly, but you should be able to slice the input and pass the corresponding parts to the appropriate model.</p>

zcajiayin: <p>Hi ptrblck, exactly, I will have to slice the input and feed to networks respectively. However, after slicing the input, I will get two parts, one for <code>mlp1()</code> and the other for <code>mlp2()</code>. The tricky part is that I need to merge the two parts back into a holistic tensor after forwarding the network. Any hints?</p>

ptrblck: <p>Assuming you have precomputed the indices, which would be used for the splitting, you could recreate the order by indexing into the result tensor:</p>
<pre><code class="lang-python">x = torch.tensor([[0.1, 0.3],
                  [0.5, 0.2],
                  [0.7, 0.8],
                  [0.6, 0.6],
                  [0.2, 0.9]])

mlp1_idx = torch.tensor([2, 4])
mlp2_idx = torch.tensor([0, 1, 3,])

x1 = x[mlp1_idx]
x2 = x[mlp2_idx]


# process
...


res = torch.zeros_like(x)
res[torch.cat((mlp1_idx, mlp2_idx), dim=0)] = torch.cat((x1, x2), dim=0)
print(res==x)
</code></pre>
<p>Depending on your work flow you might also be able to use e.g. <code>scatter</code>.</p>
ValueError: Fan in and fan out can not be computed for tensor with fewer than 2 dimensions

Ganga: <ul>
<li>
<p>Basically I want to provide my bias and weight values from the file</p>
</li>
<li>
<p>I am using Xavier iniialization to initialize the weights and bias</p>
</li>
<li>
<p>I am facing error in bias initialization<br>
have a look into my code</p>
</li>
</ul>
<pre><code class="lang-auto">import torch.nn as nn
import csv
import numpy
import numpy as np
class PrintLayer(nn.Module):
    def __init__(self):
        super(PrintLayer, self).__init__()
        m = nn.Linear(4096, 1000)
        
    
    def forward(self, x):
        # Do your print / debug stuff here
        #print("x is",x)
        return x

def init_weights(m):
      if type(m) == nn.Linear:
        print('hi')
        torch.nn.init.xavier_uniform(m.weight)
        
        torch.nn.init.xavier_uniform(m.bias)

        #for weight
        a_file1 = open("8bit_lastlayer_weight_round.txt","r")
        list_of_lists1 = []
        for line in a_file1:
          stripped_line1 = line.strip()
          line_list1 = stripped_line1.split()
          list_of_lists1.extend(line_list1)
        a_file1.close()

        b = torch.from_numpy(numpy.array(list_of_lists1,dtype= 'float32'))
        w = torch.tensor(b)
        ten = w.reshape(1000,4096)
        m.weight.data = ten

        #for bias
        a_file12 = open("8bit_lastlayer_bias.txt","r")
        list_of_lists12 = []
        for line in a_file12:
          stripped_line12 = line.strip()
          line_list12 = stripped_line12.split()
          list_of_lists12.extend(line_list12)
        a_file12.close()

        c = torch.from_numpy(numpy.array(list_of_lists12,dtype= 'float32'))
        d = torch.tensor(c)
        ten1 = d.reshape(1,1000)
        print(ten1.get_shape())
        m.bias.data = ten1
       
        #print("actual weight is",m.weight)
        #print(m.bias)
      
model = nn.Sequential(
        nn.Linear(4096, 1000),
        PrintLayer(),
         # Add Print layer for debug
        )
model.apply(init_weights)

#for input
a_file = open("8bit_lastlayer_input.txt","r")
list_of_lists = []
for line in a_file:
  stripped_line = line.strip()
  line_list = stripped_line.split()
  list_of_lists.extend(line_list)
a_file.close()
b = torch.from_numpy(numpy.array(list_of_lists,dtype= 'float32'))
x = torch.tensor(b)
x = x.reshape(1,4096)
output = model(x)

values,indices = torch.max(output,1)
print(values,indices)
np.set_printoptions(threshold=50000000,formatter={'float_kind':'{:f}'.format})
output = output.detach().numpy()
#print(output)
file = open("8bit_final_layer_output.txt","w")
file.write(str(output))
file.close()

#print(model)
#for weight and bias
#print(list(model.named_parameters()))
#print(model.state_dict())```</code></pre>

tom: <p>Xavier uniform isn’t for the bias… A typical thing to use might be 0.</p>

Ganga: <p>can you please show us the code snippet</p>

tom: <pre><code class="lang-auto">torch.nn.init.zeros_(m.bias)
</code></pre>
Keypoint training for detectron

Sahib_Majithia: <p>Hi,<br>
I have been trying to train detectron released by Facebook for Keypoint detection. I am able to train it with the coco-dataset but not on my custom dataset. I did follow steps like changing Detectron/dataset/json_dataset.py to change from person_keypoint only to my custom dataset. But I still run into issues since there is not enough documentation. Has someone trained on a custom dataset before for keypoint detection who could provide me a head-start from his learnings ?</p>

Mdhvince: <p>Could you be more precise ? Do you have some code, error to share ?</p>

xtian_carlos: <p>Hi Sahib_Majithia, Did you can train? I have the same goal and i can’t find a manual yet.</p>
Define Network (VGG) as one Sequential layer

morrohype: <p>Hello,</p>
<p>I’m trying to transfer the torchvision.models.vgg11 into one Sequential layer. You can take a look at the PyTorch source-code <a href="https://pytorch.org/docs/stable/_modules/torchvision/models/vgg.html#vgg11" rel="nofollow noopener">here</a>.<br>
As you can see the forward pass is defined as:</p>
<pre><code class="lang-auto">def forward(self, x):
        x = self.features(x)
        x = self.avgpool(x)
        x = torch.flatten(x, 1)
        x = self.classifier(x)
        return x
</code></pre>
<p>So I thought it could work like this:</p>
<pre><code class="lang-auto">pt_vgg11 = vgg11(pretrained=False)

class FlattenModule(nn.Module):
    def forward(self, x):
        x = torch.flatten(x, 1)
        return x

def VGG11_Seq():
    features = torch.nn.Sequential(
        pt_vgg16.features,
        pt_vgg16.avgpool,
        FlattenModule(),
        pt_vgg16.classifier
    )
    return features

model = VGG11_Seq()
</code></pre>
<p>(I also tried to print out the <code>pt_vgg11</code> and just take everything that’s part of the network and put it inside of a Sequential, but that won’t work either …)</p>
<p>but when I try to load weights from a trained torchvision.models.vgg11<br>
it won’t work. And also training the <code> model = VGG11_Seq()</code> won’t work. (working with CIFAR10)</p>
<p>Can anyone see the error, or can provide any tips?</p>

a_d: <p>To define the vgg model as one sequential layer you could simply do -</p>
<pre><code class="lang-auto">vgg = models.vgg16(true/false)
vgg_features = nn.Sequential(*list(vgg.children())[:-1]) #removed the Last linear layer
</code></pre>
<p>And in the forward method -<br>
<code>out = vgg_features(input)</code></p>

morrohype: <p>Thank you for your answer! Why do you remove the last linear layer?</p>

a_d: <p>You are extracting features from the pre-trained VGG and training your own classification layer, I am right to assume that? Because you are extracting features then flattening them. If you do not remove the last layer of the VGG, your output would be of the shape-<code>[batch_size, num_classes]</code>, already giving you the class probabilities.<br>
Though you can always add extra linear layers on top of the VGG network.</p>
Torchvision same random transforms on multi images

ilyes: <p>Hello,</p>
<p>I am working on an optical flow algorithm, where the input is 2 images of size HxWx3 and the target is a tensor of size  HxWx2. To do data augmentation, I need to apply the same random transformation to all the 3 tensors. Take this augmentation for example:</p>
<pre><code class="lang-auto">        aug_transforms = transforms.Compose([
            transforms.RandomHorizontalFlip(),
            transforms.RandomResizedCrop((614, 216), scale=(0.1, 1.0)),
            transforms.RandomRotation(17),
            transforms.RandomVerticalFlip(),
            transforms.RandomApply([transforms.ColorJitter()])
        ])
</code></pre>
<p>The torchvision transforms accept PIL object as an input. To solve this problem I have two questions:</p>
<ul>
<li>Can we save the random state of a given transform, so that we can apply the same randomness to img1, img2 and the flow ?</li>
<li>Can we combine all the date into a PIL object, apply transformation and retrieve back the same tensors values range we had at the beginning ?</li>
</ul>

ptrblck: <p>I would recommend to use the functional API as shown <a href="https://discuss.pytorch.org/t/torchvision-transfors-how-to-perform-identical-transform-on-both-image-and-target/10606/7">here</a>, since working with the seed might be hard and yield unwanted side effects.</p>
<p>That being said, I’m not sure at the moment, if <code>PIL</code> will accept the target images with two channels, so you might need to split the channels (or add a dummy channel to the target).</p>

ilyes: <p>Thank you for the reply. I see that this issue is faced a lot in different projects that have a tensor output. Just our of curiosity, do you know why they are using PIL instead of just numpy arrays, so we can apply the transforms on a list of tensors with different number of channels ?</p>

ilyes: <p>For people having the same issue. I found an interesting API that does data augmentation on numpy array of any channel number, can do it on multi targets, bounding boxes, masks …</p>
<pre><code class="lang-auto">import albumentations as albu
import cv2
co_aug_transforms = albu.Compose([albu.RandomSizedCrop((300, 436), 400, 940, w2h_ratio=1024/436, p=0.5),
                                  albu.Flip(),
                                  albu.ShiftScaleRotate()
                                  ])

flow = readflo(path_to_flow)
frame1 = cv2.cvtColor(cv2.imread(path_to_frame1), cv2.COLOR_BGR2RGB)
frame2 = cv2.cvtColor(cv2.imread(path_to_frame2), cv2.COLOR_BGR2RGB)
target = {}
target3 = {'image0': 'image', 'image1': 'image', 'image2': 'image'}
transformed = albu.Compose(co_aug_transforms, p=1, additional_targets=target3)(image=frame1,
                                                                                         image0=frame2,
                                                                                         image1=flow)

frame1 = transformed['image']
frame2 = transformed['image0']
flow = transformed['image1']

</code></pre>
Normalize 1-D Tensors

CogentCoder73: <p>Hello!</p>
<p>I’ve recently started a project analyzing 1-D ECG Data. However, I have to normalize the produced 1-D tensors for pre-processing. Since it doesn’t appear that torchvision.transforms supports 1-D tensors, is there a way to manually normalize 1-D tensors?</p>
<p>Thank you!</p>

a_d: <p>You could look at instanceNorm1D and layerNorm operations both provided in the torch.nn modules</p>

CogentCoder73: <p>Thank you! I will look into that!</p>
Help in building a custom cuda kernel

a_d: <p>Hello all, I want to build a custom Cuda kernel for calculating multiples IoU and adding them, to be called from a python program later. I need to speed this up and thought of using the GPU as this seems easily parallelizable.<br>
The function calcsum looks something llike this -</p>
<pre><code class="lang-auto">
void calcsum(TensorAccessor&lt;float, 3&gt; a, TensorAccessor&lt;float, 3&gt; b)
{
    float net_sum;
    for (int i=0;i&lt;a.size(1);++i)
    {
        for (int j=0;j&lt;b.size(1);++j)
        {
            auto xi = at::max(a[0][i][0], b[0][j][0]);
            auto yi = at::max(a[0][i][1], b[0][j][1]);
            auto wi = torch::clamp(torch::min(a[0][i][2], b[0][j][3])-xi, 0);
            auto hi = torch::clamp(torch::min(a[0][i][2], b[0][j][3])-yi, 0));
            auto ai = wi*hi;
            auto au = ((a[0][i][2]-a[0][i][0])*(a[0][i][3])-a[0][i][1]) + ((b[0][j][2]-b[0][j][0])*(b[0][j][3])-b[0][j][1])-ai;
            auto iou =ai/au;
            if (iou&gt;0.4)
                net_sum+=iou;

        }
    }
}

int main() {
    torch::Tensor tensor = torch::rand({1, 50000, 4});
    torch::Tensor tensor1 = torch::rand({1, 52, 4});
    auto accessor_a = tensor.accessor&lt;float, 3&gt;();
    auto accessor_b = tensor1.accessor&lt;float, 3&gt;();
    calcsum(accessor_a, accessor_b);
}

</code></pre>
<p>I have just started reading about how to parallelize loops using cuda, so I am pretty new to the field.<br>
Could someone please guide as to how I could handle the nesting of loops and how to go ahead with it…<br>
TIA</p>
Looking for methods of fixing image artifacts

Nason: <p>I’ve been working on a fully convolutional network but I’m getting some weird behaviors.<br>
My issue is that the model seems to be mixing features, <a href="https://imgur.com/a/DxAXr8f" rel="nofollow noopener">for example</a>, features which should only appear in channel 2 and 3 are appearing in channel 1. I’ve looked through my test data and there doesn’t seem to be any clear cause of this behavior, it seems to be randomly occurring.</p>
<p>I’d love to hear some suggestions on how I could fix this issue.</p>
<p>Thanks in advance,</p>
How to implement a transfer learning like training process?

Nath_NWA: <p>I’ve been working on a UNet and I’ve been advised to try a transfer learning style approach. My issue is that I can’t visualise the training procedure, I’ve got myself confused by overthinking the instructions.</p>
<p>To quote the instructions I was given.</p>
<p>“<em>I was also thinking about something along the lines of a loss function based on higher-level features. A simple approach would be to use a pre-trained net to extract some features from an intermediate layer and then calculate another loss function based on the difference between the features extracted from the ground truth and from the output images.</em>”</p>
<p>I replied:<br>
To confirm, first we get a pretrained model and feed it ground truth inputs, then we extract the resulting features from intermediate layers, and finally we calculate the loss between these extracted features and the output.</p>
<p>Reponse:<br>
“<em>You feed the output of the network as well to the pre-trained model to extract the same features as with the ground truth and then compare them to calculate the loss</em>.”</p>
<p>My question is, how would you interpret this architecture or training process? I’ve added what I think below but I’d love you hear your unbiased take.</p>
<blockquote>
<p>From my understanding, I get my pretrained UNet, then feed it ground truth images, at the same time I feed the same input to a fresh network im training, I then calculate a loss function based of the intermediate outputs of the pre-trained network and the fresh network?</p>
</blockquote>
<p>Also if you know of any similar examples of such a set up, please share as that would massively help.</p>