akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to mix model parameters?

dongfangliu: <p>Hello, all.<br>
I am a  new-bee to PyTorch.<br>
Here I have a simple idea but I do not know how to implement it with PyTorch.<br>
The question is about how to mix model parameters on the fly, to do something like weighted network parameter sum.</p>
<p>Here I got multiply <code>nn.Module</code> Instance <code>A,B,C,D</code> , they share the same network structure but have different parameter values.</p>
<p>Then I got a simple Multiple Layer Perception which is also implemented as an <code>nn.Module</code> to output a tensor <code>W</code> of size(4,1).</p>
<p>The tensor <code>W</code> works as a weight vector to be learned by the networks. And the instances <code>A,B,C,D</code> are pre-trained, and parameters are frozen to be used as the components to be weighted.</p>
<p>Then I do not know to correctly implement the <code>forward</code> function of the following class.</p>
<pre><code class="lang-auto">class Mixture(nn.Module):
    
    def __init__(self, input_dim, hidden_dim, hidden_depth,base_modules =[]):
        super().__init__()

        self.weight_network= nn.Softmax(utils.mlp(input_dim, hidden_dim, len(base_modules ),hidden_depth))
        
        self.base_modules = base_modules 
        for m in self.base_modules :
            for c in m.children():
                for param in child.parameters():
                    params.requires_grad = False
        
        self.outputs = dict()

    def forward(self, x):
        
        weights= self.weight_network(x)
        ######################### Here need help ##################
        #  Here i want to  get a network : 
        #      f = w0*param_A+w1*param_B+w2*param_C+w3*param_D
        #########################################################
        f = ?
        y = f(x)
        self.outputs['y'] = y
        
        return y
</code></pre>
<p>Thanks for any possible instructions and help.</p>

dongfangliu: <p>Help wanted! i really need that</p>

crcrpar: <p>Hi,</p>
<p>If the latency doesn’t matter, I think the snippet below will work.</p>
<pre><code class="lang-python">weights = self.weight_network(x)
f = sum(w * m(x) for w, m in zip(weights, self.base_modules))
</code></pre>
<p>I tried this here: <a href="https://colab.research.google.com/drive/163Id8tsSofW4c_Mm6_7X_XThJadTpn6e?usp=sharing" rel="noopener nofollow ugc">https://colab.research.google.com/drive/163Id8tsSofW4c_Mm6_7X_XThJadTpn6e?usp=sharing</a></p>
Can't find an install for cuda 10.2 win 64 torch 1.8 python 3.8

michaelrdonat: <p>From the Get Started page, I’m given the command:<br>
pip install torch==1.8.0+cu102 torchvision==0.9.0+cu102 torchaudio===0.8.0 -f <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a><br>
but, as admin, I get:<br>
Looking in links: <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a><br>
ERROR: Could not find a version that satisfies the requirement torch==1.8.0+cu102</p>
<p>Is there a place I can download an appropriate whl?</p>
<p>Thanks,<br>
Michael</p>
How to build a pipeline to load model in batch?

AllenD1: <p>Hey everyone,</p>
<p>currently, I have multiple models, like 10,000. I want to query these models with a single input. My current way is to query these models in a sequential way (1 by 1). However, it is not efficient enough. I am curious if there is a way to batch models and put single input to these models and get the response?</p>
<p>Thanks in advance!</p>
How to convert a CFD .VTI File into a Signed Distance Function Format

SShri: <p>Hi!<br>
I am currently trying to use Pytorch to train CFD images and wanted the input images for my network to be in a Signed Distance Function (SDF) format : <a href="https://en.wikipedia.org/wiki/Signed_distance_function" class="inline-onebox" rel="noopener nofollow ugc">Signed distance function - Wikipedia</a> . I have managed to convert the .VTI file from the CFD into a NumPy array, but the array is of the size 36,603 x 3 and so is not a feasible image to enter. However, I wish to convert this array into a size of 256 x 256 and then create an SDF mapped image. Any help would be really valuable<br>
Cheers</p>
TorchVision C++ API build error

ishanic: <p>Hello all,</p>
<p>I tried the minimal example for building a TorchVision application: <a href="https://github.com/pytorch/vision/tree/master/examples/cpp/hello_world" class="inline-onebox" rel="noopener nofollow ugc">vision/examples/cpp/hello_world at master · pytorch/vision · GitHub</a><br>
The cmake goes through generation and configuration, but it fails at the “make” with the following error -</p>
<pre><code class="lang-auto">(base) user@visionlab:~/src/vision/examples/cpp/hello_world/build$ make
CMakeFiles/hello-world.dir/build.make:82: *** target pattern contains no '%'.  Stop.
CMakeFiles/Makefile2:75: recipe for target 'CMakeFiles/hello-world.dir/all' failed
make[1]: *** [CMakeFiles/hello-world.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2
</code></pre>
<p>In the CMake, replacing line “target_link_libraries(${PROJECT_NAME} TorchVision::TorchVision)” with “target_link_libraries(${PROJECT_NAME} torchvision)” makes it work.<br>
Any thoughts on how to make the default CMake with TorchVision::TorchVision work?</p>
<p>Thanks.</p>
Manually change/assign weights of a neural network

grid_world: <p>I am using Python 3.8 and PyTorch 1.7 to manually assign and change the weights and biases for a neural network. As an example, I have defined a LeNet-300-100 fully-connected neural network to train on MNIST dataset. The code for class definition is:</p>
<pre><code>class LeNet300(nn.Module):
    def __init__(self):
        super(LeNet300, self).__init__()
        
        # Define layers-
        self.fc1 = nn.Linear(in_features = input_size, out_features = 300)
        self.fc2 = nn.Linear(in_features = 300, out_features = 100)
        self.output = nn.Linear(in_features = 100, out_features = 10)
        
        self.weights_initialization()
    
    
    def forward(self, x):
        out = F.relu(self.fc1(x))
        out = F.relu(self.fc2(out))
        return self.output(out)
    
    
    def weights_initialization(self):
        '''
        When we define all the modules such as the layers in '__init__()'
        method above, these are all stored in 'self.modules()'.
        We go through each module one by one. This is the entire network,
        basically.
        '''
        for m in self.modules():
            if isinstance(m, nn.Linear):
                nn.init.xavier_normal_(m.weight)
                nn.init.constant_(m.bias, 0)
</code></pre>
<p>To experiment with trying to change the weights for this model-</p>
<pre><code># Instantiate model-
mask_model = LeNet300()
</code></pre>
<p>To assign all of the weights in each of the layers to one (1), I use the code-</p>
<pre><code>with torch.no_grad():
    for layer in mask_model.state_dict():
        mask_model.state_dict()[layer] = nn.parameter.Parameter(torch.ones_like(mask_model.state_dict()[layer]))

# Sanity check-
mask_model.state_dict()['fc1.weight']
</code></pre>
<p>This output shows that the weights are not equal to 1.</p>
<p>I also tried the code-</p>
<pre><code>for param in mask_model.parameters():
    # print(param.shape)
    param = nn.parameter.Parameter(torch.ones_like(param))
</code></pre>
<p>But this does not work as well.</p>
<p>Help?</p>

witl: <p>Like this?</p>
<pre><code class="lang-auto">    def set_to_one(self, model):
        for name, param in model.named_parameters():
            values = torch.ones(param.shape)
            param = values
</code></pre>
<p>Edit: I forgot to add .data</p>
<pre><code class="lang-auto">    def set_to_one(self, model):
        for name, param in model.named_parameters():
            values = torch.ones(param.shape)
            param.data = values
</code></pre>
Generate a random vector that have a small distance between it and another one

Helen_R6: <p>Hi,<br>
let’s say I have a random vector <code>z1=torch.randn(1,128,requires_grad = True)</code>. Now I would like to generate another vector z2 such that ||z1-z2||&lt;epsilon. How should I do this in pytorch?<br>
Any help and suggestions would be appreciated, thanks in advance.</p>

Nifrec: <p>Perhaps generate <code>z2</code> as <code>z1</code> with some uniform noise added to it? With uniform noise you could guarantee the difference will not exceed epsilon, if you set the lower and upper bound correctly.</p>
<pre><code class="lang-python">z1=torch.randn(1,128,requires_grad = True)

epsilon = torch.tensor([0.5])

bound = (epsilon/2)/(z1.numel())

distribution = torch.distributions.uniform.Uniform(-bound, bound)
z2 = z1 + distribution.sample((1, 128))
</code></pre>
<p>Not sure if I computed the bounds optimally, but I hope the idea helps.</p>
Trouble understanding _version semantics

Nifrec: <p>Autograd does not allow backpropagation through tensors with their <code>_version</code> attribute greater than 0. However, when training a linear model, the <code>_version</code> of the learnable parameters do change each time the <code>step()</code> function of the optimizer is called, and this normally does not produce any errors. <strong>Are exceptions made for <code>nn.Parameter</code> instances? Or for leafs in the computational graph? Or is the mechanism even something else?</strong></p>
<p>Take the following example:</p>
<pre><code class="lang-python">class MyModule(nn.Module):

    def __init__(self):
        super().__init__()
        self.my_param = nn.Parameter(torch.tensor([1.0], requires_grad=True))

    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        return x * self.my_param


mm = MyModule()
optim = torch.optim.Adam(mm.parameters())

for epoch in range(2):
    optim.zero_grad()
    loss = mm(torch.tensor([2.0]))
    loss.backward()
    optim.step()
print(mm.my_param._version)
print(mm(torch.tensor([2.0]))._version)
print(loss._version)
</code></pre>
<p>This gives the following output:</p>
<pre><code class="lang-auto">2
0
0
</code></pre>
<p>Clearly the <code>_version</code> attribute of <code>my_param</code> does not stay 0.</p>
<p>The context in which I run into problems is as follows: I am more or less simulating particles using Newtonian mechanics. I want to train the initial positions, initial velocities, initial accelerations and the masses of the particles such that they end up at a specific position. Currently it works for one epoch, but the second epoch I get errors that some <code>_version</code> is not 0, and <code>torch.autograd.set_detect_anomaly(True)</code> did unfortunately not produce useful help. But it is unclear to me which variables are allowed to change in-place and which not. If only <code>nn.Parameter</code>s are allowed to change in-place, then I can start hunting for a variable that is neither a parameter nor reset at the start of the second epoch. But currently I do not know when in-place operations are allowed, so I find it hard to tell what may go wrong.</p>
How to browse the (generated) source code (in CLion e.g.)?

lithuak: <p>Hi! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I’m trying to get my head around the source base and one things that immediately gets in the way is the partition that exists between the initial C++ code and the generated part.</p>
<p>Take, for instance the /aten/src/ATen/Tensor.h</p>
<p>It looks like a promising entry point for someone like me who is getting started with understanding the code base, but then it immediately goes:</p>
<p><span class="hashtag">#include</span> &lt;ATen/core/TensorBody.h&gt;</p>
<p>which can’t be resolved by CLion because it’s just isn’t there (and as I get it is being generated during the build ending up in “/build/aten/core/”)</p>
<p>So, the source browsing stucks and the question is - what can I do here?</p>
<p>Turns out - CLion can’t be told to manually add some additional directory to search for headers - it totally relies on analyzing the CMakeLists.txt and the one in the project root dir doesn’t contain anything that could help CLion with this confusion (that’s what it (CLion) said - there is a chance I’m just missing something)</p>
<p>Adding something to CMakeLists.txt just doesn’t look right for a beginner - not sure if it wouldn’t mess up the build and then it would definitely interfere with the upstream changes and so on, so it’s better not to go down this road.</p>
<p>I’m sure that some Linux hacking can resolve this problem, the first thing that comes to mind is some union filesystem like AUFS that can be used to map both src dirs and dirs in /build with generated code together, but before going into all that, I’d rather ask - is there some standard solution to this problem?</p>
<p>How do you guys deal with it, how do you browser the source code in an “uninterrupted” way? I really hope that not everybody seriously working on Pytorch are using Emacs with some custom lisp code that does that linking for them on the fly ;))</p>
<p>Thanks!</p>
Converting a torch tensor into a jpg Image using C++ ---Libtorch

DineshRajan_T: <pre><code class="lang-auto">        torch::Tensor out_tensor = module.forward({ input_tensor }).toTensor();
		auto img_output = out_tensor;
		auto img_uint = img_output.toType(torch::kU8);
		auto img = cv::Mat(input_image.rows, input_image.cols, CV_8UC3, img_uint.data_ptr());
		cv::cvtColor(img, img, cv::COLOR_BGR2RGB);
		std::cout &lt;&lt; "Tensor converetd to Image!!" &lt;&lt; std::endl;
		cv::imwrite("./Images/new_img.jpg", img);
		

</code></pre>
<p>I can’t get the desired output. Instead I get a noisy grayscale image.</p>

DineshRajan_T: <p><a class="mention" href="/u/tom">@tom</a> Do you have any idea on this issue?</p>

ptrblck: <p>I’m not Tom, but think you might be missing a permutation in the <code>img_output</code>. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
OpenCV uses the channels-last memory layout, while your tensor might be in channels-first, so you would need to permute the dimensions before writing it out as the image.</p>

DineshRajan_T: <p>Great.I solved that and thanks</p>
Why does RamdomSampler enforce num sample >0?

atawari: <p>I have a use case where dataset length can be zero. I was just hoping that Dataloader will provide no batch with dataset length zero but instead I get below error during Dataloader initialization.</p>
<blockquote>
<p>ValueError: num_samples should be a positive integer value, but got num_samples=0</p>
</blockquote>
<p>Just to experiment, I modified the RandomSampler to not raise ValueError for dataset length zero and it seems to work fine (i.e. as expected Dataloader does not provide any batch).</p>
<p>So my question is why does RandomSampler enforce num sample &gt;0?</p>
Reinitializing classifier layer for Shufflenet V2

AKASH_L_P: <p>For different models we reshape the network and reinitialize the classifier’s  layer, the link here “<a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Finetuning Torchvision Models — PyTorch Tutorials 1.2.0 documentation</a>” does not include Shufflenet V2. Does anyone have any idea??</p>

ptrblck: <p>I don’t think there is any particular reason besides that the tutorial uses a subset of all <code>torchvision</code> models.<br>
Are you seeing any issues in applying the tutorial to ShuffleNet?</p>
Run forward on machine A, and run backward on machine B

redhairdragon: <p>Hi, I am currently working on a project about pipeline parallelism which is based on DeepSpeed. In this project, we are trying to dynamically change the pipeline size in a very efficient way.<br>
In a simplified setting of 3 machines and a model with 12 layers, each node computes 4 layers. And then node 1 may die at any moment. Before node 1 dies, it has 2 minutes to transfer data out. (Like spot instances on AWS).<br>
So before node 1 dies,<br>
node 0: layer 0-3, node 1: layer 4-7, node 2: layer 8-11<br>
After,<br>
node 0: layer 0-5, node 2: layer 6-11<br>
But node 1 may have computed several forward passes. To avoid recomputation, I will send out the intermediate tensors to other nodes and I want other nodes to finish the backward pass based on the information they receive from node 1. I wonder if it is possible.</p>
<p>The background may be confusing. Let me put a code snippet. And is it possible to make backward update the “linear” model on the “to_rank”?</p>
<pre><code class="lang-auto"> def remapping_layer(self, from_rank, to_rank):
        if self.global_rank == from_rank:
            #doing forward pass
            linear = nn.Linear(4, 5).to(self.device)
            optimizer = torch.optim.SGD(
                linear.parameters(), lr=0.1, momentum=0.9)
            optimizer.zero_grad()
            x = torch.tensor(
                [[1, 2, 3, 4.]], requires_grad=True).to(self.device)
            y = linear(x)

            self.coord_com.setTensor('y', y) #Send tensor through TCP Store
            self.coord_com.setStateDict("linear", linear.state_dict()) #Send state dict through TCP Store
            exit()
        if self.global_rank == to_rank:
            # Receive tensors and continue to do the backward pass
           # Doesn't work since  the 'y' received from from_rank doesn't associate the parameter of liner model on this machine
            linear = nn.Linear(4, 5).to(self.device)
            optimizer = torch.optim.SGD(
                linear.parameters(), lr=0.1, momentum=0.9)
            optimizer.zero_grad()

            y = torch.zeros(1, 5).to(self.device)
            y = self.coord_com.getTensor('y') # Get tensor from TCP Store

            state_dict = self.coord_com.getStateDict("linear") # Get state dict of linear
            linear.load_state_dict(state_dict)

            print(f'rank:{to_rank},{y}') #rank:0,tensor([[ 1.8351, -0.8024,  0.7445,  2.3400,  0.4823]], device='cuda:0', requires_grad=True)

            z = torch.tensor([2, 3, 4, 5., 6],
                             requires_grad=True).to(self.device)
            loss = (z-y).sum()/5
            loss.backward()
            print(f"rank: 0, before {list(linear.parameters())}") # let it = statement A

            optimizer.step()
            print(f"rank: 0, {list(linear.parameters())}") # print the same thing as statement A
            exit()
</code></pre>
UserWarning: CUDA initialization: Found no NVIDIA driver on your system

German: <p>OS - Linux, Python version - 3.6.8, Torch version - 1.7.1,<br>
NVIDIA driver - 440.33, CUDA - 10.2, GPU card version - NVIDIA Volta T100.</p>
<p>When I try to use PyTorch in interactive Python session this error message pops up:<br>
“/home/gsergei/.local/lib/python3.6/site-packages/torch/cuda/<strong>init</strong>.py:52: UserWarning: CUDA initialization:<br>
Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from <a href="http://www.nvidia.com/Download/index.aspx" rel="noopener nofollow ugc">http://www.nvidia.com/Download/index.aspx</a> (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)<br>
return torch._C._cuda_getDeviceCount() &gt; 0<br>
False”</p>
<p>Torch was installed using pip3 in home directory.</p>

ptrblck: <p>This error points towards an issue with your driver, which might be caused by e.g. a driver update without a restart etc.</p>
Data Sampler to Handle Class Imbalance

matrix: <p>I trying to train a object detector on my custom data. Data I have has issue of class imbalance, meaning only 10% of images have objects in it, other 90% images doesn’t contain any objects. So here is what I want to do to solve imbalance issue,</p>
<p>At every epoch, augment those 10% images by some factor s.t. it’s qty. is same as background images (90%), now train model with these data.</p>
<p>For example, suppose I have total 100 images among which 10 contains objects and other 90 doesn’t contain any objects. At each epoch, randomly augment these 10 images and make it’s qty. 90. Now train the model on these 90 (augmented and contains objects) + 90 (background) = 180 images. Repeat these for upcoming epochs.</p>
<p>Is there a way in PyTorch to make such <em>data loader and/or data sampler</em> to handle aforementioned scenario?</p>

ptrblck: <p>If you want a predefined number of samples from each class, you could implement a custom <code>sampler</code>, which would yield the batch indices using your desired logic.<br>
On the other hand, if your use case would allow weighted random sampling with replacement, you could use a <code>WeightedRandomSampler</code> as described <a href="https://discuss.pytorch.org/t/how-to-handle-imbalanced-classes/11264/2">here</a>.</p>
Apply Filter as a part of preprocessing

Kapil_Rana: <p>Hello,</p>
<p>I want to apply a custom 5x5 filter on image as a part of preprocessing.</p>
<p>Can someone suggest me code.</p>
<p>I have already done<br>
def HPF(img):<br>
image = np.array(img)<br>
kernel = np.array([[1,4,7,4,1],<br>
[4,16,26,16,4],<br>
[7,26,41,26,7],<br>
[4,16,26,16,4],<br>
[1,4,7,4,1]])<br>
kernel = kernel/273<br>
img_rst = cv2.filter2D(np.float32(image),-1,kernel)<br>
new_image = img_rst<br>
return new_image</p>
<p>x = Image.open("/home/iit/Kapil/Code/DIP/lenna.jpg").convert(‘RGB’)<br>
<span class="hashtag">#img</span> = TF.to_pil_image(x)<br>
transform = transforms.Lambda(HPF)<br>
img = transform(x)</p>
<p>but it does not do any kind of blurring (Gaussian Blurr)</p>

ptrblck: <p>Your method works fine using random inputs:</p>
<pre><code class="lang-python">def HPF(img):
    image = np.array(img)
    kernel = np.array(
        [[1,4,7,4,1],
         [4,16,26,16,4],
         [7,26,41,26,7],
         [4,16,26,16,4],
         [1,4,7,4,1]])
    kernel = kernel/273
    img_rst = cv2.filter2D(np.float32(image),-1,kernel)
    new_image = img_rst
    return new_image

x = PIL.Image.fromarray(np.random.uniform(0, 256, (224, 224, 3)).astype(np.uint8))

transform = transforms.Lambda(HPF)
img = transform(x)

plt.imshow(img.astype(np.uint8))
</code></pre>
GPU video memory

zzpwdaswwww: <p>gpu：4G<br>
想要训练一个程序，但是总是显示显存不够，是否有方法可以解决这个问题，所有占用显存的进程已经关闭。谢谢大家。<br>
下面是报错：<br>
RuntimeError: CUDA out of memory. Tried to allocate 16.00 MiB (GPU 0; 4.00 GiB total capacity; 2.73 GiB already allocated; 3.35 MiB free; 2.82 GiB reserved in total by PyTorch)<br>
gpu：4G<br>
想要训练一个程序，但是总是显示显存不够，是否有方法可以解决这个问题，所有占用显存的进程已经关闭。谢谢大家。<br>
下面是报错：<br>
RuntimeError: CUDA out of memory. Tried to allocate 16.00 MiB (GPU 0; 4.00 GiB total capacity; 2.73 GiB already allocated; 3.35 MiB free; 2.82 GiB reserved in total by PyTorch)<br>
gpu：4G</p>
<p>Want to train a program, but always show the video memory is not enough, whether there is a way to solve this problem, all the processes that occupy the video memory have been closed. Thank you.</p>
<p>Here is the error report:</p>
<p>RuntimeError: CUDA out of memory. Tried to allocate 16.00 MiB (GPU 0; 4.00 GiB total capacity; 2.73 GiB already allocated; 3.35 MiB free; 2.82 GiB reserved in total by PyTorch)</p>

ptrblck: <p>You could try to lower the batch size, use a smaller model, or use <code>torch.utils.checkpoint</code> to trade compute for memory.<br>
If that doesn’t work, you could use e.g. Colab, which should provide a GPU with more memory.</p>
Why my weight is between -1 to 1?

Zack_Cheng: <p>My data is normalized, each of them is a number between 0 to 1. When I print out the weight, it is a set of number between -1 to 1. But why? Do weight must be number between -1 to 1?</p>

ptrblck: <p>If you don’t apply any constraints on the parameters, they are not necessarily in the range <code>[-1, 1]</code>.<br>
E.g. <code>nn.Linear</code> layers are randomly initialized using <a href="https://github.com/pytorch/pytorch/blob/4fa47e5e7d0639c7cc0e95442c49ab43581f3003/torch/nn/modules/linear.py#L86-L91">this method</a> and thus the initial range might be smaller.</p>
Feeding the model with different sized crops

gfotedar: <p>I understand there are many posts about how to handle different sized images, but what I want to do is a bit different.</p>
<p>For each image, I want to crop a random location and create a batch of N crops in each iteration. The size of crops of course will have to be same per batch but I want to vary it randomly across different batches.<br>
I want to avoid resizing.</p>
<p>I believe I can do this outside of the dataloader, like get a prepared batch and the do the random cropping before I feed the batch into the model, however, this won’t take advantage of the “num_workers” and the parralelization in the dataloader and hence will be a bit slow.</p>
<p>I am wondering if there is a way to do this inside the dataloader. Is there a way to randomly generate a size for each batch and share this size variable such that it is accessible to the transform function for each image inside a single batch?</p>

ptrblck: <p>I think there are different valid approaches for this use case, but I think an easy one would be to use a <code>BatchSampler</code> and pass the indices for the complete batch to the <code>Dataset.__getitem__</code>.<br>
This would allow you so sample the new crop size for a complete batch as well as load and process all images from this batch inside the <code>__getitem__</code> in a loop.<br>
Have a look at <a href="https://discuss.pytorch.org/t/dataloader-sample-by-slices-from-dataset/113005/4">this post</a> for more information and an example.</p>
CUDA memory leak when following the 'Play Mario with RL' tutorial

evan1: <p>I’m trying to run the ‘Play Mario with RL’ code in a Jupyter notebook right now. The cache function had to be modified due to an error thrown when using the original code verbatim. CPU usage has no problems reaching episodes 1000+. However, running this code on my local GPU always causes the GPU’s memory to run out around episode 560. Commenting out the <code>cache</code> function seems to stop the memory from filling up, so I’m guessing the problem may be in there. Does anything seem wrong at a quick glance? Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<pre><code class="lang-auto">class Mario:
    def __init__(self, state_dim, action_dim, save_dir, checkpoint=None):
        self.state_dim = state_dim
        self.action_dim = action_dim
        self.memory = deque(maxlen=100000)
        self.batch_size = 64

        self.exploration_rate = 1
        self.exploration_rate_decay = 0.99999975
        self.exploration_rate_min = 0.1
        self.gamma = 0.9

        self.curr_step = 0
        self.burnin = 1e5  # min. experiences before training
        self.learn_every = 3   # no. of experiences between updates to Q_online
        self.sync_every = 1e4   # no. of experiences between Q_target &amp; Q_online sync

        self.save_every = 5e5   # no. of experiences between saving NN
        self.save_dir = save_dir

        # NN to predict the most optimal action - implemented in the Learn section
        self.net = DQN(self.state_dim, self.action_dim).float()
        self.net = self.net.to(device)

        if checkpoint:
            self.load(checkpoint)

        self.optimizer = torch.optim.Adam(self.net.parameters(), lr=0.00025)
        self.loss_fn = torch.nn.SmoothL1Loss()

... [CODE OMITTED] ...


    def cache(self, state, next_state, action, reward, done):
        """
        Store the experience to self.memory (replay buffer)
        Inputs:
        state (LazyFrame),
        next_state (LazyFrame),
        action (int),
        reward (float),
        done(bool))
        """
        
        state = torch.FloatTensor(np.array(state)).to(device)
        next_state = torch.FloatTensor(np.array(next_state)).to(device)
        action = torch.LongTensor([action]).to(device)
        reward = torch.DoubleTensor([reward]).to(device)
        done = torch.BoolTensor([done]).to(device)
        
        self.memory.append( (state, next_state, action, reward, done,) )

... [CODE OMITTED] ...
</code></pre>

ptrblck: <p>I cannot see any obvious issues in the posted code snippet. However, it also doesn’t show the overall usage. In the <code>cache</code> method you are appending device tensors to the <code>dequeu</code> and see an increase in memory. Are you removing these objects properly from <code>self.memory</code> and are not storing any other references to these tensors?</p>
How to add meta-features so that I can access them during training?

TinfoilHat0: <p>Hi,</p>
<p>I’m doing some image classification, and for each image, I’ve a single boolean meta-feature.</p>
<p>How should I encode the meta-feature to my dataset so that I could access it during the training<br>
when I fetch a batch from the dataloader? I’ll not pass this feature to the network that I’m training, but rather reweight the instance btw.</p>

crcrpar: <p>Hi,</p>
<p>I think making <code>Dataset</code> instance to return a batch of metadata with images (and targets) is a way if the meta-features can be represented in <code>torch.Tensor</code>.</p>
<p>For example, if we use the dataset defined below, in each iteration a <code>DataLoader</code> will generate a tuple of (image, target, metas), where each item is a <code>torch.Tensor</code>.</p>
<pre><code class="lang-python">import torch

class DatasetWithMeta(torch.utils.data.IterableDataset):

    def __init__(self, inputs, targets, metas):
        super().__init__()
        self.inputs = inputs
        self.targets = targets
        self.metas = metas

    def __getitem__(self, index):
        return (self.inputs[index], self.targets[index], self.metas[index])
</code></pre>

TinfoilHat0: <p>Perfect. Just one thing though: I believe it needs to inherit from torch.utils.data.Dataset, i.e.,<br>
<code>class DatasetWithMeta(Dataset)</code></p>
Suggestion: Helpbot for Pytorch Forums

J_Johnson: <p>I’ve noticed that many questions here are often repackaged duplicates. So perhaps a Helpbot which can accomplish the following would be useful for the Pytorch forums:</p>
<ol>
<li>Writes responses with Auto-links to the correct documentation in Pytorch Docs(this will help with Google Search results). For instance, any mention of autograd will link to <a href="https://pytorch.org/docs/stable/autograd.html" class="inline-onebox" rel="noopener nofollow ugc">Automatic differentiation package - torch.autograd — PyTorch 1.8.0 documentation</a>.</li>
<li>Links to answers that likely resolve the issue or may be closely related(this will help the questioner and Google search results).</li>
<li>Provides statistical information to Docs maintainers on what questions are asked the most so that the documentation can be reviewed and improved on.</li>
</ol>
Converting code from Tensorflow, getting wrong results (odd implementation of gradient calc)

Zador: <p>Hi, I am trying to implement some code from Tensorflow, but I am uncertain on how to implement their method of backpropogation. Here is their code:</p>
<pre><code class="lang-auto">       self.IRK_weights = np.reshape(tmp[0:q**2+q], (q+1,q))
       self.x0_tf = tf.placeholder(tf.float32, shape=(None, self.x0.shape[1]))
       self.u0_tf = tf.placeholder(tf.float32, shape=(None, self.u0.shape[1]))
       self.dummy_x0_tf = tf.placeholder(tf.float32, shape=(None, self.q)) #dummy variable for fwd_gradients


def fwd_gradients_0(self, U, x):        
        g = tf.gradients(U, x, grad_ys=self.dummy_x0_tf)[0]
        return tf.gradients(g, self.dummy_x0_tf)[0]


def net_U0(self, x):
        nu = 0.01/np.pi
        U1 = self.neural_net(x, self.weights, self.biases)
        U = U1[:,:-1]
        U_x = self.fwd_gradients_0(U, x)
        U_xx = self.fwd_gradients_0(U_x, x)
        F = -U*U_x + nu*U_xx
        U0 = U1 - self.dt*tf.matmul(F, self.IRK_weights.T)
        return U0
</code></pre>
<p>The key part that I am struggling to understand is where they calculate the x derivative of u and use the function fwd_gradients_0, where they also use a dummy variable. Here is how I tried to implement it which is seemingly wrong:</p>
<pre><code class="lang-auto">x.requires_grad_(True)
s = nn.Tanh()(self.fc1(x.float()))
s = nn.Tanh()(self.fc2(s))
s = nn.Tanh()(self.fc3(s))
u_1 = self.fc4(s)

u = u_1[:, :-1]
u_x = autograd.grad(u, x, torch.ones(u.shape).to(device), create_graph=True)[0]
u_xx = autograd.grad(u_x, x, torch.ones(u_x.shape).to(device), create_graph=True)[0]
f = - u * u_x + (0.01/np.pi)*u_xx
u_0 = u_1 - dt * torch.mm(f.double(), IRK_weights.T.double())
</code></pre>
<p>Any help would be greatly appreciated to help me understand/solve this!</p>

Zador: <p>Can’t edit but following what they did I in fact solved the problem (sorry to those who tried).</p>
<pre><code class="lang-auto">def fwd_gradients_0(self, U, x):
        z = torch.ones(U.shape).to(device).requires_grad_(True)
        g = autograd.grad(U, x, grad_outputs=z, create_graph=True)[0]
        return autograd.grad(g, z, grad_outputs=torch.ones(g.shape).to(device), create_graph=True)[0]
</code></pre>
<p>Can’t say I understand this though. What does taking the derivative w.r.t. the previous grad_output function z mean?</p>

demacdo: <p>Hey, thanks for posting your solution to this, works for me. Did you ever figure out why they take the gradients that way? I’m also trying to figure out the discrete PINNs code.</p>

Zador: <p>No never. Let me know if you ever do though!</p>

sunny1: <p>Hi, Did you guys find out to code for the derivative part? I am also trying to solve PINN problem.</p>
How to find differential in Pytorch?

sunny1: <p>I have a model with u(x,t) and i am training my model with x[100,1] and t[100,1] as tensor [100,2] to predict the u(x,t). I need to find the partial derivative of u wrt to t and double differential wrt to x to solve the PDE. I am stuck how to go for this part. I tried this piece of code. **’’’ u_pred = pn(y) f_pred = differential(u_pred,x,t) def differential(u_pred,x,t): f = [] for i in range(len(x)+1):</p>
<pre><code class="lang-auto">    dt = torch.autograd.grad(u_pred[i], t[i],allow_unused=True)[0]
    dx = torch.autograd.grad(u_pred[i], x[i], create_graph=True)[0]
    
    ddx = torch.autograd.grad(dx, x)[0]
    f_i = ddx - dt
    f = np.append(f,f_i)
return f
</code></pre>
<p>‘’’** Please help me.</p>
Would you prefer high dice score or high accuracy?

angelica-sjsu: <p>Hi all,<br>
This is more for my curiosity.<br>
I’m training a UNET model from scratch. I see it can reach 92% pixelwise accuracy while reaching 0.72 dice score.<br>
I noticed that sometime accuracy can be high but dice score is lower than the best dice score.<br>
Would you prefer to save the model when accuracy get higher or when it’s the dice score that rises?</p>
How to define a model inside a class?

Surayuth_Pintawong: <p>I modify some layers and I want to create a class for them. For example,</p>
<pre><code class="lang-auto">class mod:
    class linear(nn.Module):
       def __init__(self, d_int, d_out):
           pass
       def forward(self, x):
           pass

</code></pre>
<p>When I try to initialize this layer by</p>
<pre><code class="lang-auto">layer = mod.linear(2, 3)
</code></pre>
<p>I get an error saying</p>
<pre><code class="lang-auto">super(type, obj): obj must be an instance or subtype of type
</code></pre>
<p>Is there anyway to solve this problem?</p>
Tokenizer.batch_encode_plus uses all my RAM

Pizzabread: <p>I only have 25GB RAM and everytime I try to run the below code my google colab crashes. Any idea how to prevent his from happening. Batch wise would work? If so, how does that look like?</p>
<pre><code class="lang-auto">max_q_len = 128
max_a_len = 64

def batch_encode(text, max_seq_len):
  return tokenizer.batch_encode_plus(
      text.tolist(),
      max_length = max_seq_len,
      pad_to_max_length=True,
      truncation=True,
      return_token_type_ids=False
  )

# tokenize and encode sequences in the training set
tokensq_train = batch_encode(train_q, max_q_len)
tokens1_train = batch_encode(train_a1, max_a_len)
tokens2_train = batch_encode(train_a2, max_a_len)
</code></pre>
<p>My Tokenizer is from Huggingface</p>
<pre><code class="lang-auto">tokenizer = BertTokenizerFast.from_pretrained('bert-base-multilingual-uncased')
</code></pre>
<p>len(train_q) is 5023194 (which is the same for train_a1 and train_a2)</p>
Error building PyTorch from source

HashedViking: <p>Hello, I’m trying to build PyTorch from source with prepacked Metal backend <a href="https://pytorch.org/tutorials/prototype/ios_gpu_workflow.html#conclusion" rel="noopener nofollow ugc">as explained here</a> but have following error building PyTorch</p>
<p><strong>Python 3.8.5</strong><br>
<strong>cmake version 3.19.7</strong></p>
<pre><code class="lang-auto">[ 69%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/metal/MetalCommandBuffer.mm.o
[ 69%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/metal/MetalConvParams.mm.o
[ 69%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/metal/MetalGuardImpl.cpp.o
[ 69%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/metal/MetalPrepackOpRegister.cpp.o
[ 70%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/metal/MetalTensorImplStorage.mm.o
/pytorch/aten/src/ATen/native/metal/MetalCommandBuffer.mm:8:1: error: cannot synthesize weak property
      in file using manual reference counting
@implementation MetalCommandBuffer {
^
/pytorch/aten/src/ATen/native/metal/MetalCommandBuffer.h:6:48: note: property declared here
@property(nonatomic, weak, readonly) NSThread* thread;
                                               ^
1 error generated.
make[2]: *** [caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/metal/MetalCommandBuffer.mm.o] Error 1
make[2]: *** Waiting for unfinished jobs....
324 warnings generated.
make[1]: *** [caffe2/CMakeFiles/torch_cpu.dir/all] Error 2
make: *** [all] Error 2
Traceback (most recent call last):
  File "setup.py", line 876, in &lt;module&gt;
    build_deps()
  File "setup.py", line 354, in build_deps
    build_caffe2(version=version,
  File "/pytorch/tools/build_pytorch_libs.py", line 58, in build_caffe2
    cmake.build(my_env)
  File "l/pytorch/tools/setup_helpers/cmake.py", line 345, in build
    self.run(build_args, my_env)
  File "/pytorch/tools/setup_helpers/cmake.py", line 140, in run
    check_call(command, cwd=self.build_dir, env=env)
  File "/opt/anaconda3/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '8']' returned non-zero exit status 2.
</code></pre>

HashedViking: <p>OK, changing <code>weak</code> to <code>assign</code> fixes the problem:<br>
Inside <code>MetalCommandBuffer.h</code><br>
change<br>
<code>@property(nonatomic, weak, readonly) NSThread* thread;</code><br>
to<br>
<code>@property(nonatomic, assign, readonly) NSThread* thread;</code></p>
What would be the fastest most efficient way to pad an NCHW tensor with random values along it's H and W dimensions?

ProGamerGov: <p>I’m thinking of something like <code>torch.nn.ConstantPad2d()</code> only instead a padding with a constant value, each value added by the padding is random.</p>

KFrank: <p>Hi Pro!</p>
<aside class="quote no-group" data-username="ProGamerGov" data-post="1" data-topic="115362" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/progamergov/48/3670_2.png" class="avatar"> ProGamerGov:</div>
<blockquote>
<p>each value added by the padding is random.</p>
</blockquote>
</aside>
<p>At the cost of generating a full random tensor, you could use a sentinel<br>
value and <code>torch.where()</code>:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; torch.__version__
'1.7.1'
&gt;&gt;&gt; t = torch.ones ((2, 1, 3, 3))
&gt;&gt;&gt; sentinel = 999.9
&gt;&gt;&gt; p = torch.nn.ConstantPad2d (1, sentinel)(t)
&gt;&gt;&gt; torch.where (p == sentinel, torch.randn (p.shape), p)
tensor([[[[ 0.2294,  0.6565,  1.3840,  0.8816,  0.6239],
          [-0.1138,  1.0000,  1.0000,  1.0000,  1.4669],
          [-0.1995,  1.0000,  1.0000,  1.0000, -1.4677],
          [ 0.7898,  1.0000,  1.0000,  1.0000, -2.0605],
          [-1.1524, -0.6035, -0.0759,  0.7622,  0.0670]]],


        [[[-0.7629, -0.0068, -1.5791,  0.6111,  0.0061],
          [-2.1073,  1.0000,  1.0000,  1.0000,  0.3191],
          [-1.1106,  1.0000,  1.0000,  1.0000,  0.2106],
          [-0.1326,  1.0000,  1.0000,  1.0000,  0.3036],
          [ 0.0869, -0.6228, -0.0596, -2.4246,  1.1754]]]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
MC dropout - strange behavior

Gholle: <p>Hi,</p>
<p>I’ve noticed a strange behavior when sampling my network with MC dropout. Currently, I am using T=100 stochastic forward passes and sometimes observe a very low epistemic uncertainty (variance in the prediction over the T samples) and sometimes a very high.  Not very strange maybe.</p>
<p>What is strange is that when I try to repeat 10 such samplings (each iteration: 100 samples on the same input) then each iteration will either only produce low epistemic uncertainty or only high (they typically always surround the same value) over the 10 repeats. I would rather expect them to be mixed or yield something in between.</p>
<p>I try to figure out why this is and how I can solve this, but am pretty clueless…</p>
<p>Tried different seeds but the same pattern keeps occurring.</p>
<p>Any tips or suggestions?</p>
Indexing tensor with another tensor

zzzf: <p>I want to index a tensor with another tensor, but get an empty tensor with a different shape when the tensor is uint8:</p>
<pre><code class="lang-auto">t = torch.ones(3,12,12)
i = torch.zeros(3, dtype=torch.uint8)
print(t[:, i[1], i[2]])  # tensor([], size=(3, 0, 12, 12))
print(t[i[0], i[1], i[2]])  # tensor([], size=(0, 3, 12, 12)) 
print(t[0,0,0])  # tensor(1.)

t = torch.ones(3,12,12)
i = torch.zeros(3, dtype=torch.int8)
print(t[:, i[1], i[2]]) # tensor([1., 1., 1.])
print(t[i[0], i[1], i[2]])  # tensor(1.)
print(t[0,0,0])  # tensor(1.)
</code></pre>
<p>My torch version is 1.7.1.</p>

ragnaarok: <p>You must index with long type I think</p>
<pre><code class="lang-auto">t = torch.ones(3,12,12)
i = torch.zeros(3, dtype=torch.long)
print(t[:, i[1], i[2]])  # tensor([], size=(3, 0, 12, 12))
print(t[i[0], i[1], i[2]])  # tensor([], size=(0, 3, 12, 12)) 
print(t[0,0,0])  # tensor(1.)
</code></pre>
Use argmax as index for a new array

samsja: <p>Hi,</p>
<p>I thought first that using the argmax method on a tensor as a mask for the same tensor would be identical as the output of the max method. I was wrong.</p>
<p>I want to get the value of the index for the tensor and for an other tensor, so I really need to be able to use the argmax as indexes.</p>
<p>right now I achieved it for a matrix :</p>
<pre><code class="lang-auto">import torch

a = torch.arange(50).view(5,10)
b = torch.arange(50).view(5,10)
cx,c = b.max(dim=0)

assert (b[c,torch.arange(10)] == cx).all()

a = torch.arange(50).view(1,5,10)
b = torch.arange(50).view(1,5,10)
cx,c = b.max(dim=1)


assert (b[:,c,torch.arange(10)] == cx).all()


a = torch.arange(250).view(1,5,5,10)
b = torch.arange(250).view(1,5,5,10)
cx,c = b.max(dim=1)

assert (b[:,c,[torch.arange(5),torch.arange(10)] ] == cx).all()
</code></pre>
<p>I get the following error in the last assert :</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
&lt;ipython-input-118-390e76989bea&gt; in &lt;module&gt;
----&gt; 1 b[:,c,[torch.arange(5),torch.arange(10)] ] == cx

TypeError: only integer tensors of a single element can be converted to an index
</code></pre>
<p>I am a bit stuck and I don’t want to use python loop ( for speed purpose).</p>
<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

googlebot: <p>(b.gather(1,c.unsqueeze(1))==cx)</p>

samsja: <p>Hi,</p>
<p>it works thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Using loss from previous epoch to train current epoch

Nischal1: <p>I’m guessing using</p>
<p>optimizer.zero_grad()</p>
<p>current_error = nn.CrossEntropy (y_pred,y_true)<br>
total_error = current_error + previous_error</p>
<p>total_error.backward()<br>
optimizer.step()</p>
<p>doesn’t work.</p>

Dwight_Foster: <p>Have you tried this to see if it works?</p>
Possible Bug with torch.symeig()

Qiyao_Wei: <p>Hi all,</p>
<p>Even though torch.symeig() seems to give accurate results most of the time, I ran a test case today and noticed that the eigenvalues and eigenvectors don’t make sense. My Pytorch version is 1.8.0, see following minimal reproducible example—</p>
<p>import torch<br>
a = torch.tensor([[1,3,4],[2,2,4],[8,7,6]]).float()<br>
d,q = torch.symeig(a, eigenvectors=True)<br>
print(“matrix is”, a)<br>
print(“smallest eigenvalue looks like”, torch.matmul(a,q[:,0])/q[:,0])<br>
print(“matrix times smallest eigenvector”, torch.matmul(a,q[:,0]))<br>
print(“smallest eigenvalue times eigenvector”, q[:, 0]*d[0])<br>
print(“smallest eigenvector looks like”, q[:, 0])<br>
print(“smallest eigenvalue looks like”, d[0])</p>
<p>For which the results look like—</p>
<p>matrix is tensor([[1., 3., 4.],<br>
[2., 2., 4.],<br>
[8., 7., 6.]])<br>
smallest eigenvalue looks like tensor([ -1.6292,   0.0677, -11.9950])<br>
matrix times smallest eigenvector tensor([-1.3801, -0.0338,  2.1880])<br>
smallest eigenvalue times eigenvector tensor([-1.3801,  0.8133,  0.2972])<br>
smallest eigenvector looks like tensor([ 0.8471, -0.4992, -0.1824])<br>
smallest eigenvalue looks like tensor(-1.6292)</p>
<p>One could easily calculate eigenvalues and eigenvectors from such websites as (<a href="https://matrixcalc.org/en/vectors.html#eigenvectors%28%7B%7B1,3,4%7D,%7B2,2,4%7D,%7B8,7,6%7D%7D%29" class="inline-onebox" rel="noopener nofollow ugc">Eigenvalues and Eigenvectors</a>), and the results doesn’t look like a numerical issue to me. Is there something I am missing here? Thanks!</p>

KFrank: <p>Hi Qiyao!</p>
<aside class="quote no-group" data-username="Qiyao_Wei" data-post="1" data-topic="115303" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/qiyao_wei/48/34623_2.png" class="avatar"> Qiyao_Wei:</div>
<blockquote>
<p>a = torch.tensor([[1,3,4],[2,2,4],[8,7,6]]).float()<br>
d,q = torch.symeig(a, eigenvectors=True)</p>
</blockquote>
</aside>
<p>Your matrix, <code>a</code>, is not symmetric, but <code>torch.symeig()</code> expects a<br>
symmetric matrix to be passed in.  Use <code>torch.eig()</code> for a general,<br>
square, non-symmetric matrix.</p>
<p>(In practice, I believe that <code>symeig()</code> just uses the upper triangle of<br>
the matrix you pass in.)</p>
<p>Best.</p>
<p>K. Frank</p>