akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
DataLoader and Dataset to shuffle from slice level, not subject level

Junhao_Wen: <p>Hi,</p>
<p>I am creating a Class based on Dataset, in the function of <code>__getitem__</code>, I read each MRI and get all slices (100 slices from a MRI) into a list. Then the dataset was fitted into the DataLoader:</p>
<pre><code class="lang-python">train_loader = DataLoader(data_train,
                              batch_size=options.batch_size,
                              shuffle=True,
                              num_workers=options.num_workers,
                              drop_last=True,
                              pin_memory=True)
</code></pre>
<p>The problem is that If I set shuffle=True, the batch data is shuffled based on subject-level, and for example, If the batch_size is 16, it will give me 16 different subjects, repeating 100 times…. Actually, I do not wanna this behavior because the slices were not reallly shuffled… Do you have any ideas to shuffle from the slice level???</p>
<p>I have tried to read only one slice in the function of  <code>__getitem__</code>, but when I train the mode, it is super slow…</p>
<p>Any idea would be appreciated…</p>

smth: <p>you can write a custom Sampler, and shuffle in a more fine-grained way. See the <code>sampler</code> keyword argument, instead of <code>shuffle=True</code>. You can see some of the samplers here: <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.Sampler" rel="nofollow noopener">https://pytorch.org/docs/stable/data.html#torch.utils.data.Sampler</a></p>
<p>They are quite simple to implement, so you can implement your custom sampler that will be more aware of your dataset’s slicing that you want.</p>

Junhao_Wen: <p><a class="mention" href="/u/smth">@smth</a>  Actually, I have thought about this solution, but the problem is that the <code>__len__</code> of DataLoader and the <code>__len__</code></p>
<p>of sampler that I created were not equal, I do not see the possibility to handle that situation with the sampler.</p>
<p>I also tried to extract only one slice using <code>__getitem__</code> from the whole MRI. The problem for this approach is that the memory exploses at some time during training…</p>

smth: <p>hmmm, how about you set batch_size=1 in the DataLoader, but your custom dataset itself returns a full batch everytime <code>__getitem__</code> is called? That way you can carefully choose shuffling and other aspects by yourself in the Dataset.</p>

gkrisp9: <p>Hi, I am facing a similar problem at the time. Have you solved this problem ?</p>
Training LSTM word generator

Ozymandias: <p>I am trying to implement a simple character level names generator, the input word is provided to the model as one hot encoded tensor of shape (word_characters, 24), where the 24 is the size of the vocabulary characters.</p>
<pre><code class="lang-auto">class LSTM(nn.Module):
    def __init__(self):
        super(LSTM, self).__init__()
        
        self.n_hidden = 32
        self.n_layers = 3
        
        self.lstm = nn.LSTM(
            input_size=chars_count,
            hidden_size=self.n_hidden,
            num_layers=self.n_layers,
            batch_first=True
        )
        self.linear = nn.Linear(in_features=self.n_hidden, out_features=chars_count)
    
    def forward(self, x):
        h0 = torch.zeros((self.n_layers, x.size(0), self.n_hidden))     # x.size(0) is batch size, i.e. 1
        c0 = torch.zeros((self.n_layers, x.size(0), self.n_hidden))
        
        output, state = self.lstm(x, (h0, c0))
        output = self.linear(output)
        
        return output, state

model = LSTM()
loss = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

for i in range(1000):
    for xi, yi in dataset:
        # add batch size of 1
        xi = xi[None, :, :]  # shape = [1, 7, 24]
        yi = yi[None, :, :]  # shape = [1, 7, 24]

        optimizer.zero_grad()
        l = loss(model(xi)[0], yi)
        l.backward()
        optimizer.step()
</code></pre>
<p>Whenever I execute the training loop, I get the following error raised from this line<br>
<code>l = loss(model(xi)[0], yi)</code></p>
<pre><code class="lang-auto">
**ValueError** : Expected target size (1, 24), got torch.Size([1, 7, 24])
</code></pre>
<p>Can anyone point me what am I doing wrong? the shape of the output of my model is the same as the training label shape.</p>

HallerPatrick: <p>Hey,</p>
<p>I stripped down your problem to a minimum and ran it:</p>
<pre><code class="lang-python">import torch
from torch import nn
from torch import optim



class LSTM(nn.Module):
    def __init__(self):
        super(LSTM, self).__init__()
        
        self.n_hidden = 32
        self.n_layers = 3
        
        self.lstm = nn.LSTM(
            input_size=24,
            hidden_size=self.n_hidden,
            num_layers=self.n_layers,
            batch_first=True
        )
        self.linear = nn.Linear(in_features=self.n_hidden, out_features=24)
    
    def forward(self, x):
        h0 = torch.zeros((self.n_layers, x.size(0), self.n_hidden))
        c0 = torch.zeros((self.n_layers, x.size(0), self.n_hidden))
        
        output, state = self.lstm(x, (h0, c0))
        output = self.linear(output)
        
        return output, state

model = LSTM()
loss = nn.CrossEntropyLoss()

t = torch.zeros(1, 7, 24)
t1 = torch.zeros(1, 7, 24)

print(t.size()) # [1, 7, 24]
print(t1.size()) # [1, 7, 24]

output, hidden = model(t)
l = loss(output, t1)

</code></pre>
<p>For me this does not throw an error.</p>
<p>You may want to look into your tensor sizes again?</p>
<p>Greetings,<br>
Patrick</p>
PyTorch version of `skimage.draw.polygon`? coordinates to mask

aspiringcoder: <p>Hello everyone.</p>
<p>I’ve been trying quite a bit to figure out a solution here.</p>
<p>I’d like to go from a set of points to a masked image representation of these points. (Filling the pixels within)</p>
<p><code>skimage.draw.polygon</code> allows me to do just that.</p>
<p>Is there anyone that can assist with a PyTorch implementation?</p>
<p>Many thanks!</p>

ptrblck: <p>The internal <a href="https://github.com/scikit-image/scikit-image/blob/ed642e2bc822f362504d24379dee94978d6fa9de/skimage/draw/_draw.pyx#L195-L245"><code>_polygon</code></a> method used in <code>skimage</code> seems to iterate the passed coordinates and appends the indices which are inside the polygon, so you could try to adapt this code in PyTorch and reuse it.</p>

aspiringcoder: <p>Good find! Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> - Somehow I did not think of looking into the <code>.pyx</code>-file. This is awesome. Will post my version as its complete.</p>

Yanda_Meng: <p>I am also facing the same problem, can I ask if you have solved it ?</p>

aspiringcoder: <p>I haven’t yet.</p>
<p>I’m dissatisfied with the <code>skimage</code> implementation, as it loops over all the pixels in the image.</p>
<p>Ideally I think the operation should be vectorized, and this I haven’t had time for at present.</p>
Best practises for "turning off" parts of the model?

aspiringcoder: <p>Hi all,</p>
<p>I’m trying to do some ablation studies (or sensitivity studies in some other jargon), turning on and off different components of my model.</p>
<p>For now I’ve assumed that to avoid passing the input through a certain layer (say an <code>nn.Embedding</code>) it would be sufficient to avoid gradient contributions from that component of the model.</p>
<p>That will evidently make my model parameters larger than necessarily, with some of the layers not being updated past their initialisation.</p>
<p>Something tells me this is not exactly the best way to do these things. What would be your take? Would you rather define a specific model just for that component you want to turn on/off?</p>

Varal7: <p>You could do something like</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def init(self, args):
        self.m1 = Module1()
        if args.use_my_second_layer:
             self.m2 = Module2()
    def forward(self, x):
        h = self.m1(x)
        if args.use_my_second_layer:
             h = self.m2(x)
        return h
</code></pre>

aspiringcoder: <p>Yeah, I’ve tried something similar. But I personally think this can make your model become incredibly bloated.</p>
<p>I think its “easier” to comprehend something like</p>
<pre><code class="lang-python">class Model(nn.Module):

    def __init__(self,args) -&gt; None:
        super(Model).__init__()

        self.args = args

        self.m1 = Module1()

        self.m2 = Module2()

    def forward(x):
        
        match self.args.logic:
            case 'm1':
                x = self.m1(x)
            case 'm2':
                x = self.m2(x)
            case 'm1m2':
                x = self.m1(x)
                x = self.m2(x)
                
        return x
</code></pre>
<p>But I’m also thinking the above is not best practise either lol</p>
Error: torch.cuda.is_available() is True but .cuda() is failed

bbo: <p>I used docker to build an environment to reproduce the experiment.<br>
The environment is as follows.</p>
<p>torch==1.2.0<br>
torchvision==0.4.0</p>
<p>The contents of the docker file are as follows.</p>
<pre><code class="lang-auto">FROM nvidia/cuda:9.2-cudnn7-devel-ubuntu18.04

RUN apt-get update 
RUN apt-get install -y python3 python3-pip

# install PyTorch == 1.2.0
RUN pip3 install torch==1.2.0+cu92 -f https://download.pytorch.org/whl/torch_stable.html

# install Pillow to install torchvision
RUN python3 -m pip install --upgrade pip
RUN python3 -m pip install --upgrade Pillow

# install torchvision == 0.4.0
RUN pip3 install torchvision==0.4.0+cu92 -f https://download.pytorch.org/whl/torch_stable.html

RUN apt-get install -y vim

WORKDIR /workspace

ENV LIBRARY_PATH /usr/local/cuda/lib64/stubs
</code></pre>
<p>I was able to build the environment, but it froze when I ran the code. I tried to figure out what was causing it, and realized that .cuda() might be doing something wrong.<br>
So, I checked torch.cuda.is_available() and found that it is True, but .cuda() is not available. I am beginnner for pytorch, so I don’t know the cause of this.</p>
<p>Here is the actual bug trace I did.<br>
This is running in a docker container.</p>
<pre><code class="lang-auto">$ cat /usr/local/cuda/version.txt 
CUDA Version 9.2.148

$ python3   
Python 3.6.9 (default, Dec  8 2021, 21:08:43) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.cuda.is_available()
True
&gt;&gt;&gt; torch.__version__
'1.2.0+cu92'
&gt;&gt;&gt; torch.cuda.device_count()
1
&gt;&gt;&gt; torch.cuda.get_device_name()
'A100-PCIE-40GB'
&gt;&gt;&gt; torch.cuda.current_device()
0
&gt;&gt;&gt; torch.version.cuda
'9.2.148'

&gt;&gt;&gt; import torchvision
&gt;&gt;&gt; torchvision.__version__
'0.4.0+cu92'

&gt;&gt;&gt; T = torch.tensor([[1,2],[3,4]])
&gt;&gt;&gt; T = T.cuda()

</code></pre>
<p>Freeze when “”“T = T.cuda()”"" is executed.</p>

Caipi: <p>Maybe try different cuda versions? Cuda version 9.x might be available but not suitable for your GPU? And also why are your torch distributions so low? The newest stable torch is 1.10 i guess.</p>

bbo: <p>Thank you for your reply.<br>
I’m trying it now with version 10.0, but I’m not sure if it will work yet.<br>
The reason for the torch version 1.2.0 is that I want to match the environment of the paper I want to reproduce.</p>
How to Fine Tune own pytorch model

rajarajan: <p>Hello there am a new to pytorch , my problem is I have to fine tune my own model . I have seen example of fine tuning the Torch Vision Models , like downloading  the .pth  and start training it.</p>
<p>Like wise I have my own .pth file and Neural Network model , I want to do fine tuning .<br>
I kindly request you help with an example for my own model.</p>

imbibekk: <p>If you have your own <code>.pth</code> model file then just load it and finetune for the number of epochs you want.</p>
<pre><code class="lang-auto">import torch

model = get_model()
checkpoint = torch.load(path_to_your_pth_file)
model.load_state_dict(checkpoint['state_dict'])

finetune_epochs = 10 # number of epochs you want to finetune
for epoch in range(finetune_epochs):
         # assuming you have functions for training and validating models 
         train_model(model)   
         validate_model(model)</code></pre>

rajarajan: <p>hello there and thanks for the immediate  reply .<br>
I tried your above steps , I get the following error ,please help on this .</p>
<p>path_save="./model_new_10_300_0.005.pth"<br>
model = torch.hub.load(‘pytorch/vision:v0.5.0’, ‘resnext50_32x4d’, pretrained=False)<br>
for param in model.parameters():<br>
param.requires_grad = False</p>
<p>model.fc = nn.Linear(2048, num_classes)<br>
model<br>
print(model)<br>
checkpoint=torch.load(path_save)<br>
model.load_state_dict(checkpoint[‘state_dict’])</p>
<p>And the error is<br>
model.load_state_dict(checkpoint[‘state_dict’])<br>
KeyError: ‘state_dict’</p>

ma3252788: <p>How to modify the number of categories in the last layer?</p>

BaruchG: <p>You would access the fully connected layer and just replace it with the output argument being the number of classes that your new task has.  For example:</p>
<pre><code class="lang-auto">model = get_model()
checkpoint = torch.load(path_to_your_pth_file)
model.load_state_dict(checkpoint['state_dict'])
model.fc = nn.Linear(2048, 10) #input is whatever the output of prior layer is and output is the number of classes that you have
</code></pre>
<p>You can then decide how you would like to proceed with the training step since your last layer has random weights and the rest of your model has trained weights.  You can either just train the last layer or the whole network.</p>

Muhammad_Hussain: <p>This is not working for me.</p>
<p>‘key error: state_dict’???</p>
<p>Any idea?</p>

ptrblck: <p><code>checkpoint['state_dict']</code> assumes that <code>checkpoint</code> is a <code>dict</code> with the <code>state_dict</code> key.<br>
This doesn’t seem to be the case in your setup, so check what <code>checkpoint</code> is and what it contains.</p>