akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Cross entropy Tensor Dimensions

username: anotherone_one
cooked: <p>Hello!!<br>
I am having trouble figuring out what is wrong in here! I am having this error on the dimensions of the tensors on the Cross Entropy Loss (a classic, I know <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"> ).</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss()
classification_loss = criterion(y_logit.unsqueeze(1), y.long())
</code></pre>
<p><code>IndexError: Target 1 is out of bounds.</code></p>
<p>The dimensions and the tensors are the following (batch_size = 10):</p>
<pre><code class="lang-auto">y_logit  -&gt; 
tensor([1.0267, 0.0967, 1.1793, 1.0542, 1.4097, 1.5651, 1.5124, 1.2934, 1.9106, 0.9233])
y_logit shape -&gt; torch.Size([10, 1])

y -&gt; tensor([1., 0., 1., 1., 0., 1., 1., 0., 1., 0.])
y shape -&gt; torch.Size([10])
</code></pre>
<p>The y_logits is the output of a linear layer, without any sigmoid of softmax applied. Not sure if needed tho! and I converted it into tensor with the following line, before running the criterion: <code>y_logit = torch.from_numpy(y_logit)</code></p>
<p>Does anyone know the answer of this?<br>
Thank you in advance!</p>

username: anotherone_one
cooked: <p>Update: I had to change to nn.BCEWithLogitsLoss(), since it is not multi-class classification.</p>
Adding the tensor values based on indices

username: Neeraj_Varshney
cooked: <p>I have a tensor of values and a corresponding tensor of indices. I want to sum all the values where the index is not a special token (tokenizers).</p>
<p>For instance:</p>
<pre><code class="lang-auto">
values=tensor([1.0000, 0.1574, 0.1507, 0.2520, 0.2456, 0.2365, 0.2330, 0.2294, 0.2321,
         0.2339], grad_fn=&lt;MaxBackward0&gt;),
indices=tensor([32099,    12,    24,     3,     3,     3,     3,     3,     3,     3])
</code></pre>
<p>I want sum(0.1574, 0.1507)<br>
What is the optimal way to achieve this?</p>

username: ptrblck
cooked: <p>You could most likely create a mask using the non-special indices, apply it to the tensor, and sum it.<br>
I’m not sure, how the “spacial” indices are defined and thus how complicated it would be to “find” them.</p>

username: DoubtWang
cooked: <p>According to your description and my assume, I can use two steps to solve it.</p>
<ol>
<li>mask indices</li>
<li>select value<br>
we assume that id 3 and 32099 are the special token id.</li>
</ol>
<pre><code class="lang-auto">&gt;&gt;&gt; values
tensor([1.0000, 0.1574, 0.1507, 0.2520, 0.2456, 0.2365, 0.2330, 0.2294, 0.2321,
        0.2339])
&gt;&gt;&gt; indices
tensor([32099,    12,    24,     3,     3,     3,     3,     3,     3,     3])
&gt;&gt;&gt; mask = ~torch.eq(~torch.eq(indices, 3), torch.eq(indices, 32099))
&gt;&gt;&gt; mask
tensor([0, 1, 1, 0, 0, 0, 0, 0, 0, 0], dtype=torch.uint8)
&gt;&gt;&gt; torch.masked_select(values, mask)
tensor([0.1574, 0.1507])
&gt;&gt;&gt; torch.sum(torch.masked_select(values, mask))
tensor(0.3081)
</code></pre>
Lung nodules with CT images(dicom)

username: fighting_lhj
cooked: <p>I am a newcomer to deep learning. My tutor gave me the CT data of 30 patients a few days ago. I want to detect whether these CT images contain lung nodules, but the original data provided is not marked. I want to ask, everyone has done lung nodule recognition. Can you provide some ideas? I would be very grateful.</p>
Am I using Autocast() correctly with A2C?

username: realiti
cooked: <p>Hello,</p>
<p>I added autocast to this a2c model. It is %30 faster, and it looks like it trains well, but can’t be sure if I am doing something wrong here. I would be grateful if you can point it out any mistakes. Thanks.</p>
<p>‘’’<br>
def train_agent():</p>
<pre><code>state = env.reset()

state = torch.from_numpy(state).float().to(device)

for i_episode in range(episode_start, 4000000):       # 1000000

    log_probs = []
    values    = []
    rewards   = []
    masks     = []
    entropy = 0

    states = torch.zeros([num_steps, 720, env.shape[1]]).to(device)       

    actions = []
    dones = []
    final_reward = []

    for step in range(num_steps):

        states[step] = state

        with autocast():

            with torch.no_grad():                  

                value, action, action_log_prob = model.act(state.unsqueeze(1))

        action = action.item()

        next_state, reward, done, info = env.step(action)            

        rewards.append(reward)

        dones.append(0 if done else 1)

        actions.append(action)

        if done:

            final_reward.append(info)            

        state = next_state

        state = torch.from_numpy(state).float().to(device)

    with autocast():

        with torch.no_grad():

            next_value, _ = model(state.unsqueeze(1))            

        

        returns = compute_returns(next_value, rewards, dones)

        returns = torch.cat(returns)

        actions = torch.tensor(actions).unsqueeze(1).to(device)

        states = states.permute(1, 0, 2)

        values, action_log_probs, dist_entropy = model.evaluate_actions(states, actions)            

        advantages = returns - values

        value_loss = advantages.pow(2).mean()

        action_loss = -(advantages.detach() * action_log_probs.unsqueeze(1)).mean()
        
        value_loss_coef = 0.5

        entropy_coef = 0.01

        loss = (value_loss * value_loss_coef + action_loss - dist_entropy * entropy_coef)

    scaler.scale(loss).backward()        

    scaler.unscale_(optimizer)

    max_grad_norm = 0.5

    nn.utils.clip_grad_norm_(model.parameters(), max_grad_norm)        

    scaler.step(optimizer)

    scaler.update()

    optimizer.zero_grad()
</code></pre>
<p>‘’’</p>

username: ptrblck
cooked: <p>Your code looks correct and</p>
<aside class="quote no-group" data-username="realiti" data-post="1" data-topic="113368">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/4491bb/48.png" class="avatar"> realiti:</div>
<blockquote>
<p>It is %30 faster, and it looks like it trains well</p>
</blockquote>
</aside>
<p>is the best indication of it <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

username: realiti
cooked: <p>Thanks a lot for checking it, it is great to hear.</p>
Fill value of torchvision.transforms.functional RandomRotate

username: af_waschbaer
cooked: <p>Hi there,</p>
<p>I am confused with the documents inside <code>torchvision.transforms.functional</code> <code>RandomRotate</code>.</p>
<p><code>Fill value for the area outside the transform in the output mage is always 0.</code></p>
<p>How should I understand this behaviour?</p>
<p>Best</p>

username: ptrblck
cooked: <p>When you rotate an image you might get “borders”, which have to be filled with a specific value, which is set to <code>0</code> in this case.</p>
Change targets of torchvision dataset

username: mrdb1376
cooked: <p>Hello there.<br>
I have some data in data folder and I read them like this<br>
<code>path = os.path.join(os.getcwd(),'../../kitti/dataset/sequences/',seq)</code><br>
<code>data_in_path = torchvision.datasets.ImageFolder(path)</code><br>
the problem is that in my task the labels are not classification targets. each data has its own target which is a float tensor by shape (6,1).<br>
how can I change the classification targets in the dataset i have loaded?<br>
I also tried writing custom dataset and reading the image samples. but they were just the path of images. Is it possible for the dataloader to read images from their addresses?<br>
Thanks</p>

username: ptrblck
cooked: <p>I think the best approach would still be to write a custom <code>Dataset</code> and read the images from their path by using e.g. <code>PIL.Image.open</code>. In fact, the <code>ImageFolder</code> does the same as seen <a href="https://github.com/pytorch/vision/blob/0ec5f31f9cd17db9414f8d9d38cda64dc7e64d70/torchvision/datasets/folder.py#L193-L197">here</a>.<br>
I don’t know how the targets are created, but I assume you could either create or load them using the same <code>index</code> as used for the image paths.</p>

username: mrdb1376
cooked: <p>Thanks for answering Peter.<br>
Yes. The indexes are the same.<br>
I will try the solution you mentioned and write here about it.</p>
Closing libtorch threads

username: nitya05
cooked: <p>Hi,<br>
I am using libtorch-cpu version 1.5.1 for loading a model built in Python and then running forward and backward propagation on it. I noticed that these operations are creating multiple Pytorch threads. I tried to control these threads using at::set_num_threads(1). But, that doesn’t seem to work.</p>
<p>Also, is there a way to cleanly close the threads of PyTorch? Because I can see the threads even after the task is complete.</p>

username: nitya05
cooked: <p>Hi <span class="mention">@ptrblck_de</span>! Do you have any idea about this?</p>

username: ptrblck
cooked: <p>Unfortunately, I don’t know how threads are exactly used in <code>libtorch</code>.<br>
However, this</p>
<aside class="quote no-group" data-username="nitya05" data-post="1" data-topic="112593">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/ecb155/48.png" class="avatar"> nitya05:</div>
<blockquote>
<p>Because I can see the threads even after the task is complete.</p>
</blockquote>
</aside>
<p>sounds like a bug.<br>
Could you post a minimal, executable code snippet to reproduce this issue?</p>

username: nitya05
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
Sorry for replying late. I was experimenting with the threads of PyTorch and was trying to write a small code snippet to post here, but that helped me realize that there isn’t any problem as such. If I do inference or training tasks in a separate thread and close the thread after all the tasks are complete, the PyTorch thread pool also exits cleanly.<br>
The problem I was facing was because I was initially testing on a single threaded application which remains active always, and because of that the PyTorch threads were not exiting.</p>
<p>Thanks for your help.</p>
Could not infer dtype of CSNet

username: hazel_pacheco
cooked: <p>Hi i’m trying a code from CSNet and i have the next problem…when i a use torch.load there’s an error, now i changed it for torch.tensor and there’s another problem</p>
<p>def PSNR(pred, gt, shave_border=0):<br>
height, width = pred.shape[:2]<br>
pred = pred[shave_border:height - shave_border, shave_border:width - shave_border]<br>
gt = gt[shave_border:height - shave_border, shave_border:width - shave_border]<br>
imdff = pred - gt<br>
rmse = math.sqrt(np.mean(imdff ** 2))<br>
if rmse == 0:<br>
return 100<br>
return 20 * math.log10(255.0 / rmse)</p>
<p><span class="hashtag">#opt</span> = parser.parse_args()<br>
cuda = cuda</p>
<p>if cuda and not torch.cuda.is_available():<br>
raise Exception(“No GPU found, please run without --cuda”)</p>
<p>model = CSNet(block_size,sub_rate)</p>
<p>if model != ‘’:<br>
model.load_state_dict(torch.tensor(model))</p>
<p>image_list = glob.glob(opt.dataset+"/<em>.</em>")</p>
<p>avg_psnr_predicted = 0.0<br>
avg_elapsed_time = 0.0</p>
<p>for image_name in image_list:<br>
print("Processing ", image_name)<br>
im_gt_y = sio.loadmat(image_name)[‘im_gt_y’]</p>
<pre><code>im_gt_y = im_gt_y.astype(float)

im_input = im_gt_y/255.

im_input = Variable(torch.from_numpy(im_input).float()).view(1, -1, im_input.shape[0], im_input.shape[1])

if cuda:
    model = model.cuda()
    im_input = im_input.cuda()
else:
    model = model.cpu()

start_time = time.time()
res = model(im_input)
elapsed_time = time.time() - start_time
avg_elapsed_time += elapsed_time

res = res.cpu()

im_res_y = res.data[0].numpy().astype(np.float32)

im_res_y = im_res_y*255.
im_res_y[im_res_y&lt;0] = 0
im_res_y[im_res_y&gt;255.] = 255.
im_res_y = im_res_y[0,:,:]

psnr_predicted = PSNR(im_gt_y, im_res_y,shave_border=0)
print(psnr_predicted)
avg_psnr_predicted += psnr_predicted
</code></pre>
<p>print(“Dataset=”, opt.dataset)<br>
print(“PSNR_predicted=”, avg_psnr_predicted/len(image_list))<br>
print(“It takes average {}s for processing”.format(avg_elapsed_time/len(image_list)))</p>

username: ptrblck
cooked: <p>Which line of code creates this issue?<br>
Note that you can post code snippets by wrapping them into three backticks ``` or by using the “Preformatted text” button, which would make debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
I need to test how create a new topic

username: qindi
cooked: <h1>test</h1>
<pre><code class="lang-auto">java python
</code></pre>

username: qindi
cooked: <p>I want to test formula : $x^4$</p>

username: qindi
cooked: <pre><code class="lang-auto">$H=-\sum_{i=1}^np(x_{i})\log_2 p(x_{i})$
</code></pre>
I dont know how to pop up graph and training accuracy coding

username: zaimzaimi
cooked: <p>My question is about how to pop up graph for training and validation accuracy. Below are the codes that i use. I have try many ways but still cannot pop up graph. Can anyone help me with this <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
<h1>Train the model</h1>
<p><code>def train(n_epochs, loaders, model, optimizer, criterion, use_cuda, save_path):     '''returns trained model'''</code></p>
<pre><code># Initialize tracker for minimum validation loss
valid_loss_min = np.inf

for epoch in range(1, n_epochs+1):
    # In the training loop, I track down the loss
    # Initialize variables to monitor training and validation loss
    train_loss = 0.0
    valid_loss = 0.0

    # Model training
    model.train()
    for batch_idx, (data,target) in enumerate(trainloader):
        # 1st step: Move to GPU
        if use_cuda:
            data,target = data.cuda(), target.cuda()
  
        # Then, clear (zero out) the gradient of all optimized variables
        optimizer.zero_grad()
        # Forward pass: compute predicted outputs by passing inputs to the model
        output = model(data)
        # Perform the Cross Entropy Loss. Calculate the batch loss.
        loss = criterion(output, target)
        # Backward pass: compute gradient of the loss with respect to model parameters
        loss.backward()
        # Perform optimization step (parameter update)
        optimizer.step()
        # Record the average training loss
        train_loss = train_loss + ((1/ (batch_idx + 1 ))*(loss.data-train_loss))
  
    # Model validation
    model.eval()
    for batch_idx, (data,target) in enumerate(validloader):
        # Move to GPU
        if use_cuda:
            data, target = data.cuda(), target.cuda()
        # Update the average validation loss
        # Forward pass: compute predicted outputs by passing inputs to the model
        output = model(data)
        # Calculate the batch loss
        loss = criterion(output, target)
        # Update the average validation loss
        valid_loss = valid_loss + ((1/ (batch_idx +1)) * (loss.data - valid_loss))
  
    # print training/validation stats
    print('Epoch: {} \tTraining Loss: {:.5f} \tValidation Loss: {:.5f}'.format(
        epoch,
        train_loss,
        valid_loss))

    # Save the model if validation loss has decreased
    if valid_loss &lt;= valid_loss_min:
        print('Validation loss decreased ({:.5f} --&gt; {:.5f}). Saving model ...'.format(
              valid_loss_min,
              valid_loss))
        torch.save(model.state_dict(), 'model_transfer.pt')
        valid_loss_min = valid_loss

# Return trained model
return model
</code></pre>
<h1>Define loaders transfer</h1>
<p>loaders_transfer = {‘train’: trainloader,<br>
‘valid’: validloader,<br>
‘test’: testloader}</p>
<h1>Training the model</h1>
<p>model_transfer = train(50, loaders_transfer, model_transfer, optimizer_transfer, criterion_transfer, use_cuda, ‘model_transfer.pt’)</p>
Fairseq with MLFlow for tracking the training jobs

username: Maven123456
cooked: <p>Is there any documentation or reading material available where fairseq has been used with MLFlow for tracking training experiments?</p>
<p>I have raised the GitHub questions already the link is <a href="https://github.com/pytorch/fairseq/issues/3268" rel="noopener nofollow ugc">here</a>. I am re-posting this question here to get some more view form the community.</p>
<p>Thanks,</p>
[Solved]Conversion to LongTensor

username: Niranjan_Ramesh_Rao
cooked: <p>What happens to the tensor when I change its type from <strong>Float</strong> to <strong>Long</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/4/3484e668fc69ff9a4ac06cef5ee83a3314221517.png" data-download-href="https://discuss.pytorch.org/uploads/default/3484e668fc69ff9a4ac06cef5ee83a3314221517" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/3484e668fc69ff9a4ac06cef5ee83a3314221517_2_517x373.png" alt="image" data-base62-sha1="7uBz3IPUBIrjTmikrRcnmdDgVAH" width="517" height="373" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/3484e668fc69ff9a4ac06cef5ee83a3314221517_2_517x373.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/3484e668fc69ff9a4ac06cef5ee83a3314221517_2_775x559.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/3484e668fc69ff9a4ac06cef5ee83a3314221517_2_1034x746.png 2x" data-dominant-color="B3B3B3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1104×797 91.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is supposed to be a segmentation mask of an image, and it gets completely wrecked when I change it to LongTensor.</p>

username: ptrblck
cooked: <p>Since you are working with floating point values, you would have to be careful about transforming these values to <code>long</code>, as you might lose all information.<br>
E.g. assuming your tensor contains values in <code>[0, 1]</code> converting them to <code>long</code> could round all values to 0:</p>
<pre><code class="lang-python">x = np.random.uniform(0, 1, (100, 100, 3))
plt.imshow(x) # shows valid image
plt.imshow(x.astype(np.int64)) # blank image
print(np.unique(x.astype(np.int64)))
&gt; array([0], dtype=int64)
</code></pre>
Automatically detect input size

username: jorgerodpen
cooked: <p>I’m trying to build a multilayer perceptron for sentiment classification. I am using skorch for cross validating and to integrate a pipeline that performs the hashing trick. I want to optimise the number of features in the hashing trick and therefore the input dimension is going to change every time I change that value. I was wondering if there is a way to automatically detect the input size within the model class.</p>
<p>This is my model:</p>
<pre><code class="lang-auto">class MLP(nn.Module):
    def __init__(self, input_dim=5000, num_hidden = 1, hidden_dim=1, output_dim=4, dropout=0.5):
        # Building the network from here
        super(MLP, self).__init__()
        
        # Hidden layers
        self.linears = nn.ModuleList([nn.Linear(input_dim, hidden_dim) if i==0 else nn.Linear(hidden_dim, hidden_dim) for i in range(3)])
        
        # Output layer
        self.ol = nn.Linear(hidden_dim, output_dim)
        
        # Activation functions
        self.dropout = nn.Dropout(dropout)
    
    def forward(self, data, **kwargs):
        # To float
        X = data.float()
        
        # Hidden layers
        for i, hl in enumerate(self.linears):
            X = self.linears[i](X)
            X = F.relu(X)
            X = self.dropout(X)
        
        # Output layer
        out = self.ol(X)
        out = F.softmax(out, dim = -1)
        
        return out
</code></pre>
<p>Currently it has the input size as an argument but I want the code to automatically detect it. How can I do that?</p>

username: ptrblck
cooked: <p>I don’t think it would make a difference, but you could pass the expected input tensor to the <code>__init__</code> method and get the feature dimension via <code>input_dim = x.size(1)</code> assuming the features are in <code>dim1</code>.<br>
Alternatively, you could also reuse your code and use <code>model = MLP(x.size(1), ...)</code> instead.</p>
Custom data sampling (sequences from different recordings)

username: miturian
cooked: <p>Hi, I have some particular needs for how to sample my training data, and am hoping someone can suggest a way to do it while still benefitting from the built in utilities:</p>
<p>Dataset: N recordings, each consisting of variable number of data points</p>
<p>Network input: L consecutive data points (L much smaller than each recording). Let’s call this an L-sequence.</p>
<p>Ideally, for each batch I would like to draw m L-sequences, from random places in my data set. I would like each L-sequence to have a random midpoint, so sequence overlaps within the same epoch would be allowed. For this reason, I would like to be able to stop an epoch before all possible L-sequences have been drawn (otherwise I am passing each data point to the network L times, which is probably overkill).<br>
I also need my L-sequences to not cross boundaries between recordings.</p>
<p>L is small enough that setting m=1 incurs a pretty hefty performance cost (each epoch takes about 3 times as long, compared to the optimal batch size), so I would like to pass several L-sequences to the GPU at a time.</p>
<p>Is there a way to get most of this while still using the data loader class?</p>

username: ptrblck
cooked: <p><a href="https://discuss.pytorch.org/t/dataloader-for-a-lstm-model-with-a-sliding-window/22235/2">This post</a> explains a simple windowed <code>Dataset</code>, where non-overlapping windows would be created, while <a href="https://discuss.pytorch.org/t/how-upload-sequence-of-image-on-video-classification/24865/9">this post</a> is a bit more complicated in order to avoid “mixing” data samples from different sources.<br>
Maybe one of these two approaches could be useful for you.</p>
[URGENT]Sigmoid/Softmax for Multiclass Segmentation- Colab Attatched

username: Niranjan_Rao
cooked: <aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1c3pD-UXNjaHDafB5In5v8z2dpkp4bXsW?usp=sharing">
  <header class="source">
      <img src="https://ssl.gstatic.com/colaboratory-static/common/f6c736cbde16632a9fcd16d9ab1970b1/img/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1c3pD-UXNjaHDafB5In5v8z2dpkp4bXsW?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img width="260" height="260" src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar">

<h3><a href="https://colab.research.google.com/drive/1c3pD-UXNjaHDafB5In5v8z2dpkp4bXsW?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Please find link to the colab notebook. It has been well annotated and hope you can help me!</p>
<p>I am implementing UNET architecture for segmenting input image into 4 classes.</p>
<p><em>Input 900 grayscale images</em> → <strong>[900, 1, 256, 256]</strong><br>
<em>Mask/Ground Truth</em>               → <strong>[900, 1, 256, 256]</strong></p>
<p>Each pixel in the mask has one of the following values - <strong>0,1,2,3 (4 classes)</strong></p>
<p>I understand the output shape of my model should be:<br>
<strong>[batch_size, nb_classes, height ,width].</strong></p>
<p>In my case model outputs tensor of size = <strong>[batch_size, 4, 256, 256]</strong></p>
<p>After training using CrossEntropy Loss, I try to make predictions using: <img src="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4b3d67c22a795d03cf45648ffddf3bc5f54cd986.png" alt="image" data-base62-sha1="aJBmYnvlf5sKIzoKHERK9OcGgV8" width="650" height="225"></p>
<p>Prediction for a single image <strong>y_pred</strong> has a shape <strong>[4,256,256]</strong> whereas my mask shape is <strong>[1, 256, 256]</strong></p>
<p>How can I compare my predictions and ground truth when the shapes are mismatched.<br>
Ideally my prediction should be <strong>[1,256,256]</strong> with 4 unique values(nb_class=4)</p>
<p>I read that I am not supposed to use torch.sigmoid() with nn.CrossEntropy loss. What is the right way to get predictions?<br>
Also want to clarify if I have to one hot encode or map my masks.</p>

username: ptrblck
cooked: <p><code>nn.CrossEntropyLoss</code> expects raw logits as the model output in the shape <code>[batch_size, nb_classes, height, width]</code> for a multi-class segmentation use case, so you should remove the <code>sigmoid</code>.<br>
Also, the target should be a <code>LongTensor</code> in the shape <code>[batch_size, height, width]</code> containing the class indices in the range <code>[0, nb_classes-1]</code>.<br>
To get the predicted class labels you could use <code>pred = torch.argmax(output, dim=1)</code>.</p>
SubsetRandomSampler not splitting data correctly

username: Arvind_Subramaniam
cooked: <p>My training data has a size pf 72200, and I want to split it into 5 portions so that I can feed it in without maxing out the RAM during training. I have split my data using subsetRandomSampler, but when I check the lengths of the subsets, its not 14440 (72200), its 57.</p>
<pre><code class="lang-auto">print(train_loader.__len__())
num_partions = 5
total = train_loader.__len__()
partitions = []
partitions = [int(i*total/num_partions) for i in range(num_partions+1)]
print(partitions)

for j in range(len(partitions)-1):
  indices = range(total)
  samples = indices[partitions[j]:partitions[j+1]]
  train_sampler = SubsetRandomSampler(samples)
  train_loader_p1 = DataLoader(MyDataset(train, 12), batch_size=256, num_workers=8, pin_memory=True,sampler=train_sampler)
  print(train_loader_p1.__len__(),total)
</code></pre>
<p>This is the output I am getting:</p>
<pre><code class="lang-auto">72200
[0, 14440, 28880, 43320, 57760, 72200]
57 72200
57 72200
57 72200
57 72200
57 72200
</code></pre>
<p>The subsets have a length of 57 instead of 14440. I have checked the length of samples, and its 14440. Where am I going wrong? Also, is there a better way to do the same using random_split?</p>

username: ptrblck
cooked: <p>I guess you are mixing up the number of samples, which is given as the length of the <code>Dataset</code>, and the number of batches, which is given as the length of the <code>DataLoader</code>.</p>
<p>Based on the posted numbers, I assume <code>train_loader</code> uses a batch size of 1, so that <code>len(dataset)==len(train_loader)</code>.<br>
However, inside the loop you are using a batch size of 256. Assuming the subset has now 14440 samples, the <code>DataLoader</code> would yield 57 batches (<code>14440/256=56.40625</code>).</p>
Torchvision Datasets

username: chotog22
cooked: <p>Hello Everyone,</p>
<p>I had a question about Torchvision Datatsets. Are these datasets its own class or is it just a link where you can download the files? I’m trying run SRGAN sample program and it uses torchvision dataset but I wanted to run my own dataset but getting errors. Any help would be greatly appreciated! Thank you!</p>
<p>USING_STL = True<br>
if USING_STL:<br>
DatasetSubclass = torchvision.datasets.STL10<br>
else:<br>
DatasetSubclass = torchvision.datasets.ImageNet</p>
<p>class Dataset(DatasetSubclass):<br>
def <strong>init</strong>(self, *args, **kwargs):<br>
hr_size = kwargs.pop(‘hr_size’, [96, 96])<br>
lr_size = kwargs.pop(‘lr_size’, [24, 24])<br>
super().<strong>init</strong>(*args, **kwargs)</p>
<p>How do I replace STL10 or ImageNet dataset with my own dataset?</p>

username: Dwight_Foster
cooked: <p>The torchvision datasets are part of the dataset class but they specifically download their own dataset from the web so you cannot use them for different datasets. If you want to create your own dataset you can look at this tutorial <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="noopener nofollow ugc">here</a> or if your data is in the correct format where each folder is a class you can use the imagefolder class <a href="https://pytorch.org/vision/stable/datasets.html#imagefolder" rel="noopener nofollow ugc">here</a>.</p>

username: chotog22
cooked: <p>Thank you! I’ll definitely take a look.</p>
Type Error: only integer tensors of a single element can be converted to an index

username: Oguzhan
cooked: <p>I wrote a dataloader and dataset class that below</p>
<pre><code class="lang-auto">class Dataset():
    def __init__(self, x, y,z,u): 
        self.x = x+z
        self.y = torch.tensor([y] * len(x)+[u] * len(z) )
    def __len__(self): 
        return len(self.x)
    def __getitem__(self, i): 
        return self.x[i],self.y[i]

class DataLoader():
    def __init__(self, ds, bs): 
        self.ds, self.bs = ds, bs
    def __iter__(self):
        n = len(self.ds)
        l = torch.randperm(n) 
        for i in range(0, n, self.bs): 
            idxs_l = l[i:i+self.bs]
            yield self.ds[idxs_l]
</code></pre>
<p>And I get this error for  <code>yield self.ds[idxs_l]</code></p>
<p>*TypeError                                 Traceback (most recent call last)<br>
 in <br>
----&gt; 1 batch = next(iter(dataloader))</p>
<p> in <strong>iter</strong>(self)<br>
7         for i in range(0, n, self.bs):<br>
8             idxs_l = l[i:i+self.bs]<br>
----&gt; 9             yield self.ds[idxs_l]</p>
<p> in <strong>getitem</strong>(self, i)<br>
8         return len(self.x)<br>
9     def <strong>getitem</strong>(self, i):<br>
—&gt; 10         return self.x[i],self.y[i]</p>
<p>TypeError: only integer tensors of a single element can be converted to an index*</p>
<p>What should I do?</p>

username: Dwight_Foster
cooked: <p>I’m and not 100% sure if this is what you want but you could try</p>
<pre><code class="lang-auto">yield self.ds.gather(DIM, torch.tensor(idxs_l))
</code></pre>
<p>where dim is the dimension that you want to index on.</p>
RuntimeError: Given groups=1, weight of size [64, 3, 1, 7, 7], expected input[1, 64, 8, 64, 114] to have 3 channels, but got 64 channels instead

username: Darshan
cooked: <p>Hi pytorch Team,</p>
<p>I am converting the Slowfast pytorch model to ONNX model using torch.onnx.export().<br>
During Conversion, I am getting an error as: <strong>RuntimeError: Given groups=1, weight of size [64, 3, 1, 7, 7], expected input[1, 64, 8, 64, 114] to have 3 channels, but got 64 channels instead</strong>.<br>
Having said that, the pytorch model Inference is working fine, but do no why it is throwing above error during .pt to .onnx conversion.</p>
<p>Can you please assist me to solve this error? Would be a great help from yourside.</p>
<p><strong>Environment</strong>:<br>
Pytorch version: 1.6.0<br>
Torchvision: 0.7.0<br>
ONNX version: 1.7.0</p>
<p>One Note: Model is Slowfast. The input shape is: input0: (1,3,8,256,455), input1: (1,3,32,256,455). input0 is Slow pathway, and input1 is Fast pathway.</p>
<p>I am attaching the screenshot of the error below:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/6/36bd184721327d9bc57be362c9d19b58156a2bff.png" data-download-href="https://discuss.pytorch.org/uploads/default/36bd184721327d9bc57be362c9d19b58156a2bff" title="Screenshot from 2021-03-02 01-21-56"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/36bd184721327d9bc57be362c9d19b58156a2bff_2_690x388.png" alt="Screenshot from 2021-03-02 01-21-56" data-base62-sha1="7OeURNkOh3GRUBhIrHjwx9hfJYb" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/36bd184721327d9bc57be362c9d19b58156a2bff_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/36bd184721327d9bc57be362c9d19b58156a2bff_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/36bd184721327d9bc57be362c9d19b58156a2bff_2_1380x776.png 2x" data-dominant-color="39192F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-03-02 01-21-56</span><span class="informations">1920×1080 370 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks,<br>
Darshan</p>

username: Dwight_Foster
cooked: <p>Can you show your model code.</p>
How to get the intermediate layers output 'model.feature_list(input)' for any customized CNN like we do it in a pretrained ResNet34?

username: Kalyan
cooked: <p>I am using Resnet34 pre-trained model and trying to get the intermediate layer outputs by doing  model.feature_list(input)  (input.shape = [2,3,32,32]) the outputs are of shape [2, 64, 32, 32],  [2, 64, 32, 32], [2, 128 16 16], [2,256,8, 8], and [2, 512, 4, 4].</p>
<p>How can I write a new customized CNN code which would respond to a “model.feature_list(input)” command and provides the intermediate layer outputs.</p>

username: Dwight_Foster
cooked: <p>You could pass through each individual block. Something like this</p>
<pre><code class="lang-auto">x1 = model.layer1(input)
x2 = model.layer2(x1)
x3 = model.layer3(x2)
x4 = model.layer4(x3)
</code></pre>
<p>etc. Then return the intermediate values.</p>
How to share parameter across different tensor operations?

username: Jake_Sokol
cooked: <p>I would like to have a list of parameters (Pa, Pb, Pc) and perform several simple matrix operations on my input using these parameters.</p>
<p>For instance, consider x = tensor([1, 2, 3]). I would like to perform the operation x * tensor([Pa, Pa, Pb]) * tensor([Pb, Pc, Pa]).</p>
<p>How can I accomplish this?</p>
Efficient Weight Sharing

username: bilunsun
cooked: <p>I would like to implement a layer with weight sharing, that performs the matrix multiplication <code>M @ x</code>, with</p>
<pre><code class="lang-auto">M = [
    [a, b]
    [c, a]
]
</code></pre>
<p>where <code>a</code>, <code>b</code>, and <code>c</code> are the unique weights.</p>
<p>How can I implement this layer?<br>
I tried instantiating M as a <code>nn.Linear</code> module, then doing <code>M.weight[1][1] = M.weight[0][0]</code>. While this works for weight sharing, the forward pass complexity is still the same as a non-weight-sharing layer; same for the model size.</p>
<p>Any help would be greatly appreciated, thanks.</p>
How to access modules by name

username: Alfred
cooked: <p>Hi,<br>
I’m writing a function that computes the sparsity of the weight matrices of the following fully connected network:</p>
<pre><code class="lang-auto">class FCN(nn.Module):
    def __init__(self):
        super(FCN, self).__init__()

        
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        self.relu1 = nn.ReLU()
        self.fc2 = nn.Linear(hidden_dim, hidden_dim)
        self.relu2 = nn.ReLU()
        self.fc3 = nn.Linear(hidden_dim, hidden_dim)
        self.relu3 = nn.ReLU()
        self.fc4 = nn.Linear(hidden_dim, output_dim)
    
    def forward(self, x):

        out = self.fc1(x)
        out = self.relu1(out)
        out = self.fc2(out)
        out = self.relu2(out)
        out = self.fc3(out)
        out = self.relu3(out)
        out = self.fc4(out)

        return out
</code></pre>
<p>The function I have written is the following:</p>
<pre><code class="lang-auto">def print_layer_sparsity(model):
    for name,module in model.named_modules():
        if 'fc' in name:
            zeros = 100. * float(torch.sum(model.name.weight == 0))
            tot = float(model.name.weight.nelement())
            print("Sparsity in {}.weight: {:.2f}%".format(name, zeros/tot))
</code></pre>
<p>But it gives me the following error:</p>
<blockquote>
<p>torch.nn.modules.module.ModuleAttributeError: ‘FCN’ object has no attribute ‘name’</p>
</blockquote>
<p>It works fine when I manually enter the name of the layers (e.g.,</p>
<blockquote>
<p>(model.fc1.weight == 0)<br>
(model.fc2.weight == 0)<br>
(model.fc3.weight == 0) …</p>
</blockquote>
<p>but I’d like to make it independent from the network. In other words, I’d like to adapt my function in a way that, given any sparse network, it prints the sparsity of every layer. Any suggestions?</p>
<p>Thanks!!</p>

username: leoentersthevoid
cooked: <p>Have you tried using <code>model.children()</code>?<br>
Afterward, you can pick the child (= layer) and do <code>layer.parameters()</code>.</p>
<p>Naive way:</p>
<pre><code class="lang-auto">for layer in model.children():
    weights = list(layer.parameters())[0] # because it's a generator
    # do stuff
</code></pre>

username: Alfred
cooked: <p>Do you know if<code> layer.parameters()</code> take care of masked weights? For example, <code>model.parameters()</code> does not</p>

username: leoentersthevoid
cooked: <p>What do you mean by taking care of the masked weights?</p>

username: Alfred
cooked: <p>Sorry, I thought I had specified it. The weights are masked using a pruning function (such as <code>prune.l1_unstructured(module, name="bias", amount=3)</code>.</p>

username: Brando_Miranda
cooked: <p>here: <a href="https://discuss.pytorch.org/t/how-to-access-to-a-layer-by-module-name/83797/6" class="inline-onebox">How to access to a layer by module name? - #6 by klory</a></p>
How does one setp up the set_sharing_strategy strategy for multiprocessing?

username: Brando_Miranda
cooked: <p>where does one run the following:</p>
<pre><code class="lang-auto">import torch.multiprocessing
torch.multiprocessing.set_sharing_strategy('file_system')
</code></pre>
<p>suppose to be ran by every process or only the main one?</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/var/spool/pbs/mom_priv/jobs/486213.iam-pbs.SC", line 321, in &lt;module&gt;
    main_distributed()
  File "/var/spool/pbs/mom_priv/jobs/486213.iam-pbs.SC", line 239, in main_distributed
    mp.spawn(fn=parallel_train, args=(opts,), nprocs=opts.world_size)
  File "/homes/miranda9/.conda/envs/automl-meta-learning/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 199, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/homes/miranda9/.conda/envs/automl-meta-learning/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 148, in start_processes
    process.start()
  File "/homes/miranda9/.conda/envs/automl-meta-learning/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/homes/miranda9/.conda/envs/automl-meta-learning/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/homes/miranda9/.conda/envs/automl-meta-learning/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/homes/miranda9/.conda/envs/automl-meta-learning/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/homes/miranda9/.conda/envs/automl-meta-learning/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 58, in _launch
    self.pid = util.spawnv_passfds(spawn.get_executable(),
  File "/homes/miranda9/.conda/envs/automl-meta-learning/lib/python3.8/multiprocessing/util.py", line 427, in spawnv_passfds
    errpipe_read, errpipe_write = os.pipe()
OSError: [Errno 24] Too many open files

</code></pre>
<p>ref:</p>
<ul>
<li><a href="https://github.com/pytorch/pytorch/issues/11201" class="inline-onebox" rel="noopener nofollow ugc">Too many open files error · Issue #11201 · pytorch/pytorch · GitHub</a></li>
<li><a href="https://pytorch.org/docs/stable/multiprocessing.html#file-descriptor-file-descriptor" class="inline-onebox" rel="noopener nofollow ugc">Multiprocessing package - torch.multiprocessing — PyTorch 1.7.1 documentation</a></li>
<li><a href="https://stackoverflow.com/questions/66426199/how-does-one-setup-the-set-sharing-strategy-strategy-for-multiprocessing-in-pyto" class="inline-onebox" rel="noopener nofollow ugc">python - How does one setup the set_sharing_strategy strategy for multiprocessing in Pytorch? - Stack Overflow</a></li>
</ul>
Pytorch made many process

username: sangheonlee
cooked: <p>I was running the imagenet using DDP.</p>
<p>At the DGX station 4*V100, they make many processes<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/8/f8576d381bb067175921e3bebe67911fedf2a599.png" alt="캡처2" data-base62-sha1="zqVIQFg1rSXmneZUIwrpykanIuZ" width="372" height="180"></p>
<p>But i running the same option and same program on RTX2080Ti*8 server. They make just only 9 Process</p>
<p>I running the same code and same program… If I think the same case on RTX2080TI*8 server, I expect DGX to also make 5 processes. But it was not.</p>
<p>Why this problem, I used the same PyTorch docker on both systems.</p>
<p>And If I want to drive the number of processes, how do I contorl it?</p>

username: ptrblck
cooked: <p>What are these processes assigned to?<br>
Could you extend the drop down menu and compare both machines?<br>
Also, how many CPUs does the RTX2080Ti workstation have? Note that your DGX station reports 40, which might also be the reason why it’s able to use more processes.</p>

username: sangheonlee
cooked: <p>Now I guess, this problem is xorg or other process default program which was running at the same time. Thanks to helping me.</p>
<p>Can I ask another problem…?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/5/15057bfa306d059f1305357c565478a83209288c.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/15057bfa306d059f1305357c565478a83209288c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/15057bfa306d059f1305357c565478a83209288c_2_583x500.jpeg" alt="image" data-base62-sha1="2ZXMar68bYYvFSMsE8gESmbfaXy" width="583" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/15057bfa306d059f1305357c565478a83209288c_2_583x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/5/15057bfa306d059f1305357c565478a83209288c_2_874x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/5/15057bfa306d059f1305357c565478a83209288c.jpeg 2x" data-dominant-color="E5E8ED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">882×756 140 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Now I guess very long allreduce and Broadcast which caused by late Memcpy H2D. If you see the above figure  GPU 1,3,4 call the H2D(=green bar) at the similar time, but GPU 2 called the H2D at late. These unsync behavior make inefficient at multi-GPU training.</p>
<p>So all GPU sync on GPU2, can I solve this problem…? For example, I allocated the memcpyH2D to new stream or any nice idea…?</p>
<p>Thanks.</p>
Time/Memory keeps increasing at every iteration

username: suraj.pt
cooked: <h4>How are you measuring time?</h4>
<p>If each new iteration is taking longer, first make sure you’re measuring run time accurately <a href="https://discuss.pytorch.org/t/how-to-measure-execution-time-in-pytorch/111458" class="inline-onebox">How to measure execution time in PyTorch?</a>. Without properly synchronizing GPU tasks, your measurements will be inaccurate.</p>
<h4>Using <code>detach()</code> to reduce Autograd operations</h4>
<p>At every iteration of your network training, PyTorch constructs a computational graph of all the operations dynamically. This graph contains tensors that require gradients. When we call backward on a tensor (eg: <code>loss</code>), Autograd backpropagates through the graph that calculated the tensor and frees up the graph to save memory.</p>
<p>In your code, you might be creating some tensors that don’t require gradients (eg: hidden states across time steps in an RNN). If they continue to be part of the computational graph Autograd will backpropagate through them as well, resulting in increased time or memory consumption. Using <code>detach()</code> creates a view of this tensor that does not require gradients. <a href="https://discuss.pytorch.org/t/how-to-detach-specific-components-in-the-loss/13983/8">Refer to this example illustrating <code>detach()</code></a>.</p>
<p>A common error is to reference a tensor outside the training loop; this prevents the graph from freeing up after every iteration. Instead, use <code>item()</code> to just store the scalar value (or <code>detach()</code> if you need it as a tensor).</p>
<pre><code class="lang-auto">losses = []
for epoch in range(10):
    pred = model(x)
    loss = loss_fn(y, pred)
    loss.backward()
#   losses.append(loss)  # This accumulates history. AVOID!
    losses.append(loss.item())  # Do this instead
</code></pre>
<p><a href="https://pytorch.org/docs/stable/notes/faq.html#my-model-reports-cuda-runtime-error-2-out-of-memory">Read more about this FAQ</a></p>
<h4>Data Loading bottleneck</h4>
<p>Data loading is a CPU process, and can be a significant bottleneck (<a href="https://discuss.pytorch.org/t/data-loader-takes-a-lot-of-time-for-every-nth-iteration/10831" class="inline-onebox">Data loader takes a lot of time for every nth iteration</a>) . Take a look at <a class="mention" href="/u/rwightman">@rwightman</a>’s excellent set of tips to make loading data faster <a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19" class="inline-onebox">How to prefetch data when processing with GPU? - #19 by rwightman</a>.</p>
Autograd.grad computational cost for multidimensional output function

username: InfoMax
cooked: <p>I am having a hard time understanding the backpropagation computational cost of the two functionally equivalent approaches below. I have a function ‘model’ which takes n inputs and returns n outputs and I want to compute the gradient of the outputs with respects to the model parameters. I would expect approach 2 where I am computing the function n times to be slower than approach 1. However in this example approach 1 takes 9.3s vs 2.6s for approach 2.<br>
Is there some inner workings of autograd which would explain such large difference?</p>
<ol>
<li>Computing the gradients by looping over the outputs:</li>
</ol>
<pre><code class="lang-auto">outputs = model(inputs)
grads = [torch.autograd.grad(output, model.parameters(), only_inputs=True, retain_graph=True, allow_unused=True) for output in outputs.unbind(0)]
</code></pre>
<ol start="2">
<li>Computing the gradients by looping over the batch inputs:</li>
</ol>
<pre><code class="lang-auto">grads = [torch.autograd.grad(model(inputs[i]), model.parameters(), only_inputs=True, retain_graph=True, allow_unused=True) for i in range(inputs.shape[0])]
</code></pre>
RNN - different number of parameters in Keras vs. PyTorch

username: mmg
cooked: <p>Hi,</p>
<p>I am trying to build a simple RNN in Keras and PyTorch. Unfortunately, both have <strong>different number of parameters</strong>. <strong>Can someone explain why?</strong></p>
<p>In Keras:</p>
<pre><code class="lang-auto">model = keras.models.Sequential([
    keras.layers.SimpleRNN(1, input_shape=[None, 1])
])

model.summary()
# This returns 'Total params: 3'
</code></pre>
<p>In PyTorch</p>
<pre><code class="lang-auto">class SimpleRNN(nn.Module):
    def __init__(self):
        super(SimpleRNN, self).__init__()
        self.rnn = nn.RNN(input_size=1,
                         hidden_size=1,
                         num_layers=1,
                         nonlinearity='tanh',
                         batch_first=True
                         )
    def forward(self,X):
        out, _ = self.rnn(X)
        out = out[:,-1, :]
        return out
        
model_sr = SimpleRNN()

def count_parameters(model):
    return sum(param.numel() for param in model.parameters())

count_parameters(model_sr)
# returns 4
</code></pre>