akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Random paramters value for torchaudio augmentation

talhaanwarch: <p>Here is code</p>
<p>How can i randomly define the value of torchaudio augmentations parameters. For example time_mask_param value range is selected between 10-30</p>
<pre><code class="lang-auto">from torchvision import transforms as T
from torchaudio import transforms as TA
aug=T.Compose([
            TA.TimeMasking(time_mask_param=10),
            TA.FrequencyMasking(freq_mask_param=10),
            TA.TimeStretch(0.9),
            
            T.Resize((224,224))
            ])
                  
</code></pre>

ptrblck: <p>I guess <code>transforms.RandomChoice</code> (from <code>torchvision.transforms</code>) would work and you could use it via:</p>
<pre><code class="lang-python">transforms.RandomChoice([TA.TimeMasking(time_mask_param=10), TA.TimeMasking(time_mask_param=20)], p=[0.5, 0.5])
</code></pre>
Batching with variable size of list

dipta007: <p>What is the best way to batch with variable size lists?</p>
<p>Specifically, I have a <code>dataset</code> class that returns 2 lists containing multiple tensors of the same size.<br>
So it can be possible that one instance has 2 lists where the first one has 5 tensors of 200 size and the second one has 4 tensors of 200 size</p>
<p>And on another instance, the first list has 3 tensors of 200 size and the second one has 1 tensor of 200 size.</p>
<p>How to define the <code>dataloader</code> or <code>collate_fn</code> function to deal with it?</p>
How to use the weight parameter for F.cross_entropy() correctly?

Kexiii: <p>I’m trying to write some code like below:</p>
<pre><code class="lang-auto">x = Variable(torch.Tensor([[1.0,2.0,3.0]]))
y = Variable(torch.LongTensor([1]))
w = torch.Tensor([1.0,1.0,1.0])
F.cross_entropy(x,y,w)
w = torch.Tensor([1.0,10.0,1.0])
F.cross_entropy(x,y,w)
</code></pre>
<p>However, the output of cross entropy loss is always 1.4076 whatever w is. What is behind the weight parameter for F.cross_entropy()? How to use it correctly?<br>
I’m using pytorch 0.3:drooling_face:</p>

ptrblck: <p>You are using it correctly!<br>
However, I think there is an explanation missing on how <code>size_average</code> works regarding the weight in the <a href="https://pytorch.org/docs/master/nn.html#crossentropyloss">docs</a>.</p>
<p>Have a look at the docs of <a href="https://pytorch.org/docs/master/nn.html#torch.nn.NLLLoss">NLLLoss</a>. It states, that each loss will be divided by the sum of all corresponding class weights, if <code>reduce=True</code> and <code>size_average=True</code>.</p>
<p>In your case, since you just have one example, the loss will by divided by 10, which yields exactly the same result as the unweighted loss:</p>
<pre><code class="lang-auto">x = torch.Tensor([[1.0,2.0,3.0]])
y = torch.LongTensor([1])
w = torch.Tensor([1.0,1.0,1.0])
F.cross_entropy(x,y,w, size_average=False)
w = torch.Tensor([1.0,10.0,1.0])
F.cross_entropy(x,y,w, size_average=False)
F.cross_entropy(x,y,w, size_average=True)

loss = 10.0 * (-x[0, y] + torch.log(torch.sum(torch.exp(x))))
averaged_loss = loss / w[y]
</code></pre>
<p>If you create another sample, the loss will differ:</p>
<pre><code class="lang-auto">x = torch.Tensor([[1.0,2.0,3.0], [2.0, 1.0, 3.0]])
y = torch.LongTensor([1, 1])
w = torch.Tensor([1.0,1.0,1.0])
F.cross_entropy(x,y,w, size_average=False)
&gt; tensor(3.8152)
w = torch.Tensor([1.0,10.0,1.0])
F.cross_entropy(x,y,w, size_average=False)
&gt; tensor(38.1521)
F.cross_entropy(x,y,w, size_average=True)
&gt; tensor(1.9076)
</code></pre>
<p>In this example <code>38.1521</code> will be divided by the sum of the corresponding weights (<code>w[1]</code> for each sample), thus <code>38.1521/20. = 1.9076</code>.</p>

Kexiii: <p>Thanks a lot, it totally cleared up my confusion!</p>

pswarup: <p>Thanks for the help.</p>

thatwasepic: <p>Hi,<br>
argument ‘size_average’ is deprecated,<br>
instead of that now you use<br>
print(F.cross_entropy(x,y,w, reduction=‘sum’)) instead of ‘size_average’ = False<br>
print(F.cross_entropy(x,y,w, reduction=‘mean’) instead of ‘size_average’ = True</p>

47759476: <p>Thanks foryour help!!!</p>
RuntimeError: Expected to mark a variable ready only once

JWarlock: <p>Hi there!</p>
<p>I am trying to run a code that works perfectly well without distributed training, but fails with following error messages with distributed training<br>
<code>RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons: 1) Use of a module parameter outside the </code>forward<code>function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes2) Reused parameters in multiple reentrant backward passes. For example, if you use multiple</code>checkpoint<code> functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases yet.3) Incorrect unused parameter detection. The return value of the forward function is inspected by the distributed data parallel wrapper to figure out if any of the module's parameters went unused. For unused parameters, DDP would not expect gradients from then. However, if an unused parameter becomes part of the autograd graph at a later point in time (e.g., in a reentrant backward when using checkpoint), the gradient will show up unexpectedly. If all parameters in the model participate in the backward pass, you can disable unused parameter detection by passing the keyword argument find_unused_parameters=False to torch.nn.parallel.DistributedDataParallel.</code></p>
<p>I think I’ve identified the root cause of this issue, which is pytorch checkpoint in the image backbone.  I found that after disabling checkpoint (with_cp=False) in the model, the code can run smoothly. But if checkpoint is enabled (which is the default value to save memory), the program throws the above error.</p>
<p>You may see the compIete code of the backbone model <a href="https://github.com/HuangJunJie2017/BEVDet/blob/master/mmdet3d/models/backbones/swin.py" rel="noopener nofollow ugc">here</a>. Below is a simple illustrative example.</p>
<pre><code class="lang-auto">    def forward(self, x, hw_shape):
        for block in self.blocks:
            if self.with_cp:
                # not the lastest pytorch
                # see https://github.com/pytorch/pytorch/issues/63394
                # x = checkpoint.checkpoint(block, x, hw_shape)
                x = checkpoint.checkpoint(lambda x: block(x, hw_shape), x)
            else:
                x = block(x, hw_shape)
</code></pre>
<p>I wonder if there is a way to fix it?</p>
How to quantize a Swin transformer model to reduce its size?

MST007: <p>Hello,</p>
<p>I’m a beginner and I have a project where I need to quantize models using pytorch. I managed to find a way to quantize a resnet model but it seems like it doesn’t work on a Swin Tranformer model since this model has a different architecture. Has anyone any ideas on how to quantize a Swin transformer model?</p>

Zafar: <p>Can you give more context, s.a. links, codes to reproduce, etc.?</p>

MST007: <p>Basically, I was trying to apply the quantization aware training technique on a swin transformer model(Inspired by this work: <a href="https://leimao.github.io/blog/PyTorch-Quantization-Aware-Training/" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Quantization Aware Training - Lei Mao's Log Book</a> ) since it gives better accuracy than other techniques. However, Swin transformer has totally a different architecture, so it doesn’t work obviously on a Swin model.</p>

jerryzh168: <p>are you using eager mode quantization? have you tried fx graph mode quantization: <a href="https://pytorch.org/tutorials/prototype/fx_graph_mode_ptq_static.html" class="inline-onebox" rel="noopener nofollow ugc">(prototype) FX Graph Mode Post Training Static Quantization — PyTorch Tutorials 1.12.0+cu102 documentation</a>?</p>

jerryzh168: <p>also please take a look at api summary: <a href="https://pytorch.org/docs/stable/quantization.html#quantization-api-summary" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 1.12 documentation</a></p>
Pytorch dataloader transforms not working

Denis_Mbey_Akola: <p>Below is my dataloaders and transforms<br>
coco_train = = coco(root_dir=“./data/COCO”, image_set=“train”, year=2017, transforms=get_transform(train=True))<br>
coco_val =  = coco(root_dir=“./data/COCO”, image_set=“val”, year=2017, transforms=get_transform(train=False))</p>
<h1>
<a name="define-training-and-validation-data-loaders-1" class="anchor" href="#define-training-and-validation-data-loaders-1"></a>define training and validation data loaders</h1>
<p>data_loader = torch.utils.data.DataLoader(<br>
coco_train, batch_size=BATCH_SIZE, shuffle=True, num_workers=4, collate_fn=coco_train.collate_fn)<br>
data_loader_no_random = torch.utils.data.DataLoader(<br>
coco_val, batch_size=BATCH_SIZE, shuffle=False, num_workers=4,collate_fn=coco_val.collate_fn)</p>
<p>def get_transform(train):<br>
transforms = []<br>
# converts the image, a PIL image, into a PyTorch Tensor</p>
<pre><code>transforms.append(T.ToTensor())
if train:
    # during training, randomly flip the training images
    # and ground-truth for data augmentation
    transforms.append(T.RandomHorizontalFlip(0.5))
    
return T.Compose(transforms)
</code></pre>
<p>When I iterate over data_loader like this<br>
for image,target in data_loader:<br>
bla bla<br>
image is of datatype PIL.Image still even though transforms are applied in the custom dataset class coco.<br>
I appreciate any suggestions and work around this.<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>Check how the <code>__getitem__</code> of your custom <code>Dataset</code> is defined and where the transformation is supposed to be used.</p>
nn.TransformerEncoderLayer with d_model = 2

Laurent1: <p>Hi,</p>
<p>when using <code>nn.TransformerEncoderLayer()</code> with <code>d_model = 2</code>, I got unexpected (at least to me) results, something I do not have if <code>d_model != 2</code>, e.g.</p>
<blockquote>
<p>encoder_layer = nn.TransformerEncoderLayer(d_model=2, nhead=1, batch_first=True)<br>
src = torch.rand(2, 4, 2)<br>
out = encoder_layer(src)</p>
</blockquote>
<p>gives me</p>
<blockquote>
<p>tensor([[[ 1.0000, -1.0000],<br>
[ 1.0000, -1.0000],<br>
[ 1.0000, -1.0000],<br>
[ 1.0000, -1.0000]],</p>
<pre><code>    [[ 1.0000, -1.0000],
     [-1.0000,  1.0000],
     [ 1.0000, -1.0000],
     [-1.0000,  1.0000]]], grad_fn=&lt;NativeLayerNormBackward0&gt;)
</code></pre>
</blockquote>
<p>All values are stucked to either -1 or +1, something I  do not observe when <code>d_model != 2</code>.<br>
Could someone explain it?<br>
Thanks</p>
Fail to load package contains huggingface transformer in Torch::Deploy

JiayiFeng: <p>I try to dump a model contains huggingface transformer by torch package and then load it in Torch::Deploy. During dumping, everything is OK, but it fails during loading and throws the error message:</p>
<pre><code class="lang-auto">terminate called after throwing an instance of 'std::runtime_error'
  what():  Exception Caught inside torch::deploy embedded library:
Exception Caught inside torch::deploy embedded library:
ModuleNotFoundError: No module named 'torch._C._nn'; 'torch._C' is not a package

At:
  &lt;Generated by torch::deploy&gt;(436): _do_find_and_load
  &lt;Generated by torch::deploy&gt;(448): _find_and_load
  &lt;Generated by torch::deploy&gt;(478): _gcd_import
  &lt;Generated by torch::deploy&gt;(148): import_module
  &lt;Generated by torch::deploy&gt;(25): find_class
  &lt;Generated by torch::deploy&gt;(1526): load_global
  &lt;Generated by torch::deploy&gt;(1212): load
  &lt;Generated by torch::deploy&gt;(270): load_pickle

Aborted (core dumped)
</code></pre>
<p>Here is my model definition, only contains a huggingface tokenizer and a BERT model:</p>
<pre><code class="lang-python"># file name: bert_model.py
import torch
from transformers import BertTokenizer, BertModel

class MyBertModel(torch.nn.Module):
    def __init__(self) -&gt; None:
        super().__init__()
        self._tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
        self._model = BertModel.from_pretrained("bert-base-uncased")

    def forward(self, x):
        encoded_input = self._tokenizer(x, return_tensors='pt')
        return self._model(**encoded_input)
</code></pre>
<p>And here is how I dump the model:</p>
<pre><code class="lang-python">from torch.package import PackageExporter
from bert_model import MyBertModel

model = MyBertModel()

# Package and export it.
with PackageExporter("bert_package.pt") as e:
    e.intern('bert_model')
    e.extern("sys")
    e.extern("transformers.**")
    e.save_pickle("model", "model.pkl", model)
</code></pre>
<p>I load the dumped package in the way showed in the demo: <a href="https://pytorch.org/docs/1.12/deploy.html#loading-and-running-the-model-in-c" class="inline-onebox" rel="noopener nofollow ugc">torch::deploy — PyTorch 1.12 documentation</a></p>
<p>My huggingface transformers version is 4.20.1</p>
<p>Can anyone help? Many thanks!</p>

d4l3k: <p>discussion is at <a href="https://github.com/pytorch/multipy/issues/82" class="inline-onebox" rel="noopener nofollow ugc">Fail to load package contains huggingface transformer in Torch::Deploy · Issue #82 · pytorch/multipy · GitHub</a></p>
Cudatoolkit version relevant?

Robin_Lobel: <p><a href="http://pytorch.org" rel="noopener nofollow ugc">pytorch.org</a> suggest 2 cudatoolkit versions to install pytorch:</p>
<pre><code class="lang-auto">conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c conda-forge
</code></pre>
<p>But what happens if I just install the latest cudatoolkit (right now I think it’s 11.7) like this ?<br>
<code>conda install pytorch torchvision torchaudio cudatoolkit -c pytorch -c conda-forge</code></p>
<p>Will I run into compatibility issue ?</p>
<p>The reason I’m asking is that I’m automating the install of pytorch on multiple machines, and installing the latest pytorch is always <code>conda install pytorch torchvision torchaudio</code> however I can’t know in advance the cudatoolkit version. Is it a hard requirement for pytorch to specify the cudatoolkit version, will the latest cudatoolkit run just fine too ?</p>

ptrblck: <aside class="quote no-group quote-modified" data-username="Robin_Lobel" data-post="1" data-topic="155416">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/robin_lobel/48/21018_2.png" class="avatar"> Robin_Lobel:</div>
<blockquote>
<p>is will the latest cudatoolkit run just fine too ?</p>
</blockquote>
</aside>
<p>No, as we need to build the binaries for each new CUDA release. Right now, I have the builds ready and would need to enable the CUDA 11.7 binaries in the CI next.<br>
You would thus need to specify which dependency you want to use.</p>

Robin_Lobel: <p>Thanks, that’s good to know.</p>
<p>I noticed that if I’m not installing explicitly cudatoolkit (<code>conda install pytorch -c pytorch</code>), the minimum cudatoolkit with which pytorch is compatible will be installed (right now cudatoolkit 11.3).<br>
Am I correct, and is it safe to let pytorch tell conda its own cudatoolkit dependency rather than forcing it ?</p>
<p>I’m trying to find an easy, reproducible way to install the latest pytorch (now and in the future) with working CUDA support without having to specify the required cudatoolkit version. Ideally the latest supported cudatoolkit version, but I’m afraid there won’t be an easy solution for that.</p>

ptrblck: <p>I don’t know how <code>conda</code> will select the CUDA runtime version if none is specified. Maybe it’s picking whatever is the “first” it can find or maybe it uses some clever logic internally, but I don’t know.</p>
Is there any smaller version of RetinaNet?

vtanwar: <p>Hi,</p>
<p>I am new to object detection and want to train RetinaNet for my custom dataset having nearly 300 images only. I see that the model has 29M parameters and I know my dataset is very small for such large model.</p>
<p>This is why I want to use smaller versions of RetinaNet (as we have for YOLO).</p>
<p>Could anyone please tell me anything about small RetinaNet?</p>
<p>Thanks</p>
Reduced throughput with vector-add in libtorch

Marcel: <p>Hello,<br>
I aim to use libtorch as a tensor/linear algebra/machine learning library for a scientific code base. While assessing my code I noticed that I couldn’t get any throughput (measured in bandwidth Gb/s) from my code. I was able to reproduce this behavior with a simple vector add example. Here I get roughly half the peak bandwidth.  <strong>Now the question is, what can I do to improve the situation?</strong><br>
Here are the details:<br>
The <a href="https://github.com/Marcel-Rodekamp/TorchBandwidth" rel="noopener nofollow ugc">benchmark</a> I am doing looks something like</p>
<pre><code class="lang-auto">auto options = torch::TensorOptions()
                        .dtype(torch::kFloat64);

torch::Tensor T1 = torch::randn(N,options);
torch::Tensor T2 = torch::randn(N,options);
torch::Tensor T3 = torch::randn(N,options);

double start = omp_get_wtime();
T3 = T1+T2;
double end = omp_get_wtime();

std::cout &lt;&lt; "Timing: " &lt;&lt; end-start &lt;&lt; " s\n";
</code></pre>
<p>I did a similar thing with <code>std::vector</code>s to have some comparison. The <code>std::vector</code> version obtained peak bandwidth. Here are the results:</p>
<ul>
<li>Torch Vector Add</li>
</ul>
<pre><code class="lang-auto">* Memory Footprint   : 1.99931 Gb
* Min Execution Time : 0.171147 s
* Max Execution Time : 0.201715 s
* Mean Execution Time: 0.173588 +/- 0.00298119 s
* Mean Bandwidth     : 34.5527 +/- 0.593407 Gb/s
* Number threads     : 40
* use GPU            : false
</code></pre>
<ul>
<li>
<code>std::vector</code> Vector Add</li>
</ul>
<pre><code class="lang-auto">* Memory Footprint   : 1.99931 Gb
* Min Execution Time : 0.0994474 s
* Max Execution Time : 0.133856 s
* Mean Execution Time: 0.0999729 +/- 0.00342435 s
* Mean Bandwidth     : 59.9955 +/- 2.05501 Gb/s
* Number threads     : 40/40
* use GPU            : false
</code></pre>
<p>I am testing this on a dedicated node using 1 Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz CPU. For this particular test I use the default OpenMP environment. Here is my CMakeLists.txt:</p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(example-app)

find_package(Torch REQUIRED)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -O3 -fopenmp")

add_executable(bandwidth_torch bandwidth_torch.cpp)
target_link_libraries(bandwidth_torch "${TORCH_LIBRARIES}")
set_property(TARGET bandwidth_torch PROPERTY CXX_STANDARD 14)

add_executable(bandwidth_cpp bandwidth_cpp.cpp)
set_property(TARGET bandwidth_cpp PROPERTY CXX_STANDARD 14)
</code></pre>
<p>with a <code>c++ (Debian 10.2.1-6) 10.2.1 20210110</code>.<br>
The torch version I am using is installed via pip3</p>
<pre><code class="lang-auto">~&gt; pip3 show torch
Name: torch
Version: 1.10.0+cpu
</code></pre>
<p>The full code is available at <a href="https://github.com/Marcel-Rodekamp/TorchBandwidth" class="inline-onebox" rel="noopener nofollow ugc">GitHub - Marcel-Rodekamp/TorchBandwidth</a> (same link as above).</p>
Training hangs and overfitting

Sylvain_Ard: <p>Hi,<br>
randomly my training hangs, I would say that the program doesn’t advance anymore. Is it the GPU that overheat ?<br>
Second question : how to avoid overfitting with this program ?<br>
This is the program : <a href="https://www.dropbox.com/s/ch0ntqvdm312z0k/train.py?dl=0" class="inline-onebox" rel="noopener nofollow ugc">Dropbox - train.py - Simplify your life</a><br>
Thank you<br>
Best regards</p>
How do I make the edge_attr (the weights of each edge) in a Graph Learnable

Harsh_Choudhary: <p>I have a GNN where I am using the standard node level features and graph level features, right now I have my edge_attr(edge weights) as constant or they just depend upon how far the nodes are separated (I have x,y,z co-ordinates for nodes in my problem use case). Now I want those edge Attributes to be trainable. How do I do that? I tried defining these in the init function but since every batch of graph has different number of edges, the size is variable, One thing I can do is, define a large enough tensor in the init function of size (some large number, 1) and then use it’s subtensor of size (edge_attribute,1) in forward function. Is there a better way for this?</p>
PyTorch support for MACs with Intel processors

pr0t0n: <p>I am wondering if PyTorch supports or going to support accelerated computing on MACs with Intel processors (and not only for M1)?</p>

philipturner: <p>Discussion on this thread:</p>
<aside class="quote quote-modified" data-post="1" data-topic="151996">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/b5e925/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/pytorch-support-for-intel-gpus-on-mac/151996">PyTorch support for Intel GPUs on Mac</a> <a class="badge-category__wrapper " href="/c/metal-performance-shader/38"><span data-category-id="38" data-drop-close="true" class="badge-category " title="This category is for any question related to MPS support on Apple hardware (both M1 and x86 with AMD machines)."><span class="badge-category__name">mps</span></span></a>
  </div>
  <blockquote>
    This thread is for carrying on any discussion from: 


It seems that Apple is choosing to leave Intel GPUs out of the PyTorch backend, when they could theoretically support them. For reference, on the other thread, I pointed out that Apple did the same thing with their TensorFlow backend. When it was released, I only owned an Intel Mac mini and could not run GPU-accelerated TF. Other people may feel the same way, even though M1 is more common now. 
For their earliest (now archived) TF 2.4 backen…
  </blockquote>
</aside>
How can I check if two models are identical?

python_Dev: <p>I want to pass a model to my function but I am not sure whether the parameters will be identical and all the updates will be updated (weights, bias, gradient, etc).<br>
How can I check and do you suggest using the model as a global parameter or pass it to the function and then return it?</p>
<pre><code class="lang-auto">def batch_creator(features_train, labels_train, batch):
  modela.train()
  for batch_idx, (inpts, targts) in enumerate(zip(features_train, labels_train)):
    #iterate to get batch
    print('Batch: ',batch_idx+1)
    if batch_idx+1 == batch:
      inputs = inpts
      targets = targts
      break
  #create data to feed in LSTM    
  for video_idx, video in enumerate(inputs):
    print('Video: ',video_idx+1)
    samples = []
    for frame in video:
      #print(torch.unsqueeze(frame.to(device).type(torch.cuda.FloatTensor),0).shape)
      feature = modela(torch.unsqueeze(frame.to(device).type(torch.cuda.FloatTensor),0))
      samples.append(feature) #40x25088
      del feature
  return samples, targets    
</code></pre>
<p>(I am using the above function and I try to manage my available GPU ram)<br>
I believe using the model as a global variable will make sure that I wont have to worry about any changes not being updated in the model (though for now I don’t optimise the model)</p>
How to enable Subset to inherit methods from custom Dataset?

ash95: <p>My use case is very similar to the question asked on stack overflow. Link below</p>
<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/71242348/how-to-make-torch-utils-data-subset-inherit-methods-from-customly-defined-torch">
  <header class="source">

      <a href="https://stackoverflow.com/questions/71242348/how-to-make-torch-utils-data-subset-inherit-methods-from-customly-defined-torch" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/17555029/ilpedro" target="_blank" rel="noopener nofollow ugc">
    <img alt="ilPedro" src="https://lh3.googleusercontent.com/a/AATXAJxWQjpbIv4Vrow-P2xr_0dgtwjKdQjOy4CzOOY9=k-s256" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/71242348/how-to-make-torch-utils-data-subset-inherit-methods-from-customly-defined-torch" target="_blank" rel="noopener nofollow ugc">How to make torch.utils.data.Subset inherit methods from customly defined torch.utils.data.Dataset?</a>
</h4>

<div class="tags">
  <strong>inheritance, dataset, subset, torch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/17555029/ilpedro" target="_blank" rel="noopener nofollow ugc">
    ilPedro
  </a>
  on <a href="https://stackoverflow.com/questions/71242348/how-to-make-torch-utils-data-subset-inherit-methods-from-customly-defined-torch" target="_blank" rel="noopener nofollow ugc">06:33PM - 23 Feb 22 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Is there a way to achieve this?</p>
Implementing shared weights within resnet residual blocks

jessicaloke: <p>I’m trying to implement resnet with shared weights within each residual block. In the forward function of each block, I’m implemented the forward function twice for the conv2d layer. However, I’m facing an issue of the change between the number of filters between blocks. The conv2d which has an input of 64, outputs 128 and cannot be repeated in the forward function. What’s another way to implement shared weights of the convolution within the residual blocks?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/d/8df578a455a2b17d75739570b3d63a2e6e512d70.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8df578a455a2b17d75739570b3d63a2e6e512d70" title="Untitled drawing"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8df578a455a2b17d75739570b3d63a2e6e512d70_2_666x500.jpeg" alt="Untitled drawing" data-base62-sha1="kfPf4o4MUGzFPk2GFDtVOtNeucg" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8df578a455a2b17d75739570b3d63a2e6e512d70_2_666x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/8/d/8df578a455a2b17d75739570b3d63a2e6e512d70.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/d/8df578a455a2b17d75739570b3d63a2e6e512d70.jpeg 2x" data-dominant-color="FAF9FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Untitled drawing</span><span class="informations">960×720 31 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

JuanFMontesinos: <p>What do you mean.<br>
If you have 64 filters how do you plan to share 128? Could you provide more details?</p>

jessicaloke: <p>So, the shared weights is only within the conv2_x or conv3_x or conv4_x. The two conv layers within the same residual blocks would share weights. Similar to what’s done in [this paper] or <a href="https://arxiv.org/pdf/1702.08782.pdf" rel="noopener nofollow ugc">this paper</a>. (<a href="https://arxiv.org/pdf/1604.03640.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1604.03640.pdf</a>).</p>

JuanFMontesinos: <p>Basically to share weights you have to call the same instance as many times as you want.<br>
I think you are aware of that.</p>
<p>If you want to share part of the weights you can define the weights as nn parameters and call the convolution functional (you can concat weights or be as flexible as you want).<br>
If you change the number of filters indeen you have to decide which weights are shared manually.</p>
Selective search

JorgeRSC: <p>Hi Everyone,</p>
<p>I am new to pytorch, torchvision and I have used DNN implemented with pytorch on the jetson nano.<br>
I would like to know if there is an intruction or example on how to do selective search using pytorch or torchvision (not being part of the DNN) ?</p>
<p>Thank you for your time,<br>
Jorge</p>
Cs231's memory calculation

hitbuyi: <p>They take weights into account, bias are not included, for exact calculation,  bias should be included for memory size calculation,  is that right?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/8/18a142f78c2ac30fce0c86c150aba4e1a49fc32b.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/18a142f78c2ac30fce0c86c150aba4e1a49fc32b" title="cs231_memory"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/18a142f78c2ac30fce0c86c150aba4e1a49fc32b_2_649x500.jpeg" alt="cs231_memory" data-base62-sha1="3vSXyKBWZhr0Bz9Gj15ekaUbJvd" width="649" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/18a142f78c2ac30fce0c86c150aba4e1a49fc32b_2_649x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/18a142f78c2ac30fce0c86c150aba4e1a49fc32b_2_973x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/8/18a142f78c2ac30fce0c86c150aba4e1a49fc32b.jpeg 2x" data-dominant-color="E4E4F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">cs231_memory</span><span class="informations">1013×780 238 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Yes, for the exact calculation you could add the <code>bias</code> shape, but it can be skipped as it’s usually a small overhead.<br>
E.g. the first conv layer:</p>
<pre><code class="lang-python">conv = nn.Conv2d(3, 64, 3, padding=1)
</code></pre>
<p>would create an activation shape of <code>[batch_size, 64, 224, 224] = 3211264*batch_size</code> elements, has a <code>weight</code> of <code>[64, 3, 3, 3] = 1728</code> elements and a bias of <code>[64] = 64</code> elements.<br>
Given this large difference you can decide if the exact calculation would yield any important information.<br>
Note that the later layers have a larger difference of <code>weight/bias</code>.<br>
E.g.:</p>
<pre><code class="lang-python">conv = nn.Conv2d(256, 512, 3)
print(conv.weight.nelement())
# 1179648
print(conv.bias.nelement())
# 512

print(conv.bias.nelement()/conv.weight.nelement())
# 0.00043402777777777775
</code></pre>

hitbuyi: <p>it is so convenient to compute size of weights and bias by using Pytorch, great</p>
How to move this to cpu or detach?

ADONAI_TZEVAOT: <p>I want to move certain values outside GPU. How do I do it?<br>
The following code is not working .</p>
<pre><code class="lang-auto">loss.backward()
optimizer.step()
scheduler.step()
loss = criterion(outputs, labels)
acc = binary_acc(outputs, labels)                  
epoch_val_loss += loss.item().detach()
epoch_val_acc += acc.item().detach()
</code></pre>
<p>It is throwing the following error :</p>
<blockquote>
<p>AttributeError: ‘float’ object has no attribute ‘detach’</p>
</blockquote>
<p>I even tried this but it is not working</p>
<pre><code class="lang-auto">epoch_val_loss += loss.item().cpu()
epoch_val_acc += acc.item().cpu()
</code></pre>
<p>This throws the error :</p>
<blockquote>
<p>AttributeError: ‘float’ object has no attribute ‘cpu’</p>
</blockquote>
<p>Please help!</p>

ptrblck: <p><code>tensor.item()</code> is returning a Python scalar value, which is already detached and on the CPU.<br>
This call will fail if <code>tensor</code> contains more than a single value and thus cannot be represented by a Python scalar.</p>

ADONAI_TZEVAOT: <p>I have another doubt… the output of a neural network is stored in CPU or GPU? what should I do to get it out of GPU if its in GPU?</p>

ptrblck: <p>If depends if the parameters and also input were previously pushed to the GPU.<br>
If so, you can push the tensor back to the host via <code>tensor.cpu()</code> or <code>tensor.to('cpu')</code>.</p>

ADONAI_TZEVAOT: <p>The reason why i want to move these to CPU is because of limited memory in GPU. I have 2 more doubts</p>
<ol>
<li>Will using <code>torch.cuda.empty_cache()</code> at the end of every epoch during training help with that?</li>
<li>Will using <code>torch.cuda.empty_cache()</code> at the end of every epoch affect the way the loss is backpropagated?</li>
</ol>

ptrblck: <ol>
<li>
<p>No, as it will only free the cache and thus force PyTorch to re-allocate the memory via synchronizing <code>malloc</code> calls, which will slow down your code. You won’t avoid out-of-memory issues, but would allow other applications to use the freed GPU memory.</p>
</li>
<li>
<p>No, freeing the cache will not have any functionality impact (besides slowing down your code).</p>
</li>
</ol>
<p>Moving the output to the CPU will also not move the entire computation graph to it, but only this single value (which won’t save a lot of memory).<br>
If you are running out of memory, you could decrease the batch size, use <code>torch.utils.checkpoint</code>, the new CPU-offloading util., mixed-precision training, etc.</p>
Torch multiprocessing

ssha: <p>Hi,</p>
<p>I have some RL code implemented and am using torch.multiprocessing to collect training samples. The code runs fine but my challenge is that I want to run a separate function every n episodes to check performance metrics of current trained model, however, i cannot seem to do this. The broad structure of the multiprocessing code is as follows, the part I am stuck at is the last line in the run method (see code comment in the validation portion):</p>
<pre><code class="lang-auto">import torch.multiprocessing as mp



class Worker(mp.Process):
   def __init__(self, global_ep,  res_queue):
      self.global_ep  = global_ep
      self.res_queue = res_queue

   def record(self, global_ep):
      with self.global_ep.get_lock():
         self.global_ep += 1
   
   def run(self):
      while self.global_ep &lt; MAX_EPISODE:
         # run training loop

          if self.global_ep % VERBOSE_FREQ ==0:
             # check model performance on validation data
            """  run is called on each processor separately, however,
                 I  only want to use the global trained model to run inference once, after all processors
                have collected training samples"""
  
global_ep, res_queue = mp.Value('i', 0),  mp.Queue()

workers = [Worker(global_ep, res_queue)]
[w.start() for w in workers]
[w.join() for w in workers]
</code></pre>
<p>If i run the validate code in the portion with the comment, it runs once for each active processor since run is called individually on each processor.</p>
<p>Any ideas?</p>
<p>Thanks!</p>

vmoens: <p>Hi <a class="mention" href="/u/ssha">@ssha</a><br>
Are you training the model on each process or to collect samples?<br>
If not, I would simply create an env instance in the main process and test your model on it. Is it something you have considered?<br>
Have a look at <a href="https://github.com/facebookresearch/rl/tree/main/torchrl/trainers" rel="noopener nofollow ugc">torchrl’s trainer</a>: what we do is that we use a data collector on parallel processes that sends batches of data to the main process. The main process will take care of training the model. You can register a recording hook that will evaluate your model on an environment in the main process, i.e. distant processes are not involved.<br>
Hope it helps!</p>

sardanian: <p>You could use a shared memory object (such as a  single dimensional tensor corresponding to the number of worker processes you have) that gets updated by each worker for their completed episodes. Basically a counter for each episode.</p>
<p>Then have a separate process (or your main process) that checks that shared memory tensor and runs metrics any time it hits your N episodes.</p>
How do I share data between GPUs that is not a tensor?

PaulNaduri: <p>The torch.distributed library seems to only support sharing tensors between GPUs. I want to share arbitrary data between GPUs besides just tensors.</p>
<p>What is the elegant way of doing this?</p>
<p>Since all GPUs are on the same node, can they simply access the same data in CPU memory? If I have a FLAG variable in CPU memory and rank==0 modifies it during runtime, can rank==2 access the modification?</p>

ptrblck: <p>I’m not sure which data you would like to share as I would assume the GPU(s) should load it in the end (so a tensor should work).</p>
<aside class="quote no-group" data-username="PaulNaduri" data-post="1" data-topic="155355">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/paulnaduri/48/50983_2.png" class="avatar"> PaulNaduri:</div>
<blockquote>
<p>Since all GPUs are on the same node, can they simply access the same data in CPU memory?</p>
</blockquote>
</aside>
<p>No, GPUs cannot work on CPU data directly and you would have to move it to the device first.</p>

PaulNaduri: <p>I want to share arbitrary data, not tensors. The data will not be loaded by the GPU. For example, I want to send a dictionary from rank 0, to all devices, and the dictionary will contain only hyperparameters for training.<br>
Is there a way to do this?</p>
Identity Convolution Weights for 3 Channel Image

Aarnav_Sawant: <p>Hi,<br>
I wanted to initialize a Conv2D Kernel which returns the same image after running the image through it. This is the code I am currently using</p>
<pre><code class="lang-auto">wts = np.zeros((3,3,3,3))
nn.init.dirac_(torch.from_numpy(wts))
with torch.no_grad():
  conv_layer.weight = nn.Parameter(torch.tensor(wts,dtype=torch.float))
new_img = conv_layer(img_tensor).detach()
</code></pre>
<p>However, whenever I run the image through the Conv Layer, the new image always is disturbed in certain pixels. The means of the two images are also different. Could you guide me as to the best way to create a Convolution Filter with weights that return the same image back? I know for the single channel image case, the weights would simply be</p>
<pre><code class="lang-auto">[0, 0, 0
 0, 1, 0
 0, 0, 0]
</code></pre>
<p>but I’m unsure what it should be for a 3 channel image</p>

ptrblck: <aside class="quote no-group" data-username="Aarnav_Sawant" data-post="1" data-topic="155405">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aarnav_sawant/48/51003_2.png" class="avatar"> Aarnav_Sawant:</div>
<blockquote>
<p>but I’m unsure what it should be for a 3 channel image</p>
</blockquote>
</aside>
<p>You would need to use a grouped convolution where the posted kernel would be repeated in <code>dim0</code> (the <code>out_channel</code> dimension) as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python"># setup
in_channels  = out_channels = 7

# init
wts = torch.zeros(1, 1, 3, 3)
nn.init.dirac_(wts)
wts = wts.repeat(out_channels, 1, 1, 1)

conv_layer = nn.Conv2d(in_channels, out_channels, kernel_size=3, bias=False, padding=1, groups=out_channels)
with torch.no_grad():
  conv_layer.weight.copy_(wts)

# test
batch_size = 16
h, w = 224, 224
x = torch.randn(batch_size, in_channels, 224, 224)
new_img = conv_layer(x)

print((new_img - x).abs().max())
# tensor(0., grad_fn=&lt;MaxBackward1&gt;)
</code></pre>
How cifar.py is written

Harish_S: <p>In   <a href="https://github.com/pytorch/vision/blob/main/torchvision/datasets/cifar.py" class="inline-onebox" rel="noopener nofollow ugc">vision/cifar.py at main · pytorch/vision · GitHub</a></p>
<p>how  value  ‘c99cafc152244af753f735de768cd75f’ is taken from cifar dataet?<br>
My question is from where this md5 key is coming,ANy hints</p>
<p>train_list = [<br>
[‘data_batch_1’, ‘c99cafc152244af753f735de768cd75f’],<br>
[‘data_batch_2’, ‘d4bba439e000b95fd0a9bffe97cbabec’],<br>
[‘data_batch_3’, ‘54ebc095f3ab1f0389bbae665268c751’],<br>
[‘data_batch_4’, ‘634d18415352ddfa80567beed471001a’],<br>
[‘data_batch_5’, ‘482c414d41f54cd18b22e5b47cb7c3cb’],<br>
]</p>

Nikronic: <p>Hi,</p>
<p>These are the md5 of files inside the compressed file you would dowload. I.e.:</p>
<pre><code class="lang-bash">wget https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz
tar -xf cifar-10-python.tar.gz
ls cifar-10-batches-py/
# batches.meta  data_batch_2  data_batch_4  readme.html
# data_batch_1  data_batch_3  data_batch_5  test_batch

md5sum cifar-10-batches-py/data_batch_1
# output: c99cafc152244af753f735de768cd75f  cifar-10-batches-py/data_batch_1
# and so on for all other files listed above
</code></pre>
<p>Bests</p>

Harish_S: <p>Thanks for replying <a class="mention" href="/u/nikronic">@Nikronic</a></p>
Sliding window of 2D tensor with rollover

legoh: <p>Hi guys, I’m trying to figure out how to create a sliding window on a 2D tensor with some form of rollover when it hits the boundaries. For example, suppose we have the following tensor:</p>
<pre><code class="lang-auto">tensor([[2, 4, 6, 5, 9, 7, 3, 8, 1, 0],
        [0, 2, 1, 9, 4, 5, 6, 8, 3, 7],
        [2, 5, 8, 9, 6, 3, 0, 1, 4, 7],
        [5, 2, 1, 6, 8, 4, 9, 0, 3, 7]])
</code></pre>
<p>where each row is a batch of indexes. What I want to achieve is a sliding window of 5 elements, where the centre element is the focal point and the 2 left and 2 right neighbors form the 5 elements. For example, if we take the first row <code>[2, 4, 6, 5, 9, 7, 3, 8, 1, 0]</code>, the end result should be:</p>
<pre><code class="lang-auto">[1, 0, 2, 4, 6]
[0, 2, 4, 6, 5]
[2, 4, 6, 5, 9]
[4, 6, 5, 9, 7]
[6, 5, 9, 7, 3]
[5, 9, 7, 3, 8]
[9, 7, 3, 8, 1]
[7, 3, 8, 1, 0]
[3, 8, 1, 0, 2]
[8, 1, 0, 2, 4]
</code></pre>
<p>Tensor unfold doesn’t seem quite to get the result of this.</p>

ptrblck: <p>Manual padding and <code>unfold</code> should work:</p>
<pre><code class="lang-python">x = torch.tensor([[2, 4, 6, 5, 9, 7, 3, 8, 1, 0],
                  [0, 2, 1, 9, 4, 5, 6, 8, 3, 7],
                  [2, 5, 8, 9, 6, 3, 0, 1, 4, 7],
                  [5, 2, 1, 6, 8, 4, 9, 0, 3, 7]])

# pad borders
x_pad = torch.cat((x[:, -2:], x, x[:, :2]), dim=1)
res = x_pad.unfold(1, 5, 1)
print(res)
# tensor([[[1, 0, 2, 4, 6],
#          [0, 2, 4, 6, 5],
#          [2, 4, 6, 5, 9],
#          [4, 6, 5, 9, 7],
#          [6, 5, 9, 7, 3],
#          [5, 9, 7, 3, 8],
#          [9, 7, 3, 8, 1],
#          [7, 3, 8, 1, 0],
#          [3, 8, 1, 0, 2],
#          [8, 1, 0, 2, 4]],

#         [[3, 7, 0, 2, 1],
#          [7, 0, 2, 1, 9],
#          [0, 2, 1, 9, 4],
#          [2, 1, 9, 4, 5],
#          [1, 9, 4, 5, 6],
#          [9, 4, 5, 6, 8],
#          [4, 5, 6, 8, 3],
#          [5, 6, 8, 3, 7],
#          [6, 8, 3, 7, 0],
#          [8, 3, 7, 0, 2]],
# ...
</code></pre>
Custom dataset using torchtext (0.13.0)

p_Raj: <p>Please assist me in developing a torchtext  (0.13.0) costume data loader from a csv file. Field and LabelField are nowhere to be seen in the new torchtext module. If somebody has built it,  kindly assist me with the code?</p>

Mashood3624: <p>Check this notebook.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/github/bentrevett/pytorch-sentiment-analysis/blob/master/A%20-%20Using%20TorchText%20with%20Your%20Own%20Datasets.ipynb">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20220628-060046-RC00_457694881" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/github/bentrevett/pytorch-sentiment-analysis/blob/master/A%20-%20Using%20TorchText%20with%20Your%20Own%20Datasets.ipynb" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/github/bentrevett/pytorch-sentiment-analysis/blob/master/A%20-%20Using%20TorchText%20with%20Your%20Own%20Datasets.ipynb" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


p_Raj: <p>Thank you <a class="mention" href="/u/mashood3624">@Mashood3624</a> for your response. The Torchtext Notebook you provided is appropriate for versions 0.9.0 or earlier; data. Field(),<br>
data. LabelField(),data. TabularDataset  is missing from the most recent version.  How can we make it in 0.13.0 Version.</p>
<p>Using the torchtext documentation, I’m also attempting to accomplish this. <a href="https://pytorch.org/text/stable/_modules/torchtext/datasets/imdb.html#IMDB" rel="noopener nofollow ugc">https://pytorch.org/text/stable/_modules/torchtext/datasets/imdb.html#IMDB  </a> But unable to do it using csv</p>
Pytorch RNN implementation

laro: <p>I read about RNN in pytorch:<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.RNN.html" class="inline-onebox" rel="noopener nofollow ugc">RNN — PyTorch 2.1 documentation</a>.</p>
<p>According to the document the RNN run the following function:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9cd7f9edaedda6b5d94011f7e2977d26e2e58616.png" alt="rnn" data-base62-sha1="mnvdaCX4o7EVoEnoeXoPTe36X6S" width="357" height="50"></p>
<p>I looked on another RNN example (from pytorch tutorial):<br>
<a href="https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">NLP From Scratch: Classifying Names with a Character-Level RNN — PyTorch Tutorials 2.2.0+cu121 documentation</a>.</p>
<p>And they implemented RNN as:</p>
<pre><code class="lang-auto">import torch.nn as nn

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()

        self.hidden_size = hidden_size

        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)
        self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, input, hidden):
        combined = torch.cat((input, hidden), 1)
        hidden = self.i2h(combined)
        output = self.i2o(combined)
        output = self.softmax(output)
        return output, hidden

    def initHidden(self):
        return torch.zeros(1, self.hidden_size)

n_hidden = 128
rnn = RNN(n_letters, n_hidden, n_categories)
</code></pre>
<ol>
<li>
<p>Why the implemented function is different from the equation ?<br>
(The function doesn’t contains softmax and it does contain bias which is not shown in the code)</p>
</li>
<li>
<p>Why the code dosn’t use tanh as shown in the equation ?</p>
</li>
</ol>