akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
When i run detect.py file of yolov5 just for inference that time show me this error someone help me?

brijesh: <p>RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling <code>cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)</code></p>

ptrblck: <p>Could you post a minimal, executable code snippet to reproduce the issue as well as the output of <code>python -m torch.utils.collect_env</code>?</p>

brijesh: <p>ptrblack,</p>
<p>thank you for your time but solved this error and execute the code and it run perfectly</p>
How can I change the padding to be the same while changing the stride?

YuhskeHujisaki: <p>How can I change the padding to be the same while changing the stride?</p>
<pre><code class="lang-auto">nn.Conv2d(256,64,(1,1),stride=(1,2), padding = "same")
</code></pre>
<p>I want it to be like the code above.</p>
How will SGD work under batch data?

hanglearning: <p>I’ve learned that stochastic gradient descent (SGD) updates weights one sample after another in random order. Then I saw the following code example. My understanding is that, within one iteration, this code extracts a batch of data, say 10 samples if the batch size is set to 10, and then SGD will update weights one sample after another within these 10 samples in random order.</p>
<pre><code class="lang-auto">optimizer = torch.optim.SGD(lr=lr, params=model.parameters())
for batch_id, (inputs, labels) in enumerate(train_dataloader):
    y_hat = model(x)
    loss = F.cross_entropy(y_hat, y)
    model.zero_grad()

    loss.backward()
    optimizer.step()
</code></pre>
<p>However, I see from <a href="https://discuss.pytorch.org/t/how-sgd-works-in-pytorch/8060/1">this post</a> that @ <a href="https://discuss.pytorch.org/u/chenyuntc">chenyuntc</a> says that “SGD optimizer in PyTorch actually is Mini-batch Gradient Descent with momentum”. To my understanding, in this case, SGD does not update weights one sample after another in random order, but updates weights by processing a batch of 10 data points together, basically just no stochasticity.</p>
<p>My question is which one is true? Thanks!</p>

ptrblck: <p>Yes, the post is correct and the entire batch will be used to compute the gradients in the <code>loss.backward()</code> call. The optimizer will then use the <code>.grad</code> attribute of each parameter to perform the actual parameter update step.</p>

hanglearning: <p>Thank you very much! Then, is there an optimizer in PyTorch that does traditional SGD? I mean, iteratively randomly picking up one data sample for updating the weights.</p>
<p>Thank you</p>

ptrblck: <p>Yes, this would be the case if you specify <code>batch_size=1</code> or call <code>backward</code>, <code>optimizer.step()</code>, and <code>optimizer.zero_grad()</code> on each output sample individually.</p>

hanglearning: <p>Oh! Got it, thank you again ptrblck!!</p>
How to check whether tensor values in a different tensor pytorch?

Adam_13: <p>I have 2 tensors of unequal size</p>
<pre><code>a = torch.tensor([[1,2], [2,3],[3,4]])
b = torch.tensor([[4,5],[2,3]])
</code></pre>
<p>I want a boolean array of whether each value exists in the other tensor without iterating. something like</p>
<pre><code>a in b
</code></pre>
<p>and the result should be</p>
<pre><code>[False, True, False]
</code></pre>
<p>as only the value of a[1] is in b</p>

soulitzer: <p>To reason about this we can look at the 1-d case.</p>
<p>If <code>a = [2, 4]</code> and <code>b = [1, 2, 3]</code>, our output should be <code>[True, False]</code>. Intuitively we want to create some kind of 2 by 3 table some we can compute the pairwise equalities all at once.<br>
To do this we can use <code>torch.expand</code>:</p>
<p><code>torch.expand(a, (3, 2))</code> gives us <code>[[2, 4], [2, 4], 2, 4]]</code><br>
<code>torch.expand(b, (2, 3))</code> gives us <code>[[1, 2, 3], [1, 2, 3]]</code></p>
<p>Note that here, the shapes of a and b, are (3, 2), and (2, 3), and simply transposing b will allow us to have the same shape.<code>a1 = [[2, 4], [2, 4], [2, 4]]</code> <code>b1 = [[1, 1], [2, 2], [3, 3]]</code>.<br>
As you can see here, if we now do <code>a1 == b1</code>, which computes as <code>[[False, False], [True, False], [False, False]]</code>, we are simply computing the equality across all pairs in the cartesian product. If we simply find max of this tensor along <code>dim=0</code> we have <code>[True, False]</code> as desired.</p>
<p>Now to generalize to cases where dimension &gt; 1. If you have tensors <code>a</code> and <code>b</code>, by definition their shapes must be the same along all dimensions except one. For example if we want to compute <code>a in b</code> along <code>dim=1</code> their shapes could be <code>(x, y, z)</code> and <code>(x, w, z)</code>. Working backwards, we know that our output needs to be one-dimensional and have shape <code>(y)</code>. So the the idea here could be to compute some kind of table of with y rows and w columns where the ith output corresponds max value of the ith row.</p>
<p>Some non-tested code below based on our intuitions above is:</p>
<pre><code class="lang-auto">a1 = torch.expand(a, (w, x, y, z))
b1 = torch.expand(b, (y, x, w, z)).permute(0, 2)
t = (a1 == b1).min(dim=(1, 3)).values # shape: (w, y)
t.max(dim=0).values # shape: (y, )
</code></pre>

nastaranmarzban: <p>hi, excuse me i have the same problem as yours…can you find it? would you please help me?</p>

saluei: <p>hi, this may be help</p>
<pre><code class="lang-auto">a=torch.tensor([2,4])
b = torch.tensor([1, 2, 3])
print([(x in b)  for x in a])
</code></pre>
State_dict error in testing of a model that is trained by loading checkpoints

Tm4: <p>I trained a model using PyTorch and I stored the weights in which it had the minimum validation loss during training. Also, I stored the optimizer weights on that time.  So, I trained my model for the second time by loading the weights that I stored from the first time. I wanted to test my model and generate the outputs of my model.<br>
But, unfortunately, I faced with such an error:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b4f7cf0780d28d1bded10b69190d1640e2eb242.png" data-download-href="https://discuss.pytorch.org/uploads/default/7b4f7cf0780d28d1bded10b69190d1640e2eb242" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b4f7cf0780d28d1bded10b69190d1640e2eb242.png" alt="Capture" data-base62-sha1="hAR1VZk5lRAfrjBwkhsuKSlI7my" width="690" height="122" data-dominant-color="DAC9C9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">1417×252 21.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>When I wanted to load the weights for the second_time training I used the following scripts:</p>
<pre><code>    model = nn.DataParallel(model, device_ids = [0,1],output_device= [0,1])
    
    model = model.to(dev)

    model.load_state_dict(torch.load(path_to_the_saved_model's_weights, map_location=dev))
    
    optimizer = torch.optim.Adam(model.parameters(), lr=lr, weight_decay=2e-7)

    
    # loading optimizer
    optimizer.load_state_dict(torch.load(path to the saved optimizer's weights))
</code></pre>
<p>I only changed learning rate value during the second_time training and when I want to test the model and generating the output of the model by loading the new weights I see such an error. How can I solve the problem?</p>

ptrblck: <p>Check if your <code>DataParallel</code> usage causes the issue and if so, load the <code>state_dict</code> before wrapping the model into <code>DataParallel</code>.</p>
Making subplots using tensorboardX for pytorch

ZSheikhb: <p>I’d like to make a set of scatter plots  from a dimension of 3D tensor in the form of a combination of subplots inside <code>tensorboard</code>. The tensors have this shape like <code>torch.Size([20,31,11])</code>. Here is my current attempt</p>
<pre><code>import matplotlib.pyplot as plt
import tensorboardX as tb


def draw_scatter(writer, original, reconstructed, epoch ):
    
    matplotlib.use('Agg')
    fig  =  plt.figure(figsize=(8,5))
    for idx in range(original.shape[-1]):
        ax=plt.subplot(original.shape[-1],1,idx+1)
        ax.axis("off")
        ax.scatter(original[:,:,idx], reconstructed[:,:,idx])
        ax.set_xlabel("original")
        ax.set_ylabel("reconstructed")
    plt.tight_layout()
    fig.canvas.draw()
    writer.add_image("original/reconstructed", tb.utils.figure_to_image(fig), epoch)
writer = tb.SummaryWriter(args.logdir)
draw_scatter(writer, x_org, x_sample, epoch )
</code></pre>
<p>However, the results doesn’t look like what I was aiming for<br>
<a href="https://i.stack.imgur.com/A9LvM.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/A9LvM.png" alt="enter image description here" width="690" height="431"></a></p>
<p>How can I make subplots in tensorboard using matplotlib which each one would show the comparison between original and reconstructed data?</p>
Impact of data shuffling on results reproducibility in Pytorch

Kalyan_Katikapalli: <p>Pytorch dataloader class has the following constructor:</p>
<pre><code class="lang-auto">DataLoader(dataset, batch_size=1, shuffle=False, sampler=None,
           batch_sampler=None, num_workers=0, collate_fn=None,
           pin_memory=False, drop_last=False, timeout=0,
           worker_init_fn=None)
</code></pre>
<p>When shuffle is set to  <code>True</code> , data is reshuffled at every epoch. Shuffling the order in which examples are fed to the classifier is helpful so that batches between epochs do not look alike. Doing so will eventually make our model more robust.</p>
<p>However, I’m unable to understand  <strong>by setting shuffle=True, can we get the same accuracy value for different runs</strong> ?</p>

Kalyan_Katikapalli: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
<strong>by setting shuffle=True in data loaders, do we get the same accuracy value for different runs?</strong></p>

ptrblck: <p>You shouldn’t expect the same bitwise accurate results (and would need to set all seeds, use <code>cudnn.deterministic</code> etc. as described <a href="https://pytorch.org/docs/stable/notes/randomness.html">here</a>).</p>
<p>However, if your training procedure is “stable” you should see a small stddev in the final accuracy.</p>

Kalyan_Katikapalli: <p>If I use following code to set all the seeds,</p>
<pre><code class="lang-auto">random.seed(seed)
os.environ[‘PYTHONHASHSEED’] = str(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.enabled = False
</code></pre>
<p>will shuffling the instances,  still result in different values in different runs? <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

Kalyan_Katikapalli: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I’m unable to understand<br>
a) <strong>what is a stable training procedure?</strong> and b) <strong>how to achieve it?</strong></p>

ptrblck: <p>If setting all seeds doesn’t help, you would have to check, if your model uses one of the mentioned methods from the linked doc.</p>
<p>By “stable” I meant that your model would converge 10 out of 10 runs with a small delta in the final loss and accuracy for different seeds.<br>
“Unstable” training would depend on the seed as a hyperparameter.</p>

Ali_Aghababaei: <p>Hi,</p>
<p>Sorry to open this topic again after about 2 years.</p>
<p>I wanted to know if your problem with “shuffling” was resolved, considering that you tried to overcome the randomness as much as possible with respect to the code snippet you provided.</p>
Torch multiprocess and share

spyroot: <p>Hi Folks,</p>
<p>I’m having a strange issue that I think I have already spent more than two days troubleshooting :).  I guess it related to the example given.</p>
<pre><code class="lang-auto">x = queue.get()
x_clone = x.clone()
queue_2.put(x_clone)
</code></pre>
<p>So if tensor x is on GPU, as far as I understand, you need to clone it.</p>
<p>First question, if I have object A, which holds a reference for four tensors, what is the right way to clone an entire object with all tensors? i.e do you need the first move to CPU, put the queue, and then back to GPU? Because if detach.clone while tensor on GPU, the data always zeroes.</p>
<p>class A<br>
self._observations → tensor<br>
self._actions → tensor</p>
<pre><code class="lang-auto">def clone(self, dst):
        dst._observations = self._observations.detach().cpu()
        dst._actions = self._actions.detach().cpu()
</code></pre>
<p>then I would use</p>
<pre><code class="lang-auto">lock network_lock:
  x = A()
  network.forward(x)
  new_x = A() 
  # a new object.   x.clone() take new_x and detach() and move to CPU()
  # note that I've tried to detach() and clone() and different combinations but can't get it to work.
  # NOTE when a device is set to CPU, everything is working
  queue.put(x.clone(new_x))
</code></pre>
<p>My second question is if a tensor is on GPU, what is the right way to put an object?<br>
A that holds a bunch of tensor on the GPU?</p>
<pre><code class="lang-auto"> def clone(self, dst):
        dst._observations = self._observations.detach().clone()
        dst._actions = self._actions.detach().clone()
</code></pre>
<p>In the second example, I notice as soon as the object goes to a queue.<br>
If a tensor on GPU data becomes zero, the value is zero when the consumer picks up from the queue.</p>
<p>I see a CUDA warning and error related to grad.</p>
<pre><code class="lang-auto">[W CUDAGuardImpl.h:62] Warning: CUDA warning: invalid device ordinal (function uncheckedSetDevice)
[W CUDAGuardImpl.h:46] Warning: CUDA warning: driver shutting down (function uncheckedGetDevice)
</code></pre>
<p>I note that with small data, I see no error.</p>
<p>So in producer, I didn’t compute the gradient and tried different combinations.<br>
I did check all tensors that I serialize to a queue, and no tensor has grad attached.</p>
<p>Thank you.</p>
Replacing Torchivision with Albumentations Transforms is Lowering Performance :(

cmitra: <p>I need to switch to albumentations for more flexibility (using some custom image transforms). However, doing a simple test of the following transforms when switching from Torchvision yields lower performance:</p>
<pre><code class="lang-auto">#Imports
from torchvision import transforms as transforms
import albumentations as A
from albumentations.pytorch.transforms import ToTensorV2

#Torchvision
transforms_ = transforms.Compose([
                transforms.Resize(256),
                transforms.CenterCrop(256),
                transforms.Resize((256, 256)),
                transforms.ToTensor(),
                transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
            ])

#Albumentations:
transforms_ = A.Compose([
                A.augmentations.geometric.resize.SmallestMaxSize(256),
                A.CenterCrop(256, 256),
                A.Resize(256,256),
                A.Normalize(
                mean=[0.5, 0.5, 0.5],
                std=[0.5, 0.5, 0.5],
                ),
                ToTensorV2()
            ])
</code></pre>
<p>I thought I was very thorough, even being very particular to replace the integer-argument Resize transform with albumentations “SmallestMaxSize” transform which I am pretty sure is the equivalent.</p>
<p>What am I missing here (i.e. what is different)?</p>

ptrblck: <p>Did you check the performance using a few runs with different seeds to check the mean and stddev of the final accuracy? This could show if the difference is real or if you just had a “bad” seed.</p>

cmitra: <p>Thanks for the suggestion <a class="mention" href="/u/ptrblck">@ptrblck</a>! Unfortunately (or perhaps, fortunately), I am keeping the same seed (0) for both eval runs of my model. But do the two transforms look functionally the same? I am not too familiar with albumentations, so I was looking at their source code for the first time.</p>

ptrblck: <p>I’m not deeply familiar with the <code>albumentations</code> library but would assume both transformations yield the same outputs. The second resize transformation doesn’t seem to be necessary as the previously applied center crop already returns the desired shape. Since you are not using any random transformations (I just realized it now) it would be a good idea to test both transforms on a defined input and check the difference in their outputs.</p>

cmitra: <p>Thank you for the suggestion! After doing some digging, I found that Albumentations uses cv2.INTER_LINEAR for interpolating during resizing while Torchvision uses InterpolationMode.BILINEAR as default interpolation when resizing. This has shown to non-trivially affect performance. Hopefully, this is helpful for anyone else dealing with this issue as I know Albumentations is quite popular.</p>
<p>If anyone has any further suggestions on how to incorporate the exact same interpolation as torchvision in albumentations, I would appreciate that. For the time being, I will use torchvision.</p>
Are there any guarantees on the output ordering of ONNX model beyond topological sort?

codykala: <p>I want to convert a PyTorch model that returns multiple potentially nested tensor outputs to ONNX format. For example, the PyTorch model could return something like the following:</p>
<pre><code class="lang-auto">pytorch_model_outputs = ((tensor_1, tensor_2, tensor_3), (tensor_4,), (tensor_5, tensor_6))
</code></pre>
<p>If I convert this model to ONNX format using <code>torch.onnx.export</code>, it will flatten the outputs into a single list in the order they are returned by the graph. However, <a href="https://github.com/onnx/onnx/blob/main/docs/IR.md#graphs" rel="noopener nofollow ugc">from the docs</a> the only requirement on the ordering of nodes in the ONNX graph is that the nodes are sorted in topological order:</p>
<blockquote>
<p>The list of nodes defining the top-level computation graph MUST be ordered topologically; that is, if node K follows node N in the graph, none of the data inputs of N may refer to outputs of K.</p>
</blockquote>
<p>If I’m understanding this correctly, this leaves some ambiguity in the final ordering of the outputs. The following output orderings would be valid and possible:</p>
<pre><code class="lang-auto">onnx_model_outputs_ordering_1 = (tensor_1, tensor_2, tensor_3, tensor_4, tensor_5, tensor_6)
onnx_model_outputs_ordering_2 = (tensor_3, tensor_1, tensor_2, tensor_4, tensor_6, tensor_5)
</code></pre>
<p>In my experience so far, the output ordering has always been consistent with <code>output_ordering_1</code>, which is my desired ordering. But since <code>output_ordering_2</code> is equally possible, it begs the question: how does one know what the actual ordering of the outputs will be short of inspecting the outputs?</p>

addisonklinke: <p><a class="mention" href="/u/codykala">@codykala</a> Curious if you ever found a definitive answer to this? I imagine the ONNX order can’t change if the Python definition is already a flat tuple like <code>(tensor_1, tensor_2, ..., tensor_N)</code> but your use-case of nested tensors is an interesting one</p>
Cannot import lr scheduler

Mika_S: <p>I have pytorch compiled from master and it shows no changes to pull.</p>
<p>But I get error on importing lr_scheduler (<a href="https://github.com/pytorch/pytorch/pull/1370" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/1370</a>) .</p>
<p>Is there some way to access it ?</p>

richard: <p>How are you importing it?<br>
<code>from torch.optim.lr_scheduler import LambdaLR, StepLR, MultiStepLR, ExponentialLR, ReduceLROnPlateau</code> works for me.</p>

brightertiger: <p>I used conda / pip install on version 0.2.0_4. I faced the same issue. Code line -  “from . import lr_scheduler” was missing in the __ init __.py in the optim folder. I added it and after that I was able to import it.</p>

Dorra: <p>hi  guys,<br>
i m trying to import _LRScheduler, as follows from torch.optim.lr_scheduler import _LRScheduler<br>
but it said that there an import error, ImportError: cannot import name ‘_LRSchseduler’ from ‘torch.optim.lr_scheduler’<br>
someone has idea how I can solve that, i will be appreciable.</p>
<p>thanks ,</p>
Create a Fourier series model and optimize

armanbolatov: <p>Suppose I have the following function:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/9/49da31c1adb2e46a4a5b041b0794defec157243f.png" alt="image" data-base62-sha1="axkkKe4Jzhc3O0jxXMNCUiSlhTN" width="391" height="114"><br>
Here <code>x</code> is the input, and <code>a_i</code> and <code>omega</code> are the parameters (so there are <code>n+2</code> parameters in total).</p>
<p>Let’s say <code>n = 10</code>. I want to find the best parameters that will fit the training dataset <code>(y_i, f(x_i))</code>. I know how to optimize neural networks, but I have no idea how to apply Adam for this function.</p>
<p>How to create such a model? Could you please help?</p>
How small batch sizes affect performance on self-supervised DINO?

bryan123: <p>Hi, I am retraining DINO with my own custom dataset (~570k images).</p>
<ol>
<li>On my local computer, the maximum batch size is 32 (1 GPU RTX 3080 TI) and a single epoch takes around 1 hour 20 minutes to complete. Is it normal?</li>
<li>Does a very small batch size matter to the performance?</li>
</ol>
<p>Thank you!</p>

vdw: <p>Regarding your 2nd questions: In general, the larger the batch size, the less time needed to execute an epoch. So in terms of performance purely from the perspective of runtime of an epoch, yes, larger batches are better.</p>
<p>However, note that a fast runtime of an epoch does not necessarily mean a faster training time as the loss might go down more slowly. The large the batch, the more likely the gradients will be smaller, as the gradients will be averages across all samples in a batch. This also means that increasing the batch size typically also allows you to increase the learning rate (again, since the gradients tend to be smaller).</p>
Train multiple languages

Tornike: <p>Hi, i want to do sentiment analysis with transforms on data where some text are written in Georgian and some are written in Georgian also but with English Alphabet. what should i do? what is the pipeline of such cases? should i convert everything in 1 Alphabet or is there any model that can learn such data?</p>

vdw: <p>Kind of up to you. I probably would train 2 separate models to keep the vocabularies small(er).</p>
<p>In any case, the model does not care. Once you have created you vocabulary or vocabularies and converted all texts to sequences to word indices, the model will accept it just fine.</p>

Tornike: <p>Hi, Thank you for your reply. I kinda solved that one but another question is</p>
<p>What would u do if u had two different languages in data? 1 language is very dominant and the second language has very few samples and it needs to be just 1 model.</p>
Concatenate two encoders presentations

gesm: <p>Hi!</p>
<p>I create two encoders models to concatenate the features of two images, and then I’d like to concatenate the encoders’ output to be one input to the decoder module. My problem is with the method that I should apply to concatenate the encoders’ outputs. Any help is really appreciated! below my code</p>
<pre><code class="lang-auto"># define the NN architecture
class ConvDenoiser(nn.Module):
    def __init__(self):
        super(ConvDenoiser, self).__init__()
        ## encoder layers ##
        # conv layer (depth from 1 --&gt; 32), 3x3 kernels
        self.conv1 = nn.Conv2d(3, 128, 3, padding=1) 
        self.conv1_bn = nn.BatchNorm2d(128) 
        # conv layer (depth from 32 --&gt; 16), 3x3 kernels
        self.conv2 = nn.Conv2d(128, 64, 3, padding=1)
        self.conv2_bn = nn.BatchNorm2d(64) 
        # conv layer (depth from 16 --&gt; 8), 3x3 kernels
        self.conv3 = nn.Conv2d(64, 32, 3, padding=1)
        # pooling layer to reduce x-y dims by two; kernel and stride of 2
        self.pool = nn.MaxPool2d(2, 2)
        
        ## decoder layers ##
        # transpose layer, a kernel of 2 and a stride of 2 will increase the spatial dims by 2
        self.t_conv1 = nn.ConvTranspose2d(32, 32, 2, stride=2)  # kernel_size=3 to get to a 7x7 image output
        self.t_conv1_bn = nn.BatchNorm2d(32) 
        # two more transpose layers with a kernel of 2
        self.t_conv2 = nn.ConvTranspose2d(32, 64, 2, stride=2)
        self.t_conv2_bn = nn.BatchNorm2d(64) 
        self.t_conv3 = nn.ConvTranspose2d(64, 128, 2, stride=2)
        # one, final, normal conv layer to decrease the depth
        self.conv_out = nn.Conv2d(128, 3, 3, padding=1)
    def forward(self, x):
        ## encode ##
        # add hidden layers with relu activation function
        # and maxpooling after
        x = self.conv1(x)
        y = self.conv1(y)
        #print(a.transpose(0,3).transpose(1,2).size())
        x = F.relu(self.conv1_bn(x))
        y = F.relu(self.conv1_bn(y))
        x = self.pool(x)
        y = self.pool(x)
        # add second hidden layer
        x = self.conv2(x)
        y = self.conv2(y)
        x = F.relu(self.conv2_bn(x))
        y = F.relu(self.conv2_bn(y))
        x = self.pool(x)
        y = self.pool(y)
        # add third hidden layer
        x = F.relu(self.conv3(x))
        y = F.relu(self.conv3(y))
        x = self.pool(x)  # compressed representation
        y = self.pool(y)  # compressed representation

        z = torch.cat((x,y),1)
        
        ## decode ##
        # add transpose conv layers, with relu activation function
        z = self.t_conv1(z)
        z = F.relu(self.t_conv1_bn(z))
        z = self.t_conv2(z)
        z = F.relu(self.t_conv2_bn(z))
        z = F.relu(self.t_conv3(z))
        #x = x.squeeze(2)
        #x = x.permute(2,3,1,0)  # [w, b, c]
        # transpose again, output should have a sigmoid applied
        z = F.sigmoid(self.conv_out(z))
                
        return z

# initialize the NN
model = ConvDenoiser()
</code></pre>
How can I change this code in pytorch form?Do we have Input layer in pytorch?

maryam_abbaszadeh: <p>def build_densenet():<br>
densenet = DenseNet121(weights=‘imagenet’, include_top=False)</p>
<pre><code>input = Input(shape=(SIZE, SIZE, N_ch))
x = Conv2D(3, (3, 3), padding='same')(input)

x = densenet(x)

x = GlobalAveragePooling2D()(x)
x = BatchNormalization()(x)
x = Dropout(0.5)(x)
x = Dense(256, activation='relu')(x)
x = BatchNormalization()(x)
x = Dropout(0.5)(x)

# multi output
output = Dense(2,activation = 'softmax', name='root')(x)


# model
model = Model(input,output)
</code></pre>

ptrblck: <p>No, you don’t need to define input layers and can directly pass a tensor to the model.<br>
Take a look at <a href="https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html">this tutorial</a> for an example of a simple neural network.</p>

maryam_abbaszadeh: <p>thanks for replying, the other question is: we don’t have to change the output of denesnet121 when we pass it to GlobalAveragePooling2D()?</p>
ERROR: No matching distribution found for torch===1.7.0+cu110

Redcxx: <p>Hi I got an error while trying to install pytorch:</p>
<pre><code class="lang-auto">PS C:\windows\system32&gt; pip install torch===1.7.0+cu110 torchvision===0.8.1+cu110 torchaudio===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html
Looking in links: https://download.pytorch.org/whl/torch_stable.html
ERROR: Could not find a version that satisfies the requirement torch===1.7.0+cu110 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch===1.7.0+cu110
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/f/7fc54020d74485860c82b5c429435a190b4820ce.png" alt="image" data-base62-sha1="iejffBSH7KwAOdiUujvAfnyHK1M" width="623" height="302"></p>

ZimoNitrome: <p>What version of Python and Pip are you running?</p>
<p>Have you tried updating pip? <code>python -m pip install –upgrade pip</code></p>

Redcxx: <p>Hi I am using python 3.8.6 and pip 20.2.4</p>
<pre><code class="lang-auto">PS C:\windows\system32&gt; py -m pip install --upgrade pip
Requirement already up-to-date: pip in d:\environments\python\lib\site-packages (20.2.4)
</code></pre>
<pre><code class="lang-auto">PS C:\windows\system32&gt; py --version
Python 3.8.6
</code></pre>

Redcxx: <p>I tried to install it directly from the website but it does not work:</p>
<pre><code class="lang-auto">PS C:\windows\system32&gt; pip install https://download.pytorch.org/whl/cu110/torch-1.7.0%2Bcu110-cp38-cp38-win_amd64.whl
ERROR: torch-1.7.0+cu110-cp38-cp38-win_amd64.whl is not a supported wheel on this platform.
</code></pre>

Redcxx: <p>Ah my bad sorry, I installed the 32 bit python instead of 64 bits, so it is not supported</p>

jsktt01: <p>I solved the problem with reference to the below solution.</p>
<blockquote>
<p>If you have this issue while installing your modules from requirements.txt then you could simply add the following line to the top of your requirements.txt file</p>
</blockquote>
<p><code>-f https://download.pytorch.org/whl/torch_stable.html</code></p>
<p>ref: <a href="https://stackoverflow.com/a/70969085/9879412" class="inline-onebox" rel="noopener nofollow ugc">pytorch - Could not find a version that satisfies the requirement torch==1.7.0+cpu - Stack Overflow</a></p>
Pytorch Equivalent of a Keras code

shijinknox: <p>Hi All …<br>
I am a new comer in deep learning applications coding. I am trying to implement a simple image classification using transfer learning. I got a code in Keras for doing the same using a transfer learning technique. Could you please provide any ideas about the pytorch equivalent of the code?</p>
<p>model = Sequential()<br>
model.add(InceptionV3(include_top=False, pooling=‘max’,input_shape=(224,224,3), weights=“imagenet”))<br>
model.add(Conv2D(32, (5, 5), input_shape=(64, 64, 3)))<br>
model.add(Activation(‘relu’))<br>
model.add(MaxPooling2D((2, 2)))</p>
<p>model.add(Conv2D(64, (3, 3)))<br>
model.add(Activation(‘relu’))<br>
model.add(MaxPooling2D((2, 2)))</p>
<p>model.add(Conv2D(64, (3, 3)))<br>
model.add(Activation(‘relu’))<br>
model.add(MaxPooling2D((2, 2)))</p>
<p>model.add(Flatten())<br>
model.add(Dense(128, activation=‘relu’))<br>
model.add(Dense(29, activation=‘softmax’))<br>
model.summary()</p>

srishti-git1110: <p>Can you please see if <a href="https://pytorch.org/docs/stable/generated/torch.nn.Sequential.html" rel="noopener nofollow ugc"><code>nn.Sequential</code></a> helps?</p>
Random torch "RuntimeError: Connection refused" happened on Github Actions with macos

Ming_Wang: <p>We are working on this reinforcement learning project <a href="https://github.com/alex-petrenko/sample-factory" rel="noopener nofollow ugc">here</a> and we used GitHub Actions to build our testing CI system. Basically, we run unit tests with GitHub Actions on linux and macos. We find that the error below randomly occurs on macos.(It can pass when we rerun the job manually.) We cannot reproduce the error on our local machine. The corresponding unit test is <a href="https://github.com/alex-petrenko/sample-factory/blob/sf2/tests/algo/test_pbt.py" rel="noopener nofollow ugc">here</a></p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "&lt;string&gt;", line 1, in &lt;module&gt;
  File "/usr/local/miniconda/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/usr/local/miniconda/lib/python3.9/multiprocessing/spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
  File "/usr/local/miniconda/lib/python3.9/site-packages/torch/multiprocessing/reductions.py", line 322, in rebuild_storage_filename
    storage = torch.UntypedStorage._new_shared_filename_cpu(manager, handle, size)
RuntimeError: Connection refused
</code></pre>
<p>I’d like to know: 1. Has someone encountered this before? 2. Is this caused by our side or the GitHub Actions side? 2. If it is caused by our side, how can we debug it(since GitHub Actions is a black box to us and we cannot reproduce it locally)? 3. If it is caused by the GitHub Actions side, is there an elegant way to deal with it?</p>
How to pass arguments to datapipe map function

EvanZ: <p>Here’s an example of datapipe:</p>
<pre><code class="lang-auto">    def setup(self, stage: Optional[str]) -&gt; None:
        self.train_dataset, self.val_dataset = (
            FileLister([self._path])
            .filter(filter_fn=json_filter)
            .open_files(mode='rt')
            .readlines()
            .shuffle(buffer_size=self._buffer_length)
            .map(load_json)
            .map(self.map_data)
            .batch(batch_size=self._batch_size, drop_last=self._drop_last).collate()
            .random_split(total_length=self._buffer_length, weights={"train":self._train_fraction,\
                "valid":1-self._train_fraction}, seed=self._seed)
        )

</code></pre>
<p>I have defined <code>map_data</code> as a class method so it has access to class variables:</p>
<pre><code class="lang-auto">    def map_data(self, x):
</code></pre>
<p>This works, but it seems a bit of a hack. The interpreter throws an error if I try to pass extra arguments either like this:</p>
<pre><code class="lang-auto">.map(map_data, args)
</code></pre>
<p>or like this:</p>
<pre><code class="lang-auto">.map(map_data(args))
</code></pre>
<p>Do I need to create a partial function or something like that?</p>
Dataloader error for custom dataset

Mattheww: <p>Hello</p>
<p>I have a custom dataset that looks like this:</p>
<pre><code class="lang-auto">age gender  genre(output)
--  ------  -------------
20  1       HipHop
26  1       Jazz
31  0       Classical
20  0       Dance
</code></pre>
<p>Priorly, I used this method for splitting:</p>
<pre><code class="lang-auto">X = df.drop(columns=["genre"])
y = df["genre"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
</code></pre>
<p>Now, I would like to split the data to train_set and test_set, using random_split, <a href="https://github.com/KarhouTam/SCAFFOLD-PyTorch/blob/master/data/utils/util.py#L14" rel="noopener nofollow ugc">like this</a>, so I changed it to this:</p>
<pre><code class="lang-auto">fileName="dataset.csv"
custom_dataset  = None

with open(fileName, 'r') as f:
    custom_dataset = f.readlines()

client_dataset = torch.utils.data.Subset(custom_dataset, range(0, len(custom_dataset), 2))

val_samples_num = int(len(client_dataset) * valset_ratio)
test_samples_num = int(len(client_dataset) * testset_ratio)
train_samples_num = len(client_dataset) - val_samples_num - test_samples_num

trainset, valset, testset = random_split( client_dataset, [train_samples_num, val_samples_num, test_samples_num] )
</code></pre>
<p>But I keep getting this error:</p>
<pre><code class="lang-auto">for x, y in dataloader:
ValueError: too many values to unpack (expected 2)
</code></pre>
<p>I believe the approach to get the custom dataset in the middle of the routine is not right, and I would appreciate for any advice.</p>
<p>Thanks</p>

akuysal: <p>I executed your code blocks. I only included the following imports. There is no error similar to the ones you mentioned. It works but “client_dataset” includes half of the data in csv file. For example, when you put 6 data in dataset.csv, trainset, valset, testset will include 1 sample. So, you process 3 data instead of 6 for this case.</p>
<p>from torch.utils.data.dataset import random_split<br>
import torch</p>
How to get the commit hash used to make my specific version of pytorch?

Vedant_Roy: <p>Is there a way to check what commit hash was used to build my particular installation of Pytorch?</p>

Vedant_Roy: <p>Nevermind, found it: <a href="https://github.com/pytorch/pytorch/issues/18293" class="inline-onebox" rel="noopener nofollow ugc">Builds should include the git commit hash in torch.version.git_version · Issue #18293 · pytorch/pytorch · GitHub</a></p>
How to replace the weights/biases manually

Sarwar_Khalid: <p>Say I have 2 models with the same architecture but with different weights and biases. I want to replace some of the weights and biases of one model with the weights and biases of the other model. How would I go about doing that?</p>

Salman_Abbasi: <p>Hi, please see this link <a href="https://discuss.pytorch.org/t/copying-weights-from-one-net-to-another/1492" class="inline-onebox">Copying weights from one net to another</a></p>

Sarwar_Khalid: <p><a href="https://discuss.pytorch.org/t/how-to-assign-an-arbitrary-tensor-to-models-parameter/44082/3">This</a> post solved the problem for me.</p>
Reproducibility in Vanilla Pytorch

mike3: <p>I am getting different results for implementations in lightning and in vanilla pytorch, which to my understanding need to be equivalent, and I’d like to find out why. I have taken the code in <a href="https://towardsdatascience.com/from-pytorch-to-pytorch-lightning-a-gentle-introduction-b371b7caaf09" rel="noopener nofollow ugc">From Pytroch to Pytorch Lightning</a>, specifically the code under “full training loop for pytorch” and “full training loop for lightning”, and introduced the following changes:</p>
<ol>
<li>for both: added pl.seed_everything(42) right after the imports.</li>
<li>for lightning: disabled sanity check in the trainer.</li>
<li>for lightning: replaced self.forward calls with self.</li>
</ol>
<p>However, both codes don’t operate identically. Does anybody know why?</p>

JuanFMontesinos: <p>Pl does not really modify your code.<br>
Make sure you use the test_step (which uses torch.nograd and freezers layers such as batchnorm or dropout) in pl.<br>
Also it could be about multiprocessing in the dataloader as there are reported issues (seeds are copied and each worker return exactly the same transforms).</p>
<p>Besides that, check you can reproduce your own model, as I mentioned, pl doesn’t really change the model or the underlying tools.</p>

mike3: <p>Thank you. I will try your suggestions and report back.</p>
PyTorch Geometric: How to use label propagation algorithm?

Rocky_Owl: <p>Hi there,</p>
<p>I am quite new to PyTorch (and geometric deep learning) and am trying to use the PyTorch geometric package to implement the ‘Label propagation’ algorithm for node classification.</p>
<p><strong>Question:</strong> I am struggling to understand the documentation and don’t quite know how to implement it in code. Would someone be able to help me?</p>
<p><strong>Context</strong>: The class is called <code>LabelPropagation</code> and is listed in the <a href="https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html?highlight=label%20propagation#torch_geometric.nn.models.LabelPropagation" rel="noopener nofollow ugc">readthedocs.io</a> page.</p>
<p><strong>Attempt:</strong> Should the ‘main’ portion of the implementation look like:</p>
<pre><code class="lang-auto">class LabelPropagation(torch.nn.Module):
def __init__(self, num_layers, alpha):
        super(LabelPropagation, self).__init__()
        self.num_layers = num_layers
        self.alpha = alpha

    def forward(self, x, edge_index):
        # I don't really know what to put here
        return x
</code></pre>
<p>Any help would be greatly appreciated as I am still learning how to understand and interpret documentation.</p>

mijalapenos: <p>If by any chance you’re still interested in the answer, I found an example of the usage in the PyTorch Geometric source code: <a href="https://github.com/pyg-team/pytorch_geometric/blob/master/examples/label_prop.py" rel="noopener nofollow ugc">examples/label_prop.py</a>. I managed to run it on my custom graph thanks to that, so feel free to contact me if you need further assistance, I’ll try my best.</p>
Question about running a single model from a single GPU to multi workers

yunsang.ju: <p>Hello.<br>
Performing a single model in torchserve as multi workers increases throughput.<br>
Below is a throughput experiment based on the default_workers_per_model value and the thread setting of pythorch, MKL_NUM_THREADS value.<br>
As far as I know, GPUs only perform one thread at a time<br>
I would appreciate it if you could advise me why throughput increases as the default_workers_per_model increases.</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>number_of_netty_threads</th>
<th>netty_client_threads</th>
<th>default_workers_per_model</th>
<th>MKL_NUM_THREADS</th>
<th>job_queue_size</th>
<th>throughput</th>
</tr>
</thead>
<tbody>
<tr>
<td>100</td>
<td>100</td>
<td>1</td>
<td>1</td>
<td>1000</td>
<td>78.1</td>
</tr>
<tr>
<td>100</td>
<td>100</td>
<td>1</td>
<td>2</td>
<td>1000</td>
<td>80.44166667</td>
</tr>
<tr>
<td>100</td>
<td>100</td>
<td>1</td>
<td>4</td>
<td>1000</td>
<td>80.65833333</td>
</tr>
<tr>
<td>100</td>
<td>100</td>
<td>1</td>
<td>8</td>
<td>1000</td>
<td>80.01666667</td>
</tr>
<tr>
<td>100</td>
<td>100</td>
<td>2</td>
<td>1</td>
<td>1000</td>
<td>128.2666667</td>
</tr>
<tr>
<td>100</td>
<td>100</td>
<td>2</td>
<td>2</td>
<td>1000</td>
<td>124.65</td>
</tr>
<tr>
<td>100</td>
<td>100</td>
<td>2</td>
<td>4</td>
<td>1000</td>
<td>126.6</td>
</tr>
<tr>
<td>100</td>
<td>100</td>
<td>2</td>
<td>8</td>
<td>1000</td>
<td>122.75</td>
</tr>
<tr>
<td>100</td>
<td>100</td>
<td>4</td>
<td>1</td>
<td>1000</td>
<td>149.2083333</td>
</tr>
<tr>
<td>100</td>
<td>100</td>
<td>4</td>
<td>2</td>
<td>1000</td>
<td>146.875</td>
</tr>
<tr>
<td>100</td>
<td>100</td>
<td>4</td>
<td>4</td>
<td>1000</td>
<td>148.3666667</td>
</tr>
<tr>
<td>100</td>
<td>100</td>
<td>4</td>
<td>8</td>
<td>1000</td>
<td>145.0916667</td>
</tr>
<tr>
<td>100</td>
<td>100</td>
<td>8</td>
<td>1</td>
<td>1000</td>
<td>148.875</td>
</tr>
<tr>
<td>100</td>
<td>100</td>
<td>8</td>
<td>2</td>
<td>1000</td>
<td>149.75</td>
</tr>
<tr>
<td>100</td>
<td>100</td>
<td>8</td>
<td>4</td>
<td>1000</td>
<td>148.6166667</td>
</tr>
<tr>
<td>100</td>
<td>100</td>
<td>8</td>
<td>8</td>
<td>1000</td>
<td>149.0666667</td>
</tr>
</tbody>
</table>
</div>
How to flatten output?

princess-bell: <p>Hello there,</p>
<p>I’m currently struggling to get a <a href="https://github.com/gmum/geo-gcn" rel="noopener nofollow ugc">SGCN from this repo</a> up and running. During the training step with the FreeSolv dataset I get the following error:</p>
<pre><code class="lang-auto">mat1 and mat2 shapes cannot be multiplied (607x1600 and 64x16)
</code></pre>
<p>This is the SGCN class:</p>
<pre><code class="lang-auto">class SpatialGraphConv(MessagePassing):
    def __init__(self, coors, in_channels, out_channels, hidden_size, dropout=0):
        """
        coors - dimension of positional descriptors (e.g. 2 for 2D images)
        in_channels - number of the input channels (node features)
        out_channels - number of the output channels (node features)
        hidden_size - number of the inner convolutions
        dropout - dropout rate after the layer
        """
        super(SpatialGraphConv, self).__init__(aggr='add')
        self.dropout = dropout
        self.lin_in = torch.nn.Linear(coors, hidden_size * in_channels)
        self.lin_out = torch.nn.Linear(hidden_size * in_channels, out_channels)
        self.in_channels = in_channels

    def forward(self, x, pos, edge_index):
        """
        x - feature matrix of the whole graph [num_nodes, label_dim]
        pos - node position matrix [num_nodes, coors]
        edge_index - graph connectivity [2, num_edges]
        """
        edge_index, _ = add_self_loops(edge_index, num_nodes=x.size(0))  # num_edges = num_edges + num_nodes

        return self.propagate(edge_index=edge_index, x=x, pos=pos, aggr='add')  # [N, out_channels, label_dim]

    def message(self, pos_i, pos_j, x_j):
        """
        pos_i [num_edges, coors]
        pos_j [num_edges, coors]
        x_j [num_edges, label_dim]
        """
        relative_pos = pos_j - pos_i  # [n_edges, hidden_size * in_channels]
        relative_pos = relative_pos.float() # To not cause dtype mismatch between mat1 and mat2
        spatial_scaling = F.relu(self.lin_in(relative_pos))  # [n_edges, hidden_size * in_channels]

        n_edges = spatial_scaling.size(0)
        # [n_edges, in_channels, ...] * [n_edges, in_channels, 1]
        result = spatial_scaling.reshape(n_edges, self.in_channels, -1) * x_j.unsqueeze(-1)
        return result.view(n_edges, -1)

    def update(self, aggr_out):
        """
        aggr_out [num_nodes, label_dim, out_channels]
        """
        aggr_out = self.lin_out(aggr_out)  # [num_nodes, label_dim, out_features]
        aggr_out = F.relu(aggr_out)
        aggr_out = F.dropout(aggr_out, p=self.dropout, training=self.training)

        return aggr_out
</code></pre>
<p>I’m fairly new to ML. Do I need to flatten the output?</p>
How can I use the for statement well?

BobKim: <p>Hello everyone!<br>
I ask for your help in using the for statement for repetitive tasks.</p>
<p>The script I want to change is:<br>
As1 = curve[‘A1’]<br>
As2 = curve[‘A2’]<br>
As3 = curve[‘A3’]<br>
As4 = curve[‘A4’]<br>
As5 = curve[‘A5’]<br>
As6 = curve[‘A6’]<br>
As7 = curve[‘A7’]<br>
As8 = curve[‘A8’]<br>
…<br>
As1000 = curve[‘A1000’]</p>
<p>I want to call a function called curve 1000 times and assign it to each variable.<br>
How should I use the for statement?</p>
Failed call to cuInit: CUDA_ERROR_NOT_INITIALIZED: initialization error

HamidGadirov: <p>Once I am getting next batch from DataLoader, there is an error message:</p>
<p><code>E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:267] failed call to cuInit: CUDA_ERROR_NOT_INITIALIZED: initialization error</code></p>
<p>The program finally crashes with <code>RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR</code> once the code reaches neural net. I have recently updated tensorflow to 2.11.0, PyTorch is 1.13.0+cu117, and I have NVIDIA TITAN V GPU with <code>nvidia-smi</code>:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/3/8327add87639d4d8a35edf764715534cd5f6cf19.png" data-download-href="https://discuss.pytorch.org/uploads/default/8327add87639d4d8a35edf764715534cd5f6cf19" title="Screenshot 2022-11-28 at 19.49.29"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/3/8327add87639d4d8a35edf764715534cd5f6cf19.png" alt="Screenshot 2022-11-28 at 19.49.29" data-base62-sha1="iIfyLesUqLlfpdqtap6GcDHssfn" width="690" height="340" data-dominant-color="091206"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-11-28 at 19.49.29</span><span class="informations">1116×550 25.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I am on conda environment, I have <code>cudnn 7.6.5.32</code> and <code>nvidia-cudnn-cu11 8.5.0.96</code> installed. Any help would be appreciated!</p>

ptrblck: <p>It seems you are running into this error while using TensorFlow, so their discussion board might be a better place to ask.</p>
<aside class="quote no-group" data-username="HamidGadirov" data-post="1" data-topic="167096">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/eb9ed0/48.png" class="avatar"> HamidGadirov:</div>
<blockquote>
<p>E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:267] failed call to cuInit: CUDA_ERROR_NOT_INITIALIZED: initialization error</p>
</blockquote>
</aside>

HamidGadirov: <p>Indeed, the problem is related to TensorFlow, if I remove it training starts without issues. I also asked on the TensorFlow forum.</p>
Availbility of PyTorch cuda code

jambo6: <p>Is the pytorch cuda code for RNNs available?</p>
<p>I’d like to adapt some GRU code to a particular use case, is this viable?</p>

khushi-411: <p>Hey, Indeed PyTorch has great CUDA support for RNN and GRU. To learn more you can refer to its documentation <a href="https://pytorch.org/docs/stable/generated/torch.nn.RNN.html" rel="noopener nofollow ugc">RNN</a> and <a href="https://pytorch.org/docs/stable/generated/torch.nn.GRU.html" rel="noopener nofollow ugc">GRU</a>. Thanks!</p>
<p>PS: if you are looking forward to the source code click <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda/RNN.cu" rel="noopener nofollow ugc">here</a>.</p>

jambo6: <p>Thats great, thanks for the link</p>