akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Trying to use PINN to solve to equations, however I only manages to solve one

username: hilde
cooked: <p>Hi!</p>
<p>I am using PINN to solve to hyperbolic PDEs. However, the solution to the first equation is the only solution that is close to a true solution, and the second equation is close, but not close enough. I have two boundary conditions. Only one of the conditions are met with the solution of the network. The second boundary condition is not met. I am not sure what code I should provide to make this question easier to answer. Please let me know, and also if I should provide equations.</p>
<p>What i don’t understand is why the network is able to provide a good solution to one of the equations but not the other. These equations are dependent on each other. So any suggestion of what I should look into or investigate more to solve this is highly appreciated.</p>

username: Nikronic
cooked: <p>Hi</p>
<p>I assume you are following the work by <a href="https://github.com/maziarraissi/PINNs" rel="noopener nofollow ugc">maziarraissi/PINNs</a>. The main issue related here is that they use ‘soft constraint’ for satisfying boundary conditions and although it should work on paper, but when the boundary conditions make the loss landscape too complicated, model might not converge to a good solution at all. You can read about problems (and possible solutions) <a href="https://arxiv.org/abs/2109.01050" rel="noopener nofollow ugc">[2109.01050] (arxiv.org)</a>.</p>
<p>One of the easiest way to deal with problem is that put more emphasize on the loss terms corresponding to boundary conditions using weighted sum. This ensures that the optimization won’t ignore the BCs if the they are hard to satisfy.</p>
<p>But in a more general sense, the more complicated the loss landscape gets (more complicated BCs), it becomes harder for the model the predict a good solution and mostly the solution predicted by model is too smooth (low frequency). To enable model to capture more details, a mapping from low-dimensional input to a higher dimension seems useful <a href="https://arxiv.org/pdf/2006.09661.pdf" rel="noopener nofollow ugc">SIREN</a> and <a href="https://arxiv.org/pdf/2006.10739.pdf" rel="noopener nofollow ugc">FourFeat</a>.</p>
<p>PS: If you are in a rush to try this out, don’t feel overwhelmed by the proofs and theories; you can easily intergrate their code into yours (source code if available by authors on github)</p>
<p>Bests</p>

username: hilde
cooked: <p>Thank you for your reply! I was starting to loose my head with this issue.<br>
I will read the suggested literature and have a look at the suggested solutions. Thanks!</p>
Model in DistributedDataParallel must implement and call forward funciton

username: wqingze
cooked: <p>when I use DistributedDataParallel to parallel training, if the model must implement and call forward function, for example</p>
<pre><code class="lang-auto">class ToyMpModel(nn.Module):
    def __init__(self, dev0, dev1):
        super(ToyMpModel, self).__init__()
        self.dev0 = dev0
        self.dev1 = dev1
        self.net1 = torch.nn.Linear(10, 10).to(dev0)
        self.relu = torch.nn.ReLU()
        self.net2 = torch.nn.Linear(10, 5).to(dev1)

    def forward(self, x):
        x = x.to(self.dev0)
        x = self.relu(self.net1(x))
        x = x.to(self.dev1)
        return self.net2(x)

    def encoder(self, x):
        x = x.to(self.dev0)
        x = self.relu(self.net1(x))
        x = x.to(self.dev1)
        return self.net2(x)


def demo_model_parallel(rank, world_size):
    print(f"Running DDP with model parallel example on rank {rank}.")
    setup(rank, world_size)

    # setup mp_model and devices for this process
    dev0 = (rank * 2) % world_size
    dev1 = (rank * 2 + 1) % world_size
    mp_model = ToyMpModel(dev0, dev1)
    ddp_mp_model = DistributedDataParallel (mp_model)

    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_mp_model.parameters(), lr=0.001)

    optimizer.zero_grad()
    # outputs will be on dev1
    #outputs = ddp_mp_model(torch.randn(20, 10))
    outputs = ddp_mp_model.module.encoder(torch.randn(2, 10))
    print(outputs)
    labels = torch.randn(2, 5).to(dev1)
    loss_fn(outputs, labels).backward()
    optimizer.step()

    cleanup()
</code></pre>
<p>ToyMpModel has two function encoder and forward with the same codes, when working with DistributedDataParallel, will <code> outputs = ddp_mp_model.module.encoder(torch.randn(2, 10))</code> be work correctly, parameter in different gpu will synchronize with for example all-reduce</p>

username: ptrblck
cooked: <p>No, I don’t think your usage is will work correctly, since you are explicitly calling the internal <code>.module</code> manually and are thus skipping the <code>DDP</code> wrapper.</p>

username: wqingze
cooked: <p>so, i must wrap the codes into farword functions? DDP has a farword funcion, I am not sure this funcion has something with model’s forward</p>
What is the meaning of "template" “frame" in cuda code?

username: zwyan
cooked: <p>Seems most cuda op codes have both “op_cuda_template” and “op_cuda_frame” function.  The word"template" and “frame” are quite confusing for me… Any background on this naming conversion? Could someone help explain its? <img src="https://discuss.pytorch.org/images/emoji/apple/hugs.png?v=12" title=":hugs:" class="emoji" alt=":hugs:"></p>
How to resize a tensor in a particular order?

username: genjiii
cooked: <p>Let’s say I have a 1 x 3n tensor [a, b, c, d, e, f…]. I want to resize it into a 3 x n tensor, in the particular order [[a, d,…], [b, e,…], [c, f…]]. In other words, the order in which the numbers are populated are different than from just applying resize. How can this be done? Thanks!</p>

username: JuanFMontesinos
cooked: <p>Just take the tensor and do<br>
<code>tensor=tensor.reshape(3,n).T</code></p>
<pre><code class="lang-auto">In [1]: import torch                                                                                    

In [2]: n=10                                                                                            

In [3]: w=torch.arange(3*n)                                                                             

In [4]: w                                                                                               
Out[4]: 
tensor([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
        18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29])

In [5]: q=w.reshape(3,n)                                                                                

In [6]: q                                                                                               
Out[6]: 
tensor([[ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9],
        [10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
        [20, 21, 22, 23, 24, 25, 26, 27, 28, 29]])

In [7]: q=w.reshape(3,n).T                                                                              

In [8]: q[0]                                                                                            
Out[8]: tensor([ 0, 10, 20])
</code></pre>
<p>By this logic you can do any reshaping if the dimensions match. Because you are taking 1 out of m elements. Just replace 3 by any other number which matches and thats all.<br>
In the worst case your dimensions will mismatch, so you will have to pad manually the last values and fill the last row.</p>
How to repeat a tensor in order (with PyTorch)?

username: genjiii
cooked: <p>If I have a 3 x 768 tensor [a, b, c], how can I repeat it such that I get [[a,b,c],[a,b,c],[a,b,c]], instead of [[a,a,a],[b,b,b],[c,c,c]], which is the case when I use torch.repeat_interleave? thanks!</p>

username: pascal_notsawo
cooked: <p>I think you can do it this way:</p>
<pre><code class="lang-auto">n = 3 
m = 4 # 768 (I replace just to be able to visualize the result)

a = torch.tensor([1]*m)
b = torch.tensor([2]*m)
c = torch.tensor([3]*m)

abc = torch.stack([a, b, c], dim=0)

print(abc)

"""
tensor([[1, 1, 1, 1],
        [2, 2, 2, 2],
        [3, 3, 3, 3]])
"""

print(abc.repeat(3, 1, 1))
"""
tensor([[[1, 1, 1, 1],
         [2, 2, 2, 2],
         [3, 3, 3, 3]],

        [[1, 1, 1, 1],
         [2, 2, 2, 2],
         [3, 3, 3, 3]],

        [[1, 1, 1, 1],
         [2, 2, 2, 2],
         [3, 3, 3, 3]]])
"""
# or
print(torch.repeat_interleave(abc.unsqueeze(0), repeats=3, dim=0))
"""
tensor([[[1, 1, 1, 1],
         [2, 2, 2, 2],
         [3, 3, 3, 3]],

        [[1, 1, 1, 1],
         [2, 2, 2, 2],
         [3, 3, 3, 3]],

        [[1, 1, 1, 1],
         [2, 2, 2, 2],
         [3, 3, 3, 3]]])
"""
</code></pre>
Einsum result does not align when I try to merge two matmul together

username: coincheung
cooked: <p>Hi,</p>
<p>My code is like this:</p>
<pre><code class="lang-auto">inten = torch.randn(16, 16, 224, 224)
out13 = torch.randn(16, 8, 224, 224)
outid = torch.cat([inten, out13], dim=1)
wd = torch.randn(32, 16)
wc = torch.randn(32, 8)
wcat = torch.cat([wd, wc], dim=1)
out_wd = torch.einsum('abcd,eb-&gt;aecd', inten, wd)
out_wc = torch.einsum('abcd,eb-&gt;aecd', out13, wc)
out_cat = torch.einsum('abcd,eb-&gt;aecd', outid, wcat)
print((out_wd + out_wc - out_cat).abs().sum())
</code></pre>
<p>I would like to compute einsum with two pairs of tensor and sum up the result, and I feel that I can concat the two pairs and compute with one einsum, but the above code shows a great difference. How could I get over this please？</p>
How to have two Unets with a shared encoder?

username: meshghi
cooked: <p>Hello, I’m trying to have two Unet models that share an encoder and train them simultaneously, but I don’t know how to so. I was wondering if I could get some help here. Thanks!</p>

username: JuanFMontesinos
cooked: <p>Define one encoder and two decoders. Then for each of the decoders you call the same encoder.</p>

username: meshghi
cooked: <p>Thank you <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a> . But, how can I do the skip connections that are in Unet?</p>

username: JuanFMontesinos
cooked: <p>Well, the same way you do it for the UNet…<br>
You run the encoder, get the latent feats + their skip connections and run de decoder, then with the same latent + skip connections run the other decoder.</p>
<p>Ofc this is simpler or harder to do depending on how the UNet u are using is coded and modularized.</p>
<p>edit:<br>
for clarifiying:</p>
<pre><code class="lang-auto">Unet2Dec
__init__
self.encoder = encoder()
self.decoder1=decoder()
self.decoder2=decoder()
forward(...):
latent,skip = self.encoder(input)
o1 = self.decoder(laten,skip)
o2 = self.decoder(latent,skip)
</code></pre>

username: meshghi
cooked: <p>Thanks <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a>. One more quesiton. Can I train each of these decoders on a different dataset? I.e. decoder-1 gets trained on trainset-1 and decoder-2 gets trained on trainset-2.</p>

username: JuanFMontesinos
cooked: <p>Hmmm that is a strange question… the problem is not how to train the decoders but the encoder.</p>
<p>If you pass a sample from dataset through 1 Encoder1 Decoder 1 and then backpropagate, this will affect both e1 and d1.<br>
If you pass a sample from ds2 through e1 d2 and then backpropagate this will affect e1 and d2.</p>
<p>So the encoder will be affected by both datasets while each decoder will be affected by a single dataset.</p>
How to call ELBO loss function consecutively?

username: ItsyP
cooked: <p>Hi all,</p>
<p>I would like to ask if it is possible to call a loss function consecutively multiple times and with different parameters?</p>
<p>This image illustrates what I am trying to do:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e49aea2f432cd8db51b9ff894a146b94df6cd11c.png" data-download-href="https://discuss.pytorch.org/uploads/default/e49aea2f432cd8db51b9ff894a146b94df6cd11c" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e49aea2f432cd8db51b9ff894a146b94df6cd11c_2_422x500.png" alt="Capture" data-base62-sha1="wCkLx1vKTJ0wkc0jr4Z6MKUWeJe" width="422" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e49aea2f432cd8db51b9ff894a146b94df6cd11c_2_422x500.png, https://discuss.pytorch.org/uploads/default/original/3X/e/4/e49aea2f432cd8db51b9ff894a146b94df6cd11c.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/4/e49aea2f432cd8db51b9ff894a146b94df6cd11c.png 2x" data-dominant-color="F3F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">502×594 77.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Need to run the ELBO in step 1 for 400 epochs and the ELBO in step 2 for 600 epochs. <strong>Note:</strong> Both ELBOs have different inputs and I think that this is supposed to be done with a single instance of a model. In addition, I don’t expect people to go and dig into these models, but if you could explain how to do this using your own examples that would be great!</p>
<p>Looking forward to your answer!</p>
Generating 3d model from 2d images

username: mfcs
cooked: <p>Hey everyone!<br>
How you doing?</p>
<p>Could someone point me to one of the first deep learning papers about generating 3d models from 2d images? The base article in the case.<br>
And what would be the state of the art article in this area as well?</p>
<p>Best regards,</p>
<p>Matheus Santos.</p>
Efficient way for online cropping in GPU

username: jhp
cooked: <p>Hi I have a question about online cropping in GPU.<br>
I want to crop the corresponding region from the <code>input image</code> using the box coordinate extracted from RPN, just like roi pooling from the <code>feature map</code>.</p>
<p>Is there a way to efficiently perform it on the GPU?<br>
I would appreciate it if someone recommend a related repo.</p>
What is MyModel.module in distrubuted training

username: israrbacha
cooked: <p>i have gone through a peace of code:</p>
<pre><code class="lang-auto">if args.distributed:
model1=MyModel.module
model2=MyModel.module
</code></pre>
<p>I read the source code of Distributed training but i didn’t get the purpose of “.module”, if anyone knows?</p>

username: ptrblck
cooked: <p>The <code>.module</code> attribute holds the reference to the plain model (i.e. without the <code>DDP</code> mechanism) and <code>DDP</code> will use it internally for its execution.</p>
How do I get the --nproc_per_node=32 value from my python script?

username: Brando_Miranda
cooked: <p>How do I get the flag within my python script that I am passing to <code>torchrun</code> ? I want to set the number <code>--nproc_per_node=32</code> I am passing there automatically rather than making sure the two scripts match (note I want to set the world size myself e.g. I am using cpu parallel jobs and want to choose that value myself thus)</p>
<p>related: <a href="https://discuss.pytorch.org/t/multiprocessing-failed-with-torch-distributed-launch-module/33056/28" class="inline-onebox">Multiprocessing failed with Torch.distributed.launch module - #28 by Brando_Miranda</a></p>
Error while forwarding noise to GAN network

username: gustavo_scholze
cooked: <p>I am creating a fake noise image to forward to my Generator as following:</p>
<pre><code class="lang-python">noise = torch.randn(16, 100, 1, 1, device=var.DEVICE)
# noise shape = torch.Size([16, 100, 1, 1])

fake = netG(noise)
</code></pre>
<p>And then I get the following message when it passes through the BatchNorm layer:</p>
<pre><code class="lang-auto">File "main.py", line 108, in main
    fake = netG(noise)
  File Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "dcgan\generator.py", line 40, in forward
    return self.main(input)
  File "Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "Anaconda3\lib\site-packages\torch\nn\modules\container.py", line 141, in forward
    input = module(input)
  File "Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "Anaconda3\lib\site-packages\torch\nn\modules\batchnorm.py", line 168, in forward
    return F.batch_norm(
  File "Anaconda3\lib\site-packages\torch\nn\functional.py", line 2282, in batch_norm
    return torch.batch_norm(
</code></pre>
<p>Is it some parameter I am messing up?</p>

username: gustavo_scholze
cooked: <p>Apparently, the problem was around the Generator’s Batch Norm input, it was with a different size - it was expecting other value.</p>
Stand-alone Dilation

username: Joseph_Konan
cooked: <p>Is there a simple way to do stand-alone dilation in pytorch? Ideally it would be using a layer, but a function is okay if there is no layer available. I am not looking for a custom solution.</p>
<p>In this example, we want to pass x to the function to produce y.</p>
<pre><code class="lang-auto">import torch

x = torch.tensor([[
    [1, 1, 1, 1],
    [1, 1, 1, 1],
    [1, 1, 1, 1],
    [1, 1, 1, 1],
]])

y = torch.tensor([[
    [1, 0, 1, 0, 1, 0, 1],
    [0, 0, 0, 0, 0, 0, 0],
    [1, 0, 1, 0, 1, 0, 1],
    [0, 0, 0, 0, 0, 0, 0],
    [1, 0, 1, 0, 1, 0, 1],
    [0, 0, 0, 0, 0, 0, 0],
    [1, 0, 1, 0, 1, 0, 1],
]])
</code></pre>

username: KFrank
cooked: <p>Hi Joseph!</p>
<aside class="quote no-group" data-username="Joseph_Konan" data-post="1" data-topic="143361" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/joseph_konan/48/25213_2.png" class="avatar"> Joseph_Konan:</div>
<blockquote>
<pre><code class="lang-auto">y = torch.tensor([[
    [1, 0, 1, 0, 1, 0, 1],
    [0, 0, 0, 0, 0, 0, 0],
    [1, 0, 1, 0, 1, 0, 1],
    [0, 0, 0, 0, 0, 0, 0],
    [1, 0, 1, 0, 1, 0, 1],
    [0, 0, 0, 0, 0, 0, 0],
    [1, 0, 1, 0, 1, 0, 1],
]])
</code></pre>
</blockquote>
</aside>
<p><a href="https://pytorch.org/docs/1.10.1/generated/torch.nn.ConvTranspose2d.html" rel="noopener nofollow ugc">ConvTranspose2d</a> with an appropriate kernel performs such a dilation:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.10.2'
&gt;&gt;&gt; dilate = torch.nn.ConvTranspose2d (1, 1, 2, 2, bias = False)
&gt;&gt;&gt; dilate.weight.requires_grad = False
&gt;&gt;&gt; dilate.weight[0, 0, :, :] = torch.tensor ([[1.0, 0.0], [0.0, 0.0]])
&gt;&gt;&gt; x = torch.ones (1, 1, 4, 4)
&gt;&gt;&gt; x
tensor([[[[1., 1., 1., 1.],
          [1., 1., 1., 1.],
          [1., 1., 1., 1.],
          [1., 1., 1., 1.]]]])
&gt;&gt;&gt; dilate (x)
tensor([[[[1., 0., 1., 0., 1., 0., 1., 0.],
          [0., 0., 0., 0., 0., 0., 0., 0.],
          [1., 0., 1., 0., 1., 0., 1., 0.],
          [0., 0., 0., 0., 0., 0., 0., 0.],
          [1., 0., 1., 0., 1., 0., 1., 0.],
          [0., 0., 0., 0., 0., 0., 0., 0.],
          [1., 0., 1., 0., 1., 0., 1., 0.],
          [0., 0., 0., 0., 0., 0., 0., 0.]]]])
</code></pre>
<p>To work directly with 2d tensors, you may use <code>squeeze()</code> / <code>unsqueeze()</code><br>
to handle the leading singleton dimensions needed by <code>ConvTranspose2d</code>.<br>
You may also trim off the last row and column, if desired.</p>
<p>Best.</p>
<p>K. Frank</p>
How to define Hessian of a custom torch.autograd.Function

username: FelixPetersen
cooked: <p>My model uses custom <code>torch.autograd.Function</code>s, and I would like to define and compute the Hessian of it.<br>
Since only the first derivative is defined (in the backward method), how can I define the second derivative?<br>
I imagine it could be something like a “second_backward” method, but I haven’t found anything like that.</p>
<p>Thank you.</p>

username: KFrank
cooked: <p>Hi Felix!</p>
<aside class="quote no-group" data-username="FelixPetersen" data-post="1" data-topic="143378" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/felixpetersen/48/36491_2.png" class="avatar"> FelixPetersen:</div>
<blockquote>
<p>I would like to define and compute the Hessian of it.</p>
</blockquote>
</aside>
<p>Does <a href="https://pytorch.org/docs/1.10.1/generated/torch.autograd.functional.hessian.html" rel="noopener nofollow ugc">torch.autograd.functional.hessian()</a> fit your use case?</p>
<p>Best.</p>
<p>K. Frank</p>

username: AlphaBetaGamma96
cooked: <p>If you want to define a double backward directly within <code>torch.autograd.Function</code> you can just embed another <code>torch.autograd.Function</code> within the backward of your current <code>torch.autograd.Function</code>. An example is <a href="https://discuss.pytorch.org/t/how-to-customize-the-double-backward/17846/2">here</a></p>
Differential Privacy measures of any trained ML model

username: jarvico
cooked: <p>Hi all,</p>
<p>I wonder if there is any practical way to assess/measure the level of privacy that a trained ML model has using Pytorch?</p>
<p>I know that there are different techniques which helps to guarantee certain level of privacy to the ML models during training. For example, during training, we can apply DP-SGD algorithm to clip the gradients and add some form of noise in terms of Laplacian or Gaussian Distribution (e.g. Pytorch’s Opacus lib) . Or some other ensemble techniques like PATE has been suggested in literature for ensuring the privacy of the models.</p>
<p>But my question is: Assuming I have simple DNN-based classifier (e.g. digit classifier) which I have normally trained, how can I quantitatively measure the model’s privacy in terms of epsilon , delta values based on formal definition of differential privacy represented below?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/5/e52733c5217b7dc653cfe7ecf646a39df532bbb1.png" alt="DP" data-base62-sha1="wHbkwOgLuSC0BllbzBnGhmr2ugV" width="682" height="171"></p>
<p>Best Wishes,<br>
Jarvico</p>

username: jarvico
cooked: <p>I would appreciate anyone who can comment/help on this.</p>
How to do channel-wise scaling

username: thyeros
cooked: <p>I have a 4D tensor X in NxCxWxH like the output from Conv2d. Now, I like to scale up each channel with different constants, like</p>
<pre><code class="lang-auto">for i, each batch:
    for j, each channel:
       Y[i][j] = s[j]*W*H where s[j] is a scalar value 
</code></pre>
<p>I was able to write a for-loop to go over each image in a batch, with a bunch of view changes. I wonder if there is a vectorized way of doing per-channel constant scaling over batches.</p>

username: avish
cooked: <p>Not sure why you are multiplying each s[j] with W*H is not not same as S = S * W * H outside the loop.</p>
<pre><code class="lang-auto">S = S * W * H  # assuming S.shape() = [C]
S = S.rehape(C, 1, 1)
Y = Y * S
</code></pre>
Optimize for loops in torch.jit

username: Etienne_Perot
cooked: <p>Here is a code that is super slow (100ms !!!):</p>
<aside class="onebox githubgist" data-onebox-src="https://gist.github.com/etienne87/24c2a42d5816cfe61c1e1444f3a42a5d">
  <header class="source">

      <a href="https://gist.github.com/etienne87/24c2a42d5816cfe61c1e1444f3a42a5d" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://gist.github.com/etienne87/24c2a42d5816cfe61c1e1444f3a42a5d" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/etienne87/24c2a42d5816cfe61c1e1444f3a42a5d</a></h4>

  <h5>double_reflection_loop_torch.py</h5>
  <pre><code class="Python">torch.jit.script
def drloop(V: torch.Tensor,T: torch.Tensor) -&gt; torch.Tensor:
    n = len(V)
    Ri = torch.zeros((3,), dtype=torch.float32, device=V.device)
    Ri[0] = -T[0][1]
    Ri[1] = T[0][0]
    Ri[2] = 0
    Ris = [Ri[None]]
    
    for i in range(n - 1):   </code></pre>
    This file has been truncated. <a href="https://gist.github.com/etienne87/24c2a42d5816cfe61c1e1444f3a42a5d" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Basically it is a for loop with super simple dot products over vectors of 3 components.</p>
<p>How to optimize this? should i compute the backward myself instead?</p>
Missing class in training set

username: Zeeyuu
cooked: <p>Hi, there seems to be an issue when some class is not present in the training set but in the test/validation set. For example, for MNIST classification, if I take out all digit ‘0’ from the training set, I got the error ‘RuntimeError: expected scalar type Long but found Float’ at step ‘train_loss += criterion(output, categorized_label[i].view(-1))’  under ‘model.eval()’, likely for a . I couldn’t see why in principle missing one class during training should pose such a problem. In my setting this missing of one class is necessary so for the time being I can only add very few data for that supposedly missing class as a compromise.</p>

username: ptrblck
cooked: <p>Removing a class shouldn’t raise a <code>dtype</code> mismatch, so could you post an executable, minimal code snippet to reproduce the issue, please?</p>

username: Zeeyuu
cooked: <p>Hi thanks for the reply. Turns out it was because there was something wrong with the way I preprocess the data. Thanks for letting me know it wasn’t supposed to be the error so that I can troubleshoot it effectively.</p>
Weird inconsistency during evaluation of linear layer

username: ferrophile
cooked: <p>Consider a single linear layer as follows.<br>
<code>x</code> and <code>w</code> are vectors of the same size, and will be the input / weight of this layer.</p>
<pre><code class="lang-python">x = torch.rand(1, 63)
w = torch.rand(1, 63)
fc = Linear(63, 1, bias=False)
</code></pre>
<p>Suppose I evaluate the following block:</p>
<pre><code class="lang-python">permute = np.random.permutation(63)
fc.load_state_dict({'weight': w}, strict=False)
val1 = fc(x)[0, 0].item()
fc.load_state_dict({'weight': w[:, permute]}, strict=False)
val2 = fc(x[:, permute])[0, 0].item()

print(val1)
print(val2)
</code></pre>
<p>Note that <code>val2</code> is computed by applying the same permutation to the elements of <code>x</code> and <code>w</code>. Theoretically, <code>val1</code> and <code>val2</code> should have the same value, i.e. the dot product of <code>x</code> and <code>w</code>.</p>
<p>In practice, when I execute the block on Jupyter for multiple times with different permutations, <code>val1</code> and <code>val2</code> occasionally result in slightly different numbers, such as 14.554329872131348 vs 14.554328918457031.</p>
<p>(Pytorch ver: 1.10.0. Issue persists regardless of using CPU / GPU as device)</p>
<p>I would like to know why this situation occurs. Is it due to some sort of numerical stability issue, or are there some randomized factors that dictate how these layers are evaluated?</p>

username: KFrank
cooked: <p>Hi Hong Wing!</p>
<aside class="quote no-group" data-username="ferrophile" data-post="1" data-topic="143366" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ferrophile/48/46366_2.png" class="avatar"> ferrophile:</div>
<blockquote>
<p><code>val1</code> and <code>val2</code> occasionally result in slightly different numbers, such as 14.554329872131348 vs 14.554328918457031.</p>
</blockquote>
</aside>
<p>This is due to floating-point round-off error.  Repeat your experiment<br>
using double precision and your “slightly different numbers” will agree<br>
to about twice as many decimal places as they do in the single-precision<br>
example you posted.</p>
<p>Best.</p>
<p>K. Frank</p>
Two optimizers for two neural networks?

username: muhammadirfanzafar
cooked: <p>Hi,<br>
I am using two fully connected NNs (net_1 and net_2). Input for second NN is based on the output of first NN. Loss is calculated using outputs of both NNs. Generic code for that purpose is given as:</p>
<pre><code class="lang-auto">net_1 = Net_u()
net_2 = Net_nu()
optimizer_u = Adam(net_1.parameters(), lr=0.001)
optimizer_nu = Adam(net_2.parameters(), lr=0.001)

optimizer_u.zero_grad()
optimizer_nu.zero_grad()

u = net_1(y)
gradients = torch.ones(u.shape[0], u.shape[1])
u_y = grad(u, y, grad_outputs=gradients, create_graph=True)[0]
nu = net_2(u_y)

loss = loss_fn(u, nu)
loss.backward()
optimizer_u.step()
optimizer_nu.step()
</code></pre>
<p>Can anybody kindly tell if this is being done right and two NNs are being learned without any bias or issue?<br>
Is there any better way of doing it?</p>
<p>Thanks!</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/muhammadirfanzafar">@muhammadirfanzafar</a></p>
<p>In your solution, your input into the second model is the gradient of the first</p>
<p>A simpler solution can be the following</p>
<pre><code class="lang-auto">class Net_u(nn.Module):
    def __init__(self):
        super(Net_u, self).__init__()
        self.layer = nn.Linear(4, 5)
    
    def forward(self, x):
        x= self.layer(x)
        return x
    
class Net_nu(nn.Module):
    def __init__(self):
        super(Net_nu, self).__init__()
        self.layer = nn.Linear(5, 1)
    
    def forward(self, x):
        x= self.layer(x)
        return x

net_1 = Net_u()
net_2 = Net_nu()
optimizer_u = optim.Adam(net_1.parameters(), lr=0.001)
optimizer_nu = optim.Adam(net_2.parameters(), lr=0.001)

optimizer_u.zero_grad()
optimizer_nu.zero_grad()

y = torch.rand(2, 4)
loss = torch.nn.MSELoss()

u = net_1(y)
nu = net_2(u)

loss = loss(u, nu)
loss.backward()
optimizer_u.step()
optimizer_nu.step()
</code></pre>

username: muhammadirfanzafar
cooked: <p>My question was about use of optimizers. Defining two optimizers separately for each network is fine?</p>
Train model parallel in special task

username: Guo-Wei_Wong
cooked: <p>I have one GPU card - A100, and had 80GB GPU memory.<br>
in NLP Task  , I try to  using parallel training model, but it not work, due to data is more special</p>
<p>for example to my data.</p>
<p>sample 1 ,        5 batch, each batch size is 500<br>
sample 2,         1 batch, each batch size is 500<br>
sample 3,   1574 batch, each batch size is 500<br>
…<br>
…<br>
sample 1000000,   ? batch, each batch size is 500</p>
<p>as you see, each sample had different batch number,<br>
the best way I can do is try to flatten all of batch number to 1</p>
<p>for example:</p>
<p>sample 3-1,   1 batch, each batch size is 500<br>
sample 3-2,   1 batch, each batch size is 500<br>
…<br>
…<br>
sample 3-1574,   1 batch, each batch size is 500</p>
<p>but I don’t wanna do it, since each batch may had some relationship, if I split them, performence may not well.</p>
<p>so I want to ask is there some method can using single GPU but multiple process method.</p>
<p>here is I find the similar method.</p>
<p><a href="https://pytorch.org/docs/master/notes/multiprocessing.html#sharing-cuda-tensors" rel="noopener nofollow ugc">Multiprocessing best practices — PyTorch master documentation</a></p>
<p>but it not work to me, when I try to running it on jupyter notebook, always show cuda initial error message.</p>
Efficient batch index_select

username: Tangzy7
cooked: <p>Hi all,</p>
<p>I have tried two ways for batch index_select,  but there are still some problems.Here are weight tensor and index:</p>
<pre><code class="lang-auto">W = torch.rand(40000, 1024)
index = torch.randint(0, 40000, size=(256, 2000))
</code></pre>
<p>256 is the batch size</p>
<p>I could achieve a batch index_select by following two ways:</p>
<pre><code> first way:   W[ index, :]

 second way:   torch.gather( W.expand(256,-1,-1) , 1 , index.unsqueeze(2).expand( -1 , -1, 1024) )
</code></pre>
<p>The both result tensors are same and with same size [256, 2000, 1024]</p>
<p>The first ways cost lots of memory because the result tensor does not shared the same memory with source tensor. The second way using gather cost lots of time.</p>
<p>Do anyone have a more efficient way to achieve batch index_select?</p>

username: smth
cooked: <p>regardless of the method, you cannot have the resulting Tensor share the same memory as the source Tensor.<br>
The reason is because you aren’t asking for a slice of the original tensor, but you’re asking for particular indicees of the original Tensor. Only slices can be viewed on the memory of the original Tensor.</p>
<p>Your first way is the efficient version of doing things.</p>
<p>Alternatively, do see if this is faster (probably not, but worth a try):</p>
<pre><code class="lang-auto">W[ index.view(-1), :].view(index.size(0), index.size(1), 1024)
</code></pre>

username: Tangzy7
cooked: <p>Thanks, I also have tried this way, but the memory cost is same with W[ index, :]</p>

username: Tangzy7
cooked: <p>Actually, W is a parameter tensor required gradient.  W[ index, :] might have two elements from same source. Are these two elements will be updated independly in backpropagation?</p>

username: untrix
cooked: <p>Hi <a class="mention" href="/u/smth">@smth</a>, Will backprop work fine through a tensor output from torch.gather? Reason I became suspicious is because I found out that torch.gather on an expanded tensor (i.e., output of torch.expand) resulted in a larger storage area than the original tensor (due to the expansion I am guessing).</p>
<blockquote>
<p>Tensor Y: Shape = (12, 15, 512), Storage = 368704 bytes<br>
Tensor indices: Shape = (12, 15, 5, 512), Storage = 664 bytes<br>
Y_expanded = Y.unsqueeze(2).expand(-1, -1, 15, -1)<br>
Tensor Y_expanded: Shape = (12, 15, 15, 512), Storage = 368704 bytes</p>
</blockquote>
<p>Up until this point, size of Y_expanded == size of Y</p>
<blockquote>
<p>Y_selected = torch.gather(Y_expanded, dim=2, index=m_indices)<br>
Tensor Y_selected: Shape = (12, 15, 5, 512), Storage = 1843264 bytes</p>
</blockquote>
<p>However, after I select from Y_expanded, the storage size increases, implying a copy into new storage. Is there a way I can avoid the increase in size? Also, will backprop flow back through Y if I use this method?</p>
<p>Thanks.</p>

username: divinho
cooked: <p>bump.</p>
<p>There’s lots of people asking for this… any hope of getting it?</p>

username: thecho7
cooked: <p>All we are waiting for a savior…</p>
Semantic segmentation (multi-class) loss not giving 0 when using target as the prediction [either in NLLLoss or CrossEntropyLoss]

username: Charan0
cooked: <p>I wanted to perform semantic segmentation using PyTorch. There are a total of 103 different classes in the dataset and the targets are RGB images with only the <code>R</code> channel containing the labels. I was using <code>nn.CrossEntropyLoss</code> as my loss function. For sanity I wanted to check if using it has the expected behaviour</p>
<p>I pick a random mask from my dataset and create a categorical version of it using this custom transform</p>
<pre><code class="lang-auto">class ToCategorical:
    def __init__(self, n_classes: int) -&gt; None:
        self.n_classes = n_classes

    def __call__(self, sample: torch.Tensor):
        mask = sample.permute(1, 2, 0)
        categories = torch.unique(mask).tolist()[1:]  # get all categories other than 0
        # build a tensor with `n_classes` channels
        one_hot_image = torch.zeros(self.n_classes, *mask.shape[:-1])
        for category in categories:
            # get spacial locs where the categ is present
            rows, cols, _ = torch.where(mask == category)
            # in same spacial loc but in `categ` channel fill 1
            one_hot_image[category, rows, cols] = 1  
        return one_hot_image
</code></pre>
<p>And then I send this image as the output (prediction) and use the ground truth mask as the target to the loss function.</p>
<pre><code class="lang-auto">import torch.nn as nn

mask = T.PILToTensor()(Image.open("path_to_image").convert("RGB"))
categorical_mask = ToCategorical(103)(mask).unsqueeze(0)
mask = mask[0].unsqueeze(0)  # get only the red channel

loss_fn = nn.CrossEntropyLoss()

target = mask
output = categorical_mask

print(output.shape, target.shape)
print(loss_fn(output, target.to(torch.long)))
</code></pre>
<p>I expected the loss to be zero but to my surprise, the output is as follows</p>
<pre><code class="lang-auto">torch.Size([1, 103, 600, 800]) torch.Size([1, 600, 800])
tensor(4.2836)
</code></pre>
<p>I verified with other samples in the dataset and I obtained similar values for other samples as well. Am I doing something wrong? I expect the loss to be = 0 when the output is same as the target.</p>
<p>PS. I also know that <code>nn.CrossEntropyLoss</code> is the same as using <code>log_softmax</code> followed by <code>nn.NLLLoss()</code> but even I obtained the same value in by using nllloss as well</p>

username: ptrblck
cooked: <p>Based on the target shape and the <code>dtype</code> (<code>target</code> is a <code>LongTensor</code>) the target is expected to contain class indices and should not be one-hot encoded.<br>
Could you check the min/max values of <code>target.to(torch.long)</code>? Also <code>nn.CrossEntropyLoss</code> expects logits in the model output so you would have to fill <code>output</code> with negative/positive numbers to achieve a ~0 loss corresponding to the target classes.</p>
How do I create inter-class dependencies?

username: maierf
cooked: <p>Hello,</p>
<p>I am currently trying to improve the segmentation of an encoder-decoder network used to segment railroads.<br>
The goal is to have a network output a segmentation for the treckbed/trainrails that the train is driving on (aka ego track) be as as good (high recall/precision) as possible.<br>
For most scenes, which are mostly just one rail with no merges or forks, the network works really well, but performs poorly in scenes with merges and forks in the railroad.</p>
<p>Example mask:<br>
(Yellow = rails the train drives one: red, green neighbor rails)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e4615cac396c4efd4465485c364ea65e882feac.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/2e4615cac396c4efd4465485c364ea65e882feac" title="example2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2e4615cac396c4efd4465485c364ea65e882feac_2_690x461.jpeg" alt="example2" data-base62-sha1="6Bm7sxJt3ovwB59KrrLxn32Hp6I" width="690" height="461" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2e4615cac396c4efd4465485c364ea65e882feac_2_690x461.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2e4615cac396c4efd4465485c364ea65e882feac_2_1035x691.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/e/2e4615cac396c4efd4465485c364ea65e882feac_2_1380x922.jpeg 2x" data-dominant-color="201809"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">example2</span><span class="informations">3088×2064 146 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>A theory I came up with is to classify the rails and the trackbed separately and use the information of the rails to improve the trackbed segmentation.<br>
This should lead to less false positives due to some areas not being classified as trackbed because its not enclosed by 2 rails.<br>
Also forks and merges have a specific rail layout as seen in the example image. If the ego track was to continue to the left, the red rail on the right within the yellow area would be connected to the yellow rail on the right. Since its not connected the ego track does not fork to the left and continues forward.</p>
<p>The only idea I’ve had is to use multi-label masks adding a class which combines the rails and trackbed as one class on top of the existing classes and segment the parts with both a high probability on the rails or trackbed class and the combined class.<br>
I have not tested this yet and im not even sure if this would actually work.</p>
<p>I cant find any information on how to create a dependency of one class to another.<br>
I could improve the segmentation with some post processing, but this is only a last resort and I would much rather solve this via the NN.</p>
<p>Does anybody have an idea how to attempt to solve this problem?</p>
<p>Thanks.</p>
AssertionError: The only supported dtype for nnq.Embedding is torch.quint8

username: fardinadii
cooked: <p>Hi<br>
im trying to quantize DETR model but I faced with this error:</p>
<p>/usr/local/lib/python3.7/dist-packages/torch/nn/quantized/modules/embedding_ops.py in from_float(cls, mod)<br>
150         dtype = weight_observer.dtype<br>
151<br>
 → 152         assert dtype == torch.quint8, ‘The only supported dtype for nnq.Embedding is torch.quint8’<br>
153<br>
154         # Run the observer to calculate qparams.</p>
<p>AssertionError: The only supported dtype for nnq.Embedding is torch.quint8</p>

username: jerryzh168
cooked: <p>can you set the dtype for weight observer to torch.quint8? I think the default is torch.qint8. can you paste the code that’s used to quantize your model?</p>

username: fardinadii
cooked: <p>thank you for replay,</p>
<p>weights= ‘./model.pth’</p>
<h1>
<a name="load-model-1" class="anchor" href="#load-model-1"></a>load model</h1>
<p>model = torch.hub.load(‘facebookresearch/detr’, ‘detr_resnet50’, pretrained=False, num_classes=7)<br>
checkpoint = torch.load(weights), map_location=device)<br>
model.load_state_dict(checkpoint, strict=False)<br>
model.eval()<br>
<span class="hashtag">#model</span>.fuse()<br>
model.qconfig = torch.quantization.get_default_qconfig(‘fbgemm’)<br>
qmodel = torch.quantization.prepare(model)<br>
qmodel_comp = torch.quantization.convert(qmodel)</p>

username: jerryzh168
cooked: <p>I think you need to set the qconfig for the embedding module to be <a href="https://github.com/pytorch/pytorch/blob/1edf6f56477ea317af845a4cd65eb311737961f0/torch/ao/quantization/qconfig.py#L141" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/1edf6f56477ea317af845a4cd65eb311737961f0/torch/ao/quantization/qconfig.py#L141</a></p>
Why doesn't mean work on bool tensors?

username: kirk86
cooked: <p>Why does the following work</p>
<pre><code class="lang-auto">x = torch.tensor([True, False, True])
x.sum().div(len(x))
</code></pre>
<p>and this one not ?</p>
<pre><code class="lang-auto">x = torch.tensor([True, False, True])
x.mean()
</code></pre>
<p>numpy.mean also works on boolean arrays.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="kirk86" data-post="1" data-topic="143322">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kirk86/48/151_2.png" class="avatar"> kirk86:</div>
<blockquote>
<p>numpy.mean also works on boolean arrays.</p>
</blockquote>
</aside>
<p>It seems that <code>numpy</code> doesn’t work in the first code snippet:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.tensor([True, False, True])
x.numpy().sum().div(len(x))
&gt; AttributeError: 'numpy.int64' object has no attribute 'div'
</code></pre>
<p>so the support seems to be swapped. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<p>Feel free to create a feature request to support <code>mean</code> on <code>BoolTensor</code>s and to discuss it with the code owners.</p>

username: kirk86
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="143322">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>It seems that <code>numpy</code> doesn’t work in the first code snippet:</p>
</blockquote>
</aside>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/face_with_open_eyes_and_hand_over_mouth.png?v=12" title=":face_with_open_eyes_and_hand_over_mouth:" class="emoji" alt=":face_with_open_eyes_and_hand_over_mouth:" loading="lazy" width="20" height="20"> that’s bananas, didn’t expect that at all.<br>
Who would have thought of this divergence <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20">.</p>
<p>Cheers!</p>
Do I need to reset my Hidden and Cell State of my LSTM/GRU when training with batch size >1?

username: Patrick1
cooked: <p>I have timeseries Data that I want to classify. In my Training Data I have 3 different Classes. Each class has about 50 examples. Each example is a timeseries constisting of 190 timesteps which is about 1.9 seconds .</p>
<p>Summarized:<br>
3 Classes<br>
Each Class has about 50 examples<br>
Examples consist of 190 timesteps</p>
<p>What I did in the beginning was to train with batch size = 1. Now after my model is working and training I want to try different batch sizes. My professor told me I need to reset my hidden state and cell state when training with batch size &gt;1. So after looking through the internet I came up with my following code but I am wondering if this is right. Constently setting my hidden state to 0 inside my forward function makes me think my model should not be able to learn</p>
<p>Here is my Class</p>
<pre><code class="lang-auto">class Model_LSTM(nn.Module):

    def __init__(self, n_features, n_classes, n_hidden, n_layers):            

        super().__init__()

        self.lstm = nn.LSTM(

            input_size=n_features,

            hidden_size=n_hidden,

            num_layers=n_layers,

            batch_first=True,

            dropout=0.75,

            bidirectional=True,

        )

        self.linear = nn.Linear()

        # HE-Initialisierung

        weight = torch.zeros(n_layers,n_hidden)

        nn.init.kaiming_uniform_(weight)

        self.weight = nn.Parameter(weight)

        self.classifier = nn.Linear(n_hidden, n_classes)


    def init_hidden(self):      

        hidden_state = torch.zeros(self.lstm.num_layers,batch_size,self.lstm.hidden_size)

        cell_state = torch.zeros(self.lstm.num_layers,batch_size,self.lstm.hidden_size)

        return (hidden_state, cell_state)

    def forward(self, x):

        self.hidden = self.init_hidden()

        _, (hidden, _) = self.lstm(x)                  

        out=hidden[-1]                                  

        return self.classifier(out)
</code></pre>
Why does the evaluation time in the pruned model increase compared to the normal model?

username: m.safari
cooked: <p>I trained my model (<a href="https://github.com/facebookresearch/detectron2" rel="noopener nofollow ugc">detectron2 model</a>) and save it then load the model and evaluate it.<br>
Suppose the evaluation time is 15.34 then I prune the model like below:</p>
<pre><code class="lang-auto">def prune_model_l1_unstructured(model, layer_type, proportion):
        for module in model.modules():
            if isinstance(module, layer_type):
                prune.l1_unstructured(module, 'weight', proportion)
                prune.remove(module, 'weight')
        return model
model = prune_model_l1_unstructured(torch_model, nn.Conv2d, 0.5)
torch.save(model.state_dict(), "state_dict_prune_model.pth")
</code></pre>
<p>when I used pruned model(state_dict_prune_model.pth) to evaluate, time is 26.54 !!!</p>
<p>Does anyone know the cause?<br>
Because logically evaluation time should be less, not more</p>
When is the Graph Neutral Network inductive?

username: bowlmeat
cooked: <p>As I have learned, the statement is that <em>“GNNs are naturally inductive because they learn the same neural networks on all the nodes and edges.”</em> Does it mean that all the GNN frame like GCN and GAT can do inductive representation learning?<br>
Also I saw a blog where the writer said that he chose the GraphSage because it can achieve inductive representation learning.<br>
Does it mean only the GraphSage can do this?<br>
So, am I correct in my thinking? Thank you so much!</p>