akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How does pytorch compute a bidirectional RNN?

zamannafi: <p>Hi there, I am trying to understand the math behind a bidirectional RNN. I understand how the forward hidden state is computed but I am having trouble understanding exactly how the backward/ reverse hidden state is calculated.</p>
<p>I first made a vanilla RNN using Pytorch, then I tried to do it manually as shown in the code:</p>
<pre><code class="lang-auto">torch.manual_seed(1)
n_in, n_out = 3, 5

inpt = torch.randn([6])

inpt = inpt.view(2, 1, 3)

# print(f'inpt: {inpt}, inpt shape: {inpt.shape}')

vanilla_rnn = nn.RNN(n_in, n_out,bidirectional = True)
out, hx = vanilla_rnn(inpt)

print(f'out: {out}, \n\nout shape: {out.shape}, \n\nhx: {hx}, \n\nhx shape: {hx.shape}')

wih = vanilla_rnn.weight_ih_l0
whh = vanilla_rnn.weight_hh_l0
bih = vanilla_rnn.bias_ih_l0
bhh = vanilla_rnn.bias_hh_l0

wihr = vanilla_rnn.weight_ih_l0_reverse
whhr = vanilla_rnn.weight_hh_l0_reverse
bihr = vanilla_rnn.bias_ih_l0_reverse
bhhr = vanilla_rnn.bias_hh_l0_reverse

with torch.no_grad():
    for i in range(hx.shape[0]):
        print(hx[i])
print("--------------------------------")
print("--------------------------------")

tanh = nn.Tanh()
# hid = torch.zeros(2, 1, 5, dtype = torch.float32)
hid_forward = torch.zeros(1, 5, dtype = torch.float32)
hid_reverse = torch.zeros(1, 5, dtype = torch.float32)

for i in range(input.shape[0]):
    x = input[i]
    
    # Forward
    i_forward = x @ torch.transpose(wih, 0, 1) + bih
    h_forward = hid_forward @ torch.transpose(whh, 0, 1) + bhh
    hid_forward = tanh(i_forward + h_forward)
    
    # Reverse
    i_reverse = x @ torch.transpose(wihr, 0, 1) + bihr
    h_reverse = torch.flip(h_forward, (0,1)) @ torch.transpose(whhr, 0, 1) + bhhr
    
    hid_reverse = tanh(i_reverse + h_reverse)

print(hid_forward)
print(hid_reverse)
</code></pre>
<p>You will see the first tensor of hx matches the ‘hid_forward’ but the second tensor of hx is not matching the ‘hid_reverse’. What am I doing wrong here?</p>

zamannafi: <p>ok so I think I got it. I guess the inputs themselves need to be reversed when passing them through the backward/ reverse direction. Here is the code:</p>
<pre><code class="lang-auto">with torch.no_grad():
    for i in range(hx.shape[0]):
        print(hx[i])
print("--------------------------------")
print("--------------------------------")

tanh = nn.Tanh()
output = []
hid_forward = torch.zeros(1, 5, dtype = torch.float32)
hid_reverse = torch.zeros(1, 5, dtype = torch.float32)

# Forward
for i in range(input.shape[0]):
    x = input[i]
    
    i_forward = x @ torch.transpose(wih, 0, 1) + bih
    h_forward = hid_forward @ torch.transpose(whh, 0, 1) + bhh
    hid_forward = tanh(i_forward + h_forward)
    

# Reverse
reverse_input = torch.flip(input, (0, 1))
    
for i in range(reverse_input.shape[0]):
    x = reverse_input[i]
    
    i_reverse = x @ torch.transpose(wihr, 0, 1) + bihr
    h_reverse = hid_reverse @ torch.transpose(whhr, 0, 1) + bhhr
    hid_reverse = tanh(i_reverse + h_reverse)
    
print(hid_forward)
print(hid_reverse)
print("-----------")
</code></pre>
AttributeError in torch_geometric.transforms

Saguaro: <p>Hello,</p>
<p>I have a problem that I cannot understand: even though a module ‘torch_geometric.transforms’ has an attribute ‘AddTrainValTestMask’ according to <a href="https://pytorch-geometric.readthedocs.io/en/latest/modules/transforms.html#torch_geometric.transforms.AddTrainValTestMask" rel="noopener nofollow ugc">documentation</a>, I cannot import it. I keep receiving an error <code>AttributeError: module 'torch_geometric.transforms' has no attribute 'AddTrainValTestMask</code></p>
<p>My Pytorch version is 1.7.1</p>
<p>I took the code from <a href="https://github.com/rusty1s/pytorch_geometric/blob/master/examples/cora.py" rel="noopener nofollow ugc">here</a></p>
<p>Minimum reproducible example:</p>
<pre><code class="lang-auto">import os.path as osp

import torch
import torch.nn.functional as F
from torch_geometric.datasets import Planetoid
import torch_geometric.transforms as T
from torch_geometric.nn import SplineConv

dataset = 'Cora'
transform = T.Compose([
    T.AddTrainValTestMask('train_rest', num_val=500, num_test=500),
    T.TargetIndegree(),
])
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', dataset)
dataset = Planetoid(path, dataset, transform=transform)
data = dataset[0]


class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = SplineConv(dataset.num_features, 16, dim=1, kernel_size=2)
        self.conv2 = SplineConv(16, dataset.num_classes, dim=1, kernel_size=2)

    def forward(self):
        x, edge_index, edge_attr = data.x, data.edge_index, data.edge_attr
        x = F.dropout(x, training=self.training)
        x = F.elu(self.conv1(x, edge_index, edge_attr))
        x = F.dropout(x, training=self.training)
        x = self.conv2(x, edge_index, edge_attr)
        return F.log_softmax(x, dim=1)


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model, data = Net().to(device), data.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-3)


def train():
    model.train()
    optimizer.zero_grad()
    F.nll_loss(model()[data.train_mask], data.y[data.train_mask]).backward()
    optimizer.step()


def test():
    model.eval()
    log_probs, accs = model(), []
    for _, mask in data('train_mask', 'test_mask'):
        pred = log_probs[mask].max(1)[1]
        acc = pred.eq(data.y[mask]).sum().item() / mask.sum().item()
        accs.append(acc)
    return accs


for epoch in range(1, 201):
    train()
    log = 'Epoch: {:03d}, Train: {:.4f}, Test: {:.4f}'
    print(log.format(epoch, *test()))

</code></pre>
<p>Can somebody explain where is the problem?<br>
Thanks!</p>

ptrblck: <p>Maybe your <code>torch_geometric</code> version is too old. Note that this library is not shipped in the PyTorch installation, but is installed separately, so you would have to check its version.</p>

Saguaro: <p>Eventually it was the reason! Thank you very much!</p>
Hub search path?

markdjwilliams: <p>i all,</p>
<p>I’m part of a larger team of researchers and I’m trying to establish how we can share a repository of pre-trained model weights. In general, we do not have unrestricted internet access.</p>
<p>As things currently stand the pytorch hub checks ones of two possible locations for model weights - either <code>$TORCH_HOME/hub/checkpoints</code> or the path provided to an earlier call to <code>torch.hub.set_dir()</code>.</p>
<p>What I’m hoping to achieve is a system whereby “standard” models (such as those provided by torchvision) can be manually downloaded once into an area (which will generally be read-only for most users), and <code>torch.hub.load_state_dict_from_url()</code> should read them from that location when present. Additionally, individual developers may want to install various models from third-party sources and have those stored in their personal local user area (which has read/write permissions) for the purposes of experimentation.</p>
<p>Currently each researcher has to make their own copy of the shared model weights as we cannot isolate models shared among researchers from their local sandbox.</p>
<p>One possible solution would be a colon-separated search path, configurable via environment variable, which would be searched sequentially for cached weights. Downloaded weights would be saved into the first such path which has the appropriate access rights.</p>
<p>Has anybody looked into setting this up this way, or are there any plans to introduce such a mechanism into pytorch core in the future? As things stand it looks like I’d have to re-implement parts of <code>torch.hub</code> myself, but I imagine others may have run into similar requirements at some point.</p>
<p>Thanks,<br>
Mark</p>

ptrblck: <p>I don’t think this feature is currently implemented, but sounds like an interesting use case.<br>
Would you mind creating a feature request with a link to this topic on <a href="https://github.com/pytorch/pytorch/issues">GitHub</a>?<br>
Also, would you be interested to work on such a feature of providing multiple (user defined) locations for the hub search strategy?</p>

markdjwilliams: <p>Thank you, and apologies for the slow reply. I’ll file the feature request. I doubt I’ll be able to set aside the time to implement this properly myself, but if things change then I will certainly have a go.</p>
Need help implementing a custom sliding window

Amiama: <p>Hey guys,</p>
<p>I am running against a problem I cant solve.<br>
Lets say I have two matrices matrix A and matrix B, both are 10x10.<br>
Now I want a sliding window, 3x3 to go through matrix A and matrix B and do the following:<br>
So the image is structured, matrix A, matrix B, and what I want the result to be.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8fa06a2e03e837956bd875df1047b6eeeef60522.png" alt="image" data-base62-sha1="kuzXQHlRFn0P961fArVwkNzvx6i" width="641" height="117"></p>
<p>Next up I take this result matrix sum it up and save it at location e inside of a new matrix.</p>