akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Defining model parameters

Namjoon_Suh: <p>Hi all. I am very new to Pytorch and trying to build a simple spectral GNN by myself.<br>
Following is the code I wrote, but somehow it seems like the model parameters are not properly defined, when I create a network and do the model.parameters(). I don’t know how to fix this…<br>
I want to set W_0 and W_1 as the model parameters…</p>
<p>class Spectral_GNN(nn.Module):<br>
def <strong>init</strong>(self, input_units, hidden_units, output_units):<br>
super(Spectral_GNN, self).<strong>init</strong>()</p>
<pre><code>    def normal(shape):
        return torch.randn(size=shape)*0.01
    
    def two():
        return (normal((input_units, hidden_units)),
                normal((hidden_units, output_units)))
    
    self.W0, self.W1 = two()
    
    # Attach gradients
    params = [self.W0, self.W1]
    for param in params:
        param.requires_grad_(True)        
    
def forward(self, X, A):
    D = torch.sum(A, 1)
    D_inv = D**-0.5
    D_inv = torch.diag(D_inv)
    A_hat = D_inv * A * D_inv
    
    aggregate = torch.matmul(A_hat, X)
    propagate = torch.relu(torch.matmul(aggregate, self.W0))
    aggregate = torch.matmul(A_hat, propagate)
    propagate = torch.matmul(aggregate, self.W1)
    
    return propagate
</code></pre>

Andrei_Cristea: <p>Greetings, you can follow <a href="https://discuss.pytorch.org/t/how-to-make-a-tensor-part-of-model-parameters/51037">this prescription</a>.</p>
<p>For example simply wrap this in a <code>nn.Parameter</code>:</p>
<pre><code class="lang-auto">def normal(shape):
    return nn.Parameter(torch.randn(size=shape)*0.01)
</code></pre>
<pre><code class="lang-auto">for foo in gnn.parameters():
    print(foo)

Output:
Parameter containing:
tensor([[-0.0022]], requires_grad=True)
Parameter containing:
tensor([[0.0037]], requires_grad=True)
</code></pre>
Mixture of multinormal distributions

jwyao: <p>Hi, I am wondering if it is possible to create a mixture of multinormal distributions (covariance matrix != identity). I have looked up the examples in the official doc: <a href="https://pytorch.org/docs/stable/distributions.html#mixturesamefamily" class="inline-onebox" rel="noopener nofollow ugc">Probability distributions - torch.distributions — PyTorch 1.12 documentation</a>. But it is still unclear to me if replacing Normal with MultivariateNormal would work.</p>
<p>Thanks in advance!</p>
Implementing a mask layer with a minimum change of code

Chanwoo_Kim: <p>I wanted to implement a mask layer with a minimum change of code, and my original solution was as below.</p>
<pre><code class="lang-auto">class Mask(nn.Module):
    def __init__(self, embed_dim, num_features):
        super().__init__()
        self.embed_dim = embed_dim
        self.num_features = num_features

        self.mask_token = nn.Parameter(torch.zeros(1, embed_dim))

        self.mask=None

    def forward(self, x, mask=None):
        assert self.mask is not None or mask is not None, "'mask' must be set before forward is called or must be passed to forward"
        if mask is None:
            mask=self.mask
        assert mask.shape[0] == x.shape[0], f"batch_size is not equal"
        assert mask.shape[1] == self.num_features, f"num_features is not equal"
        assert ((mask==0) | (mask==1)).all() , "mask is not binary"
        x[mask == 0]=self.mask_token
        return x

# model initialization
model=Bert()
# insert mask layer in front of the module of interest
model.embedder=nn.Sequential(Mask(1500, 200), model.embedder)
# during forward pass, set the mask to use before calling `forward` of the model.
model.embedder[0].mask=torch.ones(1500,200)
y=model(x)
</code></pre>
<p>However, this solution becomes problematic when the module is parallelized (data parallelism). As each input to the module is split along the batch dimension, the batch dimensions of mask and <code>x</code> differ from each other. The batch dimension of <code>x</code> gets low, but the batch dimension of the mask remains the same. Do you have any other ideas on solving the issues?</p>
RN50 Differences Preprocessing with OpenCV and Pillow

donparker1: <p>I have two implementations, C++ (opencv) and Python (pillow), for doing rn50 preprocessing; however the accuracies are different. I know that the differences stem from image resizing step. torchvision  applies antialiasing, but opencv does not. And, using better interpolation in opencv reduces the gap but does not eliminate it.</p>
<p>I have tried to make resize for opencv as close to torchvision (transforms, pillow) as possible but I am still unable to reproduce. Any suggestions on how to resolve ?</p>
<p>C++</p>
<pre><code class="lang-auto">cv::Mat image = cv::imread(filename);

cv::resize(image, image, cv::Size(resize_height, resize_width), 0.5, 0.5,cv::INTER_AREA);
cv::cvtColor(image, image, cv::COLOR_BGR2RGB);

centerCrop(&amp;image, crop_height, crop_width, &amp;image);
image.convertTo(image, CV_32FC3, 1.0f / 255.0f);

 // Normalize  
cv::Mat mean(crop_height, crop_width, CV_32FC3, cv::Scalar(0.485, 0.456, 0.406));
cv::Mat std(crop_height, crop_width, CV_32FC3, cv::Scalar(0.229, 0.224, 0.225));
cv::subtract(image, mean, image);
cv::divide(image, std, image);

 auto options = torch::TensorOptions().dtype(at::kFloat);
 at::Tensor tensor = torch::from_blob(image.data, {1, crop_height, crop_width, 3}).clone();
tensor = tensor.permute({0, 3, 1, 2});
 tensor = tensor.to(at::kFloat);
</code></pre>
<p>Python:</p>
<pre><code class="lang-auto">cal_loader = torch.utils.data.DataLoader(datasets.ImageFolder("calibration_dataset/", transforms.Compose([
                transforms.Resize(256),
                transforms.CenterCrop(224),
                transforms.ToTensor(),
                normalize,

            ])),
            batch_size=batch_size, shuffle=False,
            num_workers=1, pin_memory=True)
</code></pre>