akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
CUDA + PyTorch + Package Manager on GH200

amill: <p>What is the current best practice for running PyTorch (with CUDA 12) on the NVIDIA Grace Hopper 200 nodes with package management?</p>
<p>The ideal solution would be to have the PyTorch installation within a <code>conda</code> environment, but I see that this is not yet available, as mentioned <a href="https://discuss.pytorch.org/t/conda-pytorch-cuda-12-for-arm64-and-grace-hopper/203285">here</a>.</p>
<p>The alternative I’ve been using is the NGC container, but package installations aren’t persistent when working with the Singularity <code>.sif</code> format. I can decompose the <code>.sif</code> into a writable sandbox, i.e. <code>singularity build --sandbox pytorch pytorch.sif</code>, but I would prefer not to continually rebuild between the <code>.sif</code> after each package installation, and maintaining the sandbox makes a dent in the file limit of my cluster directory.</p>
<p>A third option would be to <code>pip install</code> locally the wheels found <a href="https://github.com/pytorch/pytorch/actions/runs/9115630888?pr=126174" rel="noopener nofollow ugc">here</a>, but I would also like to have, e.g., <code>torchvision</code>.</p>
<p>I also tried building PyTorch from source in a new <code>conda</code> environment but quickly ran into issues when using the compiler in the NVIDIA HPC SDK.</p>

ptrblck: <p>All wheels are available as nightly binaries including <code>torchvision</code> so you could use these. I’m not familiar with Singularity and don’t know what requirements are needed.</p>

amill: <p>I installed the latest nightlies and have most things working, but <code>torch.compile</code> doesn’t see Triton. Do I need to build it from source? I’m not seeing it available with <code>pip</code>.</p>
Question about object localization

as2475: <p>I am building a model to detect a specific molecule in scanning electron microscopy images. This molecule is the only one I’m interested in, and there could be multiple instances of it in a single image. I initially planned to train a YOLOv8 model using images I labeled, but I’m concerned that, due to the difficulty in identifying the molecule, I may not be able to label all instances accurately, potentially introducing label noise. How much should I be worried about this, and are there strategies to address the issue? For example, if I were to train the model using only image patches where I am 100% certain the molecule is either present or absent, the training process would be easier. However, I’m unsure how to then use this model to localize the molecule in full images. I would appreciate any suggestions.</p>

Soumya_Kundu: <p>No matter what strategy you run, you need to have the right labels to gauge how well your model is doing.</p>

as2475: <p>Sure, but is there a strategy to train a network on a classification task (molecule present/absent) on a high quality dataset, then use this network for the more challenging localization task only for inference?</p>

Soumya_Kundu: <p>Assuming your patch classifier is a 100% correct,</p>
<p>You can use those weights for your detection/localisation backbone.</p>
<p>You would need to potentially retrain this network. Or may be freeze the entire network and retrain only a few layers.</p>

as2475: <p>Thanks for the reply. How would you handle the fact that the extracted patch and the original image are of different dimensions?</p>

Soumya_Kundu: <p>You’re going to feed the detector only the patch, not the original image.</p>
<p>Plus there are ways to handle difference of size anyways: <a href="https://discuss.pytorch.org/t/how-to-change-input-size-from-pretrained-model/187206" class="inline-onebox">How to Change Input Size from Pretrained Model?</a></p>
How to turn a autograd.Function into custom_ops

mksit: <p>I want to wrap an existing <code>torch.autograd.Function</code> without modifying the internal to a custom operator, but the <code>torch.library.register_autograd</code> api requires <code> backward</code> and <code>setup_context</code> functions separately. What is the proper way for turning a <code>torch.autograd.Function</code> into custom_ops?</p>
Confusion Matrix - wrong calcualtions?

Codenamics: <p>Hey,</p>
<p>I use <code>torcheval</code> for CM</p>
<pre><code class="lang-auto">model.eval()
f1 = BinaryF1Score()
bcm = BinaryConfusionMatrix()
with torch.no_grad():
    correct = 0
    total = 0
    for images, labels in test_loader:
        images = images.to(device)
        labels = labels.to(device)
        outputs = model(images)
        accuracy.update(outputs.squeeze(1), labels)
        f1.update(outputs.squeeze(1), labels)
        bcm.update(outputs.squeeze(1), labels)
    accc = accuracy.compute()
    f1_score = f1.compute()
    bcm = bcm.compute()

    print(f"Test accuracy = {accc}")
    print(f"F1 score = {f1_score}")
    print(f"Confusion Matrix = {bcm}")
</code></pre>
<p>results</p>
<pre><code class="lang-auto">Test accuracy = 0.921103835105896
F1 score = 0.8769050240516663
Confusion Matrix = tensor([[145.,  89.],
        [ 16., 374.]])
</code></pre>
<p>buttt if you add up TP + FN this is no way correct for 91%</p>

Gillian: <aside class="quote no-group" data-username="Codenamics" data-post="1" data-topic="208115">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/codenamics/48/61306_2.png" class="avatar"> Codenamics:</div>
<blockquote>
<p><code>accuracy</code></p>
</blockquote>
</aside>
<p>accuracy is not initialized in your code. I guess you used it already in a earlier loop and the results get mixed with the new ones.</p>

Codenamics: <p>accuracy is initialzied before training loop</p>
<p>but my quesition is about CM</p>

Tahamustapha_Nehdi: <p>Hi Bro<br>
can you provide use the full code s we can test with u and try to find solution ?</p>

Gillian: <p>What’s exactly your question if it’s only about CM and not about the accuracy ?</p>

Codenamics: <p>hey</p>
<p>we can close it as i figure it out</p>
<p>i swtiched to Lighting metrics</p>
<p>acc = BinaryAccuracy().to(device)(preds, labels)<br>
precision = BinaryPrecision().to(device)(preds, labels)<br>
recall = BinaryRecall().to(device)(preds, labels)<br>
cm = BinaryConfusionMatrix().to(device)(preds, labels)</p>
<p>print(f"Val Accuracy: {acc}“)<br>
print(f"Val Precision: {precision}”)<br>
print(f"Val Recall: {recall}“)<br>
print(f"Confusion Matrix:\n {cm}”)</p>
<p>Val Accuracy: 0.8060897588729858<br>
Val Precision: 0.7695390582084656<br>
Val Recall: 0.9846153855323792<br>
Confusion Matrix:<br>
tensor([[119, 115],<br>
[  6, 384]], device=‘cuda:0’)</p>
Error loading c10_cuda.dll or one of its dependencies

Dom83: <p>Hello. Please help me with this issue.</p>
<p>I had an older version of Pytorch perfectly working with an older version of ForgeUI. New versions of ForgeUI need a newer version of Pytorch though, it said <code>module 'torch' has no attribute 'float8_e4m3fn'</code> so I updated it like this:<br>
<code>python.exe -m pip install torch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 --index-url https://download.pytorch.org/whl/cu118</code><br>
but I have<br>
<code>error loading c10_cuda.dll or one of its dependencies</code><br>
I tried other versions, cu118 or cu121, still always the same error.</p>
<p>I opened c10_cuda.dll in Dependency Walker and here is what it says about the missing dependencies:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1bce3fc14dbdeddba5daed6ee56d7225c7bfca70.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/1bce3fc14dbdeddba5daed6ee56d7225c7bfca70" title="Dependency Walker c10_cuda.dll"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1bce3fc14dbdeddba5daed6ee56d7225c7bfca70_2_690x264.jpeg" alt="Dependency Walker c10_cuda.dll" data-base62-sha1="3XYMfygllspPff6WjStzvZPfK1y" width="690" height="264" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1bce3fc14dbdeddba5daed6ee56d7225c7bfca70_2_690x264.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/1/b/1bce3fc14dbdeddba5daed6ee56d7225c7bfca70.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/b/1bce3fc14dbdeddba5daed6ee56d7225c7bfca70.jpeg 2x" data-dominant-color="E8E0E0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Dependency Walker c10_cuda.dll</span><span class="informations">746×286 199 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Please help.</p>
First occurrence of unique values in a tensor

welp: <p>The torch.unique() function returns the unique values and the inverse indices,  but unlike np.unique, doesn’t return the indices of the first occurrence of these unique values. Is there any way of doing this in pytorch without using a for loop for each unique value?</p>

pietroastolfi: <p>Hi,<br>
I was wondering the same problem today, and I’ve found a tricky workaround (for sure not elegant).<br>
Here’s the example code:</p>
<pre><code class="lang-auto">a = torch.rand(2,2)
a = torch.vstack([a, a[0]])
uni, rev_idxs = a.unique(dim=0, return_inverse=True)
first_occ_idxs = rev_idxs.unique()
print(a[first_occ_idxs])
</code></pre>
<p>If anyone can suggest a more elegant solution, I would adopt it.</p>

simon-schaefer: <p>Hi,<br>
I came across the PyTorch Unique package, which implements the unique function as follows:</p>
<pre><code class="lang-auto">unique, inverse = torch.unique(x, sorted=True, return_inverse=True)
perm = torch.arange(inverse.size(0), dtype=inverse.dtype, device=inverse.device)
inverse, perm = inverse.flip([0]), perm.flip([0])
perm = inverse.new_empty(unique.size(0)).scatter_(0, inverse, perm)
</code></pre>

daniilino: <p>Hey there.</p>
<p>Maybe it’s not relevant anymore, but here’s solution that i’ve used:</p>
<pre><code class="lang-auto">def find_first_occurence(x, value):
    x_value_mask = (x==value).type(torch.int)
    idx = torch.arange(1, x.shape[-1] + 1).unsqueeze(0) # so that "0" is going to be a marker for lack of occurence
    idx = x_value_mask  * idx
    idx = torch.where(idx!=0, idx, float("nan"))
    idx = idx.sort(-1).values[:, 0]
    return idx

my_tensor = torch.tensor([
    [0 ,0 , 2],
    [0 ,1 , 1],
    [1 ,4 , 0],
])

idx = find_first_occurence(obj_nrm, 1)
</code></pre>

yclin: <p>came across this nice solution: <a>https://stackoverflow.com/questions/72001505/how-to-get-unique-elements-and-their-firstly-appeared-indices-of-a-pytorch-tenso</a></p>
Does FSDP support uneven inputs like DDP using Join Hook

pytorch-learning: <p>Does FSDP support uneven inputs like DDP using Join Hook</p>

agu: <p>It does not support the join hook.</p>

agu: <p>We have found that most people solve the uneven-batch-size-per-rank issue at the dataloader level now.</p>

windsornguyen: <p>What’s the most common method of resolving it at the dataloader level?</p>

agu: <p>The dataloader ensures every rank gets the same local batch size.</p>

pytorch-learning: <p>Thanks for your reply.</p>
Does `torch::IValue` hold and own the underlying tensor?

Fei_Liu: <p>My code needs to use a tensor (wrapped by <code>IValue</code>) 's underlying data_ptr for some raw array operations (interacting with Eigen). I want to make sure I’m not messing up the object lifetime, and I want to ask if the following c++ code is safe:</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">// x is a `torch::IValue` which holds a float tensor
float* ptr = x.toTensor().data_ptr&lt;float&gt;();
// Do I have a valid pointer from now on?
// Guarantees: `x` will remain live and constant
// Worries: the `x.toTensor()` tmp object has gone out of scope.
// Question: does `x` hold the underlying tensor and data for me
//     even though the tmp is gone?
</code></pre>
Best practice to use LayerNorm with reduced precision

YuA: <p>I want to know how people are using LayerNorm with reduced precisions (float16, bfloat16) .</p>
<p>Main questions are:</p>
<ul>
<li>Do I need to cast inputs back to float32 for accurate layer norm computation?</li>
<li>Does LayerNorm casts inputs with reduced precisions to float32 automatically?</li>
</ul>
<p>Thank you .</p>

J_Johnson: <p>What problem are you trying to solve by using higher precision in this layer?</p>
<p>Normally the LayerNorm operation is handled in whatever precision you have designated your model parameters and inputs as.</p>
<p>Are you getting an error otherwise?</p>

YuA: <p>Thank you for clarification, Mr Johnson.</p>
<p>My model (with many transformer layers) is noticablly sensitive to numerical precisions. Therefore, I am trying to find out particularly sensitive points in my model.  I am not getting errors.<br>
So, I was wondering if forcing LayerNorms to compute in float32 might be make my model more robust to numerical errors. I wanted to know if such a practice (force using float32 for LayerNorm only) is common, or, such a concer is completely irrelevant.</p>
<p>Thanks again,</p>

J_Johnson: <p>While I can’t speak for your particular case, lower precision during training generally tends to result in an additional level of model regularization (i.e. to prevent overfitting) and is desirable for this(among other reasons, such as reduced VRAM usage).</p>
<p>With that said, if a model were trained on a higher precision, lower precision during inference time may result in lower accuracy and other performance metrics than the precision it was trained on.</p>
<p>However, if I’m not mistaken, a model in <code>.eval()</code> during inference has NormLayers and other types of regularization layers turned off.</p>

YuA: <p>I didn’t know that using low precision helps regularization. I will stick to using it.</p>
<p>Based on what you suggested, it seems that forcing LayerNorm to use float32 even during mixed precision training (or, inference) is not a common practice. Thank you for your comments.</p>
<p>Actually, I created this post after looking at the source code of pytorch’s <a href="https://pytorch.org/torchtune/stable/_modules/torchtune/modules/rms_norm.html#RMSNorm" rel="noopener nofollow ugc">RMSNorm</a>.<br>
Inside this code, they first cast input to float32, and then they cast it back to the original precision at the end, perhaps for better precision of normalization.</p>
<p>Do you (or someone) know if such a practice is also disired for LayerNorm if a model is sensitive to numerical errors?</p>
<p>Thanks again!</p>
How to fix the weights and biases except two?

cnn: <p>After training a model, how can I save the parameters of the network (e.g., weights and biases) and apply them to a new equal network BUT where I want to change only the first two weights?</p>
<p>I would like to see how the loss changes when only the first two weights varies (providing a meshgrid space) and all the others are fixed.</p>
<ol>
<li>How to fix the weights and biases except two?</li>
<li>How to provide the domain in which the two free weights can span?</li>
</ol>
<pre><code class="lang-auto">class Net(nn.Module):

    def __init__(self, layer_sizes, activation=torch.tanh):
        super(Net, self).__init__()
        self.layer_sizes = layer_sizes
        self.layer_list = [x]
        self.activation = activation

        self.l1 = nn.Linear(layer_sizes[0], layer_sizes[1])
        self.l2 = nn.Linear(layer_sizes[1], layer_sizes[2])
        self.l3 = nn.Linear(layer_sizes[2], layer_sizes[3])
        self.l4 = nn.Linear(layer_sizes[3], layer_sizes[4])

    def forward(self, x):
        x = self.l1(x)
        #print(x.dtype)
        x = self.activation(x)
        x = self.l2(x)
        x = self.activation(x)
        x = self.l3(x)
        x = self.activation(x)
        x = self.l4(x)
        return x

device         = 'cpu'
layer_sizes = [1,32,32,32,1]
activation    = torch.tanh

net_u          = Net(layer_sizes, activation).to(device)

... # training

# saving 
info2exp = []
for layer in nets[0].children():
    if isinstance(layer, nn.Linear):
        info2exp.append(layer.state_dict())

torch.save([data, epochs,  info2exp], r'./dataStorage/test_v1.pt')
</code></pre>
<p>How to load them to a new network? Like this?</p>
<pre><code class="lang-auto">net_new          = Net(layer_sizes, activation).to(device)
net_new.layers.load_state_dict(net_u.layers.state_dict())

print((net_new.layers[0].weight == net_u.layers[0].weight).all())
</code></pre>
<ol>
<li>How to fix the weights and biases except two?</li>
<li>How to provide the domain in which the two free weights can span?<br>
Thanks!</li>
</ol>

JuanFMontesinos: <p>Hmmm<br>
Esentially what you save is a dicitonary.<br>
The simplest way is changing the values in the state_dict before the new network loads them.<br>
Something like</p>
<pre data-code-wrap="python"><code class="lang-python">weights = net.state_dict()
weights['layer_name1'] = ... 
weights['layer_nameN'] = ...
new_net.load_state_dict(weights)
</code></pre>
<p>To freeze certain layers: <a href="https://discuss.pytorch.org/t/how-the-pytorch-freeze-network-in-some-layers-only-the-rest-of-the-training/7088" class="inline-onebox">How the pytorch freeze network in some layers, only the rest of the training?</a></p>

J_Johnson: <p>A1. If you want to stop training on any layer of a model, just set the <code>requires_grad</code> variable to <code>False</code>. For example:</p>
<pre><code class="lang-auto">model.l1.weight.requires_grad = False
model.l1.bias.requires_grad = False
</code></pre>
<p>A2. Not clear on what you’re asking.</p>
Custom image not being classified correctly (MNIST digit recognizer)

coder100: <p>The model seems to work fine with all the images on the MNIST dataset, and with large accuracy as well. However, when I provide my own 28x28 images that are black and white, and I do invert them as well, they don’t seem to be classified as well with high accuracy.</p>
<p>I trained the set with the following transforms:</p>
<pre data-code-wrap="py"><code class="lang-py">transforms.Compose([
  transforms.ToTensor(),
  transforms.Normalize(0.5, 0.5)
])
</code></pre>
<p>And here’s how I load my own images:</p>
<pre data-code-wrap="py"><code class="lang-py">import numpy as np
import matplotlib.pyplot as plt

img = plt.imread("number.png")

def rgb2gray(rgb):
    return np.round(1 - np.mean(rgb, -1), decimals=1)

transform = transforms.Compose([
  transforms.ToTensor(),
  transforms.Normalize(0.5, 0.5)
])

img = rgb2gray(img)
img = transform(img)
</code></pre>
<p>You can view the code here: <a href="https://github.com/cursorweb/fingers-guesser-project/tree/master/mvp" rel="noopener nofollow ugc">code</a>.</p>
Shrinking one dim to 1 and keeping the last non-zero tensor

Andrew_Langdon: <p>Can’t tell if this is the right place to post this. Let me know if I’ve made a misstep in any part of this process:</p>
<p>I’m experimenting with training an LSTM model where the input is batches of sequences of embedded tokens. I created a collate_fn for the dataloader that padded every sequence in the batch with zero vectors to match the longest sequence in the batch.</p>
<p>Last LSTM output is passed to a FC layer and a sigmoid for output.</p>
<p>Model was broken at this point because I was just taking the last LSTM output, which I guess was the output after the model had seen a lot of zeros. So now I’m trying to find the output tensor corresponding to the last non-zero input and I’m using the code below, is there a better way?</p>
<pre><code class="lang-auto">class LSTMModel(nn.Module):
    def __init__(self, embedding_dim, output_dim, hidden_dim, num_layers, drop_prob=0.5):
        super().__init__()
        self.lstm = nn.LSTM(embedding_dim, hidden_dim, num_layers=num_layers, dropout=drop_prob, batch_first=True)
        self.fc = nn.Linear(hidden_dim, output_dim)
        self.sigmoid = nn.Sigmoid()
        
        
    def forward(self, X):
        lstm_out, hidden = self.lstm(X)
        
        # shape is (batch_size, max_seq_len, embedding_dim)
        X_reduced = torch.sum(X, dim=2) # shape is (batch_size, max_seq_len)
        non_zero_count = torch.count_nonzero(X_reduced, axis=1) # shape is (batch_size)
        last_non_zero_index = non_zero_count.long() - 1 # convert counts to zero-index
        
        full_first_idx = torch.arange(lstm_out.size(0)) # tried lstm_out[:, last_non_zero_index] but that was giving me a shape of (batch_size, batch_size, embedding_dim)
        last_non_zero_vectors = lstm_out[full_first_idx, last_non_zero_index]
        
        out = self.fc(last_non_zero_vectors)
        out = self.sigmoid(out)
        return out

for input_batch, labels in train_loader:
    test_model = LSTMModel(EMBEDDING_DIM, OUTPUT_DIM, hidden_dim, num_layers, drop_prob)
    test_model = test_model
    test_model(input_batch)
</code></pre>
Device Registration Mechanism for New backend

amitsrivastava78: <p>Hi All,<br>
I am new to pytorch and would like to register a new device with pytoch, can some one point me to a good documentation for the same along with some example codes.</p>
<p>Regards<br>
Amit</p>

Hacker1337: <p>Hello,<br>
I found extremely useful the file testing functionality of custom backend. It gives minimal example of new device secretly storing data on CPU RAM, and showcasing most of the things you might need.<br>
Permalink <a href="https://github.com/pytorch/pytorch/blob/7fd3b6988681557a4dfc35dc99807cdd78e805c9/test/cpp_extensions/open_registration_extension.cpp" class="inline-onebox" rel="noopener nofollow ugc">pytorch/test/cpp_extensions/open_registration_extension.cpp at 7fd3b6988681557a4dfc35dc99807cdd78e805c9 · pytorch/pytorch · GitHub</a></p>
<p>It is compiled dynamically in this file <a href="https://github.com/pytorch/pytorch/blob/7fd3b6988681557a4dfc35dc99807cdd78e805c9/test/test_cpp_extensions_open_device_registration.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/test/test_cpp_extensions_open_device_registration.py at 7fd3b6988681557a4dfc35dc99807cdd78e805c9 · pytorch/pytorch · GitHub</a><br>
however can be also compiled as a static python library in the way explained in this tutorial <a href="https://pytorch.org/tutorials/advanced/extend_dispatcher.html" class="inline-onebox" rel="noopener nofollow ugc">Extending dispatcher for a new backend in C++ — PyTorch Tutorials 2.4.0+cu121 documentation</a></p>
<p>Relevant documentation comments on the functionality used there can be found here <a href="https://pytorch.org/tutorials/advanced/privateuseone.html" class="inline-onebox" rel="noopener nofollow ugc">Facilitating New Backend Integration by PrivateUse1 — PyTorch Tutorials 2.4.0+cu121 documentation</a><br>
and here <a href="https://pytorch.org/tutorials/advanced/dispatcher" class="inline-onebox" rel="noopener nofollow ugc">Registering a Dispatched Operator in C++ — PyTorch Tutorials 2.4.0+cu121 documentation</a></p>
<p>Hope it would be helpful for someone!</p>
Batchnorm1d vs Standardisation

kkajsa: <p>As far as I understand, Batchnorm1d will do standardisation if we set <code>affine=False</code>, according to a related discussion: <a href="https://discuss.pytorch.org/t/what-is-standard-scale-of-batchnorm1d/43230">What is standard scale of BatchNorm1d?</a><br>
and the implementation of barlow twins: <a href="https://github.com/facebookresearch/barlowtwins/blob/main/main.py#L212" rel="noopener nofollow ugc">github</a></p>
<p>However, I found there are some difference between Batchnorm1d and standardisation in a toy example:</p>
<pre><code class="lang-auto">x = torch.randn(3,2) 
'''
tensor([[-0.3176, -1.0842],
        [-1.1290, -0.0545],
        [-1.7591, -0.5150]])
'''
bn = nn.Batchnorm1d(2, affine = False)
y = bn(x)
'''
tensor([[ 1.2727, -1.2655],
        [-0.1024,  1.1794],
        [-1.1703,  0.0860]])
'''

m,std = x.mean(0), x.std(0)
'''
m:
tensor([-1.0686, -0.5512])
std:
tensor([0.7227, 0.5158])
'''
(x-m)/std
'''
tensor([[ 1.0392, -1.0333],
        [-0.0836,  0.9630],
        [-0.9556,  0.0702]])
'''
bn.state_dict()
'''
OrderedDict([('running_mean', tensor([-0.1069, -0.0551])),
             ('running_var', tensor([0.9522, 0.9266])),
             ('num_batches_tracked', tensor(1))])
'''
var = x.var(0)
m,std,var
'''
tensor([-1.0686, -0.5512])
tensor([0.7227, 0.5158])
tensor([0.5222, 0.2660])
</code></pre>
<p>We can see from the above toy example, the mean calculated using <code>torch.mean()</code> is larger than that in <code>Batchnorm1d</code> by a factor of 10, while the variance is very different.</p>
<p>So I am wondering why such a difference exists?</p>

ptrblck: <p>You can refer to <a href="https://github.com/ptrblck/pytorch_misc/blob/31ac50c415f16cf7fec277dbdba72b9fb4d732d3/batch_norm_manual.py#L59-L76">this code</a> showing how <code>BatchNorm</code> layers apply the normalization.<br>
In your example the <code>var</code> calculation is wrong and you are also comparing the <code>running_*</code> stats to the actual batch stats, which is also wrong since your <code>BatchNorm1d</code> layer is in <code>training</code> mode.</p>

kkajsa: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> thanks for your reply, I have verfied the difference between <code>Batchnorm1d</code> and mean variance normalisation is due to my calculation of the standard deviation. After using <code>std = x.std(0, unbiased = False)</code>, I can get the same result as with <code>Batchnorm1d</code>. But I am wondering why we use <code>unbiased = False</code> in <code>Batchnorm1d</code>, doesn’t it make more sense with <code>unbiased = True</code> since we are using the sample standard deviation rather than that from population?  I also found it states ‘We use the unbiased variance estimate … where the expectation is over training mini-batches of size m and are their sample variances.’ in the <a href="https://arxiv.org/pdf/1502.03167" rel="noopener nofollow ugc">section 3.1 of the batchnorm paper</a></p>
The same code for deep k-means on pytorch is much bad comapred with Tensorflow. Why?

Kwin: <p>I run deep K-means with pytorch, but the results is totally bad than original tensorflow v1 implementation.  How to solve it?</p>
<p>The dkm GitHub: <a href="https://github.com/MaziarMF/deep-k-means.git" class="inline-onebox" rel="noopener nofollow ugc">GitHub - MaziarMF/deep-k-means</a><br>
My implementation: <a href="https://github.com/gaokun12/dkm_pytorch.git" class="inline-onebox" rel="noopener nofollow ugc">GitHub - gaokun12/dkm_pytorch</a></p>
How to repeat tensor elements to any size?

JosueCom: <p>Is there a way to replicate/repeat/loop the values of a tensor to any size? I know PyTorch has repeat() and expand(), which allows for repetition of a tensor, but they only works for sizes that are multiples of the original size and it may be computationally wasteful to create tensor of bigger size and then remove the extra values. My question is, there a way to create an periodic sawtooth/triangle waveform/periodic pattern with the core PyTorch package that given a tensor defining a period it will repeat the pattern to any dimension/shape?</p>
<p>If we have a tensor like this:</p>
<pre><code class="lang-python">&gt;&gt;&gt; a = th.arange(1, 3, 0.5) 
&gt;&gt;&gt; a
tensor([1.0000, 1.5000, 2.0000, 2.5000])
</code></pre>
<p>Can we get something like this:</p>
<pre><code class="lang-python">&gt;&gt;&gt; a.replicate(10)
tensor([1.0000, 1.5000, 2.0000, 2.5000, 1.0000, 1.5000, 2.0000, 2.5000, 1.0000, 1.5000])
</code></pre>

ptrblck: <p>You could <code>repeat</code> the tensor and slice it:</p>
<pre><code class="lang-python">a = torch.arange(1, 3, 0.5) 

size = 10
a.repeat(size//a.size(0)+1)[:size]
</code></pre>

JosueCom: <p>But, is there a way to do it with having to create additional value that will be sliced out? I am asking because if I need to do this in a loop, it will be computationally wasteful for periods that are long.</p>

JosueCom: <p>For my own curiosity, how does PyTorch implement repeat under the hood?</p>

JosueCom: <p>Future me here. This was the solution in case anyone may need it. What I was looking for was the sawtooth wave function.</p>
<pre data-code-wrap="python"><code class="lang-python">def sawtooth_wave(t, period = 0.5, amplitude = 1.0, phase = 0.0, move = 0.0):
    x = (t + phase) / period
    return amplitude * (x - th.floor(x)) + move
</code></pre>
<p>You can see some examples here: <a href="https://gist.github.com/wzjoriv/889e5d45e2e2c6f8492def96dccc495b" class="inline-onebox" rel="noopener nofollow ugc">Sawtooth wave function purely in pytorch · GitHub</a></p>
How does the weights being initialized?

L_Z: <p>Let’s say I am using CNN. I am assuming a certain values are being automatically selected as initial weights and then we apply backpropagation. How does these values being selected? Thanks in advance.</p>

ptrblck: <p>Each module uses its <code>reset_parameters()</code> method as seen <a href="https://github.com/pytorch/pytorch/blob/8f7d66f0c34632a0e3d0e7f70c492307faa038a6/torch/nn/modules/linear.py#L218">here</a> for <code>nn.Linear</code>.</p>
Is VBIOS version significant to report for reproducibility?

Amos_Haviv_Hason: <p>I’m trying to be as specific as possible in my report for reproducibility of the model I recently developed.</p>
<p>Across machines with different GPU models the results of training differ, understandably. But while on all our RTX 4090 machines the results are consistent, on our RTX 3090 machines there are two possible results.</p>
<p>After digging deeper, it seems that on our RTX 4090 machines, two VBIOS versions are possible - 95.02.3C.40.95 and 95.02.18.C0.75. However, the results are the same on both of them.</p>
<p>On our RTX 3090 machines, a number of VBIOS versions are possible - 94.02.42.??.?? and 94.02.26.??.??. Those machines with 94.02.42.??.?? produce the same result, and those with 94.02.26.??.?? produce the same (but different from the former) result.</p>
<p>I didn’t find any differences in the frequencies of the RTX 3090 cards though.</p>
<p>So my question is: May different VBIOS versions of the same GPU model lead to different training results? This is not the case on our RTX 4090 machines, but it might be the source of inconsistency on our RTX 3090 machines.</p>

ptrblck: <p>I doubt it, but it would be interesting to see which kernel is causing the different results.</p>

Amos_Haviv_Hason: <p>What do you mean by kernel?</p>

ptrblck: <p>The PyTorch operation causing the non-determinism.</p>