akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Generate text with torchscripted version of GPT2

RubenAlvarez: <p>Hello! I’m trying to export and perform inference over a GPT2 model from HuggingFaceBefore the conversion to torchscript, the text generation is performed by running model.generate() method from the HuggingFace model to create the response from GPT2 (<a href="https://huggingface.co/docs/transformers/v4.18.0/en/main_classes/text_generation" class="inline-onebox" rel="noopener nofollow ugc">Generation</a>)</p>
<p>However, after the conversion, the following message appear when running model.generate():</p>
<p>‘RecursiveScriptModule’ object has no attribute ‘generate’</p>
<p>Is there a way of generating text from scripted models?</p>
<p>Thanks in advance</p>
Get total memory of GPU

blackbirdbarber: <p>I checked all methods in here <a href="https://pytorch.org/docs/stable/cuda.html#module-torch.cuda" rel="nofollow noopener">https://pytorch.org/docs/stable/cuda.html#module-torch.cuda</a> and could not find single method that can bring me the GPU size of the device.</p>
<p>How to get total memory of GPU device and total available GPU memory on device?</p>

ptrblck: <pre><code class="lang-python">print(torch.cuda.get_device_properties('cuda:0'))
&gt; _CudaDeviceProperties(name='TITAN V', major=7, minor=0, total_memory=12034MB, multi_processor_count=80)
</code></pre>
<p>will give you some information about your device.<br>
Note that the CUDA context (ant other application) might take some memory, which will not be tracked by e.g. <code>torch.cuda.memory_allocated()</code> or <code>torch.cuda.memory_cached()</code>.</p>
Grid_sample() and affine_grid() float imprecision

Mehdi_Shekarnabi: <p>The torch.nn.functional.grid_sample() function requires second argument “grid” to be normalized between -1 and 1, this leads to decimal values that can cause imprecisions in the sampled output.</p>
<p>Take below code as an example, where I simulate a small 3D image and then apply an identity transform on it using grid_sample(), and show that the output is not equal to the input:</p>
<pre><code class="lang-auto">D, H, W = (7,7,7)
shape = lambda g,c,r:((g[0,:,:,:,0] - c[0])/r[0])**2 + ((g[0,:,:,:,1] - c[1])/r[1])**2 + ((g[0,:,:,:,2] - c[2])/r[2])**2 &lt;= 1
grid = torch.nn.functional.affine_grid(torch.eye(3, 4, device=device).unsqueeze(0), (1, 1, D, H, W), align_corners=True)

oval = shape(grid,[0.0,0.0,0.0],[0.4,0.4,0.6])
sphere = shape(grid,[0.0,0.0,0.0],[0.8,0.8,0.8])

intensity1 = -700.0
intensity2 = 150.0
background = -1024.0

image = torch.where(oval,intensity1,torch.where(sphere,intensity2,background)).unsqueeze(0).unsqueeze(0)

image_id = torch.nn.functional.grid_sample(image, grid,align_corners=True)

print((img2 == img1).all())
</code></pre>
<p>Converting tensor types to float64 does not change the output. Neither does, align_corners = True or False.<br>
This becomes a big problem when such small errors accumulate in bigger image sizes. Do you have any solutions or advice?</p>
Batch size and shuffle affecting evaluation

Fling: <p>Would batch size/order affect the behavior of BatchNorm or any other layer when in eval mode?</p>
<p>I have a model trained with batch size 16, and when I evaluate at batch size 16, I get my expected results. When I change the value of batchsize in evaluation, as the batch size decreases, results get worse.</p>
<p>Likewise, shuffling the Dataloaders in evaluation improves results. I assume it’s because shuffled batches are more balanced than the fixed order files, but I don’t know why that would affect results.</p>
<p>I think this is an error in my code but I’d like to ask if I’m misunderstanding intended behavior before I start working on it. I’m using Google Colab and using the same network as the one <a href="https://github.com/milesial/Pytorch-UNet/tree/master/unet" rel="noopener nofollow ugc">here</a>, if that is relevant.</p>

ptrblck: <p>No, the batch size should not have any effect on <code>BatchNorm</code> layers during <code>eval()</code> besides expected small errors potentially due to the limited floating point precision caused by a different order of operations.<br>
Your model also works for me and doesn’t show any difference:</p>
<pre><code class="lang-python">model = UNet(3, 10)
model.eval()

x = torch.randn(10, 3, 224, 224)
# create reference with full batch
out = model(x)

# iterate all samples separately
for idx, x_ in enumerate(x):
    x_ = x_.unsqueeze(0)
    out_single = model(x_)
    print((out_single.squeeze(0) - out[idx]).abs().max())

# tensor(0., grad_fn=&lt;MaxBackward1&gt;)
# tensor(0., grad_fn=&lt;MaxBackward1&gt;)
# tensor(0., grad_fn=&lt;MaxBackward1&gt;)
# tensor(0., grad_fn=&lt;MaxBackward1&gt;)
# tensor(0., grad_fn=&lt;MaxBackward1&gt;)
# tensor(0., grad_fn=&lt;MaxBackward1&gt;)
# tensor(0., grad_fn=&lt;MaxBackward1&gt;)
# tensor(0., grad_fn=&lt;MaxBackward1&gt;)
# tensor(0., grad_fn=&lt;MaxBackward1&gt;)
# tensor(0., grad_fn=&lt;MaxBackward1&gt;)
</code></pre>

Fling: <p>Thanks for the help!</p>
<p>Your mentioning precision errors was very helpful. I found that the code I copied was using a long to store the mask values so the precision errors were quite high</p>
Torch.inverse cuda error

cbd: <p>I used “a = torch.inverse(mat_inverse)” line in my code. It works fine on google colab but gives below error when run on GPU.</p>
<p>RuntimeError: CUDA error: invalid argument<br>
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.</p>
<p>With <strong>CUDA_LAUNCH_BLOCKING=1</strong> it shows below error. Is it related to memory requirement?<br>
RuntimeError: CUDA error: invalid argument</p>

ptrblck: <p>Could you post a minimal code snippet to reproduce the issue as well as the output of <code>python -m torch.utils.collect_env</code>, please?</p>

cbd: <p>It was memory issue as GPU is shared.</p>

ptrblck: <p>An <code>invalid argument</code> error is usually not raised if you are running out of memory.<br>
Do you remember which operation raised it when you saw the stack trace (in case you were running with <code>CUDA_LAUNCH_BLOCKING=1</code>)?</p>

cbd: <p>You are right. Its not memory issue. The function gives error is “torch.inverse”. Please find the details you required. Let me know if you could resolve.</p>
<pre><code class="lang-auto">PyTorch version: 1.10.0+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.2 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.10.2
Libc version: glibc-2.25

Python version: 3.6.9 (default, Jan 26 2021, 15:33:00)  [GCC 8.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-91-generic-x86_64-with-Ubuntu-18.04-bionic
Is CUDA available: True
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: Quadro P5000
Nvidia driver version: 450.119.03
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.4
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.4
[pip3] torch==1.10.0+cu113
[pip3] torchfile==0.1.0
[pip3] torchvision==0.11.1+cu113
[pip3] torchviz==0.0.2</code></pre>

ptrblck: <p>Could you post an executable code snippet to reproduce the issue?</p>

cbd: <p>Its strange but changing the array shape from (1,3,256,256) to (1, 14400, 3, 3) in below code shows error in GPU. On google colab i verified and its working fine. Any solution and cause of issue?</p>
<pre><code class="lang-auto">import torch
#mat_inverse=torch.randn(1,3,256,256).to(0)
mat_inverse=torch.rand(1, 14400, 3, 3).to(0)
a=torch.inverse(mat_inverse)
print(a)</code></pre>

cbd: <p>Have you arrived to some conclusion?</p>

Frank_Yang: <p>I also encountered this problem when excuting torch.inverse and I solved it by:</p>
<p>torch.inverse(params[‘affine’].to(‘cpu’)).to(‘cuda’)</p>

ptrblck: <p>Sorry for not following up earlier as I might have missed your follow-up.<br>
I cannot reproduce the issue in the current release (<code>1.12.0+cu116</code>) using your code so could you update PyTorch and check if you are still seeing the error?</p>
<p>CC <a class="mention" href="/u/frank_yang">@Frank_Yang</a></p>

huangxin168: <p>Thank you very much, you saved me!</p>
Compiling triton with .github/scripts/build_triton_wheel.py

GLZhu: <p>Hi community, I have built pt2.0 with gpu. The code I used is from github repo.<br>
I used python setup.py install and when run python -m torch.utils.collect_env I cannot find triton.<br>
Hence I tried to build triton with .github/scripts/build_triton_wheel.py.<br>
However during the process it is downloading cuda 12.0.<br>
I know from previous post that cuda 11.7 is prefered, why is there a difference on cuda version for code in the same repo?</p>

ptrblck: <p>The build works for me and creates a wheel without downloading any CUDA version.</p>

GLZhu: <p>Hi Sir ptrblck.</p>
<ol>
<li>is it possible to share how you build triton in pytorch directory? I want to double check to make sure my build cmd is legit.</li>
<li>I thought build pytorch with python setup.py develop will enable the build of triton by default, but it turned out not the case, I am wondering how to make sure triton is enabled in my pytorch build.</li>
<li>I used pytorch 2.0-rc2 and the pined triton commit is d54c04abe2c3e67b2139c68cdbda87b59e8dd01b I think there do exist a different cuda requirement since pytorch 2.0 stables at 11.7 and it seems that triton downloads 12.0.</li>
</ol>

ptrblck: <ol>
<li>I used <code>python .github/scripts/build_triton_wheel.py --py-version 3.8</code> which created <code>pytorch_triton-2.0.0+b8b470bc59-cp38-cp38-linux_x86_64.whl</code> and which matches the pinned commit in <code>.github/ci_commit_pins/triton.txt</code>.</li>
<li>I think you are correct and you would need to build <code>pytorch-triton</code> as a dependency as seen in my command.</li>
<li>Yes, RC2 used <code>d54c04abe2c3e67b2139c68cdbda87b59e8dd01b</code> which was updated to <code>b8b470bc59</code> recently. I don’t think Triton should and is downloading any CUDA version, but ships with <code>ptxas</code> and its own <code>libdevice.10.bc</code> library. I also did not see any CUDA download during my build and would not even know how it could install a full CUDA toolkit.</li>
</ol>
Pandas DataFrame causes Problems in CustomDataset

MikeTensor: <p>Within my <code>self.transforms</code>-Function in my CustomDataset I want to replace two columns of the original pandas DataFrame <code>dataframe</code> with the new pd.DF which are transformed (BoundingBoxes, Keypoints)</p>
<p>Unfortunately it doesn’t work within in my CustomDS (outside the CustomDS it does).</p>
<p>What I basically do is:<br>
<code>dataframe.bbox = resized_dataframe.bbox</code><br>
This works, but only if the shapes are the same (which they are and have to be, since my resize function accesses the <code>dataframe</code>.</p>
<p>That’s inside the resize function</p>
<pre><code class="lang-auto">def data_transform(img_arr, dataframe, resize, image_height_resize, image_width_resize, flip_probability):

    if resize:

        ### resize image and if needed: flipped
        transform_resize_img_bb_kyp = albumentations.Compose(
        [albumentations.Resize(height=image_height_resize, width=image_width_resize, always_apply=True),
         albumentations.HorizontalFlip(p=flip_probability)],
        bbox_params=albumentations.BboxParams(format='pascal_voc', label_fields=[]),
        keypoint_params=albumentations.KeypointParams(format="xy"))

        transformed_img_bb_kyp = transform_resize_img_bb_kyp(image=img_arr, bboxes=dataframe.bbox, keypoints=dataframe.forward_keypoint)
        bb = pd.DataFrame(transformed_img_bb_kyp["bboxes"])
        kp = pd.DataFrame(transformed_img_bb_kyp["keypoints"])
        print(f"While Transform: Shape resized Dataframe {dataframe.bbox.shape}")
        print(f"While Transform: Shape resized BoundingBox {bb.shape}")
        dataframe.bbox = bb
        dataframe.forward_keypoint = kp

        print("resized")

        return transformed_img_bb_kyp["image"], dataframe
</code></pre>
<p>The result:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d67192d44e3d03b72967952cd33f203b174e1084.png" alt="grafik" data-base62-sha1="uB3vyaw8UH0hj4Bu9DhmagOUT3e" width="505" height="157"></p>
<p>The ValueError can’t be right, since the <code>Shape resized BoundingBox</code> is a result / outcome of the <code>dataframe</code> <code>(9,4)</code><br>
It also says that the shape of the <code>Dataframe</code> itself is <code>(9, NOTHING)</code> which can’t be (and is not) true, because of the same reason as above.</p>
<p><strong>So what could be wrong here?</strong></p>
<p>I am even thinking about a workaround to just return the untouched <code>dataframe</code> and <code>resizedBoundingBoxes</code> and replace the values after the return of the <code>CustomDataset</code>, within my <code>train_function</code></p>

ptrblck: <p>Could you post a minimal and executable code snippet by adding (random) dataframes, which would reproduce the issue, please?</p>
How to create a tensor of size x based on a value (x) predicted by a network without breaking the computational graph

zabboud: <p>Hi All – Currently I’m trying to predict a kernel size through an auxiliary regression task which takes in features from a bottleneck of a CNN.</p>
<p>The regression task is predicting two values, kernel size and sigma - I am trying to generate a gaussian kernel based on that size and sigma - however, to generate the gaussian kernel, if I use torch.arange or torch.linspace it disconnects the predicted values from the computational graph. And if i still use a flag requires_grad on the torch.arange - it will be separate from the gradient from the predicted values - meaning i can’t backpropagate the gradients after computing the loss, therefore cannot train that portion of the network (I’ve visualized the graph through torchviz).</p>
<p>My question is - how can I generate a Gaussian kernel without having to disconnect the computation graph? How do i go about this problem? Any insight would be helpful! I’ve been at this problem for days and I can’t find a way out.</p>
<p>thank you!</p>
Trouble installing pytorch

Connor_Park: <p>I posted another topic under the wrong category. Trying to correct my mistake here.<br>
I am using the following command to install it from a local clone of the pytorch github.<br>
pip install -U ./ --user</p>
<p>I have get the following error after running for quite a long time.<br>
OS: Windows 10<br>
CUDA 12</p>
<p>Building Custom Rule D:/Programming/pytorch/caffe2/CMakeLists.txt<br>
– Building version 2.1.0a0+gitd1ec9a5<br>
cmake -GVisual Studio 16 2019 -Ax64 -Thost=x64 -DBUILD_CAFFE2=1 -DBUILD_ONNX_PYTHON=1 -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_GENERATOR=Visual Studio 16 2019 -DCMAKE_INCLUDE_PATH=D:\Programming\mkl\include -DCMAKE_INSTALL_PREFIX=D:\Programming\pytorch\torch -DCMAKE_PREFIX_PATH=C:\Python310\Lib\site-packages -DJAVA_HOME=C:\Program Files\Java\jdk-14.0.1 -DNUMPY_INCLUDE_DIR=C:\Users\Connor\AppData\Local\Temp\pip-build-env-tin23g2u\overlay\Lib\site-packages\numpy\core\include -DPYTHON_EXECUTABLE=C:\Python310\python.exe -DPYTHON_INCLUDE_DIR=C:\Python310\Include -DPYTHON_LIBRARY=C:\Python310/libs/python310.lib -DTORCH_BUILD_VERSION=2.1.0a0+gitd1ec9a5 -DUSE_NINJA=0 -DUSE_NUMPY=True D:\Programming\pytorch<br>
cmake --build . --target install --config Release – /p:CL_MPCount=24</p>
<p>If you have any questions or any suggestions on how to fix please let me know. I appreciate any and all assistance, thank you.</p>
The benchmark test for dynmao inductor runs stuck

fuaq: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/d/ad0c5274086613a5432e28a0acb18fba1c6b8cd0.png" data-download-href="https://discuss.pytorch.org/uploads/default/ad0c5274086613a5432e28a0acb18fba1c6b8cd0" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/d/ad0c5274086613a5432e28a0acb18fba1c6b8cd0.png" alt="image" data-base62-sha1="oGR08X19zmBHuMxlxOjgOf3Eu7m" width="690" height="48" data-dominant-color="3C3837"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1672×118 8.58 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
always stay here for a long time, no errors and no warnings.I just want to know what happend.</p>

ptrblck: <p>Cross-post from <a href="https://github.com/pytorch/pytorch/issues/95874">here</a>.<br>
Let’s please keep the discussion in one place to avoid re-debugging the same issues.</p>
Repeating tensor rows given lengths tensor

Ameen_Ali: <p>Hello!</p>
<p>Let’s say I have a length’s tensor (lengths_tensor) : len of shape [32 X count] for example :</p>
<p>tensor([1, 1, 1, 1, 3, 1, 1, 2, 1, 2, 3, 1, 2, 2, 2, 3, 4, 1, 3, 1, 2, 2, 2, 2, 2, 2, 1, 1, 2, 2, 2, 1], device=‘cuda:0’)</p>
<p>and another features tensor of shape [32 X Dimension]</p>
<p>what I want to do is for each entry (i) in the features tensor to repeat it lengths_tensor[i] times, so given the example above the first element in the feature tensor should not be repeated, and for the element in place 4 shall be repeated 3 times, so the new feature tensor should be of shape [60 x Dimension]<br>
Thanks!</p>

ptrblck: <aside class="quote no-group" data-username="Ameen_Ali" data-post="1" data-topic="173869">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ameen_ali/48/15528_2.png" class="avatar"> Ameen_Ali:</div>
<blockquote>
<p>so the new feature tensor should be of shape [60 x Dimension]</p>
</blockquote>
</aside>
<p>I’m unsure how you’ve calculated <code>60</code> elements, as the <code>sum</code> if your <code>lengths_tensor</code> is <code>57</code>.<br>
Assuming <code>57</code> is the expected shape, then <code>repeat_interleave</code> might work:</p>
<pre data-code-wrap="python"><code class="lang-python">a = torch.tensor([1, 1, 1, 1, 3, 1, 1, 2, 1, 2, 3, 1, 2, 2, 2, 3, 4, 1, 3, 1, 2, 2, 2, 2, 2, 2, 1, 1, 2, 2, 2, 1])

x = torch.arange(32)
y = x.repeat_interleave(a)
print(y)
# tensor([ 0,  1,  2,  3,  4,  4,  4,  5,  6,  7,  7,  8,  9,  9, 10, 10, 10, 11,
#         12, 12, 13, 13, 14, 14, 15, 15, 15, 16, 16, 16, 16, 17, 18, 18, 18, 19,
#         20, 20, 21, 21, 22, 22, 23, 23, 24, 24, 25, 25, 26, 27, 28, 28, 29, 29,
#         30, 30, 31])
</code></pre>
How to create (n,c,h,w) tensor with known values

ynjiun_wang: <p>Hi<br>
I would like to create a (n,c,h,w) self index tensor as below: (use (2,3,4,5) as example)</p>
<blockquote>
<p>x = torch.tensor(range(5))<br>
x = torch.stack((x,x,x,x), 0)<br>
x = torch.stack((x,x,x),0)<br>
x = torch.stack((x,x),0)</p>
</blockquote>
<p>But how to generalize to (n,c,h,w)?<br>
no problem with first step:<br>
x = torch.tensor(range(w))<br>
but how to do stack of x with h time?<br>
or is there other way to create such tensor without looping?</p>
<p>Thanks for your help.</p>

tom: <p>If you want the tensor to have one column repeated all over, you can use <code>expand</code>:</p>
<p><code>x = torch.arange(5)[None, None, None, :].expand(2, 3, 4, 5)</code></p>
<p>or, if you prefer,</p>
<p><code>x = torch.arange(5).view(1, 1, 1, -1).expand(2, 3, 4, 5)</code></p>
<p>Both create singleton dimensions where you want the expansion to go to (with indexing or view) and then expand (with stride 0) to the desired size (similar to what is done in broadcasting).<br>
If you want the exact same result (without strides) as the stacking, you can add <code>.contiguous()</code> at the end.</p>
<p>Best regards</p>
<p>Thomas</p>
If I found a typo, can I create a PR to fix it?

111357: <p>As the title says, if not who should I report it to?</p>

ptrblck: <p>Yes, you can contribute to PyTorch by sending a PR for fixes.<br>
Take a look at <a href="https://pytorch.org/docs/stable/community/contribution_guide.html">this Contribution Guide</a> to get an overview and let us know if you get stuck.</p>
Cosine similarity regularization

Paulo_Pirozelli: <p>I’m trying to add a regularization term based on the weights of two networks with the same architecture. The first part of the code is something as the following:</p>
<pre><code class="lang-auto">logits1 = model1(data)

logits2 = model2(data)

loss_fn = nn.CrossEntropyLoss()

loss1 = loss_fn(logits1, target)

loss2 = loss_fn(logits2, target)

loss = loss1 + loss2
</code></pre>
<p>After summing the two losses, I want to add a regularization term, based on the cosine similarity of the parameters in the two networks. The idea is to force the networks to have different weights; i.e., to be orthogonal. The formula for this similarity is the same as in torch.nn.CosineSimilarity().</p>
<p>Initially, I thought of storing the parameters in two lists, calculating the cosine similarity, and them adding this to the combined loss. Thus, something like:</p>
<pre><code class="lang-auto">cos = nn.CosineSimilarity(dim=0)

regularization = beta * (cos(params1, params2) ** 2)

loss += regularization
</code></pre>
<p>However, I have the impression that just adding the regularization to the loss wouldn’t actually affect the gradients, because no computational graph is created. Am I right? In this case, is there any suggestion on how to implement that?</p>

tom: <p>Why would the graph not be created? The parameters have the <code>requires_grad</code> attribute set, so this would be backpropagated into.<br>
One thing to look for is what a good value of <code>beta</code> is. If you don’t have a theory about what should be the best ratio, you could try a range that includes a value to make the loss components approximate equal.</p>
<p>Best regards</p>
<p>Thomas</p>

Paulo_Pirozelli: <p>Thank you for the answer! I was afraid this wouldn’t work.</p>
Heterogenous data elements

mv000: <p>Hello experts,<br>
I have some data that is of differing dimension, that I will create a custom model for.  That is, when I load my data in pandas it looks like</p>
<pre><code class="lang-auto">df=pd.read_csv('data.csv', usecols= ['scalar_feature','vector_feature','matrix_feature','target'])
df['vector_feature']=df['vector_feature'].apply(eval).apply(np.array)
df['matrix_feature']=df['matrix_feature'].apply(eval).apply(np.array)

df.head()
scalar_feature 	vector_feature 	matrix_feature	target
1.2 	[1,3,5] 	[[1,3,5],[2,1,1] ,[2,1,6]]	1
2.1 	[2,1,3] 	[[3,2,9],[2,2,1] ,[1,0,3]]	4
1.3 	[5,2,1] 	[[2,6,5],[2,2,3] ,[7,0,3]]	2
2.3 	[6,1,1] 	[[1,5,3],[2,4,3] ,[4,1,8]]	3

</code></pre>
<p>I can successfully create a dataloader from this dataset, but when I try to run my training loop, I get an error:</p>
<pre><code class="lang-auto">class ClassifierDataset(Dataset):
    def __init__(self,X_data,y_data):
        self.X_data=X_data
        self.y_data=y_data
   def __getitem__(self, index):
        return self.X_data[index],self.y_data[index]
   def __len__(self):
        return len(self.X_data)

train_dataset=ClassifierDataset(X_train,y_train)

train_loader=DataLoader(dataset=train_dataset, batch_size=1)

for batch_idx, (data, target) in enumerate(train_loader):
    etc...
</code></pre>
<p>TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found object</p>
<p>I have converted the objects already to numpy arrays, and so I am not sure what else I would need to do.</p>
<p>Any advice on how to achieve this?</p>
<p>Thank you very much!</p>
Distributed inference

oracal: <p>Hello, I wanna implement distributed inference for large models across multiple GPUs in single machine. However , I found that  <em><strong>torch.distributed</strong></em> APIs are mainly designed for training rather than inference. In other words, it is not convenient to use these APIs to deploy distributed inference due to some issues such as redundant gradients sync …<br>
Could you give me some suggestions about how to implement distributed inference in Pytorch? Thanks so much</p>
Can't load two C++ extensions in same module

Wesley_Neill: <p>The problem is in the title. On the C++ side I’ve developed two functionalities and registered them via:</p>
<pre><code class="lang-auto">static auto registry = c10::RegisterOperators("rays::ray_cast", &amp;ray_cast)
	.op("rays::mesh_to_heightmap", &amp;mesh_to_heightmap);
</code></pre>
<p>and on the python side, I’ve tried to load these extensions like so:</p>
<pre><code class="lang-auto">cpp_extension.load(name="mesh_to_heightmap", sources=[os.path.join(file_dir, "rays.cpp")],
                   is_python_module=False, build_directory=build_directory, extra_cflags=["/Zi"], extra_ldflags=["/DEBUG"])
cpp_extension.load(name="ray_cast", sources=[os.path.join(file_dir, "rays.cpp")],
                   is_python_module=False, build_directory=build_directory, extra_cflags=["/Zi"], extra_ldflags=["/DEBUG"])
</code></pre>
<p>However, this fails so long as both load statements are present, with the following:</p>
<pre><code class="lang-auto">Tests/models/physical/test_mesh.py:None (Tests/models/physical/test_mesh.py)
models\physical\test_mesh.py:4: in &lt;module&gt;
    from awes.torch_extensions import rays
..\awes\torch_extensions\rays.py:20: in &lt;module&gt;
    cpp_extension.load(name="ray_cast", sources=[os.path.join(file_dir, "rays.cpp")],
C:\Users\wneill\Miniconda3\envs\awes\lib\site-packages\torch\utils\cpp_extension.py:1080: in load
    return _jit_compile(
C:\Users\wneill\Miniconda3\envs\awes\lib\site-packages\torch\utils\cpp_extension.py:1318: in _jit_compile
    return _import_module_from_library(name, build_directory, is_python_module)
C:\Users\wneill\Miniconda3\envs\awes\lib\site-packages\torch\utils\cpp_extension.py:1706: in _import_module_from_library
    torch.ops.load_library(filepath)
C:\Users\wneill\Miniconda3\envs\awes\lib\site-packages\torch\_ops.py:104: in load_library
    ctypes.CDLL(path)
C:\Users\wneill\Miniconda3\envs\awes\lib\ctypes\__init__.py:373: in __init__
    self._handle = _dlopen(self._name, mode)
E   OSError: [WinError 1114] A dynamic link library (DLL) initialization routine failed
</code></pre>
<p>And sometimes it also fails with a message about “destructor implicitly defined as deleted”. On the C++ side one function calls the other, so I know that they are both working fine.</p>
How to implement a trainable staircase activation function (with many case distinctions)

KiwiPower: <p>I am trying to learn writing an activation functions that makes lots of internal case distinctions. Technically, the function picks an index <code>i</code> based on the input <code>x</code> and returns the value <code>a[i]</code>, where <code>a</code> is a trainable tensor.</p>
<p>For example, I want to implement an activation <code>Staircase</code> with</p>
<pre><code class="lang-auto">Staircase(x) = 0.  # when x &lt; 0.0
Staircase(x) = a[0]  # when 0.0 &lt;= x &lt; 0.1
Staircase(x) = a[1]  # when 0.1 &lt;= x &lt; 0.2
Staircase(x) = a[2]  # when 0.2 &lt;= x &lt; 0.3
...
Staircase(x) = a[8]  # when 0.8 &lt;= x &lt; 0.9
Staircase(x) = a[9]  # when 0.9 &lt;= x &lt; 1
Staircase(x) = 1.  # when 1 &lt;= x 
</code></pre>
<p>Regrettably, I don’t know how to implement this. I believe understanding this would help me write more complicated functions. I have tried the following to implement a staircase function (explanation below).</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

class Staircase(nn.Module):
    def __init__(self, number_of_steps ):
        super(Staircase, self).__init__()
        self.number_of_steps
        self.a = nn.Parameter( torch.rand( number_of_steps )) 
        
    # this raises an error when x is a tensor
    def forward(self, x):
        i = torch.floor( x / self.number_of_steps )
        if i &lt; 0 : return 0
        if i &gt;= self.number_of_steps : return 1.
        return self.a[i]
</code></pre>
<p>The intention is clear. What could be a way to code this in a trainable manner?</p>

tom: <p>I take it that <code>a</code> is a 1-d vector?<br>
Is it monotonically increasing between 0 and 1?</p>
<p>You could write a second function that interpolates between the levels (maybe at the mid-points 0.5, …?) and then use the goldie <code>y = (y_step - y_interp).detach() + y_interp</code>. This causes y to behave like <code>y_step</code> in terms of value and <code>y_interp</code> in terms of gradient.</p>
<p>Best regards</p>
<p>Thomas</p>
Trouble building and installing pytorch

Connor_Park: <p>I am trying to build pytorch for use with CUDA 12.<br>
While attempting the command: python setup.py install I get 5 instances of the following error:<br>
fatal error C1189: <span class="hashtag">#error:</span>  This file was generated by an older version of protoc which is (compil<br>
ing source file D:\Programming\pytorchSetup\build\caffe2\proto\caffe2.pb.cc) [D:\Programming\pytorchSetup\build\caffe2\proto\Caffe2_PROTO.vcxproj]</p>
<p>I then attempted installation using the pip install {path to pytorch local} command which ran for a long time then failed, this is the end of the log:<br>
[5860/6367] Building CUDA object caffe2\CMakeFiles\cuda_vectorized_test.dir__\aten\src\ATen\test\cuda_vectorized_test.cu.obj<br>
cuda_vectorized_test.cu</p>
<pre><code>  tmpxft_0000837c_00000000-7_cuda_vectorized_test.cudafe1.cpp

  [5861/6367] Building CUDA object caffe2\CMakeFiles\torch_cuda.dir\__\aten\src\ATen\native\transformers\cuda\flash_attn\fmha_bwd_hdim64.cu.obj
  fmha_bwd_hdim64.cu

  tmpxft_000064a0_00000000-7_fmha_bwd_hdim64.cudafe1.cpp

  [5862/6367] Building CUDA object caffe2\CMakeFiles\torch_cuda.dir\__\aten\src\ATen\native\cuda\Unique.cu.obj
  Unique.cu

  tmpxft_0000755c_00000000-7_Unique.cudafe1.cpp

  ninja: build stopped: subcommand failed.
  -- Building version 2.0.0a0+git0765dbc
  cmake -GNinja -DBUILD_PYTHON=True -DBUILD_SPLIT_CUDA=1 -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_INCLUDE_PATH=D:\Programming\mkl\include -DCMAKE_INSTALL_PREFIX=D:\Programming\pytorch\torch -DCMAKE_PREFIX_PATH=C:\Python310\Lib\site-packages -DJAVA_HOME=C:\Program Files\Java\jdk-14.0.1 -DNUMPY_INCLUDE_DIR=C:\Users\Connor\AppData\Local\Temp\pip-build-env-iut6kxu7\overlay\Lib\site-packages\numpy\core\include -DPYTHON_EXECUTABLE=C:\Python310\python.exe -DPYTHON_INCLUDE_DIR=C:\Python310\Include -DPYTHON_LIBRARY=C:\Python310/libs/python310.lib -DTORCH_BUILD_VERSION=2.0.0a0+git0765dbc -DUSE_NUMPY=True D:\Programming\pytorch
  cmake --build . --target install --config Release
  [end of output]
</code></pre>
<p>Thanks for your help!</p>

Connor_Park: <p>Just to add onto this, when I tried to build and install pytorch using pip install (local pytorch repo), if I used max jobs = 1. It got the following error.<br>
fatal error C1189: <span class="hashtag">#error:</span>  Macro clash with min and max – define NOMINMAX when compiling your program on Windows</p>

Connor_Park: <p>I have realized this is definitely not in the right category, not sure if I can move it though. Apologies.</p>
Can I declare activation function in class constructor?

DongkyuLee: <p>Hi</p>
<p>Can I declare activation function in class(Module) constructor?</p>
<p>There are example using torch::nn::sequential and using torch::functional::(kinds of activation functions) in forward function directly, but I can’t find example about activation function declaration in class.</p>
<p>Please let me know how to declare activation function in class constructor!<br>
(I’d like to see some example)</p>
<p>Thanks</p>

ptrblck: <p>You should be able to register <code>torch::nn::ReLU()</code> as any other module (e.g. <code>torch::nn::Linear</code>) in your custom class via <code>register_module</code>.</p>
Reproducibility, DataLoader: shuffle=True, using seeds

MikeTensor: <p>I am concerned about my Reproducibility.<br>
Is there a way to use <code>seed</code>s and <code>shuffle=True</code> and keep Reproducibility?</p>
<p>Let’s say I would use:</p>
<pre><code class="lang-auto">def set_seeds(seed: int=42):
    """Sets random sets for torch operations.

    Args:
        seed (int, optional): Random seed to set. Defaults to 42.
    """
    # Set the seed for general torch operations
    torch.manual_seed(seed)
    # Set the seed for CUDA torch operations (ones that happen on the GPU)
    torch.cuda.manual_seed(seed)
</code></pre>
<p>togehter with the DataLoader:</p>
<pre><code class="lang-auto">train_dataloader = DataLoader(dataset=train_data,
                               collate_fn=None,
                               batch_size=None, # how many samples per batch?
                               num_workers=1, # how many subprocesses to use for data loading? (higher = more)
                               shuffle=True,
                               pin_memory=True)
</code></pre>
<p><em>Prob. I will also splitt the Data (train, val).</em></p>
<p>How do I get the order of images keep being the same?</p>
<p>Since my problem with the DataLoader (<a href="https://discuss.pytorch.org/t/wrong-different-image-shapes-after-dataloader-bug-in-dl/173828" class="inline-onebox">Wrong/different image shapes after DataLoader; bug in DL?</a>) it seems that I have to use shuffle.</p>

ptrblck: <p>Seeding the code before iterating the <code>DataLoader</code> or rather before the iterator is created should work for a simple <code>Dataset</code> as seen here:</p>
<pre><code class="lang-python">dataset = TensorDataset(torch.arange(40).view(-1, 1))
loader = DataLoader(dataset, num_workers=2, shuffle=True, batch_size=10)

torch.manual_seed(2809)
for data in loader:
    print(data)
    
torch.manual_seed(2809)
for data in loader:
    print(data)
    
torch.manual_seed(2809)
iter_loader = iter(loader)
while True:
    try:
        data = next(iter_loader)
        print(data)
    except StopIteration:
        print("Done")
        break
</code></pre>
<p>In case you are using 3rd party libraries inside your <code>Dataset.__getitem__</code> you might need to seed these in the <code>worker_init_fn</code> if you are using multiple workers.</p>
Why no up limit of tensor dimensions for nn.Linear?

shengchunnan: <p>When I run the following code with input of extra dimensions, it reports error:<br>
<em>RuntimeError: Expected 4-dimensional input for 4-dimensional weight [64, 16, 3, 3], but got 5-dimensional input of size [8, 7, 16, 80, 80] instead</em></p>
<pre data-code-wrap="py"><code class="lang-plaintext">import torch
import torch.nn as nn

class ConvInputTest (nn.Module):

    def __init__(self):
        super(ConvInputTest, self).__init__()

        self.conv = nn.Conv2d(
            in_channels=16,
            out_channels=64,
            kernel_size=3,
            padding=1,
            stride=1)
        
    def forward(self, x):

        return self.conv(x)
    
net = ConvInputTest()
x = torch.randn(8, 7, 16, 80, 80)
y = net(x)
</code></pre>
<p><strong>However</strong>, when I test a similar case using nn.Linear, the code can run smoothly:</p>
<pre data-code-wrap="py"><code class="lang-plaintext">import torch
import torch.nn as nn

class LinearInputTest (nn.Module):

    def __init__(self):
        super(LinearInputTest, self).__init__()

        self.linear = nn.Linear(
            in_features=48,
            out_features=96)
        
    def forward(self, x):

        return self.linear(x)
    

net = LinearInputTest()
x = torch.randn(8, 7, 16, 80, 48)
y = net(x)
print(y.size())
</code></pre>
<p><strong>Why does nn.Conv2d restrict number of dimensions to 4 (or nn.Conv3d to 5, or nn.Conv1d to 3), but nn.Linear does  not have such limitations?</strong></p>

J_Johnson: <p>When you run a tensor through a Linear layer, assuming the final dim matches the input size, it just matmuls the last two dims and treats the other dims like separate batches.</p>
Image Captioning using YOLO

ASK: <p>Hello ,<br>
I have been trying to extract the bounding box coordinates of my Image using a pre-trained YOLO model as you can find in the below code snippet. Can you please help how could I extract the bounding box coordinates from the YOLO code snippet and what changes I have to made in my code of DecoderRNN to concatenate my caption embedding with the yolo coordinates and then Fed it to the LSTM.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/9/79b259d0bc68d47096ca6ccb8aef60dfa28a1b06.png" data-download-href="https://discuss.pytorch.org/uploads/default/79b259d0bc68d47096ca6ccb8aef60dfa28a1b06" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/9/79b259d0bc68d47096ca6ccb8aef60dfa28a1b06.png" alt="image" data-base62-sha1="hmzT6HlJRkzjNNI3D24HyPJlfgy" width="583" height="500" data-dominant-color="F9F8F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">784×672 18.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ASK: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/b/abb8c40dea5830e6e561b5f850c4daf539b9432e.png" data-download-href="https://discuss.pytorch.org/uploads/default/abb8c40dea5830e6e561b5f850c4daf539b9432e" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/b/abb8c40dea5830e6e561b5f850c4daf539b9432e.png" alt="image" data-base62-sha1="ov7voHSeC0R4IG4RrcliorxT9xk" width="690" height="141" data-dominant-color="F9F8FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">887×182 8.23 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
This is the YOLO code snippet</p>
How to pass image of any size to Pytorch ViT pretrained model?

stanleygeorge: <pre><code class="lang-auto">import torch
from torchvision import models

model = models.vit_b_32(pretrained=True ,image_size=320)
model.eval()
</code></pre>
<p>The above piece of code is failing at Line 3 with the below error:</p>
<pre><code class="lang-auto">ValueError: The parameter 'image_size' expected value 224 but got 320 instead. 
</code></pre>
<p>So does Pytorch’s pre-trained Vision Transformer model take only a fixed shape input image size unlike pre-trained ResNet’s which are flexible with the image size ?</p>
<p>I am shying a bit from downsizing my image as I am trying to perform Crack-detection on some metal surfaces. After downsizing to 224, the crack pixels become far too small which I believe may affect my model’s performance. When I train my model on ResNet’s, I get the optimal performance for image shapes &gt; 400px</p>

J_Johnson: <p>If pretrained, yes. 224 is the defacto size. If you do not need pretrained, you’ll need to specify the image_size and patch_size arguments.</p>
<p>Now, you might be able to get away with pretrained if you change out some layers. And then redefine the image_size, post mortem. Note, you’ll want to keep the patch_size unchanged.</p>
<p>Here is the model code:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/vision/blob/main/torchvision/models/vision_transformer.py">
  <header class="source">

      <a href="https://github.com/pytorch/vision/blob/main/torchvision/models/vision_transformer.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/main/torchvision/models/vision_transformer.py" target="_blank" rel="noopener nofollow ugc">pytorch/vision/blob/main/torchvision/models/vision_transformer.py</a></h4>


      <pre><code class="lang-py">import math
from collections import OrderedDict
from functools import partial
from typing import Any, Callable, Dict, List, NamedTuple, Optional

import torch
import torch.nn as nn

from ..ops.misc import Conv2dNormActivation, MLP
from ..transforms._presets import ImageClassification, InterpolationMode
from ..utils import _log_api_usage_once
from ._api import register_model, Weights, WeightsEnum
from ._meta import _IMAGENET_CATEGORIES
from ._utils import _ovewrite_named_param, handle_legacy_interface


__all__ = [
    "VisionTransformer",
    "ViT_B_16_Weights",
    "ViT_B_32_Weights",
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/vision/blob/main/torchvision/models/vision_transformer.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Stacking variable length chunks of a tensor (vectorized solution)

Sobir_Bobiev: <p>Is there a function that chunks a tensor along a dimension with given chunk sizes and stacks the chunks into a single tensor (also pads the chunks since they can be variable sized)?</p>
<pre><code class="lang-auto">A = torch.randn(10, 512, 768)
sizes = torch.tensor([1, 3, 6])
assert sizes.sum() == A.size(0)

out = function(A, sizes, dim=0)

assert out.shape == (len(sizes), sizes.max(), 512, 768)
</code></pre>

ptrblck: <p><code>torch.split</code> and <code>pad_sequence</code> should work:</p>
<pre><code class="lang-python">A = torch.randn(10, 512, 768)
sizes = torch.tensor([1, 3, 6])
assert sizes.sum() == A.size(0)

out = torch.split(A, sizes.tolist(), 0)
out = torch.nn.utils.rnn.pad_sequence(out, batch_first=True)
assert out.shape == (len(sizes), sizes.max(), 512, 768)
</code></pre>
Torchrun crashes when creating checkpoint, running with 2 GPUs

fabiansc: <p>I’m trying to run an mmsegmentation based script. It runs smooth well when I run it on 1 GPU, however when I try to use Torchrun (or torch.distributed.launch) and run it on my two GPUs, it runs well the first 1000 epochs, but then it crashes when it’s supposed to create a checkpoint.</p>
<p>This is the error message that I’m getting:</p>
<h1>
<a name="torchdistributedelasticmultiprocessingerrorschildfailederror-1" class="anchor" href="#torchdistributedelasticmultiprocessingerrorschildfailederror-1"></a>torch.distributed.elastic.multiprocessing.errors.ChildFailedError:</h1>
<h2>
<a name="trainpy-failed-2" class="anchor" href="#trainpy-failed-2"></a>train.py FAILED</h2>
<h2>
<a name="failures-no_other_failures-3" class="anchor" href="#failures-no_other_failures-3"></a>Failures:<br>
&lt;NO_OTHER_FAILURES&gt;</h2>
<h1>
<a name="root-cause-first-observed-failure-0-time-2023-03-01_082041-host-fabiandual2-rank-0-local_rank-0-exitcode-9-pid-67-error_file-na-traceback-signal-9-sigkill-received-by-pid-67-4" class="anchor" href="#root-cause-first-observed-failure-0-time-2023-03-01_082041-host-fabiandual2-rank-0-local_rank-0-exitcode-9-pid-67-error_file-na-traceback-signal-9-sigkill-received-by-pid-67-4"></a>Root Cause (first observed failure):<br>
[0]:<br>
time      : 2023-03-01_08:20:41<br>
host      : FabianDual2.<br>
rank      : 0 (local_rank: 0)<br>
exitcode  : -9 (pid: 67)<br>
error_file: &lt;N/A&gt;<br>
traceback : Signal 9 (SIGKILL) received by PID 67</h1>
<p>/home/fabian/miniconda3/envs/mask2form/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 20 leaked semaphore objects to clean up at shutdown<br>
warnings.warn('resource_tracker: There appear to be %d ’</p>
<p>every help would be greatly appreciated.</p>

ptrblck: <p>Do you see any error messages in the stacktrace? E.g. the OS would kill the process if it’s running out of host memory to avoid a system crash so could you monitor your RAM and check if this could be the case?</p>
Metrics available during training?

AlphaRomeo: <p>when using callbacks, I can fill it with some metrics such as filename=‘{epoch}-{val_loss:.2f}-{other_metric:.2f}’</p>
<p>Is there documentation that lists all the metrics available?</p>

ptrblck: <p>PyTorch doesn’t provide any methods to compute metrics (you could of course calculate them manually), but you could use e.g. <a href="https://torchmetrics.readthedocs.io/en/stable/">TorchMetrics</a> or other libraries providing these methods.</p>
Frequencey of evaluation steps affecting PyTorch model accuracy

vishalthengane: <p>Dear PyTorch community,</p>
<p>I observed that the frequency of evaluation steps is affecting the final accuracy of my PyTorch model. When I evaluate the model after each epoch of training (this is with a dummy <a href="https://gist.github.com/vgthengane/418595884e56d76bfe3519f2f6e36bfa" rel="noopener nofollow ugc">example</a>, where I am only training the model for an epoch), I get an accuracy of 0.7551 and a loss of 1.0198748077556585.<br>
Like below -</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
    train_one_epoch()
    evaluate()
</code></pre>
<p>However, when I evaluate the model before training, followed by training and then evaluation again, I get an accuracy of 0.7656 and a loss of 0.9733265159995692.</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
    evaluate()
    train_one_epoch()
    evaluate()
</code></pre>
<p>Similarly, if I evaluate the model twice before training, followed by training and then evaluation again, I get an accuracy of 0.7631 and a loss of 0.9899978056834762.</p>
<p>I would like to know why this is happening and how to ensure consistent results regardless of the order of training and evaluation steps. I would appreciate any suggestions or insights from the PyTorch community.</p>
<p>Here is the full code - <a href="https://gist.github.com/vgthengane/418595884e56d76bfe3519f2f6e36bfa" rel="noopener nofollow ugc">example.py</a></p>

ptrblck: <p>The small differences in accuracy might be caused by a different pseudorandom number generator behavior, which could be caused by the <code>BaseDataLoaderIter</code> as seen in <a href="https://discuss.pytorch.org/t/does-iterating-over-unshuffled-dataloader-change-random-state/173137/2">this post</a>.<br>
Could you check if re-seeding the code directly before calling <code>train_one_epoch</code> would yield the same performance?</p>

vishalthengane: <p>Re-seeding before <code>train_one_epoch</code> gives slightly different numbers, for</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
    evaluate()
    seed_everything(seed)
    train_one_epoch()
    evaluate()
</code></pre>
<p>i got <code>acc=0.7478</code> and <code>loss=1.0340169899782556</code> before it was <code>acc=0.7656</code> and <code>loss=0.9733265159995692</code></p>

ptrblck: <p>This would point to a non-deterministic execution of the actual training loop. Could you remove the <code>evaluate()</code> calls and check if the <code>train_one_epoch</code> is giving the same results for the same seeds?</p>

vishalthengane: <p>Yes, without the <code>evaluate()</code>, <code>train_one_epoch()</code> gives the same results for the same seed.</p>
Question about training the threshold for k-means clustering

smartbarbarian: <p>I need to perform k-means clustering on a set of points. To initialize the centroids, I use a method where the first centroid covers the most points within a certain distance, the second centroid covers the most points except those already covered by the first centroid, and so on. I would like to make the distance parameter trainable. How can I achieve this? Thank you in advance to anyone who takes the time to answer my question. Your help is greatly appreciated!</p>

smartbarbarian: <p>My idea is to use sigmoid as a differentiable function to clip values at a certain threshold, allowing for differentiable gradients. Is it feasible?</p>