akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Custom loss with logic operation can not backward

Ting_En_Hsu: <p>this is my Custom Loss, but the backward will not work.</p>
<pre><code class="lang-auto">def OthelloLoss_(pred, label, th):
    
    pred = torch.where(pred &gt; th, True, False)
    label = torch.where(label == 1.0, True, False)

    #ex: p:110 g:011
    pred_gt_or = torch.bitwise_or(pred, label)                          #pred+gt where 1                     =&gt;110 or 011 = 111
    denominator = torch.sum(torch.where(pred_gt_or == True, 1, 0),1)    #total pred+gt where 1 position      =&gt;p or q:111 =&gt; 3

    pred_gt_xor = torch.bitwise_xor(pred, label)                        #not correct position                =&gt;110 xor 011 = 101
    molecular =  torch.sum(torch.where(pred_gt_xor == True, 1, 0),1)    #total pred not correct num          =&gt;p xor q:101 =&gt; 2 
    position_loss = molecular / denominator                             #position loss = (not correct num) / (total postion num)
    position_loss = torch.nan_to_num(position_loss, nan=0.0)
    position_batch_loss =  torch.div(torch.sum(position_loss), len(position_loss))

    wrong_and_xor = torch.bitwise_and(pred,pred_gt_xor)                     #wrong postion                       =&gt;p and(p xor q) =  110 and 101 = 100
    wrong_num =  torch.sum(torch.where(wrong_and_xor == True, 1, 0),1)      #wrong num                           =&gt; p and(p xor q) = 100 =&gt; 1  
    wrong_penalty_loss = 0.5 * wrong_num
    wrong_penalty_batch_loss = torch.div(torch.sum(wrong_penalty_loss), len(wrong_penalty_loss))

    total_loss = position_batch_loss + wrong_penalty_batch_loss
    
    
    return total_loss
</code></pre>
<p>every epoch loss:<br>
Epoch: 1 | train_loss: 12.5797 | train_acc: 4.45% | val_loss: 12.583 | val_acc: 4.45%<br>
Epoch: 2 | train_loss: 12.5797 | train_acc: 4.45% | val_loss: 12.583 | val_acc: 4.45%<br>
Epoch: 3 | train_loss: 12.5797 | train_acc: 4.45% | val_loss: 12.583 | val_acc: 4.45%<br>
Epoch: 4 | train_loss: 12.5797 | train_acc: 4.45% | val_loss: 12.583 | val_acc: 4.45%<br>
…</p>
Non-deterministic behaviour of Huggingface Model

fdg: <p>I am currently working with a model from Huggingface (Donut) and have trained the model on a custom dataset.</p>
<p>During evaluation, I noticed, that the predictions are non-deterministic and I do not understand why that is the case.<br>
In my evaluation loop, I set model.eval() and I double checked that model.training is False during the test loop.</p>
<p>When trying to set <a href="https://pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html" rel="noopener nofollow ugc"><code>torch.use_deterministic_algorithms</code></a> to True, I get an error (CuBLAS). However, I do not understand why I would have to set this setting in the first place, as I assumed the model is deterministic in evaluation mode.</p>
<p>I would appreciate any input and am happy to provide more information as well (this is my first post  and I am unfamiliar as to what should be included in a question)</p>

ptrblck: <aside class="quote no-group" data-username="fdg" data-post="1" data-topic="179936">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/cdc98d/48.png" class="avatar"> fdg:</div>
<blockquote>
<p>However, I do not understand why I would have to set this setting in the first place, as I assumed the model is deterministic in evaluation mode.</p>
</blockquote>
</aside>
<p>There are multiple sources of non-determinism where the model’s behavior could be one of them (calling <code>model.eval()</code> would e.g. disable dropout layers and thus eliminate this source of randomness) while also algorithms themselves can create non-deterministic outputs (e.g. if the order of operations isn’t strictly defined). Calling <code>torch.use_deterministic_algorithms</code> fixes this source of randomness and non-deterministic behavior.</p>
Pytorch giving different output in single prediction and batch prediction

Rohan_Malji: <p>I have transformer model which I have quantized for android mobile when I am running a single message for prediction I am getting [7.80595, -3.5064116, -1.6516012, -0.81042784, -5.521521, -3.8056054, -5.5388055] this tensor output but when this same message is included in batch of prediction [<strong>7.798512, -3.5149465, -1.6561824, -0.80104315, -5.504274, -3.8043664, -5.5505915,</strong><br>
4.413419, -4.5582056, 1.9779913, 1.3730897, -6.670629, -2.7525914, -7.0175514, 7.798512, -3.5149465, -1.6561824, -0.80104315, -5.504274, -3.8043664, -5.5505915, 7.798512, -3.5149465, -1.6561824, -0.80104315, -5.504274, -3.8043664, -5.5505915, 7.798512, -3.5149465, -1.6561824, -0.80104315, -5.504274, -3.8043664, -5.5505915, 7.798512, -3.5149465, -1.6561824, -0.80104315, -5.504274, -3.8043664, -5.5505915, 7.798512, -3.5149465, -1.6561824, -0.80104315, -5.504274, -3.8043664, -5.5505915, 7.798512, -3.5149465, -1.6561824, -0.80104315, -5.504274, -3.8043664, -5.5505915, 7.798512, -3.5149465, -1.6561824, -0.80104315, -5.504274, -3.8043664, -5.5505915]</p>
<p>which is actually different and I am unable to find the reason behind it. can some one help me in this.</p>
Why do we need to call `torch.no_grad()` for validation loop even after calling `loss.backward()`?

Cyber_punk: <p>The typical Pytorch training loop is:</p>
<pre><code class="lang-auto">for epoch in range(n_epochs):

	# Training
	for data in train_dataloader:
		input, targets = data
		optimizer.zero_grad()
		output = model(input)
		train_loss = criterion(output, targets)
		train_loss.backward()
		optimizer.step()

	# Validation
	with torch.no_grad():
		for input, targets in val_dataloader:
            output = model(input)
            val_loss = criterion(output, targets)
</code></pre>
<p>Since the gradients are computed by <code>train_loss.backward()</code>, why do we still need to use <code>with torch.no_grad()</code> for the validation part? Isn’t the purpose of <a href="https://pytorch.org/docs/stable/generated/torch.no_grad.html" rel="noopener nofollow ugc">torch.no_grad()</a> that “<strong>Disabling gradient calculation</strong>”?</p>

ptrblck: <p>Yes, <code>no_grad</code> will disable the gradient calculation and will reduce the memory usage by deleting intermediate activations directly, which would be needed for the backward pass otherwise.<br>
Since you are never calling <code>backward()</code> during the validation phase <code>no_grad()</code> is an optional optimization.</p>
QAT training loss decreasing speed is very slow

bigtree: <p>I am working on the QAT training for my model.<br>
I compared the two training cases, i.e., training an FP32 model vs QAT based on FP32 mdoel.<br>
What I observed is the time for each epoch during training is similar.<br>
But when comparing the loss decreasing, I found the QAT is extremely slow.<br>
For example to achieve 1.5 (just an example) from 5.0 the FP32 training just needs 50 epoch. But for the QAT from 5.0 to 3.5, it has taken 6k epoch, and seems the loss decreasing is getting slower and slower.</p>
<p>BTW, all the learning rate, optimizer are the same for these two training.</p>
<p>Is there expected?<br>
Thanks.</p>

tom: <p>This might be expected because the gradient will be for a “quantized approximation” of the current parameters rather than the parameters.<br>
But so one thing you could do is first train a bit in FP32 and then do some QAT training.<br>
You should not really lose much by looking in fp32 for where to start the QAT.</p>
<p>Best regards</p>
<p>Thomas</p>

bigtree: <p>Thanks, <a class="mention" href="/u/tom">@tom</a>.<br>
Yes. If there is no other try, I think that will be the only way to accelerate my QAT training.</p>

yyd199948: <p>How did you solve it？I meet the same problem，float32 model training 1 epoch the loss can down to -9,but the qat model traing 20 epoch only down to -4.</p>
How to implement this model using BiLSTM with attention?

cjjjy: <p>I have a dataset,the each sample in dataset is &lt;Question,Document,Answer&gt; ,Answer mayebe in document or not, Now,I want implement  model using BiLSTM with attention.</p>
<p>The model have two input layers ,the one layer is question layer and the other layer is documnet layer。</p>
<p>the model output layer should predict whether each word in the Document is the start and end of the answer or not</p>
<p>For example, given a Question as input:<br>
“who discovered neptune the planet?”</p>
<p>Given Document as input :<br>
“With a prediction by Urbain Le Verrier , telescopic observations confirming the existence of a major planet were made on the night of September 23–24, 1846”</p>
<p>The Answer for Question is “Urbain Le Verrier”</p>
<p>the output layer predict word “with” is not start word or end of answer<br>
the output layer predict word “a” is not start word or end of answer<br>
the output layer predict word “prediction” is not start word or end of answer<br>
the output layer predict word “by” is not start word or end of answer<br>
the output layer predict word “Urbain” is  start word of answer<br>
the output layer predict word “Le” is not start word or end of answer<br>
the output layer predict word “Verrier” is  end word of answer</p>
<p>How to implement this model using BiLSTM with attention?</p>
List all tensors with their names and size

NightMachinery: <p>To profile the memory usage, I want to list all tensors with their name and size.</p>
<p>I have a function that can show all tensors with their size:</p>
<pre><code class="lang-auto">def pretty_size(size):
    """Pretty prints a torch.Size object"""
    assert isinstance(size, torch.Size)
    return " x ".join(map(str, size))


def dump_tensors(gpu_only=True):
    """Prints a list of the Tensors being tracked by the garbage collector."""
    import gc

    total_size = 0
    for obj in gc.get_objects():
        try:
            if torch.is_tensor(obj):
                if not gpu_only or obj.is_cuda:
                    print(
                        "%s:%s%s %s"
                        % (
                            type(obj).__name__,
                            " GPU" if obj.is_cuda else "",
                            " pinned" if obj.is_pinned else "",
                            pretty_size(obj.size()),
                        )
                    )
                    total_size += obj.numel()
            elif hasattr(obj, "data") and torch.is_tensor(obj.data):
                if not gpu_only or obj.is_cuda:
                    print(
                        "%s → %s:%s%s%s%s %s"
                        % (
                            type(obj).__name__,
                            type(obj.data).__name__,
                            " GPU" if obj.is_cuda else "",
                            " pinned" if obj.data.is_pinned else "",
                            " grad" if obj.requires_grad else "",
                            " volatile" if obj.volatile else "",
                            pretty_size(obj.data.size()),
                        )
                    )
                    total_size += obj.data.numel()
        except Exception as e:
            pass
    print("Total size:", total_size)
</code></pre>
<p>But I don’t know how to get the name of the tensors. By name, I mean, ideally, the parameter name the tensor has, but barring that, the names of the variables holding references to it.</p>

tom: <p>Tensors don’t know their names, and they might not all have names.<br>
What you can do - at the expense of speed - if your tensors require gradients is to use the anomaly mode to get the lines of the instantiations:</p>
<pre><code class="lang-python">with torch.autograd.detect_anomaly():
     a = torch.randn(5, 5, requires_grad=True)
     b = a * 2 + 1
</code></pre>
<p>then</p>
<pre><code class="lang-auto">b.grad_fn.metadata['traceback_'][-1]
</code></pre>
<p>has</p>
<pre><code class="lang-auto">'  File "&lt;ipython-...&gt;", line 3, in &lt;module&gt;\n    b = a * 2 + 1\n'
</code></pre>
<p>This only works on tensors inside the autograd graph.</p>
<p>Another approach (with better coverage) could be to use the <code>__torch_function__</code> hook to record the source line from where the tensor is created.</p>
Load pre-trained weights to new architecture

Argo: <p>Hi! I found several similar topics, but not exactly what I was looking for.<br>
Let’s Assume I have a pre-trained EfficientNetB0. I want to create a new model and tweak architecture a little bit, then I want to load weights from trained model (for every unaltered layer) and randomly init weights for new layers.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torchvision.models as models


def build_model(num_classes=5):
    model = models.efficientnet_b0(
        weights=models.EfficientNet_B0_Weights.DEFAULT,
        )
    model.classifier[1] = nn.Linear(in_features=1280, out_features=num_classes)
    return model


model_path = 'path/model.pt'

new_model = build_model(num_classes=3)

# load old weights to the new model (iterate the named_modules in both models and load the state_dict per layer)
old_model_state_dict = torch.load(model_path)
new_model_state_dict = new_model.state_dict()
for k, v in old_model_state_dict.items():
    if k in new_model_state_dict:
        new_model_state_dict[k] = v


new_model.load_state_dict(new_model_state_dict)
</code></pre>
<p>For example in this case I will get an error because I’ve changed number of output channels, but my small checker only will (maybe) work if layer is new, but not changed</p>
<p>So I want to be able to add layers, delete layers and change something in the layer and still be able to use old weights (to every other unchanged layer)</p>
<p>Would be happy to get some recommendations on the best way to do that!</p>

ptrblck: <p>I would recommend to explicitly load the parameters you need and know were not changed. You could also try to use <code>strict=False</code> in the <code>load_state_dict</code> calls, but if you are not checking the returned mismatches your code might easily beak as any small mistake might make this call a no-op.</p>

Argo: <p>I’m not quit getting there. Let me start with a simpler example, where I just want to load a pretrained weights and change number of output neurons:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torchvision.models as models


def build_model(num_classes=5):
    model = models.efficientnet_b0(
        weights=models.EfficientNet_B0_Weights.DEFAULT,
        )

    model.classifier[1] = nn.Linear(in_features=1280, out_features=num_classes)
    return model


model_path = 'path/model.pt'
num_classes = 10

checkpoint = torch.load(model_path)
model = build_model(num_classes)
model.load_state_dict(checkpoint)
</code></pre>
<p>I am getting an error because of output size mismatch. What is the right way to deal with it?</p>
<pre><code class="lang-auto">size mismatch for classifier.1.bias: copying a param with shape torch.Size([5]) from checkpoint, the shape in current model is torch.Size([10])
</code></pre>

ptrblck: <p>Either manipulate the model after loading its <code>state_dict</code> or manipulate the <code>checkpoint</code> by replacing the <code>'classifier.1.bias'</code> (and <code>weight</code>) keys with either the already used ones or new and randomly initialized tensors in the right shape.</p>
Model converges even when gradients are inf

ryanxu8959: <p>I am training a u-net for image segmentation using Lightning.<br>
In <code>trainig_step</code> I check gradients via</p>
<pre><code class="lang-auto">        for name, param in self.model.named_parameters():
            if param.grad is not None and not torch.isfinite(param.grad).all():
                log.info(f" ------ gradient of model param {name} is finite={torch.isfinite(param.grad).all()}, has nan={torch.isnan(param).any()} ------ ")
</code></pre>
<p>and i am seeing invalid gradients right at the beginning of training. However, this disappears after a few iterations and the model continue to converge. Also I enable <code>detect_anomaly</code> but the invalid gradients are not detected. For the context I am doing <code>fp16</code> training. With <code>fp32</code> there are less occurrence of invalid gradients it seems.<br>
I am wondering why the invalid gradients didn’t affect the training at all…</p>

ptrblck: <aside class="quote no-group" data-username="ryanxu8959" data-post="1" data-topic="179731">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/90db22/48.png" class="avatar"> ryanxu8959:</div>
<blockquote>
<p>For the context I am doing <code>fp16</code> training.</p>
</blockquote>
</aside>
<p>I assume you are mixed-precision training via <code>torch.amp</code> and thus also the <code>GradScaler</code>?<br>
If so, then note that you might be checking the scaled gradients, which might overflow. The <code>GradScaler</code> will then skip the parameter update and decrease the scaling factor as described in the <a href="https://pytorch.org/docs/stable/amp.html#gradient-scaling">docs</a>.</p>
`torch.linalg.eig` works on a tensor but not `eigh()`

sinAshish: <p>Hello,</p>
<p>I am trying to compute the eigenvalues and eigenvectors of a hessian matrix (H). H is computed by convolving a volumetric tensor (V) with a second-order derivatives of a gaussian kernel (G).</p>
<p>while trying to compute the eigenvalues and eigenvectors of H, I want to use eigh() since H is a symmetric matrix, but it throws this error:</p>
<pre><code class="lang-auto">*** RuntimeError: cusolver error: CUSOLVER_STATUS_INVALID_VALUE, when calling `cusolverDnSsyevjBatched_bufferSize( handle, jobz, uplo, n, A, lda, W, lwork, params, batchsize)`
</code></pre>
<p>However, when I use <code>eig()</code>, it works fine but is really slow.<br>
I’m not able to figure out a reason for this.</p>
<p>For context, H is of shape <code>BxN^3x3x3</code> where N is the size of my volumetric grid. And the grid V is filled by querying points for occupancy using the octree based algorithm described in <a href="https://openaccess.thecvf.com/content_CVPR_2019/papers/Mescheder_Occupancy_Networks_Learning_3D_Reconstruction_in_Function_Space_CVPR_2019_paper.pdf" rel="noopener nofollow ugc">MISE</a>.<br>
However, if I query the occupancy densely instead of octree-based, eigh works fine and backpropagates as well.</p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the issue as well as the output of <code>python -m torch.utils.collect_env</code>, please?</p>
Trying to build or customize an Unreal importable version of this library?

LogicPy: <p>Hi guys. I’ve written a plugin using Python, C++, and pytorch within Unreal engine. And I’d like to release my npc chat discussion plugin model to the marketplace, but there’s one huge problem. I’m unable to compile the executable with the PyTorch library because of all the doubled names in the subdirectories. Every-time a doubled name appears. I encounter an error within Visual Studio in the form of a json alert telling me there’s duplicate file names. I’m desperately searching for a solution to this. I’m hoping for some suggestions for filtering out the doubled file names, and still retain pytorch’s functionality. But, I don’t know pytorch well enough to understand how to filter out those doubled file names.</p>
<p><strong>Here’s my goal</strong>: The administrators of Unreal will only accept a plugin if it includes pytorch as a third party plugin. Therefore, I’ve included an Unreal directory called ‘Thirdparty’ where I put the modules of my software (using raw source code from Github).</p>
<p><strong>Quick suggestion to the PyTorch developers/administrators:</strong> I absolutely love your Python module you’ve created to easily enable ai communication. I just have one small request. Can you revise the current release to produce an Unreal Visual Studio third-party compatible version that doesn’t have double file names in sub directories? It would mean the world to me if you guys decided to produce that. Thank you so much for reading and possibly considering my request! <img src="https://discuss.pytorch.org/images/emoji/apple/heart.png?v=12" title=":heart:" class="emoji" alt=":heart:" loading="lazy" width="20" height="20"></p>
Trying to do predictions by using custom weights for torchvision.models.resnet50

munikumar: <p>Hi,</p>
<p>I am saving the weights of torchvision.models.resnet50 model after training it on my custom dataset using below code:</p>
<p>best_model_weights = copy.deepcopy(<strong>self.model.state_dict()</strong>)<br>
torch.save(best_model_weights, os.path.join(self.save_model_attributes_path, f"best_weights.pth"))</p>
<p><em>I am saving self.model.state_dict() correctly instead of the function self.model.state_dict</em></p>
<p>When I try to load the custom weights using the following code:</p>
<p>self.current_model = torchvision.models.resnet50<br>
current_model = self.current_model.load_state_dict(torch.load(self.best_weights_path, device))</p>
<p>I am getting the following error:</p>
<p>Traceback (most recent call last):<br>
File “C:\Macs Lab\imageClassification\prediction_model.py”, line 113, in <br>
main()<br>
File “C:\Macs Lab\imageClassification\prediction_model.py”, line 109, in main<br>
best_weights_path=best_weights_path).classify_images_and_save_predictions()<br>
File “C:\Macs Lab\imageClassification\prediction_model.py”, line 94, in classify_images_and_save_predictions<br>
predictions_dict = self.predict_images()<br>
File “C:\Macs Lab\imageClassification\prediction_model.py”, line 74, in predict_images<br>
current_model = self.current_model.load_state_dict(copy.deepcopy(torch.load(self.best_weights_path, device)))<br>
AttributeError: ‘function’ object has no attribute ‘load_state_dict’</p>

marksaroufim: <p>You’re missing a parenthesis here<br>
<code>self.current_model = torchvision.models.resnet50()</code></p>
"length" argument of pack_padded_sequence with different padding schemes

ole_jonas: <p>Hi!</p>
<p>I was wondering about the implementation of the pack_padded_sequence method from torch.nn.utils.rnn. As per my understanding pack_padded_sequence is applied to an already padded sequence and then sent to an LSTM layer. This allows us to avoid computations on the 0-padded elements in the variable length sequences that are passed to the model. From what I understand, the standard padding scheme used in pytorch is to pad at the end of each sequence until we reach the max sequence length in the batch.</p>
<p>So my question is: Can I use the pack_padded_sequence on sequences that are front-padded and/or use split padding (doing half of the padding at the head, and half at the tail) as well? Will the underlying methods begin “iterating” through each element after a non-padding element is found, or will the iteration through the elements begin at the start of each sequence, possibly leading to not doing calculations of non-padding elements due to “length” not actually coinciding with the index of the array that the actual sequence stops at?</p>
<p>Example:<br>
Would pack_padded_sequence() and then passing to LSTM work the same for these sequences?</p>
<p>[0,0,0,0,1,2,3]<br>
[0,0,1,2,3,0,0]<br>
[1,2,3,0,0,0,0]</p>
<p>(Note: all sequences in each batch would be padded in the same way, but I only use one row per padding type for brevity…)</p>

ptrblck: <p>Based on <a href="https://github.com/pytorch/pytorch/blob/96487d0d1f638ae9c665dcd3d4b79e7e2a976f59/aten/src/ATen/native/PackedSequence.cpp#L75-L94">this comment</a> from the internal implementation I assume the padding is only supported at the end of each sample.</p>
No module named 'torch'

kevinmero: <p>Hello. I am working on the anaconda base environment. When trying to execute these commands import torch or import torch.nn as nn in Jupyter or Anaconda I get a message that says: ModuleNotFoundError: No module named ‘torch’.<br>
How do I solve that without changing environment?</p>
<p>Additional data:</p>
<ul>
<li>Anaconda Navigator 2.4.0</li>
<li>Jupyter 6.5.4</li>
<li>Spyder 5.4.3</li>
<li>Pytorch 2.0.0</li>
</ul>

ptrblck: <aside class="quote no-group" data-username="kevinmero" data-post="1" data-topic="179903">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/a9a28c/48.png" class="avatar"> kevinmero:</div>
<blockquote>
<p>How do I solve that without changing environment?</p>
</blockquote>
</aside>
<p>You would need to install PyTorch into the same environment used by Jupyter.</p>

kevinmero: <p>It is installed in the base environment but the problem continues.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/4/349465bd7be9577ef58618f8c66f826d8e7401d8.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/349465bd7be9577ef58618f8c66f826d8e7401d8" title="pytorch"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/349465bd7be9577ef58618f8c66f826d8e7401d8_2_690x202.jpeg" alt="pytorch" data-base62-sha1="7v8LE14lKVTEuklYWslZW55jBSo" width="690" height="202" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/349465bd7be9577ef58618f8c66f826d8e7401d8_2_690x202.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/349465bd7be9577ef58618f8c66f826d8e7401d8_2_1035x303.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/4/349465bd7be9577ef58618f8c66f826d8e7401d8.jpeg 2x" data-dominant-color="EFF0EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch</span><span class="informations">1150×338 48.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Close by remote host

Gorgen: <p>sorry to disturb you. i don’t know why my local computer can not connect remote host. close by remote host. actually, my remote server has worked for 2 years.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/2/521cdb797eaaf308dab785f605626da4cccbd829.png" data-download-href="https://discuss.pytorch.org/uploads/default/521cdb797eaaf308dab785f605626da4cccbd829" title="remote"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/2/521cdb797eaaf308dab785f605626da4cccbd829.png" alt="remote" data-base62-sha1="bIoYHGe5kwAORWvnmFOOd4X9zHb" width="690" height="180" data-dominant-color="2C2C2C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">remote</span><span class="informations">1278×334 17.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>could you please give me a hand?</p>
<p>many thanks</p>
<p>best wishes</p>

ptrblck: <p>I’m unsure how this error is related to PyTorch, so could you post more details, please?</p>
Error with pytorch multiprocessing dataloaders on mac

Luca_Pamparana: <p>I am using pytorch version 2.0.1 with python 3.11.3 on a mac.</p>
<p>I am training a model and it works absolutely fine when I am using <code>num_workers=0</code>. I have also manually iterated over the dataset and everything is fine.</p>
<p>The problem happens as soon as I want to use multiprocessing and parallel data loading. So even setting <code>num_workers=2</code> results in the following error:</p>
<pre><code class="lang-auto">File "/Users/luca/opt/anaconda3/envs/ideep/lib/python3.11/site-packages/lightning/__init__.py", line 25, in &lt;module&gt;
    from lightning.fabric.fabric import Fabric  # noqa: E402
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/luca/opt/anaconda3/envs/ideep/lib/python3.11/site-packages/lightning/fabric/__init__.py", line 29, in &lt;module&gt;
    from lightning.fabric.fabric import Fabric  # noqa: E402
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/luca/opt/anaconda3/envs/ideep/lib/python3.11/site-packages/lightning/fabric/__init__.py", line 29, in &lt;module&gt;
    from lightning.fabric.fabric import Fabric  # noqa: E402
    from lightning.fabric.fabric import Fabric  # noqa: E402
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/luca/opt/anaconda3/envs/ideep/lib/python3.11/site-packages/lightning/fabric/fabric.py", line 21, in &lt;module&gt;
  File "/Users/luca/opt/anaconda3/envs/ideep/lib/python3.11/site-packages/lightning/fabric/fabric.py", line 21, in &lt;module&gt;
    import torch
    import torch
  File "/Users/luca/opt/anaconda3/envs/ideep/lib/python3.11/site-packages/torch/__init__.py", line 457, in &lt;module&gt;
  File "/Users/luca/opt/anaconda3/envs/ideep/lib/python3.11/site-packages/torch/__init__.py", line 457, in &lt;module&gt;
    for name in dir(_C):
    for name in dir(_C):
                    ^^
NameError: name '_C' is not defined
</code></pre>
<p>It seems importing torch between multiple processes is creating an issue. Is this a known issue and is there a fix? Like I said everything works when using a single worker.</p>