akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Calling a layer multiple times will produce the same weights?

XavierXiao: <p>Hi, I am not sure if we call a layer define in the <code>__init__</code> for multiple times, do they share weights in the traning? For example, we have a function fc1 defined as:</p>
<pre><code class="lang-auto">def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(10,10)
        ...
</code></pre>
<p>Then I called it multiple times:</p>
<pre><code class="lang-auto">def forward(self, x1,x2):
        x1 = self.fc1(x1)
        x2 = self.fc1(x2)
        ...

</code></pre>
<p>I wonder during training, whether or not these layers maintain the same weights? If not, how do we make them share weights? Thanks a lot!</p>

ptrblck: <p>Yes, as you call the same layer, the same underlying parameters (weights and bias) will be used for the computation.</p>

bhushans23: <p>Similar question if you want to read more <a href="https://discuss.pytorch.org/t/how-to-create-model-with-sharing-weight/398/2?u=bhushans23">How to create model with sharing weight?</a></p>

Hritik_Goyal: <p>I am also trying to do the same for LLMs but when I use the same module two times in forward pass, the shared memory in the GPUs just gets doubled. If they were using same shared parameter then it would not happen right??</p>

ptrblck: <p>No, that not correct since forward activations will be stored if these are needed for the gradient computation in the backward pass. Reusing parameters does not change this behavior.</p>
Find neighborhood for torch tensor

grid_world: <p>I am trying to implement a Self-Organizing Map where for a given input sample, the best matching unit/winning unit is chosen based on (say) L2-norm distance between the SOM and the input. The winning unit/BMU (som[x, y]) has the smallest L2 distance from the given input (z):</p>
<pre><code># Input batch: batch-size = 512, input-dim = 84-
z = torch.randn(512, 84)

# SOM shape: (height, width, input-dim)-
som = torch.randn(40, 40, 84)

print(f"BMU row, col shapes; row = {row.shape} &amp; col = {col.shape}")
# BMU row, col shapes; row = torch.Size([512]) &amp; col = torch.Size([512])
</code></pre>
<p>For clarity, for the first input sample in the batch “z[0]”, the winning unit is “som[row[0], col[0]]”-</p>
<pre><code>z[0].shape, som[row[0], col[0]].shape
# (torch.Size([84]), torch.Size([84]))
</code></pre>
<p><code>torch.norm((z[0] - som[row[0], col[0]]))</code> is the smallest L2 distance between z[0] and all other som units except row[0] and col[0].</p>
<pre><code># Define initial neighborhood radius and learning rate-
neighb_rad = torch.tensor(2.0)
lr = 0.5

# To update weights for the first input "z[0]" and its corresponding BMU "som[row[0], col[0]]"-
for r in range(som.shape[0]):
    for c in range(som.shape[1]):
        neigh_dist = torch.exp(-torch.norm(input = (som[r, c] - som[row[0], col[0]])) / (2.0 * torch.pow(neighb_rad, 2)))
        som[r, c] = som[r, c] + (lr * neigh_dist * (z[0] - som[r, c]))
</code></pre>
<p>How can I implement the code for:</p>
<ol>
<li>updating weights for all units around each BMU without the 2 for loops (and)</li>
<li>do it for all of the inputs “z” (here, z has 512 samples)</li>
</ol>
Why my confusion matrix goes very slow?

Met4physics: <p>Recently, I have been doing something about semantic segmentation, but my evaluation goes very slowly. I checked my code and found out that my confusion matrix computes too slowly:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.cuda.synchronize()
t2 = time.time()
conf_m.update(label.flatten(), outs.argmax(1).flatten())
torch.cuda.synchronize()
print(time.time() - t2)
</code></pre>
<p>This takes almost 20 seconds to run. My confusion matrix is a simplified version of torchvision’s implementation:</p>
<pre data-code-wrap="python"><code class="lang-python">class ConfusionMatrix(object):
    def __init__(self, num_classes):
        self.num_classes = num_classes
        self.mat = None

    def update(self, a, b):
        n = self.num_classes
        if self.mat is None:
            self.mat = torch.zeros((n, n), dtype=torch.int64, device=a.device)
        with torch.no_grad():
            k = (a &gt;= 0) &amp; (a &lt; n)
            inds = n * a[k].to(torch.int64) + b[k]
            self.mat += torch.bincount(inds, minlength=n ** 2).reshape(n, n)

    def reset(self):
        if self.mat is not None:
            self.mat.zero_()

    def compute(self, with_background=False):
        h = self.mat.float()
        acc_global = torch.diag(h).sum() / h.sum()
        acc = torch.diag(h) / h.sum(1)
        iu = torch.diag(h) / (h.sum(1) + h.sum(0) - torch.diag(h))
        if with_background is False:
            iu = iu[1:]
        recall = torch.diag(h) / h.sum(0)
        f1 = 2 * acc * recall / (acc + recall)
        return acc_global, acc, iu, f1
</code></pre>
<p>Is there a problem with my confusion matrix? Or something else I don’t know. My experiment setup:<br>
GPU: RTX 4090<br>
CPU: AMD EPYC 9654 96-Core Processor<br>
Dataset: MS COCO<br>
torch: 2.0.0+cu118</p>
<p>update: It’s weird. Sometimes it takes 20 seconds, and sometimes it takes 5 seconds.<br>
update: I send label and outs to cpu before conf_mat.update(). The time consumption now is 1.5 seconds. I believe that the problem is about CUDA, but I don’t know where.</p>
Causality for 3d CNN

Utkarsh_tiwari: <p>I was wondering how would causality be applied for conv3d:</p>
<pre><code>    self.frontend3D = nn.Sequential(
        nn.Conv3d(1, self.frontend_nout, kernel_size=(5, 7, 7), stride=(1, 2, 2), padding=(2, 3, 3), bias=False),
        nn.BatchNorm3d(self.frontend_nout),
        frontend_relu,
        nn.MaxPool3d(kernel_size=(1, 3, 3), stride=(1, 2, 2), padding=(0, 1, 1)))


def forward(self, x, lengths):
    B, C, T, H, W = x.size()
    if type(lengths) == int:
        lengths = [lengths] * B
    x = self.frontend3D(x)
</code></pre>
Prototypical Network

mna: <p>Hi there,<br>
I am using Prototypical Network for my small dataset classification.:<br>
transforms =[partial(convert_dict, ‘class’),<br>
partial(load_class_images,split=split),<br>
partial(extract_episode, n_support, n_query)]<br>
if opt[‘data.cuda’]:<br>
transforms.append(CudaTransform())</p>
<pre><code>    transforms = compose(transforms)

    class_names = []

    with open(os.path.join(split_dir, "{:s}_image_list.txt".format(split)), 'r') as f:
        for class_name in f.readlines():
             class_name = class_name.rstrip('\n').split('-')[-1].split('.')[0]  # Extracting the class name from the image name
             class_names.append(class_name)
    class_names.sort()
    print(class_names)

    ds = TransformDataset(ListDataset(class_names), transforms)
</code></pre>
<p>in this part of code : transforms =[partial(convert_dict, ‘class’), when it reads the labels in class_names list starts from index=1 and then 2 and then 0<br>
and i think it cause low accuracy for my result and i checked when i chnage the random seed it chnages the order if reading in class_names list.</p>
<p>what can i do to slove this and make it read my list from index 0?</p>
RuntimeError: Function 'LogSoftmaxBackward0' returned nan values in its 0th output

nibtehaz: <p>Hi, when running quite a deep transformer network, I am randomly seeing this error (from torch.autograd.set_detect_anomaly(True)). My entire model becomes NaN.</p>
<p>So, I’ve tried all the stuff I could find online.</p>
<ol>
<li>Check all the input and target data, so that there is no inf or nan</li>
<li>Did all normalizations properly</li>
<li>Experimented with very low lr</li>
<li>Using clip-grad-norm</li>
<li>Reduced gradient acum iter and also batch size</li>
</ol>
<p>None of them seem to improve my situation, and I am wondering what can be the cause of this.</p>
<p>I have also printed the input and output of the log softmax layer. For the step when everything breaks down and becomes NaN, none of the values seem weird, there is no nan (torch.isnan()) there is no inf (torch.isinf()), also the min and max values seem to be reasonable (between -15 to +15)</p>
<p>So, I am wondering what can be causing this? Can it be the reason that somehow the softmax results are close to 0 and numerical instability is making the log work on negative?</p>
<p>Thank you so much for your suggestions.</p>
pickle.UnpicklingError: invalid load key, '\xd0'

Matt_Morency: <pre><code class="lang-auto">f:\Projects\Kanabot Clean\Kanabot.py:18: SyntaxWarning: invalid escape sequence '\P'
  model = torch.load(f="F:\Projects\Kanabot Simple\Model\model.safetensors")
Traceback (most recent call last):
  File "f:\Projects\Kanabot Clean\Kanabot.py", line 18, in &lt;module&gt;
    model = torch.load(f="F:\Projects\Kanabot Simple\Model\model.safetensors")
f:\Projects\Kanabot Clean\Kanabot.py:1: SyntaxWarning: invalid escape sequence '\P'
  #UI load
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\Projects\Kanabot Clean\.venv\Lib\site-packages\torch\serialization.py", line 1040, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\Projects\Kanabot Clean\.venv\Lib\site-packages\torch\serialization.py", line 1258, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_pickle.UnpicklingError: invalid load key, '\xd0'.

I just did a fresh install of Torch in my virtual enviroment with just 
pip3 install torch torchvision torchaudio

I am not sure why it is wrong. 
I am running python 3.12.2 and torch 2.2.1
</code></pre>
Why named_parameters() does not return bias?

mst72: <p>I use the following code:</p>
<pre><code class="lang-auto">for k,v in model.named_parameters():
    print(k)
</code></pre>
<p>Only weight returned:</p>
<pre><code class="lang-auto">base_model.model.model.embed_tokens.weight
base_model.model.model.layers.0.self_attn.q_proj.base_layer.weight
base_model.model.model.layers.0.self_attn.q_proj.lora_A.default.weight
base_model.model.model.layers.0.self_attn.q_proj.lora_B.default.weight
base_model.model.model.layers.0.self_attn.k_proj.base_layer.weight
</code></pre>
<p>Why doesn’t it return bias? What if I want it to return bias?</p>

ptrblck: <p>Could you post the setup of the layer you are using?</p>

mst72: <p>sorry. It’s my fault. The bias of the linear layer in my model structure is false. <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=12" title=":rofl:" class="emoji" alt=":rofl:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=12" title=":joy:" class="emoji" alt=":joy:" loading="lazy" width="20" height="20"><br>
thanks your reply</p>
Multi GPU inference and write to dataframe

Diego_Hernandez: <p>Good day for all.</p>
<p>I have a dataframe as input. I need to add a new column that compute some metric using a model. The dataframe has 150000 inputs and i have 4 gpus. One way to figure out was use DDP but i have doubts about there is a good practice. (I have seen nn.DataParalell is not recommended). Some other recoomendation?. I Have also ensure the order of the metrics: For example, if dataloder splits dataframe i need to ensure, for example, the metric with index 500 has to be written in these position in dataframe, not in another(i.e 800)(For this task i use the <strong>at</strong> function of dataframe. Some other recomendation or advice?. Regards</p>

ptrblck: <p>If you are unsure about using <code>DDP</code> (I assume because no actual training of a model takes place) you could just run different processes and assign the corresponding data shards to each of them. Layer you could then create the final output based on all output shards.</p>
Find the bottleneck of suddenly slowed traning

JohnHerry: <p>I am traning on a environment that with shared CPU and memory resoure with others,  some cards of GPUS on the machine are taken by me exclusively. the training data are put on the other machine and I can access those data through the NIS system.</p>
<p>Yes today I make a train and the traning speed is good as my expect.  today I restart another traning of the same model,  but the traing speed got suddenly slow, and each step traning cost more then 100 times then ever,  There is another person traning his job, I think there is resource competation.  but I do not know how to find out which is the bottleneck. It is not the CPU ore GPU cards, because they are not taken too much. maybe local machine memory,  maybe the io through network, or maybe the disk speed on the NIS system. Is there any suggestion for finding the bottleneck while not disturbing another person’s training?</p>

ptrblck: <p>You could profile the workload via e.g. Nsight Systems to narrow down the bottleneck during the concurrent runs.</p>
Glass Degradation Site Prediction

Filippo_Portera: <p>We are working on the subject above, where n glass image frames form an example with an associate target information (x,y) where x and y a Real numbers, which determine the next breaking point.<br>
Thus, a number of l examples are given in the training set, and f examples are given in the test set.<br>
We would like to know if there exists an algorithm (LSTM, Trasformer, etc…), that could be trained and that it is able to predict glasses breaking 2D coordinates.<br>
We were thinking about training 2 independent LSTMs, one for each coordinate, but we are not sure that this will work.</p>
Problem saving nn.Module as a TorchScript module (DLRM model)

pollo_loco: <p>Hi, I am trying to create a TorchScript module of Facebook’s <a href="https://github.com/facebookresearch/dlrm" rel="nofollow noopener">deep learning recommendation model (DLRM)</a> using torch.jit.script() method. The conversion fails owing to the following runtime error:</p>
<pre><code class="lang-auto">RuntimeError: 
cannot call a value of type 'Tensor':
  File "dlrm_s_pytorch.py", line 275
        # return x
        # approach 2: use Sequential container to wrap all layers
        return layers(x)
               ~~~~~~ &lt;--- HERE
'DLRM_Net.apply_mlp' is being compiled since it was called from 'DLRM_Net.sequential_forward'
  File "dlrm_s_pytorch.py", line 343
    def sequential_forward(self, dense_x, lS_o, lS_i):
        # process dense features (using bottom mlp), resulting in a row vector
        x = self.apply_mlp(dense_x, self.bot_l)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
        # debug prints
        # print("intermediate")
'DLRM_Net.sequential_forward' is being compiled since it was called from 'DLRM_Net.forward'
  File "dlrm_s_pytorch.py", line 337
    def forward(self, dense_x, lS_o, lS_i):
        if self.ndevices &lt;= 1:
            return self.sequential_forward(dense_x, lS_o, lS_i)
                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
        else:
            return self.parallel_forward(dense_x, lS_o, lS_i)
</code></pre>
<p>To recreate the error:</p>
<ol>
<li>Clone the DLRM repository and install the requirements.</li>
</ol>
<pre><code class="lang-auto">&lt;activate virtual environment&gt;
git clone https://github.com/facebookresearch/dlrm.git
cd dlrm
pip install requirements.txt
</code></pre>
<ol start="2">
<li>Add the following line in dlrm_s_pytorch.py at after line 179 to solve a type conversion issue:</li>
</ol>
<pre><code class="lang-auto">n = n.item()
</code></pre>
<ol start="3">
<li>Add the following snippet in dlrm_s_pytorch.py after the architecture object is initialized:</li>
</ol>
<pre><code class="lang-auto">dlrm_jit = torch.jit.script(dlrm)
sys.exit() # successful exit after compiling, no need to train
</code></pre>
<ol start="4">
<li>Run the below command:</li>
</ol>
<pre><code class="lang-auto">python  dlrm_s_pytorch.py --arch-sparse-feature-size=32 --arch-embedding-size="70446-298426-33086-133729-61823" --data-size=20480  --arch-mlp-bot="256-256-128-32" --arch-mlp-top="256-64-1" --max-ind-range=400000 --data-generation=random --loss-function=bce --nepochs=5 --round-targets=True --learning-rate=1.0  --mini-batch-size=2048
</code></pre>

1chimaruGin: <p>Did you solve it?</p>
<p>I’m also facing that issue.</p>
Torch.cuda.is_available() = false

Saurav: <p>PS C:\Users\SAURAV\scrapy_projects\cuda_test&gt; nvidia-smi</p>
<p>±--------------------------------------------------------------------------------------+<br>
| NVIDIA-SMI 538.18                 Driver Version: 538.18       CUDA Version: 12.2     |<br>
|-----------------------------------------±---------------------±---------------------+<br>
| GPU  Name                     TCC/WDDM  | Bus-Id        Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |<br>
|                                         |                      |               MIG M. |<br>
|=========================================+======================+======================|<br>
|   0  NVIDIA GeForce 930MX         WDDM  | 00000000:01:00.0 Off |                  N/A |<br>
| N/A    0C    P0              N/A / 200W |      0MiB /  2048MiB |      0%      Default |<br>
|                                         |                      |                  N/A |<br>
±----------------------------------------±---------------------±---------------------+</p>
<p>±--------------------------------------------------------------------------------------+<br>
| Processes:                                                                            |<br>
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |<br>
|        ID   ID                                                             Usage      |<br>
|=======================================================================================|<br>
|  No running processes found                                                           |<br>
±--------------------------------------------------------------------------------------+<br>
PS C:\Users\SAURAV\scrapy_projects\cuda_test&gt; nvcc --version<br>
nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2023 NVIDIA Corporation<br>
Built on Wed_Feb__8_05:53:42_Coordinated_Universal_Time_2023<br>
Cuda compilation tools, release 12.1, V12.1.66<br>
Build cuda_12.1.r12.1/compiler.32415258_0</p>
<p>Pytorch installation command: pip3 install torch torchvision torchaudio --index-url <a href="https://download.pytorch.org/whl/cu121" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu121</a></p>
<p>GPU-Z information about my GPU:</p>
<h2><a name="graphics-features-1" class="anchor" href="#graphics-features-1"></a>Graphics Features</h2>
<p>DirectX<br>
12 (11_0)</p>
<p>OpenGL<br>
4.6</p>
<p>OpenCL<br>
3.0</p>
<p>Vulkan<br>
1.3</p>
<p>CUDA<br>
5.0</p>
<p>Shader Model<br>
6.7 (5.1)</p>
<p>how to solve my problem?<br>
please be specific.</p>
Binary cross entropy with logits with weights

aure_bnp: <p>Hi !</p>
<p>I am currently working with the <code>function torch.nn.functional.binary_cross_entropy_with_logits</code> <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.binary_cross_entropy_with_logits.html#torch.nn.functional.binary_cross_entropy_with_logits" rel="noopener nofollow ugc">torch.nn.functional.binary_cross_entropy_with_logits — PyTorch 2.2 documentation</a> and I have some questions. I am not sure that I have correctly grasp the difference between <code>pos_weight</code> and <code>weight</code>.<br>
pos_weight : used to give a bigger weight to the positive class than to the negative class<br>
weight : Used to give different weight to each batch ?</p>
<ol>
<li>When using pos_weight, what is the weight that is given to the negative class ? (Is it 1 ?)</li>
<li>Is it possible to change the weigth of the negative class using <code>function torch.nn.functional.binary_cross_entropy_with_logits</code> ?</li>
<li>Is it the right way to use it in a code ?<br>
(N_0 = number of instances in class 0, N_1 : number of instances in class 1)<br>
<code>weight_1 = torch.tensor([N_0/N_1])</code><br>
<code>loss = F.binary_cross_entropy_with_logits(logits, y_true, pos_weight = weight_1)</code></li>
</ol>
<p>Thank you all for your help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> !</p>
Possible to set Torchscript Environment Variables when loaded in C++?

trickyflemming: <p>Hello! I’ve been working on running TorchModules in C++. I’ve been able to successfully trace the model and export a .ts package when the model is traced on the cpu (let’s call this “cpu.ts”). I’ve run into a snag when trying to trace the model using MPS on macOS.</p>
<p>When tracing the model with MPS, I hit the typical fallback error for a missing op which is fixed by adding <code>PYTORCH_ENABLE_MPS_FALLBACK=1</code>. After adding that environment variable to my local interpreter, the model is exported successfully to another .ts file (let’s call this one “mps.ts”).</p>
<p>If I load “cpu.ts” into my C++ program, it runs fine and responds to inputs correctly. However, if I load “mps.ts” into my C++ program, the program tosses the same error about a missing op and that I should enable <code>PYTORCH_ENABLE_MPS_FALLBACK=1</code>.</p>
<p>I compiled libtorch locally with USE_MPS and various metal flags. Do I need to add PYTORCH_ENABLE_MPS_FALLBACK=1 somewhere to the CMake build script, or is there a way to pass it to the program through C++?</p>
Does for loop inside a forward pass impacts gradient computation?

CristoJV: <p>I’m working on integrating dynamic batching into a Vision Transformer (ViT) + LSTM Network. This technique involves extracting features from a series of images, with the input vector being (Batch x Sequence x C x H x W). Given that sequence lengths vary, they are adjusted through padding with empty frames to maintain uniformity.</p>
<p>To optimize processing and avoid unnecessary computations on these padded frames, which could also impact negatively gradient computations, I’ve implemented a loop that selectively applies the ViT feature extractor only on valid frames.</p>
<p>My concern revolves around the potential implications this approach might have on fine-tuning the feature extractor, particularly in relation to the gradient computations.</p>
<p>I’m curious about whether applying the feature extractor to different frames in a loop could impact gradient calculations.</p>
<p>Below is the code snippet for reference:</p>
<pre data-code-wrap="python"><code class="lang-python">class ViTLSTM_DynamicBatching(ViTLSTM):

    def forward(self, x, lengths):
        batch_size, seq_len, channels, height, width = x.shape

        # Apply the feature extractor to each frame

        max_length = max(lengths)
        device = x.device

        # Placeholder for extracted features
        features = torch.zeros(
            (batch_size, max_length, self.lstm_input_size), device=device
        )

        for i in range(batch_size):
            # Select non-padded frames for this sequence
            sequence_frames = x[i, : lengths[i]]
            # Flatten the sequence to fit the feature extractor input shape
            sequence_frames = sequence_frames.view(-1, channels, height, width)
            # Extract features
            sequence_features = self.feature_extractor(sequence_frames)[0][
                :, 0, :
            ]
            # Add the features to the placeholder
            features[i, : lengths[i]] = sequence_features

        features = features.view(batch_size, seq_len, -1)
        lengths = lengths.to("cpu").long()
        packed_features = nn.utils.rnn.pack_padded_sequence(
            features, lengths, batch_first=True, enforce_sorted=False
        )

        # Apply the LSTM
        packed_output, _ = self.ltsm(packed_features)
        ltsm_out, _ = nn.utils.rnn.pad_packed_sequence(
            packed_output, batch_first=True
        )
        # Apply pooling to the output of the LSTM
        pooled_out = torch.mean(ltsm_out, dim=1)

        # Apply the fc
        outputs = self.fc(pooled_out)
        return outputs
</code></pre>
<p>Note that in a production environment, the strategy involves directly passing the features stored in a buffer to the LSTM network, allowing both computations to be parallelized.</p>
Why am I getting this error

pak_key: <h1><a name="h-1" class="anchor" href="#h-1"></a>循环训练、测试</h1>
<p>torch.manual_seed(42)</p>
<h1><a name="torchcudamanual_seed42-2" class="anchor" href="#torchcudamanual_seed42-2"></a>torch.cuda.manual_seed(42)</h1>
<p><span class="hashtag-raw">#循环次数</span><br>
epochs = 100</p>
<h1><a name="h-3" class="anchor" href="#h-3"></a>将数据发送至设备</h1>
<p>X_blob_train,X_blob_test = X_blob_train.to(device),X_blob_test.to(device)<br>
y_blob_train,y_blob_test = y_blob_train.to(device),y_blob_test.to(device)</p>
<h1><a name="h-4" class="anchor" href="#h-4"></a>循环</h1>
<p>for epoch in range(epochs):<br>
# 训练<br>
model_4.train()</p>
<pre><code># 概率
y_logits = model_4(X_blob_train)
y_pred = torch.softmax(y_logits,dim=1).argmax(dim=1)

# 计算损失(交叉熵)(X为训练,y为测试)    
loss = loss_fn(y_pred,y_blob_test)
# 准确性
acc = acccracy_fn(y_true=y_blob_train,y_pred=y_pred)

# 优化器清零
optimizer.zero_grad()

# 反向传播
loss.backward()

# 梯度停止
optimizer.step()

### 测试
model_4.eval()
with torch.inference_mode():
    test_logits = model_4(X_blob_test)
    test_pred = torch.softmax(test_logits,dim=0).argmax(dim=1)  # 预测出最大值的索引值
    
    test_loss = loss_fn(test_logits,test_pred)
    test_acc = acccracy_fn(y_true=y_blob_test,y_pred=test_pred) 
    
    
if epoch % 10 == 0:      
    print(f"Epoch:{epoch} | Loss:{loss:.5f},Acc:{acc:.2f}%, Test_Loss:{test_loss:.5f},Test_Acc:{test_acc:2f}%")
</code></pre>
<hr>
<p>TypeError Traceback (most recent call last) Cell In[70], <a>line 19</a> <a>17</a> # 概率 <a>18</a> y_logits = model_4(X_blob_train) —&gt; <a>19</a> y_pred = torch.softmax(y_logits,dim=1).argmax(dim=1) <a>21</a> # 计算损失(交叉熵)(X为训练,y为测试) <a>22</a> loss = loss_fn(y_pred,y_blob_test) TypeError: softmax() received an invalid combination of arguments - got (NoneType, dim=int), but expected one of: * (Tensor input, int dim, torch.dtype dtype, *, Tensor out) * (Tensor input, name dim, *, torch.dtype dtype)</p>

ptrblck: <p>Your model doesn’t se to return anything. Its output is thus <code>None</code>.</p>

pak_key: <p>I need it to output what should be done</p>

ptrblck: <p>Add a <code>return</code> statement to the <code>forward</code> method of your model.</p>
Eol(end of lifecycle) of Pytorch

Byungyeon_Kim: <p>I’d like to know the end of lifecycle for PyTorch 2.2.0 and libtorch 2.2.0.<br>
Where can I find this information?</p>

ptrblck: <p>Neither of these releases receive any direct updates so I don’t see a life cycle at all. Fixes will land in the next patch release(s) and afterwards in minor updates.</p>
Python for visual code

Archelvan: <p>I have a problem …<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5f9a3832240b30388ab2e9f32828280ff6d1c41c.png" alt="image" data-base62-sha1="dDJLXfUEj3xin7bjjYYluZRomo4" width="687" height="174"></p>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/no-module-called-pytorch/104355/8">here</a> and unrelated to PyTorch.</p>
Cuda is not available for cuda 12.2

strawberry319: <p>I’ve been trying to setup the pytoch for my conda environment but keep failing. I tried downloaded the latest pytorch with this command: conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia but cuda is still not detected. I am using a virtual envirnment for this. this is my device nvidia-smi.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf8716e5382178d6501e46ebf4e0f42a2777ed4b.png" data-download-href="https://discuss.pytorch.org/uploads/default/bf8716e5382178d6501e46ebf4e0f42a2777ed4b" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf8716e5382178d6501e46ebf4e0f42a2777ed4b.png" alt="image" data-base62-sha1="rkkCJ6aq7Ag7M1jmonIQN3VNlBp" width="690" height="358" data-dominant-color="272728"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">693×360 9.37 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Maybe try installing the pip wheels with CUDA 12.1 via <code>pip install torch</code> if your conda setup causes issues.</p>
An undefined symbol problem about torch::from_blob in torch==2.1.2

wyw: <p>when I use torch::from_blob() in my C code, it can finish compile successfully, but fail when I try to import the .so file I compile, then I get a problem show:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/1/a17e56aa9783b6fac251ea8cbef7a29631a796f8.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a17e56aa9783b6fac251ea8cbef7a29631a796f8" title="WeChatb5eba05768af165fc5810cff114c6848"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a17e56aa9783b6fac251ea8cbef7a29631a796f8_2_690x24.jpeg" alt="WeChatb5eba05768af165fc5810cff114c6848" data-base62-sha1="n2DydbrBPX0jn4CMX5PBJfpX7SM" width="690" height="24" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a17e56aa9783b6fac251ea8cbef7a29631a796f8_2_690x24.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a17e56aa9783b6fac251ea8cbef7a29631a796f8_2_1035x36.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a17e56aa9783b6fac251ea8cbef7a29631a796f8_2_1380x48.jpeg 2x" data-dominant-color="3D2D2B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">WeChatb5eba05768af165fc5810cff114c6848</span><span class="informations">2500×90 58.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I really confuse about this.<br>
Any help will be greatful, thanks.</p>

wyw: <p>I solve it by change my torch from 2.1.2 to 2.2.0。</p>
Distributed.init_process_group hangs the process

Kamkov: <p>Hi, I am trying to load a model onto two GPUs using DDP. The code below causes the process to hang on Windows 11.</p>
<pre><code class="lang-auto">os.environ['MASTER_ADDR'] = '127.0.0.1' #'0:0:0:0:0:0:0:1'
os.environ['MASTER_PORT'] = '12987'
os.environ['RANK'] = '0'
os.environ['WORLD_SIZE'] = '2'

distributed.init_process_group(backend="gloo")
</code></pre>
<p>Both address formats, ipv4 and ipv6, cause hang.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/e/ae779eafe6f158267c00fc65103e0c342c0b755e.png" data-download-href="https://discuss.pytorch.org/uploads/default/ae779eafe6f158267c00fc65103e0c342c0b755e" title="изображение"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/e/ae779eafe6f158267c00fc65103e0c342c0b755e_2_690x107.png" alt="изображение" data-base62-sha1="oTpmAbcGeLWZsCwuBYBw6w0Htfw" width="690" height="107" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/e/ae779eafe6f158267c00fc65103e0c342c0b755e_2_690x107.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/e/ae779eafe6f158267c00fc65103e0c342c0b755e_2_1035x160.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/e/ae779eafe6f158267c00fc65103e0c342c0b755e.png 2x" data-dominant-color="D6DBE6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">изображение</span><span class="informations">1116×174 21.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is there a way to fix this?</p>
FSDP.scatter_full_optim_state_dict get blocked

DandelionDream: <p>When I try to load the state dict, the process will take at least 30 minutes until the NCCL timeout error occurs.</p>
<p>This is my codes:</p>

DandelionDream: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/e/de6d090e27321e4d9ee10adfcc056eeecca94707.png" data-download-href="https://discuss.pytorch.org/uploads/default/de6d090e27321e4d9ee10adfcc056eeecca94707" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de6d090e27321e4d9ee10adfcc056eeecca94707_2_668x500.png" alt="image" data-base62-sha1="vJFBy6HsuOSuZG1SqHQS02o9SSz" width="668" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de6d090e27321e4d9ee10adfcc056eeecca94707_2_668x500.png, https://discuss.pytorch.org/uploads/default/original/3X/d/e/de6d090e27321e4d9ee10adfcc056eeecca94707.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/e/de6d090e27321e4d9ee10adfcc056eeecca94707.png 2x" data-dominant-color="232424"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">811×607 95.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>