akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Prepare multivariate time series data for Seq2Seq (many-to-many of same length)

krypton: <p>I’m still struggling to get good performance with Seq2Seq models applied to time series. I still fall back on non-deep learning models. (1) Maybe I’m not preparing my data properly before feeding the Seq2Seq (2) Maybe I made a logic error when implementing SeqSeq.</p>
<p>In this discussion, I will present how I typically prepare multivariate time series data to power Seq2Seq models.</p>
<p>For example, let’s consider the following dataset:</p>
<pre><code class="lang-auto"># shape = (5, 3)
X = [[1, 2, 3],
     [4, 5, 6],
     [7, 8, 9],
     [10, 11, 12],
     [13, 14, 15]]

# shape = (5,)
y = [3.5, 6.5, 9.5, 12.5, 15.5]
</code></pre>
<p>Setting <code>seq_len = 2 </code> and <code> stride = 2</code> we obtain the following sequences:</p>
<pre><code class="lang-auto">X_first_seq = [[1, 2, 3],
               [4, 5, 6]]
y_first_seq = [[3.5], [6.5]]

X_secon_seq = [[7, 8, 9],
               [10, 11, 12]]
y_secon_seq = [[9.5], [12.5]]

X_third_seq = [[13, 14, 15]]
y_third_seq = [[15.5]]
</code></pre>
<p>If we set <code> batch_size = 2</code> we get:</p>
<pre><code class="lang-auto">batch_1 = [X_first_seq, X_secon_seq]   # shape = (2, 2, 3) = (batch_size, seq_len, input_size)
batch_2 = [X_third_seq]   # shape = (1, 2, 3) = (batch_size, seq_len, input_size)
</code></pre>
<p>As you can see, I always prepare my multivariate data to model Seq to Seqs (many-to-many of the same length).<br>
<strong>Question 1</strong>: Is this way of preparing multivariate data good? Is there another way to do it that will be more effective?</p>
<hr>
<p>In my real world data (frequency is 1 hour)  <code>X_train.shape = (12000, 11), y_train.shape = (12000) and X_test.shape = (5000, 11)</code>.<br>
So we want to forecasting 5000 hours ahead (which corresponds to approximately 7 months)<br>
I set</p>
<pre><code class="lang-auto">seq_len = 168 # 1 week
stride  = 168 # Each sequence is a slice of 1 week
batch   = 12 # Each batch contains 3 months
</code></pre>
<p><strong>Question 2</strong>. How do you set your <code>seq_len, stride and batch</code> for large ahead forecasting?</p>
<p><strong>Question 3</strong>. Should I segment the test set batches the same way as the train set batches?</p>
Implement 1D signal convolution

BWW: <p>Hi<br>
I am trying to build a FCNN to solve deconvolution. This is quite similar as inverse problem in image. I have one detected signal and one kernel function. I want to get true signal. The relationship between detected signal, kernel function and true signal is: detected signal = true signal conv kernel.<br>
The idea is quite simple: I build a FCNN</p>
<p>class DeconvolutionNN(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(DeconvolutionNN, self).<strong>init</strong>()<br>
self.layer1 = nn.Linear(149, 64)<br>
self.layer2 = nn.Linear(64, 32)<br>
self.layer3 = nn.Linear(32, 149)<br>
<span class="hashtag-raw">#self</span>.layer4 = nn.Linear(256, 149)<br>
self.relu = nn.ReLU()</p>
<pre><code>def forward(self, x):
    x = self.relu(self.layer1(x))
    x = self.relu(self.layer2(x))
    #x = self.relu(self.layer3(x))
    x = self.layer3(x)  
    return x
</code></pre>
<p>The output of this FCNN is a random true signal and I am trying to minimize the L2 loss of true signal - random true signal conv kernel.<br>
I am confused here since the torch.nn.functional.conv1d is not traditional signal convolution. It calculates the cross correlation here. I want to call Scipy.signal.convolve here but this will cause a problem that scipy won’t track the gradient. Is there any thought that how can I solve this problem?</p>
<p>Any help would be much appreciated</p>

ptrblck: <aside class="quote no-group" data-username="BWW" data-post="1" data-topic="201390">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bww/48/69460_2.png" class="avatar"> BWW:</div>
<blockquote>
<p>I am confused here since the torch.nn.functional.conv1d is not traditional signal convolution. It calculates the cross correlation here.</p>
</blockquote>
</aside>
<p>You could flip the signal to perform a convolution instead of a cross-correlation.</p>
Torch Geometric Data Handling

Boris_Martirosyan: <p>Dear community,</p>
<p>I have problems with pytorch_geometric. I am training a graph neural network, and I encounter with a lot of problems. Firstly my Data object consist from x, edge_attr, edge_index, target and shortest_path_attr which is the shortest path matrix retrieved from Floyd-Warshall algorithm and its dimensionality is NxN where N is the number of nodes. When I try to give it to dataloader I have problems with Batching and simply tensors do not stack because of their difference of dimensions (the main problem comes from shortest path matrix). But when I pad the matrices and dataloader problem solves, I encounter another problem: my model doesn’t accept the batched tensor of X attributes (node features) because the dimensions of model and data do not match if I give a batch containing more then 1 graph info. What can I do, I would appreciate any help you can provide!</p>

AJB117: <p>A bit late, but I just worked through this problem, also using the shortest-path distance matrix. What I did was just store the flattened version of each NxN matrix and then write a function for recovering the correct flattened index.</p>
<pre><code class="lang-auto">def batched_index_select(flattened_matrix, edge_index, batch):
    device = flattened_matrix.device
    sizes = torch.bincount(batch)

    batch_u = batch[edge_index[0]]

    node_offset = torch.cat(
        [torch.tensor([0]).to(device), torch.cumsum(sizes[:-1], dim=0)]
    )
    local_idx_u = edge_index[0] - node_offset[batch[edge_index[0]]]
    local_idx_v = edge_index[1] - node_offset[batch[edge_index[1]]]

    graph_offset = torch.cat(
        [torch.tensor([0]).to(device), torch.cumsum(sizes[:-1] ** 2, dim=0)]
    )

    flat_indices = graph_offset[batch_u] + sizes[batch_u] * local_idx_u + local_idx_v

    return flattened_matrix[flat_indices]
</code></pre>
How can we export the model quantized by `PyTorch 2 Export Quantization` to the binary file?

rainy: <p>I’m completely new to PyTorch.<br>
I would like to export some public model quantized on my end by using <code>PyTorch 2 Export Quantization</code> to the file in order to execute it on various platform  such as TVM or TensorRT.<br>
Could you kindly share some example codes?</p>

jcaip: <p>Example of pt2 export workflow:<br>
<a href="https://pytorch.org/tutorials/prototype/pt2e_quant_ptq_static.html#convert-the-calibrated-model-to-a-quantized-model" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/prototype/pt2e_quant_ptq_static.html#convert-the-calibrated-model-to-a-quantized-model</a></p>
<p>Here’s an example of how to export to TensorRT, but this uses the old FX workflow<br>
<a href="https://pytorch.org/TensorRT/_notebooks/vgg-qat.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/TensorRT/_notebooks/vgg-qat.html</a></p>

rainy: <p><a class="mention" href="/u/jcaip">@jcaip</a></p>
<p>Thank you for your sharing!<br>
They look helpful to understand.<br>
I understood that a simple example that would execute the model quantized by <code>PyTorch 2 Export</code> with  other platform does not yet exist in the document.</p>
A question about autograd.Function

sadegh94: <p>hello everyone<br>
I’m new here and this is my first topic.<br>
I have problem with my code.<br>
I’ve searched and didn’t find a solution.</p>
<p>The problem is I have a function that has an output shape different than input<br>
for example if input shape be (batch_no,1), my output shape will be (batch_no,2,10).<br>
the <code>torch.autograd.Function</code> makes the grad shape as the output but excpects as input.<br>
I thought this should not be an issue because we should have n_output*n_input gradians.</p>
<p>i share a dummy code here which makes the same error.<br>
I would appreciate any help</p>
<pre><code class="lang-auto">class test2(torch.autograd.Function):
    @staticmethod
    def forward(ctx,x):
        ctx.save_for_backward(x)        
        result = [np.ones((5)) for i in x]
        return torch.tensor(result)
    @staticmethod
    def backward(ctx, grad):
        x = ctx.saved_tensors
        return grad

    
inp = torch.tensor([0.5,0.5,0.5,0.5], requires_grad=True)
x = test2.apply(inp)
x.shape
x.sum().backward()
</code></pre>
<p>and i get this error:<br>
<code>RuntimeError: Function test2Backward returned an invalid gradient at index 0 - got [4, 5] but expected shape compatible with [4] </code></p>

eqy: <p>I recommend this guide <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html" rel="noopener nofollow ugc">PyTorch: Defining New autograd Functions — PyTorch Tutorials 2.3.0+cu121 documentation</a><br>
which specifies that the return should be the gradient with respect to the input (it must have the same shape as the input). In practice if an input variable is used in multiple places (multiple outputs), its gradient is the total derivative (sum of gradient contribution from each of these outputs). See also: <a href="https://en.wikipedia.org/wiki/Total_derivative#Example:_Differentiation_with_indirect_dependencies" rel="noopener nofollow ugc">Total derivative - Wikipedia</a></p>

sadegh94: <p>Thanks that’s right <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>
Distributed training issue with PyTorch estimator in AWS Sagmaker

Ashvinee: <p>I’m implementing distributed training on AWS Sagemaker utilizing PyTorch estimator.<br>
It is based on the code sourced from this <a href="https://github.com/AustinXY/GIRAFFEHD" rel="noopener nofollow ugc">link</a></p>
<p>Here is the PyTorch estimator configuration:<br>
estimator = PyTorch(…<br>
framwork_version = ‘1.13.1’,<br>
instance_count=1,<br>
py_version = ‘py39’,<br>
instancee_type = ‘ml.g5.12xlarge’,<br>
distribution = {‘torch_distributed’: {‘enabled’: True}}<br>
)</p>
<p>The training is getting stuck on the ‘generator’ line.<br>
Here is the snippet of the code.</p>
<pre><code class="lang-auto">if args.distributed:
        generator = nn.parallel.DistributedDataParallel(
            generator,
            device_ids=[args.local_rank],
            output_device=args.local_rank,
            broadcast_buffers=False,
            find_unused_parameters=True
        )

        discriminator = nn.parallel.DistributedDataParallel(
            discriminator,
            device_ids=[args.local_rank],
            output_device=args.local_rank,
            broadcast_buffers=False,
            find_unused_parameters=True
        )
</code></pre>
<p>For 4 GPUs machine instance, the local_rank number can be in between 0 and total-1.<br>
args.local_rank = 2   <span class="hashtag-raw">#between</span> 0 and 3</p>
<p>The following runtime error generated:</p>
<pre><code class="lang-auto">RuntimeError DDP expects same model across all ranks, but rank 2 has 256 params, while rank 0 has inconsistent 0 params.

Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.
</code></pre>
<p>I’m using image data with LMDB python library for generative model.<br>
I’m using it in the same manner used for the non-distributed training.</p>
<p>I would greatly appreciate any insights or suggestions on the nn.parallel.DistributedDataParallel() of generative model.<br>
How to resolve it?</p>
How much does TORCH.AMP improve performance

karambit: <p>I wanted to know that does TORCH.AMP operates same as apex amp package?</p>
<p>I applied torch.autocast and GradScaler together on my pytorch lightning gan model but training speed did not improved and even increased slightly, gpu and memory usage decreased a bit.</p>
<p>my question is that are these slight changes logical or not because I red an article which improved memory consumption dramatically using amp and checkpointing.</p>

karambit: <p>Update: after some further runs I faced no improvement and the metrics event worsened. this shows that TORCH.AMP might not working properly in my code.</p>
Version of protobuf used in libtorch-win-shared-with-deps-2.1.1+cpu.zip?

ryguy: <p>Hello, I am currently using pre-compiled pytorch package, libtorch-win-shared-with-deps-2.1.1+cpu.zip. Is there any way to know which version of protobuf was compiled against for torch_cpu.dll in this package? I see the commit-id pointing to 3.13.0.1 here, <a href="https://github.com/pytorch/pytorch/tree/v2.1.1/third_party" class="inline-onebox" rel="noopener nofollow ugc">pytorch/third_party at v2.1.1 · pytorch/pytorch · GitHub</a> , which is the same on current tip / latest release – but since this is such an old version of protobuf, I wasn’t sure if this indeed was the version used for compilation?</p>
<p>Thanks!</p>
Implementing padding mask on LSTM

floraz: <p>Hi Everyone. I am wondering if there is a way to implement padding mask on LSTM? Like in this blog: <a href="https://towardsdatascience.com/how-does-masking-work-in-an-rnn-and-variants-and-why-537bf63c306d" class="inline-onebox" rel="noopener nofollow ugc">How does masking work in an RNN (and variants) and why | by Borun Chowdhury Ph.D. | Towards Data Science</a></p>
<p>The main idea is that <em><strong>all masked values are ignored and the output and state for those time steps are just copied over from the last non-masked state.</strong></em></p>
<p>Thank you!</p>
Pytorch and CUDA environment setup issue

Ashvinee: <p>Hi, all!</p>
<p>I’m trying to setup the conda environment from <a href="https://github.com/AustinXY/GIRAFFEHD/blob/main/environment.yml" rel="noopener nofollow ugc">[link]</a> on my machine.</p>
<p>Below is my machine specification from nvidia-smi command output.<br>
±----------------------------------------------------------------------------+<br>
| NVIDIA-SMI 525.147.05 Driver Version: 525.147.05 CUDA Version: 12.0 |<br>
|-------------------------------±---------------------±---------------------+<br>
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |<br>
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |<br>
| | | MIG M. |<br>
|===============================+======================+======================|<br>
| 0 NVIDIA RTX A5000 Off | 00000000:21:00.0 Off | Off |<br>
| 30% 57C P2 68W / 230W | 1404MiB / 24564MiB | 0% Default |<br>
±------------------------------±---------------------±---------------------+</p>
<p>±----------------------------------------------------------------------------+<br>
| Processes: |<br>
| GPU GI CI PID Type Process name GPU Memory |<br>
| ID ID Usage |<br>
|=============================================================================|<br>
| No running processes found |<br>
±----------------------------------------------------------------------------+</p>
<h1><a name="the-requirement-provided-in-the-github-repo-is-1" class="anchor" href="#the-requirement-provided-in-the-github-repo-is-1"></a>The requirement provided in the GitHub repo is:</h1>
<ul>
<li>python=3.8.3</li>
<li>pytorch=1.7.0</li>
<li>torchvision=0.8.1</li>
<li>cudatoolkit=10.1<br>
=====================</li>
</ul>
<p>Setup docker environment using image <a href="http://nvcr.io/nvidia/pytorch:20.09-py3" rel="noopener nofollow ugc">nvcr.io/nvidia/pytorch:20.09-py3</a>.</p>
<p>During the setup, I received this issue:<br>
NVIDIA NVIDIA RTX A5000 with CUDA capability sm_86 is not compatible with the current PyTorch installation</p>
<h1><a name="i-went-through-this-post-where-it-was-mentioned-to-upgrade-pytorch-i-configured-new-conda-environment-which-includes-2" class="anchor" href="#i-went-through-this-post-where-it-was-mentioned-to-upgrade-pytorch-i-configured-new-conda-environment-which-includes-2"></a>I went through this post where it was mentioned to upgrade pytorch<br>
I configured new conda environment which includes:</h1>
<ul>
<li>python=3.8.3</li>
<li>pytorch=1.9.0</li>
<li>torchvision=0.10.0</li>
<li>cudatoolkit=11.1<br>
=====================</li>
</ul>
<p>Now, the training is going on fine, but the intermediate generated images look very unexpected.<br>
I’m not facing any issue when installing and training in the machine with RTX 2080.<br>
Is it because of the required env has to do something with CUDA or GPU architecture?</p>

ptrblck: <p>Install the current stable or nightly PyTorch binary with CUDA 11.8 or 12.1 and it should work.</p>

Ashvinee: <p>Thanks! it took time, but it worked.</p>
Is it safe to write to a shared memory tensor from multiple processes?

piojanu: <p>I’ve searched through the Internet and couldn’t find an answer which drives me crazy <img src="https://discuss.pytorch.org/images/emoji/apple/open_mouth.png?v=12" title=":open_mouth:" class="emoji" alt=":open_mouth:" loading="lazy" width="20" height="20"></p>
<p>In the Hogwild training example <a href="https://github.com/pytorch/examples/tree/main/mnist_hogwild" rel="noopener nofollow ugc">here</a>, why don’t we have to synchronize before the optimizer step that updates the shared model parameters? Does PyTorch take care of making this operation atomic? Where is it specified in the docs then?</p>
<p>Plz help <img src="https://discuss.pytorch.org/images/emoji/apple/pleading_face.png?v=12" title=":pleading_face:" class="emoji" alt=":pleading_face:" loading="lazy" width="20" height="20"></p>
Parent child relation between layers

Rohit_Dubey: <p>I am trying to get a dictionary output with key as the layer name and value as the parent layer name.</p>
<p>For example.</p>
<p>class Model(nn.Module):</p>
<pre><code>def __init__(self, instance_size=15, X_size=5, Uncern_size=5):

    super(MyModel, self).__init__()

    self.embedding_instance = nn.Linear(instance_size, 8)
    self.embedding_X = nn.Linear(X_size, 3)
    self.embedding_uncern = nn.Linear(Uncern_size, 3)
    self.fc1 = nn.Linear(14, 5)  
    self.fc2 = nn.Linear(5, 1)
    

def forward(self, instance, X, uncern):
    
    instance_embed = torch.relu(self.embedding_instance(instance))
    X_embed = torch.relu(self.embedding_X(X))
    uncern_embed = torch.relu(self.embedding_uncern(uncern))
    
    concatenated = torch.cat((instance_embed, X_embed, uncern_embed), dim=1)
    
    output = torch.relu(self.fc1(concatenated))
    output = self.fc2(output)

    return output
</code></pre>
<p>model =Model()<br>
The function I am trying to write should return the child-parent (key-value) relation as follows.</p>
<p>{  input_layers =&gt; [“instance”, “X”, “Uncern”],<br>
hidden_layers =&gt;{“embedding_inst” =&gt; [“instance”],<br>
“enbedding_X” =&gt; [“X”],<br>
“embedding_Uncern” =&gt; [“Uncern”],<br>
“fc1” =&gt; [“embedding_inst”, “embedding_X”, “embedding_Uncern”]},<br>
output_layers =&gt; {“fc2” =&gt; [“fc1”])}<br>
}</p>
<p>I am trying to use the model.fc1.named_children() method, but it returns a empty list. I was hoping to get the layer name “fc2” as an output so that I could write a function that produced the desired dict. Is there a way I could do it?</p>
Index list of ranges

whoab: <p>Suppose I have a T x W matrix.</p>
<p>Now also suppose I have two 1D int tensors, W1 and W2, representing the start and stop indices of the range along each row.</p>
<p>I want to index the T x [W1:W2] section out of the original matrix.</p>
<p>How do I do this efficiently?</p>

whoab: <p>This is very related: <a href="https://discuss.pytorch.org/t/range-based-indexing-based-on-another-tensor/158666/3" class="inline-onebox">Range-based indexing based on another tensor - #3 by KFrank</a></p>
Any specific reason for chosing the default batch sizes for the torch dynamo benchmark suite

Abhishek_Ghosh: <p>I was trying to test the <a href="https://github.com/pytorch/pytorch/tree/main/benchmarks/dynamo" rel="noopener nofollow ugc">TorchDynamo Benchmark Suite</a>.</p>
<p>In the model list files:</p>
<ul>
<li><a href="https://github.com/pytorch/pytorch/blob/34bce27f0d12bf7226b37dfe365660aad456701a/benchmarks/dynamo/torchbench_models_list.txt" rel="noopener nofollow ugc">torchbench_models_list</a></li>
<li><a href="https://github.com/pytorch/pytorch/blob/34bce27f0d12bf7226b37dfe365660aad456701a/benchmarks/dynamo/huggingface_models_list.txt" rel="noopener nofollow ugc">huggingface_models_list</a></li>
<li><a href="https://github.com/pytorch/pytorch/blob/34bce27f0d12bf7226b37dfe365660aad456701a/benchmarks/dynamo/timm_models_list.txt" rel="noopener nofollow ugc">timm_models_list</a></li>
</ul>
<p>Or in the yml files:</p>
<ul>
<li><a href="https://github.com/pytorch/pytorch/blob/34bce27f0d12bf7226b37dfe365660aad456701a/benchmarks/dynamo/torchbench.yaml#L3C1-L17C1" rel="noopener nofollow ugc">torchbench.yml</a><pre data-code-wrap="yml"><code class="lang-yml">batch_size:
  training:
    demucs: 4
    dlrm: 1024
    densenet121: 4
    hf_Reformer: 4
    hf_T5_base: 4
    timm_efficientdet: 1
    llama_v2_7b_16h: 1
    # reduced from 16 due to cudagraphs OOM in TorchInductor dashboard
    yolov3: 8

  inference:
    timm_efficientdet: 32
</code></pre>
</li>
<li><a href="https://github.com/pytorch/pytorch/blob/34bce27f0d12bf7226b37dfe365660aad456701a/benchmarks/dynamo/huggingface.py#L479C9-L494C18" rel="noopener nofollow ugc">huggingface.py</a></li>
</ul>
<pre data-code-wrap="python"><code class="lang-python">            model.gradient_checkpointing_enable()
        if model_name in BATCH_SIZE_KNOWN_MODELS:
            batch_size_default = BATCH_SIZE_KNOWN_MODELS[model_name]
        elif batch_size is None:
            batch_size_default = 16
            log.info(
                f"Batch size not specified for {model_name}. Setting batch_size=16"
            )


        if batch_size is None:
            batch_size = batch_size_default
            if model_name in BATCH_SIZE_DIVISORS:
                batch_size = max(int(batch_size / BATCH_SIZE_DIVISORS[model_name]), 1)
                log.info(
                    f"Running smaller batch size={batch_size} for {model_name}, orig batch_size={batch_size_default}"
                )
</code></pre>
<hr>
<p>In the above cases, where a specific batch size is associated with a particular model name, (also as in huggingface, there is the use of BATCH_SIZE_DIVISORS to divide the default batch size) is there any specific reason for chosing that particular “default” batch size in the first place? Like are these batch sizes the one which performs the best with torch dynamo?</p>
<p>Is the selection of these batch sizes just random or intentional?</p>
<p>Also, in the accuracy tests, I find that using a batch size greater than recommended leads to accuracy failure.</p>

marksaroufim: <p>They’re not batch sizes that perform best, they’re generally batch sizes that work well enough so our CI machines don’t OOM</p>
Tf.reduce_sum, tf.reduce_mean

arya47: <p>what will the equivalent code be for</p>
<pre><code class="lang-auto"> z_loss = 0.5 * tf.reduce_sum(tf.square(z_mean) + tf.exp(z_logvar) - z_logvar  - 1, axis = [1,2,3])
</code></pre>
<p>What are the pytorch equivalent for reduce_mean and reduce_sum</p>
<p>Thanks</p>

ptrblck: <p><code>torch.mean</code> and <code>torch.sum</code> would be the replacements (or call <code>.mean()</code> or <code>.sum()</code> on a tensor directly).</p>

Sushmanth_reddy_Mere: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>cosim = tf.maximum(cosine_similarity(tensor_a, tensor_b), 1e-1) ** cossim_pow</p>
<p>can we use torch.clap_min instead of tf.maximum ??for above tensor flow implementation in pytorch</p>
<p>cosim = torch.pow(torch.clamp_min(cosine_similarity(tensor_a, tensor_b), 1e-1), cossim_pow)</p>

ptrblck: <p>Yes, <code>clamp_min</code> can be used instead of <code>maximum</code> also in PyTorch. The difference is that <code>torch.max</code> would expect tensors while <code>clamp_min</code> accepts scalar values for its threshold.</p>
Hparams logs not found by tensorboard

app31t: <p>I have been logging my metrics to tensorboard as scalars but wanted organise everything better by also logging hparams. However, tensorboard somehow can‘t display the logs.</p>
<p>When I use <code>writer.add_hparams</code>, the hparams card appears in the upper left hand corner in tensorboard and files are being added to the correct logging directory. However, the hparams page always says that no logs could be found, no matter how many logs I write.</p>
<p>Even the most minimal example from the PyTorch documentation will not work:</p>
<pre><code class="lang-auto">from torch.utils.tensorboard import SummaryWriter
with SummaryWriter() as w:
    for i in range(5):
        w.add_hparams({'lr': 0.1*i, 'bsize': i},
                      {'hparam/accuracy': 10*i, 'hparam/loss': 10*i})
</code></pre>
<p>My tensorboard version is 2.16.2 . Am I doing anything wrong?</p>

app31t: <p>I was eventually able to solve it myself by downgrading protobuf via:</p>
<pre><code class="lang-auto">pip uninstall protobuf
pip install protobuf==3.20.0
</code></pre>
How to matrix-multiply 2d tensor by each vector in 3d matrix

ellat: <p>I have two matrices:</p>
<pre><code class="lang-auto">t1 = torch.randn(4,4)
t2 = torch.randn(4,1)
</code></pre>
<p>I can matrix-multiply them like so:</p>
<pre><code class="lang-auto">res = torch.matmul(t1, t2)
</code></pre>
<p>This results in a 4x1 matrix. I would like to do this multiplication many at once. <code>t1</code> remains the same, but I want to make <code>t2</code> of the size <code>[4,10,10]</code></p>
<p>If I use matmul() directly it gives me the error: <code>RuntimeError: Expected size for first two dimensions of batch2 tensor to be: [4, 4] but got: [4, 10].</code></p>
<p>How can I do this multiplication so that <code>t1</code> is multiplied by each <code>[:,r,c]</code> vector inside <code>t2</code>?</p>

ellat: <p>I solved it by rehsaping <code>t2</code> to [4,100]</p>
<p>And then multiplying the two tensors using:</p>
<pre><code class="lang-auto">res = torch.einsum('ij,jk-&gt;ik', [t1, t2])
</code></pre>
Segmentation with imbalance dataset

Deep_course: <p>Hi Everyone<br>
I Am training Unet for binary segmentation with an input size of 2048x2048 and due to an imbalance class in my dataset am using class weight. I Am training with a combined (BCE and Dice) loss. when evaluated on the test set, the model does well on the positive but also predicts the negative class positive. what are some suggestions I can to resolve this challenge? any help is greatly appreciated.</p>
Mask 3-D tensor with variable length masks

roloza7: <p>Suppose I have a tensor of shape (B, L, E) and a boolean mask of shape (B, L).</p>
<p>I’m trying to pluck the unmasked sections from the 3-D tensor and place them into another tensor of shape (B, D, E) (where D is smaller than or equal to L). I’ve looked into the gather and scatter operations but they’ve so far been hard to get my head around <img src="https://discuss.pytorch.org/images/emoji/apple/sweat.png?v=12" title=":sweat:" class="emoji" alt=":sweat:" loading="lazy" width="20" height="20"> and it’s not obvious how I would deal with padding since masks might not necessarily have the same number of zeros.</p>
<p>Any ideas/directions I could follow? Thanks!!</p>
Different results from torch.get_num_threads when using docker in x86 and arrch64 cpu

qmzznbxhl: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/8/18cbe71aedb581ea03028d5d0f00f0b103add8c8.png" data-download-href="https://discuss.pytorch.org/uploads/default/18cbe71aedb581ea03028d5d0f00f0b103add8c8" title="arrch64"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/18cbe71aedb581ea03028d5d0f00f0b103add8c8_2_690x157.png" alt="arrch64" data-base62-sha1="3xmjLXqPpaESQJw07SqRgcslJck" width="690" height="157" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/18cbe71aedb581ea03028d5d0f00f0b103add8c8_2_690x157.png, https://discuss.pytorch.org/uploads/default/original/3X/1/8/18cbe71aedb581ea03028d5d0f00f0b103add8c8.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/8/18cbe71aedb581ea03028d5d0f00f0b103add8c8.png 2x" data-dominant-color="060606"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">arrch64</span><span class="informations">869×198 18.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>i got this value from x86 cpu is 1.</p>
<p>I’m trying to control the number of threads utilized by OpenMP or Intel Math Kernel Library (MKL) within a Docker container so that it uses only the threads available inside the container instead of using those from the underlying host machine.<br>
now i need to set MKL_NUM_THREADS or OMP_NUM_THREADS;</p>
<p>is there anyone know why this happed?</p>
<p>is to blame arrch64 cpu or pytorch.</p>
Can toch.cond be used recursively?

jwillette: <p>I am wondering if torch.cond can be used recursively to define something a bit more complicated than a simple if/else function call as shown in the docs. Is there any red flag for implementing arbitrary logic with it?</p>
<p>I have a recursive call that seemed to be working fine as I developed it, but then it started ignoring true conditions and following the wrong chain (unless I have a bug somewhere). I can try to make a repro if needed, but first I thought it would be good to ask if recursion is a no-no.</p>
<p>Thanks</p>

bdhirsh: <p>If you’re able to create a small repro + file an issue on GitHub that would be helpful! I wouldn’t be surprised if this doesn’t work today (we handle cond in dynamo by statically tracing out subgraphs for the true and false branches, I’m not sure off the top of my head how that tracing step will handle the case where we are recursively calling back into the same function that invokes the cond)</p>

jwillette: <p>Ok, thanks. I will try to make a small repro soon. For now, it appeared to me that the recursion caused an error in that the condition in the recursion was already decided from a past recursion step so that something like this…</p>
<pre><code class="lang-auto">x = torch.tensor(0)
def func(x):
  if x &lt; 2:
    return func(x + 1)
  return foo(x)
</code></pre>
<p>x was always evaluated to be &lt; 2 because this was true on the first iteration, and when the value surpasses 2, it still executed the if block</p>
Every "n" batches, there is a very slow batch

whoab: <p>With num_workers = 8, here are the times it takes to load each batch (I am reading image files from disk with an imagefolder):</p>
<p>25.684549808502197<br>
7.860300064086914<br>
0.0003273487091064453<br>
2.17626690864563<br>
0.00019550323486328125<br>
0.00014281272888183594<br>
1.0835604667663574<br>
7.271766662597656e-05<br>
10.315490961074829<br>
1.0640451908111572<br>
2.1994986534118652<br>
0.00010514259338378906<br>
2.168898582458496<br>
0.00011897087097167969<br>
2.2624127864837646<br>
0.0001761913299560547<br>
19.841031312942505<br>
2.242060899734497<br>
0.00010609626770019531<br>
4.744529724121094e-05<br>
2.2600150108337402<br>
0.00014209747314453125<br>
2.28255558013916<br>
8.440017700195312e-05<br>
21.11872434616089<br>
1.1199963092803955<br>
0.00010371208190917969<br>
0.00013947486877441406<br>
2.335517644882202<br>
0.00017642974853515625<br>
2.341693639755249<br>
0.0001652240753173828<br>
19.975724935531616<br>
2.2457833290100098<br>
0.00010275840759277344<br>
0.00010085105895996094<br>
2.3120276927948<br>
0.0001964569091796875<br>
2.362861394882202<br>
0.00013828277587890625<br>
21.159653186798096<br>
1.117790937423706<br>
0.000133514404296875</p>
<p>Every num_workers, there is a very, very slow batch. Why is this? How to fix it?</p>

ptrblck: <p>The workers might not be fast enough in filling their queue with new batches and your training loop would need to wait for them.</p>
Expand dimension

se7en: <p>Hi, I want to expand dimension while doing transforming, like trans = transforms.Compose([transforms.ToTensor(), Expand_dims(trans, dim(1)])? Is there some function that Pytorch support in transforms?</p>

Balamurali_M: <p>Do that expand dims to the transformed output using reshape or view.</p>

t170815518: <p>I’m using</p>
<p><code>transforms.Lambda(lambda x: torch.unsqueeze(torch.tensor(x), 1))</code></p>
Signal handling and torch.save()

Hannes_Loeffler: <p>Hi,</p>
<p>I am thinking of  sginal handler, signal.signal()m to catch a certain range of signals like SIGINT etc.  The idea is that this handler would save out a checkpoint of the current state of a model when a signal event happens that would eventually lead to termination of the current process.  The concern is that this would happen, say, half-way through backpropagation leaving part of the weights and biases not updated/in an inconsistent state.</p>
<p>So I wonder if that is a sensible strategy at all.  What can I expect to happen in case of such events?  Is there a difference between CPU and GPU?  How would I write out checkpoints if a user presses Ctrl-C?</p>
<p>Many thanks.</p>

Maximilian_Igl: <p>Hey, wondering the exact same thing right now, did you ever find an answer or tried it and it always worked?</p>

marksaroufim: <p>This RFC has a list of hook APIs that are relevant along with some suggested improvements <a href="https://github.com/pytorch/pytorch/issues/75287" class="inline-onebox">[RFC] Consolidated and unified state_dict and load_state_dict hooks · Issue #75287 · pytorch/pytorch · GitHub</a></p>

Maximilian_Igl: <p>Hi Mark,</p>
<p>Thank you for your reply! Unfortunately, I’m still not sure I understand how to handle it.<br>
As I understand your linked post, it discusses various hooks that allow inserting computation before the state_dict is computed.</p>
<p>What I’m wondering is whether a signal handler for SIGTERM (registered with <code>signal.signal()</code>), could see the <code>state_dict</code> in a corrupted state (for example when SIGTERM is sent during the weight update step), hence making it a bad idea to save the state_dict to disk in the signal handler before exiting the program.</p>
<p>For context: This is relevant e.g. for SLURM managed clusters.</p>
<p>Thank you for your help!<br>
Best, Max</p>
Libtorch c++ segmentation fault while printing tensor

ptuser: <p>I was trying to use the C++ library on an Intel Macbook, following this tutorial – <a href="https://pytorch.org/cppdocs/installing.html" class="inline-onebox" rel="noopener nofollow ugc">Installing C++ Distributions of PyTorch — PyTorch main documentation</a><br>
The only difference is that I use the <code>libtorch</code> installed in Anaconda (pytorch version <code>2.2.2</code>).</p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.18)
project(vllm_csrc VERSION 0.1.0 LANGUAGES C CXX)

include(CTest)
enable_testing()

set(CMAKE_PREFIX_PATH "/home/anaconda3/envs/vllm/lib/python3.10/site-packages/torch/share/cmake")
find_package(Torch REQUIRED)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")

add_executable(vllm_csrc main.cpp)
target_link_libraries(vllm_csrc "${TORCH_LIBRARIES}")
set_property(TARGET vllm_csrc PROPERTY CXX_STANDARD 17)

set(CPACK_PROJECT_NAME ${PROJECT_NAME})
set(CPACK_PROJECT_VERSION ${PROJECT_VERSION})
include(CPack)
</code></pre>
<p>and the main program is a simple hello world</p>
<pre data-code-wrap="c++"><code class="lang-c++">#include &lt;iostream&gt;
#include &lt;torch/torch.h&gt;

int main(int, char**){
    std::cout &lt;&lt; "hello world" &lt;&lt; std::endl;
    
    torch::Tensor tensor = torch::rand({2, 3});
    torch::Tensor other_tensor = torch::rand({2, 3});
    torch::Tensor final_tensor = tensor + other_tensor;

    if (tensor.defined()) {
        std::cout &lt;&lt; "Tensor size: " &lt;&lt; tensor.sizes() &lt;&lt; std::endl;
        std::cout &lt;&lt; tensor &lt;&lt; std::endl;
    }
}
</code></pre>
<p>The project builds successfully. When I run the executable, I see the size, but I get a segmentation fault while printing the tensor (which is the example in the link).</p>
<pre data-code-wrap="bash"><code class="lang-bash">./vllm_csrc 
hello world
Tensor size: [2, 3]
zsh: segmentation fault  ./vllm_csrc
</code></pre>
Batch Normalization in Pytorch Model

Abhishek_Gupta: <p>Lets consider the below code</p>
<pre><code class="lang-auto">model1 = torchvision.models.resnet18(num_classes = 7)
</code></pre>
<p>I am going to use torchvision Resnet18 model. I am in doubt whther these model contains Batch Normalization layer or not.</p>

ptrblck: <p>You can <code>print</code> the model and would see it contains batchnorm layers:</p>
<pre data-code-wrap="python"><code class="lang-python">model = torchvision.models.resnet18(num_classes = 7)
print(model)
ResNet(
  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (relu): ReLU(inplace=True)
...
</code></pre>
How to export a model that without forward() function

elinliu0823: <p>Just like title,if i got a “bone” of a neural network without any specific <code>forward()</code> function like:</p>
<pre data-code-wrap="python"><code class="lang-python">def forward(x):
      raise NotImplementedError
</code></pre>
<p>Is that possible to export this model into ONNX or even TensorRT someway?</p>

ptrblck: <p>Without the <code>forward</code> implementation PyTorch wouldn’t know how the initialized modules and parameters should be used in the forward pass and I would assume the same applies to ONNX and TensorRT.</p>

elinliu0823: <p>Sorry late for the reply sir.<br>
So can i just moving the code which in inference() function into forward to make Pytorch model recognizable?</p>

elinliu0823: <p>Seems Pytorch recognize my model as expect,but there is another problem happend:</p>
<pre><code class="lang-auto">torch._dynamo.exc.UserError: Tried to use data-dependent value in the subsequent computation. 
This can happen when we encounter unbounded dynamic value that is unknown during tracing time.   
You will need to explicitly give hint to the compiler.
Please take a look at constrain_as_value OR constrain_as_size APIs.  
It appears that you're trying to get a value out of symbolic int/float whose value is data-dependent (and thus we do not know the true value.) 
The expression we were trying to evaluate is i5 &lt; 0 (unhinted: i5 &lt; 0). 
Scroll up to see where each of these data-dependent accesses originally occurred.
</code></pre>
DDP,batchnorm,two forward error

dkz: <p>When using DDP, if there is a batch normalization (BN) layer in the network and only a single GPU is employed, an error will occur after two consecutive forward passes prior to backward propagation:</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation</p>
</blockquote>
<p>However, this issue does not arise when using multiple GPUs. Why is this the case,?nn.SyncBatchNorm.convert_sync_batchnorm() has already been applied to convert batch normalization to synchronized batch normalization.</p>

ptrblck: <p>I would assume <code>DDP</code> would detect a single GPU and would execute the single GPU run. However, I also don’t understand your use case in applying <code>DDP</code> on a single device.</p>

ptrblck: <p>Your observation still does not explain your use case, so are you just trying random configs to see which one would fail?</p>

dkz: <p>The overall code is like this<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c431e28e3e68a66bb7139d48757819b86faf2ed3.png" data-download-href="https://discuss.pytorch.org/uploads/default/c431e28e3e68a66bb7139d48757819b86faf2ed3" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c431e28e3e68a66bb7139d48757819b86faf2ed3_2_378x500.png" alt="image" data-base62-sha1="rZCsF2UIKzKw2TbGDt68l93TekH" width="378" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c431e28e3e68a66bb7139d48757819b86faf2ed3_2_378x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c431e28e3e68a66bb7139d48757819b86faf2ed3_2_567x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c431e28e3e68a66bb7139d48757819b86faf2ed3_2_756x1000.png 2x" data-dominant-color="F7F7F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1250×1652 204 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>If I run <em><strong>torchrun train.py</strong></em>, an error occurs:</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [3]] is at version 3; expected version 2 instead.</p>
</blockquote>
<p>If I run <em><strong>CUDA_VISIBLE_DEVICES=2,3 torchrun --nproc_per_node 2 train.py</strong></em>, there is no error.</p>
How to use Dataset class to load a multimodal dataset?

Menan: <p>Hi!</p>
<p>I am new to pytorch, in my research i have a multimodal task. Where i have to load face images and its respective landmarks as input to the model. Is there any resources i can refer to achieve this using Dataset class of pytorch?</p>

ptrblck: <p>The <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">Data loading tutorial</a> shows how to load face images with their corresponding landmarks.</p>

Menan: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ,<br>
Thank you, my task is multimodal, so inputs will be image and landmarks (which is X in ML terms) and out will be the yaw, pitch and roll (y vector). I did find way to make it work, by using dictionaries in the <strong>getitem</strong> method. Is it the standard approach for multimodal datasets?</p>

ptrblck: <p>Yes, <code>dict</code>s are often used for multiple outputs as also seen in the linked tutorial.</p>

Menan: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> i appreciate your help!</p>

pujaltes: <p>Wouldn’t returning a <code>dict</code> in the <code>__getitem__</code> method of a Dataset lead to this <a href="https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662" rel="noopener nofollow ugc">issue</a> of accumulating ref counts? If so, what can we do to avoid this when working on a multi-modal task that requires loading &gt;20 tensors of different sizes?</p>

ptrblck: <p>Might be, but you should double check it as the post is now ~3 years old and I’m unsure if it’s still the case.</p>
Torch multiprocessing dramatic slow down a few minutes

David_Brown: <p>I am trying to implement a simple producer/consumer pattern using torch multiprocessing with the SPAWN start method. There is one consumer, the main process, and multiple producer processes.</p>
<p>The consumer process creates a pytorch model with shared memory and passes it as an argument to the producers. The producers use the model to generate data objects which are added to a Queue which is also passed as an argument from the consumer.</p>
<p>After starting the producers, the consumer will wait for items from the queue and process them in batches.</p>
<p>This works just fine at the start, but dramatically slows down after only a few minutes. From what I can tell, the producers initially generate data very rapidly but then progressively slow down after each consumer batch processing step. RAM and GPU memory both stay lower than 50% so memory leaks are not the issue. GPU utilization is also very low. From what I can tell, some of the producers simply silently stop producing. I can see in the system monitor that at any given moment, most of the processes are sleeping.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d34e0194ab43acff874533af8fdee02ed6c09ab5.png" data-download-href="https://discuss.pytorch.org/uploads/default/d34e0194ab43acff874533af8fdee02ed6c09ab5" title="Screenshot from 2024-04-19 11-43-39"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d34e0194ab43acff874533af8fdee02ed6c09ab5_2_690x306.png" alt="Screenshot from 2024-04-19 11-43-39" data-base62-sha1="u9hSaAKlsSa5VZWptOWzjLvSqYB" width="690" height="306" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d34e0194ab43acff874533af8fdee02ed6c09ab5_2_690x306.png, https://discuss.pytorch.org/uploads/default/original/3X/d/3/d34e0194ab43acff874533af8fdee02ed6c09ab5.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/3/d34e0194ab43acff874533af8fdee02ed6c09ab5.png 2x" data-dominant-color="F4EEEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2024-04-19 11-43-39</span><span class="informations">949×421 72.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What could be going on here?</p>

David_Brown: <p>I resolved the issue. The producers were occasionally hanging due to a bug in the application code, and thus the number of stalled producers increased overtime. I assumed this was caused by deadlocks in multiprocessing, but was incorrect. Tip to others experiencing deadlocks: Add timeouts inside your producers to check if it is your own code that is stalling for non-concurrency related reasons.</p>
Length of an object in a 2D image

SystemLord: <p>Hi,</p>
<p>I’m trying to train a model to measure the length of an object in a 2D image.  The object could be in an unknown orientation but I’d like the length of one axis. I’ve done some reading, and it sounds like a decent architecture would be a U net (say pre-trained resnet101) followed by a fully connected layer then a layer with one output node.</p>
<p>Does that sound reasonable?</p>
<p>Thanks.</p>
LambdaLR scheduler steps once it is instantiated

bala: <p>The following code is self-explanatory.</p>
<pre><code class="lang-auto">model = nn.Linear(2, 3)

opt = torch.optim.Adam(model.parameters(), lr=1.0)

print("Before instantiation of lr scheduler", opt.param_groups[0]['lr'])

lrs = torch.optim.lr_scheduler.LambdaLR(opt, lr_lambda = lambda epoch: (epoch+1) * 0.5)

print("After instantiation of lr scheduler", opt.param_groups[0]['lr'])

print("After instantiation of lr scheduler, accessed through the method in lr_scheduler", lrs.get_last_lr())
</code></pre>
<p>The results I obtain are:</p>
<blockquote>
<p>Before instantiation of lr scheduler 1.0<br>
After instantiation of lr scheduler 0.5<br>
After instantiation of lr scheduler, accessed through the method in lr_scheduler [0.5]</p>
</blockquote>
<p>I am wondering why the learning rate got updated even before the call on step method of lr_scheduler.</p>