akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Graph Convolutional Neural Network Issues

cwoolfo1: <p>Hello I am trying to use a GCN layer to perform regression on some data.</p>
<p>My data has 100 nodes and 2 features per node. I tried to interconnect all the nodes to start, but for some reason PyTorch Geometric doesn’t like my edge index matrix.</p>
<p>I am getting the following error:</p>
<p>ValueError: <code>MessagePassing.propagate</code> only supports integer tensors of shape <code>[2, num_messages]</code>, <code>torch_sparse.SparseTensor</code> or <code>torch.sparse.Tensor</code> for argument <code>edge_index</code>.</p>
<p>What am I doing wrong?</p>
<pre><code class="lang-auto">def make_edge_index(node_size):
    rows = []
    columns = []
    for i in range(node_size):
        for j in range(node_size):
            rows.append(i)
            columns.append(j)
    rows = np.asarray(rows)
    columns = np.asarray(columns)
    index = np.zeros(shape=(2, node_size*node_size))
    index[0,:] = rows
    index[1,:] = columns
    return index

self.input_size = int(node_size/2)
self.output_size = 4
self.edge_matrix = make_edge_index(self.input_size)
self.GCN = geom_nn.conv.GCNConv(2, self.output_size)
self.linear_layer = nn.Linear(node_size, self.input_size)


x = torch.swapaxes(x, 1, 2)
x = x.float()
x = self.linear_layer(x)
# start by scaling down all possible hits using input layer
x = torch.swapaxes(x, 1, 2)
x = self.GCN(x, self.edge_matrix)

</code></pre>
How can I get the exact torch operator of each node?

kmaeng: <p>Dear experts,</p>
<p>I am trying to iterate through the nodes in a model with an interpreter pattern and find the exact operators used. However, not sure how I can do this. For example, if I put the below print statement for each node:</p>
<pre><code class="lang-auto">print(f"{n}, {n.op}, {n.target}")
</code></pre>
<p>where <code>n</code> is a node, I get something like below:</p>
<p><code>bert_pooler_activation, call_module, bert.pooler.activation</code></p>
<p>Here, <code>bert.pooler.activation</code> is in fact <code>torch.nn.Tanh</code>, but that information is not shown at all. Is there a way for me to figure out that it is a Tanh?<br>
I tried this and that but could not figure it out.</p>
<p>Any suggestions or pointers will be highly appreciated!<br>
Thank you</p>
PyTorch and Conv2d

prabu-github: <p>I am using conv2d with large filters and my program “hangs”. Specifically, I have two (1 channel) 2048 X 2048 images. My convolution kernel is 501x501 with 48 channels i.e. I am expecting a 1 X 48 X 1548 X 1548 tensor ax output. My code below “freezes” after “out = F.conv2d(input, weight)” . What am I doing wrong?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

device = 'cuda:0'
inputs = [torch.zeros((1, 1, 2048, 2048)).to(device)]*2
weight = torch.ones((48, 1, 501, 501), dtype=torch.float32).to(device)
for (i, input) in enumerate(inputs):
    out = F.conv2d(input, weight)
    print(out.shape)
    print(torch.argmax(out))
</code></pre>
<p>EDIT: The program finished after a many minutes!! I am still unable to understand the reason for this long processing time.</p>

ptrblck: <p>The code takes a few seconds on my system but the actual performance depends on the used hardware, lib versions etc. The conv layer most likely does not support an optimized algorithm for these shapes and a <code>matmul</code> approach would be taken.</p>

prabu-github: <p>Thank you for the response <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I understand that I maybe trying shapes that are not optimized. My confusion arose from the following:<br>
For i=0, the <code>out.shape</code> is printed fast, but the program seems to slow down after that.</p>
<p>Is there is a max kernel size for which optimized algorithms are available?</p>
Code stuck at cuda.synchrize()

amsword: <p>The code hits hanging issue and the following stack shows it is stuck at cuda.synchronize(). Is there any idea of what is going on?</p>
<p>pytorch = 1.9.0, cuda=11.1</p>
<pre><code class="lang-auto">(gdb) py-bt
Traceback (most recent call first):
  &lt;built-in method _cuda_synchronize of module object at remote 0x7f0b8bc0d360&gt;
  File "/opt/conda/lib/python3.8/site-packages/torch/cuda/__init__.py", line 446, in synchronize
    return torch._C._cuda_synchronize()
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/utils/timer.py", line 165, in start
    torch.cuda.synchronize()
  File "/opt/conda/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 2372, in forward
    load_optimizer_states=True,
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1563, in _call_impl
    &gt;&gt;&gt; net = nn.Sequential(l, l)
  File "/tmp/code/act/src/qd/opt/trainer.py", line 1531, in inner_loop_step
    loss_dict = self.model_engine(self.dict_data)
  File "/tmp/code/act/src/qd/opt/trainer.py", line 1672, in inner_loop
    losses /= self.gradient_accumulate
  File "/tmp/code/act/src/qd/opt/trainer.py", line 1758, in do
    self.optimizer.zero_grad()
  File "/tmp/code/act/src/qd/pipelines/uni_pipeline.py", line 1743, in do_train
    port=self.cfg.dist_url_tcp_port,
  File "/tmp/code/act/src/qd/pipelines/uni_pipeline.py", line 4443, in train
  File "/tmp/code/act/src/qd/pipelines/uni_pipeline.py", line 1540, in ensure_train
    arguments={'iteration': start_iter},
  File "/tmp/code/act/src/qd/pipeline.py", line 679, in pipeline_train_eval_multi
    pip.ensure_train()
  File "src/qd/qd_common.py", line 3349, in execute_func
  File "src/qd/qd_common.py", line 4338, in &lt;module&gt;
(gdb) 

</code></pre>

mrshenli: <p>Looks like there is a deadlock in the code. There could be multiple reasons:</p>
<ol>
<li>Running concurrent NCCL communications. NCCL only allows one CUDA communicator to access the device a a time. If, say pipeline and ZeRO in deepspeed launches concurrent comms, it could hang.</li>
<li>User-code comm deadlock. Suppose there are two processes, X and Y. X calls commA → <code>cuda.synchronize()</code> → commB, while Y calls commB → <code>cuda.synchronize()</code> → commA. It could deadlock, as both process requires the subsequent comm op after <code>cuda.synchronize()</code> to unblock.</li>
</ol>
<p>If you didn’t launch comm ops in your user code, could you please create an issue in deepspeed repo and discuss with deepspeed experts?</p>

aljipa: <p>Is there a built-in mechanism to help debug these communication deadlocks? E.g. with thread dumps we can see which thread is waiting/holding individual locks. Or is adding logging the only option?</p>

agu: <p>Maybe setting <code>TORCH_DISTRIBUTED_DEBUG=DETAIL</code> can help?<br>
<a href="https://pytorch.org/docs/master/distributed.html#torch-distributed-debug" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/master/distributed.html#torch-distributed-debug</a></p>

aljipa: <p>Thanks Andrew, I didn’t know this existed.</p>
Memory efficient repeat with shift

Lugi: <p>So my problem is as follows: I start with a tensor of shape (h, w), and a tensor of shape (n, 2) - which is guidance for n <code>torch.roll</code> operations. Then from these 2 tensors I want to costruct a (n, h, w) tensor, where the nth slice is the (h, w) input with the <code>torch.roll</code> operation applied (with the 2 arguments guiding how much to roll over each axis coming from the (n, 2) tensor).</p>
<p>Would there be any way to do this without allocating (h, w, n) memory? Just as <code>torch.expand</code> doesn’t allocate new memory, but rather returns a view of the original tensor.</p>

smth: <p>this would be really easy and intuitive with <code>torch.dim</code>, but it requires a PyTorch nightly version (hasn’t gotten to stable releases yet): <a href="https://github.com/facebookresearch/torchdim" class="inline-onebox">GitHub - facebookresearch/torchdim: Named tensors with first-class dimensions for PyTorch</a></p>
<p>Short of that, I think your best bet is to write a Numba (CPU) or Triton (GPU) kernel.</p>
Scaled_dot_product_attention, bf16, with my attn_mask

MFajcik1: <h1>
<a name="context-1" class="anchor" href="#context-1"></a>Context</h1>
<p>Hi, I am trying to move our model from triton’s flash attention to torch2 flash attention, to benefit from <code>torch.compile</code>!<br>
However the problem lies in attention mask. Our model uses attention biasing, which I need to integrate into <code>attn_mask</code> parameter. Our model is also autoregressive, and since <code>is_causal</code> and <code>attn_mask</code> can’t be combined, I integrated causal masking into <code>attn_mask</code>. Unfortunately, it seems that flash attention currently does not support float attention masking (<code>attn_mask</code> parameter)?</p>
<h1>
<a name="environment-info-2" class="anchor" href="#environment-info-2"></a>Environment info</h1>
<pre><code class="lang-auto">GPU: RTX 3090 (but I got the same error with our code on A100)
torch==2.0.1
</code></pre>
<h1>
<a name="minimal-example-3" class="anchor" href="#minimal-example-3"></a>Minimal Example</h1>
<pre><code class="lang-python">import torch
from torch.backends.cuda import SDPBackend, sdp_kernel
import torch.nn.functional as F

backend_map = {
    SDPBackend.MATH: {"enable_math": True, "enable_flash": False, "enable_mem_efficient": False},
    SDPBackend.FLASH_ATTENTION: {"enable_math": False, "enable_flash": True, "enable_mem_efficient": False},
    SDPBackend.EFFICIENT_ATTENTION: {
        "enable_math": False, "enable_flash": False, "enable_mem_efficient": True}
}
# doesn't work
# shows UserWarning: Both fused kernels do not support non-null attn_mask
# raises RuntimeError: No available kernel.  Aborting execution.
with torch.cuda.amp.autocast(enabled=True, dtype=torch.bfloat16):
    query = torch.randn(1, 10, 1024, 64).cuda()
    key = torch.randn(1, 10, 1024, 64).cuda()
    value = torch.randn(1, 10, 1024, 64).cuda()

    attn_mask = torch.randn(1, 10, 1024, 1024).cuda()
    causal_mask = ~torch.ones(query.shape[2], key.shape[2], dtype=torch.bool).tril(diagonal=0).to(attn_mask.device)
    attn_mask = attn_mask.masked_fill(causal_mask.view(1, 1, query.shape[2], key.shape[2]), -float("inf"))

    dropout_p = 0.0

    with sdp_kernel(**backend_map[SDPBackend.FLASH_ATTENTION]):
        result = F.scaled_dot_product_attention(query, key, value, attn_mask=attn_mask, dropout_p=dropout_p)
</code></pre>
<p>Is there any way to circumvent this limitation?</p>

Jawwad_Ahmed: <p>Hi just checking<br>
Have you find a solution to this as I am facing similar issue</p>
<p>thanks</p>

MFajcik1: <p>Hi, unfortunately no. I don’t think we can use torch’s flashattention this way yet <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

Jawwad_Ahmed: <p>Hi,<br>
Yes the torch flash attention only accepts null mask I think so we gotta wait till Pytorch adds the mask feature.</p>
<p>Thanks</p>

b30: <p>Hey <a class="mention" href="/u/mfajcik1">@MFajcik1</a><br>
Did you manage to overcome the issue somehow? Maybe there’s a third-party implementation of FlashAttention that supports arbitrary masking?</p>
Reducing computational overhead when caching data for the backward pass

aabati: <p>Hello, I am trying to optimize a custom variant for a recurrent layer. I made my initial implementation in Python with PyTorch and now I am  reimplementing it with the C++ frontend, hoping that I can get some speed improvements when wrapping the C++ implementation inside <code>torch.autograd.Function</code>. So far without success.</p>
<p>I found that the overhead results from preparing / packing some intermediate results for the backward pass. Without such steps, my C++ code runs about 10% faster than my PyTorch implementation in Python. Now, I am wondering what are some best practices that I could apply to avoid such overheads? For now, I am only working on the CPU, before moving forward with a GPU implementation.</p>
<p>In a first attempt I used std::vectors to keep track of the results, and at the end of my forward pass I used <code>torch::stack</code> or <code>torch::cat</code> to combine them to a single tensor. Since this is a lot of copying it is no surprise to me that C++ version is slower than the one in Python.</p>
<p>Next, I thought it would be a good idea to pre-allocate one big tensor to cache all my intermediate results. Below is a very basic layout that illustrates my attempt:</p>
<pre><code class="lang-auto">std::vector&lt;torch::Tensor&gt; forward_pass(torch::Tensor input, torch::Tensor weights, torch::Tensor state)
{   
    auto intermediate_results = torch::empty( { 5, input.size(0), input.size(1), state.size(1), state.size(2), state.size(3) });

    // Iterate over each time step
    for (int t = 0; t &lt; input.size(1); ++t)
    {
        intermediate_results.index_put_({0, t, Slice(), Slice(), Slice(), Slice()}, /* some data tensor for the first result */);

        // ...
    }

    return { output, new_state, intermediate_results };
}
</code></pre>
<p>I was hoping that I could avoid copying and merging my tensors by pointing directly to a pre-allocated memory tensor, but I guess I was a bit naive here. I wanted to start this thread asking for some tipps and tricks, and sharing some of your experiences on how to tackle problems like these. Maybe it isn’t even a good idea to pack the results into a single tensor in the first place… but I would be happy for any suggestions or ideas.</p>
Torch.utils.tensorboard adding images with step

Ilya_Kotlov: <p>Hi, I’m using this util of visualizing:</p>
<blockquote>
<p>from torch.utils.tensorboard import SummaryWriter</p>
</blockquote>
<p>for image visualization and I’ve noticed that as more images I right each step (which is in my case epoch), the step count and points are changed, i.e:</p>
<p>When about 15 epoch were finished, although I write each epoch fixed amount of same images,<br>
only steps 1,2,3,4,8, … (some more, doesn’t matter).<br>
Now after more then 30 epochs I’ve noticed that steps 4,8 disappeared and 3 and 11 are showed.</p>
<p>Before:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/6/f622e1edd6376baaa5fd6a470eaf43a96c9844b5.jpeg" alt="image" data-base62-sha1="z7qbYJcQIQFFRnK8UGeZ6hx5l6B" width="324" height="232"></p>
<p>Now:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/f/ef28f98b9232464de4194ed4dad6c532317a1420.png" alt="image" data-base62-sha1="y7HU162NahglnfwIsSl0IUwmDG8" width="297" height="214"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/d/adf51d8ec88af837c502fce786add7d0f5f29104.png" alt="image" data-base62-sha1="oOTL6OOVwxaGwRShN0mXk1ekAII" width="293" height="222"></p>
<p>I’ve only documented the the disappearing of 4, the two last images are consequent at the step bar.</p>
<p>How can I see all the steps ?<br>
Does Tensorboard swallows them and that is a bug and bad behavior or there is something in the settings of the external api (in the browser) or some argument in the code needed to be passed for not having this behavior ?<br>
Thank you in advance.</p>
<p>It happened only for images, for loss and accuracy and other point recordings I still see all the steps.</p>

Ilya_Kotlov: <p>Oh, I think that parameter is responsibale for it, correct me if I am wrong:</p>
<blockquote>
<p>max_queue</p>
</blockquote>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4db5c8cffd7790ed52cff20ff32b510562560852.png" alt="image" data-base62-sha1="b5seFjWa7evzkwBNJlKdex1hN4K" width="683" height="55"></p>

JinroTorch: <p>This is because by default Tensorboard samples some logged results, not all.</p>
<p>If you want to increment the number of sampled images just set the “samples_per_plugin” argument when loading the Tensorboard:</p>
<p>tensorboard --logdir &lt;your_directory&gt; --samples_per_plugin “images=&lt; number_of_images&gt;”</p>
<p>Example</p>
<p>tensorboard --logdir runs --samples_per_plugin “images=10000”</p>
<p>Note: Maybe the documentation say to set “0” to load all images. However, it will not according to this reported bug <a href="https://github.com/tensorflow/tensorboard/issues/5550" rel="noopener nofollow ugc">–samples_per_plugin “images=0” is documented to show all images, but shows no images at al</a></p>
Checkpoint should be CheckpointConfig in Ray Tune tutorial

Julius_Smith: <p>The <a href="https://pytorch.org/tutorials/beginner/hyperparameter_tuning_tutorial.html" rel="noopener nofollow ugc">Ray Tune tutorial</a><br>
says <code>from ray.air import Checkpoint, session</code>, which did not work for me, while<br>
<code>from ray.air import CheckpointConfig, session</code> seems to be correct (at least compiles).</p>

ptrblck: <p>Thanks for reporting! Could you create a GitHub issue so the code owners could track and fix it?</p>
RuntimeError: mat1 and mat2 must have the same dtype, but got Long and Float

pyt0rch: <p>This code attempts to train a neural network that converts one paragraph (input) into another (output), but I keep running into this same error.</p>
<pre><code class="lang-auto">class CharPredictor(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, output_size):
        super(CharPredictor, self).__init__()
        self.rnn = nn.LSTM(input_size, hidden_size, num_layers)
        self.fc = nn.Linear(hidden_size, output_size)
    
    def forward(self, x):
        # x = nn.functional.one_hot(x, num_classes=input_size).float()
        # x = x.transpose(0, 1)
        outputs, _ = self.rnn(x)
        prediction = self.fc(outputs[:, -1, :])
        return prediction

seq_length = 100

with open(input_filename, "r") as file:
    input_text = file.read()
input_chars = sorted(set(input_text))
input_char_to_index = {c: i for i, c in enumerate(input_chars)}
sequences = [[input_char_to_index[char] for char in input_text[i:i + seq_length]] for i in range(0, len(input_text) - seq_length, seq_length)]
sequences = torch.tensor(sequences)#.long()
# input_one_hot = nn.functional.one_hot(sequences, num_classes=len(input_chars))

with open(output_filename, "r") as file:
    output_text = file.read()
output_chars = sorted(set(output_text))
output_index_to_char = {c: i for i, c in enumerate(output_chars)}
labels = [output_index_to_char[char] for char in output_text[seq_length:]]
labels = torch.tensor(labels)#.long()
# output_one_hot = nn.functional.one_hot(labels, num_classes=len(output_chars))

input_size = len(input_chars)
hidden_size = 128
num_layers = 2
output_size = len(output_chars)

model = CharPredictor(input_size, hidden_size, num_layers, output_size)
optimizer = torch.optim.Adam(model.parameters())
criterion = nn.CrossEntropyLoss()

losses = []

for epoch in range(10):
    for i in range(len(sequences)):
        optimizer.zero_grad()
        prediction = model(sequences[i].unsqueeze(0))
        loss = criterion(prediction, labels[i])
        loss.backward()
        optimizer.step()

        losses.append(loss.item())
</code></pre>

vdw: <p>You probably want to add an <code>nn.Embedding</code> layer to map your character indices to vector representation of type <code>Float</code>.</p>
Can I perform jit.trace only on specific layers of a model?

JimW: <p>Hi,</p>
<p>I received an JIT version of encoder file from my colleague, and I was appending my own decoder layer (i.e. a simple MLP for classification purpose) on top of it, and I wrapped everything as one module.</p>
<blockquote>
<p>class Combined_Net(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Combined_net, self).<strong>init</strong>()<br>
self.encoder = torch.jit.load(“myfriends_model.pth”)<br>
self.decoder = torch.nn.Linear(in_feature =, out_feature =)<br>
def forward(self, x):<br>
inter_feat = self.encoder(x)<br>
output = self.decoder(inter_feat)<br>
return output</p>
</blockquote>
<p>The code looks like something like above. But the following command crashed:</p>
<p><code>model_ts = torch.jit.trace(my_combined_net, inputs_ts, check_trace=False)</code></p>
<p>This should be caused by the fact that I was performing jit.trace on something which was already traced.<br>
So how can I perform jit.trace only on the decoder part of my network, but generate one single model_ts with JIT of both encoder and decoder?</p>
<p>Or if I do jit.trace only on decoder part, how can I combine encoder_jit and decoder_jit into one module?</p>
<p>Thanks.</p>

ptrblck: <p>I’m unsure why it failed as a warning should be raised if you are trying to trace an already traced module again:</p>
<pre data-code-wrap="python"><code class="lang-python">model = nn.Linear(10, 10).cuda()
x = torch.randn(1, 10).cuda()
model = torch.jit.trace(model, x)
model = torch.jit.trace(model, x)
# UserWarning: The input to trace is already a ScriptModule, tracing it is a no-op. Returning the object as is.
</code></pre>
<p>With that being said, note that <code>TorchScript</code> is in maintenance mode and the recommended approach is to use <code>torch.compile</code> instead.</p>
How to detect if the GPUs are AMD's or NVIDIA's

f10w: <p>Hello,</p>
<p>I would like to know if there is a way to detect the type of the GPU in terms of manufacturer.</p>
<p><code>torch.cuda.get_device_capability('cuda')</code> gives <code>(8, 0)</code> for NVIDIA A100 and <code>(9,0)</code> for AMD MI250X. It seems that the result is also <code>(9,0)</code> for NVIDIA H100, so I’m not sure how to distinguish between NVIDIA and AMD.</p>
<p>Thank you in advance for your help!</p>
Behavior explain - parameters deepcopied from model 1 do not update in model 2 where they are part of the computation

sofuncheung: <p>Hi,</p>
<p>As the title suggests, if I deepcopy a model and use the parameters of the copied model to participate the computation of the original model, why are these copied parameters not being updated despite having requires_grad = True?</p>
<p>This is perhaps a desired behavior but I’d like to know how this is done.</p>
<p>Relevant code:</p>
<p>class IDModule(nn.Module):<br>
“”"Intrinsic dimensionality wrapper module…<br>
Takes in the network, a projector (a function(D,d)-&gt; projection LinearOperator),<br>
and the target intrinsic dimensionality.</p>
<pre><code>Example usage:
id_net = IDModule(net, lambda D,d: LazyRandom(D,d), 1000)
"""

def __init__(self, net, projector, dimension=1000):
    super().__init__()

    self.d = dimension
    self._forward_net = [net]
    initnet = deepcopy(net)
    for orig_name, orig_p in initnet.named_parameters():
        if orig_p.requires_grad:
            _delchainattr(net, orig_name)
    aux = [(n, p) for n, p in initnet.named_parameters() if p.requires_grad]
    self.names, self.trainable_initparams = zip(*aux)
    self.trainable_initparams = [param for param in self.trainable_initparams]
    self.names = list(self.names)
    self.D = sum([param.numel() for param in self.trainable_initparams])
    self.subspace_params = nn.Parameter(torch.zeros(self.d))
    self.P = projector(self.D, self.d, self.trainable_initparams, self.names)

def to(self, *args, **kwargs):
    self._forward_net[0].to(*args, **kwargs)
    self.trainable_initparams = [ 
        param.to(*args, **kwargs) for param in self.trainable_initparams
    ]
    return super().to(*args, **kwargs)

def forward(self, *args, **kwargs):
    flat_projected_params = self.P @ self.subspace_params
    unflattened_params = unflatten_like(
        flat_projected_params, self.trainable_initparams
    )
    iterables = zip(self.names, self.trainable_initparams, unflattened_params)
    for p_name, init, proj_param in iterables:
        p = init + proj_param.view(*init.shape)  # will init still require gradient?
        _setchainattr(self._forward_net[0], p_name, p)
        #Mark: poking here
        #print("-"*60)
        #print( init)
        # Mark: my experiments say init does require grad but its value doesn't change.

    return self._forward_net[0](*args, **kwargs)
</code></pre>
<p>In the above code, “init” are the trainable parameters from the copied model and that participate the computation of the original model, but I have checked they don’t update when the optimization was done w.r.t. the original model’s parameters (self.subspace_params to be precise). Considering the parameters of the original model are now basically (init + M·subspace_params) but only the latter ones get updated. Why this happens? Any help would be appreciated.</p>
Quantise the wav2vec2 model

mi_gupta: <p>I am trying to quantise the wav2vec2 model using static fx mode. by=ut encountering the error: ,</p>
<pre><code class="lang-auto">from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
from torch.quantization import quantize_fx
model_f = Wav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-base-960h")

from torch.quantization import quantize_fx
m = copy.deepcopy(model_f)
m.eval()
qconfig_dict = {"": torch.quantization.get_default_qconfig("fbgemm")}
# Prepare
model_prepared = quantize_fx.prepare_fx(m, qconfig_dict,example_inputs = torch.randn(1,1,54400))
# Calibrate - Use representative (validation) data.
with torch.inference_mode():
  for _ in range(10):
    x = torch.rand(1,1,54400)
    model_prepared(x)
# quantize
model_quantized = quantize_fx.convert_fx(model_prepared)
</code></pre>
<pre><code class="lang-auto">TypeError                                 Traceback (most recent call last)
&lt;ipython-input-26-0604642dc2ff&gt; in &lt;cell line: 10&gt;()
      8 qconfig_dict = {"": torch.quantization.get_default_qconfig("fbgemm")}
      9 # Prepare
---&gt; 10 model_prepared = quantize_fx.prepare_fx(m, qconfig_dict,example_inputs = torch.randn(1,1,54400))
     11 # Calibrate - Use representative (validation) data.
     12 with torch.inference_mode():

10 frames
/usr/local/lib/python3.10/dist-packages/transformers/models/wav2vec2/modeling_wav2vec2.py in _get_feature_vector_attention_mask(self, feature_vector_length, attention_mask, add_adapter)
   1151         batch_size = attention_mask.shape[0]
   1152 
-&gt; 1153         attention_mask = torch.zeros(
   1154             (batch_size, feature_vector_length), dtype=attention_mask.dtype, device=attention_mask.device
   1155         )

TypeError: zeros() received an invalid combination of arguments - got (tuple, device=Attribute, dtype=Attribute), but expected one of:
 * (tuple of ints size, *, tuple of names names, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)
 * (tuple of ints size, *, Tensor out, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)
</code></pre>
Help with debugging - ValueError: optimizer got an empty parameter list

promach: <p>For <a href="https://gist.github.com/buttercutter/6a5b90a987840831e71f98dcc723633a/ac99a5a4d448c3bfa943bd9b6d88bda35cc14d5b#file-nlhf-py-L534" rel="noopener nofollow ugc">this NLHF code</a>, I have a noob question where my eyes / mind could not catch where went wrong at this moment yet.</p>
<blockquote>
<p>No parameters in params.</p>
<p>Traceback (most recent call last):</p>
<p>File “/Users/john/Downloads/nlhf.py”, line 567, in </p>
<p>optimizer_current_policy = AdamW_on_Lion_Optimizer(</p>
<p>File “/Users/john/Downloads/nlhf.py”, line 534, in <strong>init</strong></p>
<p>self.adamW = optim.AdamW(params=params, lr=lr, betas=adam_betas,</p>
<p>File “/Users/john/.nlp/lib/python3.9/site-packages/torch/optim/adamw.py”, line 52, in <strong>init</strong></p>
<p>super().<strong>init</strong>(params, defaults)</p>
<p>File “/Users/john/.nlp/lib/python3.9/site-packages/torch/optim/optimizer.py”, line 261, in <strong>init</strong></p>
<p>raise ValueError(“optimizer got an empty parameter list”)</p>
<p>ValueError: optimizer got an empty parameter list</p>
</blockquote>
<p>Note: Before <code>params</code> is passed into <code>AdamW_on_Lion_Optimizer()</code>, <code>params</code> is a valid non-empty parameter list.</p>

ptrblck: <p>How did you create <code>params</code>?</p>

promach: <p>See lines 567 and 577 of the code</p>
<p>Using line 577 does not cause the problem, but line 567 has the above problem.</p>

promach: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> if you add the following to just before line 567 : <code>optimizer_current_policy = AdamW_on_Lion_Optimizer(                                      params=current_policy.parameters(),                                      lr=1e-3                                 )</code> , you will notice that <code>params</code> is a valid non-empty parameter list before it is passed into <code>AdamW_on_Lion_Optimizer()</code></p>
<pre><code class="lang-auto">     # Diagnostic code to check parameters
     if not list(current_policy.parameters()):
         print("No parameters in current_policy.")
     else:
         for idx, param in enumerate(current_policy.parameters()):
             print(f"Param {idx}: requires_grad={param.requires_grad}")
             if param.requires_grad:
                 print("There are parameters that require gradients.")
             else:
                 print("No parameters require gradients.")
</code></pre>

promach: <p><code>params</code> is not actually empty when before it is first being passed into the <code>AdamW_on_Lion_Optimizer()</code> init function as <code>current_policy.parameters()</code> , so I am really confused on why it became empty when it went inside the init function.</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> : I wonder if you have any idea / comments / suggestions OR experienced something similar in your past experience ?</p>
Optimal optimizer

Kallel_Mariem: <p>When dealing with a small dataset and a complex model, what optimizer should i choose ?</p>

J_Johnson: <p>SGD and lots of Dropout. SGD has demonstrated better generalizability, which is what you want. It just takes longer.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://arxiv.org/abs/1712.07628">
  <header class="source">
      <img src="https://arxiv.org/static/browse/0.3.4/images/icons/favicon-32x32.png" class="site-icon" width="" height="">

      <a href="https://arxiv.org/abs/1712.07628" target="_blank" rel="noopener nofollow ugc">arXiv.org</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/402;"><img src="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png" class="thumbnail" width="" height=""></div>

<h3><a href="https://arxiv.org/abs/1712.07628" target="_blank" rel="noopener nofollow ugc">Improving Generalization Performance by Switching from Adam to SGD</a></h3>

  <p>Despite superior training outcomes, adaptive optimization methods such as Adam, Adagrad or RMSprop have been found to generalize poorly compared to Stochastic gradient descent (SGD). These methods tend to perform well in the initial portion of...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Compliling C++ code, cuda 11.7, python 3.9, pytorch 1.13

Roqyah_bdeen: <p>This C++ code was compiling successfully on pytorch 1.4 cuda 10.1<br>
now I got this error while I’m trying to compile the same code on  pytorch 1.13 cuda 11.7<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/b/3bdd7741ac056ac3bfc6e69fec38aa2a13189e72.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/3bdd7741ac056ac3bfc6e69fec38aa2a13189e72" title="Untitled"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/b/3bdd7741ac056ac3bfc6e69fec38aa2a13189e72_2_690x411.jpeg" alt="Untitled" data-base62-sha1="8xAERKcjBc9VKiJycmeT27HvGZI" width="690" height="411" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/b/3bdd7741ac056ac3bfc6e69fec38aa2a13189e72_2_690x411.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/b/3bdd7741ac056ac3bfc6e69fec38aa2a13189e72_2_1035x616.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/b/3bdd7741ac056ac3bfc6e69fec38aa2a13189e72_2_1380x822.jpeg 2x" data-dominant-color="3E2437"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Untitled</span><span class="informations">1603×955 263 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e4190374e79fca4ce3b567ed3afbe9e31d5a441d.png" data-download-href="https://discuss.pytorch.org/uploads/default/e4190374e79fca4ce3b567ed3afbe9e31d5a441d" title="Untitled2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e4190374e79fca4ce3b567ed3afbe9e31d5a441d_2_690x319.png" alt="Untitled2" data-base62-sha1="wxQsaSQvqAGyGQuizlmeW9Q8deB" width="690" height="319" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e4190374e79fca4ce3b567ed3afbe9e31d5a441d_2_690x319.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e4190374e79fca4ce3b567ed3afbe9e31d5a441d_2_1035x478.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/4/e4190374e79fca4ce3b567ed3afbe9e31d5a441d.png 2x" data-dominant-color="FBFAF9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Untitled2</span><span class="informations">1235×571 223 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
what sould be the wrong here please? is it possible to compile this code with these new versions?</p>

harshal_Verma: <p>I’m trying to implement the same github repo, with the same configuration of 11.7. Any idea how to overcome this ?</p>
Model Validation Loss Zero

Fatih_Haslak: <p>I am working on multiclass classification using a custom dataset with four classes. The dataset consists of 4080 images with dimensions of 2040x2040, and the classes are balanced. I am using the Convnext_tiny model with ADAM optimization (weight_decay=0.01, lr=0.0001) and crossentropy loss. My training losses are as follows: Val_loss ≈ 0.0, F1 ≈ 1. The model performs well on the validation dataset but poorly on the test dataset. How can I address this issue?</p>
<p>Convnext fully connected layer -<br>
model=models.convnext_tiny(pretrained=True)<br>
num_ftrs = model.classifier[2].in_features  # Son fully connected katmanın giriş özellik sayısı</p>
<pre><code>        model.classifier[2] = nn.Sequential(
            nn.Linear(num_ftrs, 1000),
            nn.ReLU(),
            nn.LayerNorm(1000),
            nn.Linear(1000,4),
            
        )
</code></pre>

Fatih_Haslak: <p>My train dimension 1728*1728</p>
nn.DataParallel for testing the model

hiru: <p>I am training a vit using <code>nn.DataParallel</code> and I want to remove the effect when testing. Since I am calculating few matrices based on the predictions, I want the output to be in a single tensor and in a one gpu. Is there any way for mw to turn off the effect of nn.DataParallel? or does the model.eval() handel it without any extra help?</p>

ptrblck: <p><code>model.eval()</code> won’t help, but you could use the internal <code>.module</code> to skip the data parallel wrapping via <code>out = model.module(input)</code>.</p>

hiru: <p>worked. thank you ✺◟(＾∇＾)◞✺</p>
Using torch::nn::ModuleList(which includes different blocks)

DongkyuLee: <p>Hi</p>
<p>I have some problem implement model.</p>
<p>I push different blocks to torch::nn::ModuleList, and now I want to get blocks from the torch::nn::ModuleList and do forward function.</p>
<p>However, I have no idea how to do that.<br>
In python I just do as “x = module_list [i] (x)”.<br>
In c++ I tried to write as “x = module_list [i] (x)” and “x = module_list [i] -&gt;as&lt;???&gt;()-&gt;forward(x)”<br>
(It can’t be compile with first one, and I don’t know which data type should be taken at ??? with second one.)</p>
<p>Please let me know how to solve it.</p>
<p>Thanks for reading.<br>
Dongkyu</p>

jayandra_l: <p>Hi <a class="mention" href="/u/dongkyulee">@DongkyuLee</a></p>
<p>Did you solve this problem? Facing the same issue.</p>
Torch logging problem Warning

Chihoub_Chiheb_Eddin: <p>I have stumbled on this warning while running a github code:<br>
functional.py:507: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at …/aten/src/ATen/native/TensorShape.cpp:3549.)<br>
return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]<br>
Ps: I am using wsl ubuntu, my model does not loop to train unfortunately, while i have changed the mode of the files.<br>
Any suggestions ?</p>

ptrblck: <aside class="quote no-group" data-username="Chihoub_Chiheb_Eddin" data-post="1" data-topic="196943">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chihoub_chiheb_eddin/48/63089_2.png" class="avatar"> Chihoub_Chiheb_Eddin:</div>
<blockquote>
<p>Ps: I am using wsl ubuntu, my model does not loop to train unfortunately, while i have changed the mode of the files.</p>
</blockquote>
</aside>
<p>Could you describe which mode you have changed of which files and why?</p>

Chihoub_Chiheb_Eddin: <p>thank you for your input, I have solved it by updating my environment and the model was not looping because I have mixed the parameters. However, I have a bigger (original) problem now, when I upload my checkpoint file, it does not work, I can test with its points yes, but can’t fine tune. every time it gives me a new error.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/6/a66c9adae5f94d51a2c08d18a33863601e66bcf8.png" data-download-href="https://discuss.pytorch.org/uploads/default/a66c9adae5f94d51a2c08d18a33863601e66bcf8" title="problem"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a66c9adae5f94d51a2c08d18a33863601e66bcf8_2_270x500.png" alt="problem" data-base62-sha1="nKfWAFZ6b0YRMXDV8wNHUEeCtKM" width="270" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a66c9adae5f94d51a2c08d18a33863601e66bcf8_2_270x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a66c9adae5f94d51a2c08d18a33863601e66bcf8_2_405x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a66c9adae5f94d51a2c08d18a33863601e66bcf8_2_540x1000.png 2x" data-dominant-color="303131"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">problem</span><span class="informations">1239×2286 181 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Torchaudio pipeline consistency

Lebourdais: <p>I’m trying to use the multiple Wav2Vec versions with the option of switching seamlessly between one and another.<br>
The “WAVLM_BASE” and “WAVLM_LARGE” version are supposed to be similar but don’t yield the same class.<br>
The WavLM_Base give a <code>Wav2Vec2Model</code> whereas the WavLM_Large give a <code>_Wav2Vec2Model</code>.</p>
<pre><code class="lang-auto">bundle_base = getattr(torchaudio.pipelines, "WAVLM_BASE")
base_model = bundle_base.get_model()
bundle_large = getattr(torchaudio.pipelines, "WAVLM_LARGE")
large_model = bundle_large.get_model()
assert type(base_model) == type(large_model)
</code></pre>
<p>Is this an expected result ?</p>
LSTM h_0 & c_0 storage when using One by one forward step

jodunkel: <p>Hello everyone,<br>
I would like to create a real-time anomaly checker for a sensor. To do this, I initialise a <code>self.h_0</code> &amp; <code>self.c_0</code> in <code>__init__</code> and with each <code>forward</code> step I save the state in it (very similar to that: <a href="https://discuss.pytorch.org/t/how-to-export-real-time-capable-lstm-to-onnx">https://discuss.pytorch.org/t/how-to-export-real-time-capable-lstm-to-onnx</a>). But now I have the problem that at the beginning of each new time series the old values of the previous series are stored in <code>self.h_0</code> &amp; <code>self.c_0</code>. For this reason, I always get different results when I run through the same data several times.  Is there a good solution for this Problem?</p>

jodunkel: <p>I can use an <code>if init_hidden:</code> in <code>def forward(self, input, init_hidden=False):</code> that I call at the beginning of a new time series. However, I have the problem that I can then no longer <code>torch.jit.trace()</code> the network.</p>

ptrblck: <p><code>torch.jit.trace</code> does not support data-dependent control flow or any other conditions, while <code>torch.jit.script</code> did. However, note that <code>TorchScript</code> in general is in maintenance mode and the current recommendation is to use <code>torch.compile</code> instead.</p>

jodunkel: <p>I have now solved it by outputting h_0 and c_0 and reading them back in as input. But I only have to do this if I feed it with live data. For training, I can read in the normal way, as a time series blog with many points in time. I don’t think this is the best solution, but it gives me the most freedom over h_0 &amp; c_0.</p>
<p>I want to use the net later for IOS Devises. So I have to rely on <code>torch.jit.trace</code> or <code>torch.jit.script</code> or is it also possible with <code>torch.compile</code>?</p>

ptrblck: <p>I don’t know if <code>torch.compile</code> is supported on iOS devices but would guess <code>ExecuTorch</code> would be the right approach to deploy on these devices. CC <a class="mention" href="/u/marksaroufim">@marksaroufim</a> as he would know more about the support.</p>

jodunkel: <p>Oh nice, I have now used <code>CoreMLTools</code> to get it onto the device. But I think it’s worth having a look at <code>ExecuTorch</code>.<br>
Looks really interesting, I can see the cross-platform benefit. But is there an advantage in performance, especially compared to Apple’s own <code>CoreML</code>?<br>
Do benchmarks already exist?</p>
Training speed slowdown problem when using gpu

JaehwanOh: <p>When I’m training using univariate time-series data, when I use gpu in the SamePadConv layer, I get 1.33 seconds and use cpu in the SamePadConv layer, I get 0.0002 seconds.</p>
<pre><code class="lang-auto">class SamePadConv(Module):
    def __init__(self, in_channels, out_channels, kernel_size, dilation=1, groups=1):
        super().__init__()
        self.receptive_field = (kernel_size - 1) * dilation + 1
        padding = self.receptive_field // 2
        self.conv = nn.Conv1d(
            in_channels, out_channels, kernel_size,
            padding=padding,
            dilation=dilation,
            groups=groups
        )
        self.remove = 1 if self.receptive_field % 2 == 0 else 0
        
    def forward(self, x):
        out = self.conv(x)
        if self.remove &gt; 0:
            out = out[:, :, : -self.remove]
        return out
</code></pre>
<p>As for as I know, if I use gpu, it should be faster, so why is this cause occurring?</p>
Dynamic Quantization produces inconsistent outputs

aivatoglou: <p>Dear PyTorch community,</p>
<p>I recently used PyTorch’s dynamic quantization technique to quantize the weights of a fine-tuned roberta-large-mnli model loaded from Hugging Face. I followed the steps outlined in PyTorch’s example notebook (<a href="https://pytorch.org/tutorials/intermediate/dynamic_quantization_bert_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">(beta) Dynamic Quantization on BERT — PyTorch Tutorials 2.2.0+cu121 documentation</a>).</p>
<p>Specifically, the quantization was applied exclusively to nn.Linear layers, with a dtype of qint8.</p>
<p>However, a challenge has emerged: the output logits of the quantized model seem to be affected by the batch size. Notably, the output for a single example differs depending on whether the example is part of a batch or processed individually.</p>
<p>I would like to highlight that, for inference, I loaded the base model from Hugging Face, applied dynamic quantization, and then loaded the state dict with the quantized weights.</p>
<p>Is this behavior something expected, or am I overlooking a crucial aspect of quantization?</p>
<p>Thank you.</p>

HDCharles: <p>Yeah that’s expected. Basically dynamic quantization looks at the tensor you give it and calculates the min and max values of all elements and uses that to quantized the activation. As the range gets larger, it quantizes each individual value less accurately. If you have outliers, this can affect things massively</p>
<p>You can get around this static quantization to a point, but then you doin’t handle non-static distributions as well.</p>
Could not run 'quantized::conv2d.new'

joepareti54: <p>this <a href="https://github.com/joepareti54/joepareti54/blob/main/quantized-5.ipynb" rel="noopener nofollow ugc">code</a> uses resnet and tries to quantize a CNN. The exception is ‘NotImplementedError: Could not run ‘quantized::conv2d.new’ with arguments from the ‘CPU’ backend.’</p>

james5: <p>What you are trying to do is process images (that are torch.float32 on CPU device/backend) with quantized model (with all operations quantized to torch.qint8(? or quint8 I don’t remember) on QuantizedCPU device/backend).<br>
I believe that modifying model is necessary - especially <a href="https://github.com/pytorch/vision/blob/main/torchvision/models/quantization/resnet.py#L106" rel="noopener nofollow ugc">entry</a> and <a href="https://github.com/pytorch/vision/blob/main/torchvision/models/quantization/resnet.py#L111" rel="noopener nofollow ugc">exit</a> points. You can use PyTorch’s implementation of ResNet: <a href="https://github.com/pytorch/vision/blob/main/torchvision/models/quantization/resnet.py#L111" rel="noopener nofollow ugc">vision/torchvision/models/quantization/resnet.py at main · pytorch/vision · GitHub</a>.</p>
<p>Therefore changing single line 24. to:</p>
<blockquote>
<p>model = torchvision.models.quantization.resnet18(pretrained=True)</p>
</blockquote>
<p>should be enough. Although I would check how they are fusing model.</p>

HDCharles: <p>you get that error if you do eager mode quantization without adding quant stubs to the model</p>
<p>see <a href="https://pytorch.org/docs/stable/quantization.html#post-training-static-quantization" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 2.2 documentation</a></p>
Why does modules fusion replace fused modules by nn.Identity?

soducoe: <p>My question is simple. Why replacing the fused modules by <code>nn.Identity</code> rather than simply deleting them in the function <code>fuse_known_modules</code> (<a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/fuse_modules.py#L37" rel="noopener nofollow ugc">see code</a>)?</p>
<p>The profiler shows that it introduces some non-negligible delay and kinds of suppresses the performance improvement gained by fusion. As you can see in the tracing, the overhead of the two <code>nn.Modules: Identity</code> is almost equivalent of the latency of <code>aten::batch_norm</code> in the original fp32 model, which cancels the improvement done by fusing conv1d with batch_norm.</p>

HDCharles: <ol>
<li>usually people either jit script the model or export it to get a final maximally performant version. This would remove the nn.Identity</li>
<li>if you delete random modules your model will error.</li>
</ol>
<p>if you have a model like</p>
<p>self.lin = nn.Linear<br>
self.bn = nn.BatchNorm</p>
<p>forward:<br>
y = self.bn(self.lin(x))<br>
return y</p>
<p>deleting self.bn doesn’t just remove the identity since the model doesn’t know where to send the activation after self.lin</p>
<ol start="3">
<li>it makes debugging a lot easier. We normally debug quantization by comparing activation for each module output for both the quantized and non-quantized model. If you do a fusion like</li>
</ol>
<p>location 1: A<br>
location 2: B</p>
<p>fused into</p>
<p>location 1: A+B<br>
location 2: identity</p>
<p>then you can compare the output of location 2 for both models and get an apples to apples comparison. If you compare location 1, for the top case, you’d get the output of A while for the bottom case you’d get the output of B. If you delete location 2 there’s no place that has the same location and same operations.</p>