akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is there a way to set M1 mps fallback environment variable in-code?

Sam_Lerman: <p>I want to automatically set <code>'PYTORCH_ENABLE_MPS_FALLBACK=1'</code> environment variable.</p>
<p>I tried</p>
<pre><code class="lang-auto">import os
os.environ['PYTORCH_ENABLE_MPS_FALLBACK'] = '1'
</code></pre>
<p>It didn’t work.<br>
Is there a direct way to set it in Pytorch, e.g. <code>torch.backends.mps.enable_fallback()</code>.</p>

ptrblck: <p>I would recommend exporting environment variables directly to your terminal as they are often set too late in the Python script and this won’t be read again.</p>

Sam_Lerman: <p>I’m unsure what you mean. What won’t be read again?</p>

ptrblck: <p>Certain environment variables are read only during the import of a library and often users set them too late, which can cause confusion.<br>
For this reason, I would not recommend setting any environment variables inside the Python script but to properly <code>export</code> them in your terminal before launching the Python process.</p>

tonystevenj: <p>This is helpful, thanks!</p>
Does torch.tensor([1, 2, 3, 4]) copy data twice?

youkaichao1: <p>I’m playing with different ways of converting a Python list of integer to a PyTorch array. The normal way is <code>torch.tensor([1, 2, 3, 4], dtype=torch.int64)</code>. However, I find that using Python’s array library can be much faster: <code>torch.frombuffer(array("q", [1, 2, 3, 4]))</code>.</p>
<p>Copy from <a href="https://github.com/vllm-project/vllm/issues/6879#issuecomment-2261745995" class="inline-onebox" rel="noopener nofollow ugc">[Performance]: use Python array to replace Python list for zero-copy tensor creation · Issue #6879 · vllm-project/vllm · GitHub</a> :</p>
<p>Performance micro benchmark, just for how to move a Python list to pytorch tensor:</p>
<pre data-code-wrap="python"><code class="lang-python">import array
import torch

# print header
print("N\tlist\tlist_with_array\tarray")

for N in [100, 1000, 10000, 100000, 1000000]:
    list_data = list(range(N))
    array_data = array.array('q', list_data)

    def create_from_list():
        tensor = torch.tensor(list_data, dtype=torch.int64)
        return tensor

    def create_from_list_with_array():
        tensor = torch.frombuffer(array.array("q", list_data), dtype=torch.int64)
        return tensor

    def create_from_array():
        tensor = torch.frombuffer(array_data, dtype=torch.int64)
        return tensor

    import time

    for _ in range(10):
        # warmup
        create_from_list()
    start = time.time()
    for _ in range(100):
        create_from_list()
    elapsed_list = (time.time() - start) / 100 * 1000 # ms

    for _ in range(10):
        # warmup
        create_from_list_with_array()
    start = time.time()
    for _ in range(100):
        create_from_list_with_array()
    elapsed_list_with_array = (time.time() - start) / 100 * 1000 # ms

    for _ in range(10):
        create_from_array()
    start = time.time()
    for _ in range(100):
        create_from_array()
    elapsed_array = (time.time() - start) / 100 * 1000 # ms

    print(f"{N}\t{elapsed_list:.3f}\t{elapsed_list_with_array:.3f}\t{elapsed_array:.3f}")
</code></pre>
<div class="md-table">
<table>
<thead>
<tr>
<th>N</th>
<th>torch.tensor(list)</th>
<th>torch. frombuffer(array(list))</th>
<th>torch.frombuffer(array)</th>
</tr>
</thead>
<tbody>
<tr>
<td>100</td>
<td>0.005</td>
<td>0.002</td>
<td>0.001</td>
</tr>
<tr>
<td>1000</td>
<td>0.030</td>
<td>0.014</td>
<td>0.001</td>
</tr>
<tr>
<td>10000</td>
<td>0.278</td>
<td>0.137</td>
<td>0.001</td>
</tr>
<tr>
<td>100000</td>
<td>2.684</td>
<td>1.347</td>
<td>0.001</td>
</tr>
<tr>
<td>1000000</td>
<td>26.654</td>
<td>13.759</td>
<td>0.001</td>
</tr>
</tbody>
</table>
</div><p>If this is the case, it means <code>torch.tensor(list)</code> actually copies twice, right?</p>
Is it a good idea to use float16/bfloat16 for inference?

YuA: <p>Hi</p>
<p>I am currently using bfloat16 for inference and the model seems to be performing well (with torch.amp.autocast).<br>
Is it a common practice to use reduce precision for inference as well as for training? I am not confident enough because most articles about reduced precision are focusing on training.</p>

ptrblck: <p>Yes, lower dtypes can be used for inference, too.</p>

YuA: <p>Thank you for the clear answer!</p>
Best way to get inferences for a dataset that is too large for memory

azaa: <p>I have a dataset in the form of several dozen parquet files, each of which has around 2M records. In total, I have several hundred million records, each of which need a bit of preprocessing before being input to the model.</p>
<p>I also have a trained torch_geometric model that can run on CPU. Inference time on a batch of 2048 records is about 5ms for GPU, and 20ms for CPU. For reasons related to eventual deployment, it’s preferable to work with CPU for inference.</p>
<p>What is the fastest way that I can get predictions for the entire dataset? Naively, I would just loop exactly as I did during training, store results in a cache, and - once the cache is large - save the results so far, flush the cache, and continue.</p>
<p>That said, given the light weight of the model, I imagine there is a more efficient way to do this, e.g. distributed processing or multiprocessing. But I’m not sure where to start - would appreciate any suggestions <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
How can I generate overfitting on noisy regression data using PyTorch?

Orleizera: <p>How can I generate overfitting on noisy regression data using PyTorch? Despite various attempts, PyTorch tends to generalize the data, making it challenging to memorize (overfit) the data. I modified weight initialization, used batch_size=1, made the learning rate flexible, and increased the number of layers and parameters, but it still generalizes. Are there any parameters to be changed?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/0/401cd1fb2b86f306fee233220ba53cb6c69ec427.png" data-download-href="https://discuss.pytorch.org/uploads/default/401cd1fb2b86f306fee233220ba53cb6c69ec427" title="Figure_2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/401cd1fb2b86f306fee233220ba53cb6c69ec427_2_690x259.png" alt="Figure_2" data-base62-sha1="99aiV8gTnjvfKCGFCFLNvHNpbsb" width="690" height="259" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/401cd1fb2b86f306fee233220ba53cb6c69ec427_2_690x259.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/401cd1fb2b86f306fee233220ba53cb6c69ec427_2_1035x388.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/401cd1fb2b86f306fee233220ba53cb6c69ec427_2_1380x518.png 2x" data-dominant-color="FDFDFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Figure_2</span><span class="informations">2560×963 55.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre data-code-wrap="class"><code class="lang-class">    def __init__(self):
        super().__init__()
        inSize = 1
        layers = []
        for h in hiddenSizes:
            layers.append(nn.Linear(inSize, h))
            layers.append(nn.ELU())  
            inSize = h   
        layers.append(nn.Linear(inSize, 1))          
        self.rede = nn.Sequential(*layers)
        #self.apply(self._init_weights)
    def _init_weights(self, m):
        if isinstance(m, nn.Linear):
            #nn.init.uniform_(m.weight, a=-2.0, b=2.0)
            #nn.init.kaiming_normal_(m.weight, mode='fan_in', nonlinearity='relu')
            #nn.init.xavier_uniform_(m.weight)
            nn.init.normal_(m.weight, mean=0.0, std=1.)
            if m.bias is not None:
                #nn.init.uniform_(m.bias, a=-2.0, b=2.0)
                #nn.init.zeros_(m.bias)
                nn.init.normal_(m.bias, mean=0.0, std=1.0)
    def forward(self, x):          
        return self.rede(x)

hiddenSizes = [500, 500 ,250 , 250, 125] # Layers and Parameters

lossFn = nn.MSELoss()
dataloader = DataLoader(dataset, batch_size=1, shuffle=True) # I tried other values
optimizer = torch.optim.Adam(model.parameters(), lr = .01)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=1300, gamma=.9)
# scheduler = CyclicLR(optimizer, base_lr=0.0001, max_lr=.01, step_size_up=200, mode='triangular')
</code></pre>
Testing the relative accuracy of specific outputs

johnny99: <p>Hi all,</p>
<p>big fan, long time reader, first time poster.</p>
<p>I have a general question regarding checking the accuracy of a model besides looking at the test loss and was hoping someone might have some ideas or references.</p>
<p>I’m using a simple NN as surrogate model to approximate charged particles in an electromagnetic field. My input and output vectors are phase space vectors (x, y, z, v_x, v_y, v_z) at two positions in space and are generated using a more or less conventional particle tracker. After every training epoch, besides looking at the test loss (MSELoss), I would also like to calculate the relative accuracy of every output (mean accuracy of x = ? %) so I’m using the dataloader to compute the mean relative error of batches and finally of the whole dataset.</p>
<p>Since the relative error is something like (x_out - x_pred)/x_out there are cases where x_out is very small or near 0 resulting in some numerical issues after the division. I can pinpoint this problem by setting shuffle=False: this leads to some few batches looking really bad in term of prediction accuracy (the initial dataset is more or less sorted which will lead to specific clusters being near 0) which then skews the overall result.</p>
<p>I’ve been able to ameliorate this by using z-score instead of scaling to max when generating my torch.utils.data.Dataset but the problem is still there to some extent.</p>
<p>Is there a better way to do this besides clipping the outliers or will this even out or iron itself out during later epochs? I feel that there is a simple solution for this but just can’t think of it.</p>
<p>Thanks in advance</p>

Soumya_Kundu: <p>Would definitely try a few more epochs first.</p>
<p>Otherwise, would you potentially benefit from log-based scores? Or some absolute error instead of a relative error?</p>

johnny99: <p>Thanks for the reply! I’ll give it some epochs, since after a (manual) run through datapoints I realized that the model is converging pretty well.</p>
<p>I had in fact considered using log scaling initially but since I have negative numbers and didn’t want to offset everything I decided against it. I’ll probably try a second model using symlog and see if it’s better.</p>
<p>Regarding the absolute error I think I kind of already get that by looking at the test loss. If I’m using MSELoss and z-score a loss of let’s say 0.01 (after sqrt and dividing by sqrt(n=6) to get to RMS) leads to 0.04 which means that on average my absolute error is about 4 % of the standard deviation of my specific dataset.</p>
Should forward() be banned from ModuleList with FSDP?

anonymous_user_83259: <p>I sometimes subclass ModuleList with a forward() implementation for various purposes. For example, this implements a residual sequential:</p>
<pre><code class="lang-auto">from torch import nn

class RS(nn.ModuleList):
    def __init__(self, *layers):
        super().__init__(layers)
    def forward(self, x):
        for l in self: x = x + l(x)
        return x
</code></pre>
<p>In torch 2.4, I receive a warning for using such layers in FSDP:</p>
<pre><code class="lang-auto">from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
FSDP(RS(nn.Linear(8,8)))
# UserWarning: FSDP will not all-gather parameters for containers that do not implement forward: RS(
#   (0): Linear(in_features=8, out_features=8, bias=True)
# )
</code></pre>
<p>Should subclassing ModuleList be considered an unsupported anti-pattern? Or is it more likely that this warning will disappear in the future with subclassed containers?</p>

agu: <p>Good point. I guess we do not know that your subclass of <code>nn.ModuleList</code> will override and implement <code>forward()</code>. Would it be okay to just ignore the warning?</p>
<p>The important thing is that modules that do not implement <code>forward</code> should not be wrapped with FSDP, but your subclass does implement it.</p>

anonymous_user_83259: <p>In that case, I will ignore the warning. Thank you for the response.</p>
Cannot import name 'ConvBNReLUFusion' from 'torch.quantization.fx.fusion_patterns'

Ardeal: <p>Hi,<br>
I am trying to run the code <code>https://github.com/ModelTC/MQBench</code>.<br>
In the code, <code>from torch.quantization.fx.fusion_patterns import ConvBNReLUFusion, ModuleReLUFusion</code> code report the error:</p>
<p><code>ImportError: cannot import name 'ConvBNReLUFusion' from 'torch.quantization.fx.fusion_patterns'</code></p>
<p>the version of cuda and torch I am using is: <code>torch.__version__ '2.3.1+cu118</code></p>
<p>I do see the code and comments in <code>fusion_pattern.py</code> in pytorch as follows:</p>
<h1><a name="flake8-noqa-f401-1" class="anchor" href="#flake8-noqa-f401-1"></a>flake8: noqa: F401</h1>
<pre data-code-wrap="python"><code class="lang-python">
r"""
This file is in the process of migration to `torch/ao/quantization`, and
is kept here for compatibility while the migration process is ongoing.
If you are adding a new entry/functionality, please, add it to the
appropriate files under `torch/ao/quantization/fx/`, while adding an import statement
here.
"""
from torch.ao.quantization.fx.fuse_handler import DefaultFuseHandler, FuseHandler
</code></pre>
How to profile a long-running server using `torch profiler`?

sleepwalker2017: <p>I can see torch profiler usage, it traces the execution of LLM inference.</p>
<p>But it only generate report when the inference is done.</p>
<p>I want to use it to profile vllm server, this server is launched and will not exit automatically.</p>
<p>Can torch profiler used to profile this kind of situation?</p>
<p>I mean, the server is always running, and I want to profile a duration of it.</p>
How to capture and use cross relation of target variables in neural networks

Ali_geo: <p>Dear all,</p>
<p>I want to develop a neural network with 26 input features and 20 target variables. I used pytorch <code>1.12.1</code> to construct the network. Now, I want the model to also include any possible relation between the target variables. This is how I constructed the network:</p>
<pre><code class="lang-auto">class NN(nn.Module):
    def __init__(self):
        super(NN, self).__init__()
        self.fc1 = nn.Linear(n_features, 86)  # Input layer to hidden layer
        self.norm1 = nn.BatchNorm1d(86)
        self.fc2 = nn.Linear(86, 32)  # Hidden layer to hidden layer
        self.norm2 = nn.BatchNorm1d(32)
        self.fc3 = nn.Linear(32, 32)  # Hidden layer to hidden layer
        self.norm3 = nn.BatchNorm1d(32)
        self.fc4 = nn.Linear(32, output_size)
        self.cross_layer = nn.Linear(output_size, output_size) # detecting the possible relation between the target variables

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = self.norm1(x)
        x = torch.relu(self.fc2(x))
        x = self.norm2(x)
        x = torch.relu(self.fc3(x))
        x = self.norm3(x)
        x = self.fc4(x)
        x = self.cross_layer(x)
        return x
</code></pre>
<p>In the model I have a <code>cross_layer</code> layer but it is not making any difference in the prediction. Then, how can I properly construct a neural network that can leverage any possible relation between target variables?<br>
I do appreciate any help.<br>
Cheers</p>
Bool indexing not working on mps

Amin_Jun: <p>Hello Torch Team,</p>
<p>I found un-expected behavior in my code when running on <code>mps</code> (Apple Silicon). Here’s the test case:</p>
<pre><code class="lang-auto">import torch 

a = torch.Tensor([[0, 0, 0], [1, 1, 1]]).bool().to('mps')
print(a[a])
print(a[a.cpu()])
</code></pre>
<p>I assume the outputs should not be different, but they are:</p>
<pre><code class="lang-auto">tensor([False,  True,  True], device='mps:0')
tensor([True, True, True], device='mps:0')
</code></pre>
<p>Here’s my versions:</p>
<pre><code class="lang-auto">Python 3.8.19
torch==2.1.2
</code></pre>
<p>Am I doing something wrong or am I missing something? I’d appreciate your input very much!</p>

ptrblck: <p>Do you see the same difference using the latest PyTorch release? If so, could you create a GitHub issue, please?</p>
Libtorch 2.2.0+ options

drewjenks01: <p>I noticed that after version 2.1.2 of libtorch there is no more support for the “shared-without-deps” version of the library. I need this version of libtorch since when i use the “shared-with-deps” version with bazel the cuda deps conflicts with the local ones that i need to use but this doesnt happen with the other version. How can i either adapt the shared-with-deps version or else create my own shared-without-deps version in order to fix this? Thanks!</p>
How can I trace the back propagation from loss to inputs?

JimW: <p>I am trying to train a tiny network <code>my_layer</code> which will generate an output tensor <code>g</code> ranging from 0 to 1 to control other part of numpy based error calculation.</p>
<blockquote>
<p>g = my_layer(input_tensor)  # This part is standard PyTorch setup I am familiar with<br>
numpy_error = numpy_evaluation_module(g)</p>
</blockquote>
<p>I need to treat <code>numpy_error</code> as my loss and back propagate gradients all the way back to <code>input_tensor</code>.</p>
<p>I guess I need to modify <code>numpy_evaluation_module</code> to make it a PyTorch tensor. But my observation is that, the loss value did not change at all in my case. So I guess there might be some broken places along my back propagate chain and something was not differentiable.</p>
<p>Could someone suggest some potential places for me to check? Is there any recommended method to track the back propagation flow to quickly identify where the broken variable/tensor was?</p>

ptrblck: <p>You should have seen an error pointing to a detached computation graph. If that’s not the case check if you have created a new tensor with <code>requires_grad=True</code> which will create a new computation graph in subsequent operations. In this case you could write a custom <code>autograd.Function</code> for the numpy operations or could also use <code>torch.compile</code> in recent PyTorch builds which can translate numpy operations to PyTorch.</p>
Torchvision not using pillow-simd when its installed

bseebs87: <p>dependencies = [<br>
“numpy&lt;2; sys_platform==‘win32’”,<br>
“numpy; sys_platform==‘linux’”,<br>
“pillow-simd”,<br>
“python-magic-bin==0.4.14; sys_platform==‘win32’”,<br>
“python-magic; sys_platform==‘linux’”,<br>
“pyyaml”,<br>
“scikit-learn”,<br>
“tensorboard”,<br>
“torch==2.0.0”,<br>
“torchvision==0.15.1”,<br>
“torch-tb-profiler”,<br>
“tqdm”,<br>
“typer”,<br>
“watchdog”,<br>
“matplotlib”</p>
<p>Successfully installed MarkupSafe-2.1.5 absl-py-2.1.0 certifi-2024.7.4 charset-normalizer-3.3.2 click-8.1.7 cmake-3.30.1 contourpy-1.2.1 cycler-0.12.1 filelock-3.15.4 fonttools-4.53.1 grpcio-1.65.1 idna-3.7 importlib-metadata-8.2.0 importlib-resources-6.4.0 jinja2-3.1.4 joblib-1.4.2 kiwisolver-1.4.5 lit-18.1.8 markdown-3.6 markdown-it-py-3.0.0 matplotlib-3.9.1 mdurl-0.1.2 mpmath-1.3.0 networkx-3.2.1 numpy-1.26.4 nvidia-cublas-cu11-11.10.3.66 nvidia-cuda-cupti-cu11-11.7.101 nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-8.5.0.96 nvidia-cufft-cu11-10.9.0.58 nvidia-curand-cu11-10.2.10.91 nvidia-cusolver-cu11-11.4.0.1 nvidia-cusparse-cu11-11.7.4.91 nvidia-nccl-cu11-2.14.3 nvidia-nvtx-cu11-11.7.91 packaging-24.1 pandas-2.2.2 <strong>pillow-10.4.0</strong> <strong>pillow-simd-9.0.0.post1</strong> protobuf-4.25.4 pygments-2.18.0 pyparsing-3.1.2 python-dateutil-2.9.0.post0 python-magic-0.4.27 pytz-2024.1 pyyaml-6.0.1 requests-2.32.3 rich-13.7.1 scikit-learn-1.5.1 scipy-1.13.1 shellingham-1.5.4 six-1.16.0 spidops-0.1.0 sympy-1.13.1 tensorboard-2.17.0 tensorboard-data-server-0.7.2 threadpoolctl-3.5.0 torch-2.0.0 torch-tb-profiler-0.4.3 torchvision-0.15.1 tqdm-4.66.4 triton-2.0.0 typer-0.12.3 typing-extensions-4.12.2 tzdata-2024.1 urllib3-2.2.2 watchdog-4.0.1 werkzeug-3.0.3 wheel-0.43.0 zipp-3.19.2</p>
<p>Pillow is still being installed by torchvision even though pillow-simd was installed already.  I looked in setup.py and found this:</p>
<p>def get_requirements():<br>
def get_dist(pkgname):<br>
try:<br>
return get_distribution(pkgname)<br>
except DistributionNotFound:<br>
return None</p>
<pre><code>pytorch_dep = "torch"
if os.getenv("PYTORCH_VERSION"):
    pytorch_dep += "==" + os.getenv("PYTORCH_VERSION")

requirements = [
    # TODO: Remove &lt;2 constraint! https://github.com/pytorch/vision/issues/8531
    "numpy&lt;2" if sys.platform == "win32" else "numpy",
    pytorch_dep,
]

# Excluding 8.3.* because of https://github.com/pytorch/vision/issues/4934
pillow_ver = " &gt;= 5.3.0, !=8.3.*"
pillow_req = "pillow-simd" if get_dist("pillow-simd") is not None else "pillow"
requirements.append(pillow_req + pillow_ver)

return requirements
</code></pre>
<p>but it does not seem to work correctly when installing the package.  Does it only work if building from source</p>

ptrblck: <p>You could try to install <code>torchvision</code> with <code>--no-deps</code> in case the binaries require PIL and don’t recognize PIL-SIMD.</p>
In-place operators slow down inferences

mhs4670go: <p>Hello.</p>
<p>I’m trying to apply <a href="https://arxiv.org/abs/2211.10438" rel="noopener nofollow ugc">SmoothQuant</a> to <a href="https://github.com/facebookresearch/fairseq" rel="noopener nofollow ugc">fairseq</a> NMT model. The thing is that there need some in-place updates on parameters of a model.</p>
<p>But, after in-place update, the model inference speed has become too slow. Related codes are <a href="https://github.com/mit-han-lab/smoothquant/blob/c61476d728e42ae0d8a35e7e78494edcac3237b5/smoothquant/smooth.py#L41" rel="noopener nofollow ugc">here</a>.</p>
<pre data-code-wrap="python"><code class="lang-python">@torch.no_grad()
def smooth_ln_fcs(ln, fcs, act_scales, alpha=0.5):
# ..
    ln.weight.div_(scales)
    ln.bias.div_(scales)

    for fc in fcs:
        fc.weight.mul_(scales.view(1, -1))

# def main
model.translate(..)
# apply `smooth_ln_fcs` to model
model.translate(..) -&gt; too slow
</code></pre>
<p>I don’t know why it happened. Is there any reason or clues?</p>

ptrblck: <p>Could you describe how you are measuring the performance? If you are using the GPU, are you synchronizing the code if host timers are used?</p>
Problem with a modul

Juan_David: <p>Hi, i have this problem OSError: [WinError 126] The specified module cannot be found. Error loading “C:\Users\jdzun\AppData\Local\Programs\Python\Python311\Libsite-packages\torch\lib\fbgemm.dll” or one of its dependencies. I followed the steps in the documentation and still have the same problem.</p>

ptrblck: <p>The issue seems to be related to <a href="https://discuss.pytorch.org/t/failed-to-import-pytorch-fbgemm-dll-or-one-of-its-dependencies-is-missing/201969/">this one</a>.</p>
Truncating CV Models from huggingface

Sai03: <p>Hi I was wondering if there was any way to perform model slicing/truncating on a Huggingface CV model say MobileNetv1? As far as i have searched the internet most topics are related to Transformers. I would like to know if truncating CV models are possible and if so, how? Also I would like to ask why pytorch does not have the MobilenetV1 model by default (inside torchvision.models).</p>
Implementing batched training without batching data

ldkuba: <p>Hi. I have a beginner question, which I can’t seem to find a definitive answer to anywhere.</p>
<p>From what I understand, doing batched training has 2 benefits:</p>
<ul>
<li>Batched operations can be optimized better on the GPU leading to performance increases</li>
<li>The optimizer runs after seeing multiple data samples before doing .step() which makes the model less sensitive to individual samples.</li>
</ul>
<p>My question is as follows: If we ignore the performance benefits are these two training loops equivalent?</p>
<p>Proper batching:</p>
<pre data-code-wrap="python"><code class="lang-python">model = SomeModel()
optimizer = SomeOptimizer(model.parameters(), ...)
dataloader = DataLoader(dataset, batch_size=10)

for batch_data in enumerate(dataloader):
    loss = model(batch_data)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
</code></pre>
<p>Manual batching?:</p>
<pre data-code-wrap="python"><code class="lang-python">model = SomeModel()
optimizer = SomeOptimizer(model.parameters(), ...)
dataloader = DataLoader(dataset, batch_size=10)

for batch_data in enumerate(dataloader):
    for i in range(10):
        loss = model(batch_data[i])
        loss.backward()

    optimizer.step()
    optimizer.zero_grad()
</code></pre>
<p>If they are not equivalent, what is the difference and can it be remedied without passing batched data into the model?<br>
For context, due to time pressure I’m not able to implement the model with batched input but I would still like to simulate the effects of batched learning (except for the performance benefits)</p>
<p>Thanks in advance!</p>

Origin: <p>Hi <a class="mention" href="/u/ldkuba">@ldkuba</a>. Did you find any difference? I have the same doubts. Thanks.</p>

qq-me: <p>Depends on the loss. For example MSE by default has <code>reduction = "mean"</code> argument so loss is calculated for all samples in the batch and then it calculates the mean. But when you are doing this manual batching it is equivalent to calculating the sum instead of the mean. But you are basically doing gradient accumulation and other than differences in how loss functions are calculated over the batch it should be equivalent.</p>
<p>Also some pytorch modules work on both batched (BC*) and unbatched (C*) data. But some, especially custom ones, only work on batched data, so you might get weird results by passing unbatched data. So you may want to do <code>loss = model(batch_data[i].unsqueeze(0))</code> to convert C* to BC*.</p>
How many times will __getitem__ in the data.Dataset be called?

Sail2Dream: <p>in a training loop, for example <code>:for batch_idx, (inputs, targets, info) in enumerate(self.train_loader):</code></p>
<p><strong>getitem</strong> is called once for each iteration.</p>
<p>I am trying to keep track of the number of times <strong>getitem</strong> is called, in my case the total is 8960, the epoch is 140, the batch_size is 16, and the workers is 8. I use 8 Gpus for DDP training.</p>
<p>8960/140 is 64, but then I couldn’t find the connection between the parameters. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<p>Does anyone know how 64 came about? Maybe it has something to do with batchsize, but it’s still a 4x difference.</p>

ptrblck: <p>The <code>__getitem__</code> will be called <code>len(sampler)</code> times, which in the common use cases corresponds to the number of samples, i.e. <code>len(dataset)</code>, so that all samples will be indexed, processed, and returned.</p>
Identifying memory leak/reference cycle in training loop

schroedinger: <p>I’m trying to identify the cause of a memory leak in my training loop.</p>
<pre><code class="lang-auto">    def train_one_batch(self, batch_features: BatchMoleculeFeatures) -&gt; tuple[torch.Tensor, float]:
        self.model.train()
        batch_features.to(self.device)
        self.optimizer.zero_grad()
        output_spectra: torch.Tensor = self.model(adjacency_onehot=batch_features.adjacency_onehot, per_atom_features=batch_features.per_atom_features, input_mask=batch_features.input_mask, fragment_formulas=batch_features.fragment_formulas, fragment_peaks=batch_features.fragment_peaks_binned)
        loss: torch.Tensor = self.loss_function(predicted_spectra=output_spectra, true_spectra=batch_features.true_spectrum)
        loss.backward()
        self.optimizer.step()
        # torch.cuda.empty_cache()
        return output_spectra.detach(), loss.item()
</code></pre>
<p>When checking the GPU memory usage right after loss.backward() in the first batch using nvidia-smi, I can see that 31 GiB are used (out of 44 GiB). However, during the second batch, calling loss.backward() results in OOM:</p>
<pre><code class="lang-auto">CUDA out of memory. Tried to allocate 14.40 GiB (GPU 4; 44.32 GiB total capacity; 15.14 GiB already allocated; 14.03 GiB free; 29.72 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
</code></pre>
<p>If I manually call torch.cuda.empty_cache() after each batch (as in the code line currently commented out), I don’t get OOM and the model trains without issues. I have taken a memory snapshot after three batches (after lowering the batch size such that it doesn’t OOM anymore) that can be viewed using the pytorch memory_viz tool: <a href="https://github.com/define-struct-union/debug-dump/blob/5254873e4dc432193e293bd1d4cf7934f152a902/snap.shot" rel="noopener nofollow ugc">https://github.com/define-struct-union/debug-dump/blob/5254873e4dc432193e293bd1d4cf7934f152a902/snap.shot</a></p>
<p>As you can see there’s a huge spike in memory usage right at the end of each batch, but I’m not able to find what exactly is causing this. I have furthermore used the reference cycle detector (using <code>warn_tensor_cycles()</code>) to find which tensors are being kept alive, and it has created this graph: <a href="https://github.com/define-struct-union/debug-dump/blob/5254873e4dc432193e293bd1d4cf7934f152a902/tmpz32nbdpv.html" rel="noopener nofollow ugc">https://github.com/define-struct-union/debug-dump/blob/5254873e4dc432193e293bd1d4cf7934f152a902/tmpz32nbdpv.html</a></p>
<p>It shows that both the loss and output_spectra tensors are being kept alive, but yet again I don’t see what would be the cause of this.  In particular, instead of useful object names, the graph only seems to show debug frames. However, when I run the same code outside of the debugger, then no warnings are issued and no html file is created. What other steps or tools could I use to find the exact cause of this issue?</p>
Pytorch Torch Serve

BABYYODA: <p>Hi guys, I have done an AI system that is able to detect food, and it was written in python. the model used and trained was on ultralytics YOLO. For deployment, i decided to use torch serve. However, it runs into some issue, this is how i wrote for the handler to load and run inference:</p>
<p>def initialize(self, context):<br>
# Setting up base path<br>
base_path = os.path.dirname(<strong>file</strong>)  # Path of the current file<br>
# Construct relative path for the models<br>
detection_model_path = os.path.join(base_path, “weights”, “July best.pt”)<br>
classification_model_path = os.path.join(base_path, “weights”, “classification_fish_best.pt”)</p>
<pre><code>    # Load the detection model architecture
    self.model = YOLO()  # Initialize the architecture

    # Load the model weights using torch.load
    loaded_weights = torch.load(detection_model_path, map_location=self.device)  # Use map_location if needed
    self.model.load_state_dict(loaded_weights)

    self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    self.model.to(self.device)

    # Set the model to evaluation mode
    self.model.eval()
</code></pre>
<p>is this writing correct? cos i was able to archive it and start torch serve. However, in the process of running, it has some issues, like file not found error at specific location. the  path to the location is created when everytime i started running torch serve, so i have no idea how it works, and to run inference it gives nothing, so i assume there is a problem with how the model is served</p>
<pre><code>    # Load the classification model architecture
    self.classification_model = YOLO()

    # Load the classification model weights
    classification_weights = torch.load(classification_model_path, map_location=self.device)
    self.classification_model.load_state_dict(classification_weights)
    self.classification_model.to(self.device)
    self.classification_model.eval()
</code></pre>
Shape mismatch error for vision transformer

Rohan_Bareja: <p>hello,</p>
<p>I am trying to extract features from this model (<a href="https://github.com/bioptimus/releases/tree/main/models/h-optimus/v0" class="inline-onebox" rel="noopener nofollow ugc">releases/models/h-optimus/v0 at main · bioptimus/releases · GitHub</a>) using a giant vision transformer backbone.</p>
<p>I am creating the vit-giant as in here: <a href="https://github.com/facebookresearch/dino/blob/main/vision_transformer.py" class="inline-onebox" rel="noopener nofollow ugc">dino/vision_transformer.py at main · facebookresearch/dino · GitHub</a></p>
<p>Here is my code to generate vit giant:</p>
<pre><code class="lang-auto">def vit_giant(patch_size=16, **kwargs):
    model = VisionTransformer(
        patch_size=patch_size, embed_dim=1536, depth=40, num_heads=24, mlp_ratio=5.33334,
        qkv_bias=True, norm_layer=partial(nn.LayerNorm, eps=1e-6), **kwargs)
    return model
</code></pre>
<p>Also I make following changes in class MLP to the vision_transformer.py code:</p>
<pre><code class="lang-auto">class Mlp(nn.Module):
    def __init__(self, in_features, hidden_features=None, out_features=None, act_layer=nn.SiLU, drop=0.):
        super().__init__()
        out_features = out_features or in_features
        hidden_features = hidden_features or in_features
        self.fc1 = nn.Linear(in_features, hidden_features)
        self.act = act_layer()
        self.fc2 = nn.Linear(hidden_features // 2, out_features)
        self.drop = nn.Dropout(drop)

    def forward(self, x):
        x = self.fc1(x)
        x = self.act(x)
        x = self.drop(x)
        x = self.fc2(x)
        x = self.drop(x)
        return x
</code></pre>
<p>When I am trying to extract features using eval_linear.py (<a href="https://github.com/facebookresearch/dino/blob/main/eval_linear.py" class="inline-onebox" rel="noopener nofollow ugc">dino/eval_linear.py at main · facebookresearch/dino · GitHub</a>).I get this error:</p>
<pre><code class="lang-auto">line 188, in interpolate_pos_encoding
RuntimeError: shape '[1, 15, 15, 1536]' is invalid for input of size 391680
    patch_pos_embed.reshape(1, int(math.sqrt(N)), int(math.sqrt(N)), dim).permute(0, 3, 1, 2),
RuntimeError: shape '[1, 15, 15, 1536]' is invalid for input of size 391680
</code></pre>
Normalising images

JAllsopp: <p>Hi,<br>
I have some images from a microscope, where the background is very dark and makes up most of the image, and a few very bright points in the image, so the distribution is very non-Gaussian (well a Gaussian plus a very long positive tail).</p>
<p>What would be the best way to normalise these images?</p>
<p>Thanks!</p>
Swap 4D tensors

An_Pham: <p>Dear Friends</p>
<p>I am new with pytorch tensor, i have a problems, I have a tensor [B, C, H, W], I would like to swap columms in H dimensions, or two rows in W dimension<br>
many thanks</p>

ptrblck: <p>You can directly index your tensor using the new indices. Could you post a (pseudo-)code showing your use case in case you get stuck?</p>
Is PyTorch 2.4 not compatible with Cuda 12.4?

Monarch: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f913ec61139982fd3f633afd54799dd0b1e472b1.png" data-download-href="https://discuss.pytorch.org/uploads/default/f913ec61139982fd3f633afd54799dd0b1e472b1" title="Screenshot 2024-07-29 111841"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f913ec61139982fd3f633afd54799dd0b1e472b1_2_690x270.png" alt="Screenshot 2024-07-29 111841" data-base62-sha1="zxrzH7X2H5kweyXm2SDZ0igxFV7" width="690" height="270" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f913ec61139982fd3f633afd54799dd0b1e472b1_2_690x270.png, https://discuss.pytorch.org/uploads/default/original/3X/f/9/f913ec61139982fd3f633afd54799dd0b1e472b1.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/9/f913ec61139982fd3f633afd54799dd0b1e472b1.png 2x" data-dominant-color="F4DAD6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2024-07-29 111841</span><span class="informations">1016×398 23 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>according to the website it should be.</p>
<p>but for some reason it won’t use my gpu the only answers I found here were to downgrade my Cuda version…</p>

ptrblck: <p>I don’t know which answers you are referring to but your locally installed CUDA toolkit won’t be used. PyTorch is compatible with CUDA 12.4 and the install matrix shows the command to install these binaries with the corresponding CUDA dependencies.</p>