akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Help SD on Mac M1 Pro

mmoktl: <p>Dear Sir, All</p>
<p>I use Code about Stable Diffusion WebUI AUTOMATIC1111 on Mac M1 Pro 2021 (without GPU) , when I run then have 2 error :</p>
<ol>
<li>Launching Web UI with arguments: --skip-torch-cuda-test --upcast-sampling --no-half-vae --use-cpu interrogate</li>
</ol>
<p>no module ‘xformers’. Processing without…</p>
<p>no module ‘xformers’. Processing without…</p>
<p>No module ‘xformers’. Proceeding without it.</p>
<p>Warning: caught exception ‘Torch not compiled with CUDA enabled’, memory monitor disabled</p>
<ol start="2">
<li>RuntimeError: MPS backend out of memory (MPS allocated: 9.93 GB, other allocations: 2.03 GB, max allowed: 18.13 GB). Tried to allocate 7.43 GB on private pool. Use PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 to disable upper limit for memory allocations (may cause system failure).</li>
</ol>
<p>I try install Torch, Tensoflow to active GPU (virtual) and check have GPU, but the same error, this is file web-user.bat :</p>
<p>git pull</p>
<p>@ echo off</p>
<p>set PYTHON=</p>
<p>set GIT=</p>
<p>set VENV_DIR=</p>
<p>set COMMANDLINE_ARGS= --xformers --autolaunch --skip-version-check --precision full --no-half --skip-torch-cuda-test --listen --port 9999 --disable-safe-unpickle --deepdanbooru</p>
<p>call webui.bat</p>
<p>Pls help me, many thanks</p>

bekocak: <p>Hi, have you found the solution?</p>

Valentino: <p>same exact problem, have you found any solutions?<br>
are you running on an MBP M1 base?</p>
How to convert a group of images into a image dataset format so that I can feed the data in to a CNN?

rookiearun: <p>Hello folks, I am trying to convert images generated in to  a dataset.(All I ahve is just png images in n folders and there is no label or meta data)</p>
<p>This is what I aspire to do:</p>
<p>1.I am using <strong>torch audio</strong> to convert audio formats to <strong>Mel spectrogram</strong> and save the images as Png format. <strong>Status:done</strong><br>
2.Now I have ‘n’ number of folders(classes) with images so I am curious if I could convert the newly generated images into data and target as in normal datasets, so that I can use sklearn to do the test train splits <code>sklearn.model_selection.train_test_split.</code><strong>Status:not done</strong></p>
<p>eg: fetch mnist dataset</p>
<pre><code class="lang-auto">&gt; ds_mnist = sklearn.datasets.fetch_openml(
&gt;     data_id=554,
&gt;     as_frame=False
&gt; )
</code></pre>
<p>Split data and target in to X and y</p>
<blockquote>
<pre><code class="lang-auto">dataset_X = ds_mnist .data.astype('float32')

dataset_y = ds_mnist .target.astype('int64')
</code></pre>
<blockquote>
<p>Blockquote</p>
</blockquote>
</blockquote>

ptrblck: <p>You could use <code>ImageFolder</code> to load the dataset and create the class labels based on the folders. Once this is done you would then create the split indices and use <code>Subset</code>s to create the training, validation, and test datasets.</p>
Unable to load audio file using torchaudio.load()

rookiearun: <p>So I downloaded the datasets and was trying to load the waveform using torchaudio.load(), I have given the arguments as below  :</p>
<pre><code class="lang-auto">&gt; filename = "./data/SpeechCommands/speech_commands_v0.02/yes/00f0204f_nohash_0.wav"
&gt; waveform, sample_rate = torchaudio.load(filepath=filename, num_frames=3)
&gt; print(f'waveform tensor with 3 frames:  {waveform} \n')
&gt; waveform, sample_rate = torchaudio.load(filepath=filename, num_frames=3, frame_offset =2)
&gt; print(f'waveform tensor with 2 frame_offsets: {waveform} \n')
&gt; waveform, sample_rate = torchaudio.load(filepath=filename)
&gt; print(f'waveform tensor:  {waveform}' )
</code></pre>
<p>Error message:</p>
<blockquote>
<p>Traceback (most recent call last):</p>
<p>Cell In[6], line 2<br>
waveform, sample_rate = torchaudio.load(filepath=filename, num_frames=3)</p>
<p>TypeError: get_load_func..load() got an unexpected keyword argument ‘filepath’</p>
</blockquote>
<p>I am using jupyterlab notebook to execute the codes.</p>
<p>Is there anything missing over this?</p>
<p>The <a href="https://pytorch.org/audio/stable/generated/torchaudio.load.html" rel="noopener nofollow ugc">torchaudio.load</a> documentation shows , I can give a path like object as first argument, so I was wondering, why it throws an error.</p>
<p>I am basically following this <a href="https://learn.microsoft.com/en-us/training/modules/intro-audio-classification-pytorch/2-understand-audio-data" rel="noopener nofollow ugc">tutorial</a>.</p>

ptrblck: <aside class="quote no-group" data-username="rookiearun" data-post="1" data-topic="193037">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rookiearun/48/65699_2.png" class="avatar"> rookiearun:</div>
<blockquote>
<p>The <a href="https://pytorch.org/audio/stable/generated/torchaudio.load.html">torchaudio.load</a> documentation shows , I can give a path like object as first argument, so I was wondering, why it throws an error.</p>
</blockquote>
</aside>
<p>The docs show the first argument is called <code>uri</code> instead of <code>filepath</code>, so you might want to change the argument name or just pass the <code>filename</code> implicitly without specifying the argument name.</p>

rookiearun: <p>Thank you, that was the issue and changing the argument name to uri solved the error.</p>
One cycle policy

Neofytos: <p>Hi,</p>
<p>I am using OneCycleLR scheduler with a pct_start of 1.0 (i.e. go from a small learning rate up to a maximum learning rate and then stop), but I get an error at the last training iteration:<br>
<code>computed_lr = self.anneal_func(group['max_lr'], group['min_lr'], down_step_num / self.step_size_down)</code><br>
<code>ZeroDivisionError: float division by zero</code></p>
<p>On the other hand, if I use a pct_start of 0.0 (i.e. go from a large learning rate down to a minimum learning rate), the scheduler works as expected. Is this a missed edge case, or am I using the scheduler improperly?</p>
<p>Thank you!</p>

ptrblck: <p>How did you define <code>final_div_factor</code>, which should be used for the minimal learning rate?<br>
Could this value be too high and you might thus run into a zero division?</p>

braunagn: <p><a class="mention" href="/u/neofytos">@Neofytos</a> What is the <code>max_momentum</code> set at?  I am able to reproduce the same <code>ZeroDivisionError</code> when <code>pct_start=1.0</code> and <code>max_momentum=1.0</code>.  When I change <code>max_momentum=0.99</code> (or anything less than 1.0), there is no <code>ZeroDivisionError</code>.</p>
Pytorch Debug lldb

mahuichao: <p>Hello guys, I want to debug pytorch source code from python to c++. I do some research from google and  know that gdb can do this work. But I am using Mac M1, I have no gdb and can’t install it, But I have lldb.So is there some easy ways for me to debug c++ function called by python with lldb?I see there is a file called “pytorch_lldb.py” in tools directory, does this script can help me ?</p>

mahuichao: <p>Problem Resolved, Closed.</p>

Haitao_Xiao: <p>hi,I have the same question, Could you please give me some help?</p>
Canonical way to assert tensor shape

smonsays: <p>What is the canonical way to assert that a given tensor has the correct shape, i.e. if it is known beforehand what shape it should have? Currently, I use assertions in the following way, which adds a lot of clutter to the code:</p>
<pre><code class="lang-auto">assert x.shape == torch.Size([dim1, dim2])
</code></pre>
<p>A similar question about IDE based tensor-shape checking has been asked <a href="https://discuss.pytorch.org/t/is-there-any-tool-for-tensor-shape-checking-in-ide/5985">here</a>, but has not received an answer.</p>

Oli: <p><code>assert x.shape == (dim1, dim2)</code> works aswell <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

smonsays: <p>Thanks, that looks a lot nicer already.</p>

MicahZoltu: <p>This doesn’t seem to work for 1 dimensional tensors.  Any pro tips for something that works regardless of the number of dimensions?</p>
<pre><code class="lang-py">a = torch.zeros([3])
b = torch.zeros([3, 5])
print(a.shape)
print(b.shape)
assert a.shape == (3) # error
assert b.shape == (3,5) # ok
</code></pre>

n1k31t4: <p>The assertion fails because python will turn <code>(3)</code> into simply <code>3</code>. The shape is a tuple, which means you need the comma in there: <code>assert a.shape == (3,) # works</code>.</p>
Facebook DLRM model inference using pytorch 2.0 compilation method

Tiru_B: <p>Hi,</p>
<p>Does anyone tried Facebook DLRM 10GB,100GB model with pytorch 2.0?</p>
<p>I was trying it but it got stuck, not giving any final output.10GB waited for almost 4hrs but no output.</p>
<p>But if try the same code with small models,i,e less embeddingbag with lesser sizes i can see the output.</p>
<p>Trying cpu flow using inductor as backend.</p>
Interpolate align_corners=False

isalia99: <p>I am having somewhat hard time understanding how to compute a linear interpolation for 1D tensor. For simplicity let’s take a small tensor of [1, 2, 3]. If we do the following:</p>
<pre><code class="lang-auto">import torch
from torch.nn import functional as F

a = torch.tensor([1, 2, 3], dtype=torch.float32)
a = a.unsqueeze(0).unsqueeze(0)

F.interpolate(a, scale_factor=3, mode="linear", align_corners=False)
</code></pre>
<p>Then we get the answer of :<br>
<code>tensor([[[1.0000, 1.0000, 1.3333, 1.6667, 2.0000, 2.3333, 2.6667, 3.0000,           3.0000]]])</code></p>
<p>Now I dug through the cpp code and found out this to calculate the indices:</p>
<pre><code class="lang-auto">template&lt;typename T&gt;
static inline T linear_upsampling_compute_source_index(
                          T scale, int dst_index, bool align_corners) {
  if (align_corners) {
    return scale * dst_index;
  } else {
    T src_idx = scale * (dst_index + 0.5) - 0.5;
    return src_idx &lt; 0 ? T(0) : src_idx;
  }
}
</code></pre>
<p>So whenever align_corners=False then we have the following formula to calculate the x’s for interpolation.</p>
<p>Let’s take the above example [1, 2, 3]. So the indices we have are [0, 1, 2]. To calculate the src_index given with this formula we will have <code>src_idx = 1</code> for index 0 and <code>src_idx=4</code> for index 1. Plugging these values into interpolation formula to find the value of the y0:</p>
<pre><code class="lang-auto">y = y0 + (x - x0) * ((y1-y0)/(x1-x0))
y = 1 + (0 - 1) * ((2 - 1)/(4 - 1))
y = 2/3
</code></pre>
<p>So my question is how does torch get 1.0 for the index=0 in the output I have pasted? torch version I am using is 2.1.1</p>
Torch.vmap in-place operation error

abcamiletto: <p>Hi all,</p>
<p>I have a problem with torch.vmap<br>
As clarified in the docs, torch.vmap does not support in-place operations when the number of elements of the tensor decreases.<br>
In my case, it does not seem like that is the case at first glance, but I am clearly missing something.<br>
Additionally, all the suggested workarounds I found googling are not working.<br>
Any help would be very appreciated!</p>
<blockquote>
<p>import torch</p>
<p>def faulty_function(x, y):<br>
mat = torch.zeros(x.shape[0], 3, 3)</p>
<pre><code>mat[..., 0, 0] = x * y    
return mat
</code></pre>
<p>batched_x = torch.randn(10, 1)<br>
batched_y = torch.randn(10, 1)</p>
<p>result = torch.vmap(fixed_function)(batched_x, batched_y)</p>
</blockquote>
<p>With this example, I get the following error:</p>
<blockquote>
<p>RuntimeError: vmap: inplace arithmetic(self, *extra_args) is not possible because there exists a Tensor <code>other</code> in extra_args that has more elements than <code>self</code>. This happened due to <code>other</code> being vmapped over but <code>self</code> not being vmapped over in a vmap. Please try to use out-of-place operators instead of inplace arithmetic. If said operator is being called inside the PyTorch framework, please file a bug report instead.</p>
</blockquote>
New to Pytorch, quick question

itsvnkraj: <p>Hello All</p>
<p>Newbie to pytorch, etc. Please bear with me</p>
<pre><code class="lang-auto">nvidia-smi
Fri Dec  8 14:29:51 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.147.05   Driver Version: 525.147.05   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            On   | 00000001:00:00.0 Off |                    0 |
| N/A   31C    P8     9W /  70W |      2MiB / 15360MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+


</code></pre>
<p>On the pytorch website, to install locally, i see either cuda11.8 or 12.1</p>
<p>Which one should I install please? I read all the documents all over the place and am totally confused.</p>
<p>Thanks in advance for your answer and have several nice days ahead</p>
<p>Rajini</p>

ptrblck: <p>Install the binaries with 12.1 if you are using a newer GPU. If you are stuck on e.g. Kepler architectures, use 11.8.</p>
Can't load Apple Silicon trained model to Docker cpu

brandenchan: <p>I have trained a BERT model from HuggingFace transformers with an additional linear layer on top using Apple Silicon (mps). The model is saved as follows:</p>
<pre><code class="lang-auto">torch.save(self.state_dict(), model_file_path)
</code></pre>
<p>I have no problem loading and running this model on my mac with either the ‘cpu’ device or ‘mps’. Here is the method I am using to load it:</p>
<pre><code class="lang-auto">    def load_model_from_file(self, filename):
        state_dict = torch.load(filename, map_location=self.device)
        self.load_state_dict(state_dict)
        self.eval()
        self.to(self.device)
        return self
</code></pre>
<p>But when I try to run this in a Docker container in ‘cpu’ mode, I get the following error:</p>
<pre><code class="lang-auto">2023-12-08 10:04:57 ERROR - run_s - Traceback (most recent call last):
  File "/s/scripts/run_s.py", line 188, in run
    trained_model = SModel.auto_load(**all_params)
  File "/s/s/modeling/base.py", line 56, in auto_load
    return base_model.load_model_from_file(train_model_filepath)
  File "/s/sa/modeling/BERT_class.py", line 205, in load_model_from_file
    state_dict = torch.load(filename, map_location=self.device)
  File "/usr/local/lib/python3.9/site-packages/torch/serialization.py", line 1014, in load
    return _load(opened_zipfile,
  File "/usr/local/lib/python3.9/site-packages/torch/serialization.py", line 1422, in _load
    result = unpickler.load()
  File "/usr/local/lib/python3.9/site-packages/torch/serialization.py", line 1392, in persistent_load
    typed_storage = load_tensor(dtype, nbytes, key, _maybe_decode_ascii(location))
  File "/usr/local/lib/python3.9/site-packages/torch/serialization.py", line 1366, in load_tensor
    wrap_storage=restore_location(storage, location),
  File "/usr/local/lib/python3.9/site-packages/torch/serialization.py", line 1296, in restore_location
    return default_restore_location(storage, map_location)
  File "/usr/local/lib/python3.9/site-packages/torch/serialization.py", line 381, in default_restore_location
    result = fn(storage, location)
  File "/usr/local/lib/python3.9/site-packages/torch/serialization.py", line 304, in _hpu_deserialize
    assert hpu is not None, "HPU device module is not loaded"
AssertionError: HPU device module is not loaded
</code></pre>
<p>How can I fix this?</p>
Distributed pytorch support in mpi

Saji_G: <p>hello i had installed pytorch using the command<br>
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 pytorch -c nvidia</p>
<p>and mpi using the command<br>
sudo apt-get -y install mpich &amp;  conda install -c conda-forge mpi4py</p>
<p>but while executing distributed pytorch program , mpi support error occured<br>
RuntimeError: Distributed package doesn’t have MPI built in. MPI is only included if you build PyTorch from source on a host that has MPI installed.<br>
RuntimeError: Distributed package doesn’t have MPI built in. MPI is only included if you build PyTorch from source on a host that has MPI installed.<br>
please give a solution for it</p>

ptrblck: <aside class="quote no-group" data-username="Saji_G" data-post="1" data-topic="192708">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saji_g/48/62059_2.png" class="avatar"> Saji_G:</div>
<blockquote>
<p>RuntimeError: Distributed package doesn’t have MPI built in. MPI is only included if you build PyTorch from source on a host that has MPI installed.</p>
</blockquote>
</aside>
<p>Did you follow this suggestion and build PyTorch from source with MPI support?</p>

Saji_G: <p>yes . pytorch distributed package not installed successfuly. anaconda ,cuda and mpi working properly. i can run pytorch in local system. using mpi python and c program working perfectly.<br>
but my actual demand is use distributed pytorch in hpc( 5 client nodes and 1 server node)<br>
any one can help me to install distributed pytorch with mpi support</p>
<p>my os is ubuntu 22.04 server<br>
installed anaconda<br>
cuda<br>
pytorch ( ```<br>
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia)</p>
<p>for mpi support</p>
<pre><code class="lang-auto">conda install -c conda-forge mpi4py
</code></pre>
<p>after that i checked mpi with following program.</p>
<pre><code class="lang-auto">from mpi4py import MPI

comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()

print(f"Hello from rank {rank} out of {size} processes!")

MPI.Finalize()
</code></pre>
<pre><code class="lang-auto">mpiexec -n 4 python mpi_hello.py
</code></pre>
<p>it work succefully</p>
<p>but while executing the distributed pytorch program it will not recognize mpi.</p>
<p>please help me</p>
Cannot open connection when downloading pb file and fr2en dataset

cdw72: <p><a href="http://fb-glow-assets.s3.amazonaws.com/models/lenet_mnis" class="onebox" target="_blank" rel="noopener nofollow ugc">http://fb-glow-assets.s3.amazonaws.com/models/lenet_mnis</a></p>

ptrblck: <p>Could you add more information about where you’ve found this file/code?</p>

cdw72: <p>Thank you.<br>
the first file is glow/utils/download_datasets_and_models.py<br>
The fr2en data set link in this file cannot be opened.<br>
the second file is glow/examples/bundles/lenet_mnist/CMakeLists.txt<br>
set(LENET_MNIST_GLOW_S3 “<a href="http://fb-glow-assets.s3.amazonaws.com/models/lenet_mnist%E2%80%9D" rel="noopener nofollow ugc">http://fb-glow-assets.s3.amazonaws.com/models/lenet_mnist”</a>)<br>
This url cannot be opened</p>
Help Set tf=True?

Veep_Peep: <p>Hello,<br>
Not sure this is a pytorch issue .Very new to this and was trying something from a Youtube video called:</p>
<p>AI Text and Code Generation with GPT Neo and Python | GPT3 Clone<br>
[<a href="https://github.com/nicknochnack/GPTNeo" class="inline-onebox" rel="noopener nofollow ugc">GitHub - nicknochnack/GPTNeo</a>}</p>
<p>Using Jupyter Notebook</p>
<p>This line was run:</p>
<p>generator = pipeline(‘text-generation’, model=‘EleutherAI/gpt-neo-2.7B’) # Second line<br>
OSError: Unable to load weights from pytorch checkpoint file for ‘EleutherAI/gpt-neo-2.7B’ at 'C:\Users\USER7/.cache\huggingface\transformers\0839a11efa893f2a554f8f540f904b0db0e5320a2b1612eb02c3fd25471c189a.a144c17634fa6a7823e398888396dd623e204dce9e33c3175afabfbf24bd8f56’If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.</p>
<p>I used something called Jupyter Notebook to execute each line</p>
<p>What I do not know, is where to set tf=True</p>
<p>This was the first line:<br>
!pip3 install torch==1.8.1+cpu torchvision==0.9.1+cpu torchaudio===0.8.1 -f <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a></p>
<p>which executed OK.</p>
<p>Thanks,</p>
<p>V</p>

ptrblck: <p>Based on the error message it seems that <code>Huggingface</code> uses the <code>from_tf</code> argument as mentioned e.g. <a href="https://huggingface.co/transformers/main_classes/model.html#transformers.PreTrainedModel.from_pretrained">here</a>, so you could try to pass this argument to the <code>from_pretrained</code> method, if it’s used.</p>

Veep_Peep: <p>Thank you for responding.</p>
<p>Unfortunately this is still beyond me<br>
This example had 4 lines.  And I guess it works for someone with the right set up - which I don’t have</p>
<p>not sure if I can get away with this:</p>
<p>generator = pipeline(‘text-generation’, model=‘EleutherAI/gpt-neo-2.7B’, from_tf=True)</p>
<p>guess I will blow up my system and try again!</p>
<p>I really appreciate your patience!</p>
<p>Thank you!</p>

Tentle: <p>You’re not the only one. I also experienced this problem. Setting <code>from_tf</code> to <code>True</code> only yields</p>
<pre><code class="lang-auto">OSError: Can't load weights for 'EleutherAI/gpt-neo-2.7B'. Make sure that:

- 'EleutherAI/gpt-neo-2.7B' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'EleutherAI/gpt-neo-2.7B' is the correct path to a directory containing a file named one of pytorch_model.bin, tf_model.h5, model.ckpt.
</code></pre>

Veep_Peep: <p>Thanks for pointing this out!!!</p>

Himanshu_Srivastava: <p>I pre-trained the model on custom corpus from roberta-base checkpoint and saved it in drive using save_pretrained(dir_path).<br>
Now I am trying to load this pre-trained model using from_pretrained(dir_path) .<br>
I throws the same error. If anyone found the solution, please guide me in right direction.</p>

Will_Gerstung: <p>pip install accelerate</p>
Block Gradient Flow For Specific Dimensions

abcdef123: <p>Assuming I have a VAE with a latent dimension size of 16 where I want to block the gradient flow through certain subset of those 16 dimensions: How can I do that? During forward pass I have access to the input as well as a mask tensor that says which dimension’s gradient flow should be blocked. When manually zeroing out the gradients for these dimensions after calling backward(), the gradient will still have influenced all layers before it already, so this solution is insufficient. As far as I know setting “requires_grad=False” should solve this problem in case I want to zero out the whole tensor’s gradient; but this is not the case here because I want to zero out only certain dimensions.</p>
<p>I was thinking of creating 16 linear layers of size (1024, 1) instead of creating one single linear layer of size (1024, 16) to solve this problem. I would then dynamically set the “requires_grad” flag depending on the mask tensor of the input. Is this the right way to do it or are there better solutions to this?</p>
Fine tuning on a pre-trained ViT as a backbone

hiru: <p>I am fine tuning “<a href="https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-weights/vit_small_p16_224-15ec54c9.pth" rel="noopener nofollow ugc">vit_small_patch16_224</a>” for a downstream task (learning rate is 0.001 and has a step scheduler of 0.1) dose anyone know the reason for loss curve to look like this?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/c/fc310c7477142fce160577a12206c80a7caedca2.png" data-download-href="https://discuss.pytorch.org/uploads/default/fc310c7477142fce160577a12206c80a7caedca2" title="W&amp;B Chart 12_7_2023, 7_04_01 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fc310c7477142fce160577a12206c80a7caedca2_2_690x362.png" alt="W&amp;B Chart 12_7_2023, 7_04_01 PM" data-base62-sha1="zYZplrPW0gVwRuNN5T1tKEEJeKK" width="690" height="362" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fc310c7477142fce160577a12206c80a7caedca2_2_690x362.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fc310c7477142fce160577a12206c80a7caedca2_2_1035x543.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fc310c7477142fce160577a12206c80a7caedca2_2_1380x724.png 2x" data-dominant-color="FAFCFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">W&amp;B Chart 12_7_2023, 7_04_01 PM</span><span class="informations">5056×2656 395 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Efficient grad calculation for multioutput/multihead models

matmul: <p>I have a graph-neural network (GNN) where at each node there are <code>N</code> independent readout MLPs that each produce a scalar. This means that the model has a large number of shared weights in the shared-backbone and a smaller number of indepedent weights in the readout MLPs. During training (prior to a <code>.backward()</code> call) I need the gradient of the each readout wrt the input, which is an array. I have a naive method that works where I zero <code>grad_outputs</code> except for the head of interest:</p>
<pre data-code-wrap="python"><code class="lang-python">batch_size, n_outputs = output.shape
gradients = []

for i in range(n_outputs):
    grad_output = torch.zeros_like(output)
    grad_output[:, i] = 1.0

    # Retain the graph for all outputs except the last one
    retain = True if i &lt; n_outputs - 1 else training

    gradient = torch.autograd.grad(
        outputs=output,
        inputs=input,
        grad_outputs=grad_output,
        retain_graph=retain,
        create_graph=training,
        allow_unused=True,
    )[0]

    gradients.append(gradient)

combined_gradient = torch.stack(gradients, dim=-1)
return -1 * combined_gradient
</code></pre>
<p>However, this seems inefficient as weights in the shared-backbone are the same but they might be computed <code>N</code> times. Is there a more efficient way of doing this? For example, first calculating the gradient of the backbone wrt the inputs and then outputs wrt the backbone output? Any insights appreciated!</p>
[Data Loader] What is the appropriate way to load dataset with >1M images?

Hui_Liu: <p>I have a large dataset with &gt;1M images, and i write a custom dataset like this</p>
<pre><code class="lang-auto">class MocoDataset(datasets.VisionDataset):

  def __init__(self, filepath_file=None, transform=None):
    list_of_paths = []
    with open(filepath_file, "r") as f:
      for line in f:
        line = line.rstrip("\n")
        list_of_paths.append(line)
    self.list_of_paths = np.array(
        list_of_paths
    )  # Here create a list of all image paths / paths..
    self.transform = transform
    self.loader = pil_loader

  def __len__(self):
    return len(self.list_of_paths)

  def __getitem__(self, idx):
    image_path = self.list_of_paths[idx]  # Gives the path to an image
    image = self.loader(image_path)
    # use 1 as a pseudo label for target
    return self.transform(image), 1
</code></pre>
<p>but my program got errored out like this</p>
<pre><code class="lang-auto">start training
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/workdir/moco/main_moco.py", line 671, in &lt;module&gt;
    main()
  File "/workdir/moco/main_moco.py", line 312, in main
    mp.spawn(main_worker, nprocs=ngpus_per_node, args=(ngpus_per_node, args))
  File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 240, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 198, in start_processes
    while not context.join():
  File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 140, in join
    raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 0 terminated with signal SIGKILL
...:/workdir$ /opt/conda/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 68 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
</code></pre>
<p>I used 32 workers, and batch size of only 2. I am pretty sure it is due to the dataset size because if I only load the first 10k image paths in my <em>init</em> method.  the training starts without problem</p>
<p>So:</p>
<ol>
<li>seeking some suggestions of how to implement this efficiently…</li>
<li>if my batch is 2, there will only be 2 images loaded to GPU each time?</li>
<li>even if loader is doing prefetching, it won’t try to get all the items for an epoch into CPU memory right?</li>
</ol>
<p>Thanks!!</p>

Hui_Liu: <p>actually, it seems working after I reduce workers to 2. So if too many workers it could kill the hosting process? so if I would like to use more workers I need to use more GPUs?</p>
Rewrite the loss function backward part

xianggkl: <p>Can I write my own gradients backward function?<br>
More accurately, I just want to change a small part of the original Pytorch backward of a certain forward orgorithm, where can I see the code of the certain forward orgorithm’s backpropagation steps?</p>

ptrblck: <p>Yes, you can create custom <code>autograd.Function</code>s as described <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">here</a> and can find the backward definitions in <a href="https://github.com/pytorch/pytorch/blob/b0a9641815fbe52918890bfb1e4b07daf6d97593/tools/autograd/derivatives.yaml"><code>tools/autograd/derivatives.yaml</code></a> or by searching for the kernel name in the repository.</p>

xianggkl: <p>Thanks for your reply! I check the derivatives.yaml file and I can find the function name which I want to rewrite. But I still cannot find the whole math calculations. Where do these concrete calculations of the backpropagation steps?</p>

ptrblck: <p>The derivative should either point to a few native ops or to the actual backward kernel, which you could then search for.</p>
From channels to tiles

nnnm: <p>Hi,</p>
<p>I’m new to Pytorch and I was wondering how to write a code that efficiently transforms channels into tiles. So from shape [c, n/(c/2),  n/(c/2)] to shape [1, n, n], but respecting the tiles ordering, which <code>.reshape</code> does not do. I’ll explain better with an example:</p>
<p>Let’s assume I have a tensor with shape <code>input.shape=torch.Size([4, 64, 64])</code>, and I would like an output tensor with shape <code>output.shape=torch.Size([1, 128, 128])</code>, but where each channel of the first input tensor is a tile in a quarter of the output tensor, e.g. output[64:, :64]==input[1, 64, 64].</p>
<p>But I don’t want to use assignation or for loops. The code would have to be general enough to take also e.g. an input tensor of shape [16, 32, 32], and output a tensor with size [1, 128, 128], where the 16 channels are ordered as tiles.</p>
<p>Thank you!</p>

eqy: <p>To answer this question, it might be helpful if you wrote a naive for-loop version for testing purposes and to clarify how you want to rearrange the underlying data.</p>

nnnm: <p>Hi <a class="mention" href="/u/eqy">@eqy</a>, here’s an example.</p>
<p>Let’s say I have an <code>input</code> of  4 channels all of shape [128, 128]. In this simple example, the channels all look the same. I would like to rearrange each channel like a tile in a mosaic with shape [128x2, 128x2].</p>
<pre><code class="lang-auto">output = torch.zeros(128*2, 128*2)
output[:128, :128] = input[0]
output[:128, 128:] = input[1]
output[128:, :128] = input[2]
output[128:, 128:] = input[3]
</code></pre>
<p>Since I am a new user I can only embed a single image. So here’s the <code>output</code> I would like. And each of the 4 images was a channel in <code>input</code>.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/3/a351c65619fe9267fff02f4ade98c7069591613f.png" alt="2b8454a0-bd3f-4a83-b2b3-d9950a9cfcf5" data-base62-sha1="niN1OycQfCYfjKivdfg9ckwK56D" width="262" height="252"></p>
<p>I would like the code to be flexible enough to choose based on <code>input</code> channels, how to rearrange the tiles in the <code>output</code>. Ideally, given input with shape<code> [C, H, W]</code> (with H==W), I’d like the output to have shape <code>[1, HxC/2. WxC/2]</code>. But<code> .reshape</code> or <code>.view</code> do not rearrange the pixels in the way that I’d like them to be.</p>
<p>Thanks for your help!</p>

eqy: <p>Here’s an example that more or less achieves what you want I think</p>
<pre><code class="lang-auto">import torch
input = torch.randn(4, 128, 128)
output = torch.zeros(128*2, 128*2)
output[:128, :128] = input[0]
output[:128, 128:] = input[1]
output[128:, :128] = input[2]
output[128:, 128:] = input[3]

# steps
# break input channels into 2x2
# tile means outer-channel split is H, inner channel split is W
# contiguous is not necessary unless you want the underlying data order to be changed
output2 = input.reshape(2, 2, 128, 128).permute(0, 2, 1, 3).reshape(128*2, 128*2).contiguous()
print(torch.allclose(output, output2))
</code></pre>
Optimize calculation of sub tensors

tdewolff: <p>I’m trying to optimize a piece of PyTorch code that causes the computation graph to become really large. I’m not sure if it’s possible, which is why I’d like some expert advice on this.</p>
<p>It is part of a toolkit for multi-output Gaussian processes, where data points can be in different “channels”, indicated by an integer in the first column of X (ie. channel index is in the range of <code>[0,channels)</code>). Lets say we have K channels and N input data points, for each combination of channels (K*K) we calculate and populate a sub tensor of the final tensor (shape <code>(N,N)</code>).</p>
<pre data-code-wrap="python"><code class="lang-python"># get channel indices for all data points
c = X[:,0].long() 

# create channel mask for X
m = [c==i for i in range(channels)]

# select X input for each channel
x = [X[m[i],1:] for i in range(channels)]

# find indices into final tensor for the elements of sub tensor
r = [torch.nonzero(m[i]) for i in range(channels)]

# final tensor
res = torch.empty(X.shape[0], X.shape[0])  # NxN

# loop over channel combinations
for i in range(channels):
    for j in range(channels):
        k = calculate_subtensor(i, j, x[i], x[j])
        res[r[i],r[j]] = k # k has shape (x[i].shape[0], x[j].shape[0])
</code></pre>
<p>The computation graph will have many Select, Unsqueeze, Permute, and View nodes, but the problem is that the number of nodes scales quadratically with <code>channels</code>. Can this code be improved?</p>
<p>Note that the channel index increases monotonically along the data points, something this code doesn’t make use of.</p>

tdewolff: <p>The problem is likely the fact that the loops are in Python, which makes my CPU at a 100% but my GPU only 60%. Surely, the loops can and should be unrolled so that the full computation graph can be generated. I suppose I need to play around with torch.jit?</p>
Seq2Seq with DataParallel

DodoI: <p>Folks, I am trying to run a Seq2Seq training with DataParallel.<br>
For one GPU everything works as expected but for &gt;1 I am encountering “inplace” operation error in the encoder.forward():<br>
<em>RuntimeError: Output 1 of BroadcastBackward is a view and its base or another view of its base has been modified inplace. This view is thch functions do not allow the output views to be modified inplace. You should replace the inplace operation by an out-of-place one.</em></p>
<p>My forward() is however rather trivial:</p>
<pre><code class="lang-auto">emb=self.embedding(input) #pretrained emb, with a real one is error in emb call
out,hidden=lstm(emb,hidden)
</code></pre>
<p>My question is, if I am doing it completely wrong or I have really some issue with inplace operation in my code. I am doing it as follow:</p>
<pre><code class="lang-auto">enc=someLSTMencClass() #here inside is above mentioned forward()
dec=someLSTMdecClass()

model=Seq2SeqClass(enc,dec) #inside in forward is called enc and dec
modelDP = nn.DataParallel(model, device_ids=[0,1])

for epoch in range(max_epochs):
  for (data1,data2) in train_rand_loader:
       loss = modelDP(input_tensor, target_tensor)
</code></pre>
<p>Any hint is welcome. Thanks.</p>
<p>Dodo</p>

bin_zhao: <p>try smaller batch_size</p>