akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Training loop takes a long time each epoch using TensorDataset

username: tomt
cooked: <p>Hi,</p>
<p>First of all, my dataset is loaded through a pickle file, where each variable is an np array (they are velocity components). Second, they are normalized and transformed to torch tensors. I’m training a SRGAN with low-res and high-res images btw. The dataset is around 14k images.</p>
<pre><code class="lang-auto">dataset_train = torch.utils.data.TensorDataset(LR_data_train, HR_data_train)
trainloader = torch.utils.data.DataLoader(dataset_train, batch_size=8,
                                          shuffle=True, num_workers=8, pin_memory=True)
</code></pre>
<p>I’m training my data on a NVIDIA Tesla V100, and it should not take 14 min each epoch, where each epoch contain around 1800 batches. I believe there is a bottleneck with slow IO speed, and was wondering if there are some workaround this?<br>
I believe the whole dataset is read each epoch, and I was thinking about maybe creating a custom datasetloader, or put all my tensors into a HDF5 file like</p><aside class="quote" data-post="2" data-topic="25643">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dreamyun/48/1558_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/dataloader-when-num-worker-0-there-is-bug/25643/2">DataLoader, when num_worker &gt;0, there is bug</a> 
  </div>
  <blockquote>
    Many pepole has the errors ,but there is no solution available. I hope the pytorch developers can help us solve the problem. Thanks in advance.
  </blockquote>
</aside>


username: ptrblck
cooked: <p>If you are dealing with a data loading bottleneck, I would recommend to read <a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">this post</a> which gives a good overview about possible reasons and workarounds.</p>
Different model reset strategies

username: sathvik_udupa
cooked: <p>Hi,<br>
I tried to find more information on</p>
<pre><code class="lang-auto">def weight_reset(m):
    if isinstance(m, conv1D):
        m.reset_parameters()
model.apply(weight_reset)
</code></pre>
<p>but I couldn’t find any documentation on that. What exactly is happening(will the model weights revert to definition weights?) and how is it different from weight init function on all layers?</p>
<pre><code class="lang-auto">def weight_init(m):
    if isinstance(m, conv1D)
        nn.init.xavier_uniform_(m.weight)
model.apply(weight_init)
</code></pre>

username: ptrblck
cooked: <p>The <code>reset_parameter</code> method is defined for each module (or its base class), as seen <a href="https://github.com/pytorch/pytorch/blob/f6c0a89acc929427e2e02a8215e7692455271178/torch/nn/modules/conv.py#L48" rel="nofollow noopener">here</a> for your conv layer.</p>
<p>The difference between both codes is, that the first one will use the default initialization which was specified for the particular layer, while the second approach will use your defined initialization (in your example <code>xavier_uniform</code>).</p>
Custom learn/ model classes for easier training

username: sathvik_udupa
cooked: <p>Hi,<br>
I am currently finishing up my writing my trainer class which can handle re-training by layer, prediction,  some explainability and grid search incorporation with logging(all inspired by fastai). I am relatively new to this field(&lt;8 months) and decided to do this to speed up my work on new problem statements. I am curious to know whether it is general practice or you would rather spend the time to write everything from scratch?</p>

username: ptrblck
cooked: <p>I would say it depends on your use case and the availability of the needed functionalities.<br>
Often you’ll find the wanted features in higher level APIs like fastai, Ignite or Lightning.<br>
These APIs provide you a different layer of abstraction so that you can chose the appropriate one.</p>
<p>However, if you’ve already a working code and just need to apply some form of logging, it’s sometimes more convenient (for me at least) to write this functionality myself.</p>
How to set sample input size with different width in different channels to Conv1d in pytorch

username: Yifei
cooked: <p>I’m new at deep learning, and I’m writing a 1D CNN model to train my dataset. Each data in the dataset contains three channels:  <code>(raw_data, sex, age)</code> . The sample rate of  <code>raw_data</code>  is 50Hz, and window size is 2 seconds, which means 100 samples in one data. So my data will look like this  <code>([1,2,3,4,...,100],[1],[20])</code>  where  <code>[1]</code>  means sex and  <code>[20]</code>  means age.</p>
<p>Now I want to put this data into my 1D CNN model. I know that  <code>nn.Conv1d(batch_size, channel, width)</code> . If  <code>width</code>  in all channels are the same, there is no problem. But in my case,  <code>raw_data</code>  contains 100 elements,  <code>sex</code>  and  <code>age</code>  only contain 1 element. How can I put this into my Conv1d layer?</p>
<p>Any help would be appreciated. Thank you!</p>

username: ptrblck
cooked: <p>If I understand the use case correctly, <code>raw_data</code> is some kind of temporal signal, while <code>sex</code> and <code>age</code> are some independent features.<br>
I would try to apply the conv layer on the temporal signal alone, use pass the other features e.g. to a linear layer, and concatenate the outputs afterwards.<br>
Alternatively, you could of course try to add <code>sex</code> and <code>age</code> to the temporal signal, but I’m not sure, if your conv layer will capture the information accordingly.</p>
CUDA out of memory when I store values on CPU

username: edthatalkinhorse
cooked: <p>Hi all</p>
<p>I’m new to this, but I really can’t make sense of this. I’m trying to run inference on a set on images (on GPU), and I want to aggregate the results for later processing. Even though I moved the results to CPU, appending them to a list seems to result in CUDA out of memory. Whats going on here?</p>
<p>‘’’<br>
batch_size = 1<br>
dataset = utils.ImageDataset(PATH_TO_DATA, utils.get_transform(train=False))<br>
data_loader = torch.utils.data.DataLoader(<br>
dataset, batch_size=batch_size, shuffle=False, num_workers=2)<br>
reader = utils.DatasetReader(data_loader)</p>
<h5>Set up model</h5>
<p>num_classes = 2<br>
device = torch.device(‘cuda’) if torch.cuda.is_available() else torch.device(‘cpu’)<br>
cpu_device = torch.device(“cpu”)</p>
<p>model = utils.get_instance_BB_model(num_classes)<br>
model.to(device)<br>
model.load_state_dict(torch.load(PATH_TO_MODEL))<br>
model.eval()<br>
torch.no_grad()<br>
torch.set_num_threads(1)</p>
<h4>Infer on images</h4>
<p>all_out = []<br>
for images, paths in reader.serve_images():<br>
images = list(image.to(device) for image in images)  ## Move images to GPU<br>
outputs = model(images)  ## This is the inference<br>
outputs_cpu = [{k: v.to(cpu_device) for k, v in t.items()} for t in outputs] ## Move results back to CPU</p>
<pre><code>## If I include this line, I get RuntimeError: CUDA out of memory
all_out.append(outputs_cpu)  
</code></pre>
<p>‘’’</p>

username: mmisiur
cooked: <p>Hmm, this is weird - the code seems fine to me. Have you tried debugging it, to see what exactly and when is moved to gpu and back to cpu?</p>
Balance features relevance in unsupervised learning

username: GirardR1006
cooked: <p>Hello everyone,</p>
<p>I would like to perform unsupervised learning with structured data, where some features are more relevant than others.<br>
Is there any way to specify that a feature is more important? It would correspond to giving to the network some prior knowledge; I understand that the network may fit on features that may seem not relevant for humans (which is fine, as long as it still uses the important features).<br>
I could also use a mix of supervised and non-supervised training, but I am curious to see if this feature exist somehow.</p>
<p>Thank you for your replies <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Taking median of multiple models

username: TinfoilHat0
cooked: <p>I’ve independently trained multiple models on different datasets.</p>
<p>Out of these models, I’d like to construct another model where its weights are the element-wise median of my previously trained models.</p>
<p>What’s the best way of doing this?</p>

username: Tahir
cooked: <p>Whether doing will yield any better model, i am not sure.</p>
<p>You can get median of weight tensors, e.g. like <a href="https://stackoverflow.com/questions/54310861/pytorch-median-is-it-bug-or-am-i-using-it-wrong" rel="nofollow noopener">this</a>.</p>