akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to Gain More Efficient Stateful RNN Training with Padded Sequences?

username: shawnbzhang
cooked: <p>I am trying to train an LSTM network in “stateful” mode, i.e. initializing a mini-batch’s hidden/cell state with the previous mini-batch’s last hidden/cell state. However, each of my mini-batches have sequences with <strong>padding</strong> at the ends. This is my problem: <em>I want to avoid initializing the next mini-batch’s hidden/cell state with hidden/cell states from padded time-steps.</em></p>
<p>I am able to achieve this with packing the padded sequence, like so:</p>
<pre><code class="lang-auto">class LayerNorm_LSTM_Stateful(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(LayerNorm_LSTM_Stateful, self).__init__()
        self.layernorm = nn.LayerNorm(input_dim)
        self.lstm = nn.LSTM(input_dim, output_dim, batch_first=True, bidirectional=False)
        
    def forward(self, x, lengths=None, hidden=None):
        x = self.layernorm(x)

        if hidden is None or lengths is None:
            x, hidden = self.lstm(x, hidden)
        else:
            x = pack_padded_sequence(x, lengths, batch_first=True, enforce_sorted=False)
            x, hidden = self.lstm(x, hidden)
            x, _ = pad_packed_sequence(x, batch_first=True)
    
        return x, hidden
</code></pre>
<p>However, with this packing scheme, my training time is almost 1.5x longer than without <code>pack_padded_sequence</code>. Given the auto-regressive nature of RNNs, I am hoping for a more efficient way to train — perhaps indexing the desired timestep’s hidden/cell state?</p>
<p>Does anyone have any ideas? Thank you.</p>

username: googlebot
cooked: <p>Relative performance depends both on tensor shapes and hardware/software versions… One thing to try is batch_first=False mode. If that doesn’t help, then yes, indexing is a valid approach (at least for GRUs, with LSTMs hidden/cell state distinction complicates things).</p>
<p>Some random snippet of indexing (table is (time,batch,*) , gather_pts is a 1d tensor (last indices)):</p>
<pre><code class="lang-auto">brange = torch.arange(batch_size, dtype=torch.int64, device=table.device)
y = table[gather_pts, brange]
</code></pre>
<p>or you can do the same with torch.gather</p>
<p>You can also try <a href="https://github.com/lmnt-com/haste" class="inline-onebox" rel="noopener nofollow ugc">GitHub - lmnt-com/haste: Haste: a fast, simple, and open RNN library</a>, I think it outputs correct states without packing.</p>
Choose the detail page from a lot of pages and locate the summary

username: jobsfan
cooked: <p>I use scrapy crawled a lot of html source from many many websites, there are many kinds of pages. home page, list page, article page(I call it detail page), and other mix pages, etc</p>
<p>I stored the html source page in a jl format file. each line with url, title, html as feilds.</p>
<p>my target is to use AI to recognize all the detail pages and locate the summary part so called the main content. then store them into mysql database.</p>
<p>I am so tired to write crawl rules, regrex one site by one site.</p>
<p>I suppose I need two steps to go</p>
<ol>
<li>create a model to justify which pages are detail pages against a lot of mixed pages, maybe 100000000 pages in total.</li>
</ol>
<p>method 1, parse the html source with marking html tags as 1 and No-tags as 0, which will turn the entire html source into numbers, which can be features.</p>
<p>method 2, use one-hot to mark different html tag as different type, and no html tag as another type.</p>
<p>I think the second way as feature is better.</p>
<p>then mark some with label detail or no-detail.<br>
then use it as train dataset.</p>
<p>I don’t know whether it work or not.</p>
<p>I am not very good at model a Ai learning.<br>
I just read a little about CNN model and RNN model.</p>
<p>CNN is a classifying model and features are numbers.<br>
RNN is a timeline series model and feature are one-hot. and the length of each sample is not the same.</p>
<p>as for me, I need a classifying model with one-hot features.</p>
<p>I hope I can get some advices and help here</p>

username: jobsfan
cooked: <p>Any body can help? Or just suggest a direction will be very much appreciated.</p>
Return both original and augmented images

username: safakkbilici
cooked: <p>Hello,</p>
<p>Since custom dataset returns one sample for each dict entry, I can’t combine both original and augmented images. Here is a code example</p>
<pre><code class="lang-python">def __getitem__(self, idx):
      name = self.ids[idx]
      mask_file = list(self.masks_dir.glob(name + self.mask_suffix + '.*'))
      img_file = list(self.images_dir.glob(name + '.*'))

      mask = self.load(mask_file[0])
      img = self.load(img_file[0])

      img = self.preprocess(img, self.scale, is_mask=False)
      mask = self.preprocess(mask, self.scale, is_mask=True)                     

      if self.transform != None:
          img_aug, mask_aug = self.transform((img, mask))

      if img.flatten().tolist() == img_aug.flatten().tolist():
          return {
              'image': torch.as_tensor(img.copy()).float().contiguous(),
              'mask': torch.as_tensor(mask.copy()).long().contiguous(),
              'filename': str(mask_file[0]).split("/")[-1]
          }

      else:                                                                                          
          img = torch.as_tensor(img.copy()).float().contiguous()                                     
          img_aug = torch.as_tensor(img_aug.copy()).float().contiguous()                             
          mask = torch.as_tensor(mask.copy()).long().contiguous()                                   
          mask_aug = torch.as_tensor(mask_aug.copy()).long().contiguous()                           
          return {                                                                                   
              'image': torch.cat([img, img_aug], dim = 0),                                           
              'mask': torch.cat([mask, mask_aug], dim = 0),                                          
              'filename': str(mask_file[0]).split("/")[-1]                                           
          }
     
dataset = ...
train_dataloader = DataLoader(dataset, shuffle = True, batch_size = 16)
</code></pre>
<p>Is there any easy way to combine both images?</p>

username: ptrblck
cooked: <p>I’m not sure I understand the issue correctly, but are you seeing an issue using your code? If not, I assume you would like to change it somehow, so could you explain what the desired outputs should look like?</p>
CUDA out of memory: why torch.cuda.memory_reserved() so amall?

username: cy-phy
cooked: <p>Dear all,</p>
<p>It seems I can’t get past this anymore (even with multi-gpu) and need to find a solution. Any suggestion will be of great help. Thanks!</p>
<p>I am getting following CUDA memory error:<br>
“RuntimeError: CUDA out of memory. Tried to allocate 720.00 MiB (GPU 0; 39.59 GiB total capacity; 35.53 GiB already allocated; 545.44 MiB free; 37.21 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF”</p>
<p>Question: But I print status message in the code at several places. It seems that “torch.cuda.memory_reserved(0)” is only 2.0 (2.0 MiB…?). Why allocated memory is so small? <img src="https://discuss.pytorch.org/images/emoji/apple/open_mouth.png?v=10" title=":open_mouth:" class="emoji" alt=":open_mouth:"> Also infi like “35.53 GiB already allocated” and “37.21 GiB reserved in total by PyTorch” are not matching with status message from “torch.cuda.memory_reserved(0)”. (Here I am using only one GPU)</p>
<p>**Here is the status print at different places of my code (till before it throws the error):<br>
===&gt;&gt;&gt; info: 1 – GPU status: total: 40536.1875 // reserved: 0.0 // allocated: 0.0 // free: 0.0<br>
===&gt;&gt;&gt; info: 2 – GPU status: total: 40536.1875 // reserved: 0.0 // allocated: 0.0 // free: 0.0<br>
===&gt;&gt;&gt; info: 3 – GPU status: total: 40536.1875 // reserved: 0.0 // allocated: 0.0 // free: 0.0<br>
===&gt;&gt;&gt; info: 4 – GPU status: total: 40536.1875 // reserved: 0.0 // allocated: 0.0 // free: 0.0<br>
===&gt;&gt;&gt; info: 5 – GPU status: total: 40536.1875 // reserved: 0.0 // allocated: 0.0 // free: 0.0<br>
16384 9<br>
===&gt;&gt;&gt; info: 6 – GPU status: total: 40536.1875 // reserved: 2.0 // allocated: 1.00341796875 // free: 0.99658203125<br>
===&gt;&gt;&gt; info: 7 – GPU status: total: 40536.1875 // reserved: 2.0 // allocated: 1.00341796875 // free: 0.99658203125</p>
<p>**Below is my function that prints GPU status:<br>
def get_gpu_info(gpu_n, info=’’):<br>
mb_in_byte = 1048576.0<br>
t = torch.cuda.get_device_properties(0).total_memory / mb_in_byte<br>
r = torch.cuda.memory_reserved(0) / mb_in_byte<br>
a = torch.cuda.memory_allocated(0) / mb_in_byte<br>
f = r-a  # free inside reserved<br>
print(’\n ===&gt;&gt;&gt; info: {} – GPU status: total: {} // reserved: {} // allocated: {} // free: {} '.format(info, t, r, a, f) )</p>

username: ptrblck
cooked: <p>Could you post an executable code snippet showing this behavior, please?</p>