akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Exception handling when using Datapipes

SvenDS9: <p>Hi everyone!</p>
<p>I am trying to load files using the <code>HTTPReaderIterDataPipe</code> from pytorch/data.<br>
How do I handle exceptions (e.g. timeouts) while iterating through the URLs?<br>
I would like to skip the URL causing problems and just move on to the next one. This seems to be impossible when using the functional form.</p>
<p>Is there a DataPipe (catching exceptions and moving on) exactly for this purpose or is my understanding of how to do exception handling when using DataPipes incorrect?</p>

nivek: <p>Hi,</p>
<p>Thanks for using TorchData. While <code>timeout</code> is accepted as an argument, I don’t think there is a built-in way within <code>HTTPReader</code> to handle exception in a bespoke way.</p>
<p>If you would like to skip the URL causing problems, you can consider using a <code>.filter</code> prior to <code>HTTPReader</code> to check if it is possible to establish a connection.</p>
<p>I don’t think that will fully address your issue so I think other options are:</p>
<ol>
<li>Build on top of <code>HTTPReader</code> but overrides its exception handling (probably rewrite <code>__iter__</code>)</li>
<li>Write a new DataPipe that is able to catch exception coming from a source DataPipe
<ul>
<li>I think catching the exception is feasible, I’m less sure about resuming the DataPipe/Iterator after an exception is raised</li>
</ul>
</li>
</ol>
<p>We would accept a PR for 2 if you have a good implementation. Happy to discuss further.</p>
<p>cc: <a class="mention" href="/u/ejguan">@ejguan</a></p>

SvenDS9: <p>Thanks!<br>
option 1 is what I currently use. Wouldn’t it be better to just add this functionality (as an option) to the <code>HTTPReader</code>? I have opened an issue regarding this (<a href="https://github.com/pytorch/data/issues/963" class="inline-onebox" rel="noopener nofollow ugc">Modify exception handling of online Datapipes · Issue #963 · pytorch/data · GitHub</a>)<br>
As I personally do not think that option 2 is that useful (for me at least) if continuing is impossible, I will not be working on this atm. If you think otherwise please let me know. I would be happy to help.</p>
Self-Supervised Contrastive Approaches that don't use large batch size

chandlerbing65nm: <p>This thread is dedicated to exploring the various techniques used in self-supervised contrastive learning that utilize standard batch sizes. I am seeking information on the current methods in this field, specifically those that do not rely on large batch sizes.</p>
<p>I am familiar with the SimSiam paper published by META research, which utilizes 256 batch size for 8-GPUs. However, for individuals with limited resources such as myself, access to a large number of GPUs may not be feasible. As a result, I am interested in learning about other methods that can be used with smaller batch sizes and a single GPU, such as those that would be suitable for training on 1024x1024 input images.</p>
<p>Additionally, I am curious about any more efficient architectures that have been developed in this field. This includes, but is not limited to, techniques used in natural language processing that may have applications in other areas of artificial intelligence.</p>
<p>Edited: English improved by chatGPT - I’m not a native speaker .</p>
Conv2d value cannot be converted to type uint8_t without overflow

lbadams2: <p>I’m passing a  <code>torch.Tensor</code>  with a  <code>dtype</code>  of  <code>torch.uint8</code>  to an  <code>nn.Conv2d</code>  module and it is giving the error</p>
<blockquote>
<p>RuntimeError: value cannot be converted to type uint8_t without overflow: -0.0344873</p>
</blockquote>
<p>My conv2d is defined as  <code>self.conv1 = nn.Conv2d(3, 6, 5)</code> . The error comes in my  <code>forward</code>  method when I pass the tensor to the module like  <code>self.conv1(x)</code> . The tensor has shape (4, 3, 480, 640). I’m not sure how to fix this. Here is the stack trace</p>
<blockquote>
<p>Traceback (most recent call last):</p>
<p>File “cnn.py”, line 54, in </p>
<pre><code>outputs = net(inputs)
</code></pre>
<p>File “/Users/my_repos/venv_projc/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong></p>
<pre><code>result = self.forward(*input, **kwargs)
</code></pre>
<p>File “cnn.py”, line 24, in forward</p>
<pre><code>test = self.conv1(x)
</code></pre>
<p>File “/Users/my_repos/venv_projc/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong></p>
<pre><code>result = self.forward(*input, **kwargs)
</code></pre>
<p>File “/Users/my_repos/venv_projc/lib/python3.7/site-packages/torch/nn/modules/conv.py”, line 345, in forward</p>
<pre><code>return self.conv2d_forward(input, self.weight)
</code></pre>
<p>File “/Users/my_repos/venv_projc/lib/python3.7/site-packages/torch/nn/modules/conv.py”, line 342, in conv2d_forward</p>
<pre><code>self.padding, self.dilation, self.groups)
</code></pre>
<p>RuntimeError: value cannot be converted to type uint8_t without overflow: -0.0344873</p>
</blockquote>

lbadams2: <p>Converting the tensor to a float seemed to fix it <code>self.conv1(x.float())</code></p>

albanD: <p>Hi,</p>
<p>Yes the error message is not very clear but the regular conv2d only works for float and double numbers.</p>

christophezei: <p>so how can I do inference with uint8 ? I have saved integer weights and I want to evaluate my model with that, is it possible?</p>
Merging two tensors based on time series

sashapiv: <p>Hi everyone,</p>
<p>I have two tensors:</p>
<pre><code class="lang-auto">A -&gt; (128,19,3,99,99) #(batch, date, data, data, data)
B -&gt; (128,9,223)  #(batch, date, data)
</code></pre>
<p>After encoding I will have something like that:</p>
<pre><code class="lang-auto">A2 -&gt; (128,19,10) # (batch, date, encoded data with CNN_1)
B2 -&gt; (128,9,10) # (batch, date, encoded data with CNN_2)
C -&gt; (128,28,10) # merge of A2 and B2
</code></pre>
<p>I have the records of time-series of each tensor <code>(A,B)</code>,<br>
I want to be able to merge them on <code>date</code> axis to a single tensor.</p>
<p>Here is an example of the time series of tensor A:<br>
<code>['2021-12-13', '2021-12-28', '2022-01-02', '2022-01-10', '2022-01-20', '2022-01-26', '2022-02-06', '2022-02-14', '2022-02-22', '2022-03-02', '2022-03-17', '2022-03-21', '2022-03-27', '2022-03-30', ...]</code><br>
Tensor B has different dates that do not necessarily fall on the exact dates of tensor A.</p>
<p>So at the end, the axis of date in tensor C will be something like:<br>
<code>A2,A2,B2,A2,B2,A2,B2,B2</code> depends on the original order of dates.</p>
<p>Any help will be great!</p>

ptrblck: <p>You could concatenate both tensors with their dates to a single (unsorted) one, create the sorted indices via e.g.:</p>
<pre><code class="lang-python">date = ['2021-12-13', '2022-01-02', '2022-02-14', '2021-12-28', '2022-02-06', '2022-01-10', '2022-01-20', '2022-01-26', '2022-03-17', '2022-02-22', '2022-03-02', '2022-03-21', '2022-03-27', '2022-03-30']
indices = sorted(range(len(date)), key=lambda index: datetime.strptime(date[index], "%Y-%m-%d"))
print(indices)
# [0, 3, 1, 5, 6, 7, 4, 2, 9, 10, 8, 11, 12, 13]
</code></pre>
<p>and sort the concatenated tensors again.<br>
As you might imagine, this approach would create some overhead in memory, so depending on your use case you might prefer a (slower) loop approach to avoid creating copies of the data.</p>

sashapiv: <p>Thank you!<br>
I keep the order of data so I can just make the sort on a single vector and then use it after the manipulations on datasets to restore desired order…</p>