Data Loading using Pytorch

username: Aniruth_Sundararajan
cooked: <p>I am trying to load my shanghai dataset for training . When I used the normal default Data Loader , i kept getting an empty dataset , even if my path was correct .<br>
So , i circumvented it by using torch.utils.data.subset , and the same dataset was read .</p>
<p>My Problem is , when I check the length of the subset , it is giving me correctly , the length of the dataset I have, but when I run my code for training , I keep getting a None Type Error .</p>
<p>Another issue is , if I give subset size as just 2 , then my training occurs  , only for those 2 samples .</p>
<p>My Code :<br>
shanghai_dataset = SHANGHAITECH(path_ )</p>
<p><span class="hashtag-raw">#data_loader2</span> = torch.utils.data.DataLoader(shanghai_dataset)   <span class="hashtag-raw">#This</span> is loading an empty dataset</p>
<p>data_loader2 = torch.utils.data.Subset(shanghai_dataset  , indices = [i for i in range(0 , 300)] )</p>
<p>num_epochs = 800<br>
for epoch in range(num_epochs):<br>
for batch_idx, batch in enumerate(data_loader2):<br>
video_frames = batch[batch_idx]<br>
print(batch_idx)<br>
print(video_frames)<br>
visual_features = combined_model(video_frames)<br>
visual_features = visual_features.view(-1)</p>
<pre><code>	#anomaly_scores = anomaly_detection_head(video_frames)
  labels=torch.tensor([0.0 , 0.0 , 0.0 , 0.0])
  loss = criterion(visual_features, labels )
  optimizer.zero_grad()
  loss.backward()
  optimizer.step()
  
print(f'Epoch [{epoch + 1}/{num_epochs}], Loss: {loss.item()}')
</code></pre>
<p>Output , if subset length is  300<br>
&lt;First  2 video array values are printed&gt;<br>
2<br>
None<br>
Traceback (most recent call last):<br>
File “/content/gdrive/MyDrive/Final/video_swin_transformer/train.py”, line 289, in <br>
visual_features = combined_model(video_frames)<br>
File “/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1518, in _wrapped_call_impl<br>
return self._call_impl(*args, **kwargs)<br>
File “/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1527, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “/usr/local/lib/python3.10/site-packages/torch/nn/modules/container.py”, line 215, in forward<br>
input = module(input)<br>
File “/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1518, in _wrapped_call_impl<br>
return self._call_impl(*args, **kwargs)<br>
File “/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1527, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “/content/gdrive/MyDrive/Final/video_swin_transformer/mmaction/models/backbones/swin_transformer.py”, line 652, in forward<br>
x = self.patch_embed(x)<br>
File “/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1518, in _wrapped_call_impl<br>
return self._call_impl(*args, **kwargs)<br>
File “/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1527, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “/content/gdrive/MyDrive/Final/video_swin_transformer/mmaction/models/backbones/swin_transformer.py”, line 441, in forward<br>
_, _, D, H, W = x.size()<br>
AttributeError: ‘NoneType’ object has no attribute ‘size’</p>
<p>Output if length of the subset is 2</p>
<p>Normal epoch vs loss gets printed without any issues</p>

username: Roopali_Bhavanasi
cooked: <p>I am facing a similar issue.Even I used torch.utils. DataLoader() but then I was not able read any of the data files even though my path was correct .I tried the same using subset  but then when I give index length  more than two.It is giving me null.</p>