Testing stopping at same row

username: Jordan_Howell
cooked: <p>Hello,</p>
<p>General question:</p>
<p>What do I look for if a model that is predicting a test set stops at the same row every time?</p>
<p>Nothing looks wrong to me.  No data is missing. The image is there that’s supposed to be pulling.  I tried removing that row, the same number index (new observation since the old was removed) is still getting held up.</p>
<p>I’m not sure how to trouble shoot.</p>

username: ptrblck
cooked: <p>I would start by disabling all multi-processing and multi-threading, if it’s used in your current script.<br>
If that doesn’t help, check your memory usage and make sure your are not close to running our of memory.<br>
E.g. if you are using the CPU and have a data leak (e.g. by storing the computation graph), your swap might be used, which might look like a hang, but is indeed just slow execution.</p>
<p>Recently we had a potential hang in this forum, where the validation loop was executed after a certain number of training iterations, and was unfortunately slow and didn’t print out anything, so that it again looked like a hang.</p>

username: Jordan_Howell
cooked: <p>Thanks. I’m not using multi-threading and I’m executing on GPU.  I do have it print out which row it’s on.   Is there anything else it can print so I can see it doing something?  I let the test run for  4 days at one point on the same row.</p>

username: ptrblck
cooked: <p>Could you try to execute this particular line in isolation and check, if the training would run?<br>
What kind of data are you dealing with and what do the rows represent?</p>

username: Jordan_Howell
cooked: <p>I ran 15 rows with the delinquent in the middle.  It stopped again.  I ran just the delinquent and it hangs.  I ran my validation set through and constantly hangs at 68,040.</p>
<p>I exported the deliquent out to a .csv.  The data is 6 numerical columns all centered and scaled and 8 categorical columns that are made into a categorical embedding layer. Finally, there is one image denoted in the data frame with the path to that image and I verified the images exist.</p>
<p>Here is the custom dataset (the training ran just fine):</p>
<pre><code class="lang-auto">class image_Dataset(Dataset):
    '''
    image class data set   
    
    '''
    def __init__(self, data, transform = None):
        '''
        Args:
        ------------------------------------------------------------
            data = dataframe
            image = column in dataframe with absolute path to the image
            label = column in dataframe that is the target classification variable
            numerical_columns =  numerical columns from data
            categorical_columns = categorical columns from data
            policy = ID variable
            
        '''
        self.image_frame = data
        self.transform = transform
        
    def __len__(self):
        return len(self.image_frame)
    
    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()
         
        label = self.image_frame.loc[idx, 'target']
        
#         if self.image_frame.loc[idx, 'Roof'].isna() == True:
#             pic = np.ones(3, 224,224)
#             img = pic
#             image = torch.tensor(img)
#         else:
        pic = Path(self.image_frame.loc[idx, 'location'])
        img = Image.open(pic)
        image = self.transform(img)
        
        policy = self.image_frame.loc[idx, 'policy']
        
        numerical_data = self.image_frame.loc[idx, numerical_columns]
        
        numerical_data = torch.tensor(numerical_data, dtype = torch.float)
        
        for category in non_loca_cat_columns:
            self.image_frame[category] = self.image_frame[category].astype('category')
            
            self.image_frame[category] = self.image_frame[category].astype('category').cat.codes.values
        
            
        categorical_data = self.image_frame.loc[idx, non_loca_cat_columns]
        categorical_data = torch.tensor(categorical_data, dtype = torch.int64)
            
        return image, label, policy, categorical_data , numerical_data
'''</code></pre>

username: ptrblck
cooked: <p>So the script hangs just in this special row.<br>
Could you rip out the loading and processing code into a standalone script and try to see, if you are able to load and process this particular sample?<br>
I would recommend to use an IDE and step line by line to isolate the hanging part (or add print statements to the script).</p>

username: Jordan_Howell
cooked: <p>Sorry for the late reply.  I took my kids camping.  I ran the functions outside of the data loader and they still got hung at index 879.  I don’t know why. I see the values, i see the picture.  So removed that record by using <code>test_hang = test.drop(test.index[-879])</code>.</p>
<p>Now I get the following:</p>
<pre><code class="lang-auto">Traceback (most recent call last):

  File "C:\Users\JORDAN.HOWELL.GITDIR\Documents\GitHub\Inspection_Photo_Pytorch_Model\untitled0.py", line 555, in &lt;module&gt;
    for image, label, policy, categorical_data, numerical_data in test_loader_roof:

  File "C:\Users\JORDAN.HOWELL.GITDIR\AppData\Local\Continuum\anaconda3\envs\torch_env\lib\site-packages\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()

  File "C:\Users\JORDAN.HOWELL.GITDIR\AppData\Local\Continuum\anaconda3\envs\torch_env\lib\site-packages\torch\utils\data\dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration

  File "C:\Users\JORDAN.HOWELL.GITDIR\AppData\Local\Continuum\anaconda3\envs\torch_env\lib\site-packages\torch\utils\data\_utils\fetch.py", line 47, in fetch
    return self.collate_fn(data)

  File "C:\Users\JORDAN.HOWELL.GITDIR\AppData\Local\Continuum\anaconda3\envs\torch_env\lib\site-packages\torch\utils\data\_utils\collate.py", line 79, in default_collate
    return [default_collate(samples) for samples in transposed]

  File "C:\Users\JORDAN.HOWELL.GITDIR\AppData\Local\Continuum\anaconda3\envs\torch_env\lib\site-packages\torch\utils\data\_utils\collate.py", line 79, in &lt;listcomp&gt;
    return [default_collate(samples) for samples in transposed]

  File "C:\Users\JORDAN.HOWELL.GITDIR\AppData\Local\Continuum\anaconda3\envs\torch_env\lib\site-packages\torch\utils\data\_utils\collate.py", line 81, in default_collate
    raise TypeError(default_collate_err_msg_format.format(elem_type))

TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class 'pandas.core.series.Series'&gt;
</code></pre>

username: ptrblck
cooked: <p><code>pandas.DataFrame.drop</code> should return a <code>DataFrame</code> (not sure why the error mentions a <code>Series</code> object), so you might need to convert it to an array (via <code>.values</code>) or tensor.</p>

username: Jordan_Howell
cooked: <p>So when I type:</p>
<pre><code class="lang-auto">pic = Path(test_hang.loc[879:880, 'location'].values[0])
image = self.train_transform(Image.open(pic))
</code></pre>
<p>in the console, I get a transformed tensor back.</p>
<p>When I run what is in my custom data set, I get the following error:</p>
<pre><code class="lang-auto">File "&lt;ipython-input-23-9f4883f14d32&gt;", line 175, in &lt;module&gt;
    for image, label, policy, categorical_data, numerical_data in test_loader_roof:

  File "C:\Users\JORDAN.HOWELL.GITDIR\AppData\Local\Continuum\anaconda3\envs\torch_env\lib\site-packages\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()

  File "C:\Users\JORDAN.HOWELL.GITDIR\AppData\Local\Continuum\anaconda3\envs\torch_env\lib\site-packages\torch\utils\data\dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration

  File "C:\Users\JORDAN.HOWELL.GITDIR\AppData\Local\Continuum\anaconda3\envs\torch_env\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]

  File "C:\Users\JORDAN.HOWELL.GITDIR\AppData\Local\Continuum\anaconda3\envs\torch_env\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in &lt;listcomp&gt;
    data = [self.dataset[idx] for idx in possibly_batched_index]

  File "&lt;ipython-input-23-9f4883f14d32&gt;", line 32, in __getitem__
    pic = Path(self.image_frame.loc[idx, 'location'].values[0])

AttributeError: 'str' object has no attribute 'values'
</code></pre>
<p>Here is my <code>getitem</code> once more:</p>
<pre><code class="lang-auto">class image_Dataset(Dataset):
    '''
    image class data set   
    
    '''
    def __init__(self, data, transform = None):
        '''
        Args:
        ------------------------------------------------------------
            data = dataframe
            image = column in dataframe with absolute path to the image
            label = column in dataframe that is the target classification variable
            numerical_columns =  numerical columns from data
            categorical_columns = categorical columns from data
            policy = ID variable
            
        '''
        self.image_frame = data
        self.transform = transform
        
    def __len__(self):
        return len(self.image_frame)
    
    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()

         
        label = self.image_frame.loc[idx, 'target']
        label = np.asarray(label)

        pic = Path(self.image_frame.loc[idx, 'location'].values[0])
        image = self.transform(Image.open(pic))
        
        policy = self.image_frame.loc[idx, 'policy']
        policy = np.asarray(policy)

        
        numerical_data = self.image_frame.loc[idx, numerical_columns]

        
        numerical_data = torch.tensor(numerical_data, dtype = torch.float)

        
        for category in non_loca_cat_columns:
            self.image_frame[category] = self.image_frame[category].astype('category')
            
            self.image_frame[category] = self.image_frame[category].astype('category').cat.codes.values
        
            
        categorical_data = self.image_frame.loc[idx, non_loca_cat_columns]
        categorical_data = np.asarray(categorical_data)

        return image, label, policy, categorical_data , numerical_data
</code></pre>
<p>The image seems to be working in the console but no the custom data set.  Is the index  right under the <code>getitem</code> coded correct?</p>

username: ptrblck
cooked: <p>What is the difference between the run in the terminal and your custom dataset?<br>
Apparently <code>self.image_frame.loc[idx, 'location']</code> returns a <code>str</code> for the current <code>idx</code>.<br>
I would recommend to check the <code>image_frame</code> in an IDE and make sure the <code>location</code> column contains the expected values, which can be properly indexed.<br>
The previous error pointed to a <code>pd.Series</code> object, while now a <code>str</code> is returned, to maybe it would be easier to create a plain Python <code>list</code> containing all image paths?</p>

username: Jordan_Howell
cooked: <p>Doesn’t the <code>getitem</code> have to have the</p>
<pre><code class="lang-auto">if torch.is_tensor(idx):
            idx = idx.tolist()
</code></pre>
<p>lines in it?</p>
<p>Disregard. I think I got it to work:</p>
<pre><code class="lang-auto">class image_Dataset(Dataset):
    '''
    image class data set   
    
    '''
    def __init__(self, data, transform = None):
        '''
        Args:
        ------------------------------------------------------------
            data = dataframe
            image = column in dataframe with absolute path to the image
            label = column in dataframe that is the target classification variable
            numerical_columns =  numerical columns from data
            categorical_columns = categorical columns from data
            policy = ID variable
            
        '''
        self.image_frame = data
        self.transform = transform
        
    def __len__(self):
        return len(self.image_frame)
    
    def __getitem__(self, idx):
    
        label = list(self.image_frame.loc[:, 'target'])

        #pic = Path(self.image_frame.loc[idx, 'location'])
        image = self.transform(Image.open(pathlib.Path(self.image_frame.loc[:, 'location'].values[0])))
        
        policy = list(self.image_frame.loc[:, 'policy'])
        
        numerical_data = np.asarray((self.image_frame.loc[:, numerical_columns]))

        
        #numerical_data = torch.tensor(numerical_data, dtype = torch.float)

        
        for category in non_loca_cat_columns:
            self.image_frame[category] = self.image_frame[category].astype('category')
            
            self.image_frame[category] = self.image_frame[category].astype('category').cat.codes.values
        
            
        categorical_data = np.asarray(self.image_frame.loc[:, non_loca_cat_columns])
        
        label = torch.FloatTensor(label)
        image = torch.FloatTensor(image)
        policy = torch.FloatTensor(policy)
        numerical_data = torch.FloatTensor(numerical_data)


        return image, label, policy, categorical_data , numerical_data
</code></pre>