How to attach index from pandas DF image outputs from an autoencoder or CNN

Jordan_Howell: <p>Hello,</p>
<p>I’m pulling photos from a train directory by label (train/1 or train/0) using the datasets.ImageFolder command.  I have a df or csv that has each image, target label and the location in the directory it can be found.  That said, how would I attach an output of 1000 encoder features per image or a CNN prediction back to the index as I’m using an ID associated with picture as the index?  The ultimate goal is to concatenate the output with other model outputs also associated with that id found in the index.</p>

ptrblck: <p>If you are using <code>pandas</code> to read the <code>csv</code> file, you might use <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.loc.html" rel="nofollow noopener"><code>pd.DataFrame.loc</code></a> to write to the desired image ID.<br>
Since <code>pandas</code> uses numpy arrays internally, you could access the array via <code>tensor.numpy()</code>.</p>
<p>I’m not sure, if you are also asking about the PyTorch code to extract the features, so let me know, if you need more information.</p>

Jordan_Howell: <p>Thank you for replying.  I’m trying to encode two different sets of pictures at 274*274, 3 channels, down to 1000 features, then concatenation those features with a tabular data set to ultimately predict a binary outcome (kind of like the Ludwig model).  The I’d is 1 to N pictures where 1 ID (set as the index) could have up to four pictures.  I plan to average over the features for IDs with more than one picture.</p>
<p>The tabular dataset also has the IDs as the index.  I’m just not sure how to join with the final feature sets from the encoded pictures without that ID on the photo side.</p>
<p>So, yes, any type of code would help to assign policy IDs with the picture.  The csv’s (and dataframe) column headers are:<br>
Index: ID<br>
File name<br>
Location<br>
Label</p>
<p>The directory the images are in is split between the positive class and negative class.</p>

Jordan_Howell: <p>So I’ve tried to follow <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="noopener nofollow ugc">this</a> tutorial and I keep getting an error.  My csv file has a few changes in the ordering of the columns:<br>
ID<br>
File name<br>
description<br>
target<br>
root_dir (location)</p>
<p>This is my data class I tried to do following the tutorial:</p>
<blockquote>
<p>class roofDataset(Dataset):<br>
‘’’<br>
roof data class</p>
<pre><code>'''
def __init__(self, csv_file, transform = None):
    '''
    Args:
        csv_file (string): Path to the csv file with annotations.
        root_dir (string): Directory with all the images.
        transform (callable, optional): Optional transform to be applied
            on a sample.
    '''
    self.roof_frame = pd.read_csv(csv_file)
    self.root_dir = self.roof_frame
    self.transform = transform
    
def __len__(self):
    return len(self.roof_frame)

def __getitem__(self, idx):
    if torch.is_tensor(idx):
        idx = idx.tolist()
        
    img_name = os.path.join(self.root_dir.iloc[idx,5], self.roof_frame.iloc[idx, 1])
    image = io.imread(img_name)
    roof = self.roof_frame.iloc[idx, 1:]
    roof = np.array([roof])
    sample = {'image': image, 'roof': roof}

    if self.transform:
        sample = self.transform(sample)

    return sample
</code></pre>
</blockquote>
<p>Then I ran the following to test:</p>
<blockquote>
<p>roof_dataset = roofDataset(csv_file=‘D:\CIS inspection images 0318\self_build\train\train_roof_images.csv’,<br>
transform = train_transform)</p>
<p>fig = plt.figure()</p>
<p>for i in range(len(roof_dataset)):<br>
sample = roof_dataset[i]</p>
<pre><code>print(i, sample['image'].shape, sample['roof'].shape)

ax = plt.subplot(1, 4, i + 1)
plt.tight_layout()
ax.set_title('Sample #{}'.format(i))
ax.axis('off')
show_landmarks(**sample)

if i == 3:
    plt.show()
    break
</code></pre>
</blockquote>
<p>And now I get this error:</p>
<blockquote>
<hr>
<p>TypeError                                 Traceback (most recent call last)<br>
 in <br>
5<br>
6 for i in range(len(roof_dataset)):<br>
----&gt; 7     sample = roof_dataset[i]<br>
8<br>
9     print(i, sample[‘image’].shape, sample[‘roof’].shape)</p>
<p> in <strong>getitem</strong>(self, idx)<br>
23             idx = idx.tolist()<br>
24<br>
—&gt; 25         img_name = os.path.join(self.root_dir.iloc[idx,5], self.roof_frame.iloc[idx, 1])<br>
26         image = io.imread(img_name)<br>
27         roof = self.roof_frame.iloc[idx, 1:]</p>
<p>C:\ProgramData\Anaconda3\lib\ntpath.py in join(path, *paths)<br>
113         return result_drive + result_path<br>
114     except (TypeError, AttributeError, BytesWarning):<br>
 → 115         genericpath._check_arg_types(‘join’, path, *paths)<br>
116         raise<br>
117</p>
<p>C:\ProgramData\Anaconda3\lib\genericpath.py in _check_arg_types(funcname, *args)<br>
147         else:<br>
148             raise TypeError(‘%s() argument must be str or bytes, not %r’ %<br>
 → 149                             (funcname, s.<strong>class</strong>.<strong>name</strong>)) from None<br>
150     if hasstr and hasbytes:<br>
151         raise TypeError(“Can’t mix strings and bytes in path components”) from None</p>
<p>TypeError: join() argument must be str or bytes, not ‘int64’</p>
</blockquote>

Jordan_Howell: <p>So, I think I have the class working correctly…</p>
<blockquote>
<p>class roofDataset(Dataset):<br>
‘’’<br>
roof data class</p>
<pre><code>'''
def __init__(self, csv_file, transform = None):
    '''
    Args:
        csv_file (string): Path to the csv file with annotations.
        transform (callable, optional): Optional transform to be applied
            on a sample.
    '''
    self.roof_frame = pd.read_csv(csv_file)
    self.transform = transform
    
def __len__(self):
    return len(self.roof_frame)

def __getitem__(self, idx):
    if torch.is_tensor(idx):
        idx = idx.tolist()
     
    root_dir = self.roof_frame.iloc[idx,4]
    pic = self.roof_frame.iloc[idx, 1]
    label = self.roof_frame.iloc[idx, 3]
    img_name = os.path.join(root_dir, pic)
    image = Image.open(img_name)
    policy = self.roof_frame.iloc[idx, 0]
    sample = {'image': image, 'policy': policy, 'label':label}

    if self.transform:
        image = self.transform(image)

    return image, label, policy
</code></pre>
</blockquote>
<p>So this returns the image, the label and the policy which is the ID I was referring to.  I’ve made modifications to the tutorial and got everything to print as expected with no errors!</p>
<p>I’m not sure how I’ll now feed this into an auto encoder but, the first step is done!  Thanks for the hints.</p>