DataLoader with list of features for FeatureExtract, and ROI

MikeTensor: <p>Hello everyone!</p>
<p>I try to rebuild the following paper work with the KITTI-Dataset.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/7/573bb311fd0b619993556d2bc6f383d056602853.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/573bb311fd0b619993556d2bc6f383d056602853" title="grafik"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/573bb311fd0b619993556d2bc6f383d056602853_2_690x299.jpeg" alt="grafik" data-base62-sha1="crHrCvWgDwNhuXKNM2UsrPsNm1B" width="690" height="299" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/573bb311fd0b619993556d2bc6f383d056602853_2_690x299.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/573bb311fd0b619993556d2bc6f383d056602853_2_1035x448.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/7/573bb311fd0b619993556d2bc6f383d056602853.jpeg 2x" data-dominant-color="DBDDD8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">grafik</span><span class="informations">1377×597 175 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<a href="https://openaccess.thecvf.com/content_ICCV_2019/papers/Zhu_Learning_Object-Specific_Distance_From_a_Monocular_Image_ICCV_2019_paper.pdf" rel="noopener nofollow ugc">Source | Paper</a></p>
<p>Unfortunately I have no clue how to pass the different features to the DataLoader and recall them by the different steps (image*, bounding boxes for ROI Polling, types/classes, and additional features of the KITTI_Dataset) and of course the labels: distance (which is already concated in my textfiles).</p>
<p>*<em>Image is the “main feature” from where the Features should be extracted by the FeactureExtractor.</em><br>
<em>My CustomDataset-Class looks like this right now:</em></p>
<pre><code class="lang-auto">import csv
import os
import pandas as pd
from typing import Any, Callable, List, Optional, Tuple


from PIL import Image
from torchvision.datasets.vision import VisionDataset

# df_dict = {'type': 0, 'truncated': 1, 'occluded': 2, 'alpha': 3, 'bbox': 4, 'dimensions': 5, 'location': 6, 'rotation_y': 7, 'forward_distance': 8, 'forward_keypoint': 9}
# df_types_dict = {'Car': 0, 'Van': 1, 'Truck': 2,'Pedestrian': 3, 'Person_sitting': 4, 'Cyclist': 5, 'Tram': 6, 'Misc': 7, 'DontCare': 8}

class KittiDist(VisionDataset):
    """KittiDist-Class for `KittiDist`_Dataset.

    It corresponds to the "left color images of object" dataset, for object detection.

    Args:
        root (string): Root directory where images are downloaded to.
            Expects the following folder structure if download=False:

            .. code::

                &lt;root&gt;
                    └── KittiDist --&gt; During Test: "TEST_SET"
                        ├── images
                        └── distLabels

        develop (bool, optional): If true, another File-Path is used. Could be used if
            during development another (smaller, special selected) Dataset-Folder is needed.
        train (bool, optional): Use ``train`` split if true, else ``test`` split.
            Defaults to ``train``.
        transform (callable, optional): A function/transform that takes in a PIL image
            and returns a transformed version. E.g, ``transforms.PILToTensor``
        target_transform (callable, optional): A function/transform that takes in the
            target and transforms it.
        transforms (callable, optional): A function/transform that takes input sample
            and its target as entry and returns a transformed version.


    """


    # Dataset
    # Numbers of images: 007480.png (last image)


    image_dir_name = "images"
    labels_dir_name = "distLabel"

    main_dir_name = "KITTI-Dist"
    develop_dir_name = "TEST_SET"
    # Set DevMode (Small Dataset), else whole Dataset
    # develop: bool = True

    def __init__(
            self,
            root: str,
            develop: bool = False,
            train: bool = True,
            transform: Optional[Callable] = None,
            target_transform: Optional[Callable] = None,
            transforms: Optional[Callable] = None,

    ):
        super().__init__(
            root,
            transform=transform,
            target_transform=target_transform,
            transforms=transforms,
        )

        self.images = [] # A list of all images
        self.targets = [] # A list of all Anno-Files (distLabel), xy.targets = [idx] will return a certain path.
        self.root = root
        self.train = train
        self.develop = develop
        self._location = self.develop_dir_name if self.develop else self.main_dir_name


        image_dir = os.path.join(self.root, self._location, self.image_dir_name)

        if self.train:
            labels_dir = os.path.join(self.root, self._location, self.labels_dir_name)

        # For every single Image in the path "image_dir" will be appended the Ref. to the "image"-List.
        for img_file in os.listdir(image_dir):
            self.images.append(os.path.join(image_dir, img_file))
            if self.train:
                self.targets.append(os.path.join(labels_dir, f"{img_file.split('.')[0]}.txt"))
          
    def __getitem__(self, index: int) -&gt; Tuple[Any, Any]:
        """Get item at a given index.

        Args:
            index (int): Index
        Returns:
            tuple: (image, target), where
            target is a list of dictionaries with the following keys:

            - type: str
            - truncated: float
            - occluded: int
            - alpha: float
            - bbox: float[4]
            - dimensions: float[3]
            - locations: float[3]
            - rotation_y: float
            - forward_distance: float
            - forward_keypoint: float[2]

        """
        image = Image.open(self.images[index])
        target = self._parse_target(index) if self.train else None
        if self.transforms:
            image, target = self.transforms(image, target)
        return image, target


    def _parse_target(self, index: int) -&gt; List:
        target = []
        with open(self.targets[index]) as inp:
            content = csv.reader(inp, delimiter=" ")
            for line in content:
                target.append(
                    {
                        "type": line[0],
                        "truncated": float(line[1]),
                        "occluded": int(line[2]),
                        "alpha": float(line[3]),
                        "bbox": [float(x) for x in line[4:8]],
                        "dimensions": [float(x) for x in line[8:11]],
                        "location": [float(x) for x in line[11:14]],
                        "rotation_y": float(line[14]),
                        ## Add distance and coordinates for the LIDAR-Point to which the computed distance belongs
                        "forward_distance": float(line[15]),
                        ## This should be 16:18; [18] because the last number ends on [17]. Compair with location, which ends with [13].
                        "forward_keypoint": [float(x) for x in line[16:18]],

                    }
                )
        return target

    def __len__(self) -&gt; int:
        return len(self.images)
</code></pre>
<p>The “TEST_SET” is just another folder, with less images during development.</p>
<p>Additionally my resize/augmentation function, which is not implemented into the DatasetClass:</p>
<pre><code class="lang-auto">import albumentations 
import cv2 
import numpy as np 

def resize_img_bb_kyp(img_arr, bboxes, keypoints, height, width, flip_prob):
    """
    :param img_arr: original image as a numpy array
    :param bboxes: bboxes as numpy array where each row is 'x_min', 'y_min', 'x_max', 'y_max'.
    :param h: resized height dimension of image
    :param w: resized weight dimension of image
    :param keypoints: keypoints as numpy array, each row is 'x' 'y'
    :return: dictionary containing {image:transformed (flipped?), bboxes_transformed:['x_min', 'y_min', 'x_max', 'y_max', "keypoints_transformed]}
    Adapted from: https://sheldonsebastian94.medium.com/resizing-image-and-bounding-boxes-for-object-detection-7b9d9463125a
    """
    # resize transform pipeline
    transform_img_bb_kyp = albumentations.Compose(
        [albumentations.Resize(height=height, width=width, always_apply=True),
         albumentations.HorizontalFlip(p=flip_prob)],
        bbox_params=albumentations.BboxParams(format='pascal_voc', label_fields=[]),
        keypoint_params=albumentations.KeypointParams(format="xy"))

    transformed_img_bb_kyp = transform_img_bb_kyp(image=img_arr, bboxes=bboxes, keypoints=keypoints)

    return transformed_img_bb_kyp

</code></pre>
<p>So as an overwiew:<br>
main feature: image – for CNN feature extractor<br>
feature: BoundingBoxes – for ROI pooling<br>
feature: Keypoints (maye used later)<br>
feature: type (object classes) – not sure; if I really use it, since I want to use a classifier<br>
label: distance</p>
<p>I have one big folder with all the images; another folder for the Textfiles (which containts the features, and the label).</p>
<p>So I struggle with the following questions:</p>
<ol>
<li>How to pass the several features and the label (distance) to the DataLoader?</li>
<li>How does the DataLoader look like?</li>
<li>How do I access the different features after handing it to the DataLoader?</li>
<li>Is there a smooth way, how to integrate my resize/augmentation-function into the Dataset-Class, or do I have to use pytrochs Transform? ( I am using my own resize function, since pyTorchs Transform doesnt transform the coordinates of the BoundingBoxes, and keypoints.</li>
</ol>
<p>I would really appreciate any help.<br>
– Thanks and enjoy your day.</p>
<p>P.S. This issue didn’t really helped me …</p><aside class="quote quote-modified" data-post="1" data-topic="26605">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/deeplearner17/48/3185_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/make-a-tensordataset-and-dataloader-with-multiple-inputs-parameters/26605">Make a TensorDataset and Dataloader with multiple inputs parameters</a> 
  </div>
  <blockquote>
    Hello, 
I have a dataset composed of   labels,features,adjacency matrices, laplacian  graphs in numpy format. 
I would like to build a torch.utils.data.data_utils.TensorDataset() and torch.utils.data.DataLoader() that can take labels,features,adjacency matrices, laplacian  graphs. 
To do so, l have tried the following 

import numpy as np
import torch.utils.data as data_utils

# get the numpy data
 labels_train,features_train,adjacency_train,laplacian_train=train
 labels_test,features_test,adjac…
  </blockquote>
</aside>


MikeTensor: <p>Nobody?  <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"><br>
Could you maybe give me a hint <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>You wouldn’t change anything in the <code>DataLoader</code>, but would implement a custom <code>Dataset</code> returning the desired features, targets, etc.<br>
I’m unsure how to understand points 1, 2, and 3, since you would directly pass the <code>Dataset</code> to the <code>DataLoader</code>.</p>
<p>For 4: Yes, <code>torchvision.transforms</code> or transformations from e.g. <code>augmentations</code> should work.</p>

MikeTensor: <p>Thank you very much for your reply! <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
<em>1) How to pass the several features and the label (distance) to the DataLoader?</em><br>
My CustomDataset return different features and one label.</p>
<p><strong>1. feature:</strong> image – for CNN feature extractor<br>
<strong>2. feature:</strong> BoundingBoxes – for ROI pooling<br>
<strong>3. feature:</strong> Keypoints (maye used later)<br>
<strong>4. feature:</strong> type (object classes) – not sure; if I really use it, since I want to use a classifier<br>
<strong>y: label:</strong> distance</p>
<p>So can I just return from the CustomDataset “image” and “target” (“target” as a dict or pandas DataFrame with BBoxes, Keypoints, type, and the label) and the DataLoader is fine with this?</p>
<p><em>2) How does the DataLoader look like?</em><br>
So i don’t have to adapt the DL?</p>
<p><em>3) How do I access/use the different features after handing it to the DataLoader?</em><br>
So how do I hand the different features in particular over where they are needed, ergo how to adress them?<br>
→ Image-Feature to the CNN<br>
→ BoundingBoxes (which has to be a Tensor) to the <code>roi-align</code> Function<br>
→ Keypoints …</p>
<p>→ label: “distance” for the loss-calc?</p>
<p>All the examples I saw work with <code>X</code> and <code>y</code> since there is one feature and one label.<br>
I have to tell the CNN: Use feature “image”. The <code>roi-align</code> to use the BoundingBoxes, …<br>
I don’t really understand how to do this, since there is just one <code>y</code> as a “big Tensor”?? with all features in it?</p>
<p>Like it’s totally different from accessing a pandas-DataFrame (mydataframe.boxes, …) or Dictionary (mydataset[1]) …</p>
<p>Hope that makes it a bit easier to understand, what my problem is …</p>
<p>Thanks in anyway for helping out <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> and greetings.</p>