Adjust the bounding box based on ROI of the image

Shantanu_Ghosh: <p>I have an original image of size (3518, 2800) as shown in the uploaded figure original_image.png. Now i have cropped the breast from the image and resized it to (1520, 912) as shown in the uploaded figure cropped_image.png. Following is the code for this preprocessing:</p>
<pre><code class="lang-auto">def np_CountUpContinuingOnes(b_arr):
    # indice continuing zeros from left side.
    # ex: [0,1,1,0,1,0,0,1,1,1,0] -&gt; [0,0,0,3,3,5,6,6,6,6,10]
    left = np.arange(len(b_arr))
    left[b_arr &gt; 0] = 0
    left = np.maximum.accumulate(left)
    # from right side.
    # ex: [0,1,1,0,1,0,0,1,1,1,0] -&gt; [0,3,3,3,5,5,6,10,10,10,10]
    rev_arr = b_arr[::-1]
    right = np.arange(len(rev_arr))
    right[rev_arr &gt; 0] = 0
    right = np.maximum.accumulate(right)
    right = len(rev_arr) - 1 - right[::-1]
    return right - left - 1
</code></pre>
<pre><code class="lang-auto">def ExtractBreast(img):
    img_copy = img.copy()
    img = np.where(img &lt;= 40, 0, img)  # To detect backgrounds easily
    height, _ = img.shape

    # whether each col is non-constant or not
    y_a = height // 2 + int(height * 0.4)
    y_b = height // 2 - int(height * 0.4)
    b_arr = img[y_b:y_a].std(axis=0) != 0
    continuing_ones = np_CountUpContinuingOnes(b_arr)
    # longest should be the breast
    col_ind = np.where(continuing_ones == continuing_ones.max())[0]
    img = img[:, col_ind]

    # whether each row is non-constant or not
    _, width = img.shape
    x_a = width // 2 + int(width * 0.4)
    x_b = width // 2 - int(width * 0.4)
    b_arr = img[:, x_b:x_a].std(axis=1) != 0
    continuing_ones = np_CountUpContinuingOnes(b_arr)
    # longest should be the breast
    row_ind = np.where(continuing_ones == continuing_ones.max())[0]

    return img_copy[row_ind][:, col_ind]

</code></pre>
<pre><code class="lang-auto">def save_imgs(in_path, out_path, SIZE=(912, 1520)):
    dicom = dicomsdl.open(in_path)
    data = dicom.pixelData()
    data = data[5:-5, 5:-5]
    if dicom.getPixelDataInfo()['PhotometricInterpretation'] == "MONOCHROME1":
        data = np.amax(data) - data

    data = data - np.min(data)
    data = data / np.max(data)
    data = (data * 255).astype(np.uint8)

    img = ExtractBreast(data)
    img = cv2.resize(img, SIZE, interpolation=cv2.INTER_AREA)
    cv2.imwrite(out_path, img)

    print(out_path)
</code></pre>
<p>I am calling the save_imgs function to convert the dicom image of (3518, 2800) to png image of (1520, 912). Now my groundtruth bouding boxes are not adjusted by this transformation. I am using the code to transform the bounding box:</p>
<pre><code class="lang-auto">scale_x = 912 / 2800
scale_y = 1520 / 3518

# Resize the bounding box coordinates
resized_xmin = int(bounding_box[0] * scale_x)
resized_ymin = int(bounding_box[1] * scale_y)
resized_xmax = int(bounding_box[2] * scale_x)
resized_ymax = int(bounding_box[3] * scale_y)
</code></pre>
<p>can someone please help to transform the bounding box accordingly?</p>
<p>Here is original_image.png (extracted from dicom with size: (3518, 2800))<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c4652b22f34984f4b0abdc59573e738a49c75737.png" alt="original_image" data-base62-sha1="s1okSmL1Hl26QofSjRarcCLe8o7" width="359" height="418"></p>
<p>Here is cropped_image.png (1520, 912)<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/c/fce4cb9426cdac5649aa68176609575eb40a2a03.png" alt="cropped_image" data-base62-sha1="A5cvS5AKZ2R8yGO3Ip4pQXX73Td" width="286" height="418"></p>

Shantanu_Ghosh: <p>Please help me. <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <aside class="quote no-group" data-username="Shantanu_Ghosh" data-post="1" data-topic="191836">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shantanu_ghosh/48/24369_2.png" class="avatar"> Shantanu_Ghosh:</div>
<blockquote>
<p>Now i have cropped the breast from the image and resized it to (1520, 912)</p>
</blockquote>
</aside>
<p>Cropping an image means you are subtracting parts of the image height and width while resizing would scale the image. In your bounding box manipulation you are only scaling the box coordinates using the original and new size without taking the cropping into account, so you might need to subtract the new cropping borders from the copordinates first before scaling.</p>

Shantanu_Ghosh: <p>Thanks. its actually simple:</p>
<pre><code class="lang-auto">def adjust_bounding_box(original_coords, left_crop, top_crop):
    x1, y1, x2, y2 = original_coords

    x1_new = x1 - left_crop
    y1_new = y1 - top_crop
    x2_new = x2 - left_crop
    y2_new = y2 - top_crop

    return x1_new, y1_new, x2_new, y2_new
</code></pre>
<pre><code class="lang-auto">def ExtractBreast(img, true_bounding_box):
    img_copy = img.copy()
    img = np.where(img &lt;= 40, 0, img)  # To detect backgrounds easily
    height, _ = img.shape

    # whether each col is non-constant or not
    y_a = height // 2 + int(height * 0.4)
    y_b = height // 2 - int(height * 0.4)
    b_arr = img[y_b:y_a].std(axis=0) != 0
    continuing_ones = np_CountUpContinuingOnes(b_arr)
    # longest should be the breast
    col_ind = np.where(continuing_ones == continuing_ones.max())[0]
    img = img[:, col_ind]

    # whether each row is non-constant or not
    _, width = img.shape
    x_a = width // 2 + int(width * 0.4)
    x_b = width // 2 - int(width * 0.4)
    b_arr = img[:, x_b:x_a].std(axis=1) != 0
    continuing_ones = np_CountUpContinuingOnes(b_arr)
    # longest should be the breast
    row_ind = np.where(continuing_ones == continuing_ones.max())[0]
    
    adjusted_coords = adjust_bounding_box(true_bounding_box, col_ind[0], row_ind[0])
    return img_copy[row_ind][:, col_ind], adjusted_coords
</code></pre>
<pre><code class="lang-auto">def save_imgs(in_path, SIZE=(912, 1520)):
    dicom = dicomsdl.open(in_path)
    data = dicom.pixelData()
    data = data[5:-5, 5:-5]
    extracted_breast = data
#     if dicom.getPixelDataInfo()['PhotometricInterpretation'] == "MONOCHROME1":
#         data = np.amax(data) - data

#     data = data - np.min(data)
#     data = data / np.max(data)
#     data = (data * 255).astype(np.uint8)

#     extracted_breast, row_ind, col_ind = ExtractBreast(data)
    
#     print(extracted_breast.shape)
    original_bbox = (
        df['xmin'].iloc[i], df['ymin'].iloc[i], df['xmax'].iloc[i], df['ymax'].iloc[i]
    )
#     adjusted_boxes = adjust_bounding_boxes(
#         original_boxes, extracted_breast.shape, data.shape
#     )
    
    adjusted_bbox = (
        max(0, original_bbox[0] - 5),  # Adjust top, making sure it's not less than 0
        max(0, original_bbox[1] - 5),  # Adjust left, making sure it's not less than 0
        min(data.shape[0], original_bbox[2] - 5),  # Adjust bottom, considering the new image shape
        min(data.shape[1], original_bbox[3] - 5),  # Adjust right, considering the new image shape
    )
    
    original_size = (extracted_breast.shape[1], extracted_breast.shape[0])
    extracted_breast, adjusted_bbox = ExtractBreast(data, adjusted_bbox)
#     new_size = (extracted_breast.shape[1], extracted_breast.shape[0])
#     print(original_size, new_size)
    
#     adjusted_bbox = adjust_bbox(adjusted_bbox, original_size, new_size)
#     print(adjusted_bbox)
    return extracted_breast, adjusted_bbox

</code></pre>

dhruvbird: <p>Instead of writing your own code (which you would need to validate, test, etc…) you could consider using Albumentations which allows performing image transforms which also get applied to labels such as bounding boxes, segmentation masks, etc…</p>

Shantanu_Ghosh: <p>I agree. But we need to do a lot of preprocessing for breast mammograms which are customized and albumentations wont be sufficient.</p>