How to warp an image using deformed mesh

username: ronghui
cooked: <p>I am trying to reproduce the project <a href="https://github.com/mhashas/Document-Image-Unwarping-pytorch" rel="noopener nofollow ugc">document unwarping</a>. Because i dont have enoguh data. I was trying to make fake dataset. Here is my code.</p>
<pre><code class="lang-auto">def create_grid(width=100, height=100):
    mr = width
    mc = height

    xx = np.arange(mr-1, -1, -1)
    yy = np.arange(0, mc, 1)
    [Y, X] = np.meshgrid(xx, yy)
    ms = np.transpose(np.asarray([X.flatten('F'), Y.flatten('F')]), (1, 0))

    perturbed_mesh = ms
    nv = np.random.randint(20) - 1
    for k in range(nv):
        # Choosing one vertex randomly
        vidx = np.random.randint(np.shape(ms)[1])
        vtex = ms[vidx, :]
        # Vector between all vertices and the selected one
        xv = perturbed_mesh - vtex
        # Random movement
        mv = (np.random.rand(1, 2) - 0.5)*20
        hxv = np.zeros((np.shape(xv)[0], np.shape(xv)[1] + 1))
        hxv[:, :-1] = xv
        hmv = np.tile(np.append(mv, 0), (np.shape(xv)[0], 1))
        d = np.cross(hxv, hmv)
        d = np.absolute(d[:, 2])
        d = d / (np.linalg.norm(mv, ord=2))
        wt = d

        curve_type = np.random.rand(1)
        if curve_type &gt; 0.3:
            alpha = np.random.rand(1) * 20 + 20
            wt = alpha / (wt + alpha)
        else:
            alpha = np.random.rand(1) + 1
            wt = 1 - (wt / 100)**alpha
        msmv = mv * np.expand_dims(wt, axis=1)
        perturbed_mesh = perturbed_mesh + msmv
        
    perturbed_mesh[:, 0] = np.where(perturbed_mesh[:, 0] &gt; height, height, perturbed_mesh[:, 0])
    perturbed_mesh[:, 1] = np.where(perturbed_mesh[:, 1] &gt; width, width, perturbed_mesh[:, 1])
    perturbed_mesh[:, 0] = np.where(perturbed_mesh[:, 0] &lt; 0, 0, perturbed_mesh[:, 0])
    perturbed_mesh[:, 1] = np.where(perturbed_mesh[:, 1] &lt; 0, 0, perturbed_mesh[:, 1])

#     plt.scatter(perturbed_mesh[:, 0], perturbed_mesh[:, 1], c=np.arange(0, mr*mc))
#     plt.show()

    return perturbed_mesh[:, 0], perturbed_mesh[:, 1]

    src_img = cv2.imread('source.jpg')
    src_img = cv2.flip(src_img, 0)
    scale_percent = 30 # percent of original size
    width = int(src_img.shape[1] * scale_percent / 100)
    height = int(src_img.shape[0] * scale_percent / 100)
    dim = (width, height)
    src_img = cv2.resize(src_img, dim, interpolation=cv2.INTER_AREA)

    height, width, _ = src_img.shape
    dh = height // 10
    dw = width // 10
    img = cv2.copyMakeBorder(src_img, dh, dh, dw, dw, borderType=cv2.BORDER_CONSTANT, value=(0,0,0))
    nh, nw, _ = img.shape
    xs, ys = create_grid(nh, nw)
    xs = xs.reshape(nh, nw).astype(np.float32)
    ys = ys.reshape(nh, nw).astype(np.float32)
    dst = cv2.remap(img, xs, ys, cv2.INTER_CUBIC, borderValue=0)
    cv2.imwrite('result.jpg', dst)
</code></pre>
<p>Source image is<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/8/38e3028fc916afac3658350149c129a100692855.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/38e3028fc916afac3658350149c129a100692855" title="source"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/38e3028fc916afac3658350149c129a100692855_2_271x500.jpeg" alt="source" data-base62-sha1="87f6vMcuDxyy56619VGPlAL3RTn" width="271" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/38e3028fc916afac3658350149c129a100692855_2_271x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/38e3028fc916afac3658350149c129a100692855_2_406x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/38e3028fc916afac3658350149c129a100692855_2_542x1000.jpeg 2x" data-dominant-color="F1F0F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">source</span><span class="informations">951×1750 158 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
But my result is<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/d/8d44db86b7afc8981f980dc832dddf55613b0d05.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8d44db86b7afc8981f980dc832dddf55613b0d05" title="result"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d44db86b7afc8981f980dc832dddf55613b0d05_2_271x500.jpeg" alt="result" data-base62-sha1="k9IQH0HP1Dwk7WZCAy0Vzk6e9pz" width="271" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d44db86b7afc8981f980dc832dddf55613b0d05_2_271x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d44db86b7afc8981f980dc832dddf55613b0d05_2_406x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d44db86b7afc8981f980dc832dddf55613b0d05_2_542x1000.jpeg 2x" data-dominant-color="929192"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">result</span><span class="informations">951×1750 250 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Expected result is<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/3/c36e11c19bb4eeb1674d5ffe44e9c0f14d2ed6b3.png" alt="target" data-base62-sha1="rSQVC3qWyC870kR4drHa9MgjKGD" width="298" height="499"></p>
<p>What may go wrong in my code? Or is there any pytorch equivalent to make such fake dataset? Here is the <a href="https://openaccess.thecvf.com/content_cvpr_2018/papers/Ma_DocUNet_Document_Image_CVPR_2018_paper.pdf" rel="noopener nofollow ugc">paper</a> talking about how to make fake data.</p>

username: ptrblck
cooked: <p>Based on your result vs. the expected one I would guess that your grid indices are flipped.<br>
Did you check how these values are created and made sure that the “(x, y)” coordinates are in the expected image coordinate system (often having the origin in the top left and positive axes to the right and bottom)?</p>