長方形の逆透視投影（写真中の四角形を立体補正）

# 概要

写真中の長方形（被写体）を正面向きに補正します。

以下、末尾プログラムのスクリーンショットです。

![スクリーンショット.jpg](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/226677/dc03f781-7880-1819-503a-ff55dd6fccf9.jpeg)

# 手法

次の図は透視投影された長方形とします。

![figure-1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/226677/2cbcf191-c78b-e6ef-ff7d-11336dff6499.png)
<sub>内側の四角形は立体感を出すための模様です。全てテキトーに描いたので正確な投影図ではありません。</sub>

頂点 $p_{n=(1,2,3,4)}$ の画面上の座標を $ p_n \left( s_n , t_n \right) $ とします。

<details><summary>対角線の交点 $q$ の座標</summary><div>

```math:
\begin{eqnarray}
q & = & \left( s_q , t_q \right) \\
s_q & = & \frac{
  \left( s_1 t_4 - s_4 t_1 \right) \left( s_2 - s_3 \right) -
  \left( s_2 t_3 - s_3 t_2 \right) \left( s_1 - s_4 \right)
}{
  \left( s_1 - s_4 \right) \left( t_2 - t_3 \right) -
  \left( s_2 - s_3 \right) \left( t_1 - t_4 \right)
} \\
t_q & = & \frac{
  \left( s_1 t_4 - s_4 t_1 \right) \left( t_2 - t_3 \right) -
  \left( s_2 t_3 - s_3 t_2 \right) \left( t_1 - t_4 \right)
}{
  \left( s_1 - s_4 \right) \left( t_2 - t_3 \right) -
  \left( s_2 - s_3 \right) \left( t_1 - t_4 \right)
} \\
\end{eqnarray}
```

</div></details>


$p_n$ に投影面の奥行き座標 $z_s$ を追加して

```math:
\begin{eqnarray}
p_n & = & \left( s_n , t_n , z_s \right) \\
\end{eqnarray}
```

とし、$p_n$ を $m_n$ 倍（各点別）したものが三次元空間上の $P_n$

```math:
P_n = m_n\ p_n
```

になります。線分 $P_1 P_2$ と $P_3 P_4$ は平行で同じ長さだから、Mathematica に解いてもらうと

```mathematica:Mathematica
In[1]:= Module[{
  p1 = {s1, t1, zs},
  p2 = {s2, t2, zs},
  p3 = {s3, t3, zs},
  p4 = {s4, t4, zs},
  P1, P2, P3, P4
 },
 P1 = m1 p1;
 P2 = m2 p2;
 P3 = m3 p3;
 P4 = m4 p4;
 FullSimplify[Solve[P2 - P1 == P4 - P3, {m1, m2, m3, m4}]]
]

Out[1]= {{
  m2 -> (m1 (s4 (-t1 + t3) + s3 (t1 - t4) + s1 (-t3 + t4)))/(s4 (-t2 + t3) + s3 (t2 - t4) + s2 (-t3 + t4)),
  m3 -> (m1 (s4 (t1 - t2) + s1 (t2 - t4) + s2 (-t1 + t4)))/(s4 (-t2 + t3) + s3 (t2 - t4) + s2 (-t3 + t4)),
  m4 -> (m1 (s3 (t1 - t2) + s1 (t2 - t3) + s2 (-t1 + t3)))/(s4 (-t2 + t3) + s3 (t2 - t4) + s2 (-t3 + t4))
}}
```

が得られます。しかし、この解き方は $m_1$,$m_2$ と $m_4$,$m_3$ （対角）が入れ替わった場合も成立してしまいます。2通りしかないので、うまくいく方を選択します。

$z_s$ は出てこないので任意でよいことがわかりますが、透視投影なので $z_s\gt 0$ です。指定する $p_n$ 次第では $P_n$ による形状が平行四辺形（入力形状）になりますが、長方形（出力形状）と見做します。

$m_1$ は任意の値を指定可能なので

```math:
\begin{eqnarray}
m_d & = & \frac{ 1 }{ s_2 (t_4 - t_3) + s_3 (t_2 - t_4) + s_4 (t_3 - t_2) } \\
m_1 & = & 1 \\
m_2 & = & m_d\ m_1 (+ s_3 (t_1 - t_4) + s_4 (t_3 - t_1) + s_1 (t_4 - t_3)) \\
m_3 & = & m_d\ m_1 (- s_4 (t_2 - t_1) - s_1 (t_4 - t_2) - s_2 (t_1 - t_4)) \\
m_4 & = & m_d\ m_1 (- s_1 (t_3 - t_2) - s_2 (t_1 - t_3) - s_3 (t_2 - t_1)) \\
\end{eqnarray}
```

としますが、これではうまくいかず $m_1$,$m_2$ と $m_4$,$m_3$ が入れ替わっていました。そこで、

```math:
\begin{eqnarray}
m_1 \to m_4 \\
m_2 \to m_3 \\
m_3 \to m_2 \\
m_4 \to m_1 \\
\end{eqnarray}
```

に置き換えます。

出力先の長方形の縦横比は（横 : 縦 = $P_1 P_2 : P_1 P_3$）とします。$P_n$ に対応する出力先長方形の投影面座標を $g_n (x_n, y_n, z_s)$ とすると、対応する空間座標 $G_n$ を

```math:
G_n = \frac{1}{ m_n } g_n
```

として透視投影することで立体感を打ち消す形にできます。

# プログラム

<details><summary>ソースコード HTML+JavaScript (長いので折りたたみ)</summary><div>

```javascript:
<!DOCTYPE html>
<html lang="ja">
  <head>
    <meta charset="utf-8">
    <title>画像中の四角形を立体補正</title>
    <style>
     canvas {
         border: solid 1px black;
     }
     table {
         border: solid 1px; black;
         border-collaspe: collaspe;
         border-spacing: 0;
     }
     th, td {
         border: solid 1px gray;
         padding: 4px;
     }

     .editor {
         position: absolute;
         top: 4px; left: 4px;
         z-index: 1;
     }
     .preview {
         position: absolute;
         top: 4px; left: 4px;
         z-index: 2;
     }
     .output {
         position: absolute;
         top: 0px; left: 0px;
         z-index: 2;
     }
     .noevent {
         position: absolute;
         top: 0px; left: 0px;
         z-index: 3;

         border-width: 0;
         padding: 0px;
         margin: 0px;
         background-color: rgb(0 0 0 / .5);
     }
     .rectimage {
         position: absolute;
         top: 0px; left: 0px;
         z-index: 4;

         border: solid 8px white;
         padding: 0px;
         background-color: black;
     }
     .help {
         position: absolute;
         top: 16px; left: 16px;
         z-index: 5;
         background-color: white;
     }
    </style>
  </head>
  <body>
    <canvas id="tagImageCanvas" class="editor" width="512" height="512"></canvas>
    <canvas id="tagPreviewCanvas" class="preview" hidden></canvas>
    <canvas id="tagRectangleCanvas" class="output" hidden></canvas>
    <canvas id="tagTextureCanvas" hidden></canvas>
    <p class="noevent"><canvas id="tagNoEvent" hidden></canvas></p>
    <img id="tagRectangleImage" class="rectimage" hidden>

    <table id="tagHelp" class="help" hidden>
      <tr><th>操作</th><th>内容</th></tr>
      <tr><td>ドラッグ</td><td>画像の移動</td></tr>
      <tr><td>頂点をドラッグ</td><td>頂点の移動</td></tr>
      <tr><td>[SHIFT]+ドラッグ</td><td>四角形の移動</td></tr>
      <tr><td>ホイール</td><td>画像の拡大・縮小</td></tr>
      <tr><td>[SHIFT]+ホイール</td><td>四角形の拡大・縮小</td></tr>
      <tr><td>キー：[+] または [;]</td><td>画像の拡大</td></tr>
      <tr><td>キー：[-]</td><td>画像の縮小</td></tr>
      <tr><td>キー：[=] または [0]</td><td>画像を等倍で表示</td></tr>
      <tr><td>キー：[r]</td><td>画像をドロップしたときの状態に戻す</td></tr>
      <tr><td>キー：[v]</td><td>逆変換画像の表示/非表示</td></tr>
      <tr><td>キー：[h] または [?]</td><td>操作説明を表示</td></tr>
      <tr><td colspan="2">
        逆変換画像は元画像と同じサイズです。<br>
        画像の保存は右クリックのメニューから行ってください。<br>
        ブラウザによっては画像の保存ができません。
      </td></tr>
    </table>

    <script>

     function addVec(a, b) { return a.map((v, i) => v + b[i]); }
     function subVec(a, b) { return a.map((v, i) => v - b[i]); }
     function dotVec(a, b) { return a.reduce((a, v, i) => a + v * b[i], 0); }
     function mulVec(v, m) { return v.map((v) => v * m); }
     function divVec(v, m) { return v.map((v) => v / m); }

     function vecLen(v) { return Math.sqrt(dotVec(v, v)); }
     function vecLen2(s, e) { return vecLen2(subVec(e, s)); }
     function unitVec(v) { return divVec(v, vecLen(v)); }
     function unitVec2(s, e) { return unitVec(subVec(e, s)); }
     function zoomVec(p, c, s) { return addVec(mulVec(subVec(p, c), s), c); }

     /*
      * 四角形の奥行きを算出
      *   入力： 4点の平面座標
      *   出力： 奥行き
      *
      *   m1 -- m2
      *   |      |
      *   m3 -- m4
      */

     function getQuadrangleDepth(quadrangle) {
         const [[s1, t1], [s2, t2], [s3, t3], [s4, t4]] = quadrangle;
         const d = s2 * t4 - s2 * t3 + s3 * t2 - s3 * t4 + s4 * t3 - s4 * t2;
         const m1 = (s1 * t2 - s1 * t3 + s2 * t3 - s2 * t1 + s3 * t1 - s3 * t2) / d;
         const m2 = (s4 * t1 - s4 * t2 + s1 * t2 - s1 * t4 + s2 * t4 - s2 * t1) / d;
         const m3 = (s3 * t1 - s3 * t4 + s4 * t3 - s4 * t1 + s1 * t4 - s1 * t3) / d;
         return [m1, m2, m3, 1.0];
     }

     /*
      * Drag & Drop 用クラス
      */

     class DropImage {
         // ドロップされたファイル(イベント)から image 型だけを選択する.
         static fromEvent(event) {
             return [...event.dataTransfer.files]
                 .filter((f) => f.type.match('^image/'));
         }

         // イメージ ファイルの読み込み (Promiseを返す)
         static load(blobs) {
             const files = [...blobs].map((b) => ({blob: b}));
             return new Promise((resolve, reject) => {
                 function next(files, index) {
                     const drop = files[index];
                     drop.image = new Image();
                     drop.image.onload = (() => (++index < files.length)
                                              ? next(files, index) : resolve(files));
                     drop.image.onerror = reject;
                     drop.image.src = drop.src = URL.createObjectURL(drop.blob);
                     /* 画像が不要になったら URL.revokeObjectURL で解放すること */
                 }
                 (files.length > 0) ? next(files, 0) : resolve(files);
             });
         }
     }

     /*
      * 選択領域編集
      */

     class QuadrangleSelectEditor {

         // コンストラクタ.
         constructor(canvas) {
             this.canvas = canvas;
             this.context = this.canvas.getContext('2d');
             this.image = null;
             this.ondraw = null;

             this.pointerMode = null;
             this.pointerCapture = -1;
             this.pointerDown = null;
             this.pointerData = [0, 0];
             this.mousePosition = null;

             this.reset();
         }

         // 選択状況リセット.
         reset() {
             this.image_scale = null;
             this.image_pos = null;
             this.quadrangle = null;
             this.quadrangleIndex = null;
             this.quadrangleSelect = null;
             this.quadrangleEdge = null;
             this.quadrangleEdgeSelect = null;

             const image = this.image;
             if (image) {
                 const [iw, ih] = [image.width, image.height];
                 const [cx, cy] = [iw / 2, ih / 2];
                 const [dx, dy] = [cx / 2, cy / 2];
                 this.quadrangle = [
                     [cx - dx, cy - dy],
                     [cx + dx, cy - dy],
                     [cx - dx, cy + dy],
                     [cx + dx, cy + dy],
                 ];
             }

             this.updateImagePosition();
         }

         // 拡大率の補正.
         adjustScale(scale) {
             const image = this.image;
             if (!image)
                 return scale;

             const ctx = this.context;
             const cw = ctx.canvas.clientWidth;
             const ch = ctx.canvas.clientHeight;
             const [iw, ih] = [image.width, image.height];
             const [sw, sh] = [iw / cw, ih / ch];
             const slmin = Math.floor(Math.log2(1 / Math.max(sw, sh)));
             const smin = Math.pow(2, slmin);
             return Math.min(Math.max(smin / 4, scale ?? smin), 8.0);
         }

         // 画像表示位置の更新.
         updateImagePosition() {
             const ctx = this.context;
             const cw = ctx.canvas.clientWidth;
             const ch = ctx.canvas.clientHeight;
             const csz = [cw, ch];
             this.canvas_size = csz;

             const image = this.image;
             if (!image)
                 return;

             const [iw, ih] = [image.width, image.height];
             const [sw, sh] = [iw / cw, ih / ch];
             const scale = this.adjustScale(this.image_scale);
             const gv = mulVec([iw, ih], scale);

             this.image_scale = scale;

             const [px, py] = this.image_pos ?? divVec(subVec(csz, gv), 2);
             function fix(pos, dlen, glen) {
                 const sdlt = dlen - (glen * scale);
                 return ((sdlt < 0)
                       ? Math.max(sdlt, Math.min(0, pos))
                       : Math.max(0, Math.min(pos, sdlt)));
             }
             this.image_pos = [fix(px, cw, iw), fix(py, ch, ih)];
         }

         // 画像を設定.
         setImage(image) {
             this.image = image;
             this.reset();
         }

         // 画像位置を設定.
         setImagePos(pos) {
             this.image_pos = pos;
             this.updateImagePosition();
         }

         // タグのサイズを変更.
         setTagSize(width, height) {
             this.canvas.width = Math.max(320, width);
             this.canvas.height = Math.max(240, height);
             this.updateImagePosition();
         }

         // 画像座標に変換.
         toImage(p) {
             return divVec(subVec(p, this.image_pos), this.image_scale);
         }

         // 表示座標に変換.
         toCanvas(p) {
             return addVec(mulVec(p, this.image_scale), this.image_pos);
         }

         // 対角線の交点を取得.
         getIOD() {
             const quadrangle = this.quadrangle;
             if (!quadrangle)
                 return null;

             const [[s1, t1], [s2, t2], [s3, t3], [s4, t4]] = quadrangle;
             const qd = (s1 * t2 - s1 * t3
                       + s2 * t4 - s2 * t1
                       + s3 * t1 - s3 * t4
                       + s4 * t3 - s4 * t2);
             const qx = (s1 * s2 * t4 - s1 * s2 * t3
                       + s2 * s4 * t3 - s2 * s4 * t1
                       + s3 * s1 * t2 - s3 * s1 * t4
                       + s4 * s3 * t1 - s4 * s3 * t2) / qd;
             const qy = (s1 * t2 * t4 - s1 * t3 * t4
                       + s2 * t3 * t4 - s2 * t1 * t3
                       + s3 * t2 * t1 - s3 * t2 * t4
                       + s4 * t3 * t1 - s4 * t2 * t1) / qd;
             return [qx, qy];
         }

         // 有効な形状か?
         isValidQ() {
             const iod = this.getIOD();
             if (!iod) return false;
             const check = ((v1, v2) => {
                 const s = this.quadrangle[v1];
                 const t = this.quadrangle[v2];
                 return dotVec(subVec(t, s), subVec(iod, s)) > 0;
             });
             return (check(0, 3) && check(3, 0) &&
                     check(1, 2) && check(2, 1));
         }

         // 選択状況を描画.
         draw() {
             const ctx = this.context;
             const [cw, ch] = this.canvas_size;

             ctx.fillStyle = 'gray';
             ctx.fillRect(0, 0, cw, ch);

             this.drawImage();
         }

         // 画像を描画.
         drawImage() {
             const image = this.image;
             if (!image)
                 return this.drawDropInfo();

             const scale = this.image_scale;
             const [iw, ih] = [image.width, image.height];
             const [dx, dy] = this.image_pos;
             const [dw, dh] = mulVec([iw, ih], scale);

             const ctx = this.context;
             ctx.imageSmoothingEnabled = true;
             ctx.imageSmoothingQuality = 'high';
             ctx.drawImage(image, 0, 0, iw, ih, dx, dy, dw, dh);

             this.drawQuadrangle();
         }
         drawDropInfo() {
             const ctx = this.context;

             const getPos = ((msg) => {
                 const text = ctx.measureText(msg);
                 const [cw, ch] = this.canvas_size;
                 return [
                     (cw - text.width) / 2,
                     (ch - text.hangingBaseline) / 2,
                 ];
             });
             const putText = ((msg, x, y) => {
                 ctx.strokeStyle = 'black';
                 ctx.strokeText(msg, x, y);
                 ctx.fillStyle = 'white';
                 ctx.fillText(msg, x, y);
             });

             ctx.textAlign = 'left';
             ctx.textBaseline = 'top';
             ctx.direction = 'ltr';

             ctx.font = '24px serif';
             const message = 'ここに画像ファイルをドロップしてください';
             const [mx, my] = getPos(message);
             putText(message, mx, my);

             ctx.font = '16px serif';
             const help = '[h]キーで操作説明を表示';
             const [hx, hy] = getPos(help);
             putText(help, hx, hy + 36);
         }

         // 選択領域を描画.
         drawQuadrangle() {
             const quad = this.quadrangle;
             if (!quad)
                 return;

             const ipos = this.image_pos;
             const scale = this.image_scale;
             const qpos = quad.map((p) => addVec(mulVec(p, scale), ipos));

             const ctx = this.context;

             ctx.lineWidth = 2;
             ctx.strokeStyle = this.isValidQ()
                             ? 'rgb(0 255 0 / 0.75)'
                             : 'rgb(255 0 0 / 0.75)';
             ctx.beginPath();
             ctx.moveTo(qpos[0][0], qpos[0][1]);
             [2, 3, 1].forEach((i) => ctx.lineTo(qpos[i][0], qpos[i][1]));
             ctx.closePath();
             ctx.stroke();

             ctx.fillStyle = ctx.strokeStyle;
             for (const pos of qpos) {
                 ctx.beginPath();
                 ctx.arc(pos[0], pos[1], 3, 0, Math.PI * 2);
                 ctx.fill();
             }

             if (false) {
                 ctx.lineWidth = 1;
                 ctx.strokeStyle = 'rgb(255 0 / 0.25)';
                 ctx.beginPath();
                 ctx.moveTo(qpos[0][0], qpos[0][1]);
                 ctx.lineTo(qpos[3][0], qpos[3][1]);
                 ctx.stroke();
                 ctx.beginPath();
                 ctx.moveTo(qpos[1][0], qpos[1][1]);
                 ctx.lineTo(qpos[2][0], qpos[2][1]);
                 ctx.stroke();
             }

             const [qi, rad] = this.quadrangleSelect ?? [null, 0];
             if (qi != null) {
                 ctx.lineWidth = 2;
                 ctx.strokeStyle = (this.quadrangleIndex
                                  ? 'rgb(255 255 255 / 0.8)'
                                  : 'rgb(255 255 255 / 0.4)');
                 ctx.beginPath();
                 ctx.arc(qpos[qi][0], qpos[qi][1], rad, 0, Math.PI * 2);
                 ctx.stroke();
             }

             const ei = this.quadrangleEdgeSelect;
             if (ei != null) {
                 const [e1, e2] = ei;
                 ctx.lineWidth = 2;
                 ctx.strokeStyle = (this.quadrangleEdge
                                  ? 'rgb(255 255 255 / 0.8)'
                                  : 'rgb(255 255 255 / 0.4)');
                 ctx.beginPath();
                 ctx.moveTo(qpos[e1][0], qpos[e1][1]);
                 ctx.lineTo(qpos[e2][0], qpos[e2][1]);
                 ctx.stroke();
             }

             if (this.ondraw)
                 this.ondraw(this.quadrangle);
         }

         /*
          * ユーザー操作
          */

         onUserEvent(event) {
             this.pointerData = getEventOffset(event);
             const image = this.image;
             if (!image)
                 return;

             let update = false;
             let pointer = false;
             switch (event.type) {
                 case 'pointermove':
                     this.searchShape();
                     pointer = true;
                     update = true;
                     break;
                 case 'pointerdown':
                     if (event.buttons != 1)
                         break;
                     this.setPointerDown(event);
                     this.setPointerCapture(event.pointerId);
                     if (event.shiftKey)
                         this.pointerMode = 'quadrangle';
                     else if (this.selectShape())
                         ; // NO-OP
                     else
                         this.pointerMode = 'scroll';
                     pointer = true;
                     break;
                 case 'pointerup':
                     this.setPointerDown();
                     this.setPointerCapture(-1);
                     if (this.quadrangleIndex != null) {
                         this.quadrangleIndex = null;
                         update = true;
                     } else if (this.quadrangleEdge != null) {
                         this.quadrangleEdge = null;
                         update = true;
                     }
                     this.pointerMode = null;
                     break;

                 case 'wheel':
                     {
                         const zoom = Math.pow(2, - event.deltaY / 128);
                         const mpos = getEventOffset(event);
                         if (event.shiftKey)
                             this.setZoomQuadrange(zoom, mpos);
                         else
                             this.setZoomImage(zoom, mpos);
                     }
                     break;

                 case 'keydown':
                     switch (event.key) {
                         case '+':
                         case ';':
                             this.setZoomImage(2.0);
                             break;
                         case '-':
                             this.setZoomImage(0.5);
                             break;
                         case '=':
                         case '0':
                             this.setZoomImage(1 / this.image_scale);
                             break;
                         case 'r':
                             this.reset();
                             update = true;
                             break;
                     }
                     break;
             }
             if (pointer)
                 this.movePointer(event);
             if (update)
                 this.draw();
         }

         setPointerDown(event) {
             this.pointerDown = event ? this.pointerData : null;
         }

         setPointerCapture(id) {
             const tag = this.canvas;
             const pid = this.pointerCapture;
             this.pointerCapture = id;
             if (id >= 0)
                 tag.setPointerCapture(id)
             else if (pid >= 0)
                 tag.releasePointerCapture(pid)
         }

         searchVertex(pointer, rad) {
             const mp = pointer ?? this.pointerData;
             const rad1 = rad ?? 6;
             const rad2 = rad1 * rad1;
             const qsel = this.quadrangle.map((p, i) => {
                 const dv = subVec(this.toCanvas(p), mp);
                 return dotVec(dv, dv) < rad2 ? i : null;
             }).filter((i) => (i != null));
             this.quadrangleSelect = null;
             if (qsel.length > 0) {
                 this.quadrangleSelect = [qsel[0], rad1];
                 return true;
             }
             return false;
         }
         selectVertex() {
             this.quadrangleIndex = null;
             if (this.searchVertex(this.pointerDown)) {
                 this.quadrangleIndex = this.quadrangleSelect;
                 this.pointerMode = 'pointer';
                 return true;
             }
             return false;
         }

         searchEdge(pointer, distance) {
             const elen = distance ?? 6;
             const quad = this.quadrangle.map((p) => this.toCanvas(p));
             const mp = pointer ?? this.pointerData;

             const qsel = [[0, 1], [1, 3], [3, 2], [2, 0]].map((p, i) => {
                 const [sp, ep] = [quad[p[0]], quad[p[1]]];
                 const [sev, smv] = [subVec(ep, sp), subVec(mp, sp)];
                 const [sel, sml] = [vecLen(sev), vecLen(smv)];
                 const [sen, smn] = [divVec(sev, sel), divVec(smv, sml)];
                 const scos = dotVec(sen, smn);
                 if (scos < 0) return null;
                 const [esv, emv] = [subVec(sp, ep), subVec(mp, ep)];
                 const [esl, eml] = [vecLen(esv), vecLen(emv)];
                 const [esn, emn] = [divVec(esv, esl), divVec(emv, eml)];
                 const ecos = dotVec(esn, emn);
                 if (ecos < 0) return null;
                 const dv = subVec(mp, addVec(sp, mulVec(sen, sml * scos)));
                 return vecLen(dv) < elen ? p : null;
             }).filter((i) => (i != null));

             this.quadrangleEdgeSelect = null;
             if (qsel.length > 0) {
                 this.quadrangleEdgeSelect = qsel[0];
                 return true;
             }
             return false;
         }
         selectEdge() {
             this.quadrangleEdge = null;
             if (this.searchEdge(this.pointerDown)) {
                 this.quadrangleEdge = this.quadrangleEdgeSelect;
                 this.pointerMode = 'pointer';
                 return true;
             }
             return false;
         }

         searchShape() {
             const sv = this.searchVertex();
             const se = this.searchEdge();
             return sv || se;
         }
         selectShape() {
             const sv = this.selectVertex();
             const se = this.selectEdge();
             return sv || se;
         }

         movePointer(event) {
             const ipos = this.image_pos;
             const ppos = this.mousePosition;
             const apos = ppos ?? [0, 0];
             const cpos = getEventOffset(event);
             const pmov = subVec(cpos, apos);
             const dmov = addVec(ipos, pmov);

             this.mousePosition = cpos;
             switch (this.pointerMode) {
                 case 'scroll':
                     if (!ppos) {
                         this.pointerMode = null;
                         break;
                     }
                     this.setImagePos(dmov);
                     this.draw();
                     break;

                 case 'quadrangle':
                     if (!event.shiftKey || !ppos) {
                         this.pointerMode = null;
                     } else {
                         const m = this.toImage(dmov);
                         this.quadrangle = this.quadrangle.map((v) => addVec(v, m));
                         this.draw();
                     }
                     break;

                 case 'pointer':
                     if (this.quadrangleIndex != null) {
                         const ipos = this.toImage(cpos);
                         this.quadrangle[this.quadrangleIndex[0]] = ipos;
                         this.draw();
                     } else if (this.quadrangleEdge != null) {
                         const imov = this.toImage(dmov);
                         this.quadrangleEdge.forEach((e) => {
                             this.quadrangle[e] = addVec(this.quadrangle[e], imov);
                         });
                         this.draw();
                     }
                     break;
             }
         }

         setZoomImage(scale, position) {
             if (!this.image_scale || !this.image_pos)
                 return;

             const cpos = position ?? [
                 this.canvas.width / 2,
                 this.canvas.height / 2,
             ];

             const pscale = this.image_scale;
             const nscale = this.adjustScale(pscale * scale);
             const dscale = nscale / pscale;
             this.image_scale = nscale;
             this.setImagePos(zoomVec(this.image_pos, cpos, dscale));
             this.draw();
         }

         setZoomQuadrange(scale, position) {
             if (!this.quadrangle || !this.image_pos)
                 return;
             this.quadrangle = this.quadrangle.map((q) =>
                 this.toImage(zoomVec(this.toCanvas(q), position, scale)));
             this.draw();
         }
     }

     /*
      * WebGL: 逆透視矩形描画用
      */

     const splitSize = 64;
     const triangleStripSize = splitSize * (splitSize + 1) * 2;

     function interpVertex1(u, v, s) {
         return interpVertex2(u, v, 1.0 - s, s);
     }
     function interpVertex2(u, v, s, t) {
         return [
             u[0] * s + v[0] * t,
             u[1] * s + v[1] * t,
             u[2] * s + v[2] * t,
             u[3] * s + v[3] * t,
             u[4] * s + v[4] * t,
         ];
     }
     function splitVertex(u, v) {
         const r = Array(splitSize + 1);
         for (let i = 0; i <= splitSize; i++)
             r[i] = interpVertex1(u, v, i / splitSize);
         return r;
     }
     function splitQuadrangle(quadrangle) {
         const c1 = splitVertex(quadrangle[0], quadrangle[2]);
         const c2 = splitVertex(quadrangle[1], quadrangle[3]);
         const r = Array(splitSize + 1);
         for (let i = 0; i <= splitSize; i++)
             r[i] = splitVertex(c1[i], c2[i]);
         return r;
     }
     function createTriangleStrip(quadrangle) {
         const vertex = splitQuadrangle(quadrangle);
         const r = Array(triangleStripSize);
         for (let i = 0, p = 0; i < splitSize; i++) {
             const l1 = vertex[i + 0];
             const l2 = vertex[i + 1];

             if ((i & 1) == 0)
                 for (let j = 0; j <= splitSize; j++) {
                     r[p++] = l1[j];
                     r[p++] = l2[j];
                 }
             else
                 for (let j = splitSize; j >= 0; j--) {
                     r[p++] = l1[j];
                     r[p++] = l2[j];
                 }
         }
         return r;
     }

     class RenderRectangle {
         constructor(canvas) {
             this.canvas = canvas;
             this.image = null;
             this.valid = true;

             const gl = canvas.getContext('webgl');
             if (!gl) {
                 this.valid = false;
                 alert('WebGL を初期化出来ませんでした。このブラウザでは表示できない項目があります。');
                 return;
             }
             this.context = gl;

             if (!this.initShader())
                 this.destroy();

             this.vertexBuffer = gl.createBuffer();
             this.textureCoord = gl.createBuffer();
         }

         destroy() {
             if (!this.valid)
                 return;
             this.valid = false;

             const gl = this.context;
             const shader = this.shader_program;

             if (shader)
                 gl.deleteProgram(shader);

             this.deleteBuffer(this.vertexBuffer);
             this.deleteBuffer(this.textureCoord);

             this.shader_info = null;
             this.shader_program = null;
             this.shader_attrib = null;
             this.shader_uniform = null;

             this.texture = null;
             this.texImage = null;
             this.texImageData = null;

             this.vertexBuffer = null;
             this.textureCoord = null;
         }

         deleteBuffer(buffer) {
             if (buffer)
                 this.context.deleteBuffer(buffer);
             return this;
         }

         loadShader(type, source) {
             const gl = this.context;
             const shader = gl.createShader(type);
             gl.shaderSource(shader, source);
             gl.compileShader(shader);
             if (gl.getShaderParameter(shader, gl.COMPILE_STATUS))
                 return shader;
             console.log(gl.getShaderInfoLog(shader));
             gl.deleteShader(shader);
             return null;
         }

         initShader() {
             const gl = this.context;

             const shader = gl.createProgram();

             const vertex = this.loadShader(gl.VERTEX_SHADER, [
                 'uniform mat4 uMatrix;',
                 'attribute vec4 aPosition;',
                 'attribute vec2 aTexture;',
                 'varying highp vec2 vTexture;',
                 '',
                 'void main() {',
                 '  gl_Position = uMatrix * aPosition;',
                 '  vTexture = aTexture;',
                 '}',
             ].join('\n'));

             const fragment = this.loadShader(gl.FRAGMENT_SHADER, [
                 'uniform sampler2D uSampler;',
                 'varying highp vec2 vTexture;',
                 '',
                 'void main() {',
                 '  gl_FragColor = texture2D(uSampler, vTexture);',
                 '}',
             ].join('\n'));

             this.shader_program = shader;
             if (!shader || !vertex || !fragment)
                 return false;

             gl.attachShader(shader, vertex);
             gl.attachShader(shader, fragment);
             gl.linkProgram(shader);

             if (!gl.getProgramParameter(shader, gl.LINK_STATUS))
                 return false;

             this.shader_attrib = {
                 position: gl.getAttribLocation(shader, 'aPosition'),
                 texture: gl.getAttribLocation(shader, 'aTexture'),
             }
             this.shader_uniform = {
                 matrix: gl.getUniformLocation(shader, 'uMatrix'),
                 sampler: gl.getUniformLocation(shader, 'uSampler'),
             }

             return true;
         }

         clearTexture() {
             if (this.texture) {
                 this.context.deleteTexture(this.texture);
                 this.texture = null;
                 this.texImage = null;
                 this.texImageData = null;
             }
         }

         setTexture(image) {
             this.clearTexture();

             this.texture = this.context.createTexture();
             this.texImage = image;
             this.texImageData = new Uint8Array(image.data);
         }

         drawRectangle(quadrangle) {
             const depth = getQuadrangleDepth(quadrangle);
             depth.push(depth.reduce((a, v) => a + v, 0) / 4);
             const depmin = Math.min(...depth);
             const idep = depth.map((v) => v / depmin);
             const zmax = Math.max(...idep);

             const gl = this.context;
             const width = gl.drawingBufferWidth;
             const height = gl.drawingBufferHeight;
             const aspect = width / Math.max(1, height);

             gl.viewport(0, 0, width, height);

             gl.clearDepth(1.0);
             gl.enable(gl.DEPTH_TEST);
             gl.depthFunc(gl.LEQUAL);
             gl.clear(gl.DEPTH_BUFFER_BIT);

             gl.clearColor(0.5, 0.5, 0.5, 1.0);
             gl.clear(gl.COLOR_BUFFER_BIT);

             gl.useProgram(this.shader_program);
             {
                 const znear = 0.5;
                 const zfar = zmax * 1.5;
                 const zoom = 1.0;
                 const zd = zfar - znear;
                 const m00 = zoom / aspect;
                 const m11 = zoom;
                 const m22 = (zfar + znear) / zd;
                 const m23 = - (zfar * znear * 2.0) / zd;

                 gl.uniformMatrix4fv(
                     this.shader_uniform.matrix,
                     false, new Float32Array([
                         m00, 0.0, 0.0, 0.0,
                         0.0, m11, 0.0, 0.0,
                         0.0, 0.0, m22, 1.0,
                         0.0, 0.0, m23, 0.0,
                 ]));
             }

             {
                 const rpos = [0, 1, 2, 3].map((d, i) => {
                     const p = quadrangle[i];
                     return [p[0], p[1], depth[i]];
                 });
                 const [rp1, rp2, rp3, rp4] = rpos;
                 const iw = vecLen(rp1.map((v, i) => rp2[i] - v));
                 const ih = vecLen(rp1.map((v, i) => rp3[i] - v));
                 const [cw, ch] = [width, height];
                 const [rw, rh] = [iw / cw, ih / ch];
                 const rs = Math.max(rw, rh);
                 const [sw, sh] = [rw / rs * aspect, rh / rs];
                 const [dx, dy] = [sw, sh];

                 const image = this.texImage;
                 const [[s0, t0], [s1, t1], [s2, t2], [s3, t3]] = quadrangle;
                 const [tw, th] = [image.width, image.height];

                 const vStrip = createTriangleStrip([
                     [- dx * idep[0], + dy * idep[0], idep[0], s0 / tw, t0 / th],
                     [+ dx * idep[1], + dy * idep[1], idep[1], s1 / tw, t1 / th],
                     [- dx * idep[2], - dy * idep[2], idep[2], s2 / tw, t2 / th],
                     [+ dx * idep[3], - dy * idep[3], idep[3], s3 / tw, t3 / th],
                 ]);
                 const vertex = new Float32Array(vStrip.map((v) => [v[0], v[1], v[2], 1.0]).flat());
                 const texture = new Float32Array(vStrip.map((v) => [v[3], v[4]]).flat());

                 gl.bindBuffer(gl.ARRAY_BUFFER, this.vertexBuffer);
                 gl.bufferData(gl.ARRAY_BUFFER, vertex, gl.STATIC_DRAW);
                 gl.vertexAttribPointer(this.shader_attrib.position, 4, gl.FLOAT, false, 0, 0);
                 gl.enableVertexAttribArray(this.shader_attrib.position);

                 gl.activeTexture(gl.TEXTURE0);
                 gl.bindTexture(gl.TEXTURE_2D, this.texture);
                 gl.uniform1i(this.shader_uniform.sampler, 0);

                 gl.texImage2D(gl.TEXTURE_2D, 0,
                               gl.RGBA, tw, th, 0,
                               gl.RGBA, gl.UNSIGNED_BYTE, this.texImageData);
                 gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR);
                 gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
                 gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);

                 gl.bindBuffer(gl.ARRAY_BUFFER, this.textureCoord);
                 gl.bufferData(gl.ARRAY_BUFFER, texture, gl.STATIC_DRAW);
                 gl.vertexAttribPointer(this.shader_attrib.texture, 2, gl.FLOAT, false, 0, 0);
                 gl.enableVertexAttribArray(this.shader_attrib.texture);
             }

             gl.disable(gl.CULL_FACE);
             gl.drawArrays(gl.TRIANGLE_STRIP, 0, triangleStripSize);

             gl.finish();
             gl.flush();
         }

         readPixels() {
             const gl = this.context;
             const width = gl.drawingBufferWidth;
             const height = gl.drawingBufferHeight;
             const pixels = new Uint8Array(width * height * 4)
             gl.readPixels(0, 0, width, height, gl.RGBA, gl.UNSIGNED_BYTE, pixels);
             return pixels;
         }
     }

     /*
      *
      */

     class PreviewCanvas {
         constructor(canvas) {
             this.canvas = canvas;
             this.context = canvas.getContext('2d');
         }
     }

     /*
      *
      */

     var clickTimestamp = Date.now();

     var previewImage;
     var previewCanvas;
     var rectangleCanvas;
     var quadrangleCanvas;

     function setFocus() {
         window.focus();
     }

     function getWindowInnerSize() {
         return [window.innerWidth, window.innerHeight];
     }

     function getWindowSize() {
         return [window.innerWidth - 20, window.innerHeight - 20];
     }

     function setTexture(image) {
         const [iw, ih] = [image.width, image.height];
         tagRectangleCanvas.width = iw;
         tagRectangleCanvas.height = ih;
         tagTextureCanvas.width = iw;
         tagTextureCanvas.height = ih;

         const ctx = tagTextureCanvas.getContext('2d');
         ctx.imageSmoothingEnabled = true;
         ctx.imageSmoothingQuality = "high";
         ctx.drawImage(image, 0, 0);
         rectangleCanvas.setTexture(
             ctx.getImageData(0, 0, iw, ih, { colorSpace: "srgb" }));
     }

     function updateRectImage() {
         const ic = tagImageCanvas;
         const pc = tagPreviewCanvas;
         const tc = tagTextureCanvas;

         const qc = quadrangleCanvas;
         const qi = qc.image;
         if (!qi || !qc.mousePosition) {
             pc.hidden = true;
             return;
         }

         const [mx, my] = qc.mousePosition;
         const [cw, ch] = [ic.width, ic.height];
         const [cx, cy] = [cw / 2, ch / 2];
         const [dw, dh] = [cw / 4, ch / 4];
         const caspect = cw / Math.max(1, ch);
         const [tw, th] = [tc.width, tc.height];
         const taspect = tw / Math.max(1, th);

         let rw, rh;
         if (taspect > caspect) {
             rw = Math.trunc(cw / 2.5);
             rh = Math.trunc(rw / taspect);
         } else {
             rh = Math.trunc(ch / 2.5);
             rw = Math.trunc(rh * taspect);
         }
         const rx = (mx > cx) ? 16 : (cw - rw - 8);
         const ry = (my > cy) ? 16 : (ch - rh - 8);

         pc.style.left = `${Math.trunc(rx)}px`;
         pc.style.top = `${Math.trunc(ry)}px`;
         pc.width = rw;
         pc.height = rh;

         const pi = previewImage;
         if (pi) {
             const ctx = previewCanvas.context;
             ctx.drawImage(pi[0], 0, 0, pi[1], pi[2], 0, 0, rw, rh);
         }

         pc.hidden = false;
     }

     function updatePreview() {
         const rc = rectangleCanvas;
         const iw = rc.context.drawingBufferWidth;
         const ih = rc.context.drawingBufferHeight;
         createImageBitmap(rc.canvas).then(
             (res) => previewImage = [res, iw, ih]);
     }

     function updateTagSize() {
         const [winw, winh] = getWindowSize();
         quadrangleCanvas.setTagSize(winw, winh);
         quadrangleCanvas.draw();
         updateRectImage();

         if (!tagRectangleImage.hidden)
             setViewImagePosition();
     }

     function setViewImagePosition() {
         const itag = tagRectangleImage;
         const canvas = tagRectangleCanvas;
         const [ww, wh] = getWindowSize();
         const [cw, ch] = [canvas.width, canvas.height];
         const i2w = Math.min(ww / cw, wh / ch) * 0.9;
         const [iw, ih] = [cw * i2w + 16, ch * i2w + 16];
         const [ix, iy] = [(ww - iw) / 2, (wh - ih) / 2];

         itag.style.left = `${Math.trunc(ix)}px`;
         itag.style.top = `${Math.trunc(iy)}px`;
         itag.width = iw;
         itag.height = ih;

         const [iww, iwh] = getWindowInnerSize();
         tagNoEvent.style.left = '0px;'
         tagNoEvent.style.top = '0px;'
         tagNoEvent.width = iww;
         tagNoEvent.height = iwh;
     }

     function viewImage() {
         if (!previewImage)
             return;
         const itag = tagRectangleImage;
         if (!itag.hidden)
             return;

         setViewImagePosition();
         itag.src = tagRectangleCanvas.toDataURL();
         itag.hidden = false;
         tagNoEvent.hidden = false;
         enableScroll();
     }

     function hideImage() {
         tagRectangleImage.hidden = true;
         tagNoEvent.hidden = true;
         disableScroll();

         clickTimestamp = Date.now();
     }

     function hideHelp() {
         tagHelp.hidden = true;
     }

     function getEventOffset(event) {
         return [event.offsetX, event.offsetY];
     }

     function clearEventDefault(event) {
         event.preventDefault();
         event.stopPropagation();
     }

     function dropEvent(event) {
         clearEventDefault(event);
         const images = DropImage.fromEvent(event);
         if (images.length == 1)
             DropImage.load(images).then(
                 (files) => {
                     const image = files[0].image;
                     setTexture(image);
                     quadrangleCanvas.setImage(image);
                     quadrangleCanvas.draw();
                     setFocus();
                 },
                 console.error
             );
     }

     function pointerEvent(event) {
         if (event.type == 'pointerdown') {
             hideImage();
             hideHelp();
         }
         updateRectImage();
     }

     function keydownEvent(event) {
         let help = false;
         if (event.type == 'keydown') {
             switch (event.key) {
                 case '?':
                 case 'h':
                     help = tagHelp.hidden;
                     break;
                 case 'v':
                     if (tagRectangleImage.hidden)
                         viewImage();
                     else
                         hideImage();
                     break;
                 default:
                     hideImage();
                     break;
             }
             tagHelp.hidden = !help;
         }
     }

     function keyupEvent(event) {
     }

     function wheelEvent(event) {
         updateRectImage();
     }

     /*
      *
      */

     var scrollingFlag = true;

     function preventDefault(event) {
         event.preventDefault();
     }

     function preventDefault4Key(event) {
         if ([37, 38, 39, 40].includes(event.keyCode)) {
             preventDefault(event);
             return false;
         }
     }

     function disableScroll() {
         if (!scrollingFlag)
             return;
         scrollingFlag = false;

         window.addEventListener('DOMMouseScroll', preventDefault, false);
         window.addEventListener('keydown', preventDefault4Key, false);
         window.addEventListener('touchmove', preventDefault, { passive: false });
         window.addEventListener('wheel', preventDefault, { passive: false });
     }

     function enableScroll() {
         if (scrollingFlag)
             return;
         scrollingFlag = true;

         window.removeEventListener('DOMMouseScroll', preventDefault, false);
         window.removeEventListener('keydown', preventDefault4Key, false);
         window.removeEventListener('touchmove', preventDefault, { passive: false });
         window.removeEventListener('wheel', preventDefault, { passive: false });
     }

     /*
      *
      */

     window.onload = function() {
         previewCanvas = new PreviewCanvas(tagPreviewCanvas);
         rectangleCanvas = new RenderRectangle(tagRectangleCanvas);

         quadrangleCanvas = new QuadrangleSelectEditor(tagImageCanvas);
         quadrangleCanvas.ondraw = ((q) => {
             rectangleCanvas.drawRectangle(q);
             updatePreview();
         });

         const tag = document.body;
         tag.addEventListener('dragover', clearEventDefault);
         tag.addEventListener('drop', dropEvent);

         const userEvent = ((event) => {
             if (tagRectangleImage.hidden &&
                 ((Date.now() - clickTimestamp) > 100)) {
                 quadrangleCanvas.onUserEvent(event);
             }
         });
         tagImageCanvas.addEventListener('pointerdown', userEvent);
         tagImageCanvas.addEventListener('pointerdown', pointerEvent);
         tagImageCanvas.addEventListener('pointerup', userEvent);
         tagImageCanvas.addEventListener('pointerup', pointerEvent);
         tagImageCanvas.addEventListener('pointermove', userEvent);
         tagImageCanvas.addEventListener('pointermove', pointerEvent);

         tagRectangleImage.addEventListener('pointerdown', (event) => {
             clearEventDefault(event);
             hideHelp();
         });

         window.addEventListener('pointerdown', pointerEvent);
         window.addEventListener('keydown', userEvent);
         window.addEventListener('keydown', keydownEvent);
         window.addEventListener('keyup', keyupEvent);
         window.addEventListener('wheel', userEvent);
         window.addEventListener('wheel', wheelEvent);

         window.onresize = updateTagSize;
         updateTagSize();

         disableScroll();
         setFocus();
     }

    </script>
  </body>
</html>
```

</div></details>

<p class="codepen" data-height="600" data-default-tab="result" data-slug-hash="gOEeMLX" data-user="ikiuo" style="height: 600px; box-sizing: border-box; display: flex; align-items: center; justify-content: center; border: 2px solid; margin: 1em 0; padding: 1em;">
  <span>See the Pen <a href="https://codepen.io/ikiuo/pen/gOEeMLX">
  画像中の四角形を立体補正</a> by Ikiuo (<a href="https://codepen.io/ikiuo">@ikiuo</a>)
  on <a href="https://codepen.io">CodePen</a>.</span>
</p>
<script async src="https://cpwebassets.codepen.io/assets/embed/ei.js"></script>