インタラクティブな画像(2D•3D)や動画(2D•3D)を作成する(その1)

　関わっているプロジェクトで、インタラクティブな画像(2D・3D-360°)や動画(2D・3D-360°)を作成する機会があり、その際の手順を備忘録として少しずつまとめていこうと思います。

# 今回やりたいこと
　今回は、手始めとして「2D画像をインタラクティブにする手順」についてまとめます。
　具体的には、[Flickr](https://www.flickr.com/) からダウンロードした、以下の画像の中の２頭の犬をクリッカプルにし、クリックした際に何らかのアクションを実行できるようにします。
　イメージマップを使って実現させますが、その際に [Free Online Image Map Generator](https://www.image-map.net/) などのツールでは、画像が多数になった場合や、今後対応する3D(360°)画像・動画では対応出来ないため使用しません。その代わり、[Detectron2](https://github.com/facebookresearch/detectron2) を使って、犬の位置を自動検出し、その位置情報を使ってイメージマップを作成します。

<a data-flickr-embed="true" href="https://www.flickr.com/photos/38983646@N06/3702141833/in/photolist-6D9rMB-6WyJYD-whYdR6-cZtxnw-etTsex-wXtaCo-wi9xgo-awrVod-8RDy44-q52LLC-4EwsLT-x3mBr-eXJdcP-6gUoT2-KNMuuq-d9Su2y-Jhuaq-b8U2Qr-aaPm8B-75Ry8N-oWftz3-22KSt-9iLF9P-xcU4wY-GwSCR1-7qXRNW-7KVngz-wWNuEj-rvbZNW-76QDZr-795uaz-aDhJS9-7nLbgn-22Tz5bJ-4TgZHL-aRaCtt-b9ve5-22oAGx1-DLFNK-3tB4j-bEyPQW-HBiU26-c621gE-7mBp3H-9Tfch6-uMYt9-2jv7dh-xepWm5-EeTqp-umGV9J" title="DSC_0079"><img src="https://live.staticflickr.com/3443/3702141833_7ea19dbd3d_z.jpg" width="640" height="426" alt="DSC_0079"></a>
photo by <a href="https://flic.kr/p/6D9rMB" target="_blank"> Putneypics </a>


# 手順
## 環境
- [Google Colaboratory](https://colab.research.google.com/notebooks/welcome.ipynb?hl=ja)
    - Python3
    - [Detectron2](https://github.com/facebookresearch/detectron2)

- [jQuery3.3.1](https://jquery.com/)
- [jQuery-rwdImageMaps](https://github.com/stowball/jQuery-rwdImageMaps)

## アクションを設定する対象の座標を抽出
　座標の抽出は、[Google Colaboratory](https://colab.research.google.com/notebooks/welcome.ipynb?hl=ja) 上で、[Detectron2](https://github.com/facebookresearch/detectron2) を使って行います。

　※ [Google Colaboratory](https://colab.research.google.com/notebooks/welcome.ipynb?hl=ja)の基本的な使い方は、「[Google Colabの知っておくべき使い方 – Google Colaboratoryのメリット・デメリットや基本操作のまとめ](https://www.codexa.net/how-to-use-google-colaboratory/)」 で分かりやすく説明されています。

###Detectron2のインストール
　Detectron2、その他必要なモジュールをインストールします。

```python:google_colaboratory

!pip install -U torch torchvision cython
!pip install -U 'git+https://github.com/facebookresearch/fvcore.git' 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'
import torch, torchvision
torch.__version__
```
```python:google_colaboratory
!git clone https://github.com/facebookresearch/detectron2 detectron2_repo
!pip install -e detectron2_repo
```

　イントール完了後、ランタイムを再起動する必要があります。再起動後に必要なライプラリーをimportします。


```python:google_colaboratory

import detectron2
from detectron2.utils.logger import setup_logger
setup_logger()

import numpy as np
import cv2
import random
from google.colab.patches import cv2_imshow

from detectron2.engine import DefaultPredictor
from detectron2.config import get_cfg
from detectron2.data import MetadataCatalog
```

### 座標の取得
#### 画像の読込み
　対象の画像を [Flickr](https://www.flickr.com/) からダウンロードし、読込みます。

```python:google_colaboratory

!wget https://live.staticflickr.com/3443/3702141833_7ea19dbd3d_z_d.jpg -O dog.jpg
img = cv2.imread("dog.jpg")
```

####画像内のオブジェクトを検出
　本来であれば、検出したいオブジェクトを学習させ推定を行うべきですが、今回は簡単のため学習済みモデルを使用しオブジェクトを検出します。

　※ 独自データを学習させての推定は、動画対応の際に行いたいと思います。

```python:google_colaboratory

cfg = get_cfg()
cfg.merge_from_file("./detectron2_repo/configs/COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7
cfg.MODEL.WEIGHTS = "detectron2://COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x/137849600/model_final_f10217.pkl"
predictor = DefaultPredictor(cfg)
outputs = predictor(img)
```

####検出したオブジェクトの中から、対象のマスクデータを抽出
　検出したデータの中から、今回対象にしている犬のマスクデータを取り出します。

```python:google_colaboratory

target = MetadataCatalog.get(cfg.DATASETS.TRAIN[0]).thing_classes.index("dog")
classes = np.asarray(outputs["instances"].to("cpu").pred_classes)
masks = np.asarray(outputs["instances"].to("cpu").pred_masks)[classes==target].astype("uint8")
```

>*DOG1*
>![img_mask_0.jpg](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/549006/f2b80cbd-8ae2-f4c5-4c67-9667d8cc7b7c.jpeg)
>*DOG２*
>![img_mask_1.jpg](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/549006/3a1d32a4-9ada-8dad-e976-9ea68f1adf0d.jpeg)

#### マスクデータから輪廓を抽出


```python:google_colaboratory

contours = [ cv2.findContours(m, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)[0] for m in masks]
```
> *DOG1(右) + DOG2(左)*
>![img_contours.jpg](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/549006/1018e413-5bf6-329e-55bd-f6794f7c1aea.jpeg)



#### 輪郭の座標を取得
　輪郭の情報から、imagemapで使用するため、[x1,y1,x2,y2,...,xn,yn]の形で座標を取得します。

```python:google_colaboratory

coords = [c.flatten() for contour in contours for c in contour]
```

>*DOG1*

>```
>435,105,434,106,433,106,427,112,426,112,424,114,424,115,423,116,423,117,421,119,421,120,420,121,420,122,419,123,419,124,418,125,418,126,416,128,416,129,412,133,411,133,411,134,407,138,405,138,404,139,401,139,400,138,398,138,394,134,394,133,389,128,389,127,388,126,388,125,387,124,387,123,386,122,386,121,385,120,385,118,384,117,384,115,381,112,380,112,378,110,377,110,376,109,375,109,373,107,371,107,370,106,368,106,366,108,365,108,361,112,360,112,358,114,358,115,357,116,357,120,356,121,356,125,355,126,355,148,354,149,354,155,353,156,353,163,352,164,352,178,351,179,351,232,352,233,352,240,353,241,353,245,354,246,354,248,355,249,355,253,356,254,356,259,357,260,357,265,358,266,358,267,359,268,359,269,360,270,360,271,361,272,361,275,362,276,362,278,363,279,363,280,364,281,364,282,365,283,365,286,366,287,366,291,367,292,367,295,368,296,368,297,370,299,370,301,371,302,371,305,372,306,372,307,373,308,373,309,374,310,374,311,375,312,375,315,376,316,376,319,377,320,377,330,376,331,376,334,375,335,375,338,374,339,374,340,372,342,372,343,368,347,368,348,367,349,367,358,369,360,370,360,371,361,381,361,382,360,383,360,384,359,385,359,390,354,391,354,394,351,394,349,395,348,395,339,396,338,396,334,397,333,397,330,398,329,398,327,399,326,399,324,400,323,400,322,401,321,401,320,402,319,404,319,405,318,407,318,410,321,410,323,412,325,412,326,413,327,413,330,414,331,414,342,415,343,415,349,416,350,416,351,421,356,421,357,425,361,427,361,428,362,441,362,442,361,443,361,445,359,445,356,444,355,444,352,443,351,443,348,442,347,442,346,440,344,440,343,439,342,439,341,438,340,438,337,437,336,437,333,436,332,436,325,437,324,437,320,438,319,438,318,439,317,439,316,441,314,443,316,444,316,445,317,447,317,448,318,449,318,450,319,452,319,453,318,455,318,456,317,457,317,458,316,460,316,460,315,461,314,461,313,462,312,462,293,463,292,463,283,464,282,464,280,465,279,465,278,466,277,466,274,467,273,467,270,468,269,468,266,469,265,469,264,470,263,470,262,471,261,471,259,472,258,472,254,473,253,473,251,474,250,474,248,475,247,475,245,476,244,476,240,477,239,477,234,478,233,478,229,479,228,479,225,480,224,480,220,481,219,481,205,480,204,480,198,479,197,479,196,478,195,478,192,477,191,477,189,476,188,476,187,474,185,474,184,473,183,473,182,472,181,472,180,469,177,468,177,467,176,466,176,462,172,462,171,461,170,460,170,459,169,457,169,456,168,454,168,453,167,452,167,451,166,450,166,449,165,449,164,448,163,448,161,447,160,447,158,446,157,446,156,445,155,445,145,446,144,446,141,447,140,447,137,448,136,448,116,447,115,447,114,446,113,446,112,440,106,438,106,437,105
```
>*DOG2*

>```
184,19,183,20,180,20,179,21,177,21,176,22,174,22,173,23,171,23,169,25,168,25,167,26,160,26,159,25,158,25,157,24,156,24,155,23,154,23,153,22,152,22,151,21,150,21,149,20,140,20,139,21,137,21,131,27,130,27,128,29,127,29,126,30,124,30,123,31,122,31,121,32,120,32,118,34,118,35,117,36,117,37,116,38,116,40,115,41,115,42,114,43,114,44,113,45,113,47,112,48,112,51,113,52,113,53,114,54,114,55,121,62,121,63,122,64,122,65,123,66,123,69,124,70,124,74,125,75,125,77,124,78,124,89,123,90,123,94,122,95,122,99,121,100,121,103,120,104,120,107,119,108,119,111,118,112,118,116,117,117,117,121,116,122,116,125,115,126,115,128,114,129,114,132,113,133,113,137,112,138,112,156,113,157,113,167,114,168,114,171,115,172,115,176,116,177,116,190,117,191,117,218,118,219,118,221,119,222,119,224,120,225,120,226,121,227,121,228,122,229,122,230,123,231,123,233,124,234,124,236,125,237,125,238,127,240,127,241,128,242,128,246,129,247,129,254,130,255,130,266,129,267,129,274,128,275,128,279,127,280,127,283,126,284,126,289,125,290,125,299,124,300,124,310,123,311,123,323,122,324,122,329,121,330,121,334,120,335,120,341,119,342,119,348,120,349,120,352,121,353,121,354,122,355,122,356,123,357,123,359,126,362,127,362,128,363,129,363,130,364,132,364,133,365,137,365,138,366,143,366,144,367,147,367,148,368,152,368,153,369,155,369,156,368,157,368,158,367,159,367,160,366,162,366,163,365,165,365,166,364,167,364,169,362,169,361,170,360,170,359,171,358,171,357,172,356,172,355,173,354,173,353,174,352,174,351,175,350,175,349,176,348,176,346,177,345,177,341,178,340,178,337,179,336,179,333,180,332,180,330,181,329,181,328,182,327,182,326,183,325,183,324,184,323,186,325,186,326,187,327,187,328,188,329,188,331,189,332,189,336,190,337,190,347,191,348,191,353,192,354,192,359,193,360,193,362,194,363,194,364,196,366,197,366,198,367,200,367,201,368,215,368,216,367,217,367,218,366,219,366,220,365,221,365,225,361,226,362,229,362,230,363,232,363,233,364,235,364,236,365,239,365,240,364,243,364,244,363,245,363,246,362,247,362,249,360,249,359,255,353,256,353,257,352,258,352,261,349,261,348,262,347,262,344,263,343,263,339,264,338,264,334,265,333,265,324,266,323,266,318,265,317,265,307,264,306,264,299,263,298,263,285,262,284,262,269,261,268,261,264,260,263,260,260,259,259,259,257,258,256,258,255,257,254,257,253,256,252,256,251,252,247,251,247,248,244,248,243,247,242,247,241,246,240,246,239,244,237,244,236,242,234,242,233,241,232,241,231,240,230,240,223,239,222,239,215,238,214,238,204,237,203,237,201,236,200,236,198,235,197,235,194,234,193,234,190,233,189,233,188,232,187,232,185,231,184,231,183,230,182,230,181,229,180,229,176,228,175,228,172,227,171,227,166,226,165,226,163,225,162,225,161,224,160,224,158,223,157,223,154,222,153,222,147,221,146,221,140,220,139,220,136,219,135,219,134,218,133,218,131,217,130,217,126,216,125,216,100,217,99,217,84,216,83,216,73,215,72,215,68,214,67,214,64,213,63,214,62,214,60,215,59,215,58,216,57,216,56,217,55,217,53,218,52,218,51,219,50,219,45,220,44,220,35,219,34,219,33,218,32,217,32,207,22,206,22,205,21,204,21,203,20,201,20,200,19
```

## 画像のインタラクティブ化
　Imagemap、[jQuery](https://jquery.com/)、[jQuery-rwdImageMaps](https://github.com/stowball/jQuery-rwdImageMaps) で画像をインタラクティブかつレスポンシブにします。詳細については、他で分かりやすく説明されているサイトがありますので、そちらを参照いていただければと思います。

　※ 実際のコード は以下のようになります。 コードサンプルは [こちら](https://jsfiddle.net/f4oxdah1/)

```html:html

<img src="https://live.staticflickr.com/3443/3702141833_7ea19dbd3d_z.jpg" width="640" height="426" alt="Dog" usemap="#dogmap"/>
<map name="dogmap">
    <area class="dog1" shape="poly" coords="DOG1の座標" href="#" />
    <area class="dog2" shape="poly" coords="DOG２の座標" href="#" />
</map>
```
```css:css
img[usemap] {
    max-width: 100%;
    height: auto;
}
```
```javascript:script
$(document).ready(function(e) {
    $('img[usemap]').rwdImageMaps();
});
$(".dog1").on("click", function(e){
    e.preventDefault();
    alert('DOG1');
});
$(".dog2").on("click", function(e){
    e.preventDefault();
    alert('DOG2');
});
```

# まとめ
　今回の方法で、インタラクティブな2D画像を、ほぼ自動で簡単に作成することが出来るようになります。
　今後は、この方法を発展させたインタラクティブな3D画像や動画を作成する手順を、出来るだけ簡潔にまとめていきたいと考えています。