3D Scanner AppでImmersalのマップ作成

**Create an Immersal Map using 3D Scanner App**

#はじめに
前回の記事で、REST APIを叩いて、マップ作成はできました。全てのステップはImmersal SDKを使います。今回は他のアプリからの撮影した画像をマップ作成する。

**Summary**
On the Previous Article, I compiled on how to use the Sample and REST API by using the Immersal SDK. In this article, I compiled on how to use another application to get object images and create a map using the Immersal REST API.


#準備
**Preparation**

- Immersalアカウント / Immersal Account
- iPhone 12 Pro + 3D Scanner App

3D Scanner Appのページ / 3D Scanner App download page

https://apps.apple.com/us/app/3d-scanner-app/id1419913995

#手順

##3D Scanner Appで画像撮影
**Taking Picture with 3D Scanner App**

3D Scanner Appのデモでモデルをスキャンして、「All Data」を保存します。
By watching the Scanner App Demo, scan a model and export 「All Data」 from the result.

https://www.youtube.com/watch?v=UyWoVoGbY6s


##Pythonで画像変化
**Convert images using Python**

撮影した画像はiTunesか他のアプリかでパソコンに移動します。
Copy the exported all data to your computer using iTunes or any other app.

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1156398/9bc04e90-0d73-d9d4-139c-97261f92420f.png)

まず、ImmersalはJPEG画像を受け付けません。なので画像変は必要です。Pythonで画像変化をするならPillowが必要です
The Immersal Service doesn't process JPEG images (or maybe any other type of images). So we have to convert the image first using Pillow. We have to install Pillow first if we have never install it.

```
pip install Pillow
```

その後、画像をJPGからPNGに変化します。
After that, with the code below we convert the JPEG images produced by 3D Scanner App into PNG images.

```python:image_change.py
from PIL import Image
import os

for subdirs, dirs, files in os.walk('フォルダ名/Your Folder Name'):
    for file in files:
        if file.endswith((".jpg")):
            img_path = os.path.join(subdirs, file.split('.')[0] + ".jpg")
            save_path = os.path.join(subdirs, file.split('.')[0] + ".png")
            im1 = Image.open(r'{0}'.format(img_path))
            im1.save(r'{0}'.format(save_path))
```

変化された画像から、JSONデータを作成します。
After the images have been converted, we create a JSON data with scheme of Immersal image submission.

```python:json_processor.py
import json
import os
import base64
import quaternion
import numpy as np

index = 0


data = {}

data['token'] = "自分のToken/Input your token"
data['bank'] = 0
data['run'] = 0
data['index'] = 0
data['anchor'] =  False

def ConvertToBase64(src_filepath):
    with open(src_filepath, 'rb') as imageFileAsBinary:
        fileContent = imageFileAsBinary.read()
        b64_encoded_img = base64.b64encode(fileContent)

        return b64_encoded_img


for subdirs, dirs, files in os.walk('フォルダ名/Your Folder Name'):
    for file in files:
        if file.endswith((".json")):
            f = open(os.path.join(subdirs, file))
            json_Data = json.load(f)
            
            #回転のマトリックス
            #This is camera rotation in the 4x4 Matrix
            data['r00'] = json_Data['cameraPoseARFrame'][0]
            data['r01'] = json_Data['cameraPoseARFrame'][1]
            data['r02'] = json_Data['cameraPoseARFrame'][2]
            
            data['r10'] = json_Data['cameraPoseARFrame'][4]
            data['r11'] = json_Data['cameraPoseARFrame'][5]
            data['r12'] = json_Data['cameraPoseARFrame'][6]
            
            data['r20'] = json_Data['cameraPoseARFrame'][8]
            data['r21'] = json_Data['cameraPoseARFrame'][9]
            data['r22'] = json_Data['cameraPoseARFrame'][10]
            
            #カメラの位置
            #This is Camera Position in the 4x4 Matrix
            data['px'] = json_Data['cameraPoseARFrame'][3]
            data['py'] = json_Data['cameraPoseARFrame'][7]
            data['pz'] = json_Data['cameraPoseARFrame'][11]

            #カメラのIntrinsics
            #This is Camera Intrinsics
            data['fx'] = json_Data['intrinsics'][0]
            data['fy'] = json_Data['intrinsics'][4]
            data['ox'] = json_Data['intrinsics'][2]
            data['oy'] = json_Data['intrinsics'][5]
            imageData = ConvertToBase64(os.path.join(subdirs, file.split('.')[0] + ".png"))
            data['b64'] = str(imageData, 'utf-8')
            data['index'] = index
            
            #画像一枚にアンカーにする
            #We set the first image to be Anchor
            data['anchor'] = True if index==1 else False
            f.close()
            
            #Create A savepath for your folder
            savepath = 'フォルダ名/frame_{index}.json'.format(index = index)
            with open(savepath, 'w') as fs:
                json.dump(data, fs)

            index = index + 1

```

完了したら、アップロード可能なデータを作成されます。
After the process is completed, we have created a set of JSON files which are ready to be submitted using REST API.

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1156398/34b690a6-c1ef-223a-84b2-0b1740424c8e.png)



##マップ作成
**Map Creation**

マップ作成は前回の記事で説明した、まずワークスペースリセットして。画像アップロードして、マップ作成します。
I have compiled on how to use REST API to create a map. First we reset/clear the workspace. Then we submit the JSON data and finally we create the map.


https://qiita.com/brocolly/items/ce12887ad9781aa1da53

こちらは3DScanner AppとImmersalの結果を比べます。
3D Scanner Appの結果はオリエンテーションがちょっと変。
Below is the comparison result of Mapping between 3D Scanner App and Immersal.
The 3D Scanner App orientation result is a bit strange.

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1156398/cf35f767-4832-7e2b-bcee-0a9304372125.png)

変なオリエンテーションをしても、ポイントクラウドはまだ認識されます。
Even though the orientation is strange, its still able to detect point cloud.

<img src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1156398/a4193c78-3366-3c48-240d-0f25c7e55b38.jpeg" alt="drawing" width="300"/>


##Immersalでオリエンテーションの修理
**Fixing the Orientation**

前を書いた通り、3D Scanner Appを付けば、角度がちょっと変です。なので、そのモデル角度を修理します。修理するためには、JSONを作成する時、Z-axisで90度回転します。回転のマトリックスコードはこちらになります。
As I wrote earlier, the result from the 3D Scanner is a bit strange so I want to fix it. To fix the orientation, before we submit we have to rotate the image on Z-axis for 90 degrees. Below is the code for rotation code.

```python:json_processor.py
...
                Matrix_base = [[json_Data['cameraPoseARFrame'][0], json_Data['cameraPoseARFrame'][1], json_Data['cameraPoseARFrame'][2]],
                                [json_Data['cameraPoseARFrame'][4],json_Data['cameraPoseARFrame'][5],json_Data['cameraPoseARFrame'][6]],
                                [json_Data['cameraPoseARFrame'][8],json_Data['cameraPoseARFrame'][9],json_Data['cameraPoseARFrame'][10]]]
                
                #90度で回転する
                #Rotate the images 90 Degrees
                Matrix_Rz = [[math.cos(math.radians(90)), -math.sin(math.radians(90)), 0],
                            [math.sin(math.radians(90)), math.cos(math.radians(90)), 0],
                            [0, 0, 1]]
                
                Matrix_Rotation_new = np.matmul(Matrix_base, Matrix_Rz)

                data['r00'] = Matrix_Rotation_new[0][0]
                data['r01'] = Matrix_Rotation_new[0][1]
                data['r02'] = Matrix_Rotation_new[0][2]
                
                data['r10'] = Matrix_Rotation_new[1][0]
                data['r11'] = Matrix_Rotation_new[1][1]
                data['r12'] = Matrix_Rotation_new[1][2]
                
                data['r20'] = Matrix_Rotation_new[2][0]
                data['r21'] = Matrix_Rotation_new[2][1]
                data['r22'] = Matrix_Rotation_new[2][2]
...
```

Spatial回転の参考
A theoretical reference for spatial transformation.
https://www.brainvoyager.com/bv/doc/UsersGuide/CoordsAndTransforms/SpatialTransformationMatrices.html


回転させた結果はこちらになります。
Below is the result of the rotated images.
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1156398/491ea299-95bc-874b-fc8c-53d67244dfca.png)