v3d-webによるwebカメラだけのモーションキャプチャ

# デモ

https://youtu.be/pDwhiSuiUgs


# 目次
[ 1. v3d-webの紹介](#1-3d-webの紹介)
[ 2. v3d-webのインストール](#2-v3d-webのインストール)
[ 3. Unity連携](#3-Unity連携)

# 1. v3d-webの紹介

[v3d-web](https://github.com/phantom-software-AZ/v3d-web)とは、[デモサイト](https://www.phantom-dev.com/)を見ていただければわかるとおりに, Webカメラにかかわらず、すさまじく精度が高い webカメラだけのモーションキャプチャソフトです。

ただ、インストールがTrapだらけなので、非常に使うまでが大変でした（笑）

参考に, モーションキャプチャの方法として、

- Webカメラ
- 光学式センサ
- トラッカー

などが存在しますが、Webカメラが一番手軽で、精度が低いです。ただ、上半身に限り、大きな動作を伴わなければ使用性は十分です。

# 2. v3d-webのインストール

まず, npmがないと始まらないので [Node.js](https://nodejs.org/ja/download/) をインストールします。

次に v3d-webのレポジトリをclone します。

```
clone https://github.com/phantom-software-AZ/v3d-web.git
```

README.mdに従い

```
npm install　--force
npm run build && tsc
```

を行います。

ここでは, v3d-webに必要なパッケージのインストールと、v3d-webのbuild + TypeScriptのコンパイルを行っています。

### 注意1
ちなみに、デモサイトでは React で作られているため、

```
create-react-app app-name --type typescript 
```
として、Reactアプリを作成し、

```
npm install v3d-web
```

としても、普通にバグります。


### 注意2
The following asset(s) exceed the recommended size limit と warningが出るので、
webpack.configに

```
 performance: {
    hints: false,
    maxEntrypointSize: 512000,
    maxAssetSize: 512000
}
```

と設定します. (https://stackoverflow.com/questions/49348365/webpack-4-size-exceeds-the-recommended-limit-244-kib)

### 注意3
tscは内部または外部コマンドとして認識されません

と出たら

```
npm install typescript -g
```

として、typescriptをinstallしてください

### 注意4 
Field 'browser' doesn't contain a valid alias configration

と出るので、srcフォルダ内にindex-text.ts


```index-text.ts
/*
Copyright (C) 2021  The v3d Authors.
​
    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU Affero General Public License as published by
    the Free Software Foundation, version 3.
​
    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU Affero General Public License for more details.
​
    You should have received a copy of the GNU Affero General Public License
    along with this program.  If not, see <https://www.gnu.org/licenses/>.
 */
    ​
    export * from "./v3d-web";
    export type {CloneableQuaternionMap} from "./helper/quaternion";
    export type {HolisticOptions} from "./mediapipe";
    ​
    import {V3DWeb} from "./v3d-web";
    ​
    ​
    window.onload = async (e) => {
        const vrmFile = 'Maya__Mini.vrm';
        try {
            var v3DWeb = new V3DWeb(vrmFile);
        } catch (e: any) {
            console.error(e);
        }
    ​
    };
    ​
    export {};
```

を作成してください。

## デモ

https://youtu.be/_NYunQo7efQ


# 3. Unityとの連携

このままでも十分なのですが、Blenderなどで改造した vrmファイルを突っ込むとバグります。また、パーティクルなどの高度な演出などもできません。

なので、Unityとwebsocketで連携させて、いい感じにします。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1353403/2eaeb34b-1f0b-7fa6-9739-f7df431a2404.png)

システムの構成としてはこんな感じで、pythonのサーバーとv3d-webをwebsocket通信でつなぎ、boneのデータをやり取りします。

pythonのサーバは受け取ったboneのデータをUnityに送り付けます。

## 3.1 v3d-webの設定

pose-processing.tsについて設定します。

```
private websocket = new WebSocket('ws://127.0.0.1:9001/');
```
であらかじめconnectionをpropatyとして設定します。

次に, boneのデータを送る関数を設定します。

```
 public sendBoneRotations(){
        const items : any[] = [];
        for (const [k, v] of Object.entries(this._boneRotations)){
            items.push([k, v.x, v.y, v.z, v.w]);

        }
        if (this.websocket.OPEN){
            this.websocket.send(JSON.stringify(items));
        }else if(this.websocket.CLOSED){
            console.log("Websocket failed...");            
            this.websocket = new WebSocket('ws://127.0.0.1:9001/');
            this.websocket.send(JSON.stringify(items));
        }

        }


```

あとは、この関数を process関数内に置きます。

```
        this.filterBoneRotations(lockBones);

         // here is new code!
        this.sendBoneRotations();
        // Push to main
        this.pushBoneRotationBuffer();
```

以上により,　websocketの9001番ポートにwebsocketからのデータが送られます。

注意点としては、connectionが爆発するので、接続前に必ず切ってください。

## 3.2 pythonサーバ

```
import argparse
import json
from pythonosc import udp_client, osc_message_builder, osc_bundle_builder
from SimpleWebSocketServer import WebSocket, SimpleWebSocketServer


class OscBridge(WebSocket):
    ''' Websocket to OSC bridge '''

    def __init__(self, server, sock, address):
        super(OscBridge, self).__init__(server, sock, address)

        self.oscClient = udp_client.UDPClient('127.0.0.1', 54321)

    def parseMsg(self, address, msg):
        messages = []
        if isinstance(msg, dict):
            [messages.extend(self.parseMsg(address + '/' + k, v))
             for k, v in msg.items()]

        elif isinstance(msg, list):
            if isinstance(msg[0], dict) or isinstance(msg[0], list):
                [messages.extend(self.parseMsg(address, m)) for m in msg]
            else:
                messages.append(self.createOsc(address, msg))
        else:
            messages.append(self.createOsc(address, [msg]))
        return messages

    def createOsc(self, address, params):
        msg = osc_message_builder(address=address)
        for param in params:
            msg.add_arg(param)
        return msg

    def handleMessage(self):
        ws_msgs = json.loads(self.data)

        bundle = osc_bundle_builder.OscBundleBuilder(osc_bundle_builder.IMMEDIATELY)
        for bone_inf in ws_msgs:
            msg = osc_message_builder.OscMessageBuilder(address="/VMC/Ext/Bone/Rot")
            msg.add_arg(bone_inf[0])
            for bone_i in bone_inf[1:]:
                msg.add_arg(float(bone_i))
            bundle.add_content(msg.build())

        self.oscClient.send(bundle.build())

    def handleConnected(self):
        print(self.address, 'connected')

    def handleClose(self):
        print(self.address, 'closed')


if __name__ == '__main__':
    server = SimpleWebSocketServer('127.0.0.1', 9001, OscBridge)
    server.serveforever()
```

pythonoscを使って、Unityに送り付けます。

## 3.3 Unityの設定

[uOSC]を使います。必ず **v1.1.0** をDLしてください。

unity の packageをぶち込みます。

V3DReciever.csを作成します。

```V3DReciever.cs
using System;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using VRM;

[RequireComponent(typeof(uOSC.uOscServer))]
public class V3DReciever : MonoBehaviour
{
    [SerializeField] GameObject Model;
    private Animator animator;

    [SerializeField] bool isLRBlink = false;
    
    VRMBlendShapeProxy blendShapeProxy = null;
    private static readonly Dictionary<string , BlendShapePreset> BlendPreset = new Dictionary<string, BlendShapePreset>{
        // {"Neutral", BlendShapePreset.Neutral},
        {"A", BlendShapePreset.A},
        {"I", BlendShapePreset.I},
        {"U", BlendShapePreset.U},
        {"E", BlendShapePreset.E},
        {"O", BlendShapePreset.O},
        {"Blink", BlendShapePreset.Blink},
        {"Joy", BlendShapePreset.Joy},
        {"Angry", BlendShapePreset.Angry},
        {"Sorrow", BlendShapePreset.Sorrow},
        {"Fun", BlendShapePreset.Fun},
        {"LookUp", BlendShapePreset.LookUp},
        {"LookDown", BlendShapePreset.LookDown},
        {"Blink_L", BlendShapePreset.Blink_L},
        {"Blink_R", BlendShapePreset.Blink_R},
    };

    private static readonly List<string> BoneNames = new List<string>{
        "J_Bip_C_Hips",
        "J_Bip_C_Spine",
        "J_Bip_C_UpperChest",
        "J_Bip_C_Chest",
        "J_Bip_C_Neck",
        "J_Bip_C_Head",
        "J_Adj_L_FaceEye",
        "J_Adj_R_FaceEye",

        "J_Bip_L_Shoulder",
        "J_Bip_L_LowerArm",
        "J_Bip_L_UpperArm",
        "J_Bip_L_Hand",
        "J_Bip_L_Thumb1",
        "J_Bip_L_Index1",
        "J_Bip_L_Middle1",
        "J_Bip_L_Little1",
        "J_Bip_L_Ring1",
        "J_Bip_L_Thumb2",
        "J_Bip_L_Index2",
        "J_Bip_L_Middle2",
        "J_Bip_L_Little2",
        "J_Bip_L_Ring2",
        "J_Bip_L_Thumb3",
        "J_Bip_L_Index3",
        "J_Bip_L_Middle3",
        "J_Bip_L_Little3",
        "J_Bip_L_Ring3",

        "J_Bip_R_Shoulder",
        "J_Bip_R_LowerArm",
        "J_Bip_R_UpperArm",
        "J_Bip_R_Hand",
        "J_Bip_R_Thumb1",
        "J_Bip_R_Index1",
        "J_Bip_R_Middle1",
        "J_Bip_R_Little1",
        "J_Bip_R_Ring1",
        "J_Bip_R_Thumb2",
        "J_Bip_R_Index2",
        "J_Bip_R_Middle2",
        "J_Bip_R_Little2",
        "J_Bip_R_Ring2",
        "J_Bip_R_Thumb3",
        "J_Bip_R_Index3",
        "J_Bip_R_Middle3",
        "J_Bip_R_Little3",
        "J_Bip_R_Ring3",

        "J_Bip_L_UpperLeg",
        "J_Bip_L_LowerLeg",
        "J_Bip_L_Foot",
        "J_Bip_L_ToeBase",

        "J_Bip_R_UpperLeg",
        "J_Bip_R_LowerLeg",
        "J_Bip_R_Foot",
        "J_Bip_R_ToeBase",
    };

    private static readonly Dictionary<string, string> Uni2VRMBone = new Dictionary<string, string>{
        {"hips", "J_Bip_C_Hips"},
        {"spine", "J_Bip_C_Spine"},
        {"upperChest", "J_Bip_C_UpperChest"},
        {"chest", "J_Bip_C_Chest"},
        {"neck", "J_Bip_C_Neck"},
        {"head", "J_Bip_C_Head"},
        {"leftIris", "J_Adj_L_FaceEye"},
        {"rightIris", "J_Adj_R_FaceEye"},
        
        {"leftShoulder", "J_Bip_L_Shoulder"},
        {"leftLowerArm", "J_Bip_L_LowerArm"},
        {"leftUpperArm", "J_Bip_L_UpperArm"},
        {"leftHand", "J_Bip_L_Hand"},
        {"leftThumbProximal", "J_Bip_L_Thumb1"},
        {"leftThumbIntermediate", "J_Bip_L_Thumb2"},
        {"leftThumbDistal", "J_Bip_L_Thumb3"},
        {"leftIndexProximal", "J_Bip_L_Index1"},
        {"leftIndexIntermediate", "J_Bip_L_Index2"},
        {"leftIndexDistal", "J_Bip_L_Index3"},
        {"leftMiddleProximal", "J_Bip_L_Middle1"},
        {"leftMiddleIntermediate", "J_Bip_L_Middle2"},
        {"leftMiddleDistal", "J_Bip_L_Middle3"},
        {"leftRingProximal", "J_Bip_L_Ring1"},
        {"leftRingIntermediate", "J_Bip_L_Ring2"},
        {"leftRingDistal", "J_Bip_L_Ring3"},
        {"leftLittleProximal", "J_Bip_L_Little1"},
        {"leftLittleIntermediate", "J_Bip_L_Little2"},
        {"leftLittleDistal", "J_Bip_L_Little3"},
        {"leftUpperLeg", "J_Bip_L_UpperLeg"},
        {"leftLowerLeg", "J_Bip_L_LowerLeg"},
        {"leftFoot", "J_Bip_L_Foot"},
        {"leftToes", "J_Bip_L_ToeBase"},

        {"rightShoulder", "J_Bip_R_Shoulder"},
        {"rightLowerArm", "J_Bip_R_LowerArm"},
        {"rightUpperArm", "J_Bip_R_UpperArm"},
        {"rightHand", "J_Bip_R_Hand"},
        {"rightThumbProximal", "J_Bip_R_Thumb1"},
        {"rightThumbIntermediate", "J_Bip_R_Thumb2"},
        {"rightThumbDistal", "J_Bip_R_Thumb3"},
        {"rightIndexProximal", "J_Bip_R_Index1"},
        {"rightIndexIntermediate", "J_Bip_R_Index2"},
        {"rightIndexDistal", "J_Bip_R_Index3"},
        {"rightMiddleProximal", "J_Bip_R_Middle1"},
        {"rightMiddleIntermediate", "J_Bip_R_Middle2"},
        {"rightMiddleDistal", "J_Bip_R_Middle3"},
        {"rightRingProximal", "J_Bip_R_Ring1"},
        {"rightRingIntermediate", "J_Bip_R_Ring2"},
        {"rightRingDistal", "J_Bip_R_Ring3"},
        {"rightLittleProximal", "J_Bip_R_Little1"},
        {"rightLittleIntermediate", "J_Bip_R_Little2"},
        {"rightLittleDistal", "J_Bip_R_Little3"},
        {"rightUpperLeg", "J_Bip_R_UpperLeg"},
        {"rightLowerLeg", "J_Bip_R_LowerLeg"},
        {"rightFoot", "J_Bip_R_Foot"},
        {"rightToes", "J_Bip_R_ToeBase"},
    };
    private Dictionary<string, Transform> ModelBone;
    void Start()
    {
        var server = GetComponent<uOSC.uOscServer>();
        server.onDataReceived.AddListener(OnDataReceived);

        animator = Model.GetComponent<Animator>();
        blendShapeProxy = Model.GetComponent<VRMBlendShapeProxy>();

        ModelBone = new Dictionary<string, Transform>();
        GetAllChildren(Model);
    }

    void Update()
    {
        if (blendShapeProxy == null)
        {
            blendShapeProxy = Model.GetComponent<VRMBlendShapeProxy>();
        }
    }

    void OnDataReceived(uOSC.Message message)
    {
        if (message.address == "/VMC/Ext/Bone/Rot")
        {
            var boneName = (string)message.values[0];
            if (ReferenceEquals(message.values[1], null)){
                return;
            }
            var x = (float)message.values[1];
            var y = (float)message.values[2];
            var z = (float)message.values[3];
            var w = (float)message.values[4];
            if(boneName == "leftEye" || boneName == "rightEye"){
                return;
            }
            if(boneName == "leftIris" || boneName == "rightIris"){
                return;
            }
            if (boneName == "mouth"){
               blendShapeProxy.ImmediatelySetValue(BlendPreset["A"], clipping(x));
               return;
            } 
            if (boneName == "blink"){
                if(isLRBlink){
                    blendShapeProxy.ImmediatelySetValue(BlendPreset["Blink_L"], clipping(x));
                    blendShapeProxy.ImmediatelySetValue(BlendPreset["Blink_R"], clipping(y));
                }else{
                    blendShapeProxy.ImmediatelySetValue(BlendPreset["Blink"], clipping(z));
                }
                return;
            }
            if(boneName == "iris"){
                if((int)w == -1){
                    return;
                }
                ModelBone[Uni2VRMBone["leftIris"]].transform.localRotation = ReverseZ(new Quaternion(x, y, z, w));
                ModelBone[Uni2VRMBone["rightIris"]].transform.localRotation = ReverseZ(new Quaternion(x, y, z, w));
                return;
            }
            var VRMBoneName = Uni2VRMBone[boneName];
            ModelBone[VRMBoneName].transform.localRotation = ReverseZ(new Quaternion(x, y, z, w));
        }
    }

    private void GetAllChildren(GameObject obj){
        Transform children = obj.GetComponentInChildren<Transform> ();
        if (children.childCount == 0){
            return;
        }
        foreach (Transform ob in children){
            if(BoneNames.Contains(ob.name)){
                ModelBone.Add(ob.name, ob);
            }
            GetAllChildren(ob.gameObject);
        }
    }
    public static Quaternion ReverseZ(Quaternion q)
    {
        float angle;
        Vector3 axis;
        q.ToAngleAxis(out angle, out axis);
        return Quaternion.AngleAxis(-angle, ReverseZ(axis));
    }
    public static Vector3 ReverseZ(Vector3 v)
    {
        return new Vector3(v.x, v.y, -v.z);
    }

    private float clipping(float value)
    {
        return Math.Min(Math.Max(value, 0.0f), 1.0f);
    }
}
```

V3DReciever.cs と UOscServer.csを空のオブジェクトに割り当てます。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1353403/fb1eaf6a-bade-db05-a472-2a3ec55f9985.png)

これでUnity側の設定は終わりです。

あとは 

- v3d-web
- websocket2udp.py
- Unity

の3つを起動すると、モーションキャプチャができます。

お疲れさまでした。