Script TOPでMediaPipeやーる（TouchDesigner、Python3.7.2、OpenCV4.5.2、Windows10）

# はじめに
Script TOPを使ってMediaPipeをやりまーす

# 開発環境
- TouchDesigner.2021.13610
- Windows 10
- Anaconda 
- OpenCV 4.5.2
- Python 3.7.2

# 導入
1．Dialog->Textport and DATsからバージョンチェック
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/3ed75810-a3c5-a3b1-efc5-c65d665ed9b5.png)

2．anacondaで同じ環境を作成

~~~console
conda info -e
conda create -n td python=3.7.2
conda activate td
pip install opencv-python
pip install opencv-contrib-python
pip install mediapipe
~~~

3．Edit->Preferencesの設定
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/2c4071a9-10fd-1d99-b81c-f11975fc79d1.png)

- Python 64-bit Module Pathに作成したAnacondaの場所を指定（例：***C:\Users\good_\anaconda3\envs\td\Lib\site-packages***）
- Increment Filename on Saveを***Off***

4．TouchDesignerの再起動して、mediapipeがインポートできていればOK

5．Video Device Input（TOP）、Resolution（TOP）、Script（TOP）を配置します
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/c766b113-3b9c-b02c-0abd-ff36b9889d65.png)

6．Script（TOP）の中身

~~~python
# me - this DAT
# scriptOp - the OP which is cooking
import cv2
import numpy as np
import mediapipe as mp

mp_hands = mp.solutions.hands
hands = mp_hands.Hands(max_num_hands=2,min_detection_confidence=0.3,min_tracking_confidence=0.3)

# press 'Setup Parameters' in the OP to call this function to re-create the parameters.
def onSetupParameters(scriptOp):
	return

# called whenever custom pulse parameter is pushed
def onPulse(par):
	return


def onCook(scriptOp):
	input_image = scriptOp.inputs[0].numpyArray()
	image = input_image * 255
	image = image.astype('uint8')
	image_height, image_width = image.shape[:2]

	image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
	results = hands.process(image)

	draw_image = image.copy()
	if results.multi_hand_landmarks is not None:
		for hand_landmarks, handedness in zip(results.multi_hand_landmarks,results.multi_handedness):
			for index, landmark in enumerate(hand_landmarks.landmark):
				if landmark.visibility < 0 or landmark.presence < 0:
					continue
				landmark_x = min(int(landmark.x * image_width), image_width - 1)
				landmark_y = min(int(landmark.y * image_height), image_height - 1)
				cv2.circle(draw_image, (landmark_x, landmark_y), 5, (0, 255, 0), 2)

	draw_image = cv2.cvtColor(draw_image, cv2.COLOR_RGB2BGR)
	scriptOp.copyNumpyArray(draw_image)
	return
~~~


カスタムパラメータで新しいページを作成しようとしましたが、なぜか反映されませんでした。。。

~~~python
def onSetupParameters(scriptOp):
	page = scriptOp.appendCustomPage('Custom')
	p = page.appendInt('MaxNumHands', label='Max Num Hands')
	p[0].default = 2
	p = page.appendFloat('MinDetectionConfidence', label='Min Detection Confidence')
	p[0].default = 0.5
	p = page.appendFloat('MinTrackingConfidence', label='Min Tracking Confidence')
	p[0].default = 0.5
	return


def onCook(scriptOp):
	MaxNumHands = scriptOp.par.MaxNumHands.eval()
	MinDetectionConfidence = scriptOp.par.MinDetectionConfidence.eval()
	MinTrackingConfidence = scriptOp.par.MinTrackingConfidence.eval()
~~~

<blockquote class="twitter-tweet"><p lang="en" dir="ltr">Hands<a href="https://twitter.com/hashtag/mediapipe?src=hash&amp;ref_src=twsrc%5Etfw">#mediapipe</a> <a href="https://twitter.com/hashtag/TouchDesigner?src=hash&amp;ref_src=twsrc%5Etfw">#TouchDesigner</a> <a href="https://twitter.com/hashtag/kuma_td?src=hash&amp;ref_src=twsrc%5Etfw">#kuma_td</a> <a href="https://t.co/8A8Eddtt5N">pic.twitter.com/8A8Eddtt5N</a></p>&mdash; がちもとさん@熊本 (@sotongshi) <a href="https://twitter.com/sotongshi/status/1401907132564262918?ref_src=twsrc%5Etfw">June 7, 2021</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

7．hand_landmarksのTable（DAT）を追加し、Script（TOP）を下記のように編集します。これで手のランドマークの値がTableに表示されます。

~~~python
# me - this DAT
# scriptOp - the OP which is cooking
import cv2
import numpy as np
import mediapipe as mp

mp_hands = mp.solutions.hands
hands = mp_hands.Hands(max_num_hands=2,min_detection_confidence=0.3,min_tracking_confidence=0.3)

op_hand_landmarks = op('hand_landmarks')

# press 'Setup Parameters' in the OP to call this function to re-create the parameters.
def onSetupParameters(scriptOp):
	return

# called whenever custom pulse parameter is pushed
def onPulse(par):
	return


def onCook(scriptOp):
	input_image = scriptOp.inputs[0].numpyArray()
	image = input_image * 255
	image = image.astype('uint8')
	image_height, image_width = image.shape[:2]

	image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
	results = hands.process(image)

	draw_image = image.copy()
	op_hand_landmarks.clear()
	op_hand_landmarks.appendRow(['x', 'y', 'z'])
	if results.multi_hand_landmarks is not None:
		for hand_landmarks, handedness in zip(results.multi_hand_landmarks,results.multi_handedness):
			for index, landmark in enumerate(hand_landmarks.landmark):
				if landmark.visibility < 0 or landmark.presence < 0:
					continue
				landmark_x = min(int(landmark.x * image_width), image_width - 1)
				landmark_y = min(int(landmark.y * image_height), image_height - 1)
				cv2.circle(draw_image, (landmark_x, landmark_y), 5, (0, 255, 0), 2)
				op_hand_landmarks.appendRow([landmark.x, landmark.y, landmark.z])

	draw_image = cv2.cvtColor(draw_image, cv2.COLOR_RGB2BGR)
	scriptOp.copyNumpyArray(draw_image)
	return
~~~

8．Shpere（SOP）、Transform（SOP）、Geometry（COMP）を配置します。
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/8419285b-fbd4-72fe-1e8e-e01c49259cc0.png)

9．GeometryのInstancingをONにし、Translate OPに作成したTable（DAT）を割り当てます。（Table（DAT）をGeometryへD&Dして、Translate OPを選択、Translate X, Y, ZにそれぞれTableのラベルx,y,zを割り当て）
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/2a23bd47-a65c-96e3-c4e4-e8629d62dfad.png)

10．Transform（SOP）でUniform Scaleの値を調整します。
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/cbf15dbc-be95-76c9-44db-3d204d494052.png)

<blockquote class="twitter-tweet"><p lang="ja" dir="ltr">手の位置をInstancingする<a href="https://twitter.com/hashtag/mediapipe?src=hash&amp;ref_src=twsrc%5Etfw">#mediapipe</a> <a href="https://twitter.com/hashtag/TouchDesigner?src=hash&amp;ref_src=twsrc%5Etfw">#TouchDesigner</a> <a href="https://twitter.com/hashtag/kuma_td?src=hash&amp;ref_src=twsrc%5Etfw">#kuma_td</a> <a href="https://t.co/FEQSjpKVin">pic.twitter.com/FEQSjpKVin</a></p>&mdash; がちもとさん@熊本 (@sotongshi) <a href="https://twitter.com/sotongshi/status/1401917127292952576?ref_src=twsrc%5Etfw">June 7, 2021</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

お疲れ様でした。

Script TOPを使ったサンプルがあるので参考まで
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/87262ddd-263e-c86e-180a-9c20664044e0.png)

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/99a26508-6220-3c84-445c-07b18251a182.png)