HMS ML Kitの骨格検出機能の実装入門

# 骨格検出とは
このような写真があるとします。

![mamechi1110020_TP_V4.jpg](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/704548/e4e22c5a-6c39-2f6a-5562-d14d09fc3d86.jpeg)

この写真に写っている人の頭、首、肩、肘、手、お尻、膝、足を特定するときに使われる技術は骨格検出です。

たとえば、上記の写真を骨格検出にかけると、このような結果が得られます。

![Screenshot_20201026_170005_com.sample.hmssample.ml.skeletondetectiondemo.jpg](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/704548/a64114c8-3912-03fe-d877-5fa095971aec.jpeg)

# Androidアプリに骨格検出機能を導入する方法
Androidアプリで手軽に骨格検出機能を利用する方法の一つはHMSのML Kitを導入することです。HMSのML Kitなら人の頭、首、肩、肘、手、お尻、膝、足を特定できます。

前回の[HMS ML Kitの顔検出機能の実装入門とFirebase ML Kitの顔検出機能との比較](https://qiita.com/Rei_2020/items/8dc4543bdfef3f149829)では動画による検出を紹介したので、今回は静止画による骨格検出の実装を紹介します。

## AppGallery Connectの作業
1. AppGallery Connectに入って、[My projects]を選びます。
2. リストから対象アプリに切り替えます。
3. [Project settings] -> [Manage APIs]に入って、ML Kitを有効にします。（デフォルトではすでに有効になっているはずです）

こちらもご参照ください。
[https://developer.huawei.com/consumer/jp/doc/development/HMSCore-Guides/config-agc-0000001050990353](https://developer.huawei.com/consumer/jp/doc/development/HMSCore-Guides/config-agc-0000001050990353)
[https://developer.huawei.com/consumer/jp/doc/development/HMSCore-Guides/enable-service-0000001050038078](https://developer.huawei.com/consumer/jp/doc/development/HMSCore-Guides/enable-service-0000001050038078)

## HMS SDKの導入
1. agconnect-services.jsonをプロジェクト内に配置します。
2. プロジェクトのbuild.gradleにSDKを追加します。
3. モジュールのbuild.gradleにSDKを追加します。

骨格検出関連のライブラリはこちらです。

```gradle:build.gradle
    implementation 'com.huawei.hms:ml-computer-vision-skeleton:2.0.3.300'
    implementation 'com.huawei.hms:ml-computer-vision-skeleton-model:2.0.3.300'
```

こちらもご参照ください。
[https://developer.huawei.com/consumer/jp/doc/development/HMSCore-Guides/add-appgallery-0000001050038080](https://developer.huawei.com/consumer/jp/doc/development/HMSCore-Guides/add-appgallery-0000001050038080)
[https://developer.huawei.com/consumer/jp/doc/development/HMSCore-Guides/config-maven-0000001050040031](https://developer.huawei.com/consumer/jp/doc/development/HMSCore-Guides/config-maven-0000001050040031)
[https://developer.huawei.com/consumer/jp/doc/development/HMSCore-Guides/skeleton-sdk-0000001050728362](https://developer.huawei.com/consumer/jp/doc/development/HMSCore-Guides/skeleton-sdk-0000001050728362)

## AndroidManifest.xml
HMS ML Kitの骨格検出機能の定義をAndroidManifest.xmlに追加します。

```xml:AndroidManifest.xml
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="xxx">
    ...
    <meta-data
        android:name="com.huawei.hms.ml.DEPENDENCY"
        android:value= "skeleton"/>
    ...
</manifest>
```

## レイアウト

画像を表示するImageViewと骨格を描画するオーバーレイを配置します。

```xml:main_fragment.xml
<?xml version="1.0" encoding="utf-8"?>
<layout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    >

    <androidx.constraintlayout.widget.ConstraintLayout
        android:id="@+id/main_container"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:keepScreenOn="true">

        <ImageView
            android:id="@+id/image_view"
            android:layout_width="match_parent"
            android:layout_height="match_parent"
            app:layout_constraintTop_toTopOf="parent"
            app:layout_constraintBottom_toBottomOf="parent"
            app:layout_constraintStart_toStartOf="parent"
            app:layout_constraintEnd_toEndOf="parent"
            />

        <your.package.name.OverlayView
            android:id="@+id/overlay_view"
            android:layout_width="match_parent"
            android:layout_height="match_parent"
            app:layout_constraintTop_toTopOf="parent"
            app:layout_constraintBottom_toBottomOf="parent"
            app:layout_constraintStart_toStartOf="parent"
            app:layout_constraintEnd_toEndOf="parent"
            />

    </androidx.constraintlayout.widget.ConstraintLayout>

</layout>
```

## ファイル選択ダイアログの実装
Intent.ACTION_OPEN_DOCUMENTでファイル選択ダイアログを開き、onActivityResultでファイルパスを受け取ります。

```kotlin:MainFragment.kt
private val REQUEST_CODE = 100

override fun onActivityCreated(savedInstanceState: Bundle?) {
	super.onActivityCreated(savedInstanceState)

	// わかりやすく説明するために、onActivityCreatedでファイル選択ダイアログを開くことにする
	openFileChooser()
}

// ファイル選択ダイアログの呼び出し
private fun openFileChooser() {
	val intent = Intent(Intent.ACTION_OPEN_DOCUMENT).apply {
		addCategory(Intent.CATEGORY_OPENABLE)
		type = "*/*"
	}

	startActivityForResult(intent, REQUEST_CODE)
}

// ファイル選択ダイアログの選択結果はonActivityResultコールバック経由で渡される
override fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent?) {
	super.onActivityResult(requestCode, resultCode, data)

	when (requestCode) {
		REQUEST_CODE -> {
			if (Activity.RESULT_OK == resultCode) {
				data?.data?.let { data ->
					analyzeImage(data)
				}
			}
		}
	}
}
```

## 静止画から骨格を検出する
```kotlin:MainFragment.kt
// 骨格検出オブジェクトを生成する
private val analyzer: MLSkeletonAnalyzer = MLSkeletonAnalyzerFactory.getInstance().skeletonAnalyzer

private fun analyzeImage(uri: Uri?) {
	val context = context ?: return
	val uri = uri ?: return

	// 画像をImageViewに設定する
	binding.imageView.setImageURI(uri)

	// 画像の幅と高さをオーバーレイに渡す（骨格情報の描画位置を計算するため）
	binding.overlayView.setImageSize(binding.imageView.drawable.intrinsicWidth, binding.imageView.drawable.intrinsicHeight)

	// 非同期で画像の骨格検出を行う
	analyzer.asyncAnalyseFrame(
		MLFrame.fromFilePath(context, uri)
	).addOnSuccessListener(object : OnSuccessListener<List<MLSkeleton>> {
		override fun onSuccess(results: List<MLSkeleton>?) {
			// 検索結果をオーバーレイに渡す
			binding.overlayView.setSkeletonResults(results)
		}
	}).addOnFailureListener(object : OnFailureListener {
		override fun onFailure(exception: Exception?) {
			exception?.printStackTrace()
		}
	})
}
```

## 分析結果の描画
ImageViewで表示される写真はリサイズされ、かつ真ん中に配置されるものなので、骨格を描画するときに、骨格の座標もそれに合わせて再計算します。

```kotlin:OverlayView.kt
class OverlayView : View {

    companion object {
        private val linePaint = Paint().apply {
            color = Color.RED
            style = Paint.Style.STROKE
            strokeWidth = 8.0f
        }
        private val pointPaint = Paint().apply {
            color = Color.GREEN
        }
    }

    private var skeletonResults: List<MLSkeleton>? = null
    private var imageWidth = 1
    private var imageHeight = 1
    private var scale = 1.0f
    private var offsetX = 0.0f
    private var offsetY = 0.0f

    constructor(context: Context?): super(context)

    constructor(context: Context?, attrs: AttributeSet?): super(context, attrs)

    constructor(context: Context?, attrs: AttributeSet?, defStyleAttr: Int): super(
        context,
        attrs,
        defStyleAttr
    )

    constructor(context: Context?, attrs: AttributeSet?, defStyleAttr: Int, defStyleRes: Int): super(
        context,
        attrs,
        defStyleAttr,
        defStyleRes
    )

    fun setSkeletonResults(results: List<MLSkeleton>?) {
        skeletonResults = results
        invalidate()
    }

    fun setImageSize(width: Int, height: Int) {
        imageWidth = width
        imageHeight = height
        updateScale()
    }

    private fun updateScale() {
        val scaleX = width.toFloat() / imageWidth.toFloat()
        val scaleY = height.toFloat() / imageHeight.toFloat()
        scale = min(scaleX, scaleY)

        if (scaleX > scaleY) {
            offsetX = (width.toFloat() - imageWidth.toFloat() * scale) / 2.0f
            offsetY = 0.0f
        } else {
            offsetX = 0.0f
            offsetY = (height.toFloat() - imageHeight.toFloat() * scale) / 2.0f
        }
    }

    override fun onDraw(canvas: Canvas?) {
        super.onDraw(canvas)

        canvas?.let { canvas ->
            drawSkeletonResult(canvas, skeletonResults)
        }
    }

    private fun drawSkeletonResult(canvas: Canvas, results: List<MLSkeleton>?) {
        results?.forEach { value ->
            val headTop = value.getJointPoint(MLJoint.TYPE_HEAD_TOP)
            val neck = value.getJointPoint(MLJoint.TYPE_NECK)
            val leftShoulder = value.getJointPoint(MLJoint.TYPE_LEFT_SHOULDER)
            val rightShoulder = value.getJointPoint(MLJoint.TYPE_RIGHT_SHOULDER)
            val leftElbow = value.getJointPoint(MLJoint.TYPE_LEFT_ELBOW)
            val rightElbow = value.getJointPoint(MLJoint.TYPE_RIGHT_ELBOW)
            val leftWrist = value.getJointPoint(MLJoint.TYPE_LEFT_WRIST)
            val rightWrist = value.getJointPoint(MLJoint.TYPE_RIGHT_WRIST)
            val leftHip = value.getJointPoint(MLJoint.TYPE_LEFT_HIP)
            val rightHip = value.getJointPoint(MLJoint.TYPE_RIGHT_HIP)
            val leftKnee = value.getJointPoint(MLJoint.TYPE_LEFT_KNEE)
            val rightKnee = value.getJointPoint(MLJoint.TYPE_RIGHT_KNEE)
            val leftAnkle = value.getJointPoint(MLJoint.TYPE_LEFT_ANKLE)
            val rightAnkle = value.getJointPoint(MLJoint.TYPE_RIGHT_ANKLE)

            headTop?.let { point1 ->
                neck?.let { point2 ->
                    if (point1.score > 0 && point2.score > 0) {
                        canvas.drawLine(
                            translateX(point1.pointX),
                            translateY(point1.pointY),
                            translateX(point2.pointX),
                            translateY(point2.pointY),
                            linePaint
                        )
                    }
                }
            }

            neck?.let { point1 ->
                leftShoulder?.let { point2 ->
                    if (point1.score > 0 && point2.score > 0) {
                        canvas.drawLine(
                            translateX(point1.pointX),
                            translateY(point1.pointY),
                            translateX(point2.pointX),
                            translateY(point2.pointY),
                            linePaint
                        )
                    }
                }
            }

            neck?.let { point1 ->
                rightShoulder?.let { point2 ->
                    if (point1.score > 0 && point2.score > 0) {
                        canvas.drawLine(
                            translateX(point1.pointX),
                            translateY(point1.pointY),
                            translateX(point2.pointX),
                            translateY(point2.pointY),
                            linePaint
                        )
                    }
                }
            }

            leftShoulder?.let { point1 ->
                leftElbow?.let { point2 ->
                    if (point1.score > 0 && point2.score > 0) {
                        canvas.drawLine(
                            translateX(point1.pointX),
                            translateY(point1.pointY),
                            translateX(point2.pointX),
                            translateY(point2.pointY),
                            linePaint
                        )
                    }
                }
            }

            rightShoulder?.let { point1 ->
                rightElbow?.let { point2 ->
                    if (point1.score > 0 && point2.score > 0) {
                        canvas.drawLine(
                            translateX(point1.pointX),
                            translateY(point1.pointY),
                            translateX(point2.pointX),
                            translateY(point2.pointY),
                            linePaint
                        )
                    }
                }
            }

            leftElbow?.let { point1 ->
                leftWrist?.let { point2 ->
                    if (point1.score > 0 && point2.score > 0) {
                        canvas.drawLine(
                            translateX(point1.pointX),
                            translateY(point1.pointY),
                            translateX(point2.pointX),
                            translateY(point2.pointY),
                            linePaint
                        )
                    }
                }
            }

            rightElbow?.let { point1 ->
                rightWrist?.let { point2 ->
                    if (point1.score > 0 && point2.score > 0) {
                        canvas.drawLine(
                            translateX(point1.pointX),
                            translateY(point1.pointY),
                            translateX(point2.pointX),
                            translateY(point2.pointY),
                            linePaint
                        )
                    }
                }
            }

            leftHip?.let { point1 ->
                rightHip?.let { point2 ->
                    if (point1.score > 0 && point2.score > 0) {
                        canvas.drawLine(
                            translateX(point1.pointX),
                            translateY(point1.pointY),
                            translateX(point2.pointX),
                            translateY(point2.pointY),
                            linePaint
                        )
                        neck?.let { point3 ->
                            if (point3.score > 0) {
                                canvas.drawLine(
                                    translateX((point1.pointX + point2.pointX) / 2),
                                    translateY((point1.pointY + point2.pointY) / 2),
                                    translateX(point3.pointX),
                                    translateY(point3.pointY),
                                    linePaint
                                )
                            }
                        }
                    }
                }
            }

            leftHip?.let { point1 ->
                leftKnee?.let { point2 ->
                    if (point1.score > 0 && point2.score > 0) {
                        canvas.drawLine(
                            translateX(point1.pointX),
                            translateY(point1.pointY),
                            translateX(point2.pointX),
                            translateY(point2.pointY),
                            linePaint
                        )
                    }
                }
            }

            rightHip?.let { point1 ->
                rightKnee?.let { point2 ->
                    if (point1.score > 0 && point2.score > 0) {
                        canvas.drawLine(
                            translateX(point1.pointX),
                            translateY(point1.pointY),
                            translateX(point2.pointX),
                            translateY(point2.pointY),
                            linePaint
                        )
                    }
                }
            }

            leftKnee?.let { point1 ->
                leftAnkle?.let { point2 ->
                    if (point1.score > 0 && point2.score > 0) {
                        canvas.drawLine(
                            translateX(point1.pointX),
                            translateY(point1.pointY),
                            translateX(point2.pointX),
                            translateY(point2.pointY),
                            linePaint
                        )
                    }
                }
            }

            rightKnee?.let { point1 ->
                rightAnkle?.let { point2 ->
                    if (point1.score > 0 && point2.score > 0) {
                        canvas.drawLine(
                            translateX(point1.pointX),
                            translateY(point1.pointY),
                            translateX(point2.pointX),
                            translateY(point2.pointY),
                            linePaint
                        )
                    }
                }
            }

            value.joints.forEach { point ->
                if (point.score > 0) {
                    canvas.drawCircle(
                        translateX(point.pointX),
                        translateY(point.pointY),
                        10.0f,
                        pointPaint
                    )
                }
            }
        }
    }

    private fun translateX(x: Float): Float {
        return (x) * scale + offsetX
    }

    private fun translateY(y: Float): Float {
        return (y) * scale + offsetY
    }
}
```

## リソース解放
```kotlin:MainFragment.kt
override fun onDestroy() {
	super.onDestroy()

	analyzer.let {
		try {
			it.stop()
		} catch (ioException: IOException) {
			ioException.printStackTrace()
		}
	}
}
```

実装は以上になります。


# GitHub
* 動画による骨格検出：[https://github.com/Rei2020GitHub/MyPublicProject/tree/master/ml/SkeletonDetectionDemo](https://github.com/Rei2020GitHub/MyPublicProject/tree/master/ml/SkeletonDetectionDemo)
* 静止画による骨格検出：[https://github.com/Rei2020GitHub/MyPublicProject/tree/master/ml/SkeletonDetectionDemo_Image](https://github.com/Rei2020GitHub/MyPublicProject/tree/master/ml/SkeletonDetectionDemo_Image)

# APK
* 動画による骨格検出：[https://github.com/Rei2020GitHub/MyPublicProject/blob/master/APK/SkeletonDetectionDemo.apk](https://github.com/Rei2020GitHub/MyPublicProject/blob/master/APK/SkeletonDetectionDemo.apk)
* 静止画による骨格検出：[https://github.com/Rei2020GitHub/MyPublicProject/blob/master/APK/SkeletonDetectionDemoImage.apk](https://github.com/Rei2020GitHub/MyPublicProject/blob/master/APK/SkeletonDetectionDemoImage.apk)

# 参考
* HMS：[https://developer.huawei.com/consumer/jp/](https://developer.huawei.com/consumer/jp/)
* HMS ML Kitの紹介：[https://developer.huawei.com/consumer/jp/hms/huawei-mlkit](https://developer.huawei.com/consumer/jp/hms/huawei-mlkit)
* HMS ML Kitのドキュメント：[https://developer.huawei.com/consumer/jp/doc/development/HMSCore-Guides/service-introduction-0000001050040017](https://developer.huawei.com/consumer/jp/doc/development/HMSCore-Guides/service-introduction-0000001050040017)
* HMS ML Kitの骨格検出の概要：[https://developer.huawei.com/consumer/jp/doc/development/HMSCore-Guides/skeleton-detection-0000001051008415](https://developer.huawei.com/consumer/jp/doc/development/HMSCore-Guides/skeleton-detection-0000001051008415)
* Huawei Developers：[https://forums.developer.huawei.com/forumPortal/en/home](https://forums.developer.huawei.com/forumPortal/en/home)
* Facebook Huawei Developersグループ：[https://www.facebook.com/Huaweidevs/](https://www.facebook.com/Huaweidevs/)

# ※素材について
[ぱくたそ（PAKUTASO）](https://www.pakutaso.com/)が提供している商用利用可能なフリー素材を利用させていただきました。写真のもとの場所はこちらです。
[https://www.pakutaso.com/20191137315post-24124.html](https://www.pakutaso.com/20191137315post-24124.html)