WindowsMicrophoneStreamDemoやーる（HoloLens2、Unity2020.3.0f1、MRTK2.7.2）

[HoloLensアドベントカレンダー2021](https://qiita.com/advent-calendar/2021/hololens)の6日目の記事です。
MicStreamSelectorを使いたいという要望があったので、MRTKのサンプルにあるWindowsMicrophoneStreamDemoをやってみました。

# 開発環境
- HoloLens2
- Unity 2020.3.0f1
- MRTK 2.7.2
- [MicStreamSelector](https://github.com/microsoft/MixedRealityToolkit) commit d7cb7dcd4ce18437cc81789e6e364f4beb56dea8

# 実装
1．[MRTK](https://github.com/microsoft/MixedRealityToolkit-Unity/releases/tag/v2.7.2)をunitypackageから入れます

- Microsoft.MixedReality.Toolkit.Unity.Examples.2.7.2.unitypackage
- Microsoft.MixedReality.Toolkit.Unity.Extensions.2.7.2.unitypackage
- Microsoft.MixedReality.Toolkit.Unity.Foundation.2.7.2.unitypackage
- Microsoft.MixedReality.Toolkit.Unity.TestUtilities.2.7.2.unitypackage
- Microsoft.MixedReality.Toolkit.Unity.Tools.2.7.2.unitypackage

2．シーンのWindowsMicrophoneStreamDemoを開きます
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/c5a971e7-85d3-f3db-c6ba-69c2ce52431b.png)

2．[MicStreamSelector](https://github.com/microsoft/MixedRealityToolkit/tree/main/Input/MicStreamSelector)をVS2019、Release、ARM64でビルドします

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/bea84c1a-8c8b-5189-4735-462e1893b68d.png)

3．できたDLLとMicStreamSelector->UnityAddonにある3つのスクリプトをUnityプロジェクトに追加します
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/e5d061c7-704a-53ea-b17e-ecf5d1e491b7.png)
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/d4554ce6-fa55-b6f8-8c6b-c6162c255f17.png)

4．SceneContentのInspectorビューにあるMicrophoneAmplitudeDemo.csを編集します
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/04c67f39-74dc-5ea0-232f-4a002ffaf6c1.png)

\#if MICSTREAM_PRESENT
\#end
をコメントアウトします。

wireThicknessの値をDebugTextに表示しています

~~~MicrophoneAmplitudeDemo.cs
// Copyright (c) Microsoft Corporation.
// Licensed under the MIT License.

using Microsoft.MixedReality.Toolkit.Audio;
using Microsoft.MixedReality.Toolkit.SpatialAwareness;
using UnityEngine;
using TMPro;

namespace Microsoft.MixedReality.Toolkit.Examples
{
    /// <summary>
    /// Demonstration class using WindowsMicrophoneStream (from com.microsoft.mixedreality.toolkit.micstream) to select the 
    /// voice microphone and adjust the spatial awareness mesh based on the amplitude of the user's voice.
    /// </summary>
    [RequireComponent(typeof(AudioSource))]
    public class MicrophoneAmplitudeDemo : MonoBehaviour
    {
// #if MICSTREAM_PRESENT
        public TextMeshProUGUI debugText;

        [SerializeField]
        [Tooltip("Gain to apply to the microphone input.")]
        [Range(0, 10)]
        private float inputGain = 1.0f;

        [SerializeField]
        [Tooltip("Factor by which to boost the microphone amplitude when changing the mesh display.")]
        [Range(0, 50)]
        private int amplitudeBoostFactor = 10;

        [SerializeField]
        [Tooltip("Color to use for the wireframe mesh.\nIt is recommended to use a color with an alpha of 255.")]
        private Color meshColor = Color.blue;

        private IMixedRealitySpatialAwarenessMeshObserver spatialMeshObserver = null;
        private Material visibleMaterial = null;

        /// <summary>
        /// Class providing microphone stream management support on Microsoft Windows based devices.
        /// </summary>
        private WindowsMicrophoneStream micStream = null;

        /// <summary>
        /// The average amplitude of the sound captured during the most recent microphone update.
        /// </summary>
        private float averageAmplitude = 0.0f;

        /// <summary>
        /// Cached material values used to restore initial settings when running the demo in the editor.
        /// </summary>
        private Color defaultMaterialColor = Color.black;
        private int defaultWireThickness = 0;

        private void Awake()
        {
            // We do not wish to play the ambient room sound from the audio source.
            gameObject.GetComponent<AudioSource>().volume = 0.0f;

            spatialMeshObserver = (CoreServices.SpatialAwarenessSystem as IMixedRealityDataProviderAccess)?.GetDataProvider<IMixedRealitySpatialAwarenessMeshObserver>();
            visibleMaterial = spatialMeshObserver?.VisibleMaterial;

            if (visibleMaterial != null)
            {
                // Cache the initial material settings.
                defaultMaterialColor = visibleMaterial.GetColor("_WireColor");
                defaultWireThickness = visibleMaterial.GetInt("_WireThickness");

                visibleMaterial.SetColor("_WireColor", meshColor);
            }

            micStream = new WindowsMicrophoneStream();
            if (micStream == null)
            {
                Debug.Log("Failed to create the Windows Microphone Stream object");
            }

            micStream.Gain = inputGain;

            // Initialize the microphone stream.
            WindowsMicrophoneStreamErrorCode result = micStream.Initialize(WindowsMicrophoneStreamType.HighQualityVoice);
            if (result != WindowsMicrophoneStreamErrorCode.Success)
            {
                Debug.Log($"Failed to initialize the microphone stream. {result}");
                return;
            }

            // Start the microphone stream.
            // Do not keep the data and do not preview.
            result = micStream.StartStream(false, false);
            if (result != WindowsMicrophoneStreamErrorCode.Success)
            {
                Debug.Log($"Failed to start the microphone stream. {result}");
            }
        }

        private void OnDestroy()
        {
            if (micStream == null) { return; }

            // Stop the microphone stream.
            WindowsMicrophoneStreamErrorCode result = micStream.StopStream();
            if (result != WindowsMicrophoneStreamErrorCode.Success)
            {
                Debug.Log($"Failed to stop the microphone stream. {result}");
            }

            // Uninitialize the microphone stream.
            micStream.Uninitialize();
            micStream = null;

            // Restore the initial material settings.
            if (visibleMaterial != null)
            {
                visibleMaterial.SetColor("_WireColor", defaultMaterialColor);
                visibleMaterial.SetInt("_WireThickness", defaultWireThickness);
            }
        }

        private void OnDisable()
        {
            if (micStream == null) { return; }

            // Pause the microphone stream.
            WindowsMicrophoneStreamErrorCode result = micStream.Pause();
            if (result != WindowsMicrophoneStreamErrorCode.Success)
            {
                Debug.Log($"Failed to pause the microphone stream. {result}");
            }
        }

        private void OnEnable()
        {
            if (micStream == null) { return; }

            // Resume the microphone stream.
            WindowsMicrophoneStreamErrorCode result = micStream.Resume();
            if (result != WindowsMicrophoneStreamErrorCode.Success)
            {
                Debug.Log($"Failed to resume the microphone stream. {result}");
            }
        }

        private static int maxWireThickness = 750;

        private void Update()
        {
            if (micStream == null) { return; }

            // Update the gain, if changed.
            if (micStream.Gain != inputGain)
            {
                micStream.Gain = inputGain;
            }

            if (visibleMaterial != null)
            {
                // Artificially increase the amplitude to make the visible effect more pronounced.
                int wireThickness = (int)(averageAmplitude * amplitudeBoostFactor * maxWireThickness);
                wireThickness = Mathf.Clamp(wireThickness, 0, maxWireThickness);
                visibleMaterial.SetInt("_WireThickness", wireThickness);
                debugText.text = wireThickness.ToString();
            }
        }

        private void OnAudioFilterRead(float[] buffer, int numChannels)
        {
            if (micStream == null) { return; }

            // Read the microphone stream data.
            WindowsMicrophoneStreamErrorCode result = micStream.ReadAudioFrame(buffer, numChannels);
            if (result != WindowsMicrophoneStreamErrorCode.Success)
            {
                Debug.Log($"Failed to read the microphone stream data. {result}");
            }

            float sumOfValues = 0;

            // Calculate this frame's average amplitude.
            for (int i = 0; i < buffer.Length; i++)
            {
                if (float.IsNaN(buffer[i]))
                {
                    buffer[i] = 0;
                }

                buffer[i] = Mathf.Clamp(buffer[i], -1.0f, 1.0f);
                sumOfValues += Mathf.Clamp01(Mathf.Abs(buffer[i]));
            }

            averageAmplitude = sumOfValues / buffer.Length;
        }

// #endif // MICSTREAM_PRESENT
    }
}
~~~

5．Main Cameraの下にTextMeshProのTextを作成して、SceneContentのDebugTextにアタッチしてください

6．MicStreamSelectorの設定

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/410af592-8ba7-3eb6-a2dc-72c6b1ddda45.png)

7．Build Settingsの設定
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/3cee546f-4bcb-10a6-db66-dcafc59f49d7.png)

8．ProjectSettings->Player->Publishing Settings->Capabilitiesのチェック項目（デプロイ前にPackage.appxmanifestの機能に反映されているかもチェック）

- InternetClient 
- WebCam
- Microphone
- SpatialPerception
- GazeInput

9．XR Plug-in Managementの設定（※今回、OpenXRは使いません）

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/e93bbdbd-98d8-22d8-ea7d-191f30c55296.png)

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/8add045f-9ce9-8f3a-653b-eac7203d26a5.png)

10．ビルドして、Release、ARM64でHoloLensにデプロイしましょう

# デモ
ボイスの大きさに合わせて、空間メッシュのワイヤーフレームが太くなるみたいです！

<blockquote class="twitter-tweet"><p lang="ja" dir="ltr">＊音量注意<br>WindowsMicrophoneStreamDemo<a href="https://twitter.com/hashtag/HoloLens2?src=hash&amp;ref_src=twsrc%5Etfw">#HoloLens2</a> <a href="https://twitter.com/hashtag/MRTK?src=hash&amp;ref_src=twsrc%5Etfw">#MRTK</a> <a href="https://t.co/WEM12bUJKn">pic.twitter.com/WEM12bUJKn</a></p>&mdash; がちもとさん@メタバース熊本 (@sotongshi) <a href="https://twitter.com/sotongshi/status/1466812268503760896?ref_src=twsrc%5Etfw">December 3, 2021</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

ただし、アプリ内で音出しちゃうとマイク機能が使えなくなるバグがあります。録画不可です。一度マイク機能が使えなくなると、アプリ再起動しても無理です。本体を再起動したらいけます。たぶん起動時にmicStreamのリセットが必要だと思います。

お疲れ様でした。

<!--
実装
1．はじめにHoloLens2の設定

https://qiita.com/SatoshiGachiFujimoto/items/92211b6426a5fc3aa937

MixedRealityFeatureToolでとりあえず下記にチェック入れとく
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/1e3fc9f2-453b-cd29-98ca-24233f8dc20d.png)

2．MicStreamSelectorをダウンロード

https://github.com/microsoft/MixedRealityToolkit/tree/main/Input/MicStreamSelector

3．MicStreamSelector.slnをVS2019、Release、ARM64でビルドする
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/bea84c1a-8c8b-5189-4735-462e1893b68d.png)

4．できたDLLをコピペする
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/97457a88-0327-1f31-d5c2-c9a9178bf007.png)

5．UnityAddonフォルダ以下をUnityにぶっこんでいく
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/900b0288-9cea-f6f4-5779-5e2a266e5105.png)

6．Pluginsフォルダとスクリプトをコピペ
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/6fc935bb-c05e-fb01-e886-9b8266b8b558.png)

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/663efe25-7b40-acbb-83bd-7cd5eadd1930.png)

7．参考になりそうなスクリプト

https://github.com/microsoft/MixedRealityToolkit-Unity/blob/main/Assets/MRTK/Examples/Demos/Audio/Scripts/MicrophoneAmplitudeDemo.cs

8．x64のDLLも作っときましょうか
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/1898fe6c-7b08-18b8-2628-b1770d4722e3.png)

9．Unityの再起動しとこか
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/3237da8c-1ac1-b665-b36c-39b20b2ca05b.png)

10．Microsoft.MixedReality.Toolkit.Unity.Examples.2.7.2.unitypackageを入れる

https://github.com/microsoft/MixedRealityToolkit-Unity/releases

11．シーンのWindowsMicrophoneStreamDemoをビルドしてHoloLens2でやってみよう！

\#if MICSTREAM_PRESENT
\#endif 
をコメントアウトする

ARM64なのでEditor上では動かすことはできません

~~~MicrophoneAmplitudeDemo.cs
// Copyright (c) Microsoft Corporation.
// Licensed under the MIT License.

using Microsoft.MixedReality.Toolkit.Audio;
using Microsoft.MixedReality.Toolkit.SpatialAwareness;
using UnityEngine;
using TMPro;

namespace Microsoft.MixedReality.Toolkit.Examples
{
    /// <summary>
    /// Demonstration class using WindowsMicrophoneStream (from com.microsoft.mixedreality.toolkit.micstream) to select the 
    /// voice microphone and adjust the spatial awareness mesh based on the amplitude of the user's voice.
    /// </summary>
    [RequireComponent(typeof(AudioSource))]
    public class MicrophoneAmplitudeDemo : MonoBehaviour
    {
// #if MICSTREAM_PRESENT
        [SerializeField]
        [Tooltip("Gain to apply to the microphone input.")]
        [Range(0, 10)]
        private float inputGain = 1.0f;

        [SerializeField]
        [Tooltip("Factor by which to boost the microphone amplitude when changing the mesh display.")]
        [Range(0, 50)]
        private int amplitudeBoostFactor = 10;

        [SerializeField]
        [Tooltip("Color to use for the wireframe mesh.\nIt is recommended to use a color with an alpha of 255.")]
        private Color meshColor = Color.blue;

        private IMixedRealitySpatialAwarenessMeshObserver spatialMeshObserver = null;
        private Material visibleMaterial = null;

        /// <summary>
        /// Class providing microphone stream management support on Microsoft Windows based devices.
        /// </summary>
        private WindowsMicrophoneStream micStream = null;

        /// <summary>
        /// The average amplitude of the sound captured during the most recent microphone update.
        /// </summary>
        private float averageAmplitude = 0.0f;

        /// <summary>
        /// Cached material values used to restore initial settings when running the demo in the editor.
        /// </summary>
        private Color defaultMaterialColor = Color.black;
        private int defaultWireThickness = 0;

        private void Awake()
        {
            // We do not wish to play the ambient room sound from the audio source.
            gameObject.GetComponent<AudioSource>().volume = 0.0f;

            spatialMeshObserver = (CoreServices.SpatialAwarenessSystem as IMixedRealityDataProviderAccess)?.GetDataProvider<IMixedRealitySpatialAwarenessMeshObserver>();
            visibleMaterial = spatialMeshObserver?.VisibleMaterial;

            if (visibleMaterial != null)
            {
                // Cache the initial material settings.
                defaultMaterialColor = visibleMaterial.GetColor("_WireColor");
                defaultWireThickness = visibleMaterial.GetInt("_WireThickness");

                visibleMaterial.SetColor("_WireColor", meshColor);
            }

            micStream = new WindowsMicrophoneStream();
            if (micStream == null)
            {
                Debug.Log("Failed to create the Windows Microphone Stream object");
            }

            micStream.Gain = inputGain;

            // Initialize the microphone stream.
            WindowsMicrophoneStreamErrorCode result = micStream.Initialize(WindowsMicrophoneStreamType.HighQualityVoice);
            if (result != WindowsMicrophoneStreamErrorCode.Success)
            {
                Debug.Log($"Failed to initialize the microphone stream. {result}");
                return;
            }

            // Start the microphone stream.
            // Do not keep the data and do not preview.
            result = micStream.StartStream(false, false);
            if (result != WindowsMicrophoneStreamErrorCode.Success)
            {
                Debug.Log($"Failed to start the microphone stream. {result}");
            }
        }

        private void OnDestroy()
        {
            if (micStream == null) { return; }

            // Stop the microphone stream.
            WindowsMicrophoneStreamErrorCode result = micStream.StopStream();
            if (result != WindowsMicrophoneStreamErrorCode.Success)
            {
                Debug.Log($"Failed to stop the microphone stream. {result}");
            }

            // Uninitialize the microphone stream.
            micStream.Uninitialize();
            micStream = null;

            // Restore the initial material settings.
            if (visibleMaterial != null)
            {
                visibleMaterial.SetColor("_WireColor", defaultMaterialColor);
                visibleMaterial.SetInt("_WireThickness", defaultWireThickness);
            }
        }

        private void OnDisable()
        {
            if (micStream == null) { return; }

            // Pause the microphone stream.
            WindowsMicrophoneStreamErrorCode result = micStream.Pause();
            if (result != WindowsMicrophoneStreamErrorCode.Success)
            {
                Debug.Log($"Failed to pause the microphone stream. {result}");
            }
        }

        private void OnEnable()
        {
            if (micStream == null) { return; }

            // Resume the microphone stream.
            WindowsMicrophoneStreamErrorCode result = micStream.Resume();
            if (result != WindowsMicrophoneStreamErrorCode.Success)
            {
                Debug.Log($"Failed to resume the microphone stream. {result}");
            }
        }

        private static int maxWireThickness = 750;

        private void Update()
        {
            if (micStream == null) { return; }

            // Update the gain, if changed.
            if (micStream.Gain != inputGain)
            {
                micStream.Gain = inputGain;
            }

            if (visibleMaterial != null)
            {
                // Artificially increase the amplitude to make the visible effect more pronounced.
                int wireThickness = (int)(averageAmplitude * amplitudeBoostFactor * maxWireThickness);
                wireThickness = Mathf.Clamp(wireThickness, 0, maxWireThickness);
                visibleMaterial.SetInt("_WireThickness", wireThickness);
            }
        }

        private void OnAudioFilterRead(float[] buffer, int numChannels)
        {
            if (micStream == null) { return; }

            // Read the microphone stream data.
            WindowsMicrophoneStreamErrorCode result = micStream.ReadAudioFrame(buffer, numChannels);
            if (result != WindowsMicrophoneStreamErrorCode.Success)
            {
                Debug.Log($"Failed to read the microphone stream data. {result}");
            }

            float sumOfValues = 0;

            // Calculate this frame's average amplitude.
            for (int i = 0; i < buffer.Length; i++)
            {
                if (float.IsNaN(buffer[i]))
                {
                    buffer[i] = 0;
                }

                buffer[i] = Mathf.Clamp(buffer[i], -1.0f, 1.0f);
                sumOfValues += Mathf.Clamp01(Mathf.Abs(buffer[i]));
            }

            averageAmplitude = sumOfValues / (float)buffer.Length;
        }
// #endif // MICSTREAM_PRESENT
    }
}
~~~

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/8db298c4-fc26-d6b5-549c-3cd1a12f9273.png)

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/2513caea-f17c-a710-c689-7575af808034.png)

12．MRTK Toolkitの設定

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/a6b0a964-0dba-7d3f-64a2-fa805b12c4b4.png)

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/42a23a89-ac5f-26a9-7cf0-7ddca3d9ad25.png)

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/981c4786-d5e5-3873-e4e6-33406c5a4b10.png)

13．Spatial Perceptionにチェック入れとく
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/cd1a5c7f-3edb-e8d5-d75e-84f9fe6d39e8.png)

14．ビルドするときにPackage.appxmanifestの機能の部分で空間認識にチェック入ってるか確認！反映されてないときがあるから要注意。
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/e7c97b5b-594e-a8a0-5364-ef9433335c01.png)

15．OpenXRの設定もチェック
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/6b6bd2cd-05bb-c48f-85e3-12407c4b438c.png)


16．Release、ARM64で実機にデプロイ

17．アプリ側のマイクの許可が入っていなかった

MRTK Toolkitの設定を元に戻した
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/63863/78853723-ff93-9390-72e7-ece76d1cd04e.png)

デモ
大きい音でメッシュのワイヤーフレームが太くなるらしい。。
averageAmplitudeが0のままなのでちゃんと音声とれてなさそう。。

-->