Azure AI services multi-service account & Azure OpenAIを楽しんでみる

# はじめに
[Azure AI services multi-service account](https://learn.microsoft.com/ja-jp/azure/ai-services/multi-service-resource?tabs=windows&pivots=azportal)は、一つのリソースで

- [Custom Vision](https://learn.microsoft.com/ja-jp/azure/ai-services/custom-vision-service/)
- [Document Intelligence](https://learn.microsoft.com/ja-jp/azure/ai-services/document-intelligence/)
- [Face](https://learn.microsoft.com/ja-jp/azure/ai-services/computer-vision/overview-identity)
- [Language](https://learn.microsoft.com/ja-jp/azure/ai-services/language-service/)
- [Speech](https://learn.microsoft.com/ja-jp/azure/ai-services/speech-service/)
- [Translator](https://learn.microsoft.com/ja-jp/azure/ai-services/translator/)
- [Vision](https://learn.microsoft.com/ja-jp/azure/ai-services/computer-vision/)

上記のサービスにアクセスできる優れもの。

:::note warn
2024.02.01時点で[Content Moderator](https://learn.microsoft.com/ja-jp/azure/ai-services/content-moderator/)は廃止予定
:::

https://learn.microsoft.com/ja-jp/azure/ai-services/multi-service-resource?tabs=windows&pivots=azportal

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2734332/f36d2b09-abbf-0e11-8328-285424420de6.png)

こちらと[Azure Functions](https://learn.microsoft.com/ja-jp/azure/azure-functions/functions-overview)を組み合わせて遊んでみます。


![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2734332/bf9b8a57-0bdf-d96c-eb9b-695b40399c2f.png)


https://learn.microsoft.com/ja-jp/azure/azure-functions/functions-overview

# 概要
1. Power Appsで写真を撮影 or Upload
1. Vision(Read API)で物理的な文字の読み取り
1. OpenAIで文章校正
1. Speechで音読

上記をサクっと作ります。

UIは頑張っていません。

https://twitter.com/DemodoriGatsuo/status/1752319769804689878

:::note info
動画はコチラ
:::

# Azureのリソース
- Azure AI services multi-service account 
- Azure OpenAI(gpt3.5 turbo 0613モデル)
- Azure Functions(Python 3.11)

:::note info
Key Vaultは今回割愛
:::

# 進め方
1. Vision - Read APIの関数を作成
1. OpenAIの関数を作成
1. Speech Text to Speechの関数を作成
1. VS CodeでAzure Functionsのリソース作成、デプロイ
1. Power Automateの作成
1. Power Appsに実装

:::note info
カスタムコネクタまでは頑張っていません
:::

## 1. Vision - Read API OCRの機能を構築する

<details><summary>OCRのコード</summary>

```python:OCR
def extract_text_from_image(encoded_image_data):
    """
    Extracts text from the given base64-encoded image data using the Azure Computer Vision API.

    Args:
    encoded_image_data (str): The base64-encoded data of the image from which to extract the text.

    Returns:
    str: Extracted text from the image or None if the extraction fails.
    """
    # Decode the base64-encoded data
    image_data = base64.b64decode(encoded_image_data)    
    # Authenticate
    subscription_key = os.environ["MULTI_SERVICE_KEY"]
    endpoint = os.environ["MULTI_SERVICE_ENDPOINT"]

    computervision_client = ComputerVisionClient(endpoint, CognitiveServicesCredentials(subscription_key))

    # Call API with image data
    try:
        image_stream = BytesIO(image_data)
        read_response = computervision_client.read_in_stream(image_stream,  raw=True)

        # Get the operation location (URL with an ID at the end) from the response
        read_operation_location = read_response.headers["Operation-Location"]
        operation_id = read_operation_location.split("/")[-1]

        # Call the "GET" API and wait for it to retrieve the results 
        while True:
            read_result = computervision_client.get_read_result(operation_id)
            if read_result.status not in ['notStarted', 'running']:
                break
            time.sleep(1)

        # Extract and return text if operation succeeded
        if read_result.status == OperationStatusCodes.succeeded:
            return "".join(
                line.text for text_result in read_result.analyze_result.read_results
                for line in text_result.lines
            )
    except Exception as e:
        logging.info(f"An error occurred: {e}")
        return None
```

</details>

[公式のサンプルコード](https://learn.microsoft.com/ja-jp/azure/ai-services/computer-vision/quickstarts-sdk/client-library?tabs=windows%2Cvisual-studio&pivots=programming-language-python)を関数にしただけに留まります。
`base64`を用いている理由は、`Power Automate`との値渡しを考えての設定です。

:::note warn
base64でとにかく躓きました・・・
:::

[Read API](https://learn.microsoft.com/ja-jp/azure/ai-services/computer-vision/concept-ocr)の特徴は、呼び出してから待機が発生することです。
その理由から`time.sleep`が用いられています。
呼び出し結果は、内包表記を用いて結合しました。

https://learn.microsoft.com/ja-jp/azure/ai-services/computer-vision/concept-ocr

`エンドポイント`や`秘密鍵`は**環境変数**に設定しています。
`Text to Speech`と共有できるため、**管理が楽**です。

<details><summary>gpt3.5</summary>

## 2. OpenAI - gpt3.5 turbo

```python:get_chat_completion
def get_chat_completion(message_text):
    # Configure the OpenAI API settings
    openai.api_type = "azure"
    openai.api_base = os.getenv("OPENAI_API_BASE")
    openai.api_version = "2023-07-01-preview"
    openai.api_key = os.getenv("OPENAI_API_KEY")
    _messages =  [
        {
            "role": "system",
            "content": ("Below is the text read by OCR. The following is an "
                        "introduction to the Japanese text. Please check the "
                        "content and make sure the text is appropriate. If it is "
                        "not correct, please complete it so that it can be read out. "
                        "・Don't change the numerical values in the sentences.")
        },
        {
            "role": "user",
            "content": message_text
        }
    ]
    # Process the message using the Chat API
    completion = openai.ChatCompletion.create(
        engine=os.getenv("OPENAI_API_ENGINE"),
        messages=_messages,
        temperature=0.7,
        max_tokens=500,
        top_p=0.95,
        frequency_penalty=0,
        presence_penalty=0,
        stop=None
    )

    return completion['choices'][0]['message']['content']
```

</details>

こちらも[公式のサンプルコード](https://learn.microsoft.com/ja-jp/azure/ai-services/openai/quickstart?tabs=command-line%2Cpython&pivots=programming-language-python)を関数にしただけです。
`requirement.txt`に特徴があります。

```requirement.txt
openai = 0.28
```
:::note warn
ChatCompletion APIに対応しているバージョンが、openaiのマイナーバージョンです。バージョン指定をしないと沼にはまります。
[新バージョンは下記から！](https://learn.microsoft.com/ja-jp/azure/ai-services/openai/quickstart?tabs=command-line%2Cpython-new&pivots=programming-language-python)
:::

https://learn.microsoft.com/ja-jp/azure/ai-services/openai/quickstart?tabs=command-line%2Cpython-new&pivots=programming-language-python

## 3. Speech - Text to Speech

<details><summary>Text to Speech</summary>

```python:Text to Speech
def synthesize_speech_to_audio_data(text):
    """
    Converts the given text to speech and returns the base64 encoded audio data.

    Args:
    text (str): Text to be converted to speech.

    Returns:
    str: The base64 encoded audio data as a string.
    """

    # Configure Azure Speech Service
    speech_config = speechsdk.SpeechConfig(subscription=os.environ.get('MULTI_SERVICE_KEY'), 
                                           region=os.environ.get('MULTI_SERVICE_REGION'))
    audio_config = speechsdk.audio.AudioOutputConfig(use_default_speaker=True)

    # Set the voice for speech synthesis
    speech_config.speech_synthesis_voice_name='ja-JP-KeitaNeural'

    # Create an instance of SpeechSynthesizer
    speech_synthesizer = speechsdk.SpeechSynthesizer(speech_config=speech_config, 
                                                     audio_config=audio_config)

    # Convert text to speech
    speech_synthesis_result = speech_synthesizer.speak_text_async(text).get()

    # If synthesis is successful, encode and return the audio data
    if speech_synthesis_result.reason == speechsdk.ResultReason.SynthesizingAudioCompleted:
        audio_data = speech_synthesis_result.audio_data
        return base64.b64encode(audio_data).decode('utf-8')

    # If synthesis fails, raise an exception
    elif speech_synthesis_result.reason == speechsdk.ResultReason.Canceled:
        cancellation_details = speech_synthesis_result.cancellation_details
        raise Exception(f"Speech synthesis canceled: {cancellation_details.reason}, "
                        f"Error details: {cancellation_details.error_details}")
```

</details>

こちらも[公式のサンプルコード](https://learn.microsoft.com/ja-jp/azure/ai-services/speech-service/get-started-text-to-speech?tabs=windows%2Cterminal&pivots=programming-language-python)以下略

```python
base64.b64encode(audio_data).decode('utf-8')
```
こちらで`utf-8`を明記しないと文字化けするので気を付けましょう。

:::note warn
requestsで実装する場合、`ssml.encode('utf-8')`が必要
:::

音声は`圭太`さんを採用。
[音声ギャラリー](https://speech.microsoft.com/portal)で試してみてください。

```python:function_app.py
import azure.functions as func
import azure.cognitiveservices.speech as speechsdk
from azure.cognitiveservices.vision.computervision import ComputerVisionClient
from azure.cognitiveservices.vision.computervision.models import OperationStatusCodes
from msrest.authentication import CognitiveServicesCredentials
import os
import openai
import time
import base64
import logging
from io import BytesIO

app = func.FunctionApp(http_auth_level=func.AuthLevel.FUNCTION)

@app.route(route="voicelens", methods=['POST'])
def voicelens(req: func.HttpRequest) -> func.HttpResponse:
    encoded_image_data = req.params.get('image')
    if not encoded_image_data:
        try:
            req_body = req.get_json()
        except ValueError:
            pass
        else:
            encoded_image_data = req_body.get('image')
            # 1. Read API - OCR
            image_text = extract_text_from_image(encoded_image_data)
            # 2. OpenAI
            message = get_chat_completion(image_text)
            # 3. Text to Speech
            audio_data = synthesize_speech_to_audio_data(message)

    if image_text:
        return func.HttpResponse(audio_data)
    else:
        return func.HttpResponse(
             "This HTTP triggered function executed successfully.",
             status_code=200
        )

def extract_text_from_image(encoded_image_data):
    """
    Extracts text from the given base64-encoded image data using the Azure Computer Vision API.

    Args:
    encoded_image_data (str): The base64-encoded data of the image from which to extract the text.

    Returns:
    str: Extracted text from the image or None if the extraction fails.
    """
    # Decode the base64-encoded data
    image_data = base64.b64decode(encoded_image_data)    
    # Authenticate
    subscription_key = os.environ["MULTI_SERVICE_KEY"]
    endpoint = os.environ["MULTI_SERVICE_ENDPOINT"]

    computervision_client = ComputerVisionClient(endpoint, CognitiveServicesCredentials(subscription_key))

    # Call API with image data
    try:
        image_stream = BytesIO(image_data)
        read_response = computervision_client.read_in_stream(image_stream,  raw=True)

        # Get the operation location (URL with an ID at the end) from the response
        read_operation_location = read_response.headers["Operation-Location"]
        operation_id = read_operation_location.split("/")[-1]

        # Call the "GET" API and wait for it to retrieve the results 
        while True:
            read_result = computervision_client.get_read_result(operation_id)
            if read_result.status not in ['notStarted', 'running']:
                break
            time.sleep(1)

        # Extract and return text if operation succeeded
        if read_result.status == OperationStatusCodes.succeeded:
            return "".join(
                line.text for text_result in read_result.analyze_result.read_results
                for line in text_result.lines
            )
    except Exception as e:
        logging.info(f"An error occurred: {e}")
        return None

def get_chat_completion(message_text):
    # Configure the OpenAI API settings
    openai.api_type = "azure"
    openai.api_base = os.getenv("OPENAI_API_BASE")
    openai.api_version = "2023-07-01-preview"
    openai.api_key = os.getenv("OPENAI_API_KEY")
    _messages =  [
        {
            "role": "system",
            "content": ("Below is the text read by OCR. The following is an "
                        "introduction to the Japanese text. Please check the "
                        "content and make sure the text is appropriate. If it is "
                        "not correct, please complete it so that it can be read out. "
                        "・Don't change the numerical values in the sentences.")
        },
        {
            "role": "user",
            "content": message_text
        }
    ]
    # Process the message using the Chat API
    completion = openai.ChatCompletion.create(
        engine=os.getenv("OPENAI_API_ENGINE"),
        messages=_messages,
        temperature=0.7,
        max_tokens=500,
        top_p=0.95,
        frequency_penalty=0,
        presence_penalty=0,
        stop=None
    )

    return completion['choices'][0]['message']['content']

def synthesize_speech_to_audio_data(text):
    """
    Converts the given text to speech and returns the base64 encoded audio data.

    Args:
    text (str): Text to be converted to speech.

    Returns:
    str: The base64 encoded audio data as a string.
    """

    # Configure Azure Speech Service
    speech_config = speechsdk.SpeechConfig(subscription=os.environ.get('MULTI_SERVICE_KEY'), 
                                           region=os.environ.get('MULTI_SERVICE_REGION'))
    audio_config = speechsdk.audio.AudioOutputConfig(use_default_speaker=True)

    # Set the voice for speech synthesis
    speech_config.speech_synthesis_voice_name='ja-JP-KeitaNeural'

    # Create an instance of SpeechSynthesizer
    speech_synthesizer = speechsdk.SpeechSynthesizer(speech_config=speech_config, 
                                                     audio_config=audio_config)

    # Convert text to speech
    speech_synthesis_result = speech_synthesizer.speak_text_async(text).get()

    # If synthesis is successful, encode and return the audio data
    if speech_synthesis_result.reason == speechsdk.ResultReason.SynthesizingAudioCompleted:
        audio_data = speech_synthesis_result.audio_data
        return base64.b64encode(audio_data).decode('utf-8')

    # If synthesis fails, raise an exception
    elif speech_synthesis_result.reason == speechsdk.ResultReason.Canceled:
        cancellation_details = speech_synthesis_result.cancellation_details
        raise Exception(f"Speech synthesis canceled: {cancellation_details.reason}, "
                        f"Error details: {cancellation_details.error_details}")
```

関数の認証は`Function`にしています。
HTTP要求で実装しましたが、クエリ文字列にキーが必要になります。

あとは順番に
1. OCR
1. OpenAI
1. Text to Speech

を実装します。

今回、Pythonの外部ライブラリを使いましたが、正直`requests`で実装するほうが、楽でした・・。
ただ食わず嫌いもよくないですね。逃げずにGPT先生と向き合ってみました。

**Azure Functionsの場合、Power AutomateでHTTP 要求を連発して出すよりも速さの分があります。**
Azure上でセキュアに作りこめることも魅力ですね。

# Power Platform
## 1. Power Automate
Power Appsから`contentBytes`を受け取って処理する仕組みにします。
秘密鍵は一旦`作成`アクションに格納していますが、**Key Vault** が望ましいでしょう。

[過去に記事を書いている](https://qiita.com/DEmodoriGatsuO/items/2fae27220017f5f205ea)ので、詳細は下記をご覧ください。

https://qiita.com/DEmodoriGatsuO/items/2fae27220017f5f205ea

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2734332/3105a48f-f261-8930-0a38-82ea31a78442.png)

単純にHTTP 要求を送信しているだけです。
カスタム コネクタにして楽したほうが永続的だと思います。

::: note info
今回は趣味なので割愛
:::

## 2. Power Apps
テンプレートのUIを使って、モバイルを見越したパーツを設定していきます。
戻り値は`base64`にして受け取っています。
**使い慣れている観点からです**

Text to Speechの戻り値は`wav`です。

```plaintext
"data:audio/wav;base64," & Power Automateの戻り値
```

こちらでAudioコントロールが再生できます。

Power Automateの`マイクのアイコン`の`OnSelect`に
```
// 1. varAudioに音声読み上げを格納
Set(varAudio,PowerAutomate.Run({file:{name:"",contentBytes:UploadedImage1.Image}}));
// 2. Audioコントロールをリセット
Set(varReset,false);
Set(varReset,true);
```

これで👆のつぶやきの動作が再現できます！
物理の文章を読み上げる楽しさがあります✨

# おわりに
Azure Functionsさえ進めば、Power Automate、Power Appsは1時間もかからずに作れる内容です。
**物理の文章が簡単に読み起こせるなんて夢のようですね。**

**Azureサイコー**