OpenSearchでいい感じに検索できるようにしたい初見タイムアタック（とりあえずハイブリッド検索からリランクまでの道）


タイムアタックしてきます。23:30スタート。

Elasticsearchはさわったことあったけど、OpenSearchは初めてです。


公式サイト見ながらおもむろにdocker compose upします。
https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/

.envにOPENSEARCH_INITIAL_ADMIN_PASSWORD書くの忘れずに（1敗）


opensearch-node1       | [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
opensearch-node1       | ERROR: OpenSearch did not exit normally - check the logs at /usr/share/opensearch/logs/opensearch-cluster.log

…メモリが足りない…？（2敗）

https://zenn.dev/niaeashes/articles/f65f33e390eb41

docker composeに追記したりした。うんうん。
- discovery.type=single-node

とりあえずのdocker-compose.yaml
```
version: '3'
services:
  opensearch: # This is also the hostname of the container within the Docker network (i.e. https://opensearch-node1/)
    image: opensearchproject/opensearch:2.13.0 # Specifying the latest available image - modify if you want a specific version
    container_name: opensearch
    environment:
      - discovery.type=single-node
      - bootstrap.memory_lock=true # Disable JVM heap memory swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # Set min and max JVM heap sizes to at least 50% of system RAM
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}    # Sets the demo admin user password when using demo configuration, required for OpenSearch 2.12 and later
    ulimits:
      memlock:
        soft: -1 # Set memlock to unlimited (no soft or hard limit)
        hard: -1
      nofile:
        soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data # Creates volume called opensearch-data1 and mounts it to the container
    ports:
      - 9200:9200 # REST API
      - 9600:9600 # Performance Analyzer
    networks:
      - opensearch-net # All of the containers will join the same Docker bridge network

  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:2.13.0 # Make sure the version of opensearch-dashboards matches the version of opensearch installed on other nodes
    container_name: opensearch-dashboards
    ports:
      - 5601:5601 # Map host port 5601 to container port 5601
    expose:
      - "5601" # Expose port 5601 for web access to OpenSearch Dashboards
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch:9200"]' # Define the OpenSearch nodes that OpenSearch Dashboards will query
    networks:
      - opensearch-net

volumes:
  opensearch-data1:

networks:
  opensearch-net:
```

これで、OpenSearchと、OpenSearch Dashboardが立ち上がる。
https://localhost:9200でOpenSearchに、http://localhost:5601/でOpenSearch Dashboardにアクセスできる。ユーザーadminでパスワードは.envに書いたやつ。

```https://localhost:9200
{
  "name" : "4c1d66fdfc7f",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "TAoycaGEQa6OJMAoyfXyCQ",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.13.0",
    "build_type" : "tar",
    "build_hash" : "7ec678d1b7c87d6e779fdef94e33623e1f1e2647",
    "build_date" : "2024-03-26T00:02:39.659767978Z",
    "build_snapshot" : false,
    "lucene_version" : "9.10.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}
```

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/772969/5e839e2b-4681-cee7-4b5a-de7c81c85bd4.png)

OpenSearch Dashboardはkibanaみたいなやつですね。いろいろ便利なダッシュボード作れたりサンプルデータ入れれたりクエリかけたりします。


とりあえずRTAなので？こっちはサンプルデータぶち込むのに試してつかっただけでRTAなので生API使ってきます。というかdev toolsにあるコンソール叩いていきます
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/772969/0e1dd533-eb8b-0709-32fa-8b5c93eea203.png)


どうでもいいけど_catに猫いてかわいいね。
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/772969/0c6eaacc-e875-7863-8e1a-366b3f0fcec6.png)

この辺とか見ます
https://zenn.dev/i_shinya/articles/1ab30e7f4d3144

この辺みます。
https://opensearch.org/docs/latest/ml-commons-plugin/cluster-settings/

ここみます
https://opensearch.org/docs/latest/ml-commons-plugin/custom-local-models/
とりあえず
```
PUT _cluster/settings
{
  "persistent": {
    "plugins": {
      "ml_commons": {
        "allow_registering_model_via_url": "true",
        "only_run_on_ml_node": "false",
        "model_access_control_enabled": "true",
        "native_memory_threshold": "99"
      }
    }
  }
}
```

ふむふむ
```
POST /_plugins/_ml/model_groups/_register
{
  "name": "local_model_group",
  "description": "A model group for local models"
}
```
```
{
  "model_group_id": "RzSd7I4BBmWZ2-waQ5by",
  "status": "CREATED"
}
```
モデルの登録、e5使いたい
https://huggingface.co/intfloat/multilingual-e5-large
が、OpenSearch公式が準備してないのでRTAにはつらいのであきらめる。できそうなのかどうかがわからん。

日本語対応できてるparaphrase-multilingual-MiniLM-L12-v2で妥協しておく。商用ならOpenAIの使えばいいって話もあるよね。それはこのへんみてやるらしい　https://opensearch.org/docs/latest/ml-commons-plugin/remote-models/index/


```
POST /_plugins/_ml/models/_register
{
  "name": "huggingface/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",
  "version": "1.0.1",
  "model_group_id": "RzSd7I4BBmWZ2-waQ5by",
  "model_format": "TORCH_SCRIPT"
}

```
いけたかも
```
{
  "task_id": "bDSo7I4BBmWZ2-waFJZj",
  "status": "CREATED"
}
```
```
GET /_plugins/_ml/tasks/bDSo7I4BBmWZ2-waFJZj
```
```
{
  "model_id": "cDSo7I4BBmWZ2-waG5Z4",
  "task_type": "REGISTER_MODEL",
  "function_name": "TEXT_EMBEDDING",
  "state": "COMPLETED",
  "worker_node": [
    "nUQIhBClQ9STBFX6wPwKIA"
  ],
  "create_time": 1713367421968,
  "last_update_time": 1713367442905,
  "is_async": true
}
```
完了してるっぽい？

デプロイじゃー
```
POST /_plugins/_ml/models/cDSo7I4BBmWZ2-waG5Z4/_deploy
```
```
{
  "task_id": "ezSr7I4BBmWZ2-waXJYu",
  "task_type": "DEPLOY_MODEL",
  "status": "CREATED"
}
```
でけた。

テストします
```
POST /_plugins/_ml/_predict/text_embedding/cDSo7I4BBmWZ2-waG5Z4
{
  "text_docs":[ "日本語も使えるかな？"],
  "return_number": true,
  "target_response": ["sentence_embedding"]
}
```

```
{
  "inference_results": [
    {
      "output": [
        {
          "name": "sentence_embedding",
          "data_type": "FLOAT32",
          "shape": [
            384
          ],
          "data": [
            -0.18664902,
            -0.2343629,
            0.24738024,
            -0.05488886,
            -0.040530212,
            -0.1703863,
            ...略
          ]
        }
      ]
    }
  ]
}
```

でけた。



…


もしかしてAgentsつくれる…？
https://opensearch.org/docs/latest/ml-commons-plugin/agents-tools/index/


こっちのチュートリアルのが楽かも
https://opensearch.org/docs/latest/ml-commons-plugin/agents-tools/agents-tools-tutorial/

こっちに切り替え


```
PUT /_ingest/pipeline/test-pipeline-local-model
{
  "description": "text embedding pipeline",
  "processors": [
    {
      "text_embedding": {
        "model_id": "cDSo7I4BBmWZ2-waG5Z4",
        "field_map": {
          "text": "embedding"
        }
      }
    }
  ]
}
```
```
{
  "acknowledged": true
}
```
はい。

チュートリアルにそってKNNのindexをつくります。さっきのパイプラインでembeddingする感じでmy_test_dataって名前のindexです

```
PUT my_test_data
{
  "mappings": {
    "properties": {
      "text": {
        "type": "text"
      },
      "embedding": {
        "type": "knn_vector",
        "dimension": 384
      }
    }
  },
  "settings": {
    "index": {
      "knn.space_type": "cosinesimil",
      "default_pipeline": "test-pipeline-local-model",
      "knn": "true"
    }
  }
}
```
```
{
  "acknowledged": true,
  "shards_acknowledged": true,
  "index": "my_test_data"
}
```
はい。できた。

バルクインサートします。
```
POST _bulk
{"index": {"_index": "my_test_data", "_id": "1"}}
{"text": "Qiitaで記事書くの久しぶりかも。個人的には雑な記事で許される雰囲気が好き。"}
{"index": {"_index": "my_test_data", "_id": "2"}}
{"text": "テストデータです。"}
{"index": {"_index": "my_test_data", "_id": "3"}}
{"text": "ねむいよ。"}
{"index": {"_index": "my_test_data", "_id": "4"}}
{"text": "サンプルデータで検索引っかかりやすさの差が出るデータを作るのがめんどくさい。"}
{"index": {"_index": "my_test_data", "_id": "5"}}
{"text": "本日は晴天なり"}
{"index": {"_index": "my_test_data", "_id": "6"}}
{"text": "すっごくどうでもいいけど、「アイマリンプロジェクト 「Dive to Blue」 MMD MUSIC VIDEO」がマイブームです。https:\/\/www.youtube.com/watch?v=XCzGs6rLf4s"}
```

はいった
```
{
  "took": 18,
  "ingest_took": 138,
  "errors": false,
  "items": [
    {
      "index": {
        "_index": "my_test_data",
        "_id": "1",
        "_version": 1,
        "result": "created",
        "_shards": {
          "total": 2,
          "successful": 1,
          "failed": 0
        },
        "_seq_no": 0,
        "_primary_term": 1,
        "status": 201
      }
    },
    {
      "index": {
        "_index": "my_test_data",
        "_id": "2",
        "_version": 1,
        "result": "created",
        "_shards": {
          "total": 2,
          "successful": 1,
          "failed": 0
        },
        "_seq_no": 1,
        "_primary_term": 1,
        "status": 201
      }
    },
    {
      "index": {
        "_index": "my_test_data",
        "_id": "3",
        "_version": 1,
        "result": "created",
        "_shards": {
          "total": 2,
          "successful": 1,
          "failed": 0
        },
        "_seq_no": 2,
        "_primary_term": 1,
        "status": 201
      }
    },
    {
      "index": {
        "_index": "my_test_data",
        "_id": "4",
        "_version": 1,
        "result": "created",
        "_shards": {
          "total": 2,
          "successful": 1,
          "failed": 0
        },
        "_seq_no": 3,
        "_primary_term": 1,
        "status": 201
      }
    },
    {
      "index": {
        "_index": "my_test_data",
        "_id": "5",
        "_version": 1,
        "result": "created",
        "_shards": {
          "total": 2,
          "successful": 1,
          "failed": 0
        },
        "_seq_no": 4,
        "_primary_term": 1,
        "status": 201
      }
    },
    {
      "index": {
        "_index": "my_test_data",
        "_id": "6",
        "_version": 1,
        "result": "created",
        "_shards": {
          "total": 2,
          "successful": 1,
          "failed": 0
        },
        "_seq_no": 5,
        "_primary_term": 1,
        "status": 201
      }
    }
  ]
}
```

こっから先はLLMをつけるってことなっちゃったので違うのみながらやらないと…

```
GET my_test_data/_search
```

```
{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 6,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "my_test_data",
        "_id": "1",
        "_score": 1,
        "_source": {
          "text": "Qiitaで記事書くの久しぶりかも。個人的には雑な記事で許される雰囲気が好き。",
          "embedding": [
            -0.085727125,
            -0.18462902,
            -0.13459924,
            0.2155644,
            ...
```
はいってはいる。

検索はこんな感じ
```
GET /my_test_data/_search
{
  "_source": {
    "exclude": [
      "embedding"
    ]
  },
  "query": {
    "neural": {
      "embedding": {
        "query_text": "MMD MUSIC VIDEO",
        "model_id": "cDSo7I4BBmWZ2-waG5Z4",
        "k": 5
      }
    }
  }
}
```
```
{
  "took": 13,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 5,
      "relation": "eq"
    },
    "max_score": 0.7970713,
    "hits": [
      {
        "_index": "my_test_data",
        "_id": "6",
        "_score": 0.7970713,
        "_source": {
          "text": "すっごくどうでもいいけど、「アイマリンプロジェクト 「Dive to Blue」 MMD MUSIC VIDEO」がマイブームです。https://www.youtube.com/watch?v=XCzGs6rLf4s"
        }
      },
      {
        "_index": "my_test_data",
        "_id": "3",
        "_score": 0.54350144,
        "_source": {
          "text": "ねむいよ。"
        }
      },
      {
        "_index": "my_test_data",
        "_id": "5",
        "_score": 0.5398678,
        "_source": {
          "text": "本日は晴天なり"
        }
      },
      {
        "_index": "my_test_data",
        "_id": "2",
        "_score": 0.52882004,
        "_source": {
          "text": "テストデータです。"
        }
      },
      {
        "_index": "my_test_data",
        "_id": "1",
        "_score": 0.5017807,
        "_source": {
          "text": "Qiitaで記事書くの久しぶりかも。個人的には雑な記事で許される雰囲気が好き。"
        }
      }
    ]
  }
}
```
ちなみにキーワードかえると
```
GET /my_test_data/_search
{
  "_source": {
    "exclude": [
      "embedding"
    ]
  },
  "query": {
    "neural": {
      "embedding": {
        "query_text": "アイマリンプロジェクト",
        "model_id": "cDSo7I4BBmWZ2-waG5Z4",
        "k": 5
      }
    }
  }
}
```
```
{
  "took": 15,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 5,
      "relation": "eq"
    },
    "max_score": 0.69344866,
    "hits": [
      {
        "_index": "my_test_data",
        "_id": "5",
        "_score": 0.69344866,
        "_source": {
          "text": "本日は晴天なり"
        }
      },
      {
        "_index": "my_test_data",
        "_id": "3",
        "_score": 0.66218674,
        "_source": {
          "text": "ねむいよ。"
        }
      },
      {
        "_index": "my_test_data",
        "_id": "2",
        "_score": 0.58365303,
        "_source": {
          "text": "テストデータです。"
        }
      },
      {
        "_index": "my_test_data",
        "_id": "1",
        "_score": 0.57552934,
        "_source": {
          "text": "Qiitaで記事書くの久しぶりかも。個人的には雑な記事で許される雰囲気が好き。"
        }
      },
      {
        "_index": "my_test_data",
        "_id": "6",
        "_score": 0.5615421,
        "_source": {
          "text": "すっごくどうでもいいけど、「アイマリンプロジェクト 「Dive to Blue」 MMD MUSIC VIDEO」がマイブームです。https://www.youtube.com/watch?v=XCzGs6rLf4s"
        }
      }
    ]
  }
}
```
アイマリンプロジェクトは0.5615421でめっちゃ下に出てきます。paraphrase-multilingual-MiniLM-L12-v2の限界を感じる。


ハイブリッド検索してみよう…
と思ったけどkuromojiがはいってない
```
GET _cat/plugins
```
```
4c1d66fdfc7f opensearch-alerting                  2.13.0.0
4c1d66fdfc7f opensearch-anomaly-detection         2.13.0.0
4c1d66fdfc7f opensearch-asynchronous-search       2.13.0.0
4c1d66fdfc7f opensearch-cross-cluster-replication 2.13.0.0
4c1d66fdfc7f opensearch-custom-codecs             2.13.0.0
4c1d66fdfc7f opensearch-flow-framework            2.13.0.0
4c1d66fdfc7f opensearch-geospatial                2.13.0.0
4c1d66fdfc7f opensearch-index-management          2.13.0.0
4c1d66fdfc7f opensearch-job-scheduler             2.13.0.0
4c1d66fdfc7f opensearch-knn                       2.13.0.0
4c1d66fdfc7f opensearch-ml                        2.13.0.0
4c1d66fdfc7f opensearch-neural-search             2.13.0.0
4c1d66fdfc7f opensearch-notifications             2.13.0.0
4c1d66fdfc7f opensearch-notifications-core        2.13.0.0
4c1d66fdfc7f opensearch-observability             2.13.0.0
4c1d66fdfc7f opensearch-performance-analyzer      2.13.0.0
4c1d66fdfc7f opensearch-reports-scheduler         2.13.0.0
4c1d66fdfc7f opensearch-security                  2.13.0.0
4c1d66fdfc7f opensearch-security-analytics        2.13.0.0
4c1d66fdfc7f opensearch-skills                    2.13.0.0
4c1d66fdfc7f opensearch-sql                       2.13.0.0
```


docker buildから…。えーん。（3敗）

ホントはimageにしちゃったほうがいいけどbuildで記載

```Dockerfile
FROM opensearchproject/opensearch:2.13.0
RUN opensearch-plugin install analysis-kuromoji
```

```docker-compose.yaml
version: '3'
services:
  opensearch: # This is also the hostname of the container within the Docker network (i.e. https://opensearch-node1/)
    build: .
    # image: opensearchproject/opensearch:2.13.0
    
```
とりあえずビルドしてimageを消しておく。

はいった

```
GET _cat/plugins
```
```
b81fc9256727 analysis-kuromoji                    2.13.0
b81fc9256727 opensearch-alerting                  2.13.0.0
b81fc9256727 opensearch-anomaly-detection         2.13.0.0
b81fc9256727 opensearch-asynchronous-search       2.13.0.0
b81fc9256727 opensearch-cross-cluster-replication 2.13.0.0
b81fc9256727 opensearch-custom-codecs             2.13.0.0
b81fc9256727 opensearch-flow-framework            2.13.0.0
b81fc9256727 opensearch-geospatial                2.13.0.0
b81fc9256727 opensearch-index-management          2.13.0.0
b81fc9256727 opensearch-job-scheduler             2.13.0.0
b81fc9256727 opensearch-knn                       2.13.0.0
b81fc9256727 opensearch-ml                        2.13.0.0
b81fc9256727 opensearch-neural-search             2.13.0.0
b81fc9256727 opensearch-notifications             2.13.0.0
b81fc9256727 opensearch-notifications-core        2.13.0.0
b81fc9256727 opensearch-observability             2.13.0.0
b81fc9256727 opensearch-performance-analyzer      2.13.0.0
b81fc9256727 opensearch-reports-scheduler         2.13.0.0
b81fc9256727 opensearch-security                  2.13.0.0
b81fc9256727 opensearch-security-analytics        2.13.0.0
b81fc9256727 opensearch-skills                    2.13.0.0
b81fc9256727 opensearch-sql                       2.13.0.0
```

ここを参考に切り替えます
https://opensearch.org/docs/latest/search-plugins/hybrid-search/

なお
https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
の次元は384なので上の方の設定はかえなくてよかったのね

ハイブリッド用のindexつくります
```
PUT /my-nlp-index
{
  "settings": {
    "index.knn": true,
    "default_pipeline": "test-pipeline-local-model"
  },
  "mappings": {
    "properties": {
      "id": {
        "type": "text"
      },
      "embedding": {
        "type": "knn_vector",
        "dimension": 768,
        "method": {
          "engine": "lucene",
          "space_type": "l2",
          "name": "hnsw",
          "parameters": {}
        }
      },
      "text": {
        "type": "text",
        "analyzer": "kuromoji"
      }
    }
  }
}
```

バルクインサートします。
```
POST _bulk
{"index": {"_index": "my-nlp-index", "_id": "1"}}
{"text": "Qiitaで記事書くの久しぶりかも。個人的には雑な記事で許される雰囲気が好き。"}
{"index": {"_index": "my-nlp-index", "_id": "2"}}
{"text": "テストデータです。"}
{"index": {"_index": "my-nlp-index", "_id": "3"}}
{"text": "ねむいよ。"}
{"index": {"_index": "my-nlp-index", "_id": "4"}}
{"text": "サンプルデータで検索引っかかりやすさの差が出るデータを作るのがめんどくさい。"}
{"index": {"_index": "my-nlp-index", "_id": "5"}}
{"text": "本日は晴天なり"}
{"index": {"_index": "my-nlp-index", "_id": "6"}}
{"text": "すっごくどうでもいいけど、「アイマリンプロジェクト 「Dive to Blue」 MMD MUSIC VIDEO」がマイブームです。https:\/\/www.youtube.com/watch?v=XCzGs6rLf4s"}
```
errorになった

次元がまちがってます。

DELETE /my-nlp-index

でindex消して再作成します
```
PUT /my-nlp-index
{
  "settings": {
    "index.knn": true,
    "default_pipeline": "test-pipeline-local-model"
  },
  "mappings": {
    "properties": {
      "id": {
        "type": "text"
      },
      "embedding": {
        "type": "knn_vector",
        "dimension": 384,
        "method": {
          "engine": "lucene",
          "space_type": "l2",
          "name": "hnsw",
          "parameters": {}
        }
      },
      "text": {
        "type": "text",
        "analyzer": "kuromoji"
      }
    }
  }
}
```

バルクインサート、今度は入ります。
```
{
  "took": 56,
  "ingest_took": 57,
  "errors": false,
  "items": [
    {
      "index": {
        "_index": "my-nlp-index",
        "_id": "1",
        "_version": 1,
        "result": "created",
        "_shards": {
          "total": 2,
          "successful": 1,
          "failed": 0
        },
        "_seq_no": 0,
        "_primary_term": 1,
        "status": 201
      }
    },
...
 ```

 検索かかる（簡略表記でアイマリンプロジェクトでキーワード検索。ちゃんと1件返ってきます。
 
 （kuromojiの設定がアレなので形態素解析が微妙、sudachi Bがいいなぁ（sudachi参考　https://qiita.com/mh-northlander/items/83c3888bb5fefe34be20)
 
 
 
 ```
 GET my-nlp-index/_search?q=アイマリンプロジェクト
 ```
 ```

 {
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 0.88376784,
    "hits": [
      {
        "_index": "my-nlp-index",
        "_id": "6",
        "_score": 0.88376784,
        "_source": {
          "text": "すっごくどうでもいいけど、「アイマリンプロジェクト 「Dive to Blue」 MMD MUSIC VIDEO」がマイブームです。https://www.youtube.com/watch?v=XCzGs6rLf4s",
          "embedding": [
            -0.053731408,
            -0.13152666,
            -0.065659426
            ...
 ```


 3:7の割合のハイブリッド検索設定します
 ```
 PUT /_search/pipeline/nlp-search-pipeline
{
  "description": "Post processor for hybrid search",
  "phase_results_processors": [
    {
      "normalization-processor": {
        "normalization": {
          "technique": "min_max"
        },
        "combination": {
          "technique": "arithmetic_mean",
          "parameters": {
            "weights": [
              0.3,
              0.7
            ]
          }
        }
      }
    }
  ]
}
```

検索もできた
```
GET /my-nlp-index/_search?search_pipeline=nlp-search-pipeline
{
  "_source": {
    "exclude": [
      "embedding"
    ]
  },
  "query": {
    "hybrid": {
      "queries": [
        {
          "match": {
            "text": {
              "query": "アイマリンプロジェクト"
            }
          }
        },
        {
          "neural": {
            "embedding": {
              "query_text": "アイマリンプロジェクト",
              "model_id": "cDSo7I4BBmWZ2-waG5Z4",
              "k": 5
            }
          }
        }
      ]
    }
  }
}
```
```
{
  "took": 17,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 5,
      "relation": "eq"
    },
    "max_score": 0.7,
    "hits": [
      {
        "_index": "my-nlp-index",
        "_id": "5",
        "_score": 0.7,
        "_source": {
          "text": "本日は晴天なり"
        }
      },
      {
        "_index": "my-nlp-index",
        "_id": "3",
        "_score": 0.53058726,
        "_source": {
          "text": "ねむいよ。"
        }
      },
      {
        "_index": "my-nlp-index",
        "_id": "6",
        "_score": 0.38876915,
        "_source": {
          "text": "すっごくどうでもいいけど、「アイマリンプロジェクト 「Dive to Blue」 MMD MUSIC VIDEO」がマイブームです。https://www.youtube.com/watch?v=XCzGs6rLf4s"
        }
      },
      {
        "_index": "my-nlp-index",
        "_id": "1",
        "_score": 0.04862528,
        "_source": {
          "text": "Qiitaで記事書くの久しぶりかも。個人的には雑な記事で許される雰囲気が好き。"
        }
      },
      {
        "_index": "my-nlp-index",
        "_id": "4",
        "_score": 0.00070000003,
        "_source": {
          "text": "サンプルデータで検索引っかかりやすさの差が出るデータを作るのがめんどくさい。"
        }
      }
    ]
  }
}
```

キーワード検索単体とスコアが違いますね。KNN単体だと最下位だったのと断トツのキーワード検索結果で中間の3位になっています。キーワード検索の比率あげればもうちょいあがりそうですね。
min-maxで0.3:0.7でやってるのでキーワード検索のノーマライズしたスコアが最大0.3、ベクトル検索したノーマライズしたスコアが最大0.7になっているので、_id:6の順位が下がりました。これは埋め込みモデルの性能が悪いので類似度があがらかなったからです。

```
GET /my-nlp-index/_search?search_pipeline=nlp-search-pipeline
{
  "_source": {
    "exclude": [
      "embedding"
    ]
  },
  "query": {
    "hybrid": {
      "queries": [
        {
          "match": {
            "text": {
              "query": "「アイマリンプロジェクト 「Dive to Blue」"
            }
          }
        },
        {
          "neural": {
            "embedding": {
              "query_text": "「アイマリンプロジェクト 「Dive to Blue」",
              "model_id": "cDSo7I4BBmWZ2-waG5Z4",
              "k": 5
            }
          }
        }
      ]
    }
  }
}
```
```
{
  "took": 24,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 5,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "my-nlp-index",
        "_id": "6",
        "_score": 1,
        "_source": {
          "text": "すっごくどうでもいいけど、「アイマリンプロジェクト 「Dive to Blue」 MMD MUSIC VIDEO」がマイブームです。https://www.youtube.com/watch?v=XCzGs6rLf4s"
        }
      },
      {
        "_index": "my-nlp-index",
        "_id": "5",
        "_score": 0.67498726,
        "_source": {
          "text": "本日は晴天なり"
        }
      },
      {
        "_index": "my-nlp-index",
        "_id": "3",
        "_score": 0.5904988,
        "_source": {
          "text": "ねむいよ。"
        }
      },
      {
        "_index": "my-nlp-index",
        "_id": "1",
        "_score": 0.05813713,
        "_source": {
          "text": "Qiitaで記事書くの久しぶりかも。個人的には雑な記事で許される雰囲気が好き。"
        }
      },
      {
        "_index": "my-nlp-index",
        "_id": "4",
        "_score": 0.00070000003,
        "_source": {
          "text": "サンプルデータで検索引っかかりやすさの差が出るデータを作るのがめんどくさい。"
        }
      }
    ]
  }
}
```

こんなけ長いフレーズにするとスコアが1でばっちり検索できています。



折角だし、リランクまでいけー
https://opensearch.org/docs/latest/search-plugins/search-pipelines/rerank-processor/

```
PUT /_search/pipeline/rerank_pipeline
{
  "response_processors": [
    {
      "rerank": {
        "ml_opensearch": {
          "model_id": "cDSo7I4BBmWZ2-waG5Z4"
        },
        "context": {
          "document_fields": ["text"]
        }
      }
    }
  ]
}
```
は通るけど
```
POST /_search?search_pipeline=rerank_pipeline
{
  "query": {
    "match": {
      "text": "アイマリンプロジェクト"
    }
  },
  "ext": {
    "rerank": {
      "query_context": {
        "query_text": "アイマリンプロジェクト"
      }
    }
  }
}

```

はいだめー

cross-encoder-modelsが必要なようです。

https://opensearch.org/docs/latest/ml-commons-plugin/pretrained-models/#cross-encoder-models

huggingface/cross-encoders/ms-marco-MiniLM-L-12-v2
をいれます。

```

POST /_plugins/_ml/models/_register
{
  "name": "huggingface/cross-encoders/ms-marco-MiniLM-L-12-v2",
  "version": "1.0.2",
  "model_group_id": "RzSd7I4BBmWZ2-waQ5by",
  "model_format": "TORCH_SCRIPT"
}

```

でけた
```
{
  "model_id": "cvP_7I4B7mnqBvbSU4Qd",
  "task_type": "REGISTER_MODEL",
  "function_name": "TEXT_SIMILARITY",
  "state": "COMPLETED",
  "worker_node": [
    "nUQIhBClQ9STBFX6wPwKIA"
  ],
  "create_time": 1713373138845,
  "last_update_time": 1713373151859,
  "is_async": true
}
```
モデルをデプロイして…
```
POST /_plugins/_ml/models/cvP_7I4B7mnqBvbSU4Qd/_deploy
```

リランクパイプライン作り直し
```
PUT /_search/pipeline/rerank_pipeline
{
  "response_processors": [
    {
      "rerank": {
        "ml_opensearch": {
          "model_id": "cvP_7I4B7mnqBvbSU4Qd"
        },
        "context": {
          "document_fields": ["text"]
        }
      }
    }
  ]
}
```

いけ！
```
POST my_hybrid_index/_search?search_pipeline=rerank_pipeline
{
  "query": {
    "match": {
      "text": "アイマリンプロジェクト"
    }
  },
  "ext": {
    "rerank": {
      "query_context": {
        "query_text": "アイマリンプロジェクト"
      }
    }
  }
}
```
```
{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 7.644516,
    "hits": [
      {
        "_index": "my_hybrid_index",
        "_id": "6",
        "_score": 7.644516,
        "_source": {
          "text": "すっごくどうでもいいけど、「アイマリンプロジェクト 「Dive to Blue」 MMD MUSIC VIDEO」がマイブームです。https://www.youtube.com/watch?v=XCzGs6rLf4s"
        }
      }
    ]
  },
  "profile": {
    "shards": []
  }
}
```
なんかかえってきた！

クエリの指定がキーワード検索で、1件しかあたってないのでかけらもリランクの意味がありません。

やり方あってるかわからないけど、強引にやってみます

```
POST my-nlp-index/_search?search_pipeline=nlp-search-pipeline&search_pipeline=rerank_pipeline
{ 
  "_source": {
    "exclude": [
      "embedding"
    ]
  },
  "query": {
    "hybrid": {
      "queries": [
        {
          "match": {
            "text": {
              "query": "アイマリンプロジェクト "
            }
          }
        },
        {
          "neural": {
            "embedding": {
              "query_text": "アイマリンプロジェクト",
              "model_id": "cDSo7I4BBmWZ2-waG5Z4",
              "k": 5
            }
          }
        }
      ]
    }
  }
  ,
  "ext": {
    "rerank": {
      "query_context": {
        "query_text": "アイマリンプロジェクト"
      }
    }
  }
}
```
```
{
  "took": 16,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 5,
      "relation": "eq"
    },
    "max_score": 7.644516,
    "hits": [
      {
        "_index": "my-nlp-index",
        "_id": "6",
        "_score": 7.644516,
        "_source": {
          "text": "すっごくどうでもいいけど、「アイマリンプロジェクト 「Dive to Blue」 MMD MUSIC VIDEO」がマイブームです。https://www.youtube.com/watch?v=XCzGs6rLf4s"
        }
      },
      {
        "_index": "my-nlp-index",
        "_id": "6",
        "_score": 7.644516,
        "_source": {
          "text": "すっごくどうでもいいけど、「アイマリンプロジェクト 「Dive to Blue」 MMD MUSIC VIDEO」がマイブームです。https://www.youtube.com/watch?v=XCzGs6rLf4s"
        }
      },
      {
        "_index": "my-nlp-index",
        "_id": "6",
        "_score": 7.644516,
        "_source": {
          "text": "すっごくどうでもいいけど、「アイマリンプロジェクト 「Dive to Blue」 MMD MUSIC VIDEO」がマイブームです。https://www.youtube.com/watch?v=XCzGs6rLf4s"
        }
      },
      {
        "_index": "my-nlp-index",
        "_id": "6",
        "_score": 7.644516,
        "_source": {
          "text": "すっごくどうでもいいけど、「アイマリンプロジェクト 「Dive to Blue」 MMD MUSIC VIDEO」がマイブームです。https://www.youtube.com/watch?v=XCzGs6rLf4s"
        }
      },
      {
        "_index": "my-nlp-index",
        "_id": "6",
        "_score": 7.644516,
        "_source": {
          "text": "すっごくどうでもいいけど、「アイマリンプロジェクト 「Dive to Blue」 MMD MUSIC VIDEO」がマイブームです。https://www.youtube.com/watch?v=XCzGs6rLf4s"
        }
      },
      {
        "_index": "my-nlp-index",
        "_id": "6",
        "_score": 7.644516,
        "_source": {
          "text": "すっごくどうでもいいけど、「アイマリンプロジェクト 「Dive to Blue」 MMD MUSIC VIDEO」がマイブームです。https://www.youtube.com/watch?v=XCzGs6rLf4s"
        }
      },
      {
        "_index": "my-nlp-index",
        "_id": "4",
        "_score": 7.614469,
        "_source": {
          "text": "サンプルデータで検索引っかかりやすさの差が出るデータを作るのがめんどくさい。"
        }
      },
      {
        "_index": "my-nlp-index",
        "_id": "3",
        "_score": 3.2796283,
        "_source": {
          "text": "ねむいよ。"
        }
      },
      {
        "_index": "my-nlp-index",
        "_id": "1",
        "_score": -0.10866069,
        "_source": {
          "text": "Qiitaで記事書くの久しぶりかも。個人的には雑な記事で許される雰囲気が好き。"
        }
      },
      {
        "_index": "my-nlp-index",
        "_id": "5",
        "_score": -1.6300647,
        "_source": {
          "text": "本日は晴天なり"
        }
      }
    ]
  },
  "profile": {
    "shards": []
  }
}
```

なんか分裂したけどできた！なんで分裂したんや…。

ハイブリッドリランクパイプライン作ってみます
```
 PUT /_search/pipeline/hybrid-rerank-pipeline
{
  "description": "Post processor for hybrid search",
  "phase_results_processors": [
    {
      "normalization-processor": {
        "normalization": {
          "technique": "min_max"
        },
        "combination": {
          "technique": "arithmetic_mean",
          "parameters": {
            "weights": [
              0.3,
              0.7
            ]
          }
        }
      }
    }
  ],
  "response_processors": [
    {
      "rerank": {
        "ml_opensearch": {
          "model_id": "cvP_7I4B7mnqBvbSU4Qd"
        },
        "context": {
          "document_fields": ["text"]
        }
      }
    }
  ]
}
```

はい！いい感じになりました。

```
POST my-nlp-index/_search?search_pipeline=hybrid-rerank-pipeline
{ 
  "_source": {
    "exclude": [
      "embedding"
    ]
  },
  "query": {
    "hybrid": {
      "queries": [
        {
          "match": {
            "text": {
              "query": "アイマリンプロジェクト "
            }
          }
        },
        {
          "neural": {
            "embedding": {
              "query_text": "アイマリンプロジェクト",
              "model_id": "cDSo7I4BBmWZ2-waG5Z4",
              "k": 5
            }
          }
        }
      ]
    }
  }
  ,
  "ext": {
    "rerank": {
      "query_context": {
        "query_text": "アイマリンプロジェクト"
      }
    }
  }
}
```
```
{
  "took": 15,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 5,
      "relation": "eq"
    },
    "max_score": 7.644516,
    "hits": [
      {
        "_index": "my-nlp-index",
        "_id": "6",
        "_score": 7.644516,
        "_source": {
          "text": "すっごくどうでもいいけど、「アイマリンプロジェクト 「Dive to Blue」 MMD MUSIC VIDEO」がマイブームです。https://www.youtube.com/watch?v=XCzGs6rLf4s"
        }
      },
      {
        "_index": "my-nlp-index",
        "_id": "4",
        "_score": 7.614469,
        "_source": {
          "text": "サンプルデータで検索引っかかりやすさの差が出るデータを作るのがめんどくさい。"
        }
      },
      {
        "_index": "my-nlp-index",
        "_id": "3",
        "_score": 3.2796283,
        "_source": {
          "text": "ねむいよ。"
        }
      },
      {
        "_index": "my-nlp-index",
        "_id": "1",
        "_score": -0.10866069,
        "_source": {
          "text": "Qiitaで記事書くの久しぶりかも。個人的には雑な記事で許される雰囲気が好き。"
        }
      },
      {
        "_index": "my-nlp-index",
        "_id": "5",
        "_score": -1.6300647,
        "_source": {
          "text": "本日は晴天なり"
        }
      }
    ]
  },
  "profile": {
    "shards": []
  }
}
```

本当にいい感じかは謎。

とりあえず完走（2:20）