Amazon Kendra のIndexを構築する

# なんの記事？
- AWS上でのAmazonBedrockを利用したRAGパターン実装のためにKendraのIndexを構築する
- この後、LangChainからKendraのAPIとbedrock(Claude-v2)のAPI実行によりRAG実装を目指す

# 参考ドキュメント
かきBuilders-flashの記事を参考にIndexを作成する
https://aws.amazon.com/jp/builders-flash/202302/kendra-search-system/?awsf.filter-name=*all

# Indexの作成
- Indexの作成はコンソールから実行する
- AWS Manegement ConsoleからKendraにアクセス
- リージョンは東京
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/573632/83cb3fcb-cee2-4f59-8691-28cd6132bb6b.png)
- Create Indexを押下
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/573632/9c6397fc-d44a-d6d4-830b-2fbb33b0a8b1.png)
- 新規にロールを作成
- 他はデフォルトでNextを押下
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/573632/ce3c0c36-b4bd-1fcb-28dc-3b809a21b1f5.png)
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/573632/07281df3-de18-9e1d-e43d-d29bbe4a0722.png)
- デフォルトでNextを押下
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/573632/dc7714ec-820e-79e5-3f4b-a81ebad07dc1.png)
- DeveloperEditionで構築する(今回は最初なので無料枠が適用されるはず)
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/573632/4bb33f3e-4a43-3bd1-1025-b022bf804ecc.png)
- Create を押下
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/573632/15db6f6a-ed34-febb-f8bc-1a6d00a3a98d.png)
- 作成にめっちゃ時間かかる
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/573632/3ef0b959-70d6-6cc0-4c80-3505782e51b4.png)
- 作成成功
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/573632/bd65d82d-bdfe-96bc-309e-212767bff43f.png)
# DataSouceの作成
- CloudShellからS3バケットの作成とデータ登録を行う
``` cloudshell .
[cloudshell-user@ip-10-2-77-33 ~]$ BUCKET_NAME=kendra-datasource-bucket-test-20231018
[cloudshell-user@ip-10-2-77-33 ~]$ aws s3 mb s3://${BUCKET_NAME}
make_bucket: kendra-datasource-bucket-test-20231018
[cloudshell-user@ip-10-2-77-33 ~]$ mkdir awsdoc
[cloudshell-user@ip-10-2-77-33 ~]$ pushd awsdoc
~/awsdoc ~
[cloudshell-user@ip-10-2-77-33 awsdoc]$ wget https://docs.aws.amazon.com/ja_jp/amazondynamodb/latest/developerguide/dynamodb-dg.pdf -O DynamoDB.pdf
--2023-10-18 14:31:36--  https://docs.aws.amazon.com/ja_jp/amazondynamodb/latest/developerguide/dynamodb-dg.pdf
Resolving docs.aws.amazon.com (docs.aws.amazon.com)... 99.84.133.100, 99.84.133.122, 99.84.133.24, ...
Connecting to docs.aws.amazon.com (docs.aws.amazon.com)|99.84.133.100|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 34032310 (32M) [application/pdf]
Saving to: ‘DynamoDB.pdf’

100%[==============================================================================================================================================================================================>] 34,032,310  26.4MB/s   in 1.2s   

2023-10-18 14:31:38 (26.4 MB/s) - ‘DynamoDB.pdf’ saved [34032310/34032310]

[cloudshell-user@ip-10-2-77-33 awsdoc]$ wget https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/lambda-dg.pdf -O Lambda.pdf
--2023-10-18 14:31:47--  https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/lambda-dg.pdf
Resolving docs.aws.amazon.com (docs.aws.amazon.com)... 99.84.133.105, 99.84.133.24, 99.84.133.122, ...
Connecting to docs.aws.amazon.com (docs.aws.amazon.com)|99.84.133.105|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 20743829 (20M) [application/pdf]
Saving to: ‘Lambda.pdf’

100%[==============================================================================================================================================================================================>] 20,743,829  28.1MB/s   in 0.7s   

2023-10-18 14:31:47 (28.1 MB/s) - ‘Lambda.pdf’ saved [20743829/20743829]

[cloudshell-user@ip-10-2-77-33 awsdoc]$ wget https://docs.aws.amazon.com/ja_jp/vpc/latest/userguide/vpc-ug.pdf -O VPC.pdf
--2023-10-18 14:31:57--  https://docs.aws.amazon.com/ja_jp/vpc/latest/userguide/vpc-ug.pdf
Resolving docs.aws.amazon.com (docs.aws.amazon.com)... 99.84.133.100, 99.84.133.105, 99.84.133.24, ...
Connecting to docs.aws.amazon.com (docs.aws.amazon.com)|99.84.133.100|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4310360 (4.1M) [application/pdf]
Saving to: ‘VPC.pdf’

100%[==============================================================================================================================================================================================>] 4,310,360   5.98MB/s   in 0.7s   

2023-10-18 14:31:57 (5.98 MB/s) - ‘VPC.pdf’ saved [4310360/4310360]

[cloudshell-user@ip-10-2-77-33 awsdoc]$ wget https://docs.aws.amazon.com/ja_jp/kendra/latest/dg/kendra-dg.pdf -O Kendra.pdf
--2023-10-18 14:32:06--  https://docs.aws.amazon.com/ja_jp/kendra/latest/dg/kendra-dg.pdf
Resolving docs.aws.amazon.com (docs.aws.amazon.com)... 99.84.133.122, 99.84.133.100, 99.84.133.105, ...
Connecting to docs.aws.amazon.com (docs.aws.amazon.com)|99.84.133.122|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6436120 (6.1M) [application/pdf]
Saving to: ‘Kendra.pdf’

100%[==============================================================================================================================================================================================>] 6,436,120   8.85MB/s   in 0.7s   

2023-10-18 14:32:07 (8.85 MB/s) - ‘Kendra.pdf’ saved [6436120/6436120]

[cloudshell-user@ip-10-2-77-33 awsdoc]$ wget https://docs.aws.amazon.com/ja_jp/Route53/latest/DeveloperGuide/route53-dg.pdf -O Route53.pdf
--2023-10-18 14:32:16--  https://docs.aws.amazon.com/ja_jp/Route53/latest/DeveloperGuide/route53-dg.pdf
Resolving docs.aws.amazon.com (docs.aws.amazon.com)... 99.84.133.24, 99.84.133.122, 99.84.133.100, ...
Connecting to docs.aws.amazon.com (docs.aws.amazon.com)|99.84.133.24|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8692485 (8.3M) [application/pdf]
Saving to: ‘Route53.pdf’

100%[==============================================================================================================================================================================================>] 8,692,485   9.30MB/s   in 0.9s   

2023-10-18 14:32:17 (9.30 MB/s) - ‘Route53.pdf’ saved [8692485/8692485]

[cloudshell-user@ip-10-2-77-33 awsdoc]$ popd
~
[cloudshell-user@ip-10-2-77-33 ~]$ aws s3 cp awsdoc s3://${BUCKET_NAME}/awsdoc/ --recursive
upload: awsdoc/VPC.pdf to s3://kendra-datasource-bucket-test-20231018/awsdoc/VPC.pdf
upload: awsdoc/Kendra.pdf to s3://kendra-datasource-bucket-test-20231018/awsdoc/Kendra.pdf
upload: awsdoc/DynamoDB.pdf to s3://kendra-datasource-bucket-test-20231018/awsdoc/DynamoDB.pdf
upload: awsdoc/Route53.pdf to s3://kendra-datasource-bucket-test-20231018/awsdoc/Route53.pdf
upload: awsdoc/Lambda.pdf to s3://kendra-datasource-bucket-test-20231018/awsdoc/Lambda.pdf
```
- また下記のようなテキストをS3に配置する
``` office_info.txt
■オフィス間取りについて
一般受付は2Fの東側にあります。
オフィスフロアは19Fから35Fです。
来客向け会議室は36Fの南側にあります。
展望室は37Fです。

■ワークフロー申請方法について
交通費申請はグループ会社共有Webシステムからワークフローの申請をしてください。
在宅勤務申請は自社Webシステムから申請してください。
部内備品の持ち出しはExcelにて申請してください。
```
``` cloudshell .
[cloudshell-user@ip-10-2-77-33 ~]$ mkdir officedoc
[cloudshell-user@ip-10-2-77-33 ~]$ pushd officedoc
~/officedoc ~
[cloudshell-user@ip-10-2-77-33 officedoc]$ touch office_info.txt
[cloudshell-user@ip-10-2-77-33 officedoc]$ vim office_info.txt 
[cloudshell-user@ip-10-2-77-33 officedoc]$ popd
~
[cloudshell-user@ip-10-2-77-33 ~]$ aws s3 cp officedoc s3://${BUCKET_NAME}/officedoc/ --recursive
upload: officedoc/office_info.txt to s3://kendra-datasource-bucket-test-20231018/officedoc/office_info.txt
```
- バケット確認
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/573632/00dc8fb9-86e8-dd41-56c5-53cf5a138a85.png)

- add data sourcesを押下
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/573632/8e458339-6fee-82b7-73f5-d739807cb586.png)
- 検索ボックスにS3を入力し、Amazon S3 connectorを選択
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/573632/59c2ef6b-707e-748f-af98-e1ecd7ab5642.png)
- 名前を入力
- Default language of source documents を日本語(Japanese)に設定
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/573632/d5bc67f0-9570-f19b-a8cd-6aceb98e7a3b.png)
![image.png](https://qiita-image-store.s3.ap-northeast-
1.amazonaws.com/0/573632/b81c5606-bb5c-fcef-9eba-eea1585eb211.png)
- Create New Roleを選択
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/573632/4767978c-e7a5-76e9-80b2-74e6c94eb0a3.png)
- バケットを指定し、S3バケットのプレフィックスを設定
- Sync のスケジュールを Run on demand を設定
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/573632/fd32d4f2-a9bc-4d88-80ee-143fdc295f99.png)
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/573632/7f4fcba1-7aa0-09b3-829e-3dbc83687b0b.png)
- Nextを押下
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/573632/eeaa0462-2103-6377-e001-00b0de66e203.png)
- 内容を確認してadd data source を押下
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/573632/2456f39b-4628-bbe4-56b0-d7cc8b8c3406.png)
- できた
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/573632/326a66e9-b5e5-0928-05ac-6e62d28eeede.png)
- Sync Nowを押下
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/573632/95b67fbe-0edb-1b40-a112-5ac5a973e790.png)
- Syncにめっちゃ時間がかかる
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/573632/93ba51a3-b239-242d-8c22-f7f31ea425a5.png)
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/573632/98e17c56-d066-00b9-06dc-dd8c82ed5297.png)


# 検索してみる
- Search Indexed Content をクリック
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/573632/09d42be9-67e4-f61c-3248-e15d5fdf9fb9.png)
- 画面右のスパナアイコンをクリックし、言語を日本語に設定
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/573632/f1204722-c918-1381-228d-b932f892f557.png)
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/573632/cf93365f-3904-ff19-bd28-c6cc39dcfeb3.png)

- 検索できた
![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/573632/f47b729e-16f4-46ba-9160-4f10b735eddf.png)


# 感想
- 意外とセマンティック検索にならないので、datasourceに配置するドキュメントの作りこみに工夫が必要かもしれない
- セマンティック検索にはまらない場合は単純にRAGの検索先としてはOpenSearchとかでいいと思う
- S3を簡単にdatasourceにできるのが圧倒的な強み
- ただし、高コスト

以上