movielens recommendation with VertexAI Search for Retail

## このページは

以下のチュートリアルをやってみてmovielensのデータでVertexAI Search for Retailを試してみました。

https://cloud.google.com/retail/docs/movie-rec-tutorial

## 実行環境

Cloud Shellがとても便利

## Dataの準備

```
gcloud storage buckets create gs://$PROJECT_ID-movielens-data
```

```
gcloud storage cp ml-latest/movies.csv ml-latest/ratings.csv   gs://$PROJECT_ID-movielens-data
```

## BigQueryの準備

### dataset `movielens`作成

```
bq mk $PROJECT_ID:movielens
```

### csvファイルから`movies`テーブルにload

```
bq load --skip_leading_rows=1 $PROJECT_ID:movielens.movies \
  gs://$PROJECT_ID-movielens-data/movies.csv \
  movieId:integer,title,genres
```

### csvファイルから`ratings`テーブルにload

```
bq load --skip_leading_rows=1 $PROJECT_ID:movielens.ratings \
  gs://$PROJECT_ID-movielens-data/ratings.csv \
  userId:integer,movieId:integer,rating:float,time:timestamp
```

### `movies` テーブルを productのカタログに合わせたviewを作成する

```
bq mk --project_id=$PROJECT_ID \
 --use_legacy_sql=false \
 --view "
 SELECT
   CAST(movieId AS string) AS id,
   SUBSTR(title, 0, 128) AS title,
   SPLIT(genres, \"|\") AS categories
 FROM \`$PROJECT_ID.movielens.movies\`" \
$PROJECT_ID:movielens.products
```

- id: movie_id
- title
- categories

### `ratings` を user_eventsに合わせたviewを作成する

:::note warn
- Rescale the Movielens timeline into the last 90 days. We do this for two reasons:
    - Vertex AI Search for retail requires that user events are no older than 2015. Movielens ratings go back to 1995.
    - Vertex AI Search for retail uses the last 90 days of user events when serving prediction requests for a user. Every user will appear to have recent events when we make predictions for any user later on.
:::


```
bq mk --project_id=$PROJECT_ID \
 --use_legacy_sql=false \
 --view "
 WITH t AS (
   SELECT
     MIN(UNIX_SECONDS(time)) AS old_start,
     MAX(UNIX_SECONDS(time)) AS old_end,
     UNIX_SECONDS(TIMESTAMP_SUB(
       CURRENT_TIMESTAMP(), INTERVAL 90 DAY)) AS new_start,
     UNIX_SECONDS(CURRENT_TIMESTAMP()) AS new_end
   FROM \`$PROJECT_ID.movielens.ratings\`)
 SELECT
   CAST(userId AS STRING) AS visitorId,
   \"detail-page-view\" AS eventType,
   FORMAT_TIMESTAMP(
     \"%Y-%m-%dT%X%Ez\",
     TIMESTAMP_SECONDS(CAST(
       (t.new_start + (UNIX_SECONDS(time) - t.old_start) *
         (t.new_end - t.new_start) / (t.old_end - t.old_start))
     AS int64))) AS eventTime,
   [STRUCT(STRUCT(movieId AS id) AS product)] AS productDetails,
 FROM \`$PROJECT_ID.movielens.ratings\`, t
 WHERE rating >= 4" \
$PROJECT_ID:movielens.user_events
```

movieを見て4以上の評価をしているものを `detail-page-view`として扱う

- visitorId: userId
- eventType: `detail-page-view`
- eventTime
- productDetails: {id: movie_id}


## データのインポート

### product catalogのインポート (5~10min)

```
gcloud services enable retail.googleapis.com --project $PROJECT_ID
```

gcloudがないのか画面からimport https://console.cloud.google.com/ai/retail/catalogs/default_catalog/data/catalog

:::note warn
Advanced OptionsでBucketを指定しないと新しいGCS Bucketが作成されてしまうので注意
:::


![Screenshot 2024-08-24 at 20.34.41.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/7059/3f8dc4b5-3b18-1bdd-36e9-2527dfd18698.png)

importが成功すると以下のコマンドが出てきたので、スケジュールでRetail catalog dataをimportすることもできるよう。

```
gcloud scheduler --project xxxx \
jobs create http import_catalog_xxxx \
--time-zone='America/Los_Angeles' \
--schedule='0 0 * * *' \
--uri='https://retail.googleapis.com/v2alpha/projects/xxxxx/locations/global/catalogs/default_catalog/branches/0/products:import' \
--description='Import Retail catalog data' \
--headers='Content-Type: application/json; charset=utf-8' \
--http-method='POST' \
--message-body='{"inputConfig":{"bigQuerySource":{"projectId":"xxxx","datasetId":"movielens","tableId":"products","dataSchema":"product"}},"reconciliationMode":"INCREMENTAL"}' \
--oauth-service-account-email=''
```

インポートのOperationの状態を見ることができる

![Screenshot 2024-08-24 at 20.46.03.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/7059/079eb4c0-4a4a-211f-d72e-cdcb737816c5.png)


インポートが完了すると以下のように見れるようになる

![Screenshot 2024-08-24 at 20.40.56.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/7059/09d4b997-9944-b07b-9894-22a9397c7169.png)

### user eventsをインポート(1h)

同様に User EventsもImportする

:::note warn
- Advanced OptionsでBucketを指定しないと新しいGCS Bucketが作成されてしまうので注意
:::

![Screenshot 2024-08-24 at 20.42.22.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/7059/ce7fde0d-c561-d110-572d-513ae11858b9.png)

インポートの状態を確認できる

![Screenshot 2024-08-24 at 21.10.35.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/7059/ad980ea8-473c-047d-499a-c5e5d44f9fb2.png)

User Eventデータの確認

![Screenshot 2024-08-24 at 21.11.51.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/7059/d218e0bf-2f0b-3249-1f52-529a8a62011b.png)


## Recommendationモデルのトレーニング (2 days)

モデルタイプとBusiness Objectiveなどを選択してモデルをトレーニングできる

モデルタイプ:
- Recommended for you: home pageのタイムライン向き
- Others you may like: Detailページで関連Item向き
- Frequently bought together: Addーtoーcartのすぐ後やDetailページで表示する
- Similar items: detailページ
- Buy it again: purchase historyからもう一度購入されそうなItemをお勧めする detailed page, add to cart, shopping cart, category views, home pageなどに使われる
- Page-level optimization: Automatically optimizes the entire page and catalog item recommendations with multiple recommendation panels <- どういうOptimizeになるのかあまり理解できていない
- On sale: On saleのProductをお勧めする



![Screenshot 2024-08-24 at 21.18.29.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/7059/a51b76ca-23b2-a89a-42ba-80371e04ec88.png)

モデルリスト

![Screenshot 2024-08-24 at 21.20.10.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/7059/14bdd2ea-1298-3a42-9c2a-1a3f938d71fa.png)

:::note warn
トレーニングに2日くらいかかるので待つ。なんでこんなにかかるのかは不明。
![Screenshot 2024-08-27 at 8.11.26.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/7059/cfc6eb1e-53ed-ab0d-7758-47bf54873e70.png)

:::


![Screenshot 2024-08-27 at 8.09.57.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/7059/f31555de-22ff-6108-c0e4-cc3a22b4348e.png)

## Serving Config設定

`Create Serving Config`のページから `Serving Config` を作成していく

1. トレーニングしたModelを選択
1. Price reranking, Result diversificationなどを選択する
    ![Screenshot 2024-08-27 at 8.14.15.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/7059/50921b7f-25a6-1032-35cb-50c1aa169b94.png)
1. Serving Controlは今回は何も選択肢しない

Serving Configの作成が完了 ✅

![Screenshot 2024-08-27 at 8.15.58.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/7059/95603221-a92d-1e24-e7d8-eb7eff2920c7.png)

`Model ready to query`がyesとなっているのを確認

![Screenshot 2024-08-27 at 8.16.48.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/7059/7aa2a31e-055c-96fd-a68f-4982210c6225.png)

## Predict (Evaluate)

Evaluateタブから、
- Visitor IdはOptional
- Product IDは、今回のケースではMovie IDを入れる `4993` (`"The Lord of the Rings: The Fellowship of the Ring (2001)"`)

![Screenshot 2024-08-27 at 8.19.45.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/7059/6252465d-2d38-19e5-d766-3b33d8eab685.png)

Prediction Resultが右側に出る :tada: 

![Screenshot 2024-08-27 at 8.21.12.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/7059/ac2c9f0c-9ab3-8ad6-b29c-327c1dc0bafc.png)


## Ref

- [Create personalized movie recommendations](https://cloud.google.com/retail/docs/movie-rec-tutorial) <- 今回行ったTutorial
- [Implement Vertex AI Search for retail ](https://cloud.google.com/retail/docs/overview)
- [Record Real-Time User Events](https://cloud.google.com/retail/docs/record-events)
- [Import Historical User Events](https://cloud.google.com/retail/docs/import-user-events)
- https://qiita.com/nakamasato/private/295a2a3a4d5ed2aab2d5