S3: ListBucket/ListObject(s)/DeleteObjects/PutObject コマンド

# 背景
S3 にファイルアップロードが必要になったので、調べた記録

# 利用例

## 前提
```typescript: client
import { GetObjectCommand, PutObjectCommand, ListBucketsCommand, ListObjectsCommand, ListObjectsV2Command, S3Client } from "@aws-sdk/client-s3";

const REGION = "ap-northeast-1";
const clientConfig = {
    region: REGION,
    // for Proxy
    // requestHandler: new NodeHttpHandler({
    //     httpAgent: httpsAgent,
    //     httpsAgent: httpsAgent
    // })
}
const s3Client = new S3Client(clientConfig);
const BucketName = "testBucket";
```

## ListBucketCommand
```typescript: ListBucket
const ListBucketsCommandInput = {
}
const resultListBucketsCommand = await s3Client.send(new ListBucketsCommand(ListBucketsCommandInput));
console.log("listBuckets", resultListBucketsCommand);
```

https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/classes/listbucketscommand.html


## ListObjectCommand / V2

- Prefix
    - ァイルやフォルダの取得制限がかけられるので、拡張子違いやコピーファイルなんかを一括取得。
- MaxKeys
    - 取得制限。（Default は 1000）
- ContinuationToken
    - 残データがあれば、Output.NextContinuationToken が設定されてくるので、それを設定してやれば、継続取得
- Delimiter
    - Prefix ～ Delimiter の間に該当するものを、Output.CommonPrefixes にて返してくれる。"/" で切れば、対象フォルダ内の一覧が取得出来る。<br>Prefix: "images/", Delimiter: "n" の例）
    > CommonPrefixes: [
    { Prefix: '**images**/2022-11-29_21h44_50.p**n**' },
    { Prefix: '**images**/paralells/pokemo**n**' },
    { Prefix: '**images**/paralleled_UsersIORIDow**n**' }
  ],

```typescript: ListObject
const ListObjectsV2CommandInput = {
    Bucket: BucketName,
    Prefix: "images/para",
    MaxKeys: 1,
    ContinuationToken: '1YoMH5hobuD8EitPmXzpa/8QneE8RDfziBnSTlsn1ll1pXEBAbDnzKmC60E3PkCddxgZ25VC6BKMAYiEtLnsLXfgEXzs5Mk+ikTtHV/wMrKA=',
}

// V2
// - 連続して取得する際に便利？ KeyCount や NextContinuationToken がある
const resultListObjectsV2Command = await s3Client.send(new ListObjectsV2Command(ListObjectsV2CommandInput));
console.log("listObjV2", resultListObjectsV2Command);
```

ここ見ると、start-after が便利だそうな

https://dev.classmethod.jp/articles/s3-new-api-list-object-v2/

https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/classes/listobjectscommand.html

https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/classes/listobjectsv2command.html

### 該当フォルダからの一括取得例
```typescript: 該当フォルダから一括取得
Prefix: "images/",
Delimiter: "/",
```

## DeleteObjectsCommand
一括削除。一点削除なら、DeleteObjectCommand を

- Objects
    - Key/VersionId を列挙していく。存在してない Key を列挙した場合、削除した扱いになる。

```typescript: 利用例
import { DeleteObjectsCommand, GetObjectCommand, PutObjectCommand, ListBucketsCommand, ListObjectsCommand, ListObjectsV2Command, S3Client, S3 } from "@aws-sdk/client-s3";

const deleteObjectsCommandInput = {
    Bucket: BucketName,
    Delete: {
        Objects: [
            {
                Key: "images/paralleled_AWSCLIV2.msi",
                // VersionId: "??",
            },
            {
                Key: "images/paralleled_AWSCLIV2_copied.msi",
                // VersionId: "??",
            },
            {
                Key: "images/paralleled_AWSCLIV2(2).msi",
                // VersionId: "??",
            },
        ],
    },
};
const resultDeleteObjectsCommand = await s3Client.send(new DeleteObjectsCommand(deleteObjectsCommandInput));
console.log("delete", resultDeleteObjectsCommand);
```

https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/interfaces/deleteobjectscommandinput.html

## PutObjectCommand
Key でアップロードパス（フォルダ名 / ファイル名）を指定。

::: note info
フォルダが無くても作ってくれる。
:::

```typescript: Put
import { default as path } from "path";
import { default as fs } from "fs";

const file = "%appdata%/Screenpresso/2022-11-29_21h44_50.png";
const fileStream = fs.createReadStream(file);

const putObjectCommandInput = {
    Bucket: BucketName,
    Key: `images/${path.basename(file)}`,
    Body: fileStream,
};

const putObjectCommandOutput = await s3Client.send(new PutObjectCommand(putObjectCommandInput));
console.log("PutObject", putObjectCommandOutput);
```

https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/classes/putobjectcommand.html

## Parallel Upload in lib-storage
s3/dynamo のみにある lib を使えば簡単に parallel upload も出来る

```typescript: parallel
import { default as path } from "path";
import { default as fs } from "fs";
import { Upload } from "@aws-sdk/lib-storage";

const file2 = "%appdata%/Downloads/AWSCLIV2.msi";
const fileStream2 = fs.createReadStream(file2);
const parallelUploads3 = new Upload({
    client: new S3({}) || new S3Client({}),
    params: {
        Bucket: BucketName,
        Key: `images/paralleled_${path.basename(file2)}`,
        Body: fileStream2,
    },

    tags: [
        /*...*/
    ], // optional tags
    queueSize: 4, // optional concurrency configuration
    partSize: 1024 * 1024 * 5, // optional size of each part, in bytes, at least 5MB
    leavePartsOnError: false, // optional manually handle dropped parts
});

parallelUploads3.on("httpUploadProgress", (progress) => {
    console.log(progress);
});

await parallelUploads3.done();
```

https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/modules/_aws_sdk_lib_storage.html

### blob を利用する場合の upload
:::note warn
PutObjectCommand を使うと、生成時にデータサイズが未定の為か、以下エラーが出るので注意

Body: blob の場合
> TypeError [ERR_INVALID_ARG_TYPE]: The first argument must be of type string or an instance of Buffer, 
ArrayBuffer, or Array or an Array-like Object. Received an instance of Blob

Body: blob.stream() の場合: Size 未定の為に出てるような気がするが詳細不明
> Error NotImplemented: A header you provided implies functionality that is not implemented
:::
```typescript: upload
const localUri = await fetch("https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/userguide/s3-userguide.pdf");
const localBlob = await localUri.blob();
const onProgress = (progress: number)=> {  // progress 表示用
    console.log(`${Number(progress.toFixed(2))} % is done...`);
}
await uploadImage(localBlob, `uploadings/upload_${Date.now()}.pdf`, onProgress);

async uploadImage(blob: Blob, filename: string, onStateChanged: any): Promise<string> {
    const uid = await this.getUserId();
    // https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/modules/_aws_sdk_lib_storage.html
    const parallelUploads3 = new Upload({
      client: s3Client,
      params: {
        Bucket: BucketName,
        Key: filename,
        Body: blob,
      },
      tags: [
        /*...*/
      ], // optional tags
      queueSize: 4, // optional concurrency configuration
      partSize: 1024 * 1024 * 5, // optional size of each part, in bytes, at least 5MB
      leavePartsOnError: false, // optional manually handle dropped parts
    });

    parallelUploads3.on("httpUploadProgress", (progress) => {
      let bytesTransferred = progress.loaded ?? 1;  // todo: 0 にすると、0/0 がありうるので、1 としておく
      let totalBytes = progress.total ?? bytesTransferred;
      onProgress((bytesTransferred / totalBytes) * 100);
    });

    const result = await parallelUploads3.done();
    return (result as CompleteMultipartUploadCommandOutput)?.Location ?? "";
  }
```


## GetObjectCommand

```typescript: Get
const getObjectCommandInput = {
    Bucket: BucketName,
    Key: "testFolder/lib.ts",
};
const result = await s3Client.send(new GetObjectCommand(getObjectCommandInput));
console.log("getObject", result);
```

https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/classes/getobjectcommand.html

## ListObjectVersionCommand
バージョン管理を有効にした場合

https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/classes/listobjectversionscommand.html

https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/classes/getbucketversioningcommand.html

https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/classes/putbucketversioningcommand.html

# 追加考慮事項
## マルチパートアップロード
> 通常、オブジェクトサイズが 100 MB 以上の場合は、単一のオペレーションでオブジェクトをアップロードする代わりに、マルチパートアップロードを使用することを考慮してください。

ってことで、スループットの向上や Resume のことを考えて、サイズ次第では検討必要

[上で実装済み](#parallel-upload-in-lib-storage)

https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/userguide/mpuoverview.html

## 整合性チェックや暗号化

https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/userguide/checking-object-integrity.html

https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/userguide/UsingEncryption.html

## アーカイブ
StorageClass の検討。[S3 Intelligent-Tiering](https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/userguide/storage-class-intro.html#sc-dynamic-data-access) のような自動的にアーカイブしてくれるようなものを利用する手もある。[自動化の料金](https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/userguide/storage-class-intro.html#sc-compare)は必要

https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/userguide/storage-class-intro.html

## Versioning
https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/userguide/AddingObjectstoVersioningEnabledBuckets.html

## ACL
https://docs.aws.amazon.com/ja_jp/AmazonS3/latest/userguide/about-object-ownership.html

# あとがき
とりあえず、一覧取得して、アイテムのアップロード＆ダウンロードは出来そう

SDK V3 自体の使い方が分かると、ある程度は何とかなるね。
初心に戻って、調べ方などをメモしておきたいところかな

セキュリティも考えながら完璧に行わなきゃいけないとなるとほんと大変そう。