npm package: @socialproof/file-storage
Description: File Storage SDK
Author: Mysten Labs
Latest version: 0.0.4

# `@socialproof/file-storage`

## Installation

```bash
npm install --save @socialproof/file-storage @socialproof/myso
```

## Setup

To use the file-storage SDK you will need to create a Client from the typescript SDK, and extend it with
the file-storage SDK.

```ts
import { MySoGrpcClient } from '@socialproof/myso/grpc';
import { fileStorage } from '@socialproof/file-storage';

const client = new MySoGrpcClient({
	network: 'testnet',
	baseUrl: 'https://fullnode.testnet.mysocial.network:443',
}).$extend(fileStorage());
```

The file-storage SDK currently includes all the relevant package and object IDs needed for connecting to
testnet. You can also manually configure the file-storage sdk to use a different set of ids, allowing you
to connect to a different network or updated deployment of the file-storage contracts.

```ts
import { MySoGrpcClient } from '@socialproof/myso/grpc';
import { fileStorage } from '@socialproof/file-storage';

const client = new MySoGrpcClient({
	network: 'testnet',
	baseUrl: 'https://fullnode.testnet.mysocial.network:443',
}).$extend(
	fileStorage({
		packageConfig: {
			systemObjectId: '0x98ebc47370603fe81d9e15491b2f1443d619d1dab720d586e429ed233e1255c1',
			stakingPoolId: '0x20266a17b4f1a216727f3eef5772f8d486a9e3b5e319af80a5b75809c035561d',
		},
	}),
);
```

For some environments you may need to customize how data is fetched:

```ts
import { MySoGrpcClient } from '@socialproof/myso/grpc';
import { fileStorage } from '@socialproof/file-storage';

const client = new MySoGrpcClient({
	network: 'testnet',
	baseUrl: 'https://fullnode.testnet.mysocial.network:443',
}).$extend(
	fileStorage({
		storageNodeClientOptions: {
			fetch: (url, options) => {
				console.log('fetching', url);
				return fetch(url, options);
			},
			timeout: 60_000,
		},
	}),
);
```

This can be used to implement a fetch function with custom timeouts, rate limits, retry logic, or
any other desired behavior.

## SDK Overview

The File Storage TS SDK is designed to work directly with file-storage storage nodes, or with the file-storage upload
relay. When using the file-storage SDK without an upload relay, it is important to understand that reading
and writing file-storage blobs requires a lot of requests (~2200 to write a blob, ~335 to read a blob).
The upload relay will reduce the number of requests needed to write a blob, but reads through the
file-storage SDK will still require a lot of requests. For many applications, using publishers and
aggregators is recommended, but the TS SDK can be useful when building applications where the
application needs to directly interact with file-storage or users need to pay for their own storage
directly.

The file-storage sdk exposes high level methods for reading and writing blobs, as well as lower level
methods for the individual steps in the process that can be used to implement more complex flows
when you want more control to implement more optimized implementations.

## FileStorageFile

The `FileStorageFile` API provides a higher level abstraction so that applications don't need to worry
about how data is stored in file-storage. Today it handles data stored directly in blobs, and data stored
in Quilts, but may be expanded to cover other storage patterns in the future.

### Reading files

To read files, you can use the `getFiles` method. This method accepts both Blob IDs and Quilt IDs,
and will return a `FileStorageFile`.

It is encouraged to always read files in batches when possible, which will allow the client to be
more efficient when loading multiple files from the same quilt.

```ts
const [file1, file2] = await client.fileStorage.getFiles({ ids: [anyBlobId, orQuiltId] });
```

A `FileStorageFile` works like a `Response` object from the `fetch` API:

```ts
// get contents as a Uint8Array
const bytes = await file1.bytes();
// Parse the contents as a `utf-8` string
const text = await file1.text();
// Parse the contents as JSON
const json = await file2.json();
```

A `FileStorageFile` may also have and `identifier` and `tags` properties if the file was stored in a
quilt.

```ts
const identifier: string | null = await file1.getIdentifier();
const tags: Record<string, string> = await file1.getTags();
```

### FileStorageBlob

You can also get a `FileStorageBlob` instead of a `FileStorageFile` if you have the blobId:

```ts
const blob = await client.fileStorage.getBlob({ blobId });
```

If the blob is a quilt, you can read the files in the quilt:

```ts
// Get all files:
const files = await blob.files();
// Get files by identifier
const [readme] = await blob.files({ identifiers: ['README.md'] });
// Get files by tag
const textFiles: FileStorageFile[] = await blob.files({ tags: [{ 'content-type': 'text/plain' }] });
// Get files by quilt id
const filesById = await blob.files({ ids: [quiltID] });
```

### Writing files

You can also construct a `FileStorageFile` from a `Uint8Array`, `Blob`, or a `string` which can then be
stored on file-storage:

```ts
const file1 = FileStorageFile.from({
	contents: new Uint8Array([1, 2, 3]),
	identifier: 'file1.bin',
});
const file2 = FileStorageFile.from({
	contents: new Blob([new Uint8Array([1, 2, 3])]),
	identifier: 'file2.bin',
});
const file3 = FileStorageFile.from({
	contents: new TextEncoder().encode('Hello from the TS SDK!!!\n'),
	identifier: 'README.md',
	tags: {
		'content-type': 'text/plain',
	},
});
```

Once you have your files you can use the `writeFiles` method to write them to file-storage.

Along with the files, you will also need to provide a `Signer` instance that signs and pays for the
transaction/storage fees. The signer's address will need to have sufficient `MYSO` to cover the
transactions that register the blob, and certify its availability after it's been uploaded. The
Signer must own sufficient `WAL` to pay to store the blob for the specified number of epochs, as
well as the write fee for writing the blob.

The exact costs will depend on the size of the blobs, as well as the current gas and storage prices.

```ts
const results: {
	id: string;
	blobId: string;
	blobObject: Blob.$inferType;
}[] = client.fileStorage.writeFiles({
	files: [file1, file2, file3],
	epochs: 3,
	deletable: true,
	signer: keypair,
});
```

Currently the provided files will all be written into a single quilt. Future versions of the SDK may
optimize how files are stored to be more efficient by splitting files into multiple quilts.

The current quilt encoding is less efficient for single files, so writing multiple files together is
recommended when possible. Writing raw blobs directly is also possible using the `writeBlob` API
(described below)

### Writing files in browser environments

When the transactions to upload a blob are signed by a wallet in a browser, some wallets will use
popups to prompt the user for a signature. If the popups are not opened in direct response to a user
interaction, they may be blocked by the browser.

To avoid this, we need to ensure that we execute the transactions that register and certify the blob
in separate events handlers by creating separate buttons for the user to click for each step.

The `client.writeFilesFlow` method returns an object with a set of methods that break the write flow
into several smaller steps:

1. `encode` - Encodes the files and generates a blobId
2. `register` - returns a transaction that will register the blob on-chain
3. `upload` - Uploads the data to storage nodes
4. `certify` - returns a transaction that will certify the blob on-chain
5. `listFiles` - returns a list of the created files

Here's a simplified example showing the core API usage with separate user interactions:

```tsx
// Step 1: Create and encode the flow (can be done immediately when file is selected)
const flow = client.fileStorage.writeFilesFlow({
	files: [
		FileStorageFile.from({
			contents: new Uint8Array(fileData),
			identifier: 'my-file.txt',
		}),
	],
});

await flow.encode();

// Step 2: Register the blob (triggered by user clicking a register button after the encode step)
async function handleRegister() {
	const registerTx = flow.register({
		epochs: 3,
		owner: currentAccount.address,
		deletable: true,
	});
	const result = await signAndExecuteTransaction({ transaction: registerTx });

	// Check transaction status
	if (result.$kind === 'FailedTransaction') {
		throw new Error(`Registration failed: ${result.FailedTransaction.status.error?.message}`);
	}

	// Step 3: Upload the data to storage nodes
	// This can be done immediately after the register step, or as a separate step the user initiates
	await flow.upload({ digest: result.Transaction.digest });
}

// Step 4: Certify the blob (triggered by user clicking a certify button after the blob is uploaded)
async function handleCertify() {
	const certifyTx = flow.certify();

	const result = await signAndExecuteTransaction({ transaction: certifyTx });

	// Check transaction status
	if (result.$kind === 'FailedTransaction') {
		throw new Error(`Certification failed: ${result.FailedTransaction.status.error?.message}`);
	}

	// Step 5: Get the new files
	const files = await flow.listFiles();
	console.log('Uploaded files', files);
}
```

This approach ensures that each transaction signing step is separated into different user
interactions, allowing wallet popups to work properly without being blocked by the browser.

## Writing blobs with an upload relay

Writing blobs directly from a client requires a lot of requests to write data to all of the storage
nodes. An upload relay can be used to offload the work of these writes to a server, reducing
complexity for the client.

To use an upload relay, you can add the `uploadRelay` option when adding the file-storage extension:

```ts
const client = new MySoGrpcClient({
	network: 'testnet',
	baseUrl: 'https://fullnode.testnet.mysocial.network:443',
}).$extend(
	fileStorage({
		uploadRelay: {
			host: 'https://upload-relay.testnet.file-storage.space',
			sendTip: {
				max: 1_000,
			},
		},
	}),
);
```

The `host` option is required, and indicates the url for your upload relay. Upload relays may
require a tip to be included to cover the cost of writing the blob. You can configure a maximum tip
(paid in MIST) and the `FileStorageClient` will automatically determine the required tip for your upload
relay, or you can manually configure the tip configuration as shown below.

The tip required by an upload relay can be found using the `tip-config` endpoint: (eg.
`https://upload-relay.testnet.file-storage.space/v1/tip-config`) and may either be a `const` or a `linear`
tip.

### `const` tip

A `const` will send a fixed amount for each blob written to the upload relay.

```ts
const client = new MySoGrpcClient({
	network: 'testnet',
	baseUrl: 'https://fullnode.testnet.mysocial.network:443',
}).$extend(
	fileStorage({
		uploadRelay: {
			host: 'https://upload-relay.testnet.file-storage.space',
			sendTip: {
				address: '0x123...',
				kind: {
					const: 105,
				},
			},
		},
	}),
);
```

### `linear` tip

A `linear` tip will send a fixed amount for each blob written to the fan out proxy, plus a
multiplier based on the size of the blob.

```ts
const client = new MySoGrpcClient({
	network: 'testnet',
	baseUrl: 'https://fullnode.testnet.mysocial.network:443',
}).$extend(
	fileStorage({
		uploadRelay: {
			host: 'https://upload-relay.testnet.file-storage.space',
			sendTip: {
				address: '0x123...',
				kind: {
					linear: {
						base: 105,
						perEncodedKib: 10,
					},
				},
			},
		},
	}),
);
```

## Interacting with blobs directly

In case you do not want to use the `FileStorageFile` abstractions, you can use the `readBlob` and
`writeBlob` APIs directly

### Reading blobs

The `readBlob` method will read a blob given the blobId and return Uint8Array containing the blobs
content:

```ts
const blob = await client.fileStorage.readBlob({ blobId });
```

### Writing Blobs

Thw writeBlob method can be used to write a blob (as a `Uint8Array`) to file-storage. You will need to
specify how long the blob should be stored for, and if the blob should be deletable.

```ts
const file = new TextEncoder().encode('Hello from the TS SDK!!!\n');

const { blobId } = await client.fileStorage.writeBlob({
	blob: file,
	deletable: false,
	epochs: 3,
	signer: keypair,
});
```

## Full API

For a complete overview of the available methods on the `FileStorageClient` you can reference type
[TypeDocs](http://sdk.mystenlabs.com/typedoc/classes/_mysten_file-storage.FileStorageClient.html)

## Examples

There are a number of simple
[examples you can reference](https://github.com/MystenLabs/ts-sdks/tree/main/packages/file-storage/examples)
in the `ts-sdks` repo that show things like building simple aggregators and publishers with the
file-storage SDK

## Error handling

The SDK exports all the error classes for different types of errors that can be thrown. File Storage is a
fault tolerant distributed system, where many types of errors can be recovered from. During epoch
changes there may be times when the data cached in the `FileStorageClient` can become invalid. Errors
that result from this situation will extend the `RetryableFileStorageClientError` class.

You can check for these errors, and reset the client before retrying:

```ts
import { RetryableFileStorageClientError } from '@socialproof/file-storage';

if (error instanceof RetryableFileStorageClientError) {
	client.fileStorage.reset();

	/* retry your operation */
}
```

`RetryableFileStorageClientError` are not guaranteed to succeed after resetting the client and retrying,
but this pattern can be used to handle some edge cases.

High level methods like `readBlob` already handle various error cases and will automatically retry
when hitting these methods, as well as handling cases where only a subset of nodes need to respond
successfully to read or publish a blob.

When using the lower level methods to build your own read or publish flows, it is recommended to
understand the number of shards/sliver that need to be successfully written or read for you
operation to succeed, and gracefully handle cases where some nodes may be in a bad state.

### Network errors

File Storage is designed to be handle some nodes being down, and the SDK will only throw errors when it
can't read from or write to enough storage nodes. When trying to troubleshoot problems, it can be
challenging to figure out whats going wrong when you don't see all the individual network errors.

You can pass an `onError` option in the storageNodeClientOptions to get the individual errors from
failed requests:

```ts
const client = new MySoGrpcClient({
	network: 'testnet',
	baseUrl: 'https://fullnode.testnet.mysocial.network:443',
}).$extend(
	fileStorage({
		storageNodeClientOptions: {
			onError: (error) => console.log(error),
		},
	}),
);
```

## Configuring network requests

Reading and writing blobs directly from storage nodes requires a lot of requests. The file-storage SDK
will issue all requests needed to complete these operations, but does not handling all the
complexities a robust aggregator or publisher might encounter.

By default all requests are issued using the global `fetch` for whatever runtime the SDK is running
in.

This will not impose any limitations on concurrency, and will be subject to default timeouts and
behavior defined by your runtime. To customize how requests are made, you can provide a custom
`fetch` method:

```ts
import type { RequestInfo, RequestInit } from 'undici';
import { Agent, fetch, setGlobalDispatcher } from 'undici';

const client = new MySoGrpcClient({
	network: 'testnet',
	baseUrl: 'https://fullnode.testnet.mysocial.network:443',
}).$extend(
	fileStorage({
		storageNodeClientOptions: {
			timeout: 60_000,
			fetch: (url, init) => {
				// Some casting may be required because undici types may not exactly match the @node/types types
				return fetch(url as RequestInfo, {
					...(init as RequestInit),
					dispatcher: new Agent({
						connectTimeout: 60_000,
					}),
				}) as unknown as Promise<Response>;
			},
		},
	}),
);
```

## Loading the wasm module in vite or client side apps

The file-storage SDK requires wasm bindings to encode and decode blobs. When running in node or bun and
some bundlers this will work without any additional configuration.

In some cases you may need to manually specify where the SDK loads the wasm bindings from.

In vite you can get the url for the wasm bindings by importing the wasm file with a `?url` suffix,
and then passed into the file-storage client:

```ts
import fileStorageWasmUrl from '@socialproof/file-storage-wasm/web/file_storage_wasm_bg.wasm?url';

const client = new MySoGrpcClient({
	network: 'testnet',
	baseUrl: 'https://fullnode.testnet.mysocial.network:443',
}).$extend(
	fileStorage({
		wasmUrl: fileStorageWasmUrl,
	}),
);
```

If you are unable to get a url for the wasm file in your bundler or build system, you can self host
the wasm bindings, or load them from a CDN:

```ts
const client = new MySoGrpcClient({
	network: 'testnet',
	baseUrl: 'https://fullnode.testnet.mysocial.network:443',
}).$extend(
	fileStorage({
		wasmUrl: 'https://unpkg.com/@socialproof/file-storage-wasm@latest/web/file_storage_wasm_bg.wasm',
	}),
);
```

In next.js when using file-storage in API routes, you may need to tell next.js to skip bundling for the
file-storage packages:

```ts
// next.config.ts
const nextConfig: NextConfig = {
	serverExternalPackages: ['@socialproof/file-storage', '@socialproof/file-storage-wasm'],
};
```

## Known fetch limitations you might run into

- Some nodes can be slow to respond. When running in node, the default connectTimeout is 10 seconds
  and can cause request timeouts
- In `bun` the `abort` signal will stop requests from responding, but they still wait for completion
  before their promises reject