opened
urbien
username_0
## Purpose
We have the following cases for using Hypercore with AWS S3 (and its many open source re-implementations, like https://min.io):

1. Eleven nines of durability for Hypercore feeds
2. Streaming the database from s3, this idea excited me when I read it in Andrews blog post on changing to SLEEP format in 2017. 
3. Load-balancing with multiple Hypercore replicas

## Starting point in Hypercore and what is lacking:

[hypercore-archiver](https://awesome.datproject.org/hypercore-archiver) with [random-access-s3](https://github.com/random-access-storage/random-access-s3) provide the basic method for piping a feed to S3. But random-access-s3 write method is not implemented 

## Context
While S3 supports random reads, that is a read from the offset in S3 object, there is no support for random writes:

1. S3 does not support updating file at an offset (Google S3 PUT range request)
2. S3 object update is inefficient, you need to PUT the whole file again
3. S3 Multi-part upload allows multiple chunks, which are assembled by S3 automatically into an S3 object. There is no way to update chunks
 
## Proposed approach
May be we can learn from the Search engine Lucene,. To avoid updating the whole index on every document add / update update (which is extremely costly), Lucene  writes new data into a chunk, which it calls index segment. On search it reads from all segments and merges the results. Here is how we can mimic this in Hypercore backup to S3, which has similar performance bound for our case:

1. Create a separate feed for the new segment / chunk.  
2. Dump to s3 the whole feed.
3. Merge all feeds later into one feed and dump this to S3. 
4. Before that merge, serve all peer requests to our feed from virtually merged feeds, using mounts.
S3 for backup / restore and database streaming
issue
created
pgmemk
username_1
How segmentation is going to work for editing?
comment
created
urbien
username_0
@username_1 good catch, I did not think of that. Now, here is an an idea how to address updates in data items that were already archived / backed up to S3. Maybe we could use union mounts, which are upcoming in Hypercore. Hyperdrive will use the mounts for shared folders. Union mount creates an impression that you can update friend's Hyperdrive that is today mounted read-only. Union mounts were invented by Plan9 and them over a decade were implemented and re-implemented a ton of times in Linux. They are now stable for a long time in Linux and are used for booting from an immutable source, like DVD, but still being able to customize your installation, and boot again with all your changes to immutable files saved separatly from the original DVD.
comment
created
urbien
username_0
Initial idea was to use mounts for a virtual merge. But Hyperbee does not have mounts. And Hypertrie mounts change names of the keys. Key 'a' becomes 'mountpoint/a'. 
New ideas are needed.
comment
created
urbien
username_0
perhaps this module could help?
https://github.com/little-core-labs/hypercore-multipart
comment