opened
ZenGround0
username_0
### Description
We have gotten reports from @username_1 and others that the datastore is much bigger than expected.  The expectation (I believe) is that chain data dominates the total amount of (non-sector) data stored by the filecoin node.  However at present other data with size an order of magnitude greater than chain data is stored in the datastore.

Here are the relative sizes of the different sub-stores on filecoin devnets (thanks @travisperson for grabbing these)

user:
```
$ du -h /mnt/storage/filecoins/0
76K    /mnt/storage/filecoins/0/deals
13G    /mnt/storage/filecoins/0/badger
639M    /mnt/storage/filecoins/0/chain
24K    /mnt/storage/filecoins/0/snapshots
20K    /mnt/storage/filecoins/0/wallet
8.0K    /mnt/storage/filecoins/0/staging
8.0K    /mnt/storage/filecoins/0/keystore
13G    /mnt/storage/filecoins/0
```

test
```
$ du -h /mnt/storage/filecoins/0
28M    /mnt/storage/filecoins/0/chain
16K    /mnt/storage/filecoins/0/snapshots
76K    /mnt/storage/filecoins/0/deals
8.0K    /mnt/storage/filecoins/0/staging
1.2G    /mnt/storage/filecoins/0/badger
20K    /mnt/storage/filecoins/0/wallet
8.0K    /mnt/storage/filecoins/0/keystore
1.3G    /mnt/storage/filecoins/0

28M    /mnt/storage/filecoins/1/chain
257M    /mnt/storage/filecoins/1/sealed
16K    /mnt/storage/filecoins/1/snapshots
64K    /mnt/storage/filecoins/1/deals
257M    /mnt/storage/filecoins/1/staging
454M    /mnt/storage/filecoins/1/badger
20K    /mnt/storage/filecoins/1/wallet
8.0K    /mnt/storage/filecoins/1/keystore
995M    /mnt/storage/filecoins/1

28M    /mnt/storage/filecoins/2/chain
513M    /mnt/storage/filecoins/2/sealed
16K    /mnt/storage/filecoins/2/snapshots
76K    /mnt/storage/filecoins/2/deals
261M    /mnt/storage/filecoins/2/staging
460M    /mnt/storage/filecoins/2/badger
20K    /mnt/storage/filecoins/2/wallet
8.0K    /mnt/storage/filecoins/2/keystore
1.3G    /mnt/storage/filecoins/2

28M    /mnt/storage/filecoins/3/chain
257M    /mnt/storage/filecoins/3/sealed
16K    /mnt/storage/filecoins/3/snapshots
64K    /mnt/storage/filecoins/3/deals
257M    /mnt/storage/filecoins/3/staging
455M    /mnt/storage/filecoins/3/badger
20K    /mnt/storage/filecoins/3/wallet
8.0K    /mnt/storage/filecoins/3/keystore
995M    /mnt/storage/filecoins/3

28M    /mnt/storage/filecoins/4/chain
257M    /mnt/storage/filecoins/4/sealed
16K    /mnt/storage/filecoins/4/snapshots
64K    /mnt/storage/filecoins/4/deals
257M    /mnt/storage/filecoins/4/staging
453M    /mnt/storage/filecoins/4/badger
20K    /mnt/storage/filecoins/4/wallet
8.0K    /mnt/storage/filecoins/4/keystore
994M    /mnt/storage/filecoins/4
```

The amount of data stored in `badger` is between 46x (test node 0) and 16x (other test nodes) that of the data stored in `chain`

We don't know what this data is and why it is dominating our repo size.  It appears that the `badger` directory wraps the blockstore/cborstore of the filecoin node (need to confirm this).

We need to determine where this data comes from and sketch out a plan for using our storage resources more efficiently.  

### Acceptance criteria
We have a document describing in vivid detail why we see this large resource use and understand some of our options for reducing it.  The explanation should also give us a better understanding of datastore usage patterns generally (who is storing what).  This document might live in the design docs "other" category.

### Risks + pitfalls

### Where to begin
One hypothesis is that the bloat comes from storing lots of actor state over and over.  This really only makes sense if there are lots of actors with state that changes a lot because the content addressed nature of the blockstore should be doing heavy duty deduplication automatically.
Datastore is larger than expected
issue
created
ZenGround0
username_0
We now know that the biggest source of disk usage is flushing state storage to the cborstore HAMT. 
 See [this writeup](https://docs.google.com/document/d/1-EUt0Oj5O5Simb8OWW_hVGDC3_W7iyJnnF4HgW6gsbA/edit?usp=sharing) for an explanation and steps to reproduce:
comment
closed
ZenGround0
username_0
issue
created
whyrusleeping
username_1
@username_0 your analysis is missing a few things.

- Blocks have all messages embedded directly in them. So we are storing every message once, plus an additional time for each block they are contained in (tipsets with multiple blocks will have duplicate messages)
- sector commitments are stored as a flat array in the miner actors state storage. This means that every time a miner adds a sector, it rewrites the entire list as a new object, leading to sector storing taking up `(N^2)/2` space on disk per miner.
- The HAMT is probably too wide, its current width is 256, which leads to a lower deduplication factor than we might like to have. We should investigate dropping this down to 128, 64, or even 32.
- All objects are way bigger than they need to be, since we're storing things as plain cbor with all the field names embedded. FCS will help here

cc @frrist who I know was looking into this as well.
comment
reopened
whyrusleeping
username_1
### Description
We have gotten reports from @username_1 and others that the datastore is much bigger than expected.  The expectation (I believe) is that chain data dominates the total amount of (non-sector) data stored by the filecoin node.  However at present other data with size an order of magnitude greater than chain data is stored in the datastore.

Here are the relative sizes of the different sub-stores on filecoin devnets (thanks @travisperson for grabbing these)

user:
```
$ du -h /mnt/storage/filecoins/0
76K    /mnt/storage/filecoins/0/deals
13G    /mnt/storage/filecoins/0/badger
639M    /mnt/storage/filecoins/0/chain
24K    /mnt/storage/filecoins/0/snapshots
20K    /mnt/storage/filecoins/0/wallet
8.0K    /mnt/storage/filecoins/0/staging
8.0K    /mnt/storage/filecoins/0/keystore
13G    /mnt/storage/filecoins/0
```

test
```
$ du -h /mnt/storage/filecoins/0
28M    /mnt/storage/filecoins/0/chain
16K    /mnt/storage/filecoins/0/snapshots
76K    /mnt/storage/filecoins/0/deals
8.0K    /mnt/storage/filecoins/0/staging
1.2G    /mnt/storage/filecoins/0/badger
20K    /mnt/storage/filecoins/0/wallet
8.0K    /mnt/storage/filecoins/0/keystore
1.3G    /mnt/storage/filecoins/0

28M    /mnt/storage/filecoins/1/chain
257M    /mnt/storage/filecoins/1/sealed
16K    /mnt/storage/filecoins/1/snapshots
64K    /mnt/storage/filecoins/1/deals
257M    /mnt/storage/filecoins/1/staging
454M    /mnt/storage/filecoins/1/badger
20K    /mnt/storage/filecoins/1/wallet
8.0K    /mnt/storage/filecoins/1/keystore
995M    /mnt/storage/filecoins/1

28M    /mnt/storage/filecoins/2/chain
513M    /mnt/storage/filecoins/2/sealed
16K    /mnt/storage/filecoins/2/snapshots
76K    /mnt/storage/filecoins/2/deals
261M    /mnt/storage/filecoins/2/staging
460M    /mnt/storage/filecoins/2/badger
20K    /mnt/storage/filecoins/2/wallet
8.0K    /mnt/storage/filecoins/2/keystore
1.3G    /mnt/storage/filecoins/2

28M    /mnt/storage/filecoins/3/chain
257M    /mnt/storage/filecoins/3/sealed
16K    /mnt/storage/filecoins/3/snapshots
64K    /mnt/storage/filecoins/3/deals
257M    /mnt/storage/filecoins/3/staging
455M    /mnt/storage/filecoins/3/badger
20K    /mnt/storage/filecoins/3/wallet
8.0K    /mnt/storage/filecoins/3/keystore
995M    /mnt/storage/filecoins/3

28M    /mnt/storage/filecoins/4/chain
257M    /mnt/storage/filecoins/4/sealed
16K    /mnt/storage/filecoins/4/snapshots
64K    /mnt/storage/filecoins/4/deals
257M    /mnt/storage/filecoins/4/staging
453M    /mnt/storage/filecoins/4/badger
20K    /mnt/storage/filecoins/4/wallet
8.0K    /mnt/storage/filecoins/4/keystore
994M    /mnt/storage/filecoins/4
```

The amount of data stored in `badger` is between 46x (test node 0) and 16x (other test nodes) that of the data stored in `chain`

We don't know what this data is and why it is dominating our repo size.  It appears that the `badger` directory wraps the blockstore/cborstore of the filecoin node (need to confirm this).

We need to determine where this data comes from and sketch out a plan for using our storage resources more efficiently.  

### Acceptance criteria
We have a document describing in vivid detail why we see this large resource use and understand some of our options for reducing it.  The explanation should also give us a better understanding of datastore usage patterns generally (who is storing what).  This document might live in the design docs "other" category.

### Risks + pitfalls

### Where to begin
One hypothesis is that the bloat comes from storing lots of actor state over and over.  This really only makes sense if there are lots of actors with state that changes a lot because the content addressed nature of the blockstore should be doing heavy duty deduplication automatically.
Datastore is larger than expected
issue
created
anorth
username_2
Thanks @username_0. Please summarise the findings and suggested actions here (tho IMO the full analysis and repro steps are ok in a doc). We need a few follow-up issues too outlining the various opportunities for improvement, which we can then prioritise independently.
comment
created
ZenGround0
username_0
[Meta note: My strong opinion is that the analysis and repro steps should only live in a doc.  A one sentence summary and link is WAI.  The question at hand is "what kind of doc should they live in?"  I have a similar (but weaker) opinion about design docs living as markdown on github but let's talk about that [here](https://github.com/filecoin-project/pm/issues/147).]

## Suggested Actions Related to State Storage

### [Garbage Collecting State Storage](https://github.com/filecoin-project/go-filecoin/issues/2634)

### [Our HAMT/CHAMP should be documented](https://github.com/filecoin-project/go-filecoin/issues/1268)

### [HAMT Usage optimizations](https://github.com/filecoin-project/go-filecoin/issues/2635)

## Suggested Actions Related to the go-filecoin Disk Use in General

### [Don't copy chain state unnecessarily](https://github.com/filecoin-project/go-filecoin/issues/2636)

### [Garbage Collecting Chain Data Storage](https://github.com/filecoin-project/go-filecoin/issues/2637)

### [Chain Datastore Increases on Daemon Restart](https://github.com/filecoin-project/go-filecoin/issues/2639)
I didn't look into this deeply, just observed the problem.  First step is to make sure it's not just badger trying to be clever and allocating in advance because there are a lot of writes during load

### [Stop JSON Encoding Chain](https://github.com/filecoin-project/go-filecoin/issues/1002)
this happens [here](https://github.com/filecoin-project/go-filecoin/blob/master/chain/default_store.go#L338) and [here](https://github.com/filecoin-project/go-filecoin/blob/master/chain/default_store.go#L349)

### [Merkelize Messages for Deduplication](https://github.com/filecoin-project/go-filecoin/issues/1324)
[see spec issue for latest work](https://github.com/filecoin-project/specs/issues/94)

### [Correctly Encode CBOR Data](https://github.com/filecoin-project/go-filecoin/issues/2640)

### [Miner Actor State Improvements](https://github.com/filecoin-project/specs/issues/116)
Note this is still spec work not tracked in go-filecoin issues
comment
closed
ZenGround0
username_0
issue