Repository: filecoin-project/lotus

Issue #10884: Filecoin state profiling 
Created: 2023-05-18T20:27:13Z
edited by @ZenGround0
User Story
I am a user who develops the filecoin protocol or software involved in running or using the protocol.


I want to understand what is happening with filecoin state.  I want to see the current breakdown of byte usage by the protocol to understand if there are dangerous trends that need to be addressed and get ideas for how to fix them.


I also want to be able to go back and analyze how previous changes have impacted snapshot size / protocol data usage.


I can visualize how new changes I have developed impact snapshot size / protocol data usage.


Acceptance Criteria
I can run a command or a small set of commands that gives me two things provided with a filecoin snapshot:


data on the breakdown of state usage of a chain snapshot broken down into a) state tree b) message / headers c) state churn.  Further breakdown of state and churn by i) actor type ii) actor type field and further breakdown of messages by message type


a visualization of this data for quick inspection.  This visualization will be a DiskInventoryX style treemap breakdown of the above data.



#### Deliverables
- [ ] These outputs will be judged against a first motivating task: we will use them to explain exactly how the nv19 protocol upgrade reduced the snapshot size by ~50%. https://github.com/filecoin-project/lotus/issues/11035
- [x] run this process in our own infra on a regular basis to monitor the network's condition. (https://github.com/filecoin-project/lotus/issues/11022)
- [x] Document how to use this tool (shipped at https://github.com/filecoin-project/lotus/discussions/11037)

Techincal Breakdowns
- [x] Import snapshot into something that allows for analaysis (current plan is to use lotus)
- [x] Debug and repurpose existing `lotus-shed stateroot stat` as needed
- [x] Develop similar functionalities for whole snapshot churn / messages / headers
- [x] Experiment with and decide on level of granularity, likely breaking actor types down into top level fields
- [x] Use a data output format that is good for people and plotting programs (probably json)
- [x] Plotting program: probably using [this tool]*(https://plotly.com/python/treemaps/)

Simplifying assumptions (we can revisit these)

Ignore overlap among actors to start
Don't look at individual actors to start, just actor types
All operations are over a single snapshot, forget about measuring longer time scales

Bonus Cake
visualization suggestion from andy: https://askubuntu.com/questions/73160/how-do-i-find-the-amount-of-free-space-on-my-hard-drive.  This is a good possible alternative to the python based treemap

Duplicate issue #10890: Chain datastore profiling
Created: 2023-05-19T04:50:43Z
originated by @arajasek
goal
Lotus's DB size grow at breakneck speed, build tool to better understand of Lotus's DB footprint
potential impact

reduce DB growth

questions to answer

 where is the data coming from ? Sure Filecoin has churn, but what else? Does it match the expected growth based on churn? related #10884

Closing comment (2023-06-06T19:36:59Z): Related to #10884, but focused on the actual datastores in Lotus, and how they grow.