Package: mercury-fil
mercury-pymercury-pyis a Python package that simplifies
working with historical Filecoin chain data stored on BigQuery.The results from the Mercury client arepandas.DataFrameobjects.prerequisitesInstall thegcloudCLI.Forhomebrewusers:brewinstall--caskgoogle-cloud-sdkOther installation methods arehere.installationpip install mercury-filusageMake sure you're authenticated:gcloud auth login.
You'll also need to be in the right project:gcloud config set project protocol-labs-data.💡 Only query for columns (and the height range) that you need.
A query costs $6.25/TB.simple apiThefevm_contractsandderived_gas_outputsmethods only have
2 required arguments, namely thestart_heightandend_height.It's financially costly to query all of the heights so make sure
that youonly query for heights that you need. The tables
are partitioned in BigQuery byheightat intervals of3000.The API defaults to a dry run so it computes the bytes it will
process before you decide whether to run the query job.>>>frommercuryimportClient>>>client=Client()>>>df=client.fevm_contracts(start_height=2683348,end_height=2683348+2880)>>>df'0.1322889 GiB'>>>df=client.fevm_contracts(start_height=2683348,end_height=2683348+2880,dry_run=False)heighteth_addressbyte_codebalance026847830x8b9ebed1f4a0892097b913ba938c06933065294760806040526004361061004a5760003560e01c806363a5...0E-9126842640x719e14fcb364bb05649bd525eb6c4a2d0d4ea2b76080604052600436106100a05760003560e01c8063313c...2212000000000000000.000000000226842380x719e14fcb364bb05649bd525eb6c4a2d0d4ea2b76080604052600436106100a05760003560e01c8063313c...2211000000000000000.000000000326842100x719e14fcb364bb05649bd525eb6c4a2d0d4ea2b76080604052600436106100a05760003560e01c8063313c...1211000000000000000.000000000426842040x719e14fcb364bb05649bd525eb6c4a2d0d4ea2b76080604052600436106100a05760003560e01c8063313c...1201000000000000000.000000000...............365726858970x8f81929b4b8e0a76d13e90dfb5d2316a4a163d266080604052600436106101e35760003560e01c80635c97...58400000000000000000.000000000365826858910x8f81929b4b8e0a76d13e90dfb5d2316a4a163d266080604052600436106101e35760003560e01c80635c97...57400000000000000000.000000000365926858870x8f81929b4b8e0a76d13e90dfb5d2316a4a163d266080604052600436106101e35760003560e01c80635c97...53400000000000000000.000000000366026857160x8f81929b4b8e0a76d13e90dfb5d2316a4a163d266080604052600436106101e35760003560e01c80635c97...49400000000000000000.000000000366126851320x8f81929b4b8e0a76d13e90dfb5d2316a4a163d266080604052600436106101e35760003560e01c80635c97...47400000000000000000.000000000[3662rowsx4columns]>>>df=client.derived_gas_outputs(start_height=0,end_height=2880,dry_run=False)>>>dfheightfromtovaluemethodgas_used0315f3ro3i54tule2vtdcsjdkzjkf6djx3wwe5znv5h4kwxprx...f050E-942916773812150f3sqdk3xwrfrxb77upn4jjnqzamoiuzmykavyguodsmxgh...f050E-943985968022119f3qcagyij6lb7ixssn5vxgrqekike2ujcn7b7v5jci4foq...f050E-944299392231743f3r6vel3cnsvc4ct2zymfxorxz5by4rc5tuwjb3ykbl4cn...f079191000.000000000538297645842592f3qjn4pqvipuaxyjigcbzp4zn5625646ogbqwuzmakgx5q...f033621000.0000000005380578332.....................675645342f3uxgextdo6bmkn5ax3gtfn6t3j3js5ywnojwpmc7suhyp...f040E-9216488846675646342f3uxgextdo6bmkn5ax3gtfn6t3j3js5ywnojwpmc7suhyp...f040E-9216844158675647342f3uxgextdo6bmkn5ax3gtfn6t3j3js5ywnojwpmc7suhyp...f040E-9216563776675648342f3uxgextdo6bmkn5ax3gtfn6t3j3js5ywnojwpmc7suhyp...f040E-92166919166756492051f16aum7jrmrwa7aqr2d5feu7on4emsgqh2gmxdf7qf040E-9216382158[675650rowsx6columns]selecting columnsBigQuery is a columnar data store, which means that the data is stored
on disk by column. For example, thederived_gas_outputs.methodcolumn
would be stored like so:| method | 4 | 4 | 4 | 5 | 5 | 2 | 2 |It is thus more efficient to read many records in a table by column.
Since a column usually has a non-insignificant amount of data (in bytes),
the query can become costly if you select all the columns of a table.
The financial cost is a reason why the default columns in thefevm_contractsandderived_gas_outputsmethods do not contain all of the columns listed inlilium.sh.To select the columns you need, you need to pass atupleof those column names. For example:>>>df=client.fevm_contracts(start_height=2683348,end_height=2683348+2880,columns=("height","actor_id","byte_code_hash"),dry_run=False)>>>dfheightactor_idbyte_code_hash02684783f410fropl5upuucesbf5zco5jhdagsmygkkkhgifa3pi36cdf3ab5e4b065b46d89eb1b595da8acdbb725520983d...12684264f410fogpbj7ftms5qkze32us6w3ckfugu5ivx4eoycoiaf58eb543e44897848df115f2276f7b5ffc12fa068de8c...22684238f410fogpbj7ftms5qkze32us6w3ckfugu5ivx4eoycoiaf58eb543e44897848df115f2276f7b5ffc12fa068de8c...32684210f410fogpbj7ftms5qkze32us6w3ckfugu5ivx4eoycoiaf58eb543e44897848df115f2276f7b5ffc12fa068de8c...42684204f410fogpbj7ftms5qkze32us6w3ckfugu5ivx4eoycoiaf58eb543e44897848df115f2276f7b5ffc12fa068de8c...............36572685897f410fr6azfg2lryfhnuj6sdp3lurrnjfbmpjgwco5svia48a535d2921613878ca9393b62022db4e2caaafd5a23f...36582685891f410fr6azfg2lryfhnuj6sdp3lurrnjfbmpjgwco5svia48a535d2921613878ca9393b62022db4e2caaafd5a23f...36592685887f410fr6azfg2lryfhnuj6sdp3lurrnjfbmpjgwco5svia48a535d2921613878ca9393b62022db4e2caaafd5a23f...36602685716f410fr6azfg2lryfhnuj6sdp3lurrnjfbmpjgwco5svia48a535d2921613878ca9393b62022db4e2caaafd5a23f...36612685132f410fr6azfg2lryfhnuj6sdp3lurrnjfbmpjgwco5svia48a535d2921613878ca9393b62022db4e2caaafd5a23f...[3662rowsx3columns]As with querying only data at the range of heights that you need,only
select columns that you need.supported tablesfevm_contractsderived_gas_outputsminer_sector_eventsminer_sector_infoscontributingTo contribute, you'll needPoetryto install the dependencies.To run the tests, simply runpytest.