Package: eos-etl
EOS ETLJoin Telegram GroupInstall EOS ETL:pipinstalleos-etlExport blocks, transactions and actions (Schema,Reference):>eosetlexport_blocks--start-block1--end-block500000\--provider-urihttp://api.main.alohaeos.com\--blocks-outputblocks.json--transactions-outputtransactions.json--actions-outputactions.jsonStream blockchain data continually to console:>pipinstalleos-etl[streaming]>eosetlstream-phttp://api.main.alohaeos.com--start-block500000For the latest version, check out the repo and call>pipinstall-e.[streaming]>pythoneosetl.pyTable of ContentsSchemaExporting the BlockchainRunning in DockerCommand ReferencePublic Datasets in BigQuerySchemablockstransactionsactionsExporting the BlockchainInstall python 3.6.0+https://www.python.org/downloads/Install EOS node or get access to EOS node maintained by someone else (because running your own node is not so easy).
Some docs:https://developers.eos.io/eosio-nodeos/docs/https://eosnode.tools/https://github.com/CryptoLions/EOS-MainNetMake sure it downloaded the blocks that you need by executing in the terminal:curl--requestPOST\--urlhttps://localhost:8080/v1/chain/get_info\--header'accept: application/json'You can export blocks belowlast_irreversible_block_num, there is no need to wait until the full syncInstall EOS ETL:>pipinstalleos-etlExport blocks, transactions and actions:>eosetlexport_all--start1--end499999\--provider-urihttp://api.main.alohaeos.comIn caseeosetlcommand is not available in PATH, usepython -m eosetlinstead.The result will be in theoutputsubdirectory, partitioned in Hive style:output/blocks/start_block=00000000/end_block=00000099/blocks_00000000_00000099.csv
output/blocks/start_block=00000100/end_block=00000199/blocks_00000100_=00000199.csv
...
output/transactions/start_block=00000000/end_block=00000099/transactions_00000000_00000099.csv
...
output/actions/start_block=00000000/end_block=00000099/actions_00000000_00000099.csv
...Running in DockerInstall Dockerhttps://docs.docker.com/install/Build a docker image>dockerbuild-teos-etl:latest.
>dockerimagelsRun a container out of the image>MSYS_NO_PATHCONV=1dockerrun-v$HOME/output:/eos-etl/outputeos-etl:latest\export_blocks--max-workers50--start-block30000000\--end-block30000100--provider-urihttp://your_eos_node:node_port\--blocks-output./output/blocks.csv--transactions-output./output/transactions.csv\--actions-output./output/actions.csvRun streaming to console or Pub/Sub>MSYS_NO_PATHCONV=1dockerbuild-teos-etl:latest-streaming-fDockerfile_with_streaming.
>echo"Stream to console">MSYS_NO_PATHCONV=1dockerruneos-etl:latest-streamingstream-phttp://api.main.alohaeos.com--start-block500000>echo"Stream to Pub/Sub">MSYS_NO_PATHCONV=1dockerrun-v/path_to_credentials_file/:/eos-etl/--envGOOGLE_APPLICATION_CREDENTIALS=/eos-etl/credentials_file.jsoneos-etl:latest-streamingstream-phttp://api.main.alohaeos.com--start-block500000--outputprojects/your-project/topics/crypto_eosRefer tohttps://github.com/blockchain-etl/blockchain-etl-streamingfor deploying the streaming app to
Google Kubernetes Engine.Command Referenceexport_blocksget_block_range_for_dateexport_allstreamAll the commands accept-hparameter for help, e.g.:>pythoneosetl.pyexport_blocks--help
Usage:eosetl.pyexport_blocks[OPTIONS]Exportblocks,transactionsandactions.

Options:-s,--start-blockINTEGERStartblock-e,--end-blockINTEGEREndblock[required]-p,--provider-uriTEXTTheURIoftheremoteEOSnode-w,--max-workersINTEGERThemaximumnumberofworkers.--blocks-outputTEXTTheoutputfileforblocks.Ifnotprovidedblockswillnotbeexported.Use"-"forstdout--transactions-outputTEXTTheoutputfilefortransactions.Ifnotprovidedtransactionswillnotbeexported.Use"-"forstdout--actions-outputTEXTTheoutputfileforactions.Ifnotprovidedtransactionswillnotbeexported.Use"-"forstdout--helpShowthismessageandexit.For the--outputparameters the supported type is json. The format type is inferred from the output file name.export_blocks>pythoneosetl.pyexport_blocks--start-block1--end-block500000\--provider-urihttp://api.main.alohaeos.com\--blocks-outputblocks.json--transactions-outputtransactions.json--actions-outputactions.jsonOmit--blocks-outputor--transactions-outputor--actions-outputoptions if you want to export only transactions/blocks/actions.You can tune--max-workersfor performance.get_block_range_for_date>pythoneosetl.pyget_block_range_for_date--provider-urihttp://api.main.alohaeos.com--date=2018-06-09export_all>pythoneosetl.pyexport_all--provider-urihttp://api.main.alohaeos.com--start2018-06-08--end2018-06-09You can tune--export-batch-size,--max-workersfor performance.stream>pythoneosetl.pystream--provider-urihttp://api.main.alohaeos.com--start-block500000This command outputs blocks and transactions to the console by default.Use--outputoption to specify the Google Pub/Sub topic where to publish blockchain data,
e.g.projects/your-project/topics/eos_blockchain.The command saves its state tolast_synced_block.txtfile where the last synced block number is saved periodically.Specify either--start-blockor--last-synced-block-fileoption.--last-synced-block-fileshould point to the
file where the block number, from which to start streaming the blockchain data, is saved.Use the--lagoption to specify how many blocks to lag behind the head of the blockchain. It's the simplest way to
handle chain reorganizations - they are less likely the further a block from the head.You can tune--period-seconds,--batch-size,--max-workersfor performance.Running Tests>pipinstall-e.[dev]>echo"The below variables are optional">exportEOSETL_PROVIDER_URI=http://api.main.alohaeos.com
>pytest-vvRunning Tox Tests>pipinstalltox
>toxPublic Datasets in BigQueryTODO:https://cloud.google.com/blog/products/data-analytics/introducing-six-new-cryptocurrencies-in-bigquery-public-datasets-and-how-to-analyze-them