Package: graphsense-bitcoin-etl
Graphsense fork of Bitcoin ETLThis fork is a purpose tailored version for the graphsense infrastructure.Changes preserve nulldata and other script types in the output. Original code can be found at: [upstream bitcoin etl] (https://github.com/blockchain-etl/bitcoin-etl)Install Bitcoin ETL:pipinstallbitcoin-etlExport blocks and transactions (Schema,Reference):>bitcoinetlexport_blocks_and_transactions--start-block0--end-block500000\--provider-urihttp://user:pass@localhost:8332--chainbitcoin\--blocks-outputblocks.json--transactions-outputtransactions.jsonSupported chains:bitcoinbitcoin_cashbitcoin_golddogecoinlitecoindashzcashStream blockchain data continually to console (Reference):>pipinstallbitcoin-etl[streaming]>bitcoinetlstream-phttp://user:pass@localhost:8332--start-block500000Stream blockchain data continually to Google Pub/Sub (Reference):>exportGOOGLE_APPLICATION_CREDENTIALS=/path_to_credentials_file.json
>bitcoinetlstream-phttp://user:pass@localhost:8332--start-block500000--outputprojects/your-project/topics/crypto_bitcoinFor the latest version, check out the repo and call>pipinstall-e.[streaming]>pythonbitcoinetl.pyTable of ContentsBitcoin ETLTable of ContentsSchemablocks.jsontransactions.jsontransaction_inputtransaction_outputExporting the BlockchainRunning in DockerCommand Referenceexport_blocks_and_transactionsenrich_transactionsget_block_range_for_dateexport_allstreamRunning TestsRunning Tox TestsPublic Datasets in BigQuerySchemablocks.jsonFieldTypehashhex_stringsizebigintstripped_sizebigintweightbigintnumberbigintversionbigintmerkle_roothex_stringtimestampbigintnoncehex_stringbitshex_stringcoinbase_paramhex_stringtransaction_countbiginttransactions.jsonFieldTypehashhex_stringsizebigintvirtual_sizebigintversionbigintlock_timebigintblock_numberbigintblock_hashhex_stringblock_timestampbigintis_coinbasebooleanindexbigintinputs[]transaction_inputoutputs[]transaction_outputinput_countbigintoutput_countbigintinput_valuebigintoutput_valuebigintfeebiginttransaction_inputFieldTypeindexbigintspent_transaction_hashhex_stringspent_output_indexbigintscript_asmstringscript_hexhex_stringsequencebigintrequired_signaturesbiginttypestringaddresses[]stringvaluebiginttransaction_outputFieldTypeindexbigintscript_asmstringscript_hexhex_stringrequired_signaturesbiginttypestringaddresses[]stringvaluebigintYou can find column descriptions inschemasNotes:Output values returned by Dogecoin API had precision loss in the clients prior to version 1.14.
It's caused by this issuehttps://github.com/dogecoin/dogecoin/issues/1558The explorers that used older versions to export the data may show incorrect address balances and transaction amounts.For Zcash,vjoinsplitandvalueBalancefields are converted to inputs and outputs with type 'shielded'https://zcash-rpc.github.io/getrawtransaction.html,https://zcash.readthedocs.io/en/latest/rtd_pages/zips/zip-0243.htmlExporting the BlockchainInstall python 3.5.3+https://www.python.org/downloads/Install Bitcoin nodehttps://hackernoon.com/a-complete-beginners-guide-to-installing-a-bitcoin-full-node-on-linux-2018-edition-cb8e384479eaStart Bitcoin.
Make sure it downloaded the blocks that you need by executing$ bitcoin-cli getblockchaininfoin the terminal.
You can export blocks belowblocks, there is no need to wait until the full syncInstall Bitcoin ETL:>pipinstallbitcoin-etlExport blocks & transactions:>bitcoinetlexport_all--start0--end499999\--partition-batch-size100\--provider-urihttp://user:pass@localhost:8332--chainbitcoinThe result will be in theoutputsubdirectory, partitioned in Hive style:output/blocks/start_block=00000000/end_block=00000099/blocks_00000000_00000099.csv
output/blocks/start_block=00000100/end_block=00000199/blocks_00000100_=00000199.csv
...
output/transactions/start_block=00000000/end_block=00000099/transactions_00000000_00000099.csv
...In casebitcoinetlcommand is not available in PATH, usepython -m bitcoinetlinstead.Running in DockerInstall Dockerhttps://docs.docker.com/install/Build a docker image>dockerbuild-tbitcoin-etl:latest.
>dockerimagelsRun a container out of the image>dockerrun-v$HOME/output:/bitcoin-etl/outputbitcoin-etl:latestexport_blocks_and_transactions--start-block0--end-block500000\--provider-urihttp://user:pass@localhost:8332--blocks-outputoutput/blocks.json--transactions-outputoutput/transactions.jsonRun streaming to console or Pub/Sub>dockerbuild-tbitcoin-etl:latest-streaming-fDockerfile_with_streaming.
>echo"Stream to console">dockerrunbitcoin-etl:latest-streamingstream-phttp://user:pass@localhost:8332--start-block500000>echo"Stream to Pub/Sub">dockerrun-v/path_to_credentials_file/:/bitcoin-etl/--envGOOGLE_APPLICATION_CREDENTIALS=/bitcoin-etl/credentials_file.jsonbitcoin-etl:latest-streamingstream-phttp://user:pass@localhost:8332--start-block500000--outputprojects/your-project/topics/crypto_bitcoinRefer tohttps://github.com/blockchain-etl/bitcoin-etl-streamingfor deploying the streaming app to
Google Kubernetes Engine.Command Referenceexport_blocks_and_transactionsenrich_transactionsget_block_range_for_dateexport_allstreamAll the commands accept-hparameter for help, e.g.:>bitcoinetlexport_blocks_and_transactions--help
Usage:bitcoinetl.pyexport_blocks_and_transactions[OPTIONS]Exportblocksandtransactions.

Options:-s,--start-blockINTEGERStartblock-e,--end-blockINTEGEREndblock[required]-b,--batch-sizeINTEGERThenumberofblockstoexportatatime.-p,--provider-uriTEXTTheURIoftheremoteBitcoinnode-w,--max-workersINTEGERThemaximumnumberofworkers.--blocks-outputTEXTTheoutputfileforblocks.Ifnotprovidedblockswillnotbeexported.Use"-"forstdout--transactions-outputTEXTTheoutputfilefortransactions.Ifnotprovidedtransactionswillnotbeexported.Use"-"forstdout--helpShowthismessageandexit.For the--outputparameters the supported type is json. The format type is inferred from the output file name.export_blocks_and_transactions>bitcoinetlexport_blocks_and_transactions--start-block0--end-block500000\--provider-urihttp://user:pass@localhost:8332\--blocks-outputblocks.json--transactions-outputtransactions.jsonOmit--blocks-outputor--transactions-outputoptions if you want to export only transactions/blocks.You can tune--batch-size,--max-workersfor performance.Note thatrequired_signatures,type,addresses, andvaluefields will be empty in transactions inputs.
Useenrich_transactionsto populate those fields.enrich_transactionsYou need to run bitcoin daemon with optiontxindex=1for this command to work.>bitcoinetlenrich_transactions\--provider-urihttp://user:pass@localhost:8332\--transactions-inputtransactions.json--transactions-outputenriched_transactions.jsonYou can tune--batch-size,--max-workersfor performance.get_block_range_for_date>bitcoinetlget_block_range_for_date--provider-urihttp://user:pass@localhost:8332--date=2017-03-01This command is guaranteed to return the block range that covers all blocks withblock.timeon the specified
date. However the returned block range may also contain blocks outside the specified date, because block times are not
monotonichttps://twitter.com/EvgeMedvedev/status/1073844856009576448. You can filterblocks.json/transactions.jsonwith the below command:>bitcoinetlfilter_items-iblocks.json-oblocks_filtered.json\-p"datetime.datetime.fromtimestamp(item['timestamp']).astimezone(datetime.timezone.utc).strftime('%Y-%m-%d') == '2017-03-01'"export_all>bitcoinetlexport_all--provider-urihttp://user:pass@localhost:8332--start2018-01-01--end2018-01-02You can tune--export-batch-size,--max-workersfor performance.stream>bitcoinetlstream--provider-urihttp://user:pass@localhost:8332--start-block500000This command outputs blocks and transactions to the console by default.Use--outputoption to specify the Google Pub/Sub topic where to publish blockchain data,
e.g.projects/your-project/topics/crypto_bitcoin. Blocks and transactions will be pushed toprojects/your-project/topics/crypto_bitcoin.blocksandprojects/your-project/topics/crypto_bitcoin.transactionstopics.The command saves its state tolast_synced_block.txtfile where the last synced block number is saved periodically.Specify either--start-blockor--last-synced-block-fileoption.--last-synced-block-fileshould point to the
file where the block number, from which to start streaming the blockchain data, is saved.Use the--lagoption to specify how many blocks to lag behind the head of the blockchain. It's the simplest way to
handle chain reorganizations - they are less likely the further a block from the head.Use the--chainoption to specify the type of the chain, e.g.bitcoin,litecoin,dash,zcash, etc.You can tune--period-seconds,--batch-size,--max-workersfor performance.Running Tests>pipinstall-e.[dev]>echo"The below variables are optional">exportBITCOINETL_BITCOIN_PROVIDER_URI=http://user:pass@localhost:8332
>exportBITCOINETL_LITECOIN_PROVIDER_URI=http://user:pass@localhost:8331
>exportBITCOINETL_DOGECOIN_PROVIDER_URI=http://user:pass@localhost:8330
>exportBITCOINETL_BITCOIN_CASH_PROVIDER_URI=http://user:pass@localhost:8329
>exportBITCOINETL_DASH_PROVIDER_URI=http://user:pass@localhost:8328
>exportBITCOINETL_ZCASH_PROVIDER_URI=http://user:pass@localhost:8327
>pytest-vvRunning Tox Tests>pipinstalltox
>toxPublic Datasets in BigQueryhttps://cloud.google.com/blog/products/data-analytics/introducing-six-new-cryptocurrencies-in-bigquery-public-datasets-and-how-to-analyze-them