GitHub PullRequestReviewCommentEvent at 2014-11-05 14:25:12
Repository: counterpartyd
Actor: PhantomPhreak
Payload: {"action":"created","comment":{"url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/pulls/comments/19875332","id":19875332,"diff_hunk":"@@ -1255,6 +1274,113 @@ def list_tx (db, block_hash, block_index, block_time, tx_hash, tx_index):\n \n     return\n \n+def initialise_transactions(db, bitcoind_dir):\n+    print(\"Warnings:\")\n+    print(\"- Ensure that bitcoind is stopped\")\n+    print(\"- You should reindex bitcoind after the initialisation (restart with -reindex=1)\")\n+    print(\"- This operation takes a long time\")\n+    if input('Continue the initialisation ? (y/N) : ') != 'y':\n+        return\n+\n+    start_time_total = time.time()\n+\n+    first_hash = config.BLOCK_FIRST_TESTNET_HASH if config.TESTNET else config.BLOCK_FIRST_MAINNET_HASH\n+\n+    chain_parser = ChainstateParser(os.path.join(bitcoind_dir, 'chainstate'))\n+    last_hash = chain_parser.get_last_block_hash()\n+    chain_parser.close()\n+\n+    block_parser = BlockchainParser(os.path.join(bitcoind_dir, 'blocks'), os.path.join(bitcoind_dir, 'blocks/index'));\n+    cursor = db.cursor()\n+\n+    current_hash = last_hash\n+    tx_index = 0\n+\n+    with db:\n+\n+        # prepare sqlite database\n+        logging.info('preparing database...')\n+        start_time = time.time()\n+        for table in TABLES + ['balances']:\n+            cursor.execute('''DROP TABLE IF EXISTS {}'''.format(table))\n+        first_block = block_parser.read_raw_block(first_hash)\n+        initialise(db)\n+        cursor.execute('''DELETE FROM transactions WHERE block_index >= ?''', (first_block['block_index'],))\n+        cursor.execute('''DELETE FROM blocks WHERE block_index >= ?''', (first_block['block_index'],))\n+        logging.info('database prepared in {:.3f}'.format(time.time() - start_time))\n+\n+        # parse all blocks backward\n+        while current_hash != None:\n+            start_time = time.time()\n+            block = block_parser.read_raw_block(current_hash)\n+            transactions = []\n+\n+            # get tx infos for all transactions of the block\n+            for tx in block['transactions']:\n+                tx_info = b'', None, None, None, None\n+                try:\n+                    if (config.TESTNET and block['block_index'] >= config.FIRST_MULTISIG_BLOCK_TESTNET):  # Protocol change.","path":"lib/blocks.py","position":268,"original_position":268,"commit_id":"44fc01b9da338b064c164b0a0bf413220addffa1","original_commit_id":"44fc01b9da338b064c164b0a0bf413220addffa1","user":{"login":"PhantomPhreak","id":6189829,"avatar_url":"https://avatars.githubusercontent.com/u/6189829?v=2","gravatar_id":"","url":"https://api.github.com/users/PhantomPhreak","html_url":"https://github.com/PhantomPhreak","followers_url":"https://api.github.com/users/PhantomPhreak/followers","following_url":"https://api.github.com/users/PhantomPhreak/following{/other_user}","gists_url":"https://api.github.com/users/PhantomPhreak/gists{/gist_id}","starred_url":"https://api.github.com/users/PhantomPhreak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PhantomPhreak/subscriptions","organizations_url":"https://api.github.com/users/PhantomPhreak/orgs","repos_url":"https://api.github.com/users/PhantomPhreak/repos","events_url":"https://api.github.com/users/PhantomPhreak/events{/privacy}","received_events_url":"https://api.github.com/users/PhantomPhreak/received_events","type":"User","site_admin":false},"body":"How much of this `for` loop can be abstracted away?","created_at":"2014-11-05T14:25:12Z","updated_at":"2014-11-05T14:25:12Z","html_url":"https://github.com/CounterpartyXCP/counterpartyd/pull/362#discussion_r19875332","pull_request_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/pulls/362","_links":{"self":{"href":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/pulls/comments/19875332"},"html":{"href":"https://github.com/CounterpartyXCP/counterpartyd/pull/362#discussion_r19875332"},"pull_request":{"href":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/pulls/362"}}},"pull_request":{"url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/pulls/362","id":23237802,"html_url":"https://github.com/CounterpartyXCP/counterpartyd/pull/362","diff_url":"https://github.com/CounterpartyXCP/counterpartyd/pull/362.diff","patch_url":"https://github.com/CounterpartyXCP/counterpartyd/pull/362.patch","issue_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/issues/362","number":362,"state":"open","locked":false,"title":"Fast initial synchronisation","user":{"login":"JahPowerBit","id":4574979,"avatar_url":"https://avatars.githubusercontent.com/u/4574979?v=2","gravatar_id":"","url":"https://api.github.com/users/JahPowerBit","html_url":"https://github.com/JahPowerBit","followers_url":"https://api.github.com/users/JahPowerBit/followers","following_url":"https://api.github.com/users/JahPowerBit/following{/other_user}","gists_url":"https://api.github.com/users/JahPowerBit/gists{/gist_id}","starred_url":"https://api.github.com/users/JahPowerBit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JahPowerBit/subscriptions","organizations_url":"https://api.github.com/users/JahPowerBit/orgs","repos_url":"https://api.github.com/users/JahPowerBit/repos","events_url":"https://api.github.com/users/JahPowerBit/events{/privacy}","received_events_url":"https://api.github.com/users/JahPowerBit/received_events","type":"User","site_admin":false},"body":"The synchronisation is done directly from bitcoind database files.\r\nHere the steps of the process:\r\n\r\n- step 1: prepare sqlite database \r\n- step 2: parse all blocks backward (the position of each block in blk000XX.dat files, are taken from the leveldb index (blocks/index dir)\r\n- step 3: reorder transactions\r\n- step 4: parse transactions\r\n\r\nThe main function, where these 6 steps are done, is `blocks.initialise_transactions()`:\r\nhttps://github.com/JahPowerBit/counterpartyd/blob/develop/lib/blocks.py#L1260\r\n\r\nAverage time to parse a block on an Intel Xeon 3.2Ghz with a 7200 RPM HD is about 0.1s.\r\n\r\n**Important**: bitcoind must be stopped before launching this script.\r\n\r\nMissing:\r\n- manage `get_tx_info2`\r\n- add a warning/question \"stop your bitcoind, this operations takes times, etc... are you sure you want to continue ? y/n\"\r\n- make a python lib for all files in `/lib/blockchain/` installable with pip\r\n\r\n","created_at":"2014-10-23T15:48:20Z","updated_at":"2014-11-05T14:25:12Z","closed_at":null,"merged_at":null,"merge_commit_sha":"fc451f63733b5d6efcad7593e62d4985bbbfa182","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/pulls/362/commits","review_comments_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/pulls/362/comments","review_comment_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/pulls/comments/{number}","comments_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/issues/362/comments","statuses_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/statuses/44fc01b9da338b064c164b0a0bf413220addffa1","head":{"label":"JahPowerBit:develop","ref":"develop","sha":"44fc01b9da338b064c164b0a0bf413220addffa1","user":{"login":"JahPowerBit","id":4574979,"avatar_url":"https://avatars.githubusercontent.com/u/4574979?v=2","gravatar_id":"","url":"https://api.github.com/users/JahPowerBit","html_url":"https://github.com/JahPowerBit","followers_url":"https://api.github.com/users/JahPowerBit/followers","following_url":"https://api.github.com/users/JahPowerBit/following{/other_user}","gists_url":"https://api.github.com/users/JahPowerBit/gists{/gist_id}","starred_url":"https://api.github.com/users/JahPowerBit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JahPowerBit/subscriptions","organizations_url":"https://api.github.com/users/JahPowerBit/orgs","repos_url":"https://api.github.com/users/JahPowerBit/repos","events_url":"https://api.github.com/users/JahPowerBit/events{/privacy}","received_events_url":"https://api.github.com/users/JahPowerBit/received_events","type":"User","site_admin":false},"repo":{"id":16733240,"name":"counterpartyd","full_name":"JahPowerBit/counterpartyd","owner":{"login":"JahPowerBit","id":4574979,"avatar_url":"https://avatars.githubusercontent.com/u/4574979?v=2","gravatar_id":"","url":"https://api.github.com/users/JahPowerBit","html_url":"https://github.com/JahPowerBit","followers_url":"https://api.github.com/users/JahPowerBit/followers","following_url":"https://api.github.com/users/JahPowerBit/following{/other_user}","gists_url":"https://api.github.com/users/JahPowerBit/gists{/gist_id}","starred_url":"https://api.github.com/users/JahPowerBit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JahPowerBit/subscriptions","organizations_url":"https://api.github.com/users/JahPowerBit/orgs","repos_url":"https://api.github.com/users/JahPowerBit/repos","events_url":"https://api.github.com/users/JahPowerBit/events{/privacy}","received_events_url":"https://api.github.com/users/JahPowerBit/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/JahPowerBit/counterpartyd","description":"Decentralised financial instruments in a protocol build on top of the Bitcoin blockchain","fork":true,"url":"https://api.github.com/repos/JahPowerBit/counterpartyd","forks_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/forks","keys_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/keys{/key_id}","collaborators_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/teams","hooks_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/hooks","issue_events_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/issues/events{/number}","events_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/events","assignees_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/assignees{/user}","branches_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/branches{/branch}","tags_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/tags","blobs_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/git/refs{/sha}","trees_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/git/trees{/sha}","statuses_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/statuses/{sha}","languages_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/languages","stargazers_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/stargazers","contributors_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/contributors","subscribers_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/subscribers","subscription_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/subscription","commits_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/commits{/sha}","git_commits_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/git/commits{/sha}","comments_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/comments{/number}","issue_comment_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/issues/comments/{number}","contents_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/contents/{+path}","compare_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/compare/{base}...{head}","merges_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/merges","archive_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/downloads","issues_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/issues{/number}","pulls_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/pulls{/number}","milestones_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/milestones{/number}","notifications_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/labels{/name}","releases_url":"https://api.github.com/repos/JahPowerBit/counterpartyd/releases{/id}","created_at":"2014-02-11T14:28:32Z","updated_at":"2014-10-08T10:06:52Z","pushed_at":"2014-11-05T11:48:04Z","git_url":"git://github.com/JahPowerBit/counterpartyd.git","ssh_url":"git@github.com:JahPowerBit/counterpartyd.git","clone_url":"https://github.com/JahPowerBit/counterpartyd.git","svn_url":"https://github.com/JahPowerBit/counterpartyd","homepage":null,"size":8706,"stargazers_count":1,"watchers_count":1,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":1,"default_branch":"master"}},"base":{"label":"CounterpartyXCP:develop","ref":"develop","sha":"8bd58e3010ddf37391e71607b7dbd8cdbe082d4e","user":{"login":"CounterpartyXCP","id":7490848,"avatar_url":"https://avatars.githubusercontent.com/u/7490848?v=2","gravatar_id":"","url":"https://api.github.com/users/CounterpartyXCP","html_url":"https://github.com/CounterpartyXCP","followers_url":"https://api.github.com/users/CounterpartyXCP/followers","following_url":"https://api.github.com/users/CounterpartyXCP/following{/other_user}","gists_url":"https://api.github.com/users/CounterpartyXCP/gists{/gist_id}","starred_url":"https://api.github.com/users/CounterpartyXCP/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/CounterpartyXCP/subscriptions","organizations_url":"https://api.github.com/users/CounterpartyXCP/orgs","repos_url":"https://api.github.com/users/CounterpartyXCP/repos","events_url":"https://api.github.com/users/CounterpartyXCP/events{/privacy}","received_events_url":"https://api.github.com/users/CounterpartyXCP/received_events","type":"Organization","site_admin":false},"repo":{"id":15202468,"name":"counterpartyd","full_name":"CounterpartyXCP/counterpartyd","owner":{"login":"CounterpartyXCP","id":7490848,"avatar_url":"https://avatars.githubusercontent.com/u/7490848?v=2","gravatar_id":"","url":"https://api.github.com/users/CounterpartyXCP","html_url":"https://github.com/CounterpartyXCP","followers_url":"https://api.github.com/users/CounterpartyXCP/followers","following_url":"https://api.github.com/users/CounterpartyXCP/following{/other_user}","gists_url":"https://api.github.com/users/CounterpartyXCP/gists{/gist_id}","starred_url":"https://api.github.com/users/CounterpartyXCP/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/CounterpartyXCP/subscriptions","organizations_url":"https://api.github.com/users/CounterpartyXCP/orgs","repos_url":"https://api.github.com/users/CounterpartyXCP/repos","events_url":"https://api.github.com/users/CounterpartyXCP/events{/privacy}","received_events_url":"https://api.github.com/users/CounterpartyXCP/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/CounterpartyXCP/counterpartyd","description":"Counterparty reference client","fork":false,"url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd","forks_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/forks","keys_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/keys{/key_id}","collaborators_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/teams","hooks_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/hooks","issue_events_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/issues/events{/number}","events_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/events","assignees_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/assignees{/user}","branches_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/branches{/branch}","tags_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/tags","blobs_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/git/refs{/sha}","trees_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/git/trees{/sha}","statuses_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/statuses/{sha}","languages_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/languages","stargazers_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/stargazers","contributors_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/contributors","subscribers_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/subscribers","subscription_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/subscription","commits_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/commits{/sha}","git_commits_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/git/commits{/sha}","comments_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/comments{/number}","issue_comment_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/issues/comments/{number}","contents_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/contents/{+path}","compare_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/compare/{base}...{head}","merges_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/merges","archive_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/downloads","issues_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/issues{/number}","pulls_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/pulls{/number}","milestones_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/milestones{/number}","notifications_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/labels{/name}","releases_url":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/releases{/id}","created_at":"2013-12-15T11:44:46Z","updated_at":"2014-11-01T17:09:14Z","pushed_at":"2014-11-05T13:22:16Z","git_url":"git://github.com/CounterpartyXCP/counterpartyd.git","ssh_url":"git@github.com:CounterpartyXCP/counterpartyd.git","clone_url":"https://github.com/CounterpartyXCP/counterpartyd.git","svn_url":"https://github.com/CounterpartyXCP/counterpartyd","homepage":"http://counterparty.io","size":16311,"stargazers_count":92,"watchers_count":92,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":70,"mirror_url":null,"open_issues_count":61,"forks":70,"open_issues":61,"watchers":92,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/pulls/362"},"html":{"href":"https://github.com/CounterpartyXCP/counterpartyd/pull/362"},"issue":{"href":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/issues/362"},"comments":{"href":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/issues/362/comments"},"review_comments":{"href":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/pulls/362/comments"},"review_comment":{"href":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/pulls/comments/{number}"},"commits":{"href":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/pulls/362/commits"},"statuses":{"href":"https://api.github.com/repos/CounterpartyXCP/counterpartyd/statuses/44fc01b9da338b064c164b0a0bf413220addffa1"}}}}