GitHub PullRequestReviewCommentEvent at 2014-10-07 21:59:47
Repository: fluo-io.github.io
Actor: cjnolet
Payload: {"action":"created","comment":{"url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/pulls/comments/18552342","id":18552342,"diff_hunk":"@@ -71,6 +71,41 @@ Other important improvements and features:\n \n All unit and integration tests passed.  Two [stress tests][stress] were run successfully on a 20-node cluster with one test generating 2000 integers and another generating 5000 integers.  The 5000 integer test took about 10 minutes to run after the MapReduce load of integers was complete.\n \n+##Current Status of Percolator Implementation \n+\n+The following tables identify the features necessary for a full implementation of Percolator and the current state of Fluo with regards to the implementation. Note that none of these features have been run at scale. A performance test with a synthetic workload is on the roadmap for 1.0.0.\n+\n+###Necessary features:\n+\n+| Percolator Feature         |  Details | Testing |\n+|----------|-------------------|------|\n+| ACID-compliant cross-row transactions | Provides disitributed transactions with ACID guarantees that work across Accumulo rows | Integration tests |\n+| Observers |    Perform transactions by sending read/write RPCs to Accumulo tablet servers, which in turn send read/write RPCs to HDFS.   |   Integration & cluster stress tests |\n+| Worker Process | Scans Accumulo for changed columns and invokes necessary observers as function calls on the worker. |    Integration & cluster stress tests |\n+| Timestamp Oracle | Provides strictly increasing timestamps; a property required for correct operation of the snapshot isolation protocol. . |    Integration & cluster stress tests |\n+| Oracle Failover | When an active Oracle process dies, timestamps should be served from a different Oracle. They should never go backwards. |    Integration tests |\n+| Lightweight Lock Service | Makes searches for dirty notifications more efficient. Zookeeper is used in this case. Should be low-latency, replicated, balanced, and persist to a backing store. |    Integration & cluster stress tests |\n+| Multithreaded Transasctions | Allows highly parallel execution of transactions and synchronization only during get() and commit() upon which transactions actually collide. |    Integration & cluster stress tests |\n+| Snapshot Isolation | Non-serializable isolation mostly implemented through MVCC provided through Accumulo's timestamp portion of the key. | Integration tests |\n+| Roll-back / Roll-forward semantics | Failures during the first phase of commit cause transactions to be rolled back while failures during the second phase can cause them to be rolled forward. |    Integration tests |\n+\n+\n+###Performance enhancements:\n+\n+| Percolator Feature         |  Details | Testing |\n+|----------|-------------------|------|\n+| Locality Group for NOTIFY column |  Add an extra index in the underlying r-files for the notify column where it can be scanned faster by workers.  | Integration & cluster stress tests  | \n+| Worker scan conflict resolution | Workers register random tablets they choose to scan. When two workers generate the same tablet, one of them re-generates a new random tablet. | Not yet implemented |\n+| Worker advertisement | Workers advertise through the lock service so that the system can know if they failed and any transactions upon which they are working can be rolled back or rolled forward | Integration tests | \n+| Oracle dynamic batching | Timestamps get allocated in batch to workers. This batch size should increase with load to reduce the number of RPCs. | Not yet implemented | \n+| Weak Notifications | Allows notification of a cell without the posibility of transactional conflict through a special type of column that can be notified but not written to | Integration tests |\n+| Conditional Mutations | Allows READ-MODIFY-WRITE of a mutation in a single RPC.  | Integration & cluster stress tests |\n+| Conditional Mutation batching | Delay sending of conditional mutations destined for same tablet server so they can be batched. Lowers number of RPCs, adds latency but provides better parallelism. | Integration & cluster stress tests |\n+| Read batching | Delaying of read operations on tablet server for batching | Not yet implemented |\n+| Prefetching | Reading two or more values from the same row is essentially the same cost as reading a single value. This is because an r-file block still needs to be read and decompressed for both cases. | Not yet implemented |\n+| Column readahead caching | Prediction made each time a column is read, which other columns will be read in the same transaction based on previous behavior. | Not yet implemented |","path":"_posts/release-notes/2014-10-02-1.0.0-alpha-1-release-notes.md","position":36,"original_position":36,"commit_id":"537eea11e9164c48abff0ffb1e92e97aa045cf86","original_commit_id":"537eea11e9164c48abff0ffb1e92e97aa045cf86","user":{"login":"cjnolet","id":1242464,"avatar_url":"https://avatars.githubusercontent.com/u/1242464?v=2","gravatar_id":"","url":"https://api.github.com/users/cjnolet","html_url":"https://github.com/cjnolet","followers_url":"https://api.github.com/users/cjnolet/followers","following_url":"https://api.github.com/users/cjnolet/following{/other_user}","gists_url":"https://api.github.com/users/cjnolet/gists{/gist_id}","starred_url":"https://api.github.com/users/cjnolet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cjnolet/subscriptions","organizations_url":"https://api.github.com/users/cjnolet/orgs","repos_url":"https://api.github.com/users/cjnolet/repos","events_url":"https://api.github.com/users/cjnolet/events{/privacy}","received_events_url":"https://api.github.com/users/cjnolet/received_events","type":"User","site_admin":false},"body":"You were correct about the prefetch and read-ahead caching going together. I'll change the list to reflect this.","created_at":"2014-10-07T21:59:47Z","updated_at":"2014-10-07T21:59:47Z","html_url":"https://github.com/fluo-io/fluo-io.github.io/pull/25#discussion_r18552342","pull_request_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/pulls/25","_links":{"self":{"href":"https://api.github.com/repos/fluo-io/fluo-io.github.io/pulls/comments/18552342"},"html":{"href":"https://github.com/fluo-io/fluo-io.github.io/pull/25#discussion_r18552342"},"pull_request":{"href":"https://api.github.com/repos/fluo-io/fluo-io.github.io/pulls/25"}}},"pull_request":{"url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/pulls/25","id":22380268,"html_url":"https://github.com/fluo-io/fluo-io.github.io/pull/25","diff_url":"https://github.com/fluo-io/fluo-io.github.io/pull/25.diff","patch_url":"https://github.com/fluo-io/fluo-io.github.io/pull/25.patch","issue_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/issues/25","number":25,"state":"open","locked":false,"title":"Closes #7 Adding \"Current status of percolator implementation\"","user":{"login":"cjnolet","id":1242464,"avatar_url":"https://avatars.githubusercontent.com/u/1242464?v=2","gravatar_id":"","url":"https://api.github.com/users/cjnolet","html_url":"https://github.com/cjnolet","followers_url":"https://api.github.com/users/cjnolet/followers","following_url":"https://api.github.com/users/cjnolet/following{/other_user}","gists_url":"https://api.github.com/users/cjnolet/gists{/gist_id}","starred_url":"https://api.github.com/users/cjnolet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cjnolet/subscriptions","organizations_url":"https://api.github.com/users/cjnolet/orgs","repos_url":"https://api.github.com/users/cjnolet/repos","events_url":"https://api.github.com/users/cjnolet/events{/privacy}","received_events_url":"https://api.github.com/users/cjnolet/received_events","type":"User","site_admin":false},"body":"","created_at":"2014-10-07T20:14:59Z","updated_at":"2014-10-07T21:59:47Z","closed_at":null,"merged_at":null,"merge_commit_sha":"c9ef2d01c6f1b40c136acdb7461fd6478981d4c9","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/pulls/25/commits","review_comments_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/pulls/25/comments","review_comment_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/pulls/comments/{number}","comments_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/issues/25/comments","statuses_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/statuses/537eea11e9164c48abff0ffb1e92e97aa045cf86","head":{"label":"fluo-io:fw-7","ref":"fw-7","sha":"537eea11e9164c48abff0ffb1e92e97aa045cf86","user":{"login":"fluo-io","id":8184450,"avatar_url":"https://avatars.githubusercontent.com/u/8184450?v=2","gravatar_id":"","url":"https://api.github.com/users/fluo-io","html_url":"https://github.com/fluo-io","followers_url":"https://api.github.com/users/fluo-io/followers","following_url":"https://api.github.com/users/fluo-io/following{/other_user}","gists_url":"https://api.github.com/users/fluo-io/gists{/gist_id}","starred_url":"https://api.github.com/users/fluo-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fluo-io/subscriptions","organizations_url":"https://api.github.com/users/fluo-io/orgs","repos_url":"https://api.github.com/users/fluo-io/repos","events_url":"https://api.github.com/users/fluo-io/events{/privacy}","received_events_url":"https://api.github.com/users/fluo-io/received_events","type":"Organization","site_admin":false},"repo":{"id":24653619,"name":"fluo-io.github.io","full_name":"fluo-io/fluo-io.github.io","owner":{"login":"fluo-io","id":8184450,"avatar_url":"https://avatars.githubusercontent.com/u/8184450?v=2","gravatar_id":"","url":"https://api.github.com/users/fluo-io","html_url":"https://github.com/fluo-io","followers_url":"https://api.github.com/users/fluo-io/followers","following_url":"https://api.github.com/users/fluo-io/following{/other_user}","gists_url":"https://api.github.com/users/fluo-io/gists{/gist_id}","starred_url":"https://api.github.com/users/fluo-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fluo-io/subscriptions","organizations_url":"https://api.github.com/users/fluo-io/orgs","repos_url":"https://api.github.com/users/fluo-io/repos","events_url":"https://api.github.com/users/fluo-io/events{/privacy}","received_events_url":"https://api.github.com/users/fluo-io/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/fluo-io/fluo-io.github.io","description":"Fluo project website","fork":false,"url":"https://api.github.com/repos/fluo-io/fluo-io.github.io","forks_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/forks","keys_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/keys{/key_id}","collaborators_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/teams","hooks_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/hooks","issue_events_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/issues/events{/number}","events_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/events","assignees_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/assignees{/user}","branches_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/branches{/branch}","tags_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/tags","blobs_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/git/refs{/sha}","trees_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/git/trees{/sha}","statuses_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/statuses/{sha}","languages_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/languages","stargazers_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/stargazers","contributors_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/contributors","subscribers_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/subscribers","subscription_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/subscription","commits_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/commits{/sha}","git_commits_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/git/commits{/sha}","comments_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/comments{/number}","issue_comment_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/issues/comments/{number}","contents_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/contents/{+path}","compare_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/compare/{base}...{head}","merges_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/merges","archive_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/downloads","issues_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/issues{/number}","pulls_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/pulls{/number}","milestones_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/milestones{/number}","notifications_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/labels{/name}","releases_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/releases{/id}","created_at":"2014-09-30T20:06:15Z","updated_at":"2014-10-07T21:07:19Z","pushed_at":"2014-10-07T20:45:05Z","git_url":"git://github.com/fluo-io/fluo-io.github.io.git","ssh_url":"git@github.com:fluo-io/fluo-io.github.io.git","clone_url":"https://github.com/fluo-io/fluo-io.github.io.git","svn_url":"https://github.com/fluo-io/fluo-io.github.io","homepage":"http://fluo.io","size":377,"stargazers_count":1,"watchers_count":1,"language":"CSS","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":3,"mirror_url":null,"open_issues_count":5,"forks":3,"open_issues":5,"watchers":1,"default_branch":"master"}},"base":{"label":"fluo-io:master","ref":"master","sha":"e444c7aef7be6d0365a1e6cbf8c53721b50d2f10","user":{"login":"fluo-io","id":8184450,"avatar_url":"https://avatars.githubusercontent.com/u/8184450?v=2","gravatar_id":"","url":"https://api.github.com/users/fluo-io","html_url":"https://github.com/fluo-io","followers_url":"https://api.github.com/users/fluo-io/followers","following_url":"https://api.github.com/users/fluo-io/following{/other_user}","gists_url":"https://api.github.com/users/fluo-io/gists{/gist_id}","starred_url":"https://api.github.com/users/fluo-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fluo-io/subscriptions","organizations_url":"https://api.github.com/users/fluo-io/orgs","repos_url":"https://api.github.com/users/fluo-io/repos","events_url":"https://api.github.com/users/fluo-io/events{/privacy}","received_events_url":"https://api.github.com/users/fluo-io/received_events","type":"Organization","site_admin":false},"repo":{"id":24653619,"name":"fluo-io.github.io","full_name":"fluo-io/fluo-io.github.io","owner":{"login":"fluo-io","id":8184450,"avatar_url":"https://avatars.githubusercontent.com/u/8184450?v=2","gravatar_id":"","url":"https://api.github.com/users/fluo-io","html_url":"https://github.com/fluo-io","followers_url":"https://api.github.com/users/fluo-io/followers","following_url":"https://api.github.com/users/fluo-io/following{/other_user}","gists_url":"https://api.github.com/users/fluo-io/gists{/gist_id}","starred_url":"https://api.github.com/users/fluo-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fluo-io/subscriptions","organizations_url":"https://api.github.com/users/fluo-io/orgs","repos_url":"https://api.github.com/users/fluo-io/repos","events_url":"https://api.github.com/users/fluo-io/events{/privacy}","received_events_url":"https://api.github.com/users/fluo-io/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/fluo-io/fluo-io.github.io","description":"Fluo project website","fork":false,"url":"https://api.github.com/repos/fluo-io/fluo-io.github.io","forks_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/forks","keys_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/keys{/key_id}","collaborators_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/teams","hooks_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/hooks","issue_events_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/issues/events{/number}","events_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/events","assignees_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/assignees{/user}","branches_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/branches{/branch}","tags_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/tags","blobs_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/git/refs{/sha}","trees_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/git/trees{/sha}","statuses_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/statuses/{sha}","languages_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/languages","stargazers_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/stargazers","contributors_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/contributors","subscribers_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/subscribers","subscription_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/subscription","commits_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/commits{/sha}","git_commits_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/git/commits{/sha}","comments_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/comments{/number}","issue_comment_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/issues/comments/{number}","contents_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/contents/{+path}","compare_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/compare/{base}...{head}","merges_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/merges","archive_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/downloads","issues_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/issues{/number}","pulls_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/pulls{/number}","milestones_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/milestones{/number}","notifications_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/labels{/name}","releases_url":"https://api.github.com/repos/fluo-io/fluo-io.github.io/releases{/id}","created_at":"2014-09-30T20:06:15Z","updated_at":"2014-10-07T21:07:19Z","pushed_at":"2014-10-07T20:45:05Z","git_url":"git://github.com/fluo-io/fluo-io.github.io.git","ssh_url":"git@github.com:fluo-io/fluo-io.github.io.git","clone_url":"https://github.com/fluo-io/fluo-io.github.io.git","svn_url":"https://github.com/fluo-io/fluo-io.github.io","homepage":"http://fluo.io","size":377,"stargazers_count":1,"watchers_count":1,"language":"CSS","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":3,"mirror_url":null,"open_issues_count":5,"forks":3,"open_issues":5,"watchers":1,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/fluo-io/fluo-io.github.io/pulls/25"},"html":{"href":"https://github.com/fluo-io/fluo-io.github.io/pull/25"},"issue":{"href":"https://api.github.com/repos/fluo-io/fluo-io.github.io/issues/25"},"comments":{"href":"https://api.github.com/repos/fluo-io/fluo-io.github.io/issues/25/comments"},"review_comments":{"href":"https://api.github.com/repos/fluo-io/fluo-io.github.io/pulls/25/comments"},"review_comment":{"href":"https://api.github.com/repos/fluo-io/fluo-io.github.io/pulls/comments/{number}"},"commits":{"href":"https://api.github.com/repos/fluo-io/fluo-io.github.io/pulls/25/commits"},"statuses":{"href":"https://api.github.com/repos/fluo-io/fluo-io.github.io/statuses/537eea11e9164c48abff0ffb1e92e97aa045cf86"}}}}