GitHub PullRequestReviewCommentEvent at 2014-10-07 13:46:56
Repository: adam
Actor: fnothaft
Payload: {"action":"created","comment":{"url":"https://api.github.com/repos/bigdatagenomics/adam/pulls/comments/18519689","id":18519689,"diff_hunk":"@@ -0,0 +1,45 @@\n+/**\n+ * Licensed to Big Data Genomics (BDG) under one\n+ * or more contributor license agreements.  See the NOTICE file\n+ * distributed with this work for additional information\n+ * regarding copyright ownership.  The BDG licenses this file\n+ * to you under the Apache License, Version 2.0 (the\n+ * \"License\"); you may not use this file except in compliance\n+ * with the License.  You may obtain a copy of the License at\n+ *\n+ *     http://www.apache.org/licenses/LICENSE-2.0\n+ *\n+ * Unless required by applicable law or agreed to in writing, software\n+ * distributed under the License is distributed on an \"AS IS\" BASIS,\n+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n+ * See the License for the specific language governing permissions and\n+ * limitations under the License.\n+ */\n+package org.bdgenomics.adam.algorithms.distributions\n+\n+import scala.math.abs\n+\n+object CategoricalDistribution {\n+\n+  def softmax(likelihoods: Array[Double]): CategoricalDistribution = {\n+    val sum = likelihoods.sum\n+    CategoricalDistribution(likelihoods.map(l => l / sum))\n+  }\n+}\n+\n+case class CategoricalDistribution(probabilities: Array[Double])","path":"adam-core/src/main/scala/org/bdgenomics/adam/algorithms/distributions/CategoricalDistribution.scala","position":30,"original_position":30,"commit_id":"e6d1be662fd8812fc2f12473abbd987a68216c24","original_commit_id":"e6d1be662fd8812fc2f12473abbd987a68216c24","user":{"login":"fnothaft","id":3752466,"avatar_url":"https://avatars.githubusercontent.com/u/3752466?v=2","gravatar_id":"","url":"https://api.github.com/users/fnothaft","html_url":"https://github.com/fnothaft","followers_url":"https://api.github.com/users/fnothaft/followers","following_url":"https://api.github.com/users/fnothaft/following{/other_user}","gists_url":"https://api.github.com/users/fnothaft/gists{/gist_id}","starred_url":"https://api.github.com/users/fnothaft/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fnothaft/subscriptions","organizations_url":"https://api.github.com/users/fnothaft/orgs","repos_url":"https://api.github.com/users/fnothaft/repos","events_url":"https://api.github.com/users/fnothaft/events{/privacy}","received_events_url":"https://api.github.com/users/fnothaft/received_events","type":"User","site_admin":false},"body":"I suppose I don't see what that would that buy you beyond using an array? Here we're doing invariant checking (category probabilities must sum to 1) and we have the softmax helper function for building the distribution. Neither of them are terribly difficult to implement, but if we don't implement them, the code winds up reappearing everywhere.","created_at":"2014-10-07T13:46:56Z","updated_at":"2014-10-07T13:46:56Z","html_url":"https://github.com/bigdatagenomics/adam/pull/402#discussion_r18519689","pull_request_url":"https://api.github.com/repos/bigdatagenomics/adam/pulls/402","_links":{"self":{"href":"https://api.github.com/repos/bigdatagenomics/adam/pulls/comments/18519689"},"html":{"href":"https://github.com/bigdatagenomics/adam/pull/402#discussion_r18519689"},"pull_request":{"href":"https://api.github.com/repos/bigdatagenomics/adam/pulls/402"}}},"pull_request":{"url":"https://api.github.com/repos/bigdatagenomics/adam/pulls/402","id":22263253,"html_url":"https://github.com/bigdatagenomics/adam/pull/402","diff_url":"https://github.com/bigdatagenomics/adam/pull/402.diff","patch_url":"https://github.com/bigdatagenomics/adam/pull/402.patch","issue_url":"https://api.github.com/repos/bigdatagenomics/adam/issues/402","number":402,"state":"open","locked":false,"title":"[ADAM-401] Adding read error correction.","user":{"login":"fnothaft","id":3752466,"avatar_url":"https://avatars.githubusercontent.com/u/3752466?v=2","gravatar_id":"","url":"https://api.github.com/users/fnothaft","html_url":"https://github.com/fnothaft","followers_url":"https://api.github.com/users/fnothaft/followers","following_url":"https://api.github.com/users/fnothaft/following{/other_user}","gists_url":"https://api.github.com/users/fnothaft/gists{/gist_id}","starred_url":"https://api.github.com/users/fnothaft/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fnothaft/subscriptions","organizations_url":"https://api.github.com/users/fnothaft/orgs","repos_url":"https://api.github.com/users/fnothaft/repos","events_url":"https://api.github.com/users/fnothaft/events{/privacy}","received_events_url":"https://api.github.com/users/fnothaft/received_events","type":"User","site_admin":false},"body":"Fixes #401, by adding a read error correction. This algorithm follows the following general approach:\r\n\r\n1. Reads are cut into _q_-mers, which are quality score weighted _k_-mers\r\n2. A mixture of gammas is fit to the data, where the mixture contains _m_ + 1 components (_m_ = ploidy). These components include an \"error\" class, _m_ - 1 heterozygous classes, and 1 homozygous class.\r\n3. To each _q_-mer, we assign a probability that it _is_ a \"true\" _q_-mer, given the above mixture model.\r\n4. To each read, we assign base probabilities by looking at the truth probabilities of all _k_-mers that cover the base.\r\n5. From this, we collect MLEs for the transition probabilities of each base. Our covariates include the measured quality score of the base, the position of the base in the read, and the sequenced base.\r\n6. These transition probabilities are used as priors for the fixing process. For each read, we compute the likelihoods of each base using the _q_-mer \"truth\" probabilities from step 3. We first compensate by the priors. After this, we try to fix bases in the read. We employ a coordinate ascent method, using the probabilities we calculate at each step as conjugate priors for each new fixing step.\r\n\r\nOn a small test, this appears to reduce the error rate by about an order of magnitude, as is expected. I'll be doing more tests this week, and will be putting together a writeup describing the method.","created_at":"2014-10-06T03:00:45Z","updated_at":"2014-10-07T13:46:56Z","closed_at":null,"merged_at":null,"merge_commit_sha":"d082ceec6edfb40b8f0ebb04705d52e494683c1d","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/bigdatagenomics/adam/pulls/402/commits","review_comments_url":"https://api.github.com/repos/bigdatagenomics/adam/pulls/402/comments","review_comment_url":"https://api.github.com/repos/bigdatagenomics/adam/pulls/comments/{number}","comments_url":"https://api.github.com/repos/bigdatagenomics/adam/issues/402/comments","statuses_url":"https://api.github.com/repos/bigdatagenomics/adam/statuses/e6d1be662fd8812fc2f12473abbd987a68216c24","head":{"label":"fnothaft:read-error-correction","ref":"read-error-correction","sha":"e6d1be662fd8812fc2f12473abbd987a68216c24","user":{"login":"fnothaft","id":3752466,"avatar_url":"https://avatars.githubusercontent.com/u/3752466?v=2","gravatar_id":"","url":"https://api.github.com/users/fnothaft","html_url":"https://github.com/fnothaft","followers_url":"https://api.github.com/users/fnothaft/followers","following_url":"https://api.github.com/users/fnothaft/following{/other_user}","gists_url":"https://api.github.com/users/fnothaft/gists{/gist_id}","starred_url":"https://api.github.com/users/fnothaft/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fnothaft/subscriptions","organizations_url":"https://api.github.com/users/fnothaft/orgs","repos_url":"https://api.github.com/users/fnothaft/repos","events_url":"https://api.github.com/users/fnothaft/events{/privacy}","received_events_url":"https://api.github.com/users/fnothaft/received_events","type":"User","site_admin":false},"repo":{"id":16842658,"name":"adam","full_name":"fnothaft/adam","owner":{"login":"fnothaft","id":3752466,"avatar_url":"https://avatars.githubusercontent.com/u/3752466?v=2","gravatar_id":"","url":"https://api.github.com/users/fnothaft","html_url":"https://github.com/fnothaft","followers_url":"https://api.github.com/users/fnothaft/followers","following_url":"https://api.github.com/users/fnothaft/following{/other_user}","gists_url":"https://api.github.com/users/fnothaft/gists{/gist_id}","starred_url":"https://api.github.com/users/fnothaft/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fnothaft/subscriptions","organizations_url":"https://api.github.com/users/fnothaft/orgs","repos_url":"https://api.github.com/users/fnothaft/repos","events_url":"https://api.github.com/users/fnothaft/events{/privacy}","received_events_url":"https://api.github.com/users/fnothaft/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/fnothaft/adam","description":"A genomics processing engine and specialized file format built using Apache Avro, Apache Spark and Parquet. Apache 2 licensed.","fork":true,"url":"https://api.github.com/repos/fnothaft/adam","forks_url":"https://api.github.com/repos/fnothaft/adam/forks","keys_url":"https://api.github.com/repos/fnothaft/adam/keys{/key_id}","collaborators_url":"https://api.github.com/repos/fnothaft/adam/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/fnothaft/adam/teams","hooks_url":"https://api.github.com/repos/fnothaft/adam/hooks","issue_events_url":"https://api.github.com/repos/fnothaft/adam/issues/events{/number}","events_url":"https://api.github.com/repos/fnothaft/adam/events","assignees_url":"https://api.github.com/repos/fnothaft/adam/assignees{/user}","branches_url":"https://api.github.com/repos/fnothaft/adam/branches{/branch}","tags_url":"https://api.github.com/repos/fnothaft/adam/tags","blobs_url":"https://api.github.com/repos/fnothaft/adam/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/fnothaft/adam/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/fnothaft/adam/git/refs{/sha}","trees_url":"https://api.github.com/repos/fnothaft/adam/git/trees{/sha}","statuses_url":"https://api.github.com/repos/fnothaft/adam/statuses/{sha}","languages_url":"https://api.github.com/repos/fnothaft/adam/languages","stargazers_url":"https://api.github.com/repos/fnothaft/adam/stargazers","contributors_url":"https://api.github.com/repos/fnothaft/adam/contributors","subscribers_url":"https://api.github.com/repos/fnothaft/adam/subscribers","subscription_url":"https://api.github.com/repos/fnothaft/adam/subscription","commits_url":"https://api.github.com/repos/fnothaft/adam/commits{/sha}","git_commits_url":"https://api.github.com/repos/fnothaft/adam/git/commits{/sha}","comments_url":"https://api.github.com/repos/fnothaft/adam/comments{/number}","issue_comment_url":"https://api.github.com/repos/fnothaft/adam/issues/comments/{number}","contents_url":"https://api.github.com/repos/fnothaft/adam/contents/{+path}","compare_url":"https://api.github.com/repos/fnothaft/adam/compare/{base}...{head}","merges_url":"https://api.github.com/repos/fnothaft/adam/merges","archive_url":"https://api.github.com/repos/fnothaft/adam/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/fnothaft/adam/downloads","issues_url":"https://api.github.com/repos/fnothaft/adam/issues{/number}","pulls_url":"https://api.github.com/repos/fnothaft/adam/pulls{/number}","milestones_url":"https://api.github.com/repos/fnothaft/adam/milestones{/number}","notifications_url":"https://api.github.com/repos/fnothaft/adam/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/fnothaft/adam/labels{/name}","releases_url":"https://api.github.com/repos/fnothaft/adam/releases{/id}","created_at":"2014-02-14T16:25:30Z","updated_at":"2014-06-24T16:52:53Z","pushed_at":"2014-10-06T02:01:09Z","git_url":"git://github.com/fnothaft/adam.git","ssh_url":"git@github.com:fnothaft/adam.git","clone_url":"https://github.com/fnothaft/adam.git","svn_url":"https://github.com/fnothaft/adam","homepage":"","size":5481,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"bigdatagenomics:master","ref":"master","sha":"99e6f2d824bf9b6afdaf1b0fe1e1e14daf9c45aa","user":{"login":"bigdatagenomics","id":5839438,"avatar_url":"https://avatars.githubusercontent.com/u/5839438?v=2","gravatar_id":"","url":"https://api.github.com/users/bigdatagenomics","html_url":"https://github.com/bigdatagenomics","followers_url":"https://api.github.com/users/bigdatagenomics/followers","following_url":"https://api.github.com/users/bigdatagenomics/following{/other_user}","gists_url":"https://api.github.com/users/bigdatagenomics/gists{/gist_id}","starred_url":"https://api.github.com/users/bigdatagenomics/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bigdatagenomics/subscriptions","organizations_url":"https://api.github.com/users/bigdatagenomics/orgs","repos_url":"https://api.github.com/users/bigdatagenomics/repos","events_url":"https://api.github.com/users/bigdatagenomics/events{/privacy}","received_events_url":"https://api.github.com/users/bigdatagenomics/received_events","type":"Organization","site_admin":false},"repo":{"id":14541530,"name":"adam","full_name":"bigdatagenomics/adam","owner":{"login":"bigdatagenomics","id":5839438,"avatar_url":"https://avatars.githubusercontent.com/u/5839438?v=2","gravatar_id":"","url":"https://api.github.com/users/bigdatagenomics","html_url":"https://github.com/bigdatagenomics","followers_url":"https://api.github.com/users/bigdatagenomics/followers","following_url":"https://api.github.com/users/bigdatagenomics/following{/other_user}","gists_url":"https://api.github.com/users/bigdatagenomics/gists{/gist_id}","starred_url":"https://api.github.com/users/bigdatagenomics/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bigdatagenomics/subscriptions","organizations_url":"https://api.github.com/users/bigdatagenomics/orgs","repos_url":"https://api.github.com/users/bigdatagenomics/repos","events_url":"https://api.github.com/users/bigdatagenomics/events{/privacy}","received_events_url":"https://api.github.com/users/bigdatagenomics/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/bigdatagenomics/adam","description":"A genomics processing engine and specialized file format built using Apache Avro, Apache Spark and Parquet. Apache 2 licensed.","fork":false,"url":"https://api.github.com/repos/bigdatagenomics/adam","forks_url":"https://api.github.com/repos/bigdatagenomics/adam/forks","keys_url":"https://api.github.com/repos/bigdatagenomics/adam/keys{/key_id}","collaborators_url":"https://api.github.com/repos/bigdatagenomics/adam/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/bigdatagenomics/adam/teams","hooks_url":"https://api.github.com/repos/bigdatagenomics/adam/hooks","issue_events_url":"https://api.github.com/repos/bigdatagenomics/adam/issues/events{/number}","events_url":"https://api.github.com/repos/bigdatagenomics/adam/events","assignees_url":"https://api.github.com/repos/bigdatagenomics/adam/assignees{/user}","branches_url":"https://api.github.com/repos/bigdatagenomics/adam/branches{/branch}","tags_url":"https://api.github.com/repos/bigdatagenomics/adam/tags","blobs_url":"https://api.github.com/repos/bigdatagenomics/adam/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/bigdatagenomics/adam/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/bigdatagenomics/adam/git/refs{/sha}","trees_url":"https://api.github.com/repos/bigdatagenomics/adam/git/trees{/sha}","statuses_url":"https://api.github.com/repos/bigdatagenomics/adam/statuses/{sha}","languages_url":"https://api.github.com/repos/bigdatagenomics/adam/languages","stargazers_url":"https://api.github.com/repos/bigdatagenomics/adam/stargazers","contributors_url":"https://api.github.com/repos/bigdatagenomics/adam/contributors","subscribers_url":"https://api.github.com/repos/bigdatagenomics/adam/subscribers","subscription_url":"https://api.github.com/repos/bigdatagenomics/adam/subscription","commits_url":"https://api.github.com/repos/bigdatagenomics/adam/commits{/sha}","git_commits_url":"https://api.github.com/repos/bigdatagenomics/adam/git/commits{/sha}","comments_url":"https://api.github.com/repos/bigdatagenomics/adam/comments{/number}","issue_comment_url":"https://api.github.com/repos/bigdatagenomics/adam/issues/comments/{number}","contents_url":"https://api.github.com/repos/bigdatagenomics/adam/contents/{+path}","compare_url":"https://api.github.com/repos/bigdatagenomics/adam/compare/{base}...{head}","merges_url":"https://api.github.com/repos/bigdatagenomics/adam/merges","archive_url":"https://api.github.com/repos/bigdatagenomics/adam/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/bigdatagenomics/adam/downloads","issues_url":"https://api.github.com/repos/bigdatagenomics/adam/issues{/number}","pulls_url":"https://api.github.com/repos/bigdatagenomics/adam/pulls{/number}","milestones_url":"https://api.github.com/repos/bigdatagenomics/adam/milestones{/number}","notifications_url":"https://api.github.com/repos/bigdatagenomics/adam/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/bigdatagenomics/adam/labels{/name}","releases_url":"https://api.github.com/repos/bigdatagenomics/adam/releases{/id}","created_at":"2013-11-19T23:47:57Z","updated_at":"2014-10-07T12:12:58Z","pushed_at":"2014-10-01T21:26:53Z","git_url":"git://github.com/bigdatagenomics/adam.git","ssh_url":"git@github.com:bigdatagenomics/adam.git","clone_url":"https://github.com/bigdatagenomics/adam.git","svn_url":"https://github.com/bigdatagenomics/adam","homepage":"http://bdgenomics.org/","size":58008,"stargazers_count":147,"watchers_count":147,"language":"Scala","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":47,"mirror_url":null,"open_issues_count":46,"forks":47,"open_issues":46,"watchers":147,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/bigdatagenomics/adam/pulls/402"},"html":{"href":"https://github.com/bigdatagenomics/adam/pull/402"},"issue":{"href":"https://api.github.com/repos/bigdatagenomics/adam/issues/402"},"comments":{"href":"https://api.github.com/repos/bigdatagenomics/adam/issues/402/comments"},"review_comments":{"href":"https://api.github.com/repos/bigdatagenomics/adam/pulls/402/comments"},"review_comment":{"href":"https://api.github.com/repos/bigdatagenomics/adam/pulls/comments/{number}"},"commits":{"href":"https://api.github.com/repos/bigdatagenomics/adam/pulls/402/commits"},"statuses":{"href":"https://api.github.com/repos/bigdatagenomics/adam/statuses/e6d1be662fd8812fc2f12473abbd987a68216c24"}}}}