GitHub PullRequestReviewCommentEvent at 2014-06-18 17:00:32
Repository: spark
Actor: etrain
Payload: {"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/13926606","id":13926606,"diff_hunk":"@@ -233,13 +234,73 @@ object DecisionTree extends Serializable with Logging {\n       algo: Algo,\n       impurity: Impurity,\n       maxDepth: Int): DecisionTreeModel = {\n-    val strategy = new Strategy(algo,impurity,maxDepth)\n-    new DecisionTree(strategy).train(input: RDD[LabeledPoint])\n+    val strategy = new Strategy(algo, impurity, maxDepth)\n+    // Converting from standard instance format to weighted input format for tree training\n+    val weightedInput = input.map(x => WeightedLabeledPoint(x.label, x.features))\n+    new DecisionTree(strategy).train(weightedInput: RDD[WeightedLabeledPoint])","path":"mllib/src/main/scala/org/apache/spark/mllib/tree/DecisionTree.scala","position":63,"original_position":63,"commit_id":"8e44ab81b3c7a019eb48586d75495b43987683fb","original_commit_id":"8e44ab81b3c7a019eb48586d75495b43987683fb","user":{"login":"etrain","id":1326181,"avatar_url":"https://avatars.githubusercontent.com/u/1326181?","gravatar_id":"183210d4aed0bfe86e0ff7fafd4a548d","url":"https://api.github.com/users/etrain","html_url":"https://github.com/etrain","followers_url":"https://api.github.com/users/etrain/followers","following_url":"https://api.github.com/users/etrain/following{/other_user}","gists_url":"https://api.github.com/users/etrain/gists{/gist_id}","starred_url":"https://api.github.com/users/etrain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/etrain/subscriptions","organizations_url":"https://api.github.com/users/etrain/orgs","repos_url":"https://api.github.com/users/etrain/repos","events_url":"https://api.github.com/users/etrain/events{/privacy}","received_events_url":"https://api.github.com/users/etrain/received_events","type":"User","site_admin":false},"body":"Not sure why you need to be explicit about the types here?","created_at":"2014-06-18T17:00:32Z","updated_at":"2014-06-18T17:00:32Z","html_url":"https://github.com/apache/spark/pull/886#discussion_r13926606","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/886","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/13926606"},"html":{"href":"https://github.com/apache/spark/pull/886#discussion_r13926606"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/886"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/886","id":16351362,"html_url":"https://github.com/apache/spark/pull/886","diff_url":"https://github.com/apache/spark/pull/886.diff","patch_url":"https://github.com/apache/spark/pull/886.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/886","number":886,"state":"open","title":"SPARK-1536: multiclass classification support for decision tree","user":{"login":"manishamde","id":1067223,"avatar_url":"https://avatars.githubusercontent.com/u/1067223?","gravatar_id":"341763202d03a38d0970e1007c009fc6","url":"https://api.github.com/users/manishamde","html_url":"https://github.com/manishamde","followers_url":"https://api.github.com/users/manishamde/followers","following_url":"https://api.github.com/users/manishamde/following{/other_user}","gists_url":"https://api.github.com/users/manishamde/gists{/gist_id}","starred_url":"https://api.github.com/users/manishamde/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/manishamde/subscriptions","organizations_url":"https://api.github.com/users/manishamde/orgs","repos_url":"https://api.github.com/users/manishamde/repos","events_url":"https://api.github.com/users/manishamde/events{/privacy}","received_events_url":"https://api.github.com/users/manishamde/received_events","type":"User","site_admin":false},"body":"The ability to perform multiclass classification is a big advantage for using decision trees and was a highly requested feature for mllib. This pull request adds multiclass classification support to the MLlib decision tree. It also adds sample weights support using WeightedLabeledPoint class for handling unbalanced datasets during classification. It will also support algorithms such as AdaBoost which requires instances to be weighted.\r\n\r\nIt handles the special case where the categorical variables cannot be ordered for multiclass classification and thus the optimizations used for speeding up binary classification cannot be directly used for multiclass classification with categorical variables. More specifically, for m categories in a categorical feature, it analyses all the ```2^(m-1) - 1``` categorical splits provided that #splits are less than the maxBins provided in the input. This condition will not be met for features with large number of categories -- using decision trees is not recommended for such datasets in general since the categorical features are favored over continuous features. Moreover, the user can use a combination of tricks (increasing bin size of the tree algorithms, use binary encoding for categorical features or use one-vs-all classification strategy) to avoid these constraints.\r\n\r\nThe new code is accompanied by unit tests and has also been tested on the iris and covtype datasets.\r\n\r\ncc: @mengxr, @etrain, @hirakendu, @atalwalkar, @srowen","created_at":"2014-05-27T01:31:00Z","updated_at":"2014-06-18T17:00:32Z","closed_at":null,"merged_at":null,"merge_commit_sha":"7ba44b6dc80082757e36eda6120b53add19dc2b3","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/886/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/886/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments/{number}","comments_url":"https://api.github.com/repos/apache/spark/issues/886/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/8e44ab81b3c7a019eb48586d75495b43987683fb","head":{"label":"manishamde:multiclass","ref":"multiclass","sha":"8e44ab81b3c7a019eb48586d75495b43987683fb","user":{"login":"manishamde","id":1067223,"avatar_url":"https://avatars.githubusercontent.com/u/1067223?","gravatar_id":"341763202d03a38d0970e1007c009fc6","url":"https://api.github.com/users/manishamde","html_url":"https://github.com/manishamde","followers_url":"https://api.github.com/users/manishamde/followers","following_url":"https://api.github.com/users/manishamde/following{/other_user}","gists_url":"https://api.github.com/users/manishamde/gists{/gist_id}","starred_url":"https://api.github.com/users/manishamde/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/manishamde/subscriptions","organizations_url":"https://api.github.com/users/manishamde/orgs","repos_url":"https://api.github.com/users/manishamde/repos","events_url":"https://api.github.com/users/manishamde/events{/privacy}","received_events_url":"https://api.github.com/users/manishamde/received_events","type":"User","site_admin":false},"repo":{"id":17230999,"name":"spark","full_name":"manishamde/spark","owner":{"login":"manishamde","id":1067223,"avatar_url":"https://avatars.githubusercontent.com/u/1067223?","gravatar_id":"341763202d03a38d0970e1007c009fc6","url":"https://api.github.com/users/manishamde","html_url":"https://github.com/manishamde","followers_url":"https://api.github.com/users/manishamde/followers","following_url":"https://api.github.com/users/manishamde/following{/other_user}","gists_url":"https://api.github.com/users/manishamde/gists{/gist_id}","starred_url":"https://api.github.com/users/manishamde/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/manishamde/subscriptions","organizations_url":"https://api.github.com/users/manishamde/orgs","repos_url":"https://api.github.com/users/manishamde/repos","events_url":"https://api.github.com/users/manishamde/events{/privacy}","received_events_url":"https://api.github.com/users/manishamde/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/manishamde/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/manishamde/spark","forks_url":"https://api.github.com/repos/manishamde/spark/forks","keys_url":"https://api.github.com/repos/manishamde/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/manishamde/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/manishamde/spark/teams","hooks_url":"https://api.github.com/repos/manishamde/spark/hooks","issue_events_url":"https://api.github.com/repos/manishamde/spark/issues/events{/number}","events_url":"https://api.github.com/repos/manishamde/spark/events","assignees_url":"https://api.github.com/repos/manishamde/spark/assignees{/user}","branches_url":"https://api.github.com/repos/manishamde/spark/branches{/branch}","tags_url":"https://api.github.com/repos/manishamde/spark/tags","blobs_url":"https://api.github.com/repos/manishamde/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/manishamde/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/manishamde/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/manishamde/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/manishamde/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/manishamde/spark/languages","stargazers_url":"https://api.github.com/repos/manishamde/spark/stargazers","contributors_url":"https://api.github.com/repos/manishamde/spark/contributors","subscribers_url":"https://api.github.com/repos/manishamde/spark/subscribers","subscription_url":"https://api.github.com/repos/manishamde/spark/subscription","commits_url":"https://api.github.com/repos/manishamde/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/manishamde/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/manishamde/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/manishamde/spark/issues/comments/{number}","contents_url":"https://api.github.com/repos/manishamde/spark/contents/{+path}","compare_url":"https://api.github.com/repos/manishamde/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/manishamde/spark/merges","archive_url":"https://api.github.com/repos/manishamde/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/manishamde/spark/downloads","issues_url":"https://api.github.com/repos/manishamde/spark/issues{/number}","pulls_url":"https://api.github.com/repos/manishamde/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/manishamde/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/manishamde/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/manishamde/spark/labels{/name}","releases_url":"https://api.github.com/repos/manishamde/spark/releases{/id}","created_at":"2014-02-27T00:29:08Z","updated_at":"2014-06-16T06:59:14Z","pushed_at":"2014-06-16T06:59:14Z","git_url":"git://github.com/manishamde/spark.git","ssh_url":"git@github.com:manishamde/spark.git","clone_url":"https://github.com/manishamde/spark.git","svn_url":"https://github.com/manishamde/spark","homepage":null,"size":68926,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"56c771cb2d00a5843c391ae6561536ee46e535d4","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?","gravatar_id":"a676c0bf448fcd49f588249ead719b4c","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?","gravatar_id":"a676c0bf448fcd49f588249ead719b4c","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments/{number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","created_at":"2014-02-25T08:00:08Z","updated_at":"2014-06-18T16:22:48Z","pushed_at":"2014-06-18T05:04:49Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":1729817,"stargazers_count":634,"watchers_count":634,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"forks_count":651,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":178,"forks":651,"open_issues":178,"watchers":634,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/886"},"html":{"href":"https://github.com/apache/spark/pull/886"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/886"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/886/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/886/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/{number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/886/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/8e44ab81b3c7a019eb48586d75495b43987683fb"}}}}