GitHub PullRequestReviewCommentEvent at 2014-10-16 00:54:29
Repository: cdap-spark-guide
Actor: abaranau
Payload: {"action":"created","comment":{"url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/pulls/comments/18933431","id":18933431,"diff_hunk":"@@ -0,0 +1,267 @@\n+Iterative Data Processing with Apache Spark (Beta: Standalone Only)\n+====================================================================\n+\n+`Apache Spark <https://spark.apache.org/>`_ is very popular when it comes to in-memory cluster computing for Hadoop. In this guide, you will learn how to run Apache Spark programs with CDAP.\n+\n+What You Will Build\n+-------------------\n+\n+You will build a CDAP application that exposes REST API to take in web page’s backlinks information and serve out the PageRank for the known web pages. You will:\n+\n+* Build a CDAP `Spark <http://docs.cdap.io/cdap/2.5.0/en/dev-guide.html#spark-beta-standalone-cdap-only>`_ program that computes PageRank of the web pages\n+* Build a `Service <http://docs.cdap.io/cdap/current/en/dev-guide.html#services>`_ to accept backlinks data and serve PageRank computation results over HTTP\n+* Persist and retrieve data in `Dataset <http://docs.cdap.io/cdap/current/en/dev-guide.html#datasets>`_ and use them as input for a Spark program.\n+\n+What You Will Need\n+------------------\n+\n+* `JDK 6 or JDK 7 <http://www.oracle.com/technetwork/java/javase/downloads/index.html>`_\n+* `Apache Maven 3.0+ <http://maven.apache.org/>`_\n+* `CDAP SDK <http://docs.cdap.io/cdap/current/en/getstarted.html#download-and-setup>`_\n+\n+Let’s Build It!\n+---------------\n+\n+Following sections will guide you through building an application from scratch. \n+If you are interested in deploying and running the application right away, you \n+can clone its source code and binaries from this github repository. In that case feel \n+free to skip the next two sections and jump right to Build & Run section.\n+\n+Application Design\n+~~~~~~~~~~~~~~~~~~\n+\n+Backlinks data is sent to PageRankService over HTTP (e.g. by web crawler as it processes web pages). The service persists the data into backLinks dataset upon receiving. The PageRank for known pages is computed periodically by a PageRankProgram. The program uses backLinks dataset as an input and persists results in pageRanks dataset. \n+\n+PageRankService then uses pageRanks dataset to serve PageRank for a given URL over HTTP.\n+\n+\n+|(AppDesign)|\n+\n+Implementation\n+~~~~~~~~~~~~~~\n+\n+The first step is to get our application structure set up.  We will use a standard Maven project structure for all of the source code files::\n+\n+  ./pom.xml\n+  ./README.rst\n+  ./src/main/java/co/cask/cdap/guides/pagerank/PageRankApp.java\n+  ./src/main/java/co/cask/cdap/guides/pagerank/PageRankSpark.java\n+  ./src/main/java/co/cask/cdap/guides/pagerank/BackLinksHandler.java\n+  ./src/main/java/co/cask/cdap/guides/pagerank/PageRankHandler.java\n+  ./src/main/scala/co/cask/cdap/guides/pagerank/PageRankProgram.scala\n+\n+\n+The application is identified by the PageRankApp class.  This class extends \n+`AbstractApplication <http://docs.cdap.io/cdap/2.5.0/en/javadocs/co/cask/cdap/api/app/AbstractApplication.html>`_,\n+and overrides the configure() method in order to define all of the application components:\n+\n+.. code:: java\n+\n+  public class PageRankApp extends AbstractApplication {\n+  \n+    @Override\n+    public void configure() {\n+      setName(\"PageRankApplication\");\n+      addSpark(new PageRankSpark());\n+      addService(\"PageRankService\", new ImmutableList.Builder<HttpServiceHandler>()\n+        .add(new BackLinksHandler())\n+        .add(new PageRankHandler())\n+        .build());\n+      try {\n+        ObjectStores.createObjectStore(getConfigurer(), \"backLinks\", String.class);\n+        ObjectStores.createObjectStore(getConfigurer(), \"pageRanks\", Double.class);\n+      } catch (UnsupportedTypeException e) {\n+        throw new RuntimeException(\"Will never happen: all classes above are supported\", e);\n+      }\n+    }\n+  }\n+\n+\n+In this example we’ll use Scala to write a Spark program. You’ll need to add scala as a dependency in your maven pom.xml. Also, you will need to set property app.main.class to the name of your application class for the application jar to have proper format:","path":"README.rst","position":80,"original_position":80,"commit_id":"0265a5d30401e9e3cfd92b6fd6d65d4785df594b","original_commit_id":"0265a5d30401e9e3cfd92b6fd6d65d4785df594b","user":{"login":"abaranau","id":434133,"avatar_url":"https://avatars.githubusercontent.com/u/434133?v=2","gravatar_id":"","url":"https://api.github.com/users/abaranau","html_url":"https://github.com/abaranau","followers_url":"https://api.github.com/users/abaranau/followers","following_url":"https://api.github.com/users/abaranau/following{/other_user}","gists_url":"https://api.github.com/users/abaranau/gists{/gist_id}","starred_url":"https://api.github.com/users/abaranau/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/abaranau/subscriptions","organizations_url":"https://api.github.com/users/abaranau/orgs","repos_url":"https://api.github.com/users/abaranau/repos","events_url":"https://api.github.com/users/abaranau/events{/privacy}","received_events_url":"https://api.github.com/users/abaranau/received_events","type":"User","site_admin":false},"body":"\"Also, you will need to set property app.main.class... \" - no need.","created_at":"2014-10-16T00:54:29Z","updated_at":"2014-10-16T00:54:29Z","html_url":"https://github.com/cdap-guides/cdap-spark-guide/pull/2#discussion_r18933431","pull_request_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/pulls/2","_links":{"self":{"href":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/pulls/comments/18933431"},"html":{"href":"https://github.com/cdap-guides/cdap-spark-guide/pull/2#discussion_r18933431"},"pull_request":{"href":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/pulls/2"}}},"pull_request":{"url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/pulls/2","id":22825742,"html_url":"https://github.com/cdap-guides/cdap-spark-guide/pull/2","diff_url":"https://github.com/cdap-guides/cdap-spark-guide/pull/2.diff","patch_url":"https://github.com/cdap-guides/cdap-spark-guide/pull/2.patch","issue_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/issues/2","number":2,"state":"open","locked":false,"title":"Documentation for Spark Guide","user":{"login":"rohitsinha54","id":1472867,"avatar_url":"https://avatars.githubusercontent.com/u/1472867?v=2","gravatar_id":"","url":"https://api.github.com/users/rohitsinha54","html_url":"https://github.com/rohitsinha54","followers_url":"https://api.github.com/users/rohitsinha54/followers","following_url":"https://api.github.com/users/rohitsinha54/following{/other_user}","gists_url":"https://api.github.com/users/rohitsinha54/gists{/gist_id}","starred_url":"https://api.github.com/users/rohitsinha54/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rohitsinha54/subscriptions","organizations_url":"https://api.github.com/users/rohitsinha54/orgs","repos_url":"https://api.github.com/users/rohitsinha54/repos","events_url":"https://api.github.com/users/rohitsinha54/events{/privacy}","received_events_url":"https://api.github.com/users/rohitsinha54/received_events","type":"User","site_admin":false},"body":"Will add image separately.\r\n\r\nWorking on replacing the curl command with CLI so omit those. ","created_at":"2014-10-16T00:34:10Z","updated_at":"2014-10-16T00:54:29Z","closed_at":null,"merged_at":null,"merge_commit_sha":"2f88bb2366a3fd118f8adae04bbe3f53d6cc89c2","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/pulls/2/commits","review_comments_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/pulls/2/comments","review_comment_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/pulls/comments/{number}","comments_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/issues/2/comments","statuses_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/statuses/0265a5d30401e9e3cfd92b6fd6d65d4785df594b","head":{"label":"cdap-guides:feature/docs","ref":"feature/docs","sha":"0265a5d30401e9e3cfd92b6fd6d65d4785df594b","user":{"login":"cdap-guides","id":9012786,"avatar_url":"https://avatars.githubusercontent.com/u/9012786?v=2","gravatar_id":"","url":"https://api.github.com/users/cdap-guides","html_url":"https://github.com/cdap-guides","followers_url":"https://api.github.com/users/cdap-guides/followers","following_url":"https://api.github.com/users/cdap-guides/following{/other_user}","gists_url":"https://api.github.com/users/cdap-guides/gists{/gist_id}","starred_url":"https://api.github.com/users/cdap-guides/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cdap-guides/subscriptions","organizations_url":"https://api.github.com/users/cdap-guides/orgs","repos_url":"https://api.github.com/users/cdap-guides/repos","events_url":"https://api.github.com/users/cdap-guides/events{/privacy}","received_events_url":"https://api.github.com/users/cdap-guides/received_events","type":"User","site_admin":false},"repo":{"id":25223789,"name":"cdap-spark-guide","full_name":"cdap-guides/cdap-spark-guide","owner":{"login":"cdap-guides","id":9012786,"avatar_url":"https://avatars.githubusercontent.com/u/9012786?v=2","gravatar_id":"","url":"https://api.github.com/users/cdap-guides","html_url":"https://github.com/cdap-guides","followers_url":"https://api.github.com/users/cdap-guides/followers","following_url":"https://api.github.com/users/cdap-guides/following{/other_user}","gists_url":"https://api.github.com/users/cdap-guides/gists{/gist_id}","starred_url":"https://api.github.com/users/cdap-guides/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cdap-guides/subscriptions","organizations_url":"https://api.github.com/users/cdap-guides/orgs","repos_url":"https://api.github.com/users/cdap-guides/repos","events_url":"https://api.github.com/users/cdap-guides/events{/privacy}","received_events_url":"https://api.github.com/users/cdap-guides/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/cdap-guides/cdap-spark-guide","description":"","fork":false,"url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide","forks_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/forks","keys_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/keys{/key_id}","collaborators_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/teams","hooks_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/hooks","issue_events_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/issues/events{/number}","events_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/events","assignees_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/assignees{/user}","branches_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/branches{/branch}","tags_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/tags","blobs_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/git/refs{/sha}","trees_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/git/trees{/sha}","statuses_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/statuses/{sha}","languages_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/languages","stargazers_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/stargazers","contributors_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/contributors","subscribers_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/subscribers","subscription_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/subscription","commits_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/commits{/sha}","git_commits_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/git/commits{/sha}","comments_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/comments{/number}","issue_comment_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/issues/comments/{number}","contents_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/contents/{+path}","compare_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/compare/{base}...{head}","merges_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/merges","archive_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/downloads","issues_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/issues{/number}","pulls_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/pulls{/number}","milestones_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/milestones{/number}","notifications_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/labels{/name}","releases_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/releases{/id}","created_at":"2014-10-14T19:51:33Z","updated_at":"2014-10-15T04:14:06Z","pushed_at":"2014-10-16T00:47:05Z","git_url":"git://github.com/cdap-guides/cdap-spark-guide.git","ssh_url":"git@github.com:cdap-guides/cdap-spark-guide.git","clone_url":"https://github.com/cdap-guides/cdap-spark-guide.git","svn_url":"https://github.com/cdap-guides/cdap-spark-guide","homepage":null,"size":0,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"develop"}},"base":{"label":"cdap-guides:develop","ref":"develop","sha":"2b8b75641d5f89c3614d8d4120b1d4ea968cf210","user":{"login":"cdap-guides","id":9012786,"avatar_url":"https://avatars.githubusercontent.com/u/9012786?v=2","gravatar_id":"","url":"https://api.github.com/users/cdap-guides","html_url":"https://github.com/cdap-guides","followers_url":"https://api.github.com/users/cdap-guides/followers","following_url":"https://api.github.com/users/cdap-guides/following{/other_user}","gists_url":"https://api.github.com/users/cdap-guides/gists{/gist_id}","starred_url":"https://api.github.com/users/cdap-guides/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cdap-guides/subscriptions","organizations_url":"https://api.github.com/users/cdap-guides/orgs","repos_url":"https://api.github.com/users/cdap-guides/repos","events_url":"https://api.github.com/users/cdap-guides/events{/privacy}","received_events_url":"https://api.github.com/users/cdap-guides/received_events","type":"User","site_admin":false},"repo":{"id":25223789,"name":"cdap-spark-guide","full_name":"cdap-guides/cdap-spark-guide","owner":{"login":"cdap-guides","id":9012786,"avatar_url":"https://avatars.githubusercontent.com/u/9012786?v=2","gravatar_id":"","url":"https://api.github.com/users/cdap-guides","html_url":"https://github.com/cdap-guides","followers_url":"https://api.github.com/users/cdap-guides/followers","following_url":"https://api.github.com/users/cdap-guides/following{/other_user}","gists_url":"https://api.github.com/users/cdap-guides/gists{/gist_id}","starred_url":"https://api.github.com/users/cdap-guides/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cdap-guides/subscriptions","organizations_url":"https://api.github.com/users/cdap-guides/orgs","repos_url":"https://api.github.com/users/cdap-guides/repos","events_url":"https://api.github.com/users/cdap-guides/events{/privacy}","received_events_url":"https://api.github.com/users/cdap-guides/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/cdap-guides/cdap-spark-guide","description":"","fork":false,"url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide","forks_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/forks","keys_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/keys{/key_id}","collaborators_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/teams","hooks_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/hooks","issue_events_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/issues/events{/number}","events_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/events","assignees_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/assignees{/user}","branches_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/branches{/branch}","tags_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/tags","blobs_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/git/refs{/sha}","trees_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/git/trees{/sha}","statuses_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/statuses/{sha}","languages_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/languages","stargazers_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/stargazers","contributors_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/contributors","subscribers_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/subscribers","subscription_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/subscription","commits_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/commits{/sha}","git_commits_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/git/commits{/sha}","comments_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/comments{/number}","issue_comment_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/issues/comments/{number}","contents_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/contents/{+path}","compare_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/compare/{base}...{head}","merges_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/merges","archive_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/downloads","issues_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/issues{/number}","pulls_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/pulls{/number}","milestones_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/milestones{/number}","notifications_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/labels{/name}","releases_url":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/releases{/id}","created_at":"2014-10-14T19:51:33Z","updated_at":"2014-10-15T04:14:06Z","pushed_at":"2014-10-16T00:47:05Z","git_url":"git://github.com/cdap-guides/cdap-spark-guide.git","ssh_url":"git@github.com:cdap-guides/cdap-spark-guide.git","clone_url":"https://github.com/cdap-guides/cdap-spark-guide.git","svn_url":"https://github.com/cdap-guides/cdap-spark-guide","homepage":null,"size":0,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":1,"forks":0,"open_issues":1,"watchers":0,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/pulls/2"},"html":{"href":"https://github.com/cdap-guides/cdap-spark-guide/pull/2"},"issue":{"href":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/issues/2"},"comments":{"href":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/issues/2/comments"},"review_comments":{"href":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/pulls/2/comments"},"review_comment":{"href":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/pulls/comments/{number}"},"commits":{"href":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/pulls/2/commits"},"statuses":{"href":"https://api.github.com/repos/cdap-guides/cdap-spark-guide/statuses/0265a5d30401e9e3cfd92b6fd6d65d4785df594b"}}}}