GitHub PullRequestReviewCommentEvent at 2014-07-28 17:36:11
Repository: edx-platform
Actor: srpearce
Payload: {"action":"created","comment":{"url":"https://api.github.com/repos/edx/edx-platform/pulls/comments/15477157","id":15477157,"diff_hunk":"@@ -0,0 +1,255 @@\n+.. _Package:\n+\n+######################################\n+Data Delivered in Data Packages\n+######################################\n+\n+For partners who are running courses on edx.org and edge.edx.org, edX regularly\n+makes research data available for download from the Amazon S3 storage service.\n+The *data package* that data czars download from Amazon S3 consists of a set of\n+compressed and encrypted files that contain event logs and database snapshots\n+for all of their organizations' edx.org and edge.edx.org courses.\n+\n+* :ref:`Data Package Files`\n+\n+* :ref:`Amazon S3 Buckets and Directories`\n+\n+* :ref:`Download Data Packages from Amazon S3`\n+\n+* :ref:`Data Package Contents`\n+\n+.. _Data Package Files:\n+\n+**********************\n+Data Package Files\n+**********************\n+\n+A data package consists of files that contain event data and database data.\n+\n+.. note:: All file names include the date in {YYYY}-{MM}-{DD} format.\n+\n+============\n+Event Data\n+============\n+\n+The ``{org}-{site}-events-{date}.log.gz.gpg`` file contains a daily log of\n+course events. A separate file is available for courses running on edge.edx.org\n+(with \"edge\" for the {site} in the file name) and on edx.org (with \"edx\" for\n+{site}).\n+\n+For a partner organization named UniversityX, these daily files are identified by the organization name, the edX site name, and the date. For example, ``universityx-edge-2014-07-25.log.gz.gpg``.\n+\n+An alternative option for event data is available. The\n+``{date}-{org}-tracking.tar`` file is available each week. It contains a\n+cumulative log of events in all of an organization's courses. Data for courses\n+running on both the edx.org and edge.edx.org. sites is included in this file.\n+\n+.. remove this paragraph ^ when weekly file is removed.\n+\n+For information about the contents of these files, see :ref:`Data Package\n+Contents`.\n+\n+==================\n+Database Data\n+==================\n+\n+The ``{org}-{date}.zip`` file contains views on database tables. A new file is\n+available each week. Database data as of the time of the export, for all of an\n+organization's courses on both the edx.org and edge.edx.org. sites, is included\n+in this file.\n+\n+For a partner organization named UniversityX, each weekly file is identified by\n+the organization name and its extraction date: for example,\n+``universityx-2013-10-27.zip``.\n+\n+For information about the contents of this file, see :ref:`Data Package\n+Contents`.\n+\n+.. _Amazon S3 Buckets and Directories:\n+\n+********************************************\n+Amazon S3 Buckets and Directories\n+********************************************\n+\n+Data package files are located in two different buckets on Amazon S3:\n+\n+* The **edx-course-data** bucket contains the daily\n+  ``{org}-{site}-events-{date}.log.gz.gpg`` files of course event data.\n+  \n+* The **course-data** bucket contains the weekly ``{date}-{org}-tracking.tar``\n+  file of cumulative course event data and the weekly ``{org}-{date}.zip``\n+  database snapshot.\n+\n+.. revise this paragraph ^ when weekly event file is removed.\n+\n+For information about accessing Amazon S3, see :ref:`Access Amazon S3`.\n+\n+.. _Download Data Packages from Amazon S3:\n+\n+****************************************************************\n+Download Data Packages from Amazon S3\n+****************************************************************\n+\n+#. Connect to Amazon S3 using a third-party tool or the AWS Command Line\n+   Interface. For information about connecting to Amazon S3, see :ref:`Access\n+   Amazon S3`.\n+\n+#. To download daily event files, you navigate to the **edx-course-data**\n+   bucket, and then through this directory structure to locate the files you\n+   want to download:\n+\n+   ``{org}/{site}/events/{year}``\n+\n+   The event logs in the ``{year}`` directory are in compressed, encrypted\n+   files named ``{org}-{site}-events-{date}.log.gz.gpg``.\n+\n+3. Download the ``{org}-{site}-events-{date}.log.gz.gpg`` file.\n+\n+   If your organization has courses running on both edx.org and edge.edx.org,\n+   separate log files are available for the \"edx\" site and the \"edge\" site.\n+   Repeat this step to download the file for the other site.\n+\n+4. To download a database data file, navigate to the edX **course-data**\n+   bucket. This bucket contains ``{org}-{date}.zip`` files, which are\n+   available each week. \n+\n+#. Download the ``{org}-{date}.zip`` file. \n+\n+#. To download a cumulative weekly event file, return to the **course-data**\n+   bucket. This bucket contains the ``{date}-{org}-tracking.tar`` files, which\n+   are available each week.\n+\n+.. remove this step ^ when weekly event logs are no longer available\n+\n+.. _AWS Command Line Interface: http://aws.amazon.com/cli/\n+\n+.. _Data Package Contents:\n+\n+**********************\n+Data Package Contents\n+**********************\n+\n+After you download your data package files, you must extract and decrypt the\n+contents. Each of the files you download contains one or more files of research\n+data.\n+\n+============================================================\n+Extracted Contents of ``{org}-{site}-events-{date}.gpg``\n+============================================================\n+\n+The ``{org}-{site}-events-{date}.gpg`` file contains event data for a single\n+day. After you download a ``{org}-{site}-events-{date}.gpg`` file for your\n+institution, you:\n+\n+#. Use your private key to decrypt the downloaded .gpg file. See :ref:`Decrypt\n+   an Encrypted File`.\n+\n+#. Extract the log file from the compressed .gz file. The result is a single\n+   file named ``{org}-{site}-events-{date}.log``.\n+\n+.. remove this section v through the next note when weekly file is removed\n+\n+============================================================\n+Extracted Contents of ``{date}-{org}-tracking.tar``\n+============================================================\n+\n+The ``{date}-{org}-tracking.tar`` file contains cumulative event data for all\n+of an organization's courses, running on both edx.org and edge.edx.org.\n+\n+.. note:: Over time, these files can become very large (25GB and larger). In some environments, problems such as session timeouts can occur when you download them. \n+\n+After you download the ``{date}-{org}-tracking.tar`` file for your\n+institution, you:\n+\n+#. Extract the contents of the downloaded .tar file. \n+   \n+   To balance the load of traffic to edX courses, every course is served by\n+   multiple edX servers. When you extract the contents of this file, a separate\n+   subdirectory is created for events that took place on each edX server.\n+\n+   For example, subdirectories with these names can be created:\n+\n+   ``prod-edxapp-003``\n+\n+   ``prod-edxapp-004``\n+\n+   ``prod-edxapp-005``\n+\n+   Each of these directories contains an encrypted log file of event data for\n+   every day that events occurred on that server. These event tracking data\n+   files are named ``{date}-{org}.log.gpg``.\n+\n+#. Use your private key to decrypt the extracted log files. See :ref:`Decrypt\n+   an Encrypted File`.\n+\n+.. note:: During analysis, you must combine events from each server to get a complete picture of the activity in each course. ","path":"docs/en_us/data/source/internal_data_formats/package.rst","position":185,"original_position":185,"commit_id":"5ae1aeae9e159ea6386d3da5d40e35b519632830","original_commit_id":"5ae1aeae9e159ea6386d3da5d40e35b519632830","user":{"login":"srpearce","id":3493253,"avatar_url":"https://avatars.githubusercontent.com/u/3493253?","gravatar_id":"3ab62e3a9e0114a60edd3b5337423385","url":"https://api.github.com/users/srpearce","html_url":"https://github.com/srpearce","followers_url":"https://api.github.com/users/srpearce/followers","following_url":"https://api.github.com/users/srpearce/following{/other_user}","gists_url":"https://api.github.com/users/srpearce/gists{/gist_id}","starred_url":"https://api.github.com/users/srpearce/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/srpearce/subscriptions","organizations_url":"https://api.github.com/users/srpearce/orgs","repos_url":"https://api.github.com/users/srpearce/repos","events_url":"https://api.github.com/users/srpearce/events{/privacy}","received_events_url":"https://api.github.com/users/srpearce/received_events","type":"User","site_admin":false},"body":"Line needs to be divided. Also, do we have any instructions/recommendations for how to combine these events?","created_at":"2014-07-28T17:36:11Z","updated_at":"2014-07-28T17:36:11Z","html_url":"https://github.com/edx/edx-platform/pull/4572#discussion_r15477157","pull_request_url":"https://api.github.com/repos/edx/edx-platform/pulls/4572","_links":{"self":{"href":"https://api.github.com/repos/edx/edx-platform/pulls/comments/15477157"},"html":{"href":"https://github.com/edx/edx-platform/pull/4572#discussion_r15477157"},"pull_request":{"href":"https://api.github.com/repos/edx/edx-platform/pulls/4572"}}},"pull_request":{"url":"https://api.github.com/repos/edx/edx-platform/pulls/4572","id":18934732,"html_url":"https://github.com/edx/edx-platform/pull/4572","diff_url":"https://github.com/edx/edx-platform/pull/4572.diff","patch_url":"https://github.com/edx/edx-platform/pull/4572.patch","issue_url":"https://api.github.com/repos/edx/edx-platform/issues/4572","number":4572,"state":"open","title":"Ahodges/documentation/doc764","user":{"login":"lamagnifica","id":6392274,"avatar_url":"https://avatars.githubusercontent.com/u/6392274?","gravatar_id":"08ab56e1d3585aae4a13bef6ebec6dfa","url":"https://api.github.com/users/lamagnifica","html_url":"https://github.com/lamagnifica","followers_url":"https://api.github.com/users/lamagnifica/followers","following_url":"https://api.github.com/users/lamagnifica/following{/other_user}","gists_url":"https://api.github.com/users/lamagnifica/gists{/gist_id}","starred_url":"https://api.github.com/users/lamagnifica/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lamagnifica/subscriptions","organizations_url":"https://api.github.com/users/lamagnifica/orgs","repos_url":"https://api.github.com/users/lamagnifica/repos","events_url":"https://api.github.com/users/lamagnifica/events{/privacy}","received_events_url":"https://api.github.com/users/lamagnifica/received_events","type":"User","site_admin":false},"body":"@WatsonEmily , please distribute for PM review for usability by our audience.\r\n@jhelbert , @stroilova , @shnayder , @rocha  please review for accuracy, completeness, and level of detail, particularly wrt data czars who will need to revise scripts for post-download processing.\r\n@mhoeber , @srpearce please copy edit.","created_at":"2014-07-26T20:01:55Z","updated_at":"2014-07-28T17:36:11Z","closed_at":null,"merged_at":null,"merge_commit_sha":"751d75cb73f119ffdc0f028cdcc7b2ae2c05ebf8","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/edx/edx-platform/pulls/4572/commits","review_comments_url":"https://api.github.com/repos/edx/edx-platform/pulls/4572/comments","review_comment_url":"https://api.github.com/repos/edx/edx-platform/pulls/comments/{number}","comments_url":"https://api.github.com/repos/edx/edx-platform/issues/4572/comments","statuses_url":"https://api.github.com/repos/edx/edx-platform/statuses/5ae1aeae9e159ea6386d3da5d40e35b519632830","head":{"label":"edx:ahodges/documentation/DOC764","ref":"ahodges/documentation/DOC764","sha":"5ae1aeae9e159ea6386d3da5d40e35b519632830","user":{"login":"edx","id":3179841,"avatar_url":"https://avatars.githubusercontent.com/u/3179841?","gravatar_id":"e1f6c6dc8772c47c0261cd5df2ed2814","url":"https://api.github.com/users/edx","html_url":"https://github.com/edx","followers_url":"https://api.github.com/users/edx/followers","following_url":"https://api.github.com/users/edx/following{/other_user}","gists_url":"https://api.github.com/users/edx/gists{/gist_id}","starred_url":"https://api.github.com/users/edx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edx/subscriptions","organizations_url":"https://api.github.com/users/edx/orgs","repos_url":"https://api.github.com/users/edx/repos","events_url":"https://api.github.com/users/edx/events{/privacy}","received_events_url":"https://api.github.com/users/edx/received_events","type":"Organization","site_admin":false},"repo":{"id":10391073,"name":"edx-platform","full_name":"edx/edx-platform","owner":{"login":"edx","id":3179841,"avatar_url":"https://avatars.githubusercontent.com/u/3179841?","gravatar_id":"e1f6c6dc8772c47c0261cd5df2ed2814","url":"https://api.github.com/users/edx","html_url":"https://github.com/edx","followers_url":"https://api.github.com/users/edx/followers","following_url":"https://api.github.com/users/edx/following{/other_user}","gists_url":"https://api.github.com/users/edx/gists{/gist_id}","starred_url":"https://api.github.com/users/edx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edx/subscriptions","organizations_url":"https://api.github.com/users/edx/orgs","repos_url":"https://api.github.com/users/edx/repos","events_url":"https://api.github.com/users/edx/events{/privacy}","received_events_url":"https://api.github.com/users/edx/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/edx/edx-platform","description":"Open edX, the edX platform that powers http://www.edx.org","fork":false,"url":"https://api.github.com/repos/edx/edx-platform","forks_url":"https://api.github.com/repos/edx/edx-platform/forks","keys_url":"https://api.github.com/repos/edx/edx-platform/keys{/key_id}","collaborators_url":"https://api.github.com/repos/edx/edx-platform/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/edx/edx-platform/teams","hooks_url":"https://api.github.com/repos/edx/edx-platform/hooks","issue_events_url":"https://api.github.com/repos/edx/edx-platform/issues/events{/number}","events_url":"https://api.github.com/repos/edx/edx-platform/events","assignees_url":"https://api.github.com/repos/edx/edx-platform/assignees{/user}","branches_url":"https://api.github.com/repos/edx/edx-platform/branches{/branch}","tags_url":"https://api.github.com/repos/edx/edx-platform/tags","blobs_url":"https://api.github.com/repos/edx/edx-platform/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/edx/edx-platform/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/edx/edx-platform/git/refs{/sha}","trees_url":"https://api.github.com/repos/edx/edx-platform/git/trees{/sha}","statuses_url":"https://api.github.com/repos/edx/edx-platform/statuses/{sha}","languages_url":"https://api.github.com/repos/edx/edx-platform/languages","stargazers_url":"https://api.github.com/repos/edx/edx-platform/stargazers","contributors_url":"https://api.github.com/repos/edx/edx-platform/contributors","subscribers_url":"https://api.github.com/repos/edx/edx-platform/subscribers","subscription_url":"https://api.github.com/repos/edx/edx-platform/subscription","commits_url":"https://api.github.com/repos/edx/edx-platform/commits{/sha}","git_commits_url":"https://api.github.com/repos/edx/edx-platform/git/commits{/sha}","comments_url":"https://api.github.com/repos/edx/edx-platform/comments{/number}","issue_comment_url":"https://api.github.com/repos/edx/edx-platform/issues/comments/{number}","contents_url":"https://api.github.com/repos/edx/edx-platform/contents/{+path}","compare_url":"https://api.github.com/repos/edx/edx-platform/compare/{base}...{head}","merges_url":"https://api.github.com/repos/edx/edx-platform/merges","archive_url":"https://api.github.com/repos/edx/edx-platform/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/edx/edx-platform/downloads","issues_url":"https://api.github.com/repos/edx/edx-platform/issues{/number}","pulls_url":"https://api.github.com/repos/edx/edx-platform/pulls{/number}","milestones_url":"https://api.github.com/repos/edx/edx-platform/milestones{/number}","notifications_url":"https://api.github.com/repos/edx/edx-platform/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/edx/edx-platform/labels{/name}","releases_url":"https://api.github.com/repos/edx/edx-platform/releases{/id}","created_at":"2013-05-30T20:20:38Z","updated_at":"2014-07-28T12:04:00Z","pushed_at":"2014-07-28T17:17:54Z","git_url":"git://github.com/edx/edx-platform.git","ssh_url":"git@github.com:edx/edx-platform.git","clone_url":"https://github.com/edx/edx-platform.git","svn_url":"https://github.com/edx/edx-platform","homepage":"http://code.edx.org/","size":1181775,"stargazers_count":1333,"watchers_count":1333,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"forks_count":744,"mirror_url":null,"open_issues_count":176,"forks":744,"open_issues":176,"watchers":1333,"default_branch":"master"}},"base":{"label":"edx:master","ref":"master","sha":"8a5bb72984f4df0c3c4feba2582dac67dcfa15e5","user":{"login":"edx","id":3179841,"avatar_url":"https://avatars.githubusercontent.com/u/3179841?","gravatar_id":"e1f6c6dc8772c47c0261cd5df2ed2814","url":"https://api.github.com/users/edx","html_url":"https://github.com/edx","followers_url":"https://api.github.com/users/edx/followers","following_url":"https://api.github.com/users/edx/following{/other_user}","gists_url":"https://api.github.com/users/edx/gists{/gist_id}","starred_url":"https://api.github.com/users/edx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edx/subscriptions","organizations_url":"https://api.github.com/users/edx/orgs","repos_url":"https://api.github.com/users/edx/repos","events_url":"https://api.github.com/users/edx/events{/privacy}","received_events_url":"https://api.github.com/users/edx/received_events","type":"Organization","site_admin":false},"repo":{"id":10391073,"name":"edx-platform","full_name":"edx/edx-platform","owner":{"login":"edx","id":3179841,"avatar_url":"https://avatars.githubusercontent.com/u/3179841?","gravatar_id":"e1f6c6dc8772c47c0261cd5df2ed2814","url":"https://api.github.com/users/edx","html_url":"https://github.com/edx","followers_url":"https://api.github.com/users/edx/followers","following_url":"https://api.github.com/users/edx/following{/other_user}","gists_url":"https://api.github.com/users/edx/gists{/gist_id}","starred_url":"https://api.github.com/users/edx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edx/subscriptions","organizations_url":"https://api.github.com/users/edx/orgs","repos_url":"https://api.github.com/users/edx/repos","events_url":"https://api.github.com/users/edx/events{/privacy}","received_events_url":"https://api.github.com/users/edx/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/edx/edx-platform","description":"Open edX, the edX platform that powers http://www.edx.org","fork":false,"url":"https://api.github.com/repos/edx/edx-platform","forks_url":"https://api.github.com/repos/edx/edx-platform/forks","keys_url":"https://api.github.com/repos/edx/edx-platform/keys{/key_id}","collaborators_url":"https://api.github.com/repos/edx/edx-platform/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/edx/edx-platform/teams","hooks_url":"https://api.github.com/repos/edx/edx-platform/hooks","issue_events_url":"https://api.github.com/repos/edx/edx-platform/issues/events{/number}","events_url":"https://api.github.com/repos/edx/edx-platform/events","assignees_url":"https://api.github.com/repos/edx/edx-platform/assignees{/user}","branches_url":"https://api.github.com/repos/edx/edx-platform/branches{/branch}","tags_url":"https://api.github.com/repos/edx/edx-platform/tags","blobs_url":"https://api.github.com/repos/edx/edx-platform/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/edx/edx-platform/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/edx/edx-platform/git/refs{/sha}","trees_url":"https://api.github.com/repos/edx/edx-platform/git/trees{/sha}","statuses_url":"https://api.github.com/repos/edx/edx-platform/statuses/{sha}","languages_url":"https://api.github.com/repos/edx/edx-platform/languages","stargazers_url":"https://api.github.com/repos/edx/edx-platform/stargazers","contributors_url":"https://api.github.com/repos/edx/edx-platform/contributors","subscribers_url":"https://api.github.com/repos/edx/edx-platform/subscribers","subscription_url":"https://api.github.com/repos/edx/edx-platform/subscription","commits_url":"https://api.github.com/repos/edx/edx-platform/commits{/sha}","git_commits_url":"https://api.github.com/repos/edx/edx-platform/git/commits{/sha}","comments_url":"https://api.github.com/repos/edx/edx-platform/comments{/number}","issue_comment_url":"https://api.github.com/repos/edx/edx-platform/issues/comments/{number}","contents_url":"https://api.github.com/repos/edx/edx-platform/contents/{+path}","compare_url":"https://api.github.com/repos/edx/edx-platform/compare/{base}...{head}","merges_url":"https://api.github.com/repos/edx/edx-platform/merges","archive_url":"https://api.github.com/repos/edx/edx-platform/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/edx/edx-platform/downloads","issues_url":"https://api.github.com/repos/edx/edx-platform/issues{/number}","pulls_url":"https://api.github.com/repos/edx/edx-platform/pulls{/number}","milestones_url":"https://api.github.com/repos/edx/edx-platform/milestones{/number}","notifications_url":"https://api.github.com/repos/edx/edx-platform/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/edx/edx-platform/labels{/name}","releases_url":"https://api.github.com/repos/edx/edx-platform/releases{/id}","created_at":"2013-05-30T20:20:38Z","updated_at":"2014-07-28T12:04:00Z","pushed_at":"2014-07-28T17:17:54Z","git_url":"git://github.com/edx/edx-platform.git","ssh_url":"git@github.com:edx/edx-platform.git","clone_url":"https://github.com/edx/edx-platform.git","svn_url":"https://github.com/edx/edx-platform","homepage":"http://code.edx.org/","size":1181775,"stargazers_count":1333,"watchers_count":1333,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"forks_count":744,"mirror_url":null,"open_issues_count":176,"forks":744,"open_issues":176,"watchers":1333,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/edx/edx-platform/pulls/4572"},"html":{"href":"https://github.com/edx/edx-platform/pull/4572"},"issue":{"href":"https://api.github.com/repos/edx/edx-platform/issues/4572"},"comments":{"href":"https://api.github.com/repos/edx/edx-platform/issues/4572/comments"},"review_comments":{"href":"https://api.github.com/repos/edx/edx-platform/pulls/4572/comments"},"review_comment":{"href":"https://api.github.com/repos/edx/edx-platform/pulls/comments/{number}"},"commits":{"href":"https://api.github.com/repos/edx/edx-platform/pulls/4572/commits"},"statuses":{"href":"https://api.github.com/repos/edx/edx-platform/statuses/5ae1aeae9e159ea6386d3da5d40e35b519632830"}}}}