GitHub PullRequestReviewCommentEvent at 2014-10-05 19:18:03
Repository: 500lines
Actor: cyplo
Payload: {"action":"created","comment":{"url":"https://api.github.com/repos/aosabook/500lines/pulls/comments/18436805","id":18436805,"diff_hunk":"@@ -0,0 +1,534 @@\n+Clustering by Consensus\n+***********************\n+\n+In this chapter, we'll explore implementation of a network protocol designed to support reliable distributed computation.\n+Network protocols can be difficult to implement correctly, so we'll look at some techniques for minimizing bugs and for catching and fixing the remaining few.\n+Building reliable software, too, requires some special development and debugging techniques.\n+\n+Motivating Example\n+==================\n+\n+The focus of this chapter is on the protocol implementation, but as a motivating example let's consider a simple bank account management service.\n+In this service, each account has a current balance and is identified with an account number.\n+Users access the accounts by requesting operations like \"deposit\", \"transfer\", or \"get-balance\".\n+The \"transfer\" operation operates on two accounts at once -- the source and destination accounts -- and must be rejected if the source account's balance is too low.\n+\n+If the service is hosted on a single server, this is easy to implement: use a lock to make sure that transfer operations don't run in parallel, and verify the souce account's balance in that method.\n+However, a bank cannot rely on a single server for its critical account balances!\n+Instead, the service is *distributed* over multiple servers, with each running a separate instance of exactly the same code.\n+Users can then contact any server to perform an operation.\n+\n+In a naive implementation of distributed processing, each server would keep a local copy of every account's balance.\n+It would handle any operations it received, and send updates for account balances to other servers.\n+But this approach introduces a serious failure mode: if two servers process operations for the same account at the same time, which new account balance is correct?\n+Even if the servers share operations with one another instead of balances, two simultaneous transfers out of an account might overdraw the account.\n+\n+Fundamentally, these failures occur when servers use their local state to perform operations, without first ensuring that the local state matches the state on other servers.\n+For example, magine that server A receives a transfer operation from account 101 to account 202, when server B has already processed another transfer account 101's full balance to account 202, but not yet informed server A.\n+The local state on server A is different from that on server B, so server A incorrectly allows the transfer to complete, even though the result is an overdraft on account 101.\n+\n+Distributed State Machines\n+==========================\n+\n+The technique for avoiding such problems is called a \"distributed state machine\".\n+The idea is that each server executes exactly the same deterministic state machine on exactly the same inputs.\n+By the nature of state machines, then, each server will see exactly the same outputs.\n+Operations such as \"transfer\" or \"get-balance\", together with their parameters (account numbers and amounts) represent the inputs to the state machine.\n+\n+The state machine for this application is simple:\n+\n+    def execute_operation(state, operation):\n+        if operation.name == 'transfer':\n+            if state.accounts[operation.source_account] < operation.amount:\n+                return state, False\n+            state.accounts[operation.source_account] -= operation.amount\n+            state.accounts[operation.destination_account] += operation.amount\n+            return state, True\n+        elif operation.name == 'get-balance':\n+            return state, state.accounts[operation.account]\n+\n+Note that executing the \"get-balance\" operation does not modify the state, but is still implemented as a state transition.\n+This guarantees that the returned balance is the latest information in the cluster of servers, not based on the local state on a single server.\n+\n+This may look different than the typical state machine you'd learn about in a computer science course.\n+Rather than a finite set of named states with labeled transitions, this machine's state is the collection of account balances, so there are infinite possible states.\n+Still, the usual rules of deterministic state machines apply: starting with the same state and processing the same operations will always produce the same output.\n+\n+So, the distributed state machine technique ensures that the same operations occur on each host.\n+But the problem remains of ensuring that every server agrees on the inputs to the state machine.\n+This is a problem of *consensus*, and we'll address it with derivative of the Paxos algorithm.\n+\n+Consensus by Paxos\n+==================\n+\n+Paxos was described by Leslie Lamport in a fanciful paper, first submitted in 1990 and eventually published in 1998, entitled \"The Part-Time Parliament\".\n+Lamport's paper has a great deal more detail than we will get into here, and is a fun read.\n+The references at the end of the chapter describe some extensions of the algorithm that we have adapted in this implementation.\n+\n+The simplest form of Paxos provides a way for a set of servers to agree on one value, for all time.\n+MultiPaxos builds on this foundation by agreeing on a numbered sequence of facts, one at a time.\n+To implement a distributed state machine, we use MultiPaxos to agree on each state-machine input, and execute them in sequence.\n+\n+Simple Paxos\n+------------\n+\n+So let's start with \"Simple Paxos\", also known as the Synod protocol, which provides a way to agree on a single value.\n+The name Paxos comes from the mythical island in \"The Part-Time Parliament\", where lawmakers vote on legislation through a process Lamport dubbed a Synod.\n+\n+The algorithm is a building block for more complex algorithms, so we'll set aside the bank example for the moment and use Simple Paxos to agree on today's lottery number.\n+This problem involves consensus on a single value: while there may be another lottery drawing next week, *today's* lottery number, once decided, will never change.\n+\n+The protocol operates in a series of ballots, each led by a single member of the cluster, called the proposer.\n+Each ballot has a unique ballot number based on an integer and the proposer's identity.\n+The proposer's goal is to get a majority of cluster members to accept its value, but only if another value has not already been decided.\n+\n+A ballot begins with the proposer sending a ``Prepare`` message with the ballot number *N* to the acceptors and waiting to hear from a majority.\n+\n+The ``Prepare`` message is a request for the accepted value (if any) with the highest ballot number less than *N*.\n+Acceptors respond with a ``Promise`` containing any value they have already accepted, and promising not to accept any ballot numbered less than *N* in the future.\n+If the acceptor has already made a promise for a larger ballot number, it includes that number in the ``Promise``, indicating that the proposer has been pre-empted.\n+\n+When the proposer has heard back from a majority of the acceptors, it sends an ``Accept`` message, including the ballot number and value to all acceptors.\n+If the proposer did not receive any existing value from any acceptor, then it sends its own desired value.\n+Otherwise, it sends the value from the highest-numbered ballot.\n+Unless it would violate a promise, each acceptor records the value from the ``Accept`` message as accepted and replies with an ``Accepted`` message.\n+The ballot is complete and the value decided when the proposer has heard its ballot number from a majority of acceptors.\n+\n+Returning to the example, initially no other value has been accepted, so the acceptors all send back a ``Promise`` with no value, and the proposer sends an ``Accept`` containing its value, 7-3-22.\n+\n+If another proposer later initiates lower-numbered ballot, the acceptors will not accept it.\n+If that ballot has a larger ballot number, then the ``Promise`` from the acceptors will contain 7-3-22, and the proposer will send that value in the ``Accept`` message.\n+The new ballot will be accepted, but in favor of the same value as the first ballot.\n+\n+In fact, the protocol will never allow two different values to be decided, even if the ballots overlap, messages are delayed, or a minority of acceptors fail.\n+\n+When multiple proposers make a ballot at the same time, it is easy for neither ballot to be accepted.\n+Both proposers then re-propose, and hopefully one wins, but the deadlock can continue indefinitely if the timing works out just right.\n+In a bad -- but not uncommon -- case, it can take dozens of round-trips to reach consensus.\n+\n+Multi-Paxos\n+-----------\n+\n+Reaching consensus on a single, static value is not particularly useful on its own.\n+Clustered systems such as the bank account service want to agree on a particular state (account balances) that evolves over time.\n+We use Paxos to agree on each operation, treated as a state machine transition.\n+\n+Multi-Paxos is, in effect, a sequence of simple Paxos instances (slots), each numbered sequentially.\n+Each state transition is given a \"slot number\", and each member of the cluster executes transitions in strict numeric order.\n+To change the cluster's state (to process a transfer operation, for example), we try to achieve consensus on that operation in the next slot.\n+\n+Running Paxos for every slot, with its minimum of two round trips, would be too slow.\n+Multi-Paxos optimizes by using the same set of ballot numbers for all slots, and performing the ``Prepare``/``Promise`` phase for all slots at once.\n+\n+Paxos Made .. Pretty Hard, Actually\n+-----------------------------------\n+\n+Implementing Multi-Paxos in practical software is notoriously difficult, spawning a number of papers mocking Lamport's \"Paxos Made Simple\" with titles like \"Paxos Made Practical\".\n+\n+First, the multiple-proposers problem described above can become problematic in a busy environment, as each cluster member attempts to get its state machine operation decided in each slot.\n+The fix is to elect a \"leader\" which is responsible for submitting ballots for each slot.\n+All other cluster nodes then send new operations to the leader for execution.\n+Thus, in normal operation with only one leader, ballot conflicts do not occur.\n+\n+The ``Prepare``/``Promise`` phase can function as a kind of leader election: whichever cluster member owns the most recently promised ballot number is considered the leader.\n+The leader is then free to execute the ``Accept``/``Accepted`` phase directly without repeating the first phase.\n+As we'll see below, leader elections are actually quite complex.\n+\n+Although simple Paxos guarantees that the cluster will not reach conflicting decisions, it cannot guarantee that any decision will be made.\n+Fixing this requires carefully orchestrated re-transmissions: enough to eventually make progress, but not so many that the cluster buries itself in a packet storm.\n+\n+Another problem is the dissemination of decisions.\n+A simple broadcast of a ``Decision`` message can take care of this for the normal case.\n+If the message is lost, though, a node can remain permanently ignorant of the decision and unable to apply state machine transitions for later slots.\n+So an implementation needs some mechanism for sharing information about decided proposals.\n+\n+Our use of a distributed state machine presents another interesting challenge: start-up.\n+When a new node starts, it needs to catch up on the existing state of the cluster.\n+Although it can do so by catching up on decisions for all slots since the first, in a mature cluster this may involve millions of slots.\n+Furthermore, we need some way to initialize a new cluster.\n+\n+Introducing Cluster\n+===================\n+\n+The \"Cluster\" library in this chapter implements a simple form of Multi-Paxos.\n+It is designed as a library to provide a consensus service to a larger application.\n+The application creates and starts a ``Member`` object on each cluster member, providing an application-specific state machine and a list of peers.\n+The application accesses the shared state through the ``invoke`` method, which kicks off a proposal for a state transition.\n+Once that proposal is decided and the state machine runs, ``invoke`` returns the machine's output.\n+\n+Users of this library will depend on its correctness, so it's important to structure the code so that we can see -- and test -- its correspondance to the specification.\n+Complex protocols can exhibit complex failures, too, so we will build support for reproducing and debugging rare failures.\n+\n+Message Types\n+-------------\n+\n+Cluster's protocol uses 15 different message types.\n+Using named tuples to describe each message type keeps the code clean and helps avoid some simple errors.\n+The named tuple constructor will raise an exception if it is not given exactly the right attributes, making typos obvious.\n+The tuples format themselves nicely in log messages, and as an added bonus don't use as much memory as a dictionary.\n+\n+Component Model\n+---------------\n+\n+Humans are limited by what we can hold in our active memory.\n+We can't reason about the entire Cluster implementation at once -- it's just too much, and too easy to miss details.\n+Instead, we break Cluster down into a handful of components, implemented as subclasses of ``Component``.\n+Each class is responsible for a different part of the protocol.\n+The division of the components is based on the set of protocol roles described in (Renesse, 2011).\n+\n+The components are glued together by the ``Node`` class, which represents a single node on the network.\n+Components are added to and removed from the node as execution proceeds.\n+Messages that arrive on the node are relayed to all active components, calling a method named after the message type with a ``do_`` prefix.\n+These ``do_`` methods receive the message's attributes as keyword arguments for easy access.\n+\n+The ``Node`` class also provides some convenience methods, using ``functools.partial`` to supply some arguments to the same methods of the ``Network`` class.\n+\n+Acceptor\n+........\n+\n+The ``Acceptor`` class illustrates the component model well.\n+It implements the acceptor role in the protocol, so it must store the ballot number representing its most recent promise, along with the set of accepted proposals for each slot.\n+It then responds to ``Prepare`` and ``Accept`` messages according to the protocol.\n+The result is a short class that is easy to compare to the protocol.\n+\n+Replica\n+.......\n+\n+The ``Replica`` class is the most complicated component class, as it has a few closely related responsibilities:\n+\n+* Making new proposals;\n+* Invoking the local state machine when proposals are decided;\n+* Tracking the current leader; and\n+* Adding newly started nodes to the cluster.\n+\n+The replica creates new proposals in response to ``Invoke`` messages from clients, selecting what it believes to be an unused slot and sending a ``Propose`` message to the current leader.\n+Furthermore, if the consensus for the selected slot is for a different proposal, the replic must re-propose with a new slot.\n+\n+``Decision`` messages represent slots on which the cluster has come to consensus.","path":"cluster/CHAPTER.rst","position":207,"original_position":207,"commit_id":"1d5d04d956d545fe0c8f1150d610ae7188e37d38","original_commit_id":"1d5d04d956d545fe0c8f1150d610ae7188e37d38","user":{"login":"cyplo","id":217899,"avatar_url":"https://avatars.githubusercontent.com/u/217899?v=2","gravatar_id":"","url":"https://api.github.com/users/cyplo","html_url":"https://github.com/cyplo","followers_url":"https://api.github.com/users/cyplo/followers","following_url":"https://api.github.com/users/cyplo/following{/other_user}","gists_url":"https://api.github.com/users/cyplo/gists{/gist_id}","starred_url":"https://api.github.com/users/cyplo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cyplo/subscriptions","organizations_url":"https://api.github.com/users/cyplo/orgs","repos_url":"https://api.github.com/users/cyplo/repos","events_url":"https://api.github.com/users/cyplo/events{/privacy}","received_events_url":"https://api.github.com/users/cyplo/received_events","type":"User","site_admin":false},"body":"Is this a new section, about messages ? Or does this relate to the Replica component only ?","created_at":"2014-10-05T19:18:03Z","updated_at":"2014-10-05T19:18:03Z","html_url":"https://github.com/aosabook/500lines/pull/69#discussion_r18436805","pull_request_url":"https://api.github.com/repos/aosabook/500lines/pulls/69","_links":{"self":{"href":"https://api.github.com/repos/aosabook/500lines/pulls/comments/18436805"},"html":{"href":"https://github.com/aosabook/500lines/pull/69#discussion_r18436805"},"pull_request":{"href":"https://api.github.com/repos/aosabook/500lines/pulls/69"}}},"pull_request":{"url":"https://api.github.com/repos/aosabook/500lines/pulls/69","id":17973226,"html_url":"https://github.com/aosabook/500lines/pull/69","diff_url":"https://github.com/aosabook/500lines/pull/69.diff","patch_url":"https://github.com/aosabook/500lines/pull/69.patch","issue_url":"https://api.github.com/repos/aosabook/500lines/issues/69","number":69,"state":"open","locked":false,"title":"First draft of cluster chapter","user":{"login":"djmitche","id":28673,"avatar_url":"https://avatars.githubusercontent.com/u/28673?v=2","gravatar_id":"","url":"https://api.github.com/users/djmitche","html_url":"https://github.com/djmitche","followers_url":"https://api.github.com/users/djmitche/followers","following_url":"https://api.github.com/users/djmitche/following{/other_user}","gists_url":"https://api.github.com/users/djmitche/gists{/gist_id}","starred_url":"https://api.github.com/users/djmitche/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/djmitche/subscriptions","organizations_url":"https://api.github.com/users/djmitche/orgs","repos_url":"https://api.github.com/users/djmitche/repos","events_url":"https://api.github.com/users/djmitche/events{/privacy}","received_events_url":"https://api.github.com/users/djmitche/received_events","type":"User","site_admin":false},"body":"I'm reasonably happy with the shape and content of the chapter here, so I think it's ready for a review.\r\n\r\nThe code is mostly in place, although there are a number of \"TODO\" tags that I'm still pondering.  The big one -- that affects the chapter -- is to remove the 'catchup' support entirely.  I can't quite convince myself that the implementation will still perform correctly without it, though.","created_at":"2014-07-03T23:09:18Z","updated_at":"2014-10-05T19:18:03Z","closed_at":null,"merged_at":null,"merge_commit_sha":"2934e960bc612647c18464ba177a9543a60b31f6","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/aosabook/500lines/pulls/69/commits","review_comments_url":"https://api.github.com/repos/aosabook/500lines/pulls/69/comments","review_comment_url":"https://api.github.com/repos/aosabook/500lines/pulls/comments/{number}","comments_url":"https://api.github.com/repos/aosabook/500lines/issues/69/comments","statuses_url":"https://api.github.com/repos/aosabook/500lines/statuses/1d5d04d956d545fe0c8f1150d610ae7188e37d38","head":{"label":"djmitche:master","ref":"master","sha":"1d5d04d956d545fe0c8f1150d610ae7188e37d38","user":{"login":"djmitche","id":28673,"avatar_url":"https://avatars.githubusercontent.com/u/28673?v=2","gravatar_id":"","url":"https://api.github.com/users/djmitche","html_url":"https://github.com/djmitche","followers_url":"https://api.github.com/users/djmitche/followers","following_url":"https://api.github.com/users/djmitche/following{/other_user}","gists_url":"https://api.github.com/users/djmitche/gists{/gist_id}","starred_url":"https://api.github.com/users/djmitche/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/djmitche/subscriptions","organizations_url":"https://api.github.com/users/djmitche/orgs","repos_url":"https://api.github.com/users/djmitche/repos","events_url":"https://api.github.com/users/djmitche/events{/privacy}","received_events_url":"https://api.github.com/users/djmitche/received_events","type":"User","site_admin":false},"repo":{"id":14333809,"name":"500lines","full_name":"djmitche/500lines","owner":{"login":"djmitche","id":28673,"avatar_url":"https://avatars.githubusercontent.com/u/28673?v=2","gravatar_id":"","url":"https://api.github.com/users/djmitche","html_url":"https://github.com/djmitche","followers_url":"https://api.github.com/users/djmitche/followers","following_url":"https://api.github.com/users/djmitche/following{/other_user}","gists_url":"https://api.github.com/users/djmitche/gists{/gist_id}","starred_url":"https://api.github.com/users/djmitche/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/djmitche/subscriptions","organizations_url":"https://api.github.com/users/djmitche/orgs","repos_url":"https://api.github.com/users/djmitche/repos","events_url":"https://api.github.com/users/djmitche/events{/privacy}","received_events_url":"https://api.github.com/users/djmitche/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/djmitche/500lines","description":"500 Lines or Less","fork":true,"url":"https://api.github.com/repos/djmitche/500lines","forks_url":"https://api.github.com/repos/djmitche/500lines/forks","keys_url":"https://api.github.com/repos/djmitche/500lines/keys{/key_id}","collaborators_url":"https://api.github.com/repos/djmitche/500lines/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/djmitche/500lines/teams","hooks_url":"https://api.github.com/repos/djmitche/500lines/hooks","issue_events_url":"https://api.github.com/repos/djmitche/500lines/issues/events{/number}","events_url":"https://api.github.com/repos/djmitche/500lines/events","assignees_url":"https://api.github.com/repos/djmitche/500lines/assignees{/user}","branches_url":"https://api.github.com/repos/djmitche/500lines/branches{/branch}","tags_url":"https://api.github.com/repos/djmitche/500lines/tags","blobs_url":"https://api.github.com/repos/djmitche/500lines/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/djmitche/500lines/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/djmitche/500lines/git/refs{/sha}","trees_url":"https://api.github.com/repos/djmitche/500lines/git/trees{/sha}","statuses_url":"https://api.github.com/repos/djmitche/500lines/statuses/{sha}","languages_url":"https://api.github.com/repos/djmitche/500lines/languages","stargazers_url":"https://api.github.com/repos/djmitche/500lines/stargazers","contributors_url":"https://api.github.com/repos/djmitche/500lines/contributors","subscribers_url":"https://api.github.com/repos/djmitche/500lines/subscribers","subscription_url":"https://api.github.com/repos/djmitche/500lines/subscription","commits_url":"https://api.github.com/repos/djmitche/500lines/commits{/sha}","git_commits_url":"https://api.github.com/repos/djmitche/500lines/git/commits{/sha}","comments_url":"https://api.github.com/repos/djmitche/500lines/comments{/number}","issue_comment_url":"https://api.github.com/repos/djmitche/500lines/issues/comments/{number}","contents_url":"https://api.github.com/repos/djmitche/500lines/contents/{+path}","compare_url":"https://api.github.com/repos/djmitche/500lines/compare/{base}...{head}","merges_url":"https://api.github.com/repos/djmitche/500lines/merges","archive_url":"https://api.github.com/repos/djmitche/500lines/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/djmitche/500lines/downloads","issues_url":"https://api.github.com/repos/djmitche/500lines/issues{/number}","pulls_url":"https://api.github.com/repos/djmitche/500lines/pulls{/number}","milestones_url":"https://api.github.com/repos/djmitche/500lines/milestones{/number}","notifications_url":"https://api.github.com/repos/djmitche/500lines/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/djmitche/500lines/labels{/name}","releases_url":"https://api.github.com/repos/djmitche/500lines/releases{/id}","created_at":"2013-11-12T14:03:04Z","updated_at":"2014-07-03T23:06:42Z","pushed_at":"2014-09-22T12:39:37Z","git_url":"git://github.com/djmitche/500lines.git","ssh_url":"git@github.com:djmitche/500lines.git","clone_url":"https://github.com/djmitche/500lines.git","svn_url":"https://github.com/djmitche/500lines","homepage":null,"size":962,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2,"mirror_url":null,"open_issues_count":0,"forks":2,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"aosabook:master","ref":"master","sha":"2f4ac3317409e24fbae444ffbec8a1294c85493b","user":{"login":"aosabook","id":5631751,"avatar_url":"https://avatars.githubusercontent.com/u/5631751?v=2","gravatar_id":"","url":"https://api.github.com/users/aosabook","html_url":"https://github.com/aosabook","followers_url":"https://api.github.com/users/aosabook/followers","following_url":"https://api.github.com/users/aosabook/following{/other_user}","gists_url":"https://api.github.com/users/aosabook/gists{/gist_id}","starred_url":"https://api.github.com/users/aosabook/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aosabook/subscriptions","organizations_url":"https://api.github.com/users/aosabook/orgs","repos_url":"https://api.github.com/users/aosabook/repos","events_url":"https://api.github.com/users/aosabook/events{/privacy}","received_events_url":"https://api.github.com/users/aosabook/received_events","type":"User","site_admin":false},"repo":{"id":14098121,"name":"500lines","full_name":"aosabook/500lines","owner":{"login":"aosabook","id":5631751,"avatar_url":"https://avatars.githubusercontent.com/u/5631751?v=2","gravatar_id":"","url":"https://api.github.com/users/aosabook","html_url":"https://github.com/aosabook","followers_url":"https://api.github.com/users/aosabook/followers","following_url":"https://api.github.com/users/aosabook/following{/other_user}","gists_url":"https://api.github.com/users/aosabook/gists{/gist_id}","starred_url":"https://api.github.com/users/aosabook/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aosabook/subscriptions","organizations_url":"https://api.github.com/users/aosabook/orgs","repos_url":"https://api.github.com/users/aosabook/repos","events_url":"https://api.github.com/users/aosabook/events{/privacy}","received_events_url":"https://api.github.com/users/aosabook/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/aosabook/500lines","description":"500 Lines or Less","fork":false,"url":"https://api.github.com/repos/aosabook/500lines","forks_url":"https://api.github.com/repos/aosabook/500lines/forks","keys_url":"https://api.github.com/repos/aosabook/500lines/keys{/key_id}","collaborators_url":"https://api.github.com/repos/aosabook/500lines/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/aosabook/500lines/teams","hooks_url":"https://api.github.com/repos/aosabook/500lines/hooks","issue_events_url":"https://api.github.com/repos/aosabook/500lines/issues/events{/number}","events_url":"https://api.github.com/repos/aosabook/500lines/events","assignees_url":"https://api.github.com/repos/aosabook/500lines/assignees{/user}","branches_url":"https://api.github.com/repos/aosabook/500lines/branches{/branch}","tags_url":"https://api.github.com/repos/aosabook/500lines/tags","blobs_url":"https://api.github.com/repos/aosabook/500lines/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/aosabook/500lines/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/aosabook/500lines/git/refs{/sha}","trees_url":"https://api.github.com/repos/aosabook/500lines/git/trees{/sha}","statuses_url":"https://api.github.com/repos/aosabook/500lines/statuses/{sha}","languages_url":"https://api.github.com/repos/aosabook/500lines/languages","stargazers_url":"https://api.github.com/repos/aosabook/500lines/stargazers","contributors_url":"https://api.github.com/repos/aosabook/500lines/contributors","subscribers_url":"https://api.github.com/repos/aosabook/500lines/subscribers","subscription_url":"https://api.github.com/repos/aosabook/500lines/subscription","commits_url":"https://api.github.com/repos/aosabook/500lines/commits{/sha}","git_commits_url":"https://api.github.com/repos/aosabook/500lines/git/commits{/sha}","comments_url":"https://api.github.com/repos/aosabook/500lines/comments{/number}","issue_comment_url":"https://api.github.com/repos/aosabook/500lines/issues/comments/{number}","contents_url":"https://api.github.com/repos/aosabook/500lines/contents/{+path}","compare_url":"https://api.github.com/repos/aosabook/500lines/compare/{base}...{head}","merges_url":"https://api.github.com/repos/aosabook/500lines/merges","archive_url":"https://api.github.com/repos/aosabook/500lines/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/aosabook/500lines/downloads","issues_url":"https://api.github.com/repos/aosabook/500lines/issues{/number}","pulls_url":"https://api.github.com/repos/aosabook/500lines/pulls{/number}","milestones_url":"https://api.github.com/repos/aosabook/500lines/milestones{/number}","notifications_url":"https://api.github.com/repos/aosabook/500lines/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/aosabook/500lines/labels{/name}","releases_url":"https://api.github.com/repos/aosabook/500lines/releases{/id}","created_at":"2013-11-04T02:02:53Z","updated_at":"2014-10-05T12:42:36Z","pushed_at":"2014-09-08T20:59:53Z","git_url":"git://github.com/aosabook/500lines.git","ssh_url":"git@github.com:aosabook/500lines.git","clone_url":"https://github.com/aosabook/500lines.git","svn_url":"https://github.com/aosabook/500lines","homepage":null,"size":34728,"stargazers_count":2134,"watchers_count":2134,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":362,"mirror_url":null,"open_issues_count":27,"forks":362,"open_issues":27,"watchers":2134,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/aosabook/500lines/pulls/69"},"html":{"href":"https://github.com/aosabook/500lines/pull/69"},"issue":{"href":"https://api.github.com/repos/aosabook/500lines/issues/69"},"comments":{"href":"https://api.github.com/repos/aosabook/500lines/issues/69/comments"},"review_comments":{"href":"https://api.github.com/repos/aosabook/500lines/pulls/69/comments"},"review_comment":{"href":"https://api.github.com/repos/aosabook/500lines/pulls/comments/{number}"},"commits":{"href":"https://api.github.com/repos/aosabook/500lines/pulls/69/commits"},"statuses":{"href":"https://api.github.com/repos/aosabook/500lines/statuses/1d5d04d956d545fe0c8f1150d610ae7188e37d38"}}}}