iParadigms/walker — pull request #123 by pchristopher1275 (2014-12-30 00:39:48+00:00)
Title: Trn 487a

The first part of https://jira2.iparadigms.com/browse/TRN-487: which reads

As a walker operator I want domain priorities to work relatively linear to each other. If I have domains of priority 1, 2, and 3, the 3 should run (on average) three times as many as the 1, and the 2 should run (on average) twice as much as the 1. I want priorities to be integers, minimum 1, maximum whatever we can store in an unsigned integer.
Implementation notes to complete this work:
add a table domain_counters, keyed on domain and having a counter_type column next_crawl (this is a separate table because counters cannot exist next to non-counter non-primary-keys)
add a table walker_globals with columns key text and val int to keep global shared values
in the dispatcher kick off a separate goroutine that periodically (once per dispatch_interval) reads all priorities in domain_info, finds the max, and updates walker_globals with a row: key='max_priority', val= (maybe add a hard minimum on how frequently we'll get this value, say, 1 minute, in case people set dispatch_interval to 0 or very low)
add a private datastore method to get the current maxPriority from walker_globals, which should cache it for 1 minute and re-fetch it on demand. If this value does not exist in walker_globals, use default_domain_priority
in the fetcher's datastore, select all unclaimed and dispatched domains regardless of priority. For each one, increment the domain_counters.next_crawl by that domain's priority, then select it.
If the number is >= maxPriority, try to claim this domain. If we could not claim it, move on. If we claimed it, then decrement it's domain_counters.next_crawl by maxPriority
If it's next_crawl is not >= maxPriority move on, do not claim it
Note in the code that with multiple fetchmanagers going they could increment at the same time and read outdated next_crawl values (i.e. there are concurrency issues here), but that nothing bad will happen, at worst a domain will not be crawled on perfectly even intervals
in the query for unclaimed and dispatched hosts above, use paging so that a given fetcher iterates the whole domain_info table evenly. Keep the last domain this datastore incremented next_crawl, and for successive selects use token(dom) > token()
add to the FAQ explaining what priorities are for (high-level, for an operator)
update the console to allow setting priorities within the appropriate range
add a config key default_domain_priority and every time a domain is added set it to that. Default this key to 100
I believe that captures everything. Feel free to split this into multiple pull requests or tasks if necessary.

===

This patch only implements the part of the ticket that fixes the MaxPriority to 10 (thus the allowed priorities is 1, 2, ..., 10.