piwik/piwik — pull request #6755 by diosmosis (2014-12-01 03:15:30+00:00)
Title: 5363 cron archive full concurrency

Refs #5363, make CronArchive algorithm fully concurrent by adding all API requests made to queue and executing requests in the queue in parallel. Includes:

- Job queue/processor interfaces with default implementation. Default queue/processor uses MySQL and CliMulti only, so should work in all supported environments. Code allows using custom queues/processors that use other technologies to be developed and used however.
* Default job queue uses a 'jobs' table. It is created on demand and it is possible to use multiple tables for it to have multiple queues w/ different jobs (eg, low priority jobs in one queue + high priority jobs in main queue).
* Default job processor pulls a number of jobs from the queue and executes them using CliMulti to achieve parallelism. Job hook logic is always executed in the same thread as the job processor, not the job itself.
- DI integration. Job queues & processors can be defined in DI config and referenced by name to core:archive + core:jobs-server commands.
- Cleaner CronArchive code/architecture. Uses aspect oriented programming in Piwik\CronArchive\Hooks class to separate core algorithm logic with logging, statistics & options. Algorithm logic is stored in AlgorithmRules class and is cached so concurrent code does not re-compute everything (because new CronArchive instances currently have to be created in jobFinished/Starting hooks and of course also on different machines). Algorithm output is also cleaner.
- New core:jobs-server command that can be run on extra machines to have other machines process jobs.
- Jobs are stored serialized in queues and unserialized via a new method: Piwik::secureUnserialize.
- An API method CoreAdminHome.executeJob allows Jobs to be defined that do not use URLs. The CliProcessor will use a URL to this API method to execute such jobs via CliMulti. The API method is only necessary for CliProcessor. Other processors should not need it.

Still needs more testing and I haven't reviewed everything yet.