NationalGenomicsInfrastructure/ngi_pipeline — pull request #109 by guillermo-carrasco (2014-10-30 16:04:27+00:00)
Title: Simpler logging + start script

So two main things here:

#### 1. Logging
@Galithil convinced me that using logbook + Redis was not the best idea, I'll give you that Denis. The main points are:

* `logbook` doesn't do anything that we can't do with `logging`, it makes your life easier IMHO, but just a few lines of code at the end of the day.
* The main point of using `logbook` + `RedisHandler` was that this way would be easier to log into ELK. The problem that Denis pointed out and that made me change my mind was that this makes the code send logs through the network, what happens if?
* The network is slow and the pipeline very verbose
* The redis database is down
* The machine where the database is hosted is not reachable

Instead of sorting all this out, is better to log regularly to a file (which will make the pipeline independent on the log processing system that we use) and deploy a logstash shipper that will parse the pipeline logs (probably to a file?)

#### 2. Start script
The tornado server will execute this script as background subprocesses in order to detach the tasks from the server. I need a script able to execute everything, otherwise I would have to list all available scripts that the Tornado server can execute. This is `ngi_pipeline_start.py`. The argument parsing in `bwa` style, i.e:

`./ngi_pipeline_start.py -h`
```
usage: ngi_pipeline_start.py [-h] {server,process} ...

Launch NGI pipeline

positional arguments:
{server,process} Choose the mode to run
server Start ngi_pipeline server
process Start some analysis process

optional arguments:
-h, --help show this help message and exit
```
`./ngi_pipeline_start.py process -h`
```
usage: ngi_pipeline_start.py process [-h] {flowcell,sample} ...

positional arguments:
{flowcell,sample} Choose unit to process
flowcell Start analysis of raw flowcells
sample Start the analysis of a particular sample

optional arguments:
-h, --help show this help message and exit
```
`./ngi_pipeline_start.py process flowcell -h`
```
usage: ngi_pipeline_start.py process flowcell [-h] [-p RESTRICT_TO_PROJECTS]
[-s RESTRICT_TO_SAMPLES] [-f]
demux_fcid_dir

positional arguments:
demux_fcid_dir The path to the Illumina demultiplexed fc directories
to process.

optional arguments:
-h, --help show this help message and exit
-p RESTRICT_TO_PROJECTS, --project RESTRICT_TO_PROJECTS
Restrict processing to these projects. Use flag
multiple times for multiple projects.
-s RESTRICT_TO_SAMPLES, --sample RESTRICT_TO_SAMPLES
Restrict processing to these samples. Use flag
multiple times for multiple samples.
-f, --restart-failed Restart jobs marked as 'FAILED' in Charon
```

etc...