BVLC/caffe — pull request #1384 by drdan14 (2014-10-31 05:02:12+00:00)
Title: Python log parser

While `tools/extra/parse_log.sh` is a beautiful piece of Bash engineering, it is rather convoluted and difficult to read and modify unless you're a real Bash scripting pro. So I'm introducing `parse_log.py`, a python version of the log parser, which does basically the same thing but with a few functionality changes that I prefer over the default `parse_log.sh` implementation.

Usage is very similar to `parse_log.sh`; see it with `./parse_log.py -h`:

```
$ ./parse_log.py -h
usage: parse_log.py [-h] [--verbose] logfile_path output_dir

Parse a Caffe training log into two CSV files representing training and
testing information

positional arguments:
logfile_path Path to log file
output_dir Directory in which to place output CSV files

optional arguments:
-h, --help show this help message and exit
--verbose Print some extra info (e.g., output filenames)
```

Differences between `parse_log.py` and `parse_log.sh`:

* The output tables are CSVs, not space-separated; this aids reading in the files with, e.g., Excel or Matlab's `readtable` command
* The first column is `NumIters` not `#Iters` because some programs (e.g., Matlab `readtable` again) do not like weird characters in the column headers
* You must specify the output directory for the parsed tables; I didn't like how with `parse_log.sh` the output files ended up in the current working directory
* The time of the first row of the test file is the number of seconds between when the first network test started and when it ended (which can be a few seconds to a couple of minutes), as opposed to in `parse_log.sh` where the time of the first row was typically near zero; however, the offsets are the same
* Because information is read sequentially and, in the current log format, the learning rate isn't reported until after the results of the first network test are reported, the initial value for the learning rate in the train table will always be NaN; fixing this would require a kludge to read ahead to find the first `lr = ...` line and then rewind, which isn't very aesthetically pleasing
* Unlike `parse_log.sh`, `parse_log.py` doesn't assume that `Test net output #0` is Accuracy and `Test net output #1` is loss, which turns out to depend on the network definition (e.g., it's wrong for the GoogLeNet network defined in https://github.com/BVLC/caffe/pull/1367, which has multiple loss and accuracy layers); it explicitly looks for the `accuracy = ...` and `loss = ...` lines.
* `parse_log.sh` parses train loss from lines like `Iteration N, loss = X` but `parse_log.py` parses train loss from lines like `Train net output #M: loss = X`. In retrospect, I am probably doing this wrong and should parse the loss from the `Iteration N, loss = X` line, which aggregates loss from all loss layers if there's more than one (e.g., in the above GoogLeNet implementation), so that will be changed in a subsequent commit

For your amusement, here's the first few lines the parsed training file both from `parse_log.sh` and `parse_log.py`:

`parse_log.sh`:
```
#Iters Seconds TrainingLoss LearningRate
0 51.768315 1.94563 0.002
20 61.075860 1.58658 0.002
40 70.376403 1.45276 0.002
60 79.681565 1.47321 0.002
80 88.986380 1.36988 0.002
100 149.623934 1.43918 0.002
```

`parse_log.py`
```
NumIters,Seconds,TrainingLoss,LearningRate
0,51.768378,1.945630,nan
20,61.075930,1.586580,0.002000
40,70.376478,1.452760,0.002000
60,79.681636,1.473210,0.002000
80,88.986512,1.369880,0.002000
100,149.624000,1.439180,0.002000
```