drothlis/stb-tester — pull request #107 by asimihsan (2014-02-27 15:40:18+00:00)
Title: extra/runner/report.py - use SQLite cache of file contents to speed up

On file systems where both stat'ing files and getting their contents are
very slow (two or three orders of magnitude slower) this commit improves
performance by five-fold by using an SQLite cache and other changes.

This is a RFC. Although an old version of this code is currently successfully
running on a production system this pull request is of a recently rebased
but untested version.

I would like to break out the `FileCache` object into its own importable file.
This is because on a production system using stb-tester there are other
background processes that are interested in the contents of stb-tester runner
reports but are currently unable to use the existing caches to improve their
read performance. My plan was to break this out into a `FileCache` file and
add respective integration tests to the same file (i.e. tests that go beyond
simple mocking and use an actual directory of files, reads them into the
cache, deletes the actual files, checks the DB still has them, etc.). Thoughts?

I'm concerned that the way in which I've written `_get_path`, by returning
`None`, doesn't really make sense, however it did help me make the `read`
code more concise. Suggestions as to how to express the logic here are welcome.