openmicroscopy/openmicroscopy — pull request #2919 by mtbc (2014-08-07 14:59:12+00:00)
Title: fix #7240: provide disk usage count

This PR fixes http://trac.openmicroscopy.org.uk/ome/ticket/7240. Testing is best done by people who know their way around the model graph objects and `omero.data.dir`. Pick some model objects, like `omero.cmd.DiskUsage(objects={'Image' : [1]}, includeAnnotations=True)`, and see what answer you get from submitting the requests to the server.

The disk usage calculated should include any pyramids, thumbnails, archived files, import logs, other images from the fileset, etc. entailed by the model objects, and, for `includeAnnotations=True`, any file annotations on those objects, including, for instance, an annotation on a well whose screen you specified. A file should never be double-counted.

In code review, scrutinize the population of `DiskUsageI.TRAVERSAL_QUERIES`.

Caveats:

* Import log size calculations are inaccurate due to http://trac.openmicroscopy.org.uk/ome/ticket/12513. Easiest is to just fix the size in the database, or in the sum you calculate manually. (`head -c` will verify for you that the DB size corresponds to up to `Step 4` and no further.)
* Non-existent temporary files are included if you specify `Experimenter` or `ExperimenterGroup` due to http://trac.openmicroscopy.org.uk/ome/ticket/12514.

Testing tips:

If you want to see the `DEBUG` messages in `Blitz-0.log` -- *very* handy for figuring out which files are being counted and how large the server thinks that they are -- not only will you have to adjust `etc/logback.xml` to log `omero.cmd.graphs` at `DEBUG` but you will have to comment out the filter that denies `omero.*` debug logging, thank you to @ximenesuk for that workaround.

To count up the bytes used in uploaded files, I was using lines like,
```
find ManagedRepository/user-2_3/2014-08/07/11-52-21.241 -type f -exec ls -l '{}' ';' | awk '{ sum += $5 } END { print sum }'
```
or, for a range of thumbnails (e.g., well samples for a plate),
```
ls -l Thumbnails/{25..40} | awk '{ sum += $5 } END { print sum }'
```