Initial work on image ingestion. gh-13

First commit for image ingestion work:
- Add 2 new files, one containing the ingestion routines, and one containing the tests
- Add some initial routines relating to reading the metadata file and calling the appropriate
  data reading method, and the accompanying tests for these.
Re-arrange tests_images gh-13

Add some subclasses to the image ingestion tests, to keep things tidier.
Also add a more functional test at the end, to check the entire ingestion process.
Add initial routines for reading HDF file gh-13

Add first data reading routine, for HDF files. Requires the pyhdf module

Add method for extracting the region of interest. Note that it returns
slice objects rather than lists of points, and that the slice is made from
the min and max i and j indices that were found to be inside the ROI. This is
needed because we don't have a square grid, so a simple list wouldn't be able
to give us a 2d subset array (we would get a 1d list)

Add a high level method ingest_images that would be the one users call, and
it then calls the others as required.
Add SaveGeoTiff method

Adding initial method for saving to GeoTiff. Still needs some work to generalise as it
has some hardoded bits still.

Also, it turns out that we can open hdf files with gdal, which is easier to install than
pyhdf. So also add a second read_hdf method using gdal instead, so we can choose.
Do some tidying

Split out some of the methods into separate classes in separate files to keep things a
bit tidier:
  - Methods relating to geographical transformations, creating geotiff etc are now in
    GeoTools class in ingest_images_geo_tools.py
  - Methods for reading in different file types are in DataReaders class in
    ingest_images_file_readers.py
This leaves just the top level routines in ingest_images.py

Also do some tidying of docstrings, and class names etc to comply with style guide.
Add regridding option, and method for adding image to database

Add new GeoTools method to regrid data to a regular grid, at the same time as extracting the region of
interest. This simplifies creating the geotiff, as it assumes that all pixels are same size.

Also add the image into the database, as and Image object, with an associated ImageRegion object.
Add reader for N1 data type (for MERIS and AATSR)

Add a new method to read N1 data types, using pyepr package.

Also change the way the reader methods are selected - discard "filetype" key from JSON file,
and use the instrument name to select which reader to use. This way makes more sense as we
don't anticipate having different file types for one instrument, but we do have several
instruments that may use the same file type.
Improve regridding method

Change the way we calculate the grid size to regrid to. Previously we just
used the lon/lat difference between adjacent points on the old grid. This
worked well for the VIIRS test file, but then left holes in AATSR and MERIS
when the data were regridded.

So instead of using adjacent points, we use the lon/lat differences between
opposite corners of squares that are formed by each group of 4 points:

1A------2A------3A
|       |       |
1B------2B------3B
|       |       |
1C------2C------3C

Eg take lon difference between 1A-2B, 2A-3B etc, and
   take lat difference between 2A-1B, 3A-2B etc

As before, we use the maximum value across the grid.
Use gdal not pyhdf to read HDF files

Remove the HDF reader that uses pyhdf, keeping just the gdal one.
This reduces the number of external packages that we need, and gdal
is also much easier to install.
Some refactoring/tidying

Make the GeoTools methods static, so don't need to make a pointless
GeoTools instance before using them.

Remove the old extract_region method, as we are using the new one that
regrids at the same time

Tidy the file readers a bit by adding inner functions to do the data
extraction/gridding, so we can call several times for different variables
(eg when we add the angles)

Fix some whitespace issues for PEP8 compliance
Read in viewing angles + separate file readers for each instrument

Add ability to read in viewing angles, save them to a separate
geotiff, and save the mean value in the database.

As different instruments use different dataset names for the
viewing angles. it was awkward to use the existing file readers.
So, instead of calling a reader for the file type, we call new
readers for individual instruments (which then call the old methods)

Also, as AATSR has separate data arrays for forward and nadir
directions, add some changes to deal with this. We save all the
arrays to the same file, but create two database entries - one
for each direction

Do some refactoring: make metadata, inputdir etc class variables
rather than passing them through as parameters all the time. Now
we pass the IngestImages object instance through to the other classes
instead.
Tweaks to Image model fields

Change the filename fields (archive and web locations) in the
Image model from CharField to TextField, to allow unlimited
length strings, since filenames could easily get longer than
255 characters (which is the max length guaranteed to work).

Change "direction" to a CharField, as it doesn't need to be long
and add blank=True.
Only hard code AATSR directions in one place

The AATSR direction (forward and nadir) names were hard coded in two
different files. Change so they are only defined once then saved to a
variable to be used elsewhere, so it's easier to maintain in case the
names need to change.
Use nanmean for viewing angles

Use numpy.nanmean instead of regular mean when calculating mean
viewing angles to store in database - ignores any nan values that
may crop up after regridding if our source image doesn't totally
cover the region of interest.
Add variable names and wavelengths to file readers

Set the list of wavelengths and the variable names in each instrument's reader
instead of the user having to know the variable names and putting them in the
metadata file

Then, when we create the database entry, create the InstrumentWavelength objects.
Fix regridding issue that caused empty boxes on the edges

We were getting a row of nans along the  final column and row of the
regridded array. This is because we were only searching within the
min/max values of the coordinates, but by then using int() to calculate
the grid index, we are treating the coord values as the edges of the pixel,
not the centre. Thus we need to search up to "max lon/lat + pixel width"
to fill the end boxes.

The other option was to treat our new lon/lat values as the centre of the
pixel, but GeoTiff specification assumes the corner value it is given is
for the pixel edge, so we'll stick with that.

NB in any case, we are still assuming that the "old" coordinates arrays from
the original input files are for the pixel centres (ie treat them as point
measurements).
Refactor IngestImages class

- data is now an instance attribute, rather than being passed as an argument
- Define the inputdir and outdir attributes in __init__()
- New methods ingest_all() and ingest_image() replace ingest_images(), so user
  can choose between ingesting a single image (by passing a metadata file name)
  or ingesting all the images in the inputdir
Add scaling to HDF files

Read in the scale factor and offset values from HDF file variables, then
apply to data arrays before doing any other processing.

This is not required for the N1 file reader, as there we use the high level
get_band method, instead of get_dataset, and the scaling is done automatically
behind the scenes.
Add quickview jpg plot

Add a method to produce a jpg RGB plot of the region, for quick checking.
Reference wavelengths are specified for R,G, and B and then the sensor bands
that are closest to these are extracted for the plot.

Also add a unit test for the new method.
Prevent duplicate images being ingested

Use Image.objects.get_or_create, with all fields included as query terms,
to prevent the same image object being ingested twice.