Repository: opendatacube/datacube-core

Issue:
Database integrity error ingesting data
### Expected behaviour
I've had a problem ingesting Sentinel-2 data with 1.6rc2+57.g0e4fe679 that I've managed to solve by downgrading to 1.5.4. I've tried to trace the problem back without luck. It seems that the `dataset.id` and `dataset.type.id` are not passed correctly to `psycopg2` which causes db integrity errors. Maybe it's something worth investigating.

### Actual behaviour

```
Traceback (most recent call last):
  File "/home/ubuntu/miniconda/envs/cubeenv2/lib/python3.6/site-packages/datacube/scripts/ingest.py", line 312, in process_tasks
    n_successful += _index_datasets(index, results)
  File "/home/ubuntu/miniconda/envs/cubeenv2/lib/python3.6/site-packages/datacube/scripts/ingest.py", line 269, in _index_datasets
    index.datasets.add(dataset, with_lineage=False, **extra_args)
  File "/home/ubuntu/miniconda/envs/cubeenv2/lib/python3.6/site-packages/datacube/index/_datasets.py", line 188, in add
    process_bunch(dss, dataset, transaction)
  File "/home/ubuntu/miniconda/envs/cubeenv2/lib/python3.6/site-packages/datacube/index/_datasets.py", line 137, in process_bunch
    is_new = transaction.insert_dataset(ds.metadata_doc_without_lineage(), ds.id, ds.type.id)
  File "/home/ubuntu/miniconda/envs/cubeenv2/lib/python3.6/site-packages/datacube/drivers/postgres/_api.py", line 229, in insert_dataset
    metadata=metadata_doc
  File "/home/ubuntu/miniconda/envs/cubeenv2/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/home/ubuntu/miniconda/envs/cubeenv2/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
File "/home/ubuntu/miniconda/envs/cubeenv2/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
  compiled_sql, distilled_params
File "/home/ubuntu/miniconda/envs/cubeenv2/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
  context)
File "/home/ubuntu/miniconda/envs/cubeenv2/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
  exc_info
File "/home/ubuntu/miniconda/envs/cubeenv2/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
  reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/home/ubuntu/miniconda/envs/cubeenv2/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 248, in reraise
  raise value.with_traceback(tb)
File "/home/ubuntu/miniconda/envs/cubeenv2/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
  context)
File "/home/ubuntu/miniconda/envs/cubeenv2/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
  cursor.execute(statement, parameters)

sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) null value in column "metadata_type_ref" violates not-null constraint
DETAIL:  Failing row contains (cb40d20f-73a3-4361-805e-3552c399f47c, null, null, {"id": "cb40d20f-73a3-4361-805e-3552c399f47c", "image": {"bands"..., null, 2018-07-17 12:11:20.857423+00, postgres).
[SQL: 'INSERT INTO agdc.dataset (id, dataset_type_ref, metadata_type_ref, metadata) SELECT %(id)s AS anon_1, %(dataset_type_ref)s AS anon_2, (SELECT agdc.dataset_type.metadata_type_ref \nFROM agdc.dataset_type \nWHERE agd
c.dataset_type.id = %(dataset_type_ref)s) AS metadata_type_ref, %(metadata)s AS anon_3 ON CONFLICT (id) DO NOTHING'] [parameters: {'id': UUID('cb40d20f-73a3-4361-805e-3552c399f47c'), 'dataset_type_ref': None, 'metadata': '
{"format": {"name": "NetCDF"}, "platform": {"code": "Sentinel-2A"}, "instrument": {"name": "MSI"}, "product_type": "S2MSI1C", "product_format": {"nam ... (2391 characters truncated) ... 3, 1600000.0], [5300000.0, 1600000.0
], [5300000.0, 1563738.5542817016], [5262778.866862572, 1568282.0087839924], [5266586.919003753, 1600000.0]]]}}}}'}] (Background on this error at: http://sqlalche.me/e/gkpj)
2018-07-17 12:11:20,872 12344 datacube-ingest ERROR Gather failed
Traceback (most recent call last):
File "/home/ubuntu/miniconda/envs/cubeenv2/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
  context)
File "/home/ubuntu/miniconda/envs/cubeenv2/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
  cursor.execute(statement, parameters)
psycopg2.IntegrityError: null value in column "metadata_type_ref" violates not-null constraint
DETAIL:  Failing row contains (cb40d20f-73a3-4361-805e-3552c399f47c, null, null, {"id": "cb40d20f-73a3-4361-805e-3552c399f47c", "image": {"bands"..., null, 2018-07-17 12:11:20.867503+00, postgres).
```


### Steps to reproduce the behaviour

The below ingest config worked with 1.5.4 but not with 1.6rc2+57.g0e4fe679. The command used:

```
datacube ingest -c ingest.yml --executor multiproc 4
```

```
---
description: Sentinel 2A in Equi7 grid
file_path_template: S2A/S2A_EQUI7_{tile_index[0]}_{tile_index[1]}_{start_time}.nc
global_attributes:
  acknowledgment: TestAcknowledgement
  cdm_data_type: Grid
  coverage_content_type: physicalMeasurement
  history: History
  institution: EODC GmbH
  instrument: ETM
  keywords: keywords
  keywords_vocabulary: GCMD
  license: CC BY Attribution 4.0 International License
  platform: Sentinel 2A
  product_suite: level1
  product_version: 2
  publisher_email: test@example.com
  publisher_name: Test
  publisher_url: www.test.com
  references: References
  source: EODC
  summary: Test 2
  title: Test 1
location: /home/ubuntu/Datacube/data/ingest/
measurements:
- name: 'B01'
  aliases: [band_01, B01, Band1]
  units: '1'
  dtype: uint16
  nodata: -999
  src_varname: 'B01'
  resampling_method: nearest
  spectral_definition:
    response: [0.015297, 0.067133, 0.19593, 0.357242, 0.460571, 0.511598, 0.551051, 0.587385, 0.613478, 0.650014, 0.699199, 0.74543, 0.792047, 0.862824, 0.947307, 0.993878, 1.0, 0.990178, 0.972897, 0.957042, 0.956726, 0.922389, 0.723933, 0.388302, 0.14596, 0.051814, 0.017459, 0.000303]
    wavelength: [430, 431, 432, 433, 434, 435, 436, 437, 438, 439, 440, 441, 442, 443, 444, 445, 446, 447, 448, 449, 450, 451, 452, 453, 454, 455, 456, 457]

etc...

output_type: s2a_level1c_equi7EU
source_type: s2a_level1c_granule
storage:
  chunking:
    time: 1
    x: 200
    y: 200
  crs: PROJCS["Azimuthal_Equidistant",GEOGCS["GCS_WGS_1984",DATUM["D_WGS_1984",SPHEROID["WGS_1984",6378137.0,298.257223563]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.017453292519943295]],PROJECTION["Azimuthal_Equidistant"],PARAMETER["false_easting",5837287.81977],PARAMETER["false_northing",2121415.69617],PARAMETER["central_meridian",24.0],PARAMETER["latitude_of_origin",53.0],UNIT["Meter",1.0]]
  dimension_order:
  - time
  - y
  - x
  driver: NetCDF CF
  resolution:
    x: 10
    y: -10
  tile_size:
    x: 100000.0
    y: 100000.0
```

### Environment information

* 1.6rc2+57.g0e4fe679 and 1.5.4
* miniconda environments on Ubuntu 16.04



Discussion:
Previously asked in issue #503 about other CRS and manualy added the following to both versions.

```
def _write_azimuthal_equidistant_params(crs_var, crs):
    crs_var.grid_mapping_name = 'azimuthal_equidistant'
    crs_var.latitude_of_projection_origin = crs.proj.latitude_of_origin
    crs_var.longitude_of_central_meridian = crs.proj.central_meridian
    crs_var.false_easting = crs.proj.false_easting
    crs_var.false_northing = crs.proj.false_northing


CRS_PARAM_WRITERS = {
    'albers_conic_equal_area': _write_albers_params,
    'sinusoidal': _write_sinusoidal_params,
    'transverse_mercator': _write_transverse_mercator_params,
    'lambert_conformal_conic_2sp': _write_lcc2_params,
    'azimuthal_equidistant': _write_azimuthal_equidistant_params,
}
```



Resolving patch:
diff --git a/datacube/scripts/ingest.py b/datacube/scripts/ingest.py
--- a/datacube/scripts/ingest.py
+++ b/datacube/scripts/ingest.py
@@ -144,6 +144,8 @@ def ensure_output_type(index, config, storage_format, allow_product_changes=Fals
                 raise ValueError("Ingest config differs from the existing output product, "
                                  "but allow_product_changes=False")
             output_type = index.products.update(output_type)
+        else:
+            output_type = existing
     else:
         output_type = index.products.add(output_type)
 


Test patch: