Repository: ResonantGeoData/ResonantGeoData

Issue:
psycopg2.errors.UniqueViolation: duplicate key value error
When running the demo data commands that I have, if the celery worker is set up to run in the background, an integretiy error for duplicate keys happens on the `image_entry.save()` call here:

https://github.com/ResonantGeoData/ResonantGeoData/blob/998a6c3995b4421c3632979a249fb78d66e1108f/rgd/geodata/models/imagery/etl.py#L69

The error:

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "geodata_imageentry_image_file_id_key"
DETAIL:  Key (image_file_id)=(14) already exists.
```

This is making me think that when we create a new `ImageEntry` in the tasks, there is some sort of race condition between jobs for the same `ImageFile`... which shouldn't happen? I'm not really sure what is going on here.
 

## Steps to reproduce

1. Clear the database volume
2. Apply migrations: `docker-compose run --rm django ./manage.py migrate`
3. In one session, launch the celery worker: `docker-compose up celery` and wait until ready
4. In another session, run the Landsat demo data command: `docker-compose run --rm django ./manage.py landsat_data -c 3`
    - Use the changes from #296 
5. Observe the error

## Error Message

<details>

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "geodata_imageentry_image_file_id_key"
DETAIL:  Key (image_file_id)=(14) already exists.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./manage.py", line 28, in <module>
    main()
  File "./manage.py", line 24, in main
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/opt/django-project/rgd/geodata/management/commands/landsat_data.py", line 49, in handle
    helper.load_raster_files(_get_landsat_urls(count))
  File "/opt/django-project/rgd/geodata/management/commands/_data_helper.py", line 80, in load_raster_files
    imentries = load_image_files(
  File "/opt/django-project/rgd/geodata/management/commands/_data_helper.py", line 56, in load_image_files
    result = load_image_files(imfile)
  File "/opt/django-project/rgd/geodata/management/commands/_data_helper.py", line 60, in load_image_files
    read_image_file(entry)
  File "/opt/django-project/rgd/geodata/models/imagery/etl.py", line 129, in read_image_file
    _read_image_to_entry(image_entry, file_path)
  File "/opt/django-project/rgd/geodata/models/imagery/etl.py", line 69, in _read_image_to_entry
    image_entry.save()
  File "/opt/django-project/rgd/geodata/models/common.py", line 51, in save
    super(ModifiableEntry, self).save(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 726, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 763, in save_base
    updated = self._save_table(
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 868, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 906, in _do_insert
    return manager._insert(
  File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 1268, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1410, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 98, in execute
    return super().execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "geodata_imageentry_image_file_id_key"
DETAIL:  Key (image_file_id)=(14) already exists.
```

</details>


Discussion:
If we do not run the celery worker in the background but wait until the demo data command is complete then this error does not occur. I can't put my finger out what exactly is going wrong here yet.
I'm assigning myself on this one.
This is a result of attempting to create multiple `ImageEntry` with the same `ImageFile` foreign key, which is unique by virtue of being a `OneToOneField`.

I don't understand the data model enough to determine the appropriate fix. @banesullivan is this actually a one to one correspondence, or can there be multiple `ImageEntry` for the same `ImageFile`?
> is this actually a one-to-one correspondence, or can there be multiple ImageEntry for the same ImageFile?

This is a on-to-one correspondence in practice. An `ImageEntry` is only ever created and accosiated with an `ImageFile` in `read_image_file` here:

https://github.com/ResonantGeoData/ResonantGeoData/blob/998a6c3995b4421c3632979a249fb78d66e1108f/rgd/geodata/models/imagery/etl.py#L121-L123


This should only ever be called from [`task_read_image_file`](https://github.com/ResonantGeoData/ResonantGeoData/blob/998a6c3995b4421c3632979a249fb78d66e1108f/rgd/geodata/tasks.py#L38) which is the result of an `ImageFile` being saved and it kicking off that task. 

This makes me think my helper, `get_or_create_no_commit`, could have an issue:

https://github.com/ResonantGeoData/ResonantGeoData/blob/998a6c3995b4421c3632979a249fb78d66e1108f/rgd/utility.py#L163-L170

Or one `ImageFile` is launching multiple tasks both attempting to create a new `ImageEntry` at the same time.
Perhaps it is time we do some sort of task auditing to make sure the ETL routines are all only launching one task per model
Yes. I think we'll want to re-engineer a lot of this to make task launching more explicit.
Uncovered what's going wrong here. It is a trivial typo fix.

I added a signal wrapper `skip_signal` here 

https://github.com/ResonantGeoData/ResonantGeoData/blob/998a6c3995b4421c3632979a249fb78d66e1108f/rgd/geodata/signals.py#L21-L33

that checks if a model instance has an `skip_signal` attribute and if so (regardless of value) it will skip sending the task signal. [Someone..](https://github.com/ResonantGeoData/ResonantGeoData/blame/master/rgd/geodata/management/commands/_data_helper.py#L45-L46). 🤦🏻‍♂️ ... used `skip_task` instead of `skip_signal` here:

https://github.com/ResonantGeoData/ResonantGeoData/blob/998a6c3995b4421c3632979a249fb78d66e1108f/rgd/geodata/management/commands/_data_helper.py#L45-L46

which means two tasks are being launched for the `ImageFile` model. 

You can see the `save()` call here launching the task:

https://github.com/ResonantGeoData/ResonantGeoData/blob/998a6c3995b4421c3632979a249fb78d66e1108f/rgd/geodata/management/commands/_data_helper.py#L48

and calling the ETL routine explicitly here:

https://github.com/ResonantGeoData/ResonantGeoData/blob/998a6c3995b4421c3632979a249fb78d66e1108f/rgd/geodata/management/commands/_data_helper.py#L60

We should come up with a safer way to use this `skip_signal` wrapper such that a typo like this doesn't happen again. Arbitrarily setting an attribute on an instance like this isn't a great approach
Turns out #299 didn't entirely fix this issue.

Resolving patch:
diff --git a/rgd/geodata/management/commands/_data_helper.py b/rgd/geodata/management/commands/_data_helper.py
--- a/rgd/geodata/management/commands/_data_helper.py
+++ b/rgd/geodata/management/commands/_data_helper.py
@@ -40,11 +40,12 @@ def _get_or_download_checksum_file(name):
 def _get_or_create_file_model(model, name, skip_signal=False):
     # For models that point to a `ChecksumFile`
     file_entry = _get_or_download_checksum_file(name)
-    entry, _ = model.objects.get_or_create(file=file_entry)
+    # No commit in case we need to skip the signal
+    entry, created = get_or_create_no_commit(model, file=file_entry)
     # In case the last population failed
     if skip_signal:
         entry.skip_signal = True
-    if entry.status != models.mixins.Status.SUCCEEDED:
+    if created or entry.status != models.mixins.Status.SUCCEEDED:
         entry.save()
     return entry
 


Test patch: