Repository: translate/pootle

Issue:
Migrating suggestions shouldn't fail on duplicates
I get the following when migrating old LO data, we really shouldn't have FATAL ERRORs in issues like this.  We should either fail elegantly or at least do the right thing with a duplicate. This just makes you fearful.

```
 > pootle_store:0009_move_data_from_pootle_app_suggestion
 - Migration 'pootle_store:0009_move_data_from_pootle_app_suggestion' is marked for no-dry-run.
FATAL ERROR - The following SQL query failed:
                                 INSERT INTO pootle_store_suggestion
                                (target_hash, target_f, creation_time, review_time,
                                 user_id, reviewer_id, unit_id, state,
                                 translator_comment_f, translation_project_id)
                                VALUES
                                (%s, %s, %s, %s,
                                 %s, %s, %s, %s,
                                 %s, %s)
The error was: (1062, "Duplicate entry '28723025-4723eb5aa8b0cd28c7e09433839b8fae' for key 'unit_id'"
)
```



Discussion:
Without a traceback it is difficult to tell where that error is happening.

That SQL query looks like https://github.com/translate/pootle/blob/739e804f/pootle/apps/pootle_store/migrations/0009_move_data_from_pootle_app_suggestion.py#L129-L137 which is within a `try...except IntegrityError:` so you shouldn't see that error at all.

And can't be https://github.com/translate/pootle/blob/stable/2.6.0/pootle/apps/pootle_store/migrations/0009_move_data_from_pootle_app_suggestion.py#L59 because that shouldn't trigger an `INSERT` since it is a record that already was on the DB.

It's almost definitely the first.   Its the exact same INSERT.  The migration doesn't crash so the fact that it displays is most likely a simple artifact of the situation.

https://github.com/translate/pootle/blob/stable/2.6.0/pootle/apps/pootle_store/migrations/0009_move_data_from_pootle_app_suggestion.py#L134-L135

Seems to indicate that these are indeed handled.  Would be nice to not have the scary noise though.

Is there a way to reproduce this easily?


Resolving patch:
diff --git a/pootle/apps/pootle_store/migrations/0009_move_data_from_pootle_app_suggestion.py b/pootle/apps/pootle_store/migrations/0009_move_data_from_pootle_app_suggestion.py
--- a/pootle/apps/pootle_store/migrations/0009_move_data_from_pootle_app_suggestion.py
+++ b/pootle/apps/pootle_store/migrations/0009_move_data_from_pootle_app_suggestion.py
@@ -96,6 +96,8 @@ def forwards(self, orm):
 
         # all pootle_app_suggestions corresponding to pootle_store_suggestions have been already deleted
 
+        existing_keys = list(orm['pootle_store.Suggestion'].objects.values_list('unit_id', 'target_hash'))
+
         for pas in orm['pootle_app.Suggestion'].objects.all():
             try:
                 # unit isn't deleted
@@ -106,8 +108,18 @@ def forwards(self, orm):
                         # to restore suggestion by submission
                         sub = orm['pootle_statistics.Submission'].objects.get(unit=unit, from_suggestion=pas)
                         target = u"%s" % sub.new_value
+
+                        target_hash = md5(target.encode("utf-8")).hexdigest()
+                        if (unit.pk, target_hash) in existing_keys:
+                            # Do not insert duplicate suggestions.
+                            logging.debug("failed to create duplicated suggestion from pas %d and sub %d" % (pas.id, sub.id))
+                            sub.delete()
+                            stats['DUPLICATED_SUG'] += 1
+                            continue
+                        existing_keys.append((unit.pk, target_hash))
+
                         sugg = {
-                            "target_hash": md5(target.encode("utf-8")).hexdigest(),
+                            "target_hash": target_hash,
                             "target_f": target,
                             "creation_time": pas.creation_time,
                             "review_time": sub.creation_time,


Test patch: