Repository: translate/pootle

Issue:
Transaction management error following refresh_stats
I have the following error on mlo following an update to master and following a full run of `refresh_stats`

The job indicates an issue in file `/sr/fhr/date.lang`

`pootle.core.mixins.treeitem.update_cache_job(<Store: /sr/fhr/date.lang>)`

``` pytb
Traceback (most recent call last):
  File "/var/www/sites/mozilla/env/local/lib/python2.7/site-packages/rq/worker.py", line 568, in perform_job
    rv = job.perform()
  File "/var/www/sites/mozilla/env/local/lib/python2.7/site-packages/rq/job.py", line 495, in perform
    self._result = self.func(*self.args, **self.kwargs)
  File "/var/www/sites/mozilla/src/pootle/core/mixins/treeitem.py", line 683, in update_cache_job
    instance._update_cache_job(keys, decrement)
  File "/var/www/sites/mozilla/src/pootle/core/mixins/treeitem.py", line 528, in _update_cache_job
    self.update_cached(key)
  File "/var/www/sites/mozilla/src/pootle/core/mixins/treeitem.py", line 49, in _statslog
    result = function(instance, *args, **kwargs)
  File "/var/www/sites/mozilla/src/pootle/core/mixins/treeitem.py", line 306, in update_cached
    self.set_cached_value(name, self._calc(name))
  File "/var/www/sites/mozilla/src/pootle/core/mixins/treeitem.py", line 202, in _calc
    return self._calc_wordcount_stats()
  File "/var/www/sites/mozilla/src/pootle/core/mixins/treeitem.py", line 151, in _calc_wordcount_stats
    result = self._get_wordcount_stats()
  File "/var/www/sites/mozilla/src/pootle/apps/pootle_store/models.py", line 2268, in _get_wordcount_stats
    res = self.units.filter(store__name__endswith=proj_ext) \
  File "/var/www/sites/mozilla/src/pootle/apps/pootle_store/models.py", line 1375, in units
    self.require_units()
  File "/var/www/sites/mozilla/src/pootle/apps/pootle_store/models.py", line 1478, in require_units
    self.parse()
  File "/var/www/sites/mozilla/env/local/lib/python2.7/site-packages/django/db/transaction.py", line 394, in inner
    return func(*args, **kwargs)
  File "/var/www/sites/mozilla/src/pootle/apps/pootle_store/models.py", line 1557, in parse
    self.unit_set.all().delete()
  File "/var/www/sites/mozilla/env/local/lib/python2.7/site-packages/django/db/models/query.py", line 556, in delete
    collector.collect(del_query)
  File "/var/www/sites/mozilla/env/local/lib/python2.7/site-packages/django/db/models/deletion.py", line 168, in collect
    reverse_dependency=reverse_dependency)
  File "/var/www/sites/mozilla/env/local/lib/python2.7/site-packages/django/db/models/deletion.py", line 79, in add
    if not objs:
  File "/var/www/sites/mozilla/env/local/lib/python2.7/site-packages/django/db/models/query.py", line 145, in __nonzero__
    self._fetch_all()
  File "/var/www/sites/mozilla/env/local/lib/python2.7/site-packages/django/db/models/query.py", line 966, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/var/www/sites/mozilla/env/local/lib/python2.7/site-packages/django/db/models/query.py", line 265, in iterator
    for row in compiler.results_iter():
  File "/var/www/sites/mozilla/env/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 701, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/var/www/sites/mozilla/env/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 787, in execute_sql
    cursor.execute(sql, params)
  File "/var/www/sites/mozilla/env/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 59, in execute
    self.db.validate_no_broken_transaction()
  File "/var/www/sites/mozilla/env/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 386, in validate_no_broken_transaction
    "An error occurred in the current transaction. You can't "
TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.
```



Discussion:
yep - this is the error that i found when testing this upgrade.

There were quite a lot of similar errors - all seemed to be with the fhr project - im thinking the clue is in how that project is set up

@phlax I wonder if its the format? It is a .lang file instead of .po

@hjbailey yep good spot - that is most likely the prob - ill investigate it further

@dwaynebailey i've tested this out and the error at least in the staging set up gets triggered because these stores have a state less than PARSED - it then checks on the filesystem for the files which don't exist.

can you confirm whether they exist in the migration environment - and if they do can you provide them, ta.


Resolving patch:
diff --git a/pootle/apps/pootle_store/models.py b/pootle/apps/pootle_store/models.py
--- a/pootle/apps/pootle_store/models.py
+++ b/pootle/apps/pootle_store/models.py
@@ -1541,16 +1541,18 @@ def parse(self, store=None):
             old_state = self.state
             self.state = LOCKED
             self.save()
+            keys = []
             try:
                 revision = Revision.incr()
                 for index, unit in enumerate(store.units):
+                    # Dont add duplicates
+                    if unit.getid() in keys:
+                        logging.warning(u'Unable to add duplicate unit: %s'
+                                        % unit.getid())
+                        continue
+                    keys.append(unit.getid())
                     if unit.istranslatable():
-                        try:
-                            self.addunit(unit, index, revision=revision)
-                        except IntegrityError as e:
-                            logging.warning(u'Data integrity error while '
-                                            u'importing unit %s:\n%s',
-                                            unit.getid(), e)
+                        self.addunit(unit, index, revision=revision)
             except:
                 # Something broke, delete any units that got created
                 # and return store state to its original value


Test patch: