Repository: translate/pootle

Issue:
Change unit revision during store.update() 
Currently we don't change revision during update because our `sync_stores` implementation relies on the `Store.max_unit_revision()` to be higher than the `Store.last_sync_revision` even after Web UI update during `update_stores`.
But we need to increment the revision upon `store.update()` for import functionality.

Let's increment a global revision counter in the beginning of any `store.update()` and set that new value to all units changed during this update. Note that any Web UI update will be increment revision on its own.



Discussion:
@julen, @dwaynebailey, @jleclanche I talked with @iafan about this issue and there is a kind of problem. Current approach doesn't work if `import` is run with a `sync_stores` command at the same moment.
It can't be a problem if we run all `sync` and `update` tasks within RQ background processes (see  #3583). But it's not landed and import/export process doesn't adjusted for handling it via RQ #3786.

So I have to ask once again:  are we ok with this approach?

Yeah, it's fine.

Wrong issue.

@ta2-1 Yeah last night I was wondering about the issue of doing a sync at the same time as an update.  There is a small possibility of an issue.  But I think combined with `--force` and `--overwrite` this issue could be reduced and with the sync optimisation the window is smaller.  So for now I'd say its not a critical issue since we'll eliminate it when the other RQ bits land.

@ta2-1 @jleclanche one concern I have which maybe we need to look at.

I use `--modified-since` in ``list_languages` to protect mlo from updating languages that might be hosted on Pootle but the team is actually not translating on Pootle.  So we get a list of languages that have changed.

A concern now would be that with this change `update_stores` once I've done a update against templates would force a change in revision.  That change in revision could cause me then to commit everything.  I could avoid this I guess if I know the revision used for template updates and make sure that is my last flagged revision.  But this starts to sound hairy.

Thoughts?

@dwaynebailey, if I understood you right, we do the similar thing in EN `sync-update` cycle. And `store.last_sync_revision` field is used for it.
I'm not aware how update against templates works. What does it do exactly? And how should we change revision during this process?


Resolving patch:
diff --git a/pootle/apps/pootle_store/models.py b/pootle/apps/pootle_store/models.py
--- a/pootle/apps/pootle_store/models.py
+++ b/pootle/apps/pootle_store/models.py
@@ -375,7 +375,6 @@ def __init__(self, *args, **kwargs):
         self._target_updated = False
         self._state_updated = False
         self._comment_updated = False
-        self._from_update_stores = False
         self._auto_translated = False
         self._encoding = 'UTF-8'
 
@@ -449,12 +448,14 @@ def save(self, *args, **kwargs):
                     self.state = UNTRANSLATED
                     self.store.mark_dirty(CachedMethods.WORDCOUNT_STATS)
 
-        # Updating unit from the .po file should not change its revision property,
+        # Updating unit from the .po file set its revision property to
+        # a new value (the same for all units during its store updated)
         # since that change doesn't require further sync but note that
         # auto_translated units require further sync
-        if ((not self._from_update_stores or self._auto_translated) and
-            (self._target_updated or self._state_updated
-             or self._comment_updated)):
+        revision = kwargs.pop('revision', None)
+        if revision is not None and not self._auto_translated:
+            self.revision = revision
+        elif self._target_updated or self._state_updated or self._comment_updated:
             self.revision = Revision.incr()
 
         if self.id and hasattr(self, '_save_action'):
@@ -510,7 +511,6 @@ def save(self, *args, **kwargs):
         self._target_updated = False
         self._state_updated = False
         self._comment_updated = False
-        self._from_update_stores = False
         self._auto_translated = False
 
         # update cache only if we are updating a single unit
@@ -734,10 +734,6 @@ def update(self, unit, user=None):
             self.unitid_hash = md5(self.unitid.encode("utf-8")).hexdigest()
             changed = True
 
-        if changed:
-            #TODO: check that Store.update() is in the traceback
-            self._from_update_stores = True
-
         return changed
 
     def update_wordcount(self, auto_translate=False):
@@ -1408,10 +1404,13 @@ def save(self, *args, **kwargs):
 
         if hasattr(self, '_units'):
             index = self.max_index() + 1
+            revision = None
+            if created:
+                revision = Revision.incr()
             for i, unit in enumerate(self._units):
                 unit.store = self
                 unit.index = index + i
-                unit.save()
+                unit.save(revision=revision)
 
         if self.state >= PARSED:
             self.update_dirty_cache()
@@ -1530,10 +1529,11 @@ def parse(self, store=None):
             self.state = LOCKED
             self.save()
             try:
+                revision = Revision.incr()
                 for index, unit in enumerate(store.units):
                     if unit.istranslatable():
                         try:
-                            self.addunit(unit, index)
+                            self.addunit(unit, index, revision=revision)
                         except IntegrityError as e:
                             logging.warning(u'Data integrity error while '
                                             u'importing unit %s:\n%s',
@@ -1547,6 +1547,7 @@ def parse(self, store=None):
                 raise
 
             self.state = PARSED
+            self.last_sync_revision = revision
             self.mark_all_dirty()
             self.save()
             return
@@ -1565,7 +1566,7 @@ def update_index(self, start, delta):
             index=op(F('index'), delta)
         )
 
-    def mark_units_obsolete(self, uids_to_obsolete):
+    def mark_units_obsolete(self, uids_to_obsolete, revision=None):
         """Marks a bulk of units as obsolete.
 
         :param uids_to_obsolete: UIDs of the units to be marked as obsolete.
@@ -1579,9 +1580,7 @@ def mark_units_obsolete(self, uids_to_obsolete):
             unit.store = self
             if not unit.isobsolete():
                 unit.makeobsolete()
-                # FIXME: extreme implicit hazard
-                unit._from_update_stores = True
-                unit.save()
+                unit.save(revision=revision)
                 obsoleted += 1
 
         return obsoleted
@@ -1610,12 +1609,14 @@ def remove_modified_units(self, unit_ids):
 
         return unit_ids
 
-    def update_units(self, store, uids_to_update, uid_index_map, user):
+    def update_units(self, store, uids_to_update, uid_index_map, user,
+                     revision=None):
         """Updates existing units in the store.
 
         :param uids_to_update: UIDs of the units to be updated.
         :param uid_index_map: dictionary of DB ID to index mappings.
         :param user: attribute specific changes to this user.
+        :param revision: set updated unit revision to this value.
         :return: The number of units that were actually updated.
         """
         updated = 0
@@ -1659,7 +1660,7 @@ def update_units(self, store, uids_to_update, uid_index_map, user):
                     unit.reviewed_on = None
                     unit.reviewed_by = None
 
-                unit.save()
+                unit.save(revision=revision)
 
         return updated
 
@@ -1752,6 +1753,7 @@ def update(self, overwrite=False, store=None, only_newer=False):
         self.state = LOCKED
         self.save()
 
+        update_revision = Revision.incr()
         try:
             changes = {
                 'obsolete': 0,
@@ -1792,7 +1794,8 @@ def insert(i1, j1, j2, offset):
                     new_unit_index = i1 + index + 1 + offset
                     uid = unit.getid()
                     if uid not in old_unitid_set:
-                        self.addunit(unit, new_unit_index, user=system)
+                        self.addunit(unit, new_unit_index, user=system,
+                                     revision=update_revision)
                         changes['added'] += 1
                     else:
                         update_unitids[uid] = {'dbid': old_unitids[uid]['dbid'],
@@ -1829,14 +1832,15 @@ def insert(i1, j1, j2, offset):
                     common_dbids.update(set(old_unitids[uid]['dbid']
                                             for uid in old_unitid_list[i1:i2]))
 
-
             # Step N-1: mark obsolete units as such
 
             obsolete_dbids = [old_unitids[uid]['dbid']
                               for uid in old_unitid_set -
                                          old_obsolete_unitid_set -
                                          new_unitid_set]
-            changes['obsolete'] = self.mark_units_obsolete(obsolete_dbids)
+            changes['obsolete'] = \
+                self.mark_units_obsolete(obsolete_dbids,
+                                         revision=update_revision)
 
 
             # Step N: update existing units
@@ -1849,11 +1853,19 @@ def insert(i1, j1, j2, offset):
                 common_dbids = self.remove_modified_units(common_dbids)
 
             changes['updated'] = self.update_units(store, common_dbids,
-                                                   update_unitids, system)
-
+                                                   update_unitids, system,
+                                                   revision=update_revision)
 
             self.file_mtime = disk_mtime
 
+            if (filter(lambda x: changes[x] > 0, changes) and
+                store == self.file.store):
+                if self.last_sync_revision is not None:
+                    changes['unsynced'] = \
+                        self.increment_unsynced_unit_revision(update_revision)
+
+                self.last_sync_revision = update_revision
+
         finally:
             # Unlock store
             self.state = old_state
@@ -1864,6 +1876,20 @@ def insert(i1, j1, j2, offset):
                     self.get_max_unit_revision())
                 )
 
+    def increment_unsynced_unit_revision(self, update_revision):
+        filter_by = {
+            'store': self,
+            'revision__gt': self.last_sync_revision,
+            'revision__lt': update_revision,
+            'state__gt': OBSOLETE,
+        }
+        count = 0
+        for unit in Unit.simple_objects.filter(**filter_by):
+            unit.save(revision=Revision.incr())
+            count += 1
+
+        return count
+
     def serialize(self):
         from django.core.cache import caches
 
@@ -2041,7 +2067,7 @@ def max_index(self):
         """Largest unit index"""
         return max_column(self.unit_set.all(), 'index', -1)
 
-    def addunit(self, unit, index=None, user=None):
+    def addunit(self, unit, index=None, user=None, revision=None):
         if index is None:
             index = self.max_index() + 1
 
@@ -2052,7 +2078,7 @@ def addunit(self, unit, index=None, user=None):
             newunit.submitted_on = timezone.now()
 
         if self.id:
-            newunit.save()
+            newunit.save(revision=revision)
         else:
             # We can't save the unit if the store is not in the
             # database already, so let's keep it in temporary list


Test patch:
diff --git a/tests/data/po/tutorial/ru/update_set_last_sync_revision.po b/tests/data/po/tutorial/ru/update_set_last_sync_revision.po
new file mode 100644
--- /dev/null
+++ b/tests/data/po/tutorial/ru/update_set_last_sync_revision.po
@@ -0,0 +1,20 @@
+msgid ""
+msgstr ""
+"Project-Id-Version: PACKAGE VERSION\n"
+"Report-Msgid-Bugs-To: \n"
+"MIME-Version: 1.0\n"
+"Content-Type: text/plain; charset=UTF-8\n"
+"Content-Transfer-Encoding: 8bit\n"
+"X-Generator: Pootle Tests\n"
+
+#: test.c
+msgid "1"
+msgstr ""
+
+#: fish.c
+msgid "2"
+msgstr "second"
+
+#: fish.c
+msgid "3"
+msgstr "third"
diff --git a/tests/data/po/tutorial/ru/update_set_last_sync_revision_updated.po b/tests/data/po/tutorial/ru/update_set_last_sync_revision_updated.po
new file mode 100644
--- /dev/null
+++ b/tests/data/po/tutorial/ru/update_set_last_sync_revision_updated.po
@@ -0,0 +1,28 @@
+msgid ""
+msgstr ""
+"Project-Id-Version: PACKAGE VERSION\n"
+"Report-Msgid-Bugs-To: \n"
+"MIME-Version: 1.0\n"
+"Content-Type: text/plain; charset=UTF-8\n"
+"Content-Transfer-Encoding: 8bit\n"
+"X-Generator: Pootle Tests\n"
+
+#: test.c
+msgid "1"
+msgstr ""
+
+#: fish.c
+msgid "3"
+msgstr "3"
+
+#: fish.c
+msgid "2"
+msgstr "2"
+
+#: fish.c
+msgid "4"
+msgstr "4"
+
+#: fish.c
+msgid "5"
+msgstr ""
diff --git a/tests/fixtures/models/store.py b/tests/fixtures/models/store.py
--- a/tests/fixtures/models/store.py
+++ b/tests/fixtures/models/store.py
@@ -125,3 +125,11 @@ def ru_update_save_changed_units_po(settings, russian_tutorial, system):
     po_directory = settings.POOTLE_TRANSLATION_DIRECTORY
     return _require_store(russian_tutorial, po_directory,
                           'update_save_changed_units.po')
+
+
+@pytest.fixture
+def ru_update_set_last_sync_revision_po(settings, russian_tutorial, system):
+    """Require the /ru/tutorial/tutorial.po store."""
+    po_directory = settings.POOTLE_TRANSLATION_DIRECTORY
+    return _require_store(russian_tutorial, po_directory,
+                          'update_set_last_sync_revision.po')
diff --git a/tests/models/revision.py b/tests/models/revision.py
--- a/tests/models/revision.py
+++ b/tests/models/revision.py
@@ -18,10 +18,14 @@
 @pytest.mark.django_db
 def test_max_revision(af_tutorial_po):
     """Tests `max_revision()` gets the latest revision."""
+
+    # update a store first, initial_revision = 1 after this update
+    af_tutorial_po.update(overwrite=False, only_newer=False)
+
     initial_max_revision = Unit.max_revision()
     initial_revision = Revision.get()
     assert initial_max_revision == initial_revision
-    assert initial_max_revision == 0
+    assert initial_max_revision == 1
 
     # Let's make 10 translation updates, this must also update their
     # revision numbers
@@ -35,7 +39,7 @@ def test_max_revision(af_tutorial_po):
     assert end_max_revision != initial_max_revision
 
     assert end_revision != initial_revision
-    assert end_revision == 10
+    assert end_revision == 10 + initial_revision
 
 
 @pytest.mark.django_db
diff --git a/tests/models/store.py b/tests/models/store.py
--- a/tests/models/store.py
+++ b/tests/models/store.py
@@ -12,6 +12,10 @@
 
 import pytest
 
+from pootle.core.models import Revision
+
+from .unit import _update_translation
+
 
 @pytest.mark.django_db
 def test_delete_mark_obsolete(af_tutorial_subdir_po):
@@ -121,3 +125,48 @@ def test_update_save_changed_units(ru_update_save_changed_units_po):
         if unit.target == updated_unit.target:
             assert unit.revision == updated_unit.revision
             assert unit.mtime == updated_unit.mtime
+
+
+@pytest.mark.django_db
+def test_update_set_last_sync_revision(ru_update_set_last_sync_revision_po):
+    """Tests setting last_sync_revision after store creation.
+    """
+    store = ru_update_set_last_sync_revision_po
+
+    # Parse a store during first update
+    # store.last_sync_revision is set to the next global revision
+    next_revision = Revision.get() + 1
+    store.update(overwrite=False, only_newer=False)
+    assert store.last_sync_revision == next_revision
+    assert store.get_max_unit_revision() == next_revision
+
+    # store.last_sync_revision is not changed after empty update
+    store.update(overwrite=False, only_newer=False)
+    assert store.last_sync_revision == next_revision
+
+    # any non-empty update sets last_sync_revision to next global revision
+    store.file = 'tutorial/ru/update_set_last_sync_revision_updated.po'
+    next_revision = Revision.get() + 1
+    store.update(overwrite=False, only_newer=False)
+    assert store.last_sync_revision == next_revision
+
+    # store.last_sync_revision is not changed after empty update
+    # (even if it has unsynced units)
+    item_index = 0
+    next_unit_revision = Revision.get() + 1
+    dbunit = _update_translation(store, item_index, {'target': u'first'},
+                                 sync=False)
+    assert dbunit.revision == next_unit_revision
+    store.update(overwrite=False, only_newer=False)
+    assert store.last_sync_revision == next_revision
+
+    # Non-empty update sets store.last_sync_revision to next global revision
+    # (even the store has unsynced units)
+    # There is only one unsynced unit in this case so its revision should be set
+    # next to store.last_sync_revision
+    next_revision = Revision.get() + 1
+    store.file = 'tutorial/ru/update_set_last_sync_revision.po'
+    store.update(overwrite=False, only_newer=False)
+    assert store.last_sync_revision == next_revision
+    unit = store.getitem(item_index)
+    assert unit.revision == store.last_sync_revision + 1
diff --git a/tests/models/unit.py b/tests/models/unit.py
--- a/tests/models/unit.py
+++ b/tests/models/unit.py
@@ -32,6 +32,7 @@ def _update_translation(store, item, new_values, sync=True):
 
     if 'translator_comment' in new_values:
         unit.translator_comment = new_values['translator_comment']
+        unit._comment_updated = True
 
     unit.submitted_on = timezone.now()
     unit.submitted_by = User.objects.get_system_user()