Repository: streetcomplete/StreetComplete (language: kotlin)

Issue:
Improve performance on upload: Notice if nothing changed
## Current behavior

Whenever the map data changes (download, upload) or the user creates or deletes a local edit, `MapDataWithEditsSource` notifies its listeners about the changed map data. 

Most notably amongst the listeners is the `OsmQuestController` which will re-scan the changed element(s) for quests and in turn triggers the map to refresh displaying the quests. This is done for example when the user answered a quest and it takes relatively long (a few hundred milliseconds up to a second or so).

After an edit has been uploaded, the copy of the original OSM element is updated and the edit is deleted. This triggers `MapDataWithEditsSource` to rebuild the cache (=the local changes applied to the original OSM map data) and in turn triggers notifying the listeners about changed.

## The Issue

If the element the edit has been applied to and uploaded has not been changed in the meantime by someone else, which consistitutes about 99% of the times,

- the previous original OSM element + applied edit
- and the updated OSM element after upload

are identical. In the case that the element didn't change at all, notifying the listeners that the map data changed is superfluous and could theoretically be avoided. Currently it is not avoided, which means not only are the local changes rebuilt on upload of each singular edit (I think this is not avoidable) but that updated element is scanned for quests plus the map being updated as a consequence of scanning for quests. Again, for each singular edit.

At least what could be done is to reduce the number of elements for which the listener is called to those elements that actually changed. This requires to compare the edited elements before `rebuildLocalChanges` has been called with the edited elements after it has been called.

## Challenges

- since edits and OSM elements are managed independently (`MapDataController`, `ElementEditsController`) and thus updating one and deleting the other during upload theoretically constitutes of two separate operations, it is possible that `MapDataWithEditsSource` is actually notified of a change in data twice for the same uploaded element. Needs to be confirmed is this actually the case

## Remarks

If you intend to look into this, post here first because @Helium314 vaguely registered interest to busy himself with it before.


Discussion hints:
@Helium314, I think you started some implementation in the frame of another ticket already?

I noticed that the solution for this ticket will not only improve performance on upload, but also on the normal usage for most users, as auto-upload after each change is active. So currently after each quest solution/edit for users that have auto-upload enabled, the quest icons and map data (in case of overlays displayed) are updated twice in fast succession, each with usually the same data. Thus, the solution to this ticket will speed up this normal usage of the app by a factor of 2.
> @Helium314, I think you started some implementation in the frame of another ticket already?

No, I had sone nothing on this.
But I just had a look at https://github.com/streetcomplete/StreetComplete/blob/6e4cd3d1351bae9068fb0d2032509fc021030d60/app/src/main/java/de/westnordost/streetcomplete/data/osm/edits/MapDataWithEditsSource.kt#L61
So before `rebuildLocalChanges`, we need to check whether all keys in `deleted` are in `deletedElements`, and whether all elements and geometries in `updated` are already in `updatedElements`/`updatedGeometries`.
i.e.
```kotlin
val nothingChanged = deletedElements.containsAll(deleted) && updated.all { element ->
    val key = ElementKey(element.type, element.id)
    updatedElements[key] == element && updatedGeometries[key] == updated.getGeometry(element.type, element.id)
}
```
Though elements should probably be allowed to be equal except for the timestamp and maybe version.

If all is as expected, there is no need to call `callOnUpdated`... right?
Yes, exactly, on all points, including the note about `version` and `timestamp`. Hence, the code above won't work because IIRC the `version` of the local element is never updated and the `timestamp` will always differ by at least by a few milliseconds.

I am not sure if it is necessary to compare the geometry. If the geometry changed, it means some other node changed too, so that should be caught by the first check. The geometry is aggregate/processed data from the raw OSM data anyway.

So, only everything except that should be compared. And if it is the same, the elements in deletedElements/updatedElements/updatedGeometries should be **removed** from `MapDataWithEditsSource` because the most current element is now from `MapDataController`. Plus of course, most importantly,

> If all is as expected, there is no need to call callOnUpdated... right?

this.
> I am not sure if it is necessary to compare the geometry. If the geometry changed, it means some other node changed too, so that should be caught by the first check.

Right, this is probably not necessary.

> And if it is the same, the elements in deletedElements/updatedElements/updatedGeometries should be **removed** from `MapDataWithEditsSource` because the most current element is now from `MapDataController`.

Actually deletedElements/updatedElements/updatedGeometries are cleared as part of `rebuildLocalChanges`, which is why I mentioned that this check should happen before `rebuildLocalChanges`.
And we need to rebuild anyway, because there may be other local changes on those elements. (at least currently I don't see good a way to avoid rebuilding).

When uploading, the check will only find if a single edit was done on that element.
If there were 2 edits to the element (one was just synced, one remains locally), `nothingChanged` will return false because the updated element is missing the second change.
But I think catching this case is probably much more work...
Hmm, good point. So the current solution would only work when auto-upload is on, where every single change is uploaded immediately.

I see two avenues to explore here:
1. Can the `rebuildLocalChanges` be made so that not the complete local changes must be rebuild but a few elements (what has been passed into that function) too? I.e. the local changes are being "re-based"
2. Can at least update call to the listeners be avoided by detecting that none of the elements passed into the function have changed?

> 2. Can at least update call to the listeners be avoided by detecting that none of the elements passed into the function have changed?

That _might_ be simply:
before rebuilding local changes: `val e = updated.map { get(it.type, it.id) }`
right before `callOnUpdated`: `val nothingChanged = e.all { it == get(it.type, it.id) }`
Also, even if it only works if just one edit has been made on an element, in many cases it will still be worth it because even if you have thousands of local edits, they might just as well be each on different elements and not (most) on top of each other on the same element(s)
first version: https://github.com/streetcomplete/StreetComplete/commit/4cdd85835beda00e4be3386af4c12d067abbe5e1
second version: https://github.com/streetcomplete/StreetComplete/compare/master...Helium314:StreetComplete:avoid_calling_listeners

I didn't even test whether it compiles, and may have overlooked something...
[Edit: small fix for the second version... but even if this works: it looks rather ugly and is hard to read]
Hmm, actually the second check could be done right after `rebuildLocalChanges`, and `return` immediately if nothing was changed.
I did some test:
Store the `somethingMayHaveChanged` result in a companion object, and if it's false `OsmQuestController.onUpdated` logs whether `deleted.isEmpty()` and `previousQuests` and `quests` contain the same keys / positions.
Every time this was logged, both came out true, i.e. no quests changed.
Sorry, you lost me there. I don't understand how this test was relevant. If no elements changed, of course the quests also do not change(?)
The test was
* whether the checks actually ever return that nothing is changed (nothing overlooked)
* that if they detect that nothing has changed, then really no quests change

Pull request: Don't call MapDataWithEditsSource listeners if nothing changed
fixes #4077 

This is a longer version of what was linked in https://github.com/streetcomplete/StreetComplete/issues/4077#issuecomment-1265693266, but it's not doing more work and I think it's easier to read that way.


Test command: ./gradlew :app:testDebugUnitTest --tests "de.westnordost.streetcomplete.data.osm.edits.MapDataWithEditsSourceTest"

Fix patch:
diff --git a/app/src/main/java/de/westnordost/streetcomplete/data/osm/edits/MapDataWithEditsSource.kt b/app/src/main/java/de/westnordost/streetcomplete/data/osm/edits/MapDataWithEditsSource.kt
index 844147c20a8..8e6f803918d 100644
--- a/app/src/main/java/de/westnordost/streetcomplete/data/osm/edits/MapDataWithEditsSource.kt
+++ b/app/src/main/java/de/westnordost/streetcomplete/data/osm/edits/MapDataWithEditsSource.kt
@@ -62,8 +62,35 @@ class MapDataWithEditsSource internal constructor(
             val modifiedElements = ArrayList<Pair<Element, ElementGeometry?>>()
             val modifiedDeleted = ArrayList<ElementKey>()
             synchronized(this) {
+                /* We don't want to callOnUpdated if none of the changes affects map data provided
+                 * by MapDataWithEditsSource.
+                 * This is the case if
+                 *  * All keys in deleted are already in deletedElements.
+                 *  * The modified versions of all elements in updated are the same before and after
+                 *    rebuildLocalChanges, except for the timestamp (expected to have few ms
+                 *    difference) and version (never updated locally).
+                 */
+                val deletedIsUnchanged = deletedElements.containsAll(deleted)
+                val elementsThatMightHaveChangedByKey = updated.mapNotNull { element ->
+                    val key = ElementKey(element.type, element.id)
+                    if (element.isEqualExceptVersionAndTimestamp(updatedElements[key]))
+                        null // we already have the updated version, so this element is unchanged
+                    else
+                        key to get(element.type, element.id) // elementKey and element as provided by MapDataWithEditsSource
+                }
+
                 rebuildLocalChanges()
 
+                /* nothingChanged can be false at this point when e.g. there are two edits on the 
+                   same element, and onUpdated is called after the first edit is uploaded. */
+                val nothingChanged = deletedIsUnchanged && elementsThatMightHaveChangedByKey.all {
+                    val updatedElement = get(it.first.type, it.first.id)
+                    // old and new elements are equal except version and timestamp, or both are null
+                    it.second?.isEqualExceptVersionAndTimestamp(updatedElement) ?: (updatedElement == null)
+                }
+                if (nothingChanged)
+                    return
+
                 for (element in updated) {
                     val key = ElementKey(element.type, element.id)
                     // an element contained in the update that was deleted by an edit shall be deleted
@@ -438,3 +465,10 @@ class MapDataWithEditsSource internal constructor(
         listeners.forEach { it.onCleared() }
     }
 }
+
+private fun Element.isEqualExceptVersionAndTimestamp(element: Element?): Boolean =
+    id == element?.id && tags == element.tags && type == element.type && when (this) {
+        is Node -> position == (element as Node).position
+        is Way -> nodeIds == (element as Way).nodeIds
+        is Relation -> members == (element as Relation).members
+    }


Test patch:
diff --git a/app/src/test/java/de/westnordost/streetcomplete/data/osm/edits/MapDataWithEditsSourceTest.kt b/app/src/test/java/de/westnordost/streetcomplete/data/osm/edits/MapDataWithEditsSourceTest.kt
index 77b07a834a3..50af86e1703 100644
--- a/app/src/test/java/de/westnordost/streetcomplete/data/osm/edits/MapDataWithEditsSourceTest.kt
+++ b/app/src/test/java/de/westnordost/streetcomplete/data/osm/edits/MapDataWithEditsSourceTest.kt
@@ -35,6 +35,7 @@ import org.junit.Test
 import org.mockito.ArgumentMatchers.anyLong
 import org.mockito.Mockito.verify
 import org.mockito.Mockito.verifyNoInteractions
+import org.mockito.Mockito.verifyNoMoreInteractions
 
 class MapDataWithEditsSourceTest {
 
@@ -977,6 +978,117 @@ class MapDataWithEditsSourceTest {
         verify(listener).onUpdated(eq(expectedMapDataWithGeometry), eq(expectedDeletions))
     }
 
+    @Test
+    fun `does not call onUpdated when all deleted elements are already deleted`() {
+        mapDataChangesAre(deletions = listOf(node(4)))
+
+        val s = create()
+        val listener = mock<MapDataWithEditsSource.Listener>()
+        s.addListener(listener)
+
+        val updatedMapData = MutableMapDataWithGeometry()
+        val deletions = listOf(ElementKey(NODE, 4))
+
+        mapDataListener.onUpdated(updatedMapData, deletions)
+
+        verifyNoInteractions(listener)
+    }
+
+    @Test
+    fun `does call onUpdated when not all deleted elements are already deleted`() {
+        mapDataChangesAre(deletions = listOf(node(4)))
+
+        val s = create()
+        val listener = mock<MapDataWithEditsSource.Listener>()
+        s.addListener(listener)
+
+        val updatedMapData = MutableMapDataWithGeometry()
+        val deletions = listOf(ElementKey(NODE, 3), ElementKey(NODE, 4))
+        mapDataListener.onUpdated(updatedMapData, deletions)
+
+        val expectedDeletions = listOf(
+            ElementKey(NODE, 3),
+            ElementKey(NODE, 4)
+        )
+        verify(listener).onUpdated(eq(updatedMapData), eq(expectedDeletions))
+    }
+
+    @Test
+    fun `does not call onUpdated when all updated elements stayed the same`() {
+        val ndModified = node(1, p(0.3, 0.0))
+        val pModified = ElementGeometryEntry(NODE, 1, pGeom(0.3, 0.0))
+        val ndModified4 = node(4, p(0.5, 0.4))
+        val pModified4 = ElementGeometryEntry(NODE, 4, pGeom(0.5, 0.4))
+
+        mapDataChangesAre(modifications = listOf(ndModified, ndModified4))
+
+        val s = create()
+        val listener = mock<MapDataWithEditsSource.Listener>()
+        s.addListener(listener)
+
+        val updatedMapData = MutableMapDataWithGeometry(
+            elements = listOf(ndModified, ndModified4),
+            geometryEntries = listOf(pModified, pModified4)
+        )
+        mapDataListener.onUpdated(updatedMapData, emptyList())
+
+        verifyNoInteractions(listener)
+    }
+
+    @Test
+    fun `does not call onUpdated when all updated elements stayed the same except for version and timestamp`() {
+        val ndModified = node(1, p(0.3, 0.0))
+        val pModified = ElementGeometryEntry(NODE, 1, pGeom(0.3, 0.0))
+        val ndModified4 = node(4, p(0.5, 0.4))
+        val pModified4 = ElementGeometryEntry(NODE, 4, pGeom(0.5, 0.4))
+
+        val ndModifiedWithVersion = node(1, p(0.3, 0.0), version = 2)
+        val ndModified4WithTimestamp = node(4, p(0.5, 0.4), timestamp = 123L)
+
+        val s = create()
+        val listener = mock<MapDataWithEditsSource.Listener>()
+        s.addListener(listener)
+
+        editsControllerNotifiesMapDataChangesAdded(modifications = listOf(ndModified, ndModified4))
+        originalElementsAre(ndModifiedWithVersion, ndModified4WithTimestamp)
+        verify(listener).onUpdated(any(), any())
+
+        val updatedMapData = MutableMapDataWithGeometry(
+            elements = listOf(ndModifiedWithVersion, ndModified4WithTimestamp),
+            geometryEntries = listOf(pModified, pModified4)
+        )
+        mapDataListener.onUpdated(updatedMapData, emptyList())
+
+        verifyNoMoreInteractions(listener)
+    }
+
+    @Test
+    fun `does call onUpdated when not all updated elements stayed the same`() {
+        val ndModified = node(1, p(0.3, 0.0))
+        val ndModified4 = node(4, p(0.5, 0.4))
+        val pModified4 = ElementGeometryEntry(NODE, 4, pGeom(0.5, 0.4))
+
+        val ndModifiedMoved = node(1, p(0.3, 0.1))
+        val pModifiedMoved = ElementGeometryEntry(NODE, 1, pGeom(0.3, 0.1))
+
+        val s = create()
+        val listener = mock<MapDataWithEditsSource.Listener>()
+        s.addListener(listener)
+
+        editsControllerNotifiesMapDataChangesAdded(modifications = listOf(ndModified, ndModified4))
+        originalElementsAre(ndModifiedMoved, ndModified4)
+        verify(listener).onUpdated(any(), any())
+
+        originalElementsAre(ndModified4, ndModifiedMoved)
+        val updatedMapData = MutableMapDataWithGeometry(
+            elements = listOf(ndModifiedMoved, ndModified4),
+            geometryEntries = listOf(pModifiedMoved, pModified4)
+        )
+        mapDataListener.onUpdated(updatedMapData, emptyList())
+
+        verify(listener).onUpdated(eq(updatedMapData), eq(emptyList()))
+    }
+
     //endregion
 
     //region MapDataController.Listener ::onReplacedForBBox