Repository: streetcomplete/StreetComplete (language: kotlin)

Issue:
Improve performance on upload: Only fetch newest element from API if necessary
## Current Behavior

On upload of edits to OSM data,

1. first the app downloads the newest version of the element the edit refers to,
2. then applies the edit to it locally
3. and then uploads the edited element

This means that for every singular edit, two OSM API calls are made.

## The Issue

While this makes the implementation easier, the first call the OSM API is avoidable in 99% of cases: If the OSM element hasn't been edited in the meantime since the the app downloaded it, the app could just apply the edit to the locally persisted OSM element and upload that.

Only if the OSM API responds with a HTTP 409 Conflict error, we know that the upload needs to fetch the newest version of that element, apply the edit and try again once.

This improvement not only makes the upload faster, it also avoids unnecessary internet traffic.

## Challenges

- oftentimes, multiple edits are made on the same element in a row (e.g. add building type, add housenumber, add building levels). An implementation should keep this in mind - it should not run into HTTP 409 conflicts in cases where the edit "in the meantime" has just been done by the app itself
- the OSM API returns a HTTP 409 conflict error also in case one tries to upload a change in a changeset that as already been closed. So, this case needs to be handled separately


Discussion hints:
Some crude sketch:

Current upload (in `ElementEditUploader`):
* download element
* create updates (or throw if element was changed too much)
* try uploading changes
* `ConflictException` if changeset was closed or element was changed between download and upload
* on `ConflictException`, create a new changeset and upload again

Improved upload:
* get element from `MapDataController` (in `ElementEditsUploader`)
* forward to `ElementEditUploader` and create updates from this element
* try uploading changes
* `ConflictException` if changeset was closed or if element changed since initial download
* on `ConflictException`, download the element
  * if downloaded element is unchanged, create a new changeset and upload again
  * else create changes with downloaded elements and upload again (or throw if element was changed too much)
    * if this upload results in yet another `ConflictException`, create a new changeset and upload again

The changes are already persisted to database after each upload, so I think the first *Challenge* actually is not an issue.

If what I wrote is reasonable, I'll give it a try.
That sounds reasonable

> if downloaded element is unchanged, create a new changeset and upload again

requires that all `Element`s implement `equals` correctly and I am not sure if it is a given that the `timestamp` will be the same(-> Is the locally stored element being updated after upload?)
Thanks for the hint with the timestamp!
Looks like after uploading the first change (and persisting the changed element), the local timestamp differs from timestamp of the element (re-)downloaded after the upload.
This is probably because local timestamp is more precise, containing millisecond information instead of always ending with `000`.

So how to solve this? Override `equals` to ignore small timestamp differences? Add some `equalsExceptTimestamp`? Use less precise timestamps in `createUpdates`? Something else?
Actually, you do not need to compare the elements. It should be enough to compare the `version` attribute, no?
It should be. But I'm not sure whether this is necessarily true, as I don't know enough about how OSM elements may be changed.

So if you think it's fine, I'll do a simple version check.
Every time anything is changed on an element, the version number is increased. However, I am not sure how it is done locally (-> Is the locally stored element being updated after upload?)
Yes it is, see https://github.com/streetcomplete/StreetComplete/blob/a8d915e022ddf1146b54a3a47ef262eec7baa586/app/src/main/java/de/westnordost/streetcomplete/data/osm/edits/upload/ElementEditsUploader.kt#L59
This is where I found that after uploading the edit, the only difference between downloaded element and local element is the timestamp.

Anyway, I think a simple way of checking everything except timestamp is using
`remoteElement?.copy(timestampEdited = 0) == localElement?.copy(timestampEdited = 0)`
which is not much longer than `remoteElement?.version == localElement?.version`

Pull request: Only fetch newest element from API if necessary
fixes #4078

I did *not* yet test whether the modified upload actually works, need to go for a walk first and find something to change.

Related tests are not building, and I failed at making them work, so help would be appreciated.

Test command: ./gradlew :app:testDebugUnitTest --tests "de.westnordost.streetcomplete.data.osm.edits.upload.ElementEditUploaderTest" --tests "de.westnordost.streetcomplete.data.osm.edits.upload.ElementEditsUploaderTest"

Fix patch:
diff --git a/app/src/main/java/de/westnordost/streetcomplete/ApplicationConstants.kt b/app/src/main/java/de/westnordost/streetcomplete/ApplicationConstants.kt
index 3edd6461143..c1fffa1e8e5 100644
--- a/app/src/main/java/de/westnordost/streetcomplete/ApplicationConstants.kt
+++ b/app/src/main/java/de/westnordost/streetcomplete/ApplicationConstants.kt
@@ -1,5 +1,7 @@
 package de.westnordost.streetcomplete
 
+import de.westnordost.streetcomplete.data.osm.edits.split_way.SplitWayAction
+
 object ApplicationConstants {
     const val NAME = "StreetComplete"
     const val USER_AGENT = NAME + " " + BuildConfig.VERSION_NAME
@@ -63,4 +65,10 @@ object ApplicationConstants {
         // no wiki entry, sounds like it could span large areas
         "power", "pipeline", "railway"
     )
+
+    val EDIT_ACTIONS_NOT_ALLOWED_TO_USE_LOCAL_CHANGES = setOf(
+        /* because this action may edit route relations but route relations are not persisted 
+           locally for performance reasons */
+        SplitWayAction::class
+    )
 }
diff --git a/app/src/main/java/de/westnordost/streetcomplete/data/osm/edits/ElementEditsModule.kt b/app/src/main/java/de/westnordost/streetcomplete/data/osm/edits/ElementEditsModule.kt
index 1770ecf2825..963f51ce664 100644
--- a/app/src/main/java/de/westnordost/streetcomplete/data/osm/edits/ElementEditsModule.kt
+++ b/app/src/main/java/de/westnordost/streetcomplete/data/osm/edits/ElementEditsModule.kt
@@ -12,7 +12,7 @@ import org.koin.dsl.module
 
 val elementEditsModule = module {
     factory { ChangesetAutoCloser(get()) }
-    factory { ElementEditUploader(get(), get()) }
+    factory { ElementEditUploader(get(), get(), get()) }
 
     factory { ElementEditsDao(get(), get(), get()) }
     factory { ElementIdProviderDao(get()) }
diff --git a/app/src/main/java/de/westnordost/streetcomplete/data/osm/edits/upload/ElementEditUploader.kt b/app/src/main/java/de/westnordost/streetcomplete/data/osm/edits/upload/ElementEditUploader.kt
index 3699e2acbc0..9b79af796bd 100644
--- a/app/src/main/java/de/westnordost/streetcomplete/data/osm/edits/upload/ElementEditUploader.kt
+++ b/app/src/main/java/de/westnordost/streetcomplete/data/osm/edits/upload/ElementEditUploader.kt
@@ -6,12 +6,16 @@ import de.westnordost.streetcomplete.data.osm.edits.ElementIdProvider
 import de.westnordost.streetcomplete.data.osm.edits.upload.changesets.OpenChangesetsManager
 import de.westnordost.streetcomplete.data.osm.mapdata.ElementType
 import de.westnordost.streetcomplete.data.osm.mapdata.MapDataApi
+import de.westnordost.streetcomplete.data.osm.mapdata.MapDataChanges
+import de.westnordost.streetcomplete.data.osm.mapdata.MapDataController
+import de.westnordost.streetcomplete.data.osm.mapdata.MapDataRepository
 import de.westnordost.streetcomplete.data.osm.mapdata.MapDataUpdates
 import de.westnordost.streetcomplete.data.upload.ConflictException
 
 class ElementEditUploader(
     private val changesetManager: OpenChangesetsManager,
-    private val mapDataApi: MapDataApi
+    private val mapDataApi: MapDataApi,
+    private val mapDataController: MapDataController,
 ) {
 
     /** Apply the given change to the given element and upload it
@@ -19,22 +23,40 @@ class ElementEditUploader(
      *  @throws ConflictException if element has been changed server-side in an incompatible way
      *  */
     fun upload(edit: ElementEdit, idProvider: ElementIdProvider): MapDataUpdates {
-        val element = edit.fetchElement()
+        val remoteChanges by lazy { edit.action.createUpdates(edit.originalElement, mapDataApi.fetch(edit.elementType, edit.elementId), mapDataApi, idProvider) }
+        val localChanges by lazy { edit.action.createUpdates(edit.originalElement, mapDataController.fetch(edit.elementType, edit.elementId), mapDataController, idProvider) }
 
-        val mapDataChanges = edit.action.createUpdates(edit.originalElement, element, mapDataApi, idProvider)
-
-        return try {
-            val changesetId = changesetManager.getOrCreateChangeset(edit.type, edit.source)
-            mapDataApi.uploadChanges(changesetId, mapDataChanges)
-        } catch (e: ConflictException) {
-            val changesetId = changesetManager.createChangeset(edit.type, edit.source)
-            mapDataApi.uploadChanges(changesetId, mapDataChanges, ApplicationConstants.IGNORED_RELATION_TYPES)
+        return if (edit.action::class in ApplicationConstants.EDIT_ACTIONS_NOT_ALLOWED_TO_USE_LOCAL_CHANGES) {
+            try {
+                uploadChanges(edit, remoteChanges, false)
+            } catch (e: ConflictException) {
+                // probably changeset closed
+                uploadChanges(edit, remoteChanges, true)
+            }
+        } else {
+            try {
+                uploadChanges(edit, localChanges, false)
+            } catch (e: ConflictException) {
+                // either changeset was closed, or element modified, or local element was cleaned from db
+                try {
+                    uploadChanges(edit, remoteChanges, false)
+                } catch (e: ConflictException) {
+                    // probably changeset closed
+                    uploadChanges(edit, remoteChanges, true)
+                }
+            }
         }
     }
 
-    private fun ElementEdit.fetchElement() = when (elementType) {
-        ElementType.NODE     -> mapDataApi.getNode(elementId)
-        ElementType.WAY      -> mapDataApi.getWay(elementId)
-        ElementType.RELATION -> mapDataApi.getRelation(elementId)
+    private fun uploadChanges(edit: ElementEdit, mapDataChanges: MapDataChanges, newChangeset: Boolean): MapDataUpdates {
+        val changesetId = if (newChangeset) changesetManager.createChangeset(edit.type, edit.source)
+            else changesetManager.getOrCreateChangeset(edit.type, edit.source)
+        return mapDataApi.uploadChanges(changesetId, mapDataChanges, ApplicationConstants.IGNORED_RELATION_TYPES)
+    }
+
+    private fun MapDataRepository.fetch(elementType: ElementType, elementId: Long) = when (elementType) {
+        ElementType.NODE     -> getNode(elementId)
+        ElementType.WAY      -> getWay(elementId)
+        ElementType.RELATION -> getRelation(elementId)
     }
 }
diff --git a/app/src/main/java/de/westnordost/streetcomplete/data/osm/mapdata/MapDataController.kt b/app/src/main/java/de/westnordost/streetcomplete/data/osm/mapdata/MapDataController.kt
index 526014f8047..91729b4b0e1 100644
--- a/app/src/main/java/de/westnordost/streetcomplete/data/osm/mapdata/MapDataController.kt
+++ b/app/src/main/java/de/westnordost/streetcomplete/data/osm/mapdata/MapDataController.kt
@@ -20,7 +20,7 @@ class MapDataController internal constructor(
     private val geometryDB: ElementGeometryDao,
     private val elementGeometryCreator: ElementGeometryCreator,
     private val createdElementsController: CreatedElementsController
-) {
+) : MapDataRepository {
 
     /* Must be a singleton because there is a listener that should respond to a change in the
      * database table */
@@ -175,9 +175,9 @@ class MapDataController internal constructor(
         )
     }
 
-    fun getNode(id: Long): Node? = nodeDB.get(id)
-    fun getWay(id: Long): Way? = wayDB.get(id)
-    fun getRelation(id: Long): Relation? = relationDB.get(id)
+    override fun getNode(id: Long): Node? = nodeDB.get(id)
+    override fun getWay(id: Long): Way? = wayDB.get(id)
+    override fun getRelation(id: Long): Relation? = relationDB.get(id)
 
     fun getAll(elementKeys: Collection<ElementKey>): List<Element> = elementDB.getAll(elementKeys)
 
@@ -185,10 +185,26 @@ class MapDataController internal constructor(
     fun getWays(ids: Collection<Long>): List<Way> = wayDB.getAll(ids)
     fun getRelations(ids: Collection<Long>): List<Relation> = relationDB.getAll(ids)
 
-    fun getWaysForNode(id: Long): List<Way> = wayDB.getAllForNode(id)
-    fun getRelationsForNode(id: Long): List<Relation> = relationDB.getAllForNode(id)
-    fun getRelationsForWay(id: Long): List<Relation> = relationDB.getAllForWay(id)
-    fun getRelationsForRelation(id: Long): List<Relation> = relationDB.getAllForRelation(id)
+    override fun getWaysForNode(id: Long): List<Way> = wayDB.getAllForNode(id)
+    override fun getRelationsForNode(id: Long): List<Relation> = relationDB.getAllForNode(id)
+    override fun getRelationsForWay(id: Long): List<Relation> = relationDB.getAllForWay(id)
+    override fun getRelationsForRelation(id: Long): List<Relation> = relationDB.getAllForRelation(id)
+
+    override fun getWayComplete(id: Long): MapData? {
+        val way = getWay(id) ?: return null
+        val nodeIds = way.nodeIds.toSet()
+        val nodes = getNodes(nodeIds)
+        if (nodes.size < nodeIds.size) return null
+        return MutableMapData(nodes + way)
+    }
+
+    override fun getRelationComplete(id: Long): MapData? {
+        val relation = getRelation(id) ?: return null
+        val elementKeys = relation.members.map { ElementKey(it.type, it.ref) }.toSet()
+        val elements = getAll(elementKeys)
+        if (elements.size < elementKeys.size) return null
+        return MutableMapData(elements + relation)
+    }
 
     fun deleteOlderThan(timestamp: Long, limit: Int? = null): Int {
         val elements: List<ElementKey>


Test patch:
diff --git a/app/src/test/java/de/westnordost/streetcomplete/data/osm/edits/upload/ElementEditUploaderTest.kt b/app/src/test/java/de/westnordost/streetcomplete/data/osm/edits/upload/ElementEditUploaderTest.kt
index 2682bafb3ba..778a64f6a3f 100644
--- a/app/src/test/java/de/westnordost/streetcomplete/data/osm/edits/upload/ElementEditUploaderTest.kt
+++ b/app/src/test/java/de/westnordost/streetcomplete/data/osm/edits/upload/ElementEditUploaderTest.kt
@@ -2,6 +2,7 @@ package de.westnordost.streetcomplete.data.osm.edits.upload
 
 import de.westnordost.streetcomplete.data.osm.edits.upload.changesets.OpenChangesetsManager
 import de.westnordost.streetcomplete.data.osm.mapdata.MapDataApi
+import de.westnordost.streetcomplete.data.osm.mapdata.MapDataController
 import de.westnordost.streetcomplete.data.osm.mapdata.MapDataUpdates
 import de.westnordost.streetcomplete.data.upload.ConflictException
 import de.westnordost.streetcomplete.testutils.any
@@ -14,46 +15,73 @@ import de.westnordost.streetcomplete.testutils.way
 import org.junit.Before
 import org.junit.Test
 import org.mockito.ArgumentMatchers.anyLong
+import org.mockito.Mockito.verify
 import org.mockito.Mockito.doThrow
 
 class ElementEditUploaderTest {
 
     private lateinit var changesetManager: OpenChangesetsManager
     private lateinit var mapDataApi: MapDataApi
+    private lateinit var mapDataController: MapDataController
     private lateinit var uploader: ElementEditUploader
 
     @Before fun setUp() {
         changesetManager = mock()
         mapDataApi = mock()
+        mapDataController = mock()
 
-        uploader = ElementEditUploader(changesetManager, mapDataApi)
+        uploader = ElementEditUploader(changesetManager, mapDataApi, mapDataController)
     }
 
     @Test(expected = ConflictException::class)
     fun `throws deleted exception if node is no more`() {
         on(mapDataApi.getNode(12)).thenReturn(null)
+        on(mapDataController.getNode(12)).thenReturn(node(12))
+        on(mapDataApi.uploadChanges(anyLong(), any(), any())).thenThrow(ConflictException())
         uploader.upload(edit(element = node(12)), mock())
     }
 
     @Test(expected = ConflictException::class)
     fun `throws deleted exception if way is no more`() {
         on(mapDataApi.getWay(12)).thenReturn(null)
+        on(mapDataController.getWay(12)).thenReturn(way(12))
+        on(mapDataApi.uploadChanges(anyLong(), any(), any())).thenThrow(ConflictException())
         uploader.upload(edit(element = way(12)), mock())
     }
 
     @Test(expected = ConflictException::class)
     fun `throws deleted exception if relation is no more`() {
         on(mapDataApi.getRelation(12)).thenReturn(null)
+        on(mapDataApi.getRelation(12)).thenReturn(rel(12))
+        on(mapDataApi.uploadChanges(anyLong(), any(), any())).thenThrow(ConflictException())
         uploader.upload(edit(element = rel(12)), mock())
     }
 
+    @Test
+    fun `doesn't download element if no exception`() {
+        on(mapDataApi.getNode(12)).thenThrow(NullPointerException()) // ConflictException is handled!
+        on(mapDataController.getNode(12)).thenReturn(node(12))
+        uploader.upload(edit(element = node(12)), mock())
+    }
+
+    @Test
+    fun `downloads element on exception`() {
+        on(mapDataApi.getNode(12)).thenReturn(node(12))
+        on(mapDataController.getNode(12)).thenReturn(node(12))
+        on(mapDataApi.uploadChanges(anyLong(), any(), any())).thenThrow(ConflictException()).thenReturn(null)
+        uploader.upload(edit(element = node(12)), mock())
+        verify(mapDataController).getNode(12)
+        verify(mapDataApi).getNode(12)
+    }
+
     @Test(expected = ConflictException::class)
     fun `passes on element conflict exception`() {
         val node = node(1)
         on(mapDataApi.getNode(anyLong())).thenReturn(node)
+        on(mapDataController.getNode(anyLong())).thenReturn(node)
         on(changesetManager.getOrCreateChangeset(any(), any())).thenReturn(1)
         on(changesetManager.createChangeset(any(), any())).thenReturn(1)
-        on(mapDataApi.uploadChanges(anyLong(), any()))
+        on(mapDataApi.uploadChanges(anyLong(), any(), any()))
             .thenThrow(ConflictException())
             .thenThrow(ConflictException())
 
@@ -63,10 +91,11 @@ class ElementEditUploaderTest {
     @Test fun `handles changeset conflict exception`() {
         val node = node(1)
         on(mapDataApi.getNode(anyLong())).thenReturn(node)
+        on(mapDataController.getNode(anyLong())).thenReturn(node)
         on(changesetManager.getOrCreateChangeset(any(), any())).thenReturn(1)
         on(changesetManager.createChangeset(any(), any())).thenReturn(1)
         doThrow(ConflictException()).doAnswer { MapDataUpdates() }
-            .on(mapDataApi).uploadChanges(anyLong(), any())
+            .on(mapDataApi).uploadChanges(anyLong(), any(), any())
 
         uploader.upload(edit(element = node(1)), mock())
     }
diff --git a/app/src/test/java/de/westnordost/streetcomplete/data/osm/edits/upload/ElementEditsUploaderTest.kt b/app/src/test/java/de/westnordost/streetcomplete/data/osm/edits/upload/ElementEditsUploaderTest.kt
index c833bd2ff1f..68e7e0c3f61 100644
--- a/app/src/test/java/de/westnordost/streetcomplete/data/osm/edits/upload/ElementEditsUploaderTest.kt
+++ b/app/src/test/java/de/westnordost/streetcomplete/data/osm/edits/upload/ElementEditsUploaderTest.kt
@@ -64,9 +64,11 @@ class ElementEditsUploaderTest {
         val edit = edit()
         val idProvider = mock<ElementIdProvider>()
         val updates = mock<MapDataUpdates>()
+        val node = node(1)
 
         on(elementEditsController.getOldestUnsynced()).thenReturn(edit).thenReturn(null)
         on(elementEditsController.getIdProvider(anyLong())).thenReturn(idProvider)
+        on(mapDataController.get(any(), anyLong())).thenReturn(node)
         on(singleUploader.upload(any(), any())).thenReturn(updates)
 
         uploader.upload()
@@ -84,11 +86,13 @@ class ElementEditsUploaderTest {
         val edit = edit()
         val idProvider = mock<ElementIdProvider>()
         val updatedNode = node()
+        val localNode = node()
 
         on(elementEditsController.getOldestUnsynced()).thenReturn(edit).thenReturn(null)
         on(elementEditsController.getIdProvider(anyLong())).thenReturn(idProvider)
         on(singleUploader.upload(any(), any())).thenThrow(ConflictException())
         on(mapDataApi.getNode(anyLong())).thenReturn(updatedNode)
+        on(mapDataController.get(any(), anyLong())).thenReturn(localNode)
 
         uploader.upload()
 
@@ -106,11 +110,13 @@ class ElementEditsUploaderTest {
     @Test fun `upload catches deleted element exception`() = runBlocking {
         val edit = edit(element = node(1))
         val idProvider = mock<ElementIdProvider>()
+        val node = node(1)
 
         on(elementEditsController.getOldestUnsynced()).thenReturn(edit).thenReturn(null)
         on(elementEditsController.getIdProvider(anyLong())).thenReturn(idProvider)
         on(singleUploader.upload(any(), any())).thenThrow(ConflictException())
         on(mapDataApi.getNode(anyLong())).thenReturn(null)
+        on(mapDataController.get(any(), anyLong())).thenReturn(node)
 
         uploader.upload()