Repository: streetcomplete/StreetComplete (language: kotlin)

Issue:
Upload error on too large bounding box
I am getting some crash reports.

```
de.westnordost.streetcomplete.data[https://de.westnordost.streetcomplete.data/].ApiClientException: Client request(POST https://api.openstreetmap.org/api/0.6/changeset/163137077/upload) invalid: 413 Payload Too Large. Text: "Changeset bounding box size limit exceeded."
at de.westnordost.streetcomplete.data[https://de.westnordost.streetcomplete.data/].osm.mapdata.MapDataApiClient.uploadChanges(SourceFile:42)
at de.westnordost.streetcomplete.data[https://de.westnordost.streetcomplete.data/].osm.mapdata.MapDataApiClient$uploadChanges$1.invokeSuspend(Unknown Source:16)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(SourceFile:33)
at kotlinx.coroutines.DispatchedTask.run[https://kotlinx.coroutines.dispatchedtask.run/](SourceFile:100)
at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run[https://worker.run/](SourceFile:113)
at kotlinx.coroutines.scheduling.TaskImpl.run[https://kotlinx.coroutines.scheduling.taskimpl.run/](SourceFile:89)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(SourceFile:586)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(SourceFile:820)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(SourceFile:717)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run[https://worker.run/](SourceFile:704)
Caused by: io.ktor.client.plugins.ClientRequestException: Client request(POST https://api.openstreetmap.org/api/0.6/changeset/163137077/upload) invalid: 413 Payload Too Large. Text: "Changeset bounding box size limit exceeded."
at io.ktor.client.plugins.DefaultResponseValidationKt$addDefaultResponseValidation$1$1.invokeSuspend(SourceFile:52)
... 8 more
```

Some time ago, the OpenStreetMap API introduced some API limits. 

One of these limits was a maximum bounding box size of changesets, relative to how new the user is.

I understand why it was done. A vandal moved nodes of (renamed to something vulgar) a couple of (hundreds of) ways over half the world to achieve an effect like criss-crossing barrier-tape on the rendered map with said vulgar writing on it. A side-effect was that the rendering pipeline of map renderers was completely overloaded, as suddenly millions and millions of tiles needed to be re-rendered.

The side-effect of this measure is, even though nodes of ways cannot be moved at all in StreetComplete, that this app is affected, because of how the limit was implemented. It is rather annoying that this limit wasn't implemented as a limit for the maximum distance a node (of a way) can be moved, but as a maximum bounding box size.

Well, anyway, we have to handle a 413 HTTP error code on upload now and as a response, create a new changeset, then retry the upload.


Discussion hints:
nan

Pull request: handle HTTP 413 response from OSM API
fixes #6151

Need to add tests (but not tonight). Is the code itself comprehensible, @Helium314 , @FloEdelmann , @matkoniecz ?

Test command: ./gradlew :app:testDebugUnitTest --tests "de.westnordost.streetcomplete.data.osm.edits.upload.ElementEditUploaderTest"

Fix patch:
diff --git a/app/src/main/java/de/westnordost/streetcomplete/data/osm/edits/upload/ElementEditUploader.kt b/app/src/main/java/de/westnordost/streetcomplete/data/osm/edits/upload/ElementEditUploader.kt
index 184e6118c18..5d0644a5386 100644
--- a/app/src/main/java/de/westnordost/streetcomplete/data/osm/edits/upload/ElementEditUploader.kt
+++ b/app/src/main/java/de/westnordost/streetcomplete/data/osm/edits/upload/ElementEditUploader.kt
@@ -1,10 +1,12 @@
 package de.westnordost.streetcomplete.data.osm.edits.upload
 
-import de.westnordost.streetcomplete.ApplicationConstants
+import de.westnordost.streetcomplete.ApplicationConstants.EDIT_ACTIONS_NOT_ALLOWED_TO_USE_LOCAL_CHANGES
+import de.westnordost.streetcomplete.ApplicationConstants.IGNORED_RELATION_TYPES
 import de.westnordost.streetcomplete.data.ConflictException
 import de.westnordost.streetcomplete.data.osm.edits.ElementEdit
 import de.westnordost.streetcomplete.data.osm.edits.ElementIdProvider
 import de.westnordost.streetcomplete.data.osm.edits.upload.changesets.OpenChangesetsManager
+import de.westnordost.streetcomplete.data.osm.mapdata.ChangesetTooLargeException
 import de.westnordost.streetcomplete.data.osm.mapdata.MapDataApiClient
 import de.westnordost.streetcomplete.data.osm.mapdata.MapDataChanges
 import de.westnordost.streetcomplete.data.osm.mapdata.MapDataController
@@ -22,33 +24,59 @@ class ElementEditUploader(
      *  @throws ConflictException if element has been changed server-side in an incompatible way
      */
     suspend fun upload(edit: ElementEdit, getIdProvider: () -> ElementIdProvider): MapDataUpdates {
-        val remoteChanges by lazy { edit.action.createUpdates(RemoteMapDataRepository(mapDataApi), getIdProvider()) }
-        val localChanges by lazy { edit.action.createUpdates(mapDataController, getIdProvider()) }
-
-        val mustUseRemoteData = edit.action::class in ApplicationConstants.EDIT_ACTIONS_NOT_ALLOWED_TO_USE_LOCAL_CHANGES
+        // certain edit types don't allow building changes on top of cached map data
+        val mustUseRemoteData = edit.action::class in EDIT_ACTIONS_NOT_ALLOWED_TO_USE_LOCAL_CHANGES
 
         return if (mustUseRemoteData) {
-            try {
-                uploadChanges(edit, remoteChanges, false)
-            } catch (e: ConflictException) {
-                // probably changeset closed
-                uploadChanges(edit, remoteChanges, true)
-            }
+            uploadUsingRemoteRepo(edit, getIdProvider)
         } else {
+            // we first try to apply the changes onto the element cached locally, then upload...
             try {
-                uploadChanges(edit, localChanges, false)
-            } catch (e: ConflictException) {
-                // either changeset was closed, or element modified, or local element was cleaned from db
+                val localChanges = edit.action.createUpdates(mapDataController, getIdProvider())
                 try {
-                    uploadChanges(edit, remoteChanges, false)
-                } catch (e: ConflictException) {
-                    // probably changeset closed
-                    uploadChanges(edit, remoteChanges, true)
+                    uploadChanges(edit, localChanges, false)
                 }
+                // changeset already too large -> try again with new changeset
+                catch (e: ChangesetTooLargeException) {
+                    uploadChanges(edit, localChanges, true)
+                }
+            }
+            // ...but this can fail for various reasons:
+            // - the changeset is already closed on remote
+            // - the element was modified on remote in the meantime
+            // - there's a conflict when applying the change to the locally cached element
+            // - the element does not exist in the local database (cache was deleted)
+            //
+            // In any case -> try again with remote data
+            catch (e: ConflictException) {
+                uploadUsingRemoteRepo(edit, getIdProvider)
             }
         }
     }
 
+    /**
+     *  Apply the given edit to data downloaded ad-hoc from remote, then upload it.
+     *
+     *  @throws ConflictException if element has been changed on remote in an incompatible way
+     * */
+    private suspend fun uploadUsingRemoteRepo(edit: ElementEdit, getIdProvider: () -> ElementIdProvider): MapDataUpdates {
+        // If a conflict is thrown here, it definitely means that the element has been changed on
+        // remote in an incompatible way. So, we don't catch the exception but exit
+        val remoteChanges = edit.action.createUpdates(RemoteMapDataRepository(mapDataApi), getIdProvider())
+
+        return try {
+            uploadChanges(edit, remoteChanges, false)
+        }
+        // probably changeset was closed -> try again once with new changeset
+        catch (e: ConflictException) {
+            uploadChanges(edit, remoteChanges, true)
+        }
+        // changeset too large -> also try again once with new changeset
+        catch (e: ChangesetTooLargeException) {
+            uploadChanges(edit, remoteChanges, true)
+        }
+    }
+
     private suspend fun uploadChanges(
         edit: ElementEdit,
         changes: MapDataChanges,
@@ -59,6 +87,6 @@ class ElementEditUploader(
         } else {
             changesetManager.getOrCreateChangeset(edit.type, edit.source, edit.position, edit.isNearUserLocation)
         }
-        return mapDataApi.uploadChanges(changesetId, changes, ApplicationConstants.IGNORED_RELATION_TYPES)
+        return mapDataApi.uploadChanges(changesetId, changes, IGNORED_RELATION_TYPES)
     }
 }
diff --git a/app/src/main/java/de/westnordost/streetcomplete/data/osm/mapdata/MapDataApiClient.kt b/app/src/main/java/de/westnordost/streetcomplete/data/osm/mapdata/MapDataApiClient.kt
index a58146812ae..ba8d0827db4 100644
--- a/app/src/main/java/de/westnordost/streetcomplete/data/osm/mapdata/MapDataApiClient.kt
+++ b/app/src/main/java/de/westnordost/streetcomplete/data/osm/mapdata/MapDataApiClient.kt
@@ -40,6 +40,8 @@ class MapDataApiClient(
      *                           is not the same as the one uploading the change
      * @throws AuthorizationException if the application does not have permission to edit the map
      *                                (OAuth scope "write_api")
+     * @throws ChangesetTooLargeException when the [changes] don't fit into the changeset with the given
+     *                               [changesetId] anymore.
      * @throws ConnectionException if a temporary network connection problem occurs
      *
      * @return the updated elements
@@ -60,7 +62,7 @@ class MapDataApiClient(
             return createMapDataUpdates(changedElements, updates, ignoreRelationTypes)
         } catch (e: ClientRequestException) {
             when (e.response.status) {
-                // current element version is outdated, current changeset has been closed already
+                // current element version is outdated or current changeset has been closed already
                 HttpStatusCode.Conflict,
                 // an element referred to by another element does not exist (anymore) or was redacted
                 HttpStatusCode.PreconditionFailed,
@@ -70,6 +72,9 @@ class MapDataApiClient(
                 HttpStatusCode.NotFound -> {
                     throw ConflictException(e.message, e)
                 }
+                HttpStatusCode.PayloadTooLarge -> {
+                    throw ChangesetTooLargeException(e.message, e)
+                }
                 else -> throw e
             }
         }
@@ -209,3 +214,8 @@ data class MapDataChanges(
 sealed interface ElementUpdateAction
 data class UpdateElement(val newId: Long, val newVersion: Int) : ElementUpdateAction
 data object DeleteElement : ElementUpdateAction
+
+/** While adding changes to our changeset, the API reports that the changeset limit is already
+ *  reached. We must create a new changeset */
+class ChangesetTooLargeException(message: String? = null, cause: Throwable? = null) :
+    RuntimeException(message, cause)


Test patch:
diff --git a/app/src/test/java/de/westnordost/streetcomplete/data/osm/edits/upload/ElementEditUploaderTest.kt b/app/src/test/java/de/westnordost/streetcomplete/data/osm/edits/upload/ElementEditUploaderTest.kt
index a41ed651588..6d904c1f40c 100644
--- a/app/src/test/java/de/westnordost/streetcomplete/data/osm/edits/upload/ElementEditUploaderTest.kt
+++ b/app/src/test/java/de/westnordost/streetcomplete/data/osm/edits/upload/ElementEditUploaderTest.kt
@@ -4,19 +4,23 @@ import de.westnordost.streetcomplete.data.ConflictException
 import de.westnordost.streetcomplete.data.osm.edits.ElementEdit
 import de.westnordost.streetcomplete.data.osm.edits.ElementEditAction
 import de.westnordost.streetcomplete.data.osm.edits.upload.changesets.OpenChangesetsManager
+import de.westnordost.streetcomplete.data.osm.mapdata.ChangesetTooLargeException
 import de.westnordost.streetcomplete.data.osm.mapdata.MapDataApiClient
 import de.westnordost.streetcomplete.data.osm.mapdata.MapDataChanges
 import de.westnordost.streetcomplete.data.osm.mapdata.MapDataController
 import de.westnordost.streetcomplete.data.osm.mapdata.MapDataUpdates
 import de.westnordost.streetcomplete.testutils.any
+import de.westnordost.streetcomplete.testutils.eq
 import de.westnordost.streetcomplete.testutils.mock
 import de.westnordost.streetcomplete.testutils.on
 import kotlinx.coroutines.runBlocking
 import org.mockito.ArgumentMatchers.anyBoolean
 import org.mockito.ArgumentMatchers.anyLong
 import org.mockito.Mockito.doThrow
+import org.mockito.Mockito.verify
 import kotlin.test.BeforeTest
 import kotlin.test.Test
+import kotlin.test.assertEquals
 import kotlin.test.assertFailsWith
 
 class ElementEditUploaderTest {
@@ -34,6 +38,29 @@ class ElementEditUploaderTest {
         uploader = ElementEditUploader(changesetManager, mapDataApi, mapDataController)
     }
 
+    @Test fun `create new changeset when changeset is too large`(): Unit = runBlocking {
+        val edit: ElementEdit = mock()
+        val action: ElementEditAction = mock()
+        on(edit.action).thenReturn(action)
+        on(action.createUpdates(any(), any())).thenReturn(MapDataChanges())
+
+        // current changeset is 1
+        on(changesetManager.getOrCreateChangeset(any(), any(), any(), anyBoolean())).thenReturn(1L)
+        // but when uploading using this changeset, exception is thrown
+        on(mapDataApi.uploadChanges(eq(1L), any(), any())).thenThrow(ChangesetTooLargeException())
+
+        // creating a changeset yields id 2
+        on(changesetManager.createChangeset(any(), any(), any())).thenReturn(2)
+        // and uploading changes to this changeset yields some result
+        val mapDataUpdates = MapDataUpdates()
+        on(mapDataApi.uploadChanges(eq(2L), any(), any())).thenReturn(mapDataUpdates)
+
+        assertEquals(
+            mapDataUpdates,
+            uploader.upload(edit, { mock() })
+        )
+    }
+
     @Test fun `passes on conflict exception`(): Unit = runBlocking {
         val edit: ElementEdit = mock()
         val action: ElementEditAction = mock()