Repository: digitalfabrik/entitlementcard
Pull request #2131: 2051: Provide store import information
Author: login: f1sh1918
Created: 2025-04-29T13:44:53Z | Merged: 2025-05-21T15:50:56Z

### Short Description

We get often mails that some accepting stores are not listed in the app. Therefore we provide a csv file that lists filtered stores with the filtering reason.

### Proposed Changes

<!-- Describe this PR in more detail. -->

- write a csv file with filtered stores
- add an output path to config fil (requires salt change https://git.tuerantuer.org/DF/salt/pulls/346)

### Side Effects

<!-- List all related components that have not been explicitly changed but may be affected by this PR -->

- none

### Testing

- You can just set a local path in your `config.local.yml` in the project section to test the csv output (dev)
f.e.
```
filteredStoresOutput: src/main/resources/import/filtered_stores.csv
```
- run the store import (local dev)
- Find csv file: https://staging.bayern.ehrenamtskarte.app/acceptance-stores/filtered_stores.csv (available after release deployment and once import job was triggered)

### Resolved Issues

<!-- List all issues which should be closed when this PR is merged. -->

Fixes: #2051



Diff:
diff --git a/backend/src/main/kotlin/app/ehrenamtskarte/backend/config/BackendConfiguration.kt b/backend/src/main/kotlin/app/ehrenamtskarte/backend/config/BackendConfiguration.kt
index a0757b79f8..3a8c678917 100644
--- a/backend/src/main/kotlin/app/ehrenamtskarte/backend/config/BackendConfiguration.kt
+++ b/backend/src/main/kotlin/app/ehrenamtskarte/backend/config/BackendConfiguration.kt
@@ -42,6 +42,7 @@ data class ProjectConfig(
     val smtp: SmtpConfig,
     val matomo: MatomoConfig?,
     val freinet: FreinetConfig?,
+    val filteredStoresOutput: String?,
 )
 
 data class ServerConfig(val dataDirectory: String, val host: String, val port: String)
diff --git a/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/Importer.kt b/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/Importer.kt
index 7d1d9f311c..106b1205b1 100644
--- a/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/Importer.kt
+++ b/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/Importer.kt
@@ -1,6 +1,7 @@
 package app.ehrenamtskarte.backend.stores.importer
 
 import app.ehrenamtskarte.backend.stores.importer.bayern.EhrenamtskarteBayern
+import app.ehrenamtskarte.backend.stores.importer.bayern.types.FilteredStore
 import app.ehrenamtskarte.backend.stores.importer.pipelines.BerechtigungskarteShowcase
 import org.slf4j.LoggerFactory
 
@@ -8,6 +9,7 @@ object Importer {
     fun import(config: ImportConfig): Boolean {
         val logger = LoggerFactory.getLogger(Importer::class.java)
         val project = config.project
+        val filteredStores: MutableList<FilteredStore> = mutableListOf()
 
         if (project.pipelineName == null) {
             logger.info("There is no store import pipeline for ${project.id}")
@@ -17,8 +19,8 @@ object Importer {
         return try {
             logger.info("== Pipeline ${project.pipelineName} started ==")
             when (project.pipelineName) {
-                "EhrenamtskarteBayern" -> EhrenamtskarteBayern.import(config, logger)
-                "BerechtigungskarteShowcase" -> BerechtigungskarteShowcase.import(config, logger)
+                "EhrenamtskarteBayern" -> EhrenamtskarteBayern.import(config, logger, filteredStores)
+                "BerechtigungskarteShowcase" -> BerechtigungskarteShowcase.import(config, logger, filteredStores)
                 else -> throw Error("Invalid pipeline name '${project.pipelineName}'!")
             }
             logger.info("== Pipeline ${project.pipelineName} successfully finished ==")
diff --git a/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/Logging.kt b/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/Logging.kt
index fb38f4a0c9..42a176722b 100644
--- a/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/Logging.kt
+++ b/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/Logging.kt
@@ -1,5 +1,6 @@
 package app.ehrenamtskarte.backend.stores.importer
 
+import app.ehrenamtskarte.backend.stores.importer.bayern.types.FilteredStore
 import app.ehrenamtskarte.backend.stores.importer.common.types.AcceptingStore
 import org.slf4j.Logger
 
@@ -15,8 +16,10 @@ fun Logger.logChange(store: AcceptingStore, property: String, oldValue: String?,
     logChange(storeInfo(store), property, oldValue, newValue)
 }
 
-fun Logger.logRemoveDuplicates(store: AcceptingStore, count: Int) {
-    info("Removed duplicates ($count) of '${storeInfo(store)}'")
+fun Logger.logRemoveDuplicates(store: AcceptingStore, count: Int, filteredStores: MutableList<FilteredStore>) {
+    val reason = "Removed duplicates ($count) of '${storeInfo(store)}'"
+    info(reason)
+    filteredStores.add(FilteredStore(store, reason))
 }
 
 private fun storeInfo(store: AcceptingStore): String =
diff --git a/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/bayern/EhrenamtskarteBayern.kt b/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/bayern/EhrenamtskarteBayern.kt
index 897460d96e..68f4af030e 100644
--- a/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/bayern/EhrenamtskarteBayern.kt
+++ b/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/bayern/EhrenamtskarteBayern.kt
@@ -4,8 +4,10 @@ import app.ehrenamtskarte.backend.stores.importer.ImportConfig
 import app.ehrenamtskarte.backend.stores.importer.addStep
 import app.ehrenamtskarte.backend.stores.importer.bayern.steps.DownloadLbe
 import app.ehrenamtskarte.backend.stores.importer.bayern.steps.FilterLbe
+import app.ehrenamtskarte.backend.stores.importer.bayern.steps.FilteredStoresCsvOutput
 import app.ehrenamtskarte.backend.stores.importer.bayern.steps.MapFromLbe
 import app.ehrenamtskarte.backend.stores.importer.bayern.steps.PostSanitizeFilter
+import app.ehrenamtskarte.backend.stores.importer.bayern.types.FilteredStore
 import app.ehrenamtskarte.backend.stores.importer.common.steps.FilterDuplicates
 import app.ehrenamtskarte.backend.stores.importer.common.steps.SanitizeAddress
 import app.ehrenamtskarte.backend.stores.importer.common.steps.SanitizeGeocode
@@ -24,12 +26,12 @@ object EhrenamtskarteBayern : Pipeline {
         }
     }
 
-    override fun import(config: ImportConfig, logger: Logger) {
+    override fun import(config: ImportConfig, logger: Logger, filteredStores: MutableList<FilteredStore>) {
         Unit
             .addStep(DownloadLbe(config, logger, httpClient), logger) {
                 logger.info("== Download lbe data ==")
             }
-            .addStep(FilterLbe(config, logger), logger) {
+            .addStep(FilterLbe(config, logger, filteredStores), logger) {
                 logger.info("== Filter lbe data ==")
             }
             .addStep(MapFromLbe(config, logger), logger) {
@@ -41,12 +43,15 @@ object EhrenamtskarteBayern : Pipeline {
             .addStep(SanitizeGeocode(config, logger, httpClient), logger) {
                 logger.info("== Sanitize data with geocoding ==")
             }
-            .addStep(PostSanitizeFilter(config, logger, httpClient), logger) {
+            .addStep(PostSanitizeFilter(config, logger, filteredStores), logger) {
                 logger.info("== Filter sanitized data ==")
             }
-            .addStep(FilterDuplicates(config, logger), logger) {
+            .addStep(FilterDuplicates(config, logger, filteredStores), logger) {
                 logger.info("== Filter duplicated data ==")
             }
+            .addStep(FilteredStoresCsvOutput(config, filteredStores), logger) {
+                logger.info("== Write CSV Filtered Stores ==")
+            }
             .addStep(Store(config, logger), logger) {
                 logger.info("== Store remaining data to db ==")
             }
diff --git a/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/bayern/steps/FilterLbe.kt b/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/bayern/steps/FilterLbe.kt
index 6afb0a793c..7675d15b68 100644
--- a/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/bayern/steps/FilterLbe.kt
+++ b/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/bayern/steps/FilterLbe.kt
@@ -4,6 +4,7 @@ import app.ehrenamtskarte.backend.stores.ALTERNATIVE_MISCELLANEOUS_CATEGORY_ID
 import app.ehrenamtskarte.backend.stores.MISCELLANEOUS_CATEGORY_ID
 import app.ehrenamtskarte.backend.stores.importer.ImportConfig
 import app.ehrenamtskarte.backend.stores.importer.PipelineStep
+import app.ehrenamtskarte.backend.stores.importer.bayern.types.FilteredStore
 import app.ehrenamtskarte.backend.stores.importer.bayern.types.LbeAcceptingStore
 import app.ehrenamtskarte.backend.stores.importer.matchesNa
 import org.slf4j.Logger
@@ -12,7 +13,11 @@ import org.slf4j.Logger
  * Filter and removes [LbeAcceptingStore] with invalid data.
  * These are especially stores without name, location or an invalid category.
  */
-class FilterLbe(config: ImportConfig, private val logger: Logger) :
+class FilterLbe(
+    config: ImportConfig,
+    private val logger: Logger,
+    private val filteredStores: MutableList<FilteredStore>,
+) :
     PipelineStep<List<LbeAcceptingStore>, List<LbeAcceptingStore>>(config) {
     private val invalidLocations = arrayOf("Musterhausen")
 
@@ -23,6 +28,7 @@ class FilterLbe(config: ImportConfig, private val logger: Logger) :
             store.isValidName() && store.isValidCategory() && store.isValidLocation()
         } catch (e: Exception) {
             logger.error("$store was filtered out because of an unknown exception while filtering", e)
+            filteredStores.add(FilteredStore(store, "Unknown"))
             false
         }
 
@@ -31,6 +37,7 @@ class FilterLbe(config: ImportConfig, private val logger: Logger) :
     private fun LbeAcceptingStore.isValidName(): Boolean =
         if (name.isUnavailable()) {
             logger.info("'$this' was filtered out because name '$name' is invalid")
+            filteredStores.add(FilteredStore(this, "Name is invalid"))
             false
         } else {
             true
@@ -39,6 +46,7 @@ class FilterLbe(config: ImportConfig, private val logger: Logger) :
     private fun LbeAcceptingStore.isValidLocation(): Boolean =
         if (location.isUnavailable() || invalidLocations.contains(location)) {
             logger.info("'$name' was filtered out because location '$location' is invalid")
+            filteredStores.add(FilteredStore(this, "Location is invalid"))
             false
         } else {
             true
@@ -50,6 +58,7 @@ class FilterLbe(config: ImportConfig, private val logger: Logger) :
 
         if (!valid) {
             logger.info("'$name' was filtered out because category '$category' is invalid")
+            filteredStores.add(FilteredStore(this, "Category is invalid"))
         }
 
         return valid
diff --git a/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/bayern/steps/FilteredStoresCsvOutput.kt b/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/bayern/steps/FilteredStoresCsvOutput.kt
new file mode 100644
index 0000000000..2df1315076
--- /dev/null
+++ b/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/bayern/steps/FilteredStoresCsvOutput.kt
@@ -0,0 +1,54 @@
+package app.ehrenamtskarte.backend.stores.importer.bayern.steps
+
+import app.ehrenamtskarte.backend.stores.importer.ImportConfig
+import app.ehrenamtskarte.backend.stores.importer.PipelineStep
+import app.ehrenamtskarte.backend.stores.importer.bayern.types.FilteredStore
+import app.ehrenamtskarte.backend.stores.importer.common.types.AcceptingStore
+import org.apache.commons.csv.CSVFormat
+import org.slf4j.LoggerFactory
+import java.nio.file.Files
+import java.nio.file.Paths
+
+class FilteredStoresCsvOutput(config: ImportConfig, private val filteredStores: MutableList<FilteredStore>) :
+    PipelineStep<List<AcceptingStore>, List<AcceptingStore>>(config) {
+    override fun execute(input: List<AcceptingStore>): List<AcceptingStore> {
+        val filteredOutputPath = config.project.filteredStoresOutput
+        if (!filteredOutputPath.isNullOrBlank()) {
+            writeCsvWithFilteredStores(filteredOutputPath)
+        }
+        return input
+    }
+
+    private fun writeCsvWithFilteredStores(outputPath: String) {
+        val logger = LoggerFactory.getLogger(FilteredStoresCsvOutput::class.java)
+
+        data class Col(
+            val name: String,
+            val fromRecord: (store: FilteredStore) -> String?,
+        )
+
+        val columns = listOf(
+            Col("Name") { it.name },
+            Col("Straße") { it.street },
+            Col("Hausnummer") { it.houseNumber },
+            Col("PLZ") { it.postalCode },
+            Col("Ort") { it.location },
+            Col("Grund") { it.reason },
+        )
+
+        try {
+            Files.newBufferedWriter(Paths.get(outputPath)).use { writer ->
+                CSVFormat.RFC4180.builder()
+                    .setHeader(*columns.map { it.name }.toTypedArray())
+                    .get()
+                    .print(writer).use { printer ->
+                        filteredStores.forEach { record ->
+                            printer.printRecord(columns.map { it.fromRecord(record) })
+                        }
+                    }
+            }
+        } catch (e: Exception) {
+            logger.error("Fehler beim Schreiben der CSV:", e)
+        }
+    }
+}
diff --git a/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/bayern/steps/PostSanitizeFilter.kt b/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/bayern/steps/PostSanitizeFilter.kt
index b97e6b7bbe..b8961a25a9 100644
--- a/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/bayern/steps/PostSanitizeFilter.kt
+++ b/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/bayern/steps/PostSanitizeFilter.kt
@@ -2,8 +2,8 @@ package app.ehrenamtskarte.backend.stores.importer.bayern.steps
 
 import app.ehrenamtskarte.backend.stores.importer.ImportConfig
 import app.ehrenamtskarte.backend.stores.importer.PipelineStep
+import app.ehrenamtskarte.backend.stores.importer.bayern.types.FilteredStore
 import app.ehrenamtskarte.backend.stores.importer.common.types.AcceptingStore
-import io.ktor.client.HttpClient
 import kotlinx.coroutines.runBlocking
 import org.slf4j.Logger
 
@@ -11,18 +11,24 @@ import org.slf4j.Logger
  * Filters [AcceptingStore] to prepare storing to the database.
  * Stores without longitude, latitude or postal code are removed.
  */
-class PostSanitizeFilter(config: ImportConfig, private val logger: Logger, httpClient: HttpClient) :
+class PostSanitizeFilter(
+    config: ImportConfig,
+    private val logger: Logger,
+    private val filteredStores: MutableList<FilteredStore>,
+) :
     PipelineStep<List<AcceptingStore>, List<AcceptingStore>>(config) {
     override fun execute(input: List<AcceptingStore>): List<AcceptingStore> =
         runBlocking {
             input.filter {
                 if (it.longitude == null || it.latitude == null) {
                     logger.info("'${it.name}, ${it.location}' was filtered out because longitude or latitude are null")
+                    filteredStores.add(FilteredStore(it, "longitude or latitude are null"))
                     return@filter false
                 }
                 if (it.postalCode == null) {
                     // Probably because it is outside of the state but inside the bounding box of the state
                     logger.info("'${it.name}, ${it.location}' was filtered out because its postal code is null")
+                    filteredStores.add(FilteredStore(it, "Postal code is null"))
                     return@filter false
                 }
                 true
diff --git a/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/bayern/types/FilteredStore.kt b/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/bayern/types/FilteredStore.kt
new file mode 100644
index 0000000000..d2d2476891
--- /dev/null
+++ b/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/bayern/types/FilteredStore.kt
@@ -0,0 +1,21 @@
+package app.ehrenamtskarte.backend.stores.importer.bayern.types
+
+import app.ehrenamtskarte.backend.stores.importer.common.types.AcceptingStore
+
+data class FilteredStore(
+    val name: String?,
+    val street: String?,
+    val houseNumber: String?,
+    val postalCode: String?,
+    val location: String?,
+    val reason: String,
+) {
+    constructor(
+        lbeStore: LbeAcceptingStore,
+        reason: String,
+    ) : this(lbeStore.name, lbeStore.street, lbeStore.houseNumber, lbeStore.postalCode, lbeStore.location, reason)
+    constructor(
+        store: AcceptingStore,
+        reason: String,
+    ) : this(store.name, store.street, store.houseNumber, store.postalCode, store.location, reason)
+}
diff --git a/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/common/steps/FilterDuplicates.kt b/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/common/steps/FilterDuplicates.kt
index cdf0b397c2..8920b893f5 100644
--- a/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/common/steps/FilterDuplicates.kt
+++ b/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/common/steps/FilterDuplicates.kt
@@ -2,6 +2,7 @@ package app.ehrenamtskarte.backend.stores.importer.common.steps
 
 import app.ehrenamtskarte.backend.stores.importer.ImportConfig
 import app.ehrenamtskarte.backend.stores.importer.PipelineStep
+import app.ehrenamtskarte.backend.stores.importer.bayern.types.FilteredStore
 import app.ehrenamtskarte.backend.stores.importer.common.types.AcceptingStore
 import app.ehrenamtskarte.backend.stores.importer.logRemoveDuplicates
 import org.slf4j.Logger
@@ -15,6 +16,7 @@ import java.util.Locale
 class FilterDuplicates(
     config: ImportConfig,
     private val logger: Logger,
+    private val filteredStores: MutableList<FilteredStore>,
 ) : PipelineStep<List<AcceptingStore>, List<AcceptingStore>>(config) {
     override fun execute(input: List<AcceptingStore>): List<AcceptingStore> {
         // Group by name + postal code + street to detect duplicates
@@ -31,7 +33,7 @@ class FilterDuplicates(
         // Use the last as that is perhaps the last updated/created one
         val store = last()
 
-        logger.logRemoveDuplicates(store, size - 1)
+        logger.logRemoveDuplicates(store, size - 1, filteredStores)
         val issuers = map { "'${it.districtName ?: it.freinetId.toString()}'" }.toSet().joinToString()
         logger.info("Duplicates issued by $issuers")
 
diff --git a/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/pipelines/BerechtigungskarteShowcase.kt b/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/pipelines/BerechtigungskarteShowcase.kt
index e0400a4e0f..2464cbcf89 100644
--- a/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/pipelines/BerechtigungskarteShowcase.kt
+++ b/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/pipelines/BerechtigungskarteShowcase.kt
@@ -1,10 +1,11 @@
 package app.ehrenamtskarte.backend.stores.importer.pipelines
 
 import app.ehrenamtskarte.backend.stores.importer.ImportConfig
+import app.ehrenamtskarte.backend.stores.importer.bayern.types.FilteredStore
 import org.slf4j.Logger
 
 object BerechtigungskarteShowcase : Pipeline {
-    override fun import(config: ImportConfig, logger: Logger) {
+    override fun import(config: ImportConfig, logger: Logger, filteredStores: MutableList<FilteredStore>) {
         logger.info("The pipeline for the showcase project is empty!")
     }
 }
diff --git a/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/pipelines/Pipeline.kt b/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/pipelines/Pipeline.kt
index 9f03e6c49c..5fae6d0b4f 100644
--- a/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/pipelines/Pipeline.kt
+++ b/backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/pipelines/Pipeline.kt
@@ -1,8 +1,9 @@
 package app.ehrenamtskarte.backend.stores.importer.pipelines
 
 import app.ehrenamtskarte.backend.stores.importer.ImportConfig
+import app.ehrenamtskarte.backend.stores.importer.bayern.types.FilteredStore
 import org.slf4j.Logger
 
 interface Pipeline {
-    fun import(config: ImportConfig, logger: Logger)
+    fun import(config: ImportConfig, logger: Logger, filteredStores: MutableList<FilteredStore>)
 }
diff --git a/backend/src/main/resources/config/config.yml b/backend/src/main/resources/config/config.yml
index 665141f5dd..6a23f31581 100644
--- a/backend/src/main/resources/config/config.yml
+++ b/backend/src/main/resources/config/config.yml
@@ -20,6 +20,7 @@ matomoUrl: http://localhost:5003/matomo.php
 projects:
   - id: bayern.ehrenamtskarte.app
     importUrl: https://www.ehrenamt.bayern.de/xml-json/app-daten.xml
+    filteredStoresOutput: /var/archive-data/filtered_stores.csv
     pipelineName: EhrenamtskarteBayern
     administrationBaseUrl: https://bayern.ehrenamtskarte.app
     administrationName: Ehrenamtskarte-Bayern-Verwaltung


Reviews:
user: login: f1sh1918
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-30T06:25:04Z
user: login: andrew8er
body: A few suggestions and some corrections.
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T12:56:26Z
user: login: f1sh1918
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T07:28:03Z
user: login: f1sh1918
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T07:28:24Z
user: login: seluianova
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-21T12:25:36Z
user: login: seluianova
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-21T12:36:06Z
user: login: seluianova
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-21T13:23:55Z
user: login: seluianova
body: Log-based solution sounds more generic to me (unless the idea was that this file will be checked directly by users, not only by us https://staging.bayern.ehrenamtskarte.app/acceptance-stores/filtered-stores.csv).
But the current implementation looks good. Just a few minor suggestions

❓ btw do we want to make sure that only administrators and not everyone will be able to access the file? 
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-21T13:41:45Z
user: login: andrew8er
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-21T15:10:58Z

Review comments:
diff_hunk: @@ -20,6 +20,7 @@ matomoUrl: http://localhost:5003/matomo.php
 projects:
   - id: bayern.ehrenamtskarte.app
     importUrl: https://www.ehrenamt.bayern.de/xml-json/app-daten.xml
+    filteredStoresOutput: /var/archive-data/filtered_stores.csv
path: backend/src/main/resources/config/config.yml
user: login: f1sh1918
body: not sure if this is works on debian or we need something like
```
${System.getProperty("foldername")}
```
created_at: 2025-04-30T06:25:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -31,6 +37,9 @@ class FilterLbe(config: ImportConfig, private val logger: Logger) :
     private fun LbeAcceptingStore.isValidName(): Boolean =
         if (name.isUnavailable()) {
             logger.info("'$this' was filtered out because name '$name' is invalid")
+            filteredStores.add(
+                FilteredStore(this, "Name is invalid"),
+            )
path: backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/bayern/steps/FilterLbe.kt
user: login: andrew8er
body: 🙃 This could fit on one line.
created_at: 2025-05-13T12:39:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,55 @@
+package app.ehrenamtskarte.backend.stores.importer.bayern.steps
+
+import app.ehrenamtskarte.backend.stores.importer.ImportConfig
+import app.ehrenamtskarte.backend.stores.importer.PipelineStep
+import app.ehrenamtskarte.backend.stores.importer.bayern.types.FilteredStore
+import app.ehrenamtskarte.backend.stores.importer.common.types.AcceptingStore
+import org.apache.commons.csv.CSVFormat
+import org.slf4j.LoggerFactory
+import java.io.BufferedWriter
+import java.nio.file.Files
+import java.nio.file.Paths
+
+class WriteCsv(config: ImportConfig, private val filteredStores: MutableList<FilteredStore>) :
path: backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/bayern/steps/WriteCsv.kt
user: login: andrew8er
body: 🔧 The name of this class is a little bit unspecific.
created_at: 2025-05-13T12:45:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,55 @@
+package app.ehrenamtskarte.backend.stores.importer.bayern.steps
+
+import app.ehrenamtskarte.backend.stores.importer.ImportConfig
+import app.ehrenamtskarte.backend.stores.importer.PipelineStep
+import app.ehrenamtskarte.backend.stores.importer.bayern.types.FilteredStore
+import app.ehrenamtskarte.backend.stores.importer.common.types.AcceptingStore
+import org.apache.commons.csv.CSVFormat
+import org.slf4j.LoggerFactory
+import java.io.BufferedWriter
+import java.nio.file.Files
+import java.nio.file.Paths
+
+class WriteCsv(config: ImportConfig, private val filteredStores: MutableList<FilteredStore>) :
+    PipelineStep<List<AcceptingStore>, List<AcceptingStore>>(config) {
+    override fun execute(input: List<AcceptingStore>): List<AcceptingStore> {
+        val filteredOutputPath = config.findProject().filteredStoresOutput
+        if (!filteredOutputPath.isNullOrBlank()) {
+            writeCsvWithFilteredStores(filteredOutputPath)
+        }
+        return input
+    }
+
+    private fun writeCsvWithFilteredStores(outputPath: String) {
+        val logger = LoggerFactory.getLogger(WriteCsv::class.java)
+        try {
+            val writer: BufferedWriter =
+                Files.newBufferedWriter(Paths.get(outputPath))
path: backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/bayern/steps/WriteCsv.kt
user: login: andrew8er
body: 🙃 This could fit on one line, the type is not really necessary here.
❌ The writer is never closed, I suggest you use `.use {}` to automatically close it after use.
created_at: 2025-05-13T12:52:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,55 @@
+package app.ehrenamtskarte.backend.stores.importer.bayern.steps
+
+import app.ehrenamtskarte.backend.stores.importer.ImportConfig
+import app.ehrenamtskarte.backend.stores.importer.PipelineStep
+import app.ehrenamtskarte.backend.stores.importer.bayern.types.FilteredStore
+import app.ehrenamtskarte.backend.stores.importer.common.types.AcceptingStore
+import org.apache.commons.csv.CSVFormat
+import org.slf4j.LoggerFactory
+import java.io.BufferedWriter
+import java.nio.file.Files
+import java.nio.file.Paths
+
+class WriteCsv(config: ImportConfig, private val filteredStores: MutableList<FilteredStore>) :
+    PipelineStep<List<AcceptingStore>, List<AcceptingStore>>(config) {
+    override fun execute(input: List<AcceptingStore>): List<AcceptingStore> {
+        val filteredOutputPath = config.findProject().filteredStoresOutput
+        if (!filteredOutputPath.isNullOrBlank()) {
+            writeCsvWithFilteredStores(filteredOutputPath)
+        }
+        return input
+    }
+
+    private fun writeCsvWithFilteredStores(outputPath: String) {
+        val logger = LoggerFactory.getLogger(WriteCsv::class.java)
+        try {
+            val writer: BufferedWriter =
+                Files.newBufferedWriter(Paths.get(outputPath))
+
+            data class Col(
+                val name: String,
+                val fromRecord: (store: FilteredStore) -> String?,
+            )
+
+            val columns = listOf(
+                Col("Name") { it.name },
+                Col("Straße") { it.street },
+                Col("Hausnummer") { it.houseNumber },
+                Col("PLZ") { it.postalCode },
+                Col("Ort") { it.location },
+                Col("Grund") { it.reason },
+            )
+
+            val printer = CSVFormat.RFC4180.builder()
+                .setHeader(*columns.map { it.name }.toTypedArray())
+                .build()
path: backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/bayern/steps/WriteCsv.kt
user: login: andrew8er
body: ❌ `build()` is deprecated, replace with `get()`.
created_at: 2025-05-13T12:54:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,55 @@
+package app.ehrenamtskarte.backend.stores.importer.bayern.steps
+
+import app.ehrenamtskarte.backend.stores.importer.ImportConfig
+import app.ehrenamtskarte.backend.stores.importer.PipelineStep
+import app.ehrenamtskarte.backend.stores.importer.bayern.types.FilteredStore
+import app.ehrenamtskarte.backend.stores.importer.common.types.AcceptingStore
+import org.apache.commons.csv.CSVFormat
+import org.slf4j.LoggerFactory
+import java.io.BufferedWriter
+import java.nio.file.Files
+import java.nio.file.Paths
+
+class WriteCsv(config: ImportConfig, private val filteredStores: MutableList<FilteredStore>) :
+    PipelineStep<List<AcceptingStore>, List<AcceptingStore>>(config) {
+    override fun execute(input: List<AcceptingStore>): List<AcceptingStore> {
+        val filteredOutputPath = config.findProject().filteredStoresOutput
+        if (!filteredOutputPath.isNullOrBlank()) {
+            writeCsvWithFilteredStores(filteredOutputPath)
+        }
+        return input
+    }
+
+    private fun writeCsvWithFilteredStores(outputPath: String) {
+        val logger = LoggerFactory.getLogger(WriteCsv::class.java)
+        try {
+            val writer: BufferedWriter =
+                Files.newBufferedWriter(Paths.get(outputPath))
path: backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/bayern/steps/WriteCsv.kt
user: login: f1sh1918
body: thx for the hint, i replaced the function and refactored the code a little bit
created_at: 2025-05-19T07:28:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,55 @@
+package app.ehrenamtskarte.backend.stores.importer.bayern.steps
+
+import app.ehrenamtskarte.backend.stores.importer.ImportConfig
+import app.ehrenamtskarte.backend.stores.importer.PipelineStep
+import app.ehrenamtskarte.backend.stores.importer.bayern.types.FilteredStore
+import app.ehrenamtskarte.backend.stores.importer.common.types.AcceptingStore
+import org.apache.commons.csv.CSVFormat
+import org.slf4j.LoggerFactory
+import java.io.BufferedWriter
+import java.nio.file.Files
+import java.nio.file.Paths
+
+class WriteCsv(config: ImportConfig, private val filteredStores: MutableList<FilteredStore>) :
path: backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/bayern/steps/WriteCsv.kt
user: login: f1sh1918
body: renamed to `FilteredStoresCsvOutput`
created_at: 2025-05-19T07:28:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -41,12 +43,15 @@ object EhrenamtskarteBayern : Pipeline {
             .addStep(SanitizeGeocode(config, logger, httpClient), logger) {
                 logger.info("== Sanitize data with geocoding ==")
             }
-            .addStep(PostSanitizeFilter(config, logger, httpClient), logger) {
+            .addStep(PostSanitizeFilter(config, logger, filteredStores), logger) {
                 logger.info("== Filter sanitized data ==")
             }
-            .addStep(FilterDuplicates(config, logger), logger) {
+            .addStep(FilterDuplicates(config, logger, filteredStores), logger) {
                 logger.info("== Filter duplicated data ==")
             }
+            .addStep(FilteredStoresCsvOutput(config, filteredStores), logger) {
+                logger.info("== Write CSV Filtered Stores")
path: backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/bayern/EhrenamtskarteBayern.kt
user: login: seluianova
body: ```suggestion
                logger.info("== Write CSV Filtered Stores ==")
```
nitpick to stick to the log format
created_at: 2025-05-21T12:25:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,55 @@
+package app.ehrenamtskarte.backend.stores.importer.bayern.steps
+
+import app.ehrenamtskarte.backend.stores.importer.ImportConfig
+import app.ehrenamtskarte.backend.stores.importer.PipelineStep
+import app.ehrenamtskarte.backend.stores.importer.bayern.types.FilteredStore
+import app.ehrenamtskarte.backend.stores.importer.common.types.AcceptingStore
+import org.apache.commons.csv.CSVFormat
+import org.slf4j.LoggerFactory
+import java.nio.file.Files
+import java.nio.file.Paths
+
+class FilteredStoresCsvOutput(config: ImportConfig, private val filteredStores: MutableList<FilteredStore>) :
+    PipelineStep<List<AcceptingStore>, List<AcceptingStore>>(config) {
+    override fun execute(input: List<AcceptingStore>): List<AcceptingStore> {
+        val filteredOutputPath = config.project.filteredStoresOutput
+        if (!filteredOutputPath.isNullOrBlank()) {
+            writeCsvWithFilteredStores(filteredOutputPath)
+        }
+        return input
+    }
+
+    private fun writeCsvWithFilteredStores(outputPath: String) {
+        val logger = LoggerFactory.getLogger(FilteredStoresCsvOutput::class.java)
+
+        data class Col(
+            val name: String,
+            val fromRecord: (store: FilteredStore) -> String?,
+        )
+
+        val columns = listOf(
+            Col("Name") { it.name },
+            Col("Straße") { it.street },
+            Col("Hausnummer") { it.houseNumber },
+            Col("PLZ") { it.postalCode },
+            Col("Ort") { it.location },
+            Col("Grund") { it.reason },
+        )
+
+        try {
+            Files.newBufferedWriter(Paths.get(outputPath)).use { writer ->
+                CSVFormat.RFC4180.builder()
+                    .setHeader(*columns.map { it.name }.toTypedArray())
+                    .get()
+                    .print(writer).let { printer ->
+                        filteredStores.forEach { record ->
+                            printer.printRecord(*(columns.map { it.fromRecord(record) }.toTypedArray()))
+                        }
+                        printer.flush()
+                    }
+            }
+        } catch (e: Exception) {
+            logger.info("Fehler beim Schreiben der CSV:", e)
path: backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/bayern/steps/FilteredStoresCsvOutput.kt
user: login: seluianova
body: ```suggestion
            logger.error("Fehler beim Schreiben der CSV:", e)
```
nitpick
created_at: 2025-05-21T12:36:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,55 @@
+package app.ehrenamtskarte.backend.stores.importer.bayern.steps
+
+import app.ehrenamtskarte.backend.stores.importer.ImportConfig
+import app.ehrenamtskarte.backend.stores.importer.PipelineStep
+import app.ehrenamtskarte.backend.stores.importer.bayern.types.FilteredStore
+import app.ehrenamtskarte.backend.stores.importer.common.types.AcceptingStore
+import org.apache.commons.csv.CSVFormat
+import org.slf4j.LoggerFactory
+import java.nio.file.Files
+import java.nio.file.Paths
+
+class FilteredStoresCsvOutput(config: ImportConfig, private val filteredStores: MutableList<FilteredStore>) :
+    PipelineStep<List<AcceptingStore>, List<AcceptingStore>>(config) {
+    override fun execute(input: List<AcceptingStore>): List<AcceptingStore> {
+        val filteredOutputPath = config.project.filteredStoresOutput
+        if (!filteredOutputPath.isNullOrBlank()) {
+            writeCsvWithFilteredStores(filteredOutputPath)
+        }
+        return input
+    }
+
+    private fun writeCsvWithFilteredStores(outputPath: String) {
+        val logger = LoggerFactory.getLogger(FilteredStoresCsvOutput::class.java)
+
+        data class Col(
+            val name: String,
+            val fromRecord: (store: FilteredStore) -> String?,
+        )
+
+        val columns = listOf(
+            Col("Name") { it.name },
+            Col("Straße") { it.street },
+            Col("Hausnummer") { it.houseNumber },
+            Col("PLZ") { it.postalCode },
+            Col("Ort") { it.location },
+            Col("Grund") { it.reason },
+        )
+
+        try {
+            Files.newBufferedWriter(Paths.get(outputPath)).use { writer ->
+                CSVFormat.RFC4180.builder()
+                    .setHeader(*columns.map { it.name }.toTypedArray())
+                    .get()
+                    .print(writer).let { printer ->
+                        filteredStores.forEach { record ->
+                            printer.printRecord(*(columns.map { it.fromRecord(record) }.toTypedArray()))
+                        }
+                        printer.flush()
+                    }
path: backend/src/main/kotlin/app/ehrenamtskarte/backend/stores/importer/bayern/steps/FilteredStoresCsvOutput.kt
user: login: seluianova
body: ```suggestion
                    .print(writer).use { printer ->
                        filteredStores.forEach { record ->
                            printer.printRecord(columns.map { it.fromRecord(record) })
                        }
                    }
```
Use "use" to close CSVPrinter
created_at: 2025-05-21T13:23:55Z
author_association: CONTRIBUTOR

PR comments:
user: login: connium
created_at: 2025-04-30T07:16:03Z
body: > You can just set a local path in your config.local.yml in the project section to test the csv output (dev)

Should this information be added to some kind of documentation?
author_association: NONE
user: login: f1sh1918
created_at: 2025-05-07T13:24:47Z
body: > > You can just set a local path in your config.local.yml in the project section to test the csv output (dev)
> 
> Should this information be added to some kind of documentation?

hm i don't think thats actually needed, since the `config.local.yml` in general can be used to override backend config values
author_association: CONTRIBUTOR
user: login: f1sh1918
created_at: 2025-05-19T07:28:59Z
body: > A few suggestions and some corrections.

Thx for the suggestions @andrew8er, i think i addressed all, could you please recheck
author_association: CONTRIBUTOR
user: login: bahaaTuffaha
created_at: 2025-05-20T18:48:05Z
body: How can I get the file from:  https://staging.bayern.ehrenamtskarte.app/acceptance-stores/filtered-stores.csv I tried the credentials from passpolt but that didn't work.
author_association: CONTRIBUTOR
user: login: f1sh1918
created_at: 2025-05-20T21:21:48Z
body: > How can I get the file from:  https://staging.bayern.ehrenamtskarte.app/acceptance-stores/filtered-stores.csv I tried the credentials from passpolt but that didn't work.

You can not since this job writes a file to a local path. If you execute it on staging server then it will be there.
For local testing you have to set a local path in your local config file
author_association: CONTRIBUTOR
user: login: bahaaTuffaha
created_at: 2025-05-21T09:34:34Z
body: > > How can I get the file from:  https://staging.bayern.ehrenamtskarte.app/acceptance-stores/filtered-stores.csv I tried the credentials from passpolt but that didn't work.
> 
> You can not since this job writes a file to a local path. If you execute it on staging server then it will be there. For local testing you have to set a local path in your local config file

I couldn't get any file.. I tried to signIn (dev) as store maneger koblenz/nurenburg imported stores even went to /acceptance-stores/filtered-stores.csv or /acceptance-stores/ but nothing came up can you please give us steps on how to test this and what to expect?
author_association: CONTRIBUTOR
user: login: seluianova
created_at: 2025-05-21T13:34:11Z
body: > We get often mails that some accepting stores are not listed in the app. Therefore we provide a csv file that lists filtered stores with the filtering reason.

💭 perhaps we could use the log to check this information. we basically write all filtering reasons to the import output, but do not save the log file on the server. perhaps saving the log file would also solve the reported issue

💭 maybe it would be nice to configure grafana to view logs. this could be used to check import logs and various other things... like the state of the server after release
author_association: CONTRIBUTOR
user: login: seluianova
created_at: 2025-05-21T13:49:30Z
body: > I couldn't get any file.. I tried to signIn (dev) as store maneger koblenz/nurenburg imported stores even went to /acceptance-stores/filtered-stores.csv or /acceptance-stores/ but nothing came up can you please give us steps on how to test this and what to expect?

@bahaaTuffaha this feature only applies to bayern, so you can't test it from the administration UI.
You can run stores import for bayern using "DB import online data" run configuration.
1. Set in your backend local config:
filteredStoresOutput: src/main/resources/import/filtered_stores.csv
2. Run the import. When finished, check the file locally in the file system
author_association: CONTRIBUTOR
user: login: f1sh1918
created_at: 2025-05-21T15:18:35Z
body: > Log-based solution sounds more generic to me (unless the idea was that this file will be checked directly by users, not only by us https://staging.bayern.ehrenamtskarte.app/acceptance-stores/filtered-stores.csv). But the current implementation looks good. Just a few minor suggestions
> 
> ❓ btw do we want to make sure that only administrators and not everyone will be able to access the file?

Well the csv file can be filtered and transfered to the region, so its better than a log.
Currently the file is not secured but what should be the purpose in securing it?
author_association: CONTRIBUTOR