Repository: komoot/photon (java)

Issue:
Updates no longer work with Nominatim 4.x
Nominatim has moved some of the indexing processing into Python code since version 4.x. The simple update mechansim that Photon uses thus only does half the work. In particular, addr:* parts a no longer properly matched.

If you run regular update on your Photon database you **must remain with Nominatim 3.7 for now**.

Imports still work against the newer version.


Pull request description:
Rework update mechanism to be independent from Nominatim updates
This completely changes how updates are handled with relation to Nominatim updates. Photon now installs a trigger which tracks updates by Nominatim in a private tables. When running updates, it simply collects new information for the changed items.

It wasn't possible to implement this completely without triggers (as originally discussed in #638) because there is no way to track deletions properly. I still didn't want to add the change mechanisms to Nominatim as it would require to wait for a new release of Nominatim. The proposed mechanism should work with any Nominatim version insofar as Photon is compatible with it.

Updates for interpolations and places with housenumbers are still broken. I'll address that in a separate PR.

Please note that the mechanism how to run Nominatim and Photon updates has changed completely. Refer to the README on how to set up the process now. It is in theory possible to run the Nominatim update script and Photon updater concurrently but I would advise against it. Better to run them one after the other as shown in `continuously_update_from_nominatim.sh`.

Fixes #638.

Patch:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 725cb4d..61ffadc 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -59,7 +59,7 @@ jobs:
         run: echo "cache_key=$(/bin/date -u "+%Y%W")" >> $GITHUB_ENV
         shell: bash
 
-      - uses: actions/cache@v3
+      - uses: actions/cache@v4
         with:
             path: |
                Nominatim/data/country_osm_grid.sql.gz
@@ -117,11 +117,12 @@ jobs:
       - name: Import Photon
         run: |
             java -jar target/photon-*.jar -nominatim-import -database nominatim -user runner -password foobar
+            java -jar target/photon-*.jar -nominatim-update-init-for runner -database nominatim -user runner -password foobar
 
       - name: Update Nominatim
         run: |
           nominatim replication --init
-          nominatim replication --no-index --once
+          nominatim replication --once
         shell: bash
         working-directory: data-env
         env:
diff --git a/README.md b/README.md
index e1230f4..1ba2c05 100644
--- a/README.md
+++ b/README.md
@@ -142,25 +142,56 @@ The import of worldwide data set will take some hours/days, SSD/NVME disks are r
 
 #### Updating from OSM via Nominatim
 
-In order to update nominatim from OSM and then photon from nominatim, you must start photon with the nominatim database credentials on the command line:
+To update an existing Photon database from Nominatim, first prepare the
+Nominatim database with the appropriate triggers:
 
 ```bash
-java -jar photon-*.jar -host localhost -port 5432 -database nominatim -user nominatim -password ...
+java -jar photon-*.jar -database nominatim -user nominatim -password ... -nominatim-update-init-for update_user
 ```
 
-A nominatim setup is also a requirement to have continuous updates. To keep nominatim in sync with the latest OSM changes and to update photon with nominatim afterwards run:
+This script must be run with a user that has the right to create tables,
+functions and triggers.
+
+'update-user' is the PostgreSQL user that will be used when updating the
+Photon database. The user needs read rights on the database. The necessary
+update rights will be granted during initialisation.
+
+Now you can run updates on Nominatim using the usual methods as described
+in the [documentation](https://nominatim.org/release-docs/latest/admin/Update/).
+To bring the Photon database up-to-date, stop the Nominatim updates and
+then run the Photon update process:
 
 ```bash
-export NOMINATIM_DIR=/home/nominatim/...
-./continuously_update_from_nominatim.sh
+java -jar photon-*.jar -database nominatim -user nominatim -password ... -nominatim-update
+```
+
+You can also run the photon process with the update API enabled:
+
+```bash
+java -jar photon-*.jar -enable-update-api -database nominatim -user nominatim -password ...
 ```
 
-If you have updated nominatim with another method, photon can be updated by making a HTTP GET request to `/nominatim-update`, e.g. with this command:
+Then you can trigger updates like this:
 
 ```bash
 curl http://localhost:2322/nominatim-update
 ```
 
+For your convenience, this repository contains a script to continuously update
+both Nominatim and Photon. To run it, first customize some environment
+variables according to your installation:
+
+```bash
+export NOMINATIM_DIR=/srv/nominatim/...
+export PHOTON_JAR=photon.jar
+export PHOTON_DB_NAME=nominatim
+export PHOTON_DB_USER=nominatim
+export PHOTON_DB_PASSWORD=...
+./continuously_update_from_nominatim.sh
+```
+
+If you have Nominatim < 3.7, please read the comments in the script carefully.
+
 ### Search API
 
 #### Search
diff --git a/continuously_update_from_nominatim.sh b/continuously_update_from_nominatim.sh
old mode 100644
new mode 100755
index 92976d6..6607e33
--- a/continuously_update_from_nominatim.sh
+++ b/continuously_update_from_nominatim.sh
@@ -2,7 +2,16 @@
 
 # For Nominatim < 3.7 set this to the Nominatim build directory.
 # For newer versions, this must be the project directory of your import.
-NOMINATIM_DIR=
+: ${NOMINATIM_DIR:=.}
+# Path to Photon Jar file
+: ${PHOTON_JAR:=photon.jar}
+# Name of Nominatim database.
+: ${PHOTON_DB_NAME:=nominatim}
+# PostgreSQL user name to use for update in Photon
+: ${PHOTON_DB_USER:=nominatim}
+# Password for PostgreSQL user
+: ${PHOTON_DB_PASSWORD:=}
+
 
 while true
 do
@@ -12,14 +21,13 @@ do
     # The important part here is to leave out the indexing step. This
     # will be handled by Photon.
 
-    cd $NOMINATIM_DIR
     # For Nominatim versions < 3.7 use the following:
-    # ./utils/update.php --import-osmosis --no-index
-    nominatim replication --once --no-index
+    # ./utils/update.php --import-osmosis
+    nominatim replication --project-dir $NOMINATIM_DIR  --once
 
     # Now tell Photon to finish the updates and copy the new data into its
     # own database.
-    curl http://localhost:2322/nominatim-update
+    java -jar $PHOTON_JAR -database $PHOTON_DB_NAME -user $PHOTON_DB_USER -password $PHOTON_DB_PASSWORD -nominatim-update
 
     # Sleep a bit if updates take less than a minute.
     # If you consume hourly or daily diffs adapt the period accordingly.
diff --git a/pom.xml b/pom.xml
index 3ba7e29..f0aa38e 100644
--- a/pom.xml
+++ b/pom.xml
@@ -49,7 +49,7 @@
         <dependency>
             <groupId>org.postgresql</groupId>
             <artifactId>postgresql</artifactId>
-            <version>42.7.2</version>
+            <version>42.7.1</version>
         </dependency>
         <dependency>
             <groupId>org.slf4j</groupId>
diff --git a/src/main/java/de/komoot/photon/App.java b/src/main/java/de/komoot/photon/App.java
index f9e35fc..d7ef3c6 100644
--- a/src/main/java/de/komoot/photon/App.java
+++ b/src/main/java/de/komoot/photon/App.java
@@ -29,6 +29,11 @@ public class App {
             return;
         }
 
+        if (args.getNominatimUpdateInit() != null) {
+            startNominatimUpdateInit(args);
+            return;
+        }
+
         boolean shutdownES = false;
         final Server esServer = new Server(args.getDataDirectory()).start(args.getCluster(), args.getTransportAddresses());
         try {
@@ -126,9 +131,15 @@ public class App {
         log.info("imported data from nominatim to photon with languages: " + String.join(",", dbProperties.getLanguages()));
     }
 
-    /**
-     * Prepare Nominatim updater.
-     */
+    private static void startNominatimUpdateInit(CommandLineArgs args) {
+        NominatimUpdater nominatimUpdater = new NominatimUpdater(args.getHost(), args.getPort(), args.getDatabase(), args.getUser(), args.getPassword());
+        nominatimUpdater.initUpdates(args.getNominatimUpdateInit());
+    }
+
+
+        /**
+         * Prepare Nominatim updater.
+         */
     private static NominatimUpdater setupNominatimUpdater(CommandLineArgs args, Server server) {
         // Get database properties and ensure that the version is compatible.
         DatabaseProperties dbProperties = new DatabaseProperties();
diff --git a/src/main/java/de/komoot/photon/CommandLineArgs.java b/src/main/java/de/komoot/photon/CommandLineArgs.java
index 0929bac..cc2f5a0 100644
--- a/src/main/java/de/komoot/photon/CommandLineArgs.java
+++ b/src/main/java/de/komoot/photon/CommandLineArgs.java
@@ -23,6 +23,9 @@ public class CommandLineArgs {
     @Parameter(names = "-nominatim-import", description = "import nominatim database into photon (this will delete previous index)")
     private boolean nominatimImport = false;
 
+    @Parameter(names = "-nominatim-update-init-for", description = "set up tracking of updates in the Nominatim database for the given user and exit")
+    private String nominatimUpdateInit = null;
+
     @Parameter(names = "-nominatim-update", description = "fetch updates from nominatim database into photon and exit (this updates the index only without offering an API)")
     private boolean nominatimUpdate = false;
 
@@ -60,7 +63,7 @@ public class CommandLineArgs {
     private String user = "nominatim";
 
     @Parameter(names = "-password", description = "postgres password (default '')")
-    private String password = "";
+    private String password = null;
 
     @Parameter(names = "-data-dir", description = "data directory (default '.')")
     private String dataDirectory = new File(".").getAbsolutePath();
diff --git a/src/main/java/de/komoot/photon/nominatim/DBDataAdapter.java b/src/main/java/de/komoot/photon/nominatim/DBDataAdapter.java
index b30594b..830c75f 100644
--- a/src/main/java/de/komoot/photon/nominatim/DBDataAdapter.java
+++ b/src/main/java/de/komoot/photon/nominatim/DBDataAdapter.java
@@ -25,4 +25,9 @@ public interface DBDataAdapter {
      * Check if a table has the given column.
      */
     boolean hasColumn(JdbcTemplate template, String table, String column);
+
+    /**
+     * Wrap a DELETE statement with a RETURNING clause.
+     */
+    String deleteReturning(String deleteSQL, String columns);
 }
diff --git a/src/main/java/de/komoot/photon/nominatim/NominatimConnector.java b/src/main/java/de/komoot/photon/nominatim/NominatimConnector.java
index f77e95b..f5fd0ae 100644
--- a/src/main/java/de/komoot/photon/nominatim/NominatimConnector.java
+++ b/src/main/java/de/komoot/photon/nominatim/NominatimConnector.java
@@ -13,10 +13,7 @@ import org.springframework.jdbc.core.RowMapper;
 
 import java.sql.ResultSet;
 import java.sql.SQLException;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
+import java.util.*;
 
 /**
  * Export nominatim data
@@ -38,6 +35,7 @@ public class NominatimConnector {
      */
     private final RowMapper<NominatimResult> osmlineRowMapper;
     private final String selectOsmlineSql;
+    private Importer importer;
 
 
     /**
@@ -75,7 +73,6 @@ public class NominatimConnector {
             return result;
         }
     };
-    private Importer importer;
 
     /**
      * @param host     database host
@@ -150,7 +147,9 @@ public class NominatimConnector {
 
         dataSource.setUrl(String.format("jdbc:postgres_jts://%s:%d/%s", host, port, database));
         dataSource.setUsername(username);
-        dataSource.setPassword(password);
+        if (password != null) {
+            dataSource.setPassword(password);
+        }
         dataSource.setDriverClassName(JtsWrapper.class.getCanonicalName());
         dataSource.setDefaultAutoCommit(autocommit);
         return dataSource;
@@ -172,18 +171,22 @@ public class NominatimConnector {
     }
 
     public List<PhotonDoc> getByPlaceId(long placeId) {
-        NominatimResult result = template.queryForObject(SELECT_COLS_PLACEX + " FROM placex WHERE place_id = ?",
+        List<NominatimResult> result = template.query(SELECT_COLS_PLACEX + " FROM placex WHERE place_id = ?",
                                                          placeRowMapper, placeId);
-        assert(result != null);
-        return result.getDocsWithHousenumber();
+        if (result.size() == 0)
+            return null;
+
+        return result.get(0).getDocsWithHousenumber();
     }
 
     public List<PhotonDoc> getInterpolationsByPlaceId(long placeId) {
-        NominatimResult result = template.queryForObject(selectOsmlineSql
+        List<NominatimResult> result = template.query(selectOsmlineSql
                                                           + " FROM location_property_osmline WHERE place_id = ?",
                                                           osmlineRowMapper, placeId);
-        assert(result != null);
-        return result.getDocsWithHousenumber();
+        if (result.size() == 0)
+            return null;
+
+        return result.get(0).getDocsWithHousenumber();
     }
 
     private long parentPlaceId = -1;
@@ -313,4 +316,8 @@ public class NominatimConnector {
             }
         }
     }
+
+    public DBDataAdapter getDataAdaptor() {
+        return dbutils;
+    }
 }
diff --git a/src/main/java/de/komoot/photon/nominatim/NominatimUpdater.java b/src/main/java/de/komoot/photon/nominatim/NominatimUpdater.java
index a8d134d..07aeea9 100644
--- a/src/main/java/de/komoot/photon/nominatim/NominatimUpdater.java
+++ b/src/main/java/de/komoot/photon/nominatim/NominatimUpdater.java
@@ -5,12 +5,9 @@ import de.komoot.photon.Updater;
 import de.komoot.photon.nominatim.model.UpdateRow;
 import org.apache.commons.dbcp2.BasicDataSource;
 import org.springframework.jdbc.core.JdbcTemplate;
-import org.springframework.jdbc.core.RowMapper;
 
-import java.sql.ResultSet;
-import java.sql.SQLException;
+import java.util.Comparator;
 import java.util.List;
-import java.util.Map;
 import java.util.concurrent.locks.ReentrantLock;
 
 /**
@@ -22,6 +19,33 @@ import java.util.concurrent.locks.ReentrantLock;
 public class NominatimUpdater {
     private static final org.slf4j.Logger LOGGER = org.slf4j.LoggerFactory.getLogger(NominatimUpdater.class);
 
+    private static final String TRIGGER_SQL =
+            "DROP TABLE IF EXISTS photon_updates;"
+            + "CREATE TABLE photon_updates (rel TEXT, place_id BIGINT,"
+            + "                             operation TEXT,"
+            + "                             indexed_date TIMESTAMP WITH TIME ZONE);"
+            + "CREATE OR REPLACE FUNCTION photon_update_func()\n"
+            + " RETURNS TRIGGER AS $$\n"
+            + "BEGIN\n"
+            + "  INSERT INTO photon_updates("
+            + "     VALUES (TG_TABLE_NAME, OLD.place_id, TG_OP, statement_timestamp()));"
+            + "  RETURN NEW;"
+            + "END; $$ LANGUAGE plpgsql;"
+            + "CREATE OR REPLACE TRIGGER photon_trigger_update_placex"
+            + "   AFTER UPDATE ON placex FOR EACH ROW"
+            + "   WHEN (OLD.indexed_status > 0 AND NEW.indexed_status = 0)"
+            + "   EXECUTE FUNCTION photon_update_func();"
+            + "CREATE OR REPLACE TRIGGER photon_trigger_delete_placex"
+            + "   AFTER DELETE ON placex FOR EACH ROW"
+            + "   EXECUTE FUNCTION photon_update_func();"
+            + "CREATE OR REPLACE TRIGGER photon_trigger_update_interpolation "
+            + "   AFTER UPDATE ON location_property_osmline FOR EACH ROW"
+            + "   WHEN (OLD.indexed_status > 0 AND NEW.indexed_status = 0)"
+            + "   EXECUTE FUNCTION photon_update_func();"
+            + "CREATE OR REPLACE TRIGGER photon_trigger_delete_interpolation"
+            + "   AFTER DELETE ON location_property_osmline FOR EACH ROW"
+            + "   EXECUTE FUNCTION photon_update_func()";
+
     private static final int CREATE = 1;
     private static final int UPDATE = 2;
     private static final int DELETE = 100;
@@ -43,84 +67,18 @@ public class NominatimUpdater {
         this.updater = updater;
     }
 
+    public void initUpdates(String updateUser) {
+        LOGGER.info("Creating tracking tables");
+        template.execute(TRIGGER_SQL);
+        template.execute("GRANT SELECT, DELETE ON photon_updates TO \"" + updateUser + '"');
+    }
+
     public void update() {
         if (updateLock.tryLock()) {
             try {
-                int updatedPlaces = 0;
-                int deletedPlaces = 0;
-                for (int rank = MIN_RANK; rank <= MAX_RANK; rank++) {
-                    LOGGER.info(String.format("Starting rank %d", rank));
-                    for (Map<String, Object> sector : getIndexSectors(rank)) {
-                        for (UpdateRow place : getIndexSectorPlaces(rank, (Integer) sector.get("geometry_sector"))) {
-                            long placeId = place.getPlaceId();
-                            template.update("update placex set indexed_status = 0 where place_id = ?;", placeId);
-
-                            Integer indexedStatus = place.getIndexdStatus();
-                            if (indexedStatus == DELETE || (indexedStatus == UPDATE && rank == MAX_RANK)) {
-                                updater.delete(placeId);
-                                if (indexedStatus == DELETE) {
-                                    deletedPlaces++;
-                                    continue;
-                                }
-                                indexedStatus = CREATE; // always create
-                            }
-                            updatedPlaces++;
-
-                            final List<PhotonDoc> updatedDocs = exporter.getByPlaceId(place.getPlaceId());
-                            boolean wasUseful = false;
-                            for (PhotonDoc updatedDoc : updatedDocs) {
-                                if (updatedDoc.isUsefulForIndex()) {
-                                    updater.create(updatedDoc);
-                                    wasUseful = true;
-                                }
-                            }
-                            if (indexedStatus == UPDATE && !wasUseful) {
-                                // only true when rank != 30
-                                // if no documents for the place id exist this will likely cause moaning
-                                updater.delete(placeId);
-                                updatedPlaces--;
-                            }
-                        }
-                    }
-                }
-
-                LOGGER.info(String.format("%d places created or updated, %d deleted", updatedPlaces, deletedPlaces));
-
-                // update documents generated from address interpolations
-                // .isUsefulForIndex() should always return true for documents
-                // created from interpolations so no need to check them
-                LOGGER.info("Starting interpolations");
-                int updatedInterpolations = 0;
-                int deletedInterpolations = 0;
-                int interpolationDocuments = 0;
-                for (Map<String, Object> sector : template.queryForList(
-                        "select geometry_sector,count(*) from location_property_osmline where indexed_status > 0 group by geometry_sector order by geometry_sector;")) {
-                    for (UpdateRow place : getIndexSectorInterpolations((Integer) sector.get("geometry_sector"))) {
-                        long placeId = place.getPlaceId();
-                        template.update("update location_property_osmline set indexed_status = 0 where place_id = ?;", placeId);
-
-                        Integer indexedStatus = place.getIndexdStatus();
-                        if (indexedStatus != CREATE) {
-                            updater.delete(placeId);
-                            if (indexedStatus == DELETE) {
-                                deletedInterpolations++;
-                                continue;
-                            }
-                        }
-                        updatedInterpolations++;
-
-                        final List<PhotonDoc> updatedDocs = exporter.getInterpolationsByPlaceId(place.getPlaceId());
-                        for (PhotonDoc updatedDoc : updatedDocs) {
-                            updater.create(updatedDoc);
-                            interpolationDocuments++;
-                        }
-                    }
-                }
-                LOGGER.info(String.format("%d interpolations created or updated, %d deleted, %d documents added or updated", updatedInterpolations,
-                        deletedInterpolations, interpolationDocuments));
+                update_from_placex();
+                update_from_interpolations();
                 updater.finish();
-                template.update("update import_status set indexed=true;"); // indicate that we are finished
-
                 LOGGER.info("Finished updating");
             } finally {
                 updateLock.unlock();
@@ -130,37 +88,82 @@ public class NominatimUpdater {
         }
     }
 
-    private List<Map<String, Object>> getIndexSectors(Integer rank) {
-        return template.queryForList("select geometry_sector,count(*) from placex where rank_search = ? "
-                + "and indexed_status > 0 group by geometry_sector order by geometry_sector;", rank);
-    }
+    private void update_from_placex() {
+        LOGGER.info("Starting place updates");
+        int updatedPlaces = 0;
+        int deletedPlaces = 0;
+        for (UpdateRow place : getPlaces("placex")) {
+            long placeId = place.getPlaceId();
+
+            // Always delete to catch some corner cases around places with exploded housenumbers.
+            updater.delete(placeId);
 
-    private List<UpdateRow> getIndexSectorPlaces(Integer rank, Integer geometrySector) {
-        return template.query("select place_id, indexed_status from placex where rank_search = ?" + " and geometry_sector = ? and indexed_status > 0;",
-                new Object[] { rank, geometrySector }, new RowMapper<UpdateRow>() {
-                    @Override
-                    public UpdateRow mapRow(ResultSet rs, int rowNum) throws SQLException {
-                        UpdateRow updateRow = new UpdateRow();
-                        updateRow.setPlaceId(rs.getLong("place_id"));
-                        updateRow.setIndexdStatus(rs.getInt("indexed_status"));
-                        return updateRow;
+            if (place.isToDelete()) {
+                deletedPlaces++;
+                continue;
+            }
+
+            final List<PhotonDoc> updatedDocs = exporter.getByPlaceId(placeId);
+            if (updatedDocs != null) {
+                updatedPlaces++;
+                for (PhotonDoc updatedDoc : updatedDocs) {
+                    if (updatedDoc.isUsefulForIndex()) {
+                        updater.create(updatedDoc);
                     }
-                });
+                }
+            }
+        }
+
+        LOGGER.info(String.format("%d places created or updated, %d deleted", updatedPlaces, deletedPlaces));
     }
 
-    private List<UpdateRow> getIndexSectorInterpolations(Integer geometrySector) {
-        return template.query("select place_id, indexed_status from location_property_osmline where geometry_sector = ? and indexed_status > 0;",
-                new Object[] { geometrySector }, new RowMapper<UpdateRow>() {
-                    @Override
-                    public UpdateRow mapRow(ResultSet rs, int rowNum) throws SQLException {
-                        UpdateRow updateRow = new UpdateRow();
-                        updateRow.setPlaceId(rs.getLong("place_id"));
-                        updateRow.setIndexdStatus(rs.getInt("indexed_status"));
-                        return updateRow;
-                    }
-                });
+    /**
+     * Update documents generated from address interpolations.
+     */
+    private void update_from_interpolations() {
+        // .isUsefulForIndex() should always return true for documents
+        // created from interpolations so no need to check them
+        LOGGER.info("Starting interpolations");
+        int updatedInterpolations = 0;
+        int deletedInterpolations = 0;
+        int interpolationDocuments = 0;
+        for (UpdateRow place : getPlaces("location_property_osmline")) {
+            long placeId = place.getPlaceId();
+
+            updater.delete(placeId);
+            if (place.isToDelete()) {
+                deletedInterpolations++;
+                continue;
+            }
+
+            final List<PhotonDoc> updatedDocs = exporter.getInterpolationsByPlaceId(place.getPlaceId());
+            if (updatedDocs != null) {
+                updatedInterpolations++;
+                for (PhotonDoc updatedDoc : updatedDocs) {
+                    updater.create(updatedDoc);
+                    interpolationDocuments++;
+                }
+            }
+        }
+        LOGGER.info(String.format("%d interpolations created or updated, %d deleted, %d documents added or updated", updatedInterpolations,
+                deletedInterpolations, interpolationDocuments));
+
     }
 
+    private List<UpdateRow> getPlaces(String table) {
+        List<UpdateRow> results = template.query(exporter.getDataAdaptor().deleteReturning(
+                "DELETE FROM photon_updates WHERE rel = ?", "place_id, operation, indexed_date"),
+                (rs, rowNum) -> {
+                    boolean isDelete = "DELETE".equals(rs.getString("operation"));
+                    return new UpdateRow(rs.getLong("place_id"), isDelete, rs.getDate("indexed_date"));
+                }, new Object[]{table});
+
+        results.sort(Comparator.comparing(UpdateRow::getUpdateDate));
+
+        return results;
+    }
+
+
     /**
      * Creates a new instance
      * 
@@ -170,10 +173,14 @@ public class NominatimUpdater {
      * @param username Nominatim database username
      * @param password Nominatim database password
      */
-    public NominatimUpdater(String host, int port, String database, String username, String password) {
+    public NominatimUpdater(String host, int port, String database, String username, String password, DBDataAdapter dataAdapter) {
         BasicDataSource dataSource = NominatimConnector.buildDataSource(host, port, database, username, password, true);
 
-        exporter = new NominatimConnector(host, port, database, username, password);
+        exporter = new NominatimConnector(host, port, database, username, password, dataAdapter);
         template = new JdbcTemplate(dataSource);
     }
+
+    public NominatimUpdater(String host, int port, String database, String username, String password) {
+        this(host, port, database, username, password, new PostgisDataAdapter());
+    }
 }
diff --git a/src/main/java/de/komoot/photon/nominatim/PostgisDataAdapter.java b/src/main/java/de/komoot/photon/nominatim/PostgisDataAdapter.java
index 4332fea..f7b2034 100644
--- a/src/main/java/de/komoot/photon/nominatim/PostgisDataAdapter.java
+++ b/src/main/java/de/komoot/photon/nominatim/PostgisDataAdapter.java
@@ -46,4 +46,9 @@ public class PostgisDataAdapter implements DBDataAdapter {
                     }
                 }, table, column).get(0);
     }
+
+    @Override
+    public String deleteReturning(String deleteSQL, String columns) {
+        return deleteSQL + " RETURNING " + columns;
+    }
 }
diff --git a/src/main/java/de/komoot/photon/nominatim/model/UpdateRow.java b/src/main/java/de/komoot/photon/nominatim/model/UpdateRow.java
index 705abad..dfecab6 100644
--- a/src/main/java/de/komoot/photon/nominatim/model/UpdateRow.java
+++ b/src/main/java/de/komoot/photon/nominatim/model/UpdateRow.java
@@ -1,13 +1,31 @@
 package de.komoot.photon.nominatim.model;
 
-import lombok.Data;
+import java.util.Date;
 
 /**
- * @author felix
+ * Information about places in the database that need updating.
  */
-@Data
 public class UpdateRow {
 
-    public Long placeId;
-    public Integer indexdStatus; // 1 - index, 2 - update, 100 - delete
+    private Long placeId;
+    private boolean toDelete;
+    private Date updateDate;
+
+    public UpdateRow(Long placeId, boolean toDelete, Date updateDate) {
+        this.placeId = placeId;
+        this.toDelete = toDelete;
+        this.updateDate = updateDate;
+    }
+
+    public Long getPlaceId() {
+        return placeId;
+    }
+
+    public boolean isToDelete() {
+        return toDelete;
+    }
+
+    public Date getUpdateDate() {
+        return updateDate;
+    }
 }


Test patch:
diff --git a/src/test/java/de/komoot/photon/ESBaseTester.java b/src/test/java/de/komoot/photon/ESBaseTester.java
index 48bc34e..5b1d82c 100644
--- a/src/test/java/de/komoot/photon/ESBaseTester.java
+++ b/src/test/java/de/komoot/photon/ESBaseTester.java
@@ -35,6 +35,10 @@ public class ESBaseTester {
     }
 
     protected PhotonResult getById(int id) {
+        return getById(String.valueOf(id));
+    }
+
+    protected PhotonResult getById(String id) {
         return server.getById(id);
     }
 
diff --git a/src/test/java/de/komoot/photon/ReflectionTestUtil.java b/src/test/java/de/komoot/photon/ReflectionTestUtil.java
index 19ffaea..48d4cd3 100644
--- a/src/test/java/de/komoot/photon/ReflectionTestUtil.java
+++ b/src/test/java/de/komoot/photon/ReflectionTestUtil.java
@@ -38,4 +38,9 @@ public class ReflectionTestUtil {
     public static <T> void setFieldValue(Object anObject, String fieldName, T value) {
         setFieldValue(anObject, anObject.getClass(), fieldName, value);
     }
+
+    public static <T> void setFieldValue(Object anObject, String fieldName, String subFieldName, T value) {
+        Object member = getFieldValue(anObject, fieldName);
+        setFieldValue(member, member.getClass(), subFieldName, value);
+    }
 }
diff --git a/src/test/java/de/komoot/photon/elasticsearch/ElasticTestServer.java b/src/test/java/de/komoot/photon/elasticsearch/ElasticTestServer.java
index c9bfd03..0b4a44d 100644
--- a/src/test/java/de/komoot/photon/elasticsearch/ElasticTestServer.java
+++ b/src/test/java/de/komoot/photon/elasticsearch/ElasticTestServer.java
@@ -14,8 +14,8 @@ public class ElasticTestServer extends Server {
         return new IndexSettings().setShards(1);
     }
 
-    public PhotonResult getById(int id) {
-        GetResponse response =  esClient.prepareGet(PhotonIndex.NAME,PhotonIndex.TYPE, String.valueOf(id)).execute().actionGet();
+    public PhotonResult getById(String id) {
+        GetResponse response =  esClient.prepareGet(PhotonIndex.NAME,PhotonIndex.TYPE, id).execute().actionGet();
 
         return response.isExists() ? new ElasticGetIdResult(response) : null;
     }
diff --git a/src/test/java/de/komoot/photon/elasticsearch/ImporterTest.java b/src/test/java/de/komoot/photon/elasticsearch/ImporterTest.java
index b6866b2..f73f283 100644
--- a/src/test/java/de/komoot/photon/elasticsearch/ImporterTest.java
+++ b/src/test/java/de/komoot/photon/elasticsearch/ImporterTest.java
@@ -42,6 +42,26 @@ public class ImporterTest extends ESBaseTester {
         assertNull(response.get("extra"));
     }
 
+    @Test
+    public void testAddHousenumberDoc() {
+        Importer instance = makeImporterWithExtra("");
+
+        instance.add(new PhotonDoc(4432, "N", 100, "building", "yes").houseNumber("34"));
+        instance.finish();
+        refresh();
+
+        PhotonResult response = getById("4432.34");
+
+        assertNotNull(response);
+
+        assertEquals("N", response.get("osm_type"));
+        assertEquals(100, response.get("osm_id"));
+        assertEquals("building", response.get("osm_key"));
+        assertEquals("yes", response.get("osm_value"));
+        assertEquals("34", response.get("housenumber"));
+
+    }
+
     @Test
     public void testSelectedExtraTagsCanBeIncluded() {
         Importer instance = makeImporterWithExtra("maxspeed", "website");
diff --git a/src/test/java/de/komoot/photon/nominatim/testdb/H2DataAdapter.java b/src/test/java/de/komoot/photon/nominatim/testdb/H2DataAdapter.java
index a42bded..40b2a0d 100644
--- a/src/test/java/de/komoot/photon/nominatim/testdb/H2DataAdapter.java
+++ b/src/test/java/de/komoot/photon/nominatim/testdb/H2DataAdapter.java
@@ -46,4 +46,10 @@ public class H2DataAdapter implements DBDataAdapter {
     public boolean hasColumn(JdbcTemplate template, String table, String column) {
         return false;
     }
+
+    @Override
+    public String deleteReturning(String deleteSQL, String columns) {
+        return "SELECT " + columns + " FROM OLD TABLE (" + deleteSQL + ")";
+    }
+
 }
diff --git a/src/test/resources/test-schema.sql b/src/test/resources/test-schema.sql
index 858534f..f808d3e 100644
--- a/src/test/resources/test-schema.sql
+++ b/src/test/resources/test-schema.sql
@@ -63,3 +63,11 @@ CREATE TABLE country_name (
 
 INSERT INTO country_name
     VALUES ('de', JSON '{"name" : "Deutschland", "name:en" : "Germany"}', 'de', 2);
+
+
+CREATE TABLE photon_updates (
+    rel TEXT,
+    place_id BIGINT,
+    operation TEXT,
+    indexed_date TIMESTAMP
+);