Repository: geopandas/geopandas

Issue:
Very slow when writing to GPKG
Here's my test suite for a proof: https://github.com/culebron/geodata

Run `python3.6 few.py` and `python3.6 multiple.py` to compare.

`few.py` opens a file with a lot of data, but only 2.7K records as GeoDataFrame. It writes them into GeoJSON and GPKG. In this case, GPKG driver outperforms GeoJSON.

`multiple.py` creates a 100K records dataframe and then saves it to GeoJSON and GPKG. Here, GPKG is incredibly slow.

My results:

	$ python3.6 few.py 
	writing 2.7K records to geojson 36.283805477003625
	writing 2.7K records to gpkg    20.792497718997765
	$ python3.6 multiple.py 
	100%|████████████████████████████████████████████████████████| 100000/100000 [00:03<00:00, 29996.25it/s]
	writing 100K records to geojson 61.62079200500011
	writing 100K records to gpkg    260.4413645050008

And notice that in case of `multiple.py`, the resulting GeoPackage file is only 9 megs. Which is times smaller than the file produced by `few.py`.

As I understand, the problem is that Fiona opens a session in Sqlite and creates a lock file, and it takes some time. And inspecting the code, I see GeoPandas naively writes everything 1 record at a time, which means Sqlite honestly locks it, then writes, then unlocks:

https://github.com/geopandas/geopandas/blob/master/geopandas/io/file.py#L107

    with fiona.drivers():
        with fiona.open(filename, 'w', driver=driver, crs=df.crs,
                        schema=schema, **kwargs) as colxn:
            for feature in df.iterfeatures():
                colxn.write(feature)


This should be optimized. Are there branches/pull requests for this?



Discussion:
I tried replacing the code in the original comment.

    with fiona.drivers():
        with fiona.open(filename, 'w', driver=driver, crs=df.crs,
                        schema=schema, **kwargs) as colxn:
            buf = []
            for feature in df.iterfeatures():
                buf.append(feature)
                if len(buf) > 9999:
                    colxn.writerecords(buf)
                    buf = []

            colxn.writerecords(buf)

    $ python3.6 multiple.py 
    100%|████████████████████████████████████████████████████████| 100000/100000 [00:03<00:00, 31041.45it/s]
    writing 100K records to gpkg    91.03069832099573


91 seconds instead of 260.

Increasing the batch size to 100K, it improved to 82 seconds. But I still see that sqlite locks and unlocks the db many times on the way, and actual writing speed is still slow.
@culebron Thanks for opening this issue!

So there are basically two things:

- As you mentioned, `fiona`'s API is record-oriented. For performance reasons, it would be interesting to also be able to work on full datasets / column-wise. I recently opened this issue for this: https://github.com/Toblerity/Fiona/issues/469. But, I wasn't aware of the `writerecords` method, so at least for writing this could already solve part of the problem. For reading (but also potentially for writing to prevent the need to convert the data to list of dicts), progress on that issue would be interesting.

- Apart from that, we are also working on improving the geopandas performance itself: https://github.com/geopandas/geopandas/issues/473 (by using cython). This will not directly influence the performance of working with fiona, but can speed up part of the writing time that is spend in geopandas. There is also an issue on the interaction of this work with fiona: https://github.com/geopandas/geopandas/issues/491. Your comments there are certainly welcome.

But it seems specifically for what you reported, switching to use `writerecords` seems that it could already provide a nice improvement. 
Can you explore this a bit further, or try to do a PR for that? Eg what is the effect on writing the geojson file when doing that code change? 
Looking at the code this change should be possible (as `write` is just using `writerecords` with a list of length 1 under the hood)


Ok, will do. I thought of whether GeoJSON will be affected too.

I'm quite bothered by the files performance issues, and found myself using CSV quite often via two wrapper functions (they convert GeoDataFrame to DataFrame with WKT and vice versa). But Pandas often misinterprets column format in CSV.

And regarding columns, that sounds very cool, because I thought of making a lazy dataset call for myself, splitting GDF into 1 per column.

So if GPKG can be improved, a lot of weight will fall from my shoulders.
@culebron @jorisvandenbossche I think the main speed issue is explained in the performance hints section of the [OGR SQLite format docs](http://gdal.org/drv_sqlite.html):

> The default OGR behavior is to COMMIT a transaction every 200 inserted rows. This value is surely too low for SQLite; and closing too much frequently the current transaction causes severe performance degradation.

It's my understanding that the GPKG driver is based on the SQLite driver and has the same performance issue. The docs are vague on how to increase the transaction size.

The only thing to be done in your Fiona usage is to eliminate the use of a buffer and just do this:

```python
    colxn.writerecords(df.iterfeatures())
```

There's already a buffer on the OGR side and an extra Python one accomplishes nothing.

Resolving patch:
diff --git a/geopandas/io/file.py b/geopandas/io/file.py
--- a/geopandas/io/file.py
+++ b/geopandas/io/file.py
@@ -104,8 +104,7 @@ def to_file(df, filename, driver="ESRI Shapefile", schema=None,
     with fiona.drivers():
         with fiona.open(filename, 'w', driver=driver, crs=df.crs,
                         schema=schema, **kwargs) as colxn:
-            for feature in df.iterfeatures():
-                colxn.write(feature)
+            colxn.writerecords(df.iterfeatures())
 
 
 def infer_schema(df):


Test patch: