Repository: geopandas/geopandas

Issue:
BUG: Some column names results into an error when write to a file
```
import pandas as pd
import geopandas as gpd

# Error
test = pd.DataFrame([[1, 1, 1],],columns = ['x', 'y', 'Straßenschlüssel'])

# no Error
# test = pd.DataFrame([[1, 1, 1],],columns = ['x', 'y', 'Strassenschlüssel'])

shp = gpd.GeoDataFrame(test,geometry=gpd.points_from_xy(test.x,test.y))

shp.to_file('bug')
```


Discussion:
Thanks for the report. To add more details - it affects saving to ESRI Shapefile only, as GPKG or GeoJSON works. It raises `UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdf in position 4: invalid continuation byte` due to ß in the name. 

If you specify encoding, the file saves correctly. 
```py
shp.to_file('file.shp', encoding='utf-8')
```

GeoPandas lets Fiona to take care of encoding, and its documentation says following:

> The format drivers will attempt to detect the encoding of your data, but may fail. In my experience GDAL 1.7.2 (for example) doesn’t detect that the encoding of the Natural Earth dataset is Windows-1252. In this case, the proper encoding can be specified explicitly by using the encoding keyword parameter of fiona.open(): encoding='Windows-1252'.

If you pass `encoding` to `to_file()`, it is passed to `fiona.open()` used under the hood. Honestly, I am not sure if there is anything we can do about it. If there is some solution, I think it should be in fiona.

@martinfleis thanks for the analysis.

cc @sgillies do you know if fiona should be expected to detect the field name encoding on writing automatically?
This problem comes with the (shapefile) territory. Strictly speaking, Windows-1252 is a shapefile's default encoding.

@jorisvandenbossche OGR has to know the encoding to use before the shapefiles are initialized. In fiona (or geopandas) we could test the schema field names to see if they are Windows-1252 compatible, and if not, use utf-8, but I think it would still be possible to later find that some field values are not compatible, yes?

It might be okay for geopandas to default to `encoding=utf-8`. Export would most often work. The downside would be that some older GIS desktop software wouldn't be able to read default shapefile output.

If it is not shapefile's default, I am hesitant making UTF8 the default in geopandas (as you mention, that might not be good for interoperability).

If Fiona would like to do something like checking if the field names need UTF8, that would be a nice feature I think. But I am also fine with accepting that this is a limitation of shapefile's default we have to live with. 
We should document the `encoding` option maybe more prominently in the geopandas docs then (eg the `to_file` docstring), as now you actually need to look into fiona's docs to know this option.
I want to add, that it is not the special char 'ß' only. The behaviour is strange, i.e. 'Teßt' is working as name.
> I want to add, that it is not the special char 'ß' only. The behaviour is strange, i.e. 'Teßt' is working as name.

It might be related to necessary truncating of names for shapefile as it allows only 10 characters...

Resolving patch:
diff --git a/geopandas/geodataframe.py b/geopandas/geodataframe.py
--- a/geopandas/geodataframe.py
+++ b/geopandas/geodataframe.py
@@ -508,6 +508,10 @@ def to_file(self, filename, driver="ESRI Shapefile", schema=None, **kwargs):
         The extra keyword arguments ``**kwargs`` are passed to fiona.open and
         can be used to write to multi-layer data, store data within archives
         (zip files), etc.
+
+        The format drivers will attempt to detect the encoding of your data, but
+        may fail. In this case, the proper encoding can be specified explicitly
+        by using the encoding keyword parameter, e.g. ``encoding='utf-8'``.
         """
         from geopandas.io.file import to_file
 
diff --git a/geopandas/io/file.py b/geopandas/io/file.py
--- a/geopandas/io/file.py
+++ b/geopandas/io/file.py
@@ -55,6 +55,12 @@ def read_file(filename, bbox=None, **kwargs):
     Returns
     -------
     geodataframe : GeoDataFrame
+
+    Notes
+    -----
+    The format drivers will attempt to detect the encoding of your data, but
+    may fail. In this case, the proper encoding can be specified explicitly
+    by using the encoding keyword parameter, e.g. ``encoding='utf-8'``.
     """
     if _is_url(filename):
         req = _urlopen(filename)
@@ -112,6 +118,12 @@ def to_file(df, filename, driver="ESRI Shapefile", schema=None, **kwargs):
     The *kwargs* are passed to fiona.open and can be used to write
     to multi-layer data, store data within archives (zip files), etc.
     The path may specify a fiona VSI scheme.
+
+    Notes
+    -----
+    The format drivers will attempt to detect the encoding of your data, but
+    may fail. In this case, the proper encoding can be specified explicitly
+    by using the encoding keyword parameter, e.g. ``encoding='utf-8'``.
     """
     if schema is None:
         schema = infer_schema(df)


Test patch: