Repository: geopandas/geopandas

Issue:
BUG: The dataset contains a mistake. Crimea should belong to Ukraine
- [+] I have checked that this issue has not already been reported.

- [+] I have confirmed this bug exists on the latest version of geopandas.

- [ ] (optional) I have confirmed this bug exists on the main branch of geopandas.

---

**Note**: Please read [this guide](https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports) detailing how to provide the necessary information for us to reproduce your bug.

#### Code Sample, a copy-pastable example

```python
# Your code here

import pandas as pd
import geopandas

data = geopandas.read_file(geopandas.datasets.get_path('naturalearth_lowres'))

ukraine = data.loc[data['name'] == 'Ukraine']
ukraine.plot()
```

#### Problem description

Plotting map of Ukraine.
Crimea isn't displayed, despite being a part of Ukraine. It is annexed indeed, but it is a temporary issue, that is going to be resolved 
soon. No self-respecting countries recognized Crimea's seperation. So it should be considered a part of Ukraine, obviously.

#### Expected Output
The map should show Crimea.
#### Output of ``geopandas.show_versions()``

<details>


SYSTEM INFO
-----------
python     : 3.7.12 (default, Jan 15 2022, 18:48:18)  [GCC 7.5.0]
executable : /usr/bin/python3
machine    : Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic

GEOS, GDAL, PROJ INFO
---------------------
GEOS       : 3.6.2
GEOS lib   : /usr/lib/x86_64-linux-gnu/libgeos_c.so
GDAL       : 3.4.1
GDAL data dir: /usr/local/lib/python3.7/dist-packages/fiona/gdal_data
PROJ       : 8.1.1
PROJ data dir: /usr/local/lib/python3.7/dist-packages/pyproj/proj_dir/share/proj

PYTHON DEPENDENCIES
-------------------
geopandas  : 0.10.2
pandas     : 1.3.5
fiona      : 1.8.21
numpy      : 1.21.5
shapely    : 1.8.1.post1
rtree      : None
pyproj     : 3.2.1
matplotlib : 3.2.2
mapclassify: None
geopy      : 1.17.0
psycopg2   : 2.7.6.1 (dt dec pq3 ext lo64)
geoalchemy2: None
pyarrow    : 6.0.1
pygeos     : None
/usr/local/lib/python3.7/dist-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)

</details>



Discussion:
Hi, this is not the only area in the world that can be considered _wrong_ (xref #1510).

We are using Natural Earth [dataset](https://www.naturalearthdata.com/downloads/110m-cultural-vectors/110m-admin-0-countries/). Crimea currently falls under the [following disclaimer](https://www.naturalearthdata.com/downloads/110m-cultural-vectors/110m-admin-0-countries/):

> Natural Earth Vector draws boundaries of countries according to defacto status. We show who actually controls the situation on the ground. Please feel free to mashup our disputed areas ([link](https://www.naturalearthdata.com/downloads/50m-cultural-vectors/50m-admin-0-breakaway-disputed-areas/)) theme to match your particular political outlook.

Though we decided not to do anything on this before and referred to Natural Earth, I have to admit that I wasn't aware of this and it does not make me very happy...

Drawing boundaries is a sensitive issue and we don't want to put such burden on the geopandas dev team, that is why we use NE as the data provider. [This policy](https://www.naturalearthdata.com/about/disputed-boundaries-policy/) is new (or updated) and contains "_optional “point-of-view” (POV) or “worldview” for administrative geographies."_ datasets, but only for 1:10 000 000 versions, not 1:110 000 000 we are using.

So I am not sure if we should actually do something here. I am afraid we would open a can of worms if we made an exception from the policy _use default NE data and let them deal with boundaries_.
Hi,

I am glad I had correctly understood the source of the issue, particularly that the NE dataset that comes with geopandas adopts the standard NE borders.

My suggestion is that of introducing a warning upon loading the dataset, pointing to the NE policy and to the fact that NE offers alternative views. In this way, the user is made aware of the potential issue and also of potential "solutions". Hopefully this is also a minimal "fix" that does not open a can on worms on the geopandas side or requires lots of work/changes.

To be clear: the suggestion is not that of having a warning for Crimea in particular, rather about “disputed territories” in general; people may not think there is a dispute at all and criticize that choice of words&warning, but I’d say it is good enough for the relatively simple low-res NE dataset to highlight the issue of disputed territories/ disagreements and direct people to more information and possible fixes.

Thank you for your work.
Hello there,

I've seen this map so this is why I resurrected my GitHub account to write this message,

Listen, we in Ukraine don't care what dataset you're using and what are their excuses. 

To start with, this map is WRONG, not to mention that it is also deeply offensive and problematic and this sort of excuse "whoever controls the ground" is not valid for lots of reasons. Should I explain what reasons exactly? 

There is a concept of internationally recognized borders, and everyone should respect it. If you're not, or whoever owns that dataset is not - is a perpetrator of the war crimes the aggressor states are committing. 

Like, do you show ISIS-controlled lands on your maps? Why not? 
Probably Al-Qaeda controlled? Hm? 

- Ah, human-rights abuse, kidnapping, genocide, and looting, and also their control of the land is not legitimate and unauthorized - that's why. But isn't that what's happening in Crimea as well? 

Possible solutions to this problem:
- find another dataset, like any that uses CORRECT data
- explain the situation to the current dataset maintainer (ofc they wouldn't listen because they've already made up their minds)

We would really want this to be fixed, with no excuses. 

Regards,
Oleh

>I am afraid we would open a can of worms 

Too late, by not showing the internationally recognized borders of Ukraine you already sided with russian genocidal war of conquest.
You can either fix this mistake or face the consequences.
"Drawing boundaries is a sensitive issue". Yes, it is a very sensitive issue for Ukrainian people which are dying right now for   free will  and our lives.  But you made a simple and inappropriate decision. No excuse for this. 
Hi, how is fixing the issues with Crimea not being part of Ukraine map "opening a can of worms"?

Crimea is a part of Ukraine, it's recognized as a part of Ukraine by international law, you being lazy and not wanting to fix it is completely inappropriate.

Map generation makes no sense at all if your maps don't align with reality.

You refusing to fix it is the same as agreeing with the *illegal* occupation of the area and indirectly allowing russia's crimes  against humanity to continue in Crimea.
You are messing Line of Control and Political border. Yes, you. Because you decided what data set should be used for your lib.
And in this way you are supporting russia - state sponsor of terrorism. Internationally declared.
There is an international law. You accept or deny it. There is no third option.
As per this discussion - https://github.com/nvkelso/natural-earth-vector/issues/812 - NaturalEarth does not seem to be willing to change anything.

@martinfleis we strongly suggest that you pick another provider for geo data which provides accurate and internationally accepted definitions of borders, since supporting the "de-facto" bullshit that NaturalEarth tries to spread will only degrade the usability of your library and potentially render it unusable.

They basically tell us they can change whatever they want based on their "observations" of "reality", and this clearly doesn't stand together with the spirit of OpenSource.

Set yourself free, support the freedom & humanity, use a different dataset!
Hi all,

thank you for your input and push towards change. This issue has been discussed during the last community meetings and we propose the following:

- we will create a new Python package interfacing the complete set of data provided by https://www.naturalearthdata.com, a source we've been using since early days and we're shipping today.
- the world dataset with the incorrectly drawn boundaries will be removed from the GeoPandas package. GeoPandas will then depend on the new package to load the datasets and expose possibility to specify the version of the map as per your preference regarding disputed territories.
- the default that comes as "naturalearth_lowres" will contain Ukraine within its lawful boundaries, i.e including Crimea, so when you do `gpd.read_file(gpd.datasets.get_path("naturalearth_lowres"))` as you do today, you and anyone else will get the correct version.

This solution should resolve not only this issue but also similar reported before (#1510).

Before we'll go this direction, would this be a solution you would approve? If not, we can try to come up with another that would be community approved.

Thank you all and accept my personal apologies for our slow response time on this sensitive issue.
A similar issue was brought to my attention by @mourner (of [Leaflet](https://github.com/Leaflet/Leaflet) fame, who lives in Kyiv). My approach was to switch from Natural Earth data to GeoBoundaries data - see https://www.geoboundaries.org/

You folks might wanna consider this approach if maintaining a copy of the NE data proves too cumbersome.
@IvanSanchez wasn't aware of those, thanks for the tip! It may also be a solution. Another one suggested elsewhere was to remove the provided data altogether but I'd rather keep some basic data easily accessible. Hopefully more correct but given the amount of disputed territories around the world, someone will always be affected when drawing a hard line on the ground. It is a sad reality of current geopolitics.
Hi @martinfleis, thank you for a quick response! I'm sure community will appreciate your willingness to fix this issue. I have some concern about the second point in your proposed plan, specifically:

> the world dataset with the incorrectly drawn boundaries will be removed from the GeoPandas package. GeoPandas will then depend on the new package to load the datasets and expose possibility to *specify the version of the map as per your preference regarding disputed territories*.

I think I understand the motivation, but adding a preference to allow user to choose between a correct and a russian map is like adding a preference of whether you want a real map or a map of flat earth. Though in case of the russian map there's also the added component of support for terrorism and genocide. I understand the motivation from a maintainer's point of view, but for an unaware user the optics can be seriously damaging to project reputation.

I understand that you can't prevent people from loading whatever map they want, due to the nature of open-source software, but adding a special option to do it seems unnecessary.
@blacktaxi Fully understand this point. What this solution meant to do was to not only cover the Crimea issue but also all the other. To cover all areas that are marked as disputed by naturalearth.com on the map below. 

<img width="1325" alt="Screenshot 2022-12-10 at 10 19 00" src="https://user-images.githubusercontent.com/36797143/206842957-a0f1395d-061a-43c3-8ec4-ded7e284a62c.png">

At this point, I see only one solution and that is to remove datasets like this from geopandas compeletely. After all, geopandas is not an open **data** project but open source **software** project.

What I will do now is to create a pull request that manually changes the current boundaries and assigns Crimea back to Ukraine where it belongs. This fix will be included in the 0.12.2 release that will come out within a couple of days.

The final implemented solution will be further discussed. 
The fix including Crimea as a part of Russia is now at #2670. We hope to release the new version with these boundaries by the end of today. 

The overall discussion on how to treat all disputed regions in GeoPandas will be moved to a separate issue and the solution will be implemented either fully or with a deprecation period in the next minor release (0.13).

Thank you all for the valuable feedback here and everywhere else. 

Resolving patch:
diff --git a/geopandas/datasets/naturalearth_creation.py b/geopandas/datasets/naturalearth_creation.py
--- a/geopandas/datasets/naturalearth_creation.py
+++ b/geopandas/datasets/naturalearth_creation.py
@@ -11,6 +11,7 @@
 from pathlib import Path
 from zipfile import ZipFile
 import tempfile
+from shapely.geometry import box
 
 version = "latest"
 urlbase = "https://www.naturalearthdata.com/"
@@ -89,6 +90,29 @@ def df_same(new, old, dataset, log):
             gdf = dl["override"](gdf)
         gdf = gdf.loc[:, dl["cols"]]
         gdf = gdf.rename(columns={c: c.lower() for c in gdf.columns})
+
+        # override Crimea #2382
+        if dl["file"] == "ne_110m_admin_0_countries.zip":
+            crimean_bbox = box(32.274, 44.139, 36.65, 46.704)
+            crimea_only = (
+                gdf.loc[gdf.name == "Russia", "geometry"]
+                .iloc[0]
+                .intersection(crimean_bbox)
+            )
+            complete_ukraine = (
+                gdf.loc[gdf.name == "Ukraine", "geometry"].iloc[0].union(crimea_only)
+            )
+            correct_russia = (
+                gdf.loc[gdf.name == "Russia", "geometry"]
+                .iloc[0]
+                .difference(crimean_bbox)
+            )
+            r_ix = gdf.loc[gdf.name == "Russia"].index[0]
+            gdf.at[r_ix, "geometry"] = correct_russia
+
+            u_ix = gdf.loc[gdf.name == "Ukraine"].index[0]
+            gdf.at[u_ix, "geometry"] = complete_ukraine
+
         # get changes between current version and new version
         if not df_same(gdf, gpd.read_file(dl["current"]), dl["file"], log):
             downloads[dl["file"]] = gdf


Test patch:
diff --git a/geopandas/tests/test_sindex.py b/geopandas/tests/test_sindex.py
--- a/geopandas/tests/test_sindex.py
+++ b/geopandas/tests/test_sindex.py
@@ -861,7 +861,7 @@ def test_is_empty(self):
     @pytest.mark.parametrize(
         "predicate, expected_shape",
         [
-            (None, (2, 470)),
+            (None, (2, 471)),
             ("intersects", (2, 213)),
             ("within", (2, 213)),
             ("contains", (2, 0)),