Repository: SciTools/cartopy

Issue:
contourf method significantly slower than basemap
### Description
I'm migrating my codes from Basemap to Cartopy. Generally Cartopy is always faster, however I find contourf method can take me seconds. Here's my example code.

```python
import time

import cartopy
import matplotlib.pyplot as plt
import numpy as np
import scipy.ndimage as sn
from mpl_toolkits.basemap import Basemap


#-------------------
#  Create data
#-------------------
x, y = np.mgrid[60:140:0.25, 0:45:0.25]
r = np.random.randint(0, 255, size=(320, 180), dtype='uint8')
r = sn.filters.gaussian_filter(r, (3, 3), mode='constant')

#-------------------
#  Cartopy
#-------------------
ax = plt.axes(projection=cartopy.crs.LambertConformal(
    central_longitude=100.0, central_latitude=35.0))
ax.set_extent((60,140,0,45))
for step in (20, 10, 5):
    tic = time.time()
    col = ax.contourf(x, y, r, transform=cartopy.crs.PlateCarree(),
        levels=np.arange(100,161,step))
    toc = time.time()
    points = sum(len(e.get_paths()) for e in col.collections)
    print('[Cartopy] Step: {} Points: {}  Time: {:.0f}ms'.format(
        step, points, (toc - tic) * 1000))


#-------------------
#  Basemap
#-------------------
_map = Basemap(projection='lcc', resolution=None,
    lon_0=100.0, lat_0=35.0, llcrnrlat=0,
    urcrnrlat=45, llcrnrlon=60, urcrnrlon=140)
for step in (20, 10, 5):
    tic = time.time()
    col = _map.contourf(x, y, r, levels=np.arange(100,161,step))
    toc = time.time()
    points = sum(len(e.get_paths()) for e in col.collections)
    print('[Basemap] Step: {} Points: {}  Time: {:.0f}ms'.format(
        step, points, (toc - tic) * 1000))

```
The result:
```
[Cartopy] Step: 20 Points: 123  Time: 729ms
[Cartopy] Step: 10 Points: 273  Time: 1433ms
[Cartopy] Step: 5 Points: 528  Time: 2887ms
[Basemap] Step: 20 Points: 123  Time: 118ms
[Basemap] Step: 10 Points: 273  Time: 151ms
[Basemap] Step: 5 Points: 528  Time: 179ms
```
which means Cartopy can be up to 16x slower on my old laptop. After some profiling work I find the culprit is https://github.com/SciTools/cartopy/blob/master/lib/cartopy/mpl/geoaxes.py#L1505, it seems this step is to calculate new datalim and autoscale the extent. This is how far I can get to since I don't have enough time to dig in.

- Why does this step take so much time?
- As a quick workaround, can we just check if the extent is already set (which is the most scenario imo) before autoscaling?

### Environment
- **OS**: Windows 10
- **cartopy**: 0.17.0
- **basemap**: 1.2.0
- **matplotlib**: 3.0.2
- **shapely**: 1.6.4.post1

Thanks!


Discussion:
This is a critical problem.  In trying to translate one of my oceanography class notebooks from basemap to cartopy, a 2-second basemap plot took over 2 *minutes* with cartopy.  That's prohibitive. As @crazyapril notes, the problem seems to be coming from a calculation added in 2016:
https://github.com/SciTools/cartopy/pull/730/commits/d74eeddb22cc98c7a13f0f69b3c8949f0bd00d3a
@pelson, do you remember why this was needed, and why it is done only for filled contours, not for line contours?  Is there a good test case that shows the need for this calculation?
Fundamentally cartopy's contour implementation is doing a lot more work than Basemap needs to do, as we are projecting the contour lines once they have been calculated. The commit referenced is ensuring that the data limits are respected by transforming the coordinates from source to target projection.

If this calculation is a prohibitive/expensive operation, then it is always possible to fall back to the equivalent Basemap behaviour - namely, you transform your data yourself, and pass the reprojected data to be contoured in the target project. It isn't ideal as the whole point of cartopy is that you shouldn't have to worry about this stuff (like datelines and poles), but it does highlight that cartopy is a superset of the capabilities available with Basemap.

To illustrate my meaning:

Current code:
```
ax = plt.axes(projection=target)
ax.contourf(src_x, src_y, data, transform=source)
```

Becomes:
```
ax = plt.axes(projection=target)
target_x, target_y, target_data = whatever_used_to_be_done_in_basemap(src_x, src_y, data)
ax.contourf(target_x, target_y, target_data, transform=target)
```

In the case of contouring, `whatever_used_to_be_done_in_basemap` is simply applying the projection transformation from lon, lat to the projected coordinates (meters?) via the `__call__` method of the `Basemap` object, which invokes the corresponding proj4 function.  The cartopy equivalent would be via the `CRS.transform_points` method, correct?

There is still the question of why the expensive calculation to check data limits is invoked only for contourf and not for contour.

And, I suspect the calculation is itself much more expensive than it might need to be, because it is having to navigate a tangle of python steps to do many transform calls instead of just transforming a very large number of points in one call into proj4.

I appreciate the convenience, generality, and elegance of cartopy's design.  I'm hoping that performance in some of the worst cases can be greatly improved with some optimizations.
I've experienced this issue as well, generating a lot of contourf maps in Basemap takes me 15 minutes but in Cartopy it takes 1 hour 4 minutes.

I tried working around it as suggested above with `CRS.transform_points`, but got `ValueError: x and y arrays must have the same length `. I think I'm missing something here, since surely you should be able to transform longitudes and latitudes that don't form a perfect square (same length)?

Anyway, hoping that whatever causes this slowdown can be fixed, or a workaround appears. 
To do it manually with lat/lon arrays that each define 1 dimension of a 2D region, you need to do the work of assembling them into the same size arrays. You can accomplish this using `np.meshgrid`.
So one year later I restarted the migration based on the suggestion above and in case someone may need it, just replace `ax.contourf(x, y, data, transform=source_proj)` with:
```python
ret = ax.projection.transform_points(source_proj, np.array(x),
    np.array(y), np.array(data)) # This method only accept ndarray!
xx = ret[..., 0]
yy = ret[..., 1]
data = ret[..., 2]
ax.contourf(xx, yy, data)
```
Still, I think in this case the slowdown it brings somewhat outweigh the convenience. It would be better if this behavior is explained in the documentation.
Also, I find if latitudes cross the dateline, using only `np.meshgrid` as @dopplershift suggested would cause a blank output. A bug?
Unless your data actually represent a height above ground, you shouldn't be passing that to `transform_points`, so I would try the above as:
```python
ret = ax.projection.transform_points(source_proj, np.array(x),
    np.array(y)) # This method only accept ndarray!
xx = ret[..., 0]
yy = ret[..., 1]
ax.contourf(xx, yy, data)
```
I don't think the data transformation is the bottleneck. It may make it slower, but the problem here as noted above is the calculation of the data limits. After doing some testing, even without a data transformation cartopy is MUCH slower than matplotlib with contouring.

I use cartopy in several operational scripts and it has gotten to the point where the plotting speed (specifically when using contourf) is a major limiting factor; the plots just can't keep up with the data coming in (unless we had 100's of cpu's available). As the community continues to migrate to cartopy, I think others will run into the same problem. I think this is an issue which should get some more attention to find out where exactly the bottlenecks are and what can be done. I would think that all of the contour math calculations should be vectorized, if possible, as a start.
I remember profiling all of this a while ago and the bottleneck is in the `transform_non_affine` function because we are projecting the contour lines from the source projection to target projection as mentioned above. There is a slow for loop to go through all of the geometries and project them into the new projection and currently there really isn't a great way to get around that. There is ongoing work to [merge pygeos and Shapely](https://github.com/Toblerity/Shapely/issues/782) capabilities so that you can do more vectorized calculations which I think would directly apply to this situation.

If you need it operationally fast, did you try the workaround [mentioned above](https://github.com/SciTools/cartopy/issues/1291#issuecomment-530054846)?
Greg, I'm not sure I'm understanding that workaround correctly. In my test, the data are already in axes coordinates. Cartopy is slower even when the coordinates do not need to be transformed. To explain what I mean, here's an example - I am not passing any transform kwarg to contourf - but the contourf call is far slower with a cartopy axes instance than matplotlib:

```python
import numpy as np
import cartopy.crs as ccrs
import matplotlib.pyplot as plt

lat = np.arange(-10,90,0.1)
lon = np.arange(0,100,0.1)
lons,lats = np.meshgrid(lon,lat)
data = np.random.rand(len(lat), len(lon))

fig = plt.figure(figsize=(12,9))
ax = fig.add_subplot(111, projection=ccrs.PlateCarree()) # cartopy axes
#ax = fig.add_subplot(111) # contouring on regular matplotlib axes is much faster

ax.contourf(lons, lats, data) # no data transform
```
Here's a screenshot of the profile for contouring on the cartopy axes:
![Screenshot from 2020-11-02 11-43-25](https://user-images.githubusercontent.com/45180714/97894615-b6dbdd80-1d00-11eb-8b7d-d9c16de3ff35.png)

What takes ~15 seconds to do w/ cartopy takes only ~5 seconds with matplotlib alone.
Thanks @karlwx! I did not realize that was the case even without transforms. I also don't get why the same accurate bounding box calculations aren't computed for the `contour` case, but only for the filled version here.

I agree this seems like a pretty big speed problem, especially if someone is already setting the extent manually and doesn't care about the limits being updated accurately. I think the lowest hanging fruit would be to add a keyword argument to `contourf` to tell it to ignore the accurate limit calculations, leave it True by default and allow a user to override it if they want the fast path.
https://github.com/SciTools/cartopy/blob/d12c86ca228365c61d5fb8b7121b49201f19fea8/lib/cartopy/mpl/geoaxes.py#L1574-L1580
After digging into this, it appears the issue runs deeper than we thought. While changing the above code improves speed of the contourf call itself, It turns out there are some serious slow-downs when the image is drawn for display. I added ```plt.savefig()``` to the code and here is the result:
![Screenshot from 2020-11-09 16-58-53](https://user-images.githubusercontent.com/45180714/98602372-57587180-22ae-11eb-900e-f0044442114d.png)
It looks like something to do with the matplotlib Axes draw wrapper, but I'm not entirely sure how it all fits together. 
I have run into this issue as well trying to use `contourf` via MetPy. Right now, an RTMA for 2mtr temperatures across the CONUS takes ~5min to plot the contour on an X5650. `contour` is just fine and I don't see any speed degradations. 

>  
    # @param [String] passed_run_hour
    # @param [String] passed_final_out_directory
    # @param [String] passed_netcdf_file_path
    def __makeTheGraphics(self, passed_run_hour, passed_final_out_directory, passed_netcdf_file_path):
        data = self.__getNetCDFData(passed_netcdf_file_path)
        print(list(data.variables))
        self.logging.info(f'Establishing variables...')
        # Establish the variables
        lat = data['latitude']
        lon = data['longitude']
        times = data['time']
        temp2mtr = data['TMP_2maboveground'][0]
        # Convert temp to °F
        temp2mtr.metpy.convert_units('degF')
        dp2mtr = data['DPT_2maboveground'][0]
        visibility = data['VIS_surface'][0]
        ####THIS IS WHERE MULTIPROCESSING SHOULD START####
        self.logging.info(f'Beginning creation of plot...')
        # Establish the projection
        crs = ccrs.Mercator()
        # Based upon 100dpi as default 1500x1000 Matplotlib
        fig = plt.figure(figsize=(15., 10.), frameon=False)
        # Matplotlib Adds an Axes to the figure as part of a subplot arrangement
        ax = fig.add_subplot(1, 1, 1, projection=crs)
        # Set the lat/long bounds of the image
        region = "conus"
        bounds = [(-127, -65, 23, 50)]
        ax.set_extent(*bounds, crs=ccrs.PlateCarree())
        # Add details to the map like borders and counties
        ax.add_feature(cfeature.COASTLINE.with_scale('50m'), linewidth=0.75)
        ax.add_feature(cfeature.STATES.with_scale('50m'), linewidth=0.5) # edgecolor='#0099ff'
        ax.add_feature(cfeature.BORDERS.with_scale('50m'), linewidth=1.0)
        #ax.add_feature(USCOUNTIES.with_scale('5m'))
        # Apply Title
        #plt.suptitle('2mtr Temp Test', x=0.25, y=0.25, horizontalalignment='left', verticalalignment='top', fontsize = 16)
        ax.set_title(f'Valid: {passed_run_hour}Z', loc='left')
        self.logging.info(f'Adding contours to plot...')
        #Contour Plot Of 2mtr Temp
        cf = ax.contourf(lon, lat, temp2mtr, range(-40, 120, 5), cmap=plt.cm.gist_earth_r, transform=ccrs.PlateCarree())
        c32f = ax.contour(lon, lat, temp2mtr, [32], transform=ccrs.PlateCarree(), colors='r')
        cb = fig.colorbar(cf, orientation='horizontal', extend='max')
        self.logging.info(f'Saving file -- {passed_final_out_directory}{passed_run_hour}/{region}-2mTemp.jpg')
        fig.savefig(f'{passed_final_out_directory}{passed_run_hour}/{region}-2mTemp.jpg', pad_inches=0.0, format="jpg")


![image](https://user-images.githubusercontent.com/5630428/99925200-2a26ad00-2d0b-11eb-915f-9dc68ef6c2cf.png)


I had some success improving the speed of hi-res plots like that by using xarray coarsen to resample the data to a lower resolution (did this with National Blend of Models), but it kind of defeats the purpose of the hi-res guidance in the first place. The bottom line is, the plotting time is still excessive.
I'll mess around with things a little more this evening. I pulled up some code from when I was playing with GOES16 data a year ago or so. In that code, I used `pcolormesh` to generate the image. I can't remember the difference between that and `contourf` right now.
So the difference is ```pcolormesh``` does filled grid boxes, while ```contourf``` is filled contour lines. At that resolution on a national map, ```pcolormesh``` might look okay. It will also be MUCH faster. The disadvantage of ```pcolormesh``` is it produces a "blocky" look with low resolution data. ```contourf``` is always going to be more visually-appealing, but the trade-off is how slow it is.
I'll run a benchmark between the two and also drop the images to show comparisons in their quality. This could be an alternative at least when dealing with hi-res data being displayed in CONUS.
Putting a few cross-referenced issues here with other libraries dealing with the non-affine transforms being slow.

MPL issue with slow non-affine contours:
https://github.com/matplotlib/matplotlib/issues/11464

Astropy's fix:
https://github.com/astropy/astropy/pull/7568

I think there will still be issues with the contours if one of them gets wrapped and needs to be cut along a boundary or something like that though, so the Astropy fix may not be totally suitable for our use-case.

Resolving patch:
diff --git a/lib/cartopy/mpl/geoaxes.py b/lib/cartopy/mpl/geoaxes.py
--- a/lib/cartopy/mpl/geoaxes.py
+++ b/lib/cartopy/mpl/geoaxes.py
@@ -321,6 +321,50 @@ def wrapper(self, *args, **kwargs):
     return wrapper
 
 
+def _add_transform_first(func):
+    """
+    A decorator that adds and validates the transform_first keyword argument.
+
+    This handles a fast-path optimization that projects the points before
+    creating any patches or lines. This means that the lines/patches will be
+    calculated in projected-space, not data-space. It requires the first
+    three arguments to be x, y, and z and all must be two-dimensional to use
+    the fast-path option.
+
+    This should be added after the _add_transform wrapper so that a transform
+    is guaranteed to be present.
+    """
+    @functools.wraps(func)
+    def wrapper(self, *args, **kwargs):
+        if kwargs.pop('transform_first', False):
+            if len(args) < 3:
+                # For the fast-path we need X and Y input points
+                raise ValueError("The X and Y arguments must be provided to "
+                                 "use the transform_first=True fast-path.")
+            x, y, z = (np.array(i) for i in args[:3])
+            if not (x.ndim == y.ndim == 2):
+                raise ValueError("The X and Y arguments must be gridded "
+                                 "2-dimensional arrays")
+
+            # Remove the transform from the keyword arguments
+            t = kwargs.pop('transform')
+            # Transform all of the x and y points
+            pts = self.projection.transform_points(t, x, y)
+            x = pts[..., 0].reshape(x.shape)
+            y = pts[..., 1].reshape(y.shape)
+            # The x coordinates could be wrapped, but matplotlib expects
+            # them to be sorted, so we will reorganize the arrays based on x
+            ind = np.argsort(x, axis=1)
+            x = np.take_along_axis(x, ind, axis=1)
+            y = np.take_along_axis(y, ind, axis=1)
+            z = np.take_along_axis(z, ind, axis=1)
+
+            # Use the new points as the input arguments
+            args = (x, y, z) + args[3:]
+        return func(self, *args, **kwargs)
+    return wrapper
+
+
 class GeoAxes(matplotlib.axes.Axes):
     """
     A subclass of :class:`matplotlib.axes.Axes` which represents a
@@ -1593,6 +1637,7 @@ def set_boundary(self, path, transform=None, use_as_clip_path=None):
         self.spines['geo'].set_boundary(path, transform)
 
     @_add_transform
+    @_add_transform_first
     def contour(self, *args, **kwargs):
         """
         Add the "transform" keyword to :func:`~matplotlib.pyplot.contour`.
@@ -1602,6 +1647,16 @@ def contour(self, *args, **kwargs):
         transform
             A :class:`~cartopy.crs.Projection`.
 
+        transform_first : bool, optional
+            If True, this will transform the input arguments into
+            projection-space before computing the contours, which is much
+            faster than computing the contours in data-space and projecting
+            the filled polygons. Using this method does not handle wrapped
+            coordinates as well and can produce misleading contours in the
+            middle of the domain. To use the projection-space method the input
+            arguments X and Y must be provided and be 2-dimensional.
+            The default is False, to compute the contours in data-space.
+
         """
         result = matplotlib.axes.Axes.contour(self, *args, **kwargs)
 
@@ -1621,6 +1676,7 @@ def contour(self, *args, **kwargs):
         return result
 
     @_add_transform
+    @_add_transform_first
     def contourf(self, *args, **kwargs):
         """
         Add the "transform" keyword to :func:`~matplotlib.pyplot.contourf`.
@@ -1630,8 +1686,17 @@ def contourf(self, *args, **kwargs):
         transform
             A :class:`~cartopy.crs.Projection`.
 
-        """
-        t = kwargs['transform']
+        transform_first : bool, optional
+            If True, this will transform the input arguments into
+            projection-space before computing the contours, which is much
+            faster than computing the contours in data-space and projecting
+            the filled polygons. Using this method does not handle wrapped
+            coordinates as well and can produce misleading contours in the
+            middle of the domain. To use the projection-space method the input
+            arguments X and Y must be provided and be 2-dimensional.
+            The default is False, to compute the contours in data-space.
+        """
+        t = kwargs.get('transform')
         if isinstance(t, ccrs.Projection):
             kwargs['transform'] = t = t._as_mpl_transform(self)
         # Set flag to indicate correcting orientation of paths if not ccw


Test patch:
diff --git a/lib/cartopy/tests/mpl/test_contour.py b/lib/cartopy/tests/mpl/test_contour.py
--- a/lib/cartopy/tests/mpl/test_contour.py
+++ b/lib/cartopy/tests/mpl/test_contour.py
@@ -8,6 +8,7 @@
 from matplotlib.testing.decorators import cleanup
 import numpy as np
 from numpy.testing import assert_array_almost_equal
+import pytest
 from scipy.interpolate import NearestNDInterpolator
 from scipy.signal import convolve2d
 
@@ -94,3 +95,38 @@ def test_contour_update_bounds():
     # doesn't raise with an Orthographic projection
     # GH issue 1673
     plt.draw()
+
+
+@pytest.mark.parametrize('func', ['contour', 'contourf'])
+@cleanup
+def test_contourf_transform_first(func):
+    """Test the fast-path option for filled contours."""
+    # Gridded data that needs to be wrapped
+    x = np.arange(360)
+    y = np.arange(-25, 26)
+    xx, yy = np.meshgrid(x, y)
+    z = xx + yy**2
+
+    ax = plt.axes(projection=ccrs.PlateCarree())
+    test_func = getattr(ax, func)
+    # Can't handle just Z input with the transform_first
+    with pytest.raises(ValueError,
+                       match="The X and Y arguments must be provided"):
+        test_func(z, transform=ccrs.PlateCarree(),
+                  transform_first=True)
+    # X and Y must also be 2-dimensional
+    with pytest.raises(ValueError,
+                       match="The X and Y arguments must be gridded"):
+        test_func(x, y, z, transform=ccrs.PlateCarree(),
+                  transform_first=True)
+
+    # When calculating the contour in projection-space the extent
+    # will now be the extent of the transformed points (-179, 180, -25, 25)
+    test_func(xx, yy, z, transform=ccrs.PlateCarree(),
+              transform_first=True)
+    assert_array_almost_equal(ax.get_extent(), (-179, 180, -25, 25))
+
+    # The extent without the transform_first should be all the way out to -180
+    test_func(xx, yy, z, transform=ccrs.PlateCarree(),
+              transform_first=False)
+    assert_array_almost_equal(ax.get_extent(), (-180, 180, -25, 25))