Repository: holoviz/holoviews

Issue:
Dynspread fails to handle NaN properly
#### ALL software version info
HoloViews: 1.13.2
Pandas: 0.24.2
NumPy: 1.18.1
Dask: 2.14.0

#### Description of expected behavior and the observed behavior
Calling the function `dynspread(datashade(x)),` where `x` is an element that gets correctly displayed just by calling `datashade` (e.g., a `hv.Curve` element) yields `ZeroDivisionError` if `x` is a vector full of `NaN`.

#### Screenshots or screencasts of the bug in action
![1](https://user-images.githubusercontent.com/60734967/79072979-3ca42700-7ce4-11ea-9c12-95d4a2b8b51b.png)
![2](https://user-images.githubusercontent.com/60734967/79072982-3d3cbd80-7ce4-11ea-87bf-56d03cb30a18.png)

#### Stack traceback and/or browser JavaScript console output
```
---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
C:\Anaconda3\lib\site-packages\IPython\core\formatters.py in __call__(self, obj, include, exclude)
    968 
    969             if method is not None:
--> 970                 return method(include=include, exclude=exclude)
    971             return None
    972         else:

C:\Anaconda3\lib\site-packages\holoviews\core\dimension.py in _repr_mimebundle_(self, include, exclude)
   1302         combined and returned.
   1303         """
-> 1304         return Store.render(self)
   1305 
   1306 

C:\Anaconda3\lib\site-packages\holoviews\core\options.py in render(cls, obj)
   1393         data, metadata = {}, {}
   1394         for hook in hooks:
-> 1395             ret = hook(obj)
   1396             if ret is None:
   1397                 continue

C:\Anaconda3\lib\site-packages\holoviews\ipython\display_hooks.py in pprint_display(obj)
    280     if not ip.display_formatter.formatters['text/plain'].pprint:
    281         return None
--> 282     return display(obj, raw_output=True)
    283 
    284 

C:\Anaconda3\lib\site-packages\holoviews\ipython\display_hooks.py in display(obj, raw_output, **kwargs)
    253     elif isinstance(obj, (Layout, NdLayout, AdjointLayout)):
    254         with option_state(obj):
--> 255             output = layout_display(obj)
    256     elif isinstance(obj, (HoloMap, DynamicMap)):
    257         with option_state(obj):

C:\Anaconda3\lib\site-packages\holoviews\ipython\display_hooks.py in wrapped(element)
    144         try:
    145             max_frames = OutputSettings.options['max_frames']
--> 146             mimebundle = fn(element, max_frames=max_frames)
    147             if mimebundle is None:
    148                 return {}, {}

C:\Anaconda3\lib\site-packages\holoviews\ipython\display_hooks.py in layout_display(layout, max_frames)
    218         return None
    219 
--> 220     return render(layout)
    221 
    222 

C:\Anaconda3\lib\site-packages\holoviews\ipython\display_hooks.py in render(obj, **kwargs)
     66         renderer = renderer.instance(fig='png')
     67 
---> 68     return renderer.components(obj, **kwargs)
     69 
     70 

C:\Anaconda3\lib\site-packages\holoviews\plotting\renderer.py in components(self, obj, fmt, comm, **kwargs)
    374             doc = Document()
    375             with config.set(embed=embed):
--> 376                 model = plot.layout._render_model(doc, comm)
    377             if embed:
    378                 return render_model(model, comm)

C:\Anaconda3\lib\site-packages\panel\viewable.py in _render_model(self, doc, comm)
    363         if comm is None:
    364             comm = state._comm_manager.get_server_comm()
--> 365         model = self.get_root(doc, comm)
    366 
    367         if config.embed:

C:\Anaconda3\lib\site-packages\panel\viewable.py in get_root(self, doc, comm)
    534         """
    535         doc = doc or _curdoc()
--> 536         root = self._get_model(doc, comm=comm)
    537         self._preprocess(root)
    538         ref = root.ref['id']

C:\Anaconda3\lib\site-packages\panel\layout.py in _get_model(self, doc, root, parent, comm)
    134         if root is None:
    135             root = model
--> 136         objects = self._get_objects(model, [], doc, root, comm)
    137         props = dict(self._init_properties(), objects=objects)
    138         model.update(**self._process_param_change(props))

C:\Anaconda3\lib\site-packages\panel\layout.py in _get_objects(self, model, old_objects, doc, root, comm)
    124             else:
    125                 try:
--> 126                     child = pane._get_model(doc, root, model, comm)
    127                 except RerenderError:
    128                     return self._get_objects(model, current_objects[:i], doc, root, comm)

C:\Anaconda3\lib\site-packages\panel\pane\holoviews.py in _get_model(self, doc, root, parent, comm)
    225             plot = self.object
    226         else:
--> 227             plot = self._render(doc, comm, root)
    228 
    229         plot.pane = self

C:\Anaconda3\lib\site-packages\panel\pane\holoviews.py in _render(self, doc, comm, root)
    284             kwargs = {}
    285 
--> 286         return renderer.get_plot(self.object, **kwargs)
    287 
    288     def _cleanup(self, root):

C:\Anaconda3\lib\site-packages\holoviews\plotting\bokeh\renderer.py in get_plot(self_or_cls, obj, doc, renderer, **kwargs)
     71         combining the bokeh model with another plot.
     72         """
---> 73         plot = super(BokehRenderer, self_or_cls).get_plot(obj, doc, renderer, **kwargs)
     74         if plot.document is None:
     75             plot.document = Document() if self_or_cls.notebook_context else curdoc()

C:\Anaconda3\lib\site-packages\holoviews\plotting\renderer.py in get_plot(self_or_cls, obj, doc, renderer, comm, **kwargs)
    209 
    210         # Initialize DynamicMaps with first data item
--> 211         initialize_dynamic(obj)
    212 
    213         if not renderer:

C:\Anaconda3\lib\site-packages\holoviews\plotting\util.py in initialize_dynamic(obj)
    249             continue
    250         if not len(dmap):
--> 251             dmap[dmap._initial_key()]
    252 
    253 

C:\Anaconda3\lib\site-packages\holoviews\core\spaces.py in __getitem__(self, key)
   1278         # Not a cross product and nothing cached so compute element.
   1279         if cache is not None: return cache
-> 1280         val = self._execute_callback(*tuple_key)
   1281         if data_slice:
   1282             val = self._dataslice(val, data_slice)

C:\Anaconda3\lib\site-packages\holoviews\core\spaces.py in _execute_callback(self, *args)
   1052 
   1053         with dynamicmap_memoization(self.callback, self.streams):
-> 1054             retval = self.callback(*args, **kwargs)
   1055         return self._style(retval)
   1056 

C:\Anaconda3\lib\site-packages\holoviews\core\spaces.py in __call__(self, *args, **kwargs)
    660         kwarg_hash = kwargs.pop('_memoization_hash_', ())
    661         (self.args, self.kwargs) = (args, kwargs)
--> 662         if not args and not kwargs and not any(kwarg_hash): return self.callable()
    663         inputs = [i for i in self.inputs if isinstance(i, DynamicMap)]
    664         streams = []

C:\Anaconda3\lib\site-packages\holoviews\util\__init__.py in dynamic_operation(*key, **kwargs)
    984         def dynamic_operation(*key, **kwargs):
    985             key, obj = resolve(key, kwargs)
--> 986             return apply(obj, *key, **kwargs)
    987 
    988         operation = self.p.operation

C:\Anaconda3\lib\site-packages\holoviews\util\__init__.py in apply(element, *key, **kwargs)
    980         def apply(element, *key, **kwargs):
    981             kwargs = dict(util.resolve_dependent_kwargs(self.p.kwargs), **kwargs)
--> 982             return self._process(element, key, kwargs)
    983 
    984         def dynamic_operation(*key, **kwargs):

C:\Anaconda3\lib\site-packages\holoviews\util\__init__.py in _process(self, element, key, kwargs)
    962         elif isinstance(self.p.operation, Operation):
    963             kwargs = {k: v for k, v in kwargs.items() if k in self.p.operation.param}
--> 964             return self.p.operation.process_element(element, key, **kwargs)
    965         else:
    966             return self.p.operation(element, **kwargs)

C:\Anaconda3\lib\site-packages\holoviews\core\operation.py in process_element(self, element, key, **params)
    170             self.p = param.ParamOverrides(self, params,
    171                                           allow_extra_keywords=self._allow_extra_keywords)
--> 172         return self._apply(element, key)
    173 
    174 

C:\Anaconda3\lib\site-packages\holoviews\core\operation.py in _apply(self, element, key)
    130         element_pipeline = getattr(element, '_pipeline', None)
    131 
--> 132         ret = self._process(element, key)
    133         for hook in self._postprocess_hooks:
    134             ret = hook(self, ret, **kwargs)

C:\Anaconda3\lib\site-packages\holoviews\operation\datashader.py in _process(self, element, key)
   1557         else:
   1558             new_data = array
-> 1559         return element.clone(new_data, **kwargs)
   1560 
   1561 

C:\Anaconda3\lib\site-packages\holoviews\element\raster.py in clone(self, data, shared_data, new_type, link, *args, **overrides)
    427             overrides = dict(sheet_params, **overrides)
    428         return super(Image, self).clone(data, shared_data, new_type, link,
--> 429                                         *args, **overrides)
    430 
    431 

C:\Anaconda3\lib\site-packages\holoviews\core\data\__init__.py in clone(self, data, shared_data, new_type, link, *args, **overrides)
   1155 
   1156         new_dataset = super(Dataset, self).clone(
-> 1157             data, shared_data, new_type, *args, **overrides
   1158         )
   1159 

C:\Anaconda3\lib\site-packages\holoviews\core\dimension.py in clone(self, data, shared_data, new_type, link, *args, **overrides)
    569         # Apply name mangling for __ attribute
    570         pos_args = getattr(self, '_' + type(self).__name__ + '__pos_params', [])
--> 571         return clone_type(data, *args, **{k:v for k,v in settings.items()
    572                                           if k not in pos_args})
    573 

C:\Anaconda3\lib\site-packages\holoviews\element\raster.py in __init__(self, data, kdims, vdims, **params)
    676             # Handle all forms of packed value dimensions
    677             vdims.append(alpha)
--> 678         super(RGB, self).__init__(data, kdims=kdims, vdims=vdims, **params)
    679 
    680 

C:\Anaconda3\lib\site-packages\holoviews\element\raster.py in __init__(self, data, kdims, vdims, bounds, extents, xdensity, ydensity, rtol, **params)
    322             l, b, r, t = bounds.lbrt()
    323             xdensity = xdensity if xdensity else util.compute_density(l, r, dim1, self._time_unit)
--> 324             ydensity = ydensity if ydensity else util.compute_density(b, t, dim2, self._time_unit)
    325         SheetCoordinateSystem.__init__(self, bounds, xdensity, ydensity)
    326         if non_finite:

C:\Anaconda3\lib\site-packages\holoviews\core\util.py in compute_density(start, end, length, time_unit)
   1991         return (length/(diff.total_seconds()*tscale))
   1992     else:
-> 1993         return length/diff
   1994 
   1995 

ZeroDivisionError: float division by zero
```


Discussion:
Solution found: surprisingly, the function `dynspread` is unable to handle such a trivial instance, since it  works properly only if **at least** two samples of the input vector `x` (let's say `x[5, 0]` and `x[374, 0]`) are non-NaN and are also **different between each other**. Demonstration:

![2020-04-16](https://user-images.githubusercontent.com/60734967/79471145-83569180-8002-11ea-915e-7b615e8f76bd.png)
![2020-04-16 (1)](https://user-images.githubusercontent.com/60734967/79471151-86518200-8002-11ea-9e6c-4d63ad6392e9.png)



Thanks for the report! We should be able to address this.
Glad to help. In the following days I will also make a post about legend handling in datashader-based plots, since the whole process seems to be a little awkward and cumbersome.
A PR would be gratefully accepted, but I'm not requesting one because I'm not yet sure precisely where it's most appropriate to make this change.  I do think it's a situation we already dealt with earlier for all-zero plots, so a quick check of the codebase might find where we should simply consider all-NaN to be similar to all-zero.

We're already planning to improve the legend handling for Datashader plots in HoloViews, mostly by re-implementing the features of the `shade` operation in JavaScript so that Bokeh has access to all the data it needs for building colorbars and legends. You might find https://anaconda.org/jbednar/datashade_vs_rasterize/notebook helpful in the meantime.
> A PR would be gratefully accepted, but I'm not requesting one because I'm not yet sure precisely where it's most appropriate to make this change. I do think it's a situation we already dealt with earlier for all-zero plots, so a quick check of the codebase might find where we should simply consider all-NaN to be similar to all-zero.

This seems a reasonable solution. Also remember that the anomalous behavior appears whenever the input vector does not recognize **at least two non-NaN values different between each other**: for instance, if you have a 1M data point vector with roughly 500k ones and 500k NaN, the anomalous behavior occurs again (check the code below).

![2020-04-16 (2)](https://user-images.githubusercontent.com/60734967/79503450-769d6200-8031-11ea-9540-d5dd7140dcf8.png)
![2020-04-16 (3)](https://user-images.githubusercontent.com/60734967/79503454-78ffbc00-8031-11ea-9926-79c4095bcb38.png)

With respect to the trivial "all-NaN" case presented before, this instance (mix of NaN and a constant value) can be related to real scenarios, where one may want to graphically display the existing elements within one or more database columns (`x = 1` is when the data exist, `x = NaN` is when it does not exist), as a function of time.

> We're already planning to improve the legend handling for Datashader plots in HoloViews, mostly by re-implementing the features of the `shade` operation in JavaScript so that Bokeh has access to all the data it needs for building colorbars and legends. You might find https://anaconda.org/jbednar/datashade_vs_rasterize/notebook helpful in the meantime.

Thank you for the suggestion, I will check this resource out.

Just so you know, the problem is that HoloViews tries to compute a range (max,min) or density (max-min) value, and with either zero or one distinct non-NaN values, there is no meaningful range.  So the behavior you describe is expected, we just have to work around it (using a default range or density in that case, so that the plot succeeds but shows either nothing or only a single distinct value. We've already dealt with that in many cases, but clearly there are more!
> A PR would be gratefully accepted, but I'm not requesting one because I'm not yet sure precisely where it's most appropriate to make this change. I do think it's a situation we already dealt with earlier for all-zero plots, so a quick check of the codebase might find where we should simply consider all-NaN to be similar to all-zero.

One idea that comes to my mind is the following. Given an input set of coordinates `(x,y)`, where both `x` and `y` are vectors having length of `m` samples:

```
# Compute min(x), max(x), min(y), max(y)

If min(x) != max(x):
   # Allow dynamic spreading along the x axis
else:
   # Do not spread along x 

If min(y) != max(y):
   # Allow dynamic spreading along the y axis
else:
   # Do not spread along y
```
> 
> We're already planning to improve the legend handling for Datashader plots in HoloViews, mostly by re-implementing the features of the shade operation in JavaScript so that Bokeh has access to all the data it needs for building colorbars and legends.

Cannot wait to see such features being implemented. With regards to that, I have a couple of suggestions/proposals that will be described in a separate post.

Looking at the issue description and traceback, I am rather puzzled. The output of `datashade(x)` is shown to work regardless of `NaN`s and `datashade` should map to RGB(A) space where `NaN` is a transparent pixel. By the time this gets passed to `dynspread` (and internal to datashader, `spread`) all the NaNs should be gone and the datatype should be `uint32`.

Inside `datashader.py`, the error occurs with `return element.clone(new_data, **kwargs)` where `element` is the input element to the operation (which displays fine, as RGB) and the final error is down to how the x and y densities (i.e the spatial configuration, not the array values!) are computed, resulting in a zero division. My only theory is that the xarray coordinates of `new_data` have been somehow messed up even though an operation such as spread should never change them from their original values. I'll now check my hunch by investigating further!


Resolving patch:
diff --git a/holoviews/operation/datashader.py b/holoviews/operation/datashader.py
--- a/holoviews/operation/datashader.py
+++ b/holoviews/operation/datashader.py
@@ -1565,7 +1565,8 @@ def _process(self, element, key=None):
             new_data[tuple(vd.name for vd in vdims)] = img
         else:
             new_data = array
-        return element.clone(new_data, **kwargs)
+        return element.clone(new_data, xdensity=element.xdensity,
+                             ydensity=element.ydensity, **kwargs)
 
 
 


Test patch: