Package: nandist
Nandist: Calculating distances in arrays with missing valuesThe python librarynandistenables (fast) computation of various distances in numpy arrays containing missing (NaN) values.
These distances are implemented as a drop-in replacement for distance functions in thescipy.spatial.distancemodule.The distance functions innandistcan be used as a drop-in replacement for the distance functions inscipy.spatial.distance.
The librarynandistoffers replacements forallstandalone distance functions in scipy andpartialsupport for the fast pairwise distance calculating functionscdistandpdist.Supported functions between two vectorsSupported "standalone" distance functions for calculating distances between two arrays (complete):braycurtiscanberrachebyshevcityblockcityblockcorrelationcosineeuclideanjensenshannonmahalanobisminkowskiseuclideansqeuclideanSupported functions between arrays of vectorsSupported functions for fast calculation of (pairwise) distances between multiple arrays (partial support for metrics):cdistpdistSupported distance measures incdistandpdist(passed asmetricargument):"chebyshev""cityblock""cosine""euclidean""minkowski"InstallationUsing pip:pipinstallnandistUsageA simple example for calculating the cityblock distance between (0, 1) and (NaN, 0) is shown below.>>>importnandist>>>importscipy>>>importnumpyasnp>>>>>># City-block distance between  (0, 1) and (NaN, 0)>>>u,v=np.array([0,1]),np.array([np.nan,0])>>>scipy.spatial.distance.cityblock(u,v)nan>>>nandist.cityblock(u,v)1.0You can replace the functioncityblockby any of the supported distance functions.You can get pairwise distances between arrays in two matrices usingcdist.
The NaNs do not need to be in the same component.>>>importnandist>>>importnumpyasnp>>># City-block distances between vectors A = [(0, 0), (1, NaN)] and vectors B=[(1, NaN) and (1, 1)]>>>XA,XB=np.array([[0,0],[1,np.nan]]),np.array([[1,np.nan],[1,1]])>>>Y=nandist.cdist(XA,XB,metric="cityblock")array([[1.,2.],[0.,0.]])Supported metricsSupported distance metrics are:Chebyshev:chebyshev,metric="chebyshev"Cityblock:cityblock,metric="cityblock"Cosine:cosine,metric="cosine"Euclidean:euclidean,metric="euclidean"Minkowski:minkowski,metric="minkowski"If you require support for additional distance metrics, please submit an Issue or Merge Request.How does it workInnandist, the components where a vector is NaN will be ignored (interpreted as "any number") in the distance metric.
This is achieved by replacing NaN values with zeros and correcting the resulting overestimated distance value.
Under the hood,nandistcalls functions fromscipy.spatial.distanceand then applies the corrections usingnumpylinear algebra.
This ensures that the outcomes ofnandistfunctions are equivalent toscipy.spatial.distancedistance functions when arrays are passed without NaNs in them.
In addition, all heavy computational lifting is done throughscipy, requiring only the additional computational cost of applying the corrections.Does italwayswork?No. The packagenandistperforms a correction on an overestimation of the distances when missing values are imputed as zero.
It is possible that this correction runs into the limits of floating point arithmetic.
In that case,nandistwill raise an appropriate error.
However, you don't often run into these edge cases in typical usage.