Package: simple-interpolation
simple_interpolationA Pandas implentation of the Brownian Bridge interpolation algorithm. Wiener processes are assumed to buildstd().Interpolation rocks, but doing it poorly can alter the original features of your data.Brownian bridge preserves the volatibility of the original data, if done well. Mixing that with a bit theory on the stock market (Wiener processes), we built a simple interpolation library.Readabout the algorithm in the "Brownian bridge algo" section below.Installpip install simple_interpolationHow to use# Example input dataframe, containing gaps#  (i. e. X column, values 3-5)dfXY008.0898461111.793489229.026726368.9961774711.221730588.398122698.84566771011.45470081111.4317459127.050733101310.00942011146.96467412159.541557131611.656722141911.062303152011.302763162113.04205717227.40567018238.98605719247.554964202510.46768821269.416683222710.03866523285.519665244510.184922254611.66166226479.748401274811.02311628499.298167importsimple_interpolation.simple_interpolationassiorfromsimple_interpolationimportcoreassiInterpolation, plot is optional (default False)patched_df = si.interpolate_gaps( df , plot = True )
patched_dfNo datetime column: assuming first column 'X' as X-axis
std() built with Wiener method
Will interpolate if X-column interval is more than 1.7675
Processed 0.00% of gaps
Ended interpolation, starting plotting the results..XYinterpolated00.00008.089846011.000011.793489022.00009.026726033.00008.291588144.00008.486541155.00008.736440166.00008.996177077.000011.221730088.00008.398122099.00008.84566701010.000011.45470001111.000011.43174501212.00007.05073301313.000010.00942001414.00006.96467401515.00009.54155701616.000011.65672201717.500011.35951211819.000011.06230301920.000011.30276302021.000013.04205702122.00007.40567002223.00008.98605702324.00007.55496402425.000010.46768802526.00009.41668302627.000010.03866502728.00005.51966502829.06256.58444312930.12505.50477313031.18755.62387513132.25006.27512613233.31256.63913913334.37506.39427713435.43756.79700813536.50007.88582813637.56258.53059413738.62508.92119113839.68758.94138213940.75008.90056514041.81259.03725114142.87509.36073014243.93759.91464114345.000010.18492204446.000011.66166204547.00009.74840104648.000011.02311604749.00009.2981670Effects of interpolation on seasonalityYou can check theeffects of the interpolation or gap addendum on the seasonalitiesof the signal. The next method usesFacebook's Prophetmodels (you can install viapip install fbprophet) on 4 different versions of your data:your original data,df,your data interpolated,df_i,your data with new gaps added,df_emptied,the emptied datatheninterpolated,df_emptied_i.importsimple_interpolation.prophet_sensibilityasps# This also plots the thing belowmodels=ps.prophet_sensibility(df,slicing_threshold=5000)Brownian bridge algo: the theoryTo render the equations on browser, install a LaTex rendering extension. Otherwise download it and open it on Jupyer.Allows to interpolate large gapspreserving volatilityof the series (as an input!). Read about ithere "Brownian bridge".Weiner method to obtain the relevant std()In aWiener processvolatility (variance) is $$var = \Delta_t$$ so $$std = \sqrt{var} = \sqrt{\Delta_t}$$This sets how thelocal volatilityshould be analyzed.So, if we have $std_{year}$ (or $std_{whole series}$), we can get the daily by: $$std_{year} = std_{day} \cdot  \sqrt{365} \Rightarrow std_{day} = \frac{std_{year}}{\sqrt{365}}$$So we can get the "basic building block" of the volatility by getting $std_{minute}$ in our case.Having $std_{minute}$, we then do a "bottom-up" process building the gap:$$ std_{gap} = std_{minute} \cdot \sqrt{number_of_mins_in_gap}$$(Advice from Miguel, my colleague at ING)Fixed timestepsYou can usefixed_freqargument to make theinterpolated X points rounded to a certain timestep. 'fixed_freq' timesteps defaults toNone(not fixed freq). Valid options from Pandas, see link:https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#timeseries-offset-aliasesImplementation of the rounding (you probably don't need to read this)This constraint takes usout of the brownian bridge, because for it we only interpolate themidpointsthrough:\begin{cases}
x_m = \frac{x_0 + x_1}{2} \
y_m = \frac{y_0 + y_1}{2} + std
\end{cases}But, if we round up to mins, this midpoint $x_m$ could be different than a minute-exact timestamp (imagine the first interpolated point on a gap of 3m: it would be 1.5m). Sowe round $x_m$, and search for itsassociated Y displacement$\Delta y$:\begin{cases}
x'm = x_m + \Delta x{toroundtomin} \
y'_m = y_m + \Delta y
\end{cases}To get the associated $\Delta y$ we must use theslope (derivative)at that straight line between points $(x_0, y_0), (x_1, y_1)$.So:1-Round up $x_m$ to the nearest minute(lowest,floor()-like), sowe obtain: $x'm$, $\Delta x{toroundtomin}$2- The deltas on X and Y are related by the derivative, which we are implicitly assuming linear on the brownian bridge, so it's quite straightforward to calculate $\Delta y$:$$ \Delta y := \frac{dy}{dx} \Delta x \Rightarrow  \Delta y \approx \frac{y_1 - y_0}{x_1 - x_0} \Delta x_{toroundtomin} $$So we would have everything for the Y correction.