Package: limesqueezer
OverviewLossy compression with controlled error tolerance for smooth data seriesTable of ContentsQuick Start GuideUser GuideCompressionParametersTolerancesErrorfunctionsStreamDecompressionCombining compression methodsMetaVersion NumberingChangelog1.0.12 2022-07-161.0.11 2022-07-161.0.10 2022-05-081.0.9 2022-04-031.0.8 2022-03-201.0.7 2021-12-071.0.6 2021-12-021.0.5 2021-12-021.0.4 2021-12-011.0.3 2021-11-30Quick Start Guidelimesqueezer uses numpy ndarrays types for input and output.
Package import name islimesqueezer.
Author recommends abbreviationlsRest of documentation uses this abbreviation.importnumpyasnpimportlimesqueezeraslsFor example let's make mock data.
An array of values (xdata) and an array of dependent values (ydata).
You can imagine that these have come e.g. from some simulation or measurements.x_data=np.linspace(0,1,int(1e4))y_data=np.sin(24*x_data**2)These can be simply compressed with absolute tolerance of e.g. 0.05tolerance=0.05x_compressed,y_compressed=ls.compress(x_data,y_data,tolerances=tolerance)Or maybe you have some generator-like thing that gives out numbers, e.g. some simulation step.
For sake of example there the mock generatorx0,y0=x_data[0],y_data[0]generator=zip(x_data[1:],y_data[1:])Here you use the context managerStream.
It needs to be initialise with first values of the series, here I am just going to use the first from the mock x and y data.Compressed values can then be accessed from therecordwithls.Stream(x0,y0,tolerances=tolerance)asrecord:forx_value,y_valueingenerator:record(x_value,y_value)x_compressed,y_compressed=record.x,record.yThese can then be decompressed into an spline interpolation function.function=ls.decompress(x_compressed,y_compressed)y_decompressed=function(y_data)With this function e.g. resuduals can be computed and checked against the toleranceresiduals=y_decompressed-y_datamaximum_error=np.amax(np.abs(residuals))print(f'Maximum error should be ~={tolerance}:{maximum_error:.5f}')and when plotting the output withmatplotlibyou should see itfrommatplotlibimportpyplotaspltfig,axs=plt.subplots(2,1,sharex=True)# Data and compressedaxs[0].plot(x_data,y_data,label='Original')axs[0].plot(x_compressed,y_compressed,'-o',label='Compressed')axs[0].legend()# Residuals to toleranceresiduals=y_decompressed-y_dataaxs[1].plot(x_data,y_decompressed-y_data,label='Residuals')axs[1].axhline(tolerance,label='Total tolerance',color='red')axs[1].axhline(-tolerance,color='red')axs[1].legend()fig.tight_layout()plt.show()User GuideVERY MUCH WORK IN PROGRESSCompressionParametersTolerancesKeywordtolerancesTolerances
Absolute Tolerance, Relative Tolerance and Falloff to smooth between them.Absolute tolerance -only is the defaulttolerances, Falloff determines how much the absolute error is
reduced as y value grows.
If 3 values: (relative, absolute, falloff)
If 1 values: (relative, absolute, 0)
If 1 value:  (0, absolute, 0)Allowed deviation is calculated with following function$$
deviation = Relative \cdot |Y_{data}| + \frac{Absolute}{Falloff \cdot |Y_{data}| + 1}
$$$$
D_{|Y|}^1 deviation = Relative - \frac{Absolute \cdot Falloff}{(Falloff \cdot |Y_{data}| + 1)^2}
$$To have constrain that$$
D_{|Y|}^1 deviation(Y = 0) \geq 0
$$
Means
$$
Relative \geq Absolute \cdot Falloff
$$Smooth falloff is at
$$
Falloff = \frac{Relative}{Absolute}
$$If you go over the smooth falloff limit, you make tolerance fucntion non-monotonic, so it firstdecreasesas the absolute y value increases and then starts to increase.RecommendederrorfunctionYou have some data from system of equations
For this example, let's make 100 000 datapoints along some functioninput_x=np.linspace(0,1,int(1e4))input_y=np.sin(24*input_x**2)Example of the data, compression output, and residualsOr maybe you have some generator-like thing that gives out numbers.
E.g. some simulation step
Here you use the context manager "Stream"
Initialise with first values, here I am just going to use the firstErrorfunctionsThe default,MaxAbs.
Maximum of the absolute residual$$
\max(|residual| - tolerance)
$$$$
\max(mean(|residuals|))
$$$$
\max(\max(mean(|residuals| - tolerance)), (|residuals| - tolerance)[-1])
$$Here the Residuals is actually $residuals^2$$$
\max(mean(residuals^2 - tolerance))
$$$$
\max(max(mean(residuals^2 - tolerance)), (residuals^2 - tolerance)[-1])
$$StreamContext manager and a class.Data is fed one point at the time.Context manager is used to ensure proper finishing of the compression process.withls.Stream(example_x0,example_y0,tol=1e-3)asrecord:forexample_x_value,example_y_valueingenerator:record(example_x_value,example_y_value)Using record.x or record.y in the with statement block results in
attribute error, as those attributes are generated only when
the record is closed.If you want to access the data fed to the record, you can usex_compressed,y_compressed=record.xc,record.ycto access the already compressed data andx_buffered,y_buffered=record.xb,record.ybto access the buffered data waiting more values or closing of
the record to be compressed.output_x,output_y=record.x,record.yprint(record.state)print(record)A side mote: In English language the word 'record' can be either
verb or noun and since it performs this double role of both taking
in data and being storage of the data, it is a fitting name for the objectDecompressionDecompression is done in two main steps with interpolation.
First an interpolation function is created
Then that is called.This two-step approach allows more flexible use of the data.Combining compression methodsThis compression method can be combined with lossless compressiom to achieve even higher compression ratios.
The lossless compression should be done only after the lossy compression this package provides.MetaVersion NumberingVersion code is composed of three numbers:
Major, Minor, MicroExperimental, alpha or beta versions are indicated by a 0 as one of those three.First public release starts with Major Version.
Incrementation of Major Version indicates backwards compatibility breaking change in API or fuctionality.Minor Version indicates designWhile the Minor Version is 0, the package is in alpha stage. That means features and APILater incrementation of the Minor Version signifies upgrades to the features and interfaces.
In general changes here mean changes in the design and specification, but not such that it breaks backwards compatibility
I.e. code that works withdocumentedfeatures of X.n.x will work with X.n+1.yMicro Version indicate implementation.
These are bugfixes, typo corrections, documentation clarifications.
In Micro Version incementation theintention, i.e. intended documented specification of the Minor Version is not changed,
only the implementation.Changelog1.0.12 2022-07-16Changed README to Markdown-onlyUpdated documentationSome refactoringFixed type hints to pass MyPy type checking1.0.11 2022-07-16Debug plotting improvementsAdded undocumented API for other fitting functionsMore testsProfiling and benchmarking from tests1.0.10 2022-05-08Cleaned documentation1.0.9 2022-04-03Block and stream compression are much more uniformRestructuringTestsProfiling1.0.8 2022-03-20Step-by-step style ploting of the compression.1.0.7 2021-12-071.0.6 2021-12-021.0.5 2021-12-021.0.4 2021-12-011.0.3 2021-11-30First release on PyPI.