Package: py-peak-splitting
py_peak_splittingDescriptionThis package contains an implementation for the random telegraph noise (RTN) removal tools as outlined in:Multi-level RTN removal tools for dynamic FBG strain measurements corrupted by peak splitting artefactsThe aim of this project is to find a pragmatic but effective way to remove jumps which offset the signal over a certain
amount of time, by a  certain amplitude. The article presents two methods which both follow the same principle:The problem of denoising a signal containing jumps is translated into detecting and  replacing outliers in the corresponding difference signal; once the outlier detection and replacement has been done, the cumulative sum of the de-noised difference signal is computed to arrive back at a de-noised version of the original signal.A typical case of measurement data containing jumps can be seen in Figure 1, which shows a corrupted signal before
(black) and after noise removal (blue).Figure 1. Left: Raw signal before (black) and after removing RTN/peak-splitting artefacts (blue). Right: 
Zoom-in on smaller data subset, as well as corresponding difference samples with annotations for nominal samples 
(blue), outliers (red) and outlier replacements (green). Note that mean value of the de-noised signal has not been set 
to value which represents a realistic mean value of the raw signal.For more detailed information on the two algorithms contained in this package, it is referred the accompanying
publication, which can be readhereBackground and motivationStrain  measurements  using  Fiber  Bragg  (FBG)  optical  sensors  are becoming more and more popular. However, in some
cases these measurements can become corrupted by sudden jumps in the signal, which offset the signal over the course
of  e.g. several seconds. These jumps are caused by a defect in the FBG itself which is referred to as peak-splitting.
The effects of peak-splitting artefacts on FBG strain measurements bear similarities with an additive multi-level
telegraph noise process, in which the amplitudes and occurrences of the jumps are related to fiber deformation states.Jumps in measurement data, as shown in Figure 2, can severely limit the quality or usability of data. Figure 2 shows two
raw signals with different levels of jump amplitudes relative to nominal difference between subsequent samples; the
blue annotations indicate jumps which can be clearly separated from nominal samples, whereas the red annotations
illustrate the case in which jump levels are not as straight forward to detect. The considered noise type cannot be
removed by classical linear filters (e.g. a n'th order Butterworth filter). The presence of jumps in data motivates
the need for a pre-processing tool which allows to effectively remove RTN noise.Figure 2. Multi level telegraph noise / peak splitting artefacts. Left: two raw signals containing artefacts of 
different severity. Upper right:  large jump amplitudes in comparison with differences between subsequent signal 
samples. Lower right: jump amplitudes in same order of magnitude as difference samples.The principal idea presented in the accompanying publication is to translate the de-noising of the raw signal containing
jumps, into the problem of de-noising a sample wise difference signal containing outliers. The latter is shown Figure
3,  which also shows the corresponding histograms hinting at the distribution of the nominal and outlying difference
samples. The blue and red annotations clearly indicate the outlying difference samples corresponding to relative large and
small jumps respectively.Figure 3. Sample wise difference signals corresponding to signals presented in Fig. 2. The annotations high 
light outlying difference samples which are attributed to RTN / peak splitting artefacts.In order to de-noise the difference signal, an outlier detection scheme will be used to label difference samples
corresponding to a jump in the data as an outlier.  After the outlier detection step, all outliers will be replaced by
estimates. Once the de-noising process is completed, the cumulative sum is used to obtain a realization of the de-noised
raw signal. If needed, the reconstructed signal can be corrected for a drift as well as a mean value, by maximizing
the overlap between the raw and reconstructed signals.As mentioned, the accompanying article presents two methods which follow the outlined principle. Although sharing the
underlying principle, they are different in the way they try to discriminate between nominal samples and outliers, as
well  as in sample subset selection for the replacement of individual outliers. The first method is based on using a threshold
filter for outlier detection; the high level working principle of the method is described as follows:Compute the sample wise difference signalDerive an outlier threshold level from the histogram of difference samplesReplace outliers in difference signal using a linear regression on an outlier specific buffer (which does not
contain outliers).Take the cumulative sumIf necessary, apply drift and mean correctionAs an alternative to defining a single threshold based on the full signal, the second methods is aimed at defining
threshold levels on shorter segments of the signal. The outlier detection step is based on a Hampel filter. The
high-level working principle is outlined as follows:Compute the sample wise difference signalSegment the data into short segments and compute segment wise threshold levels based on a scale estimate for the
segment's standard deviation.Perform outlier detection on individual segmentsReplace outliers in segments using a linear regression (after removing outliers from segment).Take the cumulative sumIf necessary, apply drift and mean correctionBasic use of both methods is outlined in section 1.4. A more elaborate demonstration can be found in the jupyter
notebooks contained in '''./py_peak_splitting/demo/ ''' or using the Binder which can be foundhere.Installation requirementspy_peak_splitting is written for Python 3.6+. Downloading Anaconda3 is recommended for users not familiar with Python
development. The numpy, scipy and matplotlib package are required to use all functionalities of the threshold filter
based reconstruction method, which includes an option to visualise threshold levels and overlay a kernel density estimate.
The Hampel based reconstruction method only requires the numpy pacakage.The py_peak_splitting package can be installed using one of the methods listed below:Install using Pip (Recommended)Open a terminal or command line interface (or an anaconda prompt, if you're using anaconda)run:pip install py_peak_splittingThe download and install should startManual installDownload or clone the package to a desired location on your machineExtract the zip file at that particular locationOpen a terminal or command line interface (or an anaconda prompt, if you are using anaconda)Set the current directory of the terminal/command line interface to the package's root:
PATH_TO_PACKAGE_LOCATION/py_peak_splittingrun:python install .setup.pyThe installation should startBasic examplesThis section contains the most basic usage examples. More elaborate examples are presented in a Binder which can be
foundhere. For this section a dummy variable,a, holding a 1-dimensional numpy array with corrupted
measurements will be used:type(a)>>numpy.ndarraya.shape>>(60000,)Basic example Hampel filter based reconstruction methods:frompy_peak_splitting.hampel_filterimportHampelasHah=Ha(a,th='sig',th_val=3,re='plf',nperseg=50)rec=h.reconstructionclu=h.clusterslab=h.labelsBasic example Threshold filter based reconstruction methods:frompy_peak_splitting.th_filterimportThFilterasThth=Th(arr=a)thv=th.estimate_th(aggressive=True,th=500,plot=False)rec,clu,lab=th.recon_th(th=thv,method='polyfit',nbuffer=25)rec=rec.flatten()Licensepy_peak_splitting is licensed under aCreative Commons
Attribution-NonCommercial-ShareAlike 4.0 International License. Please check the license terms below and in the license file.DisclaimerTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.