ADEV, Median, EMA, NPLC, Sample rate, averaging & the integration Time Constant - Page 1
Forum section: electronics / general

author: iMo
author_rank: Super Contributor
timestamp: November 12, 2023, 01:31:16 pm
content: I've been messing for some time with post-processing of the data coming off my DMM. I want to understand better the relationships between the ADEV's sigma(tau), Median_tau, EMA_tau, NPLC_tau, Sampling rate_tau and the integration Time Constant..Imagine the following scenario - I've been measuring with 100NPLC a voltage with a DMM. That is 4.2 secs sampling period in my case (for example 34401A, AZ on).Now, the ADEV/MADEV shows the ideal integration time tau (from its sigma(tau) graph) should be aprox 40 seconds (the tau where the minimum sigma(tau) is located). I want to achieve that number at the end, for example.With set 100NPLC I do sample the voltage, sampling period is 4.2secs, I do running median for getting rid of outliers and perhaps it does a smoothing/filtering (running median with N samples, for example 11) and I can do running EMA as a low pass as well (running EMA with say alpha=0.1, for example).I can calculate the tau of the EMA based on the alpha and N, or alpha based on tau and N (that tau is the time constant of an EMA filter).No idea how the Median affects the time constant, as it is similar to averaging, thus perhaps there is a tau as well.Now how to get all that stuff together such I fit those 40 secs? Any hints with a simple math are welcomed..

author: DH7DN
author_rank: Regular Contributor
timestamp: November 13, 2023, 10:16:49 am
content: iMo,could you please take some time and clarify the used abbreviations for the sake of the readability of your subject? I had to read it multiple times just to get the idea what's the kernel of your topic.I can imagine that not every reader is familiar with abbreviations such as ADEV, MADEV, EMA, AZ, ... and your post-processing methodsThanks in advancePS: I am trying to figure out the solution to similar problems, e. g. the relationship between the integration time and some of the parameters you mentioned.

author: iMo
author_rank: Super Contributor
timestamp: November 13, 2023, 10:56:34 am
content: ADEV/MADEV is Allan Deviation (or Modified Allan Deviation) used to characterize "stability" of something within the time (time here is the measurement duration).You sample a voltage (or frequency or any other parameter) and with time you get "sigma(tau)" chart where for a specific "tau" (it is the time you do measure) you get a "sigma" which is basically the "standard deviation" (std) of the measured signal when calculating the std over the time tau.For example you measure 10V at 10 samples per second and for tau=1sec you get sigma 1uV, for tau=100sec you get sigma 0.2uV, for tau=1000secs you get sigma 3.5uV - that means when your measurement is 1 sec long (10 samples) then your std of that measurement will be 1uV, for 100secs long (1000 samples) your std will be 0.2uV, for 1000secs long (10000 samples) your std will be 3.5uV.While looking at a graph of the ADEV/MADEV you will see a typical curve, its sigma(tau) minimum (if any) shows the measurement time "tau" where you get the most precise measurement. Outside that minimum the std of your measurement will always be worse because of various fluctuations/noise/drifts in your measurement system (DMM, counter, etc) and/or in the DUT (device under test - your voltage reference, or an OCXO for example, etc.).Now when you start to do some raw data post processing, every step you do with the raw data introduces a tau (a time constant specific for that operation) and with a series of the data manipulations you get a "final time constant", which IMHO, should be as long as it is the tau for the sigma minimum on the ADEV graph with raw data. That assumption needs to be elaborated, however..Except various taus introduced by the postprocessing the sampling rate plays a role, of course (ie. the NPLC number).Below for example the MADEV of a measurement with 10NPLC - raw data's MADEV (blue) and with applied rolling Median (over 5/11/21/31 last samples) applied. The "frequency" set to 1Hz is meaningless here as we do not measure frequency but voltage.My current understanding is the math operations - like median, EMA, average, etc. - with their "taus" resulting in a final "tau" have to be somehow aligned with the MADEV's minimal sigma(tau) of the raw measurement (for a specific sampling rate).EMA - exponential moving average - a simple low pass filter.rolling/running Median - "similar" to an Average, but it also filters out "outliers" and it does not take them into the account, better for our voltage measurements than the simple averaging, imho, I like it..PS: .. and the "integration time constant" - that is the time you "integrate" your measurement data, let say it is the "tau" (a simplification, perhaps).

author: ch_scr
author_rank: Frequent Contributor
timestamp: November 13, 2023, 02:41:46 pm
content: When looking at Allan Deviation, I foundthis explanationquite helpful.

author: iMo
author_rank: Super Contributor
timestamp: November 13, 2023, 03:14:37 pm
content: Yep, applying the info in your link above - below a real measurement.A - is the single point measurement (a lot of ADC noise)B - is the minima of sigma, when averaging for those tau=15seconds my measurement will be with smallest variance (min noisy)C - the longer I will average my measurement the worse sigma (more fluctuations/drifts).Thus averaging over aprox 37 measurements (at 10NPLC=0.402sec period with the 34401A) will do those 15secs.There is stil the question whether 10NPLC is 0.402sec sampling period with AZ on..The additional smoothing/averaging of the raw data with the rolling Median filter does following - even a single measurement will be less noisy, ie. the "A" will be around std=0.5uV (instead of the 1.5uV). In practice it means the last digits of my 7 digits DMM will be not jumping all over the place

author: Andreas
author_rank: Super Contributor
timestamp: November 13, 2023, 06:51:42 pm
content: Hmm,interesting graphs.Although I think that Allan deviation "lives" from time equidistant data.So if you throw away certain values that may explain why allan deviation looks better for low sample numbers.The question is: are those values really more at the mean value or are they only "more calm".In the rolling median filter what exactly do you really.From the median5: do you throw away 4 samples and hold the middle one or do you throw away 2 out of 5 averaging the remaining 3 samples?How does the distribution of the raw values look like?Is it gaussian noise? (or are there extreme outliers?)with best regardsAndreas

author: iMo
author_rank: Super Contributor
timestamp: November 13, 2023, 07:06:42 pm
content: The ADEV above comes from the equidistant sampling, none samples are being thrown out.All math I do is "rolling", or "moving", or "running", or "sliding", thus all samples are always considered and stored together with results. Thus every new sample generates a dozen various results which are recorded.The rolling Median (and similarly the rolling EMA, rolling Average, rolling STD, etc) works such that with every new sample coming in you do:1. insert new sample into a rolling record/array/buffer (say 5 items long), always consisting of last 5 samples (new incoming in, oldest out)2. after the insertion you sort the values in the record/array/buffer ascending or descending, it does not matter3. you take the 3rd one (the mid one in case of N=odd) as the result4. goto 1So the data stream in the entire process is continual, always with 1 sample period, no data is lost or thrown out.That median5 result stream from 3. is fed into the TimeLab then (for example)..The TimeLab can process X data streams in parallel which are read from an opened .csv file. The file also receives new records (from TeraTerm for example).My MCU reads the new voltage from 34401A, does all that processing above, and writes the results (via TT) into a .csv file, each new sample therefore generates a new csv line. TimeLab reads the opened live .csv file "online" and draws all the graphs in parallel as the new records (lines) come into the csv file.PS: below the TimeLab with opened .csv file, in top right you may see the records (each record starting with +16.. - the unix time) coming in, and you may select which column is the data for an ADEV graph. You may open several streams, each stream with different data (the "numeric field" x) and you will get several graphs which slowly grow with time..Edit: fixed typos

author: Okertime
author_rank: Contributor
timestamp: November 13, 2023, 07:55:24 pm
content: Everybody should know what Tom van Baak, the head oft the time nuts comunity, has written about Allan deviation and averaging.http://www.leapsecond.com/pages/adev-avg/Conclusion:Never ever apply running average or boxcar average data to compute Allan deviation.

author: iMo
author_rank: Super Contributor
timestamp: November 13, 2023, 08:03:17 pm
content: Quote from: Okertime on November 13, 2023, 07:55:24 pmThat is a good point.. Therefore this thread..Let us study why..OK, I think we do here something different - the goal here is not to study the stability but to artificially increase the stability such we see better, smoothed/averaged data. Thus the opposite as the Tom's goal above.Only the blue ADEV graph above represents the real situation (marked as "raw"), the pink "median5" is processed such ADEV looks better - that is the goal of this exercise - to study how to make the ADEV graph looking betterADEV tool here is a tool for seeing differences after the applying some math..The first step is to find the tau for the minimal sigma for the raw data (make ADEV with raw data), the next step is to find a process to make the ADEV graph "flat", at the level of that minimal sigma(tau), or better(??)..PS: everybody here does averaging/smoothing/integrating, every DMM does it, every SD ADC chip does it..

author: iMo
author_rank: Super Contributor
timestamp: November 14, 2023, 08:02:56 am
content: Back to the median filter - it does not make any math operation over the data values (in case the N is odd it just picks up the middle item in the buffer). Its implementation needs more resources, as you need to maintain a buffer, and do a sort algo with the data in the buffer upon each incoming sample.Interestingly I cannot find math for the median's time constant, or BW, if any. While simulating it with a 0 to 1 step (for example) there is just a propagation delay and the output copies the input.PS: what AI says:Quote

author: alm
author_rank: Super Contributor
timestamp: November 14, 2023, 09:43:21 am
content: Quote from: iMo on November 13, 2023, 08:03:17 pmTrue, but not everyone claims to derive a valid Allan deviation from it. Also, these are not running averages, where the same sample contributes to multiple output values.

author: Andy Chee
author_rank: Super Contributor
timestamp: November 14, 2023, 09:49:19 am
content: Quote from: iMo on November 12, 2023, 01:31:16 pmIs there any particular data you are measuring? The nature of the thing being measured may factor into your measurements.

author: iMo
author_rank: Super Contributor
timestamp: November 14, 2023, 10:14:38 am
content: Quote from: alm on November 14, 2023, 09:43:21 amI do not claim to derive a valid ADEV out of it either.. ADEV made of manipulated data is not the valid ADEV, as has been written above.What is the statistical difference between a "running average" calculated upon noisy data ina) a box with N unique samplesb) a box always containing the "last N samples" ?Quote from: Andy Chee on November 14, 2023, 09:49:19 amNo special data in mind, mostly static voltages with noise on them with some drifts, EMI, etc..

author: alm
author_rank: Super Contributor
timestamp: November 14, 2023, 10:50:45 am
content: Quote from: iMo on November 14, 2023, 10:14:38 amI don't claim to be an expert here, I was just going off the remark that using a running average messes with Allan deviation. To me a running average is a sliding window, not taking N unique samples, calculating an average, and then going on to the next N unique samples. I guess with the latter method you are just downsampling, with the obvious reduction in bandwidth. While the moving average gives the false impression that it preserves bandwidth but smears out any brief pulse over N samples. You see this in the link Oktertime provided: with downsampling limits the data to higher values of tau, but a sliding window average creates false values for low values of tau.

author: iMo
author_rank: Super Contributor
timestamp: November 14, 2023, 11:50:04 am
content: Yes, a running average (any average) does filtering (a low pass) and affects the ADEV graph, sure.Btw., a classic averaging of any kind is not a good method for our purposes, imho, as it includes "outliers" into the math, we usually get the outliers into our data..I was using it in past, but the median looks way better filter -a) it does low pass as well, but how to characterize it? - that is not known to me yet - here I've been looking for some related math (and sure it does LP nicely when looking at the data)b) it removes outliers, great for suppression of random spikes, emi, etc. - with N=3 (a pretty low LP effect) you already suppress a single upset samplec) it does not change the data values (with N=odd).

author: alm
author_rank: Super Contributor
timestamp: November 14, 2023, 01:06:05 pm
content: Quote from: iMo on November 14, 2023, 11:50:04 amBut the point of the article is that a running average does it worse than some other forms of averages. It produces nonsense data in the ADEV graph for lower values of tau.Quote from: iMo on November 14, 2023, 11:50:04 amOutliers suggest you have a good argument for rejecting them (say you a few times per hour a train drives by and affects your data), and that they represent a clearly different population if you make a histogram that can be cleanly distinguished. Otherwise in my opinion you are just cherry picking the samples you like best and hence are distorting the data. Making a type A uncertainty estimate based on this sounds highly suspect to me.

author: iMo
author_rank: Super Contributor
timestamp: November 15, 2023, 08:58:28 am
content: ADEV could be used to show impact of some processes applied. Thus to compare scenarios. It is just a tool you have to use with care, sure.We may argue that even the best timenut is actually using the ADEV wrongly, and the ADEV results are affected, because his lovely modern counter collects timestamps of the edges, pushes the timestamps into megabytes large fifos, and does tons of math over the data including interpolation, such he gets 21 digits at the end. With such results they show ADEV of their gadgets then.When a voltnut says his 3458A's ADEV is such and such, the question is whether the ADEV is the real one, as his meter does averaging many times in the process as well (multislope means averaging, NPLCxxx is averaging, etc etc).So, absolutely everybody does averaging and other math in some way today.My point here is to understand how to align all those processes best. You have many processes involved in a measurement, each process with its own parameters, especially the time constants of a the processes.When a meter has an "ADEV_1" (say a reference for comparison, not the real one as the real one does not exist) and I want to apply an additional math over the data (for some purposes) then I should align the additional process(es) with the ADEV_1, that is my current understanding.

author: alm
author_rank: Super Contributor
timestamp: November 15, 2023, 09:28:08 am
content: Quote from: iMo on November 15, 2023, 08:58:28 amRead the article Okertime posted again. Averaging that results in downsampling (like integration that a DMM does) just limits the tau range, but still gives truthfull results for the range of taus that can be calculated from that data. So there is no problem with this kind of averaging and ADEV.My main concern isn't ADEV: I see ADEV just as a tool to decide on the best sampling setup. But that if you filter your data in some non-standard way in which you cherry-pick data, and then report that the value is 7.1234000 V +/- 0.02 uV, that this value may be nonsense.

author: iMo
author_rank: Super Contributor
timestamp: November 15, 2023, 10:47:27 am
content: I think median does not make cherry-picking, it just filters out outliers during the "averaging process".That means if you got a sample which is off the "typical distribution", or an "expected distribution", that sample would not be "averaged" within the process (here, of course, you may ask what is the "typical or expected" distribution), thus the result of the averaging would be closer to the "reality" (again, you may ask what is the "reality"..).Occasional EMI or short fluctuations will be suppressed, but the trend close to the "statistical reality" is still maintained. How many occasional outliers it suppresses depends on number (N) of samples included into the median's calculation (especially in the rolling/moving one).In the reality median suppresses the 1/f-like fluctuations above the taus= X_times_sampling_period (nicely visible on the ADEV below with median_N applied), but it maintains the long term drifts trends, imho.At some time tau the median(N) starts to follow the raw data's ADEV (see below the taus where it crosses raw_ADEV for N=5,11,21,31), that tau(median_N) is something I want to calculate (but how?).

author: alm
author_rank: Super Contributor
timestamp: November 15, 2023, 11:44:00 am
content: Quote from: iMo on November 15, 2023, 10:47:27 amCan you define what causes these outliers are that are being filtered out and explain why they should not be considered part of the measurement and not be considered for an uncertainty estimation?Quote from: iMo on November 15, 2023, 10:47:27 amThat looks a lot like the graph under "Effect of Data Averaging (2)" inhttp://www.leapsecond.com/pages/adev-avg/. I wonder how this different this is from using a rolling window mean instead of median.