Signal processing - getting exact frequency from short ADC sample - Page 1
Forum section: electronics / projects

author: daqq
author_rank: Super Contributor
timestamp: December 13, 2019, 09:48:55 am
content: Hi guys,I'm playing around with signals, I wanted to ask whether it's possible to measure a small frequency shift on a short pulse of a signal.Assuming that I have data sampled at 100ksps @ 12bit, ignoring timing jitter, the sample size is 2ms (200 samples), is there any method that would enable me to determine the shift of a frequency of sine sampled using this with a large, 0.1Hz resolution? Basically doppler stuff, with the base signal being a pulsed sine at 10kHz. The expected shift is pretty small, 1 to 2 Hz max. I've tried playing with the raw data, using my naive applications of some methods (fft, as well as brute force correlation with a bunch of sines at different phases and different frequencies) it seems the best I can do is a difference of 3Hz-ish so far.My question is, is it even possible? If so, could you tell me the name of the methods I should look at?Thanks,David

author: EEEnthusiast
author_rank: Frequent Contributor
timestamp: December 13, 2019, 10:11:22 am
content: DavidFor sampled signals, the resolution frequency is given by the formula Fres = Fs/N where Fs is the sampling frequency and N is the number of samples. You can calculate this directly as per your signals.essentially for a 1Msps ADC, you need to capture 1Million samples to get 1Hz resolution.

author: iMo
author_rank: Super Contributor
timestamp: December 13, 2019, 10:25:03 am
content: Sure you can, with a reciprocal counter (with TIC for higher resolution).

author: daqq
author_rank: Super Contributor
timestamp: December 13, 2019, 10:39:45 am
content: QuoteThanks, unfortunately I'm stuck with the data sampled by the ADC. I know that it can be done using some other methods (high resolution Time to Digital converters and such or even simple timers clocked fast).QuoteAre you sure about this? Assuming 100Hz, sampled at 1Msps, with 1M samples, the signal will look pretty differently if it's shifted by 0.1Hz. I have a gut feeling that using some kind of sine fitting you should be able to get the 0.1Hz out of there.

author: tszaboo
author_rank: Super Contributor
timestamp: December 13, 2019, 11:25:45 am
content: What you have is a window function. Window function makes samples outside the window 0. You have a rectangular window, which is not very useful. Probably a Hamming window would be much nicer, where the frequency in question would "pop out" nicer. Check this:https://en.wikipedia.org/wiki/Window_functionYour frequency resolution is the bin size. You need a certain amount of data to have a bin. It 200/2=100 bins for you, well not really, Lets say it is 64, or 128. Your sample rate it 100KSPS, so each bin is 781 Hz or 1.562KHz (Fs/N). Thats not very useful. Clearly, you need to decrease the sample rate or increase the sample size. For both, you need the signal to be stable for longer. So FFT is not how you do it.

author: Berni
author_rank: Super Contributor
timestamp: December 13, 2019, 11:41:33 am
content: Quote from: daqq on December 13, 2019, 10:39:45 amYes you can, but the result will be fuzzy.You can fiddle with the data in various ways in order to get FFT to compute to finer resolution or try correlating against near by sine waves. But say your signal is 99.5Hz you will also get a strong signal at 99.1 99.2 99.3 99.4 99.6 99.7 99.8 99.9 100. As a result when graphing it you get a big hump around your 99.5 Hz (You have seen this hump in FFT before im sure). This is because mathematically that 1 second slice of time actually does contain all these frequencies. You can still find the 99.5 because it will be in the middle of that hump. Possible to do if what you have is a nice clean signal with no noise. But if your sine wave has some variation or jitter, or there is a lot of noise mixed in then the peak of the hump could move around a lot, making all that extra 0.1 digit of resolution useless since its just randomly jumping around.Since you are doing doppler processing its probably best to first mix the signal with the carrier. This gives you a sine wave where phase changes with your target range. If you repeat the same with a 90 degree offset you get a quadrature signal where you can instantaneously determine the phase angle. How fast this phase angle moves is your doppler speed.

author: Kleinstein
author_rank: Super Contributor
timestamp: December 13, 2019, 12:03:04 pm
content: The data set is quite short. So one could do a "brute force" least squares fit of a sine function. Depending on the algorithm used this can need quite some computations, but it is possible. The frequency resolution can be quite high, limited by the noise. A resolution in the 0.1 Hz range sounds plausible, if the signal is a real sine. Additional amplitude modulation (e.g. decay in amplitude could complicate things).Another possibility could be using a Hilbert-transformation. However the data set is rather short, with only some 20 periods. It could still work, but may need extra attention. This is essentially calculating a local phase relatively to a virtual first estimate frequency signal over bunches of some 5 periods each and than look at the time dependence of the phase that gives the difference to the first frequency estimate.This is faster than the full fit, but could still give good results, especially with longer data sets.The Method Berni just describes is about this from a different perspective.I have used that method to measure frequencies form data sets like 30000 point's over 1 s to get a frequency resolution in the 1 µHz range for a 1 kHz signal.One can get more frequency resolution from FTT too: just extend the data with leading and trailing zeros to some 4096 points or even more. It is still not easy to look at the FFT result, as more than 1 bin will have amplitude and the real frequency can be in between.

author: Marco
author_rank: Super Contributor
timestamp: December 13, 2019, 12:23:59 pm
content: Quote from: daqq on December 13, 2019, 09:48:55 amBecause you have a constrained model of the signal I'm not sure how the math works out on what frequency difference can be theoretically detected ... but I can say that you shouldn't do what you just said.Always correlate with sines and cosines and take the RMS, that way you can ignore phase. So do that with three frequencies, with the outer frequencies being beyond the min/max doppler shift you expect, then dopeak interpolationand see what you get.I suspect you shouldn't multiply with a window.

author: EEEnthusiast
author_rank: Frequent Contributor
timestamp: December 13, 2019, 12:52:54 pm
content: I suggest to Matlab this and find out. I really doubt if you can reliably resolve the frequency below Fres, with a limited sample set. The accuracy is degraded severely.

author: SiliconWizard
author_rank: Super Contributor
timestamp: December 13, 2019, 03:55:56 pm
content: This sounds like a pretty challenging endeavor.As you probably already figured, you'll get ~20 periods of your signal. A simple calculation shows that a time difference of just one sample period (10µs here) over 20 periods of a 10kHz signal is roughly equivalent to a difference of 50Hz in your input signal. This is just to give an idea of what we're looking at.Assuming the input signal is a perfect sine wave (how distorted/noisy is it? is a 12-bit ADC enough?), you could get better resolution than this, but 0.1Hz?I'd be curious to see a working algorithm giving you this kind of resolution with your above parameters, and taking into account noise/distortion and the 12-bit quantization.

author: mark03
author_rank: Frequent Contributor
timestamp: December 13, 2019, 10:49:01 pm
content: Whether this is possible or not is going to depend on the signal to noise ratio. You haven't told us much about the signal except that it is sinusoidal and pulsed. As was already mentioned, if the SNR were infinite (the "just math" case) you could have infinite frequency resolution. In reality, noise will limit you, even if it is just the quantization noise inherent in your 12-bit ADC.If you can model your signal as a sinusoid in noise, i.e. you know for certain that the sinusoid is always there (it does not turn on and off during your data capture) and there are no other signals present, just noise, then you can use methods like MUSIC or ESPRIT. These achieve much better frequency resolution than an FFT followed by peak interpolation.In any case, your next step is definitely to build some simulations in Matlab or python to find out if this will work, and how to optimize it for your situation.

author: nctnico
author_rank: Super Contributor
timestamp: December 13, 2019, 11:47:00 pm
content: Quote from: daqq on December 13, 2019, 09:48:55 amThe simplest way would be to interpolate (upsample) the signal so you get to the required resolution and then determine the frequency by looking at the zero crossings. Jitter and noise will be your biggest enemy though but if you can average the frequency of several cycles you should be able to filter out the noise. Using a narrow filter to make sure you only get the relevant frequencies of the signal will help to reduce noise. Basically a frequency meter in software.FFT and correlation (which basically comes down to FFT) don't seem like suitable methods to me because in the end the frequency resolution will be limited to the bin size.

author: David Hess
author_rank: Super Contributor
timestamp: December 14, 2019, 02:56:55 am
content: Quote from: daqq on December 13, 2019, 10:39:45 amCentroid timing and transition midpoint timing time to digital converters use curve fitting to make a more precise measurement of a waveform with known characteristics. If your signal is bandwidth limited, then sin(x)/x interpolation will also allow this.

author: coppice
author_rank: Super Contributor
timestamp: December 14, 2019, 03:04:26 am
content: Quote from: daqq on December 13, 2019, 10:39:45 amYou can get the exact frequency of any pure sine wave using 3 perfect samples - i.e. no amplitude noise, no timing noise (i.e. timing jitter), and masses of resolution. The snag comes when you realise how incredibly sensitive to noise the three sample approach really is.

author: David Hess
author_rank: Super Contributor
timestamp: December 14, 2019, 03:09:02 am
content: Quote from: coppice on December 14, 2019, 03:04:26 amSampling noise is not bandwidth limited so it causes aliasing making the solution indeterminate. If you use all of the samples, then the noise is integrated out increasing accuracy.

author: coppice
author_rank: Super Contributor
timestamp: December 14, 2019, 03:16:20 am
content: Quote from: David Hess on December 14, 2019, 03:09:02 amI didn't say the three sample approach was the most practical. However, it is widely used in things like speech analysis. See the Teager Kaiser Energy Operator (TKEO).

author: David Hess
author_rank: Super Contributor
timestamp: December 14, 2019, 03:46:10 am
content: Quote from: coppice on December 14, 2019, 03:16:20 amI just meant to point out the type of algorithm to use. In the analog domain it was common to phase lock a clean reference signal to the signal to be measured and then measure the clean reference signal at only a few points. The continuous phase comparison did the integration. But that is not feasible for burst measurements.

author: nfmax
author_rank: Super Contributor
timestamp: December 14, 2019, 08:55:30 am
content: Another approach might be to use an FFT and Hilbert transform to construct the (complex) analytic signal. Since your pulse is fairly narrow-band, the argument of this is a good approximation to the phase of the original. After unwrapping it, fit a straight line to the phase (possibly with some weighting based on amplitude or SNR, and down weighting the end regions). The slope of the fitted line is the frequency estimate.Choice of windows and weighting functions may be important. You can also try fitting a higher order polynomial, and using its linear term as your estimate.I have used this technique to analyse optical interferograms. The SNR was high, but we needed very high 'frequency' (actually fringe spacing) resolution.

author: Kleinstein
author_rank: Super Contributor
timestamp: December 14, 2019, 10:57:35 am
content: With just 200 samples I would go for the brute force least squares fit. This is rather close to the best possible suppression of noise. So the main weakness to worry about it the computational effort. Writing the "sine" function to fit not at A*sin(wt+phi) but as A1*sin(wt)+A2*cos(wt) the least square fit can be linearized quite well. So it does not need many iterations (maybe 2 or 3) if the starting point is good.It kind of depends on the HW used. An a PC and for tests one could use the fitting function of a plotting program or math-lab. So one could try with real data sets what to expect.Chances are a 100 MHz ARM based µC could to the math as fast as the data are coming in. For a similar task I reached that point with a 60 MHz Pentium CPU and a not very efficient program.

author: hamster_nz
author_rank: Super Contributor
timestamp: December 14, 2019, 07:32:19 pm
content: Made some bogus data, with an amplitude of +/- 75% at 10,000.0 Hz and 10,000.1 Hz. You can draw your own conclusions:With 12-bit resolution:Code:-99, 632, 633-98, 1203, 1203-97, 1656, 1656-96, 1946, 1947-95, 2047, 2046-94, 1946, 1946-93, 1656, 1655-92, 1203, 1202-91, 632, 632-90, 0, 0-89, -632, -633-88, -1203, -1203-87, -1656, -1656-86, -1946, -1946-85, -2047, -2046-84, -1946, -1946-83, -1656, -1655-82, -1203, -1202-81, -632, -632-80, 0, 0-79, 632, 633-78, 1203, 1203-77, 1656, 1656-76, 1946, 1946-75, 2047, 2046-74, 1946, 1946-73, 1656, 1655-72, 1203, 1202-71, 632, 632-70, 0, 0-69, -632, -632-68, -1203, -1203-67, -1656, -1656-66, -1946, -1946-65, -2047, -2046-64, -1946, -1946-63, -1656, -1655-62, -1203, -1202-61, -632, -632-60, 0, 0-59, 632, 632-58, 1203, 1203-57, 1656, 1656-56, 1946, 1946-55, 2047, 2046-54, 1946, 1946-53, 1656, 1655-52, 1203, 1202-51, 632, 632-50, 0, 0-49, -632, -632-48, -1203, -1203-47, -1656, -1656-46, -1946, -1946-45, -2047, -2046-44, -1946, -1946-43, -1656, -1655-42, -1203, -1202-41, -632, -632-40, 0, 0-39, 632, 632-38, 1203, 1203-37, 1656, 1656-36, 1946, 1946-35, 2047, 2046-34, 1946, 1946-33, 1656, 1655-32, 1203, 1203-31, 632, 632-30, 0, 0-29, -632, -632-28, -1203, -1203-27, -1656, -1656-26, -1946, -1946-25, -2047, -2046-24, -1946, -1946-23, -1656, -1655-22, -1203, -1203-21, -632, -632-20, 0, 0-19, 632, 632-18, 1203, 1203-17, 1656, 1656-16, 1946, 1946-15, 2047, 2046-14, 1946, 1946-13, 1656, 1656-12, 1203, 1203-11, 632, 632-10, 0, 0-9, -632, -632-8, -1203, -1203-7, -1656, -1656-6, -1946, -1946-5, -2047, -2046-4, -1946, -1946-3, -1656, -1656-2, -1203, -1203-1, -632, -6320, 0, 01, 632, 6322, 1203, 12033, 1656, 16564, 1946, 19465, 2047, 20466, 1946, 19467, 1656, 16568, 1203, 12039, 632, 63210, 0, 011, -632, -63212, -1203, -120313, -1656, -165614, -1946, -194615, -2047, -204616, -1946, -194617, -1656, -165618, -1203, -120319, -632, -63220, 0, 021, 632, 63222, 1203, 120323, 1656, 165524, 1946, 194625, 2047, 204626, 1946, 194627, 1656, 165628, 1203, 120329, 632, 63230, 0, 031, -632, -63232, -1203, -120333, -1656, -165534, -1946, -194635, -2047, -204636, -1946, -194637, -1656, -165638, -1203, -120339, -632, -63240, 0, 041, 632, 63242, 1203, 120243, 1656, 165544, 1946, 194645, 2047, 204646, 1946, 194647, 1656, 165648, 1203, 120349, 632, 63250, 0, 051, -632, -63252, -1203, -120253, -1656, -165554, -1946, -194655, -2047, -204656, -1946, -194657, -1656, -165658, -1203, -120359, -632, -63260, 0, 061, 632, 63262, 1203, 120263, 1656, 165564, 1946, 194665, 2047, 204666, 1946, 194667, 1656, 165668, 1203, 120369, 632, 63270, 0, 071, -632, -63272, -1203, -120273, -1656, -165574, -1946, -194675, -2047, -204676, -1946, -194677, -1656, -165678, -1203, -120379, -632, -63380, 0, 081, 632, 63282, 1203, 120283, 1656, 165584, 1946, 194685, 2047, 204686, 1946, 194687, 1656, 165688, 1203, 120389, 632, 63390, 0, 091, -632, -63292, -1203, -120293, -1656, -165594, -1946, -194695, -2047, -204696, -1946, -194797, -1656, -165698, -1203, -120399, -632, -633100, 0, 0With 14 bits:Code:-99, 3797, 3800-98, 7222, 7225-97, 9941, 9943-96, 11686, 11687-95, 12288, 12287-94, 11686, 11685-93, 9941, 9939-92, 7222, 7219-91, 3797, 3793-90, 0, -3-89, -3797, -3800-88, -7222, -7225-87, -9941, -9943-86, -11686, -11687-85, -12288, -12287-84, -11686, -11685-83, -9941, -9939-82, -7222, -7220-81, -3797, -3794-80, 0, 3-79, 3797, 3800-78, 7222, 7225-77, 9941, 9942-76, 11686, 11687-75, 12288, 12287-74, 11686, 11685-73, 9941, 9939-72, 7222, 7220-71, 3797, 3794-70, 0, -2-69, -3797, -3799-68, -7222, -7224-67, -9941, -9942-66, -11686, -11687-65, -12288, -12287-64, -11686, -11685-63, -9941, -9939-62, -7222, -7220-61, -3797, -3794-60, 0, 2-59, 3797, 3799-58, 7222, 7224-57, 9941, 9942-56, 11686, 11687-55, 12288, 12287-54, 11686, 11685-53, 9941, 9939-52, 7222, 7221-51, 3797, 3795-50, 0, -1-49, -3797, -3798-48, -7222, -7224-47, -9941, -9942-46, -11686, -11687-45, -12288, -12287-44, -11686, -11686-43, -9941, -9940-42, -7222, -7221-41, -3797, -3795-40, 0, 1-39, 3797, 3798-38, 7222, 7223-37, 9941, 9942-36, 11686, 11687-35, 12288, 12287-34, 11686, 11686-33, 9941, 9940-32, 7222, 7221-31, 3797, 3796-30, 0, -1-29, -3797, -3798-28, -7222, -7223-27, -9941, -9941-26, -11686, -11686-25, -12288, -12287-24, -11686, -11686-23, -9941, -9940-22, -7222, -7222-21, -3797, -3796-20, 0, 0-19, 3797, 3797-18, 7222, 7223-17, 9941, 9941-16, 11686, 11686-15, 12288, 12287-14, 11686, 11686-13, 9941, 9940-12, 7222, 7222-11, 3797, 3796-10, 0, 0-9, -3797, -3797-8, -7222, -7222-7, -9941, -9941-6, -11686, -11686-5, -12288, -12287-4, -11686, -11686-3, -9941, -9941-2, -7222, -7222-1, -3797, -37970, 0, 01, 3797, 37972, 7222, 72223, 9941, 99414, 11686, 116865, 12288, 122876, 11686, 116867, 9941, 99418, 7222, 72229, 3797, 379710, 0, 011, -3797, -379612, -7222, -722213, -9941, -994014, -11686, -1168615, -12288, -1228716, -11686, -1168617, -9941, -994118, -7222, -722319, -3797, -379720, 0, 021, 3797, 379622, 7222, 722223, 9941, 994024, 11686, 1168625, 12288, 1228726, 11686, 1168627, 9941, 994128, 7222, 722329, 3797, 379830, 0, 131, -3797, -379632, -7222, -722133, -9941, -994034, -11686, -1168635, -12288, -1228736, -11686, -1168737, -9941, -994238, -7222, -722339, -3797, -379840, 0, -141, 3797, 379542, 7222, 722143, 9941, 994044, 11686, 1168645, 12288, 1228746, 11686, 1168747, 9941, 994248, 7222, 722449, 3797, 379850, 0, 151, -3797, -379552, -7222, -722153, -9941, -993954, -11686, -1168555, -12288, -1228756, -11686, -1168757, -9941, -994258, -7222, -722459, -3797, -379960, 0, -261, 3797, 379462, 7222, 722063, 9941, 993964, 11686, 1168565, 12288, 1228766, 11686, 1168767, 9941, 994268, 7222, 722469, 3797, 379970, 0, 271, -3797, -379472, -7222, -722073, -9941, -993974, -11686, -1168575, -12288, -1228776, -11686, -1168777, -9941, -994278, -7222, -722579, -3797, -380080, 0, -381, 3797, 379482, 7222, 722083, 9941, 993984, 11686, 1168585, 12288, 1228786, 11686, 1168787, 9941, 994388, 7222, 722589, 3797, 380090, 0, 391, -3797, -379392, -7222, -721993, -9941, -993994, -11686, -1168595, -12288, -1228796, -11686, -1168797, -9941, -994398, -7222, -722599, -3797, -3800100, 0, -3

author: ogden
author_rank: Super Contributor
timestamp: December 14, 2019, 07:34:12 pm
content: In case of pure sine - normalize amplitude and convert it to polar form. Knowingphasorrotation speed (angular velocity) it is easy to calculate frequency (ω = 2πƒ). In such case accuracy depends on SNR of signal and ADC. It is possible to measure frequency even with just two consecutive samples - using very, very accurate ADC.

author: Marco
author_rank: Super Contributor
timestamp: December 14, 2019, 09:15:15 pm
content: You can't accurately know amplitude until you accurately know frequency.

author: Kalvin
author_rank: Super Contributor
timestamp: December 14, 2019, 09:21:22 pm
content: Would it be possible to create a sine wave so that the created sine wave would be the best fit to the captured samples? If that would be possible, we could also get the best estimate to the sampled sine wave frequency.

author: hamster_nz
author_rank: Super Contributor
timestamp: December 14, 2019, 09:49:33 pm
content: Maybe something more practical.Apply a bandpass DSP filter (to get rid of as much out of band noise as you can).Interpolate zero crossing between samples.Period is 2*(LastZeroCrossTime-FirstZeroCrossTime) /(NumberOfZeroCrossings-1)Frequency is 1/period.

author: Kleinstein
author_rank: Super Contributor
timestamp: December 14, 2019, 10:00:57 pm
content: Quote from: Kalvin on December 14, 2019, 09:21:22 pmThis is possible, e.g. with a program like Gnuplot. Here the output for the 12 Bit data given by Hamster_NZ. The Program also calculates an estimated error. Due to the likely exact integer amplitude chosen the error estimate may be a little (some 20-50%) optimistic, as the quantization error may be below typical. This would suggest some 0.5 ppm possible frequency (some 0.005 Hz for the 10 kHz example) resolution from perfect 12 Bit data.Code:gnuplot> fit a*sin(b*x)+c*cos(b*x) 'test12bit.dat' us 1:2 via a,b,c.....Final set of parameters Asymptotic Standard Error======================= ==========================a = 2046.56 +/- 0.0326 (0.001593%)b = 0.314159 +/- 2.755e-007 (8.77e-005%)c = 2.74857e-005 +/- 0.03258 (1.185e+005%)correlation matrix of the fit parameters:a b ca 1.000b 0.036 1.000c -0.001 -0.017 1.000gnuplot> fit a*sin(b*x)+c*cos(b*x) 'test12bit.dat' us 1:3 via a,b,citer chisq delta/lim lambda a b c0 4.1947679263e+01 0.00e+00 1.52e+04 2.046557e+03 3.141593e-01 2.748573e-051 2.0589137226e+01 -1.04e+05 1.52e+03 2.046317e+03 3.141567e-01 2.748573e-052 1.8897496910e+01 -8.95e+03 1.52e+02 2.046187e+03 3.141567e-01 2.748573e-053 1.8897452147e+01 -2.37e-01 1.52e+01 2.046186e+03 3.141567e-01 2.748573e-05iter chisq delta/lim lambda a b cAfter 3 iterations the fit converged.final sum of squares of residuals : 18.8975rel. change during last iteration : -2.36876e-006degrees of freedom (FIT_NDF) : 197rms of residuals (FIT_STDFIT) = sqrt(WSSR/ndf) : 0.309719variance of residuals (reduced chisquare) = WSSR/ndf : 0.0959262Final set of parameters Asymptotic Standard Error======================= ==========================a = 2046.19 +/- 0.03099 (0.001515%)b = 0.314157 +/- 2.62e-007 (8.339e-005%)c = 2.74857e-005 +/- 0.03098 (1.127e+005%)correlation matrix of the fit parameters:a b ca 1.000b 0.036 1.000c -0.001 -0.017 1.000gnuplot>

author: MasterT
author_rank: Frequent Contributor
timestamp: December 14, 2019, 11:51:55 pm
content: Correlation with a sine wave is a Goertzel algorithm. The solution is to synthesize 10-50 sets of coefficients for Goetzel tunned at 10 000 +- N x 0.1 Hz, than run a math with input data array. Last step is to find a maximum. Theoretically, there is no limits to frequency calculation, since steps could be any value, 0.1 or 0.0001 Hz. Everything would depends on SNR ratio. Here is a quick test,Data 12-bits, polluted with noise 16 - 64- 256 LSB. With higher noise level peak is drifted to the right, last value -256 is off for 0.2 HzCode:

author: nctnico
author_rank: Super Contributor
timestamp: December 14, 2019, 11:57:59 pm
content: That doesn't sound like a good approach because you need to do a lot of math and the result is still a guess. Goertzel is basically a single bin FFT. If you need to achieve a high resolution then you need many very narrow bins to try.The easiest and least processing intensive solution by far is to create a frequency meter in software using interpolated data.

author: Marco
author_rank: Super Contributor
timestamp: December 15, 2019, 08:39:30 am
content: Very simple algorithms using the FFT bins near the max bin seem to get so close to the error bound that anything more complex than that seems superfluous.https://www.researchgate.net/publication/3316697_Fast_nearly_ML_estimation_of_the_parameters_of_real_or_complex_single_tones_or_resolved_multiple_tonesFollowing the citations there are autocorrelation methods which are faster.PS. actually if I wanted something faster I'd consider doing single sample delay autocorrelation to get a good estimate, then do DFT for 3 or 5 components and do the DFT based estimation. That way you can skip the full 256 point FFT ... but is a 256 point FFT really going to be a problem?

author: mark03
author_rank: Frequent Contributor
timestamp: December 15, 2019, 05:56:27 pm
content: I'm seeing an awful lot of ad-hoc and intuition-based methods being proposed here, largely based on the FFT. While this may be useful for understanding the problem, I feel like I should repeat the point I made earlier: The best chance for success will probably come from "subspace" methods using the singular value decomposition, specifically the MUSIC and ESPRIT type algorithms. These are able to leverage a priori knowledge about the signal (the number of sinusoids) and can achieve significantly better resolution than FFT-based methods.

author: Marco
author_rank: Super Contributor
timestamp: December 15, 2019, 06:14:05 pm
content: They are ridiculously over complex for a single sinusoid. The 5 bin discrete fourier based method in the paper is within 0.6 dB of optimal ... as I said, something much more complex doesn't make a whole lot of sense.

author: Kalvin
author_rank: Super Contributor
timestamp: December 15, 2019, 06:42:10 pm
content: Quote from: hamster_nz on December 14, 2019, 07:32:19 pmIf you look at the samples from hamster_nz's original post, you should get the idea how difficult this will be with such short data even without any noise involved.

author: ogden
author_rank: Super Contributor
timestamp: December 15, 2019, 09:44:52 pm
content: Quote from: Marco on December 14, 2019, 09:15:15 pm

author: Kleinstein
author_rank: Super Contributor
timestamp: December 15, 2019, 10:00:00 pm
content: For pulsed acoustic data, I would assume there would be quite some idle time after the 2 ms pulse, just to wait for echos to decay. So the overall data rate would not be that high, maybe getting data for 1% of the time. Depending on the µC / CPU used there could be enough time to do the full least squares fit and thus get very close to the best possible (at least in case of additive white noise) result. A general nonlinear least square fit takes quite some time. However specialized to estimating DC offset, amplitude, phase and frequency the computations are not that bad anymore. In the right form (as cos and sine part) only the frequency is a nonlinear parameter so the iterations can be quite effective (exact for the amplitudes and DC offset).

author: Marco
author_rank: Super Contributor
timestamp: December 15, 2019, 10:17:55 pm
content: Quote from: ogden on December 15, 2019, 09:44:52 pmWhat? Do you want to calculate it from variance? AFAICS that will have more noise than an ideal estimate from a discrete Fourier transform at the exact frequency ... and there's not a lot of room for less than ideal estimates.You don't even really need the renormalization, but the phase based approaches using single sample lag autocorrelation need a lot of hacks to make it work well regardless. In fact to get close to the error bound they tend to only work for frequencies near zero. So it's usually a two step process, first do a bad estimate, mix down to near DC, then try again. A lot more code than Macleod's discrete Fourier domain approach (assuming you use someone else's Fourier transform code).

author: mark03
author_rank: Frequent Contributor
timestamp: December 15, 2019, 10:40:58 pm
content: Quote from: Marco on December 15, 2019, 06:14:05 pmWell, this is not my area of expertise, but I am skeptical. Macleod casts the problem as finding "the parameters (frequency, amplitude, and phase)" which would seem to rule out many techniques at the outset. Here we only care about accurate frequency estimation. Also, Macleod's only mention of the subspace methods in the paper is to dismiss them as computationally expensive. Finally, I know that these methods are used in other settings with only one tone, e.g. NMR. In fact, I run a proton-precession magnetometer (not my design) which uses the so-called "filter diagonalization method" for accurately estimating the frequency of the decaying precession signal. I'm quite sure it would not work with a simple FFT + interpolator approach.

author: ogden
author_rank: Super Contributor
timestamp: December 15, 2019, 11:50:37 pm
content: Quote from: Marco on December 15, 2019, 10:17:55 pmI just wanted to point out that you are wrong with your blanket statement. To measure signal amplitude you do not need to "accurately know signal frequency" as you say. You can measure amplitude w/o even knowing frequency. Think white noise for instance. All you want to know is "frequency band", ensure that period of lowest frequency component (of interest) is shorter than time window of measurement of amplitude.BTW method I did propose is widely used and comes from power industry where mains frequency drift shall be detected as soon as possible and accumulating enough periods for other methods (correlation/fft/zerocrossings/e.t.c.) is not an option. BTW upsampling, then reciprocal frequency calculation for full sine periods (zero crossings) is derivative. Put a little thought into it and you will see.

author: Etesla
author_rank: Regular Contributor
timestamp: December 15, 2019, 11:57:24 pm
content: Quote from: hamster_nz on December 14, 2019, 09:49:33 pmThat was my first thought as well assuming the signal your measuring is guaranteed to cross zero x times per cycle

author: coppice
author_rank: Super Contributor
timestamp: December 15, 2019, 11:59:10 pm
content: Quote from: ogden on December 15, 2019, 11:50:37 pmI suspect he meant that you need the frequency of the signal of interest before you can calculate its amplitude, rather than the total signal amplitude with noise, harmonics, etc. However, he's wrong on that too.Calculating mains power frequency is easy. With glitch suppression you can use zero crossings of the voltage waveform. The current waveform can be so distorted that you get extra zero crossings, but if the voltage waveform is that distorted something is badly wrong.

author: ogden
author_rank: Super Contributor
timestamp: December 16, 2019, 12:05:22 am
content: Quote from: coppice on December 15, 2019, 11:59:10 pmActually math is same for all sine waveforms no matter they are mains power or other kind of "signals".

author: coppice
author_rank: Super Contributor
timestamp: December 16, 2019, 12:07:01 am
content: Quote from: ogden on December 16, 2019, 12:05:22 amPerhaps you should have read what I wrote.

author: ogden
author_rank: Super Contributor
timestamp: December 16, 2019, 12:11:09 am
content: Quote from: coppice on December 16, 2019, 12:07:01 amPerhaps you shall explain what you mean by saying so.

author: coppice
author_rank: Super Contributor
timestamp: December 16, 2019, 12:21:26 am
content: Quote from: ogden on December 16, 2019, 12:11:09 amLet me expand on what I said. If you just use zero crossings on the voltage waveform to estimate the mains frequency you will have problems when people use certain equipment, like arc welders. You can apply a glitch suppression algorithm to clean up the fast transitions caused by equipment like that, which can have a huge amplitude, and very fast edges. Then, applying a zero crossing detector is a robust mechanism for measuring the mains frequency. The voltage waveform might have 30% to 40% THD under some conditions, but you just don't see extra zeros crossing in real world conditions. Try that with the current waveform, and you will have trouble. The current waveform is frequently so distorted it does show zero crossings at one of the harmonic frequencies. So, if you specifically need to extract the frequency from the current waveform you will need to use other techniques.

author: Berni
author_rank: Super Contributor
timestamp: December 16, 2019, 06:42:36 am
content: Quote from: coppice on December 16, 2019, 12:21:26 amEasy way to solve that is to run the signal trough a bandpass FIR/IIR filter to remove signals out of the band of interest, leaving you with just the sine wave you want.But still i think OP is looking at the problem from the wrong direction. If you use a quadrature mixer to convert the 100.1KHz signal down to a pair of 100Hz signals things get a lot easier (Its essentially a software defined radio at this point). Not only is this computationally cheap to do, but unlike his method of taking 1 second long 1MSPS recordings of the signal and processing them later, this downconversion method can also operate continuously on a signal, giving you a result on every sample rather than just one sample per second.Another advantage to mixer downconversion is that the sample rates can be quickly reduced. The only thing that runs at the full 1 MSPS is the mixer it self (This involves two multiply operations against a sine table and a sum to advance the table) after that the signal is low bandwith (100Hz) so you can use averaging (or a proper FIR downsampler if you like to be fancy) to drop the samplerate to something like 1KHz for all the rest of the processing, this downsampling step also filters out any out of band noise.At this point there are two 90 degree out of phase 100Hz sinewaves coming out of this. This allows you to convert them to polar form from just looking at one sample. The polar magnitude being the sine wave size and the phase being distance to doppler target. You can then unroll phase between any two samples and get the distance the doppler target has moved between those two points in time, divide that by time to get the average speed within that chosen window of time (Note this window can be placed anywhere along the stream of data)If your radar/sonar is picking up multiple doppler targets then this step is a bit more complicated because there will be multiple sinewaves in this downconverted signal. Here you might use complex input FFT to seperate them out, pick out the targets, then inverse FFT them back into a time domain signal to process same as before. This creates a sort of tracking bandpass filter for each target. This part is computationally heavy but thanks to the prior work this only needs to run at a sample rate of 1KHz so actually does not need a lot of processing power.

author: daqq
author_rank: Super Contributor
timestamp: December 16, 2019, 07:56:14 am
content: Wow. Thanks guys for the amazing replies, I'm still reading through them in detail.I'll try simulating the signal in Octave (open source matlab equivalent) and play around with it some more based on everyones advice.QuoteThat's an interesting approach. I'll try that, though I'm not sure how to apply the filter to the 'edges' of the sample window.QuoteIt is bandwidth limited, yeah.I'll have to study upon the whole interpolation math.QuoteAt the moment there's no real limit on the processing power, it's just in the guessing and estimating stage, whether or not it's possible at all.Berni: Thanks for the mixer idea, I've been thinking about it, but with such a small time frame, the more I downconvert (mix it with a high frequency getting the frequency difference), the less 'full sines' I get? Basically, if my window 2ms (200 samples) and I downconvert from 10kHz to, say, 100Hz, then, during the 2ms I don't even get one full sine.Thanks again everyone, I'll play around with it more during the week.

author: hamster_nz
author_rank: Super Contributor
timestamp: December 16, 2019, 08:02:45 am
content: Not sure if you are aware of the Hilbert Transform?https://en.wikipedia.org/wiki/Hilbert_transformWith it you can take an 'real' signal, and convert it into an I+Q signal.What might be interesting about it is that you don't need to apply it to each point.You can cut the dataset in to two halves (points 0-99 and points 100-199), then convert ONLY points 50 and 150 into I+Q (based on, then look at the phase difference between them - and see how much the phase has changed over the 100 samples. This isn't a bad as it seems, as if you are to run a FIR or IIR filter over your data to remove noise, you will end up truncating your data (for a FIR filter) or have to wait for the IIR filter to settle down.One of the interesting properties (well at least to me) of the Hilbert Transform is that you can incorporate a bandpass filter at the same time, so that might be a win too.Doing this means you don't have to worry about 'zero crossings' or anything like that - you get to pick the points you want to look at the phase of, rather than waiting for zero crossings where you know the phase is one of two values.It also hints (at least to me), that if I had the choice of 2ms of data or 8ms of data at 1/4th the sample rate, I would take the 8ms of data as any phase difference will be 4x as great, and it is far less work to bandpass filter the frequencies of interest.PS. If interested, you could convince me to code up a demo...

author: Berni
author_rank: Super Contributor
timestamp: December 16, 2019, 08:49:21 am
content: Quote from: daqq on December 16, 2019, 07:56:14 amThe point of using downconversion is to turn the real signal into a complex I/Q quadrature signal, the reduction in frequency is just a side effect that can be later used to help reduce the required processing power. It is perfectly possible to go the other way and upconvert against 1GHz to turn 100.1KHz into 1000100.1 KHz if you want more frequency.The advantage of having the signal in I/Q form is that you determine the phase by looking at only 1 sample, just converting that complex I/Q number from rectangular form to polar form (This is just calculating the sides of a right angle triangle, so just needs elementary school level math). Since phase and frequency are related via time means that you only need 2 samples from the ADC to be able to determine the frequency to any precision past 0.0001Hz. In practice however more points are needed to get to sub 1Hz precision due to the ADC not being perfect (number of bits, noise, nonlinearity etc...). There is no need to see single a zero crossing in the signal, since you know the phase in every point means you can determine the frequency pretty accurately by seeing just 1/4 of one sine wave period. This is because a sinewave will move forwards by 90 degrees in phase after 1/4 period, and there is only one unique frequency that will move its phase by 90 degrees in the given time frame of the ADC capture.So if your ADC is decent and input signal is stable you can likely use this "downmixing and polar connversion" to determine the frequency down to 0.1Hz precision from looking at only 5ms of your 1MSPS ADC data.Another advantage is that if you skip averaging after downconversion you keep 1MSPS trough the signal chain. So with only a few dozen multiply and accumulate operations per sample (after some optimizations) you stream data trough the process in a pipelined way to get 1 000 000 frequency readings per second as the output, each accurate to 0.1Hz (Or even more accurate if you use a larger phase integration window).Best of all doing this required no advanced math at all. All you have to do is multiply each sample with a sine function, then do a bit of trigonometry to find the phase angle and you are done. But the process is a bit easier to understand if you have a good idea of how complex numbers work.

author: Kleinstein
author_rank: Super Contributor
timestamp: December 16, 2019, 09:35:34 am
content: The Hilbert transform / down-conversion is a viable option if computational speed is critical. I would still use averaging before calculating the phase. In C code the ATAN2 function needed here is one of the really slow ones. With noisy data one could also get effects of noise leaving the linear range. So it makes sense to do averaging / down sampling before in the complex domain.The least squares fit gives the limit how good it can get (if the noise is white). The fitting result to the 12 Bit simulated data showed that sufficient resolution would be possible. I just realized the uncertainty number was in absolute numbers so the error bar is more like 1-1.5 ppm. So to get 0.1 Hz resolution (10 ppm) the noise could be higher by about a factor of 8. So the data would need to be good to about 9 Bits to reach the aimed for 0.1 Hz resolution.From my experience (though with considerably longer data sets and additional amplitude decay as another parameter) the Hilbert transform is slightly more sensitive to noise and may need another 2-6 dB better signals. How much depends on the exact implementation (e.g. choice of the windows for the transformation). Speed wise the Hilbert transformation is faster by maybe a factor of 10 - with the rather short data set the ATAN2 function may reduce that advantage and have the noise disadvantage a little higher as relatively short windows have to be used.

author: hamster_nz
author_rank: Super Contributor
timestamp: December 16, 2019, 09:52:59 am
content: I couldn't resist - here's using the Hilbert Transform to detect the phase change over 100 samples and extrapolating that to 100,000 samples. The input is a file containing 200 samples.Note that any signal that is a harmonic of the sample rate divided by 100 could be used... so with a sample rate of 100kS/s the signal you are analyzing could be 50kHz, 33.3kHz, 25KHz, 20kHz.. whatever. All that matters is that after 100 samples there is nominally zero phase offset. (oh, and you will need to change the magic fudge factor in the kernel if you are looking for a different frequency).Assumes that the input data has already been bandpass filtered to just around the frequencies of interest.Code:/************************************************************* poc.c : A really bad Proof of concept** Can you extract tiny frequency changes from small datasets* using the Hilbert Transform? Yes, it seems you can.** Expects a single command arg that is a list of 200 samples* Processes it assuming the sample rate is 100kS/s, and* measures the crawl in phase over 100 samples. This is then* used to extrapolate to the change of phase after 100,000 samples.** This isn't supposed to be the most correct code, just a hack* to see if it works.************************************************************/#include <stdio.h>#include <stdlib.h>#include <math.h>double data_r;double data_i;double kernel;static void build_kernel(void) {int i;for(i = -49; i < 50; i++) {if((i&1)==1) {// I pulled the scaling factor out of my datakernel[i+49] = 1.0/i/3345*2047;}}}static void hilbert(double *data, double *result) {double r = 0;int i;// Apply the kernelfor(i = -49; i < 50; i++) {r += data*kernel[i+49];}*result = r;}int main(int argc, char *argv[]) {int i;FILE *f;build_kernel();if(2 != argc) {fprintf(stderr,"Only supply data file name\n");exit(0);}f = fopen(argv,"rb");if(NULL == f) {fprintf(stderr,"Unable to open file\n");exit(0);}for(i = 0; i < 200; i++) {if(1 != fscanf(f,"%lf",data_r+i)) {fprintf(stderr,"Error reading data\n");exit(0);}data_i = 0.0;}fprintf(stderr, "Data read\n");// Just calculate the transform at two placeshilbert(data_r+49, data_i+49);hilbert(data_r+149, data_i+149);double phase49,phase149,change;phase49 = atan2(data_r,data_i);phase149 = atan2(data_r,data_i);change = (phase149-phase49)/100;/* change' needs to be wrapped into +/- PI, but I haven't done it */printf("Angle at sample 49 is %10.6f\n",phase49);printf("Angle at sample 149 is %10.6f\n",phase149);printf("Change in cycles after 100,000 samples %10.6f\n", change*100000/(2*M_PI));return 0;}Here's the output when the data is 0.1 Hz faster:Code:Data readAngle at sample 49 is -0.657761Angle at sample 149 is -0.658373Change in cycles after 100,000 samples -0.097432And 0.1 Hz slowerCode:Data readAngle at sample 49 is -0.657381Angle at sample 149 is -0.656791Change in cycles after 100,000 samples 0.093872Here's the hack I was using to generate test dataCode:#include <stdio.h>#include <math.h>#define SAMPLE_RATE (100000.0)#define FREQUENCY (9990.0)#define SCALE (2047)int main(int argc, char *argv[]) {int i;for(i = 0; i < 200; i++) {double d = sin(i/(SAMPLE_RATE/FREQUENCY)*2*M_PI)*SCALE;printf("%i\n",(int)(d));}return 0;}

author: ogden
author_rank: Super Contributor
timestamp: December 17, 2019, 02:31:21 pm
content: Quote from: Berni on December 16, 2019, 06:42:36 amI am afraid that you drive OP into wrong direction as well. Estimating precise frequency of downmixed 100Hz do not seem to be simpler than just counting zero crossings of "carrier".

author: Marco
author_rank: Super Contributor
timestamp: December 17, 2019, 03:03:04 pm
content: He isn't suggesting just downmixing it, he's suggesting downmixing it by sines in quadrature.I think after low pass filtering, the phase of the quadrature results treated as a complex vector will give you the frequency shift (side effect of the PM/FM equivalence).

author: ogden
author_rank: Super Contributor
timestamp: December 17, 2019, 03:22:00 pm
content: Quote from: Marco on December 17, 2019, 03:03:04 pmDownmixing, then measuring phase do not make any sense - if you can use same (phase) math on carrier directly. To avoid gigantic phase numbers that multiple times exceeds 2*PI you do not need to use downmixing. Just use your head.

author: Marco
author_rank: Super Contributor
timestamp: December 17, 2019, 04:22:53 pm
content: You can, but at least for low SNR signals you can't get near the error bound with a weighted linear predictor. I'm not sure the quadrature mixer method can of course.I'm also not sure how relevant using near optimal algorithms is at high SNR.

author: ogden
author_rank: Super Contributor
timestamp: December 17, 2019, 04:26:08 pm
content: Quote from: Marco on December 17, 2019, 04:22:53 pmIf you signal is noisy - apply filter.

author: Berni
author_rank: Super Contributor
timestamp: December 17, 2019, 05:00:23 pm
content: Quote from: ogden on December 17, 2019, 02:31:21 pmI am not saying it is simpler than counting zero crossings. Im saying that doing it using downconversion offers a lot of advantages like being computationally cheap, resilient to noise, offering a lot of measurements per second, able to work on very short captures of the signal (few ms rather than needing a full second of data), able to continuously pipeline data trough it rather than process it as one by one 1 second long recordings. All of that while providing well under 1 Hz resolution and involving only high school level math. Im sure there plenty more advanced ways to do this that might perform even better, but this one has quite a lot of pros versus cons.Quote from: ogden on December 17, 2019, 03:22:00 pmThe point of down-mixing here is as Marco said is to easily convert the real input signal into a pair of quadrature I/Q signals. Advantage of these quadrature signals being that its possible to compute the phase by looking at just 1 sample. No need to see a zero crossing, pick any random point on the sinewave and just look at that one sample and you will know the exact phase of it by using trigonometry. Its not the only way to get a I/Q signal from it but its a pretty attractive one because choosing a carrier frequency that is close lets you downsample the signal afterwards, as a result reducing your computational requirements while retaining the accuracy and noise resilience. Since phase and frequency are bound together by time means if you know the phase in two spots you also know the frequency. You can chose these two points anywhere along the data, the farther apart you place them the more area you will average between them and the more resolution and noise resilience you have (Tho you do have to phase unwrap the signal since it jumps back around at 360 deg)What method do you propose as being better or simpler than this?

author: Kleinstein
author_rank: Super Contributor
timestamp: December 17, 2019, 05:12:33 pm
content: The down-mixing with a quadrature mixer is mathematical similar to the Hilbert transformation. It can be a relative fast (less computational effort) way and can still be close to the optimum, at least for longer data sets. The reduced data volume is only second to getting the phase information. However there no need to get the phase information for really every point of the original data set. The phase data are smoothed anyway. So one could as well use the reduced number of points without loosing much information.A nice point about the down mixing is that this part can be done while the data are coming in and there is no need to store to whole data set. This is not so relevant here with a short data set, but it can be with 1 second or more at 1MSPS.The down conversion method is also quite simple, especially with a background from EE.The full nonlinear fit gets a better result, but with more effort. The math behind it is still not that complicated. Finding an efficient algorithm can be some extra effort.Filtering can remove out of band noise, but many of the methods discussed here (least square fit, Hilber transformation / down mixing and FFT and interpolation) are not sensitive to out of band noise anyway. So with these method an extra filtering run does not help or change a thing.Even with relatively good data the SNR can be relevant, as noise in the input data translates to noise in the frequency reading. From the Fit data it takes at least some 80 dB SNR to get 1 ppm frequency resolution or 60 dB to get the 0.1 Hz target. There is a point at rather low SNR that some of the methods may fail and could produce biased or totally off values. This tends to happen when the noise is no longer a small distortion in the linear range. So one might miss whole periods or see ambiguities in 2 pi steps in the phase.From the linked paper about FFT interpolation methods it looks like not so good method add a more or less constant factor to the noise, e.g. giving 2 or 3 times the frequency noise of the near optimum ML estimate.

author: ogden
author_rank: Super Contributor
timestamp: December 17, 2019, 05:43:07 pm
content: Quote from: Berni on December 17, 2019, 05:00:23 pmWrong. Other methods requires just single filter. IQ downmixer requires not only two mixers but two decimating filters as well.QuoteCompared to unfiltered signal - yes. As I said - other methods can and shall use filter as well.QuoteMuch less measurements per second than direct phase estimator I mentioned in myfirst post.QuoteNonsense argument. There is no other method which would need/require full second of data.QuoteNonsense again. Phasor angle estimator and even zero crossings counter can be run continuously. As I said - my proposal is the same, just do not need two mixers and two filters, needs just one filter. Dont you see.QuoteLooking at 1 sample (angle difference between two actually) do not achieve meaningful resolution.QuoteI do not see this as an argument. If filtering, counting zero crossings, calculating phase shift for sine waveform appears problem for someone then I doubt that he will succeed implementing IQ downmixer with two decimator filters, then doing essentially the same - estimating phase shift between two or more IQ samples. Complexity while doing math with single sine compared to two (IQ) is not much different. You just propose to add completely superflous step - downmixing.Your method have huge drawback - it's frequency range is kinda limited. Paper that compares many methods mentioned in this thread:https://www.researchgate.net/publication/321255243_Power_System_Frequency_Measurement_for_Frequency_RelayingFollowing are mentioned:Modified zero-crossing methods , , , , DFT with compensation , , , Orthogonal decomposition , , Signal demodulation , Phase locked loop , , Least square optimization , , , Artificial intelligence , , , , , Wavelet transform , , , , Quadratic forms , Prony method , Taylor approximation Numerical analysis

author: Berni
author_rank: Super Contributor
timestamp: December 17, 2019, 07:32:44 pm
content: Quote from: ogden on December 17, 2019, 05:43:07 pmThat paper is an excellent find.It shows these methods being fed sinewaves with various types of noise and distortion applied to them. One of them being my proposal that is here named as "Signal demodulation". On page 13 and on you will find figures 9 to 17 showing graphs of the results, notice how graph D (showing the demodulation method) is always rock solid on the right frequency no matter what kind of noise is injected into the signal, nor does it show any quantization steps.Using a FIR bandpass filter is more computationally expensive than the entire quadrature mixing process. Yes you do need to compute the mixers for every input sample to get all of the noise rejection benifits, but this cost is 2 MAC operations for the I and Q mix, and 2 additions with 2 sine table lookups for generating the mixers local oscillator. Downsampling only costs one bitshift every 1024 samples, turning this into a phase angle does require annoyingly slow division along with another trigonometry table lookup, but this is done at the slow rate so it all costs the equivalent computing power of about 0.05 of a MAC operation per input sample. So what sort of FIR filter can you build out of about 3 MAC and 2 sum and 2 lookup operations per sample?Yes with downsampling you are looking at a narrow window, but this is a good thing because it rejects any frequencies out of this band so its an advantage not a hindrance, you can still not downsample and then get the full 500KHz of bandwith with the reduced noise immunity and higher computing cost that comes with that, but you do get 1 000 000 frequency measurements per second(Noise free too as long as your phase window is large enugh). Whats even better is that the band of interest can be moved around by simply changing the rate of incrementing trough the sine table (a FIR filter needs new coefficients for this) and multiple sets of mixers can be used to track two or more doppler targets in the same signal (Other methods require FFT of filters to separate multiple sinewaves before measurement).As for the Phasor angle estimator, i agree. If you convert the signal into a phasor you know its exact frequency and job done. However can you recommend a simple and reliable algorithm for turning ADC samples into a phasor?I'm sorry if i am being annoying with this, not trying to argue or anything but just trying to get on the same page with this phasor approach.

author: ogden
author_rank: Super Contributor
timestamp: December 17, 2019, 08:18:13 pm
content: Quote from: Berni on December 17, 2019, 07:32:44 pmCartesian to polar coordinate transform. I think paper in the link below may answer this question as well.QuoteDon't be sorry at all. This is interesting topic, thank you for arguing. I believe that there is no single "ultimate" solution, every approach may have it's strong and weak points. I think following paper will give some insight into phasor approach.https://www.researchgate.net/publication/323628162_Implementation_Techniques_for_Frequency_Phasor_Estimation_in_Phasor_Measurement_Units_PMUs

author: KE5FX
author_rank: Super Contributor
timestamp: December 17, 2019, 08:36:15 pm
content: Quote from: ogden on December 17, 2019, 03:22:00 pmOptimizing the SNR performance of this sort of measurement usually comes down to reducing the measurement bandwidth, given that the target signal is a narrowband signal embedded in noise whose spectrum is potentially (DC-daylight]. It's easy to implement a narrowband filter near DC, not so easy to implement it at any arbitrary RF carrier frequency.This is as true nowadays as it was for Major Armstrong in 1918.

author: ogden
author_rank: Super Contributor
timestamp: December 17, 2019, 08:50:23 pm
content: Quote from: KE5FX on December 17, 2019, 08:36:15 pmNear DC means very limited (narrow) range of input frequencies, big group delays of filters, thus long sampling/processing periods. If this method is so good for RF, then why all frequency counters uses zero crossing count method instead?

author: Kleinstein
author_rank: Super Contributor
timestamp: December 17, 2019, 09:25:54 pm
content: Quote from: ogden on December 17, 2019, 08:50:23 pmFiltering the signal to a narrow band does not have to run as a classical FIR or similar filter. Frequency sensitivity can be a inherent more hidden feature. If one is only interested in the frequency one does not care about group delays per se. However the group delays have to be kept in mind when using a classical narrow band filter, especially for the important beginning and end. The data points at the ends are the most valuable. Use them just for filter settling is a bad idea.The usual frequency counters use the zero crossing method. This is because it is very easy and can be used at rather high frequencies.However microwave range "counters" do use different methods, like first mixing the input signal and down conversion to a, much lower frequency that is than analyzed in a different way. I don't know the details, but in modern implementation it could be well possible to use some kind of digital analysis like describes in the articles cited here.

author: nctnico
author_rank: Super Contributor
timestamp: December 17, 2019, 09:36:08 pm
content: Quote from: Berni on December 17, 2019, 07:32:44 pmI wouldn't say that. SDM shows weird startup behaviour in several graphs and it seems that the ZC (zero crossing) is determined over one period only. If you filter the input signal to a frequency band of interest and average the ZC over multiple periods (which is very applicable to the OP's intended use) you'll get very good results as well. You have to keep in mind that the paper is about tracking mains frequency which is a continuous signal and not a burst. Besides that tracking mains frequency seems to be time sensitive as in needing the answer as quickly as possible.

author: Berni
author_rank: Super Contributor
timestamp: December 17, 2019, 09:38:41 pm
content: Quote from: ogden on December 17, 2019, 08:18:13 pmYes i agree that's what my approach uses to turn a quadrature I/Q signal into phase and magnitude.But since the ADC is only spitting out real number samples where do you get the imaginary component to plug into the Cartesian input value? Or is the polar phase always just 0° or 180° ?Quote from: ogden on December 17, 2019, 08:18:13 pmI do agree there is no single best method. As it usually is in engineering every solution has its own sets of different advantages and disadvantages.Yep that paper does describe the approach to turning a sequence of real samples into a phasor:QuoteAnd it uses DFT to do it. So its pretty much just doing FFT of the ADC signal and then picking out the bin of interest, checking its phase. Doing another FFT of a signal sometime in the future, getting the phase of that too, finnaly calculating the difference between the two phases to get the full phasor.The DFT takes a horrendously large amount of computational work to calculate when you have decent sized inputs of >1024 samples. So anyone in the right mind would use FFT as it takes a lot less calculation to get the same result the DFT would give. On the upside there is no need for a input FIR filter to clean the signal since FFT can select the frequency it wants to look at, but even with FFT the computation for 1024 points takes around 25000 multiply operations and around 50000 sum operations(Or about 25 mul and 50 sum per sample). But since we are only interested in a narrow frequency window its not really necessary to calculate all of the spectrum, FFT unfortunately can't calculate just part of it, but DFT can do that. Yet if you look at what one frequency bin of a DFT actually does, you will find that its just correlating the signal against a sine and cosine of the given frequency, returning a complex number in rectangular form... much like a quadrature I/Q mixer does. Okay i admit if we are completely accurate here mixers don't do the same thing as correlation (that is integrating via sum, not via multiply), but the end result is pretty much the same thing, extracting the real and imaginary component so that it can be turned into a polar complex number.You are going for the same idea as the approach i suggested, the only difference is that my suggestion takes a bit of a computational shortcut using a mixer.Quote from: ogden on December 17, 2019, 08:50:23 pmIts because DC is a real bitch to deal with. This is the reason why analog circuitry often uses AC coupling capacitors in the signal path as soon as they are not required to work down to DC. Makes designing the analog circuit in the real world so much easier.This is also one of the reasons why mixers are usually not used with a local oscillator frequency that is identical to the signal of interest, you put the local oscillator off by a little bit on purpose so that your mixers output path can be AC coupled. Using this you move the signal of interest into an frequency area where your particular filter works best rather than the other way around for having to make your filter good where your particular signal is. In the case of FIR filters this means you need less steep of a filter, hence fewer coefficients hence easier to compute.As for why frequency counters use zero crossing, its because its the easiest thing to do in hardware and we know how to make really fast GHz speed divide by 2 clock dividers. Also they usually don't try to focus on a frequency band of interest and are designed to measure any shape of signal, not just sine waves.

author: ogden
author_rank: Super Contributor
timestamp: December 17, 2019, 11:28:23 pm
content: Quote from: Berni on December 17, 2019, 09:38:41 pmYes. You prepare real waveform for complex DFT or FFT same way.QuoteNot what I wanted to say. My bad. Can't find *that* document anymore. Will try to explain, in brief, simplest possible approach that obviously can be " extended".1) low-pass filter and normalize amplitude to (+/-) 1.0 2) run through "rising edge" zero-crossing detector to find first sample of sine period. As we have normalized waveform, calculate initial phase directly from amplitude: arcsin(amplitude1) 3) after enough (N) periods, thus zero-crossing detector "triggers", find final sample - and calculate final phase arcsin(amplitude2). 4) that's it. Calculate frequency f = (2*PI*N-arcsin(amplitude1)+arcsin(amplitude2))/(2*PI*delta_t). Example calculation for 1MSPS system, single period 10 samples, amplitude1 = 0.1 and amplitude2 = 0.2here, result is 101.61 KHz. Frequency is higher than 100KHz because phase rotated more than 2*PI. You can plug 0.1 number into both arcsins to get expected 100KHz - period exactly 10 samples.

author: ogden
author_rank: Super Contributor
timestamp: December 18, 2019, 12:56:19 am
content: Quote from: Kleinstein on December 17, 2019, 09:25:54 pmWell, well. I would like to remind what is written in the Subject of this thread and OP:Quote from: daqq on December 13, 2019, 09:48:55 am2ms means just 20 periods of the carrier signal. Nothing much you can downconvert, then low-pass filter in this case. You will hit end of the sample buffer long before low-pass filter settling. Berni may think about this as well.

author: Berni
author_rank: Super Contributor
timestamp: December 18, 2019, 08:48:17 am
content: Quote from: ogden on December 17, 2019, 11:28:23 pmYep now that is a more sensible method for extracting the phase out of a signal. Simple to do and needs even less computation.But since telling the exact zero crossing and amplitude of a noisy sine wave is a lot trickier it would still need a narrow bandpass filter in front in order to get the same amount of noise resilience. This comes with its computational cost and group delay.I admit that i am making a big deal out of noise even tho the OP did not mention it. But from experience with radar and sonar i know that these doppler signals that come out of them are often covered with all sorts of noise. Last time i had to deal with doppler radar i just used zero crossing and a fast timer to measure the periods (it also had a quadrature output so it could sense the target direction, not only speed). Simple but it worked well enough for the application it was for. But if you use some DSP magic you can often recover perfectly usable data from a signal that is 10 times smaller than the noise covering it.Quote from: ogden on December 18, 2019, 12:56:19 amOkay sorry i did loose track of the original spec (I was going on about 1MSPS 100Khz)But in any case this is not a problem since the mixer does not even need 1 complete period to work. Okay in practice if you used only 10 samples to feed the mixer you would be really sensitive to noise and distortion, but you would get the correct result if you fed in a clean sinewave.Averaging samples together as downsampling has no settling time and has a fixed predictable group delay of half the averaging window size. So in this case to get the most out of the 200 available samples one would mix the whole thing and then downsample by a factor of 100 resulting in 2 samples. These two samples are the phasors of the input signal that are 100 sampling periods apart in time(Centered on sample 50 and 150). So knowing the phase of the two and the time between them gives you the frequency. The whole thing required 400 MAC computations, 400 sums and 400 sine table lookups for the mixing and a handful of math operations to turn the 2 complex output samples into a frequency. All this with excellent noise rejection since 100 phase measurements are averaged together for each resulting sample. Im tempted to try and make this in excel to try out.But if you do require a settling time for the particular type of filter used then you can easily get around that in radar/sonar since you typically have a continuous stream of data to be fed trough processing, the OP has likely just pre trimmed the target of interest from his signal to result in 200 samples. So if you lay out your process correctly you tend to easily find some leftover data around the signal of interest to flush trough the slow filter and prime it for the real data of interest.

author: nctnico
author_rank: Super Contributor
timestamp: December 18, 2019, 10:12:16 am
content: Quote from: Berni on December 18, 2019, 08:48:17 amNot really. A single 2nd order IIR filter section (4 multiplies and 4 additions) is probably good enough. Averaging several cycles takes care of the rest of the noise. About 2 decades ago I implemented a DTMF decoder in an FPGA which uses zero cross detection (frequency counters) to detect the tones. Worked like a charm and didn't need any complex math.

author: Kalvin
author_rank: Super Contributor
timestamp: December 18, 2019, 11:39:50 am
content: One can remove the IIR-filter group delay by running the signal twice through the filter: First in normal sample order, and second time using reversed sample order for the filtered signal. At the same time one will get twice the filter order.

author: Berni
author_rank: Super Contributor
timestamp: December 18, 2019, 05:45:26 pm
content: Quote from: nctnico on December 18, 2019, 10:12:16 amYep and that is already more computation than is needed to quadrature mix the signal where filtering and phase extraction both happen in one go.But yes if the signal is reasonably clean and you don't need to be tightly selective of what frequencies you look at just zero crossings will be close enough. On a project that involves radar i did the frequency measurement even simpler by analog filtering it, sending it trough a comparator for zero crossing and finally using a timer in a MCU to count the pulse periods. It did need the signal to be a good bit out of the noise before it would reliably measure it, but that doppler radar module had enough SNR that it still detected far enough for what the client wanted, so i left it like that.But if put enough DSP effort into it to narrow your bandwidth onto what matters you can really pull out signals that are many times smaller than the noise floor itself (Tho in a lot of these cases you have to know what you are looking for, like a special tone, or frequency sweep or phase encoding etc to tell it apart from noise).

author: Kleinstein
author_rank: Super Contributor
timestamp: December 18, 2019, 07:11:30 pm
content: The iterative steps for the least square fit can be quite similar to the down-mixing method:The down mixing followed by linear interpolation to calculate the slope of the phase and thus frequency mulitplies the data with sine , cosine and a step like function from the slope calculation. The fit in the linearized form uses t*cos(wt), t*sin(wt) instead of the step functions. So the difference is not that large. The comparison also shows that it may not be a good idea to use only 2 blocks to get a start and end phase. The two steps are a rather coarse approximation to a linear slope. Using 3 blocks has the funny effect that the center block does no enter in the frequency calculation, so one does not even need to calculate the phase for the center part. Chances are the 3 block way may still be better than the 2 block version despite of ignoring 1/3 of the data. It probably gets even better with more block. However there is a limit - the down mixing to get a local phase only works well if the blocks for averaging have the length of full periods. Very short blocks may also reduce the suppression of off band noise.Most FIR filters are symmetric in time - so it does not make a difference running them forward or backwards. So no group delay involved from the start. The difference to analog filter is that such FIR filters are non causal (as the look in the future).

author: nctnico
author_rank: Super Contributor
timestamp: December 18, 2019, 10:35:52 pm
content: Quote from: Berni on December 18, 2019, 05:45:26 pmI'm not quite sure whether that is true because you'll still be aliasing the noise allover the place. And quadrature mixing doesn't do anything for averaging readings from multiple periods (which is a big advantage of using zero crossing). There is only so much you can do with signal processing; at some point it is better to use an algorithm which looks at the signal in the time-domain to make sense of it.

author: bson
author_rank: Unknown
timestamp: December 19, 2019, 06:05:22 pm
content: My guess would be cross-correlation (auto-correlation) is a more productive starting point than FFT... In fact the whole thing sounds like it would have solutions in echo detection.

author: mark03
author_rank: Frequent Contributor
timestamp: December 19, 2019, 06:49:34 pm
content: This thread has been really interesting to follow. EE version of a Rorschach test

author: SiliconWizard
author_rank: Super Contributor
timestamp: December 19, 2019, 06:55:20 pm
content: Quote from: mark03 on December 19, 2019, 06:49:34 pmAhah, sounds pretty close.

author: Marco
author_rank: Super Contributor
timestamp: December 19, 2019, 08:03:57 pm
content: Quote from: nctnico on December 18, 2019, 10:35:52 pmI kinda missed what quadrature mixing was supposed to do, it's mostly a way to get a complex signal near DC which you can use with the phase method (multiply each complex sample with the conjugate of the subsequent sample, sum them all up, take the angle of the result). The signal being near DC does do something for averaging though, any signal near DC can be trivially filtered through a simple window average before processing. Hilbert transform does not get you this.

author: nctnico
author_rank: Super Contributor
timestamp: December 19, 2019, 11:17:42 pm
content: Quote from: Marco on December 19, 2019, 08:03:57 pmBut a signal near DC also gets you into trouble with offsets. And due to aliasing you'll need to have both a bandpass filter in the analog and (at least a DC blocking filter) in the digital domain to remove any DC which gets into the ADC. All in all quadrature mixing doesn't give you any advantages because for a practical implementation you'll need just as much processing (filtering) compared to using the zero crossings method. And with quadrature mixing you can't do stuff like throwing away zero crossings which don't make sense (get rid of outlyers).

author: Berni
author_rank: Super Contributor
timestamp: December 20, 2019, 07:08:24 am
content: Okay yeah i did forget about DC offset on the input, that would cause issues for a mixer.But in terms of the aliasing filter that does need to a physical analog filter before the actual ADC. If the ADC acquires aliased samples there is no way to tell in the digital domain if the tone is a real tone or something higher that got aliased down.And yes the mixer does spit out a useless result at zero crossing, but you would generally have your downsampling window large enough to cover more than one entire period of the signal since yeah there is a lot of the carrier still left in the output signal. Most of the reason i claim it as being computational cheep is because the filter can be combined with the mixer into one MAC operation that generally takes a single cycle to execute(And also works well with SIMD).For when computational power is not so much of a concern i do admit that FFT is probably the best choice since not only does it have excellent noise resilience and selectivity, but it does it for the whole spectrum simultaneously (makes tracking multiple doppler targets easy) and the same FFT result can later be used to perform correlation of the input with the transmitted signal, since that's a nice fast way that can find an echo even when its buried under noise.

author: Kleinstein
author_rank: Super Contributor
timestamp: December 20, 2019, 09:14:44 am
content: If computational effort is not a big problem, the best solution is usually (for white noise it is the best) the least square fit. The time it needs for the fit depends on the implementation - general methods can be relatively slow. However in the right form only the frequency is a nonlinear parameter. Amplitudes and DC offset are linear and the linear approximation is thus quite good, so that the convergence can be quite good, so that only few (e.g. 2 or 3) iterative steps are needed. Each step takes a little more than twice the time of the mixing method. Because of the similarity to the mixing method, chances are that even a single approximation step could be already as accurate as the mixing method with the best block size.So the effort for the fit is not that high, if an adapted code is used and a reasonable starting point for the frequency is available.

author: hamster_nz
author_rank: Super Contributor
timestamp: December 20, 2019, 12:43:11 pm
content: Possibly useful observation...If the frequency range of the desired signal is very narrow, and the data has been filtered, and your sample rate is a multiple of four for the desired frequency you can convert the real data into I+Q data very simply.If the wavelength is 20 samples (+/- a smidgen) then:Code:data_i = (data_r[i-5]-data_r[i+5])/2;So, if 'data_r' has 200 samples of clean, filtered, valid data the following works:Code:double phase10, phase190,change;// Calculate the quadrature component of the real data in data_rdata_i = (data_r[ 10-(WAVELENGTH/4)]-data_r[10+(WAVELENGTH/4)])/2;data_i = (data_r[190-(WAVELENGTH/4)]-data_r[190+(WAVELENGTH/4)])/2;// work out phase as close to the start and end of the data as possiblephase10 = atan2(data_r,data_i);phase190 = atan2(data_r,data_i);// work out the change in phase per sample over 180 sampleschange = (phase190-phase10)/180;// print results.printf("Angle at sample 49 is %10.6f\n",phase10);printf("Angle at sample 149 is %10.6f\n",phase190);printf("Change in cycles after 100,000 samples %10.6f\n", change*100000/(2*M_PI));The key requirement is that your sample rate has to be a multiple of four of the nominal frequency of the signal of interest so you easily access data a quarter of a cycle away from any given sample - so for 10kHz you might want to use 80kS/s or 120kS/s, rather than 100kS/s.

author: nctnico
author_rank: Super Contributor
timestamp: December 20, 2019, 07:17:10 pm
content: Quote from: Berni on December 20, 2019, 07:08:24 amNo. Absolutely not. FFT (and also Goertzel) is utterly useless for detecting tones because in the frequency domain the bins are not shaped rectangular but parabolic. At the sides of the bin the signal strength will appear weaker.@hamster_nz: if there is a bandpass filter in the analog domain with sufficient suppresion you can even undersample. I have used that trick in an ultrasonic distance meter in order to use a lower samplerate.

author: Berni
author_rank: Super Contributor
timestamp: December 21, 2019, 06:13:58 pm
content: Quote from: nctnico on December 20, 2019, 07:17:10 pmI meant FFT as a process to convert the real ADC samples into complex form so that you can find the phase of that tone. We don't care about what frequency each bin is, they are just used to tell apart the signal of interest from other crap like noise, tones, harmonics etc.. Sort of like a narrow bandpass filter that happens to also convert the signal from real to complex form.Do this FFT twice on two different sections of the signal and you can get a phase difference between the two points to get the exact freqency. The wraparound integer phase can be gotten from zero crossing or calculating it from the FFT bin frequency if it has enough bins to be accurate within one whole wave.

author: iMo
author_rank: Super Contributor
timestamp: December 21, 2019, 06:15:58 pm
content: @daqq - could you post some data such the people may have something to play with?

author: nctnico
author_rank: Super Contributor
timestamp: December 21, 2019, 06:47:04 pm
content: Quote from: Berni on December 21, 2019, 06:13:58 pmNope. The result will still heavily rely on how well the actual frequency is aligned to the FFT bins. Besides that for FFT you'll need to use some kind of windowing to turn the burst signal from the OP into a seemingly continuous signal which esentially means not using all the incoming information.

author: Marco
author_rank: Super Contributor
timestamp: December 21, 2019, 08:20:57 pm
content: All in all, given the way DC makes things hard for the mixing/phasor approaches I think theFFT based methodstill is the easiest to implement.Fast enough, close to optimal, doesn't care about DC. I don't really care about the way frequencies cover bins, he said he did the Monte Carlo simulations to test his formula and it performed near optimal. I'm not going to do them myself, but unless I see someone do it and contradict him I have no reason to distrust his results.

author: RoGeorge
author_rank: Super Contributor
timestamp: March 01, 2020, 10:20:01 am
content: Without reading very carefully all the proposed solutions, the OP problem looks to me like an ideal application for a lock-in amplifier, simply because there is prior knowledge of the Tx signal.There were some quadrature based solutions mentioned, not sure if those will eventually implement an ad-hoc lock-in amplifier or not (at a draft look I perceived them as an implementation of a heterodyne).I'll go for the homodyne method first (in EE also known as a lock-in amplifier, in optics known as an interferometer) and see if it can get the required resolution. Also very important, try to make use of all of the prior knowledge, including expected S/N, expected frequency shift ranges, rate of frequency change.I never had to solve a similar problem, so my advice might be completely off.Whatever method you'll choose, "One can not find what one is not looking for", here meaning if you want to squeeze out the best results out of those few samples, youmusttake advantage of any prior knowledge you have about them, here the most important is that youhave the originalTx-ed signal, followed by the expectations about the Rx signal (expected S/N, deviation range, etc.).Chances are you might be able tovarythe Tx signal parameters, too (from the OP is not very clear if this is allowed). If true, this ability might improve the end results even more, depending on the exact application (e.g. the chirp modulation in radar/sonar type of applications).

author: ogden
author_rank: Super Contributor
timestamp: March 01, 2020, 07:15:23 pm
content: Quote from: RoGeorge on March 01, 2020, 10:20:01 amApparently you did not read your lock-in amplifier homework as well.Homodynedetectors (lock-in amplifiers) are used to demodulate either phase or amplitude, not frequency modulation/shift.

author: RoGeorge
author_rank: Super Contributor
timestamp: March 01, 2020, 07:55:55 pm
content: You could have said that without commenting about my person."Frequency modulation/shift" can always be seen as a phase modulation.

author: ogden
author_rank: Super Contributor
timestamp: March 01, 2020, 09:17:11 pm
content: Quote from: RoGeorge on March 01, 2020, 07:55:55 pmShifted carrier frequency can't be considered as modulation. Anyway lock-in amplifier requires carrier and LO frequencies to be identical which is not the case in doppler application.