6.5 Digit DMM storage resolution? - Page 1
Forum section: electronics / testgear

author: KedasProbe
author_rank: Frequent Contributor
timestamp: March 02, 2014, 03:57:51 pm
content: Hi,I was checking the resolution of the measured voltage values of my DM3068.Now I have a question, if the ADC is 24 bits shouldn't the extra bits in a double(53bits total) only be the result of multiplying with a range constant.Then why do I see much smaller resolution steps, where/why are these extra bits generated?or which math/method on this 24bit value is causing this?General info:2200000 Count 21-22 bits 6.5 Digits (DM3068 Spec. Count)16777216 Count 24 bits 7.5 Digits (DM3068: 24bit ADC Ti ADS1256http://www.ti.com/lit/ds/symlink/ads1256.pdf)DM3068 Stored Resolution Double: (.xdat file)100PLC; Range 0.22V/(50pV/100); 39 bits; 11.5 Digits100PLC; Range 2.2V/(500pV/100); 39 bits; 11.5 Digits10PLC; Range 2.2V/(500pV/10); 36 bits; 10.5 Digits1PLC; Range 2.2V/500pV; 33 bits; 9.5 Digits0.06PLC; Range 2.2V/(500pV/0.06); 28 bits; 8.5 DigitsMeasuring method:Add short or constant voltage, store the 5000 measurements to .xdat file (this contains doubles, 15 Digits)then in excel sort, stepsize to next cell, sort steps.

author: KedasProbe
author_rank: Frequent Contributor
timestamp: March 02, 2014, 05:31:04 pm
content: I was just thinking what if all measurements are 0.006PLC (the fastest)The 10PLC setting is then the average of 10PLC/0.006PLC measurements.Then the average value could have a resolution of 24bit 7.2 digits + log(10/0.006) digits = 10.4 digits

author: Andreas
author_rank: Super Contributor
timestamp: March 02, 2014, 07:11:15 pm
content: Quote from: KedasProbe on March 02, 2014, 05:31:04 pmHello,definitely not.The resolution is the rms value of the noise of your measurements against the full range where you are measuring.If (and only if) the noise is distributed gaussian then you need to average 4 measurement values to increase the resolution by a factor of 2.Below 1 NPLC the noise (mains line frequency part) will _not_ be gaussian.So look at your rms noise specifications of your meter to get the (rms) resolution. The peak to peak values of the noise are a factor of around 6 higher than the rms values for gaussian distribution.with best regardsAndreas

author: KedasProbe
author_rank: Frequent Contributor
timestamp: March 03, 2014, 08:16:31 am
content: I'm not talking about the useful resolution of the measurement. (square root n, although related to it)I want to know why the stored values have the resolution they have.If my assumption above is correct, is it common practice in DMM to measure at the same fast speed and handle higher PLC settings with averaging?

author: macboy
author_rank: Super Contributor
timestamp: March 03, 2014, 03:04:46 pm
content: Quote from: KedasProbe on March 02, 2014, 03:57:51 pmIf you are using 0.006 PLC integration time (100 microseconds!), then it is extremely likely that the instrument is internally limiting the resolution, regardless of the number of digits being displayed or captured to a data file. The effect you are seeing is rounding error in the math that is used to generate the base-10 floating point values from a 24 bit binary value with offset and gain calibration later applied. Rest assured that your meter is not measuring with 11.5 digits of resolution in 100 microseconds. It is not measuring with 6.5 digits in 100 microseconds.Floating point is strange in that many/most rational base-10 numbers can't beexactlyspecified in floating point format. Furthermore, scaling issues come into play when you start manipulating the numbers. e.g. (75.9000 - 75.8999) will not giveexactlythe same result as (0.9000 - 0.8999). Both results will be very close to 0.0001, but neither is exactly 0.0001 because that number can't be exactly represented as a double, nor can the numbers used in the calculation. The closest approximation (IEEE 754 double) is 1.00000000000000004792173602386E-4 .Rounding errors also occurs in Excel, which is accepted to have an accuracy of no more than 15 significant figures.

author: sync
author_rank: Frequent Contributor
timestamp: March 03, 2014, 03:12:39 pm
content: Quote from: KedasProbe on March 03, 2014, 08:16:31 amI think meters with an integrating ADC just adjust the integration time.

author: CaptnYellowShirt
author_rank: Frequent Contributor
timestamp: March 03, 2014, 05:39:03 pm
content: Quote from: KedasProbe on March 03, 2014, 08:16:31 amWith the HP 3457a "PLC" and "Averaging" are two totally different concepts.Adjusting the PLC parameter actually changes the way the ADC does its conversion. Among other things, it adjusts the voltage comparator ramp slope. Low PLC's give a very steep slope while high PLC's give a very long flat slope. The trade off is voltage accuracy vs time.On the HP 3457a the ADC is able to give 20 bits of resolution, which is about 1e6 counts. However, when you average those values together (after the ADC has put them into memory), the registry is much wider than 20bits. I don't know the in-memory operating precision for the 3457a, but averaging gives tends to give values in the 9 digit count range.What is this sqrt(n) and 4 samples to double the accuracy stuff? I ask because I'm in the middle of beefing up on my working knowledge of statistics and it sounds interesting, but its not something I've come across.Do you know what kind of processor your DMM uses? It seems like your step size is somewhere in the 40-41 bit range. Some processors use 40-bits as their 'extended' floating point notation.Also, if you over-range on a fixed measurement mode, what number comes out? 10.3V on a 10V range...

author: HKJ
author_rank: Super Contributor
timestamp: March 03, 2014, 07:11:41 pm
content: Quote from: macboy on March 03, 2014, 03:04:46 pmThen it must apply some random rounding to get these fine steps or else each 24 bit value would generate the same value each time.

author: Andreas
author_rank: Super Contributor
timestamp: March 03, 2014, 11:07:41 pm
content: Quote from: KedasProbe on March 03, 2014, 08:16:31 amcommon practice: only for sigma delta converters with continuous output data. (like the ADS).They are usually limited to around 5-10 NPLC in the highest resolution range.With lower NPLC the noise increases.With a multi-slope converter you can extend the integration time nearly infinite.(if your timer has enough bits to measure it).With a dual slope converter the only possibility is to adapt the integration slope to get higher PLC settings.(otherwise the integration is not continuously).with best regardsAndreas