9½ digits on Keithley 2000 when reading data remotely ? - Page 1
Forum section: electronics / testgear

author: petemate
author_rank: Regular Contributor
timestamp: November 13, 2020, 12:58:40 am
content: Hi guys,Why is it that the Keithley 2000, which is a 6½ digit DMM, shows 9½ digits when you read data through GPIB or RS232? For instance, the display shows 0.999563V, but the remote read value is 0.999562592V (output as 9.99562502E-01). Whats the cause or purpose of all these extra and probably redundant digits?

author: maginnovision
author_rank: Super Contributor
timestamp: November 13, 2020, 02:32:42 am
content: It's filtering it just gives you the actual result over remote rather than display @ valid 6.5D on display. Maybe you want the full result to process later.

author: Fungus
author_rank: Super Contributor
timestamp: November 13, 2020, 03:28:50 am
content: Quote from: petemate on November 13, 2020, 12:58:40 amIt sent you a floating point number, binary floating point numbers are imprecise when converted to decimal representation.

author: Kleinstein
author_rank: Super Contributor
timestamp: November 13, 2020, 08:03:19 am
content: The number is just a floating point number. If one carefully looks at the data, not all values are possible, but there are discrete steps. These steps are not 10 µV to get exact 6.5 digit resolution for the 10 V range. However they are also not a simple 1 µV or 0.1 µV but more a odd number (and slightly different between individual meters), as the calibration is with a numerical scale factor. The floating point number avoids extra errors from rounding, but the extra digits have limited value.

author: Fungus
author_rank: Super Contributor
timestamp: November 13, 2020, 08:13:38 am
content: Quote from: Kleinstein on November 13, 2020, 08:03:19 amYep. Any thoughts that it might secretly be accurate to 9 digits will be wrong.

author: petemate
author_rank: Regular Contributor
timestamp: November 13, 2020, 09:54:00 am
content: Thanks for your replies! But why not just round the "remote value" it to the value presented on the display? Why give the full, yet unreliable number?

author: Fungus
author_rank: Super Contributor
timestamp: November 13, 2020, 10:07:51 am
content: Quote from: petemate on November 13, 2020, 09:54:00 amThat's like asking why 1/3 has a lot of decimal places when you write it in base 10.Conversion between number bases isn't always neat and tidy.Edit: I'm assuming this transmission is using binary floating point numbers.

author: Kleinstein
author_rank: Super Contributor
timestamp: November 13, 2020, 10:44:48 am
content: More digits are also usually available in ASCII mode results. One still has a similar rounding problem: The result from the ADC is usually with a somewhat better resolution, but the steps are not so simple to represent. The additional digits avoid/reduce a rounding error. The principle is that the quantization steps from the ADC may be at some 2.34567 µV or a similar odd number and it is thus difficult to represent that in a short decimal form: either you get rounding errors or additional digits with very limited value.So the longer result send to the computer can be a little better than the 6.5 digit display, though usually not much. On the other side it does not cost much sending 1 or 2 additional bytes and the computer usually does not mind. The highest speed modes may / will show less digits or use binary data.

author: David Hess
author_rank: Super Contributor
timestamp: November 14, 2020, 07:32:58 pm
content: They might have wanted meters with different resolutions to report in exactly the same numeric format for interoperability.