Are there any conventions to indicate when numeric data is invalid? - Page 1
Forum section: electronics / beginners

author: e100
author_rank: Frequent Contributor
timestamp: March 19, 2022, 02:01:38 am
content: For example, if the speed sensor of a unidirectional motor is deemed untrustworthy then I set the value to -9 instead of 0.For invalid temperatures I choose -300 (celsius).What about things where the 'normal' measurement limits aren't clearly defined, for example a bidirectional motor that can have negative or positive measurements and the absolute limits haven't been established by experimentation. Do you just pick some arbitrary random huge positive or negative number that would be impossible to achieve without the motor self destructing before it reached that speed?

author: golden_labels
author_rank: Super Contributor
timestamp: March 19, 2022, 02:37:53 am
content: I believe that belongs to theprogramming section.If you use IEEE-754 or a similar floating point system, there isNaN. Those are designed specifically for the purpose of indicating invalid values and any calculations performed on them propagate that information withou a risk of ever accidently obtaining a valid value.Alternatively: pass the information explicitly in-band, signal it out-of-band or not report a value at all.Returning special values, that would not be possible under normal operation, is also common, but that introduces a risk of such values making into actual calculations. You must be sure that each time such a value is obtained, it must be handled and filtered out. This sounds obvious in theory, but in practice it’s guaranteed that from time to time such values will leak. For this reason do not choose them arbitrarily. Instead, try choosing values that — if they are accidently used for calculations — cause no harm. In other words: if a motor reports some absurd value that indicates “no data”, that value should not make the controller cause a short or uncontrollably spin the motor up. Instead it should lead to outputs that will stop it. To make it clear: I do not mean introducing special cases to calculations, but choosing the values themselves so — with existing calculations — they will not be harmful.

author: CatalinaWOW
author_rank: Super Contributor
timestamp: March 19, 2022, 03:22:57 am
content: You must first decide how you know the data is invalid and why you would keep known invalid data.The only reasons I can imagine for retaining such data is for debugging, in which case the actual wrong data would be what you would want to keep.

author: CatalinaWOW
author_rank: Super Contributor
timestamp: March 19, 2022, 03:26:38 am
content: If you are sure the data is invalid and you wish to keep following data processes intact an extrapolated point from prior data is probably a good choice. But the real issue is deciding how and when to deal with bad data.

author: ledtester
author_rank: Super Contributor
timestamp: March 19, 2022, 04:07:21 am
content: What's your application? Reporting the value to a user? Storing the value in a database? Communicating it to another computer (e.g. over a serial protocol)? Representing the value internally in your program?

author: langwadt
author_rank: Super Contributor
timestamp: March 19, 2022, 04:29:31 am
content: Quote from: e100 on March 19, 2022, 02:01:38 amif the temperature is controlling a heater it is probably a good idea to chose very high temperature, so you are absolutely sure the heater doesn't run amok and burn the house down

author: e100
author_rank: Frequent Contributor
timestamp: March 19, 2022, 05:34:27 am
content: Quote from: CatalinaWOW on March 19, 2022, 03:22:57 amIf two sensors provide conflicting readings then neither can be trusted.For example a level sensor at the top of a tank that says the tank is full, but the one at the bottom says it's empty.Sensor data that hasn't been updated in x amount of time (due to a comms failure) is flagged as untrustworthy.The older a measurement, the less likely it is to reflect the current reality. At some point you have to draw a line in the sand and say "no longer valid".Quote from: CatalinaWOW on March 19, 2022, 03:22:57 amI'm thinking of a distributed system where the core node is only interested in knowing what data is valid for input into its rules engine. For example, is the tank full, empty or don't know.

author: e100
author_rank: Frequent Contributor
timestamp: March 19, 2022, 05:50:56 am
content: Quote from: ledtester on March 19, 2022, 04:07:21 amFor putting the data into a rules engine.I don't know how you would display NaN on a graph for the operator to look at.Perhaps you would translate or constrain the data in the same way a 0-4-20mA current loop uses values from 0-4mA to indicate an error state.

author: bdunham7
author_rank: Super Contributor
timestamp: March 19, 2022, 06:07:07 am
content: Quote from: e100 on March 19, 2022, 05:34:27 amThat's not a good general rule and many systems don't think that way. If you have two sensors--say intake air and coolant temperature sensors on an engine--and one of them is out of range, you can keep the engine going pretty well by recognizing the out-of-range sensor as wrong and then substituting the data from the other sensor in its place. So it has to be thought through for each situation and simple, general rules won't be an optimum solution for many cases. Another relevant issue I've run into with the out-of-range data issue is if you aren't careful, you can end up with a system where it might physically reach the value you've chosen for flagging a fault. For automotive temp sensors there were systems that flagged a fault at the voltage corresponding to -40C, and occasionally they might get that cold. Same on the hot end, although IIRC, that usually only happened at temperatures reflecting severe damage.

author: ledtester
author_rank: Super Contributor
timestamp: March 19, 2022, 07:03:45 am
content: Quote from: e100 on March 19, 2022, 05:50:56 amFrom a programming perspective the value is really two pieces of information: a boolean indicating whether the value is known/is valid and a number which only has meaning if the boolean is true.The way to use such a data type is to first check the boolean and if it is true then you can access the number.This is akin to the Maybe/Option type you see in functional programming languages.You can represent both pieces of information in just the number by, as you suggest, using a special value to indicate that the value isn't valid, but your rules engine or program will still have to make this check before using the number to make sure it's valid. For instance, this happens in the C programming language with pointers as you have to check that a pointer isn't NULL before dereferencing it.I'm not sure what your rules engine gives you to work with, but one obvious idea is to store each reading as a pair of a boolean and number. Another option since you're likely going to be storing a sequence of these readings is to use two separate arrays, a boolean array and a number array.

author: rstofer
author_rank: Super Contributor
timestamp: March 19, 2022, 03:38:59 pm
content: There's nothing new in this problem. Some 50 odd years ago, there was a project where I worked involving the plotting of data received from 'sea buoys', wave height, water temperature, water salinity, etc). If they missed data or received invalid data, they just left a gap in to plot. Pretty simple solution and unquestionably correct. It's not like salinity could change over a large range instantaneously. Wave height probably increased as storms moved through and declined over time. I never asked how they dealt with 'rogue waves'.It is not uncommon to put a low pass filter on data. RPM data is the kind of thing that is often filtered. We don't need 'last digit bobble' on the display.There is an entire theme on data 'cleaning' in the area of data analysis. Datasets contain trash, they just do. Deal with it by whatever means seems appropriate. For 'large data', it is common to just ignore the entire record. Flag it, sure! But don't use it. Google 'data cleaning' for references.There must be an anticipated range of values and a value outside that range should be questioned. Division by zero is one of the ways this can occur but it's not just zero. Division by 10-18will also cause a problem if you're measuring house prices.I don't imagine there is a standard 'right answer'. It's going to depend on the application.

author: CaptDon
author_rank: Super Contributor
timestamp: March 19, 2022, 10:13:18 pm
content: One that is in common use is in NMEA 0183 GPS and maritime data strings. 'V' near the end of the string indicates VALID and 'A' in that position indicates not valid. I forget what 'A' is supposed to represent?? Maybe 'ABSENT'??

author: CatalinaWOW
author_rank: Super Contributor
timestamp: March 20, 2022, 12:33:18 am
content: I will repeat that the real answer to this is how you are going to use the data. Skipped data entries, invalid flags, and specific values intended to be used as flags are all valid approaches, but dependent on the situation.To plagiarize an example from earlier in the thread, think of this case. Two gas tank level indicators. With different readings. But in many cases both readings can be trusted. The problem is one of calibration. One says 1/4 tank with an uncertainty of 1/4 tank. The other says 3/4 of a tank with uncertainty of 1/2 tank. In this case (given the added knowledge of sensor performance) you can with high confidence say the level is between 1/4 tank and 1/2 tank.But another situation which occurs in one of my vehicles is that the wiper on the pot for the in tank float has mostly failed. It most of the time reads empty, but when contact is occasionally made the value is pretty accurate. The algorithm for dealing with this data is completely different. And the simple algorithm (ignoring zero values) has an obvious and enormous flaw since zero is in the set of valid outputs.The real answer is as rstofer and others have suggested. You have to think a lot about the problem (why the data is bad, how bad is it, how frequently is it bad, how critical it is to have each piece of data) and what your desired outcomes are before you can select a technique.Another example to think about is a sensor that suffers from occasional bursts of noise. The source of this could be anything from slip rings to sunspots, to RF interference and many others. Questions to think about. What is the magnitude of the noise relative to the signa? What is the duration of the noise relative to any control intervals? Are there any periodicities in the noise? All of those will have impacts on the selected approach.