4th order polynomial coefficients for pressure at temperature readings, Help!!! - Page 1
Forum section: electronics / projects

author: itsbiodiversity
author_rank: Regular Contributor
timestamp: August 18, 2020, 10:35:12 pm
content: I am working with an instrument that contains an RTD to account for temperature in precise pressure measurements. The designer used a fourth order polynomial to calculate correction factors a0, a1, a2, a3, and a4. Pressure readings of a standard (STD) versus a unit under test (UUT) are taken at multiple temperatures (3 temperatures) related to an ADC Count Value (3 count settings, directly correlating to temperature). Pressure readings are taken from a low range to a high range at each temperature. Can anyone help direct me to how to calculate these coefficients? I can provide an excel sheet with data if it helps. Thanks so much. Data presented below for copy/paste, but an excel sheet is shared below. Any help would be appreciated.psi °C Counts uut-15 15 5485 -15.017-13 15 5485 -13.014-11 15 5485 -11.011-9 15 5485 -9.01-7 15 5485 -7.008-5 15 5485 -5.006-3 15 5485 -3.004-2 15 5485 -2.0020 15 5485 02 15 5485 2.0013 15 5485 3.0025 15 5485 5.0047 15 5485 7.0069 15 5485 9.00711 15 5485 11.00813 15 5485 13.00815 15 5485 15.008-15 21 7386 -15-13 21 7386 -13-11 21 7386 -11-9 21 7386 -9-7 21 7386 -7-5 21 7386 -5-3 21 7386 -3-2 21 7386 -20 21 7386 02 21 7386 23 21 7386 35 21 7386 57 21 7386 79 21 7386 911 21 7386 1113 21 7386 1315 21 7386 15-15 28 9471 -14.99-13 28 9471 -12.991-11 28 9471 -10.993-9 28 9471 -8.994-7 28 9471 -6.996-5 28 9471 -4.997-3 28 9471 -2.998-2 28 9471 -1.9990 28 9471 02 28 9471 1.9993 28 9471 2.9985 28 9471 4.9977 28 9471 6.9969 28 9471 8.99611 28 9471 10.99613 28 9471 12.99615 28 9471 14.996

author: ahbushnell
author_rank: Frequent Contributor
timestamp: August 18, 2020, 10:53:46 pm
content: You can calculate the polynomial or use a lookup table. Is it a PT100 RTD? That means it's a nominal 100 ohm resistance. There is also PT1000 but not common.I would suggest a lookup table unless floating point is not a problem.Omega has information on RTD's. Here are some links.https://www.omega.com/en-us/resources/rtd-resistance-elements-principleshttps://www.omega.com/en-us/resources/temperature-transmitter-scaling-methodologieshttps://www.omega.com/en-us/resources/rtd-hubEuropean table (more common)https://assets.omega.com/pdf/tables_and_graphs/thermistor-resistance-europe.pdfUSA tablehttps://assets.omega.com/pdf/tables_and_graphs/thermistor-resistance-us.pdf

author: itsbiodiversity
author_rank: Regular Contributor
timestamp: August 18, 2020, 11:02:35 pm
content: It is a pt100 rtd. I corrected the actual RTD measurement already. This is specifically a correction for the pressure at the multiple temperatures across the range of pressure. I am not looking to calibrate the RTD, I am looking to use the variation in pressure at temperatures not standard to develop a correction curve. If the unit reads -15.036 psi at 0 °C, -15 at 20 °C, and -15.001 and 28 °C, for instance, to correct for those values. The problem I have is that I have data points where the variation differs (see attached excel file). I hope any of this makes sense.

author: itsbiodiversity
author_rank: Regular Contributor
timestamp: August 18, 2020, 11:39:24 pm
content: You know that is how I'm confused as well. I am starting to think someone left a vital piece of the puzzle out to make this difficult for future discoverers.I hope this helps,but I am trying to develop the coefficients by comparing the Standard psi to the UUT psi at multiple temperatures. At Standard temperature the coefficient is 1 (a2), it has to be. I am trying to calculate the other correction coefficients for the cold and warm temperature pressure corrections.End result, correlation coefficients for psi corrections to the uut at applied ADC temperature and applied PSI to pressure sensor.

author: itsbiodiversity
author_rank: Regular Contributor
timestamp: August 18, 2020, 11:41:21 pm
content: I believe x is (column a) and y is (column d), but at said Temperature (column b) and adc counts (column c).

author: b_force
author_rank: Super Contributor
timestamp: August 19, 2020, 12:19:33 am
content: I am missing context and error analyses here. (i am reading the word "precise")Without you can't say much about any value and therefore formula anyway

author: b_force
author_rank: Super Contributor
timestamp: August 19, 2020, 01:03:08 am
content: Quote from: DrG on August 19, 2020, 12:42:35 amIsn't that the only point to make?Context is everything, the rest is just a matter of putting in numbers, getting a trend line and fit what should be according physics?If it doesn't fit, there is something strange happening, or you should apply for a Nobel Price since you bumped into something out of the ordinary?

author: b_force
author_rank: Super Contributor
timestamp: August 19, 2020, 01:28:49 am
content: Quote from: DrG on August 19, 2020, 01:23:44 amI just even told what to do, see what relation is being expected from a physics point of view.If that doesn't fit, you either have a serious problem, or some kind of error in your system.So next step is to figure out what that error is.

author: itsbiodiversity
author_rank: Regular Contributor
timestamp: August 19, 2020, 01:34:12 am
content: you are correct. I find now the correction factors "go into a programmed" 4th degree polynomial. I was very confused. I am looking for correction factors for the uut to match std psi at any temperature. I apologize for any miscommunication.

author: itsbiodiversity
author_rank: Regular Contributor
timestamp: August 19, 2020, 01:35:27 am
content: the only file i have is an old quattro file... any way to share that?

author: Jay_Diddy_B
author_rank: Super Contributor
timestamp: August 19, 2020, 02:58:21 am
content: Hi,I am not sure that you have enough data.If am reading the data that you provided correctly:1) at 21C the device reads the nominal pressure and no correction is required.2) at 0 pressure the device reads 0 at all three temperatures.If I plot the error, that is the difference between the nominal reading and the measured reading, on a surface plot I get a shape which is twisted like a propeller:I am not sure how you correct for this with a single polynomial.Jay_Diddy_B

author: IanB
author_rank: Super Contributor
timestamp: August 19, 2020, 04:54:47 am
content: Quote from: itsbiodiversity on August 18, 2020, 10:35:12 pmThis is a regression problem.You have a measurement value compared to a standard (known) value.Therefore your measurement error is:error = standard value − measurement valueOver the range of measurements you have an error function:error = f(T, P)You want to fit a polynomial that best represents this error so you can apply it as a correction to the measured values.You say the correction factors are a0, a1, a2, a3, a4, but you need to know the form of the polynomial before you can do anything.I did a bit of experimenting in Excel and found a correction polynomial of this form was able to fit the error quite well:correction = (T-21) * (A + B*P + C*P*(T-21) + D*P^2)However, this may not be the correction polynomial you have to work with. You need to know what it is.Here is a chart showing the measurement error (blue bars) and the calculated correction (orange bars) that I obtained:(A = 5.177e-5, B = -1.131e-4, C = 5.213e-6, D = 1.565e-6)

author: itsbiodiversity
author_rank: Regular Contributor
timestamp: August 19, 2020, 08:10:52 am
content: Quote from: DrG on August 19, 2020, 02:52:24 amCondescend much? Especially while answering incorrectly.I am not looking for help with temperature information. If you think I cannot calculate RTD coefficients then please pack it up. And yes, I am using a previous manual that used the term "correction factor" - I spoke to the inventor yesterday who called it such. I'm going to say I'm safe using the term as they invented and designed it to use "correction factors". Please do not tell people how to speak - it's extremely condescending as well.I am looking to correct the PRESSURE readings across the temperature spectrum. I AM NOT LOOKING TO CORRECT THE RTD READING! I am only looking to correct for pressure at the given temperatures. There may be a specific formula used by the author of the unit that gave these "coefficients" a0 to a4 - I am going off of available information. I've had three or four engineers look at this today and we all were lacking an answer.

author: itsbiodiversity
author_rank: Regular Contributor
timestamp: August 19, 2020, 08:17:49 am
content: Is this for the pressure readings over temperature? The temperature and it's accuracy does not matter in this case - I am trying to tie factors to the pressure so that readings taken by a module at say 20 C are the same as measurements taken at 28 C. I do know the data I presented was used by the original programmer to do the statistics. I have attached an example with the columns and regression analysis. Any help figuring out what a0 through a4 would be in a scenario like this would be appreciated.

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 19, 2020, 11:55:48 am
content: Quote from: itsbiodiversity on August 19, 2020, 08:10:52 amThe naming depends on approach.Because the device behaviour is almost linear (and often modeled as linear), it makes sense to call the adjustments necessary as correction factors. This is the device-based approach.When the behaviour is modeled using a polynomial (a quartic one, in this case: \$f(x) = F_0 + F_1 x + F_2 x^2 + F_3 x^3 + F_4 x^4\$), the parameters to be fitted are the polynomial coefficients (\$F_0\$, \$F_1\$, \$F_2\$, \$F_3\$, and \$F_4\$). This is the measurement-based approach.No need for anyone to get snippy, just because they're looking at the situation from different angles, and using different nomenclature because of that.As a computational materials physicist, I do believe that in this particular case, it would be better to use a bivariate polynomial, rather than two separate polynomials (one for the RTD resistance-to-temperature curve, the other for pressure reading compensation as a function of temperature). Essentially, you would have $$f(x, y) = \sum_{i=0, j=0} F_{i j} x^i y^j$$ where \$F_{i j}\$ are the coefficients fitted to sample data (via least-squares fitting; one of the easiest tools for this is Gnuplot), \$x\$ being e.g. the ADC reading for the RTD, and \$y\$ the ADC reading for the pressure sensor, and \$f(x,y)\$ yielding the normalized pressure.Yes, I know: the original inventor didn't do it this way. It does not matter. It is a sensor combination whose behaviour is continuous and almost-but-not-exactly linear, and for this, the best model is a bivariate polynomial fitted to measured parameters, then verified that the fit looks sane (\$F(x, C)\$ and \$F(C, y)\$ look sane and predictable for all constants \$C\$), i.e. have the form predicted in literature. This is very, very common in physics, and indeed in my own field, non-QM (AKA traditional potential model) molecular dynamics with millions of atoms and more, where the potential function has aformbased upon theory and mathematical approximation, but the individual coefficients are fitted from real-world measurements. (In particular, it is very common to use one set of coefficients when modeling surfaces, and another when modeling bulk properties.)

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 19, 2020, 12:10:45 pm
content: If you provide a table with three entries – temperature count, pressure count, normalized pressure – with as many samples as you can muster, I'd be happy to provide you with a reasonable bivariate approximation, that takes in the temperature and pressure counts, and spits out normalized pressure.Or, if you want, four entries – temperature count, pressure count, actual temperature, normalized pressure – and I'll provide you with two bivariate polynomials: one spits out the actual temperature, and the other normalized pressure, when given just the temperature and pressure counts.I'll even show how I do that with Gnuplot. (Why gnuplot? It's free, takes in simple tabulated data, and can produce publication-quality graphs, making it quite popular among computational scientists.)

author: ETITsynthesizer
author_rank: Contributor
timestamp: August 19, 2020, 12:26:34 pm
content: I think Nominal Animal had the best response in this whole thread. I don't know if this is helpful but I would have tried the polynomial regression tool in excel. this gives the answer in the standard polynomial form. let the computer do the math. I have never worked with a bivariate polynomial in math before but I can dig it. I will trust that he/she knows what they are talking about since they have experience with this kind of regression.

author: itsbiodiversity
author_rank: Regular Contributor
timestamp: August 19, 2020, 01:38:10 pm
content: I believe you've got the answer here. My apologies for any explanation errors - I am working "backwards" from available knowledge in many aspects. Did you happen to look at the spreadsheet I attached on the last post? The only data that they used for this product was taking PSI readings from the Low to High Range at 3 different Temperatures. Looking at the spreadsheet it appears they have a Zero: Regression Output with x coefficients. These x coefficients are used in the formula +$A:$N$11*C7+$A:$O$11*D7+$A:$P$11*E7+$A:$Q$11*F7where N11 is an x coefficient, C7 is the temperature counts, O11 is another x coefficient, D7 is the UUT PSI reading, P11 is the third x coefficient, E7 is Temperature Counts (C7) x UUT PSI reading (D7), Q11 is the fourth x coefficient, and F7 is temperature counts squared (C7 x C7) multiplied by UUT PSI reading (D7).Applied line by line the formula does a great job of correcting for temperature area. The author of the instrument stated the following:Enter Temperature CoefficientsThis command provides for the entry of temperature coefficients that will compensate the sensor for ambient temperature conditions. The coefficients are determined during the factory calibration process.tcomp <a0> <a1> <a2> <a3> <a4> (THESE ARE WHAT I AM TRYING TO IDENTIFY)The operator specifies five coefficients, which are used in a fourth order polynomial that corrects temperature readings for the ambient temperature at the sensor.The currently effective temperature compensation coefficients can be viewed using the coef command, which is described in the user’s manual.default is 0 0 1 0 0.Does this make sense? I feel like this is close to what you were describing. I'm attaching another spreadsheet here, and I'm also downloading the program you mentioned. Thank you so much.

author: itsbiodiversity
author_rank: Regular Contributor
timestamp: August 19, 2020, 01:42:28 pm
content: Pretty sure this is the first time I've encountered anyone rude like this on here. I usually hang in the Metrology forum, which is what my career is (Chemist/Metrologist). I'm not sure if him saying "good luck with your homework" is supposed to make me feel inferior, but an answer surely would've been respected. I'm positing that if you knew the answer you'd have provided it, and if you do have the answer I'd still be appreciative.I truly appreciate those of you that have spent time and thought towards helping with this question.

author: DrG
author_rank: Super Contributor
timestamp: August 19, 2020, 02:11:15 pm
content: Quote from: itsbiodiversity on August 19, 2020, 01:42:28 pmRead me carefully...I spent plenty of time with your problem last night...repeatedly presenting graphs and coefficients and trying to help you. Trying, in fact, to understand what it is that you were trying to do. It was clearly a mistake for me to have even tried to help you and I regret spending anytime on it at all. Obviously you did not appreciate the effort and, equally obvious, is that it did not help you. I did, however. learn from the experience.I don't think you explained yourself very well and I believe you when you said how confused you were about some things about the task. You can certainly understand that it is difficult to ascertain what someone else knows or does not know on a forum.I am glad that you seem to have found your answer. I do think it is some kind of homework problem and I don't think that you explained what you were trying to do very well. I am saying good luck with your homework (or whatever it is you are trying to do) in just exactly as it says. Beyond that, I don't much care what you think or how you feel.

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 19, 2020, 02:22:12 pm
content: I don't like the idea of someone believing me because I sound trustworthy, so let me waffle on a bit about the differences between two consecutive single-variable polynomial fits, versus a single bi-variate polynomial fit. For simplicity, I'll limit to cubic (third degree) polynomials, only because it keeps the number of coefficients down.Let's say \$x\$ is the pressure reading, and \$y\$ is the temperature reading, directly from the hardware (ADC).RTDs are almost linear, but not quite. Their resistance as a function of temperature is given byCallendar–Van Dusen equation,$$R(T) = R(0) \biggr( 1 + A T + B T^2 + (T - 100) C T^3 \biggr) = R_0 + R_1 T + R_2 T^2 + R_3 T^3 + R_4 T_4$$where \$R_0 = R(0)\$, \$R_1 = R(0) A\$, \$R_2 = R(0) B\$, \$R_3 = -100 R(0) C\$, and \$R_4 = R(0) C\$; and the three coefficients, \$A\$, \$B\$, and \$C\$, are dependent on the actual sensor (material).The above equation can be solved algebraically for \$T\$ when \$R(T)\$ is known, but the expression is absolutely horrible: nobody uses it. The three coefficients are very close to zero, so \$R(T)\$ is almost linear, \$R(T) \approx R_C (T - T_0)\$. Therefore, \$T(R)\$ is almost linear too, and we can approximate it using a polynomial also. Typically, a quartic (fourth degree) polynomial is used.To measure the resistance, we use ananalog to digital converter, ADC, which converts either voltage or current to a reading. Typically – for example in the ubiquitous MAX31865 RTD-to-digital converter –, the RTD is fed from a precise current source, and the voltage (difference, infour-terminal configuration) converted to a reading using a linear ADC.To measure thepressure, there are a number of completely different types of sensors. Piezoresistive strain gauges and metal strain gauges have a pretty linear response in their stable measurement region, so we can assume the pressure reading is more or less linear function of the actual pressure. Depending on the actual sensor, some compensation due to nonlinearity may be needed, so often a polynomial fit (with zeroth and first degree terms being "large", and the higher degree terms very small; so "almost linear") is used.Every one of these is an approximation, fitted to the measured parameters. We don't have any specific model, except for, uh,"this is almost linear". (We do know that for RTDs, second and higher-degree terms are all negative for most common materials used in RTDs, for physical reasons actually.) So, the sanity check for the models is to examine a few (say, two to five) different temperatures, and see how it affects a specific (constant) pressure; and a few different pressures, to see how temperature variation affects the pressure reading. If they look to be within expected values, the overall approximation is valid.So, what's the difference between applying consecutive polynomial fits, and a single multi-variate fit? Cross terms.Consider the situation when you have two different readings, \$x\$ and \$y\$, and you have a function \$f(x, y)\$ to evaluate, but both \$x\$ and \$y\$ are only almost linear. You can calculate \$f\bigr(X(x), Y(y)\bigr)\$ where \$X(x)\$ is the compensated \$x\$ and \$Y(y)\$ is the compensated \$y\$. This is perfectly okay to do, if you have a reason and reliable fits for \$X(x)\$ and \$Y(y)\$.In this case, \$f(x, y) = x + g(y)\$, i.e. temperature \$y\$ is only used to give a correction \$g(y)\$ to the pressure \$x\$, to estimate the actual pressure.The pressure compensation is only dependent on temperature, not pressure.In fact, we could write$$f(x, y) = x + G_1 (y - y_0) + G_2 (y - y_0)^2 + G_3 (y - y_0)^3 + G_4 (y - y_0)^4$$where \$y_0\$ is the temperature at which the pressure readings are correct with zero compensation. See how there are no terms multiplying both \$x\$ and \$y\$? It means this model cannot vary the pressure compensation based on pressureandtemperature, only on temperature. I do not believe this is physically sound.Consider the case where the estimated actual pressure \$f(x, y)\$ is a bivariate (temperature reading \$y\$, pressure reading \$x\$) polynomial of degree four:$$\begin{aligned}f(x,y) &= F_{0 0} + F_{0 1} y + F_{0 2} y^2 + F_{0 3} y^3 + F_{0 4} y^4 + \\~ & ~ ~ F_{1 0} x + F_{1 1} x y + F_{1 2} x y^2 + F_{1 3} x y^3 + F_{1 4} x y^4 + \\~ & ~ ~ F_{2 0} x^2 + F_{2 1} x^2 y + F_{2 2} x^2 y^2 + F_{2 3} x^2 y^3 + F_{2 4} x^2 y^4 + \\~ & ~ ~ F_{3 0} x^3 + F_{3 1} x^3 y + F_{3 2} x^3 y^2 + F_{3 3} x^3 y^3 + F_{3 4} x^3 y^4 + \\~ & ~ ~ F_{4 0} x^4 + F_{4 1} x^4 y + F_{4 2} x^4 y^2 + F_{4 3} x^4 y^3 + F_{4 4} x^4 y^4 \\\end{aligned}$$Many of the cross terms (\$F_{i j}\$ where \$i \ne 0\$, \$j \ne 0\$) are zero or very close to zero. They describe the change in the pressure compensation as a function ofbothpressure and temperature.I believe these cross terms are needed, because we do not know exactly the pressure sensor behaviour as a function of temperature at different pressures (i.e, \$f(x, y)\$ in exact form!), but there is reason to believe that while it is likely small, there is some, due to mechanical reasons alone. (For example, thermal expansion of the strain gauge, and the fixture where the strain gauge is located, if the pressure sensor uses the strain gauge.)If mechanical, these dependencies are complex, but small. We have about zero hope of deriving a numerical description from first principles – wecancompute it, it is just way too complicated to be worth the effort.Instead, we do a set of measurements at known temperatures and pressures. If we draw a diagram with pressure on one axis, and temperature on the other axis, using a polynomial fit to a number of samples gives us an interpolated estimate within the convex hull of the samples, and handwavy extrapolated estimate outside the convex hull. (Outside the convex hull we are basically extrapolating the system behaviour, that's the hand-waving bit.) The denser the samples, the more reliable the estimate in that region.If we only do measurements along each axis (i.e., one set of samples roughly horizontally in one line, and another set of samples roughly vertically on one line), we have no information for the cross terms, and fitting the measurements separately in their single-variate polynomials makes more sense.(I like to use Gnuplot for fitting, because it allows/requires one to provide the function used for the fit, and the variables and coefficients used in the fitting. I haven't used Excel/LibreOffice Calc for this, but that does not mean it's not a valid tool also; I'm just unfamiliar with that one. I do know others use it, although Matlab/Octave is even more popular among scientists doing numerical work.)

author: itsbiodiversity
author_rank: Regular Contributor
timestamp: August 19, 2020, 02:51:43 pm
content: Thank you for such a detailed explanation, and I can clearly see how that technique would be superior. Would employing that technique produce similarly effective temperature coefficients? Is there any way to help me identify what those a0-a4 coefficients would be in both of these scenarios. I can apply this to the unit and report back the effects.

author: voltsandjolts
author_rank: Unknown
timestamp: August 19, 2020, 03:11:26 pm
content: In the first post the OP gavepsi °C Counts uut-15 15 5485 -15.017-13 15 5485 -13.014-11 15 5485 -11.011...which are:reference-pressure (psi)reference-temperature (degc)reference-temperature (counts)uut-calibrated-output (psi)Do you have uut-counts (i.e. counts input to the uut-psi-calibration which is currently wrong)?

author: IanB
author_rank: Super Contributor
timestamp: August 19, 2020, 03:17:23 pm
content: Quote from: itsbiodiversity on August 19, 2020, 02:51:43 pmThe way I did it in my post above was using Excel.I created two columns, one with the deviation and one with the correction. The correction is calculated by the correction polynomial. Then I created a third column containing (correction − deviation)^2 containing the square of the error for each measurement. By summing the error squared column I obtained a single cell containing the sum of squared errors.This is the setup phase. In the solve phase I used the Excel solver add-in to minimize the sum of squared errors by varying the polynomial coefficients.

author: itsbiodiversity
author_rank: Regular Contributor
timestamp: August 19, 2020, 03:48:09 pm
content: The UUT counts is used to calibrate to engineering units of psi. The Counts of PSI wasn't used in the author's formula, just the counts for temperature at varying pressures. Thanks for the reply.

author: itsbiodiversity
author_rank: Regular Contributor
timestamp: August 19, 2020, 03:49:33 pm
content: This may be very dense and my apologies as I am new in this area of science (obviously), but in your calculations, what would the a0, a1, a2, a3, and a4 be? I am trying to learn how to identify what those are.

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 19, 2020, 04:31:42 pm
content: The only real difficulty here is getting the picture right. Hopefully, this one will get closer!Quote from: itsbiodiversity on August 19, 2020, 01:38:10 pmYes, it opens in LibreOffice Calc just fine (don't have Windows), but it is difficult to understand what the fieldsmean. Remember, we don't have the context you do.If I've understood you correctly, you have a very high precision pressure sensor. It hasinternalquartic polynomial compensation (via five coefficients) for the pressure sensor at a specific temperature, and you are looking for how tocomputethese coefficients for different temperatures.Let's assume the following:Column A, "Standard", is the known standard pressure being measured.Column B, "Temp", is the temperature at which the measurements are made.Column C, "Temp Counts", is the raw temperature ADC value obtained from the device.Column D, "Reading", is the pressure reported by the device.We can see that the temperature compensation at temperature 21 is correct, i.e. the reported pressure matches the pressure exactly.At lower temperatures, the magnitude of the reading is too large.At higher temperatures, the magnitude of the reading is too small.To me, this indicates that the pressure sensor needs to be calibrated (using five coefficients) to work correctly at a given specific temperature, i.e. that the reported pressure is$$P = P_0 + P_1 x + P_2 x^2 + P_3 x^3 + P_4 x^4$$with the user able to supply \$P_0\$, \$P_1\$, \$P_2\$, \$P_3\$, and \$P_4\$ to the device. It is these coefficients that depend on temperature; each set of five coefficients work at a given temperature. For temperature 21, the coefficients are0 1 0 0 0, i.e. no compensation: \$P = x\$.Most importantly: We are supplying parameters to compensate the sensor behaviourat a specific temperature, not specifying the sensor behaviour at different temperatures.Let's save the first four columns of each temperature to a text file that Gnuplot can eat.temperature-15.txt:Code:#StandardTempTemp CountsReading-15155485-15.017-13155485-13.014-11155485-11.011-9155485-9.01-7155485-7.008-5155485-5.006-3155485-3.004-2155485-2.0020155485021554852.00131554853.00251554855.00471554857.00691554859.0071115548511.0081315548513.0081515548515.008The first and fourth columns are the same for the temperature 21 table, so we can skip that. (It'll just give us a linear function anyway,0 1 0 0 0).Note that Gnuplot ignores lines beginning with a#and those lines are just comments for us humans.temperature-28.txt:Code:#StandardTempTemp CountsReading-15289471-14.99-13289471-12.991-11289471-10.993-9289471-8.994-7289471-6.996-5289471-4.997-3289471-2.998-2289471-1.9990289471022894711.99932894712.99852894714.99772894716.99692894718.9961128947110.9961328947112.9961528947114.996What we want to do, is fit a fourth-degree polynomial on the fourth column, so that it yields the first column. However, we can see that there is no zero drift: when the first column is zero, fourth column is zero also. We can "enforce" that by using a polynomial without a constant coefficient. In Gnuplot:Code:P(p) = P1*p + P2*p*p + P3*p**3 + P4*p**4fit P(x) 'temperature-15.txt' using 4:1 via P1, P2, P3, P4where we use fourth column asx, and result in first column. This outputsCode:iter chisq delta/lim lambda P1 P2 P3 P40 7.4442087182e+09 0.00e+00 1.04e+04 1.000000e+00 1.000000e+00 1.000000e+00 1.000000e+001 2.2294220617e+07 -3.33e+07 1.04e+03 9.985765e-01 9.947791e-01 7.430715e-01 9.651151e-032 3.1706096563e+04 -7.02e+07 1.04e+02 9.945782e-01 9.802179e-01 2.086389e-02 -5.007133e-033 2.5079458204e+03 -1.16e+06 1.04e+01 9.946186e-01 3.960710e-01 1.535787e-05 -2.025147e-034 1.1420062378e-01 -2.20e+09 1.04e+00 9.975663e-01 2.682431e-03 8.654369e-06 -1.362051e-055 4.7602533798e-06 -2.40e+09 1.04e-01 9.990046e-01 1.557076e-05 7.926488e-07 1.567490e-086 4.7485651793e-06 -2.46e+02 1.04e-02 9.990118e-01 1.539077e-05 7.529785e-07 1.659336e-087 4.7485651793e-06 -4.80e-07 1.04e-03 9.990118e-01 1.539077e-05 7.529765e-07 1.659336e-08iter chisq delta/lim lambda P1 P2 P3 P4After 7 iterations the fit converged.final sum of squares of residuals : 4.74857e-06rel. change during last iteration : -4.80261e-12degrees of freedom (FIT_NDF) : 13rms of residuals (FIT_STDFIT) = sqrt(WSSR/ndf) : 0.000604379variance of residuals (reduced chisquare) = WSSR/ndf : 3.65274e-07Final set of parameters Asymptotic Standard Error======================= ==========================P1 = 0.999012 +/- 4.097e-05 (0.004101%)P2 = 1.53908e-05 +/- 4.78e-06 (31.06%)P3 = 7.52977e-07 +/- 2.475e-07 (32.87%)P4 = 1.65934e-08 +/- 2.544e-08 (153.3%)correlation matrix of the fit parameters:P1 P2 P3 P4P1 1.000P2 0.001 1.000P3 -0.917 -0.002 1.000P4 -0.001 -0.961 0.002 1.000which means that at temperature 15, the coefficients we should supply to the device are0 0.999012 0.0000153908 0.000000752977 0.0000000165934Running the same fortemperature-28.txtoutputs (shortened to include only the important bit)Code:Final set of parameters Asymptotic Standard Error======================= ==========================P1 = 1.0006 +/- 2.148e-05 (0.002147%)P2 = -8.84714e-06 +/- 2.51e-06 (28.37%)P3 = -5.92832e-07 +/- 1.301e-07 (21.95%)P4 = -2.39269e-08 +/- 1.339e-08 (55.97%)i.e. the device coefficients needed to get correct pressure readings at temperature 28 are0 1.0006 -0.00000884714 -0.000000592832 -0.0000000239269.The next question is, obviously, how to obtain the five coefficients for any specific temperature. We have three sets of parameters, so we can interpolate between them (and extrapolate outside them) using quadratic interpolation:$$\begin{aligned}p_{15}(x) &= 0.999012 x + 0.0000153908 x^2 + 0.000000752977 x^3 + 0.0000000165934 x^4 \\p_{21}(x) &= x \\p_{28}(x) &= 1.0006 x - 0.00000884714 x^2 - 0.000000592832 x^3 - 0.0000000239269 x^4 \\w_{15}(t) &= 98/13 - 49/78 t + 1/78 t^2 \\w_{21}(t) &= -10 + 43/42 t - 1/42 t^2 \\w_{28}(t) &= 45/13 - 36/91 t + 1/91 t^2 \\p(x, t) &= \frac{ p_{15}(x) w_{15}(t) + p_{21}(x) w_{21}(t) + p_{28}(x) w_{28} t }{ w_{15}(t) + w_{21}(t) + w_{28}(t) }\end{aligned}$$where \$w_{15}(15) = 1\$, \$w_{15}(21) = 0\$, \$w_{15}(28) = 0\$; \$w_{21}(15) = 0\$, \$w_{21}(21) = 1\$, \$w_{21}(28) = 0\$; \$w_{28}(15) = 0\$, \$w_{28}(21) = 0\$, and \$w_{28}(28) = 1\$. These fully define the nine coefficients. It turns out that \$w_{15}(t) + w_{21}(t) + w_{28}(t) = 1\$ for all \$t\$, so we can omit the divisor.When combining polynomials, evaluating the polynomials first and then scaling and summing the results is equivalent to scaling and summing the coefficients first, then evaluating the resulting polynomial. So, with the above, we can immediately describe the quadratic interpolation/extrapolation of the five configuration values to the device as a function of ambient temperature \$t\$:First coefficient is always \$0\$ (because the above fit observed there is no zero drift)Second coefficient is \$0.9946289230769221 + 0.0003833040293040879 t - 0.000006073260073258604 t^2\$Third coefficient is \$0.00008539823846153845 - 0.000006168612014652013 t + 0.00000010009663003663 t^2\$Fourth coefficient is \$0.000003624177384615384 - 0.0000002384970677655678 t + 0.000000003138913919413919 t^2\$Fifth coefficient is \$ 0.00000004226482307692308 - 0.0000000009584721611721613 t - 0.00000000005019706959706964 t^2\$These I obtained by telling Maxima to expand the expression \$p(x, t)\$, and collecting the coefficients for each exponent of \$x\$ and \$t\$.Note that if you had more samples, i.e. more temperatures (and perhaps at more than the 17 pressures), you could model the temperature dependence even better: this is the "best" approximation we can do, that should match the configuration data at the configuration temperatures and pressures, but makes no other assumptions. (Even the quadratic weight polynomials \$w_{15}(t)\$, \$w_{21}(t)\$, and \$w_{28}(t)\$ are completely defined by having them be 1 at that specific temperature, and zero elsewhere.)

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 19, 2020, 05:08:56 pm
content: Quote from: itsbiodiversity on August 19, 2020, 03:49:33 pmI assumed that you are in full control of the data conversion, obtaining raw pressure and temperature values.In other words, I showed the model I believe the device should already have incorporated, with say 5×3 = 15 coefficients.Instead, it looks like the device needs to be configured for a specific ambient temperature. The pressure sensor is factory calibrated to give correct, linear results at 21°C, but the compensation needs to be adjusted for other temperatures.My latter response shows how to calculate the five coefficients for a given temperature (in Celsius centigrade). If you want to change that to the temperature reading (5485 at 15°C, 7386 at 25°C, and 9471 at 28°C), you need to change the weight functions. In Maxima:Code:w15(t) := W00 + W01*t + W02*t^2 $w21(t) := W10 + W11*t + W12*t^2 $w28(t) := W20 + W21*t + W22*t^2 $solve([ w15(5485)=1, w15(7386)=0, w15(9471)=0,w21(5485)=0, w21(7386)=1, w21(9471)=0,w28(5485)=0, w28(7386)=0, w28(9471)=1 ], [ W00,W01,W02, W10,W11,W12, W20,W21,W22 ]);which yieldsCode:[[W00 = 34976403/3788693, W01 = -16857/7577386, W02 = 1/7577386,W10 = -3463229/264239, W11 = 14956/3963585, W12 = -1/3963585,W20 = 1350407/277027, W21 = -12871/8310810, W22 = 1/8310810]]So, we construct those polynomials, and see what \$p(x, t)\$ looks like:Code:w15(t) := 34976403/3788693 - 16857/7577386*t + 1/7577386*t^2 $w21(t) := -3463229/264239 + 14956/3963585*t - 1/3963585*t^2 $w28(t) := 1350407/277027 -12871/8310810*t + 1/8310810*t^2 $If we checkexpand(w15(t) + w21(t) + w28(t));Maxima tells us the result is always 1, so we can omit the divisor. Next, we expand \$p(x, t)\$:Code:p15(x) := 0.999012*x + 0.0000153908*x^2 + 0.000000752977*x^3 + 0.0000000165934*x^4 $p21(x) := x $p28(x) := 1.0006*x - 0.00000884714*x^2 - 0.000000592832*x^3 - 0.0000000239269*x^4 $expand( w15(t)*p15(x) + w21(t)*p21(x) + w28(t)*p28(x) );which yieldsCode:-6.891513009452426E-16*t^2*x^4+1.412920651658525E-13*t*x^4+3.655168497054565E-8*x^4+2.803896602141317E-14*t^2*x^3-7.569847447073744E-10*t*x^3+4.061479723198378E-6*x^3+9.666149779038884E-13*t^2*x^2-2.053746129643471E-8*t*x^2+9.895794611233592E-5*x^2-5.8192843849845E-11*t^2*x+1.268726552948896E-6*t*x+0.9938037796576857*xThe first coefficient given to the device is always 0, \$C_0 = 0\$, because there is no constant term (there was no zero drift in the data). Terms for \$x\$ give the first coefficient,$$C_1 = 0.9938037796576857 +1.268726552948896 \cdot 10^{-6} t - 5.8192843849845\cdot 10^{-11} t^2$$Terms for \$x^2\$ give the third coefficient, and so on, with terms for \$x^4\$ giving the fifth coefficient,$$C_4 = 3.655168497054565\cdot 10^{-8} + 1.412920651658525 \cdot 10^{-13} t - 6.891513009452426 \cdot 10^{-16} t^2$$These use the standard scientific notation, \$\text{1.2e-3} = 1.2 \cdot 10^{-3} = 0.0012\$.Because the device does this compensation, we cannot do the bivariate polynomial. (Why? Odd device. It's not like 15 multiplications and 14 additions is that much more expensive, per pressure reading, than 5 multiplications and 4 additions.)Instead, we must find the quartic polynomial the device can use to do the compensation itself, for each temperature separately. Pretty annoying, but scientifically quite valid, and I kinda know the reasoning why (it keeps policy in the userland, so to speak). Here, the cross terms appear in the interpolated form above (the last one we expanded - the one with terms including powers of \$x\$ and \$t\$), and are "hidden" in the changes of the five coefficients supplied to the device. So, as long as the numerical computation is done with more digits than are correct in the result plus sufficient digits to cover the rounding errors (1 ULP per multiplication or addition – so 4 bits or 1.2 digits per 16 multiplications or additions), this sequential fitting model is just as precise as the single bivariate one would be. So no worries!

author: itsbiodiversity
author_rank: Regular Contributor
timestamp: August 19, 2020, 05:12:51 pm
content: We are so so so so close. I am going to use Ambient temperature and the calculations below. On return to the office I will apply the coefficients and let you know the result. Thank you!

author: itsbiodiversity
author_rank: Regular Contributor
timestamp: August 19, 2020, 06:35:17 pm
content: For all you've done to get me this close I truly appreciate it. I am trying to calculate these formulas provided using t=21 (is that correct). If so I get the following:Coefficient # Calculated (a) Temp0 01 0.993830397 0.99380378 1.27E-06 5.82E-11 21 FORMULA : =0.99380378 + (1.27E-06*21)-(5.82E-11*21^2) = 0.9938303972 9.85E-05 9.90E-05 -2.05E-08 9.67E-13 21 FORMULA : =9.90E-05 + (-2.05E-08*21)+(9.67E-13*21^2) = 9.85E-053 4.05E-06 4.06E-06 -7.57E-10 2.80E-14 21 FORMULA : =4.06E-06 + (-7.57E-10*21)+(2.8E-14*21^2) = 4.05E-064 3.6555E-08 3.66E-08 1.41E-13 -6.89E-16 21 FORMULA : =3.66E-08 + (1.41E-13*21)-(6.89E-16*21^2) = 3.6555E-08I tried entering the seqence a0 = 0, a1 = 0.993830397, a2 = 9.85E-05, a3 = 4.05E-06, and a4 = 3.6555E-08 and the pressure went offscale. I also tried using Counts rather than degrees C and it also went offscale. Am I misunderstanding how to calculate those formulas above? If you're sick of helping I don't blame you!!! I will have another unit in house soon that was programmed by them prior to leaving the organization and am interested to see what is stored. Default is 0 0 1 0 0, and the module meets specifications across temperatures but hoping to "dial it in". I am kind of "pigeon holed" because the inventor/programmer did the design but all information regarding their application of the statistics is gone to the wind.

author: ahbushnell
author_rank: Frequent Contributor
timestamp: August 19, 2020, 08:08:24 pm
content: Quote from: itsbiodiversity on August 19, 2020, 02:51:43 pmSounds like going back to the source of the parameters and getting clarification would be a good idea.

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 19, 2020, 08:59:21 pm
content: Quote from: itsbiodiversity on August 19, 2020, 06:35:17 pmNo, not exactly. Also check if your environment uses ** for exponentiation and ^ for exclusive-or.For temperature \$t\$ in °C, we use the formulaeCode:a0 = 0a1 = 0.9946289230769221 + 0.0003833040293040879 * t - 0.000006073260073258604 * t * ta2 = 0.00008539823846153845 - 0.000006168612014652013 * t + 0.00000010009663003663 * t * ta3 = 0.000003624177384615384 - 0.0000002384970677655678 * t + 0.000000003138913919413919 * t * ta4 = 0.00000004226482307692308 - 0.0000000009584721611721613 * t - 0.00000000005019706959706964 * t * tso for \$t = 21\$ (°C), they area0=0,a1=1,a2=0,a3=0, anda4=0.As an example, for \$t = 24\$ (°C), they are0 +1.000330021978023 -4.992790989010987E-6 -2.91737824175825E-7 -9.652020879120904E-9.For \$t = 18\$(°C), they are0 +0.9995606593406599 +6.794530329670339E-6 +3.482382747252735E-7 +8.748473626373614E-9.The coefficients you specified, best match temperatures \$t \approx -2\$ (°C) or \$t \approx 64\$ (°C), so it is not a surprise the sensor reported pressure out of range, unless the ambient temperature was that extreme (compared to the calibrated range, +15°C to +28°C).If you want to use the temperature counts (\$c\$) instead, then useCode:a0 = 0a1 = 0.9938037796576857 + 1.268726552948896E-6 * c - 5.8192843849845E-11 * c * ca2 = 9.895794611233592E-5 - 2.053746129643471E-8 * c + 9.666149779038884E-13 * c * ca3 = 4.061479723198378E-6 - 7.569847447073744E-10 * c + 2.803896602141317E-14 * c * ca4 = 3.655168497054565E-8 + 1.412920651658525E-13 * c - 6.891513009452426E-16 * c * cAtc = 7386(which corresponds to \$t = 21°C\$), we havea0 = 0,a1 = 0.999999999999997 ≃ 1.000000,a2 = 7.453889935837843e-20 ≃ 0.000000,a3 = -2.117582368135751e-22 ≃ -0.000000, anda4 = 1.985233470127266E-23 ≃ 0.000000.Let's check the math anyways, that's always the best option. I'll use Maxima syntax, but you can adjust it trivially to SageMath, Maple, Mathematica etc.Let's use the temperature counts. The weight functions, the fitted pressure compensation functions, and the combined pressure estimate areCode:w15(t) := 34976403/3788693 - 16857/7577386*t + 1/7577386*t^2 $w21(t) := -3463229/264239 + 14956/3963585*t - 1/3963585*t^2 $w28(t) := 1350407/277027 -12871/8310810*t + 1/8310810*t^2 $p15(x) := 0.999012*x + 0.0000153908*x^2 + 0.000000752977*x^3 + 0.0000000165934*x^4 $p21(x) := x $p28(x) := 1.0006*x - 0.00000884714*x^2 - 0.000000592832*x^3 - 0.0000000239269*x^4 $p(x,t) := w15(t)*p15(x) + w21(t)*p21(x) + w28(t)*p28(x) $The temperature count at \$21°C\$ is \$7386\$, so let's look what the pressure compensation function looks like at that temperature:Code:expand(p(x,7386));It yields simplyx, like it should: no compensation.The temperature count at \$15°C\$ is \$5485\$, andexpand(p(x,5485));yields1.65934E-8*x^4+7.52977E-7*x^3+1.53908E-5*x^2+0.999012*x, which is the same asp15(x):Code:expand(p(x,5485) - p15(x));yields 0.0. Similarly for \$28°C\$ at \$9471\$:expand(p(x,9471)-p28(x))yields 0.So, we know the weighting function works, we haven't goofed there.Let's check some of the calibration values, usingp(x,t);withxfrom the fourth column in the tabulated data,tfrom the third column, and the expected result from the first column. In tabulated form, the results areCode:x │ t │ p(x,t) │ Expectd │ Error─────────┼──────┼────────────┼─────────┼─────────────────-15.017 │ 5485 │ -15.000398 │ -15 │ -0.000398-9.01 │ 5485 │ -9.000290 │ -9 │ -0.000290-3.004 │ 5485 │ -3.000912 │ -3 │ -0.0009120 │ 5485 │ 0 │ 0 │ 03.002 │ 5485 │ 2.999194 │ 3 │ 0.0008069.007 │ 5485 │ 9.000009 │ 9 │ 0.00000915.008 │ 5485 │ 15.000026 │ 25 │ 0.000026─────────┼──────┼────────────┼─────────┼─────────────────-15 │ 7386 │ -15 │ -15 │ 0-9 │ 7386 │ -9 │ -9 │ 0-3 │ 7386 │ -3 │ -3 │ 03 │ 7386 │ 3 │ 3 │ 09 │ 7386 │ 9 │ 9 │ 015 │ 7386 │ 15 │ 15 │ 0─────────┼──────┼────────────┼─────────┼─────────────────-14.99 │ 9471 │ -15.000193 │ -15 │ -0.000193-8.994 │ 9471 │ -8.999837 │ -9 │ 0.000163-2.998 │ 9471 │ -2.999864 │ -3 │ 0.0001360 │ 9471 │ 0 │ 0 │ 02.998 │ 9471 │ 2.999701 │ 3 │ -0.0002998.996 │ 9471 │ 9.000093 │ 9 │ 0.00009314.996 │ 9471 │ 14.999799 │ 15 │ -0.000201From this, we can see that the absolute errors are less than 0.001. The spreadsheet says tolerance is 0.035% of full range, which corresponds to 30×0.035/100 = 0.0105. Using the function shown above, the results are correct (compared to calibration values) with less than 0.001/30×100% = 0.0033% of (absolute) error.Let's verify the coefficient functions I gave earlier, for the temperature counts:Code:A4(t) := -6.891513009452426E-16*t^2+1.412920651658525E-13*t+3.655168497054565E-8 $A3(t) := +2.803896602141317E-14*t^2-7.569847447073744E-10*t+4.061479723198378E-6 $A2(t) := +9.666149779038884E-13*t^2-2.053746129643471E-8*t+9.895794611233592E-5 $A1(t) := -5.8192843849845E-11*t^2+1.268726552948896E-6*t+0.9938037796576857 $A0(t) := 0 $q(x, t) := A0(t) + A1(t)*x + A2(t)*x^2 + A3(t)*x^3 + A4(t)*x^4 $and repeating the table, but usingq(x, t)instead ofp(x, t), we see that the results are the same. In fact,expand(p(x,t) - q(x,t));gives a polynomial with coefficients of the order 10-21to 10-29, which is close enough to zero for us to say the two polynomials are the same to within the precision we represent them.Ifthe correctly calculated coefficients – please don't round them, all digits matter – still produce erroneous pressures, then a core assumption I made is incorrect.I assume the formula the device uses to report the pressure is \$P(p) = C_0 + C_1 p + C_2 p^2 + C_3 p^3 + C_4 p^4\$ (with \$p\$ the internal factory-calibrated sensor value, and \$P(p)\$ the pressure it provides to the user). (Mainly, because the defaults being0 1 0 0 0, it would make sense for it to correspond to \$P(p) = p\$. However,ifOP is not getting sane results, this assumption is wrong.)If we look at the spreadsheet, it has a separate cell for "constant" (zero – equal to my assumption about no zero drift!), and four X coefficients, but it is thesecondX coefficient (corresponding to \$C_2\$). Could it be that the coefficients the device wants arefor the temperatureand not the pressure? That it provides \$P(p, t)\$ to the user, internally calculating the pressure as \$P(p, t) = p + T_0 + T_1 t + T_2 t^2 + T_3 t^3 + T_4 t^4\$, with \$T_0\$ to \$T_4\$ being the four coefficients we can specify? Well, no. You see, if that was the case, we could not get \$P(p_{-}, t) = -15\$, \$P(0, t) = 0\$, and \$P(p_{+},t) = +15\$ for a constant \$t\$ but \$p_{-} \ne -p_{+}\$.In fact, I cannot imagine right now what the exact formula for \$P(p, t)\$ the device uses, if not \$C_0 + C_1 p + C_2 p^2 + C_3 p^3 + C_4 p^4\$. This does not mean it cannot be anything else; just that it is something not immediately obvious. Perhaps \$C_0 + C_1 p + C_2 p^2 + C_3 t + C_4 t^2\$ or something? I don't know, and I don't see anything specific in the spreadsheet that would help us find what it is.Quote from: itsbiodiversity on August 19, 2020, 06:35:17 pmI know how that feels. I'd like to help one team to measure pressure in a fusor, and temperature of the vacuum cask, but I seem to be the only one willing to use non-off-the-shelf parts and non-Raspberry Pi SBCs, which makes everything harder. (Although I do hear that the "I shall randomly drop USB packets because I'm overburdened right now" bugs havemostlyworked around nowadays.)Modular hardware you can build and test in units (with calibration voltage sources, compare to known good measurements) and in increasingly complex combinations, but if you use COTS hardware with LabView-only interfaces or arse-backwards 'Pi libraries that do not even check for short reads and writes, you can only hope stuff works for the duration of the experiment. (On the other hand, when something goes wrong, you get to blame other people, and not tarnish your own image.)Me, I like to make robust, reliable stuff on the cheap.

author: IanB
author_rank: Super Contributor
timestamp: August 19, 2020, 09:45:57 pm
content: Quote from: itsbiodiversity on August 19, 2020, 03:49:33 pmI'm having trouble following all the details Nominal Animal is posting, but let's simplify it a bit.There is some difference between the actual pressure reading (call it P) and the assumed true pressure (call it P*). So there is an offset between P and P* (call it deltaP).So we have:deltaP = P - P*Looking at the data table, it is clear that deltaP dependsbothon T and P.So we need to fit a correlating function of the form:deltaP = f(P, T)We might assume, for example:deltaP ~ (T-Tref) * (a0 + a1 * P + a2 * P * (T-Tref) + a3 * P^2 + a4 * P^2 * (T-Tref)) (say)You have to know the actual polynomial assumed by the designer of the equipment. What I wrote here is just for instance.Then your job is to find values of a0..a4 that best fit the data in the table. You can use Excel for that, or any other tool of your choice.To simplify matters I chose to use T-Tref because the offset appears to be zero at Tref = 21°C. But really, Tref is just another coefficient in the polynomial.At the end of the day, the only way to know what a0..a4 are is to know the form of the interpolating polynomial, and the designer has to tell you that.

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 20, 2020, 12:18:39 pm
content: Yeah, sorry about the excess verbosity, everyone.I basically just explained, step by step, how to obtain \$P(p) = P_1(t) p + P_2(t) p^2 + P_3(t) p^3 + P_4(t) p^4\$, where \$p\$ is the device-internal pressure calibrated to be correct at \$t = 21°C\$ (which corresponds to \$c = 7386\$), and how to derive \$P_1(t)\$, \$P_2(t)\$, \$P_3(t)\$, and \$P_4(t)\$ from the samples at \$t = 15°C\$, \$t = 21°C\$, and \$t = 28°C\$. (And the same for using temperature counts \$c\$ instead of temperature \$t\$.)My core assumption is/was that the five (\$P_0(t) = P_0(c) = 0\$, as there is no zero drift in the calibration data) are coefficients for powers of \$p\$, but that may not be true. It could be \$P(p, t) = P_0 + P_1 p + P_2 p^2 + P_3 t + P_4 t^2\$, or \$P(p, c) = P_0 + P_1 p + P_2 p^2 + P_3 c + P_3 c^2\$, or something else entirely.The spreadsheet sorta-kinda suggests the last one (\$P(p,c)\$) – it has no formulae for the derivation of the five coefficients used, just using the column labels as hints –, but \$P(p)\$ would be the most logical one, considering how the device is used.Actually, if we look at the spreadsheet, a possible form is \$P(p, c) = A_0 p^2 + A_1 p + A_2 c^2 + A_3 c + A_4\$. If so, the coefficients fitted to the calibration data are0.000000669808265654805 0.999876931661194 -0.0000000000881848278027671 0.000000710797927540773 0If these produce sane/good results, I'll be happy to show how to obtain these using the dataset and Gnuplot, although it is almost trivial.

author: itsbiodiversity
author_rank: Regular Contributor
timestamp: August 20, 2020, 01:34:14 pm
content: HOLY CRAP MAN!!! That was close. It actually did not cause the positive pressure linearity to vary from what I set as the "ambient" linearity; however, it caused the formerly near perfect "ambient" negative pressure values to change near identical to the un-adjust reading in Column D (-15.017 psi at a standard of -15 psi). Basically it went "backwards" to the unadjusted cold temperature value even though I am operating at ambient. Does this explanation help guide to a portion of the formula??? I am very interested how you got to this point! Of course the negative pressure was the last data point I took I thought you had it.

author: Jay_Diddy_B
author_rank: Super Contributor
timestamp: August 20, 2020, 03:15:41 pm
content: Hi,It make intuitive sense.You have a two second order polynomials, one for temperature and one for pressure.The constant term A4is the sum of the constant terms in each of the second order polynomials.On the surface plot that i shared no correction is needed at Temp = 21 (all pressures) or pressure = 0 (all temperatures)Jay_Diddy_B

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 20, 2020, 06:58:10 pm
content: Let's recap. We have five coefficients that can be defined in the device. Let's call thesea0a1a2a3a4.We do not know what these coefficients are, exactly, but we want to find out. Because of the default, we can be almost certain thata1is the first-degree coefficient for the pressure.Let's assume the internal compensation function isP(p, c) = P2p2+ P1p + C2c2+ C1c + Oo. Then, we can assume thata1= P1, i.e.P1is always the second one. The simplest option, then, would be to try the most likely possibilities, and see if they make sense. Here's the ones I would try:0.000000669808265654805 0.999876931661194 -0.0000000000881848278027671 0.000000710797927540773 0This corresponds toP2P1C2C1Oo, but we already know this didn't work.0 0.999876931661194 0.000000669808265654805 0.000000710797927540773 -0.0000000000881848278027671This corresponds toOoP1P2C1C2.0 0.999876931661194 0.000000710797927540773 0.000000669808265654805 -0.0000000000881848278027671This corresponds toOoP1C1P2C2.0.000000669808265654805 0.999876931661194 0 -0.0000000000881848278027671 0.000000710797927540773This corresponds toP2P1OoC2C1.0.000000669808265654805 0.999876931661194 0 0.000000710797927540773 -0.0000000000881848278027671This corresponds toP2P1OoC1C2.0.000000710797927540773 0.999876931661194 -0.0000000000881848278027671 0.000000669808265654805 0This corresponds toC1P1C2P2O0.There are a total of 24 possible combinations withP1in the second position, but I think there has to be some kind of logic in them, not just a random jumble, and the above ones are the ones I would test.If anyone has additional suggestions I didn't immediately see, please pipe up! Me err often, me be uncle bumblespork.(I don't justlikepeer review; my outputrequiresit.)Another option is to do ten sets of samples at different temperatures and pressures, each set with a specific coefficient set:Code:0 1 0 0 0+0.00000001 0 0 0 0-0.00000001 0 0 0 00 0 +0.00000001 0 00 0 -0.00000001 0 00 0 0 +0.00000001 00 0 0 -0.00000001 00 0 0 0 +0.000000010 0 0 0 -0.00000001Instead of ±0.00000001, it would be better to use as large value in magnitude as possible, without error in the reported pressure.Note that each of the ten sets are independent: they don't need to have samples at the same temperature and pressure, as long as there are at least five measurements, at at least three different pressures and three different temperatures. You can either do a sweep of temperature or pressure, or measure a specific pressure and temperature using each of the ten sets of parameters.The idea is that when we have a table with eight columns – the five coefficients used, the temperature (count) and pressure received, and the actual pressure being measured –, we can deduce the exact algebraic form of the compensation function the device uses. For each set of five coefficients, we do need samples at at least three different temperatures and three different pressures, so that we can differentiate between linear terms and quadratic terms. (If we suspect there are third degree terms, or higher, we need correspondingly more samples to deduce the exponent/power.)

author: IanB
author_rank: Super Contributor
timestamp: August 20, 2020, 07:12:05 pm
content: Quote from: Nominal Animal on August 20, 2020, 06:58:10 pmUnfortunately, it doesn't seem like the meaning of a0..a4 should be unknown. If we look at the OP's earlier posts, there is apparently access to the inventor of the device, and there is access to a manual. This should not be a mystery. Either the inventor or the manual should be able to say, "This is the form of the correction polynomial programmed into the device."I am perplexed that not once has the OP provided this information in plain text. It would be simple to answer if only that information were given.Quote from: itsbiodiversity on August 19, 2020, 08:10:52 amQuote from: itsbiodiversity on August 19, 2020, 03:48:09 pm

author: itsbiodiversity
author_rank: Regular Contributor
timestamp: August 20, 2020, 08:58:26 pm
content: My apologies having issues quoting. But to answer a few questions. First off the person I spoke to initially isn't responding anymore. Here is verbiage concerning the coefficients from the admin manual.Enter Temperature CoefficientsThis command provides for the entry of temperature coefficients that will compensate the sensor for ambient temperature conditions. Thecoefficients are determined during the factory calibration process.tcomp <a0> <a1> <a2> <a3> <a4>The operator specifies five coefficients, which are used in a fourth order polynomial that corrects temperature readings for the ambient temperature at the sensor.From the cal procedure:"Take your recorded readings from the 3 temperatures and enter them into the polynomial linearization program to get your coefficients."The "linearization program" I am told is a Quattro spreadsheet that I have access to. I am not able to upload the spreadsheet here or I would. I am attaching that pasted in excel to this comment. In Quattro I can see the following is used, and to great effect actually, to normalize the pressure readings on the UUT to match the standard.+$A:$N$11*C7+$A:$O$11*D7+$A:$P$11*E7+$A:$Q$11*F7where N11 is an x coefficient, C7 is the temperature counts, O11 is another x coefficient, D7 is the UUT PSI reading, P11 is the third x coefficient, E7 is Temperature Counts (C7) x UUT PSI reading (D7), Q11 is the fourth x coefficient, and F7 is temperature counts squared (C7 x C7) multiplied by UUT PSI reading (D7).The x coefficients referenced above are the four x coefficients in ZERO regression output left to right.This is all of the information I have - as it appears this team abandoned the project at some point and I'm working off of what they did. My apologies for not having further information - it has been frustrating trying to get information from them. You've all helped more than you know already.In the attached file I went ahead and wrote the equation for correction to column G labeled CORRECTED so that one can see what is going on. The confusion I am having is what within the simple formula is considered a0 through a4. Does this help at all?

author: itsbiodiversity
author_rank: Regular Contributor
timestamp: August 20, 2020, 09:11:22 pm
content: I just received this info and I hope this helps.Here is how the unit uses the coefficients:"/* 3.2 - Function to Apply Temperature Compensation */float ApplyCompensation (float rdg, int tempAdc){float tmp = (float)tempAdc;float tmp2 = tmp * tmp;float limit, result;/* Limit the result to comform to the linearization table */limit = tblCnt ? fabs(calTbl[tblCnt-1].value) * 1.5 : 17.0;/* Adjust the pressure reading using temperture polynomial */result = coefTbl+ (coefTbl * tmp)+ (rdg * (coefTbl + (coefTbl * tmp) + (coefTbl * tmp2)));/* Don't allow the result to stray too far from the table's limits */if (result < -limit)result = -limit;else if (result > limit)result = limit;/* Provide the temperature compensated pressure reading */return (result);}You may not be a connoisseur of the C language, so the distilled version is:result = a0 + (a1 * tmp) + X * (a2 + (a3 * tmp) + (a4 * tmp2 ))where: tmp is temperature ADC and X is the pressure readingI seem to recall that A0 was usually zero".

author: itsbiodiversity
author_rank: Regular Contributor
timestamp: August 20, 2020, 09:58:01 pm
content: I have just successfully programmed the first of these units thanks to everyone's help on this thread.In the end, for a similar unit to the one data was provided for the coefficients were:a0 = 0a1 = 6.18E-10a2 = 0.9938a3 = 1.16E-06a4 = -5.43749E-11a2 has a large effect on the unit. Once these were established I reprogrammed certain lines of the calibration table and the unit successfully met the 0.011 psi requirement. I am going to be diving more in to all of the information provided thus far to gain a better understanding of this type of statistics. Again - thanks. And if anyone has any input on these coefficients I am interested in what your thoughts are.

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 20, 2020, 09:59:15 pm
content: Yes! This is the key:result = a0 + (a1 * tmp) + X * (a2 + (a3 * tmp) + (a4 * tmp2 ))wheretmpis the temperature count andtmp2is that squared, andXis the uncorrected pressure reading.Using \$c\$ for temperature counts (\$c = 7386\$ corresponding to 21°C), and \$p\$ for the uncorrected pressure reading, the correction function is$$P(p, c) = a_0 + a_1 c + a_2 p + a_3 c p + a_4 c^2 p = a_0 + a_1 c + p (a_2 + a_3 c + a_4 c^2 )$$Using Gnuplot with the four-column calibration data (all three sets) indata.txt,Code:P(p, c) = a0 + a1*c + p*(a2 + a3*c + a4*c*c)fit P(x,y) 'data.txt' using 4:3:1 via a0, a1, a2, a3, a4print a0, a1, a2, a3, a4gives us a nice set of coefficients:0.00504498959142752 -6.45913664596841e-07 0.994401298866226 1.15958714191414e-06 -5.43693236648548e-11However, keepinga0zero since there is no zero drift makes still sense, soCode:P(p, c) = a1*c + p*(a2 + a3*c + a4*c*c)fit P(x,y) 'data.txt' using 4:3:1 via a1, a2, a3, a4print 0, a1, a2, a3, a4gives us the set0 6.18461239919483e-10 0.994400954117506 1.15967494730689e-06 -5.4374892229142e-11Could you try this one, itsbiodiversity?(You can useprint P(-15.017,5485)(fourth column, third column, compare result to first column) to see how well the compensation matches the calibration dataset. I'll omit the comparisons for now, to keep my posts more readable!)

author: itsbiodiversity
author_rank: Regular Contributor
timestamp: August 20, 2020, 10:10:13 pm
content: I just disconnected after battling with this one for days! I have four more to program tomorrow and will let you know how those go using the coefficients you provided. Thanks again!

author: itsbiodiversity
author_rank: Regular Contributor
timestamp: August 21, 2020, 11:27:48 am
content: Those are the coefficients NOMINAL ANIMAL. Exactly. Now I need to figure out how to calculate those myself based on any dataset using excel. I tried to download Gnuplot but had some issues installing.

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 21, 2020, 12:46:38 pm
content: Well, instead of a linear regression, you could do a brute-force (kinda-sorta gradient descent) numerical search that minimizes the largest error in magnitude at calibration samples.Given a text file with the 51 samples (each line has one sample:actual_pressure temperature_count measured_pressureobtained using default compensation coefficients), i.e.Code:-15 5485 -15.017-13 5485 -13.014-11 5485 -11.011-9 5485 -9.01-7 5485 -7.008-5 5485 -5.006-3 5485 -3.004-2 5485 -2.0020 5485 02 5485 2.0013 5485 3.0025 5485 5.0047 5485 7.0069 5485 9.00711 5485 11.00813 5485 13.00815 5485 15.008-15 7386 -15-13 7386 -13-11 7386 -11-9 7386 -9-7 7386 -7-5 7386 -5-3 7386 -3-2 7386 -20 7386 02 7386 23 7386 35 7386 57 7386 79 7386 911 7386 1113 7386 1315 7386 15-15 9471 -14.99-13 9471 -12.991-11 9471 -10.993-9 9471 -8.994-7 9471 -6.996-5 9471 -4.997-3 9471 -2.998-2 9471 -1.9990 9471 02 9471 1.9993 9471 2.9985 9471 4.9977 9471 6.9969 9471 8.99611 9471 10.99613 9471 12.99615 9471 14.996such a program reports something likeCode:Maximum error minimized to 0.002420993, using0.007697997673805127 -9.893590127282349e-07 0.9937816855224063 1.353571043339379e-06 -6.784307221818112e-11p | c | P(p, c) | P | Absolute error--------+------+---------------+-----+------------------15.017 | 5485 | -15.002188670 | -15 | -0.002188669678-13.014 | 5485 | -13.000861278 | -13 | -0.000861277627-11.011 | 5485 | -10.999533886 | -11 | +0.000466114424-9.010 | 5485 | -9.000204823 | -9 | -0.000204823422-7.008 | 5485 | -6.999876596 | -7 | +0.000123403681-5.006 | 5485 | -4.999548369 | -5 | +0.000451630784-3.004 | 5485 | -2.999220142 | -3 | +0.000779857886-2.002 | 5485 | -1.998056864 | -2 | +0.0019431363860.000 | 5485 | 0.002271363 | 0 | +0.0022713634892.001 | 5485 | 2.001600426 | 2 | +0.0016004256433.002 | 5485 | 3.001764539 | 3 | +0.0017645391945.004 | 5485 | 5.002092766 | 5 | +0.0020927662977.006 | 5485 | 7.002420993 | 7 | +0.0024209934009.007 | 5485 | 9.001750056 | 9 | +0.00175005555411.008 | 5485 | 11.001079118 | 11 | +0.00107911770813.008 | 5485 | 12.999409015 | 13 | -0.00059098508615.008 | 5485 | 14.997738912 | 15 | -0.002261087881-15.000 | 7386 | -15.000781184 | -15 | -0.000781184012-13.000 | 7386 | -13.000624947 | -13 | -0.000624947209-11.000 | 7386 | -11.000468710 | -11 | -0.000468710407-9.000 | 7386 | -9.000312474 | -9 | -0.000312473605-7.000 | 7386 | -7.000156237 | -7 | -0.000156236802-5.000 | 7386 | -5.000000000 | -5 | -0.000000000000-3.000 | 7386 | -2.999843763 | -3 | +0.000156236802-2.000 | 7386 | -1.999765645 | -2 | +0.0002343552030.000 | 7386 | 0.000390592 | 0 | +0.0003905920062.000 | 7386 | 2.000546829 | 2 | +0.0005468288083.000 | 7386 | 3.000624947 | 3 | +0.0006249472095.000 | 7386 | 5.000781184 | 5 | +0.0007811840127.000 | 7386 | 7.000937421 | 7 | +0.0009374208149.000 | 7386 | 9.001093658 | 9 | +0.00109365761611.000 | 7386 | 11.001249894 | 11 | +0.00124989441913.000 | 7386 | 13.001406131 | 13 | +0.00140613122115.000 | 7386 | 15.001562368 | 15 | +0.001562368023-14.990 | 9471 | -14.999404724 | -15 | +0.000595275661-12.991 | 9471 | -12.999373552 | -13 | +0.000626447983-10.993 | 9471 | -11.000342896 | -11 | -0.000342895540-8.994 | 9471 | -9.000311723 | -9 | -0.000311723218-6.996 | 9471 | -7.001281067 | -7 | -0.001281066740-4.997 | 9471 | -5.001249894 | -5 | -0.001249894418-2.998 | 9471 | -3.001218722 | -3 | -0.001218722096-1.999 | 9471 | -2.001703394 | -2 | -0.0017033938580.000 | 9471 | -0.001672222 | 0 | -0.0016722215361.999 | 9471 | 1.998358951 | 2 | -0.0016410492142.998 | 9471 | 2.997874279 | 3 | -0.0021257209754.997 | 9471 | 4.997905451 | 5 | -0.0020945486536.996 | 9471 | 6.997936624 | 7 | -0.0020633763318.996 | 9471 | 8.998968312 | 9 | -0.00103168816610.996 | 9471 | 11.000000000 | 11 | +0.00000000000012.996 | 9471 | 13.001031688 | 13 | +0.00103168816614.996 | 9471 | 15.002063376 | 15 | +0.002063376332i.e., if you use0.007697997673805127 -9.893590127282349e-07 0.9937816855224063 1.353571043339379e-06 -6.784307221818112e-11the compensated pressure reading is within ±0.002421 of actual pressure at every calibration point. This is just 0.008033% of the full pressure range. In other words, the absolute error is less than 1/12000th of the full range.I wrote such a program in C99 (or later) – edited to include a progress estimate:Code:// SPDX-License-Identifier: CC0-1.0// This file is in Public Domain.#include <stdlib.h>#include <stdint.h>#include <string.h>#include <stdio.h>#include <errno.h>#include <math.h>/* Each configuration is an array of five doubles. */typedef struct {double arg;} config;#define ROW_FORMAT "%14.11f %18.15f %12.9f %17.14f %22.19f"#define ERROR_FACTOR 1.50/* Calibration data entry */struct sample {double actual_pressure;double temperature_count;double measured_pressure;};static size_t max_samples = 0;static size_t num_samples = 0;static struct sample *samples = NULL;static inline void add_sample(const double actual_pressure,const double temperature_count,const double measured_pressure){/* Check if already known. */size_t i = num_samples;while (i-->0)if (samples.actual_pressure == actual_pressure &&samples.temperature_count == temperature_count &&samples.measured_pressure == measured_pressure)return;/* Grow sample buffer whenever necessary. */if (num_samples >= max_samples) {const size_t new_max = (num_samples | 1023) + 1025 - 8;void *new_ptr;new_ptr = realloc(samples, new_max * sizeof samples);if (!new_ptr) {fprintf(stderr, "Warning: Out of memory (%zu calibration samples)\n", num_samples);return;}max_samples = new_max;samples = new_ptr;}/* Add sample. */samples.actual_pressure = actual_pressure;samples.temperature_count = temperature_count;samples.measured_pressure = measured_pressure;++num_samples;}/* Returns the magnitude of the maximum absolute error for a given config, and the direction of the steepest descent. */static double error_magnitude_and_descent(const config given, config *const descent){config gradient = {{ 0.0, 0.0, 0.0, 0.0, 0.0 }};double result = 0.0;size_t i;for (i = 0; i < num_samples; i++) {const double p = samples.measured_pressure;const double c = samples.temperature_count;const double actual = samples.actual_pressure;const double reported = given.arg + given.arg*c+ p*(given.arg + c*(given.arg + c*given.arg));const double err = fabs(reported - actual);if (result < err) {result = err;gradient.arg = 1.0;gradient.arg = c;gradient.arg = p;gradient.arg = p*c;gradient.arg = p*c*c;}}if (descent) {const double scale = -result / ( gradient.arg*gradient.arg+ gradient.arg*gradient.arg+ gradient.arg*gradient.arg+ gradient.arg*gradient.arg+ gradient.arg*gradient.arg );descent->arg = gradient.arg * scale;descent->arg = gradient.arg * scale;descent->arg = gradient.arg * scale;descent->arg = gradient.arg * scale;descent->arg = gradient.arg * scale;}return result;}/* Returns the magnitude of the maximum absolute error for given coefficients. */static double error_magnitude(const config given){double result = 0.0;size_t i;for (i = 0; i < num_samples; i++) {const double p = samples.measured_pressure;const double c = samples.temperature_count;const double actual = samples.actual_pressure;const double reported = given.arg + given.arg*c+ p*(given.arg + c*(given.arg + c*given.arg));const double err = fabs(reported - actual);if (result < err)result = err;elseif (result < -err)result = -err;}return result;}/* Point cache for dense minima. */static size_t max_points = 0;static size_t num_points = 0;static config *points = NULL;static inline void push_point(const config point){if (num_points >= max_points) {const size_t new_max = (num_points | 4095) + 4097 - 8;void *new_ptr;new_ptr = realloc(points, new_max * sizeof points);if (!new_ptr) {fprintf(stderr, "Warning: Out of memory (%zu points).\n", num_points);fflush(stderr);return;}max_points = new_max;points = new_ptr;}points = point;++num_points;}static inline int pop_point(config *const to){if (num_points < 1)return 0;--num_points;if (to) *to = points;return 1;}/* Known best configuration and error magnitude. Updated after each walk. */static config best_point = {{ 0, 0, 0, 0, 0 }};static double best_error = -1.0;/* Find local minima. */static void walk(config const from){config min_point, max_point, the_point, local_point, descent;double min_error, max_error, the_error, local_error, req_error;size_t n, nmax;min_point = from;min_error = error_magnitude(min_point);push_point(min_point);local_point = min_point;local_error = min_error;nmax = 1;while (pop_point(&min_point)) {min_error = error_magnitude_and_descent(min_point, &descent);req_error = min_error;max_point.arg = min_point.arg + descent.arg;max_point.arg = min_point.arg + descent.arg;max_point.arg = min_point.arg + descent.arg;max_point.arg = min_point.arg + descent.arg;max_point.arg = min_point.arg + descent.arg;max_error = error_magnitude(max_point);while (1) {the_point.arg = max_point.arg + descent.arg;the_point.arg = max_point.arg + descent.arg;the_point.arg = max_point.arg + descent.arg;the_point.arg = max_point.arg + descent.arg;the_point.arg = max_point.arg + descent.arg;the_error = error_magnitude(the_point);if (the_error >= max_error)break;max_point = the_point;max_error = the_error;}if (min_error > max_error) {the_point = max_point; the_error = max_error;max_point = min_point; max_error = min_error;min_point = the_point; min_error = the_error;}if (local_error > min_error) {local_error = min_error;local_point = min_point;}/* Binary search for minimum between curr_point and curr_point+descent. */n = 64; /* Number of iterations */while (n-->0) {the_point.arg = (double)(0.5*min_point.arg) + (double)(0.5*max_point.arg);the_point.arg = (double)(0.5*min_point.arg) + (double)(0.5*max_point.arg);the_point.arg = (double)(0.5*min_point.arg) + (double)(0.5*max_point.arg);the_point.arg = (double)(0.5*min_point.arg) + (double)(0.5*max_point.arg);the_point.arg = (double)(0.5*min_point.arg) + (double)(0.5*max_point.arg);the_error = error_magnitude(the_point);if (min_error >= the_error) {min_error = the_error;min_point = the_point;} elseif (max_error >= the_error) {max_error = the_error;max_point = the_point;} elseif (max_error < req_error) {push_point(max_point);max_error = the_error;max_point = the_point;}}if (nmax < num_points)nmax = num_points;}fprintf(stderr, "\r" ROW_FORMAT " %.9f [%zu] ",local_point.arg, local_point.arg,local_point.arg, local_point.arg,local_point.arg, local_error, nmax);if (best_error > local_error || best_error < 0.0) {best_error = local_error;best_point = local_point;fprintf(stderr, "\n");}fflush(stderr);}static inline size_t unique5(const double v0, const double v1, const double v2, const double v3, const double v4){size_t result = 5;if (v1 == v0) result--;if (v2 == v0 || v2 == v1) result--;if (v3 == v0 || v3 == v1 || v3 == v2) result--;if (v4 == v0 || v4 == v1 || v4 == v2 || v4 == v3) result--;return result;}/* This solves the system of equationsa0 = X0 + X1*c0 + p0*(X2 + c0*(X3 + c0*X4))a1 = X1 + X1*c1 + p1*(X2 + c1*(X3 + c1*X4))a2 = X2 + X1*c2 + p2*(X2 + c2*(X3 + c2*X4))a3 = X3 + X1*c3 + p3*(X2 + c3*(X3 + c3*X4))a4 = X4 + X1*c4 + p4*(X2 + c4*(X3 + c4*X4))If a solution exists, the function returns 1, with X0,X1,X2,X3,X4 in *to.Otherwise, the function returns 0.*/static int solve(const double p0, const double p1, const double p2, const double p3, const double p4,const double c0, const double c1, const double c2, const double c3, const double c4,const double a0, const double a1, const double a2, const double a3, const double a4,config *const to){config result;const double c00 = c0 * c0, c01 = c0 * c1, c02 = c0 * c2, c03 = c0 * c3, c04 = c0 * c4,c11 = c1 * c1, c12 = c1 * c2, c13 = c1 * c3, c14 = c1 * c4,c22 = c2 * c2, c23 = c2 * c3, c24 = c2 * c4,c33 = c3 * c3, c34 = c3 * c4,c44 = c4 * c4;const double c001 = c00 * c1,c002 = c00 * c2,c003 = c00 * c3,c004 = c00 * c4,c011 = c11 * c0,c022 = c22 * c0,c033 = c33 * c0,c044 = c44 * c0,c112 = c11 * c2,c113 = c11 * c3,c114 = c11 * c4,c122 = c22 * c1,c133 = c33 * c1,c144 = c44 * c1,c223 = c22 * c3,c224 = c22 * c4,c233 = c33 * c2,c244 = c44 * c2,c334 = c33 * c4,c344 = c44 * c3;const double c0012 = c00 * c12,c0013 = c00 * c13,c0014 = c00 * c14,c0023 = c00 * c23,c0024 = c00 * c24,c0034 = c00 * c34,c0112 = c11 * c02,c0113 = c11 * c03,c0114 = c11 * c04,c0122 = c01 * c22,c0133 = c01 * c33,c0144 = c01 * c44,c0223 = c03 * c22,c0224 = c04 * c22,c0233 = c02 * c33,c0244 = c02 * c44,c0334 = c04 * c33,c0344 = c03 * c44,c1123 = c11 * c23,c1124 = c11 * c24,c1134 = c11 * c34,c1223 = c13 * c22,c1224 = c14 * c22,c1233 = c12 * c33,c1244 = c12 * c44,c1334 = c14 * c33,c1344 = c13 * c44,c2234 = c22 * c34,c2334 = c24 * c33,c2344 = c23 * c44;const double p01 = p0 * p1, p02 = p0 * p2, p03 = p0 * p3, p04 = p0 * p4,p12 = p1 * p2, p13 = p1 * p3, p14 = p1 * p4,p23 = p2 * p3, p24 = p2 * p4,p34 = p3 * p4;const double p012 = p01 * p2,p013 = p01 * p3,p014 = p01 * p3,p023 = p02 * p3,p024 = p02 * p4,p034 = p03 * p4,p123 = p12 * p3,p124 = p12 * p4,p134 = p13 * p4,p234 = p23 * p4;const double divisor = (c1224 - c0224 - c1124 + c0024 + c0114 - c0014 - c1223 + c0223 + c1123 - c0023 - c0113 + c0013) * p012- (c0012 + c1334 - c0334 - c1134 + c0034 + c0114 - c0014 - c1233 + c0233 + c1123 - c0023 - c0112) * p013+ (c0012 + c1344 - c0344 - c1244 + c0244 - c1134 + c0034 + c1124 - c0024 + c0113 - c0013 - c0112) * p014+ (c0012 + c2334 - c0334 - c2234 + c0034 + c0224 - c0024 - c1233 + c0133 + c1223 - c0013 - c0122) * p023- (c0012 + c2344 - c0344 - c1244 + c0144 - c2234 + c0034 + c1224 - c0014 + c0223 - c0023 - c0122) * p024+ (c2344 - c1344 - c0244 + c0144 - c2334 + c1334 + c0024 - c0014 + c0233 - c0133 - c0023 + c0013) * p034- (c2334 - c1334 - c2234 + c1134 + c1224 - c1124 - c0233 + c0133 + c0223 - c0113 - c0122 + c0112) * p123+ (c2344 - c1344 - c0244 + c0144 - c2234 + c1134 + c0224 - c0114 + c1223 - c1123 - c0122 + c0112) * p124- (c2344 - c0344 - c1244 + c0144 - c2334 + c0334 + c1124 - c0114 + c1233 - c0133 - c1123 + c0113) * p134+ (c1344 - c0344 - c1244 + c0244 - c1334 + c0334 + c1224 - c0224 + c1233 - c0233 - c1223 + c0223) * p234;if (divisor == 0.0)return 0;result.arg = ( ( a4*c0013 - a3*c0014 - a4*c0023 + a3*c0024 - a4*c0113 + a3*c0114 + a4*c0223 - a3*c0224 + a4*c1123 - a3*c1124 - a4*c1223 + a3*c1224) * p012+ ( -a4*c0012 + a2*c0014 + a4*c0023 - a2*c0034 + a4*c0112 - a2*c0114 - a4*c0233 + a2*c0334 - a4*c1123 + a2*c1134 + a4*c1233 - a2*c1334) * p013+ ( a3*c0012 - a2*c0013 - a3*c0024 + a2*c0034 - a3*c0112 + a2*c0113 + a3*c0244 - a2*c0344 + a3*c1124 - a2*c1134 - a3*c1244 + a2*c1344) * p014+ ( a4*c0012 - a4*c0013 - a1*c0024 + a1*c0034 - a4*c0122 + a4*c0133 + a1*c0224 - a1*c0334 + a4*c1223 - a4*c1233 - a1*c2234 + a1*c2334) * p023+ ( -a3*c0012 + a3*c0014 + a1*c0023 - a1*c0034 + a3*c0122 - a3*c0144 - a1*c0223 + a1*c0344 - a3*c1224 + a3*c1244 + a1*c2234 - a1*c2344) * p024+ ( a2*c0013 - a2*c0014 - a1*c0023 + a1*c0024 - a2*c0133 + a2*c0144 + a1*c0233 - a1*c0244 + a2*c1334 - a2*c1344 - a1*c2334 + a1*c2344) * p034+ ( -a4*c0112 + a4*c0113 + a4*c0122 - a4*c0133 - a4*c0223 + a4*c0233 + a0*c1124 - a0*c1134 - a0*c1224 + a0*c1334 + a0*c2234 - a0*c2334) * p123+ ( a3*c0112 - a3*c0114 - a3*c0122 + a3*c0144 + a3*c0224 - a3*c0244 - a0*c1123 + a0*c1134 + a0*c1223 - a0*c1344 - a0*c2234 + a0*c2344) * p124+ ( -a2*c0113 + a2*c0114 + a2*c0133 - a2*c0144 - a2*c0334 + a2*c0344 + a0*c1123 - a0*c1124 - a0*c1233 + a0*c1244 + a0*c2334 - a0*c2344) * p134+ ( a1*c0223 - a1*c0224 - a1*c0233 + a1*c0244 + a1*c0334 - a1*c0344 - a0*c1223 + a0*c1224 + a0*c1233 - a0*c1244 - a0*c1334 + a0*c1344) * p234) / (divisor);result.arg = ( ( a3*c001 - a4*c001 + a4*c002 - a3*c002 + a4*c011 - a3*c011 + a3*c022 - a4*c022 + a3*c112 - a4*c112 + a4*c122 - a3*c122) * p012+ ( a4*c001 - a2*c001 + a2*c003 - a4*c003 + a2*c011 - a4*c011 + a4*c033 - a2*c033 + a4*c113 - a2*c113 + a2*c133 - a4*c133) * p013+ ( a2*c001 - a3*c001 + a3*c004 - a2*c004 + a3*c011 - a2*c011 + a2*c044 - a3*c044 + a2*c114 - a3*c114 + a3*c144 - a2*c144) * p014+ ( a1*c002 - a4*c002 + a4*c003 - a1*c003 + a4*c022 - a1*c022 + a1*c033 - a4*c033 + a1*c223 - a4*c223 + a4*c233 - a1*c233) * p023+ ( a3*c002 - a1*c002 + a1*c004 - a3*c004 + a1*c022 - a3*c022 + a3*c044 - a1*c044 + a3*c224 - a1*c224 + a1*c244 - a3*c244) * p024+ ( a1*c003 - a2*c003 + a2*c004 - a1*c004 + a2*c033 - a1*c033 + a1*c044 - a2*c044 + a1*c334 - a2*c334 + a2*c344 - a1*c344) * p034+ ( a4*c112 - a0*c112 + a0*c113 - a4*c113 + a0*c122 - a4*c122 + a4*c133 - a0*c133 + a4*c223 - a0*c223 + a0*c233 - a4*c233) * p123+ ( a0*c112 - a3*c112 + a3*c114 - a0*c114 + a3*c122 - a0*c122 + a0*c144 - a3*c144 + a0*c224 - a3*c224 + a3*c244 - a0*c244) * p124+ ( a2*c113 - a0*c113 + a0*c114 - a2*c114 + a0*c133 - a2*c133 + a2*c144 - a0*c144 + a2*c334 - a0*c334 + a0*c344 - a2*c344) * p134+ ( a0*c223 - a1*c223 + a1*c224 - a0*c224 + a1*c233 - a0*c233 + a0*c244 - a1*c244 + a0*c334 - a1*c334 - a0*c344 + a1*c344) * p234) / (divisor);result.arg = ( ( a4*c0012 - a3*c0012 + a2*c0013 - a4*c0013 + a3*c0014 - a2*c0014 + a3*c0112 - a4*c0112 + a4*c0113 - a2*c0113 + a2*c0114 - a3*c0114) * p01+ ( a3*c0012 - a4*c0012 + a4*c0023 - a1*c0023 + a1*c0024 - a3*c0024 + a4*c0122 - a3*c0122 + a1*c0223 - a4*c0223 + a3*c0224 - a1*c0224) * p02+ ( a4*c0013 - a2*c0013 + a1*c0023 - a4*c0023 + a2*c0034 - a1*c0034 + a2*c0133 - a4*c0133 + a4*c0233 - a1*c0233 + a1*c0334 - a2*c0334) * p03+ ( a2*c0014 - a3*c0014 + a3*c0024 - a1*c0024 + a1*c0034 - a2*c0034 + a3*c0144 - a2*c0144 + a1*c0244 - a3*c0244 + a2*c0344 - a1*c0344) * p04+ ( a4*c0112 - a3*c0112 + a3*c0122 - a4*c0122 + a0*c1123 - a4*c1123 + a3*c1124 - a0*c1124 + a4*c1223 - a0*c1223 + a0*c1224 - a3*c1224) * p12+ ( a2*c0113 - a4*c0113 + a4*c0133 - a2*c0133 + a4*c1123 - a0*c1123 + a0*c1134 - a2*c1134 + a0*c1233 - a4*c1233 + a2*c1334 - a0*c1334) * p13+ ( a3*c0114 - a2*c0114 + a2*c0144 - a3*c0144 + a0*c1124 - a3*c1124 + a2*c1134 - a0*c1134 + a3*c1244 - a0*c1244 + a0*c1344 - a2*c1344) * p14+ ( a4*c0223 - a1*c0223 + a1*c0233 - a4*c0233 + a0*c1223 - a4*c1223 + a4*c1233 - a0*c1233 + a1*c2234 - a0*c2234 + a0*c2334 - a1*c2334) * p23+ ( a1*c0224 - a3*c0224 + a3*c0244 - a1*c0244 + a3*c1224 - a0*c1224 + a0*c1244 - a3*c1244 + a0*c2234 - a1*c2234 + a1*c2344 - a0*c2344) * p24+ ( a2*c0334 - a1*c0334 + a1*c0344 - a2*c0344 + a0*c1334 - a2*c1334 + a2*c1344 - a0*c1344 + a1*c2334 - a0*c2334 + a0*c2344 - a1*c2344) * p34) / (divisor);result.arg = ( ( a3*c002 - a4*c002 + a4*c003 - a2*c003 + a2*c004 - a3*c004 + a4*c112 - a3*c112 + a2*c113 - a4*c113 + a3*c114 - a2*c114) * p01+ ( a4*c001 - a3*c001 + a1*c003 - a4*c003 + a3*c004 - a1*c004 + a3*c122 - a4*c122 + a4*c223 - a1*c223 + a1*c224 - a3*c224) * p02+ ( a2*c001 - a4*c001 + a4*c002 - a1*c002 + a1*c004 - a2*c004 + a4*c133 - a2*c133 + a1*c233 - a4*c233 + a2*c334 - a1*c334) * p03+ ( a3*c001 - a2*c001 + a1*c002 - a3*c002 + a2*c003 - a1*c003 + a2*c144 - a3*c144 + a3*c244 - a1*c244 + a1*c344 - a2*c344) * p04+ ( a3*c011 - a4*c011 + a4*c022 - a3*c022 + a4*c113 - a0*c113 + a0*c114 - a3*c114 + a0*c223 - a4*c223 + a3*c224 - a0*c224) * p12+ ( a4*c011 - a2*c011 + a2*c033 - a4*c033 + a0*c112 - a4*c112 + a2*c114 - a0*c114 + a4*c233 - a0*c233 + a0*c334 - a2*c334) * p13+ ( a2*c011 - a3*c011 + a3*c044 - a2*c044 + a3*c112 - a0*c112 + a0*c113 - a2*c113 + a0*c244 - a3*c244 + a2*c344 - a0*c344) * p14+ ( a1*c022 - a4*c022 + a4*c033 - a1*c033 + a4*c122 - a0*c122 + a0*c133 - a4*c133 + a0*c224 - a1*c224 + a1*c334 - a0*c334) * p23+ ( a3*c022 - a1*c022 + a1*c044 - a3*c044 + a0*c122 - a3*c122 + a3*c144 - a0*c144 + a1*c223 - a0*c223 + a0*c344 - a1*c344) * p24+ ( a1*c033 - a2*c033 + a2*c044 - a1*c044 + a2*c133 - a0*c133 + a0*c144 - a2*c144 + a0*c233 - a1*c233 - a0*c244 + a1*c244) * p34) / (divisor);result.arg = ( ( a4*c02 - a3*c02 + a2*c03 - a4*c03 + a3*c04 - a2*c04 + a3*c12 - a4*c12 + a4*c13 - a2*c13 + a2*c14 - a3*c14) * p01+ ( a3*c01 - a4*c01 + a4*c03 - a1*c03 + a1*c04 - a3*c04 + a4*c12 - a3*c12 + a1*c23 - a4*c23 + a3*c24 - a1*c24) * p02+ ( a4*c01 - a2*c01 + a1*c02 - a4*c02 + a2*c04 - a1*c04 + a2*c13 - a4*c13 + a4*c23 - a1*c23 + a1*c34 - a2*c34) * p03+ ( a2*c01 - a3*c01 + a3*c02 - a1*c02 + a1*c03 - a2*c03 + a3*c14 - a2*c14 + a1*c24 - a3*c24 + a2*c34 - a1*c34) * p04+ ( a4*c01 - a3*c01 + a3*c02 - a4*c02 + a0*c13 - a4*c13 + a3*c14 - a0*c14 + a4*c23 - a0*c23 + a0*c24 - a3*c24) * p12+ ( a2*c01 - a4*c01 + a4*c03 - a2*c03 + a4*c12 - a0*c12 + a0*c14 - a2*c14 + a0*c23 - a4*c23 + a2*c34 - a0*c34) * p13+ ( a3*c01 - a2*c01 + a2*c04 - a3*c04 + a0*c12 - a3*c12 + a2*c13 - a0*c13 + a3*c24 - a0*c24 + a0*c34 - a2*c34) * p14+ ( a4*c02 - a1*c02 + a1*c03 - a4*c03 + a0*c12 - a4*c12 + a4*c13 - a0*c13 + a1*c24 - a0*c24 + a0*c34 - a1*c34) * p23+ ( a1*c02 - a3*c02 + a3*c04 - a1*c04 + a3*c12 - a0*c12 + a0*c14 - a3*c14 + a0*c23 - a1*c23 + a1*c34 - a0*c34) * p24+ ( a2*c03 - a1*c03 + a1*c04 - a2*c04 + a0*c13 - a2*c13 + a2*c14 - a0*c14 + a1*c23 - a0*c23 + a0*c24 - a1*c24) * p34) / (divisor);if (to) *to = result;return 1;}static size_t parse_fields(const char *line, double *values, size_t count){const char *ends;double value;size_t n = 0;while (1) {ends = line;errno = 0;value = strtod(line, (char **)(&ends));if (errno || ends == line)return n;if (*ends == '\t' || *ends == '\n' || *ends == '\v' ||*ends == '\f' || *ends == '\r' || *ends == ' ') {line = ends + 1;if (n < count)values = value;n++;} elsereturn n;}}int main(int argc, char *argv[]){double nmax;uint64_t n;size_t i;int arg;if (argc < 2 || !strcmp(argv, "-h") || !strcmp(argv, "--help") || !strcmp(argv, "/?")) {const char *argv0 = (argc > 0 && argv && argv) ? argv : "(this)";fprintf(stderr, "\n");fprintf(stderr, "Usage: %s [ -h | --help | /? ]\n", argv0);fprintf(stderr, " %s INPUT-FILE [ INPUT-FILE ]\n", argv0);fprintf(stderr, "\n");fprintf(stderr, "Each input file should contain rows of form\n");fprintf(stderr, " ACTUAL-PRESSURE TEMPERATURE-COUNT MEASURED-PRESSURE\n");fprintf(stderr, "and nothing else.\n");fprintf(stderr, "\n");fprintf(stderr, "There must be at least five unique rows,\n");fprintf(stderr, "with at least three unique actual pressures,\n");fprintf(stderr, "and at least three unique temperature counts.\n");fprintf(stderr, "The more samples, the better.\n");fprintf(stderr, "\n");return EXIT_SUCCESS;}for (arg = 1; arg < argc; arg++) {double field;char buffer, *line;FILE *in;in = fopen(argv, "r");if (!in) {fprintf(stderr, "%s: %s.\n", argv, strerror(errno));return EXIT_FAILURE;}while (1) {line = fgets(buffer, sizeof buffer, in);if (!line)break;/* Skip leading whitespace. */line += strspn(line, "\t\n\v\f\r ");/* Skip lines beginning with # or ;. */if (*line == '#' || *line == ';')continue;/* Parse three numeric fields. */if (parse_fields(line, field, 3) != 3) {/* Remove trailing whitespace. */line[strcspn(line, "\t\n\v\f\r ")] = '\0';fprintf(stderr, "%s: Invalid input: \"%s\".\n", argv, line);return EXIT_FAILURE;}/* Add as a new sample. */add_sample(field, field, field);}if (!feof(in) || ferror(in)) {fclose(in);fprintf(stderr, "%s: Read error.\n", argv);return EXIT_FAILURE;} elseif (fclose(in)) {fprintf(stderr, "%s: Read error.\n", argv);return EXIT_FAILURE;}}if (num_samples < 5) {fprintf(stderr, "Not enough calibration samples.\n");return EXIT_FAILURE;}fprintf(stderr, "Read %zu calibration samples.\n", num_samples);nmax = (double)(num_samples )* (double)(num_samples - 1)* (double)(num_samples - 2)* (double)(num_samples - 3)* (double)(num_samples - 4)* 0.66; /* Approximate, depends on sample distribution -- good'enuf */n = 0;{config point;size_t i0, i1, i2, i3, i4;for (i0 = 0; i0 < num_samples; i0++) {const double p0 = samples.measured_pressure;const double c0 = samples.temperature_count;const double a0 = samples.actual_pressure;for (i1 = 0; i1 < num_samples; i1++) {const double p1 = samples.measured_pressure;const double c1 = samples.temperature_count;const double a1 = samples.actual_pressure;if (i1 == i0)continue;for (i2 = 0; i2 < num_samples; i2++) {const double p2 = samples.measured_pressure;const double c2 = samples.temperature_count;const double a2 = samples.actual_pressure;if (i2 == i0 || i2 == i1)continue;for (i3 = 0; i3 < num_samples; i3++) {const double p3 = samples.measured_pressure;const double c3 = samples.temperature_count;const double a3 = samples.actual_pressure;if (i3 == i0 || i3 == i1 || i3 == i2)continue;for (i4 = 0; i4 < num_samples; i4++) {const double p4 = samples.measured_pressure;const double c4 = samples.temperature_count;const double a4 = samples.actual_pressure;if (i4 == i0 || i4 == i1 || i4 == i2 || i4 == i3)continue;if (unique5(a0, a1, a2, a3, a4) < 3)continue;if (unique5(c0, c1, c2, c3, c4) < 3)continue;if (unique5(p0, p1, p2, p3, p4) < 3)continue;n++;if (solve(p0, p1, p2, p3, p4, c0, c1, c2, c3, c4, a0, a1, a2, a3, a4, &point)) {if (best_error < 0 || error_magnitude(point) < ERROR_FACTOR * best_error)walk(point);}}}}}fprintf(stderr, "\r[ Estimating %.3f%% complete ]\n", (double)n * 100.0 / nmax);fflush(stderr);}}fprintf(stderr, "\n");fflush(stderr);printf(" p | c | P(p, c) | P | Absolute error\n");printf("--------+------+---------------+-----+-----------------\n");for (i = 0; i < num_samples; i++) {const double p = samples.measured_pressure;const double c = samples.temperature_count;const double actual = samples.actual_pressure;const double r = best_point.arg + best_point.arg*c+ p*(best_point.arg + c*(best_point.arg + best_point.arg*c));printf("%7.3f | %4.0f | %13.9f | %3.0f | %+15.12f\n", p, c, r, actual, r - actual);}printf("\n");printf("Maximum error minimized to %.9f, using\n", best_error);printf("%.16g %.16g %.16g %.16g %.16g\n",best_point.arg, best_point.arg, best_point.arg,best_point.arg, best_point.arg);return EXIT_SUCCESS;}Note that this does not quarantee that the absolute error is that or smaller over the entire measurement range; this minimizes the absolute error (magnitude)at calibration sample points only. It's up to you to provide enough samples to cover the device operational range.

author: itsbiodiversity
author_rank: Regular Contributor
timestamp: August 21, 2020, 02:18:13 pm
content: you are a MAD PERSON! Wow. Thanks so much for sticking with me on this.

author: IanB
author_rank: Super Contributor
timestamp: August 21, 2020, 08:29:54 pm
content: Quote from: itsbiodiversity on August 21, 2020, 11:27:48 amAs I mentioned in an earlier post, this is easily done using the Excel solver add-in (it is included with every version of Excel, but it does not show up on ribbon by default--you have to add it).With the solver add-in you can say "vary these cells to minimize the value of that cell". It becomes quite easy to do all sorts of regressions that way.

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 24, 2020, 07:06:13 pm
content: If you want to explore the gradient descent min-max optimizer, you can contact me via email, or continue discussion here.(I'm only interested in this as an optimization problem, not looking to get anything.)I noticed that I can get well over 50% acceleration using explicit AVX2 intrinsics; also a thread pool approach should work really well here.My limitation is that I don't do Windows-only code.In particular, if you have a calibration setup where you have more than one known good setup measuring the same thing as the device being calibrated, you could probably make the entire calibration process continuous.

author: b_force
author_rank: Super Contributor
timestamp: August 25, 2020, 10:17:11 pm
content: Noble amount of labor to figure all these things outBut I don't want to be much of a party pooper, but there are quite some programs available that can do very complicated curve fitting.

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 26, 2020, 02:05:38 pm
content: Note that thegradient descentoptimization isnotcurve fitting, it is actually anoptimizerthat works on the full parameter space, finding the local optimum (minimizing maximum error to calibration samples) from a given starting point.Mathematically, we have a function \$f(a_0, a_1, a_2, a_3, a_4)\$ that provides the maximum error among the calibration points when the compensation function for measured pressure \$p\$ and temperature count \$c\$ is \$P(p, c) = a_0 + a_1 c + a_2 p + a_3 p c + a_4 p c^2\$. The program assumes that only one calibration point causes the maximum error, and uses the gradient of that point – i.e., \$\left(\frac{\partial f}{\partial a_0}, \frac{\partial f}{\partial a_1}, \frac{\partial f}{\partial a_2}, \frac{\partial f}{\partial a_3}, \frac{\partial f}{\partial a_4} \right)\$ – to determine the direction in which to move in the parameter space. The length of the step is first maximized (to the maximum length that provides a smaller maximum error than the starting point), then a binary search along this hyperline segment is used to find the \$(a_0, a_1, a_2, a_3, a_4)\$ that yields the smallest maximum error along this hyperline segment.The program does a very, very large number of these descents. It uses each unique quintet among the calibration samples (each unique 5-tuple of calibration points) to use the \$\left(\begin{matrix}N\\5\end{matrix}\right) = N(N-1)(N-2)(N-3)(N-4)/(1\cdot 2 \cdot 3 \cdot 4 \cdot 5)\$ starting points for the descents. (Because of numerical instability of the naive implementation of the 5-point solver, it actually uses each of the 120 variants.)In essence, the curve \$P(p, c) = a_0 + a_1 c + a_2 p + a_3 p c + a_4 p c^2\$ is only used with sets of 5 calibration samples, and then only to get useful starting points for the gradient descent. There is no fitting at all, because the set of 5 samples define the five unknowns exactly. (The code implements this algebraic solver itself.)To get similar results from curve fitting, you use the maximum error in magnitude, \$\max\left(\left\lvert P(p_i,c_i) - A_i\right\rvert\right)\$ across the \$i\$ calibration samples (\$A_i\$ being the actual pressure being measured, \$p_i\$ the corresponding uncompensated measured pressure, and \$c_i\$ the temperature count). But, since \$\max\left(\left\lvert P(p_i,c_i) - A_i\right\rvert\right)\$ is nonlinear (piecewise quadratic with a cross term), with a non-contiguous second derivative (because the \$i\$ that causes the maximum error varies), with LOTS of local minima (thousands if not more), normal methods (including anything based on linear algebra) will not produce as good results as this gradient descent does.No need to believe me, though. Just use the curve fitting package you think does better or even just as well, and see if it discovers the minimum I reported a couple of messages back, that minimizes the maximum error among the calibration samples to 0.002421 pressure units. In comparison, least squares fitting only reaches a minimum error a magnitude larger (around 0.02, depending on details).(Actually, if you can, please do; I would appreciate the results. The ones I have access to – gnuplot, maxima, sagemath, maple, and mathematica – all use least squares fitting, which produces the same minimum error, about 0.02; or an order of magnitude worse maximum error than the gradient descent does.)

author: CatalinaWOW
author_rank: Super Contributor
timestamp: August 26, 2020, 05:31:28 pm
content: Quote from: IanB on August 21, 2020, 08:29:54 pmI heartily endorse this. There are closed form solutions to many regression problems, but the math gets difficult quickly for more complex forms. This method works easily and for a very broad class of problems. You do occasionally get convergence problems, which can often be solved by changing initial state or selecting other options in the solver setup, like bounding values for some of the coefficients or changing the form of the model.The key point is that for a variety of reasons a lowest sum of squared errors is a "best" answer. Do be aware that other definitions of best exist and may be appropriate to specific cases. Lowest maximum error for example.

author: Doctorandus_P
author_rank: Super Contributor
timestamp: August 27, 2020, 12:05:50 am
content: Why not just keep it simple?Measure some reference points, and then do linear interpolation between those points.You may need a few more points, (10 or so is usually plenty) but it makes the software much easier to understand, write, maintain, modify, etc.

author: The Electrician
author_rank: Frequent Contributor
timestamp: August 27, 2020, 01:09:59 am
content: Quote from: itsbiodiversity on August 19, 2020, 01:38:10 pm'What is the independent variable (and its powers) which the temperature compensation coefficients are multiplying?If you use the "coef" command, what are the currently effective coefficients?Edit:I see that the answer to my questions can be found in reply #39 and #41Here is how I derived a least squares solution using some matrix methods:I solved this with an HP50G calculator and got 9 correct digits for a1, and 12 correct digits for a2.

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 27, 2020, 11:49:47 am
content: Quote from: Doctorandus_P on August 27, 2020, 12:05:50 amBecause it is the device itself that internally uses \$P(p, c) = a_0 + a_1 c + a_2 p + a_3 p c + a_4 p c^2\$ to apply pressure \$p\$ and "temperature count" \$c\$ compensation to the pressure reading it provides.Quote from: The Electrician on August 27, 2020, 01:09:59 amExcept that that least squares fit (which is most commonly used) gives you maximum error an order of magnitude larger than gradient descent optimization (which yields \$a_0 = 0.007697997673805127\$, \$a_1 = -9.893590127282349\cdot 10^{-7}\$, \$a_2 = 0.9937816855224063\$, \$a_3 = 1.353571043339379 \cdot 10^{-6}\$, and \$a_4 = -6.784307221818112\cdot 10^{-11}\$).Why do I care? Because if OP is doing recalibration before giving the devices to actual users, I want those users to have the best readings they can. Least squares regression is most commonly used to fit device parameters to calibration data, but popularity does not mean it is the best method. In this instance, I have already shown how using the exact same calibration data, you can reduce the maximum error by a factor of 10. Why so many members here think that is irrelevant or not worth the effort, boggles my mind.

author: The Electrician
author_rank: Frequent Contributor
timestamp: August 27, 2020, 12:40:01 pm
content: Quote from: Nominal Animal on August 27, 2020, 11:49:47 amUsing these coefficients I get the same result you reported in reply #49:"No need to believe me, though. Just use the curve fitting package you think does better or even just as well, and see if it discovers the minimum I reported a couple of messages back, that minimizes the maximum error among the calibration samples to 0.002421 pressure units. In comparison, least squares fitting only reaches a minimum error a magnitude larger (around 0.02, depending on details)."But you pay for a smaller maximum (absolute) error with a larger residual for the GDO method, namely .009256, compared to a residual for least squares of .007665I don't find the maximum absolute error for a least squares solution to be "around .02", but rather I get .005115, which is only about twice the GDO error, not 10 times. This problem has a condition number of about 2.646E9, which is why I used rational arithmetic to solve it rather than floating point, otherwise errors in results may be larger than expected at first glance.This is often a choice to be made as to the "best" method; minimize absolute error, or residual.

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 27, 2020, 03:09:46 pm
content: Quote from: The Electrician on August 27, 2020, 12:40:01 pmRight; I apologize. I'm on a different machine (about 1Mm away from my normal machine), and remembered the difference wrong. In any case, halving the error is in my opinion still significant enough to consider.Of course, it all depends on the calibration method. For the GDO to make sense, you need 40-50 calibration samples at minimum, preferably more – I'd prefer around a hundred, with more unique temperatures. This is why I suggested research into continuous calibration, i.e. measuring a test environment whose pressure and temperature can be varied, but not controller, with at least two other measurement units, and doing GDO dynamically during measurement. (There is no need for the temperature counts and actual pressures to form a regular grid; the only critical thing is the sample density in the parameter space. I'm thinking of a small pressure vessel in a thermal bath, with more than one set of sensors, so that both pressure and temperature can be varied (but not precisely), and measured very precisely.)Quote from: The Electrician on August 27, 2020, 12:40:01 pmYes, exactly: choosing what to minimize. Least squares fitting is reliable and easy to document; gradient descent optimization to minimize error at calibration points is harder, butpotentiallymore useful for end users. I mean, if you can calibrate a device to higher precision than guaranteed, should you or should you not? What is the physical accuracy of the sensors themselves, and how fast do they drift? Would the "extra precision" lead users astray?We do not know how repeatable the calibration samples are, nor how much physical error there is in the pressure and temperature readings.For example, if for each calibration sample you had error bars, you could treat both the calibration measurements and the device coefficients as a statistical problem instead. (Then, you wouldn't have anything exact to fit to, but would need to treat the five coefficients as parameters choosing the most likely approximation.)I do know that calibrating PT100 and PT1000 temperature sensors can give very, very precise results, because they are inherently quite stable devices at these approx. room temperatures. (Self-heating due to the measurement current passing through the device can be a problem, but in my experience, it only tends to be an issue with small amounts of material to measure, and/or much lower temperatures.)As to pressure sensors, there are many different types, and we don't know which kind it is. Most of them are quite stable (strain gauge based ones for example), so I wouldguessthat their time-dependent variation is small, and precise calibration can actually give better readings in the long term (as opposed to just at the moment of calibration).It is definitely an interesting problem in my opinion, and something I'd like to see put some thought into, as an end user myself. (Not OP's device, but pressure and PT100 temperature sensors in general, in a fusor project I'm helping with for example.)

author: b_force
author_rank: Super Contributor
timestamp: August 28, 2020, 02:05:49 pm
content: Calibrating a sensor is one thing, but ever thought about the practical variables that influence the accuracy?Especially measuring very accurate temperatures is extremely difficult, sensor NOT included.Move the sensor a couple of millimetres and you're readings WILL be off.Changes the airflow a teeny tiny bit and you'r readings WILL vary.Same goes for humidity, type of mounting, mounting torque, use of thermal paste, type of thermal paste.The list goes on and on.You will be easily of ±0.5 degrees, which is even pretty generous.I have seen differences over ±2 degrees in a professional mixing device, stabilized for 2 hours.When working with samples, you should use the standard deviation with the amount of sigma you need (for accurate measurements, that's usually either 3 or 4 sigma), in combination with the standard error.Depending if you're interested in the spread or how accurate your expected average will be.I also agree to some extend with @Doctorandus_P .You should focus on what you would expect, not if your mathematical equation is going to fit to a million decimals.The default curve of a PT100 an alike isn't complicated at all, so it's not difficult to fit a regression line in there.The rest of the systems accuracy you need to calculate with the relative error formula.