Help how to fit the calibration table from ADC values - Page 1
Forum section: electronics / beginners

author: daisizhou
author_rank: Super Contributor
timestamp: March 13, 2025, 08:29:48 am
content: Hallo everyone.A long time ago, a good friend helped me make a 256-bit hexadecimal calibration table.This is a calibration table for hot wire sensors.（The accuracy is +/-2%）I lost touch with this friend due to his busy work.He told me the brief production process, but I still don't understand.I want to recreate the process myself, but I have difficulty.in other words,I hope to reproduce the same hexadecimal table as him through the ADC value.I tried to fit the curve using Origin, but unfortunately I failed,If anyone could help me I would be very grateful, thanksCode:0 12863 15286 16879 185212 196815 207418 212821 218824 225225 227730 237735 246240 254245 260250 265255 270460 276365 281770 287175 292280 295585 300790 305395 3099100 3136105 3176110 3214115 3258120 3298125 3324130 3355135 3384140 3416145 3451150 3472155 3507160 3542165 3566170 3587175 3613180 3644185 3669190 3694195 3714200 3743205 3760210 3771215 3790220 3812225 3838230 3861235 3874240 3884Code:00 00 00 02 00 05 00 08 00 0b 00 0f 00 13 00 1700 1b 00 1f 00 24 00 28 00 2d 00 32 00 36 00 3b00 40 00 46 00 4b 00 50 00 55 00 5b 00 60 00 6600 6b 00 71 00 77 00 7d 00 82 00 88 00 8e 00 9400 9a 00 a0 00 a6 00 ad 00 b3 00 b9 00 bf 00 c600 cc 00 d3 00 d9 00 e0 00 e6 00 ed 00 f4 00 fa01 01 01 08 01 0f 01 15 01 1c 01 23 01 2a 01 3101 38 01 3f 01 46 01 4d 01 54 01 5c 01 63 01 6a01 71 01 75 01 81 01 8d 01 9a 01 a6 01 b3 01 c001 cd 01 da 01 e8 01 f6 02 04 02 12 02 20 02 2f02 3d 02 4c 02 5b 02 6b 02 7a 02 8a 02 9a 02 aa02 ba 02 cb 02 db 02 ec 02 fd 03 0e 03 20 03 3203 44 03 56 03 68 03 7a 03 82 03 94 03 a6 03 b803 cb 03 dd 03 f0 04 03 04 17 04 2a 04 3e 04 5204 66 04 7a 04 8e 04 a3 04 b7 04 cc 04 e1 04 f605 0c 05 21 05 37 05 4d 05 63 05 79 05 90 05 a705 de 05 fb 06 1d 06 40 06 62 06 85 06 a8 06 c906 eb 07 0c 07 2e 07 4f 07 6d 07 8b 07 a8 07 c607 e3 08 01 08 21 08 41 08 61 08 82 08 a2 08 c208 e2 09 02 09 23 09 43 09 64 09 86 09 a8 09 ca09 ec 0a 16 0a 4b 0a 7f 0a b1 0a d2 0a f4 0b 150b 36 0b 58 0b 7e 0b a4 0b ca 0b ef 0c 15 0c 3b0c 61 0c 87 0c ad 0c dc 0d 0b 0d 3a 0d 67 0d 930d be 0d ea 0e 16 0e 44 0e 72 0e a0 0e c9 0e f00f 18 0f 3f 0f 68 0f 94 0f bf 0f eb 10 22 10 6510 a8 10 e1 11 19 11 51 11 8c 11 c8 12 04 12 3a12 71 12 a7 12 d9 13 0b 13 3c 13 7e 13 d1 14 1514 47 14 79 14 ab 14 dc 15 0e 15 40 15 7e 15 c716 2f 16 82 16 b9 16 f3 17 32 17 7f 17 a6 17 e218 39 18 72 18 af 18 e8 19 3a 19 8d 19 e4 1a 291a 65 1a a1 1b 00 1b 70 1c 08 1c 63 1c bc 1d 0b1d 59 1d 9c 1d df 1e 26 1e 72 1e cd 1f 52 20 00

author: jwet
author_rank: Frequent Contributor
timestamp: March 13, 2025, 02:30:04 pm
content: I'll try to help.It sounds like is you have an 8 bit value derived from a hot wire sensor. This sensor measures flow and you want to take the 8 bit value through a table lookup to give a flow value.A linear curve fit wouldn't work too well since you won't find a single line that goes though the points to approximate the shape, all you have is a slope and offset.What you want is a polynomial fit- it looks like a second order fit would work- the graph looks roughly parabolic. If your 8 bit input is X then the formula is as follows:Flow= a(x^2)+bx+c.You make a table in your spreadsheet with the input column X (ADC from 0 to 255) and your flow values. You then have a table with X^2 column, x and a c column. You'll have an adjacent column for the actual flow. The spreadsheet operation is called multiple regression in Excel. It will look at all you data and find the best fit (least squared error) to your data and provide you with a, b and c.Once you have a,b and c. You can generate a new table that will have your input and the calculated flow values- this is the best look up table.In your final system, you take your adc value into the table and look up a flow value- its just one to one. This is called table look up if you want a good search term.It can get confusing keeping the input and the output straight. Try to be systematic and think in terms of the formula.Also- if you have processor that can do math, you can solve the polynomial formula in your final system. An 8 bit table of 16 bit intergers would likely be larger than the code to solve that formula if you have good math support already linked in. If you're in a small system, table lookup is fine, it will take .5k of storage for the table.I hope this helps some.

author: daisizhou
author_rank: Super Contributor
timestamp: March 14, 2025, 02:50:49 am
content: Thanks, I will try the method you said.

author: Nominal Animal
author_rank: Super Contributor
timestamp: March 14, 2025, 08:51:52 am
content: If you convert the hexadecimal values to 16-bit unsigned integers with more significant byte first, you get the middle column inCode:0 0 12861 22 53 8 15284 115 156 19 16877 238 279 31 185210 3611 4012 45 196813 5014 5415 59 207416 6417 7018 75 212819 8020 8521 91 218822 9623 10224 107 225225 113 227726 11927 12528 13029 13630 142 237731 14832 15433 16034 16635 173 246236 17937 18538 19139 19840 204 254241 21142 21743 22444 23045 237 260246 24447 25048 25749 26450 271 265251 27752 28453 29154 29855 305 270456 31257 31958 32659 33360 340 276361 34862 35563 36264 36965 373 281766 38567 39768 41069 42270 435 287171 44872 46173 47474 48875 502 292276 51677 53078 54479 55980 573 295581 58882 60383 61984 63485 650 300786 66687 68288 69889 71590 731 305391 74892 76593 78294 80095 818 309996 83697 85498 87299 890100 898 3136101 916102 934103 952104 971105 989 3176106 1008107 1027108 1047109 1066110 1086 3214111 1106112 1126113 1146114 1166115 1187 3258116 1207117 1228118 1249119 1270120 1292 3298121 1313122 1335123 1357124 1379125 1401 3324126 1424127 1447128 1502129 1531130 1565 3355131 1600132 1634133 1669134 1704135 1737 3384136 1771137 1804138 1838139 1871140 1901 3416141 1931142 1960143 1990144 2019145 2049 3451146 2081147 2113148 2145149 2178150 2210 3472151 2242152 2274153 2306154 2339155 2371 3507156 2404157 2438158 2472159 2506160 2540 3542161 2582162 2635163 2687164 2737165 2770 3566166 2804167 2837168 2870169 2904170 2942 3587171 2980172 3018173 3055174 3093175 3131 3613176 3169177 3207178 3245179 3292180 3339 3644181 3386182 3431183 3475184 3518185 3562 3669186 3606187 3652188 3698189 3744190 3785 3694191 3824192 3864193 3903194 3944195 3988 3714196 4031197 4075198 4130199 4197200 4264 3743201 4321202 4377203 4433204 4492205 4552 3760206 4612207 4666208 4721209 4775210 4825 3771211 4875212 4924213 4990214 5073215 5141 3790216 5191217 5241218 5291219 5340220 5390 3812221 5440222 5502223 5575224 5679225 5762 3838226 5817227 5875228 5938229 6015230 6054 3861231 6114232 6201233 6258234 6319235 6376 3874236 6458237 6541238 6628239 6697240 6757 3884241 6817242 6912243 7024244 7176245 7267246 7356247 7435248 7513249 7580250 7647251 7718252 7794253 7885254 8018255 8192The left column is both index and the flow; the right column is the ADC value corresponding to the flow. For simplicity, lets usefto denote the flow andithe row number,ADC(f) =ADC(i) for the corresponding ADC value in the rightmost column, andH(i) for thei'th 16-bit unsigned integer in your firmware array as shown in the center column.Curve (f,ADC(f)) extremely similar to the curveH(i)*15/512, (1286+i*866/85).That is, to populateH(i), you calculatea= 1286+i*866/85. Then, you findfsuch thatADC(f) =a. Then, you setH(i) = round(512*f/15).For example, let's say you want to find H(200) (which we know is 4264).a=1286+200*866/85≃3324. We know that ADC(125)=3324, sof=125.f*512/15=125*512/15≃4267. (Difference between 4264 and 4267 is 0.07%. Obviously, instead of the table, you use the ADC curve you derive from your calibration points.)Now, I pulled the constants 15/512 = 0.029296875 (or 512/15 ≃ 34.133...) and 866/85≃10.1882 (or 85/866≃0.09815) out of my hat, by guesswork, so they're probably not exactly right. The zero at 1286 I picked from your data, so that could be slightly off, too.What I don't get is why you don't just use a look-up table that specifies the (middle) ADC reading for each flow rate,reading =adc. To find the flow, you do a simple binary search,Code:uint16_t reading;uint8_t flow(uint16_t adc) {if (adc >= reading)return 255;if (adc <= reading)return 0;uint8_t fmin = 0;uint8_t fmax = 255;while (1) {uint8_t f = fmin + (fmax - fmin)/2;if (f == fmin)break;if (reading < adc)fmin = f;elseif (reading > adc)fmax = f;elsereturn f;}if (adc - reading <= reading[f+1] - adc)return f;elsereturn f+1;}You can even get some more precision out of this via linear interpolation whenreading<adc<reading[f+1]: the result isf+xwherex = (adc - reading) / (reading[f+1] - reading),0 < x < 1.

author: daisizhou
author_rank: Super Contributor
timestamp: March 14, 2025, 12:02:53 pm
content: Thanks.This helped me understand indexes. I initially mistakenly thought that indexes were program-defined numbers.The ADC value is initially as shown below,I can only be sure of the 1286 corresponding to 0, and the other values ​​are fluctuating as the flow rate increases.Hot wire sensors are very sensitive.It may be caused by eddy currents.100 3091-3120105 3142-3159110 3213-3227115 3246-3255120 3274-3283125 3298-3325130 3318-3349135 3357-3365140 3395-3404145 3406-3427150 3439-3452155 3472-3488160 3516-3521165 3528-3540170 3547-3552175 3562-3580180 3565-3586185 3598-3618190 3621-3631195 3655-3668200 3698-3702205 3709-3714210 3723-3753215 3747-3775220 3775-3783225 3792-3820230 3818-3829235 3846-3857240 3867-3886HIGH FLOW ADC0 12863 15286 16879 185212 196815 207418 212821 218824 225225 227730 237735 246240 254245 260250 265255 270460 276365 281770 287175 292280 295585 300790 305395 3099100 3136105 3176110 3214115 3258120 3298125 3324130 3355135 3384140 3416145 3451150 3472155 3507160 3542165 3566170 3587175 3613180 3644185 3669190 3694195 3714200 3743205 3760210 3771215 3790220 3812225 3838230 3861235 3874240 3884LOW FLOW ADC0 8262.5 15625 18807.5 211010 230112.5 246415 261117.5 273320 284322.5 292625 3024

author: Nominal Animal
author_rank: Super Contributor
timestamp: March 15, 2025, 02:45:23 am
content: For what it is worth, I like to usegnuplotfor this kind of stuff. It does use theLevenberg–Marquardt algorithm, also known as damped least squares fitting, but allows you to specify the function to fit to. Just because it's free and open source, doesn't make it inferior: it's very widely used among the scientist crowd.For example, I took your initial data, converted each line toflowminmaxby removing the dashes, saved the result as "initial.txt", and then in gnuplot,Code:FIT_LIMIT=1e-10;f(x) = C0 + x*(C1 + x*(C2 + x*(C3 + x*C4)));fit f(x) "initial.txt" u (0.5*($2+$3)):($1) via C0,C1,C2,C3,C4;set xrange [2955:3945];set yrange [84:256];set key center left;plot "initial.txt" u 2:1 t " Max" w lines lc "red", f(x) t "Fit" w lines lc "black", "" u 3:1 t "Min" w lines lc "blue"(either as shown, or all in one single long line), and it displays(Click to embiggen)Furthermore,print C0, C1, C2, C3, C4outputs22595.6463484117 -24.0680606961196 0.00952409228671269 -1.65790462262155e-06 1.08058409399106e-10which means the above fit isflow(adc) = 22595.6463484117 - adc * (24.0680606961196 - adc * (0.00952409228671269 - adc * (0.00000165790462262155 - adc * 0.000000000108058409399106))Similarly, if I save "low-flow.txt" and "high-flow.txt", and tell gnuplot to plot the low and high flows on top of each other,Code:set xlabel "Low ADC";set ylabel "Low flow";set x2label "High ADC";set y2label "High flow";set xtics nomirror;set x2tics nomirror;set ytics nomirror;set y2tics nomirror;set key top left;set xrange [ 826-0.05*(3024-826) : 3024+0.05*(3024-826) ];set yrange [ 0-0.1*(25-0) : 25+0.1*(25-0) ];set x2range [ 1286-0.05*(3884-1286) : 3884+0.05*(3884-1286) ];set y2range [ 0-0.1*(240-0) : 240+0.1*(240-0) ];plot "low-flow.txt" u 2:1 axes x1y1 t "Low flow" w lines lc "red", "high-flow.txt" u 2:1 axes x2y2 t "High flow" w lines lc "blue"it shows(Click to embiggen)As you can see, I prefer to plot curves with the input on the horizontal axis, and output on the vertical.One trick is that between tasks, run "reset session" to forget old variables and settings. Very important when fitting!In particular,reset sessionFIT_LIMIT=1e-15; f(x) = C0+x*(C1+x*(C2+x*(C3+x*C4))); fit f(x) "low-flow.txt" u 2:1 via C0,C1,C2,C3,C4 ; plot "low-flow.txt" u 2:1 t 'Data' w lines lc "black", f(x) t "Fit" w lines lc "red"gives an excellent fourth-degree polynomial fit to the low flow,flow(adc) = 9.26067671535308 - adc * (0.0248355007869194 - adc * (0.0000217118632127165 - adc * (0.00000000722410135139931 - adc * 0.00000000000110185086154194)))Similarly,reset sessionFIT_LIMIT=1e-15; f(x) = C0+x*(C1+x*(C2+x*(C3+x*(C4 + x*(C5 + x*C6))))); fit f(x) "high-flow.txt" u 2:1 via C0,C1,C2,C3,C4,C5,C6 ; plot "high-flow.txt" u 2:1 t 'Data' w lines lc "black", f(x) t "Fit" w lines lc "red"gives a similar sixth-degree fit for the high flow,flow(adc) = 131.945231794756 - adc * (0.486229000989584 - adc * (0.000680541577419277 - adc * (0.000000476355184001067 - adc * (0.000000000177854571022255 - adc * (0.0000000000000331703849587813 - adc * 0.00000000000000000247116386856345)))))For further analysis, I often useawkto augment the data. For example, runningawk '{ f = 9.26067671535308 - $2 * (0.0248355007869194 - $2 * (0.0000217118632127165 - $2 * (0.00000000722410135139931 - $2 * 0.00000000000110185086154194))); if ($1>0) printf "%-4s %-4s %7.3f %+9.6f %+9.6f\n", $1, $2, f, f-$1, (f-$1)/$1; else printf "%-4s %-4s %7.3f %+9.6f\n", $1, $2, f, f-$1 }' low-flow.txt > low-flow-fit.txtcreates file "low-flow-fit.txt" with contentsCode:0 826 0.002 +0.0017342.5 1562 2.469 -0.030989 -0.0123955 1880 5.071 +0.070863 +0.0141737.5 2110 7.498 -0.001553 -0.00020710 2301 9.947 -0.052531 -0.00525312.5 2464 12.430 -0.070123 -0.00561015 2611 15.052 +0.052212 +0.00348117.5 2733 17.560 +0.060324 +0.00344720 2843 20.123 +0.123353 +0.00616822.5 2926 22.272 -0.228066 -0.01013625 3024 25.075 +0.074774 +0.002991where the third column is the fit value to the ADC reading (in second column), fourth is the absolute error compared to the flow (positive if fit is greater, negative if fit is smaller than the flow), and fifth is the relative error (±0.01 = 1%). The fourth-degree polynomial gives a pretty good (<±1.5%) fit.Very similarly,awk '{ f = 131.943050604831 - $2 * (0.486223343657538 - $2 * (0.000680535659432263 - $2 * (0.000000476351980133453 - $2 * (0.000000000177853621921368 - $2 * (0.0000000000000331702387408685 - $2 * 0.00000000000000000247115469574707))))); if ($1>0) printf "%-3s %-4s %7.3f %+9.6f %+9.6f\n", $1, $2, f, f-$1, (f-$1)/$1; else printf "%-3s %-4s %7.3f %+9.6f\n", $1, $2, f, f-$1 }' high-flow.txt > high-flow-fit.txtgenerates file "high-flow-fit.txt" with the sixth-degree fit,Code:0 1286 -0.024 -0.0236123 1528 3.168 +0.167911 +0.0559706 1687 5.690 -0.309977 -0.0516639 1852 9.087 +0.086973 +0.00966412 1968 12.199 +0.198641 +0.01655315 2074 15.717 +0.716794 +0.04778618 2128 17.792 -0.207791 -0.01154421 2188 20.342 -0.658387 -0.03135224 2252 23.359 -0.641436 -0.02672725 2277 24.624 -0.376446 -0.01505830 2377 30.187 +0.186763 +0.00622535 2462 35.566 +0.566099 +0.01617440 2542 41.187 +1.186791 +0.02967045 2602 45.760 +0.759954 +0.01688850 2652 49.805 -0.194710 -0.00389455 2704 54.238 -0.761996 -0.01385460 2763 59.545 -0.454900 -0.00758265 2817 64.660 -0.339518 -0.00522370 2871 70.022 +0.022378 +0.00032075 2922 75.314 +0.313530 +0.00418080 2955 78.856 -1.144337 -0.01430485 3007 84.629 -0.371492 -0.00437090 3053 89.934 -0.065973 -0.00073395 3099 95.431 +0.431472 +0.004542100 3136 99.997 -0.002725 -0.000027105 3176 105.083 +0.083364 +0.000794110 3214 110.066 +0.065952 +0.000600115 3258 116.029 +1.028812 +0.008946120 3298 121.641 +1.640983 +0.013675125 3324 125.393 +0.392629 +0.003141130 3355 129.979 -0.021189 -0.000163135 3384 134.387 -0.613419 -0.004544140 3416 139.390 -0.609706 -0.004355145 3451 145.043 +0.042688 +0.000294150 3472 148.530 -1.470029 -0.009800155 3507 154.513 -0.487149 -0.003143160 3542 160.724 +0.724291 +0.004527165 3566 165.125 +0.125494 +0.000761170 3587 169.077 -0.922671 -0.005427175 3613 174.108 -0.892116 -0.005098180 3644 180.318 +0.317859 +0.001766185 3669 185.505 +0.505225 +0.002731190 3694 190.863 +0.863422 +0.004544195 3714 195.280 +0.280080 +0.001436200 3743 201.902 +1.902245 +0.009511205 3760 205.911 +0.910705 +0.004442210 3771 208.557 -1.443439 -0.006874215 3790 213.227 -1.773118 -0.008247220 3812 218.800 -1.200063 -0.005455225 3838 225.628 +0.627553 +0.002789230 3861 231.898 +1.897875 +0.008252235 3874 235.542 +0.542480 +0.002308240 3884 238.397 -1.602767 -0.006678where relative error is less than 1.5% above flow rate 50, <3% above flow rate 21, increasing to up to 6% near zero.As this isn'tmydata, I don't dare massage anything to get an even better fit. Possibilities range from adding "synthesized" data points to increase the importance of small values, to changing the origin and scale. I often use the known physical model as a basis, instead of blindly using polynomials or other functions.There are many different ways one can implement these in a microcontroller, so seeing the two functions (or their logical equivalents in pseudocode) that converts the ADC readings to flow rates, would describe exactly how the look-up tables are used. In my previous post, I simply took the "known" calibration data, and fitted it to the binary data, that's all –– like throwing spaghetti at the wall to see what sticks.

author: radiolistener
author_rank: Super Contributor
timestamp: March 15, 2025, 08:21:16 am
content: Quote from: Nominal Animal on March 15, 2025, 02:45:23 amI don't know gnuplot syntax, can you please explain what means "" in your code? I tried to run it but it fails with error on this experssion.Can you please write it in octave/matlab syntax, so that it will be more easy to read?

author: Nominal Animal
author_rank: Super Contributor
timestamp: March 15, 2025, 10:21:48 am
content: Quote from: radiolistener on March 15, 2025, 08:21:16 amJust a typo I didn't notice! This forum uses .. forteletype text, which I used it elsewhere in the post, and somehow one ended up in the wrong place.Quote from: radiolistener on March 15, 2025, 08:21:16 amNo, because you do curve fitting differently in Octave/Matlab, and there are much better online guides for that than I can do, because I don't use Octave or Matlab for curve fitting.

author: radiolistener
author_rank: Super Contributor
timestamp: March 15, 2025, 10:55:59 am
content: I removed this mistype and it works. But I still want some octave solution, because I'm not familiar with gnuplot.The interpolation algorithm looks very interesting, but not easy to implement.

author: gf
author_rank: Super Contributor
timestamp: March 15, 2025, 11:22:42 am
content: Quote from: radiolistener on March 15, 2025, 10:55:59 amWhat about polyfit() and polyval()? A 6th order polynomial fits the numbers in theOPnicely (x is the first column and y is the second column).Code:p = polyfit(y,x,6);plot(0:4000,polyval(p,0:4000),y,x,".")grid onxlim([1000 4000])xlim([1000 4000])

author: daisizhou
author_rank: Super Contributor
timestamp: March 15, 2025, 11:46:38 am
content: Thanks.I really envy those of you whose mother tongue is English.Able to work more efficiently.PS:Can we use the curve clues we know so far to determine what model this hot wire sensor is?Because I checked the appearance carefully and there is no mark

author: radiolistener
author_rank: Super Contributor
timestamp: March 15, 2025, 12:31:21 pm
content: I found that Octave has implementation of Levenberg-Marquardt algorithm as leasqr() function. But it is deprecated and there is better replacement nonlin_curvefit(), but it's not clear on how to use it.UPDATED: added some fixesThere is a mix with suggestion from chat-gpt and deep-seek with my fixes which at a glance works, but it gives different results:Code:pkg load optim;data = load('initial.txt');sig = data(:, 1); % Input valuesres = 0.5 * (data(:, 2) + data(:, 3)); % Average of min and max% Model: y = f(x)f = @(C, x) C(1) + x.*(C(2) + x.*(C(3) + x.*(C(4) + x.*C(5))));% Initial parametersp0 = [3000; 1; 0.01; 1; 1];% Fit the model using nonlin_curvefit[p, ~, ~, ~] = nonlin_curvefit(f, p0, sig, res);% Debug outputfor i = 100:20:240fprintf('f(%d) = %f\n', i, f(p, i));end% Plot the resultssig_range = [84, 256];res_range = [2955, 3945];sig_fit = linspace(sig_range(1), sig_range(2), 100);res_fit = f(p, sig_fit);figure;hold on;plot(data(:, 2), sig, 'r', 'DisplayName', 'Max');plot(data(:, 3), sig, 'b', 'DisplayName', 'Min');plot(res_fit, sig_fit, 'k', 'DisplayName', 'Fit');xlabel('Output');ylabel('Input');legend show;xlim([res_range(1), res_range(2)]);ylim([sig_range(1), sig_range(2)]);hold off;grid on;grid minor;Code:f(100) = 3111.140416f(120) = 3277.967492f(140) = 3399.558056f(160) = 3498.978639f(180) = 3591.787881f(200) = 3686.036532f(220) = 3782.267454f(240) = 3873.515617It appears that it very depends on used initial parameters, but it's not clear how to properly select it and how it is selected in gnuplot. Any idea?

author: gf
author_rank: Super Contributor
timestamp: March 15, 2025, 01:04:43 pm
content: Quote from: radiolistener on March 15, 2025, 12:31:21 pmUse operator .* for element-wise multiplcation of two vectors.

author: radiolistener
author_rank: Super Contributor
timestamp: March 15, 2025, 01:13:29 pm
content: Quote from: gf on March 15, 2025, 01:04:43 pmYes, that was my mistake. I updated the code, result and plot above. Now it works with 5 parameters.The remaining question is how to properly select the initial parameters. How does Gnuplot do it?Update: it appears that with fixed version it almost doesn't depends on initial parameters, so you can use even p0 = zeros(1, 5)';With p0 = zeros(1, 5)'; it produces these coefficients:Code:C0 = 254.944666C1 = 58.293517C2 = -0.428353C3 = 0.001507C4 = -0.000002with p0 = ones(1, 5)'; it produce:Code:C0 = 265.578453C1 = 58.022773C2 = -0.425844C3 = 0.001497C4 = -0.000002with p0 = [3000; 1; 0.01; 1; 1];Code:C0 = 265.576468C1 = 58.022824C2 = -0.425845C3 = 0.001497C4 = -0.000002But then it's unclear why there is so significant difference with gnuplot for input values below 100...

author: daisizhou
author_rank: Super Contributor
timestamp: March 15, 2025, 11:59:53 pm
content: Could it be due to human error on my part or air vortex?Below is my raw ADC dataCode:0 128625 227730 237735 246240 254245 260250 265255 270460 276365 281770 287175 292280 295585 300790 305395 3099100 3136105 3176110 3214115 3258120 3298125 3324130 3355135 3384140 3416145 3451150 3472155 3507160 3542165 3566170 3587175 3613180 3644185 3669190 3694195 3714200 3743205 3760210 3771215 3790220 3812225 3838230 3861235 3874240 3884

author: Nominal Animal
author_rank: Super Contributor
timestamp: March 16, 2025, 02:39:11 am
content: Quote from: daisizhou on March 15, 2025, 11:46:38 amMy native language isFinnish, which is not related to English or other Romance languages at all. I learned English at school, and none of my English teachers were native English speakers. I only write "well" (but excessively verbosely) due to decades of practice and working on it. Not speaking much English, my spoken English is reallyRally English.If I can do it, you can too, given time and practice.Quote from: daisizhou on March 15, 2025, 11:46:38 amI don't know. I don't remember ever working with hot wire flow sensors in practice.Quote from: radiolistener on March 15, 2025, 12:31:21 pmBefore thefitcommand, you simply assign the variables with initial values.Note that theFIT_LIMITin Gnuplot defaults to only 1e-5 = 0.00001, but I force it to much smaller, which causes it to do many, many more iterations – it takes a couple of seconds on my old laptop to complete the fit.Quote from: daisizhou on March 15, 2025, 11:59:53 pmCurve fitting is detailed work. Values below 100 are outside the fit, so are irrelevant to the quality of the fit. Besides, I'm fitting forflowas a function ofADCreading, whereas radiolistener is fittingADCreading as a function offlow: transposed or inverse, whatever you want to call it.If I also approximate the ADC reading as a function of flow, I getC0=245.355,C1=58.5377,C2=-0.430615,C3=0.00151635, andC4=-0.00000198312; so thatf(100)=3111.005,f(170)=3545.466,f(240)=3873.404.radiolistener's higher coefficients don't show enough decimals to compare: usingg(x)=265.576468+x*(58.022824+x*(-0.425845+x*(0.001497-x*0.000002)))I getg(100)=3106.409,g(170)=3506.877,g(240)=3721.390, showing that the higher values drop completely outside the fit.That is why I useprint C0, C1, C2, C3, C4in gnuplot to get the double-precision values with enough decimals to represent the value exactly, and my own little utility program to convert scientific format numbers like 2.47115469574707e-18 to 0.00000000000000000247115469574707. (It simply reads in the numbers, then repeatedly converts it back to a string with increasing number of decimals, until converting back to number yields the original number.)(That has bitten me in the butt often in the past. In molecular dynamics, Embedded Atom Model is provided as samples, which are then used to reconstruct the original curve using piecewise cubic polynomials. Because the samples aren't provided at the full precision, the reconstructed piecewise curve will have a small oscillation at the upper end, close to zero. Technically, these oscillations form very weak bonds at high distances! They're just too small to normally matter much, but failing to provide the coefficients atfullcomputed precision, which would avoid or at least lessen this, always made me a bit angry: these people are scientists, and should know better! It's only a little thing, making sure their computation provides the values at maximum precision.)Gnuplotprintalways reports the values with sufficient number of decimals to represent the value exactly. For Octave/Matlab, I think settingformat longoroutput_precision(32)and then some utility to minimize the representation, would work.Here is a C command-line helper, that converts double-precision floating point numbers to their minimal decimal-only string form, that when converted back to a double, produces the exact same value. Note that it isn't the exact value of the double; that is always much longer (but finite). The helper accepts the standard forms, plus0xHHH...for the underlying storage representation (for 32-bit single-precision float if there are at most 8 hex digits, 64-bit double precision double if more). Supply one or more such values on the command line as parameters. It assumes the byte order is the same for both floating-point and integer types, and that the underlying float and double are IEEE 754 Binary32 and Binary64, respectively, because they are on all widely used computer architectures (and basically all rare ones as well).Code:// SPDX-License-Identifier: CC0-1.0// Compile and link against the math library, for example// gcc -Wall -O2 this.c -lm -o executablename#include <stdlib.h>#include <inttypes.h>#include <string.h>#include <stdio.h>#include <math.h>#include <errno.h>static inline const char *skipwhitespace(const char *from) {if (from)while (*from == '\t' || *from == '\n' || *from == '\v' ||*from == '\f' || *from == '\r' || *from == ' ')from++;return from;}static const char *parse_double_part(const char *from, double *to) {const char *end;double val;from = skipwhitespace(from);if (!from || !*from)return NULL;// Try parsing as a hexadecimal storage representation.if (from == '0' && (from == 'x' || from == 'X')) {uintmax_t storage;end = from + 2;errno = 0;storage = strtoumax(from + 2, (char **)&end, 16);if (!errno && end > from + 2) {// Single-precision?if ((size_t)(end - from) <= 10 && (uintmax_t)((uint32_t)storage) == storage && sizeof (float) == sizeof (uint32_t)) {if (to) {union {float f;uint32_t u;} binary32;binary32.u = storage;*to = binary32.f;}return end;}// Double-precision?if ((uintmax_t)((uint64_t)storage) == storage && sizeof (double) == sizeof (uint64_t)) {if (to) {union {double d;uint64_t u;} binary64;binary64.u = storage;*to = binary64.d;}return end;}}}// Try parsing as a double.errno = 0;end = from;val = strtod(from, (char **)&end);if (errno || end == from)return NULL;if (to)*to = val;return end;}static int parse_double(const char *from, double *to) {const char *end;double val;from = skipwhitespace(from);if (!from || !*from)return -1;end = parse_double_part(from, &val);if (!end)return -1;// Trailing + and - increase/decrease the value by 1 ULP.while (*end == '+' || *end == '-')if (*(end++) == '+')val = nextafter(val, +HUGE_VAL);elseval = nextafter(val, -HUGE_VAL);from = skipwhitespace(end);if (*from)return -1;if (to)*to = val;return 0;}#ifdef NAN#define NOT_A_DOUBLE NAN#else#define NOT_A_DOUBLE INFINITY#endifstatic double parse_double_strict(const char *from) {const char *end = from;double val;errno = 0;val = strtod(from, (char **)&end);if (errno || end == from || *end)return NOT_A_DOUBLE;return val;}static const char *prettyprint(double value) {static char buffer;double parsed;int digits = 0;if (!isfinite(value)) {int len = snprintf(buffer, sizeof buffer, "%f", value);if (len < 1 || (size_t)len >= sizeof buffer)return NULL;return (const char *)buffer;}while (1) {int len = snprintf(buffer, sizeof buffer, "%.*f", digits, value);if (len < 1 || (size_t)len >= sizeof buffer)return NULL;parsed = parse_double_strict(buffer);if (isfinite(parsed) && parsed == value)return (const char *)buffer;digits++;}}int main(int argc, char *argv[]) {const char *arg0 = (argc > 0 && argv && argv && argv) ? argv : "(this)";if (argc < 2 || !strcmp(argv, "-h") || !strcmp(argv, "--help")) {printf("\n");printf("Usage: %s [ -h | --help ]\n", arg0);printf(" %s NUMBER [ NUMBER... ]\n", arg0);printf("\n");printf("This program prints each number as a double-precision floating point\n");printf("number with sufficient digits to parse to the same value.\n");printf("This does not print the exact value of that number, only a decimal\n");printf("representation that when converted to double, produces the same number.\n");printf("\n");printf("Each NUMBER can be in standard decimal form, scientific form, or\n");printf("the actual storage representation in hexadecimal. You can suffix it\n");printf("with + or - to increase or decrease the value by smallest representable\n");printf("amount.\n");printf("\n");return EXIT_SUCCESS;}for (int a = 1; a < argc; a++) {const char *formatted;double value;if (parse_double(argv, &value)) {fflush(stdout);fprintf(stderr, "%s: Cannot parse value.\n", argv);return EXIT_FAILURE;}formatted = prettyprint(value);if (!formatted) {fflush(stdout);fprintf(stderr, "%s: Cannot convert value to a string.\n", argv);return EXIT_FAILURE;}printf("%s = %s\n", argv, formatted);}return EXIT_SUCCESS;}For example, if I compile and run the above,./nicedouble 2.47115469574707e-18it prints2.47115469574707e-18 = 0.00000000000000000247115469574707I can also examine the nearest representations by running./nicedouble 2.47115469574707e-18- 2.47115469574707e-18 2.47115469574707e-18+so it prints2.47115469574707e-18- = 0.00000000000000000247115469574706972.47115469574707e-18 = 0.000000000000000002471154695747072.47115469574707e-18+ = 0.0000000000000000024711546957470705which tells me I could also use "0.0000000000000000024711546957470700" to make it absolutely explicit that this does describe the value at full double precision, even if it is not the exact value of the underlying double precision number; just its shortest decimal representation.

author: daisizhou
author_rank: Super Contributor
timestamp: March 16, 2025, 03:14:46 am
content: Thanks.The gnuplot software is too difficult for me, I need to spend time learning how to use it.Currently I try to fit high flow ADC data using origin,But the result is not as expected, I don't know where the error occurred.

author: Nominal Animal
author_rank: Super Contributor
timestamp: March 16, 2025, 06:16:06 am
content: Thus far, you've been modeling the ADC reading as a function of the flow rate.Instead, try modeling the flow rate as a function of the ADC reading.The difference is which one you provide as a variable, and which one you compare against. It seems to me you're using the flow rate as the input axis, and ADC reading as the output axis; but I believe you should be using the ADC reading as the input axis, and flow rate as the output axis. It does not matter for plotting, but it matters a lot for curve fitting!You see, the mathematical model should not represent your calibration approach (where you control the flow rate, and observe the ADC reading): it should model the task you are trying to achieve, finding the flow rate when you have an ADC reading. Calibration data is just a set of input-output pairs that you want your model/approximation/firmware to replicate, while also interpolating between those points for other input values.

author: daisizhou
author_rank: Super Contributor
timestamp: March 16, 2025, 06:57:41 am
content: is that so?I am learning how to convert ADC values ​​into 0, 2, 5, 8, 11, 15, 19........

author: Nominal Animal
author_rank: Super Contributor
timestamp: March 16, 2025, 08:06:20 am
content: Quote from: daisizhou on March 16, 2025, 06:57:41 amYes!To elaborate to those who might not follow:In#16daisizhou used function \$y = a + b \, x^c\$, where \$y\$ represents an ADC reading, \$x\$ flow rate, and \$a\$, \$b\$, and \$c\$ are determined by the fit.In#18, they use the same function \$y = a + b \, x^c\$, but this time \$y\$ represents flow rate, and \$x\$ the ADC reading.The difference is not just which way the curve is plotted; it also affects the fit in very fundamental ways.Based onefunda.com hot wire theory page, the general form for the flow rate function is$$v_f = \left(\frac{I^2 R_\text{ref} \bigl( 1 + a ( T_w - T_\text{ref}) \bigr)}{b A_w ( T_w - T_f )} - \frac{a}{b} \right)^\frac{1}{c}$$where \$v_f\$ is the flow rate, \$I\$ is the current through the hot wire, \$A_w\$ is the projected (effective) hot wire surface area, \$R_\text{ref}\$ is the resistance of the hot wire at temperature \$T_\text{ref}\$, \$T_w\$ is the actual wire temperature, \$T_f\$ is the temperature of the fluid, and \$a\$, \$b\$, and \$c\$ are experimentally fitted variables, with \$c \approx 0.5\$.If we assume that the ADC reading \$x\$ is approximately a linear function of the current \$I\$, and everything else is a constant, then the flow rate \$v_f\$ should be of form$$v_f(x) = \left( C_0 + C_1 x + C_2 x^2 \right)^{C_3}$$where we'd expect \$C_3 = 1/c \approx 2\$.Unfortunately, the damped-least-squares fit of that form is extremely sensitive to changes in \$C_3\$, and I'm not sure it can find a global optimum – the best fit possible. Because of that, I'd expand it out, assuming \$C_3 = 2\$, but allowing each power of \$x\$ their own coefficient. We obviously arrive at the fourth-degree polynomial used above,$$\begin{aligned}v_f(x) &= C_0 + C_1 x + C_2 x^2 + C_3 x^3 + C_4 x^4 \\~ &= C_0 + x \Biggl( C_1 + x \biggl( C_2 + x \bigl( C_3 + x C_4 \bigr) \biggr) \Biggr) \\\end{aligned}$$Note that the above two forms are mathematically identical: the lower one is just numerically more stable, and thus often considered "better" way when floating-point numbers are used. Also note that alternating signs behave thus:$$\begin{aligned}v_f(x) &= C_0 - C_1 x + C_2 x^2 - C_3 x^3 + C_4 x^4 \\~ &= C_0 - x \Biggl( C_1 - x \biggl( C_2 - x \bigl( C_3 - x C_4 \bigr) \biggr) \Biggr) \\\end{aligned}$$(I usemaxima(wxMaximafor a nice UI) to manipulate the expressions in algebraic form.)Anyway, I got very nice fit for low flow with that.Because \$c\$ is not exactly one half, and the power form is unstable, it doesn't fit too well for the high-flow case. We can compensate by using a higher-degree polynomial. If weknowthat \$v_f(1286) = 0\$, we can enforce that too at the same time, using$$\begin{aligned}v_f(x) &= C_1 (x - 1286) + C_2 (x - 1286)^2 + \dots \\~ &= \sum_{i=1}^N C_i (x - 1286)^i \\\end{aligned}$$for up to degree \$N\$; we probably want to keep \$N\$ even, too.The best fit I got was with \$N = 6\$, in gnuplot:Code:f(x) = g(x - 1286);g(x) = x*(C1 + x*(C2 + x*(C3 + x*(C4 + x*(C5 + x*C6)))));FIT_LIMIT = 1e-20; fit f(x) "high-flow.txt" u 2:1 via C1,C2,C3,C4,C5,C6;print C1,C2,C3,C4,C5,C6giving me a high-flow functionf(x) = (x-1286)*(0.0121543669426515 + (x-1286)*(0.00000385693437341006 + (x-1286)*(-0.00000000532712686337969 + (x-1286)*(0.0000000000260948623601223 + (x-1286)*(-0.0000000000000141654567756325 + (x-1286)*0.00000000000000000247804766152225)))))that yields the following results:Code:0 1286 0.000 0.0003 1528 3.170 +0.170 +5.67%6 1687 5.689 -0.311 -5.19%9 1852 9.086 +0.086 +0.95%12 1968 12.198 +0.198 +1.65%15 2074 15.717 +0.717 +4.78%18 2128 17.792 -0.208 -1.15%21 2188 20.342 -0.658 -3.13%24 2252 23.359 -0.641 -2.67%25 2277 24.624 -0.376 -1.50%30 2377 30.187 +0.187 +0.62%35 2462 35.566 +0.566 +1.62%40 2542 41.187 +1.187 +2.97%45 2602 45.760 +0.760 +1.69%50 2652 49.805 -0.195 -0.39%55 2704 54.238 -0.762 -1.39%60 2763 59.545 -0.455 -0.76%65 2817 64.660 -0.340 -0.52%70 2871 70.022 +0.022 +0.03%75 2922 75.313 +0.313 +0.42%80 2955 78.855 -1.145 -1.43%85 3007 84.628 -0.372 -0.44%90 3053 89.934 -0.066 -0.07%95 3099 95.431 +0.431 +0.45%100 3136 99.997 -0.003 -0.00%105 3176 105.083 +0.083 +0.08%110 3214 110.066 +0.066 +0.06%115 3258 116.029 +1.029 +0.89%120 3298 121.641 +1.641 +1.37%125 3324 125.393 +0.393 +0.31%130 3355 129.979 -0.021 -0.02%135 3384 134.387 -0.613 -0.45%140 3416 139.391 -0.609 -0.44%145 3451 145.043 +0.043 +0.03%150 3472 148.530 -1.470 -0.98%155 3507 154.513 -0.487 -0.31%160 3542 160.724 +0.724 +0.45%165 3566 165.126 +0.126 +0.08%170 3587 169.077 -0.923 -0.54%175 3613 174.108 -0.892 -0.51%180 3644 180.318 +0.318 +0.18%185 3669 185.505 +0.505 +0.27%190 3694 190.863 +0.863 +0.45%195 3714 195.280 +0.280 +0.14%200 3743 201.902 +1.902 +0.95%205 3760 205.910 +0.910 +0.44%210 3771 208.556 -1.444 -0.69%215 3790 213.227 -1.773 -0.82%220 3812 218.800 -1.200 -0.55%225 3838 225.628 +0.628 +0.28%230 3861 231.898 +1.898 +0.83%235 3874 235.543 +0.543 +0.23%240 3884 238.398 -1.602 -0.67%The columns areThe flow rateThe ADC readingThe flow rate the fit function yields for the ADC reading in second columnThe absolute error between the flow ratesThe relative error of the fitNote that flow rates below 40, the absolute error is less than ±1, i.e. the fit rounds to the same flow rate or flow rate ±1. I suspect there is some noise in the data, and a "perfect" fit is neither wanted nor achievable.I would re-check the calibration data at least for flow rates 15, 21, 24, 35, 40, 45, 55, 80, because below flow rate 100 those are the ones with more than ±0.5 of absolute error. That is, other than those, the fit yields flow rates (up to 100) that round to the exact same value. Above 100, I'd recheck 115, 120, 150, 200, because those are the only ones where the relative error reaches ±1%. If you look at the calibration data, these are basically the samples where the calibration data shows a "bump" compared to any smooth curve, too.

author: radiolistener
author_rank: Super Contributor
timestamp: March 16, 2025, 08:11:44 am
content: @Nominal Animal, can you explain why there is so huge difference between gnuplotfitand octavenonlin_curvefit?

author: daisizhou
author_rank: Super Contributor
timestamp: March 16, 2025, 08:30:30 am
content: It's amazing.thank you very much.It will take me some time to read the post in depth, but I find the preliminary data to be correct.

author: Nominal Animal
author_rank: Super Contributor
timestamp: March 16, 2025, 08:31:40 am
content: Quote from: radiolistener on March 16, 2025, 08:11:44 amYes, and no.Yes: They use different methods. Gnuplotfitis described in detailhere, but the Octavenonlin_curvefit(seenonlin_residmin) has a number of backends it can use, depending on the optimization settings.No: I don't know how to make them yield the exact same approximations.It is important to note here thatit is perfectly okay to edit your calibration data used for curve fitting only, as long as you compare and plot the fit against unmodified calibration data.This is because it does not matter how you obtain the values for the free variables, as long as the results satisfy your criteria. (This is also why mathematicians often leave the details for an exercise, while those of us who onlyusemath to solve problem, are most interested in exactly those details!)It is also why I list the reproduced data for the fits I've done: so that you can evaluate for yourself how proper/good that fit is.The form of the function does matter a lot, though. You can think of it as defining the "class" of function. It is even more important when extrapolating, i.e. going outside the range of calibration data, because the form dictates the asymptotic behaviour. When interpolating between calibration data points, it is much less important.In practice, I recommend you create a separate file containing the calibration points, but that you may edit to get an even better results ––when compared to the original calibration data. That is, you "lie" to the fitter, so that it finds an even better result than it can from the original data; but it is absolutely crucial that you do not use that modified data for anything else. You definitely do not use it to compare against, or for any plots. Only fitting.In scientific papers, the methods used to obtain the empirically defined parameters –– the fit –– are not described. The methods used to obtain the fit do not matter; only the quality of the fit matters. This can feel like cheating, but really is not! The fit is our proposedthing, and the way it was found or obtained is secondary. All of physics is similar: we model reality by using whatever mathematical models that describe our measurements and observations and predict future measurements, and, perhaps, then try to explain why the model looks like that.

author: Nominal Animal
author_rank: Super Contributor
timestamp: March 18, 2025, 09:34:47 am
content: Based on PMs with daisizhou,amyk worked out how the firmware uses the calibration tablein 2021.The ADC reading is clamped to a range that corresponds to 0.000 L/min .. 240.000 L/min. That range is stored in the XRAM, for the high flow at 102c:102d, and for the high flow is 1286 .. 3884.I'll call the ADC readingx, and that rangexMin..xMax, inclusive; for high flow,xMin=1286,xMax=3884. The following is not the description of what the code does –– it uses extra steps for some strange reason! –– but what it achieves.The ADC readingxis linearly mapped to range 0.0 .. 255.0 inclusive,y = (x - xMin) * 255.0 / (xMax - xMin)and clamped (so thaty >= 0.0 && y <= 255.0). The integral partiis the look-up table index,i = floor(y) = (int)(y)f = y - (float)iand the fractional partfis used to linearly interpolate between that entry and the next:v = (1.0 - f) * TABLE + f * TABLE[i+1]This value is the flow rate in units of 240/8192 liters per minute, or exactly 0.029296875 L/min. Thus, flow rate ishighflow = v * 240 / 8192 = v * 0.029296875The steps for the low flow are similar, except the range differs, and the final units are 25/8192 liters per minute, or exactly 0.0030517578125 L/m, solowflow = v * 25 / 8192 = v * 0.0030517578125To generate the 256 16-bitTABLEvalues, we need to first determine the ADC range, and remember which maximum flow rate we're working with. (The low flow and high flow have separate TABLEs). I'll call the maximum flow raterMax(240 for high flow, 25 for low flow), and the known flow rater, both in liters per minute (L/min).Next, we take our calibration data, and map the ADC reading to 0..255, and flow rate to 0..8192. Taking theHIGH FLOW table from #4, with the flow rate in the first column and the ADC reading in the second column, the conversion is trivial in awk. Note that the mapped ADC is now in first column, and the mapped flow rate in the second column:awk '{ printf "%.9f %.9f\n", 255*($2-1286)/(3884-1286), 8192*$1/240 }' HIGH-FLOW.txt > high-flow-scaled.txtThe result ishigh-flow-scaled.txt:Code:0.000000000 0.00000000023.752886836 102.40000000039.359122402 204.80000000055.554272517 307.20000000066.939953811 409.60000000077.344110855 512.00000000082.644341801 614.40000000088.533487298 716.80000000094.815242494 819.20000000097.269053118 853.333333333107.084295612 1024.000000000115.427251732 1194.666666667123.279445727 1365.333333333129.168591224 1536.000000000134.076212471 1706.666666667139.180138568 1877.333333333144.971131640 2048.000000000150.271362587 2218.666666667155.571593533 2389.333333333160.577367206 2560.000000000163.816397229 2730.666666667168.920323326 2901.333333333173.435334873 3072.000000000177.950346420 3242.666666667181.581986143 3413.333333333185.508083141 3584.000000000189.237875289 3754.666666667193.556581986 3925.333333333197.482678984 4096.000000000200.034642032 4266.666666667203.077367206 4437.333333333205.923787529 4608.000000000209.064665127 4778.666666667212.500000000 4949.333333333214.561200924 5120.000000000217.996535797 5290.666666667221.431870670 5461.333333333223.787528868 5632.000000000225.848729792 5802.666666667228.400692841 5973.333333333231.443418014 6144.000000000233.897228637 6314.666666667236.351039261 6485.333333333238.314087760 6656.000000000241.160508083 6826.666666667242.829099307 6997.333333333243.908775982 7168.000000000245.773672055 7338.666666667247.933025404 7509.333333333250.484988453 7680.000000000252.742494226 7850.666666667254.018475751 8021.333333333255.000000000 8192.000000000For the low flow,xMin=826,xMax=3024,rMax=25, soawk '{ printf "%.9f %.9f\n", 255*($2-826)/(3024-826), 8192*$1/25 }' LOW-FLOW.txt > low-flow-scaled.txtyieldslow-flow-scaled.txt:Code:0.000000000 0.00000000085.386715196 819.200000000122.279344859 1638.400000000148.962693358 2457.600000000171.121474067 3276.800000000190.031847134 4096.000000000207.085987261 4915.200000000221.239763421 5734.400000000234.001364877 6553.600000000243.630573248 7372.800000000255.000000000 8192.000000000Our task is now to interpolate this data, so that we can calculate the second column value corresponding to each first column integer value 0..255. The values, rounded to nearest integer, are then stored in the firmware/XRAM.If we plot these curves, with the ADC value scaled to 0..255 on the horizontal axis, and relative flow scaled to 0..8192 on the vertical axis, i.e.set key top left ; set ytics mirror 0, 512, 8192 ; set xtics mirror 0, 17, 255 ; set yrange [-256:8192+256]; set xrange [-8:255+8]; plot "high-flow-scaled.txt" u 1:2 t "Scaled high flow" w lines lc "red", "low-flow-scaled.txt" u 1:2 t "Scaled low flow" w lines lc "blue"and we get a rather familiar representation of the two curves,(Click to embiggen)Note that the interpolating function here, mapping 0..255 to 0..8191, now includes both the hot wire sensor behaviour and all the nonlinearities in the electronics and ADC. Therefore, having the interpolating function match the physical response of the hot wire sensor is not important anymore.You could even use piecewise linear, or piecewise cubic interpolation (cubic splines), to interpolate between the calibration points, passing exactly through every calibration point. The device would then pretty much exactly reproduce the calibration values.

author: Nominal Animal
author_rank: Super Contributor
timestamp: March 19, 2025, 06:58:39 am
content: Here is a fully worked-out, step-by-step example of generating one of the 256×16-bit tables.Step 1.We obtain some calibration data. We control the flow, and find out the ADC reading. With flow in the first column, and the corresponding ADC reading in the second column, let's say we get file "high-flow.txt":Code:0 12863 15286 16879 185212 196815 207418 212821 218824 225225 227730 237735 246240 254245 260250 265255 270460 276365 281770 287175 292280 295585 300790 305395 3099100 3136105 3176110 3214115 3258120 3298125 3324130 3355135 3384140 3416145 3451150 3472155 3507160 3542165 3566170 3587175 3613180 3644185 3669190 3694195 3714200 3743205 3760210 3771215 3790220 3812225 3838230 3861235 3874240 3884It is very important we get a good ADC reading for zero flow, and for maximum flow. Here,maxFlow=240. The ADC reading for zero flow isminADC=1286, and the ADC reading for maximum flow ismaxADC=3884. These will be used in later steps.Step 2.We rescale the calibration data, so that flow will be between 0..8192, inclusive, and ADC reading between 0.0 .. 255.0, inclusive. This means that flowfwill becomef*8192/maxFlow, and ADC readingadcwill become(adc-minADC)*255/(maxADC-minADC). To make following steps easier, we also change the columns, so that the scaled ADC reading will be in the first column, and the scaled flow on the second column.One way to do this would be to useawk. If the calibration data was in file "high-flow.txt", thenawk '{ printf "%.6f %.6f\n", ($2-1286)*255.0/(3884-1286), $1*8192.0/240 }' high-flow.txt > high-flow-scaled.txtwill generate the data to "high-flow-scaled.txt".Another option is to use a spreadsheet program, like Excel or OpenOffice Calc or LibreOffice Calc. If you put the original flow in column A starting at row 1, and original ADC readings in column B starting at row 1, then formula=(B1-1286)*255/(3884-1286)yields the scaled ADC readings –– put it in a first-row cell, and then copy or extend down, and the program will adjust the cell reference to B2, B3, B4, and so on ––; and=A1*8192/240yields the scaled flow rates.Anyway you do it, the result should be similar to following "high-flow-scaled.txt":Code:0.000000000 0.00000000023.752886836 102.40000000039.359122402 204.80000000055.554272517 307.20000000066.939953811 409.60000000077.344110855 512.00000000082.644341801 614.40000000088.533487298 716.80000000094.815242494 819.20000000097.269053118 853.333333333107.084295612 1024.000000000115.427251732 1194.666666667123.279445727 1365.333333333129.168591224 1536.000000000134.076212471 1706.666666667139.180138568 1877.333333333144.971131640 2048.000000000150.271362587 2218.666666667155.571593533 2389.333333333160.577367206 2560.000000000163.816397229 2730.666666667168.920323326 2901.333333333173.435334873 3072.000000000177.950346420 3242.666666667181.581986143 3413.333333333185.508083141 3584.000000000189.237875289 3754.666666667193.556581986 3925.333333333197.482678984 4096.000000000200.034642032 4266.666666667203.077367206 4437.333333333205.923787529 4608.000000000209.064665127 4778.666666667212.500000000 4949.333333333214.561200924 5120.000000000217.996535797 5290.666666667221.431870670 5461.333333333223.787528868 5632.000000000225.848729792 5802.666666667228.400692841 5973.333333333231.443418014 6144.000000000233.897228637 6314.666666667236.351039261 6485.333333333238.314087760 6656.000000000241.160508083 6826.666666667242.829099307 6997.333333333243.908775982 7168.000000000245.773672055 7338.666666667247.933025404 7509.333333333250.484988453 7680.000000000252.742494226 7850.666666667254.018475751 8021.333333333255.000000000 8192.000000000You may have a different number of decimals in your table, but that's okay; just try to keep at least three decimals, preferably four to six, to avoid loss of precision.In particular, note that the first linealwayshas two zeroes, and that the last line always has255and8192. If you don't, you mistyped one of the conversions.From this point onwards, we useonlythis scaled data (file "high-flow-scaled.txt"), not the original data.Step 3.We now havealmostthe firmware table –– at least the first and last entries. The problem is that we need to use the scaled data to find the scaled ADC values corresponding to scaled flows 0, 1, 2, ..., 253, 254, 255.We can do this many ways. We can fit some function \$f(x)\$ to the entire data, and then print \$f(0)\$, \$f(1)\$, \$f(2)\$, ..., \$f(253)\$, \$f(254)\$, \$f(255)\$, or we can interpolate between the nearest values.(I believe the best method would be a cubic spline interpolation, where each interval between consecutive points is interpolated using cubic (third-degree) curve with C¹ continuity: where one curve ends and the next starts, the curves and curve tangents match. This yields a hardware table that reproduces the calibration values exactly, to within the device computation precision.)For simplicity and sake of example, let's fit the data to a sixth-degree polynomial \$f(x)\$ such that \$f(0) = 0\$ and \$f(255) = 8192\$,$$f(x) = C_1 x + C_2 x^2 + C_3 x^3 + C_4 x^4 + C_5 x^5 + \left(\frac{8192}{255^6} - \frac{C_1}{255^5} - \frac{C_2}{255^4} - \frac{C_3}{255^3} - \frac{C_4}{255^2} - \frac{C_5}{255}\right) x^6$$Remember, we fit to thescaleddata here, not to the original calibration data!In gnuplot, this isf(x) = x*(C1 + x*(C2 + x*(C3 + x*(C4 + x*(C5 + x*(8192.0/255.0**6 - C1/255.0**5 - C2/255.0**4 - C3/255.0**3 - C4/255.0**2 - C5/255.0))))))fit f(x) "high-flow-scaled.txt" u 1:2 via C1,C2,C3,C4,C5print C1, C2, C3, C4, C5, 8192.0/255.0**6 - C1/255.0**5 - C2/255.0**4 - C3/255.0**3 - C4/255.0**2 - C5/255.0which tells us2.24999635243848 0.11447694372166 -0.00197198103101253 2.3737200908095e-05 -1.04927937396057e-07 1.65996547913956e-10ie. that the fit function isf(x) = x*(2.24999635243848 + x*(0.11447694372166 - x*(0.00197198103101253 -x*(2.3737200908095e-05 -x*(1.04927937396057e-07 - x*1.65996547913956e-10)))))Step 4.We use our approximation from the previous step, to obtain the scaled flow for each scaled ADC reading 0, 1, 2, ..., 253, 254, 255. These rounded scaled flow values are the 256 16-bit values you need to put into the device, to get its results match your calibration.I can do that in gnuplot after fitting, usingdo for [x=0:255] { print int(0.5+f(x)) }(whereint()is a truncation operation, and for nonnegative values,int(0.5+y)roundsyto nearest integer).Or, I can useawk:awk 'BEGIN { for (x=0; x<=255; x++) printf "%.0f\n", x*(2.24999635243848 + x*(0.11447694372166 - x*(0.00197198103101253 -x*(2.3737200908095e-05 -x*(1.04927937396057e-07 - x*1.65996547913956e-10))))) }'Or, better yet, showing interpolated value, rounded interpolated value, and in hex:awk 'BEGIN { for (x=0; x<=255; x++) { v = x*(2.24999635243848 + x*(0.11447694372166 - x*(0.00197198103101253 -x*(2.3737200908095e-05 -x*(1.04927937396057e-07 - x*1.65996547913956e-10))))); i = int(0.5 + v); printf "%8.3f %4d %04X\n", v, i, i }}'You can even use a spreadsheet program, filling A1 to A256 with 0, 1, 2, ..., 253, 254, 255, and filling cell B1 with=ROUND(A1*(2.24999635243848 + A1*(0.11447694372166 - A1*(0.00197198103101253 - A1*(2.3737200908095e-05 -A1*(1.04927937396057e-07 - A1*1.65996547913956e-10))))))and then expanding or copying it down to cells B2, B3, ..., B253, B254, B255.In real life, I'd probably use my own program which reads in the scaled coefficients, then interpolates the 256 points, and outputs the data in the same form as the awk script, one value per line, with the un-rounded scaled flow first, then it rounded to integer in decimal, and in hexadecimal.The output from the awk script, containing the \$f(0)\dots f(255)\$ in the first column, the rounded value, and the rounded value in hexadecimal, isCode:0.000 0 00002.363 2 00024.943 5 00057.729 8 000810.711 11 000B13.880 14 000E17.225 17 001120.738 21 001524.411 24 001828.235 28 001C32.203 32 002036.308 36 002440.544 41 002944.904 45 002D49.383 49 003153.976 54 003658.677 59 003B63.483 63 003F68.389 68 004473.392 73 004978.488 78 004E83.674 84 005488.948 89 005994.307 94 005E99.750 100 0064105.274 105 0069110.879 111 006F116.563 117 0075122.325 122 007A128.166 128 0080134.084 134 0086140.080 140 008C146.154 146 0092152.307 152 0098158.538 159 009F164.850 165 00A5171.243 171 00AB177.719 178 00B2184.279 184 00B8190.925 191 00BF197.659 198 00C6204.482 204 00CC211.398 211 00D3218.408 218 00DA225.515 226 00E2232.722 233 00E9240.032 240 00F0247.447 247 00F7254.970 255 00FF262.606 263 0107270.356 270 010E278.224 278 0116286.214 286 011E294.330 294 0126302.574 303 012F310.951 311 0137319.464 319 013F328.117 328 0148336.914 337 0151345.859 346 015A354.956 355 0163364.208 364 016C373.620 374 0176383.196 383 017F392.940 393 0189402.856 403 0193412.947 413 019D423.219 423 01A7433.675 434 01B2444.318 444 01BC455.154 455 01C7466.187 466 01D2477.419 477 01DD488.856 489 01E9500.501 501 01F5512.358 512 0200524.431 524 020C536.724 537 0219549.241 549 0225561.985 562 0232574.961 575 023F588.171 588 024C601.620 602 025A615.312 615 0267629.248 629 0275643.434 643 0283657.872 658 0292672.566 673 02A1687.519 688 02B0702.735 703 02BF718.215 718 02CE733.963 734 02DE749.983 750 02EE766.277 766 02FE782.847 783 030F799.697 800 0320816.829 817 0331834.245 834 0342851.949 852 0354869.941 870 0366888.225 888 0378906.803 907 038B925.677 926 039E944.848 945 03B1964.318 964 03C4984.090 984 03D81004.165 1004 03EC1024.545 1025 04011045.231 1045 04151066.224 1066 042A1087.526 1088 04401109.137 1109 04551131.060 1131 046B1153.296 1153 04811175.844 1176 04981198.706 1199 04AF1221.883 1222 04C61245.376 1245 04DD1269.184 1269 04F51293.309 1293 050D1317.750 1318 05261342.509 1343 053F1367.586 1368 05581392.980 1393 05711418.692 1419 058B1444.721 1445 05A51471.069 1471 05BF1497.734 1498 05DA1524.716 1525 05F51552.016 1552 06101579.632 1580 062C1607.565 1608 06481635.814 1636 06641664.379 1664 06801693.258 1693 069D1722.452 1722 06BA1751.960 1752 06D81781.781 1782 06F61811.914 1812 07141842.359 1842 07321873.115 1873 07511904.181 1904 07701935.557 1936 07901967.241 1967 07AF1999.232 1999 07CF2031.530 2032 07F02064.134 2064 08102097.043 2097 08312130.256 2130 08522163.773 2164 08742197.592 2198 08962231.712 2232 08B82266.134 2266 08DA2300.855 2301 08FD2335.876 2336 09202371.196 2371 09432406.814 2407 09672442.729 2443 098B2478.941 2479 09AF2515.450 2515 09D32552.255 2552 09F82589.356 2589 0A1D2626.752 2627 0A432664.444 2664 0A682702.432 2702 0A8E2740.715 2741 0AB52779.293 2779 0ADB2818.168 2818 0B022857.340 2857 0B292896.808 2897 0B512936.574 2937 0B792976.639 2977 0BA13017.004 3017 0BC93057.669 3058 0BF23098.637 3099 0C1B3139.908 3140 0C443181.485 3181 0C6D3223.369 3223 0C973265.562 3266 0CC23308.068 3308 0CEC3350.887 3351 0D173394.024 3394 0D423437.482 3437 0D6D3481.262 3481 0D993525.370 3525 0DC53569.808 3570 0DF23614.581 3615 0E1F3659.694 3660 0E4C3705.150 3705 0E793750.955 3751 0EA73797.114 3797 0ED53843.633 3844 0F043890.518 3891 0F333937.775 3938 0F623985.410 3985 0F914033.430 4033 0FC14081.844 4082 0FF24130.658 4131 10234179.882 4180 10544229.522 4230 10864279.589 4280 10B84330.092 4330 10EA4381.041 4381 111D4432.445 4432 11504484.317 4484 11844536.666 4537 11B94589.505 4590 11EE4642.846 4643 12234696.702 4697 12594751.087 4751 128F4806.012 4806 12C64861.495 4861 12FD4917.548 4918 13364974.188 4974 136E5031.430 5031 13A75089.293 5089 13E15147.791 5148 141C5206.945 5207 14575266.771 5267 14935327.290 5327 14CF5388.522 5389 150D5450.486 5450 154A5513.204 5513 15895576.698 5577 15C95640.991 5641 16095706.106 5706 164A5772.067 5772 168C5838.899 5839 16CF5906.628 5907 17135975.281 5975 17576044.884 6045 179D6115.465 6115 17E36187.055 6187 182B6259.681 6260 18746333.376 6333 18BD6408.171 6408 19086484.097 6484 19546561.189 6561 19A16639.480 6639 19EF6719.007 6719 1A3F6799.804 6800 1A906881.910 6882 1AE26965.361 6965 1B357050.199 7050 1B8A7136.461 7136 1BE07224.191 7224 1C387313.430 7313 1C917404.221 7404 1CEC7496.608 7497 1D497590.639 7591 1DA77686.358 7686 1E067783.813 7784 1E687883.055 7883 1ECB7984.132 7984 1F308087.096 8087 1F978192.000 8192 2000The second and third column represent the same number, the second is just in decimal, and third in hexadecimal. These are the 256 values needed for the device calibration.(Because I always go through the results looking for "oddities" before trusting it as-is, I like having the unrounded number next to the results. That way, if I feel I want to nudge one result +1 or -1, I can tell how much actual difference to the fit it makes. For example, "4589.505 4590 11EE" could arguably round to either 4589 or 4590, as picking one or the other differs almost equally much from the unrounded value.)