Help requested for the mathematically illiterate  - Page 1
Forum section: electronics / beginners

author: WattsThat
author_rank: Frequent Contributor
timestamp: February 06, 2022, 09:01:56 pm
content: I'm trying to find the function for a graph. I thought I could do it in Excel with solver and while I can get a result, it's not even close. Just using the positive half of the graph, I assumed a quadratic form would give me something useful but it's not even close. I used y = a*x^2+ b*x. Was okay at the start but then diverged toward the end. So, it looks like it needs a polynomial buts there where I run out of guesses, its just not in my skill set.Hopefully I've got the image and data attached correctly. The .txt file is actually a CSV so it can be imported into Excel or whatever tool you might use.Thanks for any help you can provide.

author: ataradov
author_rank: Super Contributor
timestamp: February 06, 2022, 09:28:22 pm
content: What error do you find acceptable?Here is a simple Matlab script for polynomial fitting:Code:clear all;d = dlmread('/tmp/2/data.txt');x = d(51:end,1);y = d(51:end,2);p = polyfit(x, y, 3);y1 = polyval(p, x);figure; hold on;plot(x, y); plot(x, y1);For a 3rd degree polynomial for the positive side only it gives coefficients [-0.0002 0.0323 -0.4231 1.2352] with absolute maximum error of 1.24.Note that it is much easier to fit one half of this curve. If you want to have the complete curve, the degree goes up. The closest result to the previous one os degree 10 with 0.98 error.Also, if you exclude (0,0) point, then the fit is much better with [-0.0002 0.0330 -0.4573 1.6962], giving 0.8793 error.

author: ledtester
author_rank: Super Contributor
timestamp: February 06, 2022, 09:47:38 pm
content: Here is an online polynomial fitter:https://arachnoid.com/polysolve/When pasting the data you can leave the commas -- it will treat it as a space/separator.Entering just the pairs for x >= 0 and playing around with the degree setting yields:Code:Mode: normal x,y analysisPolynomial degree 3, 51 x,y data pairs.Correlation coefficient = 0.9998133198046751Standard error = 0.4810384937464132Output form: mathematical function:f(x) = 1.2352223392164365e+000 * x^0+ -4.2309645666560225e-001 * x^1+ 3.2322906101639952e-002 * x^2+ -1.8224862379502749e-004 * x^3Copyright (c) 2019, P. Lutus -- http://arachnoid.com. All Rights Reserved.

author: WattsThat
author_rank: Frequent Contributor
timestamp: February 06, 2022, 09:59:53 pm
content: Thank you,I must be doing something wrong. Used the formp(x) = ax^3 + bx^2 + cx + dand for a b c d I used: [-0.0002 0.0330 -0.4573 1.6962]The x value of 100 generates an output of 86. I thought perhaps it was precedence in Excel but for the life of me, I cannot find the problem.

author: TimFox
author_rank: Super Contributor
timestamp: February 06, 2022, 10:01:24 pm
content: Fitting a polynomial can give excellent resultsinterpolatingwithin the original domain (set of x-axis values).Extrapolatingoutside that domain usually gives a turnover (where it would fall again in your example).

author: ataradov
author_rank: Super Contributor
timestamp: February 06, 2022, 10:18:44 pm
content: Sorry, the accuracy of my numbers was not nearly enough. Try this [-0.000186159535457 0.033000052520777 -0.457300172655290 1.696223621363360]Matlab number display sucks.

author: T3sl4co1l
author_rank: Super Contributor
timestamp: February 06, 2022, 10:23:45 pm
content: A modified Laplacian seems to work well. Try:Code:=$B$2*(1-1/(1+ABS(A11/$B$1)^$B$3))For A11:A111 = -100:100 (x series), and B1 = 74.2, B2 = 146.72, B3 = 2.64.This gives 11 zeroes across the range, an "unfairly" good fit for only 3 parameters. It's still a bit worse than quantization noise, having a peak error around 0.8.Possibly something with a trig function (say, a polynomial inside cos, or a polynomial in cos(x)), or some manner of special function (Exceldoeshave Bessel functions) might give a more exact fit.Tim

author: WattsThat
author_rank: Frequent Contributor
timestamp: February 06, 2022, 10:52:35 pm
content: Wow! I am so impressed with this community. Thank you all. Now that I've recreated the data with a function, I'm embarrassed to admit that I've been looking in the wrong end of the microscope (an expression from a long ago departed co-worker). Read on if still you're still interested in helping the less capable.I was able to get more than good enough results with the P.Lutus polynomial fit with a 6th order equation. I'm not sure what I was doing wrong in Excel with ataradov's data but that's okay for now, I'll come back to it - and while Tim's Excel file overwhelms me, I will chew on that when I'm not trying to put a fire out, which is what this exercise is all about. Tim's result is very impressive (thank you!) but all I'm trying to do is to come up with the ability to linearize a hydraulic proportional valve's output, that's what the curve is.Now that I can recreate the data, I realize I've got it backwards because what I need is the inverse of the function. I need to pass in a value on the Y axis and come out with the X value.Could someone point me in the correct direction? Yeah, I've been solving the wrong problem.

author: WattsThat
author_rank: Frequent Contributor
timestamp: February 06, 2022, 11:01:36 pm
content: ataradov: Thank you for the additional effort. That data set is much better but I see that there is an inherent problem with using the lower order equation, things go wrong/negative in the bottom 20% of the curve.Still working the inverse function, which is what I really need.

author: IanB
author_rank: Super Contributor
timestamp: February 06, 2022, 11:18:23 pm
content: Seems like you already have good help above, but one additional point is that Excel can often do the fitting for you. Just click on the data series and say "add trendline", then adjust the trendline options appropriately. See illustration below.

author: WattsThat
author_rank: Frequent Contributor
timestamp: February 06, 2022, 11:58:54 pm
content: Quote from: IanB on February 06, 2022, 11:18:23 pmThank you, I had found that Excel feature earlier in the day and stopped at 3rd order. But for the life of me, I could not get Excel to duplicate the results with the displayed equation. Interestedly, the original dataset that ataradov posted out of Matlab was exactly the same data that I got out of Excel for 3rd order. He subsequently updated his post with a better fit.I'm still going in circles trying to figure out how to do an inverse function of what I've got - as that's what I need at this point.Input the Y, get the X.

author: calzap
author_rank: Frequent Contributor
timestamp: February 07, 2022, 12:26:12 am
content: What is the origin of the data … from real measurements, a model, or made-up? It’s best to choose a type of equation for fitting that has some relationship to the mechanism that caused the y-variable to change when the x-variable changed. For example, if the y-variable is the voltage output of an oscillator and the x-variable is time, a trigonometric function would make more sense than a polynomial. Blindly fitting a polynomial is substituting an equation for the table of data, which can be handy, but may bear no more relationship to what is going on than the table itself. And unless there is a mechanistic reason to believe otherwise, the polynomial shouldn’t used outside the range of data that was used to create it.OTOH, if there is a good mechanistic reason to believe a particular type of equation describes the process and there is a good fit, then using the equation outside the range of the data may be justified.Mike in California

author: IanB
author_rank: Super Contributor
timestamp: February 07, 2022, 12:29:02 am
content: Quote from: WattsThat on February 06, 2022, 11:58:54 pmThat depends how much programming you want to do. You should say more about where and why you need this, because it will influence the answer. Even saying where the data came from and what it represents would influence the kind of function that would fit it. Polynomials rarely represent physical systems.If you are on a computer, you could do simple linear interpolation between the points, or you could solve the y = f(x) equation numerically using Newton's method.The inverse function does not fit well to a polynomial as a whole, but you could do a piece-wise fit over different portions of the curve.There are so many options it is hard to give an answer. This kind of question is really hard to respond to, because you are not giving us any clues.

author: ledtester
author_rank: Super Contributor
timestamp: February 07, 2022, 12:57:58 am
content: Quote from: WattsThat on February 06, 2022, 11:58:54 pmOne way is to just reverse the x-y pairs (i.e. make the x's the y's and the y's the x's) and get a polynomial fit.Of course, you can only do this on a part of the function where it is either only increasing or only decreasing.

author: T3sl4co1l
author_rank: Super Contributor
timestamp: February 07, 2022, 01:20:15 am
content: There's probably a convenient formula inverting a polynomial, though I don't know of one offhand. Would have to assume it's one-to-one, i.e. not kinked, and odd order so it doesn't double back on itself; then again, those problems sound very much like just another factorization problem...More simply, just swap X and Y, plot the 0-100 part or whatever, and use the same curve fit operation.Also, the LINEST function does the same thing at the cell level, as the curve fit does graphically; this is kind of a faff so I won't go into detail, but you can look up many excellent tutorials showing its use. (The trickiest part is using the CTRL+SHIFT+ENTER array entry method, and remembering which cells in the result are which.) This would just be of interest if you want more direct control over the input data, and using the output directly in other formulas.The modified Laplace is easily inverted: try putting=(1/(1-B11/$I$2)-1)^(1/$I$3)*$I$1in cell L11 in the spreadsheet, and drag it down to fill the range. Of course, this takes the positive result.What is this going into, anyway? If an embedded application (e.g. Arduino), the exponentiation and division will probably be quite slow. Fine if you're only doing some hundreds or thousands of evaluations per second, but you'll need a more powerful microcontroller to do it much faster (one of the upper scale Arduinos, or, like, a rPi or something). Polynomials typically excel on small MCUs so I would suggest that solution, if that's the case.Oh, I didn't mention, the "even poly" is a fit (found using Solver) at double order, which is because it fits the whole curve (it's an even function, so an even order polynomial is needed; odd terms will just be noise). This is about the limit of what's feasible for Solver to do; high order polynomials converge rather poorly (takes a long time to find a solution, or gives up completely), and need tricks to help it out. The results should be similar to the regression, in any case. (Which is the biggest trick of all: solving for the polynomial directly by a rather convenientbit of math. Don't worry about reading that, just appreciate it's a "relatively" simple formula that Excel uses internally to solve the polynomial -- no need for the iterations and approximation that we are resorting to when using Solver!)Tim

author: WattsThat
author_rank: Frequent Contributor
timestamp: February 07, 2022, 01:30:56 am
content: It will eventually be a lookup table in a PLC.The answer was right in front of me, thank you ledtester for the website.

author: Cerebus
author_rank: Super Contributor
timestamp: February 07, 2022, 01:41:41 am
content: Looks like a distorted (peak compressed) cosine wave to me, and a quick manual curve fit seems to match my intuitions. Whatever this is, I suspect the underlying (pure) behaviour is sinusoidal.

author: IanB
author_rank: Super Contributor
timestamp: February 07, 2022, 02:18:54 am
content: Quote from: T3sl4co1l on February 07, 2022, 01:20:15 amI would lay odds you do know this one. A quadratic is inverted with the well known quadratic formula, a cubic can be done but the algebra is more complex, and for higher orders it is not worth the trouble. For anything above fourth order there is no algebraic solution.QuoteI think this is not true. It only needs to be "even" in the range of the fit. What it does outside that doesn't matter (if you do care, then any higher order polynomial will be a disaster). Within the range of the fit, every term can help.

author: IanB
author_rank: Super Contributor
timestamp: February 07, 2022, 02:19:45 am
content: Quote from: WattsThat on February 07, 2022, 01:30:56 amI don't get it. If it's a lookup table, why do you want to fit an equation to it?

author: WattsThat
author_rank: Frequent Contributor
timestamp: February 07, 2022, 03:55:44 am
content: Quote from: IanB on February 07, 2022, 02:19:45 amI really needed the inverse function as I had to generate the data to put into the table somehow... All I had was the curve on paper and I wanted a 256 entry table. I suppose I could have generated it by hand, given the amount of time I spent on it. But, I did learn quite a few things along the way so it was a worthwhile effort.

author: T3sl4co1l
author_rank: Super Contributor
timestamp: February 07, 2022, 04:53:22 am
content: Quote from: IanB on February 07, 2022, 02:18:54 amWell... there would be that.Well, that's for y=0 though.Oh, right, just change the constant term c_n --> c_n - y. That would do.QuoteOdd values can only ever subtract on the left side and add on the right, or vice versa; therefore they would give an increased error regardless of sign, except when zeroed.For example, turning on the trendline and cranking it to 6th order, I get "y = -2E-12 x^6 + 6E-20 x^5 - 7E-07 x^4 + 4E-14 x^3 + 0.0177 x^2 + 1E-09 x - 1.7838", notice the alternating magnitudes with none of the odd orders being above ~1e-14 except the first at a similarly minuscule 1e-9.This is different when fitting to just half the data, of course!Tim

author: IanB
author_rank: Super Contributor
timestamp: February 07, 2022, 05:05:45 am
content: Quote from: WattsThat on February 07, 2022, 03:55:44 amBut you had the table already. You attached it to the first post? I'm still confused.

author: T3sl4co1l
author_rank: Super Contributor
timestamp: February 07, 2022, 06:33:29 am
content: Do you just want it interpolated to full scale e.g. ?Code:00120226331434537640742844947104811501252135414551557165817601861196220642165226623672469257026712772287329743075317632773378347935803681378238833984408541864286438744884589469047914892499250935194529553965496559756985799581005910060101611026210363103641046510566106671066810769108701087110972110731117411175112761137711378114791158011681116821178311884118851198612087120881218912290122911239212493124941259512696126971279812899128100129101130102130103131104132105132106133107134108134109135110136111136112137113138114138115139116140117140118141119142120142121143122144123144124145125146126146127147128148129148130149131150132150133151134152135152136153137154138154139155140156141156142157143158144158145159146160147160148161149162150163151163152164153165154165155166156167157167158168159169160169161170162171163172164172165173166174167174168175169176170177171177172178173179174179175180176181177182178182179183180184181185182185183186184187185188186188187189188190189191190191191192192193193194194195195195196196197197198198199199200199201200202201203202204203205204206204207205208206209207210208211209212209213210214211215212216213217214218215219216220217221217222218223219224220225221226222227223228224229225230226231227232228233229234230235231236232237233238234239235240236241237242238243239244240245241246242247243248244249246250247251248252249253250254251255252(...Would that be integer (byte?) format or float or what? 256 seems to suggest bytes, but I have no idea what your PLC is capable of...)Tim

author: Xandinator
author_rank: Contributor
timestamp: February 11, 2022, 03:12:10 am
content: This could usehttps://en.wikipedia.org/wiki/Chebyshev_polynomialsif it ought to approximate some kernelly stuff...