Talk:Gauss–Newton algorithm

message: The 'notes' section refers to something called 'the normal equations'. However, nothing of that name is mentioned in the main section ('Description') above the notes. Could the main section be altered to explicitly introduce these normal equations and also explain why they are in need of being solved? The thing is, in the bottom of the main section is an expression for \boldsymbol \beta^{(s+1)}. This expression seems perfect for straight out computation, as the desired component  \boldsymbol \beta^{(s+1)} has been neatly isolated. Why then would we still need to to some 'solving of the normal equations'?  — Preceding unsigned comment added by 87.73.120.49 (talk) 22:24, 19 January 2016 (UTC)
thread_title:  comfusion about 'normal equations' 

message: I always thought that Newton's method, when applied to systems of equation, is still called Newton's method (or Newton-Raphson) and that the Gauss-Newton is a modified Newton's method for solving least squares problems. To wit, let f(x) = sum_i r_i(x)^2 be the least squares problem (x is a vector, and I hope you don't mind my being too lazy to properly type-set the maths). Then we need to solve 2 A(x) r(x) = 0, where A(x) is the Jacobian matrix, so A_ij = derivative of r_j w.r.t. x_i. In my understanding, Newton's method is as described in the article: f'(x_k) (x_{k+1} - x_k) = - f(x_k) with f(x) = 2 A(x) r(x). The derivative f'(x) can be calculated as f'(x) = 2 A(x) A(x)^\top + 2 sum_i r_i(x) nabla^2 r_i(x). On the other hand, the Gauss-Newton method neglect the second term, so we get the iteration A(x_k) A(x_k)^\top (x_{k+1} - x_k) = - A(x_k) r(x_k).
Could you please tell me whether I am mistaken, preferably giving a reference if I am? Thanks. -- Jitse Niesen 18:33, 18 Nov 2004 (UTC)
replies: text: It is quite possible that you are right. Please feel free to improve both this article and the corresponding section in Newton's method. I do not have an appropriate reference at hands, therefore I am unable to contribute to a clarification. - By the way, thanks for the personal message on my talk page. -- Frau Holle 22:23, 20 Nov 2004 (UTC)
replies: text: I moved the description of the general Newton's method in R^n back to Newton's method and wrote here a bit about the modified method for least squares problems.  -- Jitse Niesen 00:45, 5 Dec 2004 (UTC)
thread_title:  Name? 

message: Should the formula not be p(t)=p(t+1)-(J'J)^(-1)J'f?
I thought the RHS was supposed to look like an OLS regression estimate... —The preceding unsigned comment was added by 131.111.8.96 (talk • contribs) 15:48, 10 January 2006  (UTC)
:Please be more precise. Do you want to change the formula
:: p^{k+1} = p^k - (J_f(p^k)\,J_f(p^k)^\top)^{-1} J_f(p^k) \, f(p^k) 
:in the article to read
:: p^k = p^{k+1} - (J_f(p^k)\,J_f(p^k)^\top)^{-1} J_f(p^k) \, f(p^k)? 
:If so, can you give some more justfication than "I thought the RHS was supposed to look like an OLS regression estimate", like a reference? If not, what do you want? -- Jitse Niesen (talk) 18:12, 10 January 2006 (UTC)
replies: text: You're right. Many thanks for your comment. It should be fixed now. -- Jitse Niesen (talk) 03:10, 28 October 2006 (UTC)
thread_title:  Transform of the Jacobians in the wrong place? 

message: What is meant by "The matrix inverse is never computed explicitly in practice"? Right after it is said that a certain linear system should be solved.  How does that system gets solved if not by the inversion of that matrix?
It seems to mee also that there is a pseudoinversion going on there, and this should be made explicit in the article. -- NIC1138 18:17, 21 June 2007 (UTC)
replies: text: I have added more details on how to solve that equation without inverting the full matrix. -- Tim314 19:46, 26 September 2007 (UTC)
thread_title:  no inversion? 

message: Invertible refers to a square matrix. Since J_f(p^k) is m\times n,
it would be more exact to say it is of rank n rather than invertible.
Note that this might invalidate the following argument (using the inverse matrix), so some comment like "for the square case ..., and proper generalization is available for the rectangular case" might be needed. --AmitAronovitch (talk) 11:42, 10 January 2008 (UTC)
thread_title:  Re: Computing the step <math>\delta^k</math> 

message: I would like to see an explicit statement about the different assumptions behind Gauss-Newton and Newton method. The Newton-method-article is quite clear in its assumptions.
Now, in the Gauss-Newton method, why can the Hessian (implicitely) be approximated by    J_f(p)^\top J_f(p) . What is the geometric interpretation? Is the reason a special assumption, e.g. that the function is very close to zero near the true solution and therefore 
 \sum_{i=1}^m f_i(p) \, H(f_i)(p)  is neglectable ?
That would mean that Gauss-Newton (and also Levenberg-Marquardt!) is not suitable for finding minima where the "residual" function value is quite large, e.g. 
f_i(p) = p^2+100000000 
Or from another viewpoint: Given the funtion value and the jacobian there are infinitely many parabolas which are tangent to the function, what's the justification to guess the Hessian to be (close to)   J_f(p)^\top J_f(p)  ?
 
Without this explanation, the update rule given in the article seems too unjustified for me, where does it come from?
I do believe Gauss had good reasons for it :-)
--89.53.92.248 17:18, 6 September 2007 (UTC)
replies: text: Meanwhile I do believe that G.N. is the special case where we have a sum of squared differences (ssd) between functions and "desired function values". In pure Newton method, we would compute the jacobian and hessian of the overall ssd. In Gauss-Newton we dont approximate the ssd but we linearize each of the functions (1st order taylor series), which implies that the hessian of these linearized functions is zero and the resulting "modified" ssd is quadratic in the original parameters. However, I would still like to see an explicit derivation here. --89.53.75.102 10:17, 16 September 2007 (UTC)
replies: text:  I have made additions to the "Derivation" section (also retitled by me), which hopefully address these issues.  In answer to the above question, it's true that Gauss-Newton may have problems in the case of large residuals. Levenberg-Marquardt is more robust in those cases (although I believe Gauss-Newton is usually faster when the minimum is zero). --Tim314 19:54, 26 September 2007 (UTC)
thread_title:  Missing Assumptions - Difference between Gauss-Newton and Newton 

message: I propose that the article Levenberg-Marquardt algorithm be merged into this one.Petergans (talk) 08:28, 4 January 2008 (UTC)
replies: text: I do not think they should be merged. I view them as separate. fnielsen (talk) 09:21, 4 January 2008 (UTC)
thread_title:  Merger proposal 

message: Note that the Newton method can also be considered a special case of the GNA: for minimizing a general function \tilde{S}, choose f_i=\frac{\partial \tilde{S}}{\partial p_i}, then S=\sum {f_i}^2 has as its minima (which are exactly 0) the extrema of \tilde{S}, J_f(p) is the Hessian of \tilde{S}, and the step formula reduces to the Newton step.  —Preceding unsigned comment added by AmitAronovitch (talk • contribs) 12:17, 10 January 2008 (UTC)
thread_title:  Comment that is maybe worth adding 

message: *Least squares has been revised an expanded. This article should serve as in introduction and over-view to the two subsidiary articles, 
**Linear least squares (revised)
**Non-linear least squares (created)
which contain more technical details, but it has sufficient detail to stand on its own.
In addition Gauss-Newton algorithm (this article) has been revised. The earlier article contained a serious error regarding the validity of setting second derivatives to zero. 
Points to notice include:
*Adoption of a standard notation in all four articles mentioned above. This makes for easy cross-referencing. The notation also agrees with many of the articles on regression
*New navigation template
*Weighted least squares should be deleted. The first section is adequately covered in Linear least squares and Non-linear least squares. The second section (Linear Algebraic Derivation) is rubbish.
This completes the fist phase of restructuring of the topic of least squares analysis. From now on I envisage only minor revision of related articles. May I suggest that comments relating to more than one article be posted on talk: least squares and that comments relating to a specific article be posted on the talk page of that article. This note is being posted an all four talk pages and Wikipedia talk:WikiProject Mathematics.
Petergans (talk) 09:22, 8 February 2008 (UTC)
thread_title:  Least squares: implementation of proposal 

message: Peter, I wonder if you agree with what I changed recently in the article (see the history and explanations there of what I did). I plan to further work on it, but want to make sure there are no disagreements so far. Oleg Alexandrov (talk) 04:35, 4 March 2008 (UTC)
replies: text: I've simplified the notation by making it clearer that the normal equations etc. apply at a particular iteration. The principle is that when something applies over a range of equations it simplifies the notation by making this explicit at the beginning, rather than repeat it in each equation. I hope you will agree that as it stands there is no scope for misunderstanding. Petergans (talk) 09:55, 4 March 2008 (UTC)
thread_title:  Small recent notation changes 

message: Don't be so hasty. I've got Bjorck's book out of the library and this has helped to clarify the nature of your objections. Essentially, in data fitting the theory stands as it is. However, in optimization there is no model function. Instead of a model function, functions are defined whose sum of squares is to be minimized, so the theory won't apply. Bjorck gives the even simpler example
r_1=\beta+1,\ r_2=\lambda\beta^2+\beta-1
which is cited in R. Fletcher, Practical Methods of Optimization, Wiley, 1987. Bjorck states that the GN method is not locally convergent for |\lambda| >1. I've plotted the sum of squares and for λ>1 there is a maximum at β=0, which certainly complicates matters. The sum of squares is quartic in β so it's not surprising that there are multiple minima. See non-linear least squares#multiple minima.
What the GN article needs is a clear distinction (separate sections?) between modeling and optimization since the same criteria cannot be applied in both applications. Let me think about it and come up with a proposal. Petergans (talk) 17:35, 7 March 2008 (UTC)
replies: text: Please note that Gauss-Newton is an optimization algorithm, not a data-fitting algorithm. I do not mind you add here some theory of what happens in the data fitting case, but that should not obscure the fact that Gauss-Newton is a general algorithm used in plenty of other applications. Oleg Alexandrov (talk) 19:11, 7 March 2008 (UTC)
thread_title:  Optimization 

message: I made a google books search for Guass-Newton method. Here are the first ten results, with the book title.
#   Learning and Soft Computing
#  Numerical optimization
#  Inverse Problems and Optimal Design in Electricity and Magnetism
# Newton Methods for Nonlinear Problems
# Optimal Control of Complex Structures 
# The Sentinel Method and Its Application to Environmental Pollution Problems
#   Vehicle Routing Problem
# Applied Parameter Estimation for Chemical Engineers
#  Methodology and Tools in Knowledge-Based Systems 
# Nonlinear System Identification
At references 1, 2, 3, 6, 7, 10, the search direction is computed in terms of residuals, 
:(\mathbf{J_r}^T\mathbf{J_r})\delta\boldsymbol\beta^k=-\mathbf{J_r}^T \mathbf{r},
I was not able to find a formula at references 4, 5, 8, 9.
Nowhere in those references have I seen model functions as in this article.
With due respect to Petergans, who is surely an expert in his area of expertise, I do not believe it is appropriate to treat this article using model functions. Model functions will show up in applications of Gauss-Newton to data fitting, but they may not show up in other applications.
We need to state this article in terms of residuals. Any particular application will have its particular residuals. I do not think it is right to state this article only in terms of one particular application. Oleg Alexandrov (talk) 17:07, 8 March 2008 (UTC)
replies: text: To be more precise, the derivation of the algorithm should be in terms of residuals, not model functions. To help the readers at non-linear least squares, an expression for the shift vector using model functions can be stated. I hope this will satisfy Petergans's concern for compatibility of those articles, while stating the article in its proper framework, to be usable for all applications. Oleg Alexandrov (talk) 17:49, 8 March 2008 (UTC)
thread_title:  Gauss-Newton used in the literature 

message: The article states that the increment direction goes downhill. That would usually be called "descent direction" and would mean that \nabla S \cdot \delta \beta < 0. I wonder if this is atually true (can be proved) or if it is folklore again. Here folklore = hopefull wish, often seen in practice, true for many problems of interest, justified by heuristic thinking, etc. These have their place but should be identified as such and not confused with a mathematical statement. The word "tend" tends to be useful for that purpose. Jmath666 (talk) 17:07, 16 March 2008 (UTC)
thread_title:  Downhill? 

message: maybe i am missing something but, skimming through the back and forth, i don't see how Petergans has addressed any mathematical issue brought up by Oleg and Jmath. claiming something is "often" true, without specifying what "often" means exactly, means nothing in mathematics. citing the fact that "GN is used for data fitting of many different kinds without any concern about [ultimate] convergence...", apparently a common practice among non-math folks using this particular piece of math, also means nothing. Mct mht (talk) 00:52, 19 March 2008 (UTC)
replies: text: It may mean nothing in mathematics, but there is more in the world than mathematics :) Many algorithms are used even though we cannot prove they work. I've no problems with saying that something works "often" — in fact, I think we should say this when appropriate — but it should be made clear that this is a practical statement, not a theoretical one (a point that Jmath666 makes above), and it should also ideally carry a reference. -- Jitse Niesen (talk) 12:06, 19 March 2008 (UTC)
thread_title:  sorry to butt in ... 

message: I have implemented the algorithm given in this article in Matlab code. Hoewever I don't know how to add it to the page / format etc. It may be useful.
%GaussNewton_Demo
%Data
x = [0.038 0.1947 0.425 0.626 1.253 2.500 3.740]';
y = [0.05 0.127 0.094 0.2122 0.2729 0.2665 0.3317]';
%PLot X
X = [0.01:0.01:4];
%Inital guess
B1 = 0.9;
B2 = 0.2;
%Plot
figure(1)
cla
hold on
scatter(x,y, 'rd', 'fill')
axis auto
%Loop
for i = 1:5
    
    %Jacobian
    J = [-x./(B2+x) (B1*x)./(B2+x).^2];
    %Hessian (approx)
    H = J'*J;
       
    %Calculate residuals
    r = y - (B1*x)./(B2+x);
    %Sum of squares of residuals
    SSR = sum(r.*r)
    
    %Plot
    Y = (B1*X)./(B2+X);
    plot(X,Y, 'LineWidth', 1)
    
    %Calculate delta
    Delta = pinv(H)*J'*r;
    
    %Apply delta to parameters
    B1 = B1-Delta(1);
    B2 = B2-Delta(2);
    
    pause(1)
    
end  — Preceding unsigned comment added by MartyBebop (talk • contribs) 11:39, 12 April 2012 (UTC)
thread_title:  Demo Code 

message: I think that in the Description section, when the data fitting example is introduced, it should read
: \boldsymbol \beta^{(s+1)} = \boldsymbol \beta^{(s)} + \left(\mathbf{J_f}^\mathsf{T} \mathbf{J_f} \right)^{-1} \mathbf{ J_f} ^\mathsf{T}\mathbf{r}(\boldsymbol \beta^{(s)}).   — Preceding unsigned comment added by 129.240.215.211 (talk) 12:32, 27 May 2015 (UTC)
thread_title:  data fitting 

message: It's there a factor of 1/2 missing in the derivation of \Delta \beta ? Alzibub (talk) 11:44, 9 January 2021 (UTC)
thread_title:  Factor of 1/2 missing