Talk:Hessian matrix

message: This would really be a lot easier to understand if we could see a visual representation.. something like
Hf(x,y) = [Partial 2 derivative with respect to x2], [partial 2 derivative yx]
          [Partial 2 derivative with respect to xy], [partial 2 derivative y2]
"Hessian matrices are used in large-scale optimization problems" looks incorrect to me: for high-dimensional problems second order methods are usually taken into account only for problems with some known exploitable (sparse) structure. In general the hessian matrix is too big to be stored. First order methods are the main choice in general. --Lostella (talk) 09:44, 10 July 2013 (UTC)
thread_title: Initial discussion

message: It would be good to have at least one example of the use of Hessians in optimization problems, and perhaps a few words on the subject of applications of Hessians to statistical problems, e.g. maximization of parameters. --Smári McCarthy 16:01, 19 May 2006 (UTC)
replies: text: I agree.  BriEnBest (talk) 20:00, 28 October 2010 (UTC)

text: Me too. --Kvng (talk) 15:01, 3 July 2012 (UTC)
thread_title:  Examples 

message: The hessian displayed is incorrect, it should be 1\2 of the second derivative matrix.
Charlielewis 06:34, 11 December 2006 (UTC)
thread_title:  HUH?? 

message: I had learned that Fxy=Fyx is Youngs Thereom, not Swartz —The preceding unsigned comment was added by Lachliggity (talk • contribs) 03:02, 16 March 2007 (UTC)
thread_title:  The Thereom is Wrong 

message: It would be nice if someone could include what to do when the determinant of the hessian matrix is zero. I thought you had to check with higher order derivatives, but I'm not too sure. Aphexer 09:52, 1 June 2007 (UTC)
thread_title:  What if det H is zero? 

message: Can someone write on this topic from the point of view of Riemannian geometry?
(there should be links e.g. to covariant derivative).
Commentor (talk) 05:15, 3 March 2008 (UTC)
thread_title:  Riemannian geometry 

message: As I understand it, the Hessian describes the second-order shape of a smooth function in a given neighborhood. So is this right?:
:y=f(\mathbf{x}+\Delta\mathbf{x})\approx f(\mathbf{x}) + J(\mathbf{x})\Delta \mathbf{x} + \Delta\mathbf{x}^\mathrm{T} H(\mathbf{x}) \Delta\mathbf{x}
(Noting that the Jacobian matrix is equal to the gradient for scalar-valued functions.) That seems like it should be the vector-domained equivalent of
:y=f(x+\Delta x)\approx f(x) + f'(x) \Delta x + f(x) \Delta x^2
If that's right, I'll add that to the article. —Ben FrantzDale (talk) 04:30, 23 November 2008 (UTC)
replies: text: That appears to be right, as discussed (crudely) in Taylor_series#Taylor_series_in_several_variables. —Ben FrantzDale (talk) 04:37, 23 November 2008 (UTC)
thread_title:  Local polynomial expansion 

message: Some optimization algorithms (e.g., levmar) approximate the Hessian of a cost function (half the sum of squares of a residual) with J^\top J where J is the Jacobian matrix of r with respect to x:
:J= \frac{\partial r_i}{\partial x_j}. 
For reference, here's the derrivation. I may add it to this page, since the approximation is important and has practical applications (source).
Here's my understanding:
If we have a sum-of-squares cost function:
:f(x) = \frac{1}{2} \|r(x)\|^2 = \frac{1}{2} \sum_k r_k^2
then simple differentiation gives:
:\frac{\partial f}{\partial x_i} = \sum_k \frac{\partial r_k}{\partial x_i} r_k = J^\top r.
Then using the product rule inside the summation:
: \frac{\partial^2 f}{\partial x_i \partial x_j} = \frac{\partial}{\partial x_j}\left[\sum_k \frac{\partial r_k}{\partial x_i} r_k\right]
 = \sum_k \left[ \frac{\partial^2 r_k}{\partial x_i \partial x_j} r_k + \frac{\partial r_k}{\partial x_i}\frac{\partial r_k}{\partial x_j}\right] = \left[\sum_k (\nabla^2 r_k) r_k\right] + J^\top J
:::::=\frac{\sum_k r_k \partial^2 r_k}{\partial x_i \partial x_j} + \frac{\sum_k (\partial r_k)^2}{\partial x_i \partial x_j}
:::::=\frac{\sum_k (\partial r_k)^2}{\partial x_i \partial x_j} + {} H.O.T. for small r_ks.
With all that in mind, for small residuals we can aprpoximate the terms of H with
:H_{ij} = \frac{\partial^2 f}{\partial x_i \partial x_j} \approx \sum_k \frac{\partial r_k}{\partial x_i}\frac{\partial r_k}{\partial x_j} = \sum_k \frac{(\partial r_k)^2}{\partial x_i \partial x_j}=J^\top J.
In words this says that for small residuals, the curvature of f (in any combination of directions) is approximated by the sum of squares of the rate of change of the components of the residuals in the same directions. We ignore the curvature of the residual components weighted the residuals—they are small so they don't have much room to curve and are additionally downweighted by their (small) value. —Ben FrantzDale (talk) 13:24, 10 August 2010 (UTC)
thread_title:  Approximation with Jacobian 

message: There is a problem with notation in the lead. H(f) and H(x) are used, but the arguments are very different. In the first case, f is a function, and in the second case, x is a vector. One should try to unify the notation or at least clarify the differences. Renato (talk) 22:04, 12 September 2011 (UTC)
replies: text: Good point. I added clarification. —Ben FrantzDale (talk) 11:44, 13 September 2011 (UTC)
thread_title:  Notation 

message: Given a scalar smooth function f: \mathbb R^n \to \mathbb R, the gradient will be a vector valued function \operatorname{grad} f: \mathbb R^n \to \mathbb R^n. We can even write g := \operatorname{grad} f, a column vector where the first line is the derivative of f with respect to x_1 and so forth.
Then, the Jacobian of g, as stated in Jacobian matrix and determinant, is a matrix where the i-th column has the derivative of all components of g with respect to x_1. This is the transpose of the Hessian, or is it not? Wisapi (talk) 17:04, 20 January 2016 (UTC)
replies: text: The distinction is moot due to the symmetry of the Hessian if Clairaut's theorem holds for all the partial derivatives concerned, as in most elementary cases. If not, then consider the order of partial differentiation implied by the operators. Then one gets somewhat of a contradictory situation. Every source (Wolfram MathWorld among others) calls it simply the Jacobian of the gradient. But the same sources use the notation in the article, and \frac{\partial^2 f}{\partial x_i \partial x_j} := \frac{\partial}{\partial x_i} (\frac{\partial f}{\partial x_j}) for all i and j. I am in favor of the current statement, however, since the idea is that taking the total derivative twice yields a second derivative, without complications such as this.--Jasper Deng (talk) 17:48, 20 January 2016 (UTC)
replies: text: I also came to the conclusion (much like @Wisapi) that the expression given in the article gives the transpose of the Hessian rather than the Hessian, though I see @Jasper Deng's point with regards to partial derivative notation. Einabla (talk) 06:28, 8 June 2023 (UTC)

text: I am pretty sure the article is inconsistent with the definitions of the Jacobian and Gradient given in their respective articles (it should be the Jacobian of the Gradient, with no transpose).  Isn't it better to be consistent within Wikipedia than to be consistent with Wolfram MathWorld? 169.231.53.221 (talk) 01:02, 14 February 2024 (UTC)
thread_title:  Hessian = (transpose of?) Jacobian of the gradient. 

message: I wonder if it would be worthwhile to add a section on the coordinate independent definition of the Hessian (as a symmetric bilinear form). I know this article is "Hessian matrix," but since the Hessian matrix is just a what you get if you pick local coordinates, I feel like such a section might belong. hiabc (talk) 05:48, 11 January 2021 (UTC)
thread_title:  Coordinate independent definition 

message: I find this article to be very clearly written despite some material's being rather technical.
But one thing that seems to be missing (or vastly underemphasized) is the fact that the hessian matrix defines the quadratic term in the Taylor series of a real-valued function defined on an open subset of some euclidean space.  In other words, the hessian matrix is the matrix of the quadratic form that displays the best approximation to the function (assuming a horizontal tangent plane) at a given point.
I hope someone familiar with this and also with the appropriate version for riemannian manifolds can add this topic to the article, because it seems to be extremely important. 2601:200:C082:2EA0:F956:E02:6231:F380 (talk) 05:44, 26 January 2023 (UTC)
thread_title:  Missing section