Hessian Matrix and Optimization Problems in Python 3.8 | by Louis Brulé Naudet | Towards Data Science

Recommendations
Compatibility test performed with Python 3.8, executed on MacOS 11.3 and Linux Ubuntu Server 20.04 LTS environments.
Libraries Used : Numpy, Sympy.
pip3.8 install numpy sympy
Hessian matrices are used in large-scale optimization problems within Newton-type methods because they are the coefficient of the quadratic term of a local Taylor expansion of a function. Partial derivatives play a prominent role in economics, in which most functions describing economic behaviour posit that the behaviour depends on more than one variable. For example, a societal consumption function may describe the amount spent on consumer goods as depending on both income and wealth; the marginal propensity to consume is then the partial derivative of the consumption function with respect to income.
The Hessian matrix is also commonly used for expressing image processing operators in image processing and computer vision (see the Laplacian of Gaussian (LoG) blob detector). The Hessian matrix can also be used in normal mode analysis to calculate the different molecular frequencies in infrared spectroscopy.
The Hessian matrix of a numerical function is the square matrix, noted H(f), of its second partial derivatives. In mathematics, a partial derivative of a function of several variables is its derivative with respect to one of those variables, with the others held constant.
Example:
The gradient vector can be interpreted as the “direction and rate of fastest increase”.
If the gradient of a function is non-zero at a point p, the direction of the gradient is the direction in which the function increases most quickly from p, and the magnitude of the gradient is the rate of increase in that direction, the greatest absolute directional derivative.
In an ordered set E, an element of a part A is the largest element or maximum of A, if it belongs to A and is greater than any other element of A. The existence of a maximum is in general not guaranteed for any part of an ordered set. On the other hand, under the existence condition, such an element is unique. Similarly, the smallest element or minimum is, if it exists, an element of A smaller than any other element of A.
The objective is to determine the maximum or minimum candidate by solving the equation :
Implementation in Python 3.8 is pretty simple, and needs the “solve” function in Sympy library.
Now, we need to perform the second derivative to obtain the hessian matrix :
By the way, here is the main function of the program, which arbitrates the assignment of variables between all blocks of instructions :
In our program, we apply Schwarz’s theorem on the second partial derivatives of a function of several variables such that :
However, we will not prove this theorem, nor will we try to explain it in this article. Here is an example of hessian matrix in numpy.matrix format, for the function :
Hessian matrix that organizes all the second partial derivatives of the function x**2–1.5*x*y + y**2 is : [[2 -1.50000000000000][-1.50000000000000 2]]Determinant in the critical point {x: 0.0, y: 0.0} is : 1.75000000000000
The determinant is a scalar value that is a function of the entries of a square matrix.
It allows characterizing some properties of the matrix and the linear map represented by the matrix. In particular, the determinant is nonzero if and only if the matrix is invertible, and the linear map represented by the matrix is an isomorphism.
Thus, for positive-semidefinite and negative-semidefinite Hessians the test is inconclusive (a critical point where the Hessian is semidefinite but not definite may be a local extremum or a saddle point).
In our example, for the critical point (0; 0), the determinant is 1.75 > 0 and f’xx > 0, then, the critical point is a local minimum, so the function is strictly convex.
Tensorflow or other machine learning libraries are certainly powerful, but they are still excessively resource-intensive and can be an obstacle for low-performance machines, this article was intended to interpret a new way to build Hessian matrices, with a lighter tool for scientific computing: sympy.
Louis Brulé Naudet, holds a double degree in Law and Economics/Management from the University of Paris-Saclay.