Implementing a Decomposition Method
By: Bhanu Soni
Tags: Machine Learning, Linear Regression, Artificial Intelligence, TensorFlow, Data Science

We will implement a matrix decomposition method for linear regression.

Specifically, we will use the Cholesky decomposition, for which relevant functions exist in TensorFlow.

What is the Cholesky decomposition?

The Cholesky decomposition or Cholesky factorization is a decomposition of a Hermitian, positive-definite matrix into the product of a lower triangular matrix and its conjugate transpose.

Getting ready…

Implementing inverse methods in the previous blog (Linear Regression with TensorFlow)can be numerically unfit in most cases, especially when the matrices get very large. Another approach is to decompose the A matrix and perform matrix operations on the decompositions instead. One such approach is to use the built-in Cholesky decomposition method in TensorFlow. One reason people are so interested in decomposing a matrix into more matrices such that the resulting matrices will have assured properties that allow us to use certain methods efficiently.

The Cholesky decomposition decomposes a matrix into a lower and upper triangular matrix, say L and L’, such that these matrices are transpositions of each other.

we will solve the system,, Ax=b as LL’*x=b. we will first solve for Ly=b, then L’x=b in order to get our coefficient matrix, x.

How to do it…

We will set up the system exactly in the same way as the previous recipe. We will import libraries, initialize the graph, and create the data. Then we will obtain our A matrix and b matrix in the same way as we did in the previous blog Using the Matrix Inverse Method:

import matplotlib.pyplot as plt

import numpy as np

import tensorflow as tf from tensorflow.python.framework import ops

ops.reset_default_graph()

sess = tf.Session() x_vals = np.linspace(0, 10, 100)

y_vals = x_vals + np.random.normal(0, 1, 100)

x_vals_column = np.transpose(np.matrix(x_vals)) ones_column = np.transpose(np.matrix(np.repeat(1, 100))) A = np.column_stack((x_vals_column, ones_column))

b = np.transpose(np.matrix(y_vals)) A_tensor = tf.constant(A)

b_tensor = tf.constant(b)

2. Next we will find the Cholesky decomposition of our square matrix, A’A A’=transpose of A

Note that the TensorFlow function, cholesky(), only returns the

lower diagonal part of the decomposition. This is fine, as the upper

diagonal matrix is just the lower one, transposed.

tA_A = tf.matmul(tf.transpose(A_tensor), A_tensor)

L = tf.cholesky(tA_A)

tA_b = tf.matmul(tf.transpose(A_tensor), b) sol1 = tf.matrix_solve(L, tA_b)

sol2 = tf.matrix_solve(tf.transpose(L), sol1)

3. Now that we have the solution, we extract the coefficients:

y_intercept = solution_eval[1][0] print('slope: ' + str(slope))

print('y'_intercept: ' + str(y_intercept))

#slope: 0.956117676145 y_intercept: 0.136575513864 best_fit = []

for i in x_vals:

best_fit.append(slope*i+y_intercept)

plt.plot(x_vals, y_vals, 'o', label='Data')

plt.plot(x_vals, best_fit, 'r-', label='Best' fit line',

linewidth=3)

plt.legend(loc='upper left')

plt.show()

Data points and best-fit line obtained via Cholesky decomposition.

CONCLUSION

Keep in mind that this way of decomposing a matrix, then performing our operations on the pieces, is sometimes much more efficient and numerically stable