Add a special matrix with cache for inverse