Use stored isogeny matrices instead of recomputing on the fly