ENH: Small perf optimization to the cublas matrix.  TODO: Need to remove memory copies throughout the design, that it is what is hold back perf.