Go back to only allocating the rows and cols we need.