You need to validate that X is not sparse before this (even if `with_centering=False`), I think.