Still init normalized matrices if the VSM is lacking them for some reason