Temporary optimisation for making module loading faster

Semi-serious benchmark showed doing spectest ~1% faster with this.