Create a new classloader only when needed

This should remove the big performance hit introduced by my previous PR