Mark the factories out if no XML, should allow more GC