[Incomplete] Performance fixes

Don't pull this yet, I'm not done.

I will be breaking config formats, but that's OK because you haven't released the plugin yet.