Be more careful with async code

Don't mess with data the main thread uses in an async thread - do it on
the main thread