Various fixes to adding and removing permissions asychronously