Fix colours disappearing when permissions were removed