Remove colorme permissions from moderators