Remove annotations to ignore when removing annotations