Improve Flag setting and unsetting

Use Bitmasks to set/unset flag bits instead of checking for current
state and flipping if needed.