Remove some unnecessary object creation and switch to HashSet for better performance. See #3308