Converted the ArrayList to a HashSet, which is much faster to add/remove from.