Fix the removepoint not removing the point from internal storage.

Also, converted the point list to a hashset so that there aren't any
duplicates fo the points as that could set nasty.