Fixed dragging issue

If the add or remove button was clicked on the POI and the mouse didn't
remain exactly still the click event would not have been registered. Now
the add or remove button is no longer a handle for the sortable ULs.