Fix cleanup when last listener removed

When the last listener for a selector is removed, both the events and selectors objects need to be adjusted.  Also, use the correct key to delete the event state.