FIX: call remove observer for removeAllListeners

Forgot to add an extra conditional that will call the remove observer if
a callback exists.