Only off given handler when specified

Previously if a namespace was given then all handlers for the namespace
were off'ed event in cases where the handler to off was specified.