Don't crash if there is no address to associate

Rather, just do nothing. This prevents a crash we've seen in production at
Bonobos.

Fixes #5137