Making sure the the main address is used when rewriting workers reply to
address

Fixes #2051