Support setting remote_addr properly on requests