Have idle timeout also expose remote ip, just like request timeout #548