Make sure we compare against the connection IP.