Make connection cache aware of port and flags.