Don't do connection tracking without security groups

Quick optimisation that makes sure connection tracking flows are only applied for the interfaces that are in security groups.