Make packet inspection use sets, for slight speed improvement

About 1% on an Edison.