extensions: libxt_set, libxt_SET: check the set family too

Do not accept silently sets with wrong protocol family but reject
them with an error message. It makes straightforward to catch user
errors.

Signed-off-by: Jozsef Kadlecsik <kadlec@blackhole.kfki.hu>