libxtables: fix parsing of dotted network mask format

After upgrade from iptables 1.4.8 to 1.4.18 netmask parsing got broken:

-A foo -m policy --mode tunnel --dir in --tunnel-src 192.168.123.0/255.255.255.0 -j RETURN

With iptables 1.4.18:
iptables-restore v1.4.18: policy: bad value for option "--tunnel-src", or out of range (0-32)

This was probably broken by the augmented parser.

Reported-by: Thomas Jarosch <thomas.jarosch@intra2net.com>
Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>