Some optimizations

Firstly, eliminated a stupid if statement which could be replaced by a
for loop. Secondly, on systems with no IPv6 support, there are no
unnecessary pollfd structures being allocated anymore.