Adjust limitation on pollfd struct to eliminate some polling issues