core: Use correct fd from pollfds