You're checking the errno left over by the last recv(), not the one left by poll().