Handle problem of remove_reader being called too many times