core: fix possible integer underflow and invalid iterator deref