Don't ignore ignored items forever (fixes #816)

The new test in `ignore_test.go` doesn't really add much, since that code was not broken, but I don't think it hurts. The new integration test catches the bug. Please review for further "thinkos" in the fixed part of the walker though, since apparently there was a bunch of conditions missed by me the first time. :)