Irrefut patterns refactoring

Various bug fixes and simplifications I did as part of a patch to fix `let` patterns.

r? @catamorphism