Don't keep the expected and actual for a passed expectation

- It causes memory problems in phantom and you probably don't need it
  anyways.

Fixes #640 Fixes #690