Remove unnecessary use of XOR operator

* I had a very dark period when I would shove ^ everywhere. That
  has to be fixed and forgotten.