[RFC] Refactor codebase

cc @mlubin @IainNZ 
I'm assuming it's the same for max and min, just with a sign flip.