Straighten out time comparison

I didn't realize what Timespec was and was make my life harder for no
reason.