Enforce that we only want bytes in path related code.

Related to a bunch of the todo items in #280