Revamp build steps

* Instead of providing the binaries with the package, it'll only download the
needed binary for the users system.
* Will only build if the tests fail.
* Move `build.js` into scripts since it's not part of the actual API.
* Move binaries to `vendor` folder to keep it separate from the rest of the code.

Fixes #504. Should probably merge #508 first and then rebase this.