Update semver

This closes the loop on my work of getting semver related stuff out of cargo.

This is also built on top of my other PR, so it's not really ready until that actually merges. But it can stand for review now :)