Only guard version mismatches when given a valid semver specifier

Fixes #136