Fix the Clang warning by a better approach: use template specialization

* This adds some minor duplication of trivial code
  but give us a cleaner compile-time check.