Revert "Revert "Removed SFINAE so that error messages point to the right place""

This reverts commit ad074dcfdb6cdfb7cafbe88f8ef80ace65700d93.