Fixes for issuer and audience only being checked when they have values
Merge pull request #8 from dschenkelman/issues/7

Fixes for issuer and audience only being checked when they have values