Fixes for issuer and audience only being checked when they have values

Added unit tests and fixes. Also fixed a unit test in line #69.