ENH: Add scalar special cases to Priority getter

Exact Python scalars can never have a priority, but checking
it is expensive. This adds checks for these to the Priority
getter function.
Merge pull request #3951 from seberg/scalar-priority

ENH: Add scalar special cases to Priority getter