Refactored the token extraction to allow overrides.