> if there is a firewall but the user is not logged in because anonymous access is allowed, there is a context but no token.

Are you sure ? For me even a anonymous user have a token : https://github.com/symfony/symfony/blob/master/src/Symfony/Component/Security/Core/Authentication/Token/AnonymousToken.php