You can't separate the reference from the referent. Use references when you can, and pointers when you have to. This usually means that references are most useful in a class's public interface. References typically appear on the skin of an object, and pointers on the inside. E.g., programmers should write code in the language of the problem rather than the language of the machine.
Ambiguity is actually an asset in certain cases.
For example, during early design you might not be ready to commit to a specific representation for the handles. If you merely know that you'll need some sort of thingy that will uniquely identify and get to an object, you call the thingy a Handle. Novices often think in terms of pointers, but in reality there are downside risks to using raw pointers. Most of the time we add more layers of indirection to manage situations like these. The point is that we use the word Handle when we don't yet know the details of what we're going to do.