A system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: receiving, at a server and from a speech-enabled device that is configured to process utterances of multiple different users, an utterance from a user that requests to initiate a phone call to a recipient, wherein the recipient is identified by the utterance; determining, by the server, whether speech in the utterance matches speech corresponding to a particular known user and classifying, based on the determination, that the utterance was spoken by the particular known user of the multiple different users; in response to classifying that the utterance was spoken by a particular known user, determining, by the server, that the particular known user is associated with a personal phone number, the personal phone number associated with a mobile computing device of the particular known user; determining, by the server, a recipient number to call based on: obtaining contact entries created by the particular known user; identifying a particular contact entry from among the contact entries, where a name of the particular contact entry matches the recipient that is identified by the utterance; and determining a phone number indicated by the particular contact entry as the recipient number to call; receiving, at the speech-enabled device and from the server, the recipient number, the personal phone number and an instruction to initiate the phone call to the recipient number with the personal phone number; and in response to receiving the instruction to initiate the phone call, initiating, by the speech-enabled device and using the personal phone number, the phone call to the recipient number. receiving, at a server and from a speech-enabled device that is configured to process utterances of multiple different users, an utterance from a user that requests to initiate a phone call to a recipient, wherein the recipient is identified by the utterance; determining, by the server, whether speech in the utterance matches speech corresponding to a particular known user and classifying, based on the determination, that the utterance was spoken by the particular known user of the multiple different users; in response to classifying that the utterance was spoken by a particular known user, determining, by the server, that the particular known user is associated with a personal phone number, the personal phone number associated with a mobile computing device of the particular known user; determining, by the server, a recipient number to call based on: obtaining contact entries created by the particular known user; identifying a particular contact entry from among the contact entries, where a name of the particular contact entry matches the recipient that is identified by the utterance; and determining a phone number indicated by the particular contact entry as the recipient number to call; obtaining contact entries created by the particular known user; identifying a particular contact entry from among the contact entries, where a name of the particular contact entry matches the recipient that is identified by the utterance; and determining a phone number indicated by the particular contact entry as the recipient number to call; receiving, at the speech-enabled device and from the server, the recipient number, the personal phone number and an instruction to initiate the phone call to the recipient number with the personal phone number; and in response to receiving the instruction to initiate the phone call, initiating, by the speech-enabled device and using the personal phone number, the phone call to the recipient number.