A method for privacy-preserving face matching performed by a data processing apparatus, the method comprising: accessing, at a data processing apparatus, a first digital photograph submitted by a first user identified by a first user identifier; generating a first temporary face template in non-persistent memory for each face detected in the first digital photograph; determining, based on first user's contacts, second user identifiers that are associated with the first user identifier, the second user identifiers identifying respective second users; accessing a digital photograph index that indexes photographs by user identifiers, and that specifies, for each stored digital photograph and for each user identifier indexing the stored digital photograph, an area in the stored digital photograph in which a face of a user identified by the user identifier is located; selecting the second user identifiers, and for each of the selected second user identifiers, generating a second temporary face template in non-persistent memory for the second user identifier from each of the faces of the respective second users in the areas of the stored digital photographs indexed by the second user identifier including at least two stored digital photographs having a different view of the face of the second user identified by the second user identifier; identifying the first temporary face templates that match the second temporary face templates; for each first temporary face template that matches a second temporary face template, generating data specifying the area in the first digital photograph in which is located the face of the second user, identified by the second user identifier of the second temporary face template, wherein each temporary face template is stored in non-persistent memory and is not persisted; and deleting the first temporary face templates and the second temporary face templates after identifying the first temporary face templates that match the second temporary face template to preserve privacy. accessing, at a data processing apparatus, a first digital photograph submitted by a first user identified by a first user identifier; generating a first temporary face template in non-persistent memory for each face detected in the first digital photograph; determining, based on first user's contacts, second user identifiers that are associated with the first user identifier, the second user identifiers identifying respective second users; accessing a digital photograph index that indexes photographs by user identifiers, and that specifies, for each stored digital photograph and for each user identifier indexing the stored digital photograph, an area in the stored digital photograph in which a face of a user identified by the user identifier is located; selecting the second user identifiers, and for each of the selected second user identifiers, generating a second temporary face template in non-persistent memory for the second user identifier from each of the faces of the respective second users in the areas of the stored digital photographs indexed by the second user identifier including at least two stored digital photographs having a different view of the face of the second user identified by the second user identifier; identifying the first temporary face templates that match the second temporary face templates; for each first temporary face template that matches a second temporary face template, generating data specifying the area in the first digital photograph in which is located the face of the second user, identified by the second user identifier of the second temporary face template, wherein each temporary face template is stored in non-persistent memory and is not persisted; and deleting the first temporary face templates and the second temporary face templates after identifying the first temporary face templates that match the second temporary face template to preserve privacy.