A method for activating speaker-dependent warm words, the method comprising: receiving, at data processing hardware, audio data corresponding to an utterance spoken by a user and captured by an assistant-enabled device associated with the user, the utterance comprising a command for a digital assistant to perform a long-standing operation, wherein a long-standing operation is an operation performed by the digital assistant for an extended duration of time; after receiving the audio data corresponding to the utterance: activating, by the data processing hardware, a set of one or more warm words each associated with a respective action for controlling the long-standing operation; and associating, by the data processing hardware, the activated set of one or more warm words with only the user that spoke the utterance; and while the digital assistant is performing the long-standing operation: receiving, at the data processing hardware, additional audio data corresponding to an additional utterance captured by the assistant-enabled device; identifying, by the data processing hardware, in the additional audio data, one of the warm words from the activated set of one or more warm words; performing, by the data processing hardware, speaker verification on the additional audio data to determine whether the additional utterance was spoken by the same user that is associated with the activated set of one or more warm words; and when it is determined by the speaker verification that the additional utterance was spoken by the same user that is associated with the activated set of one or more warm words, performing, by the data processing hardware, the respective action associated with the identified one of the warm words for controlling the long-standing operation. receiving, at data processing hardware, audio data corresponding to an utterance spoken by a user and captured by an assistant-enabled device associated with the user, the utterance comprising a command for a digital assistant to perform a long-standing operation, wherein a long-standing operation is an operation performed by the digital assistant for an extended duration of time; after receiving the audio data corresponding to the utterance: activating, by the data processing hardware, a set of one or more warm words each associated with a respective action for controlling the long-standing operation; and associating, by the data processing hardware, the activated set of one or more warm words with only the user that spoke the utterance; and activating, by the data processing hardware, a set of one or more warm words each associated with a respective action for controlling the long-standing operation; and associating, by the data processing hardware, the activated set of one or more warm words with only the user that spoke the utterance; and while the digital assistant is performing the long-standing operation: receiving, at the data processing hardware, additional audio data corresponding to an additional utterance captured by the assistant-enabled device; identifying, by the data processing hardware, in the additional audio data, one of the warm words from the activated set of one or more warm words; performing, by the data processing hardware, speaker verification on the additional audio data to determine whether the additional utterance was spoken by the same user that is associated with the activated set of one or more warm words; and when it is determined by the speaker verification that the additional utterance was spoken by the same user that is associated with the activated set of one or more warm words, performing, by the data processing hardware, the respective action associated with the identified one of the warm words for controlling the long-standing operation. receiving, at the data processing hardware, additional audio data corresponding to an additional utterance captured by the assistant-enabled device; identifying, by the data processing hardware, in the additional audio data, one of the warm words from the activated set of one or more warm words; performing, by the data processing hardware, speaker verification on the additional audio data to determine whether the additional utterance was spoken by the same user that is associated with the activated set of one or more warm words; and when it is determined by the speaker verification that the additional utterance was spoken by the same user that is associated with the activated set of one or more warm words, performing, by the data processing hardware, the respective action associated with the identified one of the warm words for controlling the long-standing operation.