A method comprising: at an electronic device having one or more processors and memory: receiving a user utterance; determining, based on a text representation of the user utterance, one or more user intents corresponding to the user utterance; determining a belief state for the one or more user intents; mapping, based on a policy model, the belief state to a plurality of candidate policy actions; selecting, based on the belief state and the policy model, a policy action from the plurality of candidate policy actions; performing the policy action, including outputting results of the policy action for presentation; determining a success score for the policy action based on whether one of a plurality of types of user input is detected after performing the policy action, wherein the success score represents a degree that the performed policy action coincides with a user desired goal for the user utterance, and wherein the plurality of types of user input are each a type of user input other than a response to a structured device query; and modifying a set of parameter values of the policy model in accordance with the determined success score, wherein the user utterance is a current user utterance comprising user speech from one turn of a multi-turn dialogue conversation between a user and a digital assistant system, wherein the belief state is determined based on a previous belief state which describes a state of the multi-turn dialogue prior to receiving the user utterance, and wherein determining the belief state includes updating the previous belief state with current information and/or observations obtained based on the current user utterance. at an electronic device having one or more processors and memory: receiving a user utterance; determining, based on a text representation of the user utterance, one or more user intents corresponding to the user utterance; determining a belief state for the one or more user intents; mapping, based on a policy model, the belief state to a plurality of candidate policy actions; selecting, based on the belief state and the policy model, a policy action from the plurality of candidate policy actions; performing the policy action, including outputting results of the policy action for presentation; determining a success score for the policy action based on whether one of a plurality of types of user input is detected after performing the policy action, wherein the success score represents a degree that the performed policy action coincides with a user desired goal for the user utterance, and wherein the plurality of types of user input are each a type of user input other than a response to a structured device query; and modifying a set of parameter values of the policy model in accordance with the determined success score, receiving a user utterance; determining, based on a text representation of the user utterance, one or more user intents corresponding to the user utterance; determining a belief state for the one or more user intents; mapping, based on a policy model, the belief state to a plurality of candidate policy actions; selecting, based on the belief state and the policy model, a policy action from the plurality of candidate policy actions; performing the policy action, including outputting results of the policy action for presentation; determining a success score for the policy action based on whether one of a plurality of types of user input is detected after performing the policy action, wherein the success score represents a degree that the performed policy action coincides with a user desired goal for the user utterance, and wherein the plurality of types of user input are each a type of user input other than a response to a structured device query; and modifying a set of parameter values of the policy model in accordance with the determined success score, wherein the user utterance is a current user utterance comprising user speech from one turn of a multi-turn dialogue conversation between a user and a digital assistant system, wherein the belief state is determined based on a previous belief state which describes a state of the multi-turn dialogue prior to receiving the user utterance, and wherein determining the belief state includes updating the previous belief state with current information and/or observations obtained based on the current user utterance.