A method of sampling output audio samples, the method comprising: during a packet loss concealment event in an active voice communication session: obtaining, by data processing hardware, a sequence of previous output audio samples during a time window having a start time and an end time, the end time occurring when the packet loss concealment event commences; and at each time step of a plurality of time steps during the packet loss concealment event: generating, by data processing hardware, using a speech synthesis model, a probability distribution (164, 342) over possible output audio samples for the corresponding time step, each possible output audio sample in the probability distribution (164, 342) comprising a respective probability indicating a likelihood that the corresponding possible output audio sample represents a portion of an utterance at the corresponding time step; determining, by the data processing hardware, a temperature value for tempered sampling based on a function of a number of time steps in the plurality of time steps that precedes the corresponding time step, an initial temperature value, a minimum temperature value, and a maximum temperature value; applying, by the data processing hardware, the temperature value to the probability distribution (164, 342) to adjust a probability of selecting possible output audio samples from the probability distribution (164, 342); randomly selecting, by the data processing hardware, one of the possible output audio samples of the probability distribution (164, 342) based on the adjusted probability associated with each of the possible output audio samples; and generating, by the data processing hardware, synthesized speech using the randomly selected output audio sample. obtaining, by data processing hardware, a sequence of previous output audio samples during a time window having a start time and an end time, the end time occurring when the packet loss concealment event commences; and at each time step of a plurality of time steps during the packet loss concealment event: generating, by data processing hardware, using a speech synthesis model, a probability distribution (164, 342) over possible output audio samples for the corresponding time step, each possible output audio sample in the probability distribution (164, 342) comprising a respective probability indicating a likelihood that the corresponding possible output audio sample represents a portion of an utterance at the corresponding time step; determining, by the data processing hardware, a temperature value for tempered sampling based on a function of a number of time steps in the plurality of time steps that precedes the corresponding time step, an initial temperature value, a minimum temperature value, and a maximum temperature value; applying, by the data processing hardware, the temperature value to the probability distribution (164, 342) to adjust a probability of selecting possible output audio samples from the probability distribution (164, 342); randomly selecting, by the data processing hardware, one of the possible output audio samples of the probability distribution (164, 342) based on the adjusted probability associated with each of the possible output audio samples; and generating, by the data processing hardware, synthesized speech using the randomly selected output audio sample. generating, by data processing hardware, using a speech synthesis model, a probability distribution (164, 342) over possible output audio samples for the corresponding time step, each possible output audio sample in the probability distribution (164, 342) comprising a respective probability indicating a likelihood that the corresponding possible output audio sample represents a portion of an utterance at the corresponding time step; determining, by the data processing hardware, a temperature value for tempered sampling based on a function of a number of time steps in the plurality of time steps that precedes the corresponding time step, an initial temperature value, a minimum temperature value, and a maximum temperature value; applying, by the data processing hardware, the temperature value to the probability distribution (164, 342) to adjust a probability of selecting possible output audio samples from the probability distribution (164, 342); randomly selecting, by the data processing hardware, one of the possible output audio samples of the probability distribution (164, 342) based on the adjusted probability associated with each of the possible output audio samples; and generating, by the data processing hardware, synthesized speech using the randomly selected output audio sample.