A new pathogen, for example, might kill a large proportion of a population. A few individuals survive because they happen to be more resistant to the new pathogen. The species will be fine if pathogen A is present, but would go extinct if predator B appeared. The source of variability in a species is mutation. Sexual reproduction increases variability of offspring by shuffling gene combinations, but sex by itself doesn't introduce any new variability.
Mutations accumulate over time.
Most are neutral or detrimental, but some are useful. If a population is reduced to a small number it takes time for mutations to accumulate in the population again. If a population is reduced to the point where there is little variability left, it will be extremely vulnerable to any environmental change. If one individual is susceptible, it's likely they all are. There are thousands of seriously deleterious alleles documented in humans, but each person carries only a handful.