The reasons for this are as follows. First, remember that the main impact of genetic drift on a population is that it decreases genetic variation. Genetic drift tends to get rid of genetic variation. It makes all individuals genetically similar. As a result, there are few differences and little potential to have some higher fitness forms and some lower fitness forms, so there is little potential for natural selection to occur.
Genetic drift can also cause increases in alleles that lead to low fitness, just by chance.
Now let's consider situations in which we expect small populations to exist, that is, situations in which there will be strong effects of genetic drift. Some populations are small and remain small over long periods of time simply because they naturally occur in small areas of good habitat. For example, there are fish called desert pupfish that occur in small ponds in the desert. The ponds are too small to hold many fish, so populations are small. Each pond has a separate population from each other pond since, clearly, fish are not able to get out of the pond and walk through the desert to another pond.