Factors to consider when choosing a filter size for pour over include the brew method, desired extraction level, and filter material. The size should match the brewing device to ensure optimal water flow and prevent clogging.