Yes, the duct size can affect the CFM. For instance, the CFM performance will decrease if the bathroom fan is rated for 120 CFM with a 6-inch duct, but you have 3-inch installed. The quality of the duct can also affect the CFM output of your fan. Should you get a new bath fan, check the instructions for the required duct size. The best ducts for bathroom fans tend to be insulated, a feature that helps prevent condensation and reduce noise.