Because they can fit in more pores, they get caught in more pores, obviously. So they take longer to get through the column. It may help to think of an adult and a child walking through toy store. The adult will walk through fairly quickly, but the child is going to get distracted by all the cool toys, stopping to look and play with each one. It's going to take the kid a lot longer to walk through the toy store than it will the adult.
Polymers are the same way.
The bigger ones pass through the column uninterested in the pores in the beads, but the little ones in their curiosity have to stop to check out every little pore. So the big molecules pass through the column a lot faster than the smaller ones. What's more, we have detectors that can count how many polymer molecules are coming out of the end of the column at a given time. Because we can calculate molecular weight from elution time. So on this plot, molecular weight decreases from left to right.