Another possibility is that, although there is still some genetic variability, it can't be expressed until there is a recombination event. At this stage there can be no selection. However, eventually there will be a chance recombination event on the chromosome. Correlated responses to selection. Another barrier to indefinite improvement of a trait.
Smaller mice have a physical limitation of how much litter weight they can bear.
If you make the number of offspring large in a small mouse, you have to make the offspring smaller. These small offspring may have low fitness or even be inviable. Suppose you wanted to select a mouse with small body size and large litters. Because of the correlation, this would be difficult. Frequently, the result is that the selected line reverts back partway to its original starting point.