Linux (Radeon): Take care that post and ref divider don't grow too large

Please do a build test before. This and the other one in 996 should be replaced when Christian updates the v2 patch on the ML.

It fixes fractional Modelines for some people.