There are a lot of other factors that contribute to your credit score, but paying off loans should not materially lower your FICO score long term. However, there can be a temporary dip in your FICO score.