You should be concerned if your FICO score falls below 580, as this is considered a poor score and may significantly impact your ability to secure credit at favorable terms. Consistently declining FICO scores should also be a cause for concern, as they may indicate underlying financial issues.