Poor INP scores can significantly impact user experience by making the site feel less responsive. This can lead to reduced user satisfaction, higher bounce rates, and potentially lower SEO rankings as Google incorporates user experience metrics into its search algorithms.