[1.7.x] Fixed #23959 -- Clarified when checks automatically run.

Backport of cf2390be164bcb3c5670f73b0a7062d6141f8664 from master