You should consider replacing your gutter brackets if you notice signs of gutter sagging, detachment, or any visible damage. Additionally, if your gutters are constantly overflowing or not effectively channelling rainwater, it may indicate a problem with the brackets. Regular inspections and maintenance can help you identify when replacement is necessary.