We have something similar in our jenkins builder.  We found it was really noisy unless it was on change.

Looks like you can specify that using "change":

on_success: [always|never|change] # default: always
on_failure: [always|never|change] # default: always