Silvery Barbs doesn't care about your advantage or disadvantage. If the spell succeeds, the second roll is made as is, and the lower number is used irrespective of any effect of advantage or disadvantage on the original roll. That is, advantage is only applied to the original roll, not the one that was forced due to the effect of Silvery Barbs.