Turns out there is a simpler way of getting a set difference in bash than parsing diff output.