Add and remove the attribute from the correct slot.

The slot that we're adding/removing the attribute from may not be the same as
the attribute coming in. Make sure that they match up before we try to
add/remove them.
PR15313


git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@175684 91177308-0d34-0410-b5e6-96231b3b80d8