I think a solution would be to show a message that informs the user about what exactly the remove button is doing when they click it. Then we stay on the safe side and explain them at the same time how they can also purge the track data from the library