To remove a song from the library, click on that song from the playlist, then delete it from starred tracks.