Together with the code above, it seems like this will remove a layer that was added with a click when hovering above it (didn't test it).

In the code, I'm just missing code that tests if a given layer was actively added to the map (and not as preview). But it could well be that I don't understand something here.