Refactor unsubscribe to make the logic more clear

This is the refactor we discussed on pull request #22