Hm, you are reusing the same button for two actions? This could lead to unexpected behavior. Like what if user double-clicks. The now it adds and removes. But it would be better if it would just add. Some people do double click everything on the web.

Can you make two buttons here? So one is replaces with another completely, with different class and then it calls different things.