You can remove friends by tapping and holding (app) or right-clicking (browser game/Steam) their portrait in the friend list. A little menu will pop up allowing you to visit or delete your friend. Use the red button to remove your friend from your friend list.