Yes, you should remove them. Try to leave branches with a 45 to 50 degree angle from the trunk.