Yes, you can unequip a talisman at any time from the inventory screen. Simply select the 'Talisman' tab and choose the talisman you want to remove. Press the button prompt to unequip it.