The procedure is the same as for troops. You put Spells into the upgrade queue. Elixir Spells are upgraded with Elixir, while Dark Spells are upgraded with Dark Elixir.