Sweden then invaded Norway but agreed to let Norway keep its constitution in return for accepting the union under a Swedish king .