You use Charisma for Pact of the Blade only if you take the Hexblade subclass, which offers the Hex Warrior ability. Hex Warrior lets you touch a weapon you are proficient in to use your Charisma modifier with it.