Yes, you can summon a pact weapon in your empty hand while already wielding a normal weapon in your offhand. But unless you have two-weapon fighting, you can’t add your ability modifier to the offhand attack.