Yes, you should match up your impedances. One reason is that you will get maximum transfer of power. The other reason is that you can sometimes approach unsafe and dangerous conditions. When in doubt, it is safer to go higher in impedance to help protect the amplifier.