If you are using the Apple Wallet app, just click on ‘Remove Pass’ from the action screen.
If you are using the Google Pay app, then tap on the Remove Card’ option appearing at the bottom of your card information.