[Wallet] Do not flush the wallet in AddToWalletIfInvolvingMe(..)