First, you must have the informed consent of your employer or client. Part of being a faithful agent is warning your principal when your judgment is not as reliable as it would normally be. Your principal can then decide whether he prefers to avoid the conflict by replacing you or accept the conflict, taking the necessary precautions and hoping for the best. That decision is his, not yours. Second, you must be satisfied that you can do what is asked of you in a way that will not bring you or your profession in disrepute.
Appearances can be as important as reality.
The consent of your employer or client is part, but only part, of maintaining appearances. The rest is your responsibility, not your employer's or client's. You warned the VP of your conflict of interest. He still wants you to serve on the committee. You have much to consider.