Some potential pitfalls when setting up a Voluntary Trust include improper trust formation, mismanagement of trust assets, legal disputes, and tax implications. To mitigate these risks, it's essential to work with an experienced estate planning lawyer who can ensure the trust is correctly formed, legally valid, and well-managed.