Yes, you can be denied visitation. There is no Constitutional or otherwise inherent right to visit a prisoner, even a spouse. If they deem that the visit would be disruptive or pose a security risk, or if your husband does anything that loses him the privilege of a spousal visit, they can refuse to allow you to visit him.