Difficulties should never be used as an excuse to abuse your partner or withdraw from the relationship. If you need to remove yourself from your spouse when things go rough, your relationship isn't that essential to you, or you don't have what it takes to be in a relationship. Removing yourself like this shows that the problems are too big for just the two of you and indicates that you need help from outside the marriage. Pulling away from your partner is not okay. It's a sign of disrespect and can cause serious problems down the road. If you find yourself pulling away from your spouse, try to understand why this is happening and resolve it quickly. Discuss your concerns with your partner and get his/her perspective as well. The both of you should work on improving your relationship so it doesn't feel like you have to avoid each other to protect it.