It's not that you're not valuable - you are. But it is true that they'd get along without you if it was impossible for you to be there. Try thinking of the time you take for yourself as time when it's impossible for you to be working. Keep in mind that people who know you know how to manipulate you with guilt, even if they do so unintentionally. That is, they know that they can get what they want if they look at you with disappointment or remind you of how you're inconveniencing them.
Consistency is key.
In time, they'll change the way they approach you. Remind yourself of the gender unfairness inherent in our society and in the legal profession. The people who initially defined being a good lawyer as always putting work first usually had wives at home to take care of their families. Don't tell your colleague or client that you can't give them what they want right now because you need personal time. You don't need to explain.