Being direct with other people is a form of assertiveness. I wouldn't worry too much about this. If you're telling people what you need and you're clear about what your limits are, you're speaking up just fine. It's possible that you just need to reframe what you're picturing when you think about speaking up.