There are certain times when you need to decline an invitation. It is definitely not rude if you reasonably approach this situation. Remember, apologize, but be straightforward, or in other words, don’t beat around the bush, and offer an alternative if appropriate.