It’s not necessary to explain why you’re there. If you’re called on and prefer to remain silent, just say, “I’ll pass.” Anyone is free to simply sit and listen at meetings.