Most of the IEP meetings I have attended have included more than 5 people. If you would like to keep the meeting small, you can request input be given in writing. I would strongly suggest that you prepare for the meeting by writing an agenda of issues and concerns you would like addressed. It is also a good idea to bring someone who will support you through the meeting, and take notes for you, or remind you to ask those questions you need answered. IEP meetings can be difficult for parents because of the emotional drain of dealing with our own child's deficits.
Emotional support can help keep the process a positive one.