Look ahead of you at the amount of food on the table. Look behind you at the line of people still to come. Be sensible and polite and guage your portions accordingly. Basic common sense stuff like that. I have had great success with potlucks in the past.
Success is good planning, well executed.
If you're organizing the potluck, be sure to twist the arms of your friends to bring something good, and lots of it. I often will personally bring a large crock pot of bukkenade or other stew like substance or else two or three tartes of flesche. Offer to provide easy recipes that are transportable to anyoen who asks, and a few who dont. Remind folks that they're supposed to bring a dish to serve 8-10 people. I tell folks to bring as much food per person, as they expect to eat, volume wise.