If only a few of the birds come when you call them then you are feeding too much. You also don't want them so hungry they won't fly around the loft, but rush into the loft for feed then you are not feeding enough. Too hungry or too fat - birds don't fly. Don't change the amount of feed in a hurry. Gradually work any change into their routine so you do not upset them.
Tame birds are easier to handle and more comfortable in the loft.
It also makes working with them a pleasure. Scared pigeons are not fun to work with. I always have a little treat that I hand feed feed every bird when they are in their perches for the night. If you go into the loft and the birds are afraid of you, you have already lessened their performance on race day. The pigeons should trust you and be happy to see you.