This happens all the time, so don't take it personally. You're going to have to do a bit of work here and reach out to the guests who haven't responded. Just kindly remind them that you need to know if they're coming or not for the venue's sake so they can prepare for a specific number of guests. It's also possible they just didn't get your invitation!