There are certain quests in Deadfire that are designed with particular party members in mind. Thankfully, these quests do warn the player when they start the quest, but for players who do not care much for small interactions about every character, not much will be missed if those specific party members aren't brought along.