Special missions on the map contain poor villagers. When you open these cages by completing the mission, these villagers enter your castle and become new dwellers.