To make your quest easier, explore the entire desert area and find all of the Cactuar Statues before going to the Ruffian village.
It'll make your mission less stressful as you progress through the quest later on.