In fact, Yamato openly declared that he would join Luffy on several occasions, however, in the end, he changed his mind. Yamato decided to stay back in Wano simply because Oden started his journey by going from region to region in Wano as well.