Many events that take place in Sumeru will require you to be a certain Adventure Rank, and some of the enemies can be tough. So we wouldn't recommend getting to Sumeru before reaching at least Adventure Rank 20.