You'll have a tougher time than you would in the temperate forest, but don't be afraid to try out whatever biome you want. Even if you fail, the beauty of RimWorld is that you learn several new things with every new playthrough. An extreme desert biome is far more difficult than a regular desert, so make sure you're aware of the challenges ahead. Check out this great Wiki article on the topic for more information.