Without rear-suspension to cushion terrain, frame material and layup become an important consideration. Diverse materials and tube sizes will absorb terrain impacts differently, which can either increase or decrease your ride comfort.
Carbon can be tuned to absorb vibrations effectively whilst also being lightweight and stiff in corners or when pedaling. Carbon is expensive though and budget carbon models can often be overbuilt and have a harsh ride quality.
Steel holds a fond place amongst the hardtail community. Well-made steel frames are strong, comfortable and are far more affordable. This comes with a weight penalty though as steel is relatively heavy.
Aluminum rather unfairly got a reputation for being overly stiff and harsh in the past. However modern manufacturing methods have greatly improved the compliance of aluminum. It's also lighter than steel and is very affordable.