They have less risk of it due to higher performance aircraft, but with a much higher landing speed and greater weight and wing span, a sail plane can do a lot of damage to itself and a farmers crop landing in the wrong place. For a hang glider pilot, it is also a legitimate fear as the pilot has to discern the wind direction and location of obstacles from an altitude of 500 to 1000 ft. For a paraglider pilot, it is almost no problem at all as long as the pilot avoids power lines and pastures with canopy eating cows and territorial bulls. Retrieval for a sailplane requires a trailer and may require access to someone's property in order to remove the glider, where as a hang glider can sometimes be loaded on the back of a friendly farmers truck. Obviously, the problems vanish when you fly an aircraft that will comfortably fit in the trunk of a VW Beetle.
Preparation is the key.
If you go XC, all that is really required for easy retrieval is a central communications point and a buddy with a set of your car keys, or solid radio communications. But many adventure-some pilots have gone XC with no retrieval preparations at all. I went XC and landed in Podunkville. Another part of preparation for XC is having the proper equipment. Most of this equipment will seldom be needed, but in case you land back in the mountains with a 2 day walkout, some of these things might come in right handy.