You can use your palm sander on drywall, but an orbital sander might be a better choice as it works faster. If you don't want to apply too much pressure and you'll be better off using a round pad with drywall surfaces.
The best palm sanders will not damage anything, but you never know with other power tools. More than anything, you'll need to have the right sanding paper.