If you need to remove a drywall anchor, the best way to do it is by using a screwdriver or drill. With a screwdriver, simply unscrew the anchor from the wall. If you use a drill, set it to reverse and drill out the anchor until removed.