You should sand and wash away any left-over glue or residue. Be careful, though, that the glue isn't part of the wall rather than reside from the wallpaper.