Use dlerror() to clear any error before retrying with the mangled name

Thanks to Catalin Ontanu for reporting this, and then doing more
investigation to find and test the solution.
Add some additional checks related to instruction size