Don't search in architecture-irrelevant directories

If a library is present in both lib{32,64}, the wrong one may be picked since both are searched (depending on your actual architecture and luck)