After hours of debugging it seems that this wasn't working because of the -I or -L in the Makefile. I don't know which but removing all the references to the toolchain's includes and libs made things work much better.