Restore accidentally removed use of my_build_ext
Don't symlink libraries unless inplace build is requested
Add support for custom build directory for tests