opened
saedrna
username_0
Hi,

I am trying to port [openblas](http://www.openblas.net/), the strategy is to directly download the binary distribution and copy the include , lib and dlls to the desired place.

Here is my temporary code,

```

include(vcpkg_common_functions)
if(VCPKG_LIBRARY_LINKAGE STREQUAL "static")
    message(WARNING "openblas only support dynamic link for msvc")
    set(VCPKG_LIBRARY_LINKAGE "dynamic")
endif()
if(TARGET_TRIPLET STREQUAL "x86-windows")
    message(WARNING "current only port openblas for x64-windows")
    set(TARGET_TRIPLET "x64-windows")
endif()

set(SOURCE_PATH ${CURRENT_BUILDTREES_DIR}/src)
set(SOURCE_OPENBLAS ${SOURCE_PATH}/OpenBLAS-v0.2.19-Win64-int32)
set(SOURCE_MINGW64 ${SOURCE_PATH}/mingw64_dll)

vcpkg_download_distfile(ARCHIVE
    URLS "https://nchc.dl.sourceforge.net/project/openblas/v0.2.19/OpenBLAS-v0.2.19-Win64-int32.zip"
    FILENAME "OpenBLAS-v0.2.19-Win64-int32.zip"
    SHA512 2bb13bf5e322d35e0bef7ef1ca986653117ef74e1071108520273ee9703e6b9cbec3cd615bfac6b52965fb7d661b76628a5526a8e658628b172fb7585cde2219
)
vcpkg_download_distfile(MINGW64_DLL
    URLS "https://nchc.dl.sourceforge.net/project/openblas/v0.2.14/mingw64_dll.zip"
    FILENAME "mingw64_dll.zip"
    SHA512 8fabb10c6556686d1b04816f6ba62702f607d90cdae9c3c20e598d0330acfcbb361a3fd811402f34f896c88a232e29541ec2e98cfa9ae698b2f7c6216e2b8420
)
vcpkg_extract_source_archive(${ARCHIVE})
vcpkg_extract_source_archive(${MINGW64_DLL})

# apply patches
vcpkg_apply_patches(
    SOURCE_PATH ${SOURCE_OPENBLAS}
    PATCHES "${CMAKE_CURRENT_LIST_DIR}/config-file-use-relative-path.patch"
)

# copy include headers
file(COPY ${SOURCE_OPENBLAS}/include 
    DESTINATION ${CURRENT_PACKAGES_DIR})

# copy lib
file(COPY ${SOURCE_OPENBLAS}/lib/libopenblas.dll.a 
    DESTINATION ${CURRENT_PACKAGES_DIR}/lib)
file(COPY ${SOURCE_OPENBLAS}/lib/libopenblas.dll.a 
    DESTINATION ${CURRENT_PACKAGES_DIR}/debug/lib)
file(RENAME ${CURRENT_PACKAGES_DIR}/lib/libopenblas.dll.a ${CURRENT_PACKAGES_DIR}/lib/libopenblas.lib)
file(RENAME ${CURRENT_PACKAGES_DIR}/debug/lib/libopenblas.dll.a ${CURRENT_PACKAGES_DIR}/debug/lib/libopenblas.lib)

#copy dlls
file(COPY ${SOURCE_OPENBLAS}/bin 
    DESTINATION ${CURRENT_PACKAGES_DIR})
file(COPY ${SOURCE_OPENBLAS}/bin 
    DESTINATION ${CURRENT_PACKAGES_DIR}/debug)
file(GLOB MINGW64_DLL_FILES ${SOURCE_MINGW64}/*.dll)

file(COPY ${MINGW64_DLL_FILES}
    DESTINATION ${CURRENT_PACKAGES_DIR}/bin)
file(COPY ${MINGW64_DLL_FILES}
    DESTINATION ${CURRENT_PACKAGES_DIR}/debug/bin)

# copy cmake
file(COPY ${SOURCE_OPENBLAS}/lib/cmake
    DESTINATION ${CURRENT_PACKAGES_DIR}/share/openblas)

# copy license
file(COPY ${CMAKE_CURRENT_LIST_DIR}/LICENSE
    DESTINATION ${CURRENT_PACKAGES_DIR}/share/openblas)
file(RENAME ${CURRENT_PACKAGES_DIR}/share/openblas/LICENSE ${CURRENT_PACKAGES_DIR}/share/openblas/copyright)
```

And I have successfully get the packages in the `./vcpkg/packages/openblas_x64-windows` as below,

![snipaste_20170408_121153](https://cloud.githubusercontent.com/assets/3850191/24825624/9a14a174-1c54-11e7-821d-8a8892e54a4e.png)

But at last, I have encountered the error, and I have no idea what is this.
```
-- Performing post-build validation
Could not find proper second linker member
```

So, is this strategy OK to only copy the files but no compilation? Because openblas is C, and I think it is ABI compatible if only dynamic link is used.

If it's OK, how could I solve the `Could not find proper second linker member` problem?

Thanks,

Han
How to solve "Could not find proper second linker member"?
issue
created
ras0219-msft
username_1
Hi @username_0, thanks for starting this!

To answer the primary question up front: "Could not find proper second linker member" is an error from our internal COFF reader implementation. As part of package verification, we check several attributes of the binaries such as architecture, APPCONTAINER, and CRTs. It seems that the gcc-built import libs have some differences from the MSVC linker's import libs -- based on the error, they probably don't output the second linker member table which is used for improved link speed.

I think the best option in this case is to generate fresh import libs using the MSVC linker: see [2] for an example of this. It's probably easiest to write a short powershell script to do that.

A few comments on the overall approach (precompiled binaries):

This works as long as the library is C, doesn't allocate/free across the DLL boundary, doesn't expect shared CRT state, and has no dependencies. In this case, there is a concern that other packages might want to also use libgfortran, so that may need to be split into a separate package (with some way to verify ABI compatibility?).

Another concern with these particular libraries is that although OpenBLAS is licensed as BSD, it appears that libgfortran is only available under the GPL (note: _not_ LGPL). If we go this route, it is very important to deploy the copyright notices for all of the packaged runtime libraries as well.

Small nit: the `mingw64_dll.zip` download should probably be `openblas-0.2.14-mingw64_dll.zip`.

Finally, it looks like in 0.2.15+ the OpenBLAS project has started to support VS/MSVC directly through CMake [1]. In the long term, we would much prefer using this option and contributing back to the upstream project to make it "first-class". However, there may be reasons in the short term to prefer precompiled DLLs.

Thanks again for starting this port and let us know if you hit any other obstacles!

[1] https://github.com/xianyi/OpenBLAS/wiki/How-to-use-OpenBLAS-in-Microsoft-Visual-Studio#2-cmake-and-visual-studio
[2] https://adrianhenke.wordpress.com/2008/12/05/create-lib-file-from-dll/
comment
created
ras0219-msft
username_1
One other thing I forgot: you should use
```cmake
if(NOT VCPKG_TARGET_ARCHITECTURE STREQUAL "x64")
    message(FATAL_ERROR "openblas can only be built for x64 currently")
endif()
```
to check the architecture. This will enable it to work when building for x64-windows-static and will fail faster when someone requests x86 (with the current approach, we will just fail later, in the post-build checks which ensures you have packaged the right architecture of libraries). Unlike with static vs dynamic, it doesn't make sense to give the user x64 libraries when they requested x86 -- they can't use them!
comment
created
saedrna
username_0
I have modified the lib part, use `lib /def:libopenblas.lib /OUT:libopenblas.lib` to create the lib file.
However, it seems that the openblas binary distribution is also dependent on `msvcrt.dll` and generate another error. I know that vs2015 uses vcruntime140.dll and that's what the error means,

```
Detected outdated dynamic CRT in the following files:

    C:/workspace/build/vcpkg/packages/openblas_x64-windows/debug/bin/libopenblas.dll: msvcrt.dll
    C:/workspace/build/vcpkg/packages/openblas_x64-windows/bin/libopenblas.dll: msvcrt.dll
```

And here is the depends of libopenblas.dll, I didn't assume it also depends on vc, that's werid,
```
    libgcc_s_seh-1.dll
    libgfortran-3.dll
    KERNEL32.dll
    msvcrt.dll
    USER32.dll
```

below is how I create libopenblas.lib, and the def file is created manually and puts into the ports directory,

```
find_file(LIB_EXE lib.exe 
    PATHS "C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\bin\\amd64\\lib.exe"
          "C:\\Program Files (x86)\\Microsoft Visual Studio 15.0\\VC\\bin\\amd64\\lib.exe")
message(STATUS "${LIB_EXE}")
file(MAKE_DIRECTORY ${CURRENT_PACKAGES_DIR}/lib)
file(MAKE_DIRECTORY ${CURRENT_PACKAGES_DIR}/debug/lib)
execute_process(COMMAND "${LIB_EXE}" /machine:X64 /def:${CMAKE_CURRENT_LIST_DIR}/libopenblas.def /OUT:${CURRENT_PACKAGES_DIR}/lib/libopenblas.lib
                WORKING_DIRECTORY ${CURRENT_PACKAGES_DIR}/lib)
file(COPY ${CURRENT_PACKAGES_DIR}/lib/libopenblas.lib DESTINATION ${CURRENT_PACKAGES_DIR}/debug/lib)
file(REMOVE ${CURRENT_PACKAGES_DIR}/lib/libopenblas.exp)
```

One more comment on using cmake to build libopenblas, first it only support x86 for the time being and second it only uses plain C implementation and may be slow. If we really need to build from source of libopenblas, I think it's better to hook into something like msys or other similar subsystem.
comment
created
saedrna
username_0
Furthermore, even of I cross compile libopenblas in linux with cross compile mingw-w64, I have found that it will depend on msvcrt.dll. Below is the command to cross compile libopenblas
```
make CC=x86_64-w64-mingw32-gcc FC=x86_64-w64-mingw32-gfortran TARGET=NEHALM HOSTCC=gcc
```
comment
created
ras0219-msft
username_1
PATHS "C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\bin\\amd64\\lib.exe"
          "C:\\Program Files (x86)\\Microsoft Visual Studio 15.0\\VC\\bin\\amd64\\lib.exe")`

We ensure that the toolchain is available on the PATH inside the portfile, so you don't need to (and shouldn't!) specify any search PATHS. This helps to ensure consistent behavior across everyone's environments. Also, `find_program()` may be more appropriate here than `find_file()`.

[1] https://github.com/Microsoft/vcpkg/blob/2f8d8d8b18f8a6dd7dad6e3290074dc6044395ce/scripts/cmake/vcpkg_acquire_msys.cmake
[2] https://github.com/Microsoft/vcpkg/blob/master/toolsrc/src/PostBuildLint.cpp#L683
comment
created
saedrna
username_0
Thanks, after some searches, I think I have come to the same dead end of #23 .

Although openblas comes with a cmake system, only blas part is available without touching the fortran code. For lapack part, I didn't find a library without fortran and mingw uses msvcrt as the C runtime. 

Is it possible to port mkl? If the license is fine for vcpkg? It now has a comminity license, and I have seen others redistribute mkl freely, such as Anaconda.
comment
created
ras0219-msft
username_1
Generally, vcpkg doesn't have any problem with licenses; we do not distribute sources or binaries of the packaged libraries. So even proprietary community/evaluation licenses are conceptually fine for us.

MKL would be great to have! I see two obstacles:
1. Unfortunately our "replacement" support isn't as excellent as we'd like yet, so the experience of selecting between openBLAS and MKL will unfortunately involve manually manipulating the dependency list of everyone who needs BLAS. We're working on this!
2. I don't see a simple download on the website[1]. If the library owner insists on signing up/in to download a library, we can't package it _easily_. We might be able to fail the build and ask the user to download the archive for us, but that's a pretty terrible experience.

[1] https://software.intel.com/en-us/intel-mkl/
comment
closed
dan-shaw
username_2
issue