Fix the bug 3440.

There are two problem in the Matlab module's CMakeLists.txt.

1. According the implementation of macro PREPEND, I think it might be designed to take various IN arguments. But in fact it's not.
2. The value of MEX_LIB_DIR was not taken into account of UNIX systems.