Repository: GenericMappingTools/pygmt

Issue:
Find the GMT library by calling "gmt --show-library"
**Description of the desired feature**

Currently, PyGMT first checks if the environmental variable **GMT_LIBRARY_PATH** is defined. If yes, it tries to find the GMT library in the specified path. Otherwise, it will search for the GMT library in standard library paths (LD_LIBRARY_PATH on Linux; DYLD_LIBRARY_PATH on macOS; PATH on Windows, if I understand it correctly).

As mentioned in https://github.com/GenericMappingTools/gmt/issues/4241#issuecomment-696412420, `gmt --show-library` can return the full path of the GMT library. It can help find the GMT library especially when GMT is not installed via conda. Then the search order would be:

1. GMT_LIBRARY_PATH if defined
2. Full path returned by `gmt --show-library`
3. System-wide standard library paths

**Are you willing to help implement and maintain this feature?** Yes

Discussion:
leouieda: Thanks, @seisman! This is a good thing to have and I agree that it's worth the downsides. A few thoughts on the design of loading the library as a whole (might be out of scope for this PR but worth mentioning):

* I initially made the `Session` class load the library when it's used to avoid doing this work at import time. Mostly so we wouldn't have `GMTLibNotFoundError` at import.
* That clearly didn't work because we need to call `begin` at import for this whole thing to work. 
* So it might be time to rethink loading libgmt in the class instead of as a global instead to avoid searching for it every time. 
* Or we figure out a way to not call `begin` at import time.
leouieda: And yes, these are the trickiest things to test. Using [pytest's monkeypatch](https://docs.pytest.org/en/stable/monkeypatch.html) is usually a good way to do it so you can trigger errors yourself. 
seisman: @weiji14 Great work! 

I just made a few commits to this branch (dbfd7e29b4233c56262921877146fc85470a2f49, 6769dc3d5c6af3685352e6e7633901ae4dd01945 and c3887ce25e45e23cf6d50cfb7644898c3d7c581c). 

I tested the branch on macOS for many different cases:

- GMT_LIBRAYR_PATH defined
- PATH defined, GMT_LIBRARY_PATH undefined
- both PATH and GMT_LIBRARY_PATH defined
- both PATH and GMT_LIBRARY_PATH undefined, but DYLD_LIBRARY_PATH defined

They all work as I expect (e.g., no extra system calling if the library is already found via `GMT_LIBRARY_PATH`.

I think this PR is ready to merge. Do we want to merge it into v0.3.0?

weiji14: /test-gmt-master

> I think this PR is ready to merge. Do we want to merge it into v0.3.0?

Yes please! Once the tests pass. I think there's still some missing code coverage? But we can improve on things later in another iteration.


Patch:
diff --git a/doc/install.rst b/doc/install.rst
index 0b8c32f3..86bc5208 100644
--- a/doc/install.rst
+++ b/doc/install.rst
@@ -14,15 +14,44 @@ Installing
     <https://forum.generic-mapping-tools.org/c/questions/pygmt-q-a>`__.
 
 
+Quickstart
+----------
+
+The fastest way to install PyGMT is with the
+`conda <https://docs.conda.io/projects/conda/en/latest/user-guide/index.html>`__
+package manager which takes care of setting up a virtual environment, as well
+as the installation of GMT and all the dependencies PyGMT depends on::
+
+    conda create --name pygmt --channel conda-forge pygmt
+
+To activate the virtual environment, you can do::
+
+    conda activate pygmt
+
+After this, check that everything works by running the following in a Python
+interpreter (e.g. in a Jupyter notebook)::
+
+    import pygmt
+    pygmt.show_versions()
+
+You are now ready to make you first figure!
+Start by looking at the tutorials on our sidebar, good luck!
+
+.. note::
+
+    The sections below provide more detailed, step by step instructions to
+    installing and testing PyGMT for those who may have a slightly different
+    setup.
+
 Which Python?
 -------------
 
-You'll need **Python 3.7 or greater** to run PyGMT. Old Python versions may
-work, but there is no guarantee that PyGMT will work as expected with old versions.
+PyGMT is tested to run on **Python 3.7 or greater**. Older Python versions may
+work, but there is no guarantee that PyGMT will behave as expected.
 
-We recommend using the `Anaconda <https://www.anaconda.com/distribution>`__ Python
-distribution to ensure you have all dependencies installed and the ``conda``
-package manager available.
+We recommend using the `Anaconda <https://www.anaconda.com/distribution>`__
+Python distribution to ensure you have all dependencies installed and the
+``conda`` package manager available.
 Installing Anaconda does not require administrative rights to your computer and
 doesn't interfere with any other Python installations in your system.
 
@@ -30,18 +59,19 @@ doesn't interfere with any other Python installations in your system.
 Which GMT?
 ----------
 
-PyGMT requires Generic Mapping Tools (GMT) version 6 as a minimum, which is the latest
-released version that can be found at
+PyGMT requires Generic Mapping Tools (GMT) version 6 as a minimum, which is the
+latest released version that can be found at
 the `GMT official site <https://www.generic-mapping-tools.org>`__.
-We need the latest GMT (>=6.1.1) since there are many changes being made to GMT itself in
-response to the development of PyGMT, mainly the new
+We need the latest GMT (>=6.1.1) since there are many changes being made to GMT
+itself in response to the development of PyGMT, mainly the new
 `modern execution mode <https://docs.generic-mapping-tools.org/latest/cookbook/introduction.html#modern-and-classic-mode>`__.
 
-Compiled conda packages of GMT for Linux, macOS and Windows are provided through
-`conda-forge <https://anaconda.org/conda-forge/gmt>`__.
+Compiled conda packages of GMT for Linux, macOS and Windows are provided
+through `conda-forge <https://anaconda.org/conda-forge/gmt>`__.
 Advanced users can also
 `build GMT from source <https://github.com/GenericMappingTools/gmt/blob/master/BUILDING.md>`__
-instead, which is not so recommended but we would love to get feedback from anyone who tries.
+instead, which is not so recommended but we would love to get feedback from
+anyone who tries.
 
 We recommend following the instructions further on to install GMT 6.
 
@@ -58,61 +88,97 @@ PyGMT requires the following libraries:
 
 The following are optional (but recommended) dependencies:
 
-* `IPython <https://ipython.org/>`__: For embedding the figures in Jupyter notebooks.
+* `IPython <https://ipython.org/>`__: For embedding the figures in Jupyter
+notebooks.
 
 
 Installing GMT and other dependencies
 -------------------------------------
 
-Before installing PyGMT, we must install GMT itself along with the other dependencies.
-The easiest way to do this is using the ``conda`` package manager.
+Before installing PyGMT, we must install GMT itself along with the other
+dependencies. The easiest way to do this is via the ``conda`` package manager.
 We recommend working in an isolated
 `conda environment <https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html>`__
-to avoid issues with competing versions of its dependencies.
+to avoid issues with conflicting versions of dependencies.
 
 First, we must configure conda to get packages from the
 `conda-forge channel <https://conda-forge.org/>`__::
 
     conda config --prepend channels conda-forge
 
-Now we can create a new conda environment with Python and all our dependencies installed
-(we'll call it ``pygmt`` but you can change it to whatever you want)::
+Now we can create a new conda environment with Python and all our dependencies
+installed (we'll call it ``pygmt`` but feel free to change it to whatever you
+want)::
 
      conda create --name pygmt python=3.9 pip numpy pandas xarray netcdf4 packaging gmt
 
-Activate the environment by running::
+Activate the environment by running the following (**do not forget this step!**)::
 
     conda activate pygmt
 
-From now on, all commands will take place inside the conda virtual environment and won't
-affect your default installation.
+From now on, all commands will take place inside the conda virtual environment
+called 'pygmt' and won't affect your default 'base' installation.
 
 
 Installing PyGMT
 ----------------
 
-Now that you have GMT installed and your conda environment activated, you can
-use ``conda`` to install the latest release of PyGMT from `conda-forge <https://anaconda.org/conda-forge/pygmt>`__::
+Now that you have GMT installed and your conda virtual environment activated,
+you can install PyGMT using any of the following methods:
+
+Using conda (recommended)
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This installs the latest stable release of PyGMT from
+`conda-forge <https://anaconda.org/conda-forge/pygmt>`__::
 
     conda install pygmt
 
-or use ``pip`` to install from `PyPI <https://pypi.org/project/pygmt>`__::
+Using pip
+~~~~~~~~~
+
+This installs the latest stable release from
+`PyPI <https://pypi.org/project/pygmt>`__::
 
     pip install pygmt
 
-Alternatively, you can install the development version from the GitHub repository::
+Alternatively, you can install the latest development version from
+`TestPyPI <https://test.pypi.org/project/pygmt>`__::
+
+    pip install --pre --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pygmt
 
-    pip install https://github.com/GenericMappingTools/pygmt/archive/master.zip
+or from PyGMT's `GitHub repository <https://github.com/GenericMappingTools/pygmt>`__ 
+(slower as it downloads the whole archive)::
 
-This will allow you to use the ``pygmt`` library from Python.
+    pip install git+https://github.com/GenericMappingTools/pygmt.git#egg=pygmt
+
+Any of the above methods (conda/pip) should allow you to use the ``pygmt``
+library from Python.
 
 
 Testing your install
 --------------------
 
+Quick check
+~~~~~~~~~~~
+
+To ensure that PyGMT and its depedencies are installed correctly, run the
+following in your Python interpreter::
+
+    import pygmt
+    pygmt.show_versions()
+
+Or run this in the command line::
+
+    python -c "import pygmt; pygmt.show_versions()"
+
+
+Full test (optional)
+~~~~~~~~~~~~~~~~~~~~
+
 PyGMT ships with a full test suite.
-You can run our tests after you install it but you will need a few extra dependencies as
-well (be sure to have your conda env activated)::
+You can run our tests after you install it but you will need a few extra
+dependencies as well (be sure to have your conda environment activated)::
 
     conda install pytest pytest-mpl ipython
 
@@ -132,9 +198,9 @@ This can happen if you have multiple versions of GMT installed.
 
 You can tell PyGMT exactly where to look for ``libgmt`` by setting the
 ``GMT_LIBRARY_PATH`` environment variable.
-This should be set to the directory where ``libgmt.so``, ``libgmt.dylib`` or ``gmt.dll``
-can be found for Linux, macOS and Windows respectively.
-e.g. in a terminal run::
+This should be set to the directory where ``libgmt.so``, ``libgmt.dylib`` or
+``gmt.dll`` can be found for Linux, macOS and Windows respectively.
+e.g. on a command line, run::
 
     # Linux/macOS
     export GMT_LIBRARY_PATH=$HOME/anaconda3/envs/pygmt/lib
diff --git a/pygmt/clib/loading.py b/pygmt/clib/loading.py
index c98da792..ee46c9eb 100644
--- a/pygmt/clib/loading.py
+++ b/pygmt/clib/loading.py
@@ -6,6 +6,7 @@ through the GMT_LIBRARY_PATH environment variable.
 """
 import ctypes
 import os
+import subprocess as sp
 import sys
 from ctypes.util import find_library
 
@@ -31,9 +32,10 @@ def load_libgmt():
         If there was any problem loading the library (couldn't find it or
         couldn't access the functions).
     """
-    lib_fullnames = clib_full_names()
+    lib_fullnames = []
     error = True
-    for libname in lib_fullnames:
+    for libname in clib_full_names():
+        lib_fullnames.append(libname)
         try:
             libgmt = ctypes.CDLL(libname)
             check_libgmt(libgmt)
@@ -72,7 +74,7 @@ def clib_names(os_name):
     elif os_name.startswith("freebsd"):  # FreeBSD
         libnames = ["libgmt.so"]
     else:
-        raise GMTOSError(f'Operating system "{sys.platform}" not supported.')
+        raise GMTOSError(f'Operating system "{os_name}" not supported.')
     return libnames
 
 
@@ -86,24 +88,45 @@ def clib_full_names(env=None):
         A dictionary containing the environment variables. If ``None``, will
         default to ``os.environ``.
 
-    Returns
-    -------
+    Yields
+    ------
     lib_fullnames: list of str
         List of possible full names of GMT's shared library.
     """
     if env is None:
         env = os.environ
+
     libnames = clib_names(os_name=sys.platform)  # e.g. libgmt.so, libgmt.dylib, gmt.dll
-    libpath = env.get("GMT_LIBRARY_PATH", "")  # e.g. $HOME/miniconda/envs/pygmt/lib
 
-    lib_fullnames = [os.path.join(libpath, libname) for libname in libnames]
-    # Search for DLLs in PATH if GMT_LIBRARY_PATH is not defined [Windows only]
-    if not libpath and sys.platform == "win32":
+    # list of libraries paths to search, sort by priority from high to low
+    # Search for libraries in GMT_LIBRARY_PATH if defined.
+    libpath = env.get("GMT_LIBRARY_PATH", "")  # e.g. $HOME/miniconda/envs/pygmt/lib
+    if libpath:
+        for libname in libnames:
+            libfullpath = os.path.join(libpath, libname)
+            if os.path.exists(libfullpath):
+                yield libfullpath
+
+    # Search for the library returned by command "gmt --show-library"
+    try:
+        libfullpath = sp.check_output(
+            ["gmt", "--show-library"], encoding="utf-8"
+        ).rstrip("\n")
+        assert os.path.exists(libfullpath)
+        yield libfullpath
+    except (FileNotFoundError, AssertionError):  # command not found
+        pass
+
+    # Search for DLLs in PATH (done by calling "find_library")
+    if sys.platform == "win32":
         for libname in libnames:
             libfullpath = find_library(libname)
             if libfullpath:
-                lib_fullnames.append(libfullpath)
-    return lib_fullnames
+                yield libfullpath
+
+    # Search for library names in the system default path [the lowest priority]
+    for libname in libnames:
+        yield libname
 
 
 def check_libgmt(libgmt):
@@ -128,10 +151,5 @@ def check_libgmt(libgmt):
     functions = ["Create_Session", "Get_Enum", "Call_Module", "Destroy_Session"]
     for func in functions:
         if not hasattr(libgmt, "GMT_" + func):
-            msg = " ".join(
-                [
-                    "Error loading libgmt.",
-                    "Couldn't access function GMT_{}.".format(func),
-                ]
-            )
+            msg = f"Error loading libgmt. Couldn't access function GMT_{func}."
             raise GMTCLibError(msg)
diff --git a/setup.py b/setup.py
index cfac8b7f..45685e36 100644
--- a/setup.py
+++ b/setup.py
@@ -36,7 +36,7 @@ PLATFORMS = "Any"
 INSTALL_REQUIRES = ["numpy", "pandas", "xarray", "netCDF4", "packaging"]
 # Configuration for setuptools-scm
 SETUP_REQUIRES = ["setuptools_scm"]
-USE_SCM_VERSION = {"local_scheme": "node-and-date"}
+USE_SCM_VERSION = {"local_scheme": "node-and-date", "fallback_version": "unknown"}
 
 if __name__ == "__main__":
     setup(


Test patch:
diff --git a/.github/workflows/ci_tests.yaml b/.github/workflows/ci_tests.yaml
index 15b290f7..24c143bf 100644
--- a/.github/workflows/ci_tests.yaml
+++ b/.github/workflows/ci_tests.yaml
@@ -89,7 +89,9 @@ jobs:
         run: |
           mkdir -p ~/.gmt
           mv .gmt/* ~/.gmt
-          ls -lh ~/.gmt
+          # Change modification times of the two files, so GMT won't refresh it
+          touch ~/.gmt/server/gmt_data_server.txt ~/.gmt/server/gmt_hash_server.txt
+          ls -lhR ~/.gmt
 
       # Install the package that we want to test
       - name: Install the package
diff --git a/.github/workflows/ci_tests_dev.yaml b/.github/workflows/ci_tests_dev.yaml
index 93dfe27f..1010850c 100644
--- a/.github/workflows/ci_tests_dev.yaml
+++ b/.github/workflows/ci_tests_dev.yaml
@@ -108,7 +108,9 @@ jobs:
         run: |
           mkdir -p ~/.gmt
           mv .gmt/* ~/.gmt
-          ls -lh ~/.gmt
+          # Change modification times of the two files, so GMT won't refresh it
+          touch ~/.gmt/server/gmt_data_server.txt ~/.gmt/server/gmt_hash_server.txt
+          ls -lhR ~/.gmt
 
       # Install the package that we want to test
       - name: Install the package
diff --git a/pygmt/tests/test_clib_loading.py b/pygmt/tests/test_clib_loading.py
index 54fb89b3..0d2edc9b 100644
--- a/pygmt/tests/test_clib_loading.py
+++ b/pygmt/tests/test_clib_loading.py
@@ -1,7 +1,8 @@
 """
 Test the functions that load libgmt.
 """
-import os
+import subprocess
+import sys
 
 import pytest
 from pygmt.clib.loading import check_libgmt, clib_names, load_libgmt
@@ -23,22 +24,28 @@ def test_load_libgmt():
     check_libgmt(load_libgmt())
 
 
-def test_load_libgmt_fail():
+@pytest.mark.skipif(sys.platform == "win32", reason="run on UNIX platforms only")
+def test_load_libgmt_fails(monkeypatch):
     """
-    Test that loading fails when given a bad library path.
+    Test that GMTCLibNotFoundError is raised when GMT's shared library cannot
+    be found.
     """
-    # save the old value (if any) before setting a fake "GMT_LIBRARY_PATH"
-    old_gmt_library_path = os.environ.get("GMT_LIBRARY_PATH")
+    with monkeypatch.context() as mpatch:
+        mpatch.setattr(sys, "platform", "win32")  # pretend to be on Windows
+        mpatch.setattr(
+            subprocess, "check_output", lambda cmd, encoding: "libfakegmt.so"
+        )
+        with pytest.raises(GMTCLibNotFoundError):
+            check_libgmt(load_libgmt())
 
-    os.environ["GMT_LIBRARY_PATH"] = "/not/a/real/path"
-    with pytest.raises(GMTCLibNotFoundError):
-        load_libgmt()
 
-    # revert back to the original status (if any)
-    if old_gmt_library_path:
-        os.environ["GMT_LIBRARY_PATH"] = old_gmt_library_path
-    else:
-        del os.environ["GMT_LIBRARY_PATH"]
+def test_load_libgmt_with_a_bad_library_path(monkeypatch):
+    """
+    Test that loading still works when given a bad library path.
+    """
+    # Set a fake "GMT_LIBRARY_PATH"
+    monkeypatch.setenv("GMT_LIBRARY_PATH", "/not/a/real/path")
+    assert check_libgmt(load_libgmt()) is None
 
 
 def test_clib_names():
@@ -49,5 +56,7 @@ def test_clib_names():
         assert clib_names(linux) == ["libgmt.so"]
     assert clib_names("darwin") == ["libgmt.dylib"]
     assert clib_names("win32") == ["gmt.dll", "gmt_w64.dll", "gmt_w32.dll"]
+    for freebsd in ["freebsd10", "freebsd11", "freebsd12"]:
+        assert clib_names(freebsd) == ["libgmt.so"]
     with pytest.raises(GMTOSError):
         clib_names("meh")