Repository: opengeos/whitebox-python

Issue:
Unable to instance WhiteBoxTools in 2.3.1

Starting from yesterday I am unable to create instances of WhiteBoxTools in Python 3. I get this error:

FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.8/dist-packages/whitebox/WBT/img'

It looks like something related to this package:

WhiteboxTools_linux_amd64.zip

```
root@bdfd55668517:/src# python3
Python 3.8.10 (default, Mar 13 2023, 10:26:41) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
```
```
>>> from whitebox import WhiteboxTools
>>> wbt = WhiteboxTools()
```
```
Downloading WhiteboxTools pre-compiled binary for first time use ...
Decompressing WhiteboxTools_linux_amd64.zip ...
WhiteboxTools package directory: /usr/local/lib/python3.8/dist-packages/whitebox
Unexpected error: <class 'FileNotFoundError'>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/whitebox/whitebox_tools.py", line 278, in __init__
    download_wbt()
  File "/usr/local/lib/python3.8/dist-packages/whitebox/whitebox_tools.py", line 157, in download_wbt
    shutil.copytree(init_img_dir, new_img_dir)
  File "/usr/lib/python3.8/shutil.py", line 555, in copytree
    with os.scandir(src) as itr:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.8/dist-packages/whitebox/WBT/img'
```

Thanks!

Discussion:
@markwang0: I have the same error with a python environment I created today. It is v2.3.1 and on macOS

```python
>>> from whitebox.whitebox_tools import WhiteboxTools
>>> wbt = WhiteboxTools()
Downloading WhiteboxTools pre-compiled binary for first time use ...
Decompressing WhiteboxTools_darwin_amd64.zip ...
WhiteboxTools package directory: /Users/markwang/micromamba/envs/pygeoflood-hlm-env/lib/python3.11/site-packages/whitebox
Unexpected error: <class 'FileNotFoundError'>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/markwang/micromamba/envs/pygeoflood-hlm-env/lib/python3.11/site-packages/whitebox/whitebox_tools.py", line 278, in __init__
    download_wbt()
  File "/Users/markwang/micromamba/envs/pygeoflood-hlm-env/lib/python3.11/site-packages/whitebox/whitebox_tools.py", line 157, in download_wbt
    shutil.copytree(init_img_dir, new_img_dir)
  File "/Users/markwang/micromamba/envs/pygeoflood-hlm-env/lib/python3.11/shutil.py", line 571, in copytree
    with os.scandir(src) as itr:
         ^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/Users/markwang/micromamba/envs/pygeoflood-hlm-env/lib/python3.11/site-packages/whitebox/WBT/img'
```

Also, the error doesn't happen when I use a conda environment created 2 weeks ago (also v2.3.1). Thank you!@ekimd: Same here, starting yesterday.@giswqs: Thank you for reporting. I will try to update it tonight @jblindsay: That' my bad, @giswqs I should have told you that we were planning a release of the backend. There are just so many moving parts in releasing a new version of WbT, which is one of the reasons why it's been so darn long.@giswqs: No problem, @jblindsay! Already great to see new WBT releases! Thank you for your great work. 

I looked into the issue and I think the problem is because the file strcuture of WBT binaries have changed. The `WBT` directory used to be under the root directory, but now `WBT` is a subdirectory udner something like `WhiteboxTools_linux_amd64`, depending on the operating systems. I will update the automated routine to use the new file structure. 

v2.3.0
![image](https://github.com/opengeos/whitebox-python/assets/5016453/328fd572-8886-4c76-8856-46420ff0d9cc)

v2.4.0
![image](https://github.com/opengeos/whitebox-python/assets/5016453/5928014e-76c3-4f53-bc06-e0d777ec09d0)

@jblindsay: Ah yes, this is certainly a change that I made recently and didn't think about the consequences for the frontends. This is the result of significant changes to the build script and the fact that we are now compiling to our targets using Github Actions.@giswqs: I just released v2.3.2, which should resolve the issue. If everything works well, I can update the version to v2.4.0, the same as the WBT backend version. @ekimd: I got the same error with v2.3.2
```
/root/python3.11/lib/python3.11/site-packages/whitebox/download_wbt.py:18: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  import pkg_resources
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/root/python3.11/lib/python3.11/site-packages/whitebox/download_wbt.py", line 222, in <module>
    download_wbt()
  File "/root/python3.11/lib/python3.11/site-packages/whitebox/download_wbt.py", line 145, in download_wbt
    shutil.copytree(init_img_dir, new_img_dir)
  File "/usr/lib/python3.11/shutil.py", line 558, in copytree
    with os.scandir(src) as itr:
         ^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/root/python3.11/lib/python3.11/site-packages/whitebox/WBT/img'
```@giswqs: Interesting! It passed the GitHub Actions. Will look into it again @jblindsay: We're hoping that this will greatly simplify the process of releasing new versions.@giswqs: Try v2.3.3. It should be fixed now. #65 @ekimd: It works--thank you so much!  Your fast response time is amazing!@giswqs: Great to hear that it is working now. Closing this issue for now. 

Solution patch:
diff --git a/.github/workflows/py-check.yaml b/.github/workflows/py-check.yaml
index aa054c6..ec9aa23 100644
--- a/.github/workflows/py-check.yaml
+++ b/.github/workflows/py-check.yaml
@@ -18,25 +18,26 @@ jobs:
             fail-fast: false
             matrix:
                 os: [ubuntu-latest]
-                python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
+                python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
                 include:
                     - os: macos-latest
-                      python-version: "3.10"
+                      python-version: "3.11"
                     - os: windows-latest
-                      python-version: "3.10"
+                      python-version: "3.11"
 
         env:
             SDKROOT: /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
         steps:
             - name: CHECKOUT CODE
-              uses: actions/checkout@v3
+              uses: actions/checkout@v4
             - name: SETUP PYTHON
-              uses: actions/setup-python@v4
+              uses: actions/setup-python@v5
               with:
                   python-version: ${{ matrix.python-version }}
             - name: Install dependencies
               run: |
                   python -m pip install --upgrade pip
+                  pip install -U setuptools
                   pip install --user -r requirements_dev.txt
             - name: WBT-TEST
               run: |
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
new file mode 100644
index 0000000..842361b
--- /dev/null
+++ b/.pre-commit-config.yaml
@@ -0,0 +1,23 @@
+repos:
+    - repo: https://github.com/pre-commit/pre-commit-hooks
+      rev: v4.6.0
+      hooks:
+          # - id: check-toml
+          # - id: check-yaml
+          # - id: end-of-file-fixer
+          #   types: [python]
+          # - id: trailing-whitespace
+          # - id: requirements-txt-fixer
+          - id: check-added-large-files
+            args: ["--maxkb=500"]
+
+    # - repo: https://github.com/psf/black
+    #   rev: 24.4.2
+    #   hooks:
+    #       - id: black-jupyter
+    #         language_version: python3.11
+
+    # - repo: https://github.com/kynan/nbstripout
+    #   rev: 0.7.1
+    #   hooks:
+    #       - id: nbstripout
diff --git a/whitebox/automation.py b/whitebox/automation.py
index 2b0b6bc..314f756 100644
--- a/whitebox/automation.py
+++ b/whitebox/automation.py
@@ -45,12 +45,19 @@ print("Decompressing {} ...".format(linux_zip))
 with zipfile.ZipFile(zip_path, "r") as tar_ref:
     tar_ref.extractall(work_dir)
 
+zip_dir = os.path.join(work_dir, linux_zip.split(".")[0])
+src_dir = os.path.join(zip_dir, "WBT")
+shutil.move(src_dir, WBT_dir)
+
 if os.path.exists(new_img_dir):
     shutil.rmtree(new_img_dir)
 
 if os.path.exists(new_plugin_dir):
     shutil.rmtree(new_plugin_dir)
 
+if os.path.exists(zip_dir):
+    shutil.rmtree(zip_dir)
+
 shutil.copytree(init_img_dir, new_img_dir)
 shutil.copytree(init_plugin_dir, new_plugin_dir)
 
@@ -111,8 +118,8 @@ shutil.move(
     os.path.join(WBT_dir, "whitebox_tools"), os.path.join(work_dir, "whitebox_tools")
 )
 
-shutil.move(
-    os.path.join(WBT_dir, "whitebox_runner"), os.path.join(work_dir, "whitebox_runner")
-)
+# shutil.move(
+#     os.path.join(WBT_dir, "whitebox_runner"), os.path.join(work_dir, "whitebox_runner")
+# )
 
 f.close()
diff --git a/whitebox/download_wbt.py b/whitebox/download_wbt.py
index b0deb05..4922c93 100644
--- a/whitebox/download_wbt.py
+++ b/whitebox/download_wbt.py
@@ -5,7 +5,7 @@ def download_wbt(linux_musl=False, reset=False, verbose=True):
         linux_musl (bool, optional): Whether to download the musl version of WhiteboxTools for Linux. Defaults to False.
         reset (bool, optional): Whether to reset the WhiteboxTools installation. Defaults to False.
         verbose (bool, optional): Whether to print verbose messages. Defaults to True.
-   
+
     """
     import glob
     import os
@@ -38,10 +38,14 @@ def download_wbt(linux_musl=False, reset=False, verbose=True):
         "Darwin": "https://www.whiteboxgeo.com/WBT_Darwin/WhiteboxTools_darwin_amd64.zip",
         "Darwin-arm": "https://www.whiteboxgeo.com/WBT_Darwin/WhiteboxTools_darwin_m_series.zip",
         "Linux": "https://www.whiteboxgeo.com/WBT_Linux/WhiteboxTools_linux_amd64.zip",
-        "Linux-musl": "https://www.whiteboxgeo.com/WBT_Linux/WhiteboxTools_linux_musl.zip"
+        "Linux-musl": "https://www.whiteboxgeo.com/WBT_Linux/WhiteboxTools_linux_musl.zip",
     }
 
-    if linux_musl or ('google.colab' in sys.modules) or (os.environ.get('WBT_LINUX', False) == "MUSL"):
+    if (
+        linux_musl
+        or ("google.colab" in sys.modules)
+        or (os.environ.get("WBT_LINUX", False) == "MUSL")
+    ):
         links["Linux"] = links["Linux-musl"]
 
     if platform.system() == "Darwin" and platform.processor() == "arm":
@@ -124,11 +128,20 @@ def download_wbt(linux_musl=False, reset=False, verbose=True):
             if verbose:
                 print("WhiteboxTools package directory: {}".format(pkg_dir))
 
+            zip_dir = zip_name.split(".")[0]
+            src_dir = os.path.join(zip_dir, "WBT")
+
+            if os.path.exists(src_dir):
+                shutil.move(src_dir, pkg_dir)
+
             if os.path.exists(new_img_dir):
                 shutil.rmtree(new_img_dir)
             if os.path.exists(new_plugin_dir):
                 shutil.rmtree(new_plugin_dir)
 
+            if os.path.exists(zip_dir):
+                shutil.rmtree(zip_dir)
+
             shutil.copytree(init_img_dir, new_img_dir)
             shutil.copytree(init_plugin_dir, new_plugin_dir)
 
@@ -137,13 +150,13 @@ def download_wbt(linux_musl=False, reset=False, verbose=True):
                 exe_ext = ".exe"
             exe_name = "whitebox_tools{}".format(exe_ext)
             exe_path = os.path.join(exe_dir, exe_name)
-            runner_name = "whitebox_runner{}".format(exe_ext)
-            runner_path = os.path.join(exe_dir, runner_name)
+            # runner_name = "whitebox_runner{}".format(exe_ext)
+            # runner_path = os.path.join(exe_dir, runner_name)
 
             # grant executable permission
             if platform.system() != "Windows":
                 os.system("chmod 755 " + exe_path)
-                os.system("chmod 755 " + runner_path)
+                # os.system("chmod 755 " + runner_path)
             plugins = list(
                 set(glob.glob(os.path.join(new_plugin_dir, "*")))
                 - set(glob.glob(os.path.join(new_plugin_dir, "*.json")))
@@ -154,8 +167,8 @@ def download_wbt(linux_musl=False, reset=False, verbose=True):
 
             exe_path_new = os.path.join(pkg_dir, exe_name)
             shutil.copy(exe_path, exe_path_new)
-            runner_path_new = os.path.join(pkg_dir, runner_name)
-            shutil.copy(runner_path, runner_path_new)
+            # runner_path_new = os.path.join(pkg_dir, runner_name)
+            # shutil.copy(runner_path, runner_path_new)
 
             try:
                 os.remove(zip_name)
@@ -167,7 +180,9 @@ def download_wbt(linux_musl=False, reset=False, verbose=True):
             # # downloads the binary that is compatible with Google Colab.
             if "google.colab" in sys.modules:
                 # url = "https://github.com/giswqs/whitebox-bin/raw/master/WhiteboxTools_ubuntu_18.04.zip"
-                url = "https://www.whiteboxgeo.com/WBT_Linux/WhiteboxTools_linux_musl.zip"
+                url = (
+                    "https://www.whiteboxgeo.com/WBT_Linux/WhiteboxTools_linux_musl.zip"
+                )
 
                 zip_name = os.path.join(pkg_dir, os.path.basename(url))
                 try:
diff --git a/whitebox/whitebox_tools.py b/whitebox/whitebox_tools.py
index 7195b47..f12dbbf 100644
--- a/whitebox/whitebox_tools.py
+++ b/whitebox/whitebox_tools.py
@@ -34,7 +34,7 @@ def download_wbt(linux_musl=False, reset=False, verbose=True):
         linux_musl (bool, optional): Whether to download the musl version of WhiteboxTools for Linux. Defaults to False.
         reset (bool, optional): Whether to reset the WhiteboxTools installation. Defaults to False.
         verbose (bool, optional): Whether to print verbose messages. Defaults to True.
-   
+
     """
     import glob
     import os
@@ -67,10 +67,14 @@ def download_wbt(linux_musl=False, reset=False, verbose=True):
         "Darwin": "https://www.whiteboxgeo.com/WBT_Darwin/WhiteboxTools_darwin_amd64.zip",
         "Darwin-arm": "https://www.whiteboxgeo.com/WBT_Darwin/WhiteboxTools_darwin_m_series.zip",
         "Linux": "https://www.whiteboxgeo.com/WBT_Linux/WhiteboxTools_linux_amd64.zip",
-        "Linux-musl": "https://www.whiteboxgeo.com/WBT_Linux/WhiteboxTools_linux_musl.zip"
+        "Linux-musl": "https://www.whiteboxgeo.com/WBT_Linux/WhiteboxTools_linux_musl.zip",
     }
 
-    if linux_musl or ('google.colab' in sys.modules) or (os.environ.get('WBT_LINUX', False) == "MUSL"):
+    if (
+        linux_musl
+        or ("google.colab" in sys.modules)
+        or (os.environ.get("WBT_LINUX", False) == "MUSL")
+    ):
         links["Linux"] = links["Linux-musl"]
 
     if platform.system() == "Darwin" and platform.processor() == "arm":
@@ -153,11 +157,20 @@ def download_wbt(linux_musl=False, reset=False, verbose=True):
             if verbose:
                 print("WhiteboxTools package directory: {}".format(pkg_dir))
 
+            zip_dir = zip_name.split(".")[0]
+            src_dir = os.path.join(zip_dir, "WBT")
+
+            if os.path.exists(src_dir):
+                shutil.move(src_dir, pkg_dir)
+
             if os.path.exists(new_img_dir):
                 shutil.rmtree(new_img_dir)
             if os.path.exists(new_plugin_dir):
                 shutil.rmtree(new_plugin_dir)
 
+            if os.path.exists(zip_dir):
+                shutil.rmtree(zip_dir)
+
             shutil.copytree(init_img_dir, new_img_dir)
             shutil.copytree(init_plugin_dir, new_plugin_dir)
 
@@ -166,13 +179,13 @@ def download_wbt(linux_musl=False, reset=False, verbose=True):
                 exe_ext = ".exe"
             exe_name = "whitebox_tools{}".format(exe_ext)
             exe_path = os.path.join(exe_dir, exe_name)
-            runner_name = "whitebox_runner{}".format(exe_ext)
-            runner_path = os.path.join(exe_dir, runner_name)
+            # runner_name = "whitebox_runner{}".format(exe_ext)
+            # runner_path = os.path.join(exe_dir, runner_name)
 
             # grant executable permission
             if platform.system() != "Windows":
                 os.system("chmod 755 " + exe_path)
-                os.system("chmod 755 " + runner_path)
+                # os.system("chmod 755 " + runner_path)
             plugins = list(
                 set(glob.glob(os.path.join(new_plugin_dir, "*")))
                 - set(glob.glob(os.path.join(new_plugin_dir, "*.json")))
@@ -183,8 +196,8 @@ def download_wbt(linux_musl=False, reset=False, verbose=True):
 
             exe_path_new = os.path.join(pkg_dir, exe_name)
             shutil.copy(exe_path, exe_path_new)
-            runner_path_new = os.path.join(pkg_dir, runner_name)
-            shutil.copy(runner_path, runner_path_new)
+            # runner_path_new = os.path.join(pkg_dir, runner_name)
+            # shutil.copy(runner_path, runner_path_new)
 
             try:
                 os.remove(zip_name)
@@ -196,7 +209,9 @@ def download_wbt(linux_musl=False, reset=False, verbose=True):
             # # downloads the binary that is compatible with Google Colab.
             if "google.colab" in sys.modules:
                 # url = "https://github.com/giswqs/whitebox-bin/raw/master/WhiteboxTools_ubuntu_18.04.zip"
-                url = "https://www.whiteboxgeo.com/WBT_Linux/WhiteboxTools_linux_musl.zip"
+                url = (
+                    "https://www.whiteboxgeo.com/WBT_Linux/WhiteboxTools_linux_musl.zip"
+                )
 
                 zip_name = os.path.join(pkg_dir, os.path.basename(url))
                 try:
@@ -3048,7 +3063,7 @@ Okay, that's it for now.
 
         dem -- Name of the input raster image file. 
         output -- Name of the output vector lines file. 
-        threshold -- Threshold value (0 - infinity but typcially 1 to 5 works well). 
+        threshold -- Threshold value (0 - infinity but typically 1 to 5 works well). 
         min_length -- Minimum line length, in grid cells. 
         callback -- Custom function for handling tool text outputs.
         """
@@ -9190,11 +9205,11 @@ Okay, that's it for now.
         iterations -- Maximum iterations (if stopping criteria not reached). 
         seed -- Seed for RNG consistency. 
         prob -- Probability of random resample or resampling worst strata between [0,1]. 
-        threshold -- Objective function values below the theshold stop the resampling iterations. 
+        threshold -- Objective function values below the threshold stop the resampling iterations. 
         temp -- Initial annealing temperature between [0,1]. 
         temp_decay -- Annealing temperature decay proportion between [0,1]. Reduce temperature by this proportion each annealing cycle. 
         cycle -- Number of iterations before decaying annealing temperature. 
-        average -- Weight the continuous objective funtion by the 1/N contributing strata. 
+        average -- Weight the continuous objective function by the 1/N contributing strata. 
         callback -- Custom function for handling tool text outputs.
         """
         args = []
@@ -10897,7 +10912,7 @@ Okay, that's it for now.
         if zero_background: args.append("--zero_background")
         return self.run_tool('tributary_identifier', args, callback) # returns 1 if error
 
-    def vector_stream_network_analysis(self, streams, dem, output, cutting_height=10.0, snap=0.1, callback=None):
+    def vector_stream_network_analysis(self, streams, output, snap=0.1, callback=None):
         """This tool performs common stream network analysis operations on an input vector stream file.
 
         Keyword arguments:


Test patch:
diff --git a/tests/test_whitebox.py b/tests/test_whitebox.py
index 7b2df62..dd66e87 100644
--- a/tests/test_whitebox.py
+++ b/tests/test_whitebox.py
@@ -25,12 +25,12 @@ class TestWhitebox(unittest.TestCase):
     def test_000_something(self):
         """Test something."""
 
-    def test_command_line_interface(self):
-        """Test the CLI."""
-        runner = CliRunner()
-        result = runner.invoke(cli.main)
-        assert result.exit_code == 0
-        assert 'whitebox.cli.main' in result.output
-        help_result = runner.invoke(cli.main, ['--help'])
-        assert help_result.exit_code == 0
-        assert '--help  Show this message and exit.' in help_result.output
+    # def test_command_line_interface(self):
+    #     """Test the CLI."""
+    #     runner = CliRunner()
+    #     result = runner.invoke(cli.main)
+    #     assert result.exit_code == 0
+    #     assert 'whitebox.cli.main' in result.output
+    #     help_result = runner.invoke(cli.main, ['--help'])
+    #     assert help_result.exit_code == 0
+    #     assert '--help  Show this message and exit.' in help_result.output